3.2. Performance Evaluation Metrics
Results of fire detection are assessed and compared using two performance accuracies, user’s accuracy (a measure of the commission error) and producer’s accuracy (a measure of the omission error), given by
        
        where the 
 stands for the cardinality of a set 
S, ‘
’ is the number of fire pixels reported by the reference data product, namely MODIS MOD14/MYD14 and EUMETSAT FIR, ‘reported 
fire pixels’ is the number of pixels reported by the designed (proposed) fire detection algorithm (per analysis method), and ‘correctly reported 
fire pixels’ is the number of 
fire pixels according to the reference data products correctly reported (classified) by the designed algorithms. The performance of the fire detection method is also assessed through two performance curves, a graph of user’s accuracy against producer’s accuracy (also known as Precision-Recall curve) and ROC curve. The Precision-Recall curve is sensitive to both class skews and varying unequal/asymmetric misclassification (error) costs (i.e., when the cost of misclassifying a sample from one class is much higher than the cost of misclassifying a sample from the other class) while the ROC curve is insensitive to both [
44], and the class skews are evident in a fire detection application where the true instances of 
non-fire class occur much more often than the true instances of 
fire class. However, due to a high number of 
non-fire pixels compared to 
fire pixels, the Precision-Recall curves visually expose clearer the difference between the detection methods, in a given FAR range of interest, that is not noticeable in the ROC space  [
45]. The pixel reported as 
fire pixel by the validation fire products but is reported by the proposed fire detection method as a 
non-fire pixel or or has 
 (expected) background brightness temperature or has 
 observation (i.e., instant of completely missing data [missed pixels]), is counted as a missed detection. If an MSG-SEVIRI pixel containing more than one MODIS fire pixel is not reported as a 
fire pixel by a proposed fire detection algorithm, then all MODIS MOD14/MYD14 fire pixels belonging to the MSG pixel are missed (i.e., counted only under ‘
fire pixels’), while if reported as a 
fire pixel, all MODIS fire pixels belonging to the MSG pixel are detected (i.e., counted under ‘correctly reported 
fire pixels’ and ‘
fire pixels’). The process of deriving the best admissible CFAR at which a fire detection system can operate can be conducted during a phase that can be referred to as the ‘detection calibration phase’ using an available validation set [reference data] (in this study, the validation set is made solely of MODIS MOD14/MYD14 and EUMETSAT FIR reports). The operational CFAR (or the best CFAR for a detection method according to the reference data) can be determined based on different criteria. This study determines the operational CFAR, as a FAR, that minimizes an error function 
J, given by
        
        where 
 and 
 are the prior probability that a 
fire condition and a 
non-fire condition, respectively, will occur – they represent the proportion of correct 
fire or 
non-fire pixels to the sum of all pixels, and can be referred to as 
fire or 
non-fire true class distribution, respectively. 
 and 
 are representations of class skews. 
 and 
 are the misclassification cost of omission and cost of commission, respectively. 
 is the measured FAR while the variable 
 is the design FAR (known also as the nominal or the estimated FAR). Noting that the detection cost function 
 given by Equation (
7) assumes that the cost of correct classifications is zero, and 
 accounts of class skew [
46].
  3.3. Fire Detection Performance
User’s accuracy (
U) and producer’s accuracy (
P) computed per estimated FAR are shown in 
Figure 1a and 
Figure 1b, respectively. The two accuracies are derived from the fire results of the proposed fire detection algorithms in the study area for the period of 31 July 2007 to 4 August 2007. The validation reference data are made of a combination of EUMETSAT FIR, and MODIS MOD14/MYD14 reports. MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time (detection time is 
 ahead of Level 1.5 observation time). 
 performs better than other detection methods in terms of 
U (gives less commission errors) except at the low estimated FARs. However, the low estimated FAR region was not considered during the design of the CFAR detector (the estimation of FAR starts from 
 to 
 as indicated in 
Section 2.2). For the estimated FARs in the low FAR region (i.e., high threshold values), 
 can produce a lot of false alarms given that the assimilation window of 4D-Var is more likely to contain more missed fire samples. 
 performs worse than both 
 and 
 in terms of 
U. At high estimated FAR, all methods give similar results in terms of 
U. Overall, 
 performs better than other methods in terms of 
P (low omission error or high probability of detection), and at some estimated FARs, it performs more like 
.
Figure 2a illustrates the Precision-Recall curve (i.e., 
U vs. 
P) for a single test set in which all data for period 31 July 2007–4 August 2007 are included, with validation reference data comprising of the combination of MODIS MOD14/MYD14 and EUMETSAT FIR products, and MODIS detection time rounded to the nearest SEVIRI detection time. The performance of the detection methods when any of the three analysis is implemented is fairly modest (no method with a Precision-Recall above the shown diagonal/random classifier) and is not good due to a high number of false alarms according to the reference data products. 
 shows a better performance compared to other detection methods, as 
 dominates in Precision-Recall space, it does also dominate in ROC space as shown in ROC curves in 
Figure 2b.
 A single operational FAR, when the analysis is implemented with any of the three methods considered in this study, is derived using the error (cost) function 
 given by Equation (
7). This error function considers both commission and omission errors and is a function of the estimated FAR. It is assumed in this study that 
 and the error function simplifies then to 
. 
Figure 2c depicts a resulting 
 for a single test set made of data spanning the period 31 July 2007–4 August 2007 with a reference dataset consisting of a combination of MODIS MOD14/MYD14 and EUMETSAT FIR products, and MODIS detection time is rounded to the nearest SEVIRI detection time. The optimal design FAR point is found among discrete points ranging from 
 to 
 (with a base-10 logarithmic step of 
). 
 gives the lowest error compared with other detection methods, and its optimal point 
 FAR gives the operational FAR. The 
 FAR point is closer to the optima of 
 and 
, and the error difference (
) from 
 point to the optima of 
 and 
 is 
 and 
, respectively.
When each detection not confirmed by EUMETSAT FIR product is confirmed by any MODIS MOD14/MYD14 report in 
 before or after the detection time, the number of reported false alarms according to the validation reference data (the combination of MODIS MOD14/MYD14 and EUMETSAT FIR ) is reduced for all detection methods as shown in 
Figure 3, and the reported overall performance is then increased. In this case under consideration, fire events confirmed by the same MODIS MOD14/MYD14 report are counted as one ‘correctly reported fire’ during computation of both 
P and 
U, and the number of ‘fire pixels’ is the sum of the number of EUMETSAT FIR reports and the number found by subtracting the number of MODIS MOD14/MYD14 fire reports to the number of times one or more EUMETSAT FIR fire reports are found in the interval of 
 around a MODIS MOD14/MYD14 report (i.e., 1 to 3 fire events). The results depict an instance the validation products are used to validate a whole fire course on a pixel location and not just a fire event at a given inspection time 
k on a given pixel location.
Results of a comparison between 
, 
 and 
 detection methods at 
 FAR when only MODIS MOD14/MYD14 fire product is taken in the validation reference data (and MODIS detection time rounded to the nearest SEVIRI detection time) are depicted in 
Table 3. The number of true positives, according to a MODIS MOD14/MYD14 product, is presented (the number of true positives in 
Table 3, is presented in terms of the number of MODIS fire pixels). From the results presented in 
Table 3, we can conclude that 
 performs better than other detection methods (throughout the 5 days) in terms of detection, according to MODIS MOD14/MYD14 product. 
 and 
 at 
 FAR achieve a true positive rate of 
 and 
, respectively, of fire events reported in MODIS MOD14/MYD14 product. A moderate portion of MODIS MOD14/MYD14 reports is detected by 
 and 
 methods, and the accuracy is restricted given that small and cool fires might not be detected due to the low spatial resolution of the SEVIRI compared to the MODIS resolution.
Table 4 summarizes results for 
 (under ‘4D-Var-CD’ columns), 
 (under ‘EnKF-CD’ columns) and 
 (under ‘SIR-CD’ columns) at 
 design FAR with reference data being both EUMETSAT FIR and MODIS MOD14/MYD14 products (‘FIR’ column presents the number of EUMETSAT FIR reports and ‘
’ column portrays the number of MODIS MOD14/MYD14 reports not reported by EUMETSAT FIR ). For each of 
, 
, and 
 detection methods, 
Table 4 lists the number of true positives, according to EUMETSAT FIR product (in ‘
’ column). Among the remaining false positives of the proposed detection method, according to EUMETSAT FIR product, true positives according to MODIS MOD14/MYD14 are extracted (listed in ‘
’ column), and false positives according to both EUMETSAT FIR and MODIS MOD14/MYD14 are presented in ‘
’ column. A significant portion of false alarms, according to EUMETSAT FIR, are reduced when MODIS MOD14/MYD14 product is introduced in the validation reference dataset. Comparing true positive results presented in 
Table 3 for MODIS MOD14/MYD14 and true positive results presented in 
Table 4, the proposed fire detection algorithms detect MODIS MOD14/MYD14 fire reports at a lesser extent than they detect EUMETSAT FIR fire reports. Because, as mentioned above, of the small and cool fires that might not be detected due to the low spatial resolution of SEVIRI compared to MODIS. The degree of omission of EUMETSAT FIR fire reports by the proposed algorithms mainly depends on the level of the design FAR, and in this section, the design FAR is taken to be 
. However, the omission of EUMETSAT FIR reports at a given design FAR highly materializes while reporting fire events during times of extreme fires, and omission error arises in case the fire temperature dynamics at certain times are not far from the expected background temperature dynamics. Results of 
Table 4 show that 
 and 
 perform relatively similar and better than 
 detection method. 
 detects EUMETSAT FIR reports at 
, and detects the combination of EUMETSAT FIR fire events and MODIS MOD14/MYD14 fire events not reported by EUMETSAT FIR at 
. 
 detects EUMETSAT FIR reports at 
, and detects the combination of EUMETSAT FIR fire events and MODIS MOD14/MYD14 fire events not reported by EUMETSAT FIR at 
. All proposed detection methods report a significant number of false alarms according to MODIS MOD14/MYD14 and EUMETSAT FIR, and detect EUMETSAT FIR reports relatively well.
 Figure 4a,b present number of fire events at each time of a day (where times are expressed in terms of local solar time) for MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. The values of producer’s accuracy (or true positive rate) of the proposed detection algorithms at each time of a day are presented in 
Figure 4c,d for cases the validation reference dataset is made of MODIS MOD14/MYD14 product only and EUMETSAT FIR product only, respectively. Fires recorded by EUMETSAT FIR are significantly detected by 
 and 
 detection methods at 
 FAR with 
 outperforming other methods in detection of MODIS night fires. Their detection performance is reduced from around 
(i.e., the time of the maximum thermal radiation) to around 
(onset of the nighttime temperature decay) with respect to EUMETSAT FIR and MODIS MOD14/MYD14 results, and such performance can be viewed to be more associated with a high number of fire events reported by both MODIS MOD14/MYD14 and EUMETSAT FIR products in the time range. During time of a high number of fire events, a proposed algorithm can miss a fire event reported by the fire products but can report a fire before or after the fire product reports. Furthermore, the observed temperature in MSG 
spectral channel of an MSG pixel identified by MODIS to be affected by fire, might not deviate significantly from its expected background temperature. 
 has an overall low detection, especially during the nighttime—the time of high fluctuations in brightness temperatures due to different coupled land–atmosphere phenomena, such as orographic forcing effect. In the afternoon, its low detection is more explained by the DTC linearization from around 
 to around 
(or likewise, based on the approximate observation operator given by Equation (S8)). The detection rate of MODIS MOD14/MYD14 report is low compared to the detection rate of EUMETSAT FIR reports, and 
 and 
 perform better in detecting MODIS MOD14/MYD14 reports compared to 
 at the 4 MODIS overpass times (
 is better than all detection methods during the night). Based on the MODIS MOD14/MYD14 report, all detection methods certainly detect both daytime and nighttime fires, with 
 to a lower extent. Noting that MODIS detection time is rounded to the nearest SEVIRI detection time, and the number of ‘fire pixels’ is counted in terms of MODIS pixels to determine the true positive rate (
P), according to MODIS MOD14/MYD14 product.
 The empirical distribution of MSG 
brightness temperatures for MODIS MOD14/MYD14 reported fire pixels and EUMETSAT FIR reported fire pixels are shown in 
Figure 5a,b, respectively. MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time. For MODIS MOD14/MYD14 reported fire pixels, the temperature value of an MSG pixel containing the centers of a number of MODIS fire pixels at a given time step 
k is counted as many times as the number of the centers of the MODIS pixels. For fire detection results of 
 method, producer’s accuracy for each MSG brightness temperature value is shown in 
Figure 5c,d when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. 
 is able to detect cooler fire compared to EUMETSAT FIR, down to approximately a brightness temperature value of 
 (with 
), as shown in 
Figure 5c when only MODIS MOD14/MYD14 product is accommodated in the validation set. Some very hot fires are missed by 
, especially at the SEVIRI’s saturation point (which can be attributed to the linear approximation of the observation function and the data assimilation window that does not contain only clean samples). For fire detection results of the 
 method, producer’s accuracy for each brightness temperature value is shown in 
Figure 5g,h when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. 
 reports cold fires more frequently than 
, and can detect fires with brightness temperatures below 
 (with 
) as shown in 
Figure 5g. For fire detection results of 
 method, producer’s accuracy for each brightness temperature value is shown in 
Figure 5e,f when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. Similar to 
 and 
 methods, the 
 method is able to detect cooler fire compared to EUMETSAT FIR, even for fires with a brightness temperature value below 
 (with 
) as shown in 
Figure 5e when only the MODIS MOD14/MYD14 product makes the validation set.
An example of MSG 
images at three MSG timestamps on 3 August 2007 is illustrated in 
Table 5 to portray detection reports of 
, 
, and 
 against both MODIS MOD14/MYD14 (MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time) and EUMETSAT FIR reports. In addition, the detection reports of MODIS MOD14/MYD14 and EUMETSAT FIR at the three timestamps are provided to present and describe MODIS MOD14/MYD14 and EUMETSAT FIR products. The table provides Level 1.5 observation time (expressed in Coordinated Universal Time (UTC)) in ‘Time’ column, the pseudocolor versions of the MSG 
images in ‘Pseudocolor image’ column to better visualize the brightness temperature distribution in the images, and MODIS MOD14/MYD14 and EUMETSAT FIR fire reports overlaid on the MSG 
images in the ‘MOD14/MYD14 & FIR’ column. A great number of MODIS MOD14/MYD14 reports are not reported by EUMETSAT FIR. Essentially, since in this study, we assume that a MODIS pixel belongs to an MSG pixel into which its center falls, if a burning part of the MODIS pixel location is in a different MSG pixel, some reports would be missed by a fire detection method that uses MSG-SEVIRI data. The table also presents detection reports of 
, 
 and 
 methods on the three images, overlaid on the corresponding MSG 
images and placed under ‘4D-Var’, ‘EnKF’ and ‘SIR’ columns, respectively. 
 and 
 detect all EUMETSAT FIR reports while 
 missed, at 08:12 a.m. (corresponds to the detection time 08:00 a.m.) and at 12:42 p.m. (corresponds to the detection time 12:30 p.m.), 4 fire events and 22 fire events, respectively. 
 and 
 detect a great number of MODIS MOD14/MYD14 reports with 
 detecting less at all the three timestamps consistently. The results of 
 contain a great number of false alarms according to MODIS MOD14/MYD14 and EUMETSAT FIR products, compared to both 
 and 
, and 
 has less false alarms.