Assessment of the Representativeness of MODIS Aerosol Optical Depth Products at Different Temporal Scales Using Global AERONET Measurements

Abstract

Assessments of long-term changes of air quality and global radiative forcing at a large scale heavily rely on satellite aerosol optical depth (AOD) datasets, particularly their temporal binning products. Although some attempts focusing on the validation of long-term satellite AOD have been conducted, there is still a lack of comprehensive quantification and understanding of the representativeness of satellite AOD at different temporal binning scales. Here, we evaluated the performances of the Moderate Resolution Imaging Spectroradiometer (MODIS) AOD products at various temporal scales by comparing the MODIS AOD datasets from both the Terra and Aqua satellites with the entire global AErosol RObotic NETwork (AERONET) observation archive between 2000 and 2017. The uncertainty levels of the MODIS hourly and daily AOD products were similarly high, indicating that MODIS AOD retrievals could be used to represent daily aerosol conditions. The MODIS data showed the reduced quality when integrated from the daily to monthly scale, where the relative mean bias (RMB) changed from 1.09 to 1.21 for MODIS Terra and from 1.04 to 1.17 for MODIS Aqua, respectively. The limitation of valid data availability within a month appeared to be the primary reason for the increased uncertainties in the monthly binning products, and the monthly data associated uncertainties could be reduced when the number of valid AOD retrievals reached 15 times in one month. At all three temporal scales, the uncertainty levels of satellite AOD products decreased with increasing AOD values. The results of this study could provide crucial information for satellite AOD users to better understand the reliability of different temporal AOD binning products and associated uncertainties in their derived long-term trends.

1. Introduction

Atmospheric aerosols impact radiative energy transfer, serving as one of the most important climatic forcing factors [1] and contribute to climate change [2]. As a direct impact, atmospheric aerosols can scatter radiation back to space and therefore reduce the amount of radiation received by the Earth; and as an indirect impact, they can alter cloud properties and thus modulate the amount of radiation scattered and absorbed by clouds [3]. These direct and indirect impacts of aerosols are closely linked to aerosol optical properties such as absorption and scattering coefficients [4,5]. Aerosol optical depth (AOD) is one of the most important parameters for quantifying aerosol loading [5,6], which has been widely used in regional air quality-related applications [7,8,9], aerosol radiative forcing estimations [10], regional and global climate change modeling [11], and public health studies [12]. However, it is often challenging to accurately trace the changes of the aerosol related properties (such as AOD) due to their high variability in time and space, making it difficult to precisely estimate the aerosol radiative forcing and thus enhancing the uncertainties in modeling Earth’s climate system [2].

Aerosol optical depth measurements are primarily from two sources: ground-based measurements and space-borne AOD retrievals. The most well-known ground-based network is the AErosol RObotic NETwork (AERONET, [13]), which was first established 25 years ago by National Aeronautics and Space Administration (NASA) and the PHOtométrie pour le Traitement Opérationnel de Normalisation Satellitaire (PHOTONS; Univ. of Lille 1, CNES, and CNRS-INSU) and provides publicly-available datasets of aerosol properties. More than 1000 AERONET sites are distributed across different regions around the world, with standardized instruments (Cimel CE-318), calibrations and processing schemes. The ground-based network can provide accurate spectral AODs (at an uncertainty level of ~0.01–0.02, [13]) and associated properties [7,14]. However, the data from AERONET sites still seem to be rather limited in terms of capturing the extensive and continuous spatiotemporal heterogeneities and complex nature of global aerosol distributions [15,16].

Satellite remote sensing could potentially overcome the limitations due to its advantages in extensive data coverage and frequent observations. State-of-the-art algorithms such as Deep Blue (DB) [17], Dark Target (DT) [18], Multi-Angle Implementation of Atmospheric Correction (MAIAC) [19], and continuously updated algorithms in the Aerosol_cci project of the Europe Space Agency [20] have been successfully implemented to retrieve AODs from remotely sensed imagery. These algorithms have been tested in past and current satellite missions including the Advanced Very High Resolution Radiometer (AVHRR) [21,22,23,24], the Total Ozone Mapping Spectrometer (TOMS) [25], the Sea-Viewing Wide Field-of-View Sensor (SeaWiFS) [26,27], the Multiangle Imaging Spectroradiometer [28], the Moderate Resolution Imaging Spectroradiometer (MODIS) [29,30,31], the Visible Infrared Imaging Radiometer Suite (VIIRS) [32,33], and others. Aerosol products from these satellite instruments have also been operationally distributed and readily available for public use. Various studies have been conducted based on these satellite AOD products because of their unparalleled advantages in data acquisition. For example, AOD is one of the most widely used parameters in terms of estimating air quality (PM2.5, PM10, etc.) [33,34,35,36] and global radiative forcing [4,10,37,38,39,40].

While the satellite retrieved AOD products outperform ground-based AERONET in terms of data coverage, satellite retrievals still exhibit considerable uncertainties given the continuous efforts in the advancements of remote sensing algorithms [18,41,42]. Regional- and global-based evaluations of these remote sensing-based AOD products with ground observations have identified some major error sources [43,44,45,46,47] including inaccurate input of surface reflectance and questionable assumptions of aerosol types [6,48]. Indeed, the current community accepts the uncertainty-level in satellite AOD retrievals over land as ±|0.05 + 0.15 $\times$ AOD| [18,30,49], and over 66% of satellite-AERONET AOD pairs are within the expected interval. Note that these validation attempts were primarily based on the comparisons between the AOD retrievals from satellite snapshots and the ground-based reference (generally represented by AERONET AOD) measured at the same (or at least a very similar) time and location [9,44,50,51], and demonstrated that these AOD products are reliable in representing aerosol conditions at the overpass time.

Two additional potential uncertainty sources should also be noticed when users attempt to aggregate snapshot level-2 satellite AOD data into daily, monthly, or even longer temporal aggregated binning products. Specifically,

(1)

Polar orbiters acquire images at a fixed time of a day (for example, the local overpass times for MODIS Aqua and Terra are 1:30 pm and 10:30 am, respectively), but atmospheric aerosol loading and its associated properties could vary substantially over a very short time scale due to various emission sources, small scale meteorology effects as well as complex aerosol compositions and atmospheric processing [15,52]. Therefore, the satellite AOD retrievals and associated products may not well represent the daily aerosol conditions in theory. To address this issue, Kaufman [53] used AERONET observations to compare the daily means and mean values averaged over the MODIS Aqua/Terra time windows (i.e., synthetic MODIS data using AERONET), and they concluded that the AOD measurements at the satellite overpass time could represent daily AOD averages within an error level of 5%. While this former effort was conducted before the launch of the two MODIS instruments when real satellite data were not available, the results were only based on simulations with AERONET data. As such, the expected errors (i.e., 5%) could be different from the real satellite observations since the uncertainties in the satellite AOD retrieval algorithms were not considered. Additionally, data gaps may also exist in the AERONET AOD measurement within a day due to unfavorable weather conditions, instrument malfunctions, and many other factors [51]. However, the associated uncertainties to the daily mean AERONET AODs were not quantified [53].

(2)

Optical remote sensing suffers from cloud contamination, which results in a rather limited number of available high-quality data and remarkable observation gaps in the dataset. Indeed, a global cloud cover statistic using MODIS cloud mask products revealed that the global mean cloud coverage was 67% [54] and could vary substantially over different seasons and locations, which could significantly reduce the number of valid AOD retrievals. Moreover, this problem could be further exacerbated due to the associated issues of sub-pixel clouds, thin clouds, and their adjacency effects [55] as well as biases in retrievals [50]. Additionally, high AODs (e.g., smoke plumes, dust storms, biomass burning days) scenarios are often misinterpreted as clouds, where the signals tend to be saturated, leading to upper limitation of satellite retrievals [56,57]. This kind of misclassification could not only decrease the frequency of valid instantaneous remote sensed retrievals, especially for large values, but also further underestimate long-term satellite composites. That is, often only a few days of daily satellite AOD images with relatively lower observations were used to compose monthly (or other temporal binning) products. It remains unclear whether and to what extent these composed monthly, seasonal, or annual aerosol products could represent actual global or regional levels of aerosol concentrations. Although a recent attempt by Fan et al. (2018) was performed to examine whether the AOD trends of 53 sites detected by monthly AERONET observations could be reproduced through MODIS AOD data, the fundamental question of how satellite observation gaps could impact the reliability of monthly satellite products has never been analyzed. Similarly, Yoon et al. (2011) realized the uncertainty caused by data gaps and discarded monthly AOD data when less than five observations per month were available. However, how the uncertainties could be impacted by the changing numbers of valid data within a month has never been investigated.

Apparently, these uncertainties in satellite AOD products could further be propagated, leading to problematic estimates of the trends for changing aerosol loading and/or aerosol-induced changes in circulation, temperature, global aerosol radiative forcing, PM2.5 modeling, etc. For example, numerical simulations by Wang [40] indicated that an uncertainty level of 20% for AOD could result in a change of 15.4% for aerosol direct radiative forcing (ADRF). It is thus desirable to determine the representativeness of satellite AOD products at different temporal scales through a comparison with ground truth AODs. This could help not only understand the potential uncertainties caused by the diurnal changes in aerosol properties and the data gaps but also provide critical information to understand how the representativeness could impact the trends in aerosol loading and other related studies (such as global aerosol radiative forcing, PM2.5 modeling, etc.), when temporal binning data are required [4,8,9,58,59].

With rigorous radiometric calibrations and algorithm developments/refinements, the MODIS instruments onboard Terra and Aqua provide long-term (2000–present for Terra and 2002–present for Aqua) global satellite AOD observations. Meanwhile, the expanded network and continuous observations from the globally distributed AERONET sites have provided accurate AOD measurements over the past 25 years. Such datasets allow for comprehensive assessments of the temporal distributions of satellite AOD products with AERONENT observations. The current study was thus designed with the following objectives:

(1)

Quantify the uncertainty levels of satellite AOD products in various temporal domains based on global long-term concurrent measurements between the MODIS and AERONET observations; and

(2)

Understand the potential factors that could affect the temporal representativeness of the satellite AOD products, and discuss the future efforts that could be used to improve the validity of AOD temporal binning products and their derived long-term trends.

The rest of this paper is arranged as follows. The datasets and methods are first introduced in Section 2, followed by the assessments of satellite AOD retrievals at different temporal scales in Section 3. Then, the factors influencing the representativeness levels of temporal binning products and the implications of this work are discussed in Section 4.

2. Materials and Methods

2.1. MODIS AOD Data

The MODIS instruments onboard the Terra and Aqua satellites have provided global, near-daily AOD retrievals since 2000 (Terra) and 2002 (Aqua), where the local overpass times are 10:30 for Terra and 13:30 for Aqua. The most recent updated MODIS Level 2 AOD products were utilized in the current study (Collection 6.1 (C6.1) with MOD04_L2 for Terra and MYD04_L2 for Aqua, with a spatial resolution of 10 km (at nadir)), which were downloaded from the NASA Goddard Space Flight Center (GSFC) at https://ladsweb.modaps.eosdis.nasa.gov/. Three types of datasets (i.e., Dark Target (DT), Deep Blue (DB), and merged DT and DB product (DTB)) are included in each file, which were derived from different aerosol retrieving algorithms: the DT, DB, and specifically combined algorithms. The classical DT algorithm [18] was designed for AOD retrievals over relatively “dark” targets (lower surface reflectance) in the visible channel (e.g., vegetation or water bodies). In contrast, the DB algorithm [17] was originally designed to retrieve AOD values over relatively “bright” surfaces such as arid/semiarid areas or deserts. Both retrieval methods have been gradually updated with refinements and improvements [60,61]. The C6.1 aerosol product used in our study was operationally generated based on the newest DT and enhanced DB algorithms [15,62]. To produce more extensive high-quality data coverage, the DTB dataset was further generated by integrating the DT- and DB-based AOD retrievals using different schemes over various different land surfaces. The types of land surfaces were classified by a gridded vegetation index (e.g., the normalized difference vegetation index (NDVI)) [18]. Therefore, the DTB-based dataset was selected in this study to examine the representativeness of the MODIS aerosol products at different temporal scales. Notably, we used all quality flagged satellite AOD retrievals (Quality Flag (QF) = 1, 2, 3, indicating marginal, good, and best “confidence”, respectively) rather than performing a screening process (i.e., QF = 2 and 3, or QF = 3), in order to increase the data volume of ground-based and satellite-retrieved match-ups at hourly scale and at longer temporal bins. Indeed, sensitivity analysis with different filtering criteria (data with and without QF = 1) revealed insignificant changes in the resulting uncertainty levels when 10 AERONET sites with the greatest number of match-ups were used (see Table S1). Nevertheless, negative AOD values were excluded in this study.

2.2. AERONET AOD Data

AERONET Version 3 Level 2.0 AOD products (cloud-screened and quality-assured) obtained from the NASA GSFC website (https://aeronet.gsfc.nasa.gov/) were used as reference datasets to assess the representativeness of the MODIS AOD measurements. It is a worldwide ground-based remote sensing aerosol network [13], where a CIMEL-318 Sun photometer is installed to measure the spectral solar irradiance and sky radiances to derive AOD [14,63,64,65,66]. The network provides frequent (every 3 or 15 min) and accurate (with an uncertainty of 0.01–0.02) AOD measurements under cloud-free conditions [13,14]. Indeed, AERONET Version 3 Level 2.0 AOD observations have been considered as one of the standard resources for the validation and bias-correction purposes of satellite-derived AOD due to their high-quality data, consistency in data processing, wide global coverage, and free data access [31,57]. However, constrained and contaminated by many aspects such as cloud and instrument maintenance, AERONET data also present data gaps at different temporal scales [51], which could bring uncertainties if users do not adopt any screening criteria when taking AERONET data as reference data. This issue was carefully examined in our study (see below). Indeed, the entire AERONET AOD data archive during the period of 2000–2017 was obtained, which included data from 1051 global sites (see locations in Figure 1).

To match the central wavelength (550 nm) of the MODIS AOD products, AERONET AODs at the same wavelength were estimated with AERONET AOD values between 440- and 675-nm and with the associated Angström exponent (α) [14] [42,67], which can be expressed as:

$$AO{D}_{550nm}=\mathrm{average}\left(AO{D}_{\lambda }{\left(\frac{550}{\lambda }\right)}^{-\mathsf{\alpha }}\right), \lambda \in \left[440,675\right]$$

(1)

It has been suggested that the uncertainty introduced by such a spectral interpolation scheme is negligible [68]. Hereafter, unless otherwise stated, the AOD represents the AOD values at 550 nm.

2.3. Determination of the Satellite and AERONET Match-Ups

To assess the representativeness of the MODIS AOD retrievals over different temporal scales (i.e., hourly, daily, and monthly) using ground-based AERONET observations, match-up pairs between these two independent measurements should first be determined before further quantitative agreement analysis. The workflow used to determine the concurrent observations between satellite and point-based ground measurements over various temporal scales is shown in Figure 2a.

Hourly concurrent observations were defined as when the AERONET data were collected within ±30 min of the MODIS overpass times. For each AERONET site, the hourly mean AOD measurements between the local times of 10:00–11:00 and 13:00–14:00 were estimated when at least one timely AOD data is available during these time windows (Figure 2b), and they were considered as hourly concurrent data for the MODIS Terra and Aqua satellites, respectively. Additionally, the MODIS AODs of a 3 $\times$ 3-pixel sampling window (~30 $\times$ 30 km) centered over each station were selected, and the mean value of the valid AOD data within this window was calculated (when at least half of the pixels are valid within the 3 $\times$ 3-pixel window) to represent the satellite-derived values [42,67,69].

The calculated daily MODIS AOD data are exactly the same as the hourly MODIS AOD data if only one satellite image is collected each day. However, for high-latitude regions where more than one MODIS image was available within a day (this is because that Terra and Aqua are polar orbiters), the daily satellite data were averaged by all the available datasets. Daily AERONET AOD data could be theoretically estimated as the mean of all AERONET measurements within a day. However, we only selected AERONET data with continuous hourly observations (see below) within a day to ensure the high accuracy of the reference dataset. Figure 2c shows the temporal distribution of the data volume for the 1051 global AERONET sites, where most of the data were collected between 08:00–18:00 local time for all four seasons. Specifically, to make full use of the ground-based data, time windows of 07:00–17:00 for spring, 06:00–18:00 for summer, 08:00–16:00 for autumn, and 09:00–16:00 for winter were selected (the seasonal differences between the Northern and Southern Hemispheres were considered), where the hourly mean measurements within these windows were averaged to represent the daily AERONET AOD. Note that although the nominal data acquisition frequency of AERONET is 3 or 15 min [66], the instrument may collect more data for certain hours than others and there are remarkable variations with higher values in the afternoon than the morning during the whole day (see an example shown in Figure 2b). To eliminate potential temporal sampling bias (i.e., data collected in certain hours instead of the whole day), we selected the data only when at least one valid measurement was available for each hour during the above time windows.

Monthly satellite AOD data were compiled when at least five days of MODIS AOD data were available in that month. This convention was adopted by previous studies [70,71], and monthly satellite AOD products from ≥5 daily AOD retrievals within a month were considered as valid. On the other hand, monthly AERONET AOD data were ideally estimated as the mean values of all the daily AODs to represent the reference dataset of the monthly AOD conditions. However, daily AERONET measurements may not be available for the entire month (see Figure 2d) due to instrument anomalies and maintenance, cloud disturbance, field-of-view obstructions, and many other reasons contributing to hourly gaps [66].

To obtain an optimal threshold to determine valid monthly AERONET AOD data (i.e., monthly reference data), months with all daily AOD available were first selected, collecting a total of 67 months from the global datasets. Then, the mean values for all daily AERONET AODs within each month were first estimated (denoted as A) and compared with the mean values that were calculated using a subset of the whole month daily AODs (denoted as B). The correlations (R², see Figure 2e) between A and B over these selected 67 months were examined against the number of days used to estimate B. The results show that the correlations increased first and then plateaued at a high level of >0.95 when the number of days reached 15, suggesting that the differences between A and B should be rather small when 15 days of data are available for estimating B. Therefore, AERONET monthly AOD was considered valid when the AERONET site had at least 15 AERONET daily AOD in that month. Note that the use of 15 days of data was a compromise for the lack of full month measurements from AERONET. We acknowledge that for each match-up at the monthly scale, the numbers of daily data between satellite (≥5) and AERONET measurements (≥15) were different, and the latter has been considered as the ground reference data for monthly AOD due to its accurate measurement (AOD uncertainty level of 0.01~0.02) and frequent observations (i.e., continuous hourly observations within a day for at least 15 days within a month).

To further examine whether the combination of the two MODIS missions could improve the agreements between the satellite and ground-based AERONET, AODs from MODIS Terra and Aqua were first averaged at hourly, daily, and monthly scales (i.e., combined MODIS hourly/daily/monthly AOD, respectively) and were then compared to the corresponding AERONET composites at the same scale (i.e., combined AERONET hourly/daily/monthly AOD).

2.4. Statistical Measures for the Representativeness Analysis

The consistency between the MODIS AOD and AERONET measurements at various temporal scales was used to gauge the representativeness of the satellite products to the ground-based reference data. Several statistical measures were used to quantify the representativeness. The first group of measures included the slope (a), intercept (b), and coefficient of determination (R²) of the linear regression (see Equation (2)). The closer the slope (a) to 1 means a better agreement of the satellite aerosol to AERONET measurements; the closer the intercept (b) to 0 indicates smaller biases caused by surface reflectance assumption. R² represents to what extent satellite AOD could capture the variability of ground reference data. The second group of measures are absolute error (bias, see Equation (3)), relative error (see Equation (4)), relative mean bias (RMB, see Equation (5)), standard deviation of absolute errors (or absolute uncertainty), standard deviation of the relative errors (or relative uncertainty, i.e., uncertainty). The absolute error and relative error reveal the absolute and relative differences between satellite retrievals and ground measurements. Relative mean bias (RMB) was used to reveal the overestimation (RMB > 1) or underestimation (RMB < 1) issues of the MODIS AOD. The absolute and relative uncertainty describe the dispersion of absolute and relative biases. Hereafter, unless otherwise stated, the uncertainty of AOD represents the relative uncertainty (i.e., standard deviation of the relative errors). The third group of measures is the percentage of matching data falling within an expected error (%within error envelope (EE), Equation (6)). The %within EE variable indicates the percentage of data points falling within a predefined expected EE ($\pm \left|0.05+0.15AO{D}_{aeronet}\right|$), and a value of >66% indicates satisfactory agreement from previous studies [18,30].

$$AO{D}_{satellite}=a\times AO{D}_{aeronet}+\mathrm{b}$$

(2)

$$Absolute Error=AO{D}_{satellite}-AO{D}_{aeronet}$$

(3)

$$Relative Error=(AO{D}_{satellite}-AO{D}_{aeronet})/AO{D}_{aeronet}$$

(4)

$$\mathrm{RMB}={\overline{AOD}}_{MODIS}/{\overline{AOD}}_{AERONET}$$

(5)

$$\%\mathrm{within} \mathrm{EE}\equiv \left(AO{D}_{satellite}\in AO{D}_{aeronet}\pm \left|0.05+0.15\times AO{D}_{aeronet}\right|\right)\%$$

(6)

In addition to the above conventionally used statistical metrics, the probability of satellite derived AODs with relative uncertainties larger than 100% (POU₁₀₀) was also calculated. This new-proposed metric is defined as

$${\mathrm{POU}}_{100}\equiv \left(AO{D}_{satellite}<{\mathrm{Threshold}}_{100}\right)\%$$

(7)

where Threshold₁₀₀ is the threshold when satellite AODs have an uncertainty level of 100%. We introduced this metric mainly with the consideration that satellite AODs with large uncertainty (i.e., 100%) may cause substantial errors when they are used in other applications, and global datasets used here could help to map the spatial distributions of POU₁₀₀ for general users. The use of POU₁₀₀ is to find where and what temporal scales the satellite AOD products could have high uncertainty levels (i.e., 100%) through visualizing spatiotemporal distributions of POU₁₀₀. As shown in Equation (7), the threshold when satellite AODs have high uncertainty (i.e., uncertainty >100% (denoted as Threshold₁₀₀)) must be determined before the POU₁₀₀ calculation. Previous validation has demonstrated that the absolute error of MODIS AOD is a monotonically increasing function of AOD values (e.g., $\mathrm{absolute} \mathrm{errors}=\pm \left|0.05+0.15\times AO{D}_{AERONET}\right|$ for MODIS Collection 6 AOD over land) [18], suggesting that Threshold₁₀₀ could possibly be obtained when plotting the uncertainty estimates against the AOD based on the potential relationship of relative errors and AOD (i.e., $\mathrm{a} \mathrm{similar} \mathrm{pattern} \mathrm{as} \mathrm{relative} \mathrm{errors}=\pm \left|\frac{0.05}{AO{D}_{AERONET}}+0.15\right|$).

3. Results

The representativeness and validity of the MODIS C6.1 AOD products were evaluated via comparison with the ground truths obtained from the AERONET observations. The assessments were conducted for each MODIS instrument (Terra or Aqua) using their combined observations at various temporal scales, and the comparison statistics are presented in both global- and site-level domains.

3.1. Overall Global Statistics

Global assessment is to reveal how well MODIS AOD retrievals could represent ground-based reference data (i.e., AERONET in this study) from an overall perspective, that is, to compare all available match-ups at all sites regardless of the surface type, meteorology, and aerosol type of the sites. Figure 3a–f compares the satellite and AERONET AOD observations at both hourly and daily scales for the 1051 global AERONET sites, where the match-up pairs (83202 in total) were the spatiotemporal intersection of the two temporal-scale matched pairs to eliminate data sampling-induced differences. In general, the MODIS AODs agreed well with the AERONET observations at hourly and daily scales, as demonstrated by the high values for R² (0.76–0.81) and %within EE [18,30], low intercepts (0.02–0.03), biases (~0.06), and absolute uncertainty (~10%) as well as closer proximities to 1 for the regression slopes and RMB (1.03–1.11) for either individual instruments or the sensor combinations. Taking AERONET as the reference data, the AOD estimates of MODIS Aqua outperformed those of Terra, especially regarding the RMB values. Specifically, the RMBs were 1.11 and 1.09 for MODIS Terra at hourly and daily scales, respectively, suggesting an overestimation of ~10% for the satellite-derived AOD data. In contrast, the overestimation problem for MODIS Aqua was less significant, where the RMB values were <1.04 at the two temporal scales. The overestimation problems for the MODIS AOD products appeared to be consistent with previous validation efforts [15]. Such errors are mostly due to surface reflectance underestimations and incorrect aerosol type selection in the satellite AOD retrieval algorithm [30], which could be partially caused by residual errors from imperfections in the cloud detection method [15]. The combination of the two could slightly improve the hourly and daily representativeness of the satellite observations, with increased R² and %within EE. The differences in all of the examined statistical measures were very similar between the daily and hourly comparisons, suggesting that the matched MODIS AOD retrievals could be used to represent daily aerosol conditions from an overall perspective. Note that large discrepancies between the satellite and ground measurements can be found for low AOD values (<0.1), indicating large relative uncertainties in satellite AOD retrievals under clean aerosol conditions.

Figure 3g–i demonstrates the agreements between the monthly satellite retrievals (for the Terra/Aqua/combined product) and AERONET monthly AOD observations. In general, the agreements between the satellite and ground-based data appeared worse for the monthly data than those for the hourly and daily measurements, which can be seen from the reduced R², slope, %within EE, and RMB (see specific values in Figure 3). This demonstrates that the monthly composite of the MODIS AOD products is not as representative as the daily products. When the data are integrated from daily to monthly scales, the RMB changed from 1.09 to 1.21 for MODIS Terra and from 1.04 to 1.17 for MODIS Aqua, respectively, indicating that the satellite overestimation problem could be deteriorated through temporal bins. Noticeably, the combination of the two MODIS instruments did not improve the performance of monthly satellite AOD product. The larger disparities could also be found at lower AOD conditions, which is apparently due to the propagated errors from the hourly and daily products.

3.2. Site-Specific Assessments

Site-level evaluation was used to illustrate the details of the local representativeness. The relative metrics at three different temporal scales for each AERONET site were also estimated and color coded in Figure 4, where the performances of the satellite-derived AOD products over different global regions could clearly be revealed. To ensure statistically meaningful results and to eliminate potential seasonal bias sampling (i.e., data are selected in one or two seasons), the calculations were only conducted at those sites where the data were distributed across four seasons. For simplicity, Figure 4 only plotted the results of MODIS Terra due to its longer time period and increased data availability as well as the very similar patterns for MODIS Terra and Aqua (similar statistical results and their distributions for Aqua not shown here). As shown in Figure 4, data from many sites were excluded due to the strict match-up selection criteria. In total, 299, 159, and 72 match-up sites were selected at the hourly, daily, and monthly scales, respectively. Note that although the number of monthly satellite-ground match-up pairs at each of the 72 sites (10–62 pairs, with the mean value of 22.7) was limited due to the strict data selection requirements, the monthly representativeness map is also demonstrated here as a reference.

The agreements of the MODIS AOD and AERONET measurements showed pronounced spatial heterogeneity around the world. Although certain different distributions could be found at monthly scale, which is possibly due to the insufficient statistical volume, three temporal binning products showed generally similar spatial patterns for all selected metrics. Specifically, the high performances of the satellite AOD products, in terms of both hourly and daily scales, were mainly found in Asia, Europe, central South America, and eastern North America, with high correlations (R² > 0.7), a closer proximity to 1 for the RMBs, and low uncertainties (<50%) (see Figure 4). On the other hand, data with poor correlations (R² < 0.5) and high uncertainties were primarily located in western North America, Africa, and almost the entire Southern Hemisphere. Such patterns are generally consistent with the continent-specific validations at hourly scale conducted by Wei [42], where sites with good agreements were found in Asia, eastern North America, northern South America, and Europe, and data collected in central Australia showed bad agreements. In contrast, %within EE showed distinctive patterns compared to the other statistical measures (Figure 4d1–d3), where the values for most sites (208/299, 121/159, and 49/72 at hourly, daily, and monthly scales, respectively) exceeded the satisfactory threshold (i.e., 66%; [18]). The different patterns between %within EE and the other statistical measures were primarily due to the relatively large offset of the EE envelope (0.05), where high values of %within EE were expected for regions clean of aerosols, in contrast to the large uncertainties for low AOD values (see Figure 3). As such, the value of %within EE does not appear to be a suitable indicator to measure the spatial representativeness of satellite AOD retrievals, especially at regions with relatively clean air conditions, although it has been widely used to gauge their accuracy levels [18,30,31,72].

3.3. Global Distributions of the POU₁₀₀

Figure 5 presents the absolute errors and relative differences (estimated with Equations (3) and (4)) between the satellite and AERONET AODs as well as the standard deviations of relative errors (i.e., relative uncertainty) as a function of the AERONET AOD values, where the data were plotted at three different temporal scales. To better visualize the statistical values, data were gridded into certain numbers of bins (100 for hourly, 50 for daily, and 50 for monthly) according to the AERONET-measured AODs (x-axis), and distribution of data within each bin is illustrated as box-whisker plots (Figure 5a-1,b-1,c-1,a-2,b-2,c-2). As in Figure 4, we only plot the data from MODIS Terra. Similar to the patterns in Figure 3 and many other previous studies [18,68], the absolute errors of instantaneous observations appear to be a monotonic function of the AOD, which could be further illustrated by the fitting equations in Figure 5a-1. When plotting relative uncertainties against AODs, the numbers also demonstrate a monotonic relationship (a steady decrease) with increasing AOD at all temporal scales (Figure 5a-2,b-2,c-2). The threshold AOD values with 100% of uncertainties (i.e., Threshold₁₀₀) were identified, and the same Threshold₁₀₀ value of 0.06 was found for the three temporal scales. In other words, an uncertainty level of >100% is expected when the satellite AOD retrieval is <0.06, regardless of the temporal aggregation scheme.

The probabilities of satellite-derived AODs with relative uncertainties larger than 100% (POU₁₀₀) were thus estimated using global MODIS-AERONET match-up pairs and are illustrated in Figure 6. Basically, the value of POU₁₀₀ could be interpreted as the percentage of AERONET measurements with an AOD <0.06 based on Equation (7). The histograms of the global POU₁₀₀ were also calculated and are plotted within each panel. While the histogram mode was 0–10% across the panels, considerable numbers of regions had a POU₁₀₀ value of >50% at all three temporal scales. For example, 43 out of the 299 (i.e., 14.4%) sites showed a POU₁₀₀ of >50% at the hourly scale, meaning that more than half of the satellite observations over these areas could have an uncertainty level of >100%. Moreover, the histograms show that the percentages for POU₁₀₀ >50% were 24.5% (39/159) at the daily scale and 20.8% (15/72) at the monthly scale.

The global distribution of POU₁₀₀ shares analogous spatial patterns to those of the correlation and uncertainty estimates in Figure 4a1–a3,c1–c3. Specifically, sites with higher correlations and smaller uncertainties (green in Figure 4a1–a3,c1–c3) were found to also have small POU₁₀₀ values, which were mainly distributed across Asia, Europe, and Eastern America, etc. However, the largest POU₁₀₀ values (red in Figure 6a–c) were primarily found in the Southern Hemisphere (red in Figure 4a1–a3,c1–c3). At certain locations such as the Southern Hemisphere and the western United States, the POU₁₀₀ reached >80%, indicating that the satellite AOD retrievals were problematic in these regions. The correlations between the POU₁₀₀ values and the uncertainties were further revealed in Figure 7. The statistically significant (R² > 0.38, p < 0.01) positive relationships were found between these two accuracy measures at the hourly, daily, and monthly scales, further suggesting that the proposed metric (POU₁₀₀) could be considered as an alternative and simple metric to quantify the data quality of MODIS AOD products before they were used for other applications.

4. Discussion

4.1. Factors Influencing the Representativeness of Satellite AOD Products

The representativeness levels of satellite retrievals at different temporal scales were evaluated based on their agreements with the ground-based measurements from AERONET in this study. Previous research on the validations of satellite retrievals were primarily based on hourly observations [23,70,73], and this study further explored the potential limitations/problems with the satellite-based AOD estimations at daily and monthly scales. The results showed that the performances of satellite retrievals at the global scale varied across the three considered temporal scales, particularly between the monthly and daily/hourly scales. Three factors that could potentially contribute to the discrepancies between different time scales are: (1) uncertainties from the satellite AOD retrieval algorithms; (2) diurnal variations in aerosol conditions; (3) cloud contamination in the satellite observations; and (4) the aggregation method of AERONET measurements.

The AOD retrieval algorithms are sensitive to the choice of aerosol models and surface reflectance [29,67,74]. The choice of aerosol model is dependent on three aerosol optical properties including AOD levels, Angström exponent, and single scattering albedo (SSA) [75,76]. Revealing the relationship between the three properties and hourly uncertainty could potentially help to understand the mechanism of errors caused by the assumptions in aerosol model selection. Figure 5a3 shows that the uncertainty levels of hourly AOD have an apparent negative correlation with the magnitude of AOD (i.e., higher AOD values were associated with low relative uncertainties and POU₁₀₀ and vice versa (Figure 4c1 and Figure 6a)). Moreover, as shown in Figure 8, POU₁₀₀ was significantly correlated with the mean AERONET-measured AOD values at all temporal scales. In addition to AOD, the Angström exponent is one of the metrics that are useful for the determination of aerosol model and describing aerosol particle size [75]. Higher Angström exponent represents a greater fraction of fine mode aerosol particles, and smaller values indicate coarser particles [77]. In contrast, SSA is a measure of light extinction due to scattering, which ranges between 0 (pure absorbing aerosol) and 1 (pure scattering aerosol) [78]. While the Angström exponent could be directly obtained from the AERONET datasets (generally true) rather than MODIS products with a certain but unassured uncertainty [18,79], the SSA data provided by AERONET are insufficient to conduct a statistically meaningful validation. In practice, the SSA was partially represented with different aerosol types (i.e., continental, moderate absorbing fine, strong absorbing fine, weak absorbing fine and dust coarse types for DT models) that are associated with every MODIS AOD pixel, which were extracted from the MOD04 dataset “Aerosol_Type_Land” with the same matching scheme as that for MODIS hourly AOD. This dataset should be reliable since it was obtained using the cluster analysis of AERONET aerosol parameters. To explore the impacts of SSA on the reliability of MODIS AOD products, Figure 9a–c shows that the relative uncertainty of hourly AOD as a function of Angström exponent for different aerosol types (matchups were screened by the flags associated with the MOD04 dataset (i.e., “AOD_550_Dark_Target_Deep_Blue_Combined_Algorithm_Flag”)). For a given AOD range and aerosol type, the uncertainties of hourly MODIS AOD remained generally stable for different Angström exponents, suggesting the limited impacts of aerosol particle size distributions on the uncertainty of the satellite AOD retrievals, which is consistent with the results from Kaufman [53]. Small differences (Figure 9e) were found between moderate absorbing fine and strong/weak absorbing fine types for the low aerosol loading conditions (0.03 < AOD < 0.1), while such impacts appeared much smaller than that of the impacts of AOD magnitude.

Furthermore, the accuracy of the satellite algorithms also depends on the accuracy of surface reflectance estimations. Generally, more accurate AOD retrievals and surface reflectance are often expected from vegetated areas than urbanized and water-inundated regions [44,69,80]. Therefore, the expected different performances over different land cover types could impact POU₁₀₀ (see Figure 5). The further test of POU₁₀₀ on different land covers (see Figure 10) showed that the largest differences compared with 0.06 of the global Threshold₁₀₀ (see Figure 5a3) were found in bare soil area (Threshold₁₀₀ = 0.11), followed by water (Threshold₁₀₀ = 0.03) and crop land (Threshold₁₀₀ = 0.04), while the total area sizes only accounted for 26.0% (bare soil areas), 2.73% (inland water), and 14.30% (crop land) of the entire global land surface, respectively (estimated with the GlobeLand30 product, http://www.globeland30.org/). As such, instead of using a land cover type-specific region, a universal threshold (0.06) was suggested and used here for easier implementation by generally less-trained users.

To quantify the potential impacts of diurnal changes in the aerosol conditions, the coefficients of variation (CVs, which were estimated as the ratio between the standard deviation and mean) of the daily AERONET AOD measurements were first estimated, and the mean values for each site were shown in Figure 11. The CVs showed significant spatial heterogeneity, but briefly, more than 95% of the AERONET sites had a mean CV of <30%, and around 61% of the AERONET sites had a mean CV of <20%. Such magnitudes of the daily variations in AOD, however, resulted in negligible differences in terms of the statistical measures of hourly and daily satellite products (Figure 3a–f). To further prove the limited impacts of diurnal changes on the representativeness of composed daily satellite AODs for individual sites, the mean CVs of the AERONET site were plotted against the corresponding uncertainty estimates (i.e., Figure 12a) for data at the daily scale. The determination coefficient (R²) was 0.13, suggestion that only 13% of the uncertainties of daily AOD could be potentially explained by the diurnal variability in AOD conditions. The similarity in the uncertainties at the daily and hourly scales could be further demonstrated through their significant agreements and small differences (see Figure 12b). Indeed, such results are consistent with the study by Kaufman [53] and Schutgens [81], where the diurnal cycles only contributed to a few percent of the relative difference when the data were aggregated to a daily scale.

The representativeness of the monthly MODIS AOD products appeared to be worse than that of the daily products. The large daily gaps within a month between the ground-based and satellite measurements could be one of the major factors contributing to increased differences between satellite observations and ground truths given the limited impacts of diurnal changes. On one hand, there are few valid satellite AOD retrievals at only one or two cross times for most sites. Indeed, the global mean cloud coverage is ~67%, as suggested by the statistics from the MODIS cloud mask products [54]. Additionally, it has been acknowledged that satellites are unlikely to obtain accurate AOD retrieval in the vicinity of clouds [56,82]. As such, the probability of having high-quality MODIS AOD is less than 33% from a global mean perspective, even if the AOD algorithm could produce valid retrievals on all clear-sky days. Furthermore, the MODIS AOD retrieval algorithms could also be low-quality or even fail when improper aerosol models or surface reflectance are estimated [29,45,48,67], further reducing the number of valid satellite AOD data (denoted as N_SAOD) and causing substantial data gaps within a month. On the other hand, AERONET takes many more daily measurements to increase the probability of cloud-free measurements. Both of the above two aspects contribute to the remarkable gap differences in daily AERONET AOD and daily MODIS AOD in monthly mean calculation. Therefore, as a convention, monthly MODIS AOD products are utilized only when the N_SAOD is no less than 5 for that month; otherwise, the monthly satellite AOD products would not be utilized [70,71]. Whether this convention is reasonable and could produce satisfactory monthly binning data, we further examined the impacts of N_SAOD on the reliability of monthly MODIS AOD products and to test the validity of the conventionally used threshold (i.e., 5)

Statistical measures of the individual N_SAOD values were estimated and demonstrated in Figure 13. In general, the uncertainty and RMB values decrease with the increased N_SAOD and levelled off when the N_SAOD reaches around 15. For example, when N_SAOD increased from 5 to 15, the RMB decreased from >1.4 to <1.2, suggesting that the overestimation errors of monthly satellite products could be significantly reduced through increasing the availability of valid satellite retrievals in a month. Likewise, the uncertainty of MODIS monthly AOD showed a decrease of more than three-fold from >300% to <100% when the N_SAOD increased from 5 to 15. Significant impacts of N_SAOD on the representativeness of monthly AOD could be further revealed by scatter plots between AERONET monthly AOD and MODIS Terra/Aqua monthly AOD (see Figure 14), where the N_SAOD values were color coded for each point. Clearly, better agreements were often found with high N_SAOD values (yellow to red), and points far from the 1:1 line were apparent with small N_SAOD values (blue to green). In contrast, with the continued increases in N_SAOD, the correlations (R²) between the satellite and ground-based AOD data showed slight improvements, and the mean bias exhibited a moderate decrease. Such patterns agreed well with the results in Figure 3d–f,g–i, where the R² and the mean biases showed nonsignificant differences between the daily and monthly scales. Indeed, the sensitivity analysis of N_SAOD in this study further justified the fidelity of using 15 as the threshold to determine valid AERONET monthly data (see Figure 2e).

The uncertainty of AERONET measurements may be another factor in the uncertainty evaluated of MODIS AOD retrievals for extremely clean air conditions. The AERONET AOD had an uncertainty level of 0.01–0.02 [83], suggesting that the ground-based observations may not be appropriate to consider as “ground truth” to gauge the performance of MODIS AOD products under very thin aerosol conditions. For example, poor performances of satellite AODs have been identified at three temporal scales in some low AOD regions (such as the central Australia, see Figure 4). However, to determine whether the large uncertainties in these regions were attributed to the satellite or ground measurements requires more sophisticated measurements in the future. Additionally, the difference in sample sizes between MODIS (10 × 10 km) and AERONET (point-based) measurements may also cause potential disparities between the two AOD values. However, the associated impacts has been demonstrated to be negligible by Ichoku [69] and Schutgens [84], where sensitivity analysis with a range of spatial sampling sizes from 300 to 50 km resulted in insignificant variations.

The strict criteria used to aggregate AERONET AOD data from an hourly to daily scale (i.e., at least one valid measurement is available for each hour, see details in Section 2.3) could exclude some daily AODs when rapid changing aerosol episodes occur within a day such as cloudy weather, smoke plumes, and dust storms, among others. In practice, these episodes are not able to pass the cloud screening checks [56,82], thus reducing the number of hourly AOD within a day and therefore leading to invalid daily AOD. In contrast, MODIS daily observation within the same day may be valid if the satellite overpass time is outside the episode-occurring time window. To examine associated impacts, Figure 15 compares the AERONET monthly AODs aggregated using the most relax scheme (daily AERONET data are considered as valid when one AERONET measurement is available within a day) to that of with the current strict method. Clearly, the results of the two schemes agreed well with one another (R² = 0.96, slope = 1.03), indicating the small impacts of used criteria in daily AERONET data aggregation.

4.2. Implications and Future Efforts

The representativeness level of the monthly mean satellite AOD appears to be closely related to the data availability within a month. Such information allows us to gauge the reliability of the monthly MODIS AOD composites when N_SAOD is provided, which is very helpful for many aerosol applications such as AOD trend analysis, where monthly AOD composites are often used [7,85]. Specifically, the uncertainties of the MODIS monthly AOD products decreased with the increasing number of valid daily retrievals when N_SAOD was below 15. Therefore, the conventionally used value of 5 as the minimum number may not represent an optimal threshold for monthly satellite AOD composites, and the considerable amount of uncertainty due to data gaps exist in the monthly products and propagate to the associated trends [86]. Unfortunately, using 15 of the valid daily retrievals within a month is not a realistic expectation with MODIS satellites from a global mean perspective, which could possibly explain the inconsistency in the spatial and temporal trends derived from different AOD products [15]. Globally, the data temporal coverage and thus the N_SAOD values could vary remarkably among different satellite missions.

Figure 16a,b demonstrates the global distributions of the climatological mean N_SAOD for both the MODIS Terra and Aqua satellites, which were estimated using the entire MODIS AOD product archive between 2000 and 2017. The availability of valid MODIS AOD retrievals varies considerably over different regions, where the mean values for global terrestrial regions are 5.28 ± 4.71 and 4.77 ± 4.39 for MODIS Terra and Aqua, respectively. High N_SAOD values primarily occur in semiarid or arid areas (such as North Africa, western Asia, Australia, etc.), where the N_SAOD value can reach >15. In contrast, humid areas (such as eastern Asia, North America, and South America, etc.) are found to have much fewer valid MODIS AOD product data, where the N_SAOD values are generally <15 or even <5, which could potentially lead to substantial errors when such limited datasets are used to compose monthly products. Furthermore, similar global distribution patterns were also found when the data were integrated at monthly climatological scales (results not shown here). Notably, the combination of the two instruments (see Figure 16c) could help to increase the data availability (mean N_SAOD of 7.34 ± 6.14). Therefore, the long-term trends obtained from the combined datasets of the two instruments could be potentially be more reliable than those from individual sensors. Moreover, with the availability of AOD products from various satellite missions (such as MODIS, VIIRS, SeaWiFS, etc.) or even geostationary satellite sensors (such as the geostationary ocean color imager (GOCI), with eight hourly observations per day) [87], the use of multisource observations could be expected in the future to reduce the uncertainties caused by limited data in a month.

The reliability of the satellite products is also heavily dependent on the accuracy levels of the retrieval algorithms, and the uncertainties in the MODIS AOD products were demonstrated as a function of aerosol loads for any given temporal binning method (see Figure 5). Therefore, the concept of POU₁₀₀ could be used to evaluate the performances of satellite AOD products in the spatial domain. Similar to the N_SAOD, the entire MODIS dataset archive was also used to estimate the climatological mean POU₁₀₀ for the entire world (see Figure 16d–f). A similar spatial pattern as Figure 6 was found for POU₁₀₀, where high POU₁₀₀ values were found in regions with known small aerosol loads and good air quality such as most of the Southern Hemisphere, North America, and central North Asia. The patterns were generally the same for both individual instruments and the combination of the two instruments for terrestrial regions, where the mean values were 14.66% ± 14.35%, 16.54% ± 15.75%, and 15.74% ± 14.69% for Terra and Aqua separately and for the combined datasets, respectively. By definition, the regions with high POU₁₀₀ are supposed to have large satellite AOD uncertainties. Therefore, future applications (e.g., AOD trend analysis and validation of some climate modeled aerosol fields) of the MODIS monthly AOD products should consider not only N_SAOD within a month, but also POU₁₀₀, as demonstrated in Figure 16. Nevertheless, extensive efforts are required to improve the accuracy levels of the satellite retrievals under low aerosol loading conditions to improve the performances of satellite AOD products over regions with high POU₁₀₀ values.

Although the entire global AERONET data archive was used to study the representativeness of the MODIS satellite temporal binning AOD products, the lack of ground-based data due to the intermittent observations remains a problem [84]. For example, due to the fact that timely AERONET itself has the uncertainty of 0.01~0.02, unbiased evaluation metrics could be better for the assessment of extremely clean conditions. Additionally, when integrating the data into monthly scales, it is ideal to include ground-based daily measurements from every day in a month. However, this practice would only generate a few match-up pairs, where the month with 15 valid daily measurements was used (the applicability of this threshold has also been justified). Unfortunately, even with such a compromise, the number of match-ups for each AERONET site was not adequate to calculate statistically meaningful site-specific measures for most sites at the monthly scale, and the evaluations at seasonal and annual scales were also not possible at this stage. Moreover, the MODIS AOD products and temporal bins used in this study were based on the original spatial resolution (10 × 10 km) of the Level-2 data, which has much higher resolution than the monthly Level-3 MODIS AOD products (i.e., the 1°~×~1°, ~110~×~110 km at the equator) [85,88,89] and could better capture more details in spatial variations. Whether the spatial aggregation from Level-2 to Level-3 products could reduce or enhance the uncertainties caused by data gaps requires further efforts.

5. Conclusions

The reliability of MODIS AOD binning products at various temporal scales have been comprehensively evaluated based on global long-term ground truth (represented by the datasets strictly calculated from AERONET observations) between 2000 and 2017. There are several findings from this study, which are as follows. First, the MODIS hourly and daily observations demonstrated similar accuracy and uncertainty levels, suggesting the representativeness of the MODIS retrieval-based AODs for daily aerosol conditions at both global- and site-level assessments. Second, the overestimation errors of satellite AOD products could be enhanced when the data were aggregated into monthly scales at global-level assessment. The limitation of valid AOD retrievals from MODIS images could be the primary reason for the increased uncertainty levels of longer temporal binning products due to the data gaps at longer temporal scales largely caused by the frequent presence of clouds, and increasing the number of daily data within a month into 15 could effectively reduce the uncertainty of the commonly used monthly satellite data. Third, the uncertainty levels of the MODIS AOD products appeared to be predominantly related to the AOD magnitude at all temporal scales. The findings and discussions of this study could provide critical guidelines for assessing and improving the reliability of satellite AOD products, their derivatives (such as radiative forcing, PM2.5, etc.), and associated trends.