Machine Learning Based Algorithms for Global Dust Aerosol Detection from Satellite Images: Inter-Comparisons and Evaluation

Round 1

Reviewer 1 Report

Dear colleagues,

In this study, Lee et al. try to use the labels (vertical atmospheric features) of the CALIPSO/CALIOP as predictand and VIIRS bands as predictors to classify the global dust occurrence. Despite the remarkable results of this work, the use of satellite measurements as output is challenging. The authors themselves (L71-L78) have listed the problems of this sensor in the separation of atmospheric features. However, they omitted most of these limitations in the development of the model. Having said that, the claim that the CALIOP provides the most accurate information on the occurrence of the dust phenomenon is also very inaccurate. They used daytime CALIPSO labels, which are highly influenced by surface reflectance and sunlight. The level of contamination is so severe that studies such as Adam et al. have relied solely on CALIPSO overnight data to address this problem. I strongly suggest that the authors examine the accuracy of CALIPSO data (using terrestrial data) in a new section of the revised manuscript. For example, the amount of CALIPSO Ångström could be compared with the values ​​measured by the AERONET. Moreover, they could also use observed atmospheric phenomena from meteorological stations (the type of weather phenomenon) as predictand and report differences in results. The method used to evaluate the accuracy of the model is also not very efficient. The authors evaluated the output of their developed model by applying another model (PHYS) to VIIRS bands. As they have pointed out, machine learning techniques have been trained to achieve the best prediction of CALIPSO measurements. And obviously, their output is much closer to CALIPSO labels than those of PHYS. At the same time, using two days of data is insufficient. Given these points, I do not recommend publishing this study with the current structure of the model. I urge the authors to pay special attention to the evaluation (daytime) CALIPSO data in the revised version. Other points are presented below.

L2: Deep learning is also a machine learning algorithm. They should not be discriminated.

L50: “….confound the detection algorithm“ needs citation.

L55-58: the use of a fraction of bands in physical models could not be known as an issue or a research gap.

L67: please investigate the impact of misclassification of ice as dust and vice versa in your modeling system.

L69: As discussed above, the reliability of CALIOP measurements need to be investigated.

L71: “on the other hand” should be replaced with “but”.

L79: Again, how did you find out that “CALIOP-based dust detection is currently considered as the most reliable algorithm”

L130-L105: you only discussed that physical models do not consider all bands. It is not an issue as they are developed to only consider physically related bands.

L120-127: could please provide a plainer explanation (or reference) why a similar view angle is problematic.

L142: “…. the most reliable dust detection” please check the comments above.

L203: what about sunlight and surface reflectance impacts.

L291: since you did not develop PHYS it is better not say “we introduce…”

L370: what are the inputs of CNN. The images of 23 variables or ….?

L379: dropping visible bands for the detection of dust is not interesting but problematic. Please check why feature selection criteria drop them.

L408-L409 use units for 5x5.

L441 …. “sensor sensitivity..” remove the duplicated dot.

L448: it may be because of cloud contamination.

 

Adams, A.M., Prospero, J.M. and Zhang, C., 2012. CALIPSO-derived three-dimensional structure of aerosol over the Atlantic Basin and adjacent continents. Journal of Climate, 25(19), pp.6862-6879.

Author Response

First we would like to thank the three reviewers for their insightful comments and helpful suggestions, which have helped us improve the manuscript significantly. Below are point-by-point responses to the comments, questions and suggestions. Here we would like to summarize the major changes to the manuscript:

• We have spent significant effort explaining why we choose to use the CALIOP daytime dust detection product for this study. As we pointed out, our main purpose is to develop a global scale dust detection algorithm using satellite observations. Although the CALIOP daytime product has several limitations, it is still a reasonable choice for our study. On the other hand, we also pointed out its limitations following the suggestion of the reviewer #1.
• We explained the technical questions from the reviewer #2. Also, we revised Figure 5-11 following the suggestions/comments of reviewer #2.

 

Answer to Reviewer #1:

 

In this study, Lee et al. try to use the labels (vertical atmospheric features) of the CALIPSO/CALIOP as predictand and VIIRS bands as predictors to classify the global dust occurrence. Despite the remarkable results of this work, the use of satellite measurements as output is challenging. The authors themselves (L71-L78) have listed the problems of this sensor in the separation of atmospheric features. However, they omitted most of these limitations in the development of the model. Having said that, the claim that the CALIOP provides the most accurate information on the occurrence of the dust phenomenon is also very inaccurate. They used daytime CALIPSO labels, which are highly influenced by surface reflectance and sunlight. The level of contamination is so severe that studies such as Adam et al. have relied solely on CALIPSO overnight data to address this problem. I strongly suggest that the authors examine the accuracy of CALIPSO data (using terrestrial data) in a new section of the revised manuscript. For example, the amount of CALIPSO Ångström could be compared with the values measured by the AERONET. Moreover, they could also use observed atmospheric phenomena from meteorological stations (the type of weather phenomenon) as predictand and report differences in results. The method used to evaluate the accuracy of the model is also not very efficient. The authors evaluated the output of their developed model by applying another model (PHYS) to VIIRS bands. As they have pointed out, machine learning techniques have been trained to achieve the best prediction of CALIPSO measurements. And obviously, their output is much closer to CALIPSO labels than those of PHYS. At the same time, using two days of data is insufficient. Given these points, I do not recommend publishing this study with the current structure of the model. I urge the authors to pay special attention to the evaluation (daytime) CALIPSO data in the revised version. Other points are presented below.

 

One of the main concerns/criticism of the reviewer is the accuracy of CALIOP aerosol classification, in particular dust identification during daytime. In fact, we are well aware of the limitations of CALIOP products and pointed them out in our paper. Nonetheless, we used CALIOP data for a number of reasons, which are explained below:

• First of all, the main objective of this paper is to develop a global dust detection algorithm based on passive VIIRS observations. Although there are many dust detection algorithms available (e.g., AERONET), CALIOP operational product is widely accepted as the state-of-the-art dust identification on the global scale. Other techniques, such as AERONET and ground-based lidar may have better skills in certain circumstances but they are point observations in nature and therefore cannot meet the need for global observations. It is the reason why most recent studies of dust aerosols on the regional to global scales are using CALIOP data, including Adam et al (2012) as pointed out by the reviewer.
• Indeed, solar background radiation makes the CALIOP observations generally nosier during daytime than nighttime. However, despite the noise, the aerosol classification and dust identification during daytime are still generally accurate. Note that Adam et al. (2012) cited the study by Z. Liu et al. (2009) for their discussion of CALIOP product accuracy (see their discussion in Section 2.a). Z. Liu et al. (2009) pointed out that the main problem facing the CALIOP dust detection algorithm is that thick dust layers are often misidentified as ice clouds. However, they estimated such misclassification only accounts for ~0.7% of the “total [tropospheric] features” (see section 3.b of Z. Liu et al. 2009). Adam et al. (2012) cited this number and commented that “The most prominent error is the misclassification of Dust or Smoke as cloud, which occurs when the dust or smoke layer is thick, and optical properties become similar to those of cloud, or if the aerosol is located near a cloud layer (Z. Liu et al. 2009). Z. Liu et al. (2009) found that this type of error occurs in less than 1% of the cases in their study involving Dust;”(see their Section 2.a). In addition to Adam et al. (2012) and Liu et al. (2009), the effectiveness of CALIOP’s dust labeling has also been assessed by many validation studies. Mielonen et al. 2009 confirmed that CALIOP’s classification of dust is more reliable than classification of fine aerosols due to depolarization ratio.  Schuster et al., 2012 compared the CALIOP daytime aerosol subtype with AERONET using 3 years of daytime CALIOP collocated with AERONET data and found that dust aerosols are generally identified correctly over these 176 AERONET sites. Burton et al., 2013 showed that when compared with HSRL 80% of the CALIOP dust layers are also identified by HSRL-1 as dust or polluted dusty mix.
• Another important reason for us to use the daytime product of CALIOP is because we need the solar reflective channels from the VIIRS for dust detection. As shown in many previous studies and also confirmed by our own analysis (see our analysis of the predictor importance in table 2 and corresponding discussion), mid-infrared solar reflective bands (e.g., M09 1.38 µm, M10, 1.61 µm, M11 2.25 µm) are the most useful bands among VIIRS spectral observations for dust detection. These bands can only operate during daytime. If we use nighttime product from CALIOP, we will lose these bands and the rich information they provide for dust detection.
• Finally, we evaluated our model against CALIOP on the global testing data, which is a standard way to evaluate the performance of a ML model. The outcome is very satisfying as the model reproduced the CALIOP identified dust patterns and also extended reasonably to areas that are off-track of CALIPSO.  The PHYS method is solely for comparison purposes and not for validation purposes. As we pointed out in the paper, due to the different definition of dust, we cannot directly compare the results from PHYS to FFNN. However, we can analyze the differences between them and give hypotheses of the potential causes of the differences and further to conclude, which method is performing better under certain circumstances.  Again, the main purpose of the paper is to show that the model can reproduce the CALIOP identified dust on a global scale, rather than the model can reproduce the “true” dust patterns, which can vary a lot due to dust definition. Thus, we believe our validation method is sufficient regarding the purpose of the paper.  In our future work, we do plan on strengthen/varying the thresholds of CALIOP definition of dust such as dust loading/single dust layer and analyze the model results, and we are planning on using other means to validate our ML results, such as AERONET, or HSRL, however, it is not within this paper’s scope.

So, in short, we believe the daytime CALIOP product is the best choice for our objective of developing a global dust detection algorithm that fully utilizes the information content from VIIRS. On the other hand, we thank the review for raising this reasonable concern. We have added more discussions about the accuracy of CALIOP aerosol classification, in particular dust detection, in the revised manuscript with updated references, including Adam et al. (2012) and Liu et al. (2009).

Author Response File: Author Response.docx

Reviewer 2 Report

See attached

Comments for author File: Comments.pdf

Author Response

First we would like to thank the three reviewers for their insightful comments and helpful suggestions, which have helped us improve the manuscript significantly. Below are point-by-point responses to the comments, questions and suggestions. Here we would like to summarize the major changes to the manuscript:

• We have spent significant effort explaining why we choose to use the CALIOP daytime dust detection product for this study. As we pointed out, our main purpose is to develop a global scale dust detection algorithm using satellite observations. Although the CALIOP daytime product has several limitations, it is still a reasonable choice for our study. On the other hand, we also pointed out its limitations following the suggestion of the reviewer #1.
• We explained the technical questions from the reviewer #2. Also, we revised Figure 5-11 following the suggestions/comments of reviewer #2.

Please see our point-to-point response to your review in attached file

Author Response File: Author Response.docx

Reviewer 3 Report

In this manuscript the authors have carried out a feasibility study of using various machine learning (ML) and deep learning (DL) algorithms for the detection of dust aerosols from satellite remote sensing. They have used CALIOP and VIIRS Satellite data. The results very interesting and encouraging, even though it had several limitations such as abnormal dust detection over Antarctica and remote oceans. This is a very novel study and the authors are encouraged to fine tune and improve their algorithms and use more ground truth measurement (AERONET) in their future endeavors.

Author Response

First we would like to thank the three reviewers for their insightful comments and helpful suggestions, which have helped us improve the manuscript significantly. Below are point-by-point responses to the comments, questions and suggestions. Here we would like to summarize the major changes to the manuscript:

• We have spent significant effort explaining why we choose to use the CALIOP daytime dust detection product for this study. As we pointed out, our main purpose is to develop a global scale dust detection algorithm using satellite observations. Although the CALIOP daytime product has several limitations, it is still a reasonable choice for our study. On the other hand, we also pointed out its limitations following the suggestion of the reviewer #1.
• We explained the technical questions from the reviewer #2. Also, we revised Figure 5-11 following the suggestions/comments of reviewer #2.

 

 Answer to Reviewer 3:

 

Reviewer #3

In this manuscript the authors have carried out a feasibility study of using various machine learning (ML) and deep learning (DL) algorithms for the detection of dust aerosols from satellite remote sensing. They have used CALIOP and VIIRS Satellite data. The results very interesting and encouraging, even though it had several limitations such as abnormal dust detection over Antarctica and remote oceans. This is a very novel study and the authors are encouraged to fine tune and improve their algorithms and use more ground truth measurement (AERONET) in their future endeavors.

 

We thank the reviewer for its positive comments. We will include more ground truth measurement in our future work.

Round 2

Reviewer 1 Report

Dear Sir/Madam,

The manuscript has been satisfactorily revised.

Best

Reviewer 2 Report

See attached.

Comments for author File: Comments.pdf