Retrieval of Atmospheric Water Vapor Content in the Environment from AHI/H8 Using Both Physical and Random Forest Methods—A Case Study for Typhoon Maria (201808)

Round 1

Reviewer 1 Report (Previous Reviewer 1)

The authors answered most of comments I addressed last time. But I still insist on testing RF-based model to another typhoon. If it does work, then it could demonstrate what the authors want to show in this work.

In addition, I didn’t think ERA5 as truth value to evaluate H8 estimate of water vapor. If so, you just use ERA5 to monitor typhoon, not with remote sensing. Why not use radiosonde observation?

Author Response

Please see the attachment

Author Response File: Author Response.docx

Reviewer 2 Report (Previous Reviewer 2)

The authors have corrected the article according to comments.

I have no other comments.

Author Response

Please see the attachment

Author Response File: Author Response.docx

Reviewer 3 Report (New Reviewer)

 

Title: Retrieval of atmospheric water vapor content in the environment from AHI/H8 using both physical and random forests methods – a case study for Typhoon Maria (201808).

 

 

Authors: Zhu et al.

 

Summary:

This paper proposed using the random forest (RF) algorithm trained on the ERA-5 for fast water vapor retrieval using AHI radiance observations. Its results are then compared with 1D-Var retrievals. Although using the random forest for different retrieval applications is not new, a solid application with the RF algorithm is still worth publication as long as the algorithms are carefully designed and thoroughly evaluated. However, I have severe concerns about the procedure adopted by this study:

1. It is unclear which periods of ERA-5 reanalysis and GFS forecast are used as the training dataset. If I’m correct, the authors collected the training and testing dataset from the same period during Typhoon Maria. The consequence of this is that they will have too confident results from the RF algorithm trained on ERA-5. So it is crucial to use independent datasets to compare the retrievals from their RF algorithm trained on ERA-5 and 1D-Var.
2. However, the only independent dataset they used in this study is only 2 profiles of dropsonde observations (Figure 5). Besides, the results of 1D-Var for these 2 profiles are not included in the figure. The results are also difficult to understand without more detailed explanations. In their RF algorithm, they found that the most critical input variable is 9.6um channel, which is an ozone-sensitive channel. Why is the ozone-sensitive channel the most important predictor for water vapor retrievals? What happens if you remove 9.6um channel from your algorithm? Without additional experiments, these questions remain unsolved. 
3. In Figure 6, they verified their 1D-Var retrieval and RF-based retrieval against the WV fields. However, the numbers from those two algorithms are substantially different. How can this happen?

 

Considering all these issues, I suggest a major revision or rejection of this paper in its current form.

 

Recommendation:

Major revision or rejection 

 

 

Major comments:

1. Questions related to Methodology:

1. L134: “only IR measurements from water vapor and window bands (bands 8-16) are used in the retrieval process”: I’m confused. Did you include band 12 (9.6um) in the retrieval? If included, why do you want to include the ozone sensitive channel to retrieve WV profile? Any results removing this band?
2. In Table 2, AHI Input, you included additional brightness temperature differences between WV and window channels though you already included the Tb from those channels. Do you have a trained RF without using those differences as input variables? How is its result compared to your current result?
3. L125: “spatial resolution of 0.5 deg”: GFS forecasts also have 0.25 deg products, which are open to the public. Why did you use 0.5 deg GFS instead of the finer-resolution 0.25 deg FS? Also, it is well known within the data assimilation community that compared the ECMWF reanalysis or forecast, GFS has larger moisture biases in the upper troposphere.  
4. L196-L203: What are the periods of your training dataset for GFS, and ERA-5? Since you test your results during typhoon maria (July 4-11 in 2018), you should not use any GFS and ERA-5 during this period as the training dataset. Otherwise, your results of RF trained over ERA-5 will be overconfident.
5. L201: “with 70% (data size is 83119)”: This sample size is pretty small, if considering the sample size of one AHI full-disk scan (# of scanlines x # elements per scanline: ~5000x5000, though you have reduced amount if only retaining CSR). 
6. L196: The AHI and the GFS are temporally interpolated to the resolution of the ERA5: For this step, are you using the clear-sky AHI pixels? If so, then how did you conduct this step, since large areas contaminated by clouds are removed? How to do you treat areas without AHI CSR observations?
7. L221: “so it is necessary to use the independent ERA5 data to validate the accuracy and the temporal variation of water vapor information…”: I cannot agree with this statement. Since you are using the ERA5 as the training dataset, the results you retrieved using the RF algorithm will preserve the good and bad parts of ERA5. It is essential to include independent datasets such as radiosonde profiles to evaluate your algorithm more adequately.

 

1. L233-L234 “cost function and X”: What exactly is your X? Which variables do you include here? For your 1D-var, what are the structures of B used in your retrieval? Is B diagonal or nondiagonal? Are any error correlations between q and T considered in your B?
2. Table 3: the number of “max_feautres” is “auto”. Are you using sklearn? Which version? One important thing is that if you are using sklearn after version 1.1: the default behavior is no longer “auto” but “sqrt”.
3. Figure 1 shows that the most important variable for their RF algorithm is 9.6um channel, which is very difficult to understand, because 9.6um channel is the ozone-sensitive channel. Why does it matter so much for WV retrievals? A deeper investigation in this paper is necessary to understand this part.
4. Figure 5: you only have two independent dropsondes to evaluate your RF-based algorithms; in this figure, the 1d-var retrieval results are not shown. Please add them. 
5. Figure 6: Why are the number of 1D-Var and RF-based algorithms different for the same time snapshots?

 

 

Minor comments:

L101: “The purpose of this study is to …such as machine learning…”: This sentence is too general, replace “machine learning" with "random forest”, since machine learning itself is a very broad topic and the random forest is a very narrow subtopic. 

L138: “For your information…”: delete this sentence.

Table 1: Revise “Resolution (km)”: The resolution is 2km only at nadir. Other regions, especially those from high latitudes, experience pixel distortion with a much lower resolution.

L210: “very good quanlity”: very good quality

L332: Figures 4 shows the differences between the true values (ERA5): Any reanalysis dataset including ERA5 is never the truth. Rephrase this sentence.

 

Author Response

Please see the attachment

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report (Previous Reviewer 1)

I have no further comments.

Author Response

Thanks for your time and efforts on reviewing our manuscript.

Reviewer 3 Report (New Reviewer)

The reviewers have answered almost all my questions except two. One of these two questions is crucial, so I will state it (Point 1) again here because another reviewer also raises this issue.

 

Point 1:

The most concerning thing is the training dataset used by the random forest (RF) algorithm. The RF algorithm is trained using 70% of the collocated datasets of ERA5, AHI clear-sky pixels and the GFS forecast during the period of Typhoon Maria, and validated using 30% of the collocated dataset from the same period. 

 

I understand that 70% training dataset and 30% test dataset are randomly splitting since this t is common when tweaking the ML algorithms, and it has been implemented in different ML/AI software packages. But don’t forget the fundamental purpose of this splitting. We want the training and test datasets to include samples that cover as many corner cases as possible so that we can ensure the trained algorithm can do fair jobs on all scenarios and its performance can be evaluated adequately. 

 

Take a simple common ML task using the MNIST dataset as an example since this is a well-known case. Suppose our goal is to train a classifier to recognize hand-written digits 0-9, but all our training and test dataset mainly contains pictures of digit 7, but no other digits. In this case, you will get a well-trained classifier for digit 7, and your verification score will also be high. However, the high verification score does not mean that you have an accurate classifier that can recognize hand-written digits 0-9 because your test dataset is not representative (few test cases for numbers other than 7).

 

The same concern also exists for this study. Since the training and testing datasets are drawn from the same period, those moisture and temperature profiles are much more like each other during this period than those drawn from different periods, which will give you too confident results. To examine the real performance of your algorithm, it is vital to test your algorithm on an independent typhoon case. 

 

The authors responded to another reviewer that they are working on another typhoon case now. Before seeing the results from that independent case study, I do not have enough information to determine if this proposed method also works well on other cases at this stage.

 

Point 2:

 

This should be easy to handle. From my previous review, I asked” Point 12: Figure 5: you only have two independent dropsondes to evaluate your RF-based algorithms; in this figure, the 1d-var retrieval results are not shown. Please add them.”

 

The correspondence from the authors is “

Response 12: Thanks for your suggestion. It takes over 5 hours to use the 1DVAR algorithm to retrieve the water vapor concentration with a full-disk 15-minute AHI measurement. The 1DVAR algorithm is not very efficient in computation if aiming to obtain water vapor concentration at full resolution of the AHI measurement. In addition, 1DVAR retrieval uncertainties are difficult to be quantified. Therefore, we only invert the time used to verify the test-set results, not the time corresponding to the dropsondes. We wanted shows that except retrievals at the upper troposphere, most of the retrievals from the RF-based algorithm are even closer to the dropsonde observations than the ERA5.”

 

I have difficulty understanding your response here. All I want from you is to overlay your retrieved 1D-Var profiles on Figure 5, so that Figure 5 shows results from both 1D-Var and RF-based retrievals. 

Currently, Figure 5 only shows results from RF-based retrievals.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The Advanced Himawari Imager (AHI) onboard the Himawari-8 satellite could provide multilayer atmospheric water information with relative higher spatial and temporal resolution, compared with reanalysis data. In this work, the authors compared physical-based 1DVAR method and random forest-based from AHI/H8 to their performance on atmospheric water vapor retrievals during typhoon Maria. AHI/H8 Although the random forest method showed higher accuracy and computational efficiency than 1DVAR. It is a very interesting work. However, AHI/H8 only could provide atmospheric water information under clear sky. How it could estimate atmosphere water during typhoon period. The most important question on machine learning methods should be addressed is how to apply this training network based on one typhoon data to another typhoon. It is not unfair to compare the physical-based 1DVAR method with the random forest training retrievals, since the machine learning methods could predict well of atmosphere water dependent on the training input data. In addition, atmospheric water profile of ERA5 is relative coarse compared with AHI/H8. Why the authors have to use ERA5 as a reference when evaluated RF-based algorithm?  

 

Other comments/suggestions:

 

1)    I suggested change the title as Comparison of physically-based and random forest-based ….

2)    Discussion part needed to be expanded in more details.

3)    Table 2 and Section3, Why can GFS forecasts be used as both results and input to RF-based algorithm?

4)    Line 182-184, How is the H8 data aggregated to 0.25° and does the aggregation method affect the results?

5)    How much data is used to train the RF network? (not ratio)

6)    Line 351-354, The author have resampled H8 data to 0.25°, why the 2km data mentioned here?

7)    Fig.6(a)(b), What do different 'numbers' mean?

8)    Table.6, ‘partly clear(10%-30%)’ change to ‘partly cloud(10%-30%)’.

9)    The author may consider using the input and output of 1DVAR algorithm to train the RF algorithm, so as to realize a new algorithm with high precision, clear physical mechanism and fast speed.

Specific comments:

1)    The Figures in the paper should be more clear.

Author Response

Author Response File: Author Response.docx

Reviewer 2 Report

The aim of the research is not clearly defined in the article. It should be completed and written. It should also be emphasized that the research is an attempt ... A little more criticism is needed by the authors in this article, because of the method used.
The section on Typhoon Maria itself is missing. Why this cyclone was chosen for research. For the reader a more complete information about this typhoon should be provided. What category of typhoons can it fall into? Therefore, the paragraph with the purpose of the work should be completed!
Do the obtained results apply only to this typhoon, or can they be somehow extrapolated to other typhoons?
# page 4, lines 151-152 there is no explanation of the variables in the formula.
Table 1. Citation is missing.
#163 2.2 Methodologies. Rather methods.
Table 2. "Physical meaning"? It's not clear! Units are missing in the second column.
Table 5: percentages at each value can be removed, put in the header.
What do these studies show? No conclusions!

Author Response

Author Response File: Author Response.docx