Runoff Prediction in the Xijiang River Basin Based on Long Short-Term Memory with Variant Models and Its Interpretable Analysis

Round 1

Reviewer 1 Report

The manuscript presents an interesting work on runoff prediction in a river basin in China using LSTM and its variant models. The manuscript is well-presented and has the potential to attract interest among the researchers. There are a few concerns that need to be addressed.

The title of the manuscript may be revised. I think the words 'Research on' are redundant in the title. 

NPI was chosen as one of the input variables based on the correlation coefficient between the atmospheric circulation factor and the monthly streamflow. In Table 2, the lag at which the correlation coefficient was calculated is not mentioned. Some of the atmospheric circulation factors may have a delayed impact on the streamflow. For example, there may be a few months delay before ENSO influences the streamflow. Correlation coefficients of the atmospheric circulation factors with the streamflow at different lags could have been used to choose the input variables from the atmospheric circulation factors.

It is difficult to compare the relative performance of different models in predicting the streamflow from Figure 6. It may be revised to improve the clarity. 

The authors claim that they have used SHAM visualization tool to analyse the interpretability of the LSTM model. There is no description of this tool in the manuscript. How were Figures 10 and 11 derived? How does the information presented in Figure 10 improve the interpretability of the model? 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The four typical hydrological stations in Xijiang River Basin were selected as examples, LSTM and its variants are used to forecast runoff based on atmospheric circulation factors, meteorological factors and historical runoff, and SHAP method is used to discuss the interpretability of LSTM model in runoff prediction. The manuscript is full of workload, but there are still some details that need to be revised, especially the map involving China's sovereignty and territorial integrity:

1. In Figure 1, the nine-dash line of China map is not completely displayed! This is related to national sovereignty and territorial integrity and must be rigorous！

2. Pay attention to the writing of upper and lower corner marks in the manuscript.

3. In Figure 1, the names of rivers and weather stations in the river basin are missing.

4. Is it scientific to use the weather stations data outside the basin to forecast runoff in the basin?

5. Please give the interpolation method or necessary explanation for interpolating the missing data of LZ station.

6. Please give the process of hyper-parameter calibration of LSTM model. In addition, is the control variable method scientific for hyper -parameter calibration, and is there a situation that the LSTM results of the optimal combinations of the single best optimal hyper-parameter values are inferior to the LSTM results corresponding to the single best optimal hyper-parameter values?

7. In Figure 2, there is no legend. What do red dots and green dots represent respectively?

8. In section 4. 1.2, should lags be selected separately for each input variable at each station, rather than the same for all variables at all stations?

9. In section 4. 1.2, the expression of predicted runoff is inaccurate. Since the lag analysis is done, it means that the runoff series is known, so it is unnecessary to predict runoff.

10. In section 4.2, cross wavelet analysis is used to explore the relationship between atmospheric circulation factors and monthly runoff changes. I guess the author may want to emphasize the influence of atmospheric circulation factors on monthly runoff, but in section 4.1, Person correlation analysis has been used to analyze the correlation between them. Please further explain the difference between cross wavelet analysis and Person correlation analysis and the purpose of cross wavelet analysis.

11. The author uses a section to introduce the cross-wavelet analysis method to explore the influence of atmospheric circulation factors on monthly runoff changes, but the importance of atmospheric circulation factors to monthly runoff is not introduced in the part of Introduction.

12. In Figures 8 and 9, some dots are obscured, so it is recommended to reduce the transparency of dot colors

13. What does the number of points of a variable at the same SHAP value represent in Figure 11? It is recommended to describe and explain Figure 11 in detail.

14. The variant model of LSTM does not perform as well as LSTM. Is it because the variant model parameters are not calibrated? Analysis and discussion should be carried out.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

The manuscript has several goals: 1) identify the atmospheric circulation factors that have the greatest impact on streamflow and analyze the time lag effect between historical streamflow, precipitation, evaporation, atmospheric circulation factors, and streamflow forecasts; (2) investigate the effects of atmospheric circulation factors on streamflow changes in the Xijiang River Basin using the cross-wavelet analysis method; (3) evaluate the performance of LSTM and its variants in medium- to long-term streamflow prediction by analyzing the impact of different model structures on prediction results under different lead times; (4) enhance the credibility of the optimal model by conducting interpretability analysis based on SHAP values.

The paper is well structured. The topic is interesting and the manuscript pleasant to read.

I have several comments to share with the authors as listed in the following, consequently my suggestion accepts the manuscript after a MAJOR revision.

Comment 1

Sentence at lines 86-88: "Due to the poor interpretability of traditional machine learning methods, it is difficult to determine the direction of feature influence on the output, and the visibility of feature importance is low." Does the interpretability issue only apply to traditional ML models? Clarify this aspect well. Nowadays, there are different techniques in the literature (feature importance, feature effects, feature interactions) that make these models interpretable.

Comment 2

Enhance the existing literature review in the Introduction by including other feature importance techniques that can be used in machine learning and mentioning hydrology studies where these techniques have been used and compared.

Comment 3

SHAP is a well-known technique in the literature. Justify why this particular technique was chosen, highlighting its strengths, weaknesses, and underlying assumptions/hypotheses.

Comment 4

Section 3.4 ("Interpretable machine learning method") is not very clear. It would be convenient to provide a detailed description of the method from a mathematical point of view.

Comment 5

Typically, machine/deep learning models achieve high performance (in terms of predictive performance) because they are trained on large training sets. Do you think the size of your training set allows the model to perform adequate training? Clarify this aspect in the discussion by highlighting that the sample size of the training set is one of the key factors for the performance of such models.

Comment 6

The proposed analysis is based on monthly data. Would you expect variations in the results using daily or hourly data? In this analysis, capturing short-term patterns could be particularly useful? Clarify this point in the discussion.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have responded satisfactorily to most of the concerns raised and have improved the manuscript significantly. However, the main concern regarding the choice of the input variables is not addressed. While choosing the input variable from among the nine atmospheric circulation factors, the correlation coefficient is calculated only for zero lag. However, some of the atmospheric circulation factors may have a delayed impact on the streamflow. It is suggested that time delay impact analysis may be performed on all the atmospheric circulation factors instead of confining it to only NPI.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

It is scientific to use more weather station data for spatial interpolation. But, for basins, the boundary line of basins is the ridge line, which divides mountains into windward slopes and leeward slopes. Generally, the precipitation on windward slopes is more than that on leeward slopes, and the laws of precipitation on windward slopes and leeward slops are also different. Please explain how to consider the difference between the precipitation on windward slopes and leeward slops in the spatial interpolation.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The quality of the paper is improved. However, I have few comments to share with the authors as listed in the following, consequently my suggestion accepts the manuscript after a MINOR revision.

Comment 1
Sentence at lines 86-88: " Common Feature Importance interpretation methods include Local Interpretable Model-agnostic Explanations (LIME), Feature Importance and Partial Dependence Plots.” Note that Partial Dependence Plot is a Feature Effect tool. Probably, the authors want to highlight that the main interpretability methods are: Feature Importance, Feature Effects and Feature interactions as described in Molnar (2020). Among Feature Importance methods, we have LIME which is a local method.

Since results of a global importance measure are provided in the paper, it would be better to include only the most known/used global importance measures and the papers in which they have recently been applied.

Molnar C (2020) Interpretable machine learning. Lulu.com

Comment 2
Note that  in Equation (9) is the ML model and not the predicted value. What is N? |N| and |S|? Please explain the terms used and correct the sentence.

Comment 3
To help the reader, it would be useful to provide the mathematical formula of the SHAP importance measure (defined using the average of the absolute values of the Shapley values) in section 3.4.

Comment 4

In the sentences at lines 474-480: “In terms of data sets, the training set size allowed the models to be trained adequately. First, the training set was large enough in number to cover the diversity and complexity of the data, and the data set was taken from real data. In addition, from the results of model training, the models could achieve good performance, and there was no overfitting or underfitting phenomenon, with a certain generalization ability., the training set size allowed the model to be trained adequately.” Please quantitatively indicate the performance achieved by the models to make the reader understand the goodness of these models in the training step. Moreover, note that the last sentence has already been written at the beginning of the paragraph.

Comment 5

Note that the quality of all figures has deteriorated from the previous version of the paper.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 3

Reviewer 1 Report

The authors have addressed all the concerns raised in the previous version satisfactorily.

Reviewer 2 Report

I have no comments