The Short Time Prediction of the Dst Index Based on the Long-Short Time Memory and Empirical Mode Decomposition–Long-Short Time Memory Models

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for sending me the paper “The short time prediction of the Dst index based on the LSTM

and the EMD-LSTM models”  by Zhang et al. This paper is a well-written paper and suitable for Applied Sciences journal.

 

This study aims to develop a Dst index prediction method based on the LSTM.

 I do not have any comments except few minor comment.

 Comment:

Authors are only able to predict one-hour ahead and predicted the Dst index evolution for next 7-days (168 hours). One hour is not sufficient enough to take necessary steps to protect our satellites. Is it possible to increase the prediction window a bit (at least one/two day ahead).

 

I am also not able to understand why authors say short term prediction. They predicted the Dst-index evolution one hour ahead but for next seven days. So, they already predicted the Dst-index 6-days ahead.

 LSTM is highly successful for time series prediction. What is the advantage of EAD-LSTM over the traditional LSTM?

 Line 228: “effectively reduces the fitting error” : How EAD-LSTM method is able to reduce fitting error? I think it is over fitting error (not fitting error).

 I will suggest the authors to make their developed code public in Github. It will also be good if the authors provide the Github link in the manuscript.

Comments on the Quality of English Language

No comments

Author Response

• Authors are only able to predict one-hour ahead and predicted the Dst index evolution for next 7-days (168 hours). One hour is not sufficient enough to take necessary steps to protect our satellites. Is it possible to increase the prediction window a bit (at least one/two day ahead).

 

Yes, as you say, we can expand the forecast window. The focus of this paper is about the effectiveness of two methods for Dst index prediction. We mainly tested the effect of different parameters on the prediction results. Expanding the prediction window needs more results to support it, which is the main concern in our future works.

 

• I am also not able to understand why authors say short term prediction. They predicted the Dst-index evolution one hour ahead but for next seven days. So, they already predicted the Dst-index 6-days ahead.

 

We used historical data to predict the length of seven days to verify the validity of the predictions. The predicted values for each single hour of seven days were figured out using the true values prior to that point in time. So in fact we can only make predictions one hour ahead, which is thus called "short-term forecast".

 

• LSTM is highly successful for time series prediction. What is the advantage of EAD-LSTM over the traditional LSTM?

 

The biggest advantage of the EMD-LSTM is that it significantly improves the lag of the prediction results. Also the prediction accuracy of the EMD-LSTM during magnetic storms is better than LSTM. We made a clearer description in conclusion part.

 

• Line 228: “effectively reduces the fitting error” : How EAD-LSTM method is able to reduce fitting error? I think it is over fitting error (not fitting error).

 

EMD is an algorithm that firstly decomposes a complex raw signal into multiple components. The complexity of these components decreases in descending order, then adopts LSTM to yield a better fitting result for most of these components which have relatively low complexity. Thus the error of the final combined prediction results is somewhat reduced, particular in geomagnetic storm period. This part of the methodology has been modified for a clearer description.

 

• I will suggest the authors to make their developed code public in Github. It will also be good if the authors provide the Github link in the manuscript.

 

This is a great suggestion, but our code is still being refined and we will make it public in the future when we have completed all the work.

Reviewer 2 Report

Comments and Suggestions for Authors

Very good article. Authors succintly developed the Machine learning methods LSTM for the magnetic storm prediction. Literature review aptly demonstrate the need for more accuate prediction methods like ML and Deep learning techniques.

Only one important aspect I wish to get clarified is that why no attempt is made to include the most crucial parameters affecting the magnetic storm prediction ring current index and magnetic perturbation index is not included in the present LSTM models. (Although the authors quoted it's importance in the line 412).

Overall very good article. 

Author Response

• Only one important aspect I wish to get clarified is that why no attempt is made to include the most crucial parameters affecting the magnetic storm prediction ring current index and magnetic perturbation index is not included in the present LSTM models. (Although the authors quoted it's importance in the line 412).

 

You are correct, we have paid more attentions to the effectiveness of these methods in Dst index prediction, and want to find out the characteristics of these two models. We will, of course, add these important constraints to the future work improve the reliability of model.

Reviewer 3 Report

Comments and Suggestions for Authors

Review is attached.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Fair.

Author Response

• There is no dedicated Literature Review section. Even though the related work is mentioned in the introduction, it should be placed in a separate section and identify the research gap you are targeting. Contrast your planned contribution against the publish work. I prefer to see a table summarizing this literature contrast.

 

Thank you for this good suggestion. We have already revised and manuscript and added the table.

 

• What is your research hypothesis? Is it the study the effectiveness of deep learning in Dst index prediction? If so, why you only choose LSTM and EMD-LSTM models? You should include other recent and state of the art DL approaches to have a fair comparison.

 

We are investigating the effectiveness of deep learning in Dst index prediction. As you mentioned, we are also working on research using newer deep learning methods at the moment. LSTM and EMD-LSTM have been illustrated a good performance while other methods (e.g. ANN, CNN, etc) are not very appropriate to long time-series data prediction. This has explained in the paper.

 

• You are tackling a time series problem, and you need to have a fully detailed discussions on the data used and its characteristics, as it will greatly affect your whole experiment findings. However, Section 2 (Data) is quite small and not informative. This section needs to be fully revised, e.g. more on the data selection criteria and whether it was used by other studies? Mention the type of the problem whether its univariate or multivariate analysis?

 

A single variable is used for prediction in this study. Moreover, the Dst index is not a direct observation, the data source has already processed it and therefore it can be used directly. Since part of Dst data are newest one (to May, 2023), which has not yet been used by others. The related descriptions have been changed.

 

• In EMD-LSTM, how do you combine the predictions? Is it averaging the predictions from all models? In figure 5, it shows them as summation, which is not clear. Can you explain it.

 

The EMD decomposes the original data into several components, and the original data is obtained by summing these components, which yields the prediction result. It has been changed in manuscript for a clearer description.

 

• Was the LSTMSs in EMD-LSTM optimized, as you did with the stand alone LSTM. Can you clarify this point.

 

The LSTM in EMD-LSTM is completely identical to the stand alone LSTM.

 

• Error in Figure 5. Should be LSTM2 instead IMF2.

 

Figure 5 has been corrected.

 

• Some minor styling issues:

o Rename sections names to be more meaningful. For instance, the last section shouldn’t be named (Conclusion and discussions).

o Paragraphs in the conclusion section shouldn’t be numbered in this way.

o Refer to the figures in the text with their numbers, rather than “the above figure”.

 

All of the above details have been corrected.

Reviewer 4 Report

Comments and Suggestions for Authors

Article: The short time prediction of the Dst index based on LSTM and the EMD_LSTM models

 

General Comment:

The authors have presented a good topic of interest. However, efforts should be made to make the work of interest to ML readers; make clear what new concepts are highlighted. May be more details about the data aggregation component can be considered. However, if the work is targeting researchers in the fields of space weather, this comment can be an option.

 

Abstract:

The abstract is well written and it is clear. Th authors should revisit line 25..

 “accuracy of the LSTM model in short time period is slightly better than the EMD-LSTM model”.. and provide the actual “value” of the accuracy of LSTM model.

 

Introduction:

Well written. I suggest the authors separate the background information from empirical literature. There has to be a section for “related works”. This section should justify the stated rationale/motivation for this work as already clarified by the authors.

 

Materials and methods:

Data – should indicate what parameters and nature of collected data.

 

Results:

Well presented

 

Discussion:

Authors must separate this from conclusions. Maintain a separate “discussions” section

 

 

 

Author Response

• The authors have presented a good topic of interest. However, efforts should be made to make the work of interest to ML readers; make clear what new concepts are highlighted. May be more details about the data aggregation component can be considered. However, if the work is targeting researchers in the fields of space weather, this comment can be an option.

 

You are right, this paper is relevant to the field of machine learning. We just focus on the effectiveness of these methods in Dst index prediction by using the empirical ML methods LSTM and its upgraded version EMD-LSTM. We think this manuscript is appropriate to readers of both ML and space weather fields.

 

• The abstract is well written and it is clear. Th authors should revisit line 25.“accuracy of the LSTM model in short time period is slightly better than the EMD-LSTM model”. and provide the actual “value” of the accuracy of LSTM model.

 

RMSE (Root Mean Square Error) and CC (correlation coefficient) are the most commonly used metrics in deep learning. We believe that the data listed in the paragraph is sufficient to represent the accuracy of the model.

 

• Well written. I suggest the authors separate the background information from empirical literature. There has to be a section for “related works”. This section should justify the stated rationale/motivation for this work as already clarified by the authors.

 

Thank you for the suggestion. The section has been corrected accordingly.

 

• Data – should indicate what parameters and nature of collected data.

 

Thank you for the suggestion. This part of the narrative has been corrected accordingly.

 

• Authors must separate this from conclusions. Maintain a separate “discussions” section.

 

Thank you for the suggestion. The section has been corrected accordingly.

Reviewer 5 Report

Comments and Suggestions for Authors

The study used two machine learning models, LSTM and EMD-LSTM, to model and predict the Dst index. The LSTM model had slightly better prediction accuracy in the short time period compared to the EMD-LSTM model. However, the EMD-LSTM model was able to solve the problem of prediction lag and could better predict geomagnetic storms.  The validity and accuracy of the models were tested by modeling and predicting the Dst index for solar quiet and active periods, respectively. The predictions were then analyzed. 

The following questions need to be addressed:

The paper does not mention any comparison with other prediction models, which limits the understanding of the relative performance of the LSTM and EMD-LSTM models. I suggest referencing one or two other models for training and then comparing the success rate and timeliness of the predictions.

The paper does not discuss the potential impact of external factors or variables on the accuracy of the predictions, such as solar activity or other geomagnetic indices.

The paper does not address the computational requirements or efficiency of the LSTM and EMD-LSTM models, which could be important considerations for real-time prediction applications.

In section 2.2 on data preprocessing, the first sentence needs to be cited. In this section, you mention the sources of the test and training sets, but the title is "data preprocessing". Shouldn't you describe in detail how you processed, screened, and aligned the data? Moreover, the entire article does not provide information on the size or representativeness of the dataset used for training and testing the models.

Please pay attention to the data format. The article has many occurrences of ‘2015.01.01', as well as different types like 'May 21, 2023'.

 Figures 1 and 2 have a lot of white space. Why is there so much blank space?

  Comments on the Quality of English Language

Please pay attention to the data format.

Author Response

• The paper does not mention any comparison with other prediction models, which limits the understanding of the relative performance of the LSTM and EMD-LSTM models. I suggest referencing one or two other models for training and then comparing the success rate and timeliness of the predictions.

 

You are quite right. Our research focuses on testing the effectiveness of these two methods for Dst index prediction and testing which factors can influence the prediction results. After long term experiments, we found that these two methods work well for predicting long term series data like Dst index. Due to these experiences, we are currently conducting research on the prediction of Dst index using some other deep learning methods as well.

 

• The paper does not discuss the potential impact of external factors or variables on the accuracy of the predictions, such as solar activity or other geomagnetic indices.

 

You are correct, we have paid more attentions to the effectiveness of these methods in Dst index prediction, and want to find out the characteristics of these two models. We will, of course, add these important constraints to the future work improve the reliability of model.

 

• The paper does not address the computational requirements or efficiency of the LSTM and EMD-LSTM models, which could be important considerations for real-time prediction applications.

 

As you say. In this study, only a single parameter was used for fitting and prediction, in order to test the validity of these methods in the prediction of the Dst index. We will add other indices that have a better correlation with the Dst index to the model in the future, and we will also focus on the computational requirements or efficiency of the model.

 

• In section 2.2 on data preprocessing, the first sentence needs to be cited. In this section, you mention the sources of the test and training sets, but the title is "data preprocessing". Shouldn't you describe in detail how you processed, screened, and aligned the data? Moreover, the entire article does not provide information on the size or representativeness of the dataset used for training and testing the models.

 

As you say. We added the citation. We have changed the narrative and title of this section to make them more consistent. In addition, in the section "Results" we have stated in different parts which data were used for training in that part of the results. For example: First, we selected 90-day (90 days since January 1, 2015) and 1096-day (three years since January 1, 2015) as training data to test the performance of the LSTM model for short-term (one-hour ahead) prediction, which aims to compare the influences on the prediction accuracy at different learning rates and lengths of training data.

 

• Please pay attention to the data format. The article has many occurrences of ‘2015.01.01', as well as different types like 'May 21, 2023'.

 

Thank you for your suggestion. These details have been revised.

 

• Figures 1 and 2 have a lot of white space. Why is there so much blank space?

 

Because they are part of Figure 4. Figure 4 is complex. We have splitted it for a clear narrative.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

Review is attached.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Fair.

Author Response

• Related studies section is indeed a great addition. However, I find the summary table not informative and not readable. Refine it to make it more informative and related to shows your paper contributions. For instance, instead of Methodology Used, use ML and DL columns to show the utilized models in each study. Many other refinements are possible. Therefore, this table needs serious attention.

 

Thank you for your suggestions. We have corrected the table. In addition, we summarized the limitations of these studies, as well as the advantages of our study in the paragraphs following the table.

 

• If you are working on research using newer deep learning methods now, then add this and other future points as future work direction in the conclusion section.

 

We have corrected the manuscript.

 

• Section 7 is confusing in its current form. I see it merging both future work and threats of validity. I suggest you remove this section and merge what is appropriate with the conclusion section. In addition, create a threats of validity section just before the conclusion section to discuss the study threats of validity and how you tried to mitigate them.

 

Section 7 simply clarifies the shortcomings in this study and emphasizes our future work. Also. The last two sections are actually one paragraph, and was separated into two parts according to the comments of the other reviewer.

Reviewer 5 Report

Comments and Suggestions for Authors

The author partially answered my question, and given the space constraints, I think the current state is acceptable.

  Comments on the Quality of English Language

 Moderate editing of English language required.

 

Author Response

• Moderate editing of English language required.

 

Language has been edited.