Time-Series Feature Selection for Solar Flare Forecasting

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

In this paper, The authors introduced an ensemble method to improve the classification of solar flares. They used various feature selection techniques to identify key features in solar flare data. By combining the results of these techniques, they created a robust model for predicting solar flares. Unfortunately, the paper's figures lack clarity and fail to support the authors' claims. Given its current form, I do not recommend acceptance for publication. 

- The authors should clarify the terminology used throughout the paper. The distinction between "flare quite" and "non-flare" is unclear and should be rectified.

- The statement "As noted by He et al." requires a corresponding in-text citation to support the claim.

- The paper lacks a clear definition of "flare." Given that A and B class flares cannot be reliably detected, the authors should elaborate on the criteria used to identify flares for this study. The classification of flares into a binary system requires further explanation. Counting B events as X events could introduce noise into the data due to their unreliable detection. 

- The distinction between "non-flaring active regions" and "flaring active regions" or "flares" is ambiguous. An active region can produce multiple flares, so it is essential to clarify which is being considered.

- Partitions time spans should be defined explicitly, and the potential impact of the solar cycle on these portions should be addressed.

- Converting data to a binary classification simplifies the analysis, it is crucial to evaluate potential biases towards smaller class flares.

- Figures 5-8 present challenges in terms of readability. The error bars overlap, making it difficult to visualise the data.

- The colour scheme should be considered for colourblind readers. Overall, for Figures 5-8, I would recommend an alternative visualisation technique. 

Author Response

NOTE: PLEASE SEE THE PDF FILE WHICH INCLUDES FIGURES

 

 

Manuscript ID: universe-3172122  

Time-Series Feature Selection for Solar Flare Forecasting 

Responses to the Issues Raised by the Reviewer-1 

The authors would like to express sincere gratitude to the anonymous reviewer for their helpful and constructive comments. We also appreciate the reviewer’s insightful observations. In this document, the following sections present the responses of the authors to the issues raised by the reviewer. 

 

- The authors should clarify the terminology used throughout the paper. The distinction between "flare quite" and "non-flare" is unclear and should be rectified. 

We have added the content to explain the distinction between Flare quiet and “Non-Flare” in the manuscript. The explanation can be seen between lines 151-153 and 174 – 177 in the revised version of the manuscript. 

- The statement "As noted by He et al." requires a corresponding in-text citation to support the claim. 

We have changed the corresponding in-text citation in the Manuscript. 

- The paper lacks a clear definition of "flare." Given that A and B class flares cannot be reliably detected, the authors should elaborate on the criteria used to identify flares for this study. The classification of flares into a binary system requires further explanation. Counting B events as X events could introduce noise into the data due to their unreliable detection.  

The flares in our study are identified using data from the GOES satellite, which classifies flares into A, B, C, M, and X categories based on their peak X-ray flux. A-class flares are excluded from the SWAN-SF data due to the significant challenges associated with their detection. A-class flares, being the smallest and least intense, often fall below the detection threshold of the instruments used, particularly during periods of high solar activity when background levels are elevated. This exclusion is intentional to maintain data quality and avoid introducing noise from undetected or misclassified low-intensity events. 

In our binary classification system, we have carefully considered the detection challenges associated with B and C class flares, as well as Flare Quiet (FQ) periods, by classifying them as 'non-flaring' events. Only M and X class flares, which are more intense and reliably detected, are classified as 'flaring' events. This approach helps mitigate the risk of introducing noise into the data by avoiding reliance on the less reliably detected B-class flares. Additionally, to prevent any potential misclassification of B-class flares as more significant flares (such as X class), we explicitly separate these classes in our analysis. B-class flares are grouped with non-flaring events, maintaining the integrity of our data and ensuring that our predictive models are based on accurately detected and significant solar flaring events. 

As the other previous works investigated in Solar Flare Literature, M and X are Combined to be in a class called "flare classes". The rest of the classes are combined as "Non-Flare" classes. This Should Avoid any unreliable Detection. This is explained in the updated version of the manuscript between lines 143-158 and 179-192. 

 

- The distinction between "non-flaring active regions" and "flaring active regions" or "flares" is ambiguous. An active region can produce multiple flares, so it is essential to clarify which is being considered. 

 We have added the content to clarify this by explaining each term used and how the binary conversion is considered. The changes in the Manuscript can be seen between the lines 151-158 and 174 – 177 

 

- Partitions time spans should be defined explicitly, and the potential impact of the solar cycle on these portions should be addressed. 

Thank you for the suggestion. We explicitly mentioned the time spans for each partition in the dataset in the new version of the Manuscript. SWAN-SF dataset only covers the solar cycle:  "24". Since we have only one solar cycle for the SWAN-SF data and this cannot be investigated in depth. These details can be seen in the manuscript between lines 160 – 168. Please refer to the highlighted text in the draft. However, we plan to investigate the effect of solar cycles on our solar flare predictions by collecting data from other sources than SWAN-SF in our future directories. 

 

- Converting data to a binary classification simplifies the analysis, it is crucial to evaluate potential biases towards smaller class flares. 

Converting the data into a binary classification system simplifies the analysis, allowing us to focus on distinguishing between significant flaring events (M and X classes) and non-flaring events (B and C classes, along with Flare Quiet periods). However, we recognize the importance of evaluating potential biases that this approach might introduce, particularly towards the smaller class flares (B and C classes). 

To mitigate these biases, we conducted a thorough sensitivity analysis to ensure that our classification approach does not disproportionately ignore or misrepresent the smaller class flares. By using metrics like the True Skill Statistic (TSS), which balances sensitivity and specificity, we ensure that our model accurately distinguishes between flaring and non-flaring events without unduly favoring the more frequent non-flare classes. This metric is particularly effective in dealing with class imbalance, providing a robust measure of model performance that accounts for the potential impact of smaller flare classes within our binary system. 

- Figures 5-8 present challenges in terms of readability. The error bars overlap, making it difficult to visualise the data. 

After a deeper analysis, we discovered that our TSS values are not normally distributed, which means using confidence intervals wasn't the best approach. To improve the accuracy and readability of our results, we decided to remove the confidence intervals and present the visualizations in a more straightforward manner. 

For the references we have created the boxplots for each subset (which is tested on 4 different train – test data) of feature over different classifiers which can be seen below, and these plots can be added to the manuscript if needed. Please let us know if we should add these box-plots to the manuscript or not. 

 

Rocket: 

 

RandomForest: 

 

MiniRocket: 

 

 

- The colour scheme should be considered for colourblind readers. Overall, for Figures 5-8, I would recommend an alternative visualisation technique.  

We have changed the color schemes of the visualizations based on the readability of the color-blind readers. We used colors [ '#1f77b4',  # Navy Blue;  '#ff7f0e',  # Burnt Orange ;  '#7f7f7f',;  # Deep Grey (near black);  '#9467bd'   # Deep Purple ] 

 

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript is devoted to develop an ensemble approach that integrates feature selection methods and classifiers to improve solar flare classification employing future selection techniques including Mutual Information, Minimum Redundancy, Maximum Relevance, and Euclidean Distance, to identify the most relevant magnetic field parameters from SWAN-SF multivariate time series data from May 1, 2010, to August 31, 2018 . The manuscript may have the potential to contribute to understanding solar flare. But there are some problems in the manuscript that should be taken into account.

1.     Figure 1. When I count CMX flares taken from space weather prediction center (SWPC) for the same time interval I see that the total number of these classes flares is almost half of SWAN-SF flare number. The reason of this difference has to be presented with detail in the manuscript. 

 2.     Figure1. To increase the visibility of all flare classes the figure should be prepared in a better way. It is clearly known that the number of flare is controversial with their power. So, something is wrong in this plot because the number of B flares should be higher than the C flares. Also, the description of FQ should be presented in a better way.

 3.     Line 307. Authors presented the confidence interval calculation for a normally distributed data. Before applying it, it should be checked whether the data is normally distributed.

4.     From Figure 5 to 8. When I take into account error bars I can say that all methods show similar performance or the performance of ensemble method is not significant compared to others. This point should be clarified.

 Some small corrections

1.     From Figure 5 to 8. The error bars are not clearly separable. Please make error bars transparent and use different colors.

2.     Some references are not complete and some abbreviations are not described in the manuscript. The manuscript should be checked for typo and other errors

Overall, I think the current version of the manuscript can not be acceptable for the publication.

 

Comments on the Quality of English Language

English is fine

Author Response

NOTE: PLEASE SEE THE PDF FILE WHICH INCLUDES FIGURES.

 

 

Manuscript ID: universe-3172122  

Time-Series Feature Selection for Solar Flare Forecasting 

Responses to the Issues Raised by the Reviewer-2 

The authors would like to express sincere gratitude to the anonymous reviewer for their helpful and constructive comments. We also appreciate the reviewer’s insightful observations. In this document, the following sections present the responses of the authors to the issues raised by the reviewer. 

 

1. Figure 1. When I count CMX flares taken from space weather prediction center (SWPC) for the same time interval I see that the total number of these classes flares is almost half of SWAN-SF flare number. The reason of this difference has to be presented with detail in the manuscript. 

We appreciate your careful consideration on this. We Collected the Dataset from Swan-SF which is a well-recognized Dataset in Solar Flare Venue. They collect the data from different sources like Solar Dynamics Observatory Helioseismic and Magnetic Imager (SDO/HMI) Active Region Patches (SHARPs), managed by the Joint Science Operations Center (JSOC). It also includes Geostationary Operational Environmental Satellite (GOES) flare records, enhanced with additional data from Hinode-XRT to ensure precise flare location verification. SWPC uses GOES data and there is some Missing Data from GOES due to mission Failures which might be one of the reasons for the difference in Data count. In-depth investigation of the difference between the two data centers is beyond the scope of our research in this paper, however it would be interesting to analyze this issue in future work. 

2. Figure1. To increase the visibility of all flare classes the figure should be prepared in a better way. It is clearly known that the number of flare is controversial with their power. So, something is wrong in this plot because the number of B flares should be higher than the C flares. Also, the description of FQ should be presented in a better way.

We reverified the Figure 1 data and corrected the visualization plot now the plot supports the above statement, and the corrected Figure 1 has the proper ratio of B and C Flares. 

The Description for the FQ is given in lines 151-153 in the Updated version of the Manuscript 

 

3. Line 307. Authors presented the confidence interval calculation for a normally distributed data. Before applying it, it should be checked whether the data is normally distributed.

Thank you for your careful attention to this. We have reverified the data and come to know that data is not normally distributed. So, we have removed the error bars for better readability. We have created the boxplots for each subset (which is tested on 4 different train – test data) of feature over different classifiers which can be seen below, and these plots can be added to the manuscript if needed. Please let us know if we should add these boxplots to the manuscript or not 

 

  

Rocket: 

 

RandomForest: 

 

MiniRocket: 

 

4. From Figure 5 to 8. When I take into account error bars, I can say that all methods show similar performance, or the performance of ensemble method is not significant compared to others. This point should be clarified.

As we mentioned in the previous comment, we found that our TSS values are not normally distributed, which accounts for the overlaps in the error bars. As a result, using confidence intervals was not the most appropriate method. To enhance both accuracy and clarity, we have removed the confidence intervals and used more straightforward visualizations. Please see the updated manuscript and please let us know if you want us to add the boxplots to the manuscript. 

 

 Some small corrections 

1. From Figure 5 to 8. The error bars are not clearly separable. Please make error bars transparent and use different colors.

We have changed the Visualizations from Fig 5 – 8. 

2. Some references are not complete, and some abbreviations are not described in the manuscript. The manuscript should be checked for typo and other errors

We have corrected the References and added the abbreviations. 

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I have completed my review of the revised manuscript. I am pleased to confirm that the authors have addressed all the concerns raised in the previous review. 

Comments on the Quality of English Language

Overall, the English language in the paper is good, and the manuscript is clear and well-written. There are only a few sentences where rephrasing could enhance readability. However, these are minor issues. 

Author Response

Manuscript ID: universe-3172122  

Time-Series Feature Selection for Solar Flare Forecasting 

Responses to the Issues Raised by the Reviewer-1

 

All the questions asked by the Reviewer-1 has been answered already

Reviewer 2 Report

Comments and Suggestions for Authors

Thank to authors, the manuscript improved based on my suggestions but still there are some serious problems in the manuscript that should be corrected. 

1.     In my previous report I commented about the difference of flare number between SWAN-SF and SWPC. Authors responded that “They collect the data from different sources like Solar Dynamics Observatory Helioseismic and Magnetic Imager (SDO/HMI) Active Region Patches (SHARPs), managed by the Joint Science Operations Center (JSOC). It also includes Geostationary Operational Environmental Satellite (GOES) flare records, enhanced with additional data from Hinode-XRT to ensure precise flare location verification. SWPC uses GOES data and there is some Missing Data from GOES due to mission Failures which might be one of the reasons for the difference in Data count. In-depth investigation of the difference between the two data centers is beyond the scope of our research in this paper, however it would be interesting to analyze this issue in future work.” I) I think almost twice more flare can not detect due to mission failures. I strongly recommend checking for duplicate records, II) Authors say that “In-depth investigation of the difference between the two data centers is beyond the scope of our research in this paper.” It should be scope of this paper, because any error/duplicate count in the used data will bring erroneous results and conclusions.  This point should be clarified and discussed in the manuscript.

2.     There are also overlapping small times between different partitions: That mean some data points counted twice in the analysis. I think the multiple data points may affect the results. Therefore, the time overlapping in the partitions should be removed/corrected.  

3.     Line 187-192. Authors mentioned that “In our refined approach, B and C class flares, as well as Flare Quiet (FQ) periods, are categorized as ’non-flare’ events to prevent the misclassification of less intense flares as significant events. Only M and X class flares, known for their intensity and reliable detection, are classified as ’flare’ events.” This approximation is not correct. Although X and M class flares are very powerful, it is not correct to classify C class flares as non-flare. Some C class flares (generally stronger than C5.0) can have serious geomagnetic effects.

4.     In the response to reviewer report authors mention that “We have reverified the data and come to know that data is not normally distributed.” This information should be added to the appropriate location in the manuscript.  

Overall, I think the current version of the manuscript can not be acceptable for the publication.

Author Response

Manuscript ID: universe-3172122  

Time-Series Feature Selection for Solar Flare Forecasting 

Responses to the Issues Raised by the Reviewer-2 

1. In my previous report I commented about the difference of flare number between SWAN-SF and SWPC. Authors responded that “They collect the data from different sources like Solar Dynamics Observatory Helioseismic and Magnetic Imager (SDO/HMI) Active Region Patches (SHARPs), managed by the Joint Science Operations Center (JSOC). It also includes Geostationary Operational Environmental Satellite (GOES) flare records, enhanced with additional data from Hinode-XRT to ensure precise flare location verification. SWPC uses GOES data and there is some Missing Data from GOES due to mission Failures which might be one of the reasons for the difference in Data count. In-depth investigation of the difference between the two data centers is beyond the scope of our research in this paper, however it would be interesting to analyze this issue in future work.” I) I think almost twice more flare can not detect due to mission failures. I strongly recommend checking for duplicate records, II) Authors say that “In-depth investigation of the difference between the two data centers is beyond the scope of our research in this paper.” It should be scope of this paper, because any error/duplicate count in the used data will bring erroneous results and conclusions. This point should be clarified and discussed in the manuscript.

  

Ans: The SWAN-SF dataset reports a higher number of detected solar flares compared to SWPC, which can be attributed to several factors such as its diverse data sources. For example, SWAN-SF utilizes resources like the Solar Dynamics Observatory's Helioseismic and Magnetic Imager (SDO/HMI), which provides detailed measurements of magnetic fields, the Hinode X-Ray Telescope for precise X-ray imaging, and the Geostationary Operational Environmental Satellites (GOES). Each source contributes unique attributes; SDO/HMI offers in-depth magnetic field vectors, while Hinode XRT provides accurate spatial coordinates (latitude and longitude), crucial for precisely locating and characterizing solar flares. 

Another reason for the higher flare count in SWAN-SF is its high-frequency data collection interval. Recording data every 12 minutes ensures continuous monitoring and capturing of transient solar activities that might otherwise go unnoticed in datasets with longer intervals. Additionally, SWAN-SF employs a 12-hour sliding observation window that shifts forward by one hour to capture the next sample. This method provides a thorough temporal coverage of solar activity, allowing for more flare detections by increasing the likelihood of capturing pre-flare and flare events that might be missed by other datasets. 

To prevent duplication of data records, SWAN-SF employs a rigorous data integration technique that meticulously aligns spatial and temporal data from various sources. This alignment ensures that flare events detected by different instruments are accurately consolidated into a coherent single record. To further ensure the reliability and accuracy of its data, SWAN-SF embeds precise time stamps with each 12-minute interval data point, enabling exact tracking and historical analysis of flare events. Cross-verification across multiple platforms is integral to the dataset's validation process, ensuring that all recorded flares are genuine and not duplicated, with each flare’s characteristics such as start and end times, peak intensities, and exact positions being thoroughly compared to reconcile any observational discrepancies. 

Related citations of the above content are added in the manuscript between the lines 143 - 154    

2. There are also overlapping small times between different partitions: That mean some data points counted twice in the analysis. I think the multiple data points may affect the results. Therefore, the time overlapping in the partitions should be removed/corrected. 

Ans: Due to the small time overlaps between partitions, we ensured that no two partitions were combined for training or testing. Instead, we trained on one partition and tested on the consecutive partition to avoid any duplicate recording or data points being counted twice. This approach helps maintain the integrity of the analysis by preventing the influence of overlapping data points on the results. 

 

3. Line 187-192. Authors mentioned that “In our refined approach, B and C class flares, as well as Flare Quiet (FQ) periods, are categorized as ’non-flare’ events to prevent the misclassification of less intense flares as significant events. Only M and X class flares, known for their intensity and reliable detection, are classified as ’flare’ events.” This approximation is not correct. Although X and M class flares are very powerful, it is not correct to classify C class flares as non-flare. Some C class flares (generally stronger than C5.0) can have serious geomagnetic effects.

Ans: Thank you for your insightful feedback. We have revised the classification to distinguish between "Minor" and "Major" flares, rather than categorizing C-class flares as non-flare events. This adjustment acknowledges that certain C-class flares, particularly those above C5.0, can indeed have significant geomagnetic impacts. We believe this revised terminology better reflects the spectrum of flare intensities while preserving the distinction necessary for our model's focus on higher-impact events.  

 

4. In the response to reviewer report authors mention that “We have reverified the data and come to know that data is not normally distributed.” This information should be added to the appropriate location in the manuscript. 

Ans: Added in lines 344- 345. 

 

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

Thanks to authors, they take into account my suggestions and revise the manuscript. I have no more comment/suggestion.