Review Reports - An Intelligent Rockburst Prediction Model Based on Scorecard Methodology

Round 1

Reviewer 1 Report

The manuscript deals with developing a rockburst prediction model in the framework of a warning system in cases of underground mining. This issue is essential and challenging because rockburst is a significant hazard in mining. The procedure that was followed concerns the definition of three train datasets where the risk classes’ combination is different each time and the development of scorecards. Based on the literature research, some rockburst prediction indicators were selected to check their influence on the phenomenon. Furthermore, an application of 311 real-life rockburst cases was employed. The innovative methodology that is applied in this study shows that the machine learning techniques concern a new and promising way for rockburst prediction.

However, there are some major concerns regarding the manuscript. A significant concern with the manuscript is that some parts of the text are written like a draft. In addition, from a methodological point of view, the structure of the manuscript needs to be improved concerning the coupling between the theoretical and the experimental analysis. In this framework, the research steps must be better demonstrated. Furthermore, the English language needs to be improved in several points of the manuscript.

Moreover, the methodological quality of the manuscript needs to be improved. The authors should clarify the research problem concerning the findings of other relevant research projects and pose the research questions. Furthermore, the original contribution should be better presented and demonstrated.

Additional comments and recommendations for the improvement of the manuscript:

General comments

A better presentation of the theoretical analysis of the methodology is required.
A more in-depth discussion of the results should be provided.
From an engineering point of view, the practical considerations of the research should be further discussed.
Please check extra or fewer spaces between the words in the sentences.
Please check punctuation and sentences’ separation in the whole manuscript.
The manuscript should follow the journal’s template format. In addition, the typical format of Introduction, Materials and Methods, Results, Discussion, and Conclusion is suggested.

Abstract

General notes: The abstract is well-structured and comprehensive for the reader. However, some points need to be improved.

[Line 17] “is a” instead of “are”

[Lines 18-23] This sentence should be shorter, maybe in the point before “finally”. In addition, it should be “based on” instead of “base on”.

[Line 28] “…setting a higher value…” instead of “…setting a larger value…”

Introduction

General notes: This section describes several methods that are used for the issue, but the research gap and the innovative contribution of the study are not incorporated. Furthermore, spaces are needed between the sentences, especially before the literature citations and the parentheses.

[Line 37] “…that…” is not needed.

[Line 38] “…which results in…” instead of “…which resulting in…”.

[Line 47] “…divided into two categories…” instead of “…divided into 2 categories…”.

[Line 48] “to be” instead of “is”.

[Line 65] “a method” instead of “an method”.

[Line 73] “understand” instead of ‘understudy”

[Lines 78-79 & 82] “adaptive adjustment” and “adjust adaptively” What do these two phrases mean?

[Line 80] “is” instead of “are”

[Lines 75-84] In this paragraph, two different fonts are used. It needs to be adjusted to the journal’s template.

[Lines 82-83] “However, the study of literature…” instead of “However, the literature the study…”.

[Lines 85-87, 87-89 & 96-97] “In this paper constructs…”, “In order to verify…” and “…and the improvement the accuracy…” The sentences need to be improved.

[Line 91] The machine learning models ( LR, DT, RF) could be further explained.

Intelligent prediction model of rockburst risk based on score card method

[Line 98] In the heading of section 2 “score card” should be “scorecard”.

[Line 100] “…consists of three main steps.” Instead of “…consists of 3 main steps.”

[Lines 100-101] First, some datasets…risk level.” This sentence needs to be improved.

[Figure 1] “Construct” instead of “Constucte”.

[Line 110] ”one versus one (OVO) and one versus all (OVA)” instead of “one-to-one (OVO) and one-to-over (OVA)” ?.

[Line 126] “…consisting of three scorecards…” instead of “..consisting of 3 scorecards…”

[Lines 117-119] The sentence needs to be improved. It does not make sense.

[Lines 146-149] The interpretation of the equation’s symbols should not be in the form of a paragraph because there are no completed sentences. Furthermore, in line 147, a space is needed after “S_i”. In line 148, “δ_ij” should have the same symbol as the equation (1). In line 150, the “s_ij” should be symbolized as in the equation.

[Line 157] “…the construction of…” instead of “…the approve of construct…”

[Line 161] What does the “*” in equation (3) mean?

[Line 164] The symbols of equation (4) need to be explained.

[Line 166] “…consists of four main steps.” instead of “…consists of 4 main steps.”

[Lines 167-168] The sentence needs to be improved.

[Line 173] “…has three advantages.” instead of “…has 3 advantages.”

[Line 181] “…method that combines…” instead of “…method that combine…”.

[Line 185] “Determination of the weight of evidence for each bin” instead of “Determine the weight of evidence for each bin”.

[Lines 199-201] This sentence needs to be improved.

[Lines 203-206] This sentence needs to be improved.

[Line 215] “Determination of each weight’s indicator” instead of “Determine the weights of each indicator”.

[Line 232] Determination of the scorecard scales and compensation factors” instead of “Determine the scorecard scales and compensation factors”.

[Line 244] “ACC, FAR, MAR” An explanation could be given.

[Line 247] In the column of “Actual risk level” of Table 1, “Modrate” needs to be corrected as “Moderate”.

[Line 257] “…were divided into four bins…” instead of “…were divided into 4 bins…”.

[Lines 262-263] This sentence needs to be improved.

Application of the intelligent rockburst risk prediction model

[Lines 266-268] “Risk” instead of “Hazard” ?. Table 1 regards the classification of risk levels; None, Light, Moderate, and Strong, but in this point, the “Light” is mentioned as “weak”. The correction of these characterizations should be changed throughout the text, as in Table 5.

[Line 274] The headings of Table 2 “dataset” and “ratios of safety and hazard samples” need to have the first letter in capitals. Furthermore, the “Safety sample” and “Hazard sample” headings need to show that it is about several samples.

[Line 275] “Through the literature research…, this study selected…” instead of “Through the literature survey…, this paper selects…”

[Lines 296-297] The sentence needs to be improved.

[Line 289] The column “Parameters” of Table 3 needs units.

[Line 302] “..risk prediction model…” instead of “…risk predict model…”

[Line 303] “was” instead of “were”.

[Line 307] “WOE” or “ WofE” instead of “woe”.

[Line 309] “…more important in the rockburst risk prediction process is” instead of “…more important it is in the rockburst risk prediction process.”

[Line 310] maybe “high” is better than “large” as it is about a value. Additionally, “The” should be “the”.

[Lines 310-314] The sentence needs to be improved. Please check the punctuation.

[Line 332] “So, The IRPSC…” instead of “So, the IRPSC…”.

[Line 336] The heading of this section seems not representative regarding the content of the section.

[Line 337] “The IRPSC was applied to eight rockburst cases…” instead of “The IRPSC was applied to 8 rockburst cases…”.

[Lines 341-342] “…accurately predicted six rockburst cases out of eight…” instead of “…accurately predicted 6 rockburst cases out of 8…”.

[Line 346] A full stop is needed after “respectively” instead of a comma. How the accuracy of the model is approved?

[Pages 11-12] The parameters of Table 5 need units. Table 5 needs to be further described and discussed.

Discussion

General Comments: An in-depth discussion of the results is needed.

[Line 4] “CART”, “RF” This is the first time that CART and RF are mentioned. An explanation is needed here.

[Lines 5-6] “The sample set was divided into five groups…and the remaining four groups…” instead of “The sample set is divided into 5 groups…and the remaining 4 groups…”.

[Line 12] “TPR” and “TNR” What do they mean? The “evaluating” is not needed.

[Line 40] “…takes into account…” The word “account” should be added here.

[Line 43] “that” is not needed. “the following” maybe is better.

[Line 44] The number of this equation is (13) and not (10). Furthermore, an explanation of the symbols is needed.

Conclusion

General notes: Conclusions should be improved, focusing on the original contribution of the research.

[Line 59] “…rockburst cases…” instead of “…rock burst cases…”

[Line 60] “…was applied to the eight rockburst cases…” instead of “…was applied to the 8 rockburst cases…”

[Line 66] A comma is needed after the “evidence” and “…LR based on dataset…” instead of “…LR base on dataset…”.

[Line 78] “…set higher class weights…” instead of “…set larger class weights…”

Author Response

Dear Reviewers:

Thank you very much for your careful review of this paper and helped us find and mark the language expression defects in the paper, which has played a great role in improving the readability and quality of the paper.

Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Revised portion are marked in red in the paper.

We tried our best to improve the manuscript and made some changes in the manuscript. These changes will not influence the content and framework of the paper.

The main corrections in the paper and the responds to the comments are as flowing:

General comments

[1] A better presentation of the theoretical analysis of the methodology is required.

Response: We have improved the manuscript. i.e. the description of the model construction method is modified in Section 2, “The establishment of the IRPSC consists of three main steps. First, through the combination of rockburst cases by risk levels, the data sets of different classification tasks are obtained. Then, the scorecards for evaluating different rockburst risk level were constructed based on the data sets using chiMerge, weights of evidence(WOE) and LR. Finally, the IRPSC was constructed by integrating the scorecards.” (p. 3, lines 116–121)

The application effect is deeply analyzed in Section 3. “In the rockburst risk scorecard, the higher the absolute value of the score of a bin the more important in the rockburst risk prediction. It can be seen from table 3 that the top three rockburst risk assessment indicators in absolute scores in the rockburst risk scorecard are the σ_θ, σ_θ/σ_c and W_et, which are important indicators for predicting rockburst risk levels. The highest and lowest absolute values of the scores for the W_et are large, and the W_et is the most important indicator among the rockburst risk assessment indicators used in this paper. Reducing the W_et of the rock is the key to reducing the rock burst risk level.” (p. 11, lines 321–328)

[2] A more in-depth discussion of the results should be provided.

Response: As Reviewer suggested that we revised the article，The content of result analysis is added:

(1)The real risk level of case 1 is strong, and the model incorrectly predicts a moderate, because the scorecard f₃ identified the case as safety. The σ_θ/σ_c of case 1 is 0.58, which is in the 2nd bins and is the main reason for the incorrect prediction on the scorecard f₃. The score card f₂ incorrectly predicts case 2 as a hazard case, resulting in the actual light dangerous case 2 being predicted as a moderate. This is mainly because case 2 has high σ_θ and Wet values, in particular, especially in case 2 has a σ_θ value of 91.3 MPa and a score of -10 on the scorecard f₂, which is the lowest score for this indicator on the scorecard f₂. (p. 11, lines 360–368)

(2) In addition, the IRPSC transforms the abstract machine learning models into a tabular form, and the evaluation process is simple and transparent, which overcomes the problem of black box of machine learning models. Our results are not only accurate in predicting rockburst risk, but also in identifying the dominant factors contributing to rockburst risk in a way that previous rockburst prediction methods have not been able to do. (p. 14, lines 395–400)

[3] From an engineering point of view, the practical considerations of the research should be further discussed.

Response: As Reviewer suggested that we modify the article and point out the areas that need to be improved in the application of the model. The model predicts a high degree of accuracy. However, the model's predictions are more skewed towards moderate hazards, meaning that the model is too conservative when analysis cases with extreme hazards and safety, which is an area for further improvement. (p. 11, lines 370–372)

[4] Please check extra or fewer spaces between the words in the sentences.

Response: We are very sorry for our negligence. We carefully checked the redundant or missing spaces in the text to improve the readability of the manuscript.

[5] Please check punctuation and sentences' separation in the whole manuscript.

Response: We are very sorry for our negligence. We carefully checked the redundant or missing spaces in the text to improve the readability of the manuscript.

[6] The manuscript should follow the journal's template format. In addition, the typical format of Introduction, Materials and Methods, Results, Discussion, and Conclusion is suggested.

Response: We have made correction according to the Reviewer's suggested. The first level heading of the manuscript is changed to 1. Introduction, 2 Methods, 3 Results, 4 Discussion and 5. Conclusion

Abstract

General notes: The abstract is well-structured and comprehensive for the reader. However, some points need to be improved.

[Line 17] “is a” instead of “are”

Response: We have made correction according to the Reviewer's comments. (p. 1, lines 18).

[Lines 18-23] This sentence should be shorter, maybe in the point before “finally”. In addition, it should be “based on” instead of “base on”.

Response: We have made correction according to the Reviewer's comments. ”the multi-classified rockburst prediction problem is decomposed into three binary tasks, and” was delete. (p. 1, lines 19-25).

[Line 28] “…setting a higher value…” instead of “…setting a larger value…”

Response: We have made correction according to the Reviewer's comments. (p. 1, lines 28).

Introduction

Response: Thanks the reviewer for your kind comments. We made related revisions in the Introduction. The innovative contribution of this work is the coupled use of scorecard methodology and machine learning for rockburst risk prediction, which is a first attempt. The work not only expands the application of the method but also improves rockburst prediction. Our results are not only accurate in predicting rockburst risk, but also in identifying the dominant factors contributing to rockburst risk in a way that previous rockburst prediction methods have not been able to do.

More details are as follows:

As mentioned in the introduction, Rockburst evaluation is of great significance. (p. 2, lines 44–48) Rockburst prediction methods can be divided into two categories: empirical approach and mathematical models, with mathematical models to be divided into uncertainty theory algorithmic models and machine learning models. (p. 2, lines 49–51)

The indicator thresholds are generally determined by experts through the results of data analysis and engineering experience, however, due to the suddenness and complexity of rockburst, the thresholds for rockburst occurrence under different geological conditions are generally different and the determination of thresholds is difficult. (p. 2, lines 56–59)

In practical engineering, we not only hope that the rockburst prediction model can have high prediction accuracy, but also hope to master the risk indicator affecting the rockburst risk level and understand what indicator lead the model to make such prediction results. In other words, the ideal model not only has high prediction ability, but also has interpretability. Machine learning models are susceptible to irrelevant features and are not robust to outliers, in addition the prediction process is difficult for the human to understand, because some machine learning models have black box properties. (p. 2, lines 77–85)

the IRPSC transforms the abstract machine learning models into a tabular form, and the evaluation process is simple and transparent, which overcomes the problem of black box of machine learning models. (p. 11, lines 346–349)

[Line 37] “…that…” is not needed.

Response: We have made correction according to the Reviewer's comments. (p. 1, lines 38).

[Line 38] “…which results in…” instead of “…which resulting in…”.

Response: We have made correction according to the Reviewer's comments. (p. 1, lines 39).

[Line 47] “…divided into two categories…” instead of “…divided into 2 categories…”.

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 49).

[Line 48] “to be” instead of “is”.

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 50).

[Line 65] “a method” instead of “an method”.

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 71).

[Line 73] “understand” instead of ‘understudy”

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 84).

[Lines 78-79 & 82] “adaptive adjustment” and “adjust adaptively” What do these two phrases mean?

Response: Adaptive adjustment refers to the use of machine learning algorithm in the process of building the model, so when a new sample case is generated, the critical value and weight value of the model evaluation index can be adjusted adaptively.

As in the text “Different rockburst cases set will obtain different threshold and scores, achieving adaptive adjustment of the critical values and weight of the indicators.” (p. 10, lines 340–341)

[Line 80] “is” instead of “are”

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 98).

[Lines 75-84] In this paragraph, two different fonts are used. It needs to be adjusted to the journal's template.

Response: We have made correction according to the Reviewer's comments. Set the full text font to Palatino Linotype.

[Lines 82-83] “However, the study of literature…” instead of “However, the literature the study…”.

Response: We have made correction according to the Reviewer's comments. (p. 2, lines 98–99).

[Lines 85-87, 87-89 & 96-97] “In this paper constructs…”, “In order to verify…” and “…and the improvement the accuracy…” The sentences need to be improved.

Response: We received comments from the reviewers and improved the sentences.

modify: In order to build a rockburst prediction model with high accuracy and identify the main factors affecting the rockburst risk level, an intelligent rockburst risk prediction model based on scorecard methodology (IRPSC) was constructed using 311 real rockburst cases, and the model was used to predict rockburst cases in Riverside hydro-power stations to verify the application effect of the IRPSC. Then, the prediction effects of the IRPSC and machine learning models such as LR, CART, RF and AdaBoost are compared and analyzed; Finally, the influence of sample category weight on the missing alarm rate, false alarm rate and accuracy of rockburst risk scorecard was studied. (p. 3, lines 102–109)

original: In this paper constructs an intelligent rockburst risk prediction model based on scorecard methodology(IRPSC) with strong interpretability by studying the application of scorecard in rockburst risk prediction. In order to verify the application effect of the IRPSC, constructed the IRPSC based on 311 real rockburst cases, and the model was used to predict the rockburst cases of Riverside hydro-power stations. In addition, the predictive effectiveness of the rockburst risk scorecard for rockburst risk was compared with machine learning models such as logistic regression(LR), CART, RF and AdaBoost. Finally the effect of sample weights on the miss alarm, false alarm and accuracy of the rockburst risk scorecard was investigated.

[Line 91] The machine learning models ( LR, DT, RF) could be further explained.

Response: LR, DT, RF and AdaBoost are classical machine learning algorithms, so they are not introduced in detail in this paper. At the same time, in order to facilitate understanding, we mark the classic machine learning materials here, including these classic machine learning methods and their applications. (p. 3, lines 108).

Intelligent prediction model of rockburst risk based on score card method

[Line 98] In the heading of section 2 “score card” should be “scorecard”.

Response: Thanks the reviewer for your kind comments. We have changed the heading of section 2 to Methods. (p. 3, lines 112).

[Line 100] “…consists of three main steps.” Instead of “…consists of 3 main steps.”

Response: We have made correction according to the Reviewer's comments. (p. 3, lines 114).

[Lines 100-101] First, some datasets…risk level.” This sentence needs to be improved.

Response: We received comments from the reviewers and improved the sentences.

modify: First, through the combination of rockburst cases by risk levels, the data sets of different classification tasks are obtained. Then, the scorecards for evaluating different rockburst risk level were constructed based on the data sets using chiMerge, weights of evidence(WOE) and LR. Finally, the IRPSC was constructed by integrating the scorecards.(p. 3, lines 114–118).

original: First, some datasets of different classification tasks by combining rockburst cases of different risk level. Second, construction of scorecards for evaluating different rockburst risk level based on datasets for different classification tasks using chiMerge, weights of evidence(WOE) and LR. Finally the scorecards are integrated to obtain an intelligent rockburst risk prediction model.

[Figure 1] “Construct” instead of “Constucte”.

Response: We have made correction according to the Reviewer's comments. (p. 4, lines 150).

[Line 110] ”one versus one (OVO) and one versus all (OVA)” instead of “one-to-one (OVO) and one-to-over (OVA)” ?.

Response: We have made correction according to the Reviewer's comments. (p. 3, lines 120-122).

[Line 126] “…consisting of three scorecards…” instead of “..consisting of 3 scorecards…”

Response: We have made correction according to the Reviewer's comments. (p. 3, lines 140).

[Lines 117-119] The sentence needs to be improved. It does not make sense.

Response: This sentence has been deleted from the revised version.

[Lines 146-149] The interpretation of the equation's symbols should not be in the form of a paragraph because there are no completed sentences. Furthermore, in line 147, a space is needed after “Si”. In line 148, “δij” should have the same symbol as the equation (1). In line 150, the “sij” should be symbolized as in the equation.

Response: We deleted the paragraph before the description and corrected the format error. (p. 5, lines 164–168)

[Line 157] “…the construction of…” instead of “…the approve of construct…”

Response: We have made correction according to the Reviewer's comments. (p. 5, lines 174).

[Line 161] What does the “*” in equation (3) mean?

Response: We are very sorry for our negligence of that, In fact, the equation (3) should not contain “*”. (p. 5, lines 177)

[Line 164] The symbols of equation (4) need to be explained.

Response: equation (4) is obtained by combining equations 1, 2 and 3. The symbol explanation in the formula has been given above, so it is not explained again here. (p. 5, lines 164–169 and lines 178–179)

[Line 166] “…consists of four main steps.” instead of “…consists of 4 main steps.”

Response: We have made correction according to the Reviewer's comments. (p. 5, lines 183).

[Lines 167-168] The sentence needs to be improved.

Response: We received comments from the reviewers and improved the sentences

modify: Firstly, each risk indicator is binned using the chiMerge to obtain the threshold value for each bin of the indicator. Then calculate the WOE w_ij for each bin based on the WOE theory, the weight of indicators θ is estimated according to the maximum likelihood method of LR model. Finally determine the values for the compensation factor A and the scorecard scale B. (p. 5, lines 183–187)

original: Firstly, each risk indicator is binned to obtain the critical values l_ij, h_ij for each bin of the risk indicator. Then calculate the WOE value w_ij for each bin based on the WOE theory. The weights of the indicators are then estimated according to the maximum likelihood method in the LR. Finally determine the values for the compensation factor A and the scorecard scale B

[Line 173] “…has three advantages.” instead of “…has 3 advantages.”

Response: We have made correction according to the Reviewer's comments. (p. 5, lines 190).

[Line 181] “…method that combines…” instead of “…method that combine…”.

Response: We have made correction according to the Reviewer's comments. (p. 5, lines 199).

[Line 185] “Determination of the weight of evidence for each bin” instead of “Determine the weight of evidence for each bin”.

Response: We have made correction according to the Reviewer's comments. (p. 6, lines 203).

[Lines 199-201] This sentence needs to be improved. and [Lines 203-206] This sentence needs to be improved.

Response: We received comments from the reviewers and improved the sentences.

modify: Where is the hazard sample rate for the j-th bin of the i-th indicator, which represents the ratio of the number of dangerous samples in the j-th bin of the i-th indicator to the total number of dangerous samples in the sample set, and is the safety sample rate for the j-th bin of the i-th indicator. (p. 6, lines 212–215)

original: where and are the safety sample rate and the hazard sample rate for the j-th bin of the i-th indicator, respectively. Representing the ratio of hazard samples in the j-th bin to all hazard samples in the dataset and the ratio of safety samples in the j-th bin to all safety samples in the dataset, respectively

[Line 215] “Determination of each weight's indicator” instead of “Determine the weights of each indicator”.

Response: We have made correction according to the Reviewer's comments. (p. 6, lines 216).

[Line 232] Determination of the scorecard scales and compensation factors” instead of “Determine the scorecard scales and compensation factors”.

Response: We have made correction according to the Reviewer's comments. (p. 6, lines 234).

[Line 244] “ACC, FAR, MAR” An explanation could be given.

Response: We added a new explanation to the text，The accuracy (ACC) the ratio of the number of samples whose predicted results agree with the actual results to the total number of samples. The false alarm rate(FAR) is defined as the ratio of false alarm cases to the total number of all cases, and the miss alarm rate(MAR) is the ratio of miss alarm cases to the total number of all cases. (p. 7, lines 246–50)

[Line 247] In the column of “Actual risk level” of Table 1, “Modrate” needs to be corrected as “Moderate”.

Response: We have made correction according to the Reviewer's comments. (p. 7, lines 253).

[Line 257] “…were divided into four bins…” instead of “…were divided into 4 bins…”.

Response: We have made correction according to the Reviewer's comments. (p. 7, lines 267).

[Lines 262-263] This sentence needs to be improved.

modify: In this paper, set the expected score S*=0 when the safety odd*=1 of the rockburst risk scorecard. That is, when the safety probability of the case is greater than 0.5, the score of the case by scored by rockburst risk scorecard is greater than 0, otherwise it is less than 0. Therefore, when the score is greater than or equal to 0, the score card judges that the sample is a safety sample, otherwise it is a dangerous sample. Set the expected score S’=10 when the safety odd*=2, then the scorecard compensation factor A and scorecard scale B values is 0 and 14.42 by calculated using Eq. 8, respectively. (p. 7, lines 267-274)

original: The rockburst risk scorecard was set with an expected score of S* = 0 for Odds* = 1 and a score of S' = 10 for odd* doubling, that is, when the case is the probability of safety is greater than 0.5, rockburst risk scorecard score is greater than 0, otherwise less than 0, so the scorecard judgment case is a safety sample, when the rockburst risk scorecard score is greater than or equal to 0 , otherwise a hazard sample.We can calculated to compensation factor A and scorecard scale B value is 0, 14.42 respectively

Application of the intelligent rockburst risk prediction model

Response: We apologize for our negligence. It is more appropriate to replace “hazard” with “risk”, and “light” instead of “weak”

Response: We have made correction according to the Reviewer's comments. “Dataset” instead of “dataset”, “Ratios of safety and hazard samples” instead of “ratios of safety and hazard samples”, “Safety sample” instead of “safety sample”, and “Risk sample” instead of “hazard sample” (p. 8, lines 285)

[Line 275] “Through the literature research…, this study selected…” instead of “Through the literature survey…, this paper selects…”

Response: We have made correction according to the Reviewer's comments. (p. 8, lines 286).

[Lines 296-297] The sentence needs to be improved.

Response: We received comments from the reviewers and improved the sentences

modify: According to the probability distribution curves of the indicators of different risk levels, it can be initially determined that the prediction ability of σ_θ, σ_θ/σ_c and W_et are strong, and the prediction ability of σ_c, σ_t and σ_c/σ_t are light. (p. 8, lines 308-311)

original: According to the probability distribution curves of different rockburst risk levels indicators can be initially determined in the long-term prediction of rockburst that σ_θ, σ_θ/σ_c and W_et of rock explosion risk prediction ability is strong, and σ_c, σ_t and σ_c/σ_tof rock explosion risk prediction ability is light.

[Line 289] The column “Parameters” of Table 3 needs units.

Response: Units have been added to the parameter column of Table 3. (p. 8, lines 301)

[Line 302] “..risk prediction model…” instead of “…risk predict model…”

Response: We have made correction according to the Reviewer's comments. (p. 9, lines 313).

[Line 303] “was” instead of “were”.

Response: We have made correction according to the Reviewer's comments.(p. 9, lines 314).

[Line 307] “WOE” or “ WofE” instead of “woe”.

Response: We have made correction according to the Reviewer's comments. (p. 10, lines 327).

[Line 309] “…more important in the rockburst risk prediction process is” instead of “…more important it is in the rockburst risk prediction process.”

Response: We have made correction according to the Reviewer's comments. (p. 9, lines 319).

[Line 310] maybe “high” is better than “large” as it is about a value. Additionally, “The” should be “the”.

Response: This sentence has been rewritten.

[Lines 310-314] The sentence needs to be improved. Please check the punctuation.

Response: We received comments from the reviewers and improved the sentences

modify: It can be seen from table 3 that the top three rockburst risk assessment indicators in absolute scores in the rockburst risk scorecard are the σ_θ, σ_θ/σ_c and W_et, which are important indicators for predicting rockburst risk levels. The highest and lowest absolute values of the scores for the W_et are large, and the W_et is the most important indicator among the rockburst risk assessment indicators used in this paper. Reducing the W_et of the rock is the key to reducing the rock burst risk level. (p. 9, lines 319-325)

original: As can be seen from Table 3, the absolute values of the highest and lowest scores for the W_et are high. The W_et is the most important of the rockburst risk indicators used in this paper, and reducing the W_et of the rock is critical to reducing the rockburst risk.the top three rockburst risk indicators in the rockburst risk scorecard are the σ_θ, σ_θ/σ_c and W_et, which are important parameters for predicting rockburst risk

[Line 332] “So, The IRPSC…” instead of “So, the IRPSC…”.

Response: We have made correction according to the Reviewer's comments. (p. 10, lines 346).

[Line 336] The heading of this section seems not representative regarding the content of the section.

Response: In order to better comply with the content, we change the Application effect to Application to riverside hydro power station tunnel rockburst risk. (p. 11, lines 350)

[Line 337] “The IRPSC was applied to eight rockburst cases…” instead of “The IRPSC was applied to 8 rockburst cases…”.

Response: We have made correction according to the Reviewer's comments. (p. 11, lines 351).

[Lines 341-342] “…accurately predicted six rockburst cases out of eight…” instead of “…accurately predicted 6 rockburst cases out of 8…”.

Response: We have made correction according to the Reviewer's comments. (p. 11, lines 2355).

[Line 346] A full stop is needed after “respectively” instead of a comma. How the accuracy of the model is approved?

Response: The model improvement effect is discussed in Section 4.1. The details are as follows: The prediction accuracy of the rockburst risk scorecard is higher than that of the SVM and is about the same as that of LR, CART and AdaBoost. (p. 14, lines 390-394)

[Pages 11-12] The parameters of Table 5 need units. Table 5 needs to be further described and discussed.

Response: We added the units of early warning indicators to the table. The contents in Table 5 are further analyzed. The details are as follows: “The real risk level of case 1 is strong, and the model incorrectly predicts a moderate, because the scorecard f₃ identified the case as safety. The σ_θ/σ_c of case 1 is 0.58, which is in the 2nd bins and is the main reason for the incorrect prediction on the scorecard f₃. The score card f₂. incorrectly predicts case 2 as a hazard case, resulting in the actual light dangerous case 2 being predicted as a moderate. This is mainly because case 2 has high σ_θ and Wet values, in particular, especially in case 2 has a σ_θ value of 91.3 MPa and a score of -10 on the scorecard f₂, which is the lowest score for this indicator on the scorecard f₂.” (p. 11, lines 358-365)

Discussion

General Comments: An in-depth discussion of the results is needed.

[Line 4] “CART”, “RF” This is the first time that CART and RF are mentioned. An explanation is needed here.

Response: “CART” is classification and regression trees, RF is random forests(RF) (P2，line 70)

[Lines 5-6] “The sample set was divided into five groups…and the remaining four groups…” instead of “The sample set is divided into 5 groups…and the remaining 4 groups…”.

Response: We have made correction according to the Reviewer's comments. (p. 14, lines 378-379).

[Line 12] “TPR” and “TNR” What do they mean? The “evaluating” is not needed.

Response: We are very sorry for our negligence of that, there should be “MAR” and “FAR”. (p. 14, lines 385).

[Line 40] “…takes into account…” The word “account” should be added here.

Response: We have made correction according to the Reviewer's comments. (p. 15, lines 418).

[Line 43] “that” is not needed. “the following” maybe is better.

Response: We have made correction according to the Reviewer's comments. (p. 15, lines 422).

[Line 44] The number of this equation is (13) and not (10). Furthermore, an explanation of the symbols is needed.

Response: We have corrected the error of formula number and added explanation of the symbol:“where C₁ and C₀ are the category weight values for the hazard and safety samples respectively. N is the total number of samples, x_i, y_i is the indicator and label of the ith case in the sample set, and for hazard sample y_i = 1 and safety sample y_i = 0, p(y = 1| x_i) represents the probability that the model predicts that sample is a hazard.” (p. 15, lines 423–426)

Conclusion

General notes: Conclusions should be improved, focusing on the original contribution of the research.

Response: In order to highlight the contribution of this paper, we rewrite the conclusion. The details are as follows:“

The 311 real rockburst cases collected were used to construct the intelligent rockburst risk prediction model base on scorecard methodology (IRPSC), and the model was applied to the eight rockburst cases during the construction of a Riverside hydro-power station tunnel. the effects of category weights on the MAR, FAR and ACC of the rockburst scorecard were investigated. The following conclusions were obtained.

(1) Using 311 rockburst cases collected, an IRPSC was constructed using a ChiMerge, WOE and LR algorithm, and the model was applied to the prediction of rockburst cases in the Riverside hydro-power station tunnel. It was found that the Wet was the main indicator affecting the rockburst risk level, and the model predicted ACC, FAR and MAR of 75%, 12.5% and 12.5% respectively. The evaluation process of the IRPSC is simple and transparent, with high prediction accuracy. The IRPSC can determine the main controlling factors affecting the occurrence of rockburst.

(2)The research results of the influence of sample category weight on the prediction FAR and MAR of rock burst prediction model show that when the safety sample category weight is set to 1 and the hazard sample category weight is gradually increased from 0.5 to 10, the rockburst risk scorecard's MAR gradually decreases from 56.9% to 17.2%, the FAR increases from 1.5% to 31.4%, and the ACC decreases from 88.4% to 71.2%. Setting higher category weights of hazard sample reduces the MAR of rockburst risk scorecard , but this will increase the FAR and should be considered in practice to determine the sample category weights. (p. 15, lines 440-460)

[Line 59] “…rockburst cases…” instead of “…rock burst cases…”

Response: As mentioned above, we rewrite the conclusion

[Line 60] “…was applied to the eight rockburst cases…” instead of “…was applied to the 8 rockburst cases…”

Response: As mentioned above, we rewrite the conclusion

[Line 66] A comma is needed after the “evidence” and “…LR based on dataset…” instead of “…LR base on dataset…”.

Response: As mentioned above, we rewrite the conclusion

[Line 78] “…set higher class weights…” instead of “…set larger class weights…”

Response: We have made correction according to the Reviewer's comments. (p. 15, lines 442).

Author Response File: Author Response.docx

Reviewer 2 Report

The paper presents an intelligent rockburst prediction model based on scorecard methodology . The paper topic is in line with the journal’s.

The paper is properly structured.
The results are presented in an organic manner. All in all, the paper deserves to be considered for publication, however I would suggest to consider the comments reported below. I hope these help in improving the quality of the paper.

Introduction section - The contribution and importance of this paper are unclear and insufficiently explained. I have not anywhere seen any research gap, which is very important for proving the novelty of the paper.
Add the limitations of this method and the possibilities of its use in practice

Author Response

Dear Reviewer:

We tried our best to improve the manuscript and made some changes in the manuscript. These changes will not influence the content and framework of the paper.

The main corrections in the paper and the responds to the comments are as flowing:

[1] Introduction section - The contribution and importance of this paper are unclear and insufficiently explained. I have not anywhere seen any research gap, which is very important for proving the novelty of the paper.

More details are as follows:

[2] Add the limitations of this method and the possibilities of its use in practice

Response: We modify the article and point out the areas that need to be improved in the application of the model. The details are as follows: “The model predicts a high degree of accuracy. However, the model's predictions are more skewed towards moderate hazards, meaning that the model is too conservative when analysing cases with extreme hazards and safety, which is an area for further improvement.” (p. 11, lines 367–371)

Author Response File: Author Response.docx

Reviewer 3 Report

The reviewed article presents a topic, very important in terms of safety in underground mining, related to the prediction of rockburst. An intelligent rockburst prediction model based on scorecard methodology was developed. The model was based on the analysis of a lot of real rockburst. The developed model has great significance to the application of machine learning in rockburst risk prediction. However, I have a few minor comments to the authors, the inclusion of which should improve the quality of the work and increase its scientific value.

The authors use many abbreviations in the text. They are explained in the article - first a description, then an abbreviation. In addition, the description and abbreviation are without spaces, which makes it difficult to read. In my opinion, it would be better to make a list of abbreviations with their description at the beginning of the article.
There are many patterns in the text. They should be centred and their numbering in parentheses should be a move to the right margin. This will make them easier to find and read.
In addition, the variables from the formulas should be explained under the formulas or in the text - their description and unit should be provided.
Figure 1 should be larger, preferably on full-page. This will make it easier to understand.
Figure 2 should be fully assembled on one page, not divided into two pages. Individual charts should be marked e.g. a), b) and explained in the caption.
The title of table 1 should be connected with the table, not divided into two pages.
Table 5 should have smaller spacing between lines. This will make it possible to place that table entirely on one page.
Avoid leaving single words at the end of the paragraph, as on lines 134, 273, and page 14, line 57.
On page 14, line 38 value and unit (56 598 million) should be connected and not split into two lines.

Author Response

Dear Reviewer:

We tried our best to improve the manuscript and made some changes in the manuscript. These changes will not influence the content and framework of the paper.

The main corrections in the paper and the responds to the comments are as flowing:

The authors use many abbreviations in the text. They are explained in the article - first a description, then an abbreviation. In addition, the description and abbreviation are without spaces, which makes it difficult to read. In my opinion, it would be better to make a list of abbreviations with their description at the beginning of the article.

Response: Thank you very much for your comments. First, give the full name before the abbreviation in the text, and add a space before the explanation and abbreviation. i.e (P1-line 22, P2-line 66-67, P3-line 117, et al.).

There are many patterns in the text. They should be centred and their numbering in parentheses should be a move to the right margin. This will make them easier to find and read.

Response: We adopt the opinions of the reviewers and move the formula numbers in the text to the far right. i.e (p. 5-line 163, 171, 177, 181, p. 6-line 211, 218, 221, 237, et al.).

In addition, the variables from the formulas should be explained under the formulas or in the text - their description and unit should be provided.

Response: We adopt the opinions of the reviewer. the variables from the formulas were description in the manuscript. i.e Eq.1 (p. 5, lines 164-168), Eq.2 (p. 5, lines 172-173), Eq.3 (p. 6, lines 178-179), Eq.5 (p. 6, lines 212–215), Eq.6 (p. 6, lines 219–22), Eq.7 (p. 6, lines 222-223)

Figure 1 should be larger, preferably on full-page. This will make it easier to understand.

Response: We try to enlarge Figure 1 and set a larger font for easy reading. (p. 4, lines 150)

Figure 2 should be fully assembled on one page, not divided into two pages. Individual charts should be marked e.g. a), b) and explained in the caption.

Response: We put Figure 2 on one page, labeled each chart, and described the figure with a title. (p. 9, lines 312)

The title of table 1 should be connected with the table, not divided into two pages.

Response: We put table 1 on one page, (p. 7, lines 253).

Table 5 should have smaller spacing between lines. This will make it possible to place that table entirely on one page.

Response: Because the construction process of rock burst scorecard model is detailed in Table 5, which contains a lot of data, it is really difficult to place it on one page by reducing the spacing.

Avoid leaving single words at the end of the paragraph, as on lines 134, 273, and page 14, line 57.

Response: We try our best to adjusted the paragraph with only one word at the end of the paragraph.

On page 14, line 38 value and unit (56 598 million) should be connected and not split into two lines.

Response: Thank the reviewers for their tips. By carefully checking the contents of the paper, we have avoided the problem of units and values in two lines in the paper.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

In the revised version, the manuscript has been significantly improved according to the reviewers' comments and is suitable for publication.
However, further editing is needed. For example:
- [Lines 22-25] “based on” instead of “base on”, the explanation of the abbreviation IRPSC is not clear, “It was analyzed that … of the The results…”?
- [Lines 76-83] This needs to be improved
- [Line 112] “by risk levels”?
- [Lines 177-181] This needs to be improved
- [Line 239] “The accuracy (ACC) the ratio of”?

Author Response

n the revised version, the manuscript has been significantly improved according to the reviewers' comments and is suitable for publication.

However, further editing is needed. For example:

Query 1: [Lines 22-25] “based on” instead of “base on”, the explanation of the abbreviation IRPSC is not clear, “It was analyzed that … of the The results…”?

Response: Thank you for pointing out the mistakes, we have made it further clear. “It was analyzed that … of the IRPSC. The results…” instead of “It was analyzed that of the The results…”. (p. 1, lines 23)

the “IRPSC” is the abbreviation of the “Intelligent Rockburst Prediction Model based on Scorecard Methodology”. Where, “I” is the abbreviation of Intelligent, “R” is the abbreviation of Rockburst, “P” is the abbreviation of Prediction, “SC” is the abbreviation of Scorecard. (p. 1, lines 23, p. 2, lines 99, and p. 15, lines 423,)

Query 2: [Lines 76-83] This needs to be improved

Response: Thank you for your suggestion on this. We have rephrased and made further improvements in the revised version of the manuscript.

Original (Pre-revised version): However, in practical engineering, we not only hope that the rockburst prediction model can have high prediction accuracy, but also hope to master the risk indicator affecting the rockburst risk level and understand what indicator lead the model to make such prediction results. In other words, the ideal model not only has high prediction ability, but also has interpretability. Machine learning models are susceptible to irrelevant features and are not robust to outliers, in addition the prediction process is difficult for the human to understand and the model is poorly interpretable, because some machine learning models have black box properties

Revised Version:

Nevertheless, taking practical engineering problem into consideration, we believe that the proposed model can provide a strong basis for accurate rockburst prediction as well as assist in understanding and improving the rockburst risk indicator. Furthermore, the proposed model has both high prediction ability and interpretability. In general, machine learning based models are sensitive to the induced irrelevant features and decrepit to the outliers. Additionally, due to the presence of black box properties in some of the machine learning models, the prediction process is quite complex and has poor interpretability [26]. (p. 2, lines 74-82)

Query 3. [Line 112] “by risk levels”?

Response: As show in Fig. 1, the sequential of the rockburst risk is taken into account in the task decomposition. (Ⅰ) Dataset S1 is based on samples with light, moderate, or strong rockburst risk are the hazard samples and samples with none rockburst risk are the safety samples in dataset S1. (Ⅱ) Dataset S2 is based on samples with moderate or strong rockburst risk as the hazard sample and samples with none or light rockburst risk as the safety sample. (Ⅲ) Dataset S3 is based on samples with strong rockburst risk as the hazard sample and samples with none, light or moderate rockburst risk as the safety sample. Rockburst risk scorecard f₁, f_2,and f₃ were constructed using datasets S1, S2, and S3, respectively. (p. 3, lines 124–130)

The IRPSC was obtained by integrating the three scorecards f₁, f₂and f₃. Therefore, the sample set is divided according to the risk levels.

Query 4. [Lines 177-181] This needs to be improved

Response: Thank for the comment. We have modified the sentence as below.

Original (Pre-revised version): Firstly, each risk indicator is binned using the ChiMerge to obtain the threshold value for each bin of the indicator. Then calculate the WOE w_ij for each bin based on the WOE theory, the weight of indicators θ is estimated according to the maximum likelihood method of LR model. Finally determine the values for the compensation factor A and the scorecard scale B.

Revised Version: At first, the threshold value for each indicator bin is obtained by employing ChiMerge that bins each risk indicator. Then, using WOE theory, the weights of evidence are calculated for each bin as represented by w_ij. The weight of indicators θ is estimated using the maximum likelihood method of the LR model. Finally, the values for the compensation factor A and the scorecard scale B are computed. (p. 5, lines 174-178)

Query 5: [Line 239] “The accuracy (ACC) the ratio of”?

Response: “The accuracy (ACC) is the ratio of” instead of “The accuracy (ACC) the ratio of” (p. 7, lines 235)

Author Response File: Author Response.pdf