Round 1

Reviewer 1 Report

Dear authors, 

your paper is about cement stabilized soil, a commonly used ground reinforcement solutions in geotechnical engineering. You applied three standard machine learning methods to predict the unconfined compressive strength of cement stabilized soil. You claim that your study can provide an effective prediction tool for the unconfined compressive strength of cement stabilized soil.

I do not have the knowledge to judge about the geotechnical part, so I concentrated on the machine learning part. Please find my comments below.  

- MAE and RMSE are abbreviations, but were not introduced
- performance indicator R is what? I know only R2 (R-squared)
- Target: UCS (unconfined compressive strength) of cement-soil micture (q_u)

Paragraph 4.1
"So by continuously adjusting and optimizing the weak learner's weight in order for it to be a good learner, the loss function is also minimized and optimized" I miss here the terms "gradient descent" and iterative or sequential. Also the sentence is somewhat the other way around. 
The weights are optimized by gradient descent of the loss function. 
Eq 1 is not really helpful, without explaining what y_bar is. k is the index of the trees, the number of trees should be n. 
I would prefere F(x) = \sum_{i=1}^M \gamma_ih_i(x), where \gamma_i are the weights. Or the iterative equation F_m(x) = F_{m-1} + \gamma_m h_m(x), m is the actual step and m-1 the step before.  

Figure 4: The scheme is not really explained in the text and it is not speaking for itself. I would suggest to replace it with a more helpfull scheme or delete it.

Paragraph 4.2
The ANN is not trained by trial and error, it is trained by gradient descent (back propagation). "Multi-Layer Awareness" I have never heard, do you mean Multi-Layer Perceptron?

Figure 5 is ok, I would improve the capture. For instance: Diagramm of a fully connected ANN with one hidden layer and a single output value.  

Paragraph 4.3
Now you come up with "supervised learning", but GB and ANN before were also supervised learning methods. Maybe you should explain supervised learning as paragraph 4.1 and shift the other paragraphs bei +1.
The explanation of the SVM algorithm is not understandable.

Paragraph 4.4
Is ok, but some formulations are unclear or repetitive. Additionally I would say, that Random Search is the standard approach, because it outperforms Grid Search in most cases. 
Figure 6 is ok. 

Paragraph 4.5
Cross validation is usually used if the training data is limited to reduce bias. 
It gives a more realistic model than a simple train-test-split (if the data is limited). The term over-matching is unusual. 

Figure 7: I am not really happy with that. The green squares with "Training set" are somehow missleading, I would delete them. 
The Fold 1-5 of every coloumn is wrong, as far as I understood, the whole column 1 is "Fold 1", column 2 is Fold 2, ... 
And the legend with white box = Train model and red box = test model is not correct. It should be "train set" and "test set". 

Paragraph 4.6. 
There is no text!  

Paragraph 5.1
If you do Hyperparameter optimization, you need to hold out your test set (of 20%). Have you done that? 
If yes, you should mension that in the text.  
If no, the error estimate is too optimistic/unrealistic due to data leakage.   

Paragraph 5.2
You should explain more clearly, what you mean with "simulations". I think I can guess, what you have done, but it is not really clear. 
Calling it "sampling" instead of simulation could also be helpful. 

Figures 8-10: These figures are not "parallel coordinates plots", because parallel coordinates plots show polylines, not curves. 
Maybe they are "Andrews Curves"? 
Why do you show these figures? I don't see which conclusion can be drawn from them or that it is a valuable visualization. 
Explain what the reader can learn from them or leave them out/replace them by more useful visualizations. Maybe you could show correlations between input features and target value? 
  
Figures 11-13: You write "Histogram chart", but the shown graphs look like density curves (?), not like histograms (no intervals, no bars). 
 
Figure 14: Is it a correlation plot between measured and predicted values? If yes, the caption and the legends are confusing/wrong. 
What is the difference between Linear (Predicted) and Linear (Perfect Fit)?

Conclusion: 
ANN works best, WE and AC are most important features (for that you don't need machine learning). You wrote in the abstract, that it can be used for a prediction tool. You should pick up this point in the conclusion/outlook. 

Author Response

RESPONSES OF THE REVIEWER’S COMMENTS

Responses to Reviewer 1

Dear reviewer #1, we thank for your nice and constructive comments, which help us to improve the quality of our work.

Here are the list of answers to your comments:

1. Comment 1

- MAE and RMSE are abbreviations, but were not introduced
- performance indicator R is what? I know only R2 (R-squared)
- Target: UCS (unconfined compressive strength) of cement-soil micture (q_u)

Respond: We thank to reviewer’s comments. It was a mistake and the performance indicators has not been fully introduced in the manuscript. Performance indicator R is actually Correlation Coefficient. We had already added contents to explain about those indicators on section 4.6. The contents have been added in the revised manuscript as follows:

In this paper, to evaluate the accuracy of the model, the author used three indicators accounting for the error between the actual and predicted values were used, namely the mean absolute error (MAE) [38], root mean square error (RMSE) [39], correlation coefficient (R)[40]. R measures the correlation between the predicted value and the actual value, valid in the range [-∞, 1].The model will have high accuracy when R approaches 1. Conversely, low RMSE and MAE show better accuracy of the proposed ML algorithms. More specifically, the ideal values of RMSE and MAE are 0. On the other hand, RMSE calculates the squared root average difference, whereas MAE calculates the difference between the predicted and actual values. These values can be calculated using the following equations:

                                                                          (4)                                                               

                                                                  (5)                                                      

                                                                     (6)                                                           

where k is the number of the observations, p_o and are the measured and mean measured values of the critical, and p_t and are the predicted and mean predicted values, respectively.

2. Comment 2

Paragraph 4.1
"So by continuously adjusting and optimizing the weak learner's weight in order for it to be a good learner, the loss function is also minimized and optimized" I miss here the terms "gradient descent" and iterative or sequential. Also the sentence is somewhat the other way around. 
The weights are optimized by gradient descent of the loss function. 
Eq 1 is not really helpful, without explaining what y_bar is. k is the index of the trees, the number of trees should be n. 
I would prefere F(x) = \sum_{i=1}^M \gamma_ih_i(x), where \gamma_i are the weights. Or the iterative equation F_m(x) = F_{m-1} + \gamma_m h_m(x), m is the actual step and m-1 the step before.

Respond: We thank to reviewer’s comments. We found it very helpful to improve the introduction of GB model. We had already fixed contents on section 4.1. All changes have been marked in red color in the revised manuscript. The contents have been added in the revised manuscript as follows:

So by continuously adjusting and optimizing the weak learner's weight in order for it to be a good learner, in which the learner's weights are optimized by gradient descent of the loss function [45]. Mathematically the model can be viewed as [28]:

                                                                           (1)

where F(x) is the output model,  is the learner's weight of iteration i^th, is weak learner of iteration i^th and M is the number of iterations.

3. Comment 3

Figure 4: The scheme is not really explained in the text and it is not speaking for itself. I would suggest to replace it with a more helpfull scheme or delete it.

Respond: We thank to reviewer’s comments. We found it right and Figure 4 had been deleted in the revised manuscript.

4. Comment 4

Paragraph 4.2
The ANN is not trained by trial and error, it is trained by gradient descent (back propagation). "Multi-Layer Awareness" I have never heard, do you mean Multi-Layer Perceptron?

Respond: We thank to reviewer’s comments. It was a mistake and we had already fixed contents on section 4.2. The contents have been added in the revised manuscript as follows:

ANN can also be learned through gradient descent of the loss function, namely back propagation method….. Multi-Layer Perceptron is the most widely used ANN type.

5. Comment 5

Figure 5 is ok, I would improve the capture. For instance: Diagramm of a fully connected ANN with one hidden layer and a single output value.  

Respond: We thank to reviewer’s comments. We found it right and had already changed name of Figure 5 to “Diagramm of a fully connected ANN with one hidden layer and a single output value”

6. Comment 6

Paragraph 4.3
Now you come up with "supervised learning", but GB and ANN before were also supervised learning methods. Maybe you should explain supervised learning as paragraph 4.1 and shift the other paragraphs bei +1.
The explanation of the SVM algorithm is not understandable.

Respond: We thank to reviewer’s comments. We found it very helpful. All three machine learning method used in this study were supervised learning, so we decided to remove “supervised learning” from SVM presentation. Besides, the explanation of the SVM algorithm had been fixed on section 4.3. The contents have been added in the revised manuscript as follows:

The objective of the support vector machine algorithm is to find a hyper-plane in a multi-dimensional space that distinctly classifies the data points. To separate two layers of data points, multiple hyper-plane can be selected. But the most suitable plane is the one with the maximum margin (i.e. the maximum distance between data points of both layers).

7. Comment 7

Paragraph 4.4
Is ok, but some formulations are unclear or repetitive. Additionally I would say, that Random Search is the standard approach, because it outperforms Grid Search in most cases. 

Figure 6 is ok. 

Respond: We thank to reviewer’s comments. We found that Random Search method in this section should be fixed. The contents have been added in the revised manuscript as follows:

In machine learning, the hyper-parameter is a valuable parameter used to control the learning process, for example the number of neurons in the hidden layer of the ANN model, the kernel type of SVM, or the number of trees in the GB model.

….

The RS method can significantly reduce the number of solutions to seek before there is a high probability to find the most accurate model, on the contrary, it can be seen that to find a good solution, the GS method must have the thick mesh and takes more resources to deal with.

8. Comment 8

Paragraph 4.5
Cross validation is usually used if the training data is limited to reduce bias. 
It gives a more realistic model than a simple train-test-split (if the data is limited). The term over-matching is unusual. 

Respond: We agree with reviewer’s comments. Cross validation used when the training data is limited to reduce bias (avoid overfitting) and it seem better than simple train-test-split. The term over-matching is actually over-fitting and we had fixed it on the revised manuscript.

9. Comment 9

Figure 7: I am not really happy with that. The green squares with "Training set" are somehow missleading, I would delete them. 
The Fold 1-5 of every coloumn is wrong, as far as I understood, the whole column 1 is "Fold 1", column 2 is Fold 2, ... 
And the legend with white box = Train model and red box = test model is not correct. It should be "train set" and "test set". 

Respond: We thank to reviewer’s comments. We found it helpful and had removed green squares with "Training set" in flow chart.  Besides, the name of the legend was wrong and we had changed it to Training set and Testing set as Figure 7 in the revised manuscript. Furthermore, we think the column name should be "Iteration" and the number is from 1 to 5.

Figure 7. Flow chart of 5-fold cross-validation technique

10. Comment 10

Paragraph 4.6. 
There is no text!  

Respond: We thank to reviewer’s comments. It was a mistake and that paragraph introduced about the performance indicators used in this study. We had already added contents in the revised manuscript.

11. Comment 11

Paragraph 5.1
If you do Hyperparameter optimization, you need to hold out your test set (of 20%). Have you done that? 
If yes, you should mension that in the text.  
If no, the error estimate is too optimistic/unrealistic due to data leakage.   

Respond: We thank to reviewer’s comments. We found it very helpful and had added contents to make this section more clearly. The contents have been added in the revised manuscript as follows:

To prepare the data for the hyper-parameters tuning process, the initial data set was random divided into two sets, including training set (80%) and testing set (20%). To avoid data leakage, ML models was evaluated based on data from 5 fold CV technique which mean testing data was hidden in this step.

12. Comment 12

Paragraph 5.2
You should explain more clearly, what you mean with "simulations". I think I can guess, what you have done, but it is not really clear. 
Calling it "sampling" instead of simulation could also be helpful. 

Respond: We agreed with reviewer’s comments. We found it right to had add contents to explain more clearly. Besides, we had changed the term “simulations” to ‘sampling” in the revised manuscript. The contents have been added in the revised manuscript as follows:

From a statistical standpoint, the randomness in the data set needed to be carefully considered when comparing models. In this section, to compare the performance of the three optimized models, 300 samplings were performed taking into account the random division between training set and testing set. In these samplings, the training and test set sizes were kept the same, however the index number of the training and test data were randomly selected in the original data set. The models would be built on the training set and then validated on the testing set.

13. Comment 13

Figures 8-10: These figures are not "parallel coordinates plots", because parallel coordinates plots show polylines, not curves. 
Maybe they are "Andrews Curves"? 
Why do you show these figures? I don't see which conclusion can be drawn from them or that it is a valuable visualization. 
Explain what the reader can learn from them or leave them out/replace them by more useful visualizations. Maybe you could show correlations between input features and target value? 

Respond: We agreed with reviewer’s comments. We found that parallel coordinates plots were not useful in this situation and changed it into performance - iteration chart. Since the input data had many factors, the use of parallel plots to show correlations between the input characteristics and the target value is difficult to follow and we did not include this study.

14. 
    Comment 14

Figures 11-13: You write "Histogram chart", but the shown graphs look like density curves (?), not like histograms (no intervals, no bars). 

Respond: We thank to reviewer’s comments. Figures 11-13 were actually density curves and we had already fixed name in the revised manuscript.

15. 
    Comment 15

Figure 14: Is it a correlation plot between measured and predicted values? If yes, the caption and the legends are confusing/wrong. 
What is the difference between Linear (Predicted) and Linear (Perfect Fit)?

Respond: We thank to reviewer’s comments. Figures is correlation plot between measured and predicted values. Linear (Predicted) is the linear regression representation of the predicted data and Linear (Perfect fit) is actually the best fit line. We had changed the name of legend Linear (Perfect Fit) to Linear (Best fit).

16. Comment 16

Conclusion: 
ANN works best, WE and AC are most important features (for that you don't need machine learning). You wrote in the abstract, that it can be used for a prediction tool. You should pick up this point in the conclusion/outlook. 

Respond: We thank to reviewer’s comments. We found it right to add content to the conclusion to make this research more useful. The contents have been added in the revised manuscript as follows:

The results of this study indicated that machine learning methods, especially the ANN model, can be an effective tool for quickly predicting UCS of cement stabilized soils with excellent performance.

Author Response File: Author Response.docx

Reviewer 2 Report

I am very glad to read this well-edited manuscript and the authors have performed a good work. They developed three data-driven models to establish a relationship between the samples with different conditions and the UCS which conventionally obtained by laboratory experiments. The authors also compared the performance of these three models in the predictive capacity and ranked the importance of the fourteen influencing factors. I realized that this work provides a tool that can be applied to effectively determine the UCS in a similar geological background. Therefore, I agree to publish this manuscript if the following comments are revised.

Minor issues

1. Line 96-101: giving more information how the samples acquisition and storage. Because these samples are the fundamental information of this work, which directly determine the accuracy of the following data analysis.
2. Line 105: suggest provide all the stress-strain curves measured on other samples. More test curves will allow readers to have a macro understanding of the site where the samples drilled.
3. Line 109-110: explain the mixing radio, and why the unit is kg/m³?
4. Section 3.2: provide more literatures to show the importance of these selected parameters.
5. Line 187: explain the meaning of “hyper-parameters” or give some examples. Because some reader in civil engineering may not be very familiar with the terms of AI.
6. Line 203: the authors select the RS method but have not give a powerful reason. I suggest presenting some other reasons here, such as in the perspective of model accuracy.
7. Line 223: the authors point out the optimal hyper-parameters here, but did not illustrate how to obtain these optimized parameters. Please provide more text.
8. Line 249: I suppose these 300simulations are obtained by Monte Carlo sampling. Please give these words.

Very nice work and some other editing issues are presented in the attached pdf file.

Comments for author File: Comments.pdf

Author Response

RESPONSES OF THE REVIEWER’S COMMENTS

Responses to Reviewer 2

Dear reviewer #1, we thank for your nice and constructive comments, which help us to improve the quality of our work.

Here are the list of answers to your comments:

Point 1. Line 96-101: giving more information how the samples acquisition and storage. Because these samples are the fundamental information of this work, which directly determine the accuracy of the following data analysis.

Respond: We appreciate a reviewer's comment. We feel right to provide more information on how samples are collected and stored. The contents have been added in the revised manuscript as follows:

Soil samples were collected by pressing a cylinder into the soil block at the boreholes, then covered with a moisture-proof bag to keep original water content (Figure 2). Collected soil were stored and transported by boxcar from the collecting site to laboratory.

Figure 2. Undisturbed samples

Point 2. Line 105: suggest provide all the stress-strain curves measured on other samples. More test curves will allow readers to have a macro understanding of the site where the samples drilled.

Respond: We appreciate a reviewer's comment. We found it right and added more test curves in appendix B (Figure B). However, since the amount of data is so large, it is impossible to include all curves, so we choose some typical curves to illustrate. If the reader need, we will send the full data set.

Point 3. Line 109-110: explain the mixing radio, and why the unit is kg/m³?

Respond: We thank to reviewer's comment. We found it helpful and had explained about this feature. The contents have been added in the revised manuscript as follows:

Cement mixing ratio (i.e. the amount of cement in one m³ of the mixture) were applied to 100 kg/m³, 150 kg/m³ and 200 kg/m³.

Point 4. Section 3.2: provide more literatures to show the importance of these selected parameters.

Respond: We thank to reviewer's comment. We found it helpful and add literatues about those feature. The contents have been added in the revised manuscript as follows:

The soil type (denoted as S) seem to influence the UCS of the mixture as it determines the soil's grain composition [3] [5]. Moisture content of soil samples (denoted as Mc) also play an important role in the UCS detecting [10] [7]. The wet density of the soil (denoted as We) should also be considered as it determines the natural state of the soil prior to mixing [7]. The soil sampling depthdepth of soil samples (denoted as D) should also be taken into account when it affects the soil condition [34]. Cannot fails to mention the amount of cement (denoted as Ac) used for mixing when it determines the adhesion between soil particles [5]. Factors related to the sample after mixing such as: Specimen diameter (denoted as Di); Specimen length (denoted as L); Specimen area (denoted as A); Specimen volume (denoted as V); Mass of specimen (denoted as M); Density of specimen (denoted as De) might also affect much to the prediction [6], [28].

Point 5. Line 187: explain the meaning of “hyper-parameters” or give some examples. Because some reader in civil engineering may not be very familiar with the terms of AI.

Respond: We thank to reviewer's comment. We found it helpful and added information about hyper-parameter. The contents have been added in the revised manuscript as follows:

In machine learning, the hyper-parameter is a valuable parameter used to control the learning process, for example the number of neurons in the hidden layer of the ANN model, the kernel type of SVM, or the number of trees in the GB model.It must be asserted that hyper-parameter control the behaviors of training algorithms and have a significant effect on the performance of machine learning models [11].

Point 6. Line 203: the authors select the RS method but have not give a powerful reason. I suggest presenting some other reasons here, such as in the perspective of model accuracy.

Respond: We thank to reviewer's comment. We found it helpful and added information about RS method. The contents have been added in the revised manuscript as follows:

The RS method can significantly reduce the number of solutions to seek before there is a high probability to find the most accurate model, on the contrary, it can be seen that to find a good solution, the GS method must have the thick mesh and takes more resources to deal with.

Point 7. Line 223: the authors point out the optimal hyper-parameters here, but did not illustrate how to obtain these optimized parameters. Please provide more text.

Respond: We thank to reviewer's comment. We found it helpful and added information about how to choose optimum hyper-parameters. The contents have been added in the revised manuscript as follows:

In the process of hyper-parameter tuning, the model with the best R performance indicator was selected as the final model and the model's hyper-parameters were considered the optimum hyper-parameters

Point 8. Line 249: I suppose these 300simulations are obtained by Monte Carlo sampling. Please give these words.

Respond: We thank to reviewer's comment. We found it helpful and added information about how to conduct 300 simulations. The contents have been added in the revised manuscript as follows:

In these samplings, the training and test set sizes were kept the same, however the index number of the training and test data were randomly selected in the original data set. The models would be built on the training set and then validated on the testing set.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Dear Authors, 

the paper now makes a nice impression. Nevertheless, I still have some comments. In Line 163 it should be "neural network" instead of "neuron network". In Line 198 it should be "hyperparameters are" instead of "hyperparameter is". And although you changed the figures 8-10, I still don't like them. I see high frequency zigzag lines. It is clear, that the R value fluctuates with different hyperparameters, but the figure seem to have no benefit. A figure, where it is visible that the hyperparameter tuning leads to an improvement of R or MAE or the predictive power would be nice, but is not really required. My suggestion is to delete Figures 8-10. If the hyperparameter optimization is done correctly, the most important information are the optimized hyperparameters (given in Table 3). 

Author Response

RESPONSES OF THE REVIEWER’S COMMENTS

Responses to Reviewer 1

Dear reviewer #1, we thank for your nice and constructive comments, which help us to improve the quality of our work.

Here are the list of answers to your comments:

1. Comment 1

In Line 163 it should be "neural network" instead of "neuron network". In Line 198 it should be "hyperparameters are" instead of "hyperparameter is"

Respond: We thank to reviewer’s comments. It was a mistake and we had already fixed in the revised manuscript.

2. Comment 2

And although you changed the figures 8-10, I still don't like them. I see high frequency zigzag lines. It is clear, that the R value fluctuates with different hyperparameters, but the figure seem to have no benefit. A figure, where it is visible that the hyperparameter tuning leads to an improvement of R or MAE or the predictive power would be nice, but is not really required. My suggestion is to delete Figures 8-10. If the hyperparameter optimization is done correctly, the most important information are the optimized hyperparameters (given in Table 3). 

Respond: We thank to reviewer’s comments. We found it right and decided to delete Figures 8,9,10 as the optimum hyper-parameters was also given in Table 3.

Author Response File: Author Response.docx