Evaluation of Sampling and Cross-Validation Tuning Strategies for Regional-Scale Machine Learning Classification
Round 1
Reviewer 1 Report
The authors present a comparison of several techniques for training sample selection, including random, stratified random and deliberate sampling, as well as multiple calibration/validation procedures (Monte Carlo/Bootstrap, k-fold and leave-one-out cross-validation). Further, they compare different sample sizes and sampling on a subset.
The manuscript is concise and offers a good explanation of concepts. The structure is clear and easy to grasp. However, the topic explored in this paper is very common and well known in machine learning modelling field. Language and style are good, further proof-reading is not necessary.
Comments:
1) l. 23: please write km2 in superscript.
2) Please make sure that the fonts in table and figure descriptions match the text.
3) Table 5: please correct the description to avoid it being split by the image.
4) In study area and data section please explain more about the used dataset. E.g. have you applied any atmospheric corrections model?
Author Response
Many thanks for all of the comments and for taking the time to review this manuscript.
Author Response File: 
Author Response.docx
Reviewer 2 Report
It is a magnificent article, very well explained, clear, concise and very detailed. In which is demonstrated, with a practical example very well based theoretically, the benefits of using different selection methods, tuning approaches and the use of a representative area instead of the entire dataset.
Some minor aspects require a more detailed explanation:
In line 46, 9 references are used in a single paragraph, which is not usual and in any case distorts its usefulness.
In line 153, there should be a reference to support the statement.
In the paragraphs of lines 158, 165 and 169, the references from which the information has been obtained are missing.
I find point 1.3 very interesting, setting out the objectives of the article in the form of a question.
I had hoped to see figure 7 after figure 4, in order to have an overview of the points of each type of selection from the beginning.
There is an error in the caption of table 7, it is set the same as that of table 6 instead of the MC.
Congratulations to the authors of the text, as I believe it provides a magnificent detailed study that will be of great use to future researchers.
Author Response
Many thanks for all of the comments and for taking the time to review this manuscript.
Author Response File: Author Response.docx
Reviewer 3 Report
The paper investigates four sample selection methods and three cross-validation tuning approaches on high spatial resolution datasets.
Probs:
1. it is interesting to investigate different settings for handling high spatial resolution data.
Cons:
1. As an experimental study, the current work may be insufficient both in baselines and datasets. In fact, the methods such as SVM are hard to say SOA. It would be better to consider recent methods, especially, deep learning based ones. Moreover, more challenging data sets with lager data size and more complex data distribution should be examined.
2. The conclusion, i.e. k-fold cross validation is a good choice, is commonly accepted in the community of machine learning and image processing. It is highly expected to have more insights. I feel that more comprehensive experiments should be designed and conducted.
Author Response
Many thanks for all of the comments and for taking the time to review this manuscript.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
Thanks for addressing my comments.
Author Response
Thank you again for your comments, suggestions, and insights, and for taking the time to review this manuscript. We believe your edits greatly strengthened this work.
Reviewer 2 Report
The small details that were in the first version have been corrected, adding the bibliographical references that were missing, as well as answering the doubts that had been raised.
Now the article is even better than in its first version, so I think it should be published.
Author Response
Many thanks again for your comments, suggestions, and insights, and for taking the time to review this manuscript. We believe your edits greatly strengthened this work.
Reviewer 3 Report
the authors did not address my concerns, especially, the comparison with deep learning based methods, more complex data, and specifically designed experiments.
Author Response
Thank you again for your comments, insights, and suggestions, and for taking the time to review this manuscript. We believe that your suggestions helped to improve this manuscript. Also, we would be very interested in exploring research on deep-learning based classification methods in future works.
