Assessing Transferability of Remote Sensing Pasture Estimates Using Multiple Machine Learning Algorithms and Evaluation Structures

Round 1

Reviewer 1 Report

Dear Authors

I found the manuscript well-written, concise and with a good body of references. The approach plot to landscape is well explained and well represented. perhaps I ask to check again the body of references to be sure you do not missed recent inportant publication on AGB biomass estimation via remote sensing at some level of detail where you fee your article is outperforming literature studies.

Kind regards

Author Response

Reviewer #1 (Comments to the Author):

R1: I found the manuscript well-written, concise and with a good body of references. The approach plot to landscape is well explained and well represented. perhaps I ask to check again the body of references to be sure you do not missed recent inportant publication on AGB biomass estimation via remote sensing at some level of detail where you fee your article is outperforming literature studies.

Authors: Thank you for the positive feeback. We have added 8 new references to the paper to strengthen the points made in the paper (Lie and Huete, 1995; Tong et al., 2020; Qin and Liu, 2022; Foody et al., 2003; Jin et al., 2018; Qin et al., 2021; Morais et al., 2021; Sarle, 1996), including the point you have requested of relating our results to the AGB model performance to other studies. We found another extensive review of the use machine learning to estimate AGB (Moraise et al., 2021), which provides performance ranges of various modeling approaches. The error metrics of our interpolated model evaluations are within the ranges provided by this study as well as the review conducted by Reinermann et al., (2020) . In the text we added:

Lines 308-311: “Numerous examples of accurate site-specific SRS pasture modeling exist in the literature, and the error metrics of the interpolative evaluation of our AGB and %N models are within the range of accuracy exhibited by those studies [3,39].”

Reviewer 2 Report

The paper contains interesting research, but there are a lot of terms and information that are mentioned, but not well enough  connected or explained. For example, the vegetation index. The ten most commonly used vegetation indexes are defined - a reference is required for this, or are these the views of the author (also allowed, but need to be explained). In addition, it is somewhat unclear for the reader how the indexes are included in research. How the application of different indexes affected ML.

To my knowledge, the authors are right this study is the first assessment of the transferability of SRS pasture models, but studies of model transfer for sure exist (e.g. forests), so it would be good to research that and include this information in the introduction.

Figure 4 needs to be better explained in the text.

The main question addressed by the study is the possibility and assessment of the accuracy of the transferability of SRS pastures and the applicability of multiple models of ML algorithms and evaluation structures. These types of research are important because use of available satellite images enables research of large areas, while research is usually using smaller test areas or limited data sets. The research is also important because it includes five algorithms and two soil types.

As I wrote, the topic is relevant in the context of large pastures and specificities related to biomass. Although the researchers deal with large test areas and pastures, research related to this topic has been done on the area covered by forests and agricultural soil types. I expected that to be mentioned in the article and cited in the literature.

What is extremely interesting and has not been well addressed so far is the comparison of transferability using five machine learning regression algorithms, for which good references to previous research are also given. It is interesting and well addressed, the comparison of transferability using different machine learning regression algorithms. Although researchers have dealt with them (good references are given), so far, they have not been well structured and extensively processed. Also is well included the Xgboost (XGB) library, which has not been so far implemented in research in this way.

The inclusion of vegetation indices is very important, but I think that is not clarified enough, as well as their significance for this research (e.g., biomass assessment).

Figure 4 shows the inverse relationship between model performance and level of extrapolative evaluation. The Figure helps us to draw a conclusion of a lack of transferability for SRS pasture models, which is significant results of this study. I would like the authors to have explained in the paper this Figure better.

The conclusions by the authors based on the research are significant and in accordance with the assumptions that the selection of ML algorithms significantly affects the transferability of the SRS pasture model. In conclusion, the influences of individual models are well explained. It remains a little unclear which factor is most responsible for the obtained results (slope, moisture...).

I still stand by the fact that the authors have provided a large amount of significant information and relevant research knowledge, but which, in my opinion, are sometimes not sufficiently connected and explained. In some parts, the information in the article is more difficult to follow and connect, but in any case, the work provides a lot of new and significant information.

Author Response

Reviewer #2 (Comments to the Author):

R2: The paper contains interesting research, but there are a lot of terms and information that are mentioned, but not well enough  connected or explained. For example, the vegetation index. The ten most commonly used vegetation indexes are defined - a reference is required for this, or are these the views of the author (also allowed, but need to be explained). In addition, it is somewhat unclear for the reader how the indexes are included in research. How the application of different indexes affected ML.

Authors: Thank you for the positive feedback and the relevant point of clarification. Since the literature contains numerous studies of vegetation indices for this topic, we added them as additional predictive features for the ML models to train on. In short, in our review of the literature on SRS pasture models, we encountered ten common indices frequently used with spectral data from the Sentinel-2 platform. To clarify how we found them, we cited papers that aggregated them and wrote:

Lines 171-175: “Ten commonly used vegetation indices were calculated from the S2 bands to be included as additional features for ML modeling (Table 1). This list of indices was determined from a review of the index-based SRS pasture modeling literature, especially studies of the Sentinel-2 platform [8,28].”

To explain when the VIs were included, we wrote:

Lines 230-234: “The six ML regression algorithms were implemented for each of the eight evaluation structures. Additionally, each model fit was performed on two feature sets – S2 bands only and S2 bands + VI – resulting in 96 model fits. Only the more accurate of the feature sets was included for each model fit, leading to 48 model evaluations for publication.”

R2: To my knowledge, the authors are right this study is the first assessment of the transferability of SRS pasture models, but studies of model transfer for sure exist (e.g. forests), so it would be good to research that and include this information in the introduction.

Authors: We agree this is a useful point. Thank you for pointing it out. We’ve included new references in the introduction and discussion to provide context for our study. We added:

Lines 79-81: “This phenomenon has been observed in SRS data as well, and the challenge of model transferability has been documented for several SRS modeling applications – e.g., land cover classification [17,18] and forest biomass prediction [19,20].”

Lines 339-341: “This concept has been demonstrated for several SRS modeling applications [17,18,19,20], and researchers have reported a shift of both magnitude and direction of regression model parameters when training on different sites [19,20].”

R2: Figure 4 needs to be better explained in the text.

Authors: Great point. Figure 4 is intended to summarize the main findings of the paper, so it should be described more. We moved the figure to the results and added an additional paragraph explaining it more thoroughly. We also improved its appearance. The new paragraph is:

Lines 285-294: “To summarize our results, Figure 4 displays the means and standard deviations of the models grouped by variable, algorithm complexity, and evaluation structure. Overall, the figure illustrates how the mean model accuracies decreased as the dissimilarity of evaluation structure was increased, especially when the transferability of the models was evaluated at different locations and years. Moreover, Figure 4 highlights the extent of inaccuracy of the complex ML algorithms at increased levels of extrapolation, indicating increased over-fitting in comparison to the simpler algorithms. Additionally, the increased standard deviation of the complex algorithm accuracy is attributable to the divergence of performance between SVR and the decision tree-based algorithms which exhibited more instances of overfitting than the other algorithms.“

R2: The main question addressed by the study is the possibility and assessment of the accuracy of the transferability of SRS pastures and the applicability of multiple models of ML algorithms and evaluation structures. These types of research are important because use of available satellite images enables research of large areas, while research is usually using smaller test areas or limited data sets. The research is also important because it includes five algorithms and two soil types.

Authors: Thank you. We agree with your summary.

R2: As I wrote, the topic is relevant in the context of large pastures and specificities related to biomass. Although the researchers deal with large test areas and pastures, research related to this topic has been done on the area covered by forests and agricultural soil types. I expected that to be mentioned in the article and cited in the literature.

Authors: We agree it is useful to note previous transferability studies. Please see above, where we show the references and discussion we added to solidify this point.

R2: What is extremely interesting and has not been well addressed so far is the comparison of transferability using five machine learning regression algorithms, for which good references to previous research are also given. It is interesting and well addressed, the comparison of transferability using different machine learning regression algorithms. Although researchers have dealt with them (good references are given), so far, they have not been well structured and extensively processed. Also is well included the Xgboost (XGB) library, which has not been so far implemented in research in this way.

Authors: Thank you. We agree that the inclusion of multiple ML algorithms is important for examining impact of algorithm choice on transferability. We added more discussion and references to elaborate on the complexity of the ML algorithms that we used:

Lines 353-360: “More hyperparameters were tuned for the decision tree-based algorithms, potentially contributing to increased overfitting. Studies have demonstrated that overfitting of complex algorithms can be avoided by limiting the number of parameters used in the modeling [22,41]. However, this approach may not improve transferability for all algorithms, as Wenger and Olden (2012) reported minimal improvement for RF transferability after setting hyperparameters to constrain complexity – e.g., reducing the maximum number of nodes per tree [22].”

R2: The inclusion of vegetation indices is very important, but I think that is not clarified enough, as well as their significance for this research (e.g., biomass assessment).

Authors: Please see above for our clarifications and additions to the paper.

R2: Figure 4 shows the inverse relationship between model performance and level of extrapolative evaluation. The Figure helps us to draw a conclusion of a lack of transferability for SRS pasture models, which is significant results of this study. I would like the authors to have explained in the paper this Figure better.

Authors: Thank you, we moved Figure 4 to the results and added an accompanying paragraph.

R2: The conclusions by the authors based on the research are significant and in accordance with the assumptions that the selection of ML algorithms significantly affects the transferability of the SRS pasture model. In conclusion, the influences of individual models are well explained. It remains a little unclear which factor is most responsible for the obtained results (slope, moisture...).

Authors: Thank you for the positive feedback. We agree the most important contaminating factor is unproven in our study. We modified a discussion point to emphasize that future studies should investigate this:

Lines 382-384: “Thus, future studies should make use of larger datasets and integrate additional data types with potential to inform on these sources of bias which contaminate SRS modeling across spatial and temporal groupings.”

R2: I still stand by the fact that the authors have provided a large amount of significant information and relevant research knowledge, but which, in my opinion, are sometimes not sufficiently connected and explained. In some parts, the information in the article is more difficult to follow and connect, but in any case, the work provides a lot of new and significant information.

Authors: Thank you for the positive feedback regarding the novelty and significance of this manuscript! We hope that our reponses and revisions to the paper have improved the clarity of the study.  

Reviewer 3 Report

minor suggestion: Figure 4 should include various line types or plot symbols besides colors.

Author Response

Reviewer #3 (Comments to the Author):

R3: minor suggestion: Figure 4 should include various line types or plot symbols besides colors.

Authors: Good point! We’ve created a new plot that uses color and line types to distinguish the groupings. We think the new plot is easier to read. Please see the new version below or in the result section of the revised manuscript. 

Reviewer 4 Report

This is very interesting research and relevant to the Journal remote sensing. However, I would like to suggest the authors explicitly mention the objectives and have the conclusions section in this manuscript. In addition, the article can be checked for minor spelling and grammar mistakes. Hence, I recommend major revisions.

Author Response

Reviewer #4 (Comments to the Author):

R4: This is very interesting research and relevant to the Journal remote sensing. However, I would like to suggest the authors explicitly mention the objectives and have the conclusions section in this manuscript. In addition, the article can be checked for minor spelling and grammar mistakes. Hence, I recommend major revisions.

Authors: Thank you for the positive feedback regarding the relevancy of our paper! We agree that emphasis on objectives and conclusion will enhance the clarity of the paper. Thus, we have added some phrasing in the introduction to highlight objectives, and added a conclusion section in accordance with the journal template. We wrote:

Lines 47-49: “Therefore, the primary objective of this study is to assess the accuracy and transferability of SRS pasture models using multiple ML algorithms and evaluation structures.”

Lines 101-103: “Thus, to address our objective of assessing the accuracy and transferability of SRS pasture models, we tested six ML algorithms – LASSO, PCR, PLSR, RF, SVR, and GBM.”

Lines 395-404:

“5. Conclusion

Overall, our results demonstrated an inverse relationship between ML model performance and degree of extrapolative evaluation, providing evidence for the challenge of transferability in SRS pasture models. Moreover, we found that the relatively simple ML algorithms (LASSO, PCR, PLSR) predicted dissimilar spatiotemporal groupings more ac-curately than the complex algorithms (SVR, RF, XGB), suggesting that the less complex algorithms offer greater potential for transferable SRS pasture modeling. Regardless of model choice, our study has demonstrated the impact of test set partitioning on model performance, and we recommend future studies implement multiple spatiotemporal evaluation structures to assess the transferability of their SRS pasture models.”

Round 2

Reviewer 2 Report

After the corrections, I think that the article has been significantly improved. Congratulations to the authors, and I suggest submitting the article for publication.

Reviewer 4 Report

Thanks for revising the manuscript based on the review comments.