Machine Learning with Voting Committee for Frost Prediction

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper is about frost prediction in South America. It is not suitable for publication in its current state, and the following should be improved:

 

1. The title should be more specific.

2. This paper suggests the usefulness of the voting committee, but the description of VC in the main text is lacking. The principle should be stated more specifically.

3. Frost observation data is important for frost prediction. The description of frost observation data is lacking. The description of the frost observation network is needed.

4. Information on the number of samples used for learning, testing, and prediction should be provided. The number of days on which frost occurred and the number of days on which frost did not occur should be provided for the study period in the study area.

5. In section 3, results are expected to come first instead of discussion, but discussion comes first. It would be better to have separate result and discussion sections.

Author Response

Point-by-point response to Comments and Suggestions

Comments 1: The title should be more specific.
Response 1: We appreciate the reviewer’s valuable suggestion. In response, we have revised the title to "Machine learning with voting committee for frost prediction", which we believe better reflects the focus and methodology of the study.

Comments 2. This paper suggests the usefulness of the voting committee, but the description of VC in the main text is lacking. The principle should be stated more specifically.
Response 2: Thank you for your valuable comment. The authors understand the importance of VC definition and application to our case study on frost prediction. To address your concern, we have added a clarification to the manuscript, explaining this decision and providing context for the selected approach. Please refer to the updated Section 2.4 for the modifications. We appreciate your feedback and hope the revisions meet your expectations.

Comments 3. Frost observation data is important for frost prediction. The description of frost observation data is lacking. The description of the frost observation network is needed.
Response 3: Thank you for your insightful comment regarding the importance of frost observation data and the description of frost observation networks. We fully understand that a dedicated frost observation network would be a valuable resource for this study. We acknowledge that the identification of frost can vary in the literature, with some definitions depending on plant physiology. Additionally, considering that our study aims to compare results with a previous study that defines frost as temperatures equal to or below 6°C, we have opted to use this threshold in our analysis. To address your concern, we have added a clarification to the manuscript, explaining this decision and providing context for the selected approach. Please see Section 2. Materials and Methods in the third paragraph.

Comments 4. Information on the number of samples used for learning, testing, and prediction should be provided. The number of days on which frost occurred and the number of days on which frost did not occur should be provided for the study period in the study area.
Response 4: Thank you for your valuable comment. To address your concern, we have included detailed information on the number of samples used for training, validation, and testing. Additionally, we created a new figure to illustrate the distribution of frost and non-frost events across these periods. We kindly ask you to refer to Section 2.3 and Figure 3 in the revised manuscript for a comprehensive overview of the data distribution.

Comments 5. In section 3, results are expected to come first instead of discussion, but discussion comes first. It would be better to have separate result and discussion sections.
Response 5: Thank you for your insightful comment and suggestion regarding the structure. We agree that presenting the results first, followed by a separate discussion, would improve the clarity and organization of the manuscript. In response to your feedback, we have restructured this section to ensure the results are presented first, focusing on the key findings and supported by the corresponding figures and analysis.

Reviewer 2 Report

Comments and Suggestions for Authors

It is necessary to update the bibliography with documents from the last 4 years and that they are mostly research articles, avoiding web pages. In addition, when using the IEEE format, the appearance of the references must be in numerical order and all must appear in the literature review section.

Author Response

Point-by-point response to Comments and Suggestions

Comments 1: It is necessary to update the bibliography with documents from the last 4 years and that they are mostly research articles, avoiding web pages. In addition, when using the IEEE format, the appearance of the references must be in numerical order and all must appear in the literature review section.

Response 1: Thank you very much for your valuable feedback and suggestions. We have carefully updated the bibliography.

Reviewer 3 Report

Comments and Suggestions for Authors

This is an interesting study of the using machine learning within a regional frost prediction model.

Title: Machine learning for frost prediction in a South America region OK

Keywords: I think Frost index should be replaced as it doesn't widen the search based on words  found in the title

The paper starts by showing an agricultural concern over frost. However what is the definition of an agricultural frost? Is this based n plant physiology or is this just a temperature or observations of the "state of the ground? Or is it just observed minimum temperature or the number of days with frost observations (what is a frost observation: state of the ground??). 

It was hard for an international reader to get a sense of exactly what the index was "This index is obtained from multivariate statistical techniques applied to meteorological variables predicted by a regional model of high spatial and temporal resolution. " The problem for me was I never quite knew how a frost was defined.  Frosts are defined in a range of ways: take material damage as an example in: Richards et al (2024). Multi-model ensemble of frost risks across East Asia (1850–2100). Climatic Change, 177(4), 68. 

"Therefore, the models were developed for conditions where the minimum observed temperatures (Tobs) ≤ 6◦C."  is the model only run on days where T is likely to be <6... I did not quite understand its significance. 

Line 94-100. It would be useful for explain to the reader the topography and landscapes covered - I guess this is from the pampas plains to the Atlantic forests? 

Fig. 1: I am fortunate enough to have spent a little time in Rio Grande do Sul and Santa Catarina states, but it was still hard to recognise the regions on the map which has no scale, North arrow and lacks labels. Just a few would help. 

Figures 3-7 are very impressive though again I want to now how the frosts were defined.  What happens when you run the model on a day when frosts are very infrequent and occur in just a few places?

I guess with machine learning it is hard to know which are the variable that contribute most to the successful predictions. I guess this limits an understanding of why R1 was where the TensorFlow model presented its  best performance, while in R3 the performance was the least satisfactory.

I think Figure 8 and Table 7 should occur before the conclusions.

It would be useful for an international audience to draw conclusions about the applicability of this approach beyond South America.

 

Author Response

Point-by-point response to Comments and Suggestions

Comments 1: This is an interesting study of the using machine learning within a regional frost prediction model.
Response 1: Thank you for your positive feedback and for recognizing the relevance of our study. We are pleased to contribute to the application of machine learning techniques in regional frost prediction, highlighting its potential to enhance accuracy and improve decision-making in areas prone to frost events.

Comments 2: Title: Machine learning for frost prediction in a South America region OK Response 2: Thank you for your positive feedback.

Comments 3: Keywords: I think Frost index should be replaced as it doesn't widen the search based on words found in the title.
Response 3: Thank you for your insightful comment. The Frost Index plays a key role in our study, as it serves as a benchmark for evaluating the performance of our machine learning-based approach. This comparison is essential to demonstrate the improvements achieved with the proposed methodology. To address your concern, we have restructured the manuscript to better clarify the significance of the Frost Index in the context of our study and its role in validating the results obtained using machine learning techniques. We believe this adjustment enhances the clarity and relevance of the manuscript.

Comments 4: The paper starts by showing an agricultural concern over frost. However what is the definition of an agricultural frost? Is this based n plant physiology or is this just a temperature or observations of the "state of the ground? Or is it just observed minimum temperature or the number of days with frost observations (what is a frost observation: state of the ground??).

Response 4: Thank you for your thoughtful comment regarding the definition of agricultural frost. While we recognize that the results of our study have potential applicability to agriculture, the occurrence of frost in our analysis is not defined based on plant physiology. For this study, we considered frost occurrence as days when the observed minimum temperature in a meteorological shelter was less than or equal to 6°C. To ensure clarity for the reader, we have added a paragraph to the manuscript explaining how frost was defined and the rationale behind this choice. This addition can be found in Section 2. Materials and Methods.

Comments 5: It was hard for an international reader to get a sense of exactly what the index was "This index is obtained from multivariate statistical techniques applied to meteorological variables predicted by a regional model of high spatial and temporal resolution. " The problem for me was I never quite knew how a frost was defined. Frosts are defined in a range of ways: take material damage as an example in: Richards et al (2024). Multi-model ensemble of frost risks across East Asia (1850–2100). Climatic Change, 177(4), 68.

Response 5: In both this study and the original study where the index was developed (Rozante et al., 2019), frost occurrence is defined as days when the minimum temperature measured in a meteorological shelter is less than or equal to 6°C.

Comments 6: "Therefore, the models were developed for conditions where the minimum observed temperatures (Tobs) ≤ 6◦C." is the model only run on days where T is likely to be <6... I did not quite understand its significance.
Response 6: Thank you for your comment. The statement refers to the conditions under which the models were specifically developed and evaluated. The threshold of (Tobs) ≤ 6◦C was chosen based on climatological studies that associate minimum temperatures of 6°C or below with an increased likelihood of frost events. This criterion helps focus the model's application on scenarios where frost occurrence is plausible, ensuring that the models are evaluated under relevant conditions. However, the models themselves are not restricted to operating exclusively on days with (Tobs) ≤ 6◦C. They can be applied more broadly but were optimized and validated primarily for these conditions to improve their accuracy in frost prediction. We will clarify this in the manuscript for better understanding. Please see section 2. Materials and Methods.

Comments 7: Line 94-100. It would be useful for explain to the reader the topography and landscapes covered - I guess this is from the pampas plains to the Atlantic forests?
Response 7: Thank you for your insightful comment. Indeed, the study area encompasses a variety of landscapes and topographic features, which are crucial for understanding frost formation and prediction. We have updated Section 2, Materials and Methods, to provide a more detailed description of the topographic features of the analyzed regions. Please refer to the revised Section 2 for the updated information.

Comments 8: Fig. 1: I am fortunate enough to have spent a little time in Rio Grande do Sul and Santa Catarina states, but it was still hard to recognise the regions on the map which has no scale, North arrow and lacks labels. Just a few would help.
Response 8: Thank you for your suggestion. We have taken your feedback into account and included a new map with the improvements you mentioned, such as the scale, north arrow, and labels to help identify the regions. Additionally, we have added the topography to make the map more informative and clearer. Please see Section 2, Fig. 1.

Comments 9: Figures 3-7 are very impressive though again I want to now how the frosts were defined. What happens when you run the model on a day when frosts are very infrequent and occur in just a few places?
Response 9: Thank you for your question and insightful observation. We have clarified how frost events were defined and addressed the behavior of the model when frosts are infrequent. Kindly refer to Section 2, Materials and Methods, particularly the third paragraph, for a more comprehensive explanation.

Comments 10: I guess with machine learning it is hard to know which are the variable that contribute most to the successful predictions. I guess this limits an understanding of why R1 was where the TensorFlow model presented its best performance, while in R3 the performance was the least satisfactory.

Response 10: Thank you for your insightful comment. We agree that understanding the contribution of variables is a challenge in machine learning. In response to your observation, we have added more details to the text, discussing potential reasons for the differences in model performance between R1 and R3. We appreciate your feedback, as it has helped us improve the clarity and depth of our manuscript.

Comments 11: I think Figure 8 and Table 7 should occur before the conclusions.
Response 11: Thank you for your valuable suggestion. We agree that placing Figure 8 and Table 7 before the conclusions improves the logical flow of the manuscript. In response, we have restructured the text accordingly. Please see the updated Section 3 for the revised placement.

Comments 12: It would be useful for an international audience to draw conclusions about the applicability of this approach beyond South America.
Response 12: We thank the reviewer for this insightful comment. In response, we have expanded the discussion in the conclusion to address the potential applicability of this approach beyond South America. We emphasize that while the study focuses on a specific region, the methodology is generalizable and can be adapted to other regions with appropriate calibration and local data integration. This clarification aims to enhance the relevance of our findings for an international audience.

Reviewer 4 Report

Comments and Suggestions for Authors

I reviewed the article titled: Machine learning for frost prediction in a South America region submitted to the journal MDPI- Meteorology as consideration for a Article type of paper.

 

In this article the authors present a novel way to predict frosts in different countries of South America. The novelty presented in this article mostly relies in the use of a voting committee to decide if the Machine Learning model performs better than the frost index developed from a numerical model of regional weather for two-time scales (24 hour and 72 hours). Authors mainly found that both the Machine Learning and the numerical model based on regional weather performed similarly at the 24 hours scale, but frost predictions after 72 hours were better using the machine learning model. Authors also found spatial variation since the accuracy of both models varied per zone (R1, R2, R3). The article is quite specific but could fit the journal aim and scope.

 

However, I have few issues that need to be addressed before this article could be published. First the authors did not determine what consisted in a frost day, is it when air temperature is below 0 C ? Authors did mention a variety of definitions for frost but I don’t recall finding the definition that they used, which is of crucial importance. The authors also rely on the output of the Eta model, hence they should explain more in depth what is the Eta model and since there exists many climate model, which is only one used?

Figure 1 needs to be improved, a zoom out and a zoom in the portion under study would provide an easiest way to understand where the study takes place. Also, the way the three regions are divided needs to be better defined.

There is no need for bullet point in the text, as such the text need major rewriting.

Methodologically, I wonder why authors used 5 years and tested on one year instead of repeating their analyses using all combinations of 5 years and predict each year so authors could have used 2012, 2013,2015,2016, and 2017 to predict year 2014 and so on and compare if the results are consistent between years. Doing so you provide more robust results.

Authors did test something but did not conduct an experiment which implies manipulating variables under controlled conditions, so authors must not use the word “experiment”.

Authors should separate results and discussion. As of now the result section seems only listing figures which is not a discussion and is also not a result section. All this section needs rewriting.

The figures are hard to understand. Authors should use a mix of colors and symbols and symbols for hits and misses are almost identical.

A full discussion on the implications of their result if missing, so this article basically contains an introduction and a method section and is missing all other sections, but I think it provides interesting results so I encourage the authors to improve their manuscript since it will lead to a valuable scientific achievement.

Author Response

Point-by-point response to Comments and Suggestions

Comments 1: In this article the authors present a novel way to predict frosts in different countries of South America. The novelty presented in this article mostly relies in the use of a voting committee to decide if the Machine Learning model performs better than the frost index developed from a numerical model of regional weather for two-time scales (24 hour and 72 hours). Authors mainly found that both the Machine Learning and the numerical model based on regional weather performed similarly at the 24 hours scale, but frost predictions after 72 hours were better using the machine learning model. Authors also found spatial variation since the accuracy of both models varied per zone (R1, R2, R3). The article is quite specific but could fit the journal aim and scope.

Response 1: We appreciate your time spent reviewing our manuscript and your constructive comments. We are pleased to hear that the article was considered relevant to the aims and scope of the journal. In the revised version of the manuscript, we have more clearly highlighted the objectives of the study, in order to guide the reader as to the relevance and innovation of the research. We appreciate your noting the importance of this approach for understanding the contribution of the article.

Comments 2: However, I have few issues that need to be addressed before this article could be published. First the authors did not determine what consisted in a frost day, is it when air temperature is below 0 C ? Authors did mention a variety of definitions for frost but I don’t recall finding the definition that they used, which is of crucial importance.

Response 2: Thank you for your thoughtful comments and for raising this important point. Frost days were not defined based on temperatures below 0°C. The criteria for identifying frost days have been incorporated into the text in Section 2 in the third paragraph. We hope this clarification addresses your concern, but we are happy to provide further details if needed.

Comments 3: The authors also rely on the output of the Eta model, hence they should explain more in depth what is the Eta model and since there exists many climate model, which is only one used? Response 3: Thank you for your feedback. A brief description of the Eta model has been added to the text to address your comment. We hope this provides the necessary context, but we are happy to further elaborate if needed.

Comments 4: Figure 1 needs to be improved, a zoom out and a zoom in the portion under study would provide an easiest way to understand where the study takes place. Also, the way the three regions are divided needs to be better defined.
Response 4: Thank you for your suggestions. In response, Figure 1 has been revised to incorporate your feedback. The updated figure now includes a zoomed-out view to provide a broader context, as well as a zoomed-in view to clearly highlight the study area. Additionally, the division of the three regions has been redefined for improved clarity. We hope these adjustments meet your expectations.

Comments 5: There is no need for bullet point in the text, as such the text need major rewriting.

Response 5: Thank you for your feedback regarding the use of bullet points. We have carefully revised the manuscript to address this concern by restructuring the text into a continuous narrative. Please refer to section 2.1. Data and 2.6. Description of Experiments for the updated content.

Comments 6: Methodologically, I wonder why authors used 5 years and tested on one year instead of repeating their analyses using all combinations of 5 years and predict each year so authors could have used 2012, 2013,2015,2016, and 2017 to predict year 2014 and so on and compare if the results are consistent between years. Doing so you provide more robust results.

Response 6: The authors would like to thank the reviewer for his comments. In the development of Machine Learning models, one of the main challenges we encounter is generalization. We often obtain excellent results in the training set, but when the trained model is applied to an independent sample, the results drop considerably, sometimes making it unfeasible to use. One way to ensure the consistency of the model is to separate a completely independent sample of the data, although statistically representative of the problem under study, to apply the model as a form of hindcast to evaluate how the trained model would perform if it were being continuously used in operation. Therefore, we chose to train/validate our model in the years 2012-2016 and perform a hindcast throughout 2017. If we had also used the year 2017 for training, there would have been a possibility of contaminating our results due to the fact that the data had already been exposed to the model.

Comments 7: Authors did test something but did not conduct an experiment which implies manipulating variables under controlled conditions, so authors must not use the word “experiment”.

Response 7: The authors would like to thank the reviewer for this comment. In the context of Machine Learning, the use of the word “experiment” in reference to the process of training models is widely accepted. For example, we can cite the Weka platform (https://www.weka.io/), where the module aimed at training and testing ML models is called the "Weka Experiment Environment".

Also, we provide some references below where the term is widely used for similar objectives.

• ●  Goodfellow, I., et al. (2016) Deep Learning. MIT Press, Cambridge, MA.

• ●  http://www.deeplearningbook.org

• ●  https://www.datacamp.com/tutorial/machine-learning-experimentation-an-introduction-to-

  weights-and-biases

• ●  https://learn.microsoft.com/en-us/fabric/data-science/machine-learning-experiment

• ●  https://developers.google.com/machine-learning/managing-ml-projects/experiments

• ●  Anochi, J.A.; de Almeida, V.A.; de Campos Velho, H.F. Machine Learning for Climate

  Precipitation Prediction Modeling over South America. Remote Sens. 2021, 13, 2468. https://doi.org/10.3390/rs13132468

• Comments 8: Authors should separate results and discussion. As of now the result section seems only listing figures which is not a discussion and is also not a result section. All this section needs rewriting.
  Response 8: Thank you for your valuable suggestion regarding the structure of the manuscript. We appreciate your feedback on clearly separating the results and discussion sections. In response, we have revised the manuscript to ensure that the results section focuses on presenting the key findings, while the discussion section provides an in-depth interpretation of these results in the context of existing literature and their implications. We believe the revised sections now align better with standard practices and effectively address your concerns.

  Comments 9: The figures are hard to understand. Authors should use a mix of colors and symbols and symbols for hits and misses are almost identical.
  Response 9: Thank you for your comments regarding the figures. We agree that the symbols for hits and misses were difficult to distinguish and that the overall presentation could be improved. In response, we have revised the figures to enhance clarity:

  • ●  Improved colors and symbols: We have introduced a combination of distinct colors and symbols to clearly differentiate hits and misses, making the categories easier to identify visually.

  • ●  Higher resolution: The figures have been generated at a higher resolution, ensuring better sharpness and readability, especially for fine details.
    We hope these changes address your concerns and improve the clarity of the results presented.

  Comments 10: A full discussion on the implications of their result if missing, so this article basically contains an introduction and a method section and is missing all other sections, but I think it provides interesting results so I encourage the authors to improve their manuscript since it will lead to a valuable scientific achievement.

  Response 10: Thank you for your encouraging feedback and for recognizing the potential value of our results. We appreciate your observation regarding the need for a more comprehensive discussion of the implications of our findings. In response, we have expanded the discussion section to thoroughly interpret the results, contextualize them within the existing literature, and explore their broader implications for frost prediction and its applications. We are confident that these improvements address your concerns and contribute to the manuscript's scientific significance.

  0.  

   

   

  0.  

   

  •  

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have tried hard to account for my concerns. My only remaining concern is that a frost generally invokes freezing, so I wonder if any freezing occurs when T is merely less than 6oC. Is this not just cold weather?