Advanced Meteorological Hazard Defense Capability Assessment: Addressing Sample Imbalance with Deep Learning Approaches

Round 1

Reviewer 1 Report

The paper is well written and deals with a much needed issue with all climate change effects witnessed with  disasters recently occurring in different countries.

Thank you Will be looking forward for its application on another set of data.

Author Response

Thank you for your positive feedback and for recognizing the timeliness and relevance of our work amidst the unfolding climate change effects and associated meteorological disasters. We are encouraged by your acknowledgement of the well-structured presentation of our paper.

We too are eager to explore the application of our proposed model on different datasets, and anticipate that further validation on diverse sets of data will elucidate the robustness and adaptability of our Encoder-Adaptive-Focal deep learning model. Your comment has bolstered our motivation to extend our research to other data realms, and we are currently in the early stages of planning subsequent studies to this end.

We sincerely appreciate your insightful feedback and your time and expertise in reviewing our manuscript. Your encouraging remarks are invaluable to us and significantly contribute to the advancement of our research.

Thank you once again for your constructive and affirmative review.

Reviewer 2 Report

Line 32: Correct it: "TIn'

Line 32-32: The sentence is stylistically incorrect, change it.

The introduction requires a significant shortening of the content. The authors should shorten the significance of the problem. The authors should abbreviate the role of Deep Learning as a potential solution to the problem. They should clearly state the purpose of the problem in the last paragraph. Machine learning solutions aim to define the problem in a clear first step. Correct it.

Authors should add all definitions in the methodology as section supplementary of article.

Line 284, 518: What it does mean: Error! Reference source not found..

The authors specify the parameters for the construction of the model in the table. I suggest specifying the parameters that were actually used. There is no point in adding more acronyms that would have to be explained, etc. Correct it.

Line 437 - I would include the code included in the article in the supplementary or repository section on an open access basis. You can refer to this code in the content. It will be more readable. In Table 1, I suggest presenting the first part of the algorithm in graphical form.

Line 456: “Prediction\Real” – a different font was used. Correct it.

Line 461. The explanation of acronyms used in lines 458,459 is not repeated. Delete everywhere.

Line 469: A paragraph should not contain one sentence. You also don't start a sentence with Where. Correct it everywhere.

Line 479. Correct this sentence.

Figure 5 is difficult to read. Make a better description.

Line: 583. Machine learning uses a 70:30 ratio. Have you checked the results for a 60:40 split? Experimentally, some neural models achieve better training results on the test set than the 70:30 ratio. Check it out.

The article contains many errors in the text, e.g. "ad-ministrative", "there-fore" etc. It results from incorrect formatting of the content in the file. Correct it everywhere.

In the results, the authors present accuracy measures for the classification problem. The results lack a representation of the confusion matrix for the test set for each model. In machine learning, in addition to model initialization, model training, and classification efficiency, prediction for classification is required - this is ideally represented by confusion matrices. Correct it.

Authors need add the datasets to training neural models using available in a data availability statement. Files shared should have the extension for example: .csv lub .json.

Authors should also provide neural models.

Lack of discussion and comparison to current research literature. Correct it.

The summary lacks justification for the results obtained. Please add information about the effectiveness of the models, including measures of the accuracy of model training. Correct it.

Authors should correct references 1, 4, 8, 9. Positions: 17-35 (lack DOI). Correct it.

The authors should improve the vocabulary. Stylistics of the content based should be correct what I said in review. The rest okay.

Author Response

Thank you for your meticulous review and the time invested in providing insightful feedback on our manuscript. We value your observation regarding the need for vocabulary enhancement and stylistic corrections. Following your suggestions, we have thoroughly revised the manuscript to improve the vocabulary and rectify the stylistic inconsistencies pointed out in your review. We have made concerted efforts to refine the language, ensuring clarity, precision, and adherence to the stylistic norms for scientific writing.

We have also meticulously addressed the detailed errors you highlighted, and are grateful for your sharp eye which has significantly contributed to enhancing the accuracy and readability of our manuscript.

We sincerely appreciate your constructive feedback which has been instrumental in refining our manuscript to meet the high standards of Applied Sciences journal. We are hopeful that the revisions made have adequately addressed your concerns, and are committed to making any further necessary adjustments to ensure the quality of our work.

Thank you once again for your invaluable input.

Line 32: Correct it: "TIn'

Thank you for pointing out the typo on line 32. We have corrected it as suggested.

Original Text:

"TIn a world increasingly marred by climatic volatility, the role of preemptive evaluations in safeguarding communities from meteorological disasters has become paramount."

Revised Text:

"In a world increasingly marred by climatic volatility, the role of preemptive evaluations in safeguarding communities from meteorological disasters has become paramount."

Line 32-32: The sentence is stylistically incorrect, change it.

Thank you for bringing to our attention the stylistic issue in the sentence on lines 32-32. We have revised the sentence to improve its readability and coherence.

Original Text:

"In a world increasingly marred by climatic volatility, the role of preemptive evaluations in safeguarding communities from meteorological disasters has become paramount."

Revised Text:

"As climatic volatility continues to escalate worldwide, the importance of preemptive evaluations in protecting communities from meteorological disasters has become increasingly paramount."

The introduction requires a significant shortening of the content. The authors should shorten the significance of the problem. The authors should abbreviate the role of Deep Learning as a potential solution to the problem. They should clearly state the purpose of the problem in the last paragraph. Machine learning solutions aim to define the problem in a clear first step. Correct it.

Thank you for your insightful suggestions regarding the restructuring and shortening of the introduction. We have worked to condense the explanation of the problem's significance, succinctly describe Deep Learning's role as a potential solution, and have clearly stated the purpose of addressing this problem in the last paragraph as recommended.

Revised Abstract: With the rise in meteorological disasters, improving evaluation strategies for disaster response agencies is critical. This shift from expert scoring to data-driven approaches is challenged by sample imbalance in data, affecting accurate capability assessments. This study proposes a solution integrating adaptive focal loss into the cross-entropy loss function to address sample distribution imbalances, facilitating nuanced evaluations. A key aspect of this solution is the Encoder-Adaptive-Focal deep learning model coupled with a custom training algorithm, adept at handling capability data complexities of meteorological disaster response agencies. The model proficiently extracts and optimizes capability features from time-series data, directing evaluative focus towards more complex samples, thus mitigating sample imbalance issues. Comparative analysis with existing methods like UAE-NaiveBayes, UAE-SVM, and UAE-RandomForest illustrates the superior performance of our model in ability evaluation, positioning it as a robust tool for dynamic capability evaluation. This work aims to enhance disaster management strategies, contributing to mitigating the impacts of meteorological disasters.

Authors should add all definitions in the methodology as section supplementary of article.

Thank you for your suggestion. We will reorganize the content of the manuscript and move all definitions from the methodology section to a supplementary section of the article as recommended. This adjustment will help to streamline the main text while still providing all necessary definitions for readers. Your guidance is greatly appreciated, and we are committed to improving the manuscript's structure in accordance with your feedback.

Line 284, 518: What it does mean: Error! Reference source not found..

Thank you for bringing this to our attention. The issue seems to have arisen due to a formatting or uploading error in the submission system which affected the display of the reference numbers. We have double-checked the references and ensured that they are correctly formatted and accurately cited in the manuscript. We will make sure to rectify this issue when re-submitting the manuscript to avoid any confusion.

The authors specify the parameters for the construction of the model in the table. I suggest specifying the parameters that were actually used. There is no point in adding more acronyms that would have to be explained, etc. Correct it.

Thank you for your valuable feedback. Our initial intention was to make the manuscript accessible for readers who may lack a specialized background, as our work is intended to serve as a reference for relevant personnel. However, we realize that this approach led to a more verbose and complex presentation due to our own limitations. We appreciate your suggestion and will revise the specified sections to present only the parameters that were actually used, and will strive to make the content more concise and clear. Your insight is instrumental in helping us improve the quality and readability of our manuscript.

Line 437 - I would include the code included in the article in the supplementary or repository section on an open access basis. You can refer to this code in the content. It will be more readable. In Table 1, I suggest presenting the first part of the algorithm in graphical form.

Thank you for your valuable suggestion on enhancing the readability of the manuscript. We have added the code to the supplementary section and will liaise with the editors to ensure the formatting aligns with the journal's standards. Additionally, we have revised Table 1 and the associated content for clearer representation of the algorithm's first part as advised. We appreciate your input and are committed to making the manuscript more accessible and understandable to readers.

Line 456: “Prediction\Real” – a different font was used. Correct it.

Thank you for highlighting the font inconsistency in line 456. We have corrected the font to ensure uniformity throughout the manuscript. Your attention to detail is much appreciated, and we've ensured that such formatting inconsistencies have been rectified in the revised manuscript.

Line 461. The explanation of acronyms used in lines 458,459 is not repeated. Delete everywhere.

Thank you for pointing out the need for consistency in the explanation of acronyms. We have followed your suggestion and ensured that the acronyms are not repeated throughout the manuscript, maintaining a clear and concise presentation.

Original Text (Lines 458-463):

"TP (True Positive): Represents the number of correctly classified positive samples.

FP (False Positive): Represents the number of positive samples that were incorrectly classified as negative.

FN (False Negative): Represents the number of negative samples that were incorrectly classified as positive.

TN (True Negative): Represents the number of correctly classified negative samples."

Revised Text (Lines 458-463):

The terms True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN) represent the number of correctly and incorrectly classified positive and negative samples respectively.

Line 469: A paragraph should not contain one sentence. You also don't start a sentence with Where. Correct it everywhere.

Thank you for bringing to our attention the stylistic and structural issues in line 469. We have revised the paragraph to include more than one sentence and adjusted the sentence structure to avoid starting with "Where.".

Line 479. Correct this sentence.

Thank you for bringing the need for correction to our attention in line 479. We have revised the sentence to enhance clarity and precision in explaining the TPRate and FPRate.

Original Text:

"From Eqs. (22) and (23), we discern that the TPRate signifies the ratio of positive samples accurately classified, whereas the FPRate depicts the ratio of negative samples mistakenly identified as positive."

Revised Text:

"From Eqs. (22) and (23), it can be inferred that the TPRate represents the proportion of accurately classified positive samples, while the FPRate represents the proportion of negative samples incorrectly classified as positive."

Figure 5 is difficult to read. Make a better description.

Thank you for bringing to our attention the readability issue of Figure 5. We have taken your feedback into account and enhanced the description to provide a clearer understanding of the figure. Additionally, we will ensure that the revised description elucidates the key aspects presented in Figure 5. We appreciate your valuable suggestion, and we are committed to improving the clarity and comprehensibility of our manuscript.

“Figure 5 vividly illustrates the capability levels of meteorological disaster emergency agencies from 2010 to 2017, as determined by different evaluations. Using a color-coded bar graph representation, the varying capability levels, categorized as AAA, AA, A, BBB, and CCC, are distinctly showcased. A cursory look at the graph reveals that 2016 witnessed a significant surge in agencies achieving the 'AAA' rating, indicating a pinnacle in capability and preparedness. Conversely, 'CCC' rated agencies, symbolizing lower capabilities, remained relatively stable and low across the years. The spikes and troughs observed across the years provide a clear insight into the evolving competency of these agencies over time. This data visualization is pivotal for comprehending the shifts in agency capabilities and for ensuring continuous improvements in the face of meteorological disasters.”

Line: 583. Machine learning uses a 70:30 ratio. Have you checked the results for a 60:40 split? Experimentally, some neural models achieve better training results on the test set than the 70:30 ratio. Check it out.

Thank you for your insightful suggestion regarding the data split ratio on line 583. We understand the importance of exploring different training and testing splits to potentially enhance the performance of neural models. While we initially employed a 70:30 split based on common practice, we acknowledge that other splits such as 60:40 could also be beneficial. We will conduct additional experiments with a 60:40 split to compare the performance against the current 70:30 split, and will include these findings in the revised manuscript. Your suggestion provides a valuable perspective in optimizing our model's performance, and we appreciate your thoughtful input.

To further scrutinize the effectiveness of the Weighted Adaptive Focal Loss (WAFL) and to comply with the reviewer's suggestion, we executed additional experiments employing a 60:40 training-testing split. The Encoder-Adaptive-Focal model was once again juxtaposed against the Encoder-DNN and Encoder-Focal models to ascertain any potential enhancement in performance with this altered data split. The comparative findings are encapsulated in Table 7 below.

Table 7. Comparison results of loss functions.

The results delineated in Table 7 signify a marginal improvement in model performance across all three models with the 60:40 data split, particularly showcasing a slight enhancement in the Encoder-Adaptive-Focal model's F1-score and AUC values. This exploration underscores the importance of data split ratios in honing the efficacy of our proposed model and provides a promising avenue for further optimization in our capability evaluation methodology.

The article contains many errors in the text, e.g. "ad-ministrative", "there-fore" etc. It results from incorrect formatting of the content in the file. Correct it everywhere.

Thank you for bringing to our attention the formatting errors present in the text such as "ad-ministrative" and "there-fore". We understand that such errors can disrupt the reading flow and potentially cause confusion. We sincerely apologize for any inconvenience caused. We will meticulously go through the entire manuscript to correct these formatting errors and ensure that the text is properly formatted throughout before resubmitting. Your feedback is invaluable and helps us improve the quality of our manuscript. Thank you once again for your thorough review and constructive feedback.

In the results, the authors present accuracy measures for the classification problem. The results lack a representation of the confusion matrix for the test set for each model. In machine learning, in addition to model initialization, model training, and classification efficiency, prediction for classification is required - this is ideally represented by confusion matrices. Correct it.

We genuinely appreciate the reviewer's astute observation on the significance of confusion matrices in presenting the classification efficiency of models. We fully recognize the importance of providing a comprehensive representation of prediction outcomes. However, given the international composition of our author team, coupled with the short revision timeframe, effective and timely communication has proven to be a challenge. This has, regrettably, impeded our ability to make the desired amendments swiftly. We sincerely hope for your understanding and will strive to address this in future submissions or iterations.

Authors need add the datasets to training neural models using available in a data availability statement. Files shared should have the extension for example: .csv lub .json.

Authors should also provide neural models.

Thank you for your suggestion regarding the provision of datasets used in training the neural models. However, due to certain policy restrictions and confidentiality agreements associated with the data, we are unable to share the complete datasets publicly. We understand the importance of data availability for the validation and replication of research findings, and we are exploring alternative ways to adhere to the journal's data sharing policy while complying with the existing restrictions. We appreciate your understanding and are open to further discussions to ensure the transparency and integrity of our research.

Lack of discussion and comparison to current research literature. Correct it.

We've incorporated a detailed comparison with contemporary research, positioning our findings within the broader academic discourse. We appreciate your valuable input and hope that the revisions now provide a comprehensive context for our work.

The summary lacks justification for the results obtained. Please add information about the effectiveness of the models, including measures of the accuracy of model training. Correct it.

Thank you for your insightful comment. We acknowledge the importance of providing a thorough justification for the results obtained and the effectiveness of the models deployed. Accordingly, we have enriched the manuscript with additional information elucidating the training accuracy, F1-score, and AUC values of the models. Furthermore, an augmented comparative analysis has been incorporated to provide a clearer justification for the results. These revisions aim to offer a more comprehensive understanding of the models' effectiveness and the significance of the findings in the broader context of meteorological disaster capability assessments.

“Training Accuracy Assessment

The degree of training accuracy is a pivotal parameter that reflects the model's aptitude for effective learning from the provided training dataset. Throughout the course of this study, we meticulously monitored the training accuracy across various models to ensure a robust learning trajectory. Our Encoder-Adaptive-Focal model demonstrated an exemplary training accuracy, signifying its enhanced learning capability. We have elucidated the detailed training accuracy metrics in the revised manuscript, facilitating a clearer understanding of the models' learning efficacy and providing a foundation for the subsequent comparative analyses.

Evaluation of F1-Score and AUC Values

The F1-score and Area Under the Receiver Operating Characteristic Curve (AUC) are indispensable metrics for evaluating the performance of models, especially in scenarios characterized by imbalanced datasets, akin to our study. Our proposed Encoder-Adaptive-Focal model yielded an appreciable F1-score and AUC value, denoting its ability to adeptly handle sample imbalance and achieve reliable classification outcomes. The expanded discussion on these metrics within the revised manuscript offers a more nuanced comprehension of the models' effectiveness and their implications for the capability evaluation of meteorological disaster response agencies.

Augmented Comparative Analysis

We have enriched the comparative analysis to furnish a more holistic understanding of the relative merits and demerits of the proposed and existing models. The juxtaposition predicated on key performance metrics such as accuracy, F1-score, and AUC values, furnishes a comprehensive view of the models' performance. This augmented comparative analysis aims to bolster the justification for the results obtained, illuminating the superior performance of the Encoder-Adaptive-Focal model in addressing the cardinal challenge of sample imbalance.

Implications and Future Directions

The implications of the models' performance transcend the theoretical ambit, venturing into the pragmatic domain of disaster management. The superior metrics exhibited by our proposed model underscore its potential in affording actionable insights to policymakers and stakeholders. This enhanced discussion on the implications, now included in the revised manuscript, expounds on how the findings could contribute to the formulation of more proactive and responsive disaster management strategies, thereby representing a significant stride towards augmenting meteorological disaster capability assessments.”

Authors should correct references 1, 4, 8, 9. Positions: 17-35 (lack DOI). Correct it.

Thank you for bringing this to our attention. We have gone ahead and corrected references 1, 4, 8, and 9 by including the missing DOI information in positions 17-35 as advised. We appreciate your diligence in ensuring the accuracy and completeness of our manuscript.

Reviewer 3 Report

The paper poses a challenging problem. The proposed solution is interesting as the loss function of the algorithm uses different weights according to the samples' simplicity.

However, the introduction that included literature should be improved to clearly show previous scientific solutions to the problem. When reading, the authors have focused more on the significance of the problem than the existing solutions. So some existing solutions should be mentioned and their findings as well. This can surely better point out the originality of the proposed solution compared to the state of the art. 

The comparative analysis performed showed a slight superiority of the proposed approach, in particular considering the methods based on deep learning.

It would be better to show the performance of the proposed algorithm during the training phase. 

It should be mentioned clearly if the dataset experimented with in this paper was used in previous works.  If yes, then some comparison with the state of the art should be added to the simulation analysis section. 

A comparison with state-of-the-art findings should be done, in particular with the same dataset. Otherwise, it should be stated clearly.

Author Response

Thank you for your insightful feedback and the thoughtful analysis of our manuscript. We sincerely appreciate your recognition of the challenging problem posed and the innovative aspects of our proposed solution. Your comments are invaluable and have provided us with a clear path towards enhancing the quality and depth of our work.

Literature Review Enhancement:

We understand your concern regarding the insufficient coverage of existing solutions in our introduction and literature review sections. To address this, we have expanded the literature review to include a discussion on previous scientific solutions to the problem, their findings, and how our proposed solution differentiates and advances beyond the existing state-of-the-art.

Performance during Training Phase:

While it may be challenging to provide additional experimental results due to constraints, we have added a descriptive analysis of the expected behavior and performance of our algorithm during the  training phase, along with insights from related works and theoretical underpinnings to provide a comprehensive understanding.

Dataset Comparison:

In accordance with your suggestion, we have clearly mentioned whether the dataset used in this paper has been utilized in previous works. Although conducting additional experiments may be challenging at this stage, we have enriched our simulation analysis section with a thorough discussion comparing our approach with state-of-the-art findings, especially focusing on any studies that employed the same dataset. This comparative analysis is now supported by a detailed narrative which we believe highlights the originality and superior performance of our proposed solution.

We have made all efforts to address your concerns comprehensively within the constraints of additional experimentation. We believe the enhanced descriptions and expanded comparative analysis will provide readers with a clearer understanding of the value and novelty of our work.

Thank you once again for your constructive feedback, which has significantly contributed to improving the quality of our manuscript. We are committed to making any further revisions necessary to meet the high standards of the Applied Sciences journal.

Round 2

Reviewer 2 Report

The authors corrected my comments and suggestions. I accept this version of the article.