Towards the Generation of Medical Imaging Classifiers Robust to Common Perturbations

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article deals with the classification problem and tries to generate a medical image classifier. The authors have used X-rays and ultrasound image datasets. The review has the owing comments.

Strong points

1. The study focuses on a critical challenge in the deployment of machine learning (ML) and artificial intelligence (AI) classifiers in clinical settings, namely the lack of robustness, which has been a major hurdle for translating research findings into practical clinical applications.

2. The paper introduces a novel perturbation technique involving common random effects such as Gaussian noise, contrast adjustments, blur, rotation, and tilt. This approach is aimed at enhancing the robustness of AI/ML-based classifiers when applied to medical imaging data.

3. An innovative aspect of the study is the application of perturbations not only during the training phase but also during the testing phase. This dual-phase perturbation approach distinguishes the research from conventional methods and contributes to the overall robustness assessment.

4. The study utilizes two publicly available datasets, the PneumoniaMNIST dataset and the Breast Ultrasound Images (BUSI dataset), to demonstrate the effectiveness of the proposed perturbation approach. This choice enhances the generalizability of the findings across different medical imaging scenarios.

5. The research shows that classifiers trained with perturbed training images maintain consistent performance on unperturbed data, suggesting that the introduced perturbations during training do not compromise the classifier's ability to handle clean images.

6. A critical aspect of the study involves a comparison with the commonly used adversarial training method. This comparative analysis provides valuable insights into the relative strengths and weaknesses of the proposed perturbation approach, contributing to a more comprehensive understanding of robustness enhancement techniques.

7. The key finding indicates that classifiers trained with perturbed data exhibit improved robustness when faced with perturbed testing data compared to those developed solely on unperturbed data. This highlights a significant advantage of the perturbation approach in addressing real-world variability in medical images.

Weak points

1. The study employs two specific datasets, PneumoniaMNIST and BUSI, to illustrate the perturbation approach. The limited diversity in datasets might affect the generalization of findings to other medical imaging scenarios, raising questions about the broader applicability of the proposed technique.

2. The article does not explicitly mention the choice of evaluation metrics used to assess classifier performance. The absence of a detailed discussion on the selection of metrics may leave readers questioning the comprehensiveness and appropriateness of the chosen metrics for the specific medical imaging tasks.

3. While the study includes a comparative analysis with adversarial training, the discussion could be more elaborate. A detailed comparison of the strengths and weaknesses of both approaches, along with a discussion on specific scenarios where one method might outperform the other, would enhance the clarity of the findings.

4. The paper lacks in-depth discussion regarding the rationale behind the choice of specific perturbation parameters (e.g., magnitude of noise, level of blur). Providing insights into how these parameters were determined or varied could strengthen the methodological rigor of the study.

5. The study might benefit from discussing the computational resources required for implementing the proposed perturbation approach. Understanding the computational cost is crucial, especially for practical implementation in real-world clinical settings with potential resource constraints.

6. The paper does not explicitly address ethical considerations related to the use of medical imaging data. Given the sensitive nature of healthcare data, a discussion on privacy, consent, and data handling practices would contribute to a more comprehensive evaluation of the study.

7. While the study focuses on robustness in a controlled setting, the lack of real-world clinical validation might limit the practicality of the proposed perturbation approach. A discussion on potential challenges in translating these findings into real clinical practice would enhance the study's practical relevance.

8. The research primarily focuses on short-term robustness to perturbations. However, the paper does not delve into the long-term robustness of the proposed classifiers, which is crucial for ensuring the reliability of AI/ML systems over extended periods and varying conditions.

General comments;

1. Define all the acronyms used in the article.
2. Improve the quality of the figures used in the article.
3. Improve the cohesion and coherence of the article.
4. The literature review section is very short which must be improved.

Comments on the Quality of English Language

Moderate changes are required.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have prepared a commendable study, and I believe it will make significant contributions to the literature. Here are some comments:

1) Enumerate the original contributions in the introduction section.

2) The literature review has been well-conducted.

3) The quality and visibility of the figures could be improved.

4) The third section could be expanded by including comparisons with other approaches reported in the literature.

These suggestions aim to enhance the clarity and depth of the manuscript. Overall, the work is promising, and addressing these points would further strengthen its impact on the field.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

see the attachment 

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Extensive editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

I am really grateful to review this manuscript. In my opinion, this manuscript can be published once some revision is done successfully. I made one suggestion and I would like to ask your kind understanding. This study used perturbed image data from two public sources, applied ten convolutional neural networks and achieved the error rate of 15.2% for the prediction of pneumonia and breast lesions. I would argue that this is a good start. However, I would like to ask the authors to add a summary table for Figures 5-7: The table would increase the understanding and interest of readers. 

Comments on the Quality of English Language

Minor editing of English language required 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 5 Report

Comments and Suggestions for Authors

This study presents a promising approach to enhancing the robustness of AI/ML-based classifiers for medical imaging data by concurrently perturbing common random effects during training and testing. By utilizing perturbations such as Gaussian noise, contrast adjustments, blur, rotation, and tilt, the authors aim to create classifiers that are more resilient to variations commonly encountered in real-world medical imaging scenarios.

While the study demonstrates improved robustness of classifiers trained with perturbed data compared to those trained without perturbations, there are several aspects that warrant careful consideration and further investigation.

1. Generalizability: The authors should elaborate on the generalizability of their findings across different medical imaging modalities and datasets. It is essential to assess whether the observed improvements in robustness hold true for a broader range of imaging conditions and pathologies.

2. Evaluation Metrics: The paper should provide a comprehensive analysis of performance metrics beyond accuracy, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). This would offer a more nuanced understanding of the classifier's effectiveness across various perturbation levels.

3. Impact of Perturbation Magnitude: It would be valuable to investigate the influence of different magnitudes of perturbations on classifier performance. Assessing the trade-off between perturbation intensity and robustness could provide insights into the optimal level of perturbation for training.

4. Comparison with State-of-the-Art Techniques: While the study briefly mentions a comparison with adversarial training, a more thorough comparison with other state-of-the-art techniques for enhancing robustness, such as data augmentation strategies or domain adaptation methods, would strengthen the paper's contribution.

5. Clinical Relevance: The paper should discuss the clinical implications of the proposed approach. How would the improved robustness of classifiers impact real-world diagnostic workflows? Additionally, considerations regarding regulatory approval and integration into clinical practice should be addressed.

6. Statistical Significance: Ensure that the reported improvements in robustness are statistically significant. Conducting appropriate statistical tests and providing confidence intervals or p-values would enhance the rigor of the findings.

7. Potential Limitations: Acknowledge any limitations of the proposed approach, such as computational overhead, potential overfitting to specific perturbation types, or challenges in interpreting classifier decisions in clinical contexts.

Addressing these points would significantly strengthen the paper and provide clearer guidance for future research directions and potential applications in clinical practice.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 6 Report

Comments and Suggestions for Authors

1. The abstract may have some numerical results about the work which can attract the readers

2. Results can be improved with standard image processing metrics.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have written that they have addressed my previous comments in the response file but in the manuscript, it seems they have not incorporated my comments. It would be better if the authors specified the section, and changes in the response file and apparently the response to my previous comments provided by the authors in the response file is highly unsatisfactory and without any solid explanation. I would have expected reasonable responses from the authors and I hope in the next revision, the authors consider this point. Thanks.

Comments on the Quality of English Language

Moderate changes are required.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

After the revision, the quality of the article has improved. 

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

We thank the reviewer for their suggestions and support of the manuscript.

Reviewer 3 Report

Comments and Suggestions for Authors

author has responded all the comments

Comments on the Quality of English Language

Minor editing of English language required

Author Response

We thank the reviewer for their suggestions and support of the manuscript.

Reviewer 4 Report

Comments and Suggestions for Authors

I am really grateful to review this manuscript. 

Comments on the Quality of English Language

Minor editing of English language required 

Author Response

We thank the reviewer for their suggestions and support of the manuscript.

Reviewer 5 Report

Comments and Suggestions for Authors

Accept

Reviewer 6 Report

Comments and Suggestions for Authors

NIL.