From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review
Round 1
Reviewer 1 Report
The authors present a review of ML methods applied to CMR segmentation, with particular focus on DL approaches. The paper identifies that while the segmentation performance has reached inter-rater values, automatic provide anatomically unfeasible segmentations. Thus, the SOTA review aims to review proposals that target the reliability and robustness to mitigate the chances of outputting segmentation errors.
Overall the paper is well written and structured. I would like to congratulate the authors for the good intro and insights provided. Please find bellow my overall comments, and some line-specific ones, mainly concerning the clarity of some sentences.
Overall comments:
1. While a direct comparison is not possible among the different approaches that are being reviewed (due to the different DL models, and other strategies along the pipeline) throughout the manuscript, I argue that the authors could check for ablative studies (if performed) that were reported by the referenced works. This would allow the reader to check the improvement in performance is or not marginal in the presence of reliability/robustness measures, and also how it visually impacted the segmentation (for example).
2. There are already commercial and certified software that performs CMR segmentation. Thus, when stating that models still lack the necessary reliability and robustness to be safely translated into practice, to some extent these ideas collide. Please clarify better this overall notion.
3. When addressing the challenge to reliable segmentation, please clarify that these challenges relate to normal operation conditions, disregarding causes that can disrupt the performance of a model, for example adversarial attacks.
4. Section 3.2.2 - In the case of loss formulation, it should be stated that the vast majority of the optimization goals (e.g. cross-entropy) do no directly optimize the model with respect to the final problem task, which to some extent relates to the presence of holes and non-anatomical regions being segmented.
5. In the light of the division presented by the authors where ensemble strategies can be considered as an approach to improve segmentation. robustness or reliability?
Specific comments:
l. 40 "Before the surge ...", since first attempts of Deep Learning dates way back from its massive application in various fields, I would suggest to replace the term surge with emergence.
l. 73 do not require
l.112 and 113. Please revise the sentence. Think one of the DSC values is missing.
l.243-244 -- If these types of tools/processes have heuristics behind them or are automatically defined, via a DL model for example, mention should be made.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper presents a review of cardiac image segmentation literature. Main contributions are related to tracing deep learning methods and categorizing the literature into quality control (QC) and model improvement (MI) techniques. Quantitative evaluation comparison between the works from the two categories would be helpful for the readers to better understand the value of this categorization in the field of cardiac image segmentation.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The authors provide an overview of the state-of-the-art methods in CMR segmentation deep learning techniques. The manuscript is well written, organized and structured. It has value because it offers a comprehensive and critical analysis of the technical background of the image analysis that helps the doctor to understand the limits of the technique.
Minor recommendations:
- For the title: „From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: a Review”
- Table 1. Please rearrange so that the numbering in each column of the table is consecutive.
- Discussions and Conclusions must be different sections.
Thank you!
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The authors have address all of my comments with meaningful actions to the paper, and it was possible to observe an increase on the quality of the work. Thank you for the effort.
Reviewer 2 Report
No further comments.
