Visual Explanations of Deep Learning Architectures in Predicting Cyclic Alternating Patterns Using Wavelet Transforms

Round 1

Reviewer 1 Report

1- the contribution is limited, and should be improved 

2- the abstract  didn’t present the method results 

3- in the preprocessing phase, the authors declare: Furthermore, despite the recommendation of removing cardiac field and eye movement artifacts [17], these were not removed from the EEG signals for algorithmic simplicity.  please provide further justification.

4- The dataset possesses class imbalance, is there any possibility to balance the data instead of admitting this weakness

5- the achieved accuracy is  0.80 for MobileNet, it outperformed all the classifiers in terms of sensitivity by achieving the highest sensitivity value of 0.75. These results seems not promising and need further improvement> to justify the work contribution, authors are invited to compare the obtained results to machine learning approaches and other existing works

6-the authors incite: The baselines for the performance analysis of the best-performing models are considered from the review study conducted by Mendonca et al. [36], with global accuracy, specificity, and sensitivity values of 0.76, 0.72, and 0.78, respectively. the reference 36 seems too old, and the comparison to existing works is mandatory

minor contribution

Author Response

As a corresponding author of this manuscript, I would like to sincerely thank the respected

reviewer for spending their precious time and effort thoroughly reviewing this manuscript

and providing valuable suggestions to improve this manuscript.

Following, chronologically, the reviewer's comments will be addressed point by point.

Comment: the contribution is limited, and should be improved 

Response: We are thankful to the reviewer for the concern regarding the contribution of this work. We agree that this work has limited contribution in terms of the novelty of the method. But we would like to point out that the presented work does not focus on devising a method. Instead, it is one of the potential efforts to explore and understand the vision of computer vision-based deep learning methods for identifying Cyclic Alternating Patterns (CAP) by identifying A-phase and its subtypes using scalograms resulting for continuous wavelet transforms of EEG signal. It aims to identify the regions targeted by state-of-the-art deep learning architecture-based computer vision models for A-phase and its subtypes predictions, intending to verify if the regions targeted by these architectures resemble the regions focussed by sleep experts. This would eventually motivate the development of accurate and efficient deep learning-based methods for above mentioned biomedical tasks. We would like to stress that this work was targeted for interpretability in deep learning, and, as Van der Veer et al. [1], such will lead to approaches prioritizing interpretation over performance. Yet, our results are above the state-of-the-art average results.

Action: Inclusion of a paragraph highlighting that the present work is in the trade-off between performance and examinability while still attaining state-of-the-art results. Section 4, paragraph 1.

Comment: the abstract didn’t present the method results 

Response: We understand the concerns of the reviewer regarding the presentation of results. Agreeing with the reviewer’s suggestion, we have improved the abstract by putting accuracy, sensitivity, and specificity metrics of the best-performing model.

Action: Inclusion of the performance of the best-performing model in the abstract.

Comment: In the preprocessing phase, the authors declare: Furthermore, despite the recommendation of removing cardiac field and eye movement artifacts [17], these were not removed from the EEG signals for algorithmic simplicity.  please provide further justification.

Response: We are grateful to the reviewer’s for pointing out this concern. Pre-processing is an important step for any deep learning-based task. However, this work aims at improving the model interpretability by focusing only on a single EEG signal. To remove the indicated artifacts, the state-of-the-art works used additional signals from ECG and EOG. Such is contradictory to what we intend to do in this work, and that was the rationale for not using this cleaning procedure. In addition to that home based sleep devices (HBSD) have limited sensors.

Action: Included the justification for not doing the cardiac field and eye movement artifacts. Section 2.2.

Comment: The dataset possesses class imbalance, is there any possibility to balance the data instead of admitting this weakness.

Response: Thank you for this important indication. The reviewer is correct that the dataset is imbalanced, but we have used a cost-sensitive learning-based approach to overcome this weakness (as stated in subsection 3.3.2). The underlying principle behind this kind of learning is to assign a higher cost penalty for incorrect classifications to the class possessing a lower number of samples. Furthermore, under-sampling was not used since it might limit the model's generalizability, while oversampling may produce some synthetic sample signals that do not resemble to A-phase signal due to their spontaneous nature. Furthermore, creating synthetic biomedical signals is not recommended for such tasks.

Comment: the achieved accuracy is  0.80 for MobileNet, it outperformed all the classifiers in terms of sensitivity by achieving the highest sensitivity value of 0.75. These results seem not promising and need further improvement> to justify the work contribution, authors are invited to compare the obtained results to machine learning approaches and other existing works

Response: Thank you for pointing out this problem. We understand the reviewer’s concern regarding the classifier’s performance. First, we would like to emphasize that sensitivity is very important for the tasks considered in this study for two reasons: 1) it provides an idea that the model is correctly classifying the A-phase, 2) model’s robustness in prediction since a few components such as signals corresponding to REM, whose signal patterns are similar to A-phase. However, we agree with the reviewer’s suggestion of comparing it with other works in the literature. Therefore, we have compared our results with the reported metrics from two review studies Mendonça et al. [2] and Sharma et al. [3]. The comparative analysis suggested that despite the accuracy-explainability trade-off, the metrics achieved by the proposed model are better than the average performance metrics (the average is taken by averaging the accuracy, specificity, and sensitivity of studies reported in the respective review papers). We would like to emphasize again that this work is targeted at explainable artificial intelligence and performance, although very important for general applications, it is sacrificed when explainability is intended.

Actions: Inclusion of a comparison table and discussion of the results. Table 4 and Section 4, paragraph 1.

Comment: the authors incite: The baselines for the performance analysis of the best-performing models are considered from the review study conducted by Mendonca et al. [36], with global accuracy, specificity, and sensitivity values of 0.76, 0.72, and 0.78, respectively. the reference 36 seems too old, and the comparison to existing works is mandatory.

Response: We agree with the reviwer’s concern about the outdated study by Mendonca et al. [2]. Therefore, we have also included another updated review study by Sharma et al. [3], published in year 2022. Precisely, we have taken the average of accuracy, specificity and sensitivity from the reported studies in the review.

Action: Inclusion of a new reference for the performance comparison. Reference 37.

References

1. van der Veer, S. N., Riste, L., Cheraghi-Sohi, S., Phipps, D. L., Tully, M. P., Bozentko, K., ... & Peek, N. (2021). Trading off accuracy and explainability in AI decision-making: findings from 2 citizens’ juries. Journal of the American Medical Informatics Association, 28(10), 2128-2138.
2. Mendonça, F., Fred, A., Mostafa, S. S., Morgado-Dias, F., & Ravelo-García, A. G. (2022). Automatic detection of cyclic alternating pattern. Neural Computing and Applications, 34(13), 11097-11107.
3. Sharma, M., Lodhi, H., Yadav, R., Elphick, H., & Acharya, U. R. (2023). Computerized detection of cyclic alternating patterns of sleep: A new paradigm, future scope and challenges. Computer Methods and Programs in Biomedicine, 107471.

Reviewer 2 Report

The paper is well written and organized, however I advise the authors to look into the following minor points for improving the quality of the manuscript.

1.      I recommend to use abbreviation when they occur first in the paper. For example, author mentioned “A phase” in line 6 and its full description mentioned later in the line 25. Also, try not to give multiple abbreviation (for example: CAP).

2.      Kindly use same abbreviation for whole the paper (A phase or A-phase).

3.      Supportive reference(s) is/are required for the statement in line 17.

Author Response

As a corresponding author of this manuscript, I would like to sincerely thank the respectedreviewers for spending their precious time and effort thoroughly reviewing this manuscript and providing valuable suggestions to improve this manuscript.

Following, chronologically, the reviewer's comments will be addressed point by point.

Comment: I recommend to use abbreviation when they occur first in the paper. For example, author mentioned “A phase” in line 6 and its full description mentioned later in the line 25. Also, try not to give multiple abbreviation (for example: CAP).

Response: We are thankful to the reviewer for pointing out this mistake. We have improved the manuscript by incorporating the changes suggested by the reviewer.

Action: corrected the location when the abbreviations are defined.

Comment: Kindly use same abbreviation for whole the paper (A phase or A-phase).

Response: We are grateful to the reviewer’s for pointing out this inconsistency in the manuscript. Consequently, we have replaced the A phase with A-phase throughout the manuscript.

Action: corrected the abbreviation to A-phase.

Comment:  Supportive reference(s) is/are required for the statement in line 17.

Response: We appreciate the reviewer’s concern regarding missing references. Therefore, we have added a suitable reference in line 17.

Action: Inclusion of a new reference. Reference 17.

Reviewer 3 Report

Congratulations for your solid and interesting work. Please consider the following minor revisions :

Formal:

Meaning of the abbreviations FC used as column head in Table 2 must be provided

The meaning of all abreviations should be provided even for abbreviations that are widely used in the domain , e.g. FP (False Positive), TN (True Negative) etc.

python, gradcam, inception-v3 should start with an uppercase letter

Figure captions of Figs. 3 , 6 and 7 are too long. The explanations provided in them should be moved in text.

Typos

A singular s at the end of line 214

In References[24],[25], [27] [30] „Proceedings of the Proceedings???”

Missing end dot – line 300

The phrase from lines 93-94 needs revision. Other small corrections should be done, but the editors will ask you to fix them punctually!

Author Response

As a corresponding author of this manuscript, I would like to sincerely thank the respected reviewers for spending their precious time and effort thoroughly reviewing this manuscript and providing valuable suggestions to improve this manuscript.

Following, chronologically, the reviewer's comments will be addressed point by point.

Comment: Meaning of the abbreviations FC used as column head in Table 2 must be provided

Response: We are thankful to the reviewer for contributing to improving the readability of the paper. Following the reviewer’s suggestion, we have replaced the abbreviations with their expanded forms.

Action: correction of the abbreviation. Table 2.

Comment: The meaning of all abreviations should be provided even for abbreviations that are widely used in the domain , e.g. FP (False Positive), TN (True Negative) etc.

Response: We appreciate reviewer’s concern regarding the usage of abbreviations. Therefore, following the reviewer’s suggestion, we have replaced the first occurrences of TP, TN, FP, and FN by True Positive, True Negative, False Positive, and False Negative, respectively.

Action: correction of the abbreviations. Section 3.4.1.

Comment: python, gradcam, inception-v3 should start with an uppercase letter

Response: We are thankful to the reviewer’s for pointing out this mistake. Consequently, we have rectified it by replacing the first letter of the above-mentioned terms with uppercase.

Action: correction of the abbreviations.

Comment: Figure captions of Figs. 3 , 6 and 7 are too long. The explanations provided in them should be moved in text.

Response: We appreciate reviewer’s concern regarding long captions. We agree with reviewer’s suggestion of keeping the caption shorter. Therefore, the captions of the above-mentioned figure numbers are modified for shorter captions.

Action: correction of the figures captions. Figures 3, 6, and 7.

Comment: Typos-A singular s at the end of line 214;In References[24],[25], [27] [30] „Proceedings of the Proceedings???”;Missing end dot – line 300

Response: We are thankful to the reviewer’s for pointing out typographical errors, as these are hard to identify. We have proofread the article and removed the typographical errors.