Artificial Intelligence-Based Patient Selection for Deep Inspiration Breath-Hold Breast Radiotherapy from Respiratory Signals

Round 1

Reviewer 1 Report

This paper proposed a Deep Recurrent Neural Network (RNN) with Bidirectional Long Short-Term Memory (BLSTM) for classification of patients eligible for DIBH from analysis of their respiratory signals acquired for DIBH during acquisition of pre-treatment CT. The optimal neural network was comprised of two layers of 100 neural units which was trained using stochastic gradient descent with momentum optimizer, and the proposed BLSTM-RNN classified patients eligible for DIBH with good accuracy in the test set. These results look promising for building an accurate and robust decision system for automated assisting the radiotherapy team in assigning patients to DIBH. This paper is not suitable for publication unless the following problems are solved, so I suggest a major revision. The detailed comments are given in the following.

There are some points need to be further clarified:

1.       It may be necessary to rewrite the Introduction. The inspiration of your work must be highlighted in Introduction.

2.       The statement "the first study" in line 111 may be inappropriate.

3.       It is necessary to add the structure of the paper as a separate paragraph at the end of Introduction.

4.       The data are too few. Maybe more data can be provided to help the conclusion.

5.       It is very useful to give some sufficient and perfect conclusions. I'm sorry I didn't get them in this manuscript.

Author Response

Reviewer 1:

This paper proposed a Deep Recurrent Neural Network (RNN) with Bidirectional Long Short-Term Memory (BLSTM) for classification of patients eligible for DIBH from analysis of their respiratory signals acquired for DIBH during acquisition of pre-treatment CT. The optimal neural network was comprised of two layers of 100 neural units which was trained using stochastic gradient descent with momentum optimizer, and the proposed BLSTM-RNN classified patients eligible for DIBH with good accuracy in the test set. These results look promising for building an accurate and robust decision system for automated assisting the radiotherapy team in assigning patients to DIBH. This paper is not suitable for publication unless the following problems are solved, so I suggest a major revision. The detailed comments are given in the following.

There are some points need to be further clarified:

Comment: 1.       It may be necessary to rewrite the Introduction. The inspiration of your work must be highlighted in Introduction.

Comment: 2.       The statement "the first study" in line 111 may be inappropriate.

         Response: Although we fully agree with the reviewer’s concerns, we also could not find in the literature another work similar to ours, involving AI analising respiratory tracks for patient selection in radiotherapy. Therefore we think that writing “at our best knowledge, this is the first study” is a honest statement.

Comment: 3.       It is necessary to add the structure of the paper as a separate paragraph at the end of Introduction.

Response: we added a flowchart at the beginning of the methods section as asked also by the reviewer 2

Comment:  4.       The data are too few. Maybe more data can be provided to help the conclusion.

Response: We agree with the reviewer on this point. However, we were not able to add more data to the study, due to the large amount of work required to increase the dataset (patient treatment CT acquisition in both free breathing and breath hold, radiation oncologist’s contouring of both images, treatment planning of free breathing and breath hold) we were not able to collect new data in the short time given for the revision. Please also note, for comparison, that the Reference 31 by Lin et al which had a similar purpose as our work had 18 patients.

Comment:  5.       It is very useful to give some sufficient and perfect conclusions. I'm sorry I didn't get them in this manuscript.

         Response: The conclusion was revised in order to make it more self-explanatory, we added  the sentence “In the present work a deep neural network to predict which patients will benefit from DIBH RT from the respiratory tracks acquired early in the treatment workflow was implemented.”.

Reviewer 2 Report

Authors proposed a deep learning model to predict the patient eligibility for deep inspiration breath-hold breast cancer radiotherapy from respiratory signals. The following are my concerns which should be addressed in the revisions. 

Minor Issues:

I noticed some continuous references should be merged. For eg., line no. 43 and 44, the references change it to [1]-[4]

Check if there is a typo in line no. 48. If not what is Gy?

Section-wise outline is missing. 

Typo in line no. 271. 

Major Issues:

Abstract:

The results section of the abstract present only the abstract. I noticed the authors have mentioned the hyperparameter details in the result section. It should come in the previous paragraph(before the result). Provide details about the BLSTM-RNN model and its combinations in the methods section of the abstract. In the result section, add all results I noticed only accuracy and ROC however, it will be good to mention all the other values too. 

Introduction: 

This section is well written and easy to follow. However, I feel the related works on Artificial Intelligence and Deep Learning techniques are weaker. There have been many improvements recently in the healthcare domain through the advancement of deep learning techniques. Authors are expected to include more state-of-the-art literature. 

Figure 4 shows the proposed model is suffer from overfitting. The training accuracy is close to 85 but test accuracy is at least 10 less. This is a serious problem to be addressed. I suggest authors to adopt appropriate methods to address overfitting. 

I feel the results are inadequate. Why not other performance metrics used? I suggest authors to explore other metrics too. 

Conclusion should be support by results. 

Author Response

Comment: Authors proposed a deep learning model to predict the patient eligibility for deep inspiration breath-hold breast cancer radiotherapy from respiratory signals. The following are my concerns which should be addressed in the revisions. 

Minor Issues:

I noticed some continuous references should be merged. For eg., line no. 43 and 44, the references change it to [1]-[4]

Response: done.

Comment: Check if there is a typo in line no. 48. If not what is Gy?

Response: Gy is Gray, the unit of ionising radiation dose. Changed into “Gray (Gy)” in order to introduce the units.

Section-wise outline is missing. 

Typo in line no. 271. 

Response: corrected

Major Issues:

Abstract:

The results section of the abstract present only the abstract. I noticed the authors have mentioned the hyperparameter details in the result section. It should come in the previous paragraph(before the result). Provide details about the BLSTM-RNN model and its combinations in the methods section of the abstract. In the result section, add all results I noticed only accuracy and ROC however, it will be good to mention all the other values too. 

Response: we moved the section describing the hyperparameters in the methods section as suggested. In the abstract we also briefly describe the architecture of the neural network.

Comment: Introduction: 

Comment: This section is well written and easy to follow. However, I feel the related works on Artificial Intelligence and Deep Learning techniques are weaker. There have been many improvements recently in the healthcare domain through the advancement of deep learning techniques. Authors are expected to include more state-of-the-art literature. 

Response: The introduction was edited in order to include more recent pubblications and offer a brief overview of deep learning applications in medicine in the introduction. 

Figure 4 shows the proposed model is suffer from overfitting. The training accuracy is close to 85 but test accuracy is at least 10 less. This is a serious problem to be addressed. I suggest authors to adopt appropriate methods to address overfitting. 

Response: In the revised manuscript, we added dropout layers to minimise overfitting. As a result the scores obtained on the training and test datasets during the training phase are more close than in the previous version of the manuscript.

I feel the results are inadequate. Why not other performance metrics used? I suggest authors to explore other metrics too. 

Response: in the revised manuscript we also reported the F1 score, other than accuracy, sensitivity, specificity and AUC.

Comment: Conclusion should be support by results. 

Response: The conclusion was revised in order to make it more self-explanatory, we added  the sentence “In the present work a deep neural network to predict which patients will benefit from DIBH RT from the respiratory tracks acquired early in the treatment workflow was implemented.”.

Reviewer 3 Report

This study proposes a way to use a Deep Recurrent Neural Network (RNN) with Bidirectional Long Short-Term Memory to predict whether or not a patient will be a good candidate for the deep inspiration breath-hold (DIBH) radiation treatment for left breast cancer (BLSTM).

My remarks are as follows:

1/ the title of your paper is a paragraph lengthy; pick a few key words instead.

2/ I propose a single paragraph be used for the abstract.

3/ don't place the description of the figure in the title; instead, put it in a separate paragraph after you've quoted the figure number.

4/ Add a figure as a flowchart that includes all the steps of your proposed approach

5/ after the introduction, your paper should feature a section titled "Related Work," in which you compare and contrast the suggested efforts with other works in the same or similar fields.

6/ Provide a concluding paragraph summarizing your paper's main points and arguments after presenting them in the introduction.

6/ on line 242, go back to the line following the section heading.

7/ bolster your method by showcasing an additional results,

8/ Figure 4, must be in results section not before

9/ evaluating LSTM's performance in comparison to those of other deep learning and machine learning models

10/ If you could have used any other model, why did you pick LSTM?

11/ The conclution section: should be a longer version of your conclusion, with some additional work added as a future assignment.

Author Response

This study proposes a way to use a Deep Recurrent Neural Network (RNN) with Bidirectional Long Short-Term Memory to predict whether or not a patient will be a good candidate for the deep inspiration breath-hold (DIBH) radiation treatment for left breast cancer (BLSTM).

My remarks are as follows:

Comment: 1/ the title of your paper is a paragraph lengthy; pick a few key words instead.

Response: we thank the reviewer for this useful advice. The number of words in the title was lowered from 22 to 13.

Comment: 2/ I propose a single paragraph be used for the abstract.

Response: thank you for the suggestion, however we feel the structures abstract is more suitable for a paper in the medical field.

Comment: 3/ don't place the description of the figure in the title; instead, put it in a separate paragraph after you've quoted the figure number.

Response: done

Comment: 4/ Add a figure as a flowchart that includes all the steps of your proposed approach

Response: The workflow of the study is shown in Fig.5 of the revised manuscript.

Comment: 5/ after the introduction, your paper should feature a section titled "Related Work," in which you compare and contrast the suggested efforts with other works in the same or similar fields.

Response: We could find only one paper where an approach based on artificial intelligence was employed to select patients for DIBH. This study by Lin et al,  is now mentioned in the discussion and introduction and is reference 31. 

Comment: 6/ Provide a concluding paragraph summarizing your paper's main points and arguments after presenting them in the introduction.

Response: We edited the concluding paragraph of the introduction so that now summarizes the purpose, methods, and novelty of our work.

Comment: 6/ on line 242, go back to the line following the section heading.

Response: done

Comment: 7/ bolster your method by showcasing an additional results,

Response: we also calculated F1-score in the revised results.

Comment: 8/ Figure 4, must be in results section not before

Response: done

Comment: 9/ evaluating LSTM's performance in comparison to those of other deep learning and machine learning models

Response: We agree that this type of comparison is absolutely necessary. Following this recommendation, we could find one paper which used machine machine learning for patient selection in DIBH RT using a different approach,  by Lin et al,  is now mentioned in the discussion and introduction and is reference 31. 

Comment:  10/ If you could have used any other model, why did you pick LSTM?

Response: LSTM is specialised for anaylising signals, which is the purpose f our work.

Comment:  11/ The conclusion section: should be a longer version of your conclusion, with some additional work added as a future assignment.

Response: we added a sentence to the conclusion regarding the future works for the study and edited the conclusions to be more comprehensive.

Round 2

Reviewer 1 Report

I guess this manuscript should be accepted for publication in Applied Science, since the author has completed a thorough revision.

Author Response

Thank you very much for thorough review and support.

Reviewer 3 Report

I thank the authors for the efforts made to respond to all my comments,
I believe that this paper needs more results to validate the approach.
For the comparison with other models, you don't need to find a work in the state of the art, but you can apply other deep learning models to your database and show us the results you find to validate BLSTM.

Author Response

We apologise but we were not able to perform a thorough comparison among other deep learning for signal analysis (RNN or LSTM), but also believe that it would not add much to the article. We added a sentence in the study limitations to acknowledge that it remains to be demonstrated which is the best model for the task at hand: " Also it remains to be investigated in a comparison among models for signal analysis such as the more recent convolutional or residual LSTM [74], for instance, could improve the performance of the presented model and this could be the subject of future investigations."

Author Response File: Author Response.pdf