Integrating Speculation Detection and Deep Learning to Extract Lung Cancer Diagnosis from Clinical Notes

Round 1

Reviewer 1 Report

Development of an OCR (optical character recognition) based methods for handwritten or digital health record is really important task, however I have few comments, questions and doubts about presented solution and the manuscript overall:

1. First of all, English language must be improved. There a lot of grammar mistakes, some phrases are repeated too many times at the same paragraph, therefore sentences look poor and overall text does not sound scientific;
2. The purpose of this research is not clear because of following reasons:

• Why it is a challenge to relate cancer concepts to the date of diagnosis? And for whom? I understand that this information is useful for patient and physicians. But both sides can read it from medical statements, e-records or other documents.
• What is the main problem? It is not written what kind of problems can be solved using the proposed method. Citation “Extracting the lung cancer diagnosis can be useful to support clinical research because the diagnosis is a crucial factor for both the effective control of the disease, as well as the design of treatments”. My question – is it impossible to do that now?
• What is the applicability of this method? Let’s say authors are creating the electronic health system (I assume that Spain do have such system at the moment), but in such case all medical notes can be scanned and converted into editable and searchable data using OCR technologies. Then all history records and diagnoses can be extracted from the content. But the question is- why do we need that?
• It is not clear why it is a specific task to recognize “lung cancer diagnosis in Spanish language”.
• The contribution of this research is estimated based on previous research of the same authors. Authors give a lot of attention to their previous researches (reference number [10] and [54]). I have read these papers and as I understood the F-score has been increased from 84%  [10] to 90% for the same problem - entity recognition task.  90% results have been achieved using 4 Regex or 5 rules. However 4 rules are taken from previous research [54]. It was written that these four rules were adapted for the problem, therefore I think a few sentences about this adaptation may by useful for the reader. 

1. Figure 5 must be described in more details: no explanation for x and y axis, in the text only blue words are mentioned (no comments about green and red words). Below (Figure 2) the numbered sentences are provided, but my suggestion is to move them in another manuscript place. For example, line 269.
2. It was stated that “The collected cues are translated to Spanish using Google translate, and later they are manually corrected”, I think it is useful to provide the number of cues.
3. Technological part (for example, used CNN architecture, implemented algorithms, pseudo codes, formulas, learning metrics and etc.) is short and very abstract. For example, no explanation of abbreviation - CRF (it is Conditional Random Fields), the purpose and advantages of such layer, besides reference [47] is incorrect –no names or title of paper.
4. Authors are using BiLSTM-CRF model, but it is not explained what are the inputs and output of this model? It is not clear how training and testing data sets were prepared and what data were included in the data set
5. It is not clear how the text recognition is done. What kind of text they are working with  - hand written or computer based? Is there any structured form were diagnosis can be written, or free writing is left? 
6. Authors write that „To the best of our knowledge, there is no approach to extract cancer concepts and relate them to the patient’s natural history for the case of clinical notes written in the Spanish language.” But it‘s not clear is there any specificity or novelty in a methodological sense just because of Spanish language (bearing in mind that many diagnosis are in Latin or have codes). I think this is a matter of annotation and vocabulary, but not the advantage of techniques.
7. It suggests that the collection in Spain language was composed. How it was validated? Is this collection publicly available?
8. The sentence length analysis casts doubt on the effectiveness of such process, because authors write that the text in “these records is short and efficient”, but in section 5.3 the length is analyzed trying to find shorter sentences in the short sentence from the beginning.
9. Chapters from 3 to 6 look messy because sentences and method steps are repeated several times, the integration of different methods is unclear, some sections are very short. Therefore, the methodological section needs to be rewritten without repeating the same sentences. In addition, the journal has a requirement for a chapter “Materials and Methods: They should be described with sufficient detail to allow others to replicate and build on published results. New methods and protocols should be described in detail while well-established methods can be briefly described and appropriately cited.”
10. In Chapter 7 no precise information is provided on the validation samples (except 7.3). On which data basis those estimates were obtained?
11. Is it enough to have 5 rules? What if we increase the number of rules? Does it mean that we could expect that the results will be improved if the number of rules is increased
12. According to the final results given in Section 7.3, it is still not clear how this technique helps to relate cancer diagnosis and the date when is was diagnosed. How these results are affected if there exist some grammar mistakes?
13. After reading the manuscript the question arises - why this methodology cannot be used to extract diagnosis of other types of cancer. No specific rules or methods are included due to lung cancer case. If there is any specific (medical) information which can be used only for this type of cancer, it should be clearly denoted. The reason is that the examples of notes provided in the paper except the word “lung” can be used for other types of cancer as well. (“father dies”, “carcinoma”, “stage”, “chemotherapy“, “diagnosed”, “treated”, “date” and etc.)
14. In the Discussion chapter the additional and repeating description of the proposed method should not be included. The journal requires as follows: „Authors should discuss the results and how they can be interpreted in perspective of previous studies and of the working hypotheses. Future research directions may also be mentioned“.

Author Response

Dear Editor and Reviewers,

Please find attached the revised version of the Manuscript: “Integrating Speculation Detection an Deep  Learning to Extract Lung Cancer Diagnosis from Clinical notes”, which we resubmit for consideration to Integration and Mining of Data from Mobile Devices, Special Issue. We have discussed and implemented all Referees’ suggestions.

"Please see the attachment."

Author Response File: Author Response.pdf

Reviewer 2 Report

In this paper, the authors extend their previously published approach to detect lung cancer from clinical notes, by combining deep learning-based and rule-based methods to address the limitations of the previous work.

The manuscript is well written, apart from a few minor typos (e.g. line 571, the translation is wrong). Moreover, the description of the approach and the authors contributions are clear.

However, the main limitation I note is the absence of a methodological description of the type of validation carried out (tuning of parameters, number of experiments, type of cross-validation, division between training and testing, statistical significance test). It is a bit unfair to want to publish a study based on deep learning, but which does not go into the merits of the validation used (and by the way, uses the F1-Score as the main metric). 

It is therefore necessary to clarify these aspects in detail, as well as to make it possible to access the dataset (there is a description but no link) and possibly also the source code of the solution, for reasons of reproducibility of the results. Otherwise, I do not think I am in a position to assess the actual goodness of the method (but only that of the text).

Finally, the authors should enrich the scientific bibliography of the work with a brief overview of the modern use of deep learning and CNN techniques, not only in the NLP or medical/health domains, but are increasingly being applied to various sectors, eg. finance ([https://doi.org/10.1016/j.eswa.2020.113820]), security ([https://doi.org/10.1109/ACCESS.2020.2968718], smart cities ([https://doi.org/10.1016/j.trpro.2020.03.079], etc.

Author Response

Response to reviewer comments.

Dear Editor and Reviewers,

Please find attached the revised version of the Manuscript: “Integrating Speculation Detection an Deep  Learning to Extract Lung Cancer Diagnosis from Clinical notes”, which we resubmit for consideration to Integration and Mining of Data from Mobile Devices, Special Issue. We have discussed and implemented all Referees’ suggestions.

"Please see the attachment."

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I want to thank the authors for answers and explanations. I just have one comment - avoid narow sections such as 4.2.1 or 4.2.2. But it's more like recommendation. About figure 5: it is thus understandable that we have x and y axis, but but for the reader  it is more informative to write the name of parameter/feature instead of x and y. 

Author Response

Dear Editor and Reviewers,

Please find attached the revised version of the Manuscript: “Integrating Speculation Detection and Deep Learning to Extract Lung Cancer Diagnosis from Clinical notes”, which we resubmit for consideration to Integration and Mining of Data from Mobile Devices, Special Issue. We have discussed and implemented all Referees’ suggestions.

 "Please see the attachment."

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors have adequately addressed all my comments. In particular, I appreciated the completeness and clarity of the new section about the validation setting, as well as the publication of the source code of the approach. In addition, the manuscript has overall improved also in the other sections, especially with regard to some relevant concerns raised by the other Reviewer. There are two remaining minor issues:

1. The quality of the writing is still poor, several typos are still present;
2. The Readme of the repository is also poor, there is no precise description of how to install and run the code in order to reproduce the experiments.

Author Response

Dear Editor and Reviewers,

Please find attached the revised version of the Manuscript: “Integrating Speculation Detection and Deep Learning to Extract Lung Cancer Diagnosis from Clinical notes”, which we resubmit for consideration to Integration and Mining of Data from Mobile Devices, Special Issue. We have discussed and implemented all Referees’ suggestions.

"Please see the attachment."

Author Response File: Author Response.pdf