Parkinson’s Disease Detection from Drawing Movements Using Convolutional Neural Networks

Round 1

Reviewer 1 Report

The document describes a CNN to classify data into originating from ALzheimer's patients or healthy subjects. The methodology, subject-wise five-fold cross-validation, is correct. I fell, however, that there is no big novelty (a standard CNN was applied to an existing dataset) and that the document should be enriched in several ways, namely by performing experiments with other models and/or comparing with other results in the literature (comparison was only made with 2 other works).

  Some specific comments include:
- "There is currently no objective test for PD and the diagnosis can be confirmed only by a pathological analysis at autopsy." In the used database, the diagnosis has been confirmed by autopsy?!!!
- Since this is a time-signal, why haven't other architectures, such as recurrent networks, been applied?
- What is the reasoning behind the architecture of the chosen CNN (2 convolutional layers followed by 3 fully connected layers)? Why not use more standard structures such as AlexNet?
- I would like to have seen examples of the input (2D matrix with N x 125 dimensions)
- Please homogeneize the notation. CNN and DNN are used interchangeably. - I do not understand how you can show the results for X, Y, Z, PRE, GRID independently since they were all combined in a single image.
- I feel like some important references might be missing, such as:

[1] D. Impedovo, G. Pirlo, Dynamic Handwriting Analysis for the Assessment of Neurodegenerative Diseases: A Pattern Recognition Perspective, IEEE Rev. Biomed. Eng. 12 (2019) 209–220. doi:10.1109/RBME.2018.2840679.
[2] A. Ranjan, A. Swetapadma, An Intelligent Computing Based Approach for Parkinson Disease Detection, in: 2018 Second Int. Conf. Adv. Electron. Comput. Commun., 2018: pp. 1–3. doi:10.1109/ICAECC.2018.8479490.
[3] P. Khatamino, İ. Cantürk, L. Özyılmaz, A Deep Learning-CNN Based System for Medical Diagnosis: An Application on Parkinson’s Disease Handwriting Drawings, in: 2018 6th Int. Conf. Control Eng. Inf. Technol., 2018: pp. 1–6. doi:10.1109/CEIT.2018.8751879.
[4] C. Gallicchio, A. Micheli, L. Pedrelli, Deep Echo State Networks for Diagnosis of Parkinson’s Disease, CoRR. abs/1802.06708 (2018). http://arxiv.org/abs/1802.06708.
[5] D. Impedovo, G. Pirlo, Chapter 7: Online Handwriting Analysis for the Assessment of Alzheimer’s Disease and Parkinson’s Disease: Overview and Experimental Investigation, Ser. Lang. Process. Pattern Recognition, Intell. Syst. Pattern Recognit. Artif. Intell. (2019) 113–128. doi:https://doi.org/10.1142/9789811203527_0007.

Author Response

First of all, the authors appreciate so much the reviewer’s comments. These comments have helped to improve the paper. We have included answers in bold after every comment.

 

REVIEWER 1

The document describes a CNN to classify data into originating from ALzheimer's patients or healthy subjects. The methodology, subject-wise five-fold cross-validation, is correct. I fell, however, that there is no big novelty (a standard CNN was applied to an existing dataset) and that the document should be enriched in several ways, namely by performing experiments with other models and/or comparing with other results in the literature (comparison was only made with 2 other works).

Many thanks for this comment. In the new version, we have included a detailed table (Table 1) describing the main characteristics of previous works on PD detection using drawing movements. The two new references used the same dataset than in this paper. According to this detailed comparison, the current paper shows the best results on this dataset (the Parkinson Disease Spiral Drawings Using Digitized Graphics Tablet dataset [28]) considering a subject-wise cross validation.

  Some specific comments include:

- "There is currently no objective test for PD and the diagnosis can be confirmed only by a pathological analysis at autopsy." In the used database, the diagnosis has been confirmed by autopsy?!!!

Many thanks. We are not sure about it, probably, not. Just in case, in the new version of the paper we have removed this strong affirmation.

- Since this is a time-signal, why haven't other architectures, such as recurrent networks, been applied?

Many thanks for this comment. It is an interesting aspect to try. As we do not have enough time to run more experiments, we have decided to include this aspect as an interesting future work line in the last paragraph of the conclusions section.

- What is the reasoning behind the architecture of the chosen CNN (2 convolutional layers followed by 3 fully connected layers)? Why not use more standard structures such as AlexNet?

Many thanks for this comment. The introduction section and section 2.3 have been expanded to clarify this aspect. Mainly, there are two aspects:

1.- The CNN proposed is very similar to the CNN described in Khatamino et al. [27]. This architecture is a traditional one with two main parts. The first one contains convolutional layers for feature extraction and the second part includes fully connected layers for classification.

2.- Actually, the CNN was inspired in AlexNet architecture but with a lower number of layers. The main reason for reducing the number of layers is because the dataset in this work is very small, so we need to reduce the number of parameters to be trained.

 

- I would like to have seen examples of the input (2D matrix with N x 125 dimensions)

Thanks for this comment. In the new version, we have expanded the preprocessing description in section 2.2 and we have also included figure 2 detailing the preprocessing step. In figure 2, we show the spectrum obtained for a 3-second window from a PD patient. In this representation, it is possible to see peaks of energy corresponding to the fundamental frequency of the tremor and its harmonics.

- Please homogeneize the notation. CNN and DNN are used interchangeably.

Many thanks. The paper has been revised and all DNN references have been changed to CNN.

- I do not understand how you can show the results for X, Y, Z, PRE, GRID independently since they were all combined in a single image.

Many thanks for this comment. The single image is a 2D matrix with N x 125 dimensions. N is the number of signals considered in the CNN. N is equal to one when analyzing one signal independently and N is equal to five when using all the time series for x, y, z, pressure and grid angle.

This description has been expanded in section 2.3.

- I feel like some important references might be missing, such as:

Many thanks for these suggestions. They have been very interesting to complete the state of the art included in the introduction section.

We have added all the references except number [2] (Ranjan and Swetapadma). This reference used the same dataset, but the authors did not include details about the feature extraction or the cross-validation methodology, so we could not make a detailed and fair comparison with our work.

 

[1] D. Impedovo, G. Pirlo, Dynamic Handwriting Analysis for the Assessment of Neurodegenerative Diseases: A Pattern Recognition Perspective, IEEE Rev. Biomed. Eng. 12 (2019) 209–220. doi:10.1109/RBME.2018.2840679.

[2] A. Ranjan, A. Swetapadma, An Intelligent Computing Based Approach for Parkinson Disease Detection, in: 2018 Second Int. Conf. Adv. Electron. Comput. Commun., 2018: pp. 1–3. doi:10.1109/ICAECC.2018.8479490.

[3] P. Khatamino, İ. Cantürk, L. Özyılmaz, A Deep Learning-CNN Based System for Medical Diagnosis: An Application on Parkinson’s Disease Handwriting Drawings, in: 2018 6th Int. Conf. Control Eng. Inf. Technol., 2018: pp. 1–6. doi:10.1109/CEIT.2018.8751879.

[4] C. Gallicchio, A. Micheli, L. Pedrelli, Deep Echo State Networks for Diagnosis of Parkinson’s Disease, CoRR. abs/1802.06708 (2018). http://arxiv.org/abs/1802.06708.

[5] D. Impedovo, G. Pirlo, Chapter 7: Online Handwriting Analysis for the Assessment of Alzheimer’s Disease and Parkinson’s Disease: Overview and Experimental Investigation, Ser. Lang. Process. Pattern Recognition, Intell. Syst. Pattern Recognit. Artif. Intell. (2019) 113–128. doi:https://doi.org/10.1142/9789811203527_0007.

 

Reviewer 2 Report

This work is publishable upon addressing the following comments.

Please be consistent in writing convolutional neural networks as stated in title. There are “deep neural network”, “neural network”, etc. Abstract, please clarify the “The best obtained results”. References, the format is incorrect. Please follow the author guideline. Please provide a latest reference to replace reference [4]. Please update the content accordingly. Similarly, please provide a latest reference to replace reference [5]. Please update the content accordingly. Also, what is the typical accuracy? Introduction, please add a table to summarize the methodology, dataset and performance of related works. Introduction, authors summarized three contributions. However, these statements are the standard steps of methodology. Please be concise and discuss the innovation and technical contribution. Authors mentioned the dataset is retrieved from [26]. Does the data available in any website? Section 2.2, please explain the reason for “All the recordings were resampled to the same sampling rate of 110Hz.” Section 2.2, line 113, what is “with 2.5-second overlap.”? Section 2.2, line 116, please explain more on “125-point representation of the spectrum”. Section 2.2, please elaborate more on the power spectrum (figure and/or table should be included). Section 2.3, please justify with reasons the parameters setting clearly. Section 2.3, line 130, please clarify “1 PD patients and 0 healthy subjects”. Section 2.4, authors are suggested to cite the following article as the common practice of having cross validation. A Novel MOGA-SVM Multinomial Classification for Organ Inflammation Detection Section 3, authors are suggested to explain the reasons yielding better performance by proposed work compared to [15] and [23]. Section 4, should the algorithm use direction Z as a result?

Author Response

First of all, the authors appreciate so much the reviewer’s comments. These comments have helped to improve the paper. We have included answers in bold after every comment.

REVIEWER 2

Please be consistent in writing convolutional neural networks as stated in title. There are “deep neural network”, “neural network”, etc.

Many thanks, we have revised the paper rewriting all these terms to CNN.

Abstract, please clarify the “The best obtained results”.

This sentence has been rewritten: “The best results obtained in this work”

References, the format is incorrect. Please follow the author guideline.

Many thanks. The references have been revised accordingly.

Please provide a latest reference to replace reference [4]. Please update the content accordingly. Similarly, please provide a latest reference to replace reference [5]. Please update the content accordingly.

Many thanks, these two references have been updated and the content modified accordingly.

Also, what is the typical accuracy?

Many thanks, we have defined accuracy as the percentage of examples correctly classified the first time it is used.

Introduction, please add a table to summarize the methodology, dataset and performance of related works. Introduction, authors summarized three contributions. However, these statements are the standard steps of methodology. Please be concise and discuss the innovation and technical contribution.

Many thanks. We have added a new table (Table 1) describing the main characteristics of similar systems described in previous works. We have also included the information from this paper. After this table, we have included a concise discussion of the differences between these systems and our work, remarking the technical contribution of this work.

This table has been very useful to improve the description of our system and the comparison with previous works.

Authors mentioned the dataset is retrieved from [26]. Does the data available in any website?

Many thanks. Yes. There is one link where you can download the dataset. We have included this link to the reference: https://archive.ics.uci.edu/ml/datasets/Parkinson+Disease+Spiral+Drawings+Using+Digitized+Graphics+Tablet

Section 2.2, please explain the reason for “All the recordings were resampled to the same sampling rate of 110Hz.”

Many thanks. This explanation has been included:

“The dataset was obtained in two phases with two different sampling rates, 110 Hz and 140 Hz. In order to uniform the data, all the recordings were resampled to the same sampling rate of 110Hz.”

Section 2.2, line 113, what is “with 2.5-second overlap.”?

Many thanks. This explanation has been included:

“The sample sequence was divided into 3-second windows (330 samples per window) separated by 0.5 seconds (it means a 2.5-second overlap between two consecutive windows).”

 

Section 2.2, line 116, please explain more on “125-point representation of the spectrum”. Section 2.2, please elaborate more on the power spectrum (figure and/or table should be included).

Thanks for this comment. In the new version, we have expanded the preprocessing description in section 2.2 and we have also included figure 2 detailing the preprocessing step. In this figure 2, we show the spectrum obtained for a 3-second window from a PD patient. In this representation, it is possible to see peaks of energy corresponding to the fundamental frequency of the tremor and its harmonics.

 

Section 2.3, please justify with reasons the parameters setting clearly.

The introduction and section 2.3 have been expanded:

1.- The CNN proposed is very similar to the CNN described in Khatamino et al. [27]. This architecture is a traditional one with two main parts. The first one contains convolutional layers for feature extraction and the second part includes fully connected layers for classification.

2.- Actually, the CNN was inspired in AlexNet architecture but with a lower number of layers. The main reason for reducing the number of layers is because the dataset in this work is very small, so we need to reduce the number of parameters to be trained.

 

Section 2.3, line 130, please clarify “1 PD patients and 0 healthy subjects”.

This sentence has been reworded: “As we classify between two classes, the last layer has only one output with a sigmoid function. This output should be close to 1 for PD patients (class 1) and close to 0 for healthy subjects (class 0).”

 

Section 2.4, authors are suggested to cite the following article as the common practice of having cross validation.  A Novel MOGA-SVM Multinomial Classification for Organ Inflammation Detection.

Many thanks for the suggestion. Reference added.

Section 3, authors are suggested to explain the reasons yielding better performance by proposed work compared to [15] and [23].

Many thanks. Section 3 has been expanded remarking this aspect. The main reason for obtaining better results compared to previous works have been the use of a CNN considering the spectrum points as inputs to the CNN. Tremor is the most prevalent PD symptom and it becomes more apparent in the frequency domain: In the spectrum of the x and y coordinates, it is possible to see peaks of energy corresponding to the tremor frequency (between 3–9 Hz) and its harmonics.

 

Section 4, should the algorithm use direction Z as a result?

Many thanks. We agree that the information obtained from direction Z or the grid angle is very small, but when we remove these spectra from the 2D input matrix, we observed a small loss of performance. This loss is not statistically significant, and we did not include any specific comment in the discussion. From our point of view, it would be necessary an analysis with a larger dataset to verify if these two signals could be eliminated. We have included a comment in the last paragraph of the conclusion section where some future work is proposed.

 

 

 

Round 2

Reviewer 1 Report

The authors have performed most of my suggestions. Please try for Tables and Figures to show after being mentioned in the text. What was the statistical test made to support the "not statistically significant" claims?

Author Response

First of all, the authors appreciate so much the reviewer’s comments. These comments have helped to improve the paper. We have included answers in bold after every comment.

 

REVIEWER 1

The authors have performed most of my suggestions. Please try for Tables and Figures to show after being mentioned in the text.

Many thanks. All figures and tables have been modified accordingly.

 

What was the statistical test made to support the "not statistically significant" claims?.

We have added a new sentence:

When removing these spectra from the 2D input matrix, we observed a small loss of performance in AUC (< 1%), but not statistically significant (according to Hanley’s method [33]).

Reviewer 2 Report

Authors have addressed all of the comments.

Author Response

Authors have addressed all of the comments.

Many thanks