Review Reports

Round 1

Reviewer 1 Report

In this study, Victorino et al. proposed a machine learning model to predict wearing-off of Parkinson’s disease patients using a Wrist-Worn fitness tracker and a smartphone. Even the idea is of interest, there are some major points that need to be addressed:

1. English language and presentation should be improved significantly. There are some ambiguous parts and they make it hard to follow.

2. A critical concern is the use of small-size training data without any validation.

3. Why did the authors only use Logistic Regression and Decision Tree as their selected models? Currently, there are a lot of state-of-the-art models that may reach a better performance than LR and DT, thus the authors should evaluate and compare.

4. How to select the optimal hyperparameters of the models? This should be explained clearly.

5. The authors should compare the predictive performance to previous studies on the same data/problem.

6. Why did the authors use different hyperparameters for two participants? I think both should share the same hyperparameters to claim the efficiency of the model.

7. Machine learning & cross-validation have been used in previous biomedical studies i.e., PMID: 33036150 and PMID: 32942564. Therefore, the authors are suggested to refer to more works in this description.

8. Source codes should be provided for replicating the methods.

9. Quality of figures should be improved.

Author Response

Dear Reviewer,

We thank you for all the comments you made to our study. We deeply appreciate the time you have spent to understand our study and give insightful comments.

We have updated the manuscript based on your comments. We also attached a detailed response to each of the comments.

Author Response File: Author Response.pdf

Reviewer 2 Report

In this study, the authors attempted to develop prediction models to detect upcoming wearing-off periods of the anti-parkinson's medication. Although the idea is intersting as well as highly relevant, the application is rather limited and requires extensive revision before it is ready for publication.

 

Major:

1. The main issue with the study is the very small sample size, the authors report results from only two patients. This makes any conclusion regarding the ability of the models to predict the wearing off phenomenon very tricky especially with the very high variability between the results of the two participants as well as the low precision and the high rate of false positives. It would be optimal to add more patients to the study, however in the case this is not feasible, the authors can think of alternative ways to add more analyses to the study such as for example an additional section testing the models on simulated data. In the same vein the authors should they decide to add no further analyses must state in the title that this is a case study/preliminary analysis report.
2. Especially in the case of participant 2, the very small number of wearing off effects/symptoms reported makes it even trickier to draw any useful conclusion that can be interpolated to the population level. This of course influenced the results of the models. In real life, where the possibility of a low number of reported events is potentially high, such a model would fail to provide accurate predictions. The authors should discuss ways to optimize their model estimation on the subject level to overcome the highly possible issue of having few wearing-off events.
3. There was no mention of the dosage of the medication used in both cases throughout the manuscript. Such information should be provided.
4. Line 334: “Based on the feature importance from the models, the elapsed time since the last drug intake affected both participants’ models regardless of the sampling interval. For participant 1, the stress score and the number of steps were indicators for the wearing-off phenomenon. This result coincided with the spread of stress score for participant 1”. The biggest predictor was the time since the last dose of the drug, which makes complete sense and is intuitive. However, the rest of the features in the model, e.g. stress score and steps count were only influential in one participant not the other, raising more concerns on the generalizability of such approach. Please comment on the difference in the feature contribution between the two participants. More in depth analyses of the features, could also be performed establishing links between them, For instance, participant 2 had very low REM sleep which is know to provide information on the prognosis of PD (Schroeder et al., 2011). 
5. In the same vein, the authors should also adapt a more refined feature-selection based on previous research. If I am not mistaken sleep parameters were not included in the model. Such a highly relevant feature would probably improve the percision of the model especially when selecting relevant features ( for example as mentioned above, selecting REM density instead of total sleep duration). 
6. Methods: The description of the detection algorithms is insufficent. More details on how the tracker measures heart rate, sleep, and steps are required. What kind of sensors are implemented in the tracker? Based on which physiological/physical features?...etc.
7. Figure and table legends should be more elaborate. Legends should explain the axes of the figures.
8. Line 238: “Garmin has uniquely provided stress scores in selected Garmin fitness trackers and smartwatches based on a combination of sensors.” Please elaborate on the sensors used to measure stress in the methods section.
9. Results section, Line 255: please report exact p-values and add effect sizes.
10. In the discussion, the authors state that their results which demonstrated low precision and a high false positive rate is still acceptable. This is a very strong statement that needs to be revised. Low precision and high number of false positive alarms could introduce a stressor factor that might negatively-impact the quality of life of the patients’, acting against the main purpose of such application. It could also reduce the reliability as the patients get used ot having flase alarms that often. Therefore, the low precision of the models is alarming and requires more fine-tuning which could be achieved for example by adapting a more efficient feature selection procedure.

Minor:

1. Line 53: “PD patients have also reported effects on their sleep patterns”. Please provide more information and references on the changes in sleep patterns in PD patients.
2. The introduction has no information on the relation between wearing-off phenomenon and the changes in stress levels. Please add.
3. Can the authors add an example of the available raw data from the Garmin software to the supplements?
4. Tables 5a and 5b are confusing, please provide sufficient description in the legend of the tables.
5. Could the authors justify why they replaced missing Heart rate values with -1 instead of interpolating the values before and after as they did with other measures?
6. The number of steps provided in table 6 and figure 6 is very confusing. For example, did Participant 1 walk ~2000 steps over the whole duration of the recording with an average of 34.12 steps per day? Please clarify further in text and table legend.
7. Line 314: “On the other hand, only time after drug intake were shared features for patient’s models.” Do you mean participant 2 models? Please clarify.

References:

Schroeder, L. A., Rufra, O., Sauvageot, N., Fays, F., Pieri, V., & Diederich, N. J. (2016). Reduced Rapid Eye Movement Density in Parkinson Disease: A Polysomnography-Based Case-Control Study. Sleep, 39(12), 2133–2139. https://doi.org/10.5665/sleep.6312

Author Response

Dear Reviewer,

We thank you for all the comments you made to our study. We deeply appreciate the time you have spent to understand our study and give insightful comments.

We have updated the manuscript based on your comments. We also attached a detailed response to each of the comments.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thanks for addressing my previous comments. There is still one important concern is the use of small-size training data without any validation. It is still not yet addressed well by the authors.

Author Response

Dear Reviewer,

Again, we thank you for the comments you made to our study. We deeply appreciate the time you have spent to check the revisions in our study and give insightful comments.

We have updated the manuscript based on your comments. Similarly, we also attached a detailed response to each of the comments.

Author Response File: Author Response.pdf

Reviewer 2 Report

We thank the authors for their thorough response to our comments. We have a few minor comments to add:

• • Line 57: Sentence is not clear, please revise. What exactly is the nature of "sleep disturbances" mentioned in th text?
  • I assume that the reported p-values were not equal to zero, p-values that are less than 0.001 are reported as such: p < 0.001
  • Regarding the two-sample t-test statistics. Do the authors really report two degrees of freedom? for example the authors write: "There were significant differences between the participant’s heart rate (t(300, 765) = 1.96, p = 0.00, d = 0.995)". My question is what are the numbers in the brackets after the t-values? are these two values separated by a comma? If so, then something is wrong. Is this only one value and the comma indicates the fraction? then the authors should be consistent as they use the dot in the case of p-values and d's. Please clarify.
  • Supplements are very messy and hard to navigate through, please add proper labelling and description for each suppl. material and merge, when applicable, all suppl. information in one file/sheet.

Author Response

Dear Reviewer,

Again, we thank you for the comments you made to our study. We deeply appreciate the time you have spent to check the revisions in our study and give insightful comments.

We have updated the manuscript based on your comments. Similarly, we also attached a detailed response to each of the comments.

Author Response File: Author Response.pdf