Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper describes the development of a system for assessing physical performance tests and its validation with the prospective study on adults undergoing knee and hip arthroplasty within a rehabilitation program.

Overall, the paper is well written, the methods are described in detail, and the system's evaluation is thorough. My two main comments are about the calibration method – this would be very useful for the reader and any researcher in the field and an extra paragraph or even subsection would add great value to this paper. The other comment I have is on Figures and Tables. These should be standalone parts so that the reader can look at them and understand them by only reading the caption. This is a general rule that I suggest the authors apply to any publication. Not every reader will have time to go through a 22-page paper, and you want to present the key results in such a way that a quick skim through gives a reader a good idea if the paper is in their interest and therefore worth reading in-depth. Label all axes on graphs, write at least a sentence in the caption on what the figure/table represents, and write out all abbreviations used in the figure/table. Overall, I believe this is an important work that will be built on by other researchers and its presentation is of great importance.

Specific comments:

Introduction

-          Needs to be a good overview of all possible assessments/tests used in the field, the state of the art. You only focused on the tests that you use in your system. Please mention other (max and sub-max) tests that are also an option but have the limitations that you mentioned. Right now, the reader doesn't know what other tests exist, only that they aren't great in comparison to the three you have chosen to study.

-          Line 71 – add references to these in lab tests

Great in-depth review of apps and sensory systems for the three tests of interest.

Methods

-          Any abbreviations on Figure 4 should be written out in the caption. An additional sentence under Figure 4 would be useful - a brief description.

-          Similarly for Figure 5. Additional sentence, something like "An example of an app screenshot for results of the 30sts test." would be useful.

-          You could potentially combine the two figures and expand the caption. Left: architecture, right example of the results display. This is completely optional.

App architecture is described in great detail.

-          Figure 6 - add labels for the x and y axis, magnitude of what signal? Add to caption. Y= value could be useful to add for both red line thresholds on the line itself

-          Line 285 as well - missing space

-          Line 300-303: can you share the exact threshold you used as an example? Do you train these for an individual or are they predetermined? A time window and a threshold could be useful for other researchers developing similar applications.

-          Line 305-312: how do you ensure the distance is accurate for a remote measure? Could you estimate the distance just based on the sensors/steps count? This assumes that the user will accurately measure the distance beforehand and perform the test.

-          Line 319: could you expand on the calibration process? This paragraph is very promising and answers my previous comment, but it doesn't say how.

-          2.2.2 ten meter walking test implementation has a complicated setup to be performed remotely and correctly by the patient. I can imagine this being set up at a clinic but for a remote measure it doesn't look practical. If you have any ideas on how to simplify this in the future - please add to the discussion/future work and mention as limitation. 

-          Line 388 any reference for the thigh angle using gyroscope being the most accurate?

-          Line 437 older people (aged more than 60 y) – I recommend changing that to elderly (over 60 years old)

-          Figure 8 caption - name both devices and location. Name the test they're about to perform. Again, caption should be enough to understand the figure without reading the text. This figure is useless as is but can be made valuable. 

Overall, the text could be a bit more concise by skipping some of the more subjective phrases and sentences. Stick to the facts, skip the commentary - that belongs to the discussion section. 

Great work in keeping the reader informed on each section and how the text is structured. This is a longer paper and it is important to make the information clear throughout the paper.

Results

-          Figure 9, write out abbreviations IPA, RRA, as well as test names

-          Table 1 and 2, all abbreviations in the caption

Discussion

The calibration/recalibration process seems to be an important component of the test, ensuring more accurate results yet it is barely mentioned in the methods. This needs to be expanded and explained in more detail - how it is done, what parameters are adjusted, whether they are manually adjusted by the clinician /researcher, or whether there is an automatic process encoded within the app.

One limitation, not mentioned in the discussion is the setup for the tests. The app itself isn't ready to use on its own, at least not for the 10mwt. The setup is quite complicated, and a patient would have to have someone set it up in their environment to perform the test. 

Author Response

First of all, we wanted to thank the reviewer for their comments. We tried to address all of them and we think the paper is now significantly better than the previous version.

We are attaching a table addressing each comment from the reviewer with a specific answer. To improve the readability of the new or edited parts we highlighted them using blue text.

We also improved other minor details throughout the manuscript.

Thanks for your time,

the authors.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The stated purpose of the study was to assess an App-based system, called STEP, that automates the execution of physical performance tests to track/evaluate physical performance. If I understand things correctly the study is about highlighting ‘proof of concept’ (and validation) of the system, by showing its utility in tracking the rehabilitation of patients that have recently undergone surgery for hip or knee joint replacement. The ms would certainly of interest to readers of Biomechanics, and appears to contain results that demonstrate usefulness of the STEP system. However, the manuscript has a somewhat over-elaborate/verbose writing style, which makes it challenging to read and hence difficult to fully comprehend. I believe rewriting the many over-elaborate/ verbose sections of the ms (I provide examples of such sections below) to make them simpler, concise and clearer to understand would make the ms more comprehendible, and this would greatly strengthen its accessibility and bearing. I also have one or two other major concerns, and have highlighted many minor issues – all of which are listed below.

 

Major concerns.

1. The writing style is somewhat verbose, ‘crammed’, and/or overly expressed (elaborate). For example, the first section of the discussion (lines 633-694) is a 60-line paragraph!! Similarly, Section 3.1 has several overly long and verbose paragraph, and the section as a whole is crammed with lots and lots of diverse results. This overly crammed style makes it difficult for the reader to assimilate all of what is being said, and this makes comprehension of the key aspects challenging. I suggest that much of the ms should be rewritten in a more concise (e.g. avoid overly long paragraphs such as the first section of the discussion) and structured way (e.g. split section 3.1 into sub-sections), and this will make the ms easier to understand and will thus improve its impact.

 

2. If I understand things correctly the study is about highlighting ‘proof of concept’ of the system, by showing its utility in tracking the rehabilitation of patients that have recently undergone surgery for hip or knee joint replacement. If indeed the study is about ‘proof of concept’ for using the STEP system to assess rehabilitation, the results presented are confusing as they refer to validation and to patient factors that are associated with STEP test performance.

            My concern here is probably a consequence of the aim of the study not being well defined. For instance, which of the following best reflects the aim of the study: (abstract) “we aimed to assess the role of a System for Tracking and Evaluating Performance ………….aiming to automate the execution of physical performance tests.”; (line 162) “we propose a System for Tracking and Evaluating Patient Performance; and (line 166-67) “The system was validated during the rehabilitation of patients with knee or hip replacement surgery.”?

            I believe a better defined/more exact aim, would help in making the ms more comprehendible. Along with a better defined aim, the ms also needs to clarify how aspects such as ‘proof of concept’, validation, patient factors associated with test performance, etc., are related to the aim(s) of the study. Currently, most of the above aspects appear in the ms in the results section (i.e. like an afterthought) rather than being outlined a priori in the methods.

3. Related to point 2 above. If the study is about highlighting ‘proof of concept’ of the system, by showing its utility in tracking the rehabilitation of patients that have recently undergone surgery for hip or knee joint replacement. Then more details of the rehabilitation undertaken and the time period it was completed over are needed. This is important because one would expect that STEP test performance would improve quickly in patients who have recently undergone joint replacement surgery. However, STEP test performance may not improve that much if at all in those patients who underwent joint replacement surgery 12 months previously or longer. Connected to this point, it isn’t made that clear in the results and discussion whether the STEP system did indeed show test performance improved in joint surgery patients post rehabilitation – this aspect seems to have been lost in the results/discussion.

 

I’m wondering whether a complete rewrite/restructuring of the ms is warranted. You could considered splitting the ms into two sections: the first section could be about presenting the STEP system and describing what it does and how it works. The second section could be about showing the system’s usefulness in tracking improvements in functional abilities following rehabilitation in patients who have recently undergone joint replacement surgery? The reason the current version of the ms is difficult to read (and is overly elaborate/crammed) is because it tries to cover both these aspects (presenting the system; highlighting the system’s usefulness) all in one go.

 

Minor issues

- Line 29 “According to (13)….” Need to insert name of authors or something generic here, e.g. “according to previous research….”

 

- Lines 29-34. It is not clear why you have highlighted cardiorespiratory fitness? Surely the test used in the STEP system are clinical test used to evaluate functional ability/abilities?

 

- Lines 35-47. I don’t think this level of detail (about these tests) is needed. It would be better to simply make reference to previous research?

 

- Line 69. “A wide range of sensors is available..” This should be ‘are’ available.

 

- Line 107-08. “….The good results….” It is not clear what this sentence is referring to – needs rewording.

 

- Line 138. “Regarding the other 2 tests” – it is not clear what the other two tests are?

 

- Line 180. “Although the back end…” – what does back end mean?

 

- Line 256. “The tree tests…” I presume this should read three tests?

 

- Figure 6. What data is being presented in this figure? More details are need, e.g. in the caption, axes, and/or a footnote.

 

- Section 2.2.1 6-min walking test – its not clear where the smartphone was placed.

 

- Line 306. “length of path….is known” – clarify if the length is inputted into the App?

 

- Line 320 “After the test…..performer inputs the real number of steps” – how is the real number determined?

 

- Line 326. “..the App can detect…..” – which App are you referring to?

 

- Lines 329-30. Can we presume the two colours used need to be contrasting colours – rather than different hues of the same colour? Need to clarify.

 

- 2.2.2 10-m walking test. How does the system guard against randomly placed items in the environment (e.g. posters or pictures) that have sections of colour that are the same as the target colour?

Also, are the coloured targets set a specific (known) distance apart before the test starts? If so, how is such controlled? If the objects are set a specific distance apart, how is the width/size of the object factored in?

 

- Lines 388-394. Can you provide a reference to support what you are suggesting?

 

- 2.2.3 sit-to-stand test. Your description includes reference to rotation of the thigh. Rotation implies angular displacement. However, my understanding is that a gyroscope provides angular velocity signal. Thus clarify what signal was assessed.

 

- Line 439. Previous joint placement – need to clarify if the time since joint replacement was part of the recruitment criteria.

 

- Line 463. Need more details about the rehabilitation undertaken and the rehabilitation period.

 

- Table 1. Why is absolute difference shown? It would be better to see the direction of the difference as well as the magnitude, as this would show if PRA systematically under/over- estimated things.

 

- 3.1 Validation. The methods section needs to describe, a priori, how validation was defined and evaluated.

 

- Table 1. What are the units for the data presented?

 

- Table 2. It is unclear what data is being presented. Need more details in the caption and/or footnote.

 

- Lines 539, 543, 548. “… correlation…of distance/velocity improvement measured after the rehabilitation”. It is not clear how correlation is used to highlight improvement in test performance after rehabilitation? Wouldn’t a difference test (e.g. paired t-test; ANOVA) be more appropriate to use to show improvements in test performance?

 

- Section 3.1. There is repeated reference to Bland-Altman plots. However, without seeing such plots the reader cannot interpret what is being said – and thus the repeated reference to Bland-Altman plots is meaningless. Perhaps instead, consider including Bland-Altman agreement values (with 95% CI)? Is reference to Bland-Altman analysis even warranted, given the strong correlations shown in Figure 9? What does Bland-Altman add here?

 

- 3.2 Data analysis. It is not clear how correlation between PRA and patient physical factors relate to the aim of the study?! Inclusion of this analysis is confusing and I don’t think presented it, necessarily adds anything to the ms. You have highlighted (line 626-27), that BMI is associated with test performance. How does this help in showing ‘proof of concept’ for the STEP system? I would suggest this extra analysis (correlation between PRA and physical characteristics) is a topic for a follow-on study.

 

- Because of a lack of a clear and well-defined aim, I found the discussion ‘hard going’ (difficult to read). This is because it tries to discuss too many things concurrently, and as a result things become muddled. When you rewrite to make the ms simpler/clearer, make sure the discussion directly discusses aspects related to the aim of the study.

Comments on the Quality of English Language

see above

Author Response

First of all, we wanted to thank the reviewer for their comments. We tried to address all of them and we think the paper is now significantly better than the previous version.

We are attaching a table addressing each comment from the reviewer with a specific answer. To improve the readability of the new or edited parts we highlighted them using blue text.

We also improved other minor details throughout the manuscript.

Thanks for your time,

the authors.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have addressed all my comments. 

Author Response

Thank you for all your help.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have done an okay job at addressing the comments and suggestions I made on their original submission. However, I think further improvements are still warranted: both major and minor – see below.

 

Major concerns

1. Please consider shortening the introduction. Currently, it is too wide-ranging and too detailed, and as a result it is overwhelming (exhausting) for the reader to get through. I would suggest that shortening it to a third of its current length, could be achieved without removing any of the key points – and that such would greatly improve the readability of this ms. At the moment the reader gets drained by reading the overpowering introduction and therefore has less motivation to read the rest of the ms!

2. If I understand things correctly, the rationale put forward is the development and validation of an App that can be used for the monitoring and assessment of physical capability, with the ultimate aim that the App (system) will be used remotely, i.e. by a patient in their own home. A concern here is that two of the three assessment tasks used – 6MWT and 10MWT, typically involve walking over a level and straight line path greater than 10m. Obviously in home environments, it is unlikely there will be level and straight line paths greater than 4-5m, with some homes less than that. I realise you have factored in changes in direction in the App for the 6MWT. However, you haven’t factored in the possibility of such changes in direction for the 10MWT. Part of the rationale regarding why you chose the 6MWT and 10MWT is that such prioritize patient safety. I would suggest that having to complete these tests, particularly the 6MWT, with multiple changes in direction could compromise safety. For example, someone who has just had a knee or hip joint replacement is unlikely to want to make multiple changes in direction (within a 6 minute period), simply to complete an assessment of their functional capabilities. Some consideration of these issues need to be accounted for.

3. There seems to be a misunderstanding within the ms about rehabilitation versus assessment of functional capabilities. For example, line 20 makes reference to “rehabilitation processes, such as tracking from injury/surgery”. Tracking a person’s recovery from injury/surgery, is not the same as the actual rehabilitation they might do. Rehabilitation typically involves recommended exercises and/or stretches. There are other examples in the ms that appear to conflate rehabilitation and assessment (e.g. line 776). I would say that even the title is somewhat misleading. You have developed a system for the remote monitoring of physical capabilities. You haven’t developed a system to support rehabilitation.

 

 Minor issues

1. Line 2.  “…….in-clinic assessments……” – it is not clear what assessments are being referred to here.

 

2. Line 30. “While these tests……….”. Insert the word ‘maximal’; i.e. “While these maximal tests……….”.

 

3. “…to cover the most significant distance….”. Not clear what ‘significant’ means in this context?

 

4. Lines 52-56. Reference to a Wingate test is not required here – indeed its inclusion seems quite random/tangential to the topic of the study. I would suggest to delete.

 

5. Line 76. (e.g., saturation…..), should this read ‘blood’ or ‘oxygen’ saturation’?

 

6. Line 95. This sentence seems to be incomplete – please revise.

 

7. Line 198. As this is the first mention of CheckMe O2 device, the reader needs to be informed what this device is.

 

8. There is repeated reference to the developed App, with indication that the ‘User’ can use the App. The problem is, the ms provides no details as to where the App can be downloaded!

 

9. Figure 5. The caption refers to two example thresholds. However, there is no indication in the figure as to what or where the thresholds are (I presume the horizontal lines are the thresholds but this needs to be made clear?).

 

10. Lines 318-323. Related to the above, how are the thresholds decided/determined? Presumably these thresholds are different for each person/patient?

 

11. Line 316. “…..not expected to perform any other task……”. This seems somewhat misleading given that the test will involve multiple changes in direction.

 

12. Line 363. “……carefully counted….” – who does the counting? And how is the ‘carefully’ aspect determined?

 

13. Lines 400-429. Not clear if and how changes in direction are accounted for here.

 

14. Lines 450-451. This sentence seems redundant – i.e., it indicates something that isn’t being used! One could include lots of things that aren’t being used!!! I suggest the sentence is deleted.

 

15. Table 2. Can reporting these values to 4 decimal places be justified? You indicate that data is collected at 30Hz. This would indicate that the precision regarding time measures, is 1/30 of a second (i.e. 2 decimal places at best).

 

16. The conclusion makes reference to things that go way beyond the results of the study! Specifically, it makes reference to “accuracy of the system”, “enabling patients to administer tests at home”, and “providing support for both medical personnel and patients” – but none of these aspects where the focus of what was investigated in the study. Thus the conclusion needs to be rewritten to focus on just the findings of the study.

Comments on the Quality of English Language

The writing style of the ms is verbose and over-elaborate.  I really think shortening the text will make the ms stronger - as in it's current form reading, in particular, the introduction, will exhaust/overwhelm the reader and make it less likely they will continue reading the rest of the ms. Most other sections could also be shortened (but to a lesser extent than the introduction).

Author Response

Thank you again for your advice, it has definitely helped us further improve our manuscript.

The attached table addresses all comments and, as before, the edited text has been highlighted in blue.

We have also made some improvements and structural changes throughout the manuscript.

Thanks for all your help,

the authors.

Author Response File: Author Response.pdf