Capturing Mental Workload Through Physiological Sensors in Human–Robot Collaboration: A Systematic Literature Review

Response to Reviewer 1 Comments

1. Summary

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.

2. Questions for General Evaluation

Reviewer’s Evaluation

Response and Revisions

Does the introduction provide sufficient background and include all relevant references?

Yes/Can be improved/Must be improved/Not applicable

We have added more content to the introduction section in order to make it more complete.

Are all the cited references relevant to the research?

Yes/Can be improved/Must be improved/Not applicable

Is the research design appropriate?

Yes/Can be improved/Must be improved/Not applicable

Are the methods adequately described?

Yes/Can be improved/Must be improved/Not applicable

Are the results clearly presented?

Yes/Can be improved/Must be improved/Not applicable

Are the conclusions supported by the results?

Yes/Can be improved/Must be improved/Not applicable

3. Point-by-point response to Comments and Suggestions for Authors

Comments 1: The study reviewed papers about capturing mental workload in human-robot collaboration. I believe the manuscript needs to be clarified in many places.

Response 1: Thank you for pointing this out.

Comments 2: In line 58, the sentence is duplicated.

Response 2: We have deleted the duplicated sentence.

Comments 3: EEG offers a wider variety of extractable features compared to other measures like EDA, but the current descriptions in the manuscript are quite vague.

Response 3: We have expanded the sections regarding each of the metrics.

Comments 4: Also, some EEG features are described as ERSPs, while others are described as being in the 'time-frequency domain.' This distinction is unclear, as ERSPs are inherently defined within the time-frequency domain.

Response 4: While ERSPs are indeed defined within the time-frequency domain, none of the reviewed papers included in our analysis conducted analysis with these features. Thus, we did not include them.

Comments 5: I believe the authors should elaborate on the specific industries and tasks discussed. What are the common movement patterns or key points in each task? Revealing these details would be very informative.

Response 5: We agree. However, in order to not boggle the document, we have included a table containing this information as supplementary material.

Comments 6: I think "Combined assessment" part is too superficial. The authors mention the benefits of combining different metrics but do not elaborate on the unique contributions of each measure or how their integration improves workload assessment. Also, provide technical details as well with sufficient explanations of application contexts.

Response 6: We have moved this section to be within the discussion section, and have expanded it as well.

Comments 7: In the discussion, the authors overlooked that EEG can also be used to measure subjective aspects in human-robot interactions, such as responses to robot movement. EEG not only reflects cognitive workload but also captures nuances of subjective evaluations, including satisfaction influenced by robot performance [1]. Integrating EEG with subjective measures could provide a more holistic understanding of user experiences during robot interaction. Given that this study is a systematic review, the authors should include this broader perspective.

Response 7: Thank you for your comment. We have added a discussion regarding this topic to the discussion section.

Response to Reviewer 2 Comments

1. Summary

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.

2. Questions for General Evaluation

Reviewer’s Evaluation

Response and Revisions

Does the introduction provide sufficient background and include all relevant references?

Yes/Can be improved/Must be improved/Not applicable

Are all the cited references relevant to the research?

Yes/Can be improved/Must be improved/Not applicable

Is the research design appropriate?

Yes/Can be improved/Must be improved/Not applicable

Are the methods adequately described?

Yes/Can be improved/Must be improved/Not applicable

Are the results clearly presented?

Yes/Can be improved/Must be improved/Not applicable

Are the conclusions supported by the results?

Yes/Can be improved/Must be improved/Not applicable

3. Point-by-point response to Comments and Suggestions for Authors

Comments 1: The paper presents an interesting review of the literature on the use of physiological signals to capture mental workload in the HRC context, with a rather robust methodology used. The exposition of the content is clear and the paper is well written. I would have some suggestions below to improve the paper.

Response 1: Thank you.

Comments 2: In line 101, page 3, there is an extra round bracket before “Liu, ...”.

Response 2: We have corrected this.

Comments 3: The literature search methodology is well structured, however, I think it is missing some keywords that are often used in cognitive ergonomics (Subject 2), such as “mental effort”, “mental stress” or even just "stress". For Subject 3 I would add “physiol*” to the search. I would suggest supplementing the literature review with possible new findings emerging from the broader research, such as Gervasi, R., Aliev, K., Mastrogiacomo, L. et al. User Experience and Physiological Response in Human-Robot Collaboration: A Preliminary Investigation. J Intell Robot Syst 106, 36 (2022). https://doi.org/10.1007/s10846-022-01744-8

Response 3: We have carried out another literature search, according to your suggestions. However, we did not employ the keywords “mental stress” and "stress" because these are part of our exclusion criteria. Specifically, in our exclusion criteria, we excluded papers that only referred to physical constraints, stress conditions, or fatigue, without considering mental workload in their evaluation. We have added the results from this new search to the manuscript. Additionally, our search did not return Gervasi et al. (2022), and so it was not included in our analysis. Nonetheless, we have considered it for our discussion section.

Comments 4: The particularities of the HRC context versus a traditional one should be better highlighted to understand if there are any actual differences, in constraints or needs, that may influence methodological choices for mental workload assessment.

Response 4: We have added more content to the introduction section in order to make it more complete.

Response to Reviewer 3 Comments

1. Summary

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.

2. Questions for General Evaluation

Reviewer’s Evaluation

Response and Revisions

Does the introduction provide sufficient background and include all relevant references?

Yes/Can be improved/Must be improved/Not applicable

Are all the cited references relevant to the research?

Yes/Can be improved/Must be improved/Not applicable

Is the research design appropriate?

Yes/Can be improved/Must be improved/Not applicable

Are the methods adequately described?

Yes/Can be improved/Must be improved/Not applicable

Are the results clearly presented?

Yes/Can be improved/Must be improved/Not applicable

Are the conclusions supported by the results?

Yes/Can be improved/Must be improved/Not applicable

3. Point-by-point response to Comments and Suggestions for Authors

Comments 1: The authors provide a systematic literature review about capturing mental workload through physiological sensors, with an emphasis on HCI. I find the review well-performed.

Response 1: Thank you.

Comments 2: The introduction gives a good overview of the field. The authors also refer to three other reviews that seem to be related but seem to have a different perspective. I recommend that the authors give a short summary of the findings of these three survey articles, also related to the research questions.

Response 2: We have added more content to the introduction section in order to make it more complete.

Comments 3: Note that the research questions are not stated. I recommend adding the research questions that the survey shall answer to towards the end of Section 1; ca. Line 113 in the submitted version of the manuscript.

Response 3: We have included the research questions at the end of the introduction section.

Comments 4: In Section 2, the literature search strategy contains terms that are not explained. Please tell the reader why these keywords are relevant.

Response 4: We have modified our keywords on subject 3, to contain not only acronyms, but also their respective designation, and carried out another literature search accordingly. Additionally, we have added a list of acronyms, and respective designations at the end of the manuscript.

Comments 5: Lines 141ff: The keywords have not been introduced yet, and some of these can only be guessed. However, some of the keywords could be ambiguous, such as “resp” that could trigger to the words “respective(ly)”, “response”, etc., which probably is not intended.

Response 5: We have modified our keywords on subject 3, to contain not only acronyms, but also their respective designation. Additionally, we have added a list of acronyms, and respective designations at the end of the manuscript. We have also replaced the “resp” keyword with “respiratory” to not trigger unintended keywords, and have carried out another literature search accordingly.

Comments 6: Further, a list of abbreviations and acronyms should be added at the end of the manuscript (see manuscript template). Please refer to this list and introduce some of the terms, including reasons why these are relevant.

Response 6: We have added a list of acronyms, and respective designations at the end of the manuscript.

Comments 7: I observe that some physiological metrics would not be found using your search, such as interpreting facial expressions visually. See the survey article by Landmann (2023) https://doi.org/10.1016/j.techfore.2023.122889. However, it might be debatable how well interpreting facial expressions would work in an HRC context. At least, a discussion about this subject could form a reasoning for the chosen keywords.

Response 7: Thank you for your comment. We have added a discussion regarding this topic in the discussion section.

Comments 8: Related to abbreviations and acronyms, at the end of Table 1, the notes should be replaced with a reference to the table of acronyms. All acronyms should appear in the list, also if you choose to keep the note at the end of Table 1.

Response 8: We have added a list of acronyms, and respective designations at the end of the manuscript.

Comments 9: I wonder why you excluded conference papers (some of these conference papers also appear in periodicals). Some relevant practical research might be disseminated in conference and workshop proceedings.

Response 9: Based on your suggestion, we have conducted another literature search, now including conference papers and book chapters.

Comments 10: Lines 186ff: some of this content is already presented in different form in the methods-Section

Response 10: Thank you. We have checked and reformulated this point.

Comments 11: The fonts in most figures are too tiny, have bad contrast, and are not readable. See Figures 2, 3, 4, and 8. In its current version, Figure 3 is incomprehensible, as it is impossible to decipher the labels.

Response 11: Over the course of these revisions, we have removed the previous figures, so as to improve the document´s readability. The remaining figure´s text has been increased as well.

Comments 12: The content of Figure 2 could be presented in tables. The journal overview could be reduced in a comma-separated list of journals with one article and one journal with two. Also, Fig 2b. would be better as a table with year and number of publications. As a detail, the lines between the years do not make much sense, and a bar-graph would be better suited. (The y-value for the x-value of, e.g., 2023.4 does not result in a meaningful value.)

Response 12: We have chosen to remove these figures and instead present their information in text format.

Comments 13: I have some doubts about the usefulness of Figure 8. The percentage values are debatable, and I suppose that the number of articles would be more informative. Further, on the right side, five values seem to be equal, but with different shape. I find this difficult to interpret. Possibly, a table would be more informative.

Response 13: We have chosen to remove this figure, and instead discuss its content in the discussion section.

Comments 14: The different physiological measures are explained from Line 323 and onward. Please consider restructuring the manuscript and have a separate section treating the physiological measures (and numbering these measures with subsection numbers).

Response 14: We have reformulated these section headings.

Comments 15: The physiological measures are those found in the surveyed articles. A discussion about the completeness of these measures is lacking (see also my comment above about facial expressions). The measures include EEG, fNIRS, cardiac measures, ocular, EDA, and combined.

Response 15: We have included a discussion regarding other measures besides those found in the reviewed studies to the discussion section.

Comments 16: Line 485: do you have proof for this trend?

Response 16: We have reformulated the section regarding EEG results, and expanded upon the information provided regarding the features and ratios uses to analyze the signal.

Comments 17: The discussion section appears as a kind of in-depth information of the physiological measures. I recommend restructuring this. The first part of Section 4 is not so much a discussion, but more an in-depth information about the measures.

Response 17: We have replaced the first part of section 4 (discussion). We have also expanded upon the information provided throughout the remainder of this section.

Comments 18: The conclusion section is a bit vague, and the findings from the review “drown” in generic information. Please elaborate the findings and express the findings precisely. Please tell the reader which measures would be useful for which application area, resp. which measures would not work. For instance, some environmental conditions could be unfavorable for HRC applications.

Response 18: We have reformulated the conclusion section, so as to make it more useful to readers.

Comments 19: Line 600: I find the term "meticulous delineation" unclear in this context; "systematic description" would fit better.

Response 19: Thank you for pointing this out.

Comments 5: I understand that the authors prefer not to include the supplementary table in the main text. So, you can leave them just there. However, I suggest that they at least briefly explain the common movement patterns or key points in tasks related to physiological responses.

Response 5: We added information regarding common movement patterns or key points in tasks.

Comments 7: The revised discussion is unsatisfactory. The revision did not fully address the comment. There is no clear discussion on how EEG can measure subjective aspects, such as responses to robot movement or satisfaction influenced by robot performance. You just touched on subjective measures, not linking them to EEG. In the previous round, I provided an example to avoid confusion, but it seems the authors did not engage with the suggestion. I highly recommend referring to the example to expand on your discussion with appropriate examples.

Response 7: Thank you for your comment. We have added a discussion regarding this topic to the discussion section, using your suggestion (https://doi.org/10.3390/s23010277)

We added Table 1 of the initial manuscript. Moreover, some additional information was added throughout the manuscript to improve the flow.