Virtual Reality for Safe Testing and Development in Collaborative Robotics: Challenges and Perspectives

Round 1

Reviewer 1 Report

The presented work approaches a novel and interesting topic, human-robot collaboration. The paper offers provides an interesting read especially due to the psychological implications within HRC. 

The literature review is good, with appropriate references for each section of the paper. However, specific examples and use-case will significantly improve this work's scientific value.

Figure 1 is blur and the lower image seems to be out of context. Authors should provide further details for the bottom image.

Added value can also be provided by including a table that compares the key characteristics of prior work to highlight their differences and limitations. The authors may also consider adding a line in the table to describe the proposed solution(s).

There are a few typos especially in the abstract where words are divided unnecessary. 

Author Response

See attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript entitled “Virtual Reality for safe testing and development in collaborative robots: review and preliminary framework proposal” has been investigated in detail. After reading the paper, the reviewer has the following concerns:

• The overall quality of the paper is in general, and the implementation aspect is not clear.
• The paper suffers from language problems. The paper should be proofread by a native speaker or a proofreading agent.
• The significance of the design carried out in this paper is not well explained relative to other important works published in this field. The authors should review, comment, and compare more works that are developed recently.
• What makes the proposed method suitable for this unique task? What new development to the proposed method have the authors added (compared to the existing approaches)? These points are not clear.
• What is real novel on this research? This work is not novel how the authors claim. In general, the authors are overselling their work.
• The main contributions of the study should be clearly explained in both theoretical and practical aspects.
• This manuscript may lack the enough research results, error analysis and explanations for the publication.
• The performance of the proposed method should be better analyzed, commented and visualized in the experimental section.

In my opinion, the study has both mathematical expression and experimental deficiencies. The contribution of the study has not been clearly expressed. The novelty of the study is not emphasized in the manuscript. In summary, due to the lack of key technical discussions, comparative studies, and computational complexity analysis, I cannot recommend this manuscript to be further considered in this journal.

Author Response

see attached

Author Response File: Author Response.pdf

Reviewer 3 Report

Title:

• the authors mention review in the title. However, there is no information devoted to explain what was the methodology, how many publications were considered, what was the time interval, and so on. Therefore, I suggest removing the term review from the title.

Abstract:

• line 15 - Use upper case for extended reality
• line 16 - Use upper case for virtual reality
• line 20 - Use upper case for human-robot collaboration. Plus, introduce the acronym
• line 23 - Use upper case in the first letter for augmented reality
• I would suggest adding the phrase: "We also provide a first attempt at what could become the basis for a human 18 robot collaboration testing framework, specifically for the designing and testing factors affecting " at the end and not in the middle of the abstract.
• Probably include XR in the keywords! 

Introduction:

• The following could benefit from a reference: "The contagion risk posed by human contact had a severe socio-economic impact 37 imposing changes for industry across the board, from factories to hospitals and care 38 homes . ". There is also an extra space at the end of this sentence.
• line 50 - Use upper case for virtual reality
• line 51 - Use upper case for extended reality
• A nice add on would be to differentiate what is VR, AR, and XR. Some readers may not be familiar with such terms.
• Furthermore, such technologies deserve a better context in the introduction, i.e., what is their goal for HRC, and what are the existing opportunities? I suggest something in the line of what is described at the beginning of section 3.

HUMAN-ROBOT COLLABORATION, SAFETY AND ACCEPTABILITY:

• line 53 - Use upper case for human-robot collaboration  
• line 54 - Use upper case for human-robot interaction
• Figure 1 should appear first than table 1. It should be in page 2 and not page 4

• Another, more recente reference for: "When two agents are working together, they need to establish joint attention in order to form joint intention and execute joint actions" can be:

  • Marques, B., Silva, S. S., Alves, J., Araujo, T., Dias, P. M., & Sousa Santos, B. (2021). A Conceptual Model and Taxonomy for Collaborative Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 1–21.

• line 73 - Use upper case for human-machine interaction
• line 110 - Avoid VR some close to each other: "Using VR allows testing human interactions with different VR"
• line 114 - Avoid VR some close to each other: "The use of such digital twins allows 114 the use"
• line 114 - Use upper case for robot operating system
• Figure 1 - Consider having images side by side instead

THE USE OF VIRTUAL REALITY FOR TESTING HUMAN-ROBOT COLLABORATION:

• Line 137 - Use the acronym HRC

A FRAMEWORK FOR EXTENDED REALITY IN TESTING HUMAN ROBOT COLLABORATION:

• Line 152 - Introduce the acronym HCI
• Line 160 - Use the acronym HRC
• Line 173 - which literature? how was this collected? methodology? what was the time interval? how many works? Add more information on this subject
• Table 2 - remove the red underline of Pupilometry

VR IN TESTING COGNITIVE AND SOCIAL ASPECTS OF COLLABORATION:

• Line 207 - Use the acronym HRC
• Line 219 - Use the acronym HRI
• Line 229 - Correct the following: "acceptability, etc. users in the system. The use"
• Line 229 - Correct the following: "c.f. 230 Weistroffer et al"
• Line 242 - Use upper case for "sensorimotor contingencies"
• Line 249 - Use the acronym VR

TELEPRESENCE AND TELEOPERATION SCENARIOS

• Line 325 - Use the acronym VR
• Line 329 - Use the acronym HRI

THE USE OF AUGMENTED REALITY

• Line 343 - Use the acronym VR
• Line 343 - Use upper case for "augmented reality"

CONSIDERING OPERATOR GENDER AND AGE IN COBOT TESTING

• Line 367 - Use the acronym VR

CONCLUSIONS

• Line 417 -  Use the acronym AR
• Add ideas for future work - Also, change the title to conclusions and future work

References:

• The following should not be references but footnotes, please update them accordingly - 22, 28, 37, 68.

Author Response

Please see attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The revised manuscript entitled “Virtual Reality for safe testing and development in collaborative robots: review and preliminary framework proposal” has been investigated in detail. The points I have mentioned have not been adequately addressed.

Author Response

We answered all the Reviewer's comments in the document added to the revision. Please find them repeated below, under each respective point the Reviewer has raised by now.

• The revised manuscript entitled “Virtual Reality for safe testing and development in collaborative robots: review and preliminary framework proposal” has been investigated in detail. The points I have mentioned have not been adequately addressed.

Please note, that most points raised are not relevant to our manuscript. For example, we did not have an experimental part as this was a review paper. The rest of the points have likewise been addressed in detail, please find them below.

The manuscript entitled “Virtual Reality for safe testing and development in collaborative robots: review and preliminary framework proposal” has been investigated in detail. After reading the paper, the reviewer has the following concerns:

• The overall quality of the paper is in general, and the implementation aspect is not clear.

We reviewed the respective parts of the manuscript.

• The paper suffers from language problems. The paper should be proofread by a native speaker or a proofreading agent.

The revision was reviewed by a native British English speaker.

• The significance of the design carried out in this paper is not well explained relative to other important works published in this field. The authors should review, comment, and compare more works that are developed recently.
  
  We review key works on HRC from a time span of almost 20 years, with most recent ones published in 2022. In the revision we added more recent studies on HRC.
  
  
• What makes the proposed method suitable for this unique task? What new development to the proposed method have the authors added (compared to the existing approaches)? These points are not clear.

We write on combining subjective reports with physiological data, highlighting methodological concerns to keep in mind. We provide a categorization of variables involved in VR testing of cobot simulations for researchers to keep as a reference in designing their studies.

• What is real novel on this research? This work is not novel how the authors claim. In general, the authors are overselling their work.

Please note in the article we discuss the use of XR for testing cobots, not our own work. While the field is developing, it is still rather not widespread, especially in the case of using immersive VR and cobot digital twins, as highlighted by some of the literature we cite. We put together the existing approaches and highlight some methodological aspects that need consideration. In the revision we expanded on methodological aspects of fusion between subjective and physiological data and highlighted its main trouble insofar, namely low sample sizes.

• The main contributions of the study should be clearly explained in both theoretical and practical aspects.

As above, we are describing the existing and potential contributions of studies made by others, putting together main findings and trying to describe the overview of variables involved.

• This manuscript may lack the enough research results, error analysis and explanations for the publication.
• The performance of the proposed method should be better analyzed, commented and visualized in the experimental section.

Our paper has no experimental section, but thank you for the comment.

In my opinion, the study has both mathematical expression and experimental deficiencies. The contribution of the study has not been clearly expressed. The novelty of the study is not emphasized in the manuscript. In summary, due to the lack of key technical discussions, comparative studies, and computational complexity analysis, I cannot recommend this manuscript to be further considered in this journal.

We are neither sure what “mathematical expression” deficiencies the reviewer means, nor can we address their concern about the need for “computational complexity analysis”. We tried to write this perspective/narrative review paper in a clear way, providing an overview on existing XR solutions in collaborative robotics to a broad readership, so no computational complexity was involved, nor was any experimental section. We are hopeful that our replies satisfy the reviewer. We are happy to address any further, more specific comments.

Round 3

Reviewer 2 Report

Accept