Distributed Asymmetric Virtual Reality in Industrial Context: Enhancing the Collaboration of Geographically Dispersed Teams in the Pipeline of Maintenance Method Development and Technical Documentation Creation

Round 1

Reviewer 1 Report

The paper suggests the adoption of asymmetric VR in the industrial contexts can help enhance the collective maintainance among teams that are dispersed geographically. It is interesting to see they choose Teams for the the control group in comparison with VR. However, it is not clear what qualitative data analytic methods they used to analyse the content (interview). Besides, only descriptive statistics was used in interpreting quantitative data. I would suggest them to use ANOVA to examine the variations between two groups.

Author Response

Thank you for your review.

There should be some level of misunderstanding, since we have not used Teams-users as a control group, but rather made a qualitative exploration of how asymmetry between VR platform and Teams would work out. The data is shown for two user groups: VR-users and Teams-users since their experiences were different. Based on your comments, and to avoid possible misunderstandings, we are going to clarify this point in section 3.3.2 User Groups and Roles.

Further, to analyze interview data we have used the thematic method of qualitative data analysis – we are going to add this information to section 3.3.4 Collected Data and Analysis.

As for your last comment, we indeed used only descriptive statistics to analyze our quantitative data, collected via surveys for the following reasons – 1) the focus of this article is on qualitative exploration, rather than quantitative; 2) the user groups of 10 participants is not sufficient sample size to get valid results from statistical analysis. We have not been comparing user groups or looking for statistically verified variation in their experiences; survey results were used as supportive data – to measure expectations and experiences of two user groups and to collect their subjective perception of the asymmetric VR set-up - we are going to clarify this point in section 3.3.4 Collected Data and Analysis.

Reviewer 2 Report

The paper provides a lot of information on practical use of assymetric VR. I think it is a very good paper and nearly ready for publication. There is just one issue that I found missing in this paper. It is well known that the use of VR can cause a VR sickness in some persons. In my country any research involving VR head sets requires a consent from a bioethical commission ,  but I see no mention of the issue in your paper.  How would implementation of VR impact people  with VR sickness? Any alternative for them?

A minor issue, but Figure 5 shows unblurred faces of people, has their informed consent included publicatiin of their faces in a paper? Or maybe these are the faces of the authors.... 

Author Response

Thank you for your review.

Luckily in our country, we are allowed to carry out such a research activity without the consent of the Ethics Committee.  Regarding the risk of getting sickness from wearing the headset, we included this point in our consent form and training procedure and instructed the participants to inform the moderator immediately about such effects and take off their headset. We have clarified this point in 3.3.3 Remote user study set-up (lines 459-461)

In fact, no participants had such an effect during the user study. The investigation of an asymmetric VR approach is exactly one way to overcome such risks when adopting VR to industries since people who experience motion sickness would be able to join the collaborative sessions over Teams.

Thank you for the note to blur the faces on figure 5 – we have done so.

 

 

Reviewer 3 Report

The paper presents a combination of a VR application and a video conferencing system as a support tool for maintenance operations. I would suggest the following changes:

• lines 109-121 should be deleted because the information is redundant and does not provide relevant information,
• in line 313 I think it is figure 4,
• please clarify if there is an exchange of information between the application presented in the paper and Microsoft Teams or the video conferencing system is used only to share screen, it would be interesting if the text in the MSTeams chat could be automatically loaded with The TextBox tool,
• because you have chosen to use the term ANONYMIZED and have not presented in sufficient detail the product on which you perform maintenance operations please explain how complex the battery replacement task is, because for any industrial product this should be done very simply and easily, why you chose so simple case study
• detailed how the two groups of experts contributed specific information in the experiment given the complexity of the task chosen as a case study
• figures 7,8 and 9 are extremely difficult for readers to understand for example in figure 7 the differences described in line 507 are not visible on the graph, it would be good to find another method of presentation
• it is not very clear, at least for me, how to achieve asymmetric collaboration in VR if there is no direct data exchange between MSTeams and the VR application, please be a little more explicit on this subject, how the sound reaches the HTC headset if for example also uses wireless module
• detailed if the statements on lines 674-676 are true and for a more complex case study in which for example you try to adjust a numerically controlled machine tool

Author Response

Thank you for your review.

We will kindly follow the provided minor edits, such as deleting lines 109-121 and fixing the figure references.

Regarding the information exchange between the VR platform and Teams, we agree that it is a great idea and indeed would be interesting to investigate further. However, in the current case study, we have not developed any data exchange as you suggested. On the opposite, the goal was to investigate a simplistic and practical approach to creating an asymmetry between the VR platform and Teams, and our results demonstrated that such setup would be sufficient to facilitate the collaboration of distributed multinational departments.

Next, following your suggestions, we added a brief explanation of why the battery replacement task was selected as a case study task to section 3.3.2 User Groups and Roles (lines 407-413) and provided more details to the set-up description to the section 3.3.3 Remote user study set-up  (lines 430-449)

Finally, regarding figures 7,8,9 – they demonstrate the basic descriptive statistical analysis visualization – showing the min, max and median of the answers, which should be sufficient to demonstrate the main trends, while the textual description (e.g., line 507) expands on the data rather duplicating the information presented on the figures. To avoid any difficulties with reading the figures, we have added an explanation of how to read these (lines 511-514).

Regarding the last comment about lines 674-676, we do found an indication (based on qualitative data from expert participants) that asymmetry may decrease the number of iterations and lean waste due to enhanced and more clear communication. Our work is an initial step and we agree that more research with more complex industrial processes should be explored further.

Reviewer 4 Report

The paper describes a tool that suggests asymmetry between VR platform and Microsoft Teams, enabling distributed collaboration within 20 experts to perform maintenance operation , while guidelines on how to enhance the collaboration efficiency for low-cost distributed asymmetric VR in terms of organizational, collaboration and technology is given.

The paper is well written and organized. However some weak points must be better addressed in order to increase the quality of the paper:

• As you extensively described the use of VR to introduce the Asymmetry-VR, you have not mentioned the Augmented reality technology, which is an immerse layer that has contributed in the development of VR. Especially, in the maintenance environment, advancements have occurred that widely accepted from the industrial personnel. For example, in the  “doi:10.3390/machines8040088”, researchers developed an AR-based tool to help operators and experts to improve the efficiency of maintenance operations such that the time of maintenance task is reduced.  Could you please highlight the differences with your approach and describe how your method will be applied in the AR applications? 
• Apart from that, it is also good to refer how your approach will be combined with BIM models to enable collaboration among designers from different industries. In the https://doi.org/10.1016/j.promfg.2020.04.059,, authors described how the BIM, Augmented and Virtual Reality empowering Lean Construction Management, but it can also be combined with industry-related operation. 

Final remark: the paper assess a VR-assymetric platform and expands the literature but it is worth noting that the results show the impact of those platforms in the collaborative environment and experimental papers showing a real case study like this one are welcomed.

Author Response

Thank you for your review.

When writing this article, AR-related topics and articles (demonstrating AR-supported maintenance operations) were not included on purpose, since the context of this case study is the creation of maintenance methods and related technical documentation, rather than providing in-field guidance for maintenance technicians.

However, we will kindly follow the provided suggestions and expand our related work section with suggested papers and similar AR technology to make this work more strong. Therefore, we added suggested references – on the topic of AR to the 1. Introduction (line 65-68) and on the topic of innovative technologies and lean methodology to 2. Related Work (lines 148-150).

Regarding your question “Could you please highlight the differences with your approach and describe how your method will be applied in the AR applications?”, the answer is that context, tasks and industrial goals are different for these works – in our article, we review the phase, where maintenance instructions and technical documentation is being created, not used. And despite we acknowledge the use of AR as a second technology to facilitate asymmetry in our related work section, with this paper we explored how to expand the use of VR environments in a simple and cost-effective way.

Reviewer 5 Report

Dear Authors,

the paper is interesting and fluid reading, but sometimes too verbose.

Some comments about the paper:

1. Please explain acronyms when introduced for the first time (e.g., line 37, HCI and CSCW, or line 446, PQ).
2. Please, check the numbering of the figures; check also all references to figures (all wrong!).
3. As far as the experiment is concerned, the number of experts (20) seems too low.
4. Still, getting into the experiment, did you consider more repetitions of the tasks?
5. Have you considered administering the questionnaires in dysfunctional form, as in the Kano method?
6. In order to make the procedures more effective, have you thought about some integration with PLM?
7. There is no measure of the effectiveness of the tasks compared to current procedures, nor is there a quantification in terms of the time of the tasks performed: it would have been interesting to show a steady use of the methodology described.

Author Response

Thank you for your review.

We will follow all suggested edits to acronyms and figure references.

We acknowledge that 20 participants may sound not sufficient, and it is definitely not sufficient to run any statistical analysis and find statistically verified validation. Nevertheless, with this study, we focused on quality, not quantity and grounded this case on the knowledge of subject matter experts, which are (usually) not easy to get for such testing purposes. We highlight that this case is a qualitative investigation in the abstract, and we believe 20 experts from different countries is a good sample to get professional insight on the subject.

Further answering your questions, it is impossible to measure the effectiveness of the tasks compared to current procedures in any quantitative form. The creation of maintenance methods and technical documentation cannot be measured in a similar way as, for example, maintenance operations in the field (e.g., time and number of errors). It is a collaborative somewhat free form activity, which may be different in nature with different equipment, while the collaboration itself may be complicated because of human errors. What we found is that VR and access to VR may solve communication issues and create a shared working space for globally scattered experts, which is a better alternative than the current practices – communication over email or Teams chats. We have not considered the repetitions of tasks in this iteration – it was explorative in nature to investigate the adoption of simplistic and practical asymmetric VR set-up that would have more chances to be adopted in the industry. The integration with PLM, although sounds like a great idea, was not considered in this case study for the same reasons. As for Kano method, in the current iteration, this method does not correspond to our research question. However, we performed similar testing in the earliest phases of COVE-VR development – in case you wish to check it out, it is reference # 31, where we detail the whole development process.

Round 2

Reviewer 1 Report

I mainly questioned on the methodology in the previous round and I can see the authors added more details to clarify. I am satisfied with the modification added. 

Author Response

Thank you for the 2nd round of reviews!

We are grateful for your suggestions and happy that you found the paper in the current iteration to be satisfactory! 

Reviewer 3 Report

I think you should make a comparison between your solution and shared virtual reality / distributed virtual reality. I think that the paper is more complete at this moment and that it can be published after integrating all the observations made by the reviewers.

Author Response

Thank you for the 2nd round of reviews! We are grateful for your input and happy that you found the paper to be ready for publishing.

We agree with your point, that it would be appropriate to make a comparison between asymmetric VR  and shared virtual reality / distributed virtual reality. We have no resources to accomplish it in this article, but we added this point to future work section (lines 803-805) and we are going to persuade this point in our future projects.