Enhancing Remote Industrial Training Experiences with Asymmetric Virtual Reality: Experiences, Tools and Guidelines

Round 1

Reviewer 1 Report

This paper presents a user study with experts from the area of industrial maintenance and installation of an elevator company. The study results, in which they utilize their “Research Panel” tool, confirm that the remote training experience can be enhanced with asymmetric VR training.

Overall, I liked the paper and I think it can go in the right direction, but I have some major concerns leading to major rework:

·         Section 1:

·         Please be more specific at some points (e.g., “have a lot to offer” line 32; what exactly is a lot? Please provide at least some examples)

·         Line 50-52: I’m not sure why COVID would shift the focus to asymmetric VR, please elaborate on this connection. Wouldn’t it shift more towards remote collaboration with VR, no matter if asymmetric or symmetric? We would also need a brief introduction to how asymmetric VR actually is understood in this paper – there does not exist a unified understanding of the term.

·         Line 62: What are these potentials you are building on? I don’t see a direct connection and motivation why you would research asymmetric VR in this article besides it “can” be useful for training. Please establish a good motivation for your work and most importantly: Please establish a problem you solve. I do not see the motivation for asymmetric, but more for remote and collaborative VR. As it is now, the paper would not need asymmetry at all.

·         Line 88: Why would we need “instant reactions”? This feature is also not motivated.

·         Line 90: Please avoid platitudes. “which to 90 the authors’ knowledge,” à this is should go without saying in a research paper.

·         Please provide a specific list (1…., 2…., 3….) at the end of the introductions stating what the contributions of this paper are. Also, provide a brief overview of how the paper is structured.

·         Section 2:

·         Headlines are normally capitalized. Please check on this.

·         Line 114: The paper would need some in-depth research on early asymmetric VR systems. I think there are more suitable sources that can be used for citing asymmetric VR as the author's own work from 2022. In other words: [14] is not the first work trying to define asymmetry with VR systems. Please find a more suitable one, as this field goes way beyond the COVID pandemic. reference to the 2012 Steed paper http://reality.cs.ucl.ac.uk/projects/beaming/steed12beaming.html would have been helpful and the authors might have better understood how to use the proper terminology in the more than 50-year-old field of VR.   

·         Line 148-150: Does this not count for all educational VR systems? What is specific about asymmetry here?

·         Overall: The related-work section could be longer and a bit more in-depth. Also, the authors argue again with the COVID 19 pandemic is a motivator for asymmetric VR – still I do not see why. Please provide arguments: What related work does support this claim and why?

·         Section 3:

·         Line 247: Scenario and set-ups are not interchangeable terms.

·         Line 282: Why does the existence of different user classes facilitate the asymmetry? And what exactly are the asymmetric factors in your system? Please provide some design rationale.

·         Fig 3: Who is the VR guide user role? Please use the proposed terminology in this figure.

·         Line 296ff: I think with the Likert-scale you did also collect quantitative data, not only qualitative.

·         Line 306: What is “Virhe. Viitteen lähdettä ei löytynyt.)” please provide a translation or state that this is a name of a tool. Why is it bold?

·         Line 370: Please check on when to write numbers and when to use words for quantity expressions (e.g.,  “a senior trainer…”)

·         Section 4:

·         Fig. 8: Were the participants qualified to answer this question? Did they already have technical documentation or real-life training in comparison to VR? Also: The question is highly debatable as it is a strong leading question…

·         Overall: I would have expected at least some statistical analyses of the study data in a paper that’s main contributions are study findings. Some data is there. Please provide some basic statistical insights and significance analyses.

·         Section 5:

·         The guidelines are good and do build on related work.

·         Regarding future work: I would suggest you provide some literature for the future work directions you pointed out. E.g., there exists some work on views and asymmetric VR in educational settings

·         Section 6:

·         Line 682I think rather than highlighting the advantages of asymmetric VR for remote collaboration, the paper really provides some form of design guidelines and aspects to consider when designing asymmetric/remote VR systems.

 

 

Overall: As said, I think the paper has potential and covers a relevant topic considering current research efforts. I would argue it needs some major rework which, however, can be accomplished without re-conducting another study from scratch. I would like to see some arguments concerning the design of your learning environment. Also, the expectations of the state contributions should be lowered a bit – some arguments are rather vague and do not support the claims based on them (also the research method has a very low internal validity when speaking about asymmetry in general, I would rather like to read some specifics about how the asymmetric aspects you provided in your system can enhance *collaborative/remote* VR, which is I think this paper really targets). The contribution “confirm that the remote training experience can be enhanced with asymmetric VR training” is a bit wide, so please provide some more in-depth discussion about what are your contributions, I think there are some! I wish the authors all the best and I look forward to the revised version.

Author Response

 

In addition to rebuttal, we would like to address some issues separately:

Thank you for a very detailed list of modifications. Some of the points you raised were minor misunderstandings that occurred due to writing mistakes and now we hope that we succeed to explain them properly.

• We review asymmetry as a technological set-up to increase the access and scalability of VR technology, especially in distributed settings, which caused asymmetric experiences for different user groups. We do not contribute in any way to the asymmetric VR aspects, but rather explored the usefulness of asymmetric VR to supplement remote training session
• The roles in the study did not facilitate the asymmetry, but helped to ensure that the remote training session, supplemented by asymmetric VR, would go smooth and without problems. The roles in text correspond to the role son the figure.
• The study was initially planned to be based on expert involvement and that dictated the focus on qualitative data collection and analysis, leaving out any traditional significance analysis due to small sample size.

We hope we managed to produce a better version of the article based on your comments.

Author Response File: Author Response.pdf

Reviewer 2 Report

The research is interesting. It addresses the concept of "asymmetric VR". This approach is not usually considered in virtual reality research. However, it is relevant as implementations of this style reduce the limitations associated with VR equipment and user cubing, among other variables that influence the use of the technology. The authors are thanked for contributing to the state of knowledge on these issues.

The document is well written. The different sections provide correct and complete information. This reviewer would like to highlight the quality of the discussion section. It addresses interesting aspects of these topics. The authors have made an important effort to deliver an interesting reflection.

It is recommended:

- While the article's focus is to show usage, it is important for readers to understand the development process. It is recommended to include some simple workflow that helps to understand the process in a better way, complementing what is written. 

- Add in the methodology the characteristics of the expert panel. Indicating the characteristics of each member is important to demonstrate that the validation by this panel of a few people is correct.

- I recommend improving the conclusions section. Add the contributions to the state of knowledge so that they are explicitly shown in this section.

- Revise the references and their format. There are several references written incorrectly.

Author Response

Thank you for your review.

We have addressed your reviews, but one:

• we are not sure what you mean by "the characteristics of the expert panel". We have described the reactions of "Research panel" and showed how it looked. 
• If you mean the expert participants - we have provided additional information on their expertise and experiences in the field

Author Response File: Author Response.pdf

Reviewer 3 Report

This manuscript deals with the application of asymmetric VR in industrial training, a timely and important topic. It is based on an interesting application. The following issues need to addressed for this study to be considered for publication:

L64-67: The rationale behind using asymmetrical VR should be accompanied by a comparison of spherical or 360-degree VR video, e.g. [1]. What are differences or similarities with asymmetrical VR?

L84, L146: Authors mention multiple places the crucial term of “knowledge transfer”. As this concept is not explained, I suggest adding a short section on it exploring its connections with similar learning theories and VR e.g. [2–4].

L110: Following up on the previous comment, what is the difference between asymmetric VR and desktop-based virtual reality or virtual worlds e.g. [5]?

L295: The RQ is aimed at knowledge transfer. Please explain how the selected method addressed the RQ.

L361: What was the outline of the training? How were learners engaged? Were there specific learning activities planned?

L458: Many questions in Figure 7 are aimed at learners, but participants were in fact experts or advanced in the taught subject. As acknowledged in L664, would it not be more appropriate to focus on the suitability or experts’ acceptance of the specific training method?

L499: Figure 8 is confusing. Is the question not presumptuous? Was there no option to judge VR worse than other media or modes? Were participants exposed to other media? When and how?

L514: What is the rationale behind the selection of the particular learners? Their number is rather low. Is this not a serious limitation?

 

References

1.          Snelson, C.; Hsu, Y.-C. Educational 360-Degree Videos in Virtual Reality: A Scoping Review of the Emerging Research. TechTrends 2019, doi:10.1007/s11528-019-00474-3.

2.          Mystakidis, S. Deep Meaningful Learning. Encyclopedia 2021, 1, 988–997, doi:10.3390/encyclopedia1030075.

3.          Burke, L.A.; Hutchins, H.M. Training Transfer: An Integrative Literature Review. Human Resource Development Review 2007, 6, 263–296, doi:10.1177/1534484307303035.

4.          Garrett, M.; McMahon, M. Indirect Measures of Learning Transfer between Real and Virtual Environments. Australasian Journal of Educational Technology 2013, 29, 806–822, doi:10.14742/ajet.445.

5.          Mystakidis, S.; Berki, E.; Valtanen, J.-P. Deep and Meaningful E-Learning with Social Virtual Reality Environments in Higher Education: A Systematic Literature Review. Applied Sciences 2021, 11, 2412, doi:10.3390/app11052412.

 

 

Author Response

Thank you for your review, we have modified the article base don your comments and added four out of five suggested references.

We excluded the comparison of asymmetric VR to 360 degree educational videos, because for our context it is irrelevant - as mentioned in line 92-93; the production of any types of videos in industrial context, such as elevator shaft, is impossible due to safety concerns. 

Moreover, we review asymmetry as a technical set-up which expand the scalability and access to VR-software, which has certain benefits in the industrial context. Therefore, we have not discussed difference between "asymmetric VR and desktop-based virtual reality", since the focus of our research was to investigate how to easily and cost-efficiently provide access to VR-content; however, in discussion we  have discussed why guided asymmetric session is a better training method than if all the participants would access the VE via desktop interface.

Finally, we do understand that the number of participants is a serious limitation of the study, however, it was simply impossible to gather more participants with a similar background and expertise. However, we do not see value by expanding the participant list with people nor related to industrial context, since they posses no relevant knowledge for our case study. Therefore, we based the study on qualitative, not quantitative findings, which, considering the expertise, is highly valid for the industrial partners and will have it's influence on the company processes, while eliciting curious data for the research community as well. 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thank you for thoroughly incorporating all the reviewers' feedback. I think the paper is much better now. I think it is nearly ready for being published.

The single point from my perspective that must be included is (as mentioned last time) some kind of statistical significance analysis. The main contribution of the paper is its study. The study also did capture quantitative aspects. Nevertheless the power of the significance with only N=9, this number of totally for performing significance tests that underline the statement of the paper.

Author Response

Thank you for the review.

As requested, we have added significance analysis to the paper - we did non-parametric one-sample Wilcoxon Signed Rank Test with Bonferroni correction, which as expected showed no significance. 

--> section 3.3.1 was updated (lines 413-419)

--> section 4.2 and Figure 7 were updated (lines 577-589)

 

We hope these analysis met your requirements and now the paper is in a good shape.

Reviewer 3 Report

Authors addressed most identified issues in a satisfactory manner and provided sufficient explanations whenever they chose not to do so. The readability of the manuscript has increased significantly and can be accepted for publication.

Author Response

Thank you for the verification of the major revision. We are very happy to hear that paper is currently in a better shape.