Evaluating Design Guidelines for Intuitive, Therefore Sustainable, Virtual Reality Authoring Tools

Round 1

Reviewer 1 Report

The work presented by the authors is quite interesting while reading, if you can provide the following information it will be additive to the work.

1. How many questions were provided and environment of assessment is either collected seperately or in a group?

2. was there any specific reason for high correlation analysis or is it due to the framing approach of questionaire.

3. Is there any specific reason of doing correlation analysis and what is the author want to reflect from the analysis. Is this to reduce DG or anyother reason, need to be defined.

4.Why is it the research participants are limited to 6?

5. The conclusion lacks the end result defining the evaluation result of DG and How these guidelines are useful in real time applicability. If the conlcusion is written to the context of title will be quite reflective in understanding the outcomes.

Author Response

Dear Reviewer, we are pleased to resubmit our manuscript revised according to your comments, which considerably contributed to the improvement of the work. Please find a detailed description of the changes made below:

Point 1 – Please provide information on “how many questions were provided and environment of assessment is either collected separately or in a group?”

Response 1 – Thank you for bringing this to our attention. In the revised version, we added the information about the Likert-scale questionnaire, comprising fifteen questions and answered individually in the format of an online forms (lines 245-247) and in the focus group interview the answers were collected in an in-person group meeting with the duration of two hours (lines 277-278). In addition, a document with Supplementary Materials was sent with the revised version of the article.

Point 2 – Please provide the following information: “was there any specific reason for high correlation analysis or is it due to the framing approach of questionnaire?”

Response 2 – We appreciate your recommendation. In the revised version we included more details about the use of the correlation analysis as an indicator to evaluate the validity of the design guidelines (lines 171-173). We also added to the materials and methods section more details about how this indicator was used (lines 262-268). It was expected that guidelines with strong positive correlation would receive similar scores, therefore lower difference values, while guidelines with strong negative correlation would receive very different scores, therefore higher difference values. However, these values were obtained to serve as a demonstration of how the guidelines could be used to evaluate a VR authoring tool and to be compared with the results obtained with the second method. It was assumed that, if the results obtained from testing the tool were compatible with expectations, this would be a good indication of the validity of the guidelines as a guide to develop and evaluate more intuitive VR authoring tools.

Point 3 – Please provide information on “any specific reason of doing correlation analysis and what is the author want to reflect from the analysis. Is this to reduce DG or any other reason, need to be defined?”

Response 3 – Thank you for your comment. In the revised text we made it clearer that the correlation analysis was used as an indicator to evaluate the validity of the design guidelines (lines 171-173) by comparing the strong positive or negative correlation between pairs of guidelines with the questionnaire scores these same guidelines received. Therefore, this indicator was not used to reduce the DG. 

Point 4 – Please provide information on why the research participants is limited to 6”.

Response 4 – Thank you for highlighting this concern. The nature of our project is exploratory, focusing on relatively uncharted territories within the intuitiveness of virtual reality authoring tools. This necessitates an initial deep dive to grasp and articulate the concept thoroughly, making qualitative research an ideal approach. Following Creswell's guidelines, we purposefully selected a small, focused group of participants. Qualitative research aims to choose participants that offer the most insightful understanding of the research question, rather than aiming for a broad, random sample typical of quantitative studies. Our choice of six participants aligns with these guidelines, aiming to provide depth and clarity to our exploratory study without the need for a large sample size. To make it clearer, we added this information in Materials and Methods section (lines 180-185) and we included the suggestion that future research be carried out with a larger sample size (lines 817-819).

Point 5 – The conclusion lacks the end result defining the evaluation result of DG and How these guidelines are useful in real time applicability. If the conclusion is written to the context of the title will be quite reflective in understanding the outcomes.

Response 5 – Thank you for your comment. In the revised version, we added information about the end result defining the evaluation result of DG and how they are useful in real time applicability. We additionally summarized the the strengths and limitations of our study (lines 806-818). We also highlighted how the design guidelines are important for sustainability (859-865)

Reviewer 2 Report

As for me, the study, although it has a clear structure, is weak.

A clear assessment of the data is shown only in Figure 5. I also do not understand why this assessment tool is not mentioned in the article's abstract.

Figure 5 breaks the text, which is unacceptable.

I recommend that the authors of the article pay attention to the abstract of the article. Describe all materials and methods used in the article.

I also advise you to describe the conditions of the experiment: focus group, task being solved, software and computer support used to solve the problem, task processing time, etc. There is this data in the text, but it should be at the beginning of the practical part of the article.

In conclusion, the authors recommend sticking to their experimental scenario, which is not enough to conclude. Recommendations and conclusions should be based entirely on the experiment's results, where it is worth mentioning the pros and cons of the methods and tools used. Provide clarity in your conclusion.

Author Response

Dear Reviewer, we are pleased to resubmit our manuscript revised according to your comments, which considerably contributed to the improvement of the work. Please find a detailed description of the changes made below:

Point 1 – A clear assessment of the data is shown only in Figure 5. I also do not understand why this assessment tool is not mentioned in the article's abstract.

Response 1 – Thank you for bringing this to our attention. In the revised version, we added the sentence “using Pearson Correlation Coefficient (PCC)” in the abstract to make it clear (lines 14-15).

Point 2 – Figure 5 breaks the text, which is unacceptable.

Response 2 – Thank you for your comment. In the revised version, all the figures' positions were reviewed.

Point 3 – I recommend that the authors of the article pay attention to the abstract of the article. Describe all materials and methods used in the article.

Response 3 – We appreciate your recommendation. In the revised version we included information about the tutorials in the NVIDIA Omniverse Enterprise platform, the number of questions asked in the focus group interview (eighteen), calculating the average score value for the tool, and the use of Pearson Correlation Coefficient (PCC) for the correlation analysis (lines 9-16).

Point 4 – I also advise you to describe the conditions of the experiment: focus group, task being solved, software and computer support used to solve the problem, task processing time, etc. There is this data in the text, but it should be at the beginning of the practical part of the article.

Response 4 – Thank you for your comment. In the revised text we made it clearer by adding Figure 5, which contains a summary of the experiment conditions.

Point 5 – In conclusion, the authors recommend sticking to their experimental scenario, which is not enough to conclude. Recommendations and conclusions should be based entirely on the experiment's results, where it is worth mentioning the pros and cons of the methods and tools used. Provide clarity in your conclusion.

Response 5 – Thank you for your comment. In the revised version of the Conclusions section, we added pros and cons of the methods and tools used (lines 774-785).

Reviewer 3 Report

This paper evaluates the effectiveness of design guidelines for intuitive VR authoring tools, fills the knowledge gaps in the field of VR authoring tools, and proposes clear research objectives.

The research adopts the research method of design science and evaluates the application of design guidelines through experiments. The methodology is clear. This method can provide reliable data support and ensure the repeatability of the study.

The study invited a group of engineering students to participate in the experiment as an audience in order to better assess the practicality and intuition of the design guidelines.

Correlation analysis confirms that most design guide scores behave as expected and are ranked by use function. This adds to the credibility of the findings.

It is hoped that the author will emphasize the contribution of the article at the end of Part 1.

The references in the article may not include the latest relevant research results. Authors are advised to conduct a more comprehensive literature review to ensure consistency with the latest research in the field. In addition, authors can cite more references to provide fuller background knowledge and support.

The conclusion section of the article allows for a more definitive summary of the findings and contributions. It is recommended that authors clearly list the main findings of the study and emphasize the practical significance of the design guidelines for the development of VR authoring tools. In addition, the authors are recommended to provide further discussion, such as suggestions for potential application areas of the design guidelines and future research directions.

All in all, this paper and the proposed solution are interesting. It just needs to add more information about the above issues, which would be perfect.

The English quality of the article is very good, and the expression is clear and fluent

Author Response

Dear Reviewer, we are pleased to resubmit our manuscript revised according to your comments, which considerably contributed to the improvement of the work. Please find a detailed description of the changes made below:

Point 1 – It is hoped that the author will emphasize the contribution of the article at the end of Part 1.

Response 1 – Thank you for bringing this to our attention. In the revised version, we added that our study addresses the challenging task of making VR software more accessible to beginners and non-experts, addressing a gap in the software design for intuitive VR tools, and included a summary of our key contributions (lines 116-133).

Point 2 – The references in the article may not include the latest relevant research results. Authors are advised to conduct a more comprehensive literature review to ensure consistency with the latest research in the field. In addition, authors can cite more references to provide fuller background knowledge and support.

Response 2 – Thank you for your feedback. It's important to highlight, though, that our experiment is informed by insights from a notably recent Systematic Literature Review (SLR) published in the Sustainability journal, which is referenced in the Materials and Methods section (line 154). The authors comprehensively and consistently described the latest research in the field in this SLR (Chamusca, I.L.; Ferreira, C.V.; Murari, T.B.; Apolinario, A.L.; Winkler, I. Towards Sustainable Virtual Reality: Gathering Design 815 Guidelines for Intuitive Authoring Tools. Sustainability 2023, 15, 2924.) 

Point 3 – The conclusion section of the article allows for a more definitive summary of the findings and contributions. It is recommended that authors clearly list the main findings of the study and emphasize the practical significance of the design guidelines for the development of VR authoring tools. In addition, the authors are recommended to provide further discussion, such as suggestions for potential application areas of the design guidelines and future research directions.

Response 3 – We appreciate your recommendation. In the revised version, to make it clearer, we included information about practical significance of the design guidelines (lines 807-819), suggestions for potential application areas (lines 820-829) and future research directions (lines 838-850).

Reviewer 4 Report

1-[Abstract, Line 15]: The statement "We conclude that..." should not be included in the abstract, as this section is intended solely to summarize rather than make definitive conclusions about the study results. This type of phrasing is out of place. Consider revising this sentence to more objectively state the key findings only.

2-[Introduction, Lines 21-22]: The authors make the claim that virtual reality technology has contributed to advancing the UN's Sustainable Development Goals. However, no evidence through citations or real-world examples is provided to support this assertion regarding VR's role. Please add substantive backing to this statement by citing relevant cases and applications of VR that have demonstrably furthered specific sustainable development targets. Lacking evidence, this claim about VR's impact remains unsubstantiated.

3-[Materials and Methods, Lines 113-114]: The passage mentions following six steps aligned with a Design Science Research methodology. However, details are not provided on what each distinct step entailed and how they built upon one another. Please elaborate on the progression through the DSR steps, outlining the key activities, outcomes, and learnings generated during each one to provide transparency on this iterative process underlying the research approach.

4-[Materials and Methods, Lines 183-184]: The experiments involved participants completing Omniverse Enterprise training tasks. However, the text does not specify whether these participants had any prior product orientation, tutorials, or onboarding before embarking on the assessed VR authoring activities. Please clarify if the participants received upfront training in navigating the platform controls and interface or were expected to perform the required tasks based solely on baseline familiarity with the system.

5-[Results, Line 306]: The rationale behind resorting to examining correlation coefficient values is unclear as presented. Please explain the purpose and relevance of conducting this inter-dependency correlation analysis among the 14 design guidelines. What insight does this inter-relationship assessment lend to evaluating the guidelines as a collective set and understanding connections between them based on tool functionality? Articulating the reasoning will help justify inclusion of this supplementary analysis.

6-[Discussion, Lines 717-720]: The passage identifies the participant group composition as a potential limitation but does not elaborate on why this is the case. Please justify why the background, technical expertise level, sample size, or other attributes of the participant group selected may have negatively impacted or limited the evaluation outcomes regarding the design guidelines. More supporting explanation is needed.

7-[Conclusions, Lines 748-749]: The concluding statement that the guidelines definition may "evolve over time" within the scope of this one study is illogical and temporally impossible. Please rephrase this to avoid the false impression.

Author Response

Dear Reviewer, we are pleased to resubmit our manuscript revised according to your comments, which considerably contributed to the improvement of the work. Please find a detailed description of the changes made below:

Point 1 – [Abstract, Line 15]: The statement "We conclude that..." should not be included in the abstract, as this section is intended solely to summarize rather than make definitive conclusions about the study results. This type of phrasing is out of place. Consider revising this sentence to more objectively state the key findings only.

Response 1 – Thank you for bringing this to our attention. In the revised version, we changed the word “conclude” for “evaluated” (line 17).

Point 2 – [Introduction, Lines 21-22]: The authors make the claim that virtual reality technology has contributed to advancing the UN's Sustainable Development Goals. However, no evidence through citations or real-world examples is provided to support this assertion regarding VR's role. Please add substantive backing to this statement by citing relevant cases and applications of VR that have demonstrably furthered specific sustainable development targets. Lacking evidence, this claim about VR's impact remains unsubstantiated.

Response 2 – Thank you for your feedback. In the revised version, we added evidence to support our assertion that virtual reality technology has contributed to advancing the UN's Sustainable Development Goals (lines 24-64). The examples provided are sourced from two highly recent systematic literature reviews, offering further reading for those interested in exploring the cited cases in more depth. 

Point 3 – [Materials and Methods, Lines 113-114]: The passage mentions following six steps aligned with a Design Science Research methodology. However, details are not provided on what each distinct step entailed and how they built upon one another. Please elaborate on the progression through the DSR steps, outlining the key activities, outcomes, and learnings generated during each one to provide transparency on this iterative process underlying the research approach.

Response 3 – We appreciate your recommendation. In the revised version, to make it clearer, we included Figure 2, which synthesizes the DSR steps entailing how they built upon one another and outline the key activities, providing transparency to the process.

Point 4 – [Materials and Methods, Lines 183-184]: The experiments involved participants completing Omniverse Enterprise training tasks. However, the text does not specify whether these participants had any prior product orientation, tutorials, or onboarding before embarking on the assessed VR authoring activities. Please clarify if the participants received upfront training in navigating the platform controls and interface or were expected to perform the required tasks based solely on baseline familiarity with the system.

Response 4 – Thank you for bringing this to our attention. In the revised version, to make it clearer, we shared more details about the process of using Omniverse. The participants had a first onboarding moment to learn how to install the application needed to run the virtual machine and afterwards used the documentation in LaunchPad as a guide to execute the activities, no prior training was needed (lines 226-229).

Point 5 – [Results, Line 306]: The rationale behind resorting to examining correlation coefficient values is unclear as presented. Please explain the purpose and relevance of conducting this inter-dependency correlation analysis among the 14 design guidelines. What insight does this inter-relationship assessment lend to evaluating the guidelines as a collective set and understanding connections between them based on tool functionality? Articulating the reasoning will help justify inclusion of this supplementary analysis.

Response 5 – Thank you for your comment. In the revised version we included more details about the use of the correlation analysis as an indicator to evaluate the validity of the design guidelines (lines 171-173). We also added to the materials and methods section more details about how this indicator was used (lines 262-268). It was expected that guidelines with strong positive correlation would receive similar scores, therefore lower difference values, while guidelines with strong negative correlation would receive very different scores, therefore higher difference values. The correlation analysis was obtained to serve as a demonstration of how the guidelines could be used to evaluate a VR authoring tool and to be compared with the results obtained with the second method. It was assumed that, if the results obtained from testing the tool were compatible with expectations, this would be a good indication of the validity of the guidelines as a guide to develop and evaluate more intuitive VR authoring tools. This idea was reinforced in the results section (lines 360-363).

Point 6 – [Discussion, Lines 717-720]: The passage identifies the participant group composition as a potential limitation but does not elaborate on why this is the case. Please justify why the background, technical expertise level, sample size, or other attributes of the participant group selected may have negatively impacted or limited the evaluation outcomes regarding the design guidelines. More supporting explanation is needed.

Response 6 – Thank you for bringing this to our attention. In the revised version, we explained the participants had an engineering background, which characterizes them as final users of the tool, which is very important to evaluate a software intuitiveness, but does not provide feedback on how the software development should effectively improve. Therefore, we lack feedback on how design guidelines can help software developers evaluate an existing authoring tool with the aim of modifying it for better functions. (lines 791-800)

Point 7 – [Conclusions, Lines 748-749]: The concluding statement that the guidelines definition may "evolve over time" within the scope of this one study is illogical and temporally impossible. Please rephrase this to avoid the false impression.

Response 7 – Thank you for your comment. In the revised version, we adjusted the term “evolve” to “reviewed and modified”. (line 850).