Comparing Virtual and Real-Life Rapid Prototyping Methods for User Testing Smart City Interfaces: A Case Study

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

 

This study focuses on the integration of Virtual Reality (VR) in prototyping embedded interactive systems, examining its potential to bridge the gap between rapid prototyping and user-centered design validation. They adopted a comparative research approach by analyzing a case study: the development of a cultural smart city experience. The strengths and limitations of the paper are listed as follows:   Pros: 1) The paper is well written, background information and problem statements are clearly presented. 2) The research question is of interest to the communities of design and VR. 3) They have done several user studies, and the evaluations seem sound.   Cons: 1) The evaluation: it's not clear whether it's a between or within-subject test. Maybe I overlooked, but this information is very important for readers to understand the presented effects. 2) The evaluation: The results of normality test should be presented. 3) It's a pity that only questionnaires were used, as many physiological devices are available now. 4) Figures: I'm confused by many of the figures in this paper, for example, Figure 3, 4, 5 should be combined as one since they describe similar subjects. The captions of Figure 6, 7, 8, etc. are too simple. The figures, along with their captions, should make the figure easy to understand without referring too much contextual information in the main text. 5) Section 4 is way too short, many discussions should be moved from section 3 to section 4. 6) References: most cited papers are quite old, more recent papers (after 2020) should be added.

 

Author Response

It's not clear whether it's a between or within-subject test. Maybe I overlooked, but this information is very important for readers to understand the presented effects.

• Thank you for pointing this out, we agree with your comment. We added a sentence stating that it is a within subject design (Line 404)

The results of normality test should be presented.

• Thank you for the comment, we reported the results of the normality test in the paper and referred to the appendix with its analysis. (Line 498-501)

It's a pity that only questionnaires were used, as many physiological devices are available now.

• This is a good comment, unfortunately this was not part of this experimental design, but we aim to incorporate physiological measures in future experiments.

I'm confused by many of the figures in this paper, for example, Figure 3, 4, 5 should be combined as one since they describe similar subjects. The captions of Figure 6, 7, 8, etc. are too simple. The figures, along with their captions, should make the figure easy to understand without referring too much contextual information in the main text.

• Thank you for the feedback. We agree completely, therefore we renumbered figures and provided new description for the figures. (Line 282-286; Line 350-353)

Section 4 is way too short, many discussions should be moved from section 3 to section 

• Thank you for the comment, we have expanded the discussion with text from the result section in order to balance both sections of the paper.

Most cited papers are quite old, more recent papers (after 2020) should be added.

• Thank you for this feedback. We updated the paper with thirteen recent references to support the relevance of the study with regards to current literature.

Reviewer 2 Report

Comments and Suggestions for Authors

Comments and Suggestions for Authors

The article presents the problem of building complex systems and the need to test them at various stages of their development. This is especially important if the final product will require a lot of user interaction. The approach adopted by the authors, the assumption of which is to consult the functionality of the system already at the design stage and to create it, is most correct. The authors of the article used modern technology in the form of virtual reality (VR) for this purpose.  The results revealed the divergent roles of field testing and VR in the new product development process, highlighting the strengths of VR in the visualization of procedures. The study identifies the limitations of VR in mimicking realistic interactions and incorporating social context, highlighting the superiority of VR over paper prototypes in terms of realism and interactivity.

The introduction section is very extensive and addresses a number of issues related to the topic of the article. They distinguished in it:

·         Low fidelity prototypes for in-situ testing

·         Virtual prototyping and testing in the lab

·         High fidelity prototypes for validation testing

·         Evaluating VR prototyping

The authors correctly noted that one of the main challenges is the difficulty of developing interactive prototypes in the early stages of design using low-cost methods. Therefore, it is difficult to communicate these prototypes to the right stakeholders. The paper presents how VR prototypes can help in the process of designing new interfaces in the NPD process, focusing on the smart city application. The authors compared two assessment methods in one NPD in which the role of VR was considered. Field testing with low-fidelity prototypes was evaluated at an early stage, and the finished product was then evaluated.

The article contains the observations and opinions of experts (expert on content creation, expert on social cohesion, expert on product design, Expert on project management)

 However, I have a few questions and suggestions:

The authors used existing VR technology and questionnaires to study of the experiment. They did not clearly indicate novelties in their approach to the implementation of the project.

Line 273 The figure should be marked as Firgure 3, and the figures: a, b, c. Reference in text as Fig.3a.

Line 334 Figure 7. The presented screen views should be marked a, b, c (analogous to the note figure 3)

Lines 363-366 Studies were conducted with 5 and 2 participants. This is a very small group of participants. Were people familiar with virtual reality technology involved the study (had they used it before?).

Line 384-386 27 People (7 women) took part in the test. Please explain what was the average age of all test participants?

In the discussion section, when formulating conclusions, they should refer to specific expert assessments.

 Please refer to the negative effects of using VR technology (visual fatigue and spatial disorientation). It seems that it would be advisable to indicate the duration of individual tests, because the duration of the test may affect the comfort of the tested people.

Comments for author File: Comments.pdf

Author Response

The authors used existing VR technology and questionnaires to study of the experiment. They did not clearly indicate novelties in their approach to the implementation of the project. 

• Thank you for your comment. We acknowledge that the study does not portray itself as very novel. However, the intention of our study was to add to the limited and slightly outdated body of work concerning virtual product evaluations. Combining the research of testing in the field, virtual reality testing and final product evaluation on one use case is unique. The method we applied in our study can also be utilized on newer applications and technologies as well.

Line 273 The figure should be marked as Firgure 3, and the figures: a, b, c. Reference in text as Fig.3a.

• Thank you for this clarification, we renumbered figures and edited the text with the new numbering. (Line 280-281)

Line 334 Figure 7. The presented screen views should be marked a, b, c (analogous to the note figure 3)

• Thank you for this clarification, we renumbered figures and edited the text with the new numbering. (Line 346-348)

Lines 363-366 Studies were conducted with 5 and 2 participants. This is a very small group of participants. Were people familiar with virtual reality technology involved the study (had they used it before?).

• Good question. The social science researchers who reviewed the different prototypes had little experience with virtual reality specifically. We updated this remark in the text: “These social science researchers had limited knowledge of virtual reality.”(Line 383-384)

Line 384-386 27 People (7 women) took part in the test. Please explain what was the average age of all test participants?

• Thank you for the comment. The average reported applies to all participants. This was clarified by adding the number of men between the brackets as well. (Line 403-404)

In the discussion section, when formulating conclusions, they should refer to specific expert assessments.

• Thank you for pointing this out. We rewrote the discussion section and gave extra attention to include which specific expert reported on which specific finding.

 Please refer to the negative effects of using VR technology (visual fatigue and spatial disorientation). It seems that it would be advisable to indicate the duration of individual tests, because the duration of the test may affect the comfort of the tested people.

• Thank you for this addition. We added two references in the text referring to negative effects (line 144). In the method section we discuss how the design of the experience relates to these negative effects (line 343-344). Next to the experience being a very short VR scenario (5-10min), the study also applied an onboarding condition to become familiar with the controls and environment to counter some negative effects.  

Reviewer 3 Report

Comments and Suggestions for Authors

The work proposes using virtual reality (VR) in prototyping embedded interactive systems to balance rapid design processes with user-centered validation. It compares in situ tests with VR, highlighting VR for its capacity for visualization and discussion of procedures, despite its limitations in simulating realistic interactions, concluding that it is useful for testing user flows in complex interfaces.

Please consider the following comments to improve the work:

In the introduction section, update or add a section on related work with updated references not older than 5 years. It is important to note that some of the cited references are over 20 years old, which undermines the work's foundation.

In the methods section, the questionnaire data are publicly available in an OSF repository, but the link in the document is not accessible. Improve the discussion section by adding an analysis of the results found in the quantitative comparative test.

In the conclusions: It would be helpful to include specific data and results that support the claims made in the conclusion.

Author Response

In the introduction section, update or add a section on related work with updated references not older than 5 years. It is important to note that some of the cited references are over 20 years old, which undermines the work's foundation.

• Thank you for the comment. We updated the paper with thirteen recent references to support the relevance of the study with regards to current literature. With this update many references are included that are published after 2019.

In the methods section, the questionnaire data are publicly available in an OSF repository, but the link in the document is not accessible. Improve the discussion section by adding an analysis of the results found in the quantitative comparative test.

• Thank you for the remark. I checked the link with a colleague not related to the project and replaced the TinyURL-link with the original OSF-link in the text (line 304). We hope the problem is solved and was related to the TInyURL.

In the conclusions: It would be helpful to include specific data and results that support the claims made in the conclusion.

• Thank you for you comment. We have tried our best to update and specify which expert, participant or statistic provided which result.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

The paper proposes using virtual reality (VR) in the prototyping of embedded interactive systems to balance rapid design processes with user-centered validation. It compares on-site testing with VR, highlighting VR’s strengths in visualizing and discussing procedures, despite its limitations in simulating realistic interactions, concluding that it is useful for testing user flows in complex interfaces.

 

However, the content lacks depth and does not provide significant advances to the field. Additionally, while the references are relevant, they are not well integrated with the study’s findings, weakening the manuscript’s cohesion and rigor.

 

The paper’s conclusion fails to adequately synthesize the main findings, presenting scattered observations without a clear view of their contribution to existing knowledge. The recommendations for future research are vague and lack specificity, offering no concrete hypotheses or clear methodologies to address the identified gaps. Furthermore, the literature review is inconsistent, with references to previous studies that are not well connected to the current results, which weakens the link between this work and existing research. Additionally, the metrics used in the quantitative experiments were insufficient to fully capture the complexity of interactions in the evaluated virtual environments. This limitation affects the validity and applicability of the results, indicating a need for more robust and detailed metrics in future studies to achieve a more accurate and comprehensive evaluation.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 3 Report

Comments and Suggestions for Authors

The authors did not adequately address the questions raised, merely pointing to sections without delving into the expected details. A better design of both the experiments and entire sections of the document was anticipated. Additionally, the recommendations for future research are vague and lack specificity, offering no concrete hypotheses or clear methodologies to address the identified gaps. The literature review is inconsistent, with references to previous studies that are not well connected to the current results, weakening the link between this work and existing research. Moreover, the metrics used in the quantitative experiments were insufficient to fully capture the complexity of interactions in the evaluated virtual environments. This limitation affects the validity and applicability of the results, highlighting the need for more robust and detailed metrics in future studies to achieve a more accurate and comprehensive evaluation.