ArkaeVision VR Game: User Experience Research between Real and Virtual Paestum

Round 1

Reviewer 1 Report

This article is quite impressive.
because the large and various groups are joined for the research and the research theme is very interesting.
Moreover, the theoretical, historical background and processes have a proper logicality.

however, the related works like Arkae vision are needed to be listed up in separated chapter like 'related work'.

Author Response

Dear reviewer, 

section 1 and 1.1 have been largely revised with background works on cognition, emotions, user experience design and learning principles.

Some related works have been included in 3.1. (framework of game-based applications) and 3.4 (learning analytics, Bloom's Taxonomy of experiential learning).

Thanks,

Alfonsina Pagano

Reviewer 2 Report

Dear Authors, 

Thank you for your manuscript. You have presented an interesting take on studying an application virtual reality in cultural heritage. The technical work behind this reconstructions is paramount as the quality of the computer graphics has an AAA style. However, the scientific component needs a major revision as the document is lacking consistency that can be addressed by clearly defining the research question and hypothesis for the study. A literature section is missing from the paper and it will strengthen the manuscript significantly as it would allow you to stress out the gap your work addresses. Page limit is another component that can use improvements as the current 33 pages make the manuscript difficult to read. this is mainly because of the English improvements that are required and the extensive historical background provided, which is not needed with that level of detail. While the paper indicates that a game was developed, the design process is not well document and the word game and gamification are used interchangeably. This is worrisome as a gamification design process is not presented either. The next major item to address is the study design. Some of the challenges are findings have been already discussed in the existing literature and that is why reviewing existing works can help better highlight your contributions. Regarding the study methods, please present them in detail as the data collection and questionnaires are presented assuming the readers knows about them.  

Comments for author File: Comments.pdf

Author Response

Dear reviewer, 

Research questions

They have been explicited in the introduction of the paper, as well as in section 3.1.3 (components of the VR exploration) and 4.1 (goals of the evaluations)

Related works

Section 1 and 1.1 have been largely revised with background works on cognition, emotions, user experience design and learning principles.

Some related works have been included in 3.1. (framework of game-based applications) and 3.4 (learning analytics, Bloom's Taxonomy of experiential learning).

Historical background 

It has been reduced significantly.

Game, gamification and game design process

It is important to explain that ArkaeVision is not properly a serious-game and it
does not fully exploit the features of the gamification; yet, it is intended to leverage people's natural desires for socializing, learning, achievement, self-expression and fun; moreover, it partially follows game design elements. 

Along all the paper, it is explained and addressed the approach we used for the VR application.

Data collection

Section 4.1 has been largely revised including information about methodology used, framework of reference, analysis of quantitative and qualitative data and tools used for the evaluation (details of questions and answers). 

Thanks,

Alfonsina Pagano

Reviewer 3 Report

Brilliant work indeed! The authors provided a very comprehensive background to the design intentions of Arkaevision Archeo, which is a VR environment to offer a complete immersive user experience for Cultural Heritage (CH) tours. The UX evaluations are appropriate, while more scientific considerations should be taken into account. Regarding the potential improvement of the research work presented, the primary suggestions raised by the review is to put more focus on the details of the comparative part. The information to identify the uniqueness of the proposed system needs to be enhanced. For example, the authors conducted analytic activities on two occasions, and the functions of the system are at two different development stages, while the results were discussed at once in general. What are the disadvantages identified in the previous evaluation that enables the revision of the system? Also, what learning principles and design intentions are adopted have been proved effective comparatively, and what trends of user reactions can be proved that superior to those in other applications with similar purposes? In the discussion section, suggest put more observed facts based on the experiments to support the discussion points here. Some minor problems and suggestions are provided by the reviewer as below, please consider them as well:

• (pp. 14) Any further details about the game action/tools and how they bring the influence on the VR evaluation?
• (pp. 19) The figure is cropped incorrectly
• (pp. 19) Some examples of learning tasks would be helpful for readers to understand

Author Response

Dear reviewer,

below our comments.

Focus on the details of the comparative part of evaluations

Section 4.1 has been largely revised including information about methodology used, framework of reference, analysis of quantitative and qualitative data and tools used for the evaluation (details of questions and answers). 

Learning principles and design intentions

Some related works have been included in 3.1. (framework of game-based applications) and 3.4 (learning analytics, Bloom's Taxonomy of experiential learning).

Further details about the game action/tools 

It is important to explain that ArkaeVision is not properly a serious-game and it
does not fully exploit the features of the gamification; yet, it is intended to leverage people's natural desires for socializing, learning, achievement, self-expression and fun; moreover, it partially follows game design elements. 

Along all the paper, it is explained and addressed the approach we used for the VR application.

Examples of learning tasks

They have been explained in sections 3.4.1, 3.4.2 and 3.4.3

Thanks,

Alfonsina Pagano

Reviewer 4 Report

The paper discusses the ux design lifecycle of a cultural-heritage VR game. The paper is generally well-written and well-structured; however, sometimes, the information provided is superfluous and thus the authors could discuss their points in a more concise way. I have some concerns, which I discuss next:

1) Introduction, Motivation, and Related Work: An interesting work about UX and CH, which the authors could consider to motivate their paper more efficiently, is the recent work by Konstantakis and Caridakis [1]. Moreover, while the Introduction section has a good flow, I miss a clear motivation, leading to a research question and objectives. I would suggest to include such elements, to make clear for the reader what to expect and what the contribution of the paper is (a brief summary of it might be useful). Another concern is that while the cognition is mentioned by the authors, it is not discuss in depth. Focusing on the area of cultural-heritage games and cognition, an interesting work that the authors could consider including and discussing about (to back up the importance of cognition/cognitive functions when engaging with CH activities) is the one by Raptis et al. [2].

2) Cultural-Heritage games: I miss two major works (review papers) which discuss the design and development of games for cultural heritage [3, 4]. I would suggest the authors to discuss on them.

3) Game design: It is unclear whether the authors followed a specific model/framework to make the design decisions (e.g., Sections 3.1 and 3.2). I would suggest a clarification and justification of the decisions made. 

4) Evaluation and results: While the evaluation is discussed in a adequate level of detail, I would like to know how the analysis of the qualitative data was performed. Like before, did the authors follow a validated/established framework to analyze the qualitative data (data from observation)? Did the developed their own codes (e.g., through a theme-coding procedure)? I would recommend the authors to clarify the procedure the followed and justify the evaluation/analysis decisions they made.

[1] Konstantakis, M., & Caridakis, G. (2020). Adding Culture to UX: UX Research Methodologies and Applications in Cultural Heritage. Journal on Computing and Cultural Heritage (JOCCH), 13(1), 1-17.

[2] Raptis, G. E., Fidas, C., & Avouris, N. (2019). Do Game Designers’ Decisions Related to Visual Activities Affect Knowledge Acquisition in Cultural Heritage Games? An Evaluation From a Human Cognitive Processing Perspective. Journal on Computing and Cultural Heritage (JOCCH), 12(1), 1-25.

[3] Mortara, M., Catalano, C. E., Bellotti, F., Fiucci, G., Houry-Panchetti, M., & Petridis, P. (2014). Learning cultural heritage by serious games. Journal of Cultural Heritage, 15(3), 318-325.

[4] Anderson, E. F., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, P., & De Freitas, S. (2010). Developing serious games for cultural heritage: a state-of-the-art review. Virtual reality, 14(4), 255-275.

Author Response

Dear reviewer,

below our comments.

Introduction, Motivation, and Related Work

Research questions have been explicited in the introduction of the paper, as well as in section 3.1.3 (components of the VR exploration) and 4.1 (goals of the evaluations) .

References suggested by you have been included and discussed.

Cultural-Heritage games & Game design

It is important to explain that ArkaeVision is not properly a serious-game and it
does not fully exploit the features of the gamification; yet, it is intended to leverage people's natural desires for socializing, learning, achievement, self-expression and fun; moreover, it partially follows game design elements. 

Along all the paper, it is explained and addressed the approach we used for the VR application.

Nevertheless, reference suggested by you has been included and discussed.

Evaluation and results

Section 4.1 has been largely revised including information about methodology used, framework of reference, analysis of quantitative and qualitative data and tools used for the evaluation (details of questions and answers).

Also section 5 has been addressed in this sense.

Thanks,

Alfonsina Pagano

Round 2

Reviewer 4 Report

Thank you for submitting the revised version. It answers my concerns and I believe that the manuscript is ok to be published.

Author Response

Dear reviewer,

we have made some minimal changes as follow:

1) we have reduced the repetition rate according to the recheck report;

2) we underwent an extensive English revision;

3) we checked copyright of Figure 1 (which is licensed under the Creative Common), Figures 13 and 14 (which are publicly published, with no restrictions by authors even if no under Creative common);

4) we have no original descriptions of research conducted in experimental animals.

We have made these changes with the tracking and they are also highlighted in red color.

All the best,

Alfonsina Pagano

Author Response File: Author Response.docx