User Experience of Virtual Heritage Tours with 360° Photos: A Study of the Chapel of Dolores in Icod de los Vinos

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

In this paper, the authors present the results, in terms of quality of user experience, obtained by conducting a virtual tour using different technologies: Virtual reality headsets and smartphones.

A well-defined methodology is used. The methodology is based on a standardized user experience questionnaire. In analyzing the results, it is not clear whether one technology is definitely better than the other or whether there are different fields of application or whether there are open problems to be solved to give an answer. 

The authors should clarify these points

 

Author Response

Thank you very much for your review.

… it is not clear whether one technology is definitely better than the other or whether there are different fields of application or whether there are open problems to be solved to give an answer. The authors should clarify these points

 

It has been clarified in the second paragraph of the conclusions: “Despite the fact that Smartphone screens are considered low immersion devices and VR headsets are considered high immersion devices, similar results were obtained. Our conclusion is that in our case study, it is not necessary to have virtual reality headsets to visualize the virtual tour. However it would be interesting to elaborate on this similarity in future work.”

We also clarify at the end of the conclusion that: “As a future work we propose the analysis of scenarios where the use of virtual reality headsets is necessary compared to other types of visualizations.”

Reviewer 2 Report

Comments and Suggestions for Authors

This is an interesting research, it would be significant in the future to expand the number of participants and verify the results on other artistic heritages.

Author Response

Thank you very much for your review.

 

This is an interesting research, it would be significant in the future to expand the number of participants and verify the results on other artistic heritages.

 

Dear reviewer. In the abstract it is mentioned that this is a pilot test, which allows us to get a preliminary idea of the research. Anyway, to clarify this topic, at the end of the conclusion section we added: “In addition, we propose to extend this research with a larger and more diverse number of participants, in order to obtain more conclusive results.”

 

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Authors,

In the State of the Art section,  the content related to this type of evaluation in the heritage sector is needed, so your research can be succinctly described and contextualized with respect to previous and present theoretical background and empirical research.

In section 3.2 you should explain why did you select this evaluation test and not others, as well as on why did you select those 27 questions and not the rest of them.

Results should be further discussed what are the actual implications of the study? Authors either provide more explanations in this section or they add a discussion section

In conclusions, it would be desirable to know if authors are planning to do more evaluations to support their results.  Regarding heritage dissemination, what are the future line of work

Comments on the Quality of English Language

Authors should review their paper, as there are some Spanish words, e.g. Table 2.

Author Response

Review 3

Thank you very much for your review.

In the State of the Art section,  the content related to this type of evaluation in the heritage sector is needed, so your research can be succinctly described and contextualized with respect to previous and present theoretical background and empirical research.

Certainly, this part of the paper needed to be expanded on this topic, therefore the following text with 8 new references has been added at the end of point 2. Background

As mentioned above, virtual reality, in order to be effective as a communication element in the dissemination of heritage, must be accepted by the user. In other fields (architecture, engineering, landscaping, etc.) the measurement of user experience in virtual environments is very common [1, 2, 3]. This implies that the user experience of virtual content created on cultural heritage should be studied [4]. For example, Škola et al. [5] measure presence, engagement, and immersion in virtual reality with 360-Video in Cultural Heritage.

Verhulst et al. compare different types of Virtual and Augmented Reality technologies around cultural heritage, with a result in high enjoyment in all devices, and similar cognitive and emotional engagement levels, however with the presence higher in one of the devices [6]. Other studies compare a large number of virtual heritage resources, visible on screen, and review different ways to evaluate the performance of such applications with surveys of workload, usability, flow, and potential VR symptoms. They conclude that there is a need to access materials in easily accessible formats and with great detail. Low-resolution textures and rough 3D models can deteriorate immersion, user experience and overall application quality [7].  In other cases, they study accessibility enhancement and intangible heritage preservation, and also compare desktop media and virtual reality. The conclusions show that the virtual reality experience did not affect the effort expectancy for the desktop application, but the same experience significantly influenced the performance expectancy construct [8].

 

 

[1]	J. Saorin, C. Carbonell-Carrera, A. Jaeger and D. Díaz, "Landscape Design Outdoor–Indoor VR Environments User Experience," Land, vol. 12, no. 2, p. 376, 2023.  https://doi.org/https://doi.org/10.3390/land12020376
[2]	C. Carbonell-Carrera, J. Saorin and D. Melián Díaz, "User VR Experience and Motivation Study in an Immersive 3D Geovisualization Environment Using a Game Engine for Landscape Design Teaching," Land, vol. 10, no. 5, p. 492, 2021. https://doi.org/https://doi.org/10.3390/land10050492
[3]	Y. L. Chang, H. T. Hou, C. Y. Pan, Y. T. Sung and K. E. Chang, "Apply an Augmented Reality in a Mobile Guidance to Increase Sense of Place for Heritage Places," Journal of Educational Technology & Society, vol. 18, no. 2, pp. 166-178, 2015. https://www.jstor.org/stable/jeductechsoci.18.2.166
[4]	M. Konstantakis and G. Caridakis, "Adding Culture to UX: UX Research Methodologies and Applications," Journal on Computing and Cultural Heritage, vol. 13, no. 1, pp. 1-17, 2020. https://doi.org/https://doi.org/10.1145/3354002
[5]	F. Škola, S. Rizvić, M. Cozza, L. Barbieri, F. Bruno, D. Skarlatos and F. Liarokapis, "Virtual Reality with 360-Video Storytelling in Cultural Heritage: Study of Presence, Engagement, and Immersion," Sensors, vol. 20, no. 20, p. 5851, 2020. https://doi.org/10.3390/s20205851
[6]	I. Verhulst, A. Woods, L. Whittaker, J. Bennett and P. Dalton, "Do VR and AR versions of an immersive cultural experience engender different user experiences?," Computers in Human Behavior, vol. 125, p. 106951, 2021. https://doi.org/https://doi.org/10.1016/j.chb.2021.106951
[7]	H. Cecotti, "Cultural Heritage in Fully Immersive Virtual Reality," Virtual Worlds, vol. 1, no. 1, pp. 82-102, 2022. https://doi.org/10.3390/virtualworlds1010006
[8]	E. Selmanović, S. Rizvic, C. Harvey, D. Boskovic, V. Hulusic, M. Chahin and S. Sljivo, "Improving Accessibility to Intangible Cultural Heritage Preservation Using Virtual Reality," Journal on Computing and Cultural Heritage, vol. 13, no. 2, pp. 1-19. https://doi.org/10.1145/3377143

Reviewer 4 Report

Comments and Suggestions for Authors

This manuscript presents a methodology for developing a 360^o virtual tour based on 360^o photos of the Chapel of Nuestra 30 Señora de los Dolores, in Tenerife. The 360^o virtual tour is available for smartphones and smartphone on cardboard. Although the chapel is still open for worship, its access is limited to specific days of the month and festivities, so the creation of a virtual tour provides the possibility of being able to visit it at any time. Moreover, this 360^o virtual tour reveals rooms and spaces that, even on the days the Chapel is open, are not accessible to the public. The manuscript presents also the results of a very limited evaluation by 18 participants of a specific target group.

Unfortunately, I do not feel confidence with many aspects of this manuscript.

Regarding the background of this work: It is based on obsolete references. Moreover, the majority of the references is based on conferences. The lack of articles in journal from last five years leads the contributors of the manuscripts in pitfalls. In addition, the lack of English references, as well as the absence of DOIs caused a significant problem in their search.

Regarding the immersion: Both of the means for the representation of the virtual tour offers near zero immersion. They are based on 2D images in spherical projection, so the sense of depth is totally absent. Even if a cardboard is characterized as a head mounted display, the interactions through this device provide no physical mimic. These devices use gaze and teleporting that are not physical interaction. Thus, a comparison between two technologies that provide a near zero immersion it could be a realistic option.

Regarding the evaluation: The number of participants (N=18), as well as the target group (students of the fourth year course, Conservation and Restoration of Altarpieces, of the degree of conservation and restoration of cultural goods of the University of La Laguna), raise critical doubts about the objectivity and representativeness of the results, for any safe and useful conclusions to be drawn.

Comments on the Quality of English Language

There is a need for minor editing.

Author Response

Thank you very much for your review.

The manuscript presents also the results of a very limited evaluation by 18 participants of a specific target group.

Certainly, in the abstract it is mentioned that this is a pilot test, which allows us to get a preliminary idea of the research. Anyway, to clarify this topic, at the end of the conclusion section we added: “In addition, we propose to extend this research with a larger and more diverse number of participants, in order to obtain more conclusive results.”

Regarding the background of this work: It is based on obsolete references. Moreover, the majority of the references is based on conferences. The lack of articles in journal from last five years leads the contributors of the manuscripts in pitfalls. In addition, the lack of English references, as well as the absence of DOIs caused a significant problem in their search.

An error occurred when inserting the references automatically in the text file. It has been corrected and the DOI has been added to all possible references.

The following 12 journal and English references (11 from the last five years) have been added:

[1]	J. Saorin, C. Carbonell-Carrera, A. Jaeger and D. Díaz, "Landscape Design Outdoor–Indoor VR Environments User Experience," Land, vol. 12, no. 2, p. 376, 2023.  https://doi.org/https://doi.org/10.3390/land12020376
[2]	C. Carbonell-Carrera, J. Saorin and D. Melián Díaz, "User VR Experience and Motivation Study in an Immersive 3D Geovisualization Environment Using a Game Engine for Landscape Design Teaching," Land, vol. 10, no. 5, p. 492, 2021. https://doi.org/https://doi.org/10.3390/land10050492
[3]	Y. L. Chang, H. T. Hou, C. Y. Pan, Y. T. Sung and K. E. Chang, "Apply an Augmented Reality in a Mobile Guidance to Increase Sense of Place for Heritage Places," Journal of Educational Technology & Society, vol. 18, no. 2, pp. 166-178, 2015. https://www.jstor.org/stable/jeductechsoci.18.2.166
[4]	M. Konstantakis and G. Caridakis, "Adding Culture to UX: UX Research Methodologies and Applications," Journal on Computing and Cultural Heritage, vol. 13, no. 1, pp. 1-17, 2020. https://doi.org/https://doi.org/10.1145/3354002
[5]	F. Škola, S. Rizvić, M. Cozza, L. Barbieri, F. Bruno, D. Skarlatos and F. Liarokapis, "Virtual Reality with 360-Video Storytelling in Cultural Heritage: Study of Presence, Engagement, and Immersion," Sensors, vol. 20, no. 20, p. 5851, 2020. https://doi.org/10.3390/s20205851
[6]	I. Verhulst, A. Woods, L. Whittaker, J. Bennett and P. Dalton, "Do VR and AR versions of an immersive cultural experience engender different user experiences?," Computers in Human Behavior, vol. 125, p. 106951, 2021. https://doi.org/https://doi.org/10.1016/j.chb.2021.106951
[7]	H. Cecotti, "Cultural Heritage in Fully Immersive Virtual Reality," Virtual Worlds, vol. 1, no. 1, pp. 82-102, 2022. https://doi.org/10.3390/virtualworlds1010006
[8]	E. Selmanović, S. Rizvic, C. Harvey, D. Boskovic, V. Hulusic, M. Chahin and S. Sljivo, "Improving Accessibility to Intangible Cultural Heritage Preservation Using Virtual Reality," Journal on Computing and Cultural Heritage, vol. 13, no. 2, pp. 1-19. https://doi.org/10.1145/3377143
[9]	O. Nieva García, P. Luna González and J. Arellano Pimentel, "Comparativa de características de software para la creación de recorridos virtuales 360 en Web," Revista De Investigación En Tecnologías De La Información, vol. 9, no. 19, p. 109–117, 2022. https://doi.org/10.36825/RITI.09.19.009
[10]	H. Rahaman, E. Champion and D. McMeekin, "Outside Inn: Exploring the Heritage of a Historic Hotel through 360-Panoramas," Heritage, vol. 6, pp. 4380-4410, 2023. https://doi.org/10.3390/heritage6050232
[11]	S. Martirosov, M. Bureš and T. Zítka, "Cyber sickness in low-immersive, semi-immersive, and fully immersive virtual reality," Virtual Reality, vol. 26, pp. 15-32, 2021. https://doi.org/10.1007/s10055-021-00507-4
[12]	R. Kaplan-Rakowski and A. Gruber, "Low-immersion versus high-immersion virtual reality: Definitions, classification, and examples with a foreign language focus," in Innovation in Language Learning 2019, 2019.

 

 

Regarding the immersion: Both of the means for the representation of the virtual tour offers near zero immersion. They are based on 2D images in spherical projection, so the sense of depth is totally absent. Even if a cardboard is characterized as a head mounted display, the interactions through this device provide no physical mimic. These devices use gaze and teleporting that are not physical interaction. Thus, a comparison between two technologies that provide a near zero immersion it could be a realistic option.

We agree that in the virtual tour of our work there is no interaction beyond the fact that with the movement of the head you can visualize a scene in its entire extension and perform teleporting from one scene to another.

In the text we indicate that desktop virtual visualizations are categorized as low immersion devices [11]. (Line 166)

The following paragraph was added to Conclusions to clarify this question :

Despite the fact that Smartphone screens are considered low immersion devices and VR headsets are considered high immersion devices [27, 29], similar results were obtained. Ideally immersion also includes a simulation of acoustic, haptic, smell, taste and motion senses. However, immersion in user experience is defined as: the illusion that the virtual environment technology replaces the user's sensory stimuli by the virtual sensory stimuli. Therefore, in the virtual tours presented in this article made with 360 photos and without any other haptic device, there is only the visual sensation of immersion. Our conclusion is that in our case study without any haptic device and only visual immersion, it is not necessary to have virtual reality headsets to visualize.

Regarding the evaluation: The number of participants (N=18), as well as the target group (students of the fourth year course, Conservation and Restoration of Altarpieces, of the degree of conservation and restoration of cultural goods of the University of La Laguna), raise critical doubts about the objectivity and representativeness of the results, for any safe and useful conclusions to be drawn.

For the pilot test, students of the heritage restoration degree were selected because they are users interested in this type of tours. Due to this, the sample was only 18 participants, with the results obtained, we obtain a first assessment of the user experience.

In future works we will expand the number of users as well as the target group that may be interested in this type of virtual tour (tourists, cultural entities, students, architects, etc.) This procedure of first conducting a pilot test to expand the scope of users in subsequent works is a common procedure in research.

Round 2

Reviewer 4 Report

Comments and Suggestions for Authors

Thank you for addressing the majority of my comments. A minor: there is a missing year in reference 46.