A Virtual Reality Museum to Reinforce the Interpretation of Contemporary Art and Increase the Educational Value of User Experience

Abstract

The cultural heritage sector increasingly integrates augmented and virtual reality (VR) solutions to meet dissemination and interpretation needs for its collections. As research in the field grows, the required entertainment and learning impacts of such applications are rising. This study presents a VR museum that aims to facilitate an understanding of cultural heritage. More specifically, an exhibition was designed, curated and developed in a VR environment based on a framework that encourages the public’s interaction with the artworks and experiential learning through activities that utilize VR functionalities in a meaningful way. This framework was applied in a contemporary art museum where the description of artistic concepts is not always obvious to the general public due to the abstract forms of the artworks or the particularities of different artistic movements. This paper focuses on the application development and three user experience evaluations (museum experts, technical experts and general audience). The results were positive regarding the perceived sense of control, usability and the feelings of the user, including their sense of entertainment. Additionally, the participants valued the educational value of the developed activity types and their usefulness. Moreover, the users were interested in exploring the cultural heritage content available in the exhibition, and they would suggest the application to colleagues or friends.

1. Introduction

A lot of museums, archives, libraries and institutes managing Cultural Heritage (CH) objects have digitized and continue to digitize their collections for research, restoration and preservation purposes. In parallel, digitized artifacts are being integrated and re-used in various applications to communicate their meanings on-site and remotely. Augmented Reality (AR) and Virtual Reality (VR) applications are used to engage existing audiences and attract new ones due to the positive impact they have on visitors’ experiences [1,2]. Although both AR and VR can enhance the meaning of a CH object, AR is used in a real-life context, while VR can offer immersive experiences in new virtual worlds [3]. More specifically, AR is used in location-based applications to enhance physical exhibitions with additional material, while VR applications are used to expand existing exhibitions into virtual spaces. VR increases the building capacity of CH organizations by allowing them to display additional exhibits and allows these organizations to disseminate digital replicas of CH objects that cannot be displayed in a physical space (e.g., due to fragility, size, etc.) or that are not accessible in their original location. Moreover, as Lee et al. showed [4], an immersive VR environment can enhance a tour experience and encourage users to visit the physical space.

In the case of VR, digital replicas of 2D and 3D artifacts can be transferred into 3D worlds within unlimited simulations or game scenarios to facilitate their interpretation by users. For example, historical artifacts can be transformed into their original context of use by using objects of the past inside a 3D reconstructed monument in another age, or an art piece can be transferred to the artist’s era and social environment to highlight the artist’s intentions. CH VR applications can be classified in terms of their content into two main categories: exhibitions and historical applications [5]. The first category consists of virtual museums and art exhibitions that may be static or dynamic, displaying multiple exhibits or focusing on an artwork or artist. The second category consists of 3D reconstructed historical landmarks or buildings.

In VR exhibitions, both static and dynamic VR museum experiences, the interaction types vary, with the most common being the activation of an information 2D panel, which provides more details for the selected exhibit with multimedia content [5]. The Invisible Museum allows CH experts to create dynamic virtual exhibitions. The curators can configure the exhibits and their details and add multimedia content, and create guided tours for target groups through web interfaces. The VR visitor can navigate the exhibition through a guided tour or free exploration and by selecting an exhibit to display the provided information. What is interesting is that users can also interact with hand motions, which makes the interaction more natural [6]. In the Scan4Reco VM, the user activates the details of an exhibit, which can also display research metadata correlated with the specific areas related to the CH object. Additionally, the user can change the texture of a 3D object to display its simulated status in different time periods [7].

Another common interaction type is to grab objects by using touch controllers connected to the exhibits themselves [8] or other objects in the virtual space [9]. A VR museum dedicated to the Antikythera mechanism allows the user to grab its fragments with the touch controllers and enter their geometry to discover hidden gears and markings [8]. A timeless museum consists of multiple thematic rooms and takes into account the educational aspect of a VR museum application by providing various sensorial stimuli [9]. The user can navigate around the space, grab objects (e.g., pamphlet at the entrance), activate information panels when approaching an exhibit and hear the sounds of the surrounding environment.

The use of VR applications in museums can promote entertainment and the learning of CH [10]. Although physical museums have established their educational role by definition [11] and satisfy this role through physical exhibitions, there is still room to extend this function to VR museums [12]. As Cecotti suggests, considering learning and gaming aspects in CH VR applications can increase their learning impact with the intentional integration of learning activities and their learning impact assessment [5]. The existing work in the field of VR games for CH can contribute to this effort. As Theodoropoulos and Antoniou found [13], the intentions of VR games are not limited to entertainment, and all of the studied games can be classified as Serious Games (SGs) while having learning purposes. Moreover, those games have been reported to be an effective manner of enhancing the learning of CH and enjoyment. Thus, we suggest that the consideration of SG aspects in the design and delivery of VR museum applications might be an effective way to reinforce the interpretation of exhibits and increase the educational value of the user experience. Furthermore, in this study, we propose the integration of learning activities in a VR museum application. Moreover, the navigation and interaction typology that the presented VR museum suggests is the explorative interaction type, which, even among VR SG applications that are not used in Virtual Museums (VMs), is underrepresented [14].

At this point, it should be noted that the rigorous 3D reconstruction of the artifacts is important for delivering the intended messages and avoiding misinterpretation of the CH content. Additionally, the progress in terms of digitization methods [15,16,17] can respond to the increasing demand for highly detailed and immersive environments. Thus, CH objects and 3D environments need to be delivered using high-quality 3D models. To this end, 3D models should be balanced in terms of the intended level of detail and the number of polygons. More specifically, the 3D object should be represented economically in terms of polygons to facilitate the rendering process in the VR environment, and at the same time, it should be detailed enough in terms of geometry and texture to represent the physical object accurately.

Considering the above, this study presents a VR museum application with learning intentions. Emphasis is given to explorative interaction. More specifically, VM visitors can move freely in the virtual environment, using teleportation to avoid motion sickness, with no limitations to where they can go within the virtual space. Furthermore, the users have the ability to interact with the exhibits, utilizing the capabilities of VR, with activity types that provide different levels of interactivity and that facilitate learning [18]. The presented VR museum also focused on the rigorous representation of the CH objects in the exhibition. The results from three assessments validated the suggested VR museum set-up in terms of a sense of control, usability, usefulness, educational value and the ability to stimulate interest in the topic, and the users were also satisfied by the experience and felt positive feelings, including interest and entertainment.

2. Research Aims

The aim of this research is to assess and validate the 2gether VR museum application. The application was designed in consideration of SG factors to provide an immersive experience with increased pedagogical value. The learning intention is to encourage explorative behavior and experiential interaction in the VR museum to facilitate visitors’ understanding of contemporary art exhibits.

Based on the team’s previous work, we briefly describe the design process to explain how the proposed conceptual model was applied to the case of the 2gether VR application [18]. The conceptual model provides decision points that can support curators in expanding their curatorship in VR exhibitions. Moreover, we provide insights into how museum and technical experts can collaborate to integrate musicological and educational perspectives into VR applications so as to utilize VR functionalities in a meaningful way for the artwork and end-user.

Additionally, this paper presents how the designed experience reflected the developed VR museum set-up with details on the navigation and interaction systems, including the proposed activity types that we integrated to enhance the meaning of the exhibitions and facilitate the understanding of contemporary art.

Furthermore, we conducted three assessments of the application, focusing on museum experts, technical experts and general audiences. The assessment plan is presented, which was in alignment with the development cycles of the application, in order to timely and progressively address the elicited feedback within the timeline of the project. The results of the three assessments provide useful evidence about the impact of the VR museum experience and confirm that the application was a satisfying experience as a museum visit.

Finally, while the quality of the reconstructed artworks was appreciated by the users, we also present the process we followed to reconstruct the 3D exhibits, targeting the trade-off between the high quality of the models used for VR and polygon economy to facilitate their rendering within the game engine.

This paper is structured as follows. Section 3 starts with the factors that were considered during the design and development of the 2gether VR museum, which continues in Section 3.1 with the presentation of the VR application and the available interactive activities. Section 3.2 presents the pipeline that was followed to digitize the 3D exhibits for the VR environment. Next, Section 4 explains the assessment methodology followed, and Section 5 presents the results of the three assessments that have been conducted for the VR museum application. Section 6 and Section 7 follow with a discussion and the conclusions of this study.

3. Methodology

A VR museum is both a museum and a digital product. Thus, the delivered exhibition needs to be curated and also needs to meaningfully exploit what the selected technology can offer. In the 2gether VR museum, the first design step was to explore the VR navigation and interaction options together with the curatorship of the digital exhibition by selecting the exhibits and obtaining all of the legal licenses required to use them (Figure 1). Then, the entertainment and learning intentions were explored, and the assessment parameters were defined. This process led to the creation of a VR museum set-up to reinforce the interpretation of contemporary art and increase the educational value of the user experiences. This set-up refers to the navigation and interaction choices that were made based on the design’s conceptual framework. Finally, the development of the application consists of some key implementation tasks, such as the creation of the 3D environment, the 3D reconstruction of the exhibits, the navigation system and the implementation of the different interactive activities for the exhibits.

The design process included a significant effort to strengthen the educational value of the delivered VR experience. To do so, a series of decisions were made during the early design phase of the 2gether VR museum based on a set of conceptual factors for the design of SGs [19]. Although the 2gether VR museum is not an SG, and it was decided not to integrate game mechanics, its aim is to reinforce both learning and entertainment, two essential elements of SGs. More specifically, the 2gether team defined the parameters that applied to the museum pilot regarding the (i) learners’ specifications, (ii) learning intentions, (iii) game attributes (iv) and ways to facilitate learning in the VR experience.

Regarding learners, we decided upon a target audience of visitors above 13 years old. The curated information was intended to be presented through different types of media, such as text, sound and video, to satisfy different learning styles and preferences. Furthermore, the delivery of these media needed to vary. More specifically, the intention was to transfer information delivery types that usually take place in a physical exhibition, such as captions and guided tours, to the VR environment. At the same time, we needed to leverage the characteristics of VR, such as immersion and 3D spaces, to integrate interactions that are not possible in physical spaces. Thus, among other options, we explored the delivery of visual stimuli through 2D and 3D animations of the art pieces that enhance their meaning, allowing the users to touch the art pieces and execute hands-on activities with them, the delivery of sound information through a 3D character resembling a virtual guide and facilitating the visitors in focusing their attention on specific details of the art pieces with an interactive activity focused on the details of the artwork.

Defining the learning intentions was an important collaborative task and was carried out by technical and museum experts, and it was considered essential to establish a common understanding about how to apply a pedagogical model of CH in the VR application. Firstly, the pedagogy, aligned with the museum’s educational policy, focused on cognitive learning, which is based on the experience and can lead to the formation of a new cognitive model for the learner through experimentation and reflection [19,20,21,22]. The context of the use of the VR application was determined to be a visit to the contemporary art museum during a school visit or as part of a free time activity. A dedicated spot inside the physical exhibition was formed to host the 2gether VR application, namely the “VR spot” (Figure 2). In this spot, two VR-ready laptops and a PC were installed, which were available for the public and supported by trained museum staff.

The learning content for the application was established by art historians and museum educators who documented the information to be delivered to the audience for each selected exhibit. This could include, for example, who is the artist of an artwork, historical information concerning the artist’s era, the artistic movement and the socio/economic environment, the meaning of the artwork and what the artist intended to communicate to the public, the techniques that were used and how the meanings are being visualized on the art piece. Moreover, for each exhibit, the intended skills were documented as the expected outcome of the visitor’s interaction with the art piece, answering the question—“What do we expect the user to learn through the interaction with the exhibit?”. Overall, the aim was for the visitors to reflect on the message of an art piece and to understand the features of artistic movements, the depiction of the human body in different eras, and the use of shapes, lines, colors, shades and lighting to convey a message, the level of abstraction in human forms in different artistic movements, etc. The indicative learning objectives that combine the intended skill and the learning content are presented in

Table 1. Indicative learning objectives for the VR museum exhibits.

Artwork	Learning Objective [19,23,24]
	Intended Skill	Learning Content
Liubov Popova—Study for a protrait ©MOMus–Museum of Modern Art–Costakis Collection	The visitor to detect…	…the human face inscribed in the central triangular form.”
Aleksandr Rodchenko—Construction on White (Robot) ©MOMus–Museum of Modern Art–Costakis Collection	The visitor to recognize…	…the use of geometric forms in a composition, as one of the primary characteristics of constructivism.
Nikritin Solomon—Man in top hat ©MOMus–Museum of Modern Art–Costakis Collection	The visitor to perceive…	…the use of the dark colours, and the undefined facial features, as the depiction of the surrounding atmosphere and feelings during the war.

.

The 2gether VR museum, as mentioned before, despite being developed in a game engine, does not integrate game attributes, such as a specific game genre, game mechanics or game goals. However, what the 2gether VR museum incorporates is a narrative structure, which reflects the curatorship conducted in the virtual exhibition. More specifically, the exhibition is about the human body and how it is depicted in modern art from the late 19th century until the present day. The 2gether team documented the exhibition concept and the staging of the exhibits in chronological order. The game activities are translated into the interactive activity types that were designed for the VR museum. The activity types are presented in Section 3.1. Additionally, for the 2gether VR museum, three interactivity levels were defined (low, medium and high) depending on the amount of engagement that is needed by the user in the available activity types [18]. Another important factor considered was a sense of control. To increase the confidence of the user while using the equipment (Head-Mounted Display (HMD) device; touch controllers) and while interacting with the interfaces of the application, a tutorial activity was integrated to allow the user to practice and gradually obtain control over the application. The graphical user interface was intended to have a realistic look and feel to increase the level of immersion and, at the same time, integrate 2D interfaces that can exploit the possibilities that the game engine provides. In this way, it would be possible to allow the user to interact with the application (i) by directly grabbing 3D items in the 3D environment with the touch controllers and (ii) by pressing buttons on 2D panels inside the 3D space to access additional 2D content. Finally, usability aspects are considered to be major factors for any software application.

To facilitate learning, focused attention was applied to the exhibits and their details. First of all, designing the user experience and the graphical interface was considered to be of high importance in providing any additional information to the user without distracting the user’s attention from the artwork. At a deeper level, the activity types were designed to enable the user to focus their attention on specific details of the art pieces to understand the particularities of different artistic movements, e.g., colors, shapes, form, shading, etc. Specifically, in the context of a contemporary art exhibition, this factor was considered to be important as the forms are visualized with various levels of abstraction and complexity, and the message is not always obvious to the visitors. Additionally, based on experiential learning [25], hands-on activities were designed for the art pieces, wherein the user can interact with the 3D space by grabbing items with the touch controllers. During the hands-on activities, the user receives feedback from the environment and adjusts the response to reflect on the meaning of the artwork. The aim is to allow the user to interact with the artworks and focus on recognizing the abstract or cubistic human forms present in the artworks by understanding the different shapes, curves, lines, colors, shading and lighting and how they synthesize the overall picture. The hands-on activity depends on the artwork, and the task can be to solve a 3D puzzle of a 2D artwork or to move/synthesize parts of a 3D artwork [18]. The intended level of entertainment for the 2gether VR museum is not the level that one would seek in a game but rather lower.

Moreover, the assessment parameters were defined. In the case of 2gether, the application was perceived as a piece of software, a product, and a museum experience, and these perceptions were reflected in the selected assessment metrics. Firstly, the usability of a system is critical in the assessment of any digital application [26,27]. Additionally, it was important to evaluate the sense of control [28,29] over both VR hardware and software, as the audience might not be familiar with using VR. The results relating to usability and a sense of control would allow us to detect difficulties concerning usage (navigation, interaction, etc.) and adjust the experience accordingly. Secondly, the VR museum was assessed as a product that elicits overall satisfaction for the customers [30,31]. Thirdly, the application was assessed as a museum experience with entertainment and learning intentions. More specifically, we assessed how the integrated activity types were perceived in terms of their usefulness and their educational value. The participants were also asked open-ended questions relating to the weaknesses and strengths of the experience. In addition, we assessed the potential of the application to increase participants’ interest in contemporary art. Finally, the users were asked about their feelings during the experience, with the aim of describing the entertainment aspects as well as other positive or negative feelings.

3.1. The VR Museum Set-Up

The virtual museum consisted of two virtual spaces, the foyer and the exhibition. In the foyer, a tutorial activity takes place for the user to practice using the navigation system and interacting with the 3D environment. In the tutorial scenario, the user is asked to issue a ticket to enter the exhibition. To do so, the user should execute some steps to become familiar with using the HMD device and the touch controllers (Figure 3) (https://www.dropbox.com/sh/g91z5lt52qfsfc0/AADkIxY6VxsR1kQK5iRBSGMta?dl=0, accessed on 15 February 2023). Each tutorial step has an incremental difficulty scale for an amateur user that culminates in the combination of moving and grabbing. Firstly, the user practices selecting an option on the 2D interface. Secondly, the user is asked to look around to feel the sense of rotating their head while wearing the VR HMD device. Then, the user is asked to move in front of the ticket machine, print a ticket and grab it. Finally, the user should move to the terminal to insert the ticket and enter the exhibition.

Once the tutorial is completed, the user can navigate inside the exhibition and interact with the environment on their own. There is also the option to skip the tutorial and directly enter the museum.

After completing or skipping the tutorial, the user enters the exhibition (Figure 4). The 3D exhibition is not a representation of a physical space; rather, it is an expansion of the MOMus physical museum that does not exist in the real world. The exhibits are located either on the walls of the museum or in showcases. The visitor moves over a Π-shaped path from the entrance to the exit. Regarding the lighting, Unity’s High-Definition Render Pipeline (HDRP) [32] was used to increase the sense of realism for the end-users. The light sources are distinguished as central lighting, light coming from the windows, and the individual spotlights that illuminate each exhibition separately.

Regarding navigation, the intention was to encourage exploratory behavior [19], and the original plan was to allow the user to freely navigate around the 3D space. In the beginning, the team of museum experts was asked to test the navigation system in similar applications. It was noted that some users felt dizziness when they moved around freely with the touch controllers. This could make the experience uncomfortable and result in users quitting the VR experience too soon. For this reason, we integrated a teleporting navigation system, which allows the users to move at any point in the 3D space (Figure 5). When the user is in position A, they can press the trigger on the touch controllers to display a blue arrow on the floor. The user moves their hand around and releases the trigger to be teleported to the blue arrow.

The curated information of the exhibition is delivered through different types of stimuli (video, text, sound and animation), which the user can access through four types of interactive activities with the art pieces: focused details, virtual guides, 2D/3D animation, and hands-on activities. When the user is close to an artwork, the exhibit menu is activated, displaying the artwork’s caption and the available interactions (Figure 6). The user can read captions related to the exhibit. Additionally, the user can select “Details” to activate the focused details option, “Guide” to activate the 3D virtual guide that presents the art piece, “Animation” to bring the artwork to life and “Hands-on” to execute a task, by selecting items with the touch controllers.

Focused details aim to enable the visitors to focus their attention on specific points of interest that are highlighted in the artwork (Figure 7). These points consist of an image that is a visual detail of the exhibit and a relevant description to reinforce the understanding of the creator’s choices regarding the use of colors, forms, shading, lighting, etc., to deliver his/her message.

Two-dimensional/three-dimensional animation scenarios were used to enhance the meaning of the artworks to facilitate understanding (Figure 8). Two-dimensional animations were used for the paintings, and 3D animations were used for the sculptures. When the user triggers the animation of a painting, a 2D video of the artwork is displayed on its canvas. When the user triggers the animation of a sculpture, the 3D art piece comes alive with 3D movements. Once the amination is complete, the artworks assume their original forms and the user can replay the animations. For the exhibits where the animation activity is enabled, there is an additional text that clarifies that the animation is not part of the original artwork by the artist but rather an intervention concerning its digital copy. This was integrated to avoid communicating misleading messages to the visitors concerning the artworks:

“The movement attributed to the exhibits is not part of the original work of the artists, but an intervention to their digital copies, which has been implemented experimentally with all legal permissions under the 2gether project.”

Indicatively, the animation of Fassianos’ painting focuses on the wind that blows around the two depicted lovers (Figure 8). The wind moves the figures’ hair, the wheat that the woman holds and the red scarf that the man holds while the sound of the wind plays. At the same time, the heads of the figures move closer to highlight their intimate relationship. When the animation of Coulentianos’ artwork plays, the female figure stretches her back and moves her head upwards to facilitate the visitor’s understanding of the abstract body synthesis and the sitting pose.

When the user triggers the virtual guide option, a 3D character appears to present information relating to the exhibit (Figure 9). An audio file plays alongside a face-captured animation and various recorded body/hand motions that take advantage of Unity’s Mecanim Animation system [33].

Hands-on activities allow the user to grab parts of the artwork and execute a task to facilitate observing and understanding the composition. Indicatively, in Moralis’ artwork, the user solves a puzzle with three pieces that represent one of the two human bodies represented in the painting (Figure 10). This task enables the user to distinguish the two entangled abstract human bodies from each other, while observing the distinctive way that Moralis uses straight lines and curves to build his forms.

3.2. Three-Dimensional Reconstruction of Exhibits

AA thorough representation of the CH objects was essential for the VM. The 2D artworks were digitized by applying high-resolution images of the exhibits as textures to the digital canvases. The digitization of the 3D artworks required effort to succeed a trade-off between high quality and the number of mesh polygons used in the 3D models. The 3D models needed to be an exact replica of the original artwork from a museology perspective, and, at the same time, the number of polygons needed to be as low as possible. Photogrammetry was used as the most appropriate technique to extract an accurate digital copy. First, we took pictures of the art piece from different perspectives, and then we inserted them into the Reality capture program to extract a high-quality model (Figure 11). For Coulentianos’ artwork, 329 high-quality photos were used, while in Oppenheim’s, 213 were used (.jpg, 3024 × 4032 px). Then, we significantly decimated the output 3D model for the sake of polygon economy.

The next step was to conduct retopology in order to clean the mesh of unwanted polygons and smooth out the surfaces and photogrammetry errors. The final step was to paint the texture of the edited polygons. In the case of Coulentianos’ artwork, rigging was added to the model to enable the animation. In Oppenheim’s artwork, a disco ball 3D model was added at the end, which could not be captured by the photogrammetry due to its reflective material (Figure 12). The final step was to import the 3D models into the Unity game engine.

4. Assessment Methodology

The impact of VMs and VR applications in museums is being assessed from different perspectives, from technology and usability aspects to the overall museum visitor experience, including satisfaction, engagement, entertainment and learning [9,10,13]. Related studies usually examine VR applications in a museum visit context regarding their ability to enable visitors to learn something new, to offer them a pleasant (aesthetic), engaging and entertaining experience and to make them feel immersed or that they are somewhere else (escapism) [1,2,4]. Moreover, studies understand VR and other technologies as part of the museum experience, and this is reflected in their assessment, evaluating the overall museum experience and learning effects as well as the usefulness of the technology in terms of learning and presenting the CH content [34,35]. Additionally, VR applications are usually assessed by the visitors regarding their experience, and as Shehade and Stylianou-Lambert noticed, there is little considered regarding the point of view of museum professionals in terms of optimizing the design and development of such applications [36].

In our case, the assessment of the application in terms of CH content and usability was a priority as it was the first prototype. Firstly, it was required that the museum experts assess and adjust any content from the beginning to avoid the delivery of misleading CH information to the public. Secondly, it was required that the user experience and technology experts assess and adjust any interaction or navigation mechanisms to avoid the delivery of an application with usability issues or that is not sufficiently playable or enjoyable. Both these assessments are required to be completed before the application is released to the targeted audience. A generic usability tool would be suitable for the evaluation of an application for the first time, and the received feedback would be adequate for the detection of any major usability issues during this phase. Due to the fact that we did not use data collection in the application, we did not obtain digital input related to the user experiences during the sessions. For this reason, the moderator observed and took notes for each exhibit during the sessions. Additionally, we included open like and dislike questions to allow the user to also provide specific comments from expert or visitor points of view. Specific questions were designed regarding the usefulness and educational value of the activity types that were used, and they were related to the main features that needed to be validated by the pilot during this phase. Finally, considering the VR experience as part of a museum visit, we needed to assess whether the delivered application triggers positive feelings, including entertainment, as well as whether the visitors are satisfied with the experience that the museum offers. Moreover, although VR technologies are becoming more and more accessible and are continuously used by more people, there was a concern that a major percentage of the targeted audience might not be familiar with VR technologies. For this reason, we needed to assess to what extent the visitors feel that they have control over the VR equipment and interactions.

Subsequently, the 2gether VR museum was evaluated in three phases within the project by three different groups: museum experts, technical experts, and general audiences. The order in which the pilot assessments were carried out was prioritized based on the desired expected results that were deemed necessary to improve the application. First, the overall application was assessed by the museum experts in terms of the cultural content and how this would be approached by the VM visitors. Secondly, technical experts tested the application to detect bugs and provide expert opinions to improve user experiences by utilizing the available VR navigation and interaction technical options. Finally, an improved version was released to general audiences to elicit feedback from their point of view as the end-users of the application. Thus, the first two assessments were conducted with a pre-released version of the application, v0.0. Then, changes were made based on the feedback received and the errors detected (bugs). The result was the first released version of the application v0.1, which was tested by the general audience.

The assessment scenario for the three user groups was as follows:

In the beginning, each participant reads an information sheet and a form including details, such as the aim of the project, the purpose of the specific survey, the procedure of the assessment, the processing of any personal data, etc. Indicatively, the users are informed about the volunteering participation, their right to withdraw at any point without providing justification, the anonymous collection of any personal data and that the data will only be used for the purposes of this piece of research. A moderator is available to provide any clarification needed; furthermore, contact details are provided to the users if they wish to obtain any additional information. Finally, the users are given some time to decide whether to participate or not.

At this point, the moderator verbally asks for the participant’s permission to guide him/her by touching his/her shoulders if needed during the testing for safety reasons (e.g., in case they leave the VR interaction area and there is a risk of bumping into other people or nearby structures, stumbling on the cable of the device, etc.). Once they have agreed to participate in the research, the visitor wears the HMD device and the touch controllers with support from the moderator.

Next, the participant tries the application at their own pace, and the moderator observes and provides support as needed. At the start of the virtual experience, the user is in the museum foyer practicing the tutorial activity. During the tutorial, the user is introduced to how to use the HMD device and the touch controllers to navigate around the virtual space and interact with the environment. Secondly, the user enters the exhibition, navigates around the virtual space and interacts with the exhibits. The user tries each activity type that is available for the art pieces, namely (i) the virtual guide, (ii) the 2D animations, (iii) the 3D animations and (iv) the focus details. It must be noted that the hands-on activity type is available in the current version of the application but was not available at the time of the assessment. Thus, the users did not use the hands-on interaction option with the exhibits, although they did practice grabbing items during the tutorial. Finally, when the user has tried at least one activity type, they are informed that the test has finished. In the cases where a participant wants to stay longer in the VR experience, they can continue to explore the exhibition and interact with the rest of the art pieces at their own pace.

After testing, the user fills in a questionnaire consisting of three sections: (i) personal information and previous experience, (ii) the experience using the application and (iii) perceptions relating to the application. The questionnaire was similar for the three assessments, with some adjustments depending on the target group. More specifically, the questionnaires assess the perceived sense of control, the usability of the system and the overall satisfaction. Additional question items refer to user opinions regarding the activity types. The general audience was asked to rate the usefulness of each activity type, while museum experts were asked to rate their educational value. Additional feedback was elicited through open-ended questions concerning the weaknesses and strengths of the application. Finally, the users were asked about their motivation to explore more contemporary art and the feelings that they had during the VR experience.

4.1. Museum Experts’ Assessment

The first assessment was conducted on museum experts that work within the MOMus museum network outside of the 2gether project and that have expertise in the virtual exhibition’s cultural content, museology and museum education, e.g., art historians, curators, educators, etc. An open call invited people to participate voluntarily, and six of them were interested in participating. The purpose of this research was to initially evaluate the application by experts in the museum’s domain and their participation in the design of the VR application. More specifically, the feedback from the museum experts concerned the overall experience, focusing on the correctness of the content relating to the art exhibition (works, texts, etc.) and their opinion on how the visitor accesses this content physically and cognitively through the available interactions. As they were the first users to test the application outside of the team of developers, the intention was to also detect difficulties in terms of use and possible software errors.

During the session, the research moderator observed the users and took notes (e.g., user comments, difficulties in use, etc.). Then, the individuals who tested the application responded to a semi-structured interview that included predetermined questions related to background information (expertise and age group) and the perceived user experience. In order to better understand the experts’ feedback, the moderator asked follow-up questions when needed.

Regarding the questionnaire, in the first section, the users were asked about their age (Question—ME—Q1.1) and expertise (ME–Q1.2) (

Table 2. The questions used in the three assessments.

	Museum Experts (ME)	Technical Experts (TE)	General Audience (GA)
ME, TE, GA—Q1.1	Age group
ME, TE—Q1.2	Domain of expertise
ME—Q1.3–1.4, TE—Q1.3–1.5, GA—Q1.3–1.4	Previous experience with - Using virtual reality mask and controllers - Using similar virtual reality applications that feature cultural content	Previous experience with - Creating virtual reality applications - Using virtual reality applications - Creating applications that feature cultural content (in any technology, e.g., Unity, AR, VR, mobile, web, etc.)	Previous experience with - Using virtual reality masks and controllers - Using similar virtual reality applications that feature cultural content
TE—Q1.6, GA—Q1.3		Personal interest in cultural heritage and/or contemporary art
ME, TE, GA—Q2.1.1–2.1.3	I felt that I had the control of using the equipment (mask, controllers).
	I felt I had the control of navigation in the virtual space (teleport).
	I felt I had the control of using the 2D graphical interfaces
ME, TE, GA—Q2.2.1–2.2.10	I think that I would like to use this system frequently
	I found the system unnecessarily complex
	I thought the system was easy to use
	I think that I would need the support of a technical person to be able to use this system.
	I found the various functions in this system were well integrated (e.g., tutorial, virtual guide, artworks functionalities)
	I thought there was inconsistency in this system
	I would imagine that most people would learn to use this system quickly.
	I found the system difficult to use.
	I felt confident using the application
	I needed to learn a lot of things before I could get going with this system.
ME, TE, GA—Q2.3	How likely is that you would recommend the 2gether application to a colleague or friend?
ME—Q3.1	Would you use this application as a complementary tool in a tour or educational program?
ME—Q3.2.1–3.2.3, GA—Q3.1	To what extent do you think the following add educational value to the application? Virtual guide; focused details; animation		How do you assess the different functions of the virtual museum in terms of their usefulness to the overall experience? Virtual guide; focused details; animation
ME—Q3.3, TE—Q3.1, GA—Q3.2	What did you like about the application?
ME—Q3.4, TE—Q3.1, GA—Q3.3	What did you not like about the application and you would suggest to be different?
TE—Q3.3		To what extend the application trigger your interest in the exhibits or in contemporary art?
GA—Q4.1			To what extend the application trigger your interest in the exhibits or in contemporary art?
GA—Q4.2			When I tried the app, I felt more (multiple choice): Enjoyment/Entertainment; interest; satisfaction; admiration; disappointment; boredom; fatigue; anxiety; other

) [Appendix A]. Moreover, they were asked about their previous experience by answering two 5-point Likert scale questions (1: not at all–5: very much). The first question was related to using the VR equipment (ME—Q1.3), and the second question was related to using similar applications that feature cultural content (ME—Q1.4).

The second section consisted of three sub-sections. The users were asked about the perceived sense of control that they felt during the test [28,29]. Specifically, they responded to three 5-point Likert scale questions (1: strongly disagree–5: strongly agree) on whether they had a sense of control over (i) the equipment (ME—Q2.1.1), (ii) the navigation mechanisms (teleportation) (ME—Q2.1.2) and (iii) the interaction with the graphical elements of the artworks (exhibit menu) (ME—Q2.1.3). Next, the System Usability Score (SUS) assessment was used to draw conclusions relating to the perceived usability of the system [26,27]. This tool consists of ten questions, to which participants respond with a value on a 1–5 scale (5-point Likert scale: 1: strongly disagree–5: strongly agree) (ME—Q2.2.1–Q2.2.10). Moreover, the Net Promoter Score (NPS) assessment was used to extract the overall satisfaction of the participants [30,31]. More specifically, the users were asked to answer whether they would recommend the application to a colleague or friend by providing a response from 0 to 10 (ME—Q2.3). According to the NPS, participants who answer from 0 to 6 are considered to be dissatisfied customers (detractors), those who answer 7 or 8 are considered to be neutral (passive) and those who answer 9 or 10 are considered to be loyal customers (promoters). The final NPS score is calculated by subtracting the percentage of dissatisfied customers from that of satisfied customers (NPS score = promoters % − detractors %).

The third section assessed the user perceptions concerning the application from their expert point of view. More specifically, the museum experts were asked whether they would use the application as a complementary tool in a tour or educational program by selecting an answer from a 5-point Likert scale range (1: strongly disagree–5: strongly agree) (ME—Q3.1). Then, they rated the educational value of the virtual guide (ME—Q3.2.1), focused details (ME—Q3.2.2) and the animations (ME—Q3.2.3) by using 5-point Likert scale questions (1: not at all; 5: very much). It should be mentioned here that the hands-on feature was not evaluated at this stage as it was not integrated into the exhibition at the time of this assessment. However, the users could practice hands-on interactions in the tutorial activity.

The main objective of this assessment was to find weak points to improve the application while the application was still in the development stage. Therefore, the important results were those obtained from the moderator’s observations during the test, where difficulties in terms of use, bugs and the participants’ thoughts (think out loud) were noted and also taken from the two open-ended questions regarding what they liked and disliked about the application and what changes they would suggest (ME—Q3.3, ME—Q3.4).

4.2. Technical Experts Assessment

The second pilot group to use the VR museum assessment consisted of technical experts in the field of VR museum applications. The purpose of this research was to receive feedback regarding the technical aspects in order to improve usability when utilizing the navigation and interaction options in the VR application. The participants tested the application while thinking out loud, and then they filled in a questionnaire concerning their perceived experience and provided additional comments with open-ended questions related to the weaknesses and strengths of the application.

Similar to the previous assessment, the users were asked about their age (TE—Q1.1) and expertise (TE—Q1.2) [Appendix B]. Moreover, they were asked about their previous experiences by answering three five-point Likert scale questions (1: not at all–5: very much). These questions were related to creating and using VR applications (TE—Q1.3, Q1.4) and creating applications that feature CH content with any technology (TE—Q1.5). An additional question was asked to the technical experts concerning their interest in CH and/or contemporary art (TE—Q1.6).

The second section of the questionnaire relating to the participants’ perceived experience (TE—Q2.1–Q2.3) was the same as that used for the assessment by museum experts (Section 4.1).

As the main objective of the test was to find the strengths and the weaknesses of the application and to make improvements, the users answered the same open-ended questions concerning their likes and dislikes and were encouraged to provide suggestions (TE—Q3.1, Q3.2). Finally, they were asked a question that used a 5-point Likert scale on whether the application triggered their interest in the exhibits or in contemporary art (TE—Q3.3).

4.3. General Audience Assessment

The third pilot group of the VR museum was the general audience, the visitors of the MOMus exhibition, and they tested the improved version, application v0.1, which integrated the updates based on the two previous assessments. The visitors of the museum were asked to try the application in the VR spot, and they were informed that they could also provide their feedback after their session. Sixteen of the visitors who tried the application also provided feedback. The purpose of this assessment was to validate the updated version of the application with the defined target group, which is a general audience of people above 13 years old.

The participants were asked about their age (GA—Q1.1) and their previous experience using VR (GA—Q1.2), the use of VR applications that feature cultural content (GA—Q1.3), and their interest in CH or contemporary art (GA—Q1.4) (Table 2) [Appendix C].

The second section of the questionnaire relating to participants’ perceived experience (GA—Q2.1–Q2.3) was the same as that used in the two previous assessments (Section 4.1 and Section 4.2).

In the final section, they rated the usefulness of the virtual guide (GA—Q3.1.1), focused details (GA—Q3.1.2) and animations (GA—Q3.1.3) with 5-point Likert scale questions (1: not at all, 5: very much) regarding the overall experience.

Similar to the previous assessments, they were asked to provide their likes and dislikes in open-ended questions (GA—Q3.2, Q3.3).

Moreover, the participants were asked to rate, from 1 to 5, whether the application triggered their interest in the exhibits or contemporary art (GA—Q4.1). Finally, they were asked to choose the most prominent feelings they felt during the session from a list of positive and negative feelings (enjoyment/entertainment, interest, satisfaction, admiration, disappointment, boredom, fatigue and anxiety) (GA—Q4.2). They had the option to fill in additional feelings that were not listed.

5. Results

The following sections present the set-up of each user assessment and the respective results in detail.

5.1. Museum Experts’ Results

The individuals who participated in the trial were mainly experienced cultural professionals, such as curators, art historians and museum educators (ME—Q1.2). Regarding age, three individuals were in the 45–54 range, one individual was in the 25–34 range, and another was over 55 (ME—Q1.1). The participants had “not at all” or “a little” experience using a VR mask and touch controllers (ME—Q1.3). They had almost the same experience in terms of using similar applications that feature cultural content (ME—Q1.4). Only one user stated that they had a “moderate” experience of using similar applications. No user had “much” or “very much” experience.

The perceived experiences were generally positive. In detail, in the first three questions (ME—Q2.1.1–2.1.3), the participants answered questions relating to their sense of control during the testing session. Specifically, they responded positively (“agree” or “strongly agree”) to whether they felt that they had control over the equipment, navigation and interaction with the graphical elements of the artworks. Mostly positive responses were also provided to the questions relating to the usability of the system (ME—Q2.2.1—2.2.10). Furthermore, the users responded positively to the question concerning whether they would recommend the application to a colleague or friend, providing a score of 9 or 10, which means that the users were satisfied, being classified as promoters. Although the sample is very small to draw solid conclusions, the aforementioned results provide overall positive perspectives concerning the sense of control, usability and the satisfaction of the users, validating that the technology team was on track and that their work aligned with the needs of the museum.

More noteworthy results from this assessment are participants’ perceptions concerning the educational value of the application. In detail, they responded that they would use the application as a complementary tool to a tour or educational program, with four indicating “strongly agree” and two indicating “agree” (ME—Q3.1). Moreover, they replied to whether they think the different features (virtual guide, focused details and animations) add educational value to the application (ME—Q3.2) (Figure 13). The results showed that, in order of preference, the focused details had the highest score (29), followed by the animations (26) and the virtual guide (23).

The users liked the virtual museum environment (space and aesthetics) and that the information concerning the artworks is presented in different ways to cover the different learning preferences of the potential visitors (

Table 3. Strengths of the application (what museum experts liked, ME—Q3.3).

User	Comments
U1	“The overall environment; the options given to the user; [...] I really liked the graphics and the analysis of the art works”.
U2	“I liked [...] the architecture of the space. I also really liked the presentation of the works either on the showcases or on walls [...] I was excited by the animated presentations, I liked the script and the invention related to the properties of the art works, their morphological elements”.
U3	“Discreet and correct movement of the animations”.
U4	“The animations, the exhibition as a whole (and the space) the possibility of many options in the tour”.
U5	“The movement adds value to the aesthetics and understanding of the art work, especially in sculpture. The artworks motion is for people who do not read, and the virtual guide adds interest for those seeking for more information”.
U6	“The environment”.

). The animated artworks were also appreciated.

Specifically, in terms of sculpture, it was considered that the movement helped in understanding Coulentiano’s artwork because it “makes you understand where the head is”, as stated by U5 during the test.

The weak points of the application are grouped in Table 4 and

Table 5. Identified weak points of the application and the respective changes to resolve them—part B.

Weak Points Comments and Observations (OBs)	Issues	Changes	Version
Focused details
U2: For Rodchenko’s artwork: activating the focused details: “which are...” OB: are activated but not in the field of view.	When I click the “Details” button, I want to know that they are activated.	The focused details are always activated in the user’s field of view while the user is standing in the optimal viewing position. A sound accompanies the activation of the focused details.	0.1
Animation
U1: For Coulentiano’s artwork: “I did not notice that the animation was playing—I was expecting something to appear here (in the panel), viewing the sculpture animated”. U6: For Coulentiano’s artwork: in the animation: “Oh the exhibit is what I had to look at”.	When I press the “Animation” button, I need my attention to be focused on the artwork.	Improved positioning of graphic elements when the user is standing in the optimal viewing position	0.1
Virtual guide
U4: For Rodchenko’s artwork: “the voice of the guide is very robotic”. U5: “The guide was very robotic”.	The guide should be more natural.	Add a human voice to the virtual guide instead of the text-to-speech currently used. Add Greek.	Next Version

, as summarized in the moderator’s notes during the test (observations and thinking out loud) and the open-ended questions, ME—Q3.4. In the first column, the user comments are presented. These comments were converted into issues in the second column. The third column displays the changes made to resolve these issues. Regarding the cultural content, museum experts made specific suggestions in terms of changes to the texts used for the artworks. The changes and the updated cultural content were incorporated into the next version of the application (v0.1).

Table 4. Identified weak points of the application and the respective changes implemented to resolve them—part A.

Weak Points Comments and Observations (OB)	Issues	Changes	Version
Tutorial and hands-on
U1: “a bit difficult in my opinion - task to issue a ticket and validate it at the entrance (I just think in the beginning, you need to get familiar with the controllers, it seemed difficult as a process)”. OB: In the tutorial, lost the ticket two times. U2: reading the instructions out loud: “reach out your hand”—umm which one?”. For Katraki’s artwork: “I tried to do it with this hand too, but it did not work”. U3: “Maybe a shorter introduction”. U4: In the tutorial: follows the instructions, “now how do I grab this?”, reads the instructions, “I did not grab it, why do not I grab it?”, “I forget how to move forward”. After completing the tutorial, “Finally”. OB: Dropped the ticket. OB: The 2D panels that contain the instructions, in some cases, were out of the users’ field of view. Excessive instructions.	I want it to be easier to grab the ticket. Once I grab the ticket, I do not want to lose it so easily (user error protection). Functions should be available on both controllers for left and right-handed visitors.	The user grabs the ticket with the controller and does not need to keep pressing the button while moving to the exhibition entrance. The grab function became available on both controllers (left and right). Simplified instructions for more independent first use. Less text, more clarity. Improved positioning of the 2D panels that contain the instructions.	0.1
Navigation-teleporting
U1: For Popova’s artwork: “Every time I want to move away (from the exhibit) I will use this lever and step back?”. For Nikritin’s artwork: “I think that is the distance (from the work) that I like.” U2: “This process with the lever is difficult for me (in navigation)”. U3: “I want to go back, can I?” U5: “to get in front of them (the exhibits) you have to do a little bit of effort.”	I want to be able to move easily to a good viewing position for the artworks	The user continues to use teleportation. An optimal viewpoint was defined for each artwork where the user is automatically transported to view an exhibit (Figure 14).

Table 4. Identified weak points of the application and the respective changes implemented to resolve them—part A.

Weak Points Comments and Observations (OB)	Issues	Changes	Version
Tutorial and hands-on
U1: “a bit difficult in my opinion - task to issue a ticket and validate it at the entrance (I just think in the beginning, you need to get familiar with the controllers, it seemed difficult as a process)”. OB: In the tutorial, lost the ticket two times. U2: reading the instructions out loud: “reach out your hand”—umm which one?”. For Katraki’s artwork: “I tried to do it with this hand too, but it did not work”. U3: “Maybe a shorter introduction”. U4: In the tutorial: follows the instructions, “now how do I grab this?”, reads the instructions, “I did not grab it, why do not I grab it?”, “I forget how to move forward”. After completing the tutorial, “Finally”. OB: Dropped the ticket. OB: The 2D panels that contain the instructions, in some cases, were out of the users’ field of view. Excessive instructions.	I want it to be easier to grab the ticket. Once I grab the ticket, I do not want to lose it so easily (user error protection). Functions should be available on both controllers for left and right-handed visitors.	The user grabs the ticket with the controller and does not need to keep pressing the button while moving to the exhibition entrance. The grab function became available on both controllers (left and right). Simplified instructions for more independent first use. Less text, more clarity. Improved positioning of the 2D panels that contain the instructions.	0.1
Navigation-teleporting
U1: For Popova’s artwork: “Every time I want to move away (from the exhibit) I will use this lever and step back?”. For Nikritin’s artwork: “I think that is the distance (from the work) that I like.” U2: “This process with the lever is difficult for me (in navigation)”. U3: “I want to go back, can I?” U5: “to get in front of them (the exhibits) you have to do a little bit of effort.”	I want to be able to move easily to a good viewing position for the artworks	The user continues to use teleportation. An optimal viewpoint was defined for each artwork where the user is automatically transported to view an exhibit (Figure 14).

Six bugs were also identified during the testing, which were resolved in the next version of the application (

Table 6. Identified Bugs.

Identified Bugs
U5: “got inside” to the ticket machine	Version 0.1
U1: OB: In the exhibition: the introductory panel is not closing properly
U1: For Rodchenko’s artwork: in the focused details: relating to the animated graphical element on the 2D panel: “I thought it was something that loads” U4: “What is that? because it’s like waiting for something to appear” U4: “(I did not like)…the circles above the information/details”
U4: For Mylona’s artwork (Alex Mylona—Angel II ©MOMus): concerning a panel that is not closing properly: “is not that confusing; that is permanently open?”
U4: Concerning missing clarification text relating to the animation used for an artwork: “I needed to read this in the previous artwork as well”
U1: For Fassiano’s artwork: concerning a black frame at the end of the 2D video animation: “here when the screen went black it was very obvious.”

).

5.2. Technical Experts Results

In total, seven experts participated in the trial. They were the developers of VR and AR applications, developers that use the Unity game engine and programmers (TE—Q1.2). Almost all of them belonged to the age group of 25–34, and one was in the age group of 45–54 (TE—Q1.1).

Two of the participants had no experience with developing VR applications (TE—Q1.3) and no experience in using them (TE—Q1.4) but had “much” or “very much” experience in creating applications that feature cultural content (in any technology, e.g., Unity, AR, VR, mobile, web, etc.) (TE—Q1.5).

Three of the respondents had very much experience in creating (TE—Q1.3) and using VR applications (TE—Q1.4), with two of them also having very much experience with applications that feature cultural content (TE—Q1.5). The third respondent had no experience with cultural content applications (TE—Q1.5).

Two more users had some experience with creating VR applications (TE—Q1.3) and more experience with using such applications (moderate and much) (TE—Q1.4). One of them also had moderate experience with developing applications that feature cultural content, and the other had very much (TE—Q1.5).

The participants’ interest in CH and/or contemporary art ranged from “a little” to “very much” (TE—Q1.6) (Figure 15).

The participants’ sense of control when using the app was high. More specifically, all users felt that they had full control over the equipment (mask and controls) (“strongly agree”, TE—Q2.2.1). Regarding the navigation, they felt that they had control, with six of the users stating “strongly agree”, and one stating “agree” (TE—Q2.2.2). In terms of the level of control during the interaction with the artworks, five users felt strongly that they had control over the interactions, one user stated that they had control, choosing “agree”, and another user felt that they had a “moderate” level of control (TE—Q2.2.3). The sample is not large enough to generalize the conclusions, but it could be said that the experienced users had full control over the equipment (TE—Q2.2.1), and if anything was more difficult for users to manage, it was navigation (TE—Q2.2.2) and even more so for interaction with the interfaces (TE—Q2.2.3).

Usability was rated positively in all usability questions. These results are not presented here because they were very positive and may simply confirm that the users are familiar with such applications.

To the question of whether they would recommend the app to a colleague, six of the users were classified as satisfied users (promoters), answering with 9 or 10, while one was classified as a dissatisfied customer (detractor), answering with 6; none were neutral (passive).

Next, they answered questions concerning their likes and dislikes, and the users mentioned usability, the aesthetics of the application, the functions available in the project (animations, focused details and virtual guide) and the usefulness of such an application (TE—Q3.1–Q3.2).

The usability of the application was mentioned by three users who stated that they liked “the ease of use”, “the movement in space, the ease of navigation with only 2 buttons.” and “teleport, general handling” (TE—Q3.1). In terms of usability, improvements were also suggested (TE—Q3.2). One important suggestion was to improve the handling of the ticket in the tutorial activity, in which the user learns how to grab objects with the touch controller, a skill that is intended to be used in the exhibition during the hands-on interactions with the artworks (Figure 16). A second suggestion was to improve the interaction in terms of the focused details functionality (Figure 17). A further improvement suggested having “some kind of directional indicator at the initial stage of the application” to help unfamiliar users understand where they need to go.

Three of the users commented on the virtual environment, positively stating that they liked “the graphical interface”, “the realism it offers...” and “the high quality (representation) of the 3D and 2D exhibits” (TE—Q3.1). Suggestions included that the museum should have “more exhibits with many rooms with categories to make the user want to visit again. Not to visit it only once” (TE—Q3.2).

Regarding the aesthetics of the application, improvements were suggested in terms of “the appearance of the text which could be more readable”. Furthermore, for the 2D integrated interfaces in the tutorial, it was suggested to make them look and feel closer to the aesthetics of the exhibition (TE—Q3.2).

The participants also referred to the artwork activities. In detail, three users commented on the animations used for the artwork. Two users stated that “the animations incorporated in the art pieces complement the interactive nature of VR [...]” and “it is also very positive is the appearance of the different animations, as it adds something different”. Finally, one user stated that they liked “the animations of the images” (TE—Q3.1), but “the animations on the objects were very simple” (TE—Q3.2).

Finally, two users stated that they liked “the (focused) details in the exhibits for their understanding” and “the existence of a virtual tour guide” (TE—Q3.1).

Lastly, users responded positively to whether the app stimulated their interest in the exhibits or in contemporary art in general (“strongly agree”) (TE—Q3.3).

5.3. General Audience Results

The age of the participants ranged from 18 to over 55 years old (GA—Q1.1) (Figure 18).

The experience of the participants in terms of using CR equipment ranged from none to moderate (93.9%: none: 43,8%; a little: 18.8%; moderate: 31.3%), while one user had a lot of experience (6.3%) (GA—Q1.2) (Figure 19). Their experiences were similar in terms of using VR applications that feature cultural content, with different classifications among “none”, “a little” and “moderate” experience (GA—Q1.3). In total, 31.3% of the participants had moderate experience with VR applications (GA—Q1.2) as well as in similar applications that feature cultural content (GA—Q1.3). Additionally, most of the users attained an increased interest in cultural heritage and or contemporary art, with 81.3% (much: 12.5%; very much: 68.8%) (GA—Q1.4).

The majority of the participants felt that they had control over the VR equipment, with an 87.5% response rate (GA—Q2.1.1: strongly agree: 62.5%; agree: 25%), and control over the teleportation system, with an 81.3% response rate (GA—Q2.1.2: strongly agree: 56.3%; agree: 25%) (Figure 20). One of the users felt that they had no control, answering “disagree” to both questions. Additionally, most of the participants felt they had control when using the 2D interfaces, with a 75.1% response rate (GA—Q2.1.3: strongly agree: 56.3%; agree: 18.8%), and a relatively large percentage was neutral, with a 25% response rate.

The usability results were generally positive. More specifically, most of the users would like to use the app frequently, with a response rate of 81.3% (GA—Q2.2.1: strongly agree: 43.8%, agree: 37.5%), and 87.5% stated that the app is as complex as it should be (GA—Q2.2.2: strongly disagree: 62.5%; disagree: 25%) (Figure 21). Regarding whether the app is easy to use, 68.8% of the users agree (12.5%) or strongly agree (56.3%), while the remaining users are either neutral (18.8%) or disagree (12.5%) (GA—Q2.2.3). When asked whether the participants felt that they would need a technician to be able to use the application, the results were not encouraging. The responses were varied, with one user strongly agreeing that they would definitely need technical support and four others agreeing. In addition, five users provided neutral responses, and only the remaining six believed that they could use the application without the support of a technician (GA—Q2.2.4: disagree: 1; strongly disagree: 5).

Regarding whether the users believed that the functions were well integrated into the system, the majority of users answered positively (87.6%) (GA—Q2.2.5), and 75% of them felt that there is no inconsistency in the application (93.8%) (GA—Q2.2.6) (Figure 22).

Moreover, 81.3% of the users believed that people would learn to use the system quickly (GA—Q2.2.7: strongly agree: 62.5%; agree: 18.8%) (Figure 23) and that it is not difficult to use, with a response of 93.8% (GA—Q2.2.8: strongly disagree: 56.3%; disagree: 37.5%). In addition, 68.8% of the participants felt confident while using the application, and the remaining 31.3% were neutral (GA—Q2.2.9). Finally, almost all of the users believed that they needed to learn a lot of things before they could use the application (GA—Q2.2.10).

Regarding the NPS assessment, the results were encouraging, as all of the participants ranged from 7 to 10 (GA—Q2.3) (Figure 24). In detail, four of the participants were neutral, reporting seven (7) or eight (8). Moreover, twelve of the participants were satisfied with the experience and stated that they are likely to recommend the application to their friends and colleagues (three stated nine (9), and nine stated ten (10)). Finally, the NPS is 31.35% (Promoters 56.25% (9) − Detractors 25% (4) = 31.25% (5)), which is considered positive as it is above 0, and this is very satisfactory considering that a percentage of 50% is considered perfect.

The results regarding the usefulness of the activity types showed that all of the interactions with the artworks were well received (GA—Q3.1). In order of preference: first, the focused detail function was appreciated (87.5%); second, animation (75%); and third, the virtual guide function (68.8%) (Figure 25).

Out of the sixteen users, eleven responded to the question “What did you like about the app” (GA—Q3.2), while the question “What did you not like about the app that you would suggest to be different?” was answered by six of the participants, with feedback and one with a positive comment (“There is nothing I did not like”) (Figure 26).

The GA—Q3.2 results were grouped by their reference to (i) the overall experience, (ii) the technology, (iii) the preferences for specific functionalities/content, and (iv) user experience. Regarding the overall experience, five users reported that they liked “everything” or that they “(liked the) overall experience” or “(liked the) virtual journey”, while one said that “it is a particularly interesting experience” and that the experience “works in favor of connecting the public, but especially the non-public, with the artworks”. Some of the comments help us to understand how the use of this technology was perceived. One user reported that they liked “the sense of physical presence and how the technology highlights the art without interfering”, while another reported that they liked “the immediacy”, which can be understood as a reference to the sense of immersion that VR technology creates. In terms of the preferences for specific functionalities/content, one user appreciated “the environment and the artworks”, while one stated that they liked “the fact that I could get close to the artworks and observe them up close”. Two users appreciated the animations used in the artwork. In addition, one user appreciated the tutorial ticket functionality as they liked “when used the ticket”. Finally, regarding the experience, one user stated that they liked “the ease of use” of the application.

The GA—Q3.3 results were grouped by their reference to (i) the overall experience, (ii) the technology and (iii) the preferences for specific functionalities/content. Regarding the overall experience, one of the users stated that “As I was very skeptical about the experience, I was very pleased and certainly as improvements are made to the system, the experience will become even more real”, while another user referred to the integration of the social nature of a visit to the VM, suggesting that there should be “be other visitors”. In terms of the technology, two users referred to the equipment. In particular, one user mentioned that they did not like “the fitting of the mask”, and another said that they did not like “the relatively heavy device”. In terms of their preferences for functionalities/content, one user stated that the “animation was a bit redundant”. Finally, one user suggested that the volume of the virtual guide’s voice should be louder.

In the next question, the participants were asked whether their experience inspired their interest in the artworks and contemporary art (Figure 27). The results were positive, with 87.5% positive responses (GA—Q4.1: 62.5% strongly agree; 25% agree).

In the final multiple-choice question, the participants stated the feelings that they felt during the test (GA—Q4.2). All of the participants stated having positive feelings, indicating high interest and entertainment (Figure 28). In order of frequency, the feelings that were noted are as follows: interest, entertainment, satisfaction, admiration, anxiety, and one more feeling, which was described as “other—excitement”. Anxiety was the only negative feeling, which was noted by two of the participants and was indicated alongside positive feelings.

6. Discussion

The initial version of the application was assessed by the museum and technical experts that did not participate in the project. The feedback received led to minor and major changes to the application. One of the most important changes was related to interacting with the touch controllers to make these interactions easier for the users when grabbing items in the 3D environment. Another important change was to define an optimal viewing position to where the user is teleported when viewing each artwork. This enabled the better positioning of the 2D graphical panels of the artwork to within the field of view of the user and reduced the possibility of a user being distracted from the artwork. Moreover, the information attached to the artworks was updated based on the input of museum experts. The strengths of the application include the ease of use, the high-quality representations of the artworks and the aesthetics of the exhibition. Additionally, the museum experts appreciated the educational value of the focused details, the animations and the virtual guide functionalities of the artworks.

The next step was to release the updated version of the application and assess its performance when accessed by end-users in the dedicated VR spot inside the physical museum exhibition. The participants belonged to a wide age range (GA—Q1.1: 18–55), and despite the fact that most of them had little to no previous experience with VR and similar CH applications (GA—Q1.2, Q1.3), their sense of control was positively rated.

The usability results were encouraging (GA—Q2.2.1–Q2.2.10), with the consistency being rated as high (93.8%). Regarding the ease of use, the results were positive but could be better: 68.8% of the participants felt that the application was easy to use (GA—Q2.2.3), and 93.8% felt that the system was not difficult to use (GA—Q2.2.8). Additionally, 87.5% of the users found that the system was not unnecessarily complex (GA—Q2.2.2). On the other hand, in terms of how easy it is for someone to learn how to use the application, the results were mixed. Although 81.3% of the participants stated that they believed that most people would learn to use the system quickly (GA—Q2.2.7), regarding their experience, only six of the sixteen felt that they could use the system without technical support (GA—Q2.2.4). Finally, almost all of the users agreed that they needed to learn a lot of things before they were able to use the application (GA—Q2.2.10). The mixed results concerning how easy it is to learn how to use the system might be an indicator that the users are asked to learn a lot during the tutorial activity, which, for unfamiliar users, can lead to a high cognitive load. A solution that can be explored is the possible extension of the application, adjusting the tutorial based on the level of previous user experience.

Overall, user satisfaction was high with NPS 31.25, which resulted from zero detractors, four neutral and eleven satisfied responses as to the extent that they would suggest the application to colleagues or friends (promoters) (GA—Q2.3).

Moreover, the results concerning the usefulness of the available artwork interactions (focused details, animation, and virtual guide) were aligned with the results of the museum experts (ME—Q3.1), appreciating them with the same order of preference (GA—Q3.1).

Summing up the likes and dislikes of the participants (GA—Q3.2, Q3.3), it can be suggested that, overall, the users had positive experiences, appreciating the environment, the exhibits and the interaction with the ticket machine in the tutorial. Some of the individual’s opinions referred to the ability of the application to attract new visitors and accentuate the art without interference. It is important to note that the use of this technology can help attract new target groups, but it is the way that the technology will be used in a way that highlights the cultural content is that which maintains the interest of visitors during the experience. This is important because one of the 2gether’s priorities was to provide interactions that highlight the meaning of the artwork presented. Despite the fact that the technology was well received, the VR mask was uncomfortable to wear for some of the users.

In terms of inspiring a participant’s interest in art, the results were positive, which was also confirmed in the questions related to feelings, where the interest of the users was rated the highest (GA—Q4.1, GA—Q4.2). The users also felt entertained. It is important to mention that two of the users experienced positive feelings alongside anxiety.

Finally, one important limitation of the 2gether VR museum, which also emerged in the user feedback, is the lack of social interaction during the museum visit, which, in a real museum, is an important factor in the perception of the museum experience. The first step could be to integrate other visitors inside the virtual exhibition as non-playable characters to make the museum appear more alive. At a higher level, the visitors could be able to interact with these characters, while, at a third level, we could discuss social VR experiences [37,38,39], which could have a significant impact on social learning in the context of a museum visit [40,41].

7. Conclusions

This paper provided a conceptual framework for the design of a VR museum application with learning intentions that focuses on encouraging explorative behavior and interaction with the artworks. Additionally, this study presented and assessed the case of the developed 2gether VM, which aims to facilitate the interpretation of contemporary art through VR experiences with increased educational value. The results of three assessments were presented and, based on the feedback, improvements and future steps were discussed.

More specifically, this study presents how the described concept was applied in the developed VR museum set-up with the description of the tutorial, navigation system and interactive activities that are available when viewing 2D and 3D artworks. During the tutorial in the foyer of the virtual museum, the user practices navigation and interaction skills that will be used to interact inside the virtual exhibition. Although the navigation system does not allow the users to move freely in the virtual space in order to avoid motion sickness, it allows them to teleport to any point in the exhibition. To enable the user to find a satisfactory point of view in relation to the artwork, an optimal positioning functionality was integrated. This also allowed the developers to position the 2D graphical elements of the artwork within the user’s field of view and make them less distracting from the exhibits. In this way, the provided experience was improved significantly, and the application was considered to be easier to use by a general audience with wide age ranges and different levels of previous experience with similar applications. The interactive activities of focused detail, animations and the virtual guide were appreciated by the users in terms of their educational value and usefulness.

The fact that the provided interactions were delivered as activity types allows us to introduce those types of dynamics in the future. This would allow the users to select the activity type they want to integrate into such a VR museum set-up and adjust the content for each 2D or 3D exhibit. More types of activities can progressively be integrated into various contexts after following similar design and assessment processes.

Furthermore, the end-users validated the 2gether solution, and the assessments contributed to the evidence relating to the impact that a VR museum application can have on museum visitors. The results regarding (i) the perceived sense of control, (ii) the usability of the system, (iii) user satisfaction and (iv) their feelings during the VR experience were positive, and user feedback was considered to improve the application.

The presented assessment process proved to be very effective in our case because we managed to deliver a quality product to the audience, which was assessed during the development and was already significantly improved before it was released for the first time. One important point is that we updated the application based not only on the feedback of the internal project team but also on comments made by experts from outside of the project. Secondly, we conducted the first assessment on museum experts and then on technical experts; it was considered important to have the application validated in terms of the content, overall experience and usefulness before considering improving the technical details. Another important point is that user feedback was integrated into the project timeline and the development cycles. In this way, the application was able to be released on time and with confidence in terms of the content and its technical aspects. Additionally, it was considered important that the first evaluation of the released version was tested in the intended context of use (during a visit to the exhibition and the VR spot) with a sample of the intended end-users, in our case, general audiences.

The results of the three assessments provide evidence concerning the impact of the VR museum. They showed that the navigation system and interactions with the touch controllers significantly improved in terms of usability following the changes we integrated after the first two assessments. The application was confirmed to be easy to use for a wide audience in terms of their age range, and the sense of control was considered to be high. Moreover, the tutorial activity feature proved to be very useful when users were familiarizing themselves with the VR equipment, navigation and interaction system. The limitation that emerged is that the tutorial experience should be personalized in order to balance previous experience and cognitive load during the activity; however, this might not work the same for all users. Additionally, the educational value of the proposed activity types was validated by museum experts, and the activity types were appreciated by the end-users in terms of their usefulness. Furthermore, the user’s interest increased in terms of contemporary art, and they also felt entertained. They would also suggest the application to colleagues and friends (promoters), which is important for the museum pilot. The museum experts stated that they would also use the application in the context of an educational program or school visit. To validate the use of the application in the context of a school visit, another assessment should take place to ascertain user experiences and the learning outcomes. Finally, the quality of the reconstructed artworks was appreciated. Thus, we also presented the process we followed to reconstruct the 3D exhibits with the aim of achieving a trade-off between high-quality models for use in VR and polygon economy to facilitate their rendering within the game engine.

Regarding the learning aspects, an application such as the 2gether VR museum could be rated in terms of social, affective and cognitive impacts. In our case, the social aspect of learning was outside the scope of this project, while the affective aspect of the VR museum was rated through eliciting the users’ feelings during testing. Regarding the cognitive impact, although we elicited the opinions of museum experts on the educational value of the VR museum, we did not evaluate the actual learning impact of the application with the end-users. To assess the actual learning effect of the application, a different assessment set-up is needed that focuses on the learning outcomes, and one that is also meaningful to the participants involved in the testing process. Thus, the next step would be to test the application on general audience groups in the context of educational museum visits with, e.g., high school or university student groups.