A Gamified Augmented Reality Application for Digital Heritage and Tourism

Featured Application

An Augmented Reality application designed to allow on-the-spot museum visitors to navigate and view exhibitions using their own mobile devices by enjoying user experience multipliers like interactive content and mixed reality quiz games.

Abstract

Although Augmented Reality (AR) technology has entered many market and knowledge domains such as games and leisure activities, it remains rather limited in digital heritage. After studying the potentiality of using modern AR elements in a museum context, this paper proposes the use of additional game and educational elements in the core AR application in order to enhance the overall on-the-spot museum visitor’s experience. An agile AR application design methodology was followed by taking into account the needs of small-to-medium sized real-world museums. Moreover, a heuristic evaluation protocol was applied by a group of experts in order to test the proof-of-concept AR application, in which some novel elements were proposed such as the AR quiz game. The main findings indicate that enhanced AR experiences in museum settings can make a nice fit with the user environment, physical and perceptual abilities, known metaphors, and user position and motion in 3D space. Moreover, AR services can be provided under a minimum distraction and physical effort. As a conclusion, AR technologies are mature enough to be standardized for museum usage, while the audience seems to be ready to take advantage of the related enhanced museum experiences to maximize both user satisfaction and learning outcomes.

1. Introduction

In today’s digital cultural landscapes, people are increasingly dependent on technological tools to access digital cultural content such as musical, pictorial, and multimedia expressions that surround them [1]. In a broader sense, the digital transformation of culture has led us to the era of Smart Culture in which Information and Communication Technologies (ICT) can help citizens to benefit from improved cultural accessibility and inclusion [2]. Moreover, Cultural Technologies (CT)—which resulted from the integration of ICT into heritage organizations—have led to a change in the structure of the organizations and have affected the management of collections and the visitor’s experience [3].

CT can hardly be defined, as they include a wide range of technologies used to support the digital transformation of culture and the operation of memory institutions (museums, galleries, libraries, etc.). Such technologies may include the Internet of Things (IoT), which may help museums offer personalized services and thus complete their mission by building a better museum of education, research, and appreciation of the heritage by wider audiences [4,5].

Augmented Reality (AR) is featured as an important part of modern CT and—according to Uricchio [6]—it has become the “next big thing”, since most smartphones today have the necessary hardware and software characteristics to reproduce AR experiences (for free), and this growth has been propelled by the advertising industry. In addition, from a user-centered point of view, AR exposes visitors to an alternative interaction style, which can trigger curiosity and interest and overall can give a sense of accomplishment [7].

The main issue here is that although the use of AR has already penetrated certain market domains such as leisure and games (e.g., Pokemon Go (https://pokemongolive.com/)), it remains rather limited in the realm of other domains such as heritage education [8]. On the other hand, museums and cultural organizations are willing to experiment using new methods of democratizing culture, such as the use of mobile devices during an in-site visit, in order to reach their audiences and attract their attention [9,10]. A straightforward way to achieve these goals is to use AR for annotating museum scenes and providing extra information about collections or individual artefacts. Advances may include game elements introduced into the annotated scenes to offer gamified user experiences such as explorative navigation, playful interaction with museum artefacts, fantasy tours in ancient times, and more [11]. According to a Europe-wide study of Luna et al. [8] on this topic, it seems that most AR digital heritage proposals reconstruct spaces and buildings in outdoors applications, and only a limited number of use cases handle objects in indoors applications (excluding simple audio tour apps using mobile devices).

In the light of the new findings and recent technological advances, this study aims to introduce AR technologies into museums and galleries as a User eXperience (UX) amplifier. Apart from covering typical indoors museum AR applications that track a single object (2D or 3D artefact) and augment virtual objects and content on the top of the existing one, this paper will present a novel AR quiz game used to motivate users to revisit exhibitions. The gamified app allows on-the-spot museum visitors to navigate themselves in exhibitions using their own devices, to interact with the museum’s educational media in a dynamic and fun way, spend more time on the exhibitions, gain new knowledge from the museum visit, and pay attention to details not seen in a first pass. Moreover, this study will describe how AR markers were introduced in museum context in an aesthetically accepted way without any reduction in 3D object detection stability and overall user experience quality.

After a short explanation of the methods used for the design and development of the proof-of-concept application, it was evaluated with the participation of an expert team following a heuristic evaluation protocol especially developed for AR applications. Evaluation results discovered possible violations of the heuristic criteria, estimated the level of the violation’s severity, and finally provided recommendations for fixes and further development, which were included in the final version. The outcomes are expected to open new possibilities for museum visitor’s navigation, content visualization, and playful and educational effective interaction with museum artefacts.

1.1. Cultural Heritage and Learning

Museums are places to share not only access to artefacts and their related stories (museum narrations), but also memories, experiences, and open questions. They develop new knowledge by museological research, and they support the didactic transformation of knowledge by developing and sharing documentation used inside and outside the museum. For example, during school visits, it is quite common that teachers make first an introduction to the objectives of a museum visit and outline the exhibitions of interest, and after the museum visit, they propose an assignment or a structured discussion about what students have learned during the museum visit. Moreover, museums organize activities to offer educational proposals for families and complement formal school programs [12].

Cultural learning in museum settings is of a high environmental validity, meaning that the museological narration and documentation is placed in a wider context of the learner′s environment, thus enabling people to appreciate the museum contents, participate experiencing emotional involvement, and create the feeling of ownership. Colizzi et al. [13] outline this inherent educational nature of cultural heritage and support that it is well suited to school learning, thanks to its nature and ability to support multidisciplinary learning. Following the same philosophy, Hourdakis and Ieronimakis [14] examine the role of the museum regarding an individual’s education and see museums as catalysts for dialogue and learning. Cultural heritage and learning are linked in multiple ways, and this should come as no surprise as art and history educators in museums and schools share common professional backgrounds and content fields, although they operate in different environments and organizational contexts [15].

1.2. The Use of Augmented Reality in Museum Settings

Museums use a wide variety of demonstration and learning techniques based on the prior experience visitors may have by using technology. For example, infrastructures used in smart cities such as the Internet of Things (IoT) could be proposed in digital heritage leading to the Internet of Cultural Things (IoCT) concept. This can be perceived as evidence that interruptive technologies play a major role in transforming culture into ambient culture, as seen in a literature review conducted by McKenna [16].

Furthermore, when navigation and exploration are combined with game elements, the final outcome—from a user experience and engagement perspective—becomes really interesting. Sigouros [17] considers that ICT tools are useful in historical literacy, as long as they carry on supporting content to limit the time needed to find sources of information, provide direct and clear feedback to the user, but also allow teachers to provide their students with external motivation for learning, which in turn creates feelings of inner satisfaction.

Recent advances in mobile devices such as high screen resolutions, large mobile networking bandwidth, and increasing processing power enabled the Bring Your Own Device principle [18] to be the dominant information-sharing model in a variety of market domains, including Digital Heritage. This trend reduced the cost of service distribution, but it also released the need to share specialized hardware. For example, Saputro et al. [19] proposed the use of signal wireless sensor networks to help with the position and identification problem inside museums using the triangulation method and the Received Signal Strength Indicator (RSSI). Such solutions may give promising results, such as the distributed self-gesture and artwork recognition system of Baraldi et al. [20], which uses wearable devices (named by the authors as “ego-vision embedded devices”) but still requires the use of special equipment and user education before use. In another example, wearable devices such as Smart Glasses or AR Glasses are a kind of special equipment needed to introduce AR in museums [21]. The need for specialized hardware is removed when talking about mobile AR, in which visitors use their own device for position/orientation sensing and projection.

Practically, what AR does in a museum context is to combine real-time computer graphics and high-quality museological content with Human–Computer Interaction (HCI) modalities to enrich experiences after superimposing artificial objects on the real-world view of the museum. Thus, the playful experiences turn into learning experiences, since the digital contents have taken the form of a pedagogical processing. A few examples are the mobile guide to museum settings proposed by Damala et al. [22] and the LifePlus project, which proposed a 3D reconstruction of ancient frescos enhanced with animated avatars (artificial life) to offer dramaturgical narration in an immersive AR environment [23]. A key issue in almost all AR apps for digital heritage is the human activity monitoring, which allows the application to adapt to the user’s intentions and their physical movement in the 3D space. This sensing is usually performed through a marker or a marker-less technique to estimate where the user stands and looks, and to display the right virtual objects on the screen. Those superimposed objects include 3D representations of museum artefacts and additional information [24], or virtual objects that are part of a game concept [25]. Moreover, AR apps used for reconstructing cultural heritage for open spaces become increasingly common [26], such as those used in archaeology. In marker-less techniques, AR apps reveal features that become available from the physical objects in a scene (Kolivand et al., 2018). Alternatively, AR applications in museum settings may use IoT devices such as beacons [27] to sense the standing point of the users, but this may not be enough to obtain all the data required to accurately superimpose virtual objects on the mixed scene.

Regarding contents, mobile AR applications in heritage contexts include museological narrations of past or present times [28], 3D avatars to guide visitors through the collections of the museum [29], interactive painting and recoloring of museum artefacts [30], and even presentations of spiritual heritage elements, for example those related to the life of important personalities [31]. Quite often, learning activities hosted in museums are followed by a short assessment (e.g., self-guided quiz). Those quizzes are delivered in a paper-and-pencil form, or they can be interactive quiz games, such as in Quakequiz of Prange et al. [32]. In the majority of cases, there is no use of AR technology during this evaluation phase (quiz time).

1.3. Underlying Motivation

This paper will propose an AR quiz game designed to increase the time museum visitors interact with artefacts and offer a playful way of navigation in the 3D space and as well an alternative method of gained knowledge assessment. In this study, we accept that museum exhibitions have a native educational purpose, and in this regard, we identify three main reasons for using modern AR technologies:

Personalization: It can be said that from an educational point of view, what AR technology has to offer is mainly personalization and visualization. Museum visitors, as potential learners of new knowledge, can have a certain level of control over the learning speed or rhythm. Having museum visitors with a wide range of ages, cultural and knowledge backgrounds, it becomes easily understood why adjusting the pace of learning into one′s own needs could be so important. Interacting with the educational content of a museum in your own device and under your own rhythm of interaction is a very good example of learning speed adjustment to personal needs and preferences. Moreover, visitors can follow their own navigation route through the information space and pay attention to details of the museum artefacts they enjoy the most. AR technologies give the opportunity to make such adjustments to the needs of individuals.

Motivation: AR can maximize the learner’s motivation, especially for young learners who love storytelling, play leisure AR/VR games quite often, and can easily master mobile devices. From a constructivist approach, AR museum games can be understood not only as mediums to capture and maintain the attention of the audience but also as a way to reconstruct the meaning of the artefacts through animated storytelling inserted into the AR scenarios. This museological storytelling is working closely with visitor’s emotions and fantasy to maximize user engagement and interest. Moreover, certain game elements such as competition, challenges, conflict, control, and awards for achieving goals can work as “experience amplifiers” for all ages of museum visitors and prevent boring museum visits.

Learning Efficiency: Perceiving learning speed as one of the success criteria for technology-based educational activities, we can accept that learning in AR museum settings (e.g., using virtual reconstructions of artefacts, possibly examining alternative conditions, taking risks in a safe environment, etc.) can help master the skills and knowledge faster and/or deeper than in a classroom or in other forms of typical education. The better learning curve observed when AR applications are being used has frequently been reported in the literature [33].

Based on all of the above, the underlying motivation of this work is to explore the expected educational benefits and models of user experience enhancement caused by introducing AR navigation and games in museum settings, mainly in indoor exhibitions. The main objective is to achieve a fair balance between leisure and usefulness without retreats to other qualities such as usability and technology acceptance. The following sections present the e-Tracer AR application, which was designed according to the principles previously discussed.

2. Materials and Methods

2.1. The e-Tracer AR Application in a Nutcel

The e-Tracer AR application is an educational and guide application for museums and galleries. It was designed to allow on-the-spot museum visitors to navigate themselves in the museum exhibitions using their own devices (mainly smartphones and tablets). Currently, the e-Tracer AR application has been optimized for the permanent collections of the Silversmithing museum (http://www.piop.gr/en/diktuo-mouseiwn/Mouseio-Argyrotexnias/to-mouseio.aspx), located in the castle of Ioannina, Greece. The digital contents of the app were developed in compliance with the main objectives of the permanent museum collection, which is to preserve the technical knowledge of epirote silversmithing artists and to disseminate information about its tradition to the wider public.

The AR functionality starts by sensing a museum artefact of interest, which is being seen through the mobile’s built-in camera. The marker-based object detection mechanism triggers the projection of additional information (short description) related to this object (Figure 1), or it allows a virtual projection of animated 3D objects when available. There is an underlying multilingual support system, and currently, the information and educational content are provided to visitors in the English and Greek languages. Exhibitions in the Ioannina Silversmithing museum are mostly static and thus, in a hand-held AR setting, the object projection is expected to be performed in standard positions in the physical museum settings and in view of the visitor’s distance to the object. Our solution has been technically tested for working at a distance up to 1.5 m from the artefact (Figure 2).

What was described up to now covers mostly the typical functionality expected from an AR application in museum settings. Considering similar applications, the use of QR codes appears to be the dominant way of delivering information about artefacts. For example, Pérez-Sanagustín et al. [34] developed two ways: one using a visitor’s smartphone and QR codes and a second one using a screen located next to the exhibits, but finally, they chose the first one after measuring the effect QR codes had on visitors. QR codes with image recognition are also being used as the main object identification method in commercial AR applications such as Layar (https://www.layar.com) and Gamar (https://www.gamar.com). Thus, e-Tracer advocates the use of QR codes for indoor or outdoor use, but at the same time, it takes advantage of the aesthetically superior illustrations of artefacts, which can be equally strong at object detection. Pikov et al. [35] also used Unity to develop an indoors AR application and QR codes to integrate preprocessed 3D models, but they used a large display to make details of exhibits more accessible for visitors. In contrast, the e-Tracer AR app allows visitors to download the preprocessed 3D models on their mobile device and project the (animated) models in a dynamic zoom in/out mode.

What e-Tracer has to offer more as added-value functionality is the interactive game-like self-evaluation questionnaire, which is based on the AR component. After visiting the museum exhibitions, museum visitors are asked to participate in a quiz about the new knowledge gained from the museum visit. What is new is the AR questions that instead of asking users to select one of the available options (which is very typical in closed-type questionnaires such as the one presented in Figure 3a), they ask people to revisit museum collections, search for artefacts with specific characteristics, and scan them with the camera of their tablet. The AR app will recognize the featured object as a correct choice and will reward the user with symbolic awards (collected points and user visibility on a top-ten leaderboard). For example, in the app’s prompt: “Identify in the museum’s collections of secular artifacts a pen-case with an incorporated inkpot or otherwise “Kalamari””, users have to go back and look for the silver pen-case, such as in a treasure hunt game (Figure 3b).

The design of the AR application took into account the needs of museum visitors of all ages. Dynamic adaptation to the preferred level of difficulty is performed automatically in case of young children and school students (based on their profile previously created). Moreover, the designers took into account the findings of Sylaiou et al. [36] derived from evaluating the ARCO (Augmented Representation of Cultural Objects) application [37] regarding the wished qualities, which were found to be: intuitive User Interface (UI), high-quality 3D representations (photorealistic), sense of control, obvious exit, embedded help, and a challenging game concept.

2.2. Museum Artefact Detection

Despite the technological progress made on Wi-Fi, Bluetooth, Radio-frequency identification (RFID) technologies and more, orientation in mobile AR applications in indoor places is still an open question [22]. Market-ready solutions are based on easy-to-implement and reliable techniques in order to offer smooth visitor experiences without breaks and interruptions while keeping technology transparent. Following this trend, the e-Tracer AR scenes are subdivided into applications that require a marker to detect a museum artefact and project a 3D object on top of that. Those that do not require an additional identifier but use the museum artefact itself (displayed through the mobile camera′s image) as an identifier are still under development. The use of printed Quick Response Codes (QRC) as markers, although they are easy and cheap to be produced by museums using their own equipment, was finally avoided, mainly because they are not human readable, and thus, people may not always understand which marker corresponds to which museum artefact. Moreover, QRC markers are not aesthetically accepted in a museum or gallery context and are relatively small in sizes; thus, they may cause a loss in object detection stability. On the other hand, markers that contain a printed copy of the depicted artifact were considered more appropriate for museum settings for a number of reasons: they are aesthetically accepted, relatively small in size, cheaper to be produced with the museum’s own equipment, require relatively low device profiles, and provide effective object detection in controlled environments. In the physical environment, the AR markers were located next to the artefacts after they had received optimization for a better object detection accuracy to fix certain problems caused by the marker-less and QR code techniques (e.g., not human-readable, poor detection quality in low-lighting conditions, efficient algorithms too demanding for mobile devices). From a technical point of view, the initial markers (Figure 4a) were first pre-validated using their grayscale histogram in order to evaluate their initial suitability as markers. Images that combined a low overall contrast and a narrow and sharp grayscale histogram were not likely to be selected as target markers. In contrast, if the grayscale histogram was found wide and flat, the image had a good chance to be selected as a marker, since it contained a nice distribution of visual features.

Afterwards, potential markers were optimized according to the Vuforia’s guidelines [38] for achieving a better object detection accuracy:

• Gamma correction: Gamma encoding is applied on images usually to optimize the usage of bits during compression, by taking advantage of the non-linear manner in which humans perceive colors and light [39]. Here, gamma correction was applied with powers larger than 1 to make the shadows of the artefact darker (Figure 4b).
• Local contrast enhancement adjustments: An overall contrast/brightness adjustment helped enhance visual features, especially around corners and borders of flat surfaces. A lighter color in the background layer resulted in more feature points, while higher contrast and lower brightness on foreground features improved the feature’s quality.
• Laplacian filter: applied to detect edges, enhance them, and thus increase the points of interest, making them easily discoverable in museum rooms, even with varying lighting conditions.

Additional adjustments such as cropping images to remove areas without a density of features (for example, the barrel of the pistol in the top-right corner of Figure 4d) could improve the overall rating of the marker, but such modifications were not applied, since the markers are museum artefacts themselves, and any cropping could lead to a poor visual representation of the historical objects. Finally, the real object is being detected as its characteristics are compared with the characteristics of the models that are known in advance as a marker [40].

Before and after modifications, the pictures of the objects were evaluated against their suitability to be used as markers in the e-Tracer AR application. The evaluation criterion was based on the evaluation algorithm of the Vuforia (https://engine.vuforia.com/engine) engine, which computes the number of high contrast points. For simplicity, the overall quality is expressed in a range of 1 to 5 stars. Before any modification, the pictures of the museum artefacts were not exceeding 2 or 3 stars at maximum, while after modifications, the quality rose to 4 stars at minimum. Part of the validation process was testing against tolerance to rotation, but since the object detection engine returned a 6 Degrees of Freedom (DoF) pose of the target to the application layer, it was possible to automatically determine the amount of rotation and finally to project the 3D augmented object on the right angle.

2.3. Development Process and Tools

The overall AR app development process (Figure 5) was based on a modified Herzig [41] approach, which was used to insert game elements into a knowledge domain, such as the digital heritage in this case. The AR app development was separated from the development of the rest of the e-Tracer platform, which is about designing and developing a system that, exploiting innovative spatial interconnection technologies for multiple sites of environmental, cultural and touristic interest, will discover semantic information from multiple data sources, providing users with the ability to organize integrated touristic routes across Via Egnatia in Northern Greece. Although separated in development, the e-Tracer platform and its AR applications are being offered to customers through a single-entry point (web view).

The e-Tracer AR app was developed in Unity (Unity, https://unity.com) and uses the ARCore Software Development Κit (SDK) provided by Google (https://en.wikipedia.org/wiki/ARCore). More specifically, the e-Tracer application consumes the motion tracking, environmental understanding, and light estimation services of the SDK. The detailed 3D models of the Ioannina Silversmithing museum’s artefacts (the most heavy or not movable ones) were provided by the supervising organization (Piraeus Bank Group Cultural Foundation), while the modeling of supplementary virtual elements was performed in Google Sketchup (https://www.sketchup.com), 2018 version. Some light objects such as silversmithing hand tools were scanned using a portable Sense 3D Scanner (Cubify 3D Sense, https://www.3dsystems.com), but with moderate results, as the resulted 3D models need retouching to achieve our quality standards. The processing of the AR markers was performed in a freeware version of PaintShop Pro (https://www.paintshoppro.com). The main object detection processing is performed on the user’s mobile device, while 3D models are being automatically downloaded from the cloud services of the e-Tracer system and become available to the user on demand to allow them to examine hidden sides or details of the artefact (Figure 6).

2.4. Evaluation Method

The current e-Tracer AR prototype has been evaluated—at first—by a group of experts. Although user testing is in the future plans, the expert review method is considered equally accurate in case of skillful and knowledgeable usability experts [42,43]. Moreover, a systematic review of Dey et al. on AR usability studies published from 2005 to 2014 [44] indicated that user studies—which is the most common evaluation approach—may not be enough, and that heuristic evaluation on AR applications needs to be further developed. The usability heuristics proposed by Nielsen [45] are generally accepted in this study, but since they are oriented to productivity software and not to Digital Heritage, finally, the listing of Endsley et al. [46] was preferred, as it can better frame the multiple aspects of the AR experience design.

The classification of issues (heuristic criteria violations) according to their severity was taken from the International Software Testing Qualifications Board, which defines “the degree of impact that a defect has on the development or operation of a component or system” [47]. According to the above, four classes of heuristic criterion violations were proposed: Blocking/Critical, Major, Minor, and Insignificant.

2.5. Evaluation Protocol

In this study, four (4) experts in museum technology were employed as volunteers in the spring semester of 2020. From those, three (3) were males and one (1) was female, all in the range of 39 to 65 years old. The members of the expert panel represent four different but interconnected communities: academics, schoolteachers, artists, and museum curators. More specifically, one expert is a University professor with a specialty in multimedia applications in visual arts and museums, another one is an active educator, book author, and expert in didactic interventions using art and ICT tools, a third one is a chief curator at a national museum (of contemporary art), and the last expert is a school art educator with experience in educational applications. It is worth mentioning that those people were not involved in the development stages of the AR system or in the e-Tracer project.

The purposes and the protocol of the evaluation were explained on a one-to-one basis, according to which experts were able to ask their own questions and receive additional information about the AR app. According to the Experts Team assessment method, the experts were invited to critically review the AR system and were involved based on their experience and expertise in evaluating ICT applications in education and in digital heritage. Each member of the expert group evaluated the prototype individually, after they were instructed to explore the application through its demo materials. An online questionnaire was established using a common e-Survey engine to collect reviewer’s feedback against the heuristic criteria explained in the following section.

Overall, it was expected that the expert group would discover possible violations of the given heuristic criteria, estimate the level of severity, and finally provide recommendations for further development on the functional characteristics of the application.

3. Results

This section provides a concise and precise description of the experimental results.

Table 1. Heuristic evaluation criteria for AR [46] and experts’ responses.

A/A	Heuristic Criterion	Violations	Comments and Mitigation Actions
1	Fit with user environment and task: visualizations and metaphors should match the user’s mental models	Expert 1: □ Expert 2: ☐ Expert 3: ☒ Expert 4: ☐	Although visual metaphors were self-explained, it was not clear if the available time (gauge bar) refers to the quiz or only to the first question.
2	Form communicates function: Virtual elements should rely on existing metaphors to communicate affordances and capabilities	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	Visual metaphors are well known to users. Flashing brackets are OK, but it was recommended to display a message to prompt the user explore the physical space in AR questions.
3	Minimize distraction and overload: minimize accidental distraction as AR apps can easily become visually overwhelming	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	There is a well-stepped process in accessing the museum information (first short info and then full artifact screen).
4	Adaptation to user position and motion: virtual elements are useful and usable from the variety of viewing angles, distances, and user’s movements	Expert 1: ☒ Expert 2: ☐ Expert 3: ☒ Expert 4: ☒	The system is stable and reliable only in good lighting conditions. In poor lighting, the markers cannot be identified.
5	Alignment of physical and virtual worlds: Placement of virtual elements should make sense in the physical environment	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	Virtual objects appear nicely near markers and are aligned with architectural elements of the physical world (e.g., walls).
6	Fit with user’s physical abilities: AR experience should not require physically challenging or dangerous actions	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	No physically challenging or dangerous actions were found.
7	Fit with user’s perceptual abilities: The app should not present info in ways that fall outside of an intended user′s perceptual thresholds	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	The visual elements size, colors, and motion are well designed. Resolution is appropriate for tablets and distance to AR markers affordable.
8	Accessibility of off-screen objects: AR UI should make it easy to find/recall the items users need to manipulate when those items are outside the field of view	Expert 1: ☐ Expert 2: ☐ Expert 3: ☐ Expert 4: ☐	No violations to this heuristic were found.
9	Accounting for hardware capabilities: AR apps should meet the capabilities and limitations of the user’s mobile devices	Expert 1: ☐ Expert 2: ☒ Expert 3: ☐ Expert 4: ☒	Some of the app components could not overcome the limitations of the hardware (not all devices are supported, especially older ones).

presents the heuristic criteria used for the evaluation of the e-Tracer AR application (including the AR quiz), the possible violations identified by the members of the expert team, and their comments on how violations—or their effects on usability—could have been mitigated.

Apart from closed questions, a second part of the review was introduced a couple of weeks later to let experts express themselves regarding the advantages of AR technology when used in museums and the advances of the proposed application over conventional systems. Three out of four invited experts responded in this second part and provided valuable feedback. The questions were given to reviewers in the local language (Greek); however, English translations are provided below.

In the question (RQ1), “In your opinion, what is the advantage of using Augmented Reality during a museum tour?”, one expert made a direct comparison to the standard audio tour, and two others mentioned the immersion and interaction effect. More specifically, it was explained that AR is far better than a pause-and-play audio experience, which is the dominant model so far. One member of the expert team explained that visitors frequently complain about audio interfaces because it can easily become “annoying to search for the artefact number, type it to the audio device, press the playback button, and then go back and forth all the time because the narration may go too fast”. Such small difficulties and interruptions can ruin the overall visitor’s experience. In contrast, AR apps can sense what the visitor is looking at and display the right information without inconvenience. In addition, it was said that AR can provide a more interactive and immersive journey in the museum and actually can help visitors develop a new kind of relationship with the artefacts. In other words, apart from creating a memorable journey, AR can develop the feeling of ownership and can maximize the time visitors spend on site.

The next question (RQ2) was about the advantages of the e-Tracer AR app (“Based on the AR application you evaluated, what do you think is its biggest advantage over existing applications?”). The main advantages can be summed up as follows: the use of images instead of QR codes, the interactive learning model, the playful interaction with the artefacts, and the 3D model projection capabilities. Especially in the Silversmithing museum, there are plenty of small objects such as jewelry placed one next to the other; thus, a typical QR code marker method would have practical difficulties. It is worth mentioning that apart from the evident advantage of using 2D or thin artefacts as markers themselves, existing posters, graphical illustrations, and historical photographs already placed on the walls can be used as markers as well. An expert said that the advantage of the e-Tracer AR app is the 3D projection capabilities: “It is nice that you can rotate the 3D model and see hidden parts of it. I liked the 3D zoom in/out feature, which allows people to study details.” The projection of 3D models is of particular interest in cases of small jewelry or unique or very expensive artefacts that are protected inside glass boxes and cannot be seen from all angles or viewed up close. Another expert mentioned that the AR question functionality is the biggest advantage of the app under testing, but this is discussed below as it is more related to the next question. The third reviewer appreciated the immersion capabilities and the fact that, “Somehow, this app forces visitors to deal exclusively with them (artefacts) during the virtual tour.”

The last open question (RQ3) was about the new type of question introduced in the e-Tracer quiz app and is based on the AR technology. The question was, “Based on the AR application you evaluated, what you think is the advantage of using Augmented Reality in the self-assessment quiz?” Experts appreciated the interactive learning model being used and the play-to-learn experiences it could create to visitors, especially the younger ones. Circulating quizzes after a museum visit is not something new, nor the fact that the quiz is delivered through a mobile device. However, as an expert explained, it is the combination of AR experience, treasure game, and new knowledge self-assessment that makes this feature unique and different than other AR apps. This feature is significant because it allows people to construct complete mental shapes (concepts) about artefacts as they have the chance to revisit, have a second look at artefacts, and pay attention to details. The second expert supported that the “innovative use of the AR question gives the user the opportunity to revisit exhibits”, while the third one explained that the AR component of the quiz “applies some force to visitors to meditate on the application and thus makes them to recall the information displayed and compare it with prior knowledge”. Authors can add here that the AR questions do not simply test knowledge gained in the museum (this could be considered pretentious) but provide a playful way to (re)explore exhibitions without having missed important parts of it.

4. Discussion

This section will discuss the evaluation results of the e-Tracer AR app and will perform comparisons with the results of other studies regarding similar applications. AR apps that require special equipment such as wearables, Smart Glasses, IoT devices, and prior user education/training on how to use the special equipment are excluded from these comparisons. Those should be considered non-similar approaches in the context of everyday museum experience by wide audiences.

In general, AR technology opened up new navigation and visualization possibilities for museum visitors. It can introduce interactivity, tour guides, games, and 3D projection capabilities without a possible damage to the artefact: qualities that are needed to balance static museum content and fight visitor’s boredom. According to other researchers [8,48] and also the experts’ panel in this study, these seem to be the main advantages of the technology-supported augmentation AR has to offer in visitor’s experience.

Regarding the e-Tracer AR application, one very important usability result that emerged from the experts’ estimates is that the used visual metaphors and the overall interaction design were self-explained and well-known to users from other AR experiences and apps they may have used. One expert found that the visual metaphor for the available time that appeared as a green line (gauge bar), which progressively moves to the right of the screen as time elapses, was not very clear. More specifically, it was not clear to the reviewer if the time bar refers to the overall quiz time or the time given to the current question. This issue was noted as major, and finally, it was fixed by explaining in the help files that each question may have a different available time, according to its difficulty.

Although not a heuristic violation, one expert recommended that the AR quiz app should display a message to prompt the user to explore the physical space in the AR questions. This message, in addition to the flashing brackets, would reduce the risk of user’s laziness or boredom.

Two experts identified a problem in the system’s ability to adapt to the position and orientation of the user, especially in poor lighting conditions. It is true that the interior of the Silversmithing museum at Ioannina is quite atmospheric, leading to an intended low lighting condition; thus, the object detection algorithm of the AR app would have difficulties performing well under such conditions. This issue could be overcome if markers were placed in the center of bright spotlight cones. There could be more places in the museum where light would be enough to allow maximum performance for the object detection mechanism, but this may require either the rearrangement of museum exhibitions or the placement of additional spotlights targeting the museum artifact markers.

Another issue that was identified as blocking/critical by two of the experts was the non-adaptability of the app to poor device profiles. Actually, the e-Tracer AR app requires a relatively new hardware profile and updated software from the hosted tablets; otherwise, the app may not start at all. Luckily, the mobile devices used by the wider audiences are getting better and better, and in the near future, this will not be an issue. On the other hand, within the e-Tracer project, there is a provision for the supported museums to supply tablets to the visitors who wish to experience the AR functionality of the application.

The results of the comparison between the e-Tracer AR app and other AR applications for museums (based on literature findings and the opinions expressed by the experts) presented evidence that AR games appear to be a promising approach that is meant to create extra motivation to museum visitors, especially young children. This is in line with the e-Tracer AR design, which invested in the combination of a gamified AR experience with self-evaluation through a quiz to offer the chance to discover a different aspect of the exhibitions in a second route and to create memorable playful experiences for younger visitors. As an example, Antoniou et al. [49] agree that online Serious Games can attract the interest of more visitors and thus make cultural heritage sites known and increase physical visits to the sites. Other researchers found that museum visitors need support tools to facilitate their informal learning [50] and that game elements in AR applications for museums can support learning. Indeed, Oh et al. [51] found that the game component in the AR app they had designed made learners achieve a better learning performance than those who had had a similar AR experience but without the game element.

The e-Tracer AR app was designed to serve educational, museological, and user experience objectives. The realization of a heuristic evaluation protocol created the first evidence that it can fit very well in the user’s environment and physical movement, known concepts, mental models, navigation metaphors, physical/perceptual abilities, and museological context for which it was designed. Moreover, experts found that a unique combination of game elements, cutting-edge AR technology, immersion, interaction, and self-assessment of new knowledge was achieved in the e-Tracer AR app. It is expected that those features will be beneficial to museum guidance and the time visitors spend on exhibitions, as well as to information gathering through 3D models superimposition and finally to learning through the AR questions, which give the opportunity to revisit and pay attention to details not seen before. All of the above justify the playful design of the e-Tracer AR and informal learning components, which were both very well represented by the AR quiz component. Additional features such as the use of image markers and the projection of 3D content on top of the static artefacts on demand complete the puzzle of a modern AR application which makes exhibits appear alive, more appealing, and ready to be explored. Similar unobtrusive marker approaches can be found in Khan et al. [52], which used printed black-and-white markers detected by a smartphone’s camera, or in a case study of the Museo Diocesano of Milan [53] in which natural printed markets were used. All these studies can confirm that low-cost unobtrusive markers can be a well-working solution for low-to-medium sized museums.

On the other hand, this study may have some limitations, which are mostly related to the sample and selection of reviewers. Although expert team evaluation using heuristic criteria is a common formative evaluation and has certain advantages [54], according to the white paper of Mahmoud [55], there is a possible bias in the selection of experts. This refers to the possible future involvement of the experts in the new product, sharing a lot of similarities such as backgrounds and sources of information, having non-suitable expertise and unknown bias, if experts are outsiders. In the current study, effort was made to minimize those risks by recruiting experts who (1) will not get involved anyhow in the future product, (2) have different backgrounds (academia, typical education, museum exhibitions organization and art educators/creators) and (3) have hands-on expertise in cultural technology. The only remaining risk—according to the way Rabie describes it—is related to unknown bias, as experts are external to the project.

5. Conclusions

Most state-of-the-art AR solutions depend on the use of special equipment (such as wireless sensors, IoT devices, see-through AR glasses, or other wearable devices) to catch the attention of the museum visitors and enhance their experience. However, apart from being expensive, this special hardware needs maintenance. Moreover, education on how to use hardware should be provided to the visitors. Although most of those solutions give impressive results in usability, they suffer from the fact that they cannot be used in a context of a massive attendance of visitors.

To overcome these drawbacks, the e-Tracer AR application does not make use of any other special equipment than the user’s smartphones and tablets. The proposed application is intended to be used by numerous small-sized museums distributed in the Greek province. Local museums operate under limited financial means, and unfortunately, the number of tickets they issue each day does not justify buying expensive equipment or developing AR applications using their own means. Thus, the e-Tracer project proposed AR horizontal use-case scenarios that can get the most out of AR technology using the visitors’ own devices.

According to the above, the e-Tracer AR app was designed for the museum’s network of the Piraeus Bank Group Cultural Foundation giving emphasis on a visitor’s navigation, learning, and gaming experiences. Mobile AR technology was used to display digital versions of artefacts next to their physical position in the museum, bring objects to life, and make an interactive narration out of the available information about artefacts. The element of originality here lies in the AR quiz component used to give visitors motivation to revisit museum collections, spend more time interacting with the museum collections, and offer a memorable game-like experience, which may be of particular interest to younger visitors. Moreover, it could be used to measure growth in knowledge for in situ museum visitors.

Apart from being in line with the results of other similar AR projects for museums, e-Tracer has some unique new features to propose. The originality lies with a novel AR quiz in which questions are proposals for revisiting exhibitions, seeing artefacts from a new perspective, paying attention to details, and participating in a game-like AR activity inside the museum. Moreover, the identification of an object is based on more aesthetically accepted markers than QR codes, which may not reveal much about the artifact they correspond to and may disorientate visitors.

Overall, AR proposes a unique set of challenges for technology designers to integrate interactive AR experiences into the museum context. Apart from user’s positioning and attention, there are environmental conditions that may influence the overall user experience of the museum visitor. Those conditions are not always known in advance, and thus, developers need to follow both design guidelines and evaluation protocols before being confident that their AR software services can assess the general required usability and there are no barriers to sustainability.

Future plans of the e-Tracer AR app include the evaluation of the prototype by museum visitors via extensive trials that will be carried out during the e-Tracer project lifecycle. After a successful first evaluation of the prototype using the experts team protocol and having scalability in mind, the prototype will be further developed to handle a growing amount of work by adding more content such as 3D models, multimedia, and object’s metadata, and by supporting simultaneous users in more than one museum. Thus, the final evaluation will include cross-museum visitor experience assessment. Differences regarding AR technology acceptance and overall engagement between groups of people with different demographic characteristics will be researched as a topic of particular interest. Moreover, it will be investigated how visually impaired people could take advantage of the gamified quiz application based on successful existing paradigms on map exploration using mobile devices [56].