Augmented Reality in Cultural Heritage: An Overview of the Last Decade of Applications

Abstract

Augmented reality is a mature technology that uses the real world as a substrate and extends it by overlaying computer-generated information. It has been applied to several domains. In particular, the technology was proven to be useful for the management and preservation of Cultural Heritage. This study provides an overview of the last decade of the use of augmented reality in cultural heritage through a detailed review of the scientific papers in the field. We analyzed the applications published on Scopus and Clarivate Web of Science databases over a period of 9 years (2012–2021). Bibliometric data consisted of 1201 documents, and their analysis was performed using various tools, including ScientoPy, VOS Viewer, and Microsoft Excel. The results revealed eight trending topics of applying augmented reality technology to cultural heritage: 3D reconstruction of cultural artifacts, digital heritage, virtual museums, user experience, education, tourism, intangible cultural heritage, and gamification. Each topic is discussed in detail in the article sections, providing insight into existing applications and research trends for each application field.

1. Introduction

Augmented reality (AR) is one of the most up-to-date and advanced areas of technology. The field of AR application began in the 1990s [1] and has continuously increased in importance since then, along with technological advances and the development of information and communication technologies (ICTs). While virtual reality (VR) refers to creating a digital world that reproduces an environment that already exists in the reality, AR provides a framework for adding information to the real-world environment. AR technology allows users to perceive the world in an enhanced way; it improves the users’ experience by overlaying computer-generated information, including graphics, sounds, and sometimes touch feedback, on the real environment [2]. Thus, new metaphors of interaction with the physical world and synthetic information about the environment need to be invented, tested, and applied.

AR has been applied in various fields, from medicine to education, automotive industry, healthcare, or tourism [3]. In particular, it was proven to be useful for the management and preservation of cultural heritage (CH). There are some main purposes for which AR is used in CH: improving visitor experience, reconstruction, and exploration [4], as well as conservation and preservation [5] and bringing to life past events [6]. The technology that is most often applied in all these cases is digitization. It is used as a complementary method to preserve CH assets and to enhance traditional conservation procedures [7].

Once digitized, cultural heritage is ready to be accessed by all beneficiaries, both scientists and non-scientists. Starting with non-scientists, the large public is eager to consume this information available in digital form. In order to maximize this stream from cultural heritage towards the ordinary consumer, it is necessary to maximize the users’ absorption of heritage knowledge through such a visualization. This means that visualizations have to be adaptable, interactive, and user-oriented. On the other hand, we strongly consider scientists that deeply understand the valuable information they obtain using multi-level digitization techniques. They are striving to record everything about the artifacts: materials they are made of, weather conditions, historical period, relations between artifacts, functionalities, using procedures, etc. [8]. All these metadata must be recorded in order to be readily used or for the technology that will be available in the future [9]. For now, this information could be augmented as volume on the social network bases and using deep learning techniques for automated input [10]; validated through ontologies [11]; and multimodally visualized using VR or better AR technologies that involve visual display, gesture detection, force-feedback, and audio rendering. We consider that interactively touching the past through digital artifacts, while listening its narrative and observing its reaction to user actions, is one of the ultimate forms of cultural immersion that can be offered to the user.

On another hand, digitization is a chance of survival for the world cultural heritage under a double threat, that of natural cataclysms and that induced by direct human criminal actions. An ideal digitization technique will allow us to perfectly replicate a digitized artifact. However, we are far from this ability. In our days, the digitization results are largely visualized using different technologies, such as VR and AR, and are manipulated by the user in supervised setups using gesture-based, touch or touchless implemented metaphors. Using the early stage of 3D printing technology, scaled mockups of digitized artifacts are currently available to the large public for free and direct manipulation, without the fear of destruction and offering a personalized individual experience and knowledge on what the artifact was about or used for and how it was used [12].

Various use cases for AR/VR in the context of digital heritage are reviewed in the following. With applications in social media-based recommendations, psychiatric art therapy, theater and artistic representations, and cultural heritage dissemination AR/VR-based tools have become parts of our everyday life and are here to stay for the next decade.

Previous reviews studied the application of AR technology in CH from several viewpoints. In [13], important aspects related to VR/AR/MR technology and relevant technical requirements needed to develop CH applications are listed. The tracking, display, and interface aspects of AR applications are outlined. In [14], how AR is used in the field of history education and which AR applications might be acceptable for this field are examined. The content analysis method was used to analyze mobile AR applications in the field of CH [15]. Recently, [11] presented a survey of current ontologies and data models for AR urban environment applications in the CH field. These reviews are mainly focused on technologies and methods.

Bibliometric analyses have often been used lately to guide the future direction of research in various fields [16,17,18]. An important goal of bibliometric analysis is to generate a comprehensive overview of trends within a research domain and to examine the performance of countries, institutions, and researchers in terms of dissemination and publication. In [19], a bibliometric analysis carried out in 2016 and 2017 in the field of digital heritage is presented. The study identified that, most often, keywords are related to the technology used to develop digital heritage applications. Recently, a bibliometric analysis that outlines the use of AR for the dissemination of architectural heritage was presented in [7]. In [20], the current issues related to digital museum are overviewed using bibliometric analysis, and the relevant fields of research identified are digital humanities and VR and AR technologies. The publication trends between 2016 and 2021 in using digitization technology for CH applications are presented in [7]. This bibliometric analysis outlined that CH is increasingly adopting digitization.

AR is a dynamic research field, and it is necessary to underline the state of the research and to provide the latest trends which will allow the development of future studies. The AR tools facilitate access to cultural heritage in an interactive and engaging way [21]. The present paper aims to carry out a conclusive study on the efforts made in the last ten years for the development of AR applications used for the preservation and promotion of CH. We analyze how AR is involved in cultural heritage applications by conducting bibliometric research regarding the documents published on AR applications in CH in the last ten years. We review the relevant literature in order to investigate the following research questions:

Q1. What are the trends in using AR for cultural heritage applications?

Q2. What is the structure of the last decade of literature?

Q3. What are the main research results?

Our review is structured as follows: First, we provide insight into the research methodology we adopted in our endeavor, pointing out the document search and selection criteria, software and data extraction process, and the analysis of the results and trends. Section 3 is the core of our review, focusing on the top trending applications of AR in cultural heritage. The section uses eight of the most used keywords/research topics identified in the scientific production of the field as heading pillars, namely 3D reconstruction, digital heritage, virtual museum, user experience, education, tourism, gamification, and intangible cultural heritage. The paper ends with our detailed conclusions where we share with the reader our opinion with respect to what the future of cultural heritage dissemination technology looks like.

2. Research Method

The data used for the present study were retrieved from Scopus and Clarivate Web of Science (WoS). We selected these databases because they contain extensive peer-reviewed publications and provide effective tools for downloading data. The search was run in April 2022, using the search terms “(Augmented OR Mixed) Reality AND (Cultur* AND Heritage)”. We selected the search query that ensures the relevancy of the results to the application of AR to cultural heritage. The period selected to accomplish the goal of this research was between 2012 and 2021. A filter search was used to reduce the number of articles to only those published in English, because in most cases, the results of this field of interest are available generally in English [22] (see Figure 1). A list of the selected documents and their key information was exported into the .csv (comma-separated value) format. Based on the chosen search terms, we obtained 835 documents from WoS and 883 documents from Scopus.

Figure 1. The methodology used in this study.

In the next step, the ScientoPy scientometric tool [23] was used for data reconciliation [24]. Both datasets were merged, and all duplicate documents were removed using ScientoPy (see Figure 2). Hence, through pre-processing, 504 documents were removed from Scopus and 13 from WoS. In addition, ScientoPy was used to simplify authors’ names, accents, and abbreviations. The result contains 1201 unique entities from both datasets stored in .csv format and was used as input for further analysis of bibliographic data. We also used the Excel 360 software tool in the data analysis process.

Figure 2. The pre-processing data from Scopus and WoS databases.

ScientoPy can categorize most prominent trends using the following topic growth indicators: average growth rate (AGR), average documents per year (ADY), percentage of documents in last years (PDLY), h-index of each topic [23]. In Table 1 we present the main topics of applying augmented reality to culture heritage. The following areas were the most prominent trends identified: (a) the 3D reconstruction of cultural artifacts; (b) digital heritage; (c) virtual museums; (d) user experience; (e) education; (f) tourism; (g) intangible cultural heritage; (h) gamification.

Table 1. Growth indicators of selected trend topics.

Pos	Author Keywords	Total	AGR	ADY	PDLY	h-Index
1	3D reconstruction	155	−6.5	14.5	18.7	12
2	Digital heritage	107	−2.5	14.5	27.1	11
3	Virtual museums	78	−5.0	8.5	21.8	14
4	User experience	63	−2.5	10.5	33.3	10
5	Education	60	−2.5	5.5	18.3	12
6	Tourism	48	−0.5	7.5	31.2	10
7	Intangible cultural heritage	40	−2.0	5.5	27.5	5
8	Gamification	39	−1.0	2.0	10.3	8

Figure 3 shows the evolution of the current trending topics in the field of AR applied in CH. This plot is based on a query that returns the most popular topics according to the cumulative number of documents versus the year of publication. The trend visualization includes all documents published in the last decade. A logarithmic scale is used on the Y-axis to facilitate visualization. This evolution plot provides information regarding the evolution of the trend for the past 10 years. It can be seen that topmost popular topics followed a similar pattern. The plot shows that most of the documents follow a consistent pattern of growth. In the right diagram in Figure 3, the AGR for each trending topic between 2020 and 2021 is represented on the Y-axis, and the PDLY is represented on the X-axis. The trending topic with the highest number of documents is “3D reconstruction” and the topic with the second highest relative growth is “digital heritage”.

Figure 3. Top trending topics generated from the author keywords.

Considering the type of documents, there are 419 journal articles, 746 conferences and proceedings papers, 16 book chapters, and 20 reviews. Figure 4 shows the VOS software [25] network visualization of documents for each country, colored according to the number of publications in that country. It indicates that as of December 2021, the highest numbers of related articles were published by Italy, followed by Greece, Spain, China, and United Kingdom. This finding is a good indicator of which countries currently provide research and development in this area. In addition, it may help researchers assess their country’s impact on the field.

Figure 4. The 10 most active countries in the field of AR in cultural heritage.

Figure 5a contains the first 10 countries that have at least 5 publications. The rank indicates the country’s position according to the number of publications. The number of articles in each country is a more appropriate measure than the country’s popularity in the topic. The number of citations per institution (Figure 5b) shows a relatively similar result to the previous figure. The highest numbers of citations were observed in three countries: Italy (2344), the United Kingdom (909), and Spain (608).

Figure 5. The 10 most active countries by number of documents (a) and citations (b).

3. Top Trending Applications of AR in Cultural Heritage

Over the last decade, AR technologies had significant growth. There has been steady progress in the development of AR applications for CH, which has led to the development of engaging apps that are making a greater impact on several topics.

3.1. Three-Dimensional Reconstruction

Ancient artifacts carry invaluable information about the culture and civilization of a society. However, they are often not well preserved or are inaccessible because of their size and fragility. It is estimated that in the past ten thousand years, more than 90% of humanity’s cultural heritage has been lost [26]. The merging of 3D reconstruction and AR for cultural heritage is currently becoming an emergent technology [27].

Three-dimensional reconstruction technologies show great potential for the digital preservation of CH artifacts [28,29,30]. Research in this field started before 2010, so a review published in 2009 lists several projects related to AR and virtual heritage [31]. The 3D reconstruction process consists of a sequence of steps [32]. The first and fundamental step is obtaining the initial data collection by using several technologies, among which the relevant ones for CH artifact reconstruction are: photogrammetric 3D modeling, triangulation and time-of-flight laser scanning, structured light, stereo-calibrated optical sensors, and mobile lidar.

Photogrammetry is one of the most often used methods in the 3D modeling of CH artifacts [33,34]. However, it relies on a considerable number of images and an increased computation time for an efficient reconstruction. Therefore, it takes more time to converge. On the other hand, the 3D geometry-based modeling approach depends on a depth-sensing camera which is often too expensive for many applications. Recent works propose methods to obtain information about depth using lasers or structured light [35].

A laser scanner is a high-quality 3D solution that allows creating models by scanning any surface—from stone to wood, metal, and many more—but the hardware equipment is expensive and complex software is needed to process data [36,37,38]. The recent development of lidar technology and the integration with a mobile platform enables adequate speed and accuracy for large CH sites [39].

One image is not enough, in most cases, for a complete representation of a 3D model [32]. The artifact images may have several different viewpoints. The next step in the 3D reconstruction is the alignment of color and depth information. The alignment of range images can be performed by using the iterative closest point (ICP) approach [31]. When the registration is accomplished, all the generated 3D views are combined in a single mesh. In the final stage, the 3D model is textured for a realistic representation of the digital CH artifact. The selection of the optimal solution for the 3D reconstruction is a challenging task.

A method that combines several techniques for 3D object restoration for AR applications is presented in [40]. The solution includes obtaining the 3D point cloud of the CH artifact using a 3D scanner device, processing and reconstruction of the 3D model using Meshlab, and building the AR application for the visualization of the restored CH artifact.

AR provides a method that enables experiencing a 3D reconstruction of a CH artifact (see Figure 6) with a high level of detail and with a full-scale dimension of the space perception, without requiring the user to have specialized knowledge [41,42]. Three-dimensional models in combination with AR enable the general public to better understand and experience history through more intuitive and interactive means [43,44,45]. Furthermore, the 3D reconstruction of CH artifacts can be used for 3D replications that involve combining digital touch screens, AR applications, and high-quality 3D printing. The subjects that evaluated 3D AR considered the use of reproductions or physical 3D prints intuitive, mainly for contextual understanding and intangible experiences [46].

Figure 6. An example of mobile AR application for exploration of the 3D reconstructed Prejmer Fortified Church, a UNESCO monument from Transylvania, Romania, courtesy of Voinea et al. [51].

AR is used for the restoration of deteriorated religious objects [47] and can help to revive these important CH objects [48]. It can be used to identify the most appropriate restoration approach to produce a realistic replica, which reduces costs and speeds up the process. Using 3D reconstruction technologies, a digital model of the damaged object can be created [49], which is then used to generate an AR application [50] that helps to determine the best restoration approach [42]. The main drawback of AR technology is related to the lack of proper guidelines on how to use this technology. The limited experience with AR applications is another significant issue that limits the adoption of this technology in the reconstruction process.

3.2. Digital Heritage

UNESCO’s definition of digital heritage states the following: “Digital heritage is made up of computer-based materials of enduring value that should be kept for future generations. Digital heritage emanates from different communities, industries, sectors and regions. Not all digital materials are of enduring value, but those that are require active preservation approaches if continuity of digital heritage is to be maintained.” [52]. As such, we are interested in those applications that encompass digital heritage and technologies such as augmented reality, virtual reality, or mixed reality (MR). We review some of the influential work in this respect in the following.

Digital heritage sites benefit extensively from the advent of social media platforms and applications that integrate feedback from such platforms [53]. Balduini et al. analyzed social media traffic (more precisely Twitter messages) in Seoul, in 2011, and discovered heavy messaging regarding various points of interest in the city, such as restaurants or cafes. Starting from this, they proposed a complete system based on semantic technologies which includes an AR app that recommends locations, which were previously ranked by analyzing the opinions of users over a period of three years [53]. Another application that demonstrates the value of digital heritage in tourism is presented in [54]. Here, an AR app exploits the concept of gamification in order to aid the process of learning history.

Studies of AR applications dedicated to digital culture that are integrated in psychiatric therapy are promising. In [55], AR applications are presented as helpful tools for practitioners to deliver a form of art therapy. The acceptance of digital tools by therapists was influenced by their educational level with respect to the use of digital media. Digital tools are now embraced by researchers and practitioners [56,57]. Clinical interventions that use VR or AR are reviewed in [58], in the context of psychological and behavioral health, with the purpose of identifying how technology modulates the communication between patient and clinician.

The immersion of the visitor in the theater representation is facilitated by augmented reality in [59]. The authors bring their contribution in the form of 3D models of a historical church, as well as in the development of a multimedia system that allows participants in the show to be immersed in the virtual space. Digital heritage capitalized by means of AR applications, supported by sonic narratives [60], aids in creating a believable experience [59].

In another study pertinent to digital heritage, the digitization of cultural assets in museums in China is inspected [61]. The study involved 22 sites selected as testbeds for digital heritage applications, such as AR, VR, or multi-touch systems. The authors investigated factors that influence the successful use of the digital exhibits and patterns of behavior of the visitors in order to obtain a proper understanding of how digital culture use may be enhanced in China.

Malik et al. are investigating the perception of digital (based on AR, VR or digital displays) vs. physical (based on 3D printing) replicas of cultural heritage artifacts at the level of different stakeholders, both specialists and non-specialists, together with their applicability in different contexts [48].

The augmentation of museum exhibits with supplementary information (see Figure 7) helps build an “animated archive of cultural materials” [62], whose scope is to catalog objects alongside their descriptions and to identify multiple representations of the same object.

A step forward is made in [63] by augmenting virtual replicas of cultural heritage monument visualizations with real actors interpreting historical figures using narrative-based setups.

As for the creation of digital media content, in [64], an AR application is researched in an educational context, in an AR-based application for a design course. The app is found to aid students in concentrating and self-learning and also to raise students’ confidence.

Figure 7. A 3D-augmented book titled Constitutio Criminalis Theresiana (also known as Nemesis Theresiana). Courtesy of Duguleană M. [65].

3.3. Virtual Museums

Museums that want to go beyond their traditional experience of providing information based on audio guides and paper catalogs are starting to innovate at the level of the user experience (see Figure 8), making it more intuitive [66]. At the same time, they are adopting innovative systems able to deliver the content of their information while improving the visitor’s experience. AR museum information experience enables the visitors to learn about art collections [67,68], archaeological sites [69], or ancient buildings [53,70] in a simple and intuitive way [71]. Another advantage of using AR technologies in the museum is the ability to dynamically add a digital interactivity component to a museum exhibit. The digitally interactive component may be an AR game [72] that engages a player as a type of digital interactivity, or it may be a feature that provides an enhanced education [73,74] or some other type of content related to the physical exhibit [75]. This has the effect of increasing the visitor’s level of engagement with the artifacts and their meaning.

The necessity for developing innovative methods to digitize CH artifacts into readily deployable 3D content is becoming a priority as AR gains more attention in the CH sector. Additionally, the workflows required to transform CH artifacts into immersive 3D content are hardly supported. In [76], a practical workflow that can be utilized for 3D reconstruction to AR visualization of CH assets has been developed, and it identifies and combines a variety of open access and custom tools and services.

By using AR technologies, galleries, libraries, archives, and museums (GLAMs) can add multiple sensorial experiences that can engage visitors with CH content. One such example is the alternating reality (AltR) narrative that was introduced in [77]. This is an interweaving of two seemingly disparate narratives, one real and one virtual. During this real-time experience, the user visualizes and interacts with the virtual narrative. The Museum Coffee Table [68] is a physical surface that has been augmented so that actual objects can be used to access data about artists and their works. Using this system, the entire family may gather around the table, enjoy coffee, and enhance their museum tour by learning new information and having fun.

A study on how digital narrative content is used to communicate information regarding heritage trails through mobile apps is presented in [78]. By creating a list of the most frequently used features in successful mobile applications, it aims to make heritage trail apps more accessible and usable.

Accessibility issues in museums and exhibitions require special attention. The current museum guide solutions do not facilitate the experience of CH content for people with hearing loss or people with other disabilities. AR technology provides an immersive way for people with hearing loss that allows them to engage with museum exhibitions [79]. In [80], an AR application that uses virtual humans as sign language storytellers for on-site CH experiences is proposed.

Cataloging is an important part of the museum process that allows systematic classification of artifact information such as material, shape, function, and history. AR technologies can be used for cataloging information that describes museum artifacts in a dynamic and augmented description [62]. AR can help in the involvement of a new audience in museums through multi-dimensional environments that transpose visitors from physical to digital experiences and enable the exploration of new views [81,82]. Many studies [63] estimate that, in the near future, AR is expected to be used in a wide variety of museum exhibits that will increase visitor knowledge and interactivity. However, there are still some issues that limit the use of AR technologies, mostly related to user experience (UX) and technical knowledge required to develop AR apps [82].

Figure 8. Visualization and interaction with entire 3D scene (a) and a virtual artifact (b). Courtesy of Popovici et al. [83].

3.4. User Experience

An evaluation of AR technology potential from the point of view of stakeholder interests is performed in [84]. Stakeholders are regarded as either internal, such as a museum’s staff members, or external, such as the visitors of the museum or community representatives. The authors explore the implications of using AR on various dimensions, such as economic impact, social effects, or educational impact, in a user-based qualitative study performed in a small museum in the UK. The user study gathered stakeholders’ impressions on items pertaining to each of the dimensions where AR value was being assessed (economic, experiential, social, epistemic, historical and cultural, educational). Cultural heritage was proven to benefit greatly from adopting AR, from the perspective of both internal and external stakeholders.

The willingness of users to adopt AR-specific gadgets, such as AR smart glasses, and the potential behind such devices from the user perspective are studied in [85,86]. The study [85] used a sample of 28 visitors to a small UK-based art gallery, and it focused on identifying the driving factors that influence user preferences for smart glasses in a cultural setting. While innovativeness was perceived as a plus, the interaction was difficult for some users, even though they assimilated the smart glasses to regular glasses that they would wear. The perceived enjoyment was positive, as was the perceived usefulness.

In [86], smart glasses were used in an outdoor setting, enhancing with both audio and visual aids the exhibits at the Hecht Museum in Israel. In the experimental study, a tour guide implemented as a mobile application is compared to the AR smart glasses guide solution, with respect to usability and user acceptance, following a thorough statistical methodology. While the novelty of using smart glasses was the main attraction, the study also revealed some problems relating to the specific outdoor environment, such as lighting, that need to be addressed in the future.

Guidelines for the design of applications using wearables and in particular smart glasses are derived from an experiment at the MIT Museum in [87]. Observational data were gathered from 12 participants recruited among MIT students, freely interacting with a Glassware prototype in the Robotics Gallery, followed by a qualitative study that extracted emerging trends from text, transcribed from audio interviews.

The influence mixed reality exerts on the museum experience is modeled in [88]. The museum context is provided by the exhibition “the Ara as it was” from the Ara Pacis Museum. In the research design, the visitor experience is assessed on seven dimensions (museum information, customization, format, usability, information saving, interaction, experience) with 23 individual variables. The relatively large and diverse user sample (586 local and international visitors) provides value to the conclusions of the study, that users are moderately to highly satisfied with mixed reality integration in the museum. However, the study was only performed in one museum, and it leaves to future work questions regarding the influence that various factors, such as cultural background or previous mixed reality experiences, have on the overall mixed reality experience.

Tourism benefits from cultural heritage AR applications have been explored to enhance the attractiveness of local cultural tourism sites (see Figure 9). User experience with using mobile AR applications in the cultural heritage tourism context is modeled with Hassenzahl’s theoretical model in [89], and it is empirically confirmed in a particular implementation for urban tourism in Dublin, Ireland. The user sample was limited, and the qualitative study was not followed by a quantitative study to test its generalization power.

Figure 9. Mobile AR application for improving the experience of the tourists. Courtesy of Boboc et al. [6].

In [90], the authors attempt a case study of a complex AR application, which encompasses gestures and uses a 3D holographic interface and integration with the Meta platform. Apart from providing developmental details regarding the application, the paper also presents a quality assurance testing that takes place in Canada’s largest museum, namely Royal Ontario Museum. This preliminary testing resulted in some issues to be addressed by the development team, such as the lack of an exact description of the gestures recognized by the application or failing to provide help in the virtual immersive environment.

Another high-impact case study of an AR mobile application created as a tour guide is presented in [91], in research deployed in two directions: one regarding the detailed presentation of the developmental details and the other describing the survey designed to evaluate the app and the empirical results obtained by applying the survey in a user sample.

While the majority of usability studies deal with assessing the effects of technology on the main beneficiaries, which are the users (or the visitors, in a museum context), in [92], the evaluation takes into account three categories of individuals: the professional in the field of cultural heritage, the institution itself, and the visitor. To this end, a model is proposed, under the name MUSETECH, that is applicable for various technologies (mobile, AR, VR) in various fields (education, museums, tangible experiences). The model proposed is suitable for evaluating the impact of technology on the three categories of individuals.

A complex user-based study, with both qualitative and quantitative items, proposes discriminating between the attraction of museum visitors to AR technology in a museum gallery and their distraction by such technology by evaluating the “attentional balance” of visitors using the AR virtual magnifier (loupe) [93].

Four levels of experience of the user, in the context of dealing with VR applications for museums, are identified in [94]: artifact-related, behavior-related, spiritual level, and creation level. They are exploited by the VR application itself in the process of delivering information. A review of papers is provided, grouping them according to the specific level of user experience that is the purpose of the research in the paper. Finally, the paper proposes a model for the VR museum experience that relies on the four levels of experience discussed and acts as a design framework for VR-enabled museum exhibitions [94].

A communicative approach exploits gamification in the cultural heritage landscape in a complex application that uses VR by means of a head-mounted display, as well as AR by means of a mobile application, to allow virtual exploration of a 3D reconstructed temple and personalized interaction with the historical vestiges [95]. Preliminary evaluation of the prototype makes use of a preliminary analysis (prior to public exposure to the system), a survey completed during the museum visit, and a paper-based questionnaire filled out at the end of the visit.

The attractiveness of outdoor heritage landmarks is enhanced by means of a mobile AR application in [96], which addresses five different historical epochs. The small usability study (15 users) uses a survey with 7-point Likert scale items grouped in order to target various traits of the application, such as comprehensibility, manipulability, enjoyment, and usefulness.

We note the preference of researchers for user studies on various dimensions that translate into questionnaire items for each variable involved.

At the same time, theoretical models are empirically studied and reinforced with user-based questionnaire studies [86,87,88,89,90,91,92]. We notice a smaller preference of researchers towards proposing new models, such as in [94], a fact that may be due to the inherent difficulty of proposing and validating a new model, and further having it accepted and becoming mainstream in the research community.

3.5. Education

As we go through the third decade of the 21st century, innovations in education are needed more than ever, since many distractions compete to steal students’ attention from the educational act. Digital technologies, which young users are attracted heavily to, can be capitalized into transforming the way students learn.

VR and AR can play a crucial role, enabling users to learn and repeat experiments with minimal costs as many times as they wish [97]. In addition to the advantages of VR, AR is cheaper and can provide more interactive and cooperative communication ways between students [98].

Wearable AR gear—in particular, Google Glass—is used in [99] in a controlled experiment aimed to assess the effect of the wearables on the user learning experience, set in the context of a museum gallery of British paintings from the 19th century. An evaluation of the level of enjoyment of the users, their level of understanding and knowledge and the improvement of their skills, and their appreciation of the learning experience completed the controlled experiment [99]. Some drawbacks of using Google Glass were also identified, such as diminished social interaction between the learners using the wearables.

AR combined with a dual display and a multi-touch screen was found to help students better relate to the spatial context of historical monuments in [100].

Mobile learning by means of an AR application on a tablet used in real fieldwork learning is compared to traditional eLearning that takes place in a classroom with a regular desktop computer in [101]. The learning context is provided by heritage elements of the city Santiago of Chile. The study found that the mobile learning process significantly enhances the educational outcomes. Historical knowledge gain is the focus of the mobile AR multi-user game application presented in [102], used both in an indoor and in an outdoor setting. Bits of history are discovered in a playful manner by users, engaging them seamlessly in the learning process [103]. Serious games, i.e., “games with educational purposes”, are a valuable teaching tool in the humanities field, allowing the users to engage and be immersed in the virtual created environment [104]. Different game genres (such as adventure, strategy, or puzzle games) and different learning objectives aid in distinguishing between the many serious games reviewed in [93]: games that target cultural awareness (e.g., Discover Babylon (https://www.slj.com/story/discover-babylon-the-gaming-life, accessed on 29 September 2022, Roma Nova https://www.irit.fr/~David.Panzoli/romanova.html, accessed on 29 September 2022)), and Remembering 7th Street (http://7thstreet.org/, accessed on 29 September 2022), games that target historical reconstruction (such as The Battle of Thermopylae [105] or The Playing History (www.playinghistory.eu, accessed on 29 September 2022)), or games intended to raise heritage awareness (such as Time Explorer (https://www.seriousgamemarket.com/2010/03/time-explorer-serious-games-at-british.html, accessed on 29 September 2022) or Tate Trumps (https://www.tate.org.uk/press/press-releases/tates-new-art-game-mobiles-race-against-time, accessed on 29 September 2022)).

A large number (n = 197) of cultural heritage education programs is reviewed in [106] to identify the specific components that influence the potential of AR and VR for facilitating the learning process, making use of the Q-Edutage scale [107] in the evaluation process.

While the use of novel technologies is a trigger for students’ motivation in learning [108], it is interesting to assess the impressions of those that usually deliver education. Teachers’ views on the usefulness of AR tools in learning, as well as their willingness to continuously learn themselves and to participate in the creation of such tools, are depicted in [109]. Teachers were found to be eager to prepare continuously, participating in seminars and training sessions on new technologies. While the majority of those scrutinized had no experience with 3D modeling, all of them showed interest in learning 3D modeling techniques and would even consider participating in the development of new AR tools for learning.

The target group for the learning process in [110] consists of adults, in the context of a literature museum. An AR web application augments a literary museum, together with storytelling techniques, in an attempt to immerse visitors into the virtual universe of Svevo’s literary work, proving that even senior citizens can benefit from specially designed AR applications.

Art and multimodal technologies intertwine in education or cultural heritage applications [111] which describe, among others, an interactive environment created to mediate non-verbal communication and social collaboration, using complex audio interaction. Collaboration and multimodality are the distinctive features of VR/AR heritage applications that enable proper relations between users and the cultural context [112]. In [44], AR is used to support the visitor experience of the cultural landscape of Matera through a mobile application.

During the past decade, AR and VR have touched the educational process in various fields, such as industrial processes [97], museums [100,108], art [99], history [102,103,104], geography [102], and even literature [110]. We expect such applications to flourish in the future, as the technological pipeline is now mature and AR application development is becoming easier.

3.6. Tourism

The potential of MR technology in tourism was increasingly exploited in the last decade, due to the growing accessibility of the latest technology for 3D heritage content creation, visualization, and interactive dissemination, both in public edutainment environments and in academic/research units.

With predecessors such as Rome Reborn [113,114] and Ancient Miletus [115], Archeoguide [116] is one of the first AR-based cultural heritage projects that highlighted the influence that AR technology can instill in empowering tourism in Greece at the beginning of the 21st century and has not stopped inspiring since.

No matter which visualization solution is implemented, CAVE-based [117] with the promise of group-oriented virtual visits or HMD-based [118] with increasing quality of user-immersion or large visual displays [119] installed inside a museum, the challenges the technology had to face regarding tourist satisfaction in using the system were strongly linked with the cultural experience [87].

Studying the impact virtual reality technologies may have on tourism activities, Marasco et al. [118] proved that although a strong emotional user involvement is triggered by a virtual reality-based experience with great perceived visual appeal, the impact on the behavioral intentions to visit a cultural heritage site was not as positive as expected [118]. The authors warn that their study was realized on a single virtual tourism site, so a generalization of their results cannot be made. However, they appreciate the marketing potential posed by the VR experience, due to the emotional implications of users’ experience which translates into the propagation of positive impressions to other potential users.

The biggest step from CAVE or large display-based public VR systems was made once the mobile devices became powerful enough in order to be more than displaying devices [103]. Moreover, VR-oriented solutions started to be replaced with AR hand-held solutions as a new way to enhance user satisfaction while still preserving the cultural heritage and being able to offer a user-adapted learning experience [84].

In [120], the effect of cultural influences on the acceptance or adoption of AR applications is assessed in a study involving participants from the Republic of South Korea and from the Republic of Ireland. The aesthetics of AR is found to have a strong positive influence on the perceived usefulness, the perceived ease of use, and the perceived enjoyment of AR, in both countries.

A VR/AR multimodal application dedicated to the cultural heritage monument Jeju Island of South Korea is described in detail in [121]. A virtual assistant helps the user navigate the heritage site, while full immersion is made possible by the use of the Google Cardboard VR headset, and AR visualization and virtual handling of artifacts is also made possible by the application. Several hypotheses are expressed, in the framework provided by the Technology Acceptance Model, and verified in a user-based study involving 251 users which found that the hedonic characteristics of the application prevail over other utility-related features.

Cultural heritage valorization by means of a wearable AR project set in an art museum context is the object of study in [99]. As wearable gear, the Google Glass is considered, and the study performs a comparison of two samples, Google Glass wearers and a control group, with respect to learning outcomes, enjoyment, and influence on future behavior. The control group showed slightly greater knowledge gain than the participants in the wearable AI application, who were distracted more by the novelty of the application and did not focus as well on the learning experience. On the other hand, Google Glass wearers had better and faster access to a greater quantity of information, in a seamless fashion. Social isolation was reported as a downside of using the AR solution.

Smart glasses are used within another AR application in an art museum setting in [85]. The participants in the study positively perceived the innovative character of the application, yet they appreciated the interaction as difficult, not being accustomed to wearable equipment previously. The majority of participants found the application enjoyable and useful from a cultural tourism perspective.

No matter what the level of mixture between the real environment and the computer-generated content is, cultural heritage virtual environments are inspired by historical facts and have as their main function the visualization and interpretation of digital replicas. Once the user wants to reach the next level of virtual cultural experience, interaction with virtual artifacts has to take place on the basis of alternate scenarios that bring into hypothetical setups the known and contested historical heritage [122].

Annotated assisted on-site navigation allows for a profound multimodal exploration of cultural heritage sites [123,124]. User experience both inside and outside of museums is enhanced by the use of mobile applications that facilitate sharing of content between users and building cultural memory [123], profiting from both tangible and intangible cultural heritage.

The integration of VR/AR/MR elements into tourism activities, such as a museum visit, brings opportunities in a mixed reality project developed in Rome, in the museum Ara Pacis [81], but also brings forward some difficulties inherent to such a project. User immersion in a museum space redesigned to fit MR elements, such as audio, video, or touch, enhances the experience from various perspectives, including educational, socialization, and entertainment. The experience of users participating in a virtual tour augmented using smart glasses is evaluated in [125] through the analysis of text from reviews posted on the TripAdvisor platform. Two AR mobile applications for urban heritage capitalization are described in [89]. They are analyzed in a user study that assessed their utility, aesthetics, and interestingness, and both apps are found to enrich the tourist experience, in both outdoor and indoor environments.

3.7. Gamification

In recent years there have been some very interesting developments in the gamification space that can help change the perception of the types of applications and uses of AR in CH. The concept of gamification was defined as “the use of game design elements in non-game contexts” [126] or “game science that explores the various design techniques, and related concerns, that can be used to add game elements to existing real-world processes” [127]. More recent, another definition states that “gamification is about taking something that is not a game and applying game mechanics to increase user engagement, happiness and loyalty” [128]. The advantages offered by gamification highlighted by the above definition have been exploited also for the enhancement of the CH field by making CH more interesting and providing unique and entertaining ways for people to learn and experience these spaces. Often referred to with alternative terms such as “pervasive games” or “game-based learning” [129], gamification has been used in this field mainly in two forms: serious games (SGs) and storytelling.

SGs were created with the aim of producing constructive or educative results in addition to entertainment value. So, their objective is to support the user in achieving learning goals through a fun experience [106]. In CH, SGs are considered some of the most innovative ways to understand culture and history by using interactive technologies [130]. SGs are characterized by the following features: (a) the user’s behavior, learning, and achievement goals are more important than entertainment; (b) the design of the game is specifically created to improve the knowledge, skills, and behaviors of the target users.

There are several ways that SGs can be used to encourage more engagement with CH. For instance, the SG concepts could be used to gamify the interactions with artifacts using multimodal interfaces to create immersive experiences for users [131]. An immersive haptic VR system for recreating the experience of shooting with a bow was presented in [132]. Haptic feedback had the potential to improve the user experience, as shown in [133]. In addition, SG and gamification were used to gamify social spaces, providing opportunities for visitors to explore CH spaces in a fun and engaging way at the same time. In particular, gamification in the context of museums has been a fruitful research topic with some interesting examples of AR systems used for game-like interaction [134,135]. The enhanced AR experiences in museum settings have the potential to maximize user satisfaction and also learning outcomes [136]. In spite of the opportunities offered by indoor environments, many recent AR experiences have been designed using smartphones and tablets as supporting hardware for outdoor settings [137,138], for instance to discover CH locations [139] and to enhance experiential learning [140]. In [141], a mixed reality (MR) assisted by geoinformatics technologies for reviving historical events was presented.

The other form of gamification with great potential to revolutionize the way the users engage with CH is storytelling. Digital storytelling (DS) is defined as the evolution of traditional storytelling that combines the ancient art of telling a story and current technological possibilities [142]. Extended reality (XR) technologies have been used successfully in interactive DS applications to experience past events and meet historical characters [63] or to enhance the appreciation of classical Chinese poetry [143].

The relationship between AR technology and storytelling was also explored in [110] for an application designed to engage the visitors of a literary museum.

While gamification has been discussed as a means to provide incentives for engagement, immersive technologies also have a clear potential to support the public interested in CH, not only for tangible CH elements, such as artifacts, buildings, and historic sites, but also for intangible CH to highlight social values and traditions [144] such as traditional sports [145] or craft [146]. AR and gamification can be seen as related fields, and their success is closely related to enhancing people’s experiences and promoting learning and collaboration. The use of AR in CH will be extremely diverse in content, types of interaction, locations, and contexts, providing means for multisensorial experiences [147].

3.8. Intangible Cultural Heritage

Intangible cultural heritage (ICH) plays an important role in the life of each community because it is an expression of the history, traditions, and culture of the people who have lived and worked in a certain area. UNESCO includes in ICH the following: “oral traditions, performing arts, social practices, rituals, festive events, knowledge and practices concerning nature and the universe or the knowledge and skills to produce traditional crafts” [148].

Digital solutions could help to preserve and protect ICH. AR technology has great potential to enhance the perception of intangible cultural heritage [149]. In [150], the authors proposed an AR system designed for the digital protection of ICH that uses an intelligent terminal to fuse its content and realist video images in the scene.

AR technologies can be used to overlay ICH-related virtual decoration in living space, which brings a new enjoyable perspective to the user experience [151]. Additionally, based on AR technologies, the environment can become similar to a stage where traditional dances (for example Australian Al Ardha aboriginal dance [152] or Greek traditional dance “Syrtos in Three” [153]) can be learned.

In the field of digital display for ICH-related content, AR equipment can provide virtual try-on experiences of traditional costumes and accessories [154]. By experiencing the AR environment, users could transpose themselves to a traditional atmosphere [155]. Interactive AR walls that combine markers, projection mapping, and animations are also suitable for presenting folklore heritage, national traditions, and legends [156]. Visitors can also experience an immersed impression by using HMD in conjunction with ICH-related materials [157]. Despite all the advantages of using AR technology, one downside is the increased amount of time needed for the production process of AR applications [158].

4. Conclusions

The main goal of this study was to identify and overview the most significant topics from the last decade in the field of augmented reality and cultural heritage. We conducted an advanced search on the Scopus and WOS databases and performed a scientometric analysis on a large article dataset (n = 1201) using ScinetoPy and VOS software. In summary, the findings showed that there is a growing interest in applications of AR in many CH contexts, from 3D reconstructions to intangible cultural heritage. The growing tendency for each topic was highlighted using trend analysis indicators AGR, ADY, and PDLY.

The 3D reconstruction of cultural heritage objects is a complex task. Over the last decade, several techniques for 3D reconstruction such as photogrammetry, laser scanning, lidar, and structural light were successfully used for many CH artifacts. While these techniques provide the raw data, other techniques derived from AI such as deep learning and ontology at the same time enrich and filter the information to be used by different types of users such as researchers, museographers, or tourists. In this regard, as in the Europeana platform mentioned, a cultural heritage cloud is prepared to be densely populated and largely disseminated all around the world [159].

AR is a very effective solution for showcasing 3D reconstructed artifacts. It can also be used to identify the most appropriate restoration approach to produce a realistic replica, which reduces costs and speeds up the restoration process. Aspects as data quality, available metadata, and multi-layered information related to the real artifacts are some essential aspects that have to be considered in order to ensure a self-sustainable AR-based CH application. For many CH artifacts, 3D reconstruction combined with AR produced practical solutions, but there are still many difficulties and unresolved issues. Future research in this area may focus on real-time digitization techniques that are more computationally efficient.

Digital heritage benefits extensively from AR technologies, whose recent widespread adoption makes them a major challenge to the way we understand and study the past. The development of AR has led to important research with the aim of addressing the needs of museums, archives, and heritage institutions. Developed applications demonstrate that AR can improve several tasks in digital heritage, namely visualization of reconstructed artifacts, documentation, and contextual understanding. In addition, using AR technology, digital heritage combines creative industries’ production with the digitization of CH artifacts in order to augment the visitor experience with animated digital archives, mixing real-world remnants with digital media. It is of real importance that tourists have the opportunity not only to touch the past through 3D printed replicas of pieces that transcend ancient history but also to experiment with and thus be culturally immersed in interactive setups involved in inter-human lost activities.

AR virtual museums enable a deeper level of involvement and connection between museums and their visitors. AR has been used to make the museum contents more interesting and attractive to the users. To create AR experiences, a variety of techniques are used in order to render a vast amount of information about artifacts, mainly using a mobile device. In this way, AR technologies have the potential to increase access to and interchanging of information that describes the CH artifacts in a dynamic way. At the same time, AR is beginning to be largely used for offering new audiences in museums through multi-dimensional environments that transpose visitors from physical to digital multimodal experiences. However, the usage of AR technologies is still constrained by a few issues, most of which are related to the complex technical skills needed to develop these applications.

Available AR technology’s potential has been proved in CH both from economic and social perspectives. While specific AR gadgets are triggering willingness to use, they are proved to be more suitable in indoor setups rather than outdoor ones. Mobile technology remains one of the most accessible and promising solutions for the user-oriented AR visualization of CH. In order to obtain an effective AR app from the user experience point of view, there are some design considerations that must be addressed. Usability, intuitive visualization of AR content, and the conception of UX interfaces must all be given importance. Enjoyment and usefulness are important aspects regardless of whether we consider the user’s freedom to interact through AR technology, where smart glasses are preferred because of the user’s free hands and gesture-based implemented metaphors, rather than interaction with mobile multi-touch displays. In addition, the development of an AR app for CH requires knowledge and understanding of cultural and historical aspects.

Education has been in the spotlight in the last decade, especially in recent years. Gaining students’ attention and maintaining it is particularly difficult in the context of heavy social media and computer or mobile device usage. Recent advances in technology must be exploited to add value to the educational process, and AR, VR, and MR are not exceptions. In cultural heritage-related education tools, AR plays an important role, allowing full immersion in 3D reconstructed historical settings and/or free interaction with virtual reconstructed artifacts. Gamification is combined with AR/VR, ensuring a collaborative and engaging experience, in which learning happens seamlessly. At the same time, through the addition of visual and auditory features to the digital collections, AR increases students’ access to CH resources and their understanding.

It is hard to find CH experiences outside a tourism-based experience. We cannot imagine holidays without heritage objectives of interest to visit, locally or abroad. AR applications can help tourism by enhancing the visitor’s experience and improving CH information perception. By adding 3D visuals to the existing information, AR-based tourism applications can also give tangible advantages by enhancing the value of existing CH attractions.

Nevertheless, technology has a long way to go until the tourists will select their cultural heritage objectives as “have to visit this one” points of interest on their holiday map, despite the fact that it is mature enough to increase the visual appeal of real artifacts augmented with digital ones and thus trigger strong emotional involvement of the user in a possibly multimodal (visual, audio, textual, intangible) experience. However, tourists are starting to transform from “simple” CH content consumers, using mobile accessible devices, to digital heritage contributors, by the means of social media platforms, sharing their CH-oriented experiences and augmenting them with their own stories and memories and thus pushing forward the common cultural memory. Of course, this trend comes with the challenge of filtering this huge amount of information, classifying it, relating it to the existing information, and finally deciding if the visitor’s input was useful to humanity’s memory.

Gamification is intensely exploited in electronic educational resources, especially in the context of online learning. Cultural heritage educational applications make no exception, especially since the AR tools used for developing such solutions allow seamless integration of game-like behavior. Be it a serious game approach to CH or a storytelling approach, application areas for CH education benefit greatly from gamification. User engagement and collaboration, which are crucial for the end results of any educational process, are promoted by incorporating game-related ideas into CH applications. AR and gamification may provide a viable tool to engage participants not only in tangible CH elements, such as artifacts, buildings, and historic sites, but also in intangible CH.

In our vision, ICH is somehow an ultimate form of CH expression because it subsumes artifacts, their functionalities, the processes in which they are used, and the purposes of their use. Even if we talk about traditions (oral and gestures), performing arts, social practices, rituals, knowledge, or practices focused on any kind of human activity (e.g., skills to produce traditional crafts), addressing ICH with AR technology combined with user action validation (possibly based on some kind of ontology) could believably transpose the “cultural immersion” experience.

Although the analysis of keywords from Scopus and WOS databases provides high quality and consistency of the presented results, the inclusion of other databases can provide a more thorough identification of trending topics in the fields of augmented reality and cultural heritage.