An Augmented Reality CBIR System Based on Multimedia Knowledge Graph and Deep Learning Techniques in Cultural Heritage

Abstract

In the last few years, the spreading of new technologies, such as augmented reality (AR), has been changing our way of life. Notably, AR technologies have different applications in the cultural heritage realm, improving available information for a user while visiting museums, art exhibits, or generally a city. Moreover, the spread of new and more powerful mobile devices jointly with virtual reality (VR) visors contributes to the spread of AR in cultural heritage. This work presents an augmented reality mobile system based on content-based image analysis techniques and linked open data to improve user knowledge about cultural heritage. In particular, we explore the uses of traditional feature extraction methods and a new way to extract them employing deep learning techniques. Furthermore, we conduct a rigorous experimental analysis to recognize the best method to extract accurate multimedia features for cultural heritage analysis. Eventually, experiments show that our approach achieves good results with respect to different standard measures.

1. Introduction

Cultural heritage (CH) often fails to be a successful attraction because it cannot fully capture people’s interests. Due to legal or environmental regulations, the enjoyment of an archaeological site cannot be improved, and it is often hard to recognize relevant details in a cultural landscape or understand their meanings. In recent years, many technologies have allowed excellent results in the cultural heritage (CH) domain. On the one hand, the diffusion of new deep learning models and applications with state-of-the-art performance in many fields have found applications in this domain. Moreover, the distribution of mobile devices, thanks to the decreasing price, the high speed of new network infrastructures, and new devices, such as virtual reality viewers, allow the design and development of new applications for cultural heritage that improve the visiting experiences of cultural sites. These new technologies provide information to users quickly and intuitively. In this context, information has a strategic value in understanding the world around a user. Other new digital devices, joined with new technologies, provide the ability to interact with places or objects in real time [1,2]. Furthermore, cultural heritage goes toward decay, so future generations may not be able to access many artistic works or places. Digital cultural heritage is a set of methodologies and tools that use digital technologies for understanding and preserving cultural, or natural heritage [3]. The digitization of cultural heritage allows the fruition of artwork, from literature to paintings, for current and future generations.

Augmented reality (AR) represents a technique that has changed how art is enjoyable. The advance of information technologies has made it possible to define new ways to describe and integrate natural and digital information [4]. Undoubtedly, virtual reality allows users to be completely immersed in a computer-generated environment, hiding the real world when the device is in use. Augmented reality will enable us to superimpose information around the user without blinding them in their physical environment. Mixed reality overlays information on the natural world. It includes the ability to understand and use the environment around the user to show or hide some of the digital content. Additionally, there are new classifications of the reality–virtuality continuum, such as extended reality, in which the natural and virtual world objects are presented within a single display. Thus, comprehensive reality includes all the previously mentioned categories [5].

Most applications that extensively use multimedia objects, such as digital libraries, sensor networks, bioinformatics, and e-business applications, require effective and efficient data management systems. Due to their complex and heterogeneous nature, managing, storing, and retrieving multimedia information are more demanding than traditional data management, which can be easily stored in commercial (primarily relational) database management systems [6].

A solution to the problem of retrieving multimedia objects is to associate the objects with a specific description [7]. This description allows us to make a retrieval by similarity. How this description occurs depends on the type of object. There are two possible ways to describe an image: through metadata or visual descriptors [6]. Content-based image retrieval (CBIR) systems are a solution to use a visual descriptor. One of its primary purposes is to limit textual descriptions, using the image content to compare images in the retrieval process. Moreover, novel data structures, such as knowledge graphs [8], could combine different data by improving their informative layers [9]. In the realm of the semantic web [10], linked data is a way of publishing structured data that allows data to be linked together. The publication of linked data is based on open web technologies and standards, such as HTTP (hypertext transfer protocol), RDF (resource description framework), and URI (unified resource identifier). The purpose of this data structuring is to allow computers to read and interpret the information on the web directly. The links also make extracting data from various sources through semantic queries. When linked data link publicly accessible data, it is referred to as linked open data (LOD) [11].

In this article, we propose a system for camera picture augmentation on a mobile device that allows users to retrieve real-time information. Our framework is based on CBIR techniques, linked open data, the knowledge graph model, and deep learning tools. The augmentation task superimposes helpful information on the mobile screen. Further, a user is an active actor in our workflow because it can interact with the application to give feedback to improve future users’ experience. Furthermore, we employ deep-learning techniques to extract features from images. In particular, we use pre-trained convolutional neural networks (CNNs) as feature extractors. Moreover, we exclude classification layers from CNNs and applied max or average global pooling to have a one-dimensional feature to implement a vector-based similarity search.

We organize the rest of the paper as follows: in Section 2, we provide a review of the literature related to CBIR and augmented reality for cultural heritage; Section 3 introduces the used approach along with the general architecture of the system; in Section 4, a use case of the proposed system is described and discussed; in Section 5, we present and discuss the experimental strategy and results; eventually, Section 6 is devoted to conclusions and future research.

2. Related Works

Many scholars have extensively studied the idea of applying technology to cultural heritage. According to the UNESCO convention of 1970, “communication, audiovisual techniques, automatic data processing, and other applicable technology, as well as cultural and recreational trends should advance faster than presenting work” [12]. In this regard, augmented reality (AR) is one of the most cutting-edge technologies for the heritage area since it enables the creation of user-focused mobile applications [13]. Researchers argue that, in addition to being entertaining, visitor interaction with AR systems results in a richer CH experience [14], demonstrating experience co-creation in which the user and the provider work together to create a worthwhile result [15,16,17]. Furthermore, as mentioned in [18], AR can enhance cultural heritage sites. Visitors can explore strange environments excitedly and enjoyably [19]. Since many tourists are searching for distinctive and memorable on-trip experiences, this is one of the most significant benefits from the standpoint of supply.

Over the years, one of the technologies that has spread most in the artistic domain is augmented reality. Further, many works studied it from different points of view. In the framework of cultural heritage, the authors in [20] recognized augmented reality’s economic, experiential, social, epistemological, cultural, historical, and educational values.

Additionally, CBIR systems have a wide range of uses in cultural heritage. For example, in [21], the authors used it for 3D reconstruction and e-documentation of monumental assets.

Since most people have smartphones, providing them with this experience is incredibly inexpensive. We highlight [22], where the authors suggest a mobile app based on augmented reality to give users information on natural monuments. To further illustrate the information retrieval process, the authors of [23,24] created a CBIR system that captures images from a camera. They made use of these data to enhance the user’s displayed image. The authors of [25] introduced apps that take advantage of speeded-up robust features (SURF) and virtual reality to view data and 3D representations of cultural heritage monuments. The authors demonstrated the suitability of the features obtained using a scale-invariant feature transform (SIFT) detector for an augmented reality application in [26]. In their paper, Rodriguez et al. developed an ORB-based mobile image recognition-based augmented reality (MIRAR) framework architecture [27]. A similar ORB, a quicker approach for extracting and comparing features, is used in other papers by authors, such as [28]. The authors in [29] discussed the findings of experiments carried out during certain museum visits, together with an application that uses augmented reality and retrieval. These tests examine how such a tool can alter visitors’ educational experiences. The authors in [30] provided a method for developing mobile augmented reality (AR) applications to improve the experience of visiting cultural places by offering details and interpretations about the site’s exciting objects. The study described how different technology advancements, including mobile wireless technologies, augmented reality (AR), multimedia, and game technologies, have been combined in a cultural context to make relevant information fun and exciting.

Haugstvedt et al. [31] applied a technology acceptance model for hedonic systems to examine the determinants of intention to use an augmented reality application with historical information and pictures. The results reported in their work show that perceived enjoyment and usefulness are essential determinants of the intention to use augmented reality applications with historical information and pictures.

A case study from Italy is the “Ducale” research project, which suggests ICT solutions for the Palazzo Ducale museum in Urbino. It offers a mobile application that employs augmented reality and content-based picture retrieval to improve the experience of visiting museums. In this project, the well-known picture “Ideal City” is an example to illustrate a particular method for using digital media to preserve cultural property [32].

The discussed literature inspires our approach, but we address two weaknesses that they have. Firstly, we improved accuracy using the most recent and novel techniques based on deep neural network architectures, and lastly, we used a multimedia knowledge graph that is continuously populated, thanks to the users’ feedback. In particular, in our work, we proposed a novel method, where several descriptors are evaluated and used to achieve the best results in a real scene utilizing a mobile CBIR framework. We remove the classification layers from pre-trained CNNs, extracting deep features, which were reduced from three dimensions to one dimension, using a global pooling operation. Additionally, using augmented reality, linked open data are combined with the acquired photos to retrieve textual data and provide more information to users.

3. The Proposed System

In this section, we describe our approach. We improved a framework presented in our previous work [33], where we proposed an augmented realty process for cultural heritage that uses traditional feature extraction methods. The new novel framework contains some blocks that work offline and others that work online. The offline processes concern the multimedia knowledge graph (MKG) population, while the online ones implement the augmentation task. We populated the multimedia knowledge graph using a focused crawler for the cultural heritage domain [34]. The augmentation task works in real time, beginning with taking a picture of the mobile device camera and, through MKG and LOD, enriching it with the recognized textual details. In addition, we also considered the case where our MKG does not have the required management information with user feedback. With this strategy, the users contribute to adding new knowledge to our system, improving the experience of future ones. As shown in Figure 1, the main blocks are as follows:

Figure 1. System architecture.

• Image Loader: It loads images and preprocesses them as required from the feature extractor block with regards to the feature extraction techniques.
• Feature Extractor: It extracts features, so it takes in input of a processed image and outputs a feature vector.
• Feature Comparator: It compares the feature vector computed by the feature extractor and the feature stored in the multimedia knowledge graph. It puts on output the information used to augment the image.
• Augmenter Image: It applies augmentation using the information obtained from the feature comparator.
• Result Checker: It collects the user feedback and updates the results using LOD and MKG.
• Focused Crawler: It works offline, populating the multimedia knowledge graph and then updating and improving its content.

Furthermore, an essential component of our system is the multimedia knowledge graph [35]. It is based on cultural heritage ontology enriched with multimedia contents retrieved using a focused crawler and information extracted from linked open data. This information is employed in the augmentation task. In this work, we used deep features and ORB as local descriptors. As described above, for deep features, we use pre-trained CNNs. In particular, we chose VGG-net, residual network, inception, and MobileNet.

Concerning the pooling operation, in this work, we explored global average pooling and global max pooling. We consider a tensor of H × W × C, H is height, W is width, and C is the number of channels. The global average pooling computes the average for each matrix H × W, so the output is an array of C elements. Instead, the max average pooling calculates the max of each matrix H × W, obtaining a vector of C elements.

We briefly introduce the used CNN architectures and Oriented FAST and Rotated BRIEF (ORB) in the following.

• VGG16

The Visual Geometric Group of Oxford University proposed that VGG-Net [36] is a CNN architecture that achieved good results on the Large-Scale Visual Challenge (ILSVRC). Due to its easy architecture, it is one of the most used CNNs. The architecture involves convolutional layers with receptive filed (3 × 3), ReLU activation, and 2 × 2 max pooling layers after convolution. There are two versions, one with sixteen layers and one with nineteen. In this work, we used VGG16.

• Residual Network

Residual network (ResNet) [37] is a CNN architecture designed to mitigate the vanishing gradient effect in deep networks. The main innovation proposed in this architecture is the residual block, which adds shortcut connections. These kinds of connections skip one or more layers performing the identity mapping, and their output is added to the output of the stacked layers. The authors, over the years, proposed the architecture of ResNet with a different number of layers. In this study, we used a typical configuration with fifty layers.

• Inception

The authors introduced in [38] inception v1 and inception v2 architecture. Afterward, they proposed inception v3 [39], an improvement of v2 introducing the factorization concept. The main idea is factorizing convolution to reduce the number of connections and parameters without decreasing the network efficiency. The CNN consists of four modules implementing small factorization convolutions, factorization into asymmetric convolutions, and efficient grid size reduction.

• MobileNet

MobileNet [40] is a CNN architecture inspired by InceptionNet with the same optimization to work on mobile devices. The main contribution of this architecture is the introduction of depthwise separable convolution, which, working in two steps, firstly applies a single convolution filter for each input channel and then create a linear combination of the output using a pointwise convolution. In this work, we used MobileNetV2 [41], which is an improvement of the first version.

• ORB

ORB is a modified BRIEF descriptor based on a FAST keypoint detector. It initially employed FAST to identify the essential points. The top N spots are then determined by using a Harris corner measure. Because FAST is a rotation variation and does not compute orientation, it determines the intensity-weighted centroid of the patch with a corner situated in the middle. The vector’s direction determines the orientation from this corner point to the centroid. In order to improve the rotation invariance, moments are computed. In addition, ORB generates a rotation matrix from the patch’s orientation and then directs the BRIEF descriptors to point in that direction.

Augment Image Process

In Figure 2, we summarize the process of our system. The foremost step performed are image acquisition, feature extraction, image augmentation, visualization of augmented images, and results checking. The user interacts with the application in two cases, firstly to take a picture and lastly in case of wrong or missed matching. The process starts with a user that takes pictures on a mobile device using a mobile application. Then it sends the image to the server that performs prepossessing operations and extracts the features. Afterward, the server computes the similarity between the picture and the features contained in the multimedia knowledge graph. Then it sorts the results by similarity, performs augmentation superimposing textual information on the image, and sends information back to the mobile device. The user can suggest a resource if the server cannot find any machining. Instead, if the augmentation is wrong, the user can send feedback that the server uses to improve the experience of future users.

Figure 2. Flow chart.

In particular, to compare input pictures with images in the multimedia knowledge graph, we used cosine similarity defined as in Equation (1):

$$S_c(A,B)=cos\theta =\frac{A·B}{\parallel A\parallel \parallel B\parallel }=\frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=i}^n{A}_i^2}\sqrt{{\sum }_{i=i}^n{B}_i^2}}$$

(1)

We used Solr, a retrieval system that indexes the feature and the relations with the linked open data, to create our server application. Based on Apache Lucene, Apache Solr is an open-source search engine that offers indexing and searching tools. The targeted crawler fills our multimedia knowledge graph with linked open data. LOD, which depends on industry-standard web technologies including HTTP (hypertext transfer protocol), RDF (resource description framework) [42], and URI, is the publisher’s method of structured data that enables linking the data among them (uniform resource identifier). To obtain data from LOD, we used SPARQL [43]. A knowledge base called DBpedia was created using multiple organized bits of data from Wikipedia. For instance, according to the most recent statistics, the full DBpedia data set contains 38 million labels and abstracts in 125 different languages, 25.2 million links to images, 29.8 million links to other websites, 80.9 million links to Wikipedia, and 41.2 million links to YAGO categories. One of the most important online resources in this work was LOD DBpedia [44].

4. Use Case

In this section, we show a use case of our application. In particular, we describe two use cases: when the augmentation task receives positive feedback from the user (case 1) and negative feedback, but the user finds the correct matching in the first five results (case 2).

4.1. Use Case 1

In Case 1, we used David of Michelangelo Buonarroti as the picture we want to augment. The user takes a photo by mobile application (Figure 3a). It sends a shot to the server that computes a similarity with images contained in Multimedia Knowledge Graph and uses the information related to the more similar to augment the picture (Figure 3b). Then, it sends back information to the mobile application, and the user visualizes the correct information.

Figure 3. Use Case 1. (a) Took picture. (b) Augmented picture.

4.2. Use Case 2

In Case 2, we used the photo that depicts Apollo and Daphne of Gian Lorenzo Bernini. The user takes a picture using a mobile application (Figure 4a), then it sends a picture to the server that sends back the augmented picture (Figure 4b). The server sends to the mobile application the list of the first five best matchings (Figure 4c). The user can select the correct description from the items, and the application shows the right augmented picture (Figure 4d). The user sends negative feedback because the description does not correspond to the picture.

Figure 4. Use Case 2. (a) Took picture. (b) Augmented picture wrong. (c) List of five best matching. (d) Augmented picture.

5. Experimental Results

This section introduces the experimental strategy and shows the results with related comments and discussion. We used a vector space model to retrieve deep descriptors and key point matching for ORB. We computed the precision–recall curve and meant average precision to evaluate our proposed approach. The precision–recall curve is computed as an interpolation of the precision values for 11 standard recall values ranging from 0 to 1 step of 0.1. We used Equation (2) for the interpolation. Equation (3) shows the precision, and Equation (4) shows the recall:

$$P_{interp}\left(r\right)=\underset{r_i\ge r}{max}p\left(r_i\right)$$

(2)

$$Precision=\frac{|relevantdocuments\cap retrieveddocuments|}{\left|retrieveddocuments\right|}$$

(3)

$$Recall=\frac{|relevantdocuments\cap retrieveddocuments|}{\left|relevantdocuments\right|}$$

(4)

The mean average precision is computed as shown in Equation (5a):

$$mAP@k=\frac{{\sum }_{q=1}^{Q}AveP@k\left(q\right)}{Q}$$

(5a)

$$AvrP@k=\frac{{\sum }_{k=1}^n(P\left(k\right)\ast rel\left(k\right)}{numberofrelevantdocuments}$$

(5b)

In this work, we considered both the precision-recall curve and mAP. We used the first one to identify the best descriptor in image retrieval and the second one to figure out the best descriptor used as a classifier. It is important because we perform the augmentation using the information related to the first retrieved image. Furthermore, we computed the mAP@5 because we proposed the first five results to the user in case of negative feedback. We used a custom dataset of 100 query images to evaluate our system.

Figure 5 shows the results of the precision–recall curve, where the deep features achieved better results than ORB. Furthermore, the deep features have about the same precision. Due to this, we based our choice on MAP@1 and MAP@5. According to Table 1 and Figure 6, the feature extracted with MobileNetV2 and global average pooling as a reduction method achieved the best mean average precision at 5. Instead, MobileNetV2 and global max pooling as reduction methods achieved the best mean average precision at 1. Therefore, for our application, we chose ḿobilenetv2_avgéven if it has the second best MAP@1 and the best MAP@5.

Figure 5. Precision–recall curve.

Table 1. MAP@5 and MAP@1 for each features.

	MAP@5	MAP@1
inceptionv3_avg	0.8416	0.8716
vgg16_avg	0.8424	0.8824
resnet50_avg	0.8290	0.8590
mobilenetv2_avg	0.8586	0.9086
inceptionv3_max	0.8278	0.8878
vgg16_max	0.8417	0.8917
resnet50_max	0.8327	0.8827
mobilenetv2_max	0.8547	0.9247
orb	0.2949	0.3349

Figure 6. Mean average precision at one and five.

6. Conclusions and Future Works

This work introduced an application to augment pictures for the cultural heritage domain using deep learning techniques and multimedia knowledge graphs, improving our previous work [33]. The main differences are the introduction of a focused crawler to populate the multimedia knowledge graph and using more accurate feature extraction methods that achieved better results. The results show that the approach based on the deep learning feature extraction method achieved better results. We designed our application in a modular fashion, which is easily extensible with other descriptors or functionalities. Therefore, we proved that using pre-trained CNNs as feature extractors is a good solution for the cultural heritage domain combined with a multimedia knowledge graph to improve the user knowledge. Further, the feedback improves the result over time. In future work, we will improve the focused crawler to have more accurate results in our multimedia knowledge graph using novel techniques based on deep learning. Furthermore, we want to explore new representation learning methods to improve the feature extraction process, and we want to define and implement a strategy to analyze the user experience.