A Mobile App to Learn About Cultural and Historical Associations in a Closed Loop with Humanities Experts

Abstract

The Horizon 2020 project CrossCult aims to highlight historical and cultural associations between different characters, locations, events, venues, or artworks, to develop new strategies with which to promote intercultural and cross-border aspects of history and heritage. This paper presents a pilot app that provides graph-based visualizations of those associations, arranged by Humanities experts in relation to several reflective topics, and glued together by narratives that may present the same facts from diverse points of view. After querying a knowledge base that brings together several Linked Data resources, the associations are curated by the experts using a dedicated tool, to ensure that only meaningful associations appear on the mobile app. In turn, the app users can contribute new associations in the form of written text, which the experts can turn into new concepts and properties in the knowledge base. Here, we present the design of the mobile app and the experts’ tool, together with the results of early experiments aimed at assessing the instructional value of the proposal.

1. Introduction

History and cultural heritage are most often approached in an isolated and one-sided manner, which prevents from understanding the past as a shared and global experience [1]. However, the protection of cultural diversity and the promotion of intercultural dialog are among the most pressing contemporary challenges [2], and many scholars and policymakers have highlighted that the active usage of Internet-enabled mobile devices could be the key to realizing a shift to new models, that help people develop awareness of their own identities through an understanding of their own and others’ histories and cultures [3,4,5].

In this line, the consortium running the H2020 project CrossCult (www.crosscult.eu) has developed pilot applications aimed at highlighting historical and cultural associations between characters, locations, events, venues, or artworks. Different apps implemented different strategies to present those associations, and they were all evaluated for their potential to stimulate knowledge integration, reflection, and retention. Here, we present a pilot app that poses quiz games with graph-based visualizations of cross-border associations, arranged by Humanities experts in relation to several reflective topics, and glued together by narratives tailored to the points of interest for each venue. As illustrated in Figure 1, the app builds upon knowledge and content kept in a central repository (the CrossCult knowledge base, or CCKB), which is curated by the experts with the aid of a dedicated tool to ensure that only meaningful associations are delivered to the end users. In turn, the users can contribute new associations in the form of written text, which the experts can validate and turn into new annotations in the repository.

The paper is organized as follows. Section 2 provides an overview of the CrossCult project as a whole, plus specific details about the knowledge base that underpins our proposal. Then, the mobile app is presented in Section 3, looking at the key aspects of our approach to deliver stories, to visualize associations and to foster reflection. Section 4 describes the experts’ tool, along with the technological aids in place to help them in (i) identifying the most relevant associations in relation to selected reflective topics, and (ii) developing quiz tests involving entities that appear in the associations. Section 5 presents results about the perceived value of these solutions among smartphone users and Humanities experts from different collaborating venues. Finally, Section 6 summarizes our conclusions and describes our ongoing work.

2. CrossCult Overview

The CrossCult project has pursued the creation of an open technological platform for web/mobile applications to deliver historical and cultural contents, driven by cross-border associations and crosscutting topics. By harnessing recent advances in the areas of knowledge modelling for cultural heritage [6,7], personalization [8,9,10], visualization [11,12] and digital storytelling [13,14], the ultimate goal was to foster new ways for the citizens of EU countries and neighboring ones to appraise their past, understanding common traits and facts while embracing their potentially diverging viewpoints.

At the heart of everything, the CCKB —the central repository of knowledge of Figure 1—relies on the CIDOC Conceptual Reference Model [15,16], an international standard (ISO 21127:2006) that provides definitions and a formal structure for describing concepts and relationships used in cultural heritage documentation. The formal structure serves to break down the knowledge into semantic triples that codify individual statements in the form of subject-predicate-object expressions, to be read as “Ankara is the capital city of Turkey” or “Miguel de Cervantes authored ‘Don Quixote’”. Thereupon, the catalogue of classes and properties makes it possible to chain successive triples to represent more complex facts and come up with associations through common concepts, as in “Ankara is the capital city of Turkey, which is the successor of the Ottoman Empire, which Miguel de Cervantes fought as a soldier of the Spanish Navy Marines, founded by Charles I of Spain, who was also known as Charles V, Holy Roman Emperor”.

The project developed four pilots to assess different ways to highlight associations and foster reflection about history and cultural heritage (see [17]):

• Pilot 1 worked with the collection of paintings of the National Gallery London, providing itineraries tailored to the individual interests of each visitor.
• Pilot 2 (the focus of this paper) developed quiz games, stories, and associations between the Roman healing spas of Lugo (Spain) and Chaves (Portugal), the sanctuary of Epidaurus (Greece) and the archaeological site of Montegrotto Terme (Italy).
• Pilot 3 focused on enriching the visits in the small venue, the Archaeological Museum of Tripolis (Greece), by means of stories that bring together several exhibits in relation to current topics.
• Finally, Pilot 4 developed means to capture citizens’ reflections about the history and heritage captured in outdoor locations, driven by gamification elements that were first put into practice in the cities of Luxembourg and Valetta (Malta).

The CCKB was fed with knowledge from the aforementioned venues during the first semester of the project, back in 2016. Then, as shown in Figure 1, that knowledge was liaised with existing resources of the Linked Data initiative (linkeddata.org)—most notably, DBpedia (wiki.dbpedia.org) and Europeana (www.europeana.eu)—in order to benefit from billions of readily available triples and links to multimedia assets. As of October 2018, DBpedia alone contains more than 3 billion triples connecting more than 4.58 million entities, including 1.5 million persons, 735,000 places, 120,000 historical events, 550,000 artworks... plus 26 million links to images and 30 million links to external web pages. All that knowledge has been made available to the experts who use the CrossCult tools to develop the contents displayed by the Pilot 2 app, as we will explain in the following sections.

3. The Pilot 2 Mobile App

We designed and implemented the Pilot 2 mobile app as a generic player of experiences that bring together stories, quizzes, and associations between several cultural venues. The experiences are created by experts and stored in the CrossCult platform, in the form of a set of files with JSON format (one file per language). Figure 2 shows the starting screen of the app, which is reached after the privacy statement and an optional login screen. The figure is listing only the first experience created and tested for the aforementioned venues in Spain, Portugal, Greece, and Italy, which is titled “Water and health in ancient times”.

Upon selecting one experience, the user lands in a screen (Figure 3) that provides brief descriptions and visual contents about the different venues, with their locations displayed on an interactive map. There, the user can choose to play the quiz game together with other people visiting the same venue, or alone. Hereafter, we will explain individual gameplay only.

Once a user has started the game, she goes through different rounds focused on each one of the reflective topics chosen by the experts who created the experience. The “Water and health in ancient times” experience contains five reflective topics, namely “When and where” (general connections related to geography and chronology), “Human settlements and water”, “Water and health”, “Pilgrimage and communication routes” and “Material culture”. These topics are weaved together by a story (see Figure 4), that also gives the opportunity to display relevant visual contents.

The core of the game lies within the exploration of the associations among the selected venues, filling in some questions left blank by the experts. Figure 5 shows the mesh of concepts and connecting properties related to the reflective topic of “Pilgrimage and communication routes”, with four questions labeled ‘Q1’ to ‘Q4’. The user can drag the nodes around and click on the pale blue nodes to display a gallery of visual contents related to the corresponding concepts. If she clicks on the question nodes, a pop-up appears including a question title and a set of choices (Figure 6) that can be tailored to the users’ level of knowledge (more details in Section 4). The user’s score in the game is computed from the replies to these questions, throughout all the rounds of the game.

After each round of the game, the user is faced with a handful of highlights related to history and culture, provided by experts in relation to the messages they want to convey or the reflection they want to instigate. By clicking on the text, the user can again display a gallery of pictures and/or videos related to each fact. The user can give some quick feedback too, by clicking on one of the emojis of Figure 7 to indicate “that’s good to know”, “that makes me think”, “that’s surprising”, “I knew that already” or “who cares!”. This feedback was one of the sources of information for the experiments reported in Section 5.

Finally, when the user has gone through all the rounds of the game, her scores appear in relation to the Hall of Fame (Figure 8). This is used as an incentive to ask the user to provide new connections in the form of written text. The comments she types are sent immediately to the experts’ tool for validation, following automatic screening on the CrossCult platform for foul language or nonsensical writing. If the comments are approved, the user receives a notification, gets additional points, and sees how her updated score makes progress in the ranking.

4. The Experts’ Tool

Given a knowledge base with millions of semantic triples, it is generally possible to automatically find many different paths that connect a set of cultural venues, especially if we have rich annotations for their exhibits and we do not limit the maximum length of the chains of triples. Even if we discard trivial connections straightaway (see [18] for some of the criteria that may be applied), there is no guarantee that all those associations will be relevant and correct. In fact, DBpedia—as the formal counterpart of Wikipedia—has been criticized for the quality of the Humanities knowledge it stores [19,20,21], even if it is not usually worse than that found in traditional encyclopedias [22]. Therefore, qualified human supervision is needed to ensure that the mobile app delivers clear messages that favor reflection and retention phenomena.

In the approach we have followed in CrossCult’s Pilot 2, it is up to Humanities experts to specify the reflective topics pursued in a given experience, and to select the most relevant associations to display in the app. Our experts’ tool includes all the interfaces needed to collaboratively develop the contents of the experiences and to save the results in JSON files. It also provides association discovery mechanisms to retrieve potentially useful sequences of semantic triples from the CCKB, to identify the most relevant ones in relation to selected topics, to develop quiz tests involving the chosen entities (heritage items, characters, events, locations or artworks), and to enter new semantic triples in the CCKB, either owing to their knowledge about the venues, or by processing the comments from the app users.

To create a new experience, the experts must select the venues involved from a list of venues registered in the CrossCult platform. Then, they can specify different types of target users, in case they want to tailor the contents according to any demographic data (e.g., age or gender), to their level of knowledge about whichever subjects, or even to personality traits and cognitive/visiting styles—as explained in [23], the information about the users can be sought directly by means of profiling questionnaires, or retrieved from user profiles that keep track of their interactions with CrossCult apps in the past. For the case of our pilot experience “Water and health in ancient times”, the Humanities experts created a multiple-choice questionnaire to assess the user’s knowledge about the Roman Empire, which served to deliver different stories, different quizzes and different associations to knowledgeable users and not-so-knowledgeable ones.

Following the identification of venues and the definition of user types, the experts are faced with the main screen of the tool (Figure 9). In the upper part, there appear as many tabs as reflective topics have been defined, which can be edited and rearranged at will. Each reflective topic is characterized by a title and a set of keywords, which can be free text or selected from the normalized vocabularies of EuroVoc (http://eurovoc.europa.eu/). Optionally, the reflective topics may be limited to specific user types, too.

The mesh of associations that the users will explore in relation to a given reflective topic can be elaborated manually by the experts, by adding new nodes and edges, editing or removing existing ones, dragging around, zooming in or out, etc. A table on the left side of the screen presents the list of the semantic triples that make up the mesh for easier reference. Separately, the list can display additional triples suggested by the association discovery mechanisms, according to the knowledge captured in the CCKB (see [18] for details of their operation). When browsing the list, the experts can drag-and-drop individual triples or sequences thereof into the graph visualization. They can also tag triples as “worth considering” (thumbs up in Figure 9) or “irrelevant” (thumbs down) as indications to the other experts who collaborate in the development of the same experience.

Since the suggested triples may come in large numbers, they appear in the list ordered according to estimates of their relatedness to (i) the keywords that define the reflective topic and (ii) the keywords that define the expert’s areas of competence (also free text or borrowed from EuroVoc). Relatedness is measured by using the word embeddings of word2vec [24], which are huge collections of numbers reflecting which words appear commonly in close proximity to others in a Google News dataset. Specifically, we rate every sequence of semantic triples that takes from one venue to another in the CCKB according to Equation (1):

$$\begin{array}{ll}relatedness\left(Sequence,Topic,Expert\right)=& \frac{1}{\#\mathcal{E}\left(Sequence\right)·\#\mathcal{K}\left(Topic\right)}·\left({\displaystyle \sum _{e\in \mathcal{E}\left(Sequence\right)}\sum _{k\in \mathcal{K}\left(Topic\right)}}WCD(e,k)\right)·\\ & \frac{1}{\#\mathcal{E}\left(Sequence\right)·\#\mathcal{K}\left(Expert\right)}·\left({\displaystyle \sum _{e\in \mathcal{E}\left(Sequence\right)}\sum _{k\in \mathcal{K}\left(Expert\right)}}WCD(e,k)\right)\end{array}$$

(1)

• $WCD$: word cosine distance.
• #: the cardinality of a set.
• $\mathcal{E}\left(Sequence\right)$: the set of entities in a sequence of triples.
• $\mathcal{K}\left(Topic\right)$: the set of keywords linked to a reflective topic.
• $\mathcal{K}\left(Expert\right)$: the set of keywords in the expert’s profile.

Figure 10 shows the dialog that appears when an expert clicks on a node of the graph. Using that interface, she can link the corresponding entity to snippets of information that will make up the narrative of the reflective topic, including text and/or multimedia contents. Besides, the nodes can be turned into multiple-choice questions for the quiz game. The choices may be presented in the shape of text, image galleries, time lines or maps.

The alternatives to the correct entity may be entered manually by the experts or suggested by a recommender system running on the CrossCult platform, which simply picks from any relevant wikicategory that contains the entity. (Wikicategories are groupings of Wikipedia pages on similar subjects. A global index of such groupings in English can be found at https://en.wikipedia.org/wiki/Wikipedia:Quick_cat_index.) First of all, the relevance of the wikicategories in relation to the current reflective topic and the expert’s areas of competence is computed as indicated by Equation (2), where $\mathcal{C}\left(Wikicat\right)$ denotes the set of entities included in a given category:

$$\begin{array}{cc}relevance(Wikicat,Topic,Expert)=\hfill & \frac{1}{\#\mathcal{E}\left(Wikicat\right)·\#\mathcal{K}\left(Topic\right)}·\left(\sum _{e\in \mathcal{E}\left(Wikicat\right)}\sum _{k\in \mathcal{K}\left(Topic\right)}WCD(e,k)\right)·\hfill \\ & \frac{1}{\#\mathcal{E}\left(Wikicat\right)·\#\mathcal{K}\left(Expert\right)}·\left(\sum _{e\in \mathcal{E}\left(Wikicat\right)}\sum _{k\in \mathcal{K}\left(Expert\right)}WCD(e,k)\right)\hfill \end{array}$$

(2)

By this equation, the recommender system identifies the four most relevant wikicategories, and then it puts all the entities they contain in a unique set, from which it can pick alternatives whenever the expert clicks on the “Get recommendation” button. The selection can take into account the levels of knowledge indicated in the definition of user types (if any): compared to higher levels of knowledge, lower levels yield alternatives which are further from the correct entity in terms of $WCD$. Thus, for example, owing to the inclusion of Miguel de Cervantes in the wikicategories of “Spanish novelists” and “Roman Catholic writers”, the recommender may propose Vicente Blasco Ibáñez, Miguel Delibes or Ramón del Valle-Inclán as alternatives to the author of ‘Don Quixote’ for the knowledgeable users, and Dante Alighieri, Paulo Coelho or The Singing Nun for the not-so-knowledgeable ones.

5. Evaluation

The mobile app and the experts’ tool have been assessed in collaboration with the venues involved in the “Water and health in ancient times” experience, through experiments conducted between October 2017 and March 2018. On the one hand, the assessment of the mobile app looked at the feedback gathered through the emojis interface of Figure 7, the textual comments provided in the last screen (Figure 8) and the replies to a paper-based questionnaire, that would take up to 15 min to fill in. A total of 58 participants were recruited from among students of the Universities of Vigo, Padova and the Peloponnese, their ages ranging from 20 to 60, but with a predominance of young and middle-aged profiles, with frequent use of mobile apps.

The 58 users tried the app in one only session each, lasting for an average of 35 min. Then, they proceeded to fill in the paper-based questionnaire, which contained 31 statements to rate on a Likert scale, plus 8 questions requiring textual answers (about the user experience, the features they liked the most and the least, suggested improvements, etc.). The participants were asked to rate the 31 statements, and accordingly we got 98% of the expected ratings (a rating of “Neutral” was assumed for the gaps). In contrast, the textual answers were left as optional, and we gathered only 23% of the maximum number of replies. The anonymized dataset yields the following main results and observations:

• While the users were faced with an average of 11.7 highlights in the app, they would rate only 5.3 with one of the emojis—either because they did not read the rest of the highlights or they did not find any of the emojis to be representative enough. From among the replies, 67.6% of the users would choose interested (“that’s good to know”), reflecting (“that makes me think”) or surprised (“that’s surprising”) emojis; 27% of the users would click on the smirking (“I knew that already”) face, while only 5.4% would express indifference or lack of interest (“who cares!”).
• 91.4% of the users who reached the end of the experience (i.e., who went through all the reflective topics) would provide textual comments at the screen of Figure 8, and the average number of comments provided by those was 3.1.
• According to the data of

  Table 1. Summary of replies to the main questions about the mobile app.

  	Disagree	Neutral	Agree
  I enjoyed using the app	25.7%	43.1%	31.2%
  The story is a nice driver of the whole experience	11.3%	42.4%	46.3%
  The story made me reflect about the topics of the experience	11.5%	30.5%	58%
  The graphs of concepts and associations were informative and easy to understand	38.7%	34.6%	26.7%
  The graphs of concepts and associations made me reflect about the topics of the experience	36.9%	28.6%	34.5%
  The multiple-choice questions in the game made me reflect about the topics of the experience	5.4%	31%	63.6%
  I liked reading about historical facts and giving my feedback via emojis	6.2%	50.3%	43.5%
  The historical facts made me reflect about the topics of the experience	8.1%	33.8%	58.1%
  I enjoyed providing my reflection comments and my own associations in the end	28%	41.3%	30.7%
  I ended up with a richer understanding of life in the ancient times	3.5%	61%	35.5%
  The app triggers my curiosity about the distant venues	14.6%	45.8%	39.6%
  I think the app can enrich the experience of visiting one of its cultural venues	0%	30.2%	69.8%
  I think the app can be enriching if used BEFORE visiting one of its cultural venues	6.3%	25.8%	67.9%
  I think the app can be enriching if used AFTER visiting one of its cultural venues	14.5%	41%	45.5%
  I think the app is much more suited to tablets than smartphones due to the larger screen size	7.3%	18.1%	74.6%
  I think the app could be an interesting tool in a classroom, with teachers browsing/explaining the contents and playing the game as per the students’ instructions	0%	24.6%	75.4%

  , nearly $\frac{1}{4}$ of the users had negative opinions about the app, and more than $\frac{1}{3}$ did not perceive much added value in the graph visualizations of concepts and associations. This datum correlates with the indication that it would be better to use the app on a larger screen than offered by a smartphone, and that things would be much easier to understand with a teacher or guide browsing/explaining the contents. In contrast, the features related to the stories, the multiple-choice questions and the reflection points were appealing to an average of 90% of the users.
• 35.5% of the users indicated that they ended up with a richer understanding of life in the Roman period, while 61% had a neutral opinion. The little textual feedback gathered in this regard suggested that the statement was probably too ambitious for the relatively short time spent with the app. Besides, there is the fact that more than 78% of the users who completed the profiling questionnaire in the app were rated as knowledgeable ones, which suggests that our sample of University students already had a decent historical background.
• Finally, owing to the data gathered from the paper-based questionnaire, nearly 70% of the users agreed that the app can enrich the visit of one of its cultural venues (preferably, if used before the visit), while 40% said that it can also instigate curiosity about the distant ones. Disagreement with these statements was null or very low (14.6%).

The feedback gathered during these experiments is being used to update the implementation of the mobile app and to complete its contents. The goal is to upload the final version to the Apple Store and Google Play by February 2019, so that it can be openly advertised to visitors of the four participating venues. More extensive evaluation will be conducted during the year too, with a broader spectrum of participants recruited from actual venue visitors.

On the other hand, a small team of Humanities experts have rated aspects of perceived value of the tool to develop the experiences on a Likert scale from 1 (lowest) to 5 (highest). Despite their lack of familiarity with graph visualizations and formal artifacts for knowledge modelling, we got the following average evaluations: 3.33 for “the tool is intuitive and easy to use”, 4.17 for “the tool is useful to develop connections among multiple venues”, 4.17 for “the recommended associations are useful to develop the reflective topics” and 4.00 for “the tool facilitates the creation of multiple-choice questions that foster reflection”. It is worth noting, however, that only 12% of the labels included in the nodes of the graphs and on the connecting arrows contained the exact text retrieved from the CCKB. This means that the information in the knowledge base is not in the shape to be offered directly to the app users, which provides further justification for adopting an approach as illustrated in Figure 1, with a tool for experts mediating between the app users and the information repositories. This preliminary evaluation will be extended during 2019, both in the elaboration of the updated contents for the “Water and health in ancient times” experience and in the creation of new sample experiences.

6. Conclusions and Future Work

Notwithstanding the fact that the sample of users was small and unbalanced regarding the levels of previous knowledge about the topics of the pilot experience, the evaluation of the mobile app has yielded some findings that we consider useful towards building and publicizing the final version in 2019. The overall concept of the app is appealing, with the users apparently more interested in the Humanities content related to the cross-border associations among the venues, than in the graph-based visualizations of such associations. The meshes of concepts and connecting properties turned out to be especially hard to appraise on small screens. For the final version of the app, we are testing new mechanisms to unclutter the visualizations, and we are also considering supplementing the graph-based representation of associations with a table-based approach.

Regarding the experts’ tool, the fact that the associations retrieved from the knowledge bases inspired the definition of the reflective topics and helped start developing their contents (even if the experts ended up modifying around 90% of the text labels) can be taken as positive evidence that the concept is useful and accessible to people who are not familiar with the underlying technologies and knowledge representation models, and that the Linked Data resources brought into the CrossCult knowledge base can support the activities of Humanities experts.

In the future, we plan to check whether different formulae than those of Equations (1) and (2) would allow the technological aids of the experts’ tool to provide better results, e.g., involving geometric or harmonic averages, thresholds, and some notion of popularity for the terms in the Wikicategories. Likewise, we want to assess the impact of using different word embeddings, like those of ConceptNet Numberbatch [25] or word2vec itself but trained on collections of documents focused on cultural heritage and history (such as the ones gathered by Europeana Collections).

On the non-technical side, we are interested in defining metrics to analyze the achievements more deeply in relation to promoting reflection about cultural heritage, triggering curiosity and increasing retention. The grounds for such a study can be found in [26,27,28]. In particular, we want to gain insight into how those cognitive phenomena (and even emotional ones) can be fostered with a cross-border perspective by controversial characters or events that are viewed very differently in different countries, such as the figures of Gavrilo Princip (who is seen as a national hero by many in Serbia, whereas History textbooks in many other European countries present him as the villain who sparked the start of World War I) or Mustafa Kemal Atatürk (who was honored by UNESCO and the United Nations as a promoter of peace between nations, but is held responsible by many Greeks for acts of repression against minorities). We believe that narratives based on commonalities and differences, developed through the collaboration of an international team of experts, can explain the point of view of “the others” and, thereby, boost the kind of mutual understanding and intercultural dialog that we pursued since the inception of CrossCult.