Personalization in Digital Ecomuseums: The Case of Pros-Eleusis

Abstract

In comparison with a traditional museum, an “ecomuseum” is radically different: It is not housed in a building and does not have a collection of physical objects or artifacts. It aims to help visitors discover the tangible and intangible cultural heritage of a region through the identification of important points of interest (POIs), while offering a variety of activities and direct engagement with the region’s cultural identity. The diversity and amount of information that may be available through digital means highlight the need for supporting the visitor in selecting which POIs to visit by offering personalized content. In this paper, we present our approach for a recommendation system for an ecomuseum, through its application in the city of Eleusis, Greece. We present the approach from needs to implementation, as well as the results of a preliminary evaluation, showing promising results for its application as an engaging visitor experience for an ecomuseum. We conclude the paper with a wider discussion about personalization in this context and in a cultural heritage context in general.

1. Introduction

Personalization is defined as “the ability to provide content and services tailored to individuals based on knowledge about their preferences and behavior” [1]. Starting from e-commerce applications, in the past decades, it has known widespread usage across many sectors, including health and entertainment. The constantly increasing volume of digitized information and multimedia content, combined with the increasing ubiquity of Internet-enabled mobile devices, has made personalization all the more relevant: users rely more and more heavily on content sharing and consumption through the Web, whereas deciding amongst the alternatives has become increasingly difficult, with context-aware personalization being proposed as an effective solution [2].

Following this trend, cultural heritage (CH) has been an early active domain for personalization research. Ardissono et al. [3] and Pavlides [4] identified and reviewed several applications that attempt to put in practice research and innovation in the field of personalization and recommendation. These are designed for indoors, outdoors, as well as virtual visits to cultural heritage sites.

For more than four decades, gallery, library, archive, and museum (GLAM) institutions have been engaged in a continuous effort for digitization of their collections [5]. This extensive investment into mass digitization [6] has resulted in millions of digital heritage assets. GLAM institutions are now expected to facilitate the retrieval and reuse of this content under the FAIR principles [7] and proceed to either making it directly available online or using it to support the production of interactive digital experiences, which have become widespread in most institutions [8]. As a result, search, exploration, and retrieval have already become challenging for all users, including visitors and experience creators, and the need for personalization as a means to counter data overload has become all the more crucial [9].

Personalization is based on the assumption that the application can understand the users and their needs to provide the most relevant content [10]. Modeling the user, including their interests, prior knowledge, and preferences, as well their current context, and then eliciting their profile based on this model, is one of the main challenges of personalization in CH [3]. As visits to cultural heritage sites are mostly one-time events for the users, and, in some cases, the time dedicated to the visit is limited (in extreme cases, even only a few minutes [11]), a thorough, direct recording of the user profile might not be possible, so the “cold start” problem still remains challenging for CH recommendation applications [10]. Consequently, user profiling along with the proper characterization of the digital CH content in terms of themes and topics for recommendation purposes remain amongst the main challenges of personalization in this domain.

Personalization research in CH has been focusing on tackling the aforementioned challenges and exploring ways to inform recommendation system designs that will be meaningful in a CH context. GLAM institutions are eager to adopt new technologies; however, these technologies need to be stable and effective enough to be offered as added-value products to the visitors.

In this paper, we report our findings after putting into practice a recommendation approach for the ecomuseum of the city of Eleusis, Greece. An ecomuseum is focused on the local identity and natural and cultural history of a place [12], with a strong focus on local community participation, having sustainable development amongst its main objectives. The ecomuseum approach exceeds the concept of outdoors city guides, which highlight the main CH sites or sites, by attempting to create diverse communication pathways between the visitors and the local community. Its objective is to promote deeper understanding of the past and present of a city and create meaningful connections that will promote revisit.

The digital ecomuseum application of the city of Eleusis has been designed to cater to the diverse visitors of the city, allowing them to experience the city, both past and present, in a way tailored to their needs. The application offers a great amount of geolocalized digital content for many of the city’s main landmarks, spanning across different themes. As a result, recommendation of content has been identified as an important need to support the visitor in optimally navigating the city, both digitally and physically.

In this paper, we report on the design of a recommendation system for the ecomuseum of Eleusis. The recommendation system is offered to city visitors as the “Personalized Map View” option in the Pros-Eleusis digital ecomuseum mobile app. Through this use case, we present the identified challenges and lessons learnt of putting personalization into practice in this specific CH context.

We firstly present our motivation in Section 2 and briefly describe the digital ecomuseum of Eleusis and our adopted recommendation approach. Then, in Section 3, we situate our work within the wider domain of personalization for CH and beyond. Next, we focus on the specifics of the mobile application (Section 4) and the recommendation system, including user and content modeling (Section 5) and the algorithm (Section 6). Finally, we present the methodology (Section 7) and results (Section 8) of the evaluation of the application, concluding with a discussion on our main finding and future work (Section 9).

2. Study Motivation

Originating from France in 1971 [12], the ecomuseum concept has been universally acknowledged and used as an innovative approach for the holistic preservation, interpretation, and management of tangible and intangible cultural and natural heritage [13].

An ecomuseum is different from a traditional museum in the sense that it is not housed in a building and does not have a collection of physical objects or artifacts. It allows visitors to discover the cultural and environmental identity of a defined geographical territory by combining information provision and the engagement of the local community.

Visitors have the opportunity to explore the city, accessing indoor and outdoor monuments and points of interest in general and discovering local cultural and natural heritage through its carefully designed content, combined with the possibility of contacting representatives of the local community and experiencing the continuity of culture from past to present.

As the ecomuseum approach extends the connection of the visitor with cultural heritage beyond the boundaries of museums and historical sites, it can become an interesting use case for experimentation in the domain of personalization and recommendation. An ecomuseum is a complex system combining past and present through a network of places and people, which are also constantly updated, with new locations added or new content available for these locations. An ecomuseum can offer the amount and diversity of content that would warrant the use of a recommendation approach.

In this study, we used the ecomuseum of the city of Eleusis as our case study for the design of a recommendation approach. Eleusis is situated about 18 km northwest from the center of Athens. It has a continuous history dating back to several centuries BCE and a strong involvement in the political and religious life of Ancient Greece as well as a particular industrial recent history. Having become a point of fusion between different cultures and the center of controversy in relation to its difficult environmental issues, Eleusis has a lot to offer to a visitor interested in the distant and recent past.

The Eleusis Digital Ecomuseum App

The research project Pros-Eleusis (Personalized Routes in Eleusis—https://proseleusis.com/, accessed on 15 December 2022) explored ways to engage visitors with Eleusis, adopting a digital ecomuseum approach to digitally display the natural and cultural heritage, both tangible and intangible, of the city in situ.

The project focused on the design and implementation of software tools, content, and methodologies for authoring and experiencing personalized or predefined thematic routes in a city where the past and present meet, to simultaneously provide a path toward a sustainable future for the area. The ecomuseum of Eleusis unfolds exists for numerous points of interest within the city, relevant to its older and more recent history, its industrial past, its complex mix of local inhabitants, as well as its current cultural characteristics.

The Eleusis digital ecomuseum app aims to address the challenge of engaging visitors with the rich content created for each POI in the city, allowing them to wander in the city in a way more suitable to their preferences and to form a deeper connection with its rich heritage, feeling the desire to revisit. The general concept of the app revolves around offering the visitor rich informational content spanning across time and space as well as different options for activities within the city. The digital content is structured around a set of points of interest (POIs) in the city, which may include open spaces or buildings with different functions: archeological or other heritage sites, museums, town squares, monuments, historical buildings, spaces of ecological or environmental interest, local societies, and even restaurants and cafes.

This content is available to the user through the digital ecomuseum app. The app was designed to take into account the user’s position through GPS and deliver content in three different ways:

• Thematic routes: Different itineraries composed of a series of POIs are proposed to the user with a common theme, such as “Ancient Eleusis”. The route itinerary is fixed, proposing a certain order of visit to the user (Figure 1 left).
• Map view: All POIs are presented on a map, allowing the user to select and be directed to those of interest (Figure 1 middle).
• Personalized map view: The users provide some details about their profile and are offered a view similar to the map view; however, at each point, only a limited number of POIs are presented: those recommended by the system as the most relevant, taking into account visitor preferences and current context (Figure 1 right).

In the next section, we briefly present other recommendation approaches in the domain of cultural heritage. Then, we focus on the personalized map view of the Pros-Eleusis digital ecomuseum app.

3. Related Studies

Personalization and recommendation in cultural heritage are active research topics, explored in a variety of use cases and application types, from on-site mobile based guides to virtual museum applications. Pavlidis [3] and Ardissono et al. [4], among others, have presented comprehensive reviews of applied use cases in this domain. In this section, we present a brief overview of the approaches that are relevant to our own, discussing how they have informed our study.

Benouaret and Lenne [14] proposed a recommender system for mobile-based museum tours that adapts to the user preferences and is sensitive to their context (location, time, expertise, etc.). The system combines a semantic approach for knowledge representation using ontologies and thesauruses with a semantically enhanced collaborative filtering method to recommend artworks in the museum. They followed a hybrid approach combining user demographics with semantic and collaborative methods. The cold start issue is tackled during the visitor’s first use of the system by collecting their profile (age, gender, language, expertise in art, etc.). Then, as the visitor views and rates artworks, the semantic method is activated. Finally, when the system has built information about the user, the collaborative filtering approach is integrated. Contextual post filtering enables the generation of an optimized visiting path based on the location of the visitor, the physical environment, and the time the visitor wants to spend in the museum to ensure an optimum visitor experience and minimize fatigue. The authors’ proposed semantic context model specifies five categories of contextual information: individuality, activity (i.e., user gives rating to artwork), relationship (i.e., users perform similar activities), temporality (i.e., the time they want to spend at the museum, time spent for each artwork, and the remaining visit time), and location. The Pros-Eleusis digital ecomuseum app uses a graph data model that partially captures the same variety of information as in [14], such as ratings and features of POIs without using ontologies and thesauruses, while incorporating important contextual information such as ticket cost. We also used the notion of similarity but as a content-based similarity between different POIs and not in a collaborative filtering setting, as Pros-Eleusis is not limited to a specific location (such as a museum). It is broader, allowing users more freedom and thus increasing the difficulty in applying CF techniques. Furthermore, we did not treat constraints as part of a post-filtering technique but rather uniformly as part of the ranking problem.

The authors of [15] proposed a recommender system for POIs in a museum, which is trained with different models to produce either a ranking of unseen POIs or a single next POI as a recommendation. The recommendation problem is cast as a binary classification problem, where relevant (visited) POIs are labeled as one and irrelevant (unvisited) ones are labeled as zero. The authors took into account user demographics, user concept preferences, and onsite graph walk metrics that are treated as context, and used POIs and context pairs to train a behavioral model to predict recommended unseen POIs. They implemented a logistic regression classifier and deep neural multilayer perceptron to estimate the relevance of each POI to a given user. These models are separately trained to study which user information interaction behavior is more effective in understanding user preferences in their interactions with the IoT in smart environments. The data model is different from ours, as the authors did not use explicit user feedback (i.e., ratings); their notion of context is different and is not explicitly handled as in our case (i.e., price and time). Moreover, they used path–graph measures as similarity over routes, and the algorithmic methods used are different.

In [16], recommendations were viewed as a binary classification task in a cultural heritage environment, where artifacts are annotated with static and nonstatic information, and the classifier decides if an artifact is interesting for the user. The static content involves folksonomy, where artifacts have already been characterized/curated, while nonstatic content consists of user-generated content (UGC). The authors enhanced an already existing content-based recommender with the UGC. User tags are collected during the training step by letting users express their preferences for artifacts by entering a numerical rating and by annotating them with free-form tags. The recommender learns the user profiles and implements a naïve Bayes text categorization algorithm, which is able to classify the artifacts as interesting or not for a specific user. The main outcome is an improvement in the predictive accuracy of the tag-augmented recommender system compared with that of the traditional content-based approach. Our approach, in addition to the different data and computational model, implements a CF aspect while using additional contextual signals and constraints that are more likely to happen in a real-world scenario.

In [17], the authors envisioned a mobile context-aware tour guide. Knowledge of the user’s current location and a history of past locations are used to provide services that we expect from a real tour guide, which, according to the authors, plays the role of cartographer, librarian, navigator, and messenger. In that sense, the tour guide’s interface and usage are different from ours. Our method is not focused on recommending POIs based on user activities, but mostly opening up a platform of communication for the user to navigate the physical space and potential POIs of interest on their own.

In [18], the authors present an agent-based system that acts as an expert tourist guide for mobile users while integrating different forms of adaptation (e.g., the device being used, the user needs and preferences, and the current user location). The user provides an initial profile and provides feedback throughout the visit so that the app can dynamically change recommendations. The application of the tourist guide is for the city of Turin. The system maintains a profile of the user, including their interests, preferences, and the history of their previous visits to Turin, and exploits this profile in order to tailor its suggestions to the user preferences. It includes explicit data (sociodemographic data, such as age, gender, and profession; general interests; and data related to the visit, such as duration and free time). It also includes inferred data, such as propensity to spend and specific interests. The system also considers a set of further parameters, such as the location of the user, the time of the day, and the fact that the user is moving, and adapts the interaction while taking them into account. Even though the system overlaps with our method on some of the signals used to produce the recommended ranking, the data model and the algorithmic methods used are different, as our proposed method is a rule-based approach and calculated scores are assigned by these rules.

A strategy was proposed [19] that enables a content-based recommender to create user profiles by applying learning techniques on typical item descriptions, i.e., static content, provided by a publisher and on tags adopted by the users to annotate relevant items. The proposed method was applied as a usage example to cultural heritage personalization. Static content and UGC tags are analyzed by linguistic techniques to capture the semantics of user interests. The goal of this strategy differs from ours because the authors mainly wanted to see if the integration of this content would be useful to a content-based recommender system.

In [20], the authors present models that predict a visitor’s interests and next locations in a museum on the basis of observed behavior. Three models were proposed: one is an interest-based CF approach that predicts a visitor’s next location based on their interest in unseen exhibits, which is estimated from the time the visitor spends at the exhibits they have seen, the second approach predicts a visitor’s next location from the trajectories of other visitors, and the last one is an ensemble model that combines the predictions of the previous two models. The authors proposed producing two different rankings based on the interests and on the user trajectories and then combining them to produce the final recommendation list. Our strategy uses a variety of signals, including ratings and contextual constraints, different from that in [20]. Additionally, we have a different data model and we use a different algorithmic approach overall to produce a list of recommendations.

We present an overview of these approaches in

Table 1. Summary of existing recommendation approaches in CH and their corresponding characteristics, which are marked with a “√”.

Use Case	User Model		Algorithm-Related			Update		Input Signals
	Graph	Other	Cold start	CB	CF	Implicit	Explicit
Benouaret et al. [14]		√	√	√	√		√	Ontologies, user demographics, location, time, expertise
Hashemi et al. [15]		√		√		√	√	On-site walk graph metrics, demographics, content tags
Basile et al. [16]		√		√			√	Folksonomy, user tags, user ratings
Abowd et al. [17]	√			√			√	Location, map, POI description
Cena et al. [18]		√	√	√		√	√	Demographics, user preferences, user history, device, context (location, time, user movement), POI category, POI textual description
De Gemmis et al. [19]		√		√				Folksonomy, content description/tags, user tags
Ferrara et al. [20]		√			√	√		Time spent in POIs
Pros-Eleusis	√		√	√	√	√	√	User ratings, content tags, time, location, ticket cost, user history (visited POIs), skipped content

, focusing on the following aspects and compared them with the Pros-Eleusis ecomuseum:

• User model type: graph or other.
• Recommendation algorithm aspects: if the approach is content-based (CB), collaborative filtering (CF) and/or if it directly addresses the cold-start problem.
• User profile update method: implicit (e.g., visited, etc.) or explicit (e.g., rating, liked, etc.).
• Input signals: the different types of information that are used as input in the recommendation process.

4. Personalized Map View

The foreseen scenario of the use of the personalized map functionality of the Eleusis ecomuseum app can be summarized as follows:

The visitor arrives at the city of Eleusis, either parking their car or at the central bus station. Turning to the app for guidance, they fill in their profile as well as their preferences for the current visit in terms of available time, etc. (see Table 2 for the user profile). If they have used the app in the past, the recommendation system remembers their profile in terms of visited POIs, ratings for specific POIs, etc. The system, taking into account all available user profile information including the user current position, proposes an initial set of 10 POIs to the user. The user then has the ability to select the one they would like to visit first (Figure 2 left) and receives directions to reach it.

When arriving at the POI, the user may consume the multimedia content at will (Figure 2 middle); if they wish, they may rate the POI (Figure 2 right). Then, the system recalculates the list of recommended POIs and presents them to the user on the map. The recommendation is dynamic, taking into account each time the POI the user has already visited as well as the user’s new position. In this sense, the system does not propose a complete route from the start but rather adjusts to the user actions as the visit unfolds.

These visitor actions are recorded in the user profile so that the system can take into account this new information to provide more accurate recommendations the next time the user visits the city. The already visited POIs are not recommended to the visitor a second time.

To implement this scenario, we followed an iterative design process from requirements to implementation, in some cases having to revert and adjust previous steps. The first step was performed to attempt to more deeply understand visitor needs, informing different aspects of the recommendation system design, including the user and content (POI) model and the overall recommendation approach. We then proceeded with the design and implementation of the recommendation algorithm and system, which was evaluated and improved throughout its design and development.

5. User and Content Model

A good understanding of the user is important for any recommendation system. In the case of our ecomuseum application, the user model was iteratively refined by identifying and examining the visitor characteristics that may be relevant.

Visiting an ecomuseum that spans across a city is an activity that may take different forms according to user preferences and to external factors such as the weather or the time of day. The visitors may make the decision to walk between POIs or use their car, enter museums or monuments to explore, simply decide to lounge outside on a bench if the weather permits it, or even take a break at one of the many city restaurants or cafés. We identified the following user characteristics that may be relevant to visitor preferences, split into three main groups:

User Mobility. A prominent characteristic to take into account for recommending POIs is accessibility. If a point of interest is not easily accessible for a specific user due to its location or lack of adequate support and facilities, it should be excluded from the recommendations. To model accessibility, we followed a simplified approach, taking into account user mobility, leaving them to record whether they can move anywhere or have limitations resulting from being in a wheelchair, having other issues, or even escorting young children in a stroller.

Interests. Generic user interests relevant to the content offered are an important part of visitor preferences. Detailed interests in terms of preference for a historical period or topic are not straightforward to implicitly deduce, so we modeled such preferences by leveraging POI similarity. More specifically, the recommendation algorithm examines similarity with existing POIs that the user liked in order to recommend the next ones. In this case, the user model maintains the POIs the user has visited along with the rating the user provided for them. There are two user characteristics that could be directly declared by the user. One is their interest to interact with local city residents to better understand the local perspective, and the second is their interest in focusing on the highlights of the city, those monuments or places that are generally considered a “must-see”.

Visit Context. This category contains characteristics that may not be fixed but rather change depending on the visit context. These are relevant to the specific visit or even a specific time and may include available time for the visit, visiting alone or in a group, etc.

5.1. User Model and Profiling

The final list of user characteristics included in the user model was defined through an iterative process. The value of specific characteristics may be key in recommending or not a specific POI, representing hard constraints in the process. Others affect the recommendation process, adding to the strength of a specific recommendation. The user characteristics may also be grouped into explicit or implicit, where explicit represents those that the users are asked to fill in themselves through the application, whereas implicit are the preferences deduced by the system taking into account specific visitor actions. The application offers the possibility for the user to fill in the values of the direct profile characteristics in the app user profile page.

The user model is composed of six user characteristics relevant to the user context, demographics, and preferences. We summarize the user characteristics in Table 2, including their range of values.

Table 2. User model.

	Characteristic	Type	Value	Constraint	Elicitation
1	Time available for the visit	Visit context	Value in minutes	SOFT	Explicit: The user is presented with a dropdown menu with time durations (30, 60, 90 min, etc.)
2	Maximum cost that visitor is willing to pay for entry	Visit context	Maximum cost in EUR	SOFT	Explicit: User is presented with a dropdown list with different cost values in EUR
3	Preference for city highlights	Preferences	Yes/No	SOFT	Explicit
4	Accessibility	User mobility	Options available: I can go anywhere, I have a baby in a stroller, I am in a wheelchair, I have limited mobility	HARD	Explicit
5	Meeting with locals	Visit context	Yes/No	SOFT	Explicit
6	Visited POIs	Visit context	List of POIs the user has already visited	HARD	Implicit: Recorded automatically by the system in the user profile

Table 2. User model.

	Characteristic	Type	Value	Constraint	Elicitation
1	Time available for the visit	Visit context	Value in minutes	SOFT	Explicit: The user is presented with a dropdown menu with time durations (30, 60, 90 min, etc.)
2	Maximum cost that visitor is willing to pay for entry	Visit context	Maximum cost in EUR	SOFT	Explicit: User is presented with a dropdown list with different cost values in EUR
3	Preference for city highlights	Preferences	Yes/No	SOFT	Explicit
4	Accessibility	User mobility	Options available: I can go anywhere, I have a baby in a stroller, I am in a wheelchair, I have limited mobility	HARD	Explicit
5	Meeting with locals	Visit context	Yes/No	SOFT	Explicit
6	Visited POIs	Visit context	List of POIs the user has already visited	HARD	Implicit: Recorded automatically by the system in the user profile

5.2. Point of Interest Model

The user model characteristics were matched to corresponding POI characteristics, designed to support the need for recommendations according to the user characteristics. This resulted in a content model for the POIs with eight relevant characteristics (

Table 3. POI characteristics to support recommendations.

Variable	Value
Location	Latitude, longitude
Minimum time needed to visit the POI	Time duration
Opening hours	Weekly schedule of opening hours if applicable
Ticket cost	Price in EUR
Possibility to meet local residents	Boolean (yes/no)
Accessibility	2 options: Accessible to all Not accessible
Preference for highlights	Boolean (yes/no)
Tags	Set of labels

).

These characteristics are defined as follows:

Location: The POI geographical location in GPS coordinates on a real-life map.

Minimum time needed to visit the POI: The minimum duration is the time needed to listen to the content for this POI.

Opening hours: This field is optional. It is filled in if applicable for the specific POI.

Ticket cost: This field is optional. It is filled in if we have relevant information for the specific POI.

Possibility of meeting local residents: Filled in if applicable for the specific POI.

Accessibility: Following a simple model for accessibility, the POI is characterized as accessible if it is possible for visitors with constrained mobility to comfortably visit either as an outdoor or indoor space.

Highlights: This characteristic denotes if the POI is considered a highlight, a “must-see”, of the city for visitors.

Tags: A set of labels that represent concepts relevant to one or more POIs. They were created as a way to define content similarities between the POIs that could later be used by the personalization system during the similarity calculation process.

The process of creating these tags was iterative. We asked three history experts with knowledge of the city to read texts that described the points of interest and annotate them using either concepts from the text or general concepts that they considered relevant. We gradually collected these concepts in a single vocabulary. At first, this vocabulary was a simple list of labels, but we realized that a hierarchical structure would make it easier to read and comprehend. To decide how to organize the terms, we studied several controlled vocabularies that are widely used in the field of cultural heritage. We devised six general categories: time, society/life in society, natural environment, home and family life, people, professional life, and abstract ideas and concepts. For the final structuring of our vocabulary, we relied on the Social History and Industrial Classification (SHIC) [21], ICONCLASS [22], and Art and Architecture Thesaurus (AAT). For the terms in the time category, we adopted the class hierarchy of the CIDOC Conceptual Reference Model [23].

The combination of concepts from more than one existing vocabulary was deemed necessary because we realized that the tags we had created expanded over a wider range of topics than each one of the studied vocabularies covered. The resulting vocabulary aims to include most, if not all, aspects of life in a city during both its present and past.

The user and POI models were implemented through an appropriate data model in the system, and the recommendation app offers the possibility to the user of recording their profile characteristics through an intuitive user interface.

6. Recommender System

In this section, we discuss different aspects of our recommendation system. Before detailing our method, we present an overview of our approach. It roughly consists of the following steps:

• Initialization:
  • The similarities between POIs in the system are calculated.
  • The user’s profile is initialized with the characteristics (see User and Content Model Section) as provided in the app and their current position in the city.
• Calculation of preference for candidate POIs:
  • The preference weight for any POI is calculated based on a different set of factors. Generally, the weight is increased when:
    • It fits the user profile;
    • Its position is close to the user’s;
    • It offers a good interest-to-cost ratio, which is applicable only if the users in their profile have declared a maximum cost for this tour in terms of entrance tickets for POIs, etc., and the system calculates the amount spent already while visiting the previous POIs;
    • It offers a good interest-to-time ratio, which is applicable only if the users in their profile have declared a maximum duration for this tour. In this case, the system takes into account the expected time needed to visit a POI and the already elapsed time for the visit.
  • Constraints are applied, and if the resulting recommendations (i.e., POIs) do not fit the constraints, they are removed from the results.
• The recommended POIs are presented to the user on the personalized map view in the app.
• The user selects a POI to visit.
• After visiting the POI, the app asks the user to rate it, denoting a preference score. If the user decides not to rate the POI, the POI is assigned a neutral rating.
• The system calculates the new recommended POI as described in step 3.

6.1. Recommender System Graph Schema

Personalization and recommendation for the Pros-Eleusis app were modeled as a graph traversal problem. As such, the Pros-Eleusis digital ecomuseum personalized map view utilizes a graph-based recommender system that uses a graph with different types of nodes and edges in its core (Figure 3).

The nodes are divided into three categories, (a) users, (b) POIs, and (c) features (characteristics). (POI and feature nodes can be collectively referenced as object nodes). The nodes can be connected through edges:

• User nodes can be connected to POI nodes via HasInterest edges.
• POI nodes can be connected to feature nodes via HasFeature edges.
• POI nodes can be connected with other POI nodes via SimilarWith edges.

The edges are determined by a source node, a target node, and some weights. They are divided into three categories: (a) HasInterest, i.e., edges denoting the user’s interest in a POI; (b) SimilarWith, i.e., edges denoting the degree of similarity between two POIs; and (c) HasFeature, i.e., indicating that the POI has the characteristic described by the respective feature.

Each edge can be associated with different weight values:

• For the SimilarWith edges, there is a weight that corresponds to the degree of similarity between two POIs;
• For the HasFeature edges, there is a weight that corresponds to the degree to which the feature characterizes the POI;
• For the HasInterest edges, there are two weight values, namely strength and certainty:

  ○

  The strength value is associated with the system’s prediction of the intensity of the user’s interest in an object and takes values in the continuous interval [−1, 1];

  ○

  The certainty value relates to the reliability of that prediction, i.e., how confident the system is in the validity of this relation. It takes values in the continuous interval [0, 1].

An edge between a user and a POI indicates the system knowledge of the user’s interest in that POI.

Figure 3 presents an example of this process. To find the interest of a user in POI₃, firstly, we calculate the similarity of POI₁ and POI₄ to POI₃, accounting for the common features Feature₁ and Feature₂. We then aggregate the edges User—HasInterest→POI₁—SimilarWith→POI₃ and—HasInterest→POI₄—SimilarWith→POI₃ to predict an indirect HasInterest edge between the user and POI₃.

6.2. Pros-Eleusis Recommendations as Graph Traversals

With this data structure in place, we wanted to ask questions such as “how much does the user like a specific point of interest P?”. Because we have no prior knowledge to answer this question (i.e., our graph does not contain an explicit edge between the user and that specific POI), our system has to discover paths between the user and this POI by traversing and combining edges to derive an explicit estimated HasInterest edge between them at the end of the computation.

6.2.1. Overview of the Recommendation Algorithm

In order for the system to determine the user’s interest in a point of interest P for which it has no prior knowledge, it follows this algorithm:

• Find all other POIs for which the system is aware of the user’s interest in them (i.e., there is an outgoing HasInterest edge directly connecting the respective nodes). If there are no ratings (cold-start case), then assign the HasInterest edge with neutral rating to every POI.
  • For the POIs in step 1, explore each one’s outgoing edges to determine the existence of transitions (i.e., SimilarWith edges) to node P.
  • Keep the POIs that share common features with P.
  • Use their shared features (i.e., feature nodes), the user’s known interest in them, and a similarity metric to estimate a preference prediction for P.
• The final system prediction of the user’s interest in P is a combination of (a) the intensity of the user’s interest in POIs for which the system has knowledge of (i.e., there are explicit interest edges toward them) and (b) the degree of similarity between these POIs and P. Basically, the system has to create a new HasInterest edge between the user and P (i.e., the system’s prediction). This process takes place for all POIs for which the system does not have explicit knowledge of the user’s interest in them (i.e., through explicit feedback).
  • If some of the generated recommendations do not meet the constraints set by the user’s profile, they are filtered out, while bonuses or penalties related to the user’s preferences are applied to the remaining ones.
  • The list of POI predictions (i.e., predictions of the user’s interest in points of interest) is sorted in descending order of interest, and then the top 10 are made available to the user for consumption.

6.2.2. Offline System Updates

(a)

Creating SimilarWith edges between POIs

In order to derive the similarity edges between the POI nodes in our graph, we turned to classification methods in the literature and, in particular, to k-nearest neighbor (KNN) techniques [24].

Using KNN in our method translates as follows: we are given two sets of POIs, A and B. Our system calculates the similarity between each POI in set A with each POI in set B by calculating their Cartesian product. This procedure takes place as a preprocessing step. The similarity measure is configurable and can be changed for a variety of other metrics. We chose this metric to be the cosine similarity, defined as follows:

$$\begin{gathered} SimilarWith\left(PO{I}_a, PO{I}_b\right)=\frac{{\sum }_{\forall f \in F_A \cap F_B}^{ } PO{I}_{a,f}\ast PO{I}_{b,f}}{\sqrt{{\sum }_{\forall f \in F_a}^{ } PO{I}_{a,f}^{ 2} \ast {\sum }_{\forall f \in F_b}^{ } PO{I}_{b,f}^{ 2}}} \\ iff SimilarWith\left(PO{I}_a, PO{I}_b\right)\ge threshold,0 otherwise \end{gathered}$$

where F_x defines the set of all features of POI_x, POI_x $\in$ POIs, and POI_x,f defines the affinity value for feature f of POI_x.

We compute the similarities between POIs and combine them with known signs of user interest to estimate their interest in any POI in question. In addition, for the generated bipartite graph to remain relatively small and therefore easier to manage, similarities between POIs with a value less than a threshold are discarded. For experimental purposes, the threshold value was set to 0.1.

(b)

Creating HasInterest edges between a user and a candidate POI

When the system requests a recommendation for a user U and a POI P, this is internally translated as retrieving and composing known HasInterest and SimilarWith edges between U and a set of POIs (different from P) to create a path from U to P. This process results in a new, weighted HasInterest edge between U and P. This edge has two associated weights, namely strength and certainty, that are formally calculated as described in the following paragraph.

Suppose we want to find the interest (i.e., a weighted HasInterest edge) of user U to a POI P. Moreover, suppose that P is associated with similarity edges with M other POIs, M ⊆ POIs. A subset N ⊆ M of these POIs are connected with HasInterest edges to the node representing U.

The strength of the edge between U and P is calculated as follows [25]:

$$strength\left(U, P\right)=\frac{{\sum }_{I_k}^{ } HasInteres{t}_{\left(U, P_i\right)} \ast SimilarWit{h}_{\left(P_i, P\right)}}{{\sum }_{I_k}^{ } SimilarWit{h}_{\left(P_i, P\right)}}, P_i \ne P, P_i\in POIs,$$

where

$$\begin{gathered} I_k=kma{x}_{similarity}\left(SimilarWith\cap HasInterest\right), k \le 10, k \in \mathbb{N} \\ SimilarWith=\{ p_j | p_j \in POIs, p_j is similar with P \} \mathrm{and} HasInterest=\{ p_k | p_k \in POIs, U is interested in p_k \} \end{gathered}$$

$kma{x}_{similarity} \mathrm{is}$ a function that returns the $POIs$ of $SimilarWith\cap HasInterest$ with the highest degree of similarity

Additionally, we associate every HasInterest edge with a certainty value indicating the system confidence for the intensity of the predicted user interest. In the context of this method, this was chosen to be:

$$certainty\left(U, P\right)=ma{x}_{certainty}\left(SimilarWith\cap HasInterest\right),$$

If the user rated a specific POI, then the certainty is set to 0.5. In the case where the user dismisses the dialog box and essentially provides neutral feedback, certainty is set to 1.

(c)

Top-k POI ranking

Our system implements the described algorithm to estimate the interest in every POI in the knowledge base (KB) and rank them in descending order of user preference. This procedure combines the following steps:

• Predict the user’s interest (i.e., a weighted HasInterest (strength, certainty) edge) for every POI in KB that they have not yet visited. Visits are persistent across application runs.
• Order results in a descending order of strength, using certainty to resolve ties.

Note that we consider two intensity values to be equivalent if |v_1-v_2| ≤ d. For our experiments, d was set to 0.1.

(d)

Time and cost bonuses

For the system to promote POIs that have good interest-to-user available time and budget ratios, we consider two snapshots of the knapsack problem [25] per each proposed POI. For each POI in question, we consider a snapshot of the continuous knapsack problem with a capacity of W. We aim to find a solution for which the items (i.e., the POIs) inserted in the user backpack maximize the user’s profit. We distinguish two cases where this approach is useful: (a) maximizing the ratio of interest to the minimum time to visit a POI and (b) maximizing the ratio of interest to the cost of a ticket to visit a POI. The problem is defined as follows:

We need to maximize

$$\begin{array}{l}{\sum }_{i \in POIs}^{ } x_i{w}_i \le W, x_i \in \left\{0,1\right\}, \mathrm{while} \mathrm{also} \mathrm{maximizing} \mathrm{the} \mathrm{total} \mathrm{profit}\\ {\sum }_{i \in POIs}^{ } x_i{v}_i, x_i \in \left\{0,1\right\}\end{array}$$

In addition, we consider the following on a case-by-case basis:

For (a) $W=available budget, w_i \equiv ticketCos{t}_i, v_i=\frac{interest.strengt{h}_i{}_{ }}{ticketCos{t}_i}, i \in POIs$.

For (b) $W=available time, w_i \equiv timeRequire{d}_i, v_i=\frac{interest.strengt{h}_i{}_{ }}{timeRequire{d}_i}, i \in POIs$.

For the needs of the user study presented in this paper, we applied the following weights:

• Location: 30%;
• User budget to spend on tickets, etc.: 5%;
• Time available to dedicate on the visit: 5%;
• Interest to interact with the local community: 5%;
• Preference to prioritize the city highlights 5%;
• System prediction: 50%.

The weights are set in such a way that POIs that are both relevant to the user’s preferences and close to them in terms of location.

6.2.3. Dynamic System Updates

Dynamic updates to the user profile and context variables are triggered by user data collection and event processing. Specifically, the system receives data on the user’s preferences based on (a) the visitor profile questionnaire provided by the ecomuseum app and (b) the feedback that the users provide to the system through their interaction with it. The user provides feedback to the system (a) by choosing to skip an object and (b) by consuming a POI and providing explicit feedback to the system through content rating.

More specifically, our system receives updates upon user actions (e.g., content consumption, user position, etc.). The user can either consume an object and then be asked to rate it or skip the rating process. By rating a POI, users provide explicit feedback on how much they liked or disliked that item. To that end, the system provides a 10-star rating pop-up dialog in which the users can rate the respective content after successfully visiting it. Values between 0 and 4 are considered negative feedback, 5 is considered neutral, and values between 6 and 10 are considered positive. Internally, values are normalized in the continuous interval [−1, 1]. Omitting an object has a neutral connotation. Specifically,

• Omitting a POI is interpreted by the system as a neutral value of the user’s interest in it. For this system, this value was set to 0.
• Successful content consumption leads to the appearance of a feedback dialog in which the user can rate the related content. The score given by the user to a POI corresponds to the intensity of their interest in it.

7. Evaluation Methodology

The evaluation of a recommendation approach in a cultural heritage context is not a straightforward and simple process. Firstly, the preference of the user, declared as a rating of a specific POI, may be a result of several different factors, including the offered multimedia narrative content itself, the POI as a physical location, or even conceptual connotations and aspects of the combination of the two. Furthermore, to fully evaluate the recommended option, the user should be made aware of all the other options that the recommender system may have considered and rejected. In an evaluation context, this is impractical as it will ultimately lead to participant fatigue, which will affect the evaluation results.

Here, we opted to approach the recommender system evaluation from a visitor experience perspective. We still attempted to examine the recommendation accuracy as perceived by the participants; however, we focused on assessing how this affects the overall experience of a personalized ecomuseum visit. To this end, we invited our evaluation participants to assess the personalized map view of the app while visiting the city of Eleusis. Our evaluation objectives are summarized as follows:

Objective 1: Assess the effect of the recommendation approach on visitor experience.

Objective 2: Record the perceived effectiveness of the recommendations.

7.1. Process

The following process was followed while proceeding with the evaluation:

• Participant recruitment through online invitations.
• Meeting with the evaluators in the city of Eleusis and introductory activities:
  • Introduction to the study objectives and process;
  • Brief presentation of the Pros-Eleusis app;
  • Signing consent forms;
  • User profile questionnaire.
• City tour using the personalized map view—the user was escorted by one evaluator.
• Questionnaire and interview.

For participant recruitment, an invitation was sent to the our academic and personal mailing lists for adult individuals that would be interested in visiting Eleusis and touring the town with the support of the mobile application. They were invited to bring along friends or family members if they wished.

Participants were invited to arrive in Eleusis to meet with the evaluators at pre-agreed dates and times. After a brief introduction regarding the objectives of the study and the process, they were given a brief questionnaire recording participant information, including age, gender, whether they had visited Eleusis in the past, and general interest in history and other topics relevant to the ecomuseum themes, as well as how able and accustomed they were with walking tours. They were then asked to freely use the personalized map view of the mobile app and explore the city. They were advised to use the app at their own discretion and decide on the duration of the tour, possible breaks, number of POIs visited, etc. The evaluators explained the fact that the app offers recommendations for POIs to visit and that not all available POIs are visible at the same time on the map in personalized map view. The participants were also made aware of other two functionalities of the app, thematic routes and map view; however, they were asked not to use them.

Each visitor was discreetly escorted by one of the evaluators, who recorded user comments and issues with the app during the tour. When they felt that they would like to end the tour, they were interviewed by the evaluator escorting them (see Appendix A) about their experience and given a brief questionnaire to record their general opinion of the tour as well as more specific user experience aspects in relation to the application, content, and recommendation approach.

Apart from using the interview and questionnaire as means to record the visitor assessment of the tour, the application was designed to log data related to the user choices and visited POIs. These are presented in the following section.

Logged Data

During the tour using the personalized map view, specific information was anonymously recorded for each user. This was relevant to the itinerary and duration of the tour, as well as to the recommendation algorithm and included:

• The ordered list of visited POIs;
• Time spent at each POI;
• The total number of visited POIs;
• Total visit duration;
• POI recommendation system score;
• Indication if the participant rated the POI;
• Participant rating for the POI.

7.2. Participants

Nine adults (six women and three men) responded to the invitation. A total of 3 were between 20 and 30 years old, 3 were between 30 and 40, 1 was between 40 and 50, and 2 were above 50 years old. Four of the participants were visiting Eleusis for the first time, whereas the rest had been briefly to the city a few times in the past for business purposes. The majority of the participants were used to covering distances on foot daily. They also reported wider interests when visiting a new place beyond its local history, including the natural and urban environment and local cuisine (Figure 4).

In terms of their familiarity with Eleusis, most participants reported little to average knowledge. On a Likert scale (1, completely disagree to 5, completely agree), in the case of familiarity with ancient Eleusis, the average score was M = 2.75 with a standard deviation SD = 0.89. Similarly, in the case of modern Eleusis, the results were M = 2.63 with SD = 1.06; in the case of familiarity with the issues that the residents of Eleusis face, M = 2.75 with SD = 1.16. These results are shown in Figure 5.

8. Results

The questionnaires, log files, interviews, observations, and participant comments during the tour were recorded and analyzed. The results in terms of the effect of the recommendation approach on the visitor experience and the effectiveness of the recommendations are presented in this section.

8.1. Effectiveness of the Recommendations

The study sample size and relevant evaluation data did not permit the accurate calculation of the effectiveness of the recommendation algorithm. The evaluation, however, offered qualitative indications and insight as to this effectiveness. In this section, we report the relevant results.

Firstly, in terms of the user-perceived effectiveness of the recommendations, as we can see in Figure 6, the majority of the participants felt that the recommended POIs were interesting (S1) and mostly felt that the offered content was relevant to them (S8).

Six out of the nine participants felt that the app was indeed recommending POIs during the tour, two because they noticed that their previous POIs were taken into account and four because they saw the POIs changing (Figure 7).

Although the number of participants in this study was not sufficient to determine the effectiveness of the recommendation algorithm, Figure 8 provides insights. In most cases, we noticed that the predicted and actual score seemed to converge, if not in actual rating, then at least in the positive or negative nature of the score.

8.2. Visitor Experience and Recommendations

Generally, it is challenging to identify the degree the recommendation functionality contributed to the visitor experience in comparison with other factors such as the overall app interface, the content, and the mood of the participant at that time. In this section, we first report the overall outlook of the participants for the tour experience and then attempt to examine the role of recommendations to the user experience.

The number of visited POIs and total visit duration are presented in

Table 4. Number of visited POIs and total visit duration per participant.

Participant	Visited POIs	Total visit duration
P1	8	1 h 7 min
P2	11	1 h 47 min
P3	8	1 h 9 min
P4	7	56 min
P5	8	44 min
P6	7	1 h 2 min
P7	7	37 min
P8	4	28 min
P9	5	35 min

. The visitors, on average, visited seven POIs and spent 1 h for the whole visit. Participant P8 commented that she would have stayed longer but she had to return home earlier due to an emergency. The participants rated all the POIs that they visited, except P7 who, unintentionally, skipped the rating for the first POI she visited.

Overall, the general impression of the participants of the tour was positive. For the majority of the participants, walking did not feel tiring. Only two participants felt somewhat tired after the walk. As shown in

Table 5. Average score and standard deviation for the questionnaire statements. The statements were on a 5-point Likert scale from completely disagree (1) to completely agree (5).

Statement	Average and Standard Deviation
S1. The experience proposed POIs that were of interest to me.	M = 3.78, SD = 0.83
S2. The experience made me observe the POIs more closely.	M = 3.89, SD = 0.78
S3. I learnt something important about the city of Eleusis.	M = 3.33, SD = 0.5
S4. The experience inspired me to want to learn more about Eleusis.	M = 3.56, SD = 1.01
S5. I enjoyed the experience.	M = 3.89, SD = 1.05
S6. I had the sense that time passed quickly.	M = 3.78, SD = 0.67
S8. The content felt relevant to me.	M = 3.56, SD = 0.54
S9. I would have liked to have the experience together with another person.	M = 3.89, SD = 1.27
S10. I would use this app again when visiting Eleusis.	M = 4.11, SD = 0.93

and Figure 9, most users enjoyed the experience (S5 and S6), they felt that the content was important (S3) and that it helped them more closely observe the POIs (S2).

Most visitors considered the user experience of the personalized map view as “pleasant” and “relaxed”. They felt that it was appropriate for a city tour where you can combine the tour with breaks for coffee and food. The app, in this case, complements and supports wandering freely in the city rather than imposing a specific path or even guiding in a strong, apparent way. As one user commented, “the personalized view can be used as an aid to support wandering and exploring, that you can use as you feel like”. This approach is in line with the needs of an ecomuseum visit. The free and dynamic exploration of the city can be combined as the users desire, with stops to experience local cuisine or sample traditional products. Another visitor commented that the interaction with the recommendation system helped her explore POIs that she maybe would not have otherwise: “Having this list of 10 most recommended POIs helped me focus and choose amongst them. In the beginning, I selected 3–4 houses in a row, as I have a special interest in the architectural style of the specific period. And I noticed that the app picked up this interest of mine and continued recommending houses and neoclassical buildings. But then, amongst the recommended POIs, I also noticed a Byzantine church, so I decided to visit it, and then visited a second one, recommended to me. This type of dynamic, step by step recommendation helped me adjust the visit to the mood of the moment and not get stuck only in what I normally look for”.

During the interview, the participants were asked to compare the personalized map view with the other two options: the map view, showing all the POIs on the Map, and the thematic routes. Participants considered the concept of the thematic route experience type as more guided and appropriate when interested in a specific topic or aspect of the city and when having enough available time for it. The map view, on one hand, provides the freedom to choose amongst all available POIs; however, for some participants, it was considered to be “too chaotic” (Figure 10). Of the rest, two felt confident that they preferred it to the personalized one: the fact that the personalized map view “hides” part of the available POIs from the user can also be considered as negative. The user, at each time point, is presented with a list of POIs, who then selects one to visit. After the visit and the adjustment of this list by the system, some of the POIs that were available in the previous list and interested the user may no longer be available. According to these users, it would be useful to be able to keep these POIs for a later time.

Τhe remaining five users mentioned that they would like to be able to see all the POIs instead of only the recommended ones; however, they wanted to view them on demand or highlighted with a different color, less prominent than the recommended ones.

One user who was in favor of the personalized map view commented: “On one hand [with the personalized map view], you do not feel the pressure to follow a specific route or even complete it, and on the other, you do not get so many points on the map, as with the map view. The map view can really be overwhelming, it made me thing ‘How am I going to visit all these places?’”.

8.2.1. Community Engagement

The Pros-Eleusis app was designed to explore digital support for an ecomuseum. A vital part of the ecomuseum concept involves not only considering the location as a museum without concrete boundaries but also including the local community in the process. The COVID-19 pandemic has directly hindered the in-depth assessment of the community engagement aspects of the approach. Several of the included relevant POIs were either closed or operating under special restrictive conditions at the time of the study. In this sense, the study participants could not fully experience the visit of POIs where they could directly interact with local societies and organizations offering different activities such as workshops or seminars about local customs or even sampling local products.

In this sense, the assessment of this aspect was handled through a presentation and relevant discussion of these features with the users during the interviews. It was a common comment of all users that they would not normally expect from an application such as this to recommend POIs related to engaging with the local community. All found the idea of recommending POIs where they could sample and/or buy local products interesting. Four of the participants mentioned that they would not normally seek to participate in a local community activity while visiting a city such as Eleusis as a daily excursion. However, they would be interested to know if such an activity were available early on to be able to pass by or design their day on the fly: “If there was for example an interesting concert or an outdoors activity like shadow theater, which is nice for the children, I would have like to know in advance, at the start of the visit, to decide and make a time plan”. This participant and four others mentioned that they had not thought of such an activity while setting the profile option to engage with the local community. They suggested a more detailed presentation of the available types for such engagement in the user profile, including: attend a concert, participate in a workshop for local cuisine, buy local products, or even visit a traditional restaurant. It was also suggested that such options, if recommended by the system, should not take into account the distance factor but rather be recommended early on so that the user can take them into account while looking for an activity apart from sightseeing or a place to shop for souvenirs or take a break.

8.2.2. Proposed Improvements and Functionality

The users also proposed future extensions for the personalized view functionality of the digital ecomuseum app. These included the following:

Content recommendation within the POIs: The majority of the users felt that most POIs offered a large amount of diverse information, at times spanning across different topics, time periods, etc. Some users proposed to also apply recommendations within the content of a specific POI, allowing the users to rate specific parts of the content and, thus, the recommender system to collect more precise information about their preferences. This is an approach that needs to be further evaluated in future studies.

“Pinning” or demoting a POI: Some participants proposed ways to have more control over the recommended POIs: “I would like to be able to keep some of the recommended POIs so that the app does not make them disappear from the map”. The recommended POIs list could offer the option to keep a POI “pinned” for the next set of recommendations. Additionally, in case it is not interesting at all, they could choose to remove it from the list of recommended POIs, with the possibility of it appearing again later on if a significant part of the POIs have already been visited.

Explicitly show that recommended POIs change: Some participants felt that it should be made more evident that the recommended POIs change after visiting a POI. One proposed design was to use animation to show new POIs gradually appearing while others change color or fade and gradually disappear from the map.

Highlighting the need to rate: Two participants mentioned that they failed to rate a POI by mistake. They would have liked a confirmation of the type “Are you sure you are not going to rate this POI?” before moving forward. They both commented that they liked the corresponding POI and felt that if they had rated it the following recommendations could have been more accurate.

Most-recommended POI: Some users, both during the tour and when interviewed, expressed interest in knowing what the system would propose as the most suitable POI for them to visit amongst the list of proposed POIs. They also wondered about the criteria for the specific recommendations. At the moment, there is no evident differentiation amongst the 10 recommended POIs as to which is the most highly ranked. It would be interesting to examine the effect on the user experience of highlighting this most-recommended POI or showing the ranking of all the POIs. Color coding could be used to denote differences in the recommendation strength in the manner of a heat map to differentiate highly from the moderately recommended POIs.

Nearby POIs: There were participants that felt strongly against going back and forth when walking around to visit POIs in locations they had already passed. To this end, they suggested that it would have been useful to be able to know on demand which are the closest POIs to the one they are currently visiting, recommended or not, to avoid moving back and forth between locations.

Support for group visits: The participants also discussed the need for the app to support group visits. As is often the case, such visits are made by a family or group of friends together. Using the app individually may result in participants having to separate to follow different routes for different POIs. Although some participants felt that one device could be used per group, and the next POI could be negotiated amongst the group, others felt that it would be more practical to use an individual device with the app. In this case, the app would need to negotiate the individual visitor profiles to identify the optimum recommendations for the whole group.

Make the options for engaging with the local community more explicit: Most participants felt that the presentation of POIs where different types of interaction with the local community were offered was not straightforward enough. They proposed more detailed options offered to the participant at the user profile level as well as different handling of these types of options at the recommendation level. They mostly distinguished between POIs (1) where they can participate in an activity such watching a show or a talk or participating in a workshop; (2) buy local products; and (3) find traditional spots to have a coffee, eat, and take a break from the tour.

9. Discussion and Conclusions

In this paper, we report our findings after putting a recommendation approach for the ecomuseum of the city of Eleusis, Greece, into practice. The Pros-Eleusis digital ecomuseum application of the city of Eleusis offers geolocalized digital content for many of the city’s main landmarks, organized in three different ways: thematic routes (guiding the visitor to follow a specific path and visit POIs in a proposed order), a map view (allowing access to all available POIs), and a personalized map view, which dynamically recommends at each POI the 10 most relevant ones to visit next. Throughout the process of collecting requirements, and designing and evaluating the recommender system and the personalized map view functionality, we were able to reflect on the challenges of the different steps as well as more general issues related to the concept of recommendation in the domain of cultural heritage. Our study could provide an incentive for a more holistic understanding of personalization and recommendation in this field, taking into account experience design aspects as well as the ultimate objectives of cultural heritage visits.

The first question when considering to offer a recommendation functionality in an app supporting a museum or a cultural heritage visit, in general, is whether recommendation is really meaningful. A main criterion supporting this decision is the available content. Although the digitization efforts in the past decades have produced thousands, even millions, of digitized artifacts, this content is mainly addressed to experts to support research objectives. Digital content needs to be carefully selected, curated, and edited to be offered to the wider public. This process results in applications that offer digital content designed for a specific visit duration. In most cases, this content is organized in themes or in routes that propose a specific path through the museum, city, or cultural heritage location. The Pros-Eleusis ecomuseum app also follows this approach through the thematic routes functionality.

For a recommendation approach to be meaningful, the available digital content should be (1) organized in individual, independent units (POIs); and (2) large enough in number so that different subsets can be selected to be offered to different users. In the case of the Pros-Eleusis ecomuseum app, this is the case. More than 50 POIs are offered to the user in the map view, spanning across different historical periods, locations, and topics, making it very difficult for a visitor to experience them all in one visit. So, in this case, the Eleusis digital ecomuseum content was suitable to consider as a test case for a recommendation approach.

Having established that there was enough and diverse content available to justify a recommendation approach, the next challenge was to design the matching the user profile, context, and preferences with the content characteristics. This is also one of the main challenges when attempting to apply recommendations in cultural heritage. The relationship of a visitor and a cultural heritage object is more complex than that of a consumer looking for a specific product. Especially in the case of an ecomuseum, this object is a POI, which is a physical location combined with digital content offered by the app and forming a composite cultural heritage object. Computing the degree of interest of the user on this POI is not straightforward: Will the user like or dislike the POI because of the location itself and its esthetic value or interest to the user, or will the offered digital content, informational content, and narrative style weigh more heavily on the user rating? What are the user characteristics that can be used to predict these content preferences?

In the case of a digital ecomuseum, user context elements such as available time, amount of money to pay for entrance, and proximity in terms of location are straightforward constraints to guide recommendation. The more subtle and difficult aspects to record in the user profile and predict are those related to user preferences in terms of content themes and the attitude of the user toward culture and art. An important factor that makes the profiling process even more challenging is the cold-start issue: in the context of tourism or cultural heritage visits, the users are not willing to dedicate a long time to filling in profile information, so any recommendation algorithm usually has to proceed with minimum knowledge about the user. In the case of the Pros-Eleusis digital ecomuseum app, we opted to take advantage of the profiling information offered by the user when setting up the app to record the more straightforward user preferences that play the role of hard or soft constraints including, among others, available time for the visit, mobility issues, or a preference of seeing important highlights in priority.

We purposefully decided against including the selection of thematic concepts from the concept hierarchy (the tags associated with the POIs) in the user profile. Taking into account that the ecomuseum app already offers the possibility of selecting thematic routes, we opted to design the personalization functionality as a more free, explorative visitor experience. The visitor does not have to declare in advance what historical period or themes they would like to explore. Rather, as the visit progresses, the recommendation algorithm proposes a new set of POIs that is influenced by the visitors’ dynamic preferences and mood at that moment. An outcome of the evaluation that merits further investigation in future versions of the app is to offer possibilities for the recommendation of content available inside one POI. As some users noted, the content design of this app offered within the POIs a variety of content spanning, in some cases, different themes and historical periods. In this sense, it would be interesting to examine whether the content tagging by experts as well as the recommendation would be meaningful at this narrower level.

As already mentioned, the main objective of the recommendation approach in this context is not to identify the optimum POI according to the visitor profile but rather to offer an overall playful, pleasurable, and transformative experience connecting with the city and its ecomuseum. The application becomes a tool to provoke curiosity and support individual exploration (“I would not visit this POI by myself, but since it is recommended I will try it.”) and the recommendation algorithm to provide an incentive for reflection (“Why was this POI recommended to me? How did the system know that I will like it?”). The fact that the personalized view offers a more personal and dynamic visitor experience, through the interaction with the recommendation system, is one of its main advantages. This aspect could be further explored by dedicated studies focusing on this dialogue between the recommender system and the user and identifying ways to enhance the playful aspects of recommendation through the experience design.

This aspect is also relevant to another important challenge for the design of recommendation systems as visitor experiences: How to evaluate the effectiveness of the approach? What are the most “suitable” POIs for a specific visitor? Who is qualified to decide what “suitable” or fitting POI means? Can the visitors themselves identify reliably the more or less “interesting” POIs when asked to compare, outside the context of the specific visit? As already mentioned, the POI as an object of the recommendation is complex, and whether the visitor will like it or not is the result of many different factors. Additionally, in some cases, even the visitors themselves cannot foresee in advance what would be suitable for them. The methodology of evaluating personalization and recommendation approaches for the visitor in cultural heritage contexts should attempt to take into account these factors. Further targeted work in this field is needed to fully understand the complex implications of these factors for a recommendation approach for a cultural visit experience and to design the appropriate methodologies to evaluate it. The question remains if metrics such as the precision, accuracy, or recall are indeed significant or even relevant in this field.

As one of the users mentioned: “If it was left to me to decide, I would focus mostly on architecture and visiting neo classical houses. I would either choose a relevant thematic route, or look in the map view to select only this type of POIs. The recommendation system helped me focus on this interest but at the same time go beyond that and expand my horizons during my visit. And this is in my opinion the main strength of this app”.

In this sense, the weight of the effectiveness of personalization and recommendation for cultural heritage is shifted from proposing the optimum POI to supporting the visitor to have an optimum experience at the specific context. Moreover, having an immersive experience that helps the visitors establish a deeper connection with the city, its history, and its inhabitants is the key to guaranteeing that they will make an effort to revisit the city and explore more POIs. The evaluation results indeed produced useful insight on how to improve the recommendation system. These include additional features such as keeping interesting POIs to visit later, being able to view all POIs that are closer to a specific location, and even being able to input more details in the user profile on the way to interact with the local community of the city or take into account group visits. These suggestions need to be included in an updated design of the app and then evaluated with targeted user studies.