Flow, Staging, Wayfinding, Personalization: Evaluating User Experience with Mobile Museum Narratives

Abstract

A multitude of challenges comes into play when attempting to design (and evaluate) an interactive digital storytelling experience for use by visitors in a museum. This paper reports on the evaluation of the prototype mobile-based storytelling “guides” designed, developed and deployed as part of a research project at the Acropolis Museum in Athens, Greece. Experiences designed for different visitor profiles were evaluated several times throughout the iterative design process, in a number of on-site studies, and with more than 180 museum visitors of all ages (with this paper reporting on two studies conducted with a total of 53 users visiting individually or in pairs). The evaluation methods included ethnography (i.e., observation of visitors in the Museum’s galleries), pre- and post-experience in-depth interviews and questionnaires to measure the Users’ Experience (UX), as well as data logging. The analysis of the data focused on themes representing components of the experiences, such as interactive story plot and narration, staging and way-finding in the physical space, personalization and social interaction. Our findings confirmed that understanding UX and what makes it effective or not in the rich context of a cultural setting is a complex endeavor. The paper discusses our findings and proposes relevant recommendations for the design of digital experiences for cultural, educational, and recreational purposes.

1. Introduction

The experience economy [1] has led to an increasing adoption of the notion of ‘user experience’ by cultural institutions such as museums, which compete with other venues for visitors’ recreational time. In their attempt to broaden their offer and attract new audiences, cultural institutions are embracing the design of experiences for their visitors; that is, holistic, meaningful, personally encountered events or stories “emerging from the dialogue of a person with her or his world through action” [2] and involving a mesh of psychological, social and physiological qualities with a particular emphasis on emotions [3]. Storytelling is seen as an interpretive technique that, when used appropriately, can communicate value, cultivate experiences with emotional resonance, encourage empathy, trigger visitor’s attention and meaning-making [4,5,6].

Digital technology, especially mobile, plays a significant role in helping cultural institutions shape storytelling experiences for visitors. Mobile technologies, in particular, are credited for ensuring access to vast amounts of information, catering to differing visitor styles, interests and needs [7]. They are perceived by the cultural institutions themselves [8,9,10] as a means to bring multiple and new voices into the visitor experience, extend audience reach by attracting new/young audiences, provide varying layers of interpretation (e.g., introductory or highlights tour), support access for visitors with special needs and offer an overall engaging and interactive experience [11,12].

Hence, much of the digital offer in museums nowadays focuses on leveraging the advantages of mobile-based guided tours, whether these are museum-provided audio/multimedia guides that visitors can rent or borrow when in the museum, or applications that visitors can load onto their own smartphones and tablets (a.k.a. the “Bring Your Own Device” or BYOD model) and carry with them throughout the visit. The BYOD solution is gaining ground as it offers the advantage that most visitors already use their own devices to recreationally enhance their cultural visits (e.g., to take “arties” [13]), while it makes the benefits of mobile available to any museum, regardless of size, since the museum does not have to deal with the practical nuisances of handling and maintaining devices and their associated costs.

However, studies of the use of mobiles in museums [7,12,14,15] have revealed several considerations and issues to overcome, including visitors’ distraction of attention away from physical objects; navigation and orientation shortcomings; restricted user control over the experience; and one-size-fits-all implementations that limit personalization. Most importantly, mobile applications, whether guided tours of exhibitions or free-choice presentations of exhibits, remain predominantly information providers; narrative has been used narrowly, mostly as descriptive, scholarly prose [5]. This often didactic interpretive approach to presenting cultural content may compromise visitor engagement over time, resulting ultimately in an uncritical and ineffective experience, and diminishing any advantages exerted by the medium.

The CHESS project [16,17] set out to tackle these issues by designing mobile experiences that attempt to shift focus, from the traditional set of exhibit-centric, information-loaded descriptions, to story-centric narrations with references to the exhibits. The initial goals have been twofold. Firstly, to handle cultural content in a way that adopts an interactive drama-based storytelling approach, combined with features that promise to make mobile applications compelling (e.g., exploratory activities, interactive choices that inform a personalized and adaptive experience, games, augmented reality presentations of cultural artefacts, etc.). Secondly, to follow a user-centred participatory design approach in developing prototypes, coupled with an iterative design-evaluation-redesign cycle that involves in-depth exploration of all aspects of interaction: Through the device, between the visitor, the exhibits, exhibition space and the other visitors. Interactive experiences have been designed for and evaluated at two museums, each with a different scope and end-user requirements: The Acropolis Museum (AM) in Athens (Greece) which displays the archaeological findings of the Acropolis, and Cité de l’Espace in Toulouse (France), a science museum focusing on space and its conquest.

In this paper we concentrate on the evaluation of the prototype experiences created for the Acropolis Museum. Descriptions of the Cité de l’Espace prototypes and their evaluation can be found in [18,19]. More specifically, this paper reports on the evaluation of two sets of iteratively designed prototypes. Our goal with this evaluation has been to formulate a comprehensive understanding of the complexities of experiences that are designed to be consumed in situ, i.e., in rich and often expansive, cultural spaces. The paper is structured as follows: After an overview of the issues observed in previous related work and assessments related to mobile applications for museums, we describe our evaluation studies and present our findings, organized by theme. We conclude with a discussion and implications for the design of interactive storytelling experiences for museums.

2. Related Work and Issues Identified

Digital mobile applications exploiting cultural heritage content, from the typical multimedia guided tours to the more elaborate mobile Augmented Reality (AR) activities, have come of age, as evidenced by the plethora of research projects, system providers, software/media production companies, and successful deployments in museums and cultural sites worldwide.

As the field matures, several studies aimed at assessing the usefulness and effectiveness of handheld devices in museum settings from different perspectives have been conducted. These studies reveal that mobile museum visiting comes with a number of complex challenges that need to be addressed during design and/or deployment. The relevant aspects, as far as the design of our prototypes is concerned, have been grouped into categories of issues defined as: Information (content) delivery, balance of the visitor attention between the environment and the device, navigation, usability, social aspects, and personalization.

2.1. Content Delivery

The issue of how the cultural content is to be delivered via a mediating mobile device within a meaningful physical space containing exhibits and other mediators, is an outstanding one among researchers and practitioners in the field [20]. Several studies reported that the presence of a mobile guide increased learning and attention, and that it was a source of motivation and inspiration to try new ways of interaction with the exhibition [21,22,23]. Moreover, it also became apparent that using alternative interpretation media might have the potential of attracting new audiences [24].

Concerning the style and structure of the information provided, a general aspect to be taken into account is the density of points of interest, since studies have shown that visitors’ physical and cognitive investment declines as the visit moves forward [25]. Studies related to both outdoor [26] and indoor heritage sites [15] have reported that visitors prefer short, layered segments of audio, as it puts control into the hands of the user to decide whether or not to access additional interpretation. Danks et al. [27] note that 1.5 min is a maximum for a standing audience at a screen and that the length should be indicated to avoid visitors’ discomfort and attention drifting away. On the other hand, additional music, images, video and sound effects should only be used when they enhance the interpretation, as visitors consider these distract them from looking at the real object [14].

2.2. Screen vs. Physical Space

The dominance of the device with regards to the user’s attention and the challenge to balance the visitors’ attention between handheld devices and the exhibition space is a known and fairly well explored issue. In [28] it is reported that visitors of the Exploratorium in San Francisco emphasized a feeling of isolation both with regard to other visitors and with the rest of the exhibition. Studies involving PDAs in art museums [22] found that devices monopolize visitors’ attention, instead of fomenting an egalitarian communication between objects, visitors and the device. Woodruff et al. [28] noted that guidebooks and tours have the potential to help or hinder visitors as they strive for optimal attentional balance. They identified the ‘balancing act’ which allows visitors to focus on different stimuli when they need to.

Regarding the kind of support mobile digital content should provide, in [29] some users expected to find exactly the same content in the real exhibition and the device, while others wanted it to be different, complementary, or more informational. However, this raises cognitive issues, since the study discovered visitors had problems to establish transferences between the real and the virtual world when there were no reference points to allow the superimposition of the two kinds of explanations. Damala et al. [30] obtained a divided result, with half of the visitors considering 2D and 3D images interfered with their appropriation of a painting, while the other half said it helped them better approach the work. Overall, a more playful, discovery approach was sought.

The surrounding environmental conditions are another relevant issue considered in previous studies. Massung [26] reported that users found it difficult to locate the indicated location in crowded areas, but also that because they were too absorbed by the device, they were often unaware of other visitors and were jostled by them. In most other projects, headphones are used instead of the device’s integrated speakers to tune out the ambient noise. The same applies even more in the case of outdoor settings [26] where ambient noise increases (traffic, etc.).

2.3. Location-Awareness and Navigation

The need to know the visitor’s position in the physical space in order to support their movement and direct them to the appropriate location for the experience to work requires navigation aids and location and orientation detection subsystems. Location awareness in cultural settings remains an open issue for museum mobile applications, particularly in the case of indoor settings, in which automatic solutions are not mature enough to support accuracy of less than a 2–3 m range. As some visitors find it difficult to interpret floor plans, a solution has been to post markers (e.g., numbers, signs or even QR codes) near the exhibits or points of interest. However, such interventions can further complicate and disrupt the visitor’s immersive experience. In general terms, orientation issues have been poorly addressed [31].

2.4. Interface, Interaction and Usability

Some studies have found that although visitors who were less familiar with mobile applications, usually senior visitors, needed more initial time to understand the mediator, this did not impact negatively to their appreciation of the overall experience [30]. The main usability problems identified are related to the size of the screen: Small screens make difficult the manipulation of the displayed elements while bigger devices are cumbersome. In general, issues related to the user interface and usability can be mitigated by following a user-centered design process [14].

2.5. Social Dimension of the Visit

The social aspects of the visit have been explored by several researchers [22,23,26], yet social interaction is one of the aspects that many mobile guide technologies fail to encourage [12,32]. Isolation between different members of a group of visitors when using electronic guides has long been cited as a disruption to the social aspects of a museum visit (e.g., [25] refer to the social-cultural context, which involves interaction and communication between individuals, as the contextual model of learning). In [26] it was observed that such isolation seemed self-imposed by the visitor rather than the technology. Even though splitters require co-visitors to keep the same pace and listen to the same commentaries, discussion between visitors was rarely observed to occur. In [30], visiting the exhibition “alone” was a choice that preceded technology. The fact that in other studies, social interaction was indeed adversely affected [22], while in others it was enhanced by the ubiquity of the devices [33], seems to reinforce the hypothesis that the integration of high-tech mediators is context-dependant.

2.6. Personalization and Adaptivity

Personalization is another important aspect, which has been consistently touted as a necessity in a number of earlier projects [14,34,35]. Thematic visits and visits organized according to the user’s available time are basic features of a personalized delivery of content. In [30], the heterogeneous use of the available high- and low-tech interpretation means, evidenced that it is not possible to create a “one-size-fits-all” experience. From her study, [26] concludes that giving visitors control and choice are critical aspects, whether in the amount of information, its type (presentation format, interpretation approach), and the method adopted (amount of control and interactivity).

In CHESS, the ability to personalize one’s experience has been implemented through a profiling system and various interactions available during the experience. Prior to the experience, to address the personalization cold start problem, the visitor fills out a profiling quiz which upon completion suggests relevant stories based on persona-matching [36]. During the visit, as the user interacts with the system and additional evidence is accumulated about him/her, the individual profile is gradually refined and emphasized. The system provides actions such as skipping a story section (interpreted as high negative feedback on the corresponding activity), signalling the completion of an activity (low positive feedback on the activity), selecting options from a menu (high positive feedback on the story script branch) or not selecting options in menus (low negative feedback on the corresponding story script branches). More information on how personalization has been implemented can be found in [17].

3. Overview of the Experiences

Different sets of storytelling experiences (also called stories) were designed specifically for different visitor profiles or “personas”, i.e., imaginary yet empirically grounded descriptions of typical visitors of the Acropolis Museum [36]. Personas and the stories designed for them were conceived and evolved in a true human-centered design fashion. Prior to developing the experiences we engaged in the following:

• Collecting all kinds of data: Museum visitation numbers, demographics, museum visitor habits, styles of visit, context of a visit, web statistics, etc.
• Studying the research literature on visiting styles (and finally adopting the ant, fish, butterfly, and grasshopper model by Véron and Levasseur [36]).
• Carrying out (by an experienced ethnographer) fieldwork that included the exploration of sensitizing concepts and the conduct of in-depth interviews with explainers, curators, educators, receptionists, etc. [37].
• Observing visitors’ habits and behavior in the specific museum setting [37].
• Organizing authoring workshops with interdisciplinary teams of researchers and practitioners (a total of more than 80 professionals engaged in the course of three years), using participatory ideation techniques and “bodystorming” situated in the museum space [38].

The experiences that were designed follow largely the same general structure, starting with an introduction to the story, which is delivered through narration as the user enters the museum gallery. Then a series of object-based or theme-based episodes (or “script units”) are delivered, using narration and on-screen instructions to direct the user towards exhibits and, once the user confirms they are there, information (in disguise) is delivered relating to both the exhibit and the wider narrative. After all the exhibits have been visited, the experience concludes with a final piece of narrative, ending the story.

This paper focuses on three experiences. Each is based around a central character who tells a story around a set of objects in the museum—the characters are The Horse, Theseus and Melesso. The “Horse story” was designed in two versions, each version targeting a different persona (a child and an adult), and was evaluated in Study A, described later in this paper. The other two (“Theseus’ story” and “Melesso’s story”) were created later and evaluated in Study B. The three stories are presented in detail in Appendix A.

Each experience is delivered via a browser-based interactive application run on a tablet. The content is largely communicated through voice narration ideally experienced with a set of headphones (Figure 1). Users are directed to look at objects in the museum that parts of the narrative are built around. In addition to the narration, there are images and animations shown at times on the tablet’s screen.

All stories were developed in a highly iterative fashion, with multiple formative evaluation sessions feeding into further (re)design. Most notably, a major re-design took place between Study A and Study B using the trajectories framework proposed by [39] and described in Appendix B.

4. Evaluation Methodology

To evaluate the prototype mobile experiences from the visitors’ perspective, two main visitor studies were conducted at the AM, Study A (in December 2012, using the Horse story) and Study B (in January 2014, using the Theseus and Melesso stories). Both studies extended over a period of two days and were held, for the most part, after the Museum was closed to the public, to ensure an unobtrusive visiting experience for the visitors and ease of observing and obtaining data for the evaluators.

4.1. Participants

Representative visitors were invited to test the CHESS prototype experiences. A Call for Participants was disseminated through the project’s Advisory Board and other networks. When recruiting, we opted for a balanced sample across age and gender and tried to match the age demographics with those of our personas [36], which meant that we were looking for participants fitting in five different age range buckets: Young children (but not younger than 10), middle school-age children, young adults, adults, and people in middle or late adulthood. Although we were not able to achieve a perfect balance, we managed to collect a full set of data for a total of 53 participants who spent approximately two hours each experiencing the CHESS prototypes and participating in the evaluation.

For Study A, 25 visitors participated in and completed the evaluation. Of the 25 participants, 14 were female and 11 were male, with ages ranging from 10 to 55. For Study B, we collected a full set of data from 28 visitors, 13 female and 15 male, including three 10 year-olds.

Participants differed between the two studies. All of the younger participants paired with either a parent or a friend to experience a story (i.e., 3 pairs, 6 visitors). In Study A, 19 visitors experienced the stories individually while Study B had 10 lone visitors and 9 pairs (18 visitors).

The age and gender distribution of our participants in both studies is shown in

Table 1. Numbers and ages of participants in evaluation studies.

	Study A (12/12)		Study B (01/14)
Age Group	Female	Male	Female	Male
−11	2	1	1	2
12–17	2	2	2	1
18–29	2	3	3	8
30–49	5	3	5	3
50+	3	2	2	1
	14	11	13	15
Total	25		28

.

4.2. Data Collection-Materials and Methods

There are several practical and conceptual challenges involved in the evaluations conducted in naturalistic settings. In the particular case of the Acropolis Museum, a world-class museum visited by approximately 1.5 million people each year, coordinating empirical work that requires us to make sense of how people act, interact, and perceive a complex experience warrants the use of a mixed methodology. To overcome the challenges, our study design sought to strike a balance between different methods, including the use of observation, interviews and questionnaires, as well as the automatic recording of system logs. Specifically, both sets of studies employed the same mixed (but predominately qualitative) methods comprising:

• A general, pre-experience demographics questionnaire, administered verbally in the form of an interview;
• video and audio recording and note-taking of the observation of visitors' behavior throughout their visit, i.e., from reception and first contact with the system (which begins with the CHESS Visitor Quiz), through to a complete experience of the story in the Museum’s galleries, and the handing back of the mobile device (Figure 2);
• two semi-structured post-experience interviews per visitor/group, conducted immediately after the end of the visit. The interviews were delivered in a conversational tone in order to draw visitors out on what they experienced. The questions that served as guides (see Supplementary Documents S2) were adapted versions of instruments to measure UX and visitors’ experiences via technology in museums [12,40];
• system logs which made time stamped records (time stamps synchronized across devices) and measured such things as interaction session start and end times and total time spent on experience, number of story branches traversed, etc.

4.3. Procedure

Visitors arriving at the museum to take part in the studies were directed to a reserved room and greeted by a researcher. An introduction to the purpose of the study and the agenda was communicated verbally to each visitor, and visitors were asked to sign an informed consent form. This material is available as Supplementary Documents S1. Visitors’ demographics and familiarity with technology and with the museum’s collections were also inquired at this stage.

Upon completion of the introductory session, participants were given a tablet with a pair of over-ear headphones; in the case of a group of visitors, a Y-splitter cable was used to connect two sets of headphones to one device. A wireless lavalier microphone was fitted on each lone visitor (or on one of two visitors in the case of a group of two). The pairs of visitors were given the freedom to decide who amongst them would be handling the tablet.

For first-time visitors, profiling and subsequent personalization of the experience begins with a simple quiz, the CHESS Visitor Quiz (CVQ). The goal of the quiz is to identify user characteristics, preferences, and visiting context and to use this information to (a) recommend the most relevant experience to each user before they start, and (b) personalize aspects of the experience, such as the duration, narrative style, visuals, etc. for that particular visitor. Participants in the study were thus asked to complete the quiz before beginning the experience. Their responses to the questions of the quiz were used to determine which persona they match with the most. The CVQ questions differed depending on the visitor’s age (Figure 3). The CVQ was designed following a survey with more than 100 visitors to the Acropolis Museum [41]. Link S2 in the supplementary material points to a demonstration version of the CVQ. Further information as well as an overview of the personalization approach implemented in CHESS can be found in [17].

After completing the quiz, the system presented the user with suggestions for stories to experience. Each story was presented with a one-sentence teaser, with the recommended story listed first. In Study A (the Horse story), visitors were given a choice between the two versions of the story (see Appendix A). In Study B, visitors were presented with either Theseus’s story or Melesso’s story first. In all cases, users were free to select any story they wished regardless of the system’s recommendation; all of the participants in our studies followed the system’s recommendation.

During the visit, each visitor or group of visitors were “shadowed” by two members of the research team (Figure 2), one taking notes and the other video recording the experience for further data analysis and to be kept as record (if written agreement was provided). To ensure that visitors’ behaviour would be affected as little as possible, the researchers tried to follow the participants from a reasonable distance and would only intervene when necessary for the process.

After the end of the experience, visitors were led back to the reserved room where two semi-structured interviews took place: The first, more general, interview was conducted to gather visitors' perceptions of the storytelling (plot, flow, narrators, relationship with environment), usability, shared interactions, and their impressions of the overall experience; the second interview, conducted by a different researcher, aimed at probing visitors’ opinions on the specific functionality related to personalization. The questions to guide the interviewers are available as Supplementary Materials (Documents S2).

4.4. Data Analysis

The data collected resulted in more than 100 h of video and audio recordings, and multiple pages of accompanying notes. All video observation data, interview responses, and notes were consolidated, transcribed, and examined individually and in correlation using a qualitative methodology. Key concepts, comments, and responses were organized according to the specific themes identified when designing the interview questions, and recommendations for the design of future experiences were then derived from each theme. Additionally, for the analysis of the interview responses, we followed a method of open coding similar to the one described in [35]: We grouped together positive user comments (e.g., “cool”, “I love it”), negative comments (e.g., “ridiculous”, “I hate it”), comments regarding technical or usability problems, and, most importantly, suggestions.

Each participant in the evaluation was given a reference ID to aid in the analysis and reporting, which consists of: A code to identify the study (‘A.’ for Study A and ‘B.’ for Study B), the participant number within the study preceded by the letter ‘u’, and the participant’s gender (‘f’ for female and ‘m’ for male). For example, A.u01f indicates the 1st female participant that took part in Study A. Wherever names are used, these are pseudonyms.

5. Findings

Data collection as well as the analysis of the collected data were structured around the following set of themes, which were articulated as key (categories of) questions during the interviews:

• Story Plot and Flow: Was it clear, interesting, exciting, and immersive? Were the audio-visual assets appropriate?
• Digital vs. Physical Space: What did the visitors focus on and for how long?
• Wayfinding and Navigation: Did visitors have difficulty locating exhibits or finding where to go?
• User Interface, Interaction and Usability: Was the interface (device) easy to use and interact with, were there external issues hindering the experience?
• Pre- and Post-visit Activities: Did visitors wish to experience content at a later time? Are people interested in preparing their visit or re-visiting later?
• Social Aspects: How was the experience with other people organized around the device?
• Personalization and Adaptivity: Which aspects of the experience related to the user model, profiling, menus and preferences, and other features of each story version were appreciated by the visitor? Did the visitor appreciate (i.e., understand) the personalized experience overall?

Another way to group these questions is according to the spatial (digital or real) setting in which the action takes place (

Table 2. Classification of key aspects studied.

Where the Action is	Key Aspects
Digital & physical space	Story Plot & Flow (1); Staging, i.e., instantiating a particular plot to create a specific user experience, as an interplay of screen and museum space (2); Social Aspects (6)
Physical space	Wayfinding (3)
Digital space	User Interface, Interaction and Usability (4); Pre- and Post-Activities (5); Personalization and Adaptivity (7)

).

The findings and their interpretation are presented below, organized by theme, while the conclusions that can be drawn are discussed in the next section.

5.1. Story Plot and Flow

Considering the story-based approach, all visitors commented positively on the narrative, which was written by a professional storyteller.

● Interest, engagement

In the post-experience interviews, all users found the stories interesting and entertaining. The construct of a story itself, the first person narration, the comparisons with the contemporary world, and the informal/humorous elements injected in the story resulted in a narrative style that differed from what most visitors expected to hear in an archaeological museum.

“I really don't care about "little statues half broken". And extracting information from the labels next to them is hell for me. But this was the very first time I did care about them, because of the story.”

[B.u07m]

“I want my museum experience to be fun.”

[A.u19f]

“This was exciting. It got to me.”

[B.u27m]

“I loved it! No comparison with roaming around the museum alone.”

[A.u05f]

Visitors’ body language while experiencing the stories in the Archaic Gallery was also a telling measure of their engagement. Most visitors responded to humorous references or rhetorical questions posed by the narrator in the prose by smiling, grimacing, nodding or answering aloud. This visibly indicated their sense of immersion and personal connection to the exhibits.

“I liked the humor and the fact that it was talking to me, it was like a dialogue.”

[B.u08f]

“It was narrated in a way that made things interesting.”

[B.u14f, a 10 y. old girl]

“Not all the activities are related to horses but it is not a problem. Mythology is interesting.”

[A.u02m]

“Oh no, my time is up!”

[A.u14f, a 12 y. old girl]

“I loved the character, the narration, the voice. It was clear, not an academic text.”

[B.u15f]

● Experiencing the museum in a new way

Many visitors also considered the system to have helped them experience the museum in a new way. Especially visitors who had already visited the museum in the past were enthusiastic with this new approach to presenting the museum content.

“Very interesting because I have been here before and there were things I knew nothing about and were highlighted and I saw them in a different way. It was immersive in a way, because it was so interesting.”

[A.u08f]

“I have been here many times but had not noticed Hippalektryon (the horse-rooster statue). It is a big success that I now noticed the exhibit because I can realize its importance and rarity.”

[A.u06f]

However, visitors also identified various shortcomings.

● Fragmented—not a story, no real plot

“I think it was bits and pieces, a narration with different flows.”

[B.u01m]

“I expected that the things said should be related to the plot, the frame of helping horses, but not all were.”

[A.u03m]

“There was a theme rather than a plot, with subplots.”

[A.u07m]

“It kept asking me to go on with her "story" but I was wandering what the "story" is all about.”

[B.u04m]

The absence of a story was more prominent in the case of the Horse experience evaluated in Study A. Despite a successful initial setting of the plot (“the horse stuck in a museum time warp” or “the horse as a guide” depending on the user’s choice), the majority of users perceived the story was thereafter too informational content-heavy. They felt it was more a developed audio guide rather than a plot, and suggested that the general concept and role of the horse in the plot should have been more emphasized, and that this would have contributed to their involvement in the story. Other comments indicating that the users treated the experience as an elaborated audio guide include:

“With this guide, I experienced only a selection of the museum exhibits. 99% of the exhibits were missing from the story; it felt like cheating the museum.”

[B.u18f]

“The story was nice and what you had to do here was nice. […] But I would've liked it more if you could feel “the rush”. Maybe more sound effects would help to make you feel that you are more part of it.”

[A.u09f]

● Too static

One of the main negative responses concerning the mobile storytelling experience was the fact that the experience was “too static”: Visitors felt they stayed too long at the same spot listening to long narrations, and there was not enough interaction with the exhibits.

“The story was interesting but just standing there listening, made me feel awkward like something was missing.”

[B.u01m]

“Moving is interesting, standing is not.”

[A.u15m]

“I would have liked some more movement, particularly at the last part of the visit.”

[A.u06f]

5.2. Digital vs. Physical Space

Most visitors found the visual assets presented on their handheld device interesting, reporting that media assets aided their understanding. Nevertheless:

● Too much looking at the screen

Similarly to other evaluations concerning mobile devices in museums, an important observation was that the majority of users, especially younger ones, were fully absorbed by the imagery shown on the device (which they appreciated) and spent more time looking at the screen than observing the exhibits. Being digital natives, this form of presentation felt natural to them. On the other hand, most adult visitors felt obliged to constantly look at the screen to avoid missing information (Figure 4a). This was in fact a cause of stress for some visitors as they realized that they should be looking at the objects but their attention was drawn to the screen. So they were not sure where to focus.

It seems that the provision of rich visual material on the digital device without explicit or implicit prompting to interact with the environment can trap visitors’ attention in the screen, thus impeding a fundamental goal of both visitors and the museum, namely the observation of exhibits. Redesigning the stories with the help of the trajectories framework, helped to alleviate but not eliminate this issue (Figure 4b).

“While standing there I was constantly looking at the screen. I was afraid that I would be shown something and miss it.”

[A.u20f]

“I felt like I had to look at the screen even though I did not want to, even though my eyes were forced to look around.”

[A.u04f]

“I sensed my eyes drifting to the exhibits. I wanted more interaction with the exhibits.”

[A.u06f]

“Too little time with the exhibits because I was looking at the exhibit only when there was an image of the exhibit on the screen.”

[A.u03m]

● Didn’t know when to look at the screen

“I need clear directions where to look because otherwise I am absorbed by the screen.”

[A.u06f]

“It's difficult to have 3 eyes, one on the tablet, one on the statue, one on its label. I ignored the screen.”

[A.u07m]

“At some points I felt awkward just having to look at the screen all the time. I wanted to know specifically where to look.”

[A.u04f]

5.3. Wayfinding

Regarding the number and size of physical transitions (movement within the gallery), visitors reported that they liked moving around in the gallery and they would not mind moving back and forth in a nonlinear fashion. This is related to comments about the story flow, where visitors said that they do not like to be stuck in the same place for a long time, especially when they are not required to take specific actions (such as observing an exhibit’s details or playing a game). Yet, whenever location transitions are required to guide the visitor to a particular location, the visitor needs to be somehow prompted and guided.

We considered (and implemented both of) the following main way-finding approaches: (i) Photo-based navigation support which has shown to be robust, with visitors found their way to the correct location with relative ease throughout both experiences’ automatic continuation based on location-tracking mechanisms and (ii) explicit confirmation through visitor input. The spatial proximity of the exhibits in the Archaic Gallery affects negatively the accuracy of location tracking technologies, making them inappropriate for this physical space. Consequently, we implemented the latter approach, depicting a photo of the corresponding destination on the screen and requesting from the visitor to verify successful arrival at the target destination by selecting “I’m there”.

For example, in Study A, evaluating the Horse stories, a visitor commented on the photo-based navigation: “I’m not good with finding my way around in general so I felt more relaxed when I saw that there was a photo of the object I had to look for so it would not be so difficult to find. But on the other hand, I felt pressure as if I was taking exams: “And if I do not find it what happens?” I was not sure if I did it correctly. In this case it was more like doing homework, no sense of helping the horses escape with this activity. In general, I had no problem finding the exhibits, but the search should be integrated in the story.” [A.u05f].

In the prototypes evaluated in Study B, we mainly adopted a hints-based approach, where the narrating character asks the visitor to find the particular exhibit and provides a set of clues to do so. In this way, rather than disrupting the storytelling experience, navigation is indirectly achieved through its integration in the experience. If the visitor does not succeed in locating the target destination, the system displays an annotated photo including minimal navigation directions. For the story to continue, the visitor needs to confirm that she has actually reached the right place.

The results of the evaluation showed, similarly to our previous studies at the Cité de l’Espace [18], that photo-based navigation support worked well, as long as the photos were designed to illustrate precise locations and exhibits. What is perhaps more surprising is that all but one of the users enjoyed and even preferred the alternative, i.e., the explorative approach generated by the “hint-based” navigation. Even when visitors seemed to wander around before locating their destination, they felt it was part of the exploration and adventure aspect of the story. One of the participants even suggested that the system should only provide a textual description of the exhibit (without picture), in order to make the process more challenging. Nonetheless, the participant who did not appreciate the explorative navigation may be considered representative of another user group who prefers a simple, guided experience, with immediate, direct navigation assistance to avoid getting lost in big rooms.

5.4. User Interface, Interaction and Usability

With regards to usability, both the observations and visitors’ responses showed that, overall, the interface was regarded as straightforward and easy to use, even by visitors not experienced with touch screen devices or smart phones. However, there was also a range of usability issues identified. Many issues were specific to the different versions of the prototypes and were corrected easily in future system versions; nonetheless some issues may have general applicability to mobile system design and are likely to show persistence.

As the interface was designed in a way that brought the audio–visual elements in focus with the control elements hidden and available upon user request, many visitors missed such functionality, and requested immediately accessible “pause”, “skip”, or “history” buttons. Such functionality was added in later versions of the interface and was accessible to the user pressing on the character avatar in the lower left of the screen (Figure 5). Additionally, it was observed that participants who were familiar with the use of tablets tried to perform typical tap-based operations (such as zoom in/out, drag, etc.), even in cases where a single image identical to the exhibit was displayed.

The weight of the mobile device is another important factor to be taken into account, as some visitors felt tired by carrying the tablet and holding it the appropriate way for looking at the screen. The idea of using smart phones was greeted with mixed feelings, as some visitors felt it would be limited, whereas others felt it would help not to look constantly at the screen.

5.5. Pre- and Post-Visits

Pre-visit experiences were not considered necessary by the majority of users. On the other hand, a post-visit experience, especially a web account of the story that was experienced in the museum, was favoured by several users. This is linked with the frustration expressed by participants about the resources being limited in the experience in the Archaic Gallery. Since the screen presented exhibits from other parts of the museum, some visitors expressed interest in being able to see them after the story. Most visitors were very positive with the idea of keeping parts of the story for later or to re-experience them after the story: “I would have liked an option to keep some things for later. After each activity maybe. But not to stand there for longer.” [A.u04f].

This material could be used as a souvenir or to show their friends and relatives. The notion of souvenir was received with special interest and visitors said that they’d like a personalized memento of their experience: “I would take it [i.e., the souvenir]. Not sure if I would look at it but maybe I would give it to somebody else who was not there with me.” [A.u04f].

5.6. Social Aspects

It is well-known that museum visits are usually done in group (of families, pairs, friends, etc.). Thus, during the interview, visitors were asked about their opinions regarding individual vs. group experiences. Their responses varied, depending on their personal visiting style: While some visitors expressed preference for a mostly individual visit (with the possibility to exchange impressions at some points), others expressed the wish for a constantly shared experience.

In accordance with the issues identified in other studies reviewed in Section 2, our studies found that practical encumbrances (e.g., the need to tie everyone together with headphones on a single device because it is too noisy to hear through the device’s speakers) limited conversation between members. Nevertheless, visitors adopted non-verbal techniques, expressions and body language strategies, to communicate and share the experience (Figure 6).

Observations of pairs of visitors, in both Study A (3 pairs) and Study B (9 pairs) confirmed observations from previous ethnographic studies conducted at the Acropolis Museum as part of the CHESS project [36]. In particular, the role that one group member takes can impact that member’s or even the group’s engagement with the content. For example, in the case of parents visiting with children, the experience was mainly led by the latter (who usually carried the device); parents acted mainly as content mediators, thus sacrificing having an experience of their own for the sake of facilitating their children [19].

5.7. Personalization

There was considerable evidence that personalization of the experiences was desired by visitors.

Focusing on the user role, we identified three main categories, namely spectator, participant and contributor. A spectator has no personal involvement, she simply follows the story (e.g., traditional audio–visual narrations). A participant is assigned with some task that she needs to accomplish in the current production, as it is usually the case with games, quizzes and AR applications. Finally, a contributor needs to actively participate by providing content (such as text, drawing, photograph, opinion, etc.). We observed that while many visitors expressed a high interest in productions requiring their involvement, some stated that they would absolutely dislike to be assigned tasks requiring specific actions from their part. Such a dislike was justified with various reasons, such as fear of failing, preference on dramatic tension offered by narratives, general preference on spectator role and so on.

The duration of the story parts (also analysed by other researchers) seems to be an important personalization factor, as some visitors tended to generally prefer short ones, while others were more in favour of longer ones. The same was observed for the total duration of the experience, in relation to which two extreme types of participants arise. On the one hand, some participants kept their experience as short as possible, omitting almost all optional subparts. They were concerned about the provision of “skipping” functionalities by the storytelling system and their main objective was to avoid getting bored. On the other hand, other participants wanted to join almost every optional subpart and their main objective was to explore the story space, without missing something important. These visitors commented that there were some points during their experience where they would like to ask for additional information about the specific topics or about related exhibits that were close by.

Some key challenges are directed towards effective personalization. The observations revealed that superficially small differences in age or interests of visitors may have considerable impact on the appropriateness of the digital activities and the delivered experiences. Personalization needs to be dynamic and highly targeted through levels of information and a careful appreciation of visitors’ age and development. The personalized experience may also be facilitated by further control of content delivery, visibility of story structure and explicit identification of content as essential or secondary and loosening of the connection between the physical environment and content where the link is not essential.

6. Discussion

In this section we summarize the findings of the evaluation and discuss the implications of our work for the design, deployment and evaluation of future experiences and means to support them. We consider the results from the different but complementary perspectives that we used to structure our studies and analysis: The story perspective; the staging perspective (digital and physical space and moving through it); the interaction perspective; the social perspective; and the personalization perspective. Key insights are distilled into a brief list of tips offered at the end of each section.

6.1. Story Plot and Flow (Story Perspective)

The findings indicate overall positive sentiments towards the experiences. Considering storytelling, most users favoured narrative constructs that are not commonly used in museum prose: Humor, links to everyday contemporary life, an informal tone, the perhaps surprising use of unconventional characters, such as a horse, as a story plot device. However, it took a few iterations to reach a story design that was not too static and informational. The overall development of the story evaluated in Study A was regarded as somewhat disappointing, since the story content was predominantly informational and “static”. The teaser and introduction to the plot were described as interesting hooks for the experience, but after initial focus, the story then reverted to content with a loose apparent relationship to the plot. In Study B, after the story was re-designed (using the framework described in Appendix B) and new stories were created and evaluated, although fewer, such comments persisted.

This indicates that maintaining a sound and coherent plot throughout significantly benefits the perceived experience. The general plot, the actual “glue” that keeps together the informational script units, should be constantly present from start to finish in order to maintain coherence and engagement. Nevertheless, creating an experience that balances the immersive qualities of fiction with the authoritative factual interpretation of the past is a challenging exercise. Storytelling requires archaeologists and museum curators to abandon the objective, scientific voice typically used in their prose and may represent, to some, a sacrifice of authority and scholarship [4].

Engagement may be also enhanced by generally adopting a discovery approach. These requirements suggest that, for system authors, considerable effort may need to be spent if stories that last over a complete visitor experience are to be developed and successfully deployed.

The results also highlighted the importance of audio quality, not merely in terms of being able to hear the narration, but also how interesting and engaging real actors were to visitors. In other words, first person narration, humor, rhetorical questions addressed to the user, and references to her everyday life help strengthen personal relevance and immersion in the story. Special attention should be given to the quality of the narrated audio in order to convey the emotional aspect of the story and bring the character(s) to life. The evaluation indicated that long narrative sections are not favored by many visitors. There was evidence that visitors did not enjoy staying at one location for longer than they deemed it necessary. As indicated by researchers in other studies (see Section 2), script units should be kept short, and a timeline would be useful to see how far one is in the story. The results may indicate that personalizing visiting paths may be important for many visitors, or they are likely to modify their navigation to suit their preferred visiting style. The ability to break the fixed links between content and exhibits so that media may be consumed in many locations emerged as a common requirement. Control by visitors over how much content, exactly what it is, and where they experience it, does appear to be a significant requirement. Again, making it clear when content is directly related to exhibits, and when it may be consumed in any location is important. In this respect, the mobile application should distinguish (with a visual cue or even in the narration) the script units that directly relate to exhibits from the ones that do not. The application should also indicate the user’s action to be performed at each moment (listen, walk, explore exhibits, look at the screen, find next exhibit, etc.).

A summary of insights extracted from the findings with regards to the story perspective can be found in

Table 3. Insights (pros, cons) from findings on the story perspective, and implications for design.

Story Perspective (Plot & Flow)
Positives	Narrative constructs such as humor, links to everyday contemporary life, an informal tone, and unconventional characters.
Drawbacks	Story too informational and “static”, even after improvement.
Implications for design	Balance the immersive qualities of fiction with authoritative factual interpretation of the past but do not overflow the visitor with information. Use the engagement power of fiction to spark visitor interest on the facts. Keep story sections short, provide a “story progress bar”, inform visitor whether content is related or not to exhibits. Give the visitor (the illusion of) control.

.

6.2. The All-Consuming Screen (Staging Perspective)

Considering the balance between digital and physical space, visitors were observed to be highly engaged with the visual on-screen content. As reported by other evaluations [22,28], many adult visitors considered the visuals a distraction from the primary focus, the exhibits. Younger visitors, on the other hand, regarded the visual content as the most enjoyable part of the experience (as opposed perhaps to narrative). Visitors appeared to spend considerable time looking at the tablet screens, even when walking around the space. This may indicate that visual material should only be provided when visitors are stationary for both safety reasons and to reinforce that visuals should primarily be an addition to the overall experience, not a distraction [26]. However, taking into account the outcome of Study A regarding visitors’ dislike of staying long at one place, it is important to include dynamic elements. In the case of script units not related to exhibits, the presentation should be enriched with visual/audio/motion effects so as to make up for the lack of physical observation points; in the case of script units related to exhibits, the application should encourage a more detailed exploration of the object (through the narrator or by including enhanced functionalities). Although rich media production is important, a carefully selected subset of objects/cases must be defined so that these dynamic/interactive elements do not overpower or break the overall storytelling experience.

A summary of insights extracted from the findings with regards to the staging perspective can be found in

Table 4. Insights from findings on the staging perspective, and implications for design.

Staging Perspective (Digital vs. Physical Space)
Positives	In some cases, visitors felt that the visuals augmented the experience by bringing forth exhibit details that were not visible otherwise or related informational content and artifacts.
Drawbacks	The all-encompassing screen.
Implications for design	Prompt visitors to observe the exhibits. Provide clear paradigms to the visitor on where to focus (screen or exhibits), e.g., ○ weave them into the story ○ with visual/audio cues for focus switch (dimming screen, beeps) ○ pointers on where the visitor should be looking at on the screen Avoid having important content presented to the visitor while she has to navigate inside the site. It is not possible to focus equally on wayfinding and on understanding the offered content. Avoid presenting on screen images from exhibits that the visitor can view in the physical space. Avoid providing information on an object or location before the visitor reaches it/can view it.

.

6.3. Moving and Searching through Exploration

Considering the navigation through the space, when visitors were moving through the AM galleries, few reported problems in finding where to go. Navigational information from both the story characters and the interface (e.g., through images) was effective. This success was due to the relatively small size of the galleries and the fact that most exhibits were within the visitor’s field of view but may not apply to other cultural sites with larger sized exhibits, points of interest that are not directly visible, or open-air spaces with complex and changing views. Furthermore, visitors did not seem to be bothered to move in a nonlinear fashion.

An important feature to pair with location awareness is orientation detection because the visitor’s physical perspective influences the narration. Current technical solutions that can be deployed to capture the user’s orientation include using the device’s compass.

Overall, a flexible yet accurate approach to navigation appears to be necessary to suit specific requirements of visitors and of the deployment site, if the system is to effectively guide visitors. An approach that utilised the enjoyment of exploration may allow some compromises in system accuracy to not negatively impact visitors' experience. Other user roles (e.g., guided vs. explorative) or visiting styles (e.g., systematic vs. selective) should also be taken into account.

A summary of insights extracted from the findings with regards to the navigation perspective can be found in

Table 5. Insights from findings on the navigation perspective, and implications for design.

Navigation Perspective (Wayfinding & Navigation)
Positives	Almost all visitors were able to locate the exhibits even though no localization system was in place. Photo and descriptive instructions for navigation were seen as a more explorative and appealing way to navigate, by some users.
Drawbacks	In some cases, visitors were lost or found a different exhibit than the one indicated without realizing it. There was too much standing still in one place.
Implications for design	Take into account: ○ Visiting style (e.g., ant vs. grasshopper/butterfly) ○ user’s preferred level of interactivity (guided vs. explorative) ○ that visitors participate in an experience on-site, standing up and moving in physical space. There is only so much time before they get tired, physically or mentally so keep the stories short (short enough to be completed in one visit) Exploration: ○ No need to be too explicit, make it a game ○ encourage visitors to move back and forth, to revisit/pass by exhibits they can now see from a different point of view, having heard their story ○ avoid narration while the visitor is walking towards a goal; it adds to one’s cognitive load and can be distracting Orientation: ○ Detecting a user’s orientation (e.g., with a compass or even AR) makes things easier Take into account the options of QR codes or NFC tags as ways for the user to check in at a specific location indoors during the experience.

.

6.4. Varying Levels of Control According to Interaction Style (Interaction Perspective)

Considering usability problems experienced by visitors, most had been resolved between various iterations. Comments from participants suggested that user control points should be more explicit/obvious and that touch-screen affordances and other enhanced functionalities (such as zoom in/out, rotate, drag) should be taken advantage of. Other important requirements are the ability to skip content, to go back, and to request additional information (e.g., display some pop-up control on the screen whenever related script pieces are available). On the other hand, the appearance or not of menus should be included as a dynamic option (depending on visitor profile and visit circumstances). Additionally, given the size and weight of tablets but the reduced dimensions of smartphone screens, iPad® Mini and similar sized tablets seem to be a good middle solution when it comes to selecting amongst devices currently on the market.

An issue for future consideration concerning interaction is how to design people’s uses of mobile and AR interfaces to be suitably respectful to the exhibits, the environment and to each other. Do we want to encourage people to run around and point mobile phones into the faces (or other bits) of statues, or do we need to design for slower, ritualistic and reverential interactions that engender solitude and reflection [42]?

A summary of insights extracted from the findings with regards to the interaction perspective can be found in

Table 6. Insights from findings on the interaction perspective, and implications for design.

Interaction Perspective (Control)
Positives	Some visitors liked to be guided by the storytelling experience.
Drawbacks	There was no actual interaction with the story plot. The story felt too guided and linear. The visitors would have liked, at points, to break the experience and focus on an irrelevant exhibit that caught their attention.
Implications for design	Give the visitor the illusion of control over the story in particular. The visitors’ pre-conception of the experience matters: Do not promise interactivity and control if there is going to be none. Provide for the visitors who prefer guided experiences but also for those who prefer more explorative and interactive ones.

.

6.5. Before, During, and After Continuity and Sense of Ownership

Considering pre- and/or post-visit experiences, the observations suggested that persistence of information was an important consideration. As in findings from previous studies of visitors at the Cité de l’Espace and the AM, the close link between location of exhibits and the content delivered (either as information provided by the museum at exhibits, or digital content) has the result that visitors are unable to consume all the content that they wish to on-site at the time of experiencing the story. The ability to save interesting details for later, see what has been previously visited, take a “souvenir” from the visit (e.g., something created by the user), and explore the exhibits appearing in the story after it finishes may be desired by a number of visitors. This requirement is in part driven by the social pressures to keep visiting groups together, but also that standing still at a particular exhibit and consuming all the content that might be relevant is not always considered necessary by visitors. In Study A, this issue was accentuated when the information did not relate to any particular exhibit. Both kinds of content are consumable at any location, for instance may be browsed in cafes or other relation points in a site, away from the pressures to keep up with the crowd, or move away from exhibits so others can enjoy.

Another interesting idea for future consideration is the notion that visitors need to be able to tell their own stories of what happened to them after an experience. This is a different kind of storytelling from the museum telling its story to the visitors, as the visitors are now telling their stories to each other (or maybe even back to the museum which can provide important feedback and possibly profiling information). A post-visit experience should introduce specific concepts to support functionality that would allow participants to capture, select, filter, recombine and publish documentation from experiences in various ways, from posting a story on a social media account to creating a wall display in a classroom.

A summary of insights extracted from the findings with regards to the temporal experience perspective can be found in

Table 7. Insights from findings on the temporal experience perspective, and implications for design.

Temporal Interaction (Pre- and Post-Visit Activities) Perspective
Positives	Visitors liked the pre-quiz.
Drawbacks	There was no option for “take away” content or a digital souvenir.
Implications for design	Design an engaging pre-experience profiling quiz. Provide options for post-visits & “souvenirs”: ○ Allow content to be carried away from exhibits (e.g., ”Add to cart”, “Save for later”) ○ Present/provide a textual transcript of the story. ○ Display a comic strip with visual/textual highlights of the visit. ○ Make available (optional) additional information on the exhibits.

.

6.6. Experiencing with Others (Sharing Perspective)

Considering both individual and shared group experiences, opportunities for human-to-human interaction appears to be important for some visitors at AM. Although the experience was primarily designed and evaluated as an individual experience, some of the participants noted that they wished to share the experience with other people. Increasing the number of mobile devices in the group or using multiple headphones with splitters to isolate from the ambient noise are obvious but partial solutions to the practical issue on how to support a group. A more insightful approach would be to design activities to support co-experiencing and meaningful collaboration among visitors who share the same experience; yet this is a challenging next step in the nascent field of design for shared visitor experiences in museums [43].

In the case of families, most parents attempted to structure the visit of their children, for example by helping them to interpret the experience, and by guiding them around different exhibits. However, parents experienced difficulties when they had lost track of where their children were in the experience; they did not have sufficient knowledge; they could not hear what their children were listening to; or the children were showing signs of frustration or boredom and parents did not know how to move on or skip unessential content. Possible solutions to these issues may be to allow synchronization of progress when families are using several devices; to create extra channels of information to parents, which they could use in this structuring work; to use multi headphone splitters; and to provide sufficient information about the structure of a visit to allow choices to be made [19].

A summary of insights extracted from the findings with regards to the social perspective can be found in

Table 8. Insights from findings on the social perspective, and implications for design.

Social Interaction Perspective
Positives	The stories were perceived by some visitors, who visited the museum alone, as companions.
Drawbacks	The stories were individual experiences, not designed for social interaction.
Implications for design	Offer frequent and meaningful choices as part of the plot. Make the group think and explore, give them challenges, make their choices have moral weight. Ensure that their actions influence how the story unfolds and its outcome. Include activities that promote social interaction and collaboration (e.g., every member of the group should have a role to play in order to reach a goal). If more devices are available, provide each group member the opportunity to experience a different story. For example, they may follow a different central character that has a different point of view and information to share. Visitors can be prompted to exchange information from each in order to piece together a bigger picture.

.

6.7. What Type of Visit Am I in the Mood for Today?

Visitors experienced personalized and adapted stories to a moderate degree which was dependent upon their answers given in the pre-visit profiling quiz. A number of different visitor types were identified, those having a focused interest on a few, specific topics, and perhaps expressing indifference to other topics, and those expressing interest in almost every topic covered. The studies also identified that visitors varied in their desire to actively participate in the experience. Some preferred a more passive spectator’s role while others sought a more active, even contributing, role. The current productions, such as the Augmented Reality activities, required a certain amount of user input which was not always desirable by all visitors [44]. Personalization and adaption systems need to take further account of the visiting roles that users may prefer. The adaption required for these visitor roles also interacts with the flexibility needed to support the varied preferences in content consumption. Some visitors appear to skip through content to find what interests them, while others prefer to experience all content in depth. These user preferences have a considerable impact on the process of authoring new and existing stories.

Finally, personalizing the visitor’s experience inevitably conflicts with the inherently social activity of visiting a museum, thus placing additional tension on how to support individualized experience while also facilitating social interaction.

A summary of insights extracted from the findings with regards to the personalization perspective can be found in

Table 9. Insights from findings on the personalization perspective, and implications for design.

Personalization & Adaptivity Perspective
Positives	The visitors liked the idea that the story is personalized.
Drawbacks	It has not been proven that personalization really made a difference.
Implications for design	Highlight recommended choice (e.g., annotate with a star) Pace controlling (hurry up, skip, replay buttons) Carefully selected time-estimation & planning mechanisms ○ e.g., Time-based adaptation only in case of hard constraints Provide a way to request more information (e.g., background, related or nearby exhibits) Provide options, possibly different versions of the same story, to accommodate for visitors with different availabilities in time.

.

6.8. Limitations of the Evaluation

Our studies have limitations that should be considered in the interpretation of our findings. Firstly, we designed the studies to take place in the natural environment of one of the top museums of the world, the Acropolis Museum, rather than the simulated space of a laboratory. Nevertheless, although we achieved this ambitious goal, we had to succumb under the weight of the practical problems related to observing and recording study participants while the museum was open (e.g., crowding by other visitors around exhibits, “photos not allowed” policy, too much noise, etc.) and thus perform most of the studies after-hours. This can compromise the ecological validity of the studies, which set out to observe the complete visitor experience within a real-world setting, in all its complexities.

A further limitation relates to the data logs. While our system was designed to log data for each participant, such as total time spent on experience, number of story branches traversed, etc., only a partial view of this data was available to the interviewer at the time of the in-depth interview that immediately followed the participant’s in-gallery experience. The interviewers were able to view the raw data but informative data analytics were not in place to associate the visitor’s actions on the tablet with specific points in time recorded in the video. Hence, the interviewers had to sort through the data manually, at the time of the interview, to identify participants’ interaction points of interest in order to probe the interviewee to reflect on them. With the advent of data visualization solutions also for museums [45], to overcome this limitation, in future developments of our work we will consider automatic visualization tools to support museum curators in analysing and making sense of visitors’ movements and interactions.

7. Conclusions

Our work builds on a growing body of research and practice that strives to enrich museum visits by digital means. Specifically, the goal of the mobile prototypes designed, developed and evaluated at two very different museums in the context of the CHESS project was to make the museums more attractive and relevant to a wider range of visitors by creating personalized interactive storytelling experiences.

The evaluation of the experiences deployed at the Acropolis Museum of Athens provided insights on real users’ interactions in a visiting context which is challenging. The studies have revealed issues concerning both favorable and less favorable aspects of the deployed system. Overall, visitors had a very positive response to the experience, indicating that more unconventional, e.g., storytelling, approaches to engage with cultural content may greatly contribute towards more compelling visiting experiences.

At the same time, the evaluation confirmed the findings of previous studies arguing that the attempt to integrate several heterogeneous elements in a single seamless experience carries with it several possible breakdowns and pain points. Issues such as balancing the attention of the visitor between the screen and the physical space, ensuring proper navigation and orientation, combining an engaging story plot with interpretive content and catering for the needs of different user profiles, point towards design modifications that form the nucleus of our current research efforts. A follow-up project (H2020 EMOTIVE) is already underway with the objective to research the emotive potential of interactive storytelling experiences that resonate with visitors [5], informed by the lessons learned from this evaluation and the considerations presented in this paper.