Next Article in Journal / Special Issue
TinajAR: An Edutainment Augmented Reality Mirror for the Dissemination and Reinterpretation of Cultural Heritage
Previous Article in Journal
A Comparative Study of Skeuomorphic and Flat Design from a UX Perspective
Previous Article in Special Issue
Interactive Tools for the Preservation, Dissemination and Study of Silk Heritage—An Introduction to the SILKNOW Project
Article

Flow, Staging, Wayfinding, Personalization: Evaluating User Experience with Mobile Museum Narratives

1
Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, 157 84 Athens, Greece
2
Athena Research and Innovation Center, 15125 Maroussi, Greece
*
Author to whom correspondence should be addressed.
Multimodal Technol. Interact. 2018, 2(2), 32; https://doi.org/10.3390/mti2020032
Received: 24 April 2018 / Revised: 6 June 2018 / Accepted: 6 June 2018 / Published: 11 June 2018
(This article belongs to the Special Issue Digital Cultural Heritage)

Abstract

A multitude of challenges comes into play when attempting to design (and evaluate) an interactive digital storytelling experience for use by visitors in a museum. This paper reports on the evaluation of the prototype mobile-based storytelling “guides” designed, developed and deployed as part of a research project at the Acropolis Museum in Athens, Greece. Experiences designed for different visitor profiles were evaluated several times throughout the iterative design process, in a number of on-site studies, and with more than 180 museum visitors of all ages (with this paper reporting on two studies conducted with a total of 53 users visiting individually or in pairs). The evaluation methods included ethnography (i.e., observation of visitors in the Museum’s galleries), pre- and post-experience in-depth interviews and questionnaires to measure the Users’ Experience (UX), as well as data logging. The analysis of the data focused on themes representing components of the experiences, such as interactive story plot and narration, staging and way-finding in the physical space, personalization and social interaction. Our findings confirmed that understanding UX and what makes it effective or not in the rich context of a cultural setting is a complex endeavor. The paper discusses our findings and proposes relevant recommendations for the design of digital experiences for cultural, educational, and recreational purposes.
Keywords: user experience (UX) evaluation; human–computer interaction; mobile technologies; digital storytelling; digital heritage; archeological museum user experience (UX) evaluation; human–computer interaction; mobile technologies; digital storytelling; digital heritage; archeological museum

1. Introduction

The experience economy [1] has led to an increasing adoption of the notion of ‘user experience’ by cultural institutions such as museums, which compete with other venues for visitors’ recreational time. In their attempt to broaden their offer and attract new audiences, cultural institutions are embracing the design of experiences for their visitors; that is, holistic, meaningful, personally encountered events or stories “emerging from the dialogue of a person with her or his world through action” [2] and involving a mesh of psychological, social and physiological qualities with a particular emphasis on emotions [3]. Storytelling is seen as an interpretive technique that, when used appropriately, can communicate value, cultivate experiences with emotional resonance, encourage empathy, trigger visitor’s attention and meaning-making [4,5,6].
Digital technology, especially mobile, plays a significant role in helping cultural institutions shape storytelling experiences for visitors. Mobile technologies, in particular, are credited for ensuring access to vast amounts of information, catering to differing visitor styles, interests and needs [7]. They are perceived by the cultural institutions themselves [8,9,10] as a means to bring multiple and new voices into the visitor experience, extend audience reach by attracting new/young audiences, provide varying layers of interpretation (e.g., introductory or highlights tour), support access for visitors with special needs and offer an overall engaging and interactive experience [11,12].
Hence, much of the digital offer in museums nowadays focuses on leveraging the advantages of mobile-based guided tours, whether these are museum-provided audio/multimedia guides that visitors can rent or borrow when in the museum, or applications that visitors can load onto their own smartphones and tablets (a.k.a. the “Bring Your Own Device” or BYOD model) and carry with them throughout the visit. The BYOD solution is gaining ground as it offers the advantage that most visitors already use their own devices to recreationally enhance their cultural visits (e.g., to take “arties” [13]), while it makes the benefits of mobile available to any museum, regardless of size, since the museum does not have to deal with the practical nuisances of handling and maintaining devices and their associated costs.
However, studies of the use of mobiles in museums [7,12,14,15] have revealed several considerations and issues to overcome, including visitors’ distraction of attention away from physical objects; navigation and orientation shortcomings; restricted user control over the experience; and one-size-fits-all implementations that limit personalization. Most importantly, mobile applications, whether guided tours of exhibitions or free-choice presentations of exhibits, remain predominantly information providers; narrative has been used narrowly, mostly as descriptive, scholarly prose [5]. This often didactic interpretive approach to presenting cultural content may compromise visitor engagement over time, resulting ultimately in an uncritical and ineffective experience, and diminishing any advantages exerted by the medium.
The CHESS project [16,17] set out to tackle these issues by designing mobile experiences that attempt to shift focus, from the traditional set of exhibit-centric, information-loaded descriptions, to story-centric narrations with references to the exhibits. The initial goals have been twofold. Firstly, to handle cultural content in a way that adopts an interactive drama-based storytelling approach, combined with features that promise to make mobile applications compelling (e.g., exploratory activities, interactive choices that inform a personalized and adaptive experience, games, augmented reality presentations of cultural artefacts, etc.). Secondly, to follow a user-centred participatory design approach in developing prototypes, coupled with an iterative design-evaluation-redesign cycle that involves in-depth exploration of all aspects of interaction: Through the device, between the visitor, the exhibits, exhibition space and the other visitors. Interactive experiences have been designed for and evaluated at two museums, each with a different scope and end-user requirements: The Acropolis Museum (AM) in Athens (Greece) which displays the archaeological findings of the Acropolis, and Cité de l’Espace in Toulouse (France), a science museum focusing on space and its conquest.
In this paper we concentrate on the evaluation of the prototype experiences created for the Acropolis Museum. Descriptions of the Cité de l’Espace prototypes and their evaluation can be found in [18,19]. More specifically, this paper reports on the evaluation of two sets of iteratively designed prototypes. Our goal with this evaluation has been to formulate a comprehensive understanding of the complexities of experiences that are designed to be consumed in situ, i.e., in rich and often expansive, cultural spaces. The paper is structured as follows: After an overview of the issues observed in previous related work and assessments related to mobile applications for museums, we describe our evaluation studies and present our findings, organized by theme. We conclude with a discussion and implications for the design of interactive storytelling experiences for museums.

2. Related Work and Issues Identified

Digital mobile applications exploiting cultural heritage content, from the typical multimedia guided tours to the more elaborate mobile Augmented Reality (AR) activities, have come of age, as evidenced by the plethora of research projects, system providers, software/media production companies, and successful deployments in museums and cultural sites worldwide.
As the field matures, several studies aimed at assessing the usefulness and effectiveness of handheld devices in museum settings from different perspectives have been conducted. These studies reveal that mobile museum visiting comes with a number of complex challenges that need to be addressed during design and/or deployment. The relevant aspects, as far as the design of our prototypes is concerned, have been grouped into categories of issues defined as: Information (content) delivery, balance of the visitor attention between the environment and the device, navigation, usability, social aspects, and personalization.

2.1. Content Delivery

The issue of how the cultural content is to be delivered via a mediating mobile device within a meaningful physical space containing exhibits and other mediators, is an outstanding one among researchers and practitioners in the field [20]. Several studies reported that the presence of a mobile guide increased learning and attention, and that it was a source of motivation and inspiration to try new ways of interaction with the exhibition [21,22,23]. Moreover, it also became apparent that using alternative interpretation media might have the potential of attracting new audiences [24].
Concerning the style and structure of the information provided, a general aspect to be taken into account is the density of points of interest, since studies have shown that visitors’ physical and cognitive investment declines as the visit moves forward [25]. Studies related to both outdoor [26] and indoor heritage sites [15] have reported that visitors prefer short, layered segments of audio, as it puts control into the hands of the user to decide whether or not to access additional interpretation. Danks et al. [27] note that 1.5 min is a maximum for a standing audience at a screen and that the length should be indicated to avoid visitors’ discomfort and attention drifting away. On the other hand, additional music, images, video and sound effects should only be used when they enhance the interpretation, as visitors consider these distract them from looking at the real object [14].

2.2. Screen vs. Physical Space

The dominance of the device with regards to the user’s attention and the challenge to balance the visitors’ attention between handheld devices and the exhibition space is a known and fairly well explored issue. In [28] it is reported that visitors of the Exploratorium in San Francisco emphasized a feeling of isolation both with regard to other visitors and with the rest of the exhibition. Studies involving PDAs in art museums [22] found that devices monopolize visitors’ attention, instead of fomenting an egalitarian communication between objects, visitors and the device. Woodruff et al. [28] noted that guidebooks and tours have the potential to help or hinder visitors as they strive for optimal attentional balance. They identified the ‘balancing act’ which allows visitors to focus on different stimuli when they need to.
Regarding the kind of support mobile digital content should provide, in [29] some users expected to find exactly the same content in the real exhibition and the device, while others wanted it to be different, complementary, or more informational. However, this raises cognitive issues, since the study discovered visitors had problems to establish transferences between the real and the virtual world when there were no reference points to allow the superimposition of the two kinds of explanations. Damala et al. [30] obtained a divided result, with half of the visitors considering 2D and 3D images interfered with their appropriation of a painting, while the other half said it helped them better approach the work. Overall, a more playful, discovery approach was sought.
The surrounding environmental conditions are another relevant issue considered in previous studies. Massung [26] reported that users found it difficult to locate the indicated location in crowded areas, but also that because they were too absorbed by the device, they were often unaware of other visitors and were jostled by them. In most other projects, headphones are used instead of the device’s integrated speakers to tune out the ambient noise. The same applies even more in the case of outdoor settings [26] where ambient noise increases (traffic, etc.).

2.3. Location-Awareness and Navigation

The need to know the visitor’s position in the physical space in order to support their movement and direct them to the appropriate location for the experience to work requires navigation aids and location and orientation detection subsystems. Location awareness in cultural settings remains an open issue for museum mobile applications, particularly in the case of indoor settings, in which automatic solutions are not mature enough to support accuracy of less than a 2–3 m range. As some visitors find it difficult to interpret floor plans, a solution has been to post markers (e.g., numbers, signs or even QR codes) near the exhibits or points of interest. However, such interventions can further complicate and disrupt the visitor’s immersive experience. In general terms, orientation issues have been poorly addressed [31].

2.4. Interface, Interaction and Usability

Some studies have found that although visitors who were less familiar with mobile applications, usually senior visitors, needed more initial time to understand the mediator, this did not impact negatively to their appreciation of the overall experience [30]. The main usability problems identified are related to the size of the screen: Small screens make difficult the manipulation of the displayed elements while bigger devices are cumbersome. In general, issues related to the user interface and usability can be mitigated by following a user-centered design process [14].

2.5. Social Dimension of the Visit

The social aspects of the visit have been explored by several researchers [22,23,26], yet social interaction is one of the aspects that many mobile guide technologies fail to encourage [12,32]. Isolation between different members of a group of visitors when using electronic guides has long been cited as a disruption to the social aspects of a museum visit (e.g., [25] refer to the social-cultural context, which involves interaction and communication between individuals, as the contextual model of learning). In [26] it was observed that such isolation seemed self-imposed by the visitor rather than the technology. Even though splitters require co-visitors to keep the same pace and listen to the same commentaries, discussion between visitors was rarely observed to occur. In [30], visiting the exhibition “alone” was a choice that preceded technology. The fact that in other studies, social interaction was indeed adversely affected [22], while in others it was enhanced by the ubiquity of the devices [33], seems to reinforce the hypothesis that the integration of high-tech mediators is context-dependant.

2.6. Personalization and Adaptivity

Personalization is another important aspect, which has been consistently touted as a necessity in a number of earlier projects [14,34,35]. Thematic visits and visits organized according to the user’s available time are basic features of a personalized delivery of content. In [30], the heterogeneous use of the available high- and low-tech interpretation means, evidenced that it is not possible to create a “one-size-fits-all” experience. From her study, [26] concludes that giving visitors control and choice are critical aspects, whether in the amount of information, its type (presentation format, interpretation approach), and the method adopted (amount of control and interactivity).
In CHESS, the ability to personalize one’s experience has been implemented through a profiling system and various interactions available during the experience. Prior to the experience, to address the personalization cold start problem, the visitor fills out a profiling quiz which upon completion suggests relevant stories based on persona-matching [36]. During the visit, as the user interacts with the system and additional evidence is accumulated about him/her, the individual profile is gradually refined and emphasized. The system provides actions such as skipping a story section (interpreted as high negative feedback on the corresponding activity), signalling the completion of an activity (low positive feedback on the activity), selecting options from a menu (high positive feedback on the story script branch) or not selecting options in menus (low negative feedback on the corresponding story script branches). More information on how personalization has been implemented can be found in [17].

3. Overview of the Experiences

Different sets of storytelling experiences (also called stories) were designed specifically for different visitor profiles or “personas”, i.e., imaginary yet empirically grounded descriptions of typical visitors of the Acropolis Museum [36]. Personas and the stories designed for them were conceived and evolved in a true human-centered design fashion. Prior to developing the experiences we engaged in the following:
  • Collecting all kinds of data: Museum visitation numbers, demographics, museum visitor habits, styles of visit, context of a visit, web statistics, etc.
  • Studying the research literature on visiting styles (and finally adopting the ant, fish, butterfly, and grasshopper model by Véron and Levasseur [36]).
  • Carrying out (by an experienced ethnographer) fieldwork that included the exploration of sensitizing concepts and the conduct of in-depth interviews with explainers, curators, educators, receptionists, etc. [37].
  • Observing visitors’ habits and behavior in the specific museum setting [37].
  • Organizing authoring workshops with interdisciplinary teams of researchers and practitioners (a total of more than 80 professionals engaged in the course of three years), using participatory ideation techniques and “bodystorming” situated in the museum space [38].
The experiences that were designed follow largely the same general structure, starting with an introduction to the story, which is delivered through narration as the user enters the museum gallery. Then a series of object-based or theme-based episodes (or “script units”) are delivered, using narration and on-screen instructions to direct the user towards exhibits and, once the user confirms they are there, information (in disguise) is delivered relating to both the exhibit and the wider narrative. After all the exhibits have been visited, the experience concludes with a final piece of narrative, ending the story.
This paper focuses on three experiences. Each is based around a central character who tells a story around a set of objects in the museum—the characters are The Horse, Theseus and Melesso. The “Horse story” was designed in two versions, each version targeting a different persona (a child and an adult), and was evaluated in Study A, described later in this paper. The other two (“Theseus’ story” and “Melesso’s story”) were created later and evaluated in Study B. The three stories are presented in detail in Appendix A.
Each experience is delivered via a browser-based interactive application run on a tablet. The content is largely communicated through voice narration ideally experienced with a set of headphones (Figure 1). Users are directed to look at objects in the museum that parts of the narrative are built around. In addition to the narration, there are images and animations shown at times on the tablet’s screen.
All stories were developed in a highly iterative fashion, with multiple formative evaluation sessions feeding into further (re)design. Most notably, a major re-design took place between Study A and Study B using the trajectories framework proposed by [39] and described in Appendix B.

4. Evaluation Methodology

To evaluate the prototype mobile experiences from the visitors’ perspective, two main visitor studies were conducted at the AM, Study A (in December 2012, using the Horse story) and Study B (in January 2014, using the Theseus and Melesso stories). Both studies extended over a period of two days and were held, for the most part, after the Museum was closed to the public, to ensure an unobtrusive visiting experience for the visitors and ease of observing and obtaining data for the evaluators.

4.1. Participants

Representative visitors were invited to test the CHESS prototype experiences. A Call for Participants was disseminated through the project’s Advisory Board and other networks. When recruiting, we opted for a balanced sample across age and gender and tried to match the age demographics with those of our personas [36], which meant that we were looking for participants fitting in five different age range buckets: Young children (but not younger than 10), middle school-age children, young adults, adults, and people in middle or late adulthood. Although we were not able to achieve a perfect balance, we managed to collect a full set of data for a total of 53 participants who spent approximately two hours each experiencing the CHESS prototypes and participating in the evaluation.
For Study A, 25 visitors participated in and completed the evaluation. Of the 25 participants, 14 were female and 11 were male, with ages ranging from 10 to 55. For Study B, we collected a full set of data from 28 visitors, 13 female and 15 male, including three 10 year-olds.
Participants differed between the two studies. All of the younger participants paired with either a parent or a friend to experience a story (i.e., 3 pairs, 6 visitors). In Study A, 19 visitors experienced the stories individually while Study B had 10 lone visitors and 9 pairs (18 visitors).
The age and gender distribution of our participants in both studies is shown in Table 1.

4.2. Data Collection-Materials and Methods

There are several practical and conceptual challenges involved in the evaluations conducted in naturalistic settings. In the particular case of the Acropolis Museum, a world-class museum visited by approximately 1.5 million people each year, coordinating empirical work that requires us to make sense of how people act, interact, and perceive a complex experience warrants the use of a mixed methodology. To overcome the challenges, our study design sought to strike a balance between different methods, including the use of observation, interviews and questionnaires, as well as the automatic recording of system logs. Specifically, both sets of studies employed the same mixed (but predominately qualitative) methods comprising:
  • A general, pre-experience demographics questionnaire, administered verbally in the form of an interview;
  • video and audio recording and note-taking of the observation of visitors' behavior throughout their visit, i.e., from reception and first contact with the system (which begins with the CHESS Visitor Quiz), through to a complete experience of the story in the Museum’s galleries, and the handing back of the mobile device (Figure 2);
  • two semi-structured post-experience interviews per visitor/group, conducted immediately after the end of the visit. The interviews were delivered in a conversational tone in order to draw visitors out on what they experienced. The questions that served as guides (see Supplementary Documents S2) were adapted versions of instruments to measure UX and visitors’ experiences via technology in museums [12,40];
  • system logs which made time stamped records (time stamps synchronized across devices) and measured such things as interaction session start and end times and total time spent on experience, number of story branches traversed, etc.

4.3. Procedure

Visitors arriving at the museum to take part in the studies were directed to a reserved room and greeted by a researcher. An introduction to the purpose of the study and the agenda was communicated verbally to each visitor, and visitors were asked to sign an informed consent form. This material is available as Supplementary Documents S1. Visitors’ demographics and familiarity with technology and with the museum’s collections were also inquired at this stage.
Upon completion of the introductory session, participants were given a tablet with a pair of over-ear headphones; in the case of a group of visitors, a Y-splitter cable was used to connect two sets of headphones to one device. A wireless lavalier microphone was fitted on each lone visitor (or on one of two visitors in the case of a group of two). The pairs of visitors were given the freedom to decide who amongst them would be handling the tablet.
For first-time visitors, profiling and subsequent personalization of the experience begins with a simple quiz, the CHESS Visitor Quiz (CVQ). The goal of the quiz is to identify user characteristics, preferences, and visiting context and to use this information to (a) recommend the most relevant experience to each user before they start, and (b) personalize aspects of the experience, such as the duration, narrative style, visuals, etc. for that particular visitor. Participants in the study were thus asked to complete the quiz before beginning the experience. Their responses to the questions of the quiz were used to determine which persona they match with the most. The CVQ questions differed depending on the visitor’s age (Figure 3). The CVQ was designed following a survey with more than 100 visitors to the Acropolis Museum [41]. Link S2 in the supplementary material points to a demonstration version of the CVQ. Further information as well as an overview of the personalization approach implemented in CHESS can be found in [17].
After completing the quiz, the system presented the user with suggestions for stories to experience. Each story was presented with a one-sentence teaser, with the recommended story listed first. In Study A (the Horse story), visitors were given a choice between the two versions of the story (see Appendix A). In Study B, visitors were presented with either Theseus’s story or Melesso’s story first. In all cases, users were free to select any story they wished regardless of the system’s recommendation; all of the participants in our studies followed the system’s recommendation.
During the visit, each visitor or group of visitors were “shadowed” by two members of the research team (Figure 2), one taking notes and the other video recording the experience for further data analysis and to be kept as record (if written agreement was provided). To ensure that visitors’ behaviour would be affected as little as possible, the researchers tried to follow the participants from a reasonable distance and would only intervene when necessary for the process.
After the end of the experience, visitors were led back to the reserved room where two semi-structured interviews took place: The first, more general, interview was conducted to gather visitors' perceptions of the storytelling (plot, flow, narrators, relationship with environment), usability, shared interactions, and their impressions of the overall experience; the second interview, conducted by a different researcher, aimed at probing visitors’ opinions on the specific functionality related to personalization. The questions to guide the interviewers are available as Supplementary Materials (Documents S2).

4.4. Data Analysis

The data collected resulted in more than 100 h of video and audio recordings, and multiple pages of accompanying notes. All video observation data, interview responses, and notes were consolidated, transcribed, and examined individually and in correlation using a qualitative methodology. Key concepts, comments, and responses were organized according to the specific themes identified when designing the interview questions, and recommendations for the design of future experiences were then derived from each theme. Additionally, for the analysis of the interview responses, we followed a method of open coding similar to the one described in [35]: We grouped together positive user comments (e.g., “cool”, “I love it”), negative comments (e.g., “ridiculous”, “I hate it”), comments regarding technical or usability problems, and, most importantly, suggestions.
Each participant in the evaluation was given a reference ID to aid in the analysis and reporting, which consists of: A code to identify the study (‘A.’ for Study A and ‘B.’ for Study B), the participant number within the study preceded by the letter ‘u’, and the participant’s gender (‘f’ for female and ‘m’ for male). For example, A.u01f indicates the 1st female participant that took part in Study A. Wherever names are used, these are pseudonyms.

5. Findings

Data collection as well as the analysis of the collected data were structured around the following set of themes, which were articulated as key (categories of) questions during the interviews:
  • Story Plot and Flow: Was it clear, interesting, exciting, and immersive? Were the audio-visual assets appropriate?
  • Digital vs. Physical Space: What did the visitors focus on and for how long?
  • Wayfinding and Navigation: Did visitors have difficulty locating exhibits or finding where to go?
  • User Interface, Interaction and Usability: Was the interface (device) easy to use and interact with, were there external issues hindering the experience?
  • Pre- and Post-visit Activities: Did visitors wish to experience content at a later time? Are people interested in preparing their visit or re-visiting later?
  • Social Aspects: How was the experience with other people organized around the device?
  • Personalization and Adaptivity: Which aspects of the experience related to the user model, profiling, menus and preferences, and other features of each story version were appreciated by the visitor? Did the visitor appreciate (i.e., understand) the personalized experience overall?
Another way to group these questions is according to the spatial (digital or real) setting in which the action takes place (Table 2).
The findings and their interpretation are presented below, organized by theme, while the conclusions that can be drawn are discussed in the next section.

5.1. Story Plot and Flow

Considering the story-based approach, all visitors commented positively on the narrative, which was written by a professional storyteller.
● Interest, engagement
In the post-experience interviews, all users found the stories interesting and entertaining. The construct of a story itself, the first person narration, the comparisons with the contemporary world, and the informal/humorous elements injected in the story resulted in a narrative style that differed from what most visitors expected to hear in an archaeological museum.
I really don't care about "little statues half broken". And extracting information from the labels next to them is hell for me. But this was the very first time I did care about them, because of the story.”
[B.u07m]
I want my museum experience to be fun.”
[A.u19f]
This was exciting. It got to me.”
[B.u27m]
I loved it! No comparison with roaming around the museum alone.”
[A.u05f]
Visitors’ body language while experiencing the stories in the Archaic Gallery was also a telling measure of their engagement. Most visitors responded to humorous references or rhetorical questions posed by the narrator in the prose by smiling, grimacing, nodding or answering aloud. This visibly indicated their sense of immersion and personal connection to the exhibits.
I liked the humor and the fact that it was talking to me, it was like a dialogue.”
[B.u08f]
It was narrated in a way that made things interesting.”
[B.u14f, a 10 y. old girl]
Not all the activities are related to horses but it is not a problem. Mythology is interesting.”
[A.u02m]
“Oh no, my time is up!”
[A.u14f, a 12 y. old girl]
I loved the character, the narration, the voice. It was clear, not an academic text.”
[B.u15f]
● Experiencing the museum in a new way
Many visitors also considered the system to have helped them experience the museum in a new way. Especially visitors who had already visited the museum in the past were enthusiastic with this new approach to presenting the museum content.
Very interesting because I have been here before and there were things I knew nothing about and were highlighted and I saw them in a different way. It was immersive in a way, because it was so interesting.”
[A.u08f]
I have been here many times but had not noticed Hippalektryon (the horse-rooster statue). It is a big success that I now noticed the exhibit because I can realize its importance and rarity.”
[A.u06f]
However, visitors also identified various shortcomings.
● Fragmented—not a story, no real plot
I think it was bits and pieces, a narration with different flows.”
[B.u01m]
I expected that the things said should be related to the plot, the frame of helping horses, but not all were.”
[A.u03m]
There was a theme rather than a plot, with subplots.”
[A.u07m]
It kept asking me to go on with her "story" but I was wandering what the "story" is all about.”
[B.u04m]
The absence of a story was more prominent in the case of the Horse experience evaluated in Study A. Despite a successful initial setting of the plot (“the horse stuck in a museum time warp” or “the horse as a guide” depending on the user’s choice), the majority of users perceived the story was thereafter too informational content-heavy. They felt it was more a developed audio guide rather than a plot, and suggested that the general concept and role of the horse in the plot should have been more emphasized, and that this would have contributed to their involvement in the story. Other comments indicating that the users treated the experience as an elaborated audio guide include:
With this guide, I experienced only a selection of the museum exhibits. 99% of the exhibits were missing from the story; it felt like cheating the museum.”
[B.u18f]
The story was nice and what you had to do here was nice. […] But I would've liked it more if you could feel “the rush”. Maybe more sound effects would help to make you feel that you are more part of it.”
[A.u09f]
● Too static
One of the main negative responses concerning the mobile storytelling experience was the fact that the experience was “too static”: Visitors felt they stayed too long at the same spot listening to long narrations, and there was not enough interaction with the exhibits.
The story was interesting but just standing there listening, made me feel awkward like something was missing.”
[B.u01m]
Moving is interesting, standing is not.”
[A.u15m]
I would have liked some more movement, particularly at the last part of the visit.”
[A.u06f]

5.2. Digital vs. Physical Space

Most visitors found the visual assets presented on their handheld device interesting, reporting that media assets aided their understanding. Nevertheless:
● Too much looking at the screen
Similarly to other evaluations concerning mobile devices in museums, an important observation was that the majority of users, especially younger ones, were fully absorbed by the imagery shown on the device (which they appreciated) and spent more time looking at the screen than observing the exhibits. Being digital natives, this form of presentation felt natural to them. On the other hand, most adult visitors felt obliged to constantly look at the screen to avoid missing information (Figure 4a). This was in fact a cause of stress for some visitors as they realized that they should be looking at the objects but their attention was drawn to the screen. So they were not sure where to focus.
It seems that the provision of rich visual material on the digital device without explicit or implicit prompting to interact with the environment can trap visitors’ attention in the screen, thus impeding a fundamental goal of both visitors and the museum, namely the observation of exhibits. Redesigning the stories with the help of the trajectories framework, helped to alleviate but not eliminate this issue (Figure 4b).
While standing there I was constantly looking at the screen. I was afraid that I would be shown something and miss it.”
[A.u20f]
I felt like I had to look at the screen even though I did not want to, even though my eyes were forced to look around.”
[A.u04f]
I sensed my eyes drifting to the exhibits. I wanted more interaction with the exhibits.”
[A.u06f]
Too little time with the exhibits because I was looking at the exhibit only when there was an image of the exhibit on the screen.”
[A.u03m]
● Didn’t know when to look at the screen
I need clear directions where to look because otherwise I am absorbed by the screen.”
[A.u06f]
It's difficult to have 3 eyes, one on the tablet, one on the statue, one on its label. I ignored the screen.”
[A.u07m]
At some points I felt awkward just having to look at the screen all the time. I wanted to know specifically where to look.”
[A.u04f]

5.3. Wayfinding

Regarding the number and size of physical transitions (movement within the gallery), visitors reported that they liked moving around in the gallery and they would not mind moving back and forth in a nonlinear fashion. This is related to comments about the story flow, where visitors said that they do not like to be stuck in the same place for a long time, especially when they are not required to take specific actions (such as observing an exhibit’s details or playing a game). Yet, whenever location transitions are required to guide the visitor to a particular location, the visitor needs to be somehow prompted and guided.
We considered (and implemented both of) the following main way-finding approaches: (i) Photo-based navigation support which has shown to be robust, with visitors found their way to the correct location with relative ease throughout both experiences’ automatic continuation based on location-tracking mechanisms and (ii) explicit confirmation through visitor input. The spatial proximity of the exhibits in the Archaic Gallery affects negatively the accuracy of location tracking technologies, making them inappropriate for this physical space. Consequently, we implemented the latter approach, depicting a photo of the corresponding destination on the screen and requesting from the visitor to verify successful arrival at the target destination by selecting “I’m there”.
For example, in Study A, evaluating the Horse stories, a visitor commented on the photo-based navigation: “I’m not good with finding my way around in general so I felt more relaxed when I saw that there was a photo of the object I had to look for so it would not be so difficult to find. But on the other hand, I felt pressure as if I was taking exams: “And if I do not find it what happens?” I was not sure if I did it correctly. In this case it was more like doing homework, no sense of helping the horses escape with this activity. In general, I had no problem finding the exhibits, but the search should be integrated in the story.” [A.u05f].
In the prototypes evaluated in Study B, we mainly adopted a hints-based approach, where the narrating character asks the visitor to find the particular exhibit and provides a set of clues to do so. In this way, rather than disrupting the storytelling experience, navigation is indirectly achieved through its integration in the experience. If the visitor does not succeed in locating the target destination, the system displays an annotated photo including minimal navigation directions. For the story to continue, the visitor needs to confirm that she has actually reached the right place.
The results of the evaluation showed, similarly to our previous studies at the Cité de l’Espace [18], that photo-based navigation support worked well, as long as the photos were designed to illustrate precise locations and exhibits. What is perhaps more surprising is that all but one of the users enjoyed and even preferred the alternative, i.e., the explorative approach generated by the “hint-based” navigation. Even when visitors seemed to wander around before locating their destination, they felt it was part of the exploration and adventure aspect of the story. One of the participants even suggested that the system should only provide a textual description of the exhibit (without picture), in order to make the process more challenging. Nonetheless, the participant who did not appreciate the explorative navigation may be considered representative of another user group who prefers a simple, guided experience, with immediate, direct navigation assistance to avoid getting lost in big rooms.

5.4. User Interface, Interaction and Usability

With regards to usability, both the observations and visitors’ responses showed that, overall, the interface was regarded as straightforward and easy to use, even by visitors not experienced with touch screen devices or smart phones. However, there was also a range of usability issues identified. Many issues were specific to the different versions of the prototypes and were corrected easily in future system versions; nonetheless some issues may have general applicability to mobile system design and are likely to show persistence.
As the interface was designed in a way that brought the audio–visual elements in focus with the control elements hidden and available upon user request, many visitors missed such functionality, and requested immediately accessible “pause”, “skip”, or “history” buttons. Such functionality was added in later versions of the interface and was accessible to the user pressing on the character avatar in the lower left of the screen (Figure 5). Additionally, it was observed that participants who were familiar with the use of tablets tried to perform typical tap-based operations (such as zoom in/out, drag, etc.), even in cases where a single image identical to the exhibit was displayed.
The weight of the mobile device is another important factor to be taken into account, as some visitors felt tired by carrying the tablet and holding it the appropriate way for looking at the screen. The idea of using smart phones was greeted with mixed feelings, as some visitors felt it would be limited, whereas others felt it would help not to look constantly at the screen.

5.5. Pre- and Post-Visits

Pre-visit experiences were not considered necessary by the majority of users. On the other hand, a post-visit experience, especially a web account of the story that was experienced in the museum, was favoured by several users. This is linked with the frustration expressed by participants about the resources being limited in the experience in the Archaic Gallery. Since the screen presented exhibits from other parts of the museum, some visitors expressed interest in being able to see them after the story. Most visitors were very positive with the idea of keeping parts of the story for later or to re-experience them after the story: “I would have liked an option to keep some things for later. After each activity maybe. But not to stand there for longer.” [A.u04f].
This material could be used as a souvenir or to show their friends and relatives. The notion of souvenir was received with special interest and visitors said that they’d like a personalized memento of their experience: “I would take it [i.e., the souvenir]. Not sure if I would look at it but maybe I would give it to somebody else who was not there with me.” [A.u04f].

5.6. Social Aspects

It is well-known that museum visits are usually done in group (of families, pairs, friends, etc.). Thus, during the interview, visitors were asked about their opinions regarding individual vs. group experiences. Their responses varied, depending on their personal visiting style: While some visitors expressed preference for a mostly individual visit (with the possibility to exchange impressions at some points), others expressed the wish for a constantly shared experience.
In accordance with the issues identified in other studies reviewed in Section 2, our studies found that practical encumbrances (e.g., the need to tie everyone together with headphones on a single device because it is too noisy to hear through the device’s speakers) limited conversation between members. Nevertheless, visitors adopted non-verbal techniques, expressions and body language strategies, to communicate and share the experience (Figure 6).
Observations of pairs of visitors, in both Study A (3 pairs) and Study B (9 pairs) confirmed observations from previous ethnographic studies conducted at the Acropolis Museum as part of the CHESS project [36]. In particular, the role that one group member takes can impact that member’s or even the group’s engagement with the content. For example, in the case of parents visiting with children, the experience was mainly led by the latter (who usually carried the device); parents acted mainly as content mediators, thus sacrificing having an experience of their own for the sake of facilitating their children [19].

5.7. Personalization

There was considerable evidence that personalization of the experiences was desired by visitors.
Focusing on the user role, we identified three main categories, namely spectator, participant and contributor. A spectator has no personal involvement, she simply follows the story (e.g., traditional audio–visual narrations). A participant is assigned with some task that she needs to accomplish in the current production, as it is usually the case with games, quizzes and AR applications. Finally, a contributor needs to actively participate by providing content (such as text, drawing, photograph, opinion, etc.). We observed that while many visitors expressed a high interest in productions requiring their involvement, some stated that they would absolutely dislike to be assigned tasks requiring specific actions from their part. Such a dislike was justified with various reasons, such as fear of failing, preference on dramatic tension offered by narratives, general preference on spectator role and so on.
The duration of the story parts (also analysed by other researchers) seems to be an important personalization factor, as some visitors tended to generally prefer short ones, while others were more in favour of longer ones. The same was observed for the total duration of the experience, in relation to which two extreme types of participants arise. On the one hand, some participants kept their experience as short as possible, omitting almost all optional subparts. They were concerned about the provision of “skipping” functionalities by the storytelling system and their main objective was to avoid getting bored. On the other hand, other participants wanted to join almost every optional subpart and their main objective was to explore the story space, without missing something important. These visitors commented that there were some points during their experience where they would like to ask for additional information about the specific topics or about related exhibits that were close by.
Some key challenges are directed towards effective personalization. The observations revealed that superficially small differences in age or interests of visitors may have considerable impact on the appropriateness of the digital activities and the delivered experiences. Personalization needs to be dynamic and highly targeted through levels of information and a careful appreciation of visitors’ age and development. The personalized experience may also be facilitated by further control of content delivery, visibility of story structure and explicit identification of content as essential or secondary and loosening of the connection between the physical environment and content where the link is not essential.

6. Discussion

In this section we summarize the findings of the evaluation and discuss the implications of our work for the design, deployment and evaluation of future experiences and means to support them. We consider the results from the different but complementary perspectives that we used to structure our studies and analysis: The story perspective; the staging perspective (digital and physical space and moving through it); the interaction perspective; the social perspective; and the personalization perspective. Key insights are distilled into a brief list of tips offered at the end of each section.

6.1. Story Plot and Flow (Story Perspective)

The findings indicate overall positive sentiments towards the experiences. Considering storytelling, most users favoured narrative constructs that are not commonly used in museum prose: Humor, links to everyday contemporary life, an informal tone, the perhaps surprising use of unconventional characters, such as a horse, as a story plot device. However, it took a few iterations to reach a story design that was not too static and informational. The overall development of the story evaluated in Study A was regarded as somewhat disappointing, since the story content was predominantly informational and “static”. The teaser and introduction to the plot were described as interesting hooks for the experience, but after initial focus, the story then reverted to content with a loose apparent relationship to the plot. In Study B, after the story was re-designed (using the framework described in Appendix B) and new stories were created and evaluated, although fewer, such comments persisted.
This indicates that maintaining a sound and coherent plot throughout significantly benefits the perceived experience. The general plot, the actual “glue” that keeps together the informational script units, should be constantly present from start to finish in order to maintain coherence and engagement. Nevertheless, creating an experience that balances the immersive qualities of fiction with the authoritative factual interpretation of the past is a challenging exercise. Storytelling requires archaeologists and museum curators to abandon the objective, scientific voice typically used in their prose and may represent, to some, a sacrifice of authority and scholarship [4].
Engagement may be also enhanced by generally adopting a discovery approach. These requirements suggest that, for system authors, considerable effort may need to be spent if stories that last over a complete visitor experience are to be developed and successfully deployed.
The results also highlighted the importance of audio quality, not merely in terms of being able to hear the narration, but also how interesting and engaging real actors were to visitors. In other words, first person narration, humor, rhetorical questions addressed to the user, and references to her everyday life help strengthen personal relevance and immersion in the story. Special attention should be given to the quality of the narrated audio in order to convey the emotional aspect of the story and bring the character(s) to life. The evaluation indicated that long narrative sections are not favored by many visitors. There was evidence that visitors did not enjoy staying at one location for longer than they deemed it necessary. As indicated by researchers in other studies (see Section 2), script units should be kept short, and a timeline would be useful to see how far one is in the story. The results may indicate that personalizing visiting paths may be important for many visitors, or they are likely to modify their navigation to suit their preferred visiting style. The ability to break the fixed links between content and exhibits so that media may be consumed in many locations emerged as a common requirement. Control by visitors over how much content, exactly what it is, and where they experience it, does appear to be a significant requirement. Again, making it clear when content is directly related to exhibits, and when it may be consumed in any location is important. In this respect, the mobile application should distinguish (with a visual cue or even in the narration) the script units that directly relate to exhibits from the ones that do not. The application should also indicate the user’s action to be performed at each moment (listen, walk, explore exhibits, look at the screen, find next exhibit, etc.).
A summary of insights extracted from the findings with regards to the story perspective can be found in Table 3.

6.2. The All-Consuming Screen (Staging Perspective)

Considering the balance between digital and physical space, visitors were observed to be highly engaged with the visual on-screen content. As reported by other evaluations [22,28], many adult visitors considered the visuals a distraction from the primary focus, the exhibits. Younger visitors, on the other hand, regarded the visual content as the most enjoyable part of the experience (as opposed perhaps to narrative). Visitors appeared to spend considerable time looking at the tablet screens, even when walking around the space. This may indicate that visual material should only be provided when visitors are stationary for both safety reasons and to reinforce that visuals should primarily be an addition to the overall experience, not a distraction [26]. However, taking into account the outcome of Study A regarding visitors’ dislike of staying long at one place, it is important to include dynamic elements. In the case of script units not related to exhibits, the presentation should be enriched with visual/audio/motion effects so as to make up for the lack of physical observation points; in the case of script units related to exhibits, the application should encourage a more detailed exploration of the object (through the narrator or by including enhanced functionalities). Although rich media production is important, a carefully selected subset of objects/cases must be defined so that these dynamic/interactive elements do not overpower or break the overall storytelling experience.
A summary of insights extracted from the findings with regards to the staging perspective can be found in Table 4.

6.3. Moving and Searching through Exploration

Considering the navigation through the space, when visitors were moving through the AM galleries, few reported problems in finding where to go. Navigational information from both the story characters and the interface (e.g., through images) was effective. This success was due to the relatively small size of the galleries and the fact that most exhibits were within the visitor’s field of view but may not apply to other cultural sites with larger sized exhibits, points of interest that are not directly visible, or open-air spaces with complex and changing views. Furthermore, visitors did not seem to be bothered to move in a nonlinear fashion.
An important feature to pair with location awareness is orientation detection because the visitor’s physical perspective influences the narration. Current technical solutions that can be deployed to capture the user’s orientation include using the device’s compass.
Overall, a flexible yet accurate approach to navigation appears to be necessary to suit specific requirements of visitors and of the deployment site, if the system is to effectively guide visitors. An approach that utilised the enjoyment of exploration may allow some compromises in system accuracy to not negatively impact visitors' experience. Other user roles (e.g., guided vs. explorative) or visiting styles (e.g., systematic vs. selective) should also be taken into account.
A summary of insights extracted from the findings with regards to the navigation perspective can be found in Table 5.

6.4. Varying Levels of Control According to Interaction Style (Interaction Perspective)

Considering usability problems experienced by visitors, most had been resolved between various iterations. Comments from participants suggested that user control points should be more explicit/obvious and that touch-screen affordances and other enhanced functionalities (such as zoom in/out, rotate, drag) should be taken advantage of. Other important requirements are the ability to skip content, to go back, and to request additional information (e.g., display some pop-up control on the screen whenever related script pieces are available). On the other hand, the appearance or not of menus should be included as a dynamic option (depending on visitor profile and visit circumstances). Additionally, given the size and weight of tablets but the reduced dimensions of smartphone screens, iPad® Mini and similar sized tablets seem to be a good middle solution when it comes to selecting amongst devices currently on the market.
An issue for future consideration concerning interaction is how to design people’s uses of mobile and AR interfaces to be suitably respectful to the exhibits, the environment and to each other. Do we want to encourage people to run around and point mobile phones into the faces (or other bits) of statues, or do we need to design for slower, ritualistic and reverential interactions that engender solitude and reflection [42]?
A summary of insights extracted from the findings with regards to the interaction perspective can be found in Table 6.

6.5. Before, During, and After Continuity and Sense of Ownership

Considering pre- and/or post-visit experiences, the observations suggested that persistence of information was an important consideration. As in findings from previous studies of visitors at the Cité de l’Espace and the AM, the close link between location of exhibits and the content delivered (either as information provided by the museum at exhibits, or digital content) has the result that visitors are unable to consume all the content that they wish to on-site at the time of experiencing the story. The ability to save interesting details for later, see what has been previously visited, take a “souvenir” from the visit (e.g., something created by the user), and explore the exhibits appearing in the story after it finishes may be desired by a number of visitors. This requirement is in part driven by the social pressures to keep visiting groups together, but also that standing still at a particular exhibit and consuming all the content that might be relevant is not always considered necessary by visitors. In Study A, this issue was accentuated when the information did not relate to any particular exhibit. Both kinds of content are consumable at any location, for instance may be browsed in cafes or other relation points in a site, away from the pressures to keep up with the crowd, or move away from exhibits so others can enjoy.
Another interesting idea for future consideration is the notion that visitors need to be able to tell their own stories of what happened to them after an experience. This is a different kind of storytelling from the museum telling its story to the visitors, as the visitors are now telling their stories to each other (or maybe even back to the museum which can provide important feedback and possibly profiling information). A post-visit experience should introduce specific concepts to support functionality that would allow participants to capture, select, filter, recombine and publish documentation from experiences in various ways, from posting a story on a social media account to creating a wall display in a classroom.
A summary of insights extracted from the findings with regards to the temporal experience perspective can be found in Table 7.

6.6. Experiencing with Others (Sharing Perspective)

Considering both individual and shared group experiences, opportunities for human-to-human interaction appears to be important for some visitors at AM. Although the experience was primarily designed and evaluated as an individual experience, some of the participants noted that they wished to share the experience with other people. Increasing the number of mobile devices in the group or using multiple headphones with splitters to isolate from the ambient noise are obvious but partial solutions to the practical issue on how to support a group. A more insightful approach would be to design activities to support co-experiencing and meaningful collaboration among visitors who share the same experience; yet this is a challenging next step in the nascent field of design for shared visitor experiences in museums [43].
In the case of families, most parents attempted to structure the visit of their children, for example by helping them to interpret the experience, and by guiding them around different exhibits. However, parents experienced difficulties when they had lost track of where their children were in the experience; they did not have sufficient knowledge; they could not hear what their children were listening to; or the children were showing signs of frustration or boredom and parents did not know how to move on or skip unessential content. Possible solutions to these issues may be to allow synchronization of progress when families are using several devices; to create extra channels of information to parents, which they could use in this structuring work; to use multi headphone splitters; and to provide sufficient information about the structure of a visit to allow choices to be made [19].
A summary of insights extracted from the findings with regards to the social perspective can be found in Table 8.

6.7. What Type of Visit Am I in the Mood for Today?

Visitors experienced personalized and adapted stories to a moderate degree which was dependent upon their answers given in the pre-visit profiling quiz. A number of different visitor types were identified, those having a focused interest on a few, specific topics, and perhaps expressing indifference to other topics, and those expressing interest in almost every topic covered. The studies also identified that visitors varied in their desire to actively participate in the experience. Some preferred a more passive spectator’s role while others sought a more active, even contributing, role. The current productions, such as the Augmented Reality activities, required a certain amount of user input which was not always desirable by all visitors [44]. Personalization and adaption systems need to take further account of the visiting roles that users may prefer. The adaption required for these visitor roles also interacts with the flexibility needed to support the varied preferences in content consumption. Some visitors appear to skip through content to find what interests them, while others prefer to experience all content in depth. These user preferences have a considerable impact on the process of authoring new and existing stories.
Finally, personalizing the visitor’s experience inevitably conflicts with the inherently social activity of visiting a museum, thus placing additional tension on how to support individualized experience while also facilitating social interaction.
A summary of insights extracted from the findings with regards to the personalization perspective can be found in Table 9.

6.8. Limitations of the Evaluation

Our studies have limitations that should be considered in the interpretation of our findings. Firstly, we designed the studies to take place in the natural environment of one of the top museums of the world, the Acropolis Museum, rather than the simulated space of a laboratory. Nevertheless, although we achieved this ambitious goal, we had to succumb under the weight of the practical problems related to observing and recording study participants while the museum was open (e.g., crowding by other visitors around exhibits, “photos not allowed” policy, too much noise, etc.) and thus perform most of the studies after-hours. This can compromise the ecological validity of the studies, which set out to observe the complete visitor experience within a real-world setting, in all its complexities.
A further limitation relates to the data logs. While our system was designed to log data for each participant, such as total time spent on experience, number of story branches traversed, etc., only a partial view of this data was available to the interviewer at the time of the in-depth interview that immediately followed the participant’s in-gallery experience. The interviewers were able to view the raw data but informative data analytics were not in place to associate the visitor’s actions on the tablet with specific points in time recorded in the video. Hence, the interviewers had to sort through the data manually, at the time of the interview, to identify participants’ interaction points of interest in order to probe the interviewee to reflect on them. With the advent of data visualization solutions also for museums [45], to overcome this limitation, in future developments of our work we will consider automatic visualization tools to support museum curators in analysing and making sense of visitors’ movements and interactions.

7. Conclusions

Our work builds on a growing body of research and practice that strives to enrich museum visits by digital means. Specifically, the goal of the mobile prototypes designed, developed and evaluated at two very different museums in the context of the CHESS project was to make the museums more attractive and relevant to a wider range of visitors by creating personalized interactive storytelling experiences.
The evaluation of the experiences deployed at the Acropolis Museum of Athens provided insights on real users’ interactions in a visiting context which is challenging. The studies have revealed issues concerning both favorable and less favorable aspects of the deployed system. Overall, visitors had a very positive response to the experience, indicating that more unconventional, e.g., storytelling, approaches to engage with cultural content may greatly contribute towards more compelling visiting experiences.
At the same time, the evaluation confirmed the findings of previous studies arguing that the attempt to integrate several heterogeneous elements in a single seamless experience carries with it several possible breakdowns and pain points. Issues such as balancing the attention of the visitor between the screen and the physical space, ensuring proper navigation and orientation, combining an engaging story plot with interpretive content and catering for the needs of different user profiles, point towards design modifications that form the nucleus of our current research efforts. A follow-up project (H2020 EMOTIVE) is already underway with the objective to research the emotive potential of interactive storytelling experiences that resonate with visitors [5], informed by the lessons learned from this evaluation and the considerations presented in this paper.

Supplementary Materials

The following are available online at https://www.mdpi.com/2414-4088/2/2/32/s1, Documents S1: Study introductory information and consent form. Documents S2: Interview questions and guide. Additionally, the following are available online, Video S1: The CHESS experience promo video https://youtu.be/fZRiE7VR-xw. Video S2: Excerpt showing visitors taking part in the evaluation of CHESS experiences (Study A), December 2012, https://youtu.be/v26X8mCGVKg. Video S3: Excerpt from the introduction to Theseus’ story, https://youtu.be/9s_7hLEZLmw. Video S4: One of the multimedia productions that form part of Melesso’s experience, the story of the Athenian women who sought revenge for their husbands’ deaths by using their peplos’ pins, https://youtu.be/mfohmO1MBR8. Video S5: One of the multimedia productions that form part of Melesso’s experience, the myth of Arachne and goddess Athena Ergane (weaver), https://youtu.be/_LXa4HC8S5A. Video S6: The CHESS Project: Augmented Reality at the Acropolis Museum’s Peplos Kore exhibit https://youtu.be/K-A1ZC30OJE. Link S1: Project website: http://www.chessexperience.euLink S2: demonstration system of the CHESS Visitor Quiz, http://chess.madgik.di.uoa.gr:10005/cvs-acropolis/.

Author Contributions

M.R. and A.K. conceived and designed the studies presented in this paper; managed the larger research team performing the experiments; analyzed the data; and wrote the paper.

Funding

This research was funded by the Seventh Framework Programme European Union CHESS Project (www.chessexperience.eu), grant number 270198. The authors are also funded by the follow-up H2020 Framework Programme European Union EMOTIVE project, grant number 727188.

Acknowledgments

The authors wish to acknowledge the President and Chief Curator of the Acropolis Museum, Professor Dimitrios Pantermalis, Stamatia Eleutheratou, Niki Dollis, Laia Pujol, and the staff of the museum for their help in facilitating the studies. We also wish to express our gratitude to the researchers and evaluation support team, especially Vassilis Kourtis, Maria Vayanou, Manolis Synodinos, Timo Engelke, Jens Keil, Nikolaos Begetis and Peter Tolmie, to the author of the stories Lyo Kalovyrnas, and to all other members of the CHESS project. Last but not least, we thank all of the users who came to the museum to participate in our studies. Funds for covering the costs to publish in open access have not been received.

Conflicts of Interest

The authors declare no conflict of interest. The funding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

Appendix A

Eight stories were authored for the Acropolis Museum in the course of the CHESS project, each dealing with one or more themes provided by the Museum’s curators. Each story went through a number of design iterations (in some cases up to fifteen), while one of the stories tested in these studies also had variations, addressing different personas (Figure A1). A brief description of each of the three stories (the Horse story, Theseus’ story, Melesso’s story) and excerpts are presented below. The Horse story was the first to be authored and thus was tested in Study A. Theseus’ and Melesso’s stories were authored at a later time following a trajectories model (Appendix B), which was applied as a result of the lessons learned from Study A.
All excerpts were narrated by actors and recorded in a professional studio. Pauses in the narrative or content that has been removed from this presentation are indicated with [...].

Appendix A.1. Horse Story—Brief Description and Excerpts

“A horse and his friends are stuck in another time dimension. Will you help them find their way back to their time?”
The main character of the story is a horse who brags about his importance in ancient Greece. This story has two different plots (aimed at two different visitor personas). In the first plot version, “A horse needs help to go back to his time”, the Horse presents the museum as a time gap in which he and his friends are trapped. The Horse needs the visitor’s help to gather his friends and return to their time. In the process, he talks about horses and other related mythological beings (gods, monsters, etc.) that are either represented on exhibits in the museum’s Archaic Gallery or displayed in visual form on the tablet. In the second plot version, “The horse guides you through the Athens of 500 BC”, the Horse acts as a guide and presents notable horses, monsters and heroes of the archaic Athenian society.
Figure A1. The same story becomes a different experience, adapting the plot, media assets and narration style to different personas; here the horse character in the version of the story for (a) an art enthusiast; (b) a young visitor who prefers a more exploratory, action-packed adventure.
Figure A1. The same story becomes a different experience, adapting the plot, media assets and narration style to different personas; here the horse character in the version of the story for (a) an art enthusiast; (b) a young visitor who prefers a more exploratory, action-packed adventure.
Mti 02 00032 g0a1

Selected Excerpts from the Story Follow

Hello, my friend. There’s not much time, so I’m only going to say this once. You think you’re in a museum but you’re not.
This is a—I don’t know if I can trust you. This is not a museum, though it looks like one. It’s a time travel machine. Silly humans did not listen to us animals and some of us are stuck here now.
You see, this is not about the Acropolis, as you’ve been told. It’s about what was on that hill before the Acropolis that you see today was built.
I, the horse, am now in charge of getting us all back to our time; that is, around 500 BC. Anyway, I’m divine. My father was the god Poseidon himself. So that makes me a kind of god too. Ancient Greeks simply loved horses and made up lots of stories about us. We were very popular, so there were lots of statues of horses on the Acropolis.
There are 11 marble statues of horses on this floor and quite a few smaller ones, like the one on your screen.
To start with, there’s four of us horses ... Go look for us!
[…]
Aren’t we gorgeous? We come from a chariot, which is a horse-drawn carriage specially made for racing and warfare. This is part of the decoration of a temple and there once was a rider behind us, probably a god, which is now lost. You know, gods never rode on horseback, only in chariots—it’s more dignified really. For example, you can see on your screen goddess Artemis, or perhaps her brother, god Apollo, riding a chariot.
But I’m getting carried away… Like I said, you’re not in a museum but in a time-travel machine, which can take us back 2600 years ago! Things were a bit different then: we had distant relatives that were half-horse and half-men, or even winged horses.
They were called Centaurs and they were fierce creatures that loved war and merry-making, often at the same time. Not all of them were ruthless beasts though. One of my distant Centaur cousins, Chiron, was very wise. He was even the teacher of Asclepius, the god of medicine.
[…]
Uh-oh! Stand real still! There’s a monster here, right behind you ... No, don’t turn around yet! Turn around really slowly and no matter what you do, don’t look her in the eye! She’s the Medusa! You have to be very cunning when approaching this monster, or she’ll turn you to stone, if you look at her straight in the eyes! That was her special power. There’s an even scarier Medusa, in a showcase behind the pillar.
Go and find Medusa in one of the show cases. You’ll see two of her.
Ghastly, isn’t she? Medusa was one of the three Gorgon sisters. She had snakes instead of hair. If you come close to the glass, you will hear them hissing! Many had tried to kill her but they all turned to stone when they gazed in her fierce eyes. Go ahead, do you dare look at those eyes inside the showcase?
[…]
Let’s look for my cousin Pegasus, a winged-horse that could fly. He was one of the greatest stars of ancient mythology—definitely in the top 5! But before you see him… well, this winged horse had a monster of a mother, a real monster! Can you find her?
See also Video S2: Excerpt from the evaluation of the CHESS project (Study A), December 2012. https://youtu.be/v26X8mCGVKg.

Appendix A.2. Theseus’ Story—Brief Description and Excerpts

“Theseus, the famous hero of Athens needs your help to build an army after exiting the Cretan labyrinth! Join him in an exciting adventure to get all sorts of human, animal and divine aides!”
This story is narrated by Theseus, prince and chief hero of Athens. It engages the visitor to perform a series of feats (i.e., to collect “powers”), which in practice guide him/her to various exhibits. The story is fairly linear, in the sense that there is a beginning and an end, but there are several detours, leading the visitor back to the main plot.

Selected Excerpts from the Story Follow

[Failing transmission with static, in the vein of R2D2’s projection of Princess Leia. Theseus speaks in person]
Hello? Is anyone there? Can you hear me? [more static, transmission fails momentarily]
Oh finally! Someone’s found me! Please stay with me, this is serious! My name is Theseus. Prince of Athens. I’ve come to the far island of Crete to save my fellow Athenians from a terrible monster, the Minotaur. This monster is half bull half man and lived in a labyrinth. After a fierce fight I managed to kill him.
Now we are on our way out of the labyrinth, with the cunning help of Ariadne, the island’s princess. Ariadne gave me a ball of yarn after entering the maze and I used it to leave a trail to find the exit. The problem is that the king’s army is probably waiting for us outside. If so, how are we going to make it back to the ship to go back to Athens?
Here’s where you come in. I need you to gather the mightiest warriors you can find there and send them to me with the help of your device.
[...]
See this huuuge stone er… thing taking up so much space? To the left is my good ol’ friend Hercules fighting a Triton, which is another monster—it was like Monsters Inc. back then, monsters here, monsters there, monsters everywhere! Hercules is a fantastic hero, who performed all sorts of feats. He slew a monster with nine snake heads called Lernaia Hydra, he killed the Stymphalian Birds, who ate people and had bronze beaks and sharp metallic feathers they could launch at their victims. He was good and noble, and the gods allowed him to live with them in their palace on Mount Olympus, because he had helped them defeat the Giants. Hercules was the most famous hero in Athens until I came along, that is, and became even more famous.
Hey, stop admiring Hercules, you’re supposed to be helping me! So what do you think, should we take Hercules with us?

Appendix A.3. Melesso’s Story—Brief Description and Excerpts

“Melesso, a noble Athenian woman, talks about her life in the city of Athens. She has many things to say! Join her journey of memories, choose the ones you want her to share with you, and learn about the historical events that affected her life.”
The year is 474 BC. Archaic Acropolis has been destroyed by the Persians as they flee after their defeat in the second Persian War. The story is told by Melesso, an Athenian aristocrat in her late 50s, who recounts what happened in the war as she comes to the Acropolis, only 6 years after its destruction, to give a statuette as an offering (anathema) to Athena. She chooses Athena Promachos instead of Athena Ergane, because she also asks her goddess to protect her son who is away fighting the Persians. The memory of the ravages of war is still fresh, as she sees her city’s most sacred site scorched and destroyed. What at first seems like a fictional story is gradually fleshed out with accounts of the character’s daily life, climaxing when the visitor actually witnesses the anathema and, more importantly, sees the protagonist’s name inscribed on a real artefact 2.500 years ago (Figure A2). The connection between past and future is no longer cognitive or vague but experiential. This storyline connects the past (Archaic Athens) to the future (Classical Athens) on its very cusp, when the old world is dying and a new one is dawning. Those worlds are also linked to the present day.
The story is less explorative than Theseus’ story, but much longer, with many branches and opportunities for personalization.
Figure A2. Visitors experiencing “Melesso’s story” witness, at the culmination of the story, the character’s name inscribed on a real artefact of 2500 years ago.
Figure A2. Visitors experiencing “Melesso’s story” witness, at the culmination of the story, the character’s name inscribed on a real artefact of 2500 years ago.
Mti 02 00032 g0a2

Selected Excerpts from the Story Follow

Hello, my name is Melesso. I’m 2545 years old. Once, a long time ago, I walked on this very soil you are standing now, feeling proud. And scared. This is my story.
[...]
You’re probably walking up the inclined walkway now; to your left you can see small statues and pottery that tell the story of how we, the citizens of Athens, get married. Love doesn’t come into it at all, as all marriages are arranged by our fathers. I too got married when I became a woman at 15, to a man twice my age, as is the rule. I was lucky, for my husband, Thersis, was a good man. Anyway, you can take a closer look at them on your way back. I am in a hurry now, because what I am carrying is heavy. You may accompany me if you like to the temple—well, what is left of it…
[...]
You must be shocked by the fact that I was married off to a stranger twice my age when I was only 15, but you need to understand that things were very different in my time. We women of Athens led rather different lives to what is considered right and normal now. Even free women of the upper classes like me were not citizens and had no right to speak in public, vote or appear in court as a juror. We belonged to our fathers till we were old enough to bear children and then were married off to men chosen by our fathers. On the other hand, we weren’t cooped up at home all the time. We did take a very serious role in the religious rites of our city, which were part and parcel of public life, much more so than what is custom today.
[...]
When my first child, a son, was born I came to the Acropolis to dedicate one of my most expensive dresses at the temple of goddess Artemis. …Can you see Artemis smiling from the center of the gallery? Do you see the hole in her hand? It was made to hold her bow and arrows. How appallingly white and faded she looks now! Lift up your lens to see her in all her former splendor and vivid colors. [Augmented Reality activity]
See also Videos S4, S5 and S6 in the supplementary material for examples of multimedia productions that visitors could view on the tablet as part of the Melesso experience.

Appendix B

The trajectories framework, introduced by Benford et al. [39], and applied successfully in the design of other cultural experiences [42], was deemed necessary to alleviate the issues of story plot disconnect and general flow of the experience reported by many users in Study A. In other words, our approach was to use the existing experiences as a basis and re-structure them to include the stages that we felt they were missing e.g., connecting to the plot, drawing attention to the physical exhibit, etc.
We analyzed the Acropolis Museum stories to assess whether there was an existing trajectory-like structure and found that there were parts of the story that fitted in to the Approach, Engage, Experience and Disengage stages of the trajectory (Table A1) but without any distinguishing characteristics; a lot of these features were embedded within the story and did not always come in a useful order (e.g., in one story information about an object is delivered before the user is even told to approach it).
Table A1. 6-stage local trajectory for an Acropolis Museum experience.
Table A1. 6-stage local trajectory for an Acropolis Museum experience.
Connect (to Plot)ApproachEngageExperienceDisengage(Re-) Connect (to Plot)
Transition between the overarching story and the exhibit-based informationA teaser to draw the visitor in + Instructions on navigation to the point of interestPractical: A shift of focus from the overarching story to being involved in the exhibit-based information-telling the system you are there Conceptual: Drawing initial interest in the storyDraw attention to details of physical exhibits (or digital assets) + Deliver main story contentClosure of the exhibit-based information segment. Generic phrases that do not affect the plot but give some sense of story e.g., “But I’m getting carried away”Transition between the overarching story and the exhibit-based information
Storytelling is a key feature in the AM experiences—the whole experience is a story and each exhibit-based episode is both a story in itself and a part of the wider story. There are explicit points in the local trajectory where the exhibit-based story is connected from and connected to the overarching story, but there are also points within the Experience stage of the local trajectory where further connections are made via references to the wider story (e.g., “So now we have not just an athlete but a real winner amongst us!”). The interaction between the overarching story (global trajectory) and the exhibit-based stories (local trajectories) can be seen in Figure A3.
Figure A3. An example of an annotated local trajectory applied to a script unit of the Theseus story.
Figure A3. An example of an annotated local trajectory applied to a script unit of the Theseus story.
Mti 02 00032 g0a3

References

  1. Pine, B.J., II; Gilmore, J.H. The Experience Economy: Work Is Theater & Every Business a Stage; Harvard Business School Press: Boston, MA, USA, 1999. [Google Scholar]
  2. Hassenzahl, M. User Experience and Experience Design. In Encyclopedia of Interaction Design, 2nd ed.; Interaction Design Foundation: Aarhus, Denmark, 2012. [Google Scholar]
  3. Law, E.L.-C. The Measurability and Predictability of User Experience. In Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS), Pisa, Italy, 13–16 June 2011; pp. 1–9. [Google Scholar]
  4. Wright, A. Storytelling and Interpretation: Examining Best Practice at Heritage Sites; University of York: York, UK, 2017. [Google Scholar]
  5. Perry, S.; Roussou, M.; Economou, M.; Young, H.; Pujol, L. Moving Beyond the Virtual Museum: Engaging Visitors Emotionally. In Proceedings of the 23rd International Conference on Virtual Systems and Multimedia-VSMM, Dublin, Ireland, 31 October 2017. [Google Scholar]
  6. Johnsson, E. Telling Tales. A Guide to Developing Effective Storytelling Programmes for Museums; Adler, C., Ed.; London Museums Hub, Museum of London: London, UK, 2006. [Google Scholar]
  7. Tallon, L.; Walker, K. (Eds.) Digital Technologies and the Museum Experience: Handheld Guides and Other Media; AltaMira Press: Lanham, MD, USA, 2008. [Google Scholar]
  8. American Alliance of Museums, Museums Association, and Fusion Research + Analytics. Mobile in Museums Study—2012: A Survey of American Alliance of Museums (US) and Museums Association (UK) Members; AAM Press: Washington, DC, USA, 2012. [Google Scholar]
  9. Victoria & Albert Museum, Fusion Research & Analytics LLC, and Frankly, Green + Webb Ltd. Understanding the Mobile V&A Visitor: Autumn 2012; V & A: London, UK, 2012. [Google Scholar]
  10. Natural History Museum and Fusion Research & Analytics LLC. Natural History Museum: Understanding the Mobile Visitor; NHM: London, UK, 2013; Available online: http://www.nhm.ac.uk/resources-rx/files/nhm-mobile-report-july-2013-127910.pdf (accessed on 9 June 2018).
  11. Proctor, N. Mobile Apps for Museums: The AAM Guide to Planning and Strategy, 2nd ed.; AAM Press: Washington, DC, USA, 2011. [Google Scholar]
  12. Othman, M.K. Measuring Visitors’ Experiences with Mobile Guide Technology in Cultural Spaces; The University of York: York, UK, 2012. [Google Scholar]
  13. Smith, L.F.; Smith, J.K.; Tinio, P.P.L. Time spent viewing art and reading labels. Psychol. Aesthet. Creat. Arts 2017, 11, 77–85. [Google Scholar] [CrossRef]
  14. Kuflik, T.; Stock, O.; Zancanaro, M.; Gorfinkel, A.; Jbara, S.; Kats, S.; Sheidin, J.; Kashtan, N. A visitor’s guide in an active museum. ACM J. Comput. Cult. Heritage 2011, 3, 1–25. [Google Scholar] [CrossRef]
  15. Damala, A. Interaction Design and Evaluation of Mobile Guides for the Museum Visit: A Case Study in Multimedia and Mobile Augmented Reality Synthesis; Conservatoire National des Arts et de Metiers: Paris, France, 2009. [Google Scholar]
  16. Pujol, L.; Roussou, M.; Poulou, S.; Balet, O.; Vayanou, M.; Ioannidis, Y. Personalizing interactive digital storytelling in archaeological museums: The CHESS project. In Archaeology in the Digital Era, Proceedings of the 40th Annual Conference of Computer Applications and Quantitative Methods in Archaeology (CAA), Southampton, UK, 26–29 March 2012; Earl, G., Sly, T., Chrysanthi, A., Murrieta-Flores, P., Papadopoulos, C., Romanowska, I., Wheatley, D., Eds.; Amsterdam University Press: Amsterdam, The Netherlands, 2013; pp. 77–90. [Google Scholar]
  17. Vayanou, M.; Karvounis, M.; Kyriakidi, M.; Katifori, A.; Manola, N.; Roussou, M.; Ioannidis, Y. Towards Personalized Storytelling for Museum Visits. In Proceedings of the 6th International Workshop on Personalized Access, Profile Management, and Context Awareness in Databases (PersDB 2012), Istanbul, Turkey, 31 August 2012. [Google Scholar]
  18. Rennick-Egglestone, S.J.; Roussou, M.; Brundell, P.; Chaffardon, C.; Kourtis, V.; Koleva, B.; Benford, S. Indoors and outdoors: Designing mobile experiences for Cité de l’Espace. In Proceedings of the 6th NODEM (Network of Design and Digital Heritage) 2013 Conference: Beyond Control—The Collaborative Museum and its Challenges, Stockholm, Sweden, 1–3 December 2013; pp. 89–97. [Google Scholar]
  19. Rennick-Egglestone, S.J.; Brundell, P.; Koleva, B.; Benford, S.; Roussou, M.; Chaffardon, C. Families and mobile devices in museums: Designing for integrated experiences. ACM J. Comput. Cult. Heritage 2016, 9, 11. [Google Scholar] [CrossRef]
  20. Wyman, B.; Smith, S.; Meyers, D.; Godfrey, M. Digital Storytelling in Museums: Observations and Best Practices. Curator Mus. J. 2011, 54, 461–468. [Google Scholar] [CrossRef]
  21. Falk, J.H.; Dierking, L.D. Enhancing Visitor Interaction and Learning with Mobile Technologies. In Digital Technologies and the Museum Experience: Handheld Guides and Other Media; Tallon, L., Walker, K., Eds.; AltaMira Press: Lanham, MD, USA, 2008; pp. 19–33. [Google Scholar]
  22. Vom Lehn, D.; Heath, C. Displacing the object: Mobile technologies and interpretive resources. In Proceedings of the ICHIM03, International Cultural Heritage Informatics Meeting, Paris, France, 8–12 September 2003. [Google Scholar]
  23. Grinter, R.E.; Aoki, P.M.; Hurst, A.; Szymanski, M.H.; Thornton, J.D.; Woodruff, A. Revisiting the Visit: Understanding How Technology Can Shape the Museum Visit. In Proceedings of the Computer Supported Collaborative Work (CSCW), New Orleans, LA, USA, 16–20 November 2002; ACM Press: New Orleans, LA, USA, 2002; pp. 146–155. [Google Scholar]
  24. Lombardo, V.; Damiano, R. Storytelling on mobile devices for cultural heritage. New Rev. Hypermedia Multimedia 2012, 18, 11–35. [Google Scholar] [CrossRef][Green Version]
  25. Falk, J.H.; Dierking, L.D. The Museum Experience Revisited; Routledge: London, UK; New York, NY, USA, 2016. [Google Scholar]
  26. Massung, E. Visitor Reception to Location-based Interpretation at Archaeological and Heritage Sites. In Thinking beyond the Tool: Archaeological Computing and the Interpretive Process; BAR International Series; Archaeopress: Oxford, UK, 2012. [Google Scholar]
  27. Danks, M.; Goodchild, M.; Rodriguez-Echavarria, K.; Arnold, D.; Griffiths, R. Interactive Storytelling and Gaming Environments for Museums: The Interactive Storytelling Exhibition Project. In Proceedings of the 2nd International Conference on Technologies for E-Learning and Digital Entertainment, LNCS 4469, Hong Kong, China, 11–13 June 2007; Springer: Berlin/Heidelberg, Germany, 2007; pp. 104–115. [Google Scholar]
  28. Hsi, S. A study of user experiences mediated by nomadic web content in a museum. J. Comput. Assist. Learn. 2003, 19, 308–319. [Google Scholar] [CrossRef]
  29. Woodruff, A.; Aoki, P.M.; Hurst, A.; Szymanski, M.H. Electronic Guidebooks and Visitor Attention. In Proceedings of the 6th International Cultural Heritage Informatics Meeting (ICHIM), Milan, Italy, 3–7 September 2001; Archives and Museum Informatics: Milan, Italy, 2001; pp. 437–454. [Google Scholar]
  30. Damala, A.; Cubaud, P.; Bationo, A.; Houlier, P.; Marchal, I. Bridging the Gap between the Digital and the Physical: Design and Evaluation of a Mobile Augmented Reality Guide for the Museum Visit. In Proceedings of the 3rd International Conference on Digital Interactive Media in Entertainment and Arts—DIMEA ’08, Athens, Greece, 10–12 September 2008; ACM Press: New York, NY, USA, 2008; pp. 120–127. [Google Scholar]
  31. Moussouri, T.; Roussos, G. Conducting Visitor Studies Using Smartphone-Based Location Sensing. ACM J. Comput. Cult. Heritage 2015, 8, 1–16. [Google Scholar] [CrossRef][Green Version]
  32. Woodruff, A.; Szymanski, M.H.; Aoki, P.M.; Hurst, A. The Conversational Role of Electronic Guidebooks. In Proceedings of the 3rd International Conference on Ubiquitous Computing, Atlanta, GA, USA, 30 September–2 October 2001; Abowd, G.D., Shafer, S., Brumitt, B., Eds.; Springer: Berlin, Germany, 2001; pp. 187–208. [Google Scholar]
  33. Zurita, G.; Nussbaum, M. A constructivist mobile learning environment supported by a wireless handheld network. J. Comput. Assist. Learn. 2004, 20, 235–243. [Google Scholar] [CrossRef]
  34. Ardissono, L.; Kuflik, T.; Petrelli, D. Personalization in cultural heritage: The road travelled and the one ahead. User Model. User Adapt. Interact. 2011, 22, 73–99. [Google Scholar] [CrossRef]
  35. Damiano, R.; Gena, C.; Lombardo, V.; Nunnari, F.; Pizzo, A. A stroll with Carletto: Adaptation in drama-based tours with virtual characters. User Model. User Adapt. Interact. 2008, 18, 417–453. [Google Scholar] [CrossRef]
  36. Roussou, M.; Katifori, A.; Pujol, L.; Vayanou, M.; Rennick-Egglestone, S.J. A Life of Their Own: Museum Visitor Personas Penetrating the Design Lifecycle of a Mobile Experience. In Proceedings of the CHI 2013 Extended Abstracts on Human Factors in Computing Systems—CHIEA’13, Paris, France, 27 April–2 May 2013; ACM Press: New York, NY, USA, 2013; pp. 547–552. [Google Scholar]
  37. Tolmie, P.; Benford, S.; Greenhalgh, C.; Rodden, T.; Reeves, S. Supporting group interactions in museum visiting. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing—CSCW’14, Baltimore, MD, USA, 15–19 February 2014; ACM Press: New York, NY, USA, 2014; pp. 1049–1059. [Google Scholar]
  38. Roussou, M.; Pujol, L.; Katifori, A.; Chrysanthi, A.; Perry, S.; Vayanou, M. The museum as digital storyteller: Collaborative participatory creation of interactive digital experiences. In Proceedings of the 19th annual Museums and the Web Conference, Chicago, IL, USA, 8–11 April 2015. [Google Scholar]
  39. Benford, S.; Giannachi, G.; Koleva, B.; Rodden, T. From Interaction to Trajectories: Designing Coherent Journeys through User Experiences. In Proceedings of the CHI ’09: SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA, 4–9 April 2009; ACM Press: New York, NY, USA, 2009; pp. 709–718. [Google Scholar]
  40. Roto, V.; Vermeeren, A.P.O.S.; Väänänen-Vainio-Mattila, K.; Law, E.L.-C. User Experience Evaluation–Which Method to Choose? In Proceedings of the 13th IFIP TC 13 International Conference, Human-Computer Interaction—INTERACT 2011, Lisbon, Portugal, 5–9 September 2011; Springer: Lisbon, Portugal, 2011; pp. 714–715. [Google Scholar]
  41. Antoniou, A.; Katifori, A.; Roussou, M.; Vayanou, M.; Karvounis, M.; Kyriakidi, M.; Pujol-Tost, L. Capturing the Visitor Profile for a Personalized Mobile Museum Experience: An Indirect Approach. In Proceedings of the 24th ACM Conference on User Modeling, Adaptation and Personalisation (UMAP 2016), Halifax, NS, Canada, 13–17 July 2016; ACM Press: New York, NY, USA, 2016. [Google Scholar]
  42. Fosh, L.; Benford, S.; Reeves, S.; Koleva, B.; Brundell, P. ‘See Me, Feel Me, Touch Me, Hear Me’: Trajectories and Interpretation in a Sculpture Garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems—CHI’13, Paris, France, 27 April–2 May 2013; ACM Press: New York, NY, USA, 2013; pp. 149–158. [Google Scholar]
  43. Katifori, A.; Perry, S.; Vayanou, M.; Pujol, L.; Chrysanthi, A.; Ioannidis, Y. Cultivating mobile-mediated social interaction in the museum: Towards group-based digital storytelling experiences. In Proceedings of the MW2016: Museums and the Web 2016, Los Angeles, CA, USA, 6–9 April 2016. [Google Scholar]
  44. Keil, J.; Pujol, L.; Roussou, M.; Engelke, T.; Schmitt, M.; Bockholt, U.; Eleftheratou, S. A digital look at physical museum exhibits: Designing personalized stories with handheld Augmented Reality in museums. In Proceedings of the Digital Heritage International Congress (DigitalHeritage), Marseille, France, 28 October–1 November 2013; IEEE: Marseille, France, 2013; Volume 2, pp. 685–688. [Google Scholar] [CrossRef]
  45. Lanir, J.; Kuflik, T.; Sheidin, J.; Yavin, N.; Leiderman, K.; Segal, M. Visualizing museum visitors’ behavior: Where do they go and what do they do there? Pers. Ubiquitous Comput. 2017, 21, 313–326. [Google Scholar] [CrossRef]
Figure 1. Visitors handle a tablet and don headphones to experience and interact with a personalized story at the Acropolis Museum (AM).
Figure 1. Visitors handle a tablet and don headphones to experience and interact with a personalized story at the Acropolis Museum (AM).
Mti 02 00032 g001
Figure 2. Visitors were “shadowed” by one researcher taking notes and another recording their actions.
Figure 2. Visitors were “shadowed” by one researcher taking notes and another recording their actions.
Mti 02 00032 g002
Figure 3. (a) A young visitor completing the CHESS Visitor Quiz before beginning her museum experience; (b) one of the questions of the quiz used for user profiling/persona matching.
Figure 3. (a) A young visitor completing the CHESS Visitor Quiz before beginning her museum experience; (b) one of the questions of the quiz used for user profiling/persona matching.
Mti 02 00032 g003
Figure 4. (a) Study A showed that users spent most of their experience time fixating on the screen. (b) The stories were redesigned to encourage observation of the exhibits by having the characters bring the visitors’ attention to details (e.g., “Find me…”, “Look at my scaly tail…”).
Figure 4. (a) Study A showed that users spent most of their experience time fixating on the screen. (b) The stories were redesigned to encourage observation of the exhibits by having the characters bring the visitors’ attention to details (e.g., “Find me…”, “Look at my scaly tail…”).
Mti 02 00032 g004
Figure 5. Screenshots showing the approaches to wayfinding and interaction: control points to ensure that the visitor is in the right place for the story to continue and branching points with recommendations for choices that the system considers would interest the specific user and detours or functionalities for pausing, skipping, or saving for later.
Figure 5. Screenshots showing the approaches to wayfinding and interaction: control points to ensure that the visitor is in the right place for the story to continue and branching points with recommendations for choices that the system considers would interest the specific user and detours or functionalities for pausing, skipping, or saving for later.
Mti 02 00032 g005
Figure 6. Social interaction is limited by the need to wear headphones but visitors adopted non-verbal methods and body language strategies to communicate and share the experience.
Figure 6. Social interaction is limited by the need to wear headphones but visitors adopted non-verbal methods and body language strategies to communicate and share the experience.
Mti 02 00032 g006
Table 1. Numbers and ages of participants in evaluation studies.
Table 1. Numbers and ages of participants in evaluation studies.
Study A (12/12)Study B (01/14)
Age GroupFemaleMaleFemaleMale
−112112
12–172221
18–292338
30–495353
50+3221
14111315
Total2528
Table 2. Classification of key aspects studied.
Table 2. Classification of key aspects studied.
Where the Action isKey Aspects
Digital & physical spaceStory Plot & Flow (1);
Staging, i.e., instantiating a particular plot to create a specific user experience, as an interplay of screen and museum space (2);
Social Aspects (6)
Physical spaceWayfinding (3)
Digital spaceUser Interface, Interaction and Usability (4);
Pre- and Post-Activities (5);
Personalization and Adaptivity (7)
Table 3. Insights (pros, cons) from findings on the story perspective, and implications for design.
Table 3. Insights (pros, cons) from findings on the story perspective, and implications for design.
Story Perspective (Plot & Flow)
PositivesNarrative constructs such as humor, links to everyday contemporary life, an informal tone, and unconventional characters.
DrawbacksStory too informational and “static”, even after improvement.
Implications for design
  • Balance the immersive qualities of fiction with authoritative factual interpretation of the past but do not overflow the visitor with information. Use the engagement power of fiction to spark visitor interest on the facts.
  • Keep story sections short, provide a “story progress bar”, inform visitor whether content is related or not to exhibits.
  • Give the visitor (the illusion of) control.
Table 4. Insights from findings on the staging perspective, and implications for design.
Table 4. Insights from findings on the staging perspective, and implications for design.
Staging Perspective (Digital vs. Physical Space)
PositivesIn some cases, visitors felt that the visuals augmented the experience by bringing forth exhibit details that were not visible otherwise or related informational content and artifacts.
DrawbacksThe all-encompassing screen.
Implications for design
  • Prompt visitors to observe the exhibits.
  • Provide clear paradigms to the visitor on where to focus (screen or exhibits), e.g.,
    weave them into the story
    with visual/audio cues for focus switch (dimming screen, beeps)
    pointers on where the visitor should be looking at on the screen
  • Avoid having important content presented to the visitor while she has to navigate inside the site. It is not possible to focus equally on wayfinding and on understanding the offered content.
  • Avoid presenting on screen images from exhibits that the visitor can view in the physical space.
  • Avoid providing information on an object or location before the visitor reaches it/can view it.
Table 5. Insights from findings on the navigation perspective, and implications for design.
Table 5. Insights from findings on the navigation perspective, and implications for design.
Navigation Perspective (Wayfinding & Navigation)
Positives
  • Almost all visitors were able to locate the exhibits even though no localization system was in place.
  • Photo and descriptive instructions for navigation were seen as a more explorative and appealing way to navigate, by some users.
Drawbacks
  • In some cases, visitors were lost or found a different exhibit than the one indicated without realizing it.
  • There was too much standing still in one place.
Implications for design
  • Take into account:
    Visiting style (e.g., ant vs. grasshopper/butterfly)
    user’s preferred level of interactivity (guided vs. explorative)
    that visitors participate in an experience on-site, standing up and moving in physical space. There is only so much time before they get tired, physically or mentally so keep the stories short (short enough to be completed in one visit)
  • Exploration:
    No need to be too explicit, make it a game
    encourage visitors to move back and forth, to revisit/pass by exhibits they can now see from a different point of view, having heard their story
    avoid narration while the visitor is walking towards a goal; it adds to one’s cognitive load and can be distracting
  • Orientation:
    Detecting a user’s orientation (e.g., with a compass or even AR) makes things easier
  • Take into account the options of QR codes or NFC tags as ways for the user to check in at a specific location indoors during the experience.
Table 6. Insights from findings on the interaction perspective, and implications for design.
Table 6. Insights from findings on the interaction perspective, and implications for design.
Interaction Perspective (Control)
PositivesSome visitors liked to be guided by the storytelling experience.
Drawbacks
  • There was no actual interaction with the story plot.
  • The story felt too guided and linear.
  • The visitors would have liked, at points, to break the experience and focus on an irrelevant exhibit that caught their attention.
Implications for design
  • Give the visitor the illusion of control over the story in particular.
  • The visitors’ pre-conception of the experience matters: Do not promise interactivity and control if there is going to be none.
  • Provide for the visitors who prefer guided experiences but also for those who prefer more explorative and interactive ones.
Table 7. Insights from findings on the temporal experience perspective, and implications for design.
Table 7. Insights from findings on the temporal experience perspective, and implications for design.
Temporal Interaction (Pre- and Post-Visit Activities) Perspective
PositivesVisitors liked the pre-quiz.
DrawbacksThere was no option for “take away” content or a digital souvenir.
Implications for design
  • Design an engaging pre-experience profiling quiz.
  • Provide options for post-visits & “souvenirs”:
    Allow content to be carried away from exhibits (e.g., ”Add to cart”, “Save for later”)
    Present/provide a textual transcript of the story.
    Display a comic strip with visual/textual highlights of the visit.
    Make available (optional) additional information on the exhibits.
Table 8. Insights from findings on the social perspective, and implications for design.
Table 8. Insights from findings on the social perspective, and implications for design.
Social Interaction Perspective
PositivesThe stories were perceived by some visitors, who visited the museum alone, as companions.
DrawbacksThe stories were individual experiences, not designed for social interaction.
Implications for design
  • Offer frequent and meaningful choices as part of the plot. Make the group think and explore, give them challenges, make their choices have moral weight. Ensure that their actions influence how the story unfolds and its outcome.
  • Include activities that promote social interaction and collaboration (e.g., every member of the group should have a role to play in order to reach a goal).
  • If more devices are available, provide each group member the opportunity to experience a different story. For example, they may follow a different central character that has a different point of view and information to share. Visitors can be prompted to exchange information from each in order to piece together a bigger picture.
Table 9. Insights from findings on the personalization perspective, and implications for design.
Table 9. Insights from findings on the personalization perspective, and implications for design.
Personalization & Adaptivity Perspective
PositivesThe visitors liked the idea that the story is personalized.
DrawbacksIt has not been proven that personalization really made a difference.
Implications for design
  • Highlight recommended choice (e.g., annotate with a star)
  • Pace controlling (hurry up, skip, replay buttons)
  • Carefully selected time-estimation & planning mechanisms
    e.g., Time-based adaptation only in case of hard constraints
  • Provide a way to request more information (e.g., background, related or nearby exhibits)
  • Provide options, possibly different versions of the same story, to accommodate for visitors with different availabilities in time.
Back to TopTop