2.1. Interactive Narrative
Digital media such as mobile applications, web applications, or digital games have transformed stories, which are told in an interactive format. In interactive narratives, interactors create or influence a dramatic storyline through their actions [15
]. Examining interactive narrative scholarship provides insight into how the design of digital media can evoke visitors’ engagement with cultural heritage artifacts in museum exhibits. Ryan describes interactive narrative as providing narrative engagement to interactors through the concepts of immersion and interactivity. Ryan aggregates various forms of interactive narrative experience through a framework that identifies diverse modes of interactivity in terms of the interactor’s involvement with the two narrative elements: character and plot [16
]. Her framework consists of two binary pairs—internal vs. external (describing involvement with the character) and ontological vs. exploratory (describing involvement with the plot) that are combined to offer four different types of interactive narrative experience. This framework maps out various interactive narrative experiences of the interactor such as playing the role of the protagonist or of the creator who constructs a story.
Ryan’s framework informs the framework we propose in this paper toward identifying interaction as diverse modes of engagement with a narrative role. Ryan considers narrative as a mental phenomenon; however, her framework does not offer a way to represent physical experience, thus it seems to offer a limited understanding of interactivity. In her view, immersion happens when the interactor transports him/herself into the virtual world by relating to an avatar in the story and imagining the details of the storyworld, which takes place in one’s mind [17
In contrast, Murray embraces physical experience by viewing narrative as a “threshold” experience between the imagined and the real worlds [18
]. She proposes a “threshold object” as a physical device that appears both in the story and the real world [19
]. A threshold object not only immerses one in the story, but also assists with one’s participation. This participation provides the pleasure of agency to the interactor to actively play a narrative role and transform the narrative outcome. Murray gives the example of an arcade game interface to show how an object one engages with within a story can draw one into the story. Murray states: “Ideally, every object in a digital narrative, no matter how sophisticated the story, should offer the interactor as clear a sense of agency and as direct a connection to the immersive world as I felt in the arcade holding a six-shooter-shaped laser gun and blasting away at the outlaws in Mad Dog McCree ” [18
] (p. 146).
Murray uses the term diegetic to describe how an object exists in both the physical and virtual worlds to help interactors engage with a story. The concept of diegetic is also used in film. Film utilizes the term to distinguish background sounds, which cannot be heard by characters in the movie but are added for the audience’s ears only to enhance the emotional impact on the viewer, from actual sounds that are heard by the characters in the movie [20
]. Along with the concept of diegetic, we propose to use the concept of non-diegetic. For example, a theme song played in the background of a movie would be non-diegetic, since the actors within the story cannot hear it. In contrast, the sound of a gun blast in the story would be diegetic, since the actors within the story can hear it. The concepts of diegetic and non-diegetic inform the proposed framework in this paper by showing that the tangible and embodied interactions in a narrative space shape narrative engagement and interpretation of the story. This paper extends interactive narrative research by examining studies on tangible and embodied interactions to inform how physical interactions can be utilized for narrative engagement.
2.2. Tangible and Embodied Narrative Systems
While museums have been implementing digital technologies in their exhibits, the projects typically focus on educational goals at the expense of experimenting with multi-modal technologies and interactions to reach those goals. Examination of existing non-museum projects that integrate embodied interaction, digital media, and narrative can guide the design of interactive experiences in museums. A growing number of interactive narrative systems have used tangible and embodied interfaces to integrate the interactor’s physical environment. For example, a digital tabletop that tracks physical objects can enable moving around tangible figurines to examine the various aspects of a story as if playing with chess pieces [21
]; virtual reality (VR) environments for games can provide sensory feedback that leads to immersion in the scene [22
]; and educational interfaces can augment learning by integrating the story with tangible puzzle pieces [23
Several frameworks on tangible and embodied interaction provide insight on how to integrate digital media and tangible/embodied interaction. Ullmer and Ishii focus on how tangible representation of digital data allows manipulation of computationally mediated digital information [3
]. This view has been called the “data-centered view” [2
]. Although Ullmer and Ishii’s framework does not provide specific guidance with respect to narrative design, we can look to the Triangles as an example of a tangible narrative system that shows a data-centered design, as each tangible triangular piece represents different narrative elements such as characters, events, setting, and dialogue [23
]. The magnetic triangular pieces can be stitched together in a variety of potential combinations, each creating different configurations of three stories (Cinderella 2000, Galapagos!, and The Digital Veil). The interactor thus manipulates the triangles, thereby potentially assembling a variety of configurations and hearing different parts of the story. For example, in the story of Cinderella 2000, snapping together the image of Cinderella’s evil stepmother’s face with the image of her house would play the audio clip of Cinderella’s step mother yelling at Cinderella. An example of a similar interactive system in a museum exhibit is an installation in the Exploratorium in San Francisco that allows visitors to explore phytoplankton populations in the world’s oceans [24
]. The installation consists of a physical ring that functions as a digital magnifying glass placed on top of an interactive multi-touch tabletop. Visitors can move the physical ring anywhere on the tabletop to look at the type and proportion of phytoplankton in different areas of the oceans. The physical ring utilized in the Exploratorium successfully promotes visitors’ participation. However, such interactions remain relatively simple (e.g., browsing or navigating) compared to the interactions resulting from Triangles cited above.
Other researchers focus on the human body as shaping embodied engagement in the world, thus proposing integration of computer systems with our physical experiences [25
]. The turn to the human body has inspired how the tangible system has become a resource for human action, in which meaning is created [26
]. This view has been called the “expressive-movement-centered view” [2
]. This literature proposes utilizing the sensory richness of the forms integrated with the interactive system to support the interactor’s meaning-making. An example of a tangible interface that allows expressive movement in a narrative is the Nintendo Wiimote game console device [28
]. The device can be held in one’s palm and allows the interactor to make bodily gestures such as shaking, pointing, and waving using one’s arms in order to participate in the game. For example, when playing virtual golf, the Wii console provides sensory feedback that evokes thoughts and imagination, as if one were engulfed in the golf field as a golf player. An example of a sensory experience provided in a museum is the installation of the Lunch Counter Simulation at the Center for Civil and Human Rights in Atlanta [29
]. The installation physically re-enacts the sit-in by providing a countertop, four bar stools, and headphones where visitors can sit and listen. Visitors hear the voices of people yelling and pounding on the table, and feel the vibration through the bar stools, thus, sensing the brutality and tension of the scene almost as though they were present during the actual event. Although the multi-sensory experience evokes thoughts about a poignant historic event, visitors have less interactivity and bodily engagement compared to those offered by non-museum projects.
Other researchers have devised tangible interactive systems for human sense-making in a spatial environment where the meaning of a tangible interaction, instead of being predefined by the designers, is created ‘in situ’, through situated social positioning and sensorimotor coupling [30
]. For instance, the interactive system NOOT provides a playback device through physical tags with RFID that can create audio-recordings so that interactors can revisit conversations in a relevant spatial setting [30
]. An example of a museum piece that provides space for social interaction is Youtopia, a multi-touch tabletop installation with tangible objects that teaches sustainability [10
]. The table displays an interactive map, and interactors can manipulate tangible stamps, menu blocks, and an information ring tool on the tabletop for land-use planning. The tabletop setup provides space for multiple interactors to gather around the map, and to discuss and negotiate their choices in order to make sustainable decisions.
Other works highlight the potential of tangible and embodied interactions for engaging with narrative perspectives and supporting interpretation. For example, based on the storytelling convention of multiple point-of-view narratives, Tangible Spatial Narratives utilizes tangible pawns on a digital tabletop to tell stories of an event from various points of view [21
]. The pawns represent characters in a story and can be moved around the tabletop surface, which shows a spatial arrangement of the narrative setting from a top-down view. The story is mapped within the spatio-temporal environment through the tangible pawns that represent different narrative points of view, allowing the interactor to examine various parts and layers of the story by placing and moving actual, physical pawns around the tabletop. A similar example, from a museum exhibit, is the interaction designed for the Hague and the Atlantic Wall: War in the City of Peace [31
]. In this exhibit, visitors carry around six smart replicas of objects (e.g., tea bag, sugar packet, travel pass, arm band, mug, dictionary), which they can place in interactive kiosks to hear an audio version of the story of the construction of the Atlantic Wall during WWII in the voice of a German soldier, or a civilian, or a civil servant. What is successful about this exhibit is the way interaction is implemented to highlight diverse viewpoints of the people within a historical event.
Examining frameworks on tangible and embodied interaction as well as existing museum and non-museum tangible and embodied narrative projects, like those discussed above, sheds light on the design direction museum projects can take to better integrate interactive narrative and embodied interactions to engage visitors with narrative perspectives. We see the potential of using tangible and embodied interactions to engage with narrative perspectives in interpreting a story in projects such as Tangible Spatial Narratives [21
] and in the interactions designed for pieces such as the Hague and the Atlantic Wall: War in the City of Peace [31
]. Inspired by these works, we propose a narrative framework to further develop similar projects.
2.3. Toward a Narrative Framework
We extracted the following narrative design elements from the literature on interactive narratives and the aforementioned examples: (1) physical engagement, (2) narrative role, and (3) narrative consequence. These elements can help engage interactors with narrative perspectives to interpret a story. Table 1
below illustrates the elements that emerge in each project.
Our examination of these elements in turn led us to devise the three spectra, (1) diegetic vs. non-diegetic, (2) internal vs. external, and (3) ontological vs. exploratory that comprise the framework we propose in the next section. In our previous work on tangible narrative frameworks, we examined existing tangible narrative systems and described the ways in which tangible media engage interactors with narrative systems across seven categories [32
]. Here we look more broadly at both tangible as well as embodied interactions, while at the same time focusing on narrative engagement with cultural artifacts in museum contexts. Hence, we narrowed down our proposed framework to three categories, which we call “spectra,” that focus on engagement with a narrative role that helps visitors relate to the original users of a given cultural artifact.
The three spectra that comprise the TENF come from Ryan’s framework on interactivity describing internal vs. external and ontological vs. exploratory interactions, and Murray’s concept of diegetic. Ryan’s concepts depict interactivity through a narrative role and consequences; Murray’s concept of a diegetic interaction describes how a tangible and embodied interaction offers engagement with a narrative. As the projects illustrated in Table 1
make evident, the way(s) in which interactors physically engage with the narrative through tangible objects and/or bodily movements is an essential part of the interactive narrative experience. These interactions can be diegetic or non-diegetic, which takes into account the way an interactor is positioned with respect to the narrative through their physical (object and bodily) interactions. The three spectra are described in detail in the next section.