Embodied Engagement with Narrative: A Design Framework for Presenting Cultural Heritage Artifacts

Abstract

An increasing number of museum exhibits incorporate multi-modal technologies and interactions; yet these media divert visitors’ attention away from the cultural heritage artifacts on display. This paper proposes an overarching conceptual structure for designing tangible and embodied narrative interaction with cultural heritage artifacts within a museum exhibit so that visitors can interact with them to comprehend their cultural context. The Tangible and Embodied Narrative Framework (TENF) consists of three spectra (diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory) and, considering how different interactions map along these three spectra, can guide designers in the way they integrate digital media, narrative, and embodiment. In this paper, we examine interactive narrative scholarship, existing frameworks for tangible and embodied interactions, and tangible and embodied narrative projects. We then describe the design of the TENF and its application to the pilot project, Mapping Place, and to the case study project, Multi-Sensory Prayer Nuts. The findings indicate that embodied engagement with artifacts through a narrative role can help visitors (1) contextualize the meaning of artifacts and (2) make personalized connections to the artifacts. Based on this work, we suggest design recommendations for tailoring the use of the TENF in the cultural heritage domain: simulate cultural practices, associate visitors with cultural perspectives, and provide simultaneous digital feedback. We conclude by describing future directions for the research, which include generating other possible projects using the TENF; collaborating with other designers and museum professionals; and exploring applications of the TENF in museum spaces.

1. Introduction

This research presents a Tangible and Embodied Narrative Framework (TENF) to design digital media that engage visitors with cultural heritage artifacts in museums. Concepts of embodiment posit that our physical state and situated-ness in the world provide an interactional context and bodily engagement for sense-making [1]. Tangible user interfaces, which couple computational resources with real-world objects and our physical environment through embedded digital media, allow interactors to have an embodied interaction with computational data through gestural and expressive interactions [2,3]. Current ways of presentation (e.g., descriptive texts, images, audiovisual guides) of cultural heritage artifacts (e.g., pottery, bronze/wooden tools, masks, garments) lack context and narrativity and do not connect visitors with the artifacts [4,5]. The sensory experience of touching cultural heritage artifacts offers a numinous experience that elicits the meaning of an object [6] and forges an intimate personal connection with an artifact and its creators [5]. Designers and museum professionals have attempted to address this phenomenon by creating tangible and embodied interactions such as tangible replicas or hybrid mixed media environments that integrate the material properties and digital representation of artifacts so that visitors are stimulated to explore and reflect on their meaning [7,8]. Using such tangible interactions, on the one hand, transforms how a museum displays deliver content to visitors through various hands-on experiences; they support reflective thinking [9], promote collaborative learning [10], and evoke a sense of discovery and curiosity [11]. On the other hand, tangible interactions divert visitors’ attention away from the actual content, requiring them to first learn how to deal with an interface, which takes time and effort that they could be spending on reflecting on and gaining a true understanding of the content [12]. As museum professionals emphasize that museum exhibits should evoke visitors’ narrative imagination [13] and facilitate their active participation and interpretive processes [14], the design of tangible and embodied interactions should align with these goals.

This research uses mixed methods to demonstrate how tangible interactions can be designed to engage museum visitors with cultural heritage artifacts. It first examines interactive narrative scholarship and existing frameworks for tangible interaction to highlight the need for a narrative framework. Findings from the literature review and a study of existing frameworks are used to develop a comprehensive framework, the TENF, which engages visitors with cultural heritage artifacts and related practices. The paper then illustrates the TENF through two projects: a pilot project, the Mapping Place, of a multi-touch tabletop that visitors can use to tell stories; and a case study, Multi-Sensory Prayer Nuts, of a tangible replica of prayer nuts. It then suggests design recommendations for tailoring the use of the TENF in the cultural heritage domain. Finally, it presents the limitations of the research and future directions to generate other possible projects using the TENF.

2. Literature Review

2.1. Interactive Narrative

Digital media such as mobile applications, web applications, or digital games have transformed stories, which are told in an interactive format. In interactive narratives, interactors create or influence a dramatic storyline through their actions [15]. Examining interactive narrative scholarship provides insight into how the design of digital media can evoke visitors’ engagement with cultural heritage artifacts in museum exhibits. Ryan describes interactive narrative as providing narrative engagement to interactors through the concepts of immersion and interactivity. Ryan aggregates various forms of interactive narrative experience through a framework that identifies diverse modes of interactivity in terms of the interactor’s involvement with the two narrative elements: character and plot [16]. Her framework consists of two binary pairs—internal vs. external (describing involvement with the character) and ontological vs. exploratory (describing involvement with the plot) that are combined to offer four different types of interactive narrative experience. This framework maps out various interactive narrative experiences of the interactor such as playing the role of the protagonist or of the creator who constructs a story.

Ryan’s framework informs the framework we propose in this paper toward identifying interaction as diverse modes of engagement with a narrative role. Ryan considers narrative as a mental phenomenon; however, her framework does not offer a way to represent physical experience, thus it seems to offer a limited understanding of interactivity. In her view, immersion happens when the interactor transports him/herself into the virtual world by relating to an avatar in the story and imagining the details of the storyworld, which takes place in one’s mind [17].

In contrast, Murray embraces physical experience by viewing narrative as a “threshold” experience between the imagined and the real worlds [18]. She proposes a “threshold object” as a physical device that appears both in the story and the real world [19]. A threshold object not only immerses one in the story, but also assists with one’s participation. This participation provides the pleasure of agency to the interactor to actively play a narrative role and transform the narrative outcome. Murray gives the example of an arcade game interface to show how an object one engages with within a story can draw one into the story. Murray states: “Ideally, every object in a digital narrative, no matter how sophisticated the story, should offer the interactor as clear a sense of agency and as direct a connection to the immersive world as I felt in the arcade holding a six-shooter-shaped laser gun and blasting away at the outlaws in Mad Dog McCree ” [18] (p. 146).

Murray uses the term diegetic to describe how an object exists in both the physical and virtual worlds to help interactors engage with a story. The concept of diegetic is also used in film. Film utilizes the term to distinguish background sounds, which cannot be heard by characters in the movie but are added for the audience’s ears only to enhance the emotional impact on the viewer, from actual sounds that are heard by the characters in the movie [20]. Along with the concept of diegetic, we propose to use the concept of non-diegetic. For example, a theme song played in the background of a movie would be non-diegetic, since the actors within the story cannot hear it. In contrast, the sound of a gun blast in the story would be diegetic, since the actors within the story can hear it. The concepts of diegetic and non-diegetic inform the proposed framework in this paper by showing that the tangible and embodied interactions in a narrative space shape narrative engagement and interpretation of the story. This paper extends interactive narrative research by examining studies on tangible and embodied interactions to inform how physical interactions can be utilized for narrative engagement.

2.2. Tangible and Embodied Narrative Systems

While museums have been implementing digital technologies in their exhibits, the projects typically focus on educational goals at the expense of experimenting with multi-modal technologies and interactions to reach those goals. Examination of existing non-museum projects that integrate embodied interaction, digital media, and narrative can guide the design of interactive experiences in museums. A growing number of interactive narrative systems have used tangible and embodied interfaces to integrate the interactor’s physical environment. For example, a digital tabletop that tracks physical objects can enable moving around tangible figurines to examine the various aspects of a story as if playing with chess pieces [21]; virtual reality (VR) environments for games can provide sensory feedback that leads to immersion in the scene [22]; and educational interfaces can augment learning by integrating the story with tangible puzzle pieces [23].

Several frameworks on tangible and embodied interaction provide insight on how to integrate digital media and tangible/embodied interaction. Ullmer and Ishii focus on how tangible representation of digital data allows manipulation of computationally mediated digital information [3]. This view has been called the “data-centered view” [2]. Although Ullmer and Ishii’s framework does not provide specific guidance with respect to narrative design, we can look to the Triangles as an example of a tangible narrative system that shows a data-centered design, as each tangible triangular piece represents different narrative elements such as characters, events, setting, and dialogue [23]. The magnetic triangular pieces can be stitched together in a variety of potential combinations, each creating different configurations of three stories (Cinderella 2000, Galapagos!, and The Digital Veil). The interactor thus manipulates the triangles, thereby potentially assembling a variety of configurations and hearing different parts of the story. For example, in the story of Cinderella 2000, snapping together the image of Cinderella’s evil stepmother’s face with the image of her house would play the audio clip of Cinderella’s step mother yelling at Cinderella. An example of a similar interactive system in a museum exhibit is an installation in the Exploratorium in San Francisco that allows visitors to explore phytoplankton populations in the world’s oceans [24]. The installation consists of a physical ring that functions as a digital magnifying glass placed on top of an interactive multi-touch tabletop. Visitors can move the physical ring anywhere on the tabletop to look at the type and proportion of phytoplankton in different areas of the oceans. The physical ring utilized in the Exploratorium successfully promotes visitors’ participation. However, such interactions remain relatively simple (e.g., browsing or navigating) compared to the interactions resulting from Triangles cited above.

Other researchers focus on the human body as shaping embodied engagement in the world, thus proposing integration of computer systems with our physical experiences [25]. The turn to the human body has inspired how the tangible system has become a resource for human action, in which meaning is created [26,27]. This view has been called the “expressive-movement-centered view” [2]. This literature proposes utilizing the sensory richness of the forms integrated with the interactive system to support the interactor’s meaning-making. An example of a tangible interface that allows expressive movement in a narrative is the Nintendo Wiimote game console device [28]. The device can be held in one’s palm and allows the interactor to make bodily gestures such as shaking, pointing, and waving using one’s arms in order to participate in the game. For example, when playing virtual golf, the Wii console provides sensory feedback that evokes thoughts and imagination, as if one were engulfed in the golf field as a golf player. An example of a sensory experience provided in a museum is the installation of the Lunch Counter Simulation at the Center for Civil and Human Rights in Atlanta [29]. The installation physically re-enacts the sit-in by providing a countertop, four bar stools, and headphones where visitors can sit and listen. Visitors hear the voices of people yelling and pounding on the table, and feel the vibration through the bar stools, thus, sensing the brutality and tension of the scene almost as though they were present during the actual event. Although the multi-sensory experience evokes thoughts about a poignant historic event, visitors have less interactivity and bodily engagement compared to those offered by non-museum projects.

Other researchers have devised tangible interactive systems for human sense-making in a spatial environment where the meaning of a tangible interaction, instead of being predefined by the designers, is created ‘in situ’, through situated social positioning and sensorimotor coupling [30]. For instance, the interactive system NOOT provides a playback device through physical tags with RFID that can create audio-recordings so that interactors can revisit conversations in a relevant spatial setting [30]. An example of a museum piece that provides space for social interaction is Youtopia, a multi-touch tabletop installation with tangible objects that teaches sustainability [10]. The table displays an interactive map, and interactors can manipulate tangible stamps, menu blocks, and an information ring tool on the tabletop for land-use planning. The tabletop setup provides space for multiple interactors to gather around the map, and to discuss and negotiate their choices in order to make sustainable decisions.

Other works highlight the potential of tangible and embodied interactions for engaging with narrative perspectives and supporting interpretation. For example, based on the storytelling convention of multiple point-of-view narratives, Tangible Spatial Narratives utilizes tangible pawns on a digital tabletop to tell stories of an event from various points of view [21]. The pawns represent characters in a story and can be moved around the tabletop surface, which shows a spatial arrangement of the narrative setting from a top-down view. The story is mapped within the spatio-temporal environment through the tangible pawns that represent different narrative points of view, allowing the interactor to examine various parts and layers of the story by placing and moving actual, physical pawns around the tabletop. A similar example, from a museum exhibit, is the interaction designed for the Hague and the Atlantic Wall: War in the City of Peace [31]. In this exhibit, visitors carry around six smart replicas of objects (e.g., tea bag, sugar packet, travel pass, arm band, mug, dictionary), which they can place in interactive kiosks to hear an audio version of the story of the construction of the Atlantic Wall during WWII in the voice of a German soldier, or a civilian, or a civil servant. What is successful about this exhibit is the way interaction is implemented to highlight diverse viewpoints of the people within a historical event.

Examining frameworks on tangible and embodied interaction as well as existing museum and non-museum tangible and embodied narrative projects, like those discussed above, sheds light on the design direction museum projects can take to better integrate interactive narrative and embodied interactions to engage visitors with narrative perspectives. We see the potential of using tangible and embodied interactions to engage with narrative perspectives in interpreting a story in projects such as Tangible Spatial Narratives [21] and in the interactions designed for pieces such as the Hague and the Atlantic Wall: War in the City of Peace [31]. Inspired by these works, we propose a narrative framework to further develop similar projects.

2.3. Toward a Narrative Framework

We extracted the following narrative design elements from the literature on interactive narratives and the aforementioned examples: (1) physical engagement, (2) narrative role, and (3) narrative consequence. These elements can help engage interactors with narrative perspectives to interpret a story.

Table 1. Narrative design elements for a framework identified in existing projects.

Type	Project Title	Physical Engagement	Narrative Role	Narrative Consequence
Non-museum Projects	Triangles	Triangle pieces represent characters, events, and setting	Interactor is a narrator overseeing the story	Story unfolds as the interactor stitches together the triangles
	Nintendo Wiimote	Interface enables gesturing as if one were immersed in the game world	Interactor is an actor in the game world	Appropriate game play can lead to winning or losing
	Tangible Spatial Narratives	Tangible pawns represent characters in the story	Interactor is a focalizer to examine the story	Moving around the pawns shows different parts of the story
	NOOT	Physical tags record different parts of a conversation	Interactor is a creator to record the story	The playback device plays the audio recordings when triggered
Museum Projects	Interaction to view Plankton Populations	Physical ring on an interactive tabletop functions as a magnifying glass	Interactor is an external focalizer	Moving around the physical ring shows the plankton populations
	Lunch Counter Simulation	Physical bar table and stools reenact a historic event for interactors to be immersed in	Interactor is a focalizer in the actual scene	Audio from the sit-in event is played without any interactivity
	Youtopia	Tangible objects on an interactive tabletop enable manipulation of the story	Interactor is a creator of a virtual city	Interactors develop a city in diverse ways that are sustainable
	The Hague and the Atlantic Wall exhibit	Tangible objects represent different perspectives to engage in the historic event	Interactor is a focalizer to examine the story	Moving around the objects shows different parts of the story

below illustrates the elements that emerge in each project.

Our examination of these elements in turn led us to devise the three spectra, (1) diegetic vs. non-diegetic, (2) internal vs. external, and (3) ontological vs. exploratory that comprise the framework we propose in the next section. In our previous work on tangible narrative frameworks, we examined existing tangible narrative systems and described the ways in which tangible media engage interactors with narrative systems across seven categories [32]. Here we look more broadly at both tangible as well as embodied interactions, while at the same time focusing on narrative engagement with cultural artifacts in museum contexts. Hence, we narrowed down our proposed framework to three categories, which we call “spectra,” that focus on engagement with a narrative role that helps visitors relate to the original users of a given cultural artifact.

The three spectra that comprise the TENF come from Ryan’s framework on interactivity describing internal vs. external and ontological vs. exploratory interactions, and Murray’s concept of diegetic. Ryan’s concepts depict interactivity through a narrative role and consequences; Murray’s concept of a diegetic interaction describes how a tangible and embodied interaction offers engagement with a narrative. As the projects illustrated in Table 1 make evident, the way(s) in which interactors physically engage with the narrative through tangible objects and/or bodily movements is an essential part of the interactive narrative experience. These interactions can be diegetic or non-diegetic, which takes into account the way an interactor is positioned with respect to the narrative through their physical (object and bodily) interactions. The three spectra are described in detail in the next section.

3. Tangible and Embodied Narrative Framework (TENF)

The proposed TENF consists of the three spectra introduced above, diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory, to help designers integrate the design elements of physical engagement, narrative role, and narrative consequences in various ways (see Figure 1). We describe the three spectra in detail here, and then discuss how they come together to form the TENF.

3.1. Diegetic vs. Non-Diegetic

The diegetic vs. non-diegetic spectrum (see Figure 1) describes the interactor’s position as he or she interacts within the narrative environment, which determines the physical modality of interaction. The diegetic vs. non-diegetic interaction can be distinguished by whether the interactor can physically situate him/herself within the story. An interaction is diegetic when a tangible and embodied interaction (e.g., with an artifact or an installation by taking a certain action or making a certain gesture) motivates the interactor to immerse him/herself (i.e., project his or her body) in the context (time and place) of the story. The interactor’s actions directly influence the story as if s/he had a physical presence in the time and place of the story. The interactor gains agency from his/her ability to move around and interact with the narrative environment through firsthand experience. An example of an interface that evokes diegetic interaction would be a race-car game which the interactor can control through a wearable or VR interface and have an experience close to that of actually racing, and feeling as though s/he were really driving the race car.

A tangible and embodied interaction is non-diegetic when it is limited to symbolic representations of narrative elements which the interactor conceptualizes to interact with and interpret the story, without being transported to the time and place of the story. In a non-diegetic interaction, the narrative environment is a conceptual representation of the story such as a map, narrative structure, or character dynamics. The interactor can manipulate these symbolic forms through the physically represented narrative environment. The interactor gains agency with his/her ability to access, navigate, or organize the story through abstract representations. An example of an interface that evokes non-diegetic interaction would be a race-car game, in which the interactor utilizes a remote control that simulates driving a race car seen on the screen, without really driving the car. In this non-diegetic situation, the interactor is fully aware of being outside the race car. Another example of an interface that evokes a non-diegetic interaction would be story blocks or puzzles that the interactor can reposition in order to alter the plot.

The diegetic or non-diegetic interaction offers physical experiences to imagine and relate to the cultural context such as time and place. When visitors have a diegetic interaction with an interface, rich-sensory metaphors and experiences evoke thoughts and imagination related to the cultural scene. These immersive interactive experiences can evoke a specific context of the artifact that is essential for understanding the value and importance of the artifact. For example, learning more about a culture by holding a dinner plate in one’s hands enriches the information that the plate conveys to visitors. When visitors have a non-diegetic interaction, physical properties of the interface promote reasoning about abstract concepts that underlie the story without drawing the visitors into the world of the story. A non-diegetic interaction can help interactors comprehend abstract concepts grounded in artifacts. For example, in a chess game, as the two opponents manipulate the chess pieces, they learn how to strategize. Such interactive systems with physical representations can promote reasoning of the dynamics of a complex structure of information.

3.2. Internal vs. External

The internal vs. external spectrum (see Figure 1) from Ryan’s framework captures the interactor’s narrative perspective in the story. While the diegetic vs. non-diegetic spectrum involves the embodiment of the interactor distinguished by whether the interactor can physically situate him/herself within the story, the internal vs. external spectrum involves a mental identification of the interactor with a narrative perspective. Ryan defines an internal interaction as one in which “users project themselves as members of the virtual world by identifying with an avatar, who can be shown from either a first-person or a third-person perspective” [16] (p. 108), such as playing the role of a race car driver. In an internal interaction, the interactor becomes an actor or a witness of the events, thus focalizing and interpreting the events in the story from a first-person perspective. An example of an internal mode would be the interactor playing the role of a protagonist in the story to intervene in events, or following the story to focalize and become a witness of events.

Ryan defines an external interaction as one in which “users are situated outside the virtual world. They either play the role of a god who controls the virtual world from above, or they conceptualize their own activity as navigating a database” [16] (p. 108). In this case, the interactor does not have an insider’s, first-person perspective. S/he only has access to ascertainable information related to the happenings of the story. S/he may impart an avatar’s thoughts and feelings within the story, but would not be identified with an avatar within the story. An example of an external mode would be viewing/listening to the narration of a story through navigating a database, or creating a story as god would, through character dynamics.

The internal or external interaction offers specific narrative perspectives that visitors can assume so that they can engage with artifacts and related practices. When visitors have an internal interaction, they play the roles of people in the time and place of historic events; thus they experience the narrative emotionally and empathize with the cultural dimension of the practices. When visitors have an external interaction, they examine the cultural concepts and context while playing the role of an omnipotent narrator or of an external member revisiting the historic events. Visitors’ interactions as they navigate or re-configure the story help them comprehend and connect with the original cultural practices.

3.3. Ontological vs. Exploratory

Ryan’s ontological vs. exploratory spectrum (see Figure 1) captures how the interactor interacts with the story’s plot. Ryan explains that narrative engagement is ontological if the interactor’s actions result in altering the events that influence the raw materials of the story; and exploratory if the interactor navigates the story to influence the organization of the story being told. The ontological mode offers the interactors’ actual participation to change the story. As a protagonist or as the creator of a story or when playing any other role, the interactor takes actions or decisions that lead to forking paths in the story. An example of an ontological experience would be assembling tangible puzzle blocks that represent narrative events or characters to generate stories.

In contrast to the ontological mode, in the exploratory mode, as the interactor interacts, he or she has access to different parts of the story and chooses his or her own navigational path, although without the ability to alter the outcome of the story. An example of an exploratory interaction would be handling tangible pawns to choose the point of view and location on a map to view stories from multiple points of view, while not intervening in any of the events in the story.

The ontological or exploratory interaction helps to communicate information and guide visitors’ experiences. As museum exhibits usually have complex stories to tell, interactivity through an ontological or exploratory interaction can effectively assist comprehension of such information. Ontological interactions with freedom to construct the plot or change the ending of the story can provide visitors an opportunity to express creativity and apply their knowledge of the concepts they have learned. The exploratory interaction, where visitors navigate the story without changing the ending of the story, can help familiarize visitors with how the artifact was used in particular scenarios. With cultural history museums presenting culturally distant information, exploratory interactions can assist visitors in imagining the cultural context of artifacts and journeying to the past.

3.4. Structure of the Tangible and Embodied Narrative Framework

The TENF provides an overarching conceptual structure for designing, generating, or analyzing tangible and embodied narrative interactions. Any tangible narrative project can be mapped onto the TENF according to how it falls along each spectrum. Thus, eight new combinations of diegetic/non-diegetic, internal/external, and ontological/exploratory can be generated, with each combination representing a different type of narrative engagement. Therefore, paying close attention to all three of the spectra can guide the design of museum exhibits to utilize physical engagement for visitors’ interactions with cultural artifacts and practices from the perspectives held in their original context.

Using the framework, a project can be mapped on a continuum of each of the three spectra. For example, as shown in the positioning of the dots in Figure 2, the Tangible Spatial Narratives [21] project has a moderate degree of non-diegetic interaction because the tangible pawns on a map blend symbolic and mimetic forms of representation; it has a moderate degree of external interaction, as the interactors identify as external members in the time and place of events; and it has a moderate degree of exploratory interaction, since the interactors examine and navigate different parts of the story instead of making any changes to it (see Figure 2).

As seen in Figure 3 below, the Nintendo Wiimote game console device [28] can be positioned in moderate degrees of diegetic interaction because it expresses mimetic elements, such as gesturing as if playing golf to simulate the story; it has completely internal interaction, since the interactor plays the role of an internal member, such as a golf athlete who takes part in the story; and it has a moderate degree of ontological interaction, as the interactor’s game play determines the direction the story takes, such as winning or losing the game (see Figure 3). As seen in the case of the two projects, different degrees of the three spectra on the TENF reflect various types and depth of tangible and embodied narrative engagement.

As can be seen from the above descriptions of the three spectra, diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory, the Tangible and Embodied Narrative Framework helps designers understand various types of tangible and embodied narrative engagement through the varying degrees of the three spectra. Our goal in designing the TENF is to enable designers to incorporate tangible and embodied interactions within a museum exhibit. There is potential in utilizing this framework when digitally recreating cultural artifacts and practices through multimodal technologies and interactions.

4. The Pilot and Case Study Projects

In the two projects below, Mapping Place, and Multi-Sensory Prayer Nuts, we wanted to create an interactive installation using digital and tangible media to simulate traditional cultural practices. We investigated how we could provide narrative engagement with a digitally recreated artifact through tangible and embodied interaction, with the aim of supporting comprehension of cultural concepts and heritage within a museum context. We developed and explored the TENF for this purpose. We developed the pilot project Mapping Place while we were still conceptualizing the framework; therefore, the process of designing and evaluating it guided us to formulate the TENF. We then explored how we might apply the TENF in the case study project Multi-Sensory Prayer Nuts.

4.1. Pilot Project: Mapping Place (2013–2014)

The Mapping Place was displayed in the exhibition, Mapping Place: Africa beyond Paper, from 28 February to 6 June 2014 in the Robert C. Williams Museum of Papermaking at Georgia Tech. The project was designed and developed by students, including the lead author of this paper, in Synlab at Georgia Tech and Ryerson University, under the guidance of the co-author, professor Ali Mazalek. It emulated a Lukasa board, a hand-sized wooden tablet studded with beads and shells used as a mnemonic device by the Luba peoples in the Congo (see Figure 4). The shape, color, location, and arrangement of these ornaments on the board represent stories and are meant to help learn about the identities, genealogy, and history of the tribe [33]. In Mapping Place, in order to highlight the non-linguistic meaning-making practices of the Lukasa board, we created an interactive tabletop that visitors could use to tell stories about their own families, friends, and neighborhood [34].

The Mapping Place consists of a multi-touch tabletop with tangible shells, two adjacent wall projections, and a wall-mounted television screen. To begin telling stories, visitors place a shell on the interactive tabletop, which represents a story. With each shell placed on the tabletop, icons of the story elements (girl, boy, woman, man, house, neighborhood, or a pet) are displayed around it. The tabletop consists of virtual beads of several different colors and shapes, which can be dragged into one of the story element categories. Once a virtual bead is dragged on top of a story icon, the bead transforms into a story-bead and a connection with a story-shell is formed (see Figure 5 and Figure 6). A corresponding animation is displayed on a wall projection to encourage visitors to share their stories with other visitors. Visitors can rearrange their story by dragging their story-shells or rearranging the connected story-beads. Once they are finished creating stories, they lift their tangible shells to see their own story-map: story-beads with virtual icons attached to the story-shell.

Evaluation of the Mapping Place

In order to assess visitors’ comprehension of the concepts of the Lukasa board based on their experience of using the interactive tabletop, we conducted a user study by inviting fourteen primary school-age children (the main target demographic of the exhibition) to interact with the system in groups of two to four. We observed the interactions and conducted semi-structured interviews with each of the groups. We used qualitative methods, in particular grounded theory [35], to analyze the data collected in the user study. The results of the study from the perspective of learning the abstract concepts of the Lukasa board are reported in [34]. Here, we describe the insights derived from the study that informed the design of the TENF.

The evaluation of Mapping Place revealed that although the interactive tabletop provides vivid examples to which visitors could relate while learning about the Lukasa board, they needed to have prior knowledge of how the Luba people utilized this particular artifact. Specifically, the cultural gap hindered visitors’ ability to make sense of cultural concepts purely based on their interactions around the table. While the interface was successful as a storytelling application, it did not offer an embodied experience of the cultural practice of the Lukasa board that was similar to how the Luba people would have engaged with it. Our findings led us to see the value of a framework that can guide the design of experiences that allow visitors to grasp the cultural and historical perspectives in the provenance of the artifacts.

When we were designing Mapping Place, we realized that a framework would be useful for expressing and thinking about the narrative design elements: physical engagement, narrative role, and narrative consequence. In terms of physical engagement, we based our design of Mapping Place on the way the Lukasa board was used in practice. The artifact served as an abstract tangible representation of stories. The creator of a Lukasa board decides how the various beads, shells, and carvings they place on the board will represent different tribal stories. The creator (or another tribe member) can then run their hands over the board in order to give an oral performance and interpretation of the stories it contains. This physical engagement with the beads and shells on the Lukasa board helped us see that we could define the physical interaction in Mapping Place as a non-diegetic interaction, as opposed to diegetic. In terms of narrative role, we related the visitor to a creator of a Lukasa board who recalls memories and narrates the story; hence, we saw that the interactor engages with an external perspective on the story as a narrator, as opposed to being an internal member in the story. In terms of narrative consequence, we allowed freedom for visitors to create their own story maps; we saw that this offered them an ontological interaction with the story, rather than exploratory. These reflections helped identify the three spectra for the TENF, diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory, to reflect those narrative design elements. Figure 7 illustrates how Mapping Place maps along the three spectra.

Our evaluation of the Mapping Place also taught us the importance of tailoring the framework for the cultural heritage domain. Decisions about where to situate an interactive design on each of the three spectra can help to frame visitor experiences in relation to the cultural heritage artifacts visitors are encountering in the museum. For example, by positioning the Mapping Place interactions as non-diegetic rather than diegetic, we were able to intentionally relate the visitors’ experiences to the abstract mapping practices of the Lukasa board. Similarly, the internal vs. external spectrum can be used to consider how visitors will relate to the perspectives held by the Luba people when they use a Lukasa board as external narrators. Lastly, the ontological vs. exploratory spectrum captures how visitors can create their own story map as a process of mnemonic meaning-making in an ontological mode. We therefore conceived of the following case study, Multi-Sensory Prayer Nuts, which intentionally focuses on utilizing the framework as a way toward priming interactors with the original cultural perspectives.

4.2. Case Study: Multi-Sensory Prayer Nuts (2014–2016)

The Multi-sensory Prayer Nuts project was conducted and led by four students in Synlab at Georgia Tech and Ryerson University to imagine the potential of the design space for tangible interaction with cultural heritage in a museum [36,37]. The project was not exhibited in a museum space, and therefore did not involve consultation with museum professionals. It focused on the prayer nut, a devotional object that laypeople used during the 16th century in the Netherlands (see Figure 8). In general, the prayer nut was meant to be a portable miniature of a sacred space, such as a cathedral. A prayer nut, as a kind of rosary, consists of two boxwood hemispheres around six centimeters in diameter with a clasp to open and close it. The interior shows intricate carvings on each hemisphere representing biblical scenes, and the exterior shows Gothic architecture. Fragrances were infused into the prayer nut or the prayer nut was used with beads made of scented materials. The prayer nut engaged various senses including sight, touch, and smell in order to assist with religious practices. Used during personal religious experiences, the prayer nut shows the dependence of spirituality on material objects during the 16th century [38,39,40].

Based on historic texts about the prayer nut, we focused on three different sensory aspects—tactile, somatic, and olfactory—in order to place the artifact in its original context and use. We created three different replicas of the artifact and connected it with sensors and microcontrollers to allow interaction; we then connected it with computers, monitors, and speakers to provide audio- and video animations about the contextual information in response to interactions.

The Multi-Sensory Prayer Nuts project consisted of three prototypes—Visual Voyage, Experiencing Spirituality, and Scents of Power—each highlighting different sensory practices with the prayer nut. The first prototype, Visual Voyage, focuses on tactile aspects of the intricate carvings of biblical scenes in the prayer nut and how these scenes can support religious contemplation. In many ways, interacting with an actual prayer nut in a spiritual context was compared to interacting with the Bible. First, the intricate carving required a magnifying glass to view it, similar to how one might have used a magnifying glass to read scriptures. Also, opening and closing the prayer nut was likened to opening and closing the Bible, or even the panels of a triptych [40].

The Visual Voyage prototype we designed provides a wooden sphere with carvings of biblical scenes. The surface of the prayer nut that the interactor can hold has three tactile points that can be activated upon gently touching its surface (see Figure 9). The interactor can then see corresponding scripture on the screen relating to the carving and an enlarged carving on the screen as if s/he were contemplating an actual prayer nut using a magnifying glass.

The second prototype, Experiencing Spirituality, focuses on breath as a sensory-bodily aspect of prayers, and how prayers might have assisted in moving from a secular place to a sacred place in the mind. In historical practice, prayer nuts were used to navigate through one’s religious practices while also serving as a symbol of wealth and piety among lay people [38]. In the interactive project, a screen shows a merchant from the 16th century holding the prayer nut. The design intention is to encourage the interactor to hold the prayer nut in a similar way. When the interactor holds the prayer nut, the screen shows a close-up of hands holding the prayer nut, while one can hear the sound of breathing and marketplace noises. This is to reference that the prayer nut is used to assist one in finding peace in one’s heart. Holding the prayer nut for eight seconds causes the background to change from a market scene to a cathedral, along with the sounds that change to liturgy and a choir singing (see Figure 10).

The third prototype, Scents of Power, focuses on the belief in the magical power of the olfactory aspects of the prayer nut. During the 16th century, people put “sweet-smelling ingredients” such as herbs, oils, and dried flowers in their prayer nuts [41]. They believed such scents had magical power to ward off evil [42]. To emulate the original scents of the prayer nut, we brushed the inner surface of the prototype with essential oils, such as lavender and cinnamon. We attached the prototype to a table with a projection overlaid from the top, projecting images into the prayer nut. Opening and closing the prayer nut causes the projection to show animations of the scents (see Figure 11).

The above-described prototypes of the Multi-Sensory Prayer Nuts enable interactors to identify with a fictional character to discover and find meaning through all three spectra, and to have diegetic, internal, and exploratory narrative experiences, as shown in Figure 12 below. The project falls within the diegetic side of the spectrum because visitors can make gestures similar to those the actual user of the prayer nuts would have made, thereby empathizing with them. This interaction displays a moderate degree of the diegetic interaction because it contains some symbolic elements, while suggesting that interactors immerse themselves in the experience. The project falls within the internal side of the spectrum because it positions the interactor to identify with the original people who used the artifact, i.e., the prayer nuts. It shows a moderate form of internal interaction because interactors only focalize through the perspectives of the original people rather than actively intervening in the outcome. The project falls within the exploratory side of the spectrum because visitors can see how the prayer nuts are used in their original time and place, yet they cannot change the progression of the narrative. It shows a strong form of an exploratory interaction because interactors are able to examine the process of using the prayer nuts following a predesigned path (see Figure 12).

The prayer nut replica was modeled in 3D software and it was printed in 3D using plastic. In the 3D model, we simplified the carvings of the prayer nut such that it could be printed in plastic, while providing a look and feel that was sufficiently similar to the original. To replicate the intricate carvings of the relief sculpture in the interior of the prayer nut, we utilized a photographic image converted to a 3D model through software for conversion. Various types of sensors were embedded within the plastic prayer nut and connected to microcontrollers and computers. The Visual Voyage utilized capacitive paint to sense touching on different parts of the carvings. For Experiencing Spirituality, we utilized a capacitive sensor to determine how long the interactor may be holding the prototype. For Scents of Power, we utilized a magnetic sensor to detect the opening and closure of the prayer nut. The audio and video were displayed through the Processing program upon interaction.

4.2.1. Evaluation of the Multi-Sensory Prayer Nuts

The goal of the user test was to examine how interactors’ embodied and sensory interaction, combined with the narrative role, helps interactors understand more about an object and its use. We evaluated the three prototypes in a lab environment with 13 participants (eight male and five female), 11 of whom were graduate students at Georgia Tech. We invited participants in ones, twos, or threes and conducted pre- and post-task semi-structured interviews before and after the interactive sessions. The pre-task interview asked about the usual ways participants encounter multi-modal experiences in their everyday lives. The post-task interview prompted participants to relate how they utilized their sensory experiences to engage with and make meaning from the artifact. We asked open-ended questions such as “How did you make meaning from your experience?” Such a question required participants to talk about their experiences; we then followed up with corresponding questions to participants’ responses.

The three prototypes for the interaction session were randomly placed in a triangular formation. We did not limit the time, and the participants spent on average 1~3 min with each prototype. In front of each interactive prototype, we created pictorial descriptions of how to interact during each session. Each prototype was accompanied by a visual guide on how to interact with the prototype, along with a short description of the historical meaning of such an interaction with the actual artifact. This was to ensure the interactors would not get confused about what they were to do with the prototype. We video-recorded and later transcribed the interviews and the participants’ interactions with the prayer nuts, and analyzed the interview results through qualitative methods, in particular, discourse analysis [43]. We derived two findings: the project enabled visitors to (1) contextualize the artifacts and to (2) make personalized connections. We discuss these findings in detail below.

Contextualize Artifacts

The exploratory and diegetic interactions in the Multi-Sensory Prayer Nuts project offered a hands-on experience on how the artifact was used in its originating cultural context. The audio and visual cues showing the usage of the prayer nuts organized in an exploratory interaction triggered participants to reflect on the people who historically used prayer nuts and on the religious activities associated with them, which was our intention in the design. During the post-task interview, participants shared the fact that sensory interactions with the prayer nuts helped them contextualize the use of the prayer nuts by providing information that would not have been available through a simple text or visual display. As one participant stated, “It is more that, if I just saw the object, I would be, ‘oh, this is a pretty item to use in the 16th century.’ Instead I am like, ‘oh, people would take this on the street, and maybe look at it, smell it, and this is how it made them feel’.” This comment shows that the exploratory interaction supported participants in imagining how the artifact was used in its original cultural context and appreciating its meaning within that provenance.

Most importantly, presenting sensory interactions within a narrative supported contextualizing the meaning of interaction through cultural perspectives. Through a diegetic interaction, participants learned what it would be like to be transported into the context of the cultural practice. Participants took the perspective of a potential user of the prayer nuts in the 16th-century Netherlands so they could approximate that person’s experience. They were able to understand ‘why’ the artifact was meaningful to the original users of the prayer nuts. As another participant shared, “By touching it, it tells you more about what the objects were really meant for … Because it is now there for you to know these things exist. It is there for you to understand how, when, and why people used these things a long time ago.” Clearly, this participant’s sensory experience had triggered her to identify the value and meaning of the artifact based on its religious context. Another participant described how the multi-sensory experience helped contextualize the artifact: “You feel like you are actually experiencing the object and thinking about the context. The evocative illustrations, and the use of sound, all these senses … it really puts it in context and it is not just an artifact divorced from context.” The above remark shows that being able to handle the prototypes of the Multi-Sensory Prayer Nuts and having an exploratory and diegetic interaction prompted participants to make connections with the past, and furthermore, evoked contextual meanings of the artifact that are essential for understanding its value and importance.

Make Personalized Connections

Overall, participants were motivated to imagine the use of the prayer nuts and learn more about them. In addition, offering an internal narrative perspective led participants to empathize with the cultural dimension of the practices. As one participant shared, “Before, I didn’t know about the prayer nuts, but now I know what they are, and I know how to use them; I can feel some similar feelings with them. So it was very informative and interesting.” This participant’s hands-on experience with the prayer nuts offered her a way to personalize that experience and relate to the thoughts and feelings of the original users of the prayer nuts. Another participant, who assumed the first-person perspective as a hypothetical 16th-century figure, said: “As I was touching the object, I felt like I was becoming that person, because I was interacting with the object, and I was also experiencing what those people were experiencing in the 16th century. So, I felt I was entering that person’s life as opposed to just reading about it.” With these words, this participant sounded like she was bridging the gap between herself and the actual users of the prayer nuts, which was the outcome we had intended in designing the Multisensory Prayer Nuts through an internal interaction.

The internal interaction provided through sensory experience offered participants personalized connections with the artifacts and practices. Participants’ interactions led some to describe their experiences by making connections to relevant personal memories. Some compared the prayer nuts to modern objects with which we engage, such as toys, cell phones, and jewelry. One participant said: “It is really a fun and an enchanting idea to me, to think of little worlds contained in small things, and I guess that was the appeal of toys that I grew up with, like Polly Pocket and Mighty Max, which is a boy version. I loved them a lot and played with them a lot.” This participant’s ability to relate to the prayer nuts with his personal memories and experiences helped him understand the meaning and value of the artifact. As another participant shared, “I was really having this intimate moment with this object, which I think is what these objects tend to be about. They are personal.” Indeed, for this participant, too, encountering and handling the prayer nuts offered new routes to making a personalized connection with a different time period and culture. These participants’ reports show that an internal interaction offers an opportunity to relate one’s current experiences to the artifact’s provenance, thereby empathizing and making personal connections.

This evaluation of the case study project Multisensory Prayer Nuts was limited to a few (13) interviewees who represent only a type of users, graduate students, in a lab setting. Thus, the results do not provide empirical data for generalizable conclusions. Despite this limitation, the case study showed us that the TENF could serve as a map for providing various types of embodied interactions combined with a narrative role, to understand a cultural artifact and its related uses. The two findings of the Multisensory Prayer Nuts project about how participants contextualize artifacts and make personalized connections show the design consequences of a project that used a combination of diegetic, internal, and exploratory interactions, per the spectra of the TENF.

4.3. Conclusion to the Pilot and Case Study Projects

Drawing on the pilot project Mapping Place, as well as our literature review, we were able to define the Tangible and Embodied Narrative Framework based on three spectra that would engage visitors with cultural perspectives through tangible and embodied interactions. We saw that the TENF could support visitors’ narrative engagement, thus bridging the gap between their own cultural perspectives and that of the originating culture. Therefore, we designed a case study project in which we could consider the three spectra: diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory, in our design process. In the Multi-Sensory Prayer Nuts, we more fully explored how we could utilize the TENF: by considering the design possibilities of the three spectra, we designed the interactions we wanted museum visitors to have which would induce their narrative involvement with cultural perspectives. Applying the TENF to the Multi-Sensory Prayer Nuts showed us that interactive museum experiences can bring contextualized and individualized experiences of cultural heritage artifacts. Both projects helped us identify the importance of paying special attention to the different cultural contexts of artifacts and related practices when designing tangible narrative projects for cultural museums.

5. Discussion

This paper presented research that proposes a narrative design method for creating tangible and embodied interaction to engage museum visitors with cultural artifacts and their related practices. It reviewed the literature on interactive narrative and researched existing tangible narrative frameworks and projects to develop a comprehensive framework, the Tangible and Embodied Narrative Framework. We used this framework to explore its application in a pilot project, Mapping Place, and in a case study project, Multi-Sensory Prayer Nuts.

5.1. Design Recommendations

Based on the two projects and reflecting on the TENF, we found that the TENF provides guidance on how to integrate cultural heritage, digital media, narrative, and embodied interaction in a cohesive way to support visitors’ meaningful engagement with artifacts and related practices. The design recommendations discussed below can guide future designers in using the TENF for designing interactions with cultural heritage artifacts and related practices. They provide insight into how the Tangible and Embodied Narrative Framework (TENF) can be applied specifically in designing interactions for cultural heritage museum contexts. These recommendations can guide designers in integrating narrative interaction with embodiment in their projects in order to offer engagement with cultural heritage artifacts, which can help visitors journey back to the original, likely foreign, cultural context of an artifact. They imply that sense-making activities should be designed based on a cultural dimension; and the design should assist visitors in interpreting the meaning of an artifact based on its provenance.

5.1.1. Simulate Cultural Practices

Digitally designed artifacts (e.g., the prayer nuts) and their interactions can simulate cultural forms and practices in diegetic or non-diegetic ways to offer physical engagement. Digitally recreating a practice (e.g., sensory experiences with the prayer nuts) does not necessarily mean replicating the experience; but rather, the design must abstract and present some aspects of a cultural form to support visitors in drawing connections between the digital replica and the original artifact. Such a design could include the interactor’s bodily gestures (e.g., pointing, dragging, holding, shaking) and sensory experiences (e.g., smelling, touching, hearing) paired with symbolic meaning (e.g., mnemonic practices, meditation, passage of time) associated with the artifact. In these ways, designers can utilize the integrated physical and digital forms to recreate cultural experiences and evoke thoughts about the original practice.

5.1.2. Associate Visitors with Cultural Perspectives

A good design for tangible and embodied interaction must assign an internal or external narrative role associated with the original context that would best engage the visitor with the perspective held by the people of the original culture. Interactions must associate visitors with cultural perspectives that can help interpret the meaning of an artifact or cultural practice in its original context. The designer must take special care to consider how the original users interpreted their practices associated with an artifact. If the artifact embodies a unique and unfamiliar meaning that is difficult to grasp, priming the visitor regarding its original use prior to interaction may be necessary to help visitors comprehend the underlying meaning of the artifact.

5.1.3. Provide Simultaneous Digital Feedback

To achieve an effective design of tangible and embodied interactions, simultaneous digital feedback should be provided in ontological or exploratory ways as interactions unfold. When visitors receive feedback upon a particular gesture they make, they are able to adjust their gesture by making a more intentional one in the next moment; and down the line, they learn about and understand the customs and traditions of the practices of another culture. Seeing the consequences of their interactions, visitors will remain interested to engage with the digitally designed artifacts and their related practices. Also, such feedback will enable visitors to interpret the meaning of tangible and embodied interactions grounded in the provenance of the artifact. Providing feedback through audio, video, or sensory cues can avoid confusion in making meaning out of unfamiliar, therefore unique, interactions with an artifact (e.g., prayer nuts). In doing so, the design can support grounding visitors’ interpretation somewhere closer to what it would be like to perform a particular cultural practice (i.e., multisensory experiences of religious rituals using the prayer nuts) in its provenance. Also, sensory cues provide embodied metaphors that map on to previous experiences to leverage the interactor’s comprehension of abstract concepts [44].

5.2. Limitations and Future Directions

Reflecting on the design, development, and evaluation of the TENF as applied to the case study project Multi-Sensory Prayer Nuts led us to identify four limitations and future directions of research. First, the authors’ conclusions from the Multi-Sensory Prayer Nuts are limited to only one combination (diegetic, internal, and exploratory) out of eight possible combinations of spectra that comprise the proposed framework. Thus, the findings are not generalizable; designing a project using another combination, for example, non-diegetic, external, and exploratory, would yield different results. Conducting several other projects in which the interactions map along the spectra of the framework in different ways would show other uses and design consequences of the TENF and would provide different means of reflection. For example, a non-diegetic, external, and ontological interaction may prompt the interactor to grasp abstract concepts and a diegetic, internal and exploratory interaction may have the interactor immerse him/herself in the provenance of the artifact and, thus, empathize with the original people who used it. Future tangible narrative projects can evaluate uses of different combinations of the spectra for their effectiveness in engaging interactors.

Another limitation of the study is in the way we conceptualized the interactions as being fixed on a point in a spectrum by the designers. However, we learned that throughout an interaction, one can move along the spectrum. In this way, the TENF can be used to observe and track the interactor’s experience as it fluctuates during his or her interaction. The interactor’s experience can begin as being non-diegetic when he or she is unable to relate his or her experience to the narrative, and it may move toward being a diegetic experience, as s/he makes a connection between his or her embodied experience and the narrative. In the Multi-Sensory Prayer Nuts, the non-diegetic interaction prompts the interactor to navigate different parts of the story, while the diegetic interaction supports him or her to also reflect on the original context of the artifact. To avoid this interpretive slippage, designers of future tangible narrative projects can further consider interaction as a process of forging narrative engagement.

Furthermore, the framework was primarily designed and applied from the authors’ perspective. We have not exposed it to other designers or museum curators yet; nor did we incorporate their feedback and perspectives. Designers can create future projects that offer interactive experiences in museum spaces using the Tangible and Embodied Narrative Framework in collaboration with museum curators to identify its value or limitations. Co-designing with the participation of museum professionals can better integrate interactive technologies into existing museum systems [45]. Given museum professionals’ emphasis on exhibits that evoke visitors’ narrative imagination and facilitate their active participation, future designers’ efforts should align with their goals.

While developing and conducting the case study project in a lab setting enabled us to freely explore potential design spaces, it also limited us as we were not able to examine the in situ reactions of visitors. Implementing the project in a museum setting may raise new issues, challenges, and political debate around authenticity and technoheritage [46]. Future research may explore challenges around digital cultural heritage when exhibiting tangible replicas of cultural heritage artifacts in museums.

5.3. Conclusions

The Tangible and Embodied Narrative Framework (TENF) provides designers with an overarching conceptual model for designing tangible and embodied narrative experiences with cultural heritage artifacts and practices for museum visitors. The TENF is comprised of three spectra (diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory), and can guide designers to consider where the interactions they design might fall along these spectra in order to integrate digital media, narrative, and embodiment in a way that engages visitors with cultural heritage content. The various ways that interactions can be positioned along the three spectra suggest a vision toward generating new types of projects with tangible and embodied interaction with narrative. The digitally recreated cultural practices informed by the TENF can help visitors interact with cultural artifacts from the perspectives held at the original time and place of such artifacts and practices. The goal is to help visitors imagine and come closer to the lived experiences of the users of the original artifact so that they can comprehend the meaning of an artifact in its provenance. Through the pilot project, Mapping Place, and the case study project, Multi-Sensory Prayer Nuts, the authors illustrated applications of the TENF. Likewise, museums can utilize the TENF to incorporate multimodal technologies and interactions in their presentations of cultural heritage artifacts. The new directions suggested by the TENF can transform future museum exhibits to be even more experiential, individualized, and participatory.