Choreographic and Somatic Approaches for the Development of Expressive Robotic Systems

Abstract

As robotic systems are moved out of factory work cells into human-facing environments questions of choreography become central to their design, placement, and application. With a human viewer or counterpart present, a system will automatically be interpreted within context, style of movement, and form factor by human beings as animate elements of their environment. The interpretation by this human counterpart is critical to the success of the system’s integration: “knobs” on the system need to make sense to a human counterpart; an artificial agent should have a way of notifying a human counterpart of a change in system state, possibly through motion profiles; and the motion of a human counterpart may have important contextual clues for task completion. Thus, professional choreographers, dance practitioners, and movement analysts are critical to research in robotics. They have design methods for movement that align with human audience perception; they can help identify simplified features of movement that will effectively accomplish human-robot interaction goals; and they have detailed knowledge of the capacity of human movement. This article provides approaches employed by one research lab, specific impacts on technical and artistic projects within, and principles that may guide future such work. The background section reports on choreography, somatic perspectives, improvisation, the Laban/Bartenieff Movement System, and robotics. From this context methods including embodied exercises, writing prompts, and community building activities have been developed to facilitate interdisciplinary research. The results of this work are presented as an overview of a smattering of projects in areas like high-level motion planning, software development for rapid prototyping of movement, artistic output, and user studies that help understand how people interpret movement. Finally, guiding principles for other groups to adopt are posited.

1. Introduction

Domin: ...Man is a being that does things such as feeling happiness, plays the violin, likes to go for a walk, and all sorts of other things which are simply not needed. No, wait. Which are simply not needed for activities such as weaving or calculating. A petrol engine doesn’t have any ornaments or tassels on it, and making an artificial worker is just like making a petrol engine. The simpler you make production the better you make the product. What sort of worker do you think is the best?

Helena: The best sort of worker? I suppose one who is honest and dedicated.

Domin: No. The best sort of worker is the cheapest worker. The one that has the least needs. What young Rossum invented was a worker with the least needs possible. He had to make him simpler. He threw out everything that wasn’t of direct use in his work, that’s to say, he threw out the man and put in the robot. Miss Glory, robots are not people. They are mechanically much better than we are, they have an amazing ability to understand things, but they don’t have a soul. Young Rossum created something much more sophisticated than Nature ever did—technically at least!

—RUR (Rossums’ Universal Robots) Čapek (2004)

Bringing together choreography and engineering raises a question about whether both fields are concerned with the same academic values and inquiries. That is, are they motivated by the same organizing principles and can their practices be complementary, and further mutually beneficial, when brought together? In the above excerpt from the play where the term “robot” was coined, the immediate intertwining of robotics and the arts can be observed. From the first moment the concept of a “robot” was conceived there was an idea injected that movement could be divided between its efficient, functional parts and its extraneous, expressive parts. Thus, the idea of efficiency and function has permeated robotics since the word was introduced in a 1920 play (Čapek 2004). Long before they were called robots, automatic mechanical machines, automata, have existed since antiquity and as Truitt writes “are mimetic objects that dramatize the structure of the cosmos and humankind’s role in it” Truitt (2015), pointing to the long held mysticism and romance associated with complex machines. These concepts of efficiency and mysticism create a contradictory dichotomy that continues to influence the field of robotics today that may be resolved by considering the vastness of human motion as investigated through body-based methods in this paper.

At the same time, a choreographer Rudolf Laban (1879–1958) was working to establish a system for movement, now called the Laban/Bartenieff Movement System (LBMS), that helped codify learning from work in the arts and helped frame a rejection of industrialization and return to nature (Bradley 2008). Thus, both the idea of a robot and this somatic body-based movement system that will be leveraged heavily in the methods presented here are in part a reaction to the industrial revolution where mechanical efficiency—and its abstract ideal—was taken to hyperbolic levels not previously seen. Dancers’ technique and training often reflects the ethos of developing efficiency through iteration of motor tasks, continually refined and streamlined. But, dance performance requires the use of something “more”—maybe the same thing Domin and Helena discuss as missing from the concept of a robot.

Thus, as much as physical phenomenon, the distinction between human and robot movement is a philosophical distinction. In broad, reductive strokes, we argue that both roboticists and choreographers aim to do the same thing: to understand and convey subtle choices in movement within a given context. One of the fundamental emergent questions from this categorization is about the lens through which constituents in each field methodologically approach how to generate, interpret, and reproduce different movement from the idiosyncratic to the robotic. Consider two branches of inquiry, one concerned with internal experience and the other with external measurement as a model for understanding the differences in assumptions generated from each field.

Different branches of philosophical inquiry provide critical, albeit artificial, lenses by which to address this complex question. Two such branches, rhetoric and phenomenology, are useful in understanding subjective and objective experience, though notably from different angles. Rhetoric, at least in the classical (Aristotelian) sense, investigates “the available means of persuasion” in any given situation (Kennedy et al. 2006). Outside of ancient Greek life, contemporary rhetoric can be understood as a practice to examine the function, efficiency, and efficacy of persuasive communication—starting with linguistic, and moving to the visual, bodily, energetic, posthuman, and so on (Barnett and Boyle 2016). Alternatively, phenomenology provides a lens through which to understand subjective, first-person experience. Championed by Edmund Husserl at the turn of the twentieth century, phenomenology urges a “bracketing” of experience such that an individual approaches stimuli as if for the first time (Husserl 2012). Phenomenology and rhetoric are not, strictly speaking, incongruous. One can approach the creation of rhetorical artifacts (i.e., speeches, visualizations, choreographies, etc.) phenomenologically, just as one can probe first-person experience rhetorically. These lenses are artificial constructs—they offer a way by which to view the world but their limitations are neither absolute nor universal (Hawhee 2004).

Returning to the questions of engineers and choreographers, a problem can occur when either one becomes too beholden to their value systems. For engineers, the problem occurs when valuing efficiency and function (i.e., rhetoric) trump user experience (i.e., phenomenology). Conversely, for choreographers, the problem occurs when first-person experience (of the performers) is not observable by an external viewer (and therefore, maybe one day, quantified). As with any practice, balance is imperative. To be able to address complex questions pertaining to generating, interpreting, and reproducing movement, relying too heavily on rhetorical or phenomenological aims can be limiting in a way that may be invisible or simply not salient for the agents. This is one reason, among many, to support bringing engineers and choreographers together into one shared space, and, for a robotics lab, to employ phenomenologically motivated practice in our research.

In the largest sense, the purpose of the methods presented here is to explore what human bodies can express in movement, what ideas machines can express in movement and whether they are the same things. Can a robot expressively communicate? If we assert that all movement is expressive (which we do), then can a robot express the same set of things as a human? The other big idea is to utilize the choreography of the body itself, and its expressive capabilities, to “move” engineers into an embodied experience that could inform their work both in design and context creation. Meaning making through movement—engineering through body. Thus, our approach is pragmatic: we want to understand the phenomenon of how people create such vastly varied motion profiles that communicate complex intent. This knowledge is contained inside body-based movement training and somatic practice1 where practitioners hone their own movement capabilities by expanding their array of choices. External methods, typically employed in the sciences and in engineering such as motion capture, photography, force plates, and the like, can work to document the result of a movement pattern but do not have access to choices made by a human in focus, motivation, sensation, memory, prior muscle patterning (and re-patterning), etc. The practice of honing these choices is one of embodiment—a body of knowledge that cannot be known but only moved.

1.1. Systems of Embodied Motion

There are a multitude of body-based practices (Foster 2004), including very well known ones, such as Alexander Technique (Alexander 1990), Feldenkrais Technique (Feldenkrais 1972), Body Mind Centering (Cohen et al. 2012), Pilates Method (Latey 2001) and Contact Improvisation (Pallant 2006) to name just a few, but they all share some commonalities in their approach to experiencing the body from an internal, rather than external, perspective (Hanna 1980). Indeed in philosophy and research in the arts a question can be posed of what kinds of ideas can be expressed through movement (Elgin 2010) and what kind of learning is inherently kinesthetic (Abrahamson 2004; Goldman Schuyler 2010; Lindgren and Johnson-Glenberg 2013). A field of work known as choreology (Hall 1964; Sutil 2015) defines methodology that enables deeper thinking about what defines movement. In robotics, a separation between functional, task-based movement and “expressive motion” profiles (only useful for the purposes of human responses) has been motivated. LBMS frames efficiency as the process of selecting from all that is possible, the clearest choices for the richness of human physical expression to occur. Recognize the word play that happens when one considers “the efficiency of enjoying the task and being in relationship with another” or “the expressiveness of a clear, decisive, precise movement through space”. Thus, all movement is “expressive” and “functional”—and the tools honed through time spent in a dance studio will help to create more expressive robotic systems that have broader, more successful functionality in dynamic environments.

The LBMS system of embodied motion (Bartenieff and Lewis 1980; Hutchinson and Guest 1996; Hackney 1998; Maletic 1987; Laban and Ullmann 1966, 1971; Studd and Cox 2013) is mainly communicated through certification programs through the Laban/Bartenieff Institute of Movement Studies (LIMS) and Integrated Movement Studies (IMS), where successful participants become Certified Movement Analysts (CMAs) and Certified Laban/Bartenieff Movement Analysts (CLMAs), respectively. The emphasis on in-person training is part of the programs’ philosophy: the material cannot be “understood” without bodily, or embodied, participation. A system of movement analysis taught in these programs and initiated by Laban and his student Irmgard Bartenieff, LBMS is utilized in many professional contexts, such as therapy, consulting, and research, in addition to dance and choreography. Unlike other academic disciplines, this field does not publish regularly (Groves et al. 2007) and does not offer doctoral degrees. Nevertheless, this body-based approach to research has much to offer the field of robotics.

The work in LBMS takes into account the larger patterns of human movement identified as Thematic Dualities. One such theme is Function/Expression (F/E), which clarifies that the need for expressive robotic systems is a practical, functional pursuit. It is important to note that finding patterns is in fact the primary experience of our body moving in relationship to our environment. It is a process of differentiation that ultimately allows for synthesis—understanding that while function and expression can be perceived as opposites, they are in fact inseparable. In the LBMS curriculum, these themes are described with a mobius strip topology, indicating that the ideas are one and the same. This is a primary influence on our point of view regarding expressive robotic systems.

Further, LBMS provides a series of interrelated, qualitative lenses, termed Body (answering “What?” about movement), Effort (“How?”), Space (“Where?”), and Shape (“Why?”), which contribute to notation systems, for practitioners to use in finding pattern and meaning in movement. The overlaps in these categories are explicated through Affinities (part of the theory of Space Harmony (Laban and Ullmann 1971)). The idea of Affinities is simply that there are large patterns that can be identified through relationships among Body, Effort, Shape, and Space. For example, certain kinds of expression tend to occur in particular directions in Space. This relationship is actually rooted in the body’s design, (once again foregrounding the idea of body-based movement as a form of knowing) for example Light Weight Effort is linked to the Center of Levity, located in the Upper Body—(up)—and Strong Weight Effort is linked to the Center of Gravity located in the Lower Body—(down). As humans we associate particular movement expression with certain tasks and series of changes in spatial location that allows us to create and contextualize meaningful communication. Note, that this context, most generally, is comprised of the summative experience of a lived life—thus, meaning making in movement is a function of culture, prior experience, as well as immediate situational context.

1.2. Choreography as Body-Based Research

Collaboration with dance practioners brings largely universal movement principles into the coterie of source inspiration and awareness for engineers. Movement knowledge empowers lab students to find context and meaning in seemingly commonplace movements. How does a hand wave meaning “hello” differ from a hand wave of “move out of the way”? When tempo and frequency change, is the movement “the same”? Importantly, movement awareness shows how narrative is drawn from any scale of movement in space. For instance, a Roomba floor robot skids along the floor in a continuous motion. Possible assigned narratives include subservience, a snake-like creep, and a treadmill. How does this type of movement and resulting narrative affect the perception of the robot overall? The Roomba takes up space within an owner’s home, and this new addition necessarily alters their daily story. How will that owner move differently or rearrange their space as a result of the robot? What feelings or moods will it impart in its human counterparts? It is critical engineers are given tools to understand the perceptions and influence that stem from human-facing machines.

The methods utilized by choreographers help create meaningful movement2 and organize the execution of large-scale performances, which may not take on citations in academic journals, but are nonetheless important ways in which society organizes knowledge and makes sense of experience. For example, Rainer’s “Trio A” initially didn’t look like a dance because she used pedestrian-style movements that do not have the typical virtuousic leg extensions, etc. associated with dance (Lambert-Beatty 2008; Rainer 1966); this work was heavily influenced by the context of the larger social movements connected with her and others working at the Judson Memorial Church at the same time (Banes 1983, 2011). Thus, the innovative, groundbreaking movement profiles put forth by Rainer puzzled the dance community and audiences but revealed that pedestrian movement is already expressive and masterful (Elgin 2010). Innovation in any discipline is often associated with disorientation of existing patterns; here it is specifically new movement patterns that help express a greater variety of ideas. Moreover, the philosophy embodied by her work (Rainer 2006) informed minimalism in other art forms like fashion, music, and theater (Copeland 1993; Lambert 1999).

1.3. Improvisational Technology

Improvisation is an important process used to support choreography (movement design) and dance training and utilized in performance that is distinct enough to be called out on its own. It also employs a large role in the somatic approaches taken by the RAD Lab—activities such as movement hour incorporate individual and group improvisation activities (described in Section 2). These activities are tools for improvising in the body, each highlighting different aspects of movement and providing important opportunities to explore and learn about movement. Just as we can improvise with external technologies—such as musical instruments or software programs—we can improvise using the body. Forsythe has worked extensively to explicate these strategies, which he employs in training, choreography, and performance (Forsythe 2012).

One improvisational technique from dance is contact improvisation. As the name suggests, this form of dance is improvised, sometimes to music, and emphasizes contact (with other humans, walls, floors, etc.) This embodied approach to studying movement improvisation can provide valuable experience to roboticists interested in the spontaneous design of movement for bodies which interact with their environment. Further, a mature improvisational practice develops deep understanding of what choices are available and which may be generated in response to an exigency or problem Goehr (2014). As roboticists, improvisational techniques are useful at two levels. First, if a human is designing robot motion (such as a gait, or the path a robot should take through a warehouse), it is useful to be able to improvise instructions to the robot—allowing the designer to quickly iterate on their instructions and get feedback immediately about how their instructions are interpreted on the robot platform. On the other hand, we are moving toward robots that need to “improvise” their own movement—for example, autonomous vehicles that must react to the actions of human drivers around them. One exercise in contact improvisation is to walk in a group around a room, with each person improvising their path, and gradually increasing the speed until everyone is running around the room and must manage to avoid collisions by dodging other people at the last second, or making contact with them in a way that avoids injury. As an exercise for roboticists, this helps us understand and analyze techniques for communicating intent, and using momentum to maintain control and avoid damage—techniques very relevant to problems studied by roboticists.

Another important concept from improvised dance is an embodied understanding of creative flow and an appreciation for the complexity of the human body. As most people have experienced, improvising dance is difficult. It does not feel natural at first. But with practice, it becomes easier to slip into a mental state where there is no self-consciousness and the dance is done as an instinctive reaction to music, previous movement, or the movements of other people around you. Setting an intention to move beyond known or familiar movement patterns supplements this work (Peters 2009). This embodied experience often leads to a deep appreciation of the complexity of the human body and movement: when improvising, we choose naturally from an extremely large set of possible movements. The calculations done by the human nervous system clearly outpace our best digital optimizers and controllers. A practice in improvisation also gives an appreciation for the usefulness of constraints: trying to improvise a completely unconstrained dance is much more difficult than improvising “walking forward as if you are moving through knee-high water,” for example. In robotics constraints are similarly useful: general-purpose robots are much more difficult to engineer than ones which only perform or respond in specific contexts. Indeed, after careful consideration of the body the task of creating a robot which compares in expressivity begins to seem nearly impossible. Analytical models, which acknowledge the nonlinearity of something as complex as improvisation, have been used to try and describe this behavior inside structured improvisations in performance as in Özcimder et al. (2016) or to generate diverse movement through chaotic models (Bradley and Stuart 1998). Our goal in this article is to extract embodied understanding that can guide in the development of expressive tools, rather than debating the possibility of biological processes in machines.

1.4. Tools for Creators

Creativity is a varied process even within the arts (Kaufman et al. 2005). It’s also a topic that computers have helped facilitate (Isaacson 2014; Lubart 2005), even in sensory fields like cooking (Pinel et al. 2015). Similarly, external technologies can augment human movement designers, providing tools such as randomness, or immediate visualization, which can inspire new creative directions as described in LaViers (2018). One well-known adopter of such external technologies to aid in motion design was Merce Cunningham (Copeland 2004). For instance, he would use coin flips and the I Ching to randomly determine the sequence of movement in his choreography, and said of this technique, “the feeling that I have when I compose in this way is that I am in touch with a natural resource far greater than my own personal inventiveness could ever be” (Cunningham 1997). Similarly, Cunningham also adopted computer technology, such as the LifeForms software (Calvert et al. 1993; Schiphorst 1993) which is a graphical tool for choreographers to compose dances using animated representations of human dancers.

Robot programming methods for inexperienced users use several techniques to make the process more approachable, with an emphasis on reconfigurability of programs for industrial applications (Rossano et al. 2013). These include visual (icon and data flow) languages, tangible interfaces (Sefidgar et al. 2017), CAD and VR interfaces (Whitney et al. 2017a), natural language instructions (Tellex et al. 2011), as well as lead-through or “learning by demonstration” (Argall et al. 2009). Research into human-robot collaboration often combines natural language commands with social cues such as direction of gaze and posturing of the body (such as pointing) (Whitney et al. 2017b) and processes from the arts (El-Jiz and Rodrigues 2017). In general, there is a trade-off between the simplicity and ease-of-use of a language or interface and its expressivity—the size of the sets of robot configurations and/or paths that it can specify.

Abstractions, such as instructions about repeating movements and modifying stored movements in time, space, and resulting style, can help increase the expressivity of a language or interface while maintaining a compact, understandable specificiation. Similarly, in choregraphic notation, there is a range of media and languages used for specifying movement—from raw video, to Motif and the more explicit full Labanotation (Guest 1977), to style modifiers such as level and Effort changes. Robot programming similarly requires a range of tools for different use cases, informed by an understanding of the choregraphic (aka movement design) context and task requirements. Somatic methods can help roboticists understand the subtleties of movement differences (for example, between a chop and a point gesture), and to help define design criteria for robotics tools (for example, the desire for a tight development loop to encourage improvisation and fast prototyping of robot motion). To this end, commercial solutions for robotic prototyping have been developed, including by a startup spin-off of the RAD Lab, AE Machines, shown in Figure 1, as well as by LabView, Google Blockly, Scratch, and Lego Mindstorms, Technic, and Boost Blocks.

1.5. Expressive Robotics

Motion for robots that is termed “expressive” has been used to describe movement used in social and human-facing settings. Breazeal’s Kismet (Breazeal 2004), one of the first socially expressive robots, employed imitations of facial features in order to create engaging interactions through a mechanical system. On the other hand, Knight’s candy delivering systems (Knight et al. 2015) focused on engagement between human and robots can be created through contextual interaction (situated in this case around the Halloween holiday). Other work has attempted to recreate autonomous behavior around musicality (de Sousa Junior and Campos 2011; Michalowski et al. 2007; Weinberg et al. 2009; Schoellig et al. 2014) (even in functional tasks like surgery (Siu et al. 2010)). Inside social robotics, the idea of dance is a strong abstraction for connection through movement (Okamoto et al. 2014; Peng et al. 2015). Such dance-inspired motion is often described with affective labels (Breazeal and Scassellati 1999; Knight and Gray 2012; Knight and Simmons 2014) or, similar to Domin in the play that spawned the term “robot”, as motion features orthogonal to function (Knight and Simmons 2015). Extensive work in animation has explored similar veins of work (Bradley and Stuart 1998; Brand and Hertzmann 2000; Gillies 2009; Liu et al. 2005; Torresani et al. 2007). For example, Etemad and Arya (2016) validates motion profiles via lay viewers under labels of “happy”, “sad”, “tired”, “energetic”, “feminine”, and “masculine”. LBMS has also been utilized in several such academic publications in the field of robotics (Knight and Simmons 2014, 2015; Patla 1982; Barakova et al. 2015; Huang and Hudak 2003; Knight and Simmons 2016; Lourens et al. 2010; Masuda and Kato 2010; Masuda et al. 2009; Rett and Dias 2007; Rett et al. 2008). Work in Gielniak et al. (2010) has shown that just adding variation to movement makes it seem more human-like to human viewers. Other work has focused on functional interpretation, termed legibility and predictability, in narrow contexts (Dragan et al. 2013).

Studies in the psychology of adoption of robotic technologies in the household outline the importance of social intelligence and perceived animacy. These studies show that by expressing character traits, a robotic system can imply levels of social intelligence that increases the likelihood of adoption. Hendriks et al. (2011) and Young et al. (2011) tell us that it is natural for humans to try and extract information from robotic actions, subsequently attributing intentionality to robot movement characteristics and decision making. In Forlizzi and DiSalvo (2006), for instance, user study participants describe the movements of a robot vacuum cleaning system, the Roomba as “cute” or “pathetic”, even though such a correlation may not have been intentioned. There are also instances where people name their Roomba robot, thus giving it an added social identity. The achievement of social assimilation albeit by coincidence and not by intention, prompts the user to associate decisions made by the robot to its personality traits, as opposed to the functional algorithm that determines its movements. In Darling et al. (2015) researchers explore the relationship between empathic concern and the effect of stories in the interaction with robotic systems. This is done through a user study, where participants are asked to strike and destroy a robot insect, with a mallet. Some of the miniature robot insects are given a backstory as described to participants, and some are not. Results of the study show that people are less likely to strike the insects with a backstory, thus proposing a relationship between empathy towards robots and the existence of a priming backstory. This indicates that constructing the design, movement, and context surrounding future robots could lead to increased social acceptance. Similarly, choreographers leverage and create elements, e.g., a program note, to manipulate audience experience.

2. Methods: Embodied Practices for Roboticists Developed from Choreography and Somatics

Members of the RAD Lab are exposed to the different aspects of choreography, improvisation, and LBMS as a tool for meaning making through recognizing patterns, reconciling paradox, contexting (meaning-making), and gaining respect for the complexity of human movement. We approach these ideas through both functional and expressive movement experiences that can illuminate the body as basis for our “knowing” of the world in order to inform our engineering work. This section highlights a small subsection of the activities in the lab with choreographic tools and often a somatic perspective, which have become important, foundational methods for our research.

2.1. Bringing Movement to the Foreground

Humans move all day everyday. So, unless one is trained to isolate and inspect it, as dancers and choreographers and somatic practitioners are, it is invisible. This activity brings movement to the foreground, even for engineers and inexperienced movement practitioners; at the same time, participants in a new group setting will learn each others names. The activity is a good way to begin a new project or workshop on embodied methods.

The exercise is a twist on a standard, simple name game. In the classic version of this game, participants stand in a circle (as in Figure 2) and go around the circle, saying their name at the same time as making a personal movement. The first participant (usually a facilitator) will say: “Hi, I’m Amy,” and in time with their name will make a move, which can be a combination of postural and gestural change and may involve weight shift. Typically the stakes of the room and length of the workshop can determine how involved or complex this first movement is. It should both invite participants to move more than they are expecting, but also respect the boundaries implied by the context of the workshop. For example, if participants are in business suits and skirts, a large jumping jack movement will only serve to alienate participants. Typically a large gesture mixed with a more subtle postural change sets an easy tone for future participants to match—they can exaggerate and choose a more athletic movement for their name or they can use a familiar, pedestrian gesture like a wave.

Next, the facilitator instructs the whole room to repeat their name and their movement in unison. This is a magical moment for group building. The facilitator sets the tone by breaking the “movement taboo” that can exist in engineering and now the whole group will follow suit. It can be helpful for the facilitator to have people in the group that they know will be comfortable with the action and join in—this will help along participants who may be resistant to such an odd request for their context. The exercise now shifts to the next participant who will repeat the actions of the facilitator. This time the group will repeat the facilitator and the first participant’s name in sequence, forcing memory and building the shared movement experience of the group. This repeats until everyone in the room has had a turn, and a seemingly impossible feat has been accomplished: an entire room of engineers is dancing together.

If time allows, the twist on this exercise illuminates this feat further. Now the facilitator allows each participant to go around the room in turn and correct their fellow participants in the execution of their personal movement. To keep plausible deniability, the facilitator may not want to demonstrate, but simply choose an eager participant to go first, offering encouragement or asking questions about how the movement should be done to spur corrections. Now the participants are engaging in iterative choreography. Often, they’ll need to work hard to articulate how a movement should be done, particularly as questions arise. This offers the facilitator a chance to foreground taxonomies (like LBMS) which may help in this process, e.g., by using their own expertise to highlight and articulate more nuanced differences in execution. Eventually, this provides a point of comparison that brings the focus back to an engineering perspective: how can everyone be doing the same thing if we all have different shaped bodies? Indeed, it is a high-level notion of movement that is needed to make such a claim, and the tools inside LBMS provide ways to improve and highlight successful replication.

2.2. Weight, Flow, Breath, and Group Sensing

Standing in a circle, the facilitator begins by activating weight in relationship to gravity by bouncing, jiggling, and vibrating both core and limbs standing with two feet on the ground. As the movement range increases the facilitator invites the participants to activate a conscious awareness of their breath, activating flow through sounding and allowing a connection with the others in the room. Focusing on breath allows an Inner/Outer (I/O) relationship to be established in which participants are feeling their body’s inner space in relationship to the general space. The intensity, speed, and space of the movements continue to increase by adding weight shifting from the core not only up and down, but also forward and backward and sideways. This establishes the ability to locomote and move in a shared rhythm. In doing this the group is literally “warming up”—core body temperature is raised, and mobilization and oxygenation of muscles occur. Breath support is activated and helps to establish a shared space and connection between participants. This attuning to inner sensation can be effective preparation for a shared, embodied research session.

A second exercise allows participants to go into a deeper relationship with one’s own body and breath. Beginning with a partner seated on the floor back to back, participants are instructed to close their eyes and bring their attention to their breath as they sit in relationship to another person. After several breaths and a prompt to observe their natural breathing rhythms participants are asked to bring their attention to Lengthening and Shortening in their torso, investing in the Vertical dimension with their breath. After a few breaths here, they are encouraged to place their hands on their abdomens and with each breath focus on Bulging and Hollowing in the Sagittal dimension. Finally, with hands on either side of their torso, to send breath into their side space as they invest in Widening and Narrowing in the Horizontal dimension. By attending to breath in relationship to another person and with the addition of self touch for kinesthetic and proprioceptive awareness, participants begin to sense more deeply into the shaping of their torsos and the role of breath.

Another short exercise can further develop the sense of the group as a whole, bringing a sense of play as well. One participant, a volunteer, exchanges eye contact with the participant to their left. Using this eye contact as a nonverbal line of communication, the pair attempts to clap at the same time. This is trickier than it sounds because there is a tendency to lose eye contact, which can feel culturally awkward, initially, and undervaluing the complexity of simultaneous action, participants tend not to give the task their full attention. However, after passing this around the circle a few times, most participants receive the satisfaction of simultaneous action, emphasized with the sound of clapping, and the energy in the room will invariably change. The participants are ready for more. Next, to increase the complexity of the task, participants are instructed to add the possibility of locking focus with others across the circle and increasing speed of the exercise. This warmup builds shared focus, group sensing, prolonged awareness during the exercise (you never know when the focus will be directed to you). It also demonstrates how synchrony in dance is temporal, spatial, and qualitative. Dancers who synchronize well attend to many factors, can observe them, and have a range of choices in their body movement repertoire to adapt, adjust, and, thus, synchronize. This training is additive and not reductive. The abstraction of synchrony (it’s an abstraction, not an absolute truth as in same physical behavior) needs additive choice in order to exist on the multiplicity of morphologies of human bodies.

2.3. Kinesphere Exploration Through Spatial Pulls

Laban defined the Kinesphere as “…the sphere around the body whose periphery can be reached by easily extended limbs without stepping away from that place which is the point of support when standing on one foot, which we shall call the stance/place.” (Von Laban 1966). This idea includes more than a roboticist’s notion of workspace, but it is similar. There are many ways to explore the dynamic space around which humans move. Primarily, in the study of the Kinesphere, LBMS maps an approach to Kinesphere that includes Zones, Levels, Reach Space, Pathways, Forms, and Directions. Patterns of Body Organization which are underlying movement pattern identification based on human motor development that support relationship of part to whole provide another way to internally organize movement within the Kinesphere. “Our form, with its upright vertical stance, and bilateral symmetry, is organized through relationships among upper and lower body, right and left sides of the body, front and back, core and periphery” (Studd and Cox 2013).

One exercise in particular highlights spatial organization keyed around a room versus a body reference frame—exploration through Spatial Pulls. To explore the Kinesphere through Spatial Pulls, participants, moving around in a dance studio, are instructed to move while attending to specific directions within the room, e.g., “Place High”, “Place Middle”, and “Place Low”. Initially, these directions are the same for every participant, keyed off the room, and using the hand, or another body part available to all participants, comfortably extended, as the body part drawn to each direction. Gradually, the notion of a space centered around each mover individually, allows for instructions keyed from each participant’s body, e.g., “Forward Right”, which will result in more varied movement for participants with different facings. Another changing condition can be the extent to which the hand moves into the space: a shorter extension is in Near-Reach Space, while, conversely, full extension of the hand is in Far-Reach Space, which can also invite locomotion. Finally, the body part which initiates the movement can change: elbow, hip, top of the head, nose, etc. Inviting participants to debrief the exercise, e.g., answering “When do you do these actions in your daily life? or “How did different spatial prompts make you feel?”, reveals commonalities and differences in experience that highlight the complexity and malleability of meaning-making through movement.

These changing conditions highlight several lessons: (1) the body is an adaptable and accommodating phenomenon to changing requests, (2) what delineates a particular body part is not always clear, and (3) space has meaning. This framework abstracts away the physical properties of the moving parts, focusing on high-level movement ideas common to all movers. “Place High” is a different place for every mover based on different heights, shoulder flexibility, and ability to rise onto the ball of the foot. Moreover, each mover has different prior experiences—maybe positive or negative or something complexly in between—with this kind of movement, which may create strong or subtle emotional responses to the components of the activity. That is why, for understanding this framework, one cannot just rely on going through the related literature that covers these details. Instead, one has to get inside a movement studio, experience the different spatial pulls, move through different spatial levels, and imagine the different spatial instructions while moving around.

2.4. Written and Embodied Movement Observation

A short exercise in movement observation might simply involve bringing up a YouTube video of some movement and having participants write descriptively about the video, communicating it to someone who has never seen it. A sentence might read as follows (this one was written about Mikhail Baryshnikov in Twyla Tharp’s “Push Comes to Shove”): The dancer enters a foggy spotlight emerging from shadows upstage. He deftly flips a tan hat into the air and pops it on his head with a nonchalant air. Holding his hands limp, softly locked in a spatial location while creating a myriad of choices with his core underneath. This description requires participants to watch movement in great detail and make sense of the pattern. To take the exercise a level deeper, participants can write about distinct movement examples and switch samples to see if a naive reader can determine what the original movement sequence entailed.

To continue to work on participants listening and movement observation skills—from an embodied perspective—participants are placed in pairs and prompted to engage in what is often referred to as a mirroring exercise. With partners facing each other, one is directed to improvise various movements which can include pedestrian movements, weight shifts, and arm and leg movements, while their partner is working real time to mirror/imitate the exact movements that they are observing. This tool is used to hone movement observation skills and the ability to make movement choices in the moment. Replicating exactly is not possible—because both participants have different bodies (different ranges of motion different lengths of bones)—and this brings up a chance to describe strategies and choice in motion creation.

Labanotation is a well-known toolset that utilizes Laban’s work to record movement sequences—mostly for archival purposes Guest (1977). A more common shorthand notation system is called Motif, which is useful in practice for applying (and learning) the tools in LBMS. While Labanotation is a precise record of a specific movement phrase, often recorded for posterity, Motif is a description of a gross movement idea or pattern. Each form utilizes its symbols in slightly different ways. Utilizing these notation schemes can clarify observations and record executions of movement. Given the inability to perfectly recreate motion between two distinct platforms, as in the mirroring exercise described previously, our group favors use of Motif as a flexible shorthand that allows certain movement features to be foregrounded over others. The tool of Motif helps designers consider which features or broad patterns of a movement are most important for a given context; the mechanics of the tool are described through the diagrams in Figure 3.

Another useful investigation revolves around the LBMS component of Effort (Laban and Lawrence 1947); Studd and Cox (2013) and reading/interpreting Effort Motifs. Effort can be simply described as the “quality” of a movement—the mover’s inner attitude, motivation, and intention as revealed in observable movement. Activities designed to allow participants to explore the choices in this space can evolve as follows:

• Engage in free movement improvisation in response to the stimuli of varying musical selections
• Engage in free movement improvisation in response to words (adjectives) that evoke qualitative movement responses
• Playing children’s games, like Red Rover, Ring-Around-The-Rosey, and Duck-Duck-Goose, which naturally elicit playful, dynamic motion profiles (which easily produce short moments of complex Effort constellations, such as States and Drives, providing fodder for discussion)
• Break into groups that are each given an Effort Motif, e.g., three symbols describing distinct motion quality, and have the group create a phrase of movement that uses them, explicating a context (note that it’s unlikely, bordering on impossible, that two groups will come up with the same motion sequence)
• Show movement sequences that are generated from the same Motif side-by-side
• Discuss as a group: What is essential about the movements and their commonalities? How is this revealed through the quality descriptors in the Motif? What makes the sequences different and “mean” different things?

2.5. Establishing the Malleability of Meaning in Movement

This scale of “meaning” in movement—whether narrative (“Romeo loves Juliet”), emotional (“that part seemed sad”), or aesthetic (“that reminds me of a flower blooming”)—is utilized to the choreographer’s discretion and this utility impacts the audience’s subjective experience and interpretation. To illustrate this, an exercise described here allows students to create mini-narratives and manipulate contexts from the pedestrian (“a shopping trip to the mall”) to the surreal (“swimming through Saturn’s rings”) with a single gesture.

All participants sit against one wall of a large, open studio. One person is selected to begin the exercise. The person is then directed to “pick a scene, place, idea, event, dream, etc. It can be an atom floating across the universe, it can be a dragon lying in an abandoned mythical forest, or it can be a monk sitting atop a hill. There are no limitations to the who, what, where, when, why. Decide what or who you will be within this scene. Are you the heat coming off a loaf of fresh bread? Are you Cleopatra on her throne? Now when you stand up and enter the large open area of the space, you will be this. No speaking or verbal clues. You will move like it and commit to it. Go.” Now this first participant, the “scene leader” goes into the space without revealing their internal image or desired role; they simply begin moving in the space. After the scene leader begins, the facilitator selects students at random and instructs them to, “Decide what you think this is. What scene/what action/what is happening? What space is being created? And then select how you will insert yourself into this.” This participant now enters the space, similarly without revealing what they think is going on or talking in anyway; they simply begin moving, in earnest, their role. This continues for all of the participants until a dynamic, moving scene has been created.

In one example, the scene leader laid on their stomach with arms outstretched, holding an imaginary item, and keeping one eye closed. They rolled back and forth on their stomach with constrained breathing and bound overall flow. It was revealed later that they were a World War I soldier fighting in the trenches. Without this knowledge, other members of the class cohort decided for themselves what was happening and joined the scene. The first “joiner” began moving in a curved path, on their hands and feet with hips in the air, similar to a children’s camp “bear walk”. They would occasionally pause and switch their focus up from side to side. This was a labored movement pattern, with dramatic weight shifting and a strong energy. It was revealed later that the first joiner believed the scene to be a safari and inserted themselves as a large hunter-target.

In each case, the participants are told, “once you join, continue what you are doing until the exercise ends”. At the end of the exercise, each participant had joined the scene and committed to their interpretation. After a few minutes, the facilitator ends the exercise and participants stop moving, and then each individual revealed what they believed the scene to be. The interpretations of one single scene can be varied: from “gas along Saturn’s rings” to “children on Easter” or rather unified: “journalist reporting on war”, “soldier patrolling an area”, and “medic caring for the wounded”. Participants altered between joining early in the scene and later in the scene. Additionally, as a further variation, a subset of participants can exit a scene once it had been “finalized”, and then asked to reinterpret the scene from their original image. This creates a cycle of constantly deciding and participating based on the new visual representation, examples are shown in Figure 4.

The lessons learned can be numerous. For example, one person is a “hunter” when another is the “buffalo”. How does this change when the hunter disappears? Relationships created through movement and space are altered through perspectives, additions, and subtractions. Confirmation bias exists in performance as audience members process what may be purposefully ambiguous or unstructured. One scene leader might choose a very specific theatrical action, like the scene leader on their stomach, while the joiner may use an entirely improvisational, constant, and freeform movement pattern. This exercise illustrates the malleability of meaning in movement. In real time, participants can observe how movement patterns reorganize into new narratives, for individual performers and for the performance area as a whole.

3. Results: Progress and Principles in Developing Expressive Robotic Systems

The methods described above have been utilized in a myriad of robotic projects, in the RAD Lab and outside it. Moreover, in successful funding proposals, these methods have been explicitly touted for their potential for dramatic leaps forward inside robotic control and human-robot interaction. Through this work, a few guiding themes have emerged, which are discussed here.

3.1. Irreplaceable Body-Based Research in Robotics

The understanding gained from the exercises described in the previous section cannot be learned from a textbook. This work is of the body, not just the mind, and is necessarily passed on through embodied practice. What role does breath play in movement? As most motion capture systems ignore this aspect of motion, one way to answer is through the exercises described in Section 2.2. From this work, researchers have worked to include the notion of breath in virtual and robotic agents, including work in Pakrasi and LaViers (2018). A bodily understanding of the role of improvisation (Section 1.3) and spatial commands (Section 2.3) in movement design has led to new languages for robotic motion specification (Jang Sher et. al. 2018; Nilles et. al. 2018). Further, as seen in Huzaifa et al. (2016); Huzaifa and LaViers (2016), body-based exploration can lead to novel robot designs. Just as biomedical researchers collaborate with medical practitioners and visit hospitals to understand the activities and stakeholders within, roboticists building expressive systems for human interaction can gain much spending time in a dance studio working with and learning from choreographers and dance professionals. In our case, artistic output of the lab has stretched performance of robotic systems and helped us better understand how changing the context around a given physical machine can change perception of it (Cuan et al. 2018).

3.2. The Case for Manual Movement Design

Spending time in a dance studio with a choreographer illuminates how malleable motion interpretation can be (as in Section 2.5). For example, contrary to work where specific models are given for movement that is “happy” and “sad”, our follow on work (Heimerdinger and LaViers 2017) has shown that these models are limited. Thus, tools that allow human movement designers choice in developing motion profiles, as in Huang and Hudak (2003), are critical to this intersection of work. Yet, robotics venues have balked at creative tools and methodologies as being too “ad-hoc” with reviewers offering constructive feedback like “The key postures are also manually provided for different styles—this does not sound very scientific. I think it is necessary to automate such process to claim a contribution on stylizing the movements” and “if a different designer had done the transformation, very different characters might have emerged. The design process is highly subjective and does not generalise”. However, when viewed as a creative tool, these are strengths of such “ad-hoc” methods. Again, the many myriad of ways human motion may be translated to a set of fewer or distinct degree of freedom system guarantees choice in the process. Premature automation and restrictive optimization may end up limiting the exhibited behavior of robotic systems. On the other hand, think of the many myriad ways motion as simple as a Roomba has been interpreted in the inherently varied contexts it inhabits in human homes. Offering this choice to a human is not only the way to empower collaborators in dance but also a reasonable step toward making useful tools.

3.3. Objective, Qualitative Movement Observation (to Support Subjective Conclusions)

In pursuit of automation that is sensitive to movement expression, then, we motivate the need for a pipeline of activity which starts with observation of movement. Only through noticing the objective mechanisms at play with qualitative methods (e.g., written description) can we begin to quantify and automate aspects of expression through movement. The diagram in Figure 5 highlights this take away, motivating the need for two areas of future work: qualitative methods for the description of objective movement phenomena and quantitative methods for the analysis of subjective phenomena. The former is a way of characterizing all of the methods presented in Section 2, while the latter describes the results presented in Section 3. For example, every conversation spurred on by the embodied practices employed here is a chance for engineers and dancers to describe the experience of moving in their own bodies and viewing movement in others. On the other hand, every output of expression through movement that interfaces with some technological element or another requires quantification.

These lenses also reveal how important objective description is to seeing mechanisms in movement that can be translated to automated systems. It is tempting for engineers to watch an animate character that may serve inspiration for a robot companion and say: “I’m going to replicate this happy movement in my robot design”. As seen in Pakrasi and LaViers (2018), this lacks objective explication of movement phenomena that can be translated to an engineered system. Instead engineers need to break down the behaviors that are contributing to that character’s “happy” affect within its given context (Heimerdinger and LaViers 2017, 2018). In this light, an engineer can then see: “Ah, the frequency of motion in the vertical dimension is modulated when the character’s internal state varies, and an increased frequency of the period of the motion correlates with a happier seeming character”. This can be translated into quantitative design elements: period of oscillations. Moreover, it motivates an important separation between “style” (how a platform is moving) and “affect” (how a human responds to motion). Expressive robotic systems will have wide capacity for a range of motion styles that, when aspects of context and characteristics of a human counterpart are accounted for, can communicate a desired affect to a human counterpart.

4. Conclusions: Toward Widespread Collaboration between Dancers and Roboticists

...the truest creativity of the digital age came from those who connected the arts and sciences.

—The Innovators (Isaacson 2014)

There is inherent excitement around collaborations between engineering and the arts—as evidenced by the quote above. The sum of the embodied, relational work presented here implies that, no, a machine cannot express all the things a human can express, but that humans themselves can express more things alongside and in relation to machines. To this end, this paper has provided background on tools and techniques from choreography and somatics and how those have been used in the development of more expressive robotic systems. We’ve posited that a missing piece from much of the prior work in robotics is embodied practice alongside artistic practitioners. To that end, we’ve presented methods developed in the RAD Lab with our many collaborators to provide embodied experiences for an interdisciplinary group working in robotics. To show that these approaches can be effective and spur innovation, we’ve shared connections to projects that have benefited within our group. Finally, framing principles that have resulted from this work have been shared.

Robotics is at a critical point where researchers are working to bring complex mechanical machines outside of controlled factory environments, where they are walled-away from human counterparts. This demands new understanding of how humans view and create movement that experts like dance artists, choreographers, and somatic practitioners already understand. Broadly speaking, the methods highlighted in this paper work toward the production of expressive robotic systems: systems that have a greater variety of choice in the generated motion profiles. This makes it easier for humans to design movement. It makes it easier for humans to interpret movement. Is such a system more life-like? Yes, because it’s more expressive. Is it dancing? No, that’s the humans operating it, designing it, selecting moving patterns over time, and extending themselves through technology. Thus, as a more refined paintbrush or a camera with better dynamic range, with improved, more expressive robotic technology, the machine extends the artist into the 21st century.