ClothSurface: Exploring a Low-Cost Prototyping Tool to Support Ideation for Shape Displays

Abstract

A shape-changing user interface is a type of interface that interacts with users by changing its physical form. Although researchers have been extensively studying shape-changing user interfaces, relevant research on its various design aspects—including tools and methods—remains limited. Prototyping shape-changing interfaces often requires sophisticated equipment and knowledge, which makes this sphere of design unwelcoming for designers with limited resources and technical knowledge (e.g., design students). In this study, we propose ClothSurface—a simple and low-cost prototyping tool to design for shape displays—and explore its use through a series of design sessions. The results reveal that ClothSurface can allow inexperienced designers to illustrate their ideas and to explore the design space of shape displays.

1. Introduction

A shape-changing user interface is a relatively new sphere of design that is yet to be fully explored. Consequently, there exist only limited methods and tools for designing shape-changing interfaces. This study proposes a tool for prototyping shape-changing interfaces, with a focus on shape displays [1]. Shape displays refer to a type of shape-changing interface that primarily has a flat surface.

In interaction design, brainstorming is typically done through sketches [2,3]. Then, low-fidelity prototypes, like paper prototypes, are created to facilitate further discussion and testing [4]. In the field of human-computer interaction (HCI), researchers working with shape-changing interfaces have created numerous prototypes to illustrate and test technological advancements and relevant designs [5,6,7]. However, these prototypes are sophisticated. Building and altering them requires in-depth technical knowledge and/or time and effort.

We propose ClothSurface, a primitive and low-cost prototyping tool for low-fidelity prototyping of shape-changing interfaces. It requires simple physical resources and a rather limited skill level to operate. It is accessible to designers with limited resources and technical know-how, such as design students. ClothSurface enables an instant change in design, thereby supporting efficient design exploration and critique. In addition, it filters the interactive and dynamic aspects of shape-changing interface design in a manifestation that only requires a small amount of resources. Its easy-to-change nature enables the exploration of the design space.

1.1. Challenges of Designing for a Shape-Changing Interface

A number of studies have explored the design sphere of shape-changing interfaces [8,9]. This has led to the emergence of different design perspectives, such as the tight coupling of a physical shape and digital information [6] and jamming [5,7]. There have also been specific technological advancements in this sphere of design, such as weight-and-volume-changing devices [5,10] and programmable jamming systems [5]. Studies have also investigated the user experience of shape-changing interfaces, such as determining the effectiveness of shape-changing mobile devices [11,12], understanding the experience of a shape-changing bench [13], and identifying user perceptions of shape-changing handheld devices [14]. Roudaut et al. [15] explored different alternatives of technical implementation for the same design concept; their exploration mainly focuses on technical aspects.

Through a workshop with 25 experts in the field of shape-changing interfaces, Alexander et al. [16] derived 12 grand challenges for research on shape-changing interfaces. Design is one of the major areas of challenges. The first challenge is design for temporality [16]. Traditional design tools, such as sketches and prototypes of static forms, offer tangible manifestation of ideas that multiple designers and/or other stakeholders can compare and comment on. However, sketches and static prototypes cannot conveniently present the dynamic and interactive properties of shape-changing interfaces. This limitation makes design critiquing [17] difficult. Another challenge is to design the form and dynamics of shape-changing interfaces to fulfil the needs and desires of people [18]. The tools and methods that support designers to design for action, perception, and reaction are necessary [19].

There is a growing number of design tools and methods supporting the exploration of shape-changing interfaces. Hummels et al. [20] discuss and reflect on several movement-based interaction design techniques. They suggest that more tools are needed to support designers to explore designers’ movement with their own bodies. Designing with designers’ bodies and movement has been an emerging topic in the research on interaction design. For example, Wilde et al. [21] presented a framework on analyzing body-based (or “embodied”) ideation methods, which are based on the notion of estranging the relationship between designers’ bodies and physical materials. Torres et al. [22] proposes the notion of “Conversation with Materials”, which emphasizes on designers’ reflections in bodily interactions with materials. They suggest principles of facilitating Conversation with Materials. Based on the principles, they developed a system that allows designers to bodily interact with various actuators controlled by different signals. In a user study conducted with 18 designers, they demonstrated that designers could reflect on the interactions with the actuators, elicit memory, and engage in conceptual design thinking.

1.2. Physical Prototyping with Readily Available Objects

Since the popularity of personal computers, there has been hype surrounding the adoption of digital tools for design. An overemphasis on digital design tools prevents designers from valuing the merits of the bodily experience of physical manipulation and improvisation. Sass [23] indicated that using available objects to create, illustrate, and prototype enables designers to experience and experiment design ideas without technical restrictions. Further, researchers have emphasized the importance of handmade tools (e.g., [24,25]). An example of a handmade tool is a prototype. In the context of architecture studios, KhakZand and Babaei [26] found that students were able to choose readily available objects to innovate and express their ideas. Further, within the context of product design, Bramston [27] indicated that creating a junk model enables designers to understand the actual nature of problems.

Buchenau and Suri [28] indicated that prototypes aiming to simulate the experience of a design enables designers to bodily experience a design and build empathy toward existing users, explore and evaluate design ideas, and communicate these ideas. From this perspective, prototypes do not need to be visually appealing. Quick-and-dirty prototyping can be applied in the context of interaction design [29]. This method costs little in terms of time and money. The physical manipulation of materials enables idea exploration for inexperienced designers, particularly in the case of design tasks that involve physical artefacts [30].

2. Prototypes as Filters and Manifestations

Lim et al. [31] present a framework for analyzing prototypes. They conceptualize prototypes as tools for exploring design space. Their focus is on a prototype’s role in supporting designers’ generative and evaluative discovery. They presented two notions with regard to prototypes. The first notion is that prototypes serve as filters in a design process. The idea is that prototypes filter those aspects of design ideas that are relevant to the current stage of the design process. The filtered aspects should be relevant and sufficiently interesting for the designers to explore the design space and alternatives. In their framework, they identified five filtering dimensions: appearance, data, functionality, interactivity, and spatial structure. These are five areas of design ideas that may be filtered in or out in various types of prototypes. One example that they illustrated in their paper is a two-dimensional (2D) floor plan in the early stage of a house design process. A 2D floor plan filters the spatial aspects of rooms and enables designers to focus on the spatial arrangement of spaces. Aspects such as the appearance of the house can be left to be addressed for a later stage, when a different prototype (e.g., three-dimensional (3D) model of a house) is created to filter the aspects relevant to that stage.

Lim et al.’s second notion is that prototypes serve as manifestations of design ideas. Creating prototypes enable designers to externalize their ideas. Designers iteratively interact with the tangible prototypes. Thus, the manifestation becomes a collaborator and discussant that “talks back” to them [32]. This notion resembles Torres et al.’s idea of “Conversation with Materials” [22]. Prototypes also enable multiple designers to understand each other’s ideas in design collaboration. In design collaboration, the externalization of ideas is essential in creating a common understanding of the ideas under the discussion among peer designers. In design literature, it has been suggested that managing shared mental models (e.g., [33]) and intersubjectivity (e.g., [34]) is essential in collaborative design. In Lim et al.’s framework, there are three manifestation dimensions: material, resolution, and scope. Material is the medium used to create a prototype. Resolution is the level of details or sophistication of a prototype; this dimension is often referred to as “fidelity” in the literature. Scope refers to the extent of the manifestation of an idea. One example that can illustrate the concept of scope dimension is manifesting a search engine. A prototype that only manifests a button and a search box has a smaller scope than a prototype that manifests the entire layout of a webpage, with page title, background, a search box, and a button.

Together with the abovementioned two notions, Lim et al., posit that “the most efficient prototype is the most incomplete one that still filters the qualities the designer wants to examine and explore” ([31], p.8). Their framework of a prototype is adopted in the conceptualization of prototypes in the present study. In particular, the present study focuses on rough and physical prototypes that a designer can quickly create to manifest and revise design ideas, while ensuring that the relevant aspects of ideas are not filtered out.

The current study focuses on shape displays. The following subsections review the landscape of the existing prototyping tools for shape displays in the framework of prototypes presented in [31]. This is not meant to be an exhaustive list. The key is to identify the limitations in the current landscape and, thus, to present the motivation of proposing ClothSurface. Instead of focusing on the technicalities of prototyping a shape-changing interface (such as the overview in [35]), the review focuses on how the prototypes serve as manifestations that filter relevant aspects for designers to explore the design space of shape-changing interfaces [31].

Table A1. Overview of Prototype Tools or Methods.

Prototype Tools or Methods	Filtering Dimension	Manifestation Dimensions
Sketching [36]	Filtering dimensions addressed: Appearance: 2D illustration of shape, limited illustration of shape change Data: Data type Functionality: Functional specification Interactivity: Users’ input and shape-change as output Spatial Structure: 2D manifestation of arrangement of shapes Filtering dimensions not addressed: Nil	Materials: Pencil and paper; change requires erasing and re-sketching Resolution: Flat manifestation of physical shapes and spatial structure; static manifestation of shape-change (e.g., in multiple drawings); rough manifestation of interactivity and functionality—basic specification of inputs, outputs, and functions Scope: Static representation of shapes or states of shape change.
Design Fiction [39]	Filtering dimensions addressed: Data: data type Functionality: Functions in context Interactivity: Users’ relationship with shape-changing artifacts in context Filtering dimensions not addressed: Appearance; Spatial Structure	Materials: Pencil and paper or word processor on a computer; changes require modifying the text/images in the fictional materials Resolution: Limited manifestation of physical shapes and shape-change (only in simple graphical illustrations); description of data, functionality, and interactivity in use scenarios/contexts Scope: Mainly focused on background and context of use
ClothSurface (the current paper)	Filtering dimensions addressed: Appearance: Shape, physical, and dynamic illustration of shape change Data: Data type Functionality: Hand gestures as inputs; dynamic shape-change as outputs Interactivity: Users’ interactions; adaptive affordances Spatial Structure: Spatial arrangement of shapes within the box area Filtering dimensions not addressed: Nil	Materials: ClothSurface: top-opened box, spandex/elastic cloth, and everyday objects (e.g., balls, cubes, water bottles); changes require moving hands and/or objects underneath the spandex Resolution: Manifestation of shape and shape change is dynamic and tangible; rough interactivity with manual operation; tangible manifestation of functionality and data type; overall manifestation of spatial structure Scope: limited to the size of the top-opened box
Prototype with Mechanical Pin-Actuation Displays e.g., [1,40,42]	Filtering dimensions addressed: Appearance: Shape, physical, and dynamic illustration of shape change; colors (with projection e.g., [1] or AR-displays [42]) Data: Data type; data size Functionality: Hand gestures as inputs; dynamic shape-change as outputs Interactivity: User interactions; adaptive affordances Spatial Structure: Spatial arrangement of shapes Filtering dimensions not addressed: Nil	Materials: Mechanical pin-actuators, hardware and software system(s) controlling the actuators; change requires significant amount of effort in programming and hardware re-configuration Resolution: High level of details in the shape change and accurate manifestation of interactivity, functionality, and spatial structure; appearance may be rough, depending on the number of pins and the color projections or AR-displays Scope: Typically a fully functional standalone prototype
Prototype with Elastic Deformable Displays (e.g., [44,45,46])	Filtering dimensions addressed: Appearance: Shape, physical and dynamic illustration of shape change; colors (with projections) Data: Data type; data size Functionality: Hand gestures as inputs; shape-change as outputs; color projections as outputs Interactivity: User interactions; vivid color projections/displays in respond to users’ actions Spatial Structure: Spatial arrangement of physical and visual elements Filtering dimensions not addressed: Nil	Materials: Elastic displays or projection screens, hardware and software system(s) sensing deformation of screens and controlling the color projections; change requires significant amount of effort in programming and hardware re-configuration Resolution: High level of details in using deformation as inputs, shape change or projections as outputs and accurate manifestation of interactivity, functionality, appearance, and spatial structure Scope: Typically a fully functional standalone prototype
Prototyping Toolkits with Mechanical Pin-Actuation (e.g., [48,49])	Filtering dimensions addressed: Appearance: Shape, physical and dynamic illustration of shape change; potentially colors (e.g., [49]) Data: Data type; data size Functionality: Hand gestures as inputs; shape-change as outputs Interactivity: User interactions; shape change in response to users’ actions Spatial Structure: Spatial arrangement of physical and visual elements Filtering dimensions not addressed: Nil	Materials: Modular pin-actuators; change requires low technical skills and some time in re-writing scripts Resolution: High level of details in using deformation as inputs, shape change or projections as outputs and accurate manifestation of interactivity, functionality, appearance, and spatial structure Scope: Typically a fully functional standalone prototype

The format of the table is inspired by that of the analysis in [31]. Since one of the motivations of the present study is to propose a quick-and-dirty prototype tool for shape-changing interface, additional comments on the ease of change are made in the manifestation dimension of material.

in the Appendix A presents an overview of the review.

2.1. Sketching

Designers can sketch shape-changing ideas using a pen and paper. Rasmussen et al., [36] investigated the design space of shape-changing interfaces by analyzing sketches made by 21 experts. They invited researchers working in the sphere of shape-changing interfaces to brainstorm and sketch ideas for either a shape-changing radio or a shape-changing mobile phone. The motivation was to examine the design space of shape-changing interfaces by analyzing the concepts of the experts through their sketches. Their analysis was conducted from three perspectives: vocabulary [8], use of metaphors [37], and affordances [38]. Sketching can serve the two purposes of a prototype as a filter and a manifestation; moreover, sketching can address all the filtering dimensions with graphical illustrations and textual annotations. Pen and paper are the only materials needed to create sketches, thereby making sketching easier to change. However, the manifestation of shape-changing ideas with sketching lies in the low level of details, because the aspects of 3D appearance and tangibility cannot be easily addressed.

2.2. Design Fiction

An alternative of prototyping shape-changing interface with the benefits of ease of change is design fiction. Sturdee et al., [39] used the medium of a user manual to envision what a game with a shape-changing interface could be like. They concluded that design fiction can be used in combination with a prototype of a shape-changing interface in the future. The design fiction of a shape-changing interface manifests a broad scope with regard to the context of use. In Sturdee et al.’s case, it highlights the background and entertaining implications of games with a shape-changing interface, and enables designers to focus on these aspects when exploring the design space. However, the physicality is absent in such a prototype. The materials used are text and images, which can be paper-based (e.g., pencil and paper) or computer-based (e.g., a word processing software). It should be easy to change and be made compatible with rapid iterative change and design critique. In terms of other filtering dimensions, the spatial structure and appearance of a shape-changing interface are almost completely filtered out (except in certain images). Such a prototype is not suitable if designers wish to explore different forms of shape change tangibly and dynamically.

2.3. Prototype with Mechanical Pin-Actuation Displays

In the literature, there is a group of prototypes that serve as manifestations of design concepts of shape displays. These prototypes filter performance and interactivity of shape-display design concepts with a high level of details. inFORM [1] is a prototype of a shape display that can guide users with dynamic physical constraints and dynamic affordances. As a manifestation of such design ideas, inFORM is a prototype with a high level of details in representing performance and interactivity. The upward and downward actuation speeds of the pins are accurately represented. The interactive and haptic aspects of a shape display in dynamically providing affordance, user-guiding constraints, and object manipulation in response to users’ behavior are represented vividly. Further, built on inFORM, TRANSFORM [40] is a prototype of the concept of shape-changing furniture. It filters the dynamic affordance in the concept of shape-changing furniture for the visitors to the Milano Design Week. Haptic Edge Display [41] is another example of such a prototype, but in the context of mobile devices.

In addition to the filtering dimensions of performance and interactivity, certain prototypes based on the pin-actuation mechanism represent appearance as well. Sublimate [42] is a prototype of a design idea of an augmented reality (AR) display combined with a shape display. As a prototype, it filters the aspects of appearance, interactivity, and functionality. Along the filtering dimension of appearance, color transparency in the AR display manifests the idea of what it looks like if an illustration can hover over a shape display. Relief [43] is an actuated shape display covered with elastic fabric (Lycra) with a color projection from the top. Relief is a prototype that filters appearance for designers to focus on in addition to the filtering dimensions of interactivity and performance.

The materials used in these prototypes comprise mechanical actuations. Changing them often requires a certain re-configuration of hardware and/or re-programming of software for which a certain level of technical skill is required.

2.4. Prototype with Elastic Deformable Displays

A group of prototypes built with elastic deformable displays can filter the dimensions of appearance, functionality, and interactivity. Harrison and Hudson [44] present prototypes that manifested the idea of dynamically changeable physical buttons on a display. Müller et al., [45] and Watanabe et al., [46] present prototypes that use an elastic surface and a projector to manifest the idea of a shape-changing display with high level of details in the aspects of appearance. Further, TableHop [47] is a prototype that manifests the idea of a user interface that can change shape, color, and vibrates. It is built on spandex fabric and transparent electrodes. Such prototypes are solid in technical implementation and manifest different ideas of shape displays with a high level of detail in filtering functionality, interactivity, and appearance. They are created with a strong focus on the illustration of specific concepts and to show technical feasibility; however, they are not designed for ease of change and rapid iteration.

2.5. Prototyping Toolkits with Technical Elements

There are prototyping toolkits that consist of technical implementation and are designed for change. The design of these toolkits aims to reduce the technical skills required to build prototypes for shape-changing interfaces. In this context, Hardy et al., [48] proposed ShapeClip. The toolkit is a set of pin-actuators that increase their height in response to the color they detect. The users of the toolkit need to control the color displayed on a regular color display. Another example is Everitt and Alexander’s PolySurface [49]. The authors present a step-by-step process of fabrication, which consists of data segmentation, laser cutting, assembly with super glue, visualizations design with HTML, height design based on ShapeClip, and interaction control based on HTML. These toolkits can support the creation of prototypes with a decent level of details in functionality and interactivity. These toolkits make it possible for designers with knowledge of basic scripting, such as JavaScript and HTML, to create prototypes of shape displays. Foldio [50] is a toolkit belonging to this category. It is based on printable electronics and folding assembling process. The aim of these toolkits is creating functional prototypes. Although flexibility is built in these toolkits, it is not feasible to support instant and iterative idea changes such as those in design critique.

2.6. Limitations of Existing Prototypes and Motivation of ClothSurface

Prototypes based on mechanical pin-actuation elements or deformable displays can manifest ideas of shape-changing interface in a tangible and interactive fashion. The physicality and interactivity enable designers to interact with and understand ideas. However, these prototypes are typically built for manifesting specific design ideas; they are not created for rapid changes, which often occur in design exploration and design critique. Prototype toolkits based on mechanical pin-actuation elements are designed for changes. However, the changes supported are not sufficiently rapid for instant alterations.

One easy-to-change prototyping option is sketching with pen and paper. However, sketches are fundamentally static and 2D. The temporality and interactivity of design ideas are filtered out. Designers can accompany their sketches with gestures to enact behaviors of shape-changing interface. However, dynamic shape change cannot be easily represented with hand gestures. In other words, sketches may filter out too much of the aspects relevant to designers’ exploration of shape changing ideas. Parkes and Ishii [51] pointed out that a design language beyond sketches is needed for the aspects of motion and physical transformation. Design fiction is also relatively easy to change. However, it is characterized by zero to a very low level of details of manifesting forms and shape change.

What is missing is a prototyping tool for designing shape displays that (1) filter the key aspects of shape display design ideas (e.g., shape change, interactivity) and (2) manifest designers’ ideas that immediately “talk back” to the designers to enable instant reflection and critique. If a prototype is intended for instant reflection and/or critique on shape display ideas, a high fidelity in the prototype may not be needed. The prototyping tool fulfilling the gap must be simple and still retain aspects that are relevant to the exploration of the design space.

The current study is an attempt to fill this gap by introducing ClothSurface, a prototyping tool that manifests ideas for shape-changing interfaces (particularly shape displays) with materials allowing rapid changes. It filters dynamic and interactive behaviors of shape change for designers to explore the design space of shape displays.

3. ClothSurface: A Prototyping Tool for Shape Displays

We propose a tool to prototype ideas for shape displays called ClothSurface (The name ‘ClothSurface’ was coined at the time of writing this paper for the purpose of easy reference. During the study, the tool was simply referred to as “the new prototyping tool.”). The tool consists of a piece of spandex cloth mounted on a partially open box (Figure 1), and a number of everyday objects: a ball, a medium-sized box, a box of staples, a tape roll, and three water bottles. The spandex is treated as the shape-changing surface. The everyday objects were provided for the easy simulation of different forms of surfaces. The everyday objects were not in the initial design. They were added after two pilot study sessions with four design students (two per session). The participants were unable to simulate complicated concepts that needed more than two hands. Refinements were made afterwards by providing the everyday objects. Three water bottles were provided to support the objects temporally if the designer needed to simulate shapes that needed more than two hands.

3.1. Benefits of ClothSurface

The main idea behind ClothSurface was to create an elastic surface that enabled designers to put their hands beneath it to simulate different types of shape-changing behavior (Figure 1c). Other designers or users can sit next to a designer who is using the tool in order to experience this behavior. The design of ClothSurface is based on the Wizard-of-Oz technique [52]. Designers can enact the behavior of the shape-changing interface with their own bodies (mostly hands) [21] to externalize their ideas. This can facilitate collaborative design for shape displays [32].

The designer using the tool can observe how users or other designers interact on the cloth (the surface) and they can respond by changing the shape. They can also use their hands to control the rate and motion of the transformation of the shape change. ClothSurface can support embodied approaches of ideation [21]. Designers can rely on their bodies to imagine shape change ideas that are unexpected, surprising, or even magical. Merritt et al. posit that shape-changing interfaces have capabilities to offer such ‘imagined physics’ [53]. ClothSurface is a tool to support the ideation of imagined physics.

ClothSurface is intended to be used by designers at early stages of their design process. The focus is to support designers in manifesting their ideas for shape displays while filtering the design aspects that they can explore. Jung et al. [54] reported a process of design exploration with sketches and physical materials. ClothSurface can be a useful tool in early stage of such design exploration. For example, ClothSurface can serve as an early version of prototype for designs such as Relief [43]. In later stages of the design process, it is no doubt that designers need to create more prototypes with relevant aspects like interactivity for design evaluation—for example, for user testing.

3.2. Research Questions

The research questions of the explorative study detailed in the following sections are given below: Can ClothSurface enable designers to filter the design aspects of design exploration or brainstorming? If yes, what can be filtered? Can designers use ClothSurface to manifest shape-changing interactions? If yes, how can manifestation be done?

4. Method

We conducted a series of design sessions that consisted of a sketching stage followed by a prototyping stage using ClothSurface. The rationale underlying this two-stage arrangement was to simulate situations wherein designers begin with sketching and then proceed to prototyping. Sixteen volunteer master’s students (5 males, 11 females, all 22–31 years old) majoring in interaction design participated in eight design sessions (two volunteers per session). Their work experience ranged from 0 to 5 years. They were all international students from five different countries or regions. All of them indicated that they had either never heard of shape-changing interfaces, or had heard of them but had very limited to no experience with it. The teams were formed voluntarily at the time of registering for the study.

4.1. Design Tasks

The participants were asked to design either a tabletop radio or an office phone. For each of the two artefacts, the generated design needed to achieve two purposes: a focus on pragmatic use and hedonic use. In other words, each artefact had two design tasks, which were adopted from [36]. They were chosen because being pragmatic and hedonic are two broad purposes behind the use of shape-change interfaces [8]. In addition, a focus on pragmatic and hedonic uses has been shown to result in different shape-changing designs [8]. Given our focus on surface-based shape-changing interfaces, the original mobile phone task from [36] was modified to an office phone task. The design tasks were as follows:

• 1A Radio—Volume (pragmatic): Please sketch one or more examples of how physical changes in shape can be used to indicate the volume level on a radio. Your answer should explain how the user sees and changes the volume level.
• 1B Radio—Genre/Channel (hedonic): Please sketch one or more examples of how physical changes in shape can be used to indicate the genre of music playing on a radio. Your answer should explain how the user sees and changes the channel.
• 2A Office Phone—Volume (pragmatic): Please sketch one or more examples of how physical changes in shape can be used to indicate an office phone’s volume level (e.g., silent, low, normal or high). Your answer should also illustrate how the user changes the volume level.
• 2B Office Phone—Emotion (hedonic): Please sketch one or more examples of how physical changes in shape can be used to convey the emotion of the voice messages on an office phone. Your answer should illustrate how the messages are played on an office phone.

4.2. Session Procedure

In each session, there were two participants. After they arrived at the session venue, they filled in a questionnaire about their demographics and background. The researcher then introduced them to the topic of shape-changing user interfaces using a slideshow that consisted of videos showing three design cases of shape-changing surfaces [40,55,56].

Then, the pairs of participants were randomly assigned to the radio or office phone condition and were given the associated design task. They were given a pen and paper to sketch and generate ideas for 15 minutes. The participants were also told that they could work together in any way they preferred (e.g., work separately and then discuss later, or work and discuss together throughout the process). After the 15-minute sketching stage, they described their ideas to a researcher recording them on video. The researcher then introduced them to ClothSurface. The participants continued to prototype their ideas and their design process. Subsequently, they simulated their ideas using ClothSurface, while being recorded on a video camera. Finally, they were interviewed about their feedback. The rationale behind the procedure was to simulate a design process where designers start brainstorming with sketches and then proceed to make tangible prototypes.

5. Results

5.1. Sketching and ClothSurface: Filtering Dimensions Addressed

This section presents the analysis of the filtering dimensions addressed in the prototypes in the sketching and ClothSurface stages.

Table A2. Filtering Dimensions in the Sessions Assigned to the Radio Design Tasks (1A & 1B).

Se.	Sketching Stage	ClothSurface Stage
S1	Filtering dimensions addressed: Appearance: Shapes (ripple, rings, bricks, music instruments), mention of color change Interactivity: Hand gestures (rotating, dragging, stacking) and shape change with written notes, sound feedback Spatial structure: Relative arrangement of elements Filtering dimensions not addressed/discussed: Data; Functionality	Filtering dimensions addressed: Appearance: Shapes (ripple, rings, bricks, music instruments); Interactivity: Rhythm of emerging ripple or music instruments to reflect music mood/genre, users’ gestures (rotating, dragging) Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S3	Filtering dimensions addressed: Appearance: Shapes (wave, bubbles/spheres, rings), different states of shape change Interactivity: Hand gestures (swiping, pressing, pinching). Spatial structure: Relative positions of elements Filtering dimensions not addressed/discussed: Data; Functionality	Filtering dimensions addressed: Appearance: Shapes (wave, bubbles/spheres, rings) and shape change Interactivity: Speed shape change in response to users’ gestures; height/depth indicating music volume; rhythm of waves indicating music genre; swiping to change channel. Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S6	Filtering dimensions addressed: Appearance: Shapes (round-shape radio nob, fountain, peak) Interactivity: Turning a nob as changing channels, moving the nob to an end to change the volume, rhythm of shape change indicating emotions of voice messages, height of a peak represents music volume; pressing and pinching the peak to change volume; moving the hill to change channel Spatial Structure: Relative arrangement of shape-change surface and other elements Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure	Filtering dimensions addressed: Appearance: Shapes (peak) and shape change Interactivity: Pressing and pinching the surface to change volume; moving the peak on the surface to change channel; moving a peak towards the center to change volume; moving another peak to change channel Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S7	Filtering dimensions addressed: Appearance: Shapes (wave form, bar, slope, pointy surface) Interactivity: Height of a bar indicating the tempo of the music being played, moving a bar to change the channel, moving another bar to change volume; wave hands up/down above a surface to change volume, wave hands left/right to change channels Spatial Structure: Relative arrangement of the shape-changing surface and a loudspeaker Filtering dimensions not addressed/discussed: Data; Functionality	Filtering dimensions addressed: Appearance: Shapes (wave form, bar, slope, pointy surface) Interactivity: Speed of a bar in response to users’ gesture to change volume; rhythm of the wave-like surface indicating music genre; wave hands up/down above a surface to change volume, wave hands left/right to change channels Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure

The format of the table is inspired by the analysis reported in [31].

and

Table A3. Filtering Dimensions in the Sessions Assigned to the Office Phone Design Tasks (2A & 2B).

Se.	Sketching stage	ClothSurface stage
S2	Filtering dimensions addressed: Appearance: Shapes (wave, flat surface, facial expressions/emojis, envelope, vertical bars) Interactivity: Hand gestures (dragging, raising hands) Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure	Filtering dimensions addressed: Appearance: Shapes (wave, vertical bars) Interactivity: Speed of shape change in response to gesture input; users’ gestures (dragging an emerged slider); rhythm of waves indicating emotions of voice messages Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S4	Filtering dimensions addressed: Appearance: Shapes (hill, wave, wrapped surface, flat surface, pointy surface) Interactivity: Hand gestures (pressing, opening, tapping), rhythm of shape change indicating emotions of voice messages Spatial Structure: Relative arrangement of shape-change surface and other elements Filtering dimensions not addressed/discussed: Data; Functionality	Filtering dimensions addressed: Appearance—shapes (smooth and pointy surfaces, spiral surface) and shape change Interactivity—pressing and pinching the surface to change volume; rhythm of wave movement to indicate different emotions Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S5	Filtering dimensions addressed: Appearance: Shapes (wave, round-shaped hill, balloon, phone icon) Interactivity: Hand gestures (poking a round-shaped hill), nodding head to answer calls, voice inputs Functionality: Detecting fraud calls Spatial Structure: Relative positions of a shape-changing surface and an office room Filtering dimensions not addressed/discussed: Data;	Filtering dimensions addressed: Appearance: Shapes (waves, emerged phone icon, exclamation mark icon, phone icon) Interactivity: Pressing and pinching the surface to change volume; rhythm of wave movement to indicate volume; swiping the surface to listen to voice messages; pressing the surface to ignore the voice messages Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure
S8	Filtering dimensions addressed: Appearance: Icons/emojis/letters, slope, wave, nob, disc/circular bar Interactivity: Pressing different parts of a slope to change volume, or wave hands on the slope to change volume; press a disc harder/softer to change volume; turning a nob to change volume; moving a bar to change volume; vibration of a bar indicates emotion of voice messages Spatial Structure: Relative positions of a shape-changing surface and an office room; relative arrangements of elements indicating volume and messages Not addressed/discussed filtering dimensions: Data; Functionality	Filtering dimensions addressed: Appearance: Icons/emojis/letters; waving surface; round-top hill Interactivity: Slide left/right a peak on the surface to change volume, pressing an icon emerged from a surface to listen to a voice message; pressing a surface harder/softer to change volume; rhythm of a waving surface indicates the volume; rhythm of a moving disc to indicate the emotion of voice messages; sliding an emerged sphere to change volume Filtering dimensions not addressed/discussed: Data; Functionality; Spatial Structure

The format of the table is inspired by that of the analysis reported in [31].

in Appendix B summarize the analysis for the sessions assigned to the two design tasks, respectively. The given design task mentioned the functions required in the generated ideas and the relevant data (e.g., the data on the volume level for a volume-changing function). Therefore, these two filtering dimensions were considered to be addressed by the prototypes only if the participants addressed data and functionality other than the aspects mentioned in design tasks.

In the sketching stage, the prototype (i.e., the sketches) of the sessions addressed appearance, interactivity, and, for most sessions, spatial structure. In the ClothSurface stage, spatial structure was an aspect that was filtered out.

Data and functionality dimensions were omitted in both stages in almost all the sessions. The participants appeared to be very focused on the functions and data, given the design tasks during the design sessions. The only exception was that in the sketching stage of session S5, the participants thought of a new function for detecting fraud phone calls.

5.2. ClothSurface for Manifestation of Design Ideas

This section presents an analysis of how the participants used ClothSurface to manifest design ideas of shape displays for the two design tasks.

Table 1. Manifestation Dimensions and Use of Materials in the Sessions Assigned to the Radio Design Tasks (1A & 1B).

Se.	ClothSurface Prototype—Manifestation Dimensions	Highlights of the Use of Materials in Manually Manifesting Shape—Changing Interactions (often Accompanied by Verbal Descriptions)
S1	Materials used: ClothSurface, hands, water bottle, tape roll, box of staples Resolution: Manual simulation with ClothSurface; peer participant acting as a user Scope: Only manifested one interaction at a time	S1L operated ClothSurface, S1R acted as the user: When participant S1R (acting as a user) dragged on the ClothSurface, participant S1L’s fingers moved out/inwards around the tape roll to manifest an expanding/shrinking ring emerged from the surface (Figure 2a). A water bottle supporting a tape roll was used to manifest the initial state of the ring. Participant S1L moved her fingers upwards/downwards to manifest the rhythm of a jumping ring emerging from the surface S1R-operated ClothSurface, S1L acted as the simulator: Participant S1R held the tape roll under the spandex to manifest a ring. Participant S1L turned it with her right hand to manifest an interaction to change the channel.
S3	Materials: ClothSurface, hands, ball, tape roll Resolution: Manual simulation with ClothSurface; peer participant acting as a user Scope: Only manifested one interaction at a time	S3L operated ClothSurface, S3R acted as the user: Participant S3L emerged a tape roll from the surface with an upward and downward motion to manifest a jumping ring. When the peer participant S3R (acting as a user) pushed into the hole of the tape roll, S3L put her fingers around the tape roll and moved outwards to manifest an expanding ring. Then S3L emerged the tape roll near the edge of the surface to manifest other rings moving away as one ring expanded. Participant S3L used the up-and-down motion of the jumping tape roll to represent the genre of music. S3R operated ClothSurface, S3L acted as the user: Participant S3R emerged a ball from the surface to manifest a round-top hill. She moved the ball upward and downward as she put her other hand above the surface to pick and press the ball up and down (Figure 2b). The height of the round-top hill represented the volume.
S6	Materials: ClothSurface, hands, ball, water bottle, tape roll Resolution: Manual simulation with ClothSurface; peer participant acting as a user Scope: Only manifested one interaction at a time	S6L operated ClothSurface and acted as the user: Participant S6L emerged a ball from the surface with a water bottle supporting it in order to manifest a round-top hill. Then, he held a tape roll with his right hand, emerging from the surface to manifest a nob raised from the surface. He used his left hand to hold the tape roll from above the surface to manifest a user moving the nob toward or away from the round-top hill to manifest an interaction of adjusting volume. Participant S6L moved the ball up/down to manifest different heights of the round-top hill indicating different volume levels. Participant S6L moved around the nob he manifested with the tape roll to manifest an interaction of changing the channel S6R operated ClothSurface and acted as the user: Participant S6R emerged his right index finger from the surface to manifest a pointy hill on the surface (Figure 2c). He pulled and pushed the pointy hill with his left hand above the surface to change volume. Participant S6R moved the pointy hill that he manifested with his right index finger to different locations on the surface. This was manifest in an interaction of changing channels.
S7	Materials: ClothSurface, hands, tape roll, two water bottles, medium-size box Resolution: Manual simulation with ClothSurface; peer participant acting as a user Scope: Only manifested one interaction at a time	S7L operated ClothSurface, S7R acted as the user: Participant S7L moved his fingers rapidly below the surface to manifest a wave-like moving surface, which represented a channel playing fast-paced music. Participant S7L emerged a tape roll (supported with a water bottle) and the cap of another water bottle from the surface to manifest two nob-like hills on the surface. While participant S7R pointed into the tape roll with her right hand and moved forward and backward, S7L moved the tape roll simultaneously to manifest an interaction of changing volume (Figure 2d). While participant S7R pointed at a water bottle with her right hand and moved left and right, S7L moved the water bottle simultaneously to manifest an interaction of changing the channel S7R operated ClothSurface, S7L acted as a user: While participant S7L waved his right hand upward above the surface, S7R emerged both her hands from the surface to manifest an emerging hill-like shape. While participant S7L waved his right hand to the left and right above the surface, S7R emerged both her index fingers from the surface to manifest different pointy wave-like shapes emerged (which represented different channels).

and

Table 2. Manifestation Dimensions and Use of Materials in the Sessions Assigned to the Office Phone Design Tasks (2A & 2B).

Se.	ClothSurface Prototype—Manifestation Dimensions	Highlights of the Use of Materials in Manually Manifesting Shape—Changing Interactions (often Accompanied by Verbal Descriptions)
S2	Materials: ClothSurface, hands, ball, tape roll, box of staples Resolution: Manual simulation with ClothSurface; peer participant acting as the user Scope: Only manifested one interaction at a time	S2L operated ClothSurface, S2R acted as the user: When S2R (acting as a user) moved her hand upward above the ClothSurface, participant S2L moved a tape roll upward to pretend to be a raising disc from surface (Figure 2e). S2L moved a box of staples emerged from ClothSurface left/right as S2R dragged left/right with her finger. Participant S2L moved a ball emerged from ClothSurface to enact a smooth wave, with a slow rhythm indicating a calm voice message.
S4	Materials: ClothSurface, hands, ball, medium-size box Resolution: Manual simulation with ClothSurface; peer participant acting as the user Scope: Only manifested one interaction at a time	S4L operated ClothSurface, S4R acted as the user: Participant S4L emerged a ball from the surface to manifest a round-top hill. He moved the ball upward and downward as the peer participant S4R put her hand above the surface to pick and press the ball up and down. The height of the round-top hill represented the volume. Participant S4L emerged the corner of a medium-size box to manifest a hill with a pointy top, which indicated a voice message with a strong emotion (Figure 2f). A round-top hill (manifested with an emerged ball) indicated a “soft” emotion. Participant S4L grabbed the spandex and pulled it downwards to manifest a downward spiral shape, which indicated a negative emotion.
S5	Materials: ClothSurface, hands, ball, water bottle Resolution: Manual simulation with ClothSurface; peer participant acted as a user Scope: Only manifested one interaction at a time	S5L operated ClothSurface, acted as the user: Participant S5L moved a ball upward and downward beneath the spandex. The magnitude of the vertical motion indicated the volume level. S5L hovered her right hand above the spandex and gradually moved downward. Simultaneously, she reduced the aptitude of the vertical motion of the ball to manifest an interaction of reducing the volume level. S5L tapped the ball emerged from the spandex (Figure 2g). Then, S5L emerged her fingers from the spandex to manifest an interaction of playing a voice message. She moved her fingers upward and downward to manifest a wave-like shape change, which indicated the voice message was being played. S5R operated ClothSurface and acted as the user: Participant S5R moved his left hand up and down beneath the spandex to indicate an incoming call. He tapped his right hand on the spandex and moved backwards. Simultaneously, he reduced the speed of the vertical motion of his left hand to manifest an interaction of reducing volume. Participant S5R emerged a water bottle from the spandex with a vertical motion to manifest an exclamation mark emerged from the surface, to indicate a fraud or suspicious call.
S8	Materials: ClothSurface, hands, medium box, tape roll, water bottle Resolution: Manual simulation with ClothSurface; peer participant acting as the user Scope: Only manifested one interaction at a time	S8L operated ClothSurface, S8R acted as the user: While S8R tapped and moved her left index finger to the left and right on the surface, participant S8L simultaneously moved her index finger (emerged from the surface) to the left and right at the same time, to manifest an interaction of changing volume. S8L held a medium box below the spandex to manifest an emerging icon (Figure 2h). As S8R tapped the medium box above the spandex, participant S8L moved the box downwards to manifest a respond to the user’s tapping. Then, S8L continuously moved her fingers beneath the spandex to manifest a wave-like moving surface, which indicated a voice message being played S8R operated ClothSurface, S8L acted as the user: While S8L tapped and moved her right index finger in a circular path on the surface, S8R moved a ball (emerged from the surface) in a circular path and upwards at the same time, to manifest an interaction of increasing volume. S8L tapped a tape roll held by S8R beneath the spandex. Then, S8R moved the tape roll down to manifest a respond to the user’s tap and played the voice message. S8R kept tilting a tape roll beneath the spandex to indicate an unread voice message. The speed of the tilting motion indicated the emotion of the message. Fast speed indicated angry/strong emotion. Slow speed indicated a calm emotion.

summarize the analysis. The analysis focuses on how the participants used ClothSurface to manifest different shape-changing interactions. The participants used their hands and the provided physical objects underneath the spandex of ClothSurface to manually manifest changes in the surface. Two participants in each session worked together to manifest interactions. The participant who operated the ClothSurface responded by moving objects or their hands at the moment when the peer participant acting as the user attempted to interact with the prototype.

While manifesting their prototypes, many of the participants accompanied it with a verbal explanation. In doing so, they frequently used deixis such as ‘this’ and ‘that’ to describe shapes, shape changes and movements (e.g., ‘a shape likes this’ or ‘it moves like this’). The use of deixis was often accompanied with a pointing finger or the slight movement of an item held under the cloth.

The interactions found in the answers were dragging, swiping, pinching, pressing, tapping and pushing. In participant S1L’s answer, users could drag ring-like shapes to expand them (to increase the music volume). The participants have been labelled as follows: the letter ‘S’, the session number and the letter ‘L’ or ‘R’ to refer to which side of the camera the participant was sitting e.g., participant S3L refers to the participant sitting on the left of the camera in the third session. Four of the answers assumed that the shape-changing interface had hand-tracking capability and involved the use of hovering hand gestures (e.g., swiping) above ClothSurface instead of touching it. Five answers included a button-like shape for users to press, tap or push.

The resolution and scope manifestation dimensions were very similar across the sessions. In terms of resolution, the prototypes with ClothSurface were manually operated and, thus, offered some details on interactivity. The level of interactivity was sufficient for peer participants to act as users and to attempt interacting with them; the scope of the manifestation was limited to one interaction at a time.

5.3. Participants’ Feedback

In the interviews, the participants gave overall positive feedback about ClothSurface. Three participants (S4L, S6R and S6L) said that ClothSurface allowed designers to illustrate their ideas quickly. Four participants (S1R, S1L, S8R, and S8L) appreciated that they could use a ClothSurface prototype to simulate and touch their concept. S1R said that when she saw her partner interacting with her prototype, she noticed aspects that she had missed and started thinking about solutions. Participants S2R and S8R made similar comments. S7R asked S7L a question about his design, which prompted him to start thinking of a solution.

Participant S8L appreciated the possibility for her to pretend to be a user and touch a prototype. She said the tool allowed her to become a user and interact with a shape-changing interface that she could touch. This feature gave her a concrete idea of what her partner’s concept was.

The participants acknowledged that they discovered some problems with their ideas when they were able to simulate their concepts and that ClothSurface helped them identify issues they had not thought of in the sketching stage. Participant S3R said that the experience of touching a prototype stimulated him and made him improve his idea. Participant S4R made similar comments.

However, the participants made some negative comments. Participants S8L and S8R thought that the objects provided perhaps limited the ideation process. Participant S3L wanted to create the items by herself, so that she could use exactly the shapes she wanted. Participant S6R said that the tool itself might limit designers’ ideation if they do not know the final device well enough. Participants S5R and S5L thought that users could understand how a prototype based on ClothSurface works only if they used their imagination. Participant S4L pointed out that it was not easy to use it to simulate shape changes with inward or downward transformation.

6. Discussion

6.1. ClothSurface to Support Filtering and Manifestation

Although this was the first time the participants were introduced to ClothSurface, they managed to manifest various ideas of shape display interactions using it. Although the range of interactions manifested were not an exhaustive list of all possible interactions with shape displays, it was a demonstration that ClothSurface can support designers to explore the design space of shape displays in a simple manner. From the Sketching stage to the ClothSurface stage, the spatial structure was filtered out. The participants appeared to focus on the appearance and interactivity of different shape display ideas with ClothSurface.

In the design sessions, the prototypes produced with ClothSurface usually filtered out data, functionally, and spatial structure. Peer designers who have engaged in the same design tasks can easily understand other designers’ ideas prototyped with ClothSurface. The prototypes are abstract and proximate. Therefore, designers need to verbally explain their concepts alongside their ClothSurface prototypes. Occasionally, they can refer to their sketches to support their explanation (e.g., which part of their design ideas they were illustrating). The strength of ClothSurface is that it facilitates design discussions with a rough, approximate and tangible prototype that all the participating designers can easily understand, refer to and modify.

The scope in the manifestation was limited to one interaction at a time; this was because ClothSurface required manual operation. Therefore, one designer can only manage a limited amount of manifestation at a time. This might be the reason why spatial structure dimension was filtered out in the ClothSurface stage. Spatial structure addressed in the sketching stage was about the relative arrangements of elements. The limited scope prevented the participants from manifesting multiple elements and/or interactions, thereby causing participants to focus on one interaction at a time. This became a filtering mechanism to guide designers to focus on the design of one interaction at a time.

6.2. Limitations

The current form of ClothSurface does have a few limitations, for example, the risk of letting the user notice that the shape-changing interface prototype is manually simulated. Prototypes created with ClothSurface are not ready for user testing. ClothSurface is aimed to be used in the early stage of a design process, when designers are exploring ideas among their peers. Ideas generated with ClothSurface are not guaranteed to be technically possible as well. Further technical investigation on generated ideas is needed.

The design sessions in the current study were limited in terms of time and constrained to a specific room. Further, the iterative nature of design was not fully considered in the current study, although at least two participants returned to their sketches and modified them. Future studies can investigate the use of ClothSurface in a relatively lengthier design process.

ClothSurface can be tested with experts in shape displays in the future. They may have a rich vocabulary of shape changing interface and thus rely less on the physicality offered by ClothSurface. On the other hand, it is also possible that the physicality and manual-operated interactivity may still be useful to experts when they engage in rapid iterations of design ideas and design critique.

7. Conclusions

Currently, there is a lack of low-cost prototyping tools for shape-changing interfaces. This study proposed ClothSurface—a low-cost prototyping tool for shape-changing interfaces. In a series of sessions, ClothSurface was shown to provoke explorations of shape-changing interfaces by designers who are new to this sphere of design. Designers can use ClothSurface in their process of ideating shape-changing interfaces; moreover, ClothSurface supports communication and discussions among designers as well. Designers can manifest their ideas using ClothSurface and discuss these with their peers on filtered aspects, particularly appearance (shape change) and interactivity.

The design of ClothSurface is not intended to represent a complete solution to the challenge of prototyping shape-changing interfaces. From sketches on paper to fully functional prototypes, the process in between may involve many design iterations. It is hoped that the low-cost and ever-ready nature of ClothSurface can encourage designers to try out their ideas beyond sketching, without worrying about the technical details involved in technology-enabled prototypes. It is hoped that ClothSurface can complement the explorations of future scenarios through tangible prototypes.