The Study of Bending and Twisting Input Modalities in Deformable Interfaces

Abstract

The deformable input provides users with the ability of physical operation equipment to interact with the system. In order to facilitate further development in flexible display interactive technology, we devised FlexSheet, an input device that can simulate the deformation environment. This paper presents two forms of deformation input, bending and twisting, with regard to three selection techniques. We conduct a controlled experiment to select discrete targets by combining two input forms and three selection strategies, taking into account the influence of visual feedback. Further, we use the deformation angle to reflect the degree of deformation and map it to the experimental variables. In accordance with the experimental results, we analyze the experimental performance under three evaluation indexes and prove the viability of our selection technology in bending and twisting input modes. Finally, we provide suggestions on the control level in bending and twisting input modes, respectively.

1. Introduction

General displays have achieved good performance after technical iterations, like liquid crystal display (LCD), which have been widely used in everyday life. Limited to its own structural characteristics, it can only express two-dimensional information and the degree of freedom of interaction is limited to the screen surface [1]. With the introduction of the concept of organic user interfaces (OUIs) [2], the research on deformable user interfaces (DUI) [3,4] and its interaction technology is becoming increasingly extensive. The interactive feature of flexible devices is that we can change their shape in accordance with different functions, such as using deformation as input parameters to open music [5,6], manipulate maps [7], and even prove that it can be used to input passwords, which enhances interactive security during the process of innovation [8], thereby providing an advantage that traditional screen technology does not have. This additional input mode not only greatly increases the bandwidth of human-computer interaction [9] but also inspires researchers to explore greater possibilities of this interaction technology.

The design of the deformation simulation prototype is the premise and foundation of this kind of researches, and the main objective is to simulate the input environment of deformation equipment. Certain researchers use electrical components as deformation sensors [5,8,10,11,12]. The principle is that when electrical components are deformed, their voltage changes accordingly. Moreover, different degrees of deformation can be simulated by tracking the reflective ball attached to the surface [7,13]. There are some other related work, such as the use of optical sensors [14,15] and the use of fiber-based deformation sensor clusters [16] to detect human intentional deformation, etc. Our implementation scheme is based on the OptiTrack motion capture system. In this system, multiple infrared cameras collect reflective identification points in space in real time and calculate the intersection of multiple direction vectors after completing target correction and recognition in order to determine the world coordinates. In addition, this infrared optical positioning technology can effectively eliminate visible light interference, improve system robustness, and yield a high data transmission rate, which is similar to what has been found in extant research [17].

There are numerous studies on enhancing interactive expressiveness based on prototypes. The works of deformation gesture has attracted the interest of many researchers. The remarkable feature of gesture operation is that it can give users a more intuitive experience and a strong sense of participation in the interaction process. Some researchers directly bind specially designed deformable gestures with functional requirements to improve interactive bandwidth, such as controlling smart TV [18]. A larger trend is that researchers collect user-defined deformation gesture consensus sets [19,20] to understand users’ natural interaction habits and solve some practical problems.

In the process of exploring deformation gestures, researchers found that devices using bending gestures are the most effective for rapid response to continuous bipolar variables [21], and suggested to maintain two orders of magnitude [22]. If it is a one handed operation, the deformation gesture in the upper right corner is a better choice [23].

In addition to taking the deformation degree of the equipment directly as the input parameter [5,7,17], many researchers have also discussed the input potential of the combination of deformation and other input methods, such as pressure [24,25], touch [26,27], and even provide new interaction methods for people with visual impairment by combining hearing [28] and kinesthetic [27].

Certain researchers believe that the advantage of deformation input is not only a single command binding. In other words, this interactive technology has a vivid metaphor, which is related to the user’s behavior and ideas in a few cases. For example, by reading [13], we can simulate the feeling of actual physical paper on the deformation device. When we need to turn the page [10], it is like using real documents. Scenes in virtual games—such as fishing, archery, or golf games are similar examples [11], and embodied in intelligent wearable healthcare equipment [29].

In this study, we design a prototype called FlexSheet that supports deformation as input and based on the OptiTrack motion capture system. This prototype can provide the interactive characteristics of flexible devices to simulate a deformable input environment (Figure 1). We call this the deformation controller. The deformation controller consists of two thin and soft polyvinyl chloride (PVC) parts of size 20 cm × 20 cm on which the reflective traceable ball is fixed. We use two input forms: bending and twisting. In addition, two visual feedback mechanisms are set up—full vision (FV) and partial vision (PV). Simultaneously, we use an angle to describe the degree of deformation. With the increase in force feedback [20,30], the angle of deformation becomes increasingly larger. For the quantitative analysis of deformation, we mapped the angle to the experimental variables and took the maximum bending angle of 180° in the horizontal state. Finally, through a simple input form, we checked the user’s ability to control deformation input in order to obtain relevant design guidelines.

The remainder of the paper is organized in the following manner. Section 2 introduces the experimental materials, including input mode, visual feedback, and selection technologies. Section 3 describes controlled experiments used to study deformation input performance. Section 4 analyzes the experimental results from the perspective of three indexes: completion time (MT), error rate (ER), and crossing times (NC); thereafter the best order of magnitude suggestions under the two input modes are provided, respectively. Section 5 discusses and summarizes the paper.

2. Materials Instruction

2.1. Input Modes

Most flexible screen studies have demonstrated that bending and twisting are two basic types in the design of deformation gestures. Combined with our research content, we provide the following design guidelines for deformation gestures:

• The deformation gestures must be universal (that is, they can be realized on as many materials as possible), because different materials allow different forms of deformation; plastic sheets are the most similar type of flexible display material that is currently available [18];
• The deformation gestures must give the feeling of using real thing [19];
• The deformation gestures must have a high degree of consistency in orientation [10];
• The deformation gestures must have powerful powerful metaphors [19].

According to the above design principles, we examined two kinds of deformation gestures: bending and twisting. The basic operation example is depicted in Figure 2.

2.2. Visual Feedback

As depicted in Figure 3, we used two visual feedback conditions: FV and PV. Under FV conditions the cursor is always visible, as illustrated in Figure 3a,b. Under PV conditions, the cursor is only displayed at the initial state of the selection task and is hidden once the selection task is initiated, as depicted in Figure 3c,d. Under PV conditions, the user must depend on previous experience in FV conditions to complete target selection. This simulates a situation in which expert users may learn to use deformation input intuitively and without excessive cursor support or visual cues, which is similar to the behavior that expert users display when tagging menus [31].

2.3. Selection Technologies

Once the cursor enters the target, the user must have a mechanism to identify and select options in the general graphical user interface (GUI), as general determination measures are usually performed by clicking the buttons on the mouse. Consequently, we tested three alternative technological solutions to replace the traditional mouse functions:

• Click: Move the cursor by bending or twisting the FlexSheet. When the cursor appears within the target rectangle, click the task button to complete the target selection;
• Dwell: Bending or twisting the FlexSheet to move the cursor until it appears within the target, then keep the cursor in the target for a specified period of time to complete the target selection (in our experiment, a delay of 0.7 seconds was used);
• Quick Release: Move the cursor by bending or twisting the FlexSheet. When the cursor appears in the target, quickly remove the FlexSheet from the deformed state.

3. Controlled Experiment

Through this experiment, our objective was to study the human ability to complete the task of selecting discrete string targets when using deformation inputs. This mainly includes the influence of visual feedback on the selection process and selection results, actual performance of three target selection strategies as well as the control level under different input modes.

3.1. Participants

We recruited 12 participants, including ten males and two females, aged 23–31 years, with an average age of 26 years. All participants are right-handed and have some user graphical interface experience, but have no experience of using deformation input.

3.2. Apparatus and Environment

The experiment was conducted on Lenovo computers running Windows10, 3.8 GHz, screen size of 27 inches and resolution of 1920 by 1080 pixels. The experiment also used a motion capture device, OptiTrack, which is composed of eight cameras with Prime13/13W specification and connected through Cisco switch. The deformable plate is made of a plastic sheet (20 cm × 20 cm), and markers accessories is fixed to the prototype. The products in the experiment were compiled by the NatNetSDK3.0.1 provided by OptiTrack official website and run in the Visual Studio 2015 environment.

3.3. Task

Our long-term goal is to explore the general design criteria for deformation interactions, with a particular emphasis on developing design guidelines for deformation interactions based on bending and twisting modes.

We used a serial target selection task. Control the cursor to move vertically by deforming the FlexSheet. The deformation value of 180° is uniformly mapped to 256 pixels. Within the distance of 256 pixels, we draw a set of continuous rectangular targets. In each experiment, a target rectangle to be selected will be highlighted in purple. The user’s task is to apply appropriate deformation and move the cursor to the target rectangle with different distances and different widths. When the cursor enters the range of the target rectangle, the target rectangle changes from purple to green to give visual feedback to the subject.

It should be noted that the cursor is displayed in the initial state of the task. Since we provide two different visual feedback mechanisms, when the cursor starts to move, the full visual feedback supports the full display of the cursor, while the cursor under partial visual feedback will be hidden.

After the target to be selected gives color feedback, the subject needs to use the three selection technologies we provide to quickly complete the target selection, and the experimental task is over. Subjects will then repeat this task many times under different combinations of conditions.

3.4. Performance Measures

MT is the dependent variable of the experimental task, which represents the time it takes to select each target, defined as the time (milliseconds) from moving the cursor after the target appears until the target is selected; ER means the number of incorrect selections in the target selection task accounts for the total number of selections the ratio; NC represents the cumulative number of crossovers that enter the target and then leave the target when selecting each target. The functions of these evaluation indicators complement each other. ER and MT provide us with an overview of the overall success rate, and NC provides us with an indication of the degree of control of the user’s application of deformed input.

3.5. Procedure and Design

A within-subjects full factorial design with repeated measures was used. The independent variables were selection modes (Bending and Twisting), selection technologies (Click, Dwell and Quick Release), visual feedback condition (FV, PV), the distance from the starting point to the target (D = 37, 72, 109, 144) and the target width (W = 15, 18, 22.5, 30, 45). The units of distance and width are represented by the deformation angle.

For each visual feedback, each participant will be exposed to two input modes. In each input mode, participants need to complete the experiments of all three selection technologies. For each selection technology, 20 D-W combination conditions are included and presented in random order. The experiment order was counterbalanced among participants using a Latin Square design.Each participant repeated the entire task three times to complete a total of 8640 individual tasks. In summary, the experiment consisted of:

• Selection modes (Bending and Twisting);
• Visual feedback conditions (FV and PV);
• Selection technologies (Click, Dwell and Quick Release);
• Target distance (D = 37, 72, 109, 144);
• Target width (W = 15, 18, 22.5, 30, 45);
• 12 Participants
• 3 Repetitions

Before the experiment, we explained the task, and performed a brief demonstration. Participants were given a short warm-up set of trials to familiarize themselves with input modes, visual feedback and selection technologies. Participants were instructed to perform the task as quickly and accurately as possible. Participants can take a short break during the experiment. The experiment lasted approximately 1 hour for each participant. To analyze the experimental results, we recorded the relevant parameters of each target selection task, including input modes, target size, target distance, time spent, errors, and crossing times.

4. Results

We analyzed the collected measures by conducting a repeated measures ANOVA (α = 0.05) that used the three-interaction technique and two-visual feedback condition as independent variables. Further, we used Greenhouse-Geisser corrected values in the analysis when the assumption of sphericity was violated (tested with Mauchly’s test). Pairwise t-tests with Bonferroni corrections were used for post hoc tests. The test was considered an outlier if the time required to complete the task exceeded two standard deviations of the average completion time of the task. A total of 282 outliers were eliminated, which accounted for 3.3% of the collected data.

4.1. Time

As depicted in Figure 4, in bending mode, we found that different selection techniques had no significant effect on performance time (F_1.342,14.764 = 2.504, p = 0.129). Further, we found no significant interaction between interaction technique and visual feedback (F_2,22 = 3.286, p = 0.056), as shown in Figure 5.

Although there was no significant difference in the performance of time between the three interactive technologies, it can be found that the most time was spent in Dwell, followed by Quick Release, and the then Click.

As illustrated in Figure 6, in twisting mode, we found that different selection techniques had no significant effect on performance time (F_1.165,12.818 = 2.284, p = 0.154). As depicted in Figure 7, we found a significant interaction between technology and visual feedback (F_1.171,12.882 = 4.749, p = 0.044). The visual feedback condition has a significant effect on Click time (p = 0.002) as well as on Dwell and Quick Release times (p < 0.001, p = 0.002). The overall trend is that the Quick Release time under PV is significantly higher than that under FV (p = 0.004, p = 0.034); there was no significant difference among all technologies (p > 0.1).

4.2. Accuracy

In bending mode, we found that selection technology has a significant effect on the accuracy (F_1.243,13.673 = 7.9, p = 0.01), as illustrated in Figure 8. The post hoc tests revealed no significant differences among the technologies except Click and Quick Release (p = 0.021) (p > 0.07).

As illustrated in Figure 9, there was a significant interaction between selection techniques and visual feedback (F_2,22 = 3.609, p = 0.044).

The level of visual feedback had no obvious effect on the error rate for Click (p = 0.604); however, a significant effect was found on the error rates for Dwell and Quick Release (p = 0.006, p = 0.005). The general trend was that the error under PV was significantly higher than that under FV. The post hoc tests revealed that under FV conditions, there were no significant differences between technologies; under PV conditions (p > 0.1), Click and Quick Release were significant (p = 0.012). In addition, there are no significant differences among other technologies (p > 0.1).

As depicted in Figure 10, in twisting mode, we found a main effect on the error rate of different selection techniques (F_1.167,12.839 = 11.812, p < 0.001).

The post hoc tests showed significant differences among all selection technologies (p = 0.02, p = 0.00, p = 0.047). Further, as illustrated in Figure 11, we found no significant effect between selection techniques and visual feedback (F_1.305,14.35 = 1.406, p > 0.2).

4.3. Indication of Control

As illustrated in Figure 12, in bending mode, we found that this technique had a significant impact on the number of performance crosses (F_1.254,13.791 = 8.527, p = 0.008). The post hoc tests revealed that there was no significant difference among all technologies, except Dwell and Quick Release (p = 0.022) (p > 0.05).

As presented in Figure 13, there was no significant interaction between technology and visual feedback (F_1.769,19.464 = 2.771, p = 0.092).

As depicted in Figure 14, in twisting mode, we found that different techniques had a significant effect on the NC (F_1.353,14.885 = 17.716, p < 0.001).

There was no significant difference between Click and Dwell (p = 0.032), as revealed by the post hoc tests, but there was no significant difference between Click and Dwell (p = 0.066, p = 0.069).

As depicted in Figure 15, there was no significant interaction between technology and visual feedback (F_1.218,13.397 = 5.783, p = 0.026). In addition, the level of visual feedback has no significant effect on the NC of Click (p = 0.084), Dwell (p = 0.169), and Quick Release (p = 0.009). The general trend was that the NC under PV condition is significantly higher than that under FV condition. In FV condition, Click and Quick Release (p = 0.011), Dwell and Quick Release (p = 0.007) differed significantly among different technologies, except Click and Dwell (p = 1). Under PV conditions, Click and Quick Release (p = 0.01) differed significantly from Dwell and Quick Release (p = 0.007), except for Click and Dwell (p = 1).

4.4. Optimum Number of Deformation Levels

One of the intentions of this research is to ascertain the number of independent bending levels (nLevels) a user can distinguish under a given level of device performance. Except the Quick Release, when nLevels ≤ 8, all selection technology error values between 1–2%, and there was no significant difference in the error rate (Figure 16). Except Quick Release, when nLevels ≤ 6, the NC of FV was stable around 1.1 in each selection technology; moreover, all the selection technologies of NC had no significant differences (Figure 17). In bending mode, these results indicate that six levels of deformation were a reasonable number in bending mode.

Except the Quick Release, when nLevels ≤ 8 and when all selection technology error values are between 1% and 3%, there were no significant differences between the error rates of the technology selected (Figure 18). In addition to Quick Release, when nLevels ≤ 8, of all stable technologies of NC in 1.1%, there was no significant difference (Figure 19). In twisting mode, these results indicate that eight levels of deformation were a reasonable number for humans to comfortably control in twisting mode.

5. Discussion and Conclusions

In this study, we examined the interaction technology of a flexible screen and designed a simulation prototype called FlexSheet, which can support deformation inputs. Among with the OptiTrack system, a deformation input environment was constructed. In this context, we summarized two basic deformation models: bending and twisting. In order to study the ability of humans to control deformation, we introduced the deformation angle as a reflection of the degree of deformation. Further, we designed a discrete object selection experiment using three selection techniques (Click, Dwell, and Quick Release) and two types of visual feedback FV and PV to simulate novice and expert users. According to the experimental performance of participants, a series of performance indicators were analyzed in detail. The results revealed that in two input modes, less time was spent on simple Click than Dwell and Quick Release, a finding that is in keeping with our hypothesis. The main reason is that the Click selection is more in line with a user’s operating habits. Another reason is that the other two selection methods need to be implemented with a certain time delay, thereby reducing the selection efficiency. In terms of accuracy, both models had similar performance, with the lowest error rate for Click, followed by Dwell, and then Quick Release. In terms of stability (NC), from the input mode, the number of twisting intersections was lesser than that of bending intersections, but the overall performance is similar. From the perspective of different selection technologies, the average number of intersections to Dwell is the lowest, followed by Click, and then Quick Release. This is consistent with the feedback of participants that rapid release is more difficult to control than other technologies, and it is easier to remain within the target range using bending input, which leads to higher time cost, error rate, and crossing times. According to our measurement and analysis, the maximum deformation grade can be divided into six levels in bending and eight levels in twisting. The division basis of deformation levels comes from the mapping of deformation angle to width variables in the task, and the division results are reflected by user feedback. It is worth noting that in each index analysis, the overall performance of FV is better than that of PV.

Further, although we discussed the specific performance of the two deformation input modes, respectively, we believe that the research on the combination of the two input modes has good potential in future research on flexible screen technology. In addition, we believe that the research on ergonomic models based on deformation input is of great significance. We are optimistic regarding the future of flexible interactive technology, and our study can provide designers a few useful guidelines for designing interactive techniques based on flexible screen.