Tangible User Interface and Mu Rhythm Suppression: The Effect of User Interface on the Brain Activity in Its Operator and Observer

Abstract

The intuitiveness of tangible user interface (TUI) is not only for its operator. It is quite possible that this type of user interface (UI) can also have an effect on the experience and learning of observers who are just watching the operator using it. To understand the possible effect of TUI, the present study focused on the mu rhythm suppression in the sensorimotor area reflecting execution and observation of action, and investigated the brain activity both in its operator and observer. In the observer experiment, the effect of TUI on its observers was demonstrated through the brain activity. Although the effect of the grasping action itself was uncertain, the unpredictability of the result of the action seemed to have some effect on the mirror neuron system (MNS)-related brain activity. In the operator experiment, in spite of the same grasping action, the brain activity was activated in the sensorimotor area when UI functions were included (TUI). Such activation of the brain activity was not found with a graphical user interface (GUI) that has UI functions without grasping action. These results suggest that the MNS-related brain activity is involved in the effect of TUI, indicating the possibility of UI evaluation based on brain activity.

1. Introduction

Tangible user interface (TUI) is a type of UI that allows a person to interact with digital information through the physical environment [1]. This type of UI involves the use of tangible objects that bridges the gap between digital information and physical space, and is characterized by its UI function based on the manipulation (e.g., grasping and moving) of that objects [2]. Because TUI accepts specific user actions on the objects as inputs, the result of each action can be easily predicted, which is thought to bring more intuitiveness in comparison with other UIs. By combining physical and digital representations, TUI provides more fun as well as intuitiveness to its users. Its applications in the design community include those focused on user experience (e.g., [3,4]) and learning at museums [5].

The intuitiveness of various TUIs provided at museums is likely to have an effect on the experience and learning of not only those who are operating them, but also of those who are just watching it. TUI can make observers understand the purpose of each user action more easily, because the user action on the tangible object is visible.

Such characteristics have led to its applications to education (e.g., [6,7,8]), providing more fertile environment for education than conventional GUI [9]. For example, as proposed by Hornecker et al. [10], TUI allows a variety of interaction styles and also has known social aspects. Its application to collaboration was proposed early on (e.g., [11,12]). TUI is also known to have an effect on its observers. When a person wants to use a UI for the first time, he/she often starts by watching someone actually operating it. Through such an observation to learn the operation, he/she can more easily find an interest in the operation and will become more motivated to operate the UI by himself/herself. Therefore, it is important to evaluate the intuitiveness of a UI for the observer.

There are a lot of UI evaluation methods, including those based on subjective, behavioral, and psychological approaches (e.g., [13,14,15]). However, most studies to date have focused on the effect on the user. The effect on observers has been studied about their behaviors or experiences, but it has only been investigated about physiological responses in few cases [16,17,18].

In the present study, we take a neuroscientific approach based on a brain function that is activated by observing the action of others. There are a set of neurons that fire both by performing and observing the same action. These are referred to as mirror neurons, first identified in specific areas in the brain of macaque monkey [19].

It has been reported that mirror neurons play an important role in the understanding of the intention of an action of others [20]. In humans, fMRI studies have shown that an MNS-like function involves a plurality of brain areas. MNS in human is far more complex than that in macaque monkey, and has been reported to be involved in imitation and empathy [21] as well as understanding of the purpose of an action of others [22].

EEG has been proposed as a convenient means to monitor the activity of MNS [23]. Performing and observing an action both suppress the alpha band rhythm in the central sulcus [24]. The alpha band rhythm around the central sulcus is called mu rhythm. Mu rhythm suppression is enhanced by observing a goal-directed action or performing a social task [25,26]. In particular, the band power in 10–12 Hz was previously reported as a relatively sensitive indicator of the motor cortex activity [25,27].

One characteristic of TUI is that it involves physical actions of the user to grasp a tangible object and the like, which are readily visible to its observers (e.g., [9,28]). On the other hand, it is known that MNS is activated by the observation of the action of others. MNS also responds to the movement of hands and tools shown in an introductory video for museum [29]. In particular, it shows a pronounced response to the action of grasping an object [30]. With regard to the action of the operator of UI, such a grasping action makes TUI different from GUI, a more common type of UI [2]. What kind of effect on MNS is caused by watching others operating TUI, in comparison with GUI?

By monitoring the brain activity related to MNS during the operation of different UIs (i.e., different information acquisition processes), it will be possible to evaluate the effect of each UI in terms of empathy, understanding of intentions, and the like. Because UI is primarily designed for its users, there is a non-negligible effect on a person actually operating it, besides its observer. Therefore, in the present study, we conducted two types of experiment to understand the effect of TUI on both its operator and observer in terms of the MNS-related brain activity. First, we examined the effect of the observation of UI on the MNS-related brain activity in the observer (Experiment 1). Next, we investigated the same brain activity in the operator (Experiment 2). Based on the results of both experiments, the effect that was characteristic of TUI on the brain activity could be addressed.

2. Materials and Methods

2.1. Experiment 1

In Experiment 1, the brain activity in the observer watching the UI operation from behind was monitored to investigate the effect on the observer.

2.1.1. Subjects

The subjects were 15 right-handed students (21.9 ± 1.2 years old). All subjects signed the informed consent form. The present study was approved by the Ethical Committee of Kyushu University. No subject had a history of a psychiatric or neurological disorder.

2.1.2. Experimental Conditions

In the present study, each subject sat behind and watched an assistant (hereafter actor) operating a TUI.

One of the art appreciation systems provided at the Louvre DNP Museum Lab was simplified into three different experimental TUIs composed of a screen and either tangible objects or a touch panel with object thumbnails. The effect of TUI was investigated in three conditions, i.e., using these different types of UI (two TUI and one GUI; Figure 1).

We adopted two TUI conditions to compare presence/absence of corresponding relation between the selected object and the result, and third GUI condition to compare presence/absence of the grasping motion.

In the first condition, a set of small porcelain models (a total of eight different porcelains) was adopted as UI (TUI/OBJECT condition). In each task, the actor grasped one model and moved it onto a holder in front of the screen. As a result, the screen displayed a porcelain picture that corresponded to the model. Then the actor hided the porcelain picture from the screen by returning the model to the initial position. The actor conducted this task for all of the eight models (i.e., a total of eight tasks of displaying/hiding a porcelain picture).

In the second condition, eight identical can models were used as UI (TUI/CAN condition). The actor conducted a total of eight tasks of displaying/hiding a porcelain picture as in the first condition, except that the appearance of each model was not predicative of the corresponding porcelain picture.

In the third condition, thumbnails of eight different porcelains provided on the touch panel served as UI (GUI condition). In each task, the actor touched one thumbnail with a finger, as a result of which the screen displayed a porcelain picture that corresponded to the thumbnail. Then, the actor moved only a hand onto the holder in front of the screen. After that, the actor moved the hand and touched the same thumbnail to hide the porcelain picture from the screen. Then the actor returned the hand to the holder. The actor conducted a total of eight tasks of displaying/hiding a porcelain picture as in the other conditions.

2.1.3. Experimental Procedure

Each subject put on an EEG electrode cap and sat 2 m behind the actor (Figure 2). During the first 70 s, the subject stayed at rest looking at a fixation point displayed on the screen. Then, the actor started to operate the UI. The operation time for each picture was about 8.7 s, and a total of eight operation tasks were conducted which included all the eight different pictures in a random order. This set of tasks (about 70 s) was repeated once again. EEG was recorded while the subject was at rest and watching the actor’s manipulation, and data from the resting state and the second set were used for the subsequent analysis. We carried out the above process for each of the three conditions, wherein the order of these conditions was counter-balanced between the subjects. Each subject was instructed to sit still throughout the EEG measurement watching the hand of the actor during the manipulation and the screen during each picture displayed on the screen.

2.1.4. EEG Measurement and Analysis

A 64-channel Ag-AgCl electrode cap, an EGI NetAmps 200 amplifier, and Netstation acquisition software (Electric Geodesics, Inc., Eugene, OR, USA) were used for the EEG measurement. The average of all channels was used as the reference, and sampling was done at 250 Hz, with a band-pass filter ranging from 0.3 to 100 Hz. FFT was applied to EEG segments (4.09 s interval, 1024 points, Hanning window), which were overlapped by 50% (EMSE Data Editor 5.3, Source Signal Imaging, Inc., La Mesa, CA, USA). Data from the first 10 s were excluded from the study for reason of transients in attention. The average power in 10–12 Hz was adopted as the mu rhythm power. Each power value obtained was converted into a logarithm to ensure the normality of the data. EMSE Suite Data Editor 5.3 Release Candidate 3 (Source Signal Imaging, Inc.) was used for data analysis. Mu rhythm suppression in the observer watching the actor’s action was adopted as the index of MNS.

Mu rhythm suppression in each condition was calculated using data from the resting state as reference. Because the sensorimotor area showed mu rhythm suppression, EEG was analyzed in two ROIs: left central (LC) (4 electrodes around C3) and right central (RC) (4 electrodes around C4).

The main effects of conditions (the TUI/OBJECT condition, the TUI/CAN condition, and the GUI condition) and brain areas (LC and RC), and their interactions were tested by two-way ANOVA. For multiple comparisons, we used modified sequentially rejective bonferroni procedure. Before significance testing, outliers in each subject were identified by Smirnov-Grubbs test (p < 0.01). All the outlier channels were excluded from the subsequent statistical analyses.

2.2. Experiment 2

In Experiment 2, the brain activity in the operator actually operating UI was monitored to study the effect on the operator.

2.2.1. Subjects

The subjects were 18 right-handed students (22.1 ± 1.57 years old). Other details were the same as described above for Experiment 1.

2.2.2. Experimental Conditions

An ACTION condition, which involved the user action but no visual information as a results of each actions, was included in addition to the three conditions in Experiment 1 (i.e., four conditions in total). The fourth condition was adopted to compare presence/absence of visual feedback of the operation result. The eight identical can models in the TUI/CAN condition were also used in this condition (Figure 3). The operator conducted a total of eight tasks as in the TUI/CAN condition, except that no porcelain picture was displayed on the screen.

2.2.3. Experimental Procedure

Each subject put on an EEG electrode cap and sat in front of the screen. The operator stayed at rest for the first 60 s and then started to operate the UI. The operation time for each picture was about 13 s, and a total of eight operation tasks were conducted which included all the eight different pictures in a random order. This set of tasks (about 104 s) was repeated once again. EEG of the operator at rest and during the two sets of tasks was recorded, and data from the resting state and the second set were used for the subsequent analysis. We carried out the above process for each of the four conditions, wherein the order of these conditions was counter-balanced between the subjects.

The subject was instructed to operate to a sound stimulus presented at a constant rhythm, for the sake of the consistency of the movement. The subject was also instructed to sit as still as possible during the EEG measurement and to keep looking at the manipulating hand during the manipulation and at the screen during a picture was displayed, so that the eye movement would be in line with the operation process. The subject went through enough explanation and exercise before the experiment.

2.2.4. EEG Measurement and Analysis

For comparison with Experiment 1, the 10–12 Hz component of EEG was analyzed as in Experiment 1.

Mu rhythm suppression in the operator was adopted as the index of brain activities. The main effects of conditions (the TUI/OBJECT condition, the TUI/CAN condition, the GUI condition, the ACTION condition) and brain areas (LC and RC), and their interactions were tested by two-way ANOVA. The other analysis procedure was the same as Experiment 1.

3. Results

3.1. Experiment 1

Mu rhythm suppression in each brain area due to UI was examined by comparing the power values in the mu band in the resting state and during the operation, by one sample t-test (Figure 4). As a result, RC in the TUI/CAN condition showed a significant difference (t (14) = −2.46; p = 0.028). In contrast, no brain area showed a significant difference in the TUI/OBJECT condition or the GUI condition. Based on the data from the resting state and the data from the operation, mu rhythm suppression was determined for LC and RC in different conditions (TUI/OBJECT condition, TUI/CAN condition, and GUI condition; Figure 4).

Next, the main effect of conditions (TUI/OBJECT condition, TUI/CAN condition, and GUI condition) and brain areas (LC and RC), and their interactions were tested by two-way ANOVA. The ANOVA results confirmed an interaction between condition and brain area (F(2, 28) = 4.245; p = 0.025; η2p = 0.233), and a main effect of condition in RC (F(2, 28) = 4.234; p = 0.025; η2p = 0.232). By t-test, there was a tendency that the TUI/CAN condition resulted in a higher suppression in comparison with other two conditions (t (14) = 2.50; p = 0.077 in TUI/CAN vs. TUI/OBJECT, t (14) = 2.02; p = 0.077 in TUI/CAN vs. GUI).

3.2. Experiment 2

As in Experiment 1, mu rhythm suppression in each brain area due to UI was examined by comparing the power values in the mu band in the resting state and during the operation, by one sample t-test (Figure 5). As a result, there are significant differences (t (17) = −3.30; p = 0.004 in TUI/CAN at RC, t (17) = −2.50; p = 0.023 in TUI/CAN at LC, t (17) = −2.32; p = 0.033 in TUI/OBJECT at LC, t (17) = −2.79; p = 0.013 in TUI/OBJECT at RC). In contrast, in the GUI condition and the ACTION condition, no brain area showed a significant difference. Based on the data from the resting state and the data from the operation, mu rhythm suppression was determined for LC and RC in different conditions (TUI condition, CAN condition, GUI condition, and ACTION condition; Figure 5).

Then, the main effect between conditions (the TUI/OBJECT condition, the TUI/CAN condition, the GUI condition, and the ACTION condition) and between brain areas (LC and RC) were tested by two-way ANOVA. The ANOVA results confirmed a main effect of condition (F(2.3, 39.12) = 6.572; p = 0.002; η2p = 0.279). By t-test, it was shown that the TUI/OBJ condition and the TUI/CAN condition resulted in a significant suppression in comparison with the GUI condition and the ACTION condition (t (17) = 3.66; p = 0.012 in TUI/OBJECT vs. GUI, t (17) = 3.65; p = 0.012 in TUI/OBJECT vs. ACTION, t (17) = 3.12; p = 0.019 in TUI/CAN vs. GUI, t (17) = 2.74; p = 0.042 in TUI/CAN vs. ACTION).

4. Discussion

In Experiment 1, we found that the brain activity in the observer varied depending on the type of UI (Figure 4).

In the TUI/CAN condition, watching the UI operation resulted in an elevated activity in RC in the observer in comparison with the resting state. This is consistent with the report that watching the action of others results in mu rhythm suppression in the somatosensory cortex, which roughly corresponds to RC [23,31]. Therefore, it is suggested that the brain activity was induced in this MNS-related brain area. Among brain areas, only RC showed the activity in this experiment. We suppose this was because the user action took place in the left visual hemifield of the observer [29,32] rather than the observer imagined imitating the right hand action of the operator as a result of watching it. This activity tended to be higher in the TUI/CAN condition in comparison with the TUI/OBJECT condition and the GUI condition, indicating that it was characteristic of the TUI/CAN condition. The MNS activity reflects the immediate goal of an action [22]. Thus we had expected mu rhythm to be suppressed in the TUI/OBJECT condition, but found the suppression only in the TUI/CAN condition. It has been reported that the observation of a highly unfamiliar action results in an elevated activity in the motor cortex [33,34]. In the TUI/CAN condition in the present study, the result of each action is unpredictable, which is not seen in ordinary UIs. We suppose this caused the elevated activity in the sensorimotor area.

On the other hand, the response of the observer did not seem to differ between the TUI/OBJECT condition and the GUI condition, in spite of the presence/absence of the grasping. In this experiment, any of the two conditions did not even show activation in the same brain area, indicating that they did not activate the MNS. It has been reported that the observation of a familiar task results in a lower activity in the inferior frontal gyrus and the premotor cortex in comparison with an unfamiliar task [35], and that repeated grasping of the same object leads to weaker mu rhythm suppression [36]. We suppose that the repeated stimulus presentation in the present study hindered the effect of the grasping action from being detected.

In Experiment 2, we found that the brain activity in the operator also varied depending on the type of UI (Figure 5).

Areas around the somatosensory cortex were active in the TUI/OBJECT condition and the TUI/CAN condition, suggesting the possible induction of a brain activity related to the brain activity seen in Experiment 1 [23,31]. The brain areas that were active in this experiment included not only LC, an area activated by performing a right hand action, but also RC. The visual information of each user action was presented primarily in the left visual hemifield, whereas the visual information of the output as the result of the action was presented primarily in the right visual hemifield. Processing of such visual information possibly had some effect [29,32]. These activities tended to be higher in the TUI/OBJECT condition in comparison with the GUI condition and the ACTION condition. The TUI/OBJECT condition and the GUI condition differed in that only the former involved the grasping action. In an fMRI study, the action of reaching for an object and grasping it and the action of only reaching for that object differed in that the former resulted in an elevated activity in the anterior intraparietal sulcus (aIPS; [37]). In our experiment, the aIPS activity due to grasping probably had an effect on the sensorimotor area. The TUI/OBJECT condition and the ACTION condition involved the same user action, and only differed in that the former displayed a porcelain picture on the screen as the result of each action. The elevated activity in LC and RC in the TUI/OBJECT condition, in comparison with the ACTION condition, suggests that the visual output as the result of each action had some effect on the activity of sensorimotor area.

Similar results were also obtained in the TUI/CAN condition, except that RC showed a particularly pronounced activity in the TUI/CAN condition, as seen in Experiment 1. It is possible that the unpredictability of the result of each action enhanced the observer’s attention [33,34].

In the present study, we showed that MNS in the observer was more active in the TUI/CAN condition than in the GUI condition: i.e., the effect of TUI on its observers was demonstrated based on the brain activity.

We had also expected to detect the effect of the grasping action [2] as an MNS activity in the TUI/OBJECT condition, in comparison with the GUI condition. However, we did not obtain such a result in the present study. One possibility is that the observer recognized the TUI as UI from the movement of the objects, but failed to pay attention to the action of “grasping” the objects. On the other hand, in the TUI/CAN condition, the unpredictability of the result of each action seemed to affect the MNS-related brain activity. We suppose that the factor of unfamiliarity attracted the attention and interest of the observer.

However, the brain activity in the operator did not show a difference between the TUI/CAN condition and the TUI/OBJECT condition. These conditions involved the same action as the ACTION condition, but activated the brain activity in the somatosensory cortex because they were provided with a screen to output the results. In comparison with the GUI condition, the TUI/CAN condition and the TUI/OBJECT condition resulted in an elevated activity in the same brain area because they involved the “grasping” action.

It is suggested that the same brain activity defined by mu rhythm suppression in this area is activated in the observer not only by a hand action but also by the addition of an unpredictable nature to UI, reflecting the degree of interest.

Altogether, the mu rhythm suppression reported in the present study might represent a brain activity that is activated by the combination of UI function, grasping action, and interest in UI, but not by either of them alone. Therefore the mu rhythm suppression in LC and RC can be used as an index in the evaluation of “grasping” action-based TUIs.

5. Conclusions

In this research, we adopted the following approaches for TUI, which have been hardly studied so far. One is an approach from brain activity and the other is an approach from the influence on surrounding observers.

To understand the possible effect of TUI, the present study focused on the mu rhythm suppression in the sensorimotor area reflecting execution and observation of action. By investigating the brain activity, we showed the possibility that a TUI involving a goal-directed action can activate a sensorimotor area that reflects MNS, both in the operator and the observer. We thought it indicated a part of the effectiveness of TUI from the influence on brain activity. It is one of the significance to positively adopt TUI.

Moreover, we also showed the possibility that the monitoring of MNS-related brain activities in the observer can be a novel UI evaluation method.