Expressive Interaction Design Using Facial Muscles as Controllers

Abstract

Here we describe a proof-of-concept case study focusing on the design and development of a novel computer interface that uses facial muscles to control interactivity within a virtual environment. We have developed a system comprised of skin-mounted electrodes that detect underlying muscle activity through electromyography. The signals from the electrodes are filtered and smoothed, then used as input data to an application that displays a virtual environment with a 3D animated avatar. The user’s expressions control the facial movements of the avatar, thus conveying user emotions through real-time animation of a representative face in a virtual scenario. To achieve this, we collaborated with our Public and Patient Involvement focus group to discuss concepts and design appropriate interactions, while simultaneously developing a prototype system. Programmers and 3D artists worked together to create a system whereby individual user facial muscles are connected to 3D animated models of the same muscle features represented in an avatar, providing the user with an option to receive visual and numerical feedback on the extent of their muscle control. Using the prototype system, people can communicate facial expressions virtually with each other, without the need for a camera. This research is part of an on-going project to develop a facial muscle rehabilitation system that can be optimized to help patients with conditions such as hypomimia.

1. Introduction

The motivation for this research was to design a novel therapeutic paradigm for facial neurorehabilitation, focusing on the development of a technological platform backed by relevant neuroscientific knowledge. To date, therapies to rehabilitate facial muscles have received little attention, since the focus of research has been on maintaining and improving limb and hand movements that are essential for daily activities. However, facial muscle disorders are challenging social issues that not only affects physical competence, but the psychological wellbeing of patients, their family and friends. Parkinson’s Disease, for example, is a progressive nervous system disorder that can lead to a loss of muscle control in different parts of the body; people with the condition may find themselves increasingly unable to express their emotions to others if their facial muscles deteriorate (known as hypomimia or ‘masked face’). There may also be an impact on clarity of speech and associated psychological well-being, increasing the person’s need for support. Regaining the ability to express themselves will encourage individuals to become more confident in social environments and gain independence.

The human face conveys a wealth of subtle information, including mood and emotion as well as identity [1,2] and as a species, we have evolved to interpret these signals as part of our communication strategy. There are at least 43 muscles in the face, which move in concert to create expressions ranging from happiness to anger and despair [3]. These facial muscles are attached to the bones of the skull and enable us to perform important functions in daily life, including mastication through movement of the tongue and mouth, articulation of speech and forming expressions (See Figure 1).

With the advancement of media technology and Internet access over the last 50 years, digital communication has become commonplace, despite the loss of physical information experienced by users. Although streamed video enables remote participants to hear and see each other in real time, subtle body language may be lost off-screen. Another factor that can reduce the integrity of the experience is that in some multiuser contexts (notably gaming), participants select an avatar as their representative, a virtual figure that can often be customised. Since traditional game controllers are designed for two hands, they have limitations in respect to the manipulation of virtual characters. The controls are optimised for enabling large body movements, essential for gameplay, but do not generally allow players to alter characters’ facial expressions. There are several reasons for this—for example: (i) there are not enough buttons, combos or fingers to work the controls; (ii) there is not sufficient time to move the character and simultaneously modify the face; (iii) facial reactions during gameplay situations are more often instinctive than selected.

This might not seem important in the context of an action game, but with ever-increasing enthusiasm for social, multi-player gaming between groups of friends, personalisation and sharing live facial reactions can enhance what is essentially a remote experience, by making it more lifelike. Moreover, avatars that can appear to express emotions are becoming more prevalent as people spend more time in digital spaces, for both work and leisure [4,5]. The continuing appeal of remote networking and the ability to choose one’s virtual representation means that virtual multiuser spaces have proliferated—and the ability to communicate effectively is critical. We argue that it is a fundamental responsibility of developers to provide inclusive systems that will enable all people, whatever their physical or mental characteristics, to be involved in these spaces. Although artificial intelligence (AI), as deployed in computer games, can use context to determine and activate an appropriate expression on virtual agents (e.g., non-player characters, or NPCs) or on players’ avatars (virtual representations of the player), there may be drawbacks. Whilst AI-generated expressions can lend credibility to an agent’s performance, AI-determined player avatar expressions are not as reliable or personal as the player’s true expression.

One way round this problem is for people to allow a webcam to capture their faces while they are communicating—not to display the video, but so that an algorithm can determine their expressions and underlying emotions. This information can then be represented in their avatar’s expression. Although many systems for interpreting human expressions from video data are robust [6,7,8], they do not consider the challenges experienced by people with reduced facial muscle capacity, nor those who have headwear or glasses that obscure facial features. Our research is therefore exploring an alternative method for capturing facial movement data, using face-mounted sensors.

This approach has several potential advantages. For example, in a virtual environment, users need to wear headsets, but computer vision for expression recognition works better with full face video data. Currently available popular headsets are bulky and cover much of the top half of the face.

Another benefit is mobility—the fact that the system does not require the user to be constantly facing a camera, which frees them from location constraints. We are experiencing a gradual transition away from screen-based interfaces with technology, towards more intuitive and human-centred styles of interaction that use gestures and speech, and which require more kinetic freedom [9]. The development of lightweight sensors that can detect changes in movement, pressure, temperature and humidity has enabled wearables, offering users a selection of new tools [10]. Our system can contribute an alternative portable device that has a novel interaction modality in the form of a facial expression interface with technology.

Furthermore, a camera can only detect significant movement of the surface of the skin, which resembles the perceptive abilities of a human observer. Face-mounted sensors, on the other hand, can register subtle amounts of muscle movement beneath the surface, enabling expression. The data provides an indication of user intent, as well as a measurement of their performance. Our system is therefore capable of offering detailed feedback to users based on their ability to flex relevant muscles, which suggests a potential for muscle training and rehabilitation. We believe this could help people with Parkinson’s Disease (PD) or those who have facial paralysis caused by stroke, trauma or COVID-19. Additionally, alternative interfaces for interacting with technology have the potential to support users who are unable to manipulate traditional controllers with their hands, or to encourage interaction in people with different kinds of communication disorders.

PD is a long-term degenerate, incurable condition that affects the central nervous system. When cells in the brain stop working correctly, a chemical called dopamine is not produced in insufficient amounts, resulting in the inability to control body movements, including those of the face. People living with hypomimia may be offered treatment in the form of one-to-one physiotherapy sessions with a specific therapist, which incurs a significant time investment from the healthcare professional, in addition to patients needing to travel to a clinic. It can also be difficult for patients and physiotherapists to build the confidence required to facilitate treatment within a relatively short time slot. Whilst there are a number of tried and tested therapies documented as being effective in treating facial palsy, including selective serotonin reuptake inhibitors, massage and mirror therapy [11], there is no standard, effective treatment currently being used.

We propose to develop a tool that motivates facial muscle movement in people with facial muscle impairments, targeting specific muscles and integrating immediate feedback. The device is intended mainly for home use, to support independence, choice and personal control, offering a flexible pace and a motivating routine. Feedback generated over time will be stored and shared with the patient’s physician to provide an overview of progress.

The rest of this paper describes our proof-of-concept case-study, the development of a system that deploys a novel hands-free facial muscle interface connecting to a computer application, using muscle movement data to control facial muscle animations and facial expressions of a 3D avatar.

2. Materials and Methods

An overview of the project development is presented in Figure 2: Project Flow Diagram. This shows how ideation and development have progressed over time, in a parallel configuration that incorporates a feedback loop with our Public and Patient Involvement (PPI) focus group.

Our initial design was developed after collaborating with our focus group. Using their feedback, we then planned our system using a combination of art, animation and engineering tools. The design was motivated by feedback from the focus group, with reference to established neurological methodologies and previous successes in the use of virtual reality technologies for rehabilitation. The development team were concurrently investigating facial muscle movements and how to create a suitable control system. This involved tracking skin movement on volunteers, 3D modelling and animating an avatar head, capturing individual facial muscle movements and linking user muscle data to the avatar face in a virtual environment. A detailed explanation of the flow diagram (Figure 2) and the associated technology is provided in the subsequent sections ‘Concept and Interaction Design’ and ‘System Development’.

3. Concept and Interaction Design

The team was committed to a participatory design approach, involving the potential end-users of the system from the earliest stages of concept design. We therefore collaborated with Parkinson’s UK (https://www.parkinsons.org.uk; accessed on 15 August 2022) in order to recruit some volunteers to form a small focus group. During our initial consultation with the focus group, our PPI volunteers with PD provided feedback and advice enabling us to gain insights into the challenges facing those who experience symptoms of hypomimia. Some of the comments demonstrate that the psychological impact of facial muscle deterioration cannot be overstated.

‘Difficulty swallowing and speaking clearly made me feel very self-conscious, so I avoided going out to public places.’

‘Low self-esteem prevented me from enjoying a social life and burdened the rest of my family too… I hate having photographs taken as I focus on the mask effect. It makes me sad that I always take the photos on family occasions when we look back there is no reflection that I was there.’

‘The general public only recognise the physical symptoms and it’s the non-visible that are usually the most impactful on you psychologically… Easier to use coping strategies and avoid situations leaving you even more isolated and vulnerable.’

The system we envisage will be equipped with sensors that measure small electrical signals generated by facial muscles; these signals from the patient’s facial muscles interact with a virtual reality (VR) environment which reacts to the user’s effort, offering a pleasurable and motivating encounter with a responsive avatar. Moreover, the output received by the user is perceived immediately and maps their muscle movements to a personalised visual and auditory experience that can provide more nuanced and useful feedback than viewing themselves in a mirror.

The rationale for the interaction design is based around the challenge of providing rehabilitation for weakened facial muscles through self-motivated physiotherapy. It is grounded in two established neurological practices—Action Observation Therapy (AOT) and Motor Imagery—and in the use of VR to engage and immerse the user within a stimulating context.

3.1. Neurological Basis of Design Concept

Action Observation Therapy (AOT) supports rehabilitation through the promotion of neural plasticity, whereby new networks can be established if old ones have been lost. When we perform an action, specific neurons in the brain are activated; however, even if we just see that action being performed, ‘mirror’ neurons are activated. Visuo-motor connections are being made, probably to enable us to understand the intentions of others [12]. According to Sarasso et al. [13]: ‘AOT has an effect in improving motor function regardless of the disease and the severity of motor impairment’.

AOT involves initially showing the patient a desired action, which activates these mirror neurons, thereby reinforcing the existing cortical networks and facilitating the rebuilding of the damaged ones [13]. Lundqvist [14] found that facial expressions are contagious, while Ravaja et al. [15] observed that the emotional expressions of virtual characters (VCs) influenced human emotional responses. We know that there is positivity associated with smiling and being smiled at, triggering endorphin production [16], and therefore our concept includes responsive avatars that can perform positive facial expressions and react to user facial input.

While AOT has been shown to work for gait improvement in Parkinson’s Disease [17], Bek et al. [18] and Caligiore et al. [12] note that Motor Imagery (imagining performing the action) works well in combination with AOT. Moreover, Mezzarobba et al. [19] found that sound was helpful for providing feedback to users learning how to modify their gait, as it was less distracting than visual cues; they therefore recommend a multimodal approach for designing interactive systems.

3.2. Using VR for Motivation and Delivery

Virtual Reality (VR) has been used successfully in stroke rehabilitation [20,21] and with AOT in Buccofacial Apraxia rehab [22]. Developing our device as a VR experience will offer the user a non-invasive, fully immersive training environment that provides feedback in real time, showing them how their reactions affect a situation. It is hoped that such immediate feedback will be motivating for the user who can immediately see the benefits of the device and will then hopefully continue to want to use it for sustained improvement. VR is gaining in popularity as a fun leisure pursuit and is particularly suitable for providing users with an intriguing and interactive environment in the comfort of their own home. Indeed, home-use was identified as a major benefit by our PPI group:

‘I think that there is a need for something like this device that people could access in the privacy of their own home. Freedom to choose the place and time for therapies and exercise is paramount to its effectiveness and its portability would open its use to people living in isolated communities.’

‘I would use it. It would enable me to be active in managing my symptoms and be able to do so at a time and place to suit my own lifestyle. It is portable so I could use it in my own home and even if on holiday or away from home it would be portable enough to carry.’

Initially, development focused on capturing movement of the voluntary muscles, such as the zygomaticus major (see Figure 1), which is responsible for smiling, since this is such an important expression for communicating with others [23]. Technology that collects personal information can be perceived as invasive, sharing data over which we may have little control, but which are often hidden from others, such as pulse, temperature and moisture. Facial expressions, on the other hand, have the potential to be changed at will, as well as reflecting our natural state of mind. In other words, people are more capable of controlling the data they share via an emotional interface than they are when using typical sports monitoring wearables.

Emotion recognition for VR has been gaining traction, with recent studies deploying different technologies to explore the topic—for example, capturing data using a Kinect [24], and using machine learning (ML) to train systems with facial electromyogram (EMG) sensors [25] or train systems with photo reflective sensors embedded in a headset [26]. Our proposed early solution does not need to rely on ML algorithms to determine expressions from limited data, because we have developed a model that directly and immediately translates individual muscle movements from a real user into their virtual ‘muscular’ equivalents on an avatar within a 3D environment.

VR offers a number of distinct benefits over other conventional approaches, including privacy in the context of not being directly observed during practice sessions, accessibility and a bespoke experience. It also offers deep immersion that facilitates engagement with optimal performance. Links between behavioural change and gamification have been explored by Charles et al. [20], an aspect that was also addressed by our PPI group volunteers, offering their feedback on the initial concept:

‘Its modern technological approach would engage the younger generation and I think people would be more motivated to maintain a programme long term.’

‘…keeping motivated in management of exercises is crucial in Parkinson’s. Ensuring a regular and timely routine is maintained helps stop apathy and complacency setting in and achieving visible goals is down to maintaining long term routines, no excuses.’

The device and associated system would offer the opportunity for long-term management and maintenance of muscle strength and flexibility.

4. System Development

Our development environment of choice for this work was Unity (https://unity.com/; accessed 15 August 2022), as it provides a ready-made engine for rapid prototyping of interactive 3D applications. Unity is a cross-platform game engine that can be used to create 2D and 3D games, simulations and other experiences for mobile, console, desktop and virtual reality deployment. For the development of this prototype, two teams worked concurrently—artists and programmers, both adapting their techniques to meet the demands of the project. The art team modelled and animated a range of heads, using Autodesk Maya (https://www.autodesk.co.uk/; accessed 15 August 2022). These showed bone and simplified muscle structure underneath a skin that could be rendered semi-transparent. Animation of facial expressions matched the underlying muscle movement, allowing for differences between right and left. The programming team focused on collecting muscle movement data using non-invasive sensors connected to an Arduino microcontroller (https://www.arduino.cc/; accessed 15 August 2022) via a SEN0204 Electromyogram (EMG) Sensor by OYMotion (http://www.oymotion.com/; accessed 15 August 2022). This involved filtering the sensor signals and calibrating the output so that it could be used as a control mechanism. The procedures followed and associated results are described in the following sections that present this work.

4.1. Analysing Skin Movement

Eight volunteers were recruited from the research team academics and associated students, comprising five women and three men, ages ranging from 21–59 years. This was a small and expedient sample, used for initial visual measurement of surface skin movement, and not intended to be representative of the target users, who would be expected to include a full spectrum of ages, skin types and facial shapes.

The volunteer’s facial skin movements were captured and analysed whilst presenting different expressions (such as smiling), using comparative photography taken from different angles and 14 ‘nodes’ (A–N) lightly marked on the skin (see Figure 3 and Figure 4).

The skin movement was measured by noting the direction of movement and distance of each ‘smiling’ node compared to the node’s original position on the neutral face.

The values were estimated using a scale of 0–300 units, where 100 units equalled the node diameter (actual value 8 mm sticky dot) and the direction of movement was estimated using an approximate ‘compass’. This data was subsequently used by artists to inform the design and animation of the 3D models.

4.2. 3D Modelling and Animation

The art team produced a range of 3D heads using Autodesk Maya. These were based on anatomical drawings, dot measurement data and photographs. The artists were able to model both the surface ‘mask’ that is visible and the underlying muscles most important for enabling emotional expressions (see Figure 5).

It was important that individual virtual facial muscle and skin animation sequences, although pre-rendered, could be precisely controlled by the user deploying their personal facial muscles. To achieve this, the animations were created using blend shape deformers on individual 3D objects (see Figure 6). When exported as FBX files into the development environment used by the programming team, the animation sequences could be handled using scripts, meaning that input data from external sensors could control the amount of movement being displayed.

4.3. Bioelectrical Interface for Capturing Facial Muscle Movement

A critical aspect of the work was to be able to capture facial muscle movement using external sensors and use this as input data for a computer application. In order to transfer muscle movement data through a current to a computer application, the information received first had to be converted into a format the application could recognise. We used an Arduino microcontroller with a SEN0204 EMG Sensor to manage the capture of sensor data and this was integrated with Unity using the Ardity library (https://ardity.dwilches.com; accessed 15 August 2022) that enables serial communication.

At first, a standard analogue input sensor (potentiometer) was tested for its potential to control the procedural animation of a 3D object in a Unity scene. When it was confirmed that an analogue sensor could control the scale of a 3D primitive, the same input was successfully mapped to a blend shape animation sequence.

Following this, different kinds of bio-sensor components were tested on forearm and cheek muscles (see Figure 7). The quantity, range and amplitude of signals produced depends on the current activity and the mass of that muscle group, with more mass resulting in higher impulses of electrical currents. This means that facial muscle movements produce less signals than limb movements, for example, and that therefore the output from facial muscles can be more precise. The OYMotion electrode plate was only suitable for detecting large muscle movement, so it was replaced with smaller Ambu Blue Monitoring electrodes (https://www.ambu.com/cardiology/clinical-evidence/ambu-bluesensor-ecg-electrodes; accessed 15 August 2022) and then with Skintact RT34 resting electrocardiogram (ECG) electrodes (https://www.cardiodepot.co.uk/s/F10505_skintact; accessed 15 August 2022). Sonogel GELE100 electrode gel was used to ensure good contact between surface and sensor.

Both EMG and ECG electrodes detect electrical signals, so they were interchangeable for this experiment. EMG readings measure the strength and speed of signals transmitted by motor neurons, and for the prototype, two electrodes had to be placed between the ends of the zygomaticus muscle, along the longitudinal midline and ideally centrally, not near the tendon at the cheekbone nor the motor point at the corner of the mouth (see Figure 7). EMG signal readings are mostly in the frequency range 50–150 Hz, but as there is a lot of noise, a reference electrode is also used to provide a stable baseline [27]. This should be located on electrically neutral tissue, which in this case, was the user’s forehead (see Figure 7 and Figure 10). The SEN0240 sensor integrates a filtering circuit and an amplified circuit. It amplifies minimal sEMG within ±1.5 mV 1000 times and depresses noises (especially power frequency interference) by using differential input and an analog filter circuit (https://github.com/oymotion/EMGFilters; accessed 15 August 2022).

The team then focused on deploying appropriate sensors to record the electrical activities of the muscles underlying the facial skin and to translate this data into a signal that could be interpreted by the Unity engine.

5. Results and Discussion

The measurements taken from student and staff volunteers presenting neutral and smiling faces are shown in Figure 8. Importantly, these results demonstrate the variability in skin movement between the participants in the small study, despite each person presenting the same identifiable expression. For example, although Node D (placed at the side of the mouth—see Figure 3 and Figure 8) shows a large amount of skin movement across all participants, the lowest value was 8 mm (100 on the scale used) and the highest value was 22.4 mm (280 on the scale). By comparison, Node A (placed in centre of chin) shows a much smaller range of movement (between 0.8 mm and 6.8 mm), but still with much variation between participants.

It will be interesting to find out whether there exists a comparable range in the amount of subcutaneous muscle flexing taking place in this group, although we anticipate that age and facial shape are more likely causes of variation in skin movement. This requires further exploration.

The programming team successfully created an interface that was able to detect bioelectrical signals from the facial muscles of an individual, as well as from other muscle groups in the forearm. The interface supplied input data to an application that presented users with a human avatar. Users could control the animation of the avatar by flexing their own facial muscles.

This was achieved by using ECG sensors connected to an EMG signal transmitter board that amplified and filtered the signals before sending them to a microcontroller. In Arduino scripts, the data was further manipulated by mapping the received values (ranging from <0 to >600) to a new scale, 1–100. For all users, the system had to be initially calibrated to take into account individual variations in muscle movement and associated electrical activity. A baseline threshold was set depending on the user’s neutral expression (no voluntary muscle activity). This then displayed 0 on the 1–100 scale, whereas the same user trying to move the muscle group of choice would output higher values.

The filtered sensor values were still very jittery (see Figure 9), so an algorithm was used to smooth them before passing the data via a serial connection as input to Unity. This involved taking an average signal value over a short time period and outputting an integer in the range 1–100. The Arduino–Unity serial connection has a limit on the amount of data that can be sent before it is overloaded, but three sensors worn on the face were able to send data successfully at a rate of approximately 10 bps at a baud rate of 115,200, which was sufficient to cover the zygomaticus muscle on one side of the face (see Figure 9).

Facial expressions on the avatar face could then be controlled programmatically by applying a weight (directly received as input data from the sensors) to each blend shape animation, on a scale of 0–100. In the test version, the face mesh was split into 4 blend shapes—lower and upper right, lower and upper left. Figure 9 shows the process of linking real facial expressions to virtual ones, initially controlling the lower right side of the face.

One of the challenges was to capture and manipulate the data so that it represented the current state of muscle contraction, rather than the current change in value. The corresponding 3D animation in Unity had to reflect the same amount of muscle change, and it was also important that the system could retain a numerical indication of muscle strength at any given moment. In terms of using this as a tool for supporting rehabilitation, this information would enable people to monitor muscle strength over time.

The prototype solution enables someone wearing three topical sensors (on cheek and forehead) to gradually change their expression from neutral to smiling, and in doing so, control the facial expression of an avatar. In Figure 10, you can see the change in facial expression of the avatar on the screen as a direct response to the output from the facial muscles via the sensors. This facial movement is animating in real time as the user flexes their zygomaticus muscle on the right side of their face.

We have shared these preliminary results with our PPI focus group and invited them to give us their feedback. We are particularly interested in people’s reactions to the use of technology to support rehabilitation. Comments from the focus group have been positive:

‘For me, it’s engaging with the innovative technology which will cross all age boundaries, both for patients’ benefit but also the … designers. It is good to see the latest gaming technologies being utilised not only to bring potential effective therapies but bringing fun element to the public. Cross generational thinking is the way for the future.’

‘Great to see the work progress… it looks very exciting.’

‘Addresses the important psychological benefits that are usually missed in conventional therapies.’

5.1. Future Plan

The next stage will be to bring an early prototype to our PPI group volunteers and discuss different aspects of the design. We plan to work collaboratively on the ergonomic features (wearability and ease of use) of the proposed headset, and also on the VR user interface and associated training paradigms to engage users with interactive activities that are aimed at providing muscle rehabilitation.

Meanwhile, there is work to be done on refining sensor outputs and targeting different facial muscles, by connecting more SEN0240 signal transmitter boards to Arduino, which can manage up to six simultaneous inputs. This would enable us to test how effective the system would be as a remote emotional interaction interface; six inputs would enable the capture of left and right zygomaticus muscles as well as the frontalis muscle from two local users via one microcontroller. We hope to be able to detect and transmit a variety of smiles, frowns and skewed expressions.

Additionally, we plan to implement and test the interface as part of a networked multi-user VR environment. We envisage the system having potential for online social settings or group therapy. For example, it could be used by PwP (People with Parkinson’s) in a virtual meeting, with the added benefit that the user’s avatar would be able to compensate for any deterioration in muscle control so that the user still fully expressed themselves.

The prototype must also be capable of collecting and storing motor data using different facial biotypes, in order to build a relevant model of data associated with forming different expressions. It became evident while testing the device that the precise positioning of the facial electrodes was critical for obtaining good results, so it might be necessary to create bespoke physical solutions that acknowledge the individuality of users. Creating an inclusive dataset is necessary to underpin work that analyses user experience and progress; it may be possible to identify intentional expressions from weak muscle signals and provide benchmarks for comparisons.

5.2. Future Impact

The inability to control facial muscles can affect the ability to show emotion and this can be very upsetting for individuals, their families and their friends, potentially leading to loss of confidence in social settings. If this project is successful in developing an interactive system that helps rehabilitate weakened facial muscles, it could have a huge impact, lifting a barrier to social contact and assisting with psychological and physical health plans. Independent rehabilitation therapy supports patients to regain control and become proactive in their health care.

Furthermore, there is scope to repurpose the interface so that it can be used to support the deployment of emotionally expressive avatars in any virtual setting, and to customise the biosensor technology so as to facilitate a range of conditions.