A Review on Measuring Affect with Practical Sensors to Monitor Driver Behavior

Abstract

Using sensors to monitor signals produced by drivers is a way to help better understand how emotions contribute to unsafe driving habits. The need for intuitive machines that can interpret intentional and unintentional signals is imperative for our modern world. However, in complex human–machine work environments, many sensors will not work due to compatibility issues, noise, or practical constraints. This review focuses on practical sensors that have the potential to provide reliable monitoring and meaningful feedback to vehicle operators—such as drivers, train operators, pilots, astronauts—as well as being feasible for implementation and integration with existing work infrastructure. Such an affect-sensitive intelligent vehicle might sound an alarm if signals indicate the driver has become angry or stressed, take control of the vehicle if needed, and collaborate with other vehicles to build a stress map that improves roadway safety. Toward such vehicles, this paper provides a review of emerging sensor technologies for driver monitoring. In our research, we look at sensors used in affect detection. This insight is especially helpful for anyone challenged with accurately understanding affective information, like the autistic population. This paper also includes material on sensors and feedback for drivers from populations that may have special needs.

1. Introduction

Emotions and affective expressions play a critical role in decision-making, learning, and other cognitive functions, but current technology is, for the most part, incapable of taking our emotions into account. Affective computing, supported through practical sensors, provides a possible solution to this problem. Taking driving as a context, by monitoring and reacting to the emotions or underlying signals from drivers, affective computing enhances interactions between humans and technology, with the ultimate goal to improve safety. The vehicle can be equipped with an intelligent support system to monitor the driver, monitor the driver’s behaviors, provide feedback to the driver, and even take control of the vehicle if necessary.

For long-term use and adoption, sensors should be practical: they should not require the user to spend significant time activating the device, experience discomfort when using the device, or spend significant time maintaining the device. These non-intrusive sensors can take the form and function of skin-contact wearables that measure unintentional signals as well as surface-borne sensors that collect intentional signals from the driver. An example of a practical sensor might be a group of pressure sensors embedded into a vehicle’s surfaces, powered and monitored by the vehicle to detect a user’s interactions with seats, safety accessories, armrests and the steering wheel. In-vehicle sensors already face challenges because of acoustic noise, electromagnetic noise and compatibility issues related to integrating with a central processor; for instance, a scalp-mounted electroencephalograph (EEG) sensor would be challenging to apply in vehicles because of noise sensitivity and data-processing requirements. Beyond engineering considerations, it would be unrealistic to expect daily drivers to apply the EEG sensor before each trip because it would be a new behavior and a time-consuming departure from the driver’s routine in a world where it is already difficult to get people to wear seatbelts. Wearable sensing devices such as watches and eyeglasses that fit into a driver’s established routine are more practical. Video cameras, reviewed in 2016 by Fernández et al. [1], are practical in the sense that users do not need to activate or touch them, but cameras and microphones also introduce privacy concerns and produce high bandwidth data that requires processing. New soft and textile-embedded sensor formats are promising because they can be fitted to vehicle interiors, and because they can detect safety-relevant activity using body contact data that is not as personally identifiable as video and audio streams.

Figure 1 illustrates a pressure sensor for tracking a driver’s grip pressure in (a) a body-worn format, and (b) a vehicle surface format. Wearable sensors such as the glove in (a) are more practical for daily use than, for example, blood sampling to measure glucose [2] or cortisol levels, or neural implants to detect brain activity in animal studies. Such invasive sampling can validate conclusions drawn from proxy signals available at the body surface, but a sensor glove is better for daily wear from the user’s viewpoint. However, for this grip-tracking application, the driver would have to modify their behavior to put gloves on, would need to keep the gloves charged, and would need to initiate wireless communication between the gloves and the vehicle. For those reasons, the steering wheel format in Figure 1b is more practical than gloves for grip tracking. Measurements of other signals that vary with driver stress levels, such as pulse rate, skin surface temperature, and skin conductance, often rely on skin-to-sensor contact that cannot be guaranteed on vehicle surfaces—even a steering wheel, if the driver is wearing mittens. In this physiological-sensing realm, wearables are unparallelled. Previous reviews have carefully considered wearable sensors for driving safety [3], wearable sensors for emotion recognition [4,5], and combinations of wearable and in-car sensors for detecting driver drowsiness [6,7] and distraction [7,8].

The current review covers the recent (since 2000) state of the literature on sensors that monitor driving behaviors, including emotions experienced while driving, and sensors designed for non-driving contexts that can detect emotional and physiological states applicable to transportation safety. We distinguish wearable sensors from vehicle surface-borne sensors, and consider where each of these sensor types may find the most practical application for monitoring driver behavior, identifying a general trend of wearable sensors for physiological measurements and surface-borne sensors for driver–vehicle interactions. We also provide a summary of emerging sensor technologies to study affective states, discuss concerns for their practicality in a driving situation, and their potential to contribute to future research on driver safety.

1.1. Affective States and Affect Detection

We provide a brief description of the literature’s use of affective states and emotions in the context of affect detection, as well as recommendations for further reading in related research areas. The foundation of affective computing is informed by theory of emotion. This field of research seeks to develop “computational systems that recognize and respond to the affective states (e.g., moods and emotions) of the user” as described by Calvo and D’Mello in their 2010 article [9], which gives a comprehensive review on the overlap of emotion research and affect detection. There is a rich history on the definition of emotion, emotional expression, and emotional experience [10,11]. Picard’s work [12] instituting affective computing steers clear of addressing a definition of emotion directly, instead defining emotional experience and moving on to affect detection. Picard (1995) uses “sentic state, emotional state, and affective state interchangeably. These refer to your dynamic state when you experience an emotion. All you consciously perceive in such a state is referred to as your emotional experience” [12]. Affect detection is possible by way of a person revealing their emotional expression, through the motor system, or “sentic modulation” [12]. James individually [13], and later with Lange [14], provides a theory of emotion that links physiological changes in the sympathetic nervous system (SNS), a part of the autonomic nervous system (ANS), to emotional expressions [13,14,15]. Physiological-based affect detection leverages sensors to detect changes in a person’s SNS and ANS. The James-Lange theory of emotion is used in several studies of affective states: Calvo and D’Mello [9]; Ekman, Levenson, and Friesen [16]; Critchley et al. [17]; AlZoubi, D’Mello, and Calvo [18]; and Baker et al. [19]. For further reading, see also work by Smith and Lazarus [20], Darwin [21], and Dalgleish [22].

For specific definitions of emotions, methods and assessment tools used in each study may vary. When given, we will summarize the definition of an affective state, or methods, used in a specific study. For Lazarus (1993), emotions include anger, anxiety, fright, sadness, and happiness, among others; and this research also describes an overlap of stress and emotion [10]. Russell (2003) defines core affect to include a pleasure scale (happy, sad) as well as an arousal scale (fatigue, drowsiness, tense, alertness) [11]. The affective states of frustration, confusion, engaged concentration, delight, surprise, boredom, and neutral were examined by Baker et al. [19], and defined as follows:

“Frustration was defined (for participants) as dissatisfaction or annoyance. Confusion was defined as a noticeable lack of understanding, whereas engaged concentration was a state of interest that results from involvement in an activity. Delight was defined as a high degree of satisfaction. Surprise was defined as wonder or amazement, especially from the unexpected. Boredom was defined as being weary or restless due to lack of interest. Participants were given the option of making a neutral judgment to indicate a lack of distinguishable affect. Neutral was defined as no apparent emotion or feeling.”

In relation to driving, an exhaustive list of which emotions have the greatest influence is not fully known, as discussed in Section 6. To frame that exploration, Figure 2 takes Russell’s Affective Circumplex [23] and marks areas likely, but not yet fully studied, to represent concerns for driver safety; we list some current studies linking affect and safe/unsafe driving behaviors in

Table 1. Representative studies on the relationship between affect detection and driver behavior.

Affective State	Reference(s)	Effect on Driver Behavior
Anger and Anxiety (vs. contempt and fright)	Roidl et al. (2014) [32]	Higher driving speed, stronger acceleration, speed limit violation for a longer time
Stress	Westerman and Haigney (2000) [43]	Higher (self-reported) lapses, errors, and violations
Happiness and Calmness (vs. anger)	Steinhauser et al. (2018) [44]	Lower driving speed and speed variability, longer distance to lead car
Fatigue	Philip et al. (2005) [46]	More inappropriate line crossings
Drowsiness	Lee et al. (2016) [48]	More near-crash events and lane excursions

. Figure 2 should be considered a broad initial guideline for researchers considering the impact of emotions on driver behavior, but it should be used very cautiously as a definitive conclusion on which emotions are involved in driving and how they influence behavior. Therefore, the circumplex can be used to guide which sensors are appropriate to consider when monitoring drivers, based on the sensor’s history of studies relating it to certain affective information.

1.2. Methods

This review used Google Scholar to identify recent (since 2000) literature on driving-relevant affect detection using sensors that are compatible with vehicle environments. Keywords searched include driver behavior, driving safety, sensors, soft sensors, wearable sensors, and affect detection. The authors excluded results based on relevance to this review’s focus and redundancy of a topic covered by cited works. During the editorial process, reviewers suggested additional references. Ultimately, more than 110 references are cited for further reading.

2. Previous Work Relating Affect to Driver Behaviors/Physiological Signals

Endowing intelligent systems with an ability to understand implicit interaction cues, such as the person’s intention, attitude, and their likes and dislikes creates more meaningful and natural interactions between the human operator and the intelligent system [24]. However, current technology cannot seamlessly interpret emotions and affective states that convey implicit communication. Responding to emotions is integral to typical social interaction and increases ways humans and technology communicate. Additionally, human–machine interaction (HMI) that relies solely on explicit commands ignores the potential gain of implicit communication, which can be significant as evidenced from experimental psychology [25]. Affective computing provides a possible solution to this problem. To establish affect-sensitive HMI, the role and potential of implicit communication is important [26]. By monitoring and reacting to the emotions or underlying signals from users, affective computing enhances interactions between humans and technology. Dr. Picard’s book [27] established a springboard for affective computing. Recent advancements in this research area have moved toward wearables and other practical sensors, leveraged machine-learning analysis techniques, and expanded the range of application areas.

Aside from trait, personality, and other personal factors [28,29,30,31], traffic and environmental situations that contain certain appraisal factors (e.g., whether another driver was accountable) can lead to a driver’s development and experience of emotions [32]. Several representative examples, although not exhaustive, are outlined in Table 1. Emotions and the accompanying attributions of traffic situations create a motivational tendency to show certain behaviors [33,34]. Such behaviors, if dangerous, may lead to negative consequences and compromise one’s own safety and the safety of other road users [29,35,36]. For example, angry drivers tend to drive faster, commit more traffic violations, display hostile gestures, honk more frequently, and underestimate risky situations, as evidenced in questionnaire, simulator, and naturalistic driving studies [35,37,38,39]. These behaviors are considered aggressive and unsafe to other vehicles. Furthermore, individuals who scored higher on the Driving Anger Expression Index are 2.5 times more likely to damage their vehicles in anger and twice as likely to crash than individuals who scored lower on the Driving Anger Expression Index [40]. Other work [41] revealed that drowsiness had the largest impact on increased crash rates, more than other inattention scenarios; while stress was linked to minor crashes [42].

In the following, a brief summary of the methods or definitions used to study the affective states listed in Table 1 is given. Roidl et al. [32] used the Driving Anger Scale to measure anger, the State Trait Anxiety Inventory to measure anxiety, and a modified Geneva Emotion Wheel to measure contempt and fright. Westerman and Haigney [43] examined the Driver Behaviour Inventory and the Driver Behaviour Questionnaire to study stress. Steinhauser [44] studied happiness, calmness, and anger during driving through a combination of (1) asking participants to self-select and re-live a previously-experienced life event related to each emotion and (2) by playing music related to each emotion, as validated by Jefferies et al. [45]. No further definitions of the affective states were given to participants. Philip et al. [46] used Grandjean’s definition of fatigue [47] as “a gradual and cumulative process associated with a disinclination towards effort, eventually resulting in reduced performance efficiency.” Lee et al. [48] collected physiological measures of drowsiness and measures on the Johns Drowsiness Scale.

2.1. Measuring Affect Based on Physiological Signals

Previous research has shown that physiological signals could classify affective states induced by on-road driving with 97% accuracy [49], with heart rate and skin conductivity having the highest correlations with driver stress. Physiological signals are not appropriate indicators of emotion for every application. Respiration has been linked to being indicative of emotional states [50]. It is a slowly-changing signal that does not provide information in enough time to prevent a driving-related accident [51], but it may provide insight into the relationship between driver emotional response and behavior. Tracking multiple physiological signals was judged as a favorable approach in previous research [52,53,54], and should be examined in work that seeks to predict and respond to physiology-based changes in emotion.

2.2. Measuring Affect to Improve Driver–Vehicle Interactions

Aside from physiological signals, researchers have been examining the degree to which affect-sensitive driver interfaces can be used to infer and support a driver’s affective state, safety, and comfort [55]. The causal association between emotion and performance has long been documented. Drivers who are stressed or angry are more likely to exhibit unsafe and dangerous behaviors and violations [29,56,57,58]. Since the driving task heavily involves integrating visual information and coordinating motor responses, researchers have been exploring the use of other senses for the monitoring of driver’s affective state. For example, a speech-based emotion recognition system with an adaptive noise cancellation technique that filters out ambient noise from driving has shown promise in classifying positive, neutral and negative emotions [59]. Nass and colleagues examined whether characteristics of a vehicle voice can influence driver’s affective state and driving performance [60], and the results showed that when the driver’s emotion matched vehicle voice emotion, drivers had fewer accidents, attended more to the road, and spoke more to the vehicle. A recent article emphasized the importance of using natural driver-car communication to understand a driver’s affective state and needs as well as to provide a human-like assistance system [61]. This approach has the advantages of being adaptive to various driving situations, drivers’ propensities and coping strategies, and the uncertainty of traffic behaviors [62]. Recent findings suggest that, in addition to matching vehicle voice to driver’s affect, vehicle voice showing empathy via a voice assistant led to the largest improvement of negative emotions and was also positively perceived by angry and sad drivers [63].

3. Soft and Wearable Sensor Technologies Applicable to Monitoring Driver Behavior

This section reviews wearable sensors and soft surface-borne sensors that can measure some aspect of a driver’s affective state and provide data that could be used in the future to study possible improvements in driver safety. As discussed in the introduction, embedded and wearable sensors are practical formats for in-vehicle sensing. We divided the review into two branches: sensors that monitor largely-involuntary physiological signals, and sensors that monitor driver–vehicle interaction.

3.1. Sensor Technologies for Affect Detection Based on Physiological Signals

Physiological signals generated from the human body include brain electrical activity (electroencephalography, EEG), skin temperature, heart rate and other aspects of the heart’s electrical activity (electrocardiogram, ECG/EKG), eye blink rate, blood flow and oxygenation (SpO₂), muscle current (electromyography, EMG), skin conductance changes due to sweating (galvanic skin response, GSR, or electrodermal activity, EDA), and respiration rate and volume. Such signals are usually involuntary, except in the sense that muscle signals and respiration events can sometimes originate from intentional body motions or speech. Physiological signals have previously been investigated for emotion recognition [4,64]. In the latter study [64], GSR, skin temperature, and heart rate were collected with an armband wearable sensor that the authors suggested could work with drivers.

Driving-specific studies that use wearable physiological sensors to investigate a safety-relevant emotional response include wired GSR and heart rate sensors measuring stress in a street driving environment [65]. Even though vehicles introduce electronic and acoustic noise, a seated driver produced fewer motion artifacts in the GSR and heart rate data than in related studies on ambulatory subjects.

More recently, GSR, SpO₂, respiration and ECG data were collected from drivers using wearable sensors with the goal of recognizing task difficulty-induced stress [66]. EMG sensors applied to subjects’ facial muscles detected facial expressions originating from anger in simulated driving tests [67]. Heart rate and skin conductance electrodes provided insight into stress in subjects taking a simulated driving test in a later study, with visible feedback on drivers’ stress levels provided by real-time data processing [68]. Like the Healey studies, these groups used stick-on ECG and EMG electrodes and other physiological sensors that attached directly to the body; data collection was wireless in newer reports. Such biomedical electrodes are useful for proof-of-concept studies and high-quality data collection for a fundamental understanding of the relationship between physiological signals and emotions experienced while driving.

However, armband [69,70] and eyeglass-based [71] sensors are more practical than ECG electrodes for widespread use, because they are fast to apply and may already be part of a driver’s everyday routine. Researchers recently studied driver drowsiness using the infrared proximity sensor built into Google Glass eyeglasses to measure blink rate using a thresholding algorithm [72] and determined that they were able to detect operator drowsiness.

Softer, stretchier electronic and optical materials have emerged over the past 10 years, making it possible to collect physiological data from textile-like surfaces and even from skin-contacting conformal sensors. A recent overview of this fast-moving research area [73] described applications in healthcare, consumer electronics, and robotics. Although no driving-specific sensors were mentioned, physiological signals commonly used to detect stress (for example, pulse rate) are measurable with soft materials, and new emotion-relevant applications like wearable sweat quantification and analysis [74] are now possible thanks to skin-conforming materials.

These sensors are lighter, more breathable and more comfortable than ECG electrodes, but most drivers are not yet accustomed to applying stickers or tape to their skin, plus powering the devices is still an early-stage technology that uses radiofrequency (RF) power transmission or thin-film batteries. Therefore, researchers are also moving physiological sensors to vehicle interior surfaces. In-vehicle sensors have successfully monitored drivers’ heart rates for detecting drowsiness [75], using electrically conductive fabric wrapped around the steering wheel. ECG electrodes on the steering wheel have also been studied for driver identification from biometric signatures [76,77]. Heart rate and respiratory rate sensors were embedded in vehicle seats based on piezoresistive textiles [78]. Soft, surface-embedded sensors measured physiological data in a vehicle seat in road tests [79] where, beyond detecting driver stress, the authors suggested the passive seat and steering wheel ECG could improve safety by detecting underlying heart conditions. A problem they addressed that is not present in skin-adhesive sensors was drivers’ failure to consistently grip the steering wheel; they proposed to fill data gaps with lower-resolution heart-rate data measured from redundant sensors in the vehicle seat. The missing hand itself could also indicate inattention, for example from texting or holding a cellphone. A recent study investigated the use of non-contact, capacitive coupled ECG embedded in the back support of a driver’s seat in a simulator to estimate driver’s fatigue [80]. Results indicated that there was good correlation between conventional ECG and cECG signals and that cECG signals had higher quality over time. Although this study only had male participants and used one type of clothing, it demonstrated feasibility for monitoring dynamics of heart rate variability using non-contact, more practical ECG methods.

Table 2. Wearable/in-vehicle physiological sensors and the connection between the sensed signal and affective state.

Physiological Sensor	Reference	Affective State(s) Sensed	Scope and Context (Driving Only)
Heart rate, galvanic skin response (GSR) wearable biomedical sensors	Healey and Picard (2005) [49]	Stress	Driving test on roads: 24 subjects on at least a 50 min route
GSR, SpO2, respiration, and electrocardiogram (ECG) wearable biomedical sensors	Ranjan Singh and Banerjee (2010) [66]	Fatigue, stress	Driving test on roads: 14 subjects including taxi drivers
Heart rate and GSR wearable biomedical sensors, plus wearable biofeedback using a visible indicator	MacLean et al. (2013) [68]	Stress, emotional regulation	Simulated driving test: 11 subjects with driving experience and no history of epilepsy or autism
Heart rate, GSR, and temperature from armband; Polar heart monitor chest strap	Nasoz et al. (2010) [69]	Fear, frustration, boredom	Simulated driving test: 41 subjects
Eye blink rate, from smart glasses-correlated with braking response time and lane deviation	He et al. (2017) [72]	Drowsiness	Simulated driving test: 23 subjects.
Heart rate variability, from ECG electrodes made from conductive fabric on steering wheel	Yu (2009) [75]	Fatigue, drowsiness	Simulated driving test: 2 subjects
Heart rate using ECG electrodes on body, and eye movement	Oliveira et al. (2018) [81]	Drowsiness	Driving test on roads: 20 subjects
GSR, SpO2, Respiration, and ECG embedded in vehicle seat	Baek et al. (2009) [79]	Task-induced stress	Driving test on roads: 4 subjects with at least 5 years driving experience
Facial electromyography (EMG)	Saikalis and Lee (2019) [67]	Anger	Simulated driving test: 11 subjects
Electroencephalography (EEG) and heart rate	Hassib, Braun, Pfleging, and Alt, (2019) [82]	Negative emotions induced by music	Simulated driving test: 12 subjects
Capacitive coupled ECG embedded in back support of vehicle seat	Bhardwaj and Balasubramanian (2019) [80]	Fatigue	Simulated driving test: 20 male subjects

compares the above-listed physiological studies.

3.2. Measuring User Activity Based on Driver–Vehicle Interaction

Besides involuntary physiological signals like those reviewed in Section 3.1, drivers interact with vehicle surfaces by gripping, tapping, leaning, and other hand or whole-body motions that give insight into their attention level and affective state. These body motions may be intentional, as in steering wheel motions made by a driver following a route, or unintentional, such as fidgeting. Safety is also improved if body position information helps plan airbag deployment during a collision.

Intentional motions for steering, braking, and acceleration are already collected by vehicle instrumentation, but body position is not. The following sensor technologies are able to capture body position and other driver-vehicle interactions based on proximity, pressure, and acceleration.

Table 3. Emerging sensor technologies that can capture driving-relevant user interaction signals.

Interaction Sensor Format	Reference	Interaction Category and Possible Affective State(s)	Context of Study (Driving/Other)
Wrist-worn accelerometry on both driving hands	Bi et al. (2017) [83]	Handling secondary tasks (texting, eating), distraction, drowsiness	Road driving tests with 6 subjects, 75 different trips
Capacitive proximity sensors in vehicle seats	Durgam and Sundaram (2019) [85]	Driver posture, sudden braking, panic	Other: validating occupant position in video vs sensor data
Capacitive proximity sensors in vehicle headrests	Ziraknejad et al. (2015) [86]	Head position, drowsiness	Other: validating head position detection in lab tests
Wearable capacitive pressure sensor	Pouryazdan et al. (2016) [87]	Fidgeting, inattention	Other: detecting restless leg motion
Stretchable optical strain sensors in athletic tape	Harnett et al. (2017) [88]	Muscle tension, stress	Other: detecting weight bearing activity in lab tests, proof of concept
Stretchable optical strain sensors in gloves	Leber et al. (2018) [89]	Hand motion, distraction	Other: detecting hand configuration in lab tests, proof of concept
Resistive textile pressure sensors in trousers	Skach et al. (2018) [91]	Body posture, social behavior, engagement	Other: Classification of 19 different postures and gestures, 36 subjects
Resistive foam pressure sensors in an office chair back and seat	Skach et al. (2017) [94]	Seated body position, social behavior, engagement	Other: Correlation of body position and speaking role during conversation, 27 subjects
Electrical impedance tomography touch detection on 3D objects	Zhang et al. (2017) [95]	Hand motion/grip shape, coordination level related to alertness	Other: User interface for computers, games, toys: demonstration
Wrist-worn electrical impedance tomography sensor	Zhang et al. (2016) [96]	Hand positions, excitement/hostility	Other: classifying hand gestures

covers such emerging wearable and vehicle surface-borne sensor technologies for measuring driver–vehicle interactions. Previous studies in this category often focus on activity recognition rather than emotion recognition, and, likely because these wearable and soft sensor materials are an emerging field, many of the papers summarized below and listed in Table 3 emphasize the new sensor technology itself rather than applications such as monitoring transportation activities. However, some recent papers do apply wearable and surface-borne user interaction sensors to driving. Researchers used accelerometer-equipped smart watches to track hand motion [83], making the connection to driver monitoring by correlating acceleration and gyroscope readings with non-steering secondary task motions during road-driving tests. Another group developed soft piezoresistive fabric steering wheel sensors, not for heart rate measurement as described above, but for detecting grip pressure, location, and swiping gestures [84].

In contrast to resistive pressure sensors which require direct contact, capacitive sensing can detect changes to electric fields extending above and around electrode surfaces. This feature makes capacitive sensing a good match for driver-vehicle interactions like head or torso position where the driver is not contacting the surface at all times. Capacitive proximity sensing has been applied to vehicle seats for detecting driver posture and possible sudden braking [85], and researchers investigated its feasibility for measuring driver head position [86], which is an indicator of drowsiness and a critical input for active restraint systems during a crash.

Emerging soft technologies are already monitoring subtle body motions in non-driving contexts using skin-like wearable sensors. For example, a wearable capacitive sensor was demonstrated to detect restless leg motion [87]. Soft, deformable optical materials made it possible to measure shape changes in a leg-worn athletic tape caused by weight bearing [88], and hand motions in a glove [89] equipped with all-polymer strain sensors. Body motions like fidgeting and slouching are also visible in images; Fernández et al. [1] gives a comprehensive overview of camera-based sensors for detecting motions relevant to driver fatigue and inattention. Soft optical sensors might complement or replace some types of image sensing, while their optical readouts might simplify wiring in vehicle applications where signals must be transmitted across a gap between moving parts. A recent study proposed methodology to standardize the processing of camera-based sensing data while taking into account individual differences, randomness in driver behaviors, and driver head motion tracking [90]. Another recent wearable sensor used resistive textile sensors embedded in trousers to classify body postures [91]; such sensors can capture shifts in weight that are difficult to pick up on camera, and can collect information related to a wearer’s focus, participation and engagement without video recording. Surface-borne sensors are contrasted with obtrusive video cameras for sensing human activity in smart environments [92].In recent office- and home-environment studies, researchers integrated thin resistive and capacitive sensors into soft surfaces for human activity recognition. An electronic textile couch was equipped with capacitive presence sensors, with a focus on sleep apnea intervention [93]. In another smart furniture experiment, observers recorded engagement, laughter, speaking and listening behaviors in seated subjects and correlated them with data from chairs fitted with resistive pressure-sensing pads [94]. These sensor formats are compatible with upholstered interior vehicle surfaces. The experiments generally determine pressure location by sampling a large array of sensor electrodes. Electrical impedance tomography can collect touch-location information with only a few (typically 8) electrodes using a scanning approach. Touchpads were created by painting surfaces with conductive paint on which consecutive resistance measurements were collected at pairs of electrodes along the edge of the conductive surface [95]. Electrical impedance tomography in a wrist-wearable format has also been used to classify hand gestures [96].

The key advantage of vehicle surface-borne sensors is that they are transparent to the user. In contrast to wearable textile sensors or wearable electronics where the user is acutely aware of the sensors, surface-borne sensors may collect body position or hand grip data without disturbing the user. Only recently it has become practical to measure body motions from these unobtrusive sensors instead of video or human observation methods. The link between this kind of sensor data and drivers’ affective states needs further clarification from the human-computer interaction community and comparison with questionnaires and physiological data.

4. Relationship between Driver Behavior and Roadway Safety

Making practical use of affect-sensitive sensor data to improve safety is a layered problem with solutions at the technology layer, the behavior modification layer, and the policy layer. In a study that merged physiological data (ECG heart monitoring) and vehicle data (speed, acceleration, fuel consumption, and pedal position), researchers went beyond characterizing individual driver behavior, suggesting that locations where multiple drivers experienced stress could help “map potentially dangerous road segments and intersections” [97]. Such information on human factors can complement and help interpret speed and braking patterns already captured by road sensors and external surveillance cameras [98], and geometric road characteristics like curvature and elevation collected by mobile phone sensor data analysis with an eye toward adjusting speed limits on rural roads with horizontal curves [99]. These examples suggest that real-time crash prevention is not the only goal for in-vehicle and wearable driving sensors. For instance, patterns of human stress reactions to specific traffic conditions could effectively distill years of human driving experience into safer algorithms for self-driving cars. The benefits of these advanced sensors can also extend to pedestrians and passengers of vehicles without sensors in the form of road repairs, warning signs, and traffic re-routing.

A well-integrated monitoring and assistance system is likely to maximize the intended safety benefits while minimizing barriers to adoption. From a user’s perspective, having an intelligent system is only part of the solution, the other requirement is user’s acceptance, adoption, and cooperation. In their conceptual framework, Lee and See described the processes from receiving information on a display to calibrate trust in automation and to develop reliance on automation, and how this process is influenced by individual, organizational, cultural, and environmental contexts [100]. For affect-sensitive driver interfaces to efficiently monitor and support drivers, the sensors, technologies, interfaces, users, and the operating environment (vehicle itself and supportive infrastructure) should be designed jointly and as one whole system [101].

5. At-Risk Example Population: Drivers with Autism Spectrum Disorders

The literature shows that sensors used for affect detection do provide information relevant to monitoring driver behavior. However, most studies do not include drivers with autism. This population is covered as an at-risk example of drivers with special needs to consider. Specifically, individuals with autism have deficits in accurately expressing explicit cues of affect, making forms of affect detection that rely on generalized facial expressions or neurotypical vocal tones less reliable. Individuals with autism spectrum disorders (ASD), are not devoid of affective expressions [102,103,104], but their own understanding of them and interpretation by others is limited [105,106]. For example, a person with ASD might smile when actually in pain. Furthermore, facial feature interpretation can be challenging because of their difficulties in displaying the expected range of facial expressions [107,108] or mismatch with their vocal tone. Therefore, monitoring another communication signal can be very informative for detecting changes in emotions for this population. An affect-sensitive system that can interpret the changing emotions of a driver and react with useful and appropriate feedback could be transformative. This at-risk population may require more training hours than neurotypical individuals, before driving skills are acquired at a safe level. As a focus discussed in future directions, driving simulators will be targeted to develop the intervention infrastructure.

Autism rates are growing, and the challenges autism presents to daily life abound. Research suggests prevalence rates of autism have increased in the last four decades from 1 in 10,000 to an estimated 1 in 68 children and 1 in 42 boys, based on the latest CDC report [109]. Individuals with autism are characterized by having difficulties with social interaction and communication, and a tendency to fixate on limited interests and repetitive behaviors [110]. The symptoms can range in degree from mild to severe, which is why autism is a spectrum disorder and generally described as autism spectrum disorders, or ASD. Even though there is increasing research in technology-assisted autism intervention, there is a paucity of published studies that specifically address how to automatically detect and respond to affective states of individuals with ASD. Such ability could be critical given the importance of human affective information in human–technology interaction [27,111] and the significant impacts of the affective factors of children with ASD on the intervention practice [112,113,114].

People with autism do have changing physiological signals that indicate reactions to their experiences [102,103,104]. Detecting subtle markers of changes in emotions is important in autistic therapies. Trained therapists make their best interpretations but could be further assisted by advancements in affective computing. Previous work demonstrated that affect-sensitive closed-loop human–robot interaction improved performance and enhanced enjoyment for a small group of children with ASD [115]. Advancements in sensors and interpretation of signals between drivers with ASD and technology are needed. An intelligent driving simulator that can detect the affective states of a person with ASD and interact with him/her based on such perception could have a wide range of potential impacts. A clinician could use the history of the person’s affective information to analyze the effects of the intervention approach. With the record of the activities and the consequent emotional changes in the person with ASD, a driver training system could learn individual preferences and affective characteristics over time and thus could alter the manner in which it responds to the needs of different drivers with ASD.

6. Conclusions and Future Directions

For a full understanding of driver behavior and its relationship to safety, sensors must capture both unintentional physiological signals correlated with fatigue/stress/affective states, and voluntary interaction signals (for example, steering, braking, gripping) coming from the driver’s response to those states. The general pattern that emerged from our literature review in Section 3 is that: wearable, skin-contacting sensors are a practical means for successfully capturing unintentional physiological signals. Surface-borne sensors are more practical than wearables, as discussed in the glove-vs.-wheel example of Figure 1, but are more difficult to use for physiological sensing than for user activity recognition because unreliable skin contact adds noise to most physiological signals. Perhaps for this reason, wearables dominated the physiological sensors reviewed in Table 2, which was limited to driving applications only. Meanwhile, driver–vehicle interaction sensing can be successful with either a wearable or surface-borne approach. Table 3, our review of sensors that capture user-interaction signals, had a relatively even split between wearable and surface-borne sensors for detecting driver–vehicle interaction. Since these studies were so recent, we did not narrow the applications to a driving context. As these emerging user interaction sensor technologies mature, the practical advantages of in-vehicle surface sensors may give them an edge over wearables.

Figure 3 puts our review of affect detection (Table 1), physiological sensors (Table 2), and user-interaction sensors (Table 3) in context with the larger picture of driver behavior. A third category of sensors, vehicle data sensors, refers to braking, steering, acceleration and other mechanical signals available from vehicle computers. This sensor layer in the second row of the diagram is the link between driver behavior and possible safety interventions.

This review suggests utilizing implicit communication by analyzing affective information gathered from physiological data of a person during affect-sensitive interactions with an intelligent system. The intelligent system, such as an advanced driver training system, will take in processed physiological signals and apply an affective model which maps the signals to an affective state. Then that system will make decisions about altering the interaction to respond appropriately to the affective state. The intelligent system is trying to emulate the human ability to detect, interpret, and influence affective states. Although such systems will not be able to precisely define a user’s internal motivations, the information can be used as feedback to improve HMI and skill learning. Closed-loop interaction is achievable, after open-loop analysis to process the signals into samples of features and build affective models to relate feature samples to an affective state. The current climate of high acceptance of wearable electronics in daily life, data-driven solutions, and demand for more communication between humans and machines is ripe for advancements in affective computing. Teenagers may be willing adopters of such technology and could be the first generation to witness the future fruits of affective computing experiments, implemented on common computing devices during closed-loop interactions in everyday life.

Conducting a comprehensive study of physiological signal analysis during driving situations, with an open-ended broad list of emotions would be a useful next step. Such a study could be modeled after AlZoubi, D’Mello, and Calvo’s work on the exploration of computer-based learning situations [18]. This previous work collected data on which emotions are likely to occur in a learning activity. Participants were 27 adults that completed a learning module on a computer. Participants then watched their 45-min session again and gave labels to every 20 s of the experience. The affective states they could choose from included: boredom, confusion, curiosity, delight, flow/engagement, frustration, surprise, neutral (no affect), and an “other” category. This research provided important information on which emotions are more likely to be experienced in a learning situation. A similar study centered on a driving task would be of great benefit to quantify which emotions are more prevalent while driving. These insights can then guide the deployment of sensors and the integration of unintentional and intentional signals that will support driver monitoring, assistance, and intervention.

Additionally, future research should systematically compare the feasibility and efficacy of emerging surface-borne and other practical sensors in a driving context and investigate the potential for monitoring a driver’s affective state and implications for training and interventions. This work needs to be conducted in simulators as well as naturalistically with a focus on improving safety and well-being of the drivers. Practical applications should be envisioned beyond real-time intervention in individual vehicles. Large-scale statistics on drivers’ affective states in response to common driving situations could offer valuable training insights for driverless vehicle algorithms. Mapping stress and distraction could suggest better design rules for future roadways. Affective mapping might also pinpoint where to spend transportation funds on roadway modifications that improve safety not only for individual sensor-equipped vehicles, but also for cyclists, pedestrians, and drivers with autism who do not outwardly express affective states in the same manner as the majority of drivers. As we pointed out earlier, clinically-disadvantaged populations, such as individuals with autism spectrum disorders [52,104,108], can especially benefit from tailored in-vehicle intelligent systems that monitor vehicle control behaviors and the underlying physiological states. Practical wearable and surface-borne sensors coupled with already available vehicle data sensors provide the means to connect affect detection to driver, vehicle, and transportation system interventions.