Improving Driver Emotions with Affective Strategies

Abstract

Drivers in negative emotional states, such as anger or sadness, are prone to perform bad at driving, decreasing overall road safety for all road users. Recent advances in affective computing, however, allow for the detection of such states and give us tools to tackle the connected problems within automotive user interfaces. We see potential in building a system which reacts upon possibly dangerous driver states and influences the driver in order to drive more safely. We compare different interaction approaches for an affective automotive interface, namely Ambient Light, Visual Notification, a Voice Assistant, and an Empathic Assistant. Results of a simulator study with 60 participants (30 each with induced sadness/anger) indicate that an emotional voice assistant with the ability to empathize with the user is the most promising approach as it improves negative states best and is rated most positively. Qualitative data also shows that users prefer an empathic assistant but also resent potential paternalism. This leads us to suggest that digital assistants are a valuable platform to improve driver emotions in automotive environments and thereby enable safer driving.

1. Introduction

Affective automotive user interfaces show potential for an improvement of driver safety through the detection of performance-influencing states, such as anger, and their mitigation through strategic interaction [1]. Thanks to recent advancements in psychophysiological computing and new technologies that allow for contactless sensing of human emotions, affective interfaces might soon become feasible in everyday environments [2]. We see automotive user interfaces as perfect pioneering grounds for user-aware systems, as drivers and passengers are confined in a limited space with high availability of sensors placed in the car interior.

In this work, we take the viability of psychophysiological sensing of emotions as a given prerequisite for affective in-car systems and focus on interaction techniques to mitigate the effects of negative emotional states. Unbridled driver emotions have been shown to contribute to unsafe driving [3] and are thus a prime use case for adaptive, user-aware interfaces in the car. We present strategies to counteract negative emotional states from related work and evaluate four approaches which we deem promising for an in-car application. Sixty participants experienced these interface concepts and provided us with insights on their effects on driving behavior, physiological data, and user experience. The presented study focusses on the effects of anger and sadness behind the wheel, as these are two extreme affective states a minimum viable product for automotive use will need to be able to take into account [4]. As core findings, we suggest natural voice interaction with subliminal cues (e.g., light) as suitable modalities for an interaction with positive outcome and recommend a strategic focus on empathy, without patronizing the user. The goal of this work is to help designers of in-car user interfaces to chose the right strategies when reacting to negative driver emotions.

2. Related Work

Previous research has by and large acquired knowledge on the fundamentals of automotive user interfaces and affective computing, which we aggregate in the following section. The presented influencing strategies from related studies are clustered according to their elementary operating principles and modalities, and eventually served as inspiration for the concepts we conceive.

2.1. Automotive User Interfaces

Interaction with in-car user interfaces is usually performed as a secondary task [5] in addition to the more important, highly safety-critical main task of driving. Automotive UIs are therefore benchmarked towards their effects on distraction, driving performance, and the workload they induce on the driver [6]. Modern automotive UIs aim for a natural experience with multimodal input channels [7], they can include persuasive abilities, for example, to inspire economic driving behavior [8] or discourage speeding [9], and they can be adaptive to the driver state, e.g., by taking into account the induced cognitive load and adapt the displayed information [10], or by detecting fatigue through psychophysiological measures and adapting air condition settings to vitalize the driver [11]. The assessment of emotional states provides another perspective towards adaptive UIs, which constitutes the starting point for this work.

2.2. Emotions on the Road

Emotions occur in every conceivable situation, so naturally also during driving. The affective state of the driver is influenced by environmental (roads, weather), situational (traffic), and interactional factors (conversations, user interfaces) and thus fluctuates continuously [12]. Especially actions caused by negative emotions, like anger or sadness, have been shown to impair driving performance in a negative way [3,13]. In the course of the preceding research, we came to know that drivers experiencing negative emotions would approve a system which would help them to improve their emotional state [4]. Although this idea has been proposed before, e.g., by Nass et al. [14], most driver models only take into account cognitive load, distraction, and the physical state of the driver but do not consider emotions and their triggers [15]. For adaptive, affect-integrated interaction to become reality, the software behind automotive UIs needs to have a more holistic view of the driver which includes their long-term traits and preferences, as well as short-term states, such as emotions [1,16].

2.3. Affective Interfaces

Affective states have been shown to bias user actions and cognitive processes [17], which is why they can have effects on road safety [1]. In order to understand these emotional states, they are often classified within categories of basic emotions [17] or on continuous scales with the dimensions arousal and valence [18]. In the driving context, medium activation is seen as optimal level of arousal [19], and positive valence is generally desired as a sign of good user experience [16] (see Figure 1). The motivation behind affective interfaces is to sense, interpret, and respond appropriately to human emotions [20] with the aim of positively influencing user behavior, which in the context of driving means a prospective improvement of road safety [21].

An affective in-car system must, first of all, be aware of the driver’s state [22], which is comprised, e.g., of physical features, mental workload, fatigue, the level of distraction, and emotions [16,23]. Psychophysiological measures like heart rate variability, skin conductance, skin temperature, breathing rate, or EEG can be used to deduce driver states [24,25,26,27,28]. At the same time, unobtrusive contactless technologies like audio-visual sensors or eye-tracking might have lower initial hurdles regarding acceptance in cars: a video stream of the driver can be used to extract facial action coding units, which in combination with voice features can inform an emotion detection system [29,30], and measures of pupil diameter can be used to assess mental workload [31]. An in-car environment provides a great starting point for such systems, as users are confined to a limited space and all kinds of sensors are highly common in the interior of a modern car and widely accepted by users.

2.4. Designing Affective In-Car Assistants

Nass et al. were first to investigate an assistant which can adapt to the user’s emotion [14]. They report fewer accidents, better attention, and higher willingness to communicate when the system’s voice mirrored the driver’s emotion. In a follow-up study, their team also found that negative effects on driving performance from frustrating events can be mitigated through positive voice interaction [32]. Hoedemaeker and Neerincx present an in-car interface which adapts its informational content based on the detected cognitive load of the driver [10]. Williams et al. explored another facet of emotional interaction in the car with a social robot, which also had positive effects on driving performance [33]. We take this aspect of adaptive and social interaction and connect it to an affect-integrated driver model in order to intervene within the driver–vehicle interaction loop [1,16,34]. Figure 1 shows the driver state taxonomy used in many related studies, which defines dangerous states with extreme values of arousal in combination with negative valence (e.g., anger and sadness) and identifies medium arousal and positive valence as optimal driving state. In the following, we outline possible approaches to affective interaction in the car, which we suppose to differ regarding efficiency and acceptance [8].

2.4.1. Incentivization

A classic behavior regulation strategy working with incentives and punishment, also known as the “carrot and stick” method, has been suggested by Wang et al. [35]. In their prototype, road users can give direct feedback to other drivers by showing appreciative or disapproving external display contents. On a practical level, however, positive as well as negative reinforcement can have adverse long-term effects on social interactions, and especially the punishment of unwanted behavior is seen as highly undesirable by users [36].

2.4.2. Distraction

In the automotive domain, distraction from the driving task is generally advised to be avoided. Drivers can, however, be distracted from negative affective states as long as the stimuli do not influence driving safety. Previous work has shown concepts to tackle negative states by changing routing options towards routes with a better view [33,37]. Another widespread concept to distract drivers from negative emotions is using adaptive music playback in order to nudge them towards more positive feelings [38,39,40,41].

2.4.3. Driver State Display

The idea to display the detected driver states within a graphical UI has been evaluated in recent studies, e.g., by Völkel et al. [42], within the AwareCar project at MIT [23], and in our own research [43]. Such a display can keep drivers informed regarding their current capacities and support in-vehicle systems to improve driver performance and safety [23]. Drivers generally seem open towards continuous driver state displays, provided the criticality of the situation is easily discernible [42,43]. If such a system incorporates notifications, the timing needs to be adaptive to the level of receptiveness and the driver’s attentional states [44]. Williams et al. also suggest to communicate this information to other road users in order to improve driving safety [33].

2.4.4. Gamification

Several concepts by Schroeter and Steinberger introduced game design aspects to automotive user interfaces [45,46,47]. They use gamification to trigger engaging tasks for distracted drivers, to give them feedback on their driving performance, and to increase situational awareness with proactive markers. The resulting re-engagement and better hedonic qualities of these systems can, however, come accompanied with increased visual distraction [46].

2.4.5. Affective Symmetry

Systems which mimic empathy by reflecting the detected driver state might possibly be accepted well by users due to similarity attraction theory [48]. Previous work in the automotive domain has found according indications for visual avatars [26] as well as voice assistants [14] and social robots [33]. Studies report positive effects on emotions and driving performance when the system matches the driver’s state [14,21,33].

2.4.6. Reappraisal

After facing negative emotional events, affective states can be regulated through reappraisal of the experienced situation [49]. In the context of digital systems, this means a system provides another perspective with a less negative outlook for the user. Harris and Nass have shown evidence for positive effects on emotions and driving performance for reappraisal in frustrating driving situations [32].

2.4.7. Subliminal Influencing

In contrast to above described explicit strategies, emotions can also be influenced with subliminal cues. Temperature control has been shown to mitigate effects of high arousal [11,38] and lighting can also have an influence on affective states [50]. In particular, blue light is shown in related work to have a calming effect [51,52].

2.4.8. Summary

Each of these strategies incorporates a driver–vehicle interaction loop, meaning the system senses driver data, estimates an emotional state, and reacts accordingly [34]. Different modalities with varying degrees of blatancy are used to influence the driver towards an optimal driving state, which harbours the risk of introducing side effects such as distraction or perceived paternalism [53]. For a realization of such affective strategies in the car, their potential impact on the driver needs to be assessed, so we can exclude unfit approaches and advance more promising concepts.

2.5. Research Question

Affective automotive user interfaces follow the goal of detecting unsafe driver states and influencing the driver so that they can restore safe driving performance. Based on previous research, we assume an in-car system can influence the emotional driver state through interaction. The outstanding question we answer with this work is whether any strategy or modality is to be preferred in order to improve the driver’s negative emotions.

3. Concept

The wealth of different strategies named above spurred us to consult a specialist focus group with the goal of distilling the most promising approaches for a future concept. Twelve UI/UX researchers of the BMW Group (two designers, three human factors engineers, three psychologists, four computer scientists) took part in a one hour session where we introduced examples from related work we had compiled beforehand and discussed applicable study designs and which emotional states should be chosen as starting points. They also reviewed ethical and practical implications of the introduced strategies.

In summary, incentives and punishment were ruled out as they might have too many possible side effects, and gamification was seen as a fitting approach for automated vehicles but offered too much distracting elements for manual driving. Affective symmetry was discussed in detail as some participants thought it might intensify negative emotions but also likely increase acceptance of the system. Subliminal cues were also evaluated as promising; however, they were seen as harder to assess than explicit stimuli because users might not perceive the influence under certain conditions. In the end, the focus group agreed on four interaction strategies which are to be tested using a within-subject design against a baseline UI which does not offer any influencing factors. We considered two distinct emotional states, Anger and Sadness, as reasonable between-subject groups. Four concepts result from combinations of the discussed strategies and are defined as follows:

3.1. Ambient Light

The strategy Ambient Light is realized using a controllable LED strip attached to the lower edge of the dashboard, so the light would be perceived in the driver’s peripheral field of view. The subliminal light cue would fade in slowly when activated and the background of the central information display (CID) would also change its color tone. Participants in the group Anger receive a purple-blue light to calm them down, while users in the group Sadness see a soft green-yellow light. These color schemes were derived from work on the effects of color on emotions by Valdez and Mehrabian [51].

3.2. Visual Notification

The main idea for this strategy is to give the driver a graphical visualization of their current state without active interaction. After discussing the display of physiological measures, we ultimately decided to abstract these values into a simple emoticon representation (cf. Figure 2). Detected anger is represented by an angry face and sadness is shown with a sad face. These graphical representations are displayed as full-screen notifications on the CID during the influencing phase of the experiment.

3.3. Voice Assistant

This strategy incorporates a proactive spoken dialog when the user is found to be in a negative affective state. The voice samples were recorded beforehand by a male voice actor. Participants are addressed with the recommendation “I detect that you are distracted. Would you like to listen to some radio to concentrate better?” to which they could answer via speech or touch screen input. If the recommendation is accepted, a sample from a local radio station is played back (short host commentary and an emotionally neutral song).

3.4. Empathic Assistant

This approach is based on the previous strategy but enhanced with empathic interaction. The voice assistant addresses participants in the Anger group with “Hey, are you alright? I can understand that you are a bit angry, sometimes I feel the same way. How about some music to take your mind off things?” and the ones in the Sadness group with the phrase “Hey, are you alright? I can see that you are lost in thought, I feel the same way sometimes. How about some music to cheer you up a bit?”. The played radio sample differed from the one used in strategy Voice Assistant but was also held as emotionally neutral as possible.

4. User Study

We performed a simulator study (n = 60) to investigate whether negative driver emotions can be improved through affective interaction. The primary goal of this study was to identify promising approaches for affective computing in the car and their effects on driving performance, user experience, and workload.

4.1. Hypotheses

We expect positive effects on safety and user experience if a system can influence the driver’s emotional state. We also assume that not all strategies we designed are equally well suited for the driving context. The hypotheses we derived from our research and previous work can be summarized as follows:

Hypothesis 1 (H1).

Driving performance is impaired by negative emotions, which can be mitigated through interaction [4,14,15].

Hypothesis 2 (H2).

A decrease in negative emotions results in a change of cognitive load and physiological signals [24,25,26,27,28].

Hypothesis 3 (H3).

The tested strategies differ regarding their effect on the driver’s emotional state (cf. Section 3).

Hypothesis 4 (H4).

Emotional states differently influence preferences for interaction and their effects on the driver [3,13].

Hypothesis 5 (H5).

Users prefer subliminal influencing over direct interaction as it is less patronizing [44].

4.2. Study Design

The study was designed with the goal of comparing the effects of different UI strategies on driver emotions while driving. As independent variables, participants experienced the four strategies introduced above plus a Baseline interface without interaction. The sequence of strategies was permuted using a $5\times 5$ latin square. Subjects were assigned to one of two groups of induced emotions, either Sadness or Anger, resulting in a mixed (within-between-subject) design. Emotions were induced using autobiographical recollection. This method requires the participant to think of and write down an experience from their past which had elicited the desired emotion. In order to recall this emotion while driving, participants have to recount the the story aloud, preferably in a setting protecting their privacy. Autobiographical recollection has been identified as the effective emotion elicitation method in previous research [54,55].

We collected dependent measures on driving performance, eye glances, the driver’s workload and emotional state, as well as subjective feedback. Driving performance was quantified using the car’s position within the simulation to calculate headway variability and the standard deviation of lane position (SDLP) [56]. The driver’s workload was assessed after each ride using the driving activity load index (DALI) questionnaire [57]. Before and after every concept interaction, participants also gave an estimation on the current strength of the induced emotion on a one-item scale. Physiological sensors provided data streams on galvanic skin response (GSR) and heart rate, an eye-tracker recorded the driver’s gaze, and a driver camera allowed the analysis of emotional states from facial expressions. In the end, subjects were asked to rank the experienced strategies according to their general preference, and to explain their feelings towards each strategy in a short unstructured interview.

4.3. Participants

Sixty-three participants aged 22–61 years (M = 31.58, SD = 10.10) took part in the study. After discarding three datasets due to missing or corrupted measures, we take into account n = 60 (42 male, 18 female), of whom 30 participants each experienced induced Anger or Sadness respectively. The majority of participants was familiar with graphical and voice-activated automotive UIs (57), and had never before interacted with a system which considered their emotional state (51).

4.4. Apparatus

The study was conducted in a high-fidelity driving simulator at the BMW driving simulation center. Figure 3 shows the basic setup, consisting of a fixed-base BMW 5-series (F10) mockup in front of a curved 180° screen. The driving simulation was realized with BMW’s proprietary simulation software SPIDER [58] and projected by five high lumen projectors. The visual components of the concepts were presented on a Microsoft Surface tablet situated at the original position of the central information display, the ambient light was realized using a Philips Hue LightStrip attached to the lower edge of the dashboard, and auditory parts were played via the car’s speaker system. The concepts and simulation were controlled from a separated experimenter room with video and audio surveillance and an audio backchannel.

Physiological readings to analyze the emotional state of the driver were collected using the iMotions research platform [59] in combination with a Shimmer 3 sensor [60] measuring heart rate and galvanic skin response through adhesive electrodes. An instance of the Emotient FACET software [61] was also connected to iMotions to detect emotional states from facial expressions and a Tobii X2-60 [62] served as eye-tracking device.

4.5. Procedure

At the beginning of the experiment, we explained the study goal of comparing affective interfaces and the following procedure. Participants filled out a short questionnaire on demographics and a consent form, during which the experimenter assigned them to an experiment group based on their age and gender. We did this to ensure balanced distributions within the two between groups and the five latin square permutations. Participants then sat down inside the car to adjust the seat and get accustomed to the driving task during a generous familiarization ride. The driving task required participants to follow a white car in moderate highway traffic.

The first part of the emotion elicitation task was again performed outside the car. Participants sat down at a table and wrote down a personal experience that made them feel very sad or angry, depending on the group they belonged to. They were given a time window of 10 minutes to fill their paper and remember every little detail about the experience. Then, they would be outfitted with adhesive electrodes for the physiological sensing and the eye tracker was subsequently calibrated when they were back inside the car.

At the beginning of every ride, participants absolved the second part of the autobiographical recollection task. They recounted the story they wrote down before to themselves for five minutes whilst driving. Their directive was to conjure up the emotional state they were in when they experienced the remembered events. During this elicitation phase, the audio channel to the simulator was disconnected for privacy reasons, but the experimenter could still observe the subject via video. After this, they were asked to rate the intensity of their emotion and after a short free ride they experienced one influencing strategy and another query of their emotional state after some time had passed. This was repeated five times, once for each concept and for the baseline. In the end, participants were asked to rank the strategies according to their personal perception and to justify their decision in a short unstructured interview.

4.6. Limitations

The study was conducted in a driving simulator due to safety reservations. We expect comparable outcomes as in a real driving context [63]. Participants were employees of the BMW Group; however, we exercised due care with recruiting to avoid a biased user sample.

5. Results

We present the results split by the between-subject groups Anger and Sadness. The within-subject variable Baseline serves as ground truth to compare the effects of each strategy. The statistical information given comes from an analysis of variance if not stated otherwise. Values for direct comparisons were calculated using a t-test.

5.1. Driving Performance

Drivers had to follow a vehicle and keep the distance throughout the ride, which they were generally able to do as headway variability was not notably different between strategies or emotions. The lane offset (SDLP) in contrast shows significant differences between strategies ($F(4,0.012)=3.810,p=0.005$) and groups ($F(1,0.040)=12.483$, p < 0.001). Figure 4 illustrates that drivers in the Anger group showed higher lateral deflection than drivers with induced Sadness. Concept Ambient Light also led to higher lateral deviations compared to Baseline ($p<0.001$). Deterioration of driving performance is common for interactions with in-vehicle UIs. However, only the maximum SDLP of $0.23\pm 0.07$ m for the strategy Ambient Light in the Anger condition is close to the threshold for unsafe straddling of $0.25$ m as defined by Brookhuis et al. [6].

5.2. Eye Tracking

Eye-tracking data provides no indication of increased visual activity for any strategy. However, we found that gaze heatmaps of participants in group Anger showed more agitated gaze behavior than those of participants in group Sadness. There is also tendency for a more downcast look with angry drivers (see Figure 4).

5.3. Driver Workload

The DALI questionnaire provides self-ratings for driver workload, which showed no differences between groups or strategies for mental and visual workload or induced stress. The mental workload, however, was rather high for all participants with values of $5.6\pm 2.1$ for group Anger and $5.4\pm 1.9$ for group Sadness (scale 0–9), which might stem from the induced emotional strain they experienced.

Auditory workload shows significant differences between strategies ($F(4,14.441)=4.905$, p < 0.001), with increased values for Empathic Assistant compared to Baseline (p = 0.022) and Ambient Light (p = 0.003), as illustrated in Figure 5. The ratings for pleasure of use show significant differences between strategies ($F(4,13.769)=2.469$, p = 0.045), with only $EmpathicAssistant$ scoring significantly better than $Baseline$ (p = 0.044).

5.4. Emotional Self-Assessment

Drivers rated the intensity of the induced emotional state before and after interacting with each concept. An ANOVA with repeated measures shows differences between measures ($F(1,264.007)=221.465,$ p < 0.001) and interaction effects between strategies ($F(4,3.582)=3.005$, p = 0.019). Statistically non-significant tendencies show that Visual Notification helped less to reduce sadness and anger than Baseline, while the Empathic Assistant led to the highest decrease of negative emotions (see Figure 6).

5.5. Emotion Recognition

Emotion recognition from facial expressions was used to analyse the driver’s valence during the ride. Figure 7 shows that there were no significant differences between strategies for the Anger group but more negative expressions than among the Sadness group ($F(1,670.30)=4.276,p=0.039$). Participants with induced sadness also showed a high variance for positive valence.

5.6. Physiological Measures

Besides emotion detection through video, we also collected physiological data during the rides. An analysis of galvanic skin response (GSR) peaks per minute shows significant differences between groups ($F(1,692.694)=33.083,p<0.001$) but not between strategies (see Figure 8). Measurements of heart rate provide no differences between strategies or groups.

5.7. Subjective Feedback

Participants provided a subjective ranking of the experienced strategies and gave further feedback in a final interview. The ranking (see Figure 9) shows significant differences between the four strategies (F_(3,9.460) = 17.358, p < 0.001). Baseline was excluded as it contains no interaction. There are no differences between emotion groups. In direct comparison, concept Visual Notification is placed significantly worse than the other concepts (p < 0.001). In the following, we provide short overviews of the feedback we received for each concept.

5.7.1. Ambient Light

In addition, 17/60 (28.3%) participants state that the light influenced their emotions positively, while 5 (8.3%) say the opposite. One-third perceived the light as disturbing. Many participants suggested that Ambient Light could be used to support one of the more interactive approaches to improve the overall experience.

5.7.2. Visual Notification

The only visual strategy in the test was the most criticized instance, as 26 participants (46.3%) see it as disturbing or stressful. In addition, 17/60 (28.3%) see no use for such a system at all, while 12/60 (20%) could imagine an improved implementation which gives continuous live feedback on the driver state. Other suggestions for improvement involve a way to control the display to mitigate felt paternalism.

5.7.3. Voice Assistant

Feedback by 23/60 participants (31.7%) draws this strategy as second to the Empathic Assistant. However, some (9/60, 15%) prefer the Voice Assistant. Eight participants (13%) find it distracting and seven (11.7%) would have preferred personalized music instead of the radio. Other issues voiced in the feedback were the computer voice and perceived paternalism.

5.7.4. Empathic Assistant

Furthermore, 19/60 participants specifically prefer this strategy over the Voice Assistant, 17 (28.3%) noticed no difference between the two, and 12 (20%) disliked the Empathic Assistant. Nine participants (15%) found this approach distracting, and general feedback also contained comments on the artificial computer voice and perceived paternalism.

6. Discussion

Our results suggest that not all modalities are equally suitable for an interaction with emotionally engaged drivers. We discuss the hypotheses from Section 4.1 and the most meaningful learnings we draw for future work.

6.1. Effects of Negative Emotions on Driving Performance

Participants in our study had no major problems to complete the primary driving task. SDLP data shows imperfect performance from sad drivers and significantly worse performance in the angry condition. However, no group showed dangerous swerving or accidents. Most influencing strategies did neither improve nor worsen the performance significantly. Only ambient light increased lane exceedance compared to the baseline. The measures of driving performance only show tendencies of negative effects of emotions which were previously found in other work. Our experiment design does not allow the scrutinization of this aspect, as a third group without emotional influences would have been needed. Therefore, we cannot confidently accept the hypothesis that driving performance is impaired by negative emotions and that this can be mitigated through interaction (H1). We assume that driving in real traffic offers more uncertainty and might expose the effects of negative affective states, and with that the efficiency of influencing strategies, more clearly.

6.2. Connections between Workload and Emotional State

We hypothesized that an improvement of the affective state also improves the experienced workload of the driver and affects physiological measures (H2). We cannot completely verify this assumption as the workload was consistent for all strategies, no matter whether they achieved influencing the driver or not. An analysis of physiological signals, however, shows changes for galvanic skin response and facial impressions which support a connection to the emotional state of the driver. We can explain these findings with the theoretical view that emotional state and workload are not directly connected and rather are two parameters to quantify the user’s current state. The concept Empathic Assistant, for example, caused an increase of auditory workload but managed to change physiological measures of affect into a more positive direction. This has implications for the evaluation of affective automotive user interfaces and can also explain why no differences in driving performance could be found although negative emotions were decreased: because the driver’s workload was elevated simultaneously.

6.3. Efficacy of Emotion Influencing Strategies

The initial question motivating this work was whether different strategies to mitigate emotions have different effects on the emotional state of the driver (H3). We can accept this claim as the Empathic Assistant turns out to be most effective in reducing negative emotions, both for sad and angry drivers. Measures of facial emotions as well as subjective pleasure and the comparison ranking show this strategy as a promising approach towards emotion improvement. Natural interaction has been a trend in automotive interfaces for some time and just as with social robots, a human touch to the interaction seems to allow a closer connection and thus more emotional influence on the driver. Hypothesis 4 assumes that the driver’s emotion is a distinguishing feature for interaction preferences and the efficacy of the system. We cannot see differences regarding preferred interaction methods between the groups. Drivers in the Sadness group, however, reacted more positively to voice interaction than those in the Anger group (see Figure 7).

6.4. Perception of Paternalism

Although paternalism has been named as an issue in interviews, participants did not prefer the subliminal concept Ambient Light over more explicit influences. Concept Visual Notification, however, was assessed as too patronizing and also ranked worse than all other concepts. We conclude that the proactive recommendations in concepts Voice Assistant and Empathic Assistant were not perceived as overly patronizing, although they actively intervened in the driver’s freedom of action. Thus, we dismiss H5, which states that users might prefer subliminal influencing over direct interaction due to perceived paternalism. Subjective feedback suggests using subliminal cues like ambient light in addition to explicit interactions to promote a positive atmosphere in the car.

7. Conclusions and Future Work

One take-away we can derive from the presented results is that a driver state model needs to distinguish between different manifestations of the driver’s current condition. Emotions and workload are distinct phenomena which can have disparate implications on safety and interact with each other. We propose a driver model consisting of a set of long-term traits, like personality or expertise, and short-term traits like emotions, physiology, or cognitive load [16].

Another key finding we want to emphasize is the efficacy of speech-based interfaces, invigorating the momentary trend of natural user interfaces and the “speech-first” movement within the automotive industry. Our concept Empathic Assistant might be a viable starting point for empathically enhanced voice assistants in the car. Subliminal methods like ambient light can be used to improve the overall atmosphere in the interior; however, potential distractions caused by the visual stimulus have to be controlled.

Our interpretation of a driver state display was not accepted by participants, which extends our understanding of continuous driver state displays insofar as negative representations might have to be avoided. If the UI gives feedback on the emotional state of the driver, it should do so in a more abstract way, giving the driver more control on interpretation and reaction. Participants also found fault with a perceived paternalism which most likely contributed to their aversion towards the Visual Notification concept, which needs to be prevented in future implementations.

Upcoming questions for affective in-car interfaces should focus on open questions like when to trigger interactions and whether proactive system behavior is subject to limitations regarding distraction or social acceptance. The underlying classification algorithms will be needed to adapt to user reactance and include more data points than currently envisioned. Can driving behavior, street conditions, the type of journey, or the company within the vehicle be decisive for a better driver model? How can we interact with groups of people and which social problems arise for affective interaction in a multi-user environment? In addition, how will these systems perform over time? We expect exciting research in this young field which might not come without disappointments and surprises, but we are sure that this research contributes to the big picture of driver safety.