Next Article in Journal
Kernel-Based Ensemble Learning in Python
Next Article in Special Issue
Information Needs and Visual Attention during Urban, Highly Automated Driving—An Investigation of Potential Influencing Factors
Previous Article in Journal
A Big Data Analytics Approach for the Development of Advanced Cardiology Applications
Previous Article in Special Issue
Does Information on Automated Driving Functions and the Way of Presenting It before Activation Influence Users’ Behavior and Perception of the System?
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Information
Volume 11
Issue 2
10.3390/info11020061
information-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

From HMI to HMIs: Towards an HMI Framework for Automated Driving

by
Klaus Bengler
1,*,
Michael Rettenmaier
1,
Nicole Fritz
2 and
Alexander Feierle
1
1
Chair of Ergonomics, Technical University of Munich, Boltzmannstr. 15, 85748 Garching, Germany
2
Robert Bosch GmbH, Corporate Research, Robert-Bosch-Campus 1, 71272 Renningen, Germany
*
Author to whom correspondence should be addressed.
Information 2020, 11(2), 61; https://doi.org/10.3390/info11020061
Submission received: 15 December 2019 / Revised: 10 January 2020 / Accepted: 20 January 2020 / Published: 25 January 2020
(This article belongs to the Special Issue Automotive User Interfaces and Interactions in Automated Driving)
Browse Figures
Versions Notes

Abstract

:
During automated driving, there is a need for interaction between the automated vehicle (AV) and the passengers inside the vehicle and between the AV and the surrounding road users outside of the car. For this purpose, different types of human machine interfaces (HMIs) are implemented. This paper introduces an HMI framework and describes the different HMI types and the factors influencing their selection and content. The relationship between these HMI types and their influencing factors is also presented in the framework. Moreover, the interrelations of the HMI types are analyzed. Furthermore, we describe how the framework can be used in academia and industry to coordinate research and development activities. With the help of the HMI framework, we identify research gaps in the field of HMI for automated driving to be explored in the future.

1. Introduction

For decades, researchers and engineers have developed human machine interfaces (HMIs) to ensure and to emend appropriate interaction between conventional vehicles and their drivers. A user-centered approach became inevitable in the development of concepts for human machine interaction. An automotive HMI consists mainly of output channels that provide information about the system state to the driver (e.g., via displays and auditory signals), input channels to receive the driver’s input (e.g., via buttons, steering wheel, and pedals), and a dialog logic to specify the relationships among input, output, and context parameters.
The ongoing implementation of advanced driver assistance systems (ADAS) towards higher levels of automation, on the one hand, offers the possibility of creating new ways to communicate and therefore changes the interaction between driver, vehicle, and the surrounding traffic system. Further development of sensor technology increases its detection range and enables the automated vehicle (AV) to create an awareness of its surroundings including other human road users (HRU) such as pedestrians, cyclists, and drivers of conventional vehicles. Moreover, the ongoing development of automation accompanies the AV’s growing responsibility of the driving task. These facts enable the AV to use new types of HMIs such as communicating via the vehicle dynamics, informing other HRUvia external interfaces, or providing guidance to the AV’s passengers.
On the other hand, the introduction of AVs into traffic entails the necessity of additional innovative communication strategies towards the AV user. From SAE Level 3 and higher, the AV’s “driver” becomes a passenger [1]. Consequently, automation gives the passenger access to new opportunities such as engaging in non-driving related activities (NDRA) which results in less attention to the driving scene. Thus, there are new aspects that have to be designed and have been investigated concerning in-vehicle communication, for example, a request to intervene, passenger’s information need, and NDRA. In parallel to the development of ADAS and automated driving functions, in-vehicle information systems and infotainment systems increase their functionality and offer an unimaginable amount of information and potential interactions.
Moreover, a possibly distracted passenger is no longer available for established communication with surrounding HRU. Thus, it becomes necessary to investigate not only the AV-passenger interaction, but also communication strategies with surrounding traffic. At present, different HRU communicate implicitly via the trajectory of the car (e.g., maintain speed, accelerate, and decelerate) and explicitly via flashing lights or using hand gestures [2]. In the context of automated driving, communication strategies could be derived from the mentioned approaches of human-to-human communication. AVs could communicate their intention implicitly via the trajectory or explicitly via external HMIs.
To introduce successfully AVs in traffic, communication with the passenger, as well as with other HRU, is essential. The communication strategies of different HMIs integrated in one single vehicle must not only perform individually but also in combination and synchronization with each other. Therefore, development is progressing from an HMI to HMIs.

2. Motivation

Originally, the development of different HMIs was located in different departments of a company or spread over different companies if infotainment was integrated via, for example, terminal mode concepts such as Apple carplay or Mirrorlink. Additionally, in research, studies were designed to answer specific research questions, mainly focusing on one HMI aspect. We argue that it is going to be vital for successful communication and interaction to consider the synchronization and consistency of the different HMIs. Furthermore, an HMI needs to be activated in the right context and at the right time in order for it to successfully communicate the relevant information. This paper aims to introduce an HMI framework that can be used for the coordination of different research and development activities. The HMI framework should highlight the influence of different factors on the HMI selection and it should visualize the interrelations between the HMI types themselves. Previous approaches that addressed different interactive surfaces inside the vehicle [3] did not provide a holistic understanding of internally and externally communicating HMI solutions and their interrelations. The HMI types presented are defined in the ISO/TR 21,959 [4], whereby the interrelations between the different HMIs and a holistic view are discussed in this paper. This framework is motivated from an ergonomic perspective and is not intended to present a technical architecture for the implementation of automotive HMIs. Nevertheless, the basic structure and ergonomic requirements could serve as input for technical architectures.

3. HMI Framework

The HMI framework is pictured in Figure 1. It describes the interrelations between the influencing factors and the AV’s selection of HMI types and HMI content in a given situation. The influencing factors (upper part of Figure 1, see also Section 3.1) consist of the static infrastructure for the operation of the AV, the dynamic elements that change as the situation evolves, and the automation system with its actual capabilities. Moreover, the addressees of the AV’s communication belonging to the dynamic elements supplement the influencing factors. The lower part of the HMI framework (Figure 1) presents the five different HMI types (Section 3.2) that the AV possesses. We advocate for consistent terminology of these HMI types that are elaborated in the following sections.
Different HMIs are disjunct to each other and are clustered in the way they communicate with the addressees, either internally with the AV’s passenger(s) or externally with other road users, such as drivers of conventional vehicles, cyclists, or pedestrians. The term passenger in this publication refers to the occupant of a vehicle for automation levels 0 to 5 according to SAE [1], regardless of whether the passenger currently has a driving or supervising task.
The dynamic HMI (dHMI) is part of both external and internal communication. It communicates with the AV’s passenger and to surrounding HRU via the AV’s dynamics due to the influence of conventional driving dynamics or due to specifically designed trajectories. In addition to the dHMI, the vehicle HMI (vHMI), the infotainment HMI (iHMI), and the automation HMI (aHMI) complement the group of internally communicating HMIs. The vHMI shares information about the vehicle’s condition, whereas the iHMI offers additional interfaces for NDRA. The aHMI comprises the communication of all relevant information about the system status as well as current and future activities of the driving automation system (DAS). The external HMI (eHMI) is part of the external communication and communicates its information to surrounding HRU via interfaces at the AV’s surface.
The framework is structured as a circuit. The influencing factors affect the selection and the content of the HMI types, resulting in the use of the most appropriate HMI types in a given scenario. As a result, the AV’s communication influences the addressees’ behavior, and thereby ultimately the interaction between the AV and its surrounding. Moreover, in the process of driving the static infrastructure, the dynamic elements, as well as the capabilities of the automation system itself change over time. This in turn influences the suitability and the selection of the HMI types and their content.

3.1. Influencing Factors

The influencing factors contain all attributes in the respective scenario affecting how the AV’s communication strategy needs to be in a given situation to enable successful human-AV interaction. We adapted the influencing factors’ clustering from the specification methodology of the collaborative research project UR: BAN [5] and adapted by the project @City [6]. The influencing factors include static and dynamic elements of a scene, the automation system, and the AV’s interaction partners in the scenario.

3.1.1. Static Infrastructure

The first factor influencing the suitable communicational means is the static infrastructure. The static infrastructure consists of all elements in the scenario that remain constant over time. Among others, this includes the road type (geometry, speed limits) with possible lanes, entrances and exits, and visual constrictions either for the AV’s sensors or for the AV’s passengers or surrounding HRU. Additionally, a decisive factor is the vulnerable road users (VRU) infrastructure. Moreover, the prevailing traffic rules influence the interaction and communication needs between the AV and other HRU. Using the example of an urban road section, the mentioned components would be listed as essential static characteristics and attributes (see Figure 2).

3.1.2. Dynamic Elements

The dynamic elements in the scenario are all attributes that are not static but change over time, and thus affect the AV’s interaction strategy. This includes both high-dynamic elements that change their location within seconds and low-dynamic elements that change their location within minutes to hours. Figure 3 shows the enrichment of the static infrastructure with dynamic elements. In particular, the surrounding HRU such as pedestrians, cyclists, or drivers of conventional cars and the AV itself including the driving strategy and individual maneuvers (speed, acceleration, and position) of the aforementioned HRU belong to the high-dynamic elements. In addition, low-dynamic elements such as the weather and lighting conditions affect the efficiency of different HMI types especially in the case of external communication. Furthermore, present nonmoving objects, such as double parking vehicles and other road constrictions such as construction sites, can lead to visual constrictions, and thereby affect the selection of adequate HMI types and content. We classify these objects as low-dynamic elements, although they do not move in the given scenario, because they could dynamically change their location over time.

3.1.3. Automation System

The automation system strongly influences the selection and the content of the five HMI types. Common industrial standards (e.g., SAE J3016 [1]) classify automated systems according to their functional scope via different levels of automation. As the functional scope of the automation system increases, the role of the traditional driver changes from active operator (during manual driving), to passive monitoring (during partially automated driving), to passive passenger (during conditionally, highly, and fully automated driving) within a specific operational design domain (ODD). As the role of the passenger changes, the information need does, too [7]. Therefore, the AV’s present automation level is decisive for the type and amount of information presented to the passenger. If the AV is driving in partially automated mode (SAE Level 2), the passenger has to monitor the current driving situation and the automation system to be able to take over vehicle control if system limits are reached. The passenger, therefore, needs information about automation system status and maneuvers [8,9,10]. Whereas, if the AV is driving in conditionally, highly, or fully automated mode (SAE Levels 3, 4, and 5) the passenger could be involved in NDRA. Therefore, the passenger’s role differs between different levels of automation (driver vs. monitoring passenger vs. passive passenger) as does the passenger’s state (e.g., attention), which needs to be considered when selecting the content of internal interfaces. Furthermore, the level of automation could change within a scenario. This could be the case if functional limits are reached and the passenger has to take over vehicle control following an automated ride or if the vehicle switches between highly and partially automated driving because the vehicle cannot guarantee a system fallback in a given ODD. These transitions need to be designed adequately with consideration of the current state of the passenger to ensure an optimal level of mode awareness and readiness.
The functional scope of the automation system further determines the AV’s external communication needs and strategies with other HRU. With increasing functionality of the automation systems, the passenger becomes less involved in the actual driving task, therefore, driver-related cues that are available nowadays in human interaction in traffic (such as eye contact and hand gestures) would no longer be available and, instead, would have to be replaced via the AV’s HMIs.

3.1.4. Addressees

The addressees of the AV’s communication are part of the dynamic elements. They are subdivided into passengers, especially the person on the driver’s seat and surrounding HRU. Depending on whom the AV intends to communicate with, different interfaces are suitable. For internal communication processes the use of vHMI, iHMI, aHMI, and dHMI are appropriate, whereas to communicate with other HRU, external communication via dHMI and eHMI is suggested. The selection of the correct HMI type is also influenced by the dynamics of the addressees and the AV in a given scenario. For example, during low relative speed scenarios clear communication via dHMI can reach its limits. The same is true in scenarios where the AV stands still. Here, the usage of eHMI may be necessary to reach clear communication with other HRU.
Moreover, the personal factors of the passenger and the surrounding HRU affect the communication process. Personal factors include both long-term personal characteristics such as personality, gender, age, driving experience, and driving style, as well as short-term characteristics such as the current vigilance level and attention [5].

3.2. HMI Types

3.2.1. Automation HMI (aHMI)

The aHMI comprises all interfaces in the vehicle interior that enable the passenger to interact with the DAS. As the automation level rises the scope of the automation system increases, and therefore the role of the passenger changes (see Section 3.1.3). Therefore, the aHMI has to communicate the system status of the automation transparently to the passenger to guarantee mode awareness and allow safe, efficient, and comfortable transitions between automation levels.
Studies have shown differences in informational needs according to the automation level [7], traffic situations [11], and the individual passenger [12], as well as differences in the extent to which passengers show changes in driver state, for example, fatigue [13]. Therefore, the aHMI should adapt to the traffic context, passenger’s state, information needs, and the individual differences of the passenger to ensure sufficient trust, acceptance, and safety for the DAS.
The HMI-elements used in the aHMI can be subdivided into visual, auditory, and tactile elements. From haptic-tactile to auditory to visual, the information rate increases and the perception delay time also increases [14]. Therefore, visual HMI elements are used, in particular, for monitoring and communicating the state of the DAS, whereas haptic-tactile and auditory signals, such as tones, are mainly used for warnings. Especially for warnings (in the case of a request to intervene), multimodality seems to be advantageous, since the integration of several perception modalities can result in shorter reaction times and faster perception of a higher amount of information [15,16]. In addition, the redundant provision of information results in lower error rates in the operation of technical devices [16,17].
As a visual HMI element, the instrument cluster, in particular, is considered the primary interaction element for communicating the system state and further detailed information in automated driving [18]. The instrument cluster is used in almost all studies on automated driving (e.g., [8,19,20,21,22]). In addition, head-up displays have become established that can inform passengers in the primary field of view about system status, emerging situations, and maneuvers without eyes off road (e.g., [8,19,23,24,25]). Augmented reality head-up displays, as an extension of conventional head-up displays, have been used in automated driving because they visualize maneuvers, mark detected objects directly in the environment (e.g., [23,26,27]) and can support the driver in take-over situations (e.g., [28,29]). Due to their good peripheral perception, LED strips are particularly advantageous for communicating changes in the system status (transition between automation modes or to manual driving), and therefore are used, for example, in the windshield, to provide adequate mode and situation awareness (e.g., [8,21,30,31,32]). However, the monitor in the center console has also been used as part of the aHMI [19]. Although semantics have been used in auditory aHMI (e.g., [25,33]), the aHMI design has been characterized by warning tones that have been used in the case of a request to intervene (e.g., [30,34,35,36,37]) due to their fast perception. Haptic-tactile warnings during a request to intervene in automated driving, so far, have been implemented primarily by vibration of the driver’s seat (e.g., [33,38]). However, this use was not unimodal but was integrated into a multimodal HMI (e.g., [33,39]). Figure 4 shows the most common visual HMI-elements used in previous studies.
In future research, it is important to consider how the large number of different HMI elements can be combined into an overall concept so that the requirements for the various automation levels according to SAE and the role of the passenger can be met. A particular focus must be placed on automation levels 4 to 5, which have hardly been investigated to date. At the same time, the use of automated driving in series production can lead to long-term effects that require adaptation of the aHMI over time. In particular, an adaptation to the individual differences of the passengers could be necessary. Carsten et al. [40] has provided recommendations that should be considered in the design of aHMIs.

3.2.2. Vehicle HMI (vHMI)

In contrast to the aHMI, the vHMI represents the HMI in the vehicle interior without automation reference. It enables the information presentation of the vehicle’s condition and the interaction with its settings. For this purpose, information is communicated visually or auditorily. Auditory elements are warning tones (e.g., loss of tire pressure). As visual elements, control lights (e.g., low fuel capacity or engine malfunction) are located in the instrument cluster in the driver’s direct field of vision. Settings such as the air conditioning system are located in the center console so that the front passenger can also manipulate it [41]. New technologies such as gesture [42] and voice control [43] offer completely new possibilities for operating the vHMI. Several vHMI elements are legally required and need to fulfill minimum requirements.

3.2.3. Infotainment HMI (iHMI)

Passengers driving in higher automation levels (SAE Level 3 to 5), where continuous monitoring of the ADS is no longer necessary, can engage in NDRA [1]. The iHMI is intended to enable performing NDRA beyond the classic infotainment offering (radio and telephone) and to guarantee its safe performance. Studies by [44,45] have shown that passengers want to be engaged in auditory NDRA such as talking to passengers, listening to music, calling, as well as visual NDRA such as watching the surrounding environment, watching videos on a tablet, and using their phone (e.g., reading or writing) during automated driving. This selection of visual NDRAs is not only offered via the display in the center console, but portable devices such as smartphones and tablets would inevitably have to be introduced as interaction elements.
Visual NDRAs can compete for space and presentation resources with the aHMI if these are placed on the instrument cluster or the display in the center consol. At the same time, coordination between aHMI and iHMI must be ensured in such a way that the passenger is guided back comfortably and safely into the vehicle control loop when taking over driving the vehicle due to a request to intervene or a driver-initiated takeover, in order to ensure controllability. In particular, mobile devices that are brought into the vehicle have not yet been considered as HMI elements and are seen as a future challenge.

3.2.4. External HMI (eHMI)

External HMIs are interfaces installed on or projecting from the external surface of a vehicle. Indicators and brake lights are legally required realizations of eHMIs and highly standardized. Via the eHMI, the AV communicates with other HRU and provides information about the AV’s status or its behavior. External HMIs could be clustered in auditory eHMIs [46,47] and visual eHMIs. In the context of the electrification of transport especially in low speed scenarios, hybrid electric or pure electric vehicles emit substantially less noise than comparable vehicles with combustion engines [48]. Due to the missing engine sound, electric vehicles are less perceivable particularly for blind or visually impaired pedestrians [49]. Auditory eHMIs could help these people, since quieter electric AVs are detected later in approaching scenarios than vehicles with a combustion engines [48]. To counteract this fact, auditory eHMIs could have the potential to support communication between electric AVs and VRU. At present, visual eHMIs are almost exclusively investigated in research for AVs. Figure 5 structures visual eHMIs in four different categories, marking a vehicle as “driving automated” [50,51], light-strips [52,53,54] communicating via light patterns, displays [55,56,57,58,59] showing text or symbols, and laser projections [60,61,62,63] projecting the message of the eHMI on the street.
Depending on the eHMI type, different amounts of information and semantics can be transmitted to the recipient. Labelling a vehicle as “automated” via an eHMI provides the most general information content by communicating only the AV’s status. Light-strips execute light patterns, which for example communicate by running from one side of the light-strip to the other side, or by flashing. Projections and displays are able to communicate the highest density of information, as these interfaces can be designed rich in detail.
At present, there are many studies researching the communication of AVs with pedestrians and other road users via external HMIs. Communicating via an eHMI increases the pedestrians’ perceived safety [58,64,65] and the level of comfort [66] when crossing the street in case the AV shows its intention to yield via an eHMI. Moreover, eHMIs could reduce the time pedestrians need to cross the street in case the AV cedes the right of way [62]. Additionally, Rettenmaier et al. [56] investigated the interaction between AVs and manual car drivers at bottleneck scenarios. External HMIs have been used to negotiate the right of way at road constrictions increasing the traffic flow and safety [56]. Other applications are situations in which the AV is already stationary or moving slowly and can no longer communicate via its dynamics. One example is a deadlock scenario, which could be resolved by communicating externally via eHMIs [67].
In addition to the potential of eHMIs, there are still challenges to be explored and solved in the future. Currently, there are no defined standards or minimal requirements of eHMIs. Open questions need to be answered such as: “Which medium should eHMIs use?”, “Which color should eHMIs be?” [68,69], or “Where should eHMIs be mounted?” [70]. Moreover, researchers discuss whether eHMIs should inform other HRU about the AV’s intention or whether the automated system should call the surrounding traffic for action.

3.2.5. Dynamic HMI (dHMI)

Following the notion that vehicle dynamics are not merely functional but also a form of nonverbal communication among road users (e.g., [71,72]), the term dHMI describes the AV’s intended or unintended communication of information to passengers and surrounding HRU via vehicle dynamics. Vehicle dynamics include both vehicle translatory dynamics (e.g., speed, lateral, and longitudinal accelerations) and vehicle rotational dynamics (e.g., active pitch or role motions of the vehicle’s chassis). In automated driving, both forms of vehicle dynamics could be either intentionally designed to transport information to passengers and surrounding HRU, or they could be a result of functional driving maneuvers that are unintentionally understood to be communicative by HRU in a given context. The term dHMI covers both ways, and thereby emphasizes the importance of considering vehicle dynamics design for AVs.
Today, road users already commonly communicate via vehicle dynamics in traffic encounters. Human drivers use purposeful vehicle movement to communicate intent and to negotiate and cooperate with others (e.g., [72,73]). Observation studies investigating the communication behavior of road users in pedestrian crossing scenarios have found that vehicle movement is the most common form of communication [74,75]. Similar results have been found from observations at bottleneck scenarios where drivers use offensive and defensive driving strategies to communicate their intent to either go first or let the other driver pass [76,77]. Furthermore, surrounding HRU use a vehicle’s movement to make assumptions about the driver’s mood and character [71]. Video analyses of traffic encounters show that unexpected motion behavior in manual [72] or in partially automated driving [71] can lead to discomfort and misunderstandings among HRU. Therefore, to replace the human driving behavior in automated driving, a considerate dHMI has to be designed that covers the communicational aspect of vehicle movement and is meaningful to passengers and surrounding HRU by matching their prior experiences and expectations. The potential of the dHMI is currently investigated for both internal and external communication.
From the perspective of external communication, the AV needs to clearly communicate its intent to other HRU to achieve safe and comfortable interactions in mixed traffic encounters. Several experimental studies have investigated the potential of the dHMI for external communication during pedestrian road crossings and lane changing and merging. During pedestrian road crossing, vehicle movement provides sufficient information for pedestrians to decide whether it is save to cross [57,78,79,80]. Furthermore, different motion design parameters and their combination (e.g., speed, onset of deceleration, and deceleration rate) affect the time HRU need to recognize the AV’s intent to yield [81,82] and their gap acceptance [83]. During lane changing and merging, the AV needs to clearly communicate both its intent to change lanes and its willingness to cooperate when other drivers want to merge into the lane (if the AV is the lag vehicle). Research has shown that the perceived cooperation of an AV is strongly influenced by the driving pattern before the lane change (in addition to an early communication of the AV’s intent via the indicator) [84,85]. Considering release conditions, an efficient automated lane change maneuver is more accepted by other HRU than one that is too conservative [86]. The perception of the AV’s willingness to cooperate as a reaction to a lane change announcement by a human driver is affected by the reaction time of the AV and the amount of deceleration and speed reduction [87].
In addition to external communication, the dHMI can be a new channel to communicate the AV’s intent to the passenger. Research on lane change maneuvers in partially automated driving shows, that active pitch and role motions can be used to feedback information on the system’s state and behavior (e.g., upcoming maneuver) to the passenger [88]. A dHMI is less accepted if the driver is no longer responsible for supervising the system [89]. Therefore, the acceptance of the dHMI by the passengers could depend on their involvement in the driving task, and, therefore on the functional scope of the automated system.
In summary, the dHMI is a powerful interface in many different situations to communicate to both passengers and surrounding HRU. It is crucial to consider the dHMI when designing an AV’s interaction strategy because the AV’s driving dynamics can be communicative even if they were not intended to be by designers and engineers (see also [72]). Research should further investigate the potential of the dHMI and how it could be combined with other HMIs (e.g., considering matching of dHMI and an additional visual eHMI [90]) to achieve one holistic communication strategy. Furthermore, the interplay of driving parameters and personal factors of the passenger and surrounding HRU needs to be considered during dHMI design (e.g., age see [83]).

4. Effect of Influencing Factors on HMI-Interrelations

As the functionality of the automated system becomes more and more advanced, the human machine interaction becomes more complex. In high levels of automation, the AV will have to be capable of communicating to passengers and surrounding HRU at the same time. During the design of an AV’s communication strategy the effect of the influencing factors on the HMI types will have to be considered. In particular, adapting to rapid changes in the influencing factors during passing the scenario must be ensured. This interplay leads to changes in the HMI interrelations. Relevant interrelations are the concurrence for space, content transitions, practicability, consistency, and the chronological coordination. The five interrelations are represented by the connections between the different HMI types in Figure 1.

4.1. Concurrence for Space in the Interior

The concurrence for display space especially involves the in-vehicle communication strategy. The iHMI, the vHMI, and the aHMI use the same devices to some extent to communicate to the AV’s passenger. Depending on the automation level, the passenger’s information need varies. For example, during manual driving, the vHMI is more important than the aHMI since automated driving is not activated or available. When driving automated, the information of the aHMI concerning the capabilities and the maneuver of the AV becomes more important for the passenger. Since the display space is limited, an appropriate selection of relevant information must be made depending on the characteristic of the automation system.

4.2. Content Transitions

The second interrelation between the five HMI types concerns content transitions, which means that depending on the automation level the same information content changes from one HMI-type to another. An example is displaying the vehicle’s speed. During driving in automation level 0 the speed belongs to vHMI since it informs about a driver input. When activating the cruise control in automation level 1 or driving in automation level 2 and higher, displaying the speed switches from vHMI to aHMI since the vehicle’s speed is controlled by the automated system.

4.3. Feasibility

Feasibility is based on the fact that in some situations particular HMI types are no longer available, and therefore other HMI types would take on the communication. For example, in situations when the automated vehicle drives slowly (e.g., deadlock scenarios) the communication capabilities of the dHMI are limited, because the AV can no longer change longitudinal and lateral dynamics. Another example is when communication via lateral dynamics is not possible due to obstacles on the road. In these cases, the eHMI could have the potential to communicate externally with surrounding road users and it could replace the communication of the dHMI.

4.4. Consistency

The HMI framework and its components show that as compared to previous concepts, the important requirements of consistency have to be interpreted on a higher level of complexity. Assuming consistency of one HMI type means that information and dialog design is free from contradictions, the understanding in the framework is that contradicting messages over different HMI types have to also be avoided (see also [90]).

4.5. Chronological Coordination and Timing

The fifth interrelation between the HMI types concerns the chronological coordination of the five HMIs, as well as the time slots in which the HMI types have the capability to communicate to the passenger or to surrounding HRU. Since almost no research has been done analyzing the temporal coordination of internal and external communication as well as the coordination of dHMI and eHMI, there are still plenty of research gaps in the future.
Moreover, in addition to the chronological coordination of the different HMI types, it is essential to research the time slots in which the respective HMI has to communicate to ensure enough time to perceive and process the HMI’s message. The identified time slots are strongly linked with the coordination of the HMI types because the time slots affect and limit the sequence and coordination of displaying the HMIs.

5. Future Work

Current research and studies cited in this contribution show that HMI concepts and their evaluation are mainly focused on one single HMI type or the combination of two different types in combination with selected levels of automation. It seems that all HMI types should be synchronized to communicate simultaneously. This assumption has to be investigated in detail in the future. Cramer et al. [91] presented indications that aHMI and dHMI could benefit from a given amount of asynchronicity, presenting the dHMI information in advance to aHMI. In future work, the temporal dialog coordination and the HMIs’ synchronization have to be interpreted at a higher level of complexity. In addition, this coordination and synchronization could lead to increased complexity regarding body control modules (and vehicle bus communication), as different HMI types could be implemented on different on-board networks or modules. The strong dependency of the HMI types and the resulting additional coordination effort could increase demands on processing power and data exchange. Although this publication does not address technical feasibility, the holistic framework provides information on the interrelations that should be taken into account from an ergonomic perspective when using all HMI types in a future implementation.
In addition, it is necessary to investigate settings communicating via different HMI types and to include scenarios in which at least one communication fails and thus influences the communication of the other HMI types. This could mean that a passenger does not react to a request to intervene or a VRU misunderstands the AV’s communication. The consequences of these unpredictable changes of situations need to be communicated to the passenger and to other HRU.

6. Conclusions

This work provides insights about the different HMI types in AVs, clustered according to their belonging to internal and external communication. In addition to a detailed explanation of which information could be assigned to which HMI, this paper analyzes and summarizes factors that influence HMI type selection and HMI content. The HMI framework combines the HMI types, their influencing factors, and the relationship between both parts. Moreover, the interrelation between the different HMI elements due to changes in the influencing factors’ characteristics are presented. We created the framework to emphasize the importance of considering all HMI types when researching the interaction strategies of AVs with its passenger or surrounding HRU. Changing the design of a single interface could lead to an alteration of the influencing factors, which thereupon affects the HMI selection. Therefore, a holistic approach considering internal and external communication is indispensable when researching interaction strategies in the context of automated driving.

Author Contributions

Conceptualization, K.B., N.F., M.R., and A.F.; methodology, K.B., N.F., M.R., and A.F.; investigation, K.B., N.F., M.R., and A.F.; writing—original draft preparation, K.B., N.F., M.R., and A.F.; writing—review and editing, K.B., N.F., M.R., and A.F.; visualization, K.B., N.F., M.R., and A.F.; supervision, K.B.; and funding acquisition, K.B. All authors have read and agreed to the published version of the manuscript.

Funding

The German Federal Ministry of Economics and Energy funded this research within the project @City: Automated Cars and Intelligent Traffic in the City (grant number 19A17015B).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. SAE International. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles; J3016; SAE International: Warrendale, PA, USA, 2018. [Google Scholar]
  2. Fuest, T.; Sorokin, L.; Bellem, H.; Bengler, K. Taxonomy of Traffic Situations for the Interaction between Automated Vehicles and Human Road Users. Adv. Hum. Asp. Transp. 2018, 597, 708–719. [Google Scholar] [CrossRef]
  3. Strömberg, H.; Bligård, L.-O.; Karlsson, M. HMI of Autonomous Vehicles-More than Meets the Eye; Springer: Cham, Switzerland, 2019; Volume 823, pp. 359–368. [Google Scholar] [CrossRef]
  4. ISO. Road Vehicles: Human Performance and State in the Context of Automated Driving: Part 2–Considerations in Designing Experiments to Investigate Transition Processes; ISO/PRF TR 21959-2; ISO: Geneva, Switzerland, 2019. [Google Scholar]
  5. Graichen, M.; Nitsch, V.; Färber, B. A Meta-perspective on Research Activities in UR: BAN Human Factors in Traffic. In UR: BAN Human Factors in Traffic: Approaches for Safe, Efficient and Stress-Free Urban Traffic; Bengler, K., Drüke, J., Hoffmann, S., Manstetten, D., Neukum, A., Eds.; Springer Vieweg: Wiesbaden, Germany, 2018; pp. 29–46. ISBN 978-3-658-15418-9. [Google Scholar]
  6. @City. Automatisiertes Fahren in der Stadt. Available online: https://www.atcity-online.de/ (accessed on 11 December 2019).
  7. Beggiato, M.; Hartwich, F.; Schleinitz, K.; Krems, J.F.; Othersen, I.; Petermann-Stock, I. What Would Drivers Like to Know during Automated Driving? Information Needs at Different Levels of Automation; 7. Tagung Fahrerassistenz: Munich, Germany, 2015. [Google Scholar]
  8. Othersen, I. Vom Fahrer zum Denker und Teilzeitlenker: Einflussfaktoren und Gestaltungsmerkmale nutzerorientierter Interaktionskonzepte für die Überwachungsaufgabe des Fahrers im teilautomatisierten Modus; AutoUni–Schriftenreihe; Springer Fachmedien Wiesbaden: Wiesbaden, Germany, 2016. [Google Scholar] [CrossRef]
  9. Drüke, J.; Semmler, C.; Bendewald, L. The “HMI tool kit” as a Strategy for the Systematic Derivation of User-Oriented HMI Concepts of Driver Assistance Systems in Urban Areas. In UR: BAN Human Factors in Traffic; Bengler, K., Drüke, J., Hoffmann, S., Manstetten, D., Neukum, A., Eds.; Springer Fachmedien Wiesbaden: Wiesbaden, Germany, 2018; pp. 53–74. [Google Scholar]
  10. Van den Beukel, A.P.; van der Voort, M.C. Design considerations on user-interaction for semi-automated driving. In Proceedings of the 35th FISITA World Automotive Congress 2014, Maastricht, The Netherlands, 2–6 June 2014. [Google Scholar]
  11. Richardson, N.; Michel, B.; Zimmermann, A.; Diermeyer, F. Erfassung und Bewertung des Informationsbedarfs von Lkw-Fahrern während hochautomatisierter Fahrt. 9. VDI-Tagung-Der Fahrer im 21. Jahrhundert. 2017. Available online: https://www.researchgate.net/publication/321358983_Erfassung_und_Bewertung_des_Informationsbedarfs_von_Lkw-Fahrern_wahrend_hochautomatisierter_Fahrt (accessed on 14 January 2019).
  12. Feierle, A.; Danner, S.; Steininger, S.; Bengler, K. Information Needs and Visual Attention in Urban Highly Automated Driving–Investigation of Potential Influencing Factors. Information 2020, Accepted. [Google Scholar] [CrossRef] [Green Version]
  13. Radlmayr, J.; Feldhütter, A.; Frey, A.; Jarosch, O.; Marberger, C.; Naujoks, F.; Weinbeer, V.; Bengler, K. Drowsiness and fatigue in conditionally automated driving–Towards an integrative framework. Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2018 Annual Conference. Waard, D.D., Brookhuis, K., Coelho, D., Fairclough, S., Manzey, D., Naumann, A., Onnasch, L., Röttger, S., Toffetti, A., Wiczorek, R., Eds.; 2019. Available online: https://mediatum.ub.tum.de/1510012 (accessed on 2 January 2020).
  14. Fecher, N.; Hoffmann, J. Fahrerwarnelemente. In Handbuch Fahrerassistenzsysteme: Grundlagen, Komponenten und Systeme für Aktive Sicherheit und Komfort; Winner, H., Hakuli, S., Lotz, F., Singer, C., Eds.; Springer Fachmedien Wiesbaden: Wiesbaden, Germany, 2015; pp. 675–685. [Google Scholar]
  15. Pfeffer, S. Wahrnehmungspsychologische Untersuchung zum Thema Visueller, Haptischer und Akustischer Kanal; Studienarbeit; Uni Stuttgart: Stuttgart, Germany, 2007. [Google Scholar]
  16. Schmid, M.; Maier, T. Technisches Interface Design; Springer: Berlin/Heidelberg, Germany, 2017. [Google Scholar]
  17. Zühlke, D. Der Intelligente Versager–Das Mensch-Technik-Dilemma; Primus Verlag: Darmstadt, Germany, 2005. [Google Scholar]
  18. Feierle, A.; Bücherl, F.; Hecht, T.; Bengler, K. Evaluation of Display Concepts for the Instrument Cluster in Urban Automated Driving. In Human Systems Engineering and Design II; Ahram, T., Karwowski, W., Pickl, S., Taiar, R., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 209–215. [Google Scholar]
  19. Albert, M.; Lange, A.; Schmidt, A.; Wimmer, M.; Bengler, K. Automated Driving–Assessment of Interaction Concepts Under Real Driving Conditions. In Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, Las Vegas, NV, USA, 26–30 July 2015; Ahram, T., Karwowski, W., Schmorrow, D., Eds.; Elsevier: Amsterdam, The Netherlands, 2015; pp. 2832–2839. [Google Scholar]
  20. Forster, Y.; Hergeth, S.; Naujoks, F.; Krems, J.; Keinath, A. User Education in Automated Driving: Owner’s Manual and Interactive Tutorial Support Mental Model Formation and Human-Automation Interaction. Information 2019, 10, 143. [Google Scholar] [CrossRef] [Green Version]
  21. Feldhütter, A.; Härtwig, N.; Kurpiers, C.; Hernandez, J.M.; Bengler, K. Effect on Mode Awareness When Changing from Conditionally to Partially Automated Driving. In Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018), Florence, Italy, 26–30 August 2018; Bagnara, S., Tartaglia, R., Albolino, S., Alexander, T., Fujita, Y., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 314–324. [Google Scholar]
  22. Petermeijer, S.; Doubek, F.; de Winter, J. Driver response times to auditory, visual, and tactile take-over requests: A simulator study with 101 participants. In Proceedings of the 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Banff, AB, Canada, 5–8 October 2017. [Google Scholar]
  23. Feierle, A.; Beller, D.; Bengler, K. Head-Up Displays in Urban Partially Automated Driving: Effects of Using Augmented Reality. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 27–30 October 2019; pp. 1877–1882. [Google Scholar] [CrossRef]
  24. Prasch, L.; Geßler, F.; Reinhardt, J.; Bengler, K. You do the talking. Passengers are happy when the automation decides on cooperation. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018. [Google Scholar]
  25. Naujoks, F.; Forster, Y.; Wiedemann, K.; Neukum, A. Improving Usefulness of Automated Driving by Lowering Primary Task Interference through HMI Design. J. Adv. Transp. 2017, 2017, 1–12. [Google Scholar] [CrossRef] [Green Version]
  26. Wintersberger, P.; von Sawitzky, T.; Frison, A.-K.; Riener, A. Traffic Augmentation as a Means to Increase Trust in Automated Driving Systems. In Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, Cagliari, Italy, 18–20 September 2017; Paterno, F., Ed.; ACM: New York, NY, USA, 2017; pp. 1–7, ISBN 978-1-4503-5237-6. [Google Scholar]
  27. Schömig, N.; Wiedemann, K.; Naujoks, F.; Neukum, A.; Leuchtenberg, B.; Vöhringer-Kuhnt, T. An Augmented Reality Display for Conditionally Automated Driving. In Adjunct Proceedings, 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, Toronto, ON, Canada, 23–25 September 2018; with Doctoral Colloquium on September 22; Association for Computing Machinery: New York, NY, USA, 2018; pp. 137–141. ISBN 978-1-4503-5947-4. [Google Scholar]
  28. Damböck, D.; Weißgerber, T.; Kienle, M.; Bengler, K. Evaluation of a Contact Analog Head-Up Display for Highly Automated Driving. In Advances in Human Aspects of Road and Rail Transportation; Stanton, N.A., Ed.; CRC Press: Boca Raton, FL, USA, 2012. [Google Scholar]
  29. Eriksson, A.; Petermeijer, S.M.; Zimmermann, M.; de Winter, J.C.F.; Bengler, K.J.; Stanton, N.A. Rolling Out the Red (and Green) Carpet: Supporting Driver Decision Making in Automation-to-Manual Transitions. IEEE Trans. Hum. Mach. Syst. 2019, 49, 20–31. [Google Scholar] [CrossRef] [Green Version]
  30. Yang, Y.; Karakaya, B.; Dominioni, G.C.; Kawabe, K.; Bengler, K. An HMI Concept to Improve Driver’s Visual Behavior and Situation Awareness in Automated Vehicle. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018. [Google Scholar]
  31. Löcken, A.; Heuten, W.; Boll, S. AutoAmbiCar. In Adjunct Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Ann Arbor, MI, USA, 24–26 October 2016; Green, P., Ed.; ACM: New York, NY, USA, 2016; pp. 57–62. ISBN 978-1-4503-4654-2. [Google Scholar]
  32. Van den Beukel, A.P.; van der Voort, M.C. Driving automation & changed driver’s task-effect of driver-interfaces on intervention. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden, 19–22 June 2016. [Google Scholar]
  33. Kalb, L.; Streit, L.; Bengler, K. Multimodal Priming of Drivers for a Cooperative Take-Over. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018. [Google Scholar]
  34. Radlmayr, J.; Ratter, M.; Feldhütter, A.; Körber, M.; Prasch, L.; Schmidtler, J.; Yang, Y.; Bengler, K. Take-Overs in Level 3 Automated Driving–Proposal of the Take-Over Performance Score (TOPS). In Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018), Florence, Italy, 26–30 August 2018; Bagnara, S., Tartaglia, R., Albolino, S., Alexander, T., Fujita, Y., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 436–446. [Google Scholar]
  35. Yang, Y.; Götze, M.; Laqua, A.; Dominioni, G.C.; Kawabe, K.; Bengler, K. A method to improve driver’s situation awareness in automated driving. Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2017 Annual Conference; Waard, D.D., Toffetti, A., Wiczorek, R., Sonderegger, A., Röttger, S., Bouchner, P., Franke, T., Fairclough, S., Noordzij, M., Brookhuis, K., Eds.; 2017. Available online: https://www.researchgate.net/profile/Yucheng_Yang/publication/335992600_A_method_to_improve_driver's_situation_awareness_in_automated_driving/links/5d89f160458515cbd1be5b44/A-method-to-improve-drivers-situation-awareness-in-automated-driving.pdf (accessed on 2 January 2020).
  36. Feldhütter, A.; Kroll, D.; Bengler, K. Wake Up and Take Over! The Effect of Fatigue on the Take-over Performance in Conditionally Automated Driving. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018. [Google Scholar]
  37. Körber, M.; Gold, C.; Lechner, D.; Bengler, K. The influence of age on the take-over of vehicle control in highly automated driving. Transp. Res. Part F Traffic Psychol. Behav. 2016, 39, 19–32. [Google Scholar] [CrossRef] [Green Version]
  38. Petermeijer, S.M.; Cieler, S.; de Winter, J.C.F. Comparing spatially static and dynamic vibrotactile take-over requests in the driver seat. Accid. Anal. Prev. 2017, 99, 218–227. [Google Scholar] [CrossRef] [PubMed]
  39. Petermeijer, S.; Bazilinskyy, P.; Bengler, K.; de Winter, J. Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop. Appl. Ergon. 2017, 62, 204–215. [Google Scholar] [CrossRef] [PubMed]
  40. Carsten, O.; Martens, M.H. How can humans understand their automated cars? HMI principles, problems and solutions. Cogn. Technol. Work. 2019, 21, 3–20. [Google Scholar] [CrossRef] [Green Version]
  41. Knoll, P. Anzeigen für Fahrerassistenzsysteme. In Handbuch Fahrerassistenzsysteme: Grundlagen, Komponenten und Systeme für Aktive Sicherheit und Komfort; Winner, H., Hakuli, S., Lotz, F., Singer, C., Eds.; Springer Fachmedien Wiesbaden: Wiesbaden, Germany, 2015; pp. 659–673. [Google Scholar]
  42. Stecher, M.; Michel, B.; Zimmermann, A. The Benefit of Touchless Gesture Control: An Empirical Evaluation of Commercial Vehicle-Related Use Cases. In Advances in Human Aspects of Transportation; Stanton, N.A., Ed.; Springer International Publishing: Cham, Switzerland, 2018; pp. 383–394. [Google Scholar]
  43. Braun, M.; Broy, N.; Pfleging, B.; Alt, F. Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction. J. Multimodal User Interfaces 2019, 13, 71–88. [Google Scholar] [CrossRef]
  44. Hecht, T.; Feldhütter, A.; Draeger, K.; Bengler, K. What Do You Do? An Analysis of Non-driving Related Activities During a 60 Minutes Conditionally Automated Highway Drive. In Human Interaction and Emerging Technologies; Ahram, T., Taiar, R., Colson, S., Choplin, A., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 28–34. [Google Scholar]
  45. Pfleging, B.; Rang, M.; Broy, N. Investigating user needs for non-driving-related activities during automated driving. In MUM 2016, Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, Rovaniemi, Finland, 13–15 December 2016; Alt, F., Ed.; The Association for Computing Machinery, Inc.: New York, NY, USA, 2016; pp. 91–99. ISBN 978-1-4503-4860-7. [Google Scholar]
  46. Deb, S.; Strawderman, L.J.; Carruth, D.W. Investigating pedestrian suggestions for external features on fully autonomous vehicles: A virtual reality experiment. Transp. Res. Part F Traffic Psychol. Behav. 2018, 59, 135–149. [Google Scholar] [CrossRef]
  47. Mahadevan, K.; Somanath, S.; Sharlin, E. Communicating Awareness and Intent in Autonomous Vehicle-Pedestrian Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; ACM: New York, NY, USA, 2018; pp. 1–12. [Google Scholar] [CrossRef]
  48. Garay-Vega, L.; Hastings, A.; Pollard, J.K.; Zuschlag, M.; Steams, M.D. Quieter Cars and the Safety of Blind Pedestrians: Phase I; United States. National Highway Traffic Safety Administration: Washington, DC, USA, 2010. [Google Scholar]
  49. Regulation (EU) No 540/2014 of the European Parliament and of the Council. 2014. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32014R0540 (accessed on 2 January 2020).
  50. Joisten, P.; Alexandi, E.; Drews, R.; Klassen, L.; Petersohn, P.; Pick, A.; Schwindt, S.; Abendroth, B. Displaying Vehicle Driving Mode–Effects on Pedestrian Behavior and Perceived Safety. In Human Systems Engineering and Design II; Ahram, T., Karwowski, W., Pickl, S., Taiar, R., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 250–256. [Google Scholar]
  51. McNamara, P. Car of the Year: 2046: BMW Vision Next 100 Explored. Available online: https://www.carmagazine.co.uk/car-news/first-official-pictures/bmw/car-of-the-year-2046-bmw-vision-next-100-car-may-2016/ (accessed on 28 November 2019).
  52. Faas, S.M.; Baumann, M. Yielding Light Signal Evaluation for Selfdriving Vehicle and Pedestrian Interaction. In Human Systems Engineering and Design II; Ahram, T., Karwowski, W., Pickl, S., Taiar, R., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 189–194. [Google Scholar]
  53. Lagström, T.; Lundgren, V.M. AVIP-Autonomous Vehicles’ Interaction with Pedestrians: An Investigation of Pedestrian-Driver Communication and Development of a Vehicle External Interface. Master’s Thesis, Chalmers University of Technology, Gothenburg, Sweden, 2015. [Google Scholar]
  54. Daimler AG. Automated Driving: How Do We Develop Trust between Humans and Machines? Available online: https://www.daimler.com/innovation/case/autonomous/future-insight-2.html (accessed on 28 November 2019).
  55. Semcon. Who Sees You When the Car Drives Itself? Available online: https://semcon.com/smilingcar/ (accessed on 28 November 2019).
  56. Rettenmaier, M.; Pietsch, M.; Schmidtler, J.; Bengler, K. Passing through the Bottleneck-The Potential of External Human-Machine Interfaces. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; pp. 1687–1692. [Google Scholar] [CrossRef]
  57. Clamann, M.; Aubert, M.; Cummings, M.L. Evaluation of Vehicle-to-Pedestrian Communication Displays for Autonomous Vehicles. In Proceedings of the Transportation Research Board 96th Annual Meeting, Washington, DC, USA, 8–12 January 2017. [Google Scholar]
  58. De Clercq, K.; Dietrich, A.; Núñez Velasco, J.P.; de Winter, J.; Happee, R. External Human-Machine Interfaces on Automated Vehicles: Effects on Pedestrian Crossing Decisions. Hum. Factors 2019, 61, 1353–1370. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  59. Government Technology. How Can a Smiling Car Make Roads Safer? Available online: https://www.govtech.com/question-of-the-day/Question-of-the-Day-for-09212016.html (accessed on 28 November 2019).
  60. Mitsubishi. Mitsubishi Electric Introduces Road-Illuminating Directional Indicators. Illuminated Projections on Road Surfaces Expected to Help Avoid Accidents; Mitsubishi: Tokyo, Japan, 2015. [Google Scholar]
  61. Volkswagen. Let There Be Light! Available online: https://www.volkswagen-newsroom.com/en/stories/let-there-be-light-4255 (accessed on 14 January 2019).
  62. Dietrich, A.; Willrodt, J.-H.; Wagner, K.; Bengler, K. Projection-Based External Human Machine Interfaces–Enabling Interaction between Automated Vehicles and Pedestrians. In Proceedings of the DSC 2018 Europe VR, Antibes, France, 5–7 September 2018. [Google Scholar]
  63. Daimler AG. Autonomes Fahren Wird Immer Greifbarer. Available online: https://www.daimler.com/innovation/autonomes-fahren/experteninterview.html (accessed on 28 November 2019).
  64. Chang, C.-M.; Toda, K.; Sakamoto, D.; Igarashi, T. Eyes on a Car: An Interface Design for Communication between an Autonomous Car and a Pedestrian. In Proceedings of the 9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, Oldenburg, Germany, 24–27 September 2017; ACM: New York, NY, USA, 2016; pp. 65–73. [Google Scholar] [CrossRef]
  65. Habibovic, A.; Lundgren, V.M.; Andersson, J.; Klingegård, M.; Lagström, T.; Sirkka, A.; Fagerlönn, J.; Edgren, C.; Fredriksson, R.; Krupenia, S.; et al. Communicating Intent of Automated Vehicles to Pedestrians. Front. Psychol. 2018, 9, 1336. [Google Scholar] [CrossRef] [PubMed]
  66. Böckle, M.-P.; Brenden, A.P.; Klingegård, M.; Habibovic, A.; Bout, M. SAV2P: Exploring the Impact of an Interface for Shared Automated Vehicles on Pedestrians’ Experience. In Proceedings of the 9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, Oldenburg, Germany, 24–27 September 2017; ACM: New York, NY, USA, 2017; pp. 136–140. [Google Scholar] [CrossRef]
  67. Matthews, M.; Chowdhary, G.; Kieson, E. Intent Communication between Autonomous Vehicles and Pedestrians; Cornell University: Ithaca, NY, USA, 2017. [Google Scholar]
  68. Werner, A. New Colours for Autonomous Driving: An Evaluation of Chromaticities for the External Lighting Equipment of Autonomous Vehicles. Colour Turn 2018. [Google Scholar] [CrossRef]
  69. Faas, S.M.; Baumann, M. Light-Based External Human Machine Interface: Color Evaluation for Self-Driving Vehicle and Pedestrian Interaction. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting; Sage Publications: Los Angeles, CA, USA, 2019; Volume 63, pp. 1232–1236. [Google Scholar] [CrossRef]
  70. Eisma, Y.B.; van Bergen, S.; ter Brake, S.M.; Hensen, M.T.T.; Tempelaar, W.J.; de Winter, J.C.F. External human-machine interfaces: The effect of display location on crossing intentions and eye movements. Information 2019, 11, 13. [Google Scholar] [CrossRef] [Green Version]
  71. Brown, B.; Laurier, E. The Trouble with Autopilots: Assisted and Autonomous Driving on the Social Road. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems; ACM: New York, NY, USA, 2017; pp. 416–429. ISBN 978-1-4503-4655-9. [Google Scholar]
  72. Risto, M.; Emmenegger, C.; Vinkhuyzen, E.; Cefkin, M.; Hollan, J. Human-Vehicle Interfaces: The Power of Vehicle Movement Gestures in Human Road User Coordination. In Proceedings of the Ninth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Manchester Village, VT, USA, 26–29 June 2017; pp. 186–192. [Google Scholar]
  73. Müller, L.; Risto, M.; Emmenegger, C. The Social Behavior of Autonomous Vehicles. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct; ACM: New York, NY, USA, 2016; pp. 686–689. ISBN 978-1-4503-4462-3. [Google Scholar]
  74. Dietrich, A.; Bengler, K.; Portouli, E.; Nathanael, D.; Ruenz, J.; Wu, J.; Merat, N.; Madigan, R.; Lee, Y.M.; Markkula, G.; et al. interACT D.2.1 Preliminary Description of Psychological Models on Human-Human Interaction in Traffic; 2018; Available online: https://www.interact-roadautomation.eu/wp-content/uploads/interACT_WP2_D2.1-_PsychologicalModels_v1.0_approved_Uploadwebsite.pdf (accessed on 2 January 2020).
  75. Dey, D.; Terken, J. Pedestrian Interaction with Vehicles: Roles of Explicit and Implicit Communication. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; ACM: New York, NY, USA, 2017; pp. 109–113. ISBN 978-1-4503-5150-8. [Google Scholar]
  76. Imbsweiler, J.; Ruesch, M.; Palyafári, R.; Puente León, F.; Deml, B. Entwicklung einer Beobachtungsmethode von Verhaltensströmen in kooperativen Situationen im innerstädtischen Verkehr. In Proceedings of the 32. VDI/VW-Gemeinschaftstagung, Fahrerassistenz und automatisiertes Fahren, Wolfsburg, Germany, 8–9 November 2016. [Google Scholar]
  77. Rettenmaier, M.; Requena Witzig, C.; Bengler, K. Interaction at the Bottleneck–A Traffic Observation. In Human Systems Engineering and Design II; Ahram, T., Karwowski, W., Pickl, S., Taiar, R., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 243–249. [Google Scholar]
  78. Rothenbücher, D.; Li, J.; Sirkin, D.; Mok, B.; Ju, W. Ghost driver: A field study investigating the interaction between pedestrians and driverless vehicles. In Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA, 26–31 August 2016; pp. 795–802. [Google Scholar]
  79. Pillai, A.K. Virtual Reality Based Study to Analyse Pedestrian Attitude towards Autonomous Vehicles; KTH, Media Technology and Interaction Design, MID; Aalto University: Espoo, Finland, 2017. [Google Scholar]
  80. Rodríguez Palmeiro, A.; van der Kint, S.; Vissers, L.; Farah, H.; de Winter, J.C.F.; Hagenzieker, M. Interaction between pedestrians and automated vehicles: A Wizard of Oz experiment. Transp. Res. Part F Traffic Psychol. Behav. 2018, 58, 1005–1020. [Google Scholar] [CrossRef] [Green Version]
  81. Ackermann, C.; Beggiato, M.; Bluhm, L.-F.; Krems, J. Vehicle movements as implicit communication signal between pedestrians and automated vehicles. In Proceedings of the 6th Humanist Conference, The Hague, The Netherlands, 13–14 June 2018. [Google Scholar]
  82. Fuest, T.; Michalowski, L.; Traris, L.; Bellem, H.; Bengler, K. Using the Driving Behavior of an Automated Vehicle to Communicate Intentions-A Wizard of Oz Study. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 3596–3601. [Google Scholar]
  83. Beggiato, M.; Witzlack, C.; Krems, J.F. Gap Acceptance and Time-To-Arrival Estimates As Basis for Informal Communication between Pedestrians and Vehicles. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; ACM: New York, NY, USA, 2017; pp. 50–57. ISBN 978-1-4503-5150-8. [Google Scholar]
  84. Kauffmann, N.; Winkler, F.; Naujoks, F.; Vollrath, M. “What Makes a Cooperative Driver?” Identifying parameters of implicit and explicit forms of communication in a lane change scenario. Transp. Res. Part F Traffic Psychol. Behav. 2018, 58, 1031–1042. [Google Scholar] [CrossRef]
  85. Potzy, J.; Feuerbach, M.; Bengler, K. Communication Strategies for Automated Merging in Dense Traffic. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; pp. 2291–2298. [Google Scholar]
  86. Potzy, J.; Sophie, F.; Siedersberger, K.-H.; Bengler, K. Manual Drivers’ Evaluation of Automated Merging Behavior in Dense Traffic: Efficiency Matters. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 27–30 October 2019; pp. 3454–3460. [Google Scholar]
  87. Kauffmann, N.; Naujoks, F.; Winkler, F.; Kunde, W. Learning the “Language” of Road Users-How Shall a Self-driving Car Convey Its Intention to Cooperate to Other Human Drivers? In Advances in Human Factors and Systems Interaction; Nunes, I.L., Ed.; Springer International Publishing: Cham, Switzerland, 2018; pp. 53–63. ISBN 978-3-319-60366-7. [Google Scholar]
  88. Cramer, S.; Kaup, I.; Siedersberger, K.-H. Comprehensibility and Perceptibility of Vehicle Pitch Motions as Feedback for the Driver during Partially Automated Driving. IEEE Trans. Intell. Veh. 2019, 4, 3–13. [Google Scholar] [CrossRef]
  89. Lange, A. Gestaltung der Fahrdynamik beim Fahrstreifenwechselmanöver als Rückmeldung für den Fahrer beim Automatisierten Fahren. Ph.D. Thesis, Technical University of Munich, Munich, Germany, 2018. [Google Scholar]
  90. Othersen, I.; Cramer, S.; Salomon, C. HMI for External Communication-Kann Die Fahrzeugbewegung als Kommunikation Zwischen Einem Fahrzeug und Einem Fußgänger Dienen? VDI-Berichte Nr. 2360; 2019; Available online: https://www.researchgate.net/publication/338209963_HMI_for_external_communication_-_Kann_die_Fahrzeugbewegung_als_Kommunikationskanal_zwischen_einem_Fahrzeug_und_einem_Fussganger_dienen (accessed on 2 January 2020).
  91. Cramer, S.; Lange, A.; Bültjes, S.; Klohr, J. Design Possibilities for Vehicle Roll Motions as Feedback for the Driver during Automated Driving. In AutomotiveUI ’17: Proceedings of the 9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications; ACM: New York, NY, USA, 2017; pp. 152–157. [Google Scholar]
Information 11 00061 g001 550
Figure 1. The HMI framework including the factors influencing the HMI characteristics and selection on the upper part, and the different HMI types dynamic HMI (dHMI), vehicle HMI (vHMI), infotainment HMI (iHMI), automation HMI (aHMI), and external HMI (eHMI) on the lower part.
Figure 1. The HMI framework including the factors influencing the HMI characteristics and selection on the upper part, and the different HMI types dynamic HMI (dHMI), vehicle HMI (vHMI), infotainment HMI (iHMI), automation HMI (aHMI), and external HMI (eHMI) on the lower part.
Information 11 00061 g001
Information 11 00061 g002 550
Figure 2. Static characteristics of an urban road scenery. The static influencing factors in this example include a two-lane road, a bike lane, and a zebra crossing. The traffic signs provide further information about the prevailing traffic rules. Additionally, the scenery includes houses and trees.
Figure 2. Static characteristics of an urban road scenery. The static influencing factors in this example include a two-lane road, a bike lane, and a zebra crossing. The traffic signs provide further information about the prevailing traffic rules. Additionally, the scenery includes houses and trees.
Information 11 00061 g002
Information 11 00061 g003 550
Figure 3. Dynamic and static characteristics of an urban road scenery. The dynamic influencing factors in this example include the AV (red) and additional moving HRU. The arrows represent the HRU’s trajectories. The parking lorry in the center is also part of the dynamic elements because it could potentially continue driving anytime.
Figure 3. Dynamic and static characteristics of an urban road scenery. The dynamic influencing factors in this example include the AV (red) and additional moving HRU. The arrows represent the HRU’s trajectories. The parking lorry in the center is also part of the dynamic elements because it could potentially continue driving anytime.
Information 11 00061 g003
Information 11 00061 g004 550
Figure 4. Most common visual HMI-elements used in aHMIs. (a) Instrument cluster, head-up display, and monitor in the center console used for presenting automation state and maneuvers [19]. (b) Augmented marking of a front vehicle and the control range in the head-up display [23]. (c) LED-strip in the windshield to visualize the automation mode using color-coding [21].
Figure 4. Most common visual HMI-elements used in aHMIs. (a) Instrument cluster, head-up display, and monitor in the center console used for presenting automation state and maneuvers [19]. (b) Augmented marking of a front vehicle and the control range in the head-up display [23]. (c) LED-strip in the windshield to visualize the automation mode using color-coding [21].
Information 11 00061 g004
Information 11 00061 g005 550
Figure 5. Categorization of different visual eHMI types clustered in the groups labelled as “automated” [51], light-strips [54], display [59], and projection [63].
Figure 5. Categorization of different visual eHMI types clustered in the groups labelled as “automated” [51], light-strips [54], display [59], and projection [63].
Information 11 00061 g005

Share and Cite

MDPI and ACS Style

Bengler, K.; Rettenmaier, M.; Fritz, N.; Feierle, A. From HMI to HMIs: Towards an HMI Framework for Automated Driving. Information 2020, 11, 61. https://doi.org/10.3390/info11020061

AMA Style

Bengler K, Rettenmaier M, Fritz N, Feierle A. From HMI to HMIs: Towards an HMI Framework for Automated Driving. Information. 2020; 11(2):61. https://doi.org/10.3390/info11020061

Chicago/Turabian Style

Bengler, Klaus, Michael Rettenmaier, Nicole Fritz, and Alexander Feierle. 2020. "From HMI to HMIs: Towards an HMI Framework for Automated Driving" Information 11, no. 2: 61. https://doi.org/10.3390/info11020061

APA Style

Bengler, K., Rettenmaier, M., Fritz, N., & Feierle, A. (2020). From HMI to HMIs: Towards an HMI Framework for Automated Driving. Information, 11(2), 61. https://doi.org/10.3390/info11020061

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop