From HMI to HMIs: Towards an HMI Framework for Automated Driving

: During automated driving, there is a need for interaction between the automated vehicle (AV) and the passengers inside the vehicle and between the AV and the surrounding road users outside of the car. For this purpose, di ﬀ erent types of human machine interfaces (HMIs) are implemented. This paper introduces an HMI framework and describes the di ﬀ erent HMI types and the factors inﬂuencing their selection and content. The relationship between these HMI types and their inﬂuencing factors is also presented in the framework. Moreover, the interrelations of the HMI types are analyzed. Furthermore, we describe how the framework can be used in academia and industry to coordinate research and development activities. With the help of the HMI framework, we identify research gaps in the ﬁeld of HMI for automated driving to be explored in the future.


Introduction
For decades, researchers and engineers have developed human machine interfaces (HMIs) to ensure and to emend appropriate interaction between conventional vehicles and their drivers.A user-centered approach became inevitable in the development of concepts for human machine interaction.An automotive HMI consists mainly of output channels that provide information about the system state to the driver (e.g., via displays and auditory signals), input channels to receive the driver's input (e.g., via buttons, steering wheel, and pedals), and a dialog logic to specify the relationships among input, output, and context parameters.
The ongoing implementation of advanced driver assistance systems (ADAS) towards higher levels of automation, on the one hand, offers the possibility of creating new ways to communicate and therefore changes the interaction between driver, vehicle, and the surrounding traffic system.Further development of sensor technology increases its detection range and enables the automated vehicle (AV) to create an awareness of its surroundings including other human road users (HRU) such as pedestrians, cyclists, and drivers of conventional vehicles.Moreover, the ongoing development of automation accompanies the AV's growing responsibility of the driving task.These facts enable the AV to use new types of HMIs such as communicating via the vehicle dynamics, informing other HRUvia external interfaces, or providing guidance to the AV's passengers.
On the other hand, the introduction of AVs into traffic entails the necessity of additional innovative communication strategies towards the AV user.From SAE Level 3 and higher, the AV's "driver" becomes a passenger [1].Consequently, automation gives the passenger access to new opportunities such as engaging in non-driving related activities (NDRA) which results in less attention to the driving scene.Thus, there are new aspects that have to be designed and have been investigated concerning in-vehicle communication, for example, a request to intervene, passenger's information need, and NDRA.In parallel to the development of ADAS and automated driving functions, in-vehicle information systems and infotainment systems increase their functionality and offer an unimaginable amount of information and potential interactions.
Moreover, a possibly distracted passenger is no longer available for established communication with surrounding HRU.Thus, it becomes necessary to investigate not only the AV-passenger interaction, but also communication strategies with surrounding traffic.At present, different HRU communicate implicitly via the trajectory of the car (e.g., maintain speed, accelerate, and decelerate) and explicitly via flashing lights or using hand gestures [2].In the context of automated driving, communication strategies could be derived from the mentioned approaches of human-to-human communication.AVs could communicate their intention implicitly via the trajectory or explicitly via external HMIs.
To introduce successfully AVs in traffic, communication with the passenger, as well as with other HRU, is essential.The communication strategies of different HMIs integrated in one single vehicle must not only perform individually but also in combination and synchronization with each other.Therefore, development is progressing from an HMI to HMIs.

Motivation
Originally, the development of different HMIs was located in different departments of a company or spread over different companies if infotainment was integrated via, for example, terminal mode concepts such as Apple carplay or Mirrorlink.Additionally, in research, studies were designed to answer specific research questions, mainly focusing on one HMI aspect.We argue that it is going to be vital for successful communication and interaction to consider the synchronization and consistency of the different HMIs.Furthermore, an HMI needs to be activated in the right context and at the right time in order for it to successfully communicate the relevant information.This paper aims to introduce an HMI framework that can be used for the coordination of different research and development activities.The HMI framework should highlight the influence of different factors on the HMI selection and it should visualize the interrelations between the HMI types themselves.Previous approaches that addressed different interactive surfaces inside the vehicle [3] did not provide a holistic understanding of internally and externally communicating HMI solutions and their interrelations.The HMI types presented are defined in the ISO/TR 21,959 [4], whereby the interrelations between the different HMIs and a holistic view are discussed in this paper.This framework is motivated from an ergonomic perspective and is not intended to present a technical architecture for the implementation of automotive HMIs.Nevertheless, the basic structure and ergonomic requirements could serve as input for technical architectures.

HMI Framework
The HMI framework is pictured in Figure 1.It describes the interrelations between the influencing factors and the AV's selection of HMI types and HMI content in a given situation.The influencing factors (upper part of Figure 1, see also Section 3.1) consist of the static infrastructure for the operation of the AV, the dynamic elements that change as the situation evolves, and the automation system with its actual capabilities.Moreover, the addressees of the AV's communication belonging to the dynamic elements supplement the influencing factors.The lower part of the HMI framework (Figure 1) presents the five different HMI types (Section 3.2) that the AV possesses.We advocate for consistent terminology of these HMI types that are elaborated in the following sections.
Different HMIs are disjunct to each other and are clustered in the way they communicate with the addressees, either internally with the AV's passenger(s) or externally with other road users, such as drivers of conventional vehicles, cyclists, or pedestrians.The term passenger in this publication refers to the occupant of a vehicle for automation levels 0 to 5 according to SAE [1], regardless of whether the passenger currently has a driving or supervising task.
The dynamic HMI (dHMI) is part of both external and internal communication.It communicates with the AV's passenger and to surrounding HRU via the AV's dynamics due to the influence of conventional driving dynamics or due to specifically designed trajectories.In addition to the dHMI, the vehicle HMI (vHMI), the infotainment HMI (iHMI), and the automation HMI (aHMI) complement the group of internally communicating HMIs.The vHMI shares information about the vehicle's condition, whereas the iHMI offers additional interfaces for NDRA.The aHMI comprises the communication of all relevant information about the system status as well as current and future activities of the driving automation system (DAS).The external HMI (eHMI) is part of the external communication and communicates its information to surrounding HRU via interfaces at the AV's surface.conventional driving dynamics or due to specifically designed trajectories.In addition to the dHMI, the vehicle HMI (vHMI), the infotainment HMI (iHMI), and the automation HMI (aHMI) complement the group of internally communicating HMIs.The vHMI shares information about the vehicle's condition, whereas the iHMI offers additional interfaces for NDRA.The aHMI comprises the communication of all relevant information about the system status as well as current and future activities of the driving automation system (DAS).The external HMI (eHMI) is part of the external communication and communicates its information to surrounding HRU via interfaces at the AV's surface.The framework is structured as a circuit.The influencing factors affect the selection and the content of the HMI types, resulting in the use of the most appropriate HMI types in a given scenario.As a result, the AV's communication influences the addressees' behavior, and thereby ultimately the interaction between the AV and its surrounding.Moreover, in the process of driving the static infrastructure, the dynamic elements, as well as the capabilities of the automation system itself change over time.This in turn influences the suitability and the selection of the HMI types and their content.The framework is structured as a circuit.The influencing factors affect the selection and the content of the HMI types, resulting in the use of the most appropriate HMI types in a given scenario.As a result, the AV's communication influences the addressees' behavior, and thereby ultimately the interaction between the AV and its surrounding.Moreover, in the process of driving the static infrastructure, the dynamic elements, as well as the capabilities of the automation system itself change over time.This in turn influences the suitability and the selection of the HMI types and their content.

Influencing Factors
The influencing factors contain all attributes in the respective scenario affecting how the AV's communication strategy needs to be in a given situation to enable successful human-AV interaction.We adapted the influencing factors' clustering from the specification methodology of the collaborative research project UR: BAN [5] and adapted by the project @City [6].The influencing factors include static and dynamic elements of a scene, the automation system, and the AV's interaction partners in the scenario.

Static Infrastructure
The first factor influencing the suitable communicational means is the static infrastructure.The static infrastructure consists of all elements in the scenario that remain constant over time.Among others, this includes the road type (geometry, speed limits) with possible lanes, entrances and exits, and visual constrictions either for the AV's sensors or for the AV's passengers or surrounding HRU.Additionally, a decisive factor is the vulnerable road users (VRU) infrastructure.Moreover, the prevailing traffic rules influence the interaction and communication needs between the AV and other HRU.Using the example of an urban road section, the mentioned components would be listed as essential static characteristics and attributes (see Figure 2).

Influencing Factors
The influencing factors contain all attributes in the respective scenario affecting how the AV's communication strategy needs to be in a given situation to enable successful human-AV interaction.We adapted the influencing factors' clustering from the specification methodology of the collaborative research project UR:BAN [5] and adapted by the project @City [6].The influencing factors include static and dynamic elements of a scene, the automation system, and the AV's interaction partners in the scenario.

Static Infrastructure
The first factor influencing the suitable communicational means is the static infrastructure.The static infrastructure consists of all elements in the scenario that remain constant over time.Among others, this includes the road type (geometry, speed limits) with possible lanes, entrances and exits, and visual constrictions either for the AV's sensors or for the AV's passengers or surrounding HRU.Additionally, a decisive factor is the vulnerable road users (VRU) infrastructure.Moreover, the prevailing traffic rules influence the interaction and communication needs between the AV and other HRU.Using the example of an urban road section, the mentioned components would be listed as essential static characteristics and attributes (see Figure 2).

Dynamic Elements
The dynamic elements in the scenario are all attributes that are not static but change over time, and thus affect the AV's interaction strategy.This includes both high-dynamic elements that change their location within seconds and low-dynamic elements that change their location within minutes to hours.Figure 3 shows the enrichment of the static infrastructure with dynamic elements.In particular, the surrounding HRU such as pedestrians, cyclists, or drivers of conventional cars and the AV itself including the driving strategy and individual maneuvers (speed, acceleration, and position) of the aforementioned HRU belong to the high-dynamic elements.In addition, low-dynamic elements such as the weather and lighting conditions affect the efficiency of different HMI types especially in the case of external communication.Furthermore, present nonmoving objects, such as double parking vehicles and other road constrictions such as construction sites, can lead to visual constrictions, and thereby affect the selection of adequate HMI types and content.We classify these objects as low-dynamic elements, although they do not move in the given scenario, because they could dynamically change their location over time.

Dynamic Elements
The dynamic elements in the scenario are all attributes that are not static but change over time, and thus affect the AV's interaction strategy.This includes both high-dynamic elements that change their location within seconds and low-dynamic elements that change their location within minutes to hours.Figure 3 shows the enrichment of the static infrastructure with dynamic elements.In particular, the surrounding HRU such as pedestrians, cyclists, or drivers of conventional cars and the AV itself including the driving strategy and individual maneuvers (speed, acceleration, and position) of the aforementioned HRU belong to the high-dynamic elements.In addition, low-dynamic elements such as the weather and lighting conditions affect the efficiency of different HMI types especially in the case of external communication.Furthermore, present nonmoving objects, such as double parking vehicles and other road constrictions such as construction sites, can lead to visual constrictions, and thereby affect the selection of adequate HMI types and content.We classify these objects as low-dynamic elements, although they do not move in the given scenario, because they could dynamically change their location over time.

Automation System
The automation system strongly influences the selection and the content of the five HMI types.Common industrial standards (e.g., SAE J3016 [1]) classify automated systems according to their functional scope via different levels of automation.As the functional scope of the automation system increases, the role of the traditional driver changes from active operator (during manual driving), to passive monitoring (during partially automated driving), to passive passenger (during conditionally, highly, and fully automated driving) within a specific operational design domain (ODD).As the role of the passenger changes, the information need does, too [7].Therefore, the AV's present automation level is decisive for the type and amount of information presented to the passenger.If the AV is driving in partially automated mode (SAE Level 2), the passenger has to monitor the current driving situation and the automation system to be able to take over vehicle control if system limits are reached.The passenger, therefore, needs information about automation system status and maneuvers [8][9][10].Whereas, if the AV is driving in conditionally, highly, or fully automated mode (SAE Levels 3, 4, and 5) the passenger could be involved in NDRA.Therefore, the passenger's role differs between different levels of automation (driver vs. monitoring passenger vs. passive passenger) as does the passenger's state (e.g., attention), which needs to be considered when selecting the content of internal interfaces.Furthermore, the level of automation could change within a scenario.This could be the case if functional limits are reached and the passenger has to take over vehicle control following an automated ride or if the vehicle switches between highly and partially automated driving because the vehicle cannot guarantee a system fallback in a given ODD.These transitions need to be designed adequately with consideration of the current state of the passenger to ensure an optimal level of mode awareness and readiness.
The functional scope of the automation system further determines the AV's external communication needs and strategies with other HRU.With increasing functionality of the automation systems, the passenger becomes less involved in the actual driving task, therefore, driverrelated cues that are available nowadays in human interaction in traffic (such as eye contact and hand gestures) would no longer be available and, instead, would have to be replaced via the AV's HMIs.

Addressees
The addressees of the AV's communication are part of the dynamic elements.They are subdivided into passengers, especially the person on the driver's seat and surrounding HRU.Depending on whom the AV intends to communicate with, different interfaces are suitable.For internal communication processes the use of vHMI, iHMI, aHMI, and dHMI are appropriate, whereas to communicate with other HRU, external communication via dHMI and eHMI is suggested.The

Automation System
The automation system strongly influences the selection and the content of the five HMI types.Common industrial standards (e.g., SAE J3016 [1]) classify automated systems according to their functional scope via different levels of automation.As the functional scope of the automation system increases, the role of the traditional driver changes from active operator (during manual driving), to passive monitoring (during partially automated driving), to passive passenger (during conditionally, highly, and fully automated driving) within a specific operational design domain (ODD).As the role of the passenger changes, the information need does, too [7].Therefore, the AV's present automation level is decisive for the type and amount of information presented to the passenger.If the AV is driving in partially automated mode (SAE Level 2), the passenger has to monitor the current driving situation and the automation system to be able to take over vehicle control if system limits are reached.The passenger, therefore, needs information about automation system status and maneuvers [8][9][10].Whereas, if the AV is driving in conditionally, highly, or fully automated mode (SAE Levels 3, 4, and 5) the passenger could be involved in NDRA.Therefore, the passenger's role differs between different levels of automation (driver vs. monitoring passenger vs. passive passenger) as does the passenger's state (e.g., attention), which needs to be considered when selecting the content of internal interfaces.Furthermore, the level of automation could change within a scenario.This could be the case if functional limits are reached and the passenger has to take over vehicle control following an automated ride or if the vehicle switches between highly and partially automated driving because the vehicle cannot guarantee a system fallback in a given ODD.These transitions need to be designed adequately with consideration of the current state of the passenger to ensure an optimal level of mode awareness and readiness.
The functional scope of the automation system further determines the AV's external communication needs and strategies with other HRU.With increasing functionality of the automation systems, the passenger becomes less involved in the actual driving task, therefore, driver-related cues that are available nowadays in human interaction in traffic (such as eye contact and hand gestures) would no longer be available and, instead, would have to be replaced via the AV's HMIs.

Addressees
The addressees of the AV's communication are part of the dynamic elements.They are subdivided into passengers, especially the person on the driver's seat and surrounding HRU.Depending on whom the AV intends to communicate with, different interfaces are suitable.For internal communication processes the use of vHMI, iHMI, aHMI, and dHMI are appropriate, whereas to communicate with other HRU, external communication via dHMI and eHMI is suggested.The selection of the correct HMI type is also influenced by the dynamics of the addressees and the AV in a given scenario.For example, during low relative speed scenarios clear communication via dHMI can reach its limits.The same is true in scenarios where the AV stands still.Here, the usage of eHMI may be necessary to reach clear communication with other HRU.
Moreover, the personal factors of the passenger and the surrounding HRU affect the communication process.Personal factors include both long-term personal characteristics such as personality, gender, age, driving experience, and driving style, as well as short-term characteristics such as the current vigilance level and attention [5].

Automation HMI (aHMI)
The aHMI comprises all interfaces in the vehicle interior that enable the passenger to interact with the DAS.As the automation level rises the scope of the automation system increases, and therefore the role of the passenger changes (see Section 3.1.3).Therefore, the aHMI has to communicate the system status of the automation transparently to the passenger to guarantee mode awareness and allow safe, efficient, and comfortable transitions between automation levels.
Studies have shown differences in informational needs according to the automation level [7], traffic situations [11], and the individual passenger [12], as well as differences in the extent to which passengers show changes in driver state, for example, fatigue [13].Therefore, the aHMI should adapt to the traffic context, passenger's state, information needs, and the individual differences of the passenger to ensure sufficient trust, acceptance, and safety for the DAS.
The HMI-elements used in the aHMI can be subdivided into visual, auditory, and tactile elements.From haptic-tactile to auditory to visual, the information rate increases and the perception delay time also increases [14].Therefore, visual HMI elements are used, in particular, for monitoring and communicating the state of the DAS, whereas haptic-tactile and auditory signals, such as tones, are mainly used for warnings.Especially for warnings (in the case of a request to intervene), multimodality seems to be advantageous, since the integration of several perception modalities can result in shorter reaction times and faster perception of a higher amount of information [15,16].In addition, the redundant provision of information results in lower error rates in the operation of technical devices [16,17].
As a visual HMI element, the instrument cluster, in particular, is considered the primary interaction element for communicating the system state and further detailed information in automated driving [18].The instrument cluster is used in almost all studies on automated driving (e.g., [8,[19][20][21][22]).In addition, head-up displays have become established that can inform passengers in the primary field of view about system status, emerging situations, and maneuvers without eyes off road (e.g., [8,19,[23][24][25]).Augmented reality head-up displays, as an extension of conventional head-up displays, have been used in automated driving because they visualize maneuvers, mark detected objects directly in the environment (e.g., [23,26,27]) and can support the driver in take-over situations (e.g., [28,29]).Due to their good peripheral perception, LED strips are particularly advantageous for communicating changes in the system status (transition between automation modes or to manual driving), and therefore are used, for example, in the windshield, to provide adequate mode and situation awareness (e.g., [8,21,[30][31][32]).However, the monitor in the center console has also been used as part of the aHMI [19].Although semantics have been used in auditory aHMI (e.g., [25,33]), the aHMI design has been characterized by warning tones that have been used in the case of a request to intervene (e.g., [30,[34][35][36][37]) due to their fast perception.Haptic-tactile warnings during a request to intervene in automated driving, so far, have been implemented primarily by vibration of the driver's seat (e.g., [33,38]).However, this use was not unimodal but was integrated into a multimodal HMI (e.g., [33,39]).Figure 4 shows the most common visual HMI-elements used in previous studies.
according to SAE and the role of the passenger can be met.A particular focus must be placed on automation levels 4 to 5, which have hardly been investigated to date.At the same time, the use of automated driving in series production can lead to long-term effects that require adaptation of the aHMI over time.In particular, an adaptation to the individual differences of the passengers could be necessary.Carsten et al. [40] has provided recommendations that should be considered in the design of aHMIs.

Vehicle HMI (vHMI)
In contrast to the aHMI, the vHMI represents the HMI in the vehicle interior without automation reference.It enables the information presentation of the vehicle's condition and the interaction with its settings.For this purpose, information is communicated visually or auditorily.Auditory elements are warning tones (e.g., loss of tire pressure).As visual elements, control lights (e.g., low fuel capacity or engine malfunction) are located in the instrument cluster in the driver's direct field of vision.Settings such as the air conditioning system are located in the center console so that the front passenger can also manipulate it [41].New technologies such as gesture [42] and voice control [43] offer completely new possibilities for operating the vHMI.Several vHMI elements are legally required and need to fulfill minimum requirements.

Infotainment HMI (iHMI)
Passengers driving in higher automation levels (SAE Level 3 to 5), where continuous monitoring of the ADS is no longer necessary, can engage in NDRA [1].The iHMI is intended to enable performing NDRA beyond the classic infotainment offering (radio and telephone) and to guarantee its safe performance.Studies by [44,45] have shown that passengers want to be engaged in auditory NDRA such as talking to passengers, listening to music, calling, as well as visual NDRA such as watching the surrounding environment, watching videos on a tablet, and using their phone (e.g., reading or writing) during automated driving.This selection of visual NDRAs is not only offered via the display in the center console, but portable devices such as smartphones and tablets would inevitably have to be introduced as interaction elements.
Visual NDRAs can compete for space and presentation resources with the aHMI if these are placed on the instrument cluster or the display in the center consol.At the same time, coordination between aHMI and iHMI must be ensured in such a way that the passenger is guided back comfortably and safely into the vehicle control loop when taking over driving the vehicle due to a request to intervene or a driver-initiated takeover, in order to ensure controllability.In particular, mobile devices that are brought into the vehicle have not yet been considered as HMI elements and are seen as a future challenge.In future research, it is important to consider how the large number of different HMI elements can be combined into an overall concept so that the requirements for the various automation levels according to SAE and the role of the passenger can be met.A particular focus must be placed on automation levels 4 to 5, which have hardly been investigated to date.At the same time, the use of automated driving in series production can lead to long-term effects that require adaptation of the aHMI over time.In particular, an adaptation to the individual differences of the passengers could be necessary.Carsten et al. [40] has provided recommendations that should be considered in the design of aHMIs.

Vehicle HMI (vHMI)
In contrast to the aHMI, the vHMI represents the HMI in the vehicle interior without automation reference.It enables the information presentation of the vehicle's condition and the interaction with its settings.For this purpose, information is communicated visually or auditorily.Auditory elements are warning tones (e.g., loss of tire pressure).As visual elements, control lights (e.g., low fuel capacity or engine malfunction) are located in the instrument cluster in the driver's direct field of vision.Settings such as the air conditioning system are located in the center console so that the front passenger can also manipulate it [41].New technologies such as gesture [42] and voice control [43] offer completely new possibilities for operating the vHMI.Several vHMI elements are legally required and need to fulfill minimum requirements.

Infotainment HMI (iHMI)
Passengers driving in higher automation levels (SAE Level 3 to 5), where continuous monitoring of the ADS is no longer necessary, can engage in NDRA [1].The iHMI is intended to enable performing NDRA beyond the classic infotainment offering (radio and telephone) and to guarantee its safe performance.Studies by [44,45] have shown that passengers want to be engaged in auditory NDRA such as talking to passengers, listening to music, calling, as well as visual NDRA such as watching the surrounding environment, watching videos on a tablet, and using their phone (e.g., reading or writing) during automated driving.This selection of visual NDRAs is not only offered via the display in the center console, but portable devices such as smartphones and tablets would inevitably have to be introduced as interaction elements.
Visual NDRAs can compete for space and presentation resources with the aHMI if these are placed on the instrument cluster or the display in the center consol.At the same time, coordination between aHMI and iHMI must be ensured in such a way that the passenger is guided back comfortably and safely into the vehicle control loop when taking over driving the vehicle due to a request to intervene or a driver-initiated takeover, in order to ensure controllability.In particular, mobile devices that are brought into the vehicle have not yet been considered as HMI elements and are seen as a future challenge.

External HMI (eHMI)
External HMIs are interfaces installed on or projecting from the external surface of a vehicle.Indicators and brake lights are legally required realizations of eHMIs and highly standardized.Via the eHMI, the AV communicates with other HRU and provides information about the AV's status or its behavior.External HMIs could be clustered in auditory eHMIs [46,47] and visual eHMIs.In the context of the electrification of transport especially in low speed scenarios, hybrid electric or pure electric vehicles emit substantially less noise than comparable vehicles with combustion engines [48].Due to the missing engine sound, electric vehicles are less perceivable particularly for blind or visually impaired pedestrians [49].Auditory eHMIs could help these people, since quieter electric AVs are detected later in approaching scenarios than vehicles with a combustion engines [48].To counteract this fact, auditory eHMIs could have the potential to support communication between electric AVs and VRU.At present, visual eHMIs are almost exclusively investigated in research for AVs. Figure 5 structures visual eHMIs in four different categories, marking a vehicle as "driving automated" [50,51], light-strips [52][53][54] communicating via light patterns, displays [55][56][57][58][59] showing text or symbols, and laser projections [60][61][62][63] projecting the message of the eHMI on the street.External HMIs are interfaces installed on or projecting from the external surface of a vehicle.Indicators and brake lights are legally required realizations of eHMIs and highly standardized.Via the eHMI, the AV communicates with other HRU and provides information about the AV's status or its behavior.External HMIs could be clustered in auditory eHMIs [46,47] and visual eHMIs.In the context of the electrification of transport especially in low speed scenarios, hybrid electric or pure electric vehicles emit substantially less noise than comparable vehicles with combustion engines [48].Due to the missing engine sound, electric vehicles are less perceivable particularly for blind or visually impaired pedestrians [49].Auditory eHMIs could help these people, since quieter electric AVs are detected later in approaching scenarios than vehicles with a combustion engines [48].To counteract this fact, auditory eHMIs could have the potential to support communication between electric AVs and VRU.At present, visual eHMIs are almost exclusively investigated in research for AVs. Figure 5 structures visual eHMIs in four different categories, marking a vehicle as "driving automated" [50,51], light-strips [52][53][54] communicating via light patterns, displays [55][56][57][58][59] showing text or symbols, and laser projections [60][61][62][63] projecting the message of the eHMI on the street.[51], light-strips [54], display [59], and projection [63].
Depending on the eHMI type, different amounts of information and semantics can be transmitted to the recipient.Labelling a vehicle as "automated" via an eHMI provides the most general information content by communicating only the AV's status.Light-strips execute light patterns, which for example communicate by running from one side of the light-strip to the other side, or by flashing.Projections and displays are able to communicate the highest density of information, as these interfaces can be designed rich in detail.
At present, there are many studies researching the communication of AVs with pedestrians and other road users via external HMIs.Communicating via an eHMI increases the pedestrians' perceived safety [58,64,65] and the level of comfort [66] when crossing the street in case the AV shows its intention to yield via an eHMI.Moreover, eHMIs could reduce the time pedestrians need to cross the street in case the AV cedes the right of way [62].Additionally, Rettenmaier et al. [56] investigated the interaction between AVs and manual car drivers at bottleneck scenarios.External HMIs have been used to negotiate the right of way at road constrictions increasing the traffic flow and safety [56].Other applications are situations in which the AV is already stationary or moving slowly and can no longer communicate via its dynamics.One example is a deadlock scenario, which could be resolved by communicating externally via eHMIs [67].
In addition to the potential of eHMIs, there are still challenges to be explored and solved in the future.Currently, there are no defined standards or minimal requirements of eHMIs.Open questions need to be answered such as: "Which medium should eHMIs use?", "Which color should eHMIs be?" [68,69], or "Where should eHMIs be mounted?"[70].Moreover, researchers discuss whether eHMIs should inform other HRU about the AV's intention or whether the automated system should call the surrounding traffic for action.

Dynamic HMI (dHMI)
Following the notion that vehicle dynamics are not merely functional but also a form of nonverbal communication among road users (e.g., [71,72]), the term dHMI describes the AV's intended or unintended communication of information to passengers and surrounding HRU via  [51], light-strips [54], display [59], and projection [63].
Depending on the eHMI type, different amounts of information and semantics can be transmitted to the recipient.Labelling a vehicle as "automated" via an eHMI provides the most general information content by communicating only the AV's status.Light-strips execute light patterns, which for example communicate by running from one side of the light-strip to the other side, or by flashing.Projections and displays are able to communicate the highest density of information, as these interfaces can be designed rich in detail.
At present, there are many studies researching the communication of AVs with pedestrians and other road users via external HMIs.Communicating via an eHMI increases the pedestrians' perceived safety [58,64,65] and the level of comfort [66] when crossing the street in case the AV shows its intention to yield via an eHMI.Moreover, eHMIs could reduce the time pedestrians need to cross the street in case the AV cedes the right of way [62].Additionally, Rettenmaier et al. [56] investigated the interaction between AVs and manual car drivers at bottleneck scenarios.External HMIs have been used to negotiate the right of way at road constrictions increasing the traffic flow and safety [56].Other applications are situations in which the AV is already stationary or moving slowly and can no longer communicate via its dynamics.One example is a deadlock scenario, which could be resolved by communicating externally via eHMIs [67].
In addition to the potential of eHMIs, there are still challenges to be explored and solved in the future.Currently, there are no defined standards or minimal requirements of eHMIs.Open questions need to be answered such as: "Which medium should eHMIs use?", "Which color should eHMIs be?" [68,69], or "Where should eHMIs be mounted?"[70].Moreover, researchers discuss whether eHMIs should inform other HRU about the AV's intention or whether the automated system should call the surrounding traffic for action.Following the notion that vehicle dynamics are not merely functional but also a form of nonverbal communication among road users (e.g., [71,72]), the term dHMI describes the AV's intended or unintended communication of information to passengers and surrounding HRU via vehicle dynamics.Vehicle dynamics include both vehicle translatory dynamics (e.g., speed, lateral, and longitudinal accelerations) and vehicle rotational dynamics (e.g., active pitch or role motions of the vehicle's chassis).In automated driving, both forms of vehicle dynamics could be either intentionally designed to transport information to passengers and surrounding HRU, or they could be a result of functional driving maneuvers that are unintentionally understood to be communicative by HRU in a given context.The term dHMI covers both ways, and thereby emphasizes the importance of considering vehicle dynamics design for AVs.
Today, road users already commonly communicate via vehicle dynamics in traffic encounters.Human drivers use purposeful vehicle movement to communicate intent and to negotiate and cooperate with others (e.g., [72,73]).Observation studies investigating the communication behavior of road users in pedestrian crossing scenarios have found that vehicle movement is the most common form of communication [74,75].Similar results have been found from observations at bottleneck scenarios where drivers use offensive and defensive driving strategies to communicate their intent to either go first or let the other driver pass [76,77].Furthermore, surrounding HRU use a vehicle's movement to make assumptions about the driver's mood and character [71].Video analyses of traffic encounters show that unexpected motion behavior in manual [72] or in partially automated driving [71] can lead to discomfort and misunderstandings among HRU.Therefore, to replace the human driving behavior in automated driving, a considerate dHMI has to be designed that covers the communicational aspect of vehicle movement and is meaningful to passengers and surrounding HRU by matching their prior experiences and expectations.The potential of the dHMI is currently investigated for both internal and external communication.
From the perspective of external communication, the AV needs to clearly communicate its intent to other HRU to achieve safe and comfortable interactions in mixed traffic encounters.Several experimental studies have investigated the potential of the dHMI for external communication during pedestrian road crossings and lane changing and merging.During pedestrian road crossing, vehicle movement provides sufficient information for pedestrians to decide whether it is save to cross [57,[78][79][80].Furthermore, different motion design parameters and their combination (e.g., speed, onset of deceleration, and deceleration rate) affect the time HRU need to recognize the AV's intent to yield [81,82] and their gap acceptance [83].During lane changing and merging, the AV needs to clearly communicate both its intent to change lanes and its willingness to cooperate when other drivers want to merge into the lane (if the AV is the lag vehicle).Research has shown that the perceived cooperation of an AV is strongly influenced by the driving pattern before the lane change (in addition to an early communication of the AV's intent via the indicator) [84,85].Considering release conditions, an efficient automated lane change maneuver is more accepted by other HRU than one that is too conservative [86].The perception of the AV's willingness to cooperate as a reaction to a lane change announcement by a human driver is affected by the reaction time of the AV and the amount of deceleration and speed reduction [87].
In addition to external communication, the dHMI can be a new channel to communicate the AV's intent to the passenger.Research on lane change maneuvers in partially automated driving shows, that active pitch and role motions can be used to feedback information on the system's state and behavior (e.g., upcoming maneuver) to the passenger [88].A dHMI is less accepted if the driver is no longer responsible for supervising the system [89].Therefore, the acceptance of the dHMI by the passengers could depend on their involvement in the driving task, and, therefore on the functional scope of the automated system.
In summary, the dHMI is a powerful interface in many different situations to communicate to both passengers and surrounding HRU.It is crucial to consider the dHMI when designing an AV's interaction strategy because the AV's driving dynamics can be communicative even if they were not intended to be by designers and engineers (see also [72]).Research should further investigate the potential of the dHMI and how it could be combined with other HMIs (e.g., considering matching of dHMI and an additional visual eHMI [90]) to achieve one holistic communication strategy.Furthermore, the interplay of driving parameters and personal factors of the passenger and surrounding HRU needs to be considered during dHMI design (e.g., age see [83]).

Effect of Influencing Factors on HMI-Interrelations
As the functionality of the automated system becomes more and more advanced, the human machine interaction becomes more complex.In high levels of automation, the AV will have to be capable of communicating to passengers and surrounding HRU at the same time.During the design of an AV's communication strategy the effect of the influencing factors on the HMI types will have to be considered.In particular, adapting to rapid changes in the influencing factors during passing the scenario must be ensured.This interplay leads to changes in the HMI interrelations.Relevant interrelations are the concurrence for space, content transitions, practicability, consistency, and the chronological coordination.The five interrelations are represented by the connections between the different HMI types in Figure 1.

Concurrence for Space in the Interior
The concurrence for display space especially involves the in-vehicle communication strategy.The iHMI, the vHMI, and the aHMI use the same devices to some extent to communicate to the AV's passenger.Depending on the automation level, the passenger's information need varies.For example, during manual driving, the vHMI is more important than the aHMI since automated driving is not activated or available.When driving automated, the information of the aHMI concerning the capabilities and the maneuver of the AV becomes more important for the passenger.Since the display space is limited, an appropriate selection of relevant information must be made depending on the characteristic of the automation system.

Content Transitions
The second interrelation between the five HMI types concerns content transitions, which means that depending on the automation level the same information content changes from one HMI-type to another.An example is displaying the vehicle's speed.During driving in automation level 0 the speed belongs to vHMI since it informs about a driver input.When activating the cruise control in automation level 1 or driving in automation level 2 and higher, displaying the speed switches from vHMI to aHMI since the vehicle's speed is controlled by the automated system.

Feasibility
Feasibility is based on the fact that in some situations particular HMI types are no longer available, and therefore other HMI types would take on the communication.For example, in situations when the automated vehicle drives slowly (e.g., deadlock scenarios) the communication capabilities of the dHMI are limited, because the AV can no longer change longitudinal and lateral dynamics.Another example is when communication via lateral dynamics is not possible due to obstacles on the road.In these cases, the eHMI could have the potential to communicate externally with surrounding road users and it could replace the communication of the dHMI.

Consistency
The HMI framework and its components show that as compared to previous concepts, the important requirements of consistency have to be interpreted on a higher level of complexity.Assuming consistency of one HMI type means that information and dialog design is free from contradictions, the understanding in the framework is that contradicting messages over different HMI types have to also be avoided (see also [90]).

Chronological Coordination and Timing
The fifth interrelation between the HMI types concerns the chronological coordination of the five HMIs, as well as the time slots in which the HMI types have the capability to communicate to the passenger or to surrounding HRU.Since almost no research has been done analyzing the temporal coordination of internal and external communication as well as the coordination of dHMI and eHMI, there are still plenty of research gaps in the future.
Moreover, in addition to the chronological coordination of the different HMI types, it is essential to research the time slots in which the respective HMI has to communicate to ensure enough time to perceive and process the HMI's message.The identified time slots are strongly linked with the coordination of the HMI types because the time slots affect and limit the sequence and coordination of displaying the HMIs.

Future Work
Current research and studies cited in this contribution show that HMI concepts and their evaluation are mainly focused on one single HMI type or the combination of two different types in combination with selected levels of automation.It seems that all HMI types should be synchronized to communicate simultaneously.This assumption has to be investigated in detail in the future.Cramer et al. [91] presented indications that aHMI and dHMI could benefit from a given amount of asynchronicity, presenting the dHMI information in advance to aHMI.In future work, the temporal dialog coordination and the HMIs' synchronization have to be interpreted at a higher level of complexity.In addition, this coordination and synchronization could lead to increased complexity regarding body control modules (and vehicle bus communication), as different HMI types could be implemented on different on-board networks or modules.The strong dependency of the HMI types and the resulting additional coordination effort could increase demands on processing power and data exchange.Although this publication does not address technical feasibility, the holistic framework provides information on the interrelations that should be taken into account from an ergonomic perspective when using all HMI types in a future implementation.
In addition, it is necessary to investigate settings communicating via different HMI types and to include scenarios in which at least one communication fails and thus influences the communication of the other HMI types.This could mean that a passenger does not react to a request to intervene or a VRU misunderstands the AV's communication.The consequences of these unpredictable changes of situations need to be communicated to the passenger and to other HRU.

Conclusions
This work provides insights about the different HMI types in AVs, clustered according to their belonging to internal and external communication.In addition to a detailed explanation of which information could be assigned to which HMI, this paper analyzes and summarizes factors that influence HMI type selection and HMI content.The HMI framework combines the HMI types, their influencing factors, and the relationship between both parts.Moreover, the interrelation between the different HMI elements due to changes in the influencing factors' characteristics are presented.We created the framework to emphasize the importance of considering all HMI types when researching the interaction strategies of AVs with its passenger or surrounding HRU.Changing the design of a single interface could lead to an alteration of the influencing factors, which thereupon affects the HMI selection.Therefore, a holistic approach considering internal and external communication is indispensable when researching interaction strategies in the context of automated driving.

Figure 1 .
Figure 1.The HMI framework including the factors influencing the HMI characteristics and selection on the upper part, and the different HMI types dynamic HMI (dHMI), vehicle HMI (vHMI), infotainment HMI (iHMI), automation HMI (aHMI), and external HMI (eHMI) on the lower part.

Figure 1 .
Figure 1.The HMI framework including the factors influencing the HMI characteristics and selection on the upper part, and the different HMI types dynamic HMI (dHMI), vehicle HMI (vHMI), infotainment HMI (iHMI), automation HMI (aHMI), and external HMI (eHMI) on the lower part.

Figure 2 .
Figure 2. Static characteristics of an urban road scenery.The static influencing factors in this example include a two-lane road, a bike lane, and a zebra crossing.The traffic signs provide further information about the prevailing traffic rules.Additionally, the scenery includes houses and trees.

Figure 2 .
Figure 2. Static characteristics of an urban road scenery.The static influencing factors in this example include a two-lane road, a bike lane, and a zebra crossing.The traffic signs provide further information about the prevailing traffic rules.Additionally, the scenery includes houses and trees.

Figure 3 .
Figure 3. Dynamic and static characteristics of an urban road scenery.The dynamic influencing factors in this example include the AV (red) and additional moving HRU.The arrows represent the HRU's trajectories.The parking lorry in the center is also part of the dynamic elements because it could potentially continue driving anytime.

Figure 3 .
Figure 3. Dynamic and static characteristics of an urban road scenery.The dynamic influencing factors in this example include the AV (red) and additional moving HRU.The arrows represent the HRU's trajectories.The parking lorry in the center is also part of the dynamic elements because it could potentially continue driving anytime.

Figure 4 .
Figure 4. Most common visual HMI-elements used in aHMIs.(a) Instrument cluster, head-up display, and monitor in the center console used for presenting automation state and maneuvers [19].(b)Augmented marking of a front vehicle and the control range in the head-up display[23].(c) LEDstrip in the windshield to visualize the automation mode using color-coding[21].

Figure 4 .
Figure 4. Most common visual HMI-elements used in aHMIs.(a) Instrument cluster, head-up display, and monitor in the center console used for presenting automation state and maneuvers [19].(b) Augmented marking of a front vehicle and the control range in the head-up display [23].(c) LED-strip in the windshield to visualize the automation mode using color-coding [21].