Special Issue "Test and Evaluation Methods for Human-Machine Interfaces of Automated Vehicles"

A special issue of Information (ISSN 2078-2489). This special issue belongs to the section "Information Applications".

Deadline for manuscript submissions: 31 May 2020.

Special Issue Editors

Dr. Frederik Naujoks
Website
Guest Editor
BMW Group, Munich, Germany
Interests: usability; human–machine interaction; human factors; automated driving
Dr. Sebastian Hergeth
Website
Guest Editor
BMW Group, Munich, Germany
Interests: Usability; Human-Machine Interaction; Human Factors; Automated Driving
Dr. Andreas Keinath
Website
Guest Editor
BMW Group, Munich, Germany
Interests: Usability; Human-Machine Interaction; Human Factors; Automated Driving
Dr. Nadja Schömig
Website
Guest Editor
Wuerzburg Institute for Traffic Sciences, Veitshöchheim, Germany
Interests: Usability; Human-Machine Interaction; Human Factors; Automated Driving
Ms. Katharina Wiedemann
Website
Guest Editor
Wuerzburg Institute for Traffic Sciences, Veitshöchheim, Germany
Interests: Usability; Human-Machine Interaction; Human Factors; Automated Driving

Special Issue Information

Dear Colleagues,

Today, OEMs and suppliers can rely on commonly agreed and standardized testing and evaluating methods for in-vehicle human–machine interfaces (HMIs). These have traditionally focused on the context of manually driven vehicles and put the evaluation of minimizing distraction effects and enhancing usability at their core (e.g., AAM guidelines or NHTSA visual distraction guidelines).

However, advances in automated driving systems (ADS) have already begun to change the driver’s role from actively driving the vehicle to monitoring the driving situation and being ready to intervene in partially automated driving (SAE L2). Higher levels of vehicle automation will likely only require the driver to act as a fallback ready user in case of system limits and malfunctions (SAE L3) or could even act without any fallback within their operational design domain (SAE L4). During the same trip, different levels of automation might be available to the driver (e.g., L2 in urban environments, L3 on highways). These developments require new test and evaluation methods for ADS, as available test methods cannot be easily transferred and adapted.

For example, The ADS HMI should be capable of informing the user about the current mode and minimize confusion about the status of the ADS and the user’s current responsibilities (e.g., whether the ADS is functioning properly, ready for use, unavailable for use or requesting a transition of control from the ADS to the user). While ADS might allow new and more comfortable seating positions and engagement in nondriving-related tasks that were not allowed in manual driving, these might generate motion sickness or lower the user’s availability for a transfer of control. As the driving task is no longer actively fulfilled by the driver, distraction by nondriving-related tasks might turn into controlled engagement. ADS might behave differently than manually driven vehicles, which might generate a need for external HMIs or standardized motion patterns to adequately interact with non-equipped traffic participants.

This Special Issue welcomes theoretical papers as well as empirical studies that deal with these new challenges by proposing new and innovative test methods in the evaluation of ADS HMIs in areas such as (but not limited to) the topics below:

- Mode awareness and mode indicators;
- Testing of minimum HMI requirements;
- Driver state in the context of ADS (e.g., distraction or drowsiness);
- Trust in ADS;
- External HMIs for ADS;
- Guidelines for HMIs for ADS;
- Motion sickness in ADS;
- Validity of test settings (on-road, driving simulators, etc.);
- Learnability and usability of ADS;
- Comfortable and pleasurable user experience of ADS.

Dr. Frederik Naujoks
Dr. Sebastian Hergeth
Dr. Andreas Keinath
Dr. Nadja Schömig
Ms. Katharina Wiedemann
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All papers will be peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Information is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1000 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • Automated driving
  • Human–machine interface
  • Test methods
  • User studies
  • Evaluation

Published Papers (6 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

Open AccessArticle
Supporting Drivers of Partially Automated Cars Through an Adaptive Digital In-Car Tutor
Information 2020, 11(4), 185; https://doi.org/10.3390/info11040185 - 30 Mar 2020
Abstract
Drivers struggle to understand how, and when, to safely use their cars’ complex automated functions. Training is necessary but costly and time consuming. A Digital In-Car Tutor (DIT) is proposed to support drivers in learning about, and trying out, their car automation during [...] Read more.
Drivers struggle to understand how, and when, to safely use their cars’ complex automated functions. Training is necessary but costly and time consuming. A Digital In-Car Tutor (DIT) is proposed to support drivers in learning about, and trying out, their car automation during regular drives. During this driving simulator study, we investigated the effects of a DIT prototype on appropriate automation use and take-over quality. The study had three sessions, each containing multiple driving scenarios. Participants needed to use the automation when they thought that it was safe, and turn it off if it was not. The control group read an information brochure before driving, while the experiment group received the DIT during the first driving session. DIT users showed more correct automation use and a better take-over quality during the first driving session. The DIT especially reduced inappropriate reliance behaviour throughout all sessions. Users of the DIT did show some under-trust during the last driving session. Overall, the concept of a DIT shows potential as a low-cost and time-saving solution for safe guided learning in partially automated cars. Full article
Show Figures

Figure 1

Open AccessArticle
Standardized Test Procedure for External Human–Machine Interfaces of Automated Vehicles
Information 2020, 11(3), 173; https://doi.org/10.3390/info11030173 - 24 Mar 2020
Abstract
Research on external human–machine interfaces (eHMIs) has recently become a major area of interest in the field of human factors research on automated driving. The broad variety of methodological approaches renders the current state of research inconclusive and comparisons between interface designs impossible. [...] Read more.
Research on external human–machine interfaces (eHMIs) has recently become a major area of interest in the field of human factors research on automated driving. The broad variety of methodological approaches renders the current state of research inconclusive and comparisons between interface designs impossible. To date, there are no standardized test procedures to evaluate and compare different design variants of eHMIs with each other and with interactions without eHMIs. This article presents a standardized test procedure that enables the effective usability evaluation of eHMI design solutions. First, the test procedure provides a methodological approach to deduce relevant use cases for the evaluation of an eHMI. In addition, we define specific usability requirements that must be fulfilled by an eHMI to be effective, efficient, and satisfying. To prove whether an eHMI meets the defined requirements, we have developed a test protocol for the empirical evaluation of an eHMI with a participant study. The article elucidates underlying considerations and details of the test protocol that serves as framework to measure the behavior and subjective evaluations of non-automated road users when interacting with automated vehicles in an experimental setting. The standardized test procedure provides a useful framework for researchers and practitioners. Full article
Show Figures

Figure 1

Open AccessArticle
The Impact of Situational Complexity and Familiarity on Takeover Quality in Uncritical Highly Automated Driving Scenarios
Information 2020, 11(2), 115; https://doi.org/10.3390/info11020115 - 20 Feb 2020
Abstract
In the development of highly automated driving systems (L3 and 4), much research has been done on the subject of driver takeover. Strong focus has been placed on the takeover quality. Previous research has shown that one of the main influencing factors is [...] Read more.
In the development of highly automated driving systems (L3 and 4), much research has been done on the subject of driver takeover. Strong focus has been placed on the takeover quality. Previous research has shown that one of the main influencing factors is the complexity of a traffic situation that has not been sufficiently addressed so far, as different approaches towards complexity exist. This paper differentiates between the objective complexity and the subjectively perceived complexity. In addition, the familiarity with a takeover situation is examined. Gold et al. show that repetition of takeover scenarios strongly influences the take-over performance. Yet, both complexity and familiarity have not been considered at the same time. Therefore, the aim of the present study is to examine the impact of objective complexity and familiarity on the subjectively perceived complexity and the resulting takeover quality. In a driving simulator study, participants are requested to take over vehicle control in an uncritical situation. Familiarity and objective complexity are varied by the number of surrounding vehicles and scenario repetitions. Subjective complexity is measured using the NASA-TLX; the takeover quality is gathered using the take-over controllability rating (TOC-Rating). The statistical evaluation results show that the parameters significantly influence the takeover quality. This is an important finding for the design of cognitive assistance systems for future highly automated and intelligent vehicles. Full article
Show Figures

Figure 1

Open AccessFeature PaperArticle
Repeated Usage of an L3 Motorway Chauffeur: Change of Evaluation and Usage
Information 2020, 11(2), 114; https://doi.org/10.3390/info11020114 - 18 Feb 2020
Abstract
Most studies on users’ perception of highly automated driving functions are focused on first contact/single usage. Nevertheless, it is expected that with repeated usage, acceptance and usage of automated driving functions might change this perception (behavioural adaptation). Changes can occur in drivers’ evaluation, [...] Read more.
Most studies on users’ perception of highly automated driving functions are focused on first contact/single usage. Nevertheless, it is expected that with repeated usage, acceptance and usage of automated driving functions might change this perception (behavioural adaptation). Changes can occur in drivers’ evaluation, in function usage and in drivers’ reactions to take-over situations. In a driving simulator study, N = 30 drivers used a level 3 (L3) automated driving function for motorways during six experimental sessions. They were free to activate/deactivate that system as they liked and to spend driving time on self-chosen side tasks. Results already show an increase of experienced trust and safety, together with an increase of time spent on side tasks between the first and fourth sessions. Furthermore, attention directed to the road decreases with growing experience with the system. The results are discussed with regard to the theory of behavioural adaptation. Results indicate that the adaptation of acceptance and usage of the highly automated driving function occurs rather quickly. At the same time, no behavioural adaptation for the reaction to take-over situations could be found. Full article
Show Figures

Figure 1

Open AccessArticle
External Human–Machine Interfaces: The Effect of Display Location on Crossing Intentions and Eye Movements
Information 2020, 11(1), 13; https://doi.org/10.3390/info11010013 - 24 Dec 2019
Cited by 2
Abstract
In the future, automated cars may feature external human–machine interfaces (eHMIs) to communicate relevant information to other road users. However, it is currently unknown where on the car the eHMI should be placed. In this study, 61 participants each viewed 36 animations of [...] Read more.
In the future, automated cars may feature external human–machine interfaces (eHMIs) to communicate relevant information to other road users. However, it is currently unknown where on the car the eHMI should be placed. In this study, 61 participants each viewed 36 animations of cars with eHMIs on either the roof, windscreen, grill, above the wheels, or a projection on the road. The eHMI showed ‘Waiting’ combined with a walking symbol 1.2 s before the car started to slow down, or ‘Driving’ while the car continued driving. Participants had to press and hold the spacebar when they felt it safe to cross. Results showed that, averaged over the period when the car approached and slowed down, the roof, windscreen, and grill eHMIs yielded the best performance (i.e., the highest spacebar press time). The projection and wheels eHMIs scored relatively poorly, yet still better than no eHMI. The wheels eHMI received a relatively high percentage of spacebar presses when the car appeared from a corner, a situation in which the roof, windscreen, and grill eHMIs were out of view. Eye-tracking analyses showed that the projection yielded dispersed eye movements, as participants scanned back and forth between the projection and the car. It is concluded that eHMIs should be presented on multiple sides of the car. A projection on the road is visually effortful for pedestrians, as it causes them to divide their attention between the projection and the car itself. Full article
Show Figures

Figure 1

Open AccessArticle
How Do eHMIs Affect Pedestrians’ Crossing Behavior? A Study Using a Head-Mounted Display Combined with a Motion Suit
Information 2019, 10(12), 386; https://doi.org/10.3390/info10120386 - 06 Dec 2019
Cited by 1
Abstract
In future trac, automated vehicles may be equipped with external human-machine interfaces (eHMIs) that can communicate with pedestrians. Previous research suggests that, during first encounters, pedestrians regard text-based eHMIs as clearer than light-based eHMIs. However, in much of the previous research, pedestrians were [...] Read more.
In future trac, automated vehicles may be equipped with external human-machine interfaces (eHMIs) that can communicate with pedestrians. Previous research suggests that, during first encounters, pedestrians regard text-based eHMIs as clearer than light-based eHMIs. However, in much of the previous research, pedestrians were asked to imagine crossing the road, and unable or not allowed to do so. We investigated the e ects of eHMIs on participants’ crossing behavior. Twenty-four participants were immersed in a virtual urban environment using a head-mounted display coupled to a motion-tracking suit. We manipulated the approaching vehicles’ behavior (yielding, nonyielding) and eHMI type (None, Text, Front Brake Lights). Participants could cross the road whenever they felt safe enough to do so. The results showed that forward walking velocities, as recorded at the pelvis, were, on average, higher when an eHMI was present compared to no eHMI if the vehicle yielded. In nonyielding conditions, participants showed a slight forward motion and refrained from crossing. An analysis of participants’ thorax angle indicated rotation towards the approaching vehicles and subsequent rotation towards the crossing path. It is concluded that results obtained via a setup in which participants can cross the road are similar to results from survey studies, with eHMIs yielding a higher crossing intention compared to no eHMI. The motion suit allows investigating pedestrian behaviors related to bodily attention and hesitation. Full article
Show Figures

Figure 1

Planned Papers

The below list represents only planned manuscripts. Some of these manuscripts have not been received by the Editorial Office yet. Papers submitted to MDPI journals are subject to peer-review.

Title: Methodological Considerations Concerning Motion Sickness Investigations During Automated Driving

Authors: Dominik Mühlbacher (WIVW), Katharina Reinmüller (Audi), Lena Rittger (Audi), Markus Tomzig (WIVW)

Abstract: Automated driving vehicles will allow all occupants to spend their time with various non-driving related tasks like relaxing, working, or reading during the journey. However, approx. one third of people are susceptible to motion sickness which limits the comfort of automated vehicles. Therefore, it is necessary to investigate the phenomenon of motion sickness while automated driving and to develop countermeasures. As most existing studies concerning motion sickness are fundamental research studies, a methodology for driving studies is missing yet. This paper discusses methodological aspects for investigating motion sickness in the context of driving including measurement tools, test environments, sample, and ethical restrictions. Additionally, methodological considerations guided by different underlying research questions and hypotheses are provided. Selected results from own studies concerning motion sickness during automated driving which were conducted in a motion-based driving simulation and a real vehicle are used to support the discussion.

 

Title: Human-Machine Interfaces in Automated Driving Systems: An extensive meta-analysis on the current state-of-the-art

Authors: Heikoop, D. D., Gürses, S. S., Happee, R., & Hagenzieker, M. P.

Abstract: Human-Machine Interfaces (HMIs) are gaining increasing interest among designers of automated driving systems. Vehicle designers and OEMs implement their HMIs for various reasons and in a plethora of designs, ranging from blind spot warning systems to vehicle control devices, while researchers investigate manoeuver suggestion systems to haptic guidance assistants. An all-encompassing overview of the literature is currently lacking, however, which this meta-analysis will provide. Using Google Scholar through Harzing’s Publish or Perish, a detailed overview of the literature was made. Particularly, this meta-analysis focused on HMIs between driver and automated vehicle, being specified as a four-wheeled public road vehicle. Four topics (Human Machine/Computer Interface/Interaction) and its plural, five adjectives (automated, autonomous, self-driving, cooperative, and driverless) and seven nouns (vehicle, car, pod, shuttle, driving, bus, and transport OR transit) made for (4*2*5*7=)280 search queries. Exclusion search words were, due to their abundance of irrelevant hits, ‘pedestrian’, ‘wheelchair’, ‘military’, and ‘business’. All queries combined provided 15,455 results, of which 7,183 papers remained after duplicate removal, 719 after title filtering, 451 after abstract filtering, and ultimately, 364 papers were deemed relevant to this study after having been fully read. These relevant papers were consequentially categorised based on study type: 167 experimental studies, 61 literature studies, and 106 patents. Those that fit neither category (30) were categorised as other. Then, an extensive accumulation of over 20 different characteristics, ranging from type of HMI modality to level of simulator fidelity, and from amount of participants to questionnaire type used, resulted in a wide knowledge base of the current state-of-the-art concerning HMIs in automated driving systems, to be used by, among others, designers, OEMs and researchers.

 

Title: The system desirability scale: A scale for the assessment of a system’s or a product’s user appeal

Authors: Purucker, C., Befelein, D., Tomzig, M., Naujoks, F., Hergeth, S., & Keinath, A.

Abstract: The empirical testing of a system’s usability is a standard procedure in the automotive sector when it comes to the development of new input and output devices, as well as novel infotainment systems, driver assistance systems and vehicle automation functions. While most established procedures focus on core aspects of system usage and the system’s distraction potential when used while driving, the customer appeal is usually not a core module in these participant tests, although such a combined assessment would yield several advantages. In the current work, we propose an assessment scale for the measurement of a system’s (or a product’s) desirability – the capability to elicit a strong desire to own or use the product. We derive the concept from two expert studies and investigate the scale’s psychometric qualities in a study with N = 30 participants. A post-survey study with N = 21 participants from the main study provides an additional outlook on the temporal development of the measure.

 

Title: Usability evaluation – Advances in experimental design in the context of automated driving Human-Machine-Interfaces

Authors: Deike Albers, Jonas Radlmayr, Alexandra Löw, Annika Boos, Sebastian Hergeth, Frederik Naujoks, Andreas Keinath, Klaus Bengler

Abstract: The projected introduction of conditional automated driving systems to the market has sparked multifaceted research on human-machine-interfaces (HMIs) for such systems. By moderating the roles of the human driver and driving automation system, the HMI is indispensable in avoiding side effects of automation such as mode confusion, misuse, and disuse. In addition to safety aspects, the usability of HMIs plays a vital role in facilitating appropriate trust and acceptance towards the automated driving system. This paper aggregates common research methods and findings based on an extensive literature review. Empirical studies as well as frameworks and review articles are included. In the first part, findings and conclusions are presented focusing on study characteristics such as use cases, dependent variables, testing environments or participant samples. In the second part the methods and findings are discussed critically considering minimum requirements for automated driving defined by the NHTSA. The paper concludes with a derivation of recommended study characteristics framing a proposal for an experimental design to assess the usability of HMIs in the context of automated driving. The advised selection of scenarios and metrics will be applied in a future validation study series comprising a driving simulator experiment and three real driving experiments on test tracks in Germany, the USA, and Japan.

Keywords: Conditionally Automated Driving, Human-Machine-Interface, Usability, Validity, Method Development

 

Title: Methodological approach towards evaluating the effects of non-driving related tasks during partially automated driving

Authors: Schmidt, C., Rauh, N., Naujoks, F., Hergeth, S., Krems, J. F., & Keinath, A.

Partially automated driving (PAD, SAE Level 2) features provide steering and brake / acceleration support to the driver, whereas the driver must constantly supervise the support feature and steer, brake or accelerate as need to maintain safety. Thus, PAD has the potential for contributing to increases in comfort and road safety. However, previous research indicates that supervising automation may create unintended side effects such as reduced situation awareness. In addition, drivers may increasingly engage in non-driving related tasks (NDRTs). Accordingly, some research has focused on the effects of NDRTs on the driver during PAD. At the same time, there are no broadly established methodologies and guidelines for this research area such as driver distraction guidelines for manual driving.

The current project’s goal was to take first steps towards developing a methodology for systematically evaluating the effects of NDRTs during PAD. The methodologies for manual driving built the basis for this and were extended to PAD. In the course of three studies, two in a simulator and one on a closed circuit, the methodology was evaluated and refined. Two generic take-over situations, addressing the system limits of a given PAD regarding longitudinal and lateral control, were implemented as test scenarios to evaluate drivers’ monitoring and take-over capabilities while engaging in different NDRTs (e.g. radio tuning task). The results indicate that the methodology was sensitive to detect differences between the NDRTs’ influences on the drivers’ take-over and especially monitoring capabilities.

Keywords: Partially automated driving; non-driving related tasks; take-over situations; methodology development; user studies (simulator, closed circuit)

 

Title: Comparison of Methods to Evaluate the Influence of Automated Vehicle’s Driving Behavior on Pedestrians: Wizard of Oz, Virtual Reality, and Video

Tanja Fuest 1*, Elisabeth Schmidt 2 and Klaus Bengler 1

1   Chair of Ergonomics, Technical University of Munich

2   BMW Group, New Technologies

Abstract: Integrating automated vehicles into mixed traffic involves several challenges. Questions regarding vehicle marking or additional external human machine interfaces must be clarified. In addition, a driving behavior that is understandable for all human road users has to be designed to ensure a smooth and safe traffic. Several studies were already carried out to investigate these issues, especially regarding the communication between automated vehicles and pedestrians. These studies used different methods, e.g. videos, virtual reality, or Wizard of Oz vehicles. However, the transferability of these studies to real life traffic situations is still unknown. Therefore, we replicated the same study design in the three different settings: videos, virtual reality, and Wizard of Oz. In all studies, participants stood at the roadside in a shared space. An automated vehicle approached from the left, using different driving profiles characterized by changing speed to communicate its intention of letting the pedestrians cross the road. Participants were asked to recognize the intention of the automated vehicle and to press a button as soon as they had understood this intention. Results revealed differences in the intention recognition time between the three study setups, as well as in the correct intention rate. The results from different study settings can therefore only be compared to a limited extent.

 

Title: Mode Awareness – What is it and how can it be measured?

Abstract: In SAE level 2, the driver has to monitor the traffic situation and system performance at all times, whereas the system assumes responsibility within a certain ODD in SAE level 3. The different responsibility allocation in these automation modes requires the driver to always be aware of the currently active system and its limits to ensure a safe drive. For that reason, current research focuses on identifying factors that might promote mode awareness. There is however no gold standard for measuring mode awareness and different researches have different approaches to this highly complex construct. This circumstance surely complicates the comparability and the validity of study results. We thus propose a measurement method that combines both the knowledge and the behavior pillar of mode awareness. The latter is represented by the relational attention ratio in manual, level 2 and level 3 driving as well as the controllability of a system limit in level 2. The behavior aspect of mode awareness is operationalized by a questionnaire on the participants’ mental model of the automation systems after an initial instruction as well as an extensive enquiry following the automated driving sequence. Further assessments of system trust, engagement in non-driving related tasks and subjective mode awareness are proposed.

Back to TopTop