Does the Driver Understand the Warning? Comprehension of the Request to Intervene

Featured Application

The authors predict that the results of their work will prove helpful in designing HMIs for use in automated vehicles. The results of the analysis show the characteristics of the pictograms with the greatest potential to induce correct driver reaction.

Abstract

Understanding the information conveyed by symbols is extremely important from the point of view of road safety. The symbols used on road signs and integrated into the vehicle must be clear and legible. This is especially important given the lack of regulations and standardization in the field of modern driving automation systems. Manufacturers are allowed to choose the way of informing the driver about the need to intervene in a situation when the vehicle is unable to continue driving in automated mode. The authors proposed to assess the comprehension of the information displayed by several different icons used to inform the driver about the need to take control. For this purpose, they used the adapted ISO 9186 Comprehension Test. The research was carried out on two different groups of drivers—passenger car drivers and professional bus drivers. A total of nine different pictograms were tested. The results indicate that drivers associate the red color with information about danger. The same icon in different colors may have a different effect on drivers, often leading to wrong responses.

1. Introduction

Proper comprehension of signs has been found to be a significant factor for traffic safety improvement [1,2,3]. Due to the complexity of road information, there is a strong need to support the driver with homogenized pictorial messages [4]. What is more, such messages need to meet numerous conditions to support the driver effectively. One of these conditions is comprehensibility, recognized as one of the key requirements in the usability-based approach [5,6]. In this context, comprehensibility, sometimes described as guessability, is a measure of cost to the user of a product when performing a task for the first time (the cost stems from uncertainty about how to do the task correctly). The higher the guessability, the lower the cost [6]. With the fast development of vehicle technologies, including powertrains and advanced driver assistant systems (ADAS), instrument cluster interfaces are also becoming complex, increasing the demand on drivers [7]. Guessability is especially important in those fields where symbols and signs do not have to conform to a worldwide standard. This is the case in the automotive industry, characterized by rapid technological changes and, consequently, requiring the introduction of new symbols and messages. While some of the ADAS symbols have already been a subject of extensive research and were standardized [3], most emerging technologies are independently developed by OEMs and producers and a significant lack of standardization is observed [8]. What is, therefore, particularly needed is research on icons. They can communicate information to the driver in a language-free and space-efficient manner [9], which is why the use of pictograms and symbols is highly recommended in road traffic [10]. Symbols convey messages in a concise form. In a “busy” environment, they are more noticeable, more quickly understood, and, therefore, more effective than a written message, as well as more persuasive than a spoken message. This observation is supported by studies on the duration of fixation on signs, which revealed its positive correlation with user’s difficulty in understanding the displayed sign [11,12] and in extracting information from it [13]. These findings are of great significance for drivers because slow comprehension of the observed signal may be reflected in extended response time (as when excessively long fixations are needed to understand the meaning of the pictogram) and wrong reaction to an event (as when the icon is misunderstood). According to McDougall et al. [14], the focus in research on symbols should be on features such as familiarity, concreteness, complexity, meaningfulness, and semantic distance. The self-evident features, such as color and shape, or ones that concern relations to other symbols, such as distinctiveness, are of lesser importance. Familiarity is the frequency with which a given icon has been encountered. Icons are concrete if they depict real objects, materials, or people. Those that do not are abstract. Icons are regarded as complex if they are intricate or contain many details, and they are considered simple if they contain few elements or details. Meaningfulness refers, rather obviously, to how meaningful people perceive icons to be. Finally, semantic distance is the closeness of the relationship between what is depicted in an icon and what it is intended to represent [1].

Numerous methods of testing comprehension have been discussed in the literature [15,16,17,18,19], as this field developed intensively in the 1990s. Studies have been conducted in various contexts to improve not only the symbols used but also the methods for testing them. The multicultural test conducted in Austria, Hungary, the Czech Republic, and Spain including 33 symbols of variable-message signs for 243 variants revealed a problem with the cross-cultural interpretation of signs, as only 28 signs reached the 85% comprehensibility level [4]. In a different study [20], symbols that were to be implemented in public spaces in a cross-cultural environment (airport) were successfully tested with various methods. Another cross-cultural study by Shinar et al. [21] showed a significant variation in the results of comprehension tests across four different countries (Canada, Finland, Israel, and Poland). The study by Yakut [22], designed to evaluate the implementation of signs for road traffic in Turkey as part of unification with other European countries, is also worth mentioning. Conducted using the multiple-choice questionnaire format, covering 20 different traffic signs and including 210 participants, the study initially showed good comprehensibility of the new signs (over 81%). It was followed by a broader study [2], carried out using open-ended questionnaires among 1487 participants from Ankara, and focused on assessing the comprehensibility of 30 selected traffic signs. Its results showed that most of the signs were not well known, only 12 of them were correctly identified by more than 70% of the participants, and five were identified as signifying the opposite of their actual meaning by more than 10% of the sample. These two significantly different results argue methods used for testing. A cross-cultural study was also performed in the context of preferences for unimodal and multimodal interfaces in take-over-request (TOR) situations [23] but yielded no significant comprehension and intelligibility results. In that study, responses of 1692 participants from 91 countries were analyzed in terms of opinions on various types of auditory, visual, and vibrotactile messages in five scenarios that differed in the level of urgency. The results are in line with the assumptions of our study, as they suggest that for urgent TOR situations, the most preferable signals are multimodal ones, while some simplification into an audio-only interface is preferred for system readiness signals.

Like the study reported in this paper, some research has used high-class simulators to assess the usability and comprehensibility of different interfaces in take-over-request scenarios [24]. Results clearly imply that the modality of the interface is crucial for proper reaction and that “the warning design in automated vehicles must be clearly differentiated from that of conventional manual driving vehicles.” [24]. Another simulation-based study was performed by Wang et al. [25], who tested whether visual and auditory icons would benefit the effectiveness of audio-visual multimodal warnings. Their study showed that simple multimodal signals were interpreted more quickly, which implied faster reaction, while more complex visual signals increased the intelligibility of the signs. These two facts do not combine into a single implementable conclusion, but they are an important result that shows the need for further research in this specific area.

2. Materials and Methods

System-initiated transitions in automated vehicles have to be introduced via different Human-Machine Interfaces (HMIs) by issuing a Request to Intervene (RtI). In the Trustonomy project, the effectiveness of individual HMI solutions was measured in two ways: Through the assessment of reaction time and through the assessment of ergonomics. The time-based approach, which is the focus of the article, is mainly based on comparing response parameters across different HMIs, complemented by additional subjective support measures based on the levels of message comprehension, perceived utility, and task load reported by participants.

The authors conducted research on the comprehension of messages conveyed by icons used to inform drivers about the need to intervene while driving an automated vehicle. A survey based on a standardized ISO 9186 Comprehension Test was used for this purpose [19]. The evaluation procedure proposed by ISO was matched to the pilot requirements and served as an inspiration to develop a tailor-made comprehension test. Using the questionnaire thus developed, the authors examined the effect of different HMI symbols as perceived by the participants. More specifically, the comprehension test was used for testing different referents and variants of an HMI element. A referent is an idea or object that a graphical symbol is intended to represent, while a variant is an alternative graphical design for a given referent. The participants (present in the laboratory) were to complete an online questionnaire, responding to three types of close-ended questions: “What do you think this symbol means?”, “What action should you take in response to this symbol?”, and “To what extent does this icon convey a Request to Intervene (RtI)?”. Each participant completed the same questionnaire. Please see Supplementary Materials for the comprehension test used to examine the effect of the pictograms designed by the ITS team.

In the beginning, the participants were asked to answer the first two questions, i.e., to guess what the symbol meant and what actions one should perform after noticing it. Then, after all pictograms were reviewed using the first two questions, the respondents were familiarized with the meaning of individual icons and asked to answer the third question, thus reporting the level of comprehension of each icon. All answers were provided in written form. The results were analyzed based on descriptive statistics.

The study was conducted in two locations: Warsaw (Motor Transport Institute; ITS) and Bednary near Poznań (Solaris). Different groups of users and different types of pictograms were tested in each location. In the ITS laboratory, the sample consisted of 81 subjects (51% women, 49% men) aged 18–65. These people had a category B driving license and had no experience in driving a vehicle equipped with ADAS. On the other hand, Solaris invited their test drivers to take part in the research, namely, professional drivers with a bus driving license. All of these 10 subjects were men aged 30–40. In the course of their employment with a bus manufacturer, these people had prior contact with many new features implemented in the brand’s vehicles. Nevertheless, they had not participated in the Trustonomy project before, did not know the research methodology, and, most importantly, had never seen the icons designed by the Solaris team before the experiment. As some Solaris buses were fitted with MobilEye Shield+, some of the test drivers may have used the device in the past.

In Warsaw, the understanding of 3 different icons designed by ITS employees was examined. All of them were implemented in a driving simulator equipped with an L4 level automation system (according to SAE). The icons that were assessed are shown in Figure 1.

In Bednary, 6 different pictograms were examined in the context of driving a bus equipped with ADAS, 3 pictograms that were implemented in MobilEye Shield+ devices (ME) and 3 others designed by Solaris (SOL) engineers that were intended to appear on the instrument cluster (Figure 2).

The study was performed in accordance with national and international standards. The research objective and method did not meet the conditions set out in the WMA Declaration of Helsinki that would have made it necessary to apply for approval from an independent ethics committee. Nevertheless, national standards for psychological research and assessment were applied, and the analysis was performed by a qualified specialist. The documentation and results are anonymized, stored, and processed in accordance with GDPR and national psychological regulations.

3. Results

This section, divided into three subsections, presents the results of research conducted in accordance with the methodology described in Section 2. Section 3.1 and Section 3.2 show the results of subjective message comprehension assessment (3.1—the study in the ITS laboratory, 3.2—the study at Solaris). Section 3.3 presents the results obtained in the ITS laboratory by asking the respondents an additional question about the best icon location.

3.1. ITS Results

The results of the CT (comprehension test) are shown below. Figure 3 provides information about the understanding of the RtI message conveyed by each of the icons presented in Figure 1. Correct answers are marked with a green rectangle. Almost half of the respondents stated that the “red hands on steering wheel” icon warned about the need to intervene, and this was the best result. Somewhat fewer participants, 31%, indicated that the same icon in natural color signaled RtI. The drop in comparison to the red icon may be due to drivers becoming accustomed to the way information is encoded on the dashboard. According to ISO 2575 “Road vehicles—Symbols for controls, indicators, and tell-tales” [26], red color is used to warn drivers. This type of lighting informs them about the car’s potential problem, safety issues, or immediate failure points. On the other hand, the orange wheel icon was not regarded as RtI-signaling. Respondents tended to think that the symbol suggested driving in automated mode, the system is in control of the vehicle (38%), locked steering wheel (30%), or had no idea what it meant (22%). This is also reflected on the right panel of Figure 3, showing that drivers either had no idea what they should do when noticing such a symbol (26%) or thought they should relax because the vehicle was in automated mode (44%). Both answers were incorrect.

Respondents reported that when noticing the “hands on wheel” icons, they were supposed to put their hands on the steering wheel (red 47%, natural 47%). Although this was not the best answer, it might be considered half-correct because regaining control (correct answer; red 30%, natural 12%) necessitates putting one’s hands back on the steering wheel. It should also be noted that some drivers misunderstood the meaning of the symbols by reading the pictogram as conveying the hand-off-wheel message. This was the case for 21% of respondents interpreting the natural-colored icon.

Next, participants were familiarized with the meaning of the symbols and asked to assess the extent to which each icon warned about RtI (Figure 4). It was the red icon that turned out to communicate such a warning most effectively, with the largest proportion of respondents (80%) indicating answers 4 (26%) and 5 (54%).

Participants did not rate the orange icon well. According to 41% of respondents (the second highest number of answers), the icon reflected its intended meaning to a very small extent.

3.2. Solaris Results

Figure 5 and Figure 6 provide information about the understanding of the RtI message conveyed by each of the icons presented in Figure 2. Correct answers are marked with a green rectangle. All the icons turned out to be comprehensible to the drivers. The most problematic one was the lane departure icon (implemented in MobilEye). Only 60% of the participants understood its meaning, while the others read its message as “the system is in control of the vehicle” (30%) or “the system has detected hands off the steering wheel” (10%). Both yellow icons (designed by Solaris) scored high on comprehensibility (90% each), but it is worth noting that 10% of drivers understood them as conveying the opposite of their intended message (“the system is in control of the vehicle). The same icon in red had a 100% level of comprehension, which may be related to the meaning of red in the automotive industry.

As regards the understanding of what action should be performed when the icon is displayed (Figure 6), four out of the six tested pictograms received 100% of correct answers. The icon with the worst result was the steering wheel on a yellow background, only 60% of the drivers knew that when seeing it displayed (on the dashboard, Head-up Display, etc.), they should react to avoid an accident. Seventy percent of the respondents would have reacted correctly to the lane departure icon. Both pictograms can be considered moderately comprehensible to the average driver and therefore need to be refined.

The participants were then asked to specify to what extent the presented icons conveyed RtI (Figure 7). For this purpose, a simple scale from 1 to 5 was used, where 5 was the highest rating. The results show that the red car and the red pedestrian are the best pictograms. They clearly show that a dangerous situation is imminent (red color) and, at the same time, define the type of threat. The average rating given to these icons is 4.60. The third place also belongs to an icon in red, a warning triangle on a red background, with an average score of 3.70 (−0.90). The same pictogram on a yellow background scored 3.50, while the lane departure icon and the steering wheel icon were rated the worst (3.20 in both cases). Based on these and previous results, these two icons cannot be regarded as correctly representing RtI.

3.3. Icon Location

At the end of the experiment held in the ITS laboratory, participants were asked to choose the best location of the RtI icon. The results are shown in the chart below (Figure 8). The largest number of drivers believed that displaying a big icon on the entire screen of the digital instrument cluster was the best way to give a visual warning conveying the need to take control (46%). About a third of the participants pointed to the windscreen display (HUD), while a small icon on the cluster panel received the fewest votes. This result is valuable because most vehicle manufacturers develop systems that inform the driver of their status by displaying or greying out a small icon on the dashboard (see, e.g., Skoda Travel Assist).

4. Discussion

The purpose of this research was to check how drivers understood the warnings conveyed by new pictograms used to inform them about the immediate need for the human driver to take back control of the vehicle. The results clearly indicate that using ISO 9186-1 [19] could improve the process of introducing new visual HMIs, including icons and dynamic information, facilitate the implementation process, and enhance user experience with new ADAS technologies. This supports the results reported by Campbell et al. [3] and justifies further implementation of this method in ADAS testing and development. From the perspective of users’ needs and preferences, this research indicates that for safety-critical signals, such as RtI, users prefer the red color. This might be connected with the typical coding rules for safety signs, namely, the ISO 3864 series [27], where various levels of urgency are indicated by yellow, orange, and red. This might also stem from people’s habituation to these rules being observed in other areas of life as well. On the other hand, while participants of our research fulfilled the same questionnaire, it might have an influence on the interpretation of the results. Nevertheless, our results clearly emphasize that the randomization of questions would not affect the results to a meaningful extent.

Short reaction time in the event of danger on the road is crucial for safe and efficient intervention. Reaction time depends on icon complexity. According to McDougall et al. [14], concrete icons depict real objects, materials, or people. The results obtained in the Trustonomy project support this thesis. However, the authors would like to extend it to include the activity factor, which means the pictogram itself should suggest the expected reaction (both hands-on-wheel icons from the first set were considered more obvious than the orange wheel icon). On the other hand, the results for the first set of icons (level 4 automation) suggest that showing only hands-on-wheel may not be enough. Novice automation users may understand the icon as conveying the opposite meaning, namely, as an instruction to remove their hands from the steering wheel. This case shows how important it is for driver training to be properly tailored to provide knowledge about the correct operation of the system and about the way the vehicle communicates with the driver. Moreover, HMI icon designers should consider introducing the environmental contextualization factor (as in ADAS) to provide the context of the driving scenario that requires driver reaction (type of danger). The results have shown that two of the three contextualized symbols (MobilEye icons) were understood better than the non-contextualized ones. Both pedestrian and car-warning icons were highly rated in terms of conveying the need to intervene. This leads to the conclusion that, apart from suggesting the expected action, a symbol should also facilitate regaining situational awareness by introducing the type of danger. In their future research, the authors would like to further investigate this issue, focusing especially on the actual take-over times in vehicles of different automation levels. More precisely, they intend to study the effect of action and/or danger-suggesting pictograms.

In the era of ADAS technology, the authors express the opinion that all drivers who use new supporting features should be treated as inexperienced, regardless of whether they are prospective drivers or professionals. This thesis is in line with previous studies, e.g., with the ones by Ng and Chan [1,28] and Al-Madani [29], where familiarity with rules was an important factor determining the level of sign comprehensibility. It is also supported by McDougal’s findings [14] and consistent with the results reported by Yun and Yang [24], indicating that the interfaces of “normal” driving should be clearly distinct from those of automated driving features, which will help achieve appropriate driver reaction. Additionally, the proper choice of modality was underlined, as some of the interfaces had multiple times longer reaction times than others.

The way to improve the intelligibility of icons and thus increase road safety is to introduce driving automation training that would improve drivers’ knowledge about the systems and their HMIs [8]. However, different producers use different methods (not only visual but also haptic or audio signals) to communicate system status to the driver, and there are no large-scale programs focused on the education of drivers. Drivers can take part in individual courses, such as SKODA Auto Szkoła [https://www.szkola-auto.pl/; accessed: 17 June 2022], but these are expensive. Some studies provide clear evidence of the effect of sign familiarity and driving experience on sign comprehensibility. In the study by Shinar et al. [21], infrequent signs were found to be more likely to be miscomprehended and less likely to be correctly learned. Similar results in the context of driving experience were obtained by Ng and Chan [1,28] and in Al-Madani’s [28] extensive research conducted in Bahrain, Qatar, and the United Arab Emirates. As reported by Kirmizioglu and Tuydes-Yaman [2], education was also provided as a solution to low comprehensibility problems in Turkey: “The signs that are partially known are the easiest to improve by simply focusing on the full extent of their meaning in driver education courses…”

An important limitation of this research is that it was conducted on a culturally and ethnically uniform group of Polish drivers. As reported by Siebenhandl et al. [4] and Shinar et al. [21], and as found in the FaceSymbol project [20], the same symbols might be interpreted in different ways by different cultural groups. This should be taken into account in research carried out by vehicle producers and OEMs during system development.