Research and Design Considerations for Presentation of Non-Safety Related Information via In-Vehicle Displays during Automated Driving

Featured Application

This paper presents a review of current knowledge regarding factors affecting presentation of non-safety related information on in-vehicle displays. A set of proposed design guidelines, based on literature review, are presented along with open research questions.

Abstract

As automated vehicles become more prevalent on roadways, it is necessary to study driver behaviors in interacting with such systems. With higher levels of vehicle automation, drivers may become less engaged with the roadway environment. As a result, how to effectively bring non-safety related information (e.g., guide and service sign content) to a driver’s attention is an open research question. In this review, we summarize current literature on three domains of research, including: (1) the design and effectiveness of traditional road signage, (2) human factors considerations in vehicle automation design, and (3) current design guidelines for in-vehicle information presentation. Based on the review, including empirical studies, we identify knowledge relevant to communicating road signage information in automated vehicles. We propose a framework highlighting various factors that could determine the effectiveness of in-vehicle messaging. The framework is intended to motivate future research on development of in-vehicle interfaces for highly automated driving.

1. Introduction

With the rapid development of sensor and computing technologies, personal vehicles are now capable of gathering large amounts of information on vehicle status and the roadway environment, as well as making proximity estimates and predicting potential driving events. Recent advances in vehicle automation have envisioned future driving without the need for drivers to attend to the road. Ford previously announced delivery of a high production fully automated vehicle by 2021 [1]. Volvo, Nissan, Honda, Toyota, and BMW have all promised similar timelines [2,3,4,5]. Federal and state legislation is also responding to this rapid technological change [6,7]. These vehicles will be fully equipped with information systems for navigation, communication, and entertainment, which will continuously capture much of the driver’s attention inside the vehicle. As a result, there is a potential shift in information communication from driver-roadway interaction to driver and in-vehicle display interaction. In this case, drivers may be less likely to notice road signs and rather increasingly rely on a GPS device to notify them of a speed limit change or beginning of a school zone. While in-vehicle notification of such information can overcome issues with road signs, such as poor visibility due to weather and maintenance problems, it poses significant human factors challenges on how to effectively deliver signage information while minimizing potential distraction and keeping the driver aware of the road environment.

Despite decades of research on in-vehicle notification designs (see [7] for a complete review), most studies have focused on presenting information that is critical for driving safety, such as collision warnings and lane departure warnings. In contrast, how to effectively present information that is non-safety critical and secondary to driving but important for a trip (e.g., specific roadside services), remains unexplored, especially in the context of highly automated driving. As driver attention could differ based on the relevance or importance of a piece of information, results of studies that have focused on the impact of using in-vehicle information systems on manual driving [8,9,10] or the effectiveness of collision warning designs [11,12,13] may be informative but may not necessarily generalize to non-safety critical notifications under automated driving. If a display notification is not directly relevant to a primary driving task, drivers may exhibit lower levels of attention, which may be compounded by degraded alertness during automated driving [14,15,16].

This literature review provides an overview of the current understanding of driver interaction with road signage and automated driving technologies as well as factors that affect driver performance, including characteristics of the driver, the environment, vehicle automation, and the in-vehicle display. We identify research gaps that need to be addressed to inform the design of in-vehicle messaging of road signage information under highly automated driving conditions and propose a framework to guide further empirical investigations.

2. Roadway Signage and Human Factors Issues

Roadway signage is one of the most common forms of traffic control device. Signs utilize words, pictorial elements, or a combination of these to convey information [17]. Although extensive design guidelines have been developed for roadway signs [18], these guidelines do not apply to ensuring readability and comprehension of signage information when presented via an in-vehicle display. Sanders and McCormick identified five ergonomic principles related to the development of traffic signs, including: (1) spatial compatibility, (2) conceptual compatibility, (3) physical representation, (4) familiarity, and (5) standardization. Spatial compatibility refers to a sign’s physical position and orientation in space [19]. For example, in a right-side driving environment, a stop sign is always placed to the driver’s right at an intersection. Conceptual compatibility refers to the degree to which symbols and words presented on a sign match driver expectations. For example, a stop sign is always octagonal in shape, so drivers associate the shape with the act of stopping. Physical representation refers to the degree to which the content of a sign represents reality. In this case, the sign needs to accurately inform a driver of a roadway circumstance (e.g., falling rocks). Familiarity refers to the extent of the driver’s experience with a specific sign. Some drivers may encounter signs that are uncommon (e.g., minimum freeway speed limit) and may be confused as to the meaning. This lack of familiarity can be an issue, if the sign is designed to serve as a warning or in cases of driving in foreign countries. The last guideline is standardization, which refers to the level of consistency in the design of the signs. Shape, color, and pictorial elements should be consistent for each specific type of sign; otherwise, driver interpretation can be compromised if different designs are used for the same purpose. Shinar & Vogelzang [20] found that pairing pictorial information with text on road signage presentation improved interpretation accuracy, even when the sign was unfamiliar to the driver. While the presentation of roadway signage information through an in-vehicle display potentially allows for more creative designs, as well as greater flexibility in timing and duration of communication, such displays may still benefit from consideration of existing ergonomic principles for roadway signs due to driver prior adaptation.

Beyond the formatting of signs, the physical location of the sign on the roadway is another important human factors consideration. Although many drivers have experience with different types of roadway signs, there are barriers to recognition due to the signing environment and sign configuration [21,22]. For example, driver awareness of a road sign can be easily affected by weather (e.g., fog vs. clear), lighting conditions (e.g., night vs. day), vegetation (e.g., tree branches covering a sign), sign legibility (e.g., font size of a street name being too small to see), and language used in signs (e.g., foreign drivers face difficulty in understanding local signs).

In addition, driver compliance with a road sign may be low, even if they comprehend the sign, as they may not associate the sign with a necessary action. For example, a driver may ignore the need to stop at a stop sign when there is no traffic at an intersection. In a comparison of studies on driver compliance with conventional stop signs between 1931 to 1999, traffic counts revealed a sharp decline in the percentage of drivers who made a full stop at stop signs from 47% in 1931 to 1% in 1996 [23] (p. 2775, Table XIV). Drivers in full violation of a stop sign increased from 42% in 1931 to 97% in 1996. With higher levels of vehicle automation, drivers may not need to perform some or all aspects of the driving task, thus their visual scanning and vigilance of the roadway may degrade, as compared to manual driving [24]. As a result, driver processing of signage on the road could be further limited or even abandoned. In contrast, in-vehicle messaging of signage information presents an opportunity to address signage limitations and draw driver attention to such information.

3. Highly Automated Driving

Many car manufacturers are releasing increasing numbers of automated vehicles (e.g., Tesla (Model Y), Volvo (SC-90), BMW (i8), General Motors (Volt) and Ford (Mustang, F150)), which promise to improve safety and reduce the number of accidents and injuries caused by manual driving [25]. The Society for Automotive Engineers (SAE) has defined six levels of automated vehicles to determine the capabilities of a vehicle with varying degrees of vehicle automation [26]. Among the six levels of automation, vehicles with SAE Level 1 automation provide vehicle longitudinal position control, only (e.g., adaptive cruise control). Level 2 automation can maintain lane position and adaptively control speed, but the driver is responsible to monitor the roadway to safely respond to hazards and determine the suitability of using certain automated functions. Examples of Level 2 automated vehicle technologies include adaptive cruise control, active lane-keeping, and forward collision warning systems [27]. These types of technologies are generally referred to as Advanced Driving Assistance Systems (ADAS). Several manufacturers have included some, if not all, of these technologies in current generations of vehicle models.

SAE Level 3 automation represents “partial driving automation”. At this level, the human driver is not required to monitor automated vehicle control under specific driving circumstances. The vehicle automation makes informed decisions about driving tasks, but the driver must remain available to takeover control, if the automation is incapable of processing specific conditions. Unfortunately, when Level 3 is active, a driver may be disengaged from vehicle monitoring and, therefore, not retake control in a timely manner [28]. Some companies claim to have produced vehicles that reach Level 3 automation [29,30]. The Audi R8 is the only current production vehicle that achieves this SAE level. The vehicle can take drivers to destinations within mapped areas, as a driver tells the car where they want to go. However, the driver is still expected to be prepared to takeover control when necessary.

While higher levels of vehicle automation have the potential to ultimately reduce crashes involving driver error, no current production vehicles exist that achieve higher SAE levels of driving automation than Level 3. There are also several human factors concerns that have been expressed in relation to autonomous driving systems [31], including: automation mode confusion, locus of control, overreliance, and driver underload. Mode confusion refers to the potential for the driver to misunderstand their responsibilities during automated driving. Locus of control refers to who (the driver or automation) has final decision authority in response to hazard situations and whether the allocation of authority is consistent with situational demands. For example, if a driver has enabled automated vehicle control, and there is a hazard in the roadway, the car may respond without ceding control to the driver and violate expectations. Overreliance occurs when drivers attempt to engage vehicle automation, as frequently as possible, despite know system limitations. Overreliance can lead to degraded driver situation awareness and capability to takeover control when necessary. Finally, driver underload refers to the situation when automated operation of the vehicle leads to driver disengagement and boredom.

Each of these potential human factors issues can affect a driver’s capability to process non-safety related information presented in the vehicle cockpit. For example, if the driver is unsure about the state of vehicle automation, there may be confusion as to whether the vehicle will correctly navigate to a pre-programmed route (e.g., taking a food exit on a highway). In this case, the driver may attempt to takeover vehicle control. On the other hand, drivers may also fully trust the vehicle and may fail to takeover control when the vehicle erroneously passes the intended exit. This situation can result in driver re-routing. Highway driving situations, like this scenario, can be dangerous when there is driver uncertainty in the use of automation and any attempt to immediately reverse a vehicle control action. Consequently, there is a need for effectively communicating automation state as well as non-safety related information during automated driving. There is a need to consider the interaction of the driver and the vehicle automation when designing messaging formats.

3.1. The Out-of-the-Loop (OOTL) Performance Problem

A major human factors concern with highly automated systems is that human use typically involves monitoring activity, which can lead to boredom, vigilance decrements and degraded operator system/situation awareness. Such behavioral and cognitive states can produce performance decrements, for example, when drivers need to takeover automated vehicle control [32]. A major difference between automated and manual driving is that driver perception of visual information is altered because their attention may be focused internally to the vehicle and not on the driving environment [33]. During manual driving, the driver is an active observer of roadway conditions and responds accordingly. In contrast, in automated driving, the driver becomes a passive observer, as they are not in direct control of the vehicle. In this passive-viewing state, the driver is likely to disengage from all information related to driving and, therefore, less likely to perceive vehicle messaging on operational control (e.g., speed), maneuvering (e.g., lane changes, cornering), route navigation, and systems status. This lack of awareness, or OOTL performance problem, can translate to decrements in driver takeover during automated driving.

Out-of-the-loop performance problems can result from cognitive disengagement from a task and/or a lack of physical interaction with an operational system. When a driver is not required to control steering, acceleration, and/or braking, an out-of-the-loop state can arise [34]. In addition to the physical control loop, this problem can also develop when a driver is disengaged from a cognitive control loop [34]. In this case, the driver loses situation awareness on the current state of the vehicle either because they are not viewing the roadway, they are disengaged from the driving task due to a secondary task, or the vehicle has simply taken over control from the driver. Related to this, research on mind wandering while driving has revealed negative effects on driving performance due to cognitive disengagement [35,36,37,38]. When a driver’s mind becomes disengaged from the driving task, a range of impairments in driving performance can occur, including increased variability in vehicle speed control, slower reaction time, reduced visual scanning of the environment, and poorer recognition/memory of the visual environment [35,36,37,38]. Finally, driver physical and cognitive control loops are intertwined, as physical control of a vehicle provides neuromuscular feedback to the driver, which can then be cognitively translated into vehicle heading corrections through adjustments in steering [39].

Both administrative and design methods can be used to address the out-of-the-loop performance problems in complex systems. For example, driver takeover of vehicle control can be scheduled in a predictable manner. Merat et al. compared driver performance during a takeover request under conditions where automated driving to manual driving alternated at regular, system-based intervals, or based on the duration of a driver’s gaze being away from the road [40]. The study found that driver engagement was higher with system-based intervals as it allowed for expectation and preparation. When drivers expect a takeover event, they are primed to attend to the dynamic situation inside and out of the vehicle, such as automation states, navigation information and lane position, speed, surrounding traffic, etc. By priming driver attentional allocation to these elements, and supporting situation awareness, they are able to resume control more safely and prevent out-of-the-loop performance decrements.

The out-of-the-loop performance problem and takeover control are critical to the design of in-vehicle messaging. A driver may need to assume vehicle control to achieve a particular business destination. In this situation, a road sign display and takeover notification may occur simultaneously. This coupling of information in the vehicle and external to the vehicle may present significant challenges because the driver needs to channel attentional resources to multiple (separate) stimuli. A solution to this problem is to present road sign information through in-vehicle displays. This would consolidate information on the external environment and the vehicle itself in the same spatial location. This approach may improve driver performance when taking over control and navigating to a desired destination.

Nonetheless, processing of in-vehicle message displays represents another secondary task to the primary driving task. Several studies have observed that drivers tend to engage in non-driving-related activities during highly automated driving. As a result of the driver not manually operating the car, they are more likely to engage in other tasks that divert attention away from driving [41]. However, recent studies in this area have demonstrated conflicting results regarding the effects of engagement in a secondary activity on takeover performance. Some studies have found that non-driving related tasks have a similar effect to that of distracted driving during manual driving [42,43]. Other studies, in contrast, have shown non-driving-related activities to be beneficial for takeover performance [14,44]. For example, Miller et al. found that driver fatigue was lower when engaged in a non-driving-related task, such as watching a television show on a tablet or reading, during an automated drive [14]. Without non-driving-related tasks, drivers showed signs of drowsiness, which is a dangerous condition when vehicle takeover may be required. Clark and colleagues examined takeover performance as drivers voluntarily engaged in non-driving-related activities during automated driving [45,46]. In general, they found that such activities did not impair driver takeover performance, which is consistent with findings of benefit from a secondary task on a primary vigilance task [47]. Furthermore, regardless of the modality of the voluntarily secondary activity, participants showed consistent takeover performance [45]. However, there is some additional evidence that longer activity engagement tends to be associated with slower takeover response after a notification [46].

3.2. Driver Mistrust of Driving Assistance Systems

Another factor that impacts driver performance during highly automated driving is the degree of trust between the human and the machine. The nature of automated driving means that vehicle automation is making decisions on behalf of the driver and, as a result, the driver must trust that the vehicle is operating safely [48]. Previous research in this area has shown that use of simulated autonomous vehicles can result in significant physiological stress [49]. Other work has shown that driver trust in automated vehicle technologies varies depending on the type of automation [50]. For example, drivers tend to trust “side-view assist” technology more than “active lane keeping” systems [50]. The degree of trust between driver and vehicle in autonomous driving is of interest because trust can determine the likelihood of driver usage of automation [1]. A lack of trust may result in driver disregard of, for example, in-vehicle messaging display. This situation could be critical for safety related warnings, but may have less impact for non-safety related information, such as trip information. A driver may choose to ignore a notification of a nearby service station with little consequence; whereas, ignoring GPS guidance to a hidden turn could be more costly from a trip perspective. Driver trust has not been investigated across safety and non-safety related messaging systems. In either case, implementations of ADAS should ensure that content of in-vehicle messaging is kept up-to-date to foster driver trust in the use of such systems and potentially mitigate the variations in driver trust observed among specific ADAS [50,51].

4. Advanced In-Vehicle Displays

In-vehicle displays, and in-vehicle information systems, provide additional information to the driver. As a result, they can have significant effects on driver behavior. The implementation of in-vehicle information systems (IVIS) in cars can increase the number of tasks that a driver must perform, concurrently [52]. During manual driving, use of IVISs has been observed to degrade performance by overloading the driver, thus resulting in safety concerns [53,54]. A previous naturalistic driving study with 100 participants showed that 78% of crashes and 65% of near-crashes were the result of driver distraction, of which in-vehicle technology use accounted for roughly one-quarter of the events [55]. While this study examined manual driving, driver distraction due to different types of in-vehicle displays could also affect the capability of drivers to respond quickly to automation takeover requests.

Research has found that drivers tend to adapt their behavior in various ways to compensate for the presence of IVISs, but these adaptations vary between being beneficial and detrimental [56,57,58]. One of the issues is that IVISs create secondary tasks for the driver, which in some cases may increase workload to the point that driver response to environmental events is negatively impacted [59]. For example, IVIS displays that require manual input were found to increase workload and were related to increased, center line crossings, and off-road accidents in a simulator study [53]. Compounding these effects, drivers also differ in the way they interact with IVIS displays. Higher risk drivers demonstrate longer eyes-off-road times (EOR) than lower risk drivers, even when performing tasks that the former group rated as higher risk, such as typing in a street address while driving [60]. However, driving performance with an IVIS has been found to improve with time, as individuals become more proficient with systems, and tends to follow the power law of practice [57].

While it is known that IVIS displays impact driver performance during manual driving, those observed performance decrements may not apply to automated driving. When the driver is not required to physically control the car during highly automated driving, an IVIS can be used for more than simply displaying navigation information or vehicle status information. A recent survey of user preferences for activities during automated driving revealed that instead of doing nothing, people prefer listening to music or entertainment, engaging in communication or productivity tasks during an automated drive [61]. Consequently, an effective IVIS display of messages may recapture driver attention from the secondary activity. Much of the work in this area has focused on messaging safety-critical information (e.g., forward collision warnings [12,62]) to a driver in manual driving, as reviewed below, but few studies have considered non-safety-related information.

4.1. NHTSA Guidelines for In-Vehicle Messaging

NHTSA developed a series of design guidelines for driver vehicle interfaces including those for presenting information during manual driving [7]. The current guidelines state that messaging content needs to be designed to pose minimal additional workload and not obstruct a driver’s capability to process information from the roadway [7]. As processing information from an in-vehicle display may demand the same pool of perceptual and cognitive resources a driver needs to operate the vehicle, poor interface design could lead to distracted driving [63]. In specific, NHTSA recommendations state that displays should support tasks that can be completed with sequential glances that are brief enough not to affect driving and tasks that do not require the driver to make time sensitive responses [7]. These guidelines are supported by previous work that has empirically investigated driver distraction (for a comprehensive review see [64]). During manual driving, effective messaging content needs to be informative without increasing workload, but these needs may become more complex in automated driving, given the general underload issue, and sudden unexpected spike of workload in the event of an automation takeover.

The design of in-vehicle messaging for automated driving needs to consider the three phases of driver display information processing, including: extraction, recognition, and interpretation [62]. Extraction relates to how easily the message can be perceived by the driver; recognition refers to the structure of the message and whether it accurately represents the information it is trying to convey to the driver; and interpretation depends on the capability of the driver to understand the message [7]. While these phases have been considered in manual driving systems design, they may also apply to automated driving, with the additional factor that drivers will likely disengage from the driving task. For example, NHTSA released guidelines for how the elements in a visual display should be presented to optimize the three phases of message presentation and processing. Specifically, the use of certain representations can aid in driver processing of information. One example is the use of displays that present the criticality of warnings in terms of scaled distance to a crash point. Another example is the use of symbolic or pictorial information to aid understanding without the need to read text [7]. NHTSA also recommends symbolic or iconic images to give meaning to analog displays, such as a collision warning, so that drivers do not need to read for interpretation. Spatial information can be used to represent intersection configurations and lane locations. Representational information can be used when specific locations or destinations need be conveyed to the driver, such as a merge lane position [7]. The use of stimulus-response compatibility, or consistency between the type of display and type of information being displayed, allows these specific representations to aid driver understanding of information and the effectiveness of such design elements has been supported empirically [7,65,66].

Another issue is the amount of data being presented in a display, which can result in display clutter. Research has indicated that the amount of attention devoted to a display increases with an increasing amount of clutter [67]. Pankok and Kaber extended these findings and observed that in higher workload situations, drivers exhibit shorter glances to displays to account for increased clutter [68]. This behavior occurs despite drivers allocating insufficient time (with short glances) to process information. Altogether, these findings suggest that the design of in-vehicle messaging needs to pay careful attention to driver workload, the ability of the drivers to process information, and limiting display clutter to assist the driver in perceiving necessary information.

4.2. Advanced Display Technologies

Advanced Driver Assistance Systems are being developed to offload some cognitive demands from drivers to vehicle automation. Many vehicles now include visual status displays to support drivers in automated driving tasks. Since it is difficult to predict where a driver will be looking at each moment, the type and amount of information to display, as well as presentation formats, remain open questions. For example, Head-Up Displays (HUDs) are being integrated in vehicles as part of a current trend in vehicle engineering [69,70]. These displays present information on the windshield of the vehicle so that it is within the driver’s field of view as they gaze at the roadway [71]. For automated driving, HUDs are being considered for presenting important trip-related information [72]. These displays have shown promise in reducing eyes-off-the-road and are more effective than Head-Down Displays (HDDs) in the presentation of navigation and safety information [73]. However, a major issue is such displays can consume driver attention to exclusion of the roadway, even though the road scene is generally within the visual field [59,70,74,75]. In a recent study of the effectiveness of HUDs in automated driving [70], researchers measured driver situation awareness when viewing driving videos with and without an HUD. It was found that not only increasing complexity in a HUD impaired driver situation awareness, but also that driver situation awareness was in general better when there was no HUD. It is perhaps not surprising that when much information was augmented (i.e., complex condition), drivers became cognitively overloaded, and their attention was consumed by the display. However, even under the minimal condition when only road signs and traffic lights that required an action was augmented, driver situation awareness was still poorer than the no HUD condition. Considering a stop sign or a red light which require an action would be safety-critical information, the lack of benefit of HUD as found in this study further alarms the need to investigate how to present trip-related, but non-safety-critical, information to drivers using advanced in-vehicle displays. Research should investigate how to design the HUD, and very importantly, also consider how drivers will become used to visually process such advanced displays. Thus far, there has been little work done to test the effectiveness of advanced information displays for the presentation of trip-related road sign information during automated driving.

Multi-modal warnings are another type of display that are becoming increasingly prevalent in automated vehicles. They provide a viable solution for delivering information to the driver related to a takeover request, where the driver must re-establish situational awareness on the vehicle and roadway environment [76]. There is a growing body of work investigating how best to combine modalities of information presentation for more effective driver alerts and understanding how perceived urgency can affect driver response times during takeover [56,77,78]. General findings indicate that response time for drivers was better in conditions where alerts were pictorial versus text-based and when the perceived urgency of the alert was greater. However, these results were observed when a threat was present in the roadway, resulting in a takeover request from a vehicle to a driver. There has been little work on how multimodal warnings affect driver processing of non-safety-critical information, such as signage for food and lodging.

4.3. In-Vehicle Display of Roadway Conditions

With the development of highly advanced automobile technologies, such as connected vehicles, IVISs have also been proposed as a potential alternative, or supplement, to road signage [23,79,80,81]. Compared to conventional signs, IVIS displays have the advantages of being less susceptible to poor weather conditions and presenting messages that are tailored to traffic conditions (current and anticipatory) as well as driver information needs. For example, lodging information can be displayed more frequently to a driver during evening vs. daytime hours. However, only a small number of studies have been conducted to guide the presentation of in-vehicle displays of roadway conditions. In one study, Lee and his colleagues found that in-vehicle messages, such as warnings about “icy roadway” and “accident in lane”, were much more effective when presented as redundant information, in addition to road signs, than when presented alone [56]. Caird and colleagues examined the effectiveness of in-vehicle displays of traffic light notifications and found that notifications presented 8 to 12 s before arriving at an intersection reduced the frequency of drivers running yellow lights [79]. In a study by Creaser & Manser, drivers were provided with in-vehicle speed limit information [80]. Although the in-vehicle presentation did not lead to significant improvements in driver speed control, drivers rated the in-vehicle information as favorable and helpful when following an unfamiliar route. In-vehicle presentation of information about surrounding traffic has also been found to improve driver anticipatory behavior in both manual driving [82] and driving with automation [83]. As compared to receiving takeover requests and automation capability information, drivers would be more likely to take over an automated vehicle when also provided with surrounding traffic information [83]. This finding highlights the importance in considering communicating roadway information to cognitively engage drivers in automated driving. The questions are when and how to present such information. For example, how frequently should roadway information be presented, and what roadway information (safety-critical vs. non-safety-critical) should be presented.

5. Factors That Influence the Effectiveness of In-Vehicle Messaging

Based on the above review of advanced vehicle technologies, there are several cognitive and engineering factors that need to be taken into consideration when designing in-vehicle messaging systems. Multiple factors may significantly affect the way messaging content is presented to a driver. In this section, we address driver characteristics (including cognition), environmental factors, engineering of the automated vehicle, and engineering of the in-vehicle display, which might play a role in the effectiveness of message delivery. Figure 1 provides a conceptual illustration and summary of these specific factors.

The characteristics of the driver are major factors that can affect the efficacy of in-vehicle messaging during automated driving. State and trait-based characteristics can have different effects on driver capability and abilities to perceive information on the road; thus, driver states and traits are considered separately in this review. The distinction here is necessary because states can be influenced by tendencies that result from trait level characteristics [84].

5.1. Driver Characteristics

Trait-based characteristics of a driver, such as perceptual, attentional and cognitive abilities, as well as personality profiles, make a driver prone to certain tendencies or states during driving [84]. For example, limited attentional resources may result in an inability for the driver to store relevant information in working memory, such as the relative speed of other cars on a highway. This working memory limitation can result in a safety hazard if the driver attempts to change lanes. Similarly, the driving style may also play a role in determining whether an individual driver benefits from a specific in-vehicle display design [70]. The following subsections review additional trait characteristics that may affect driver ability to perceive and interpret in-vehicle messaging information.

5.1.1. Cognitive Capabilities and Messaging Display Design

Drivers’ cognitive capabilities in attention and working memory have been shown to be significant predictors of driving performance and safety [85,86]. In specific, spatial attention has been identified as being a strong predictor vehicle and traffic awareness. Spatial attention refers to the allocation of attention to locations in the visual field [87]. Spatial attention is affected by peripheral and central cues, which involve reflexive orienting and volitional orienting, respectively [88,89]. During driving, an individual must direct their attention to several different locations in the vehicle to develop a complete “picture” of vehicle status. This process is related to automated driving because the driver may be required to reallocate attention from a secondary task to a display in the car during a takeover request. The design of in-vehicle displays should support this shift of attention. For example, auditory cues have shown promise for capturing and diverting attention to visual information that is spatially coupled with the auditory cue [90]. Therefore, any messages presented at an in-vehicle visual display will benefit from a pairing of auditory information to best alert the driver to the information.

Another related consideration is individual differences in spatial attention [91,92,93,94]. To effectively design messaging content, individual attentional capability must be considered, as messaging may fail to enter a driver’s state of awareness or unnecessarily capture too much attention. These factors are especially important for older drivers who may need accommodations due to reduced attention capability with age-related changes. Design practices, such as decreasing visual clutter, providing multi-modal messaging, as well as using advanced cues (such as an early notification of upcoming signage) to reduce simultaneous competition of attention could all be considered.

Driver working memory is also a critical cognitive factor, as driving is a complex task that demands storing and manipulation of information in real-time [95]. Research has shown that high working memory capacity novice drivers were less affected by increasing cognitive load in a driving task. They performed better in lane changes, identified developing hazards in video clips more quickly by using more efficient eye movements in a dual-task condition, and were less likely to experience instances of inattention during driving [96].

Taken together, the above results suggest that lower-cognitive capacity individuals perform worse on hazard perception performance partially because they fixate less on the hazard, which greatly reduces their ability to identify, interpret, and respond quickly to the hazard [95]. These observations lead to the inference that in-vehicle messaging should consider a driver’s working memory as it varies among individuals and throughout the life span. While automated driving does not require a driver to be actively engaged in the task of driving, the need for the driver to be ready to takeover vehicle control can result in drastic changes in cognitive loads. Working memory available at the moment of a takeover request will largely determine the capability of a driver to safely operate an automated vehicle. If designed inappropriately, an in-vehicle message may be presented with poor timing and contribute to driver cognitive load; for example, presenting novel service logo information when a takeover is required, as a vehicle exits a highway. Instead, such messaging could be provided in advance when driver cognitive load is low during highway driving. Such a design could potentially mitigate changes in driver cognitive load.

5.1.2. Technology Experience and Acceptance

The degree to which drivers engage vehicle automation and other modern technologies may be determined in large part by their experience with and acceptance of the technology. For example, surveys about driver acceptance towards automated vehicle technologies have shown that drivers tend to vary in comfort in using adaptive cruise control and active lane-keeping across driving situations [97]. In specific, drivers were more comfortable with automation during free-flowing traffic than in stop-and-go traffic situations [97]. The degree to which drivers utilize these technologies is also affected by their level of prior experience. Prior research has shown that drivers who are accustomed to adaptive cruise control tend to be faster at responding to event notifications during automated driving [98].

Older adults are a demographic of particular interest, when considering technology acceptance, as they stand to benefit greatly from vehicle automation. Some research has shown that adaptive cruise control and lane-keeping systems can improve overall vehicle safety [99,100]. However, older drivers tend to exhibit less trust in vehicle automation, likely due to a general lack of experience and technical support [101]. Compared to younger and middle-aged drivers, older drivers also have greater adaptation to roadside signage. This experience could have an impact on how older drivers adapt to any transition from roadside displays to in-vehicle displays. Furthermore, research has found that older drivers engage in more conversational secondary tasks during automated driving while younger drivers tend to be immersed in electronic device use [46]. Considering in-vehicle delivery of signage information during automated driving, how secondary activities impact the use of in-vehicle information communication remains an open and important research question.

For in-vehicle messaging content to be effective, this means considering individual differences including age differences with appropriate notification times to allow drivers to respond and intuitive interface design to promote adoption. The implementation of in-vehicle messaging of signage and interface design should consider existing guidelines and the standards for designing in-vehicle communication systems [7,62,66].

5.1.3. Driver States

Driver states are characteristics of a driver at a given moment, and therefore are dynamic and situational [102]. Driver states are influenced by a driver’s current and prior tasks, as well as environmental conditions. Driver states that may be considered in in-vehicle signage information delivery design include, but are not limited to, driver boredom, sleepiness, and fatigue, as well as driver vigilance. Despite being separate, these state measures are often related and impact each other.

A concern with Level 3 automated driving (according to the SAE taxonomy) is that a driver may become bored during highly automated driving with activity being limited to passively monitoring the state of the vehicle [103,104]. This is of concern because states of boredom can result in performance decrements, such as slower response times and higher variability in responses [105,106,107]. Previous work has demonstrated that drivers tend to cope with boredom by using approach or avoidance strategies [108]. In an approach strategy, a driver may refocus their attention to increase the amount of stimulation they receive from the driving task; thus, they are “approaching” the task. Heslop showed that drivers tend to use avoidance strategies when engaged in boring driving tasks resulting in driver distraction [105]. During automated driving, drivers may tend towards this avoidance strategy and seek stimulation from sources other than monitoring the state of the vehicle. There is research that shows drivers who are required to monitor automated systems demonstrate more signs of drowsiness than drivers who are actively engaged in secondary tasks such, as reading a book or watching a video [14]. This presents an opportunity for in-vehicle messaging of non-safety-related information, such as specific service information to potentially reduce the likelihood of driver boredom. On the other hand, during a critical event, such as a takeover, drivers could experience a sudden spike in workload and the presentation of in-vehicle messages may pose undesirable distractions, as the driver tries to comprehend the current situation. Distraction can degrade driver situation awareness under normal and abnormal hazard conditions [109]. However, a close examination is needed on whether the detrimental effects of boredom may be offset by secondary task performance and if any benefit outweighs the cost of task switching between a secondary task (e.g., sign identification) and the driving task (e.g., takeover).

5.2. Characteristics of the Environment

The external environment also plays a role in influencing driver’s processing of information from an in-vehicle display and how attending to an-in-vehicle display may impact their performance in operating a highly automated vehicle. For example, environmental characteristics, such as lane width, quality of roadway markings, and the presence of trees or buildings near the road can influence driver performance in speed maintenance and maneuvering [110,111,112,113]. Thus, depending on the location where a takeover is taking place, a driver’s performance at the maneuvering/control interaction level can be negatively impacted by environmental features [110]. Specific structural features of the roadway such as lane width, road markings, and the presence of trees or buildings near the road all affect driver vehicle control, including speed [113]. Time of day has also been shown to be a significant factor in the rate and severity of automobile crashes with nighttime driving being more dangerous than daytime driving due to several factors such as fatigue [114,115]. In nighttime driving, there is lower luminance which can affect visual processing of stimuli on the roadway and increase processing times [116]. Depending on the time of the day, the relevance of certain information also changes (e.g., lodging information may be more relevant in the evening). However, it remains unclear how these variations in road environment conditions can affect driver interaction with automation; specifically, the transfer of control between the automation and the human driver as well as interpretation of in-vehicle messaging.

From an engineering standpoint, these environmental factors need to be considered when providing messaging that might result in a takeover scenario for the driver. In-vehicle messaging displays need to smartly respond to roadway configuration, the time of day and luminance levels to promote ease of information processing for the driver. Roadway structure may not be as easily addressed in terms of messaging content; for example, drivers should be informed of inadequate lane width markings for automated vehicle control. Such messaging needs to be timely and salient to ensure vehicle and driver safety.

5.3. Characteristics of Vehicle Automation

Vehicles with Level 3 or 4 automation can monitor and maintain control of driving tasks with the expectation that a human will take over control. Levels 1 or 2 vehicle automation still require the human to maintain control of the vehicle for the majority of time [26]. These different levels (or modes) of automation create a set of factors that could potentially impact the development of in-vehicle messaging content. Specifically, automation reliability is crucial to ensuring safe and effective use of new technologies. Automation reliability is necessary to minimize demands on driver vigilance over extended periods of time when there are limited opportunities for physical engagement in vehicle control. Reliability is also necessary to offset driver performance decrements when boredom and vigilance decrements occur [117]. However, reliable automation comes with its own set of pitfalls. As the level of highly reliable automation increases, there is an increased chance of performance impairment when the automation fails. This is particularly true for Levels 2 and 3 vehicle automation, as the driver is disengaged from a significant part of the driving task but is still expected to takeover vehicle control when needed. As automation handles vehicle control and simple road hazards, drivers become disengaged [46], and their skills to handle takeovers in the event of automation failures degrade, especially for more complex situations [118].

One possible function for in-vehicle messaging is to serve as a mitigation of automation-induced vigilance problems by providing a secondary task that might prevent the onset of boredom during an automated drive; however, there is no empirical evidence of this relationship. Proper tuning of the timing, type and amount of information is necessary to ensure messaging is helpful without resulting in increased workload on the driver in information processing. Another concern regarding characteristics of messaging displays is the amount of feedback that is provided by the automated system. As a result of the driver being disengaged from the driving task, recovery of control can differ in quality depending on the design of warnings [46,119,120]. Informative feedback could potentially alleviate the negative consequences of higher levels of automation by keeping the driver regularly apprised of information relevant to the driving task. However, drivers may experience some frustration with frequent messaging notifications from the vehicle for no acute reason. A potential method is to establish specific messaging intervals depending on the criticality of the message. However, this design approach requires further investigation for effectiveness.

5.4. Characteristics of In-Vehicle Displays

Substantial human factors research has investigated the interaction between drivers and vehicle information displays. Increasing vehicle automation leads to new challenges in display design. Several display characteristics, influential in human use, have been identified including the type (e.g., head-up vs. head-down), modality of information presentation, information format, and information load. As noted, Head-Up Displays (HUDs) and Head-Down Displays (HDDs) each come with some advantages but also pose unique challenges for presenting road sign information during automated driving. HUDs may provide the benefit of reduced re-fixation time; however, they can absorb driver attention with the amount of visual information, particularly during a takeover request. In contrast, HDDs may reduce visual load but could lead to longer response times, as visual attention must first be directed to the display and then re-directed to the roadway, if needed. Therefore, HUDs could be more suitable for supporting driver perception of road environment cues, while HDDs may be more beneficial for reduced information load when a longer response time is available (e.g., a notification in advance of a takeover).

It is also important to ensure consistency in interface design for vehicle systems that present the same type of information. Such displays should always occur at the same location, either on a HUD or HDD. HDDs require greater use of visual cues to direct driver attention to displays, as a result of the display not being co-located with the roadway in the same way as a HUD. As of the time of this review, there has been no comparison of HUDs and HDDs for presenting roadway signage information and the utility of either type of display remains an open research question.

Regarding modalities of display information attention, the combination of visual, auditory and even haptic cues could be effective in guiding spatial attention of a driver to an in-vehicle display. For non-safety related messaging, the use of these cues can improve driver’s processing of upcoming exits that may be of interest, such as food or lodging locations. Modalities of perceptual cueing and their combinations should be examined for information presentation using HUDs and HDDs, as specific pairings may vary in effectiveness during automated driving.

Regarding the amount of information presented through an in-vehicle display, the format of content needs to be optimized to support driver information processing and to allow enough time for decision-making on takeover requests. Research has shown that pictorial information paired with text tends to be more effective than just text-based information and should be applied for display design [20]. This approach can also help to reduce display clutter and increase readability. Effective organization of visual information on the in-vehicle display can also improve driver preference and driving performance while using the in-vehicle displays. In a recent study, Chen and colleagues [121] found that as compared to a visual interface with six panels of information, drivers reported higher preference toward a design with three panels, and also showed better vehicle control when using a design with four panels. This finding suggests the importance of exploring the visual organization of in-vehicle displays. In addition to optimizing visual communication, auditory cues can also aid in reducing driver visual attention load [122]. Therefore, auditory menu cues along with visual presentation of other displays should be utilized to reduce driver workload. Research has also demonstrated that use of auditory cues for menu navigation is beneficial in orienting driver spatial attention [123]. The required processing time for each modality of information presentation, or combination of modalities, should determine the use of modalities for specific situations, such as planned takeovers, emergency takeovers, and ambient communication of trip-related information.

For in-vehicle display design, determining an appropriate information load may be critical to maintaining driver alertness and display engagement. Considering the general concern for driver disengagement during automated driving, this may be a key challenge to support driver performance and vehicle safety. The display information load may also need to be adjusted to the dynamics of vehicle automation during driving. For example, when a driver is required to takeover vehicle control, they are more likely to be in a potential overload vs. underload condition; thus, safety-critical information should be emphasized. Dynamically minimizing the amount of attention that a driver needs to allocate to processing display information is expected to promote driver capability to respond to takeover requests effectively and safely. The goal of any in-vehicle display design should be to maintain a proper level of information load while considering the state of automation, the driver, and their interaction.

6. Proposed Design Guidelines

One objective of this review is to provide a preliminary set of guidelines to design effective non-safety related in-vehicle messaging content. Previous attempts have been made to provide human factors guidelines for roadway signage and interface design for elderly drivers [124]. NHTSA proposed a series of guidelines for in-vehicle displays, but these guidelines focus primarily on manual driving, whereas the conditions of automated driving could involve differing concerns [7]. As research on in-vehicle message delivery of road signage information during automated driving is still in its infancy, the preliminary guidelines proposed here are intended to provide a starting-point for how in-vehicle messaging should be designed to leverage aspects of vehicle automation. The guidelines are meant to motivate designers to address specific human performance issues associated with automated driving vs. identifying exact solutions for specific driver disengagement situations.

Table 1. Proposed design guidelines and outstanding research questions for the presentation of non-safety related information via in-vehicle display.

Design Aspect	Guidelines Based on Existing Literature	Related Research Questions
Presentation of Road Signage	Road signage should use structures familiar to drivers from other physical signs [18]. Signage should be physically representative of what it is trying to convey (i.e., lane closure signs should represent a lane being closed) [18]. Pictorial information with text is more effective for conveying sign information [20].	Should the presentation of road sign information follow the standard road conventions, when presented during highly automated driving? If both on-road signage and in-vehicle displays are available, how would drivers cope with potential incongruence between the two information sources?
Driver Attention	Multimodal cues (auditory and visual) should be leveraged to guide driver attention effectively [75,77,78]. Messaging cue design should accommodate older or disabled drivers [94,125]. Keeping drivers engaged during highly automated driving can benefit safety [126,127].	Are auditory, visual, or multimodal cues most effective for presenting non-safety related information to a driver? How do these cues need to be altered to accommodate older drivers when presenting non-safety-related information? Can we leverage in-vehicle presentation of non-safety-related information to periodically engage a driver with the road environment?
Communicating Automation State	It should be assumed that drivers are disengaged from the driving task during highly automated driving [42]. Indicate status of vehicle control to driver using effective cues that allow appropriate response times (i.e., whether the vehicle is in control of specific functions or not) [86,122].	How should information regarding automation state be presented to a driver? What information does a driver need regarding the state of vehicle automation?
Displaying Information	HUDs should be leveraged to reorient drivers to roadway information [128]. Displays should optimize information to be informative with minimal display clutter [54].	Should the presentation of safety-critical vs. non-safety critical information be different? Given the potential to increase visual clutter, should non-safety-related information be presented via perceptually augmented displays?
Driver Interaction	Display content should minimize driver off-road glance time such that messages are easily interpreted without requiring the driver to look away from the road for extended periods of time [7]. Visual messages and menus should also allow for auditory navigation to facilitate interaction [122].	Does non-safety-related information significantly impact driver off-road glance time?

presents the proposed guidelines and their description as well as the supporting literature.

7. Future Research Directions

As previously mentioned, the limited body of research in this area has focused on in-vehicle display of safety-critical messages. For example, Politis and colleagues investigated the use of multimodal displays in conveying safety critical handover of control from an automated vehicle to driver [78]. In this study, the authors tested multimodal abstract (pictorial) or language-based warnings and compared effects on driver capability to resume control of the vehicle. Delivery of non-safety-related but trip-related information (e.g., available local services) remains unexplored. Since driver attention allocation may vary according to the relevance of a message to concurrent tasks, the findings on in-vehicle messages of safety-related information may not generalize to messages of non-safety-critical information.

Due to the nature of previously published guidelines, and supporting research focus on manual driving, it is necessary to conduct additional research on the proposed in-vehicle display guidelines. In specific, it will be important to investigate driver performance when signage is presented either on the roadway, an in-vehicle display, or both. In addition, for in-vehicle presentation, the spatial location of displays should also be examined. Driver visual behavior, vehicle control and hazard response should all be considered in such investigations. At the same time, despite the non-safety related nature of specific service (logo) signs, presentation of this content may have a significant effect on driver performance during a hazard scenario. Driver performance should be investigated in terms of manual and automated vehicle control to compare how automated driving may alter hazard negotiation. It is possible that signage information, which has been presented using specific formats, may need to be restructured to aid effective driver interpretation of information during automated driving via an in-vehicle display.

Additional work should focus on the effects of individual characteristics (demographic, trait-based, or situational) and driver engagement on vehicle control performance during presentation of messaging content. For example, aging results in several physical and cognitive changes that can affect driving, as noted in the previous sections. Therefore, age groups should be compared to investigate how perceptual and cognitive declines impact driver responses to non-safety-related messaging. Also noted in previous sections, automated driving can result in driver disengagement due to passive monitoring of the roadway. As such, it is necessary to investigate how drivers behave during automated vehicle control and how their capability to perceive and attend to in-vehicle messages is affected by disengagement from the driving task.

Furthermore, future work should explore whether the current standards of road signage can be translated to in-vehicle display design or whether new formats of exit information could improve driver performance. The structure of the current road signage should be evaluated for effective in-vehicle display, particularly when the driver is not in control of the vehicle and not attending to information external to the vehicle. The timing of takeover notifications, relative to non-safety-related messaging, should also be explored to determine the minimum presentation time of each message for drivers to perceive and respond effectively without compromising driving safety.

8. Conclusions

The present review sought to provide a framework of the factors contributing to the effectiveness/utility of in-vehicle messaging content during automated driving. We also proposed preliminary guidelines for designing such content for future investigations. It is apparent from the body of literature that not only does automated vehicle technology need to be leveraged to effectively deliver in-vehicle messaging, but the characteristics of human attention and cognition need to be considered along with messaging design options as well as environmental/situational factors. The guidelines provided in this review are meant to aid designers in identifying and target specific automated driving issues that related to the design of in-vehicle information displays. The guidelines are also expected to provide some basis for future investigations on the development of messaging content that is safe and informative for drivers during automated driving.