Beep and Show, Don’t Tell: Multimodal Feedback Strategies for Driver-Automation Coordination in Critical Driving Situations

Abstract

In level-3 automated driving, control shifts between the system and the human driver, making timely and effective communication crucial in conflict situations. To explore how different communication modalities affect the driver’s understanding, trust, and performance in such situations, we conducted two complementary studies: a focus group and an interactive user study. The focus group revealed a preference for multimodal, tailored feedback, with visual information most frequently favored. The interactive user study tested these findings in practice by asking participants to confirm the system suggestion or take over. The results from the interactive study showed the importance of a pre-explanation signal that draws attention to the situation, and a clear visual marking of the proposed resolution. While most participants stated a preference for speech-augmented feedback, spoken and written explanations were frequently overlooked or reported as distracting during active conflict scenarios, suggesting a gap between stated preference and in-task utility. These results indicate that in urgent conflict situations, concise visual cues are more effective than detailed verbal explanations, offering guidance for the design of future level-3 vehicle interfaces.

1. Introduction

The increasing deployment of automated driving systems introduces novel challenges for human-vehicle interaction as part of Human–robot interaction (HRI), particularly in the context of unexpected conflicts in traffic. At level-3 [1], the Autonomous Vehicle (AV) can drive itself under certain conditions, performing all dynamic driving tasks like acceleration, braking, and steering. The key aspect is that the system monitors the environment and the driver must remain available and be able to take over when requested via a Takeover request (TOR) [2]. AVs have been demonstrated to possess the capacity to detect and analyze complex conflict situations; however, the manner in which the system’s solution is communicated to the human operator remains an area of exploration.

Conflict situations, i.e., situations in which automated vehicle behavior is challenged by environmental uncertainty, system limitations, or interaction with other road users, have been examined from multiple perspectives. These include interaction-focused analyses [3] and methodological discussions of scenario evaluation [4,5]. Together, this work illustrates the challenges associated with communicating about conflicts in automated driving. One prominent direction in this space is the use of multimodal communication, combining visual, auditory, and textual feedback to convey system intent, urgency, and context. Visual cues are commonly used to support spatial understanding, while speech and other explanatory forms are explored as a means of increasing transparency and trust. Studies in instructional driving contexts show that spoken explanations of the situation can complement visual guidance and support learning and confidence when time pressure is low [6]. These findings provide useful context, although interactive conflict situations impose additional temporal and attentional demands.

The present study builds upon recent work [7,8] on how multimodal strategies—such as visual, auditory, and combined feedback—can support drivers in rapidly understanding the situation and taking appropriate action. Its objective is to examine how different modalities of conflict communication influence driver perception, situational awareness, and cooperative decision-making in controlled scenarios. This paper provides empirical evidence on how multimodal communication strategies affect human understanding and trust in shared autonomy. Our study combines a focus group with an interactive simulator study to examine participants’ perceptions of concise visual cues and pre-warning signals in urgent conflict scenarios, compared with text- and speech-based explanations that were sometimes perceived as distracting. Our findings extend the existing body of research on takeovers by highlighting the circumstances in which multimodality facilitates, and those in which it impedes, high-pressure human–automation coordination.

2. Related Work

2.1. Conflict Scenarios in Autonomous Driving

Conflict handling in automated driving has become a central research topic. Investigating how conflicts arise and how they can be communicated effectively is crucial for safe human–automation collaboration. Tekkesinoglu and Kunze emphasize that conflict communication must be understood as an integrated feedback loop rather than an isolated warning system [3]. Related methodological contributions propose systematic evaluation frameworks to assess safety-critical scenarios and human–automation coordination, such as the automated assessment approach by Morales-Alvarez et al. [4] and scenario-based exploration frameworks discussed by Chaczko et al. [5].

Recent work also points to broader research agendas. Literature reviews highlight the variety of conflicts and interaction challenges in highly automated driving [9,10,11]. These surveys emphasize the complexity of driver behavior, the variability of situational factors, and the methodological challenges in evaluation. Complementary to this, methodological advances in adaptive conflict detection and resolution strategies, such as predictive driver modeling [12,13], gender-sensitive approaches [14], and contextual adaptation [15], underline the diversity of current perspectives.

2.2. Feedback Modalities and Human Factors

While this study focuses on level-3 automated driving, the attentional mechanisms underlying driver response to multimodal warnings are well-established in manual and remote driving, sharing common foundations with the challenges addressed here. Research using fine-grained tracking tasks has shown that verbal task modality degrades driving performance following psycholinguistic timing, driven by cognitive rather than purely perceptual load, establishing that resource competition between communication channels and vehicle control is a fundamental property of human attention [16]. Studies on prolonged visual distraction from in-vehicle displays further demonstrate a non-linear impairment of lateral and longitudinal control, where a threshold is reached beyond which the negative impacts plateau rather than infinitely intensify, underscoring the critical impact of distraction timing [17]. A meta-analysis across 308 individual studies confirmed that while both auditory and tactile warnings significantly reduce response times compared to no-warning baselines, tactile alerts specifically outperform visual-only alerts in direct comparisons; however, trimodal configurations fail to yield statistically significant response-time improvements over bimodal warnings [18]. This systemic limitation in user response dynamics is mirrored in pedestrian alert studies, where system limits, specifically false alarms rather than modality count alone, introduce severe alert fatigue, anxiety, and excessive driver caution [19]. These modality-conflict dynamics demonstrate that warning content, timing, and channel selection jointly determine safety outcomes in both remote operation and in-vehicle human–robot co-driving contexts [20,21].

These insights from manual and remote driving contexts converge with a growing body of work specifically examining feedback modalities in automated driving, where timing, modality, and workload similarly shape driver awareness, trust, and performance. Kim et al. [22] show that timing and modality of feedback significantly influence takeover performance. Multimodal feedback is widely discussed as a promising strategy, combining visual, auditory, and even haptic cues to balance workload and redundancy [23,24,25,26]. Bellem et al. [27] highlight that comfort in automated driving is closely tied to how feedback is delivered, while Körber [28] emphasizes trust calibration as a determinant of system acceptance.

Non-driving related tasks (NDRTs) represent a particular challenge for driver re-engagement. Yoon et al. [29], Hu et al. [30], and Hungund et al. [31] show how distraction and task complexity influence takeover times. Studies on driver workload [32,33], calibration of trust [34], and the use of physiological or contextual indicators [35,36] further reinforce the need for adaptive and personalized feedback strategies.

2.3. Augmented and Multimodal Interfaces

Research has also investigated novel interface modalities to support conflict communication. Augmented Reality (AR) has emerged as a promising channel for delivering intuitive information about vehicle intent and environmental hazards. Kunze et al. [37], Langlois et al. [38], and Kettle et al. [39] show that AR can effectively increase situational awareness and trust. Recent studies extend this by focusing on adaptive AR feedback under dynamic conditions [40,41].

Beyond visual augmentation, conversational and speech-based interaction mechanisms are increasingly considered. Early work on speech interfaces in vehicles demonstrated their potential to reduce visual distraction [42,43]. A related line of research looks at the problem of generating explanations of driving situations with natural language [44], with growing attention to methods for automatically producing such explanations, especially since the rise in large vision–language models [45,46,47]. More recent studies explore conversational agents and natural language interaction as a means of explaining automated vehicle behavior and intentions [48,49,50,51,52]. These approaches underline the shift from purely reactive warning systems to more transparent, dialog-oriented designs. However, prior work does not provide consistent conclusions on the usefulness of spoken explanations in automated driving contexts. Spoken explanations have been found to be beneficial when combined with visual guidance in low-urgency, instructional settings, but they can also increase cognitive load when poorly timed or overly detailed [6].

2.4. Methodological and Exploratory Approaches

Several contributions stress the importance of methodological rigor and systematic evaluation in the design and assessment of human-vehicle interaction in automated driving. For example, Naujoks et al. [53] discuss human factors methodologies for evaluating TORs, while Salis et al. [12] and Qu et al. [54] propose predictive modeling approaches to understand driver intent and capacity.

At the same time, exploratory reports point to emerging directions. Studies investigating new multimodal designs [55], the use of function-specific feedback [56], or innovative use of context-sensitive information [57], demonstrate the experimental breadth of current work. These efforts reflect the broader trend of integrating adaptive, multimodal, and context-aware strategies into future vehicle–human interfaces.

Across these diverse contributions, three common threads emerge: (1) conflict detection and communication in automated driving is technically feasible but requires human-centered integration [3,4,5]; (2) multimodal and adaptive feedback mechanisms outperform unimodal solutions in maintaining situational awareness and trust [22,23,26]; and (3) systematic frameworks and augmented interfaces represent the next step toward holistic evaluation and deployment [37,38,41]. These insights define the current state of the art and motivate our work, which aims to further integrate adaptive multimodal feedback strategies in conflict communication for automated driving.

2.5. Automated Driving Simulators

Simulation is an important methodological tool in autonomous driving research, as it enables systematic investigation of driving behavior and system performance under controlled conditions. Existing simulation platforms address different components of the driving stack and operate at varying levels of abstraction. Traffic- and policy-oriented simulators, such as SUMO [58], ScenarioNet [59], and Waymax [60], are commonly used for large-scale interaction modeling and policy evaluation, but typically do not support a human driver in the loop.

Among simulators that support direct human control, AirSim [61] and CARLA [62] are widely used open-source platforms. While AirSim is commonly used for perception and control studies, CARLA is designed to support urban driving scenarios with multi-agent traffic and closed-loop interaction. Building on these capabilities, game-engine-based frameworks [7] and CARLA-based extensions [8,63] support more specific use cases, including conflict simulation, shared autonomy, and human-centered interaction studies. For the needs of our study, we selected CARLA as an established framework, and adopted a toolkit built on top of it, which allows simulation of conflict scenarios and TORs.

Our work adds to the existing literature on level-3 handover through four findings. First, we show that a two-stage alert combining a beep and a mirror icon is effective at drawing driver attention before the main information is presented. Second, showing the planned trajectory was consistently well-received and helped drivers make decisions under time pressure. Third, there was a gap between what participants said they preferred and what actually helped them: most said they preferred speech explanations, but during the scenarios these were often missed or ignored. Fourth, whether speech is useful depends on the situation, as it helped most when the visual scene gave little information on its own. Overall, our findings suggest that handover interfaces work best when visual trajectory cues are the primary channel and spoken explanations are reserved for situations where the visual scene is unclear. Given the exploratory scope and limited sample sizes of both studies, these findings are best read as hypothesis-generating rather than confirmatory, consistent with common practice in early-stage HRI/Human–machine interface (HMI) research.

3. Simulation of Conflict Situations

3.1. Simulation Framework

For our study, we adopted as a foundation a CARLA-based toolkit [8] for simulation of conflict scenarios and TORs (cf. Figure 1). We extended it to support behavior-agent-based autonomous driving, conflict-specific trigger points, and enhanced takeover logic. We added to the conflict definitions a trigger point (a pre-defined location on the map). When this point was reached by the vehicle, we showed the notification for takeover or confirmation of the suggestion. For the experimental trials, predefined trajectories were implemented on CARLA’s Town04 and Town05 maps to ensure consistent replication of conflict scenarios across participants (cf. Figure 2). We also added textual descriptions to the conflict scenarios, which contain a description and reasoning of the situation in natural language, which can read to the user by a text-to-speech model when a TOR is initiated.

Figure 1. Third-person view of CARLA simulation environment after a test run with remnants of visual cues in the background (green-colored waypoint fragment). The highway environment is part of Town04 and used for the vanishing lane markings conflict.

Figure 2. Top-down view of predefined conflict trajectories (- - -) implemented on CARLA’s Town04 (left) and Town05 (right) maps, showing the sequence from start waypoints to conflict zones represented by the X (images were generated using the CARLA simulator environment).

To create an immersive driving experience, we placed the main camera in the interior of the car (Mercedes Coupe 2020) and disabled external views. Conflicts were triggered either at predefined points on the map or, in one of the scenarios, when the confidence of the used lane detection model [64] dropped below defined thresholds (<0.6 for warning, <0.3 for mandatory takeover). These thresholds were established empirically through iterative pilot testing prior to the study: we varied the confidence cutoffs during development and observed the resulting trigger timing and driving behavior in the simulator, selecting the values that produced consistent, well-timed conflict onset without prematurely interrupting normal driving or triggering warnings too late for a meaningful driver response. No formal calibration of the underlying lane-detection confidence scores was performed; the thresholds instead reflect a pragmatic choice tuned for reliable and reproducible triggering of the two-stage warning/takeover escalation used across all participants.

In these cases the system issued a warning, offered the driver a choice to continue with the suggested maneuver, or required a mandatory takeover via TOR. The HMI elements described in Section 3.3 communicated the conflict and suggested maneuver, while the takeover was done via key press that was explained beforehand and also displayed on the screen. This setup enabled us to systematically vary urgency and the level of driver involvement across different conflict situations, including road narrowing, loss of lane information, and negotiation with other vehicles.

For the implementation of the interactive user study, our simulation framework models conflict situations as a sequence of phases (cf. Figure 3): an initial autonomous driving phase of about two minutes, during which participants performed a NDRT to deliberately draw attention away from the street (see Section Non-Driving Related Task (NDRT)). The system then detects a potential conflict and issues pre-takeover cues, followed by an explicit TOR. At this point, the driver must decide whether to assume manual control or confirm the system’s suggested maneuver, after which the scenario continues in manual or supervised automated driving. Shortly after overcoming the conflict, the simulation was terminated at a nearby final waypoint.

Figure 3. Timeline of a simulated conflict situation: the driver starts in autonomous mode with a NDRT, receives early pre-TOR cues, transitions into takeover, and either drives manually or confirms the system’s suggestion. The color transition from gray to red indicates decreasing time to initiate an action.

3.2. Conflict Types

The conflicts examined in our study (cf. Figure 4) can be located within a broader taxonomy of challenges in autonomous driving. Generally, conflicts arise when the system reaches the boundaries of its domain (e.g., the system cannot resolve a given situation without violating traffic rules) or when its technical capabilities are affected by failure, such as sensor malfunction that limits the system’s perception or decision-making capabilities.

Figure 4. Mockup images of examined conflict types narrowing road (left), vanishing lane markings (middle), and intentional conflict (right). Colored arrows indicate the planned trajectory of the vehicles, while the black arrow represents necessary communication to solve the conflict.

Although many empirical studies have focused on specific instances of these categories—such as issues with lane markings, interactions with pedestrians or cut-in vehicles, or environmental conditions—our work concentrates on three particularly relevant conflict types. These represent common and safety-critical situations where system limits and human intervention intersect. To structure these conflicts, we draw on the framework proposed by Gold et al. [65], which classifies conflicts according to urgency (time budget to react), predictability (dependencies on contextual factors), criticality (safety risk), and driver response (complexity of human intervention). We have already transferred this framework to feasible simulation [8]. For our purposes, urgency and driver response are particularly decisive, as these factors directly affect the design of human–machine interaction during conflict resolution. The selected conflict scenarios (narrowing road, vanishing lane markings, and negotiation) were chosen because they represent situations in which current level-3 AVs may encounter ambiguity or uncertainty that cannot always be resolved through purely rule-based decision making. In each scenario, multiple reasonable actions are possible, requiring contextual interpretation of the environment and consideration of implicit traffic norms. These situations therefore constitute representative examples of circumstances in which future AVs may benefit from driver involvement, making them suitable test cases for investigating communication and coordination mechanisms between the vehicle and its human operator.

3.2.1. Narrowing Road

In this scenario (cf. Figure 4, left), the AV approaches a section of road in Town05 that narrows due to a parking vehicle (cf. Figure 5). While driving in automated mode, the vehicle detects the reduction in available lane width and issues a request for human input. The driver, engaged in a NDRT, is required either to confirm the system’s proposed maneuver or to take over control to safely continue. The situation creates a time-critical conflict, as the available space decreases rapidly and clear communication of the system’s intent is essential for timely driver response. Although the system is technically capable of identifying a feasible solution, it faces uncertainty because doing so would require breaking its internal rule set (e.g., crossing a solid line or temporarily entering the traversal lane).

Figure 5. Narrowing road conflict in CARLA. A parking, non-moving vehicle on the right side represents an obstacle leading to trajectory adjustments to reach the next waypoint (green).

3.2.2. Vanishing Lane Markings

In this conflict scenario (cf. Figure 4, middle), the AV encounters a stretch of road in Town04 where lane markings are gradually disappearing, for example due to poor maintenance, construction work, or adverse weather conditions (cf. Figure 6). As the markings fade, the automated driving system loses reliable lane-keeping information and can no longer guarantee safe operation. This use case would also be observed if the AV’s sensor technology fails. The system therefore issues a request for human input. The conflict emphasizes the challenge of reduced environmental cues for automated driving and the need for timely communication of system limitations.

Figure 6. Vanishing lane markings conflict in CARLA. Bad weather conditions (rainy) lead to limitations and uncertainty. The small display (center-bottom) shows detected lanes and their confidence levels. A TOR (center-top) is initiated when detection confidence falls below threshold.

3.2.3. Negotiation

In this scenario, the AV approaches a situation in Town05 that requires negotiating priority with another road user (no Vehicle-to-everything (V2X)/Vehicle-to-vehicle (V2V)), such as a merging vehicle at an intersection or a vehicle entering from a side road (cf. Figure 4, right). While the automation can propose a strategy, it may lack full certainty about the intentions of the other driver (car in the opposite lane with its lights on, cf. Figure 7). The system therefore requests human input, either to confirm the proposed maneuver or to take over control. This conflict highlights the difficulties automated systems face in ambiguous social interactions on the road and the importance of clear human-vehicle communication in such situations.

Figure 7. Negotiation conflict in CARLA with green colored waypoint. Both the ego vehicle’s lane and the adjacent lane are experiencing partial obstruction. It is imperative that both vehicles possess the same rights to traverse the narrow section and that effective communication be established among them.

3.3. Multimodal Feedback

Building on such frameworks, current research increasingly focuses on extending conflict simulation with adaptive feedback mechanisms and multimodal communication strategies in HMIs, ensuring that simulated scenarios not only test technical capabilities but also capture the human-centered aspects of conflict resolution.

In our study, we concentrated on visual and auditory feedback modalities. While haptic feedback is technically feasible and increasingly employed in modern vehicles through mechanisms such as steering-wheel vibrations, torque feedback, and pedal-based warnings, it plays a comparatively minor role in many everyday driving interactions compared to visual and auditory channels [66]. Furthermore, in level-3 automated driving, drivers are frequently expected to disengage from active vehicle control, which can limit the availability and effectiveness of haptic feedback as a communication modality. Our aim was to focus on the most effective and commonly applied modalities in the automotive domain to allow direct comparison between different visualizations and auditory strategies.

3.3.1. Visual Communication

The implemented visual feedback modalities focused on three main aspects. First, intention visualization was realized through trajectory visualization and projected waypoints (cf. Figure 8). This enabled drivers to anticipate the vehicle’s planned maneuvers. In addition to the currently recommended trajectory, the system displayed the vehicle’s initially planned trajectory. The initial trajectory served as a reference for the driver’s decision-making process by making the underlying conflict explicit. Displaying both trajectories allowed participants to understand why a deviation from the original plan was being considered and to assess the consequences of alternative actions. This is particularly relevant in conflict situations, where effective coordination requires not only knowledge of the AV’s current recommendation but also an understanding of the reasoning that led to it. Such visualizations could be implemented in reality through windshield projections or Head-Up Displays (HUDs).

Figure 8. Scenario narrowing road in CARLA. A parking, non-moving vehicle on the right side represents an obstacle leading to trajectory adjustments. Shown initially planned trajectory (yellow) and suggested trajectory marked in red, which would force the AV to drive into the opposite lane.

State visualization was supported by bounding boxes that highlighted relevant obstacles (cf. Figure 9). Bounding boxes were implemented as wireframe cubes around objects of interest (pre-defined), representing potential sources of conflicts.

Figure 9. Scenario intentional conflict in CARLA. Both the ego vehicle’s lane and the adjacent lane are experiencing partial obstruction (barrier and parking car with lights off). State visualization of potential conflict sources is done via bounding boxes (orange), supplemented by planned trajectory (yellow).

Textual descriptions of the situation were displayed on the top of the screen (cf. Figure 6) for part of the scenarios, to evaluate how the presence of written textual information would affect the driver’s decision. The text size was selected to ensure readability from the driver’s viewing position in the simulator while minimizing occlusion of the driving scene. Prior to the study, the display was visually inspected and iteratively adjusted by the researchers to ensure that all text could be read comfortably under the experimental conditions. No formal readability assessment against established visual-ergonomics standards (e.g., ISO 15008 [67]) was conducted; this is consistent with the study’s focus on communication modality and timing rather than typographic optimization, but we acknowledge it as a limitation and recommend that future iterations of the interface be validated against such standards before deployment.

HUD-based attention guidance was provided via symbolic cues, such as alert symbol and conflict-related icons (cf. Figure 10). The indicators were placed in the area of the rearview mirror because this location is close to the driver’s forward line of sight and therefore allows information to be presented with minimal gaze deviation from the roadway. It was implemented as an overlay because we could not modify the interior of the car in CARLA. Furthermore, the mirror region is commonly used for status and warning indicators in experimental driving simulators, reducing the likelihood that additional visual elements would obscure relevant parts of the traffic scene. Other options, such as augmenting other vehicles with contextual icons or using foveated rendering to direct driver attention, were considered for future work.

Figure 10. Scenario vanishing lane markings with HUD elements, prototypically placed in the rearview mirror. The red triangle symbolizes a general warning, while the icon on the right shows the type of conflict.

3.3.2. Auditory Communication

The study covered two modalities of auditory feedback. Feedback tones provided alerts and warnings, signaling the onset of a conflict or the need for driver action. Additionally, voice prompts were included to communicate conflict descriptions in natural language. Because the experimental scenarios followed predefined trajectories and conflict situations, all voice messages were scripted in advance and manually triggered according to the active scenario. This approach ensured consistent wording and timing across participants while avoiding the need for real-time natural-language generation. The voice prompts, which were 1–2 sentences long, briefly described the conflict situation.

4. Experimental Study

To inform the design and evaluation of conflict communication in automated driving, we conducted two complementary studies: a focus group and an interactive user study, both following the same overall protocol (cf. Figure 11).

Figure 11. The protocol for both the focus group and the user study incorporated a sequence of events, as illustrated here. This sequence commenced with an introduction to the background of the project, followed by the administration of a demographic questionnaire. Subsequent to this, a series of runs were conducted, incorporating a combination of conflict type and HMI feedback. Subsequent to the execution of the test runs, the NASA TLX was employed only in the user study. The two procedures were concluded with a post-study questionnaire.

The focus group (Section 4.1) aimed to explore expectations regarding conflict detection in automated vehicles. Using a semi-structured discussion format supported by pre-recorded driving scenarios, eight participants from mixed backgrounds engaged in open reflection. The sessions revealed user needs and design implications for transparency and trust in conflict communication, highlighting both preferred and controversial approaches.

Building on these insights, the interactive user study (Section 4.2) evaluated the usability and effectiveness of different communication strategies in an interactive CARLA simulation. The study employed a within-subjects design, exposing participants to three conflict scenarios combined with two feedback modalities. Dependent measures included task performance, response time, trust ratings, and perceived clarity.

The two studies are closely interlinked. The focus group informed scenario refinement, helping identify the most compelling and ambiguous situations for further testing. It also guided the selection of communication strategies, prioritizing feedback styles that participants considered either particularly effective or controversial. The subsequent user study is designed to test these hypotheses in practice, providing empirical evidence on how different feedback modalities affect driver understanding, trust, and performance in conflict situations.

The questionnaires, data, and results from both the focus group and the user study are publicly available [68].

4.1. Study 1: Focus Group

The focus group was conducted to explore how users perceive and interpret different feedback modalities in conflict situations during automated driving. The goal was to identify which individual modalities and combinations (e.g., projected waypoints with audio alerts) are perceived as most urgent, clear, trustworthy, and intuitive. These insights informed the selection of communication strategies for the subsequent user study (Section 4.2).

Participants were presented with short video clips of pre-recorded conflict scenarios (see Section 3.2). Each clip demonstrated a specific feedback modality or a combination of modalities, including visual and auditory cues (see Section 3.3). Following each clip, participants discussed their impressions in a moderated session. They were asked to comment on how well they understood the situation, how urgent it felt, how much they trusted the system’s response, and how intuitive the communication appeared.

4.1.1. Study Design

A total of eight participants with a diverse range of backgrounds (N = 8) were included in the focus group. The median age of the participants was 36.5 years (range: 34–73 years). All participants were licensed drivers, with varying levels of driving experience (median driving duration: 4.5 h per week) and prior exposure to Advanced Driver Assistance System (ADAS). The median trust level in AVs was determined to be 2.5 on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Participants were recruited from the local university and community. A brief overview of the focus group questionnaire is shown in Table 1.

Table 1. Overview of the focus-group questionnaire, including question blocks and response formats.

Section	Question(s)	Response Format/ Scale Meaning
Pre-discussion	Age, gender, driving frequency, prior ADAS experience, definition of “conflict”, desired information in unexpected situations	Demographics and open-ended responses
Pre-discussion	Trust in AVs to handle unexpected situations	5-point Likert scale
Scenario-based feedback	Perceived urgency, understanding of vehicle intent, confidence, trust in vehicle, and perceived amount of feedback	5-point Likert scale
Scenario-based feedback	Most helpful feedback channel; amount of feedback; suggested changes	Multiple choice/ open-ended responses
Post-discussion	Overall helpfulness, inappropriate/annoying moments, change in view of automated driving, movie analogy, other thoughts, conflict-related icons	Open-ended responses/ icon selection

A 3 × 3 Latin Square counterbalanced the three conflict scenarios (narrowing road, vanishing lanes, negotiation) with three feedback variants, producing nine possible orders. Eight were used for N = 8 participants, one per participant. One order was not assigned. The omitted sequence was selected for practical reasons and did not systematically favor any particular scenario or feedback modality. As a result, each scenario and feedback variant remained approximately balanced across presentation positions, with deviations of at most one occurrence per cell. Given the exploratory nature of the study and the near-balanced distribution, substantial order effects are considered unlikely. Each viewed three clips (one per scenario) with distinct feedback variants, ensuring no repeats. The assignment balanced first-position exposure and distributed scenarios and variants across positions 1–3; small deviations (±1 per cell) reflect N < 9. After each clip, brief discussion and ratings minimized memory carryover while preserving order control.

The discussions were transcribed and analyzed using thematic coding. Participant responses were clustered along key dimensions (urgency, clarity, trust, intuitiveness) to identify promising modality combinations and highlight areas of concern.

4.1.2. Results

Participants expressed a clear preference for multimodal and tailored feedback. Visual information was most frequently favored, though opinions differed regarding its presentation. While some participants preferred trajectory-based or permanent visualizations, others favored shorter, situational feedback to reduce distraction.

Across modalities, the most helpful forms of feedback were reported to be trajectory visualizations, clear text or short voice messages, audio warnings, and HUDs with strong visual clarity (cf. Figure 12). Suggestions for improvement emphasized enhancing HUD readability, refining the timing of feedback—such as providing pre-TOR signals—and simplifying the information presented to avoid cognitive overload.

Figure 12. Participant preferences for conflict communication modalities and design qualities from the focus group (N = 8).

To complement the thematic analysis, we conducted participant-level quantitative triangulation. Endorsements of design elements (visual, audio, hybrid) and qualities (transparency, minimalism) were coded per participant and summarized as proportions with exact 95% Confidence Intervals (CIs). Within-participant contrasts (e.g., visual vs. audio) used McNemar’s exact test with odds ratios; per-clip ratings (clarity, urgency, trust, intuitiveness) were analyzed via Friedman tests and Wilcoxon signed-rank follow-ups (Holm-adjusted). Inter-rater reliability was evaluated by agreement and Cohen’s $\kappa$. Results mirrored qualitative trends: visual feedback was endorsed by 6/8 (75%, 95% CI 34–96%) vs audio 4/8 (50%, 16–84%); the contrast was nonsignificant (p = 0.29, OR = 3.0, 95% CI 0.3–∞). Hybrid and transparent designs each reached 6/8 (75%, 34–96%), minimalism 4/8 (50%, 16–84%). Per-clip ratings showed scenario-level effects consistent with Study 2 (e.g., higher urgency, lower clarity for vanishing lanes). Given N = 8 and binary/ordinal data, exact and nonparametric tests provide valid small-sample inference and clarify which contrasts reach significance.

4.2. Study 2: Interactive Simulator Study

We conducted a controlled interactive user study in a driving simulator configured as a SAE level-3 AV. Participants were distracted during observing the vehicle driving. They were either instructed to perform a NDRT or were conversing with the interviewer while the vehicle was in automated mode until a situation arose that required confirmation or takeover.

4.2.1. Ethical Approval, Duration, Participants Recruitment and Compensation

The user study was conducted after approval by a university ethical committee. The participants were voluntarily taking part in the study and were recruited with leaflets or announcement in social networks. The duration of the study was between 45 and 60 min, and we compensated the participation with lunch vouchers.

The study took place in two university campuses. In one campus, fifteen participants participated in the study using the racing simulator rig, and the other five conducted it on a computer with a gaming steering wheel (cf. Section Simulator). To assess potential hardware and site effects, we compared the racing-rig group (Aalto, N = 15) against the gaming-wheel group (Lund, N = 5) on takeover rate, NASA-TLX subscales, and global trust ratings (Table 2). No comparison approached significance (all p > 0.27, Mann–Whitney U or Fisher’s exact test), and effect sizes were negligible to small (Cliff’s $\delta$ between −0.32 and +0.31; takeover rate 35.6% vs. 33.3%, OR = 1.10, p = 1.0). These results are consistent with comparable outcomes across the two configurations, though we note that hardware and study site were fully confounded, and the small gaming-wheel subsample limits power to detect smaller effects.

Table 2. Comparison of racing-rig and gaming-wheel groups across key measures. All effect sizes are Cliff’s $\delta$; interpretation follows conventional thresholds (negligible < 0.15, small < 0.33).

Measure	Racing Rig (n = 15)	Gaming Wheel (n = 5)	p	Cliff’s $\mathit{\delta }$
Takeover rate	35.6%	33.3%	1.000	- (OR = 1.10)
Mental Demand	46.7 ± 28.7	45.0 ± 20.9	0.965	+0.03
Physical Demand	24.3 ± 21.1	20.0 ± 21.5	0.567	+0.19
Temporal Demand	42.7 ± 27.6	49.0 ± 23.3	0.511	−0.21
Performance	32.0 ± 29.7	18.0 ± 10.4	0.597	+0.17
Effort	40.7 ± 28.1	29.0 ± 8.9	0.510	+0.21
Frustration	29.3 ± 29.0	27.0 ± 20.5	0.930	−0.04
Comfortable relying	3.40 ± 0.91	3.80 ± 0.84	0.485	−0.21
Explanation appropriate	3.53 ± 1.06	3.00 ± 0.71	0.317	+0.31
Feel safe	3.47 ± 0.99	3.40 ± 1.14	0.891	+0.05
Appropriate decisions	3.73 ± 0.88	4.20 ± 0.45	0.277	−0.32

4.2.2. Study Design

The study employed a mixed design. Each participant experienced the three mentioned conflict types: (1) narrowing road, (2) vanishing lane markings, and (3) negotiation (cf. Figure 4) with another vehicle. These represent distinct challenges for human–automation interaction in takeover situations. Communication strategies were counterbalanced across participants. This ensures that every conflict–strategy combination is represented across the sample, without participants encountering the same conflict multiple times.

To support conflict resolution, two communication patterns were evaluated (cf. Table 3)—based on the knowledge gained in the focus group (cf. Section 4.1): (1) visual feedback (Visual only) and (2) visual feedback combined with audio and textual information (Visual + Text/Speech). Levels of urgency were conveyed through dynamic visual changes and audio signals. Each TOR, before the presentation of the visual or audio information, was indicated with an audio signal (beep), and a warning icon in the rearview mirror (cf. Section 3.3).

Table 3. Overview of simulator feedback conditions and scenario-specific elements.

Condition	Included Elements	Scenario-Specific Elements
Visual only	Visual trajectory/conflict visualization in the HUD	Visual content adapted to the scenario context
Visual + Text/Speech	Visual trajectory/conflict visualization in the HUD, pre-warning cue (beep + red warning icon), textual or spoken explanation	Explanation content adapted to the scenario context
Shared across all conditions	Driving task, scenario structure, TOR audio signal and icon

Sensitivity Analysis

Given the sample sizes involved (N = 20 for the interactive study, N = 8 for the focus group), we conducted a sensitivity analysis to contextualize our statistical results. For the within-subjects comparisons in the interactive study, the design had 80% power ($\alpha =0.05$, two-sided) to detect effects of Cohen’s d ≈ 0.66 or larger; for the focus group, the corresponding minimum detectable effect was d ≈ 1.16. This suggests that the study was well positioned to detect medium-to-large effects, but not small ones. Accordingly, the negligible effect sizes observed for the text-vs-no-text comparisons (Section 4.2.3) are consistent with the absence of a practically meaningful difference, while still leaving open the possibility of smaller effects that the study was not powered to detect. As reported in Section 4.2.3, the bootstrap confidence intervals for these effect sizes did not fall entirely within the conventional negligible-effect bound, so this is best read as consistent with no meaningful difference rather than as confirmed equivalence.

Procedure

After an introduction and training drive, participants completed three experimental trials, each featuring one conflict type paired with one communication strategy (cf. Figure 13). After the main trial, the participants were played the same scenario with the alternative communication strategy and were asked if they noticed any difference in the way of communication. If they did not, we did not immediately tell them the difference in order to not bias them for the following scenarios. We explained it after the end of the study. After each trial, participants provided short ratings on clarity, trust, urgency, action confidence, perceived helpful elements, and expectancy. At the end of the three trials, participants completed a NASA-TLX [69] workload assessment (cf. Section NASA Task Load Index) and a global trust questionnaire (cf. Section Global Trust), and were asked to rank the strategies by preference. Finally, a brief post-study interview was conducted to collect qualitative impressions (cf. Section Overall Impressions).

Figure 13. Illustration of the interactive user study protocol, showing the sequence from an initial manual practice drive to automated control with an NDRT, conflict initialization and TOR, the addition of visual elements, and the three conflict scenarios (vanishing lanes, narrowing road, negotiation) that result in either manual takeover or confirmation of the vehicle’s suggested maneuver.

Objective measures include the resolution choice (takeover or following the suggested path). Subjective measures consist of per-trial ratings, workload (NASA-TLX), global trust ratings, strategy preference rankings, and qualitative feedback from the interviews.

Simulator

We conducted the study with 15 participants on a racing simulator rig (see Figure 14): a Simucube 2 Sport racing simulator wheel, Heusinkveld Sprint pedals, racing seat, and a 34” wide curved monitor. We set the wheel’s force feedback motor to give a constant resistance to mimic the feel of driving a real car. The wheel and pedals were connected as virtual joysticks to a PC with an Nvidia RTX3090 graphics card. On the simulator rig, we ran CARLA 0.9.15 and adapted an open-source toolkit for conflict simulation [8] (see Section 3.1) for performing the interactive user study. The other 5 participants used a similar setup with a 34” wide curved monitor, Logitech G29 steering wheel and the same version of CARLA.

Figure 14. The racing simulation rig (Simucube 2 Sport racing simulator wheel, Heusinkveld Sprint pedals, racing seat, 34” wide curved monitor) used for the interactive user study.

We found no significant differences between the two hardware configurations on any measure, though this comparison should be treated as inconclusive rather than confirmatory given the small gaming-wheel subsample and the confound with study site (see Section 4.2.1).

Non-Driving Related Task (NDRT)

During autonomous driving, participants engaged in NDRTs that were implemented on a separate handheld device to ensure that the primary driving visualization (i.e., the simulator screen and feedback modalities) remained unaffected by direct visual or spatial overlap. The NDRT was therefore designed as an independent secondary task without interference with the vehicle interface.

In cases where participants did not actively engage with the provided NDRT (e.g., by focusing primarily on the driving scene), the interviewer initiated a neutral conversation to maintain a comparable level of cognitive engagement across participants. This ensured that participants were not consistently operating in a fully idle state, which would not reflect typical level-3 automated driving conditions.

It needs to be mentioned that the NDRT engagement still varied between participants in terms of attentional distribution (e.g., reading vs. conversational engagement). However, because all NDRTs were primarily visual in nature and spatially separated from the driving interface, their influence is expected to affect general cognitive load rather than selectively bias specific feedback modalities. Nevertheless, this variability represents a limitation of the study and may have introduced differences in available attentional resources during takeover and interpretation of system feedback.

4.2.3. Results

The demographics and driving experience of the participants in the user study are presented in Table 4. The study was conducted in a university campus and most of the participants (75%) were in the age group 25–34 years. Overall 4 female participants were part of the study. 7 participants had experience with ADAS and 12 had experience with driving simulators or racing games.

Table 4. Participant demographics (N = 20).

Variable	Distribution
$N_{\mathrm{Participants}}$	20
Age	18–24: 2 (10%); 25–34: 15 (75%); 35–44: 3 (15%)
Gender	Female: 4 (20%); Male: 16 (80%);
Driving hrs/week	$\mu =5.075$
Use of ADAS	Yes: 7 (35%); No: 13 (65%)
Simulator experience	Yes: 12 (60%); No: 8 (40%)

Preliminary Questions

Before the study, we asked the participants about their definition of conflict in the context of automated driving. Conflict was most often described as unexpected situations (7), followed by technical malfunctions or limitations (5) and differences between driver expectations and vehicle actions (4), while fewer responses mentioned ethical or interpersonal conflicts (2), obstacles (2), accidents (1), or spatial conflicts (1) (two participants gave two responses). When asked what information they would want in such situations, participants most frequently mentioned visual (6) and audio (6) cues. They also emphasized the need for a situation summary (4), environment details (4), and information about intent (4). Less often, they requested risk information (3), possible resolutions (3), proximity cues (3), or technical details (2) and a TOR (2). On average, participants gave a neutral-to-slightly-positive response to the statement I would fully trust an automated vehicle to handle unexpected situations on its own (3.0 ± 0.85) on a 5-point scale (1 = strongly disagree, 5 = strongly agree).

Scenario Rating

After each scenario, the participants were asked to evaluate the Clarity, Trust, Urgency, Confidence and Anticipation on a 1–5 Likert scale. The summary is displayed on Figure 15. The scenario Narrowing Road was the most clear one, with highest trust and least experience of urgency. The scenario Vanishing Lanes was rated as the least clear and felt most urgent. Most of the participants could not understand what was happening and why the vehicle wanted to pass over the control. The placement of the screen with the projected detected lanes was problematic for most participants—some did not notice it at all, and even the ones who did could not relate the visual information to the situation. In this scenario, the audio descriptions were found the most helpful for understanding the situation from all three scenarios, when a voice message asked the participants to take manual control because of lost lanes.

Figure 15. Ratings ($\mu \pm \sigma$) for the three scenarios narrowing road, negotiation, and vanishing lane markings.

Ratings differed significantly across scenarios for trust (${\chi }^2\left(2\right)=10.92$, p = 0.004, Kendall’s W = 0.27), urgency (${\chi }^2\left(2\right)=8.03$, p = 0.018, W = 0.20), and anticipation (${\chi }^2\left(2\right)=7.75$, p = 0.021, W = 0.19), but not for understanding (${\chi }^2\left(2\right)=4.85$, p = 0.089) or confidence (${\chi }^2\left(2\right)=0.68$, p = 0.710). Holm-corrected post hoc Wilcoxon tests showed that Narrowing Road elicited significantly higher trust than both Negotiation (p = 0.009, r = 1.00) and Vanishing Lane Markings (p = 0.020, r = 0.80), and higher anticipation than Vanishing Lane Markings (p = 0.011, r = 0.71). Vanishing Lane Markings elicited marginally higher urgency than Narrowing Road (Holm-p = 0.056) and significantly lower anticipation than both other scenarios (both Holm-p < 0.02). These results statistically confirm the pattern visible in Figure 15: the Narrowing Road scenario was best understood and most trusted, while Vanishing Lane Markings was the most urgent and least anticipated.

Evaluation of Communication Strategies

We asked the participants to rank the communication strategies they observed—Visual only or Visual + Text/Speech. Then we asked them to provide free-text explanations about each separate element of the communication: Notification (beep signal and a red warning icon in the mirror), Visual strategy, Text + Speech. We manually classified the sentiment of these free-text explanations into three categories: clearly positive, clearly negative, and a suggestion for improvement for the strategy. Because this classification was performed by a single researcher no second coder or reliability check was applied here, and the categorization should therefore be treated as an indicative rather than independently validated classification of participant sentiment.

On the direct question about preferred strategy, twelve (12) of the participants marked the strategy including speech as their preferred choice, and eight (8) chose the visual-only strategy. However, when the sentiment labels of the free-text explanations were considered (cf. Figure 16), the results look different. Our findings suggest that notifications (beep + icon) are a reliable first step to capture attention and should be a default element of conflict communication design. In contrast, textual/speech explanations must be kept extremely short or context-specific, as prolonged sentences risk distracting the driver in urgent scenarios. The visual explanation (showing the suggested path for the correction) was positively evaluated by fifteen (15) participants. There were suggestions to add more descriptive HUD elements, and that the color of the proposed trajectory should be changed. The textual/speech explanations were positively rated by six (6) participants, seven (7) were certain that they are either not helpful, could be ignored or can even be harmful in urgent scenarios as they would distract the driver from the visual processing of the situation. There were also suggestions that speech can be helpful, but the written text either will not be noticed or it should be placed on a better location, or that it can be distracting. There were also suggestions for showing short and clear messages, instead of full sentences.

Figure 16. Evaluation of communication strategies.

During the user study, after the main run of each scenario, we showed each participant the alternative communication strategy for the same scenario, and asked the participants whether they noticed any difference between the two runs. If they did, we asked them if they thought the textual explanation was helpful, and we counted the number of people who were certain that the scenario with the textual descriptions was helpful. Table 5 summarizes the results. Less than half of the participants noticed the textual explanations, and a small part of them were certain that they are helpful. Note that the evaluation at the end of the study was done after we explained the distinction between the strategies, and the participants reconsidered that the textual explanations are actually helpful. However, during the scenarios themselves, these explanations were often overlooked or disregarded. These results indicate that textual descriptions could be helpful for situation awareness, but their use needs to be carefully designed.

Table 5. Perceived usefulness of voice descriptions across scenarios.

Scenario	Difference Noticed	Helpful
Negotiation	7	3
Narrowing Road	8	5
Vanishing Lane Markings	8	5

Manual takeovers occurred in 35.5% of text trials and 34.5% of no-text trials, with no detectable difference between conditions (${\chi }^2\left(1\right)=0.01$, p = 1.0, Fisher’s exact test; OR = 1.05, $\varphi =0.01$). Consistent with these mixed impressions, explanatory text also did not reliably change subjective ratings: trust (3.71 ± 1.22 vs. 3.83 ± 1.00), understanding (4.13 ± 0.96 vs. 4.07 ± 1.10), urgency (3.61 ± 1.15 vs. 3.52 ± 1.12), confidence (4.39 ± 0.62 vs. 4.48 ± 0.74), and anticipation (3.74 ± 0.89 vs. 3.48 ± 1.24) were statistically indistinguishable between text and no-text trials (all p > 0.35, Mann–Whitney U). Effect sizes for all five comparisons were negligible (Cliff’s $\delta$ between −0.12 and 0.08), and the sensitivity analysis (Section Sensitivity Analysis) indicates the study was adequately powered to detect medium-sized effects, suggesting these results are unlikely to be an artifact of insufficient power. To formally assess equivalence, we computed bootstrap 95% confidence intervals for each $\delta$ and compared them against the conventional negligible-effect bound (±0.15); none of the five intervals fell entirely within this bound (e.g., understanding: [−0.29, 0.26]; confidence: [−0.38, 0.14]). As with the hardware comparison (Section 4.2.1), we therefore report these results as consistent with no meaningful difference rather than confirmed equivalence, since our sample size does not provide sufficient precision to formally establish it.

NASA Task Load Index

The NASA Task Load Index (NASA-TLX) questionnaire [69] was included as a post-trial measure to assess perceived workload during the different automated driving conditions and to provide a manipulation check for the cognitive demands induced by the experimental setup. In particular, it allows characterizing whether the implemented feedback and NDRT combination resulted in meaningful variations in perceived mental and temporal demand.

The reported values represent aggregated scores across all experimental conditions and participants, providing an overall estimate of perceived workload in the study rather than condition-specific effects ($\mu$ and $\sigma$). Overall, participants reported moderate workload levels (cf. Table 6). Mental (46.3 ± 26.5) and temporal demand (44.3 ± 26.2) were highest, while physical demand was low (23.3 ± 20.7). Performance scores were also low (28.5 ± 26.7), suggesting participants felt effective. Effort (37.8 ± 25.0) and frustration (28.8 ± 26.6) were mid-range, pointing to cognitive and temporal demands as the main workload contributors.

Table 6. NASA-TLX scores ($\mu \pm \sigma$, 0–100).

Subscale	Score
Mental Demand	46.3 ± 26.5
Physical Demand	23.3 ± 20.7
Temporal Demand	44.3 ± 26.2
Performance	28.5 ± 26.7
Effort	37.8 ± 25.0
Frustration	28.8 ± 26.6

Global Trust

At the end of the study, we asked the participants to evaluate their trust in the system with four questions shown in Table 7. Participants rated the appropriateness of the vehicle’s decisions relatively high (3.85 ± 0.81). Perceived safety (3.45 ± 1.00), the appropriateness of the explanations (3.40 ± 0.99), and comfort with the communication strategies (3.50 ± 0.89) were all above the scale midpoint, although ratings showed high variability across participants.

Table 7. Global trust ($\mu \pm \sigma$, 1–5 scale).

Question	Score
I would be comfortable relying on this communication strategy in similar situations.	3.50 ± 0.89
The amount of explanation was appropriate.	3.40 ± 0.99
I feel safe when using this strategy.	3.45 ± 1.00
The vehicle makes appropriate decisions or recommendations under time pressure.	3.85 ± 0.81

Overall Impressions

At the end of the study, we asked the participants Would you want this system in a real car? The responses indicated cautious acceptance: Yes: 9; Maybe: 10; No: 1.

In the post-study questionnaire, we asked the participants what they liked, disliked, and would improve in the scenarios they observed. The answers were in free-text format and we performed thematic coding of the responses. Participants liked the use of visual cues (8 mentions) and beep or notification signals (7), which they felt were effective for capturing attention and guiding action. Textual explanations were also appreciated (4), mainly when they clarified what the system was doing. At the same time, textual explanations were the most frequent point of criticism (6), often described as distracting and poorly placed, and participants noted that warnings were sometimes issued too late (4). Other concerns included confusing visual displays (3), overlapping cues causing overload (2), and slow interaction steps that felt unsafe (2).

Participants suggested improvements such as earlier warnings, clearer visual guidance with less text, and more contextual information (e.g., pedestrians, traffic, speedometer). They also noted issues with vehicle motion and the mirror display, and recommended simpler interactions (one-click, voice, immediate takeover) or automatic action after a timeout. Overall, they emphasized the need for more realistic scenarios and adjustable levels of communication detail, pointing to adaptive and configurable interfaces as a promising direction.

5. Discussion

Given the exploratory nature and limited sample sizes of both studies (see Section Sensitivity Analysis for a sensitivity analysis), the following findings should be interpreted as hypothesis-generating rather than definitive evidence, warranting confirmation in larger, more demographically diverse samples. The study’s findings underscore the importance of flexible communication strategies, as participants interpreted conflicts in diverse ways. The ratings assigned to each scenario demonstrated a strong correlation between intuitive mappings between system output and driver perception and clarity and trust. The narrowing road scenario was well understood; however, the scenario of the vanishing lanes revealed significant shortcomings in visual communication.

The evaluation of communication strategies was found to be inconsistent. The pre-warning cues, such as the audible alert and the mirror icon, were met with universal approval, and the visual trajectory displays were generally trusted. Conversely, the impact of speech and text explanations was more polarizing. While some participants found them beneficial due to the additional context they provided, many others found them distracting or did not take them into account at all. Spoken explanations were most useful when visual cues were weak, such as in the vanishing lanes scenario or in lower-urgency situations, such as the narrowing road scenario, where participants had more attentional capacity. These explanations were more often reconsidered as useful only after the scenarios had ended, but mostly were not noticed or were found distracting. Written on-screen text was often unnoticed or considered unhelpful during interactive driving.

The workload ratings indicated moderate mental and temporal demands, while trust in system decisions was above average but varied across individuals. The findings indicate that concise visualizations and clear notifications are effective for conflict communication. In contrast, text and speech require careful integration to avoid information overload and ensure relevance.

Together with the focus group findings, the results indicate that preferences for multimodal feedback do not translate uniformly into effective use during driving. The usefulness of additional modalities depended on visual clarity, urgency, and timing. This explains the inconsistent evaluations of communication strategies and highlights the need for context-sensitive integration rather than fixed modality combinations.

Our results extend and, in one case, complicate prior findings on multimodal HMI design for automated driving. The visual trajectory display was consistently well-received, confirming Colley et al. [51], who similarly found a concise spatial-plus-intent AR display among the most preferred of four visualization concepts. The pre-warning cues were met with universal approval, in line with Alyamani et al. [19], who found that combining all three warning modalities was associated with the highest reported stress and distraction, suggesting lighter-weight alerting can be preferable to maximal redundancy. In contrast, the vanishing-lanes advantage we observed for speech contrasts with Avetisyan et al. [50], who found satisfaction shifted toward combined visual-auditory delivery as information volume increased, regardless of visual scene clarity; this may reflect that their explanations accompanied hypothetical videos rather than an active driving task. Most notably, the gap we found between stated preference for speech and its actual disregard under time pressure contrasts with Wang et al. [48], who found conversational agents preferred and associated with safer takeover behavior; we attribute this to their agents delivering brief prompts during low-workload monitoring, whereas ours carried full explanatory content during an active, time-critical conflict. This gap also echoes a broader tension in explainable AI, where explanation modality may need to shift with situational urgency rather than remain fixed [6].

5.1. Limitations

The present study is not without its limitations. Both studies used relatively small samples (N = 8 and N = 20), which is common in exploratory HMI and simulator research but limits the precision of effect estimates and the ability to detect subtle differences between the presented feedback strategies. Moreover, the simulator sample was predominantly young and male, whereas previous work indicates that age and gender can influence takeover behavior and conflict perception [14]. The present results should therefore be interpreted as hypothesis-generating and may not transfer directly to older drivers or more gender-balanced populations.

Initially, the evaluation was conducted using a limited set of feedback modalities. Alternative approaches, such as augmenting surrounding vehicles with contextual icons or implementing foveated rendering to direct attention, were considered but not implemented. Secondly, the implementation of the textual feedback occurred through a “Wizard-of-Oz” approach, as opposed to the utilization of trained models, which limits the inferences that can be drawn about their real-time performance. Thirdly, it is important to note that all evaluations were conducted within a controlled simulation environment. This environment, by its very nature, is incapable of fully capturing the complexity and stress associated with real-world driving conditions.

Relatedly, the two simulator setups differed in hardware (a Simucube base with Heusinkveld pedals at one site vs. a Logitech wheel at the other), and this difference was fully confounded with study site: all racing-rig participants were recruited at one site and all gaming-wheel participants at the other, so any apparent hardware effect cannot be separated from a site, experimenter, or protocol effect. We report a formal comparison across the two configurations in Section 4.2.1; no measure showed a significant difference and effect sizes were negligible to small, but the small gaming-wheel subsample (N = 5) leaves this comparison underpowered, so the result should be read as inconclusive rather than as confirmed hardware equivalence.

Furthermore, the inclusion of only a limited number of conflict scenarios and a relatively small, mixed participant pool constrained the generalizability of findings. Finally, aspects such as timing, escalation strategies, and potential cognitive overload from multimodal feedback were not systematically examined and remain for future research.

Additionally, text and HUD element sizing were tuned through iterative visual inspection rather than validated against formal legibility standards, which may limit the precision of conclusions drawn about textual/HUD readability specifically.

The sentiment classification of free-text responses in the interactive study (cf. Section 4.2.3) was conducted by a single coder without an independent reliability check, unlike the focus group’s thematic coding; future work should employ multiple coders with reported inter-rater agreement.

5.2. Outlook

In future work, we plan to complement the present questionnaire-based evaluation with objective behavioral and physiological measures. In particular, we aim to integrate eye tracking into our simulator and semi-autonomous research vehicle to capture gaze allocation, fixation behavior, and attention shifts between the HUD, road scene, and NDRTs. Coupling these measures with physiological indicators such as heart rate variability would enable a more fine-grained analysis of workload, arousal, and trust dynamics during conflict resolution and TOR.

A second direction is to move from the currently static multimodal interface toward adaptive feedback strategies. The present results provide initial guidance on when concise visual cues are sufficient and when richer verbal explanations may be beneficial; these patterns can inform adaptive policies that adjust modality, timing, and level of detail based on conflict type, time budget, and driver state. In our ongoing work, we are preparing to deploy and iterate these interfaces in a real semi-autonomous research car at Aalto University, which will allow us to study adaptation under more varied real-world conditions, including motion cues, lighting changes, and mixed traffic.

Third, extending the evaluation beyond the simulator is crucial for external validity. The campus-car studies will serve as an intermediate step between controlled laboratory scenarios and deployment in production vehicles, helping to validate whether the observed benefits of pre-alerts and trajectory visualization transfer to real traffic contexts, and to identify any unanticipated usability issues that only emerge under genuine risk and environmental complexity.

Finally, we see opportunities to align the interface more explicitly with emerging HMI standardization efforts. The present HUD and alert concepts can be mapped to aviation-inspired design principles, such as centralized, prioritized alert management similar to Engine Indicating and Crew Alerting Systems (EICAS) in aviation, and future iterations should be evaluated against automotive HMI guidelines for legibility, placement, and consistency. Beyond academic dissemination, we therefore intend to translate our findings into practitioner-oriented summaries and support transfer into industrial design processes and contribute to more ergonomic, standardized multimodal interfaces in AVs.

6. Conclusions

This study examined communication strategies for conflict situations in automated driving through controlled simulated scenarios. The findings indicate that concise visualizations and clear pre-warning cues are most effective in supporting driver comprehension and trust in level-3 automated driving. Conversely, speech and text explanations necessitate meticulous design to avoid distraction, and their usefulness depends on urgency and clarity of the situation. Despite the study’s limitations, it underscores the significance of scenario-specific, multimodal strategies for transparent conflict communication and establishes a foundation for subsequent research in real-world contexts.

While the present work is situated in the domain of automated driving, the findings extend to broader questions of shared autonomy in HRI. Many HRI settings involve urgent coordination where the human must quickly ascertain the system state and intentions. Examples include collaborative robots in manufacturing, teleoperation of drones, or assistive robots in healthcare. Across these domains, the challenge lies not only in the dissemination of information to the human but also in the calibration of trust and attention under time constraints. The findings of this study indicate that brief, salient cues (e.g., auditory alerts or minimal visual trajectories) are particularly efficacious in capturing attention and facilitating rapid decision-making. In contrast, longer verbal explanations may overload users in time-critical situations. This underscores the necessity for adaptive multimodal feedback strategies that can transition from concise notifications to more detailed explanations, dependent on situational urgency and the user’s state. For HRI, this necessitates the design of transparent and context-sensitive interfaces, achieving a balance between rapid comprehension and profound understanding.