A User-Centered Teleoperation GUI for Automated Vehicles: Identifying and Evaluating Information Requirements for Remote Driving and Assistance

Abstract

Teleoperation emerged as a promising fallback for situations beyond the capabilities of automated vehicles. Nevertheless, teleoperation still faces challenges, such as reduced situational awareness. Since situational awareness is primarily built through the remote operator’s visual perception, the graphical user interface (GUI) design is critical. In addition to video feed, supplemental informational elements are crucial—not only for the predominantly studied remote driving, but also for emerging desk-based remote assistance concepts. This work develops a GUI for different teleoperation concepts by identifying key informational elements during the teleoperation process through expert interviews (N = 9). Following this, a static and dynamic GUI prototype was developed and evaluated in a click dummy study (N = 36). Thereby, the dynamic GUI adapts the number of displayed elements according to the teleoperation phase. Results show that both GUIs achieve good system usability scale (SUS) ratings, with the dynamic GUI significantly outperforming the static version in both usability and task completion time. However, the results might be attributable to a learning effect due to the lack of randomization. The user experience questionnaire (UEQ) score shows potential for improvement. To enhance the user experience, the GUI should be evaluated in a follow-up study that includes interaction with a real vehicle.

1. Introduction

As of 2025, driverless cars remain a rare sight, especially in Europe, even though they exist and are permitted by legislation [1]. However, if there is an automated vehicle (AV) on a public road, chances are high that a human is still in the loop. Major AV companies like Waymo [2] or Cruise [3] continue to rely on remote operators for monitoring, guidance, and driving [4]. From a car user’s perspective, attitudes toward autonomous and remote driving also vary. A McKinsey survey [5] of around 1500 car owners in China, Germany, and the United States found that consumers consider remote driving superior to autonomous driving in several aspects, including safety and adaptability to unexpected situations. Additionally, about 56% of respondents indicated they would be more likely to purchase an av if it included remote-driving features, as this could allow for broader usability across different road types and weather conditions. Thus, besides automation technology companies, actual teleoperation providers like Valeo [6], DriveU [7], Einride [8], and Fernride [9] have emerged.

If human involvement is essential for autonomous driving, then so is the user interface for interacting with the vehicle. In addition to intuitive interaction, situational awareness of the remote operator presents a particular challenge in teleoperation [10,11]. Since situational awareness relies mainly on the remote operator’s visual perception [12], particular emphasis is placed on designing the graphical user interface (GUI) to effectively convey essential information to the remote operator, enabling quick and safe decision-making and acting.

Therefore, our work aims to develop a user-friendly GUI focused on the necessary informational elements that complement the video feed for safe operations, following a user-centered design process, as follows:

• Defining teleoperation process steps for remote driving and remote assistance.
• Identifying essential GUI elements through expert interviews.
• Evaluating a static and dynamic variant of the developed GUI within the teleoperation process through an online study.

Typically, teleoperation GUIs present a camera feed along with additional information like the vehicle speed. Several studies have already explored optimal video presentation in teleoperation, focusing on aspects such as required video quality [13], camera perspective [14], and field of view [15]. In addition, research has examined a range of display concepts [13,16] and output devices, including head-mounted displays [17,18].

In contrast, this work focuses on the supplemental informational elements and how a holistic GUI design should look on a conventional monitor to accommodate all necessary information on a single display surface. The following summarizes the literature on evaluating teleoperation GUIs and deriving design recommendations. Table 1 provides an overview of the teleoperation concepts considered in the publications, the implementation approaches, setups (including output devices, displayed elements, and input devices), and participants. The table also distinguishes whether the evaluations focused on the overall GUI or individual display elements.

Based on expert interviews or observations, publications propose design recommendations or requirements for teleoperation interfaces without presenting a fully developed GUI in their work [19,20]. For instance, Graf and Hussmann [20] identified 80 general information requirements, such as object display and vehicle position, which they narrowed down to 20 essential elements considered critical for safe teleoperation. In contrast, Tener and Lanir [19] illustrated their GUI element suggestions in design examples. However, the proposed informational elements were neither integrated into a unified GUI nor empirically validated.

While the work by Lindgren and Vahlberg [21] presented a holistic GUI, their study evaluated a click dummy and derived general design guidelines, including information prioritization and the use of coding strategies, such as color, to enhance clarity. Nevertheless, their work did not explicitly examine the impact of individual GUI elements.

Further developing this line of research, Gafert et al. [22] and Bodell and Gulliksson [23] shifted the focus toward analyzing individual GUI components. Gafert et al. [22] were the first to translate the requirements defined by Graf and Hussmann [20] into 19 concrete GUI elements, which were then implemented into two display variants—one featuring the complete set of informational elements and the other encompassing a reduced display configuration. In a real driving study involving the teleoperation of a miniature vehicle, the density and relevance of displayed information were examined across the orientation and navigation phases. Bodell and Gulliksson [23] also conducted a user study to explore how camera images, maps, and sensor data can be effectively presented to enhance safety and efficiency. Other informational display elements, such as steering angle, turn signal indicators, or vehicle model information, were not included in their GUI and evaluation.

Table 1. Overview of related work regarding graphical user interfaces for teleoperation.

	Source	Implementation			Setup			Participants		Evaluation
		None	Simulation	Real World	Output Device	Display Elements	Input Device	No.	Experience	Display Elements	Overall GUI
Remote Driving	Tener and Lanir [19]	Observations/ interviews			Ultra-wide Monitor	Video feed, side mirrors, rear view, vehicle speed, other driving information → derive 25 teleoperation challenges and design guidelines	Steering wheel and pedals	8/14	Remote operators/automotive industry professionals, academic teleoperation researchers	(✓)
	Graf and Hussmann [20]	Interviews				80 collected requirements/20 extracted for safe AV teleoperation		18/10	Automotive industry professionals	✓
	Gafert et al. [22]			Miniature vehicle	Head-mounted display	36 requirements represented in 19 GUI elements: video feed, rear view, navigation map, vehicle speed, other driving information, task information, weather, …	Steering wheel and pedals	16	None	✓	✓
	Bodell and Gulliksson [23]		Gazebo		2 Monitors	360° video feed, vehicle speed, bird’s-eye view map, planned path of automation, LiDAR data	Steering wheel, gamepad controller	?	None	✓	✓
	Lindgren and Vahlberg [21]		Click Dummy		3 Monitors	Video feed, side mirrors, rear view, bird’s eye view, vehicle speed, other driving information, notification list, top bar with system information, time and weather → derive general design guidelines	Mouse	9	Engineers with experience in remote operation, remote operators		✓
Remote Assistance	Kettwich et al. [24]		Click dummy		6 Monitors, touchscreen	Video feed, overview map, vehicle data, disturbances; bird’s eye view, planned path of automation	Mouse, touch	13	Control center professionals		✓
	Schrank et al. [25]		Click dummy		6 Monitors, touchscreen	Video feed, overview map, vehicle data, disturbances; bird’s eye view, planned path of automation	Mouse, touch	34	University or state-certified technical degree (remote operator criteria acc. to German law)		✓
	Tener and Lanir [26]		Click dummy		Ultra-wide monitor, monitor on top, tablet	Video feed; rear view; video feed, status bar, planned path of automation, obstacle marking, notifications, vehicle status, weather, …	Touch	14	Experts in remote operation		✓

Table 1. Overview of related work regarding graphical user interfaces for teleoperation.

	Source	Implementation			Setup			Participants		Evaluation
		None	Simulation	Real World	Output Device	Display Elements	Input Device	No.	Experience	Display Elements	Overall GUI
Remote Driving	Tener and Lanir [19]	Observations/ interviews			Ultra-wide Monitor	Video feed, side mirrors, rear view, vehicle speed, other driving information → derive 25 teleoperation challenges and design guidelines	Steering wheel and pedals	8/14	Remote operators/automotive industry professionals, academic teleoperation researchers	(✓)
	Graf and Hussmann [20]	Interviews				80 collected requirements/20 extracted for safe AV teleoperation		18/10	Automotive industry professionals	✓
	Gafert et al. [22]			Miniature vehicle	Head-mounted display	36 requirements represented in 19 GUI elements: video feed, rear view, navigation map, vehicle speed, other driving information, task information, weather, …	Steering wheel and pedals	16	None	✓	✓
	Bodell and Gulliksson [23]		Gazebo		2 Monitors	360° video feed, vehicle speed, bird’s-eye view map, planned path of automation, LiDAR data	Steering wheel, gamepad controller	?	None	✓	✓
	Lindgren and Vahlberg [21]		Click Dummy		3 Monitors	Video feed, side mirrors, rear view, bird’s eye view, vehicle speed, other driving information, notification list, top bar with system information, time and weather → derive general design guidelines	Mouse	9	Engineers with experience in remote operation, remote operators		✓
Remote Assistance	Kettwich et al. [24]		Click dummy		6 Monitors, touchscreen	Video feed, overview map, vehicle data, disturbances; bird’s eye view, planned path of automation	Mouse, touch	13	Control center professionals		✓
	Schrank et al. [25]		Click dummy		6 Monitors, touchscreen	Video feed, overview map, vehicle data, disturbances; bird’s eye view, planned path of automation	Mouse, touch	34	University or state-certified technical degree (remote operator criteria acc. to German law)		✓
	Tener and Lanir [26]		Click dummy		Ultra-wide monitor, monitor on top, tablet	Video feed; rear view; video feed, status bar, planned path of automation, obstacle marking, notifications, vehicle status, weather, …	Touch	14	Experts in remote operation		✓

The publications presented so far all focus on remote driving. In addition to traditional remote driving workstations, startups and research initiatives are now introducing novel teleoperation workstations with alternative desk-based control and display concepts, primarily for remote assistance of the AV. Click-based remote assistance systems offer an intuitive way to issue high-level commands, with users favoring minimal involvement in the dynamic driving task unless the scenario feels unsafe [27,28]. This requires different user interface specifications [29] and research in remote assistance [30].

Implementations by Cruise [31], Motional [32,33], and Zoox [34] display an abstract representation of the surroundings alongside multiple video streams, including road layout and detected objects. Motional [32] also visualizes traffic lights on the map. However, these startups do not provide access to the entire GUIs, including supplemental informational elements, likely due to competitive reasons.

Both industrial implementations and research on remote assistance, especially in GUI design, remain limited. Kettwich et al. [24] designed a remote operator GUI with six screens and a touchscreen, based on a systematic analysis of use cases [35]. First, the GUI was evaluated as a click dummy by control center professionals, using offline videos. Building on this setup, Schrank et al. [25] conducted another study simulating scenarios in which an AV requires remote assistance. Although the click dummy achieved good usability and acceptance scores in both studies, it was not designed to determine which informational elements were necessary at specific points to achieve these results. Similarly, the results from the online user study by Tener and Lanir [26] allow for general conclusions about their simulation-based click dummy, revealing good usability scores and relatively positive user acceptance, but lacking information on the influence of individual informational elements and the appropriate timing of their display during the teleoperation process.

Consequently, research questions arise regarding the selection and timing of informational elements within a GUI and whether there are distinct differences in the information required for remote driving versus remote assistance. Given the absence of a clearly defined teleoperation process, an initial analysis is necessary to identify the informational elements required at each stage. Furthermore, this work presents and evaluates a GUI prototype that incorporates the informational elements identified as essential. This prototype serves as a basis for assessing the effectiveness of the selected elements. The detailed methodology is outlined in the following section.

2. Methodology

The development of the GUI for remote operation followed the user-centered design process [36]: (1) understand context of use, (2) specify user requirements, (3) design solutions, (4) evaluate against requirements. In the first step, the context of use was defined through interviews, which analyzed the teleoperation process for both remote driving and remote assistance. Given the limited user base, primarily within startups developing remote driving systems, teleoperation experts were consulted during system development. Based on the context of use, the experts assessed the importance of collected informational elements in a second step during the interviews to specify requirements. These requirements formed the basis for the GUI design, which was implemented as a click dummy in a third step. In the fourth step, the GUI was evaluated in an online study.

The following sections describe the approach used in the expert interview and the online study. The interviews and studies were conducted with ethical approval from the ethics committee of the Technical University of Munich (2024-79-NM-BA, 2024-82-NM-BA).

2.1. Expert Interview

To design a display concept for teleoperation, it was first necessary to understand the interaction process between a remote operator and the AV. According to the authors, no publications on the teleoperation process currently exist, so this process was developed through expert interviews. Therefore, nine international experts (median age: 33 with $SD=8$) were recruited, each with a minimum of one year of teleoperation experience, while six had three or more years of experience. Two participants were working as CEOs responsible for business development; three were researchers, and four were working as engineers or product managers responsible for technical solutions. Seven participants had already operated a remote-controlled vehicle (exclusive of toys).

To guide the participants through a generic interaction process, we defined an imaginary scenario in which each participant acts as a remote operator responsible for an AV fleet capable of self-driving but still requiring human support in unsolvable situations. An AV requires support by either (A) remotely driving the vehicle or (B) providing remote assistance to the vehicle. Due to the odd number of participants, five participants were randomly assigned to (A) remote driving (P_RD), and four participants to (B) remote assistance (P_RA) without considering the experts’ specifications.

Task I: For the first task, participants were provided with an empty template (Figure 1) with five predefined sections, ranging from self-driving to teleoperation and back, separated by vertical blue lines. The participants’ task was to define actions executed by either the AV or the remote operator during the interaction process. Subsequently, these actions were to be placed as blue-bordered boxes within one of the dotted boxes related to either the AV or the remote operator. The predefined actions in the first and fifth sections could be modified, and additional actions could be added. Multiple actions could be placed in series or parallel, with different durations represented by the length of the boxes. If actions spanned multiple predefined sections, they could also be placed across the blue section lines. Beyond that, participants were asked to assess the importance of a standardized teleoperation process for themselves and the industry.

In the next step, we aimed to determine which informational elements should be shown on a GUI for the remote operator in each section of the teleoperation process. As part of a literature review, more than 60 GUIs in the fields of teleoperation and in-vehicle systems were previously analyzed, and their display elements were compiled into a morphological box, resulting in 57 categories of informational components.

Task II: For the second task, the experts were asked to evaluate the 57 informational elements. The participants were provided with the same template, containing the predefined five sections of the teleoperation process (Figure 1) and were invited to rate the relevance of each informational element within these sections {0: irrelevant, 1: not necessary, but nice to have, and 2: necessary}. Thereby, the evaluation only assessed the permanent display of information. This task did not cover short-term visuals such as warnings (e.g., for making the remote operator aware of low tire pressure).

2.2. Online Study

Based on the results of the expert interviews (analyzed in Section 3.2), we designed a GUI, shown in Figure 2, containing the informational elements that were classified as at least 1: not necessary, but nice to have. The GUI uses the area of the vehicle’s engine hood as a dashboard for displaying vehicle parameters. Additional informational elements are overlaid in the top border area of the screen to minimize interference with the video stream.

For remote monitoring during self-driving in the transition sections, two display alternatives with varying levels of information density were developed. The first display variant corresponds to the GUI in teleoperation mode (Figure 2), ensuring a static GUI throughout the teleoperation process. The second display variant (Figure 3) features a reduced number of elements tailored to the teleoperation mode, allowing the GUI to dynamically adapt across the stages of the teleoperation process, incorporating insights from the expert interviews. When switching to teleoperation, both GUI variants change their appearance from a green to a blue tint, indicating that human input is required.

The online study aimed to evaluate and compare the static and dynamic GUI variants while validating the experts’ assessment of the necessity of displaying certain informational elements during teleoperation. To simulate real-world teleoperation, we developed an interactive online prototype that participants could click through, enabling a remote study without requiring peripherals like pedals or steering wheels by focusing solely on remote assistance. Therefore, we took a series of photos of a scenario in which the ego vehicle was driving about 50 m through an American neighborhood. To proceed to the next interactive screen and move the vehicle forward, participants had to click on the road in the picture. Otherwise, the GUI showed an error message.

Task I: After an introduction to teleoperation, the participants were required to interact with the static GUI while paying attention to the display elements. After starting the first interactive screen by clicking a button, the participant monitored the AV in autonomous mode until the AV requested the participant to support it. The participant guided the AV through the scenario by clicking a waypoint in the desired direction on the street surface. After solving the scenario, the participant was informed that the AV could continue driving autonomously. Hence, control was handed over to the AV. After the interaction with the prototype, the participants evaluated the static GUI using the system usability scale (SUS) [37] and the short version of the user experience questionnaire (UEQ) [38].

Task II: In the second task, participants were introduced to the dynamic GUI and had to solve the same scenario with this display variant again. We did not make any additional changes except for the reduced set of display elements during autonomous driving mode. After completing the SUS [37] and UEQ [38] again, participants were asked to identify their preferred GUI option and offer suggestions for potential improvements.

Task III: Finally, the participants were asked to complete a card-sorting task by allocating each informational element of the GUI to one of the following categories: {0: irrelevant, 1: not necessary, but nice to have, and 2: necessary}. Thus, each informational element was represented by a card containing an icon and a textual designation.

Participants: For the online study, a total of $N=36$ participants (4 female, 32 male) with an average age of 26.3 years ($SD=6.0$) were recruited. All participants, except for one, had a valid driver’s license. Of these, 28 participants reported having 10 or fewer years of driving experience, seven had 11 to 20 years, and one had more than 30 years. Fourteen participants drove almost every day, five a few days a week, twelve a few days a month, and four a few times a year. Thirty-one participants reported only short-distance travel (<200 km per round trip), three participants reported middle-distance travel (201 km to 500 km per round trip), and one participant reported long-distance travel (>500 km per round trip). The participants’ average affinity for technology interaction (ATI) (short version) score was $M=4.42$ with $SD=1.08$, and $\alpha =0.89$.

3. Results

The following section summarizes the results from the expert interviews on the teleoperation process and the evaluation of informational elements, as well as the results from the online study assessing the designed GUI as a click dummy.

3.1. Teleoperation Process

Before designing the GUI, the first step was to conduct an expert interview to define the context of use. Each expert, assisted by the interviewer, specified a teleoperation process in the given template (Figure 1) by defining different actions for the AV or remote operator. Since the experts did not modify the self-driving sections s1 and s5, we focus only on the transition from self-driving to teleoperation (s2), teleoperation by the remote operator (s3), and the transition from teleoperation to self-driving (s4). Similar actions were grouped wherever possible to compare the resulting process steps. Tasks executed by the AV are plotted in orange, and those executed by the remote operator in purple. The more frequently the experts mentioned a particular action, the higher the number and the more intense the color.

For remote driving (

Table 2. Specified teleoperation process steps for remote driving, including the number of mentions.

	Transition from Self-driving to Teleop.			Teleoperation (Remote Driving)			Transition from Teleop. to Self-driving
Autonomous Vehicle	$\downarrow$	Drive autonomously (sense, plan, act)	3	$\downarrow$	Send situation data	3	$\downarrow$	Drive autonomously (sense, plan, act)	1
		Send situation data	3		Wait	1		Send situation data	2
		Realize difficult traffic scenario	1		Follow driving commands	3		Follow driving commands	1
		Send notification	3		Monitor teleoperation	3		Monitor teleoperation	1
		Execute minimal risk maneuver	4		Offer assistance	1		Intervene if something goes wrong	1
		Offer interaction concept	1		Intervene if something goes wrong	1		Check confidence for self-driving	2
		Wait	1		Approve self-driving	1		Approve self-driving	2
		Approve teleoperation	1					Transition to self-driving	1
								Disconnect	1
Remote Operator	$\downarrow$	Wait	1	$\downarrow$	Take over for teleoperation	1	$\downarrow$	Send driving commands	1
		Receive notification	2		Select interaction concept	1		Monitor execution of driving commands	2
		Assess situation	2		Send driving commands	5		Guide vehicle back in safe state	3
		Take over for teleoperation	2		Monitor execution of driving commands	3		Request self-driving	3
		Select interaction concept	1		Guide vehicle back in safe state	2		Monitor self-driving	1
								Disconnect	2

), the main process starts with the AV driving autonomously and continuously sending situation data while the remote operator is waiting. However, an expert emphasized that teleoperation is only profitable if waiting time is utilized for other tasks, like fleet monitoring. Once the AV recognizes a problem, it sends a notification to the remote operator while executing a minimal risk maneuver. P2_RD suggested that an optimal takeover request design should enable the vehicle to ask for support. After receiving the notification, the remote operator assesses the situation. If the AV reached a safe state, the remote operator takes over for teleoperation and chooses one of the interaction concepts offered by the AV that the AV has to approve. Thus, the remote operator sends driving commands, which the AV monitors and executes consecutively, whereby the AV can offer remote operator assistance or automatically intervene if necessary. In turn, the AV’s execution is monitored by the remote operator. Once the difficult situation is bypassed, the remote operator guides the AV back into a safe state and requests autonomous driving. P3_RD emphasized that the vehicle must inform the remote operator when self-driving is available again. Therefore, the AV checks its confidence level for self-driving, approves the request, and transitions back to autonomous driving. Afterward, the remote operator monitors the AV until both disconnect.

For remote assistance (

Table 3. Specified teleoperation process steps for remote assistance, including the number of mentions.

	Transition from Self-driving to Teleop.			Teleoperation (Remote Assistance)			Transition from Teleop. to Self-driving
Auton. Vehicle	$\downarrow$	Assist with ADAS	2	$\downarrow$	Assist with ADAS	3	$\downarrow$	Assist with ADAS	2
		Collect information for learning	1		Collect information for learning	1		Collect information for learning	1
		Send notification	4		Stay in safe state	1		Check if vehicle is back on orig. trajectory	1
		Execute minimal risk maneuver	3		Approve teleoperation	1		Transition to self-driving	2
		Stay in safe state	1		Execute driving commands	2		Disconnect	1
		Offer predefined commands	1
Remote Operator	$\downarrow$	Identify task	2	$\downarrow$	Assess situation	2	$\downarrow$	Monitor self-driving	1
		Receive notification	2		Plan trajectory	3		Intervene transition if sth. goes wrong	1
		Connect to vehicle	1		Define autom. transition to self-driving	1		Check if vehicle can be stopped	1
		Assess situation	2		Confirm trajectory for execution	2		Stop the vehicle	1
		Take over for teleoperation	2					Hand over to AV	1
		Request teleoperation	2

The steps highlighted in gray are optional and only required when automated handover is not possible, necessitating a manual transition from a stationary position.

), the experts described the process during the transition from self-driving to teleoperation (s2), similar to the remote driving procedure. However, the interaction is continuously supported by ADAS while the vehicle collects information for learning. Furthermore, the experts emphasized that the remote operator first connects to the AV to assess the situation when a notification from the AV is received, checked, and confirmed. If the AV reaches a safe state, the remote operator can request or accept teleoperation and take over. In the specific scenario of waypoint guidance [39], the remote operator plans a trajectory by setting a series of waypoints. P4_RA proposes that this trajectory can also include a point where the transition to autonomous driving is performed automatically. Once the remote operator confirms the final trajectory, they monitor the AV’s execution. Thereby, the AV checks whether it is back on the originally planned trajectory and, if so, transitions back to self-driving. The remote operator can intervene in this transition if something goes wrong. If the transition to autonomous driving is not performed automatically, the remote operator tries to stop the vehicle and manually hands over to the AV. As a final step, the AV disconnects from the remote operator.

In the post-task survey, the question “How important is a standardized teleoperation process for you?” was answered with 4.6 on average on a 5-point Likert scale (1: very unimportant, 5: very important). The importance of a standardized teleoperation process for the industry was assessed at 4.7. However, P1_RA warned that overly detailed governmental standardization may hinder development, while P3_RA noted that higher standardization improves efficiency but is challenging due to varying conditions. P2_RD raised key research questions, including the optimal operator-to-vehicle ratio, minimum latency, expected market penetration of Level 5 autonomy, and the benefits of remote driving over assistance.

3.2. Informational Elements

The experts rated each collected informational element on a scale of $\{0,1,2\}$, regarding their assigned teleoperation concept (remote driving or remote assistance) and the respective section (s1–s5) in the teleoperation process. The average results are visualized in

Table 4. Heat map for remote assistance, remote driving, and their comparison based on the experts’ evaluation of 57 informational elements identified through the literature review {0: irrelevant, 1: not necessary, but nice to have, and 2: necessary}. (x) marks elements intended for the GUI but omitted due to the scenario and click dummy limitations.

	Part of GUI	Part of red. GUI	Remote Assistance					Remote Driving					Comparison
			s1	s2	s3	s4	s5	s1	s2	s3	s4	s5	s1	s2	s3	s4	s5
Vehicle Parameters
Vehicle Speed	x	x	1.75	1.75	2.00	1.75	1.75	1.40	2.00	2.00	2.00	1.40	−0.35	0.25	0.00	0.25	−0.35
Engine RPM			0.25	0.25	0.25	0.25	0.25	0.20	0.60	0.80	0.60	0.20	−0.05	0.35	0.55	0.35	−0.05
Gear	x		0.25	0.50	1.00	0.50	0.25	0.00	0.80	1.20	0.80	0.00	−0.25	0.30	0.20	0.30	−0.25
Mileage			0.00	0.00	0.00	0.00	0.00	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40
Range			0.25	0.00	0.00	0.00	0.25	0.60	1.20	1.00	0.60	0.60	0.35	1.20	1.00	0.60	0.35
Energy Consumption			0.00	0.00	0.00	0.00	0.00	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40	0.40
Energy Level (Gas/Battery)			0.75	0.50	0.50	0.50	0.75	1.00	1.60	1.20	1.00	1.00	0.25	1.10	0.70	0.50	0.25
Oil Level			0.00	0.00	0.00	0.00	0.00	0.20	0.00	0.20	0.00	0.20	0.20	0.00	0.20	0.00	0.20
Oil Temperature			0.00	0.00	0.00	0.00	0.00	0.20	0.00	0.20	0.00	0.20	0.20	0.00	0.20	0.00	0.20
Open Doors	x		0.50	0.50	1.00	0.50	0.50	0.60	1.00	1.20	1.00	0.60	0.10	0.50	0.20	0.50	0.10
Used Seats	x		0.25	0.50	1.00	0.50	0.25	0.40	1.60	1.00	0.80	0.40	0.15	1.10	0.00	0.30	0.15
Fastended Seat Belts			0.25	0.50	0.50	0.50	0.25	0.60	0.80	0.80	0.40	0.60	0.35	0.30	0.30	−0.10	0.35
Lateral & Longitudinal Accelerations (G-Forces)	x	x	0.75	1.00	1.00	1.00	0.75	0.40	1.00	1.40	1.00	0.40	−0.35	0.00	0.40	0.00	−0.35
Tire Pressure			0.25	0.25	0.25	0.25	0.25	0.40	0.20	0.40	0.20	0.40	0.15	−0.05	0.15	−0.05	0.15
Traction			0.50	0.50	0.75	0.50	0.50	0.20	0.60	1.20	0.60	0.20	−0.30	0.10	0.45	0.10	−0.30
Steering Angle (Front Wheel Angle)	x		0.25	0.75	1.50	0.75	0.25	0.40	1.60	2.00	1.60	0.40	0.15	0.85	0.50	0.85	0.15
Light	x		0.25	0.75	1.00	0.75	0.25	0.40	1.00	1.20	1.00	0.40	0.15	0.25	0.20	0.25	0.15
Turn Signal	x	x	0.50	1.00	1.50	1.00	0.50	0.60	1.60	2.00	1.60	0.60	0.10	0.60	0.50	0.60	0.10
Hazard Lights	x	x	0.50	1.00	1.00	1.00	0.50	0.60	1.60	2.00	1.60	0.60	0.10	0.60	1.00	0.60	0.10
Vehicle Model Information			0.75	0.75	0.75	0.75	0.75	0.40	1.00	0.60	0.60	0.40	−0.35	0.25	−0.15	−0.15	−0.35
Park Brake	x	x	0.50	1.00	1.00	1.00	0.50	0.60	1.20	1.60	1.20	0.60	0.10	0.20	0.60	0.20	0.10
Window Heating			0.00	0.25	0.25	0.25	0.00	0.40	0.20	0.40	0.20	0.40	0.40	−0.05	0.15	−0.05	0.40
Air Condition			0.00	0.25	0.25	0.25	0.00	0.40	0.20	0.40	0.20	0.40	0.40	−0.05	0.15	−0.05	0.40
Local Time			0.75	0.75	0.75	0.75	0.75	0.60	1.40	1.00	1.00	0.60	−0.15	0.65	0.25	0.25	−0.15
System Parameters
Overall System Status	x	x	1.33	1.33	1.33	1.33	1.33	1.40	1.20	1.20	1.20	1.40	0.07	−0.13	−0.13	−0.13	0.07
Sensor Quality			0.67	0.67	0.67	0.67	0.67	0.60	1.20	1.00	1.00	0.60	−0.07	0.53	0.33	0.33	−0.07
Latency	x		0.75	0.75	1.50	0.75	0.75	0.80	1.40	1.60	1.40	0.80	0.05	0.65	0.10	0.65	0.05
Network Quality	x	x	1.00	1.00	1.50	1.00	1.00	0.80	1.60	1.60	1.60	0.80	−0.20	0.60	0.10	0.60	−0.20
Communication
Microphone for Cabin Communication (On/Off)			0.50	0.75	0.75	0.75	0.50	1.60	2.00	2.00	2.00	1.60	1.10	1.25	1.25	1.25	1.10
Microphone for Outside Communication (On/Off)	x	x	0.50	1.50	1.50	1.50	0.50	0.80	1.60	1.60	1.60	0.80	0.30	0.10	0.10	0.10	0.30
Outside Sound (On/Off)	x	x	0.50	1.50	1.50	1.50	0.50	0.40	2.00	2.00	2.00	0.40	−0.10	0.50	0.50	0.50	−0.10
Cabin Sound (On/Off)			0.50	0.75	0.75	0.75	0.50	1.00	2.00	2.00	2.00	1.00	0.50	1.25	1.25	1.25	0.50
Chat (Textual Communication with Passengers)			0.25	0.50	0.50	0.50	0.25	1.20	0.60	0.60	0.60	1.20	0.95	0.10	0.10	0.10	0.95
Inside Camera	x	x	0.50	1.00	1.25	1.00	0.50	1.20	1.20	1.20	1.20	1.20	0.70	0.20	−0.05	0.20	0.70
Teleoperation
Realtime Evaluation of Teleoperation Input	x		0.00	0.67	1.33	0.67	0.00	0.40	0.60	1.40	1.00	0.40	0.40	−0.07	0.07	0.33	0.40
Control Owner	x	x	1.50	1.75	2.00	2.00	1.50	1.20	1.80	1.80	1.80	1.20	−0.30	0.05	−0.20	−0.20	−0.30
Control Mode (Interaction Concept)	x	x	0.50	1.00	2.00	1.00	0.50	0.40	1.80	1.40	1.40	0.40	−0.10	0.80	−0.60	0.40	−0.10
Environment
Camera Sensor Video Stream	x	x	1.25	2.00	2.00	2.00	1.25	0.80	2.00	2.00	2.00	0.80	−0.45	0.00	0.00	0.00	−0.45
Further Sensor Video Stream (e.g. LiDAR)	(x)		0.50	0.75	1.50	0.75	0.50	0.40	1.00	1.00	1.00	0.40	−0.10	0.25	−0.50	0.25	−0.10
Obstacle Highlighting and Abstraction	x		0.50	0.75	1.50	0.75	0.50	0.60	1.80	1.60	1.80	0.60	0.10	1.05	0.10	1.05	0.10
Traffic Sign Highlighting and Abstraction	x		0.50	0.75	1.25	0.75	0.50	0.60	1.60	1.40	1.60	0.60	0.10	0.85	0.15	0.85	0.10
Traffic Participant Highlighting			0.00	0.25	0.75	0.25	0.00	0.40	1.40	1.20	1.40	0.40	0.40	1.15	0.45	1.15	0.40
Traffic Lights Highlighting and Abstraction	(x)		0.25	0.50	1.00	0.50	0.25	0.40	1.60	1.40	1.60	0.40	0.15	1.10	0.40	1.10	0.15
Weather Conditions	x	x	1.00	1.00	1.00	1.00	1.00	0.60	1.40	1.00	1.00	0.60	−0.40	0.40	0.00	0.00	−0.40
Perspective Mode Selection	x	x	0.50	1.00	1.25	1.00	0.50	0.60	1.60	1.20	1.40	0.60	0.10	0.60	−0.05	0.40	0.10
Prediction
Predicted Ego Trajectory	x	x	0.50	1.00	1.75	1.00	0.50	0.40	1.20	2.00	1.60	0.40	−0.10	0.20	0.25	0.60	−0.10
Predicted Ego Position	(x)		0.25	0.75	1.50	0.75	0.25	0.60	0.80	1.40	0.80	0.60	0.35	0.05	−0.10	0.05	0.35
Predicted Traffic Participant Trajectory	(x)		0.25	0.50	1.25	0.50	0.25	0.40	0.60	0.80	0.60	0.40	0.15	0.10	−0.45	0.10	0.15
Predicted Traffic Participant Position	(x)		0.50	0.75	1.50	0.75	0.50	0.40	0.60	0.80	0.60	0.40	−0.10	−0.15	−0.70	−0.15	−0.10
Predicted Collision	(x)	(x)	0.50	1.00	1.75	1.00	0.50	0.40	1.00	1.40	1.00	0.40	−0.10	0.00	−0.35	0.00	−0.10
Explainable AI
Confidence Score			0.75	0.75	0.75	0.75	0.75	0.80	1.20	0.40	1.00	0.80	0.05	0.45	−0.35	0.25	0.05
Explanation			0.50	0.50	0.50	0.50	0.50	0.80	1.40	0.60	1.20	0.80	0.30	0.90	0.10	0.70	0.30
Routing
Compass			0.25	0.25	0.25	0.25	0.25	0.60	0.80	0.60	0.60	0.60	0.35	0.55	0.35	0.35	0.35
Navigation Direction (e.g. next turn)	x	x	1.00	1.00	1.50	1.00	1.00	0.60	1.80	1.60	1.60	0.60	−0.40	0.80	0.10	0.60	−0.40
Map	x	x	1.25	1.25	1.75	1.25	1.25	1.20	1.20	0.80	0.80	1.20	−0.05	−0.05	−0.95	−0.45	−0.05
Route (Long-term path on map)	x	x	0.75	1.00	1.50	1.00	0.75	1.20	1.00	0.60	0.60	1.20	0.45	0.00	−0.90	−0.40	0.45
Trip Information (ETA, distance, etc.)	x	x	1.00	1.00	1.50	1.00	1.00	1.00	1.00	0.60	0.80	1.00	0.00	0.00	−0.90	−0.20	0.00

Comparison: Positive numbers indicate more importance for remote driving, negative numbers for remote assistance; s1: self-driving vehicle in autonomous state, s2: transition from self-driving to teleoperation, s3: teleoperation by remote operator, s4: transition from teleoperation to self-driving, s5: self-driving vehicle in autonomous state.

with darker shades and higher values indicating greater relevance of an element.

For remote assistance, the experts consider informational elements like engine RPM, mileage, range, energy consumption and level, oil level, temperature, and the compass to be irrelevant, regardless of the teleoperation process section (s1–s5). In contrast, elements like vehicle speed, control owner and mode, and the camera sensor video stream appear to be mostly necessary at all times.

Although the remote driving heat map is comparable, it shows more high-necessity ratings (value $2.00$), emphasizing the need for additional informational elements such as steering angle, turn signals (used in conjunction with hazard lights), as well as a microphone for cabin communication, outside sound, cabin sound, and the predicted ego trajectory. However, P1_RA, P2_RD, and P3_RD noted that the need to toggle cabin or outside sound could be eliminated by implementing a system that only forwards relevant conversations to the remote operator. Furthermore, a few ratings—like energy level, range, used seats, vehicle model information, and local time—break the symmetry and are most important during the transition to remote driving (s2), as the remote operator may need this information to gain sufficient situational awareness.

The comparison heat map compares both evaluations for remote assistance and remote driving. The cells’ values within $[-2.0,2.0]$ were calculated by subtracting the remote assistance from the remote driving values. Hence, positive numbers (marked in green) indicate informational elements that were more important for remote driving. Vice versa, negative numbers (marked in blue) indicate elements considered more important for remote assistance. Notably, map, route, and trip information were rated as more critical for remote assistance, whereas traffic signs, lights, participant, and object highlighting and abstraction were considered more relevant for remote driving.

Overall, expert assessments of remote assistance and remote driving concepts do not differ significantly. However, the predominantly green shading in the comparison column indicates that many informational elements were considered slightly more important for remote driving ($0.22$ higher, on average, compared to remote assistance; Table 4).

3.3. Online Study

Design Derivation: Since the online study focuses on evaluating a novel GUI for remote assistance, the GUI concept was derived from the results of the expert interviews on remote assistance, particularly considering the highest ratings of the elements within the teleoperation section, s3, for the static GUI variant, and the importance across all sections for the reduced GUI (Table 4). The derived design for the click dummy (Figure 2) only contains the informational elements that were at least classified as nice to have, but not necessary by every expert, i.e., an average value of at least 1.00.

However, traffic light highlighting and abstraction were excluded because the scenario in the online study does not include any traffic lights, but the approach would be applied analogously to the traffic signs in the GUI. As the scenario does not include moving participants, no predicted traffic participant trajectory or position was highlighted. The explicit display of the further sensor video stream (e.g., LiDAR) and predicted collisions was omitted, as detected objects on the road were already highlighted with red line markings, and collisions were not possible due to the fixed scenario screens to click through. Similarly, the predicted ego position can be inferred from the predicted ego trajectory, visualized as a driving corridor. The resulting set of informational elements is indicated with an x and marked red in the column Part of GUI in Table 4. All elements omitted for the reasons mentioned above are marked with (x). The reduced GUI (Figure 3) contains fewer informational elements during the monitoring of autonomous self-driving that were, on average, considered as 1: nice to have, but not necessary in sections other than teleoperation (s1, s2, s4, s5).

Evaluation: The online study aimed to evaluate and compare the developed static and dynamic GUI. Since waypoints could only be placed on the road surface, it was possible to track misclicks. Misclicks occurred 10 times while using the GUI_Static and 14 times with GUI_Dynamic (

Table 5. Comparison results of the static and dynamic GUI.

	GUIStatic			GUIDynamic
	$\mathit{Mdn}$	$\mathit{M}$	$\mathit{SD}$	$\mathit{Mdn}$	$\mathit{M}$	$\mathit{SD}$
Misclicks	10	–	–	14	–	–
Task Completion Time (s)	61.73	76.7	37.7	41.25	64.8	69.6
SUS Score	70.00	68.5	12.8	78.75	76.6	12.5
UEQ Score	0.56	0.41	0.90	0.63	0.67	0.92
Pragmatic Quality	0.88	0.65	1.08	1.13	0.94	1.11
Hedonic Quality	0.38	0.16	1.10	0.50	0.40	1.08

).

Guiding the vehicle via clicks through the situation using GUI_Static took an average of 76.7 s whereas GUI_Dynamic reduced this time by 16%. As the Shapiro–Wilk test shows no normal distribution for the task completion time, the Wilcoxon test was applied to assess significant differences. The results reveal that using the GUI_Dynamic was significantly faster than using the GUI_Static ($z=-3.97$, p = $0.00007$) with an effect size of $r=0.66$, indicating a strong effect according to Cohen [40].

The participants rated the usability of GUI_Static , on average, with a SUS score of $68.5$, putting it within the higher “marginal” acceptability range and labeling it with the adjective “ok” [37]. In comparison, the GUI_Dynamic received a SUS score of $76.6$, ranking as “acceptable” while shifting the adjective to “good”. Due to the lack of normal distribution, a Wilcoxon test was conducted for the SUS scores, revealing that the SUS score for the GUI_Dynamic was significantly higher compared to the GUI_Static ($z=-4.11$, $p=0.00004$), with an effect size of $r=0.69$, indicating a strong effect [40].

The user experience of the GUI_Static was rated lower compared to the GUI_Dynamic in pragmatic, hedonic, and overall quality within the neutral evaluation range (Table 5). The GUI_Dynamic scored the highest rated value in pragmatic quality, reaching a positive evaluation. Although the UEQ data is normally distributed, the t-test showed no significant difference in the overall UEQ score ($t=-2.007$, $p=0.0525$), but there is a significant difference ($t=-2.160$, $p=0.0377$) for the pragmatic quality between GUI_Dynamic and GUI_Static with a small practical effect ($r=0.36$) [40]. For the hedonic quality, the t-test shows no significant difference ($t=-1.355$, $p=0.1842$).

In the post-task survey, users rated their preference between GUI_Static and GUI_Dynamic; 4 favored GUI_Static, 11 preferred GUI_Dynamic, 9 liked both, and 12 saw no difference.

Informational Elements: The results of the card sorting are visualized in

Table 6. Comparison of experts’ and participants’ ratings of informational elements during remote assistance, and their differences: positive values indicate higher participant ratings (pink), negative values indicate higher expert ratings (turquoise) {0: irrelevant, 1: not necessary, but nice to have, and 2: necessary}.

	Experts	Participants	Comparison
Vehicle Parameters
Vehicle Speed	2.00	1.97	−0.03
Gear	1.00	1.42	0.42
Open Doors	1.00	1.50	0.50
Used Seats	1.00	0.64	−0.36
Lateral & Longitudinal Accelerations (G-Forces)	1.00	0.44	−0.56
Steering Angle (Front Wheel Angle)	1.50	1.69	0.19
Light	1.00	1.28	0.28
Turn Signal	1.50	1.67	0.17
Hazard Lights	1.00	1.39	0.39
Park Brake	1.00	1.31	0.31
System Parameters
Overall System Status	1.33	1.50	0.17
Latency	1.50	1.00	−0.50
Network Quality	1.50	0.92	−0.58
Communication
Microphone for Outside Communication (On/Off)	1.50	0.72	−0.78
Outside Sound (On/Off)	1.50	0.64	−0.86
Inside Camera	1.25	0.56	−0.69
Teleoperation
Realtime Evaluation of Teleoperation Input	1.33	1.44	0.11
Control Owner	2.00	1.47	−0.53
Control Mode (Interaction Concept)	2.00	1.28	−0.72
Environment
Camera Sensor Video Stream	2.00	1.75	−0.25
Obstacle Highlighting and Abstraction	1.50	1.53	0.03
Traffic Sign Highlighting and Abstraction	1.25	1.56	0.31
Weather Conditions	1.00	0.81	−0.19
Perspective Mode Selection	1.25	1.06	−0.19
Prediction
Predicted Ego Trajectory	1.75	1.56	−0.19
Routing
Navigation Direction (e.g. next turn)	1.50	1.33	−0.17
Map	1.75	1.14	−0.61
Route (Long-term path on map)	1.50	1.42	−0.08
Trip Information (ETA, distance, etc.)	1.50	0.81	−0.69

. The experts and participants columns contain the respective average ratings of each informational element during the teleoperation section (s3) for remote assistance, while the Comparison column highlights their differences. Positive values (marked in pink) indicate elements rated higher by participants, whereas negative values (marked in turquoise) indicate elements rated higher by experts. The majority of vehicle parameters were rated significantly higher by the participants. On the other side, informational elements related to system parameters, communication, teleoperation, and routing were rated notably higher by the experts. Overall, four participants stated in their improvement suggestions that the GUIs, in general, contained too much information for the task. Three participants criticized the prototype’s limited interaction and the resulting challenges. Furthermore, three participants expressed a preference for the ability to customize the GUI to better suit their individual needs rather than relying on a default configuration.

4. Discussion and Limitations

Following the results, this section discusses their outcomes and limitations to lay the foundation for future research and ensure the proper interpretation of the findings.

4.1. Teleoperation Process

The following general process steps are derived from the remote driving and remote assistance processes defined during the expert interviews (Table 2 and Table 3), assuming the remote operator is not yet in the loop. It should be noted that the experts may not have explicitly stated certain aspects, as these were presumed to be self-evident, such as the AV also transmitting situational data to the remote operator during remote assistance.

• AV recognizes complex scenario or problem.
• AV notifies the remote operator while executing a minimal risk maneuver.
• Remote operator receives notification, connects to AV, and assesses situation.
• If the AV reached a safe state, the remote operator takes over for teleoperation.
• Thereby, the AV can offer different teleoperation concepts.
• Remote operator chooses a teleoperation concept (remote driving or remote assistance).
• AV has to approve the teleoperation with the chosen interaction concept.
• Execution of the dynamic driving task through the selected teleoperation concept (specific individual steps).
• Thereby, the AV checks and approves confidence for self-driving.
• Handing over to self-driving upon request of the remote operator or offer from the AV, or through automated transfer (especially in remote assistance).
• Remote operator monitors self-driving and can intervene if something goes wrong.
• Remote operator and AV disconnect.

Step (8) is further divided into substeps, with remote driving involving more low-level commands, whereas remote assistance remains at a high level due to the continued presence of autonomous features. This would allow an automatic transition back to self-driving for remote assistance. However, the handshakes in steps (4) and (10) remain unclear.

If the remote operator is already connected to the AV and is assessing the situation, they can request teleoperation and take over, according to step (4). The question remains—Does the remote operator request teleoperation intervention manually, or does the AV enter teleoperation mode autonomously and await commands? Similarly, step (10) involves the second handover transitioning back to self-driving. Further research should be conducted to determine which handover option is promising regarding transition time and reliability, as well as if and how an AV could automatically switch back to autonomous driving.

A further field of research is the right timing of a handover. Typically, the vehicle is handed over to the remote operator from a minimal risk state, which is likely to be a standstill. However, a transition process that enables seamless switching between autonomous driving and teleoperation without entering a minimal risk maneuver remains largely unexplored. Likewise, the transition back to self-driving can occur either from a standstill or while in motion.

Overall, experts generally consider standardizing the teleoperation process to be important or very important. Nevertheless, standardization should not hinder development but should instead allow adaptability across different applications and conditions. Variations in interaction, such as the handover to self-driving (step (10)), present significant challenges to establishing a standardized teleoperation process.

4.2. Informational Elements

The evaluation of informational elements reduced the number of collected potential display components by more than half, minimizing the GUI and ensuring the remote operator’s attention remained focused on essential information.

The initial experts’ assessment of informational elements indicates that features such as maps, route, and trip information are considered more critical for remote assistance, aligning with existing applications in Section 1. In contrast, traffic signs, traffic lights, participant and object highlighting, and abstraction were rated as more important for remote driving, likely because the remote operator is directly responsible for executing the driving task and maintaining vehicle control. Therefore, the remote operator may need to gain a more detailed situational awareness than in remote assistance. The overall higher rated relevance of informational elements in remote driving also suggests an increased need for situational awareness in this teleoperation mode.

However, it is essential to note that the experts came from various fields of teleoperation applications. Teleoperation in passenger transport focuses on safety, comfort, and user support in complex traffic situations, while logistics emphasizes operational efficiency with less focus on user experience, often in controlled environments such as warehouses. While both domains rely on remote human intervention, the presence of passengers rather than cargo shapes distinct operational priorities. Since the experts were randomly assigned to the remote driving and remote assistance groups, it is possible that one sector was overrepresented in a group. This may explain why, in remote driving, communication elements were considered more important, likely due to the higher proportion of experts from the passenger transport sector compared to the remote assistance group, which may have included more experts from logistics applications. This effect can be mitigated by increasing the participant pool or by evenly distributing the experts based on their prior experience.

Despite the small number of experts ($N=9$) and random distribution, the relatively symmetrical color distribution in the heat map in Table 4 shows that the elements are perceived as equally necessary or unnecessary, whether for remote driving or assistance.

Comparing the expert’s rating with the results from the participants of the online study (Table 6), the assessments for remote assistance can largely be confirmed by the larger participant sample. However, participants rated vehicle parameters as more important than experts, whereas experts placed greater emphasis on elements related to communication, map, trip information, latency, and network quality indicators, as well as control owner and mode display. This may be because participants in the online study did not have to interact with these elements, making the information less relevant in the click dummy scenario without an actual driving task. It also shows that the experts take a different perspective compared to the laypersons due to their prior experience.

Overall, it was noted in the online study that the GUI was overloaded with information and should be further reduced. However, it is again crucial to consider that many elements were not needed in the click task without a vehicle connection. In the next step, a user study involving the execution of teleoperation using a real-world vehicle should be conducted.

In the future, it is conceivable that the informational elements could be individually arranged and customized by the remote operator, as suggested by the three participants.

4.3. Online Study

The expert interview findings were translated into a GUI prototype, implemented in a static and dynamic variant using a click dummy. However, one-third of participants reported not noticing any difference between the two versions. Still, the dynamic GUI received a significantly higher rating in terms of usability. This result may be influenced by the presentation order, as participants were always shown the static GUI first, followed by the dynamic version. Consequently, increased familiarity with the system in the second trial could have led to better evaluations.

Similarly, the faster task completion times observed with the dynamic GUI may be attributed to the fixed order of presentation and potential learning effects. Additionally, both GUI variants were tested using the same scenario, which may have amplified the impact but led to better comparability.

The number of misclicks can likely be attributed to participants’ curiosity, as they attempted to challenge the system and explore its limitations. Nevertheless, the GUI variants achieved a marginally higher, acceptable-to-good SUS score, with pragmatic quality in the user experience falling within the (almost) green range. In contrast, the hedonic quality received lower ratings, remaining within a neutral range, possibly due to the GUI being evaluated only in a prototypical click dummy without real vehicle integration. Three participants even criticized the lack of proper interaction. Therefore, the results of usability and user experience cannot be solely assigned to the GUI but are also influenced by the interaction with the prototype. Future studies should explore the combined impact of interaction and display concepts on usability and system performance, considering various interaction approaches and real-world applications. Thus, studies should be conducted under real-world conditions, such as video streaming with associated transmission latencies and interaction with a vehicle.

Generally, it is important to note that the online study involved participants without prior teleoperation experience, in order to capture their intuitive impressions of the visual interface without bias from existing teleoperation systems and to reach a broader participant pool. However, it must be acknowledged that the participants were likely to have been naïve with respect to teleoperation. For instance, feedback indicating that the GUI contained too much information may be more reflective of their lack of experience than of actual usability issues. Similarly, differences in the evaluation of display elements compared to expert assessments (Table 6) could be explained by the participants’ lack of familiarity.

Nevertheless, 30% of the participants preferred the dynamic gui (33.3% noticed no difference and 25% liked both), which was generally rated higher regarding usability and user experience. Therefore, the display should adapt throughout the teleoperation process, providing only the necessary elements at each stage. However, the dynamic GUI and its reduced display during the automated driving monitoring phase still need to be further investigated with a real vehicle connection. In particular, the remote operator notification, including the switch to the scenario, and the subsequent handover to teleoperation, require further investigation.

5. Conclusions

As the core of this work, a graphical user interface (GUI) for the teleoperation of AVs was developed following the user-centered design process. First, process steps were defined by experts, outlining tasks to be performed by the AV or remote operator. Thus, a major unresolved issue was the handshake to the remote operator and the transition back to self-driving. Based on these steps, the experts evaluated the necessity of informational elements gathered from the state of the art. Building on the results, a click dummy was created, and the necessity of the informational elements was reassessed in an online study with a larger sample. Additionally, a comparison was made between a static and a dynamic GUI, showing reduced display elements during the automation monitoring phase. Although one-third of the participants reported no noticeable difference between the GUIs, the usability of the dynamic GUI was rated significantly higher. Overall, both GUIs received good system usability scale (SUS) ratings. Task completion time was also significantly faster with the dynamic GUI. Nevertheless, the lack of counterbalancing the order of the static and dynamic conditions limits direct comparisons, as any observed improvements in the dynamic condition may result from learning effects. However, the user experience questionnaire (UEQ) score suggests that there is still potential for improvement in terms of user experience. Based on the expanded insights, the GUI design should be further refined and integrated into existing interaction concepts. This enables iterative development, including future studies involving interaction with a vehicle. Overall, the developed display concept could serve as a standard interface for presenting key information in teleoperation research.