A Helping Human Hand: Relevant Scenarios for the Remote Operation of Highly Automated Vehicles in Public Transport

Abstract

Remote operation bears the potential to roll out highly automated vehicles (AVs, SAE Level 4) more safely and quickly. Moreover, legal regulations on highly automated driving, e.g., the current law on highly automated driving (SAE Level 4) in Germany, permit a remote supervisor to monitor and intervene in driving operations remotely in lieu of a safety operator on board AVs. In order to derive requirements for safe and effective remote driving and remote assistance of AVs and to create suitable human-centered design solutions for human-machine interfaces (HMIs) that serve this purpose, a set of 74 core scenarios that are likely to occur in public transport AVs under remote operation was compiled. The scenarios were collected in several projects on the remote operation of AVs across a variety of contexts including interviews with and observations of control center staff, video analyses from naturalistic road events, and interviews with safety operators of AVs. A hierarchical system that is based on interactions of central actors was used to structure the scenarios. The set explicates relevant cases in remote operation, which may help improve workplaces for remote operation both by combatting human factors issues such as distraction and fatigue, and by boosting usability, user experience, trust, and acceptance. As the catalogue of scenarios is not exhaustive, scenarios may be added as knowledge of the remote operation of AVs progresses. Further research is needed to validate and adapt the scenarios to specific conceptualizations of remote operations.

1. Introduction

As mobility demands grow while calls for more sustainable and environment-friendly travel options are becoming more vocal, the transportation sector is facing dramatic changes. Not only has there been a wave of technological innovations in driving automation, electromobility, and mobile network bandwidth [1,2,3,4,5]. There is also a previously unseen boom in innovative means of public transport such as ride-sharing and other flexible on-demand mobility solutions that may be serious alternatives to individual mass mobility [6,7]

However, even though automation technologies have demonstrated sharp improvements, there are still plenty of cases where a vehicle’s automation might be overwhelmed. Complex, multilayered sets of quickly evolving traffic situations particularly in urban mixed traffic environments present extreme challenges to highly automated vehicles (AVs). Some of these cannot be resolved by the automation alone as they exceed the vehicle’s Operational Design Domain, or ODD, i.e., the defined conditions in which the AV can operate autonomously in a safe way [8]. Also, a human on-board operator is not mandatory to serve as a fallback solution for highly automated vehicles (Level of Automation Four according to SAE’s terminology [8]). Instead, remote operation by a human operator may be a viable solution to enable automated driving without specifying every possible ODD, which is the requirement of fully automated driving according to SAE Level 5. This endeavor is likely not be fulfilled for a very long time, if ever. From the perspective of the substitution model, remote operation can be conceptualized as a substitute for the primary controller, in this case, the driving automation [9]. The remote operator could observe automated driving operations and intervene when the AV’s driving automation capabilities are exceeded. This approach is becoming increasingly feasible as computers’ processing speed and capacity have shown a sharp incline and high-bandwidth low-latency communication technologies with the option to prioritize certain kinds of data such as 5G have been widely rolled out [10].

In order to identify critical situations where remote operation could be used as an effective and efficient approach to support or resume highly automated driving, an extensive collection of scenarios with relevance to remote operation has been collected using a multimethod approach (see Section 2). This collection will help notice and address challenges in the practical use of remote operation solutions and support the human-centered design process of interfaces for remotely operating vehicles.

1.1. Regulation, Standardization, and Conceptualization of Remote Operation of AVs

In addition to technological leaps forward, legal environments have become more favorable toward using remote operations on public roads as well. For instance, the German Road Traffic Act (“Straßenverkehrsgesetz”) has been modified last year so it now explicitly permits AV of Level 4 on German roads—as long as they are monitored and controlled, if necessary, by a human operator coined “Technical Supervisor” (“Technische Aufsicht”) [11]. This supervisor can be either on board the vehicle or at another location. Thus, remote vehicle operations are now legally feasible on German roads. Also, in the UK, Sweden, Japan and a few US states, laws and regulations that require remote supervision of highly automated cars without an on-board driver have been passed [12]. Moreover, driverless operation of vehicles on public roads is now possible in major European countries like France and the UK [13] as well as at least 41 states of the US [14].

The standardization of remote operations has also seen tremendous steps forward. The latest update of the SAE’s Taxonomy for Driving Automation Systems [8] includes two conceptualizations of remote vehicle operations: Remote Driving and Remote Assistance. In Remote Driving (RD), a human operator is executing “real-time performance of […] the DDT (i.e., dynamic driving task such as braking, steering, or accelerating)” ([8] p. 19). Thus, remote driving resembles the conventional way of driving a vehicle: by initiating direct low-level driving maneuvers, including lateral and longitudinal motion control, right when the situation requires them. The Remote Driver, who is the actor to execute Remote Driving, may overrule the vehicle automation’s driving tasks.

In contrast, Remote Assistance (RA) is defined as an “event-driven provision, by a remotely located human, of information or advice to an ADS (i.e., automated driving system) equipped vehicle in driverless operation in order to facilitate trip continuation when the ADS encounters a situation it cannot manage” ([8] p. 18). Thus, the Remote Assistant supports the vehicle by providing high-level guidance on how to deal with certain situations that are not part of the automation’s ODD. This advice is provided well before the challenging situation and must not be time-critical. In addition, the automation must be capable of processing the high-level information provided and translate it into direct driving maneuvers. Examples for guidance range from simple “giving clearance” cases in which the vehicle requests an assessment of the situation from the Remote Assistant on how to proceed in a certain situation, e.g., when it is uncertain whether an identified object is an actual obstacle, to more demanding “setting trajectories or waypoints” cases that require the Remote Assistant to determine a pathway or waypoints of a pathway that the AV follows to circumvent an obstacle.

1.2. Real-World Tests of Remote Operation

The remote operation of AVs is currently being tested in a variety of real-world laboratories. These labs usually are collaborations between research-focused and industrial partners which examine the feasibility of novel technologies in a real-world setting, aiming at maximizing ecological validity. Thanks to their incorporation into naturalistic settings within the intended context of use, real-world labs offer invaluable insights into the in situ application of devices or systems that have previously only been investigated in higher controlled oftentimes experimental, yet less realistic environments. Thus, situations and phenomena are more likely to occur that may not have been observed in a more controlled setting. Since they demonstrate the interplay of the technology with users and other actors in a less standardized environment, they yield scenarios with a larger external validity, i.e., transferring them to other (real-world) contexts may be facilitated.

The German Aerospace Center (DLR) is involved in real-world labs that use both RD and RA for remote operation. Figure 1 displays examples of vehicles that are remotely operated within these projects. Regarding RD, the modular electric AV concept “U-Shift” that caters to different urban mobility demands, including transportation of people and goods, encompasses Remote Drivers in its system architecture [15]. Pertaining to RA, DLR is engaged in urban mobility projects that provide last-mile shuttle services from major hubs of transportation, e.g., train stations, to the final destination. In the “Hamburg Electric Autonomous Transportation”, or “HEAT”, project, a self-driving minibus ran along a fixed route through the Harbor District of the city incorporated in the public transport provider “Hochbahn”’s network [16]. “The Real-World Lab Hamburg (“Reallabor Hamburg”) provided on-demand service from a suburban railway hub to nearby neighborhoods that could be booked via a cellphone application [17]. The Berlin-based “KIS’M” project aims at demonstrating an AI-based system for connected mobility and at examining the interaction between human operators in the control center with passengers of remote-controlled AVs [18], utilizing an RA approach.

Figure 1. Remotely operated vehicles in investigated projects. (a) Modular vehicle concept “U-Shift”. Reprinted with permission from Ref. [19]. 2022, DLR; (b) Shuttle “EasyMile” used in real-world transportation laboratory “Reallabor Hamburg”.

1.3. Rationale and Objectives

Since the remote operation of AVs has not been widely rolled out so far, there is limited knowledge about concrete use cases and scenarios that are most relevant to it. However, being aware of events that may occur during remote operation is pivotal for several reasons: (1) It enables an ecologically valid determination of ODD thresholds for a vehicle’s automation, (2) helps bridge those thresholds, and (3) feeds into the derivation of requirements for the task and workplace design regarding the remote operation (both RD and RA) by a Technical Supervisor.

Therefore, a method of approximation via adjacent roles and workplaces needs to be taken. This includes the study of today’s already existing control centers for public transport as they execute tasks of monitoring and resolving disturbances that are comparable to those of remote operators. Further, gaining insights into workplaces of operators on board of already in real-world laboratories operating highly automated shuttles may be helpful as well. In this vein, control center staff has been interviewed about their expectations on remote-operators’ tasks [20] and has been confronted with a first prototype for remote operation [21]; a study on on-board operators’ tasks and human-machine interfaces (HMIs) is currently carried out [22].

In spite of the lack of research opportunities regarding actual remote operators, there is an urgent need for an initial compilation of use cases and scenarios in remotely operating AVs. This is an important element of the user-centered design process as user requirements are derived from them. As presented in Figure 2, initially in this process the authors of this paper followed, and observations and expert interviews in a context that is similar to the future context of use are conducted. From their results, both potential tasks and potential scenarios are derived. This paper focuses on deriving potential scenarios. Next, both tasks that the users will have to execute and scenarios they will be exposed to are used to compile user requirements. These, in turn, need to be addressed while designing the prototype of the remote-control workplace. Whether they were met or not is subsequently evaluated in user studies.

Figure 2. Empirical application of the user-centered design process by the authors. Observations and expert interviews serve as a basis both for deriving potential tasks and, which is the focus of this paper, on deriving potential scenarios. Both help derive user requirements that in turn inform the design of a workplace prototype, which will be validated by conducting user tests and evaluations. These results feed back to investigations of the context of use that has been initially investigated.

This procedure adapts the user-centered design process depicted in Figure 3 as specified by ISO (Section 7) [23]. First, the context of use needs to be understood and specified (box “Understanding and specifying the context of use”). There are different approaches to do so: One way is describing the context of use, of which tasks of the users are an essential characteristic. This approach is being pursued by the authors across several real-world laboratories for future mobility in Germany where they investigate tasks and human-machine interfaces of highly automated shuttle buses that are supervised by a human operator on board the vehicle [22]. Since on-board operators execute tasks similar to remote operators, this approach serves as an approximation to the tasks of remote operators that barely exist in urban road traffic to this date.

Figure 3. The user-centered design process is adapted from ISO [23]. Scenarios are a central element of understanding and specifying the context of use and determine user requirements that in turn help produce design solutions. Adapted from Ref. [22], 2019, ISO.

Another way is the specification of “as-is scenarios”. This is the core objective of this paper. It contains an extensive list of so-called “Is Scenarios”, as defined below. Thus, compiling scenarios and defining tasks are parallel steps that are both based on the empirical data, from sources such as interviews and observations. In a subsequent step, user requirements can be derived from both scenarios and tasks (“Specifying user requirements”). Eventually, these will be used to generate design solutions (“Producing design solutions”). This approach of basing the design on requirements that were factually articulated by potential users helps designing interactions in a user-friendly way that may increase safety, efficacy, ease of use, and prevent task overload, fatigue, and a lack of situational awareness that may increase the risk for accidents. The interactions, in turn, will facilitate the specification of HMIs and, eventually, help design workplaces for remote operation both in research and industrial application.

In addition to considerations of HMI design, compiling use cases and scenarios is also essential for creating a framework that can be used as a basis for interdisciplinary dialogue between engineers, computer scientists, mobility researchers, human factors specialists, and decision-makers on how to conceptualize and further develop remote operation. Furthermore, it may also be used in driving automation and transportation research (see Section 4).

This paper proposes an initial catalogue of use cases and scenarios in which remote operation, operationalized either as Remote Driving or Remote Assistance at SAE Level 4, supports vehicle automation. It is highlighted that this catalogue is a living conceptual document. As it contains statements from a limited number of sources, it does not claim to be exhaustive.

2. Materials and Methods

The following section will outline the process of user-centered design in which scenarios for remote control are a central element. Second, the process of collecting scenarios will be described before a system for systematically structuring the scenarios will be proposed.

2.1. Process of Collecting Scenarios

The scenarios that have been collected both from control centers and the operation of highly automated vehicles (see Figure 4).

Figure 4. Contexts from which scenarios were extracted and methods used in the process.

2.1.1. Control Centers

To date, many real-world labs have tested automated shuttles with on-board operators in order to expand public transport services. AVs in real-world labs are usually operated with the assistance of a steward on board the shuttle who, among other things, monitors the traffic situation and the technical vehicle status and, if necessary, intervenes and initiates appropriate measures depending on the situation.

Remotely operated shuttles without onboard operators, however, will differ fundamentally regarding their interactions. Instead of interacting with the AV’s on-board operators, the control center will interact directly with the AV while the tasks of the onboard operator will be shifted to the control center. However, only crude concepts and isolated prototypes for control centers to monitor, supervise, and, if necessary, remote-control AVs without on-board operators exist at the moment. Thus, in a first step, the roles and activities of today’s control centers in public transport were analyzed by means of observation and interviewing. Participatory observation and expert interviews with control center staff in Hamburg and Braunschweig, Germany, helped examine the working equipment, tasks, roles, and collaborations in a control center for teleoperation in public transport in general. More importantly, these methods yielded scenarios with potential relevance for remote operation.

First, observation was used as a tool to collect essential data. It includes the description of behavioral and temporal patterns, the consequences for control center staff and their environment, as well as the spatial relationship of the control center employee with other people. Observations were characterized by the following attributes:

• Time sampling: Control center staff’s behaviors were observed during a fixed time interval by researchers.
• Unstructured observation: Observations were conducted in a holistic way. The researcher entered the field with some general ideas of what might be salient, but not of what specifically will be observed, i.e., without using any pre-determined objectives, schedules, or variables (cf. [24]).
• Naturalistic setting: The process involved observing and studying the spontaneous behavior of the control center employees in their natural environment.

Second, based on these observations, several expert workshops yielded a set of categories which were subsequently used for two card-sorting studies. In a first study, interdisciplinary traffic researchers clustered the categories, assigned concrete task sets to them, and finalized them. In a second study, expert interviews were conducted using the card-sorting approach [20] to identify tasks and roles in future control centers.

2.1.2. Highly Automated Vehicles

Structured in-depth interviews were carried out with three onboard operators of automated shuttles (SAE Level Four [13]) integrated into Hamburg’s public transport system as part of the HEAT project [11]. These interviews focused on control center tasks, disruptions, work experience, current and future workplace design [25]. From the disruptions mentioned, scenarios were derived.

Videoclips from the EU CityMobil2 project [1,3] were analyzed focusing on the interaction of AVs with other road users to generate scenarios. The main objective of CityMobil2 was to implement different demonstrations of AVs in five European cities as a part of local public transport [12]. Before the analysis, literature research and workshops with four traffic experts led to a set of categories that were applied to analyze the videos of AVs on the road. The categories were chosen to represent the events of interest as exhaustively as possible. The categories were mutually exclusive, precisely defined and their wording was simplistic. They included interactions with vehicles, pedestrians, cyclists, and infrastructure. In accordance with this categorization scheme, naturalistic video clips from the AV demonstrations were evaluated. They showed highly automated shuttles from the cities of La Rochelle, France, and Trikala, Greece, and were recorded as part of the EU project CityMobil2. The videoclips were shot independently from the raters who categorized them and therefore not influenced by them. In order to categorize them, the videos were analyzed regarding incidents, the main events were noted down and grouped regarding the interactions of the AV with other road users, including vulnerable road users (VRUs), and the infrastructure, e.g., traffic lights. These interaction categorizations served as a basis for generating scenarios.

2.2. System of Structuring Scenarios

In order to highlight similarities among scenarios, a hierarchical structure with three levels is proposed. It consists, from top to bottom, of use case clusters (UCCs), use cases (UCs), and scenarios.

The terminology for these terms is based on Ulbrich et al. [26] and Wilbrink et al. [27]. It is illustrated in Figure 5 and will be defined in the following paragraphs.

Figure 5. Relations between use case cluster (UCC), use case (UC), scenario, and scene. Adapted with permission from Ref. [27]. 2018, interACT.

A scene describes a snapshot of the environment. It includes a scenery (e.g., lane networks, stationary elements, and environmental conditions), dynamic elements (e.g., dynamic objects’ states and attributes), self-representations of actors, and observers (e.g., actors’ and observers’ states and attributes, skills, and abilities) as well as the relationships between those entities. A scenario is defined as a temporal development of different scenes within a sequence of scenes. In order to characterize this temporal development, events and actions, as well as objectives, might be specified. Unlike a scene, a scenario describes a period of time. Scenarios start with an initial scene and can be visualized using interaction diagrams (cf. Figure 6). A use case is a functional description of a technical system and its behavior for a specific use. Use cases can comprise numerous different scenarios, but a scenario can only contain a certain number of scenes arranged in a certain order. A use case cluster comprises similar use cases.

Figure 6. Interaction diagram for the scenario with the cause “Vehicles parked in second row”. The lane is blocked, e.g., due to vehicles parked in second row. This causes delays or disturbs the onward journey of the shuttle. The shuttle waits due to the blocked lane (1a). The shuttle informs the remote operator that no further travel is possible due to an obstacle (1b). The remote operator is in bidirectional contact with the passengers and switches to the shuttle’s cameras to get an overall view (2). The remote operator finds out whether it is possible to bypass the obstacle. The remote operator sets a new trajectory, e.g., by setting waypoints, selecting a trajectory, or steers the AV manually around the obstacle, gives clearance, and permits the AV to continue (3). If this is the case, the shuttle can continue its journey. If a bypass cannot take place due to e.g., constructional conditions, then the remote operator contacts the police or another authority (4a). The passengers are proactively informed about the further procedure and possible delays (4b). The police or another authority drive to the shuttle and solve the blockade (5).

For research on human-machine interaction, the focus on singular static scenes does not suffice to describe processes of interaction. On the other side, the system-based level applied in use cases is too abstract to pay enough attention to these processes. Therefore, the focus of this paper will be on scenarios. Particularly, it will list and categorize various scenarios in a uniform structure to lay the groundwork for a scaffolding of scenarios pertaining to remotely operating AVs.

The scenarios presented in this paper were compiled based on interviews and observations in control centers of public transport (see previous section). They are inspired by Geis and Tesch’s notion of Is Scenarios [28]. These are events or chains of events occurring in a naturalistic setting. They are characterized by a “narrative, textual description of actions that a certain user applies in order to attend to one or several tasks (translated by the authors)” [28] (p. 71). They describe components of the context of use in interplay with the perspective of the interviewed or observed person. According to Dzida and Freitag [29], Is Scenarios are the central source for identifying demands and deriving user requirements. Even though tasks may be included in an Is Scenario, they are not in the focus—unlike in Use Scenarios, which describe the implementation of tasks by the user. Rather, as Is Scenarios investigate the interrelations between tasks, the relevance of specific resources is elucidated. This, in turn, may facilitate the identification of previously concealed demands. Furthermore, Is Scenarios may help surface the intertwining of various actors.

As processes of interaction are vital to understand what is happening in urban mixed traffic settings including remote-controlled HAVs, they are used here to provide a scaffolding on the uppermost level, i.e., the level of use case clusters. On this top level, actors play a significant role. This paper defines “actor” in accordance with the United Modeling Language. Thus, an actor “specifies a role played by a user or any other system that interacts with the subject” [27] (p. 586). It emphasizes that actors are not limited to human users such as remote operators: “Actors may represent roles played by human users, external hardware, or other subjects” [27] (p. 586).

Following this definition, the following actors are used in the compilation of scenarios:

• Remote Operator, who may be both Remote Driver of Remote Assistant in SAE’s [8] terminology,
• Highly Automated Vehicles at SAE Level 4,
• Passengers,
• Infrastructure, e.g., crosswalks, traffic lights, or intelligent road-side infrastructures (including road-site units, i.e., sensors along the road that scan the traffic in their immediate surroundings and submit this information to a traffic management center, cf. [28]), and
• Other Actors, e.g., emergency services, control center staff, and other road users.

All of these five actors may be interacting with any other actor in a given scenario within a specific context that influences them. Figure 7 includes the most relevant actors and their interrelations.

Figure 7. Most relevant actors interacting in the compiled scenarios on remote operation of highly automated vehicles. The remote operator, the highly automated vehicle, the passengers, and the infrastructure collaborate with each other to complete the driving task at SAE Level 4. Adapted with permission from Ref. [30]. 2022, DLR (CC BY-NC-ND 3.0).

On the second-to-top level, scenarios are grouped into use cases. Here, a use case is defined as a functional description of a technical system and its behavior for a specific use. Use cases can comprise numerous different scenarios, but a scenario can only contain a certain number of scenes arranged in a certain order [14].

On the third-to-top level, scenarios are listed. In order to facilitate comparability, every scenario is structured in a chain of cause, event, and consequence, as Figure 7 represents. The sequence consists of the following elements: a cause that the event is attributed to, the central event, and the consequences that arise from it.

A generic template is proposed that is used for every scenario:

Due to <Cause>, <Event> takes place. This results in <Consequence>.

The event and its consequence, in turn, lead to certain measures that are required to resolve the event. These required measures, however, are not part of the presented catalogue of scenarios. Adding them would be beyond the scope of this paper since its focus is on events in remote operation, their causes, and consequences. The required measures will be addressed in future publications.

It is important to note that the scenario does not take place in ignorance of the concrete context in which it happens. Rather, all its stages are embedded in this context (see Figure 8). Additionally, the actors that interact throughout the scenario represent another level of analysis that accompanies the chain of cause, event, and consequence, and might also be involved in the measures required for resolving the event.

Figure 8. The chain of cause, event, and consequence provides the scaffolding for each scenario. In future iterations of the scenario catalogue, required measures may be added.

3. Results

The following section outlines the structure of the scenario catalogue, provides the entire compilation of scenarios, and concludes with an exemplary scenario and its related interaction diagram.

3.1. Structure and Catalogue of Scenarios

Figure 9 presents an organigram of the structure of the compiled scenarios of scenarios, organized in use case clusters and use cases. The central interactions can be considered the main body of the scenario collection. They are structured in a way that enables an interaction of one actor with any of the remaining ones. In addition, use case clusters (UCCs) regarding the remote operator’s state, contextual factors, and technical malfunctions related to peripheral factors that are not directly based on interactions.

Figure 9. Structure of use case clusters (UCCs, top row) and use cases (UCs, rows below top row). The core of the scenario collection is made up of interactions between different actors (central interactions). In addition, UCCs regarding the remote operator’s state, contextual factors, and technical malfunctions related to peripheral factors that are not directly based on interactions. RO = Remote Operator, AV = Highly Automated Vehicle, P = Passengers, I = Infrastructure, OA = Other Actors.

Table 1 is a comprehensive list of scenarios in remote operation compiled. It is structured as follows: the overarching classification categories use case cluster and use case, the defining elements of the scenario consisting of cause, event, and consequence, a column for every of the five actor categories, and the mode of remote operation.

Table 1. Comprehensive list of scenarios in remote operation compiled. A classificatory number indicates the use case cluster (UCC), use case (UC) and Is Scenario (Sc) in the following way: <N_UCC> . <N_UC> . <N_Sc>. × indicates that an option applies for a given scenario.

Use Case Cluster	Use Case	Cause	Event	Consequence	Description 2 (Is Scenario)	Actors					Mode of ROn 1
						RO 1	AV 1	P 1	I 1	OA 1	RD 1	RA 1
1 Interaction RO with Passengers	1.1 Conflicts	Vandalism	AV 1 is damaged (inside and/or outside)	AV cannot continue its ride.	1.1.1 Due to vandalism, the AV is damaged (inside and/or outside). This results in the AV being unable to continue its ride.	X	X			X	X	X
		NA 1	Passenger abuses intercom	Control center employee is unnecessarily distracted	1.1.2 Due to an undefined cause, a passenger abuses the intercom. This leads to unnecessary distraction of the control center staff.	X		X			X	X
		NA	Passenger does not want to get off	Delay	1.1.3 Due to an undefined cause, a passenger does not want to get off the AV. This leads to a delay of the AV.	X		X			X	X
		NA	Disputes between passengers	AV cannot continue its journey	1.1.4 Due to an undefined cause, there are disputes between the passengers. This leads to the AV not being able to continue its journey.	X	X	X			X	X
	1.2 Incidents requiring intervention by RO	Mobility-impaired passenger would like to use AV	Mobility-impaired passenger receives support from the control center for using the AV	Mobility-impaired passenger uses AV	1.2.1 Due to a request of a mobility-impaired passenger, they receive support to use the AV. This leads to the mobility-impaired passenger being able to use the AV.	X	X	X			X	X
		Emergency	Control center contacts passengers	Passengers is able to appropriately react to emergency	1.2.2 Due to an emergency, the control center contacts the passengers. This leads to the passengers becoming enabled to appropriately react to the emergency.	X		X			X	X
		Emergency	RO opens the door between regular stops	Passengers can leave the AV between regular stops	1.2.3 Due to an emergency, the doors of the AV are opened by the RO between regular stops. This leads to the fact that passengers can leave the AV between regular stops.	X	X	X			X	X
		Medical emergency	Passenger needs immediate medical treatment	Onward journey might be delayed	1.2.4 Due to a medical emergency in the AV, a passenger needs immediate medical treatment. This might delay the onward journey.	X	X	X			X	X
		NA	Passenger/RO finds ownerless suitcase	Possible security risk	1.2.5 Due to an undefined cause, a passenger or the RO finds an ownerless suitcase. This may lead to a security risk.	X		X			X	X
		Technical malfunction	AV stops	Onward journey of the AV delayed, restricted or not possible	1.2.6 Due to a technical malfunction, the AV stops. This leads to the fact that the onward journey of the AV is delayed, restricted or not possible.	X	X				X	X
		NA	Passenger would like to exit between regular stops	Onward journey delayed	1.2.7 Due to an undefined cause, a passenger wants to exit the AV between regular stops. This leads to a delay in the onward journey of the AV.	X		X			X	X
		Bulky objects inside the AV	Doors blocked	Doors do not close, AV cannot continue its journey, onward journey delayed	1.2.8 Due to bulky objects inside the AV, the doors are blocked. This leads to the doors not closing and the AV not being able to continue its journey. The onward journey of the AV is delayed.	X	X	X			X	X
		AV crowded	Passengers block the door	Doors do not close, AV cannot continue its journey, or onward journey is delayed	1.2.9 Due to the AV being crowded, passengers block the door. This leads to the doors not closing and the AV not being able to continue its journey or the onward journey being delayed.		X	X			X	X
		Spilled liquids, scattered food	AV’s interior dirty	Passenger satisfaction declines	1.2.10 Due to spilled liquids or scattered food in the interior of the AV, the AV’s interior is dirty. This leads to declining passenger satisfaction.		X	X			X	X
	1.3 Incidents not requiring intervention by RO	Emergency	Passenger applies emergency brake	AV stops, door opens, passengers can get off	1.3.1 Due to an emergency, a passenger applies the emergency brake. This leads to the doors opening and the passengers being able to get off.		X	X			X	X
2 Interaction RO with AV	2.1 Emanating from AV	Sensors detect obstacle	AV stops	AV cannot continue its journey/onward journey is delayed, AV waits for clearance from RO until it can resume ride	2.1.1 Due to obstacles detected by sensors of the AV, the AV stops. This leads to the fact that the AV cannot continue its journey or the journey being delayed. The AV waits for the RO to give clearance so it can resume its ride.	X	X					X
		Sensors detect obstacle (e.g., other road users, parked vehicles, trees, foliage, animals), technical malfunction	AV stops	AV cannot continue its journey/onward journey is delayed, AV waits for the RO to give clearance, select a trajectory or set waypoints so that AV can resume ride	2.1.2 Due to obstacles detected by sensors of the AV, the AV stops. This leads to the fact that continuing the journey is not possible or delayed. The AV waits for the RO to give clearance, select a trajectory or set waypoints so that it can resume its ride.	X	X				X	X
		Technical malfunction	RO receives an error/malfunction message	Resolving the error/malfunction	2.1.3 Due to a technical malfunction, the RO receives an error/malfunction message. This leads to the RO resolving the error/malfunction.	X	X				X	X
		AV is partially defective but still operable	AV stops	AV cannot resume ride/is delayed. AV waits for the RO to give clearance, select a trajectory or set waypoints so that AV can resume ride. Resuming ride only possible for AV until a safe stopping point with restrictions	2.1.4 Due to the AV being partially defective but still operable, the AV stops. This leads to the fact that resuming the ride is not possible for the time being and is delayed. The AV waits for the RO to give clearance, select a trajectory or set waypoints so that the AV can continue its ride to a safe stopping point.	X	X					X
		Objects or parts of the AV on fire	Fire alarm activated	AV stops, cannot continue its journey	2.1.5 Due to objects or parts of the AV being on fire, the fire alarm is activated. This results in the AV stopping and not being able to continue.		X	X			X	X
		Smoke emitted by passengers from cigarette, e-shisha, etc.	Fire alarm activated	AV cannot continue its journey or onward journey is delayed	2.1.6 Due to passengers emitting smoke from cigarettes, e-shishas, etc. in the AV, the fire alarm is activated. This results in the AV stopping and not being able to continue or the onward journey being delayed.		X	X			X	X
		Technical malfunction	RO brakes too hard	Wheels block, critical situation or accident may occur	2.1.7 Due to a technical malfunction of the AV, the RO brakes too hard. This makes the wheels block, a critical situation or accident may occur.	X	X				X
		Unforeseen situation (traffic jam, route change, dispatching error, etc.)	AV battery’s state of charge low	AV cannot reach the planned charging station or AV cannot complete the roundtrip as planned	2.1.8 Due to an unforeseen situation (traffic jam, route change, dispatching error, etc.), the battery has a low state of charge. This leads to AV not being able to reach the planned charging station or to complete the roundtrip as planned.	X	X				X	X
		AV conducted roundtrip with high energy consumption, e.g., air conditioning, traffic jam	State of charge not sufficient for another roundtrip	AV needs to be charged earlier than planned	2.1.9 Due to a roundtrip with high energy consumption, the state of charge is not sufficient for another roundtrip. This leads to the fact that AV needs to be charged earlier than planned.	X	X				X	X
		Technical malfunction	Charging not possible	AV cannot continue its journey	2.1.10 Due to a technical malfunction, charging the AV’s battery is not possible. This leads to the AV not being able to continue its journey.	X	X		X		X	X
		Demand for charging stations exceeds supply	No free charging stations	Charging not possible	2.1.11 Due to the demand for charging stations exceeding the supply, the AV cannot find a free charging station. This leads to the fact that the AV cannot be charged.	X	X		X		X	X
		Allocated charging station is out of order	Charging not possible	AV cannot continue its journey	2.1.12 Due to the allocated charging station being out of order, charging is not possible. This leads to the fact that the AV cannot continue its journey.	X	X		X	X	X	X
	2.2 Emanating from RO	RO’s misbehavior	RO is inattentive or distracted, does not recognize street signs	AV overrides RO’s input because it is no longer compliant with traffic rules	2.2.1 Due to misbehavior of the RO during direct control, the inattentive or distracted RO does not recognize the street signs. This leads to the AV overriding the RO’s input because it is no longer compliant with the traffic rules.	X	X				X
		Accident on driving route	Closure of parts of the route	AV cannot continue route as planned (with or without stopping), planned route is adjusted	2.2.2 Due to an accident on the AV’s route, there is a closure of parts of the AV’s planned route. This means that the AV cannot continue its route as planned (with or without stopping). The route must be adjusted.	X	X					X
		Major event, protest, construction site, etc.	Change of route	Changed order of stops; stops may be omitted, RO contacts passengers	2.2.3 Due to a major event, protest, construction site, etc., there is a change in the route. This leads to a change in the order of the stops and, if necessary, stops might be skipped. The RO contacts the passengers and lets them know about the changes.	X	X	X			X	X
		RO’s misbehavior	RO brakes too hard	Wheels block, accident	2.2.4 Due to the misbehavior of the RO, the AV brakes too hard. This leads to the wheels blocking and the risk of a critical situation or an accident.	X	X				X
		Distraction RO	RO accidentally goes off-track	Critical situation or accident	2.2.5 Due to a distraction of the RO during remote driving, the AV goes off-track. This leads to a critical situation or an accident.	X	X				X
		Technical malfunction, latency too high	RO accidentally goes off-track	Critical situation or accident	2.2.6 Due to a technical malfunction and a too-high latency in direct control, the AV accidentally goes off-track. This leads to a critical situation or an accident.	X	X				X
		AV is urgently needed	No complete charging possible	Range of the AV is restricted	2.2.7 Due to an urgent need of the AV, it is not possible to fully charge the AV’s battery. This leads to a limited range of the AV.	X	X		X		X	X
	2.3 Source unclear	Soiling of the AV’s interior camera	Camera images from the AV’s interior cannot be recognized by RO	RO may not notice passenger interactions (e.g., disputes)	2.3.1 Due to a soiled camera inside the AV, the RO cannot see the camera images inside the AV. This leads to the RO not noticing passenger interactions (e.g., disputes).	X	X			X	X	X
		Soiling of the sensors	Sensors limited in their functionality	AV cannot continue its journey or onward journey delayed	2.3.2 Due to soiled sensors on the AV, the functionality of the sensors is limited. This leads to the AV not being able to continue its journey or in a delayed onward journey.	X	X		X		X	X
		Connectivity issues between control center and AV	RO loses connection to AV	RO cannot establish contact with AV	2.3.3 Due to connectivity issues between the control center and the AV, the RO loses connection to the AV. This results in RO no longer being able to contact the AV.	X	X		X		X	X
		Connectivity issues of audio stream from AV to control center	Strong noise interference	RO cannot understand passengers	2.3.4 Due to connectivity issues of the audio stream between the control center and the AV, there is strong noise interference, resulting in the RO not being able to understand the passengers.	X	X	X			X	X
		Connectivity issues of video stream from AV to control center	Desired camera image is not transmitted in sufficient quality	AV can only continue journey with restrictions	2.3.5 Due to connectivity issues of the video stream between the control center and the AV, the desired camera image is not transmitted in sufficient quality. This leads to the fact that the AV can only continue its journey to a limited extent.	X	X				X	X
3 Interaction AV with infrastructure	3.1 Road-site units	Unmapped construction site	Road-site units support AV while maneuvering through unmapped construction site	AV can continue its journey without stopping	3.1.1 Due to an unmapped construction site, road-site units support the AV while maneuvering through the unmapped area. This leads to the AV being able to continue its journey without stopping.		X		X
4 Interaction RO with infrastructure	4.1 Road-site units	Road infrastructure is disrupted	Unstable/disturbed connection from the AV to the road-site units	RO is informed and forwards error so that it can be fixed as quickly as possible	4.1.1 Due to a disruption of the road infrastructure, the connection from the AV to the road-site units is unstable or disturbed. This leads to RO being informed about the unstable or disturbed connection. The RO forwards the fault to the relevant actors as quickly as possible so that the faults can be fixed as soon as possible.	X	X		X		X	X
5 Interaction AV with other actors	5.1 Other road users	Vehicle or VRU is occluded by object or outside the focus area	AV reacts abruptly to suddenly appearing vehicle or VRU	Critical situation or accident	5.1.1 Due to the occlusion of a vehicle or VRU 1 by another object or because a vehicle or VRU is outside of the focus area, the AV reacts abruptly to the suddenly appearing vehicle or VRU. This leads to a critical situation or an accident.		X			X
		Several vehicles reach an intersection without a right-of-way sign at the same time and congest it	Deadlock occurs at an intersection without a right-of-way sign (left-yields-to-right rule)	RO gives clearance	5.1.2 Due to vehicles reaching the intersection from all directions at the same time, a deadlock occurs. This leads to the RO giving clearance and permitting the AV to proceed if the other road users do not resolve this situation beforehand.	X	X			X		X
		VRU or animal suddenly or unexpectedly crosses the intersection	AV performs emergency braking maneuver	Critical situation or accident	5.1.3 Due to a suddenly crossing or unexpected VRU or animal, the AV has to perform an emergency braking. This leads to a critical traffic situation or an accident.	X	X			X	X
		VRU’s misbehavior or technical malfunction	AV collides with VRU	AV cannot continue its journey	5.1.4 Due to a misbehavior of the VRU or a technical malfunction, the AV collides with a VRU. This leads to the AV being unable to continue its journey.	X	X			X	X	X
		Vehicles parked in second row	Pathway blocked	Continuation of ride delayed; RO assesses situation and guides AV around the obstacles	5.1.5 Due to vehicles parked in the second row, the AV’s pathway is blocked. This leads to a delay in the onward travel of the AV. The RO gives clearance to the AV to continue, sets a new pathway, e.g., by setting waypoints or selecting a pathway, or steers the AV manually around the obstacles.	X	X			X	X	X
		Accident	AV stops	AV cannot continue its journey	5.1.6 Due to an accident, the AV stops. This leads to the AV not being able to continue its journey.	X	X				X	X
		Driver’s distraction, difficult weather conditions, etc.	Collision of another vehicle with AV	Damage to the AV by another vehicle, AV cannot continue its journey, damage must be fixed and passengers must be informed	5.1.7 Due to the driver’s distraction, difficult weather conditions, etc., another vehicle collides with the AV. This leads to the AV being damaged by the other vehicle and being unable to continue its journey. The RO conducts procedure as with accidents in general (i.e., contacts passengers, informs blue light organization, organizes a substitute vehicle if necessary, documentation, etc.).	X	X	X		X	X	X
		Major event, protest, construction site, etc.	Traffic jam	Onward journey is delayed	5.1.8 Due to a major event, protest, construction site, etc., a traffic jam occurs. This leads to a delay in the onward journey of the AV.	X	X			X	X	X
6 State RO	6.1 Sickness	NA	RO sick	RO cannot fulfill supervisory duty	6.1.1 Due to an undefined cause, the RO is sick. This leads to the RO being unable to fulfill their supervisory duties or to perform their tasks.	X	X				X	X
	6.2 Distraction	Fatigue, stress, distraction in general, etc.	RO distracted	RO cannot fulfill supervisory duty	6.1.2 Due to fatigue, stress, distraction in general, etc., the RO is distracted. This leads to the RO being unable to fulfill their supervisory duties or to perform their tasks.	X	X				X	X
7 Contextual Factors	7.1 Weather conditions	Difficult weather conditions	Overload or soiling of the sensors on the vehicle	AV cannot continue its journey	7.1.1 Due to difficult weather conditions, the sensors on the AV are overloaded or soiled. This leads to the fact that they cannot detect the environment accurately so the AV cannot continue its journey.		X
		Difficult weather conditions, such as icy roads, fog, heavy rain, or snowfall	AV goes off-track, no other parties involved	Critical situation or accident, onward journey is delayed, restricted, or not possible	7.1.2 Due to difficult weather conditions, such as icy roads, fog, heavy rain, or snowfall, the AV goes off-track. There are no other parties involved. This leads to a critical situation or accident and to the fact that the continuation of the AV’s journey is delayed, restricted, or not possible.		X
		Difficult weather conditions, such as icy roads, fog, heavy rain, or snowfall	AV comes off the road, there are other parties involved	Accident with internal and external damage, onward journey is delayed, onward journey is delayed, restricted, or not possible	7.1.3 Due to difficult weather conditions, such as icy roads, fog, heavy rain, or snowfall, the AV goes off-track. There are other people involved. This leads to an accident with internal and external damage and the fact that the continuation of the AV’s journey is delayed, restricted, or not possible.		X			X
		Flood, fire, etc.	Traffic routing changed, road closed, speed limit changed	Planned route changed, planned stops skipped	7.1.4 Due to a flood, fire, etc., for example, the traffic routing was changed, a road was closed, or the speed limit was changed. This leads to the planned route being changed and planned stops being skipped.	X	X					X
8 Technical malfunction	8.1 Internal cause	Technical malfunction	Doors do not close	AV cannot continue its journey	8.1.1 Due to a technical malfunction, the doors of the AV do not close. This leads to the AV not being able to continue its journey.		X	X
		Technical malfunction	Doors do not open	Passengers can neither get on nor off	8.1.2 Due to a technical malfunction, the doors of the AV do not open. This leads to passengers not being able to get on or off and the AV not being able to continue its journey.		X	X
		Leakage in tank or pipes	AV loses liquid	AV cannot continue its journey; may obstruct other road users	8.1.3 Due to a leakage in the tank or pipes, the AV loses liquid. This leads to the fact that it is not possible for the AV to continue its journey and other road users being obstructed.		X			X
		GPS receiver malfunction or connectivity issues	AV cannot be localized	Monitoring and controlling of the AV by RO not possible	8.1.4 Due to a defective GPS receiver or connectivity issues, the AV cannot be localized. This leads to the fact that monitoring and controlling the AV by the RO is not possible.	X	X				X	X
		Technical malfunction	AV’s fire detector disturbed	Passengers’ safety cannot be guaranteed	8.1.5 Due to a malfunction, the AV’s fire detector is disturbed. This leads to the fact that the safety of the passengers cannot be guaranteed.	X	X				X	X
		Technical malfunction	Air conditioning out of order	Overheating of the passenger cabin; reduction of comfort or impairments of passengers’ health	8.1.6 Due to a technical malfunction, the air conditioning in the AV is out of order. This leads to an overheating of the passenger cabin and to a reduction in comfort or impairments of passengers’ health.		X	X
		Technical malfunction	Signal and/or brake light fails	RO receives an error message	8.1.7 Due to a technical malfunction, a signal and/or brake light of the AV fails. This leads to the RO receiving an error message.	X	X				X	X
		Technical malfunction	Headlight cleaning system defective	Limited visibility of the AV	8.1.8 Due to a technical malfunction, the headlight cleaning system of the AV is defect. This leads to limited visibility of the AV.		X
		Technical malfunction	Problems with thermal management	Danger of overheating of the vehicle’s battery or its interior	8.1.9 Due to a technical malfunction, the AV has problems with the thermal management. This leads to an overheating of the vehicle’s battery or its interior.		X	X
		Failure of parts of the sensors	MRM 1	AV cannot continue its journey and may endanger passengers and other road users due to AV’s location on the road	8.1.10 Due to a partial failure of the AV’s sensor system, the AV performs an MRM. This leads to the AV being unable to continue its journey and possibly endangering passengers and other road users due to AV’s location on the road.		X	X
		Interface between AV and control center disturbed	Target and actual direction of travel do not align	AV drives on the wrong pathway	8.1.11 Due to a malfunction in the interface between the AV and the control center, the AV’s planned and actual driving directions do not align. This leads to the AV driving on the wrong pathway.	X	X					X
		Interface between AV and control center disturbed	Target and actual speed do not align	AV drives at the wrong speed	8.1.12 Due to a malfunction in the interface between the AV and the control center, the desired and actual speed of the AV do not align. This leads to the AV driving at the wrong speed. The AV decides to adapt the speed according to the traffic regulations.	X	X					X
	8.2 External cause	Spiky objects on the roadway	Damaged tire	AV can continue to drive if necessary but with increased risk of accident; onward journey is delayed	8.2.1 Due to spiky objects on the road, a tire of the AV is damaged. This leads to the AV being able to continue driving if necessary but with increased risk of accident. The onward journey is delayed.		X
		Collision with an obstacle: vehicle hits curb or boundary of construction site	Damage to the AV, possibly further damage	AV cannot continue its journey	8.2.2 Due to a collision with an obstacle, such as a curb or a boundary of a construction site, the AV is damaged. Further damage may occur. This leads to the AV not being able to continue its journey.		X
	8.3 Unknown cause	NA	AV stops	AV cannot continue its journey; onward journey is delayed	8.3.1 Due to an undefined cause, the AV stops. This leads to the fact that the AV cannot continue its journey or the onward journey is delayed.		X
		Technical malfunction	AV cannot be controlled remotely	AV may endanger passengers and other road users	8.3.2 Due to a technical malfunction, the AV cannot be controlled remotely. This leads to the AV not being able to continue driving or the onward journey being delayed. Passengers and other road users may be endangered.	X	X				X	X
		Connectivity issues	RO cannot connect to AV	AV no longer complies with all safety regulations, may endanger passengers and other road users	8.3.3 Due to connectivity issues, the RO cannot connect to the AV. This leads to the fact that the AV no longer complies with all safety regulations. Passengers and other road users may be endangered.	X	X				X	X
		Technical malfunction or transmission error	RO fails to identify the road signs	Onward journey is delayed	8.3.4 Due to a technical malfunction or transmission error, the RO does not recognize the road signs. This leads to the onward journey being delayed.	X	X				X
		Poor object identification	Unclear detection situation	AV cannot continue its journey, AV waits for the RO to give clearance so it can continue journey	8.3.5 Due to poor object identification, the detection situation of the AV is unclear. This leads to the AV not being able to continue its journey. The AV waits for the RO to give clearance so it can continue its journey.	X	X

¹ NA = Not specified, AV = Highly Automated Vehicle (SAE Level 4), I = Infrastructure, MRM = Minimal Risk Maneuver, OA = Other Actors, P = Passengers, RA = Remote Assistance, RD = Remote Driving, RO = Remote Operator, ROn = Remote Operation, VRU = Vulnerable Road User (e.g., pedestrian, cyclist). ² According to standardized template.

Overall, 74 scenarios were compiled. These were subsumed under 15 use cases, which in turn were grouped into eight use case clusters, five of which are central interactions, and the remaining three are regarded as peripheral factors. Both use cases and use case clusters are stated in Figure 8. A sequential number (N) was given to each use case cluster (UCC), use case (UC), and Is Scenario (Sc), in the following fashion:

<N_UCC> . <N_UC> . <N_Sc>

From left to right, Table 1 includes the following columns: Use case cluster, Use case, Cause, Event, Consequence, and the descriptive Is Scenario. In the section “Actors”, each actor involved is checked with an “X” if involved. If not involved, the respective column remains blank. The same system is used to indicate the Mode of Remote Operation. Here, an “X” indicated that the scenario is valid for Remote Driving, Remote Operation, or both.

3.2. Example for Scenario “Vehicles Parked in Second Row”

An example for a scenario is from the Use Case Cluster “Interaction AV with Other Actors”, Use Case “Other Road Users”. The Causes in this scenario are “Vehicles parked in second row”. This leads to the Event “Driveway blocked” which in turn triggers the following Consequence: “Continuation of ride delayed; RO needs to assess situation and intervene”. The following Actors are involved in this scenario: Remote Operator, Highly Automated Vehicle, and Other Actors. The scenario may occur both in Remote Driving and Remote Assistance. Finally, the full description of an Is Scenario reads as follows:

“Due to vehicles parked in the second row, the AV’s lane is blocked. This leads to a delay in the further travel of the AV. The RO allows the AV to continue, sets a new trajectory, e.g., by setting waypoints or selecting a pathway, or steers the AV manually around the obstacle.”

Figure 6 above shows an interaction diagram for this scenario. It displays the main actors in this scenario. Every scenario in Table 1 can be translated into an interaction diagram like this.

4. Discussion

This paper proposes an initial draft for a catalogue of scenarios that might help when creating design solutions for the remote operation of AVs. It is published preliminarily with a remaining need for evaluation, validation, and modification in future iterations. Even though the scenarios presented here originate from diverse sources, particularly real-world labs, idiosyncrasies of other contexts of use will need to be considered by revalidating and extending the catalogue.

In spite of these constraints, the catalogue fulfills several purposes. First, it serves as a starting point for designing novel interfaces for remote-controlling highly automated vehicles—and has, in fact, already done so. Based on these scenarios, the authors of this paper designed an HMI for a workplace for remote operation in an online study with experts employed by control centers in public transport [21]. Incorporating the results from the evaluation study, a workplace for remotely assisting AVs has been set up at DLR’s Braunschweig premises.

Second, the catalogue is suitable to test and validate HMIs in teleoperations of means of public transport. For instance, the workplace described above will be tested and validated using the catalogue of scenarios presented here. Thanks to the workplace’s integration in realistic road simulations and DLR’s fleet of highly automated vehicles [31], a validation study with high ecological validity will be carried out. Using both quantitative, performance-based indicators and qualitative interview and questionnaire methodology, a group of experts from public transportation facilities and associations and other potential users of a remote operation workplace was exposed to a selection of the scenarios presented here [32]. Hence, a bidirectional relationship is established between the scenarios and the workplace: Not only will the study guide the process of improving the workplace for the needs of the remote operator to execute their tasks safely, effectively, efficiently, while ensuring an optimal task load, keeping the operator in the loop and preventing fatigue and monotony. It will also help reassess and refine the compiled scenarios.

Third, Operational Design Domains (ODDs) may be derived from the scenarios. Thus, the catalogue will help specify the context and boundaries of safe teleoperation and create if-then contingencies between a certain contextual factor and its adequate driving mode. Hence, the remote operation can be incorporated in a wider framework of highly automated driving that encompasses different modes of operation, e.g., relying on input from the driving automation [33] and the infrastructure [34,35] in addition to remote operation. This multimodal, holistic approach is conceptualized within the framework of Managed Automated Driving [36].

Fourth, the catalogue will help to identify the most safety-relevant scenarios for teleoperation in public transport. This will be done, for example, by conducting a study that makes experts and operators in public transportation review the scenarios and have them rate (1) the probability of a scenario’s occurrence and (2) its criticality for safe remote operation. A resulting priority classification will help understand the most pressing safety-relevant scenarios, among others, and provide a pathway to address them effectively. Subsequently, they will be used to derive user requirements to further improve the existing HMI of DLR’s prototypical remote operation workstation. In addition, they will also help create adaptive HMIs solutions that consider the remote operator’s current state and adapt the interface to accommodate it. This approach is pursued within the European Union’s Horizon 2020 project “Hi-Drive”, among others, and may be suitable to defragment the transition between various operational contexts [37].

Fifth, the identified user requirements may also facilitate creating a checklist for assessing the quality of a remote operator’s workstation from a Human Factors perspective. Critically, guidelines for selecting and training remote operators could be interpolated from these requirements. This is highly relevant to rolling out highly automated vehicles since the role of having a Technical Supervisor to remotely monitor AVs and intervene, if necessary, i.e., a remote operator, was put forward as a requirement for SAE Level Four driving operations by several legislators on the German and European level [11,38]. Thus, remote operation is likely to be an inevitable prerequisite for highly automated driving.

Finally, the collection of scenarios may also be of importance for mobility research in adjacent disciplines. For instance, it could contribute to IT mobility services that require a holistic user-centered view on future mobility systems, embedded in a network of various means of transport, and feed data into a comprehensive mobility data space that is used to exchange mobility data. This is currently examined by the project “GAIA-X 4 ROMS” that aims at supporting and remote-operating automated and networked mobility services [39].

In spite of its numerous benefits, the catalogue of scenarios comes with certain limitations. Particularly, it must be noted that the catalogue does not claim to be exhaustive in several regards. First, not all the interactions between the actors proposed are addressed in the catalogue. Second, not every use case cluster or use case is considered at the same level of depth and detail which affects the balance of scenarios across use case clusters and use cases. The focus of this catalogue is on the projects and contexts that have been presented initially in this paper. Filling the gaps of the presented framework and coming up with more use case clusters, use cases, and scenarios is a quest for further research and inevitably depends on the context of use, the vehicles investigated, and their level of automation, as well as on the mode of remote operation and its concrete conceptualization. Also, categorizing the scenarios’ consequences, e.g., by severity or impact, is left to future empirical investigations. Appropriate categorization systematics may be part of prospective iterations of this scenario catalogue, which is considered a living document that is permanently updated as research progresses. The same is true for deriving required measures for each scenario. Third, the categories used for analyzing the scenarios might not be mutually exclusive in certain cases. For instance, the listed consequences of some of the scenarios may already contain required measures even though deriving measures will be the subsequent step in the user-centered design process. This is because for comprehending these scenarios, indicating the required measures is inherently necessary. Moreover, only if the scenario is understood correctly, adequate design solutions can be derived from it. Further research is needed to evaluate and modify the categories used in this framework, if necessary, as well as to disentangle remaining unclarities in the categories assigned.

At any rate, the remote operation of vehicles is likely to become a vital element of highly automated driving systems. Analyzing typical scenarios that may occur when remotely operating highly automated vehicles is a first but essential step towards enabling remote operation while designing with human needs at the center of attention. While the presented scenarios focus on the notion of controlling one vehicle at a time, feasible remote-operation solutions may need to be able to supervise several vehicles simultaneously. Also, communication of the remote operator with other road users is a research area that has not been investigated yet to the knowledge of the authors. Relying on external HMI solutions for highly automated vehicles (e.g., [40,41]) may be a feasible approach.

All in all, the presented catalogue of scenarios is an important milestone for bringing highly automated vehicles onto the roads as it is a precursor for testing and validating them under realistic conditions. By being able to tackle the scenarios presented, the remote operation will significantly improve the operation of AVs and therefore bring us a small but vital step closer to fully automated mobility. And even in the case that fully automated driving (SAE Level 5) may be achieved one day, certain scenarios in fully automated public transport, such as passenger emergencies or vehicle malfunctions, may always require human support. Hence, the remote operation may be more than a preliminary technology to bridge the gap to a certain level of automated driving. It may be a long-lasting alternative to human operators on-site without compromising on the unique and sometimes irreplaceable abilities and skills of humans.