Authority Transfer According to a Driver Intervention Intention Considering Coexistence of Communication Delay

Abstract

Recently, interest and research on autonomous driving technology have been actively conducted. However, proving the safety of autonomous vehicles and commercializing autonomous vehicles remain key challenges. According to a report released by the California Department of Motor Vehicles on self-driving, it is still hard to say that self-driving technology is highly reliable. Until fully autonomous driving is realized, authority transfer to humans is necessary to ensure the safety of autonomous driving. Several technologies, such as teleoperation and haptic-based approaches, are being developed based on human-machine interaction systems. This study deals with teleoperation and presents a way to switch control from autonomous vehicles to remote drivers. However, there are many studies on how to do teleoperation, but not many studies deal with communication delays that occur when switching control. Communication delays inevitably occur when switching control, and potential risks and accidents of the magnitude of the delay cannot be ignored. This study examines compensation for communication latency during remote control attempts and checks the acceptable level of latency for enabling remote operations. In addition, supplemented the safety and reliability of autonomous vehicles through research that reduces the size of communication delays when attempting teleoperation. It is expected to prevent human and material damage in the actual accident situation.

1. Introduction

With the advent of the 4th Industrial Revolution, autonomous driving technology is developing [1,2]. Many companies are researching and developing autonomous driving technology, and there is a great interest in autonomous driving as many open sources such as Autoware and Apollo are provided [3,4,5,6]. However, autonomous driving technology still has many problems. In Figure 1, Autonomous driving technology is still unstable, according to a report released in 2022 on the lifting of autonomous driving by the California Department of Motor Vehicles [7].

Due to the instability of autonomous driving technology, we are facing many problems such as safety, laws, and regulations [8,9,10,11]. Also, there are many variables on real roads. There are many things to consider, such as erased lanes, broken traffic lights, damaged road surfaces, and sensor errors due to bad weather. To improve this problem, many studies are being conducted as a means to assist autonomous driving, such as monitoring, haptic, and teleoperation [12,13,14,15,16]. Teleoperation is used in many places such as Mars exploration robots and rescue robots [17,18].

However, teleoperation through these robots is not suitable for roads. Teleoperation on the road has many things to consider, such as traffic rules, sudden obstacles, pedestrians, etc. [19]. It is necessary to comply with the rules of lane and signal and to move the autonomous vehicle to an optimal safe place in the field. Taking into the SAE J3016:2021, and BSIPAS 1880:2 accounts, the teleoperation can be divided into three teleoperation stages which are remote driving, remote assistance, and remote monitoring [20,21].

The concept of teleoperation, for example, is shown as Figure 2 of the vehicle can be largely classified into three ideas which are: direct control, shared control, and trajectory guidance [22]. Firstly, the concept of direct control which represents a human operator handling the sensing data, and the planning of the vehicle trajectory, directly provides control to the vehicle [23,24]. Second, shared control allows remote drivers to gain control of autonomous vehicles through control commands such as steering, brake, and throttle [22,25]. Finally, the trajectory guidance allows the vehicle to drive through the route presented by the remote driver received from the estimated route and trajectory planning [22].

This study proposes a way to solve the problem through teleoperation by taking control of the vehicle in situations where the driver needs to intervene, such as hardware, software, and emergencies in which the autonomous vehicle is driving on the road.

In addition, when attempting teleoperation, communication delays between the autonomous vehicle and the teleoperated inevitably occur [26,27]. Communication delay that occurs when teleoperation is operated is a potential threat. Ignoring delays and attempting remote manipulation will lead to even greater accidents. One of the key functions when attempting to remotely operate a vehicle is to detect the intention of the remote operator to intervene in the operation. In the preceding study, the remote operator manipulates the steering to detect the driver’s intention to intervene and transfer the control of the vehicle to the remote operator [16,28]. In addition, to compensate for the instability of system performance due to communication delays, there is also a study through a predictor model that predicts and compensates for delays [26,27,29,30].

Through this study, we would like to compensate for communication delays caused by taking control of autonomous vehicles and talk about the allowable range of communication delays that allow autonomous driving.

Contribution

The contribution of the study is shown below:

• Control authority transfer is regarded as a crucial aspect of autonomous driving until full-automation technologies are fully commercialized. Despite dedicated efforts to transfer authority from autonomous vehicles to local human drivers, some remain unclear when it comes to scenarios including utilizing the teleoperation of the remote vehicle. Due to this fact, in this study, a control transfer mechanism was proposed to compensate for the communication delay between the remote vehicle and the control center to guarantee control transfer within a specific required time. In addition, not only the proposed method can be applied to the scenarios containing the teleoperation of ground vehicles, but it can also be applied to the field that the application is targeting wireless teleoperation of mobile robots, surgical robots, and so on.
• One another aspect of this study is that it is based on actual measured communication delays. As long as the communication was constructed based on wireless communication, inherently it contains passive and erroneous communication delay latency. Therefore, in this study, we measured an actual communication delay between the remote vehicle and the control center to reference and manage the parameters that supplement the proposed control transfer algorithm. In the future, large-scale deployment scenarios can be expected including valet parking service, minimum risk state takeover when autonomous is in malfunctioning state, and so on.

2. Methodology

There are several considerations for the teleoperation of autonomous vehicles.

First, we need to confirm the location of the decision of the transfer of control. We need to talk about where to give commands to transfer control, such as whether control is handed over from an autonomous vehicle or whether control is taken from a control center. Figure 3a,b as shows process according to decide 88 to hand over control, that the result of the control transfer differs depending on the control 89 location.

Second, it is necessary to consider the communication delay that occurs when attempting teleoperation. It is necessary to check how to compensate for the communication delay that occurs in the process of transferring control, and to set standards for how much delay is allowed [26,30]. Finally, as shown in Figure 4, the remote driver operates the steering during autonomous driving and transfer control, indicating that the autonomous vehicle needs to detect the driver’s intervention intentions. There are many ways to bring control, but this study adopted a method in which the autonomous vehicle detects the driver’s intention to intervene and hands over control by the remote driver manipulating the steering wheel. In previous studies, Sonoda et.al’s research was conducted as follows. the research of Sonoda et.al was conducted on local vehicles [16]. It may not be sufficient in a system for remotely controlling autonomous vehicles. Firstly, the presence of communication delays can result in delayed reaction times and uncertain criteria for remote controllability, depending on the magnitude of the communication delay. Secondly, by far of the author’s knowledge, no method is proposed for transferring control from the autonomous driving vehicle to the remote operator and back to autonomous driving. teleoperation is appropriate when there is no problem with the Automatic Driving System (ADS), but it is uncertain to transfer control to the local vehicle when there is a problem with the ADS. However, assuming the technological advancements and sufficient safety of the ADS, it would be feasible to assume control during situations such as traffic congestion or parking, where the driver needs to take over. Therefore, in this study, in addition to the research of Sonoda et.al, we aim to provide criteria for tolerating certain levels of communication delay during control transfer and discuss potential strategies to address this issue [16].

2.1. Driver Intervention Intention

When the driver intervenes in the ADS operation by using the steering wheel, the system shifts from AD mode to shared authority mode. It is assumed that the ADS controls the vehicle with a proportional-differential (PD) controller for the lateral control of the vehicle [31]:

$${\tau }_{\mathrm{ads}}=-K_{\mathrm{p}}(\theta \left(t\right)-{\theta }_d\left(t\right))-K_d\dot{\theta }\left(t\right),$$

(1)

where

${\tau }_{\mathrm{ads}}:\mathrm{Torque}\mathrm{induced}\mathrm{to}\mathrm{the}\mathrm{steering}\mathrm{column}$

$K_{\mathrm{p}}:\mathrm{Coefficient}\mathrm{of}\mathrm{the}\mathrm{proportional}\mathrm{control}\mathrm{term}$

$K_d:\mathrm{Coefficient}\mathrm{of}\mathrm{the}\mathrm{differential}\mathrm{term}$

$\theta :\mathrm{Instant}\mathrm{angle}\mathrm{of}\mathrm{the}\mathrm{steering}$

${\theta }_d:\mathrm{Desired}\mathrm{angle}\mathrm{of}\mathrm{the}\mathrm{steering}$

$\dot{\theta }:\mathrm{Derivative}\mathrm{of}\mathrm{the}\mathrm{instant}\mathrm{angle}\mathrm{of}\mathrm{the}\mathrm{steering}$

2.2. Control Transfer Mechanism

When the system initiates control transfer from the vehicle to the teleoperator, the proportional gain of the lateral control of the vehicle is changed by the following:

$${\displaystyle \frac{d{K}_{\mathrm{p}}}{dt}}=-G·\mathrm{sgn}\left(K_{\mathrm{p}}\left(t\right)\right){\left|K_{\mathrm{p}}\right|}^p,$$

(2)

where

$G:\mathrm{gain}\mathrm{of}\mathrm{the}\mathrm{changing}\mathrm{speed}\mathrm{of}\mathrm{the}{K}_{\mathrm{p}}\left(t\right)$

$\mathrm{sgn}:\mathrm{denotes}\mathrm{the}\mathrm{sign}\mathrm{of}\mathrm{the}\mathrm{variable}$

$p:\mathrm{parameter}\mathrm{used}\mathrm{to}\mathrm{converge}\mathrm{in}\mathrm{finite}\mathrm{time}$

When communication delays co-exist, according to the magnitude of the communication delay, there is a chance that control transfer may not be finished in time. Therefore, in this study, the authority transferring method considering the communication delay is proposed. $p_0$ is the initial parameter for the p, $\eta$ is the gain parameter to increase the p to transfer the authority fast, and $\tau \left(t\right)$ is the communication delay at time t.

$$p=p_0+\eta (K_{\mathrm{p}}(t-\tau \left(t\right))-K_{\mathrm{p}}\left(t\right)),$$

(3)

where

$p_0:\mathrm{initial}\mathrm{parameter}\mathrm{for}\mathrm{the}p$

$\eta :\mathrm{gain}\mathrm{parameter}\mathrm{to}\mathrm{increase}\mathrm{the}p\mathrm{to}\mathrm{transfer}\mathrm{the}\mathrm{authority}\mathrm{fast}$

As shown in Figure 5, an authority transfer is initiated after an intervention is started. However, when communication delays co-exist, according to the magnitude of the communication delay, there is a chance that control transfer may not be finished in time. From the red dotted line in Figure 5, the $K_{\mathrm{p}}$ value is delayed by delay and is again compensated by Equation (3) with the ideal control transmission value.

3. Experiment

In this section, an experimental setup is shown.

General Setup

To gather the communication delay profiles, a 5G/LTE modem and ethernet network are used (UX-50C, LG, Republic of Korea). Also, GPS is used to compute the communication delay between the control center and the teleoperated vehicle (PA1616S, Adafruit, New York, NY, USA) with an external GNSS antenna (GPS ACTIVE 28 dB, Changhong, China). As for the middle operational system, ROS-noetic is used (Ubuntu, 20.04 LTS). Finally, to control the simulation vehicle, a wheel joystick (G29, Logitech, Swiss) was used. In addition, Matlab (R2019 a, Mathworks) is used to demonstrate the teleoperated vehicle in the framework of the vehicle dynamics block set.

Table 1. Simluation conditions used in the experiment.

Simulation Trial	Delay (ms)	Gain (G)	Gain ($\mathit{\eta }$)	${\mathit{p}}_0$
1	34	22	-	0.5
2	100	22	-	0.5
3	200	22	-	0.5
4	300	22	-	0.5
5	400	22	-	0.5
6	500	22	-	0.5

shows the proportional gain with the magnitude change of the communication delay from 34 ms to 500 ms.

4. Result

4.1. Communication Delay

Figure 6 shows a communication delay profile gathered between the control center and the teleoperated vehicle. The x-axis represents time and the y-axis represents the probability density. The mean value of the communication delay was 34 ms and the maximum value was 325 ms.

4.2. Authority Transfer of Outer and Inner Controller

Two types of methods can be considered to compensate for communication delays during control transmission. First, Figure 7 used an inner transfer controller, and Figure 8 an outer transfer controller. The x-axis represents time and the y-axis represents proportional gain. In addition, the solid line represents the ideal control transfer result and the dotted line represents the delayed control transfer result due to the influence of the communication. outer transfer controllers have irregular results compared to inner transfer controllers.

4.3. Authority Transfer According to the Communication Delay

Figure 9 shows the proportional gain with the change in communication delay size from 34 ms to 500 ms shown in Table 1. This figure shows that the greater the communication delay, the more difficult it is to transfer permissions on time. The solid line represents the ideal control transfer and the dotted line represents the delay control transfer.

4.4. Simulation Results Based on Condition

This section shows the results of the proposed method of intervening in the self-driving vehicle to hand over control. Indicates the amount of communication delay based on the change in the gain value. The solid line in the graph represents the ideal control transfer, and the dotted line represents the delayed control transfer. The x marker corrects the delayed control transmission according to the proposed method.

Figure 10 is the simulation result value reflecting the change amount of G shown in

Table 2. Simulation result value according to the amount of change in the value of G.

Simulation Trial	Delay (ms)	Gain (G)	Gain ($\mathit{\eta }$)	${\mathit{p}}_0$
1	34	5	1000	0.5
2	100	10	1000	0.5
3	200	22	1000	0.5
4	300	30	1000	0.5
5	400	50	1000	0.5
6	500	100	1000	0.5

. I changed the G value from 5 to 100. show the most ideal results when we reflect 22.

Figure 11 shows the simulation results based on the change amount of $\eta$ shown in

Table 3. Simulation result value according to the amount of change in the value of $\eta$.

Simulation Trial	Delay (ms)	Gain (G)	Gain ($\mathit{\eta }$)	${\mathit{p}}_0$
1	34	22	100	0.5
2	100	22	300	0.5
3	200	22	700	0.5
4	300	22	1000	0.5
5	400	22	2000	0.5
6	500	22	3000	0.5

. changed the $\eta$ value from 100 to 3000, and the value of 1000 looks most stable.

Figure 12 is the simulation result value reflecting the change amount of G shown in

Table 4. Simulation result value according to the amount of change in the value of $p_0$.

Simulation Trial	Delay (ms)	Gain (G)	Gain ($\mathit{\eta }$)	${\mathit{p}}_0$
1	34	22	1000	0.1
2	100	22	1000	0.3
3	200	22	1000	0.5
4	300	22	1000	0.7
5	400	22	1000	0.9
6	500	22	1000	1.0

. changed the G value from 0.1 to 1.0. The delay transmission shown in dotted lines is rapidly changing with the change of $p_0$. When the driver intervenes, there is a risk of an accident when the control is suddenly transferred and the vehicle is safely evacuated. show the most ideal results when we reflect 0.5.

4.5. Control Transition Method Using the Proposed Method

Table 5. Conditions for issuing control commands considering communication delays.

Simulation Trial	Delay (ms)	Gain (G)	Gain ($\mathit{\eta }$)	${\mathit{p}}_0$
1	34	22	1000	0.5
2	100	22	1000	0.5
3	200	22	1000	0.5
4	300	22	1000	0.5
5	400	22	1000	0.5
6	500	22	1000	0.5

shows Figure 13 by referring to the simulation results according to the previous conditions and reflecting the most stable results. The solid line represents the ideal control transfer and the dotted line represents the delayed control transfer. The x marker shows the result of the recovered control transfer. The proposed method compensates for communication delays and allows for stable teleoperation through smoother control transfer.

5. Discussions

5.1. Communication Delay Depending on Control Position

In Figure 7 and Figure 8 communication delay according to the location of the control transfer. Transferring the control transition from the inside was more stable than controlling the vehicle exterior switching. However, it is hard to say that giving control commands inside the vehicle is unconditionally good. Depending on the operation method and environment, it seems that the control location should be considered appropriately.

5.2. Criteria for Communication Delays That Allow Teleoperation

Figure 9 shows the check the result value according to the amount of change in communication delay that occurs when the control is transferred from the autonomous vehicle. Communication delay increased sharply at 500 ms. However, it is necessary to establish a clear standard for teleoperation depending on the delay size by acquiring driving data through the subject in subsequent studies.

5.3. Teleoperation Method Considering Communication Delay

Figure 13 delay can be compensated through the proposed method by applying arbitrary delay. As shown in the table, we set the experimental value from 34 ms to 500 ms, and the result is the same as the image. This allows remote drivers to step in to take control of the vehicle, evacuate the vehicle to safety, or replace it in urgent situations.

5.4. How to Protect against Malicious Cyberattacks

Communication security is one of the important parts when attempting teleoperation. A malicious attack from the outside can cause another accident. There are several ways to solve these problems.

Protecting against malicious cyberattacks originating from external sources during teleoperation operations encompasses several key strategies. These strategies include the utilization of firewalls for the meticulous management of network connections and the incorporation of Virtual Private Networks (VPNs) to ensure secure data transmission. Furthermore, authentication and authorization management play a pivotal role in verifying remote users’ identities and administering their permissions, thereby enhancing overall security. The implementation of network monitoring and intrusion detection systems aids in identifying anomalous activities, consequently fortifying security measures. Routine application of security patches and updates serves to rectify known vulnerabilities, while user education and training foster heightened cybersecurity awareness. Employing encryption technology ensures the secure protection of data, while integrity checks verify the integrity of systems and data, thus effectively detecting unauthorized alterations. The involvement of Red Teams and penetration testing helps in the discovery of vulnerabilities and the enhancement of security measures. Lastly, the formulation of an incident response plan allows for swift and effective responses in the event of a cyberattack.

6. Conclusions

Communication delays inevitably occur when transferring authority from the autonomous vehicle to the remote driver. research is needed to reduce communication delays. In this study, we confirmed the teleoperation method considering the communication delay between the autonomous vehicle and the control center. Especially, when communication delays coexist, authority transfer can be delayed due to the communication delay and may potentially not be able to control the vehicle immediately in emergency status. Commercialization of autonomous driving technology requires demonstrating safety and reliability. To compensate for this, we believe that teleoperation can prevent unforeseen accidents. In particular, teleoperation will play a big role in lanes where there are many things to consider, such as road conditions and signal rules. autonomous driving vehicles will be commercialized in the near future, and research on teleoperation to compensate for this should also be actively conducted. A lot of research should be done, such as compensating for the aforementioned communication delays and two-way control. However, it is also important that communication delay can be influenced by various driving conditions such as rain, snow, network bandwidth, and also regions. Therefore, a careful parameter should be chosen to guarantee control transfer can be fulfilled within the required time.