Network Latency in Teleoperation of Connected and Autonomous Vehicles: A Review of Trends, Challenges, and Mitigation Strategies

With remarkable advancements in the development of connected and autonomous vehicles (CAVs), the integration of teleoperation has become crucial for improving safety and operational efficiency. However, teleoperation faces substantial challenges, with network latency being a critical factor influencing its performance. This survey paper explores the impact of network latency along with state-of-the-art mitigation/compensation approaches. It examines cascading effects on teleoperation communication links (i.e., uplink and downlink) and how delays in data transmission affect the real-time perception and decision-making of operators. By elucidating the challenges and available mitigation strategies, the paper offers valuable insights for researchers, engineers, and practitioners working towards the seamless integration of teleoperation in the evolving landscape of CAVs.


Introduction
CAVs are continuously developing to provide safer and more convenient transportation.These vehicles can prevent many road accidents due to human errors [1].The Society of Automotive Engineers (SAE) J3016 (2016) defines six levels of driving autonomy for on-road CAVs, from level 0 to level 5 [1].Higher levels have more automated driving features, making them entirely driverless vehicles [1,2].In levels 0 to 2, a human driver is required to be present at all times.These levels have supporting/assisting features (e.g., warning systems).In level 3, human drivers are also present.Still, these vehicles are said to be conditionally automated, i.e., the vehicle can handle a few situations (such as lane changing, autonomous emergency braking (AEB)), and the driver should be monitoring and ready to take over the vehicle at any instant.Level 4, which includes higher automation features, has an automated driving system (ADS) that can handle most dynamic driving tasks without human intervention [1,3].Ultimately, the vehicles are expected to handle all situations and become fully autonomous in SAE level 5. Today, level 2-4 vehicles are being tested and deployed in the market [3,4].For example, Tesla's level 2 with autopilot system [5] and level 3 with the fully self-driving mode [6] are commercially available [1,3].
Even with remarkable advancements in CAV development, it would be unrealistic to anticipate zero system failures.It is widely acknowledged that CAVs may not handle all road incidents and will depend on human decisions [7], as the real world can be very uncertain.In this case, a human operator is required to take over the vehicle.It may not be through physical access but by remotely assisting/operating the vehicles in challenging situations [1].This remote access is known as teleoperation.The word 'tele' comes from a Greek word meaning "at a distance" [8].It involves a remote human operator to monitor, assist, and control a vehicle from a distant location for some manoeuvres or edge situations [7].For CAVs, it is also known as remote driving or tele-driving.
One particular use case of remote driving to enhance vehicle safety is by bringing the vehicle to the minimal risk condition (MRC), a stable and safe stopped state when a problem occurs [1].Additionally, integrating human teleoperators in the over-the-network control loop can improve the vehicle's operational efficiency by broadening the vehicle's operational design domain (ODD) allowing it to cope with challenging environmental conditions such as weather, visibility, speed limits, types of roadway, and motorways [3].
The communication system is an essential aspect of teleoperation.Maintaining the safety and security of the teleoperation communication systems is important, as these systems should be uninterrupted and transmit data in real time [9].The British Standards Institution (BSI) Flex 1886 recommends that ADS have appropriate autonomous capabilities to perform a minimal risk manoeuvre (MRM) safely (as a fallback solution to achieve an MRC) when connectivity is compromised or lost.The communication system can be compromised by system malfunction/failure, cyber-attack, or simply network issues (e.g., latency or bandwidth constraints).The communication capabilities of fourth-generation (4G), fifth-generation (5G), and future generations of wireless networks can facilitate the remote driving of CAVs [3].A few companies have already demonstrated that 5G networks are reliable for the teleoperation of remote vehicles, and they expect promising results in future experiments [10].Various networking technologies, including Ethernet, wireless fidelity (WiFi), General Packet Radio Service (GPRS), third generation (3G), long-term evolution (LTE), 4G, and 5G [10], have been considered in the literature through real-world implementations, simulations, and emulations.However, ensuring the availability of 5G and newer technologies on every road at all times might not be realistically achievable [10].Similarly, the availability of remote operators 24/7 might be an issue.Moreover, the CAVs are expected to generate approximately 10 gigabytes (GB) of data every minute [11], suggesting that even advanced networks such as 5G will require optimisation [10].
Network latency or delay is one of the major challenges that can cause disruptions in communication systems [1,3,[7][8][9][10]12].Latency can affect the quality of sensory data from the vehicle to the remote station and control command data from the station to the vehicle, in turn degrading the operator's and teleoperation performance.It can also cause overand under-steering of the remote vehicle.Moreover, longer and variable (time-varying) latency is an even more significant problem, making the control problem very challenging.The remote operators can tolerate latency up to a threshold beyond which it becomes unmanageable.For example, during vehicle teleoperation, Zhang [3] states that a constant latency of less than 170 milliseconds (ms) has a minor impact and is easily manageable whilst the operator can adapt to latency below 300 ms.However, latencies (with less variance) between 300 ms to 500 ms become challenging and significant for the operator to handle the vehicle (at slow speeds).Latencies above 700 ms make it nearly impossible for the teleoperator to interact with the vehicle in a timely manner.

Comparison with Existing Surveys
A summary comparison with relevant survey papers is provided in Table 1.Several related survey papers have contributed to the realm of teleoperation, focusing on aspects such as teleoperation research scope, architectures, taxonomies, advanced communication links, and challenges of mobile robots [8,12,13] and CAVs [1,3,7,10,14,15].
Opiyo et al. [13] emphasise architecture, communication links, and situational awareness in teleoperation.While some studies present teleoperation methods and enhancement techniques, Farajiparvar et al. [12] review approaches for time delay mitigation, focusing on control theory, user interface designs, and time-series prediction models.Similarly, Moniruzzaman et al. [8] comprehensively review teleoperation challenges and examine existing teleoperation methods and enhancement techniques (with limited coverage of latency compensation techniques).However, these survey papers revolve around mobile robots, which may not be valid for CAVs.
Other recent survey studies focus on CAV teleoperation.For instance, Bogdoll et al. [14] provide a taxonomy for the teleoperation systems, outlining recent terminology in the field.
Likewise, Mutzenich et al. [1] stress integrating remote operator roles into industry-standard taxonomies and use cases for regulatory frameworks.They also highlight the importance of designing control interfaces to maximise operator situational awareness.Additionally, Zhang [3] presents a vision of intelligent teleoperation systems powered by artificial intelligence and advanced networks (e.g., 5G), discussing their potential benefits and challenges.Tener and Lanir [7] analyse significant challenges in CAV teleoperation and provide design guidelines for future teleoperation interfaces.Amador et al. [10] compile works on the remote operation of road vehicles, categorising them based on the level of human intervention and identifying challenges in deployment across technological, regulatory, and commercial domains.Zhao et al. [15] survey remote driving challenges and solutions of latency, driving feedback, support control, and initiatives.They briefly provide the impacts of latency on remote drivers and review a few mitigation strategies.Although the existing surveys acknowledge the significance and challenges in the context of CAV teleoperation, extensive research focusing on network latency as a challenge and mitigation strategies specific to this domain remains scarce.Many of these surveys have insufficiently covered the network latency aspects of CAV teleoperation.
To the best of our knowledge, the current literature lacks comprehensive studies on the impact of network latency and mitigation methods during teleoperation, particularly in the CAV domain.this paper proposes a comprehensive and systematic review that explores the trends, impact, and state-of-the-art latency mitigation strategies that apply to CAV teleoperation, further offering valuable insights for researchers and practitioners in this field.The contributions of this paper are as follows:

•
It thoroughly examines the various network quality-of-service (QoS) parameters and their relation with latency, outlining their significance in the context of teleoperation, further indicating the sources of latency in system aspects of CAV teleoperation.• It critically analyses state-of-the-art latency mitigation methods for control and perception latency, discussing their advantages and disadvantages for compensating time-varying delays during CAV teleoperation.Additionally, it presents up-to-date initiatives and standardisation efforts that have contributed to remote driving.

Paper Structure
The organisation and structure of the paper is shown in Figure 1.Section 2 covers the systematic review process.Section 3 discusses the challenges of teleoperation with a focus on network latency.Section 4 provides the nature of network latency and its impact

Search Process
The literature for this survey was selected based on various inclusion and exclusion criteria.The papers were searched and classified mainly on relevance to the topic, focusing on keywords such as network latency, teleoperation, and CAVs.The search process included alternate keywords (in different combinations) as shown in Figure 2.However, in this process, it became apparent that a limited number of papers focused on addressing the requirements of the teleoperation of CAVs.To cover this literature gap, papers on teleoperation from diverse domains have been considered and compared with the CAV requirements.
The selection process for this review was performed according to the PRISMA guidelines [16] and is depicted in Figure 3.Most of the screened papers are from secondary sources that are peer-reviewed and published articles, surveys, journals, and conference papers, including a few from the non-academic literature from the policy landscape, such as technical reports and white papers.The papers include studies with diverse research methodologies such as frameworks, simulation-based, theoretical analysis, experimental and empirical studies to provide a comprehensive overview.Another criterion was to include literature published within a specified time frame to maintain timeliness.This survey covers literature within the last ten years.The literature was gathered and acquired from publicly available research databases and search engines such as Google Scholar, BASE, CORE, Science Gov, refseek, ACM library, ScienceDirect Elsevier, Springer, Semantic Scholar, MDPI library, ResearchGate, IEEE Xplore digital library, Wiley library, and Frontiers.

Data Analysis
The total number of papers collected was approximately 230.Among these, about 25% were excluded/screened out, considering their relevance to the inclusion and exclusion criteria.The remaining papers were reviewed and included in this survey paper.
Using different combinations of the keywords mentioned above, the included papers were further categorised into broad topics, including survey papers in relevant fields, papers from networking and communications, papers that focus on network latency and teleoperation, papers on latency mitigation methods, and industry initiatives and standard reports, as shown in Figure 4.Each category was again divided based on its application domains, namely, CAV and other (non-CAV) domains.As inferred from the chart, the majority of screened papers focus on the CAV domain across various categories.However, a divergence is observed in the latency teleoperation and mitigation methods categories.In these areas, the proportion of papers from other domains surpasses those directly related to CAVs, as fewer papers have addressed these topics in the literature.This disparity and scarcity of studies reflect the importance of further research in this direction.

Teleoperation Challenges
CAV teleoperation or remote driving has a distinctive set of challenges.CAVs heavily rely on multiple sensors and communication networks, unlike mobile robots or other vehicles.Since CAVs are connected to networks, they face increased cyber threats.Further, as CAVs are expected to operate in the real world, they can be dynamically operated, over long distances and with a full range of speed environments [3,7].They are also expected to make safe and ethical decisions.Hence, they require a robust, reliable and seamless communication system.
Communication systems can be compromised by network latency during CAV teleoperation.It is widely acknowledged in the literature that network latency is one of the major challenges of teleoperation, which can cause significant disruptions in communication systems and performance [1,3,[7][8][9][10]12].Long and variable latency poses an even more significant challenge, making it nearly impossible to operate effectively.Even today, it is an unsolved problem.
Latency can affect the quality of feedback from sensory data (visual, audio, and haptic) of the vehicle to the teleoperation station, making it difficult for the operator to acquire the current state and environment of the vehicle.Simultaneously, it affects the control command data from the station to the vehicle, in turn degrading the overall teleoperation performance in the control loop.Certain studies have indicated that latency can influence the decision making, behaviour, and mental workload of remote drivers, causing overand under-steering of the remote vehicle and leading them to irregularly adjust their pressure on the acceleration and brake pedals [7,8,15], which thereby increases the driver's anger and frustration [17].Latency simultaneously impacts the vehicle in terms of energy consumption, motion sickness, and degrading the comfort of passengers [15].
It has been hypothesised in the literature that latency would have detrimental effects on the performance of both teleoperation and operator perception [18].The performance will only decrease with increasing latency.Different types of latency (from various sources) can distinctly impact performance and adaptation strategies based on individuals [18].For example, the effect of latency will be larger on complex system tasks than on simple ones [19].Further, the level of automation can improve the system's ability to tackle latency, thereby increasing the trust in the system.
Another major challenge of teleoperation is situational awareness limitations [7,8].Remote operators may need added data to improve their perception, including viewpoint shifting and adjustment, map merging, depth, spatial awareness, and multi-sensory feedback.All these data need a good system design and interface to avoid a cognitive workload for remote operators, as it can trigger confusion and distractions that can lead to accidents.
There are also other teleoperation challenges, such as lack of physical sensing (e.g., force, sound), requiring improvement in the user interface with sensor fusion, bandwidth constraints, control issues such as lack of model estimation, uncertainties, and reliable local autonomy, and other issues related to human factors and workload requirements [7,8].

Teleoperation System Model
An overview of the CAV teleoperation system architecture is shown in Figure 5.The system architecture for the teleoperation of CAVs includes (1) a teleoperation station with a suitable human-machine interface (HMI) or control interface, (2) a remote CAV, and (3) a wireless communication network [10].The communication network is responsible for seamless data transmission and operational efficiency between the vehicle and the station.The operated CAV transmits perception data streams obtained from sensors (e.g., video data from the cameras) to the teleoperation station via onboard communication networks for "uplink" transmission to a remote operator.The remote operator receives the visual and other perception data, and then, sends back the control data (e.g., steering and braking commands) using a teleoperation controller/interface through the "downlink" of the same communication network.

Wireless Technologies Used in Teleoperation
Wireless technologies play a pivotal role in teleoperation systems by enabling seamless control and communication between the operator and the teleoperation station.It is crucial to maintain the safety and security of the teleoperation communication systems, as these systems should be uninterrupted and transmit data in real-time [9].The wireless technologies used in teleoperation with their performance and latency requirements are listed in Table 2.These are technologies that have been used across various domains.The technology performance is determined by various network parameters, including frequency, bandwidth, coverage, data rate, throughput, and latency.The most influential factors for a general teleoperation use case include latency, throughput (user data rate), data rate (maximum capacity), and coverage.Hence, only these parameters are mentioned in the table.
There has been a drastic evolution in network technologies over time, with each iteration achieving improved performance and unlocking new applications, in turn, decreasing the effects of latency.A standout example is the evolution of cellular networks, which have revolutionised the way we communicate and access data.They have evolved from first-generation 1G (from 1980) to fifth-generation 5G (from 2020 onwards) networks, where each generation is an improvement on the previous one [20], providing better coverage, higher frequencies, faster data transfers, lower latency, higher reliability, and mobility, among other improvements.
The main factors for network data traffic growth are increasing populations with smart devices, applications, and sensors, leading to the Internet of Everything (IoE) [20].The statistics of the International Telecommunication Union (ITU) state that just global cellular traffic will rise to 607 exabytes (EBs) per month by 2025 and 5016 EB by 2030 [20].With the continuous demand for network traffic and emerging applications (e.g., CAVs), the current technologies will have difficulty sustaining the requirements.
Latency can cause disruptions to applications relying on communication technologies.Different technologies for teleoperation exhibit various levels of latency based on their QoS factors.For example, experimental results have shown that 3G networks have a mean latency of 217 ms and 205 ms for vehicle control and video streaming, respectively.Whereas the next generation showed a slight improvement, with 4G networks having a mean latency of 183 ms for video streaming and 107.2 to 110.3 ms for vehicle controls [21].The present and future generations of networks, including 5G and sixth generation, are expected to have much lower latency and better network quality (e.g., high frequencies, fast data transfers, high reliability, mobility) [20], satisfying the requirements and leading to efficient teleoperations.For example, teleoperation using 5G networks can have latency as low as 60 ms but as high as 260 ms in some cases [22].
As inferred from the table, the wireless technologies used in teleoperation can be broadly classified as cellular and non-cellular.Cellular technologies, especially the recent advances, provide reliable and high-speed connectivity over large geographical areas.Many studies that have used 3G, 4G, and 5G networks for teleoperation have demonstrated that latency and bandwidth requirements could be met for teleoperation applications, particularly for CAVs and remote scenarios requiring long-distance operations.In such scenarios, wired connections are infeasible.However, they have limitations in extremely remote areas such as rural areas or due to heavy traffic and congestion in urban areas, which can lead to potential latency or signal degradation.Some studies have revealed that there is an evident difference between 4G and 5G networks for teleoperation [23].
The 4G networks are just able to support the necessary infrastructure for basic remote driving capabilities [10].The relatively high data rates and improved reliability over 3G allow sufficient quality of video and control transmission, which is essential for this use case.However, as seen in the table, latency of 100 ms can pose challenges for real-time responsiveness, leading to potential difficulty in vehicle control.Moreover, the bandwidth limitations can restrict the parallel processing and amount of sensor data.Despite these constraints, 4G serves as a standard for demonstrating the feasibility of remote driving.
Going one step further, 5G networks significantly enhance remote driving capabilities with their ultra-low latency (on the order of 10 ms), high data rates (up to 10 Gbps), and significantly improved reliability [10,24].They offer large volumes of data that can be quickly and reliably transmitted.These features are satisfactory for various autonomous applications, including multiple high-resolution sensor data, responsive control, logistics, and emergency services [3].They can even ensure reliable management of fleets of remotely operated vehicles.
The non-cellular classification includes diverse technologies that are more suitable for applications other than CAV teleoperation.Technologies such as WiFi, Bluetooth, and Zigbee are commonly used for short-range communications with overall good network performance.These are well suited for applications in controlled and confined environments, as in robotics.Satellite communication performs excellently in areas where other traditional terrestrial networks are impractical, with high latency given its large coverage area.However, it is expensive and is sensitive to environmental factors.Both cellular and non-cellular technologies have been emulated/simulated for more flexible, scalable, and secure experimentation.On one hand, these experiments attempt to allow the replication of real-world networks in controlled environments without risking physical systems.On the other hand, they offer the opportunity to define parameters and effectively validate and test various teleoperation applications in different scenarios and conditions.The parameters of simulated/emulated networks depend on how the user defines them.For example, in a simulation experiment, driving control had a latency of 170 ms and flight control had 1.5 to 3 s [28].Another example, an NS3-IP-based emulator, had a latency of 16-19 ms [10].The simulated ROS-based virtual network adapter had 64-74 ms, Rosbridge had 270 ms and WiFi had 445 ms [8,27].However, these networks are primarily implemented in simulations and may not always yield realistic results due to real-world networks' inherent uncertainty and dynamic nature.
Studies have also used other communication technologies in teleoperation, including the Internet through a wired Ethernet cable, VLAN, radio link, inter-process communication (IPC) wireless network, umbilical cable, dual-tone multi-frequency (DTMF) and multimodal radio frequency (RF), VideoLAN Client (VLC), and acoustics [8,30].

Network Latency Characteristics
Network quality in a system (e.g., teleoperation) can be defined by its QoS and quality of experience (QoE) [31,32], also known as objective and subjective factors, respectively.The objective factors include the network performance parameters.Meanwhile, the subjective factors include user performance parameters such as user interface, video quality, driving precision, comfort level, manoeuvre security, and interruptions [32].
Network latency is a QoS component that produces a time delay when transferring data across a communication network [33,34].Usually, it is calculated as end-to-end delay or round-trip time (RTT), which refers to the time taken for data transfer from source to destination and back again to the source.Networks with longer delays resulting in a noticeable lag have high latency.A sufficient amount significantly affects the application system performance, and more elevated levels can cause system failures.

End-to-End Delay
The latency sources in this teleoperation feedback loop are shown in Figure 6.Latency sources and components of CAV teleoperation in an end-to-end loop include sensor exposure delay (e.g., camera capture delay) (∆S d ), encoding delay (∆E d ), data communication delay in terms of transmission (vehicle to the station (∆T d,1 ) and station to vehicle (∆T d,2 )), processing or decoding delay (∆D d ), visual display delay (∆V d ), operator response time (∆O d ), and vehicle actuation delay (∆A d ) [25,35,36].It is challenging to mitigate all these different sources of latency in a system [15].
The one-way latency is calculated as the sum of scheduling, transmission, and retransmission time [37].The information might be re-transmitted if lost.The sum of delays from camera capture to display is called capture-to-display latency or glass-to-glass latency [25], which is also the one-way uplink latency.The capture-to-display latency (∆ C2D ) is given by Similarly, on the one-way downlink (DL) side, control-to-response latency is the sum of delays from sending operator control commands to the vehicle actuation response.This latency also includes control command processing, encoding, and decoding delays.However, these delays are negligible when compared to delays in capture-to-display latency.Hence, they are not considered in the equation.The modified control-to-response latency (∆ C2R ) can be represented as Finally, the overall end-to-end latency (∆ E2E ) or RTT is the sum of delays of all the components from uplink to downlink over the network.The end-to-end latency can be represented in an equation as  Human operators have demonstrated the ability to perceive delays as low as 10 to 20 ms [28].A latency of around 300 ms has a noticeable impact on the operator behaviour.In contrast, latencies of 1 s or more can drastically affect real-time operations.Such delays often cause operators to adjust strategies, transitioning from open-loop to closed-loop control methods, including start-and-stop or move-and-wait, sometimes even disregarding the visual feedback [18,19].The cognitive threshold for maintaining real-time performance is typically within 0.4 s.Upon receiving feedback from the vehicle, operators must respond within 0 to 3 s, with a minimum response time of 0.2 s [39,40].

Modelling Network Delay
For network latency, a major problem is with the variability (unpredictable fluctuations) of the delay rather than its magnitude, which degrades the teleoperation performance [29], suggesting that there needs to be a deterministic delay [41].
Researchers have modelled delays using simulations and emulations or have gathered data from real communication networks.These delays are injected into simulated or emulated networks.The delays used in these experiments can be classified as constant (fixed time delay) or varying (incorporating network jitter) [42].Uddin and Ryu [43] categorise some predictive methods based on constant and variable time delays.Further, the end-to-end delay can be symmetric or asymmetric [43].Symmetric means the uplink and downlink delays are approximately equal.Meanwhile, asymmetric delays refer to different delay values in the uplink and downlink, which is usually the case for teleoperation, i.e., the uplink sensor data delays are greater than the downlink control command delays.
However, it is crucial to evaluate teleoperation performance on delay values close to real-world networks exhibiting many fluctuations.Studies that model real-world network delays tend to have varying delays that can fit a predefined time sequence (such as a distribution or a function) or model them as a random process [42].It has been shown that the generalised extreme value (GEV) distribution can model delays for mobile communication data [35,42].The GEV distribution has been used to model variable delay in various studies [35,42,[62][63][64].The probability density function (PDF) of GEV(ξ, µ, σ) is defined as where x is a random variable for the delay value, ξ is the shape parameter, µ is the location parameter, and σ is the scale parameter.Based on extreme value theory, if ξ > 0, the distribution has a heavy tail with µ − σ ξ as the lower bound.The authors in Perez et al. [65] characterise the function of RTT delays as a defined delay threshold (where it is not very significant), rapidly and linearly decaying, and then, ending with a long tail.Examples of such functions include piecewise linear [66], algebraic, logistic, and log-logistic [67].Thus, Perez et al. model delays using the following algebraic function: where where T is the interaction lag and T m is the model parameter.
In another example, Zhou et al. [68] use variable time delays calculated by combining constant terms for the main delay with periodic trigonometric functions of different periods for small delay variations together with normally distributed noise, as shown below: where T 1 and T 2 are periods of trigonometric terms and ω(0, 1) is normally distributed noise.Likewise, Zhou et al. [69] model delays as a sine wave distribution.Bacha et al.
Some studies have also considered collecting and using real-world network data for teleoperation.Zheng et al. [42] measure the delay between Michigan and California.Guo et al. [85] collect variable delay from real network data using the User Datagram Protocol (UDP).Saparia et al. [86] collect data from 4G LTE networks.Kebria et al. [87] use the Internet to collect delay data between Australia and Scotland.Sridhar et al. [88] experiment on a physical network using Amazon Web Services for Uttar Pradesh and Karnataka.Other studies that used real networks include [4,29,[89][90][91].

Interdependency between Network Parameters and Latency
Certain network objective factors influence network latency and teleoperation [18,22,29,[31][32][33][34]41,60,65,[89][90][91][92].Latency is directly impacted by distance, vehicle density, handover between cells, speed, mobility, and packet loss.Latency increases as the remote vehicle is farther away from the teleoperation station, which increases the overall RTT [32,34,90].Wireless networks have limited bandwidth, and increased network traffic from vehicle density can result in congestion, causing delays in data transmission and increasing latency [32,91].Increasing handover between cells might lower the throughput and increase the latency [90].Teleoperation at higher speed demands even faster reaction times, and higher mobility also decreases the operator situation awareness [22,60,65].Latency measures the delay in a packet's arrival at the destination, and more latency leads to fewer packets arriving per unit of time, which might lead to higher packet loss [33].
Meanwhile, some parameters may have indirect relationships, such as bandwidth, throughput, signal strength, data rate, and coverage.For instance, although less bandwidth increases latency during peak usage, more bandwidth does not necessarily mean more data, particularly if latency is high [33,34].A low-latency and -bandwidth network indicates that throughput will also be low.With a large bandwidth and low latency, throughput will be higher, and network connection will be more efficient [33,34,90].Latency becomes lower with better signal strength.However, it is not a linear correlation.Signal strength may influence the throughput but not the latency [89,90].As long as the signal strength is not too poor, the network latency is maintained within the acceptable limits for teleoperation.Higher data rates typically lead to lower latency, as more data can be transmitted and processed quickly and vice versa [32,91].Coverage can be less in rural areas than in urban, which can lead to more latency, indicating that latency depends on location as well [32].Some studies have shown that latency has a more significant effect on teleoperation than actuator delay [18].Moreover, latency has a significant impact on video streaming methods such as field of view (FOV), orientation, camera viewpoint, depth perception, video quality, and frame rate [22], which might affect the situational awareness of the operator.
Further, latency depends on the wireless technologies being used.Different network types and mediums provide different QoS and have diverse requirements and effects on latency [90].

Network Delay Impact on Various Teleoperation Domains
Teleoperation is generally used as a fallback solution to maintain the safety of a vehicle in cases of failure scenarios such as malfunction, challenging weather, confusing situations, collision, or other such situations [93].Other use cases of teleoperation include places where it is dangerous or difficult for humans to reach, such as telerobotics for space and underwater exploration, satellite communications, use of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) for reconnaissance missions, search and rescue, or surveillance in military/defensive applications, and also for telesurgery in medical domains [40].The impact of network latency and requirements across various domains are detailed in Table 3.Several studies have explored teleoperation as a critical use case within their respective domains.Although teleoperation is used in different critical applications, most studies employ robots as their teleoperation system.Some use mobile robots, while others require single-or dual-arm manipulator robots (such as telesurgery).Nevertheless, a consistent finding across these studies is the detrimental impact of latency on teleoperation performance and operational safety, ranging from compromised mission effectiveness to increased mortality risks in critical applications such as telesurgery.It is universally acknowledged that having lower latency is the best case for effective operation.The minimum acceptable latency threshold is shown to be influenced by factors such as required data rates, teleoperation distance, and, most importantly, the choice of wireless technologies.
Even within domains seemingly disparate from traditional teleoperation, such as gaming, parallels can be drawn with teleoperation regarding identical setups and the importance of responsive interactions from the operators.Thus, while the specifics and unique demands may vary across domains, the overarching concern remains the mitigation or compensation of latency to ensure optimal teleoperation performance and safety.

CAV Teleoperation Requirements
Considering various domains, most of the literature on teleoperation uses mobile robots with or without manipulators.Comparatively, there is less research conducted on CAV teleoperation [8,12].Despite the similarities of challenges faced by teleoperated robots and CAVs, the latter operate in significantly more complex environments, encountering far greater challenges [7].For example, CAVs have higher stakes and operate in adverse weather conditions, long distances from remote stations, at higher speeds, and involve ethical decision making.Hence, more research is recommended in this area.
The network requirements of CAVs are different from those of teleoperation in other domains.For example, compared to robotic teleoperations, CAVs involve realistic, uncertain, long-distance environments with usually higher vehicle speeds.The requirements of CAV teleoperation are shown in Table 4.Most studies provided requirements for latency thresholds and data rates, as the most critical QoS parameters for CAV teleoperation are the uplink data rates and the downlink latency [100].The effect of latency thresholds can be classified as less impact, acceptable, difficult, and impossible thresholds.As mentioned earlier, using different wireless technologies for teleoperation can induce different latency thresholds.Summarising Table 4, it is evident that the downlink latency threshold is relatively lower than the uplink.By combining all the overall latency thresholds, it can be concluded that less than 170 ms has less impact on performance, and between 250 ms and 300 ms has an acceptable effect, with the average uplink latency in the range of 50 to 120 ms and downlink latency in the range of 20 to 80 ms (with some outliers).As the latency becomes higher, such as greater than 300 ms, the operator performance seems to degrade, making it difficult.Further degradation is seen above 700 ms and 1 s, making teleoperating almost impossible.
Similarly, regarding data rate, it can be inferred that CAVs have significantly higher uplink data needs than downlink needs.This is because CAV sensor perception (such as camera, lidar, and radar sensors) takes up more data than the control commands.The average uplink data rate can range from 8 to 50 megabits per second (Mbps), as camera data themselves take about 8 Mbps, while data rates for the downlink are in the range of 0.25 to 5 Mbps.
Correspondingly, there are network QoS and other environmental requirements for CAV teleoperation not covered in Table 4, such as a bandwidth of about 30 to 100 MHz, depending on urban or rural environments, a throughput greater than 3 Mbps [65], visual quality with a minimum or average resolution of 640 × 480 along with approximately 150 • view angle FOV, and a fluent image of 25 to 30 frames per second (FPS) is sufficient for teleoperators [90].Further, the service reliability is preferred to be 99% for uplink and 99.999% for downlink [32,91].Suppose the teleoperation service is scaled up in the coming years.In that case, it will be feasible for the current networks to handle a vehicle density of about 10 vehicles per kilometre square (veh/km 2 ) [91].Most studies focus on driving at low speeds, such as less than 50 km/h [32].However, a few studies suggest the maximum speed teleoperated vehicles can tolerate is 100 to 250 km/h, above which performance will degrade significantly [41].Therefore, to avoid such degradation, it is recommended to perform remote driving at low-to-medium speeds rather than at high speeds.With regards to distance, CAV teleoperations can be conducted over long distances, for example, 5200 km in [65] and 19,000 km in [90].

Latency Mitigation Strategies
Most delay compensation or mitigation approaches can be broadly classified into control, perception, and network optimisation methods.These methods are reviewed further in the following subsections.However, it is worth noting that, in this paper, it is assumed that the network optimisation methods are already performed.Hence, they are covered very briefly.This paper primarily focuses on the control and perception approaches.A comprehensive list of latency mitigation strategies for control and perception is provided in Table 5 and Table 6, respectively.A practical summary of these strategies highlighting their advantages and disadvantages is later provided in Table 7.
Predictive methods have been shown to be well suited for delay compensation on both the downlink and uplink of teleoperation [70,113].In the realm of predictive methods, passivity-based and predictive display-based methods have been most commonly used.They have been shown to reduce inconsistency.However, the performance degrades with increasing and variable time delays.A broad classification of latency mitigation methods is demonstrated in Figure 7.

Control Latency Approaches
Control latency is the delay in data transmission of control commands from the operator to the vehicle (i.e., on the downlink).A list of latency mitigation strategies for control is provided in Table 5.On the one hand, earlier researchers focused on mathematical models for accurately predicting and compensating for undelayed controller signals [45].These methods are well known to be model-based approaches [69].They are usually modelled according to control theorems, out of which the most extensively researched compensation methods are based on the perspective of passivity theory [44,51,114].Passivity-based control methods predict the signals after the delay without knowledge of the magnitude of the delay.They ensure the stability and transparency of the bilateral teleoperation systems.
A few examples include wave variable-based methods, scattering signals, time-domain passivity analysis (TDPA)-based methods [48,77,79,80], and model-mediated teleoperation (MMT) [69,80,115].The major drawback of these methods is that they require knowledge of system dynamics.Moreover, their performance degrades due to model errors and complicated conditions with variable time delay, large disturbances, or extensions to multilateral systems [69].On the other hand, model-free approaches based on data-driven models have been developed.These methods have grown popular as they are more robust and do not require any knowledge of system dynamics [69].They are adaptable and can deal with disturbances and uncertainties while making predictions.A popular example is neural networks (NNs) [69,116].Time-series prediction methods effectively mitigate the effects of control latency by predicting or imitating the operator behaviour [12].They tend to be more adaptable to non-linear data.In the literature, most studies employ or combine traditional NN methods such as auto-encoder (AE), variational auto-encoder (VAE), recurrent neural network (RNN), and long short-term memory (LSTM) for time-series data [12].Among these, RNN [74,117] and LSTM [55,69,74,118,119] are more suitable for time-series prediction.However, sometimes RNNs cannot learn long-term dependencies, whereas LSTMs have the advantage of retaining both long-and short-term memory.A limitation of LSTMs is that they are incompatible with dynamic output lengths and may require retraining [12].More recent methods, for instance, sequence-to-sequence (Seq2Seq) models [120] and generative adversarial networks (GANs) [121] have shown great performance for time-series prediction, and they can tackle these limitations.GANs learn the distributions of time series well and also adapt to new data, but might not capture dependencies.Meanwhile, Seq2Seq has a better mapping of input and output data relationships.
Other significant control prediction methods include state estimation filters, motion models, linear regression, proportional-derivative models, statistical models, and probabilistic models [116], among others.Linear regression is a simple approach with good accuracy for smooth and linear motion.However, it is prone to errors and overshooting [122].State estimation filters, such as Kalman filters (and their extensions), are commonly followed by motion models and are robust against fluctuations [122].However, they are sensitive to noise and computationally expensive.Proportional-derivative methods achieve stability without knowing system dynamics, but their robustness may come at the cost of transparency [45].Statistical models represent the control loop in the form of equations; for example, auto-regressive models, moving average models, and their extensions.However, these methods are unsuitable for non-linear and dynamic data, which can be expected in the real world [12].
Other control approaches aim to minimise delay, including supervisor control, control autonomy augmented reality [113], and adaptive-based control [70,113].Approaches such as supervisor control involve the operator sending high-level commands, minimising communication requirements and delay, where the remote robot or vehicle needs planning and control algorithms.However, it does not involve continuous teleoperation [114].Onboard control autonomy capabilities can be activated to mitigate the effects of delay.Such approaches can be classified as adjustable autonomy, collaborative control, mixedinitiative control, and sliding autonomy [51].They are all impacted by communication delays and are computationally expensive [45].[54] Robotic teleoperation Controlhandwritten letter drawing K-means, Gaussian mixture model (GMM), hidden semi-Markov models (HSMMs), and linear quadratic tracking (LQT) for motion recognition and segmentation [48] Bilateral robot teleoperation

Control and force feedback
Force controller and timedomain passivity approach (TDPA) [125] Bilateral teleoperation Improve control Integral-order and fractionalorder PD controller, along with frequency-domain analysis for maximum upper bound of delay, Lyapunov-Krasovskii functional [118] Train to ground communication State and network prediction LSTM and high-degree polynomial linear regression (HPLR)

Perception Latency Approaches
Perception latency is the delay in the perception system at the remote operator's end.A list of latency mitigation strategies for perception is provided in Table 6.Predictive displays generate the probable vehicle response on the delayed or predicted view based on the current actions of the operator.They have been shown to effectively withstand and compensate for the negative impacts of latency during teleoperation [42].These methods can be classified as offline or online [129].The former involves using prior knowledge of static environments and operator behaviour models before making predictions.The latter is suitable for dynamic environments and makes near real-time predictions to provide instantaneous feedback.
During remote driving, the operator can receive sensory feedback in three main forms: visual, audio, and haptic [7].However, most studies do not focus on audio and haptic feedback.Therefore, visual data are significant for the remote operator.Predictive display methods mainly focus on visual feedback prediction.However, most of the research on predictive displays uses simulation-based experiments [36].
Earlier offline approaches estimate and predict images that rely on first-order prediction techniques, depending on the system, delays, and actions of the operator [45].They are restricted to state prediction and future pose estimation.For example, estimating the visual impact of control commands, heading angles, and steering wheel angles, predicting position and trajectory, or by using other dynamic, curvature, and geometric models [52,129].However, reliance on geometric models for feature extraction and matching in low-textured scenes may lead to inaccuracies [129].Moreover, first-order techniques are ineffective for highlatency, high-speed, and long-distance teleoperations and do not include factors related to the uncertainty of future events.Integrating these first-order approaches with recent state-of-theart approaches that use artificial intelligence (AI) and NNs is challenging [46,51].
Similar to the model-free control approaches in the previous section, the modern online prediction approaches involve deep neural networks (DNNs).They mainly involve data-driven models for time-series predictions of future-perspective video frames [8].These approaches have been shown to give good results in terms of performance.The advanced methods for predictive display include the use of generative AI methods for pixel synthesis (e.g., GANs) [46,129,130], pixel transformation and time-series methods (e.g., LSTMs) [61], and probabilistic models [8].Alternate methods that reduce the burden on computation and the interface can be considered, such as motion and content separation and extracting higher-level features in the visual feedback.Though DNNs offer accurate predictions for non-linear and long-term signals, they require large amounts of data to train for this accuracy, which can be costly in terms of computation, memory, and time [15].Additionally, they are considered black-box models, which affects their transparency and trustworthiness, especially in safety-critical applications such as remote driving.Recent studies have also demonstrated that NNs' predictions can be altered/manipulated by pixel adversarial attacks [131,132].
Other visual feedback enhancement techniques for enhancing teleoperators' overall situational awareness (and not for delay compensation) include exocentric view, automatic view adjustment, stereoscopic vision, virtual environment, vision-based object tracking, and predictive systems [51].

Network Optimisation Approaches
These optimisation methods target the latency during data transmission at the network level.Parvez et al. [135] provide a survey on latency reduction solutions for 5G networks.They divide the approaches into radio access network (RAN) solutions (e.g., short frame/packets, new waveform designs, mmWave aggregation), core network solutions (e.g., high-speed backhaul, mobile edge computing (MEC)/fog-network architectures), and caching solutions (e.g., distributed, centralised caching).
Various approaches have been proposed to optimise the network for reduced latency.For instance, Hollinghurst et al. [136] use redundant messages to exploit randomness across multiple paths, leading to low latency.Similarly, Hui et al. [137] consider a redundancyaware federated learning architecture for efficient and cooperative vehicular networking and improved data quality accuracy.Ndikumana et al. [138] propose intelligent infotainment caching models in autonomous vehicles, where they use deep learning to predict contents that need to be cached.Their method can minimise content downloading delays.Belogolovy et al. [38] reduce latency by using multiple links, enabling rate control and scheduling in combination with slice-by-slice video processing.Hui et al. [139] develop a smart-contract-based secure edge computing architecture, which offers low-cost services for vehicles in 6G networks.Heryana et al. [140] focus on reducing video streaming latency in vehicle teleoperation through network factors.They apply UDP and the Real-Time Messaging Protocol (RTMP), tune the encoder, and further apply data compression.In another representative work, Zhang et al. [141] utilise DetNet for telesurgery, integrating timesensitive networking (TSN), software-defined network (SDN), and other technical features for a more deterministic network with low latency and jitter.Qiong et al. [142] propose deep Q-learning to predict the optimal minimum contention window (MCW) for improved vehicular communication and age fairness.Kousaridas et al. [100] analyse the requirements of QoS predictions and discuss the architecture of a prediction model for 5G V2X applications, specifically for remote driving use cases.Further to that, Barmpounakis et al. [143] use LSTMs for QoS prediction of 5G networks for CAVs.[145,146] and cooperative intelligent transport systems (C-ITSs) [146].The 3rd Generation Partnership Project (3GPP) develops standards for wireless networks, inclining towards cellular vehicle-to-everything (C-V2X) [145].The IEEE developed communication protocols and standards for the Internet of Vehicles (IoV) and intelligent transport systems (ITSs) among various interfaces, including vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and so on [31].
Standards for CAVs have been established by SAE standard J3016 since 2014, which has defined taxonomies on driving automation systems and levels [10,14,147], where SAE J3016 focuses on defining remote assistance and remote driving.
However, there is not a sufficient number of standards for vehicle teleoperation that provide the requirements and responsibilities of remote driving.
Recognising this gap, the BSI has developed several publicly available specifications (PASs) for the remote teleoperation of CAVs.According to BSI PAS 1886, the roles of remote operators include remote monitoring, remote assistance, and remote driving to supervise, support, and directly control the vehicle, respectively [9].The remote operator is responsible for continuously monitoring remotely at all SAE levels, including fully automated vehicles [10].SAE levels 0-3 require remote driving from human operators as a fallback solution to achieve MRC, sometimes including level 4. Remote assistance is primarily performed in SAE levels 3-5.Furthermore, BSI PAS 1884 includes a section on training for remote operators, ensuring the quality and proficiency of this critical role [148].
In addition to standards, some efforts from white papers and industry reports address the current state, use cases, and requirements for safe CAV teleoperation.Projects such as Endeavour-WP15B [149], 5GAA [24,106], 5GCroCo [108,109], and 5GMobix [110] have contributed insights and recommendations, particularly concerning impacts and requirements for CAV teleoperation and network latency.Moreover, the initial deliverable of the SAVOR project [150] undertakes experiments and provides recommendations on remote monitoring and teleoperation of CAVs.

Method Correlation with CAVs
There are very few studies on delay mitigation methods for CAVs.Nevertheless, it is feasible to adapt methodologies employed in other domains to fill this gap, provided the distinctive requirements of CAVs are taken into consideration.However, it is essential to note that the developed methods should be optimised and minimise any additional latency they introduce in the teleoperation control loop [8].A potential solution is using predictive feedback methods, such as the predictive model-free and online approaches, which are well suited for addressing both control and perception delays, respectively.The developed methods should focus on dynamically adjusting their sensor retrieval and driving behaviour based on the network conditions.One can achieve safe and reliable teleoperation even while predicting the short-term or immediate future [12], i.e., during latency, it is sufficient to predict approximately the number of frames based on the visual update rates (the cameras FPS).Further, to improve the operator situational awareness, it is also recommended to use multi-sensory feedback (e.g., cameras along with lidars, radars, audio, and haptic).However, appropriate sensor choice is critical for the sensor fusion to be safe and reliable, since camera-based sensors may require higher computational power and are sensitive to lighting conditions.Meanwhile, active sensors such as lidar and radar can be affected by interference from other sensors [173].
Moreover, most studies use simulations to experiment and test the teleoperation environments [36].This is a cost-effective solution, yet it may not always provide realistic results due to simplistic assumptions and constrained conditions.Again, real-world experiments also have their downsides, including being expensive to build and, most importantly, not reproducible for further research.

Conclusions and Future Directions
This paper provides a systematic literature review on the impact of network latency during teleoperation in CAVs.First, the latest trends and existing challenges have been discussed, followed by a system model of teleoperation along with sources of latency, the wireless technologies in teleoperation, and the impact of network latency across domains.Performing the teleoperation of CAVs is challenging, as they operate in significantly complex and dynamic environments with high mobility over long distances.The specifics and unique demands may vary across domains, but a consistent finding is the detrimental impact of network latency on teleoperation performance.The CAV teleoperation system has a distinctive set of requirements, including low latency, sufficient data rates, and high reliability.The most critical QoS parameters for CAV teleoperation are the uplink data rates and the downlink latency.This is because CAV sensor perception has significantly greater data needs than control commands, and a delay in control commands is more alarming than a delay in perception.For ideal CAV teleoperation, an acceptable latency threshold is preferred to be less than 250 ms or 300 ms, with the average uplink latency in the range of 50 to 120 ms and downlink latency in the 20 to 80 ms range.This threshold correlates with a few safety-critical applications such as telesurgery, UAV, nuclear, ordnance disposal, and some network games.Similarly, when it comes to data rate, the average uplink data rate can range from 8 to 50 Mbps, while data rates for downlink are in the range of 0.25 to 5 Mbps.The best wireless technology candidates to meet these needs are cellular networks, which provide high-speed communication and sufficient coverage for long-distance CAV teleoperation.As these technologies continue to evolve, from 4G to 5G and anticipated 6G, they play a crucial role in shaping the future of autonomous and remotely operated vehicles.The groundwork was laid by 4G by providing the basic infrastructure necessary for initial teleoperation capabilities, despite its limitations in latency and bandwidth.The advent of 5G significantly enhanced ultra-low latency, high data rates, and robust reliability, enabling real-time high-definition video streaming and precise remote control during teleoperation.This has made 5G the current standard for advanced remote driving applications, enhancing safety and operational efficiency.While still in the conceptual and development stage, the next generation 6G networks will be better suited to sustain emerging applications and network traffic.
Then, state-of-the-art latency mitigation and compensation strategies have been analysed.The latencies can occur in different sources of the CAV teleoperation system, including transmission delay, operator response time, encoding and decoding, visual display, sensors, and actuator delays, which makes delay mitigation quite challenging.The predictive methods are well suited for latency mitigation in both teleoperation control and perception approaches.Recent studies use model-free and online predictive approaches for control and perception latencies, respectively.Among these, the most used are deep neural networks for time-series predictions.In addition, there are also other equally capable methods involving regression, probabilistic models, and local autonomy.
Enlightened by the aforementioned analysis, this paper proposes the following future research directions:

•
More empirical studies across diverse environmental settings would be beneficial in determining the true and accurate CAV requirements.This also would establish a comprehensive understanding of CAV teleoperation.It is important to consider the impact of different levels of network reliability with varying QoS parameters, particularly latencies and bandwidths.For example, the studies can include testing performance in uncertain and dynamic environments, the trade-off between video quality and control responsiveness, and quantifying the robustness of predictive algorithms.

•
When developing latency mitigation methods, it would be beneficial to integrate various approaches.For example, traditional statistical methods (i.e., first-order prediction techniques) can be integrated with modern AI-based approaches to overcome their limitations.This will allow the methods to be effective and reliable against non-linear dynamic data (such as high-latency and high-speed teleoperations) without heavily relying on complex and accurate system models.Another example, integrating predictive methods with probabilistic approaches can provide the likelihood and uncertainty of predictions.This would help in determining the reliability of the developed method.• During teleoperation system experimentation, it is recommended to replicate realworld network conditions (including communication networks and injected delays) as closely as possible, as they offer robust evaluation by extensively testing the system on time-varying and fluctuating latency.Note that teleoperation in the presence of time-varying latency is more challenging than under constant and predictable latency.

•
While there are a few white papers and initiatives from industry projects and alliances for CAV teleoperation, there is a lack of sufficient standardisation in critical aspects such as network latency requirements to guarantee consistent performance evaluation, data security protocols to protect sensitive information and ensure the integrity of communication, and interoperability between different teleoperation system compo-

Figure 1 .
Figure 1.Summary diagram showing the structure of this paper.

Figure 2 .
Figure 2. Keywords for search criteria.

Figure 4 .
Figure 4. Paper taxonomy classified for CAVs and other domains.

Figure 5 .
Figure 5. System architecture of CAV teleoperation control loop.

Figure 6 .
Figure 6.Different sources of latency in CAV teleoperation control loop.The latency in the diagram is shown by red lines, which show approximately the magnitude of each component (for illustration purposes).The blue colour region is the uplink data from sensors to the operator perception interface.Similarly, the green region is the downlink, the control data from the operator station to the vehicle actuators.For example, the latency values in different sources can be [25,38]: • For camera capture, ∆S d = 17-33 ms at a frame rate of 30 fps. • For data encoding, ∆E d = 17-50 ms.• For data transmission, ∆T d,1 = 25-50 ms.• For video decoding, ∆D d = 17-32 ms.

Table 2 .
Wireless technologies used for teleoperation.

•
For digital visual interface (DVI) and liquid-crystal display (LCD), ∆V d involves a refresh time of 10 ms, a display time of 16.7 ms at 60 fps, and a frame buffer of 17 ms.• For joystick control command transmission, ∆T d,2 = 32 ms.• For sensors and actuators, ∆S d + ∆A d is roughly 50 ms.• Network jitter is approximately below 150 ms.• For LTE, the average handover latency is 40 ms.

Table 3 .
Impact of network latency on teleoperation across domains.

Table 5 .
Mitigation strategies for control latency.

Table 6 .
Mitigation strategies for perception latency.

Table 7 .
Cont.Standards bodies and industry alliances play a crucial role in maintaining quality, consistency, interoperability, and reliability across industries and domains by providing formal guidelines, products, services, responsibilities, specifications, and requirements.There are numerous standards for vehicular communications.For example, the Institute of Electrical and Electronics Engineers (IEEE) and European Telecommunications Standards Institute (ETSI) developed standards for dedicated short-range communication (DSRC) (IEEE 802.11p or IEEE 1609 based)