A Comprehensive Literature Review on Modular Approaches to Autonomous Driving: Deep Learning for Road and Racing Scenarios

Highlights

What are the main accomplishments of this study?

• A comprehensive analysis of deep learning techniques in both on-road and autonomous racing cars, highlighting distinct challenges and requirements for each context.
• The identification of critical challenges for future research, to ensure safety and performance in autonomous systems.

What are the implications of the main findings?

• The detailed evaluation of planning methods and performance metrics points to opportunities to refine existing methodologies and identify emerging research areas that can guide the development of more efficient, robust, and scalable autonomous driving technologies.
• The challenges identified in sensor fusion, environmental robustness, and computational efficiency imply that addressing these issues is critical to progress in autonomous systems.

Abstract

Autonomous driving technology is advancing rapidly, driven by integrating advanced intelligent systems. Autonomous vehicles typically follow a modular structure, organized into perception, planning, and control components. Unlike previous surveys, which often focus on specific modular system components or single driving environments, our review uniquely compares both settings, highlighting how deep learning and reinforcement learning methods address the challenges specific to each. We present an in-depth analysis of local and global planning methods, including the integration of benchmarks, simulations, and real-time platforms. Additionally, we compare various evaluation metrics and performance outcomes for current methodologies. Finally, we offer insights into emerging research directions based on the latest advancements, providing a roadmap for future innovation in autonomous driving.

1. Introduction

In the past decade, there has been a significant increase in traffic accidents worldwide. According to the latest report by the World Health Organization [1], traffic accidents have become the eighth-leading cause of death worldwide. Each year, approximately 1.3 million people lose their lives in road accidents, while an additional 20 to 50 million suffer non-fatal injuries, many of which result in permanent disabilities. Many sectors, including the transportation and urban planning sectors, as well as road authorities and lawmakers, play a vital role in addressing this challenge. To reduce accidents and improve road safety, autonomous driving technologies have become essential.

In line with these objectives, the automotive industry and the scientific community have focused on developing innovative technologies, particularly those aimed at enhancing the safety and performance of road vehicles [2,3]. These advances contribute to increased safety and road efficiency and a significant reduction in the risk of traffic accidents, lowering the number of fatalities on the road. In recent decades, autonomous systems have significantly advanced, offering promising improvements in driver safety and vehicle quality.

Existing studies have investigated modular systems utilizing various techniques, including classical methods, mathematical modeling, optimization, machine learning, and deep learning. In particular, deep learning has recently shown remarkable effectiveness in autonomous driving scenarios, as demonstrated by Zhao et al. [4]. The success of deep learning has led to its widespread adoption in autonomous driving. These techniques are used individually in modular components and end-to-end approaches [5,6,7].

End-to-end deep learning approaches bypass the traditional modular hierarchy, handling complex tasks autonomously through a single architecture. This approach streamlines workflows by eliminating the need for separate modules, which is shown to be effective in certain autonomous driving scenarios.

This systematic review uniquely analyzes deep learning approaches for autonomous vehicles in road and race environments. In the on-road context, autonomous driving components must navigate challenges such as traffic congestion, obstacles, parking, pedestrian detection, and managing vehicle speed in areas with congested traffic signals. Autonomous racing presents unique challenges, including speed control, optimized acceleration and braking, and maintaining track position, identifying the best trajectories for shorter lap times, and monitoring high-speed opponents to ensure safety and competitiveness.

This paper explains how deep learning addresses these challenges, focusing on the integration of the components of the modular system from existing studies. The main contributions can be summarized as follows:

• This review is the first to present a comparison of state-of-the-art approaches that address deep learning methods between autonomous driving on the road and racetrack scenarios.
• We highlight existing deep learning approaches for modular systems, including perception, planning, control, end-to-end approaches, and safety.
• We also describe both scenarios’ benchmarks, simulations, and real-time platforms.
• In addition, we discuss evaluation metrics, state-of-the-art performance, and their comparisons.
• Finally, we outline the potential challenges and research directions from the state-of-the-art.

The rest of this paper is organized as follows: Section 2 describes the background overview of autonomous driving, autonomous racing cars, and modular systems. Section 3 elaborates on recent surveys conducted for autonomous driving, incorporating various deep learning techniques. Section 4 presents the research questions addressed and describes the methodology used to prepare this systematic review. Section 5 presents the characteristics of the selected studies and a synthesis of the results achieved. Focusing on each of the defined research questions, Section 6 presents an in-depth analysis of the articles included in our study, providing a comparative analysis of existing approaches as well as proposing future research directions. Section 7 presents the limitations of this study. Section 8 discusses the performance, real-time challenges, and future directions of existing modular autonomous driving systems in on-road and racing scenarios. Section 9 presents a case study focused on the integration and combination of state-of-the-art methods in the creation of autonomous driving systems. Finally, the conclusions of this systematic review are presented in Section 10.

2. Background

This section provides an overview of autonomous driving, one of the fastest-growing technologies enabling vehicles to navigate and operate autonomously. The Society of Automotive Engineers (SAE) has established a taxonomy for autonomous vehicles, standardized under the “SAE J3016 Levels of Driving Automation”. This framework defines six levels of driving automation, ranging from Level 0 (complete manual) to Level 5 (fully autonomous) [8]. Figure 1 provides a visual summary of these levels of automation, clearly understanding the progression from manual to fully autonomous vehicle control.

The scientific and industrial communities have recently focused on investigating Level 5 automation, the highest level of SAE taxonomy. This level represents fully autonomous vehicles operating without human intervention in all driving conditions. Achieving this level of automation is considered a critical milestone in the advancement of autonomous driving technology.

2.1. On-Road Autonomous Vehicles

The literature shows that the investigation focuses on the quality and maturity of self-driving vehicles from the perspective of road and race scenarios. However, the concept of an autonomous vehicle appeared for the first time in 1925, when Francis Houdini presented a remotely controlled prototype that traveled 19 km/h between Broadway and Fifth Avenue. Between the 1980s and 1995, Dickmanns and Zapp [9] integrated computer systems into vehicles, allowing speed control of up to 63 km/h on stationary roads. Over time, autonomous vehicles have progressively advanced, incorporating a variety of approaches.

The autonomous vehicle challenges were initially organized as the Grand and Urban Challenges in 2004 and 2007, respectively. The Defense Advanced Research Projects Agency (DARPA) launched the Grand Challenge to spur innovation in unmanned ground vehicle navigation. DARPA organized various competitions successfully and won millions of dollars by demonstrating industrial interest and improving the quality of autonomous vehicles [10].

2.2. Autonomous Racing Cars

There are three autonomous racing car platforms—F1/tenth, Formula Student, and Formula 1; a detailed discussion is presented in Section 6.3. The first autonomous racing car competition took place in 2006. It was designated as the first competition in which vehicles are fully autonomous on large-scale racing tracks, considering Formula F1. The teams in this autonomous racing challenge are the world’s leading university research groups. The winner completed the race, reaching a top speed of 124 km/h. Roborace is organizing several series of autonomous races; the most recent series of races was held from 2020 to 2022, including the Beta season and DevBot2.0. This automobile championship provides a platform for technological advancement capable of accelerating the commercialization of fully autonomous vehicles and implementing advanced driving assistance systems (ADASs) to increase safety and performance. In the 2020 competition, the TUM Autonomous Motorsport team from the Technical University of Munich won the challenge, securing a prize of USD 1 million and achieving an average track speed of 218 km/h [11].

2.3. Modular System

Although modular structures have been extensively studied, our contribution focuses on bridging these established architectures across distinct domains—public roads and high-speed racing environments—as well as critically evaluating the transferability of techniques such as perception and planning across contexts.

The modular system is the leading autonomous driving procedure, providing three different modules.

Perception involves gathering information and extracting relevant information from the vehicle’s environment. Perception can be divided into two categories: (1) environmental perception, which involves understanding the context of the environment, such as identifying obstacles, detecting traffic signs, and performing semantic data classification, and (2) localization, which pertains to the vehicle’s ability to determine its position relative to the environment as well as mapping and tracking the path for maneuvers. Perception provides essential input for planning modules.

Planning involves making deliberate decisions to achieve higher-order objectives, guiding the vehicle from a start location to a target location while avoiding obstacles and optimizing based on specific heuristics. Path planning is often divided into global planning (route planning) and local planning, which encompasses trajectory and behavioral planning.

Control refers to the vehicle’s ability to carry out the actions defined by higher-level processes. This includes lateral and longitudinal control, such as managing acceleration, braking, drifting, and steering.

Figure 2 illustrates an autonomous driving pipeline, comprising hardware such as sensors, computing devices, servers, buses, and connectivity, as well as software components such as operating systems, a modular system, safety, and simulations.

3. Related Work

Scopus data (TITLE-ABS-KEY (( “autonomous driving” OR “autonomous vehicle” OR “autonomous racing”) AND (“perception” OR “planning” OR “control” OR “safety”))AND (“machine learning” OR “deep learning” ) AND PUBYEAR > 2019) indicate a substantial growth in publications on autonomous driving since 2019, with an exponential rise in recent years (Figure 3), reflecting the increasing focus on autonomous driving within transportation systems. This surge signifies growing attention from academic and industrial sectors toward the research and development of autonomous vehicle technologies as they become central to future mobility.

Several studies have examined the integration of deep learning techniques in autonomous driving, focusing on key tasks such as perception, planning, control, and safety. These reviews consistently emphasize the central role that deep learning plays in improving the performance and reliability of autonomous systems.

To provide a structured and insightful overview of existing surveys, the literature review is organized thematically across core components of autonomous driving systems.

3.1. Perception-Focused Surveys

Perception remains a fundamental module, often addressed through object detection, segmentation, localization, and sensor fusion. Morooka et al. [13] presented a comprehensive review of deep learning applications in various components of the modular system in autonomous driving. The study focused on advances in perception (such as object detection, localization, and segmentation), planning (including trajectory generation and prediction), and control (covering steering and speed optimization). It highlighted the significant role deep learning plays in improving the accuracy and robustness of each component, demonstrating its growing impact on improving the overall performance of autonomous driving systems. Golroudbari and Sabour [14] focused on improving autonomous driving systems by analyzing deep learning approaches applied to specific components of the modular system. These included obstacle detection, sensory perception, behavior modeling, lane detection, navigation, and path planning, all aimed at enhancing the overall performance of autonomous driving systems. Specific reviews focused on perception [15,16,17,18], emphasizing deep learning techniques for tasks such as object detection and semantic segmentation. LiDAR-based perception surveys were found in [18,19,20,21], while SLAM-based perception reviews were discussed in [22,23].

3.2. Planning and Trajectory Prediction

Planning has been extensively surveyed, integrating both machine learning and deep learning techniques. Reviews such as [24,25,26,27] evaluated planning systems based on trajectory generation, prediction, and behavioral reasoning under dynamic road conditions. Bachute and Subhedar [28] presented a study with a broader scope, with an in-depth exploration of self-driving technologies, using a combination of deep learning and machine learning techniques. They compared the performance of existing methods by analyzing key metrics such as mean intersection over union (mIoU) and average precision, among others. Their review focused on specific components of the modular system, including motion planning, lane detection, localization and mapping, pedestrian detection, traffic sign detection, auto-parking, security, and fault diagnosis.

3.3. Control Systems

Control modules, including lateral and longitudinal vehicle control, have been extensively studied in surveys like [29,30], which incorporate both classical and learning-based control strategies.

3.4. End-to-End Learning

Several reviews focus on end-to-end autonomous driving architectures where deep neural networks learn the entire pipeline from perception to control. These were examined in [31,32,33], covering both imitation learning and reinforcement learning frameworks.

3.5. Safety in Autonomous Driving

Safety-related survey articles analyze risk mitigation, redundancy, and robustness. Jing Ren, Raymond Ning Huang, Jing Ren [34] reviewed deep learning techniques by applying various aspects of autonomous vehicles, particularly fault diagnosis, data security, and vehicle communication systems. The authors highlighted how deep learning has become a cornerstone for optimizing vehicle-to-vehicle (V2V) and vehicle-to-everything (V2X) communication. Broader safety-oriented discussions were found in [35,36,37,38].

3.6. Large Language Models and Vision-Language Models

The emerging literature highlights the role of large language models (LLMs) and vision-language models (VLMs) in autonomous systems. Li et al. [39] reviewed how LLMs support human-like reasoning in self-driving. They investigated both modular system pipelines and end-to-end systems, concluding that LLM models enhance decision-making, perception, and human–machine interaction in autonomous vehicles. The LLM4Drive survey [40] explored planning, perception, and question-answering modules enhanced with LLMs. Multimodal models (MLLMs) were covered in [41], focusing on integration with motion planning and perception. Zhou et al. [42] analyzed VLMs in perception, decision-making, and scene understanding. These studies marked a shift toward knowledge-driven and context-aware driving systems. However, despite their potential, LLMs and vision-language models present significant limitations.

Although LLMs and language-vision models have shown promise in improving semantic understanding, decision-making, and contextual awareness in autonomous systems Cui et al. [43], Zheng et al. [44], several limitations require critical consideration. These models often inherit biases from their training data, which may reflect geographic, cultural, or environmental biases. In the context of driving, this can result in inadequate generalization to road conditions, signaling styles, or underrepresented human behaviors.

In addition, the integration of LLMs raises ethical and operational issues, including how decisions made by the models can be interpreted, verified, or certified for safety-critical applications. However, issues such as data privacy, accountability in failure scenarios, and explainability of language-based decisions still pose major challenges.

Furthermore, LLMs and VLMs are computationally intensive, which limits their real-time applicability without significant model compression or dedicated hardware optimization.

3.7. Simulation Modalities in Autonomous Driving Research

Simulation environments constitute an important basis for the development and validation of autonomous driving systems. The existing literature distinguishes between several simulation modalities, each of which offers different degrees of realism and interaction.

Fully simulated environments include computer games or virtual worlds where both the vehicle and its surroundings are entirely synthetic. Several reviews have examined the use of these environments for training and benchmarking AI agents, especially in early reinforcement learning experiments. For example, Kaur et al. [45] provided a comprehensive survey on simulators for testing self-driving cars, discussing different simulation platforms and their applications in autonomous vehicle development. More recently, Ref. [46] examined several mainstream simulators, detailing their features, including accessibility and underlying engines, and presented a structured overview of various open-source datasets. They also investigated notable virtual competitions, highlighting their reliance on simulators and datasets.

In a semi-simulated framework scenario, real-world driving footage is used as input, and an AI agent attempts to make decisions based on the video stream. This setup has been discussed in research exploring imitation learning and video prediction for decision-making. Shen et al. [47] presented Sim-on-Wheels, a framework that integrates virtual events into real-world driving scenarios, enabling safe and realistic testing of autonomous vehicles.

Semi-physical environments involve scaled-down physical environments where small vehicles equipped with real sensors drive on miniature tracks. Reviews in this area typically focus on simulation-to-real-world transfer, sensor calibration, and domain adaptation. Caleffi et al. [48] provided an overview of small-scale self-driving cars, summarizing current autonomous platforms and focusing on software and hardware developments in this field.

In real-world physical systems, full-scale autonomous vehicles operating in real-world road conditions represent the highest level of fidelity. Several studies have focused on real-world deployment challenges, including sensor fusion, system safety, and real-time performance. Fremont et al. [49] explored formal scenario-based testing approaches that span both simulation and real-world environments, highlighting methods to bridge the gap between the two.

Each modality plays a unique role in the development pipeline. A comprehensive understanding of their trade-offs helps researchers select the appropriate test environment for specific phases of algorithm design and evaluation.

3.8. Comparative Context and Scope

While many existing surveys concentrate on isolated components, our review contributes by connecting these components across two domains—on-road and autonomous racing—highlighting their adaptation to specific context-specific requirements. We aim to bridge traditional modular reviews with a comparative perspective that accounts for scenario-specific constraints.

We provide an overview of the modular system, including perception, planning, control, safety, benchmarks, simulation, and real-time autonomous driving. Figure 4 illustrates the hierarchical structure of modular autonomous driving systems, which offers a conceptual overview of how perception, planning, control, and evaluation modules interact, serving as a foundation for understanding the various methods.

This paper provides a detailed analysis and comparison of the unique challenges and requirements for both road and race autonomous vehicles. This analysis demonstrates how artificial intelligence (AI) technologies can be adapted and optimized for varying automotive environments. Our comparison highlights the adaptability and efficiency of AI algorithms in managing the fast-paced decision-making required in racing scenarios, in contrast to the complex and dynamic conditions faced by on-road vehicles.

4. Materials and Methods

The current systematic review of the literature was conducted based on the preferred reporting items for systematic reviews and meta-analyses (PRISMA) [50].

4.1. Research Motivations

Autonomous driving technology is advancing rapidly, driven by the integration of advanced intelligent systems. Among the various techniques used, deep learning has proven to be highly effective in autonomous driving scenarios. The success of deep learning techniques has led to their widespread use in autonomous driving, which makes it necessary to explore in-depth analyses by examining previous studies. Our study aims to systematically review the proposed deep learning approaches for autonomous vehicles, covering both on-road and racing environments. It seeks to highlight existing deep learning approaches for modular systems, describe benchmarks of simulation and real-time platforms, discuss evaluation metrics, state-of-the-art performances and their comparisons, and outline potential challenges and future research directions. Additionally, we aim to fill what we perceive as a gap in similar studies that often focus on specific modular system components within either on-road or racing car environments.

4.2. Research Questions

This study aims to answer the following research questions:

• RQ1: What are the deep learning approaches used in modular autonomous driving systems on the road and in racing scenarios?
• RQ2: What safety and robustness techniques from machine learning and deep learning are used in autonomous driving on the road and in racing scenarios?
• RQ3: What are the existing datasets used for machine learning/deep learning techniques in autonomous driving?
• RQ4: What performance evaluation metrics are used to evaluate the modular system in autonomous driving on the road and in race scenarios?

4.3. Information Sources and Databases

We implemented a systematic approach to gather the most recent studies, covering the period from 2020 to 2025, from a variety of journals and conferences across multiple database sources, including SCOPUS, Science Direct, IEEE Xplore, Web of Science, ACM, Taylor & Francis, and MDPI. We developed a search strategy to identify relevant literature following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines during all stages MJ et al. [50]. Figure 5 presents the paper screening and selection processes.

4.4. Search Strategy and Key Terms

Our search strategy was customized for three databases using the following search terms: “autonomous driving” OR “autonomous vehicles” OR “autonomous racing” AND “perception” OR “planning” OR “control” OR “safety” AND “machine learning” OR “deep learning” OR “DL” OR “ML”. All searches covered the period from 2020 to 2025 and included journal and conference papers, early access papers, and book chapters.

4.5. Eligibility Criteria and Quality Assessment

The selection of articles to be considered in our study was made according to a set of criteria that prioritized high-quality publications from leading journals and conferences in the relevant areas. The search focused on mapping the existing literature on machine learning/deep learning approaches used in modular autonomous driving systems. The search covered the years 2020 to 2025, excluding all papers with a publication date outside this period. The search was not restricted to specific countries or regions and was of global scope. Only papers written in English were considered. Furthermore, only papers published in the research areas of computer science, engineering (excluding subfields not aligned, such as biomedical engineering, and civil engineering, among others), transportation, automation control systems, robotics, or related areas were included in this study. Short articles, posters, editorial materials, letters, and articles not aligned with the scope of this study were excluded. To ensure the relevance of each selected paper, we critically reviewed the abstracts of each one, focusing only on those directly related to our topic.

Table 1. Inclusion and exclusion criteria.

	Inclusion	Exclusion
Type of Study	Original research papers, review papers, technical reports, data papers	Posters, short papers, editorials, letters
Language	English	Non-English
Countries/Regions	Not restricted	-
Publication Year	January 2020 to December 2025	Pre-2020 or Post-2025
Intervention	Machine learning/deep learning approaches	Other approaches rather than machine learning or deep learning
Scope	Deep learning approaches for modular autonomous driving systems	Other focus rather than deep learning approaches for modular autonomous driving systems

summarizes the inclusion and exclusion criteria used in the selection of research articles.

4.6. Data Extraction

Our search yielded a substantial number of articles from which we extracted key themes, main contributions, discussions, future directions, and conclusions to form the foundation of this systematic review. To achieve this, two of the authors independently and extensively analyzed the 155 articles included in the study.

4.7. Risk of Bias Assessment

To assess potential bias in the included studies, we developed a custom checklist, taking into account best practices for systematic reviews. This checklist considered the clarity of the study objectives, data quality, methodological transparency and consistency, completeness of reporting, and potential conflicts of interest. Each of the 155 studies was independently assessed by two of the authors (K.H. and S.J.), and points of disagreement were resolved through discussion between the authors. The overall risk of bias was classified as low, moderate, or high for each study, according to these criteria.

4.8. Effect Measures

Although we did not conduct a meta-analysis, we extracted available effect measures from the included studies. These included precision, recall, F1 score, mean absolute error, root mean square error, average displacement error, final displacement error, and accuracy. Where applicable, we present these values in the Section 5 for comparative analysis.

4.9. Reporting Bias Assessment

In this systematic review, no statistical tools were used to assess reporting bias. However, we qualitatively considered potential bias by examining the trends in reporting in similar studies and identifying any patterns of selective reporting of results.

4.10. Registration and Protocol

This systematic review was not registered in PROSPERO or any other protocol registry. No formal review protocol was published prior to conducting the review.

5. Results

5.1. Characteristics of the Selected Studies

From the research carried out in previously identified databases, 155 articles were identified, focused on deep learning techniques in modular autonomous driving systems. Of these, 6 were published in 2020, 57 in 2021, 47 in 2022, 17 in 2023, 25 in 2024, and 3 in 2025. Regarding the scope, of the 155 included in the study, 31 focus on the perception module ($20\%$), 48 on the planning module ($31\%$), 31 on the control module ($20\%$), 35 on end-to-end approaches ($22.5\%$), and 10 ($6.5\%$) on safety and robustness issues.

5.2. Synthesis of Results

The results of this research are presented in the Section 6, which is divided according to the scope of the articles that make up this study: (a) articles focused on deep learning approaches for the perception module of modular autonomous driving systems; (b) articles focused on deep learning approaches for the planning module of modular autonomous driving systems; (c) articles focused on deep learning approaches for the control module of modular autonomous driving systems; (d) articles focused on deep learning end-to-end approaches for modular autonomous driving systems; and (e) articles focused on deep learning approaches for the safety of modular autonomous driving systems.

6. Major Findings

In this section, we present an in-depth analysis of the papers included in our study. For each research question, we not only present the findings reached through the systematic review carried out, but also provide a comparative analysis of existing approaches and propose future research directions.

6.1. RQ1: What Are the Deep Learning Approaches Used in Modular Autonomous Driving Systems on the Road and in Racing Scenarios?

For better organization, and given that each of the analyzed articles focuses mainly on one of the modules of the modular autonomous driving system, the answer to research question one (RQ1) is presented in three parts, each of which relates to one of the modules of the modular system, perception, planning, and control. Thus,

Table 2. Comparison of modular system components between autonomous on-road vehicles and racing cars.

Modular System Module	Components	Autonomous On-Road Vehicles	Autonomous Racing Cars
Perception	Sensor Fusion	Tracks pedestrians, vehicles, and road signs using cameras, LiDAR, and radar.	Detects track boundaries, other cars, and obstacles using cameras, LiDAR, radar, and GPS.
	Object Detection	Pedestrians, cyclists, road signs, other vehicles, traffic signals, and weather conditions.	Track conditions such as turns, barriers, and opponents.
	Lane and Road Recognition	Road lanes, lane markings, road edges, and free space to drive.	Track boundaries, racing lines, and potential off-track hazards
Planning	Path Planning	Manage trajectory, traffic rules, collision obstacles and road geometry.	Optimal trajectory, avoiding collisions and overtaking at high speeds.
	Decision-Making	Safe driving decisions include stopping at signals, changing lanes, and walking to pedestrians.	Aggressive decision-making, such as high-speed overtaking, defending positions, and responding to track conditions.
	Speed Control	Balances fuel efficiency and safety based on traffic, road conditions, and speed limits.	Maximizes speed while maintaining control, particularly during cornering and on straights.
Control	Steering Control	Smooth steering for safe navigation, considering curves, traffic, and road signs.	Makes precise, rapid steering adjustments to maintain the optimal racing line.
	Throttle & Braking Control	Adjust throttle and braking to maximize fuel efficiency and safety, including emergency braking.	Precise throttle and braking to optimize lap times, especially during sharp corners and accelerations.
	Stability and Traction Control	Ensure vehicle stability in varying weather and road conditions, minimizing skidding and loss of control.	Maximizes traction, especially during cornering, to maintain grip and minimize oversteer/understeer.
	Vehicle Safety	Monitors vehicle health and adapts for comfort and fuel efficiency.	Monitors vehicle performance in real-time for performance optimization.

provides a detailed comparison of perception, planning, and control components for road versus race vehicles, highlighting how each domain presents distinct challenges and system demands.

6.1.1. Perception

Perception is the initial module of the modular system that is based on sensory data. Sensors contain cameras, radar, LiDAR, and the global positioning system (GPS). Table 2 illustrates components concerning perception for both types of vehicles, including object detection, lane or road recognition, boundary prediction, and obstacles or opponents detection, which are captured by the sensors. Here, we focus on the existing methodologies of the perception module, including sensor data based on 2D and 3D. These sensory data are crucial in autonomous driving since they detect, recognize, and segment obstacles surrounding vehicles. For example, CNNs are widely used for object detection and segmentation across 2D and 3D inputs in both on-road and racing scenarios [8,51]. Therefore, autonomous driving has been used in various urban, rural, and highway environments. During these environments, multiple studies have been conducted on the different instances surrounding autonomous driving, including traffic congestion, blocks, pedestrians, speed congestion, traffic signals, and other obstacles. In contrast, the perception module for autonomous racing cars follows specific tracks to detect and segment objects, racetrack lanes, and opponents at high speeds.

• On-Road Perception Techniques

This section elaborates on state-of-the-art perception methods in autonomous on-road vehicles, including object detection, segmentation, depth estimation, localization, and mapping using deep learning models. Table 2 lists the main perception components.

Object detection methods are typically categorized into one-stage (e.g., SSD, FCOS, YOLO) and two-stage (e.g., Faster R-CNN) CNN-based approaches. One-stage detectors such as the YOLO family are widely used for real-time applications due to their speed and efficiency [52,53]. Despite detecting small objects, newer models such as YOLOv5 [54] and YOLOv8 [55] show significant accuracy improvements. Jia et al. [56] proposed a structural re-parameterization technique in YOLOv5 to enhance the detection of small traffic elements.

Low-light scenarios addressed by [57] utilized LLE-UNET to improve image clarity and object detection, enhancing decision-making for road elements. Other advances integrate large language models (LLMs) to convert sensor data into natural language prompts, improving contextual reasoning and decision-making [58,59]. Vision-language models (VLMs) such as CLIP [60] and VLM-Auto [61] further support scene interpretation and driver-assistance systems (ADAS).

Three-dimensional (3D) object detection has gained traction using data from LiDAR and stereo cameras. BEVDetNet [62] employs a bird’s-eye view and semantic segmentation for low-latency performance. Multistage approaches such as Faster R-CNN integrated with feature pyramids improve the detection of occluded and small-scale objects [63]. Other notable models include Stereo-CenterNet [64], RCBi-CenterNet [65], and Pseudo-Stereo [66], which use stereo geometry and virtual views.

Sensor fusion approaches, such as CameraRadarFusion-Net [67], combine radar and camera data to enhance detection across wide fields of view. The combination of CNN and DRL [68,69] has been employed to enhance environment sensing, tracking, and navigation.

For segmentation, dense feature pyramid networks [70] and multi-scale FPN [71] have been used to delineate roads and detect clusters of pedestrians. Semantic models such as GCNet [72] and A3T-GCN [73] are used for understanding urban scenes and predicting traffic flow.

In SLAM, deep learning enhancements include Siamese-YOLOv5 [74] for loop closure and 4Seasons [75] for robust localization under varying weather. Hybrid-dense SLAM approaches [76] merge sparse and dense mapping techniques.

• Racing Car Perception Techniques

In this section, we focus on various approaches to the perception of autonomous racing cars, comprising deep learning algorithms used to detect racetracks, track boundaries or lanes, and identify opponents, as specified in Table 2.

Formula One-based racing cars have recently been progressively promoted in different studies and innovations. Zhang [77] explored autonomous racing, focusing on perception and reinforcement learning in two phases. The first part describes the perception stack using RADAR and LiDAR-camera systems to detect and track race cars, achieving high accuracy and speed. The second part investigates the improvement of a reinforcement learning agent for simulated autonomous racing, along with the incorporation of long short-term memory (LSTM) networks and soft actor-critic algorithms to enhance the agent’s driving capabilities.

Balakrishnan et al. [78] detected and recognized objects using global features, contours, shape matching, and color analysis using matching algorithms and convolution layers, and their approach appears in an F1-based simulator by avoiding obstacles and opponents. Betz et al. [11] conducted a study on perception that utilized various algorithms by integrating sensor fusion techniques, e.g., camera, LiDAR, and radar. First, they performed preprocessing using conditional removal, Voxel downsampling, and ground filtering (employing the Ray ground filter based on Autoware). Teeti et al. [79] emphasized the impact of adverse weather conditions, using cycle generative adversarial networks (CycleGANs) alongside various deep learning methods, including tiny-YOLOv3, Scaled-YOLOv4, and EfficientDet to support object detection.

However, in Formula Student-based autonomous racing, real-time cars are utilized on racetracks, guided by strategically placed cones.

TraCon [80] used an innovative methodology for cone detection that uses YOLOv5 combined with CSPDarkNet for feature extraction and PANet (feature integration).

Strobel et al. [81] investigated depth prediction and clustering by incorporating YOLOv3, ResNet, and ReKTNet with the Perspective-n-Point method for sensor fusion. Large et al. [22] analyzed and compared EKFSLAM and GhaphSLAM for SLAM. The performance of GraphSLAM outperformed that of EKFSLAM. zheng Peng et al. [82] provided a novel method for SLAM and navigation that utilized open sensor-fusion-based sources in YOLOv3-tiny (cone detection), vector regular registration (positioning), and graph factor (optimization).

Ref. [83] introduced a multimodal sensor fusion and object tracking method designed for autonomous racing, focusing on scenarios that require high speed and low latency.

• Comparative Analysis and Challenges

Perception takes data from the different sensors individually or together (fusion), which investigates the perception part using various approaches and assesses performance using multiple evaluation metrics. Furthermore, each approach reveals its advantages and disadvantages during the investigation. In this section, we illustrate the limitations of existing studies in both scenarios.

The perception of on-road vehicles includes various components. These components have been investigated progressively, incorporating deep learning techniques, and some studies need to be further improved. Therefore, Ref. [84] compared stage one and two models with various feature extractors, while the two-stage model outperformed in detecting tiny or occluded objects. However, two-stage detectors are slow in real-time and struggle with dense traffic, adverse weather, and poor illumination. The detection of tiny or lightweight objects in real-time outperformed in static environments but faced challenges in dark or adverse weather conditions [54,55,56]. Fast and accurate object detection is another challenge in auto-driving scenarios, and this has been investigated extensively. However, it faced difficulties in reducing the sample size of the train, the model parameters, the adjustment of the parameters and the flooring, adverse weather conditions, and computational complexity [52]. Mohapatra et al. [62] improved object detection in dynamic environments like diverse weather and light by using LiDAR point clouds, but faced challenges in detecting small or distant objects. Stereo CenterNet [64] performed better in detecting 3D objects by estimating depth, but struggled to detect distant and blocked objects in dynamic environments. The model excelled in marking road indicators with different colors but struggled with segmenting faded markings and required significant computational resources [70]. The approach detected and segmented pedestrians from dense crowds, demonstrating robust performance. However, it requires significant computational resources, faces challenges in real-time detection, and struggles with varying postures of pedestrians and extreme occlusions [71].

Autonomous racing perception incorporates various components that have been progressively investigated using deep learning techniques, although some areas still require further improvements.

Teeti et al. [79] researched the robustness of various object detection methods, improving performance in diverse weather conditions, but also faced computational overhead due to the augmentation process. Optimization of the factor graph based on sensor fusion achieves highly accurate and reliable vehicle odometry, enhancing localization precision and robustness in dynamic environments. However, it still struggles with sensor synchronization and failure and requires fine-tuning [82].

6.1.2. Planing

Planning techniques are responsible for planning a path while driving from source to destination, categorized into global, local, and behavior planning, as described in Table 2. These categories have been evaluated by incorporating various approaches, including classical, sample-based, graph-based, interpolating curve-based, optimization-based, and deep learning-based methods. Deep and reinforcement learning approaches, particularly RNNs and DRL, have been adapted for trajectory prediction and decision-making in both urban and high-speed racing contexts.

As stated, this paper focuses on state-of-the-art deep learning approaches for trajectory prediction, trajectory optimization, lap time, minimum curvature, decision-making, overtaking maneuvers, etc. The distinction between autonomous racing cars and on-road vehicles in planning modules lies in the fact that racing cars are designed to maintain planning at high speeds, while on-road vehicles must navigate various types of environments, such as urban, rural, and highway settings.

• On-Road Path Planning Techniques

In this section, we investigate the autonomous vehicle planning stage, which presents significant challenges, including local and global planning methods, focusing on deep learning and reinforcement learning techniques. Ref. [85] investigated complex traffic scenarios that integrate driver-style inference with deep reinforcement learning to improve safety and efficiency. Thus, a variational autoencoder (VAE) based on gated recurrent units (GRUs) is employed to extract driver-style features from historical trajectory data and is used to train a motion planning strategy within reinforcement learning. This research explores the enhancement of the efficiency of interaction-aware trajectory planning for autonomous vehicles. Ref. [86] introduced a knowledge distillation approach, training a smaller network to mimic the predictive capabilities of a larger network by reducing computational costs without sacrificing accuracy, ensuring that the distilled network significantly accelerates trajectory optimization. Hui et al. [87] used an encoder–decoder for trajectory prediction and also worked on trajectory correction using a DNN. ResNet has been used for trajectory prediction, reducing the gradient vanishing problem, and ResNet34 has been used to estimate the trajectory [88]. TridentNet [89] is a conditional variational autoencoder for trajectory generation. VTGNet [90] is a vision-based trajectory generation with two phases: feature extraction for trajectory planning using MobileNet with fully connected layers, and LSTM for trajectory prediction and generation. LSTM-based models have been employed for trajectory prediction and generation [90,91]. Zhang et al. [92] predicted trajectories sequentially with specific time intervals. Each interval creates a trajectory that builds on previous trajectories by supporting adversarial robustness and attack. F-Net [93] combines RNN and CNN to estimate trajectories using fusion-based data. Trajectory prediction utilizes a monocular image DNN with YOLOv3, combined with the extended Kalman filter, to determine angular and linear velocity [94]. LSTM based on spatial–temporal attention has been used to estimate the trajectory of historical trajectories within a specific interval to be ranked between the nearest vehicles [95]. Greer et al. [96] employed the auxiliary loss function in ResNet-50 to predict multiple trajectories and paths. The trajectory predictions exhibited various uncertainties when using deep learning, and the proposed PRIME approach feasibly and reliably predicted trajectories by employing a model-based method as a generator. In addition, it used learning-based models to choose the trajectories as evaluators [97]. TrajVAE [98] incorporated a generative adversarial network (GAN), a variational autoencoder, and LSTM models by generating trajectories. GANs with features of temporal logic-based syntax have been utilized to predict the trajectory [99]. Attention-based interaction-aware trajectory prediction (AI-TP) [100] consolidated the generative attention network (GAT) with ConvGRU to predict trajectories.

Graph-based spatial–temporal convolutional networks [101] have been utilized to predict and estimate trajectories within sequential temporal contexts, incorporating a graph convolution network (GCN). SCOUT [102] is a graph-based trajectory prediction that supports the attention-based graph neural network (GNN); this method organizes social consistency during trajectory prediction and maintains vulnerable other users in a specific manner. Hierarchical-GNN [103] estimates the trajectories of vehicles by analyzing the interactions and relationships between multiple maneuvers. Li et al. [104] explored improving decision-making in self-driving cars using deep reinforcement learning, addressing complex scenarios and efficient data use. They implemented the DRL-EPKG model and combined human driving knowledge with a soft actor-critic framework, which determines both vertical and horizontal vehicle behaviors. A novel behavior-aware trajectory prediction model (BAT) [105] considers the behavior, interactions, priorities, and positions of surrounding vehicles, informed by traffic psychology and human behavior, providing a robust and efficient solution for predicting vehicle trajectories, improving safety and reliability. Ref. [106] utilized the lightweight SqueezeNet model for human-like trajectory planning by imitating human driving behavior and planning efficient and safe trajectories. Ref. [107] introduced a human-like planning method for navigating lane changes and intersections. This method mimics various human driving styles to emulate realistic and safe human-like driving behaviors. vision-language-planning (VLP) [108] enhances autonomous driving systems (ADS) by integrating the reasoning capabilities of language models that incorporate agent-centric and self-driving-car-centric learning paradigms to improve scene understanding and decision-making, demonstrating VLP’s superiority in open-loop planning, perception, prediction, and generalization. DriveLLM [43] integrates LLMs to improve decision-making with common sense reasoning by addressing challenges in traditional rule-based systems. DriveLLM incorporates cyber-physical feedback for continuous learning and improved performance in complex scenarios, enabling interaction with human input while protecting against adversarial attacks. “Drive as You Speak” [109] explores the integration of LLMs to create more human-like interactions, allowing natural language communication, contextual understanding, and continuous learning in autonomous vehicles. It investigates the potential of LLMs for improved reasoning, personalization, and transparency in driving, demonstrating enhanced decision-making and motion-planning capabilities.

Here, we illustrate various state-of-the-art approaches for planning that incorporate reinforcement learning. The Markov decision process (MDP) and DRL have been investigated for navigation and path planning, respectively [110]. Xu et al. [111] used an actor-critic approach for trajectory generation and decision-making in trajectory planning to achieve optimal policies.

Longitudinal trajectory planning was implemented using the Adam optimizer and deep Q-learning, formulated with MDP [112]. Luis et al. [113] introduced multi-agent trajectory planning using centralized convolutional deep Q-learning (DQL) and a modified reward function for rewards and penalties. Hierarchy-based reinforcement learning has been used for trajectory planning by supporting LSTM for historical monitoring, and a hybrid reward has been used to improve performance [114]. Reinforcement learning and multi-agent reinforcement learning (MARL) methods have been used to generate trajectory optimization [115] and trajectory planning [116], respectively.

• Racing Car Planning Techniques

Here, we discuss the state of the art in planning autonomous race cars that incorporate deep learning techniques.

Thus, Jain and Morari [117] predicted the trajectories to solve the waypoint and optimal lap time by maintaining the limit handling in high-speed racing vehicles, incorporating Bayesian optimization. Trajectory planning during interactive overtaking that incorporates conventional methods to rely on predicting the behavior of other vehicles proposed a novel RL-based approach to generate trajectories, and a safety layer intervenes if infeasible trajectories are generated, selecting a safe and suboptimal alternative [118]. An unscented Kalman filter combined with a Bayesian estimator was used to address the uncertainty in predicting future trajectories in LUCIDGames [119].

Mix-Net [120] is a structural deep learning method based on the LSTM encoder–decoder to estimate the trajectory, position, and motion of the opponent cars.

The generation of a trajectory from unseen data using reinforcement learning managed computationally outperformed the unmatched parameter method compared to the traditional method [121]. Weaver et al. [122] presented a novel method for real-time trajectory generation in autonomous racing, employing dynamic movement primitives (DMPs). This approach allows racing cars to control deviations that handle accelerating targets, which is crucial for precise control at handling limits. Evans et al. [123] introduced a modification planner that identifies paths by avoiding obstacles and adapts to obstacles using deep reinforcement learning to adjust waypoints when blocks occur The deep deterministic policy gradient (DDPG) has also been employed to capture short-term trajectories with MARL and a linear-quadratic Nash game approximation for low-level trajectories or reference points [124]. RaceMOP [125] is a novel mapless online path planner for multi-agent autonomous racing that uniquely combines an artificial potential field (APF) planner with residual policy learning (RPL) to enable long-horizon planning and robust collision avoidance during overtaking.

Feed-forward artificial neural networks (ANNs) have been used to identify optimal trajectories and tap time, outperforming other traditional methods [126]. Traditional methods are needed to prevent uncertainty and other constraints. Therefore, the TOAST method has introduced trajectory optimization and tracking by deploying a feed-forward network [127].

Sim2Real [128] uses reinforcement learning to leverage optimal trajectories and lap time. Reinforcement learning with a proxy reward for progress predicts optimal trajectories and lap time [129]. ResRace [130], based on the modified artificial potential field (MAPF), produces the optimal DRL-predicted navigation policy.

Weiss et al. [131] predicted opponent vehicle future trajectories by incorporating LSTM, which maintains the interaction area during racing. Q-learning has been applied to behavior inspections and overtaking maneuvers with opponent vehicles on both straight and curved tracks. Gaussian process and Stochastic-MPC have been used for interaction-based methods dealing with overtaking procedures. The procedure involves planning and maintaining optimal trajectories [132,133]. Tian et al. [134] propose a balanced reward-inspired proximal policy optimization (BRPPO) algorithm to improve the quality of training and decision-making of autonomous racing vehicles to navigate complex tracks, incorporating reinforcement learning. BRPPO used a balanced reward function that considers both historical and current rewards to minimize collisions during sharp bends.

• Comparative Analysis and Future Research Directions

In order to present an overview of the performance of previous approaches that incorporate deep learning in autonomous driving in road vehicles and racing cars, we include a performance analysis in

Table 3. Comparison of planning methods between autonomous racing cars and autonomous vehicles.

Platform	Reference	Approach	Modular	Performance
Autonomous Vehicles	[85]	VAE + GRU and RL	Trajectory planning	Average route completion degree (100%), number of collisions N is 6, Number of deadlocks is 0, and average running time is 542.56 s
	[101]	Graph-spatial-temporal-CNN with GRU	Trajectory prediction	Average RMSE 1.52 with all vehicles, and Average RMSE 1.49 with one vehicle
	[102]	SCOUT: Attention-based GCN	Interaction, trajectory prediction	Average displacement error (ADE)/final displacement error (FDE), InD dataset: 0.46/1.03
	[104]	SAC and VAE	Decision-making	Success rate outperformed
	[109]	LLMs	Motion planning and decision-making
Autonomous Racing Cars	[118]	RL	Trajectory planning for overtaking maneuvers	Success rate up to 92%, time per planning cycle 1.5 ms
	[126]	ANN: feed-forward neural network	Optimal trajectory and lap time	MAE = ±0.27, and ±0.11 at curvature point, 9000 times faster performance than traditional methods
	[125]	Multi-agent: artificial potential field (APF) planner with residual policy learning (RPL)	Path planning	Collision ratio of IC = 0.33% and IC = 0.42%
	[134]	Balanced reward-inspired proximal policy optimization (BRPPO)	Decision-making to navigate complex tracks	Number of collisions is 0 on all tracks
	[122]	Dynamic movement primitives (DMPs)	Trajectory generation	Mean lap times between the acceleration goal DMP (Mean = 134.75, SD = 0.85) and Velocity goal DMP (M = 136.87, SD = 1.34)

, which highlights how diverse planning approaches cater to the specific demands of autonomous driving in both on-road and racing scenarios. For on-road vehicles, models such as VAE + GRU and SCOUT show strong accuracy metrics (e.g., RMSE, ADE/FDE), which suggest that these approaches are effective in structured, lower-speed environments with frequent decision-making points. However, the reliance on high-dimensional data (e.g., graph-based spatiotemporal models) poses challenges in scalability and interpretability. In contrast, racing-focused methods like BRPPO and ANN emphasize fast decision-making and robustness under high-speed conditions, with metrics reflecting lower collision rates and faster planning cycles. These results underscore a practical challenge. While on-road systems prioritize reliability and accuracy, racing systems demand agility and low-latency response. This contrast points to an opportunity for hybrid planners that can adaptively balance speed and precision based on environmental context and task complexity.

The planning components of autonomous driving on roads have been extensively investigated using deep learning techniques. Furthermore, the encoder–decoder deep neural network accurately predicts trajectories in dynamic environments, but it still faces challenges, such as the need for significant computational resources and the difficulty of dealing with highly unpredictable driving scenarios [87]. The graph-based temporal CNN-based trajectory prediction enhances the accuracy in diverse driving scenarios. It still struggles with complex interactions in highly congested scenarios, and it also needs to extend its training to ensure robust performance in diverse behavior and patterns [101]. Trajectory prediction efficiently incorporates lane-cross behavior and destination points but faces computational resources and adverse driving conditions [91]. TridentNet generates trajectory dynamically using the conditional generative model effectively, still facing the issue of rare and highly unpredictable events and limited training due to limited computational resources [89]. TrajVAE [98] generated the trajectory for dynamic environments but needs to be extended computationally and faces problems in real-time scenarios.

The planning components of autonomous racing cars have been drastically investigated using deep learning techniques. Thus, TC-Driver [121] used reinforcement learning to drive decisions on predicted trajectories, optimize performance by learning robustly, and face challenges when unseen track layouts. Real-time optimal trajectory planning for autonomous vehicles using machine learning algorithms, enabling efficient lap time simulation and improving performance, but faces challenges when involving opponents and sudden change events [126]. Simultaneous trajectory optimization and tracking using shared neural network dynamics effectively balance accuracy and computational efficiency, which enhances real-time performance, but this approach faces challenges with noisy data and limited computational capacity and adaptability [127]. DeepRacing AI [135] employed advanced algorithms for agile trajectory planning, which allows it to navigate racing circuits efficiently, but it faces problems in real-time decision-making.

6.1.3. Control

This section discusses the final module—control systems based on lateral and longitudinal control. Lateral control manages path tracking by regulating steering angles and inputs from lateral controls, which are based on reference lines and track boundaries determined during the planning phase. Likewise, longitudinal control manages vehicle acceleration, braking, and throttle settings to maintain a specific velocity, with inputs based on speed, position, or direction. Autonomous racing cars handle high-speed environments on a specific racetrack by incorporating various methods, including classical controls, Model Predictive Controls (MPC), learning-based, drifting, and optimization controls. Therefore, lateral and longitudinal control is investigated individually or simultaneously to control steering angles, speed, velocity, acceleration/deceleration, position, side slope, and drifting, incorporating deep learning techniques. On-road autonomous vehicles must control and monitor all situations around traffic environments at a limited speed to avoid other obstacles. Due to differing environments, we should investigate the approaches between racing and on-road vehicle controls.

Table 4. Listing and comparison of the control modules within the pipeline of a modular system for both autonomous racing cars and autonomous vehicles.

Platform	Reference	Approach	Modular	Performance
Autonomous Vehicles	[136]	Proposed a human-like neural network	Longitudinal motion control	Control style consistency and convergence rate.
	[137]	VLM	Literal controls	Driving Score (DS) 6.87, route completion (RC) 18.08 and infraction score (IS) 0.42
	[138]	DRL with DQN	Vehicles control in difficult environments	Maximizes success rate with minimizes collision rate
	[139]	CNN with pre-training as well as maintaining overfitting	Lateral control: steering angle estimation	Improve training and generalization to prevent over-fitting
	[140]	Introduced a novel approach—robust adaptive learning control (RALC)	Predicting uncertainties and lateral tracking controls	Tracking performance and errors: lateral deviation and heading errors are evaluated on an eight-shaped track where the adhesion coefficient is = 1.
Autonomous Racing Cars	[132]	Gaussian process model and model predictive control (MPC)	Control safely overtaking opponents	Lateral and longitudinal error are (0.02 and 0.026) means and (0.006 and 0.006) variance respectively
	[141]	Mathematical models	Learning MPC	20th iteration lap time (ILT-20) is 5.0
	[142]	Model-free DRL, Dreamer for Sim2Real	Controls	Lap time on four tracks
	[130]	ResRace: MAPF and model-free-DRL	Control policy	Lap time on five tracks

provides a detailed comparison of control strategies across on-road and racing scenarios, showcasing how different deep learning models handle lateral and longitudinal control under different conditions. In road environments, methods such as robust adaptive learning control (RALC) and human-like neural networks emphasize control precision and smooth trajectory tracking, often reporting low lateral deviation and improved convergence. However, these methods may fail under rapidly changing conditions, where fast adaptability is required. Conversely, racing-focused models like model-free DRL (e.g., Dreamer) and ResRace prioritize lap-time minimization and rapid control response. Although effective for high-speed maneuvers, these approaches often struggle with stability and interpretability. The differences in evaluation metrics, from infraction scores to variance in tracking errors, highlight a broader trade-off—optimizing for safety and interpretability in on-road scenarios versus maximizing performance and agility in racing scenarios. Future research could benefit from hybrid control architectures that take advantage of the reliability of on-road controllers with the adaptability of racing strategies, especially for urban high-speed applications and adversarial settings.

• Lateral and Longitudinal Control in On-Road Vehicles

This section reviews the latest advances in the autonomous control of road vehicles, focusing on longitudinal and lateral control systems supported by deep learning techniques. We explore approaches applicable to various driving environments, including urban, rural, and highway settings, while taking into account maneuvers, obstacles, and complex traffic conditions. Deep learning methods are increasingly being used to optimize vehicle control, enabling systems to manage limit handling and meet the challenges of various traffic scenarios.

Several deep learning techniques have been associated with MPC, showing promising results in autonomous driving. Thus, we summarize some MPC-based deep learning techniques. DDPG combined with MPC has been used to control vehicle speed under poor and inefficient conditions on sidewalks in dynamic traffic conditions, maintaining crowd-sourced information about road conditions and ensuring adequate vehicle performance [143]. The Deep Koopman neural network operator is a data-driven MPC approach that controls dynamic vehicles based on unpredictable and nonlinearities from infinite dimensionality [144]. Linear parameter varying (LPV) is a regression-based learning method used for MPC to control non-linear path and lateral problems [145].

Collecting camera-based data, Yin [139] investigated the estimation of steering angles by incorporating the CNN with a pre-trained network to avoid overfitting. Vision-based neural networks [146] were introduced to control the tracking under uncertainty in group vehicles. The vehicles follow the leader of the group of cars, and the leader vehicle leads the other member vehicles. Salunkhe et al. [147] controlled CAN-bus and data optimization and managed energy consumption, incorporating MLP and CNN neural network methods. Ref. [136] introduced a novel deep learning approach to enhance control as human-like, specifically addressing longitudinal motion, which is designed with a structure that mirrors the driver’s control mechanism, making the model more interpretable and applicable to vehicle dynamics, maintaining and improving performance consistency and convergence rate. The robust adaptive learning control (RALC) approach, introduced based on Lyapunov-like theory, enhances tracking control performance in unpredictable environments. This method also improves tracking performance by considering previous tracking control experiences [140]. CarLLaVA [137] is a novel vision language model designed for autonomous driving using only camera input, which leverages the LLaVA vision encoder in closed-loop driving scenarios. It used a semi-disentangled output representation, combining path predictions and waypoints to enhance literal control. Experiments demonstrate CarLLaVA’s superiority over existing methods, particularly in lateral control. DriveMLM [148] is a framework for autonomous driving that uses LLMs, which bridges the divide between language-based decisions and vehicle control through standardized decision states. DriveMLM employed a multimodal LLM, which uses data from various sensors and inputs to make driving decisions and provide explanations.

Ref. [138] explored DRL to improve autonomous vehicle control, addressing the complexities of diverse and varying weather conditions, using Deep-Q networks (DQNs) to train AVs for challenging conditions such as heavy traffic and adverse weather. Prathiba et al. [149] used deep reinforcement learning and genetic algorithms to manage complex road structures, traffic congestion, safety comfort, and vehicle energy maintenance. DRL has been used to optimize traffic flow and generate or predict new routes to reach the destination in emergency situations [150]. DDPG and PPO have been adapted for continuous control in complex navigation tasks, enabling vehicles to manage unstructured intersections or high-speed overtaking in racing environments [151]. Dong et al. [152] used DRL to manage connected autonomous vehicles, maintain safety and movement, and suggest lane changes. Deep reinforcement learning has also been used to manage the unsignalized intersections of connected vehicles, utilizing a proximal policy optimization algorithm and generalized advantage estimation to ensure the reliability and accuracy of traffic flow [153]. Energy consumption includes fuel consumption, battery backup from source to destination, estimated range, driving rounds, etc. Reinforcement learning is used to control the collision problem in autonomous vehicles by framing collision prevention as a Markov decision problem. This approach utilizes an actor-critic policy to make decisions that ensure the safety of the cars [154]. Traffic-aware autonomous driving (TrAAD) [155] is a novel method used to improve the performance of autonomous vehicles by integrating traffic simulation into imitation learning and optimizing speed control to improve traffic flow and energy consumption, which involves a two-phase training process—the autonomous vehicle learns optimal acceleration through reinforcement learning, and the learned behavior is integrated into a comprehensive autonomous driving framework.

• High-Speed Control Strategies in Racing Cars

First, we discuss autonomous racing car controls, which deal with high-speed vehicles with high accuracy and outstanding performance. The primary purpose of this survey is to address the question “How to control and reduce the lateral and longitudinal error rate during racing on the track?”. We designed control-based approaches by incorporating deep learning techniques. Here, we examine the lateral and longitudinal controls, focusing on steering angles, acceleration, braking, velocity, and other throttle adjustments.

MPC is the most promising strategy for addressing autonomous racing constraints, which, we hope, will perform increasingly well in the autonomous racing world, particularly with exceptionally dynamic models. Ref. [141] introduced a novel method for autonomous racing, which enhances learning model predictive control (LMPC) by focusing on learning the error dynamics. Instead of directly modeling vehicle dynamics, it learns the difference between a nominal physics-based model and real-world data.

The MPC method employs the Model Predictive Path Integral (MPPI) control, supported by a multilayer neural network, to navigate dynamic environments. However, it may perform inappropriately in real-time environments, and model-free-based reinforcement learning is used to optimize the cost function [121]. The Deep Koopman operator [156] is a data-driven, model-free learning method used for MPC problems. It controls the growing dimensionality and produces optimal racing lanes without existing knowledge of the vehicle. Here, a Gaussian process-based stochastic MPC is used for optimal trajectory planning and overtaking during racing [133]. BAYESRACE [157] was invented to investigate the lateral side slip by incorporating a Gaussian process while maintaining a kinematics-based MPC.

Deep reinforcement learning has been used to control velocity to ensure the limit of high-speed autonomous racing cars from uncertainty dynamics and investigate the minimum trajectory and lap time [129,142]. Evans et al. [158] deployed reinforcement learning control for vehicle position, velocity, and rewards to achieve optimal trajectory, reduce curvature, and optimize lap time.

• Comparative Analysis and Challenges

We discuss the performance and limitations of the controls in both scenarios and list some performance of recent studies in Table 4.

Therefore, the components and comparison of both driving scenarios are included in Table 2. Initially, we list the most recent studies on the cons and pros of autonomous driving based on the road. The other reinforcement learning model effectively enabled the avoidance of chain collisions by learning safe and adaptive maneuvers. However, it faces challenges in real-time decision-making under extreme conditions [154]. The deep learning-based control algorithm provides improved decision-making, efficient path planning, and adaptive control, allowing vehicles to navigate complex environments with improved precision and responsiveness [139].

Finally, we list the most recent existing studies on the performance and challenges of autonomous racing in control modules.

Salvaji et al. [159] investigated using deep reinforcement learning (DRL) to control an autonomous Formula SAE, trained reinforcement learning algorithms in simulation, exploring the challenges of applying RL to autonomous navigation, especially the gap between simulation and real-world performance. Two algorithms, DQN and TD3, were benchmarked. The Gaussian process model is effectively used to predict opponent movements and provide probabilistic estimates that improve decision-making, but it struggles in dense racing fields [132]. The method designed a signal-specific tailored reward based on improving and promoting faster lap times with precise maneuvers, but it faces challenges like the potential for leading to unintended behaviors if the reward is not carefully balanced [158]. Residual policy learning has led to faster convergence and significantly refined control in high-speed environments, but it faces problems in real-time [130]. The hierarchical control system improves coordination among cooperative teams in competitive autonomous racing by structuring decision-making into layers, optimizing individual vehicle performance and team strategies, and enhancing race efficiency. However, there is a need to improve real-time coordination in highly dynamic and competitive environments [124].

• End-to-end approaches

End-to-end approaches have emerged as alternatives to traditional modular autonomous driving pipelines. These methods employ deep learning models where sensor inputs, such as images, LiDAR, point clouds, etc., are directly mapped to control actions, potentially simplifying the system architecture and improving its latency. This section presents a structured overview of fully end-to-end and partially end-to-end systems, highlighting their applications in road and racing environments. Figure 6 illustrates the architectural differences between fully and partially end-to-end pipelines.

Sometimes, end-to-end approaches overlap in autonomous vehicles in both on-road and racing scenarios. However, the methodologies in both cases differ due to different environments and objectives. A summary of selected approaches across domains is provided in

Table 5. Summary of representative end-to-end approaches for autonomous driving on the road and in racing scenarios.

Platform	Reference	Approach	Modular	Performance
Autonomous Vehicles	[161]	LLMs	Perception to controls	Ability to reason and solve long-tailed cases
	[162]	DSUNet (depthwise separable convolutions)	Planning: Lane detection and path prediction	Static/dynamic MAE: estimated curvature: 0.0046/0.0049, lateral offset: 0.18/0.11
	[44]	VLMs	Perception to planning	Accuracy 66.5
	[7]	ShuffleNet V2: proximal policy optimization (PPO) algorithm with curriculum learning and actor-critic	e2e: perception to controls	Best: Collision rate is 63%, waypoint distance is 2.98 m, speed is 8.65 km/h, total reward is 2025, and timestep is 374.
	[163]	Multimodal LLMs	Perception to planning	Avg L2 (m) 0.29 s
	[164]	Fuses RL with optimization-based methods	Explainable and robust motion planning and control approach	Online Computing time PPO-based in few millisecond(very fast) and optimization-based 120–150 ms (very slow) and Proposed (Hybrid) < 100 ms (acceptable)
Autonomous Racing Cars	[165]	Extreme Learning Machine (ELM)	Planning and control	Optimal lap times 8.46 s, Mean deviation from racing line 0.0832 m and violation time 0.46 s
	[166]	RL	e2e: Perception to trajectory planning	Success rate around 80%
	[5]	Direct policy learning with CNN and LSTM	e2e: Perception to controls (steering angles and throttles)	Average lap time: (84.19, 142.4), at highest speed: (70 mph, 60 mph), competed laps: (65, 50) track1 and track2, respectively
	[6]	CNN, transfer learning, DQN	e2e process for Sim2Real	Lap time: (AUT: 23 s, BRC: 56 s, GBR: 48 s, MCO: 42 s)
	[167]	Imitation learning (IL): IRL	Trajectories to controllers	CBFs-value for safety guarantee

, comparing architectures, performance metrics, and control strategies. On-road systems increasingly incorporate vision-language models (VLMs) and large language models (LLMs), enabling semantic understanding and long-tail reasoning, which is essential for urban driving where interpretability and robustness are priorities. However, models such as ShuffleNet V2 and DSUNet, while accurate in static tasks like lane detection, may underperform in dynamic or edge-case scenarios. In contrast, racing-focused models like extreme learning machines (ELMs) and CNN + LSTM based on reinforcement learning architectures are optimized for speed and responsiveness, excelling in lap time and trajectory adherence, but often at the cost of explainability and safety. These trade-offs highlight an ongoing challenge: aligning architectural simplicity with robustness and transferability. The field may benefit from partially modular approaches that blend interpretability with performance, particularly in sim-to-real transitions and multi-task scenarios.

In the following subsections, a more detailed analysis of the end-to-end approaches proposed for both scenarios is made.

• Fully End-to-End Systems: On-Road Scenarios

Fully end-to-end systems are predominantly used in structured environments such as urban and highway driving. These systems often use convolutional neural networks (CNNs), recurrent layers such as GRUs [168] or LSTMs [169], as well as reinforcement learning (RL) techniques [170,171] to learn control policies from raw sensory data. For example, Anzalone et al. [7] proposed an end-to-end reinforcement learning-based framework using proximal policy optimization (PPO) and curriculum learning to improve driving policies in complex scenarios. Similarly, Agarwal et al. [172] integrated monocular vision with deep reinforcement learning for urban trajectory planning and control, enabling real-time collaboration and collision avoidance. Other approaches include Coopernaut [173], which emphasizes vehicle-to-vehicle perception, and integrates vision and language systems such as SimpleLLM4AD [44], where driving decisions are reformulated as a series of visual question-and-answer tasks. These systems use graph vision transformers and visual question answering (GVQA) to analyze the scene and plan coherent actions. Imitation learning has also been widely applied, for example, in CNN-based behavioral cloning approaches [174], where end-to-end lateral controllers are trained directly from human demonstrations. The authors of [161] have made advancements in this area by incorporating large language models (LLMs) to mimic human-like reasoning and memory in long-tail driving scenarios.

• Fully End-to-End Systems: Racing Scenarios

In racing environments, fully end-to-end systems aim to maximize control performance at high speeds with minimal latency. Wadekar et al. [5] proposed a deep neural network (DNN) to estimate steering, throttle, and braking based on input from vision sensors, simplifying modular planning. Reinforcement learning has proven to be particularly effective in competitive environments. Deep latent competition (DLC) [175] models policy learning within a latent visual space to interact with opponents in racing simulations. On the other hand, Conditioning for action policy smoothness (CAPS) [176] applies soft actor-critic algorithms for smooth policy transitions using visual data, while other approaches use imitation learning for high-speed control with formal safety constraints through control barrier functions (CBFs) [167]. The transition from a simulation environment to the real environment is a relevant topic in the racing context. Methods such as DeepRacer [177], which integrates point cloud and stereo vision with reinforcement learning, aim to bridge the performance gap between simulation and physical environments.

• Partially End-to-End Approaches and Hybrid Methods

Partial end-to-end systems offer a balance between the adaptability of learning-based models and the structure of modular pipelines. These methods typically learn specific segments of the pipeline, such as perception-to-planning or planning-to-control, while retaining rule-based or classical approaches for other components. DeepSTEP [178] introduced a novel end-to-end perception architecture that comprises a self-attention mechanism to leverage temporal information, improving overall efficiency and performance by integrating detection and localization into a single unified pipeline. Hu et al. [164] addressed the challenge of autonomous vehicles merging onto highways from on-ramps and proposed a novel approach that fuses RL with optimization-based methods, leveraging the strengths of both techniques, and achieving a balance between smoothness, computational efficiency, explainability, and robustness.

EMMA [163] is a new end-to-end multimodal model for autonomous driving developed by Waymo. EMMA leverages a multimodal large language model foundation to directly transform raw camera sensor data into driving outputs like trajectories and object detection. The model represents inputs and outputs as natural language, enabling it to perform various driving tasks within a unified language space using task-specific prompts.

SuperDriverAI [179] is an end-to-end autonomous driving system that improves robustness and interpretability by using deep neural networks, processing image data to control steering, throttle, and braking.

DriveVLM and DriveVLM-Dual [180] are novel autonomous driving approaches that harness the power of VLMs to better understand and plan the scene. The core innovation, DriveVLM, employs a unique reasoning approach through scene description, analysis, and hierarchical planning. Recognizing VLMs’ limitations, DriveVLM-Dual has emerged as a hybrid integrating VLM capabilities with traditional autonomous driving pipelines for better spatial awareness and computational efficiency. VLM-AD [181] introduced a novel approach that incorporates VLMs to supervise and improve AD training, which enables AD models to learn richer feature representations, leading to improved planning accuracy and reduced collision rates on the nuScenes dataset, leveraging VLM driving knowledge into end-to-end AD pipelines through high-quality datasets of behavioral text annotations.

Lee and Liu [162] introduced the depth separable convolutional network (DSUNet) to determine the lane of the roads and predict the path involving CNN and UNet. UNet is a convolutional-based encoder–decoder network for semantic segmentation.

Three-dimensional (3D)-LiDAR-based point clouds and 2D-camera-based images were used for trajectory planning, where point cloud images were converted into grayscale images using gradient transformation. Both data images on similar scales used attention-based autoregressive-GRU to predict trajectories [182]. Conditional DQN with fuzzy logic as an end-to-end approach has been used to predict the planning of the directional trajectory by supporting global route planning. It also maintains planning commands such as steering angles and throttles [183]. TransFuser [184] is an end-to-end imitation learning model invented for multiple sensor-fusion converted to the same dimension to perform in high-dynamic and complicated environments, controlled like multi-directional traffic congestion and communication.

Kalaria et al. [165] proposed a novel framework that uses an extreme learning machine (ELM) to learn and adapt the vehicle tire model online to operate at their handling limits to achieve optimal lap times. DeepRacing [185] has been introduced to capture trajectory predictions from a vision-based perspective. It is a partial end-to-end approach from the sensor to planning. Mammadov [166] focused on an end-to-end approach using raw lidar data and velocity information using RL. The RL agent’s ability to navigate without prior map information contrasts with traditional methods that rely on precise mapping and path planning. The gradient-free optimizer has been used across various domains of high-speed autonomous racing, such as planning and control. This method has been extensively used for trajectory planning and behavior planning [186]. Optimization of the soft actor-critic-based trajectory and overtaking maneuvers from vision-based input is also a partially end-to-end method [187].

• Comparative Analysis and Challenges

End-to-end and hybrid systems present contrasting trade-offs. Fully end-to-end approaches offer optimized integration and potential performance gains, especially in structured environments. However, they often lack interpretability and require extensive training data. In contrast, partially end-to-end and hybrid models provide systems with greater modular clarity while selectively leveraging learning-based adaptability. These methods are particularly well-suited for scenarios that require explainability, certification, or domain adaptation. Despite significant progress, challenges remain. Although fully end-to-end systems often address rare driving events, hybrid methods can suffer from interface incompatibilities between learned and rule-based components. In this regard, it is important to continue research on simulation-to-real-world transfer, uncertainty estimation, and hierarchical control frameworks to address these limitations and move toward building robust and generalizable autonomous systems.

6.2. What Are the Safety and Robustness Machine Learning/Deep Learning Techniques Used in Autonomous Driving on the Road and in Racing Scenarios?

Safety is the most critical debate in the environment of autonomous driving. Suppose that the researchers are not appropriate for the safety and comfort by organizing the limited handling of vehicles. Thus, technology may be a dangerous zone. However, safety must consider presenting their novel studies, especially in deep learning techniques, since they are based on historical data, which can occur under different constraints, including inadequate interpretation, verification, overfitting/underfitting, etc. Therefore, we review some state-of-the-art studies that have focused on safety.

• Safety in Road Scenarios

In this section, we focus on the safety of autonomous vehicles on the road by tackling several instances, such as other vehicles and their interaction, route or road lane, signal, turning points, pedestrians, vehicle speed, and other technical challenges. Taking into account the safety constraints, the SAE developed a safety level standard that was adopted by the National Highway Traffic Safety Administration (NHTSA), Department of Transportation, USA.

Therefore, we will present some state-of-the-art approaches to handling safety verification by incorporating deep learning. Safety improvements in autonomous driving have been driven by fundamental advances in perception and control algorithms supported by deep learning techniques.

Abrecht et al. [188] addressed the critical safety concerns associated with the use of deep learning, which develops a structured approach for “safety concerns” to systematically analyze and mitigate potential risks arising from the unique characteristics of deep neural networks. Wang et al. [189] proposed using LLMs for intelligent decision-making in behavior planning, supplemented by a safety verifier, which is based on two case studies demonstrating that LLMs can improve driving performance and safety within simulated environments. Chen et al. [190] proposed a novel hierarchical DRL framework that enables autonomous vehicles to avoid collisions, particularly with pedestrians. The system combines a traditional PID controller for path-following with a DRL-based collision avoidance agent activated upon vulnerable road users’ VRU detection.

Autonomous and manual vehicle safety solutions have enabled 5G intelligent transportation systems associated with LSTM and decision-making layers of the probability matrix [191]. Multi-source and multi-outcome-based vehicles safely and progressively predict the direction of arrival and transmission constraints. They estimate geometry-based coordinate limitations using the spare block recovery problem with SBLNet to predict the direction of arrival and transmission to ensure the safety of autonomous vehicles [192].

Deep neural networks have been used to compute the safety level of autonomous vehicles, analyze instances from the on-board unit, and identify breakdowns originating from the on-board unit [193].

Zhu et al. [194] proposed a CNN-based vehicle-pedestrian detection method that extracts several huge features to maintain the exchanges. Squeeze-Net has been presented to extract some other features of traffic by promoting safety interaction. Xing et al. [195] investigated safety within connected automated vehicles (CAVs) performing deep-RNN and LSTM, which predicts trajectories, maintains longitudinal and lateral behavior, and highlights the stage of energy consumption.

• Safety Concerns in Racing Scenarios

We highlight some state-of-the-art safety measures in high-speed vehicles under challenges, incorporating deep learning approaches.

Chen et al. [196] manipulated the camera vision and vehicle speed inputs with reinforcement learning to identify unsafe behaviors. They used the Hamilton–Jacobi (HJ) approach to maintain safety within constrained MDPs.

End-to-end deep reinforcement learning (RL) and imitation learning methods have been utilized in what is known as the deep imitative reinforcement learning (DIRL) approach. It is used on visual data to reliably and accurately identify control policies by validating safety in both simulation and real-time environments [197].

Autonomous racing vehicles have ensured high-speed vehicle controls and safety concerning track boundary changes and, for this purpose, presented the predictive safety filter by supporting the imitation learning method that learns from human expertise and handles non-linear high-dynamic systems [198].

• Safety in Adversarial and Edge-Case Scenarios

While significant attention has been given to perception and planning modules, safety remains an underexplored critical component of autonomous driving systems, with the importance of robustness to adverse conditions and rare driving events being particularly important. Vision-based systems, especially those based on CNNs, have demonstrated susceptibility to adversarial attacks, such as imperceptible changes in inputs that can lead to drastic changes in outputs. Recent studies have proposed input sanitization, adversarial training, and model distillation as mitigation strategies.

In addition, handling edge cases is a concern that must be addressed urgently. Autonomous vehicles must be prepared to respond to scenarios that, although rare, pose a high risk, such as pedestrians crossing the street or sudden lane closures. To address this, systems increasingly incorporate out-of-distribution detection, uncertainty estimation, and scenario-based testing. LLM-enhanced architectures, such as DriveLLM [43] and SimpleLLM4AD [44], offer promising directions by introducing contextual reasoning and generalization capabilities for long-tail events.

Simulation environments and stress test datasets are also being developed to assess safety under unpredictable conditions. However, industry-applicable standards are not yet available. Bridging the gap between model performance and certifiable safety is essential for the real-world deployment of autonomous driving systems.

6.3. What Are the Existing Datasets Used for Machine Learning/Deep Learning Techniques in Autonomous Driving?

Based on our interest in autonomous vehicles, we illustrated the existing datasets used for machine learning/deep learning techniques. These datasets are based on simulation and real-world environments, which are categorized into full-simulators, semi-simulators, semi-real, and real-world. Deep learning learns objectives or environments from historical data, and as a consequence of our intention, investigations will be conducted on existing dataset benchmarks. Those datasets have been explored in previous work, and some datasets have also been developed by considering their objectives.

This section summarizes the datasets for autonomous driving for racing and on-road applications separately in

Table 6. Datasets that are frequently utilized in autonomous driving on-road vehicles.

Datasets	Sensors	Purpose	Developed	Type
CARLA	LiDAR, cameras	Perception, planning, control, sensor fusion, edge case testing	Computer game with game world and AI agents	Fully Simulation
LGSVL	LiDAR, cameras	Perception, localization, V2X interaction, control	Computer game world using Unity, AI agent simulation
AirSim	LiDAR, cameras	Perception, reinforcement learning, control	Game-based simulation using Unreal Engine with drone/car agents
KITTI	LiDAR, cameras	Perception (object detection, SLAM), sensor calibration	Real video data with AI agents making decisions on pre-existing video	Semi-simulation
Waymo Open	LiDAR, cameras	Perception, planning evaluation, tracking	Real-world recordings processed with autonomous AI modules
nuScenes	LiDAR, cameras	Sensor fusion, prediction, 3D object detection	Real-world dataset with annotated scenes used for training AI agents
Duckietown	Cameras, GPS, IMU	Lane detection, control, end-to-end learning	Scale model car with sensors running in a physical, scaled-down town	Semi-Real
Tesla Autopilot	LiDAR, cameras, radar	Perception, autopilot control, real-world planning	Real car with sensor suite collecting data from actual roadways	Real-World
Waymo	LiDAR, cameras, radar	Perception, prediction, planning, localization, control	Autonomous vehicles operating and recording in real environments
Cruise	LiDAR, cameras, radar	Mapping, decision-making, motion planning, control	Real vehicles with AI-driven, collecting real data

and

Table 7. Datasets that are progressively used in autonomous racing cars.

Datasets	Sensors	Purpose	Developed	Type
Sim4Racing	Cameras, IMU	End-to-end driving, high-speed control, reinforcement learning	Game engine simulation with virtual racing environment and AI agents	Full simulation
TORCS	Cameras, wheel encoders	Planning, lane keeping, control, speed optimization	Classic racing simulator used for training AI agents
DeepRacer (Sim)	Cameras, LiDAR	End-to-end policy learning, control	Cloud-based virtual simulation platform for reinforcement learning
FormulaTrainee	Video footage (real track)	Decision-making from video frames, imitation learning	AI agent trained using pre-recorded video of real race tracks	Semi-simulation
DriverNet	Dash cam videos	Lane following, control	Real driving video data
F1TENTH	Cameras, LiDAR, IMU	Planning, real-time obstacle avoidance, racing policy control	Onboard sensors operating in real environment	Semi-Real
DonkeyCar	Cameras, IMU	End-to-end learning, behavioral cloning	DIY scale car platform with sensors trained in physical tracks
Audi RS5 (AutoDrive)	Cameras, radar, LiDAR	Real-time planning, high-speed control, safety-critical navigation	Full-sized car with sensor suite tested on race tracks	Real-World
Indy Autonomous Challenge	Cameras, LiDAR, GPS, IMU	Full autonomy in high-speed race scenarios, perception, planning	Full-size open-wheel race cars equipped with sensors in real competitions
Roborace	Cameras, LiDAR, radar, GPS, IMU	High-speed autonomous racing, multi-agent interaction, real-time planning	Real full-size electric race cars with autonomous control tested in real racing circuits

, respectively.

• Real-Time and Simulations on-road vehicles

Several different real-time simulators and simulations have been developed in some popular industries to ensure that vehicles perform well in the real world dynamically and that humans can easily understand and benefit from the modern contemporary world. These vehicles compete in diverse environments, including maintaining weather conditions day and night and managing urban, rural, and highway environments. Researchers have developed different simulations in the field of autonomous vehicles. When researchers introduce novel concepts in their studies to evaluate real-time cars, which can be very expensive, they often use simulators as a test bed to conduct their studies.

Consequently, we illustrate some real-time autonomous and simulation-based vehicles and hardware and software perspectives. Several sensory input sources have been used to capture the environment of autonomous detection that can be demonstrated in further processing; sensor inputs, including cameras, LiDAR, radar, GPS, IMU, and other sensor fusion types for GPUs, especially NVIDIA, are being used for computing platforms. Table 6 summarizes and compares the real-time and simulator-based datasets.

First, the real-time scenarios are most promisingly used as dataset benchmarks in their research work, such as Kitti (Karlsruhe Institute of Technology and Toyota Technological Institute), Oxford RobotCar, NuScenes, Drive360, Udacity, DAVIS, Cityscapes, HDD (Honda Research Institute Driving Dataset ), BDDV (Berkeley DeepDrive Video), Comma.ai, PandaSet, and so on.

Secondly, some simulators are being investigated during research as datasets, including CARLA, Udacity, CARSIM, Comma.ai, 9-DoF Driving Simulator, GTA-v, rFpro, NGSIM (Next Generation Simulation), etc.

• Real-Time and Simulations racing cars

In 2017, a company in the UK invited a series of autonomous racing. They developed two versions of race cars—Devbot (versions 1.0 and 2.0) and Beta Serious—which have a maximum speed of 180 km/h. The Nvidia PX2 and Speedgoat Mobile Target Machine control the engines of race cars, utilizing several input sources to gather and process environmental information. The sensory devices include front and rear cameras, LiDARs, Radars, and GPS/IMU. These are integrated into a fully electric, Le Mans Prototype (LMP) chassis with rear-wheel drive, as used in Roborace. However, this series has been featured in several competitions, such as the ALPHA (2018–2019) and BETA (2020–2021) series, and has been accomplished through participation from various universities and researchers. Additionally, simulations targeting Roborace have also been developed, including the Roborace simulator and Learn-to-Race (L2R) simulators (handles RL problems). Researchers and companies have also investigated the components of the modular system that can be utilized in Roborace environments.

Now, the most prominent racing competition for full-scale autonomous racing cars is the Indy Autonomous Challenge, launched in 2021. Inspired by the DARPA Series, it features competitions at speeds of up to 290 km/h [199]. These cars have a central computing platform to control the race car engine, which contains a fully electric Indy light chassis, and rear wheel drive power. In 2021 and 2022, Indianapolis Motor Speedway and Las Vegas Motor Speedway organized competitions to compete with a stationary car and an opponent vehicle, respectively [11]. Several groups of university students also participated in the rounds.

However, simulations have been developed for researchers to easily access the IAC environment, such as the Ansys simulator and the LGSVL simulator. Researchers have recently investigated various studies involving simulation to validate their findings. Moreover, other simulators for autonomous racing cars are being utilized as investigation testbeds, including TORCS, Carla Simulator, Gran Turismo Sports, and a hardware-in-the-loop (HIL) simulator. These datasets belong to autonomous racing platforms that use the components of the modular system, which are listed and summarized in Table 7. According to our investigation, most of the existing papers annotated their dataset by considering real-time or simulation tracks during their experiments.

6.4. What Performance Evaluation Metrics Are Used to Evaluate the Modular System in Autonomous Driving on the Road and in Racing Scenarios?

Now, we describe the evaluation metrics utilized to evaluate the modular system of both scenarios in state-of-the-art studies, including mean absolute error (MAE), mean squared error (MSE), root mean square error (RMSE), accuracy, F-measure, precision, recall, the area under the curve (AUC), mean Jaccard, and IOU.

Occasionally, planning has been evaluated based on MAE, MSE, RMSE, lap time, trajectory fit ratio (TFR), mean time between boundary failures (TBFs), distance between boundary failures (DBFs), and tracking errors. The lap time represents the duration it takes for cars to complete a circuit, traveling from the source to the destination, which is represented as follows:

$$T_o=T_a+T_c+T_s+T_b$$

(1)

where $T_o$ represents the optimal lap time during racing, $T_a$ is the period accelerating from the start line to the first corner, $T_c$ is the time taken to navigate the corners around the track, $T_s$ is denoted as the period of the straight portion of the racetrack where maximum speed is achieved, and $T_b$ is the time spent braking before entering corners.

$TFR$ has been used to evaluate the precision of the compared trajectory prediction models, which demonstrate the degree of fit between the predicted trajectory and the real-time trajectory [87], which is mathematically represented as follows:

$$TFR=(1-{\displaystyle \frac{abs\left(a{e}_1\right)+abs\left(a{e}_2\right)+....+abs\left(a{e}_n\right)}{L}})\times 100\%$$

(2)

where $a{e}_i$ is the difference between the $i_{t}h$ output coordinate and its actual value, and L denotes the average length of trajectories.

However, control performance has been examined using several metrics, including MAE, MSE, RMSE, lateral and longitudinal error, heading error, average displacement error (ADE), final displacement error (FDE), learned model velocity optimization (LMVO), and LMVO plus failure prediction and the intervention module (FIM). $ADE$ estimates the actual position of autonomous vehicles within a specific period, which is represented as follows:

$$ADE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{d}_i$$

(3)

where n denotes the total number of positions, $d_i$ is the displacement error at $ith$ positions, and $d_i$ utilizes the Euclidean distance between the actual and desired positions. $FDE$ is used to measure the error between the final position and its corresponding desired position to complete a trajectory; it is also calculated using Euclidean distance [102].

7. Limitations of the Study

The systematic review presented in this article comprehensively summarizes the proposed approaches for modular autonomous driving systems. It primarily relies on research articles collected using a search strategy that prioritized databases such as SCOPUS, Science Direct, IEEE Xplore, Web of Science, ACM, MDPI, and Taylor & Francis databases. Consequently, related articles may have been excluded from this study because they were not found in the considered databases. Furthermore, the search was restricted to publications from the last five years (2020–2025), as well as publications written in English, which also limited the number of articles analyzed. Another limitation is that we mainly used manual screening, without the assistance of automation or AI tools, which may have introduced human error.

8. Real-Time Challenges and Future Directions

This section evaluates the performance, real-time challenges, and future directions of existing modular autonomous driving systems in on-road and racing scenarios.

8.1. Real-Time Challenges for On-Road and Racing Scenarios

In this section, we present the real-time challenges in autonomous driving and provide an overview of the issues faced in both environments.

8.1.1. On-Road Autonomous Vehicles

Initially, many studies focused on specifying pose detection and classification faults to detect tiny objects and adverse weather conditions in real-time [62]. Adapting to unpredictable changes (e.g., sudden obstacles, traffic changes) in the environment creates problems in real-time [14]. Consequently, during auto-driving, making instant and reliable decisions in real-time by ensuring safety and compliance [140,149]. Vehicle-to-vehicle and vehicle-to-everything communications—communication with other vehicles and infrastructures to improve situational awareness [140,152]. Handling unexpected situations and sensor failures presents a challenge in real-time environments [151]. In addition, efficiently calculating the safest and most efficient routes in real-time and managing computational resources are significant challenges [143,151].

8.1.2. Autonomous Racing Cars

First, by maintaining safety and control to navigate at high speeds, racing cars cause challenges in real-time [117]. Thus, it is necessary to decide between dynamic racing and overtaking opponents in real-time face problems [132,133]. Rapid sensor data processing to make immediate adjustments poses a real-time problem [185]. It is challenging to calculate and adjust the trajectory to obtain optimal racing lines in real-time [132,133]. Another critical aspect involves ensuring safety by avoiding collisions with opponents and adhering to track boundaries at high speeds [143,158]. Lastly, it is essential to manage energy consumption and computational resources during races to sustain performance [117,143].

8.2. Research Directions for On-Road and Racing Scenarios

This section outlines future research directions in deep learning for autonomous driving, focusing on both on-road and racing applications. We highlight the challenges of integrating deep learning techniques across the components of the modular system, emphasizing the need for innovation in perception, planning, and control.

8.2.1. Perception

Recent advancements in perception for autonomous driving open up promising research directions. The IS-YOLOv5+VNP model [54] integrates YOLOv5 with versatile network pruning (VNP) to improve the detection of lightweight objects. Future work could extend this approach for real-time applications, especially for detecting smaller objects such as traffic signs and signals. In addition, CycleGAN combined with object detection methods [79] has shown promise in improving performance under adverse weather conditions. Training models with augmented datasets that simulate such conditions could significantly enhance the reliability of detection in challenging environments.

Additionally, GCNet [72] and two-stage object detection models [84] could be further developed by integrating sensor fusion techniques (e.g., camera and LiDAR data). Extending these methods to incorporate temporal data frames may also improve the detection accuracy, particularly for moving objects.

8.2.2. Planning

Planning is critical for autonomous driving, with trajectory prediction and lane detection forming the core of decision-making processes. TC-Driver [121] uses RL for trajectory planning, and combining RL with recurrent neural networks while optimizing regularization through Bernoulli policies could improve real-time performance.

ANN-based trajectory optimization [126] is another area for exploration, particularly for non-stationary autonomous racing, where detecting opponents and optimizing overtaking maneuvers could greatly improve performance. TOAST [127] employs model predictive control (MPC) for optimal trajectory tracking, which could benefit from integrating RL to learn optimal policies through adaptive environmental interactions.

In addition, MixNet [120] introduces interaction-based motion planning, which could be expanded by incorporating graph neural networks (GNNs) to handle complex interactions with surrounding objects. The deep encoder–decoder model [87] could be improved by handling lateral and longitudinal boundaries, and transforming path values into graph representations could facilitate the use of GNNs. AI-TP [100] offers an attention-based approach to multimodal trajectory prediction, which could be refined to handle more complex interactions.

8.2.3. Control

In control systems, deep learning continues to show potential in improving vehicle handling. More advanced deep learning models could improve DRL-based lane change during racing [152] for better efficiency and performance.

As demonstrated by RALC [154], adaptive learning-based tracking control could be further developed using a hardware-in-the-loop (HIL) system for real-time testing. Improving training data quality by focusing on realistic driving patterns could enhance model performance and generalization.

8.2.4. End-to-End Approach

End-to-end learning, which integrates perception, planning, and control, is a growing area in autonomous driving research. Hierarchy-based control [124] uses MARL for high-level planning and low-level control. Future research could focus on integrating both levels into a unified framework using multi-agent approaches.

End-to-end trajectory planning using sensor fusion data in the GRU model [182] offers another avenue to improve generalization. Similarly, DQN with fuzzy logic [183] for motion planning could be extended to include obstacle detection and traffic signal interaction, offering a more comprehensive solution for real-world driving challenges. Aoki et al. [179] addressed the challenges of unexpected road scenarios and cooperation with human drivers, presenting a practical approach based on imitation learning that could focus on extending to handle more practical situations and allowing safe interaction between autonomous and human-driven vehicles.

8.3. Certainty of Evidence

Regarding the assessment of the overall certainty of the evidence on the topics studied, we considered that the findings related to the perception and planning modules presented moderate confidence due to the consistency of the methodologies and the reproducibility of the metrics. In contrast, we understand that evidence on safety and robustness should be classified as low certainty due to the smaller number of studies and the methodological variability existing in them.

9. Integration of State-of-the-Art Modules: A Case Study

To more clearly demonstrate the state-of-the-art methods of different modules and their combinations for creating autonomous driving systems, we built a conceptual modular autonomous driving system based on the main methods identified and analyzed in this study.

• Perception module: The perception stage uses PointPillars [15], a 3D object detection framework capable of efficiently processing LiDAR point clouds for real-time object localization. Its voxelization of point cloud data allows for low-latency obstacle detection, critical for downstream trajectory planning. For visual perception, the use of YOLOv5 [94] allows for robust and lightweight object detection from RGB cameras, allowing redundancy in multisensory fusion pipelines.
• Planning module: For the planning subsystem, we propose the use of SCOUT [96], a spatiotemporal graph-based model capable of predicting trajectories with interaction awareness. SCOUT predicts the future movements of surrounding agents and proposes safe and dynamically feasible paths. Its graph-based structure allows explicit modeling of interactions between multiple agents, making it suitable for complex urban environments.
• Control module: The control layer incorporates robust adaptive learning control (RALC) [134], which ensures stability in the face of modeling inaccuracies and external disturbances. By integrating learning-based disturbance estimation with classical feedback control, RALC is suitable for dynamically uncertain conditions.
• Deployment environment: To test the system, we propose the use of the CARLA simulator [45], which provides several urban, suburban, and highway scenarios. The possibility of simulating sensors (LiDAR, cameras, GPS, IMU) enables realistic sensor fusion, as well as scenario-based system evaluation.

While each module individually demonstrates state-of-the-art performance, integrating them presents challenges. In cases of perception errors, such as adverse weather conditions or occlusions, planning accuracy can decrease significantly. Using a graph-based method for the planning module may result in delays in real-time execution, given the computational overhead that these entail. On the other hand, controllers such as RALC, although robust, depend on accurate and frequent trajectory updates, which can be compromised, for example, by network communication and computational issues. Thus, future research could focus on tightly coupled learning frameworks that jointly optimize perception, planning, and control, in addition to quantifying uncertainty at each stage to mitigate cascading failures.

The modular structure of autonomous systems varies significantly between the road and racing domains, as illustrated in Figure 7. While traffic-oriented architectures prioritize traffic element perception and rule-compliant planning, focusing on safety and rule compliance, racing systems emphasize dynamic road perception and aggressive trajectory optimization, focusing on optimization of performance metrics such as lap time. These differences fundamentally shape the design, evaluation, and integration challenges faced by each application domain.

10. Conclusions

This survey paper offers valuable information by comprehensively analyzing deep learning techniques in both on-road and autonomous racing cars, highlighting their distinct challenges and requirements. On-road vehicles emphasize safety and adaptability in complex, dynamic environments while racing scenarios demand precision, high-speed control, and real-time decision-making.

In perception, deep learning has significantly advanced object detection, semantic segmentation, localization, and sensor fusion, improving navigation in complex environments. State-of-the-art CNN VLM models are frequently used, but challenges remain, particularly in dealing with adverse weather and unpredictable conditions.

Deep learning techniques such as trajectory prediction, generation, optimization, and decision-making have been proven to be effective in planning. Notably, RNNs, generative models, LLMs, VLMs, and reinforcement learning have made crucial contributions. However, real-time decision-making in dense traffic and high-speed racing remains a significant hurdle.

Deep learning has greatly benefited control systems, including steering, throttle, stability, traction, and safety. LLMs, reinforcement learning, and model predictive control have shown substantial promise in both on-road and racing applications. However, the complexity of control in dynamic environments demands further refinement.

End-to-end approaches, which integrate perception, planning, and control into unified deep learning models, have laid the groundwork for fully autonomous systems. However, these approaches often struggle with complex edge cases and maintaining robust safety in real-time scenarios.

This paper underscores critical challenges for future research, including improving the robustness of models in diverse environmental conditions, improving real-time computational efficiency, and strengthening sensor fusion techniques. Addressing these challenges will be pivotal in ensuring the safety and performance of autonomous systems.

Ultimately, this survey encourages ongoing collaboration between academia and industry to advance autonomous driving technologies. By extending the capabilities of deep learning, we can foster new innovations in both on-road vehicles and autonomous racing, enhancing performance and transforming the future of mobility.