DeepReinforcement and IL for Autonomous Driving: A Review in the CARLA Simulation Environment

Abstract

Autonomous driving is a complex and fast-evolving domain at the intersection of robotics, machine learning, and control systems. This paper provides a systematic review of recent developments in reinforcement learning (RL) and imitation learning (IL) approaches for autonomous vehicle control, with a dedicated focus on the CARLA simulator, an open-source, high-fidelity platform that has become a standard for learning-based autonomous vehicle (AV) research. We analyze RL-based and IL-based studies, extracting and comparing their formulations of state, action, and reward spaces. Special attention is given to the design of reward functions, control architectures, and integration pipelines. Comparative graphs and diagrams illustrate performance trade-offs. We further highlight gaps in generalization to real-world driving scenarios, robustness under dynamic environments, and scalability of agent architectures. Despite rapid progress, existing autonomous driving systems exhibit significant limitations. For instance, studies show that end-to-end reinforcement learning (RL) models can suffer from performance degradation of up to 35% when exposed to unseen weather or town conditions, and imitation learning (IL) agents trained solely on expert demonstrations exhibit up to 40% higher collision rates in novel environments. Furthermore, reward misspecification remains a critical issue—over 20% of reported failures in simulated environments stem from poorly calibrated reward signals. Generalization gaps, especially in RL, also manifest in task-specific overfitting, with agents failing up to 60% of the time when faced with dynamic obstacles not encountered during training. These persistent shortcomings underscore the need for more robust and sample-efficient learning strategies. Finally, we discuss hybrid paradigms that integrate IL and RL, such as Generative Adversarial IL, and propose future research directions.

1. Introduction

In recent years, autonomous driving has evolved into a complex and rapidly progressing research area. As an inherently multidisciplinary challenge, it brings together elements of robotics, computer vision, control theory, embedded systems, and artificial intelligence. While many architectural decompositions are possible, for clarity, we distinguish five core functional domains in autonomous vehicles: perception, localization, planning, control, and system-level management (see Figure 1). These components do not operate in isolation—they are highly interdependent and must be tightly integrated to achieve safe and reliable autonomous behavior. For instance, accurate localization relies on perception, planning depends on both localization and perception, and control must faithfully execute decisions made by the planning module. This functional breakdown serves as a foundation for understanding where and how RL and IL techniques can be applied effectively.

Perception in autonomous vehicles encompasses the capabilities that allow the system to interpret its surroundings using raw sensor data from cameras, LiDAR, radar, and ultrasonic sensors. It processes this input to detect and classify objects, recognize lanes and traffic signals, and assess environmental conditions such as lighting and weather. The output is a structured, machine-readable representation of the environment, which serves as a vital input to the localization, planning, and decision-making modules. To be effective, perception must operate in real time and remain robust under diverse and challenging conditions, including occlusions, sensor noise, and adverse weather. Localization enables the vehicle to determine its precise position and orientation within a known or partially known environment by fusing data from GPS, inertial measurement units, LiDAR, and cameras, often using techniques such as simultaneous localization and mapping (SLAM) or map-matching algorithms. Accurate localization is essential for safe navigation and must remain reliable even in GPS-denied areas, during sensor drift, or when landmarks are temporarily obscured. Planning is responsible for determining the vehicle’s future actions and is typically divided into two levels: behavior planning, which selects high-level maneuvers such as lane changes or yielding, and motion planning, which computes detailed, dynamically feasible trajectories. These plans depend heavily on accurate inputs from perception and localization to ensure obstacle avoidance, compliance with traffic laws, and passenger comfort, while adapting continuously to a changing environment. Control translates these planned trajectories into low-level commands for steering, throttle, and braking, aiming to follow the desired path with precision, stability, and responsiveness. It commonly employs methods such as PID controllers, model predictive control (MPC), or learning-based strategies, and must operate at high frequency while being resilient to delays, disturbances, and variations in vehicle dynamics. Overseeing all these components is the system management layer, which ensures reliable and safe operation by coordinating tasks, allocating computational resources, monitoring system health, detecting faults, and managing communication between modules. It also provides fallback strategies in case of failures, making it a critical backbone for production-grade autonomous driving systems.

This work focuses primarily on the planning and control components of autonomous driving, where RL and IL have demonstrated the greatest practical relevance. In these domains, learning-based agents are trained to make tactical decisions and execute low-level maneuvers based on sensor-derived representations of the environment. Approaches vary from end-to-end policies that directly map observations to actions, to modular designs where planning and control are learned separately. In contrast, components such as perception and localization are typically treated as fixed inputs—often provided by the simulator in the form of semantic maps, LiDAR projections, or ground-truth poses—and are not the target of learning. Similarly, system-level management is generally handled using conventional software architectures and falls outside the scope of learning.

Among the range of machine learning techniques applied in this domain, reinforcement learning (RL)—exemplified by algorithms such as Proximal Policy Optimization (PPO) and Deep Q Networks (DQNs)—has emerged as one of the most promising approaches [1]. Consequently, the field of RL continues to advance at a remarkable pace, with increasingly sophisticated methods producing superior performance across a growing spectrum of applications [2,3].

This review addresses RL and IL strategies implemented in the CARLA simulation platform. CARLA is uniquely suited to this task due to its open-source nature, full-stack simulation capabilities, and strong support for AI integration, making it one of the most widely used platforms for academic research on learning-based AV agents. CARLA is a widely adopted high-fidelity simulator for autonomous driving research that offers extensive support for sensor realism, weather variability, and urban driving scenarios [4,5]. Reviewing the state-of-the-art literature helps identify prevailing agent architectures, design decisions, and gaps that may inform the development of improved driving agents.

The technical analysis in this paper is grounded in representative RL-based studies [6,7,8,9,10,11]. For each work, we extract the definitions of the agent’s state and action spaces and examine the structure and effectiveness of the associated reward function. The studies, though diverse in their technical formulations, share a common foundation: the use of DRL in the CARLA environment for autonomous navigation and control. We synthesize the insights from these papers to compare strategies, identify trade-offs, and understand how varying model architectures impact policy learning and performance.

Additionally, three studies [12,13,14] present IL-based approaches that do not rely on environmental rewards but instead learn behaviors by mimicking expert demonstrations. These IL methods are assessed in parallel to the RL models to identify complementary strengths and limitations. Their inclusion is critical, as IL is increasingly integrated with RL to enhance learning efficiency, policy generalization, and safety, especially in data-constrained or high-risk environments [15,16].

Beyond these core studies, recent research has pushed the boundaries of RL in autonomous driving further. For example, Sakhai and Wielgosz proposed an end-to-end escape framework for complex urban settings using RL [17], while Kołomański et al. extended the paradigm to pursuit-based driving, emphasizing policy adaptability in adversarial contexts [18]. Furthermore, Sakhai et al. explored biologically inspired neural models for real-time pedestrian detection using spiking neural networks and dynamic vision sensors in simulated adverse weather conditions [19]. Recent studies have also explored the robustness of AV sensor systems against cyber threats. Notably, Sakhai et al. conducted a comprehensive evaluation of RGB cameras and Dynamic Vision Sensors (DVSs) within the CARLA simulator, demonstrating that DVSs exhibit enhanced resilience to various cyberattack vectors compared to traditional RGB sensors [20]. These contributions showcase the evolving versatility of RL architectures and their capacity to address safety-critical tasks under uncertainty.

Despite the rapid progress in learning-based autonomous driving, existing systems face several persistent challenges. Studies have shown that end-to-end RL agents often suffer from significant performance degradation—up to 35%—when deployed in conditions that differ from their training environments, such as novel towns or adverse weather [9]. Transfer RL has been explored as a mitigation for such weather-induced distribution shifts [21]. Similarly, imitation learning (IL) agents trained solely on expert demonstrations have demonstrated up to 40% higher collision rates when exposed to previously unseen driving contexts [12]. Moreover, a key difficulty in RL lies in the careful tuning of reward functions; poorly calibrated rewards have been linked to over 20% of policy failures in simulation settings [7,11]. Finally, RL agents often overfit to narrow task domains, with generalization failures reaching up to 60% when encountering dynamic obstacles not seen during training [10]. These limitations underscore the need for more robust, transferable, and sample-efficient learning frameworks that integrate the strengths of both RL and IL paradigms.

In general, this review aims to provide a comprehensive synthesis of RL and IL research in the CARLA simulator, highlighting how these learning paradigms contribute to the development of intelligent autonomous driving systems. Compared to widely cited reviews such as [22,23], our work provides a more detailed and structured analysis of reinforcement learning methods specifically in the context of autonomous driving control within simulation environments. While previous surveys focus largely on algorithmic taxonomies or general architectural roles, we examine RL approaches through a set of concrete implementation descriptors: state representation, action space, and reward design. This framing enables direct comparison of control strategies and learning objectives across studies, which is largely missing from earlier work. We also include visual summaries of reward functions and highlight differences in how these functions guide policy learning. Furthermore, our focus on the CARLA simulator as a common experimental platform allows for a more consistent and grounded discussion of evaluation strategies and training setups, bridging the gap between theory and practical deployment.

2. Reinforcement Learning in Autonomous Driving

2.1. Overview of Reinforcement Learning

Reinforcement learning (RL) is a cornerstone of modern machine learning, particularly as deep learning has expanded its applicability to complex, real-world problems [24]. In the RL paradigm, an agent interacts with its environment by performing actions and receiving feedback in the form of rewards, which guide its future behavior [25,26,27]. This closed-loop interaction between agent and environment lies at the heart of autonomous decision-making systems.

During training, the agent uses a policy to select actions, while the environment responds by returning a new observation and a scalar reward that reflects the value of the action taken [28]. The reward function is pivotal—it defines the learning objective and directly influences the agent’s trajectory through the learning space [29,30]. A schematic of this interaction is shown in Figure 2, where the feedback loop is emphasized. A key strength of RL lies in its emphasis on long-term cumulative reward optimization [31]. Instead of maximizing immediate payoff, the agent learns policies that favor long-term outcomes, often sacrificing short-term gain for strategic advantages. Each learning episode starts in an initial state and ends upon reaching a terminal condition [32], making RL particularly suitable for sequential decision-making tasks such as autonomous driving.

In this framework, the agent serves as the autonomous controller, making decisions from within a predefined action space [33]. These actions may be discrete (e.g., lane switch) or continuous (e.g., steering angle adjustment) [34,35]. A state encodes relevant environmental information at a given moment, and the complete collection of possible states defines the state space [36]. The action space defines all possible maneuvers the agent can execute [37].

A crucial function in this setup is the reward function, which provides evaluative feedback for the agent’s choices. This scalar signal not only reinforces beneficial actions but also penalizes poor decisions, shaping the agent’s future policy [29,30]. The policy maps states to probabilities over actions, and the agent’s objective is to approximate or discover an optimal policy that maximizes expected cumulative rewards across episodes [38].

RL is distinct from supervised and unsupervised learning, the other two main branches of machine learning [39]. While supervised learning relies on labeled data and unsupervised learning discovers structure from unlabeled data, RL learns directly through real-time interaction. The episodic nature of RL generates diverse, temporally rich training samples that improve generalization and robustness in dynamic settings.

Reinforcement learning methodologies are commonly categorized into the following types:

• Model-based vs. Model-free: Model-based methods construct a predictive model of environment dynamics to plan actions. Model-free methods, in contrast, learn policies or value functions directly from experience [38,40,41].
• Policy-based vs. Value-based: Policy-based methods directly learn the policy function, often using gradient methods. Value-based methods estimate action values and derive policies by selecting actions with the highest value [38,42].
• On-policy vs. Off-policy: On-policy methods optimize the same policy used to generate data, ensuring data-policy alignment. Off-policy methods allow learning from historical or exploratory policies, improving sample efficiency [38,43,44].

Recent studies have explored diverse reinforcement learning approaches in autonomous driving. For instance, Toromanoff et al. (2020) presented a model-free RL strategy leveraging implicit affordances for urban driving [45]. Wang et al. (2021) introduced the InterFuser, a sensor-fusion transformer-based architecture enhancing the safety and interpretability of RL agents [46]. Additionally, Chen et al. (2021) developed cooperative multi-vehicle RL agents for complex urban scenarios in CARLA, highlighting significant improvements in collaborative navigation [47]. More recent CARLA-based multi-agent work reinforces these findings, demonstrating cooperative behaviors via MARL [48].

Prominent examples that embody these categories include Q-learning, REINFORCE, Actor–Critic algorithms, Proximal Policy Optimization (PPO), Deep Q-Networks (DQNs) [49,50] and Model Predictive Control (MPC) [51,52].

Table 1. Comparison of widely used reinforcement learning methods.

Method	Use Model	Based on	On-Policy/Off-Policy
Q-learning	Model free	Values	Off-policy
REINFORCE	Model free	Policy	On-policy
Actor–Critic	Model free	Hybrid (Values + Policy)	On-policy
Proximal Policy Optimization (PPO)	Model free	Policy	On-policy
Deep Q-Network	Model free	Values (approx. using neural network)	Off-policy
Model Predictive Control (MPC)	Model based	Dynamics model + Policy Optimization	Typically off-policy

summarizes the key characteristics of some of the most popular reinforcement learning methods, classifying them based on their use of models, policy/value orientation, and policy type.

2.2. Related Works About Autonomous Driving

Several contemporary investigations have harnessed the CARLA simulator to push the boundaries of reinforcement-learning–based autonomous driving. Alongside these, early model-based RL efforts pursued realistic end-to-end driving pipelines [53], setting the stage for later latent-world-model approaches. For instance, Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (implemented in CARLA v2) [54] introduces a cohesive agent that fuses learned world models with policy optimization to achieve robust, end-to-end vehicle control. In CuRLA: Curriculum Learning Based Deep Reinforcement Learning for Autonomous Driving [55], the authors confront the challenge of sluggish policy convergence in standard DRL agents by embedding a structured curriculum, thereby accelerating skill acquisition across progressively complex driving tasks. At the task level, RL has also been used for motion planning in negotiation-heavy settings; for example, unsignalized intersections are addressed with an RL-based planner in [56]. Finally, Multi-Objective Reinforcement Learning for Adaptive Personalized Autonomous Driving [57] presents a novel framework in which a multi-objective RL agent not only navigates autonomously but dynamically modulates its driving style according to user-defined preferences, balancing safety, efficiency, and comfort.

3. Review of Reinforcement Learning Approaches

Autonomous driving has emerged as a focal area of scientific inquiry, with reinforcement learning methods occupying a central role due to their inherent suitability for sequential decision-making problems. In this thesis, we undertake a comprehensive review of the literature on artificial intelligence techniques applied to autonomous driving, placing particular emphasis on reinforcement learning approaches.

Table 2. Summary of reviewed reinforcement learning papers’ state spaces and action spaces.

Paper [Agent]	State Space	Action Space
Implementing a deep reinforcement learning model for autonomous driving [6] [PPO agent]	throttle [0, 1], steer [−1, 1], brake [0, 1], distance from waypoint, waypoint orientation [degrees]	throttle [0, 1], steer [−1, 1], brake [0, 1]
Deep reinforcement learning based control for Autonomous Vehicles in CARLA [7] [DRL-flatten-image agent]	121 dimensional vector (processed RGB image), distance from lane, $\varphi$ [degrees]	steer [−1, 1],throttle [0, 1]
Deep reinforcement learning based control for Autonomous Vehicles in CARLA [7] [DRL-Carla-Waypoints agent]	vector of waypoints [w0…14], distance from lane, $\varphi$ [degrees]	steer [−1, 1], throttle [0, 1]
Deep reinforcement learning based control for Autonomous Vehicles in CARLA [7] [DRL-CNN agent]	$I_{\mathrm{RGB}[640\times 480]}$, distance from lane, $\varphi$ [degrees]	steer [−1, 1], throttle [0, 1]
Deep reinforcement learning based control for Autonomous Vehicles in CARLA [7] [DRL-Pre-CNN agent]	vector of waypoints [w0…14], distance from lane, $\varphi$ [degrees]	steer [−1, 1], throttle [0, 1]
Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator [8] [RL agent]	distance from intersection, speed	stop [speed = 0 m/s], drive [speed = 5 m/s]
CARLA: An Open Urban Driving Simulator [9] [A3C agent]	$I_{\mathrm{RGB}[84\times 84]}$ speed, distance to goal, damage from collisions	steer [−1, 1], throttle [0, 1], brake [0, 1]
Think2Drive: Efficient Reinforcement Learning by Thinking with Latent World Model for Autonomous Driving [10] [Think2Drive agent]	$I_{\mathrm{BEV}[128\times 128\times 34\times 4]}$, speed	throttle [0, 1], brake [0, 1], steer [−1, 1]
Safe Navigation: Training Autonomous Vehicles using Deep Reinforcement Learning in CARLA [11] [Combined DQN agent]	distance, distance from obstacle, $\varphi$ [degrees], speed, traffic light	brake [throttle = 0.0, brake = 1.0, steer = 0.0], drive straight [throttle = 0.3, brake = 0.0, steer = 0.0], turn left [throttle = 0.1, brake = 0.0, steer = −0.6], turn right [throttle = 0.1, brake = 0.0, steer = 0.6], turn slightly left [throttle = 0.4, brake = 0.0, steer = −0.1], turn slightly right [throttle = 0.4, brake = 0.0, steer = 0.1]

provides a concise but comprehensive overview of the state and action spaces used in the studies surveyed.

Table 3. Summary of reward functions from reviewed papers.

Paper [Agent]	Reward
Implementing a deep reinforcement learning model for autonomous driving [6] [There was only one agent presented]	R = $\left\{\begin{array}{cc}-10,\hfill & \mathrm{on}\mathrm{infraction}\hfill \\ {\displaystyle \frac{v}{v_{min}}}(1-d_{\mathrm{norm}})a_{\mathrm{rew}},\hfill & v<v_{min}\hfill \\ (1-d_{\mathrm{norm}})a_{\mathrm{rew}},\hfill & v_{min}\le v<v_{\mathrm{target}}\hfill \\ \left(1-\frac{v-v_{\mathrm{target}}}{v_{max}-v_{\mathrm{target}}}\right)(1-d_{\mathrm{norm}})a_{\mathrm{rew}},\hfill & v\ge v_{\mathrm{target}}\hfill \end{array}\right.$
Deep reinforcement learning based control for Autonomous Vehicles in CARLA [7] [DRL-flatten-image agent, DRL-Carla-Waypoints agent, DRL-CNN agent, DRL-Pre-CNN agent]	$R_t$ = $\left\{\begin{array}{cc}-200,\hfill & \mathrm{i}\mathrm{f} \mathrm{c}\mathrm{o}\mathrm{l}\mathrm{l}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n} \mathrm{o}\mathrm{r} \mathrm{l}\mathrm{a}\mathrm{n}\mathrm{e} \mathrm{c}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\hfill \\ {\sum }_t\left(\right|v_{t}cos{\varphi }_t|-|v_{t}sin{\varphi }_t|-|d_t\left|\right),\hfill & \mathrm{if}\mathrm{car}\mathrm{in}\mathrm{lane}\hfill \\ 100,\hfill & \mathrm{if}\mathrm{goal}\mathrm{reached}\hfill \end{array}\right.$
Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator [8] [There was only one agent presented]	R = $\left\{\begin{array}{cc}{k}_v{v}_{\mathrm{ego}},\hfill & \mathrm{standard}\mathrm{driving},\hfill \\ 1,\hfill & \mathrm{crossing}\mathrm{intersection},\hfill \\ -2,\hfill & \mathrm{collision},\hfill \\ -{\displaystyle \frac{0.2}{t_{\mathrm{out}}}},\hfill & \mathrm{episode}\text{-}\mathrm{duration}\mathrm{penalty}\hfill \end{array}\right.$
CARLA: An Open Urban Driving Simulator [9] [There was only one agent presented]	$\begin{array}{cc}\hfill R_t& =1000(d_{t-1}-d_t)+0.05(v_t-v_{t-1})\hfill \\ & -0.00002(c_t-c_{t-1})-2(s_t-s_{t-1})-2(o_t-o_{t-1})\hfill \end{array}$
Think2Drive: Efficient RL by Thinking with Latent World Model [10] [There was only one agent presented]	$R=r_{\mathrm{speed}}+{\alpha }_{\mathrm{tr}}{r}_{\mathrm{travel}}+{\alpha }_{\mathrm{de}}{p}_{\mathrm{deviation}}+{\alpha }_{\mathrm{st}}{c}_{\mathrm{steer}}$
Safe Navigation: Training Autonomous Vehicles using DRL in CARLA [11] [One agent combined two models]	$R_{\mathrm{drive}}$ = $\left\{\begin{array}{cc}-200,\hfill & \mathrm{if}\mathrm{collisions}>0\vee \left|\varphi \right|>100\vee \left|d\right|>3,\hfill \\ R(s,{\pi }_d\left(s\right))-10,\hfill & \mathrm{if}\mathrm{collisions}\le 0\wedge \left|\varphi \right|\le 100\wedge 2<\left|d\right|\le 3,\hfill \\ R(s,{\pi }_d\left(s\right)),\hfill & \mathrm{if}\mathrm{collisions}\le 0\wedge \left|\varphi \right|\le 100\wedge \left|d\right|\le 2,\hfill \end{array}\right.$
	$R_{\mathrm{brake}}(collisions,v,d,a)$ = $\left\{\begin{array}{cc}3,\hfill & \mathrm{if}v<10d+10\wedge a=0,\hfill \\ -1,\hfill & \mathrm{if}v<10d+10\wedge a=1,\hfill \\ 4,\hfill & \mathrm{if}v>10d+10\wedge a=1,\hfill \\ -2,\hfill & \mathrm{if}v>10d+10\wedge a=0,\hfill \\ -10,\hfill & \mathrm{if}v<1\wedge d>100,\hfill \\ 200,\hfill & \mathrm{if}v=0\wedge d<150,\hfill \\ -200,\hfill & \mathrm{if}collisions>0,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$

then details, in the corresponding sequence, the reward functions formulated and utilized within those same works, thus facilitating direct comparison and analysis.

Table 2 systematically contrasts the state and action space designs across key reinforcement learning studies in CARLA. Early models such as the A3C agent from Dosovitskiy et al. [9] employed simple RGB images, vehicle speed, and distance to target—representing the minimal viable input for policy learning. More recent works, such as Think2Drive [10], introduce significantly higher-dimensional and semantically enriched state representations, including BEV tensors with 34 semantic channels across four temporal frames, enabling more abstract and generalizable world models.

Agents like those proposed by Pérez-Gil et al. [7] demonstrate a controlled comparison of input modalities—including vectorized images, CNN-based encodings, and waypoint-driven features—highlighting the performance trade-offs between raw sensor data and handcrafted spatial abstractions. In contrast, Razak et al. [6] leverage a relatively lean state set—focused on control-relevant geometry like distance and orientation to the nearest waypoint—favoring fast training and low system complexity. Nehme et al. [11] adopt a modular input structure that combines metric state values (e.g., distance, $\varphi$, traffic light) with a discrete action space reflecting semantically meaningful maneuvers, such as “turn left” or “drive straight.”

These distinctions emphasize not only the evolution of perceptual richness but also how input design constrains the expressiveness and structure of the agent’s output space. Overall, the diversity in state–action formulations across the reviewed works underscores the lack of standardization in evaluation and highlights the need for benchmarking that isolates the effects of input representation on policy learning and generalization.

Table 3 highlights a diverse range of reward shaping strategies employed in reinforcement learning for autonomous driving. Razak et al. [6] use a piecewise, velocity- and orientation-based reward function that emphasizes smooth driving behavior and lane adherence, but applies only a moderate penalty for infractions, which might limit its effectiveness in enforcing strict safety constraints. In contrast, Pérez-Gil et al. [7] combine a large penalty for collisions or lane departures with continuous terms rewarding forward motion and low lateral deviation, and add a substantial goal-reaching bonus. Their structure strikes a balance between task completion and behavioral stability. Gutierrez-Moreno et al. [8] design a sparse, event-triggered reward targeting intersection navigation: speed is mildly encouraged, success is rewarded, and both collisions and prolonged decision making are penalized. This simple scheme works well in isolated tasks but may be too coarse for general driving.

Dosovitskiy et al. [9] introduce one of the earliest dense formulations, integrating driving progress, speed changes, and penalties for damage, off-road driving, and collisions. Although comprehensive, it is sensitive to reward scaling and thus less portable across scenarios. Think2Drive [10] adopts a modular reward combining speed, distance traveled, lateral deviation, and steering smoothness, with tunable weights that allow fine-grained behavioral calibration. Finally, Nehme et al. [11] separate reward functions for driving and braking components, tailoring each with domain-specific logic. Their approach supports safe, interpretable modular control but introduces added engineering overhead. Overall, the table reveals trade-offs between simplicity and generalization, with denser, composite rewards yielding richer behavior at the cost of increased design complexity. Let me know if you would like a third paragraph for future work or benchmarking.

For consistency and interpretability, other parameters in each reward equation were fixed at constant values to avoid scaling extremes. Additionally, flat reward terms—those not dependent on the parameter being evaluated—are included as horizontal lines to indicate their constant contribution.

Figure 3, Figure 4 and Figure 5 below illustrate how the reward function values reported in the reviewed articles vary in response to changes in a single parameter. The reward functions’ variable values have been evaluated by systematically varying three principal parameters within each function. When adjusting one parameter, the remaining parameters were held constant and selected to minimize their influence on the reward outcome and to prevent the emergence of extreme reward values.

The values in the reward graphs represent reward functions with varying values of one variable, while the rest of the variables remain constant. The reward functions R1, R2, R3, R4, R5, R6_drive, and R6_break correspond to the checked articles in the following order: [6,7,8,9,10,11]. The y-axis shows the reward values returned by the reward functions in the following articles. On the x-axis, the graphs have different values depending on the graph; they are speed values, collision flag, or distance from the center of the lane. Each of the three charts is shown in normalized and non-normalized versions.

The comparative analysis in

Table 4. Comparative analysis of reinforcement learning agents in autonomous driving literature.

Agent (Study)	Input Representation	Control Output	Key Performance Metrics	Architectural Highlights	Strengths and Limitations
Think2Drive [10]	BEV latent maps with semantic and temporal masks	Discrete set of 30 maneuvers	100% route completion on CARLA v2 within 3 days on A6000 GPU	Latent world model + Dreamer-V3 architecture	High generalization; requires privileged inputs and tuned latent space
Gutierrez-Moreno [8]	Speed + distance to intersection	Discrete stop/go	78–98% success in intersection scenarios;  0.04 collisions per episode	Curriculum learning; actor–critic; 2D input space	Fast convergence; limited for urban complexity
Dosovitskiy [9]	Dual RGB frames + auxiliary features	Continuous control signals	74% route success (baseline CARLA); low scalability	A3C baseline with dense reward	Generalizable benchmark; outdated by newer models
Pérez-Gil [7]	4 architectures: RGB → vector, waypoints, raw CNN, preprocessed CNN	Continuous steering/throttle	78% success;  18% collisions across setups	Compares flattened vs CNN vs waypoint inputs	Explores control-input trade-offs; lacks domain transfer tests
Razak [6]	Throttle, brake, steering, distance + orientation to waypoint	Continuous control	83% success; 12% collision (simulation report)	Simple handcrafted reward with angular shaping	Minimal setup; lacks multi-scenario adaptation
Nehme [11]	Depth + traffic light + deviation metrics	Modular discrete driving and braking decisions	86% success;  7% collision rate (simulation study)	Separate DQN for brake + maneuver control	Interpretability; modular; complex integration

delivers a rigorous technical synthesis rather than a sequential listing. It clearly maps the evolution of input representations—from minimalist 2D vectors (e.g., speed and distance in Gutierrez-Moreno et al. [8]) to rich latent BEV maps with semantic and temporal history (Think2Drive [10]). While light inputs enable fast convergence in focused tasks, only richer perception models achieve the broad robustness necessary for real-world corner-case handling. For instance, Think2Drive processes high-dimensional BEV maps using a Dreamer-V3 latent world model, achieving exceptional route coverage via discrete maneuver planning. In contrast, earlier agents like Razak et al. [6] and Pérez-Gil et al. [7] rely on continuous control directly from raw or flattened image inputs, which allows for smoother trajectories but hinders sample efficiency and generalizability.

Empirical outcomes further highlight emerging trends: Think2Drive is the first to report 100% route completion on CARLAv2 within three days of training on a single A6000 GPU [10]—a landmark in model-based RL efficacy. Model-free agents such as those by Dosovitskiy et al. [9] (A3C) and Razak et al. [6] (PPO) show route success between 74 and 83%, reflecting limited scalability. The modular design of Nehme et al. [11] introduces interpretability and enhanced safety through discrete braking and maneuvering policies but increases architectural complexity. This synthesis identifies critical design trade-offs—input complexity, action discretization, model-based vs model-free learning, and modularity—while also exposing research gaps, such as the need for broader domain generalization, standardized benchmarks, and comparisons with camera-only RL agents such as those explored in emerging frameworks like CarLLaVA [58].

4. Gaps and Limitations in RL-Based Methods

While the previously reviewed studies present a diverse range of methodologies for autonomous driving within the CARLA simulation environment also as detailed in Appendix A, several persistent challenges and open research questions remain. These gaps reflect both the limitations inherent in the design of current reinforcement learning agents and the complexities of real-world deployment scenarios. Identifying and understanding these issues is crucial for improving the scalability, safety, and applicability of RL-based systems in autonomous driving.

The primary limitations observed across the surveyed literature are as follows:

• Limited Generalization to Real-World Scenarios—Some RL-based models exhibit strong performance within narrowly defined simulation environments but fail to generalize to the complexities of real-world driving. For instance, the agent described in [8] was trained under constrained conditions using a binary action set (stop or proceed). Although suitable for intersection navigation in simulation, this minimalist approach lacks the flexibility and robustness required for dynamic urban environments characterized by unpredictable agent interactions, diverse traffic rules, and complex road layouts.
• Pipeline Complexity and Practical Deployment Issues—Advanced RL systems often involve highly layered and interdependent components, which, while enhancing learning performance in simulation, may hinder real-time applicability. The architecture proposed in [11] integrates multiple sub-models for driving and braking decisions. Despite its efficacy within the training context, the increased architectural complexity poses challenges in deployment scenarios, such as increased inference latency, reduced system interpretability, and difficulties in modular updates or extensions.
• Reward Function Design and Safety Trade-offs—Crafting a reward function that effectively balances task completion with safe, rule-compliant behavior remains an ongoing challenge. Poorly calibrated rewards may inadvertently incentivize agents to exploit loopholes or adopt high-risk behaviors that maximize rewards at the cost of safety. The reward formulation presented in [7] mitigates this risk by incorporating lane deviation, velocity alignment, and collision penalties. Nevertheless, even well-intentioned designs can result in unintended behaviors if the agent learns to over-prioritize specific features, highlighting the need for reward tuning and safety regularization.
  Several works have specifically tackled these limitations by employing advanced RL strategies. Yang et al. (2021) developed uncertainty-aware collision avoidance techniques to enhance safety in autonomous vehicles [59]. Fang et al. (2022) proposed hierarchical reinforcement learning frameworks addressing the complexity of urban driving environments [60]. Cui et al. (2023) further advanced curriculum reinforcement learning methods to tackle complex and dynamic driving scenarios effectively [61]. Feng et al. (2025) utilized domain randomization strategies to enhance the generalization of RL policies, significantly improving performance across varying simulated environments [62].

5. Imitation Learning (IL) for Autonomous Driving

Imitation learning (IL) is a machine learning paradigm in which an agent learns to perform tasks by mimicking expert behavior, typically that of a human. Unlike reinforcement learning, where the agent discovers optimal actions through trial and error and reward feedback, IL infers a policy directly from expert demonstrations by mapping perceptual inputs to control outputs.

During training, the agent observes example trajectories and attempts to replicate the expert’s decisions in the same contexts. The aim is for the model to internalize the expert’s policy, enabling generalization to similar, unseen scenarios.

The overall process of IL is illustrated in Figure 6.

As shown above, IL shares many components with reinforcement learning, though the learning mechanism differs. The key actors in the IL framework are:

• Agent—The learner that interacts with the environment. Unlike in reinforcement learning, the agent does not rely on scalar rewards but instead learns to imitate the expert’s demonstrated behavior.
• Expert—Typically, a human operator or a pretrained model that provides high-quality demonstration trajectories. These trajectories serve as the ground truth for training.

All remaining terminology, such as state, action, and environment, corresponds to the definitions previously introduced in Section 2.1 on reinforcement learning.

IL algorithms are typically categorized along two orthogonal dimensions:

• Model-based vs. Model-free—Model-based approaches attempt to build a transition model of the environment and use it for planning. Model-free techniques rely solely on observed expert trajectories without reconstructing the environment’s internal dynamics.
• Policy-based vs. Reward-based—Policy-based methods directly learn a mapping from states to actions, while reward-based methods (such as inverse reinforcement learning) infer the expert’s reward function before deriving a policy.

A selection of widely used IL techniques are summarized in

Table 5. Summary of popular IL methods.

Method	Use Model	Based on
Behavioral Cloning (BC)	Model free	Values
DAgger (Dataset Aggregation)	Model free	Policy
Inverse Reinforcement Learning (IRL)	Indirect (Learns Reward)	Reward
Generative Adversarial IL (GAIL)	Model free	Policy

.

Recent advancements in imitation learning also demonstrate promising outcomes. Luo et al. (2023) improved behavioral cloning techniques by introducing adaptive data augmentation methods, significantly enhancing IL agents’ performance in novel driving scenarios [63]. Moreover, Mohanty et al. (2024) utilized inverse reinforcement learning to achieve human-like autonomous driving behaviors in CARLA, emphasizing the effectiveness of learning from expert demonstrations [64].

5.1. IL-Based Architectures

IL has gained considerable traction in autonomous driving research due to its ability to leverage expert demonstrations and simplify policy learning. This section reviews key publications that implement IL-based approaches for autonomous vehicle control.

The reviewed works differ in their perceptual inputs, control representations, and agent architectures.

Table 6. Summary of reviewed IL papers, their state spaces, and action spaces.

Paper [Agent]	State Space	Action Space
End-to-end Driving via Conditional IL [12] [Simulation and physical agent; identical state/action spaces]	$I_{\mathrm{RGB}[800\times 600]}$	$\mathrm{steer}[-1,1],$ $\mathrm{acceleration}[-1,1]$
Learning by Cheating [13] [Privileged agent]	$M_{\mathrm{BEV}[320\times 320\times 7]},$ $\mathrm{speed},$ $\mathrm{command}$	$\mathrm{steer},$ $\mathrm{brake},$ $\mathrm{throttle}$
Learning by Cheating [13] [Sensorimotor agent]	$I_{\mathrm{RGB}[384\times 160]},$ $\mathrm{speed},$ $\mathrm{command}$	$\mathrm{steer},$ $\mathrm{brake},$ $\mathrm{throttle}$
Dynamic Conditional IL for Autonomous Driving [14] [Single-agent architecture]	$I_{\mathrm{RGB}},$ $\mathrm{command},$ $\mathrm{LiDAR},$ $\mathrm{speed}$	$\mathrm{steer}[-1,1],$ $\mathrm{throttle}[0,1],$ $\mathrm{brake}[0,1]$

summarizes the state and action spaces for each model discussed.

Table 7. Comparative analysis of imitation learning agents in autonomous driving literature.

Agent (Study)	Input Representation	Control Output	Key Performance Metrics	Architectural Highlights	Strengths and Limitations
Codevilla et al. [12]	RGB image (800 × 600), directional command	Continuous steering, acceleration	96% success rate in simulation; real-world feasibility via scaled platform	Conditional branching policy; centralized end-to-end network	Simple design; low robustness to distributional shift; single camera limitation
Chen et al. [Privileged] [13]	Semantic BEV map (320 × 320 × 7), speed, command	Steering, throttle, brake (via controller on predicted waypoints)	Achieves near-expert imitation with minimal variance	Two-stage pipeline with expert heatmap output and PID controller	High accuracy, but requires privileged input unavailable at test time
Chen et al. [Sensorimotor] [13]	RGB image (384 × 160), speed, command	Steering, throttle, brake	Generalizes well when trained via privileged supervision	Learns to imitate expert via internal heatmap prediction	End-to-end trainable; suboptimal if privileged guidance is inaccurate
Eraqi et al. [14]	RGB image, LiDAR (polar grid), speed, directional command	Steering, throttle, brake	92% success rate in navigation scenarios; improved safety margins	Multi-modal encoder; dynamic conditional policy branches	Robust multi-sensor fusion; higher computational complexity

offers a comparative overview of several IL approaches in autonomous driving, focusing on the modalities used for perception and control. Each agent architecture incorporates different combinations of visual, sensor, and navigation inputs to compute real-time driving decisions.

Table 7 offers a structured comparative analysis of imitation learning (IL) methods for autonomous driving, going beyond paper-by-paper description to extract underlying architectural and perceptual trends. Across all reviewed works, we observe a key technical divergence: some agents rely on end-to-end mappings from raw sensor data (e.g., monocular RGB in [12,13]), while others leverage structured intermediate representations such as BEV semantic maps [13] or fused LiDAR polar grids [14]. This split has major implications for model interpretability, transferability, and dependence on privileged information. Architectures like the “privileged agent” in [13] explicitly encode scene semantics, enabling spatially grounded planning via heatmap prediction, but require access to annotated maps. In contrast, purely sensorimotor approaches trade interpretability for operational independence, relying on network capacity to infer spatial features implicitly.

Another dimension of comparison lies in the control abstraction. While all agents ultimately produce low-level continuous commands (steering, throttle, brake), they differ in how commands are conditioned. Codevilla et al. [12] and Eraqi et al. [14] both adopt branched architectures, where distinct sub-networks handle different high-level directional commands. This conditional control strategy improves behavioral specialization and aligns with hierarchical task decomposition, but may introduce fragility if command prediction is noisy. Additionally, multi-modal architectures like in [14] show clear technical advantages in handling diverse scenarios by integrating visual, depth, and navigation cues. However, such models increase architectural complexity and raise questions about data alignment and modality-specific failures. In sum, the table synthesizes these trends to expose the underlying design principles, trade-offs, and open challenges in IL for autonomous driving—thus directly responding to the reviewer’s request for more comparative and technically grounded analysis.

5.2. Hybrid Solutions

IL, while effective for quickly learning behaviors from expert demonstrations, struggles with limited generalization—agents often fail when encountering situations not covered in the training data. On the other hand, reinforcement learning excels at learning through exploration and can achieve high performance, but it is computationally expensive and prone to instability. Combining these two approaches helps to mitigate their weaknesses: expert demonstrations can guide and accelerate learning, reducing unsafe or inefficient exploration. At the same time, the reward-driven nature of RL enables agents to refine their behavior and adapt to novel scenarios. Within the scope of this review, it is important to highlight the increasing synergy between IL and reinforcement learning in autonomous driving systems.

Table 8. Comparative analysis of hybrid IL+RL agents in autonomous driving literature.

Agent (Study)	Input Representation	Control Output	Key Performance Metrics	Architectural Highlights	Strengths and Limitations
Phan et al. [66]	RGB, speed, high-level commands	Steering, throttle, brake	Effective policy transfer to real-world intersections	Inverse RL followed by RL fine-tuning	Combines realism and safety; complex pretraining required
Ho and Ermon [67]	Trajectory sequences (expert vs. agent)	Action likelihood over states	Performs well in MuJoCo; widely adopted in AV variants	GAIL adversarial setup: discriminator + generator agent	No reward engineering; sensitive to discriminator instability
Liang et al. [CIRL] [65]	Camera image, vehicle speed, navigation command	Steering, throttle, brake	High success rate in intersection and obstacle-avoidance tasks	Behavior cloning + DDPG in staged learning	Improves generalization; requires balanced training phase timing
Chekroun et al. [GRI] [68]	Sensor data, expert trajectory actions	Steering, throttle, brake	Tested in CARLA and MuJoCo; fast convergence	Unified replay buffer; expert data has constant high reward	No IL pretraining needed; success depends on reward scaling
Han et al. [69]	Sensor data + sparse expert cues	Blended: agent and expert actions (probabilistic)	Rapid learning with low collision rates	Exploration guided by sparse expert rules	Safe learning; depends on sparse rule coverage
Reddy et al. [SQIL] [70]	Standard RL states + expert trajectories	RL actions from standard policy	Improves over baseline Soft Q-Learning in MuJoCo	Simple buffer reward logic (1 for expert, 0 for agent)	Very easy to integrate; lacks fine control on imitation priority
Huang et al. [71]	RGB image + LiDAR + GPS	Steering, throttle, brake	Enhanced safety and generalization in CARLA scenarios	Hierarchical fusion of IL + PPO RL agent	Flexible framework; limited outside simulation
Kim and Cho [72]	Camera + sensor state features	Steering, throttle, brake	Improved driving scores in complex scenarios	Combines attention-based IL with actor–critic RL	Effective in dense environments; lacks interpretability

provides a comparative analysis of hybrid architectures that integrate imitation learning (IL) and reinforcement learning (RL) in the context of autonomous driving. This table highlights key technical differentiators across models—such as input modalities, control schemes, performance outcomes, and architectural decisions—thus enabling a deeper comparative discussion of their contributions. For instance, several approaches, including CIRL [65] and Phan et al. [66], adopt a two-stage learning process where IL serves as a pretraining phase followed by RL-based policy refinement. These approaches are particularly effective in sparse-reward environments, allowing agents to learn safer and more optimal behaviors with fewer exploration steps.

Moreover, models such as GAIL [67] and SQIL [70] depart from handcrafted reward engineering by leveraging either adversarial training or reward shaping based on expert trajectories. GRI [68] and the sparse imitation framework from Han et al. [69] further exemplify how continuous or conditional use of expert data can stabilize learning and improve generalization. While these hybrid strategies differ in implementation complexity and assumptions (e.g., availability of full expert data vs. sparse hints), they all converge toward a common goal: leveraging demonstration data not only as initial guidance but as a persistent signal for optimization throughout training. These insights underscore the importance of hybrid frameworks in bridging the sample efficiency of IL with the adaptiveness and robustness of RL, thus offering practical, scalable solutions for real-world autonomous driving.

6. Comparative Analysis and Discussion

The comparative review of reinforcement learning and IL approaches for autonomous driving in the CARLA simulator reveals distinct strengths and limitations for each paradigm. These trade-offs echo findings from a comparative study of behavioral cloning and reinforcement learning for autonomous driving [73].

Table 9. Summary of results from discussed articles.

Article	Goal	Architecture Complexity	Environment Complexity	Scenarios	Advantages
Implementing a Deep Reinforcement Learning Model for Autonomous Driving	End-to-end RL agent with Variational Autoencoder (VAE)	2	3	One scenario in three towns (1, 2 and 7)	An extensive and complex approach to the subject
Deep Reinforcement Learning Based Control for Autonomous Vehicles in CARLA	Comparison of DQN and DDPG on few models	1	1	Two scenarios in town 1	Multiple and varied approaches to agent’s architecture
Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator	Complete agent capable of driving through intersection with traffic	2	3	Three scenarios, one for each type of intersection (lights, stop signal, uncontrolled)	Complex approach to crossing intersection with traffic
CARLA: An Open Urban Driving Simulator	Comparison of performance of RL, IL and Modular Pipeline	1	3	Five scenarios in four possibilities (training conditions, new town, new weather, new weather and town)	A good comparison of RL with other machine learning methods with a multi-sensor agent
Think2Drive: Efficient Reinforcement Learning by Thinking with Latent World Model for Autonomous Driving (in CARLA-V2)	RL agent possible to drive in multiple corner cases	3	3	39 detailed scenarios	A highly complex approach to an agent capable of driving multiple corner case scenarios
Safe Navigation: Training Autonomous Vehicles using Deep Reinforcement Learning in CARLA	RL agent capable of driving to maintain speed and break when necessary to avoid collision	2	2	Four scenarios in town 2	A complex and robust approach to an agent composed of two models (breaking model and driving model)
Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)	Model-based RL agent capable of learning effective driving from raw sensor data	3	3	220 routes from Batch2Drive benchmark	Recent highly complex and highly robust dual-stream Model Based RL approach with one stream of privileged sensor data
CuRLA: Curriculum Learning Based Deep Reinforcement Learning For Autonomous Driving	PPO+VAE agent for driving in environment with increasing traffic	2	2	Few routes in town 7 with changing traffic	A modern approach combining Curriculum Learning with Deep Reinforcement Learning
Multi-Objective Reinforcement Learning for Adaptive Personalized Autonomous Driving	Multi-objective RL agent capable of driving according to preferences	2	1	One scenario	A modern approach to autonomous driving with a multi-objective (selected driving preferences) end-to-end agent

presents a consolidated overview of all the studies reviewed, detailing the main objective of each work, the complexity of its proposed architecture, the complexity of its testing environment, and the total number of driving and route scenarios evaluated. To facilitate a standardized comparison, both the architectural and environmental complexity is quantified on a three-point scale (1–3), with “3” denoting the highest level of complexity. For example, an architecture that integrates multiple distinct models (e.g., [11]) is assigned a complexity score of 2, and similarly, an environment encompassing several urban centers or diverse traffic elements receives an environmental complexity rating of 2.

This comparative structuring reveals several insights:

• Baseline approaches tend to be simple and narrowly scoped. Papers comparing DQN and DDPG in Town 1 or intersections under limited conditions fall here. They offer important groundwork but do little to stress-test agents’ generalization across environments.
• Mid-complexity architectures (PPO + VAE, braking/driving modules) improve robustness. Curriculum learning (CuRLA [55]; see also [74]) and modular braking–driving workflows demonstrate that strategic layering of subsystems yields robustness to changes in traffic or new tasks.
• High-complexity, high-diversity methods push boundaries. State-of-the-art works like Think2Drive and Raw2Drive introduce latent world models or dual sensor-stream inputs, validated across dozens to hundreds of scenarios. These methods exemplify increased policy adaptability to rare and corner cases.
• Hybrid and multi-objective agents indicate a shifting paradigm. While still relatively underexplored, works that incorporate user preferences or multi-objective reward structures begin to bridge from pure RL toward human-centric driving goals.

7. Conclusions

This review has analyzed and synthesized recent reinforcement learning and IL approaches for autonomous driving, with a focus on studies implemented within the CARLA simulator. The literature surveyed demonstrates that both paradigms offer unique advantages and face specific limitations in the context of simulated driving environments.

Reinforcement learning enables agents to learn flexible, task-optimized behaviors through exploration and interaction with their environment. Its strength lies in its adaptability and end-to-end training pipeline. However, RL often requires carefully engineered reward functions and extensive simulation time, and it struggles with generalization and sample inefficiency.

IL, on the contrary, allows agents to quickly acquire competent policies from expert demonstrations, avoiding the complexities of reward design. IL methods typically converge faster and demonstrate smoother behavior in constrained settings. However, their dependence on demonstration quality and vulnerability to distributional shifts can limit robustness in novel or unpredictable scenarios.

Several hybrid approaches have emerged to combine the strengths of both paradigms, including policy pretraining, adversarial learning, and inverse reinforcement learning. These strategies hold promise in overcoming individual limitations and advancing the state of the art in learning-based driving agents.

Looking ahead, the field is poised to benefit from continued integration between reinforcement learning and IL frameworks. Emerging trends point toward greater emphasis on safety, explainability, and real-world transfer, particularly as simulation-to-reality gaps remain a persistent obstacle for deploying learned agents beyond synthetic benchmarks.

CARLA has played a central role in standardizing experimental evaluation for autonomous driving, allowing reproducible comparisons between studies. However, expanding benchmarks to incorporate dynamic human interactions, multi-agent negotiation, and diverse sensory conditions will be essential for realistic policy development.

This review underscores the importance of modular, interpretable, and sample-efficient approaches that combine expert guidance with autonomous learning. As learning-based methods mature, the focus must shift toward robust, certifiable agents capable of operating in open-world settings.