Predictive Risk-Aware Reinforcement Learning for Autonomous Vehicles Using Safety Potential

Abstract

Safety remains a central challenge in autonomous driving: overly rigid safeguards can cause unnecessary stops and erode efficiency. Addressing this safety–efficiency trade-off requires specifying what behaviors to incentivize. In reinforcement learning, the reward provides that specification. Conventional reward surrogates—such as distance gaps and time-to-collision (TTC)—depend on instantaneous geometry and often miss unfolding multi-vehicle interactions, whereas sparse terminal rewards provide no intermediate guidance. Accordingly, we adapt Safety Potential (SP)—a short-horizon, time-weighted path-overlap forecast—into a dense reward-shaping term that provides a predictive risk-aware signal for anticipatory throttle/brake control. In the CARLA v0.9.14 roundabout environment, SP attains 94% success with 3% collisions; in percentage points, this is 16.00, 13.00, and 5.75 higher success and 18.75, 9.50, and 7.25 lower collisions than No-Safe, Distance, and TTC, respectively. Adding a lightweight reactive guard at inference further reduces collisions to below 1% without sacrificing success. These results indicate that injecting a predictive, overlap-based risk measure directly into the reward supplies temporally consistent safety cues and clarifies the trade-off between progress and risk in reinforcement-learning-based vehicle control.

1. Introduction

Reinforcement learning (RL) is increasingly used in autonomous driving, yet much of the literature emphasizes algorithmic sophistication while reward design remains comparatively underexplored [1,2]. In safety-critical road settings, the reward is not a peripheral detail; it encodes what the agent should value and therefore governs the balance between efficiency and safety. This work adopts a reward-centric stance. Rather than modifying the learning algorithm, we study how shaping the reward can deliver safer behavior without adding architectural complexity.

Most existing RL rewards for on-road driving use reactive risk surrogates constructed from instantaneous geometry, for example distance or gap penalties [3] and time-to-collision (TTC) thresholds [4]. Because these quantities are computed from a single snapshot, they frequently raise false alarms when vehicles are close but on diverging paths. Prior work has explored proactive warning and predictive risk estimation [5,6,7], but these signals have seldom been used directly as step-wise reward terms within the RL training loop. Figure 1 illustrates that, even in visually similar traffic scenes, reactive gap/TTC surrogates and a predictive path-overlap signal can yield very different risk assessments. Moreover, works that introduce predictive cues typically compare only against a vanilla baseline, without disentangling how the new reward differs from conventional distance/TTC shaping; thus, the specific contribution of reward design remains unclear [8,9,10]. We introduce a predictive path-overlap reward that estimates short-horizon overlap between the ego’s planned trajectory and surrounding vehicles, yielding a temporally consistent risk signal that encourages anticipatory throttle and brake. We then conduct matched, reward-to-reward comparisons against TTC and the gap to isolate the contribution of reward design.

To isolate the effect of reward shaping, the rest of the stack is deliberately simple: a standard off-policy RL algorithm (Soft Actor–Critic) [11] handles longitudinal speed control and a rule-based Pure Pursuit controller [12] provides lateral steering. Experiments are conducted in the unsignalized CARLA [13] roundabout with a single learning agent (ego) interacting with many surrounding vehicles (NPCs; non-player-controlled vehicles, used here interchangeably with “surrounding vehicles”). Although many works measure or signal risk, comparatively few feed either predictive or reactive risk estimators back into RL as rewards or auxiliary signals and evaluate them on concrete simulator tasks with side-by-side baselines. Our study addresses this gap by operationalizing a lightweight overlap forecast as a step-wise, dense reward and comparing it against conventional snapshot-based rewards in this roundabout setting, showing that careful reward shaping can improve safety without additional model complexity.

1.1. Literature Review

A longstanding difficulty in reinforcement learning is the credit assignment problem (CAP): when rewards are sparse, delayed, or weakly informative, attributing credit or blame to specific decisions becomes unreliable and destabilizes learning [14]. Reward shaping was proposed precisely to mitigate CAP by providing additional, task–relevant feedback while preserving the original objective, and early studies showed that carefully designed shaping signals can accelerate learning and improve robustness across control domains [15,16].

In on–road driving, most shaping terms have been reactive and snapshot–based. Nageshrao et al. [17] penalize small headways when the ego–lead gap falls below a safety margin, whereas Huang et al. [18] grant positive reward within a desired spacing band. TTC has likewise been used directly as a safety reward: Zhu et al. [4] employ TTC as a dense term for velocity control, but they do not isolate their effect against alternative safety rewards under identical training conditions.

Beyond reactive measures, predictive approaches have been explored: Lv et al. [8] (lane change) add a short-horizon binary predictor to TTC/gap; Wang et al. [9] (roundabout) train a discrete IP–SAC with a binary penalty on predicted-path intrusion, reporting up to 90% SR and 10% CR; Dong and Guo [10] (intersection) use a transformer-based risk score as reward. These studies report setting-specific gains but rely on binary or black-box signals, do not isolate improvements over TTC/gap under matched conditions, and provide limited interpretability and fine-grained risk discrimination.

NVIDIA [19] introduced the Safety Force Field (SFF) and the notion of safety potential, where SP denotes the degree of overlap among the unions of vehicles’ predicted trajectory sets when each agent follows prescribed safety procedures; SFF is the mechanism that seeks to reduce this overlap. The public exposition is largely conceptual and oriented toward globally coordinated control rather than an ego-centric learner. Closer to our setting, Suk et al. [20] train an SP predictor and then modulate longitudinal acceleration with a rule-based controller in CARLA, reporting increases in number of arrivals and longer accident-free time; the decision policy itself is not learned via RL.

Recent task-specific autonomous-driving reinforcement learning work includes the following. At intersections, Leng et al. [21] treat safety as a cost and project the chosen action onto a safe set when the expected collision or lane-departure cost exceeds a threshold. Yu et al. [22] estimate uncertainty over the return with an ensemble or a distributional critic and trigger conservative fallback when uncertainty is high. In roundabouts, Gan et al. [23] train a single PPO policy with a dynamically weighted reward and a compact, state-compressed representation, reporting a 35–80% success rate, while Lin et al. [24] combine a new Q-value estimation approach for discrete decisions with a TTC-based constraint and a tracking controller, citing a 1–2% collision rate. Overall, across recent autonomous-driving RL, most methods rely on conventional risk measures and focus on acting on them rather than on designing them; when new rewards are proposed, matched reward-to-reward comparisons under identical conditions are rare, leaving the contribution of reward design underexplored.

1.2. Contributions

We introduce Safety Potential for reward shaping in a standard off-policy SAC agent. In the unsignalized CARLA roundabout, SP outperforms the strongest baseline (TTC), showing a 5.75 percentage-point gain in success and a 7.25 percentage-point reduction in collisions. To isolate reward effects, we compare rewards head-to-head under matched architectures and budgets; performance improves via a simple predictive overlap signal that quantifies risk continuously, is readily extensible, and can also serve as a standalone risk metric or auxiliary signal.

The remainder of the paper is organized as follows. Section 2 describes the methods—predictive risk shaping, reactive baselines, and the experimental setup—and specifies the environment characteristics and task difficulty. Section 3 presents results and analysis: quantitative metrics with statistical comparisons and qualitative case studies, and includes ablations that (i) toggle a simple hard guard and (ii) add Safety Potential as a state feature to separate reward effects from observation design. Section 4 concludes with limitations and directions for future work.

2. Method

This section presents a predictive risk–aware reinforcement-learning agent that augments the reward with an SP term while keeping the rest of the pipeline standard and off-policy. We first outline the overall pipeline, then define the fixed 49-dimensional ego-centric state and the action space, followed by the reward design (sparse and speed terms together with the SP shaping term) and the computation of SP from short-horizon path overlap with time weights. Finally, we describe the SAC training procedure with replay-based mini-batch updates and the hyperparameters used across experiments.

2.1. Overview

We implement a CARLA agent with rule-based lateral steering (Pure Pursuit) and SAC for longitudinal control. At each step, the agent observes the state and outputs both steering and throttle/brake; the simulator applies the combined control, but only the longitudinal component is used for learning. The resulting transition—state, longitudinal action, reward (including SP shaping), and next state—is logged to a replay buffer. During off-policy training, the learner samples minibatches from the buffer: the critic estimates Q-values from state–action pairs, and the actor is updated to favor actions with higher critic value. An overview of this control and learning framework is shown in Figure 2.

2.2. State Representation

The agent observes a 49-dimensional ego-centric state $s\in {\mathbb{R}}^{49}$ composed of ego features and up to six surrounding-vehicle slots; when fewer than six are present, the unused slots are zero-padded so the vector length remains 49 regardless of traffic count. To mitigate training instability from heterogeneous feature scales, all scalar state variables are normalized: distances and speeds are divided by 20, accelerations by 10, angular terms use a compact encoding in [−1, 1], and values are clipped to bounded ranges.

$$s=\left[ s_{\mathrm{ego}};s_{\mathrm{prog}};s_{\mathrm{look}};s_{\mathrm{sur}} \right],\mathrm{total}\mathrm{dimension}=49$$

(1)

$$s_{\mathrm{ego}}=\left[ v, {\theta }_v, a, {\theta }_a \right],s_{\mathrm{prog}}=\left[ g \right],s_{\mathrm{look}}=\left[ d_{\ell }, {\theta }_{\ell } \right]$$

(2)

$$s_{\mathrm{sur}}=\left[ s_1, \dots , s_6 \right],s_k=\left[ d_k, {\theta }_{d,k}, v_k, {\theta }_{v,k}, {\theta }_{\mathrm{yaw},k}, a_k, {\theta }_{a,k} \right]$$

(3)

The overall state dimensionality is 49 (4 for ego, 1 for progress, 2 for lookahead, and 6 × 7 for surrounding vehicles). $g\in [0,1]$ denotes the goal–progress ratio (0 at start, 1 at goal), and $(d_{\ell },{\theta }_{\ell })$ is the lookahead target derived from the local route. See Appendix A for complete nomenclature and normalization details.

2.3. Action and Control

At each step, the actor network receives the current observation and outputs a single continuous longitudinal command $a\in [-1,1]$. We interpret positive a as throttle and negative a as brake:

$$\mathrm{throttle}=max(a,0),\mathrm{brake}=max(-a,0).$$

(4)

Steering is generated by a Pure Pursuit controller rather than the learned policy. We follow a fixed global route and, at each step, construct a speed-adaptive local path from the ego’s nearest point. The Pure Pursuit lookahead point is selected on the constructed local route at 5 m ahead of the ego, from which the steering angle is computed. The simulator then applies the neural policy’s throttle/brake together with this steering command.

2.4. Safety Potential: Discretized, Time-Weighted Overlap Risk

To keep the two-dimensional overlap computation tractable while preserving spatial fidelity, both the ego route (generated after the agent action) and each surrounding-vehicle continuation are discretized into short longitudinal rectangular cells; the per-frame Safety Potential is then derived from weighted cell intersections.

2.4.1. Route Discretization and Footprinting for SP

For each surrounding vehicle, we generate a short-horizon rule-based prediction by combining two simple elements: (i) a few body-aligned points sampled along the vehicle’s yaw to cover the immediate footprint and (ii) lane-following waypoints smoothly extended ahead from the current position and heading. Waypoints are sampled at a fixed 1 m spacing. Along this route, each NPC yields $max(2v+5, 10)$ rectangular polygons (including the body footprint), while the ego route yields $max(2v, 8)$ polygons, where v is the vehicle speed (m/s) under the 1 m discretization. These deterministic, rule-based routes provide the waypoint sequences, which are converted to rectangular cells for overlap calculation.

2.4.2. Overlap Area Definition

Each waypoint is converted to a rectangular polygon oriented with the local path; the width is $W=2.8 \mathrm{m}$ (vehicle width) and the length is $L=1 \mathrm{m}$. Denote the ego polygons by $p_i$($i=0,\dots ,N_e-1$) and the k-th surrounding vehicle’s polygons by $q_{k,j}$ ($j=0,\dots ,N_o-1$). The rectangular intersection area is written $\mathrm{Area}(p_i\cap q_{k,j})$.

2.4.3. Normalized Weights (Time Weight)

To emphasize more imminent interactions, we apply exponential time weights along forward polygon indices and treat a small number of leading NPC cells as unit-weight body boxes, so that near-term overlaps are penalized more strongly, as illustrated in Figure 3.

$$u_{e,i}={\displaystyle \frac{i}{N_e-1}},u_{o,k,j}=\left\{\begin{array}{cc}0,\hfill & j<J_{\mathrm{body}},\hfill \\ {\displaystyle \frac{j-J_{\mathrm{body}}}{N_o-1-J_{\mathrm{body}}}},\hfill & j\ge J_{\mathrm{body}},\hfill \end{array}\right.,$$

(5)

$$w_i=exp(-\eta u_{e,i}),v_{k,j}=\left\{\begin{array}{cc}1,\hfill & j<J_{\mathrm{body}},\hfill \\ exp(-\eta u_{o,k,j}),\hfill & j\ge J_{\mathrm{body}},\hfill \end{array}\right..$$

(6)

where $J_{\mathrm{body}}$ denotes the number of initial body-aligned cells and $\eta >0$ is the decay coefficient along the path. We set $J_{\mathrm{body}}=5$ to approximately span one vehicle length, and fix $\eta =0.5$ since larger $\eta$ more aggressively down-weights distant cells and may underestimate emerging risk.

2.4.4. Safety Potential Aggregate

The Safety Potential at time t is the maximum, over surrounding vehicles k, of the time-weighted overlap summed across ego and other-vehicle polygons:

$${\mathrm{SP}}_t=\underset{k}{max}\sum _{i=0}^{N_e-1}\sum _{j=0}^{N_o-1}min\left(w_i, v_{k,j}\right) \mathrm{Area}\left(p_i\cap q_{k,j}\right).$$

(7)

2.4.5. Comparison of Pre-Collision Risk Metrics

We generated collision episodes using a random policy and, for each episode, recorded Safety Potential (SP), Time-to-Collision (TTC), and forward distance at relative timesteps $\tau =30,\dots ,2$ before collision (one step = $0.1$ s; total = $2.9$ s). For visualization, we computed the per-step average across episodes and estimated 95% confidence intervals via bootstrap across episodes. To place the proxies on a comparable scale, TTC and distance were sign-flipped so that larger values indicate higher risk, and each metric was then normalized to the [0, 1] range by min–max scaling.

In the episode-aligned averages shown in Figure 4, SP rises approximately monotonically as collision approaches and shows markedly smaller variability across episodes. By contrast, TTC and distance often show nonmonotonic trends, with dips or plateaus in the middle of the trajectory, and they do not consistently peak at collision. This indicates that SP supplies a denser and more temporally consistent signal before collision, whereas TTC and distance are susceptible to short-term fluctuations caused by relative geometry and motion. As a result, they are less suitable as dense shaping signals for learning.

2.5. Reward Design

The episodic reward combines sparse terminal/event signals that determine success or failure with dense shaping terms that guide safe, efficient, stepwise behavior. Sparse terms dominate final returns, while dense terms (safety and speed shaping) help the agent trade efficiency against risk during training.

2.5.1. Safety Potential Reward

Safety Potential is a predictive, overlap-based risk measure. The SP penalty includes a state cost and an action-scaled component that activates above a threshold; braking (negative longitudinal action) reduces the action-scaled penalty, which prevents persistent penalty accumulation in congested or stopped situations:

$$r_{\mathrm{sp}}\left(t\right)=-{\lambda }_{\mathrm{sp}} {\mathrm{SP}}_t-{\lambda }_{\mathrm{sp},a} 1_{\{{\mathrm{SP}}_t>{\sigma }_{\mathrm{sp}}\}} a_t {\mathrm{SP}}_t.$$

(8)

We use ${\sigma }_{\mathrm{sp}}=7$ and ${\lambda }_{\mathrm{sp}}={\lambda }_{\mathrm{sp},a}=0.005$. Here $a_t\in [-1,1]$ is the longitudinal control.

2.5.2. Sparse Event Rewards

Terminal outcomes and checkpoint events are modeled as a single sparse component:

$$r_{\mathrm{sparse}}=\left\{\begin{array}{cc}-10,\hfill & \mathrm{on}\mathrm{collision},\hfill \\ +5,\hfill & \mathrm{on}\mathrm{goal}\mathrm{reached},\hfill \\ +1,\hfill & \mathrm{on}\mathrm{predefined}\mathrm{intermediate}\mathrm{waypoint},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right. .$$

(9)

Here, $r_{\mathrm{sparse}}$ aggregates both terminal events and non-terminal checkpoints.

2.5.3. Speed Reward

Speed shaping encourages maintaining a target speed $v^{\star }=8\mathrm{m}/\mathrm{s}$ but is gated by SP to avoid incentivizing unsafe acceleration into predicted overlaps. The shaping rewards small deviations from $v^{\star }$ under low risk, penalizes large overspeeding, and applies a small negative term to discourage persistent stalling when the vehicle remains idle without braking:

$$r_{\mathrm{speed}}=\left\{\begin{array}{cc}{\displaystyle 0.02 max\left(0, 1-{\displaystyle \frac{|v_t-v^{\star }|}{v^{\star }}}\right),}\hfill & \mathrm{if}{v}_t\le v^{\star }+2,{\mathrm{SP}}_t<3,\hfill \\ -0.2 {\displaystyle \frac{v_t-v^{\star }}{v^{\star }}},\hfill & \mathrm{if}{v}_t>v^{\star }+2,\hfill \\ -0.015,\hfill & \mathrm{if}{\mathrm{SP}}_t<0.01,v_t<0.5,a_t<0.1,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right. .$$

(10)

All dense shaping weights are intentionally small relative to terminal rewards so that collisions and goal attainment determine final returns, while SP gating ensures the speed objective does not distort behavior under risky conditions.

2.5.4. Total Reward

The per-step reward is the sum of dense shaping and terminal/event terms (the terminal term is nonzero only at the corresponding events):

$$r_t=r_{\mathrm{sp}}\left(t\right)+r_{\mathrm{sparse}}\left(t\right)+r_{\mathrm{speed}}\left(t\right).$$

(11)

2.6. Training Procedure (Soft Actor–Critic)

We train a stochastic policy ${\pi }_{\theta }(a\mid s)$ using the Soft Actor–Critic (SAC) framework with two Q-function critics $Q_{\varphi }^{\left(1\right)},Q_{\varphi }^{\left(2\right)}$ to mitigate overestimation. The critics are optimized by minimizing the soft Bellman residual with L2 regularization:

$$L_Q\left(\varphi \right)=\mathbb{E}\left[\sum _{i=1}^2{\left(Q_{\varphi }^{\left(i\right)}(s_t,a_t)-{\widehat{Q}}_t\right)}^2\right]+{\lambda }_{\mathrm{wd}}{\parallel \varphi \parallel }_2^2.$$

(12)

with target

$${\widehat{Q}}_t=r_t+\gamma \left(\underset{i=1,2}{min}{Q}_{{\varphi }^{-}}^{\left(i\right)}(s_{t+1},a_{t+1})-\alpha log{\pi }_{\theta }(a_{t+1}\mid s_{t+1})\right), a_{t+1}\sim {\pi }_{\theta }(·\mid s_{t+1}).$$

(13)

and the actor is updated to maximize the entropy-regularized objective:

$$L_{\pi }\left(\theta \right)=\mathbb{E}\left[\alpha log{\pi }_{\theta }(a_t\mid s_t)-Q_{\varphi }(s_t,a_t)\right],a_t\sim {\pi }_{\theta }(·\mid s_t).$$

(14)

We follow a standard SAC setup with automatic temperature tuning for $\alpha$, soft target-network updates for the critics, and Adam optimizers with gradient regularization. At each control step, the agent receives an observation, selects a continuous longitudinal action, and the resulting tuple $(s_t,a_t,r_t,s_{t+1},\mathrm{done})$ is appended to a replay buffer; the reward $r_t$ already includes the SP-based shaping term (Section 2.4). Training proceeds off-policy: the actor continually collects experience, and at fixed intervals a minibatch is drawn from the buffer to (i) update the critics using soft target-network updates, (ii) update the policy under the entropy-regularized objective, and (iii) adjust the entropy temperature via the standard dual formulation. The training hyperparameters are summarized in

Table 1. Training hyperparameters. Initialized from SAC defaults [11] and minimally adjusted for stability; the same configuration is used for all experiments. $h_i$ denotes the i-th hidden layer.

Parameter	Value
Replay buffer capacity	200,000
Batch size	256
Update frequency (env steps per optimize)	5
Actor/Critic/$\alpha$ learning rate	$2\times {10}^{-4}/2\times {10}^{-4}/1\times {10}^{-4}$
Discount factor $\gamma$	0.99
Soft update coefficient $\tau$	0.001
Target entropy	$-1.0$
Initial log-$\alpha$	$-1.5$
L2 regularization (critic)	$1.0\times {10}^{-4}$
Actor network layer sizes (input $\to h_1\to h_2\to$ output)	$49\to 256\to 256\to 1$
Critic network layer sizes (input $\to h_1\to h_2\to$ output)	$50\to 256\to 256\to 1$

.

2.7. Experimental Setup

2.7.1. Traffic Generation

Background traffic is spawned over the whole map with a random count of 180–190 vehicles per episode. NPCs run the built-in autopilot (CARLA Traffic Manager) with the following parameters:

• headway to the leader: uniformly 6–20 m;
• desired speed scaling: uniformly $\pm 10\%$;
• lateral lane offset: uniformly $\pm 0.1$ m;
• ignore_lights: fixed $100\%$ (all NPCs ignore traffic lights to ensure steady inflow into the roundabout);
• ignore_vehicles: fixed $20\%$ (all NPCs occasionally ignore yielding, inducing near-collision conflicts).

This configuration increases interaction difficulty and exposes the ego policy to diverse risky situations, while being identical across all learning variants. Traffic density and driver-behavior parameters are calibrated not to replicate any particular real-world distribution, but to create a deliberately harsh environment that accelerates learning and magnifies performance differences. For the calibration baseline under the same protocol, see Section 2.7.3.

2.7.2. Episode Protocol and Termination

Each episode runs for up to $T=700$ fixed simulation ticks. An episode terminates early on (i) successful exit via the designated branch (goal reached), (ii) any collision with dynamic or static actors, or (iii) reaching the step limit. The experimental environment and representative training scenes are shown in Figure 5.

2.7.3. Rule-Based Baseline (Calibration)

Because this setup is not a standardized community benchmark with a turnkey evaluation suite, a calibration step is necessary. We therefore use CARLA’s rule-based stack (BehaviorAgent) as a baseline, and we also treat the No-Safe variant (Section 2.7.4) as an additional reference metric for policy performance.

To calibrate task difficulty, we evaluate BehaviorAgent (“normal” configuration) on the same route and traffic protocol. Across 400 episodes, this baseline yields a collision rate of 20.5%. A post-hoc heuristic audit classifies 6.0% of episodes (≈29% of collisions) as primarily the responsibility of surrounding vehicles. These figures characterize the task as deliberately challenging; subsequent policy results should be interpreted in light of this difficulty level.

2.7.4. Agent Variants and Risk Shaping

We compare four longitudinal controllers that are identical in architecture and learning, differing only in the per-step risk shaping. Lv et al. [8] is a representative on-road RL work that explicitly applies safety rewards by combining TTC and distance gap; we adopt this formulation as the basis for our baselines. To isolate each surrogate, we separate the combined safety term into two baselines and couple each penalty to the longitudinal action. For fairness, baseline settings were lightly tuned to the roundabout scenario.

• No-Safe: no dense per-step risk penalty.
• Distance (forward-distance shaping). Based on Equation (13) [8]; detections are limited to a $\pm 10°$ forward cone to reduce false alarms. Unlike [8], we use a one-sided gap penalty (no penalty for large gaps) since the speed reward (Equation (10)) already captures efficiency. We set the gap margin to 15 m. With ${\lambda }_d=0.5$, the shaping term is as follows:

  $$r_{\mathrm{dist}}\left(t\right)=-{\lambda }_d max\left(0, 1-{\displaystyle \frac{d_t}{15}}\right)-{\lambda }_d 1_{\{d_t<10\}} a_t\left(1-{\displaystyle \frac{d_t}{15}}\right).$$

  (15)
• TTC (time-to-collision shaping). Following Section 4.3 and Equation (13) of Lv et al. [8]—which computes ${\tau }_t=\Delta x/\Delta v$ and uses a 2.5 s margin—we adopt TTC as the surrogate and then apply roundabout-specific filters: we take the minimum over valid detections, ignore opposite-heading traffic, and disable the term when the ego is stationary. With ${\lambda }_{\tau }=0.1$, the shaping term is

  $$r_{\mathrm{ttc}}\left(t\right)=-{\lambda }_{\tau } max\left(0, 2.5-{\tau }_t\right)-{\lambda }_{\tau } 1_{\{{\tau }_t<1.5\}} a_t\left(2.5-{\tau }_t\right).$$

  (16)
• SP (ours): Safety Potential (time-weighted overlap; see Section 2.4) is used as the per-step shaping signal. The SP-based training procedure is summarized in Algorithm 1.

Algorithm 1: SP-based Training Procedure

1: Initialize networks and parameters $(\theta ,{\varphi }_1,{\varphi }_2,\alpha )$.

2: for episode $=1,\dots ,N$ do

3:    Spawn the ego vehicle and NPCs.

4:    while not (collision ∨ goal ∨ max_step) do

5:        Set $N_t$ (NPCs to observe at step t) and compose the state $s=[s_{\mathrm{ego}};s_{\mathrm{prog}};s_{\mathrm{look}};s_{\mathrm{sur}}]$.

6:        Plan a local ego trajectory ${\xi }_{\mathrm{ego}}$.

7:        Determine the steering command with the Pure Pursuit controller.

8:        Sample throttle $a\sim {\pi }_{\theta }(a\mid s)$, apply (steer, a), and step the environment to obtain $s^{\prime }$.

9:        Replan the ego local trajectory ${\xi }_{\mathrm{ego}}^{\prime }$ and update predicted trajectories for NPCs ${\left\{{\zeta }_u^{\prime }\right\}}_{u\in N_t}$.

10:      For each $u\in N_t$, compute $S{P}_u=\mathrm{overlap}\left({\xi }_{\mathrm{ego}}^{\prime }, {\zeta }_u^{\prime }\right)$; then set $S{P}_t={max}_{u\in N_t}S{P}_u$.

11:      Compute reward r and store $(s,a,r,s^{\prime },\mathrm{done})$.

12:      Periodically update $\theta ,{\varphi }_1,{\varphi }_2,\alpha$ with the SAC optimizer.

13:   end while.

14:   Destroy the ego vehicle and all NPCs for the next episode.

15:  end for.

Output: learned longitudinal control policy ${\pi }_{\theta }^{\mathrm{new}}$.

3. Results

This section evaluates the policies in CARLA through quantitative analysis of task completion, safety, and driving quality, and qualitative diagnostics of risk-aligned behavior and representative collision cases. A separate ablation study examines the effects of enabling a lightweight hard guard at inference and appending a normalized SP term to the state.

3.1. Quantitative Results

We evaluate performance along two primary driving objectives—task completion (success rate, SR) and safety (collision rate, CR)—and assess driving quality via the speed-reward component. Because the safety-related reward terms differ in both scale and formulation, direct comparisons of raw episode returns are not appropriate; reporting per-component metrics isolates task success, safety, and control quality.

3.1.1. Success and Collision Rates over Training

Figure 6 shows that the SP–shaped agent climbs out of the low-performance regime earlier than all baselines and passes the rule-based reference (gray band; see Section 2.7.3) after fewer samples, indicating superior sample efficiency. As training proceeds, its success curve continues to rise smoothly while the collision curve steadily declines, with markedly fewer late-stage oscillations. This stability suggests that the predictive, overlap-based signal provides consistent guidance even as traffic interactions become more frequent, helping the policy consolidate rather than regress.

By contrast, the Distance and TTC variants do not raise success or reduce collisions to the same extent. In Figure 4, their pre-crash curves exhibit higher dispersion and weaker temporal consistency; when used as rewards, such variability can destabilize training. Overall, SP dominates the safety–task frontier: at comparable success, it yields fewer collisions, and at comparable collisions, it achieves higher success. The thin SP+Guard band corresponds to pairing SP with a hard reactive constraint (see Section 3.4.1) and marks the attainable upper region under such a guard.

Deterministic, post-training evaluation in the same environment (

Table 2. Performance of learned policies after training (400 episodes in the same environment). Arrows indicate the preferred direction for each metric, and boldface highlights the best value in each column.

Method	Success Rate ↑	Collision Rate ↓	Timeout Rate ↓	Steps (Mean)
No-Safe	78.00%	21.75%	0.25%	249.5
Distance	81.00%	12.50%	6.50%	286.3
TTC	88.25%	10.25%	1.50%	345.2
SP (ours)	94.00%	3.00%	3.00%	369.2

) shows the same preference for SP: with networks frozen and actions taken as the deterministic mean (SAC samples only during training), SP attains the highest success and the lowest total failure (collisions + timeouts) while using more steps on average—consistent with safer, more deliberate behavior without additional mechanisms.

3.1.2. Speed Reward Comparison

Figure 7 plots the speed–reward of Equation (10) with a 30k-step moving average to suppress short-term noise and highlight training trends. The metric increases when the agent, under safe conditions, tracks the target speed closely while avoiding overspeeding and unnecessary stops; it is designed to isolate longitudinal regulation quality from safety shaping.

The SP variant attains the highest and most stable plateau late in training, indicating tighter adherence to the target-speed band with fewer over-/underspeed excursions. Distance improves steadily but settles below SP, suggesting more frequent deviations and occasional stop-and-go behavior. TTC also improves but remains below SP, typically with a slower ramp and mid-training oscillations consistent with reactive corrections. No-Safe shows the largest fluctuations, with sharp spikes and dips, reflecting unstable throttle–brake regulation.

This speed–reward serves as a proxy for longitudinal driving quality—smooth attainment of cruising speed with minimal idling or overshoot—independent of SR and CR. In this comparison, SP improves the speed–reward without degrading safety outcomes.

3.2. Stress-Test Scenarios

We run two within-distribution stress tests (200 episodes each). Test A changes driving style at the training density (headway 5–10 m, target speed +10–30%, lateral offset $\pm 0.2$ m). Test B increases traffic to 250–260 vehicles while keeping the training style. We avoid unrealistically easy or broken settings: densities below training make the task too simple, while densities above Test B can cause a standstill and hide policy differences. Likewise, making driving style more conservative than training would make the task easier, and making it much more aggressive would make responsibility unclear in crashes. These two setups are therefore slightly harder than training and are meant to test robustness.

Table 3. Stress-test results under distribution shift (200 episodes each). Arrows indicate the preferred direction for each metric, and boldface highlights the best value in each column.

Method	Test A: Aggressive Driving				Test B: High Density
	Success ↑	Collision ↓	Timeout ↓	Steps (Mean)	Success ↑	Collision ↓	Timeout ↓	Steps (Mean)
No-Safe	74.0%	25.0%	1.0%	260.6	69.0%	30.5%	0.5%	276.9
Distance	79.0%	10.5%	10.5%	274.4	80.4%	9.8%	9.8%	293.9
TTC	89.5%	9.0%	1.5%	347.9	78.5%	17.5%	4.0%	387.9
SP (ours)	90.0%	5.5%	4.5%	365.2	92.0%	3.5%	4.5%	397.7

shows that in Test A, all methods behave similarly to the training setting; safety metrics (SR/CR) remain comparable to training, indicating that the style change does not materially affect safety. In Test B, heavier congestion and tighter gaps raise collisions overall. No-Safe and TTC show the largest increases in collision rate, consistent with point-to-point measures struggling in dense, closely packed traffic. Distance shows a small drop in collisions but a clear rise in timeouts, indicating progress stalls under crowding. By contrast, SP holds up better: collision rates do not rise in the same way, and success remains competitive, suggesting less overfitting to the training setup.

3.3. Qualitative Study

We complement the quantitative results with two qualitative analyses. We examine risk-aligned behavior around hazard onset by time-aligning events and summarizing action, speed, SP, and TTC, and we review representative collision cases to highlight typical failure modes and recovery patterns.

3.3.1. Risk-Aligned Behavior Around Hazard Onset

Although TTC and SP are not perfect ground-truth measures of risk, their tail regimes are less prone to false alarms and align with genuinely hazardous states. Leveraging this, the trigger in the training environment is defined as the first time both TTC $\le 1.5$ s and SP $>15$ hold; events are aligned at step 0 and analyzed over a 3 s window spanning the preceding 5 steps (0.5 s) and the subsequent 25 steps (2.5 s).

Figure 8 shows that, relative to the alternatives, the SP-shaped policy initiates braking earlier and with greater magnitude before the hazard materializes, and its control trajectory around step 0 evolves more smoothly with fewer reversals. Following the trigger, TTC rises with a steeper initial slope and remains on a higher plateau, while SP drops more rapidly and stabilizes at a lower level, indicating quicker clearance and more stable resolution of the hazardous interaction. Speed recovers more slowly yet more monotonically, suggesting a deliberate choice to maintain a safety buffer once risk has been suppressed. Overall, the policy reallocates effort to earlier preemption, achieves stronger and more sustained TTC recovery with persistently low SP, and accepts slower post-event speed recovery as the trade-off for consistent risk reduction across events.

3.3.2. Collision Case Analysis

After training, we analyzed four representative collision cases from additional rollouts using the SP policy to inspect typical failure modes and the limits of single-step scalar risk surrogates. Figure 9 and Figure 10 show spatial overviews of the collisions and the temporal risk profiles for the 30 steps preceding each accident. Neither SP nor TTC exhibits a universally monotonic trend toward collision; although SP typically increases and TTC typically decreases as collision approaches, both metrics fluctuate with local geometry and interactions. This indicates the learned policy optimizes a mixture of objectives and that no single surrogate can guarantee safety in all situations. While SP provides a more forward-looking signal useful for learning, reactive measures remain valuable complements for real-time safety assurance.

Analysis of Collision Scenarios and SP/TTC Distributions:

• Case 1. The ego (blue box) changes from the inner to the outer lane as a red vehicle enters the same outer lane from an entrance. Because the first path overlap occurs only on a far segment of the ego’s plan, SP rises late as the vehicles converge, consistent with the delayed increase before the collision.
• Case 2. The ego (blue box) merges from an entrance into the inner lane while a white vehicle is already traveling there. SP briefly rises as a passing outer–lane car dominates, dips when it clears, then rises again as the inner–lane conflict becomes critical. Because SP takes the maximum overlap across surrounding vehicles, this temporary dominance understated the inner-lane risk; the ego kept merging, and a collision followed.
• Case 3. The ego (blue box) moves outward from the inner lane while a reddish-orange vehicle on the outer lane restarts from a stop. Because the first path overlap lies farther along the ego’s plan, SP stays at a moderate level without a sharp rise; risk is flagged but not high enough to trigger a stronger response, and a collision follows.
• Case 4. A red vehicle is blocked by a lead car ahead, while the ego (blue box) attempts to pass from behind. The path overlap is partial and oblique to the body, so SP stays very low (<5) despite footprint intersection, leading to underestimated risk.

Overall, these cases indicate that a single scalar risk proxy can leave blind spots in complex interactions. When used solely as a reward signal, SP does not fully capture atypical temporal profiles or spatial distributions of risk, leaving some patterns under-penalized. The base learner remains SAC; we modify only the reward by adding an SP shaping term. Because the critic’s Q-value aggregates multiple objectives (speed, progress, arrival), the policy does not necessarily halt when SP rises and can drift into high-risk configurations. In the sequel, we therefore consider complementary design choices that improve robustness.

3.4. Ablation Study

We evaluate two small variants while keeping the learner and training setup unchanged: (i) Safety-augmented Policies in the Training Environment, which adds a lightweight hard guard only at inference; and (ii) SP as a State Feature, which appends a normalized SP scalar to the observation to test stability in the training environment.

3.4.1. Safety-Augmented Policies in the Training Environment

Using the learned policy, we conduct additional rollouts in the same training environment (Town 3; identical route and traffic protocol with 180–190 NPCs) while enforcing a lightweight reactive guard only at execution time. The guard intervenes on longitudinal control by applying full brake whenever $\mathrm{TTC}<2.0$ s or when the nearest object within a $40°$ forward cone ($\pm 20°$ about the ego heading) is closer than 8 m; steering remains unchanged and is generated as in the base setup. Each method (with/without the guard) is evaluated over 400 episodes to average out the small absolute differences.

Table 4. Performance of guard-enabled variants. Success, collision, timeout rates, and average steps per episode are reported (400 evaluation episodes). Arrows indicate the preferred direction for each metric, and boldface highlights the best value in each column.

Method	Success ↑	Collision ↓	Timeout ↓	Steps (Mean)
No-Safe + Hard Guard	90.50%	4.50%	5.00%	329.7
Distance + Hard Guard	82.75%	3.00%	14.25%	382.0
TTC + Hard Guard	87.25%	2.50%	10.25%	414.7
SP + Hard Guard	92.75%	0.25%	7.00%	412.6

shows that the guard consistently reduces collisions across methods. Notably, the SP policy benefits the most: combining predictive shaping (SP) with a simple reactive brake rule achieves the lowest collision rate while maintaining competitive success. Out of a total of 400 episodes, only a single collision occurred under SP+Guard. This incident was primarily caused by the behavior of an opponent vehicle, rather than the ego’s policy.

3.4.2. SP as a State Feature

We append a scalar ${\mathrm{SP}}_t/30$ to each observation, expanding the observation to 50 dimensions. Because SP is derived from near-future path overlap with surrounding vehicles, adding it as a state feature injects predictive interaction information, though only in a compressed scalar form. Evaluations are conducted on 400 episodes in the same environment as training.

As shown in

Table 5. Evaluation in the same environment as training; 400 episodes per method. Arrows indicate the preferred direction for each metric, and boldface highlights the best value in each column.

Method	Success Rate ↑	Collision Rate ↓	Timeout Rate ↓	Steps (Mean)
No-SP (=No-Safe)	78.00%	21.75%	0.25%	249.5
SP state only	89.25%	8.00%	2.75%	292.5
SP reward only	94.00%	3.00%	3.00%	369.2
SP state + reward	94.25%	1.75%	4.00%	339.2

, exposing SP only as a state feature improves success and reduces collisions relative to the No-SP baseline, but remains below the reward-only variant. Using SP solely as a reward yields larger gains in both success rate (SR) and collision rate (CR). Combining state and reward provides a small additional benefit over either alone and also lowers the mean step count, indicating quicker episode completion under the same task conditions.

4. Discussion and Conclusions

This study introduced Safety Potential as a predictive risk signal in a SAC pipeline for longitudinal control and used it for reward shaping; in CARLA v0.9.14 roundabouts under a matched SAC setup, it outperformed distance- and TTC-based rewards on both quantitative metrics and qualitative driving behavior. We also evaluated a lightweight inference-time hard guard and a state-augmented variant that appends a normalized SP term. Overall, predictive overlap signals provide a practical basis for risk-aware policy learning; when used as rewards or states, they yield safer behavior and are expected to measure risk more faithfully.

Despite these promising results, several limitations remain. The action space was restricted to throttle and brake, which constrained evasive options and may have placed excessive burden on longitudinal control. Observations relied on simulator ground-truth states, including actors that would often be occluded in reality. In addition, there was no head-to-head comparison against standardized baselines in an identical environment, and community benchmarks for this specific task setting (roundabout, longitudinal-only control with fixed lateral policy) are largely absent, making fair cross-paper comparison difficult.

Looking ahead, several extensions are worth exploring. First, broaden the action space to include steering and richer maneuvers, enabling policies to exploit both lateral and longitudinal options. Second, move beyond the full-state assumption by using perception-derived inputs that reflect occlusion, noise, and latency, so the system is trained and evaluated under deployment conditions. Finally, adopt or help establish standardized benchmarks and reporting protocols for this task, and perform matched-environment comparisons against public baselines, with code, scenarios, and seeds released for reproducibility. These steps would better test the robustness and practical value of predictive risk shaping in reinforcement learning for autonomous driving.