Emergent Pedestrian Safety in a World-Model Driving Agent Under Adversarial Interaction Without Explicit Safety Rewards

Featured Application

Safety evaluation of world-model reinforcement learning agents for autonomous driving under adversarial pedestrian interaction, using the Jaywalkers Library, a configurable CARLA benchmark with three adversarial pedestrian archetypes, to diagnose emergent avoidance strategies and identify failure mechanisms under sparse-reward training.

Abstract

Pedestrian interaction remains a central safety challenge for autonomous driving, particularly under non-compliant or adversarial pedestrian behavior. Existing research and evaluations predominantly test against rule-following pedestrians, leaving a gap in understanding how learning-based agents handle worst-case interactions. We introduce the Jaywalkers Library, a novel configurable benchmark in CARLA with three adversarial pedestrian archetypes (Intruder, Indecisive Crosser, and Protester). We evaluate a DreamerV3 agent trained with sparse rewards, where the only pedestrian-specific signal is a terminal collision penalty. Evaluation employs a frozen-policy protocol with explicit train–test separation. Safety behavior is decomposed into endpoint outcomes, evasion dynamics, and efficiency costs. Under nominal conditions, the agent achieves high route completion and generalizes to an unseen town, whereas under adversarial exposure, an archetype-sensitive evasion strategy emerges. The agent swerves at speed against dynamic pedestrians but decelerates against the slow-moving Protester. Collision rates reveal a counterintuitive difficulty ordering in which the Protester is the hardest, followed by the Intruder, with the Indecisive Crosser as the most survivable. These findings show that a sparse terminal penalty suffices for emergent pedestrian avoidance in a world-model agent, but that effectiveness is bounded by the world model’s ability to predict pedestrian persistence.

1. Introduction

Urban autonomous driving demands reliable interaction with pedestrians, yet pedestrian behavior is weakly constrained, highly variable, and often non-compliant with traffic norms [1,2]. Empirical studies report that pedestrians frequently violate regulations, engage in risk-taking, and adapt their actions strategically based on vehicle speed, distance, and perceived driver intent [3,4,5]. Such behaviors (late crossings, gap exploitation, deliberate obstruction, sudden trajectory changes) contribute disproportionately to near-miss events and urban accidents [6,7]. For an autonomous vehicle, risk is therefore driven less by static scene layout and more by rapidly changing interaction structure: short-gap entries, hesitation, freeze–resume behavior, or persistent roadway occupation [8,9]. Several works further emphasize that pedestrian behavior reflects implicit negotiation and intent signaling rather than purely reactive or stochastic motion [10,11,12].

Simulation platforms such as CARLA [13], SUMO [14], and LGSVL [15] have become standard tools for developing and evaluating autonomous driving systems, and benchmarks built on these platforms assess perception accuracy, navigation success, and traffic rule compliance [16,17,18]. However, pedestrian behavior in these simulators is typically scripted using rule-based or social-force-inspired models that follow sidewalks, cross at designated locations, and exhibit limited deviation from nominal trajectories [19,20]. While appropriate for average-case evaluation, such models fail to represent the intentional or adversarial interactions documented in the traffic safety literature. A small number of works explore adversarial agents or environment perturbations to expose rare failure cases, but these typically target adversarial vehicles or sensor noise rather than pedestrians as strategic actors [21,22,23]. Evaluating safety in the presence of rare critical events has motivated scenario-based testing, importance sampling, and rare-event acceleration techniques [24,25,26], yet scenario-based approaches often lack systematic coverage of adversarial human behavior [27] and most studies report collision occurrence as the sole outcome without analyzing emergent behavioral patterns such as conservatism, hesitation, or late reaction [28,29]. These two gaps—the absence of adversarial pedestrian scenarios and the reliance on collision-only metrics—motivate the adversarial benchmark and multi-axis diagnostic framework proposed in this work.

Reinforcement learning (RL) has been widely applied to autonomous driving tasks including lane following, intersection handling, and end-to-end urban navigation [30,31]. Model-free methods such as Deep Q-Networks (DQNs), Proximal Policy Optimization (PPO), and Soft Actor–Critic (SAC) demonstrate promising performance but suffer from high sample complexity and limited generalization [32]. Model-based RL has gained attention as a means of improving sample efficiency and long-horizon planning; Dreamer and its successors learn latent world models and optimize policies through imagination-based rollouts, enabling reasoning about future consequences under partial observability [33,34,35,36]. Reward design remains a central factor: many prior studies include explicit pedestrian-aware shaping, including distance penalties, time-to-collision (TTC) penalties, and rule-based yielding incentives, to stabilize learning and reduce collisions. Although effective for optimization, dense safety shaping can obscure whether the resulting behavior is learned through dynamics prediction or injected directly through engineered penalties. In model-based driving specifically, most evaluations combine rich perception with rich reward shaping under cooperative traffic, leaving it unclear how much pedestrian safety can emerge from world-model planning when such shaping is deliberately excluded [37,38]. This motivates the present study’s deliberate exclusion of pedestrian-specific reward terms, isolating the safety behavior that emerges from world-model dynamics prediction alone.

Yet model-based RL agents have been evaluated almost exclusively under nominal or weakly stochastic traffic [18,39,40]. In adversarial interactions, risk may emerge late or depend on strategic responses from other agents, requiring planning over latent representations that can underrepresent rare but safety-critical futures [41]. Prior adversarial testing and rare-event methods have primarily targeted model-free controllers or rule-based planners, with limited focus on evaluating the robustness of learned world models in closed-loop interaction [42]. Three problems remain insufficiently addressed: (i) systematic evaluation of world-model agents under adversarial pedestrian interaction rather than cooperative traffic; (ii) disentangling emergent safety behavior from dense reward injection; and (iii) diagnostic analysis of how an agent fails, not only whether it collides, but what evasion strategy it attempts and where that strategy breaks down.

This paper addresses these problems by pairing a controlled adversarial evaluation suite with a DreamerV3 [33] world-model agent whose reward deliberately excludes pedestrian-distance, TTC, and yielding terms. Existing evaluation paradigms address parts of this problem in isolation: adversarial testing and falsification methods [21,24,26] search for worst-case failures but typically do not categorize them by interaction type, while scenario-based benchmarks [16] cover nominal traffic without adversarial pedestrian stress. The present framework combines controlled archetype-specific adversarial scenarios with multi-axis diagnostic decomposition, positioning it between worst-case search and scripted scenario suites. To create reproducible adversarial stress, we introduce the Jaywalkers Library, a CARLA-integrated benchmark comprising three pedestrian archetypes (Intruder, Indecisive Crosser, and Protester) defined by observable motion and geometry rather than assumed psychology. Training uses standard CARLA traffic in which adversarial-like pedestrian interactions occur only incidentally and at low frequency; evaluation then raises both the frequency and severity of these interactions to controlled stress levels using a frozen policy. Rather than reporting only collision rates, we decompose behavior into endpoint safety outcomes, safety-margin distributions, evasion dynamics, and efficiency costs.

Results show that the agent develops archetype-sensitive evasion: it maintains speed and swerves against dynamic pedestrians, while decelerating and attempting a lateral bypass against the slow-moving Protester. Collision rates reveal an unexpected difficulty ordering in which the slow-moving obstruction is hardest, exposing a world-model prediction failure against persistent obstacles whose non-clearing behavior is underrepresented in training. These findings demonstrate that a sparse terminal penalty produces emergent, adaptive pedestrian-avoidance behavior while identifying the specific prediction bottleneck that limits its effectiveness.

The main contributions of this paper are:

• frozen-policy evaluation protocol: a controlled stress-test design with explicit train–test separation, pre-specified metrics, and archetype-specific adversarial scenarios spanning collision pressure and deadlock pressure conditions;
• diagnostic decomposition of safety behavior: a multi-axis evaluation framework that separates endpoint outcomes from evasion dynamics (lateral offset, speed drop, collision speed), enabling mechanism-focused analysis beyond collision rate alone;
• Jaywalkers Library: a novel CARLA-integrated adversarial pedestrian suite with three archetypes (Intruder, Indecisive Crosser, Protester) that serves as the experimental apparatus enabling the above analysis;
• observation space design: a compact three-channel image representation (inverse depth, risk-ordered semantic segmentation, route projection) that foregrounds collision-relevant structure without task-specific feature engineering.

Applying this framework, we find that the agent develops archetype-sensitive evasion strategies despite receiving no explicit pedestrian-safety supervision.

The remainder of the paper is organized as follows. Section 2 defines the Jaywalkers Library and its three adversarial archetypes. Section 3 presents the methodology: agent architecture, reward design, training, and evaluation protocol. Section 4 reports results for driving competence, per-archetype adversarial evaluation, and cross-scenario comparison. Section 5 interprets the emergent evasion strategy, analyzes the distinct failure mechanisms exposed by each archetype, and discusses reward design and world-model limitations. Section 6 concludes.

2. Jaywalkers Library: Adversarial Pedestrian Benchmark

The Jaywalkers Library is a specialized scenario suite designed to stress-test autonomous pedestrian interaction within the CARLA simulator [13] (Car Learning to Act, an open-source high-fidelity urban driving simulator providing configurable road networks, sensor models, and traffic scenarios for autonomous driving research). Unlike standard evaluation protocols that rely on cooperative or weakly stochastic agents, the benchmark is strictly interaction-centric: each behavior is defined by deterministic geometric constraints and precise timing rules rather than high-level intent labels or latent psychological states. It is implemented as a runtime scenario layer that controls pedestrian behavior independently of the driving policy; the policy receives no privileged intent annotations and must infer risk from observations alone. This design targets a practical evaluation question: how does a closed-loop driving policy behave when pedestrians create short reaction windows, ambiguous crossing-state transitions, or prolonged route blockage? Three principles guide the library’s construction: interaction-driven specification, where scenarios are defined through observable state transitions such as gap timing, crossing phase, and blockage persistence rather than intent assumptions; late risk emergence, where critical risk appears only after initially benign motion; and distinct failure-pressure modes, spanning both collision pressure and deadlock pressure. The three archetypes map to principal modes of adversarial pedestrian–vehicle interaction identified in the traffic safety literature: sudden gap exploitation, the most frequent pre-crash pedestrian scenario [6]; hesitation and freeze–resume behavior, a documented source of driver uncertainty at unsignalised crossings [43,44]; and deliberate roadway occupation, reflecting the strategic dominance pedestrians gain over yield-obligated autonomous vehicles [45]. The current benchmark contains three adversarial archetypes (Intruder, Indecisive Crosser, and Protester), each implemented as an independent behavior controller attached to CARLA walker actors (Figure 1), with per-archetype parameter ranges summarized in Table 1. All parameter ranges are grounded in empirical pedestrian behavior data, spanning the adversarial tail of observed distributions by sampling from the dangerous end of documented crossing speeds [46,47], evasive freeze–resume dynamics [43,44], gap acceptance thresholds [48], and pedestrian–autonomous-vehicle strategic interaction [45], rather than replicating any single study’s measurements. The Jaywalkers Library is used here primarily for evaluation through frozen-policy stress tests; it is not presented as an exhaustive human behavior model, but as a compact and reproducible adversarial suite for mechanism-focused safety analysis. The Indecisive Crosser uses a mixed profile in evaluation, alternating between safe and freeze behavioral modes episode-by-episode, producing within-archetype diversity while keeping a single evaluation condition. All three archetypes are evaluated in the training town (Town02) and in an unseen town (Town01); the unseen-town condition constitutes the out-of-distribution test.

Figure 1. Jaywalkers Library archetypes in CARLA: Intruder, Indecisive Crosser, and Protester. The benchmark spans short-window collision pressure, ambiguous crossing dynamics, and persistent blockage. Top-down view: blue rectangle = ego vehicle, filled circle = pedestrian.

Table 1. Jaywalkers Library parameterization by archetype.

Parameter	Intruder	Indecisive (Safe/Freeze)	Protester
Speed (m/s)	$[2.0,3.2]$	$[1.8,2.4]$/$[1.1,1.8]$	$2.0$
Depart lead time (s)	—	$[2.2,2.8]$/$[1.2,1.8]$	—
Freeze TTC (s)	—	$[0.7,1.0]$/$[1.2,1.8]$	—
Resume TTC (s)	—	$[1.2,1.6]$/$[2.0,2.6]$	—
Freeze duration (s)	—	$[0.3,0.6]$/$[0.7,1.3]$	—
Continue/block dist. (m)	$[5.0,8.0]$	$[5.0,8.0]$	—
Block duration (s)	—	—	$10.0$

Behavior Taxonomy

The Intruder is a pedestrian spawned near the ego route that commits to crossing at a designated anchor point A on the route. The commit timing is adversarially chosen: using closing-speed estimation, the pedestrian starts crossing at the moment that places it in the conflict zone simultaneously with the ego if the ego maintains current speed, leaving no deceleration margin. Once committed, the pedestrian walks through A and continues 5–$8\mathrm{m}$ beyond it at $2.0$–$3.2\mathrm{m}/\mathrm{s}$, a range that spans the upper walking-to-running transition observed in field studies, where normal crossing averages $1.2$–$1.4\mathrm{m}/\mathrm{s}$ and running pedestrians reach $1.5$–$4.0\mathrm{m}/\mathrm{s}$ [46,47]. The primary adversarial contribution lies in the commit-timing logic, which eliminates the deceleration margin that real-world gap acceptance normally preserves [48].

The Indecisive Crosser departs early, before the ego enters the tight conflict window, but may freeze mid-crossing when the ego TTC drops below a threshold. Two parameterized profiles alternate across episodes: a safe profile (speed $1.8$–$2.4\mathrm{m}/\mathrm{s}$, freeze TTC $0.7$–$1.0\mathrm{s}$, resume TTC $1.2$–$1.6\mathrm{s}$, freeze duration $0.3$–$0.6\mathrm{s}$) and a freeze profile (speed $1.1$–$1.8\mathrm{m}/\mathrm{s}$, freeze TTC $1.2$–$1.8\mathrm{s}$, resume TTC $2.0$–$2.6\mathrm{s}$, freeze duration $0.7$–$1.3\mathrm{s}$). These ranges are grounded in empirical pedestrian-avoidance data: Li et al. [43] measured backward-avoidance initiation at TTC $=1.6\mathrm{s}$ with a deceleration-to-stop phase of $\sim 0.73\mathrm{s}$, and Oxley et al. [44] reported age-dependent decision times of $0.6$–$1.4\mathrm{s}$. The pedestrian resumes only after TTC recovers above the resume threshold and the minimum freeze duration has elapsed, creating behavioral hysteresis. This non-monotonic freeze–resume dynamic tests replanning under sustained ambiguity.

The Protester moves to a designated blocking position on the ego route at $2.0\mathrm{m}/\mathrm{s}$, then stands still or repositions laterally to maintain obstruction for a configurable block duration (default $10\mathrm{s}$) before slowly leaving the roadway. The $10\mathrm{s}$ duration sits at the upper end of empirically observed pedestrian road-occupancy events; average waiting times at unsignalized crossings range from 5 to $8\mathrm{s}$ [48]. This duration is also consistent with the strategic dominance that pedestrians gain over yield-obligated autonomous vehicles, as analyzed by Millard-Ball [45]. This archetype shifts pressure from immediate collision avoidance to patience under sustained route blockage. Table 1 summarizes the parameter ranges for all three archetypes.

3. Methodology

This section describes the agent under evaluation, a DreamerV3-based driving agent with a compact multi-modal observation design and ego-frame route conditioning, together with the reward formulation, training configuration, and evaluation protocol.

DreamerV3 [33] is a model-based reinforcement learning algorithm that learns a compact latent world model from environment experience and trains an actor–critic entirely on imagined trajectories, without additional environment interaction during the learning phase. Its core is the Recurrent State Space Model (RSSM), which maintains a two-part latent state: a deterministic recurrent component $h_t$, updated from the previous state and the previous action, and a stochastic discrete component $z_t$, inferred from $h_t$ and the current encoded observation. From any latent state $(h_t,z_t)$, the world model unrolls imagined future trajectories under the current policy; the actor is trained to maximize predicted discounted returns over these imagined rollouts, and the critic provides value estimates. At inference, the frozen actor maps the current latent state to a continuous action without querying the environment model.

3.1. Observation Space

The learning algorithm is DreamerV3 [33], an established world-model RL method among the broader range of reinforcement learning approaches applied in CARLA [18]; the design contribution of this work lies in the observation representation fed to it. Rather than supplying raw camera images, we construct a three-channel input that encodes geometry, semantic risk, and route intent in forms that foreground collision-relevant information and suppress irrelevant visual variance. At time step t, the policy receives:

$$o_t\triangleq \left(I_t^{\mathrm{inv}},I_t^{\mathrm{seg}},I_t^{\mathrm{wp}},e_t,u_{t-1}\right),$$

(1)

where $I_t^{\mathrm{inv}}\in {[0,1]}^{H\times W}$ is scaled inverse depth, $I_t^{\mathrm{seg}}\in {[0,1]}^{H\times W}$ is risk-ordered semantic segmentation, $I_t^{\mathrm{wp}}\in {[0,255]}^{H\times W}$ is a route projection channel, $e_t=[v_t,a_t,r_t,e_{\mathrm{lat},t},e_{\psi ,t}]\in {\mathbb{R}}^5$ is ego-state (speed, longitudinal acceleration, yaw rate, lateral route error, heading error), and $u_{t-1}=[{\delta }_{t-1},{\ell }_{t-1}]\in {\mathbb{R}}^2$ is the previous action (steering and signed longitudinal command). As $o_t$ comprises heterogeneous inputs (image tensors and numerical vectors), it is represented as a structured tuple rather than a flat vector concatenation; Figure 2 illustrates the same components flowing left-to-right into the encoder, consistent with the assembly direction of Equation (1).

Figure 2. Designed observation-to-action pipeline. The CarlaJaywalkersEnv wrapper (left) encapsulates the CARLA simulator, the Jaywalkers Library, and a route planner. At each step the environment emits a multi-signal observation $o_t$ and a scalar reward $r_t$. The observation design is a contribution of this work: three image channels (inverse depth, risk-ordered segmentation, and a camera-projected route channel) encode geometry, semantic risk, and navigation intent in a compact representation that foregrounds collision-relevant structure. These are concatenated (⊕) and fed alongside numerical ego-state and previous action to DreamerV3, which applies a shared CNN and multilayer perceptron (MLP) encoder, maintains Recurrent State Space Model (RSSM) latent state $(h_t,z_t)$ via world-model imagination, and drives an actor–critic that produces the steering and longitudinal action $({\delta }_t,{\ell }_t)$, fed back to the simulator. The agent learns entirely from this closed loop; no explicit collision-avoidance rules are injected.

The route projection channel ($I_t^{\mathrm{wp}}$) encodes route intent as an image channel rather than a numerical vector. The route planner produces $K=10$ ego-frame waypoints with spacing $\Delta d=3\mathrm{m}$ ($30\mathrm{m}$ look-ahead). These waypoints are projected onto the camera view using perspective geometry and rendered as connected line segments with dot markers at each waypoint, producing a single-channel $H\times W$ image $I_t^{\mathrm{wp}}\in [0,255]$. This visual encoding is concatenated with inverse depth and segmentation as a third image channel, allowing the convolutional neural network (CNN) encoder to jointly extract spatial features from geometry, semantics, and route intent.

The scaled inverse depth channel ($I_t^{\mathrm{inv}}$) transforms depth to emphasize near-field geometry:

$$I_t^{\mathrm{inv}}(u,v)={\displaystyle \frac{\frac{1}{d_t(u,v)}-\frac{1}{d_{max}}}{\frac{1}{d_{min}}-\frac{1}{d_{max}}}}\in [0,1].$$

(2)

Here $(u,v)$ are pixel coordinates, $d_t(u,v)>0$ is the raw depth at pixel $(u,v)$, and $d_{min}$, $d_{max}$ are the sensor’s near and far clipping distances. This allocates representational precision where collision risk is concentrated.

The risk-ordered segmentation channel ($I_t^{\mathrm{seg}}$) maps semantic labels to a single-channel risk ordering. Critically vulnerable classes are assigned high intensities (e.g., pedestrian as the highest level), while drivable/safe background classes are assigned low values (e.g., roads near the minimum). The map is fixed across all runs. This encoding provides a structured prior on what is dangerous without encoding what to do; no intent or future-state labels are injected. Raw RGB was not used: at $128\times 128$, it carries appearance variation from lighting, shadows, and surface texture that is irrelevant to collision risk. The world model would spend representational capacity compressing this variance rather than learning task-relevant dynamics. Semantic segmentation eliminates this source of noise, following established practice in simulation-based driving RL [49,50].

For encoder fusion and world-model integration, the three image channels (inverse depth, segmentation, and route projection) are encoded by a shared CNN and fused with numerical channels:

$$z_t=f_{\theta }\left(I_t^{\mathrm{inv}},I_t^{\mathrm{seg}},I_t^{\mathrm{wp}},e_t,u_{t-1}\right),$$

(3)

where $f_{\theta }$ is the CNN encoder with learned parameters $\theta$, and $z_t$ is the latent state used for imagination rollouts and actor–critic optimization in DreamerV3. The complete observation-to-action pipeline is illustrated in Figure 2.

3.2. Action Space

The policy outputs a 2D continuous action: steering ${\delta }_t\in [-1,1]$ and a signed longitudinal command ${\ell }_t\in [-1,1]$. At execution, throttle and brake are computed as $g_t=\max ({\ell }_t,0)$ and $b_t=\max (-{\ell }_t,0)$.

3.3. Reward Function

The step reward combines a dense gate-based progress signal with lightweight regularization:

$$r_t=r_{\mathrm{gate},t}-k_{\mathrm{speed}}{p}_{\mathrm{speed},t}+r_{\mathrm{terminal},t}.$$

(4)

where $r_{\mathrm{gate},t}\in \mathbb{R}$ is the gate-based route-progress reward, $k_{\mathrm{speed}}>0$ is the speed-penalty weight, $p_{\mathrm{speed},t}\ge 0$ is the quadratic speed-excess penalty, and $r_{\mathrm{terminal},t}\in \mathbb{R}$ is the sparse terminal reward signal.

Route progress uses a gate mechanism: virtual gates are placed every $2\mathrm{m}$ along the route, and each gate collected yields a reward that decreases linearly from $5.0$ at the route center to $0.5$ at the gate boundary ($0.85\mathrm{m}$ lateral half-width). Speed regulation applies a quadratic penalty above $30\mathrm{km}/\mathrm{h}$; below this threshold no speed penalty is applied.

Terminal rewards assign $+100$ for reaching the goal, $-50$ for timeout, $-100$ for leaving the route or colliding with a vehicle, and $-300\varphi \left(v_t\right)$ for a pedestrian collision, where the quadratic speed factor $\varphi \left(v_t\right)=1+{(v_t/30)}^2$ (with $v_t$ in km/h) escalates the penalty at higher speeds. The magnitudes follow two design principles: the goal reward ($+100$) must dominate the cumulative progress reward along a typical $200\mathrm{m}$ route (approximately $+100$ from gates) to make route completion clearly preferable to stalling, and the pedestrian collision penalty ($-300$ base, up to $-600$ at target speed) must be the single largest negative signal in the system, substantially exceeding off-route and vehicle collision penalties, so that the world model learns pedestrian collisions as the worst possible outcome even under sparse supervision.

Regarding the sparse pedestrian-safety design: the reward intentionally excludes all commonly used pedestrian-safety shaping: distance-to-pedestrian penalties, TTC-based shaping terms, explicit yielding or braking incentives, and lateral-deviation, heading-error, lane-invasion, and no-progress penalties (all set to zero weight). The key design choice is that pedestrian safety receives only a sparse, terminal signal: the $-300\varphi \left(v_t\right)$ penalty triggers once at collision and provides no gradient over approach distance, closing speed, or time-to-collision. This formulation is the defining design choice this study is built to examine. We deliberately train with sparse pedestrian-safety supervision to study what avoidance strategy emerges, and then stress-test that emergent strategy against adversarial jaywalking scenarios the agent has never encountered.

3.4. Training Environment and Pedestrian Exposure

The agent is trained in a single DreamerV3 run in Town02 with the configuration detailed in Table 2. Each training episode follows a randomly generated route of approximately $200\mathrm{m}$, drawn from the Town02 road network with random turns at intersections. The training environment includes non-adversarial crossing pedestrians that periodically cross roads at $0.3$–$2.0\mathrm{m}/\mathrm{s}$. These pedestrians are spawned at fixed distances ahead of the ego vehicle ($30,60,90,120,150,180\mathrm{m}$) and cross $8\mathrm{m}$ laterally, waiting 2–$4\mathrm{s}$ at each side before crossing again. Additional pedestrians walk only on sidewalks, providing background foot traffic; approximately $10\%$ of all spawned pedestrians are stationary.

Table 2. Training configuration and core hyperparameters.

Setting	Value
Image resolution/field of view (FOV)	$128\times 128$/${100}^{\circ }$
Action repeat	2
Dynamics hidden/deter/stoch/discrete	512/512/32/32
Encoder CNN depth/kernel/minres	32/4/4
Encoder MLP layers × units	$5\times 1024$
Batch size/sequence length	16/64
Model/actor/critic learning rate (LR)	$1\times {10}^{-4}$/$3\times {10}^{-5}$/$3\times {10}^{-5}$
Imagination horizon	32
Discount/$\lambda$	0.997/0.95
Training steps	${10}^7$
Simulation step/effective control rate	0.05 s/10 Hz

These pedestrians occasionally produce adversarial-like interactions by coincidence (e.g., crossing at an inopportune moment), but they have no adversarial timing, no ego-reactive behavior, and no deliberate route blockage. The three Jaywalker archetypes used for evaluation are never injected during training. The policy is frozen after training and evaluated without any test-time learning or adaptation. All results reported in this paper use a single frozen checkpoint. The training convergence curve is shown in Figure 3. The full training configuration is listed in Table 2.

Figure 3. Training convergence of the DreamerV3 main run over $\sim 1.4\mathrm{M}$ environment steps. Raw episode return (faint dots) and exponential moving average (EMA)-smoothed return (bold line, $\alpha =0.99$). The checkpoint for all frozen-policy evaluations was selected from the stabilized region.

3.5. Evaluation Protocol

The frozen policy is evaluated in two phases: a driving-competence baseline without pedestrians, and a zero-shot adversarial jaywalker evaluation using the three archetypes from the Jaywalkers Library. The evaluation targets three objectives:

• Identify the pedestrian-avoidance strategy that emerges when a world-model RL agent is trained with only sparse collision penalties in a pedestrian-populated urban environment.
• Assess the robustness of this emergent avoidance behavior against a taxonomy of increasingly adversarial jaywalking scenarios (Intruder, Indecisive Crosser, Protester).
• Characterize the failure modes that appear and the limitations of sparse-penalty reward design they reveal for pedestrian safety.

Phase 0: Driving competence baseline. Phase 0 establishes that the agent can drive competently before we study its response to adversarial pedestrians. Two conditions are evaluated without any pedestrians: the training town (Town02, 1000 episodes) and an unseen town (Town01, 1000 episodes). High route completion in both towns confirms basic driving competence; comparable performance across towns indicates world-model generalization to novel road geometry rather than route memorization.

Phase 1: Adversarial jaywalker evaluation. All jaywalker scenarios are run on Town02 with the jaywalker archetype as the independent variable. Each evaluation episode uses a $60\mathrm{m}$ route, and the jaywalker is spawned $45\mathrm{m}$ ahead along the route, ensuring a pedestrian encounter in every episode. A total of 1000 episodes are run per-archetype. The agent has never encountered any of these archetypes during training.

3.6. Metrics and Operational Definitions

Driving competence is measured by route completion (%), mean speed (km/h), off-route termination rate (%), and collision rate with vehicles or static objects (%). Pedestrian safety is assessed by pedestrian collision rate (%), near-miss count (minimum TTC $<1.5\mathrm{s}$ without collision), off-route termination rate (%), median minimum TTC, and TTC distributions across encounters. Throughout this paper, time-to-collision is computed as ${\mathrm{TTC}}_t=D_t/v_t$, where $D_t$ is the Euclidean distance from the ego vehicle to the nearest pedestrian and $v_t$ is the ego speed at step t. This is a conservative, worst-case proximity indicator that does not account for pedestrian velocity or heading; it bounds the collision time from below, overestimating danger when the pedestrian is already clearing the ego’s path.

For proportions (collision rate, near-miss rate), 95% Wilson confidence intervals are reported. For the TTC median, bootstrap confidence intervals are computed from $10,000$ resamples. Pairwise differences are tested with ${\chi }^2$ tests (proportions) and Mann–Whitney U tests (TTC distributions), with overall TTC differences assessed via Kruskal–Wallis.

An encounter is an episode segment where ego-to-pedestrian TTC drops below $3.0\mathrm{s}$ (the conflict window). Episodes where TTC never drops below $3.0\mathrm{s}$ are non-encounters, meaning the pedestrian self-resolved before any conflict materialized. The evasion strategy during each encounter is classified by two vehicle-dynamics measures applied within the conflict window: maximum lateral offset from the route centreline and maximum speed drop from the pre-encounter speed. These produce four categories: lateral only, braking only, combined (both present), and none (neither present). Classification uses vehicle dynamics, not raw action commands: the DreamerV3 actor operates at high frequency and routinely saturates the $[-1,+1]$ action bounds during normal driving, so vehicle dynamics, filtered by physical inertia, provide a more faithful readout of what the car actually did. Both quantities are also reported jointly as a continuous evasion dynamics distribution (Figure 4), computed for all encounters including those ending in collision.

Figure 4. Evasion dynamics across all 2773 pedestrian encounters, split by adversarial archetype. Each hexbin shows the joint density of maximum lateral offset from the route centreline (x-axis) and maximum speed drop from pre-encounter speed (y-axis). Collision episodes are overlaid as individual crosses. The Protester panel shows the widest spread along both axes, reflecting combined lateral-and-longitudinal evasion; the density for dynamic archetypes concentrates in the low-speed-drop region.

4. Results

This section presents the frozen-policy evaluation results. We first establish driving competence (Section 4.1), then report the adversarial jaywalker evaluation with aggregate and per-archetype metrics (Section 4.2), and finally present the cross-scenario comparison across all encounters (Section 4.3).

4.1. Driving Competence Baseline

Figure 3 shows the training curve for the main DreamerV3 run. Episode return stabilizes after approximately $\sim 1.4\mathrm{M}$ training steps, confirming that the evaluated checkpoint represents a converged policy. Table 3 reports Phase 0 results: the agent achieves high route completion in both the training town (Town02) and an unseen town (Town01), with no pedestrian-related failures (no pedestrians are present), confirming basic driving competence and world-model generalization to novel road geometry.

Table 3. Driving competence baseline (Phase 0, no pedestrians, 1000 episodes each). High completion in both towns confirms competence and generalization.

Condition	Town	Completion (%)	Speed (km/h)
Seen town	Town02	98.9	24.9
Unseen town	Town01	100.0	25.1

4.2. Adversarial Jaywalker Evaluation

Table 4 summarizes the Phase 1 evaluation across all three adversarial archetypes. Collision rates range from $25.2\%$ (Indecisive Crosser) to $51.7\%$ (Protester), with the Intruder at $40.0\%$. Median minimum TTC varies from $0.25\mathrm{s}$ (Protester) to $0.87\mathrm{s}$ (Indecisive Crosser). Off-route rates are uniformly low across all archetypes ($1.3$–$1.9\%$), confirming route-following competence under adversarial stress.

Table 4. Adversarial jaywalker evaluation results (Phase 1, 1000 episodes each). NM/ep = near-misses per episode; TTC_med = median minimum time-to-collision over encounters with TTC data. Bracketed values are 95% confidence intervals (Wilson for proportions, bootstrap for TTC median).

Archetype	Collision (%)	Off-Route (%)	NM/ep	TTCmed (s)	Completion (%)	Speed (km/h)
Intruder	40.0 [37.0, 43.1]	1.9	0.48 [0.45, 0.51]	0.47 [0.42, 0.52]	54.5	22.5
Indecisive Crosser	25.2 [22.6, 28.0]	1.3	0.45 [0.42, 0.48]	0.87 [0.80, 0.91]	71.0	23.2
Protester	51.7 [48.6, 54.8]	1.9	0.39 [0.36, 0.42]	0.25 [0.21, 0.28]	39.9	21.4

The Intruder times its crossing to arrive at the ego’s path simultaneously with the vehicle. The $40.0\%$ collision rate is accompanied by a median minimum TTC of $0.47\mathrm{s}$, and $32.5\%$ of encounters produce no detectable evasion response; when evasion is initiated, combined lateral-and-longitudinal maneuvers dominate, with a median speed drop of $3.9\mathrm{km}/\mathrm{h}$ and route completion of $54.5\%$. The Indecisive Crosser departs earlier but introduces trajectory uncertainty through its freeze–resume behavior, producing the lowest collision rate ($25.2\%$), highest route completion ($71.0\%$), and a median minimum TTC of $0.87\mathrm{s}$; the freeze–resume pattern still produces elevated near-miss rates ($0.45$ per episode), and in $40.0\%$ of encounters no active evasion is detected, with the lowest median speed drop across archetypes at $2.9\mathrm{km}/\mathrm{h}$. The Protester enters the lane and moves slowly to obstruct the ego vehicle, never self-clearing the path; it produces the highest collision rate ($51.7\%$), lowest median minimum TTC ($0.25\mathrm{s}$), and lowest route completion ($39.9\%$), but also elicits the strongest evasion effort with a median speed drop of $8.8\mathrm{km}/\mathrm{h}$, and collision speeds are the lowest of all archetypes, indicating that the agent decelerates substantially before impact.

Collision and near-miss outcomes are binary, so non-parametric proportion tests apply directly. For TTC, Kolmogorov–Smirnov tests reject normality across all archetypes, motivating the use of rank-based tests. All pairwise collision-rate differences are statistically significant (${\chi }^2$ test, $p<{10}^{-7}$ for all pairs). TTC distributions differ significantly across all three archetypes (Kruskal–Wallis $H=341.5$, $p<{10}^{-74}$; pairwise Mann–Whitney $p<0.001$ for all pairs). Near-miss rates differ significantly between the Protester and the other two archetypes ($p<0.01$), but the Intruder–Indecisive Crosser difference is not significant ($p=0.12$), indicating that despite different collision outcomes, these two archetypes produce comparable near-miss exposure.

4.3. Cross-Scenario Comparison

Figure 4 shows the joint distribution of lateral offset and speed drop for all 2773 encounters, split by archetype. The density concentrates in the low-speed-drop region for all archetypes, but the Protester panel shows the widest spread along the speed-drop axis, consistent with combined lateral-and-longitudinal evasion; collision episodes (crosses) appear in all panels, with the Protester panel showing the highest concentration. The minimum TTC distributions in Figure 5 reveal further archetype-dependent structure: the Intruder distribution is compressed near zero, reflecting adversarial timing that eliminates the reaction window; the Indecisive Crosser shows a wider spread with a substantial fraction above the $1.5\mathrm{s}$ near-miss threshold, consistent with its earlier departure providing more temporal margin; and the Protester distribution clusters at very low TTC values ($0.25\mathrm{s}$ median), reflecting collisions that occur at close proximity during the bypass attempt despite prior deceleration. Figure 6 shows that Protester collisions occur at the lowest speeds among all archetypes, confirming that the agent decelerates before impact, while Intruder collisions occur at the highest speeds, consistent with timing-compressed encounters where the agent has little opportunity to reduce speed. Representative trajectories in Figure 7 illustrate these patterns qualitatively: against the Intruder and Indecisive Crosser the agent executes tight lateral swerves at near-cruise speed, while against the Protester it decelerates and attempts a wider lateral bypass, with collisions occurring at low speed during the bypass attempt. The collision rate ordering across archetypes does not follow intuitive difficulty expectations (Table 4): the slow-moving, predictable Protester produces the highest collision rate ($51.7\%$), followed by the adversarially timed Intruder ($40.0\%$), with the Indecisive Crosser proving most survivable ($25.2\%$). This ordering and its implications are examined in the Discussion.

Figure 5. Distribution of minimum time-to-collision (TTC) per encounter across adversarial archetypes. Horizontal lines indicate the near-miss threshold ($1.5\mathrm{s}$) and critical threshold ($1.0\mathrm{s}$). The Intruder distribution is compressed near zero, reflecting adversarial timing that eliminates the reaction window.

Figure 6. Split-violin plot of ego speed during collision episodes, per-archetype. Left half (blue): speed at closest approach (min-TTC moment); right half (orange): speed at impact. Horizontal lines show medians. Protester collisions occur at the lowest speeds; Intruder collisions at the highest.

Figure 7. Representative evasion and collision trajectories for each adversarial archetype ($3\times 2$ grid). Each row is one archetype; left column: successful evasion, right column: collision. Paths are colored by time (viridis); gold markers indicate the min-TTC pair. (a) Intruder evasion: tight swerve at near-cruise speed. (b) Intruder collision: the pedestrian arrives before lateral clearance is achieved. (c) Indecisive Crosser evasion: agent deviates laterally while maintaining speed. (d) Indecisive Crosser collision: freeze–resume transition narrows the evasion window. (e) Protester evasion: agent decelerates and executes a wide lateral bypass. (f) Protester collision: agent yields but collides during the bypass attempt. Quantitative coverage across all 2773 encounters is in Figure 4.

5. Discussion

5.1. Emergent Evasion as a Reward-Rational Policy

The evaluation reveals an archetype-sensitive evasion strategy: against dynamic pedestrians (Intruder, Indecisive Crosser), the agent maintains near-cruise speed and relies on lateral displacement, while against the slow-moving Protester it decelerates substantially (median speed drop $8.8\mathrm{km}/\mathrm{h}$ versus $2.9$–$3.9\mathrm{km}/\mathrm{h}$ for dynamic archetypes; Table 4) and occasionally stops before attempting a lateral bypass. Figure 4 confirms that the majority of encounters concentrate in the low-speed-drop region, with an overall median of $4.7\mathrm{km}/\mathrm{h}$ ($\sim 16\%$ of cruise speed). This modulation is the expected outcome of a deliberate methodological choice: as discussed in Section 3.3, the only pedestrian-specific signal is a sparse terminal collision penalty, with proximity shaping, TTC penalties, and yielding incentives all excluded. Any dense safety term would make it impossible to attribute avoidance to world-model predictions rather than injected rules, so any evasive behavior observed here must emerge from consequence prediction in latent imagination.

The dense progress reward is a prerequisite for driving: without it, the agent has no incentive to advance between sparse terminal events. This creates a structural opportunity cost for yielding: each second stopped is progress foregone, while the collision penalty only pays off when a collision actually occurs. Against training distribution pedestrians that cross and clear, evasion without stopping is almost always safe, so the world model rarely imagines a collision from maintaining speed. The agent therefore converges on non-yielding evasion as its default strategy.

Against the Protester, however, the agent departs from this default strategy: its world model assigns high enough collision probability to override the progress incentive, producing substantial deceleration. The failure is not that the agent refuses to yield; it does yield. Rather, the world model predicts the pedestrian will eventually clear the lane, consistent with transient road-stopping pedestrians seen during training, and the agent commits to a lateral bypass through a gap that never opens (Figure 7f).

This opportunity-cost mechanism generalizes beyond the gate formulation used here: any dense progress reward creates the same trade-off between advancement and safety. Under sparse terminal supervision, the agent calibrates its longitudinal response to estimated collision risk, succeeding when that estimate is accurate and failing when persistent obstacles are misclassified as transient.

5.2. How Adversarial Behaviors Expose Distinct Failure Mechanisms

The non-obvious collision rate ordering (Protester > Intruder > Indecisive Crosser; Table 4) reveals that each archetype exposes a structurally distinct failure mechanism.

The Protester produces the highest collision rate ($51.7\%$) despite being the most perceptually salient threat, exposing a world-model prediction failure. The agent does recognize and respond to the slow-moving lane-center obstruction: it deploys its strongest evasion effort (largest median speed drop $8.8\mathrm{km}/\mathrm{h}$) and occasionally stops entirely. The failure is not in detection or braking but in prediction: the agent’s world model appears to forecast that the pedestrian will eventually clear the lane, consistent with the transient road-stopping pedestrians seen during training, and commits to a lateral bypass through a gap that never opens. This is the most diagnostic finding of the study: it demonstrates that a world-model agent can correctly perceive a threat, respond with appropriate deceleration, and still fail because its learned dynamics model misrepresents pedestrian persistence. Figure 6 reinforces this interpretation: Protester collisions occur at the lowest speeds, confirming that the agent has already decelerated before impact: the collision occurs during the bypass attempt, not from failure to brake.

The Intruder’s adversarial timing exposes a reaction-time failure, producing a $40.0\%$ collision rate driven by insufficient reaction time rather than strategic inadequacy. The agent survives the majority of encounters ($54.5\%$ completion), but $32.5\%$ of encounters produce no detectable evasion, indicating they unfold too rapidly for any response. This is a geometric failure: when the pedestrian arrives simultaneously with the vehicle, no evasion can succeed regardless of strategy. Collision speeds are the highest across archetypes, as shown in Figure 6, consistent with the absence of a deceleration window, and the TTC distribution in Figure 5 confirms that most Intruder encounters fall below the critical threshold.

The Indecisive Crosser proves most survivable ($25.2\%$ collision, $71.0\%$ completion) largely because its encounters are partly self-resolving. The freeze–resume pattern creates prediction errors, but the earlier departure provides enough temporal margin for evasion to succeed in most cases. The $40.0\%$ no-evasion rate reflects encounters where the pedestrian’s own resume phase clears the threat before the agent needs to react, making the Indecisive Crosser partly cooperative by accident.

5.3. Reward Design and World-Model Limitations

These results provide empirical grounding for a qualified reward design principle: a dense progress reward combined with a sparse terminal collision penalty produces an agent that calibrates its evasion response to perceived threat level, but whose effectiveness is bounded by the accuracy of its learned world model. The Protester’s $51.7\%$ collision rate (Table 4) reveals that the world model’s prediction of pedestrian persistence is the binding constraint: the agent yields but misjudges how long the pedestrian will remain, committing to a bypass through a gap that never opens.

The bottleneck for safety is not the reward function alone but the interaction between reward design and world-model fidelity. As Figure 4 shows, the agent’s willingness to decelerate against the Protester demonstrates that the sparse penalty can incentivize yielding when the perceived collision risk is high enough. What fails is the world model’s estimate of how long the risk persists, a training distribution problem, not a reward design problem. This suggests that curriculum design may be more effective than reward shaping for improving Protester performance: training with more persistent lane-blocking pedestrians would directly improve the world model’s prediction of non-clearing behavior, without introducing the interpretability cost of dense safety shaping.

The Protester failure suggests two non-exclusive remedies. Curriculum-based approaches would expose the agent to persistent lane blockages during training, improving world-model fidelity for non-clearing obstacles. Reward-based approaches (TTC penalties, yielding incentives) would directly penalize close-proximity operation, potentially producing conservative braking regardless of world-model accuracy. The trade-off is interpretability: a shaped agent may brake correctly, but its braking cannot be attributed unambiguously to world-model consequence prediction rather than reward injection. The present study establishes the unshaped baseline, characterizing what emerges in the absence of such terms and identifying the specific world-model prediction failure that future interventions should target.

5.4. Limitations

This study has several limitations that scope the findings and point to natural extensions:

• Single training run: results characterize one DreamerV3 agent trained with a single random seed; exact numerical rates may vary across runs. The experimental infrastructure supports replication, and multi-seed evaluation is a direct next step. To support reproducibility, all training hyperparameters are reported in Table 2, the evaluation uses a single frozen checkpoint across all 3000 adversarial episodes (1000 per-archetype), and the Jaywalkers Library scenarios are fully deterministic given their parameter seeds.
• Simulator-bound evaluation: the study is conducted entirely in CARLA Town02 and Town01. Beyond the standard sim-to-real gap, three aspects are particularly susceptible: pedestrian detection range (dependent on simulated camera optics), the TTC metric (dependent on simulated kinematics), and swerve dynamics (dependent on tire and chassis models absent from CARLA’s default physics). Real-world transfer would require validation on each of these axes independently.
• No retraining ablations: adding TTC shaping or yielding incentives to verify that braking emerges under richer supervision would further strengthen the reward-design argument. Such ablations are a natural next step; the present contribution is the characterization of the sparse-supervision baseline that those ablations would extend from.
• Scripted-only pedestrian archetypes: the three jaywalker archetypes follow pre-scripted trajectories and do not react to the ego vehicle. Reactive pedestrians, ones that freeze or yield because they detect an approaching car, introduce an additional feedback loop that may elicit qualitatively different avoidance behaviors, and represent a structurally important gap in the current taxonomy.
• No model-free baseline: without a model-free comparison (e.g., PPO or SAC trained under the same reward contract), it is not possible to isolate the contribution of world-model planning to the observed archetype-sensitive evasion strategy. The behavior may be a consequence of the reward signal alone rather than a product of world-model consequence prediction; a matched model-free baseline would resolve this ambiguity.

6. Conclusions

This paper examined whether a DreamerV3 world-model agent, trained with a sparse terminal collision penalty as its only pedestrian-specific supervision, develops effective collision avoidance when confronted with adversarial jaywalkers in CARLA. To support this investigation, we introduced the Jaywalkers Library, a configurable benchmark comprising three adversarial pedestrian archetypes (Intruder, Indecisive Crosser, and Protester) grounded in documented real-world jaywalking patterns.

The agent achieves near-perfect route completion under nominal conditions and generalizes to an unseen town, confirming basic driving competence. Under adversarial stress, an archetype-sensitive evasion strategy emerges without any explicit evasion incentive: the agent swerves at near-cruise speed against dynamic pedestrians while decelerating substantially against the slow-moving Protester. The collision rate ordering is counterintuitive: the Protester causes the highest collision rate ($51.7\%$), followed by the Intruder ($40.0\%$), with the Indecisive Crosser proving most survivable ($25.2\%$), and each archetype exposes a different failure mechanism: a world-model prediction failure against the Protester, a reaction-time failure against the Intruder, and self-resolving encounters against the Indecisive Crosser.

These findings demonstrate that a sparse terminal penalty is sufficient for emergent, adaptive pedestrian avoidance in a world-model agent, but that the agent’s effectiveness is bounded by its world model’s ability to predict pedestrian persistence. The primary bottleneck is training distribution rather than reward design, suggesting that, within simulation-based training, curriculum-based interventions (exposing the agent to persistent lane blockages during training) may be more effective than dense reward shaping for improving safety against non-clearing obstacles. The Jaywalkers benchmark provides a new mechanistic diagnostic needed to reveal these failure modes, which would remain invisible under standard held-out traffic evaluation.