Open-World Critical Scenario Recognition and Maneuver-Level Generation for Autonomous Driving Simulation Testing ^†

Abstract

As autonomous driving moves toward large-scale deployment, controllable and efficient simulation testing has become a primary means of ensuring system safety. However, in open-world environments, existing scenario catalogs often fail to cover the full spectrum of potential traffic situations, while rare yet high-risk critical scenarios are even harder to obtain. This scarcity renders traditional random sampling and parameter-sweeping strategies ineffective for identifying unknown risks. This study addresses two core challenges: (1) incomplete scenario catalogs hindering unknown critical scenario recognition and (2) insufficient critical samples, where generated scenarios struggle to balance physical realism and edge case coverage. To tackle the first challenge, we propose an open-world recognition method integrating transformers, random forests, and extreme value theory for precise unseen sample detection. Outlier and validity filtering ensure clustering reliability, and random forest activation patterns cluster unknown samples into meaningful groups to expand the scenario catalog. Experiments show the overall F1_macro improved by 2.3 percentage points over SOTA MDENet, with its clustering accuracy surpassing iterative-AutoNovel by 6.2 percentage points. For the second challenge, we introduce a reinforcement-learning-based maneuver-level generation method. It extracts maneuver semantics from trajectories, constructs a low-dimensional parameter space, and models parameter correlations via a multivariate multimodal distribution. A dual-layer LSTM agent with a composite reward iteratively optimizes policies toward high-risk edge scenarios. The results outperformed RLBE; longitudinal and lateral reconstruction errors were reduced by 32.7% and 15.3%, respectively, while high-risk time steps and the collision rate increased by 4.3% and 5.1%, respectively. Finally, we develop a CARLA-based scenario-driven simulation framework, integrating recognized and generated scenarios into closed-loop testing on high-risk road segments. CAS failure cases validate the generated scenarios’ physical feasibility and extreme challenge. Targeted augmentation of scarce critical scenarios enriches the test library and ensures broader coverage of real-world driving conditions.

1. Introduction

1.1. Background: Safety-Critical Scenarios in Autonomous Driving

The rapid advancement of deep learning and vehicle-infrastructure integration has expedited the shift from assisted to conditionally autonomous driving. However, this progress conflicts with society’s near-zero tolerance for safety failures and the high cost of real-world validation. A RAND Corporation report [1] showed that proving the statistical safety of autonomous vehicles via real-world mileage alone requires  880 million kilometers of testing, equivalent to centuries of continuous driving. This impracticality has driven academia and industry to adopt scenario-based testing (SBT), which uses a limited set of representative critical and edge scenarios to replace exhaustive road testing.

Traditional validation methods each have inherent limitations. Open-road tests are costly, non-repeatable, and dominated by long-tail events (<0.1% critical scenarios), and scripted track tests lack complex multi-agent interactions, while virtual simulation enables scalable, controllable execution at low cost. Platforms like CARLA provide high-fidelity environments and flexible scenario customization, making them ideal for safety-critical evaluation [2].

Despite these benefits, conventional simulation-based pipelines rely heavily on manual scripting or random perturbations, failing to cover emerging long-tail risks in open-world settings. Real-world traffic is inherently dynamic; road geometry, agent behaviors, and driving patterns evolve constantly, leading to unknown risk scenarios unrecorded in existing catalogs. Existing rule-based or fixed-label recognition methods depend on predefined scenario categories, often resulting in overconfident misclassifications when facing novel high-risk patterns [3]. While open-set recognition (OSR) has been explored in computer vision [4], its direct application to multi-agent spatiotemporal trajectories faces major challenges due to high dimensionality, strong temporal coupling, and long-tail imbalance. These issues underscore the need for an open-world scenario recognition framework that extracts robust motion semantics and statistically models extreme events.

1.2. Gaps in Current Scenario Recognition and Generation

Even when critical scenarios are identified, their scarcity remains a bottleneck for system robustness validation. Existing scenario generation methods often rely on manual rules, trajectory perturbations, or random parameter sampling, failing to capture the semantic structure of driving behavior and potentially generating physically implausible trajectories. Early optimization-based methods (e.g., genetic algorithms [5] or Bayesian optimization [6] targeting a low time to collision (TTC)) struggle with high-dimensional trajectory-level decision variables and tend to converge to suboptimal solutions. Recent work [7] shows that driving behaviors can be decomposed into atomic maneuver units (e.g., acceleration, braking, and lane changing), enabling low-dimensional maneuver parameterization. Yet, current methods have two key unresolved limitations: (1) recognition, as most methods assume closed-set scenario types and lack the ability to detect open-world “unknown” safety-critical cases, and (2) generation, as rule- or perturbation-based methods cannot effectively model nonlinear parameter dependencies, resulting in limited diversity and low realism. These limitations call for an integrated framework combining open-world recognition with data-driven maneuver-level generation, enabling automated catalog expansion and edge case synthesis in a unified pipeline.

1.3. Our Approach

To tackle these challenges, this study proposes a full recognition-generation-validation framework for open-world autonomous driving scenario testing. The main contributions are as follows:

• We design TRFE, a trajectory representation and feature extraction framework that uses transformers for spatiotemporal encoding, fuses random forests with extreme value theory for novelty detection, filters outliers and invalid samples, and performs online clustering to adaptively expand the scenario catalog.
• We introduce MCSG, a reinforcement learning (RL)-based scenario generator that decomposes trajectories into maneuver units, models a multimodal low-dimensional parameter space, and employs a dual-layer LSTM agent with a composite reward to search for physically feasible, high-risk edge scenarios.
• We build a CARLA-based scenario-driven simulation system forming a closed testing loop; recognized and generated scenarios run on high-risk road segments, and failure modes of the CAS obstacle avoidance system provide real-time feedback to optimize policies.

Together, these components form a unified “recognition-generation-simulation” workflow enabling systematic testing and automated expansion of critical scenario databases.

1.4. Paper Structure

The remainder is organized as follows. Section 2 reviews related work. Section 3 presents the methodology. Section 4 reports experimental results. Section 5 discusses limitations and future directions. Section 6 concludes the paper. This paper is an extended version of our conference paper [8].

2. Related Work

2.1. Critical Scenario Mining and Identification

Critical scenario identification discovers accident-correlated driving trajectories from real-world data. Existing methods fall into two categories: rule-based and data-driven.

Rule-based approaches rely on expert-defined catalogs or handcrafted features. They are simple to implement yet lack coverage and generalization, as expert knowledge cannot fully capture high-dimensional real traffic dynamics. Early studies used distance-based metrics and hierarchical clustering [9,10] but focused on ego-centric interactions, ignoring multi-agent dynamics.

Deep learning has shifted focus to automated representation learning. Autoencoders, CNN-RNN hybrids [11], and self-supervised embedding models [12] map multi-vehicle trajectories to low-dimensional spaces for clustering and retrieval, reducing manual feature engineering. However, most assume a closed set (all classes known during training), forcing unseen patterns into existing categories and failing to address open-world uncertainty.

Open-set recognition (OSR) frameworks (OpenMax [2], EVT calibration [13], reconstruction-based detection [14], prototype learning [15], and GAN-based synthesis [16]) detect out-of-distribution samples. However, most target image classification; direct extension to spatiotemporal trajectories remains challenging, leaving open-world critical scenario discovery unsolved.

2.2. Scenario Generation and Simulation

Generating high-quality critical scenarios is key for evaluating autonomous driving robustness under rare events. Real driving data’s long-tail distribution drives research on scenario generation.

Early methods used rule-based or stochastic sampling (Monte Carlo [17] or KDE-based estimation [18]) to generate realistic normal scenes but failed to bias sampling toward boundary or failure cases.

Optimization and RL-based methods now dominate for targeting corner cases. Optimization frameworks use Bayesian optimization [19], evolutionary algorithms [20], or genetic operators [21] to search high-risk parameter regions. RL-based approaches control scenario evolution via policy learning [22], with recent extensions [23] integrating generative model constraints to balance realism and adversariality, though they are constrained by training data coverage. Many methods ignore maneuver semantic correlations, reducing physical plausibility of extreme trajectories. Key unresolved issues include insufficient edge case bias, poor realism-adversariality balance, and inadequate parameter dependency modeling. A recent comprehensive review [24] further highlights that existing scenario libraries rarely cover the full autonomous-level spectrum of the transportation system, calling for unified CAV testing frameworks that integrate diverse scenario types.

2.3. Simulation Platforms for Autonomous Driving Testing

Simulation platforms enable scalable, safe autonomous driving evaluation, accelerating safety validation by orders of magnitude [25]. Simulators include open-source platforms (CARLA [2] and LGSVL [26]) with flexible APIs; industrial tools (PreScan [27] and CarSim [28]) with hardware-in-the-loop support; and specialized environments (MATLAB R2025a and Simulink [29]) for algorithm verification. Beyond vehicle-level testing, simulation is also used to study human-factor risks in mixed-autonomy settings. For instance, Shao et al. [30] showed that mental workload, driving style, and ramp geometry jointly determine driver takeover time, underscoring the need for scenario libraries that cover safety-critical human–vehicle interaction events.

3. Methodology

3.1. Overall Framework

A “recognition–generation–simulation” pipeline for autonomous driving SBT is proposed, integrating TRFE (open-world recognition), MCSG (maneuver-level generation), and CARLA closed-loop simulation (Figure 1).

Multi-source driving data is processed by TRFE to identify known and unknown high-risk scenarios (expanding scenario catalogs). MCSG decomposes trajectories into maneuver units (solving high-dimensional search inefficiency), models multimodal parameters, and uses RL to search high-risk configurations. Both scenarios are converted to OpenSCENARIO for CARLA simulation, with feedback iteratively optimizing modules (ensuring library diversity and criticality).

3.2. Open-World Critical Scenario Recognition (TRFE)

Transformer-Based Representation Learning

Traffic scene data are represented as ego-centric occupancy grid sequences. Each time step t yields a 2D grid $G_t\in {\mathbb{R}}^{H\times W}$ describing the surrounding environment. To prepare the input for the transformer, $G_t$ is first flattened into a 1D vector $g_t$ and then augmented with a temporal positional encoding $p_t$ to obtain $x_t=g_t+p_t$. The full sequence $(x_1,\dots ,x_T)$ is fed into a standard transformer encoder $f(·)$, which leverages multi-head self-attention to capture long-range temporal dependencies across all time steps and produces a global scene feature vector h.

To learn robust spatiotemporal representations without relying on large labeled datasets, the encoder is first pretrained via a self-supervised temporal order prediction task. For each scene sample, the input sequence is shuffled into one of N permutations ${\mathrm{shuffle}}_i\left(X\right)$, each assigned a pseudo-label $c_i$. The model is trained to predict the correct temporal ordering, encouraging it to internalize the dynamic evolution patterns of traffic trajectories. The pretraining loss is

$${\mathsf{L}}_{\mathrm{pre}}=-{\displaystyle \sum _{i=1}^N}{\mathbf{1}}_{\{y=c_i\}}log\alpha \left(f\left({\mathrm{shuffle}}_i\left(X\right)\right)\right)$$

(1)

After pretraining, the encoder is fine-tuned on the labeled set $D_{\ell }=\left\{(G_i,y_i)\right\}$ by appending a two-layer MLP classification head. The head maps the global feature h to a probability distribution over K known scenario classes:

$$\widehat{\mathbf{y}}=\mathrm{Softmax}(W_2\sigma (W_{1}h+b_1)+b_2)$$

(2)

Fine-tuning minimizes the standard cross-entropy loss over the labeled training set:

$${\mathcal{L}}_{\sup }=-{\displaystyle \frac{1}{M}}{\displaystyle \sum _{i=1}^M}log{\widehat{\mathbf{y}}}_i\left[y_i\right]$$

(3)

Once fine-tuning converges, the classification head is discarded. The encoder f is retained as a fixed feature extractor, and the global feature $h=f\left(G\right)$ is passed to the open-set recognition module described next.

3.3. Model Components

3.3.1. Unknown-Screening (RF-EVT)

The Unknown Class Screening (UCS) module determines whether an input scene belongs to a known category or should be flagged as unknown. Given the transformer feature h, a random forest of T trees independently predicts a class label for each tree:

$${\widehat{y}}^{\left(t\right)}=T_t\left(h\right)\in \{1,\dots ,K\}$$

(4)

Rather than using only the majority-vote result, the full vote distribution is retained. The vote count for class k is

$$v_k={\displaystyle \sum _{t=1}^T}\mathbf{1}\left(T_t\left(h\right)=k\right)$$

(5)

This vote vector $\mathbf{v}={[v_1,\dots ,v_K]}^{\top }$ provides a non-probabilistic measure of how well the input matches each known class. For known-class samples, votes concentrate on the correct class; for unknown samples, votes scatter across classes or cluster in low-count regions. To statistically model this tail behavior, extreme value theory (EVT) is applied. For each class k, a threshold $u_k$ (95th percentile of training vote counts) is selected, and the excess values $x_i=u_k-v_k^{\left(i\right)}$ are fitted with a generalized Pareto distribution (GPD):

$$f(x;\xi ,\beta )={\displaystyle \frac{1}{\beta }}{\left(1+\xi {\displaystyle \frac{x}{\beta }}\right)}^{-{\displaystyle \frac{1}{\xi }}-1}$$

(6)

At inference, the tail-survival score for class k is computed as follows:

$$s_k=1-F_k\left(x^{\ast }\right)$$

(7)

where $x^{\ast }=u_k-v_k^{\ast }$ for the test sample. If the maximum score across all classes falls below a threshold $\tau$ (i.e., ${max}_k{s}_k<\tau$), then the scene is declared unknown. Otherwise, it is assigned to the best-matching known class:

$$\widehat{k}=arg\underset{k}{max}{s}_k$$

(8)

This three-stage pipeline—transformer feature extraction, RF vote counting, and EVT tail modeling—enables statistically principled detection of out-of-distribution scenarios without requiring explicit negative samples.

The RF is fitted on the frozen encoder features after supervised fine-tuning and is not updated during joint clustering optimization; the RF vote vector enters only as a fixed similarity signal, introducing no gradient discontinuity into the joint loss.

3.3.2. New-Value Preprocessing

Not every sample flagged as unknown by the UCS module is suitable for clustering. Some may be outliers, misclassified samples, or low-quality features that would degrade clustering performance. The New-Value Preprocessing (NVP) module addresses this by managing an FIFO buffer B of a capacity W and applying two sequential filtering steps before passing samples to the clustering stage.

When the buffer reaches capacity, outlier detection is first applied. For each unknown sample feature $h_i$, a k-nearest neighbor score measures its isolation in feature space:

$$\mathrm{OutlierScore}\left(h_i\right)={\displaystyle \frac{1}{|N_i|}}\sum _{j\in N_i}{\parallel h_i-h_j\parallel }_2^2$$

(9)

Samples whose scores exceed a threshold $\delta$ are discarded as outliers, yielding a cleaned set $B_{\mathrm{clean}}$.

A second validity filter then checks whether each remaining sample is sufficiently distant from all known class centers $c_k$, ensuring it carries genuinely novel information rather than being a borderline known sample:

$$d_{min}\left(h_i\right)=\underset{k}{min}{\parallel h_i-c_k\parallel }_2$$

(10)

Only samples exceeding a minimum margin are retained, forming the final set $B_{\mathrm{fil}}\subseteq B_{\mathrm{clean}}$, which is forwarded to the online clustering model. This layered filtering ensures that the clustering stage receives high-quality candidates with genuine potential to form new scenario classes.

3.3.3. Online Clustering (RFAPs)

The Online Clustering (OC) module takes the filtered set $B_{\mathrm{fil}}$ and groups unknown samples into new scenario classes, which are then incorporated into the existing catalog. Unlike static clustering algorithms, the OC module operates in an online fashion, continuously updating class assignments as new data arrive.

The key innovation is using random forest activation patterns (RFAPs) to measure inter-sample similarity. When a feature vector h traverses a decision tree, it follows a unique path through internal nodes to a leaf. This path is encoded as a sequence of node indices $r_i^b$ for tree b. Two samples that share longer common paths in more trees are considered more similar. The pairwise similarity is

$$S_{ij}=1-{\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{\displaystyle \frac{\left|\right\{o\mid r_i^b\left[o\right]\ne r_j^b\left[o\right]\left\}\right|}{|r_j^b|}}$$

(11)

This path-based similarity captures structural relationships in the feature space that are not accessible from Euclidean distances alone, providing richer guidance for clustering unknown scenarios. The transformer encoder is jointly optimized with two heads—a clustering head ${\alpha }_u\left(h\right)$ for unknown samples and a classification head ${\alpha }_{\ell }\left(h\right)$ for known classes—using three complementary loss terms.

The clustering loss uses the RFAP similarity matrix S to pull together similar unknown samples and push apart dissimilar ones:

$${\mathsf{L}}_{\mathrm{cluster}}={\displaystyle \frac{1}{W^2}}\sum _{i,j}\left[S_{ij}log\left({\alpha }_u{\left(h_i\right)}^{\top }{\alpha }_u\left(h_j\right)\right)+(1-S_{ij})log(1-{\alpha }_u{\left(h_i\right)}^{\top }{\alpha }_u\left(h_j\right))\right]$$

(12)

The classification loss preserves recognition accuracy on known classes using labeled data:

$${\mathcal{L}}_{\mathrm{cat}}=-\sum _i{y}_{i}log{\alpha }_{\ell }\left(h_i\right)$$

(13)

The consistency loss enforces stable feature representations under data augmentation, improving generalization:

$${\mathcal{L}}_{\mathrm{cons}}={\displaystyle \frac{1}{W}}\sum _i{∥{\alpha }_u\left(h_i\right)-{\alpha }_u\left({\widehat{h}}_i\right)∥}_2^2$$

(14)

The three terms are combined with a time-varying weight $\omega \left(t\right)$ on the consistency term:

$$\mathsf{L}={\mathsf{L}}_{\mathrm{cat}}+{\mathsf{L}}_{\mathrm{cluster}}+\omega \left(t\right){\mathsf{L}}_{\mathrm{cons}}$$

(15)

Once training converges, high-confidence clusters are promoted to new scenario classes. Specifically, a cluster $C_q$ is promoted if its mean intra-cluster RFAP similarity exceeds 0.85 and its minimum distance to all existing class centers exceeds the validity margin $d_{min}$ used in NVP. The number of new clusters Q is not fixed in advance; it is determined automatically by the online clustering head ${\alpha }_u$, which outputs a soft assignment over a maximum of $K_{max}=5$ candidate clusters per round, and clusters with fewer than 20 samples after convergence are discarded. Promoted clusters receive new class labels and are appended to the labeled training set $D_{\ell }$. The expanded class set ${\mathcal{C}}^{\prime }=\mathcal{C}\cup {\mathcal{C}}_{\mathrm{new}}$ is fed back into the UCS module, which is retrained from the current checkpoint (not from scratch) on the augmented $D_{\ell }$, closing the “recognition–discovery–update” loop and enabling continuous adaptive expansion of the scenario catalog.

Gradient competition is managed by (1) initializing joint optimization from the converged supervised checkpoint and (2) gradually ramping up $\omega \left(t\right)$ to stabilize the shared representation during early joint training.

3.4. Maneuver-Level Multimodal Parameter Space

To avoid the high dimensionality and lack of semantic abstraction inherent in trajectory-level parameterization, MCSG first decomposes continuous driving trajectories into discrete maneuver units (e.g., braking, lane change, and cut in). For the representative lane change scenario, each maneuver is compactly described by a seven-dimensional parameter vector:

$$\mathit{x}={\left[\begin{array}{c}{d}_{\mathrm{trigger}},v_{\mathrm{start}},t_{\mathrm{start}},v_{\mathrm{end}},t_{\mathrm{end}},v_{\mathrm{final}},t_{\mathrm{scenario}}\end{array}\right]}^{\top }$$

(16)

The seven parameters and their physical meanings are listed in

Table 1. The 7-dimensional maneuver parameter vector, with units, value ranges from highD, and OpenScenario mapping.

Param.	Description	Unit	Range	OpenScenario Field
$d_{\mathrm{trigger}}$	Gap to lead vehicle at trigger	m	[5, 80]	EntityCondition/relativeDistance
$v_{\mathrm{start}}$	Speed at maneuver start	m/s	[10, 40]	LongitudinalAction/targetSpeed
$t_{\mathrm{start}}$	Scene start to maneuver start	s	[0, 10]	StartTrigger/delay
$v_{\mathrm{end}}$	Speed at lane change end	m/s	[10, 40]	LongitudinalAction/targetSpeed
$t_{\mathrm{end}}$	Lane change duration	s	[1, 8]	LaneChangeAction/dynamicsDuration
$v_{\mathrm{final}}$	Speed after maneuver	m/s	[10, 40]	LongitudinalAction/targetSpeed
$t_{\mathrm{scenario}}$	Maneuver end to scene end	s	[0, 10]	StopTrigger/delay

. Each parameter is extracted directly from real trajectory data and maps to a corresponding OpenScenario action; longitudinal maneuvers (acceleration/deceleration) are encoded as LongitudinalAction with the target speed and dynamics, and the lateral lane change is encoded as LaneChangeAction with a duration $t_{\mathrm{end}}$. This one-to-one mapping ensures that any parameter vector sampled by MCSG can be directly executed in CARLA without additional post-processing.

Naïve approaches model parameters either independently or as a single Gaussian, both of which fail to capture the nonlinear correlations among maneuver variables. An independent model factorizes the joint distribution:

$$p_{\mathrm{ind}}\left(x\right)={\displaystyle \prod _{j=1}^D}{p}_j\left(x_j\right)$$

(17)

while a unimodal Gaussian assumes a single mode:

$$p_{\mathrm{uni}}\left(x\right)=\mathcal{N}(x;\mu ,\mathbf{\Sigma })$$

(18)

Real driving data exhibit multiple behavioral modes (e.g., aggressive versus cautious lane changes), and thus a multivariate multimodal distribution is needed. MCSG fits this via kernel density estimation (KDE), which places a Gaussian kernel at each observed data point and sums them:

$$p_{\mathrm{multi}}\left(x\right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{K}_h(x-x^{\left(i\right)})$$

(19)

where the multivariate Gaussian kernel is

$$K_h\left(\mathit{z}\right)={\displaystyle \frac{1}{{\left(2\pi \right)}^{D/2}{\left|{\mathbf{\Sigma }}_k\right|}^{1/2}}}exp\left(-{\displaystyle \frac{1}{2}}{\mathit{z}}^{\top }{\mathbf{\Sigma }}_k^{-1}\mathit{z}\right)$$

(20)

For example, the joint distribution of the trigger distance d and relative speed $\mathrm{\Delta }v$ in a highway lane change scenario is modeled as follows:

$$p(d,\mathrm{\Delta }v)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \frac{1}{2\pi h^2}}exp\left(-{\displaystyle \frac{{(d-d^{\left(i\right)})}^2+{(\mathrm{\Delta }v-\mathrm{\Delta }{v}^{\left(i\right)})}^2}{2h^2}}\right)$$

(21)

The bandwidth matrix ${\mathbf{\Sigma }}_k$ is determined by first constructing an initial estimate via Scott’s rule and then selecting the optimal H through cross-validation on the log-likelihood. This multimodal distribution preserves the complex correlations among maneuver parameters and serves as a realism prior in the RL reward, preventing the agent from generating physically implausible trajectories.

3.5. RL-Based Edge Case Generation (MCSG)

3.5.1. MDP Formulation

The scenario generation problem is formulated as a Markov decision process (MDP). At each step, the agent observes the current maneuver parameter configuration and edits it to push the scenario toward higher-risk regions. The state $s_t$ is the current parameter vector:

$${\mathit{s}}_t={\left[\begin{array}{c}{d}_{\mathrm{trigger}},v_{\mathrm{start}},t_{\mathrm{start}},\dots \end{array}\right]}_t$$

(22)

The action ${\mathbf{a}}_t$ is a parameter edit, and the state transition follows

$$s_{t+1}=f(s_t,a_t)$$

(23)

The reward function balances three objectives: maximizing the proportion of high-risk time steps (measured via RSS safe distance violations), encouraging collisions, and penalizing low-likelihood parameter combinations to preserve physical realism. The RSS-based high-risk proportion is

$${\mathrm{highrisk}}_{\mathrm{RSS}}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{k=1}^T}\mathsf{I}\left(d\left(k\right)<d_{min}\left(k\right)\right)$$

(24)

The composite reward is

$$R\left(\mathit{\theta }\right)=\alpha {\mathrm{highrisk}}_{\mathrm{RSS}}+\beta {\mathrm{l}}_{\mathrm{collision}}-\lambda \left[-log{p}_{\mathrm{multi}}\left(\mathit{\theta }\right)\right]$$

(25)

where $\alpha$, $\beta$, and $\lambda$ balance risk, collision, and realism terms, respectively. The negative log-likelihood penalty $-log{p}_{\mathrm{multi}}\left(\mathit{\theta }\right)$ acts as a realism prior, preventing the agent from converging to physically implausible parameter combinations.

Physical feasibility is enforced by (1) constraining parameters to ranges derived from real highD trajectories and (2) CARLA’s physics engine, which clips actuator commands to the vehicle model’s feasible range at execution time.

3.5.2. Policy Learning

The policy ${\pi }_{\varphi }(a_t\mid s_t)$ is implemented as a two-layer LSTM network, which is well suited for capturing sequential dependencies in the parameter editing process. The policy is optimized via the REINFORCE policy gradient algorithm. The gradient of the expected return is

$${\nabla }_{\varphi }J\left(\varphi \right)={\mathbb{E}}_{\tau \sim {\pi }_{\varphi }}[\sum _t{\nabla }_{\varphi }log{\pi }_{\varphi }({\mathit{a}}_t\mid {\mathit{s}}_t)\left(R\left(\tau \right)-b\right)]$$

(26)

where b is a baseline to reduce variance. Parameters are updated via

$$\varphi \leftarrow \varphi +\eta \widehat{{\nabla }_{\varphi }J\left(\varphi \right)}$$

(27)

To improve search efficiency, a coarse-to-fine sampling strategy is adopted. The agent first explores broadly across the parameter space to identify high-risk regions and then refines its search within those regions. This hierarchical strategy accelerates convergence toward optimal high-risk edge case parameter combinations while avoiding premature local optima.

4. Experiments and Results

This section introduces the experimental set-up and then reports the results for TRFE-based open-world critical scenario recognition, MCSG-based maneuver-level generation, and CARLA closed-loop validation.

4.1. Experimental Set-Up

4.1.1. Datasets and Scenario Taxonomy

Experiments were conducted on the public highD [31] and INTERACTION [32] trajectory datasets (highway and urban scenes). The highD dataset has 6 German highway segments (16.5 h, 25 Hz, 110,000+ trajectories, car following and lane change), and the INTERACTION dataset has 3 subsets (roundabouts, ramp merging, and unsignalized intersections) from the US, China, and Germany (vehicles, cyclists, and pedestrians, 10 Hz) (Figure 2).

Raw trajectories were categorized into 17 critical scenario classes based on an expert-defined catalog, as detailed in

Table 2. The 17 critical scenario classes used in TRFE experiments.

ID	Scenario Class	Dataset	Samples	Labeling Criterion
1	Highway free cruise	highD	∼2600	Constant speed, no lane change, large headway
2	Highway car following	highD	∼2600	Follows lead vehicle, TTC < 4 s
3	Highway cut in	highD	∼2600	Adjacent vehicle merges into ego lane, lateral gap < 2 m
4	Highway cut out	highD	∼2600	Ego-lane vehicle departs, sudden deceleration
5	Highway left lane change	highD	∼2600	Ego changes to left lane, displacement > 3.5 m
6	Highway right lane change	highD	∼2600	Ego changes to right lane, displacement > 3.5 m
7	Ramp entering	INTERACTION	∼2600	Merges from on-ramp, lateral shift > 50 m
8	Ramp exiting	INTERACTION	∼2600	Diverges to off-ramp, lateral shift > 50 m
9	Urban cut in	INTERACTION	∼2600	Cut-in on urban road, TTC < 3 s
10	Urban cut out	INTERACTION	∼2600	Sudden departure from urban lane
11	Roundabout entering	INTERACTION	∼2600	Enters roundabout, yields to circulating traffic
12	Roundabout exiting	INTERACTION	∼2600	Exits roundabout, crosses conflict zone
13	Roundabout circulating	INTERACTION	∼2600	Navigates inside roundabout
14	Unsignalized left turn	INTERACTION	∼2600	Unprotected left turn, gap acceptance
15	Unsignalized right turn	INTERACTION	∼2600	Right turn, pedestrian and cyclist conflict
16	Unsignalized straight cross	INTERACTION	∼2600	Straight crossing at unsignalized intersection
17	Unsignalized merging	INTERACTION	∼2600	Two streams merge at unsignalized junction

. The 17 classes defined the initial known set for the open-world protocol; the RFAP clustering module could discover and incorporate additional classes beyond this taxonomy as unknown samples accumulated. Uneven distributions were mitigated via resampling and outlier removal, yielding 43,800 balanced labeled samples. Each scenario was represented by 4 occupancy grids sampled uniformly within a 1.5-s window, regardless of the native frame rate of the source dataset (25 Hz for highD and 10 Hz for INTERACTION); no interpolation between frames was performed, and thus no interpolation artifacts were introduced into the grid values. The sample splits were 70%, 15%, and 15% (training, validation, and testing, respectively).

4.1.2. Evaluation Metrics and Baselines

TRFE used the open-set ACC and macro-averaged F1 (known and unknown samples) with a clustering accuracy $ACC\_cluster$. The baselines were CAC [33], ARPL [34], FCPN [35], CEVT [36], and MDENet [37] (open set), with STAE+K-Means, STAE+HC [38], iterative Euclidean, and iterative AutoNovel [39] (clustering).

MCSG used the trajectory fidelity ($RMS{E}_t$/$RMS{E}_s$) to validate trajectory realism; closed-loop stability and controller failure were assessed separately in the CARLA simulation experiments (Section 4.1.2), with criticality metrics ($P_{\mathrm{highrisk}}$, CR). The baselines were GACS [21], IBO [19], ConScenD [40], CCSF [41], RLTester [23], and RLBE [42].

For the CARLA closed-loop tests, the focus was on triggering near-misses or collisions in CAS across road geometries (no SUT comparison). Evaluating the generated scenarios against multiple SUT implementations to contextualize failure modes across different controller designs remains for future work.

4.2. Open-World Critical Scenario Recognition TRFE

4.2.1. Open-Set Recognition Protocol

An incremental protocol emulated open-world deployment. Initially, two classes were treated as known; in each subsequent round, three additional classes were introduced as unknown samples. TRFE had to reject them as unknown, cluster them via RFAPs, and incorporate the resulting clusters into the known set before the next round begins, repeating the process until all 17 classes were covered (five rounds of three new classes after the initial two). New classes were added in order of increasing visual similarity to the existing known set, following the ordering in Table 2, to create a progressively harder recognition task. To avoid data leakage, the test split for each round contained only samples from classes already incorporated into the known set plus the current round’s unknown samples; samples from future rounds were withheld entirely. Each round was repeated five times with different random seeds, and the results were averaged. The reported ACC and F1_macro are macro-averages across all test rounds.

4.2.2. Open-Set Recognition Performance

Table 3. Open-set critical-scenario recognition performance (higher numbers are better).

Method	ACC	F1_macro
CAC	0.985	0.817
ARPL	0.971	0.822
FCPN	0.958	0.858
CEVT	0.978	0.875
MDENet	0.976	0.883
TRFE (ours)	0.972	0.906

reports the open-set ACC and F1_macro for TRFE and the baselines.

TRFE achieved the best F1_macro score (0.906) despite having a slightly lower ACC than MDENet (0.972 vs. 0.976), reflecting enhanced rare and scenario recognition (critical for safety testing).

4.2.3. Unknown-Class Clustering

Table 4. Unknown-class clustering accuracy (ACC_cluster).

Method	ACC_cluster
STAE + K-Means	0.427
STAE + HC	0.516
Iterative Euclidean	0.612
Iterative AutoNovel	0.902
TRFE (ours)	0.964

compares the clustering accuracy with the baselines.

TRFE’s $ACC\_cluster$ (0.964) outperformed the iterative AutoNovel by 6.2 percentage points (RFAP provides more discriminative similarity than feature-space distances).

4.2.4. Ablation and Parameter Sensitivity

Figure 3 confirms that the UCS, NVP, and RFAPs are critical; No-UCS degraded unknown detection, No-NVP caused unstable clusters, and No-OC weakened cluster alignment.

Figure 4 shows the optimal parameters: $\tau =5$ (EVT threshold), $T=500$ (RF trees), and $W=2000$ (buffer size).

4.3. Maneuver-Based Critical Scenario Generation MCSG

MCSG decomposes TRFE-labeled trajectories into seven-dimensional maneuver parameter vectors and uses RL to search high-risk configurations (two-layer LSTM policy, Adam optimizer).

Regarding the training details, MCSG trained for 3000 episodes; policy parameters were updated in batches of 50 episodes using the REINFORCE gradient estimator. The two-layer LSTM policy network had 32 hidden units per layer, Dropout 0.2, an Adam optimizer (learning rate of $1\times {10}^{-4}$), and gradient clipping at 1.0. For the reward weights, the composite reward was $R=h_{\mathrm{RSS}}$ (normalized high-risk time step ratio, non-collision) or $R=0.25$ (collision), where the collision bonus of 0.25 was selected via grid search to maximally promote edge-case generation without destabilizing training. All CARLA closed-loop tests used the same CAS configuration: a depth camera and LiDAR with a 10-m detection range, RSS-based safe distance computation, and maximum-braking response on high-risk detection. The sensor and controller settings were held constant across all road geometries and scenario types.

4.3.1. Trajectory Fidelity

Figure 5 shows that the generated trajectories aligned closely with the real ones (longitudinal and lateral directions), verifying maneuver-based parameterization realism.

4.3.2. Comparison with Baseline Scenario Generators

Table 5. Comparison of scenario generation methods.

Method	RMSE_t ↓	RMSE_s ↓	${\mathit{P}}_{\mathbf{highrisk}}$↑	CR ↑
GACS	0.763	4.379	0.683	0.530
IBO	0.931	5.181	0.631	0.425
ConScenD	0.327	3.764	0.311	0.262
CCSF	0.816	4.771	0.726	0.627
RLTester	0.463	3.950	0.724	0.621
RLBE	0.352	3.968	0.747	0.631
MCSG (ours)	0.237	3.362	0.779	0.663

shows that MCSG outperformed all the baselines (best fidelity and criticality):

4.3.3. Multivariate Multimodal Parameter Modeling

Multimodal KDE (MCSG) achieved log-likelihood $e^{-12}$ (versus independent univariate $e^{-95}$, unimodal Gaussian $e^{-17}$), better capturing maneuver correlations. Figure 6 shows the generated collision samples’ (red) cluster within real safe regions (green).

4.3.4. Ablation Studies

Figure 7 confirms that maneuver parameterization, multimodal modeling, and RL search are necessary for realism/criticality balance.

4.4. CARLA-Based Closed-Loop Evaluation

The CARLA-based simulation system integrates the outputs of TRFE and MCSG into a closed-loop testing pipeline. Recognized and generated scenario trajectories are standardized into OpenDRIVE road network files and OpenScenario action scripts and then executed in CARLA against the built-in collision avoidance system (CAS) as a black box system under test (SUT).

TRFE and MCSG trajectories are converted to OpenDRIVE and OpenScenario scripts and tested on CARLA’s CAS (black box SUT).

4.4.1. Scene Conversion and CAS Under Test

Road fragments extracted from the highD and INTERACTION datasets were converted to OpenDRIVE files using Osm2lanelet. Maneuver parameters output by MCSG were mapped to OpenScenario action sequences that controlled adversarial vehicle behavior. Road fragments were converted to OpenDRIVE files, and maneuver parameters were mapped to OpenScenario actions. The CAS used a depth camera and LiDAR (10-m range) to compute RSS safe distances (high-risk and collision feedback for MCSG RL).

4.4.2. Simulation Results on Typical and Rare Critical Scenarios

Two categories of scenarios were tested. First, TRFE extracted real critical scenarios from the datasets (e.g., highway cut ins and ramp merging) and converted them to CARLA scripts to verify that the simulation system faithfully reproduced real-world risk patterns. Second, MCSG generated optimized edge-case scenarios by searching for parameter combinations that maximized the RSS violations and collision probability. TRFE extracted real safe critical scenarios, and MCSG optimized the parameters to generate near-miss and collision cases. Tests on four geometries (sharp curve, narrow road, four-way intersection, and T junction) show that the MCSG scenarios triggered collisions consistently (Figure 8).

MCSG generated rare scenarios (diverging, merging, and congestion) on custom CARLA maps, enriching the test library.

Overall, the experiments demonstrate that (1) TRFE effectively identified and clustered unknown critical scenarios (high macro-F1 and clustering accuracy); (2) MCSG generated realistic and highly critical trajectories; and (3) the pipeline reliably triggered collisions across road geometries, enabling effective scenario-based testing.

5. Discussion

This work addresses autonomous driving safety evaluation via a “recognition–generation–simulation” pipeline. The open-world TRFE framework discovers unknown high-risk scenes and expands the scenario catalogue incrementally; via self-supervised pretraining, RF-EVT screening, and RFAP-based clustering, TRFE achieved a 0.906 macro-F1 score (open set) and 0.964 clustering accuracy on the HighD and INTERACTION datasets, outperforming MDENet (+2.3 percentage points for macro-F1 score) and iterative AutoNovel (+6.2 percentage points for clustering accuracy). This enhances robustness to distribution shifted and enriched data-driven scenario cataloguing.

Building on TRFE, MCSG addresses the scarcity of high-risk samples and realism-criticality balance in generation. By decomposing trajectories into maneuver units, modeling multivariate multimodal parameter spaces (KDE), and using a two-layer LSTM RL agent (RSS and collision reward), MCSG outperformed the variants (no maneuver abstraction, unimodal modeling, and non-RL search) with a lower trajectory RMSE and higher high-risk and collision rates, validating the necessity of the joint maneuver space, risk-aware reward, and RL policy design.

CARLA-based closed-loop validation confirmed the pipeline effectively stress-tested the SUT by reconstructing risky configurations across diverse geometries via OpenDRIVE and OpenScenario scripts, systematically pushing scenarios to safety boundaries while preserving physical consistency.

Several limitations point to future directions. For the recognition module (TRFE), a CNN-transformer hybrid front-end could provide stronger spatial inductive bias than row-major flattening; representation probing is recommended to verify the encoder learns structural rather than velocity features; the GPD tail model may degrade under severe domain shift or sensor dropout, motivating online threshold recalibration; replacing the Euclidean distance in NVP with cosine similarity or a learned metric and replacing single-center class boundaries with nearest-prototype or density-based margins would improve robustness to complex class distributions; and the 17-class taxonomy defines the initial known set and can be expanded by the RFAP clustering module as unknown samples accumulate. For the generation module (MCSG), the seven-dimensional parameter vector captures maneuver semantics but not sub-second micro-behaviors such as jerk profiles; the single-adversary design can be extended to coordinated multi-agent scenarios with scalable density estimators (e.g., normalizing flows); replacing the deterministic RSS reward with a probabilistic risk field model and adding explicit jerk constraints would improve physical fidelity; a Fuzzy–TD3 hybrid that adapts reward weights to the instantaneous safety margin [43] is a promising direction; and true within-episode closed-loop adversarial control remains future work. For evaluation, extended multi-stage interactions may require longer temporal windows or hierarchical representations, and an ablation of coarse-to-fine parameter boundary settings and comparison across multiple SUT implementations are recommended future experiments.

6. Conclusions

This paper proposed a unified framework for autonomous driving safety-critical scenario mining and testing, comprising TRFE (open-world recognition), MCSG (RL-based generation), and a CARLA-based simulation system. TRFE identifies and clusters unknown risky scenes, outperforming SOTA baselines in open-set recognition (0.906 macro-F1 score) and clustering (0.964 accuracy score) on highway and urban datasets. MCSG leverages maneuver abstraction, multivariate multimodal modeling, and deep RL to generate realistic trajectories with higher high-risk exposure and collision rates than baselines.

Integrated into CARLA via OpenDRIVE and OpenScenario, these scenarios enable closed-loop CAS testing across diverse road geometries, stably triggering near-misses and collisions and supplementing real data with rare critical scenes. The unified pipeline enhances scenario-based testing efficiency and coverage, providing a practical foundation for systematic autonomous driving safety validation.