A Survey on Data-Driven Scenario Generation for Automated Vehicle Testing

Abstract

Automated driving is a promising tool for reducing traffic accidents. While some companies claim that many cutting-edge automated driving functions have been developed, how to evaluate the safety of automated vehicles remains an open question, which has become a crucial bottleneck. Scenario-based testing has been introduced to test automated vehicles, and much progress has been achieved. While data-driven and knowledge-based approaches are hot research topics, this survey is mainly about Data-Driven Scenario Generation (DDSG) for automated vehicle testing. Rather than describe the contributions of every study respectively, in this survey, methodologies from various studies are anatomized as solutions for several significant problems and compared with each other. This way, scholars and engineers can quickly find state-of-the-art approaches to the issues they might encounter. Furthermore, several critical challenges that might hinder DDSG are described, and responding solutions are presented at the end of this survey.

1. Introduction

Traffic accident reduction has always been a hot topic in industry and academia [1,2,3]. Automated Driving (AD) is a promising approach to reducing accidents [4]. The rapid development of AD has been witnessed in the past decade. Some companies, such as Baidu, Tesla, and Audi, claim that they have developed high-level automated vehicles (AVs) [5,6]. Before being accepted by the market, high-level AVs must be thoroughly validated and evaluated to prove that they are safer than human driving, or at least as same [7]. However, testing AVs is still challenging [8]. On the one hand, AVs are becoming increasingly more complicated. For example, some Machine Learning (ML) technologies have been applied to AD for tasks including object recognition and classification and have achieved good results [9]. However, many of them, such as Convolutional Neural Networks (CNN), and Reinforcement Learning (RL), are not fully interpretable, which makes it significantly challenging to test AVs based on traditional methods, such as mileage-based testing [10,11]. On the other hand, there are many occasional scenarios in the physical world where AVs are required to drive [12]. The number of scenarios an AV might encounter in natural driving traffic is theoretically infinite. Ref. [13] indicates that millions of miles of road testing may be required to prove the reliability of a driverless vehicle, which is unfeasible and unaffordable.

Scenario-Based Testing (SBT) is a well-investigated and promising method for AV testing. There are many projects and initiatives about the SBT of AVs, including Pegasus [14], euroFOT [15], AdaptIVe [16], Sakura [17], StreetWise [18], and so on. SBT has already been utilized in software engineering [19,20]. The purpose of SBT is to prove that the System Under Test (SUT) can work as designed or at least safely without getting itself hurt or jeopardizing the safety of other traffic participants in the whole Operational Design Domain (ODD) [6].

One of the cornerstones of SBT is the scenario database, including diverse and critical scenarios. There are two kinds of methods to obtain it, data-driven or knowledge-based. The most significant difference between them is their reliance on expert knowledge. Knowledge-based methods usually require substantial expertise to generate needed scenarios, such as ontology-based methods [21,22]. On the contrary, data-driven methods generate scenarios primarily by exploiting information contained in source data. It is worth noting that “data-driven” does not mean expert knowledge is not needed in all relevant processes of scenario generation. Actually, specialist knowledge is utilized in almost all data-driven methods or as complementary to them, more or less [23,24]. While knowledge-based SBT has been proven to be a practical approach in some circumstances [22], it is limited to the domain designed by experts [25], and it cannot derive SUT performance in natural traffic [15]. Therefore, this survey focuses on Data-Driven Scenario Generation (DDSG).

Data-driven scenario generation for AV testing has become a hot topic, and much progress has been made. It would be helpful for researchers and engineers to read surveys summarizing critical points of relevant studies, especially the ones published in the past three years. A problem-oriented survey is provided to make it easier for researchers and engineers to find solutions for their problems of data-driven scenario generation quickly. Solutions for the same issues are grouped and compared. For example, dimension reduction techniques, such as Principal Component Analysis (PCA), and t-distributed Stochastic Neighbor Embedding (t-SNE), are employed in many studies. A comparison of them could help scholars and engineers make the best choice. It is worth noting that while this survey is mainly about data-driven approaches for scenario generation, some studies taking advantage of expert knowledge to enhance data-driven scenario generation are also considered in this survey.

There are already some reviews about SBT of AVs [12,26,27,28]. SBT of AVs is one of the topics of [28], but the latest literature is not considered. The approaches for SBT are taxonomized in [26] without some essential topics not discussed in detail, such as the improvement of the original Accelerated Evaluation (AE) [29] and the construction of metrics considering several aspects. In [27], methods for SBT are categorized into three classes: coverage-oriented, unsafe-scenario-oriented, and indicator-estimation oriented. And these methods are analyzed based on simulation results. However, most of these reviews group related studies based on the overall methodologies rather than the specific problems they try to solve.

The motivation of this paper is to introduce, analyze, and compare state-of-the-art methodologies for Data-Driven Scenario Generation (DDSG) and solutions to related sub-problems. Furthermore, pointing out some remaining problems in DDSG. Scholars and engineers can make the best choice for AV testing among the existing methods or dig deeper to tackle the remaining problems in scenario-based testing (SBT). Therefore, the contributions of this survey include four parts:

• State-of-the-art methodologies used for DDSG, such as Reinforcing Learning (RL), Accelerated Evaluation (AE), and so on, are generally introduced. The generation of customized scenarios for the VUT is also covered by this survey, which cannot be found in existing reviews.
• Solutions to sub-problems involved in these methodologies are described in detail. These sub-problems include source data collection, scenario identification, and criticality metrics used for scenario evaluation.
• Some remaining problems are pointed out, and responding potential solutions are provided.

2. Framework

Some crucial problems must be solved to generate needed scenarios, including collecting authentic source data, identifying interested scenarios hidden in source data, generating scenarios for AV testing, evaluating derived scenarios, and so on. Since methodologies for DDSG might diverge a lot from each other, based on these to-be-solved problems, a typical framework of DDSG is obtained as shown in Figure 1, which includes mainly four steps:

Step 1: Source Data Obtaining. DDSG mainly relies on information extracted from data instead of expert knowledge. Therefore, source data is crucial to DDSG [26]. Source data can be collected through physical or virtual approaches. The former includes collecting Natural Driving Data (NDD) and accident data et al. Sometimes, data from the real world is integrated into relevant laws, regulations, or standards, which can also be used as data sources. With the developing of simulation techniques, the fidelity of the virtual world used for AV testing is improving gradually, leading to the increasing importance of the later ones [30,31].

Step 2: Scenario Identification. Scenario identification aims to identify or detect scenarios that have already happened and are hidden in source data. Based on these identified scenarios, parameter ranges and distributions of logical scenarios can be elicited. On the other hand, some accident scenarios can be directly used for AV testing, such as accident and near-collision scenarios [32].

Step 3: Scenario Generation. Many companies have collected a large amount of NDD [28]. However, on the one hand, the number of scenarios existing in the real world is theoretically infinite. It is almost an impossible mission to collect enough data in which all possible parameter combinations in ODD can be directly identified [33]. Moreover, most scenarios recorded by NDD are dull, and critical scenarios are rare [15]. As for accident databases, the diversity of identified scenarios cannot be secured. Therefore, it is necessary to generate scenarios that are rarely observed in the natural world but are critical for AVs.

Step 4: Scenario Evaluation. Based on pre-defined metrics, the quality of scenarios can be measured, which will be employed to quantify SUT performance. This survey focuses on the criticality of scenarios.

After analyzing hundreds of studies, this survey focusses on several hot topics, including source data collection, scenario identification, diverse scenario generation, critical/challenging scenario generation, customized scenario generation, and criticality metrics of scenarios. The remains of this survey are arranged as follows: Section 2 provides some basic term definitions. Several source data collection methods are described in Section 3. Section 4 and Section 5 introduce essential procedures and methodologies for scenario identification and generation, respectively. At the end of Section 3, Section 4 and Section 5, conclusions of relevant sections are made. Finally, conclusions about existing methodologies for DDSG are made, and several challenges are pointed out in Section 6.

3. Definitions

3.1. Scene, Scenario, and ODD

Although many studies about SBT of AVs exist, the definitions of scenario and scene remain open [34]. Considering rationality and popularity, relative term definitions in Pegasus [14] are adopted in this survey. In Pegasus, a scenario is a temporal sequence of scenes. A scene is a snapshot of the environment, including dynamic entities such as vehicles and pedestrians and static entities such as traffic signs and lights [35].

Based on the level of abstraction, scenarios can be classified into three categories, functional, logical, and concrete scenarios [20]. Functional scenarios are described by natural languages, such as cut-in and car-following scenarios, and have the highest level of abstraction. Parameter ranges and distributions can be found in logical scenarios, which have less abstraction than functional scenarios. Each parameter has an exact value in concrete scenarios, meaning concrete scenarios are less abstract than logical and functional scenarios. Concrete scenarios selected to execute are called test cases [36]. If with no specific notation, scenarios mentioned in the following refer to concrete scenarios.

ODD is the operational design domain where the SUT should work as designed [6]. For a better description of ODD, a 5-layer model [22] is developed for highway scenarios, which includes Road-level (Layer 1), Traffic Infrastructure (layer 2), Temporary Manipulation of L1 and L2 (layer 3), Objects (Layer 4), and Environment (Layer 5). Furthermore, [37] extends it by adding the sixth layer, Digital Information. Furthermore, a modified 6-layer model is presented in [38] to describe traffic scenarios in urban appropriately, which includes Road Network and Traffic Guidance Objects (Layer 1), Roadside Structures (Layer 2), Temporary Modification of L1 and L2 (Layer 3), Dynamic Objects (Layer 4). Environment Conditions (Layer 5), Digital Information (Layer 6).

3.2. Critical, Challenging Scenarios

There are many terms used in different studies to describe scenarios with different characteristics, including critical scenarios, challenging scenarios, boundary scenarios, complex scenarios, corner cases, edge cases, etc. However, there are no standard definitions for them now. While usually applied as synonyms [39,40,41], differences between these terms can be found in some studies.

The term criticality can be found in many studies (see Section 7), but its definition in SBT remains an open question. In [42], critical metrics are surrogate measures for analyzing conflict potential or the severity of microscopic scenarios. In [43], criticality indicates the temporal or spatial closeness to a potential collision in a driving scene/scenario, or the magnitude of the dynamic driving reaction required to prevent an accident. Furthermore, a critical metric quantifies the criticality of a scene or scenario. Since the definitions described in [43] are more comprehensive, this survey adopts them. In terms of critical scenarios, the definition provided by [44] is adopted: critical scenarios are scenarios in which the ego vehicle leads or nearly leads to collisions. The criticality of a scenario can only be obtained after scenario execution.

Challenging scenarios and complex scenarios are usually used as alternatives to each other [44]. In [45,46], challenging and complex scenarios refer to significantly difficult scenarios for ego vehicles to pass safely. They insist that how challenging or complex a scenario is can be determined before scenario execution. Ref. [47] regards scenarios that are very difficult to master as challenging scenarios and assumes that an increasing difficulty of a scenario leads to a surge of failure possibility of the SUT. In this survey, challenging or complex scenarios are the ones that are challenging for the SUT to pass without directly or indirectly leading to a crash. It is worth noting that challenging or complex scenarios are not always critical, which can only be determined after the execution.

In [28], boundary scenarios are those whose execution results are in the proximate area around the boundary between safe and unsafe. Behavior mode boundaries are considered guardians for searching critical scenarios in [48]. As for corner and edge cases, [49] believes only corner cases with unusual or novel conditions can be considered edge scenarios. In this survey, the definition presented by [27] is adopted, which is that boundary scenarios/cases are the ones that exist around the performance boundaries of the SUT, around which a small change to scenario parameters might lead to significantly different execution results, such as leading to a collision or not. Corner and edge scenarios/cases are the ones that are extremely rare in the real world. The slight difference between corner cases and edge cases is ignored. For more term definitions used for SBT, [49,50] are advised.

4. Source Data Collection

The authenticity and coverage of source data are crucial to the quality of a scenario database. Data from the physical world, such as Natural Driving Data (NDD) and accident datasets, are exploited in many studies. In the meantime, some studies use simulation-based methods to acquire source data. This section introduces methods of obtaining NDD, accident data, and virtual data, and some conclusions are given at the end of this section.

4.1. Natural Driving Data

Natural Driving Data (NDD) is traffic data collected in the physical world. There are two main approaches for NDD collection: the floating vehicle-based and the fixed sensor-based. Floating vehicles (FVs) are usually equipped with many sensors, such as lidars, millimeter-wave radars, inertial navigation systems, global positioning systems, and cameras. Theoretically, FVs can collect NDD at any place where FVs can go at any time. By fusing data from different sensors, high-quality data can be collected. Some big companies have gathered lots of NDD using FVs, including Baidu, Waymo, Volvo, and so on [51]. Well-founded projects such as Pegasus, SPMD project [52], et al. also have their own NDD databases. However, it is costly to maintain a large fleet of FVs. Due to the tremendous investment required, many related companies or institutes do not share/publicize the source data, such as Pegasus and TNO [53].

In [54], floating vehicles equipped with two laser scanners, two front cameras, and IMU/GUSS are used to collect traffic data in China, as shown in Figure 2. 3D-semantic labels and 3D-bounding boxes in ApolloScape dataset make it a hot dataset for object detection and orientation estimation algorithms. Since this dataset includes many urban traffic scenes, it can also be used to generate scenarios for AV testing.

Fixing sensors on roadside infrastructures or drones to collect NDD is a comparatively more affordable method. Drones equipped with cameras are mentioned in many studies [55,56,57,58]. With the help of computer vision technology, trajectories of traffic participants can be extracted precisely. In the highD dataset [55], mean errors of lateral and longitudinal positions between computer-based and human-based methods are less than 0.03 m. However, compared with FV-based methods, the observed area and duration of the fixed sensor-based method are pretty limited and are sensitive to harsh circumstances. For example, if cameras are the only employed instruments, extreme weather, such as foggy or rainy, could make it significantly challenging to extract accurate NDD from the recorded videos. However, if several types of sensors are applied for data collection, the negative influence of unfavorable weather can be mitigated.

In [59], to collect traffic data in an intersection in Aschaffenburg, Germany, 14 8-layer lidars and eight cameras with different viewpoints are installed on traffic lights and lamp posts more than 5 m above the ground.

Traffic scenarios in NDD are naturally feasible, and SUT performance in natural traffic can be derived from that in scenarios generated based on NDD (see Section 5). Since the number of scenarios that might happen in natural driving traffic is almost infinite, an enormous amount of NDD is needed for AV testing. However, it requires heavy investments to collect NDD based on FVs, while fixed sensors can only monitor a specific area. While many large-scale NDD datasets are not publicly available, some institutes share their NDD data for non-commercial purposes. In [12], 25 public NDD datasets are compared in several aspects, including bird’s-eye or first-person view, containing or not containing data in different weather, notation types, etc. Ref. [60] summarizes 37 NDD datasets and 22 kinds of commercial or simulation software for AV testing. For the completeness of this section, 27 traffic datasets are provided in

Table 1. 27 NDD dataset. “Trajectory” indicates if trajectories of all traffic agents are explicitly available in the dataset.

Number	Dataset	OpenSource	Method	Sensor	Trajectory
1	100-car [61]	Yes	FV-based	Camera, GPS, radar	No
2	A*3D [62]	Yes	FV-based	Lidar, Camera	No
3	ApolloScape [54]	Yes	FV-based	Camera, Lidar, GPS/IMU	No
4	Argoverse [63]	Yes	FV-based	Lidar, Camera	Yes
5	Bdd100k [64]	Yes	FV-based	Camera, Lidar, GPS/IMU	No
6	CamVid [65]	Yes	FV-based	Lidar, Camera	No
7	Cityscapes [66]	Yes	FV-based	Camera	No
8	CitySim [67]	Yes	Fixed sensor-based	Camera	Yes
9	Five Roundabouts [68]	Yes	Fixed sensor-based	Lidar, Camera	Yes
10	H3D [69]	Yes	FV-based	Cameras, LiDAR and GPS/IMU	No
11	InD [56]	Yes	Fixed sensor-based	Camera	Yes
12	INTERACTION [70]	Yes	Fixed sensor-based	Camera	Yes
13	KAIST [71]	Yes	FV-based	Camera, Lidar, GPS/IMU	No
14	KITTI [72]	Yes	FV-based	Camera, Lidar, GPS/IMU	No
15	Ko-PER [73]	Yes	Fixed sensor-based	Lidar, Camera	Yes
16	Lyft Level 5 [59]	Yes	FV-based	Lidar, Camera	No
	NGSIM [58]	Yes	Fixed sensor-based	Camera	Yes
17	nuScenes [59]	Yes	FV-based	Radar, Lidar, Camera, GPS/IMU	No
18	Oxford RobotCar [74]	Yes	FV-based	Camera, Lidar, GPS/IMU	No
19	RondD [57]	Yes	Fixed sensor-based	Camera	Yes
20	SPMD [52]	Yes	FV-based	VAD, ASD, RSD, et.al.	No
21	Stanford Drone [75]	Yes	Fixed sensor-based	Camera	Yes
22	BDDDD [76]	Yes	Fixed sensor-based	Camera	Yes
23	TRAF [77]	Yes	FV-based	Camera	Yes
24	TDCDBD [78]	Yes	FV-based	Camera	No
25	TAF-BW [79]	Yes	FV-based	Camera	Yes
26	Udacity [80]	Yes	FV-based	Camera	No
27	Waymo Open [81]	Yes	FV-based	Cameras, LiDAR and GPS/IMU	No

, including the latest ones.

4.2. Accident Data

Each accident scenario identified in accident databases involves at least one crash, which means accident scenarios are naturally challenging to AVs [82]. Theoretically, all approaches of NDD collection can be applied to accident data gathering. In [83], many accident data collection methods adopted in developed and developing countries are introduced and compared. Considering the rarity of accidents and the affordability of relevant instruments, accident reports and videos are most widely used for accident data extraction [84,85]. However, accident reports or videos might ignore some essential details, which means some remedies might be necessary [86]. Many countries have invested a lot in accident data collection, resulting in many famous accident databases, such as CIDAS [87] and NAIS [88] from China, GIDAS [89] from German, GES [90] from the US, ASSESS [91] from Europe, OTS [92] and STATS19 [92] from the UK. Several accident datasets are analyzed in [93].

However, as shown in

Table 2. 14 Accident datasets. “In-Depth” indicates an in-depth accident dataset.

Number	Dataset	Open Source	In-Depth	Region	Source of Raw Data
1	US-Accidents [96]	Yes	No	USA	MapQuest Traffic and Microsoft Bing Map Traffic
3	CIDAS [97]	No	Yes	China	accident report
4	Dubai [98]	No	No	Dubai	accident report
5	GIDAS [99]	No	Yes	German	accident report
6	KIDAS [100]	No	Yes	Korea	accident report
7	Korean Freeway [101]	No	No	Korea	accident report
8	NAIS [88]	No	Yes	China	accident report
9	GES [67]	Yes	Yes	USA	accident report
10	OSM [102]	Yes	No	Global	accident report
11	SHUFO [103]	No	Yes	Shanghai	accident report
12	Singapore [104]	No	No	Singapore	accident report
13	UKIDAS [105]	No	Yes	UK	accident report
14	CADP [106]	Yes	No	Global	Video

, some valuable accident datasets are not available for most researchers, such as In-depth accident databases from German and China [89,94]. The Crash Investigation Sampling System (CISS) database of the National Highway Traffic Safety Administration is the only in-depth accident database freely available to the public [95]. Moreover, accident scenarios only fill a small part of ODD, indicating that they cannot be leveraged to prove AV safety in the whole ODD.

4.3. Virtual Data

In the virtual world, the positions and behaviors of traffic participants can be manipulated as desired, and complex scenarios can be produced at a low cost. With the advancement of simulation technology, it becomes possible to gather virtual but high-fidelity data in simulation experiments.

Microscopic traffic simulation is an efficient method to quickly get a large amount of virtual data for DDSG. In [31], accident scenario data is obtained by microscopic traffic simulation on SUMO [107]. To create a more realistic traffic circumstance, a digital twin of the natural static environment in a district is reconstructed on Unity in [108]. Then virtual vehicles, pedestrians, and other traffic participants are added to the virtual world to generate high-fidelity traffic scenarios. Reviews of traffic simulation technologies and popular simulators can be found in [109,110].

The behavior models employed in microscopic traffic simulations are usually conservative and lack complicated interactions among each other, which are significantly essential for AVs [111]. To this end, it is an excellent option to replace the crucial virtual traffic participants (such as drivers, pedestrians, et al.) with real ones while other elements remain virtual. To investigate the interaction between human drivers and automated vehicles in dilemma areas, [30] equips human drivers with Virtual Reality (V.R.) instruments to immerse them in the virtual world. Similarly, in [112], Mixed Reality (M.R.) technology is adopted to collect pedestrian-vehicle interaction data. Augmented Reality (AR) instruments are integrated with a scenario generation methodology in [113] to simulate a natural traffic environment. However, simulation efficiency is somewhat limited since not all elements in these experiments are virtually and automatically generated.

There are several advantages of generating source data based on the simulation environment. First, the generation of virtual traffic data is more time efficient. Different from collecting traffic data in the real world, the number of virtual traffic data generated within one second could be significant given sufficient computing resources. Second, it is less labor-intensive. Third, semantic information is 100% right, which is of great value for the perception system test of an AV. However, there might always be a simulation-to-real gap [114]. Rendering realistic traffic scenarios will be a hot research topic in the next decade.

4.4. Conclusions of Source Data Collection

This section introduces methods for collecting NDD, accident data, and virtual scenario data. NDD is naturally feasible, and SUT performance in natural traffic can be derived based on SUT performance in scenarios generated based on NDD (see Section 6). However, it is not easy to gather enough NDD without heavy investment. Accident data can be used to extract accident scenarios. Each of these accident scenarios involves at least one accident and is challenging for AV testing. Considering the efforts needed for collecting scenario data in the real world, it would significantly contribute to the community if big companies share their traffic data, such as Alphabet, Tesla, Didi, and Baidu. It must be pointed out that it is not possible or feasible to identify scenarios that cover all space of the ODD only using data collected in the physical world.

In the virtual world, almost all elements needed for a scenario, including behaviors of all traffic participants, weather, roadside buildings, and so on, can be simulated and manipulated. With the development of simulation technologies, the efficiency of generating high-fidelity scenario data is increasing continually. It is reasonable to believe that virtually generated data would play an increasingly significant role in DDSG.

5. Scenario Identification

Scenarios identified in source data can be applied to derive ranges and distributions of scenario parameters. The critical ones among them can be directly used for the SBT of AVs. For example, critical scenarios generated by adding noise to accident scenarios are utilized by Waymo to test AVs [32]. Highly efficient scenario identification methods are needed to mind a large amount of source data. This section summarizes studies about scenario identification, and a comparison of different approaches is made at the end of this section. It is worth noting that while the topic of this survey is DDSG, considering rule-based methods can be used to label data for scenario identification based on Unsupervised Machine Learning (UML), studies about rule-based methods are also considered in the section.

5.1. Region of Interest

Before scenario identification, spatial and temporal ranges of engaging scenarios should be determined to ignore factors having almost no influence on ego and eliminate unnecessary data. However, a scenario’s duration and Region Of Interest (ROI) are still open to discussion. Ref. [26] suggests that a duration of about 10 seconds is enough for most scenarios.

For the ROI of highway scenarios, an eight-vehicle model, as shown in Figure 3, is proposed only to consider vehicles near ego [115]. For specific values of $L_f$ and $L_b$, 60 m is advised in [115]. Ref. [116] argues that ROI should be relative to the longitudinal velocity of ego, and an ROI considering safety distance is presented, as shown in Figure 4. A time gap of 1.8 s is suggested in [45,117]. Ref. [118] holds similar opinions and argues that remaining lateral distance should also be considered.

5.2. Feature Dimension Reduction

For different scenarios number of parameters may vary a lot. To evaluate the performance of AVs in the whole ODD, complicated scenarios involving multiple parameters are necessary. Dimension reduction is essential to overcome dimension explosion and make it more feasible to identify or generate scenarios based on features in the latent space in the following steps. While there are many techniques for DR [115], three of them are frequently used for DDSG, including Principal Component Analysis (PCA) [119], Singular Value Decomposition (SVD) [120], t-Distributed Stochastic Neighbor Embedding (t-SNE) [121].

PCA is a linear analysis technique that transfers original parameters to independent principal components by mapping parameters from the original to a latent space without losing too much variance [115]. PCA is widely adopted because it is robust and costs a low level of computational power [122]. Inspired by PCA, [123] proposes Principal Feature Analysis (PFA) which maps the most important features rather than all features to the latent space. In [124], PCA is applied to transform high-dimensional trajectories to low-dimensional feather vectors, and then the noise is added to the feather vectors to generate critical scenarios. However, since PCA works based on the contribution of parameters to the variance of the result, it belongs to a statistical method. This usually makes the feather vector not physically interpretable [125].

While SVD is one of the key parts of PCA, SVD itself can also be applied for dimension reduction. To reduce dimensions for scenario description, SVD is adopted in [126] to map the original scenario parameters to a space with fewer dimensions. Furthermore, the Possibility Density Function (PDF) of the resulting low-dimension parameters is estimated by Kernel Density Estimation (KDE) and sampled to generate new scenarios. Although PCA and SVD are both linear analysis techniques without supervision, there is a significant difference between them [127]. The goal of PCA is to maximize the variance of the original parameters, while SVD aims to minimize reconstruction error. Furthermore, PCA needs no iteration, while SVD does.

t-SNE is a visualization and dimension reduction tool based on possibility differences [128]. High-dimension parameters are mapped to a lower-dimension (usually 2 or 3) space by t-SNE without losing significant structure in the original data. In other words, adjacent points in the original space are still near each other with a high possibility in the new space, which PCA cannot achieve. To cluster trajectories in different scenarios, [129] combined Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [130] with PCA, SVD, and t-SNE, respectively. Simulation results show t-SNE based DBSCAN outperforms the others. However, every DR method has its edges. For more details about dimension reduction, [131] are suggested.

5.3. Rule-Based Methods

Rule-based methods identify scenarios mainly based on pre-defined constraints or rules, such as thresholds, parameter ranges, etc. For simple scenarios that involve no complicated maneuvers or many participants, several thresholds or rules defined by experts might be enough for scenario identification. Based on pre-defined thresholds of decelerations, the lateral distance between ego and lane lines, or the time before or after the maneuver, braking, turning, lane-change, and cut-in scenarios are identified in [23,132,133].

In Rule-based approaches designed to identify complicated scenarios, scenarios are usually interpreted as combinations of elementary blocks, which could be states, events, activities, or maneuvers. Different temporal sequences of these building blocks belong to different function scenarios [134,135,136].

In [137], activities are the elementary blocks of scenarios. There are three lateral activities (acceleration, cruising, and deceleration) and three longitudinal activities (lane following, turn left, and turn right). Every trajectory can be regarded as a combination of these activities. Rules-based scenario templates are designed by experts and include the relative locations of traffic participants and their activities and speed. Gap-closing and cut-in scenarios in NDD are detected. The authors of [137] claim that this approach can identify all scenarios in NDD. But no example involving complex maneuvers is provided.

In [138], a maneuver is defined as the intentional transfer between different states of a participant. A layered maneuver model for urban scenario identification is designed, which includes vehicle state maneuvers, infrastructure maneuvers, and object-related maneuvers. Vehicle state maneuvers include acceleration, keeping velocity, deceleration, and reversing. Infrastructure maneuvers include following lanes, lane-change, approaching a junction, and so on. Object-relate maneuvers include the following and approaching objects, et al. Different combination patterns of these maneuvers are categorized into different functional scenarios. A field experiment of left turning in a junction is presented to validate this methodology. Then random sampling is used to generate scenarios based on the scenario parameter ranges extracted from the field data. However, it is difficult for this method to detect scenarios involving maneuvers not considered in the pre-defined maneuver model.

Rule-based models are enhanced by an unsupervised clustering technique in [45]. First, data in highD is clustered by a Hierarchical Clustering Algorithm (HCA) to identify scenarios that include at least one challenger. Second, the maneuvers of challengers are classified by a rule-based decision tree. Third, based on trajectories, the maneuvers of challengers are classified into one of the six functional scenarios in the PEGASUS Project. This method is scenarios in highD dataset, and 67,455 concrete scenarios belonging to the pre-defined six functional scenarios are extracted.

The studies mentioned above have shown that the rule-based method can effectively identify scenarios in source data. However, the rules designed by different experts might vary, and it demands a lot of expertise, and complicated scenarios are still challenging for rule designers. Moreover, it is difficult for rule-based methods to identify scenarios that are unknown to experts, which, to some extent, limits the diversity of the resulting scenarios.

5.4. Unsupervised Machine Learning

Rule-based methods are quite effective in detecting known scenarios with pre-defined rules. However, more powerful tools are needed to search for scenarios that have unknown patterns or are too complicated to be described by rule-based models. Machine learning is an effective tool for pattern recognition and has been applied in many fields [139]. Unsupervised Machine Learning (UML) identifies patterns without expert knowledge. The methods frequently used for scenario identification are unsupervised clustering techniques, such as K-Means, K-Medoids, and Hierarchical Clustering Algorithms (HCA) [45,115,140,141]. There are two major problems with detecting scenarios using UML. First, the durations of scenarios from the same function or logical scenario might vary. Second, some important features might hide in a latent instead of explicit space.

For the first problem, techniques that can transfer time series to feature vectors are advised, such as Dynamic Time Warping [115] and Variational Autoencoder [142]. Ref. [142] maps four parameters of car-following scenarios, including the speed of the ego and the following vehicle, car-following distance, and ego acceleration, to a 2-dimensional latent space by Variational Autoencoder (VAE), which can process temporal sequences. Furthermore, the distribution of one of the two latent variables, named z, is analyzed based on NGSIM and highD, respectively. Results of numerical experiments indicate that z contains enough information for scenario identification.

The first problem can also be solved by decomposing the scenario as a combination of states and actions. In [53], signals are sliced to search event or activity indicators such as extrema, inflections, or saddle points. Then, signal slices are clustered by an agglomerative HCA. This method can be used for data labeling. A similar approach can also be found in [141].

For the second problem, source data are usually mapped to a latent space and then processed by UML to extract information in the latent space. The combination of a dimension reduction algorithm and a clustering algorithm can be found in [115,143,144,145]. In [145], PCA and K-Means are used to identify representative traffic scenarios in 72,336-h urban-motorway NDD. First, all source data are divided into two datasets based on the driving direction. Then, each dataset is processed by PCA, and two principle components accounting for more than 90% variance of all eigenvalues for each dataset are obtained. Finally, K-Means is applied to cluster similar scenarios in the latent space generated by PCA. Four representative scenarios are identified.

The introduction and analysis above show that, based on UML techniques, especially clustering techniques, implicit and explicit patterns of scenarios can be automatically extracted for scenario identification. Unlike rule-based methods, UML-based ones could obtain unknown scenarios, contributing to the coverage of the resulting scenarios. However, some patterns identified by UML might be uninterpretable, resulting in unreasonable results.

5.5. Supervised Machine Learning

Supervised Machine Learning (SML) techniques, such as Support Vector Machine (SVM), Recurrent Neural Networks (RNN), can identify scenario patterns based on labeled source data. In [25], scenarios are represented as state sequences. Rules are designed to label data with state tags. Different tag sequences represent different scenarios. Tag sequences can be used as scenario templates to identify scenarios in source data. The supervised classifier trained by a small amount of labeled data could label other unlabeled data automatically. In [146], A rule-based method is applied to tag source data with state labels, and logical scenarios are described as label sequences. An RNN model is trained by these labeled scenario data and then exploited to detect scenarios in unlabeled source data. This methodology is evaluated by a case of detecting lane-change scenarios in an NDD dataset [146].

Unlike rule-based and UML-based methods, SML combines expert knowledge with machine learning techniques, achieving high efficiency in scenario identification without losing too much interpretability. It is worth noting that with inappropriate labels, SML cannot obtain reasonable scenarios, like UML methods.

5.6. Conclusions of Scenario Identification

This section introduces solutions to three problems: filtering out unnecessary scenario data, reducing feature dimensions, and identifying scenarios recorded in source data. Limiting the spatial and temporal ranges allows scenario data of interest to be extracted from the source data. After that, three widely used dimension-reducing techniques are introduced and compared for high-dimensional scenarios. Finally, Rule-based, UL-based, and SL-based methods are introduced.

Rule-based, UML, and ML techniques are described for the last problem. Rule-based methods identify scenarios relying on rules designed by experts, which means that the performance of rule-based methods highly depends on experts. It would be pretty tricky to achieve good results for complex scenarios without time for adjusting the rules. UML techniques can identify previously unknown scenarios with no expert knowledge. However, the results obtained by UML might be uninterpretable or even feasible. SML-based methods can be regarded as a tradeoff between rule-based and UML-based methods. The training data for SML must be labeled, which can be achieved with or without expert knowledge. And implicit patterns in the training data can be extracted by SML. But UML-based methods also have similar disadvantages compared with rule-based and UML-based ones. Experts may be needed for data labeling, and the resulting scenarios may be unreasonable.

To make good use of source data, it will be a trend to apply a rule-based method to identify known scenarios and exploit UML to find unknown scenarios. After that, a feasibility analysis is needed to filter out unreasonable scenarios. Furthermore, based on all identified scenarios, some labels, such as maneuver types, road topologies, weathers, et al., could be tagged to a certain amount of the source data. Finally, A high-performance scenario detector is trained based on the labeled data and applied to detect all other known or unknown scenarios.

6. Scenario Generation

6.1. Diverse Scenario Generation

Coverage and criticality of scenarios are essential to prove AVs’ safety in ODD. Scenarios covering large spaces in ODD can be used to test AVs in various circumstances. Critical scenarios can quickly find faults in the SUT, helping developers and engineers improve AVs’ safety. Almost all scenarios detected in accident databases are challenging, but their diversity is limited. Therefore, it is necessary to generate diverse, critical, and customized scenarios for the SBT of AVs.

6.1.1. Random Sampling

Random sampling can generate concrete scenarios based on parameter ranges and distributions in logical scenarios. Monte Carlo might be the most popular random sampling technique applied for DDSG. Theoretically, random sampling can be used to produce all possible concrete scenarios in ODD if logical scenarios are available.

A 60-hour traffic video is recorded in [15] to generate car-following scenarios. Based on car-following scenarios identified from the recorded video, parameter distributions are fitted using Kernel Density Estimation (KDE). These parameters are velocity reduction, total braking time, and end velocity. Ten thousand concrete scenarios are generated by random sampling on parameter distributions.

The most significant advantage of DDSG based on NDD using random sampling is that performance of the SUT in natural traffic can be derived based on SUT performance in the generated scenarios, if the distributions of scenario parameters are given [147]. However, it is significantly inefficient to execute numerous scenarios while only a small proportion of them is critical [148]. For this reason, many studies use random sampling as a baseline to show the superiority of other methodologies, such as Reinforce Learning and Accelerated Evaluation [29,147,148].

6.1.2. Combinatorial Testing

Combinatorial Testing (CT), also named N-wise testing, is a widely used tool in software testing in which different value combinations of inputs are regarded as different scenarios [149]. CT is proposed based on the hypothesis that most errors in a system happen because of the interaction of influencing factors [150]. Kuhn and Reilly analyzed test reports and found that 70% of errors of Mozilla are caused by the interaction of two factors and 90% by the interaction of three factors [151]. While CT can be implemented for knowledge-based or data-driven scenario generation [21,84,152,153,154], The latter is the topic of this section.

CT is utilized in [153] to find errors in the Lane Deviation Warning (LDW) system. First, influencing factors are parameterized, and all continuous parameters are discretized. Second, scenarios are generated based on CT with the different significance of parameters being considered. A metric named “complex index” is defined to measure the complexity of each scenario. Third, concrete scenarios generated by CT are clustered by an HCA based on weighted distance. Scenarios in each cluster are stitched together, deriving several continuous scenarios. Finally, the continuous scenarios are executed on a virtual test platform. Simulation results show that the more complex a scenario is, the more possible errors of the SUT are found, and for a scenario with a complexity of 0.3, there is more than a 70% chance of making the LDW system fail.

CT can generate complex scenarios given parameter ranges, and the N-wise coverage metric can be applied to measure the coverage of the obtained scenario database. However, CT can only generate scenarios based on discrete values, which means parameters with continuous ranges must be discretized. For high-dimension scenarios, this might lead to a dimension explosion. Therefore, CT is mainly adopted to generate scenarios for low-level AVs.

6.1.3. Mutation Testing

Mutation Testing (MT) is a method for evaluating the quality of test sets. In MT, mutants of original test cases are generated based on pre-defined mutation operators. One original test case can derive one or more mutants. If SUT performance in a mutant scenario differs significantly from that in the original test case, the mutant will be killed. If not, the mutant stays alive. The smaller the proportion of the killed mutants among all mutants is, the better the original test set is.

Ref. [155] is a typical study of generating scenarios based on MT and Feature-Interaction-Coverage-Sampling (FIC-Sampling), as shown in Figure 5. The methodology in [155] can be divided into eight steps. In step 1, scenario templates named feature models are designed. Test cases can be obtained by modifying parameters in scenario templates. In step 2, FIC-Sampling is adopted to select parameters for concrete scenarios. FIC-Sampling is a technique similar to CT. For convenience, scenarios generated in the second step are called father scenarios. In step 3, mutants of scenarios generated in the second step are generated by pre-defined mutant operators and are called son scenarios. In step 4, father and son scenarios are all executed, and SUT performance in these scenarios is recorded. In step 5, if the performance in a son scenario is quite different from the performance in the corresponding father scenario, the son scenario will be killed and, if not, saved. The mutant metric is the ratio of the number of killed and saved son scenarios. In step 6, if the pre-defined stop conditions are meted, iteration stops, and the son scenarios generated in the last iteration are regarded as the most diverse scenarios. If not, son scenarios become parent scenarios and iterate steps 3 to 6. Two high-performance AEB systems are tested in [155], resulting in 11,145 scenarios from 30 high-quality scenario sets, better than random sampling and experts. Since FIC-Sampling can be regarded as a variant of CT, the methodology described in [155] can be considered a combination of MT and CT. The performance of this methodology highly depends on the pre-designed scenario templates, which can not only effectively avoid generating unreasonable scenarios but also stop the generation of unknown but valuable scenarios simultaneously.

MT can also be used to obtain diverse path planners. For a path planner, the weight combinations in the cost function determine its style, which may consider several aspects, such as safety, comfort, energy efficiency, etc. Different weight combinations lead to different driving styles. In [156], MT mutates the weights of cost functions, which guide the path planners to find the optimal trajectories for AVs. In the experiment, 42 path planners with different weight combinations are generated, which can be used to obtain more diverse maneuvers of vehicles around the VUT. Inspired by the Genetic Algorithm (GA), a variant of MT, SceGene, is proposed by [111]. In SceGene, scenario features are encoded as genes on chromosomes. Iteration processes, such as crossover, mutation, and selection, are activated to generate new scenarios based on initial scenarios. A microscopic driving model is adopted to repair scenarios that are not valid. This methodology is exploited to generate 1000 diverse merging scenarios in simulation experiments. The authors of [111] claim that SceGene can automatically generate diverse and realistic traffic scenarios. However, the repair rate of the merging scenario is 34.4%, indicating the unstable performance of this method. The repair rate is calculated by (repaired number)/(total number).

Similar to CT, the boosting number of scenario parameters is catastrophic for MT because many iterations would be needed to achieve an excellent mutant score. The iterative processes in MT require many computation resources, and with no guiding processes for critical scenario generation, many of the resulting scenario are not critical for the VUT.

6.2. Critical Scenario Generation

6.2.1. Accelerated Evaluation

Considering most scenarios generated by random sampling are not safety critical, an original Accelerated Evaluation (AE) based on statistical sampling is proposed by [29]. In [29], unusual behavior models of the other primary vehicles are obtained by sampling from an accelerated distribution $f^{*}\left(x\right)$, which is obtained by skewing the original distribution $f\left(x\right)$ in NDD by Importance Sampling (IS). Simulation results show that the efficiency of AE is 300 to 100,000 times better than road testing in generating critical traffic scenarios. Therefore, AE can be divided into the following six steps.

• Collect a large amount of NDD.
• Identify target scenarios.
• Fit the original Probability Density Function (PDF) $f\left(x\right)$ of each scenario parameter.
• Skew the original PDF, deriving a modified PDF $f^{*}\left(x\right)$, which will lead to more radical behaviors of traffic agents or rare scenarios.
• Random sampling is conducted based on the modified PDF $f^{*}\left(x\right)$ to generate accelerated scenarios, and then applied to test the VUT.
• The accelerated scenarios are statistically skewed back to obtain the performance of the VUT in natural traffic.

In step 3, the original PDF $f\left(x\right)$ can be fitted by Piecewise mixture distribution models [157] or KDE [147,158] to reduce the gap between the real one and the statistical one. In step 4, Importance Sampling (IS) is innovatively used in [29] to make step 6 feasible. The hyperparameters needed for IS can be obtained based on random searching or other search algorithms, such as the cross-entropy method [159] and GA [160].

In [159], the lane-change scenario, as shown in Figure 6, is described by three parameters, $v\left(t_{LC}\right),v_L\left(t_{LC}\right), R_L\left(t_{LC}\right)$. $t_{LC}$ is the time when the lane marker is crossed by the center line of the Lane-Change Vehicle (LCV). $v_L$ and $v$ are the velocities of LCV and the VUT, respectively. $R_L$ is the distance gap between the LCV and the LKV. Time to collision ${\mathrm{TTC}}_L=\frac{R_L}{{\dot{R}}_L}$ is used as the criticality metric of the lane-change scenario. Since $v=v_L-{\dot{R}}_L$, then a lane-change scenario can be described by a vector $x=\left[\begin{array}{ccc}{v}_L& {\mathrm{TTC}}_L^{-1}& R_L^{-1}\end{array}\right]$. The distribution of $v_L$ in NDD is adopted. The function of IS is to skew the Probability Density Function (PDF) of ${\mathrm{TTC}}_L^{-1}$ and $R_L^{-1}$, which are $f_{{\mathrm{TTC}}_L^{-1}}$ and $f_{R_L^{-1}}$, resulting in two modified PDFs of them $f_{{\mathrm{TTC}}_L^{-1}}^{*}$ and $f_{R_L^{-1}}^{*}$. The similarity between the original PDF $f$ and its modified PDF $f$ is measured by the likelihood in Equation (1). Then, aiming for the best $L$ close to 1, The cross-entropy method is used to search for the appropriate hyperparameters of IS, ${\vartheta }_{{\mathrm{TTC}}_L^{-1}}$ and ${\vartheta }_{R_L^{-1}}$. Experiment results show that test efficiency in lane-change scenarios is improved by IS-based AE by 200 to 20,000 times compared with natural driving testing.

$$L\left(R_L^{-1}=x,{\mathrm{TTC}}_L^{-1}=y\right)=\frac{f_{{\mathrm{TTC}}_L^{-1}}\left(x\right)·f_{R_L^{-1}}\left(y\right)}{f_{{\mathrm{TTC}}_L^{-1}}^{*}·f_{R_L^{-1}}^{*}}$$

(1)

It should be noted that the core of AE is obtaining the accelerated $f^{*}\left(x\right)$, which means that all approaches that can obtain the accelerated $f^{*}\left(x\right)$ can be used for AE [148,161,162]. An AE strategy based on kernel methods is proposed by [148]. The function of kernel methods is mapping scenario parameters to a latent space. This strategy includes six steps. First, some initial scenarios are generated by random sampling and executed, resulting in the initial training set in the original space ${\left\{\left(X_i,Y_i\right)\right\}}_{i=1}^n$, where $X$ is the scenario parameter set, and $Y$ is a binary set that indicates if the responding scenarios are critical or not. Second, the training set is mapped by a kernel function $\varphi \left(x\right)$ to a latent space, resulting scenario in the latent space, ${\left\{\left(\varphi \left(X_i\right),Y_i\right)\right\}}_{i=1}^n$. Third, a Support Vector Machines (SVM) classifier is trained by latent scenarios to find the linear boundary between critical and non-critical scenarios in the latent space. Forth, another set of random scenarios ${\left\{\left(\varphi \left(X_j\right)\right)\right\}}_{j=1}^{\mathrm{m}}$ are generated and mapped by $\varphi \left(x\right)$. Fifth, GMM is used to approximate the PDF of $\varphi \left(X_i\right)$, and leads to an approximation model $\tilde{f}\left(\varphi \left(X_i\right)\right)$, from which the modified distribution of $\varphi \left(X_i\right)$, ${\tilde{f}}^{*}(\varphi \left(X_i\right)$, is derived. Sixth, $f^{*}\left(X_i\right)$ is acquired by adjusting the marginal distribution of ${\tilde{f}}^{*}(\varphi \left(X_i\right)$ to make it closer to the area dominated by critical latent scenarios. The SVM classifier identifies these critical latent scenarios, and its accuracy needs to be improved by iterations. Simulation experiments about cut-in scenarios are carried out to validate and evaluate this strategy. Simulation results imply kernel model-based AE is 60,000 times better than random sampling. The biggest disadvantage of this method is that many iterations are needed to train the SVM classifier with random latent scenarios, which consumes a lot of computing resources. Therefore, this strategy can be improved further by detecting the performance boundary in the latent space based on fewer scenario executions.

Since the accelerated distribution function $f^{*}\left(x\right)$ is the unbiased estimation of the original estimation $f\left(x\right)$, the VUT performance in the generated scenarios can be skewed back to that in natural traffic. However, obtaining high-accuracy $f\left(x\right)$ usually requires lots of effort for data collection. Moreover, the generated scenarios based on AE are not always feasible, although feasibility analysis can eliminate this problem [127].

6.2.2. Search Algorithms

Finding the most critical scenarios is a Worst Case Searching (WCS) problem, which can also be regarded as an optimization problem. Many search algorithms have been used for WCS, including Genetic Algorithm (GA) [163], Particle Swarm Optimization (PSO) [164], multi-arm searching [165], Bayesian optimization algorithm [166], Rapidly-exploring random trees [167], Differential Evolution Algorithm (DEA) [168], and so on. A toolchain for critical scenario generation based on search algorithms is introduced in [44]. There are three keys for scenario generation based on search algorithms: define an appropriate fitness function, find a balance between exploration and exploitation, and reduce the search space as much as possible.

Fitness functions are crucial to searching algorithm-based WCS [168]. Based on pre-defined fitness functions, search algorithms can adaptively search for the worst case/scenario. Drivable area minimization is an efficient method to generate critical scenarios for motion planners. The drivable area indicates the solution space where the VUT operates appropriately without leading to any collision. Therefore, the drivable area is not relevant to the performance of the VUT. In [169], drivable areas are minimized by adjusting the initial state parameters of traffic participants using Evolutionary Algorithms (EA), such as DE and PSO. Simulation experiments involving multiple vehicles are carried out to generate critical highway and intersection scenarios. Experiment results show that by iteratively adjusting maneuvers of more than 10 traffic agents, both DE and PSO can find scenarios with little drivable space for the ego in both highway or intersection areas, and collisions are found in several highway scenarios. In contrast, no collision is produced in the intersection scenario, although several scenarios with small drivable areas are generated.

To our knowledge, no general metric can comprehensively quantify all properties of all scenarios. For example, Time-To-Collision (TTC) might be enough to measure the safety of car-following scenarios, but it is not suitable for intersection scenarios. A fitness function may consider several aspects by including several metrics. Combining metrics concerning different aspects of different aspects might be a good option to find scenarios with various characteristics (see Section 7).

How to obtain the best fitness function is still an open question [170]. Several templates for designing fitness functions are provided in [171]. However, given a suitable fitness function, it might still be challenging for search algorithms to find the most critical scenarios because of the high-dimension ODD and the increasingly complicated AVs. Therefore, a search strategy must balance exploration and exploitation, and the search space should be reduced as much as possible to find the best target more efficiently [163,172].

To enhance local searching ability, [163] combines a GA with a local fuzzer to generate the most critical trajectories of Non-Player Characters (NPCs) to minimize the safety of the SUT, and some faults of Baidu Apollo are found in a simulation experiment. A two-stage method is proposed in [173]. In the first stage, the optimization algorithm obtains diverse collision scenarios. In stage two, the purpose is to find the best parameter combination for path planners to avoid collision in the most critical scenarios found in stage one. A five-module method is proposed in [165]. These five modules include the Exploration and Exploitation Module based on the multi-arm bandit method, the Parameter Moving Probability Determination Module for ensuring the change of influencing factors, the Step Size Determination Module for searching size controlling, and the Memory Function Module for avoiding repeat searching. Simulation experiments are carried out to generate critical car-following scenarios. Experiment results show that 174 critical scenarios are generated based on the methodology proposed in [165] with 1251 scenarios executed.

Performance boundaries are the ones near which scenarios tend to lead to apparent changes in AV behaviors. Boundary scenarios near performance boundaries are valuable for AVs, and they can guide the generation of critical scenarios by reducing the search space [167,174,175,176]. In [174], decision tree classification and multi-objective population-based search algorithms are combined to find performance boundaries of critical regions. The critical scenarios found by the search algorithm can increase classification accuracy. Simultaneously, critical regions identified by the classifier can enhance the efficiency of critical scenario searching. An experiment testing an AEB system shows that 731 distinct, critical scenarios are found within 24 h.

To generate scenarios with realism, severity, and exposure considered, a heuristic algorithm is applied in [40]. First, scenarios in NDD are identified and classified into two sets, the critical and normal sets. After that, parameter distributions are fitted, and dependencies between parameters are obtained by regression analysis. Then, heuristic algorithms are applied to find the critical and high-coverage scenarios in the normal and high-coverage sets. The risk potential field and the diversity of parameter combinations are utilized to measure the criticality and exposure of scenarios, respectively. A new metric quantifying the difference between two scenarios is proposed to eliminate duplicate scenarios. This method is applied to an unpublished dataset, and 229 critical and 2065 high-coverage scenarios are found based on the critical and high-coverage datasets, respectively.

Although search algorithm-based approaches have been proven effective in some studies [111,167,174,175,176], numerous iterations are unavoidable and consume a lot of computation resources, especially for high-dimensional critical scenario generation.

6.2.3. Reinforcement Learning

Similar to the search algorithm-based method, guided by a pre-defined reward function, Reinforcement Learning (RL) searches for critical scenarios adaptively without prior knowledge about the SUT [11], as shown in Figure 7. A DDSG methodology based on Neural Architecture Search is proposed in [177]. A policy-gradient RL algorithm is applied in Neural Architecture Search. The longitudinal safe distance based on Responsibility Sensitive Safety (RSS) method is used to quantify the criticality of the situation in which ego should be responsible for the possible accident. A reward function is specifically designed, which gives a bonus to parameter combinations leading to a crash.

Q-learning is applied in [178] to search for critical scenarios. The reward function is a criticality metric that considers distance headway, Time-To-Collision (TTC), and the longitudinal acceleration required for collision avoidance. The numerical experiment shows that the proportion of critical scenarios is 37.13% more than that of randomly generated scenarios. RL-based on Long and Short-Term Memory (LSTM) architecture is applied in [179] to find the worst perception scenario. This method is evaluated by several experiments utilizing three datasets, EVB, KITTI, and BDD100K.

Adaptive Stress Testing (AST) based on deep RL is proposed by [180] to test decision-making systems. AST searches for the worst scenarios by adaptively changing the behaviors of the agents around the VUT to maximize the pre-defined reward. Simulation results indicate that deep RL is more efficient than a Monte Carlo Tree Search (MCTS) algorithm. However, iterations involved in AST require many high-fidelity simulations, which is costly and time-consuming. A methodology combining AST and the backward searching algorithm is proposed to tackle this problem [181]. First, critical scenarios in the low-fidelity simulation environment are generated by AST. Then these critical scenarios are utilized as expert demonstrations of the backward searching algorithm to find critical scenarios in the high-fidelity simulation environment. Simulation results show that the percentage of high-fidelity simulations is reduced to 4.7% without losing many rewards.

Like search algorithm-based methods, RL-based methods also belong to the falsification approach. If the VUT fails in a scenario, a conclusion that the VUT is not safe enough can be made. However, if no critical scenarios are found, it is not reasonable to claim that the VUT is safe. Furthermore, RL-based methods usually need many high-fidelity simulations, which needs sufficient computing efforts.

6.2.4. Others

A model-driven adversarial testing strategy is presented by [182], in which adversarial trajectories for NPCs are continuously generated based on a pre-defined anchor-template hierarchy structure, and a lower-level controller is designed to track these trajectories. Unlike other iteration-based methods, the VUT is tested continuously in one scenario. However, the duration of searching critical scenarios based on this method would cost much more time than one iteration of other methods, such as RL-based testing. Experiments involving highway scenarios, including several traffic agents and one VUT, are carried out to prove the effectiveness of this strategy. The simulation results show that the strategy proposed by [182] can find critical scenarios within 15 seconds. But no baseline is provided to measure the efficiency of this strategy.

6.3. Customized Scenario Generation

If the VUT is unavailable, a surrogate model of the VUT is usually utilized to generate critical scenarios. Five SMs are utilized and compared in [183], including RBF, Kriging, QP, IDW, XGB, and SVR. In simulation experiments, a modified IDM is used as the VUT. Simulation results indicate that different SMs lead to scenarios with different characteristics, and IDW-based SM is the best. It generates a large percentage of critical scenarios while exploiting only 2.5% of the test resource used by random searching. A multi-start optimization approach assisted by a seed-filling technique is described in [184] and applied in [185] for critical scenario searching based on a GPC-based SM. First, a lot of random scenarios are generated and simulated. Then these initial scenarios are labeled with critical or non-critical tags. Then they are used as the training data set for GPC. Simulation experiments of the car-following scenario are presented to evaluate this method, which is described by three parameters, including the speed of the ego car, the speed of the leading car, and the aperture angle of the radar sensor of the ego car. Simulation results show that the boundary between critical and non-critical scenarios is successfully found. However, it will need a large number of random samples for high-dimension scenarios. Therefore, it is necessary to find a method that can derive a high-performance SM without consuming too many computational resources.

There are some studies proposing methodologies for SM optimization. Most of them follow the process depicted in Figure 8. The method mentioned in [184] is further improved by an adaptively enhanced GPR-based surrogate model [166]. Ref. [166] presents an Adaptive Testing Scenario Library Generation (ATSLG) methodology using a Bayesian optimization scheme. Gaussian Process (GP) plays a crucial part in this methodology. There are two roles for GP in [166]. Gaussian Process Regression (GPR) estimates the dissimilarity between the surrogate model and the VUT. Gaussian Process Classification (GPC) estimates observations of unexecuted scenarios. GPR is applied to classify the parameter space into two sets, suboptimal scenarios and optima scenarios. The unexecuted scenarios whose observation GPR models cannot estimate accurately with high possibility are called informative scenarios. In each iteration, informative scenarios are selected and executed to provide information to improve GPR models and GPC models. This method is applied to enhance the AE method in [185] by adaptively find the most informative scenarios based on an optimized SM. Simulation results show that 18 times more critical cut-in scenarios are generated. Similar techniques can also be found in [183,186,187].

In [188], a surrogate model based on Artificial Neural Networks (ANN) is iteratively improved by iteratively generating diverse samples. Experiments of generating car-following scenarios involving three vehicles are described, and TTC is adopted as the criticality metric. Simulation results show that after 2560 scenario executions, the MSE between the results estimated by the ANN and the real ones is 0.0097.

ANN-based SM can better model high-dimension logical scenarios when enough training data are provided. The GP-based SM usually performs better when the training set size is limited. As illustrated above, some studies claim that their methods can generate customized scenarios effectively and reduce testing efforts significantly. However, no cases involving complicated AV systems are described or applied in the industry.

6.4. Conclusions of Scenario Generation

The tasks of this section are to introduce methodologies for generating diverse, critical, and customized scenarios. A qualitative comparison of the strategies mentioned above is presented in

Table 3. Qualitative comparison of scenario generation strategies. “Diversity”: Capable of generating diverse scenarios that can fill the whole ODD. “Criticality”: Capable of finding a large percentage of critical scenarios. “Knowledge”: a certain amount of expert knowledge is needed. “Iteration”: Some iterations are required. “Naturality”: testing results can be mapped to that in natural traffic. “Scalability”: Capable of generating high-dimensional critical scenarios without consuming many resources. “Y”: Yes. “N”: No. “Y/N”: Yes, for some cases, and NO, for others. For example, random sampling can generate natural traffic scenarios if sampling on the parameter distributions in natural traffic, and vice versa.

Type	Method	Diversity	Criticality	Knowledge	Iteration	Naturality	Scalability
Diversity-Oriented	Random Sampling	Y	N	N	N	Y/N	Y
	CT	Y	Y	N	N	N	Y
	MT	Y	N	Y	Y	N	N
Criticality-Oriented	AE	N	Y	Y/N	Y/N	Y	Y
	Search Algorithms	N	Y	Y/N	Y	N	N
	RL	N	Y	Y/N	Y	N	N

. The section on customized scenario generation is mainly about obtaining a good SM to assist in the generation of high-quality scenarios. Therefore, the strategies for customized scenario generation are not included in Table 3.

As for generating diverse scenarios, random sampling can achieve the best diversity, and testing results can be mapped to natural traffic. But scenarios generated based on NDD have only a small percentage of critical scenarios. CT discretizes continuous scenario parameters and generates scenarios by combining different parameters’ discrete values. A greater granularity of parameter discretization leads to more scenarios requiring more computing resources. MT generates scenarios relying on pre-defined mutation rules, which experts must design. Moreover, the resulting scenarios, including non-critical ones, would be generated and executed for AV testing, leading to unnecessary wasting of efforts. The biggest drawback of diversity-oriented methods is that many boring scenarios might be generated and executed, significantly reducing test efficiency. It is a promising research topic to reduce the execution of boring scenarios without scarifying coverage.

AE, search algorithm-based, and RL-based methodologies aim to generate critical scenarios. Unlike the other two methodologies, AE can achieve naturality and criticality at the same time [147]. However, expert knowledge might be crucial to tune vital parameters for some statistical sampling techniques, such as IS. Since the original parameter PDF is essential to AE-based methods, the completeness of source data is essential to AE. Search algorithms have been proven effective in some cases, but the involved iteration processes often require much computation, especially for high-dimensional critical scenario generation. Although RL has been proven more efficient than search algorithms in generating specific scenarios [180], both need expert knowledge to design a fitness or reward function. It is a significant challenge and worth more attention to find the most critical scenario for the VUT based on limited sources.

Customized scenarios could be generated based on high-performance SMs to minimize unnecessary execution. While many surrogate models can be chosen, such as RBF, Kriging, QP, IDW, XGB, and SVR [183], each is suitable for certain circumstances. It might be good to build several surrogate models and choose the best one for the SUT based on their functional performance.

7. Criticality Metrics of Scenarios

Criticality metrics of scenarios are of great importance in DDSG. On the one hand, criticality metrics can be used to identify or generate scenarios for DDSG. On the other hand, the safety of the SUT can also be measured by criticality metrics. Criticality metrics used in DDSG are listed and compared in [43,189]. This survey categorizes criticality metrics into five classes: trajectory-based, maneuver-based, energy-based, uncertainty-based, and combination-based. They are described in the following sections.

7.1. Trajectory-Based

Trajectory-based criticality metrics are the ones that can be calculated based on the whole or part of the trajectories of traffic participants. Time headway (THW) [190], Gap Time (GT), and Distance Head Way (DHW) quantify the spatial or temporal gap between two traffic participants without considering their velocities. Time-to-Collision (TTC) is the time to collide if two participants keep their velocities unchanged. TTC is almost the most widely used criticality metric. However, there are many scenarios in which the velocities of traffic participants change, and more than two participants exist. Therefore, some TTC variants are designed to measure the safety of the whole scenario, such as Worst Time To Collision (WTTC) [191], Time To Closest Encounter (TTCE) [192], Time Exposed TTC (TET) [193], minimal normalized positive enhanced time-to-collision (mnpETTC) [184], Time Integrated TTC (TIT), Time to Zebra (TTZ) [194] et al.

Post Encroachment Time (PET) [195] is the time gap between one traffic agent leave and another one enters a conflict area, as shown in Figure 9 and Equation (2). PET is more suitable than TTC for the criticality quantification of intersection scenarios.

$$\mathrm{PET}\left(Ve{h}_1,Ve{h}_2,\mathrm{CA}\right)=t_{\mathrm{enter}}\left(Ve{h}_1,CA\right)-t_{\mathrm{leave}}\left(Ve{h}_2,CA\right)$$

(2)

However, TTC and its variants are available only after scenario execution. For scenarios unexecuted, it is impossible to be measured by trajectory-based metrics.

7.2. Maneuver-Based

Maneuver-based criticality metrics (MBSMs) are the ones that are relative to collision-avoiding maneuvers, such as braking and steering. Criticality metrics relative to braking behaviors include Time To Brake (TTB), Deceleration to Safety Time (DST), Brake Threat Number (BTN), Required Longitudinal Acceleration, Longitudinal Jerk, and so on [190,196]. Criticality metrics involving steering maneuvers include Time To Steer (TTS), Steer Threat Number (STN), Required Lateral Acceleration (RLA), Required Longitudinal Acceleration (RLA), Required Lateral Acceleration, Lateral Jerk, et al. In some MBSMs, all kinds of collision-avoiding maneuvers are considered, including Time to Maneuver (TTM), Required Acceleration (RA), and Time to Reaction (TTR).

Based on trajectories generated by a path planner, some information about scenarios can be derived. Several criticality metrics based on trajectories are designed by [197], including SafePathInv, UnsafePercent, NarrowInv, AvgEffort, MinEffort, NarrowInv, and CriticalTime. SafePathInv is the inverse of the number of safe paths available to the VUT. UnsafePercent is the percentage of paths that will lead to a collision among all safe paths for the VUT. AvgEffort is the average effort the VUT needs to pay to pass the scenario safely. All safe paths are analyzed to obtain the absolute values of steering and acceleration to follow each safe path. MinEffort is the minimum effort required to follow a safe path. Critical Time is the minimal time available before the VUT is unable to avoid a crash. Numerical experiments indicate that most difficult scenarios in the NGSIM dataset have a score of 3.22, while a rule-based attacker designed by [197] can generate adversarial scenarios with a score of 3.95. However, this method heavily relies on the performance of the path planner. Different path planners may lead to different results.

Maneuver-based metrics do not rely on the performance of the VUT, which makes them suitable for measuring scenario criticality before executions. However, some maneuvers-based metrics, such as STN, BTN, SafePathInv, cannot separate similar scenarios with different criticality.

7.3. Energy-Based

If a collision is not avoidable, it is necessary to reduce the accident’s severity. The kinematic energy released in a collision is directly relative to crash severity [198]. Therefore, the kinematic energy of traffic participants can be used as a criticality metric. In [31], the kinematic energy of the ego at the moment of avoiding the collision is regarded as part of the Scenario Risk Index (SRI).

The kinematic energy of a traffic agent is relevant to its weight, which is not always available. Therefore, energy-based metrics often work as a complement to other criticality metrics.

7.4. Uncertainty-Based

Many factors may contribute to the crash in a scenario, such as road friction and velocity variance of vehicles. Therefore, possibility-based metrics are proposed to capture these uncertainties. The temporal variation of estimated collision speed between a vehicle and a pedestrian in a crosswalk scenario is quantified by Pedestrian Risk Index (PRI) in [199]. Crash Potential Index (CPI) [200] is the average crash possibility if the required deceleration exceeds the maximal available deceleration in the scenario. The maximal available deceleration in the scenario is described as a distribution relevant to objective factors such as the road material and performance of the braking system. Parameters in CPI are obtained by calibration based on traffic data in an intersection in NGSIM dataset in [200]. No application case of this metric is described in [200].

In natural traffic, a minor behavior change of the traffic participants around the ego might lead to a fetal accident [201]. Ref. [202] insists that a scenario is critical if the VUT cannot predict the trajectory of at least one participant. In [203], all possible trajectories of NPCs are predicted, and Monte-Carlo simulations are carried out to estimate Time-To-Critical-Collision-Probability (TTCCP) to consider the uncertainty of their behaviors.

7.5. Combination-Based

Different metrics with different properties can measure scenarios from different perspectives. Five metrics are exploited in [170] to guide evolutionary algorithms to find scenarios with different properties, including criticality. Furthermore, as a general metric that can measure the safety of all scenarios does not exist now, combining different metrics is a good option.

On the one hand, several criticality metrics can be adaptively used to measure scenario criticality. An adaptive methodology to measure scenario safety is described in [204]. Several criticality metrics concerning different aspects are selected based on multidimensional criticality analysis. A situation awareness module is designed to identify the type of the current scenario. Applicable metrics in the current scenario are calculated, and those exceeding pre-defined thresholds are weighted and summed together, resulting in the safety of the current scenario. In [178], a combination-based safety, including longitudinal acceleration, time headway, and TTC, is applied as a reward function to guide critical scenario generation based on RL.

On the other hand, metrics concerning several other aspects except safety may be integrated. The production of exposure, severity, and controllability is used to quantify the risk of a scenario in [158,205]. Exposure is the expected happening possibility of a scenario. Severity is the expected possibility of collision if no backup operators are available. Controllability is the ratio of expected collision possibility with and without backup operators.

7.6. Conclusions of Criticality Metrics of Scenarios

Five classes of criticality metrics are considered in this section. The trajectory-based criticality metrics can be calculated given all trajectories of traffic participants in a scenario or all positions in a scene. Maneuver-based ones are proposed to measure the difficulty of avoiding an accident. Energy-based ones are applied to measure the severity of a crash. Uncertainty is the key to uncertainty-based criticality metrics. More uncertainty in a scenario would lead to more challenges for the SUT. Unlike other criticality metrics, a combination-based critical metric integrates several metrics concerning different aspects, resulting in a more comprehensive metric. However, to our knowledge, no single criticality metric can be utilized for all scenarios. Therefore, researchers are advised to design or adopt appropriate criticality metrics for different scenarios. A general and objective criticality metric for all scenarios does not exist by far.

8. Discussions and Conclusions

This work decomposes methodologies described in relevant studies into solutions to several fundamental problems about DDSG, including source data collection, scenario identification, scenario generation, and criticality metrics of scenarios. Involved techniques for similar problems are analyzed and compared with each other. Conclusions about each methodology mentioned in this survey can be found at the end of the related section. To avoid unnecessary repetition, they are not included in this section. Some hot research topics are summarized as follows:

• Develop a toolchain that can generate good-quality traffic-scenario data on simulation platforms to reduce the efforts and investments for gathering source data in the real world.
• Build a methodology that can effectively and efficiently identify known and unknown scenarios in source data without sacrificing feasibility to use all source data fully.
• Use different methodologies to generate diverse, critical, and natural scenarios to meet different requirements in different development stages.
• Find a strategy to obtain high-performance Surrogate Models (SM) based on limited resources.
• Design a general criticality metric that can objectively quantify the criticality of all scenarios.

There are also some significant problems in AV testing, rather than only existing in DDSG, and the authors of this survey think it is necessary to point out. Here are some of the most crucial ones:

• Simulation fidelity and computing power need improvement. Simulation with high fidelity is crucial to executing scenarios and can contribute to source data generation. Simulation technology has been widely utilized in SBT. However, on the one hand, many studies utilize simulation techniques to execute scenarios. On the other hand, no software companies claim that their software can replace experiments in the real world. If it is impossible to replace the real world with a virtual one, it will be helpful to quantify the gap between them, which can let us know how much we can trust the simulation results. Moreover, it is of great significance to use low-fidelity simulations to reduce high-fidelity simulations, which are more expensive and consume more time.
• There are no conclusions on how many of what scenarios are enough for AV testing. There is an infinite number of scenarios in the physical world. It is impossible to test AVs in all of them. An embarrassing dilemma is that many studies propose many scenario-based methods to test AVs, and no one concludes how many of what scenarios are enough for AV testing. It is significantly vital to draw a terminal line for this endless Marathon.
• Data sharing is crucial for AV testing. Safety-critical events hidden in NDD are crucial for DDSG. However, because of the Curse of Rarity (CoR) [206], a large amount of NDD would significantly contribute to DDSG. While some open datasets are available for researchers, giant companies like Tesla, Baidu, Didi, et al. hold a large amount of NDD privately. Furthermore, because many functions are complete black boxes, it is hard for a researcher or an engineer to generate customized scenarios for the VUT. It is reasonable to believe that more comprehensive cooperation between industry and academia can tremendously enhance the development of SBT of AVs.
• The unignorable gap between ideology and reality deserves more attention: While one of the aims of developing AVs is to reduce traffic accidents to zero, achieving zero accidents in practice is severely challenging. It might be good to mitigate the public expectation to an appropriate level to let more un-perfect but good AVs be tested in natural traffic. This way, AVs will evolve and collect more valuable data for researchers.