Vision Transformer for Detecting Critical Situations and Extracting Functional Scenario for Automated Vehicle Safety Assessment

Abstract

Automated Vehicles (AVs) are attracting attention as a safer mobility option thanks to the recent advancement of various sensing technologies that realize a much quicker Perception–Reaction Time than Human-Driven Vehicles (HVs). However, AVs are not entirely free from the risk of accidents, and we currently lack a systematic and reliable method to improve AV safety functions. The manual composition of accident scenarios does not scale. Simulation-based methods do not fully cover the peculiar AV accident patterns that can occur in the real world. Artificial Intelligence (AI) techniques are employed to identify the moments of accidents from ego-vehicle videos. However, most AI-based approaches fall short in accounting for the probable causes of the accidents. Neither of these AI-driven methods offer details for authoring accident scenarios used for AV safety testing. In this paper, we present a customized Vision Transformer (named ViT-TA) that accurately classifies the critical situations around traffic accidents and automatically points out the objects as probable causes based on an Attention map. Using 24,740 frames from Dashcam Accident Dataset (DAD) as training data, ViT-TA detected critical moments at Time-To-Collision (TTC) ≤ 1 s with 34.92 higher accuracy than the state-of-the-art approach. ViT-TA’s Attention map highlighting the critical objects helped us understand how the situations unfold to put the hypothetical ego vehicles with AV functions at risk. Based on the ViT-TA-assisted interpretation, we systematized the composition of Functional scenarios conceptualized by the PEGASUS project for describing a high-level plan to improve AVs’ capability of evading critical situations. We propose a novel framework for automatically deriving Logical and Concrete scenarios specified with 6-Layer situational variables defined by the PEGASUS project. We believe our work is vital towards systematically generating highly reliable and trustworthy safety improvement plans for AVs in a scalable manner.

1. Introduction

With the advancement of autonomous driving technology, expectations for continuing commercial success of Automated Vehicles (AVs) are rapidly increasing [1,2]. Society of Automotive Engineers defines AVs as vehicles capable of navigating, controlling, and avoiding risks partly or totally without human assistance and divides them into six levels of driving automation [3]. In reality, most current commercial AVs are evaluated at Level 2 or 3, i.e., capable of partial or conditional automation [4]. AVs are viewed as the critical components of the Intelligent Transportation System to contribute to traffic flow enhancement. In particular, as driver negligence accounts for more than 90% of the causes of traffic accidents, AVs are attracting attention as a safer alternative transportation mode [5,6]. However, there are still hurdles to meeting the above-mentioned expectations due to errors in recognizing surrounding objects during autonomous driving and operational control transfer in a moment of emergency [7]. AVs may risk human drivers and passengers, especially when they transfer the control back to the human drivers [8,9]. Some researchers tried to leverage Vehicle-To-Everything technologies to help AVs become more promptly and accurately aware of dangerous circumstances based on the information provided by the nearby vehicles, pedestrians, and traffic infrastructure over a high-speed wireless network [7,10,11,12]. Evaluation criteria are being discussed for safety standards and operations specifically for AVs [10]. The AV safety assessment is a part of the New Car Assessment Program that tests the safety level of vehicles. Various evaluation criteria have been proposed based on ISO 26262 [13], an international automotive functional safety standard for preventing accidents caused by defects in the Electric and Electronic system. Simulation-based accident scenario studies for safety evaluation have been discussed [14,15]. However, most previous studies were based on limited evaluation criteria for temporary test drive permits. Current scenarios do not sufficiently reflect traffic conditions and unique accident patterns in various countries and regions of concern. Test plans based on such imperfect accident scenarios cannot fully assess AVs’ safety functions on the road. Therefore, it is necessary to develop test scenarios based on actual accident data to obtain a broader spectrum of trustworthy safety criteria for AVs [16,17]. However, obtaining actual AV accident data is highly challenging and costly. To address the limited availability of the actual AV accident data for research purposes, we employ real accident data of human-driven vehicles (HVs) as a source of scenario extraction because HVs are known to exhibit similar patterns of accidents as AVs, according to recent studies [18,19,20]. Specifically, we base our analysis on the Dashcam Accident Dataset (DAD), which contains video clips of collisions [21]. The lack of objectivity in accident scenarios due to human experts’ intervention is a limitation. Hence, data-oriented analysis has been emphasized [17]. Artificial Intelligence (AI) has emerged as a replacement for human experts for deriving unbiased accident scenarios based on data [22]. Virdi et al. [23] used deep recurrent neural networks such as Long Short-Term Memory (LSTM) algorithm to analyze the sequential pattern of accidents. However, previously employed machine learning algorithms for accident modeling come short in accountability. Hence, explainable Artificial Intelligence (XAI) technologies are emerging to interpret the inference made by the AI transparently. XAI improves the trustworthiness of AIs by analyzing the relationship between input data, model parameters, and inference outcomes. Highly reliable XAI results were produced in various problem domains [24,25,26]. In particular, Transformer neural network with Self-Attention-based feature encoding layers identifies critical parts of the input data [27]. With the explanation capability, Transformer showed outstanding performance for continuous sequence analysis in Natural Language Processing (NLP) [27], Speech-to-Text [28,29], Reinforcement Learning (RL) [30,31], and Computer Vision (CV) [32,33,34]. Motivated by the success of Transformers in various previous research works, this study aims to make the following novel contributions through the overall approach we illustrated in Figure 1. We accurately classify actual road accidents and critical situations using Vision Transformer-Traffic Accident (named ViT-TA), an Attention-based Transformer algorithm customized for computer vision data. Specifically, ViT-TA was applied to a stream of accident video frames recorded by HVs. Here, we assume that the ego vehicles in the video have AV safety functions. From the time-series of visual Attention maps, we can identify the Spatio-temporal characteristics of critical situations around accidents, such as road conditions and dangerous movements of the vehicles referred to as Target Vehicles (TVs) causing the accidents [35]. Based on the ViT-TA-assisted situation interpretation, we derive intuitive Functional accident scenarios using 6-Layer information defined by the PEGASUS project [14,36,37,38].

Figure 1. Overview of Vision Transformer-assisted safety assessment scenario extraction.

Our study is presented in the following order: In Section 2, we draw the novelty of this study by reviewing the related works on the safety evaluation of AVs and traffic accident scenarios. In Section 3, we describe Dashcam Accident Dataset (DAD) preprocessing and ViT-TA architecture for modeling situations around traffic accidents. In Section 4, we evaluate the performance of ViT-TA. Section 5 explains how we derived Functional accident scenarios and suggests a framework that automatically generates more concrete and highly objective safety test scenarios. Finally, Section 6 makes concluding remarks.

2. Related Work

In this section, we review the research on various methods for evaluating the safety of AVs. AVs have to fulfill safe driving tasks under various operating circumstances. Traditionally, human experts have composed the operational scenarios based on their knowledge and experience. However, experts’ subjectivity and intervention can make scenarios non-systematic and impractical [36]. The safety assessment procedures written by human experts cannot wholly and quantitatively cover various critical situations [17]. Various studies considered alternative approaches based on simulations and AI algorithms to resolve the shortcomings of manual scenario authoring by humans. The simulation-based methods can systematically generate scenarios in a scalable manner. Lim et al. [39] simulated AVs to assess their capability of satisfying the minimum safety requirements under various scenarios. Kim et al. [40] evaluated AV’s controllability during the fault-injected test simulations. Their study concluded with a list of safety concepts and indicators that should appear in test scenarios. Park et al. [41] developed scenarios for evaluating safety measures during take-over situations on virtually simulated highways. Similarly, Abbas et al. [42] demonstrated various dangerous situations during autonomous driving through a virtual simulator based on the Grand Theft Auto V game. Recently, simulation methods have been studied in conjunction with AI technologies. Strickland et al. [43] used Bayesian Convolutional LSTM (Conv-LSTM) to predict collision risks during the autonomous and assisted driving mode. They generated collision scenarios using traffic simulators such as VISSIM and SSAM. Jenkins et al. [10] created accident scenarios by feeding V2X data such as vehicle speed and directions to Recurrent Neural Networks. They measured the Kernel Density similarity between the synthesized scenarios and the actual accident data. Fremont et al. [44] proposed comprehensive scenario-based experiments to generate 1294 safety assessment cases. However, some of the scenarios based on the simulations may not be valid in the real world. Therefore, looking into the actual accident data as a source for more concrete and realistic scenarios is imperative. Yao et al. [45] implemented a method for tracking traffic participants recorded in first person-view videos to predict potential accidents. Maaloul et al. [46] used a heuristic method based on Farneback Optical Flow to detect motions that can lead to highway accidents. Morales et al. [47] presented a semantic segmentation method for detecting road obstacles in ego-vehicle videos. They computed obstacle trajectories using monocular vision and an extended Kalman filter. Ghahremannezhad et al. [48] devised a real-time road boundary detection method to point out the vehicles unsafely crossing the boundaries. Agrawal et al. [49] developed a system that promptly alarms the emergency services upon traffic accidents detected by a deep neural network. Kim et al. [26] presented a traffic accident detection method based on a Spatio-temporal 3D convolutional neural network that consists of a pre-activation ResNet and a Feature Pyramid Network. These works focused on improving the accident detection accuracy and paid less attention to the causality analysis necessary for accident scenario composition. Sui et al. [50] clustered accident reports to obtain six crash scenarios based on the time of the crash, visual obstruction, and pre-crash behavior of the car and two-wheeled vehicles. Some of the scenarios they discovered resembled those provided by the Euro New Car Assessment Program. Furthermore, the scenarios reflected the unique characteristics of different countries. Holland et al. [51] designed virtual roads in which crashes occurred based on Open Street Maps and applied the NLP technique to generate the crash scenario from a text description of the crash event. The scenarios specified the real features of roads, such as the curvature, the number of lanes, and speed limits. Geng et al. [52] constructed an ontology of road elements, traffic participants, and their inter-relationships. They computed the probability of accidents on urban roadways given the ontological information. They claimed that the explicit specification of the inter-relationship can be used to interpret the accidents in detail, and the result of such interpretation can be reflected in the safety test scenarios for AVs. Yuan et al. [53] identified nine field variables such as time, space environment, and accident participation from real accident data to describe 4320 compelling risk scenarios. Given those variables, they used an artificial neural network to compute the probability of accidents. Park et al. [35] mined text reports of accidents on the community roads to extract keywords for traffic safety situations. Using an NLP technique and the keywords, they followed a 5-layer format to compose 54 functional scenarios that can be tested on urban arterials and interactions. Lenard et al. [54] studied typical patterns of pedestrian accidents and proposed pattern variation methods to generate scenarios pertaining to less common and severe accident circumstances. These previous research works did not examine the accident patterns in the context of AVs. The AI-based methods mentioned above could not automatically pinpoint the objects that caused accidents. Thus, these methods still relied on human experts to conclude the causality. As mentioned earlier, the correctness of the interpretation made by human experts is questionable. Karim et al. [55] proposed the Dynamic Spatial-Temporal Attention (DSTA) model for not only anticipating traffic accidents but also for identifying probable causes from Car Crash Da (CCD) [56]. While their work proposed an intriguing technical direction for interpreting accident scenes, they did not fine-tune their model to reliably detect exact points where the critical situation starts to develop. Neither of the works aimed to study a method for generating safety test scenarios for AVs. To address the limitations of the previous works, we customized an Attention-based Transformer neural network (ViT-TA). Our network is trained to account for the causes of traffic accidents recorded in the Dashcam Accident Dataset [21]. Given accurately pointed reasons, we can more reliably reflect the interpretation of the critical situations onto the safety test scenarios for AVs. Unlike DSTA, ViT-TA does not restrict the understanding of accidents only regarding the pre-recognized surrounding cars. Therefore, ViT-TA can also highlight non-vehicle objects as a probable cause of accidents that otherwise could be overlooked by DSTA or human experts. We derived AV-oriented safety test scenarios that happen at the moment with a TTC shorter than the typical Perception-Reaction-Time (PRT) of human drivers in critical situations. We believe our method can improve the objectivity of the safety test scenarios for AVs as they are based on the accurate accounting of actual accidents by ViT-TA.

3. Methodology

In this section, we present the overall procedure of deriving AV accident scenarios for safety assessment, as shown in Figure 2. The process consists of three steps. First, image frames are extracted from ego-vehicle video clips containing traffic accidents between Target-Vehicles (TVs). Each frame is categorized into four classes, Normal, Pre-Collision, Collision, and Post-Collision, according to TTC values considering the conventional PRT of AVs. Second, a ViT-TA neural network model is fine-tuned to produce Attention maps that visually indicate critical factors of dangerous situations. Finally, accident scenarios are composed based on the analysis result returned by ViT-TA.

Figure 2. Framework for deriving AV accident scenarios based on real-world accident data set.

3.1. Dataset

We used the Dashcam Accident Dataset (DAD) that contains 620 accident First Point of View (FPV) videos collected between 2014 and 2016 through dash cameras mounted on HVs in six Taiwanese cities (Taipei, Taichung, Yilan, Tainan, Kaohsiung, and Hsinchu) [21]. The proportions of the accident types are as follows: 42.6% motorbike-to-car accidents, 19.7% car-to-car accidents, 15.6% motorbike-to-motorbike accidents, and 20% other miscellaneous types; 3% of the accidents occurred during night time. In addition to accident clips, 1130 normal traffic videos are included. The DAD contains relatively more challenging videos showing accidents taking place in complicated roads and crowded streets. The size of each frame is 224 × 224. We used 455 videos for training and 165 videos for testing. Each 4 s video with 720p resolution comprises 100 frames played back at 25 FPS (Frames Per Second). Note that we assumed the HV of FPV to be an ego-vehicle with AV functions.

Most previous studies focus mainly on the very moment of the collision itself. As shown in Figure 3, we classified each video frame into one of four classes based on TTC to understand the precursors and aftermaths of accidents. The four classes are normal situation before an accident (Normal), a situation that can lead to a collision (Pre-Collision), a moment of collision (Collision), and a situation after the crash (Post-Collision). Most studies regarding HV accidents focus on the situation at TTC < 2.5 s, considering the typical PRT of human drivers [57]. PRT is defined as the time interval at which the driver first represents a response to an object or situation in the driver’s view. PRT is divided into four stages: perception, identification, motion, and reaction. The American Association of State Highway and Transportation Officials (AASHTO) also uses the cognitive response time of 2.5 s to design safe roads [58]. However, technical advancements helped HVs avoid unexpected critical situations much quicker. For example, most recent vehicles have an automatic transmission that does not require the overhead of changing forward gears [59,60,61]. Advanced Driver Assistance Systems, such as Lane Keeping Assist and Forward Collision-Avoidance Assist, have helped vehicles respond to critical situations at PRT < 2.5 s [62,63,64,65].

Figure 3. Labeling the DAD video frames with critical situations.

Considering AVs are more dexterous than the HVs, we define the Pre-Collision situation as the moment when TTC < 1 s, as shown in Figure 4. It is assumed that AV passengers are well adjusted with no fear to the reduced PRT in a near collision situation. Videos in the DAD were mainly about the accidents between nearby TVs, not the FPV ego vehicles. However, the FPV ego vehicles we assumed to be the AVs were well within reach of the dangerously moving TVs at TTC < 1. Therefore, the situation around the nearby TVs’ accidents can be regarded as a critical situation in which the automated ego vehicles become vulnerable to becoming involved in another accident as a chain reaction.

Figure 4. Accident video frame labeling based on TTC more suitable for understanding critical situations for AVs.

After screening 45,500 frames, we learned that Pre-Collision situations started to be captured at the 74th frame on average. The Normal driving situation is set at the 25th frame and forward up to TTC to the frame just before the Pre-Collision state. The scene of the accident lasted 0.5 s on average. Frames after 0.5 s were labeled as Post-Collision. We cropped out the first 24 frames marked Normal from every video to avoid the imbalance among the four classes in terms of data amount. Hence, the total number of frames for training and testing ViT-TA was 24,740, as shown in Table 1.

Table 1. The Number of Frames per Class.

Cropped-Out Normal Frames	Normal	Pre-Collision	Collision	Post-Collision
20,760	6684	6691	6614	4751
	24,740

3.2. Vision Transformer-Traffic Accident (ViT-TA)

Vision Transformer (ViT) [66] is a Transformer algorithm [27] applied to images and videos. The original Transformer was designed to conduct Sequence-to-Sequence (seq2seq) modeling based on Encoder and Decoder with the layers computing scaled dot-product Attention values. The Attention values represent the importance of a token in converting a sequence to another sequence. For ViT, a token is a patch of image pixels, whereas a token for seq2seq modeling is conventionally a natural language word. In this study, we customized ViT to devise ViT-TA (ViT-Traffic Accident). We first present how ViT-TA computes the embedding of objects and their positions as follows. ViT-TA dissects a C × C image into patches. The content of each P × P patch is embedded into a latent vector with a length of D through linear projection. A token of the class label is embedded into a D-length latent vector so that ViT-TA can learn to classify the scene in an image. We refer to the latent vector constructed through this process a content embedding. Then, ViT-TA encodes each patch position according to Equation (1), where p is the offset of the patch from top left-hand corner of an image, D is the length of the vector containing encoded vectors, and i = 0, 1, …, D. Position embedding has the exact dimensions of the content embedding. The position encoding plays a crucial role in representing the relative location of a patch to others. Moreover, the patch positions are essential for interpreting critical situations leading to accidents.

$$pe(p,i)=sin\left(\frac{p}{{10000}^{\frac{2i}{D}}}\right)$$

(1)

ViT-TA combines the content and the position embedding by vector addition. Finally, a (C/P)2 × D input matrix is generated. ViT-TA feeds the input matrix to the Transformer Encoder block, with its architecture shown in Figure 5. Each Transformer Encoder block is a network of normalization (Norm), Multi-Head Attention (MHA), and Multi-Layer Perceptrons (MLP) blocks. Each Attention Head in the MHA block computes scaled dot-product Attention scores of the image patches according to Equation (2). Q, K, and V are matrices of initially random parameters for computing the Attention score of every patch, and dk is the dimension of the split segment of combined embedding on each of N Attention Heads.

$$Attention(Q,K,V)=softmax\left(\frac{Q{K}^t}{\sqrt{D_k}}\right)V$$

(2)

Figure 5. Processing of accident scenes through Vision Transformer for Traffic Accidents (ViT-TA).

ViT-TA computes Attention scores in parallel and feeds the normalization of the output from every Attention Head to MLP, which uses Gaussian Error Linear Unit as the non-linear activation function. We stripped the Transformer decoder block from the original Transformer and let MLP determines the class of a given image. During training, parameters in Q, K, and V matrices converge, and the Transformer Encoder block learns to associate the computed Attention scores of patches with the classification result of MLP. Upon detecting critical situations, we regard the objects appearing over the image patches with high Attention scores as the probable causes of accidents. Note that we did have to encode the sequence of image frames explicitly using RNNs such as LSTM favored by many previous traffic studies. Instead, the explicit class labeling and the position encoding prevent ViT-TA from becoming confused with the proper order of transitions during critical situations. The MHA in ViT-TA as a rich semantic representation of image frames can be modeled in parallel. Thus, ViT-TA has a training speed advantage over RNNs as well.

4. Evaluation of ViT-TA Model

Processing of accident scenes through Vision Transformer for Traffic AccidentsWe preprocessed DAD with Open CV (v4.5) and implemented ViT-TA with Python (v3.9.7) and Tensorflow (v2.6.0). We empirically configured the ViT-TA structure to produce an optimal model, as shown in Table 2 and Table 3. The Dense network at the end is connected to the Softmax layer to classify the scene in an image frame. We set the patch size to 32 × 32. We had a total of 12 Attention Heads in the Transformer Encoder. The length (D) of the combined embedding of each patch was 768. A set of 24,740 video frames from DAD was divided into train and test data at a ratio of 8:2. We varied the batch size between 16, 32, and 64. For normalization and model generalization, we used Categorical Cross Entropy as a loss function with label smoothing set to 0.2 [67]. For optimization, we employed Rectified Adam (RAdam) [68], which complements the limitations of Adaptive Moment Estimation (Adam). We iterated the training process up to an epoch of 1000 with early stoppage enabled. We trained ViT-TA and evaluated it on a Window 10 Deep Learning machine with Intel(R) Xeon(R) Gold 5220R CPU, a 192GB RAM, and an NVIDIA GeForce RTX 3090 GPU. As a result, we fine-tuned the model with a total of 87,466,855 parameters.

Table 2. Configuration of ViT-TA Structure.

Layer	Output Shape	Parameter
Input	[(None,224,224,3)]	0
Conv2D Embedding	(None,7,7,768)	2,360,064
Reshape	(None,49,768)	0
Class Token Embedding	(None,50,768)	768
Transformer/Position Embedding	(None,50,768)	38,400
Transformer/Attention Head 0∼11	(None,50,768),(None,12,None,None)	7,087,872
Transformer/LayerNormalization	(None,50,768)	1536
Lambda ExtractToken	(None,768)	0
Flatten	(None,768)	0
Batch normalization	(None,768)	3072
Dense	(None,11)	8459
Batch normalization	(None,11)	44
Dense	(None,4)	48

Table 3. Configuration of ViT-TA Training.

Parameters	Description
Data (Train included validation, Test)	19,792 (80%), 4948 (20%)
Batch size	16, 32, 64
Loss	Categorical Cross Entropy/label smoothing = 0.2
Optimizer	RAdam (Rectified Adaptive Moment Estimation)
Epochs	1000
Learning Rate	1 × 10−4∼1 × 10−6/monitor = validation accuracy/patience = 2
Early Stoppage	monitor = validation accuracy/patience = 5

For the evaluation of the model, we use the following metrics: (1) accuracy (Equations (3)); (2) precision (Equations (4)); (3) recall (Equations (5)); and (4) f1-score (Equation (6)) using precision and recall. A True Positive (TP) is an outcome where the model correctly predicts the positive class. A True Negative (TN) is an outcome where the model correctly predicts the negative class. A False Positive (FP) is an outcome where the model incorrectly predicts the positive class. A False Negative (FN) is an outcome where the model incorrectly predicts the negative class.

$$Accuracy=\frac{TP+TN}{TP+FN+FP+TN}$$

(3)

$$Precision=\frac{TP}{TP+FP}$$

(4)

$$Recall=\frac{TP}{TP+FN}$$

(5)

$$F1-score=\frac{2*precision*recall}{precision+recall}$$

(6)

The learning was conducted until the validation accuracy reached a peak of 99%, as shown in Table 4. After the training, ViT-TA achieved the highest test accuracy of 94% when we set the batch size to 64. According to the confusion matrix, ViT-TA remarkably never misclassified Collisions as Normal situations. A small portion of pre-collision situations was falsely identified as collisions. Despite a few FPs and FNs, we were not hampered in identifying the precursors of crashes as Attention maps were given, as shown in Figure 6.

Table 4. ViT-TA learning Result.

			Loss	Accuracy
Batch Size	64	Train	0.6023	0.9980
		Test	0.6869	0.9401
	32	Train	0.6014	0.9989
		Test	0.7083	0.9348
	16	Train	0.6056	0.9987
		Test	0.7159	0.9338
Class	Precision	Recall	f1-Score	Support
Normal	0.9581	0.9537	0.9558	1296
Pre-Collision	0.9241	0.9372	0.9306	1339
Collision	0.9386	0.9261	0.9323	1354
Post-Collision	0.9398	0.9447	0.9422	959
Average	0.9402	0.9404	0.9402	-
Total Accuracy	0.9400			4848

Figure 6. Illustrating the critical situation with Attention maps and scores generated by ViT-TA.

Attention scores of the image patches are visualized by the transparencies. High transparency indicates a high Attention score. For Normal situations, the entire image was nearly opaque, as the Attention scores of the patches were universally low. When the pre-collision situation was detected, ViT-TA pinned a focal point on the objects involved in the collision, as shown in Frame #70 in Figure 6. Interestingly, the focal point fades out over the subsequent frames that eventually contain the scenes of Collisions and Post-collision. Starting from the Pre-collision state, the total Attention scores of the image frames continuously increased until they reached the peak at Post-collision (Frame #85). Such visually salient transition of the Attention scores over the frames and the critical objects highlighted at the Pre-collision state helped us understand how the situation unfolded. Karim et al. (2022) [55] also trained DSTA with the DAD to point out the entities involved in a collision. Karim et al. set an arbitrary threshold of crash probability exceeding 50% to identify a critical point. With such thresholds, the mean TTC was approximately 4.5 s. However, the critical moments actually occurred mainly at TTC ≤ 1 s, according to our in-depth screening of the DAD videos. Perception-To-Response Times (PRT) of human drivers and AVs are typically 2.5 s and 1.5 s, respectively. Considering the significantly shorter PRT, the credibility and usefulness of the arbitrary thresholds set for DSTA are questionable. Focusing on the critical moments at TTC < 1 s, our ViT-TA outperformed DSTA with an average accuracy 34.92% higher than DSTA. As introduced earlier, DSTA recognized cars prior to classifying the video scenes. However, we argue that such object recognitions must be performed afterward, especially when accident scenario writing is a concern. DSTA can overlook unclassified objects as crucial factors for accidents. Removing the biases by not restricting ViT-TA only to vehicles was another reason for outperforming DSTA.

5. Composition of Vulnerable Traffic Accident Scenarios Based on ViT-TA

In this section, we share our experience composing a safety assessment scenario for AVs based on the preliminary analysis result returned by ViT-TA. Research projects such as PEGASUS [38], CETRAN [69], ENABEL-S [70] studied AV safety scenarios. In particular, through the PEGASUS project, academic researchers and industry partners, such as BMW and Audi, systematized methods and requirements for the safety of autonomous driving functions. In this study, we defined the template for composing what PEGASUS refers to as Functional scenarios that describe critical situations at a high level, as shown in Figure 7 [37]. The Functional scenarios specify the AV functions to be evaluated under certain circumstances expressed with 6-Layer information such as road levels, traffic infrastructure, events, objects, environments, and digital information (Figure 7).

Figure 7. Functional, Logical, and Concrete scenarios defined by PEGASUS based on 6-Layer information.

In the following, we explain each section in the sample Functional scenario illustrated in Figure 8. The safety Function Requirement section states the objective of safety evaluation. For instance, the goal is to test how AVs respond to the sudden stopping of target vehicles ahead. The Traffic Flow Pattern section describes how target vehicles moved, for example, lane changes. The Location section specifies where the critical situation took place. The Visualization section illustrates the critical situation. The template offers an area for full-text description of safety functions required under the critical situation. The Road Geometry section describes road level information such as the number of lanes and road alignments. The Object Movement section describes the maneuvers of the objects involved in the critical situation. This section specifies the details such as vehicle types, driving lane of the vehicles, and the location of the target vehicles relative to the position of the ego vehicle. Finally, the raw snapshot of the scene showing the critical situation and its attention map is attached at the top of the scenario template to highlight the factors contributing to the critical situation. Examples of the attention maps for some of the critical situations recorded in the DAD are shown in Figure 8.

Figure 8. Illustrating the Critical situation of attentions of the ViT-TA for Extraction functional AV Scenarios.

The first case shows a two-wheeled vehicle suddenly cutting in and crashing into the car ahead, slowing down on the third straight lane during the daytime. The second case shows a car suddenly switching lanes and colliding with another vehicle on the first lane during nighttime. The third case shows the accident ahead due to a car that ran a red light. The last case shows the collision between two vehicles ahead simultaneously crossing lanes on the curved road during nighttime. ViT-TA was able to pinpoint the objects involved in the collisions. We can regard that the highlighted things are likely to invoke a critical situation for the ego vehicle having AV functions. The visual cues provided by ViT-TA can help safety inspectors describe Functional accident scenarios. We can plan a test that evaluates the AV’s capability of recognizing and evading the critical situations described in the Functional scenario. Figure 9 shows a Functional scenario describing the first case shown in Figure 8.

Figure 9. Functional Scenario #1: Evaluation plan for the sudden accident between Target Vehicles ahead.

However, the Functional scenarios do not provide the configuration of situational values necessary to define a precise and concrete safety evaluation plan. The PEGASUS project introduced Logical and Concrete scenarios that specify the range and initialization of the situational variables, as shown in Figure 7. It was not feasible to extract the exact values of the situational variables such as TTC, vehicle speed, acceleration, the distance between objects, lane numbers, and object trajectories from the 2D camera. Values of the situational variables can be obtained from 3D point cloud data recorded by depth cameras such as LiDAR. Given the depth data, we propose a reference framework for automating the writing process of Logical and Concrete scenarios with minimal human intervention, as illustrated in Figure 10 and Figure 11.

Figure 10. Framework for automation of Logical and Concrete scenarios.

Figure 11. An example of a Logical and Concrete Scenario for AV safety assessment.

We first extract features from data collected by the sensors on AVs. We train ViT-TA with the feature embeddings seen under Normal, Pre-collision, Collision, and Post-collision situations. After the training, ViT-TA detects and highlights objects contributing to the critical conditions. Through object recognition [71], semantic segmentation, and image classification techniques [72,73], the types and geometry of the highlighted objects can be inferred. Upon detection of the first appearance of the critical objects, video frames up to the Post-collision point are iterated to compute the values of the situational variables, such as speed and direction of the vehicles and the distance between critical objects. The situational variables can be randomly adjusted for scenario variations to cover peculiar AV accident patterns not typically seen in HV accidents [74]. Finally, we can adopt text generation techniques [51] to fill out the natural language description of the safety function required under critical situations. Understanding the text accident reports requires recognizing named entities and their inter-relations that can be mined with various machine learning techniques [75,76]. We expect that such an automation framework minimizes human intervention and promises a highly objective scenario for the safety assessment of AVs. To improve the credibility of the scenarios further, we plan to employ the latest development of statistical explanation techniques in addition to the Transformer-based visual interpretation approaches [25,26,77,78].

6. Conclusions

In this paper, we presented a custom Vision Transformer (ViT-TA) for automatically pointing out the clues from videos containing the scenes of critical situations leading to potential traffic accidents. With 24,740 frames from Dashcam Accident Dataset (DAD) used as training data, ViT-TA could classify video scenes with an average f1-score of 94. ViT-TA’s average accuracy was 34.92% higher than DSTA in detecting the critical moment at TTC < 1, considering AVs’ typical Perception-Reaction-Time (PRT) time. While most previous AI-based methods focus on catching the accidents, ViT-TA’s attention map highlighting the critical objects helped us understand how the situations unfolded to put hypothetical ego vehicles with AV functions at risk. Based on the ViT-TA-assisted interpretation, we demonstrated the composition of Functional scenarios conceptualized by the PEGASUS project for describing a high-level plan to test AVs’ capability of evading critical situations. The objectivity and trustworthiness of the Functional scenario can be improved through our approach compared to the manual scenario composition by humans. Using actual accident data, we can derive more plausible scenarios unforeseen by the simulation-based methods. However, from the 2D video frames, we could not derive a Logical and Concrete scenario specified with the values of situational variables defined as 6-Layer information by the PEGASUS project such as TTC, vehicle speed, directions, and distance between objects. The situational variables can be initialized by analyzing depth information captured by sensors such as LiDAR. We laid out a framework for automating the scenario authoring process by incorporating ViT-TA modeling LiDAR data, object recognition algorithms, and generative-adversarial neural networks for synthesizing text description of safety function requirements in a Logical and Concrete scenario. Our study is a significant step toward generating highly reliable safety improvement plans for AVs.