An Integrated Automated Driving Risk Indicator in Urban Mixed Traffic Environments

Featured Application

This study can monitor the safety of mixed traffic flow situations and can identify hazardous urban facilities and sections requiring improvement.

Abstract

In this study, a novel methodology is proposed to evaluate automated driving safety in mixed traffic environments, including autonomous vehicles (AVs) and manually driven vehicles (MVs). An open-source AV dataset obtained from a real-world autonomous mobility testbed in Korea was used for methodology development and evaluations. The driving behavior was evaluated using well-known promising indicators, including the standard deviation of the vehicle speed, acceleration noise, standard deviation of the lane offset, time to collision (TTC), and deceleration to avoid a crash (DRAC). Min-max and max-min normalization was performed to unify the units of the evaluation indicators. The importance of each driving safety indicator was derived through the Analytical Hierarchy Process (AHP) performed by traffic experts, and the weights were estimated based on the average of the collected importance. The normalized indicators were integrated to obtain the automated driving risk score (ADRS), which is regarded as a measure of automated driving safety. The automated driving safety degraded considerably in road sections where right turns were made at intersections and that had a bus stop. Hazardous driving events of AVs were visualized, which is useful for monitoring mixed traffic safety and developing effective countermeasures for proactive road safety management.

1. Introduction

Identifying where, when, and how often hazardous driving situations occur on a road is essential for effectively preventing traffic crashes. There is considerable interest in analyzing mixed traffic consisting of autonomous vehicles (AVs) and manually driven vehicles (MVs) in terms of traffic safety because AVs and MVs have different driving behaviors. The level of autonomy and market penetration of AVs are important factors in determining the performance of mixed traffic streams. Guériau & Dusparic suggested that the introduction of low-level AVs will increase the number of traffic conflicts [1]. Yao et al. reported that AV penetration below 50% in a connected AV environment increased the risk of rear-end collisions and resulted in a low time to collision (TTC) [2]. The safety-prioritizing conservative behavior of AVs operated by Google’s Waymo is threatened by adjacent MVs [3]. Mahdinia et al. confirmed that the introduction of AVs into traffic flow changes the driving behavior of MVs [4]. AVs are also designed to mimic human drivers for safe implementation [5]. Consequently, identifying the hazardous driving scenarios for AVs resulting from the interaction between AVs and MVs is fundamental for developing useful countermeasures to prevent traffic crashes in mixed traffic in the era of automated driving systems. Thus, we propose a methodology for determining the level of automated driving safety.

2. Research Background

To evaluate the safety of driving on urban roads under mixed traffic conditions, various methodologies for evaluating safety using driving trajectory data have been investigated, and suitable evaluation indicators have been selected for urban areas. Kamrani et al. proposed various safety evaluation indicators for driving behavior that can be used to analyze collected data to determine driving variability and accident frequency at intersections. The standard deviation of speed is one such metric [6]. This metric is one of the simple statistics used to express data variability. Zheng et al. found that the standard deviation of speed is related to traffic oscillations, where the accident rate increases with the standard deviation of speed [7]. The standard deviation of speed has been found to be correlated with safety in several previous studies [8,9]. Jeong & Oh proposed a methodology for evaluating the effect of introducing active vehicle safety systems (AVSSs) into traffic using the acceleration noise as an index [10]. Ko et al. also used the speed and acceleration noise as evaluation indicators for traffic flow [11]. The acceleration noise is the standard deviation of acceleration and reflects the degree of dispersion of the change in speed. The larger the acceleration noise is, the lower the driving safety is [12]. The acceleration noise has been adopted in many studies as an alternative indicator to evaluate safety. In this study, we used conflict-related indicators to analyze the safety of traffic on urban roads.

Liu et al. used the distance of a vehicle from the center of a lane as an evaluation indicator of the lateral safety [13]. The lane position was defined as the center-to-center distance between the lane and the vehicle, and the standard deviation of the lane position was used to evaluate the lateral safety. Jung et al. also evaluated lateral safety on the basis of the standard deviation of the lane position [14]. A driving simulation was used to evaluate the lateral driving safety of elderly drivers involved in high-speed driving at 140 km/h or higher. The data collected in the driving simulation was used to calculate the standard deviation of the lane position using the center-to-center distance between the vehicle and the lane. Surrogate safety measures (SSMs) are often used to analyze conflict-related safety. One of the most common SSMs is the time to collision (TTC). The TTC is defined as the time in which a collision will occur if the current speed is maintained [15,16]. The TTC is utilized as an indicator of collision occurrence and is often used to comparatively analyze the safety of road environments and drivers [17]. Another widely utilized metric is the deceleration rate to avoid a crash (DRAC), which is calculated on the basis of the times to approach the vehicle in front when the investigated vehicle is driving at different speeds [18]. Unlike the TTC and other metrics, the DRAC has the advantage of reflecting driving behavior that can avoid accidents [19]. Among SSM-based driving safety evaluation metrics, there is the Post-Encroachment Time (PET), which represents the time when a leading vehicle passes a specific point until the following vehicle passes it. PET is an essential SSM that can analyze driving safety in urban areas, particularly at intersections. However, since the collected data does not provide the coordinate information of the leading vehicle or all adjacent vehicles, it was excluded from this study.

In previous studies, longitudinal and lateral maneuvering characteristics as well as conflicts have been used as evaluation indicators to analyze driving safety, and these evaluation indicators have been found to be correlated with crashes. In the present study, the three aforementioned indicators were integrated into an evaluation index for analyzing driving behavior. The driving safety of roads was evaluated using driving trajectory data collected for AVs (rather than crash records) and by analyzing simulations of AVs. The driving behavior data of AVs driving alongside real MVs can be used to perform a continuous and detailed analysis in 0.1 s increments and provide more realistic analysis results than simulations do.

3. Methodology

In this study, data was collected on AVs driving on real roads and safety evaluation indicators were derived for analyzing safety in real mixed traffic situations. In Step 1, the collected AV data was preprocessed to generate data related to individual vehicle driving behavior and conflicts. The data was combined with location data to analyze the driving behavior of AVs according to the road facilities they pass through. In Step 2, indicators for evaluating driving safety in urban areas were derived from the preprocessed AV driving behavior data by road facility using the evaluation indicators utilized in previous studies. Five evaluation indicators were derived to evaluate longitudinal, lateral, and inter-vehicle safety. In Step 3, each road facility was evaluated on the basis of an integrated risk score for driving safety indicators. The indicators were normalized to enable analysis using the same units. The automated driving risk score (ADRS) was derived by summing the normalized indicators based on the differing levels of importance for each indicator. In Step 4, five categories of road risk were established. Figure 1 shows the analysis procedure of the methodology developed in this study for evaluating the driving safety of urban road facilities using data on AVs driving in urban areas.

3.1. Data Collection and Preprocessing

In this study, an analysis was performed on autonomous driving data collected by the Sejong Autonomous Driving Control Center Open Lab. The data provided by the Open Lab were driving records from 16:20~17:20 and 20:20~21:50 for 8 days from 1 to 3 May and 8 to 12 May in 2022. The AVs circulated clockwise around the roads near Sejong City Hall over a total length of 8.6 km with 2 to 4 lanes in each direction (3 lanes for bus stops and 4 lanes for intersections). The data were collected for vehicles passing through the autonomous driving pilot area five times a day, with an average travel time of 30 min. The autonomous driving pilot ran through signalized intersections, roundabouts, signalized and unsignalized crosswalks, bus stops, and school zones. The AVs were manually driven in the school zone. This study utilized only AV driving data for analysis, and driving data when passing through school zones was excluded from the analysis. Figure 2 shows the actual AV driving data visualized on a map.

The AVs used in the analysis utilize LiDAR, Radar sensors and cameras installed on the front, rear, sides, and top of the vehicle to recognize on the road. In addition, the Electronic Control Unit (ECU) installed in the AV manages the body and autonomous driving. It collects vehicle location and driving information through video recorder, Digital tachograph, and GPS. For object recognition and data collection, cameras were installed at the front, forward left and right, and rear left and right of the AV. The front-facing cameras have a field of view (FOV) of 60 degrees, while the rear left and right cameras have a FOV of 60 degrees, the forward left and right cameras have a FOV of 110 degrees. The details and photos of the analysis vehicle are taken from the publicly available data of Sejong Autonomous Driving Control Center and are shown in Figure 3.

Two types of data were used: C-ITS standard collection data and the Sejong Techno Park (TP) dataset. The C-ITS standard data were collected for AVs by the control center and include probe vehicle data (PVD) consisting of driving vehicle information, such as location information and brake operation; TIM, which provides traffic information, including weather; road side alert (RSA), which provides road hazard warnings; and SPaT data, which are signal presentation data collected from signal controllers. The Sejong TP data includes vehicle maneuvering and location information of AVs and sensor recognition information. In this study, only Sejong TP data on sensor perception were analyzed to assess the driving safety of vehicles.

The components of the Sejong TP data are shown in

Table 1. Sejong TP dataset.

List		Field Name	Description	Unit
Vehicle data		timestamp	Timestamp	Epoch time in milliseconds, UTC
		AutoDrivingStatus	Autonomous driving Mode	0 = off, 1 = on
		wheelSpeed	Wheel speed	Average of 4 (FL to RR) wheel speeds, m/s
		yawRate	Tilt of the vehicle	Deg/sec
		turnSignal	Turn signals	0: OFF, 1: RIGHT, 2: LEFT
		emergencyLight	Emergency lights	0: Off, 1: ON
		wiperStatus	Wiper operation status	0: Off, 1: ON
		mdpsAngle	Angle of motor-driven power steering	Deg (clockwise: -)
		transmission	Gear status	P: 0, R: 1, N: 2, D: 3
		brakeStatus	Brake operation status	0: unavailable, 1: Off, 2: ON
Vehicle location data		timestamp	Timestamp	Epoch time in milliseconds, UTC
		latitude	Latitude	Degree
		longitude	Longitude	Degree
		speed	Speed	m/s
		heading	Vehicle orientation	Degree
Cognition	Obstacle	timestamp	timestamp	Epoch time in milliseconds, UTC
		obstacleId	Object ID	-
		obstacleHeading	Orientation of objects	Degree (relative angle to the vehicle)
		obstacleType	Type of objects	Dummy data (0)
		obstacleWidth	Width of the object	m
		obstacleLngSpeed	Longitudinal relative speed	m/s
		obstacleLateralSpeed	Lateral relative speed	m/s
		obstacleLngAbsSpeed	Longitudinal absolute speed	m/s
		obstacleLateralAbsSpeed	Lateral absolute velocity	m/s
		obstacleDistance	Distance from the object	m
	Lane	timestamp	Timestamp	Epoch time in milliseconds, UTC
		laneId	ID of the lane	-
		laneNumber	Lane number	Driving lane, where the lane number increases to the right
		laneCurvatureLeft	Curvature with lane—left	Degree
		laneCurvatureRight	Curvature with lane—right	Degree
		laneDistanceLeft	Distance from lane—left	cm
		laneDistanceRight	Distance from lane—right	cm
		laneSlopeLeft	Slope with lane—left	Degree
		laneSlopeRight	Slope with lane—right	Degree

. The perceptual information in the Sejong TP dataset includes vehicle data, location information, and object and lane information. In this study, a safety analysis was performed on each item using location data (such as the timestamp, latitude and longitude, and speed) and cognitive data (such as the object orientation, longitudinal absolute velocity, distance from the object, and left–right distance from the lane). This study utilized only data from vehicles operating in autonomous driving mode for analysis. The determination of whether a vehicle was in autonomous or manual driving mode was based on the “AutoDrivingStatus” field. This means that only data where the “AutoDrivingStatus” field was set to 1 was used for analysis.

The data provided by the Open Lab included items that were not collected. Different companies operate AVs in the autonomous driving pilot section, and Open Lab manages the items that can be collected for data management and analysis. Owing to the different driving algorithms and bylaws of AV companies, some data is not provided even though Open Lab requests collection of these data. In the collected data, the “type of object” field of the perception-object data was provided as dummy data (an uncollectable item). Among the collected data, vehicle recognition and lane recognition were performed using cameras. Video information captured by the cameras was interpreted through object recognition to distinguish whether the recognized object was a vehicle or a lane. Data for the “orientation of object” field was collected for all objects with orientations between −45° and 45°. Up to 64 different objects could be recognized from the collected data. In this study, we analyzed data close to the car in front of the investigated car as the data of the car in front. Therefore, we set the value of the distance to the object as “distance to the car in front” when the direction of the object is close to 0°. We created a new data field for the lane offset to process the collected data. The lane offset was taken as the distance of the investigated vehicle from the center of the lane and was calculated as the distance from the left lane minus the distance from the right lane. A positive value of the lane offset indicated a rightward skew relative to the lane center.

We used the geolocation of the collected data to determine the roadway facilities through which the AV passed. An AV was defined as passing through a road facility when it entered the area of influence of the road facility. The influence area of each road facility was based on the stopping distance suggested by the road plan and the geometry of the Road Design Manual. All the intersections and crosswalks were set to have a total influence area of 80 m, including 30 m of the stopping distance required for traveling at 50 km/h and 50 m of the distance traveled over 3 s to ensure a sight distance after the stop line. The area of influence for bus stops was defined as the area where buses enter and exit to arrive at the stop. The data and preprocessing process used in the analysis are shown in Figure 4.

3.2. Driving Safety Indicators for AVs

This study developed a new indicator capable of analyzing driving safety from multiple perspectives, including longitudinal and lateral aspects as well as vehicle-to-vehicle interactions. In this study, to evaluate intervehicle interactions, longitudinal and lateral safety indicators were derived on the basis of the results of other studies. The standard deviation of the speed and acceleration (acceleration noise, AN) was selected as the longitudinal indicator. This indicator compares the magnitude of changes in the speed and acceleration, where the larger the difference is, the lower the safety is because of inconsistent driving through each section of the city center. The standard deviation of the speed is a useful metric for analyzing urban networks with large variations in speed [20]. The acceleration noise is a useful metric for analyzing frequent changes in acceleration and deceleration caused by traffic conflicts because networks include many signalized intersections [21,22]. The standard deviation of the speed and acceleration noise are defined in Equations (1) and (2), respectively.

$$\mathrm{S}\mathrm{p}\mathrm{d}=\sqrt{{\displaystyle \frac{1}{\mathrm{n}-1}}{\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left(\mathrm{v}-\overline{\mathrm{v}}\right)}^2}$$

(1)

where $\mathrm{S}\mathrm{p}\mathrm{d}$: Standard deviation of the speed $(\mathrm{k}\mathrm{m}/\mathrm{h})$

$\mathrm{n}$: Number of data points in the analyzed section

$\mathrm{t}$: Unit of time (s)

$\mathrm{v}$: Driving speed $(\mathrm{k}\mathrm{m}/\mathrm{h})$

$\overline{\mathrm{v}}$: Average speed $(\mathrm{k}\mathrm{m}/\mathrm{h})$

$$\mathrm{A}\mathrm{N}=\sqrt{{\displaystyle \frac{1}{\mathrm{n}-1}}{\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left(\left|\mathrm{a}\right|-\overline{\left|\mathrm{a}\right|}\right)}^2},$$

(2)

where $\mathrm{A}\mathrm{N}$: Acceleration noise ($\mathrm{m}/{\mathrm{s}}^2$)

$\mathrm{a}$: Acceleration ($\mathrm{m}/{\mathrm{s}}^2$)

$\overline{\mathrm{a}}$: Average acceleration ($\mathrm{m}/{\mathrm{s}}^2$)

The standard deviation of the lateral deviation was selected as a lateral evaluation indicator in this study. This indicator compares the change in lateral steering maneuvers, where the larger the value of the indicator is, the worse safety is. In previous studies, lateral steering maneuvers for changing lanes have been shown to lead to poor driving safety [23]. The standard deviation of the lane offset is defined in Equation (3).

$$\mathrm{L}\mathrm{D}=\sqrt{{\displaystyle \frac{1}{\mathrm{n}-1}}{\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left(\mathrm{d}-\overline{\mathrm{d}}\right)}^2},$$

(3)

where $\mathrm{L}\mathrm{D}$: Standard deviation of the lateral deviation ($\mathrm{m}/{\mathrm{s}}^2$)

$\mathrm{o}$: Lateral deviation ($\mathrm{m}/{\mathrm{s}}^2$)

$\overline{\mathrm{o}}$: Average lateral deviation ($\mathrm{m}/{\mathrm{s}}^2$)

TTC and DRAC were defined as indicators of the safety of vehicle interactions in this study. The TTC is the time (s) in which an accident will occur if the leading and trailing vehicles maintain their current speeds and paths. The TTC is typically used for analyzing traffic conflicts [17,24]. A TTC value lower than a set threshold can indicate that a conflict has occurred. In this study, we set the threshold to 1.5 s, which is often used in urban center analysis, and define that a conflict occurs if the TTC is below or equal to 1.5 s [25]. The DRAC is the deceleration rate required for the trailing vehicle to avoid a collision and is a useful metric for analyzing crashes at unsignalized intersections [26]. The American Association of State Highway and Transportation Officials (AASHTO) has defined the threshold DRAC as 3.4 $\mathrm{m}/{\mathrm{s}}^2$ and that a conflict occurs when the DRAC is exceeded [27,28]. The TTC and DRAC are defined in Equations (4) and (5), respectively.

$$\mathrm{T}\mathrm{T}\mathrm{C}={\displaystyle \frac{\mathrm{d}}{{\mathrm{v}}_{\mathrm{f}}-{\mathrm{v}}_{\mathrm{l}}}},$$

(4)

where $\mathrm{T}\mathrm{T}\mathrm{C}$: Time to collision (s)

${\mathrm{v}}_{\mathrm{f}}$: Speed of the following vehicle (m/s)

${\mathrm{v}}_{\mathrm{l}}$: Speed of the leading vehicle (m/s)

$\mathrm{d}$: Distance between the leading and following vehicles (m)

$$\mathrm{D}\mathrm{R}\mathrm{A}\mathrm{C}={\displaystyle \frac{{{(\mathrm{v}}_{\mathrm{f}}-{\mathrm{v}}_{\mathrm{l}})}^2}{\mathrm{d}}},$$

(5)

The evaluation indicator for the occurrence of a conflict was selected as the average of the TTC and DRAC values aggregated in 0.1 s increments for each urban road facility. In this study, driving safety was evaluated on the basis of AV data collected for 8 days of driving. The average of the evaluation indicators (the standard deviation of the speed and acceleration, standard deviation of the lane offset, and average of the TTC and DRAC) was used to evaluate each urban road facility. Using an average evaluation indicator reduces the error incurred daily.

3.3. Normalization of the Safety Evaluation Indicator and Derivation of ADRS Using AHP Methodology

Considering that each evaluation indicator has a different value range, the indicator values were converted into raw units to directly compare the indicator. This study normalized the five evaluation indicators collected in this study to have the same units.

The driving safety becomes worse as the value of standard deviation of speed, acceleration noise, lateral deviation, and DRAC increases. On the other hand, the decrease in TTC makes the driving safety worse. Therefore, min-max normalization was performed for standard deviation of speed, acceleration noise, lateral deviation, and DRAC. Also, max-min normalization was performed for TTC to normalize the risk score to increase as the value of all evaluation indicators increases. The equations for min-max normalization are defined in Equation (6) and max-min normalization in Equation (7).

$${\mathrm{X}}^{\mathrm{N}}={\displaystyle \frac{\mathrm{x}-{\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}{{\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}},$$

(6)

$${\mathrm{X}}^{\mathrm{N}}={\displaystyle \frac{{\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\mathrm{x}}{{\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}},$$

(7)

where ${\mathrm{X}}^{\mathrm{N}}$: Normalized evaluation indicator

$\mathrm{x}$: Evaluation indicator

${\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$: Maximum value of the evaluation indicator

${\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$: Minimum value of the evaluation indicator

The summed metric was defined as the automated driving risk score (ADRS), which is defined in Equation (8). ADRS is derived by multiplying the weight set for each driving safety indicator according to its importance by the normalized indicator and summing them.

$$\mathrm{A}\mathrm{D}\mathrm{R}\mathrm{S}=\sum {\mathsf{\omega }}_{\mathrm{i}}{\mathrm{X}}_{\mathrm{i}}^{\mathrm{N}}={\mathsf{\omega }}_1\mathrm{S}\mathrm{p}{\mathrm{d}}^{\mathrm{N}}+{\mathsf{\omega }}_2\mathrm{A}{\mathrm{N}}^{\mathrm{N}}+{\mathsf{\omega }}_3\mathrm{L}{\mathrm{D}}^{\mathrm{N}}+{\mathsf{\omega }}_4\mathrm{T}\mathrm{T}{\mathrm{C}}^{\mathrm{N}}+{\mathsf{\omega }}_5\mathrm{D}\mathrm{R}\mathrm{A}{\mathrm{C}}^{\mathrm{N}},$$

(8)

where ${\mathsf{\omega }}_{\mathrm{i}}$: Weight of the evaluation indicator

The importance of each driving safety indicator was derived using the Analytic Hierarchy Process (AHP), which is commonly used in multi-criteria decision-making methodologies. The AHP can be weighted by averaging the magnitudes of the importance of each indicator collected by experts. Since there is no absolute standard for judging driving safety in the collected AV driving data, this study performed AHP to determine the objective importance of each evaluation metric. The importance derived from the pairwise comparison of each indicator is organized into a matrix, and when the eigenvalue λ of the matrix is the maximum, the eigenvector becomes the weight for each indicator. To determine whether the weights collected by each expert are consistent, the Consistency index (CI) is used to determine the consistency. The formula for CI is defined in Equation (9).

$$\mathrm{C}\mathrm{I}=(\mathsf{\lambda }-\mathrm{n})/(\mathrm{n}-1),$$

(9)

where $n$ is the number of factors, which are fixed to $n=5$ in this study as five indicators are utilized in the analysis. Consistency can be judged based on the ratio of the derived CI to the defined Random Index (RI) by the Consistency Ratio (CR) [29]. If the CR is less than 0.1, the CI is considered consistent. The equation for CR is defined in Equation (10), and the RI defined by the number of factors is presented in

Table 2. Random index [29].

Dimension	2	3	4	5	6	7
RI	0.00	0.58	0.90	1.12	1.24	1.32

. The number of factors is 5 according to the number of endpoints, so the RI of this study was set to 1.12.

$$\mathrm{C}\mathrm{R}=\mathrm{C}\mathrm{I}/\mathrm{R}\mathrm{I},$$

(10)

The AHP survey of experts was designed to prioritize which of the five driving safety indicators used in this study best represents driving safety and to aggregate the results of driving safety pairwise comparisons by indicator. This study aggregated the AHP survey results of 14 traffic safety experts. All transportation experts collected in this study have conducted research in the field of traffic safety for over five years and were recruited from researchers working at transportation research institutes such as the Korea Institute of Civil Engineering and Building Technology, the Korea Railroad Research Institute and the Korea Transport Institute. CR-based analysis showed that there was consistency in the importance of ratings collected from all experts. The average of the importance ratings per metric collected from each expert was set as the metric weight for this study. The weighted responses of individual experts are presented in

Table 3. AHP results—Weight of Evaluation Index and CRs by Traffic Safety Expert.

Experts No.	1	2	3	4	5	6	7	8	9	10	11	12	13	14	AHP
Std. of speed	0.43	0.54	0.50	0.27	0.15	0.03	0.07	0.07	0.12	0.48	0.08	0.15	0.27	0.21	0.24
Acceleration noise	0.04	0.19	0.26	0.06	0.51	0.25	0.28	0.26	0.07	0.28	0.12	0.54	0.04	0.07	0.21
Lateral deviation	0.26	0.15	0.13	0.17	0.03	0.14	0.04	0.50	0.06	0.14	0.04	0.07	0.14	0.03	0.14
TTC	0.14	0.06	0.07	0.03	0.07	0.51	0.50	0.03	0.50	0.03	0.50	0.03	0.47	0.54	0.25
DRAC	0.12	0.06	0.03	0.46	0.24	0.07	0.11	0.13	0.26	0.07	0.26	0.21	0.08	0.14	0.16
CR	0.05	0.08	0.05	0.07	0.06	0.10	0.08	0.05	0.09	0.10	0.09	0.09	0.07	0.08	-

, and the average of the weights provided by all experts is highlighted in bold. The AHP methodology was used to derive the weights of the evaluation indicators. It was found that TTC was the most important indicator in determining driving safety, followed by standard deviation of speed, acceleration noise, DRAC, and lateral deviation in decreasing order of importance.

Equation (11) shows the formula for the ADRS with the weights derived from the AHP. As shown in the derived AHP, ${\omega }_1$ through ${\omega }_5$ are defined as 0.24, 0.21, 0.14, 0.25, and 0.16, respectively.

$$\begin{array}{c}\mathrm{A}\mathrm{D}\mathrm{R}\mathrm{S}=\sum {\mathsf{\omega }}_{\mathrm{i}}{\mathrm{X}}_{\mathrm{i}}^{\mathrm{N}}\\ =0.24\times {\mathrm{S}\mathrm{p}\mathrm{d}}^{\mathrm{N}}+0.21\times {\mathrm{A}\mathrm{N}}^{\mathrm{N}}+0.14\times {\mathrm{L}\mathrm{D}}^{\mathrm{N}}+0.25\times {\mathrm{T}\mathrm{T}\mathrm{C}}^{\mathrm{N}}+0.16\times {\mathrm{D}\mathrm{R}\mathrm{A}\mathrm{C}}^{\mathrm{N}},\end{array}$$

(11)

4. Results

4.1. Analysis of AV Maneuvering for Driving Safety Evaluation

The average, standard deviation, maximum, and minimum values of the daily evaluation indicator values derived in the analysis are summarized for each urban road facility. The average and standard deviation listed in Table 2, Table 3,

Table 4. Analysis of Std. of Speed (km/h).

Section	Average	Standard Deviation	Maximum	Minimum
Signalized_straight	21.97	15.05	48.67	0.00
Signalized_crosswalk	34.54	12.44	49.98	0.00
Bus stop	37.35	12.42	49.99	0.00
Signalized_right turn	17.83	8.99	43.18	0.06
Unsignalized_crosswalk	15.88	8.58	34.83	0.00
Traffic island_right turn	19.73	6.04	29.67	8.98
Roundabout	19.96	3.74	34.69	0.17
Total network	30.72	12.81	50.27	0.00

,

Table 5. Analysis of Acceleration Noise ($\mathrm{m}/{\mathrm{s}}^2$).

Section	Average	Standard Deviation	Maximum	Minimum
Signalized_right turn	0.02	3.89	7.63	−6.81
Traffic island_right turn	−0.26	3.23	8.03	−8.51
Unsignalized_crosswalk	−0.25	2.90	8.74	−3.88
Roundabout	−0.19	2.74	7.21	−6.72
Signalized_crosswalk	−0.35	2.18	5.53	−8.71
Bus stop	−0.06	2.08	7.84	−7.02
Signalized_straight	0.29	1.92	7.63	−7.28
Total network	0.01	1.42	8.94	−9.63

and

Table 6. Analysis of The Lane Offset (cm).

Section	Average	Standard Deviation	Maximum	Minimum
Traffic island_right turn	0.24	1.80	6.55	−4.39
Signalized_right turn	0.33	1.32	7.98	−3.89
Roundabout	−0.03	0.75	2.04	−7.25
Signalized_straight	0.47	0.56	6.18	−1.20
Unsignalized_crosswalk	0.09	0.33	0.98	−1.25
Signalized_crosswalk	0.06	0.32	0.87	−1.88
Bus stop	0.01	0.26	1.17	−2.05
Total network	0.11	0.78	7.98	−8.68

are the means of the average and standard deviation of the daily speed, acceleration, lane offset, TTC, and DRAC values. The maximum and minimum values of the aggregated total speed, acceleration, lane offset, TTC, and DRAC are shown. The statistical results are analyzed to identify the driving behavior of AVs in mixed traffic and areas where the driving safety is reduced by conflicts.

Table 4 presents descriptive statistics of the speed used to calculate the driving safety evaluation metrics. The basic speed statistics show that the average speed of the entire section is 30.72 km/h, and the standard deviation is 12.81 km/h. As a result of AVs complying with the 50 km/h speed limit in the city center, the highest speed is 50.27 km/h. The lowest speed is 0 km/h because AVs stop at red lights. The largest average standard deviation of the speed is found for the road section going straight through the intersection, i.e., the section with the lowest longitudinal safety. This section has a higher standard deviation (15.05 km/h) compared with the other sections because drivers stop at the signal and accelerate without steering. The highest average speed (37.35 km/h) is found for the road section with a bus stop. This result is obtained because drivers passing through a section with a bus stop do not need to slow down if no other vehicles are entering or exiting the stop. Therefore, the average speed for this section is higher than that of other road facilities at which drivers are required to slow down. The lowest speed is 8.98 km/h, and the average speed is 19.73 km/h where right turns are made passing through a section with a traffic island. This result indicates that right-turning traffic passes through the section with a traffic island without stopping, unlike traffic that stops before passing through signalized intersections.

The basic acceleration statistics are presented in Table 5. The section with the lowest driving safety was where right turns are made to pass through the signalized intersection, for which the standard deviation of the acceleration was 3.89 $\mathrm{m}/{\mathrm{s}}^2$. The section with the next lowest driving safety was where right turns are made passing the traffic island, with 3.23 $\mathrm{m}/{\mathrm{s}}^2$. To make right turns when passing through an intersection, drivers must decelerate, accelerate and steer according to the signal. The driver must make evasive maneuvers to avoid crashes with other vehicles passing through the intersection. Therefore, the driving safety of this section was found to be poor because of the large acceleration caused by conflicts with other vehicles.

Descriptive statistics for the lane offset are presented in Table 6. The lowest lateral driving safety was found for the section in which right turns are made passing a traffic island, where the standard deviation of the lane offset was 1.80 cm. The sections with the next lowest lateral driving safety were those in which right turns are made passing through signalized and rotary intersections, with standard deviations of the lane offset of 1.32 and 0.75 cm, respectively. All three sections require steering maneuvers to pass through, and the section with the traffic island requires that the curb be avoided while driving.

Table 7. Analysis of TTC (s).

Section	Average	Standard Deviation	Maximum	Minimum
Traffic island_right turn	2.94	1.11	5.00	0.68
Bus stop	2.97	0.96	4.99	0.72
Signalized_crosswalk	3.13	0.96	5.00	1.01
Roundabout	3.17	1.02	5.00	0.82
Signalized_right turn	3.20	1.09	5.00	0.65
Unsignalized_crosswalk	3.23	1.08	4.99	0.85
Signalized_straight	3.37	0.94	4.99	0.68
Total network	3.09	1.04	5.00	0.63

summarizes the basic statistical analysis for the TTC. The TTC is the time to crash if the same speed is maintained in the same lane, where the smaller the TTC is, the lower the driving safety is. At specific points in time during data collection for this study, if the leading vehicle was too far ahead, the aggregated TTC or DRAC values were measured as excessively large or small, indicating that driving safety was not at risk. In the analysis, TTC values exceeding 5 s were removed as not being meaningful. The section with the lowest average TTC (2.94 s) was found for the section where right turns are made passing the traffic island, indicating the lowest driving safety. Driving through traffic islands creates conflicts with other vehicles traveling through the intersection and results in a low TTC because the vehicle does not slow down to accommodate oncoming right-turning vehicles. The section with the bus stop had the second lowest driving safety, with an average TTC of 2.97 s. The low driving safety of this section resulted from conflicts of vehicles with buses entering and exiting the bus stop.

Table 8. Analysis of DRAC ($\mathrm{m}/{\mathrm{s}}^2$).

Section	Average	Standard Deviation	Maximum	Minimum
Bus stop	4.00	1.76	7.99	0.00
Signalized_crosswalk	3.63	1.91	7.99	0.00
Traffic island_right turn	3.23	1.73	7.98	0.00
Roundabout	2.82	1.60	8.00	0.00
Signalized_right turn	2.78	1.61	7.97	0.00
Signalized_straight	2.77	1.65	7.96	0.00
Unsignalized_crosswalk	1.92	1.42	7.71	0.00
Total network	3.41	1.91	8.00	0.00

summarizes the descriptive statistical analysis of the derived DRAC by section. The DRAC values corresponding to accelerations of more than 8 $\mathrm{m}/{\mathrm{s}}^2$ and decelerations of less than 0 $\mathrm{m}/{\mathrm{s}}^2$ were removed as outliers. The analysis showed that the section in which vehicles passed through the bus stop had the largest average DRAC of 4.00 $\mathrm{m}/{\mathrm{s}}^2$. This result was obtained because of conflicts between vehicles and buses entering and exiting the bus stop, similar to the results for the TTC.

Figure 5 shows the visualization results for the TTC and DRAC, which are conflicting indicators. The data aggregated at 0.1 s intervals is visualized in Figure 5a for points where the TTC is 1.5 s or lower [25] and in Figure 5b for points where the DRAC is 3.4 $\mathrm{m}/{\mathrm{s}}^2$ or greater [27,28]. The TTC visualization results show that many conflicts occur when right turns are made at signalized intersection with and without traffic islands, and the DRAC visualization results show that conflicts occur at bus stops and roundabouts.

The TTC-based visualization results (Figure 6) showed that many conflicts occurred while making right turns at intersections. For right turns with traffic islands, conflicts occurred at the point where lane changes were made and at the crosswalk of the traffic island. For the section with the signalized intersection at which right turns were made, conflicts occurred at the point at which lane changes were made.

4.2. Level of Automated Driving Safety

The classification criteria for the ADS and the ADRS results are summarized in

Table 9. Criteria for Levels of Automated Driving Safety (ADS) and ADRS Results.

Description	Criteria for ADRS	ADS Level	Section	ω1SpdN	ω2ANN	ω3LON	ω4TTCN	ω5DRACN	ADRS
Very good	ADRS < 0.321	A	Unsignalized crosswalk	0.103	0.105	0.006	0.081	0.000	0.296
			Roundabout	0.000	0.088	0.043	0.116	0.069	0.317
Good	0.321 ≤ ADRS < 0.476	B	Signalized straight	0.242	0.000	0.027	0.000	0.066	0.334
Moderate	0.476 ≤ ADRS < 0.535	C	Signalized crosswalk	0.186	0.028	0.005	0.139	0.132	0.490
Poor	0.535 ≤ ADRS < 0.648	D	Signalized right turn	0.112	0.212	0.094	0.099	0.066	0.583
			Bus stop	0.186	0.017	0.000	0.232	0.161	0.595
Very poor	0.648 ≤ ADRS	E	Traffic island right turn	0.049	0.141	0.137	0.249	0.101	0.677

. The ADRS results were divided into five classes with equal intervals of 20 percentiles. These levels of automated driving safety (ADS) were labeled A, B, C, D, and E in order of decreasing ADRS values. The section classified as ADS E was the one in which right turns were made passing a traffic island and had an ADRS of 0.677. Compared with the other sections, the section in which a right turn was made at a traffic island had the lowest TTC, corresponding to the inter-vehicle safety. The section classified as ADS D was the bus stop and the right turn at a signalized intersection. The section in which vehicles passed through the bus stop had a high TTC and DRAC value and standard deviation of the speed. This section was found to have poor driving safety because of conflicts with buses entering and exiting the stop rather than because of steering maneuvers of individual vehicles. The right turn at the intersection had the highest acceleration noise value of 0.212 due to the acceleration and deceleration caused by the signal. Making a right turn at signalized intersections was associated with poor longitudinal and transverse driving safety of individual vehicles rather than intervehicle safety.

The ADS developed in this study can be used to evaluate and visualize the safety of urban road facilities in mixed traffic with AVs. Figure 7 shows a map of urban road facilities based on the ADS level. Visualization makes it possible to view sections and areas with poor driving safety and plan areas that could be installed with road-safety facilities. In future studies, real-time data collected from AVs could be used to evaluate and visualize ADS levels to reduce the number of crashes and conflicts by employing variable speed limits and signal control.

5. Conclusions

This study analyzed AV data collected on real roads to evaluate the driving safety of individual urban road facilities under mixed traffic conditions. Sejong TP data, which includes vehicle control and positioning information and data on the distance and speed of the leading vehicle, were selected from the analyzed data. Among the evaluation indicators that can be derived from existing studies and AV data, we selected five to analyze driving safety in urban areas. The selected metrics can be used to evaluate the longitudinal and lateral driving safety of individual vehicles and the driving safety for interactions between vehicles. In this study, the standard deviations of the speed and the lane offset, acceleration noise, and the average of the TTC and DRAC were selected as evaluation indicators of safety. These indicators were derived from data of AVs driven for one day. The driving safety of each road facility was evaluated based on the average value of the metrics determined for a total of eight days. The evaluation indicators were converted to the same units by min–max normalization and then integrated by summation, the ADRS, to evaluate the driving safety of each road facility. A safety rating was set by establishing five classes of ADRS values, ranging from A to E, with equidistant 20% intervals.

The analysis showed that road sections classified as ADS E were those at which right turns were made passing through traffic islands. Because when passing through a traffic island, the AV must navigate while avoiding collisions with the traffic island and other vehicles. In the event of a conflict, drivers slowed down to pass the island without stopping, resulting in poor traffic safety. However, low driving safety of individual vehicles was found for sections where right turns were made passing through a signalized intersection because vehicles accelerate after slowing down and stopping when passing through the intersection. Visualization of conflict points at sections where right turns were made passing traffic islands revealed that conflicts occurred at the stop line in front of the crosswalk of the traffic island, based on both TTC and DRAC metrics. Pavement markings and signs could be installed to induce vehicles to pause before the intersection as a countermeasure for reducing the number of conflicts.

The limitations of this study include that the collected real-world AV data do not cover all urban roadway facilities. In future studies, the evaluation methodology developed in this study could be used to analyze data for AVs passing through more urban facilities, such as three-way intersections, intersections at which left turns are made, tunnels and underpasses, and overpasses. In this study, all the metrics that can be used to evaluate driving safety were not used, and only metrics that could be calculated from the data and are suitable for urban analysis were utilized. Only a single indicator of lateral safety, the standard deviation of the lane offset, was used in this study. In future studies, lateral safety could be more accurately evaluated by collecting data on lateral acceleration and steering maneuvers or by deriving lateral safety metrics from the collected data. All the ADRS metrics were normalized and summed using the same weights, although each metric reflects safety to a different extent and is based on different criteria. In future studies, an integrated score could be derived based on all available metrics. A regression model or machine learning model that can predict the frequency of conflicts and accidents in a section could be used to derive the correlation coefficient or variable importance of each evaluation indicator, and the indicators could be summed using different weights to produce a more accurate ADRS.

This study was conducted under limited conditions. It utilized driving trajectory data from AVs operating in Korea. All AVs operating in Korea are currently in pilot operation and are regulated to operate during off-peak hours to ensure safe driving. Consequently, sections deemed hazardous during peak hours may appear safe in this study’s results. This study prioritized considering the reduction in driving safety caused by road structure. Sections with high traffic volume experience frequent vehicle interactions, leading to more conflicts and congestion. Therefore, sections with higher traffic volume compared to others may appear to have the most dangerous driving safety during peak hours. This study utilized driving data from off-peak hours to reduce potential biases caused by traffic volume. Future research could analyze AVs driving during all hours to identify sections where driving safety risks change with traffic volume.

Errors may occur during the data preprocessing stage used in this study. This study set the leading vehicle as the one with an angle closest to zero relative to the collected leading vehicle information. That is, the vehicle directly in front of the AV was designated as the leading vehicle. However, on curved sections or right-turn sections, the angle between the AV and the leading vehicle may deviate significantly from zero, potentially preventing its recognition as the leading vehicle. This study estimates that such errors could occur for up to approximately 3 s when the AV passes through roundabouts and right-turn sections. Consequently, an AV driving the same route 5 times per day for 2 h and 30 min could experience a maximum error of 60 s, representing 0.6% of the total route time. This study determined that the error would not significantly impact the results’ variability. However, data reliability could be enhanced by expanding adjacent vehicle information, such as through a multi-hypothesis tracker algorithm.

There is insufficient evidence to determine whether the evaluation metrics proposed in this study are associated with actual crashes and conflicts involving AVs. This is because the proposed metrics are indicators related to vehicle conflicts and driving behavior, and they do not consider whether the driving behavior of AVs is hazardous. Future research could collect AV crash data and related driving data to analyze crash risks associated with driving behavior and conflicts. Simulating the environments in which AVs operate could also demonstrate how the proposed evaluation metrics relate to crashes.

The AV data used in this study was analyzed based on the driving records of a single vehicle and not different types of vehicles. This data selection may have led to biased results because the analysis did not reflect the characteristics of different AVs by type and operating company but only the driving behavior of a single vehicle. In future studies, data for multiple types of vehicles, such as SUVs or shuttle-type AVs, could be used to evaluate road safety in general and compare different driving behaviors by vehicle type. The data collected do not account for mixed traffic but were obtained only for single AVs traveling on city streets. Owing to the nature of the pilot area, AVs were allowed to operate only during off-peak hours to reduce the number of conflicts and accidents caused by AVs. In future studies, more accurate safety evaluation results could be analyzed by considering various scenarios, such as multiple AVs driving together and during peak traffic hours.