Leveraging UAV Capabilities for Vehicle Tracking and Collision Risk Assessment at Road Intersections

Abstract

Transportation agencies continue to pursue crash reduction. Initiatives include the design of safer facilities, promotion of safe behaviors, and assessments of collision risk as a precursor to the identification of proactive countermeasures. Collision risk assessment includes reliable prediction of vehicle trajectories. Unfortunately, in using traditional tracking equipment, such prediction can be impaired by occlusion. It has been suggested in recent literature that unmanned aerial vehicles (UAVs) can be deployed to address this issue successfully, given their wide visual field and movement flexibility. This paper presents a methodology that integrates UAVs to track the movement of road users and to assess potential collisions at intersections. The proposed methodology includes an existing deep-learning-based algorithm to identify road users, extract trajectories, and calculate collision risk. The methodology was applied using a case study, and the results show that the methodology can provide beneficial information for the purpose of measuring and analyzing the infrastructure performance. Based on vehicle movements it observes, the UAV can communicate its collision risk to each vehicle so that the vehicle can undertake proactive driving decisions. Finally, the proposed framework can serve as a valuable tool for urban road agencies to develop measures to reduce crash risks.

1. Introduction

It has been prognosticated that unmanned aerial vehicles (UAVs) will play a vital role in various application or context areas of transportation systems management. This is motivated by the success of UAVs in other sectors and domains including photography, photogrammetry, agriculture, terrain mapping, monitoring, disaster relief and rescue operations, and recreational purposes. Due to these applications, the emerging global market for drone-enabled services has been valued at over 12.7 billion USD [1,2]. Moreover, it is predicted that the industry will lead to the creation of more than 100,000 new jobs [3]. According to recent literature, seven million small UAVs have already been deployed in the airspace for commercial use in various domains including real estate, insurance, and agriculture.

In the transportation sector, engineers have investigated various ways in which UAV technology can be applied to enhance transportation operations, and drone-based solutions are being developed and tested to increase the efficiency of transportation in general and freight transportation in particular [4,5]. Recognizing the immense potential of UAV technology in transportation, the US Congress, in 2012, passed legislation that required the Federal Aviation Authority (FAA) to integrate small drones into the airspace by 2015 [6]. That legislation increased the number of research efforts in this area. Most of this research work focused on traffic flow analysis [7,8], vehicle detection [9], and highway infrastructure management [10].

Another UAV application area in transportation is the risk assessment of traffic safety [4]. The safety of public system users is considered a key indicator of the social pillar of sustainable development [11,12,13]. Public safety has been ranked in at least one previous study as the most important assessment indicator for infrastructure sustainability [11]. The safety costs of crashes are immense, as families often experience great pain and suffering when they lose a loved one through an accident. Road traffic crashes cause 1.3 million deaths annually [14] and over 20 million people suffer non-fatal injuries, with many incurring a disability as a result of their injury [14]. Traffic safety can extend beyond the social to the economic pillar of sustainable development, particularly when safety costs are converted into dollars using unit crash costs [15]. The global economic cost is immense: over 580 billion USD annually, costing most countries as much as 3% of their gross domestic product [16]. Efforts to address road traffic safety can be categorized by the factors that affect crashes: vehicle defect, driver incapacitation or inattention, road design and management, enforcement, driver education, and the environment (weather). Of these, only road management and design are under the direct control of the road agency. Road safety management includes traffic monitoring and crash risk assessment. Transportation agencies devote significant investment, and safety researchers go to great lengths to help reduce injuries and deaths associated with transportation systems by designing safer facilities, monitoring road traffic, and promoting safe driving behaviors. However, current research efforts in UAV-based assessment of traffic safety risk are rather limited. Kim and Chervonenkis studied the detection of emergency and abnormal traffic situations with a UAV artificial vision system but acknowledged the limitations of the efficacy of their algorithm [17]. Sharma et al. proposed a multi-UAV coordinated vehicular network to analyze driving behavior for improving traffic safety [18]. However, their work is only applicable in scenarios where more than two UAVs are available, which is hard to generalize at present.

In the application area of traffic safety risk assessment, UAVs have two advantages: First, UAVs are portable, flexible, and robust. Traditional video data collection by land-based cameras mounted on tall physical structures has several limitations including restrictions of the field of view posed by the height of the camera and camera tilt angle. These impair accuracy in tracking the trajectories of the road vehicles being monitored. In addition, the time-consuming and labor-intensive installation process of mounting cameras on tall buildings prohibits the timely implementation and maintenance of ground-based traffic monitoring. UAVs offer a convenient means to address these limitations as it is possible to easily and quickly dispatch them to the site of interest and to adjust their spatial locations and camera positions. For automated vehicles, onboard sensors such as cameras and Lidar suffer from the problem of limited coverage range and occlusion. The vehicle is limited not only qualitatively (in terms of the precision and the richness of the delivered information) but also quantitatively (in terms of the range of its sensors) [19]. Onboard sensors often fail to detect persons and objects blocked by trees, vehicles, building corners, and other obstacles, and have difficulty sensing road users that are not in the same lane/direction as them. Under these conditions, drivers may not easily notice pedestrians in a timely manner and avoid collisions. Sensor fusion can address this problem to a limited extent [20]. A UAV overcomes these limitations because it has a global bird-eye view that generates comprehensive telemetric data on cyclists, pedestrians, and other mobile entities in the image recording.

The second advantage of UAVs arises from the realization that it is still challenging to realize a large-scale ground-based vehicle-to-everything (V2X) network at the current time and in the near future. In a V2X network, the mobile entities represent nodes in an integrated connected network and thereby communicate directly with each other and with roadside infrastructure. The resulting information network is termed vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), or vehicle-to-pedestrian (V2P) networks. If the communication is with a data center or information technology network, then the network is a vehicle-to-network (V2N). A common term that combines all these types of communication, providing vehicle links with various recipients, is vehicle-to-everything (V2X). Cooperative V2X communications are intended to support a variety of use cases in risk detection including do-not-pass warning, forward collision warning, parking discovery, queue-ahead warning, curve speed warning, optimal speed advisory, and other contexts that enhance traffic safety and efficiency [21,22]. However, the major drawback of a V2X network is that its effectiveness hinges on the number of vehicles/facilities that are equipped with communication capabilities, because non-equipped vehicles are completely invisible to equipped vehicles. In addition, published research suggests that dedicated short-range communications or DSRC (a major data transfer technology used in V2X networks) is often plagued with issues of reliability, efficiency, and productivity, particularly at high traffic volumes [23]. Moreover, security issues including dynamic network typology, attack prevention, and network scalability may impair the efficacy of complex V2X networks [24]. Further, there is the issue of interoperability: the multiplicity of principal stakeholders—automotive manufacturers, public transport providers, municipalities, and transport authorities—makes it difficult to achieve fully connected systems. For these reasons, full and effective deployment of V2X systems may not be realized in the near future. To address this gap, UAVs could potentially play a critical role by facilitating communications between connected vehicles and other vehicles, infrastructure, etc., without requiring that all these neighbors be connected to each other. In addition, the flexible nature of UAV operations is such that they can facilitate macroscopic and microscopic characterization and analysis of the traffic stream. UAV connectivity to vehicles, infrastructure, and pedestrians can enable intelligent and real-time communications. Having this capability is useful for safe and efficient connected and autonomous vehicle (CAV) operations. By virtue of their accuracy, complexity, range, and availability of the traffic data they generally capture, UAVs have opened up new opportunities in the field of traffic monitoring, management, and analyses.

The potential benefits of UAV in intersection safety management are underscored by the fact that intersection safety is a top priority at the local, state, and national levels. According to Federal Highway Administration (FHWA), over 50 percent of all fatal and injury crashes occur at or near intersections [25]. Due to the complexity of mixed traffic flows at intersections, each of these accidents often involves multiple vehicles, pedestrians, motorcycles, and trucks. Consequently, transportation agencies including the National Highway Transportation Safety Administration, Federal Highway Administration, and Institute of Transportation Engineers continue to support the development of safety initiatives to reduce collision risk at intersections. The FHWA, in particular, has sponsored the investigation of crash causation factors and evaluating alternative intersection designs that facilitate the safe movement of pedestrians and bicyclists. Recently, the agency has reiterated the encouragement of edge computing platforms to facilitate real-time actions (detection of traffic events and subsequent decision-making) to enhance safe operations at signal-controlled intersections [25].

Against this background, this paper investigates the potential utilization of UAVs for assessing collision risk at intersections. The objectives of the paper are twofold: (1) propose a framework that uses data obtained from UAV and V2X connectivity to track the movement of road users and to assess potential collisions at intersections, and (2) demonstrate the framework using a case study involving an intersection. The proposed framework, facilitated using machine-learning models, is intended to enhance the extraction and analysis of reliable trajectory data and detection of collision risk. The proposed framework can help traffic engineers to assess safety conditions at intersections, recognize root causes of intersection safety hazards (as a precursor to identifying appropriate safety countermeasures), and design intersections for improvement not only in the current era but also in the prospective era of CAVs.

The rest of the paper is organized as follows: Section 2 reviews related work and Section 3 introduces the proposed framework. A case study is presented in Section 4. Section 5 provides overall concluding remarks of this work.

2. Related Work

In previous research, UAV applications in road safety management have been investigated. A few researchers have proposed frameworks that use photographs from drones to reconstruct accident scenes. Others have compared the use of UAVs to other alternatives for traffic monitoring including manned drones, helicopters, and road patrol vehicles [26] and have carried out multiple criteria analyses to identify the most cost-effective monitoring platform. They found that UAV has a lower cost compared to helicopters and is quicker to deploy compared to road patrols, and concluded that UAV is the best option for incident management. As UAV technology continues to develop, research attention is turning towards the processing of drone images captured at different shooting angles and altitudes and improving the quality of reconstructed scenes. For example, researchers have proposed low-cost methods that use UAV photogrammetry and other techniques to reconstruct traffic accident scenes [27] and assessment of reconstruction quality of the images using the concepts of peak signal-to-noise ratio and structural similarity [28]. Relatively few research efforts have addressed the UAV applications in safety risk assessment. Risk assessment entails a detailed analysis of vehicle trajectories extracted from UAV-based videos. From the trajectories, potential conflicts, high-risk lanes, and risky maneuvers can be identified and crash occurrence could be predicted. In [29], authors developed a framework to investigate crash risk at freeway interchange merging areas using data exported from a UAV and incorporated a driver behavior model to identify the factors of risky driving behavior. Other researchers have explored UAV applications in smart transportation and have used trajectories from the optical flow model for traffic parameter extraction, driver behavior analysis, and congestion detection [30].

The task of accurately extracting trajectories is one of the most challenging aspects of the UAV-based risk assessment process. Such difficulty is exacerbated by the heterogeneity that often characterizes the surveillance scene. For example, the scene at an intersection may be densely crowded and consist of objects that vary in their nature and features, and the presence of a substantial number of object classes (vehicles, pedestrians, or bicycles) with multiple interactions and behaviors. In addition, the recognition of specific activities can be challenging. Manual monitoring and review of large amounts of video data may be cumbersome and impractical. Therefore, accurate extraction of trajectories from videos is one of the most critical as well as challenging requirements for video-based applications. The task of tracking multiple trajectories is termed multi-object tracking (MOT). In MOT tasks, challenges that are encountered include occlusion, initialization and termination of tracks, the similarity of appearance, and interactions among different objects. In recent years, the rapid development of convolutional neural network deep-learning-based MOT algorithms with high computing speed and accuracy have been proposed to facilitate the task. Most existing MOT research can be placed into one of two categories: detection-based tracking (DBT) and detection-free tracking (DFT). The difference between them is that DFT performs detection, matching objects with trajectories, and tracking simultaneously. DBT, on the other hand, conducts detection and tracking tasks separately: objects are first detected and then linked to identify the trajectories. In recent tracking studies [31,32], benchmarks have been established for DBT models. Bose et al. proposed a framework for detecting and tracking multiple interacting objects with due cognizance of fragmentation [31]. In their experiments, 89 out of 94 moving objects were correctly tracked and 762 merges and splits were detected. DFT models, on the other hand, are free of pre-trained object detectors but require manual initialization of a fixed number of objects in the first frame [33,34]. It has been realized by at least one researcher [35] that simultaneous detection and tracking can be carried out using a detection model. DFT models attract significant research attention because they can address disappearing objects or emerging objects in the image frame. DBT is generally more time-consuming compared to DFT because the total time used for the DBT algorithm is the sum of time spent by two components.

In the literature, the assessment of collision risk at road intersections has been identified as a critical task yet to be addressed in the domain of transportation safety research [36,37,38,39]. The American National Standard [40] listed intersections as a locational context that is due for critical safety evaluation. The Standard defines an intersection as an area which “(a) contains a crossing or connection of two or more roadways not classified as driveway access and (b) is embraced within the prolongation of the lateral curb lines or, if none, the lateral boundary lines of the roadways”. If the distance along a road between two areas meeting the two criteria is less than 33 ft (10 m), then both areas and the connecting roadway are considered to be parts of a single intersection. Collision risk indicators are developed to quantify the potential risk for road users. Quantitative risk indicators include time-to-collision (TTC), time-to-brake (TTB), and time-to-steer (TTS), of which TTC is the most widely adopted. TTC is calculated as the duration between a reference timestamp and the time of the first impact between the vehicles if the concerned vehicles maintain their current speed vectors. In recent research, risk models that comprehensively consider vehicle motion/location, driver behavior, and road geometry information have been proposed [41,42,43]. In an effort to describe collision risks more reliably, researchers have proposed a number of metrics—TTC confidence levels, duration of the risk, configuration of the risk apart from the TTC value—as risk indicators to support driver decisions [19]. Similarly, a collision map with options for the ego vehicle to prevent or mitigate collision was also proposed by [44]. The review of literature also showed that the development of V2X network and cloud computing have enabled cooperative collision avoidance (CCA) and therefore brought the risk detection tasks to a more real-time and proactive level. According to CCA-related studies, CCA systems use vehicle-to-vehicle (V2V) communications [45,46,47] or vehicle-to-infrastructure (V2I) communications [45] to detect the possibility of accidents and to achieve cooperative collision avoidance. Studies have shown that in advanced CCA systems, vulnerable road users (VRU) can be recognized and warning messages sent accordingly. The US Department of Transportation (DOT) estimates that V2V can potentially mitigate as much as 82% of all crashes in the country that involve unimpaired drivers, thereby prospectively preventing thousands of fatalities and billions of dollars in property damage and economic loss [48]. Gelbal et al. introduced a pedestrian collision warning and avoidance system for road vehicles based on V2X communication signals from pedestrians’ smartphone apps that are used to detect them and their locations using dedicated short-range communications (DSRC) [20]. Du et al. incorporated a Model Predictive Control approach in V2V communication systems and proposed a method for autonomous driving vehicles to avoid crashes in a mixed traffic stream that contains aggressive human drivers exhibiting errant lane-changing behavior [42].

3. Methodology

The proposed framework consists of three main stages (Figure 1). The first stage addresses trajectory extraction and the second stage performs risk assessment. In the first stage, the CenterTrack model [35], trained using UAV-captured traffic videos, is applied in order to obtain real-time and historical trajectories of each road user. In the second and third stages, the crash risk associated with each road user is determined. The scale of the frames and speed of every road user are first calculated using results from the first stage. The crash risk between each pair of tracked road users is then estimated by calculating the time-to-collision (TTC) between them. The implementation details and further discussion are provided in subsequent subsections of this section of the paper.

3.1. Trajectory Tracking

Accurate trajectory tracking is a key requirement for the effective generation of profiles. Given an input video sequence, the multi-object tracking (MOT) task is required to locate multiple objects, maintain their identities, and yield their individual trajectories. In our scenario, the objects refer to road users (vehicles, motor vehicles, and pedestrians) at the intersection where the volume of objects is typically large. In addition, given the dynamic traffic pattern, we require an MOT model to capture the trajectories of road users quickly and accurately. Recent literature suggests that convolutional neural network (CNN)-based multi object tracking algorithms are promising approaches for doing this. As discussed in the previous section of this paper, CNN-based tracking algorithms fall into two categories: detection-based tracking (DBT) and detection-free tracking (DFT). In this paper, we recognize that traffic monitoring is inherently time-sensitive, and therefore, we use a DFT algorithm which is faster than the DBT algorithm. Zhou et al. developed a CenterTrack model, which is a simultaneous detection and tracking algorithm that is simple, fast, and accurate [35] and therefore, is a perfect fit for our case study demonstration. Therefore, we developed our method as a further enhancement of the CenterTrack model. CenterTrack identifies each object through its center point and then regresses to the height and width of the object’s bounding box. Specifically, it produces a low-resolution heatmap and a size map. In addition to the original output channels in CenterTrack, we introduce herein a new channel for object classification purposes. Figure 2 presents the structure of the tracking model. At time t, we are given an image of the current frame and the previous frame, as well as the heatmap of tracked objects from the previous frame. The heatmap is formed by the distribution of the confidence score of object centers. First, the heatmap and frames go through the convolutional layers separately and then are concatenated to feed into another sequence of convolutional layers. The output from the entire network includes object classification, displacement prediction, height and width of bounding boxes, and a heatmap for the current frame. The original loss function of Centertrack consists of three components: focal loss, size, and local location. The focal loss, which is the loss of object detection ($L_k$), is presented in Equation (1). In Equation (1), $Y_{xyc}\left(=0,1\right)$ indicates the ground truth heatmap of annotated objects. ${\hat{Y}}_{xyc}$ is the detected heatmap and N is the number of objects. α, β are hyperparameters for the focal loss. Compared to cross-entropy loss, the focal loss is an improved version of object detection by assigning greater weight to difficult-to-classify or easily misclassified entities. Therefore, the focal loss is more suitable for detection tasks under complex contexts such as drone-captured intersection images, where the “effective detection region” (i.e., regions occupied by road users) is relatively small compared to the background. The size prediction is learned by the loss function ($L_{\mathrm{size} }$) in Equation (2) and is supervised at the center locations. In Equation (2), $s_i$ is the bounding box size of the i-th object at location $P_i$ and ${\hat{S}}_{p_i}$ is the detected size. The offset calculated as the displacement of object centers is learned using the loss function ($L_{off}$) shown as Equation (3), where $p_i^{\left(t-1\right)}-p_i^{\left(t\right)}$ captures the difference in location of the object in the current frame $p_i^{\left(t\right)}$ and the previous frame $p_i^{\left(t-1\right)}$ and ${\widehat{D}}_{p_i^{\left(t\right)}}$ denotes the displacement at time t at location $P_i$ learned by the model.

$$L_k=\frac{1}{N}{{{\displaystyle \sum }}^{ }}_{xyc}\{\begin{array}{l}{\left(1-{\hat{Y}}_{xyc}\right)}^{\alpha }\log \left({\hat{Y}}_{xyc}\right) if Y_{xyc}=1\hfill \\ {\left(1-Y_{xyc}\right)}^{\beta }{\left({\hat{Y}}_{xyc}\right)}^{\alpha }\log \left(1-{\hat{Y}}_{xyc}\right) Otherwise\hfill \end{array}$$

(1)

$$L_{\mathrm{size} }=\frac{1}{N}{{{\displaystyle \sum }}^{ }}_{i=1}^N\left|{\hat{S}}_{p_i}-s_i\right|$$

(2)

$$L_{off}=\frac{1}{N}{\displaystyle \sum }_{i=1}^N\left|{\widehat{D}}_{p_i^{\left(t\right)}}-\left(p_i^{\left(t-1\right)}-p_i^{\left(t\right)}\right)\right|$$

(3)

3.2. Data Preparation and Risk Assessment

The crash risk of road users can be evaluated using the trajectory extracted at the previous stage of the methodology.

Table 1. Data for risk assessment.

Data	Notes
Scale	Match the video to real-world scales
Categories	Log all categories of different road users
$x_t,y_t$	Location of the center of the bounding box at time t
$H_t$	Height of the center of the bounding box at time t
$W_t$	Width of the center of the bounding box at time t
$V_t$	The speed at time t
$C_t$	Category of the detected road user at time t

presents a summarized set of data (and their notations) used for the risk assessment. First, the data are prepared to obtain the scale of frames and speed of road users. We assume that the length of a typical vehicle is 4 m, and the width is 1.7 m. A scale can be obtained by aligning detection boxes of vehicles in the video sequence with real dimensions of vehicles. The speed of road users is calculated using Equation (4) below:

$$v\left(t\right)=\frac{\sqrt[]{{\left(x_t-x_{t-{\Delta }_t}\right)}^2+{\left(y_t-y_{t-{\Delta }_t}\right)}^2}}{{\Delta }_t}\times scale$$

(4)

where ${\Delta }_t$ is the video frame frequency and the unit of speed is meters/second.

After the data preparation, data from each road user are assigned a unique ID. The data include the center of its bounding box, height and width of its bounding box, speed, and the category it belongs to in every frame. We adopted a widely used risk assessment parameter, the time-to-collision (TTC), as the measure of risk. TTC was first developed in 1972 [49]. The initial definition of TTC is “the time required for two vehicles to collide if they continue at their present speed and on the same path”. A lower TTC value corresponds to higher conflict severities and a TTC smaller than 2.5 s is typically taken as critical [50]. Hence, TTC is generally perceived to be a primary and efficient measure in traffic safety assessment. In this paper, for any two objects (e.g., object 1 and object 2 in Figure 3), the TTC is calculated using Equations (5)–(9) below:

$$Relative speed, \widehat{v_{rela}}=\widehat{v_2}-\widehat{v_1}$$

(5)

$$\left|v_{rela}\right|=\sqrt[]{v_1{}^2+v_2{}^2-2\left|v_1\right|\left|v_2\right|cos\alpha }$$

(6)

$$Distance, l=\sqrt[]{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2}$$

(7)

$$Projected speed, \widehat{v_{projected}}=\widehat{v_{rela}}\times cos\theta$$

(8)

$$TTC, ttc=\frac{l}{\left|v_{rela}\right|}$$

(9)

From the proposed model, the TTC value of each pair of all tracked road users can be easily achieved and road safety can be assessed at both macroscopic and microscopic levels. From the macroscopic perspective, a risk profile of the studied area at every time step can be established by identifying road users that exhibit the critical TTC. From the microscopic perspective, an individual road user is informed of their TTC relative to neighboring entities so that the road user can undertake an appropriate maneuver to enhance their safety. The case study section of this paper presents a detailed demonstration of the assessment.

3.3. Performance Evaluation

The success of the proposed framework is determined by how many risky TTCs it can correctly detect, which in turn depends on the accuracy of the trajectory tracking task. The TTC ground truth is obtained by feeding the true trajectory data into the Risk Assessment module. To better fit the UAV scenario, when training the trajectory tracking model, we used video clips provided by VisDrone [51], which consist of 56 video clips with 24,198 frames captured by UAVs. The trained model is tested on a test set containing 16 video clips with 6322 frames. In this paper, we consider six categories of road users: pedestrian, bicycle, car, van, truck, bus, and motorcycle. We used Multi-Object Tracking Accuracy (MOTA) to evaluate the tracking results [52]. MOTA is calculated using Equation (10) below:

$$\mathrm{MOTA}=1-\frac{{{{\displaystyle \sum }}^{ }}_t\left(m_t+f{p}_t+mm{e}_t\right)}{{{{\displaystyle \sum }}^{ }}_{t}g{t}_t}$$

(10)

where $m_t, f{p}_t, mm{e}_t, and g{t}_t$ are the number of misses, false positives, mismatches, and ground truth trajectories (road user trajectories), respectively, at time t. As shown in

Table 2. Evaluation of the trajectory tracking model.

Dataset	MOTA
Train set	64.89
Test set	63.12

, the tracking algorithm gives 64.89 MOTA on the training set and 63.12 MOTA on the testing set.

In using the extracted trajectories to produce risk profiles of a studied area, the true positive rate and false negative rate were logged as an evaluation matrix. Using 2.5 s as a threshold, the TTC between each pair of road users was labeled as safe vs. risky. A true positive means both ground truth and our proposed framework detect the TTC between each pair of road users as risky. A false positive means our proposed framework indicates a TTC as risky while the ground truth shows it is safe. A true negative refers to situations where both ground truth and our proposed framework indicate that the TTC is safe. Similarly, a false negative means that our proposed framework gives a safe TTC while the TTC is risky in the ground truth dataset. As shown in

Table 3. Evaluation of the risk assessment model.

Evaluation Metric	Value
Accuracy 1	78.2%
True positive rate	80%
False positive rate	31%

¹ Accuracy = (number of true positive cases + number of true negative cases)/total number of cases.

, our model yields a true positive rate of 80% and a false negative rate of 31%. For all the detected TTCs, 78.2% of the model results fall into the ground-truth categories of risky or safe designations.

4. Case Study and Analysis

To illustrate the analysis framework of the UAV-based risk assessment proposed in the sections above, a case study was conducted using drone images captured at an intersection in Tianjin, China. As shown in Figure 4, this is a 4-way intersection. The video data were provided by an open-source dataset [51] that includes intersection videos taken under various conditions including sunny weather, good light, and no electromagnetic interference (which could influence the stability of the video pictures at a vertical angle). The movements and interactions between vehicles in this intersection were captured at a frame frequency of 30 fps.

4.1. Road Traffic Performance Characterization

The UAV-captured data, after being processed by deep-learning networks, offer numerous applications to support road monitoring and management. As a proof of concept, we present here how these findings could be applied to measure the performance of the study area in terms of the safety and efficiency of traffic movement. Transportation performance measures, sometimes referred to as measures of effectiveness (MOEs), are quantitative estimates on the performance of a transportation facility, and include the level of service, crash frequency, and travel time [53,54]. Proper evaluation of transportation facility performance has always been supported by legislation [55]. At the current time, cities are growing at an unparalleled pace, particularly in Asia and South America. As such, there is a growing demand for information on traffic growth trends to support general transportation administration and management, and the development and evaluation of road safety policies. In this context, the proposed UAV framework can be beneficial to road and traffic managers because it is capable of generating large amounts of traffic data in real time.

The capability of the developed framework to generate traffic performance data is due to the inherent structure of the deep-learning network used in the framework: the detection results help identify the composition of road users and the tracking results help measure the speeds and directions of the road users. In the studied area (Figure 5), the intersection occupants are two-wheelers (36.6%), vans (2.3%), bicycles (10.1%), pedestrians (14%), and automobiles (36%).

From the results of the tracking analysis of the UAV data, the speeds of road users of different categories can be presented (Figure 6). The tracking analysis excludes the phase where road users wait for green traffic light signals. Of the road users that pass through the intersection, motor vehicles are those that show highest speeds as expected. In addition, the speed range for the motor vehicles is widest compared to all the other road users. In contrast, the travel speed of pedestrians and vans are relatively stable. This information could be used to generate several useful measures of the intersection performance. For example, the travel time index (TTI), which is travel time divided by the free-flow travel time, can be calculated [56]. A TTI value of 1.00 indicates travel at the free-flow speed, while a TTI value of 2.00 indicates travel that is twice as long, compared to free-flow conditions. The vehicle speed outcomes can be compared to target or design speeds to assess relative benefit. In analyzing the traffic performance of the study area, it is desirable to incorporate local input values; however, this is outside the scope of this study.

4.2. Risk Profiles

At each time step, the trajectories of all studied road users were tracked using the deep-learning-based model developed in this study. Then, the crash risk for every pair of road users was estimated using the TTC equation provided as Equations (5)–(9). It may be noted that only positive TTCs are considered in this study and TTCs smaller than 2.5 s are labeled as risky. For any road user, if the minimum correlated TTC is risky, the road user is labeled “risky”. Figure 7 presents a series of consecutive macroscopic risk profiles where risky road users are highlighted by their bounding box; the number indicated at the top of the box is the value of the most critical TTC value correlated to the road user in the box. As indicated by Figure 7, dynamic variation of the intersection risk profile is captured by videos. In addition to vehicles, pedestrians and bicycles that are risky are also identified by the proposed framework. The microscopic risk profiles can be obtained by extracting information for an individual road user. Figure 8 presents the risk profile of an individual vehicle and highlights the neighbors that have a “risky” level of TTC with respect to the individual vehicle in question. The individual vehicle is indicated by a red circle in the figure. Other road users that are risky are marked with blue boxes. The number indicated above each box is the TTC value between the road user in the box and the studied vehicle.

The benefits of such risk profiles are twofold. First, when transmitted (by the UAV) to the individual road user, the microscopic risk profile can help the road user become aware of potential crashes in its surroundings. This is considered particularly important in the prospective era of autonomous vehicles (AVs) because the AV’s in-vehicle detectors including cameras and Lidar may fail to identify all potential crashes or hazardous situations due to their narrow detection range and detection challenges. As such, the UAV not only serves as a robust source of information relating to a broad view of the surroundings and wider spatial characterization of the environment, but also makes available accurate data regarding potential risks. Second, the macroscopic risk profile provided by the UAV can provide useful insights to urban planners and transportation managers in their efforts to assess the safety level of an intersection, identify risky road users, and analyze reasons for potential collisions. If there exists a centralized control platform, then the platform could convey real-time warning messages to connected risky road users with appropriate collision-avoiding maneuver suggestions to mitigate collision. The risk profile patterns can be identified by summarizing data from the same intersection. In the studied area, 72% of potential collisions are formed between vehicles. In the remaining 28% of potential collisions (Figure 9), 40% are caused by pedestrians and vehicles and 30% of collisions happen between trucks and cars. The road agency overseeing the operations of the intersection may be interested in investigating the reasons why risk occurs so often between pedestrians and cars, and therefore, can recommend the construction of pedestrian-dedicated facilities to mitigate these problems. Figure 10 logged all locations where “risky” road user interactions are prevalent. It can be observed that the most critical potential crashes occurred in the upper right corner of the intersection. This may be due to a large number of bicycles and pedestrians who typically occupy that area, where they share the lane with vehicles. As a result, it is difficult for vehicles to undertake safe turning maneuvers. Intersection designers can also use the results of such analysis as a basis to carry out intersection improvements.

The results of this study are consistent with a national effort to assess safety risks at road sections and intersections. In 2009, the US Federal Highway Administration (FHWA) conducted a program to address pedestrian safety concerns by developing and researching effective tools and countermeasures and by coordinating projects, plans, and discussions with State and local officials and safety advocates [56]. The initiative has been echoed by efforts at the local level for road administration. For example, the Chicago Department of Transportation completed an extensive pedestrian crash analysis to identify specific crash factors and characteristics including when and where pedestrian crashes occurred, road user categories that were involved in pedestrian crashes, and the contributing factors related to the pedestrian crash. The report advocated the construction of marked crosswalks, in-road state stops for pedestrians’ signs, and pedestrian refuge islands at roads and intersections considered to be risky [57]. In addition, the 2012 Chicago Pedestrian Plan identified opportunities and ongoing plans to increase the safety of the city’s pedestrians. Similarly, the City of Austin developed a Pedestrian Safety Action Plan based on a comprehensive analysis of intersections considered dangerous [58]. The risk profiles extracted by UAVs, as demonstrated in this paper, would facilitate such programs and enable them to be more efficient and focused. As depicted in Figure 11, the information exchange between UAVs, connected vehicles, and transportation agencies facilitates the dissemination of microscopic and macroscopic risk profiles, and helps identify appropriate safety countermeasures.

4.3. Risk Prediction

In assessing the risk associated with an intersection, it is also of interest to predict the risk of a vehicle at subsequent time steps. To predict such future crash risk, we deployed a Random Forest classifier, a supervised learning algorithm. Random Forests create decision trees on randomly selected data samples, obtain predictions from each tree and select the best solution through voting. The major advantage of Random Forest is that it provides an indicator of the feature importance which offers insights on features that are influential in distinguishing the data samples. Such information makes the model interpretable and facilitates the pre-emptive identification (before a crash occurs) of vehicles associated with “critical” interactions so that risk could be mitigated in a proactive manner. It may be noted that only cars are included as studied objects. This is because the moving pattern of car–other pairs (that is, cars and other road users) is different. The features fed into the random forest model include speed, location, safety condition of the studied vehicle and its neighbors, together with TTC and distance between them in the five consecutive previous time steps. The output of the classifier is either “safe” or “risky”. “Safe” means that for the studied vehicle, the smallest predicted TTC at the next time step is greater than 2.5 s, and “risky” means the smallest predicted TTC is less than 2.5 s. From the Random Forest model, we obtain the importance of different features regarding future risk prediction, which is calculated by a Gini Importance value that sums over the number of splits (across all trees) that include the feature, proportionally to the number of samples it splits. A higher Gini Importance value indicates the feature is more likely to be the essential difference between different categories. Figure 12 presents the top 5 important features and their levels of relative importance. A “dangerous road user” refers to the neighbor with the smallest TTC with respect to the studied vehicle in the last time step. Features with higher importance contribute more when predicting risky vehicles, indicating that we can observe these features to predict the future potential risk of a vehicle. The threshold of these features could also be extracted from the Random Forest classifier. The threshold values are not discussed in this paper because the threshold values are very specific to the studied area and the time of capturing the video, and cannot be generalized. According to the Random Forest classifier, the vehicle speeds in the previous time steps are most related to its future safety condition. The status of the vehicle’s neighbors, particularly the location and speed of its dangerous neighbor, also plays a significant role in determining a vehicle’s future safety condition.

In practice, warning messages could be generated based on these results and sent to the vehicles concerned to remind them to be aware of the imminent danger of traffic collision. For example, when the speed of a vehicle exceeds the speed threshold specified in the classifier, the vehicle could be alerted to reduce speed. Over the past decade, there has been an upsurge in the availability of collision-warning systems in cars sold in the US [59]. The benefits of pre-collision warning systems have been verified by the Insurance Institute for Highway Safety (IIHS) whose data suggest that collision warning reduces rear-end accidents by 27 percent [60]. Wider adoption of collision warning systems could be anticipated considering the rapid advancement of autonomous driving technologies. Currently, the collision-warning systems are mainly powered by on-vehicle ranging sensors (e.g., cameras and radar) and are limited to forward collision warning (FCW), pedestrian detection system (PDS), and lane departure warning (LDW). Vukadinovic et al. proved that cellular-V2X systems increased reliability of communication performance under increasing congestion on the wireless channel but the UAV-based V2X systems have not attracted much attention in safety analysis [61]. The UAV-based collision warning system addresses the inadequacy of onboard sensors and therefore opens up new potential for collision avoidance systems for autonomous driving cars. It should be noticed that the design of such a UAV-based warning system should be adapted to local intersection data which can be retrieved from videos of local intersection traffic. In addition, as depicted by Figure 12, the location of a vehicle plays a key role in identifying its risk, which aligns with our finding in the section above that a critical region exists at an intersection. The critical region may be a result of improper intersection design including inadequate sight distance and inappropriate traffic light phase and visibility. Recognition of regions of high risk is critical in safety management; however, identifying these regions could be a challenging task. The results of this analysis can serve as a basis for addressing this task.

5. Concluding Remarks

This study presents a methodology to assess traffic safety at intersections utilizing UAV video. The methodology explores the potential crashes between each pair of road users by extracting their trajectories from video images and calculating their time-to-collision values. To develop the trajectories of all detected road users efficiently and accurately, the study used a deep-learning-based multi-object tracking algorithm. The trajectory data were re-scaled for the risk assessment. Then, a method was suggested to compute the crash risk between pairs of road users by calculating the time-to-collision between them. The paper shows how the data provided by the UAV (including road user composition, their speed distributions, and TCC values) can help road safety managers to identify conflicts and other problem areas, develop targeted countermeasures, and measure the general performance of intersections and other road facilities. Based on the TTC value, “risky” road users whose smallest TTC is less than a threshold (2.5 s in this paper) are identified, and a macroscopic risk profile can be established and presented to the road agency that manages the intersection. An individual road user can acquire its own microscopic risk profile from the UAV so that it can make its safe and informed movement decisions accordingly. In the case study, we demonstrated how our framework could assist intersection management in the current era of human driving, and more importantly, in the future era of autonomous driving. The results showed that by investigating consecutive macroscopic risk profiles, the spatial-temporal pattern of risk profiles can be observed. The results of this study suggest that it is possible to use UAV-captured videos to identify critical zones where potential collisions happen most frequently, and to identify the riskiest road user categories. Urban planners and intersection managers may find these results useful in their efforts to improve traffic control, design configuration, and ultimately, safety at intersections. In addition, the paper deployed a Random Forest model to predict the safety condition of a vehicle by utilizing historical risk profiles, and the results suggest that the travel speed is the most critical factor of a vehicle’s future safety condition. The speed of the vehicle’s neighbor was also found to be influential. With the proposed model, traffic engineers can be placed in a better position to propose efficient countermeasures to enhance road safety at intersections. Moreover, the proposed model can provide CAVs information that is helpful for making informed driving decisions and make data available for traffic engineers that may be considering intersection improvements from design or operating policy perspectives.

There are a number of limitations of this study, which are indicative of possible future work improvements. First, the VisDrone dataset does not provide the geometry and coordinates of the road infrastructure; therefore, in this study, we did not consider the impact of road geometry on the collision risk. For example, where a vehicle approaches the intersection via a misaligned road segment, the time-to-collision can be influenced by such anomalous geometry of the roadway. Ideally, the TTC calculation should reflect such anomaly. In future studies, this limitation may be addressed by the use of datasets that include more detailed information (such as lane directions, skew angles, and geometry) that captures any irregularities associated with the roadway infrastructure. Secondly, the current research work uses the VisDrone dataset, which is not only related to a rather limited range of roadway geometric contexts (that is, intersections only) but also involves a narrow range of tilt angles and traffic conditions. Therefore, future work could address geometric contexts (roundabouts, straight or curved highway segments, and so on), and a wide range of site perspective views (i.e., camera tilt angles) and traffic conditions.