1. Introduction
With the rapid development of automatic-driving technology, many functions of low-level advanced driver-assistance systems have been implemented in an increasing number of vehicles. However, for high-level automatic-driving systems, it is imperative to develop safer and more intelligent decisions and control for automated vehicles under increasingly complex traffic scenes. As a typical traffic scene with a high incidence of accidents, unsignalized intersections have been investigated by many researchers for decision making and control to promote driving safety and efficiency [
1,
2].
As a classical method, behavior prediction for surrounding vehicles in traffic environment has proved to be an efficient way of dealing with decision-making problems. Zyner et al. [
3] leveraged the long short-term memory (LSTM) recurrent neural network (RNN) to predict the intention of the driver when a vehicle enters an intersection, contributing to the decision making of an autonomous vehicle. A decision-making framework is proposed by Samyeul in [
4] for autonomous vehicles to predict the future trajectory of observed vehicles and to delineate the potentially dangerous collision area to help navigate the intersection safely and efficiently. In [
5], a motion-planning method for autonomous vehicles is introduced via rapidly exploring the random-tree algorithm. To solve motion-planning problems in environments with dynamic obstacles, the algorithm combines the RRT algorithm and the configuration-time space to improve the quality of the planned trajectory. Ramyar et al. [
6] present a data-driven technology using the Takagi–Sugeno fuzzy model to simulate and predict driver behavior at intersections, thereby further improving prediction accuracy.
Model predictive control (MPC) as a commonly used control method has been widely exploited in decision control of autonomous driving at intersections. In [
7], a bilevel MPC algorithm is established for the coordination of autonomous vehicles at intersections, and a distributed sequential quadratic-programming (QP) method is leveraged to solve the intersection-level optimization problem. Zhao et al. [
8] developed a collaborative-driving algorithm for connected and automated vehicles at unsignalized intersections based on MPC, and a decentralized controller was advanced to control each vehicle to pass through the intersection smoothly. A probabilistic model was devised to predict the trajectory of the target vehicle [
9], and afterwards was integrated within a collision-avoidance model. Katriniok et al. proposed a distributed MPC approach that enables multiple vehicles to pass through an intersection simultaneously with a safe and efficient manner [
10]. A study was conducted concerning the decision-making control in intersections with multiple surrounding vehicles [
11], wherein a robust MPC is responsible for searching security breakthrough in the studied scene, and meanwhile, planning the optimal trajectory.
In recent years, partial observable Markov decision processes (POMDP) have been progressively employed for autonomous-driving decisions at intersections. Bouton et al. [
12] defined the traffic problem at unsignalized intersections as a POMDP, and the Monte Carlo sampling method was adopted to solve the problem. Shu et al. [
13] proposed a method for decision-making control for left-turning intelligent vehicles based on the key turning points at intersections, and a partially observable Markov model was employed to solve the optimal speed sequence in the left-turn process. Kye et al. [
14] introduced an intent-aware autonomous-driving decision-making method at unsignalized intersections, where the intents of traffic participants were modeled as dynamic Bayesian networks, and the intent-aware decision-making problem was modeled as a POMDP based on the inference results. Hubmann et al. [
15] considered the occlusion generated by static objects and dynamic objects at the same time, and a general autonomous-driving strategy based on POMDP was advanced under urban conditions. In [
16], a POMDP framework was proposed for online autonomous driving in different situations.
Machine-learning algorithms, such as reinforcement learning (RL), are also widely exploited in the field of decision control. Deep RL (DRL) combines the perception ability of deep learning and the decision-making capability of RL, performing well in solving continuous motion-control problems [
17,
18]. Islee et al. [
19] investigated the effectiveness of DRL in dealing with intersection decision-control problems. Through comparison study, a deep Q network enables the learning of strategies better than common heuristic methods for different indicators, such as traffic time and traffic rate; however, the generalization ability is limited. Shi et al. proposed a coordinated control method with proximal policy optimization in a vehicle-road-cloud integration system, and a policy of the connected vehicles was learned by RL to across the intersection safely [
20]. Chen et al. [
21] proposed an autonomous intersection-management system based on DRL, and a braking safety-control model was applied to ensure the safety of each autonomous vehicle at the intersection. Zhou et al. [
22] established a vehicle-following model for intelligent vehicles based on RL to improve driving behavior at intersections. By specifying an effective reward function, the model can be learned and works well under different conditions to improve fuel consumption, safety, and driving efficiency.
In view of the research status of autonomous-driving decision making and control at integrated intersections, planning methods based on state-prediction results of environmental vehicles usually quantify the degree of risk of intersection collisions, and rule-based strategies are proposed to make decisions for intelligent vehicles. However, rule-based strategies exhibit poor generality, and the formulation of rules depends on the practical experience, greatly affecting the effectiveness of the algorithm. Problems in the decision and control of intelligent vehicles at intersections are complex and involve multifactor coupling [
23]. Crossing an intersection is a complex driving behavior [
24]; thus, it is necessary to simplify the intersection-scene model to a certain extent to make decision rules depending on the quantified degree of risk, leading to certain differences between the simplified scene and the actual scene [
25]. Generally, a POMDP model requires a large amount of computation. Although Monte Carlo sampling can mitigate this concern, the required discretization of the motion space will also lead to deteriorated accuracy to some extent [
13]. The method based on model prediction strongly relies on the accuracy of the established model; thus, many factors should be considered comprehensively in the modeling process to achieve a satisfactory control effect [
8]. In contrast to the above methods, a specific control model is not required in RL due to its model-free characteristic. Decision making for straight intelligent vehicles at intersections is a continuous action-control problem, and thus it is well-believed that the decision-making control problem of intelligent vehicles at intersections can be solved by an RL method.
Motivated by these conditions, in this study, a decision-and-control model based on RL is designed. The main contributions of this study are as follows. (1) A method is proposed to judge the priority of crossing the intersection based on a speed prediction by an autoregressive integrated moving average (ARIMA), and to calculate the expected speed of an autonomous vehicle. (2) A decision-and-control model is constructed based on speed prediction and RL. The model incorporates the expected speed guided by the RL model to converge in the optimal direction, thereby saving the learning time of the agent. (3) A multiobjective decision-making control-effect-evaluation system is established with the consideration of success rate, speed punishment, safety, traffic efficiency, and comfort.
The remainder of this article is structured as follows. In
Section 2, the geometric model of the road and the circular model of the vehicle body are established, and a mathematical analysis of the intersection confluence trajectory data is presented. In
Section 3, a decision-and-control method based on speed prediction, RL, and evaluation methods is introduced in detail. In
Section 4, the simulation and effects validation are addressed.
Section 5 draws the main conclusions of this study.
2. Intersection Confluence Condition Modeling
To better analyze the decision-making process and explain the mathematical model of the subsequent decision-making control, the road-geometry model of the research object should be constructed first. When passing through the intersection, a vehicle generally has three directions to go, as shown in
Figure 1a, where a-f represents the possible driving direction of the vehicle. As shown in
Figure 1b, the relationship between two vehicles can basically be divided into three types: irrelevant (1–4), cross (5–7), and confluent (8,9). The areas with probability of collision are marked with a yellow box in the figures. Under the confluence condition, two vehicles will eventually drive into the same lane; therefore, the potential collision area is longer than in other conditions. This scene not only includes the decision-making and control problem in the process of two vehicles when passing through the intersection, but also contains the continuous influence between two vehicles after confluence. Therefore, this paper selects b and d for subsequent modeling and analysis.
Figure 2 illustrates the road-geometry model under the conditions of two-way single-lane confluence. Straight and turning vehicles enter the intersection from different junctions and eventually converge into the same lane. The center lines of the east–west and north–south lanes at the intersection are labeled as
and
, respectively;
through
represent the stop line at the intersection; and
is the confluence point of the two vehicles.
2.1. Circular Model of Vehicle Body
The trajectory shown in
Figure 2 shows only the centroid movement process of the straight and turning vehicles without considering the actual geometric size of the vehicle. In a real driving scenario, the geometric size of the vehicle body cannot be ignored to avoid the potential risk of collision in the process of two vehicles converging at the intersection. Therefore, a circular model, which has been widely adopted in studies on vehicle collisions, is used to represent the vehicle body profile hereinafter, as shown in
Figure 3. By this manner, the radius of the circular model can be calculated by
where
and
denote the width and length of the vehicle, respectively; and
denotes the radius of the body circle.
Considering the circular model of the vehicle body, the actual motion trajectory of straight and turning vehicles under the confluence condition at the intersection is shown in
Figure 4. The trajectory of the turning vehicle is assumed to be composed of two straight lines and a 1/4 arc, in which TR and GS denote the turning section and straight section of the turning vehicle, respectively;
denotes the radius of the arc; and
represents the chord length of the arc. In the TR phase, the vehicle turns to the right and eventually merges to the same lane as a straight vehicle. As the two vehicles become closer, the risk of collision increases; therefore, this is the area that our research is focused on.
The critical condition of collision judgment can be determined by (2), where
denotes the coordinate of the straight vehicle and
represents the coordinates of the turning vehicle.
According to (2), we can easily establish areas where collisions may occur during the confluence of two vehicles, as shown in
Figure 5. The CA area in red enclosed by the lines, which is 2
r from the centerline of the trajectories
y2 and
x1, is the only latent collision area of the two vehicles before the confluence point
, since the distance between the two vehicles will not be less than 2
r outside this area.
According to the different positions and speeds of the two vehicles in the confluence process, there are several feasible solutions to (2). Nonetheless, it is difficult to solve all the vehicle positions where the body circles are tangential, while the important points in the CA area can be analyzed. We selected the confluence point and the turning point of the turning vehicle for analysis.
By finding the point on the track of the turning vehicle at a distance of 2
r from the merging point, it can be observed that when the straight and turning vehicles are located around the merging point and end of the GS segment, the body circles of the two vehicles touch in tangent, as shown in
Figure 6. Another important body circle, C2, is obtained by taking the symmetry position of the merging point
about the straight line
.
The analysis shows that there will be no collision between the two vehicles before the straight vehicle reaches the position of the C2 body circle. When any of the two vehicles reach the position of C3 in advance, the rear vehicle should pass through the intersection in the vehicle-following mode after the prior one. Hence, the body circles C1, C2, and C3 represent the key positions in the proposed road-geometry model approximately with regard to collision avoidance at intersections. A detailed analysis of the three body circles is presented in the following content.
2.2. Statistical Analysis of Intersection Confluence Trajectory Data
To mimic the actual working conditions in this study, the intersection-trajectory dataset published by Open ITS [
26] was imported to conduct the driving-behavior analysis at intersections. Open ITS is a traffic-data resource-sharing platform built jointly by different research institutes and enterprises. Open ITS provides trajectory data of intersections involving a total of 60 confluence conditions.
Figure 7 shows one example of the dataset, which contains information about the position, speed, and acceleration change with time, of straight and turning vehicles.
A speed that is too high or too low at an intersection is dangerous, and as such it is necessary to determine the speed and acceleration threshold of vehicles driving through the intersection. Grouped statistics are made according to the passing order of vehicles going straight at intersections, and the statistical results are shown in
Figure 8 and
Figure 9, where
Figure 8 shows the speed and acceleration distribution of straight and turning vehicles in the case of a straight vehicle going before a turning vehicle, and
Figure 9 sketches the distribution when the straight vehicle gives way.
The statistical results reveal that under the two working conditions the velocity and acceleration are mainly distributed in a certain range. The vehicle speed is mainly distributed in the range of 0 to 8 m/s, and the acceleration is mainly distributed in the range of −2 to 2 m/s2, thereby providing a reference basis for boundary-condition setting.