Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning

Abstract

Autonomous maneuver decision making is the core of intelligent warfare, which has become the main research direction to enable unmanned aerial vehicles (UAVs) to independently generate control commands and complete air combat tasks according to environmental situation information. In this paper, an autonomous maneuver decision making method is proposed for air combat by two cooperative UAVs, which is showcased by using the typical olive formation strategy as a practical example. First, a UAV situation assessment model based on the relative situation is proposed, which uses the real-time target and UAV location information to assess the current situation or threat. Second, the continuous air combat state space is discretized into a 13 dimensional space for dimension reduction and quantitative description, and 15 typical action commands instead of a continuous control space are designed to reduce the difficulty of UAV training. Third, a reward function is designed based on the situation assessment which includes the real-time gain due to maneuver and the final combat winning/losing gain. Fourth, an improved training data sampling strategy is proposed, which samples the data in the experience pool based on priority to accelerate the training convergence. Fifth, a hybrid autonomous maneuver decision strategy for dual-UAV olive formation air combat is proposed which realizes the UAV capability of obstacle avoidance, formation and confrontation. Finally, the air combat task of dual-UAV olive formation is simulated and the results show that the proposed method can help the UAVs defeat the enemy effectively and outperforms the deep Q network (DQN) method without priority sampling in terms of the convergence speed.

1. Introduction

With the progress of combat mode and the expansion of combat scale, modern air combat gradually extends from the within visual range (WVR) air combat to the beyond visual range (BVR) air combat [1]. Unmanned aerial vehicles (UAVs) are more and more widely used in military tasks such as investigation, monitoring and target attack [2] because of their low cost, strong mobility and high concealment. Due to the limitations of a single UAV’s mission and combat capability, autonomous multi-UAV cooperative air combats have become a research hotspot in recent years [3].

Autonomous air combat maneuver decision making refers to the process of automatically generating the maneuver control decisions of UAVs based on mathematical optimization and artificial intelligence [4], which requires that UAVs have independent capabilities for autonomous sensing, information processing and decision-making abilities [5]. At present, there are many autonomous decision making methods for UAV air combat maneuver control, which can be roughly divided into two categories: Analytical solution methods and intelligent optimization methods. Analytical solutions include matrix games, influence diagrams, differential games, etc. Matrix game method [6] uses a linear program to find the optimal solution in a short decision time, which cannot guarantee global optimization. In addition, this method needs to introduce expert experience to design the income matrix in line with the actual air combat, which is time consuming and laborious. The maneuver decision making method based on influence graph [7,8] can intuitively express the air combat model of key factors such as threat, situation and pilot’s subjective preference through influence graph, but it is difficult to obtain the analytical solution by this method, and the calculation time is long. So the influence graph cannot meet the real-time performance of air combat decision making. In [9,10], the knowledge of game theory is introduced into air combat to realize the one-to-one autonomous maneuver decision of UAV, and the method in [9] solved the curse of dimension problem by using fuzzy theory. However, the method in [9] does not take into account the current state of the enemy aircraft when designing the state, and lacks confrontation simulation results. The method used in [10] has complex calculation and poor real-time performance, and is not suitable for the high dynamic environment. The differential game method [11] is the most practical decision making model for studying air combat games. However, due to the difficulty in setting the performance function, the huge amount of calculation and the ill condition after the model is simplified, although the differential game theory has been developed for many years, it has not produced a more reasonable description of actual air combat. In [12], researchers propose an air-to-air confrontation method based on uncertain interval information, but only analyze the influence of different factors on air combat effect, and do not consider the maneuver model of UAV. In addition, the method in [12] needs to calculate the revenue and expenditure matrix, which is cumbersome and has low real-time performance. In short, the analytical solution method needs to accurately model and describe the decision model, which cannot be applied to the air combat scene without model or incomplete environment information, and cannot meet the requirements of intelligent air combat.

Intelligent optimization methods mainly include expert system method [13], neural network method [14] and some other optimization algorithms such as fuzzy tree, particle swarm optimization [15] and reinforcement learning. The maneuver decision making method based on the expert system has mature technology and is easy to implement, but its disadvantage is that the establishment of the knowledge base is complex and it is difficult to fully cover all air combat situations. The maneuver decision making based on the neural network has strong robustness and learning ability, but it is a supervised learning method, which cannot be applied without a training set. While the application of neural networks in air combat decision making has practical value, it is worth further exploration and improvement. A maneuver decision making method based on the fuzzy tree is proposed in [16], which can guide UAVs to make more targeted maneuver decisions according to a real-time combat situation. However, the design of fuzzy tree is difficult and the hyper-parameters are complex and diverse, and expert experience needs to be introduced. In [17], researchers use dynamic game and particle swarm optimization to realize multi-agent task allocation and confrontation. This method will make the payment matrix of both parties become huge with the increase of the number of agents, and the income matrix needs to be designed manually. Therefore, subjective factors have a great impact on the experimental results. In addition, the simulation result is a two-dimensional plane without considering the maneuver control model of UAV, which is very different from the real maneuver.

Reinforcement learning [18] is an intelligent optimization method that uses the “trial and error” method to interact with the environment, learns from the environment and improves the performance with time [19]. It overcomes the shortcomings of complex modeling, difficult sample marking and cumbersome solutions of other methods, and can produce a series of decision sequences considering long-term effects through self-interactive training without manual intervention. It is a feasible modeling method for autonomous decision making of air combat maneuvers in artificial intelligence [20,21]. The autonomous maneuver decision making problem of air combat based on deep reinforcement learning is studied in [22,23,24,25,26]. In [22,23], researchers verify the performance of the algorithm by building a high simulation combat platform, and has good experimental results. However, the reward function in [23] is sparse, and the reward is 0 in most states of each round, which is not conducive to network training. The robust multi-agent reinforcement learning (MARL) algorithm framework is used in [24] to solve the problem that the reinforcement learning algorithm cannot converge due to the unstable environment in the training process. However, the simulation environment in [24] is a two-dimensional plane and the simulation test initialization is fixed, which makes it hard to be applied in the dynamic confrontation scenarios. Many aspects of UAV situation assessment is considered in [25], but UAV uses absolute coordinates as the state input, which is highly dependent on spatial characteristics. In [26], researchers use Monte Carlo reinforcement learning to carry out research. The biggest problem is that the agent needs to complete a complete air combat process to evaluate the reward. Moreover, the above references consider the one-to-one air combat scenario, which has limited reference value for the research of multi-aircraft cooperative autonomous control. There are few studies on multi-agent confrontation using reinforcement learning algorithms. In [27], a novel autonomous aerial combat maneuver strategy generation algorithm based on state-adversarial deep deterministic policy gradient algorithm (SA-DDPG) is proposed, which considers the error of the airborne sensor and uses a reward shaping method based on maximum entropy inverse reinforcement learning algorithm. However, the reliance on expert knowledge in the design of reward functions in this paper is not conducive to extension to more complex air combat environments. In [28] researchers propose an air combat decision-making model based on reinforcement learning framework, and use long short-term memory (LSTM) to generate a new displacement prediction. However, the simulation experiments in [28] rely on an off-the-shelf game environment, which is not conducive to the extension of the study and it studies the air combat problem of searching for observation station in a non-threatening environment, which differs significantly from the air combat mission of this paper. Based on the MARL method, the simulation in [29] of multiple UAVs arriving at their destinations from any departure points in a large-scale complex environment is realized. However, the modeling environment is planar, and a sparse reward function is used, and only distance penalty is considered. The method for maneuver decision making of multi-UAV formation air combat in [30] is robust. However, there are no simulation results, and there are only three maneuver behaviors. The deep deterministic policy gradient (DDPG) algorithm is used in [31] to realize the maneuver decision of the dynamic change of UAV quantity in the process of swarm air combat. The algorithm has robustness and expansibility, but the waypoint model is used in this paper, which cannot describe the maneuver characteristics of UAV. Other researches on air combat based on reinforcement learning, the intelligent decision making technology for multi-UAV prevention and control proposed in [21,32]. The control method of UAV autonomous avoiding missile threat based on deep reinforcement learning introduced in [33,34]. Deep reinforcement learning is used in [35] to build an intelligent command framework and so on. These studies focus on the feasibility of reinforcement learning methods in solving some air combat problems, which has little correlation with our autonomous maneuver decision making problem, but provides some ideas for our research. In addition, uniform sampling is used in [21,22,23,24,25,26,29,30,32], which means that the probability of all experiences in the experience pool being extracted and utilized is the same, thus ignoring the different importance of each experience, resulting in long training time and extremely unstable.

Generally speaking, at present, the research on air combat maneuver decision making based on reinforcement learning mainly focuses on single UAV confrontation tasks, and the research on multi-UAV confrontation and multi-UAV cooperation are in the initial exploration stage. These studies have one or more of the following problems: Dimension explosion, rewards are sparse and delayed, simple simulation environment, lack of maneuver model, incomplete situation assessment and random uniform sampling leads to slow training.

In this paper, an autonomous maneuver decision making method based on deep reinforcement learning is proposed for dual-UAV cooperative air combat. The main contributions are as follows. First, aiming at the problems of dimension explosion, sparse and delayed rewards and incomplete situation assessment, we discretize the continuous air combat state space into 13 dimensions for dimension reduction and quantitative description of air combat states. Then the situation assessment model is established based on the relative location between the UAV and the target. Second, a reward function is designed according to the situation assessment results which includes the real-time gain due to maneuver and the final combat winning/losing gain. Such a design helps to solve the problem of sparse and delayed reward in the games of long-running time for ending. Third, aiming at the problem of slow convergence caused by random sampling in conventional DQN learning, an improved priority sampling strategy is proposed to accelerate the convergence of the DQN network training. Fourth, we apply and modify the designed autonomous maneuver decision making method for the typical task of dual-UAV olive formation air combat, which enables the UAVs to own the capability of collision avoidance, formation and confrontation. Finally, the proposed method is validated by simulation using practical fixed-wing UAV models and compared with the DQN learning method without priority sampling. The simulation results show that our method can make the two UAVs defeat the enemy effectively and improve the performance in terms of the convergence speed.

The following parts are arranged as follows: Section 2 is the problem formulation. Section 3 is the air combat confrontation algorithm based on deep reinforcement learning. Section 4 is the description of typical air combat scenarios and the design of dual-UAV cooperative autonomous maneuver strategy. Section 5 conducts simulation analysis. Section 6 summarizes the full paper.

2. Problem Formulation

This paper studies the autonomous maneuver control decision of multi-UAV BVR cooperative tracking and close combat. Maneuver control model and situation assessment are the premises of UAV maneuver decision. Therefore, in the following sections, we will elaborate on UAV autonomous maneuver decision making from three aspects: Maneuver control model, situation assessment and maneuver control decision.

2.1. UAV Dynamic Model

As shown in Figure 1, in the ground coordinate system, the $ox$, $oy$ and $oz$ are the east, north and vertical directions respectively. The motion model of UAV in the coordinate system is given by

$$\left\{\begin{array}{c}\dot{x}=vcos\gamma sin\psi ,\hfill \\ \dot{y}=vcos\gamma cos\psi ,\hfill \\ \dot{z}=vsin\gamma ;\hfill \end{array}\right.$$

(1)

where x, y and z represent the position of the UAV in the coordinate system, v represents the current speed direction of the UAV, $\dot{x}$, $\dot{y}$ and $\dot{z}$ represent the change rate of v in the three coordinate axis directions, $v^{\prime }$ is the projection of v on the $xoy$ plane, $\gamma$ is the angle between $v^{\prime }$ and v, and $\gamma$ represents the pitch angle, $\psi$ is the angle between $v^{\prime }$ and $oy$ axis, and $\psi$ represents the yaw angle. In the same coordinate system, the dynamic model of UAV can be expressed as

$$\left\{\begin{array}{c}\dot{v}=g\left(n_x-sin\gamma \right),\hfill \\ \dot{\gamma }=\frac{g}{v}\left(n_{z}cos\mu -cos\gamma \right),\hfill \\ \dot{\psi }=\frac{g{n}_{z}sin\mu }{vcos\gamma },\hfill \end{array}\right.$$

(2)

where g is the gravitational acceleration. $n_x\in \mathbb{R}$ and $n_z\in \mathbb{R}$ represent tangential overload and normal overload, and $\mu \in \left[-\pi ,\pi \right]$ represents the roll angle around v. $\left[n_x,n_z,\mu \right]\in {\mathbb{R}}^3$ are the feasible basic control parameters in the UAV maneuver control model and they jointly control the direction and magnitude of UAV speed. $\left[n_x,n_z,\mu \right]$ are often used as the command for air combat maneuver decision making.

Figure 1. Motion model of UAV.

2.2. Situation Assessment Model

As shown in Figure 2, denote by $P_{{\mathrm{U}}_t}$ and $P_{{\mathrm{T}}_t}$ the position of UAV and target at time t. Denote by ${\phi }_{{\mathrm{U}}_t}$ the angle between the vectors $P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}$ and $v_{{\mathrm{U}}_t}$, named as the lag angle, and similarly, ${\phi }_{{\mathrm{T}}_t}$ for the angle between the vectors $P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}$ and $v_{{\mathrm{T}}_t}$, named as lead angle, which are defined as

$${\phi }_{{\mathrm{U}}_t}=arccos\left(\frac{v_{{\mathrm{U}}_t}\left(P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}\right)}{∥v_{{\mathrm{U}}_t}∥∥P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}∥}\right),0\le {\phi }_{{\mathrm{U}}_t}\le \pi ,$$

(3)

$${\phi }_{{\mathrm{T}}_t}=arccos\left(\frac{v_{{\mathrm{T}}_t}\left(P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}\right)}{∥v_{{\mathrm{T}}_t}∥∥P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}∥}\right),0\le {\phi }_{{\mathrm{T}}_t}\le \pi ,$$

(4)

where $D_{{\mathrm{U}}_t{\mathrm{T}}_t}=∥P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}∥$ is the distance between UAV and target.

Figure 2. Situation assessment model.

Based on the attack model [36] and evaluation function [31,37], the effective attack range of a UAV in the air combat is a cone with an axis in the direction of $v_{{\mathrm{U}}_t}$ and angle of ${\phi }_m$, which is truncated by a ball of radius $D_{max}$ as shown in Figure 2, where $D_{max}$ represents the attack range of weapons. Similarly, we can define the cone-shape attack range for the target. The UAV should try to follow the target as much as possible. That is, the smaller the ${\phi }_{{\mathrm{U}}_t}$ is, the greater the probability of UAV successfully attacking the target is. On the contrary, the larger the ${\phi }_{{\mathrm{T}}_t}$ is, the greater the probability of the target successfully attacking UAV is. Therefore, we define

$${\eta }_{A_t}={\phi }_{{\mathrm{U}}_t}+{\phi }_{{\mathrm{T}}_t}$$

(5)

to reflect the changes of the angle situation between the target and the UAV in the process of air combat confrontation, and there is $0\le {\eta }_{A_t}\le 2\pi$. The smaller the ${\eta }_{A_t}$ is, the more means that the UAV is in pursuit posture confrontation the target. In addition to angle, distance is also an important factor in air combat. Denote by $D_{min}$ the minimum distance that a UAV can reach to the target for safety. When $D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{min}$, it means that the target is in the blind zone of UAV radar detection, and the UAV has the risk of collision [38]. Thus, the distance situation of UAV in air combat can be defined as

$${\eta }_{D_t}=\frac{D_{max}-D_{{\mathrm{U}}_t{\mathrm{T}}_t}}{D_{max}-D_{min}},$$

(6)

where $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ is distance which can be as long as 50km in the air combat. The larger the ${\eta }_{D_t}$ is, the closer the distance between the UAV and the enemy is. Outside the $D_{min}$, the smaller the $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ and lag angle are, the greater the probability of success of UAV attack on the target is. Combining angle situation and distance situation, we define

$${\eta }_t=w_1{\eta }_{A_t}+w_2{\eta }_{D_t},$$

(7)

to evaluate the real-time comprehensive situation during UAV air combat, where $w_1$ and $w_2$ are scale factors, which represent the influence of different situation factors on UAV situation assessment. Moreover, since the value range of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ is very different, we have to balance the effects of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ by using $w_1$ and $w_2$.

Therefore, the air combat maneuver decision making problem of UAV can be regarded as an optimization problem,

$$\underset{\left[n_{x_{\mathrm{t}}},n_{z_t},{\mu }_t\right]\in \mathsf{\Lambda }}{max}\sum _{t=t_0}^{t_n}{\eta }_t\left(n_{x_{\mathrm{t}}},n_{z_t},{\mu }_t\right).$$

(8)

where $\mathsf{\Lambda }$ denotes a set of UAV maneuver control commands. ${\eta }_t\left(n_{x_{\mathrm{t}}},n_{z_t},{\mu }_t\right)$ means ${\eta }_t$ is the function of $n_{x_{\mathrm{t}}}$, $n_{z_t}$, and ${\mu }_t$, where $\left[n_{x_{\mathrm{t}}},n_{z_t},{\mu }_t\right]$ has the same meaning as (2). Thus (8) means maximize the sum of UAV situation from the beginning to the end of air combat. It is difficult to get the optimal solution because the objective function is complex high-order nonlinear. Next, we will use deep reinforcement learning to solve this problem.

3. Maneuver Decision Modeling by Deep Q Network

3.1. Reinforcement Learning

Reinforcement learning is a method for the agent to optimize maneuver strategy. The air combat maneuver decision making problem discussed in this paper belongs to the model-free reinforcement learning problem. Markov decision process (MDP) is usually used as the theoretical framework of model-free reinforcement learning, and the final objective of the reinforcement learning is to solve the MDP by deducing an optimum policy [39], which is described by quaternion array $\left[S,A,R,\gamma \right]$, where S represents state space, A represents action space, R represents reward function and $\gamma$ represents discount factor. Reinforcement learning uses the state action value function Q to evaluate the value of action taken in the current state [18], which is defined as

$$\begin{array}{cc}{Q}_{\pi }\left(s,a\right)& =E_{\pi }\left[G_t|s_t=s,a_t=a\right]\hfill \\ & =E_{\pi }\left[{\displaystyle \sum _{k=0}^{\infty }}{\gamma }^k{r}_{t+k+1}|s_t=s,a_t=a\right],\hfill \end{array}$$

(9)

where $s\in S$, $a\in A$, $r\in R$. In addition, in order to facilitate the calculation, the following simplified processing is usually implemented [40],

$$Q_{\pi }\left(s,a\right)=E_{\pi }\left[r_{t+1}+\gamma Q\left(s_{t+1},a_{t+1}\right)|s_t=s,a_t=a\right].$$

(10)

Reinforcement learning finds the optimal strategy ${\pi }_{{}^{\ast }}\left(s\right)$ by finding the optimal action value function $Q_{{}^{\ast }}\left(a|s\right)$, i.e.,

$$Q_{\ast }\left(s,a\right)=\underset{\pi }{max}{Q}_{\pi }\left(s,a\right).$$

(11)

As long as the maximum action value function is found, the corresponding strategy ${\pi }_{\ast }$ is the solution of the reinforcement learning problem. In order to solve the dimension disaster problem of (11), deep reinforcement learning algorithm [41] is proposed, and transforms table value learning into parameters fitting of the neural network, i.e.,

$$Q_{\pi }\left(s,a\right)=Q_{\pi }\left(s,a,\theta \right),$$

(12)

where $\theta$ is the parameters of neural network and $Q\left(s,a;\theta \right)$ is called online Q network. Therefore, the solution of reinforcement learning problem can be expressed as

$${\pi }_{{}^{\ast }}\left(s\right)=arg\underset{\pi }{max}{Q}_{\pi }(s,a,\theta ).$$

(13)

3.2. State Space

The state of UAV can quantitatively reflect the current air combat information. We consider designing the state space of UAV from the following three aspects: The first is the maneuver characteristics of UAV. The second is the relative situation between the UAV and the target. The third is target dynamic embedding prediction of the situation in real-time combat.

In this paper, we use the following 13 variables to form the state space, $v_{\mathrm{U}}$, ${\gamma }_{\mathrm{T}}$, ${\gamma }_{\mathrm{U}}$, ${\phi }_{\mathrm{U}}$, ${\phi }_{\mathrm{T}}$, $v_{\mathrm{U}}-v_{\mathrm{T}}$, ${\psi }_{\mathrm{T}}$, ${\psi }_{\mathrm{U}}$, $D_{\mathrm{UT}}$, $z_{\mathrm{U}}$, $z_{\mathrm{U}}-z_{\mathrm{T}}$, ${\psi }_{\mathrm{UT}}$, ${\gamma }_{\mathrm{UT}}$. ${\gamma }_{\mathrm{UT}}$ represents the angle between $P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}$ and the $oxy$ plane, and ${\psi }_{\mathrm{UT}}$ represents the angle between projection vector of $P_{{\mathrm{T}}_t}-P_{{\mathrm{U}}_t}$ on the $oxy$ plane and $ox$ axis as shown in Figure 2. In order to unify the range of each state variable and improve the efficiency of network learning, each state variable is normalized to a range, as shown in Table 1.

Table 1. The state space for the DQN model.

State	Definition	State	Definition	State	Definition
$s_1$	${\displaystyle \frac{v_{\mathrm{U}}}{v_{max}}}$	$s_2$	${\displaystyle \frac{{\gamma }_{\mathrm{U}}}{2\pi }}$	$s_3$	${\displaystyle \frac{{\phi }_{\mathrm{U}}}{2\pi }}$
$s_4$	${\displaystyle \frac{v_{\mathrm{U}}-v_{\mathrm{T}}}{v_{max}-v_{min}}}$	$s_5$	${\displaystyle \frac{{\gamma }_{\mathrm{T}}}{2\pi }}$	$s_6$	${\displaystyle \frac{{\phi }_{\mathrm{T}}}{2\pi }}$
$s_7$	${\displaystyle \frac{D_{\mathrm{UT}}}{D_{thres}}}$	$s_8$	${\displaystyle \frac{{\gamma }_{\mathrm{UT}}}{2\pi }}$	$s_9$	${\displaystyle \frac{v_{\mathrm{U}}}{v_{max}}}$
$s_{10}$	${\displaystyle \frac{{\psi }_{\mathrm{U}}}{2\pi }}$	$s_{11}$	${\displaystyle \frac{{\psi }_{\mathrm{U}}}{2\pi }}$	$s_{12}$	${\displaystyle \frac{z_{\mathrm{U}}}{z_{max}}}$
$s_{13}$	${\displaystyle \frac{z_{\mathrm{U}}-z_{\mathrm{T}}}{z_{max}-z_{min}}}$

$v_{max}$ and $v_{min}$ represent the maximum and minimum speed of UAV respectively. $z_{max}$ and $z_{min}$ represent the maximum and minimum safe altitude of UAV flight respectively. $Dthres$ is the distance threshold, and represents the starting distance of close combat. Therefore, the state space can be defined as $s=[s_1,s_2,\dots .,s_{13}]$.

3.3. Action Space

As mentioned above, $\left[n_{x_{\mathrm{t}}},n_{z_t},{\mu }_t\right]\in \mathsf{\Lambda }$ constitute the action space of UAV. Different deep reinforcement learning algorithms such as DDPG and DQN have different design methods for the action space. Due to the huge state space, using DDPG and other algorithms to train continuous maneuvering strategies will cause difficulty in neural network convergence, and the maneuver process of UAV can be regarded as a combination of some basic actions [42], thus this paper adds eight action commands on the basis of the basic air combat maneuver [43] divided by NASA, and finally discretizes the UAV action space into 15 actions, as shown in Figure 3 and Table 2. This approach reduces the difficulty of UAV training, and compared with the basic seven actions, this method can make the UAV carry out constant speed, acceleration and deceleration control in each direction, which is closer to the real flight mode of the UAV.

Figure 3. UAV maneuver library.

Table 2. Maneuver library.

No.	Maneuver	Control Values
		${\mathbf{n}}_{\mathbf{x}}$	${\mathbf{n}}_{\mathbf{z}}$	$\mathbf{\mu }$
$a_1$	forward maintain	0	1	0
$a_2$	forward accelerate	2	1	0
$a_3$	forward decelerate	−1	1	0
$a_4$	left turn maintain	0	8	$-arccos\left(1/\phantom{18}8\right)$
$a_5$	left turn accelerate	2	8	$-arccos\left(1/\phantom{18}8\right)$
$a_6$	left turn decelerate	−1	8	$-arccos\left(1/\phantom{18}8\right)$
$a_7$	right turn maintain	0	8	$arccos\left(1/\phantom{18}8\right)$
$a_8$	right turn accelerate	2	8	$arccos\left(1/\phantom{18}8\right)$
$a_9$	right turn decelerate	−1	8	$arccos\left(1/\phantom{18}8\right)$
$a_{10}$	upward maintain	0	8	0
$a_{11}$	upward accelerate	2	8	0
$a_{12}$	upward decelerate	−1	8	0
$a_{13}$	downward maintain	0	8	$\pi$
$a_{14}$	downward accelerate	2	8	$\pi$
$a_{15}$	downward decelerate	−1	8	$\pi$

The UAV selects an action $a\in A$ according to the state s and outputs it to the environment. After format conversion, the UAV is guided to fly according to the command,

$$\begin{array}{c}A=\left\{a_1,a_2,\dots ,a_m\right\}\subseteq \mathsf{\Lambda },m=15,\hfill \end{array}$$

(14)

$$\begin{array}{c}{a}_i=\left[n_x,n_z,\mu \right],i=1,2,\dots ,15.\hfill \end{array}$$

(15)

3.4. Reward Function

Reward function [44,45,46,47] is the feedback signal obtained by the agent in the process of interaction with the environment, which is used to evaluate the effect of the agent executing a certain action strategy. Therefore, reasonable design of reward function can effectively improve the convergence speed of the system [45]. The reward $r_t$ in this paper consists of two parts, which is defined as follow.

$$r_t=R_t+R_{{\eta }_t}.$$

(16)

In (16), $R_t$ is the evaluation of the final result of air combat, which defines as (17).

$$R_t=\left\{\begin{array}{cc}C,\hfill & \mathrm{if}\mathrm{UAV}\mathrm{wins},\hfill \\ -C,\hfill & \mathrm{if}\mathrm{Target}\mathrm{wins},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(17)

C is a constant and greater than 0. When the UAV meets the following conditions, it is considered that “UAV wins” is established and a success reward C is given. First, an optimal shooting distance threshold $D_{attack}$ based on expert experience is defined, and there is $D_{attack}\le D_{max}$. Then if the distance between UAV and the target meets $D_{min}<D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{attack}$ and the angle meets ${\phi }_{{\mathrm{U}}_t}\le {\phi }_m$ and ${\phi }_{{\mathrm{T}}_t}\ge \pi -{\phi }_m$ at the same time, it is considered that the UAV has the best shooting conditions against the target, and the “UAV wins” condition is established. Similarly, if the distance between UAV and the target satisfies $D_{min}<D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{attack}$ and the angle satisfies ${\phi }_{{\mathrm{T}}_t}\le {\phi }_m$ and ${\phi }_{{\mathrm{U}}_t}\ge \pi -{\phi }_m$ at the same time, the condition “Target wins” is established and we give the failure penalty $-C$.

$R_{{\eta }_t}$ in (16) is used for the real-time evaluation of maneuver decision making, which is defined as

$$R_{{\eta }_t}={\eta }_t-{\eta }_{t-1}=w_1\left({\eta }_{A_t}-{\eta }_{A_{t-1}}\right)+w_2\left({\eta }_{D_t}-{\eta }_{D_{t-1}}\right).$$

(18)

(18) indicates the change of situation of $\mathrm{UAV}$ during air combat. If $R_{{\eta }_t}>0$, the situation at time t is better than that at time $t-1$. The maneuver strategy $a_{t-1}$ adopted by UAV from $s_{t-1}$ to $s_t$ is reasonable, and we give a positive reward. On the contrary, if $R_{{\eta }_t}<0$, we give a negative penalty. $w_1$ and $w_2$ indirectly affect the maneuver decision making of UAV by influencing $r_t$. Considering the different importance of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ under different $s_t$, an evaluation method of maneuver decision making based on $w_1$ piecewise adjustment is proposed. In this paper, $w_1$ is set as a piecewise function,

$$w_1=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{D}_{{\mathrm{U}}_t{\mathrm{T}}_t}>D_{max},\hfill \\ W_1\hfill & \mathrm{if}{D}_{attack}\le D_{{\mathrm{U}}_t{\mathrm{T}}_t}\le D_{max},\hfill \\ W_2\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(19)

where $W_1$ and $W_2$ are constants, and $0<W_1<W_2$. When $D_{{\mathrm{U}}_t{\mathrm{T}}_t}>D_{max}$ holds, UAV should give priority to adjust $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ to quickly approach the target. When $D_{{\mathrm{U}}_t{\mathrm{T}}_t}\le D_{max}$ holds, UAV should consider adjusting ${\phi }_{{\mathrm{U}}_t}$ and $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ at the same time, and the smaller the $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ is, the larger the $w_1$ is. $w_2$ is a constant and there is no need to set a piecewise function for $w_2$ because the relative importance of $w_2$ will change with $w_1$.

Remark 1.

If we change one of $w_1$ and $w_2$, the influence of angle advantage and distance advantage on UAV maneuver decision will change relatively. Therefore, this paper discusses the design of $w_1$. The setting method of $w_1$ can consider the following three types: Fixed value, piecewise function as (17), or continuous function as $w_1\propto D_{{\mathrm{U}}_t{\mathrm{T}}_t}$. The fixed value represents that the influence of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ on UAV maneuver decision is fixed. The piecewise function represents the influence of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ on UAV maneuver decision, which is changed in a limited number of different cases. The continuous function represents the influence of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ on UAV maneuver decision, which changes in real-time according to the current state of UAV. In this paper, we set $w_1$ as a piecewise function with the following considerations. $r_t$ is a comprehensive evaluation of UAV at time t on ${\eta }_{A_t}$, ${\eta }_{D_t}$ and $R_t$ and these three contents are independent of each other. If $w_1\propto D_{{\mathrm{U}}_t{\mathrm{T}}_t}$, $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ and ${\phi }_{{\mathrm{U}}_t}$ are coupled with each other. Then with the change of $D_{{\mathrm{U}}_t{\mathrm{T}}_t}$, the change of $w_1$ represents the synchronous change of the importance of ${\eta }_{A_t}$ and ${\eta }_{D_t}$ to UAV maneuver decision making, which cannot reflect the different importance of the two advantage functions in different stages of air combat. In addition to the above theoretical analysis, this paper also gives the comparison results in the experimental stage. That is, $w_1$ is set as a fixed value, piecewise function and continuous function proportional to distance respectively, and analyzes the changes of the loss function in three cases, and further explains the rationality of $w_1$ as a piecewise function.

3.5. Priority Sampling and Network Training

In this paper, the DQN algorithm is used to realize the self-learning of UAV maneuver control strategy [41,48], and an improved priority sampling strategy is proposed to accelerate the training and learning process. Experience replay mainly includes two key steps of “experience storage” and “sampling replay”. Here, it is mainly to improve “sampling replay”. The basic idea is to assign a priority to each sample in the experience pool. When selecting experience, we prefer to choose the experience with high priority. First, the data in the experience pool is marked according to importance, that is, the greater the value of data to network training is, the more important it is, and the higher the corresponding priority is. Then sample the labeled data, that is, the higher the priority is, the greater the probability that the sample is extracted is. Finally, the extracted samples are used for the weighted training of the network. Next, we will introduce priority sampling in detail from three aspects: Sample labeling, sampling and network training.

$p_i$ is used to indicate the importance of the ith sample $(s_i,a_i,r_i,s_{i+1})$. A reasonable approach is to use the following TD error ${\delta }_i$ to assign $p_i$ [49],

$$y_i=r_i+\gamma \underset{a^{\prime }}{max}Q\left(s_{i+1},a^{\prime };{\theta }^{\prime }\right),$$

(20)

$${\delta }_i=y_i-Q\left(s_i,a_i;\theta \right).$$

(21)

where $y_i$ is called the target Q value, and ${\delta }_i$ represents TD error. Since $\left|{\delta }_i\right|\ge 0$, and the larger the $\left|{\delta }_i\right|$ is, the more important the sample i is [49], and the higher the probability of being sampled should be. In order to avoid accidental factors that cause $\left|{\delta }_i\right|$ of some samples to be too large and the sampling probability of some samples with lower priority to be close to 0 resulting in the decrease of sampling diversity, $\left|{\delta }_i\right|$ is limited in $\left[0–1\right]$ by using tanh function, i.e.,

$$p_i=tanh\left(\left|{\delta }_i\right|+\sigma \right),$$

(22)

where $\sigma$ is a positive number, so that $p_i=\sigma >0$ at ${\delta }_i=0$. Then the sampling probability of sample i is expressed as

$$P\left(i\right)=\frac{p_i^{\alpha }}{{\sum }_k{p}_k^{\alpha }},$$

(23)

where $\alpha$ is the priority factor, which is used to adjust the priority of the sample. The larger the $\alpha$ is, the larger the $P\left(i\right)$ will be. When $\alpha =0$, the above equation will degenerate into uniform sampling, and k is the number of samples.

Remark 2.

The definition of $P\left(i\right)$ is not unique. Two variants are proposed in the [49]. The second variant is $p_i=\frac{1}{rank\left(i\right)}$. $rank\left(i\right)$ represents the rank of sample i sorted according to $\left|{\delta }_i\right|$. Considering the simplicity of code implementation, we use $p_i=\left|{\delta }_i\right|+\sigma$.

Prioritized replay introduces bias because it changes distribution of sampled data in an uncontrolled fashion, and therefore changes the solution that the $Q\left(s,a;\theta \right)$ will converge to. Generally, the important sampling (IS) weights ${\lambda }_i$ can be used to correct this error,

$${\lambda }_i={\left(\frac{1}{N}·\frac{1}{P\left(i\right)}\right)}^{\beta },$$

(24)

where N represents the experience pool capacity and $\beta$ represents the compensation degree. If $\beta =1$, the non-uniform probability is fully compensated. Then update the network parameters by using ${\lambda }_i{\delta }_i$ instead of ${\delta }_i$. For stability reasons [50], we will standardize ${\lambda }_i$ as follow,

$${\lambda }_i=\frac{{\left(N·P\left(i\right)\right)}^{-\beta }}{{max}_j{\lambda }_j}.$$

(25)

After obtaining the sample data needed for network training through priority sampling, we input it into $Q\left(s,a,{\theta }^{\prime }\right)$ and $Q\left(s,a,\theta \right)$ to update $\theta$. DQN adjusts $\theta$ through gradient descent method during training, and the loss function after adding importance sampling weight ${\lambda }_i$ is

$$L\left(\theta \right)={\delta }_i{\lambda }_i,$$

(26)

and its gradient is

$$\frac{\partial L\left(\theta \right)}{\partial \theta }={\lambda }_i\frac{\partial Q(s_i,a_i;\theta )}{\partial \theta }.$$

(27)

Finally, in order to collect enough samples for network training, DQN algorithm uses $\epsilon -greedy$ strategy [41,48] to select actions, i.e.,

$$a_t=\pi \left(s\right)=\left\{\begin{array}{cc}\arg \underset{a\in A}{max}Q(s_t,a;\theta ),\hfill & \mathrm{if}\mathrm{num}>\epsilon ;\hfill \\ \mathrm{random}A,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(28)

where $\mathrm{num}$ is a random number of 0–1.

The above UAV BVR autonomous maneuver decision algorithm is summarized in the form of pseudo code as shown in Algorithm 1.

Algorithm 1 DQN with proportional prioritization

1: Initialize online network Q with random parameters $\theta$ Initialize target network $Q^{\prime }$ with random parameters ${\theta }^{\prime }$ Initialize replay buffer M, Initialize hyper-parameters $D_{max}$, $D_{min}$, $V_{max}$, $\gamma$, ${\phi }_{\mathrm{m}}$, $w_2$, $W_1$, $W_2$, $\beta$, $z_{min}$, $z_{max}$, $\sigma$, $D_{\mathrm{BVR}}$, $V_{min}$, $D_{\mathrm{WVR}}$, $\alpha$, a, b, C, k, K, $D_{attack}$. 2: for $episode=1$ to N do 3:       Initialize the initial state of air combat 4:       Receive initial observation state $s_1$ 5:       for $t=1$ to T do 6:             With probability $\epsilon$ select a random action $a_t$ 7:             Otherwise select $a_t={max}_{a}Q\left(s_t,a;\theta \right)$ 8:             UAV executes action $a_t$, and target executes action according to its policy 9:             Receive reward $r_t$ and observe new state $s_{t+1}$ 10:           Store transition $(s_t,a_t,r_t,s_{t+1})$ in M 11:           Sample a mini batch of N transition $(s_{t+1},a_t,r_t,s_{t+1})$ from M with priority $P\left(i\right)=p_i^{\alpha }/{\sum }_k{p}_k^{\alpha }$ 12:           Compute importance-sampling weight ${\lambda }_i={\left(N·P\left(i\right)\right)}^{-\beta }/{max}_j{\lambda }_j$ 13:           Set $y_i=r_i+\gamma {max}_{a^{\prime }}Q\left(s_{i+1},a^{\prime };{\theta }^{\prime }\right)$ 14:           Compute TD-error ${\delta }_i=y_i-Q\left(s_i,a_i;\theta \right)$ 15:           Update transition priority $p_i^{}\leftarrow \left|{\delta }_i\right|$ 16:           Perform a gradient descent step on ${\lambda }_i{\left(y_i-Q\left(s_i,a_i;\theta \right)\right)}^2$ with respect to the network parameters $\theta$ 17:           Every K steps reset ${\theta }^{\prime }=\theta$ 18:     end for 19: end for

The current state of the UAV is $s_t$. The online Q network selects and executes the action $a_t$ based on $\epsilon -greedy$ to transfer the UAV to the next state $s_{t+1}$, and obtain the reward $r_t$. Save $(s_t,a_t,r_t,s_{t+1})$ to the experience pool, and repeat the above steps until the number of samples in the experience pool meets the requirements. Select samples from the experience pool according to the priority $P\left(i\right)$ to train the neural network, and calculate the importance sampling weight ${\lambda }_i$ of the selected samples. Use these samples to train the network parameters. That is, first, input $s_t$ into the online Q network, and input $s_{t+1}$ into the target Q network. Second, calculate the weighted mean square error according to (25), and use (27) to update the online Q network’s parameters. At the same time, TD error ${\delta }_i=y_i-Q\left(s_i,a_i;{\theta }^{\prime }\right)$ is obtained. According to ${\delta }_i$, the priority of the selected samples in the experience pool is updated, and the target network parameters are updated after a certain number of times or rounds of training. In the training process, the $\epsilon$ should be increased slowly, so that the UAV can choose the optimal action according to the value function with greater probability. When the error is close to 0 or there is no obvious change, the training is stopped, and the trained neural network is saved to obtain the air combat maneuver strategy of dual-aircraft formation

$$\pi {\left(a|s\right)}_{\mathrm{U}}=arg\underset{a\in A}{max}Q\left(s,a,\theta \right).$$

(29)

4. Olive Formation Air Combat as an Example

4.1. Task Description

This paper takes a typical two UAVs olive formation combat scene as an example. As shown in Figure 4, two UAVs perform tracking, defense and attacking tasks.

Figure 4. Task description.

UAV air combat can be divided into three stages: Take-off stage, BVR tracking stage and WVR attack stage. During the take-off stage, the two UAVs took the enemy aircraft as the target and continuously accelerated to the target direction. In the BVR tracking stage, the nose of one UAV faces the target and the nose of the other UAV backs the target. Two UAVs fly in olive formation to maintain the continuity of attack and defense. In this stage, the trajectory formed by the UAV from flying towards the target to flying back to the target and then flying towards the target is a circle or ellipse, so it is called olive formation. The process of WVR combat is also called dog fight. The two UAVs find the best angle and distance to attack the enemy and avoid entering the attack range of the target at the same time. We assume that the UAVs can accurately obtain any information they want, and then control the speed, yaw and roll through autonomous maneuver decision making, so as to track, defend and attack the target.

There are three problems to be considered for dual-aircraft autonomous maneuver control decisions: First, how to conduct inter aircraft collision avoidance? Second, how to make the two UAVs form an olive formation to maintain the continuity of attack and defense? Third, how to make the two UAVs maneuver independently to realize BVR tracking and short range attack? Next, we solve the above three problems by designing a hybrid autonomous maneuver strategy of obstacle avoidance, formation and confrontation.

4.2. Collision Avoidance and Formation Strategy

As described in Section 4.1, $\mathfrak{A}=\left\{{\mathrm{U}}_1,{\mathrm{U}}_2\right\}$ is used to represent our UAV set. We use U to represent any UAV in $\mathfrak{A}$, and use $\tilde{\mathrm{U}}$ to represent U’s friendly aircraft. Denote by $D_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}$ the distance between U and $\tilde{\mathrm{U}}$ at time t. First, if $D_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}<D_{min}$, no matter whether the UAV meets the firing conditions or not, it must avoid collision between UAVs. When $D_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}<D_{min}$, the fast and effective obstacle avoidance method between UAVs is to change the flight altitude between itself and friendly aircraft. Therefore, the collision avoidance strategy is as follows,

$$O{\left(a|s\right)}_{\mathrm{U}}=\left\{\begin{array}{ccc}{a}_{11},& \mathrm{if}& D_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}<D_{min},{\mathrm{U}}^z\ge {\tilde{\mathrm{U}}}^z,\\ a_{14},& \mathrm{if}& D_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}<D_{min},{\mathrm{U}}^z<{\tilde{\mathrm{U}}}^z;\end{array}\right.$$

(30)

${\mathrm{U}}^z$ and ${\tilde{\mathrm{U}}}^z$ represent the heights of $\mathrm{U}$ and $\tilde{\mathrm{U}}$ respectively. The UAV with a higher altitude adopts the accelerated ascent strategy, and the UAV with a lower altitude adopts the accelerated descent strategy. Second, in order to ensure that the UAVs can realize olive formation flight, and considering the flight characteristics and ease of control of the UAV, we use continuous uniform left turn to realize circling flight,

$$H{\left(a|s\right)}_{\mathrm{U}}=a_4.$$

(31)

However, strategy (31) is not enough. We also need to let the two UAVs know when to start flying in olive formation, who starts first and how to switch. It is assumed that ${\mathrm{U}}_1$ is the leader and ${\mathrm{U}}_2$ is the wingman. In the BVR tracking stage, the leader first uses the strategy (31). It can be seen from (2) that the decision making time required by the UAV for a circle is

$$T_{olive}=\frac{2\pi vcos\gamma }{g{n}_{z}sin\mu }.$$

(32)

Finally, the maneuver control in takeoff stage, BVR tracking stage and WVR attack stage is realized by $\pi {\left(a|s\right)}_{\mathrm{U}}$.

Therefore, the maneuver strategy of UAV at time t can be expressed by the following equation.

$$\mathsf{\Pi }{\left(a|s\right)}_{\mathrm{U}}=\left\{\begin{array}{cc}O{\left(a|s\right)}_{\mathrm{U}},\hfill & \mathrm{if}{D}_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}<D_{min};\hfill \\ H{\left(a|s\right)}_{\mathrm{U}},\hfill & \mathrm{if}{D}_{{\mathrm{U}}_t{\tilde{\mathrm{U}}}_t}\ge D_{min},\hfill \\ \hfill & D_{\mathrm{WVR}}\le D_{{\mathrm{U}}_t{\mathrm{T}}_t}\le D_{\mathrm{BVR}},\hfill \\ \hfill & \mathsf{\Pi }{\left(a|s\right)}_{\tilde{\mathrm{U}}}\ne H{\left(a|s\right)}_{\tilde{\mathrm{U}}};\hfill \\ \pi {\left(a|s\right)}_{\mathrm{U}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(33)

$D_{\mathrm{BVR}}$ represents the distance threshold of BVR air combat, and $D_{\mathrm{WVR}}$ represents the distance threshold of WVR air combat. If $D_{{\mathrm{U}}_t{\mathrm{T}}_t}\ge D_{\mathrm{BVR}}$, our UAV belongs to take-off stage. If $D_{\mathrm{WVR}}\le D_{{\mathrm{U}}_t{\mathrm{T}}_t}\le D_{\mathrm{BVR}}$, our UAV belongs to BVR tracking stage. If $D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{\mathrm{WVR}}$, our UAV belongs to the WVR air combat stage.

The autonomous maneuver decision algorithm of the dual-UAV olive formation is sketched in Algorithm 2.

Algorithm 2 Maneuver strategy of two UAVs olive formation in air combat

1: Load trained neural network $Q\left(s,a,\theta \right)$. 2: Initialize the state of the leader and wingman $\left(s_{\mathrm{U}1,0},s_{\mathrm{U}2,0},s_{\mathrm{T},0}\right)$, 3: Initialize target maneuver strategy $\pi {\left(a|s\right)}_{\mathrm{T}}$. 4: for $step=1$ to $maxstep$ do 5:       for U in $\mathfrak{A}$ do 6:             Calculate $D_{\mathrm{UT}}$ 7:             Calculate $D_{\mathrm{U}\tilde{\mathrm{U}}}$ 8:             Execute $a_t=\mathsf{\Pi }{\left(a|s\right)}_{\mathrm{U}}$ 9:             $\left[\Delta v,\Delta \gamma ,\Delta \psi \right]$ is obtained according to (2) 10:           $\left[\Delta x,\Delta y,\Delta z\right]$ is obtained according to (1) 11:           Get the next state $s_{\mathrm{U},\mathrm{t}+1}$ 12:           $s_{\mathrm{U},\mathrm{t}}=s_{\mathrm{U},\mathrm{t}+1}$ 13:     end for 14:     The target moves to the next state $s_{\mathrm{T},t+1}$ according to the strategy $\pi {\left(a|s\right)}_{\mathrm{T}}$ 15:     if $D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{attack},{\phi }_{{\mathrm{U}}_t}\le {\phi }_m,{\phi }_{{\mathrm{T}}_t}\ge \pi -{\phi }_m$ then 16:           UAVs win 17:           break 18:     end if 19:     if $D_{{\mathrm{U}}_t{\mathrm{T}}_t}<D_{attack},{\phi }_{{\mathrm{T}}_t}<{\phi }_m,{\phi }_{{\mathrm{U}}_t}>\pi -{\phi }_m$ then 20:           target win 21:           break 22:     end if 23:     $s_{\mathrm{T},\mathrm{t}}=s_{\mathrm{T},\mathrm{t}+1}$ 24: end for

First, the trained neural network $Q\left(s,a,\theta \right)$ and the maneuver strategy of the target $\pi {\left(a|s\right)}_{\mathrm{T}}$ are loaded. Initialize the state $\left(s_{{\mathrm{U}}_1,0},s_{{\mathrm{U}}_2,0},s_{\mathrm{T},0}\right)$ of UAVs and target, where $s_{{\mathrm{U}}_1,0}$ and $s_{{\mathrm{U}}_2,0}$ are the initial state of our two UAVs respectively, and $s_{\mathrm{T},0}$ is the initial state of target. For each UAV, the distance $D_{\mathrm{UT}}$ and the distance $D_{\mathrm{U}\tilde{\mathrm{U}}}$ are calculated, and then the maneuver strategy $a_t$ is obtained according to (33). If the distance between UAV and the enemy is less than $D_{min}$, the collision avoidance strategy (30) is implemented. If the UAV is in the takeoff stage, the UAV selects the maneuver strategy according to (29). If both UAVs enter the BVR tracking state, the leader first executes the strategy (31), and the number of execution steps is obtained according to (32), while the wingman continues to select $a_t$ according to (29). When the leader completes a circle according to (32) and (31), the flight strategies of the leader and the wingman are exchanged. In the BVR tracking stage, our two UAVs constantly change their maneuver strategies to maintain the continuity of attack and defense. In the WVR attack stage, both UAVs use (29) to complete the short-range combat. If either of the enemy and our UAVs satisfies (17), the air combat ends.

5. Simulation

This paper uses Python language to establish the air combat environment model of dual-aircraft olive formation tracking and attacking, and establishes the DQN network model based on the PyTorch module.

5.1. Simulation Setup

The air combat environment parameters are set as shown in Table 3.

Table 3. Design of the simulation parameters.

Variable	Value	Variable	Value	Variable	Value
$D_{max}$	1000	$D_{min}$	200	$D_{attack}$	500
C	10	$V_{max}$	300	b	10
$\gamma$	0.9	${\phi }_{\mathrm{m}}$	45	$w_2$	20
$\beta$	0.4	$V_{min}$	90	$z_{min}$	1000
$z_{max}$	12,000	$D_{threshold}$	10,000	a	5
$W_1$	30	$W_2$	40	$\alpha$	0.6
k	5000	K	300	$D_{\mathrm{BVR}}$	20,000
$\sigma$	0.01	$D_{\mathrm{WVR}}$	10,000

The parameters in the DQN model are set as follows. According to the definition of state space and maneuver library, it is obvious that DQN has 13 input states and 15 output Q values. The online Q network and the target Q network are constructed using a fully connected network. The network has three hidden layers, 512, 1024 and 512 units respectively. The output layer has no activation function, and the other layers are tanh layers. The learning rate is 0.001 and the discount coefficient is 0.9. The size of the experience pool is 5000, the number of samples taken in batch during training is 64, and the target network is updated every 300 steps. In the process of air combat simulation, the decision cycle T is set to 1s, and one set contains 100 decision steps. If any of the following conditions are met: The height of UAV is greater than $z_{\max }$ or less than $z_{\min }$, or any UAV meets (17). The round of training is completed.

In order to verify the effectiveness of the DQN algorithm based on priority sampling and dynamic adjustment of scale factor proposed in this paper, we compare the loss values in the training process of the DQN network under the following four conditions while ensuring the same initial conditions of the simulation: $w_1$ is set as a fixed value, piecewise function and continuous function proportional to distance respectively, and introduce the priority sampling. The loss value represents the difference between the online network and the target network. The larger the loss value is, the larger the network optimization space at this stage is. The greater the fluctuation of loss value is, the worse the convergence performance of the neural network is.

5.2. Simulation Results

Figure 5 shows the results of two UAVs formation air combat. Red and green are our two UAVs, and blue is the target. The initial positions of the three UAVs are fixed, namely (0,0,3000), (20,000,20,000,4000), (500,500,3000), and the heading angle and pitch angle are initialized randomly.

Figure 5. Combat result.

The target moved in a straight line at a constant speed. Our two UAVs take different maneuver control decisions in different air combat stages, and the two UAVs cooperate to complete the tracking, defense and attack tasks of the target. During the takeoff stage, our two UAVs tracked the target from a distance of about 30,000 m. In the BVR stage, the leader (red) first executes the strategy $H{\left(a|s\right)}_{\mathrm{U}}$, while the wingman (green) continues to track the target. When the leader hovers for one circle, its strategy is changed to track the target, and the wingman switches the flight strategy to $H{\left(a|s\right)}_{\mathrm{U}}$. The two UAVs cooperate to maintain the continuity of attack and defense. When the distance between UAV and target is less than $D_{\mathrm{BVR}}$, two UAVs enter the close attack phase. Figure 5 shows that our two UAVs attack the target from the rear of the target. Figure 6, Figure 7, Figure 8 and Figure 9 more clearly show the changes of various parameters of both sides in the process of air combat in Figure 5. The abscissas of Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 are the decision time, and the ordinates are different air combat parameters.

Figure 6. Change of the distance in the process of air combat.

Figure 7. Change of the lag angle in the process of air combat.

Figure 8. Change of the single step reward in the process of air combat.

Figure 9. Change of the cumulative reward in the process of air combat.

Figure 6 shows the change of the distance between the two sides in the process of air combat, where red is the change curve of the distance between the leader and the target, and green is the change curve of the distance between the wingman and the target, and black is the change of the distance between our two UAVs. The ordinate is the distance in meters. It can be seen that the distance between the UAV and the target will increase when flying in circles, and the distance between the UAV and the target will continue to shorten in the stages of BVR tracking and WVR air combat. Figure 7 shows the change of the lag angle of the UAVs of our two UAVs, where red is the change of the lag angle between the leader UAV and the target, and green is the change of the lag angle between the wingman UAV and the target. The ordinate is the lag angle in radians. It can be seen that the lag angle of our UAV changes from 0 to $\pi$ when flying in circles, and the lag angle remains at a low level during the pursuit and close combat.

Figure 8 shows the change of the reward function of our two UAVs in the whole process of air combat, in which red is the leader and green is the wingman. The abscissa is $r_t$. It can be seen that our UAV is flying in circles $r_t$ fluctuated obviously, and the situation of pursuit and close combat remained positive. When the target enters the attack range of UAV, the angle advantage function is added, and our situation rises obviously. Figure 9 shows the change of cumulative $r_t$, which also shows an upward trend as a whole. Figure 10 shows the variation diagram of loss values during training when $w_1$ are $w_1=-20$ and piecewise function (19), in which red is $w_1=-20$. Figure 11 shows the variation diagram of loss values during training when $w_1$ are $w_1\propto D_{{\mathrm{U}}_t{\mathrm{T}}_t}$ and piecewise function (19), in which red is $w_1\propto D_{{\mathrm{U}}_t{\mathrm{T}}_t}$.

Figure 10. Comparison of loss with different forms of scale factors 1.

Figure 11. Comparison of loss with different forms of scale factors 2.

Figure 12 shows the error comparison diagram of introducing priority sampling and not introducing priority sampling on the premise that $w_1$ is a piecewise function (19). The blue curve represents the DQN algorithm considering priority sampling, and the red curve represents the DQN algorithm without priority sampling. The abscissas of the above three figures represent training steps, and the ordinates represent loss values, and the comparison shows the changing trend of loss in the whole training process. In addition, because the amount of data to be presented is too large, in order to avoid affecting the clarity of the display results, and on the premise of not affecting the loss change trend, Figure 10, Figure 11 and Figure 12 choose to record the loss value every 300 training times. It is not difficult to see from the above figures that the fluctuation amplitude and range of the red curve are much larger than that of the blue curve. The network converges better when $w_1$ is a piecewise function. The priority sampling can effectively improve the learning efficiency of agents and accelerate the convergence speed of the neural network.

Figure 12. Comparison of convergence results before and after priority sampling.

In summary, by modeling and simulation, this paper solves the following problems that exist in the state of the art: (i) Dimension explosion solved by discretizing the air combat state space to be finite and using neural network to learn the decision making model; (ii) Sparse and delayed reward solved by designing a real-time reward function based on situation assessment; (iii) Slow convergence solved by using improved priority sampling strategy; and (iv) Inadaptation to the real air combat maneuver control solved by incorporating the real UAV dynamic model and the comprehensive situation assessment model which is verified in the classical two agents olive formation scenario.

6. Conclusions

Based on reinforcement learning theory, an improved maneuver decision algorithm for UAV autonomous air combat is proposed in this paper. First, the UAV dynamic model and situation assessment model are established, and the UAV state space and action space are improved to solve the dimension explosion problem and make the UAV maneuver more flexible. Second, aiming at the problems of delayed reward and poor guidance ability, a reward function design method based on adaptive adjustment of the relative situation and the scale factor is proposed. Third, an improved priority sampling strategy is proposed to speed up the learning rate. Fourth, based on the dual-UAV olive formation task, a hybrid maneuver strategy of collision avoidance, formation and confrontation is proposed to realize dual-UAV cooperative autonomous air combat decision making. The simulation results show that the improved method can effectively improve the efficiency of the UAV learning confrontation maneuver strategy, and the UAV air combat maneuver decision model based on deep reinforcement learning can realize strategy with self-learning. The improved deep reinforcement learning method has a faster training speed and a more stable effect.