Scalable Pursuit–Evasion Game for Multi-Fixed-Wing UAV Based on Dynamic Target Assignment and Hierarchical Reinforcement Learning

Highlights

What are the main findings?

• A hierarchical collaborative game framework combining hierarchical reinforcement learning methods and target allocation methods is proposed for autonomous pursuit–evasion.
• The hierarchical reinforcement learning method based on trajectory prediction and stable auxiliary gradients effectively improves the win rate and generates a smooth flight path.

What are the implications of the main findings?

• The simulation of various large-scale pursuit–evasion games demonstrates the advantages of this framework in terms of training time and large-scale scalability.
• The hierarchical reinforcement learning method based on trajectory prediction and stable auxiliary gradients provides a feasible approach for application to real UAVs.

Abstract

The unmanned aerial vehicle (UAV) pursuit–evasion game is the fundamental framework for promoting autonomous decision-making and collaborative control of multi-UAV systems. Faced with the limitations of current deep reinforcement learning methods in terms of transferability and generalization for scalable multi-fixed-wing UAV pursuit–evasion game scenarios, this paper proposes a hierarchical collaborative pursuit–evasion game framework based on target allocation and hierarchical reinforcement learning. The framework comprises three layers: target allocation layer, maneuver decision-making layer, and flight control layer. The target allocation layer employs a dynamic target assignment method based on a dynamic value adjustment mechanism, decomposing the multi-vs.-multi pursuit–evasion game into several one-vs.-one confrontations. The maneuver decision-making layer utilizes a maneuver decision-making method based on trajectory prediction and hierarchical reinforcement learning to generate adversarial maneuver commands. The flight control layer adopts a stable gradient-assisted reinforcement learning flight controller to ensure stable UAV flight. Comparisons with other algorithms across 3V3, 6V6, 9V9, and 12V12 scenarios demonstrate that the proposed method achieves high win rates in diverse game scales. The comparison results also demonstrate the advantages of the framework proposed in this paper in terms of training efficiency and large-scale scalability.

1. Introduction

With the intelligence level improvement of unmanned aerial vehicles (UAVs), they are currently used widely in many fields, such as regional search and rescue [1], relaying communications [2], and disaster relief [3]. Among them, the protection of civil airport airspace is a key issue of people’s livelihood security that needs to be urgently addressed at present. Illegal intrusion of UAVs may interfere with flight take-offs and landings and cause safety accidents. Using fixed-wing drones to drive away or capture illegal intrusion UAVs has become a new response measure [4]. UAV pursuit–evasion games provide a basic framework to promote cooperative autonomous decision-making of multi-UAV system. Through a pursuit–evasion game, a fixed-wing UAV can effectively maneuver to the rear of the opponent UAV and capture it. The key in this process is maneuver decision-making, and reinforcement learning shows great potential in this aspect.

In recent years, deep reinforcement learning has achieved significant accomplishments in various games, such as Alpha Go [5], OpenAI Five [6], AlphaStar for StarCraft [7], Suphx for Mahjong [8], and DeepNash for Stratego [9]. By modeling the pursuit–evasion game process as a Markov process, single-agent reinforcement learning has been applied to one-to-one pursuit–evasion games, such as approximate dynamic programming [10], Proximal Policy Optimization (PPO) [11], Deep Deterministic Policy Gradient (DDPG) [12], Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3) [13], and Soft Actor Critic (SAC) [14]. A multi-UAV pursuit–evasion game is a scenario in which two or more UAVs engage in a game confrontation. Reinforcement learning methods are emerging as highly effective approaches to solving this problem [15].

Multi-agent reinforcement learning (MARL) has made remarkable progress in the field of multi-UAV collaboration and gaming. For a multi-UAV game, reference [16] proposed a MARL method based on the leadership–follower architecture, which improves the collaborative efficiency of multiple aircraft through hierarchical strategy optimization. However, this method fails to solve the problem of the sharp increase in training time in large-scale scenarios. Qu et al. improved the value decomposition mechanism of the QMIX algorithm, which enhanced the stability of multi-drone swarm games. However, in scenarios with more than 10 drones, the win rate still decreased significantly [17]. For multi-UAV pursuit and evasion, Lei et al. adopted an attention mechanism to enhance the credit allocation capability of MAPPO, alleviating the performance degradation caused by scale expansion, but did not consider the impact of flight trajectory oscillation on practical applications [18]. Although these latest studies have optimized the synergy effect in specific scenarios, they still have not simultaneously addressed the two core pain points of large-scale adaptability and flight stability.

The collaborative decision-making of heterogeneous UAVs/Unmanned Ground Vehicles (UGVs) in complex three-dimensional environments is also a current research hotspot. Reference [19] has realized a pursuit and evasion game between UAVs and UGVs in a three-dimensional complex environment. The communication constraints and autonomy framework design of multi-UAV systems are the key bottlenecks in their practical application. Recent studies have pointed out that communication delays and link interruptions in high-dynamic scenarios can lead to the failure of collaborative strategies, and the balance between distributed autonomy and centralized collaboration is the core to solving this problem [20]. Most of the existing methods optimize the performance of a single module through communication fault-tolerant algorithms but lack robust design for the entire chain of collaborative decision-making—flight control. However, most of the existing heterogeneous collaborative research focuses on task division scenarios, with insufficient adaptability to high-dynamic pursuit and evasion game scenarios. The differences in maneuverability characteristics between fixed-wing and rotorcraft UAVs (such as speed and overload limits) make it difficult to balance the real-time performance and effectiveness of collaborative strategies.

Compared to the aforementioned games, the challenges encountered in multi-UAV pursuit–evasion games are as follows:

(1) Flight trajectory oscillation: Hierarchical RL is commonly used to decouple maneuver confrontation and flight control, but RL-based flight controllers often induce violent trajectory oscillations during high-overload engagements, restricting practicality.

(2) Poor scalability across UAV scales: Fixed network architectures of multi-agent RL (MARL) methods (e.g., MAPPO, QMIX) require retraining for different-scale scenarios, and their win rates and training efficiency degrade sharply as the number of UAVs increases, due to credit allocation issues and exponential growth in sample/training time.

In UAV pursuit–evasion game scenarios, UAVs must simultaneously master stable flight while learning to defeat opponents and ensure survival, significantly increasing the complexity of maneuver strategy acquisition. Consequently, hierarchical reinforcement learning approaches are proposed for pursuit–evasion game. These typically consist of a top-level tactical selector and a bottom-level controller [21,22,23]. Generally, the top-level strategy selects between offensive and defensive tactics [23] or chooses discrete maneuvers from a finite set [21], while the bottom-level strategy controls the maneuvering flight. A hybrid action hierarchical architecture is used in the literature [24], where the continuous layer controls maneuvers and the discrete layer assigns targets. A hierarchical structure is used in the literature [25], where each agent’s virtual targets are generated by the upper layer and maneuver commands are generated by the lower layer.

Reinforcement learning controllers employed in these studies often focus solely on reaching desired states without considering flight path smoothness, frequently resulting in violent oscillations during high-overload engagements.

The second challenge lies in the applicability of deep reinforcement learning methods across scenarios with varying numbers of UAVs. Current multi-agent reinforcement learning (MARL) has demonstrated remarkable performance in multi-UAV cooperative pursuit–evasion games; the representative ones include the MAPPO algorithm [26] based on policy gradient and the QMIX algorithm [27] based on value decomposition. Reference [28] proposed a multi-agent reinforcement learning algorithm in response to the sparsity and instability of sample data in reinforcement learning algorithms, which demonstrated better training results. Reference [25] enhances win rates by applying hierarchical reinforcement learning to a multi-UAV coordinated game. While these papers demonstrate strong performance of multi-agent reinforcement learning, they require retraining for different-scale scenarios, limiting the method’s generalizability. Reference [29] has developed a Gaussian-enhanced MARL framework for developing scalable avoidance strategies in dynamic tracking scenarios, but it can only be used for avoidance and not for tracking. Reference [30] addresses the scalability issue in multi-agent reinforcement learning by parameterizing actor and critic networks using attention-based neural network structures. However, it exhibits a significant decline in win rates as the scale increases. The monotonicity constraint of multi-agent reinforcement learning based on value decomposition fails when the scale increases, and the credit allocation variance of the multi-agent reinforcement learning algorithm based on policy gradient gradually increases with the increase in scale [31]. Additionally, all these methods suffer from the problem that training time and sample size for multi-agent reinforcement learning increase exponentially with scale, limiting their application to large-scale pursuit–evasion game scenarios.

To address the two challenges outlined above, and effectively meet the protection requirements of civil airport airspace, this paper proposes a hierarchical collaborative pursuit–evasion game framework. It decomposes the multi-UAV collaborative pursuit–evasion game problem into three layers: target allocation, maneuver decision-making, and flight control. The target allocation layer performs dynamic target assignment. The maneuver decision-making layer generates maneuver commands using deep reinforcement learning based on the assigned targets. The flight control layer controls UAV flight according to the maneuver commands. The main contributions of this paper can be summarized as follows:

(1) A modular hierarchical framework that decomposes multi-UAV pursuit–evasion into target allocation, maneuver decision-making, and flight control layers, resolving the scalability bottleneck of MARL methods.

(2) A stable auxiliary gradient (SAG)-enhanced RL flight controller, which introduces angular acceleration constraints to eliminate trajectory oscillations without sacrificing control accuracy.

(3) A trajectory prediction-guided hierarchical RL maneuver decision-making method, using polynomial fitting to predict opponent positions and shape pre-tracking rewards, significantly improving one-on-one confrontation win rates.

(4) Dynamic target allocation algorithm based on situation advantage and threat level, with a dynamic value adjustment mechanism to achieve rapid, coordinated multi-target assignment and avoid resource concentration.

2. Problem Modeling

2.1. Game Model of Pursuit–Evasion with Multi-Fixed-Wing UAV

The scenario of this article is civil aviation airspace protection. There is an illegal UAV swarm intrusion, and our side needs to dispatch fixed-wing UAVs to engage in a pursuit–evasion game. The multi-fixed-wing UAV pursuit–evasion game refers to a scenario where two or more fixed-wing drones engage in maneuvering confrontation within space. By maneuvering around to the opponent’s tail and pointing the nose at the opponent, they then use a capture net or other ways to capture the opponent’s drone. During the process, it is necessary to simultaneously maneuver to prevent the opponent’s drone from reaching the tail of one’s own UAV. A schematic diagram of the game scenario is shown in Figure 1, where the orange cone represents the captured area of the UAV, and the green cone represents the capture area of the UAV. In this article, the range of the captured area is set to an angle of less than 20 degrees, the capture area is set to an angle of less than 4 degrees, and the distance is less than 4 km. The parameters are derived from the physical characteristics of fixed-wing unmanned aerial vehicles (UAVs) and the requirements of actual combat scenarios. Moreover, 4 km is the effective detection distance of the on-board sensors of small fixed-wing UAVs (such as electro-optical pods), 4° is the effective aiming angle for capture actions (such as launching capture nets), and 20° is the dangerous angle threshold for being locked by the enemy. These settings do not change with the number of drones, ensuring the consistency of the evaluation criteria in scenarios of different scales.

Note that the current study assumes noise-free state information (e.g., position, velocity, attitude) and no communication latency/loss. This simplification is intended to focus on validating the core hierarchical framework’s scalability and cooperative decision-making effectiveness, isolating the performance of dynamic target allocation, trajectory prediction-guided maneuvering, and stable flight control from non-ideal environmental interference. This is a standard foundational research paradigm in multi-UAV system optimization [32,33], and the proposed framework is designed to support subsequent integration of robustness mechanisms for real-world deployment.

2.2. Fixed-Wing UAV Model

This paper adopts JSBSIM for aircraft simulation. JSBSIM is a nonlinear six-degrees-of-freedom aircraft simulation platform. Its authenticity has been verified by NASA. At present, JSBSIM has been widely applied in academic and industrial research [34].

$$\begin{array}{l}\dot{x}=u\cos \theta \cos \psi +v(\sin \varphi \sin \theta \cos \psi -\cos \varphi \sin \psi )+w(\cos \varphi \sin \theta \cos \psi +\sin \varphi \sin \psi )\\ \dot{y}=u\cos \theta \sin \psi +v(\sin \varphi \sin \theta \sin \psi +\cos \varphi \cos \psi )+w(\cos \varphi \sin \theta \sin \psi -\sin \varphi \cos \psi )\\ \dot{z}=u\sin \theta -v\sin \varphi \cos \theta -w\cos \varphi \cos \theta \end{array}$$

(1)

$$\begin{array}{l}\dot{V}={\displaystyle \frac{u\dot{u}+v\dot{v}+w\dot{w}}{V}}\\ \dot{\alpha }={\displaystyle \frac{u\dot{w}-w\dot{u}}{u^2+w^2}}\\ \dot{\beta }={\displaystyle \frac{\dot{v}V-v\dot{V}}{V^2\cos \beta }}\end{array}$$

(2)

$$\begin{array}{l}\dot{p}={\displaystyle \frac{I_{xz}(I_x-I_y+I_z)pq+(I_z{I}_y-I_z^2-I_{xz}^2)qr+I_{z}L+I_{xz}N}{I_x{I}_z-I_{xz}^2}}\\ \dot{q}={\displaystyle \frac{M+(I_z-I_x)pr+(r^2-p^2)I_{xz}}{I_y}}\\ \dot{r}={\displaystyle \frac{I_{xz}L+I_{z}N+(I_x^2+I_z^2-I_x{I}_y)pq-I_{xz}(I_x-I_y+I_z)qr}{I_x{I}_z-I_{xz}^2}}\end{array}$$

(3)

$$\begin{array}{l}\dot{\varphi }=p+\tan \theta (r\cos \varphi +q\sin \varphi )\\ \dot{\theta }=q\cos \varphi -r\sin \varphi \\ \dot{\psi }=(r\cos \varphi +q\sin \varphi )/\cos \theta \end{array}$$

(4)

where $\varphi ,\theta ,\psi$ are the roll, pitch, and yaw flight attitude angles, respectively. $p,q,r$ are the angular velocity in the body coordinate system. $x,y,z$ are the aircraft position in the ground coordinate system. $u,v,w$ are the velocity components in the x, y, and z directions, respectively. $V,\alpha ,\beta$ are the velocity, angle of attack, and angle of sideslip, respectively.

Since this paper adopts JSBSIM, which includes actuator simulation, the actions output by the neural network in this paper are rudder deviation instructions. These instructions will pass through the actuator stage and then be input into the aircraft simulation model, which is also a manifestation of the high fidelity of JSBSIM. Moreover, due to the adoption of high-fidelity aerodynamics in the model simulation of JSBSIM, stall situations may also occur in flight control simulation. The impact of environmental stroke was not taken into account in this article.

The controls are $u=[{\delta }_e,{\delta }_a,{\delta }_r,{\delta }_T]$, including elevator, aileron, rudder command, and throttle command. The UAV coordinate system and angle definition are shown in Figure 2.

2.3. MDP Process

A Markov Decision Process is formulated by the tuple $<S,A,r,P,{\rho }_0,\gamma ,T>$ [35]. Here, S denotes the state space, A denotes the action space, the transition probability function $P:S\times A\times S\to [0,1]$ and the reward function $r:S\times A\to ℝ$ describe the dynamics of the environment, ${\rho }_0:S\to [0,1]$ describes the initial state distribution, T denotes the episode horizon, and $\gamma \in [0,1)$ is the discount factor.

2.4. PPO Algorithm

In this paper, PPO [36] is employed as the foundational architecture for policy optimization. The optimal policy parameterized by PPO serves as the learning objective, achieved by maximizing the expected cumulative return $\mathcal{J}({\pi }_{\theta })$:

$$\mathcal{J}({\pi }_{\theta })=E[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t)}]$$

(5)

where $R(s_t,a_t)$ is the return. $\gamma$ is the coefficient. A clipped alternative objective function is defined, which limits the maximum deviation between the old and new policies to maintain training stability of the policy network:

$$L^{clip}(\theta )=E_t[min(r_t(\theta ){\widehat{A}}_t,clip(r_t(\theta ),1-\epsilon ,1+\epsilon ){\widehat{A}}_t)]$$

(6)

where $r_t\left(\theta \right)={\pi }_{\theta }/{\pi }_{\theta old}$ denotes the probability ratio, ${\widehat{A}}_t$ represents the estimated advantage function, and $ϵ$ is the hyperparameter that clips the ratio. This paper employs Generalized Advantage Estimation (GAE) to compute advantage estimation, defined as

$${\widehat{A}}_t={\displaystyle \sum _{l=0}^{\infty }(\gamma \lambda }{)}^l{\delta }_{t+l}$$

(7)

where ${\delta }_t=R_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)$ represents the temporal difference (TD) error, and $V\left(s_t\right)$ is the value function output by the critic. The scaling factor $\lambda \in \left[\mathrm{0,1}\right]$ controls the bias–variance trade-off.

The critic network is trained to minimize the mean squared error (MSE) loss. MSE calculation is obtained by the mean of the errors of the target reward $G_t$ and the predicted value $V_{\varphi }(s_t)$.

$${\mathcal{J}}^{critic}(\varphi )=E_t[{(G_t-V_{\varphi }(s_t))}^2]$$

(8)

An entropy bonus term for the policy is added to the objective function to enhance exploration. The optimization objective for the actuator network becomes

$${\mathcal{J}}^{actor}(\theta )=L^{clip}(\theta )+\sigma H({\pi }_{\theta })$$

(9)

where $\sigma$ is the hyperparameter for scaling the entropy contribution.

3. Hierarchical Maneuvering Decision-Making Algorithm

The hierarchical decision-making algorithm proposed in this paper mainly consists of three layers, namely, the target allocation layer, the maneuvering decision-making layer, and the aircraft control layer, thereby solving the maneuvering decision-making problem of the scalable multi-UAV cooperative pursuit–evasion game. The overall framework diagram is shown in Figure 3.

This paper adopts a hybrid control mode of centralized optimization at the target allocation layer and distributed decision-making at the execution layer. The target allocation layer conducts centralized target allocation based on global situation information to ensure coordination. The maneuvering decision-making layer and flight control layer adopt distributed reinforcement learning to enhance the real-time response speed.

3.1. Reinforcement Learning Aircraft Controller with Stable Auxiliary Gradients

The core objective of this module is to ensure that the UAV can accurately respond to the pursuit–evasion game instructions when performing tasks such as airspace protection at civil airports, while maintaining smooth flight and avoiding secondary risks caused by trajectory oscillations.

3.1.1. Observation Space and Action Space

(1) Observation space

The observation space established in this paper primarily includes the deviation between the desired state variables and the current state variables, as well as the current flight state variables of the UAV. These mainly comprise $\Delta h$ altitude difference, $\Delta \psi$ heading difference, $\Delta V$ velocity difference, h current flight altitude, $\sin \varphi$ sin of roll angle, $\cos \varphi$ cos of roll angle, $\sin \theta$ sin of pitch angle, $\cos \theta$ cos of pitch angle, $(Vx,Vy,Vz)$ velocity components along the three axes and velocity magnitude V, and angular velocities p, q, r, etc.

(2) Action Space

The action space includes continuous variables for pitch rudder deflection, yaw rudder deflection, roll rudder deflection, and throttle ${\delta }_a,{\delta }_e,{\delta }_r\in [-1,1],{\delta }_t\in [0,1]$.

3.1.2. Reward Function

Expected value error rewards are proposed as follows.

Define the altitude error reward as $r_h=e^{-{(\Delta h/s_h)}^2}$

• where $\Delta h$ is the difference between the expected altitude and the current altitude, and $s_h$ is the scale factor for altitude error.

Define the heading error reward as $r_{\psi }=e^{-{(\Delta \psi /s_{\psi })}^2}$

• where $\Delta \psi$ is the difference between the expected heading and the current heading, and $s_{\psi }$ is the scale factor for heading error.

Define the velocity error reward as $r_V=e^{-{(\Delta V/s_V)}^2}$

• where $\Delta V$ is the difference between the expected velocity and the current velocity, and $s_V$ is the scale factor for velocity error.

The overall reward is $r_{error}={(r_h·r_{\psi }·r_V)}^{1/3}$.

3.1.3. Stabilizing the Auxiliary Gradient

(1) Smoothness measurement

This paper formally presents a practical metric for measuring the oscillation degree of the unmanned aerial vehicle (UAV) flight trajectory under a given strategy π.

The most commonly used quantitative measures are longitudinal acceleration [37], lateral acceleration [38], and jerk [39]. Unlike autonomous vehicles, unmanned aerial vehicles travel in three-dimensional space, and their trajectory oscillations are mainly caused by continuous changes in flight attitudes. A smooth trajectory should maintain a stable angular velocity.

Therefore, in this paper, angular acceleration is adopted as the key metric for this smoothness. In this case, the one-step smoothness metric (OSSM) $\kappa (s,a)$ is defined as the norm of angular acceleration:

$$\kappa (s,a)=‖\alpha ‖=‖{\displaystyle \frac{d\mathsf{\omega }}{dt}}‖$$

(10)

where $\mathsf{\omega }={[p,q,r]}^T$ is the angular velocity vector and α is the angular acceleration.

The multi-step smoothness metric (MSSM) is defined as the superposition of the one-step smoothness metric, and an auxiliary network $\mathcal{H}(s,a)$ is used to approximate it:

$$\mathcal{H}(s,a)=\mathbb{E}\left[\sum _{k=0}^T{\gamma }^k\kappa (s_{t+k},a_{t+k})\right]$$

(11)

It is constrained to be less than a certain upper bound ${\mathcal{K}}_t$.

The objective of the auxiliary network is to accurately predict cumulative angular acceleration. Therefore, its loss calculation ${\mathcal{J}}^{Aux}(\psi )$ is carried out by predicting the difference between multi-step angular acceleration and the actual value.

$${\mathcal{J}}^{Aux}(\psi )={\mathbb{E}}_t\left[{\left(\left[\sum _t{\gamma }^t\kappa (s_t,a_t)\right]-{\mathcal{H}}_{\psi }({\mathit{s}}_t)\right)}^2\right]$$

(12)

(2) Three-layer nested optimization problem

We aim to maximize the cumulative rewards while meeting the smoothness constraints:

$$\underset{\pi }{\max }\mathbb{E}\left[\sum {\gamma }^{k}r(s,a)\right]\hspace{1em}\mathrm{s}.\mathrm{t}.\hspace{1em}\mathbb{E}\left[\sum _{k=0}^T{\gamma }^k\kappa (s_{t+k},a_{t+k})\right]\le {\mathcal{K}}_t$$

(13)

Construct the Lagrange function, introduce the Lagrange multiplier $\widehat{\lambda }$ ($\widehat{\lambda }$ ≥ 0, physical meaning: the penalty coefficient for violating the smoothness constraint), and transform it into an unconstrained optimization problem through the Lagrange relaxation method.

$$\underset{\widehat{\lambda }\ge 0}{\min }\underset{\theta }{\max }\left[\mathcal{J}({\pi }_{\theta })-\widehat{\lambda }\left({\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _t{\gamma }^t\kappa ({\mathit{s}}_t,{\mathit{a}}_t)\right]-{\mathcal{K}}_t\right)\right]$$

(14)

Since the $\mathcal{J}({\pi }_{\theta })$ (value function) and $\mathcal{H}(s,a)$ need to be learned from the data, the problem is ultimately decomposed into a three-layer nested optimization problem, with the following objectives for each layer:

Inner layer (max): $\underset{\theta }{\max }(L^{clip}(\theta )+\sigma H({\pi }_{\theta }))$, the strategy strives to maximize the cumulative reward.

Middle layer (min): $\underset{\theta }{\min }({\mathcal{H}}_{\psi }(\mathit{s},{\pi }_{\theta }(\mathit{s}))-{\mathcal{K}}_t)$, the strategy strives to control the estimated cumulative angular acceleration to keep it below the upper bound of the constraint.

Outer layer (min): $\underset{\widehat{\lambda }\ge 0}{\min }\mathbb{E}\left[-\widehat{\lambda }({\kappa }_{\psi }(\mathit{s},{\pi }_{\theta }(\mathit{s}))-{\mathcal{K}}_t)\right]$, adjust the penalty coefficient, increase when the constraint is violated, and decrease when the constraint is satisfied.

Under the PPO framework, the above objective is approximated through the proxy loss function. The trajectories sampled by the current policy are utilized to avoid resampling for each update. Finally, the loss function of the PPO-SAG actor network is obtained:

$$\mathcal{J}({\pi }_{\theta },\psi )=L^{clip}(\theta )+\sigma H({\pi }_{\theta })-\widehat{\lambda }\left({\mathbb{E}}_{(\mathit{s},\mathit{a})\sim {\rho }_{\pi }}\left[{\kappa }_{\psi }(s,a)\right]-{\mathcal{K}}_t\right)$$

(15)

(3) Gradient calculation and parameter update

The three-layer optimization objective is achieved by alternately updating the policy parameters $\theta$, auxiliary network parameters $\psi$, and Lagrange multipliers $\widehat{\lambda }$ through the gradient descent method.

(a) Major Gradient

Optimize the policy ${\pi }_{\theta }$ using the PPO algorithm to maximize cumulative rewards:

$${\nabla }_{\theta }{\mathcal{J}}_{\pi }(\theta )={\nabla }_{\theta }(L^{clip}(\theta )+\sigma H({\pi }_{\theta }))$$

(16)

(b) Stable Auxiliary Gradient

By using the chain differentiation rule, the stable auxiliary gradient can be obtained as

$${\mathsf{\nabla }}_{\theta }{\mathcal{J}}^{Aux}=-\widehat{\lambda }·\mathbb{E}\left[{\mathsf{\nabla }}_a{\mathcal{H}}_{\psi }(\mathit{s},\mathit{a}){|}_{\mathit{a}={\pi }_{\theta }(\mathit{s})}·{\mathsf{\nabla }}_{\theta }{\pi }_{\theta }(\mathit{s})\right]$$

(17)

The auxiliary gradient is used to penalize those actions that cause high-angular acceleration.

(c) Lagrange multiplier update

Since the outer layer is a min problem, it needs to be updated in the opposite direction of the gradient, and the formula is

$$\widehat{\lambda }\leftarrow \widehat{\lambda }-{\eta }_{\lambda }{\mathsf{\nabla }}_{\widehat{\lambda }}\mathcal{J}=\widehat{\lambda }+{\eta }_{\lambda }\mathbb{E}\left[{\mathcal{H}}_{\psi }(\mathit{s},{\pi }_{\theta }(\mathit{s}))-{\mathcal{K}}_t\right]$$

(18)

where ${\eta }_{\lambda }$ is the learning rate of the multiplier. If ${\mathcal{H}}_{\psi }>\mathcal{K}$ (constraint violation), $\widehat{\lambda }$ will increase; otherwise, it will decrease.

The overall training framework is shown in the following Figure 4. The pseudo-code of the SAG-PPO algorithm is shown in Algorithm 1.

Algorithm 1: SAG-PPO:
1	Initialize the policy network ${\pi }_{\theta }$ and the value network $V_{\varphi }$ Initialize the auxiliary network ${\mathcal{H}}_{\psi }$ Initialize the Lagrange multiplier λ = 0.1 Initialize the hyperparameters and the upper bound ${\mathcal{K}}_t$ of smoothness
2	for step = 1 to Max-steps do
3	env step
4	get (s, a, r, s’, κ, done) and store them in buffer
5	If reach the buffer length
6	calculate critic loss ${\mathcal{J}}^{critic}(\varphi )=E_t[{(G_t-V_{\varphi }(s_t))}^2]$
7	calculate the gradient $\nabla {\mathcal{J}}^{critic}(\varphi )$ and update critic parameters
8	calculate the main loss $\mathcal{J}({\pi }_{\theta })=L^{clip}(\theta )+\sigma H({\pi }_{\theta })$
9	calculate the loss of the auxiliary network $J^{Aux}(\theta )=-\widehat{\lambda }\left({\mathbb{E}}_{(\mathit{s},\mathit{a})\sim {\rho }_{\pi }}\left[{\kappa }_{\psi }(\mathit{s},\mathit{a})\right]-{\mathcal{K}}_t\right)$
10	$\mathcal{J}({\pi }_{\theta },\psi )=\mathcal{J}({\pi }_{\theta })+J^{Aux}(\theta )$ total loss is obtained
11	calculate the gradient $\nabla \mathcal{J}({\pi }_{\theta },\psi )$ and update actor parameters
12	calculate the gradient ${\mathsf{\nabla }}_{\psi }{\mathcal{J}}^{Aux}$ and to update the auxiliary network parameters
13	$\widehat{\lambda }\leftarrow \widehat{\lambda }-{\eta }_{\lambda }{\mathsf{\nabla }}_{\widehat{\lambda }}\mathcal{J}=\widehat{\lambda }+{\eta }_{\lambda }\mathbb{E}\left[{\mathcal{H}}_{\psi }(\mathit{s},{\pi }_{\theta }(\mathit{s}))-{\mathcal{K}}_t\right]$ Update the Lagrange multiplier

The PPO-SAG controller adopts a fully connected network (hidden layer size (128, 128)), with an input dimension of 14 (observation space: height difference, course difference, angular velocity, etc.) and an output dimension of 4 (control instructions: roll, pitch, yaw, throttle), forward propagation time $O(D_{in}\times D_{hidden}+D_{hidden}+D_{hidden}\times D_{out})$ ($D_{in}=14$, $D_{hidden}=128$, $D_{\mathrm{out}}=4$), and the output time of a single control command for a single unmanned aerial vehicle is approximately 0.8 ms. The stable auxiliary gradient and Lagrange multiplier update are only performed during the training phase. During the inference phase, the trained policy output instructions are directly called, with no additional computational overhead.

3.2. Maneuver Decision-Making Layer Based on Trajectory Prediction and Hierarchical Reinforcement Learning

In the scenario of civil airport airspace protection, this layer provides precise pursuit guidance or evasion decisions for security UAVs by predicting the trajectories of illegally invading UAVs, thereby enhancing the effectiveness of protection tasks. The overall framework is shown in Figure 5.

3.2.1. Trajectory Prediction

By predicting the movement trend of the opponent UAV, the position of the opponent UAV after a certain period of time can be obtained, thereby forming a pre-tracking state. (Figure 6 presents a schematic diagram of the pre-tracking.) This can effectively increase the win rate. Therefore, this paper proposes a prediction of opponent UAV positions based on polynomial fitting.

Since the movement of opponent UAV satisfies the basic motion equations, it is impossible for sudden changes to occur in a short period of time. Therefore, it is feasible to predict the trajectory by using the polynomial fitting method.

$$\left[\begin{array}{c}{x}_{ep}(t)\\ y_{ep}(t)\\ z_{ep}(t)\end{array}\right]=P_{3\times 3}\left[\begin{array}{c}{t}^2\\ t\\ 1\end{array}\right]=\left[\begin{array}{ccc}{p}_1^x& p_2^x& p_3^x\\ p_1^y& p_2^y& p_3^y\\ p_1^z& p_2^z& p_3^z\end{array}\right]\left[\begin{array}{c}{t}^2\\ t\\ 1\end{array}\right]$$

(19)

$$\left\{\begin{array}{l}{p}_1^x={\displaystyle \frac{1}{2·\Delta t^2}}(x_e(t)-2·x_e(t-\Delta t)+x_e(t-2·\Delta t))\hfill \\ p_2^x={\displaystyle \frac{1}{2·\Delta t^2}}(3·x_e(t)-4·x_e(t-\Delta t)+x_e(t-2·\Delta t))\hfill \\ p_3^x=x_e(t)\hfill \end{array}\right.$$

(20)

$$\begin{array}{l}{x}_e(t+\Delta t)=p_1^x·\Delta t^2+p_2^x·\Delta t+p_3^x\\ y_e(t+\Delta t)=p_1^y·\Delta t^2+p_2^y·\Delta t+p_3^y\\ z_e(t+\Delta t)=p_1^z·\Delta t^2+p_2^z·\Delta t+p_3^z\\ V_{ex}(t+\Delta t)={\displaystyle \frac{x_e(t+\Delta t)-x_e(t)}{\Delta t}}\\ V_{ey}(t+\Delta t)={\displaystyle \frac{y_e(t+\Delta t)-y_e(t)}{\Delta t}}\\ V_{ez}(t+\Delta t)={\displaystyle \frac{z_e(t+\Delta t)-z_e(t)}{\Delta t}}\end{array}$$

(21)

where $x_e(t)$, $x_e(t-\Delta t)$, and $x_e(t-2·\Delta t)$ represent the X-axis positions of the other party at the current moment, the moment before $\Delta t$, and the moment before $2·\Delta t$, respectively. $x_e(t+\Delta t)$ represents the predicted X-axis position of the target at the moment after $\Delta t$.

3.2.2. Observation Space and Action Space

(1) Observation space

This paper has already carried out target allocation at the upper level. Therefore, UAVs take the observed space variables as input and generate action output through strategic model reasoning. This method extracts features from battlefield information and normalizes them, taking the basic state of our UAV and the geometric state of both sides’ UAV information as input.

Specifically, the basic state variables include the UAV’s altitude h, the magnitude of velocity in the x direction $V_x$, the magnitude of velocity in the y direction $V_y$, the magnitude of velocity in the z direction $V_z$, the angular velocity q, the magnitude of velocity V, and the rate of velocity change ΔV. The relative geometric state information mainly includes the state of the opponent UAV at the current moment $stat{e}_{t_0}=[\Delta x_{t_0},\Delta y_{t_0},\Delta z_{t_0},{\theta }_{Los\_body}^{t_0},{\psi }_{Los\_body}^{t_0},V_x^{t_0},V_y^{t_0},V_z^{t_0}]$, the predicted state of the opponent UAV $stat{e}_{t_0+\Delta t}$ after time $\Delta t$, and the predicted state of the opponent UAV $stat{e}_{t_0+2\Delta t}$ after time $2\Delta t$, where ${\theta }_{Los\_body},{\psi }_{Los\_body}$ are the pitch and heading of the line of sight vector in the aircraft coordinate system. To achieve the tactical behavior of pre-tracking, we combine the predicted position of the opponent UAV with the current position information as the input. The observed variables and their definitions are shown in

Table 1. Observation space.

Variable Type	Variable Symbol	Meaning
Basic state variables	h	Height
	$V_x$	Velocity value in the x direction
	$V_y$	Velocity value in the y direction
	$V_z$	Velocity value in the z direction
	$V$	Magnitude of speed
	$\Delta V$	Change in velocity
	q	Pitch Angle velocity
Relative geometric state variables	$stat{e}_{t_0}$	Current status of opponent aircraft
	$stat{e}_{t_0+\Delta t}$	Status of opponent UAV after $\Delta t$ time
	$stat{e}_{t_0+2\Delta t}$	Status of opponent UAV after $2\Delta t$ time

.

(2) Action space

Since this paper employs hierarchical reinforcement learning, in the upper layer, $\Delta \psi ,\Delta h,\Delta V$ changes in heading, altitude, and speed are used as actions. Each action component is five discrete decision spaces, shown in

Table 2. Action space.

Action Variables	Scope
$\Delta h$ (km)	{−0.2, −0.1, 0, 0.1, 0.2}
$\Delta \psi$ (radian)	{−π/6, −π/12, 0, π/12, −π/6}
$\Delta V$ (mach)	{−0.2, −0.1, 0, 0.1, 0.2}

.

3.2.3. Reward Shaping

(1) Situation reward

In reference [40], angular situation and distance situation were considered separately. However, in the context of this paper, to meet the termination conditions, both angular and distance conditions need to be satisfied simultaneously. Therefore, this paper introduces a comprehensive consideration of distance and angular situation.

Angular situation: In order to guide the UAV towards the opponent’s UAV, relatively strict angular reward conditions were adopted.

$$Ta=\left\{\begin{array}{cc}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}10\hfill & if q_{Los}<4/180\times pi\hfill \\ 1+2{\displaystyle \frac{15/180\times pi-q_{Los}}{15/180\times pi}}\hfill & elseif 4/180\times pi<q_{Los}<15/180\times pi\hfill \\ 1-{\displaystyle \frac{\alpha -15/180\times pi}{35/180\times pi-15/180\times pi}}\hfill & elseif 15/180\times pi<q_{Los}<35/180\times pi\hfill \\ 0\hfill & \mathrm{else}\hfill \end{array}\right.$$

(22)

$$Td=\left\{\begin{array}{c}1,d\le d_A\\ e^{1-d/d_A},d>d_A\end{array}\right.$$

(23)

The distance situation is considered as follows: when the distance is less than the termination distance of the UAV, the distance situation is 1; when it is less than the termination distance, the distance situation value decays.

By coupling the angular situation and the distance situation, taking into account the opponent UAV threat, the situation reward can be obtained as

$$R_{adv}={\omega }_1\times Ta\times Td-{\omega }_2\times T{a}_{\mathrm{e}}\times Td$$

(24)

where $T{a}_{\mathrm{e}}$ represents the angle situation value of the opponent UAV, with ${\omega }_1$ set to 1 and ${\omega }_2$ set to 0.7.

(2) Pre-tracking reward

This is a direct incentive for the agent to perform pre-tracking. Directly encourage the agent to aim the nose of the aircraft at the future position of the opponent UAV rather than its current position. This is a direct means to increase the win rate. By calculating the LOS angle, a reward is given, where $q_{Los}$ is the LOS angle and $\sigma$ is $4/180\ast pi$. Through the $q_{Los}^{t_0},q_{Los}^{t_0+\Delta t},q_{Los}^{t_0+2\Delta t}$ calculation and weighting, the result can be obtained:

$$r_{lead}=0.5e^{-(q_{Los}^{t_0}-\sigma )}+0.3e^{-(q_{Los}^{t_0+\Delta t}-\sigma )}+0.2e^{-(q_{Los}^{t_0+2\Delta t}-\sigma )}$$

(25)

(3) Win/lose rewards

$$r_{win-lose}=\left\{\begin{array}{cc}5\hfill & \hfill ,win\\ -5\hfill & \hfill ,lose\end{array}\right.$$

(26)

3.2.4. Hierarchical Reinforcement Learning Algorithm

During the upper-level training process, the low-level model network parameters are frozen, and the high-level model network parameters are updated. During the training process, the action output of the upper-level agent serves as an observation for the lower-level agent. The maneuvering decision algorithm based on hierarchical PPO is shown in Algorithm 2.

The pseudo-code of the hierarchical PPO algorithm is as follows.

Algorithm 2: hierarchical PPO
1	Load low level model $M_{low}$ and set it to eval mode Initialize actor, critic parameters, and hyperparameters
2	for step = 1 to Max-steps do
3	get action from actor
4	transfer action to low level model
5	$M_{low}$ output action to env
6	env step
7	get (s, a, r, s’, κ, done) and store them in buffer
8	If reach the buffer length
9	calculate critic loss ${\mathcal{J}}^{critic}(\varphi )=E_t[{(G_t-V_{\varphi }(s_t))}^2]$
10	calculate the gradient $\nabla {\mathcal{J}}^{critic}(\varphi )$ and update critic parameters
11	calculate actor loss $\mathcal{J}({\pi }_{\theta })=L^{clip}(\theta )+\beta H({\pi }_{\theta })$
12	calculate the gradient $\nabla \mathcal{J}({\pi }_{\theta })$ and update actor parameters

In this network layer, polynomial fitting is adopted to predict the future position of the enemy, and a linear least squares problem is solved based on K historical trajectory points, with a time complexity of O (K³). In practice, K is taken as 3, which is equivalent to a constant time of O (1), and the time consumption for a single prediction is approximately 0.5 ms.

Hierarchical reinforcement learning inference The structure of the upper-level policy network (MLP + GRU) is fixed (hidden layer size [128, 128]), independent of the scenario scale. The input feature dimension is 31, the time complexity of forward propagation is $O(D_{in}\times D_{mlp}+D_{mlp}\times D_{gru}+D_{gru}^2+D_{gru}\times D_{hidden}+D_{hidden}\times D_{out})$ ($D_{in}$ = 31, $D_{hidden}$ = 128, $D_{mlp}$ = 128, $D_{gru}$ = 128, $D_{out}$ = 3), and the reasoning time for a single unmanned aerial vehicle is approximately 1.2 ms. The reasoning of N drones can be executed in parallel, so the time consumption is the same as that of a single drone.

3.3. Dynamic Target Allocation Algorithm Based on Dynamic Value Adjustment

In response to the demand for multi-security UAVS to collaboratively protect the airport airspace, this algorithm rationally allocates the intervention tasks for illegal UAVs through dynamic value adjustment. It avoids the concentration or waste of resources and ensures the comprehensiveness and efficiency of protection.

3.3.1. Situation Assessment

Based on the game theory algorithm, we have established a pursuit–evasion game situation assessment model. In the northeast coordinate system of the earth coordinate system, assuming the position of the i-th unmanned aerial vehicle (UAV) on the red side is $(x_{ri},y_{ri},z_{ri})$ and its velocity vector is ${\overrightarrow{V}}_{ri}=(V{x}_{ri},V{y}_{ri},V{z}_{ri})$, and the position of the j-th UAV on the blue side is $(x_{bj},y_{bj},x_{bj})$ and its velocity vector is ${\overrightarrow{V}}_{bj}=(V{x}_{bj},V{y}_{bj},V{z}_{bj})$, the relative position vector can be obtained as $\overrightarrow{D}=(x_{bj}-x_{ri},y_{bj}-y_{ri},z_{bj}-z_{ri})$. According to the geometric model of the pursuit–evasion game situation, the relative entry angle between the red UAV and the blue UAV can be calculated as

$$A_{\alpha }^{ij}=\arccos ({\displaystyle \frac{\overrightarrow{D}·{\overrightarrow{V}}_{ri}}{‖\overrightarrow{D}‖‖{\overrightarrow{V}}_{ri}‖}}),\hspace{1em}{A}_{\alpha }^{ji}=\arccos ({\displaystyle \frac{-\overrightarrow{D}·{\overrightarrow{V}}_{bj}}{‖\overrightarrow{D}‖‖{\overrightarrow{V}}_{bj}‖}})$$

(27)

The advantage of the red UAV over the blue UAV is defined based on the angle of entry as

$$A^{ij}={\displaystyle \frac{\pi -A_{\alpha }^{ij}}{\pi }}$$

(28)

Similarly, the threat level of the blue drone to the red drone is defined as

$$A^{ji}={\displaystyle \frac{\pi -A_{\alpha }^{ji}}{\pi }}$$

(29)

Thus, the situation advantage of the red party UAV over the blue party UAV is obtained as follows:

$$S^{ij}=A^{ij}-A^{ji}$$

(30)

According to the game theory method, $-S^{ij}$ represents the situation advantage of the blue UAV over the red UAV.

3.3.2. Dynamic Target Allocation Method

According to the above situation calculation method, the situation matrix can be obtained as

$$\left[\begin{array}{cccc}{S}^{11}& S^{12}& \dots & S^{1j}\\ S^{21}& S^{22}& \dots & S^{2j}\\ \dots & \dots & \dots & \dots \\ S^{i1}& S^{i2}& \dots & S^{ij}\end{array}\right]$$

(31)

When allocating targets, it is necessary to consider both the situation advantage of our aircraft over the opponent and the threat posed by the opponent to our aircraft. Therefore, by integrating the situation advantage ($S_{ij}$ > 0) and the threat ($S_{ij}$ < 0), the formula is

$$U_{ij}=\alpha ·\max (S_{ij},0)+\beta ·\max (-S_{ij},0)$$

(32)

where $\alpha$ represents the advantage weight ($\alpha >0$), indicating the degree of importance attached to the situation advantage, fixed at 1; $\beta$ represents the threat weight ($\beta >0$), indicating the degree of importance attached to the threat.

Then the target allocation model can be constructed as

$$\begin{array}{l}\max {\displaystyle \sum _{i=1}^{N_a}}{\displaystyle \sum _{j=1}^{N_t}}{U}_{ij}·x_{ij}\\ \mathrm{s}.\mathrm{t}.{\displaystyle \sum _{j=1}^{N_t}}{x}_{ij}=1\hspace{1em}\mathsf{\forall } j\in [1,\dots ,N_a]\\ \hspace{1em}\hspace{1em}{x}_{ij}\in \left\{0,1\right\}\end{array}$$

(33)

where $x_{ij}\in \left\{0,1\right\}$ represents the decision variable. When it is 1, it indicates that the target j is assigned to UAVi; when it is 0, it indicates that it is not assigned to UAVi. The optimization target is $\max {\displaystyle \sum _{i=1}^{N_a}}{\displaystyle \sum _{j=1}^{N_t}}{U}_{ij}·x_{ij}$. Since each UAV in this article can only be assigned one target, the constraint condition is ${\displaystyle \sum _{j=1}^{N_t}}{x}_{ij}=1\hspace{1em}\mathsf{\forall } j\in [1,\dots ,N_a]$. However, many of our UAVs can jointly pursuit the same opponent UAV. Therefore, in this paper, there is no constraint that the target can only be assigned to one of our UAVs. Thus, this problem cannot be solved by the traditional auction algorithm. If the matrix game method is used to transform it into a hybrid game problem, the number of our strategies is 6⁶, which is too large and the solution time is too long, and it cannot meet the real-time requirements. Therefore, this paper proposes the use of lightweight collaborative allocation rules.

This paper proposes a dynamic value adjustment mechanism to reduce the attraction of a friendly UAV to other friends when it chooses to pursue an opponent UAV, avoid excessive concentration, and naturally incentivize the selection of suboptimal but unsaturated targets.

The procedure is as follows:

1. Initialization: Calculate the value matrix $U$ based on the original situation matrix S.

2. Find the global best pairing: Locate the current maximum value $U[i*,j*]$ in $U$.

3. Allocation: Allocate opponent UAV $j*$ to friendly UAV $i*$.

4. Dynamic devaluation: Reduce the attractiveness of opponent aircraft j* to all other friendly UAVs: $U[i,j*]=U[i,j*]*\delta ,i\in [1,\dots N_a]$, where $\delta$ is the depreciation factor.

5. Update matrix: Set $U[i*,j]$ to the minimum value (or mark it as assigned) to prevent duplicate selection.

6. Repeat steps 2 to 5 until all friendly machines have been allocated or there are no available targets.

By devaluing the selected target, it encourages other friendly UAVs to seek new targets. This algorithm is highly efficient in calculation: in each round, it only needs to find the maximum value of the matrix (O(N²)), iterate N times, and the total complexity is O(N³), with extremely high real-time performance. Figure 7 shows the dynamic target allocation process.

Theorem 1.

The dynamic target allocation algorithm proposed in this paper is a 1/2 approximation algorithm for the maximum weight matching problem. That is, the total utility value of the solution obtained by the algorithm satisfies $w(M)\ge {\displaystyle \frac{1}{2}}w(M^{\ast })$, where $M^{\ast }$ is the global optimal solution.

Proof: 

We formalize the many-to-many objective assignment problem as a Maximum Weight Matching problem. □

Bipartite Graph: Construct a bipartite graph $G=(R,B,E)$, where R is the set of red drones (friendly aircraft), B is the set of blue drones (opponent aircraft), and E is the edge set, where each edge $(i,j)$ connects $i\in R$ to $j\in B$. The weight of each edge $(i,j)$ is $w_{ij}=U_{ij}$, which is the defined utility value that integrates the situation advantage and threat. All weights $w_{ij}>0$.

We will prove it by comparing the greedy solution X with the optimal solution X. For each edge X in the optimal match, it must “conflict” with at most two edges in the greedy match X.

If $(i,j)$ is also selected by the greedy algorithm, then it exists simultaneously in $M$ and $M^{\ast }$.

If $(i,j)$ is not selected, it must be because during the operation of the greedy algorithm, when edge $(i,j)$ still has a chance to be selected, the algorithm selects another edge with a greater weight, which is connected to $i$ or $j$.

Specifically, there might be an edge $(i,k)\in M$ (that is, the blue side $k$ is eventually assigned to the red side $i$), and $w_{ik}>w_{ij}$.

Or there might be an edge $(l,j)\in M$ (that is, the blue side $j$ is eventually allocated by the red side $l$), and $w_{lj}>w_{ij}$.

Since a friendly machine can only be assigned one target and an opponent machine can be assigned multiple times but not repeatedly from the same friendly machine, an edge can “block” at most pairs of edges in the optimal solution. Therefore,${\displaystyle \sum _{(i,j)\in M^{*}}{w}_{ij}}\le {\displaystyle \sum _{(p,q)\in M}2w_{pq}}$, which is equivalent to $w(M)\ge {\displaystyle \frac{1}{2}}w(M^{\ast })$.

The algorithm in this paper adds a dynamic devaluation mechanism on the basis of the standard greedy algorithm. This operation will not undermine the proof of the 1/2 approximation ratio mentioned above. The devaluation operation is carried out only after one edge has been added to the match. The proof process focuses on the decision made when the algorithm selects edges, and this decision is based on the weights at that time. Devaluation only affects subsequent choices. Since we always choose the edge with the current maximum weight, and after devaluation, the weights of some edges become smaller, this may actually make it less likely for the algorithm to make “poor” choices in subsequent selections, and it may even obtain a better solution than the standard greedy algorithm. Therefore, the quality of the solution of the algorithm in this paper will at least not be worse than that of the standard greedy algorithm.

$$w(M_{\mathrm{ours}})\ge w(M_{\mathrm{greedy}})\ge {\displaystyle \frac{1}{2}}w(M^{\ast })$$

(34)

4. Simulation Results

Using a single machine (AMD Ryzen 9 9800X CPU 4.5 GHz, 24-core, one NVIDIA GeForce RTX 3090 24 G GPU, 128 GB RAM), all experiments were run three times using different random seeds. In this article, the important parameter settings of the PPO algorithm are listed in

Table 3. Parameter settings of PPO algorithm.

Name	Value
n_rollout_threads	32
use_valuenorm	True
hidden_sizes	[128, 128]
activation_func	relu
gain	0.01
use_recurrent_policy	True
recurrent_n	1
learn_rate	0.0005

.

4.1. Simulation Comparison of Flight Control Layer

In the flight control layer, the PPO algorithm is adopted to train the flight control. The expected state instructions are given every 12 s, and the network is sampled and trained from the process. Since the update of the policy network takes into account the influence of both the primary gradient and the auxiliary gradient simultaneously, this paper compares the training effect of using the auxiliary network with that of not using it.

The training results of the rewards curve are shown in Figure 8. From the overall trend of the data, it can be seen that the performance with/without the stable auxiliary gradient is roughly the same. This study indicates that adding the proposed smooth auxiliary gradient does not significantly reduce learning efficiency.

In addition, to prove the effectiveness of the auxiliary gradient proposed in this paper, the instruction responses of the trained network were compared, and an expected state instruction was given every 12 s.

Figure 9a shows the curves of the expected state and the actual flight state. The green dots in Figure 9a represent the expected states. It can be seen from the figure that both with and without the auxiliary network can reach the expected state within the specified time, indicating that the auxiliary network does not affect the state tracking effect of the main network. Figure 9b shows the variation curve of the UAV attitude angles, from which it can be seen that the attitude angle oscillation is relatively intense without the auxiliary network. Figure 9c shows the OSSM variation curve, from which it can be seen that the auxiliary network has demonstrated an excellent OSSM suppression effect. Figure 9d shows the three-dimensional flight trajectory, and the two are quite similar.

4.2. Simulation Comparison of Maneuvering Decision-Making Layer

4.2.1. Trajectory Prediction Simulation

In this section, we quantitatively evaluate the accuracy of the polynomial trajectory prediction used in this paper. We selected a flight trajectory for online trajectory prediction and used the root mean square error of position as the evaluation index. The results of the trajectory prediction are shown in Figure 10.

In Figure 10a,b, the blue UAV represents the position of the UAV at the current moment, the green UAV represents the predicted position 1 s later, and the red UAV represents the predicted position 2 s later. It can be seen from this that the polynomial trajectory prediction method in this paper has a relatively high prediction accuracy. Figure 10c shows the variation curve of trajectory prediction error and overload over time. It can be seen from this that when the overload of UAV flight changes significantly, the trajectory prediction error will increase, but the maximum error can be maintained within 100 m.

4.2.2. The Training Results of Hierarchical Reinforcement Learning

In the maneuver decision-making layer, the red UAV is driven by hierarchical reinforcement learning and the blue UAV is driven by rules. Ablation experiments were conducted at the maneuvering decision-making level. By comparing the hierarchical reinforcement learning method with and without trajectory prediction, the effectiveness of trajectory prediction was demonstrated.

Figure 11a shows the rewards curve. It can be seen that with prediction, there is a faster convergence speed, and the rewards it eventually converges to are larger than those without prediction. Figure 11b shows the win rate curve. Eventually, the algorithm with prediction converges to around 0.6 and the algorithm without prediction converges to around 0.43. Therefore, it can be proved that trajectory prediction has a significant effect on improving the win rate of pursuit–evasion game. In addition, to demonstrate the effectiveness of the trained agents, the confrontation process between the final trained red agent and the rule-driven blue agent is presented.

Figure 12 provides screenshots of the confrontation process in the tacview visualization software. Initially, the red side and the blue side face each other head-on, with a distance of 26.4 km between the two sides. In the subsequent 70 s and 93 s, the red and blue sides adopt similar maneuvers, forming a single loop. Then, at 143 s, the red side clearly gains a situation advantage through trajectory prediction. On the 266 s and 390 s, the red side demonstrates obvious pre-tracking tactical behavior. Finally, at 546 s, the red side reaches the termination condition, and the red side wins. It can be seen from the process that the red side can effectively utilize trajectory prediction to gain a situation advantage.

4.3. Target Allocation Simulation

To demonstrate the target allocation process, taking 6V6 as an example, the heat maps of the target allocation utility matrix U for 35 s and 156 s are presented, and the target allocation results are labeled. The results are shown in Figure 13 and Figure 14.

Figure 13a and Figure 14a show the three-dimensional situation of the UAV at that time, while Figure 13b and Figure 14b present the target allocation utility matrix U and the allocation results. It can be seen from this that the target allocation algorithm proposed in this paper can effectively allocate targets based on the situation, and the proposed dynamic devaluation mechanism avoids allocating the same target to a certain extent. For example, in Figure 14b, the target allocated by the red UAV6 is the blue UAV4, rather than the blue UAV1. Meanwhile, the allocation method proposed in this paper can also generate allocation focus. For instance, in Figure 13b, the red UAV1, UAV2, and UAV5 are all allocated as the blue UAV3. This enables the production of a coordinated pursuit effect. It has a better synergy effect compared with the previous fixed one-to-one allocation model.

This paper also analyzes the hyperparameters in target allocation. Parameter optimization includes situation weight optimization and target devaluation factor optimization.

Situation weight optimization: In order to obtain the optimal situation weight factor parameters, this paper compares the influence of different situation weight factors on the win rate. Experiments were conducted on different weight factors in three scenarios, namely, 3V3, 6V6, and 9V9, and the win rate values under different weight factors were obtained. We took the β values as 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9.

As can be seen from Figure 15, when the situation weight factor is 0.7, it has the highest win rate. The reason is that when the situation weight factor is too high, the agent tends to adopt defensive tactics and is difficult to form a coordinated offensive strategy; thus, the win rate is not high. When the situation weight factor is too low, the agent ignores the threat of an opponent UAV and is easily lost.

Target devaluation factor optimization: The devaluation factor reduces the focusing effect among multiple UAV and encourages friendly UAVs to seek other targets. However, if the value is too small, it may make it difficult to form a coordinated tactical effect and reduce the win rate. Therefore, this paper optimizes the devaluation factor $\delta$, with values of 0.6, 0.7, 0.8, and 0.9.

As can be seen from Figure 16, when the scale increases, it is even more necessary to reduce the target depreciation factor, thereby prompting friendly UAVs to search for other targets. Taking all factors into consideration, the $\delta$ is taken as 0.8.

4.4. Overall Algorithm Comparison

To demonstrate the generalization of the algorithm proposed in this paper in a multi-UAV pursuit–evasion game at different scales, simulations were conducted at 3V3, 6V6, 9V9, and 12V12 scales. The comparison algorithms include MAPPO, QMIX, and the matrix game [41]. References [42,43] demonstrate that these two algorithms have achieved good results in multi-UAV pursuit–evasion game. The MAPPO and QMIX algorithms adopt the maneuvering decision-making method of hierarchical reinforcement learning in reference [43]. The red side uses the above-mentioned algorithms, while the blue side uses the rule-based target allocation method. The rule-based target allocation method and the matrix game target allocation algorithm only differ from the algorithm proposed in this paper at the target allocation layer. The same algorithm is used at both the maneuvering decision-making layer and the flight control layer. Among them, the rule-based target allocation method is as follows: Initially allocate the target closest to one’s own side. After merging with the opponent aircraft, switch the target to this one.

Three evaluation indicators were selected: win rate, decision-making time, and training time consumption. Thus, the advantages and disadvantages of the algorithm can be examined from multiple aspects. Under the same initial conditions, 30 Monte Carlo simulations were conducted to obtain the win rates of each algorithm.

Figure 17 shows the comparison results of the method proposed in this paper with the MARL method and the matrix game algorithm. From Figure 17a, the win rate of our method and the matrix game algorithm increases as the scale increases, while the win rate of the MARL algorithm decreases as the scale increases. This is related to the credit allocation problem that the MARL method faces in large-scale game scenarios. In the case of small-scale game scenarios, the QMIX algorithm has the highest win rate. When the scale increases, our method has the highest win rate.

As shown in Figure 17b, the MARL method has the shortest single-step decision-making time consumption, followed by our method. The matrix game optimization target allocation algorithm takes a relatively long time and struggles to meet the real-time requirements of the pursuit–evasion game. Moreover, as the scale increases, the time consumption of the matrix game optimization target allocation algorithm increases sharply. As can be seen from Figure 17c our method does not require retraining with changes in scale, while the training time of MARL algorithms increases sharply with the increase in scale.

To demonstrate the reliability of 30 simulation times, we conducted more simulation times to obtain the changes in the win rate, thereby proving the effectiveness of the number of simulation times in the evaluation of algorithm quality.

As shown in Figure 18, the win rate difference between 30 and 40 simulations is ≤ 2.8% (average 1.9%), and between 30 and 50 simulations is ≤ 3.5% (average 2.3%). A two-sample t-test (α = 0.05) was conducted to verify the difference between win rates of 30 and 50 simulations. The results show p-values > 0.05 for all algorithms, indicating no statistically significant difference, confirming that 30 Monte Carlo simulations are sufficient for reliable performance evaluation.

The blue opponents in the above simulation results are generally rule-driven. Therefore, in order to better demonstrate the advantages of the method proposed in this paper over other methods, we will compare the method proposed in this paper with the MAPPO, QMIX, and matrix game methods. Conduct 30 Monte Carlo confrontations on a scale of 3V3, 6V6, 9V9, and 12V12 to obtain the win rate, which is presented in Figure 19.

The elements of the matrix heat map in Figure 19 represent the win rates of the game between the method in the row and the method in the column. As can be seen from Figure 19, the method proposed in this paper does not have an advantage over the MAPPO, QMIX, and matrix game methods when the scale is relatively small. However, as the scale increases, the win rate gradually rises. When playing against MAPPO and QMIX in 12V12, the win rate can reach about 80%.

The experimental results show that although MAPPO and QMIX have an advantage in decision-making time consumption, their training time consumption far exceeds that of our method, and their win rate in large-scale scenarios is significantly lower than that of the hierarchical allocation method proposed in this paper. This proves the advantages of our method in terms of training efficiency and large-scale scalability. The matrix game target allocation algorithm obviously cannot meet the requirements of real-time decision-making. Due to the existence of allocation delay during target allocation, the win rate is not as high as that of our method.

To conduct a detailed analysis of the confrontation process, this article provides screenshots of the 3V3 confrontation process as shown in Figure 20.

The above figure shows the 3V3 simulation confrontation process. In Figure 20a, at the initial moment, the red and blue sides formed a formation and faced each other head-on. In Figure 20b, a UAV in the blue square reaches the termination condition because it ignores the threat behind the tail. Then, at 253 s and 392 s, the other two drones on the blue side meet the termination conditions. It can be seen from the confrontation process that the entire confrontation process is significantly shortened compared to the 1V1 confrontation process. The reason lies in the synergistic effect brought about by rapid target allocation in a many-to-many confrontation situation. During the confrontation, the blue UAVs failed in the end because they only focused on pursuing and neglected to evade.

4.5. Analysis of the Impact of Termination Conditions

In this article, the capture area is set with a line of sight angle < 4° and a distance < 4 km, and the captured area is set with a line of sight angle < 20° and a distance < 4 km. To further study the impact of termination conditions on scalability, we adopted different termination conditions. To obtain the win rate, 30 Monte Carlo confrontations are conducted between the method proposed in this paper and the rule opponent. The selection of termination conditions is divided into similar conditions, simple conditions, and strict conditions.

Among them, the similar conditions are as follows: the capture area has a line of sight angle < 5° and a distance < 5 km, and the captured area has a line of sight angle < 30° and a distance < 5 km.

The simple conditions are as follows: the capture area has a line of sight angle < 60° and a distance < 8 km, and the captured area has a line of sight angle < 90° and a distance < 8 km.

The strict conditions are as follows: the capture area has a line of sight angle of less than 4° and a distance of less than 2 km, and the captured area has a line of sight angle of less than 4° and a distance of less than 2 km.

It can be seen from

Table 4. Table of termination conditions and win rate.

	Initial Conditions	Similar Conditions	Simple Conditions	Strict Conditions
3V3	63.8%	60.8%	53.4%	25.4%
6V6	76.5%	73.5%	48.3%	32.5%
9V9	84.7%	79.4%	55.6%	38.5%
12V12	88.0%	85.6%	59.1%	41.3%

that under similar conditions, the termination condition has little impact on the win rate of the method proposed in this paper. However, under simple conditions, the win rates of the red and blue sides are comparable. The reason is that the termination condition is too simple, which leads to too much randomness in meeting the termination condition, thus resulting in comparable win rates. Under harsh conditions, due to overly strict termination conditions, many scenarios ended in draws. When the scale increased, as the method proposed in this paper could produce a synergistic effect, the win rate rose to a certain extent. This also reflects the rationality of the termination conditions set in this article.

4.6. Failed Case Analysis

By analyzing the failed cases, the deficiencies of the method proposed in this paper can be obtained. We obtained a failed case in a 3V3 scenario, and the confrontation process is shown in Figure 21.

Figure 21 shows a failed case of a 3V3 scenario pursuit and evasion game. In Figure 21a, at the initial moment, the red and blue sides form a formation and face each other directly. In Figure 21c, a red UAV is simultaneously targeted and attacked by three blue UAVs. Then, at 156 s, this red UAV meets the termination condition. Subsequently, as the number of blue UAVs is greater than that of the red UAVs, another red UAV meets the termination condition at 166 s. Finally, at 352 s, the last red UAVs reach the termination condition one by one under the coordinated pursuit of the two blue UAVs. It can be seen from the confrontation process that the red side lost because the blue side just formed a coordinated pursuit tactic.

4.7. Ablation Experiment

To demonstrate the effectiveness of each module of the method proposed in this paper, it was verified through ablation experiments. The comparison algorithms mainly include the following:

(1) Baseline (Ours): The complete algorithm.

(2) Ablation variant 1: Ours w/o DTA, removing the dynamic target assignment and replacing it with the nearest distance assignment method.

(3) Ablation variant 2: Ours w/o TP, removing trajectory prediction.

(4) Ablation variant 3: Ours w/o SAG, removing the stable auxiliary gradient.

(5) Ablation variant 4: Ours w/o HL, removing the hierarchical reinforcement learning framework and using single-layer reinforcement learning.

The comparison results are shown in

Table 5. Multi-scale comparison results.

	3V3		6V6		9V9		12V12
	Win Rate	Average OSSM	Win Rate	Average OSSM	Win Rate	Average OSSM	Win Rate	Average OSSM
Ours (baseline)	63.8%	0.32	76.5%	0.35	84.7%	0.38	88.0%	0.34
Ours w/o DTA	56.3%	0.33	50.1%	0.36	42.6%	0.39	31.5%	0.32
Ours w/o TP	52.3%	0.31	68.5%	0.36	73.5%	0.35	76.7%	0.31
Ours w/o SAG	60.6%	0.85	72.5%	0.83	80.3%	0.81	83.5%	0.86
Ours w/o HL	51.6%	1.23	59.5%	1.18	64.6%	1.33	71.5%	1.21

.

By comparing the baseline and Ours w/o DTA in Table 5, in the 12v12 scenario, the win rate of Ours w/o DTA is 57.5% lower than that of the benchmark group, while in the 3v3 scenario, it is only 7.5% lower. This indicates that the role of dynamic DTA is more crucial in large-scale scenarios. The comparison between Ours w/o TP and the baseline shows that the win rate of the baseline is 9–12% higher than that of Ours w/o TP, which proves the effectiveness of trajectory prediction in improving the win rate. In the comparison between Ours w/o SAG and the baseline, OSSM was 2.7 times that of the baseline group, and the win rate decreased by 4.5–5.9%, proving that SAG ensures smoothness without sacrificing game performance. The comparison between the baseline and Ours w/o HL proves the effectiveness of hierarchical reinforcement learning in solving the coupling of agent flight control and maneuvering confrontation training and improving the win rate.

From the above simulation results, it can be seen that compared with the traditional fixed allocation and deep reinforcement learning game methods, the scalability of this framework can adapt to security requirements of different scales, and it can be quickly deployed without repeated training. Moreover, the win rate can still remain at around 80% in large-scale scenarios, meeting the protection requirements of civil airport airspace.

5. Conclusions

To address the universality of deep reinforcement learning methods in pursuit–evasion game scenarios of different scales and the stability of flight trajectories, this paper combines hierarchical reinforcement learning methods with target allocation methods and proposes a hierarchical collaborative game framework. The hierarchical collaborative game framework is divided into the target allocation layer, the maneuvering decision-making layer, and the flight control layer. To solve the problems of rapid target allocation, a dynamic target allocation layer based on a dynamic value adjustment mechanism is proposed. To improve the win rate of one-on-one pursuit–evasion game, a maneuvering decision-making method based on trajectory prediction and hierarchical reinforcement learning is proposed. To generate a smooth flight trajectory, a reinforcement learning flight control method with stable auxiliary gradients is proposed. The method proposed in this paper is effective in adversarial scenarios of any scale and has stronger interpretability compared with multi-agent reinforcement learning (MARL) methods. This explainability is verified and quantitatively supported by both the structural design and the experimental results.

Firstly, the hierarchical collaborative framework breaks down the complex multi-drone pursuit and evasion problem into three functionally independent modules: target allocation, maneuvering decision-making, and flight control. The input and output logic of each module is aligned with human tactical intuition, and the structure is transparent. Secondly, the ablation experiment (Table 5) quantified the performance contributions of each module: for instance, after removing the dynamic target allocation module, the win rate in the 12V12 scenario dropped by 57.5%, clearly indicating that it is the core of large-scale collaboration. After removing trajectory prediction, the win rate decreased by 9% to 12%, verifying its guiding role in maneuvering decisions. Thirdly, the decision rules of the method are traceable. The dynamic target allocation builds a utility function based on the situation advantage and threat level, and the maneuverable decision-making achieves pre-tracking through trajectory prediction, all of which are consistent with human decision-making logic rather than the black-box joint learning of the MARL method. Finally, the scale-independent training of the framework (Figure 17c) essentially stems from modular disassembly, avoiding the credit allocation problem that is difficult to explain in the MARL method. These results collectively indicate that the interpretability of the proposed method is not only at the structural level but also quantitatively verified through experiments.

The method proposed in this paper can be applied to the fixed-wing UAV clusters for civil airport security. As a defensive control system, it can perform the tasks of driving away and capturing illegally invading UAVs, effectively protecting the airspace of civil airports. Future research will further explore communication and sensors, introduce dynamic leader selection mechanisms and local communication mechanisms between UAVs, and better enhance the collaborative capabilities among UAVs.