Target Allocation and Air–Ground Coordination for UAV Cluster Airspace Security Defense

Highlights

What are the main findings?

• The proposed PM-HDM framework significantly improves target allocation, pursuit coordination, and interception efficiency of UAV clusters in multi-target confrontation scenarios.
• The integration of UGV-based dynamic recharging extends SI-UAV endurance and enhances the autonomy and sustainability of air–ground collaborative security systems.

What is the implication of the main finding?

• PM-HDM offers a robust and adaptive solution for real-time airspace defense, enabling UAV clusters to rapidly respond to dynamic multi-target threats and maintain interception superiority in complex scenarios.
• The proposed approach provides a practical and scalable foundation for constructing intelligent, autonomous airport defense systems, strengthening the safety, resilience, and operational stability of low-altitude civil airspace.

Abstract

In this paper, we propose a cooperative security method for unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to address the scenario of unauthorized rogue drones (RDs) intruding into an airport’s restricted airspace. The proposed method integrates artificial intelligence techniques with engineering solutions to enhance the autonomy and effectiveness of air–ground cooperation in airport security. Specifically, the MADDPG algorithm enables the Security Interception UAVs (SI-UAVs) to autonomously detect and counteract RDs by optimizing their decision-making processes in a multi-agent environment. Additionally, Particle Swarm Optimization (PSO) is employed for distance-based target assignment, allowing each SI-UAV to autonomously select intruder targets based on proximity. To address the challenge of limited SI-UAV flight range, a power replenishment mechanism is introduced, where each SI-UAV automatically returns to the nearest UGV for recharging after reaching a predetermined distance. Meanwhile, UGVs perform ground patrols across different airport critical zones (e.g., runways and terminal perimeters) according to pre-designed patrol paths. The simulation results demonstrate the feasibility and effectiveness of the proposed security strategy, showing improvements in the reward function and the number of successful interceptions. This approach effectively solves the problems of target allocation and limited SI-UAV range in multi-SI-UAV-to-multi-RD scenarios, further enhancing the autonomy and efficiency of air–ground cooperation in ensuring airport security.

1. Introduction

In modern airport security management, the response strategy of UAV clusters has become a key factor of great strategic significance and cannot be ignored. With the rapid development in UAV technology, UAVs have evolved from a single surveillance role to a comprehensive platform with multiple functions such as monitoring, cross-domain cooperative operations, communication jamming, and data relay. At the same time, thanks to the rapid progress in artificial intelligence algorithms, especially the breakthroughs in the fields of deep learning, reinforcement learning, and computer vision, UAVs have been able to realize autonomous decision-making and control to a certain extent. This autonomous control capability is extremely important in the field of airport security, allowing UAVs to perform complex tasks, such as detecting and tracking unauthorized RDs within restricted airspace, without human intervention or only remote monitoring, thus greatly improving the efficiency of security management [1,2].

This study investigates adversarial strategies for UAV clusters, not only helping to improve UAV protection capabilities but also optimizing the response mechanisms of UAV clusters in the face of complex threats to airport security. In terms of multi-target tracking, dynamic decision-making, and interference mitigation, the collaboration and information sharing of each UAV in the cluster are crucial [3]. By conducting research on intelligent confrontation strategies for UAV clusters, potential security vulnerabilities can be predicted and identified in advance, providing data support and decision-making foundations for protecting critical airport infrastructure. Especially when countering unauthorized rogue drones, each UAV in the cluster can collaborate through real-time information exchange and decision-making, effectively resisting interference and attacks, thus enhancing the survival probability of the security system [4,5]. This not only guarantees the safety of the airport airspace but also provides more flexible and adaptive response strategies for tactical deployment. Therefore, an in-depth study of the confrontation strategies of UAV clusters is of great strategic importance for enhancing the emergency response capabilities of modern airport security systems and improving the effectiveness of UAVs in complex security environments.

Currently, deep reinforcement learning (DRL) algorithms have become an important tool in the field of autonomous decision-making for UAVs. DRL learns optimal strategies through interaction with the environment, enabling UAVs to make fast and accurate decisions when facing complex and changing security scenarios [6]. In terms of motion planning, collaborative DRL significantly improves the ability of UAVs to perform long-range motion planning in the face of unpredictable obstacles by integrating human experience and machine learning [7,8]. The value function algorithm indirectly obtains the agent’s strategy by iteratively updating the value function with the goal of finding the optimal value of the value function when the value function is iterated to the optimal value. The strategy gradient algorithm directly adopts the function approximation method to establish a strategy network, obtains the reward value by selecting actions through the strategy network, and optimizes the strategy network parameters along the gradient direction to obtain the maximum reward value of the optimized strategy [9,10,11].

Despite these advances, the existing DRL-based UAV cluster research still faces critical limitations when applied to real-world airport airspace defense—where dynamic intrusions, resource constraints, and air–ground synergy are unavoidable. Specifically, three key gaps remain unaddressed:

First, the current studies mostly focus on target management paradigms with fixed SI-UAV-to-RD quantity ratios, and their adaptability to dynamic-scale scenarios has not been fully validated. Even commonly used multi-agent DRL algorithms like MADDPG tend to produce suboptimal decisions in such cases: their target-allocation logic fails to adapt to heterogeneous quantity ratios, and their collaborative mechanisms lack flexibility under resource constraints—ultimately leading to decision delays or cooperative failure.

Second, to further enhance the effectiveness of airport airspace defense, operational planning needs to incorporate SI-UAV endurance factors into its design. Most studies still assume SI-UAVs have unlimited flight range, ignoring the practical need for energy replenishment during prolonged defense tasks—an oversight that can cause SI-UAVs to lose maneuverability mid-mission and create vulnerabilities in the airspace defense system.

The existing studies have made certain explorations in UAV-UGV collaboration, including cooperative path planning for persistent tasks, task scheduling optimization in hierarchical multi-agent systems, and resource allocation in multi-agent operations. Notably, UAV-UGV collaboration also involves leveraging UGVs’ potential as mobile resupply/charging stations or supplementary situational awareness nodes—roles that can effectively extend SI-UAV endurance and provide communication support when SI-UAV resources fluctuate. However, the value of such UAV-UGV collaboration, especially regarding UGVs as rule-based support systems for DRL-trained SI-UAV clusters in airport airspace defense scenarios, remains to be further explored, and it has promising application scenarios.

Considering the above three factors comprehensively, this study develops a dynamic resource-scheduling mechanism integrated with an adaptive air–ground coordination architecture, with its innovations focusing on the following three aspects:

(1)

By integrating Particle Swarm Optimization (PSO) with MADDPG, we optimize multi-SI-UAV target assignment based on the principle of real-time distance minimization. This resolves the multi-target ambiguity issue of standalone MADDPG and enhances the coordination of interception tasks, directly tackling the challenge of dynamic SI-UAV-to-RD quantity ratios.

(2)

We use PSO to organize SI-UAVs into cooperative units while maintaining global swarm communication. Goal-oriented resource allocation prevents SI-UAVs from falling into Nash equilibrium and adapts to dynamic changes in swarm size, overcoming the flexibility deficit in traditional DRL-based collaborative mechanisms.

(3)

Mobile UGVs function as rule-based dynamic resource nodes: They patrol preset routes and trigger SI-UAV energy replenishment when a predefined flight distance threshold is reached. This rule-based cyclic resource management enables DRL-trained SI-UAVs to sustain continuous defensive operations—solving the SI-UAV endurance limitation—without the need to integrate UGVs into the learning framework itself.

To ensure that readers can systematically follow the research process and verify the effectiveness of the above innovations, the remainder of this paper is structured as follows: Section 2 presents a comprehensive review of related work, covering multi-UAV cooperation, air–ground coordination, and DRL/PSO-based optimization methods—laying a theoretical foundation for the proposed framework. Section 3 formalizes the research problem and establishes key system models, including the kinematic models of SI-UAVs and UGVs, and the SI-UAV return-to-charge mechanism that supports endurance extension. Section 4 details the design of the proposed PSO-MADDPG hybrid decision-making framework (PM-HDM), with specific elaborations on observation/action space definitions, reward function design, and PSO-based target allocation logic—fully unfolding the core innovation of this study. Section 5 describes the experimental setup, analyzes the simulation results, and discusses the performance of the PM-HDM framework in addressing dynamic intrusions and resource constraints. Finally, it concludes the study and outlines future research directions to further optimize the air–ground coordinated defense system.

2. Related Works

While UAVs have inherent bottlenecks such as limited endurance, limited payload, and insufficient anti-jamming capability, UGVs possess advantages including long ground endurance, high load capacity, and stable communication relay. Through air–ground coordination, a dynamic endurance supply chain and multi-dimensional security support system can be established; by complementing heterogeneous resources, the adaptability of UGVs and UAVs in complex airport security environments is significantly enhanced.

The study presented in [12] proposes a low-cost joint deployment method of UAVs and UGVs on air and ground. UGVs are responsible for ground transportation, and UAVs are responsible for high-altitude distribution. The whole system is developed based on Robot Operating System (ROS), which can compare and optimize the energy consumption and time consumption, and solves the problem of efficiency and cost of the traditional distribution methods, which is especially suitable for distribution in urban or residential areas.

The study presented in [13] investigates the problem of UAV and UGV formation tracking under event-triggered communication. To solve the high cost problem associated with communication, a hierarchical control strategy is proposed, which contains distributed event-triggered filters and local controllers. The strategy effectively realizes formation tracking and reduces the communication overhead, which is of practical significance for multi-UAV and ground vehicle joint system applications in complex environments.

The study presented in [14] investigates the fixed-time distributed tracking problem for heterogeneous UAV and unmanned ground vehicle systems under denial of service (DoS) attacks and external interference. The literature proposes a fuzzy control-based fixed-time secure control strategy (FXFSC), which performs state estimation through fuzzy switching observers to ensure that the system maintains stable tracking under attacks and disturbances, and ensures that the synchronization error and tracking error converge to a predefined small neighborhood.

The study presented in [15] investigates the task scheduling problem for hierarchical multi-agent systems (HMASs). Aiming at the challenges of NP-hard complexity and variable-size decision space of the problem, a knowledge-assisted evolutionary algorithm (SAETS-HMAS) is proposed. It integrates domain knowledge based on the strongly typed genetic programming (STGP) framework to ensure the feasibility of solutions, and constructs a knowledge–data dual-driven transferable surrogate model (with a multi-head scheme to improve accuracy) that can be transferred across different task settings to reduce computational costs. The scheme is verified in UAV/UGV wildfire-fighting simulations and the SwarmBang real-world robot system, enhancing the efficiency and reliability of task scheduling.

The study presented in [16] studies the cooperative path planning of UAVs and UGVs for persistent surveillance tasks in urban environments. To address issues such as limited UAV coverage and building occlusion, the cooperative path planning problem is formulated as a large-scale 0–1 optimization problem. A hybrid EDA-GA algorithm (combining the advantages of global and local search) is proposed, and a sweep-based method is used to generate circular paths. Meanwhile, an online local adjustment strategy is designed to cope with changes in ground coverage requirements. Simulation results show that this scheme can reduce the time required for complete coverage, and the solution quality and stability of the hybrid algorithm are better than those of EDA or GA alone.

In multi-UAV collaborative tasks, traditional DRL cannot effectively handle the collaboration and competition among agents; therefore, multi-agent deep reinforcement learning (MADRL) has emerged to improve the collaborative efficiency of UAV swarms by considering interactions between agents [17,18]. Therefore, this paper aims to explore the heterogeneous synergy between SI-UAV clusters and UGV clusters for airport security tasks through the Multi-Agent Deep Deterministic Policy Gradient-based algorithm [19] in order to improve security task efficiency and autonomy in complex airport security environments.

In order to improve the operational stability and security mission execution efficiency of UAVs in complex security environments, researchers are continuously exploring new path planning and cooperative operation strategies. Some studies have designed threat models based on actual situations that consider factors, such as ground radar, unauthorized aerial obstacles, and terrain, constructed comprehensive threat models, and established multi-constraint objective optimization models by combining temporal and spatial constraints, so as to solve the problem of collaborative path planning for multiple UAVs in complex environments using optimization algorithms [20]. In addition, a study proposed combining the consensus packet algorithm (CBBA) with independent deep reinforcement learning (IDRL) for path planning, which effectively improves the search efficiency and avoids collisions between UAVs; meanwhile, for the obstacle avoidance problem in stochastic environments [21], the researcher used the combination of multi-agent reinforcement learning (MARL) and model predictive control (MPC) to propose a new training and execution framework to optimize the decision-making process [22]. In terms of multi-UAV cooperative operations, a capture algorithm based on the MADDPG algorithm has been proposed [23]. By designing three sub-networks—3-vs.-1 capture sub-network, 2-vs.-1 interception sub-network, and 1-vs.-1 interception sub-network—corresponding to different interception phases, respectively, the method enables the algorithm to adapt to the environment where the number of UAVs is dynamically decaying, and solves the problem of capturing and defending UAVs with limited interception and defensive capabilities and communication ranges, and interception-related challenges.

The study presented in [24] proposes the DP-MADDPG algorithm in the context of UAV cluster intelligence, which overcomes the problems of traditional MADDPG’s tendency to fall into local optima and slow convergence in dynamic environments through the introduction of a decomposition mechanism and preferred experience replay (PER) mechanism. The study presented in [25] proposes a meta-learning-based deep reinforcement learning method for multiple intelligences, integrating model-independent meta-learning with MADDPG, and rapidly achieving end-to-end learning across different tasks by improving the playback mechanism and introducing the forgetting mechanism, which enhances the adaptability and generalization ability of the UAV cluster.

The Fuzzy-MADDPG algorithm presented in [26] utilizes fuzzy logic to synthesize UAV residual energy, target information similarity, and mission completion status, enabling automatic adjustment of search tendencies and providing decision support for UAVs under varying communication conditions. This approach demonstrates particular robustness under network interference. To optimize search efficiency, the method integrates the Maximum Entropy Reinforcement Learning (MERL) model to reduce information redundancy via spatial information entropy and an enhanced reward function. Simulation results indicate that it achieves higher target coverage and lower energy consumption under constrained network conditions.

To address the target assignment and path planning problem, the authors in [27] propose the STAPP method, which simultaneously incorporates target assignment and path planning into a multi-agent system (MAS) optimization framework, thereby reducing computational complexity and enhancing adaptability to dynamic environments.

In response to the communication stability challenges in UAV swarms, the study in [28] introduces a communication network reconstruction method based on the decoupled MADDPG (DE-MADDPG) algorithm. This method augments the traditional MADDPG by adding a local critic network for each agent and alleviates the sparse reward problem through reward function decoupling and reshaping. Additionally, a virtual leader–follower model is introduced to enhance system stability. Simulation experiments demonstrate that DE-MADDPG outperforms conventional methods in terms of success rate, adaptability, and error rate.

The PD-MADDPG algorithm proposed by [29] improves the efficiency and adaptability of unmanned systems in continuous dynamic environments through parallel decoupling. Meanwhile, the CEL-MADDPG algorithm proposed by [30], combined with course-based experiential learning, significantly improves the task completion rate when performing complex single-target interception tasks.

In the studies mentioned above, the focus is often on single-target scenarios, whereas in real-world complex environments, multiple RD targets may need to be addressed simultaneously when SI-UAVs are deployed [31]. This makes it especially important to study the autonomous confrontation strategies between multi-SI-UAVs and multiple RDs. This cooperative tasking capability in multi-target situations [32] not only requires SI-UAV clusters to quickly recognize and assign targets, but also to adjust their strategies in real time to respond to changing security scenarios and the dynamic interaction of multiple RDs [33]. This necessitates the development in algorithms suited for multi-target tasking, path planning, and cooperative strategies to ensure that SI-UAV clusters maintain an advantage and effectively complete multi-target pursuit and interception missions.

In research on SI-UAV cluster multi-target confrontation, many algorithms and models have been proposed to address the task of multi-target pursuit and confrontation in complex environments [34,35]. Among them, MADDPG performs well in multi-target decision-making scenarios because it can handle interactions, cooperation, or competition between multiple agents in a multi-agent environment. Traditional single-agent reinforcement learning algorithms usually assume a static environment, meaning there is no direct interaction between agents. However, in a multi-agent environment, the actions of multiple agents not only affect their own rewards but also directly influence the states and decisions of other agents [36,37].

For the multi-target pursuit task, the authors in [38] introduce the Multi-Target Intelligent Assignment Model (MTIAM), whose core consists of a Multi-Target Assignment Formulation (MTAF) and an improved multi-target gray wolf optimizer. By explicitly accounting for the relative kinematics and multi-attribute information of both UAVs and targets, the framework achieves near-optimal assignment performance. Building on this, the study presented in [39] devises an end-to-end airborne target-management pipeline. An evaluation index system that integrates angle, distance, and energy-efficiency metrics is first constructed; an enhanced hybrid particle-swarm optimizer is then employed for multi-target allocation, while improved Double Q-learning generates maneuvering decisions for each UAV. Simulation results demonstrate that the pipeline enables efficient tracking and neutralization of multiple targets. Recognising that Double Q-learning is inherently single-agent and tends to misestimate state–action values in shared environments, the authors in [40] develop a cooperative approach built upon MADDPG. A Voronoi-diagram-based reward mechanism is introduced to balance exploration against goal tracking, and a centralised critic is adopted during training to mitigate the instability caused by non-stationary policies. Empirical evaluations show that the resulting system significantly improves the swarm’s capture capability in dynamic scenarios.

3. Problem Description and Modeling

3.1. UAV and UGV Motion Modeling

In the study of multi-UAV and multi-UAV confrontation, it is first assumed that all UAVs fly at a fixed altitude and UGVs move on the ground at a fixed speed to construct a continuous dynamic scenario. The 2D graphical kinematic models of the UAVs and UGVs are shown in Figure 1, respectively, where $d_{\mathit{uav}}$ and $d_{\mathit{ugv}}$ represent the radii of the UAVs and UGVs, respectively. If the distance between the midpoints of two UAVs is less than the diameter of the UAV, it is considered a collision. To simplify the process of the UAV returning to the nearest UGV for updating, it is stipulated that the return is considered successful when the difference in distance between the UAV centroid and the UGV centroid is extremely small.

Figure 1. Kinematic model of UAV and UGV.

Then the kinematic equations of UAV and UGV are shown in (1):

$$\left\{\begin{array}{cc}\hfill p^t& =\left[\begin{array}{c}{p}_x^t\\ p_y^t\end{array}\right]=\left[\begin{array}{c}{x}^{t-1}+v_x^t·\Delta t\\ y^{t-1}+v_y^t·\Delta t\end{array}\right]\hfill \\ \hfill v^t& =\left[\begin{array}{c}{v}_x^{t-1}+a_x^t·\Delta t\\ v_y^{t-1}+a_y^t·\Delta t\end{array}\right]\hfill \end{array}\right.$$

(1)

where $p_x^t$, $v_x^t$, and $a_x^t$ denote the position, velocity, and acceleration of the UAV and UGV on the x-axis at time t, respectively; $p_y^t$, $v_y^t$, and $a_y^t$ denote the position, velocity, and acceleration of the UAV and UGV on the y-axis at time t, respectively; and $\Delta t$ denotes the time interval. The UAV is limited by the maximum velocity $v_{\max }$ and the maximum acceleration $a_{\max }$ due to its dynamic constraints and safety considerations. As for the UGV, it is set to always travel at a constant speed $v_{ugv}$ on the ground along a predetermined route at a uniform speed. Therefore, the UAV should satisfy the constraints in (2):

$$\left\{\begin{array}{cc}\hfill \left|v^t\right|& \le v_{\max }\hfill \\ \hfill \left|a^t\right|& \le a_{\max }\hfill \end{array}\right.$$

(2)

The specific parameters mentioned above will be given in Section 5.

3.2. SI-UAV Return to Resupply Model

Due to the limited range of the UAV, in order to help the SI-UAV return to the nearest UGV in time to replenish the range when it is insufficient, this paper establishes the SI-UAV return model as shown in (3):

$$\left\{\begin{array}{cc}\hfill d& =\sqrt{{\left(x^t-x^{t-1}\right)}^2+{\left(y^t-y^{t-1}\right)}^2}\hfill \\ \hfill d^t& =d^{t-1}+d\hfill \end{array}\right.$$

(3)

Each SI-UAV is equipped with a dedicated odometer, where $d_t$ denotes the displacement difference between the SI-UAV at time t and its displacement at the previous time step. If the sampling interval is sufficiently small, $d_t$ can be approximated as the SI-UAV’s instantaneous flight distance. When $d_t$ exceeds the predefined threshold $d_{\max }$, the SI-UAV searches for the nearest UGV that is not currently supplying energy to another SI-UAV and autonomously navigates to its coordinates. Once a SI-UAV initiates the return process toward a specific UGV, the state of that UGV is marked as “True,” indicating that it is occupied and cannot accept other SI-UAVs simultaneously. However, to prevent the system from being locked in the event of abnormal conditions—such as UAV destruction, mission reassignment, or communication delays—a timeout-based fail-safe mechanism is incorporated. If the SI-UAV fails to reach the designated UGV within a specified timeout period $t_{timeout}$, the UGV automatically resets its state to “False,” assuming that the return mission has failed and thereby releasing the charging resource. In the experimental setup, $t_{timeout}$ is set to 10 min, which reflects a reasonable balance between UAV speed and endurance; this parameter can be flexibly adjusted in practical deployments according to the specific flight characteristics of different UAV platforms. When the SI-UAV successfully arrives within the timeout window, the charging process is executed, and upon completion, the UGV state is reset to “False,” allowing it to serve other UAVs thereafter.

3.3. MADDPG and PSO

3.3.1. MADDPG

MADDPG is a type of multi-agent deep reinforcement learning (MADRL), where agents are trained to adaptively perform different behaviors based on varying environments. When multiple agents act in the same environment simultaneously, there may be competition, collaboration, or both. In such scenarios, a single-agent algorithm like DDPG is no longer suitable because the actions of each agent influence the states and rewards of the others. To address this issue, MADDPG employs a “centralized training–decentralized execution” mechanism, as illustrated in Figure 2: each agent makes decisions independently based on its local observations (via the actor network), while a shared critic network integrates global state–action information to optimize the policy during training, enabling synergy between distributed control and centralized value evaluation. Additionally, MADDPG uses an experience replay buffer to store the global state–action–reward transitions encountered by multiple agents, and it performs centralized policy optimization via batch sampling, supporting the synergy between decentralized decision-making and global experience.

Figure 2. Centralized training and decentralized execution.

The update methods for the agent and critic networks vary for each agent of the MADDPG. The goal of the critic network is to minimize the Bellman error of the values and update the weights of the critics using the time difference (TD) error. As an example, the update formula for the critic network for agent i is shown in (4):

$$\begin{array}{c}L\left({\theta }^{Q_i}\right)=E_{s_t,a_t,r_t,s_{t+1}}\left[{\left(Q_i\left(s_t,a_t;{\theta }^{Q_i}\right)-y_i\right)}^2\right]\hfill \end{array}$$

(4)

where is the Q value output from the critic network, and the target Q value is calculated as in (5):

$$\begin{array}{c}{y}_i=r_t+\gamma Q_i^{\prime }\left(s_{t+1},a_{t+1}^{\prime };{\theta }^{Q_i^{\prime }}\right)\hfill \end{array}$$

(5)

Here, $Q_i^{\prime }$ is the target critic network, and the parameters of the target network are updated only at regular intervals. $a_{t+1}^{\prime }$ is the next action generated by the agent through its target actor strategy, $\gamma$ is the discount factor, and $r_t$ is the current reward. ${\theta }^{Q_i}$, the parameter of the critic network, is updated through the gradient descent method is updated to minimize the loss function $L\left({\theta }_i\right)$, and the update formula is shown in (6):

$$\begin{array}{c}{\theta }^{Q_i}\leftarrow {\theta }^{Q_i}-{\alpha }_Q{\nabla }_{{\theta }^{Q_i}}L\left({\theta }^{Q_i}\right)\hfill \end{array}$$

(6)

The goal of the actor is to make the chosen action maximize the Q value given by the critic network, and in the case of agent i, the update formula is (7):

$$\begin{array}{c}{\nabla }_{{\theta }^{{\pi }_i}}J\left({\theta }^{{\pi }_i}\right)=E_{s_t}\left[{\nabla }_{a_i}{Q}_i\left(s_t,a_t;{\theta }^{Q_i}\right){\nabla }_{{\theta }^{{\pi }_i}}{\pi }_i\left(a_t\mid s_t;{\theta }^{{\pi }_i}\right)\right]\hfill \end{array}$$

(7)

Since actor outputs a deterministic strategy, ${\nabla }_{{\theta }^{{\pi }_i}}{\pi }_i(\left.a_t\right|s_t;{\theta }^{{\pi }_i})$ can obtain (8) by the chain rule:

$$\begin{array}{c}{\nabla }_{{\theta }^{{\pi }_i}}J\left({\theta }^{{\pi }_i}\right)=E_{s_t}\left[{\nabla }_{a_i}{Q}_i\left(s_t,a_t;{\theta }^{Q_i}\right){\nabla }_{{\theta }^{{\pi }_i}}{a}_t\right]\hfill \end{array}$$

(8)

Here, ${\nabla }_{a_i}{Q}_i(s_t,a_t;{\theta }^{Q_i})$ is the gradient of the critic network for the action, indicating the value of the action, and ${\nabla }_{{\theta }^{{\pi }_i}}{a}_t$ is the gradient of the actor network for the action. The parameter ${\theta }^{{\pi }_i}$ of the actor network is updated according to Equation (9) by gradient ascent:

$$\begin{array}{c}{\theta }^{{\pi }_i}\leftarrow {\theta }^{{\pi }_i}+{\alpha }_{\pi }{\nabla }_{{\theta }^{{\pi }_i}}J\left({\theta }^{{\pi }_i}\right)\hfill \end{array}$$

(9)

where ${\alpha }_{\pi }$ is the learning rate of the actor network.

On the other hand, the update for the target network is realized by soft update, which means that the parameters of the target network will only be updated slightly each time based on the current network parameters, and the formula for the soft update is (10):

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }$$

(10)

where ${\theta }^{\prime }$ is a parameter of the target network, $\theta$ is a parameter of the current network, and $\tau$ is a coefficient of soft update that takes the value of $0<\tau \ll 1$.

3.3.2. PSO

The Particle Swarm Optimization (PSO [41]) algorithm simulates the intelligent behavior of a flock of birds, such as foraging, and achieves optimization by iteratively updating the particles’ positions and velocities. Each particle tracks its own historical optimal solution (pBest) and the group’s optimal solution (gBest), dynamically adjusting its search direction based on the fitness function. This allows the algorithm to efficiently approximate the globally optimal solution in a multi-dimensional solution space.

The particle updates are implemented in vector form. For particle i at generation k we denote its position by the m-dimensional vector ${\mathbf{x}}_i^k\in {\mathbb{R}}^m$ and its velocity by ${\mathbf{v}}_i^k\in {\mathbb{R}}^m$. The position is updated by adding the velocity:

$${\mathbf{x}}_i^{k+1}={\mathbf{x}}_i^k+{\mathbf{v}}_i^{k+1}.$$

(11)

The velocity update used is the standard vector-valued PSO rule:

$${\mathbf{v}}_i^{k+1}=\omega {\mathbf{v}}_i^k+c_1{\mathbf{r}}_1\odot \left({\mathbf{pbest}}_i-{\mathbf{x}}_i^k\right)+c_2{\mathbf{r}}_2\odot \left(\mathbf{gbest}-{\mathbf{x}}_i^k\right),$$

(12)

where $\omega$, $c_1$, and $c_2$ are scalar coefficients (inertia, cognitive, and social weights); ${\mathbf{pbest}}_i\in {\mathbb{R}}^m$ is the personal best position vector of particle i; $\mathbf{gbest}\in {\mathbb{R}}^m$ is the global best position vector across the swarm, ${\mathbf{r}}_1,{\mathbf{r}}_2\in {[0,1]}^m$ are random vectors with independently sampled components in $[0,1]$; and “⊙” denotes element-wise multiplication. All the operations in (11) and (12) are, therefore, understood component-wise and are mathematically consistent in the vector setting.

For each particle the algorithm evaluates a scalar fitness value based on its current position; if the current fitness is better than the particle’s historical best, the personal best ${\mathbf{pbest}}_i$ is updated accordingly. The global best $\mathbf{gbest}$ is the position associated with the best scalar fitness observed across the population. Stopping is determined by the usual criteria (e.g., maximum iterations or convergence of the objective).

4. PSO-MADDPG Hybrid Decision-Making Algorithm (PM-HDM)

4.1. Definition of the Problem

When RDs from unknown sources invade, the multi-SI-UAV response scenario for addressing multi-RD intrusions can be described as follows: In a limited operational airspace, two groups with distinct operational goals—SI-UAVs and RDs—operate within the same space, each with maneuvering capabilities aligned with their respective roles. It is assumed that all SI-UAVs share identical dynamic performance and must make collaborative, consistent decisions to neutralize the RDs; by contrast, the RDs leverage their maneuverability to stay as far away from the SI-UAVs as possible and attempt to evade interception. Before the response process begins, the PSO algorithm groups the SI-UAVs according to different operational objectives.

Due to the background situation involving RDs intruding into a controlled airspace from an unknown source, SI-UAVs will share real-time position and velocity information with each other during the mission, while also obtaining the real-time position and velocity information of the RDs. Additionally, the RDs are equipped with sensing capabilities, allowing them to passively detect or actively scan to acquire the position and velocity data of all SI-UAVs—this may be due to the integrated sensors or communication devices of the RDs. This bidirectional information exchange allows both sides to dynamically adjust their behavior, thereby increasing the complexity of airspace management.

To enhance the practical applicability and technical challenge of the research, these RDs are given higher maneuverability, creating an asymmetric pursuit–avoidance scenario between the slower SI-UAVs and the faster evading RDs. Furthermore, while the endurance of autonomous UAV systems (SI-UAVs) is limited (due to range constraints), the RDs do not face such limitations—this presents a more stringent condition for the autonomous UAV system to perform its tasks. This paper adopts a deep reinforcement learning algorithm to construct a maneuver decision-making model suitable for such dynamic scenarios, training and optimizing the model through simulations of pursuit–avoidance interactions. Unlike traditional methods, the target allocation strategy for autonomous UAV systems is based on a distance-adaptive mechanism (instead of random allocation), enabling a rapid response and efficient interception of the RDs. This design focuses on minimizing the time required to neutralize the RDs and improving the overall efficiency of airspace security management.

4.2. Algorithm Overview

The PSO-MADDPG hybrid decision-making system is illustrated in Figure 3, which primarily employs MADDPG for the collaborative decision-making of UAV clusters in confrontation maneuver tasks. PSO is executed only once per episode, and it is triggered right before the first step of that episode. At this single trigger point, to handle situations with unequal numbers of UAVs, the environment sends the position of each UAV to the PSO algorithm, which also acquires real-time position information of all SI-UAVs and RDs; the algorithm then computes the optimal target allocation scheme through particle-swarm optimization. After the predetermined number of iterations, the PSO algorithm delivers the allocation results to the MADDPG algorithm, which not only fixes the specific RD target that each SI-UAV must track throughout the current episode (ensuring each SI-UAV maintains a clear and stable pursuit target during the entire training episode, thereby avoiding decision-making hesitation and cooperation failures caused by target switching in traditional methods) but also controls each SI-UAV to interact with the environment according to the targets assigned by PSO. The data generated during this interaction is stored in the shared experience pool to train each agent individually; once training is complete, the agents are controlled by both their respective actor networks and the energy-replenishment system to perform the task. In other words, when a SI-UAV reaches its maximum range, the energy-replenishment system takes over, guiding the SI-UAV to search for the nearest UGV that is not currently supplying energy to another SI-UAV and autonomously navigate to it for recharging, while at other times, the SI-UAV is controlled by the actor network to engage the RD.

Figure 3. Flowchart of PM-HDM.

4.3. Multi-Agent Collaborative Adversarial Modeling

4.3.1. Observation Space

In the confrontation scenario of UAV clusters, the observation space $S_{si-{uav}_i}$ for SI-UAVs is defined according to (13):

$$\left\{\begin{array}{c}{S}_{si-ua{v}_i}=v_i+p_i+S_{tar}+S_{team}+S_{ugv}\hfill \\ S_{tar}=[v_{tar},p_{tar}-p_i]\hfill \\ S_{team}=[v_{team},p_{team}-p_i]\hfill \end{array}\right.$$

(13)

where $vi,pi$ represent the speed and position of the SI-UAV i, respectively, and $Star$ is the set of speeds of the target pursued by UAV i and the relative position between the target and the aircraft. $Steam$ is the set of speeds and relative positions of all other SI-UAVs except SI-UAV i. $vtar,ptar,vteam$, and $pteam$ represent the speeds and positions of the target and the friendly UAVs, respectively. $S_{ugv}$ represents the absolute positions of all UGVs. It should be noted that $S_{ugv}$ is not involved in the training process; instead, it is solely utilized to identify the nearest UGV once a SI-UAV’s endurance reaches the threshold.

For the RD the observation space $S_{R{D}_i}$ is defined according to (14):

$$\left\{\begin{array}{c}{S}_{r{d}_j}=v_j+p_j+S_{si-uav}\hfill \\ S_{si-uav}=[v_{si-uav},p_{si-uav}-p_j]\hfill \end{array}\right.$$

(14)

where $v_j$ and $p_j$ represent the speed and position of the RD j, respectively, and $Ssi-uav$ is the set of the speeds of all SI-UAVs and their respective relative positions to RD j. $v_{si-uav}$ and $p_{si-uav}$ represent the speed and position of the SI-UAV, respectively. Since the UGVs do not participate in the confrontation, but only patrol ground targets while simultaneously serving as charging stations for the SI-UAVs, no observation space is defined for them.

4.3.2. Action Space

Whether it is an SI-UAV or an RD, their action space is the same. In order to make the control system closer to the real scenario, the action space of the UAV is set as a continuous two-dimensional vector:

$$A=\left[F_x,F_y\right]$$

(15)

where $F_x,F_y$ denote the thrust force on the x-axis and y-axis of the UAV, respectively; then, naturally the acceleration of the UAV can be expressed as (16):

$$a=\left[a_x,a_y\right]=\left[{\displaystyle \frac{F_x}{m_{uav}}},{\displaystyle \frac{F_y}{m_{uav}}}\right]$$

(16)

where $m_{uav}$ denotes the mass of the UAV.

As for the UGVs, we always let the UGVs travel at a constant speed on the ground at a constant speed, so the acceleration is set to zero. Then combining the UAV and UGV models in Section 3.1, the dynamics models of UAV and UGV can be obtained. In addition, considering the performance limitations of the UAV itself, a maximum acceleration $a_{\max }$ is set for each UAV in the experiment, and the specific parameters will be given in Section 5.

4.3.3. Reward Function

In the UAV cluster confrontation scenario, the task of each SI-UAV is to get closer and closer to its target RD and issue an attack behavior (with the purpose of expelling or neutralizing the target RD). In this scenario the collision initiated by the SI-UAV is regarded as an attack behavior, and the determination conditions for collision occurrence have been given in Section 3.1, the collision reward function of the SI-UAV is shown in (17):

$$r_{c_i}=\left\{\begin{array}{c}10,\mathrm{if}\mathrm{collision}\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(17)

In which obtaining a positive reward (with a value of 10) is limited to a collision with the target RD, the distance reward function for an SI-UAV is shown in (18):

$$r_{d_i}=-\sqrt{{(x_{tar}-x_i)}^2+{(y_{tar}-y_i)}^2}$$

(18)

where $x_{tar},y_{tar},x_i,y_i$ represent the coordinates of the target RD and the coordinates of its SI-UAV i, respectively.

Therefore, the total reward function of the SI-UAV can be expressed as

$$r_{si-uav}=r_{c_i}+r_{d_i}$$

(19)

For each RD, it does not know which SI-UAV is pursuing itself, so the task of each RD is to stay away from all SI-UAVs and avoid collision with SI-UAVs. The collision reward function of the RD is shown in (20):

$$r_{c_i}=\left\{\begin{array}{c}-10,\mathrm{if}\mathrm{collision}\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(20)

Here, a negative reward (with a value of −10) is assigned for a collision with any SI-UAV. In addition, each collision will cause the maximum speed of the RD to decay—this decay is used to characterize the degree to which the RD’s maneuverability is impaired; once the maximum speed decreases to zero, the RD is considered rendered non-operational (i.e., unable to continue unauthorized intrusion). The distance reward function for the RD is shown in (21):

$$r_{d_i}=\sqrt{{(x_{\min }-x_i)}^2+{(y_{\min }-y_i)}^2}$$

(21)

where $x_{min},y_{min},x_i,y_i$ represent the coordinates of the nearest SI-UAV and the coordinates of its RD i, respectively. Assuming that the length and width of the simulation experimental site are both m, in order to prevent the RD from running out of the boundary, the boundary reward function on the coordinate axes as expressed in (22) and (23) is also set for the RD:

$$r_{x_i}=\left\{\begin{array}{c}0,\mathrm{if}\left|x\right|<{\displaystyle \frac{m}{2}}-2\hfill \\ \left({\displaystyle \frac{m}{2}}-\left|x\right|\right)·15,\mathrm{if}{\displaystyle \frac{m}{2}}-2<\left|x\right|<{\displaystyle \frac{m}{2}}\hfill \\ \max \left(e^{2\left|x\right|-m},30\right),\mathrm{if}\left|x\right|>{\displaystyle \frac{m}{2}}\hfill \end{array}\right.$$

(22)

$$r_{y_i}=\left\{\begin{array}{c}0,\mathrm{if}\left|y\right|<{\displaystyle \frac{m}{2}}-2\hfill \\ \left({\displaystyle \frac{m}{2}}-\left|y\right|\right)·15,\mathrm{if}{\displaystyle \frac{m}{2}}-2<\left|y\right|<{\displaystyle \frac{m}{2}}\hfill \\ \max \left(e^{2\left|y\right|-m},30\right),\mathrm{if}\left|y\right|>{\displaystyle \frac{m}{2}}\hfill \end{array}\right.$$

(23)

The terms $r_{x_i}$ and $r_{y_i}$ represent the boundary penalty rewards of the i-th RD in the horizontal and vertical directions, respectively. Taking $r_{x_i}$ as an example, the field length is m. When $\left|x\right|\le \frac{m}{2}-2$, the RD remains well within the boundary and receives no penalty. When $\frac{m}{2}-2<\left|x\right|\le \frac{m}{2}$, the RD approaches the boundary and a linearly increasing penalty is applied to guide it back toward the valid region. When $\left|x\right|>\frac{m}{2}$, the RD has crossed the boundary and an exponentially increasing penalty is imposed to strongly discourage further deviation. To prevent excessive or unstable reward values after boundary violation, an upper bound is applied to the penalty term.

The total reward function of RD i can be expressed as follows:

$$r_{r{d}_i}=r_{c_i}+r_{d_i}+r_{x_i}+r_{y_i}$$

(24)

4.3.4. PSO-Based Target Allocation

This section briefly explains the rationale for selecting the PSO algorithm for target allocation in the current context and describes the specific allocation mechanism adopted in this study. When the problem is simplified to a one-to-one linear allocation with a well-defined cost matrix, classical allocation methods such as the Hungarian algorithm or auction mechanisms become highly attractive: they can efficiently provide (near-)optimal solutions to such bipartite graph matching problems. However, the target allocation task examined in this paper exhibits several practical characteristics that limit the applicability of these classical methods: (i) Allocation relationships may be many-to-one or many-to-many (multiple SI-UAVs may be assigned to a single RDs, and vice versa). (ii) The objective function to be minimized is nonlinear and inherently represented as a continuous spatial quantity.

PSO was selected for its pragmatic balance between solution quality, implementation flexibility, and scalability. Specifically, PSO supports direct encoding of complex allocation schemes and custom adaptability functions, and it can also incorporate distance, energy, and capacity constraints if required in later extensions. It handles non-convex nonlinear objective functions without linearizing the problem and easily parallels particle evaluation to reduce runtime. Furthermore, PSO inherently supports soft constraints or additional constraints through penalty terms within the fitness function, facilitating the integration of operational rules.

In a UAV clustering mission, the number of SI-UAVs must be at least equal to the number of RDs. This requirement stems from the nature of the mission: if there are n RDs invading our airspace, at least n SI-UAVs are needed, with each RD being confronted by at least one SI-UAV. The principle for allocating SI-UAVs is illustrated in Figure 4: Assuming that there are m SI-UAVs and n RDs in the scenario, the PSO algorithm calculates whether m divides n. If m divides n and the result of the division is s, then each RD is pursued by s SI-UAVs. If it is not divisible and the remainder after division is h, then there will be h RDs pursued by $s+1$ SI-UAVs, and the rest of the RDs pursued by s UAVs, and this distribution scheme ensures that the number of pursuers for each RD is as balanced as possible. Once an assignment vector has been obtained from a particle (after the continuous position is projected to the discrete index domain), the PSO algorithm evaluates a scalar fitness for that assignment. Let m denote the number of RDs and n the number of SI-UAVs. Let us represent a discrete assignment by

$$\mathbf{z}=\left(z_1,z_2,\dots ,z_m\right)\in {\{0,\dots ,n-1\}}^m,$$

where $z_i$ is the index of the SI-UAV assigned to RD i. The fitness function for assignment $\mathbf{z}$ is defined as the total Euclidean distance between each RD and its assigned SI-UAV:

$$f\left(\mathbf{z}\right)=\sum _{i=1}^{m}g\left(p_{si-ua{v}_{z_i}},p_{r{d}_i}\right),$$

(25)

$$g\left(p_{si-ua{v}_{z_i}},p_{r{d}_i}\right)=\sqrt{{\left(x_{si-ua{v}_{z_i}}-x_{r{d}_i}\right)}^2+{\left(y_{si-ua{v}_{z_i}}-y_{r{d}_i}\right)}^2},$$

(26)

where $p_{si-ua{v}_{z_i}}=(x_{si-ua{v}_{z_i}},y_{si-ua{v}_{z_i}})$ denotes the coordinates of the SI-UAV assigned to RD i, and $p_{r{d}_i}=(x_{r{d}_i},y_{r{d}_i})$ denotes the coordinates of RD i. The objective is to minimize $f\left(\mathbf{z}\right)$, i.e., to find an assignment that minimizes the sum of distances between RDs and their assigned SI-UAVs.

Figure 4. Principles of PSO quantity allocation.

By incorporating PSO goal assignment before training begins, we ensure that each SI-UAV has a clear objective, enhancing the agent’s task focus in each round of confrontation. The subsequent training process is based on this goal assignment, improving both the autonomy and effectiveness of the confrontation task.

In our implementation, the PSO assignment routine is invoked at environment reset—i.e., at the start of every training episode—which increases computational overhead but yields measurable performance benefits. The primary computational cost of PSO stems from particle fitness evaluations and particle updates. Let P denote the number of particles, I the number of iterations. Each fitness evaluation requires distance calculations for all n targets, so a single fitness call has complexity $O\left(n\right)$. Updating all P particles in one iteration, therefore, costs $O(P·n)$, and running the algorithm for I iterations leads to an overall time complexity of $O(I·P·n)$. Since PSO is executed at every stage, the total overhead incurred by PSO scales linearly with the number of stages.

The pseudo-code for the PSO algorithm is shown in Algorithm 1, and the pseudo-code for PM-HDM is shown in Algorithm 2:

Algorithm 1 PSO Pseudocode.

Input:

m: Number of SI-UAVs

n: Number of RDs

adversaries: List of SI-UAVs

agents: List of RDs

num particles: Number of particles

Output:

best_assignment: Index of SI-UAVs assigned to each RD

//Step 1. Initialization Stage

1:      Initialize particles (particles, velocities, best_positions, best_fitnesses)

2:      global_best_position ← None

3:      global_best_fitness ← ∞

//Step 2. Iteration Process

4:      For each iteration from 1 to iterations:

5:          For each particle:

6:               Compute fitness(particle.position)

7:               If current fitness <particle’s local best fitness:

8:                     Update particle’s local best position and fitness

9:               If current fitness <global best fitness:

10:                   Update global best position and fitness

//Step 3. Update the velocity and position of each particle

11:          For each particle:

12:               Update velocity: update_velocity(particle.velocity, particle.position, …)

13:               Update position: particle.position += particle.velocity

14:               Correct position: enforce_constraints(particle.position, m, n)

15:      Return global_best_position

Algorithm 2 PM-HDM Pseudocode.

Input:

s: State space

a: Action space

r: Reward function

e: Environment

n: Number of UAV

$\gamma$: Discount factor

$\tau$: Soft update coefficient

B: Experience replay buffer size

best_assignment: PSO Assignment Results

//Step 1.Initialization

1:     For each agent i:

2:         Initialize Actor network ${\mu }_i\left(s\right)$

3:         Initialize Critic network $Q_i(s,a)$

4:         Initialize target networks ${\mu }_i^{\prime }$ and $Q_i^{\prime }$

5:         Initialize experience replay buffer B

//Step 2. Perform target assignment before each training (PSO)

6:     For each iteration:

7:         best_assignment = PSO(m, n, SI-UAVs, RDs)

8:         For each SI-UAV i:

9:                  SI-UAV_i.target = best_assignment[i]

//Step 3. Each UAV selects an action based on the current state

10:        For each UAV i:

11:               $a_i={\mu }_i\left(s_i\right)$ # Select action via Actor network

12:               Execute action $a_i$ in environment

13:               Observe reward $r_i$ and next state $s_i^{\prime }$

14:               Store $(s_i,a_i,r_i,s_i^{\prime })$ in buffer B

//Step 4. Update Critic and Actor networks

15:         For each UAV i:

16:               Sample a batch of experiences from B

17:               Compute TD error: ${\delta }_i=r_i+\gamma Q_i^{\prime }(s_i^{\prime },{\mu }_i^{\prime }\left(s_i^{\prime }\right))-Q_i(s_i,a_i)$

18:               Update Critic network using TD error

19:               Update Actor network by maximizing $Q_i$

20:               Soft-update target networks: ${\mu }_i^{\prime }\leftarrow \tau {\mu }_i+(1-\tau ){\mu }_i^{\prime }$

21:               Soft-update target networks: $Q_i^{\prime }\leftarrow \tau Q_i+(1-\tau )Q_i^{\prime }$

5. Experiments and Conclusions

5.1. Parameter Description

In the simulation experiment of cluster confrontation, the task simulation step is set to 0.1 s, and the task site is defined as a two-dimensional plane of 100 m × 100 m, represented by the coordinate range $-50\le x,y\le 50$. The coordinates are used to calculate the distances involved in the algorithm. During scene initialization, all UAVs are randomly deployed at specific positions within the map. The detailed settings of UAV dynamics and the learning-related hyperparameters are summarized in Table 1. Specifically, the table lists the initial motion conditions (initial velocity, maximum speed, and maximum acceleration for both SI-UAVs and RDs), collision-related dynamics (impact deceleration, UAV mass, and radius), and the core training parameters of the reinforcement learning algorithm (actor and critic learning rates, discount factor, soft-update coefficient, replay buffer size, batch size, update frequency, and the total number of training episodes).

Table 1. Parameter settings.

Attributes of Rounding Up	Variable Names	Parameters
Initial velocity of UAV	$v_{io}$	0 m/s
Maximum speed of SI-UAV	$v_{si-uav}$	15 m/s
Maximum speed of RD	$v_{rd}$	20 m/s
Maximum acceleration of SI-UAV	$a_{si-uav}$	$8{\mathrm{m}/\mathrm{s}}^2$
Maximum acceleration of RD	$a_{rd}$	$10{\mathrm{m}/\mathrm{s}}^2$
Impact deceleration of RD	$v_{ac}$	0.25 m/s
UAV mass	$m_{uav}$	1 kg
UAV radius	$d_{radius}$	0.5 m
Actor learning rate	${\alpha }_{\mathrm{actor}}$	0.01
Critic learning rate	${\alpha }_{\mathrm{critic}}$	0.01
Discounted rate	$\gamma$	0.95
Soft-update parameter	$\tau$	0.01
Replay buffer size	/	${10}^6$
Batch size	/	1024
Update frequency (samples per update)	/	100
Number of episodes	/	20,000

The number of UGVs in the task is always 4 with a constant speed of 2.5 m/s and initial positions of (−45, 45), (5, 45), (−45, −5), (5, −5). The paths of the first and third UGVs are the same, and the paths of the second and fourth UGVs are the same, and the path planning is shown in Figure 5.

Figure 5. UGV ground patrol path.

The blue solid line is the path of UGVs patrolling on the ground, and the orange dashed line is the return path after patrolling the respective area.

In each UAV training round, the round ends if any of the following occurs: the maximum speed of all RDs is reduced to 0, or the number of simulated steps reaches the maximum limit. At this point, the environment will be reset, and a new training round will begin. When the experience pool is full, the MADDPG algorithm enters the training state and updates the neural network parameters using the Adam optimizer.

5.2. Simulated Environment

The simulation experiments in this paper were conducted on a Windows 11 environment with Python 3.6 and TensorFlow 1.8, based on the MPE platform. This platform provides a multi-agent reinforcement learning environment, which has been modified and extended to design and train the UAV cluster adversarial model. The mission scenario is a two-dimensional finite square airspace under security surveillance, containing several SI-UAVs and RDs, aiming to enable multiple SI-UAVs to collaborate in neutralizing intrusive RDs for airspace protection.

In Section 5.3, the simulation environment for MPE-based multi-UAV cluster interactive maneuver decision-making is described. In this environment, red solid circles represent SI-UAVs, green solid circles represent RDs, and black solid circles represent the UGVs patrolling the ground. The SI-UAVs use the PSO-MADDPG hybrid decision-making algorithm for independent training, while the RDs use the MADDPG algorithm for their independent training. The escape process of the RD focuses mainly on avoiding interception by SI-UAVs by monitoring the distance between them.

5.3. Experimental Simulation Results

Based on the simulation environment and parameter settings mentioned above, this study employs two algorithms to train the SI-UAVs for the multi-UAV roundup maneuver decision model: the traditional MADDPG algorithm and the proposed PM-HDM algorithm. To comprehensively evaluate their performance, a series of experiments under different scenarios are designed for comparative analysis. It should be noted that during all training processes, the policies of both the RDs and SI-UAVs are generated by the actor–critic networks of the MADDPG algorithm and are learned autonomously through adversarial training. During subsequent visualization, as well as when detecting collisions and interception times, all policies are fixed. In each comparative experiment, the initial parameters and training mechanisms of the RDs are kept completely consistent. Figure 6 and Figure 7 illustrate the reward function curves of five SI-UAVs pursuing two RDs, trained with both algorithms without distance constraints.

Figure 6. PM-HDM reward curve (5–2).

Figure 7. Conventional MADDPG algorithm reward curve (5–2).

The horizontal axis of this graph represents the total number of training rounds, while the vertical axis shows the rewards for individual UAVs (averaged over multiple independent experiments with different random seeds). The curves in different colors indicate the reward changes of different UAVs: the first five solid curves represent the rewards of the five SI-UAVs, and the two dashed curves represent the rewards of the two RDs. Initially, there is no significant difference in the reward values between the two algorithms. The results from multiple independent experiments show that after 3000 rounds of training, the five SI-UAVs driven by PM-HDM achieved a reward value of approximately −352.07 ± 166.67, while the five SI-UAVs driven by traditional MADDPG achieved a reward value of approximately −937.79 ± 34.75. Although the standard deviation for PM-HDM is larger than that of traditional MADDPG, by the time the training reaches 5000 rounds, the standard deviation for PM-HDM decreases to a level comparable to that of the traditional MADDPG algorithm. In the long term, by 20,000 training rounds, the reward value for the five SI-UAVs driven by PM-HDM stabilizes at around −415.10 ± 45.14, while the traditional MADDPG algorithm stabilizes at around −1934.89 ± 70.41. Moreover, the reward values of the two RDs in the traditional MADDPG algorithm are also higher compared to PM-HDM, further demonstrating the effectiveness and robustness of PM-HDM.

In addition, as shown in the zoomed-in region, during the training rounds from 1000 to 4000, the five SI-UAVs driven by PM-HDM consistently engaged with the two RDs, leading to a significant decrease in RD rewards due to repeated interactions. In contrast, the SI-UAVs driven by traditional MADDPG failed to effectively engage with the RDs, resulting in no significant reduction in the RD’s reward values. This highlights the superior long-term optimization capabilities of PM-HDM compared to traditional MADDPG, a conclusion that is further supported by the changes in the reward curves after 5000 rounds.

As depicted in Figure 8 and Figure 9, the traditional MADDPG algorithm records a low or even zero number of collisions across multiple episodes. Here, “episode” refers to an independent test round conducted after training: once the training of all UAVs’ policies is completed, their strategies are fixed, and the UAV cluster is set to engage in 100 rounds of confrontation to generate the data presented. For instance, in Episode 3 (the third test round), only RD 1 achieves 22 collisions, while several rounds result in no collisions at all. This suggests the algorithm’s lack of global coordination, particularly when the target is dispersed and attempting to escape. The SI-UAVs often switch targets, leading to a sharp decline in effective collisions. In contrast, the PM-HDM algorithm consistently records a high number of collisions in all episodes. In fact, the collisions of RD 2 surpass 200 in some episodes (e.g., 281 collisions in Episode 4 and 471 in Episode 94), which demonstrates the effectiveness of its target allocation mechanism. This mechanism minimizes decision hesitation and facilitates more intense interception. Furthermore, Table 2 presents the average number of interceptions of the RDs for both algorithms, further highlighting the superior target allocation and collaborative pursuit capabilities of PM-HDM.

Figure 8. Collision count curve for PM-HDM (5–2).

Figure 9. Collision count curve for conventional MADDPG algorithm (5–2).

Table 2. Collision average (5–2).

Algorithm	RD1	RD2
MADDPG	7.95	1.12
PM-HDM	25.15	48.13

In terms of interception performance, it was observed after multiple experimental trials that the traditional MADDPG algorithm exhibits chaotic target assignment due to the absence of a definite pursuit objective. As a result, by the end of each episode, the two RDs only experience a slight reduction in velocity, typically within $0–4\mathrm{m}/\mathrm{s}$. In contrast, under the PM-HDM algorithm, the two RDs suffer significantly greater deceleration: one RD shows a velocity decrease of $5–10\mathrm{m}/\mathrm{s}$, while the other completely loses maneuverability before the end of the episode. This outcome further confirms the higher interception efficiency and coordinated pursuit capability achieved by the PM-HDM algorithm.

In addition to the comparison between MADDPG and PM-HDM, we conducted further experiments comparing the PSO-based PM-HDM algorithm with a Greedy strategy for target allocation, where the benchmark is minimizing the total distance from all SI-UAVs to the RDs in the allocation results. The results of these experiments, based on 100 trials with random positions for the SI-UAVs and RDs, demonstrate the advantages of PSO in achieving more efficient and stable task allocation.

In these trials, PSO outperformed the Greedy strategy in 77% of the cases, while 22% of the trials showed that PSO performed equally as Greedy. Only 1% of the trials showed that PSO performed worse than Greedy. On average, PSO achieved a total distance of 197.77, which is significantly lower than Greedy’s average total distance of 217.07. Additionally, PSO demonstrated a smaller standard deviation (51.59 vs. 57.69), indicating greater stability and efficiency in task allocation.

The comparison of PSO and Greedy strategies is summarized in Table 3.

Table 3. Comparison of PSO and Greedy strategies (5–2).

Result	PSO	Greedy
PSO Better	77%	-
PSO Equal to Greedy	22%	-
PSO Worse	1%	-
Average Total Distance (m)	197.77	217.07
Standard Deviation (m)	51.59	57.69

Figure 10 and Figure 11 show the results of the simulation experiments of both algorithms after completing the training and after visualization of the UAV without battery power limitation.

Figure 10. PM-HDM confrontation process (5–2).

Figure 11. Traditional MADDPG algorithm confrontation process (5–2).

The red curve in the figure represents the motion trajectory of the SI-UAV, the green curve represents the RD’s trajectory, and the black solid circle represents the UGV, which patrols the ground based on the routes planned in the first section. From the UAV trajectories in both figures, we can see that in the traditional MADDPG algorithm, the average distance between the SI-UAVs and RDs is greater. This is due to the target-locking flaw in each UAV. When two RDs escape in the same direction, the SI-UAVs driven by MADDPG can still maintain basic pursuit. However, at the crucial moment when the RDs scatter and escape, the lack of a target allocation mechanism causes the SI-UAVs to fall into a Nash equilibrium dilemma. In contrast, PM-HDM introduces a target assignment module via the PSO algorithm, allowing each SI-UAV to maintain a clear purpose. When the RDs disperse and escape, the SI-UAVs ensure that the distance between them remains relatively small, ultimately forming an effective interception formation during the border interception phase.

To address the SI-UAV endurance limitations in practical airspace security operations, an intelligent air–ground collaborative replenishment system is constructed. The ground UGV cluster not only undertakes regional patrol tasks but also serves as mobile charging station nodes within a dynamic energy supply network. The experiment adopts a distance-threshold-based energy management model, where all SI-UAVs are configured with a 200-m flight range threshold. In the experimental setup with return replenishment, once an SI-UAV’s cumulative flight distance reaches 200 m, its navigation system takes over control, locates the nearest available UGV node, and guides the SI-UAV along a shorter route to the UGV for energy replenishment. After refueling, control is returned to the PM-HDM algorithm, allowing the SI-UAV to resume tracking of the RD. Figure 12 and Figure 13 compare the simulation results of scenarios with and without return replenishment under the SI-UAV range limitation condition.

Figure 12. Non-return replenishment mechanism.

Figure 13. Return replenishment mechanism.

In the experiment without the return mechanism, when the number of training steps reaches between 350 and 400, the SI-UAVs’ speed decreases after hitting the distance limit. This results in increased distance from the RDs, making it impossible for the SI-UAVs to catch up, which ultimately hinders the mission’s success. In contrast, in the experiment with the return mechanism, four SI-UAVs also reach the distance limit between steps 350 and 400, triggering the return mechanism. These UAVs resupplied at step 450, resumed pursuit, and were able to approach the RDs again at step 500. This highlights the importance of the return-to-recharge mechanism. Without it, once a UAV hits the distance threshold, it permanently loses maneuverability, leading to significant losses and creating vulnerabilities in the defense system. On the other hand, the experimental group with the return mechanism successfully enables dynamic resource recycling through the synergy of the intelligent scheduling system and rapid replenishment. To further evaluate the generalization performance of PM-HDM, this study extends the experimental scenarios to assess its capability in handling multi-target intrusion. Figure 14 and Figure 15 compare the training effects of seven SI-UAVs intercepting three RDs.

Figure 14. PM-HDM reward curve (7–3).

Figure 15. Conventional MADDPG algorithm reward curve (7–3).

It can be observed that even when the number of attacking drones is increased to seven, the reward values of PM-HDM-driven attacking drones remain higher than those exhibited by the traditional MADDPG algorithm. Between 1000 and 4000 training rounds, PM-HDM shows similar effects to those in previous experiments, where the SI-UAVs gradually decrease the reward values of the RDs while increasing their own reward values during the optimization process. In contrast, the traditional MADDPG algorithm does not significantly improve the reward values of the SI-UAVs during training, nor does it cause a significant decrease in the reward values of the RDs. Additionally, in the traditional MADDPG experiment, when the number of RDs increases to three—coupled with the SI-UAVs’ inability to maintain stable tracking of their assigned targets—the RDs often disperse and evade successfully after the training phase. This leaves the SI-UAVs in a prolonged Nash equilibrium, unable to effectively pursue the intruding RDs, thereby allowing the RDs to maintain high reward values in the later stages of the experiment.

In the scenario with seven SI-UAVs and three RDs, Figure 16 and Figure 17 illustrate that PM-HDM significantly outperforms the traditional MADDPG algorithm in terms of the number of collisions between attacking and RDs. Under the traditional MADDPG algorithm, the number of collisions fluctuates significantly, and effective collisions are often not achieved, highlighting its weak target allocation and coordination capabilities in a multi-target environment. In contrast, PM-HDM improves target assignment by integrating PSO, ensuring that each UAV has a clear target and minimizing decision hesitation. This results in consistently high collision counts across all rounds, demonstrating superior task orientation, synergy, robustness, and tactical adaptability, particularly when responding to multiple RDs. Table 4 shows the average number of collisions for each RD, further emphasizing the growing gap between the traditional MADDPG algorithm and PM-HDM as the number of RDs increases. This illustrates that the drawbacks of the traditional MADDPG algorithm become more pronounced with more RDs.

Figure 16. Collision count curve for PM-HDM (7–3).

Figure 17. Collision count curve for conventional MADDPG algorithm (7–3).

Table 4. Collision average (7–3).

Algorithm	RD1	RD2	RD3
MADDPG	0.41	1.33	1.9
PM-HDM	19.37	16.49	28.02

In terms of interception performance, observations from multiple experimental trials indicate that, under the traditional MADDPG algorithm, the three RDs exhibit only a slight reduction in velocity by the end of each episode, typically within $0–2\mathrm{m}/\mathrm{s}$. In contrast, under the PM-HDM algorithm, the deceleration of the three RDs is significantly greater, generally ranging from $8–15\mathrm{m}/\mathrm{s}$. This result further confirms the superior interception efficiency and coordinated pursuit capability achieved by the PM-HDM algorithm in the scenario involving seven SI-UAVs and three RDs.

In the seven SI-UAV vs. three RD scenario, similar to the previous 5–2 experiment, we conducted a further comparison of the PSO-based PM-HDM algorithm with the Greedy nearest-neighbor allocation strategy. This comparison was also based on 100 randomized trials with different initial positions for the SI-UAVs and RDs, allowing us to assess the efficiency and stability of task allocation in a larger swarm setting.

In these trials, PSO strictly outperformed the Greedy strategy in 93% of the cases, while no trials resulted in equal performance between PSO and Greedy. Only 7% of the trials showed that PSO performed worse than Greedy. On average, PSO achieved a total assignment distance of 256.23 m, which is substantially lower than Greedy’s average total distance of 300.04 m. The standard deviation of total distance for PSO (69.08 m) is also smaller than that for Greedy (81.09 m), indicating that PSO not only reduces travel distance but also yields more stable assignments under random initializations. The numerical comparison is summarized in Table 5.

Table 5. Comparison of PSO and Greedy strategies (7–3).

Result	PSO	Greedy
PSO Better	93%	-
PSO Equal to Greedy	0%	-
PSO Worse	7%	-
Average Total Distance (m)	256.23	300.04
Standard Deviation (m)	69.08	81.09

Figure 18 and Figure 19 present the visualized simulation results of both algorithms in dynamic airspace interception scenarios. Analysis of these two figures reveals that the PM-HDM algorithm achieves effective dynamic tracking of targets: regardless of the evasion tactics employed by the RDs, it maintains a close distance to the intruding RDs and initiates interception actions once within effective range. In contrast, with the increased number of RDs, the traditional MADDPG algorithm can still pursue one or two targets at the initial stage of confrontation. Yet once the three RDs disperse in three directions in the later stages, traditional MADDPG clearly fails to complete the interception task due to its inherent limitations. This comparison confirms that by integrating the PSO strategy, PM-HDM markedly enhances the target allocation efficiency and cooperative performance of multi-agent systems in complex confrontation environments. It performs exceptionally well in addressing multi-target dynamic game scenarios, exhibiting greater robustness and tactical flexibility.

Figure 18. PM-HDM confrontation process (7–3).

Figure 19. Traditional MADDPG algorithm confrontation process (7–3).

5.4. Conclusions

In this paper, the effectiveness and robustness of PM-HDM in UAV cluster defensive operations for airport security are verified through simulation experiments. The results demonstrate that the algorithm achieves more efficient target assignment and coordinated interception: it quickly allocates RDs to SI-UAVs while maintaining stable interception performance—critical for countering unauthorized intrusions in restricted airport airspace—and overcomes the limitations of traditional MADDPG algorithms in multi-target defensive scenarios. Furthermore, PM-HDM exhibits stable optimization during training, with reward values increasing steadily and minimal fluctuations, ensuring consistent response efficiency even under dynamic changes, such as sudden RD intrusions near runways or terminal perimeters. The integration of the air–ground cooperative replenishment mechanism effectively addresses the SI-UAV endurance constraint, significantly improving task efficiency and operational sustainability for round-the-clock airport security operations.

Due to the complexity and resource constraints of conducting full-scale UAV experiments, this study currently relies on simulation scenarios that approximate real-world airport operations. For future work, we plan to validate PM-HDM through controlled physical experiments. Specifically, UAVs and UGVs will be appropriately modified to meet experimental requirements, enabling controlled flight and operational tests, while laboratory setups will simulate key operational areas of an airport. These experiments will allow assessment of the algorithm’s real-world performance, reliability, and operational feasibility, further strengthening its applicability in practical airport security systems.

In summary, PM-HDM provides an efficient and stable solution for UAV cluster defensive tasks tailored to airport security, and its future validation through physical experiments will enhance its potential for practical deployment, ensuring a robust defense against unauthorized aerial intrusions and safeguarding airport operations.

5.5. Analysis of Dynamic Reallocation Limitations and Future Directions

A core limitation of the current PM-HDM design is that the PSO-based target allocation is performed once at the start of each episode and is not updated during the episode. In highly dynamic scenarios this single-shot assignment can be insufficient. Specifically, three classes of runtime events are not addressed by the present implementation:

• Arrival of new threats: If new intruders appear after episode start (i.e., the number of RDs changes), no runtime reallocation is provided.
• Target neutralization: When an assigned RD is neutralized, the reassignment of its former SI-UAVs is undefined.
• Agent unavailability: If an SI-UAV temporarily leaves the mission (e.g., return-to-base, loss, or communication delay), its tasks remain unattended.

To improve robustness, we propose and plan to evaluate the following concise dynamic reallocation mechanisms.

5.5.1. Reserve Agents

Keep a small set of uncommitted reserve SI-UAVs that share the same policy parameters as active agents. Reserves immediately assume assignments when new threats appear, agents fail, or targets are neutralized. This approach requires minimal online computation but slightly reduces the initial active force (i.e., fewer agents at $t=0$).

5.5.2. Event-Triggered Incremental Reallocation

Favor light-weight, event-driven responses to bound runtime cost. For example, the system can employ local greedy repair, which assigns the nearest available SI-UAV(s) to newly arrived or vacated tasks (complexity $O\left(n\right)$); warm-started PSO, which for larger changes runs a short PSO instance initialized from the previous assignment to accelerate convergence toward a globally informed solution; and a priority-queue plus negotiation mechanism, which maintains a ranked task queue (by threat level or distance) and enables agents to locally renegotiate assignments using predefined tie-breaking rules to support decentralized responsiveness when central computation is costly or delayed.

5.5.3. Implementation Trade-Offs

Each strategy trades off computation, responsiveness, and optimality: reserves and greedy repair maximize responsiveness with minimal compute but may be suboptimal; warm-started PSO yields better global allocations at higher but controlled cost; and decentralized negotiation increases fault-tolerance but requires reliable communication and conflict resolution.

5.5.4. Planned Evaluation

We will (i) instrument the simulator to generate the three event types above, (ii) implement reserve-agent and warm-started PSO variants, and (iii) compare these methods on mission success (collision/neutralization counts), reallocation latency, and additional computational overhead.

In summary, the one-shot PSO allocation is a modelling simplification. We consider hybrid schemes—reserve agents for fast failover, local repairs for trivial events, and warm-started PSO for significant changes—to be a pragmatic roadmap. These incremental developments will be evaluated to guide the evolution of PM-HDM toward a dynamically adaptive system suitable for continuously changing operational environments.