A Graph-Aided Hierarchical Decision Framework for UAV Swarm Interception Under Saturation Incursions

Highlights

What are the main findings?

• The proposed HM-MF-DDPG framework achieves a 93% interception success rate in large-scale UAV swarm saturation incursion scenarios (50 Blue vs. 25 Red), significantly outperforming DDPG and MF-DDPG baselines in both convergence speed and task efficiency.
• The graph-enhanced weighted mean field approximation with attention mechanisms effectively breaks the homogeneity assumption of traditional mean-field methods, enabling more accurate modeling of heterogeneous agent interactions in dynamic adversarial environments.

What are the implications of the main findings?

• The hierarchical multi-scale decision architecture provides a scalable and practical solution for real-time cooperative interception under saturation incursions, supporting robust task allocation and collision-free flight control without central coordination.
• The framework’s strong generalization across varying no-fly zone configurations and swarm scales demonstrates its potential for deployment in complex, real-world UAV swarm defensive operations under diverse tactical conditions.

Abstract

The interception of saturation incursions by Unmanned Aerial Vehicle (UAV) swarms presents critical challenges in multi-agent coordination, including the curse of dimensionality, heterogeneous interaction effects, and multi-scale decision-making requirements. This paper proposes a Hierarchical Multi-scale Mean-Field DDPG (HM-MF-DDPG) framework augmented by graph sampling and aggregation networks to address these challenges. The framework introduces three key innovations: (1) a graph-enhanced weighted mean-field approximation that employs attention mechanisms to dynamically assess the contextual importance of neighboring agents, overcoming the homogeneity limitation of conventional mean-field methods; (2) a hierarchical decision architecture that separates strategic coordination (via graph attention networks) from low-level flight control (via improved gated recurrent units with situational awareness modulation); and (3) a distributed target assignment mechanism formulated as a potential game and solved via parallel auction algorithms, enabling collision-free allocation without central coordination. Extensive simulations in a constructed UAV swarm interception environment demonstrate that the proposed framework achieves a 93% interception success rate with 50 interceptors against 25 intruders, outperforming Deep Deterministic Policy Gradient (DDPG) and Mean-Field DDPG (MF-DDPG) baselines in both convergence speed and task efficiency. The framework exhibits robust generalization across varying No-Fly Zone (NFZ) configurations and swarm scales, providing a scalable solution for cooperative interception under saturation incursions.

1. Introduction

The rapid advancement of Unmanned Aerial Vehicle (UAV) swarm technology is profoundly transforming operational paradigms, driving the evolution of multi-agent operations from single-platform independent operations toward distributed and collaborative swarm-based engagements [1]. Within this trajectory, the employment of UAV swarms to execute coordinated swarm intrusions have emerged as a significant operational pattern, demonstrating considerable effectiveness in practical operation scenarios [2]. Consequently, the development of effective cooperative interception strategies against incoming UAV swarms executing saturation incursions has become a critical challenge for aerial protection systems, with its importance increasingly accentuated [3,4]. Saturation incursion refers to a deliberate large scale coordinated intrusion where the number of simultaneously attacking UAVs significantly exceeds the defender’s available instantaneous engagement channels, aiming to overwhelm and saturate the defense system. This task necessitates that defenders also employ UAV swarms to achieve efficient tracking, interception, and interception of adversarial agents. The core challenges reside in distributed real-time decision-making, dynamic target allocation, and the maintenance of swarm cohesion under strongly adversarial conditions [5,6].

In the domain of cooperative decision-making for UAV swarms, numerous scholars have proposed diverse solutions from various perspectives. Du et al. noted that traditional Multi-Agent Reinforcement Learning (MARL) algorithms are often suitable for small-scale cooperative tasks but frequently encounter limitations when applied to large-scale swarms due to the high-dimensional state-action spaces and the challenges posed by partially observable environments [7]. In recent years, Deep Reinforcement Learning (DRL) methods, such as Deep Q-Networks (DQN) and Actor-Critic frameworks, have significantly enhanced algorithmic scalability. However, further exploration is still required in areas like multi-agent cooperative mechanism design, reward function construction, and environmental modeling. For instance, He et al. addressed the joint optimization problem of task allocation, communication base station association, and flight strategy for multi-UAV distributed sensing scenarios [8]. They proposed an algorithm based on multi-agent independent learning and a composite action Actor-Critic, modeling the complex mixed-integer nonlinear programming problem as a cooperative Markov game. This approach enabled cooperative optimization of UAV sensing, communication, and flight strategies through distributed Reinforcement Learning (RL), achieving favorable outcomes in system utility and energy consumption control. Gong et al. focused on UAV cooperative air operation and tactical interception tasks, employing value decomposition networks and independent Q-learning to design Centralized Training and Distributed Execution (CTDE) strategies [9]. By incorporating formation maneuver tactics and interception line assessment mechanisms, dynamic target allocation and cooperative interception were achieved. Nevertheless, this algorithm is primarily applicable to small-scale formation engagements and struggles to scale to large-scale swarm operations. Lowe et al. introduced the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, employing a CTDE framework to stabilize training, demonstrating strong performance in mixed cooperative-competitive environments [10]. Addressing the challenge of UAV swarm cooperative exploration in unknown environments, Wang et al. proposed a path planning method based on Multi-Agent Soft Actor-Critic (MASAC). They constructed a reward function using breadth-first search to guide exploration and leveraged convolutional neural networks to process grid map information, exhibiting promising generalization capabilities and cooperative efficiency in collaborative exploration tasks [11].

Recently, Graph Neural Networks (GNNs) have gained traction in MARL for UAV swarms due to their ability to model complex relational structures. Yang et al. proposed the Graph-based Policy Mean Field (GPMF) algorithm, which integrates a Graph Attention network (GAT) with partially observable mean field theory to better capture the influence of neighboring agents’ feature information, outperforming standard mean field methods in large-scale UAV cooperative roundup tasks [12]. Similarly, research on collaborative multi-target tracking has demonstrated that GNN-based frameworks can effectively handle the time-varying dimensions of agent observation spaces and enable efficient inter-agent communication, leading to higher tracking success rates and fairness. These studies underscore the potential of combining graph-based relational modeling with MARL for complex swarm coordination problems.

In the specific area of target allocation, auction-based mechanisms have proven to be pivotal distributed solution approaches for UAV swarm task allocation problems. Chen et al. introduced a distributed task allocation method grounded in swarm benefits optimization, which quantifies individual benefit variation to directly optimize overall swarm utility [13]. This method, incorporating spatial distance, task value, and temporal decay, demonstrated superior performance compared to consensus-based algorithms like Consensus-Based Bundle Algorithm (CBBA). The integration of such auction mechanisms within learning frameworks offers a promising path toward dynamic and conflict-free resource allocation.

Despite significant progress in MARL, substantial challenges persist when addressing large-scale UAV swarm cooperative tasks. As the number of agents increases, the joint state and action spaces expand exponentially, leading to the so-called “curse of dimensionality” [14,15]. This renders many traditional MARL algorithms computationally intractable [16,17]. While independent learning can alleviate this complexity to some extent, it often fails to capture the behavioral interdependencies crucial for effective swarm coordination. The CTDE framework, exemplified by MADDPG, offers a compromise; however, its scalability remains limited when the number of interacting agents grows substantially [18,19].

To tackle scalability issues, current mainstream approaches often introduce mean field theory, approximating the influence of other agents through a single mean effect [12,20]. The foundational work by Yang et al. on mean field MARL demonstrated its efficacy in simplifying large-scale agent interactions [12]. However, these methods are typically predicated on a homogeneity assumption, treating all neighboring agents as equally influential. This presents a significant limitation in practical UAV swarm confrontation scenarios. In reality, the impact of different agents on cooperative outcomes varies considerably based on their relative positions, capability states, and tactical roles. For instance, an adversarial UAV on a direct collision course poses a vastly different threat level compared to a distant target [21]; similarly, a friendly unit with ample munitions in a critical position holds substantially higher cooperative value than a damaged or peripheral unit [14]. Existing mean field approximations struggle to capture such heterogeneity, potentially leading to suboptimal swarm policies.

Furthermore, pursuit-interception problems inherently exhibit multi-temporal and multi-spatial scale characteristics: at the microscopic level, agents must execute flight control maneuvers (e.g., acceleration, turning); at a macroscopic tactical scale, they need to implement maneuvers such as encirclement, pincer movements, and target selection; over an extended operational timeline, they must adhere to high-level strategic intents like area denial and interception destruction. Most existing MARL methods operate on a single fixed temporal scale for decision-making, failing to organically integrate these interrelated yet distinct layers [22,23]. This often results in myopic policies that perform well in immediate tasks but struggle to support long-term, holistic operational objectives [24,25,26]. The need for hierarchical architectures that can manage different decision scales is echoed in recent surveys on autonomous and intelligent UAV swarms, which highlight the importance of frameworks capable of bridging intuition and data across multiple levels of abstraction.

Addressing these challenges, this paper proposes a novel hierarchical multi-scale mean field DDPG framework augmented by graph sampling and aggregation networks for large-scale UAV swarm cooperative pursuit-interception tasks. The primary contributions of this work are threefold:

(1)

Graph-Enhanced Mean Field Approximation: We introduce an attention mechanism based on Graph Sample and Aggregation (GraphSAGE) to enhance traditional mean field methods. Instead of employing simple averaging, our model learns to compute a weighted mean field—dynamically assigning influence weights based on each neighbor’s encoded features and their contextual relationships within the swarm communication graph. This enables more accurate and context-aware modeling of multi-agent interactions, effectively mitigating the dimensionality catastrophe while preserving the richness of inter-agent dependencies.

(2)

Hierarchical Multi-Scale Decision Architecture: We design a hierarchical policy structure that explicitly separates and manages decision-making processes across different temporal and spatial scales. The high-level layer, based on a graph attention network, is responsible for strategic swarm coordination and dynamic target allocation. The low-level layer employs an improved Gated Recurrent Unit (GRU) actor network, incorporating a situational awareness modulation mechanism to achieve refined individual UAV motion and flight control. The enhancement of the GRU strengthens the agent’s capability to model long-term temporal dependencies and predict future trajectories, which is crucial for the success of pursuit-evasion tasks.

(3)

Distributed Target Allocation via Dynamic Potential Game: We formulate the dynamic target allocation problem as a potential game model and solve it efficiently using a distributed auction algorithm. This model enables UAVs to perform distributed, collision-free target assignment without central coordination, thereby maximizing the overall swarm operational utility in adversarial settings.

The remainder of this paper is organized as follows: Section 2 describes the environmental modeling for the UAV swarm interception pursuit problem. Section 3 elaborates on the distributed auction-based target allocation strategy. Section 4 details the design of the GraphSAGE-based hierarchical multi-scale mean field DDPG algorithm. Section 5 presents the experimental setup, results, and comparative analysis. Section 6 concludes the paper by summarizing the innovations and significance of this work and outlining future research directions.

2. Task Scenario and Situational Awareness Modeling

2.1. Task Scenario

The task scenario is established within a bounded two-dimensional simulated airspace, wherein Red UAVs are tasked with approaching Blue High-Value Assets (HVAs), while Blue UAVs perform defensive interception operations. The operational environment contains multiple No-Fly Zones (NFZs) representing terrain obstacles or restricted airspace that impose navigational constraints on all aerial platforms. Crucially, these NFZs must not be encroached upon by either Red or Blue UAVs, with violations resulting in mission failure. Red UAVs must therefore plan penetration routes that evade both NFZ constraints and Blue UAVs interception efforts, while Blue UAVs must simultaneously pursue Red UAVs intruders and maintain NFZ avoidance, thereby facing identical environmental restrictions. This unified constraint framework ensures that both swarms integrate terrain awareness and obstacle avoidance into their tactical decision-making, reflecting realistic operational conditions where physical and synthetic barriers affect all agents equally. The specific task configuration is illustrated in Figure 1.

Figure 1. Schematic diagram of task scenario.

2.2. Perception and Engagement Models

In the following, let ${\mathbf{x}}_i$ denote the position of UAV $i$ in three-dimensional space, and similarly ${\mathbf{x}}_j$ for UAV $j$. The Euclidean distance between them is denoted by $\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel$, where $\parallel \cdot \parallel$ represent the standard $L_2$ norm.

Radar Detection Model: A probabilistic detection model is employed. Each UAV $i$ is equipped with a radar sensor. The probability that UAV $i$ detects UAV $j$ is given by

$$P_{detect}\left(i,j\right)=\left\{\begin{array}{c}1-exp\left(-{\displaystyle \frac{d_0^2}{{‖{\mathbf{x}}_i\left(t\right)-{\mathbf{x}}_j\left(t\right)‖}^2}}\right), if\left|\left|{\mathbf{x}}_i-{\mathbf{x}}_j\right|\right|\le R_{radar}\\ 0 , otherwise\end{array}\right.$$

(1)

where $R_{\mathrm{radar}}$ is the maximum detection range, $d_0$ is a detection capability parameter characterizing the sensitivity at close range. The detection probability employs an exponential decay. This formulation is motivated by the radar range equation: the received signal-to-noise ratio (SNR) falls off approximately as $1/\parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }^4$ for free-space propagation, and for a given detection threshold the probability of detection is a monotonically decreasing function of distance. The exponential model is a widely used continuous approximation that captures the smooth degradation of sensing performance with range while maintaining analytical tractability, as adopted in prior UAV swarm simulation studies.

Communication Model: Communication between any two UAVs $i$ and $j$ occurs over a finite distance. The condition for establishing communication is

$$‖{\mathbf{x}}_i\left(t\right)-{\mathbf{x}}_j\left(t\right)‖\le R_{comm}$$

(2)

where $R_{\mathrm{comm}}$ is the maximum communication range. The communication model adopts a deterministic binary condition: two UAVs can communicate if their distance is within $R_{comm}$. This is a standard abstraction derived from the fact that a communication link is established when the received signal power exceeds a sensitivity threshold, which translates directly to a maximum range under a given transmit power and propagation environment. The binary model is appropriate for our focus on decision-making strategies, as it simplifies information exchange without sacrificing the key structural property of limited connectivity.

Engagement Success Probability Model: A probabilistic engagement model is constructed based on UAV engagement characteristics. The probability of successfully engaging a target UAV $j$ by UAV $i$ is

$$P_{engage}\left(i,j\right)=\left\{\begin{array}{c}{p}_{0}exp\left(-{\displaystyle \frac{\left|\left|{\mathbf{x}}_i-{\mathbf{x}}_j\right|\right|}{d_{eff}}}\right), if\left|\left|{\mathbf{x}}_i-{\mathbf{x}}_j\right|\right|\le R_{engage}\\ 0 , otherwise\end{array}\right.$$

(3)

where $R_{\mathrm{engage}}$ is the maximum engagement range, $p_0$ is the baseline engagement success probability (typically achieved at short range), and $d_{\mathrm{eff}}$ is the engagement effectiveness decay distance. The engagement success probability is modeled linearly decreasing from a baseline $p_0$ at zero distance to zero at the maximum effective engagement range $d_{eff}$. This linear form captures the principle that close-range engagements are more reliable due to reduced kinematic error, shorter time-of-flight, and lower chance of evasion. Similar linear lethality functions are commonly used in weapon effectiveness analyses and provide a transparent relationship between engagement geometry and success likelihood, which aids interpretability in multi-agent reinforcement learning.

2.3. Situational Assessment Model

The primary task of the Blue UAVs is to prioritize the interception and neutralization of Red UAVs that are easier to destroy. By inputting real-time state information of both sides, the situational advantage of a Blue UAV relative to a Red UAV is quantitatively assessed. This assessment provides reference inputs for the potential functions involved in the situational awareness computation within high-level decision-making, thereby facilitating intelligent target allocation.

This situational assessment model focuses on analysis from three key dimensions. Range advantage evaluates the relative distance between Blue and Red UAVs, prioritizing targets with lower approach costs. Angular advantage analyzes the relative azimuth and entry angles to ensure the Blue UAV can occupy a favorable attack posture. Speed advantage compares the velocity vectors of both sides to determine the capability for timely interception or effective pursuit. The system comprehensively considers the position and velocity information of both Red and Blue UAVs, as illustrated in Figure 2.

Figure 2. Schematic diagram of situation assessment.

In Figure 2, $d_{ij}$ represents the distance between Blue UAV $i$ and Red UAV $j$, ${\theta }_{ij}$ is the azimuth angle of Red UAV $j$ relative to Blue UAV $i$, $v_i$ is the velocity of Blue UAV $i$ and $v_j$ is the velocity of Red UAV $j$. The core of the assessment is the computation of a comprehensive situational advantage value $S_{ij}$. A higher value indicates a greater advantage for Blue UAV $i$ in intercepting Red UAV $j$. The comprehensive situational advantage is calculated as

$$S_{ij}=w_d\cdot D_{ij}+w_{\mathrm{\theta }}\cdot A_{ij}+w_v\cdot V_{ij}$$

(4)

where $w_d,w_{\theta },w_v$ are the weight coefficients, $D_{ij}$, $A_{ij}$, and $V_{ij}$ are the normalized advantage values, each ranging within $[0,1]$.

The distance advantage function $D_{ij}$ for a Blue UAV $i$ relative to Red UAV $j$ is designed as follows:

$$D_{ij}=e^{-d_{ij}}$$

(5)

As the distance decreases, the advantage increases, and the value of $D_{ij}$ becomes larger.

The angular advantage function $A_{ij}$ for a Blue UAV $i$ relative to Red UAV $j$ is considered only when the relative angle ${\theta }_{ij}$ is within the range $[-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}]$:

$$A_{ij}=\left\{\begin{array}{ll}{e}^{-\left|{\theta }_{ij}\right|}& \left|{\theta }_{ij}\right|<{\displaystyle \frac{\pi }{2}}\\ 0& {\displaystyle \frac{\pi }{2}}<\left|{\theta }_{ij}\right|<\pi \end{array}\right.$$

(6)

The smaller the relative angle ${\theta }_{ij}$ between the Blue and Red UAVs, the greater the angular advantage $A_{ij}$.

The speed advantage function $V_{ij}$ for a Blue UAV $i$ relative to Red UAV $j$ is defined based on the speed ratio:

$$V_{ij}=\left\{\begin{array}{l}0.25\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{v}_i\le 0.5v_j\\ {\displaystyle \frac{v_i}{v_j}}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}0.5v_j<v_i<2v_j\\ 1\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{v}_i\ge 2v_j\end{array}\right.$$

(7)

When the speed ratio between the Blue UAV $i$ and the Red UAV $j$ lies between 1:2 and 2:1, the speed advantage of the Blue UAV is proportional to the speed ratio. This formulation aligns with principles used in adaptive collision avoidance and pursuit-evasion scenarios, where maintaining an advantageous speed relative to a dynamic target is critical for successful interception.

3. Target Allocation Model

To achieve comprehensive and efficient interception of the Red UAV swarm by the Blue UAV swarm, this paper constructs a collaborative target allocation mechanism based on a hierarchical RL decision framework. This mechanism comprises a high-level strategic layer and a low-level execution layer. The high-level layer is responsible for dynamic and cooperative interception task assignment at the global level, which is formulated as a potential game problem aimed at maximizing defense effectiveness. This problem is solved via a distributed auction algorithm to accomplish target allocation. The auction algorithm, a distributed optimization method grounded in benefit maximization, enables each Blue UAV to compute interception benefits in parallel and resolve target conflicts through iterative bidding and negotiation, ultimately achieving maximization of overall defense effectiveness.

3.1. Interception Benefit and Optimal Target Selection

Each Blue UAV $i$ computes the interception benefit $b_{ij}$ for each target $j$ based on local observations and global situational information. The benefit function comprehensively considers interception success probability, target threat value, and interception time, formulated as

$$b_{ij}=({\omega }_1·p_{ij}+{\omega }_2·v_{ij}-{\omega }_3·t_{ij})·(1+{\alpha }_{ij})$$

(8)

where

$p_{ij}$ denotes the interception success probability of UAV $i$ against target $j$ (derived from relative position, velocity, and other factors);

$v_{ij}$ represents the threat value of target $j$ (determined by threat level, type, etc.);

$t_{ij}$ is the estimated interception time (i.e., the time required for UAV $i$ to intercept target $j$);

${\omega }_1,{\omega }_2,{\omega }_3$ are weight coefficients that balance the importance of different factors;

${\alpha }_{ij}$ is the GAT attention weight (see Section 4.2), which dynamically adjusts the interception benefit based on cooperative relationships between UAVs.

Each UAV $i$ computes the benefits for all targets in parallel and selects the target with maximum benefit as its initial bid object:

$$j_i^{*}=arg\underset{j}{\max }{b}_{ij}$$

(9)

UAV $i$ then transmits a bid message to the swarm communication network, declaring its bid for target $j_i^{\ast }$.

3.2. Distributed Negotiation and Conflict Resolution

During the distributed auction process, the system maintains a target allocation status vector:

$$A=\left[a_1,a_2,\dots ,a_M\right]$$

(10)

where $a_j$ represents the current status of target $j$. $a_j=0$: target unallocated; $a_j=i$: target allocated to UAV $i$; $a_j=\mathrm{pending}$: target under negotiation.

The system also maintains a benefit matrix $\mathbf{B}\in {\mathbb{R}}^{N\times M}$, where $N$ is the number of Blue UAVs and $M$ is the number of Red UAVs. The element $\mathbf{B}\left[i\right]\left[j\right]$ represents the benefit of Blue UAV $i$ intercepting Red UAV $j$:

$$\mathbf{B}={\left[b_{ij}\right]}_{N\times M}$$

(11)

When multiple UAVs bid for the same target $j$, the system automatically identifies the conflict set:

$$C_j=\{i\in I|j_i^{*}=j and b_{ij}>\mathrm{\theta }\}$$

(12)

where $\theta$ is the benefit threshold, ensuring that only valid bids participate in conflict resolution. For conflicting target $j$, arbitration is performed using a weighted comprehensive benefit:

$$W_{ij}={\lambda }_1·b_{ij}+{\lambda }_2·c_{ij}+{\lambda }_3·r_{ij}$$

(13)

where $c_{ij}={\displaystyle \frac{1}{t_{ij}+\mathsf{ϵ}}}$ is the interception urgency factor; $r_{ij}={\displaystyle \frac{E_i}{{\displaystyle \sum _{k\in C_j}{E}_k}}}$ is the resource suitability factor ($E_i$ denotes the interception effectiveness of UAV $i$); and ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are weight coefficients.

The distributed arbitration algorithm employs an iterative bidding mechanism. First, benefit announcements are posted: each participating UAV $i(i\in C_j)$ broadcasts its weighted benefit message:

$$m_i=\left(i,j,W_{ij},t_{\mathrm{bid}}\right)$$

(14)

Each UAV receives all bid information and computes

$$i_{win}=arg\underset{k\in C_j\cup i}{\max }{W}_{kj}$$

(15)

If $i_{\mathrm{win}}=i$, the UAV declares itself the winner; otherwise, it awaits the winner’s declaration.

A majority voting mechanism is employed to confirm the winner:

$$V_{\mathrm{confirm}}=\left\{k\in C_j|\mathrm{acknowledges}\hspace{0.33em}{i}_{\mathrm{win}}\right\}$$

(16)

The winning condition is

$${\displaystyle \frac{\mid V_{\mathrm{confirm}}\mid }{\mid C_j\mid }}\ge \gamma$$

(17)

where $\gamma$ is the consensus threshold (typically $\gamma >0.5$)

For unsuccessful UAVs $i$ target reselection is required. The available target set is updated as

$$J_{available}=J\backslash \left\{j\in J|a_j\ne 0\right\}$$

(18)

When recalculating benefits, opportunity costs are considered:

$$b_{ij}^{new}=b_{ij}-\mu ·\underset{k\ne i}{\max }{b}_{kj} , for j\in J_{available}$$

(19)

where $\mu$ is the opportunity cost coefficient, designed to prevent excessive competition.

3.3. Convergence and Allocation Matrix

The algorithm convergence condition is

$${\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^M\mid x_{ij}^{\left(t\right)}-x_{ij}^{\left(t-1\right)}\mid }}=0$$

(20)

or when the maximum number of iterations $T_{\max }$ is reached. After each iteration, the global potential function is updated:

$$\mathrm{\Phi }(t)={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^M{x}_{ij}^{\left(t\right)}\cdot b_{ij}}}$$

(21)

Convergence is achieved when

$$\mid \mathrm{\Phi }(t)-\mathrm{\Phi }(t-1)\mid <\epsilon$$

(22)

The algorithm iterates until the allocation scheme stabilizes, outputting the optimal interception allocation matrix $X$. This process rapidly converges to a globally near-optimal solution that maximizes the defense potential function $\mathrm{\Phi }\left(s,\alpha \right)$. The potential function is defined as

$$\mathrm{\Phi }\left(s,\alpha \right)={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^M{x}_{ij}\cdot b_{ij}}+C_{cooperative}}$$

(23)

where the $C_{cooperative}$ captures synergistic effects among UAVs through the GAT attention mechanism (e.g., avoiding coverage overlap).

Through this auction mechanism, the Blue UAVs can efficiently and reliably allocate interception tasks, maximizing overall defense effectiveness while avoiding threat vulnerabilities and resource waste. The complete target allocation algorithm pseudocode is presented in Algorithm 1:

Algorithm 1: Target allocation

Initialize: $A,X,B\leftarrow 0; t\leftarrow 0, \mathrm{converged}\leftarrow \mathrm{false}; {\Phi }_{crr}\leftarrow {\Phi }_{prev}\leftarrow 0$   while $t<T_{max}$ and not converged do           // Phase 1: GAT benefit computation for each unassigned UAV $i$ in parallel do       for each free target j do             $B\left[i\right]\left[j\right]\leftarrow \left({\omega }_1·p_i+{\omega }_2·v_j-{\omega }_3·{\tau }_i\right)·\left(1+{\alpha }_{ij}\right)$             //${\alpha }_{ij}$ denotes the GAT attention weight defined in Equation (33) of Section 4.2       end for   end for         // Phase 2: Distributed bidding for each unassigned UAV $i$ in parallel do       $j^{\mathrm{*}}$ ← best free target for $i$       adjust bid if contested       broadcast bid $(i,j^{\mathrm{*}})$ end for // Phase 3: Conflict detection and resolution for each target $j$ do       $C$ ← set of UAVs that bid for $j$       if $\left|C\right|>1$ then             $\Delta \leftarrow$ benefit spread among $C$             if $\Delta <\delta$ then                   // Complex conflict: weighted multi-attribute resolution                   compute weighted score W for each $i\in C$ (benefit, urgency, resource)                   $i_{win}\leftarrow UAV$ with highest $W$                   if majority acknowledges $i_{win}$ then                       if $\exists i_{old}$ s.t. $X\left[i_{old}\right]\left[j\right]=1$ then $X\left[i_{old}\right]\left[j\right]$ ← 0 // clear previous assignment of target j                       if $\exists j_{old}$ s.t. $X\left[i_{win}\right]\left[j_{old}\right]=1$ then $X\left[i_{win}\right]\left[j_{old}\right]$ ← 0 //clear previous assignment of UAV $i_{win}$                               $X\left[i_{win}\right]\left[j\right]\leftarrow 1$                   else                           mark pending and request rebid                   end if               else                     // Simple conflict: highest benefit wins (tie broken by resource)                     $i_{win}\leftarrow argmax-benefit\left(tie:max{E}_i\right)$                     if $\exists i_{old}$ s.t. $X\left[i_{old}\right]\left[j\right]$ = 1 then                                              $X\left[i_{old}\right]\left[j\right]\leftarrow 0$                     if $\exists j_{old}$ s.t. $X\left[i_{win}\right]\left[j_{old}\right]=1$ then                                              $X\left[i_{win}\right]\left[j_{old}\right]\leftarrow 0$                     $X\left[i_{win}\right]\left[j\right]\leftarrow 1$                 end if         else if $\left|C\right|=1$ then                 $i_{win}\leftarrow sole-bidder$                 if $\exists i_{old}$ such that $X\left[i_{old}\right]\left[j\right]=1$ then                                              $X\left[i_{old}\right]\left[j\right]\leftarrow 0$                 if $\exists j_{old}$ such that $X\left[i_{win}\right]\left[j_{old}\right]=1$ then                                              $X\left[i_{win}\right]\left[j_{old}\right]\leftarrow 0$                         $X\left[i_{win}\right]\left[j\right]\leftarrow 1$         end if end for           // Phase 4: Convergence check                                              ${\Phi }_{prev}\leftarrow {\Phi }_{crr}$                                              ${\Phi }_{crr}\leftarrow {\Sigma }_{i,j:x\left[i\right]\left[j\right]=1}B\left[i\right]\left[j\right]+GA{T}_{coop}$ if $\left|{\Phi }_{crr}-{\Phi }_{prev}\right|<\epsilon$ or high-priority assigned or all $UAVs$ assigned then         $converged\leftarrow true$ end if                                              $t++$   end while   return X

This distributed auction-based target allocation mechanism ensures efficient and conflict-free task assignment while maintaining scalability for large-scale swarm operations. The integration of GAT-derived attention weights enables context-aware benefit calculation, while the two-stage conflict resolution process balances computational efficiency with allocation optimality.

4. Hierarchical Multi-Scale Mean Field Deep Deterministic Policy Gradient Algorithm

To address the challenges inherent in large-scale UAV swarm cooperative pursuit-interception tasks, this paper proposes a novel Hierarchical Multi-Scale Mean Field Deep Deterministic Policy Gradient (HM-MF-DDPG) framework. The training of the HM-MF-DDPG algorithm aims to jointly optimize the high-level graph attention network, the auction strategy, and the low-level actor-critic networks through experience replay and gradient descent, maximizing the global cumulative expected return. The core principle lies in utilizing the weighted mean field as a bridge connecting high-level and low-level policies, while simultaneously updating the parameters of both policy layers via gradients generated by the low-level critic networks.

4.1. Deep Deterministic Policy Gradient Algorithm

The Deep Deterministic Policy Gradient (DDPG) algorithm is a DRL method specifically designed for continuous action spaces [10]. It is built upon the actor-critic framework, integrating the core ideas of deterministic policy gradients and Deep Q-Networks. The DDPG algorithm architecture comprises four neural networks. The actor network represents the policy function $\mu (s|{\theta }^{\mu })$ which takes a state $s$ as input directly outputs a deterministic continuous action $s$. The critic network represents the value function $Q(s,a|{\theta }^Q)$, receiving both state $s$ and action $a$ to evaluate the quality of the state-action pair and output the corresponding Q-value. To enhance training stability, DDPG introduces corresponding target networks for both the actor and critic: the target actor network ${\mu }^{\prime }(s|{\theta }^{{\mu }^{\prime }})$ and the target critic network $Q^{\prime }(s,a|{\theta }^{Q^{\prime }})$. The parameters ${\theta }^{\prime }$ of these target networks are updated through a soft update mechanism, slowly tracking the parameters $\theta$ of the online networks, thereby smoothing the learning process.

The actor network is updated via the deterministic policy gradient theorem, with the objective of maximizing the Q-value provided by the critic network. The performance objective is defined as

$$J\left({\mathrm{\theta }}^{\mathrm{\mu }}\right)={\mathbb{E}}_s\left[Q\left(s,\mathrm{\mu }\left(s|{\mathrm{\theta }}^{\mathrm{\mu }}\right)\right)|{\mathrm{\theta }}^Q\right]$$

(24)

The policy gradient is computed as:

$${\nabla }_{{\theta }^{\mu }}J\left({\theta }^{\mu }\right)\approx {\mathbb{E}}_{s_t}\left[{\nabla }_{a}Q\left(s,a|{\theta }^Q\right){|}_{s=s_t,a=\mu \left(s_t\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right){|}_{s=s_t}\right]$$

(25)

The actor parameters are then updated via gradient ascent:

$${\mathrm{\theta }}^{\mathrm{\mu }}\leftarrow {\mathrm{\theta }}^{\mathrm{\mu }}{\mathrm{\alpha }}_{\mathrm{\mu }}{\nabla }_{{\mathrm{\theta }}^{\mathrm{\mu }}}J+\left({\mathrm{\theta }}^{\mathrm{\mu }}\right)$$

(26)

This adjustment directs the actor network to produce actions that yield higher Q-values.

The critic network is updated by minimizing the temporal difference (TD) error. The target Q-value is computed as:

$$y_t=r_t+\gamma Q^{\prime }\left(s_{t+1},{\mu }^{\prime }\left(s_{t+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$$

(27)

where $r_t$ is the immediate reward and $\gamma$ is the discount factor.

The loss function for the critic is

$$L({\theta }^Q)={\mathbb{E}}_{s_t,a_t,r_t,s_{t+1}}[{(y_t-Q(s_t,a_t|{\theta }^Q))}^2]$$

(28)

Critic parameters are updated via gradient descent:

$${\mathrm{\theta }}^Q\leftarrow {\mathrm{\theta }}^Q{\mathrm{\alpha }}_Q{\nabla }_{{\mathrm{\theta }}^Q}L\left({\mathrm{\theta }}^Q\right)$$

(29)

The critic learns to accurately evaluate the value of actions taken by the actor network.

Target network updates employ soft updates to ensure training stability:

$$\left\{\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}\\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}\end{array}\right.$$

(30)

where $\tau \ll 1$ is a small constant (e.g., 0.001).

4.2. Graph Attention Network

When processing environment information sets, UAVs should not treat all information equally. Instead, they should dynamically and discriminatively allocate limited resources based on current task requirements, focusing on more critical environment information. The UAV swarm is modeled as a graph $G=\left(V,E\right)$, where each UAV represents a node, and the cooperative relationships between UAVs represent edges.

For a Blue UAV $i$ in the graph, it possesses its own feature vector $h_i$ (e.g., position, velocity, munition status). The query $q_i$ is computed by transforming the feature $h_i$ of node $i$ through a learnable weight matrix $W_q$:

$$q_i=W_q\cdot h_i$$

(31)

This represents “what node $i$ wants to know.” For neighboring nodes $j$ of node $i$ (including itself), their features $h_i$ are transformed by $W_k$ and $W_v$, respectively, yielding keys $k_j=W_k\cdot h_j$ and values $v_j=W_v\cdot h_j$. The key $k_j$ is used for matching with the query, while the value $v_j$ represents the information to be ultimately aggregated.

Subsequently, the weighted mean field for node $i$ is computed. First, the correlation between node $i$ and each of its neighbors $j$ is calculated. In GAT, this is typically performed using a single-layer feedforward network $a$ followed by a $LeakyReLU$ activation function:

$$e_{ij}=LeakyReLU(a^T\cdot [q_i\left|\right|k_j])$$

(32)

where $\left|\right|$ denotes vector concatenation. The attention weights are then computed by applying $Softmax$ normalization to the scores of all neighbors $j$ of node $i$:

$${\alpha }_{ij}=exp\left(e_{ij}\right)/{\displaystyle \sum _{k\in N_i}exp(e_{ik})}$$

(33)

Here, ${\alpha }_{ij}$ represents the degree of “attention” that UAV $i$ should allocate to UAV $j$ when executing high-level target allocation. Finally, the weights are applied to the value vectors and summed to obtain the new feature $h_i^{\prime }$ for node $i$, which constitutes the weighted mean field for node $i$:

$$h_i^{\prime }={\displaystyle \sum _{j\in N_i}{\alpha }_{ij}\cdot v_j}$$

(34)

This weighted mean field $h_i^{\prime }$ captures the comprehensive influence of other agents in the swarm on agent $i$. It is not a single average value but rather personalized contextual information customized for each agent. The high-level policy leverages this $h_i^{\prime }$, rich in cooperative information, along with global state information (Red UAVs, NFZs, etc.), to construct the potential game model and solve for the optimal allocation matrix via the distributed auction algorithm. The attention network computation is illustrated in Figure 3.

Figure 3. Schematic diagram of the attention network structure.

4.3. Weighted Mean Field Approximation

In the multi-UAV cooperative pursuit task, we set up the mixed-reward function to guide the UAVs to complete their respective tasks according to the task objectives of both sides, as shown below.

Within this framework, the weighted mean field serves as the core enabler of efficient hierarchical coordination. Acting as a cooperative signal, the weighted mean field transmits the macro-level strategy of the high-level policy to each individual agent in the low-level layer, thereby achieving distributed decision-making while preserving global consistency. Unlike traditional mean field theory, which simplifies agent interactions to interactions with the average behavior of the swarm, the weighted mean field introduced in this paper dynamically evaluates the relative importance of neighboring agents through an attention mechanism, rendering coordination more targeted and intelligent.

Consider a system comprising $N$ agents. For any agent $i$, its weighted mean field ${\overline{a}}_{-i}$ is defined as the weighted sum of the actions of all other agents:

$${\overline{a}}_{-i}={\displaystyle \sum _{j\in N(i)}{a}_{ij}\cdot a_j}$$

(35)

where $a_j$ denotes the action of neighboring agent $j$, and ${\alpha }_{ij}$ is the attention weight of agent $i$ with respect to agent $j$, satisfying ${\displaystyle \sum _{j\ne i}{\alpha }_{ij}=1}$ and ${\alpha }_{ij}\ge 0$. These weight coefficients ${\alpha }_{ij}$ are crucial, as they quantify the relative magnitude of agent $j$’s influence from the decision-making perspective of agent $i$.

A core task of the high-level policy is to leverage global information to compute agent-specific attention weights ${\alpha }_{ij}$ for each agent $i$, thereby generating a guiding weighted mean field ${\overline{a}}_{-i}^{\mathrm{high}}$. Specifically, the high-level policy models the entire swarm as a graph $G=\left(V,E\right)$ and computes the weights using a graph attention network. For agents $i$, and $j$, the unnormalized attention score $e_{ij}$ is calculated. Subsequently, the final attention weights are obtained by normalizing the scores via the Softmax function.

The generated high-level weighted mean field ${\overline{a}}_{-i}^{\mathrm{high}}$ is injected as key input into the decision-making process of the low-level policy. In the actor network, agent $i$’s actor ${\pi }_{\mathrm{low}}\left(o_i,{\overline{a}}_{-i}^{\mathrm{high}}\right)$ takes as input both the local observation $o_i$ (e.g., self-state, assigned target, nearby threats) and the high-level weighted mean field ${\overline{a}}_{-i}^{\mathrm{high}}$. This enables the agent to not only base its decisions on local information but also possess a broader strategic perspective, aligning with the swarm’s overall strategic rhythm when determining its own continuous action $a_i=\left(v_x,v_y\right)$. In the critic network, the critic $Q\left(s_i,a_i,{\overline{a}}_{-i}\right)$ guides the actor’s update by evaluating the value of executing action $a_i$ in state $s_i$ given that the neighbor’s behavior follows a certain mean field ${\overline{a}}_{-i}$. The ${\overline{a}}_{-i}$ used during training is typically sampled from the experience replay buffer or derived from the target network. The evaluation of action-value $Q\left(s_i,a_i,{\overline{a}}_{-i}\right)$ by the low-level critic is influenced by the ${\overline{a}}_{-i}^{\mathrm{high}}$ produced by the high-level policy. Through the chain rule, the gradient of the value function with respect to the high-level policy parameters ${\theta }_{\mathrm{high}}$ can be back-propagated:

$${\nabla }_{{\theta }_{\mathrm{high}}}J\approx \mathbb{E}\left[{\nabla }_{{\overline{a}}_{-i}}Q(s_i,a_i,{\overline{a}}_{-i})\cdot \nabla {\theta }_{high}{\overline{a}}_{-i}^{\mathrm{high}}\right]$$

(36)

This gradient indicates how the high-level policy should adjust its output weighted mean field to maximize the long-term returns obtained by low-level agents. In other words, the high-level policy learns to generate a “coordination field” that optimally guides the swarm, while the low-level policy learns to act optimally within the constraints of this field. The two layers achieve end-to-end collaborative optimization through the medium of the weighted mean field.

4.4. GRU Learning Framework

The GRU is a type of recurrent neural network designed to address the vanishing or exploding gradient problems encountered when training traditional Recurrent Neural Networks (RNNs) on long sequences. By introducing gating mechanisms that selectively retain or forget information, GRUs effectively capture long-term dependencies in time series. The core of a GRU consists of a reset gate and an update gate, both of which are vectors with elements ranging between 0 and 1, controlling information flow. The update gate determines how much past information needs to be passed to the future, helping the model capture long-term dependencies. The reset gate determines how much past information needs to be “forgotten,” assisting the model in discarding historical information irrelevant to future predictions. The GRU structure is illustrated in Figure 4.

Figure 4. Schematic diagram of GRU structure.

Assume that at time step $t$, the input to the GRU is the current input vector $x_t$ and the hidden state $h_{t-1}$ from the previous time step. The update gate and reset gate are computed as follows:

$$z_t=\sigma \left(W_z\cdot x_t+U_z\cdot h_{t-1}+b_z\right)$$

(37)

$$r_t=\sigma \left(W_r\cdot x_t+U_r\cdot h_{t-1}+b_r\right)$$

(38)

where $\sigma$ is the sigmoid function; $W_z,U_z$ are weight matrices; $b_z$ is the bias vector; $z_t$ closer to 1 indicates “remembering” more past information. $W_r,U_r$ are weight matrices; $b_r$ is the bias vector; $r_t$ closer to 0 indicates “resetting” (forgetting) more past information.

Next, the candidate hidden state is computed. This state incorporates the current input and historical information filtered by the reset gate, representing the potential new state given the current input:

$${\tilde{h}}_t=\tanh \left(W_h\cdot x_t+U_h\cdot \left(r_t\odot h_{t-1}\right)+b_h\right)$$

(39)

Here, $\odot$ denotes element-wise multiplication (Hadamard product). If $r_t$ is close to 0, the term $U_h\cdot \left(r_t\odot h_{t-1}\right)$ becomes very small, meaning the previous hidden state $h_{t-1}$ is largely ignored, and the candidate state is computed based only on the current input $x_t$. This is suitable for scenarios requiring the discarding of irrelevant historical information.

Finally, the GRU’s final output is computed by mixing the old hidden state and the new candidate state through the update gate:

$$h_t=\left(1-z_t\right)\odot h_{t-1}+z_t\odot \widetilde{h_t}$$

(40)

Here, $h_t$ is a weighted average of $h_{t-1}$ and $\widetilde{h_t}$; $z_t$ determines the weight of the new information $\widetilde{h_t}$; $\left(1-z_t\right)$ determines the weight of the old information $h_{t-1}$. If $z_t$ is close to 1, then $h_t\approx \widetilde{h_t}$, meaning the old state is almost completely replaced by the new state. If $z_t$ is close to 0, then $h_t\approx h_{t-1}$, meaning the current input is ignored and the old state is fully retained. This final hidden state $h_t$ is passed to the next time step and also typically serves as the output for the current time step.

The GRU module in the low-level policy functions as a temporal information processor. It endows individual UAVs with the capability to make intelligent decisions based on historical sequence data in partially observable environments, enabling them to better understand the evolution of the dynamic environment and thus execute more effective and coordinated pursuit-interception actions. It is key to achieving intelligent flight control, as opposed to simple reactive control.

4.5. HM-MF-DDPG Algorithm

The objective of the low-level critic network $Q_{{\phi }_i}$ is to accurately estimate the action-value function. Its loss function employs the standard temporal difference (TD) error, augmented with a regularization term to prevent overfitting:

$$L\left({\phi }_i\right)=E_{\left(o_i^t,a_i^t,r_i^t,o_i^{t+1},{\overline{a}}_i^t\right)\sim D}\left[{\left(y_i^t-Q_{{\phi }_i}\left(o_i^t,a_i^t,{\overline{a}}_i^t\right)\right)}^2\right]+{\lambda }_c\left|\right|{\phi }_i|{|}^2$$

(41)

where the target value $y_i^t$ is computed using target networks to stabilize training:

$$y_i^t=r_i^t+\mathsf{\gamma }{Q}_{{\phi }_i^{\prime }}\left(o_i^{t+1},{\mathrm{\mu }}_{{\theta }_i^{\prime }}\left(o_i^{t+1},{\overline{a}}_i^{t+1}\right),{\overline{a}}_i^{t+1}\right)$$

(42)

Here, ${\phi }_i^{\prime }$ and ${\theta }_i^{\prime }$ are the parameters of the target critic and target actor networks, $\gamma$ is the discount factor, and ${\mathrm{\lambda }}_c$ is the L2 regularization coefficient. Critic parameters ${\phi }_i$ are updated by minimizing this loss function:

$${\phi }_i\leftarrow {\phi }_i-{\mathrm{\alpha }}_c{\nabla }_{{\phi }_i}L\left({\phi }_i\right)$$

(43)

where ${\mathrm{\alpha }}_c$ is the critic learning rate.

The parameters of the low-level actor policy ${\mathrm{\mu }}_{{\mathrm{\theta }}_i}$ are updated via the deterministic policy gradient theorem. Its objective is to output actions that maximize the Q-value evaluated by the critic. The gradient is computed as follows:

$${\nabla }_{{\theta }_i}J\left({\theta }_i\right)\approx E_{\left(o_i^t,{\overline{a}}_i^t\right)\sim D}\left[{\nabla }_{a_i}{Q}_{{\varphi }_i}\left(o_i,a_i,{\overline{a}}_i\right){|}_{a_i={\mu }_{{\theta }_i}\left(o_i^t,{\overline{a}}_i^t\right)}\cdot {\nabla }_{{\theta }_i}{\mu }_{{\theta }_i}\left(o_i^t,{\overline{a}}_i^t\right)\right]{\overline{a}}_i^t$$

(44)

Subsequently, the actor parameters ${\theta }_i$ are updated along the gradient ascent direction:

$${\theta }_i\leftarrow {\theta }_i+{\alpha }_a{\nabla }_{{\theta }_i}J\left({\theta }_i\right)$$

(45)

where ${\mathrm{\alpha }}_a$ is the actor learning rate.

The update of the high-level GAT network policy parameters $\mathrm{\xi }$ constitutes the core of the entire algorithm. The objective is to adjust the graph attention network so that the weighted mean field ${\overline{a}}_i^t$ it generates guides low-level agents to execute actions that maximize the global Q-value. The gradient originates from the low-level critic network and back-propagates through the weighted mean field ${\overline{a}}_i^t$:

$${\nabla }_{\mathrm{\xi }}J\left(\mathrm{\xi }\right)\approx E_{S_t\sim D}\left[{{\displaystyle \sum }}_{i=1}^N{\nabla }_{{\overline{a}}_i}{Q}_{{\varphi }_i}\left(o_i,a_i,{\overline{a}}_i\right){|}_{a_i={\mathrm{\mu }}_{{\mathrm{\theta }}_i}\left(o_i^t,{\overline{a}}_i^t\right)}\cdot {\nabla }_{\mathrm{\xi }}{\mathsf{\pi }}_{\mathrm{high}}\left(S_t;\mathrm{\xi }\right)\right]$$

(46)

The update of high-level parameters $\xi$ also follows gradient ascent:

$$\xi \leftarrow \xi +{\alpha }_{\mathrm{high}}{\nabla }_{\xi }J\left(\xi \right)$$

(47)

where ${\mathrm{\alpha }}_{\mathrm{high}}$ is the learning rate for the high-level policy.

To further enhance training stability, all target networks are updated using a soft update strategy, slowly tracking the parameters of the online networks:

$${\phi }_i^{\prime }\leftarrow \tau \cdot {\phi }_i+\left(1-\tau \right)\cdot {\phi }_i^{\prime }$$

(48)

$${\theta }_i^{\prime }\leftarrow \tau \cdot {\theta }_i+\left(1-\tau \right)\cdot {\theta }_i^{\prime }$$

(49)

$${\xi }^{\prime }\leftarrow \tau \cdot \xi +\left(1-\tau \right)\cdot {\xi }^{\prime }$$

(50)

where $\mathrm{\tau }\ll 1$. The HM-MF-DDPG algorithm structure is illustrated in Figure 5 and the pseudo-code of the HM-MF-DDPG algorithm is detailed in Algorithm 2.

Algorithm 2: HM-MF-DDPG Algorithm

Initialize High-level GAT network ${\pi }_{high}\left(\xi \right)$ and target network ${\pi }_{high}\left({\xi }^{\prime }\right)$ for episode = 1 to $M$ do         Initialize state $S_1$         for $t=1$ to $T$ do                 // High-level: allocation and mean field                 ${\mathrm{\Phi }}_t,{\overline{a}}_i^t={\pi }_{high}(S_t;\xi )$                 for each agent $i$ do                         Assign target $j^{*}$ from ${\mathrm{\Phi }}_t$                         Build local observation $o_i^t$ (self, target, environment, neighbors)                         $a_i^t={\mu }_{{\theta }_i}(o_i^t,{\overline{a}}_i^t)+{\mathcal{N}}_t$ // exploration noise                 end for                 Execute joint action $A_t=(a_1^t,\dots ,a_N^t)$                 Get reward $R_t=(r_1^t,\dots ,r_N^t)$ and next state $S_{t+1}$                 Store $\left(S_t,A_t,R_t,S_{t+1},{\overline{a}}_i^t\right)$ in $D$                 // Model updates (every step, G times)                 for update = 1 to $G$ do                         Sample mini-batch $B$ from $D$                         for each agent $i$ do                                 // Update low-level Critic                                 $y_i^t=r_i^t+\gamma Q_{{\phi }_i^{\prime }}(o_i^{t+1},{\mu }_{{\theta }_i^{\prime }}(o_i^{t+1},{\overline{a}}_i^{t+1}),{\overline{a}}_i^{t+1})$                                 $L\left({\phi }_i\right)=(y_i^t-Q_{{\phi }_i}(o_i^t,a_i^t,{\overline{a}}_i^t){)}^2$                                 ${\phi }_i\leftarrow {\phi }_i-{\alpha }_c{\nabla }_{{\phi }_i}L$                                 // Update low-level Actor                                 ${\nabla }_{{\theta }_i}J\approx {\nabla }_{a_i}{Q}_{{\phi }_i}\left(o_i^t,a_i,{\overline{a}}_i^t\right){\mid }_{a_i={\mu }_{{\theta }_i}}\cdot {\nabla }_{{\theta }_i}{\mu }_{{\theta }_i}\left(o_i^t,{\overline{a}}_i^t\right)$                                 ${\theta }_i\leftarrow {\theta }_i+{\alpha }_a{\nabla }_{{\theta }_i}J$                         end for                         // Update high-level policy                         ${\nabla }_{\xi }J\approx {\sum }_i{\nabla }_{{\overline{a}}_i}{Q}_{{\phi }_i}\left(o_i^t,a_i^t,{\overline{a}}_i\right){\mid }_{a_i={\mu }_{{\theta }_i}}\cdot {\nabla }_{\xi }{\pi }_{high}\left(S_t;\xi \right)$                         $\xi \leftarrow \xi +{\alpha }_{high}{\nabla }_{\xi }J$                         // Soft update target networks                         for each agent $i$ do                                 ${\phi }_i^{\prime }\leftarrow \tau {\phi }_i+\left(1-\tau \right){\phi }_i^{\prime }$                                 ${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$                         end for                         ${\xi }^{\prime }\leftarrow \tau \xi +\left(1-\tau \right){\xi }^{\prime }$                 end for         end for end for

Figure 5. HM−MF−DDPG algorithm structure diagram.

The high-level policy learns to produce a weighted mean field and target allocation scheme that maximizes global cooperative effectiveness, while the low-level policy learns to execute optimal distributed control actions under the guidance of this scheme, ultimately accomplishing complex cooperative pursuit-interception tasks. This training procedure ensures the synergistic optimization of the high-level and low-level policies.

5. UAV Swarm Interception Strategy Based on HM-MF-DDPG

5.1. State Space Design

The state space composition is fundamentally identical for both Blue and Red UAVs. For a Blue UAV, the state space primarily consists of three components: real-time kinematic state, detected target information, and intercept target information, denoted as

$$S^i=\left[I_{GRU}^i,I_{detect}^i,I_{intercept}^i\right]$$

(51)

The real-time kinematic state records the UAV’s position ($x_i\left(t\right),y_i\left(t\right)$), velocity $v_i\left(t\right)$, and heading angle ${\phi }_i\left(t\right)$:

$$I_{GRU}^i=\left[x_i\left(t\right),y_i\left(t\right),v_i\left(t\right),{\phi }_i\left(t\right)\right]$$

(52)

Considering the practical limitations in intercept scenarios where each UAV can detect and track only a limited number of targets, the UAV records information for up to three targets that can be reliably tracked by its radar. The detected target information is structured as

$$I_{detect}^i=\left[d_1^i,{\beta }_1^i,d_2^i,{\beta }_2^i,d_3^i,{\beta }_3^i\right]$$

(53)

where $d_k^i$ and ${\beta }_k^i$ ($k=1,2,3$) represent the distance and azimuth angle of the k-th detected target relative to UAV $i$. If fewer than three targets are detected, the corresponding distance and azimuth values are set to zero.

When interception task begins, the distance and azimuth of the assigned target relative to the Blue UAV are incorporated into the state space as intercept target information:

$$I_{intercept}^i=\left[d_i^j,{\beta }_i^j\right]$$

(54)

where $d_i^j$ and ${\beta }_i^j$ denote the relative distance and angle between Blue UAV $i$ and its assigned target $j$.

The state space of a Red UAV follows a similar structure, consisting of its real-time kinematic state $I_{EUAV}^j$, detected target information $I_{e\_detect}^i$, and attack task information $I_{HVA}^j$:

$$S_E^j=\left[I_{EUAV}^j\hspace{0.33em},\hspace{0.33em}{I}_{e\_etect}^i\hspace{0.33em},\hspace{0.33em}{I}_{HVA}^j\right]$$

(55)

Here, $I_{EUAV}^j$ and $I_{e\_detect}^i$ are identical in structure to the Blue UAV. $I_{HVA}^j$ is the relative distance $d_j^{HVA}$ and azimuth ${\beta }_j^{HVA}$ of HVA with respect to Red UAV $j$:

$$I_{HVA}^j=\left[d_j^{HVA}\hspace{0.33em},\hspace{0.33em}{\beta }_j^{HVA}\right]$$

(56)

5.2. Action Space Design

UAV motion commands primarily consist of linear acceleration and angular acceleration. The action space is defined by linear acceleration $u$ and angular acceleration $w$:

$$A=\left[u,\hspace{0.33em}w\right]$$

(57)

5.3. Reward Function Design

In this subsection, we design composite reward functions for both Blue and Red UAVs, consisting of episodic rewards and step rewards. Episodic rewards are granted upon task termination to evaluate overall mission success (e.g., interception success reward, boundary penalty, collision penalty). Step rewards are provided at each time step to guide immediate behaviors (e.g., reward for moving toward the assigned target, time penalty). This “sparse + dense” reward structure addresses the credit assignment problem in large-scale adversarial environments: the episodic reward provides a clear mission objective to avoid local optima, while the step reward alleviates reward sparsity and accelerates early training convergence.

The weights of each reward component are determined via grid search and ablation experiments. Taking the step reward of Blue UAVs as an example, we fix the Blue-to-Red ratio at 2:1 and test different ratios between the progress reward weight ${\alpha }_1$ and the time penalty weight ${\alpha }_2$ over 50 training episodes. The results show that the optimal balance between fast target approach and collision avoidance is achieved when ${\alpha }_1/{\alpha }_2=5:1$. All reward components are normalized to $\left[-\mathrm{1,1}\right]$ to ensure training stability.

The reward function for Red UAVs is designed to guide them to penetrate interception and attack the high-value asset (HVA). The distance reward in its step reward encourages Red UAVs to approach the Blue HVA, while the time penalty urges them to complete the task as soon as possible. This adversarial reward design allows the policies of both sides to compete with each other during training, eventually converging to a neighborhood of the Nash equilibrium.

5.3.1. Blue UAV Reward

The total reward $R_i^B$ for a Blue UAV $i$ consists of an episodic reward $R_{i,episode}^B$ and a step reward $R_{i,step}^B$:

$$R_i^B=R_{i,episode}^B+R_{i,step}^B$$

(58)

The episodic reward $R_{i,episode}^B$ is granted upon task termination (success or failure) and includes an interception reward, a boundary penalty, and a collision penalty:

$$R_{i,episode}^B=r_{intercept}+r_{boundary}+r_{collision}$$

(59)

where $r_{intercept}$ is the reward when a Blue UAV successfully intercepts a Red UAV, $r_{boundary}$ is the penalty when a Blue UAV flies outside the predefined task area, and $r_{collision}$ is the penalty when a Blue UAV collides with any other entity.

The step reward $R_{i,step}^B$ guides the UAV’s immediate behavior at each time step and comprises a progress reward and a time penalty:

$$R_{i,step}^B={\alpha }_1\cdot r_{progress}+{\alpha }_2\cdot r_{time}$$

(60)

where ${\alpha }_1,{\alpha }_2$ are weighting coefficients, $r_{progress}$ is a positive reward given when the Blue UAV moves toward its assigned target, and $r_{time}$ is a fixed negative reward used to encourage the Blue UAV to complete the interception as soon as possible.

5.3.2. Red UAV Reward

The total reward $R_j^R$ for a Red UAV $j$ also consists of episodic and step rewards, but its design objective is to guide successful destruction of the Blue HVA:

$$R_j^R=R_{j,episode}^R+R_{j,step}^R$$

(61)

The episodic reward $R_{j,episode}^R$ includes an escape reward, a boundary penalty, and a collision penalty, granted upon task termination:

$$R_{j,episode}^R=r_{escape}+r_{boundary}+r_{collision}$$

(62)

The step reward for Red UAVs at each time step is

$$R_{j,step}^R={\beta }_1\cdot r_{distance\_to\_HVA}+{\beta }_2\cdot r_{time}$$

(63)

where ${\beta }_1,{\beta }_2$ are weighting coefficients, $r_{distance\_to\_HVA}$ encourages Red UAVs to approach the high-value asset, and $r_{time}$ is a time penalty.

5.4. Neural Network Architecture

The HM-MF-DDPG algorithm adopts CTDE framework. The neural networks consist of low-level actor and critic networks for each agent, and a high-level graph attention network (GAT) shared across all agents.

5.4.1. High-Level Graph Attention Network

The high-level GAT produces the weighted mean field ${\overline{a}}_i^t$ for the low-level critics. Its inputs consist of features from all Blue UAVs and Red targets, as well as the heterogeneous graph structure, which includes communication and perception edges. The architecture of the high-level network is shown in Figure 6.

Figure 6. High-level GAT architecture.

The initial feature $h_i^0$ of a Blue UAV node comprises its own state and the state of its assigned target. The feature $g_j$ of a Red UAV node includes its position, velocity, and distance to the HVA. The network consists of feature projection layers and graph attention layers. In each layer, every node aggregates information from its neighbors via the attention mechanism. For a Blue UAV node $i$ and its neighbor $j$, the attention coefficient ${\mathrm{\alpha }}_{ij}$ is computed as described in Section 4.2. After multiple propagation layers, each Blue UAV node outputs a final embedding $h_i^{final}$ that encapsulates rich cooperative information among UAVs.

5.4.2. Low-Level Actor Network

The low-level actor network ${\mu }_{\theta }$ acts as a distributed controller, generating continuous action commands based on the agent’s local observation. The input layer has dimension $d_{in}^a=12$, comprising the agent’s own state, the state of its assigned target, environmental information, and the high-level GAT-generated weighted mean field ${\overline{a}}_i^t$. The hidden layers consist of two fully connected (FC) layers and one GRU layer. The first FC layer (FC1) performs initial encoding and nonlinear transformation of the 12-dimensional input. The GRU layer takes the output of FC1 as the current time step input, combines it with the previous hidden state $h^{t-1}$, and outputs a new hidden state $h^t$. This GRU encodes the target’s motion trends, historical interaction information, and predicts its future trajectory. The second FC layer (FC2) further transforms the GRU output. The output layer has dimension $d_{out}^a=2$, mapping the FC2 output to a 2-dimensional vector, which is then passed through a Tanh activation function to compress it into the range $\left[-1,1\right]$. This design enables the agent to leverage historical information for long-term prediction while adjusting the aggressiveness of its actions based on real-time threats. The actor network structure is depicted in Figure 7.

Figure 7. Low−level actor network architecture.

5.4.3. Low-Level Critic Network

The low-level critic network $Q_{\mathrm{\phi }}$ evaluates the value of the action taken by the actor, providing gradient direction for policy updates. The input layer concatenates the agent’s local observation $o_i^t$, its own action $a_i^t$, and the high-level GAT output ${\overline{a}}_i^t$, resulting in a 16-dimensional input vector. The hidden layers employ three fully connected layers for value function approximation: FC1 performs high-capacity encoding of the concatenated features, FC2 further abstracts and refines features, and FC3 prepares for the final Q-value output. The output layer has dimension $d_{out}^c=1$, producing a scalar representing the expected cumulative return $Q\left(o_i^t,a_i^t,{\overline{a}}_i^t\right)$ given the state, the agent’s own action, and the neighbors’ mean field. The low-level critic network architecture is shown in Figure 8.

Figure 8. Low-level critic network architecture.

The critic network comprises three fully connected hidden layers with dimensions 256, 128, and 64 respectively, each followed by a ReLU activation. The input to the critic is a 16-dimensional vector formed by concatenating the local observation $o_i^t$ (12-dim), the agent’s own action $a_i^t$ (2-dim), and the weighted mean field ${\overline{a}}_i^t$ (2-dim, derived from the high-level GAT output). The output layer produces a scalar Q-value estimating the expected cumulative return. This architecture provides sufficient capacity to approximate the complex value function in the large-scale swarm interception task.

6. Experiments

6.1. Experimental Setup

We constructed a simulation training environment tailored for UAV swarm cooperative decision-making research and conducted multi-agent interception task training. The simulation parameters are set as shown in Table 1.

Table 1. Simulation environment parameters.

Parameter	Value
Task area (m × m)	$500\times 500$
Blue UAV speed (m/s)	$v_i(t)\in [1,2.5]$
Red UAV speed (m/s)	$v_j(t)\in [1.5,3]$
UAV linear acceleration (m/s2)	$u(t)\in [-1,1]$
UAV angular acceleration (rad/s2)	$w(t)\in [-\pi /6,\pi /6]$
Maximum radar detection range (m)	$d_{detect}=50$
Maximum radar detection angle (°)	${\theta }_{detect}=75$
Maximum inter-UAV communication range(m)	$d_{com}=50$
Engagement range (m)	$d_{attack}=20$
Engagement angle (°)	${\theta }_{attack}=120$
Effective engagement range (m)	$d_{kill}=15$
Number of NFZs	$n=5$
Learning rate	$\alpha =0.005$
Discount factor	$\gamma =0.98$
Batch size	$batch\_size=128$
Number of training episodes	$T=2000$
Maximum steps per episode	$t_{\max }=700$
Weighting coefficients for $R_{i,step}^B$	${\alpha }_1=0.7,{\alpha }_2=0.3$
Weighting coefficients for $R_{j,step}^R$	${\beta }_1=0.7,{\beta }_2=0.3$

6.2. Training Process

In the large-scale UAV swarm interception task, Blue UAVs were trained using the HM-MF-DDPG algorithm, while Red UAVs were trained using the MF-DDPG algorithm. To verify the convergence of the algorithm training, the loss values of the neural networks were recorded, as shown in Figure 9 and Figure 10.

Figure 9. Blue UAV actor and critic network losses.

Figure 10. Red UAV actor and critic network losses.

In the early stages of training, UAVs adopted random action strategies for exploration, resulting in relatively high losses for both actor and critic networks. As the number of training episodes increased, the actor and critic losses for both sides gradually decreased. Due to the adversarial nature of the task, the loss values exhibited considerable fluctuations during training but overall trended toward convergence.

To verify the effectiveness of the algorithm training, the average rewards per episode for both sides were recorded as performance metrics, as illustrated in Figure 11.

Figure 11. Average episode reward for Blue and Red UAV swarm.

In the initial training phase, both sides employed random action strategies, resulting in large negative rewards. As training progressed, both swarms continuously learned, and the reward values increased. After approximately 600 episodes, the average reward for Blue UAVs stabilized, while Red UAVs converged after around 1000 episodes.

The task success rate is defined as the ratio of the number of successfully intercepted Red UAVs to the total number of Red UAVs. In the experiment, models of the Blue UAVs at different training steps were selected to conduct confrontation tests against a fixed opponent, and the corresponding task success rate curve is shown in Figure 12.

Figure 12. Interception success rate for Blue UAV.

During the early training phase, the interception success rate was low due to random exploration. As training advanced, more Blue UAVs successfully intercepted evading targets, leading to a significant increase in success rate, which eventually converged. As shown in the figure, the interception success rate converges to approximately 93%.

6.3. Validation Process

In the large-scale UAV swarm intercept scenario, we set the number of Blue UAVs to $N=50$, Red UAVs to $M=25$, and NFZs to $n=5$. Blue UAVs made maneuvering decisions using the trained HM-MF-DDPG model, while Red UAVs employed an MF-DDPG model. As illustrated in Figure 13, the UAV swarm, using the trained HM-MF-DDPG model for autonomous intelligent task decision-making, successfully avoided terrain NFZs, maintained internal swarm stability, and effectively pursued and intercepted Red UAVs, thereby accomplishing the intercept task.

Figure 13. Snapshot of simulation with 50 Blue UAVs, 25 Red UAVs, 5 NFZs.

To evaluate the generalization capability of the algorithm, we varied the number of NFZs while keeping the numbers of Blue and Red UAVs fixed at 50 and 25, respectively. In all cases, the Blue UAV swarm successfully Intercepted the Red UAV swarm. The execution processes under different NFZs configurations are depicted in Figure 14 and Figure 15.

Figure 14. Snapshot of simulation with 50 Blue UAVs, 25 Red UAVs, without NFZ.

Figure 15. Snapshot of simulation with 50 Blue UAVs, 25 Red UAVs, 7 NFZs.

To further assess the algorithm’s effectiveness under varying tactical situations, we altered the swarm sizes and initial postures while fixing the number of NFZs at $n_{NFZ}=5$. The execution processes under different configurations are shown in Figure 16 and Figure 17.

Figure 16. Snapshot of simulation with 70 Blue UAVs, 35 Red UAVs, 5 NFZs.

Figure 17. Snapshot of simulation with 90 Blue UAVs, 45 Red UAVs, 5 NFZs.

As the numbers of both Blue and Red UAVs increase, the environment becomes more dynamic and complex, raising the difficulty of the intercept task. In scenarios with 70 Blue UAVs against 35 Red UAVs, and 90 against 45, the majority of Blue UAVs were still able to accurately intercept evading Red UAVs, achieving task success rates of approximately 88%. When the number of NFZs was small, the Blue UAV swarm could leverage terrain advantages to pursue Red UAVs. As the number of NFZs increased, the swarm had to avoid these zones while maintaining internal stability, which elevated the risk of collisions. Overall pursuit performance was influenced by factors such as the number and spatial distribution of NFZs, yet the task success rate remained above 90%.

We defined two metrics to evaluate the penetration effectiveness of the Blue UAV swarm: interception success rate (ratio of successfully intercepted Red UAVs to total Red UAVs) and task efficiency (ratio of episodes in which all Red UAVs were successfully intercepted to total validation episodes). To validate the effectiveness of the HM-MF-DDPG algorithm, we fixed the number of Red UAVs at 45 and the number of NFZs at 5, while varying the number of Blue UAVs. The resulting metrics are plotted in Figure 18.

Figure 18. Interception success rate & task efficiency under different numbers.

The number of Blue UAVs was gradually increased from 10 to 90, corresponding to Blue-to-Red ratios ranging from 2:9 up to 2:1. As shown in the figures, when the number of Blue UAVs was less than that of Red UAVs, the Red side held a numerical advantage, and due to attack characteristics, not all Red targets could be intercepted. When the Blue UAV count increased to 50 or more, the pursuit success rate rose substantially. At a 1:1 force ratio, the pursuit success rate reached 75%, with a task efficiency of approximately 60%. At a 2:1 force ratio, the pursuit success rate increased to 95%, and task efficiency reached 95%. Therefore, when facing an evading Red UAV swarm, appropriately increasing the number of Blue UAVs can significantly enhance pursuit effectiveness.

6.4. Algorithm Comparison

We further compare the proposed HM-MF-DDPG with three well-established MARL baselines: MADDPG, QMIX, and MAPPO. To ensure a fair comparison, all algorithms are evaluated in the same simulation environment and task setting (Blue UAVs to $N=50$, Red UAVs to $M=25$, and NFZs to $n_{NFZ}=5$). Since these baseline methods were originally designed for general multi-agent cooperation and do not inherently include target assignment mechanisms, we augment each with a common rule-based assignment module: each Blue UAV selects the closest unassigned Red UAV as its target based on Euclidean distance, and the assignment is recomputed at each decision step. The assigned target identity and its relative state are concatenated into the local observation of the respective agent.

For fair comparison, the three baselines are adapted to the continuous-action interception task as follows: MADDPG employs parameter-shared actors with a centralized critic that processes all agents’ states via permutation-invariant mean pooling. QMIX is implemented as a continuous variant with per-agent actors and critics, whose individual Q-values are combined through a monotonic mixing network conditioned on the global state. MAPPO adopts parameter-shared actor-critic networks with a transformer-based global state encoder and is trained via PPO with generalized advantage estimation. Figure 19 presents the learning curves of the four algorithms, and Table 2 reports the quantitative metrics at convergence.

Figure 19. Comparison of average episode rewards.

Table 2. Performance comparison with MARL baselines.

Algorithm	Interception Success Rate (%)	Task Efficiency (%)	Avg. Episodic Reward
HM-MF-DDPG	93.2 ± 1.5	91.6 ± 2.1	20.47 ± 2.54
MADDPG	71.4 ± 4.3	57.8 ± 5.1	13.45 ± 4.26
QMIX	68.9 ± 5.0	52.1 ± 6.2	11.13 ± 3.65
MAPPO	79.2 ± 3.1	70.5 ± 3.9	15.28 ± 3.51

HM-MF-DDPG outperforms all baselines by a substantial margin across all metrics. MAPPO achieves the best baseline results due to its on-policy nature and global encoding, yet its success rate remains 14 percentage points lower, suffering from myopic target selection and duplicated assignments. MADDPG and QMIX degrade more severely as the number of agents grows, with high variance and slow convergence caused by the curse of dimensionality and the limitations of their factorization assumptions.

These results confirm that the proposed HM-MF-DDPG offers substantial advantages over state-of-the-art MARL methods in large-scale UAV swarm interception. Its hierarchical design, graph-enhanced mean field, and auction-based allocation collectively address the scalability, heterogeneity, and coordination challenges that degrade the performance of conventional MARL algorithms in this domain.

6.5. Ablation Study

To rigorously quantify the contribution of each core component in the proposed HM-MF-DDPG framework, we conducted a comprehensive ablation study. Five model variants were constructed by systematically removing or replacing individual modules, while keeping all other settings identical to the full model. All variants were trained and evaluated in the canonical scenario with Blue UAVs to $N=50$, Red UAVs to $M=25$, and NFZs to $n_{NFZ}=5$. The evaluated variants are as follows:

HM-MF-DDPG: The complete proposed framework, including graph-enhanced weighted mean field, hierarchical decision architecture, auction-based distributed target assignment, and the situational-awareness-modulated GRU in the low-level actor.

w/o Graph Attention (w/o GAT): The high-level graph attention network is replaced by a conventional equal-weight mean field approximation. All neighboring agents contribute equally to the mean field, i.e., ${\overline{a}}_i={\displaystyle \frac{1}{\left|N_i\right|}}{\sum }_{j\in N_i}{a}_j$. The hierarchical structure and auction module are retained.

w/o Hierarchical Architecture (w/o Hier): The two-level decision hierarchy is removed. A single-layer policy network (an MLP-based actor-critic) is used, which receives the same local observation augmented with the equal-weight mean field and the identity of the target assigned by a fixed rule. Target assignment is performed by a simple closest-target heuristic (each Blue UAV pursues the nearest unassigned Red UAV), eliminating the high-level GAT and the auction mechanism. The low-level GRU is retained for trajectory prediction.

w/o Auction Mechanism (w/o Auction): The distributed auction-based target allocation is replaced by a greedy heuristic. At each decision step, every unassigned Blue UAV independently selects the target that maximizes the instantaneous interception benefit $B\left[i\right]\left[j\right]$ without any conflict resolution or iterative bidding. The high-level GAT and low-level GRU remain unchanged.

w/o GRU (w/o GRU): The gated recurrent unit in the low-level actor network is substituted by a standard fully connected layer with the same number of hidden units. The actor therefore loses its temporal modelling capability and processes only the current observation without explicit historical context. All other components are kept intact.

Table 3 summarises the interception success rate, task efficiency (fraction of episodes in which all Red UAVs are intercepted), and average episodic reward at convergence for each variant.

Table 3. Ablation study results.

Variant	Interception Success Rate (%)	Task Efficiency (%)	Avg. Episodic Reward
HM-MF-DDPG	93.2 ± 1.5	91.6 ± 2.1	20.47 ± 2.54
w/o GAT	85.7 ± 2.8	80.2 ± 3.4	15.23 ± 3.64
w/o Hier	76.4 ± 3.5	68.9 ± 4.2	14.92 ± 4.58
w/o Auction	88.1 ± 2.0	83.7 ± 2.6	17.28 ± 3.02
w/o GRU	90.5 ± 1.8	87.4 ± 2.3	18.67 ± 2.81

The results clearly attribute the performance gains to each component. Removing graph attention (w/o GAT) causes a 7.5% drop in success rate, confirming that attention-based neighbor weighting is essential for capturing heterogeneous interactions. Disabling the hierarchy (w/o Hier) yields the largest degradation (−16.8%), demonstrating that separating strategic coordination from low-level control is critical. Replacing the auction mechanism (w/o Auction) reduces success rate by 5.1%, reflecting the importance of conflict-free assignment. Excluding the GRU (w/o GRU) results in a smaller but consistent loss (−2.7%), indicating that temporal modeling provides additional refinement. Overall, all three innovations, graph-enhanced mean field, hierarchical architecture, and auction-based allocation, contribute meaningfully, with the hierarchical decomposition playing the most vital role.

7. Conclusions and Future Work

7.1. Conclusions

This paper proposed a HM-MF-DDPG framework augmented by GraphSAGE for large-scale UAV swarm cooperative interception under saturation incursions. The framework introduces three key innovations: (1) a graph-enhanced weighted mean field approximation that employs attention mechanisms to overcome the homogeneity limitation of conventional mean field methods; (2) a hierarchical decision architecture separating strategic coordination (via graph attention networks) from low-level flight control (via improved GRUs); and (3) a distributed target assignment mechanism formulated as a potential game and solved via parallel auction algorithms.

Extensive simulations in a custom-built environment demonstrated that HM-MF-DDPG achieves a 93% interception success rate with 50 Blue UAVs against 25 Red intruders, outperforming DDPG and MF-DDPG baselines in both convergence speed and task efficiency. The framework exhibits robust generalization across varying NFZ configurations and swarm scales, maintaining success rates above 90% under different tactical situations.

7.2. Future Work

Several directions merit further investigation. First, extending the framework to full three-dimensional space will incorporate altitude-dependent kinematics and sensor constraints, enabling application to more realistic operational scenarios. Second, the robustness of the proposed method under realistic communication conditions, including packet loss, latency, and bandwidth limitations, should be systematically evaluated, as communication degradation may significantly impact the distributed auction mechanism and the graph-based mean field computation. Third, the current study assumes ideal sensing; introducing sensor noise and detection uncertainty would allow for the assessment of the framework’s resilience under imperfect observations. Fourth, a computational cost analysis of the hierarchical architecture, particularly the trade-off between allocation optimality and real-time performance on resource-constrained onboard hardware, is necessary to evaluate deployability. Additionally, extending the framework to heterogeneous UAV teams with diverse kinematic capabilities and payload configurations, as well as incorporating resilience mechanisms against agent failures, represent important steps toward practical swarm defense applications in contested environments.