A Multi-UAV Cooperative Decision-Making Method in Dynamic Aerial Interaction Environments Based on GA-GAT-PPO

Highlights

What are the main findings?

• A hierarchical decision-making framework that decouples low-level maneuver control from high-level cooperative assignment is proposed, which effectively mitigates cross-level non-stationarity in hierarchical multi-agent reinforcement learning for multi-Unmanned Aerial Vehicle (UAV) systems.
• A Geometry-Aware Graph Attention Network (GA-GAT) with embedded kinematic feasibility constraints is developed, which achieves superior coordination efficiency and zero-shot scalability in asymmetric multi-UAV confrontational scenarios.

What are the implications of the main findings?

• The proposed framework provides a computationally efficient, real-time applicable solution for multi-UAV decision-making in safety-critical dynamic airspace, with broad civilian prospects in UAV swarm coordination, emergency response and infrastructure inspection.
• The physics-informed graph learning paradigm established in this work can be extended to general multi-agent cooperative decision-making scenarios, including swarm robotics, distributed sensing and autonomous air traffic management.

Abstract

Autonomous task assignment in multi-unmanned aerial vehicle (UAV) systems operating in dynamic and safety-critical airspace environments is highly challenging due to complex spatial interactions and rapidly changing relative geometries. This paper proposes a hierarchical decision-making framework that bridges individual maneuvering behaviors with cooperative task allocation in multi-agent aerial systems. First, a high-fidelity single-agent maneuver model is learned using a physics-consistent simulation environment, where spatial advantage is evaluated based on relative distance and angular relationships within a kinematically feasible interaction zone (KIZ). Subsequently, a Geometry-Aware Graph Attention Network (GA-GAT) is developed to address scalable multi-agent assignment problems. Unlike conventional approaches that rely on flat feature representations, the proposed method explicitly incorporates kinematic feasibility constraints into the attention mechanism via a novel gating module, enabling efficient relational reasoning under dynamic conditions. The proposed framework is applicable to a range of civilian and safety-oriented scenarios, including UAV swarm coordination, emergency response monitoring, infrastructure inspection, and autonomous airspace management. Simulation results demonstrate that the GA-GAT-based approach significantly outperforms heuristic baselines in terms of coordination efficiency and overall system performance in complex multi-agent environments. This study highlights that decoupling maneuver-level control from high-level coordination provides a scalable and computationally efficient solution for real-time multi-UAV decision-making in safety-critical applications. The proposed framework is designed for general multi-agent coordination problems in civilian aerial applications.

1. Introduction

Autonomous multi-UAV systems operating in dynamic and safety-critical airspace environments have attracted increasing attention due to their broad applications in civilian domains such as emergency rescue coordination, infrastructure inspection, and environmental monitoring [1]. In such highly dynamic scenarios, including cooperative and competitive multi-agent interaction scenarios such as task allocation, collision avoidance, and spatial coordination, multiple UAVs must make real-time, coordinated maneuver decisions to safely handle complex spatial interactions and rapidly evolving relative geometries [2]. The decision-making process is challenged not only by the rapid maneuverability of surrounding agents but also by the intricate spatial relationships within the swarm [3,4]. Therefore, a pivotal challenge lies in establishing a scalable mechanism that can seamlessly map complex global spatial configurations to optimal individual task assignments in real time. It should be noted that the interaction patterns considered in this work are abstract representations of multi-agent spatial relationships and are not intended to model military-specific scenarios.

From a methodological perspective, these problems can be naturally formulated as multi-agent decision-making tasks involving interaction-aware coordination and dynamic resource allocation. Current research in multi-UAV decision-making primarily encompasses expert systems, optimization-based methods, game-theoretic approaches, and deep reinforcement learning, with recent cutting-edge explorations even leveraging Large Language Models to empower multi-UAV task planning [5]. To mimic the cognitive process of human operators, early decision-making models heavily relied on expert systems to construct logical reasoning chains [6]. These methodologies systematically decompose complex decision processes into sequential functional modules, such as situational assessment [7,8,9], target intention prediction [10,11,12], and strategic decision-making [13,14,15,16]. To address specific operational constraints, optimization-based methods [17,18] and advanced cooperative path planning algorithms [19] have been employed to derive collision-free optimal trajectories and coordinated strategies.

While these techniques offer high mathematical precision, they often suffer from heavy computational overhead, making it difficult to achieve the millisecond-level responsiveness required in dynamic multi-UAV environments. Game-theoretic frameworks [20,21,22] provide a rigorous mathematical foundation for modeling competitive and interactive behaviors among multiple agents. By formulating interactions as incomplete information dynamic games [23,24,25,26], researchers have addressed uncertainty and derived optimal responses through equilibrium-based analysis. However, the practical application of these approaches is often hindered by the “curse of dimensionality”; as the number of UAVs increases, the computational complexity of solving for equilibrium or performing recursive inference grows exponentially. Such scalability bottlenecks limit their applicability in large-scale, high-dimensional multi-agent scenarios.

Parallel to game-theoretic approaches, Deep Reinforcement Learning (DRL) has demonstrated significant potential in autonomous cooperative control of UAVs [27] by mapping complex state inputs to high-fidelity maneuver commands. Initially, research focused on one-on-one interaction scenarios, where algorithms such as DQN and DDPG were employed to train agents capable of basic maneuvering behaviors [28,29]. To address team-level coordination, subsequent studies transitioned toward multi-agent reinforcement learning (MARL), utilizing centralized training with decentralized execution (CTDE) frameworks such as MADDPG and MAPPO to foster emergent cooperation [30,31]. In addition, an alternative approach utilizing the Tactical Pursuit Point method is introduced to handle complex dynamic scenarios, constructing a structured decision space based on defensive tactical bases [32]. However, these methods often struggle with “hierarchical complexity” in high-fidelity six-degree-of-freedom environments, where a single policy must simultaneously learn long-horizon coordination and short-horizon control, leading to inefficient exploration.

To mitigate this issue, Hierarchical Reinforcement Learning has been introduced to achieve temporal and functional abstraction. For instance, the Leader-Follower Multi-Agent Proximal Policy Optimization framework decomposes the decision-making process into multiple levels [33]. The top level focuses on high-level coordination and role assignment, while lower levels handle maneuver guidance and execution. This decomposition reduces the learning burden and enables the agents to bridge the gap between abstract coordination objectives and precise control actions.

Despite these advances, existing methods face critical scalability and flexibility challenges. Most DRL architectures rely on Multi-Layer Perceptrons (MLPs) that require fixed-dimensional inputs, making them unsuitable for scenarios with a varying number of agents. In addition, these models often treat the environment as a flat feature space, failing to explicitly capture spatial relational dependencies among agents, which are essential for efficient task assignment in large-scale swarm systems.

To address this limitation, Graph Neural Networks (GNNs) have been introduced to model multi-UAV systems as dynamic graphs. For example, the GraphZero-PPO framework utilizes GraphSAGE to extract relational features, enabling permutation invariance and scalability across different system sizes [34]. However, direct scaling of GNN-based methods still faces challenges such as distribution shift and feature over-smoothing, which may degrade performance in dense multi-agent scenarios.

To effectively resolve the three aforementioned limitations—namely, inadequate temporal feature extraction, computational bottlenecks in large-scale coordination, and the lack of physical constraints in abstract decision spaces—this paper proposes a hierarchical multi-agent decision-making framework. Specifically, our proposed solutions strictly map to these challenges: First, to address the limitation of static feature extraction, we incorporate a Transformer-based architecture integrated with an Actor-memory module to capture long-term temporal dependencies from historical interactions. Second, to overcome computational complexity and ensure scalability, we propose a decoupled hierarchical framework, complemented by a Geometry-Aware Graph Attention Network (GA-GAT) that optimizes target pairing through relational reasoning across diverse $M\times N$ scenarios. Third, to bridge the gap between abstract decisions and physical flight limits, we model the interaction constraints through a kinematically feasible interaction zone (KIZ), enabling physically consistent and interpretable decision-making.

It is crucial to distinguish our framework from existing vanilla GNN-MARL architectures such as standard MAPPO-GNN. Conventional graph-based methods typically rely on dense neighborhood aggregation. In dense multi-UAV scenarios, this approach is theoretically prone to feature over-smoothing and fundamentally fails to account for physical flight constraints. In contrast, our GA-GAT represents a structural improvement rather than a mere algorithmic combination. By introducing the KIZ gating mechanism, the framework explicitly prunes physically irrelevant connections in the graph topology. This structurally mitigates the risk of over-smoothing and ensures that the attention mechanism focuses exclusively on geometrically meaningful interaction patterns, enabling physically executable and scalable coordination.

It is essential to clarify the theoretical improvement of our hierarchical architecture compared to conventional decoupling strategies. While separating task assignment from maneuver control is a common paradigm, traditional hierarchical MARL frequently suffers from a “kinematic gap”: high-level policies operating in abstract discrete spaces often generate assignment commands that are physically unexecutable by low-level continuous controllers, leading to severe cross-level non-stationarity. Our approach structurally resolves this flaw through Kinematics-Aware Decoupling. Rather than a naive separation, our framework structurally bridges the two layers by embedding the KIZ directly into the high-level GA-GAT topology. This mechanism strictly bounds abstract relational reasoning with low-level physical executability, thereby theoretically preventing unfeasible assignments and ensuring highly stable cross-layer policy convergence.

The main contributions of this research are summarized as follows:

• Physics-Embedded Relational Topology for Constraint-Aware Graph Learning: We introduce a physics-informed relational modeling mechanism in which kinematic feasibility is embedded directly into the graph attention topology. Instead of treating interaction geometry as auxiliary features, the proposed mechanism reformulates physical constraints as structural reweighting factors in the attention computation, improving interpretability and stability in dense multi-agent scenarios.
• Multi-Scale Relational Reasoning via Coupled GA-GAT–Transformer Architecture: To address the mismatch between local interaction geometry and global coordination, we establish a two-scale relational inference pipeline. The graph neural network captures pairwise kinematic dependencies, while the Transformer encoder models higher-order coordination patterns, enabling consistent performance under varying system scales.
• Cross-Level Non-Stationarity Mitigation through Structural Decoupling: We propose a structural decoupling strategy that separates target assignment learning from low-level maneuver adaptation. By stabilizing maneuver policies during assignment optimization, the framework mitigates cross-level non-stationarity in hierarchical multi-agent reinforcement learning and improves convergence robustness.

The remainder of this paper is organized as follows. Section 2 establishes the system model and formulates the problem, including UAV kinematic equations and the spatial advantage assessment model. Section 3 formulates the multi-UAV coordination task as a Partially Observable Markov Decision Process (POMDP), where the state space, action space, and reward functions are defined. Section 4 presents the proposed hierarchical framework, including the Transformer-based Actor-memory module and the GA-GAT-based assignment mechanism. Section 5 provides simulation results and performance analysis in complex M vs. N scenarios. Finally, Section 6 concludes the paper.

2. Problem Formulation and Modeling

In this section, a decision-level kinematic model and a maneuver-based control abstraction are introduced to provide a realistic yet tractable physical basis for multi-UAV cooperative aerial interaction. Each UAV is abstracted as a point-mass kinematic system for high-level maneuver decision-making, while the execution of maneuver commands and the enforcement of aerodynamic constraints are handled by the JSBSim (v1.1.6, https://jsbsim.sourceforge.net/, accessed on 22 April 2026) flight dynamics simulator. As analytically detailed in Figure 1, the representation of maneuver control inputs is not merely illustrative; it establishes the formal kinematic boundaries of the continuous action space ${\mathcal{A}}_i$. Specifically, the figure visualizes the non-linear relationship between the control commands (load factors $n_x,n_z$ and bank angle $\mu$) and the resulting spatial trajectory variations (e.g., flight path angle $\gamma$ and turning radius). By explicitly defining this mapping, Figure 1 demonstrates the physical constraints under which the low-level maneuver policy must operate, ensuring that the actions selected by the reinforcement learning algorithm are aerodynamically executable within the JSBSim simulation environment.

2.1. State Representation

To focus on high-level decision-making and multi-agent coordination, each aircraft is modeled as a simplified point-mass kinematic system. High-fidelity aerodynamic effects and actuator constraints are handled by the JSBSim flight dynamics simulator.

The state of the i-th aircraft is defined in a fixed inertial Cartesian coordinate frame as

$${\mathbf{s}}_i=\left[x_i,y_i,z_i,v_i,{\psi }_i,{\gamma }_i\right],$$

(1)

where $x_i$, $y_i$, and $z_i$ denote the aircraft position, $v_i$ is the airspeed magnitude, ${\psi }_i$ represents the heading angle, and ${\gamma }_i$ denotes the flight-path angle. The horizontal plane is assumed to be flat, and the vertical axis points upward.

2.2. Equations of Motion

To maintain consistency with the discrete maneuver commands, the aircraft’s motion is governed by a three-degree-of-freedom point-mass dynamic model. The state derivatives are expressed as a function of the tangential load factor $n_x$, the normal load factor $n_z$, and the bank angle $\mu$:

$$\begin{array}{cc}\hfill {\dot{x}}_i& =v_{i}cos{\gamma }_{i}cos{\psi }_i,\hfill \\ \hfill {\dot{y}}_i& =v_{i}cos{\gamma }_{i}sin{\psi }_i,\hfill \\ \hfill {\dot{z}}_i& =v_{i}sin{\gamma }_i,\hfill \\ \hfill {\dot{v}}_i& =g(n_{x,i}-sin{\gamma }_i),\hfill \\ \hfill {\dot{\gamma }}_i& ={\displaystyle \frac{g}{v_i}}(n_{z,i}cos{\mu }_i-cos{\gamma }_i),\hfill \\ \hfill {\dot{\psi }}_i& ={\displaystyle \frac{g{n}_{z,i}sin{\mu }_i}{v_{i}cos{\gamma }_i}},\hfill \end{array}$$

(2)

where g is the gravitational acceleration. In this formulation, the control vector is defined as ${\mathbf{u}}_i={[n_{x,i},n_{z,i},{\mu }_i]}^{\top }$. This representation directly maps the discrete maneuvers to the continuous-time state evolution, where $n_{z,i}$ and ${\mu }_i$ collectively determine the curvature and orientation of the flight path. To strictly bridge the continuous-time kinematic differential equations with the discrete-time optimization of the reinforcement learning framework, we employ a multi-rate simulation architecture supported by the JSBSim engine. In the context of the POMDP formulation, Equation (2) serves as the core physics engine driving the state transition function $\mathcal{T}\left(s_{t+1}\right|s_t,a_t)$. To suppress discretization errors and ensure numerical stability during policy training, a dual-loop timing scheme is implemented. At the physics level, the kinematic states are integrated using a 4th-order Runge-Kutta solver at a high internal frequency. Conversely, the RL policy generates tactical maneuver commands at a lower decision frequency. The discrete maneuver commands are smoothly interpolated by low-level autopilots across the intermediate physics steps. Furthermore, to prevent gradient explosion and ensure neural network stability during PPO updates, the raw unbounded kinematic outputs from the integration are strictly normalized and clipped before being mapped into the observation space $\mathcal{O}$. This rigorous treatment guarantees that the policy learns from aerodynamically valid and numerically stable trajectory transitions.

2.3. Spatial Advantage and Game Dynamics

To quantitatively evaluate the game engagement effectiveness, we establish a situational assessment model that maps the kinematic states to tactical advantages. Instead of relying on absolute coordinates, the model focuses on the relative geometry between the ego UAV i and the target j.

2.3.1. Relative Geometric Formulation

Let the state of each UAV be defined by its position $\mathbf{p}={[x,y,z]}^T$ and ${\mathbf{v}}_i$ denote the velocity vector of the UAV in the inertial frame, which can be derived from its airspeed magnitude $v_i$, flight-path angle ${\gamma }_i$, and heading angle ${\psi }_i$ as ${\mathbf{v}}_i={[v_{i}cos{\gamma }_{i}cos{\psi }_i,v_{i}cos{\gamma }_{i}sin{\psi }_i,v_{i}sin{\gamma }_i]}^T$. The relative geometry is governed by the line-of-sight (LOS) vector ${\mathbf{d}}_{ij}={\mathbf{p}}_j-{\mathbf{p}}_i$, with the engagement distance $R_{ij}=\parallel {\mathbf{d}}_{ij}\parallel$.

The tactical orientation is characterized by the Azimuth Angle (${\varphi }_{az}$) and the Aspect Angle (${\varphi }_{asp}$), as illustrated in Figure 2. These angles are derived from the inner product of the velocity vectors and the LOS vector:

$${\varphi }_{az}=arccos\left({\displaystyle \frac{{\mathbf{v}}_i·{\mathbf{d}}_{ij}}{\parallel {\mathbf{v}}_i\parallel R_{ij}}}\right),{\varphi }_{asp}=arccos\left({\displaystyle \frac{{\mathbf{v}}_j·{\mathbf{d}}_{ij}}{\parallel {\mathbf{v}}_j\parallel R_{ij}}}\right)$$

(3)

In this formulation, ${\varphi }_{az}$ represents the deviation of the target from the ego UAV’s velocity vector, while ${\varphi }_{asp}$ signifies the orientation of the target’s tail relative to the LOS. These parameters are directly coupled with the flight-path angles (${\gamma }_i,{\psi }_i$) and $v_i$ defined in the kinematic model in Equation (2), ensuring physical consistency across the decision-making hierarchy.

2.3.2. Situation-Adaptive Advantage Assessment

The relative geometry used in close-range multi-agent interactions is illustrated in Figure 2. From a computational deployment perspective, the geometric parameters extracted from this relative spatial topology are not merely analytical metrics. They are dynamically fed into the spatial advantage model to formulate the posture reward $r_{post}$, thereby explicitly driving the reinforcement learning policy to seek tactically favorable geometries. The overall spatial advantage $S_{adv}(i,j)$ is derived by aggregating four critical interaction factors using a dynamic weighting strategy. The global advantage function is formulated as:

$$S_{adv}(i,j)=\mathbf{w}{\left(\mathcal{S}\right)}^T·{\mathbf{S}}_{ij}=w_{ang}{S}_{ang}+w_{dist}{S}_{dist}+w_{vel}{S}_{vel}+w_{alt}{S}_{alt}$$

(4)

where ${\mathbf{S}}_{ij}={[S_{ang},S_{dist},S_{vel},S_{alt}]}^T$ represents the normalized score vector for angle, distance, velocity, and altitude, respectively. The weights $\mathbf{w}\left(\mathcal{S}\right)$ are dynamically adjusted based on the current interaction configuration $\mathcal{S}$ (e.g., close-range maneuvering, head-on configuration, or pursuit configuration). The detailed mathematical definitions of each scoring component are summarized in

Table 1. Mathematical formulations of spatial advantage scoring functions.

Component	Notation	Mathematical Formulation
Angular	$S_{ang}$	${\displaystyle 1-\frac{|{\varphi }_{az}|+|{\varphi }_{asp}|}{2\pi }}$
Distance	$S_{dist}$	${\displaystyle \left\{\begin{array}{cc}1.0\hfill & R_{ij}\le R_w\hfill \\ exp\left(-\frac{{(R_{ij}-R_w)}^2}{2{\sigma }_d^2}\right)\hfill & R_{ij}>R_w\hfill \end{array}\right.}$
Velocity	$S_{vel}$	${\displaystyle \frac{v_i}{v_{op}}·exp\left(-\frac{2|v_i-v_{op}|}{v_{op}}\right)}$
Altitude	$S_{alt}$	${\displaystyle \left\{\begin{array}{cc}exp\left(-\frac{\Delta h^2}{2{\sigma }_h^2}\right)\hfill & \Delta h\le 0\hfill \\ 1.0\hfill & 0<\Delta h\le {\sigma }_h\hfill \\ exp\left(-\frac{{(\Delta h-{\sigma }_h)}^2}{2{\sigma }_h^2}\right)\hfill & \Delta h>{\sigma }_h\hfill \end{array}\right.}$

.

In Table 1, $R_w$ denotes the optimal interaction range, ${\sigma }_d$ and ${\sigma }_h$ are the decay coefficients for distance and altitude, and $v_{op}$ is the optimal reference velocity calculated dynamically based on the relative closing rate.

To ensure real-time computational efficiency for reinforcement learning while maintaining high-fidelity tactical evaluation, we constructed a two-stage offline-to-online evaluation pipeline guided by established multi-criteria assessment methods [35]. Instead of computing complex fuzzy logic inference matrices online—which would introduce severe computational overhead and reward non-stationarity during MARL training—the final weight vector $w\left(\mathcal{S}\right)$ is formulated as a convex combination of a static baseline weight vector $w_{sta}$ and a situation-adaptive dynamic weight vector $w_{dyn}\left(\mathcal{S}\right)$:

$$w\left(\mathcal{S}\right)=\alpha ·w_{sta}+(1-\alpha )·w_{dyn}\left(\mathcal{S}\right)$$

(5)

where the blending coefficient $\alpha \in [0,1]$ governs the trade-off between baseline stability and adaptability. Through extensive empirical MARL tuning, $\alpha =0.5$ was identified as the optimal balance.

It is crucial to emphasize that these parameters are not heuristically guessed. The static weight vector $w_{sta}$ is derived from an offline data-driven variance analysis (via the CRITIC method), which mathematically established that angular alignment is the dominant determinant of close-range tactical superiority, setting $w_{sta}={[0.80,0.10,0.05,0.05]}^T$. Specifically, the static baseline weight vector is established as $w_{sta}$, following the CRITIC-based optimization framework established in [35]. These values represent the normalized and optimized results derived from offline engagement data analysis, where angular alignment is mathematically identified as the most significant contributor to tactical success. To ensure the robustness of the reward signal across various engagement scales, these optimized values serve as the nominal constants in our situational advantage model.

Furthermore, the dynamic weights $w_{dyn}\left(\mathcal{S}\right)$ are strictly calculated using the Fuzzy Multi-Criteria Decision Making (Fuzzy MCDM) methodology established in [35]. To prevent online non-stationarity and computational bottlenecks, the fuzzy logic evaluation was executed offline. We rigorously evaluated the interaction space across four representative geometric configurations (self-advantage, counterpart-advantage, mutual-advantage, and mutual-disadvantage) using the Fuzzy MCDM framework. The resulting optimized weight vectors for each configuration were then crystallized into robust lookup tables, serving as a stationary and high-fidelity reward shaping mechanism for the RL agent. These configurations are summarized in

Table 2. Dynamic weight vectors $w_{dyn}\left(\mathcal{S}\right)$ based on interaction configurations.

Configuration ($\mathcal{S}$)	Angle (${\mathit{w}}_{\mathit{ang}}$)	Distance (${\mathit{w}}_{\mathit{dist}}$)	Speed (${\mathit{w}}_{\mathit{vel}}$)	Altitude (${\mathit{w}}_{\mathit{alt}}$)
Self-advantage	0.332	0.291	0.209	0.168
Counterpart-advantage	0.325	0.210	0.278	0.287
Mutual-advantage	0.239	0.328	0.278	0.155
Mutual-disadvantage	0.111	0.313	0.403	0.173

.

2.3.3. Kinematic Interaction Zone and Time-Integrated Effectiveness Decay Model

Precise mapping of agent interaction feasibility traditionally relies on high-fidelity 6-DOF simulations and complex surrogate models [36]. While such approaches capture non-linear boundaries defined by multidimensional kinematic factors, querying these heavy models at every integration step for $M\times N$ agent pairs is computationally prohibitive for MARL. To mitigate this computational bottleneck and the sparse reward problem inherent in binary interaction outcomes, we abstract the interaction using a time-integrated effectiveness decay model based on a KIZ, as illustrated in Figure 3.

Rather than simulating explicit trajectories, this model quantifies the continuous reduction in an agent’s operational effectiveness when operating within a valid interaction geometry. The effectiveness index, $H_j\left(t\right)$, of agent j is updated at each simulation step based on the cumulative interaction contributions from other agents:

$$H_j(t+1)=max\left(0,H_j\left(t\right)-\sum _{i\in {\mathcal{N}}_{interacting}}\mathbb{I}(i,j)·D_{ij}\left(t\right)\right)$$

(6)

To enhance robustness in dense multi-agent interactions, the aggregation in Equation (6) adopts a max-based formulation, which limits excessive accumulation of interaction effects. This helps avoid unstable scaling when multiple agents are present.

The linear decay term further attenuates historical interactions over time, preventing long-term accumulation of outdated influence.

We note that this formulation is heuristic in nature. Alternative smooth aggregation schemes, such as softmax-based weighting, may be explored in future work to provide improved differentiability and theoretical guarantees.

To model feasibility in close-range interactions, a simplified KIZ is introduced to determine whether an agent lies within a valid interaction geometry. The KIZ is implemented as an indicator function enforcing hard boundaries based on relative distance and off-trajectory angular constraints:

$$\mathbb{I}(i,j)=\left\{\begin{array}{cc}1,\hfill & R_{min}\le R_{ij}\le R_{max}\mathrm{and}\left|{\varphi }_{az}\right|\le {\varphi }_{az,max},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

We note that the current KIZ formulation adopts a hard-threshold gating mechanism for computational efficiency. While effective in practice, such binary gating may introduce discontinuities in the interaction signal. Future work may consider smooth approximations, such as sigmoid-based soft gating, to improve differentiability and learning stability.

Only when the relative geometry satisfies the kinematic constraints is the interaction considered valid. When an agent enters the KIZ, a continuous effectiveness decay rate is applied to represent the impact of the interaction. The decay rate is defined as:

$$D_{ij}\left(t\right)=min\left(D_{max},D_{\mathrm{base}}{f}_{ang}{f}_{dist}\right),$$

(8)

where $D_{\mathrm{base}}$ denotes the nominal decay coefficient, and $D_{max}$ limits the maximum instantaneous reduction rate. The angular and distance shaping factors respectively capture the favorability of the interaction geometry:

$$f_{ang}=max\left(0,1-{\displaystyle \frac{|{\varphi }_{az}|}{{\varphi }_{az,max}}}\right)max\left(0,1-{\displaystyle \frac{|{\varphi }_{asp}|}{{\varphi }_{asp,max}}}\right),$$

(9)

$$f_{dist}=max\left(0,{\kappa }_{dist}-{\displaystyle \frac{R_{ij}}{R_{max}}}\right).$$

(10)

These shaping terms encourage geometrically advantageous interactions, assigning higher effectiveness impact to close-range aligned or pursuit-like configurations while suppressing unfavorable conditions. The resulting process serves as a continuous surrogate for interaction effectiveness evaluation, allowing agents’ operational effectiveness to decay continuously over time. An agent is considered functionally inactive and removed from further interactions once its effectiveness state reaches zero, i.e., $H_j\left(t\right)\le 0$.

2.4. Discrete Maneuver Command Library

To bridge the gap between high-level tactical decision-making and low-level flight dynamics, the continuous control space of the UAV is discretized into a finite set of maneuver primitives $\mathcal{M}=\{a_1,a_2,\dots ,a_{11}\}$. The design of the 11 maneuver primitives in

Table 3. Control command encoding for the discrete maneuver library based on gravity compensation.

Index	Maneuver Description	${\mathit{n}}_{\mathit{x}}$	${\mathit{n}}_{\mathit{z}}$	Bank ($\mathit{\mu }$)
$a_1$	Constant-velocity level flight	$sin\gamma$	$cos\gamma$	0
$a_2$	Maximum acceleration	$sin\gamma +1$	$cos\gamma$	0
$a_3$	Maximum deceleration	$sin\gamma -1$	$cos\gamma$	0
$a_4$	Vertical climb	$sin\gamma$	$cos\gamma +5$	0
$a_5$	Vertical dive	$sin\gamma$	$cos\gamma -5$	0
$a_6$	Right-turn climb	$sin\gamma$	$cos\gamma +5$	$-{\mu }_0$
$a_7$	Left-turn dive	$sin\gamma$	$cos\gamma -5$	${\mu }_0$
$a_8$	Left-turn climb	$sin\gamma$	$cos\gamma +5$	${\mu }_0$
$a_9$	Right-turn dive	$sin\gamma$	$cos\gamma -5$	$-{\mu }_0$
$a_{10}$	Left level turn	$sin\gamma$	$cos\gamma$	${\mu }_0$
$a_{11}$	Right level turn	$sin\gamma$	$cos\gamma$	$-{\mu }_0$

Note: $n_x$ and $n_z$ are the tangential and normal load factors; $\gamma$ represents the flight-path angle; ${\mu }_0$ is the preset bank angle for turning maneuvers.

represents a deliberate trade-off between control fidelity and the tractability of reinforcement learning. In large-scale multi-agent environments, expanding the action space to continuous domains or highly granular discrete sets may significantly increase the exploration complexity and make policy learning more challenging. To address this, the primitive library is constructed to cover representative boundary conditions of the UAV’s operational flight envelope, such as maximum load factors, bank angles, and steady-state flight. The underlying high-fidelity JSBSim environment further provides continuous aerodynamic responses between these commands, allowing the framework to retain physically meaningful behavior. We note that this discretization simplifies the decision space at the cost of reduced action resolution. Nevertheless, it provides a practical balance for stable learning in dense $M\times N$ scenarios. Future work may consider expanding the maneuver set or adopting hybrid action representations to improve flexibility.

As summarized in Table 3, each maneuver $a_i$ is parameterized by a triplet of control setpoints: the tangential load factor $n_x$, the normal load factor $n_z$, and the bank angle $\mu$. These parameters directly drive the aircraft’s kinematic state derivatives defined in Equation (2). Specifically, $n_x$ modulates the longitudinal acceleration ${\dot{v}}_i$, while the coupling of $n_z$ and $\mu$ determines the evolution of the flight-path angle ${\dot{\gamma }}_i$ and the heading rate ${\dot{\psi }}_i$.

The design of the command setpoints incorporates gravity feed-forward compensation to neutralize inherent environmental disturbances. As formulated in Equation (2), the gravity-related terms $-gsin{\gamma }_i$ and $-gcos{\gamma }_i$ continuously influence the UAV acceleration and flight-path rate. To counteract these effects, corresponding trigonometric compensations are embedded directly in the maneuver command definitions summarized in Table 3. For example, during level turning maneuvers the normal load factor is set to $n_z=cos\gamma$, which cancels the vertical gravity component and maintains a steady flight-path angle.

To bridge the high-level kinematic commands with the six-degree-of-freedom JSBSim aerodynamic simulator, a low-level proportional–integral–derivative (PID) autopilot is implemented. This inner-loop controller converts the compensated command setpoints $(n_x,n_z,\mu )$ into actuator inputs, including throttle, elevator, and aileron deflections, thereby ensuring physically consistent aerodynamic execution of the tactical decisions.

3. POMDP Formulation for Multi-Agent Aerial Interaction

In complex M vs. N multi-agent interaction scenarios, the decision-making process is modeled as a Partially Observable Markov Game (POMG). This framework captures the interactive and potentially competitive nature of the environment, the need for cooperative coordination among agents, and the inherent uncertainty due to limited sensing capabilities.

3.1. POMG Framework

This framework is formally defined by the tuple $(\mathcal{I},\mathcal{S},\left\{{\mathcal{A}}_i\right\},\mathcal{P},\left\{{\nabla }_i\right\},\left\{{\mathcal{O}}_i\right\},\Omega ,\gamma )$. In this formulation, $\mathcal{I}=\{1,\dots ,M\}$ represents the set of cooperative agents, while $\mathcal{S}$ denotes the global state space that encompasses the kinematic states and system status of all participating agents. The action space for each agent i is defined by ${\mathcal{A}}_i$, with the state transition probability governed by the mapping $\mathcal{P}:\mathcal{S}\times \mathcal{A}\to \Delta \left(\mathcal{S}\right)$. The immediate feedback for agent i is provided by the reward function ${\nabla }_i:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$. Due to partial observability, agents rely on the set of local observations ${\mathcal{O}}_i$, which are generated according to the observation probability distribution $\Omega :\mathcal{S}\times \mathcal{A}\to \Delta \left(\mathcal{O}\right)$. Finally, $\gamma \in [0,1)$ serves as the discount factor for long-term reward aggregation.

3.2. Observation Space

Due to onboard sensing constraints, agents operate under partial observability within a limited detection radius $R_{det}$. To isolate and evaluate the core decision-making algorithm, we adopt an idealized observation model within this interaction range. Specifically, for any entity within the detection sphere ($R_{ij}\le R_{det}$), the ego agent is assumed to have instantaneous and noise-free access to its kinematic state. This abstraction intentionally removes lower-level sensing uncertainties (e.g., signal attenuation or communication delays) to focus on high-level decision-making performance. The detailed composition of the observation space is presented in

Table 4. Configuration of the local observation space ${\mathcal{O}}_i$ for the UAV agents.

Category	Feature	Symbol	Dim.
Ego State	Altitude	$h_i$	1
	Roll angle	${\varphi }_i$	2
	Pitch angle	${\theta }_i$	2
	Body-frame velocity	${\mathbf{v}}_i^b$	3
	Calibrated airspeed	$v_{c,i}$	1
Target Info	Relative X-velocity	$\Delta u_{ij}$	1
	Relative altitude	$\Delta h_{ij}$	1
	Azimuth angle (ATA)	${\varphi }_{az}$	1
	Target aspect angle (AA)	${\varphi }_{asp}$	1
	Relative distance	$R_{ij}$	1
	Side indicator	$I_{side}$	1
Safety Info	Relative position	$\Delta {\mathbf{p}}_{col}$	3
	Inverse safety distance	$D_{col}^{-1}$	1

. It should be noted that the current observation space assumes ideal, noise-free sensing within the detection radius. This assumption is deliberately made to isolate and evaluate the pure decision-making and relational reasoning capabilities of the GA-GAT framework without entangling them with state-estimation errors. For practical engineering deployments, this idealized observation space would be coupled with upstream multi-sensor fusion modules to process raw noisy signals into the structured state representations required by our policy.

3.3. Hierarchical Action Space

Consistent with the proposed decoupled framework, the action space is decomposed into two distinct levels. This hierarchical design allows the agent to handle high-level coordination and low-level maneuvering separately.

3.3.1. High-Level Action (Task Assignment)

The upper-level policy outputs a discrete selection action to determine the target interaction pairing:

$$a^{assign}=T_{target}\in \{1,2,\dots ,N\}$$

(11)

where N is the number of detected counterpart agents. This action defines the coordination objective for the lower-level controller.

3.3.2. Low-Level Maneuver Action (Flight Control)

The lower-level policy executes precise 6-DOF control by selecting a discrete command from the maneuver library $\mathcal{M}$ defined in Table 3:

$$a=m_{cmd}\in \{a_1,a_2,\dots ,a_{11}\}$$

(12)

By decoupling $a^{assign}$ and a, the dimensionality of the action space is significantly reduced, avoiding the exponential growth typical in joint continuous-discrete spaces.

3.4. Hierarchical Reward Function Design

To support the proposed hierarchical learning framework, the reward design is decomposed into two stages corresponding to maneuver learning and high-level coordination. The objective of Phase I is to learn physically feasible and stable maneuver behaviors, while Phase II focuses on cooperative decision-making and overall system effectiveness. Such staged reward decomposition reduces cross-level interference between maneuver generation and coordination optimization, thereby improving training stability and policy specialization. The components of hierarchical reward are listed in

Table 5. Definitions of hierarchical reward components.

Phase	Reward	Purpose
Phase I: Maneuver-Level
	$r_{alt}$	Penalizes unsafe low-altitude flight and excessive downward velocity.
	$r_{post}$	Encourages favorable spatial configurations based on orientation alignment and interaction distance.
Phase II: Coordination-Level
	$r_{assign}$	Encourages cooperative task allocation and penalizes redundant assignments.
	$r_{eff}$	Rewards improvements in system-level effectiveness.
	$r_{surv}$	Encourages maintaining operational capability and penalizes failure states.
	$r_{time}$	Applies a constant time penalty to encourage efficient completion.

. The specific weight coefficients for all reward components across both phases are detailed in

Table 6. Reward weight coefficients.

Phase	Parameter	Value	Description
Phase I: Maneuver-Level Weights
	$w_{post}$	1.0	Base weight for interaction posture optimization
	$w_{alt}$	0.5	Weight for altitude safety penalty (soft constraint)
Phase II: Coordination-Level Weights
	$w_{eff}$	2.0	High priority weight for overall system effectiveness
	$w_{assign}$	1.0	Weight for cooperative task assignment
	$w_{surv}$	1.0	Weight for maintaining operational capability
	$w_{time}$	0.05	Small penalty to encourage efficiency without inducing premature termination

.

3.4.1. Maneuver-Level Reward

During Phase I, the reward focuses on learning stable and physically meaningful maneuver behaviors without introducing task-level objectives. The maneuver reward is defined as

$$r_{maneuver}=w_{alt}{r}_{alt}+w_{post}{r}_{post}$$

(13)

where $r_{alt}$ penalizes unsafe low-altitude flight, and $r_{post}$ evaluates the relative interaction posture.

• Altitude Safety Reward

To prevent unsafe low-altitude maneuvers, a safety penalty is introduced when the aircraft descends below predefined altitude thresholds. The altitude reward consists of two components reflecting vertical velocity risk and altitude violation:

$$r_{alt}=P_v+P_h$$

(14)

where $P_v$ penalizes excessive downward velocity near the safety altitude, and $P_h$ penalizes severe altitude violations below a critical threshold.

• Interaction Posture Reward

To encourage favorable spatial configurations, the posture reward is defined based on the relative geometry between the ego agent and interacting agents:

$$r_{post}=r_{ori}(AO,TA)·r_{dis}\left(R\right)$$

(15)

where $AO$ denotes the angle-off between the agent heading and the line of sight, $TA$ represents the aspect angle, and R is the relative distance. The orientation term encourages alignment toward the interaction direction, while the distance term promotes operation within effective interaction ranges.

3.4.2. Coordination-Level Reward

After maneuver policies are stabilized, Phase II introduces coordination-level rewards to guide cooperative multi-agent behavior. The reward is defined as

$$r_{coord}=w_{assign}{r}_{assign}+w_{eff}{r}_{eff}+w_{surv}{r}_{surv}+w_{time}{r}_{time}$$

(16)

where each component corresponds to a key coordination objective.

• Task Assignment Reward

To promote cooperative allocation, the assignment reward $r_{assign}$ evaluates the effectiveness of the selected pairing. As implemented in our environment, it is formulated as a weighted sum of three distinct components:

$$r_{assign}=r_{adv}+r_{coop}+r_{global}$$

(17)

The individual advantage reward $r_{adv}$ encourages the agent to select targets against which it holds a favorable spatial situation. It is defined by the difference in bidirectional situational advantage $S_{adv}$ between the ego agent i and the assigned target j:

$$r_{adv}=2\left(S_{adv}(x_i,x_j)-S_{adv}(x_j,x_i)\right)$$

(18)

where $x_i$ and $x_j$ denote the state vectors of the respective agents. To prevent redundant assignments and resource waste, the team cooperation reward $r_{coop}$ penalizes multiple friendly agents targeting the same enemy. Let k be the total number of friendly agents assigned to target j. The cooperation term is defined as:

$$r_{coop}=\left\{\begin{array}{cc}{c}_{bonus},\hfill & \mathrm{if}k=1\hfill \\ -p_{dup}·k,\hfill & \mathrm{if}k>1\hfill \end{array}\right.$$

(19)

where $c_{bonus}$ is a small reward for dispersed allocation, and $p_{dup}$ is the penalty coefficient for overlapping assignments. To ensure that the learned policy outperforms a naive greedy approach, the global optimality reward $r_{global}$ evaluates the overall assignment set ${\mathcal{A}}_{current}$ against a greedy, non-overlapping baseline ${\mathcal{A}}_{greedy}$. It normalizes the difference by the number of friendly agents N:

$$r_{global}={\displaystyle \frac{1}{N}}\left(\sum _{(i,j)\in {\mathcal{A}}_{current}}\Delta S_{i,j}-\sum _{(i,j)\in {\mathcal{A}}_{greedy}}\Delta S_{i,j}\right)$$

(20)

where $\Delta S_{i,j}=S_{adv}(x_i,x_j)-S_{adv}(x_j,x_i)$. A positive $r_{global}$ indicates that the multi-agent policy has discovered an assignment strategy that is globally superior to the localized greedy heuristic. Finally, the aggregated $r_{assign}$ is normalized and clipped to the range $[-1,1]$ to ensure training stability.

• System Effectiveness Reward

Changes in system-level effectiveness yield a positive reward:

$$r_{eff}=r_{base}\Delta N_{eff}$$

(21)

where $\Delta N_{eff}$ denotes the number of agents whose effectiveness has transitioned below the operational threshold.

To maintain system robustness, a survival-oriented reward is introduced as

$$r_{surv}=\left\{\begin{array}{cc}w{\displaystyle \frac{E}{E_{max}}},\hfill & E>0\hfill \\ P_{fail},\hfill & E\le 0\hfill \end{array}\right.$$

(22)

where E denotes the remaining system effectiveness.

• Time Penalty

A small constant penalty is applied at each timestep

$$r_{time}=P_{time}$$

(23)

to encourage efficient task completion and avoid unnecessarily prolonged interactions.

It is crucial to theoretically justify the strict separation between the coordination-level reward and the maneuver-level reward in our hierarchical formulation. Rather than a heuristic design choice, this decoupling is a mathematical necessity to resolve the credit assignment problem inherent in multi-scale Markov Decision Processes. The high-level assignment policy operates at a macro-temporal resolution, driven by sparse, event-based tactical outcomes. Conversely, the low-level maneuver policy operates at a micro-temporal resolution, requiring dense, continuous spatial advantage signals to guide aerodynamic flight. Aggregating these distinct signals into a unified, flattened reward function would induce severe cross-level gradient interference: the high-frequency dense posture rewards would numerically overwhelm the sparse global tactical signals, trapping the assignment policy in local optima. Therefore, the structural separation of these reward functions guarantees gradient isolation, ensuring that each policy layer optimizes its respective objective without temporal or scaling conflicts. The empirical necessity of this separation is intrinsically corroborated by the performance collapse of flattened baseline architectures that fail to resolve such intertwined reward signals in complex $M\times N$ scenarios.

The separation between maneuver-level and coordination-level rewards is designed to reflect their different temporal characteristics and learning objectives. While this decomposition improves training stability in practice, we acknowledge that further ablation studies would be valuable to isolate its contribution more rigorously.

3.5. State Transition and Termination Conditions

The temporal evolution of the environment is governed by the non-linear 6-DOF kinematic model detailed in Section 2.2, ensuring high-fidelity state transitions. A simulation episode concludes when one of the following terminal criteria is met: (1) Task Completion, achieved when all target agents reach an inactive state; (2) System Failure, triggered if all cooperative agents become inactive; (3) Time Expiration, where the interaction duration exceeds the maximum threshold $T_{max}$; or (4) Boundary Violation, occurring if any agent exits the designated operational airspace defined by $[X_{min},X_{max},Y_{min},Y_{max}]$.

Before detailing the specific neural architectures, it is crucial to emphasize how the aforementioned POMG formulation strictly guides the algorithmic design of the hierarchical framework. Rather than a mere formal restatement, the POMG establishes the explicit computational interfaces for policy learning. Specifically, the defined observation space $\mathcal{O}$ strictly dictates the input feature dimensions for both the Transformer-based memory module and the GA-GAT node features. The defined hierarchical action space $\mathcal{A}$ directly configures the structure of the Actor network’s output layers, ensuring that the policy mapping $\pi \left(a\right|o)$ remains mathematically consistent and aerodynamically executable. Furthermore, the customized reward functions $\mathcal{R}$ serve as the exact empirical return signals that dictate the gradient updates within the PPO objective function. Through this rigorous mapping, the POMG formulation provides the foundational blueprint for both policy learning and evaluation.

4. Methodology: Hierarchical GA-GAT-PPO Framework

The POMG formulation provides the structural basis for the design of the proposed learning framework. Specifically, the observation space defines the input representation to the policy network, the action space determines the structure of the actor outputs, and the reward function directly corresponds to the optimization objective used in PPO training. This mapping ensures consistency between the formal problem definition and the implemented learning architecture.

We propose a hierarchical reinforcement learning framework that decouples the complex multi-UAV game problem into two manageable sub-problems: low-level maneuver execution and high-level coordination decision-making. This section details the training phases and the network architectures of the proposed GA-GAT-PPO algorithm. The overall architecture of the proposed GA-GAT-PPO framework is illustrated in Figure 4 and Figure 5.

4.1. Overall Framework Architecture

The overall training procedure of the proposed hierarchical framework is summarized in Algorithm 1. The proposed decision-making hierarchy is structured as a dual-loop control system, decoupling high-level strategic coordination from low-level maneuver execution through a frequency-separated architecture. The inner loop, designated as the Maneuver Loop, employs a Recurrent PPO agent to map local ego-observations directly to discrete flight commands, such as load factors and bank angles. This layer is primarily optimized for low-level tactical agility and aerodynamic stability within a high-fidelity flight envelope.

Superimposed on this is the Tactical Loop (Outer Loop), which utilizes a GA-GAT to process the global game topology. By extracting relational features from the dynamic interaction between cooperative nodes and counterpart nodes, the outer loop generates optimal target assignments for the entire swarm. A critical aspect of this hierarchical approach is the phased training transition: the parameters of the Maneuver Loop are frozen during the Tactical Loop’s optimization. This sequential freezing mechanism ensures that high-level coordination decisions are grounded in consistent and predictable maneuver capabilities, effectively mitigating the non-stationarity issues common in multi-agent reinforcement learning. To clarify the inter-layer interaction, the communication between high-level assignment and low-level control is governed by a target-conditioned state remapping mechanism. Once the high-level assignment policy ${\pi }_{high}$ determines the target index $T_i$ for agent i, this index is utilized to transform the global state $S_{global}$ into a target-centric local observation $O_{i,low}=f(S_{global},T_i)$. This mapping function filters out irrelevant environmental information and re-centers the kinematic features onto the specifically assigned counterpart. Consequently, the low-level maneuver policy ${\pi }_{low}$ operates on a standardized input space, focusing solely on executing optimal tactical maneuvers against the selected target. This structural decoupling ensures that the high-level policy handles the combinatorial complexity of task allocation while the low-level policy focuses on the physical fidelity of maneuver execution, significantly mitigating the non-stationarity typically found in multi-agent training.

Algorithm 1 Hierarchical Two-Stage Training Framework

1:  Input: Maneuver library $\mathcal{M}$, geometric prior ${\mathcal{B}}_{geo}$, training iterations $N_I,N_{II}$

2:  Initialize: Maneuver policy ${\pi }_{\theta }$ and critic $V_{\varphi }$

3:  Initialize: Assignment policy ${\pi }_{\Theta }$ and centralized critic $V_{\Phi }$

4:

5:  Phase I: Maneuver Policy Learning

6:  for episode $=1,\dots ,N_I$ do

7:       Collect trajectories $\tau =\{o_t,a_t,r_t\}$ using ${\pi }_{\theta }$

8:       Incorporate geometric prior ${\mathcal{B}}_{geo}$ to guide action sampling

9:       Estimate advantages ${\widehat{A}}_t$ via GAE

10:       Update $(\theta ,\varphi )$ using PPO objective

11:  end for

12:

13:  Phase II: Collaborative Assignment Learning

14:  Freeze maneuver policy ${\pi }_{\theta }$

15:  for episode $=1,\dots ,N_{II}$ do

16:       Initialize environment and construct interaction graph ${\mathcal{G}}_t$

17:       for $t=1,\dots ,T$ do

18:           Encode graph features ${\mathbf{h}}_i\leftarrow \mathrm{GA}\text{-}\mathrm{GAT}\left({\mathcal{G}}_t\right)$

19:           Compute assignment distribution ${\pi }_{\Theta }$ via attention-based matching

20:           Sample assignment action $a_t^{assign}\sim {\pi }_{\Theta }$

21:           Execute maneuver action $a_t\sim {\pi }_{\theta }(·\mid a_t^{assign})$

22:           Store transition $(S_t,{\mathbf{a}}_t,R_t,S_{t+1})$

23:       end for

24:       Update centralized critic $V_{\Phi }$ using value loss

25:       Update assignment policy ${\pi }_{\Theta }$ via MAPPO objective

26:  end for

27:

28:  Output: Trained policies ${\pi }_{\theta },{\pi }_{\Theta }$

4.2. Phase I: Maneuver Policy Learning via Recurrent PPO

In the first phase, we train a unified maneuver policy ${\pi }_{{\theta }_m}$ in a high-fidelity 1v1 interaction environment. The goal is to maximize the expected cumulative reward defined in Equation (13).

To improve the robustness and generalization capability of the learned policy, we adopt a self-play training paradigm with a dynamic opponent pool. Instead of relying on fixed or heuristic opponents, the opponent pool consists of historical checkpoints of the ego-agent’s policy collected at different stages of training. Each checkpoint is associated with an ELO rating, which is updated based on its win/loss performance in simulated interactions.

During training, opponents are sampled from this pool according to their ELO scores, typically selecting agents with comparable or slightly varied skill levels. This mechanism introduces diverse and progressively challenging interaction scenarios, effectively preventing overfitting to specific opponent behaviors and mitigating catastrophic forgetting of previously acquired maneuver strategies. Moreover, it provides an implicit curriculum learning process, where the difficulty of opponents naturally increases as the policy improves.

4.2.1. PPO Optimization Objective

To handle the partial observability of the interaction environment, we integrate a Gated Recurrent Unit (GRU) into the policy network. The objective function is defined as:

$$L^{CLIP}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t)\right]$$

(24)

where $r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|o_t,h_{t-1})}{{\pi }_{{\theta }_{old}}\left(a_t\right|o_t,h_{t-1})}}$ denotes the probability ratio, ${\widehat{A}}_t$ is the advantage estimate computed via Generalized Advantage Estimation, and $ϵ$ is the clipping hyperparameter.

4.2.2. Recurrent Actor-Critic Architecture

The maneuver agent adopts a recurrent Actor-Critic architecture to mitigate the effects of partial observability and ensure temporal consistency in high-dynamic interaction scenarios. The processing pipeline of the Actor network is structured into three distinct stages: spatial feature extraction, temporal encoding, and policy generation. The detailed architecture of the GA-GAT actor–critic network is shown in Figure 6. First, the raw observation vector ${\mathbf{o}}_t$ is mapped to a latent feature space ${\mathbf{z}}_t$ via a MLP:

$${\mathbf{z}}_t=\sigma ({\mathbf{W}}_z{\mathbf{o}}_t+{\mathbf{b}}_z)$$

(25)

where $\sigma$ denotes the ReLU activation function. To capture the historical tactical context, the feature vector is fed into a GRU layer, which maintains a hidden state ${\mathbf{h}}_t$ updated as follows:

$${\mathbf{h}}_t=\mathrm{GRU}({\mathbf{z}}_t,{\mathbf{h}}_{t-1})$$

(26)

This recurrent mechanism allows the agent to maintain an internal representation of the game state even during transient sensor dropouts. Finally, the Actor maps the hidden state to a maneuver distribution $\pi \left(a_t\right|{\mathbf{o}}_t,{\mathbf{h}}_{t-1})$ through a Categorical head:

$$\pi \left(a_t\right|{\mathbf{o}}_t,{\mathbf{h}}_{t-1})=\mathrm{Softmax}({\mathbf{W}}_{\pi }{\mathbf{h}}_t+{\mathbf{b}}_{\pi })$$

(27)

To accelerate training and prevent the agent from getting stuck in suboptimal loops, we inject domain knowledge directly into the action distribution. As shown in the PPO Actor design, the Beta distribution parameters $\alpha$ and $\beta$ are dynamically adjusted based on the relative geometry:

$${\pi }_{\theta }\left(a\right|s)\leftarrow (1-\lambda ){\pi }_{\theta }\left(a\right|s)+\lambda \mathcal{B}({\alpha }_{geo},{\beta }_{geo})$$

(28)

When the counterpart agent is within the KIZ, the priors bias the agent towards advantage-seeking maneuver patterns, significantly reducing the exploration space during the early training stages.

Complementarily, the Critic network shares the feature extraction and GRU backbone to estimate the state-value function $V(s_t,{\mathbf{h}}_t)={\mathbf{W}}_v{\mathbf{h}}_t+b_v$. This value estimate serves as the baseline for calculating the advantage ${\widehat{A}}_t$ via GAE. By decoupling the advantage calculation from the specific maneuver choice, the framework achieves stable policy gradient updates, enabling the UAV to execute smooth and geometrically consistent trajectories.

4.3. Phase II: Collaborative Assignment Policy Learning

In the second phase, the maneuver policy ${\pi }_{{\theta }_m}$ is fixed, and a set of M cooperative agents and N counterpart entities are initialized to train the assignment policy ${\pi }_{{\varphi }_{assign}}$.

The multi-agent system is represented as a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where nodes correspond to agents and edges encode relative kinematic relationships. The objective is to learn a permutation-invariant policy that maps each cooperative agent $i\in {\mathcal{N}}_{coop}$ to a counterpart index $j\in {\mathcal{N}}_{cp}$.

The policy is optimized using a MAPPO-style framework with centralized training and decentralized execution. The reward signal ${\mathcal{R}}_{coord}$ evaluates collective system performance rather than individual outcomes, promoting coordinated assignment behaviors.

4.3.1. GA-GAT-Based Actor Computation

The assignment actor encodes the dynamic relational structure into a latent representation via a cascaded GA-GAT architecture. Unlike standard graph models that rely purely on feature similarity, the proposed model incorporates geometric constraints directly into the attention mechanism, enabling differentiation between agents with similar states but distinct relative configurations.

Specifically, each GA-GAT layer performs weighted aggregation modulated by a kinematic gating term. For the first layer, the relational representation ${\mathbf{h}}_i^{\left(1\right)}$ is obtained by integrating relative position $\Delta {\mathbf{p}}_{ij}$ and velocity $\Delta {\mathbf{v}}_{ij}$ into the attention score:

$$e_{ij}^{\left(k\right)}={\mathbf{a}}^{\left(k\right)T}\mathrm{LeakyReLU}\left({\mathbf{W}}^{\left(k\right)}[{\mathbf{h}}_i\Vert {\mathbf{h}}_j]+{\mathrm{MLP}}_{geo}(\Delta {\mathbf{p}}_{ij},\Delta {\mathbf{v}}_{ij})\right)$$

(29)

To rigorously formulate how the GA-GAT architecture processes kinematic constraints, we introduce the Kinematic Masking Function ${\mathcal{M}}_{KIZ}$, which mathematically embeds the physical interaction boundaries directly into the attention mechanism. Instead of applying a standard Softmax to the raw attention score $e_{ij}^{\left(k\right)}$, the kinematics-aware attention coefficient ${\alpha }_{ij}^{\left(k\right)}$ is computed by incorporating the mask:

$${\alpha }_{ij}^{\left(k\right)}={\displaystyle \frac{exp(e_{ij}^{\left(k\right)}+{\mathcal{M}}_{KIZ}(i,j))}{{\sum }_{m\in {\mathcal{N}}_i}exp(e_{im}^{\left(k\right)}+{\mathcal{M}}_{KIZ}(i,m))}}$$

(30)

where the masking function ${\mathcal{M}}_{KIZ}(i,j)$ is mathematically defined based on the KIZ:

$${\mathcal{M}}_{KIZ}(i,j)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathrm{node}j\mathrm{is}\mathrm{within}KI{Z}_i\hfill \\ -\infty ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(31)

This mathematical formulation ensures that if a target agent j is physically unreachable or tactically irrelevant according to the KIZ constraints, the exponential term $exp(-\infty )$ evaluates to exactly zero. Consequently, the attention weight ${\alpha }_{ij}^{\left(k\right)}$ collapses to zero, completely severing the message-passing edge between i and j. This provides the rigorous mathematical foundation for how our architecture structurally prunes physically unexecutable connections, thereby concentrating the network’s relational reasoning strictly on valid tactical topologies and mitigating feature over-smoothing. The output of the first layer is computed by concatenating K independent heads, followed by Layer Normalization (LN) and a residual connection to preserve ego-state integrity:

$${\mathbf{h}}_i^{\left(1\right)}=\mathrm{ReLU}\left(\mathrm{LN}\left(\underset{k=1}{\overset{K}{\Vert }}\sum _{j\in {\mathcal{N}}_i}{\alpha }_{ij}^{\left(k\right)}{\mathbf{W}}^{\left(k\right)}{\mathbf{o}}_j+{\mathbf{o}}_i\right)\right)$$

(32)

The second GA-GAT layer refines these local representations to capture higher-order relational patterns. While the first layer focuses on pairwise interactions, the second layer aggregates contextual information across the graph, enabling the model to encode global structural dependencies.

$${\mathbf{h}}_i^{\left(2\right)}=\mathrm{ReLU}\left(\mathrm{LN}\left(\mathrm{Avg}\left(\sum _{j\in {\mathcal{N}}_i}{\alpha }_{ij}^{\left(K\right)}{\mathbf{W}}^{\left(K\right)}{\mathbf{h}}_j^{\left(1\right)}\right)+{\mathbf{h}}_i^{\left(1\right)}\right)\right)$$

(33)

This stacked GA-GAT architecture ensures that the learned representations are not only permutation-invariant, accommodating a variable number of participants, but also deeply coupled with the 3-DOF kinematic constraints of the engagement. This structural inductive bias significantly accelerates convergence and enhances the interpretability of the target assignment policy in complex M vs. N scenarios.

Building upon these physically grounded embeddings, the actor employs a Scaled Dot-Product Assignment Mechanism to transform relational features into tactical decisions. To perform assignment, node embeddings are projected into role-specific latent spaces. For each cooperative agent i, a query vector ${\mathbf{Q}}_i$ is computed, while each counterpart node j is mapped to a key vector ${\mathbf{K}}_j$:

$${\mathrm{Score}}_{ij}={\displaystyle \frac{{\mathbf{Q}}_i{\mathbf{K}}_j^T}{\sqrt{d_k}}}$$

(34)

This score represents the relative compatibility between agent–target pairs in a shared embedding space.

However, instantaneous assignment based solely on current scores may lead to frequent switching due to dynamic state variations. To improve temporal consistency, a recurrent refinement module is introduced. The raw affinity scores are fed into a Target-GRU unit, which integrates the historical assignment context with the current situational scores. This temporal correction is further augmented by a learnable Inertia Gate, which applies a bias to the previously assigned target:

$${\mathrm{Logits}}_{ij}=\mathrm{GRU}({\mathrm{Score}}_{ij},{\mathbf{h}}_{t-1}^{target})+exp\left({\alpha }_{log}\right)·⊮(j={\mathrm{last}\_\mathrm{target}}_i)$$

(35)

where ${\alpha }_{log}$ is a learned parameter that adaptively regulates the cost of switching targets.

Finally, to ensure the validity of the decision space, an adversarial action mask $\mathbf{M}$ is applied to the refined logits to suppress neutralized or out-of-range entities. The final assignment probability distribution $\pi \left(a_{ij}\right|s)$ is generated through a masked Softmax operation:

$$\pi \left(a_{ij}^{assign}\right|s)={\displaystyle \frac{exp\left({\mathrm{Logits}}_{ij}\right)·M_{ij}}{{\sum }_{k}exp\left({\mathrm{Logits}}_{ik}\right)·M_{ik}}}$$

(36)

This multi-stage pipeline, combining graph-based relational encoding with temporal refinement, enables the policy to produce stable and consistent assignment decisions under dynamically evolving conditions.

4.3.2. Structure-Aware Critic and Global Value Evaluation

The Structure-Aware Critic is designed to provide a comprehensive evaluation of the global multi-agent state under the centralized training paradigm. To ensure feature consistency across the system, the Critic shares the same GA-GAT backbone as the Actor for initial state encoding. By leveraging the kinematic gating term $\mathcal{G}(\Delta {\mathbf{p}}_{ij},\Delta {\mathbf{v}}_{ij})$, the Critic extracts interaction-relevant patterns and produces geometry-aware node embeddings ${\mathbf{h}}_i^{\left(GNN\right)}$, which serve as the foundation for global relational reasoning.

Although the GA-GAT module effectively captures pairwise interaction patterns through localized message passing, its receptive field is inherently limited by neighborhood aggregation. As the number of agents increases, long-range dependencies among spatially distributed entities become difficult to model using GNNs alone. To address this limitation, the node embeddings generated by the GA-GAT layers are further processed by a Graph Transformer module, enabling global relational reasoning across the entire interaction graph. To incorporate domain-specific geometric structure, additional encodings are introduced, including node degree, role identity, and normalized spatial distance:

$${\mathbf{h}}_i^{\left(transformer\right)}={\mathbf{h}}_i^{\left(GNN\right)}+{\mathrm{MLP}}_{deg}\left(d_i\right)+{\mathrm{MLP}}_{role}\left(r_i\right)+{\mathrm{MLP}}_{dist}\left({\mathrm{dist}}_i\right)$$

(37)

where $d_i$, $r_i$, and ${\mathrm{dist}}_i$ denote the node degree, role indicator (cooperative or counterpart), and normalized distance to the global reference frame, respectively. A multi-head self-attention mechanism is then applied to capture long-range dependencies across the graph:

$${\mathrm{Attn}}^{\left(k\right)}({\mathbf{Q}}_i,{\mathbf{K}}_j)=\mathrm{Softmax}\left({\displaystyle \frac{\left({\mathbf{W}}_Q^{\left(k\right)}{\mathbf{h}}_i\right){\left({\mathbf{W}}_K^{\left(k\right)}{\mathbf{h}}_j\right)}^T}{\sqrt{d_{head}}}}\right)$$

(38)

Following the transformer refinement, the Critic employs a role-aware attention pooling strategy to aggregate global information into a compact representation. Instead of standard mean pooling, features are aggregated separately for cooperative and counterpart groups using learnable attention weights:

$${\mathbf{f}}_{coop}=\sum _{i\in {\mathcal{N}}_{coop}}\mathrm{Softmax}\left({\mathbf{w}}_c^T{\mathbf{h}}_i\right){\mathbf{h}}_i,{\mathbf{f}}_{cp}=\sum _{j\in {\mathcal{N}}_{cp}}\mathrm{Softmax}\left({\mathbf{w}}_p^T{\mathbf{h}}_j\right){\mathbf{h}}_j$$

(39)

In addition, a cross-edge pooling module aggregates interaction features between the two groups to capture overall interaction intensity:

$${\mathbf{f}}_{edge}={\displaystyle \frac{1}{|{\mathcal{E}}_{cross}|}}\sum _{(u,v)\in {\mathcal{E}}_{cross}}{\mathrm{MLP}}_{edge}\left({\mathbf{e}}_{uv}\right)$$

(40)

The final global representation is constructed via concatenation:

$${\mathbf{f}}_{global}=[{\mathbf{f}}_{coop}\Vert {\mathbf{f}}_{cp}\Vert {\mathbf{f}}_{edge}\Vert \mathrm{Avg}\left(\mathbf{h}\right)]$$

(41)

This holistic feature vector is then mapped through a multi-layer regressor to estimate the state value $V\left(s\right)$. By integrating geometry-aware graph representations with global transformer-based reasoning, the Critic provides a stable and informative baseline for policy optimization.

The resulting representation captures both local interaction structures and global coordination dependencies, enabling effective evaluation in large-scale and dynamically evolving multi-agent systems.

5. Simulation Results and Analysis

In this section, we evaluate the performance of the proposed hierarchical GA-GAT-PPO framework through high-fidelity numerical simulations. The evaluation focuses on individual maneuvering agility, team-level tactical coordination, and zero-shot scalability across varying game scales. During training, policy evaluation is performed every 10 episodes and averaged over five independent random seeds to ensure statistical reliability.

To ensure the reproducibility of our experiments and provide a comprehensive view of the training process, the detailed hyperparameter configurations for the proposed GA-GAT PPO framework and the MvN simulation environment are summarized in

Table 7. Key hyperparameters of the proposed GA-GAT framework.

Parameter	Value
Environment and Rollout
Parallel environments N	4
Rollout length T	2000
Total timesteps	${10}^7$
Switch penalty ${\lambda }_{\mathrm{switch}}$	$0.05$
Network Architecture
Node feature dimension	18
Edge feature dimension	6
Hidden dimension $d_h$	128
Number of GA-GAT layers	2
Normalization	LayerNorm
Residual connection	Layer 2
RNN hidden dimension	128
RNN sequence length	8
PPO Optimization
Optimizer	Adam
Actor learning rate	$1\times {10}^{-4}$
Critic learning rate	$5\times {10}^{-4}$
PPO epochs	5
Mini-batches	32
Clip ratio $ϵ$	$0.2$
Discount factor $\gamma$	$0.99$
GAE parameter $\lambda$	$0.95$
Value loss coefficient	$0.7$
Entropy coefficient	$0.02\to 0.001$
Max gradient norm	$1.0$

.

5.1. Experimental Setup

The simulation environment is hosted on a workstation equipped with an Intel Core i7-10875H CPU and an NVIDIA GeForce RTX 3060 GPU running Python 3.8 and PyTorch 1.2. The game theater is modeled as a $15\times 15\times 10$ km³ airspace. The initial engagement configuration is summarized in

Table 8. Initial configuration of the air game scenarios.

Parameter	Value
Scenario type	Head-on swarm engagement
Team size	$N_r\in \{4,6\}$ (Red), $N_b\in \{4,7\}$ (Blue)
Initial altitude	$\mathrm{20,000}$ ft
Initial speed	800 ft/s
Team center separation	${0.1}^{\circ }$ latitude
Initial heading	Red: $0^{\circ }$, Blue: ${180}^{\circ }$
Agent initialization	Agents are initialized around the team center with small random offsets
Position perturbation	$\pm {0.03}^{\circ }$ in latitude and longitude
Heading perturbation	$\pm 3^{\circ }$
Altitude perturbation	$\pm 500$ ft

.

To guarantee the stringency and absolute fairness of the comparative analysis, a rigorous evaluation protocol was established to ensure that performance gaps reflect genuine algorithmic advancements rather than subjective hyperparameter mis-tuning. For heuristic baselines, performance is dictated by deterministic operational rules devoid of trainable hyperparameters, making them immune to tuning biases. For deep reinforcement learning baselines, we utilized community-benchmarked, official hyperparameter configurations to prevent manual bias. Crucially, the performance degradation observed in standard MAPPO architectures is fundamentally structural, not parametric. Standard MLP-based networks require fixed-dimensional inputs. In dynamic $M\times N$ environments with varying numbers of active agents, these baselines are forced to employ extensive zero-padding. This padding severely corrupts the input feature distribution and fundamentally disrupts policy convergence when scenarios dynamically change. Because our proposed GA-GAT architecture natively processes variable-sized graphs without padding, the comparative results represent a strict structural superiority rather than a mere artifact of hyperparameter tuning.

To establish a rigorous benchmark, we compare the proposed method against three representative baselines, including a random assignment strategy that allocates targets without tactical reasoning, a greedy nearest-target strategy that tends to cause assignment redundancy, and an auction-based heuristic coordination method commonly adopted for distributed task allocation.

To ensure the rigorous reproducibility and physical fidelity of the proposed framework, it is crucial to articulate the basis for the selected system parameters. Unlike standard algorithmic hyperparameters, the KIZ thresholds are not set heuristically; they are strictly derived from the physical constraints of the UAV’s hardware, representing the maximum effective range of onboard sensors and weapon engagement envelopes. Consequently, treating these physical boundaries as tunable variables for sensitivity analysis would compromise the sim-to-real fidelity of the aerodynamic model.

Furthermore, the weight coefficients and reward scaling factors within the hierarchical formulation were established to mathematically normalize the gradient variance during PPO updates, preventing gradient explosion from dense maneuver rewards. Rather than relying on exhaustive and computationally prohibitive sensitivity grid-searches, the robustness of these coefficients is empirically validated through cross-scenario generalization. The exact same configuration of reward scalings and architectural weights was uniformly applied across all evaluated scales. The successful convergence of the policy across these diverse topological variations without scenario-specific re-tuning serves as robust empirical evidence that the chosen parameters are structurally stable and fundamentally grounded, rather than delicately over-fitted.

• Statistical Evaluation Protocol

All comparative and ablation results are averaged over multiple independent random seeds. The reported curves correspond to mean performance, while the shaded regions indicate the standard deviation, reflecting run-to-run variability. In addition, representative training curves are provided to illustrate convergence behavior and training stability. These curves show the evolution of loss terms and rewards over time and are intended for qualitative analysis rather than statistical comparison.

The parameters used in this study are selected based on a combination of physical considerations and empirical tuning. While the chosen configuration demonstrates consistent performance across multiple scenarios, further sensitivity analysis could provide deeper insight into parameter influence.

5.2. Phase I: Acquisition of Tactical Maneuvering Primitives

The first phase trains a unified maneuver policy ${\pi }_{{\theta }_m}$ in a single-agent maneuver training environment based on the Lightweight Air-to-Air Game. This phase ensures that high-level decisions are grounded in robust individual flight control.

Training Convergence with PKI

As illustrated in Figure 7, we present the training convergence of the initial maneuver model guided by Prior Knowledge Injection (PKI). The learning curve demonstrates rapid and stable convergence, reaching a high asymptotic reward within approximately $2\times {10}^7$ steps. By explicitly confining early random actions to tactically viable regions, this mechanism ensures that the model quickly accumulates meaningful game experiences, thereby establishing a robust foundational policy for the subsequent multi-agent coordination stage.

5.3. Phase II: Multi-UAV Tactical Coordination and Scalability

After freezing the maneuver policy, the GA-GAT-based assignment module is trained to optimize target pairing in complex M vs. N scenarios.

5.3.1. Training Convergence and Ablation Analysis

Figure 8 presents the training dynamics of the proposed GA-GAT-PPO framework. During training, the policy loss fluctuates around a stable range after the early exploration phase, while the value loss gradually decreases, indicating the progressive improvement of the critic estimation. The entropy term steadily declines, suggesting that the policy transitions from exploratory behavior to a more confident decision-making strategy.

In addition, the evaluation reward exhibits a clear increasing trend and eventually stabilizes at a high level, demonstrating the effectiveness and convergence capability of the proposed framework. The shaded bands denote the local standard deviation computed using a rolling window over the reward sequence, reflecting the short-term variability of policy performance during training. This visualization differs from the ablation analysis in Section 5.6, where the shaded regions represent the standard deviation across multiple random seeds. Evaluation is conducted every 10 training episodes and averaged over five independent seeds.

5.3.2. Analysis of 4v4 Scenario

Figure 9 illustrates the dynamic interaction process in the multi-agent scenario, where the GA-GAT-driven policy demonstrates effective coordination capability. By adaptively updating the assignment links, the swarm achieves a coordinated distribution of interaction effort based on the relative priority of each target. This relational reasoning enables agents to balance between focused allocation on high-priority entities and efficient multi-target coverage, ensuring robust system behavior under highly dynamic conditions.

The interaction evolution is further quantified in Figure 10. The proposed method maintains consistently higher spatial advantage values throughout the interaction, while the counterpart’s effectiveness decreases rapidly after the initial interaction phase.

Figure 10 and Figure 11 illustrate how the framework translates spatial configuration advantages into system-level performance. Following an initial interaction phase, cooperative agents establish a clear advantage around step 300 in Figure 10. The mean spatial advantage of the cooperative group increases steadily, while that of the counterpart group declines significantly. This geometric superiority directly correlates with the effectiveness metrics in Figure 11: the cooperative swarm maintains high system effectiveness and operational stability, whereas the counterpart effectiveness gradually decreases to zero under coordinated interactions.

To further analyze the coordination mechanism, Figure 12 illustrates the temporal evolution of assignment decisions. Each row corresponds to a target entity, while the color indicates the agent assigned at each timestep. The assignment patterns exhibit adaptive adjustments, particularly during intermediate stages. These variations reflect the system’s ability to dynamically redistribute coordination responsibilities as the spatial configuration evolves. In certain situations, temporary reassignment is utilized not only for direct interaction but also to guide maneuver trajectories, enabling agents to reposition and establish favorable configurations. As the scenario progresses and the interaction structure becomes clearer, the assignment patterns gradually stabilize, forming longer continuous segments. This behavior demonstrates that the proposed framework achieves a balance between temporal consistency and adaptive flexibility under dynamic conditions.

5.3.3. Analysis of 6v7 Scenario

To further validate scalability, the same GA-GAT architecture is deployed in a 6v7 scenario to evaluate performance under asymmetric agent scales.

As illustrated in Figure 13, the interaction evolves through three representative phases. Initially, both groups maintain a frontal configuration while preserving spatial dispersion. Subsequently, the learned policy drives a subset of agents to execute lateral repositioning maneuvers, breaking symmetry and forming localized density advantages. Finally, through coordinated spatial compression and dynamic assignment reallocation, the system progressively reduces the maneuvering space of counterpart agents, leading to full status degradation.

The framework’s scalability is further validated in Figure 14 and Figure 15. Despite the numerical disadvantage, the GA-GAT-based coordinator maintains higher spatial efficiency, with cooperative agents establishing dominant positional advantages earlier than in smaller-scale scenarios. As shown in Figure 14, the consistent suppression of counterpart advantage values confirms the effectiveness of the learned interaction patterns in dense environments. This translates to accelerated effectiveness reduction for the larger group in Figure 15, where coordinated interactions reduce the active agent ratio to zero within a shorter duration compared to the 4v4 case.

Furthermore, the assignment heatmap in Figure 16 demonstrates that the coordination mechanism effectively manages the increased state space, exhibiting adaptive reassignment behavior without sacrificing stability. These results highlight the robustness of the proposed framework in handling heterogeneous scales and complex interaction topologies.

5.4. Scalability Evaluation

To further evaluate the scalability of the proposed framework, we extend the experiments to larger-scale scenarios, including 5v5 and 8v8 configurations, in addition to the previously considered 4v4 and 6v7 settings. A representative interaction process in the 8v8 scenario is illustrated in Figure 17, showing that the proposed method maintains coordinated behavior and effective target suppression under increased agent density.

For each scenario, we conduct Monte Carlo simulations with randomized initial conditions to ensure statistical reliability. The performance is evaluated in terms of success rate and combat effectiveness. The results are summarized in

Table 9. Performance under different swarm sizes (averaged 1000 Monte Carlo runs).

Scenario	Success Rate (%)	Effectiveness Ratio	System Eff.
4v4	$85.3$	$3.87$	$45.4$
5v5	$86.2$	$3.26$	$22.8$
6v7	$93.2$	$7.52$	$39.1$
8v8	$95.7$	$7.89$	$37.8$

.

As the swarm size increases, a slight degradation in success rate can be observed, which is expected due to the increased interaction complexity and the expansion of the joint state-action space. Nevertheless, the proposed method maintains consistently high performance across all tested scales, demonstrating strong robustness under varying system sizes.

An interesting trend can be observed from Table 9. While the success rate remains relatively stable, the system effectiveness exhibits a decreasing tendency as the swarm size increases. This suggests that the proposed policy increasingly prioritizes outcome-oriented coordination, achieving high success rates by leveraging more intensive interactions and resource consumption.

This observation reveals a trade-off between coordination optimality and efficiency preservation. In larger-scale scenarios, the policy focuses on maximizing overall success, potentially at the cost of reduced residual combat capability. Such behavior reflects a shift from efficiency-oriented coordination to outcome-oriented strategies as system scale increases.

From a scalability perspective, this trade-off also provides insight into the potential limitations of the proposed framework. As system effectiveness continues to decrease with further increases in scale, the available resources for sustained coordination may become insufficient. In such cases, the coordination advantage may no longer compensate for the efficiency loss, which could eventually lead to performance degradation.

This robustness can be attributed to the structure-aware design of the framework. The physics-embedded relational topology enables consistent modeling of interaction constraints, while the multi-scale GA-GAT–Transformer architecture captures both local dependencies and global coordination patterns. In addition, the hierarchical decoupling mechanism stabilizes maneuver execution, reducing sensitivity to scale variations.

From a scalability perspective, this trade-off provides an important indicator of the boundary conditions of the proposed framework. As system effectiveness continues to decrease with increasing scale, the available resources required to sustain coordinated behavior may become insufficient. In such a regime, the coordination advantage can no longer compensate for the efficiency loss, marking a transition point beyond which further scaling is expected to degrade overall performance. This analysis suggests that the scalability of the proposed method is ultimately constrained not by a fixed number of agents, but by the balance between coordination gain and resource consumption.

5.5. Statistical Evaluation and Robustness Analysis

All baselines are implemented under consistent training and evaluation settings. For learning-based methods, commonly adopted hyperparameter configurations are used to ensure fair comparison.

To assess the framework’s robustness against spatial stochasticity, 1000 independent Monte Carlo trials were conducted for each scenario using distinct random seeds. To emulate realistic deployment variations and prevent the neural network from overfitting to fixed geometric patterns, a structured stochastic initialization strategy was adopted. Specifically, different agent groups were assigned distinct operational centers, and the initial coordinates $(x_i,y_i)$ of each UAV were generated by applying a $10\%$ uniformly distributed spatial perturbation relative to their respective group centers. This ensures that the evaluated performance gains are statistically significant and robust across diverse spatial configurations.

The statistical distributions are visualized in Figure 18. To rigorously evaluate the specific contribution of our graph-based architecture, we introduced a standard Flat MAPPO as a reinforcement learning baseline alongside Greedy and Auction methods. As shown in the effectiveness ratio and system performance plots, Flat MAPPO achieves a competitive success rate of 81.0% in the symmetric 4v4 scenario, outperforming traditional heuristic methods. However, its performance exhibits a severe degradation (dropping to 18.0%) when directly transferred to the asymmetric 6v7 scenario.

This drastic degradation reveals a key limitation of standard MLP-based MARL in dynamic multi-agent environments. Since traditional MLPs require fixed-dimensional inputs, adapting a model trained in 4v4 to a 6v7 scenario requires zero-padding or observation truncation. Consequently, agents lose critical spatial awareness, and the system behavior deteriorates into uncoordinated interactions with reduced performance. In contrast, heuristic methods such as Auction, which rely on rule-based allocation, demonstrate relatively stronger robustness to scale variation.

In contrast, the proposed GA-GAT framework naturally accommodates varying numbers of agents without input truncation. The 95% confidence intervals in Figure 18c further demonstrate that the proposed approach achieves not only higher but also more consistent performance. In the 6v7 asymmetric case, the GA-GAT framework maintains a dominant 93.2% success rate and achieves an effectiveness ratio of 7.57—nearly six times that of the Greedy method—while maintaining a high operational retention rate. The combination of high effectiveness ratio and strong success rate highlights that decoupling maneuver primitives and adopting permutation-invariant graph structures is essential for achieving zero-shot scalability in dense $M\times N$ multi-agent interaction scenarios.

Detailed numerical results, including computational overhead and system retention statistics, are summarized in

Table 10. Detailed Performance Statistics from 1000 Monte Carlo Trials. The bold formatting to indicate the best-performing method and results for clarity and consistency in the table presentation.

Assignment	Scenario	Success Rate (%)	Effectiveness Ratio	System Eff.
Greedy	4v4	$68.0\pm 2.9$	$1.83$	$20.3$
Auction	4v4	$77.1\pm 2.6$	$1.72$	$18.2$
Flat MAPPO	4v4	$81.0\pm 2.4$	$2.65$	$28.5$
GA-GAT (Ours)	4v4	$\mathbf{85.3}\pm \mathbf{2.2}$	$\mathbf{3.87}$	$\mathbf{45.4}$
Greedy	6v7	$63.1\pm 3.0$	$1.29$	$4.4$
Auction	6v7	$79.1\pm 2.5$	$1.91$	$16.2$
Flat MAPPO	6v7	$18.0\pm 2.4$	$0.45$	$2.1$
GA-GAT (Ours)	6v7	$\mathbf{93.2}\pm \mathbf{1.6}$	$\mathbf{7.57}$	$\mathbf{39.1}$

. The average computation time highlights the efficiency of parallelized inference. While heuristic methods suffer from combinatorial growth in assignment complexity as the system scales, and flat MARL methods fail to generalize across dimensions, the graph-based approach maintains near-linear computational overhead. This efficiency supports real-time applicability in dynamic multi-agent coordination environments.

5.6. Ablation Study

To evaluate the contribution of key architectural components, ablation experiments are conducted by removing specific modules from the proposed GA-GAT framework. As shown in Figure 19, the complete GA-GAT model is compared with its degraded variants as well as two baseline graph models, namely a GAT and a vanilla GNN aggregator. All models are trained under identical hyperparameter settings, and the reported results are averaged over five independent runs with different random seeds.

The vanilla GNN baseline employs a standard neighborhood aggregation mechanism similar to GraphSAGE, which performs feature propagation without explicitly modeling interaction constraints. As a result, the model tends to suffer from feature over-smoothing in dense multi-agent scenarios and converges to a significantly lower evaluation reward. This phenomenon arises because agents with similar state features may correspond to substantially different interaction priorities depending on their relative geometric relationships.

In contrast, the proposed GA-GAT incorporates kinematic gating mechanisms that explicitly embed KIZ-related constraints into the attention computation, enabling the network to distinguish geometrically meaningful interaction patterns. Furthermore, removing either the Transformer layer or the GRU-based memory module leads to noticeable performance degradation. The Transformer contributes to capturing global relational dependencies among agents, while the GRU module introduces temporal consistency into the decision process. Without these components, the policy becomes less capable of modeling cooperative interaction dynamics and maintaining stable coordination strategies. Overall, the full GA-GAT architecture achieves the highest evaluation reward and the most stable convergence behavior. The shaded regions in Figure 19 indicate the standard deviation across different random seeds, reflecting the stability of each architecture during training.

As demonstrated in the preceding analyses, a two-tiered statistical protocol is established to rigorously validate the proposed framework and ensure full transparency. First, to confirm the statistical significance and training reliability of the performance gains, the core comparative evaluations and ablation studies are aggregated across multiple independent training runs initialized with distinct random seeds. In these macroscopic performance plots, the solid lines represent the mean episodic returns, and the enveloped shaded regions denote the standard deviation (run variance), thereby proving that the algorithmic superiority is robust against environmental stochasticity. Second, for the analysis of auxiliary microscopic dynamics, such as the specific loss convergence behaviors, cross-seed averaging would inherently neutralize the distinct sequential meaning of the gradients. Therefore, these internal optimization metrics are presented using data extracted from a representative median-performing run, where the shaded bands indicate intra-episode smoothing variance rather than cross-seed variance. This decoupled approach guarantees both the rigorous statistical validity of the overall performance and the microscopic interpretability of the training dynamics.

6. Conclusions

This paper addresses the problem of autonomous task assignment in multi-agent systems operating in dynamic and safety-critical environments, where overall performance critically depends on precise maneuver execution and structured spatial interactions. To bridge the gap between individual-level control and system-level coordination, a hierarchical GA-GAT-PPO framework is proposed, integrating a high-fidelity maneuver policy with a geometry-aware graph-based coordination module.

By explicitly encoding relational spatial dependencies, the proposed GA-GAT enables scalable and adaptive assignment decisions under dynamically evolving conditions. Extensive simulation results demonstrate that the proposed method consistently outperforms conventional heuristic and flat MARL baselines in terms of effectiveness retention and overall task success rate. Furthermore, ablation studies verify that explicit relational inductive bias is essential for robust coordination in multi-agent systems.

The results indicate that decoupling low-level control from high-level coordination provides a scalable and computationally efficient paradigm for complex $M\times N$ interaction problems. The proposed framework is not limited to aerial systems and can be extended to broader multi-agent applications, including swarm robotics, distributed sensing, emergency response, and autonomous traffic management.

In the current study, we adopt an idealized setting with homogeneous agents and simplified communication assumptions in order to isolate the core decision-making and coordination capabilities of the proposed framework. While demonstrating significant theoretical advantages in simulation, real-world deployment of the GA-GAT-PPO framework faces several sim-to-real challenges. First, communication delays and packet losses can result in stale state observations, potentially degrading the synchronization of the graph-based policy. Second, the KIZ formulation relies on precise kinematics; real-world sensor noise could cause fluctuating attention masks and assignment chattering. Finally, the Transformer module imposes non-trivial computational overhead during real-time inference, challenging the power and thermal budgets of UAV edge-computing platforms. Acknowledging these hardware and environmental constraints provides a clear trajectory for our subsequent research.

Future work will focus on incorporating communication constraints, extending the framework to heterogeneous agents, and validating the approach in more realistic environments.