Multi-Agent Deep Reinforcement Learning (MADRL)-Based End-to-End Formation Control for UAV Swarm with Dynamic Topology

Abstract

While MADRL has demonstrated significant potential in Unmanned Aerial Vehicle (UAV) swarm control, traditional architectures often rely on fixed-dimensional observation spaces. This rigid structural constraint severely limits the swarm’s adaptability in dynamic environments, particularly when facing sudden topological changes such as node failures or dynamic reinforcements. To overcome these limitations, this paper proposes an end-to-end UAV swarm motion control framework incorporating a state-modulated Graph Attention Network (GAT). By modeling the swarm as a dynamic interaction graph, the proposed method dynamically aggregates neighbor features using attention weights modulated by the agents’ real-time kinematic states. Furthermore, a virtual structure combined with an auction mechanism is introduced to achieve precise formation planning and target allocation. Evaluated in the Genesis 3D physics engine, the proposed Prioritized Experience Replay (PER)-MADDPG-Graph Attention Network (GAT) algorithm exhibits superior robustness and spatial adaptability. Extensive experiments, including dynamic node reduction and addition scenarios, confirm that the proposed framework seamlessly maintains swarm configurations without catastrophic policy degradation, outperforming baseline MADRL methods in both convergence speed and control precision.

1. Introduction

In recent years, UAV swarms have been increasingly deployed in complex missions [1], such as area surveillance [2,3], search [4] and rescue [5], and distributed sensor networks. Compared to single-agent systems, UAV swarms [6,7] offer improved fault tolerance and broader spatial coverage. However, effective cooperative control remains a significant challenge, as the swarm must simultaneously navigate toward target destinations, avoid collisions, and maintain specific spatial configurations in highly dynamic environments.

To address the complex coordination tasks of UAV swarms, numerous motion control strategies have been proposed in recent literature. Traditional control paradigms, such as robust control [8] based on semidefinite programming, sliding mode control [9], and behavior-based algorithms [10], have established foundational frameworks for maintaining swarm formations. To further refine performance, two notable advances have been proposed. First, integrating Control Barrier Functions ensures safety-critical obstacle avoidance [11]. Second, advanced path-following strategies like decoupled nonuniform guiding vector fields improve coordination accuracy on complex 3D manifolds by decoupling convergence and coordination terms [11]. Furthermore, integrating Control Barrier Functions with disturbance observers ensures rigorous, safety-critical obstacle avoidance and prescribed tracking performance for UAVs [12]. However, these model-based methods typically rely on explicit mathematical models and fixed environmental assumptions, severely limiting their adaptability when confronted with highly dynamic environments or unexpected topological disruptions [13].

Consequently, MADRL has emerged as a practical paradigm for end-to-end autonomous control [14,15]. Algorithms such as the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) allow agents to learn cooperative strategies—such as obstacle avoidance, target encirclement, and trajectory planning—through continuous trial-and-error interactions. By mapping environmental observations directly to continuous control commands, MADRL circumvents the need for explicitly programmed heuristic rules [16,17]. Furthermore, to address the scalability bottlenecks in multi-agent interactions, recent studies have begun integrating Graph Neural Networks with reinforcement learning, treating the swarm as a dynamic graph to efficiently aggregate spatial features among neighbors [18]. Beyond spatial coordination, UAV swarms increasingly function as complex nonlinear networked systems within industrial and surveillance applications. In these settings, addressing subsystem heterogeneity and data privacy under external disturbances has become a critical frontier. This has prompted the development of advanced federated learning and offset-free distributed control frameworks [19].

Despite these significant theoretical advancements, applying conventional MADRL and graph-based extensions to realistic 3D UAV swarms still presents several critical limitations. First, traditional Actor–Critic architectures require fixed-dimensional observation spaces [20], restricting agents to a pre-defined number of neighbors. Although recent hybridized frameworks utilize Graph Neural Networks [21] to handle variable topologies, their attention mechanisms predominantly rely on static spatial proximity. They lack the explicit integration of the UAV’s dynamic kinematic intents (i.e., continuous linear and angular velocities). When swarms undergo highly dynamic flight or sudden reconfigurations, this purely distance-based perception causes topological lag, frequently leading to catastrophic inter-agent collisions. Second, existing on-policy MARL algorithms suffer from severe sample inefficiency in complex continuous control tasks. Consequently, many studies validate their algorithms using simplified dataset-driven or 2D point-mass kinematics [22,23], which fail to capture the 3D underactuated and strongly coupled dynamics of actual quadrotors. Finally, most existing MADRL approaches utilize purely reactive reward shaping. While this encourages basic aggregation, it often results in loose, unpredictable swarm morphologies, making it difficult to enforce strict geometric formations required for specific tactical tasks.

To address these limitations, this paper proposes an off-policy, sample-efficient end-to-end 3D UAV swarm motion control framework incorporating a state-modulated GAT [24] and PER [25]. By modeling the swarm as a dynamic interaction graph, the GAT module processes variable-length neighbor information, freeing the network from fixed-topology constraints. Unlike conventional graph-based methods that rely purely on spatial distance, our attention weights are dynamically modulated by the agents’ real-time kinematic states. This structural contribution establishes a physical kinematics-driven semantic topology, ensuring rapid proactive collision avoidance even under sudden swarm reconfigurations. Furthermore, the integration of PER significantly mitigates the sample inefficiency bottleneck in 3D environments. To achieve strict geometric control, a virtual structure [26] method combined with a decentralized auction mechanism is introduced for formation planning and target allocation.

The main contributions of this paper are summarized as follows:

• 3D End-to-End MDP Formulation: We construct a comprehensive Markov Decision Process (MDP) for 3D quadrotor control. A composite reward function is designed, incorporating attitude constraints, velocity penalties, and collision avoidance, bridging the gap between high-level navigation decisions and low-level physical actuation.
• State-Modulated GAT for Dynamic Topologies: We introduce a GAT module that dynamically aggregates neighbor features based on kinematic intent vectors. This design inherently accommodates variable neighbor counts, enabling the swarm to seamlessly adapt to sudden topological changes (e.g., node failures or additions) without requiring retraining.
• Auction-Based Formation Planning: We integrate a virtual structure approach with a decentralized auction mechanism to translate macroscopic swarm topologies into explicit relative-position tracking tasks, effectively minimizing the total maneuvering cost during formation reconfiguration.
• 3D Physical Validation: The proposed PER-MADDPG-GAT algorithm is evaluated within the Genesis 3D physics engine. Extensive ablation studies confirm that the framework maintains high control precision and robustness under dynamic node reduction and reinforcement scenarios, significantly outperforming baseline MADRL methods.

The remainder of this paper is organized as follows. Section 2 details the proposed MADDPG-based UAV swarm control algorithm, including the MDP modeling, the state-modulated GAT module, and the formation planning mechanism. Section 3 presents the experimental settings, comparative results, and ablation studies in the 3D simulation environment. Finally, Section 4 concludes the paper.

2. GAT-Based UAV Swarm Control Algorithm

2.1. MDP Modeling for 3D End-to-End Swarm Control

Based on the core principles of reinforcement learning (RL), the highly dynamic 3D swarm coordination problem must be formally formulated into a Markov Decision Process (MDP) model to achieve end-to-end control of the UAV swarm [27]. It should be clarified that in this specific context, “end-to-end” refers to the direct policy mapping from local observation states to continuous kinematic commands, effectively bypassing traditional explicit path planning and trajectory optimization modules. Meanwhile, the underlying attitude and motor executions are still governed by pre-defined low-level controllers for fundamental flight stability. As shown in Figure 1, this section elaborates on the formal definitions from three dimensions: the high-dimensional state space, the low-level action space, and the composite reward function with physical constraints. Note that Figure 1 was conceptually drafted by the authors; AI (gemini-3.1-pro) was employed solely for non-substantive visual polishing—including color palette, shading, and aesthetic refinement—with all elements rigorously verified against the original draft to ensure scientific integrity.

State Space: In a multi-UAV swarm, the state space serves as the sole interface for the neural network to perceive physical world information. The rationality and completeness of its design directly determine the upper bound of algorithm convergence and control stability. Assuming there are N UAVs in the environment, to ensure that the agent grasps precise kinematic boundaries and avoids dangerous maneuvers that violate physical characteristics (such as extreme roll/pitch overloads), a single UAV i must first obtain the observation of its own ego-attitude ${\mathbf{\Theta }}_i={[{\varphi }_i,{\theta }_i,{\psi }_i]}^T$, representing the roll, pitch, and yaw angles, respectively.

To depict the absolute motion state of the UAV in 3D space, the network must be fed the UAV’s position vector ${\mathbf{p}}_i={[x_i,y_i,z_i]}^T$, linear velocity vector ${\mathbf{v}}_i={[v_{x_i},v_{y_i},v_{z_i}]}^T$, and angular velocity vector ${\mathit{\omega }}_i={[{\omega }_{x_i},{\omega }_{y_i},{\omega }_{z_i}]}^T$ in the global coordinate system. The aforementioned physical quantities collectively constitute the agent’s self-state observation component ${\mathbf{o}}_i^{\mathrm{self}}={[{\mathbf{\Theta }}_i^T,{\mathbf{p}}_i^T,{\mathbf{v}}_i^T,{\mathit{\omega }}_i^T]}^T$.

To accomplish cooperative swarm tasks, relying solely on isolated individual information is far from sufficient to maintain the swarm formation. Adequate relative motion information of neighbors must be provided to enhance the UAV’s perception of the swarm’s spatial topology. Constrained by the communication range and bandwidth limits of real physical systems, a local topology judgment mechanism is introduced: setting a maximum communication radius $d_c$, state resolution is only performed for nodes within this range. In addition, the selection of the K nearest neighbors is analytically determined by balancing the geometric requirements of 3D collision avoidance with the communication bandwidth constraints of physical UAVs. From a spatial perspective, tracking the three closest neighbors effectively covers the immediate interaction neighborhood required for reliable collision evasion in a 3D partition. Furthermore, from a topological learning perspective, blindly increasing K not only escalates the communication payload but also introduces “over-smoothing” risks inherent to graph neural networks, which blurs the distinct local kinematic intents of the agents and degrades the responsiveness of the swarm. Under this rule, agent i acquires the relative position vector ${\mathbf{p}}_{ij}={\mathbf{p}}_j-{\mathbf{p}}_i$ and the relative velocity vector ${\mathbf{v}}_{ij}={\mathbf{v}}_j-{\mathbf{v}}_i$ of neighbor j. Simultaneously, to guide the swarm in completing navigation tasks, the relative distance vector ${\mathbf{p}}_{ig}={\mathbf{p}}_g-{\mathbf{p}}_i$ between the agent and the task target point (or the absolute hovering point in the desired swarm configuration) is introduced. The above interaction and target guidance features are combined to form the environmental observation component ${\mathbf{o}}_i^{\mathrm{env}}$.

To accelerate the convergence of the neural network, all observation vectors undergo scaling and normalization. The final, complete state space of UAV i is defined as ${\mathbf{o}}_i={[{\mathbf{o}}_i^{\mathrm{self}},{\mathbf{o}}_i^{\mathrm{env}}]}^T$.

Action Space: A quadrotor UAV is a highly underactuated and strongly coupled non-linear system. To discard traditional attitude resolution controllers and achieve a direct mapping from network commands to real physical torques, the action space design must balance control precision with dynamic boundaries. The system adopts a continuous action space, where the output of the Actor network is designed as a 4D normalized vector ${\mathbf{u}}_i={[u_i^1,u_i^2,u_i^3,u_i^4]}^T$, with each component $u_i^k\in [-1,1]$.

In the Genesis engine, directly outputting unconstrained rotational speeds easily leads to rotor stall or overspeed crashes. Assuming the baseline motor rotational speed of the quadrotor in a fully loaded hover state is ${\mathsf{\Omega }}_{\mathrm{hover}}$, this paper constructs a relative control mapping mechanism based on this baseline speed. The outputs of the Actor network are used to perform differential adjustments on the low-level propellers:

$${\mathsf{\Omega }}_i^k=(1+c·u_i^k){\mathsf{\Omega }}_{\mathrm{hover}}$$

(1)

where c is the physical scaling coefficient for actions, set to $c=0.8$ in this study. Under this hard constraint, the true physical rotational speed of a single motor is strictly confined within the closed interval $[0.2{\mathsf{\Omega }}_{\mathrm{hover}},1.8{\mathsf{\Omega }}_{\mathrm{hover}}]$, ensuring that system control commands always evolve smoothly around the steady-state hovering condition.

Composite Reward Function: In quadrotor UAV swarm control, the attitude stability of the UAV is of paramount importance. During the initial learning phase, a lack of reasonable attitude constraints can cause the UAV to tumble or lose control, violating basic physics. Therefore, attitude stability terms are designed to guide the UAV toward smooth and controllable flight. First, excessive roll and pitch angles are penalized to suppress severe attitude inclination:

$$r_i^{\mathrm{att}}=-\lambda \mathrm{att}\left(\left|{\varphi }_i\right|+\left|{\theta }_i\right|\right)$$

(2)

where $\lambda \mathrm{att}>0$ is the weight coefficient that adjusts the impact of this term within the overall reward space, ensuring the UAV maintains a level attitude during flight and improving stability.

In addition to attitude angle deviations, excessive angular velocity also leads to flight instability. Particularly during swarm flight and cooperative maneuvers, violent attitude changes can disrupt the formation structure and increase the risk of collision. Thus, a penalty term based on the magnitude of the angular velocity is introduced to suppress rapid spinning behaviors:

$$r_i^{\mathrm{ang}}=-{\lambda }_{\omega }\left|{\mathit{\omega }}_i\right|$$

(3)

where $\lambda \omega >0$ is the angular velocity penalty weight. Restricting angular velocity effectively reduces attitude oscillations, guiding the UAV toward smoother flight.

During swarm flight tasks, the yaw stability of the UAV is crucial for maintaining heading and coordinating maneuvers. Therefore, an exponential reward function based on yaw deviation is introduced. Let ${\psi }_i$ be the yaw angle of the i-th UAV. After normalizing it to $(-\pi ,\pi ]$, the yaw stability reward is defined as:

$$r_i^{\mathrm{yaw}}=exp(-{\lambda }_{\psi }\left|{\psi }_i\right|)$$

(4)

where ${\lambda }_{\psi }>0$ is the decay coefficient. This reward assumes a larger value when the yaw angle aligns with the target direction and decays rapidly as the deviation increases, guiding the UAV to maintain a stable heading. Combining these three terms, the attitude stability reward function formulated in this paper is expressed as:

$$r_i^{\mathrm{stab}}=r_i^{\mathrm{att}}+r_i^{\mathrm{ang}}+r_i^{\mathrm{yaw}}$$

(5)

For UAVs to form a cohesive swarm, stable flight attitude is the fundamental prerequisite. During the pre-defined swarm formation process, each UAV must gradually navigate to its designated position within the formation. Therefore, a reward function is designed based on the relative distance between the UAV’s position and the target point:

$$r_i^{\mathrm{step}}={\lambda }_{\mathrm{step}}\left(\left|{\mathbf{p}}_i^t-{\mathbf{p}}_g^t\right|-\left|{\mathbf{p}}_i^{t+1}-{\mathbf{p}}_g^{t+1}\right|\right)$$

(6)

where $\lambda \mathrm{step}>0$. Using the difference between consecutive frames serves as a dense feedback signal, allowing the UAV to explicitly perceive whether each step brings it closer to the target. Compared to a sparse reward granted only upon reaching the destination, this approach provides stronger guidance. However, while this design encourages general movement toward the target, achieving precise arrival and steady hovering requires an additional, more direct penalty:

$$r_i^{\mathrm{goal}}=-{\lambda }_{\mathrm{goal}}\left|{\mathbf{p}}_i-{\mathbf{p}}_g\right|$$

(7)

This term penalizes the absolute distance between the UAV and the target, reaching its maximum (zero) when the UAV arrives. By combining these two navigation rewards, the UAV exhibits superior performance in precise navigation sub-tasks.

In practical scenarios, as the number of UAVs increases, collision avoidance during swarm formation becomes critical. The UAVs must learn an autonomous obstacle avoidance policy, for which the following collision penalty function is designed (Equation (8)):

$$r_i^{\mathrm{collision}}=\left\{\begin{array}{cc}-k,\hfill & d<d_{\min }\hfill \\ -k{\left({\displaystyle \frac{d-d_{\mathrm{avoid}}}{d_{\mathrm{avoid}}}}\right)}^{0.2},\hfill & d_{\min }\le d<d_{\mathrm{avoid}}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

where k is a positive penalty coefficient, and d represents the distance between the UAV and its nearest neighbor. When d is less than $d_{\mathrm{avoid}}$, the UAV perceives the need to evade; when d falls below $d_{\min }$, indicating that a collision has occurred, the UAV receives the maximum penalty.

In summary, the total reward space of the UAV comprises three components: attitude stability, swarm navigation, and collision avoidance, as expressed in Equation (9):

$$r_i=r_i^{\mathrm{stab}}+(r_i^{\mathrm{step}}+r_i^{\mathrm{goal}})+r_i^{\mathrm{collision}}$$

(9)

2.2. State-Modulated Graph Attention Mechanism

Traditional MADDPG algorithms concatenate neighbor observations into a fixed-length vector, which fails when the neighbor count is time-varying. To resolve this, the observation space of the i-th UAV is first decoupled as ${\mathbf{o}}_i={[{\mathbf{o}}_i^{\mathrm{self}},\Delta {\mathbf{P}}_i]}^T$. Here, ${\mathbf{o}}_i^{\mathrm{self}}$ contains the ego-states (velocity, attitude, angular velocity), and $\Delta {\mathbf{P}}_i={\{\Delta {\mathbf{p}}_{ij}\}}_{j=1}^N$ denotes the relative positions of all neighbors within the sensing range. The relative position $\Delta {\mathbf{p}}_{ij}$ is defined as:

$$\Delta {\mathbf{p}}_{ij}=\left\{\begin{array}{cc}{\mathbf{p}}_j-{\mathbf{p}}_i,\hfill & j\ne i\hfill \\ 0,\hfill & j=i\hfill \end{array}\right.$$

To prevent the curse of dimensionality and reflect the varying importance of spatial information under different motion patterns, a state-modulated intent vector is designed. The critical kinematic state ${\mathbf{s}}_i$ is defined as:

$${\mathbf{s}}_i={[{\mathbf{v}}_i^T,{\mathit{\omega }}_i^T]}^T$$

where ${\mathbf{v}}_i$ and ${\mathit{\omega }}_i$ represent the linear and angular velocities, respectively. This state is linearly mapped to an intent vector ${\mathbf{q}}_i$ via a parameter matrix ${\mathbf{W}}_q$:

$${\mathbf{q}}_i={\mathbf{W}}_q{\mathbf{s}}_i$$

This mapping explicitly endows the agent with dynamic attention preferences. For instance, during high-speed cruising, forward neighbors receive higher attention, whereas during aggressive turning maneuvers, lateral awareness is prioritized. To process the neighbor information, the raw relative position $\Delta {\mathbf{p}}_{ij}$ is linearly projected into a latent Key-Value space via learnable parameter matrices ${\mathbf{W}}_k$ and ${\mathbf{W}}_v$. Specifically, the spatial descriptor ${\mathbf{k}}_{ij}$ and the interaction feature ${\mathbf{v}}_{ij}$ are formally defined as:

$${\mathbf{k}}_{ij}={\mathbf{W}}_k\Delta {\mathbf{p}}_{ij}$$

$${\mathbf{v}}_{ij}={\mathbf{W}}_v\Delta {\mathbf{p}}_{ij}$$

Here, ${\mathbf{k}}_{ij}$ encapsulates the geometric relationship of neighbor j, acting as an identity tag for matching, while ${\mathbf{v}}_{ij}$ embeds the actual spatial context to be aggregated. Using the intent vector ${\mathbf{q}}_i$ and the spatial descriptor ${\mathbf{k}}_{ij}$, the raw attention score is calculated via a scaled dot-product model to mitigate vanishing gradients:

$$e_{ij}={\displaystyle \frac{{\mathbf{q}}_i^T{\mathbf{k}}_{ij}}{\sqrt{d_k}}}$$

A Softmax function normalizes these scores to obtain the final attention weight $a_{ij}$:

$$a_{ij}={\displaystyle \frac{exp\left(e_{ij}\right)}{{\sum }_{k=1}^{N}exp\left(e_{ik}\right)}}$$

Finally, the interaction features ${\mathbf{v}}_{ij}$ are aggregated using these weights to generate a fixed-dimensional spatial context representation ${\mathbf{h}}_i^{\mathrm{attn}}$ through a non-linear activation function $f_{\mathrm{attn}}$:

$${\mathbf{z}}_i={\displaystyle \sum _{j=1}^N}{a}_{ij}{\mathbf{v}}_{ij}$$

$${\mathbf{h}}_i^{\mathrm{attn}}=f_{\mathrm{attn}}\left({\mathbf{z}}_i\right)$$

Unlike traditional graph-based MARL models that require exhaustive tuning for hyperparameters, the proposed state-modulated GAT is naturally robust. By mapping the intent query ${\mathfrak{q}}_i$ directly to the UAV’s real-time kinematic state $s_i$, the attention weights $a_{ij}$ are driven by actual flight mechanics rather than abstract hidden variables. Consequently, the parameter matrices ($W_q,W_k,W_v$) simply align spatial geometry with the drone’s movement intent, making the network much less sensitive to parameter shifts. Additionally, using a lightweight, single-head attention structure prevents the model from overfitting to fixed swarm formations.

2.3. Network Architecture and Algorithm Workflow

The proposed GAT–Actor network aims to fuse ego-perception with spatial relationship modeling. The network processes the raw observation by extracting the aforementioned ${\mathbf{h}}_i^{\mathrm{attn}}$ using the state-modulated GAT module. This effectively compresses the variable-length relative position matrix $\Delta {\mathbf{P}}_i$ into a fixed-length feature, drastically reducing the training complexity in dynamic topologies. Specifically, the Actor is parameterized by a Multi-Layer Perceptron (MLP) with two hidden layers of 256 neurons.

Similarly, the centralized Critic network is enhanced to process the joint feature states and actions of all agents, providing a stable, low-variance value gradient for Actor updates. The Critic also adopts an MLP architecture with 256 hidden units. Both networks are trained using the Adam optimizer. Coupled with the PER mechanism—which assigns higher sampling probabilities to transitions with high Temporal-Difference (TD) errors—the proposed PER-MADDPG-GAT algorithm forces the network to focus on critical interaction scenarios, significantly boosting convergence stability.

For real-world UAV applications, the proposed decentralized framework offers three practical advantages: The computational complexity of the local GAT module is $\mathcal{O}(K·d_f)$, where K is the neighbor count and $d_f$ is the feature dimension. With a fixed $K=3$, the complexity reduces to $\mathcal{O}\left(d_f\right)$, making the onboard computation independent of the total swarm size N and ensuring high scalability. Each UAV only broadcasts its low-dimensional kinematic states (position and velocity) to nearby neighbors. This avoids global state broadcasting, keeping the communication payload minimal and strictly within the limits of standard embedded wireless modules. The actor network utilizes a single-layer attention mechanism and a shallow MLP. This lightweight architecture requires minimal floating-point operations, enabling low-latency forward inference that satisfies the high-frequency execution requirements of embedded flight controllers.

2.4. Swarm Formation Planning and Node Allocation

While the GAT module provides dynamic perceptual capabilities, purely reactive rewards fail to enforce strict geometric formations. Therefore, a virtual structure approach is adopted to model the high-level objectives. Assuming the geometric center of the swarm is the origin of the reference frame, the predefined topology is represented by a set of desired relative node positions:

$$\mathcal{T}=\left\{{\mathbf{p}}_k^{\mathrm{ref}}\in {\mathbb{R}}^3\mid k=1,2,\dots ,N\right\}$$

where ${\mathbf{p}}_k^{\mathrm{ref}}$ is the expected position vector of the k-th virtual node relative to the swarm center.To match physical UAVs with these virtual nodes optimally, a decentralized auction mechanism is employed. The bidding cost $c_{ik}$ for UAV i to claim node k is defined by the Euclidean distance:

$$c_{ik}=∥{\mathbf{p}}_i-{\mathbf{p}}_k^{\mathrm{ref}}∥$$

Based on this cost matrix, a linear assignment algorithm efficiently minimizes the total maneuvering cost during topology reconfiguration. This mechanism, seamlessly integrated with the UAV-based control policy, yields a highly robust cooperative control solution for dynamic environments.

3. Experimental Validation and Results Analysis

3.1. Experimental Parameter Settings

To ensure the simulation accurately reflects the dynamic characteristics of multi-UAV cooperative tasks in the real world, a 3D flight environment is constructed using the Genesis physics engine. This platform provides rigid-body dynamics for the quadrotors. The physical attributes of the UAVs and the swarm parameters for the cooperative tasks are rigorously defined.

Table 1. Simulation Environment Parameters and Reward Function Weights.

Parameter Name	Symbol	Value
Initial Number of UAVs	N	5
Desired Swarm Radius	R	0.3 m
Target Flight Altitude	$h_{\mathrm{target}}$	1.0 m
Simulation Frequency	f	100 Hz
Max Steps per Episode	$T_{max}$	500
Total Training Episodes	$Episodes$	30,000
Batch Size	$N_{\mathrm{batch}}$	256
Map Dimensions	–	$6\mathrm{m}\times 6\mathrm{m}\times 3\mathrm{m}$
Target Tracking Reward	$w_{\mathrm{dist}}$	3.0
Proximity Guidance Reward	$w_{\mathrm{prox}}$	300.0
Altitude Maintenance Reward	$w_h$	1.0
Velocity Tracking Reward	$w_v$	1.0
Attitude Stability Reward	$w_{\mathrm{pose}}$	10.0
Action Smoothness Penalty	$w_{\mathrm{act}}$	0.02
Actor Learning rate	${\eta }_a$	$1\times {10}^{-4}$
Critic Learning rate	${\eta }_a$	$1\times {10}^{-3}$
Replay Buffer Capacity	$N_{buffer}$	$1\times {10}^6$
PER prioritization exponent	${\alpha }_{PER}$	0.6
PER importance sampling exponent	${\beta }_{PER}$	0.4

details the specific physical parameters, task settings, and reward function weights, which form the baseline for evaluating the effectiveness and stability of the control policies.

The coefficients of the joint reward function were empirically determined based on strict hierarchical priorities defined by actual UAV flight requirements. First, the quadcopter must maintain a stable attitude and fly in a manner consistent with the laws of physics. Second, the UAV must avoid collisions in space, which aligns with real-world mission requirements. Finally, there is a coefficient for mission completion, which serves to gradually guide the UAV’s movement toward achieving the mission objective. During the training process described in this paper, although the convergence rate of the DRL policy exhibits moderate sensitivity to the absolute values of these weights, the final flight performance remains highly robust as long as this relative order of priority is strictly maintained. Conversely, if the weights of the reward coefficients are adjusted arbitrarily, the first challenge encountered is that training becomes ineffective; the drone will be unable to achieve normal flight capabilities even after tens of thousands of training iterations.

3.2. Experimental Results and Analysis

This section validates the performance of the multi-UAV swarm algorithm integrated with the GAT module within the Genesis environment. Initially, four baseline and proposed algorithms are trained under identical task scenarios to form specific swarm configurations. Subsequently, utilizing the converged policies of these four algorithms, disturbance rejection tests are conducted in two dynamic topology scenarios to rigorously verify the effectiveness of the GAT module in handling dynamic interactive environments.

To evaluate the learning efficiency and final policy quality of each algorithm, Figure 2 presents the average reward curves over 30,000 training episodes for MADDPG, PER-MADDPG, MADDPG-GAT, and the proposed PER-MADDPG-GAT. The solid lines represent the mean values across 5 independent trials utilizing different random seeds, while the shaded regions denote the standard deviations, reflecting the volatility and stability of the training process. Observing the overall trends, all four algorithms experience a brief decline in rewards during the initial exploration phase, followed by a gradual ascent and eventual convergence as the policies are optimized. However, significant disparities emerge in convergence speed and final steady-state rewards: the original MADDPG algorithm exhibits the lowest learning efficiency. Its reward recovery is sluggish, remaining at a relatively low level throughout the training cycle. This indicates that relying solely on basic multi-agent gradient policies is insufficient to rapidly find optimal solutions when dealing with high-dimensional state spaces and complex cooperative tasks. A comparison between MADDPG and MADDPG-GAT reveals that the introduction of the graph attention mechanism plays a decisive role in elevating the policy’s performance ceiling. The slope of the MADDPG-GAT curve is significantly steeper than that of the baseline algorithm, and its final converged reward is substantially higher. This demonstrates that by dynamically extracting key features of neighbor nodes, the GAT enhances the agents’ accuracy in perceiving the environmental topology, enabling them to learn more efficient cooperative strategies. The PER-MADDPG-GAT algorithm proposed in this paper combines the advantages of both mechanisms, exhibiting optimal comprehensive performance. After the initial exploration, this algorithm ascends with the steepest gradient, reaching a high-score region unmatched by the others at approximately 10,000 episodes, and consistently maintains the highest average reward level. The exceedingly narrow shaded region in the later stages of training indicates that PER-MADDPG-GAT maintains stable convergence across different random seeds, showcasing exceptional robustness. Furthermore, to rigorously validate this performance enhancement, a Welch’s t-test was conducted on the converged average rewards. The statistical analysis yielded a p-value of $p<0.05$ when comparing the proposed PER-MADDPG-GAT against the baseline algorithms, providing strong statistical evidence that the improvement in policy quality is indeed significant. Upon the convergence of the four algorithms, two sets of cooperative navigation experiments are established to verify their configuration maintenance capabilities and navigation precision. After the UAVs are randomly initialized in the 3D space, the policy must output low-level actions based on current states to rapidly form a stable, predefined geometric configuration. Subsequently, the system relocates the target points in the environment to guide the swarm in executing simple straight-line and complex rectangular trajectory navigation, respectively. The stability of the policies is verified by observing the swarm’s motion trajectories and configuration retention. Figure 3 displays two motion trajectories of the PER-MADDPG-GAT algorithm during the testing phase. It is evident that after spawning from the origin, the UAVs rapidly adjust their attitudes and positions to form a swarm at the yellow star marker in Figure 3a. They then navigate toward the target point to complete the mission. Throughout this process, the trajectory remains remarkably smooth without noticeable formation dispersion or collisions, proving the algorithm’s stability in foundational navigation tasks. In Figure 3b, four continuous target points are set. The results clearly show that even when the swarm is required to execute 90-degree turns, the policy successfully maintains the geometric shape during target navigation.

To further quantitatively evaluate the performance, we compare the swarm errors of the four algorithms across these two tasks. The swarm error E is defined in Equation (10):

$$E={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left|{\left|{\mathbf{p}}_i-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{\mathbf{p}}_j\right|}_2-R\right|$$

(10)

where N is the total number of UAVs, R is the predefined expected radius of the swarm, and ${\mathbf{p}}_i$ is the position of the i-th UAV. As shown in Figure 4, PER-MADDPG-GAT consistently maintains the lowest and most stable error among the four algorithms. This result not only verifies the baselines’ capabilities to complete basic tasks but also highlights the precision advantage of the proposed method, providing a benchmark for evaluating performance under more complex conditions.

The ablation studies in this section primarily aim to verify the adaptability and stability of the GAT module when the number of UAVs and the topological structure change. Two scenarios involving dynamic quantity variations are designed: UAV failure (node reduction) and dynamic UAV reinforcement (node addition). The four aforementioned algorithms are assigned to execute these tasks to validate the GAT module’s superiority in processing UAV interaction relationships in complex environments. First, to verify the robustness and adaptability of the PER-MADDPG-GAT algorithm in extreme scenarios such as sudden hardware failures or communication disruptions, an ablation experiment with dynamically reducing UAV numbers is designed. The scenario is initialized with a 5-UAV swarm executing a straight-line navigation task. At step 200 of the simulation, a failure mechanism is artificially introduced, forcing one UAV to abruptly exit the swarm. At this juncture, the remaining 4 UAVs must perceive the topological change without human intervention, rapidly reconfigure from the original pentagonal formation into a square formation, and sustain this new geometry to complete the remaining navigation task.

The test results are shown in Figure 5. The UAV swarm initially gathers and maintains a stable pentagonal formation. Upon reaching the failure trigger point, the compromised UAV detaches. The remaining four UAVs respond swiftly to the topological shift, adjusting their relative positions to autonomously reconstruct the formation from a pentagon into a square. Post-reconfiguration, they continue to maintain the swarm shape and fly steadily toward the target point. The overall trajectory is smooth, exhibiting no system crashes or route divergence due to node loss, directly proving the proposed algorithm’s adaptive capability in handling dynamic node dropouts. To visually demonstrate this topological reconfiguration, Figure 6 presents snapshots of the experimental scene in the Genesis environment before and after the dynamic reduction.

To quantitatively analyze the GAT module’s contribution in such dynamic scenarios, Figure 7 compares the system error variations of the four algorithms before and after the failure. Initially, all algorithms successfully guide the UAVs to form a stable swarm. At step 200, when one UAV fails, the geometric center of the swarm abruptly shifts, necessitating a recalculation of the configuration allocation; consequently, the swarm errors for all algorithms spike. For the two algorithms lacking the GAT module, MADDPG (red curve) oscillates violently and fails to converge, indicating that the UAVs cannot maintain stable flight in the Genesis environment and the entire swarm enters a state of persistent oscillation. PER-MADDPG (light blue curve) converges, but its swarm error does not decrease, implying that the swarm fails to reform into a square geometry post-failure. These two algorithms demonstrate that a fixed-dimensional state space cannot accommodate variations in input dimensions, inevitably leading to swarm destabilization.

Conversely, algorithms incorporating the GAT module exhibit significant advantages. Specifically, PER-MADDPG-GAT, after a brief perturbation, sees its error curve drop rapidly and stabilize near zero. This indicates that the GAT module effectively compensates for the information vacuum caused by the missing node by dynamically adjusting the attention weights of the remaining neighbors, enabling the swarm to quickly adapt to the new topology (a 4-UAV swarm), thereby verifying the mechanism’s exceptional robustness in dynamic topological environments. To further validate the algorithm’s adaptability in swarm scaling and dynamic task allocation, an ablation experiment involving dynamic UAV reinforcement is designed. The initial scenario features 5 UAVs executing a straight-line navigation task. At step 200, a new reinforcement UAV is introduced into the system. The swarm must perceive the integration of the new node, reallocate target points, and autonomously expand and reconfigure from the initial pentagonal topology to a hexagonal one without human intervention, subsequently flying to the destination while maintaining the new shape.

Figure 8 illustrates the 3D trajectory of the PER-MADDPG-GAT algorithm during this dynamic reinforcement task. Initially, the 5 UAVs rapidly assemble into a stable pentagon (marked by the yellow star). At step 200, as the reinforcement UAV cuts in from the flank, the swarm reacts promptly to this perturbation; the original UAVs proactively adjust their positions to yield space for the new node. As indicated by the blue hexagram, the swarm successfully accommodates the reinforcement, smoothly completing the topological transition from a pentagon to a hexagon. Post-reconfiguration, the 6 UAVs maintain a tight hexagonal formation until reaching the destination (black star). The entire reinforcement process features a smooth trajectory with no collisions or drastic evasive maneuvers between the old and new nodes, proving the algorithm’s stability when handling dynamic node integration. Figure 9 similarly displays the swarm variations before and after the reinforcement’s arrival in the Genesis environment.

Figure 10 further quantifies the performance disparities among the four algorithms when confronting swarm scaling. At the exact moment the reinforcement UAV joins at step 200 (indicated by the dashed line), the relative relationships between UAVs alter. MADDPG and PER-MADDPG, lacking the graph attention mechanism, exhibit severe adaptability failures. As illustrated, the error for PER-MADDPG escalates sharply post-step 200 and shows a diverging trend. This demonstrates that a rigid network structure struggles to effectively process the interactive information of the added node, resulting in the complete collapse of the swarm configuration. In stark contrast, MADDPG-GAT and PER-MADDPG-GAT exhibit profound robustness. Benefiting from the inherent adaptability of graph neural networks to variable-length inputs, the UAV instantaneously assigns attention weights to the newly joined node. The experimental data shows that PER-MADDPG-GAT experiences only marginal error fluctuations at the moment of reinforcement integration before rapidly reconverging to a steady state.

4. Conclusions

This paper presented an end-to-end 3D motion control framework based on a state-modulated GAT for UAV swarms in dynamic topologies. To overcome the fixed-dimensionality constraints of traditional MARL, the proposed PER-MADDPG-GAT algorithm dynamically aggregates variable-length neighbor features using kinematic-dependent attention weights. Additionally, integrating a virtual structure with a decentralized auction mechanism ensures precise macroscopic formation control.

Simulations in the Genesis 3D physics engine confirm that our framework seamlessly adapts to sudden topological shifts, such as node failures or reinforcements, under strict rigid-body constraints. Compared to baseline methods, it significantly improves convergence stability, trajectory precision, and robustness. Future work will focus on addressing communication latency and deploying the framework on physical quadrotors.

While the proposed framework demonstrates clear advantages in adapting to dynamic topologies, the current empirical validation is bounded by small-scale swarm variations and comparisons with MADDPG-based architectures. To address these limitations, we plan to extend this work in three main directions. First, to overcome the computational bottleneck of simulating full 3D rigid-body dynamics for numerous agents, we will utilize distributed computing clusters to scale the experiments to larger swarms under severe communication noise. Second, we aim to benchmark our framework against more recent on-policy or sequence-modeling multi-agent algorithms, such as MAPPO and Transformer-based architectures. Alongside this, we will conduct granular ablation studies to isolate and quantify the specific performance contribution of the kinematic state-modulation mechanism versus standard distance-based attention. Finally, we will continue working toward bridging the sim-to-real gap by testing the control policy on physical quadrotors.