Joint Trajectory and Power Optimization for Loosely Coupled Tasks: A Decoupled-Critic MAPPO Approach

Highlights

What are the main findings?

• Proposed a novel MAPPO-DC algorithm for multi-UAV communication, introducing a hierarchical framework and a composite reward function to optimize resource allocation in loosely coupled tasks.
• Demonstrated MAPPO-DC’s enhanced performance in simulation, achieving improved communication efficiency, user fairness, and crash avoidance compared to IPPO and standard MAPPO.

What are the implications of the main findings?

• Offers a promising, simulation-validated algorithmic foundation for future real-world multi-UAV communication systems, promising improved network performance and task completion in complex environments.
• Provides insights into designing advanced MARL algorithms capable of managing credit assignment and balancing individual agent performance with global objectives in decentralized systems.

Abstract

Multi-unmanned aerial vehicle (UAV) systems are crucial for establishing resilient communication networks in disaster-stricken areas, but their limited energy and dynamic characteristics pose significant challenges for sustained and reliable service provision. Optimizing resource allocation in this situation is a complex sequential decision-making problem, which is naturally suitable for multi-agent reinforcement learning (MARL). However, the most advanced MARL methods (e.g., multi-agent proximal policy optimization (MAPPO)) often encounter difficulties in the “loosely coupled” multi-UAV environment due to their overly centralized evaluation mechanism, resulting in unclear credit assignment and inhibiting personalized optimization. To overcome this, we propose a novel hierarchical framework supported by MAPPO with decoupled critics (MAPPO-DC). Our framework employs an efficient clustering algorithm for user association in the upper layer, while MAPPO-DC is used in the lower layer to enable each UAV to learn customized trajectories and power control strategies. MAPPO-DC achieves a complex balance between global coordination and personalized exploration by redesigning the update rules of the critic network, allowing for precise and personalized credit assignment in a loosely coupled environment. In addition, we designed a composite reward function to guide the learning process towards the goal of proportional fairness. The simulation results show that our proposed MAPPO-DC outperforms existing baselines, including independent proximal policy optimization (IPPO) and standard MAPPO, in terms of communication performance and sample efficiency, validating the effectiveness of our tailored MARL architecture for the task. Through model robustness experiments, we have verified that our proposed MAPPO-DC still has certain advantages in strongly coupled environments.

1. Introduction

After sudden disasters such as earthquakes, floods, and building collapses, ground cellular networks often become paralyzed due to physical damage or power outages, forming “communication islands”. In this situation, rapidly deploying multiple unmanned aerial vehicles (UAVs) to form a temporary, self-organizing aerial communication network has become one of the most promising solutions [1,2,3,4].

UAVs have the advantages of flexible deployment and high maneuverability, making them very suitable for scenarios with sparse and unevenly distributed users [5]. The main challenge of UAV-assisted communication lies in its limited onboard energy, which limits flight endurance and signal transmission power. Therefore, it is crucial to effectively allocate communication resources among multiple UAVs [6,7]. Designing effective resource allocation strategies is crucial for providing sustainable and reliable services under strict energy constraints. However, in highly dynamic scenarios involving multiple UAVs and mobile users, solving resource allocation problems constitutes a complex sequential decision-making process. Traditional methods based on static optimization are not suitable for this task, which in turn highlights the enormous potential of learning-based adaptive strategies.

1.1. Related Works

The problem of resource allocation in UAV-assisted communication has been extensively studied in academia. These studies can be broadly categorized into methods based on traditional optimization and those based on learning methods. Our analysis begins with an overview of traditional optimization techniques, followed by the evolution and application of reinforcement learning paradigms in this domain.

1.1.1. Traditional Optimization Methods

Early research mainly relied on traditional optimization techniques to solve the resource allocation problem in multi-UAV communication networks. These works typically model resource allocation problems as precise mathematical formulas, utilizing methods such as convex optimization and successive convex approximation (SCA) to optimize UAV trajectories and transmission power under idealized channel models [4,8]. Other methods include using block coordinate descent (BCD) to decouple complex variables [8], or applying game theory to simulate interactions between multiple UAVs and users [9].

Although traditional optimization methods can theoretically achieve optimal or near-optimal solutions under static conditions with perfect information, they face significant limitations in practical multi-UAV communication scenarios serving multiple users as follows:

• Computational complexity and real-time decision-making: Traditional methods in UAV trajectory and resource allocation typically transform optimization problems into finding optimal solutions for non-convex problems, which often require complex mathematical designs (e.g., iterative SCA or BCD methods) and are computationally intensive. This makes it impossible for them to make real-time decisions in highly dynamic environments where UAVs and users are constantly moving [10].
• Prior information dependency and model robustness: The performance of traditional optimization methods largely relies on accurate and real-time system state information (e.g., precise channel models, user locations, interference levels) [4]. However, in highly dynamic UAV communication networks, it is difficult to obtain accurate information in real time, which makes traditional optimization methods susceptible to environmental uncertainty and subsequently affects communication performance.
• Scalability: As the number of UAVs and users increases, the dimensionality of optimization problems grows rapidly, leading to combinatorial explosion and making traditional methods unsuitable for large-scale dynamic scenarios.
• Local optimality of heuristics: Simpler heuristic approaches, such as greedy algorithms [11], while computationally less demanding, often suffer from inherent sub-optimality and local convergence, failing to achieve desirable global communication performance across the network.

These issues point out the limitations of traditional optimization methods in highly dynamic UAV communication systems and highlight the importance of feedback based learning algorithms.

1.1.2. Single-Agent Reinforcement Learning Approaches

In order to overcome the limitations of traditional optimization methods, some scholars use reinforcement learning (RL) for communication resource allocation. RL learns the optimal strategy through direct interaction with the environment, unlike traditional optimization methods that rely on fixed computational modes. This method is very suitable for solving complex sequential decision-making problems in multi-UAV communication scenarios.

To simplify the problem, initial deep reinforcement learning (DRL) research was typically based on a single-agent system framework, which can represent a UAV or centralized controller. For example, some studies focus on trajectory design using algorithms such as deep Q-network (DQN) [12,13], while assuming fixed power control or user association schemes [14]. However, this decoupling method ignores the strong coupling between these variables. After recognizing the inherent relationship between the position, transmission power, and communication performance of UAVs, the mainstream trend in academia has shifted towards joint optimization of continuous variables, especially the joint design of UAV trajectories and transmission power. Solutions based on advanced algorithms such as the deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and proximal policy optimization (PPO) [15,16,17] have been shown to significantly outperform single-variable optimization. These studies consolidate the consensus that joint optimization is the key way to improve the efficiency of UAV communication.

However, in high dynamic scenarios where multiple UAVs provide communication services to multiple users, the DRL framework of a single agent, even as an independent learner (IL) applied to a single UAV (e.g., IPPO [18,19]), inherently faces severe challenges:

• High-dimensional action space: In scenarios where multiple UAVs provide communication services to multiple users, the dimension of the action space increases sharply with the number of UAVs and users, making it difficult for single-agent methods to effectively explore and learn optimal strategies.
• Non-stationarity and convergence difficulties: When multiple independent agents run in a shared environment, the dynamics of the environment become non-stationary from the perspective of each agent due to the constantly changing strategies of other agents. This non-stationarity leads to convergence difficulties, often causing algorithms to fall into local optima or requiring excessively long training time, severely limiting the upper limit of communication performance.
• Lack of global coordination: When multiple independent agents run in a shared environment, due to their complete independence, the lack of coordination between agents leads to malicious interference and limited global communication performance, such as the following:

  -

  Resource contention and interference: Uncoordinated power allocation and trajectory planning can cause severe inter-UAV interference, degrading overall network capacity and user throughput.

  -

  Collision avoidance failures: Without explicit coordination, UAVs may collide, leading to mission failure and communication service disruption.

  -

  Suboptimal coverage and fairness: Individual UAVs optimizing solely for their local objectives may neglect underserved areas or users, compromising network-wide coverage and fairness in communication quality.

  -

  Credit assignment problem: When negative feedback (e.g., collision, network congestion) occurs, it is difficult for an independent learner to determine whether it is an error in one’s own strategy or the influence of other agent strategies, resulting in low learning efficiency.

These limitations emphasize the necessity of transitioning from single-agent DRL to multi-agent DRL, which can clearly handle coordination and cooperation in complex multi-UAV communication environments.

1.1.3. Multi-Agent Reinforcement Learning Approaches

Therefore, after recognizing the inherent need for coordination in multi-UAV systems, the focus of research has shifted from single-agent DRL to MARL. While simple MARL implementations, such as the ILs mentioned above, still struggle with non-stationarity and credit assignment, the centralized training with decentralized execution (CTDE) framework [20] has emerged as the gold standard. Within CTDE, a centralized critic leverages global information during training to guide the learning of individual, decentralized actors. This architecture is adopted by algorithms ranging from the pioneering multi-agent deep deterministic policy gradient (MADDPG) [21] to the high-performing MAPPO [22]. Due to its outstanding collaborative performance and training stability, MAPPO is widely regarded as one of the most advanced state-of-the-art (SOTA) methods [23,24], especially in complex multi-UAV communication service scenarios. These CTDE-based MARL algorithms directly solve the problem of dimension explosion in single-agent methods and the lack of global coordination in IL methods, enabling the following:

• Enhance communication network performance: Through global information sharing and centralized evaluation, the system can collaboratively optimize the trajectories and resource allocation of multiple UAVs, effectively reduce interference, and improve network capacity and overall user throughput.
• Robust collision avoidance: Centralized training enables agents to learn collaborative strategies to avoid collisions, ensuring the safe and stable operation of multi-UAV system.
• Improving resource management and fairness: Coordinated methods help to allocate resources more fairly, optimize coverage, and thus improve the overall quality and user satisfaction of communication services.

However, while improving the performance of MARL in resource allocation tasks, some scholars have focused on the structural characteristics of task scenarios and introduced the concept of “task coupling structure” [25], defining it as the extent to which the optimal behavior of an agent depends on the behavior of other agents. For example, the classic QMIX algorithm [26] is specifically designed for tasks with strictly monotonic cooperative structures. These studies emphasize the necessity of designing algorithms that appropriately match the coupling characteristics of specific problems.

In the scenario where multiple UAVs provide communication services, the task scenario essentially presents a “loosely coupled” structure. Although UAVs must cooperate to avoid mutual interference and ensure continuous system operation, they are largely autonomous in providing services to users within their respective service areas. Therefore, mainstream MARL algorithms based on CTDE, such as standard MAPPO, may be limited due to their “over concentration” or “over coupling” properties as follows:

• Suppression of individualized performance: The single centralized critic of standard MAPPO is designed specifically for strongly coupled tasks to evaluate global returns. This mechanism may inadvertently dilute the unique contributions of individual agents while promoting overall cooperation, inhibiting exploration of personalized and local optimal behaviors. In loosely coupled communication tasks, a single UAV may have different user groups and local optimization objectives, which may hinder achieving maximum individual communication performance.
• Ambiguous credit assignment in loosely coupled settings: When receiving global rewards (or punishments), in a loosely coupled environment, a single centralized critic may find it difficult to accurately allocate credit or responsibility to various agents. The ambiguity of credit assignment may slow down learning speed or lead to the formation of suboptimal strategies, especially when the influence of agents is more localized, which can affect the overall efficiency and fairness of communication services.

Therefore, designing a MARL algorithm that not only includes the required communication fairness objectives, but also customizes it specifically for the loosely coupled task characteristics generated by complex multi-UAV communication tasks remains an important and challenging research direction, and current methods have not fully solved it.

1.2. Motivation

In MARL, advanced CTDE algorithms are crucial for coordinating multiple UAVs. Although pioneering methods like MADDPG [21] effectively extend DDPG to multi-agent systems and can handle continuous control tasks such as trajectory and power optimization, their dependence on deterministic strategies and inherent stability challenges limits their applicability to our specific problems. MAPPO [22,23,24] provides significant advantages for joint optimization of continuous variables such as UAV trajectory and power allocation in high dynamic and safety-critical multi-UAV communication networks. MAPPO is built on a stable and robust PPO algorithm, using a random strategy that inherently promotes better exploration in complex, high-dimensional continuous action spaces. This feature is crucial for discovering diverse and energy-efficient strategies in dynamic UAV environments, ensuring robust learning against uncertainty. In addition, MAPPO’s improved stability and ease of adjustment make it a more reliable choice for formulating policies that can ensure safe operation (such as collision avoidance) and continuous communication services under strict energy constraints. Therefore, while pursuing high communication performance and system robustness, MAPPO has become the preferred benchmark for complex multi-UAV communication tasks.

However, there are still two limitations to current research. Firstly, most current research mainly focuses on maximizing the total rate in optimizing objectives [13,22,23], often neglecting fairness among users. Although methods such as maximizing minimum user performance [8] or using the jain fairness [27] index consider fairness among users, they have the problem of ignoring overall system performance or easily falling into local optima in complex optimization environments. Proportional fairness (PF) is a classic indicator that balances performance and fairness, and its application has not been explored to a large extent within this advanced MARL framework.

Secondly, although joint optimization is the most ideal way to improve the overall performance of the system, it is computationally infeasible to directly apply the PF objective function to a fully joint MARL optimization framework. On the other hand, if hierarchical optimization is adopted to avoid this problem, the resulting loosely coupled task structure poses new challenges to the MARL algorithm itself. Simple methods like IPPO are plagued by non-stationarity and cannot capture subtle but crucial inter-agent coupling, such as collision avoidance. This is because sparse global negative feedback signals, such as collision penalties, are often attributed by each independent learner to the randomness of their own strategy or environmental noise, making it difficult to form effective collaborative avoidance strategies. On the contrary, CTDE algorithms designed for mainstream strongly coupled scenarios such as MAPPO may suppress the exploration of individual optimal performance strategies in loosely coupled scenarios due to their overly concentrated value estimation, resulting in poor overall system performance.

Therefore, designing a MARL algorithm that not only considers fairness objectives, but also tailors it to the loosely coupled task characteristics generated by hierarchical optimization has become an important research direction.

1.3. Contribution

To address the aforementioned challenges, this paper proposes a hierarchical framework centered on a novel MARL algorithm, termed MAPPO with decoupled critics (MAPPO-DC). In the upper layer, an efficient user-to-UAV assignment is performed using a lightweight clustering algorithm. In the lower layer, our proposed MAPPO-DC algorithm enables each agent to learn tailored policies for trajectory and power control. Specifically, by redesigning the critic’s update rule, MAPPO-DC allows each agent to distinguish shared global value from its local, task-specific advantages, achieving a sophisticated balance between global coordination and individualized exploration. Furthermore, a composite reward function, congruent with the MAPPO-DC architecture, is devised based on reward construction principles to guide the learning process toward the proportional fairness objective.

The main contributions of this paper are as follows:

• We design an efficient hierarchical framework to mitigate the combinatorial explosion caused by an overly large joint optimization space.
• Based on a detailed analysis of the scene’s task coupling, we propose the MAPPO-DC algorithm tailored for the loosely coupled task scenario inherent to the hierarchical framework. A composite reward is also designed for this architecture to ensure both inter-agent coordination and individual agent performance in loosely coupled settings.
• The simulation results demonstrate that our proposed MAPPO-DC algorithm outperforms IPPO, MAPPO, and other baseline algorithms. In addition, we analyze the intrinsic behavioral mechanisms of different algorithms under energy constraints.

Theoretically, MAPPO-DC is positioned as a CTDE-based algorithm with a multi-critic architecture. Relative to IPPO, it mitigates environmental non-stationarity by incorporating global states into the critic. Relative to standard MAPPO with a single centralized critic, MAPPO-DC decouples the value estimation to prevent the ambiguity of credit assignment caused by global reward averaging in loosely coupled tasks. Furthermore, unlike value-decomposition methods (e.g., VDN, QMIX) that rely on the monotonicity assumption of the global value function, MAPPO-DC requires no such constraints. Consequently, it is particularly well-suited for the loosely coupled communication scenarios presented in this work, which require equal emphasis on inter-agent coordination (e.g., safety constraints) and individual agent performance (e.g., customized power and trajectory control).

2. System Model

We consider a scenario, as illustrated in Figure 1, where a fleet of N energy-constrained UAVs provides wireless communication services to K users distributed over a two-dimensional plane. The set of UAVs is denoted by $\mathcal{N}=\{1,\dots ,N\}$, and the set of users is denoted by $\mathcal{K}=\{1,\dots ,K\}$. We use the index $i,j\in {\mathcal{N}}_{j\ne i}$ for UAVs and $k\in \mathcal{K}$ for users. At any given time step t, the position of the ith UAV is represented by its 3D Cartesian coordinates ${\mathbf{q}}_{\mathbf{i}}\left(t\right)\in {\mathbb{R}}^3$. Similarly, the position of the kth user is ${\mathbf{w}}_{\mathbf{k}}\left(t\right)\in {\mathbb{R}}^3$. The users are randomly distributed within the operational area [4,5,8]. As shown in Figure 1, multiple UAVs fly within this area to provide communication services to users. The specific parameters of the scenario, adjusted based on [28,29], are detailed in

Table A1. Simulation parameters settings.

Parameter	Notion	Value
Spatial Area (m)	$\mathcal{A}$	1000 m × 1000 m × 50 m
Number of Agents (UAVs)	N	5
Number of users	K	30
User Movement Speed	$v_u$	1 m/s
UAV Maximum Speed	$v_{max}$	25 m/s
Safe Distance Between UAVs	$d_{safe}$	10 m
Range of Served Users		[4, 8]
Flight Altitude Range		[20 m, 40 m]
Max. Communication Power	$P^{comm}$	0.1 W
Min. Allocated Power	$P_{min}$	1 × 10−6 W
Allocatable Bandwidth	$B_{total}$	10 Mbps
Hovering Power	$P^{hov}$	150 W
Motion Power Range	$(P_{min}^{fly},P_{max}^{fly})$	[150 W, 500 W]
Max. Battery Capacity	$E_{max}$	65,000 J
Number of Training Scenarios		1000
Number of Test Scenarios		200
Max. Training Episodes		100,000
Time Steps per Episode	$T_{max}$	200

.

2.1. Transmission Model

The communication link adopts a probabilistic line-of-sight (LoS) model [4], where the channel gain is not deterministic but is the expected value of the channel gains under LoS and non-loS (NLoS) conditions. The probability of LoS depends on the elevation angle between the UAV and the user.

The 3D distance between UAV i and user k is given by

$$d_{i,k}\left(t\right)=∥{\mathbf{q}}_i\left(t\right)-{\mathbf{w}}_k\left(t\right)∥$$

(1)

The horizontal distance between UAV i and user k is calculated as

$$d_{i,k}^{2D}\left(t\right)=\sqrt{{\left(x_i\left(t\right)-x_k\left(t\right)\right)}^2+{\left(y_i\left(t\right)-y_k\left(t\right)\right)}^2}$$

(2)

And the elevation angle between UAV i and user k is calculated as

$${\theta }_{i,k}\left(t\right)={\displaystyle \frac{180}{\pi }}arctan\left({\displaystyle \frac{h_i}{d_{i,k}^{2D}\left(t\right)}}\right)$$

(3)

The LoS probability is given by

$$P_{i,k}^{LoS}\left(t\right)={\displaystyle \frac{1}{1+a·exp\left(-b·\left({\theta }_{i,k}\left(t\right)-a\right)\right)}}$$

(4)

where a and b are empirical parameters dependent on the environment, set to 9.61 and 0.16, respectively [28].

Building upon free-space path loss and differentiating between LoS and NLoS conditions, the path loss for LoS and NLoS links are, respectively,

$$P{L}_{i,k}^{LoS}\left(t\right)=20{\log }_{10}\left(d_{i,k}\left(t\right)\right)+{\xi }_{LoS}$$

(5)

and

$$P{L}_{i,k}^{NLoS}\left(t\right)=20{\log }_{10}\left(d_{i,k}\left(t\right)\right)+{\xi }_{NLoS}$$

(6)

To calculate the average channel gain, we first convert the path losses to linear scale,

$${\overline{PL}}_{i,k}\left(t\right)=P_{i,k}^{LoS}\left(t\right)·{10}^{{\displaystyle \frac{-{\xi }_0-P{L}_{i.k}^{LoS}\left(t\right)}{10}}}+\left(1-P_{i,k}^{LoS}\left(t\right)\right)·{10}^{{\displaystyle \frac{-{\xi }_0-P{L}_{i,k}^{NLoS}\left(t\right)}{10}}}$$

(7)

Consequently, the average channel gain (in linear scale) is obtained by converting the average path loss,

$$h_{i,k}\left(t\right)={\beta }_0·{\overline{PL}}_{i,k}\left(t\right)$$

(8)

where ${\xi }_{LoS}$ and ${\xi }_{NLoS}$ are the excessive path loss for LoS and NLoS links, respectively, and  ${\beta }_0$ is the channel power gain at a reference distance of 1 m [29].

We employ an orthogonal frequency division multiple access (OFDMA) scheme, allocating independent frequency bands to each UAV to avoid co-channel interference among them. Let ${\mathcal{U}}_i\left(t\right)\subseteq \mathcal{K}$ be the set of users served by UAV i. According to the Shannon–Hartley theorem, for a user $k\in {\mathcal{U}}_i\left(t\right)$, the received signal-to-interference-plus-noise ratio (SINR) can be simplified as the signal-to-noise ratio (SNR) and expressed as

$$SN{R}_{i,k}\left(t\right)={\displaystyle \frac{P_{i,k}\left(t\right)h_{i,k}\left(t\right)}{B_k\left(t\right)N_0}}$$

(9)

where $P_{i,k}\left(t\right)$ is the transmit power allocated by UAV i to user k at time t, $h_{i,k}\left(t\right)$ is the channel gain between them, $N_0$ is the noise power spectral density, and $B_k\left(t\right)$ is the bandwidth allocated to the user. For analytical convenience, we assume an equal bandwidth allocation scheme, where each served user $k\in {\mathcal{U}}_i\left(t\right)$ receives $B_k\left(t\right)={\displaystyle \frac{B_{total}/N}{\left|{\mathcal{U}}_i\left(t\right)\right|}}$. The purpose of this assumption is to simplify the analysis of the problem and focus on key variables [18,20], ensuring fairness among users at the bandwidth level, while concentrating research on trajectory and power.

The achievable instantaneous throughput for user k at time t is

$$C_k\left(t\right)=B_k\left(t\right){\log }_2\left(1+SN{R}_{i,k}\left(t\right)\right)$$

(10)

2.2. Energy Consumption Model

The energy consumption of a UAV is primarily generated by its propulsion system, which includes both hovering and motion components. Based on the quadrotor energy consumption model in [30] and incorporating common simplifications of rotary-wing aircraft aerodynamics from [31], factors such as climb, descent, and blade profile drag are neglected to focus the research on the long-term joint optimization strategy for trajectory and power. Thus, the total power consumption $P_i\left(t\right)$ for UAV i at time t can be approximated [4] as

$$P_i\left(t\right)=P_i^{fly}\left(t\right)+P^{comm}$$

(11)

where $P^{comm}$ is the communication power. The propulsion power $P_i^{fly}\left(t\right)$ is calculated based on the UAV’s velocity vector,

$$P_i^{fly}\left(t\right)=P^{hov}+c_{move}·{∥{\mathbf{v}}_i\left(t\right)∥}^2$$

(12)

where $P^{hov}$ is the hovering power, ${\mathbf{v}}_i\left(t\right)$ is the velocity vector of UAV i, and $c_{move}$ is the motion-related power coefficient [4]. The remaining energy of UAV i at the next time step can be calculated by subtracting the consumed energy from the current remaining energy,

$$E_i\left(t\right)=E_i(t-1)-P_i\left(t\right)·\Delta t$$

(13)

where $E_i\left(t\right)$ is the remaining energy of UAV i, $\Delta t$ is the duration of a time step.

2.3. Problem Formulation

This study aims to maximize the system’s PF throughput by jointly optimizing the UAVs’ trajectories $\mathcal{Q}=\left\{{\mathbf{q}}_i\left(t\right)|\forall i\in \mathcal{N},\forall t\right\}$ and power allocation strategies $\mathcal{P}=\left\{P_{i,k}\left(t\right)|\forall i,k,t\right\}$ within a finite flight period and under energy constraints. This objective is achieved by maximizing the log-sum of the users’ long-term average throughputs, as formally expressed in the optimization problem (P1),

$$\begin{array}{ccccc}\left(\mathrm{P}1\right):& {\displaystyle \underset{\mathbf{Q},\mathbf{P}}{\max }}& {\displaystyle \sum _{k\in \mathcal{K}}}\log \left({\overline{R}}_k\right)& & \\ & \mathrm{s}.\mathrm{t}.& \left(C1\right):& ∥{\mathbf{q}}_i(t+1)-{\mathbf{q}}_i\left(t\right)∥\le v_{max}·\Delta t,& \forall i,t\\ & & \left(C2\right):& H_{min}\le z_i\left(t\right)\le H_{max},& \forall i,t\\ & & \left(C3\right):& ∥{\mathbf{q}}_i\left(t\right)-{\mathbf{q}}_j\left(t\right)∥\ge d_{safe},& \forall i\ne j,t\\ & & \left(C4\right):& {\displaystyle \sum _{k\in {\mathcal{U}}_i\left(t\right)}}{P}_{i,k}\left(t\right)\le P^{comm},& \forall i,t\\ & & \left(C5\right):& E_i\left(t\right)\ge 0,& \forall i,t\\ & & \left(C6\right):& K_{min}\le \left|{\mathcal{U}}_i\left(t\right)\right|\le K_{max},& \forall i,t\end{array}$$

(14)

where ${\overline{C}}_k={\displaystyle \frac{1}{T_{max}}}{\displaystyle \sum _{t=0}^{T_{max}-1}}{C}_k\left(t\right)$ is the long-term discounted cumulative throughput for user k, and $C_k\left(t\right)$ is the instantaneous throughput defined in (10). Constraints C1 to C6 correspond to the UAV maximum velocity, flight altitude, collision avoidance, communication power, battery endurance, and user serving load, respectively.

Problem (P1) is a mixed-integer non-convex optimization problem, characterized by highly coupled variables, high-dimensional state-action spaces, and the highly dynamic nature of UAV communication environments. Due to these complexities, including the problem being NP-hard and the partial observability from each agent’s perspective, it is intractable for traditional optimization methods to obtain real-time optimal solutions. Therefore, we model this problem as a multi-agent partially observable Markov decision process (POMDP). We then employ a MARL approach, specifically our proposed MAPPO-DC framework, to learn an approximately optimal policy that addresses these challenges and maximizes the long-term cumulative reward, with a carefully designed reward function aligned to the objective of (P1).

3. Method

Although the end-to-end MARL method with a fully joint action space is theoretically superior, the explosion of fully joint action space combinations at our scene scale (N UAVs and K users) makes existing algorithms computationally unfeasible. More importantly, in most communication scenarios, the decision time scale of user clusters is much larger than the decision time of trajectory and power control. This natural separation of time scales provides a convincing theoretical basis for adopting a hierarchical optimization framework. The concept of decomposition has also been validated in traditional optimization fields. For example, reference [8] employs the BCD method to solve a similar problem, decomposing the complex joint optimization task into more tractable subproblems that are solved alternately. This inspires our approach, suggesting that problem decomposition, in traditional or learning-based methods, is a critical path toward finding feasible and efficient solutions.

Therefore, we adopted a hierarchical architecture. At the upper level, we adopted a user clustering algorithm tailored to our scenario, which combines nearest neighbor assignment with intelligent user offloading. At the bottom level, our proposed MAPPO-DC algorithm is used to jointly optimize UAV trajectory planning and power allocation. The hierarchical framework decouples user association from trajectory control, which inevitably sacrifices some global optimality. Specifically, the heuristic clustering based on nearest distance may lead to suboptimal load balancing in boundary cases where a user could be better served by a slightly more distant but less congested UAV (to maximize SINR via power headroom). However, finding the globally optimal assignment requires solving a combinatorial optimization problem ($O\left(N_{uav}^{N_{user}}\right)$), which is prohibitive for real-time control. Our proposed intelligent user offloading mechanism mitigates this impact by reassigning users from overloaded UAVs to neighboring ones, ensuring that the physical capacity constraints are met. While not mathematically globally optimal, this approach provides a robust Pareto improvement over static assignments, balancing computational latency with system throughput.

The time complexity of our proposed heuristic clustering algorithm is $O(N_{uav}·N_{user})$, and in the current scenario (30 users, 5 UAVs), it only requires microsecond-level computation. In contrast, finding the theoretical global optimal solution for user allocation is a combinatorial optimization problem with a complexity of $O\left(N_{uav}^{N_{user}}\right)$. As the number of UAVs and users increases, the computational cost of this heuristic algorithm only increases linearly, while the computational cost of finding the global optimal solution increases exponentially. Therefore, our hierarchical architecture actually obtained a high-quality suboptimal solution through heuristic methods at an extremely low time cost, leaving a sufficient computational time window for real-time control of the lower-level MARL. This ensures the real-time capability of the system during large-scale expansion.

The specific meanings corresponding to the symbols are shown in

Table 1. Main notions.

Notion	Description
i	Agent (UAV) Index: The i-th agent (UAV)
k	User Index: The k-th user
t	Time Step: The current discrete time step
${\mathbf{q}}_{\mathbf{i}}\left(t\right)$	UAV Position: Position coordinates of UAV i at time step t
${\mathbf{w}}_{\mathbf{k}}\left(t\right)$	User Position: Position coordinates of user k at time step t
$h_{i,k}\left(t\right)$	UAV-to-User Channel Gain: Channel gain from UAV i to user k at time step t
$C_k\left(t\right)$	User Instantaneous Throughput: Instantaneous throughput of user k at time step t
${\mathbf{o}}_i\left(t\right)$	Local Observation: Local information observed by agent i at time step t
${\mathbf{s}}_t$	Global State: Aggregated global information from all agents’ observations
$\overline{C_k}\left(t\right)$	User Historical Throughput: Historical throughput of user k
${\mathbf{a}}_i\left(t\right)$	Action: Action executed by agent i at time step t
${\pi }_{{\theta }_i}$	actor Policy: The agent’s policy network
${\theta }_i$	actor Parameters: Parameters of the actor network
$P_{i,k}\left(t\right)$	UAV Allocated Power: Power allocated by UAV i to user k
$P_i\left(t\right)$	Total UAV Power Consumption: Total power consumption of UAV i at time step t
$E_i\left(t\right)$	Remaining UAV Energy: Remaining energy of UAV i at time step t
${\mathcal{U}}_i\left(t\right)$	User Cluster: The cluster of users served by UAV i
$V_{{\varphi }_i}$	critic Value Function: The agent’s value network
${\varphi }_i$	critic Parameters: Parameters of the critic network
$V_{{\varphi }_i^{\prime }}$	Target critic: The target network for the critic, used to stabilize training, whose parameters are a soft update of ${\varphi }_i$
$\mathcal{B}$	Experience Replay Buffer: A buffer used to store past experiences
$r_i\left(t\right)$	Reward: Reward for agent i at time step t
$\gamma$	Discount Factor: The discount rate for future rewards, set to 0.92
$d_t$	Done Flag: A flag indicating if the episode terminates at time step t (1 if done, 0 otherwise)
$y_{i,t}$	TD Target: The temporal-difference target used to train the critic
${\widehat{A}}_{i,t}$	Advantage Function: Measures the advantage of an action compared to the average, used to train the actor
L	Loss Function: The objective function to be minimized (or maximized)
${\rho }_t\left({\theta }_i\right)$	Policy Ratio: The ratio comparing the new and old policies
$\epsilon$	PPO Clipping Parameter: The hyperparameter in PPO used to limit the magnitude of policy updates
$T_{epochs}$	Training Epochs: The total number of training epochs

.

3.1. User Clustering

As mentioned above, incorporating discrete user association decisions into the MARL action space leads to a combinatorial explosion. Furthermore, the decision frequency for user association is typically lower than that for trajectory and power control. Therefore, we employ an efficient heuristic-based clustering algorithm to periodically handle the user association problem, thereby constraining the action space of the lower-layer MARL task to continuous control variables. The procedure for hierarchical control, including periodic clustering, is detailed in Algorithm 1.

Algorithm 1 User Clustering via Nearest-Neighbor Assignment and Intelligent Offloading

1: Input: UAV set $\mathcal{N}$, user set $\mathcal{K}$, max users per UAV $N_{max}$, UAV positions ${\left\{{\mathbf{q}}_i\left(t\right)\right\}}_{i\in \mathcal{N}}$, user positions ${\left\{{\mathbf{w}}_{\mathbf{k}}\left(t\right)\right\}}_{k\in \mathcal{K}}$, total time steps T, clustering period $T_{cluster}$ 2: Initialize: UAV positions ${\mathbf{q}}_i\left(0\right)$, user positions ${\mathbf{w}}_{\mathbf{k}}\left(0\right)$, initial user-UAV association map ${\mathcal{U}}_i\left(0\right)$ 3: for time step $t=0$ to $T-1$ do 4:      if $t\left(\mathrm{mod}{T}_{cluster}\right)==0$ then // Periodically update clusters 5:       ${\mathcal{U}}_i\left(t\right)\leftarrow \mathbf{UpdateUserClusters}(\mathcal{N},\mathcal{K},{\mathbf{N}}_{max},{\left\{{\mathbf{q}}_{\mathbf{i}}\left(\mathbf{t}\right)\right\}}_{\mathbf{i}\in \mathcal{N}},{\left\{{\mathbf{w}}_{\mathbf{k}}\left(\mathbf{t}\right)\right\}}_{\mathbf{k}\in \mathcal{K}})$ 6:     else 7:      ${\mathcal{U}}_i\left(t\right)\leftarrow {\mathcal{U}}_i(t-1)$ // Maintain previous association 8:     end if 9:     Execute lower-layer MAPPO-DC policy based on ${\mathcal{U}}_i\left(t\right)$ to determine UAV velocities ${\mathbf{v}}_i\left(t\right)$ and powers $P_{i,k}\left(t\right)$ 10:     Update UAV positions: ${\mathbf{q}}_i(t+1)\leftarrow {\mathbf{q}}_i\left(t\right)+{\mathbf{v}}_i\left(t\right)\Delta t$ 11: end for 12: Function $\mathbf{UpdateUserClusters}(\mathcal{N},\mathcal{K},{\mathbf{N}}_{max},{\left\{{\mathbf{q}}_{\mathbf{i}}\left(\mathbf{t}\right)\right\}}_{\mathbf{i}\in \mathcal{N}},{\left\{{\mathbf{w}}_{\mathbf{k}}\left(\mathbf{t}\right)\right\}}_{\mathbf{k}\in \mathcal{K}})$ 13: Output: Final user-UAV association map $\mathcal{U}\left(t\right)=\left\{{\mathcal{U}}_1\left(t\right),\dots ,{\mathcal{U}}_N\left(t\right)\right\}$ Phase 1: Nearest-Neighbor Assignment 14: Initialize: clusters ${\mathcal{U}}_i\left(t\right)\leftarrow \varnothing$ for all $i\in \mathcal{N}$ 15: for each user $k\in \mathcal{K}$ do 16:     Find the nearest UAV: $i^{\ast }\leftarrow arg\underset{i\in \mathcal{N}}{min}{\mathbf{w}}_k\left(t\right)-{\mathbf{q}}_i\left(t\right)$ 17:     Assign user k to UAV $i^{\ast }:{\mathcal{U}}_{i^{\ast }}\left(t\right)\leftarrow {\mathcal{U}}_{i^{\ast }}\left(t\right)\cup \left\{k\right\}$ 18: end for Phase 2: Intelligent Offloading 19: Initialize: Overloaded UAV set ${\mathcal{N}}_{\mathrm{over}}\leftarrow \left\{i\in \mathcal{N}\left|\right|{\mathcal{U}}_i\left(t\right)>N_{max}\right\}$ 20: while ${\mathcal{N}}_{\mathrm{over}}\ne \varnothing$ do 21:     Select an overloaded UAV $i\in {\mathcal{N}}_{\mathrm{over}}$ 22:     Identify the user to offload: $k_{off}\leftarrow arg\underset{k\in {\mathcal{U}}_i\left(t\right)}{max}{\mathbf{w}}_k\left(t\right)-{\mathbf{q}}_i\left(t\right)$ 23:     Define candidate receiver UAVs: ${\mathcal{N}}_{\mathrm{cand}}\leftarrow \left\{j\in \mathcal{N}\setminus \left\{i\right\}\left|\right|{\mathcal{U}}_j\left(t\right)<N_{max}\right\}$ 24:     if ${\mathcal{N}}_{\mathrm{cand}}\ne \varnothing$ then 25:      Find the best receiver UAV: $j^{\ast }\leftarrow arg\underset{j\in {\mathcal{N}}_{\mathrm{cand}}}{min}{\mathbf{w}}_{k_{\mathrm{off}}}\left(t\right)-{\mathbf{q}}_j\left(t\right)$ 26:      Update clusters: ${\mathcal{U}}_i\left(t\right)\leftarrow {\mathcal{U}}_i\left(t\right)\setminus \left\{k_{off}\right\},{\mathcal{U}}_{j^{\ast }}\left(t\right)\leftarrow {\mathcal{U}}_{j^{\ast }}\left(t\right)\cup \left\{k_{off}\right\}$ 27:     else 28:      Break // No available UAV to receive the user 29:     end if 30:      Update overloaded set: ${\mathcal{N}}_{\mathrm{over}}\leftarrow \left\{i\in \mathcal{N}\left|\right|{\mathcal{U}}_i\left(t\right)>N_{max}\right\}$ 31: end while Safety truncation for rare unsolvable cases 32: for each UAV i with $\left|{\mathcal{U}}_i\left(t\right)\right|>N_{max}$ do 33:     Sort users in ${\mathcal{U}}_i\left(t\right)$ by distance to ${\mathbf{q}}_i\left(t\right)$ in ascending order 34:     Keep the nearest $N_{max}$ users: ${\mathcal{U}}_i\left(t\right)\leftarrow$ first $N_{max}$ users in sorted list 35: end for 36: return Final clusters $\mathcal{U}\left(t\right)$

3.2. State Space

The MAPPO-DC algorithm adheres to the CTDE paradigm, which requires aggregating the local observations of all agents to form a global state. State space is partitioned into two components: the local observation space and the global state space.

The self-centric local observation for agent i, denoted as ${\mathbf{o}}_i\left(t\right)$, comprises

$${\mathbf{o}}_i\left(t\right)=\left[{\mathbf{q}}_{\mathbf{i}}\left(t\right),{\left\{{\mathbf{q}}_{\mathbf{j}}\left(t\right)-{\mathbf{q}}_{\mathbf{i}}\left(t\right)\right\}}_{i,j\in \mathcal{N},j\ne i},{\left\{{\mathbf{q}}_i\left(t\right)-{\mathbf{w}}_{\mathbf{k}}\left(t\right)\right\}}_{k\in {\mathcal{U}}_i\left(t\right)},{\left\{\overline{C_k}\left(t\right)\right\}}_{k\in {\mathcal{U}}_i\left(t\right)},E_i\left(t\right)\right]$$

(15)

It includes the agent’s absolute position ${\mathbf{q}}_{\mathbf{i}}\left(t\right)$, its relative positions to other UAVs ${\left\{{\mathbf{q}}_{\mathbf{j}}\left(t\right)-{\mathbf{q}}_{\mathbf{i}}\left(t\right)\right\}}_{i,j\in \mathcal{N},j\ne i}$, the relative positions ${\left\{{\mathbf{q}}_i\left(t\right)-{\mathbf{w}}_{\mathbf{k}}\left(t\right)\right\}}_{k\in {\mathcal{U}}_i\left(t\right)}$ to its served users ${\mathcal{U}}_i\left(t\right)$, the historical average throughputs of these users ${\left\{\overline{C_k}\left(t\right)\right\}}_{k\in {\mathcal{U}}_i\left(t\right)}$, and its own remaining energy $E_i\left(t\right)$, where $\overline{C_k}\left(t\right)={\displaystyle \frac{1}{t}}{\displaystyle \sum _{\tau =1}^t}{C}_k\left(\tau \right)$. To maintain a fixed input dimension, information related to the served users is padded to the maximum users a UAV can serve ($K_{max}$).

The global state ${\mathbf{s}}_t$ is formed by concatenating the local observations of all agents, thereby containing complete information about all dynamic elements in the system. It is constructed as

$${\mathbf{s}}_t=\left[{\left\{{\mathbf{q}}_i\left(t\right)\right\}}_{i\in \mathcal{N}},{\left\{{\mathbf{w}}_{\mathbf{k}}\left(t\right)\right\}}_{k\in \mathcal{K}},{\left\{\overline{C_k}\left(t\right)\right\}}_{k\in \mathcal{K}},{\left\{E_i\left(t\right)\right\}}_{i\in \mathcal{N}}\right]$$

(16)

It consists of the set of all UAV positions ${\left\{{\mathbf{q}}_i\left(t\right)\right\}}_{i\in \mathcal{N}}$, the set of all user positions ${\left\{{\mathbf{w}}_k\left(t\right)\right\}}_{k\in \mathcal{K}}$, the historical average throughputs of all users ${\left\{\overline{C_k}\left(t\right)\right\}}_{k\in \mathcal{K}}$, and the remaining energy of all UAVs ${\left\{E_i\left(t\right)\right\}}_{i\in \mathcal{N}}$.

3.3. Action Space

At each time step t, the action of agent i, denoted as ${\mathbf{a}}_i\left(t\right)\in {\mathbb{R}}^{3+K_{max}}$, is composed of two continuous vectors: a trajectory control action ${\mathbf{a}}_i^{traj}\left(t\right)\in {\mathbb{R}}^3$ and a power allocation action ${\mathbf{a}}_i^{pwr}\left(t\right)\in {\mathbb{R}}^{K_{max}}$.

$${\mathbf{a}}_i\left(t\right)=\left\{{\mathbf{a}}_i^{traj}\left(t\right),{\mathbf{a}}_i^{pwr}\left(t\right)\right\}$$

(17)

3.3.1. Trajectory Planning

For the trajectory planning action ${\mathbf{a}}_i^{traj}\left(t\right)$, the agent’s action aims to determine its velocity or displacement for the next time step. Specifically, the actor network outputs a mean value ${\mu }_i^{traj}\left(t\right)\in {\mathbb{R}}^3$ for a displacement vector $\Delta {\mathbf{q}}_i\left(t\right)={[\Delta x,\Delta y,\Delta z]}_{i,t}$. The actual displacement increment is sampled from a normal distribution centered at this mean,

$${\mathbf{a}}_i^{traj}\left(t\right)=\Delta {\mathbf{q}}_i\left(t\right)\sim \mathcal{N}\left({\mu }_i^{traj}\left(t\right),{\sigma }_{traj}^2\right)$$

(18)

where ${\sigma }_{traj}$ is a learnable or fixed standard deviation. The UAV’s next position is updated as ${\mathbf{q}}_i(t+1)={\mathbf{q}}_i\left(t\right)+\Delta {\mathbf{q}}_i\left(t\right)$. The actual velocity of UAV i at time t is then ${\mathbf{v}}_i(t+1)={[\Delta x,\Delta y,\Delta z]}_{i,t}/\Delta t$, which effectively defines the desired velocity for the current time step.

3.3.2. Power Allocation

For the power allocation action ${\mathbf{a}}_i^{pwr}\left(t\right)$, the agent’s action aims to decide how to distribute its total communication power $P_{max}$ among its served users. We implement this through a Softmax-based mechanism. Specifically, the actor network outputs a vector of raw logit values, ${\mathbf{l}}_{\mathbf{i}}\left(t\right)=\left[l_{i,1}\left(t\right),l_{i,2}\left(t\right),\dots ,l_{i,K_{max}}\left(t\right)\right]\in {\mathbb{R}}^{K_{max}}$. For a user $k\in {\mathcal{U}}_i\left(t\right)$, the allocated power $P_{i,k}\left(t\right)$ is obtained by applying a Softmax function to the corresponding logit values and scaling by the total available power,

$${\mathbf{a}}_i^{pwr}\left(t\right)=P_{i,k}\left(t\right)=P_i^{comm}·{\displaystyle \frac{exp\left(l_{i,k}\left(t\right)\right)}{{\displaystyle \sum _{k\in {\mathcal{U}}_i\left(t\right)}}exp\left(l_{i,k}\left(t\right)\right)}}$$

(19)

The elements of the $K_{max}$-dimensional vector ${\mathbf{l}}_{\mathbf{i}}\left(t\right)$ are the unnormalized raw logit values. These values are transformed into power allocation ratios via the Softmax function, thereby enabling continuous power control for the served users.

3.4. Reward

To maximize both overall system coordination and individual agent performance, we adapt the reward structure for MAPPO-DC by classifying each reward component based on its contribution to either coordination or individual performance. Specifically, the collision penalty $r^{coll}\left(t\right)$, PF reward $r^{pf}\left(t\right)$, and user satisfaction reward $r^{bonus}\left(t\right)$, which contribute to agent coordination, are categorized as shared rewards $r^{share}\left(t\right)$. Conversely, the position guidance reward $r^{pos}\left(t\right)$, motion smoothness penalty $r^{smooth}\left(t\right)$, boundary penalty $r^{bound}\left(t\right)$, and crash penalty $r^{crash}\left(t\right)$, which relate to an agent’s individual behavior, are categorized as individual rewards. Since the individual rewards differ for each agent, the total reward for each agent is also distinct

$$r_i\left(t\right)=w_{share}·r^{share}\left(t\right)+w_{ind}·r_i^{ind}\left(t\right)$$

(20)

where $r^{share}\left(t\right)$ is the reward common to all agents, $r^{ind}\left(t\right)$ is the reward unique to agent i, and w are the weighting coefficients for each sub-reward,

$$r^{share}\left(t\right)=w_{coll}·r^{coll}\left(t\right)+w_{pf}·r^{pf}\left(t\right)+w_{bonus}·r^{bonus}\left(t\right)$$

(21)

$$r_i^{ind}\left(t\right)=w_{pos}·r_i^{pos}\left(t\right)+w_{smooth}·r_i^{smooth}\left(t\right)+w_{bound}·r_i^{bound}\left(t\right)+w_{crash}·r_i^{crash}\left(t\right)$$

(22)

To align with the primary objective in (P1), we design the main component of the reward function based on the instantaneous sum-log-rate, which serves as a dense signal to promote proportional fairness. The PF reward at time t is defined as

$$r^{pf}\left(t\right)=\sum _{k\in \mathcal{K}}\log \left(C_k\left(t\right)+{\epsilon }_{pf}\right)$$

(23)

where $C_k\left(t\right)$ is the instantaneous throughput of the user k, and ${\epsilon }_{pf}$ is a small positive constant to ensure numerical stability for logarithmic operation.

To encourage the fulfillment of a basic Quality of Service (QoS), we introduce an additional reward for exceeding a throughput threshold,

$$r^{bonus}\left(t\right)=\sum _{k\in \mathcal{K}}\mathbb{I}\left(C_k\left(t\right)>C_{thresh}\right)$$

(24)

where $\mathbb{I}(·)$ is the indicator function, which equals 1 when the condition is true and 0 otherwise, and $C_{thresh}$ is the throughput threshold.

To guide exploration within a vast action space and under strict energy constraints, we introduce a position guidance reward $r_i^{pos}\left(t\right)$ through reward shaping. This reward penalizes the distance between a UAV and the geometric center of its user cluster served by ${\mathbf{U}}_i\left(t\right)$, thus constraining the exploration space, preventing the model from converging to local optima and accelerating convergence.

$$r_i^{pos}\left(t\right)=-∥{\mathbf{q}}_i\left(t\right)-{\mathbf{U}}_i\left(t\right)∥$$

(25)

where the geometric center of the user cluster ${\mathbf{U}}_i\left(t\right)$ is calculated as

$${\mathbf{U}}_i\left(t\right)={\displaystyle \frac{1}{\left|{\mathcal{U}}_i\left(t\right)\right|}}\sum _{k\in {\mathcal{U}}_i\left(t\right)}{\mathbf{w}}_{\mathbf{k}}^{\mathbf{2}\mathbf{D}}\left(t\right),\mathrm{if}\left|{\mathcal{U}}_i\left(t\right)\right|>0$$

(26)

where ${\mathbf{U}}_i\left(t\right)\in {\mathbb{R}}^2$ is the 2D coordinate vector of the target cluster center for UAV i at time t, ${\mathcal{U}}_i\left(t\right)$ is the set of user indices assigned to UAV i, $\left|{\mathcal{U}}_i\left(t\right)\right|$ is the number of users in that cluster, and ${\mathbf{w}}_{\mathbf{k}}^{\mathbf{2}\mathbf{D}}\left(t\right)\in {\mathbb{R}}^2$ is the 2D projection of user k’s position vector onto the horizontal plane. ${\displaystyle \sum _{k\in {\mathcal{U}}_i\left(t\right)}}{\mathbf{w}}_{\mathbf{k}}^{\mathbf{2}\mathbf{D}}\left(t\right)$ is the sum of the position vectors of all users within the cluster.

To ensure that the UAVs generate physically feasible and energy-efficient smooth trajectories, we penalize abrupt changes in their actions,

$$r_i^{smooth}\left(t\right)=-\sum _{i\in \mathcal{N}}∥{\mathbf{a}}_i^{traj}\left(t\right)-{\mathbf{a}}_i^{traj}(t-1)∥$$

(27)

A collision penalty is incurred when the distance between any two UAVs is less than the safety distance $d_{safe}$,

$$r^{coll}\left(t\right)=\sum _{j\in \mathcal{N},j>i}{r}^{coll}·\mathbb{I}\left(∥{\mathbf{q}}_i\left(t\right)-{\mathbf{q}}_j\left(t\right)∥<d_{safe}\right)$$

(28)

where $r^{coll}$ is a large negative constant.

A boundary penalty is applied when a UAV reaches the scenario boundaries,

$$r_i^{bound}\left(t\right)=r^{bound}·\mathbb{I}\left({\mathbf{q}}_i\left(t\right)\notin \mathcal{A}\right)$$

(29)

where $\mathcal{A}$ represents the allowed 3D flight region and $r^{bound}$ is a negative constant. Unlike the collision and energy depletion penalties, $r_i^{bound}\left(t\right)$ is a relatively smaller negative value. This distinction is made because users are randomly distributed within the scenario; if a UAV’s cluster center is near the edge, the policy should be allowed to explore the boundary without incurring an excessive penalty, which would otherwise overly constrain the search space.

An energy depletion penalty is applied when the UAV’s remaining energy drops below zero,

$$r_i^{crash}\left(t\right)=r^{crash}·\mathbb{I}\left(E_i\left(t\right)\le 0,E_i(t-1)>0\right)$$

(30)

where $r^{crash}$ is a large negative constant.

The following rationale behind this composite reward shaping is threefold:

• Guidance: The dense signal $r_i^{pos}\left(t\right)$ constrains the exploration space, guiding agents towards valid topological formations early in training.
• Constraints: $r_i^{smooth}\left(t\right)$ acts as a soft constraint to enforce physical feasibility, while $r_i^{crash}\left(t\right)$ and $r_i^{bound}\left(t\right)$ serve as hard safety constraints (sparse penalties) to prune unsafe state-action regions.
• Optimization: Finally, $r^{pf}\left(t\right)$ and $r^{bonus}$ drive the policy towards the core objective of fair communication performance.

This decomposition ensures that agents learn how to fly safely before optimizing how to communicate efficiently, and the weights of each reward are shown in

Table A2. Network parameter table.

Parameter	Symbol	Value
Discount Factor	$\gamma$	0.99
GAE Lambda	$\lambda$	0.95
PPO Clip Range	$\epsilon$	0.2
PPO Epochs	$K_{epochs}$	5
Actor Learning Rate	$a_{actor}$	1 × 10−5 (Linear Decay)
Critic Learning Rate	$a_{critic}$	5 × 10−5 (Linear Decay)
Entropy Coefficient	$c_{ent}$	0.02
Batch Size	B	800 (4 × 200)
Max Gradient Norm		0.5
Optimizer		Adam (eps = 1 × 10−5)
Target critic soft update	$\tau$	0.01
Shared reward weight	$w_{share}$	1.1
Individual reward weight	$w_{ind}$	0.9
Collision penalty weight	$w_{coll}$	−80
PF reward weight	$w_{pf}$	1.0
Weight of user satisfaction reward	$w_{bonus}$	5.0
Position guided reward weight	$w_{pos}$	0.8
Motion smoothing penalty weight	$w_{smooth}$	−0.1
Boundary penalty weight	$w_{bound}$	−10
Cracked penalty weight	$w_{crash}$	−200

.

3.5. MAPPO-DC Algorithm

The MAPPO-DC algorithm proposed in this paper is built upon the MAPPO architecture and likewise adheres to the CTDE paradigm. Its workflow is detailed in Algorithm 2.

Algorithm 2 Multi-Agent Proximal Policy Optimization with Decoupled critics (MAPPO-DC)

1: Initialize: For each agent $i\in \mathcal{N}$: 2:     actor network ${\pi }_{{\theta }_i}$ with parameters ${\theta }_i$ 3:     critic network $V_{{\varphi }_i}$ with parameters ${\varphi }_i$ 4:     Target critic network $V_{{\varphi }_i^{\prime }}$ with parameters ${\varphi }_i^{\prime }\leftarrow {\varphi }_i$ 5: Initialize: Replay buffer $\mathcal{B}$ 6: for episode = 1, 2, ... do 7:     Reset environment and get initial global state ${\mathbf{s}}_{\mathbf{0}}$ and local observations ${\left\{{\mathbf{o}}_i\left(0\right)\right\}}_{i\in \mathcal{N}}$ // Decentralized Execution & Data Collection 8:     for t = 1, 2, ..., T do 9:        for each agent $i\in \mathcal{N}$ in parallel do 10:         Select action ${\mathbf{a}}_i\left(t\right)\sim {\pi }_{{\theta }_i}\left(·|{\mathbf{o}}_i\left(t\right)\right)$ 11:        end for 12:        Execute joint action $\mathbf{a}\left(t\right)={\left\{{\mathbf{a}}_i\left(t\right)\right\}}_{i\in \mathcal{N}}$ in the environment 13:        Receive next global state ${\mathbf{s}}_{t+1}$, next local observations ${\left\{{\mathbf{o}}_i(t+1)\right\}}_{i\in \mathcal{N}}$, and individual rewards ${\left\{r_i\left(t\right)\right\}}_{i\in \mathcal{N}}$ 14:        Store transition $\left({\mathbf{s}}_t,\left\{{\mathbf{o}}_i\left(t\right)\right\},\left\{{\mathbf{a}}_i\left(t\right)\right\},\left\{{\mathbf{r}}_i\left(t\right)\right\},\mathbf{s}(t+1),\left\{{\mathbf{o}}_i(t+1)\right\}\right)$ in buffer $\mathcal{B}$ 15:     end for // Centralized Training with Decoupled critics 16:     for training step = 1, 2, ..., $T_{epochs}$ do 17:        Sample a batch of trajectories from $\mathcal{B}$ 18:        for each agent $i\in \mathcal{N}$ in parallel do 19:           // — critic Update — 20:           Get global states from the batch 21:           Calculate TD-target: $y_i\left(t\right)=r_i\left(t\right)+\gamma \left(1-d\left(t\right)\right)V_{{\varphi }_i^{\prime }}\left(\mathbf{s}(t+1)\right)$ 22:           Update critic ${\varphi }_i$ by minimizing the loss: $\mathcal{L}\left({\varphi }_i\right)=\left[{\left(V_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)-y_i\left(t\right)\right)}^2\right]$ 23:           // — actor Update — 24:           Get local observations ${\mathbf{o}}_i$ from the batch 25:           Compute advantage estimates ${\widehat{A}}_i\left(t\right)$ using GAE with critic $V_{{\varphi }_i}$ 26:           Update actor ${\theta }_i$ by maximizing the PPO-Clip objective: 27:            ${\mathcal{L}}_{\mathrm{clip}}\left({\theta }_i\right)=\left[min\left({\rho }_t\left({\theta }_i\right){\widehat{A}}_i\left(t\right),clip\left({\rho }_t\left({\theta }_i\right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_i\left(t\right)\right)\right]$ 28:        end for 29:      end for // Update Target Networks 30:     for each agent $i\in \mathcal{N}$ do 31:       ${\varphi }_i^{\prime }\leftarrow \tau {\varphi }_i+(1-\tau ){\varphi }_i^{\prime }$ 32:     end for 33: end for

The values of the network parameters in Algorithm 2 are shown in Table A2. The network architectures for the actor and critic are depicted in Figure 2 and Figure 3, respectively.

The actor network employs a shared backbone followed by two independent heads for trajectory and power control, which share weights. A Tanh activation function is used on the trajectory output to constrain the velocity, while a Softmax layer normalizes the logits for power allocation. In the input stage, the remaining energy, being a one-dimensional parameter, is expanded to an 8-dimensional vector via dimension expansion to balance its feature scale relative to other input variables.

Specifically, the global state space is dominated by high-dimensional vectors representing UAV and user positions. Directly concatenating the scalar energy state $E_i\left(t\right)$ would cause a dimension imbalance problem [15], where the critical energy constraints are prone to be submerged by high-dimensional location features during gradient backpropagation. To address this, we employ a dimension spread technique (also known as feature embedding) to map the scalar energy to an $N_e$-dimensional vector. We set $N_e=8$ based on the following two considerations:

• Feature Scale Alignment: This value brings the energy feature representation to a comparable order of magnitude as the user state vectors (e.g., historical throughput), preventing any single feature group from dominating the network’s initial learning phase. This setting is consistent with the findings in [15], where an expansion to $N_e=10$ was shown to be essential for the agent to successfully learn energy-aware navigation tasks.
• Convergence Stability: By increasing the dimensionality of the energy input, we effectively increase the number of synaptic weights connecting the energy state to the first hidden layer. This amplifies the gradient signal related to energy consumption, thereby avoiding convergence to local optima where agents maximize throughput but fail to satisfy flight endurance constraints.

In contrast to the single critic in standard MAPPO, our MAPPO-DC algorithm, inspired by the structure of IPPO, assigns a dedicated critic network with distinct parameters ${\varphi }_i$ for each agent. Figure 3 illustrates the architecture for critic $V_{{\varphi }_i}$. Unlike IPPO, each critic in MAPPO-DC receives the global state as input. Although all critics share the same input of the global state $\mathbf{s}\left(t\right)$, they learn personalized value functions through their unique interactions with their respective paired actors.

Specifically, each critic $V_{{\varphi }_i}$ solves a regression problem. Its objective is to find a set of parameters ${\varphi }_i^{\ast }$ that minimizes the prediction error for future cumulative rewards. This objective can be formally expressed as

$${\varphi }_i^{\ast }=arg{\underset{{\varphi }_i}{min}}_{\tau \sim P\left(\tau |{\overrightarrow{\pi }}_{\overline{\theta }}\right)}\left[\sum _{t=0}^H{\left(V_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)-G\left(t\right)\right)}^2\right]$$

(31)

where $G\left(t\right)={\displaystyle \sum _{l=0}^{\infty }}{\gamma }^l{r}_{t+l+1}$, the true value of the discounted return $G\left(t\right)$ is only known at the end of an episode. In practice, we use the TD-target $y_i\left(t\right)=r_i\left(t\right)+\gamma \left(1-d\left(t\right)\right)V_{{\varphi }_i^{\prime }}\left(\mathbf{s}(t+1)\right)$ as its unbiased estimator. Therefore, the gradient update for critic $V_{{\varphi }_i}$ follows

$${\nabla }_{{\varphi }_i}L\left({\varphi }_i\right)\propto {\mathbb{E}}_{{\overrightarrow{\pi }}_{\overrightarrow{\theta }}}\left[\left(V_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)-y_i\left(t\right)\right){\nabla }_{{\varphi }_i}{V}_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)\right]$$

(32)

where expectation ${\mathbb{E}}_{{\overrightarrow{\pi }}_{\overrightarrow{\theta }}}$ is taken over the state-action stream generated by the joint policy of the entire team.

Although the input is the global state, the critic’s objective is not simply to learn the unconditional value of the state. Instead, it implicitly learns a conditional value function: given the state, what is the team’s expected future return assuming that the agent will follow policy? This implicit conditioning is achieved through the statistical averaging effect of the stochastic gradient descent.

Decomposing the joint action into agent i’s action $a_i\left(t\right)$ and the joint action of all other agents $a_{-i}\left(t\right)$. Both the reward $r_i\left(t\right)$ and the state transition $p\left(s(t+1)\left|\mathbf{s}\right(t),a(t)\right)$ are functions of these two components. From the perspective of critic $V_{{\varphi }_i}$, we obtain the following:

• $a_i\left(t\right)\sim {\pi }_{{\theta }_i}\left(·|o_i\left(t\right)\right)$: This is generated by its paired actor i, and its behavior pattern is tightly coupled with the critic’s learning process through the advantage function ${\widehat{A}}_i\left(t\right)$.
• $a_{-i}\left(t\right)\sim {\pi }_{{\theta }_{-i}}\left(·|o_{-i}\left(t\right)\right)$: This is generated by all other actors. Since the output of critic $V_{{\varphi }_i}$ does not directly influence the updates of other agents, the behavior of $a_{-i}\left(t\right)$ can be treated as part of the environmental stochasticity from the perspective of critic $V_{{\varphi }_i}$’s optimization process, introducing high-variance noise to the value estimation of $r_i\left(t\right)$ and $\mathbf{s}(t+1)$.

According to the law of total expectation, we can decompose the expected gradient update,

$${\mathbb{E}}_{\mathbf{s}\left(t\right)}\left[\nabla V_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)\right]={\mathbb{E}}_{\mathbf{s}\left(t\right)}\left[{\mathbb{E}}_{{\mathbf{a}}_i\sim {\pi }_{{\theta }_i}|\mathbf{s}\left(t\right)}\left[{\mathbb{E}}_{{\mathbf{a}}_{-i}\sim {\pi }_{{\theta }_{-i}}|\mathbf{s}\left(t\right)}\left[\nabla V_{{\varphi }_i}\left(\mathbf{s}\left(t\right)\right)\right]\right]\right]$$

(33)

The update for critic $V_{{\varphi }_i}$ is driven by the advantage signal ${\widehat{A}}_i\left(t\right)$. From its point of view, the actions of other agents $a_{-i}\left(t\right)$ introduce high-variance noise into the global state observation $\mathbf{s}\left(t\right)$, as these actions lack a direct causal link to agent i’s policy update. By performing a stochastic gradient descent over a large number of samples, critic $V_{{\varphi }_i}$ learns to average out this noise and focus on the stable correlation between the behavior of its paired actor and the resulting reward.

This statistical decoupling enables each critic to function as a specialized value estimator. It effectively learns a projection from the global state to a subspace most relevant to its actor’s role, enabling precise and personalized credit assignment even within a centralized training framework. In contrast, standard MAPPO employs a single, centralized critic that regresses towards a global return shared by all agents. While this mechanism promotes cooperation, in loosely coupled tasks, it dilutes agent i’s local reward signal within the global average, leading to credit assignment ambiguity for individual contributions and potentially stifling the exploration of individually optimal behaviors. MAPPO-DC addresses this issue by equipping each agent with its own dedicated critic.

4. Simulation Results

To ensure fairness in all hyperparameter adjustments, baseline comparisons, and ablation studies, all experiments were conducted using the same random seed. In addition, a dedicated scenario library was established before the training, consisting of a training scenario set and a testing scenario set. In the training set, only the initial positions of the UAV and the user are initialized. The test set not only initializes the location, but also fixes the user trajectory. This design ensures that the training process can explore as wide a range of strategies as possible, while ensuring fairness and reproducibility in the testing phase. The ratio of the training scenario set to the testing scenario set is set to 5:1.

In addition, in all MARL algorithms, the learning rates of critic and actor networks adopt linear decay. In order to continuously monitor the performance trend of the model, the current model is tested every 100 training cycles in the entire testing scenario set. The average performance of all test scenarios is recorded as the performance metric of the current model. To ensure that the test results do not affect the training dynamics, the testing process is completely independent of the training process.

To ensure a fair comparison, all experiments were implemented using Python 3.8 and PyTorch 1.12 on a platform with Intel i9-13900K CPU, GeForce RTX 4080 GPU. All reinforcement learning algorithms were trained for a fixed budget of 100,000 episodes. For the MARL baselines (IPPO, MAPPO, MAPPO-DC, MASAC, MADDPG), we maintained identical backbone network architectures (e.g., three hidden layers with 64 units each) and common hyperparameters, differing only in algorithm-specific parameters (e.g., entropy coefficients for MASAC). Additionally, non-learning baselines (greedy, APF, convex) were implemented using standard configurations from the literature to serve as rigorous benchmarks.

4.1. Comparative Study of Clustering Algorithms

We compared our proposed clustering method with the widely used K-means algorithm. Although K-means assigns a small number of users with low variance to each user cluster [32], its objective function (minimizing intra-cluster variance) is purely geometric and independent of communication constraints. This may result in users being assigned to geometrically optimal but physically distant UAVs, leading to service interruptions. Our method aims to avoid this critical issue by prioritizing geographical proximity.

Figure 4 illustrates that K-means achieves a more balanced user load distribution. Figure 5 demonstrates that in terms of individual user performance, the Nearest Neighbor Assignment + Smart Offloading scheme exhibits superior performance for each user, as users consistently experience optimal or suboptimal channel conditions. Figure 6 shows that in terms of overall system performance, the two schemes are comparable. However, in terms of the presence of unserved users, the Nearest Neighbor Assignment + Smart Offloading scheme ensures that all users are served across all episodes and all time steps, provided that the number of crashed UAVs is less than two.

We trained the MAPPO-DC algorithm architecture combining nearest neighbor assignment and intelligent offloading under 5 different random seeds, and counted the number of truncation operations mentioned in Algorithm 1, as well as the average and variance of other key indicators, the results are shown in

Table 2. Statistical data of 5 random seeds.

	Number of Truncation Operations (Crashed ≥ 2)	TP	PF	Crashed Final Zeroing Episode
Statistical Value	3 ± 1.5810	58.0991 ± 0.1108	430.3395 ± 0.1489	17,480 ± 572.9317
Range	(1, 5)	(57.9731, 58.2774)	(430.0815, 431.0025)	(16,750, 18,200)

. From the results, it can be observed that in the entire training episode (100,000 episodes), there were only a maximum of 5 truncation operations, all of which occurred before the crashed zeroing episode, indicating that this situation occurred very rarely; In addition, we observed that the variance of TP and PF under different random seeds is very small, while the variance of the zero episode of crashed is relatively large. This is because the performance limit that the model can reach is relatively high. It can also be said that the best solution explored by the model in the current scenario is relatively fixed. The plan to zero crashed was explored in the early stage of training, when the model is in the random exploration stage, so the variance is relatively large.

In summary, prioritizing the scenario requirements, we find that ensuring all users are served is preferred over strictly balancing the UAV load. Therefore, we select the Nearest Neighbor Assignment + Smart Offloading scheme as the final user clustering solution.

4.2. Clustering Frequency Experiment

The scenario studied involves mobile ground users that show high dynamism. Consequently, the clustering algorithm must dynamically update and iterate to ensure an effective follow-up service by the UAVs. This experiment seeks to determine the optimal dynamic clustering frequency.

In this experiment, the optimal clustering frequency is jointly determined by two parameters: the final test performance (average of the last 1000 episodes) and the episode count for stable crash zeroing (the episode after which the crash count consistently remains zero).

From the throughput (TP) and PF curves in Figure 7 and the numerical results in

Table 3. Comparison table of performance of different clustering frequencies.

Clustering Frequency	Final Test Reward	Final Test TP (Mbps)	Final Test PF	Crashed Final Zeroing Episode
10	61,661.97	58.069	430.08	16,300
20	54,723.71	53.801	424.92	10,800
40	54,520.61	54.264	425.52	10,400
50	56,240.68	54.108	425.36	13,900

, we observe that a clustering frequency (CF) of 10 performs relatively poorly early in training compared to other frequencies. However, during the convergence stage, CF = 10 yields the highest TP and PF. In contrast, the Reward curve in Figure 7 and the Pareto frontier scatter plot in Figure 8 indicate that CF = 10 requires the longest time for convergence and the slowest stable crash zeroing among the four tested frequencies. The lower clustering frequency (less frequent user cluster reassignment) stabilizes the user groups faced by the agents during training, allowing them to learn superior long-term strategies. However, since the environment changes more slowly, it takes longer to accumulate experience and fully explore the strategy space, resulting in slower convergence.

While CF = 10 exhibits the slowest convergence speed compared to higher frequencies, it ultimately achieves the highest converged TP and PF scores. This presents a trade-off between training efficiency and final inference performance. From a practical deployment perspective, model training is a one-time offline cost, whereas the converged policy’s performance dictates the system’s long-term utility during online execution. Since CF = 10 provides a more stable interaction window that allows agents to learn superior long-term strategies—albeit requiring more episodes to generalize across varying topologies—we prioritize the final performance gains over training speed. Therefore, we select a clustering frequency of 10 as the final parameter setting.

4.3. Comparison with Non-Learning Algorithms

We selected the greedy algorithm [11,33] as the primary non-learning baseline for comparison. This algorithm is a typical method that maximizes the instantaneous reward currently available. This study demonstrates the superiority of learning-based algorithms over non-learning ones under energy-constrained conditions.

Since the Greedy algorithm is a non-learning method, it does not undergo a learning and convergence process. Therefore, we compare its average performance across 200 test scenarios. For MAPPO-DC, we load the converged model and evaluate its performance on the same 200 test scenarios. The “average crashed” metric here refers to the average number of crashed UAVs in the test scenarios.

It can be seen from

Table 4. Average performance record of MAPPO-DC and greedy algorithm.

Metric	Average TP	Average PF	Average Crashed	Jain’s Index	Jain’s Std
Greedy	55.252	411.03	2.5	0.625	0.0733
MAPPO-DC	58.069	430.08	0	0.767	0.052

that our algorithm demonstrates a 5.826% improvement in TP, a 16.7 point increase in PF, and a 22.72% improvement in the Jain’s fairness index compared to the Greedy algorithm. Furthermore, as depicted in the performance curves from our experiments, the average crash count for MAPPO-DC decreases from an initial value of 0.26 to a stable convergence at 0 after approximately 20,000 training epochs. In contrast, the Greedy algorithm, which focuses on maximizing instantaneous rewards (where the position reward has the highest weight), leads to frequent and drastic maneuvers that rapidly deplete energy, ultimately failing to sustain continuous service. Consequently, its crash count remains at a persistently high level.

In conclusion, this experiment validates the need to use learning-based algorithms to effectively manage energy consumption in energy-constrained scenarios.

4.4. Comparison with Decoupled Optimization Methods

In the decoupled optimization experiment, we compare our method with two common approaches from existing research: trajectory optimization based on the artificial potential field (APF) method [34,35] and power optimization using a convex optimizer [8]. We denote the scheme combining MAPPO-DC for power optimization and APF for trajectory optimization as MAPPO-DC-P+APF-T. The scheme combining MAPPO-DC for trajectory optimization and convex optimization for power control is denoted as MAPPO-DC-T+Convex-P.

Figure 9 reveals that the advantage of joint optimization with MAPPO-DC over decoupled methods is substantial, primarily because the joint optimization scheme enables global observation and effective credit assignment. The results show that the MAPPO-DC joint optimization scheme outperforms the decoupled schemes in both PF and crash avoidance.

In the MAPPO-DC-P+APF-T scheme, the UAV trajectories are controlled by the non-learning APF algorithm. Since the APF method cannot adjust the trajectories based on the energy remaining of the UAV’s, the crash count fails to decrease. Although it achieves a high TP, its PF remains consistently low.

In the MAPPO-DC-T+Convex-P scheme, although MAPPO-DC can reduce the crash count to a low level by controlling flight energy consumption—which constitutes a major part of the total energy—it lacks control over communication power. While communication energy consumption is relatively small, the lack of direct control over this component prevents the critic from accurately assessing the risks associated with total energy expenditure, leading to a credit assignment failure. As shown in Figure 9, around the 19 k training epoch mark, the crash curve for the MAPPO-DC-T+Convex-P scheme begins to rise instead of fall. This occurs because the policy, after 19 k epochs, starts to push the energy limits in pursuit of higher PF, causing a temporary resurgence in crashes until the final policy stabilizes.

In summary, this experiment demonstrates that in energy-constrained scenarios, a learning algorithm must have control over the entire energy consumption profile to effectively mitigate the risk of UAV crashes and ensure continuous service to all users within the specified mission time.

4.5. Comparison with Other MARL Algorithms

The MAPPO-DC algorithm we propose is designed by integrating the multi-critic structure of IPPO [19,20] into the MAPPO framework. This design solves the problems in loosely coupled tasks by balancing the coordination between intelligent agents and the individual performance of individual agents. Therefore, in this experiment, we mainly compared MAPPO-DC with its basic algorithms, MAPPO and IPPO. In addition, we also compared the MASAC and MADDPG algorithms. Although we conducted extensive parameter tuning experiments, the MADDPG algorithm experienced gradient explosion after about 5500 episodes of training. Therefore, we did not add the training curve of MADDPG to Figure 10.

From the training time of each algorithm in

Table 5. Training time for each MARL algorithm.

Algorithm	MASAC	MADDPG	IPPO	MAPPO	MAPPO-DC
Training time for 100 k episodes (h)	195	about 120	120	105	102

, it can be seen that, compared to off policy IPPO, MAPPO, and MAPPO-DC algorithms, on policy MASAC and MADDPG algorithms require longer training time due to the need to maintain a larger relay buffer during the training process, while MASAC requires more computation time due to the inclusion of a double-Q network to eliminate high estimation bias. In addition, due to the gradient explosion that occurred during MADDPG training up to 5.5 k, we estimated the training time for 100 k episodes based on the time taken for the first 5.5 k episodes.

As shown in Figure 9 and Figure 10, both the MAPPO-DC-T+Convex-P scheme in Section 4.4 and the IPPO experiment in this section suffer from credit assignment failure, which introduces high-variance noise into the decision-making process. Specifically, for agent i in IPPO, its critic only receives observations from its corresponding actor. Consequently, when the actions of other agents result in negative feedback for agent i, it can neither identify the source of this negative feedback nor devise a solution, a phenomenon known as credit assignment failure. In the MAPPO-DC-T+Convex-P architecture, MAPPO-DC treats the uncontrollable communication power as noise from environmental non-stationarity. Similarly, in the IPPO architecture, each agent i treats all other agents $j(j\ne i)$ as noise from environmental non-stationarity. After the 21,000 training epochs, IPPO’s crash curve also begins to rise, which, while increasing TP, leads to a decrease in PF. At this stage, IPPO treats crashes as a sacrificial parameter in exchange for higher TP and PF.

MADDPG failed to converge due to severe overestimation bias and the limitations of deterministic policies in stochastic environments, leading to gradient explosion early in training. Meanwhile, MASAC suffered from initial overfitting followed by a collapse into sub-optimal local safety policies. The failure of MADDPG and the instability of MASAC highlight a critical theoretical insight: in highly dynamic scenarios with random user mobility, off-policy methods (MASAC, MADDPG) significantly underperform on-policy methods (IPPO, MAPPO, MAPPO-DC). This is primarily because the historical experiences stored in the replay buffer become obsolete rapidly due to environmental non-stationarity (distributional shift), causing the critic to learn from inaccurate state distributions. In contrast, on-policy methods, which optimize based on fresh samples from the current policy, demonstrate superior adaptability. Consequently, MADDPG is excluded from Figure 10, and a detailed analysis of both algorithms is provided in Appendix A.

In scenarios where strong agent coordination is not required, MAPPO suffers from credit assignment ambiguity. That is, when agents i and j execute their respective actions and the critic receives positive feedback, it cannot precisely determine whether the positive feedback originated from agent i or agent j. As seen in the comparison between the MAPPO-DC and MAPPO curves in Figure 10, their reward, TP, collision, and out-of-bounds curves are nearly identical. However, the crash curve for MAPPO-DC decreases and converges to zero much faster than that of MAPPO. This indicates that the decoupled critics provide a clearer and more direct penalty signal for unsafe actions, enabling the agents to learn safe operational boundaries more quickly. The crash and PF curves further reveal that by rapidly pruning the vast unsafe regions of the state-action space, MAPPO-DC allows agents to focus their exploration on promising areas for performance optimization. This leads to the initial faster improvement in PF seen in Figure 10. Furthermore, MAPPO-DC achieves a higher final PF than MAPPO, confirming that the decoupled critic design successfully achieves our objective of maximizing individual agent performance while maintaining necessary coordination.

To quantitatively evaluate the efficacy of credit assignment mechanisms, we introduce the Variance of Advantage Estimates as a metric. In MARL, high advantage variance typically indicates ambiguous credit assignment, where an agent’s value estimation is heavily severely perturbed by the non-stationary behaviors of others (noise). Conversely, a lower variance implies that the critic has successfully isolated the agent’s specific contribution from the environmental noise, leading to more stable and precise policy updates.

As can be seen from Figure 11, the mean and standard deviation of the advantage variance of MAPPO-DC are the lowest, which confirms that the decoupled critic architecture can effectively isolate the specific contributions of each agent in loosely coupled scenarios, reducing the noise introduced by the dynamic behavior of other agents. The advantage variance of MAPPO is higher than that of MAPPO-DC, indicating that centralized critics introduce coupling noise when dealing with non-stationarity, that is, the value estimation of agents is sensitive to the action fluctuations of other agents; IPPO exhibits relatively low advantage variance, but its error bars show significant dispersion, indicating that although agents often adopt strategies that ignore interactions (resulting in low variance in steady states), they exhibit severe instability when inevitable conflicts occur, as evidenced by the large lower limit of the error bars; MASAC exhibits the highest advantage variance because it is a maximum entropy algorithm designed to encourage exploration, and compared to trust region-based methods such as PPO, its policy randomness and non-strategic nature of value estimation introduce higher variance in advantage computation.

In conclusion, the simulation experiment results validate our core hypothesis: the decoupling critic mechanism in MAPPO-DC effectively solves the problem of credit assignment failure caused by the introduction of high variance noise in IPPO, as well as the problem of credit assignment ambiguity in the MAPPO centralized critic, which suppresses individual performance. The MAPPO-DC algorithm achieves a more efficient and secure learning process without affecting the necessary coordination of multi-agent tasks. In addition, we compared the experimental results of MASAC and MADDPG and found that the off-policy-based algorithms (MASAC, MADDPG) performed worse than the on-policy-based algorithms (MAPPO-DC, MAPPO, IPPO) in highly dynamic random environments.

4.6. Model Robustness Experiment

To validate the robustness of the proposed algorithm under severe uncertainty and explore its applicability boundaries, we conducted additional experiments with enhanced stochasticity. Note that MADDPG is excluded from this analysis as it failed to converge even in the baseline scenario (Section 4.5); evaluating it in this more complex environment would yield no additional analytical value. We introduced the following three major environmental stressors:

• Co-channel Interference: Switching from OFDMA to a full frequency reuse scheme, where all UAVs share the entire spectrum, thereby introducing strong inter- UAV interference.
• Heterogeneous Bandwidth: Replacing the equal bandwidth division with an inverse-distance-weighted allocation scheme (an edge-user protection strategy) to simulate uneven resource distribution.
• Energy Consumption Noise: Adding Gaussian perturbations to the energy consumption model to mimic unmodeled external disturbances (e.g., wind gusts).

As illustrated in Figure 12, despite the increased environmental uncertainty, both MAPPO and MAPPO-DC successfully converged, demonstrating inherent robustness. The converged system throughput is lower compared to the interference-free scenario (Section 4.5). From the user’s perspective, although full frequency reuse increases the available bandwidth per user, the SINR degrades significantly due to severe co-channel interference. This is particularly detrimental to users at the cluster edges, who suffer from high interference power from neighboring UAVs, leading to a net reduction in TP and PF.

A critical observation from the crash curves (comparing Figure 10 and Figure 12) is that with energy noise, the number of episodes required for crash zeroing increased by approximately 15 k. This indicates the model needs more time to learn a “safety margin”—effectively reserving extra battery to buffer against $3\sigma$ noise fluctuations. Comparing this to the MAPPO-DC-T+Convex-P scheme (Section 4.4) reveals a key insight: In this experiment, the noise follows a statistical distribution, which the critic network can approximate, allowing the actor to learn a robust margin. In contrast, the convex optimizer in Section 4.4 behaves as a non-stationary or adversarial “black box”—slight position changes cause drastic, non-linear jumps in power consumption. This structural unpredictability is harder to adapt to than statistical noise, explaining why the hybrid convex scheme failed to eliminate crashes completely, whereas the end-to-end MAPPO-DC succeeded here.

The trends for MASAC and IPPO remain consistent with Section 4.5. MASAC exhibits instability (overfitting then local optima), and IPPO sacrifices safety (high crashes) for short-term reward gains.

In this strongly coupled scenario, MAPPO and MAPPO-DC achieved similar converged performance (MAPPO-DC: TP ≈ 23.80 Mbps, PF ≈ 357.61; MAPPO: TP ≈ 23.98 Mbps, PF ≈ 356.34). This similarity arises because both algorithms utilize global states for value estimation, enabling global coordination. However, MAPPO-DC achieved crash zeroing significantly faster than MAPPO. This confirms that even under strong coupling, the decoupled critic architecture provides clearer credit assignment for individual safety constraints, accelerating the learning of safe flight policies.

Figure 13 presents the Advantage Variance in the strongly coupled scenario. Compared to the loosely coupled scenario (Figure 11), the advantage variance for all algorithms increased. This confirms that co-channel interference elevates the coupling strength, making the system performance more dependent on complex multi-agent coordination rather than individual actions, thereby blurring the credit assignment. IPPO exhibits the lowest variance. However, this does not indicate superior credit assignment. Instead, it reflects a trivial solution: to mitigate interference penalties without coordination capabilities, IPPO agents learned a “spatial repulsion” strategy—staying far apart to minimize interaction. This artificially reduces the variance of interaction terms. Consistent with previous results, MASAC shows the highest variance, confirming its poor robustness against uncertainty.

Unlike the loosely coupled case, MAPPO-DC shows slightly higher variance than MAPPO here. This reveals the following trade-off of decoupling:

• Smoothing Effect of Centralization: MAPPO’s centralized critic predicts a unified team reward, effectively averaging out individual fluctuations caused by intense interference, resulting in a statistically smoother estimate.
• Challenge of Decoupling: MAPPO-DC attempts to isolate individual contributions from this highly entangled signal. In strong coupling, this “forced decoupling” is computationally challenging and more sensitive to interference noise, leading to slightly higher variance.

This explains why MAPPO-DC’s PF is marginally lower (due to variance-induced gradient noise), but its safety learning remains superior (as safety constraints remain largely local).

Comparing the loosely coupled (Section 4.5) and strongly coupled (Section 4.6) scenarios allows us to delineate the following clear operational boundaries:

• MAPPO (Centralized Critic): Optimal for Interference-Limited Scenarios where maximizing system capacity via complex coordination is the sole priority. If the mission tolerates slower safety convergence to squeeze out the absolute maximum throughput, standard MAPPO is preferable.
• MAPPO-DC (Decoupled Critic): Dominates in Safety-Critical and Rapid-Response Missions. It offers comprehensive advantages in loosely coupled tasks and remains highly competitive in strongly coupled settings. Its ability to swiftly learn collision-avoidance policies (fast Crash Zeroing) makes it the robust choice for real-world deployments, offering the best trade-off between optimality, safety, and training efficiency.

5. Conclusions

This paper addresses the challenge of energy-constrained and fairness-aware resource allocation within multi-UAV emergency communication networks. We proposed a hierarchical framework centered around MAPPO-DC, a novel MARL algorithm specifically designed for loosely coupled task structures, which arise from decoupling user association from trajectory and power control. The framework employs a lightweight clustering algorithm for efficient user assignment, while MAPPO-DC enables UAV agents to learn customized control policies. The algorithm successfully maximizes the system’s PF throughput while ensuring the necessary coordination for the multi-agent task, providing a resource allocation strategy that balances both efficiency and fairness for energy-limited emergency communication networks.

Multiple simulation experiments have drawn the following conclusions: Firstly, the joint optimization of trajectory and power control is essential for reliable operation. When decoupling is introduced into the methodology, the uncontrolled subsystem acts as an irreducible “black box,” inevitably leading to credit assignment failures and persistent crashes. Secondly, our proposed MAPPO-DC algorithm significantly outperforms standard benchmarks such as IPPO and MAPPO. It successfully avoids the non-stationarity pitfalls of independent learning (IPPO) while demonstrating superior sample efficiency and performance compared to the overly centralized value learning of standard MAPPO. This validates the effectiveness of specialized credit assignment for loosely coupled tasks. Finally, our robustness analysis across varying coupling levels reveals a clear operational boundary: MAPPO-DC demonstrates superior efficacy and safety convergence in loosely coupled environments, while retaining high adaptability and competitive performance even when subjected to strong co-channel interference. This validates MAPPO-DC as a robust, safety-aware solution suitable for diverse dynamic communication missions.

Future work can extend this research in several directions. The energy consumption model could be refined to incorporate more complex factors such as wind and acceleration. Additionally, the inclusion of Vehicle-to-Vehicle (V2V) communication could be explored as a means to introduce more explicit coordination, further enhancing cooperative efficiency.