CCTD-MARL: Coupled Communication-Task Decoupling Framework for Multi-Agent Systems Under Partial Observability

Abstract

Although multi-agent reinforcement learning (MARL) has achieved significant success in various domains, its deployment in real-world scenarios remains challenging, particularly in communication-constrained environments involving multi-task coupling. Existing methods suffer from two limitations: (1) the inability to effectively integrate and process incomplete state from disparate agents, and (2) a lack of robust mechanisms for handling complex multi-task coupling. To address these challenges, we propose the Coupled Communication-Task Decoupling (CCTD) framework. CCTD introduces two critical innovations: first, a distributed state compensation mechanism to process historical data, thereby reconstructing accurate global states from partial observations; second, a hierarchical architecture that systematically decomposes complex tasks into manageable subtasks while preserving their interdependencies. Thanks to its modular design, CCTD can integrate with existing MARL algorithms and allow for flexible combination of various subtasks. Extensive experiments demonstrate that CCTD outperforms baseline methods, achieving a 10% improvement in communication reception rate and superior performance across all subtasks in multi-task environments.

1. Introduction

Multi-agent reinforcement learning (MARL) [1] has achieved success in solving complex problems across diverse domains, including autonomous robots and multiplayer game scenarios [2,3,4,5]. Despite these advancements, the practical implementation of MARL is still hindered by two challenges [6].

The first challenge is to solve the non-stationarity [7] in multi-agent environments. The Centralized Training Decentralized Execution (CTDE) [8] framework is commonly adopted to mitigate this issue by coordinating a joint policy during training and execution in a decentralized manner. Therefore, each agent relies on local observations and incomplete communication information [9] to make decisions. This limitation frequently leads to suboptimal decision-making because agents may lack the critical information needed to accurately assess the global state and select the best action, making it difficult for agents to cooperate with others. In cooperative settings, information exchange becomes essential to bridge this gap, enabling the aggregation of localized information and fostering effective inter-agent collaboration [10,11].

The second challenge we face is how to manage complex multitasking scenarios [12]. Traditional MARL algorithms are designed mainly to handle single-task scenarios. When applied to complex multi-task environments, they often encounter problems such as low learning efficiency, high computational resource requirements, and poor scalability due to high-dimensional state and action spaces [13,14]. They have difficulty decomposing tasks and making coordinated decisions effectively. Hierarchical reinforcement learning (HRL) [15] demonstrates the potential to solve these problems through its hierarchical architecture, which decomposes decision-making tasks into multiple levels, bridging high-level abstract goals with low-level specific actions [16]. Agents can learn independently at different levels, reducing the dimensionality of search space and accelerating convergence [17].

In this study, the term “coupled multi-task” denotes scenarios where subtasks (e.g., search and delivery) exhibit interdependence: the information acquired from the search task dictates the target of the delivery task, while the execution of delivery modifies the spatial distribution of agents, thereby influencing the efficiency of subsequent search processes. Formally, for a set of subtasks $\mathcal{Z}=\{z_1,z_2,\dots ,z_k\}$ in a multi-agent system, the coupling relationship is defined by the following two key properties:

1.

State Dependence: The state of subtask $z_i$ at time t, denoted as $s_{z_i}\left(t\right)$, is a function of the state of other subtasks ${\left\{s_{z_j}(t-1)\right\}}_{j\ne i}$ and the joint action $\mathbf{a}(t-1)=(a_1(t-1),\dots ,a_n(t-1))$ executed by all agents at time $t-1$, i.e., $s_{z_i}\left(t\right)=f_i({\left\{s_{z_j}(t-1)\right\}}_{j\ne i},\mathbf{a}(t-1))$, where $f_i$ is a task-specific transition function.

2.

Reward Coupling: The cumulative reward of the system $R_{\mathrm{total}}={\sum }_{t=0}^{\infty }{\gamma }^{t}r\left(t\right)$ depends on the combined performance of all subtasks, with the immediate reward $r\left(t\right)=g\left({\left\{r_{z_i}\left(t\right)\right\}}_{i=1}^k\right)$ being a non-separable function of the individual subtask rewards $\left\{r_{z_i}\left(t\right)\right\}$, where g denotes the reward aggregation mechanism.

Distinct from standard hierarchical reinforcement learning (HRL), which treats subtasks as independent modules, the proposed CCTD preserves these interdependencies via a shared State Compensation Layer. This design ensures that high-level state transitions are guided by reconstructed global states rather than noisy belief distributions.

To address these issues, we introduce the Coupled Communication-Task Decoupling (CCTD). Our contributions include the following:

• We propose a dynamic state compensation framework that enables agents to reconstruct accurate global states from partial observations and incomplete communication information.
• We design a multi-agent hierarchical architecture that effectively decomposes complex, coupled tasks while maintaining inter-task dependencies and achieving efficient coordination.
• We provide a flexible integration mechanism that allows for flexible combination with existing MARL algorithms and specific task solutions.

2. Background

2.1. Multi-Agent Reinforcement Learning

MARL extends traditional reinforcement learning to environments with multiple agents. Problems are typically formalized as a decentralized partially observable Markov decision process (Dec-POMDP) [18,19]. It can be defined by the tuple $\langle I,S,\left\{A_i\right\},P,\left\{O_i\right\},R,\gamma \rangle$, where I denotes the set of agents, S is the global state space, $\left\{A_i\right\}$ is the action spaces, $\left\{P\left(s^{\prime }\right|s,a)\right\}$ is the transition dynamics, $\left\{O_i\right\}$ is the local observation spaces, $R(s,a)$ is the shared reward function, and $\gamma$ is the discount factor. All agents have the same objective, which is to learn a joint policy $\pi =({\pi }_1,\dots ,{\pi }_n)$ that maximizes the expected reward $\underset{\pi }{max}{E}_{\pi }\left[{\sum }_{t=0}^{\infty }R(s_t,a_t)\right]$.

A key challenge in MARL, different from single-agent RL, is environmental nonstationarity; the Markov property fails to hold due to the policy of other agents. To address these issues, the Centralized Training with Decentralized Execution (CTDE) framework leverages global information during training while maintaining decentralized policies. As a classic CTDE-based approach, the QMix [20] algorithm has established strong performance in multi-agent scenarios. Building on the QMix architecture, the Agent Permutation Invariant Network (API) [21] introduces further advancements: it retains the core QMix network structure while fully leveraging its inherent permutation equivariance and permutation invariance properties. These properties not only effectively reduce the dimensionality of input data but also significantly accelerate the convergence rate of the training process [22].

2.2. Hierarchical Deep Reinforcement Learning

Hierarchical deep reinforcement learning (HDRL) combines the representational power of deep learning with the task abstraction capabilities of hierarchical reinforcement learning. Foreign studies have combined the advantages of hierarchical reinforcement learning and deep learning to propose multiple algorithms such as the Hierarchical Deep Q-Network (h-DQN) [23], Hindsight Experience Replay (HER) [24], and Hierarchical Actor-Critic (HAC) [25]. These algorithms solve problems in large-scale spaces and with sparse rewards and improve the ability to learn strategies for task completion. Domestic studies focus more on applications. Methods such as path planning for UAV swarms, intelligent traffic signal control, and multi-agent adversarial strategy generation in military systems have been proposed. These methods optimize decision-making and communication efficiency through hierarchical reinforcement learning.

2.3. Multi-Agent Communication Architectures

Effective communication architectures play an important role in cooperative multi-agent systems, which leads to more stable algorithm performance. Although existing MARL algorithms have shown promising results in some environments, they face challenges when deployed in complex scenarios. These algorithms are typically validated in relatively ideal simulation environments, but they lack adaptability and robustness in complex dynamic environments, especially when facing challenges such as high communication delays and information loss. Therefore, they are not fully applicable to the problems we need to solve.

Differentiable Inter-Agent Learning (DIAL), Reinforced Inter-Agent Learning (RIAL) [26] and CommNet [27] were among the early works to introduce communication in the field of multi-agent deep reinforcement learning. Taking DIAL as an example, since decentralized training methods (such as Independent Q Learning) face non-stationarity problems when dealing with Dec-POMDP problems, centralized training and communication between agents are adopted to solve the Dec POMDP problem. The intuitive solution is the RIAL (Reinforced Inter-Agent Learning) method. RIAL combines DRQN with Independent Q Learning to select actions and communication messages. In the agent’s policy network, in addition to generating actions in the usual sense, the generated messages are also considered as discrete actions. At the same time, the learned messages are explicitly transmitted among agents to enhance the agents’ perception of the environment, thereby overcoming the problem of environmental non-stationarity to a certain extent. In terms of training, RIAL still uses the non-centralized training method of Independent Q Learning, where each agent is trained based on the error of its Q-network.

3. Methodology

In this section, we present an approach to address complex decision-making in multiple-task scenarios under the limited communication condition. The scenarios targeted by this method mainly involve the collaborative tasks of intelligent agent clusters in environments with communication limitations. As shown in Figure 1, this approach consists of two crucial components: the hierarchical reinforcement learning module integrated with the API algorithm and the communication optimization module, which address multi-task decision-making and optimize communication, respectively.

3.1. Hierarchical Reinforcement Learning

To mitigate the exponential growth of the state-action space in homogeneous multi-agent systems, we integrate the API architecture [21] into our framework. This integration is tailored to address the core challenges of our scenario: 12 homogeneous UAVs performing coupled search-delivery tasks under communication constraints, where agent input ordering is arbitrary but information redundancy must be eliminated.

Our task context (12 homogeneous UAVs, coupled subtasks, communication latency) makes API a critical component for three key reasons: 1. Eliminate Redundant Search Space: For 12 homogeneous agents, there are $12!$ (479 million) distinct input orderings that carry identical information. Standard MLPs treat these orderings as unique, leading to enormous computational waste and slow convergence—critical issues in time-sensitive logistics tasks. 2. Ensure Consistent Decision-Making: Coupled subtasks (search guiding delivery, delivery altering search conditions) require stable state representations. API’s permutation invariance ensures that agent input order does not affect decision outputs, preventing erratic behavior when switching between subtasks. 3. Adapt to Dynamic Communication: Communication latency often causes asynchronous state updates (agents receive partial, out-of-order data). The API’s ability to aggregate information independent of input sequence ensures robust state fusion, complementing our RNN-based state compensation module. Specifically, the state compensation framework aligns asynchronous, delayed local observations into a unified global state $S_{global}^t$, which eliminates the noise caused by communication latency. This compensated global state is then fed into the API-HPN architecture, where the permutation-invariant aggregation ensures that the order of agent inputs does not affect the final state representation. The synergy between these two components addresses the core challenge of partial observability: the state compensation framework resolves information incompleteness caused by communication constraints, while the API eliminates redundant computations caused by agent input permutations. Together, they enable efficient and consistent decision-making for coupled multi-task scenarios. To balance permutation invariance and representational capacity, we adopt a Hypernet-based API (API-HPN) with the following three key steps:

• Hyperweight Generation: A shared Hypernetwork takes each agent’s spatiotemporal feature $x_i$ (position, velocity, orientation) as input, generating customized weights $W_i=\mathrm{HyperNet}\left(x_i\right)$. This adapts to individual agent states while maintaining homogeneity.
• Invariant Aggregation: The input layer computes latent embeddings via symmetric summation: $e={\sum }_{i=1}^{12}(x_i·W_i)+b$. Symmetric summation ensures e is unchanged by agent input permutations, eliminating redundant computations.
• Equivariant Mapping: For entity-specific actions (e.g., assigning delivery targets to UAVs), submodular weights are remapped to the original agent order, Achieving Permutation Equivariance (APE). This preserves action-agent correspondence while retaining invariant aggregation benefits.

By embedding API into both the high-level task chooser (subtask selection) and low-level policies (action execution), CCTD reduces the effective input space by a factor of $1/12!$, drastically accelerating convergence in coupled, communication-constrained scenarios.

Our proposed framework employs a hierarchical reinforcement learning (HRL) methodology to address multi-task coupled decision-making in constrained communication environments. The model consists of three parts: the hierarchical learning structure, the API algorithm integration, and the environmental interaction interface.

We formalize the hierarchical architecture as a Markov Decision Process (MDP). The high-level controller (HLC) operates at fixed discrete time steps $T=\{0,1,2,\dots \}$, where its core responsibility is to select a subtask z from the task set $\mathcal{Z}=\{\mathrm{Search},\mathrm{Delivery}\}$. At each decision step $t\in T$, the HLC observes the local observation information of all agents, denoted as $o_i$ for agent i, and makes a subtask selection based on the aggregated local observations. During the training phase, the HLC is trained on top of the pre-trained low-level controller (LLC), with the objective of maximizing the cumulative long-term reward. The detailed training procedure of the HLC is presented in Algorithm 1.

Algorithm 1 Training Procedure of HLC Based on Pre-trained LLC (MDP Formulation)

1: Initialization: 2: Pre-train Low-level Controller (LLC) until convergence 3: Initialize High-level Controller (HLC) parameters ${\theta }_{\mathrm{HLC}}$ randomly 4: Define task set $\mathcal{Z}=\{\mathrm{Search},\mathrm{Delivery}\}$, fixed discrete time steps $T=\{0,1,2,\dots \}$ 5: Set total training episodes $E_{\mathrm{total}}$, maximum steps per episode $t_{\max }$ 6: Initialize replay buffer $\mathcal{B}$ for HLC 7: for episode $e=1$ to $E_{\mathrm{total}}$ do 8:     Reset environment and agents, get initial local observations $\{o_1^0,o_2^0,\dots ,o_N^0\}$ (N is number of agents) 9:     Set current high-level time step$t=0$ 10:     while$t<t_{\max }$ do 11:           HLC Decision Making: 12:           Aggregate local observations $O^t=[o_1^t,o_2^t,\dots ,o_N^t]$ 13:           HLC selects subtask $z^t\sim {\pi }_{{\theta }_{\mathrm{HLC}}}\left(z\right|O^t)$ (policy $\pi$) 14:           Assign $z^t$ to all agents, LLC executes subtask $z^t$ for one fixed time step 15:           Move to next high-level time step $t\leftarrow t+1$ 16:           Get new local observations $\{o_1^t,o_2^t,\dots ,o_N^t\}$ after subtask execution 17:           Calculate immediate reward $r^t$ based on subtask performance 18:           Store transition $(O^t,z^t,r^t,O^{t+1})$ into $\mathcal{B}$ 19:     end while 20:     HLC Update: 21:     Sample mini-batch of transitions from $\mathcal{B}$ 22:  Compute cumulative long-term reward $R={\sum }_{k=t}^{t_{\max }}{\gamma }^{k-t}{r}^k$ (discount factor $\gamma$) 23:  Update ${\theta }_{\mathrm{HLC}}$ by minimizing the negative expected reward ${\nabla }_{{\theta }_{\mathrm{HLC}}}\mathbb{E}\left[R\right]$ 24: end for 25: Output trained HLC and pre-trained LLC as the CCTD-MARL hierarchical controller

In the high-level controller of hierarchical reinforcement learning, a task-decision agent is designed to select a specific task; it is responsible for determining high-level goals based on the environmental state but does not directly output actions to interact with the environment, thereby supporting the switching and selection of subtasks.

In the low-level controller, agents are built for executing the specific tasks selected by the high-level controller. They are responsible for generating actions and directly controlling the movement of the agents.

Both high-level and low-level controllers are required to possess decision-making capabilities tailored to different granularities. To this end, we leverage API algorithms and optimization rules to enhance the overall control performance. The specific details of the API algorithm have been elaborated in the preceding section.

For the part that interacts with the environmental platform, we customize a multi-task simulation environment. Distinct action interfaces are configured for different tasks to control the agents. This modular design supports a plug-and-play capability, allowing for the flexible construction of the following multi-task experimental scenarios:

• Low-Level Controller

The low-level controller is designed to generate specific policies for different subtasks. For each subtask, the task executor constructs the corresponding state and action space by analyzing both the subtask requirements and available resources. The task executor can load the most suitable reinforcement learning (RL) algorithm or optimization solution to perform action selection. Through this modular approach, the low-level controller effectively handles the diverse and complex demands of various subtasks in the system.

The task scenario modeled in this paper is a logistics distribution task, where the entities on the map are divided into logistics delivery agents and material-receiving vehicles. We focus on controlling multiple logistics delivery agents that must perform search and material delivery tasks at different points. Under ideal conditions, the agent cluster should autonomously handle collaborative search and material allocation. To address the complexity of this scenario, we decompose it into two distinct components: collaborative search and material allocation, and construct separate solutions for each.

• High-level controller

At the upper layer of subtask execution, we model the high-level controller as a decision-making and task assignment framework. This framework involves a painstaking process of refining and integrating relevant state information, which encompasses details about the agent’s state, such as their current positions, available resources, and task goals, including time constraints or material requirements.

The agent leverages the information to train a set of low-level task policies. Upon completion of the low-level task training, the agent further utilizes the global information to train a high-level selector, which is designed to select the optimal low-level task for the current system state. This hierarchical decision-making framework provides strategic guidance for the coordinated operation of the entire system, thereby enabling the agent to efficiently converge toward the overarching task objectives.

3.2. State Compensation Framework

In multi-agent scenarios, communication latency often leads to incomplete information, resulting in interruptions and information loss, and causes agents to perceive the states of other agents and environmental entities at temporally misaligned timesteps. To address this asynchrony, we develop a state prediction framework that aligns all positional information to a unified timestep. In this paper, we design a virtual validation multi-agent environment to verify the performance of our proposed CCTD, where a random communication delay of 0–10 s is present.

In scenarios with low latency, we construct a historical state completion module, where the delayed information is filled by using historical location data. The model offers a simplified approach to dealing with latency. Specifically, it enables direct completion relying on historical states, and the current state information is complemented by the agents’ historical state. The historical state completion mechanism can directly call historical data to update the current state. It is a degradation processing method with low computational resources and is suitable for scenarios with low latency and limited computational resources of the model. Through this completion method, the agents can still maintain the consistency of the state without the support of the model.

In scenarios involving high latency or data loss, we employ an RNN-based module for state prediction. Input to the RNN: The module takes a concatenated sequence of two types of time-series data as input: (1) own history: the agent’s own past 10 timesteps of position, velocity, and orientation in the global Cartesian coordinate system; (2) received states: the delayed, timestamped position and target information received from other agents. Output of the RNN: The output is the predicted current position and orientation of the target agent, represented in the same global Cartesian coordinate system used for input. This ensures seamless alignment with the unified timestep of the state prediction framework. Training loss function: We train the RNN using Smooth L1 Loss between the predicted position and the ground-truth position obtained from real-time synchronized data. The RNN module adopts a 2-layer LSTM architecture with 128 hidden units per layer, followed by a fully connected layer (64 neurons, ReLU activation) and an output layer (6 neurons for position and orientation prediction). We use layer normalization after each LSTM layer to stabilize training, and apply dropout (rate = 0.2) to the fully connected layer to prevent overfitting. This loss function is robust to outliers and provides stable gradient updates, which is critical for handling noisy communication data. Policy change impact: Changes in agent policies (e.g., switching from “Search” to “Delivery”) do not affect the state completion process, as the state completion mechanism is purely based on spatiotemporal positional information and is independent of subtask types. This ensures the stability and generality of the state prediction module across different task modes.

This module is trained on historical trajectory data to accurately predict missing states and estimate the current positions of agents, thereby ensuring the integrity of global information. The training dataset consists of 100,000 trajectory sequences collected from real-world UAV logistics simulations, with a sampling frequency of 1 timestep. Each sequence includes 1000 consecutive timesteps of spatiotemporal data (position: x/y/z coordinates in the global Cartesian coordinate system; velocity: x/y/z components) for one UAV. We split the dataset into training (80%), validation (10%), and test (10%) sets, with data augmentation applied via Gaussian noise injection (standard deviation = 0.01) to position and velocity features to enhance the robustness of the model. When entering the latency processing mode, the model activates the RNN-based model, which processes historical trajectory to complete the state representation, ensuring the accuracy of the agent’s state information in a high-latency environment. The training dataset comes from historical records, including agent positions and execution results. The actual data is formed by collecting real-time position information during actual task execution processes. According to the hierarchical task instructions issued, these instructions specify what each agent will do, making the trajectory of the agent a clear target to follow at the current target point, preventing the trajectory volatility caused by directly predicting the change in the agent’s trajectory target, which may lead to inaccurate situation prediction.

Supported by this prediction module, each agent broadcasts its timestamped position and target information. The framework then synchronizes asynchronous data by extrapolating positions from time $t-n$ to the unified current timestep t. These synchronized states, which specifically include the prediction of agents and the prediction of vehicles, serve as inputs for both the hierarchical and low-level decision-making networks. By leveraging RNN for state compensation, the framework effectively processes time-series data to enhance localization accuracy. This ensures robust state awareness and maintains task performance despite communication delays. Algorithm 2 shows the overall algorithm flow.

To ensure the coordination and efficiency of the overall task, the HRL structure implements a “Context-Aware” coupling mechanism. The high-level controller (HLC) outputs a current observation vector $o_i$, which is fed into the low-level controller (LLC) along with the compensated state. This ensures that when switching from ‘Search’ to ‘Delivery’, the momentum and environmental context are preserved.

Algorithm 2 Distributed State Alignment and Position Prediction

Input: $\mathcal{A}=\{a_1,\dots ,a_n\}$—Set of agents; t—Target timestamp (global Cartesian coordinate system); D—Historical trajectory buffer (stores $\{s_k,t_k,v_k\}$ for each agent, where $s_k$ = position, $t_k$ = timestamp, $v_k$ = velocity); ${\delta }_{thresh}$—Delay threshold (unit: s); N—Length of historical sequence for RNN input; d—Dimension of agent state vector (e.g., $d=6$ for x/y/z coordinates, x/y/z velocity) Output: $S_{global}^t$—Time-aligned global state matrix of size $n\times d$ (global Cartesian coordinate system) 1: Initialize $S_{global}^t$ as an empty matrix of size $n\times d$   ▹ Initialize global state matrix 2: for each agent $a_i\in \mathcal{A}$ do 3:     Retrieve last received state tuple $(s_{t_{last}},t_{last}^i,v_{last})$ from D for $a_i$ 4:  Boundary Check: If no historical data exists in D for $a_i$, set ${\widehat{s}}_i^t=\mathbf{0}$ and skip to Line 13                ▹ Handle first communication 5:     Calculate communication delay: $\Delta t_i=t-t_{last}^i$ ▹ Time difference between target and last received timestamp 6:     if $\Delta t_i<{\delta }_{thresh}$ then 7:         ${\widehat{s}}_i^t\leftarrow s_{t_{last}}+v_{last}·\Delta t_i$   ▹ Linear dead reckoning (vector operation in global Cartesian system) 8:     else 9:         $H_i\leftarrow$ Extract historical sequence ${\left\{(s_{t-k},v_{t-k})\right\}}_{k=0}^N$ from D for $a_i$ ▹ Get past N timesteps of position/velocity 10:          Padding Check: If length of $H_i<N$, pad with initial state $\mathbf{0}$ to length N 11:        ${\widehat{s}}_i^t\leftarrow \mathrm{RNN}\left(H_i\right)$ ▹ RNN prediction (only spatiotemporal data, independent of subtask type) 12:     end if 13:     $S_{global}^t[i,:]\leftarrow {\widehat{s}}_i^t$             ▹ Assign predicted state to global matrix 14: end for 15: return $S_{global}^t$                 ▹ Time-aligned global state matrix

4. Experiments

In this section, our trial evaluation is conducted by comparing the basic multi-agent reinforcement learning algorithm to understand the performance effect of our method in the multi-task coupling scenario in a restricted communication environment. We present three types of reinforcement learning experiments to verify the effectiveness of CCTD from various perspectives. The first type compares CCTD with classical multi-agent reinforcement learning algorithms without communication optimization, the second compares CCTD with classical multi-agent reinforcement learning algorithms without hierarchical reinforcement learning, and the third proposes a set of ablation experiments to evaluate different components of CCTD. In the experiment, we used the following indicators to measure the performance effect of the algorithm: environmental reward, task target achievement degree, communication-optimized communication compensation rate, and communication compensation accuracy rate.

The agent can be rewarded for taking actions to interact with the environment. With the correct reward design, actions that benefit the task goal will be more rewarded. Therefore, the environment reward can measure the action of the agent during the task. The degree of achievement of the mission target is the task completion degree of the UAV in the search task and material delivery task. The communication compensation rate refers to the compensation effect of the UAV after communication optimization. The communication compensation accuracy is the gap between the data obtained in the communication optimization and the real data.

4.1. Environments

The experiment environment is a self-defined simulation scenario. In this scenario, there are five supply receiver vehicles on the ground. Among receiver vehicles, some are stationary while others are in motion, and their locations are unknown to the drones. The twelve logistics drones are manipulated by the algorithm to collaboratively complete the search and material delivery tasks, delivering supplies to the vehicles. At the same time, the environment can simulate multiple communication environments. In the low-interference environment, there is a 0–5 s communication delay. In the high-interference environment, besides a 0–10 s communication delay, there also exists communication disruption. In a high-interference environment, communication interference devices around vehicles will be activated, disturbing the drones’ communications and the disturbed drones will be disconnected from the drone communication network, unable to send or receive messages. The closer the drone is to the interference devices, the higher the probability of communication interruption. Additionally, the communication between any two drones may be lost if the distance between these two drones is too far. In this case, they cannot receive each other’s messages. But they can receive messages from other drones and also can send messages to other drones. All simulations and training processes were conducted in strict accordance with the hyperparameter settings presented in

Table 1. Simulation and training hyperparameters.

Parameter	Value	Parameter	Value
Map Size	$1000\times 1000$ m	Discount Factor ($\gamma$)	0.99
Number of UAVs	12	Batch Size	32
Communication Range	200 m	RNN Hidden Units	128
Delay Dist. ($\Delta t$)	$\mathrm{Uniform}(0,10)$ s	Learning Rate	$5\times {10}^{-4}$
Reward (Search)	$+2.0$ per area	Reward (Delivery)	$+10.0$
Step Penalty	$-0.1$	Optimizer	Adam

.

• Reward Settings

During the process of an agent’s actions in the environment, the agent can receive little reward when exploring positions that have not been detected. Once the agent discovers a logistics receiving vehicle, it also obtains corresponding rewards. When the agent delivers supplies successfully, it can receive a relatively substantial reward. In order to encourage the agent to complete the task in a short time, the system has set up a penalty mechanism for each step. As the number of action steps continues to increase, it accumulates gradually to motivate the agent to achieve the task goal as soon as possible.

4.2. Baseline

• MAPPO [28]: A MARL algorithm based on PPO (Proximal Policy Optimization), belonging to the policy gradient method. It is suited for complex policy and continuous action space scenarios. As a non-communication, non-hierarchical baseline, it serves to verify the necessity of communication optimization and task decomposition in coupled multi-task scenarios.
• QMix [20]: A value-decomposition MARL algorithm, designed for complex team tasks in discrete action spaces. It adopts a centralized training and decentralized execution paradigm but lacks explicit communication mechanisms and hierarchical task management, acting as a baseline for validating the effectiveness of our proposed communication compensation and hierarchical architecture.
• QMix-communicate MARL: An improved version of QMix with an explicit communication mechanism based on CommNet [27]. Each agent encodes its local observation (position, velocity, orientation, and partial environmental information) into a 32-dimensional feature vector, which is explicitly transmitted to all other agents through a fully connected communication network. The received messages are aggregated with the agent’s own observation via element-wise summation, enhancing the global state awareness of each agent. This baseline is used to compare the performance of different communication optimization strategies.
• QMix-Hierarchical MARL: A hierarchical extension of QMix, built on the Hierarchical Deep Q-Network (h-DQN) [23] architecture. It decomposes the complex coupled task into two independent subtasks (search and delivery) at the high level, with a meta-controller selecting subtasks and a low-level controller executing specific actions for each subtask. The meta-controller and low-level controllers are both trained using the QMix value-decomposition framework, but lack mechanisms to preserve subtask interdependencies and handle communication latency. This baseline is used to verify the advantages of our CCTD framework in maintaining task coupling and adapting to communication-constrained environments.

In our experiments, we tested four baselines and the CCTD proposed in the same environment to evaluate their performance. Agents interact with the environment and receive rewards based on their actions. We plot the final reward of each episode on the y axis and the episode number on the x axis. This visualization allows us to observe the learning progress of agents over time.

Figure 2 shows the performance of CCTD and the other four baselines in the same environment. We can see that the MAPPO and QMix show similar performance; the performance of these two baselines is the worst. From this, we can see that traditional reinforcement learning algorithms such as MAPPO and QMix are inadequate in handling coupled multi-tasks like those in this environment. QMix-com adds explicit communication based on QMix; its performance is better than the previous baselines. Furthermore, agents’ acquired rewards transition from negative to positive in the final stage of training, indicating that clear communication plays a certain role in improving the performance of the agent. QMix-hir combines the hierarchical architecture with the QMix algorithm; its performance is inferior only to CCTD, and its reward is greatly improved compared with the previous three baselines. When dealing with coupled multi-tasks, the hierarchical architecture can decouple tasks. Therefore, QMix-hir achieves good performance in this scenario. Finally, we can see that the performance of the CCTD is the best. This is because CCTD not only has a hierarchical architecture but also has a communication compensation module. In this environment, there exists communication interference. CCTD could effectively relieve the interference through the communication compensation module. Therefore, its rewards are higher than the four baselines all the time. And we can see that its learning efficiency is also higher than others from the slope of the curve.

To evaluate the overhead introduced by the CCTD framework, as

Table 2. Quantitative comparison of algorithm complexity.

Algorithm	Inference Time (ms)	Train Time (s/1000 Steps)
MAPPO	4.5	12.4
QMix	3.2	8.5
QMix-com	4.6	13.8
QMix-Hir	5.8	15.2
CCTD (Ours)	7.2	19.8

shows, we measured the average inference time and training time per step on a machine with an NVIDIA RTX 4090 GPU and Intel i7-10700K CPU.

While CCTD increases the inference time by approximately 24% compared to QMix-Hir due to the RNN-based state compensation, this overhead is negligible in real-world UAV applications where the control cycle is typically 20–50 ms. The performance gain in complex tasks justifies this modest increase in computational demand.

4.3. Experimental Results

In this section, we verify the effects of the communication compensation module. In this scenario, each agent encodes its observation information and sends it to other agents in the form of communication packets. And they receive communication packets from other agents. In this way, they bridge communication. To better measure communication quality, we design a metric called effective information reception rate, which refers to the quotient of the communication packets agents receive in fact and the communication packets agents receive in theory. We verify the effectiveness of the communication compensation module by comparing the change in the effective information reception rate after the agent receives communication compensation. As Figure 3 shows, it presents 500 episode results. The less transparent and more volatile discounts are plotted from the raw data; each episode’s data is the average of all agents’ effective information reception rate. The smoother curve is plotted by smooth-processed data, which reflects the overall effects of communication. It can be seen from Figure 3 that the communication compensation module improves the communication effect by about 10%.

Furthermore, to intuitively demonstrate the predictive performance of the proposed module, we conducted a series of comparative experiments under different communication delay scenarios. Two methods were implemented for communication compensation: the proposed RNN-based method and the traditional historical-location-based method. We evaluated their performance across hundreds of episodes under 0–5 s and 5–10 s delay conditions.

The experimental results are illustrated in Figure 4 and Figure 5, and

Table 3. A detailed data comparison between the two methods (average error).

Method	0–5 s Delay Error	5–10 s Delay Error
RNN-based	≈19 m	≈21 m
Historical-location	≈27 m	≈32 m

. Note that the unit of the vertical axis in the figures represents $10\mathrm{m}$. As shown in the results, the RNN-based method significantly outperforms the historical-location-based method in both delay scenarios. Specifically, in the 0–5 s delay test (Figure 4), the average prediction error of the RNN method converges to approximately $1.9$ (about $19\mathrm{m}$), whereas the historical method’s error continues to rise, exceeding $2.7$ (about $27\mathrm{m}$). In the more challenging 5–10 s delay scenario (Figure 5), the RNN method maintains a stable error of around $2.1$ (about $21\mathrm{m}$), while the error of the historical method increases to over $3.2$ (about $32\mathrm{m}$).

The RNN-based approach consistently maintains lower prediction errors and demonstrates much smaller fluctuations (as indicated by the narrower shaded regions), which proves its superior stability and robustness in handling stochastic communication delays.

4.4. Ablation Studies

In this section, we analyze the contribution of different parts of CCTD to better verify the effectiveness of our proposed methods. We removed the communication compensation module in CCTD to verify it is useful for improving the performance of agents. We also remove the hierarchical architecture in the CCTD. The ablation studies are designed as follows:

• CCTD w/o com: Using CCTD without the communication compensation module
• CCTD w/o hir: Using CCTD without hierarchical architecture

Figure 6 shows the results of our ablation studies. As we can see, CCTD has a significant performance improvement compared to CCTD without com, showing that the communication compensation module is not a decisive factor, but it also plays a role in improving the performance of agents. In addition, CCTD w/o hir performs worse than CCTD. This indicates that it is a crucial component of CCTD, as it is helpful to decompose complex tasks and enhance learning efficiency. All the above results confirm that each component in CCTD is necessary and important to deal with coupled multi-tasks in this scenario.

5. Conclusions and Discussion

This paper introduces the Coupled Communication-Task Decoupling (CCTD) framework, which includes a hierarchical decision-making architecture and state compensation framework. The hierarchical decision-making architecture decomposes decision-making into two complementary layers; the upper-layer policy dynamically selects task-critical actions and rapidly changes the task of the agent. For the specific task, the lower-layer policy then executes action based on compensated information.

Through our state compensation framework, CCTD effectively resolves communication issues among multiple agents, the framework’s effectiveness is quantitatively validated through two metrics: (1) Predicted Distance error (<20 m) and (2) communication reception rate (improving by 10% compared with baseline), and achieves superior performance across all subtasks in multi-task environments. Beyond the scope of autonomous agent clusters, the architectural principles of CCTD—specifically its ability to reconstruct missing state information and decouple interdependent processes—possess high transferability to other data-intensive cyber-physical systems. A promising future direction lies in the domain of smart building energy management and industrial diagnostics. For instance, identifying operational faults in Air Handling Units (AHUs) within complex HVAC systems often faces challenges such as sensor delays, intermittent connectivity, and non-stationary thermal dynamics.

The state compensation framework proposed in this study could be adapted to enhance the reliability of fault detection and load forecasting by processing real-world labeled datasets, such as those encompassing operational data from office, auditorium, and hospital buildings [29]. By reconstructing missing thermal or flow-rate parameters through time-series prediction, our approach could facilitate more resilient building automation. The method proposed in this paper can be applied to such engineering fields and holds practical significance for real-world implementation.