Multi-Agent Deep Reinforcement Learning for Coding-Aware and Energy-Balanced Routing in Dynamic Drone Networks ^†

Highlights

What are the main findings?

• Throughput Enhancement: We discovered that integrating coding opportunities into routing metrics significantly enhances network throughput and transmission efficiency. This supports the high-capacity communication requirements of low-altitude UAV networks, enabling the robust relay of bandwidth-intensive multimedia traffic, such as real-time video and high-resolution imagery.
• Optimization Superiority: The study demonstrates that applying reinforcement learning to routing optimization outperforms traditional algorithms by effectively balancing conflicting performance metrics. This approach significantly improves communication efficiency and overcomes the computational complexity and solvability challenges inherent in traditional network coding schemes.

What is the implication of the main finding?

• Foundations for the Low-Altitude Economy: This research provides a scalable and intelligent networking solution essential for the ubiquitous deployment of next-generation autonomous UAV swarms and the burgeoning low-altitude economy.
• Enhanced Collaborative Capabilities: By successfully applying multi-agent reinforcement learning to complex topology management, the protocol significantly increases communication efficiency and throughput. This capability supports larger data payloads and robust information interaction, which are critical for collaborative aerial missions.

Abstract

By incorporating opportunistic coding, network throughput is enhanced, resulting in improved overall performance. However, applying this paradigm to Flying Ad-hoc Networks (FANETS) faces significant challenges due to the highly dynamic topology caused by the high-velocity mobility of UAVs, alongside the NP-hard complexity of identifying optimal coding opportunities in rapidly evolving aerial network architectures. To address these challenges, this paper proposes a novel coding-aware routing protocol based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG). We formulate the routing problem as a multi-agent continuous decision-making process, employing the MADDPG algorithm to optimize routing policies in real-time through decentralized execution and centralized training. To maximize network utility, we design a comprehensive reward function that integrates coding benefits, throughput, energy distribution, and end-to-end delay, ensuring a balance between throughput maximization and the energy sustainability of individual UAV nodes. Simulation results demonstrate that the proposed protocol significantly outperforms state-of-the-art coding-aware routing protocols in terms of throughput, Packet Delivery Ratio (PDR), and transmission delay, exhibiting superior robustness in highly dynamic FANET scenarios. Notably, at a network density of 20 UAVs, MARL-CAR outperforms COPE, DCAR, TSCAR, and RLCAR in terms of coding ratio by 32.23%, 18.93%, 20.35%, and 5.5%, respectively. This research provides a scalable and intelligent networking solution for the next generation of autonomous UAV swarms and collaborative aerial missions.

1. Introduction

The burgeoning low-altitude economy has catalyzed the ubiquitous deployment of Unmanned Aerial Vehicles (UAVs) across diverse sectors, capitalizing on their superior maneuverability, rapid deployment, and robust Line-of-Sight (LoS) communication links [1,2]. Especially in mission-critical scenarios—such as post-disaster search and rescue, emergency communication restoration, and persistent wide-area surveillance—the operational efficacy of a solitary UAV is inherently limited by its onboard energy reserves and constrained sensing swathe. Consequently, the paradigm shift toward multi-UAV collaborative swarms, structured as Flying Ad-hoc Networks (FANETs) [3,4,5], has become an essential trajectory for achieving ubiquitous coverage and seamless information interaction. Unlike traditional Mobile Ad-hoc Networks (MANETs), FANETs represent a highly volatile network architecture that maintains real-time situational awareness for ground control stations through multi-hop relaying, even in infrastructure-depleted or denied environments [6,7].

The primary objective of routing protocols is to guarantee stringent Quality of Service (QoS) for heterogeneous network traffic, facilitating high-throughput and reliable packet delivery [8,9,10]. However, the transition from terrestrial MANETs to FANETs introduces significant challenges, characterized by high-velocity mobility and the resultant sporadic topology reconfigurations. These inherent dynamics render conventional topology-based routing paradigms suboptimal; Maintaining global link-state consistency incurs prohibitive signaling overhead and intolerable convergence latencies [11]. In this context, geographic routing mechanisms—which execute forwarding decisions based on localized spatial coordinates—have emerged as a dominant paradigm due to their inherent scalability and resilience against rapid topological fluctuations [12].

Nevertheless, in practical mission environments, standalone routing strategies often struggle to sustain ultra-reliable communication amidst the frequent link outages and time-varying channel stochasticity inherent to FANETs [13,14]. The conventional store-and-forward mechanism inevitably precipitates retransmission redundancy and spectral inefficiency when link stability is compromised. To circumvent this bottleneck, Network Coding (NC) has been integrated into FANETs as a transformative transmission paradigm [15]. Moving beyond simple forwarding, NC enables intermediate relay nodes to computationally combine multiple ingress data streams into a single encoded egress packet. This approach exploits the inherent broadcast nature of the wireless medium, effectively minimizing transmission frequency and attenuating packet loss probabilities. Empirical evidence underscores that NC not only amplifies network throughput but also bolsters transmission robustness and curtails end-to-end latency within highly dynamic aerial environments [16].

The efficacy of NC in FANETs hinges on the strategic identification of coding opportunities [17]. A valid coding opportunity is instantiated when intersecting data flows at a relay node can be encoded, provided that the respective downstream recipients possess the requisite side information for successful decoding. Maximizing these opportunities is the primary conduit for unlocking the gain of NC, a process intrinsically coupled with path selection. Specifically, routing decisions dictate the spatial interaction of data flows, which directly governs the density of potential coding intersections [18]. Consequently, the development of Coding-Aware Routing (CAR) protocols—capable of proactively perceiving and engineering coding opportunities—is paramount to augmenting the systemic performance of autonomous UAV networks.

Although coding-aware routing holds great promise, existing solutions remain ill-suited for the highly dynamic FANET environment, primarily due to three critical limitations: (a) Calculating optimal coding opportunities is inherently difficult in rapidly changing environments. Traditional schemes relying on static or quasi-static topology assumptions fail to accommodate the frequent link intermittency and topology shifts characteristic of UAV networks [19]. (b) Existing algorithms typically adopt a greedy approach to maximize the sheer number of coding opportunities. However, this myopic focus may force data flows to detour through congested nodes, inadvertently degrading end-to-end latency, causing uneven energy depletion, and overloading computational resources, thereby failing to balance throughput with other vital QoS metrics [20]. (c) There exists a complex, non-linear coupling between routing decisions and aggregate network performance (e.g., latency, energy, and reliability). In high-mobility FANETs, analytically characterizing this mapping via closed-form mathematical models results in a non-convex optimization problem where computational complexity scales exponentially with network size, rendering it impractical for real-time decision-making [21,22,23].

Crucially, while multi-objective optimization or sophisticated heuristics could theoretically address these individual constraints, they encounter a fundamental contradiction between computational latency and network dynamics in high-speed FANETs. Traditional iterative solvers often require extensive time to converge, which is incompatible with the short topology coherence time of UAVs moving at speeds. By the time a heuristic algorithm generates a routing decision, the physical link state may have already changed, rendering the solution obsolete.

In response to the aforementioned limitations, this paper proposes MARL-CAR, a novel Multi-Agent Deep Reinforcement Learning-based Coding-Aware Routing protocol. By modeling the FANET as a cooperative multi-agent system, MARL-CAR leverages the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to facilitate collaborative routing decisions among UAVs. This enables near-instantaneous execution, making reinforcement learning a functional necessity for real-time routing in transient aerial environments rather. The selection of MADDPG as the core methodology is significant because it effectively addresses the non-stationarity of dynamic FANETs through its centralized training and decentralized execution (CTDE) paradigm. This approach not only actively uncovers latent coding opportunities but also masters the complex non-linear mapping between distributed routing actions and global network performance.

To mitigate the deficiencies of traditional algorithms—which often sacrifice holistic performance in a myopic pursuit of coding opportunities—MARL-CAR incorporates multi-dimensional metrics into its reward function design, including throughput, end-to-end latency, and coding potential. Furthermore, we introduce specific parameters such as Coding Benefit and Energy Distribution. These are critical not only for maximizing throughput via network coding but also for ensuring the long-term sustainability of energy-constrained UAV nodes by maintaining network-wide energy equilibrium. The key contributions and innovations of this work are as follows:

• We model the coding-aware routing problem in FANETs as a Multi-Agent Markov Decision Process. By adopting the CTDE framework, we enable UAVs to make independent routing decisions based on local observations while learning a cooperative global policy that adapts to the dynamic and time-varying topologies typical of UAV networks.
• We introduce two novel parameters: Coding Benefit and Energy Distribution. The former synthesizes coding potential with hop-count overhead to optimize path efficiency, while the latter focuses on balancing the residual energy variance among UAV nodes to maintain network-wide energy equilibrium.
• We design a comprehensive multi-dimensional reward function and employ the MADDPG algorithm for optimization. Distinct from traditional schemes, our reward mechanism explicitly integrates end-to-end delay alongside coding opportunities, throughput, and energy consumption. This approach allows for dynamic strategy adjustment that prevents the load imbalance issues prevalent in conventional schemes while guaranteeing low-latency transmission.
• We conduct extensive simulation experiments to provide an in-depth analysis of MARL-CAR’s performance. The results demonstrate significant improvements in key metrics, particularly coding success rate, transmission throughput, and delay reduction, compared to state-of-the-art benchmarks.

The structure of the paper is as follows: Section 2 reviews the related work on code-aware routing. Section 3 outlines the problem and establishes the network model. The details of the MARL-CAR mechanism are provided in Section 4. Section 5 discusses the experimental setup and results, followed by the conclusion in Section 6.

2. Related Works

In the field of network coding, one of the key challenges is how to efficiently identify and utilize coding opportunities within routing protocols. Early approaches, such as the COPE protocol introduced by Katti et al. [16], laid the foundation for coding-aware routing by demonstrating that selecting paths with abundant coding opportunities could significantly improve network throughput. However, COPE adopts a decoupled architecture that segregates route discovery from coding detection, thereby constraining its efficacy in highly dynamic FANET topologies. This separation limited the efficiency of coding-aware routing. To address this, the ROCX [18] protocol was introduced, which synchronized the detection of coding opportunities with the route discovery process. This refinement not only increased the efficiency of discovering coding opportunities but also enhanced the overall performance of coding-aware routing protocols.

Le et al. [24] developed Distributed Coding-Aware Routing (DCAR) to overcome the two-hop coding limitation in COPE, enabling coding opportunities to be extended beyond localized paths. By employing a novel Coding-Aware Routing Metric (CRM), DCAR detected coding benefits along the entire routing path rather than just among immediate neighbors. Building on this distributed approach, Rafiee and Mirjalily [25] improved routing reliability by integrating fuzzy logic mechanisms. Their protocol utilized a fuzzy logic controller to select optimal forwarding nodes by comprehensively weighing link quality and potential coding opportunities. However, while these algorithms performed well in relatively static networks, they struggled with the rapid topological changes in FANETs, where frequent link failures incurred prohibitive maintenance overheads.

Addressing the computational and energy constraints of wireless nodes, He and Yang [26] proposed the CARTA protocol for Wireless Sensor Networks. CARTA minimized route discovery overhead by leveraging a tree-based addressing scheme within a Multi-Root Multi-Tree topology to calculate coding opportunities. Despite its low overhead, the reliance on a structured tree topology made it difficult to adapt to the decentralized and highly mobile nature of FANETs. To further enhance coding probability, Shao et al. [27] developed the TSCAR protocol, which introduced traffic shaping. TSCAR employed an active waiting mechanism to buffer and align asymmetric data flows at intermediate nodes, artificially creating coding opportunities. Although this significantly boosted the coding ratio, the introduced queuing delay was often unacceptable for latency-sensitive UAV control tasks.

Parallel to the advancement of coding-aware mechanisms, extensive research was devoted to enhancing the robustness and adaptability of general routing protocols in FANETs through intelligent optimization algorithms. To address the slow convergence of traditional protocols in dynamic topologies, Zhou et al. [28] proposed a bidirectional Q-learning routing strategy based on Ad hoc On-Demand Distance Vector Routing (AODV), which significantly accelerated the convergence speed of the learning process. Addressing the local optima problem inherent in greedy forwarding, Wang et al. [29] incorporated Particle Swarm Optimization to refine next-hop selection. Furthermore, Jiang et al. [30] integrated fuzzy logic with depth-first search to construct Q-value routing tables, improving the cooperation between ground equipment and UAV nodes. Recognizing the critical constraints of energy consumption in UAV swarms, Zhu et al. [31] utilized Ant Colony Optimization to calculate energy-efficient routes for massive machine-type communications, effectively prolonging network lifetime.

In essence, prior studies on coding-aware routing predominantly focused on maximizing coding opportunities within quasi-static or energy-constrained networks (e.g., MANETs or WSNs). Conversely, while recent intelligent routing protocols successfully applied learning-based algorithms to handle FANET dynamics, they often overlooked the throughput gains offered by network coding. The inherent volatility of UAV-based FANETs demands a paradigm shift that amalgamated these two strengths. Motivated by the superior decision-making capabilities of Reinforcement Learning (RL) demonstrated in recent works [32,33], we proposed a novel coding-aware routing protocol specifically tailored to meet the dynamic challenges of FANETs.

3. System Model and Problem Formulation

In this section, we outline the research problem of code-aware routing and introduce the corresponding network model. We frame the routing problem within the context of RL, with the aim of maximizing the coding opportunities available to the system. By modeling this problem as an optimization task, our goal is to identify an optimal routing path that effectively balances data transmission requirements with network constraints. The specific process is outlined below.

3.1. Problem Formulation and Network Modeling

Before establishing the mathematical model, it is crucial to define the core mechanism of coding-aware routing. Unlike traditional store-and-forward protocols, coding-aware routing exploits the broadcast nature of wireless media to combine multiple data streams at intermediate relay nodes. Specifically, when two data flows intersect at a relay UAV, the node can perform an XOR operation on the packets (e.g., $p_{enc}=p_1\oplus p_2$) and broadcast the encoded packet. This process, contingent upon the receiving nodes possessing the necessary decoding information (i.e., the native packets of the opposing flow), significantly reduces transmission frequency and bandwidth consumption. Consequently, the routing objective shifts from simply finding the shortest path to identifying paths rich in such coding opportunities.

We define the coding-aware routing problem as a sequential decision-making process, which is addressed using a Multi-Agent Reinforcement Learning (MARL) approach. As shown in Figure 1, the trained MARL-CAR algorithm functions as the intelligent agent within this framework, learning optimal routing decisions through continuous interaction with the network environment. The state space represents the agent’s observations, incorporating critical factors such as the UAV’s topological structure and the intersection status of data flows. This state space enables the agent to understand and adapt to the dynamically changing network environment. At the same time, the reward function integrates important elements such as data transmission requirements and network constraints, guiding the agent’s learning process. To align with the core objectives of coding-aware routing, the reward function explicitly considers both the discovery of network coding opportunities and the overall system throughput, ensuring that the agent prioritizes paths that maximize network efficiency and resource utilization. By observing the state space through reinforcement learning, the agent converges to an optimal policy to maximize the network’s efficiency and resource utilization.

Figure 1. Framework for code-aware routing using Multi-Agent Reinforcement Learning in a drone network.

3.2. Coding-Aware Network Model for FANETs

In this paper, we present a coding-aware network model for Flying Ad-hoc Networks, integrating graph theory with network coding to enhance data transmission efficiency. The network model is represented as an undirected graph $G=\{U,E,W\}$, where U is the set of UAVs, E represents the set of communication links between UAVs, and W denotes the set of edge weights that reflect the coding potential between UAVs.

The network graph accounts for both stable and dynamic coding opportunities. In the stable coding graph, each vertex corresponds to a flow handled by a UAV, with the value associated with each vertex representing the average number of packets currently in the queue for that flow. An edge between two vertices signifies that the corresponding flows satisfy the conditions for network coding, allowing them to be combined during transmission. If a subgraph of the coding graph forms a complete graph, all flows within that subgraph can be encoded together in a single transmission, thereby improving transmission efficiency.

In contrast, the dynamic coding graph emphasizes the potential coding opportunities introduced by new flows. It captures the free-ride effect, where the effective queue length of a new flow can be reduced by encoding it with existing flows. This dynamic graph helps identify and leverage these opportunities, thereby optimizing network performance by incorporating the coding benefits from newly introduced traffic.

However, it is critical to acknowledge that maximizing coding opportunities is not always equivalent to maximizing network throughput or robustness. Under certain conditions, coding-aware decisions can be harmful to the overall system performance. For instance, in scenarios with highly asymmetric traffic or heterogeneous link rates, forcing data streams to converge at a specific relay node may introduce significant queuing delays, as high-rate flows must wait for lower-rate flows to complete the XOR operation. Moreover, if the routing protocol myopically prioritizes coding intersections, it may select circuitous paths that involve excessive detours. Such detours escalate the hop-count overhead, increase end-to-end latency, and accelerate the energy depletion of battery-constrained UAV nodes. These decisions become detrimental when the cumulative network penalty—in terms of delay and energy—outweighs the theoretical bandwidth gain derived from network coding.

To navigate these trade-offs, we propose the coding-aware routing protocol MARL-CAR, which seeks to balance the topological distance and coding potential. The routing decision is framed as a MDP, where the optimization problem is expressed as follows:

$$max{\displaystyle \frac{\left({\sum }_{i=1}^{\left|l\right|}{w}_{l_i,l_{i+1}}\left(l\right)\right)}{\left|l\right|}},$$

(1)

where L denotes the complete set of all feasible routing paths from the source UAV to the destination UAV. For a specific path $l\in L$, $l_i$ represents the i-th UAV node in the sequence, and $\left|l\right|$ denotes the total number of nodes in that path. The edge weight $w_{l_i,l_{i+1}}\left(l\right)$ for each segment of the path is derived from the coding potential. The objective is to identify a path that minimizes transmission overhead while simultaneously maximizing coding potential.

Within this framework, the objective function is mapped to a reinforcement learning value function, allowing the agent to continually evaluate and refine its policy based on received rewards. By incorporating coding opportunities and other key network performance metrics into the reward structure, the learned policy effectively guides route selection to reduce transmission overhead while exploiting coding opportunities to enhance throughput and reliability. Such DRL-based routing has been demonstrated to adaptively cope with rapid topology changes and improve overall network performance compared to traditional heuristic methods, making it particularly suitable for the highly dynamic context of FANETs.

3.3. Formulation of Network Coding-Aware Routing Metric

To systematically identify coding opportunities within the routing framework, we formalize the network topology with respect to flow dynamics. Let a data stream f, originating from a source UAV $u_s$ and destined for a target UAV $u_d$, be denoted by the sequence of nodes traversing its path. For any intermediate relay UAV $u_v$, we characterize its topological relationship to flow f via two sets: the Predecessor Set ${\mathcal{P}}_f\left(u_v\right)$ and the Successor Set ${\mathcal{S}}_f\left(u_v\right)$. Accordingly, the local one-hop neighbor set of $u_v$ is defined as $\mathcal{N}\left(u_v\right)$.

Drawing upon the solvability criteria established in [16], the validity of executing XOR-based network coding at node $u_v$ is contingent upon the decoding capability of the next-hop recipients. Specifically, for two intersecting flows $f_1$ and $f_2$, the topology must satisfy a reciprocal overhearing constraint: the designated receiver of flow $f_1$ (denoted as $s_1\in {\mathcal{S}}_{f_1}\left(u_v\right)$) requires prior access to the native packet of flow $f_2$. This access is guaranteed only if $s_1$ falls within the transmission coverage of the predecessor of flow $f_2$ (denoted as $p_2\in {\mathcal{P}}_{f_2}\left(u_v\right)$). This geometric constraint manifests in two topological forms: either via direct opportunistic listening (where $s_1\in \mathcal{N}\left(p_2\right)$) or via reverse traffic symmetry (where $s_1$ is identical to $p_2$). Symmetrically, the receiver of flow $f_2$ must simultaneously possess connectivity to the predecessor of flow $f_1$ to ensure mutual decodability.

To integrate these topological opportunities into the routing metric, we quantify the Coding Gain ($\mathcal{G}$). Assume a packet $P_l$ containing candidate routing information traverses a path of length h. As it propagates, it identifies potential coding relays. Consider a specific coding-capable UAV $u_c$ ($1<c<h$) where a set of flows intersect. Let $K\left(u_c\right)$ denote the number of concurrent streams. The transmission capability is governed by the set of link rates $\mathsf{\Phi }\left(u_c\right)=\{\mu \left(f_k\right)\mid 1\le k\le K\left(u_c\right)\}$, where $\mu \left(f_k\right)$ represents the link capacity for the k-th flow. Since the encoded packet is broadcast to multiple recipients simultaneously, the effective transmission duration is constrained by the bottleneck link, defined as:

$${\mu }_{min}\left(u_c\right)=\underset{1\le k\le K\left(u_c\right)}{min}\mu \left(f_k\right).$$

(2)

The local coding gain $\mathcal{G}\left(u_c\right)$ is formulated to reflect the bandwidth efficiency derived from this broadcast mechanism. It is calculated as the product of the theoretical reduction in transmission slots (i.e., $K\left(u_c\right)-1$) and the ratio of the bottleneck rate to the aggregate unicast rate:

$$\mathcal{G}\left(u_c\right)=(K\left(u_c\right)-1)·{\displaystyle \frac{{\mu }_{min}\left(u_c\right)}{{\sum }_{k=1}^{K\left(u_c\right)}\mu \left(f_k\right)}}.$$

(3)

Finally, since the packet $P_l$ exploits encoding opportunities across multiple hops, the total utility is derived by aggregating the local gains from all identified coding nodes. The Cumulative Coding Gain is expressed as:

$${\mathcal{G}}_{total}\left(P_l\right)=\sum _{1<c<h}\mathcal{G}\left(u_c\right).$$

(4)

4. Components of MARL-CAR

This section is dedicated to a comprehensive exposition of the proposed reinforcement learning-based framework for routing optimization. Initially, we bridge the gap between the routing decision problem and the learning model by formulating the task as a Markov Decision Process. Within this context, the critical components of the learning environment are rigorously defined, with a specific focus on the mathematical characterization of the state space, the discrete action set, and the reward function design essential for guiding the policy update. Building upon this theoretical foundation, the MARL-CAR algorithm is formally introduced. The section concludes with a systematic delineation of the algorithm’s operational workflow, elucidating the iterative interaction mechanisms between the agents and the network environment.

4.1. Cooperative Coding-Aware Routing: A Multi-Agent Learning Approach

In pursuit of the optimal solution for the objective function defined in Equation (1), this study constructs a distributed learning architecture based on MARL, specifically utilizing the MADDPG algorithm. The schematic representation of this framework is depicted in Figure 1, which strictly adheres to the CTDE paradigm to balance global training stability with local execution flexibility. Within this networked environment, each UAV is conceptualized as an autonomous intelligent agent i. These agents continuously interact with the dynamic FANET environment to optimize routing performance. Mathematically, the decision-making evolution is formalized as a Markov Decision Process (MDP). In each time slot t, agent i perceives its immediate surroundings to form a local state observation $s_{i,t}$. Subsequently, based on this observation, it generates a routing decision $a_{i,t}$, which contributes to the system-level joint action vector $a_t$. Following the execution of these actions, the environment transitions to a new global state $S_{t+1}$ and feeds back a specific reward signal $r_{i,t}$. Through this iterative feedback loop, agents progressively refine their policies, converging towards a equilibrium that represents a near-optimal cooperative strategy.

The coding-aware routing problem is formally modeled as a multi-agent extension of an MDP, characterizing the discrete-time stochastic control framework by the tuple $(S, A, P, R)$. In this tuple:

• S denotes the global state space, where $S_t\in S$ represents the configuration of the entire system at time t.
• A denotes the joint action space, representing the combination of actions from all agents.

The system dynamics are governed by the state transition probability matrix P, which describes the stochastic response of the environment. It is defined as the probability of transitioning to the next global state $S^{\prime }$ given the current state S and the joint action a:

$$P(S^{\prime }\mid S,a)=\mathbb{P}[S_{t+1}=S^{\prime }\mid S_t=S,a_t=a].$$

(5)

Additionally, R represents the reward function set, shared or distributed among agents to encourage cooperation. In the MADDPG framework, each agent’s decision-making strategy is encapsulated by a deterministic policy, denoted as ${\mu }_i$, which establishes a direct mapping from its local observation to a specific action. This is formally expressed as:

$$a_{i,t}={\mu }_i\left(s_{i,t}\right).$$

(6)

The local state $s_{i,t}$ encapsulates the network information perceived by UAV i, serving as the input to its Actor network to characterize the local routing scenario. Since routing decisions dynamically alter the network environment, the state transition is inherently determined by the joint actions of all UAVs.

In the proposed MARL-CAR framework, the local state of agent i at time slot t is formally defined as the tuple $s_{i,t}=(P_i(u_s,u_d),a_{i,t-1})$. This composite vector is designed to integrate the Service requirement data with the agent’s historical decision context. The specific components are characterized as follows:

• Service requirement data $P_i(u_s,u_d)$: This component identifies the data packet currently under the management of agent i, explicitly specifying its source UAV $u_s$ and targeted destination UAV $u_d$.
• Historical routing action $a_{i,t-1}$: This represents the forwarding decision executed by the agent during the preceding time interval, serving as temporal feedback for policy updates.

Correspondingly, the action variable $a_{i,t}$ denotes the routing trajectory selected by agent i for the current transmission. Mathematically, it is expressed as $a_{i,t}=l_{i,t}\{u_s,u_1,u_2,\dots ,u_d\}\in L$, where u represents the sequence of UAV nodes constituting the selected link.

To quantify agent efficacy and characterize the caliber of routing decisions, we establish the reward function, denoted as R, as the fundamental evaluation metric. In this study, we construct a reward mechanism designed to promote the selection of links that exhibit superior coding potential and transmission stability, while simultaneously penalizing routes plagued by poor link quality. Since the formulation of the reward signal fundamentally shapes the optimization trajectory, our objective is to guide the agents toward paths within FANETs that offer a synergy of maximized coding opportunities and high throughput. To achieve this, the proposed composite reward function synthesizes four critical performance dimensions: coding benefits, throughput, energy distribution, and transmission latency.

Building upon the analysis of Cumulative Coding Gain presented in the previous section, we derive a specific reward component to address coding efficiency. For a routing decision made by agent i, the coding-related reward is formulated as the net utility derived from the selected path $l_i$:

$$r_{i,code}={\mathcal{G}}_{total}\left(P_{l_i}\right)-\mathcal{H}\left(P_{l_i}\right),$$

(7)

where $P_{l_i}$ represents the specific trajectory resulting from action $a_{i,t}$, and $\mathcal{H}\left(P_{l_i}\right)$ quantifies the path length in terms of hop count.

This formulation establishes a balance between throughput maximization and latency minimization. The component ${\mathcal{G}}_{total}\left(P_{l_i}\right)$ serves as an incentive, encouraging agents to traverse topologically dense regions with abundant coding opportunities, thereby enhancing spectral efficiency. Conversely, $\mathcal{H}\left(P_{l_i}\right)$ imposes a distance-based penalty to curb excessive path lengths. By integrating these competing terms, the reward function steers the policy toward an optimal throughput-delay trade-off, deterring circuitous paths that offer only marginal coding gains.

To accurately evaluate transmission efficiency, we bifurcate the throughput reward mechanism into two directional streams: uplink and downlink. For a given agent i, the uplink throughput metric is calculated as:

$$r_{i,up-thr}=\mathrm{avg}\left(\sum _{l_{s,d}\in a_{i,t}}{\displaystyle \frac{t{x}_t\left(l_{s,d}\right)}{b{w}_t\left(l_{s,d}\right)·\left|{\Delta }_t\right|}}\right).$$

(8)

In this context, the action $a_{i,t}$ selects the specific link $l_{s,d}$ bridging the source UAV s and the destination UAV d. The variable $t{x}_t\left(l_{s,d}\right)$ measures the cumulative data transfer across this link over the duration ${\Delta }_t$, whereas $b{w}_t\left(l_{s,d}\right)$ denotes the channel capacity (bandwidth). The operator $\mathrm{avg}(·)$ is applied to smooth out idle periods, thereby deriving the mean effective transmission rate.

Symmetrically, the downlink component focuses on the reception quality and is expressed as:

$$r_{i,down-thr}=\mathrm{avg}\left(\sum _{l_{d,s}\in a_{i,t}}{\displaystyle \frac{r{x}_t\left(l_{d,s}\right)}{b{w}_t\left(l_{d,s}\right)·\left|{\Delta }_t\right|}}\right),$$

(9)

where $r{x}_t\left(l_{d,s}\right)$ indicates the quantity of data successfully decoded at the receiver end derived from the source.

Finally, to establish a holistic view of link quality, the aggregate throughput reward is formulated as a weighted sum of these two components:

$$r_{i,thr}=\varphi ·r_{i,up-thr}+(1-\varphi )·r_{i,down-thr}.$$

(10)

where, the control parameter $\varphi \in [0, 1]$ allows the Traffic Engineering (TE) optimizer to balance the priority between uplink transmission and downlink reception based on network requirements.

Myopically prioritizing coding gains and throughput can lead to disproportionate energy consumption, thereby precipitating premature node exhaustion. To prolong the network lifetime, we incorporate an energy cost metric designed to discourage the overuse of energy-constrained nodes. The cost function for a specific UAV u is defined as:

$$c\left(u\right)=1-{\displaystyle \frac{E_{res}\left(u\right)}{E_{init}\left(u\right)}},$$

(11)

where $E_{init}\left(u\right)$ signifies the full battery capacity of the UAV at the onset of the mission, and $E_{res}\left(u\right)$ denotes the remaining energy available at the specific instant of packet transmission.

This study adopts a simplified energy consumption model to prioritize the marginal energy impact of routing decisions—specifically communication and processing overhead—on the electronic operational lifetime of the nodes. Assuming uniform initial energy across the swarm, a depletion in residual energy corresponds to an elevated consumption cost. Furthermore, we introduce the concept of energy distribution, denoted as $d\left(u\right)$, which is formulated as:

$$d\left(u\right)={\displaystyle \frac{2}{\pi }}arctan(E_{res}\left(u\right)-\overline{E}\left(u\right)),$$

(12)

where, $\overline{E}\left(u\right)$ signifies the mean energy level of one-hop neighbors surrounding UAV u, derived via the exchange of HELLO messages.

In summary, the specific individual reward component attributed to energy consumption for agent i is expressed as:

$$r_{i,energy}=\sum _{u_k\in a_{i,t}}[{\alpha }_{1}d\left(u_k\right)-{\alpha }_{2}c\left(u_k\right)-g],$$

(13)

where, the summation iterates over the UAV nodes $u_k$ in the path selected by agent i. g accounts for the baseline energy drain of the drone over time, while ${\alpha }_1$ and ${\alpha }_2$ serve as weighting coefficients (${\alpha }_1=0.05,{\alpha }_2=0.5$).

To capture dynamic network characteristics and the impact of traffic load on latency, the single-hop delay $D\left(l_{j,k}\right)$ between any two nodes in the selected path is formulated as the aggregate of queuing, processing, and transmission delays:

$$D\left(l_{j,k}\right)=T_{queue}\left(j\right)+T_{proc}\left(j\right)+{\displaystyle \frac{L_{pkt}}{b{w}_t\left(l_{j,k}\right)}},$$

(14)

where $T_{queue}\left(j\right)$ represents the queuing delay that escalates as traffic load approaches node capacity, effectively indicating congestion; $T_{proc}\left(j\right)$ accounts for the processing delay induced by routing lookups and network coding (XOR) operations; and the transmission term is defined by the ratio of packet size $L_{pkt}$ to the instantaneous link bandwidth $b{w}_t\left(l_{j,k}\right)$.

The reward component for delay is defined as a penalty term. Since the reinforcement learning agent seeks to maximize the reward, a negative sign is applied to the cumulative delay to encourage the selection of low-latency paths:

$$r_{i,delay}=-\sum _{l_{j,k}\in a_{i,t}}D\left(l_{j,k}\right).$$

(15)

By synthesizing the incentives for coding potential, throughput, energy conservation, and delay minimization, the comprehensive individual reward function for agent i at time t is formulated as:

$$r_{i,t}={\xi }_1·r_{i,code}+{\xi }_2·r_{i,thr}+{\xi }_3·r_{i,energy}+{\xi }_4·r_{i,delay},$$

(16)

where, ${\xi }_1,{\xi }_2,{\xi }_3$, and ${\xi }_4$ represent the distinct weights assigned to each component (${\xi }_1+{\xi }_2+{\xi }_3+{\xi }_4=1$), allowing for flexible tuning to balance the trade-offs between coding efficiency, high throughput, energy balance, and low latency according to specific mission requirements.

Finally, to foster cooperation and optimize the overall performance of the UAV swarm in the CTDE framework, we define the global reward $R_t$. This metric aggregates the individual rewards of all agents to guide the centralized critic network, ensuring that local optimization aligns with the system-wide objectives:

$$R_t={\displaystyle \frac{1}{N}}\sum _{i=1}^N{r}_{i,t}.$$

(17)

4.2. Learning Algorithm in MARL-CAR

Having established the modeling framework for the coding-aware routing problem, the subsequent phase focuses on its resolution. To this end, we propose MARL-CAR, a routing algorithm predicated on the MADDPG framework. The algorithm is expressly designed to simultaneously maximize coding opportunities and system throughput, while minimizing end-to-end latency. As illustrated in Figure 2, MADDPG extends reinforcement learning based on policy gradients to multi-agent domains by adopting the paradigm of CTDE.

Figure 2. MARL-CAR based on MADDPG algorithm.

Within the MARL-CAR architecture, each UAV acts as a distinct agent i. The routing policy for agent i is parameterized as a local deterministic policy function, denoted as ${\mu }_i\left(s_i\right)$, which relies solely on its own local observations $o_i$. Conversely, the centralized action-value function for agent i, utilized during training to guide cooperation, is represented by $Q_i(s,a)$, where s denotes the global state information and $A=\{a_1,\dots ,a_N\}$ represents the joint actions of all N agents. These functions are approximated via artificial neural networks, designated as the Actor network and the centralized Critic network, respectively. The optimization of direct output actions is achieved through a policy gradient ascent algorithm. Furthermore, to enhance stability in this multi-agent setting, the MADDPG framework incorporates a target network technique, which employs structural replicas of the training networks to compute target values. Given that the application environment is a dynamically evolving FANET characterized by non-stationarity induced by multiple interacting UAVs, the utilization of an MADDPG-based approach offers significant advantages in solving this specific routing model compared to independent single-agent learning.

4.3. The Network Model

This paper implements an Actor-Critic architecture to effectively optimize routing policies within continuous domains. The centralized action-value function, which underpins agent i’s learning process, is approximated by a neural network with parameters ${\theta }_i^Q$. regarding action selection at time step t, the process is governed by a deterministic policy ${\mu }_i$. This policy maps local observations $s_{i,t}$ to specific actions $a_{i,t}$, a relationship formally defined as:

$$a_{i,t}={\mu }_i(s_{i,t}\mid {\theta }_i^{\mu }),$$

(18)

where ${\theta }_i^{\mu }$ represents the learnable weights characterizing the Actor (policy) network of agent i.

In the phase of centralized training, the evaluation of actions is performed by the Critic network, utilizing holistic state information. Once the instantaneous reward $r_{i,t}$ is received and the system transitions to the next global state $S_{t+1}$, the formulation of the centralized state-value function adheres to the principles of the Bellman equation:

$$Q_i^{\mu }(S_t,a_{1,t},\dots ,a_{N,t})=\mathbb{E}[r_{i,t}+\gamma Q_i^{\mu }(S_{t+1},{\mu }_1\left(s_{1,t+1}\right),\dots ,{\mu }_N\left(s_{N,t+1}\right))],$$

(19)

where, $S_t$ represents the global state vector encompassing observations of all agents, $\gamma \in [0,1)$ represents the discount factor, and $\mu =\{{\mu }_1,\dots ,{\mu }_N\}$ denotes the set of current deterministic policies for all N agents.

Furthermore, to stabilize training, a target Q-value $y_j$ is computed for each sample j in a mini-batch utilizing target network structures. Unlike DDPG, the target calculation in MADDPG requires the next-state joint actions of all agents:

$$y_j=r_{i,j}+\gamma Q_i^{\prime }(S_{j+1},{\mu }_1^{\prime }(s_{1,j+1}\mid {\theta }_1^{{\mu }^{\prime }}),\dots ,{\mu }_N^{\prime }(s_{N,j+1}\mid {\theta }_N^{{\mu }^{\prime }})),$$

(20)

where the index j refers to the j-th transition sample randomly drawn from the replay buffer $\mathcal{D}$ during the centralized training phase. Each sample j in the mini-batch represents a historical experience tuple $(S_t,a_t,r_t,S_{t+1})$ that was originally recorded at some preceding time step t. $Q_i^{\prime }$ and ${\mu }_k^{\prime }$ denote the target Critic network and target Actor networks respectively, parameterized by ${\theta }_i^{Q^{\prime }}$ and ${\theta }_k^{{\mu }^{\prime }}$. These target parameters are soft-updated copies of the online networks. By employing the randomized sampling index j for training rather than sequential time t, the algorithm effectively decorrelates the transition data, thereby ensuring gradient stability and improving policy convergence.

Subsequently, the parameters ${\theta }_i^Q$ of the Critic network for agent i are updated via gradient descent by minimizing the loss function. This loss is defined as the mean squared error over a mini-batch of size M, assessing the discrepancy between the estimated Q-value and the target value:

$$Loss\left({\theta }_i^Q\right)={\displaystyle \frac{1}{M}}\sum _{j=1}^M{(y_j-Q_i(S_j,a_{1,j},\dots ,a_{N,j}\mid {\theta }_i^Q))}^2.$$

(21)

Accordingly, the expected return objective function J for agent i is mathematically characterized as:

$$J\left({\theta }_i^{\mu }\right)=\mathbb{E}\left[r_{i,t}\right].$$

(22)

Applying the chain rule to the gradient of this objective function, the update mechanism for the Actor network of agent i is derived (the deterministic policy gradient). Crucially, since the centralized Critic takes joint actions as input, the gradient is computed with respect to agent i’s action $a_i$, evaluated at the actions currently chosen by all agents:

$${\nabla }_{{\theta }_i^{\mu }}J\approx {\displaystyle \frac{1}{M}}\sum _{j=1}^M\left[{\nabla }_{a_i}{Q}_i(S_j,a_1,\dots ,a_N\mid {\theta }_i^Q){|}_{a_k={\mu }_k\left(s_{k,j}\right)\forall k}·{\nabla }_{{\theta }_i^{\mu }}{\mu }_i(s_{i,j}\mid {\theta }_i^{\mu })\right],$$

(23)

where, ${\nabla }_{a_i}{Q}_i$ represents the gradient of the Critic’s output with respect to agent i’s action, and ${\nabla }_{{\theta }_i^{\mu }}{\mu }_i$ is the gradient of the Actor network’s output with respect to its own parameters.

Finally, the parameters ${\theta }_i^{Q^{\prime }}$ and ${\theta }_i^{{\mu }^{\prime }}$ corresponding to the target networks undergo a soft update process using a smoothing technique.

The detailed procedure of the proposed routing strategy is summarized in Algorithm 1. Specifically, this study develops a multi-agent training framework where each UAV, acting as an intelligent agent, interacts with the dynamic topology to optimize its forwarding policy through a centralized training and decentralized execution paradigm.

Algorithm 1 Execution Procedure of the MADDPG-Driven Routing Protocol

Require:  Replay Buffer $\mathcal{D}$, UAV network Global State $S_t$, Local Observation Set ${\mathbf{s}}_t=\{s_{1,t},\dots ,s_{N,t}\}$, System Parameters: Soft Update Rate $\tau$, Discount Factor $\gamma$.     1: Setup: For every agent $i\in \{1\dots N\}$: Instantiate Critic $Q_i(S,\mathbf{a}\mid {\theta }_i^Q)$ and Actor ${\mu }_i(s_i\mid {\theta }_i^{\mu })$.     2: Synchronize target weights: ${\theta }_i^{Q^{\prime }}\leftarrow {\theta }_i^Q$ and ${\theta }_i^{{\mu }^{\prime }}\leftarrow {\theta }_i^{\mu }$ across all i.     3: for episode $m=1$ to M do     4:     Reset exploration noise process $\sigma$.     5:     Acquire initial system state $S_1$ and observation vectors ${\mathbf{s}}_1$.     6:     for time step $t=1$ to T do     7:         Compute action $a_{i,t}={\mu }_i(s_{i,t}\mid {\theta }_i^{\mu })+{\sigma }_t$ for each agent i.     8:        Perform joint action ${\mathbf{a}}_t$, observe reward set ${\mathbf{r}}_t$, next global state $S_{t+1}$, and new observations ${\mathbf{s}}_{t+1}$.     9:         Archive experience tuple $(S_t,{\mathbf{a}}_t,{\mathbf{r}}_t,S_{t+1},{\mathbf{s}}_{t+1})$ into buffer $\mathcal{D}$.   10:         for agent $i=1$ to N do   11:             Draw a random batch of samples from $\mathcal{D}$.   12:             Compute target value $y_j$ via target networks (Equation (20)).   13:             Refine Critic $Q_i$ by minimizing the loss function (Equation (19)).   14:             Optimize Actor ${\mu }_i$ via policy gradient ascent (Equation (23)).   15:         end for   16:         Perform soft update on target parameters for all i:         ${\theta }_i^{Q^{\prime }}\leftarrow \tau {\theta }_i^Q+(1-\tau ){\theta }_i^{Q^{\prime }}$,         ${\theta }_i^{{\mu }^{\prime }}\leftarrow \tau {\theta }_i^{\mu }+(1-\tau ){\theta }_i^{{\mu }^{\prime }}$.   17:     end for   18: end for   19: Upon termination of the procedure, the MARL-CAR framework yields the converged routing policy.

4.4. MARL-CAR Implementation

Predicated on the agent training architecture previously defined, we now delineate the specific operational protocol for the coding-aware routing strategy. Figure 3 provides a schematic representation of the MARL-CAR workflow, which is segmented into four integral stages: environmental sensing, flooding-based topology discovery, the iterative training regimen, and the deployment phase.

Figure 3. Flowchart of the MARL-CAR system architecture, detailing the interaction between the training regime and the inference deployment.

Within the MARL-CAR architecture, the flooding stage acts as the linchpin for network discovery. This mechanism relies on the cyclical dissemination of HELLO beacons, enabling UAVs to share critical link-state metrics. Such data aggregation serves as a prerequisite, provisioning the necessary environmental context for both the MADDPG training loop and active routing decisions. The neighbor discovery handshake proceeds via a specific logic: When node A intercepts a HELLO transmission from node B, it locally buffers the packet and tentatively flags B as an asymmetric neighbor. Subsequently, if node B identifies its own unique ID within a reciprocal message from A, it upgrades A’s status to a symmetric neighbor. This confirmation sequence successfully instantiates a valid bidirectional communication link between the pair.

A primary advantage of the proposed MARL-CAR framework lies in its utilization of the CTDE paradigm. This architecture effectively mitigates signaling overhead, a critical concern in highly dynamic FANETs. During the centralized training phase, a global Critic network is trained offline using comprehensive state information. This process can be conducted within a simulated environment or on a ground-based server, thereby avoiding the consumption of the aerial network’s limited operational bandwidth. Upon policy convergence, only the lightweight decentralized Actor networks are deployed onto the individual UAV nodes.

The operational workflow during the training regime is strictly governed by the procedural logic delineated in Algorithm 1. This process unfolds as follows: Driven by local observations $s_{i,t}$, each agent i determines and executes a specific action $a_{i,t}$, balancing exploration and exploitation via mechanisms such as $ϵ$-greedy strategies. The execution of this joint routing decision ${\mathbf{a}}_t$ inevitably modifies the network environment, prompting a dynamic update in the link state. Subsequently, the system calculates a composite reward $r_{i,t}$ that synthesizes coding potential, throughput metrics, latency, and energy constraints. Leveraging this feedback alongside the global state $S_t$, the centralized Critic network $Q_i$ undergoes optimization via gradient descent to minimize the loss function, whereas the Actor network ${\mu }_i$ is refined using the chain rule. Concurrently, the parameters of the target networks are subjected to soft updates. This iterative loop persists until the termination conditions are satisfied, yielding a fully converged multi-agent routing system.

The execution phase commences once the training regimen has successfully converged. In this phase, each UAV independently makes routing decisions based solely on its local observations, eliminating the requirement for real-time global state synchronization. The real-time signaling overhead is restricted to standard, periodic HELLO beacon exchanges. Since neighbor discovery is inherent to geographic routing, this approach imposes no additional requirement for global consistency. Within this deployment stage, MARL-CAR leverages the fully trained decentralized actors to autonomously determine routing trajectories. Regardless of environmental volatility or flight dynamics, the framework consistently establishes links that optimize both coding potential and data transmission rates, guided by the inference of the learned policy ${\mu }_i$.

5. Results

5.1. Simulation Environment

The performance evaluation of the proposed MARL-CAR protocol was conducted utilizing the NS3-Gym framework [34]. This middleware facilitates a seamless interface between the OpenAI Gym reinforcement learning toolkit and the NS-3 network simulator, effectively bridging the gap between environmental state observations and agent actions. The simulation scenario is defined within a $2\mathrm{km}\times 2\mathrm{km}$ geographical area populated with 20 to 50 UAV nodes. To mimic realistic flight patterns, we employed the Gaussian-Markov mobility model. The network traffic is generated using Constant Bit Rate (CBR) data streams, allowing us to rigorously assess the routing protocol’s efficacy under varying load conditions.

In alignment with the formulation detailed in Section 4, the hyperparameters governing the energy-related reward are assigned as ${\alpha }_1=0.05$ and ${\alpha }_2=0.5$. Furthermore, the fundamental energy cost unit is established at $g=1$, while the discount factor applied to the reinforcement learning updates is set to $\gamma =0.9$. The weighting coefficients ${\xi }_1,{\xi }_2,{\xi }_3,{\xi }_4$ reflect the mission-specific priorities. In this study, we set ${\xi }_1=0.4$ to explicitly guide the agents toward discovering latent coding opportunities—a primary objective of MARL-CAR. The remaining weights are distributed equally ($\xi =0.2$) to provide a neutral baseline for general network performance, ensuring that no single metric becomes a bottleneck for the others. A detailed enumeration of the experimental configuration is presented in Table 1.

Table 1. Simulation Parameters.

Parameter	Value
Simulation Duration	1000 s
Simulation Area	$2\mathrm{km}\times 2\mathrm{km}$
Movement Pattern	Gaussian Markov
Number of drones	20∼50
UAV Flight Speed	$15\mathrm{m}/\mathrm{s}$
Coding Scheme	Random Linear Network Coding
Data Transmission Type	Constant Bit Rate
CBR Rate	20 k–200 kbps
Packet Size	512 bytes
Discount Factor	$\gamma =0.9$
Reward Weights	${\alpha }_1=0.05,{\alpha }_2=0.5$
Baseline Energy Consumption	$g=1$
Reward Function Coefficients	${\xi }_1=0.4$, ${\xi }_2=0.2$, ${\xi }_3=0.2$, ${\xi }_4=0.2$

To validate the performance of the proposed MARL-CAR, four representative algorithms are selected as benchmarks. It is worth noting that existing schemes predominantly rely on centralized decision-making or target network topologies that differ significantly from the dynamic environments considered in this study. To ensure a rigorous and fair performance evaluation, we re-implemented these baselines within the same NS3-Gym framework, subjecting them to identical mobility models, energy constraints, and traffic loads. This approach allows for a direct comparison between MARL-CAR’s explicit multi-metric reward mechanism and the performance of established distributed protocols.

• COPE [16]: This protocol bridges theory with practice by mixing unicast packets to boost throughput, yet its efficiency is heavily contingent on specific traffic patterns. Its decoupled mechanism—detecting coding opportunities only after path establishment—restricts its adaptability in highly dynamic FANET topologies.
• DCAR [24]: By introducing a CRM, this protocol enables coding detection along the entire path to overcome COPE’s two-hop limitation. However, the excessive signaling overhead required for path discovery in expansive regions hinders its performance in rapidly evolving aerial networks.
• TSCAR [27]: This scheme utilizes traffic shaping to align asymmetric flows and artificially create coding opportunities. Nevertheless, the inherent queuing delay caused by its active waiting mechanism often violates the stringent latency constraints of real-time UAV control tasks.
• RLCAR [35]: A single-agent RL framework developed to improve routing flexibility in dynamic environments. While it enhances adaptability, it lacks the collaborative optimization and cross-layer coding gains essential for managing complex, multi-node UAV swarms.

5.2. Reward Convergence and Learning Efficiency

To quantitatively evaluate the training effectiveness of the proposed framework, this section analyzes the cumulative reward convergence curves, which synthesize critical performance indicators such as network throughput, latency, and energy consumption. Figure 4 illustrates the dynamic evolution of reward values for the two algorithms employing reinforcement learning: the proposed MARL-CAR and the single-agent baseline RLCAR. As a sequential decision-making process modeled as a Markov Decision Process, these reward trajectories initiate from zero and exhibit distinct convergence trends across different discount factors $\gamma$. This comparison reveals the robustness and exploration capabilities of each learning architecture in optimizing network coding decisions within a dynamic Environment.

Figure 4. Cumulative reward convergence curves for MARL-CAR and RLCAR under various discount factors $\gamma$ in a 30-UAV scenario.

In the experimental evaluations conducted with a swarm density of 30 UAVs, MARL-CAR demonstrates superior learning efficiency, characterized by the steepest initial gradient and the capability to reach a steady state significantly faster than the RLCAR baseline. Specifically, the multi-agent architecture attains a higher asymptotic reward ceiling, outperforming the single-agent alternative by a substantial margin. This performance gain is primarily attributed to the CTDE paradigm. By leveraging global state information during the training phase, MARL-CAR effectively mitigates the non-stationarity issues inherent in multi-UAV environments, allowing the agents to learn a globally cooperative policy that optimizes multi-hop coding opportunities.

In contrast, while the RLCAR approach eventually achieves convergence, it exhibits inferior performance in terms of both convergence upper bounds and training stability. As evidenced by the more pronounced stochastic fluctuations in its reward curves, the single-agent model struggles to maintain a robust policy amidst the rapid topological reconfigurations and frequent link intermittency of FANETs. Furthermore, the performance of RLCAR is more sensitive to the selection of the discount factor $\gamma$, whereas MARL-CAR maintains a more resilient convergence profile. The outcomes demonstrate that the cooperative multi-agent framework of MARL-CAR effectively models the intricate non-linear dependencies linking decentralized routing decisions to system-wide efficiency, thereby achieving higher cumulative rewards relative to standalone learning schemes.

5.3. Comparative Analysis of Network Coding Performance

The coding ratio, defined as the proportion of coded packets to the total number of transmitted packets, serves as a primary metric for evaluating coding efficiency. As illustrated in Figure 5a, a positive correlation between the coding ratio and the number of UAVs is observed across all considered protocols.

Figure 5. Comparative evaluation of coding efficiency between MARL-CAR and baseline protocols under different swarm sizes. The subfigures illustrate: (a) the proportion of coded packets, (b) the successful decoding rate, and (c) the cumulative coding benefit.

Notably, MARL-CAR consistently achieves the superior performance. At a network density of 20 UAVs, MARL-CAR outperforms COPE, DCAR, TSCAR, and RLCAR by 32.23%, 18.93%, 20.35%, and 5.5%, respectively. This performance margin expands significantly as the UAV population scales, which is ascribed to the adaptive capability of MARL-CAR agents to adaptively exploit dynamic coding opportunities through the reinforcement learning mechanism.

Given the inherent volatility of wireless links, the decoding ratio (the percentage of packets successfully recovered at the destination) serves as a critical proxy for transmission stability. As depicted in Figure 5b, while MARL-CAR maintains the highest decoding rate among the cohorts, the overall decoding success rate remains relatively independent of UAV density. This phenomenon suggests that mere node density is not the sole determinant of transmission stability; rather, effectiveness is governed by the synergy between coding opportunities and link quality.

Furthermore, we evaluate routing efficiency through cumulative coding benefits. As shown in Figure 5c, MARL-CAR achieves the highest coding benefit values. A higher coding benefit indicates that more packets are transmitted via encoded piggybacking, which effectively translates to reduced resource consumption and minimized equivalent routing hops for data completion. The increasing trend in coding benefits as the network scales further validates MARL-CAR’s superiority in optimizing path selection within complex topologies.

5.4. Discussion of the Transmission Performance

Beyond the specific metrics of network coding, we extended the evaluation to assess general transmission performance metrics, including average throughput, end-to-end delay, Packet Delivery Ratio (PDR), and energy consumption per packet. In this experimental setup, the network topology consists of a fixed swarm of 30 UAVs with identical velocity and mobility models. The primary variable is the data flow rate, manipulated via the CBR, which allows us to simulate varying service demands and evaluate the network’s load-bearing capacity.

5.4.1. Throughput and Delay Analysis

Figure 6a depicts the mean throughput, quantified as the successful data transfer rate along an arbitrary trajectory. As the volume of data traffic amplifies, a consistent rising trend in throughput is observable for every protocol. In the low-load phase, the throughput values for all four protocols closely track the input CBR rate, indicating that the traffic load is within the manageable capacity of the network. However, as the flow rate escalates, a performance divergence becomes apparent. MARL-CAR consistently reflects the maximum throughput among the baselines. Although the simulation did not extend indefinitely, it is anticipated that the throughput for all protocols would eventually converge to a saturation plateau.

Figure 6. Impact of varying data flow rates on basic transmission metrics. (a) Average throughput and (b) Average delay comparison among MARL-CAR and benchmarks.

As illustrated in Figure 6b, we analyze how escalating data loads influence transmission latency. The results indicate a period of stability where all four protocols sustain negligible delay, provided the data generation rate stays under the threshold of 140 kbps. A sharp escalation in delay beyond this point suggests that a CBR rate of approximately 150 kbps approaches the optimal load threshold of the network. Notably, MARL-CAR sustains the lowest transmission delay throughout the test. This advantage is attributed to the inclusion of coding benefits within the MARL-CAR reward function; by optimizing for path efficiency, the algorithm effectively selects routes with fewer hops, thereby reducing cumulative transmission latency.

The PDR, defined as the ratio of successfully received packets at the destination to those generated by the source, is illustrated in Figure 7a. The experimental results demonstrate a clear inverse correlation between PDR and traffic load across all evaluated protocols. As the flow rate increases from 20 kbps to 200 kbps, the PDR for all schemes exhibits a declining trend due to intensified network congestion and buffer overflows. Notably, the proposed MARL-CAR consistently maintains the highest PDR across the entire flow rate range, with the performance gap widening significantly under high-load conditions. This sustained superiority stems from MARL-CAR’s ability to intelligently identify and exploit multi-hop coding opportunities through its multi-agent reinforcement learning mechanism. By optimizing coding decisions in real-time, MARL-CAR effectively reduces the number of required transmissions, thereby alleviating channel contention and expanding the network’s effective capacity. Furthermore, the collaborative nature of the agents allows for better adaptation to dynamic traffic patterns, mitigating packet drops under heavy load and bolstering overall delivery reliability in high-density UAV environments.

Figure 7. Evaluation of network reliability and efficiency. (a) Packet delivery ratio and (b) Energy consumption per packet under varying data flow rates.

5.4.2. Reliability and Energy Efficiency

Finally, we analyzed the energy cost associated with transmitting a single packet, as shown in Figure 7b. The results indicate a non-linear U-shaped relationship where the energy cost per packet is minimized at intermediate flow rates (around 120 kbps). At low traffic volumes, the cost is higher due to the static power consumption being distributed over fewer packets, while at high-load territories, congestion-induced retransmissions drive the energy expenditure upward. Across all scenarios, MARL-CAR demonstrates superior energy efficiency. Notably, at the peak load of 200 kbps, MARL-CAR maintains an energy cost of approximately $5.8\times {10}^3$ J/Packet, significantly lower than COPE’s $8.4\times {10}^3$ J/Packet. This advantage is attributed to MARL-CAR’s intelligent coding-aware mechanism, which maximizes the information per transmission. By effectively reducing redundant packets, it alleviates the energy strain on battery-constrained UAV nodes while maintaining robust PDR performance.

6. Conclusions

In conclusion, this paper presented MARL-CAR, a multi-agent deep reinforcement learning-based routing protocol that effectively addresses the challenges of high mobility and dynamic topologies in FANETs. By modeling the routing process as a cooperative decision-making task, we developed a composite reward mechanism that balances coding opportunities, throughput, energy equilibrium, and end-to-end latency. Comprehensive simulations within the NS3-Gym environment demonstrate that MARL-CAR significantly outperforms state-of-the-art benchmarks in terms of coding efficacy, transmission reliability, and energy efficiency. However, we acknowledge that these simulation-based results provide approximations of complex aerial environments and may inherit inherent modeling errors. To enhance the rigor of our findings, subsequent research will focus on integrating exact solutions from analytical models to further validate and calibrate the learned neural networks. Furthermore, we plan to transition from synthetic environments to real-world UAV testbeds to verify the protocol’s robustness and adaptability across heterogeneous topologies in practical mission-critical scenarios.