Energy-Harvesting-Assisted UAV Swarm Anti-Jamming Communication Based on Multi-Agent Reinforcement Learning

Highlights

What are the main findings?

• This paper proposes an energy-harvesting-assisted anti-jamming communication framework for unmanned aerial vehicle (UAV) swarm networks, aiming to optimize the long-term trade-off between transmission success rate and energy consumption.
• A multi-agent reinforcement learning-based solution is developed, enabling each cluster head to independently learn its channel selection, power control, and energy-harvesting time allocation policies through sharing spatiotemporal correlations among neighboring nodes.

What are the implications of the main findings?

• The proposed solution shows superior performance in simulations, achieving a higher transmission success rate with lower energy consumption across varying numbers of channels.
• The research provides a novel anti-jamming framework for intelligent spectrum management in dynamic jamming environments, offering guidance for the design of practical anti-jamming and energy-efficient UAV networks.

Abstract

Considering that the unmanned aerial vehicles (UAVs) are susceptible to both co-channel interference and malicious jamming with limited onboard battery energy, this paper proposes an energy-harvesting-assisted anti-jamming communication framework for UAV swarm networks. Specifically, we first model the problem as a decentralized partially observable Markov decision process (Dec-POMDP), aiming to achieve a long-term trade-off between data transmission success rate and energy consumption. Then we propose a multi-agent independent advantage actor–critic (IA2C)-based energy-harvesting-assisted anti-jamming communication solution, which enables each cluster head (CH) to learn its transmit channel, power, and energy harvesting time policy independently. By constructing a time-space-based extended Dec-POMDP, the spatiotemporal correlations among neighboring nodes are learned by allowing adjacent agents to share discounted local observations. Extensive simulations show that, compared with the benchmark schemes, the proposed scheme improves the average cumulative reward and average cumulative success rate by 17.26% and 10.37%, respectively, while achieving a higher transmission success rate with lower energy consumption under different numbers of available channels.

1. Introduction

1.1. Motivation and Contribution

With the rapid development of wireless communication technology, the Internet of unmanned aerial vehicles (I-UAVs) has been widely deployed due to its scalability and flexibility [1,2,3]. In many scenarios, I-UAVs rely on UAV-to-UAV links for communication via short-range line-of-sight transmissions [4,5]. However, due to the open nature of the wireless spectrum, I-UAVs are prone to external malicious jamming as well as intra-cluster co-channel interference, which degrades communication reliability [6,7]. Therefore, mitigating jamming and interference is crucial for ensuring reliable communications in I-UAV applications [8].

Recent advances in artificial intelligence have accelerated the adoption of multi-agent reinforcement learning (MARL) in intelligent anti-jamming systems [9,10]. Even without prior knowledge of the jammer, UAVs can learn effective anti-jamming policies through interaction with the environment. However, in typical multi-UAV scenarios, a centralized single-agent reinforcement learning architecture often suffers from poor scalability. In contrast, independent MARL enables each UAV to learn a policy based on its local observations, yet the overall anti-jamming performance can be compromised by partial observability of the environment and the lack of information exchange among agents, which may further lead to unstable learning. To address these limitations, collaborative multi-agent anti-jamming communication schemes have been proposed, where agents can collaborate by sharing training information to learn their own policies [11,12,13,14].

Additionally, as UAVs are primarily battery-powered, the limited onboard energy budget imposes a fundamental constraint on mission execution. Moreover, due to energy limitations, UAVs cannot always transmit at the desired power level, which may reduce the robustness of UAV anti-jamming communication. Therefore, it is essential to jointly consider energy consumption and reliable low-latency communication in UAV networks [15,16].

Motivated by these facts, we investigate the joint optimization problem of spectrum allocation, power control, and the time-splitting ratio between energy harvesting and data transmission for UAV swarm anti-jamming communication. In particular, we establish an energy-harvesting-assisted multi-agent anti-jamming communication framework to achieve a long-term trade-off between the data transmission success rate and transmission energy consumption. The major contributions are summarized as follows:

• We formulate a multi-domain energy-efficient anti-jamming communication optimization problem to achieve a long-term trade-off between data transmission success rate and energy consumption. Furthermore, this problem is modeled as a finite-horizon decentralized partially observable Markov decision process (Dec-POMDP).
• We propose an energy-harvesting-assisted anti-jamming communication scheme based on the multi-agent independent advantage actor–critic (IA2C) approach. In this scheme, each cluster head (CH) obtains discounted observations from neighboring CHs by applying a spatial discount factor based on the network topology. As a result, each CH agent can sequentially reduce the influence of decisions made by other CH agents on its own anti-jamming decision-making. Moreover, each CH maintains its own actor–critic network and performs distributed training using advantage-based updates.
• Simulation results show that the proposed scheme effectively improves the data transmission success rate while reducing energy consumption. In addition, the proposed scheme consistently achieves superior performance under varying numbers of channels.

1.2. Related Work

In recent years, anti-jamming and energy consumption issues in UAV networks have attracted extensive attention. To improve the overall anti-jamming performance of UAV swarms, researchers have increasingly focused on cooperative multi-agent reinforcement learning approaches [17]. For example, Yao et al. of [11] proposed a collaborative anti-jamming scheme that considered only inter-user interference. By exchanging the training information among users, the proposed scheme enhances the system’s anti-jamming capability. Additionally, Yan et al. of [18] proposed a cooperative MARL-based anti-jamming framework. By sharing their current channel selections and local learning experiences, the UAV pairs can cooperatively learn their own policies by relying on previously observed information and predicting the actions of others.

Many conventional approaches reduce communication energy consumption via efficient resource allocation or protocol and algorithm optimization, yet they do not fundamentally address energy scarcity in UAV networks [19,20]. Energy harvesting enables UAVs to scavenge ambient energy for data transmission, reducing reliance on conventional power supplies and supporting flexible deployment and reliable communication [21]. For example, Sekander et al. of [22] proposed that UAVs can harvest energy from solar and wind sources. In [23], Yang et al., demonstrated that low-altitude UAVs can act as radio frequency (RF) energy transmitters to power nearby energy-constrained devices. Moreover, Peng et al. of [24] proposed a resource allocation scheme based on simultaneous wireless information and power transfer for energy saving and endurance optimization in UAV communication, where UAVs can perform both energy harvesting and data transmission through a dedicated passive reflecting array.

To enhance the reliability of UAV communication networks and improve system energy efficiency under jamming, energy-harvesting-assisted anti-jamming schemes have been extensively investigated. For example, Yang et al. of [25] studied a maritime communication system assisted by a UAV-mounted reconfigurable intelligent surface under malicious jamming. They further proposed an adaptive energy-harvesting scheme for simultaneous information transmission and energy collection, thereby improving the anti-jamming robustness and endurance of UAVs. In [26], Dang-Ngoc et al., considered the limited onboard energy of UAVs and developed a cooperative time-switching relay protocol, enabling UAVs to jointly harvest wireless energy, forward information, and generate artificial interference against eavesdroppers. Moreover, since jamming signals are also RF signals, Ji et al. of [27] exploited the primary user’s transmitted signal as a dedicated RF source for UAV selection and energy harvesting. However, in UAV scenarios, energy-harvesting-based anti-jamming research has largely been confined to relay-assisted settings, where UAVs act as relays to amplify and forward the received signals [28], rather than being directly applied to UAV anti-jamming communication networks.

2. System Model and Problem Formulation

In this section, we elaborate on the anti-jamming communication model for UAV swarms, including the network model, wireless transmission model, mobility model, energy harvesting model, and optimization problem.

2.1. Network Model

As shown in Figure 1, the energy-harvesting-assisted UAV swarm anti-jamming communication system consists of M clusters, N jammers, and a fixed ground energy source, where each cluster consists of a CH UAV, and I cluster-member (CM) UAVs. Assume that all the clusters form a simple logical queueing network topology $\mathrm{G}(\mathcal{M},\epsilon )$, where $\mathcal{M}=\left\{1,2,\dots ,M\right\}$ denotes the set of all CHs and satisfies the logical order structure of $1–2–\cdots –M$, and $\epsilon$ is the set of communication links between all CHs in the topological network. ${\mathcal{N}}_m=\{j\in \mathcal{M}|(j,m)\in \epsilon \}$ denotes the set of all neighboring CHs of CH m with ${\mathcal{M}}_m={\mathcal{N}}_m\cup m$. We adopt a fixed sweep jamming pattern, where each jammer periodically scans all communication links according to a preset channel. The system operates relying on multiple time slots (TSs) with the duration $\tau$ and an index set of $\mathcal{T}=\{1,2,\dots ,T\}$, where the environment remains unchanged during each TS. In addition, within the considered region, the ground energy source is located at a fixed center position [29]. For highly mobile and large-scale missions, the model can be naturally extended by deploying multiple energy stations across different sub-regions.

2.2. Wireless Transmission Model

Due to the scarcity of spectrum resources in the complex electromagnetic environment, the frequency band in the system can be divided into L mutually orthogonal channels, and all UAVs and jammers jointly multiplex these L channels. Assume that different UAVs can reuse the same channel, but the CH can only select a single channel to communicate with its CM in each TS. Let ${\mathrm{g}}_{m,i}^{t,l}$ denote the channel gain from CH m to its CM i on channel l in TS t, which is determined by Rayleigh small-scale fading and large-scale path loss. Then, the channel gain between CH m and its CM i over all channels in TS t is denoted as ${\mathit{G}}_{m,i}^t={\left[g_{m,i}^{t,l}\right]}_{l=1:L}$. Similarly, let ${\overline{g}}_n^{t,l}$ denote the jamming channel gain from jammer n on channel l in TS t. It is assumed that each CH knows only the channel gains of its own intra-cluster links over all channels, but cannot predict the transmission conditions of other UAV clusters in advance.

Let us denote the selected transmit channel and transmit power from CH m to its CM i in TS t as $l_{m,i}^t$ and $p_{m,i}^t$, respectively. Then, the transmit channel and power vector of CH m in TS t can be given by ${\mathit{L}}_m^t={\left[l_{m,i}^t\right]}_{i=1:I}$ and ${\mathit{P}}_m^t={\left[p_{m,i}^t\right]}_{i=1:I}$, respectively. In addition, the jamming channel and power from jammer n in TS t are denoted as ${\overline{l}}_n^t$ and ${\overline{p}}_n^t$, respectively. Then, the jamming channel and power vector of jammer n in TS t can be given by ${\overline{\mathit{L}}}^t={\left[{\overline{l}}_n^t\right]}_{n=1:N}$ and ${\overline{\mathit{P}}}^t={\left[{\overline{p}}_n^t\right]}_{n=1:N}$, respectively. Moreover, since all links share the limited spectrum resources, the transmission between CH m and its CM i on channel l in TS t is subject to three types of interference: intra-cluster co-channel interference, inter-cluster co-channel interference, and external malicious jamming. Denote the resulting total interference by $V_{m,i}^{t,l}$, i.e.,

$$V_{m,i}^{t,l}=\sum _{k=1}^I\mathit{1}\{l_{m,k}^t=l,k\ne i\}{p}_{m,k}^t{g}_{m,k}^{t,l}+\sum _{q=1}^M\sum _{i=1}^I\mathit{1}\{l_{q,i}^t=l,q\ne m\}{p}_{q,i}^t{g}_{q,i}^{t,l}+\sum _{n=1}^N\mathit{1}\{{\overline{l}}_n^t=l\}{\overline{p}}_n^t{\overline{g}}_n^{t,l}$$

(1)

Note that $\mathit{1}\{·\}$ is the indicator function, which equals 1 if the condition is satisfied and 0 otherwise.

The data transmission rate between CH m and its CM i on channel l in TS t can be expressed as

$$R_{m,i}^{t,l}={B\log }_2(1+{\displaystyle \frac{p_{m,i}^t{g}_{m,i}^{t,l}}{{\sigma }^2+V_{m,i}^{t,l}}})$$

(2)

where ${\sigma }^2$ is the noise variance of the additive white Gaussian noise (AWGN) and B is the bandwidth of each channel.

Considering that a jammer may switch its jamming channel during the data transmission time, the transmission time of CH m in TS t, denoted as $T_m^{\mathrm{S},t}$, can be divided into two time periods, $T_1$ and $T_2$, such that $T_1+T_2=T_m^{\mathrm{S},t}$. Let $V{1}_{m,i}^{t,l}$ and $V{2}_{m,i}^{t,l}$ denote the total interference on the link between CH m and its CM i on channel l in TS t during $T_1$ and $T_2$, respectively. Accordingly, the achievable data rate differs between the two interference conditions. Let $R{1}_{m,i}^{t,l}$ and $R{2}_{m,i}^{t,l}$ denote the data transmission rates between CH m and its CM i on channel l in TS t during $T_1$ and $T_2$, respectively. Then the maximum number of bits that can be transmitted from CH m to its CM i on channel l in TS t is given as

$$K_{m,i}^{t,l}=T_m^{\mathrm{S},t}{R}_{m,i}^{t,l}$$

(3)

If the jammer switches jamming channels during the data transmission phase, then

$$K_{m,i}^{t,l}=T_{1}R{1}_{m,i}^{t,l}+T_{2}R{2}_{m,i}^{t,l}$$

(4)

2.3. Mobility Model

We assume that all UAVs and jammers move within the same range and synchronize their position information updates at the beginning of each TS. Each CH and jammer follows a 3D Gaussian Markov mobility model [30], and its trajectory can be expressed as

$$\left\{\begin{array}{c}\begin{array}{c}{v}_{m\left(n\right)}^t=\mu v_{m\left(n\right)}^{t-1}+(1-\mu ){\overline{v}}_{m\left(n\right)}^t+\sqrt{1-{\mu }^2}\Delta v_{m\left(n\right)}^t\hfill \end{array}\hfill \\ \begin{array}{c}{d}_{m\left(n\right)}^t=\mu d_{m\left(n\right)}^{t-1}+(1-\mu ){\overline{d}}_{m\left(n\right)}^t+\sqrt{1-{\mu }^2}\Delta d_{m\left(n\right)}^t\hfill \end{array}\hfill \\ \begin{array}{c}{p}_{m\left(n\right)}^t=\mu p_{m\left(n\right)}^{t-1}+(1-\mu ){\overline{p}}_{m\left(n\right)}^t+\sqrt{1-{\mu }^2}\Delta p_{m\left(n\right)}^t\hfill \end{array}\hfill \end{array}\right.$$

(5)

where $v_{m\left(n\right)}^t$, $d_{m\left(n\right)}^t$, and $p_{m\left(n\right)}^t$ denote the velocity magnitude, horizontal angle, and spatial angle of CH m/jammer n in TS t, respectively. ${\overline{v}}_{m\left(n\right)}^t$, ${\overline{d}}_{m\left(n\right)}^t$, and ${\overline{p}}_{m\left(n\right)}^t$ denote the corresponding mean values. Additionally, $\Delta v_{m\left(n\right)}^t$, $\Delta d_{m\left(n\right)}^t$, and $\Delta p_{m\left(n\right)}^t$ denote the corresponding random deviations, all of which obey Gaussian random distribution with zero mean and variance ${\sigma }_M^2$. $\mu \in [0,1]$ is the correlation coefficient, which quantifies the degree of randomness in mobility.

The trajectory of each CM follows the reference point (RP) mobility model [31]. Each CM moves within a radius of $R_{\mathrm{CM}}$ centered on the RP, where the RP is determined within a radius of $R_{\mathrm{RP}}$ centered on each CH. Assuming that the maximum communication distance between a CH and its CM is $D_{\max }$, the sum of the RP mobility radius and the CM mobility radius should not exceed $D_{\max }$, i.e., $R_{\mathrm{CM}}+R_{\mathrm{RP}}\le D_{max}$.

2.4. Energy Harvesting Model

Considering the limited onboard battery energy of UAVs, the communication performance among UAVs may be affected. We assume that each CH is equipped with an energy-harvesting (EH) circuit-based RF receiver architecture, which uses semiconductor rectifying elements to convert the RF signal transmitted by the energy source into electrical energy for data transmission. Specifically, the RF energy harvesting module consists of multiple receiving antennas, a band-pass filter, a rectifier circuit, and a low-pass filter. To reduce computational complexity, we adopt a linear energy-harvesting model [32], in which the harvested energy $E^{\mathrm{H}}$ is assumed to be proportional to the received RF power from the energy source, i.e.,

$$E^{\mathrm{H}}=\eta T^{\mathrm{H}}{P}_{\mathrm{RF}}$$

(6)

where $\eta$ is the fixed linear RF-energy conversion efficiency of the rectifier circuit, $T^{\mathrm{H}}$ is the energy harvesting time, and $P_{\mathrm{RF}}$ is the RF signal receiving power.

In addition, we adopt a time-switching-based receiver architecture [33], where each CH is equipped with a dedicated RF energy harvester and an RF information receiver. Since energy harvesting and data transmission cannot be performed simultaneously, each CH must decide the time allocation between energy harvesting and data transmission. Taking CH m as an example, let $T_m^{\mathrm{H},t}$ denote the energy-harvesting duration of CH m during the action phase in TS t, which satisfies $T_m^{\mathrm{H},t}+T_m^{\mathrm{S},t}=T_{\mathrm{ACT}}$. The energy $E_m^{\mathrm{H},t}$ harvested from the energy source is given by

$$E_m^{\mathrm{H},t}=\eta T_m^{\mathrm{H},t}{P}_{\mathrm{RF}}$$

(7)

Let ${\omega }_m^t$ denote the fraction of the execution-phase duration allocated to energy harvesting for CH m in TS t, i.e.,

$${\omega }_m^t={\displaystyle \frac{T_m^{\mathrm{H},t}}{T_{\mathrm{ACT}}}}$$

(8)

In addition, we assume that all CMs within a cluster have the same energy-harvesting duration and data-transmission duration to ensure synchronized intra-cluster transmissions.

2.5. Problem Formulation

To simultaneously maximize the total number of transmitted bits and minimize the total transmission energy consumption, we formulate an energy-harvesting-assisted optimization problem for anti-jamming communication. Specifically, each CH determines a time-allocation factor between energy harvesting and data transmission to balance harvested energy and communication energy. Moreover, the anti-jamming performance is improved by jointly optimizing spectral-domain channel selection and power-domain power control. Accordingly, the resulting optimization problem is formulated as

$$\begin{array}{cc}\hfill & \underset{({\mathit{L}}^t,{\mathit{P}}^t,{\mathit{\omega }}^t)}{max}\mathbb{E}\left[\sum _{t=1}^T\sum _{m=1}^M\left({\displaystyle \frac{K_m^t}{E_m^{S,t}}}\right)\right]\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & s.t.K_{m,i}^t\ge K,\forall m,\forall i,\forall t\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & E_m^{\mathrm{R},t-1}+E_m^{\mathrm{H},t}\ge E_m^{\mathrm{S},t},\forall m,\forall t\hfill \end{array}$$

(11)

where ${\mathit{\omega }}^t=\left[{\omega }_1^t,\cdots ,{\omega }_M^t\right]$ denotes the energy-harvesting time fraction of all CHs in TS t with respect to the execution-phase duration. $K_m^t={\sum }_{i=1}^I{K}_{m,i}^t$ denotes the total number of successfully transmitted bits by CH m in TS t, and K denotes the minimum required number of transmitted bits per TS. $E_m^{\mathrm{S},t}={\sum }_{i=1}^I{E}_{m,i}^{\mathrm{S},t}$ denotes the total transmission energy consumption of CH m in TS t. Moreover, $E_m^{\mathrm{R},t-1}$ denotes the residual energy of CH m at the end of TS $t-1$, which depends on the harvested energy and the transmission energy consumption in TS $t-1$, as well as the residual energy in TS $t-2$, i.e.,

$$E_m^{\mathrm{R},t-1}=min\left(E_m^{\mathrm{R},t-2}+E_m^{\mathrm{H},t-1}-E_m^{\mathrm{S},t-1},E_{\max }\right)$$

(12)

where $E_{\max }$ is the maximum battery capacity of each UAV.

Constraint (10) indicates that, in each TS, the total number of bits transmitted by each CH to all its CMs must be no less than the minimum required number of transmitted bits K. Constraint (11) indicates that, in each TS, the residual energy of each CH from the previous TS, together with the energy harvested in the current TS, must be sufficient to support data transmission in the current TS.

3. Proposed Energy-Harvesting-Aided Anti-Jamming Communication Framework

In this section, we first introduce the multi-agent anti-jamming communication framework, and then formulate the anti-jamming communication problem as a Markov game.

As shown in Figure 2, the multi-agent anti-jamming communication framework for UAV swarms is presented. From the perspectives of UAV pairs, the anti-jamming communication procedure can be divided into the following four stages:

3.1. Framework

• Wideband Spectrum Sensing Phase $T_{\mathrm{WBSS}}$: Each CH first performs wideband spectrum sensing to perceive the current spectral environment and obtain relevant channel information, while also collecting its local state information for decision-making.
• Decision-making phase $T_{\mathrm{ACK}}$: Each CH allocates the time between energy harvesting and data transmission based on the observed intra-cluster channel information and its residual energy, and determines the transmission channel and transmit power for communicating with its CMs.
• Execution phase $T_{\mathrm{ACT}}$: Each CH first harvests energy during the energy-harvesting phase according to the selected time-splitting ratio. After the harvesting interval, it switches to the transmission phase and initiates synchronized data transmission to all its CMs on the selected channel with the selected transmit power.
• Learning Phase $T_{\mathrm{LEARN}}$: The CHs collaboratively learn the jammer’s channel-interference patterns and the energy-loss levels from environmental feedback, thereby providing training information for energy-efficient anti-jamming decisions in the next TS.

By enabling each CH agent to obtain discounted observations from neighboring CHs and to fuse them with local observations, the proposed multi-agent anti-jamming communication framework allows each CH to independently determine intra-cluster communication resource allocation and the energy-harvesting time ratio, while providing learning experiences for the upcoming TS.

3.2. Dec-POMDP Design

Due to the limited observation capability of UAVs and the highly dynamic UAV network topology, we propose a multi-agent reinforcement learning framework and model the optimization problem as a Dec-POMDP, denoted as $\left\{G,\mathcal{S},\mathcal{A},\mathcal{O},P,R\right\}$. where G represents the UAV network topology, $\mathcal{S}$ is the global observation space, $\mathcal{A}$ is the joint action space, $\mathcal{O}$ is the joint observation space, P is the observation transition probability, and R is the system reward function.

3.2.1. Observation Space

We assume that each CH can acquire channel information via wideband spectrum sensing. However, due to limited sensing capability, UAVs cannot obtain global information. Therefore, the observation space of each CH is characterized by the following information: the transmission channel states for the intra-cluster communications between the CH and all CMs in the current TS, and the residual energy of the CH itself in the current TS. Accordingly, the observation of CH m in TS t, denoted as ${\mathit{o}}_m^t$, can be expressed as

$${\mathit{o}}_m^t=\left[{\mathit{G}}_m^t,E_m^{\mathrm{R},t}\right]$$

(13)

where ${\mathit{G}}_m^t={\left[{\mathit{G}}_{m,i}^t\right]}_{i=1:I}$. Accordingly, ${\mathcal{O}}_m\stackrel{\Delta }{=}\left\{{\mathit{o}}_m^t\right\}$ denotes the observation space of CH m. Then, the joint observation space can be expressed as $\mathcal{O}\stackrel{\Delta }{=}{\mathcal{O}}_1\times \cdots \times {\mathcal{O}}_M$.

Let $s^t$ denote the global observation in TS t, including the transmission channel states for the communication between all CHs and all CMs, the residual energies of all CHs, and the jamming channel information. Accordingly, ${\mathit{s}}^t$ can be expressed as

$${\mathit{s}}^t=\left[{\mathit{G}}^t,{\mathit{E}}^{R,t},\overline{{\mathit{L}}^t}\right]$$

(14)

where ${\mathit{G}}^t=\left[{\mathit{G}}_1^t,\cdots ,{\mathit{G}}_M^t\right]$ denotes the channel gain vector for all intra-cluster communications in TS t, and ${\mathit{E}}^{\mathrm{R},t}=\left[E_1^{\mathrm{R},t},\cdots ,E_M^{\mathrm{R},t}\right]$ denotes the current residual-energy vector of all CHs in TS t. The global observation space $\mathcal{S}$ is denoted as $\mathcal{S}\stackrel{\Delta }{=}\left\{{\mathit{s}}^t\right\}$.

3.2.2. Action Space

Each UAV cluster is modeled as an agent, where the CH independently determines the communication channel and transmit power for data transmission to all CMs. Moreover, during the execution phase, to ensure sufficient communication energy for serving all CMs, the CH allocates time between energy harvesting and data transmission to balance harvested energy and transmission energy consumption. Accordingly, the action space of the CH agent consists of channel selection and power allocation for transmissions to its CMs, as well as time allocation between energy harvesting and data transmission in the execution phase. Then the action of the CH m in TS t can be expressed as

$${\mathit{a}}_m^t=\left[{\mathit{L}}_m^t,{\mathit{P}}_m^t,{\mathit{\omega }}_m^t\right]$$

(15)

Accordingly, the action space of CH m is defined as ${\mathcal{A}}_m\stackrel{\Delta }{=}\left\{{\mathit{a}}_m^t\right\}$. Then, the joint action space is defined as $\mathcal{A}\stackrel{\Delta }{=}{\mathcal{A}}_1\times \cdots \times {\mathcal{A}}_M$.

3.2.3. System Reward

According to the optimization problem (9)–(11), under the constraint that sufficient communication energy is guaranteed, we aim to maximize the total number of transmitted bits over all data-transmission links while minimizing the total energy consumption over all links. Considering the significant magnitude difference between the transmitted-bit number and the transmission energy consumption, exponential normalization is adopted to reduce the scale gap while preserving the relative trends of the two terms, thereby promoting smoother and more reliable convergence. The reward is then defined as the ratio of the normalized transmitted-bit number to the normalized transmission energy consumption, thereby improving reward balance and training stability. Taking the CH m as an example, the reward in TS t is expressed as

$$r_m^t=\left\{\begin{array}{cc}{\displaystyle \frac{r\left(K_m^t\right)}{r\left(E_m^{\mathrm{S},t}\right)}},\hfill & {\displaystyle \sum _{i=1}^I}{E}_{m,i}^{\mathrm{S},t}\le E_m^{\mathrm{R},t-1}+E_m^{\mathrm{H},t}\hfill \\ \xi ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(16)

Here, to ensure that each CH performs effective data transmission subject to a communication-energy constraint, we set $\xi <0$ as the penalty factor. Specifically, when the available communication energy of CH m is insufficient, the environment assigns a penalty.

Moreover, $r\left(E_m^{\mathrm{S},t}\right)$ is the utility function of the transmission-phase energy consumption of CH m in TS t, which is expressed as:

$$r\left(E_m^{\mathrm{S},t}\right)=exp\left(-{\displaystyle \frac{{\eta }_E}{{\sum }_{i=1}^I{E}_{m,i}^{\mathrm{S},t}}}\right)$$

(17)

where ${\eta }_E$ is the weight coefficient.

Similarly, $r\left(K_m^t\right)$ denotes the utility function of the number of transmitted bits from CH m to all its CMs in TS t. Assuming that the communication energy in the current TS is sufficient, the reward function for the number of transmitted bits of CH m in TS t is given by

$$r\left(K_m^t\right)=\left\{\begin{array}{cc}exp\left(-{\displaystyle \frac{{\eta }_K}{K_m^t}}\right),\hfill & K_m^t\ne 0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(18)

where ${\eta }_K$ is the weight coefficient, $K_m^t={\sum }_{i=1}^{I}1\left\{K_{m,i}^t\ge K\right\}{K}_{m,i}^t$ denotes the total number of successfully transmitted bits from CH m to all its CMs in TS t. If the amount of data transmitted $K_{m,i}^t$ from CH m to CM i is no less than the minimum packet threshold K, the transmission is regarded as successful.

In addition, the system reward obtained from the environment in TS t is defined as the average reward over all CH agents, i.e.,

$$r^t={\displaystyle \frac{1}{M}}\sum _{m=1}^M{r}_m^t$$

(19)

Based on the proposed Dec-POMDP, the objective of the optimal global policy is to collaboratively maximize the expected discounted cumulative system reward, expressed as

$$\begin{array}{c}\hfill {\Pi }^{*}\in \arg \underset{\Pi }{\max }\mathbb{E}\left[\sum _{t\in \mathcal{T}}{\gamma }^{t-1}{r}^t\right]\end{array}$$

(20)

where $\gamma \in [0,1)$ is the discount factor that reflects the importance of future system rewards, and $\Pi =[{\pi }_1,\dots ,{\pi }_M]$.

4. Proposed Energy-Harvesting-Assisted Anti-Jamming Communication Scheme Based on Multi-Agent IA2C

In this section, we introduce a spatial discount factor to extend the existing Dec-POMDP model into the spatial domain. Based on the network topology, the CHs exchange observation information with their neighboring CHs. We then present the proposed multi-agent IA2C anti-jamming communication scheme.

4.1. Time–Space-Based Extended Dec-POMDP Model

Under the current UAV topology, we assume that each CH can not only observe its local observation information but also obtain the discounted observation information from neighboring CHs. Therefore, the observation information ${\tilde{\mathit{o}}}_m^t$ of the CH agent m in TS t can be expressed as the concatenation of the local observation information and the discounted neighboring information, i.e.,

$${\tilde{\mathit{o}}}_m^t={\mathit{o}}_m^t\cup \alpha {\left\{{\mathit{o}}_j^t\right\}}_{j\in {\mathcal{N}}_m}$$

(21)

where $\alpha$ is a spatial discount factor introduced to balance the greedy control and cooperative control of each agent. When $\alpha \to 1$, each agent can access the full observations of its neighboring agents, thereby exhibiting global cooperation performance. When $\alpha \to 0$, the rewards contributed by more distant agents are increasingly attenuated, so that each agent focuses more on its local surroundings and tends to follow a more greedy policy. When $0<\alpha <1$, each agent strikes a balance between local decision-making and cooperation with neighboring agents. Thereafter, each CH agent executes actions according to its own actor network, and accordingly obtains its local reward $r_m^t$ and global reward $r^t$.

In the fully collaborative scenario, all CH agents will receive the same global reward. However, by introducing the spatial discount factor $\alpha$, the influence of other CH agents’ decisions can be sequentially reduced on each CH’s anti-jamming decision according to the network topology. Let $d(m,j)=\left|m-j\right|$ denote the absolute logical distance between CH m and CH j. The logical distance is obtained based on the predefined logical order among CHs. Then, the global reward ${\tilde{r}}_m^t$ of CH m in TS t can be expressed as

$${\tilde{r}}_m^t=\sum _{d=0}^{M-1}\left(\sum _{j\in \mathcal{M}\left|d(m,j)=d\right.}{\alpha }^d{r}_j^t\right)$$

(22)

4.2. Energy-Harvesting-Assisted Multi-Agent IA2C Anti-Jamming Communication Algorithm

Based on the spatio-temporal extended Dec-POMDP model, and considering the difficulty of fully sharing global information, this section proposes an energy-harvesting-assisted UAV swarm multi-agent IA2C anti-jamming communication scheme. This scheme follows a centralized training and distributed execution (CTDE) framework, as shown in Figure 3, where the CH of each UAV cluster acts as an agent. Each CH integrates the observations of neighboring CHs with its own local observation through a spatial discount factor, and maintains an independent local actor network. Based solely on the observation information, each CH independently makes decisions on time allocation, channel selection, and power control. In addition, each agent maintains its own critic network, which is updated during training using its local observation and the joint actions of all CH agents. The detailed training procedure is provided in Algorithm 1.

Algorithm 1 Proposed Energy-Harvesting-Assisted Multi-Agent IA2C Anti-Jamming Algorithm

1: Input: UAV network parameters, maximum number of training episodes E, maximum number of training steps T, sampling batch size $\Xi$.   2: Initialize neural network parameters for each CH.   3: for $e=1$ to E do   4:     Each CH obtains its current position information, intra-cluster communication channel information, and initial energy state.   5:     for $t=0$ to $T-1$ do   6:         for $m=1$ to M do   7:             Obtain the current observation ${\mathit{o}}_m^t$, and then compute ${\tilde{\mathit{o}}}_m^t$ using the spatial discount factor $\alpha$ based on the network topology.   8:             Sample an action ${\mathit{a}}_m^t$ according to the policy ${\pi }_m^t$.   9:             Obtain other agents’ actions and the reward $r_m^t$ from environmental feedback, and then compute the global reward ${\tilde{r}}_m^t$ according to the network topology. 10:             Obtain the updated cluster positions and the next observation ${\tilde{\mathit{o}}}_m^{t+1}$. 11:             Store the interaction transition $〈{\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t,{\tilde{\mathit{o}}}_m^{t+1},{\tilde{r}}_m^t,{\nu }_m^t〉$ into the local replay buffer. 12:         end for 13:         Each CH updates the critic network parameters according to Equation (26). 14:         Each CH updates the actor network parameters according to Equation (31). 15:     end for 16: end for 17: Output: The actor network parameters ${\left\{{\theta }_m\right\}}_{m\in \mathcal{M}}$ and critic network parameters ${\left\{{\vartheta }_m\right\}}_{m\in \mathcal{M}}$.

During distributed execution, the CH of each UAV cluster makes decisions on energy harvesting and resource allocation based solely on its local observations. Taking CH m as an example, in TS t, it first obtains the observation ${\tilde{\mathit{o}}}_m^t$ via spectrum sensing. Then, CH m feeds ${\tilde{\mathit{o}}}_m^t$ into its local actor network to produce the corresponding action probability distribution ${\pi }_m^t$, from which it samples and executes an action ${\mathit{a}}_m^t$. Specifically, ${\mathit{a}}_m^t$ specifies the time allocation ratio between energy harvesting and data transmission, the selected channel for data transmission, and the transmit power. Finally, the environment assigns CH m a global spatial reward ${\tilde{r}}_m^t$ according to the network topology and the absolute logical distance between CH m and other CHs.

During the training process, the CH of each cluster continuously interacts with the environment and stores the generated experience trajectories ${\displaystyle \langle {\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t,{\tilde{\mathit{o}}}_m^{t+1},{\tilde{r}}_m^t,{\nu }_m^t\rangle },m=\{1,2,\cdots ,M\}$ in its respective replay buffer. By utilizing a replay buffer, the temporal correlation between consecutive experiences is effectively mitigated, which not only improves sample efficiency but also bolsters the overall stability of the training process. It should be noted that the algorithm is an off-policy learning method, so each experience pool can obtain the joint action vector of all CH agents, i.e., ${\mathit{a}}^t=[a_1^t,\cdots ,a_M^t]$. Each CH $m(m\in \mathcal{M})$ uses its local observation ${\mathit{o}}_m^t$ together with the joint action ${\mathit{a}}^t$ to compute the action-value function. When the number of effective interaction trajectories reaches the maximum capacity of the replay buffer, these samples are used to train the local actor network and the critic network. The specific steps are as follows:

(1) Let $\mathcal{D}$ denote the experience replay buffer, and $\mathcal{B}\in \mathcal{D}$ represent a mini-batch sampled from it, with size $\left|\mathcal{B}\right|$. For a K-step transition starting from TS t, define $t_K=t+K$. The cumulative discounted reward ${\widehat{R}}_m^t$ in TS t is given by:

$${\widehat{R}}_m^t=\sum _{\tau =t}^{t_K-1}{\gamma }^{\tau -t}{\tilde{r}}_m^{\tau }$$

(23)

Then, the local state–action value target is computed using the temporal-difference (TD) method as

$${\tilde{R}}_m^t={\widehat{R}}_m^t+{\gamma }^K{\nu }_t$$

(24)

where ${\nu }_t=Q_{{\vartheta }_m^{-}}({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t)$, and ${\vartheta }_m^{-}$ denotes the parameters of the target critic network of CH m.

(2) Update the critic network parameters: First, construct the mean-squared-error (MSE) loss function of the critic network, which is given by

$$L\left({\vartheta }_m\right)={\displaystyle \frac{1}{2\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}{\left({\tilde{R}}_m^t-Q_{{\vartheta }_m}({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t)\right)}^2$$

(25)

The input to the critic network consists of the observation of CH m and the joint action vector of all CH agents, which alleviates, to some extent, the non-stationarity of the environment caused by policy updates of other agents. Here, ${\vartheta }_m$ denotes the parameters of the online critic network of CH agent m. Furthermore, the critic network parameters are updated via gradient descent on the above loss function as

$${\vartheta }_m\leftarrow {\vartheta }_m-\beta \nabla L\left({\vartheta }_m\right)$$

(26)

where $\nabla L\left({\vartheta }_m\right)$ is given by

$$\nabla L\left({\vartheta }_m\right)={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}\left({\tilde{R}}_m^t-Q_{{\vartheta }_m}\left({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t\right)\right){\displaystyle \frac{\mathsf{\partial }{Q}_{{\vartheta }_m}\left({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t\right)}{\mathsf{\partial }{\vartheta }_m}}$$

(27)

(3) Update the actor network parameters: First, construct the loss function of the actor network, which is given by

$$L\left({\theta }_m\right)={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}-ln{\pi }_{{\theta }_m}\left({\mathit{a}}_m^t\left|{\tilde{\mathit{o}}}_m^t\right.\right)A_m^t+{\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}\sum _{\mathit{a}\in {\mathcal{A}}_m}{\pi }_{{\theta }_m}\left(\mathit{a}\right|{\tilde{\mathit{o}}}_m^t)ln{\pi }_{{\theta }_m}\left(\mathit{a}\left|{\tilde{\mathit{o}}}_m^t\right.\right)$$

(28)

Here, the first term contains the advantage function

$$A_m^t=Q_{{\vartheta }_m}({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t)-V_m\left({\tilde{\mathit{o}}}_m^t\right)={\tilde{R}}_m^t-Q_{{\vartheta }_m}({\tilde{\mathit{o}}}_m^t,{\mathit{a}}^t)$$

(29)

and the second term is an entropy-regularization term. Furthermore, taking the gradient of the above loss yields

$${\nabla }_{{\theta }_m}L\left({\theta }_m\right)=-{\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}\left(A_m^t{\displaystyle \frac{\mathsf{\partial }ln{\pi }_{{\theta }_m}\left({\mathit{a}}_m^t\left|{\tilde{\mathit{o}}}_m^t\right.\right)}{\mathsf{\partial }{\theta }_m}}\right)$$

(30)

Then, the actor network parameters are updated via gradient descent as

$${\theta }_m\leftarrow {\theta }_m-\beta {\nabla }_{{\theta }_m}L\left({\theta }_m\right)$$

(31)

In summary, the proposed scheme constructs a spatio-temporal extended Dec-POMDP model by introducing a spatial discount factor, enabling each CH agent not only to rely on local observations but also to selectively integrate neighboring information based on the network topology, thereby achieving a flexible trade-off between local greediness and global cooperation. Meanwhile, it adopts a CTDE architecture, where each CH agent maintains an independent actor network for real-time decision-making, while utilizing a critic network to incorporate global action information during the training phase, effectively mitigating the non-stationarity issue in multi-agent environments.

5. Simulation Results

In this section, we present simulation results to validate the performance of our proposed energy-harvesting-assisted multi-agent IA2C anti-jamming communication scheme for a UAV swarm scenario. In our simulation, all UAV clusters and jammers are flying within a 500 m × 500 m area with an initial height from the interval of $[100\mathrm{m},180\mathrm{m}]$. The other default simulation parameters are listed in

Table 1. System parameters settings.

Parameter	Notion	Value
Number of UAV clusters	M	4
Number of cluster members	I	2
Number of jammers	N	4
Number of available channels	L	6
Transmit power of UAVs	$p_{m,i}^t$	$[20,23,26]$ dBm
Transmit power of jammers	$p_n^t$	30 dBm
Antenna gain of UAVs/jammers		3 dBi
Power of AWGN	${\sigma }^2$	$-174$ dBm
Timeslot	$\tau$	$2.18$ s
Execution stage	$T_{\mathrm{ACT}}$	$0.98$ s
Time ratio factor	${\omega }_m^t$	$[0,0.25,0.5,0.75,1]$
Jamming period	$T_J$	$2.28$ s
Correlation weight factor	$\mu$	$0.6$
Gaussian variable	${\sigma }_M^2$	2
Maximum communication range	$D_{\max }$	100 m
The radius of motion of the RP	$R_{\mathrm{RP}}$	99 m
The radius of movement of CMs	$R_{\mathrm{CM}}$	1 m
Bandwidth	B	180 kHz
Data packet size	K	300 KB
Energy Harvesting Efficiency	$\eta$	$0.8$
Spatial discount factor	$\alpha$	1

and

Table 2. Training parameters settings.

Parameter	Notion	Value
Number of episodes	E	5000
Maximum number of steps per episode	T	400
Learning rate parameter	$\beta$	0.002
Batch size	$\left|\mathcal{B}\right|$	32
Discount factor	$\gamma$	0.99

.

To demonstrate the superiority of the proposed scheme, this section compares it with the following baseline schemes:

• Consensus-update-based anti-jamming scheme (CU) [34]: Each CH agent adopts a consensus-update mechanism to perform distributed training, allocating the time for energy harvesting and data transmission, while simultaneously deciding the channel selection and transmit power.
• Multi-agent DDRQN-based anti-jamming scheme (DDRQN) [35]: Each CH agent employs the multi-agent DDRQN method. It makes independent decisions using only local observations, including the time allocation ratio between energy harvesting and data transmission, the transmission channel, and the transmit power.
• Received-signal-strength-indication-based anti-jamming scheme (RSSI): Each CH agent randomly allocates the time between energy harvesting and data transmission, and then performs data transmission on the channel with the smallest path loss using the maximum transmit power.
• Random scheme: Each CH agent randomly allocates the time between energy harvesting and data transmission according to its local observations, and randomly selects the transmission channel and transmit power.

5.1. Comparison of Convergence Performance of Different Schemes

Figure 4 compares the convergence performance of the system average cumulative reward. The two schemes that share neighboring observations (IA2C and CU) achieve significantly better convergence performance and higher converged average rewards than the other three benchmark schemes. Moreover, compared with the two non-intelligent schemes (RSSI and Random), DDRQN exhibits better convergence due to its learning capability. In addition, although CU converges faster than the proposed IA2C scheme, IA2C attains the highest system average cumulative reward after convergence among all schemes. This is because the CU scheme updates its parameters using the averaged critic network parameters of all neighboring CHs, which accelerates convergence. In contrast, the proposed IA2C scheme introduces a spatial discount factor to emphasize local decision-making while incorporating partial information from neighboring CHs via information exchange, thereby enabling more effective policy learning and yielding a higher reward after convergence. This design improves each agent’s ability to learn an effective policy from local observations augmented with neighborhood CHs, thereby further enhancing anti-jamming performance.

Figure 5 compares the convergence performance of different schemes in terms of data transmission success rate. All schemes that share neighboring observations converge quickly to a high success rate, with the proposed IA2C scheme achieving the best performance. This is because each CH in the proposed scheme collaborates with its neighboring CHs, and the spatial discount factor encourages each agent to focus more on local decision-making during training. As a result, each CH can allocate time between energy harvesting and data transmission more effectively and make better channel selection and power allocation decisions.

Figure 6 compares the convergence performance of the average cumulative transmission energy loss of different schemes. All intelligent schemes can effectively reduce this energy loss. In contrast, the proposed multi-agent IA2C scheme leverages a spatial discount factor to encourage each agent to focus on locally relevant decision-making during training. This design better balances harvested energy and transmission energy consumption under anti-jamming operation in the spectrum and power domains. These simulation results validate the effectiveness of inter-UAV information exchange and demonstrate its advantage in improving the anti-jamming performance of UAV clusters in energy-limited environments.

5.2. Performance Comparison Versus the Number of Available Channels

Figure 7 compares the average cumulative reward of different schemes versus the number of available channels. The average cumulative reward of the IA2C scheme and the CU scheme increases as the number of available channels increases, and the proposed IA2C scheme achieves a significantly higher average cumulative reward than the benchmark schemes. This is because IA2C introduces a spatial discount factor to encourage each CH agent to focus on locally relevant decision-making, while information exchange among neighboring agents increases the effective information available to each CH. Together, these mechanisms enable better channel selection and power allocation decisions and improve anti-jamming performance.

Figure 8 compares the data transmission success rates of the different schemes versus the number of available channels. As the number of available channels increases, the data transmission success rates of both IA2C and CU increase. This is because a larger spectrum provides each cluster with more candidate channels, allowing it to select a higher-quality channel for communication. Consequently, co-channel interference among clusters is reduced, and the UAV swarm becomes more adaptable to diverse communication environments. Moreover, the proposed IA2C scheme consistently achieves the highest average data transmission success rate versus the number of available channels. Because IA2C employs a spatial discount factor to emphasize local decision-making and leverages neighbor information exchange to enrich each CH’s effective observations.

Figure 9 compares the average cumulative transmission energy loss of different schemes versus the number of available channels. As the number of available channels increases, both IA2C and CU maintain lower average cumulative transmission energy loss, with IA2C performing slightly better. This is because more available channels allow each cluster to select a higher-quality channel, which reduces inter-cluster co-channel interference and enables stable communication at lower transmit power. Moreover, IA2C employs a spatial discount factor to emphasize local decision-making and leverages information exchange with neighboring CHs to enrich each CH’s effective observations. In summary, as the number of available channels increases, the per-cluster transmission energy loss under IA2C and CU remains low and relatively stable.

6. Conclusions

To simultaneously ensure anti-jamming performance and improve energy efficiency for UAV swarm networks in complex electromagnetic environments, we propose an energy-harvesting-assisted multi-agent anti-jamming resource allocation framework. Within this framework, each CU learns its channel, power, and time allocation policy through utilizing the spatiotemporal correlations among neighboring CHs. By aggregating observations from neighboring CHs and leveraging the multi-agent IA2C method, each CU can progressively mitigate the influence of decisions made by other CH agents. Simulation results demonstrate that the proposed scheme consistently outperforms the baseline schemes across varying numbers of channels. Future work will consider (1) dynamic clustering topologies, (2) joint resource allocation and trajectory optimization, (3) integrated routing and resource allocation, and (4) power-splitting architecture.