Collaborative Search Algorithm for Multi-UAVs Under Interference Conditions: A Multi-Agent Deep Reinforcement Learning Approach

Abstract

Unmanned aerial vehicles (UAVs) have emerged as a promising solution for collaborative search missions in complex environments. However, in the presence of interference, communication disruptions between UAVs and ground control stations can severely degrade coordination efficiency, leading to prolonged search times and reduced mission success rates. To address these challenges, this paper proposes a novel multi-agent deep reinforcement learning (MADRL) framework for joint spectrum and search collaboration in multi-UAV systems. The core problem is formulated as a combinatorial optimization task that simultaneously optimizes channel selection and heading angles to minimize the total search time under dynamic interference conditions. Due to the NP-hard nature of this problem, we decompose it into two interconnected Markov decision processes (MDPs): a spectrum collaboration subproblem solved using a received signal strength indicator (RSSI)-aware multi-agent proximal policy optimization (MAPPO) algorithm and a search collaboration subproblem addressed through a target probability map (TPM)-guided MAPPO approach with an innovative action-masking mechanism. Extensive simulations demonstrate superior performance compared to baseline methods (IPPO, QMIX, and IQL). Extensive experimental results demonstrate significant performance advantages, including 68.7% and 146.2% higher throughput compared to QMIX and IQL, respectively, along with 16.7–48.3% reduction in search completion steps versus baseline methods, while maintaining robust operations under dynamic interference conditions. The framework exhibits strong resilience to communication disruptions while maintaining stable search performance, validating its practical applicability in real-world interference scenarios.

1. Introduction

With characteristics such as low costs, good mobility, flexibility, and ease of deployment, unmanned aerial vehicles (UAVs) have found extensive applications in tasks such as communication coverage [1,2], reconnaissance [3,4], and search and rescue [5,6]. A notable advantage of multi-UAV missions over single-UAV operations is the enhancement of task efficiency and reliability, particularly in complex and large-scale task management scenarios [7]. This enhancement is achieved by leveraging the collaborative capabilities of multiple UAVs, which rely on real-time communication to ensure effective coordination and collaboration. The success of a multi-UAV collaborative mission is contingent upon the design of a coordination algorithm that ensures both efficient task execution among the UAVs and reliable communication between them.

Multi-UAV collaborative search is one of the typical application scenarios of multi-UAV systems. Such scenarios include post-disaster relief [8,9], coordinated targeting during operations [10,11], border patrol [12,13], and perimeter monitoring [14,15]. These tasks usually involve characteristics such as unknown target locations, wide-area coverage, and time sensitivity. Therefore, multi-UAV collaborative search algorithms often aim to maximize target detection rates, area coverage, and search efficiency. It is evident that a taxonomy of search methods exists and that these can be classified into two distinct categories: static search and dynamic search [16,17,18].

In static search, UAVs perform searches according to pre-planned routes, such as the Zamboni search [19] and spiral search [20], covering the entire task area to identify targets. In static environments or when there is no prior knowledge of the task area, static searches can yield good results due to their simplicity and ease of implementation, but they are often difficult to apply in dynamic environments.

In contrast, a dynamic search is more suitable for dynamic environments. Since the state information in a dynamic environment may change at any time, UAVs in dynamic searches make collaborative search decisions in each time slot based on their observations and communications with other UAVs, offering greater flexibility and applicability, thereby fostering extensive research in this area. Zheng et al. [21] proposed a distributed real-time-search path planning method based on the framework of distributed model predictive control. In order to address the issue of time-sensitive moving targets in uncertain dynamic environments, Yang et al. [22] provided an intelligent collaborative task planning scheme for UAV swarms based on hybrid artificial potential field and ant colony optimization. In [23], the authors investigated a multi-layer aerial computing network search scenario, and a multi-agent deep reinforcement learning method is proposed based on parameter sharing and action masking. To tackle the challenge of target searches by UAV swarms in unknown three-dimensional environments, paper [24] introduced a cooperative target search method based on the improved wolf pack algorithm, which enhances the efficiency of target search operations. In order to enhance the scalability and cooperation of UAV swarm search methods, the authors of [25] designed a distributed cooperative search method based on multi-agent reinforcement learning algorithms, which can operate efficiently in complex, large-scale environments. The author of [26] presented a weighted area model mapped by a Monte Carlo drift prediction model, and three reinforcement learning algorithms were employed to address the weighted area coverage issue during actual rescue processes. Paper [27] considered the issues of collaborative control and search area coverage in multi-UAV cooperative searches and used an enhanced binary logarithmic learning algorithm to address the problem.

Recent advances in multi-UAV cooperative systems have progressively addressed three critical aspects of collaborative operations: communication reliability, search efficiency, and interference resilience. While early studies primarily focused on optimizing search patterns and coverage strategies, the fundamental role of communication in enabling effective coordination has gained increasing recognition. Initial approaches to maintaining connectivity relied heavily on distance-based models, as demonstrated by local communication network architecture [28] and relay UAV deployment strategy [29]. Subsequent refinements incorporated more sophisticated connectivity maintenance techniques, including priority-based positioning systems [30] and deep Q-learning pheromone models [31], marking important steps toward robust multi-UAV operations.

The field has witnessed parallel developments in interference-resistant coordination, with notable contributions including robust trajectory planning using convex optimization by [32] and overlapped sensing schemes for improved transmission efficiency [33]. Decentralized architectures have emerged as a promising direction, exemplified by consensus-based ant colony optimization [34] and attention mechanism-enhanced planning for communication-restricted scenarios [35]. Bio-inspired techniques have shown particular promise, with bird flocking algorithms [36] and game-theoretic approaches [37] demonstrating effective coordination in complex environments. Recent learning-based methods have pushed the boundaries further, with deep reinforcement learning frameworks for joint computation offloading [38] and advantage dueling Q-learning algorithms setting new benchmarks for autonomous operations in unknown environments [39]. Various strategies have been proposed to handle communication disruptions, ranging from metacognitive decision-making [40] to information compensation methods [41], while dynamic adaptation techniques have been explored through multi-UAV deployment [29] and adaptive search strategies [34].

Despite these advancements, existing solutions exhibit three key limitations: (1) most distance-based connectivity models fail to account for external interference, (2) many learning approaches neglect real-time spectrum constraints, and (3) few frameworks successfully integrate communication and search optimization. While studies like [42,43] advanced search efficiency, they overlooked communication constraints, and approaches such as [44,45] addressed communication limitations without considering spectrum contentions. Even recent works like the human–UAV interaction system [46] and information fusion methods [47] retain centralized coordination elements, while the minimum time search algorithm [48] maintains dependence on ground station connectivity.

Our work bridges these gaps through three key innovations: first, by developing the first fully decentralized framework that jointly optimizes spectrum access and search efficiency; second, through a novel integration of RSSI-aware MAPPO for dynamic spectrum coordination; and third, via TPM-guided MAPPO with action masking for constrained search optimization. This comprehensive approach advances beyond current solutions by simultaneously addressing interference resilience and real-time adaptability while maintaining superior performance metrics, as clearly demonstrated in

Table 1. Comparative analysis of related works.

Study	Spectrum Collaboration	Search Collaboration	Jammer Resilience	Real-Time Operation	Joint Optimization
[28,29,30,31]	✔	✔	✘	✘	✘
[32,33]	✔	✘	✘	✔	✘
[34,35,38,39,40,41,43,44,45,46,47,48]	✘	✔	✘	✔	✘
[36,37,42]	✘	✔	✘	✘	✘
our work	✔	✔	✔	✔	✔

✔: The feature is fully implemented, ✘: The feature is not implemented.

. The table highlights our unique position as the only solution that successfully integrates all four critical dimensions of multi-UAV cooperative systems under interference conditions.

In order to cope with the foregoing issues, we propose a collaborative search scheme for multi-UAVs under interference conditions based on a specifically designed multi-agent reinforcement learning algorithm. The significant contributions are as follows:

• A collaborative search model for multi-UAVs under interference conditions is established. Based on this model, an optimization problem is constructed with the goal of minimizing the total search time by jointly optimizing the channel selection and search heading angles of the UAVs. To reduce the complexity of solving the problem, the optimization is decomposed into two subproblems, corresponding to spectrum collaboration under interference conditions and undisturbed search collaboration.
• An MAPPO-based spectrum collaboration algorithm that only requires the RSSI of the channels is proposed to enhance the algorithm’s real-time responsiveness, allowing each UAV to complete the collaboration without needing to know the channel’s state information between other nodes.
• An MAPPO-based search collaboration algorithm that takes the TPM of the task area as a shared message is designed to optimize the heading of the UAVs. In addition, action masking based on the current state and the previous state is employed in the search collaboration to improve the convergence speed of the algorithm. The policy network is also delicately designed with two convolutional neural networks to match the two-dimensional TPM.

The rest of the study is organized as follows: Section 2 describes the UAV search model and the UAV communication model. The problem formulation is elaborated in Section 2.4. The MAPPO-based spectrum collaboration and search collaboration are detailed in Section 2.4. The numerical results are presented in Section 3. The paper concludes in Section 4.

2. System Model

As shown in Figure 1, N reconnaissance UAVs collaboratively search an area containing M stationary targets at altitude H. For each reconnaissance UAV, the number of targets is unknown, and its perception of the environment is based on previous reconnaissance data or pertinent prior knowledge. Upon entering the task area, the reconnaissance UAVs employ onboard sensors to locate targets, collect reconnaissance data, and send this information to the ground control station in real time. The ground control station aggregates the detection data from all UAVs and subsequently broadcasts this information to each UAV, facilitating a collaborative search among multiple UAVs. Furthermore, there are several jammers in the task area that disrupt communication between the ground control station and the UAVs.

2.1. UAV Search Model

To establish a multi-UAV collaborative search model, the task area $\Omega$ is divided into $N_x\times N_y$ discrete grids, with each grid being a square of side length L. Here, L represents the flight distance of the UAVs during one search decision cycle. Any grid within the task area can be represented by the coordinates of its center point $(x,y)$, $0\le x\le N_x-1$, $0\le y\le N_y-1$.

To simplify the search space of the UAVs, we assume that each UAV can only move to one of the 8 neighboring grids, thus providing 8 feasible heading angles, denoted as $\mathcal{D}=\left\{0,1,\cdots ,7\right\}$, with the corresponding direction vectors represented as $\mathcal{Z}=\left\{[1,0],[1,-1],[0,-1],[-1,-1],[-1,0],[-1,1],[0,1],[1,1]\right\}$. Considering the turning angle constraints for the UAVs, we assume that the change in flight direction between two adjacent time slots cannot exceed 45°. Let the flight direction of the UAV i at time slot t be denoted as $d_i^t$. Then, at time slot $t+1$, the feasible flight direction set for the drone is ${\mathcal{D}}_i^{t+1}=\left\{d_i^t-1,d_i^t,d_i^t+1\right\}mod8$. Since the distance traveled by the UAVs when flying diagonally ($d_i^t\in \left\{0,2,4,6\right\}$) is $\sqrt{2}$ times that of flying along the edge ($d_i^t\in \left\{1,3,5,7\right\}$), to ensure that the UAV is always centered in the grid when making a search decision, we assume that the corresponding flight speeds for the two types of flight directions are $\sqrt{2}v$ and v. Let ${\mathbf{z}}_i^t$ represent the position of the drone at time t; then, the positional relationship of UAV i within two consecutive decision cycles can be expressed as

$${\mathbf{z}}_i^{t+1}={\mathbf{z}}_i^t+\mathcal{Z}\left(d_i^t\right).$$

(1)

Assuming that the UAV conducts reconnaissance of the target using a downward-looking circular scanning method, as shown in Figure 1, the detection radius of the onboard sensors is $r_c$, the detection probability is $P_d$, and the false alarm probability is $P_f$. Thus, the probability distribution for the UAVs detecting a target can be expressed as

$$p\left(a=1|l\right)=\left\{\begin{array}{cc}{P}_d& l\le l_c\\ P_f& l>l_c\end{array}\right.,$$

(2)

where $p\left(a=1|r\right)$ represents the probability of the UAV detecting the target at a distance l and $l_c=\sqrt{r_c^2+H^2}$. To reduce the complexity of probability calculations, we assume that there is at most one target in each grid of the task area.

2.2. Target Probability Map

To describe the probability distribution of the target in the task area at different time slots, we introduce the target probability map (TPM) [49]. The TPM for the task area $\Omega$ at time slot t can be defined as

$${\mathbf{TPM}}_{\Omega }^t\stackrel{\Delta }{=}\left\{p_{\left(x,y\right)}^t|x=0,1,\cdots ,N_x-1,y=0,1,\cdots ,N_y-1\right\},$$

(3)

where $p_{\left(x,y\right)}^t$ represents the probability of a target at grid $\left(x,y\right)$ at time slot t. The TPM update formula derived from Bayes’ theorem is as follows:

$$p_{\left(x,y\right)}^{t+1}=\left\{\begin{array}{cc}{p}_{\left(x,y\right)}^t& \mathrm{undetected}\\ {\displaystyle \frac{P_d{p}_{\left(x,y\right)}^t}{P_f+\left(P_d-P_f\right)p_{\left(x,y\right)}^t}}& a_{(x,y)}^t=1\\ {\displaystyle \frac{\left(1-P_d\right)p_{\left(x,y\right)}^t}{1-P_f+\left(P_f-P_d\right)p_{\left(x,y\right)}^t}}& a_{(x,y)}^t=0\end{array}\right.,$$

(4)

where $a_{(x,y)}^t$ denotes the detection result of the UAV for the grid $\left(x,y\right)$ at time slot t. In general, $0<P_f<0.5<P_d<1$, and it can be easily proven that $p_{\left(x,y\right)}^{t+1}\ge p_{\left(x,y\right)}^t$ for $a_{(x,y)}^t=1$ and $p_{\left(x,y\right)}^{t+1}\le p_{\left(x,y\right)}^t$ for $a_{(x,y)}^k=0$. For the grid $\left(x,y\right)$ in which UAVs bear no prior knowledge of the probability of the presence of a target, $p_{\left(x,y\right)}^t=0.5$. As the search proceeds, $p_{\left(x,y\right)}^{\infty }\to 0or1$. Let ${\eta }_0$ and ${\eta }_1$ be the thresholds for determining the absence and presence of a target, respectively. Once $p_{\left(x,y\right)}^k\le {\eta }_0$, it can be concluded that there is no target within the grid $\left(x,y\right)$; once $p_{\left(x,y\right)}^k\ge {\eta }_1$, it can be determined that a target exists within grid $\left(x,y\right)$.

To reduce the update complexity of (4) and shorten the decision time for the search task, a nonlinear transformation can be applied to $p_{\left(x,y\right)}^k$, enabling the nonlinear update in (4) to be changed into a linear update. By setting $Q_{(x,y)}^k=ln\left({\displaystyle \frac{1}{p_{\left(x,y\right)}^k}}-1\right)$ as [50], Equation (4) can be rewritten as

$$Q_{(x,y)}^{t+1}=\left\{\begin{array}{cc}{Q}_{(x,y)}^t& \mathrm{undetected}\\ Q_{(x,y)}^t+ln{\displaystyle \frac{P_f}{P_d}}& a_{(x,y)}^t=1\\ Q_{(x,y)}^t+ln{\displaystyle \frac{1-P_f}{1-P_d}}& a_{(x,y)}^t=0\end{array}\right..$$

(5)

Given that $ln{\displaystyle \frac{P_f}{P_d}}<0$ and $ln{\displaystyle \frac{1-P_f}{1-P_d}}>0$, when $a_{(x,y)}^t=1$, it follows that $Q_{(x,y)}^{t+1}\le Q_{(x,y)}^t$; when $a_{(x,y)}^t=0$, it follows that $Q_{(x,y)}^{t+1}\ge Q_{(x,y)}^t$. Therefore, for $Q_{(x,y)}^t$, the decision thresholds can also be used to determine whether a target exists. For $Q_{(x,y)}^t\le {\tilde{\eta }}_1$, it is considered that there is a target within the grid $\left(x,y\right)$; for $Q_{(x,y)}^t\ge {\tilde{\eta }}_0$, it is assumed that there is no target within the grid $\left(x,y\right)$, where ${\tilde{\eta }}_1=ln\left({\displaystyle \frac{1}{{\eta }_1}}-1\right)$ and ${\tilde{\eta }}_0=ln\left({\displaystyle \frac{1}{{\eta }_0}}-1\right)$.

From the perspective of multi-UAV collaborative search, the TPM of the task area $\Omega$$\mathbf{TP}{\mathbf{M}}_{\Omega }$ contains the probability distribution of targets within each grid of the task area, serving as an important basis for UAVs to make collaborative search decisions. Therefore, we assume that each UAV maintains its own TPM, referred to as LTPM (local TPM). As the search task progresses, each UAV updates its LTPM. Additionally, there is a global TPM, denoted as GTPM, maintained at the ground control station (GCS). The GTPM is updated based on the LTPM of each UAV, and in turn, the GTPM is used to update each UAV’s LTPM, as illustrated in Figure 2, where ${\mathbf{LTPM}}_i^t$ and $\mathbf{GTP}{\mathbf{M}}^t$ represent the LTPM of UAV i and the GTPM of the GCS at time slot t, respectively. The yellow grid represents the UAV’s reconnaissance area (the updated portion of the LPTM). Such an updating process is essential for achieving collaborative search among multiple UAVs.

Specifically, each decision cycle of a search task $T^S$ can be divided into four stages, as shown in Figure 3. In $T_1^S$, each UAV performs the search task and updates its LTPM based on the search results. In $T_2^S$, each UAV sends its updated LTPM to the GCS. During $T_3^S$, the GCS fuses the LTPMs and updates the GTPM with the fused results. In $T_4^S$, the GCS broadcasts the updated global TPM to each UAV. The UAVs use the received GTPM as their new LTPM and continue with the search task. Such a cycle is repeated until the search task is completed. Essentially, this is a centralized updating process of the TPM utilizing the GCS, which allows each UAV to receive updates from all other UAVs’ LTPMs, enabling collaborative search among all UAVs throughout each decision cycle.

2.3. Channel Model

As mentioned above, to update the GTPM and LTPM, the UAVs and the ground control station must communicate during the second and fourth stages of each search decision cycle. In addition, there are jammers in the vicinity of the task area that randomly select channels to release interference. To quantitatively describe the TPM update process under interference conditions, this section will model the channels between the UAVs and the ground control station, the UAVs and the jammer, and the jammer and the ground control station.

The channels between the UAVs and the ground control station, as well as between the UAVs and the jammer, are all considered air-to-ground channels. For the sake of simplicity, the GCS denoted as G and the jammer denoted as J will hereinafter be collectively referred to as ground nodes denoted as X. Considering the impact of obstacles between the air and ground, such as trees and buildings, a mixed channel model of line-of-sight (LoS) and non-line-of-sight (NLoS) is employed to establish the air-to-ground channels between the UAVs and the ground nodes [51,52]. At each time slot, the channel between the UAVs and the ground nodes X can be expressed as

$$g_{i,X}^t=P_{i,X}^{\mathrm{LoS}}\left(t\right)g_{i,X}^{\mathrm{LoS}}\left(t\right)+P_{i,X}^{\mathrm{NLoS}}\left(t\right)g_{i,X}^{\mathrm{NLoS}}\left(t\right),$$

(6)

$$g_{i,X}^{\mathrm{LoS}}\left(t\right)=20lg{d}_{i,X}^t+20lg{f}_c+20lg{\displaystyle \frac{4\pi }{c}},$$

(7)

$$g_{i,X}^{\mathrm{LoS}}\left(k\right)=20lg{d}_{i,X}^k+20lg{f}_c+20lg{\displaystyle \frac{4\pi }{c}}+{\psi }_{NLoS},$$

(8)

$${\theta }_{i,X}^t={\displaystyle \frac{180}{\pi }}arcsin{\displaystyle \frac{h}{d_{i,X}^t}}.$$

(9)

For the ground-to-ground channel between the jammer and the GCS, both large-scale fading and small-scale fading are considered [53]. Therefore, the channel between the two can be written as

$$h_{J,G}^t={\alpha }_{J,G}^t{f}_{J,G}^t,$$

(10)

where ${\alpha }_{J,G}^t$ represents large-scale fading, which can be calculated using (7). On the other hand, $f_{J,G}^t$ signifies the small-scale fading that is frequency-dependent and assumed to follow an exponential distribution with a mean of 1 and a variance of 0.

2.4. UAV Communication Model

As previously mentioned, during the second stage of the search decision cycle $T_2^S$, each UAV will transmit the updated LTPM back to the GCS. Due to the presence of interference, the available channels may be subject to random disruption, rendering it impossible to pre-assign a channel for each UAV to return the LTPM. Consequently, each UAV can only dynamically select a channel during $T_2^S$. Ensuring the successful transmission of the LTPM necessitates the avoidance of interference channels by the UAVs, whilst also preventing the selection of the same channel as other UAVs. If all UAVs can successfully return information through appropriate channel selection, avoiding interference or mutual disturbance, this is referred to as achieving spectrum collaboration among the UAVs.

Assuming that M channels are available and each channel has a bandwidth of B, the received signal-to-noise ratio (SNR) at the GCS for UAV i on channel m during time slot $t^{\prime }$ of $T_2^S$ can be expressed as follows:

$${\gamma }_{i,m}^{t^{\prime }}={\displaystyle \frac{P_u}{{10}^{\frac{g_m\left(t^{\prime }\right)}{10}}{\displaystyle \sum _{\genfrac{}{}{0pt}{}{n\in \mathcal{N}\hfill }{n\ne i\hfill }}}\left(c_{n,m}^{t^{\prime }}{P}_u{10}^{-\frac{g_{i,X}^{t^{\prime }}}{10}}+{\chi }_m^{t^{\prime }}{P}_J{10}^{-\frac{h_{J,G}^{t^{\prime }}}{10}}+B{\sigma }^2\right)}}.$$

(11)

where $P_u$ denotes the transmission power of the UAVs. It is assumed that all UAVs possess identical transmission power. $\mathcal{N}=\left\{1,2,\cdots ,N\right\}$ is defined as the set of all reconnaissance UAVs, and $c_{i,m}^{t^{\prime }}$ represents the channel selection of the UAV in time slot $t^{\prime }$, which is defined as follows:

$$c_{i,m}^{t^{\prime }}=\left\{\begin{array}{cc}1,& \mathrm{uav}i\mathrm{selects}\mathrm{channel}m\mathrm{at}\mathrm{slot}{t}^{\prime }\\ 0,& \mathrm{otherwise}\end{array}\right..$$

(12)

It is assumed that, within a designated time slot, each UAV is permitted to select only a single channel at most. Furthermore, it is posited that a channel can be selected by no more than one UAV. Consequently, the following conditions must be met by $c_{i,m}^{t^{\prime }}$:

$$\sum _{m=1}^M{c}_{i,m}^{t^{\prime }}\le 1,$$

(13)

$$\sum _{i=1}^N{c}_{i,m}^{t^{\prime }}\le 1.$$

(14)

In (11), $P_J$ denotes the transmission power of the jammer, whilst ${\chi }_m^t$ serves as an indicator of whether channel m is subject to interference from the jammer at time slot $t^{\prime }$, as defined below:

$${\chi }_m^{t^{\prime }}=\left\{\begin{array}{cc}1,& \mathrm{channel}\mathrm{m}\mathrm{is}\mathrm{jammed}\mathrm{at}\mathrm{slot}{\mathrm{t}}^{\prime }\\ 0,& \mathrm{otherwise}\end{array}\right..$$

(15)

Furthermore, in (11), $\mathcal{M}$ is employed to denote the set of all available channels, and ${\sigma }^2$ is used to represent the power spectral density of Gaussian white noise.

In the context of the given parameters ${\gamma }_{G,i}^{t^{\prime }}$, the achievable rate of UAV i over channel m to the GCS at slot t can be expressed in accordance with Shannon’s formula as follows:

$$R_{i,m}^{t^{\prime }}=B{log}_2\left(1+{\gamma }_{i,m}^{t^{\prime }}\right).$$

(16)

It is imperative to acknowledge the distinction between the search decision cycle and the spectrum decision cycle, which operate on disparate temporal scales. The search decision pertains to the direction in which the UAV flies and is typically measured in seconds, while the spectrum decision is related to alterations in channel status and is usually measured in milliseconds. Therefore, the utilization of t and $t^{\prime }$ is employed to differentiate between search decision time slots and spectrum decision time slots. Figure 4 illustrates the relationship between the search decision cycle and the spectrum decision cycle.

During the $T_2^S$ stage of the search decision cycle, there are $N_c$ spectrum decision cycles. Therefore, the total amount of data returned by UAV i in t-th search decision cycle is

$$R_i^t=\sum _{t=1}^{N_c}\sum _{m=1}^M{R}_{i,m}^t.$$

(17)

In $T_4^S$, the GCS disseminates the updated GTPM over all channels, while all UAVs listen on all channels. As this stage does not involve channel selection among multiple UAVs, we only consider spectrum collaboration during $T_2^S$. Furthermore, we assume that the broadcast messages in $T_4^S$ can always be successfully received by each UAV.

Problem Formulation: The objective of multi-UAV collaborative search tasks is to enhance the efficiency of the search process within the designated task area while detecting the targets dispersed within that area. This paper delineates the completion of the search operation within the task area as follows: For any $p_{\left(x,y\right)}\in \mathbf{TP}{\mathbf{M}}_{\Omega }$, it holds that $p_{\left(x,y\right)}\ge {\eta }_1$ or $p_{\left(x,y\right)}\le {\eta }_0$, where $p_{\left(x,y\right)}$ is the probability of the target’s presence in grid $\left(x,y\right)$. Consequently, it can be concluded that under the specified threshold, a determination can be made regarding the presence or absence of targets in any grid within the task area.

For brevity, a factor $f_{\left(x,y\right)}$ is introduced to denote whether the grid $\left(x,y\right)$ has been previously searched at slot t. As outlined below,

$$f_{\left(x,y\right)}^t=\left\{\begin{array}{cc}1& p_{\left(x,y\right)}^t\ge {\eta }_{1}or{p}_{\left(x,y\right)}^t\le {\eta }_0\\ 0& \mathrm{otherwise}\end{array}\right..$$

(18)

Based on (18), we can define the coverage rate of the search task in time slot t as follows:

$${\eta }_c^t={\displaystyle \frac{{\displaystyle \sum _{y=0}^{N_y-1}}{\displaystyle \sum _{x=0}^{N_x-1}}{f}_{\left(x,y\right)}^t}{N_x{N}_y}}.$$

(19)

We have discussed the updating process of the TPM in Section 2. During $T_1^S$, following an independent search, UAV i updates its ${\mathbf{LTPM}}_i^t$ to ${\mathbf{LTPM}}_i^{\tilde{t}}$ at time slot t. It can be deduced from (2) and (5) that ${\mathbf{LTPM}}_i^{\tilde{t}}$ is related to ${\mathbf{LTPM}}_i^t$, the position of UAV i during time slot t, the detection radius $r_c$ of the onboard sensor, the detection probability $P_d$, and the false alarm probability $P_f$. The relationship between ${\mathbf{LTPM}}_i^{\tilde{t}}$ and ${\mathbf{LTPM}}_i^t$ can thus be described by the following function:

$${\mathbf{LTPM}}_i^{\tilde{t}}=\phi \left({\mathbf{LTPM}}_i^t,{\mathbf{z}}_i^t,r_c,P_d,P_f\right).$$

(20)

During $T_3^S$, the GCS updates $\mathbf{GTP}{\mathbf{M}}^t$ using ${\mathbf{LTPM}}_i^{\tilde{t}}\left(i=1,\cdots ,N\right)$. In a similar manner, the relationship between the two can be expressed as a function given by

$$\mathbf{GTP}{\mathbf{M}}^{t+1}=\varphi \left({\left\{{\mu }_i^t·{\mathbf{LTPM}}_i^{\tilde{t}}\right\}}_{i=1}^N,r_c,P_d,P_f\right),$$

(21)

$${\mu }_i^t=\left\{\begin{array}{cc}1& R_i^t\ge R_{th}\\ 0& R_i^t<R_{th}\end{array}\right..$$

(22)

During $T_4^S$, each UAV is responsible for receiving the broadcast message $\mathbf{GTP}{\mathbf{M}}^{t+1}$ from the ground station and updating its own ${\mathbf{LTPM}}_i^{\tilde{t}}$. The following has been established:

$${\mathbf{LTPM}}_i^{t+1}=\mathbf{GTP}{\mathbf{M}}^{t+1}.$$

(23)

It is posited that at the beginning of the UAV’s task, the following equation holds:

$${\mathbf{LTPM}}_1^0={\mathbf{LTPM}}_2^0=\cdots =\mathbf{GTP}{\mathbf{M}}^0.$$

(24)

With the above discussion, it is therefore possible to propose a joint optimization model that incorporates both spectrum collaboration and search collaboration, with the objective of minimizing the number of decision cycles given by (P1) as follows:

$$\begin{array}{c}\left(\mathrm{P}1\right)\underset{\left\{d_i^t\right\},\left\{c_{i,m}^{t,t^{\prime }}\right\}}{min}\underset{T}{argmax}{\eta }_c^T\hfill \\ s.t.C1:d_i^{t+1}\in \left\{\left\{d_i^t-1,d_i^t,d_i^t+1\right\}mod8,8\right\}\hfill \\ C2:0\le {\mathbf{z}}_i^t\left(0\right)\le N_x-1,0\le {\mathbf{z}}_i^t\left(1\right)\le N_y-1\hfill \\ C3:d_i^t\in \left\{0,1,\cdots ,8\right\}\hfill \\ C4:c_{i,m}^{t,t^{\prime }}\in \left\{0,1\right\}\hfill \\ C5:{\mathbf{z}}_i^{t+1}={\mathbf{z}}_i^t+Z\left(d_i^t\right)\hfill \\ C6:{\displaystyle \sum _{m=1}^M}{c}_{i,m}^{t^{\prime }}\le 1\hfill \\ C7:{\displaystyle \sum _{i=1}^N}{c}_{i,m}^{t^{\prime }}\le 1\hfill \\ C8:{\mathbf{LTPM}}_i^{\tilde{t}}=\phi \left({\mathbf{LTPM}}_i^t,{\mathbf{z}}_i^t,r_c,P_d,P_f\right)\hfill \\ C9:\mathbf{GTP}{\mathbf{M}}^{t+1}=\varphi \left({\left\{{\mu }_i^t·{\mathbf{LTPM}}_i^{\tilde{t}}\right\}}_{i=1}^N,r_c,P_d,P_f\right)\hfill \\ C10:{\mu }_i^t=\left\{\begin{array}{cc}1& R_i^t\ge R_{th}\\ 0& R_i^t<R_{th}\end{array}\right.\hfill \\ C11:{\mathbf{LTPM}}_i^{t+1}=\mathbf{GTP}{\mathbf{M}}^{t+1}\hfill \\ C12:{\mathbf{LTPM}}_1^0={\mathbf{LTPM}}_2^0=\cdots =\mathbf{GTP}{\mathbf{M}}^0\hfill \end{array}.$$

(25)

In (P1), $d_i^t$ denotes the search decision variable, which represents the heading angle of the UAV i during the t-th search decision, while $c_{i,m}^{t,t^{\prime }}$ signifies the spectrum decision variable, which indicates the channel selection during the $t^{\prime }$-th spectrum decision within the t-th search decision cycle. T is the number of search decision cycles when the search task is completed. Constraint $C1$ represents the nonlinear constraint on the heading angles of the UAV between two adjacent decision cycles, while constraint $C2$ ensures that the UAV’s flight trajectory remains within the task area. Constraints $C3$ and $C4$ specify the value ranges for $d_i^t$ and $c_{i,m}^{t,t^{\prime }}$, respectively. Furthermore, since the spectrum decision variable has been redefined as $c_{i,m}^{t,t^{\prime }}$, the variable $c_{i,m}^{t^{\prime }}$ in Equations (13) and (14) is correspondingly changed to $c_{i,m}^{t,t^{\prime }}$.

The optimization problem in (25) is nonlinear in nature and involves multiple integer variables, in addition to the inter-dependency of several variables. The search decision variable $d_i^t$ determines the distance between the UAV and the GCS, jammer, and other UAVs. This, in turn, affects the channel state and influences spectrum collaboration. The spectrum decision variable $c_{i,m}^{t,t^{\prime }}$ determines the amount of information sent back by each UAV during the $T_2^S$ of the search decision cycle, which in turn affects search collaboration. Additionally, problem (P1) incorporates real-time variables such as the small-scale fading factor of channel $f_{J,G}^t$, random channel interference ${\chi }_m^t$ from the jammer, and the detection and false alarm probabilities of onboard sensors, which further introduce randomness into the search task. Convex optimization methods and heuristic algorithms are limited by their efficiency and their inability to adapt to complex time-varying environments. In the next section, the original joint optimization problem (P1) is to be transformed into two sub-optimization problems: spectrum collaboration and search collaboration. Subsequently, multi-agent reinforcement learning methods will be employed to achieve real-time collaboration among multiple UAVs in spectrum and search operations.

3. Proposed Algorithm

Multi-UAV spectrum collaboration is the foundation for achieving multi-UAV search collaboration. However, search decisions also impact spectrum decisions, as illustrated by (22), where the value of ${\mu }_i^t$ is related to both channel selection and the positions of the UAVs. This is due to the utilization of an absolute threshold. In order to decompose problem (P1) into two independent subproblems, the absolute threshold in Equation (22) is replaced with a relative threshold. That is, given the positions of the UAVs, spectrum decisions will only make maximum effort for collaboration, and Equation (22) can be rewritten as

$${\widehat{\mu }}_i^t=\left\{\begin{array}{cc}1& R_i^t\ge {\tau }_0{R}_{max}^t\\ 0& R_i^t<{\tau }_0{R}_{max}^t\end{array}\right..$$

(26)

In (26), $R_{max}^t$ denotes the maximum amount of information that each UAV is capable of transmitting to GCS during the t-$th$ search decision cycle, contingent on their respective positions. Meanwhile, ${\tau }_0\in \left(0,1\right]$ signifies the threshold value. Consequently, this enables the formulation of the optimization problem (P2) solely concerning spectrum collaboration, which is written as follows:

$$\begin{array}{c}\left(\mathrm{P}2\right)\underset{\left\{c_{i,m}^t\right\}}{max}{\displaystyle \sum _{i=1}^{N_c}}{R}_i^t\hfill \\ s.t.C1:c_{i,m}^t\in \left\{0,1\right\}\hfill \\ \begin{array}{c}C2:{\displaystyle \sum _{m=1}^M}{c}_{i,m}^{t^{\prime }}\le 1\hfill \\ C3:{\displaystyle \sum _{i=1}^N}{c}_{i,m}^{t^{\prime }}\le 1\hfill \end{array}\hfill \end{array}.$$

(27)

In addition, the UAV’s heading angle is optimized based on the spectrum decisions, with the objective of minimizing the number of search decision cycles required to complete the search task. This results in the following optimization problem (P3):

$$\begin{array}{c}\left(\mathrm{P}3\right)\underset{\left\{d_i^t\right\}}{min}\underset{T}{argmax}{\eta }_c^t\hfill \\ s.t.C1:d_i^{t+1}\in \left\{\left\{d_i^t-1,d_i^t,d_i^t+1\right\}mod8,8\right\}\hfill \\ C2:0\le {\mathbf{z}}_i^t\left(0\right)\le N_x-1,0\le {\mathbf{z}}_i^t\left(1\right)\le N_y-1\hfill \\ \begin{array}{c}C3:d_i^t\in \left\{0,1,\cdots ,8\right\}\hfill \\ C4:{\mathbf{z}}_i^{t+1}={\mathbf{z}}_i^t+Z\left(d_i^t\right)\hfill \end{array}\hfill \\ \begin{array}{c}C5:{\mathbf{LTPM}}_i^{\tilde{t}}=\phi \left({\mathbf{LTPM}}_i^t,{\mathbf{z}}_i^t,r_c,P_d,P_f\right)\hfill \\ C6:\mathbf{GTP}{\mathbf{M}}^{t+1}=\varphi \left({\left\{{\mu }_i^t·{\mathbf{LTPM}}_i^{\tilde{t}}\right\}}_{i=1}^N,r_c,P_d,P_f\right)\hfill \\ C7:{\mu }_i^t=\left\{\begin{array}{cc}1& R_i^t\ge R_{th}\\ 0& R_i^t<R_{th}\end{array}\right.\hfill \\ C8:{\mathbf{LTPM}}_i^{t+1}=\mathbf{GTP}{\mathbf{M}}^{t+1}\hfill \\ C9:{\mathbf{LTPM}}_1^0={\mathbf{LTPM}}_2^0=\cdots =\mathbf{GTP}{\mathbf{M}}^0\hfill \end{array}\hfill \end{array}.$$

(28)

At this point, the original optimization problem (P1) has been decomposed into two distinct subproblems: spectrum decision and search decision. Prior to implementing reinforcement learning as a solution to these two subproblems, a concise overview of multi-agent reinforcement learning and MAPPO will be provided. Subsequently, an MAPPO-based approach for both spectrum cooperation and search cooperation will be presented.

3.1. Structure of MARL

Multi-agent reinforcement learning is typically modeled as a Partially Observable Markov Decision Process (POMDP). POMDP is generally depicted as a six-tuple $\left(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathcal{O},\gamma \right)$, where $\mathcal{S}$ signifies the global state information, $\mathcal{O}=\left\{{\mathcal{O}}_1,{\mathcal{O}}_2,\cdots ,{\mathcal{O}}_N\right\}$ denotes the joint observation space of multiple agents, ${\mathcal{O}}_i$ indicates the local observation space of agent i, $\mathcal{A}=\left\{{\mathcal{A}}_1,{\mathcal{A}}_2,\cdots ,{\mathcal{A}}_N\right\}$ denotes the joint action space of multiple agents, and ${\mathcal{A}}_i$ represents the action space of agent i. At time step t, based on the local observation $o_i^t\in {\mathcal{O}}_i$ of the environment, agent i selects an action $a_i^t\in {\mathcal{A}}_i$ and receives a reward $r_i^t\in \mathcal{R}$ from the environment. Under the joint actions of multiple agents ${\mathbf{a}}_t=\left[a_1^t,a_2^t,\cdots ,a_N^t\right]$, the current global state $s^t\in \mathcal{S}$ transitions to the next state $s^{t+1}\in \mathcal{S}$ with some probability $p\left(s^{t+1}|s^t,{\mathbf{a}}_t\right)\in \mathcal{P}$. At time step $t+1$, agent i will obtain a local observation $o_i^{t+1}$, which is a part of the new global state $s^{t+1}$. The cumulative reward of agent i$G_i^t$ at time step t can be expressed as follows:

$$G_i^t=\sum _{l=0}^{\infty }{\gamma }^l{r}_i^{t+l},$$

(29)

where $\gamma \in \left(0,1\right)$ is the discount factor that describes the influence of future rewards on the current cumulative return.

3.2. PPO-Based MARL

Proximal policy optimization (PPO) [54] is a policy optimization algorithm that has been employed in reinforcement learning since its inception by the OpenAI research team in 2017. It is currently regarded as one of the most prevalent and pragmatic algorithms within the domain of reinforcement learning.

In reinforcement learning, an agent engages with the environment to generate trajectories $\tau =\left(s_0,a_0,r_0,s_1,\cdots ,s_T\right)$, with the objective of attaining enhanced cumulative rewards. The expected total reward can be expressed as follows:

$$J\left(\theta \right)=\sum _{\tau }r\left(\tau \right)p\left(\tau |\theta \right),$$

(30)

where $\theta$ represents the parameters of the policy network, $r\left(\tau \right)={\displaystyle \sum _{t=0}^{T-1}}{\gamma }^t{r}_t$ signifies the total reward accumulated by the agent throughout the trajectory, and $r_t$ denotes the reward received by the agent at time step t. $p\left(\tau |\theta \right)$ signifies the probability of the trajectory occurring under the policy ${\pi }_{\theta }$. To maximize the reward, the derivative of Equation (30) is taken, yielding the following form of the policy gradient

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{\tau \sim p\left(\tau |\theta \right)}\left[A_{\theta }\left(s_t,a_t\right){\nabla }_{\theta }log{p}_{\theta }\left(a_t|s_t\right)\right],$$

(31)

where $A_{\theta }\left(s_t,a_t\right)=q_{\theta }\left(s_t,a_t\right)-V_{\varphi }\left(s_t\right)$ denotes the advantage function, while $q_{\pi }\left(·\right)$ and $V_{{\pi }_{\phi }}\left(·\right)$ represent the action value function and the value function under the given policy ${\pi }_{\theta }$, respectively. The advantage function describes the relative quality of actions given a particular state. During the policy improvement process, if $A_{\theta }\left(s_t,a_t\right)>0$, the probability $p_{\theta }\left(a_t|s_t\right)$ is increased; otherwise, it is decreased. To enhance sample efficiency, an importance-sampling method is employed to collect samples, and Equation (31) can be rewritten as

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{\tau \sim p\left(\tau |\theta \right)}\left[\sum _{t=0}^T{\displaystyle \frac{p_{\theta }\left(a_t|s_t\right)}{p_{{\theta }^{\prime }}\left(a_t|s_t\right)}}{A}_{{\theta }^{\prime }}\left(s_t,a_t\right){\nabla }_{\theta }log{p}_{\theta }\left(a_t|s_t\right)\right].$$

(32)

The objective function corresponding to the gradient in the above equation is as follows:

$$J\left(\theta \right)={\mathbb{E}}_{\tau \sim p\left(\tau |\theta \right)}\left[{\displaystyle \frac{p_{\theta }\left(a_t|s_t\right)}{p_{{\theta }^{\prime }}\left(a_t|s_t\right)}}{A}_{{\theta }^{\prime }}\left(s_t,a_t\right)\right],$$

(33)

where $p_{{\theta }^{\prime }}\left(a_t|s_t\right)$ denotes the policy that interacts with the environment, and $p_{\theta }\left(a_t|s_t\right)$ signifies the target policy. It is evident that the closer the two are, the more optimal the training outcomes. To this end, PPO-clip imposes constraints on the difference between them within a specified range through the implementation of a clipping function. The objective function associated with PPO-clip is as follows:

$$\begin{array}{c}{J}^{clip}\left(\theta \right)={\mathbb{E}}_{\left(s_t,a_t\right)}\left[min\left(r_t\left(\theta \right)A_{{\theta }^{\prime }}\left(s_t,a_t\right),\right.\right.\hfill \\ \left.\left.clip\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right)A_{{\theta }^{\prime }}\left(s_t,a_t\right)\right)\right]\hfill \end{array},$$

(34)

where $r_t\left(\theta \right)={\displaystyle \frac{p_{\theta }\left(a_t|s_t\right)}{p_{{\theta }^{\prime }}\left(a_t|s_t\right)}}$. $clip\left(x,l,r\right)\stackrel{\Delta }{=}max\left(min\left(x,r\right),l\right)$, which restricts x to the interval $\left[l,r\right]$. Here, $\epsilon$ is a hyperparameter that represents the range for truncation.

In a multi-agent scenario, PPO is extended to multi-agent PPO (MAPPO) [55], where each agent possesses a policy network. The agent interacts with the environment based on its local observations in order to update its policy network. As a result, Equation (34) thus becomes

$$\begin{array}{c}{J}^{clip}\left({\theta }^i\right)={\mathbb{E}}_{\left(s_t,a_t\right)}\left[min\left(r_t\left({\theta }^i\right)A_{{\theta }^{\prime }}\left(o_t^i,a_t^i\right),\right.\right.\hfill \\ \left.\left.clip\left(r_t\left({\theta }^i\right),1-\epsilon ,1+\epsilon \right)A_{{\theta }^{\prime }}\left(o_t^i,a_t^i\right)\right)\right]\hfill \end{array}.$$

(35)

MAPPO is a reinforcement learning method that employs a combination of centralized training and decentralized execution (CTDE). During the centralized training process, the value network of agent i is configured to learn a centralized value function $V_{\varphi }^i\left(s_t\right)$ based on the global state $s_t$, with the objective function being

$$L\left({\varphi }^i\right)={\mathbb{E}}_{s_t}\left[{\left(V_{{\varphi }^i}\left(s_t\right)-{\widehat{r}}_t^i\right)}^2\right],$$

(36)

where ${\varphi }^i$ signifies the value network parameters of the agent i, and ${\widehat{r}}_t^i$ denotes the total return received by the agent i in global state $s_t$. Following the acquisition of $V_{{\varphi }^i}\left(s_t\right)$, the generalized advantage estimation (GAE) can be utilized to estimate $A_{{\theta }^i}^{GAE}\left(s_t,a_t^i\right)$, as demonstrated in the subsequent equation:

$$A_{{\theta }^i}^{GAE}\left(s_t,a_t^i\right)=\sum _{l=0}^{\infty }{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^i,$$

(37)

where ${\delta }_t^i={\widehat{r}}_t^i+\gamma V_{{\varphi }^i}\left(s_{t+1}\right)-V_{{\varphi }^i}\left(s_t\right)$ denotes the temporal difference (TD) error, and $\lambda \in \left[0,1\right]$ is an additional hyperparameter that has been introduced in GAE.

3.3. MAPPO-Based Spectrum Collaboration

We define the POMDP elements corresponding to the spectrum collaboration optimization model (P2).

(1) Observation and state: In the $t^{\prime }$-$th$ spectrum decision cycle, the local observation ${\mathbf{o}}_{i,t^{\prime }}^c$ of UAV i comprises the received signal strength indication (RSSI) of all M channels: that is, ${\mathbf{o}}_{i,t^{\prime }}^c=\left[RSS{I}_{i,1}^{t^{\prime }},RSS{I}_{i,2}^{t^{\prime }},\cdots ,RSS{I}_{i,M}^{t^{\prime }}\right]$. Here, $RSS{I}_{i,m}^{t^{\prime }}$ represents the RSSI of the m-$th$ channel observed by UAV i, defined as

$$RSS{I}_{i,m}^{t^{\prime }}=\sum _{\genfrac{}{}{0pt}{}{n\in \mathcal{N}\hfill }{n\ne i\hfill }}\left(c_{n,m}^{t^{\prime }-1}{P}_u{10}^{-{\displaystyle \frac{g_{n,X}^{t^{\prime }-1}}{10}}}+{\chi }_m^{t^{\prime }-1}{P}_J{10}^{-{\displaystyle \frac{h_{J,G}^{t^{\prime }-1}}{10}}}+B{\sigma }^2\right).$$

(38)

The first term of Equation (38) signifies the power received by UAV i from other UAVs through channel m. The subsequent term denotes the power received by UAV i from the jammer through channel m. The final term accounts for the power of the Gaussian white noise received by UAV i on channel m. Moreover, we merely concatenate the local observations of all agents to form global state information ${\mathbf{s}}_{t^{\prime }}^c$, which is given by

$${\mathbf{s}}_{t^{\prime }}^c=\left[{\mathbf{o}}_{i,t^{\prime }}^c,{\mathbf{o}}_{i,t^{\prime }}^c,\cdots ,{\mathbf{o}}_{i,t^{\prime }}^c\right].$$

(39)

In order to differentiate between the same type of parameters for spectrum collaboration and search collaboration, superscripts “c”and “s” are utilized. For example, ${\mathbf{o}}_{i,t^{\prime }}^c$ and ${\mathbf{s}}_{t^{\prime }}^c$ denote local observations and global state information in spectrum collaboration, while ${\mathbf{o}}_{i,t}^s$ and ${\mathbf{s}}_{t^{\prime }}^s$ are employed to represent local observations and global state information in search collaboration. Other parameters are denoted in a similar fashion.

(2) Action: In the $t^{\prime }$-$th$ spectrum decision cycle, UAV i randomly selects a channel m from the set of all channels $\mathcal{M}=\left\{1,2,\cdots ,M\right\}$ to transmit back to the GC based on its local observation ${\mathbf{o}}_{i,t^{\prime }}^c$. Thus, $a_{i,t^{\prime }}^c=m,m\in \mathcal{M}$.

(3) Reward: In order to optimize the objective function of the given optimization problem (P2) and thereby achieve spectrum collaboration among the UAVs, it is necessary to design a reward function that will encourage each agent to maximize its own achievable rate to GCS. Hence, the following reward function is therefore defined:

$$r_{i,t^{\prime }}^c=\left(1-{\varsigma }_{i,t^{\prime }}\right)R_i^{t^{\prime }}.$$

(40)

Furthermore, the multiplicative channel switching factor ${\varsigma }_{i,t^{\prime }}$ was intentionally incorporated with the objective of reducing the channel switching frequency of the UAVs and further cutting down on system overhead, which is defined as follows:

$${\varsigma }_{i,t^{\prime }}=\left\{\begin{array}{cc}0,& a_{i,t^{\prime }}^c=a_{i,t^{\prime }-1}^c\\ {\varsigma }_0,& a_{i,t^{\prime }}^c\ne a_{i,t^{\prime }-1}^c\end{array}\right..$$

(41)

where ${\varsigma }_0$ is a hyperparameter. In the absence of channel switching between two consecutive spectrum decision cycles, the reward remains as it is. Conversely, when a channel switch occurs, the reward is reduced.

With regard to the network architecture, both the policy network and value network comprise multiple fully connected layers. It is important to acknowledge that during spectrum collaboration, UAVs located at the same position and under the same observation should select different channels. This implies that the strategies employed by different UAVs are distinct. Consequently, it is essential that each MAPPO agent incorporates a separate policy network and value network.

In conclusion, the proposed MAPPO-based spectrum collaboration algorithm under jamming conditions for (P2) is summarized in Algorithm 1.

Algorithm 1: MAPPO-based spectrum collaboration under jamming conditions.

3.4. MAPPO-Based Search Collaboration

In a similar manner, the POMDP elements with regard to the search collaboration model (P3) are defined as follows.

(1) Observation and state: In the t-$th$ search decision cycle, the local observation ${\mathbf{o}}_{i,t}^s$ of UAV i encompasses the ${\mathbf{LTPM}}_i^t$ it carries, the coordinates of the current grid it is located within, and its heading angle from the last search decision cycle, represented as ${\mathbf{o}}_{i,t}^s=\left[{\mathbf{LTPM}}_i^t,{\mathbf{z}}_i^t,d_i^{t-1}\right]$. As for the global state information ${\mathbf{s}}_t^s$, this comprises three constituent elements: $\mathbf{GTP}{\mathbf{M}}^{t,*}$, obtained from the specified combination of ${\mathbf{LTPM}}_i^t$ from all UAVs, is written as

$$\mathbf{GTP}{\mathbf{M}}^{t,*}=\varphi \left({\left\{{\mathbf{LTPM}}_i^{\tilde{t}}\right\}}_{i=1}^N,r_c,P_d,P_f\right);$$

(42)

${\left\{{\mathbf{z}}_i^t\right\}}_{i=1,\cdots N}$ is the coordinates of the current grids occupied by all UAVs; ${\left\{d_i^{t-1}\right\}}_{i=1,\cdots ,N}$ is the heading angles of all UAVs in the previous time slot. Therefore, the following expression can be used to represent ${\mathbf{s}}_t^s$:

$${\mathbf{s}}_t^s=\left[\mathbf{GTP}{\mathbf{M}}^{t,*},{\left\{{\mathbf{z}}_i^t\right\}}_{i=1,\cdots N},{\left\{d_i^{t-1}\right\}}_{i=1,\cdots ,N}\right];$$

(43)

(2) Action: In the t-$th$ search decision cycle, the available actions for the UAVs are the eight given heading angles, denoted as $a_{i,t}^s=d_i^t$, $d_i^t\in \left\{0,1,\cdots ,7\right\}$. However, to ensure that the actions of the UAVs satisfy constraints $C1$ and $C2$ in (P3) and also to accelerate the training speed, an action-masking mechanism is introduced. With regard to the boundary constraints in $C1$, action masking is primarily related to the current grid coordinates of the UAV, denoted as $mas{k}_1\left({\mathbf{z}}_i^t\right)$. The masked actions differ when the UAV is in different grids. As for the heading angle constraint in $C2$, action masking is contingent on the UAV’s heading angle from the preceding search decision, written as $mas{k}_2\left(d_i^{t-1}\right)$. Since both ${\mathbf{z}}_i^t$ and $d_i^{t-1}$ can be obtained from the current observation ${\mathbf{o}}_{i,t}^s$, the final masked actions can be represented as follows:

$$mask\left({\mathbf{o}}_{i,t}^s\right)=mas{k}_1\left({\mathbf{z}}_i^t\right)\cup mas{k}_2\left(d_i^{t-1}\right).$$

(44)

In addition, to prevent all available actions from being masked, we introduce a hovering action into the action set. When all flight directions are masked, the UAV will hover, and in the next time slot after hovering, it can choose any of the eight flight directions.

(3) Reward: As the UAV conducts its search, the uncertainty of the task area undergoes a gradual decrease. Consequently, the variation in the uncertainty of the task area between two adjacent search decision cycles can be utilized to delineate the reward obtained from the UAV’s search. Firstly, it is necessary to define the uncertainty of the task area, and the uncertainty function of grid $(x,y)$ in the t-$th$ search decision cycle is defined as follows:

$${\eta }_{\left(x,y\right)}^t\stackrel{\Delta }{=}{e}^{-K_q\left|Q_{(x,y)}^t\right|},$$

(45)

where $K_q$ is a constant greater than 0. When $p_{\left(x,y\right)}^t=0.5$, the corresponding ${\eta }_{\left(x,y\right)}^t=1$, indicating that the uncertainty of the grid $(x,y)$ is at its maximum. When $p_{\left(x,y\right)}^t=0$ or $p_{\left(x,y\right)}^t=1$, the corresponding ${\eta }_{\left(x,y\right)}^t=0$, which means that the presence of a target in grid $(x,y)$ can be fully determined, resulting in the lowest possible uncertainty for grid $(x,y)$.

The joint reward for the UAV’s collaborative search during the t-$th$ search decision cycle can be expressed as follows:

$$r_t^s=-\sum _{y=0}^{N_y-1}\sum _{x=0}^{N_x-1}{\eta }_{\left(x,y\right)}^t.$$

(46)

Finally, an explanation is provided of the network architecture. Given that the $\mathbf{GTP}{\mathbf{M}}^{t,*}$ in the agent’s observations is two-dimensional data, a convolutional network is specifically used to extract features from it. Additionally, to avoid feature loss, pooling layers are not utilized. Subsequent to flattening the convolved data, it is concatenated with other data from the observations and fed into a fully connected multilayer network for further feature extraction. The masked action is then combined with the features and fed into a fully connected network, ultimately outputting action probabilities through a softmax layer. The value network also utilizes convolutional layers to process two-dimensional data in its input. Figure 5 illustrates the architecture of the policy network. Similarly to spectrum collaboration, each agent also employs a set of independent policy and value networks during search collaboration. The MAPPO-based search collaboration algorithm for (P3) is outlined in Algorithm 2.

Algorithm 2: MAPPO-based search collaboration.

Once the N spectrum collaboration policy networks and N search collaboration policy networks are obtained from Algorithm 1 and Algorithm 2, respectively, and each UAV is loaded with one spectrum collaboration policy network and one search collaboration policy network, the N UAVs are then capable of performing joint search tasks under interference conditions, as illustrated in Figure 6.

3.5. Computational Complexity Analysis

The computational complexity of the proposed algorithm is fundamentally determined by three key factors: (1) the state-space dimensionality of the agents, (2) the action-space dimensionality, and (3) the architectural configuration of both the policy and value networks. For the search coordination agent, the policy network processes an $N_s\times N_s$ TPM matrix through two convolutional layers followed by three fully connected layers, as shown in Figure 5, resulting in a complexity of $\mathcal{O}\left(K_0{N}_s^2\right)$, where $K_0$ represents the combined complexity factor of these operations. The corresponding value network shares a similar structure with slightly different fully connected layer dimensions, yielding a complexity of $\mathcal{O}\left(K_1{N}_s^2\right)$. In contrast, the spectrum coordination agent’s networks eliminate convolutional layers and operate on state variables proportional to the number of agents N, with complexities of $\mathcal{O}\left(K_{3}N\right)$ for the policy network and $\mathcal{O}\left(K_{4}N\right)$ for the value network. The total algorithm complexity combines these components as $\mathcal{O}\left(K_0{N}_s^2\right)+\mathcal{O}\left(K_1{N}_s^2\right)+\mathcal{O}\left(K_{3}N\right)+\mathcal{O}\left(K_{4}N\right)$. Given that the grid resolution $N_s$ typically exceeds the number of UAVs, N, the overall complexity is dominated by the quadratic term $\mathcal{O}\left(N_s^2\right)$, establishing a square relationship with the TPM matrix size. This analysis reveals that larger search areas or finer grid partitions will quadratically increase computation time, suggesting two practical optimization strategies: employing UAVs with greater coverage to reduce $N_s$ or strategically partitioning the mission area into subregions before algorithm application to maintain computational efficiency while preserving search effectiveness. The quadratic scaling behavior provides clear guidance for balancing algorithmic performance with operational requirements in real-world deployments.

4. Simulation and Results

In this section, we present the simulation scenario along with three baseline methods. The experimental results are then described and analyzed.

4.1. Parameter Setting

In the simulation, the mission area is assumed to be a rectangular region of 5 km × 5 km. The UAV search collaboration cycle and the spectrum collaboration cycle are set to 10 s and 100 ms, respectively, with 16 channels available. The flight speed v of the UAV is 20 m/s, and since the grid length is the distance that the UAV travels during a search–collaboration cycle, it is set to 200 m. Therefore, the entire mission area can be divided into a 25 × 25 grid. The UAV has a detection radius of 300 m, allowing it to simultaneously scan a 3 × 3 grid within its field of view. In addition, three jammers are strategically deployed to disrupt UAV communications. Each jammer employs a sweeping interference pattern characterized by deterministic channel switching through all M available channels, and the jammers share the same sweeping period but maintain random initial phase offsets. Further simulation parameters are given in

Table 2. Simulation parameters of the environment.

Parameter	Description	Value
N	Number of UAVs	3
M	Number of targets	15
h	UAV flight altitude	100 m
$P_d$	Detection probability	0.9
$P_f$	False alarm probability	0.1
${\eta }_0,{\eta }_1$	Thresholds for determining the absence and presence of a target	0.05, 0.95
$f_c$	Carrier frequency	2 GHz
B	Channel bandwidth	1 MHz
$P_u$	UAV power	23 dbm
$P_J$	Jammer power	60 dbm
$b_1$,$b_2$	Environmental constants	0.13, 11.9
${\psi }_{NLoS}$	NLoS link’s additional attenuation factor	20 dB
${\sigma }^2$	White noise power spectral density	−100 dbm/MHz
${\tau }_0$	Threshold for the information transmitted to the GCS	0.9
${\varsigma }_0$	Hyperparameter for channel switching	0.1
${\gamma }^c,{\gamma }^s$	Discount factor	0.99
${\lambda }^c,{\lambda }^s$	GAE parameter	0.95
${\epsilon }^c,{\epsilon }^s,$	Clip factor	0.2
$l{r}_a^c$,$l{r}_c^c$	Policy/critic network learning rate of MAPPO-based spectrum collaboration	5 × 10−4, 5 × 10−4
$l{r}_a^s$,$l{r}_c^s$	Policy/critic network learning rate of MAPPO-based search collaboration	5 × 10−4, 5 × 10−3

.

4.2. Performance Benchmark

To evaluate the performance of the proposed MAPPO-based method in both spectrum collaboration and search collaboration, three baseline methods are used for comparison.

(1) IPPO: IPPO-based spectrum collaboration method or search collaboration method. The network structure, observations, and rewards are identical to those in the MAPPO-based method. However, a key distinction is that, during the training phase, the critic network of IPPO solely utilizes the local observations of each agent as input, as opposed to the global state.

(2) QMIX: QMIX-based spectrum collaboration methods or search collaboration methods. The observations and rewards for the agents are exactly the same as those in the MAPPO-based method; the difference lies in the network structure utilized by each agent, which is QMIX-based, with a set of Q networks deployed for each agent. The architecture of these Q networks is identical to that of the corresponding MAPPO method, with all inputs as local observations. The outputs of all agents’ Q networks are then processed through a mixing Q network to obtain the global action value function, with the input to the mixing Q network as the global state.

(3) IQL: IQL-based spectrum collaboration methods or search collaboration methods. The observations and rewards for the agents are also the same as those in the MAPPO-based method. The distinguishing factor pertains to the deployment of a set of Q networks by each agent. The architecture of these Q networks is identical to that of the corresponding MAPPO method, with each agent’s Q network receiving local observations as input.

4.3. Performance Evaluation

First, we conducted simulations to evaluate the convergence of the proposed Algorithm 1 and compare it with other baseline methods. The training process involved 1000 episodes, with each episode consisting of 1000 steps, and the reward of all agents for each episode was calculated. As illustrated in Figure 7, the MAPPO-based and IPPO-based spectrum collaboration methods demonstrate faster convergence and attain higher final convergence values. The proposed MAPPO-based spectrum collaboration achieves throughput improvements of 68.7% and 146.2% over QMIX and IQL, respectively. In contrast, QMIX exhibits slightly lower performance, while IQL demonstrates the poorest performance. In the context of spectrum collaboration, policy gradient-based reinforcement learning methods exhibit superior performance in comparison to those based on value functions. Despite the fact that MAPPO uses global states during training, the IPPO’s critic network, which only utilizes local observations, demonstrates performances very close to that of MAPPO. This is because the RSSI observed by different agents has a strong correlation, resulting in a high degree of similarity between local and global observations.

Following the convergence of the reward curve of the proposed MAPPO-based algorithm, a random selection of four consecutive steps from a single episode was conducted, where each step can be regarded as a spectrum decision cycle, to analyze the channel selection of the UAV during these four slots, as illustrated in Figure 8. The grey grids represent the interfered channels, the blue grids represent the available channels, and the yellow grids indicate the channels selected by the UAV. It is evident that in each decision cycle, the UAV not only avoids interfering channels but also avoids selecting the same channel as other UAVs. This outcome demonstrates the efficacy of the proposed algorithm in facilitating spectrum collaboration among UAVs in the presence of interference.

A simulation was also conducted on the effect of the weighting factor ${\varsigma }_{i,t^{\prime }}$ in Equation (40) of the spectrum collaboration section. Under MAPPO-based spectrum collaboration, Figure 9a,b present the channel selections of the UAVs over 18 spectrum collaboration cycles for ${\varsigma }_{i,t^{\prime }}=0$ and ${\varsigma }_{i,t^{\prime }}=0.1$, respectively. Distinct colors are employed to differentiate the various channels. It is evident from these figures that when ${\varsigma }_{i,t^{\prime }}=0.1$, the UAVs can achieve spectrum utilization while reducing the number of channel switchings, thereby lowering the system overhead required for spectrum collaboration.

In Figure 10, a comparison of the convergence between the proposed MAPPO-based search collaboration algorithm and other baseline methods is presented. It is assumed that the agents achieve complete spectrum collaboration in each search decision cycle, i.e., ${\mu }_i^t\equiv 1$ in Equation (21). It is further assumed that all methods start with the same initial environment. During the training process, 1000 episodes were conducted, with each episode consisting of a maximum of 500 steps, thereby ensuring that the agents could complete the search task in each episode. As is evident in Figure 10, the proposed MAPPO-based algorithm and the IPPO-based algorithm demonstrate comparable performances, thus validating the efficacy of the proposed approach. This observation aligns with the findings of the convergence simulation of spectrum collaboration, wherein the performance of both algorithms is found to be similar. The underlying reason for this consistency in performance can be attributed to the assumption of complete spectrum collaboration, i.e., the condition where ${\mathbf{LTPM}}_i^t=\mathbf{GTP}{\mathbf{M}}^t$. This assumption results in the global states utilized by MAPPO becoming equivalent to the local states employed by IPPO during the training process. Furthermore, the performances of QMIX and IQL are relatively similar, as both are trapped in local optima, which also indicates that in collaborative search scenarios, policy gradient-based reinforcement learning algorithms still outperform value-function-based reinforcement learning algorithms.

To validate the performance of the proposed MAPPO-based search collaboration algorithm, the search trajectories of the UAV were generated by both the proposed algorithm and baseline methods under the same initial conditions. The distribution of the targets within the task area is shown in Figure 11a. Based on the distribution of the targets, the corresponding initial TPM can be generated, as illustrated in Figure 11b. The methodology for generating the TPM entails the creation of a two-dimensional Gaussian probability distribution by Equation (47), centered at the location of each target in the sequence, which is then utilized to update the corresponding elements in the TPM. The remaining positions in the TPM are populated with ${\eta }_0$, signifying the absence of targets.

$$p_{\left(x,y\right)}^0=1-\prod _{m=1}^M\left[1-{\displaystyle \frac{1}{2\pi {\sigma }_0{\sigma }_1}}{e}^{-{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{{\left(x-z_m^0\right)}^2}{2{\sigma }_0^2}}+{\displaystyle \frac{{\left(y-z_m^1\right)}^2}{2{\sigma }_1^2}}\right)}\right],$$

(47)

As illustrated in Figure 12, the search trajectories of the UAVs are depicted under various search collaboration methods, with complete spectrum collaboration. The red, yellow, and green poly-lines in the figure represent the motion trajectories of UAV 1, UAV 2, and UAV 3, respectively, with pentagrams indicating their starting points. To more clearly illustrate the UAVs’ movement paths, we label every 4th trajectory point with its corresponding step number. The crosses in the figure denote the targets. It is evident that, in comparison to alternative baseline methods, the MAPPO-based collaborative search algorithm requires a fewer number of steps to complete the search of the task area under identical initial conditions. It is discernible that the areas suspected of harboring targets are predominantly searched by a solitary UAV, which indicates that effective search collaboration is indeed attained among the diverse UAVs. Furthermore, the UAV flight trajectories satisfy the boundary constraint $C1$ and turning constraint $C2$ specified in (P3), thereby demonstrating the effectiveness of the action-masking mechanism. Due to the presence of the turning constraint, UAVs require more steps when executing turns of 90 degrees or more, resulting in circular trajectories, which better align with the actual flight characteristics of the UAVs. As demonstrated in Figure 12c,d, while the QMIX- and IQL-based methods attain a certain degree of spectrum collaboration, they exhibit substantial redundancy in their steps, signifying the presence of further optimization potential and the possibility of being trapped in local optima.

In Figure 13, a comparison is drawn between the coverage rates of the four algorithms under complete spectrum collaboration. Utilizing the TPM from Figure 11b as the initial TPM, the coverage rate commences at 78% due to the fact that 22% of the grid cells in this TPM continue to exhibit uncertainty. In comparison to the baseline algorithms, the proposed MAPPO-based algorithm requires the fewest steps, and for the majority of the search process, the coverage rate of the MAPPO-based algorithm exceeds that of the other algorithms, thereby indicating superior performance. However, between 10 and 20 steps, the IPPO- and QMIX-based methods demonstrate a higher coverage rate, suggesting that the proposed MAPPO-based algorithm provides a local optimum. This indicates that there is potential for enhancing the search efficiency.

We also conducted a comparative analysis of the coverage rates of four distinct search collaboration methods under MAPPO-based spectrum collaboration. As depicted in Figure 14. Still utilizing the TPM from Figure 11b as the initial TPM, our analysis revealed that the “perfect” curve, representing an MAPPO-based collaborative search under complete spectrum collaboration, exhibited the optimal performance. Our experiments demonstrate that even with partial spectrum collaboration, the MAPPO-based search algorithm consistently outperforms baselines with 16.7–48.3% reductions in search completion steps. However, the coverage rate of the perfect collaboration is consistently greater than that of the MAPPO-based spectrum collaboration algorithm throughout the search process, as it achieves complete spectrum collaboration during the whole process. This simulation result reinforces the notion that spectrum collaboration is a significant factor impacting search collaboration. It is evident that the proposed spectrum collaboration algorithm is yet to attain full efficacy in achieving spectrum collaboration.

5. Conclusions

In this paper, we have developed a multi-UAV collaborative search scheme under interference conditions based on multi-agent reinforcement learning. An optimization problem was formulated to minimize the total search time by optimizing both the selection of channels and the search heading angles of the UAVs. The original problem was decomposed into two separate parts and solved by proposing RSSI-MAPPO and TPM-MAPPO. The simulation results indicated that our proposed scheme can ensure the collaborative search of UAVs under interference conditions, significantly reducing search times compared to other baseline algorithms. Our future work will focus on the following: (1) distributed LTPM exchange under interference and (2) game-theoretic meta-learning solutions for adaptive jammers, extending beyond the current fixed-pattern jamming scenario.