Multi-UAV Cooperative Search in Partially Observable Low-Altitude Environments Based on Deep Reinforcement Learning

Highlights

What are the main findings?

• Propose a multi-agent deep reinforcement learning algorithm named Normalizing Graph Attention Soft Actor-Critic (NGASAC) to solve the problem of multi-UAV cooperative search in low-altitude partially observable environments.
• To address the multi-UAV collaborative search problem in real scenarios, we propose a phased search strategy.

What are the implications of the main findings?

• NGASAC integrates a normalizing flow (NL) layer and a multi-head graph attention network (MHGAT). The normalizing flow technique maps traditional Gaussian sampling to a more complex action distribution, thereby enhancing the expressiveness and flexibility of the policy. Simultaneously, by constructing a multi-head graph attention network that captures “obstacle–target” relationships, the algorithm improves the UAVs’ ability to learn and reason about complex spatial topologies, leading to significantly better performance in cooperative search and stable surveillance of hidden targets.
• The phased search strategy achieves adaptive allocation of search resources by dynamically responding to changes in the target state, thereby enhancing search efficiency and task success rate.

Abstract

Multi-Unmanned Aerial Vehicle (Multi-UAV) cooperative search represents a cutting-edge research direction in the field of unmanned aerial vehicle applications. The use of multi-UAV systems for low-altitude target search and area surveillance has become an effective means of enhancing security capabilities. In practical scenarios, UAVs rely on onboard sensors to acquire environmental information; however, due to the limited perceptual range of these sensors, their observation capabilities are inherently local and constrained. This paper investigates the problem of multi-UAV cooperative search in partially observable low-altitude environments, where each UAV possesses a circular sensing range with a finite radius. Target location information is only obtained when a target enters the field of view of any UAV. The objective is to achieve cooperative search and sustain continuous surveillance while ensuring safety among UAVs and with the environment. To address this challenge, we propose a novel multi-agent deep reinforcement learning (MADRL) algorithm named Normalizing Graph Attention Soft Actor-Critic (NGASAC). This algorithm integrates a normalizing flow (NL) layer and a multi-head graph attention network (MHGAT). The normalizing flow technique maps traditional Gaussian sampling to a more complex action distribution, thereby enhancing the expressiveness and flexibility of the policy. Simultaneously, by constructing a multi-head graph attention network that captures “obstacle–target” relationships, the algorithm improves the UAVs’ ability to learn and reason about complex spatial topologies, leading to significantly better performance in cooperative search and stable surveillance of hidden targets. Simulation results demonstrate that the NGASAC algorithm markedly outperforms baseline methods such as Multi-Agent Soft Actor-Critic (MASAC), Multi-Agent Proximal Policy Optimization (MAPPO), and Multi-Agent Deep Deterministic Policy Gradient (MADDPG) across multiple evaluation metrics, including success rate, task time, and obstacle avoidance capability. Furthermore, it exhibits strong generalization performance and robustness.

1. Introduction

In recent years, UAV technology has undergone rapid development, driving significant technological advancements in fields such as surveying, aerial photography, and emergency response [1]. With continuous improvements in visual sensors, embedded processors, and data processing techniques, UAV systems are increasingly deployed for target search tasks in high-risk or even hostile environments [2]. Multi-UAV cooperative search [3,4] involves the systematic coordination of multiple drones to explore unknown areas and locate targets. Through strategic collaboration among UAVs, this approach significantly enhances the efficiency and robustness of search and surveillance operations. As a result, it has been widely adopted in critical applications including disaster relief, power line inspection, and forest fire monitoring [5,6,7].

This paper focuses on the problem of cooperative dynamic target search in low-altitude overhead surveillance scenarios. Its theoretical foundation can be traced to the predator–prey interaction model—a classic problem extensively studied in robotics and control theory [8]. Within this framework, the searching UAVs act as searcher, dynamically formulating strategies based on environmental information to approach and continuously monitor the target as quickly as possible, while the target acts as an evader, striving to maintain maximum distance to avoid detection [9]. However, most existing studies assume full observability—that both pursuers and evaders have complete access to each other’s information [10]—which significantly diverges from real-world applications characterized by information asymmetry and partial observations limited by sensor capabilities. Therefore, investigating cooperative UAV search strategies under partial observability is both more practical and challenging.

The search evasion problem can be regarded as a pursuit evasion game. Traditional approaches often employ differential game theory, solving the Hamilton–Jacobi–Isaacs (HJI) equation to derive optimal strategies. Alternatively, some studies use expert rules or geometry-based cooperative control methods for multi-UAV coordination [11]. However, in real-world environments with complex obstacle distributions and highly uncertain target behavior, it is difficult to establish accurate dynamic models [12]. Moreover, targets may repeatedly disappear and reappear, further complicating the design of conventional search policies [13].

With the rapid development of artificial intelligence, DRL—a major branch of machine learning—has demonstrated great potential in agent decision-making. By learning through trial-and-error interactions with the environment, DRL satisfies the requirement for real-time online decision-making and is an ideal approach for control problems in unknown environments [14,15]. It does not rely on precise environmental dynamics and can directly approximate control policies or optimize control parameters via neural networks [16]. Furthermore, learning-based methods exhibit superior generalization and adaptability in unfamiliar environments compared to traditional control approaches [17].

MADRL has gradually emerged as an effective way to address pursuit evasion problems due to its capacity to handle cooperative and competitive interactions among agents. After a detailed analysis of the specific challenges in search evasion scenarios, we propose the Normalizing Graph Attention Soft Actor-Critic (NGASAC) algorithm. This approach builds upon the Soft Actor-Critic (SAC) architecture [18] and extends it into a Centralized Training with Decentralized Execution (CTDE) framework [19] suitable for multi-agent decision-making under partial observability. To overcome the limitations of conventional action sampling methods that often rely on simplistic Gaussian distributions—which restrict policy expressiveness [20,21]—we introduce a normalizing flow (NF) layer inspired by [22]. This technique uses invertible transformations to map the original sampling distribution into a more complex action distribution. Updated in an end-to-end manner within the MADRL framework, it enhances the flexibility and diversity of policy representations. Additionally, while many MADRL algorithms based on the actor-critic framework use Multi-Layer Perceptron (MLP) to construct centralized critics [21,23], we incorporate a Multi-Head Graph Attention Network (MHGAT) into the critic network, drawing inspiration from [24]. This enables explicit modeling of the spatial relationships among UAVs, obstacles, and targets, thereby improving situational reasoning in complex environments. Our main contributions are as follows:

(1) We developed a practical multi-UAV cooperative search model under partial observability in low-altitude environments, incorporating overhead perception, partial observability, dynamic target behavior, and multiple safety constraints.

(2) We proposed the NGASAC algorithm, which integrates NF and MHGAT to significantly enhance policy expressiveness and spatial relational reasoning, with ablation studies validating the effectiveness of each module.

(3) Extensive simulations demonstrate that the proposed algorithm outperforms existing state-of-the-art methods in success rate, task efficiency, and robustness, and exhibits superior generalization across various obstacle configurations, numbers of UAVs, and unseen task scenarios.

The remainder of this paper is organized as follows: Section 2 reviews related work in cooperative search and MADRL; Section 3 provides a formal model of the multi-UAV cooperative search problem in partially observable low-altitude environments; Section 4 formulates the corresponding Partially Observable Markov Decision Process (POMDP) and elaborates on the proposed NGASAC algorithm; Section 5 presents extensive experimental evaluations and results analysis; and Section 6 concludes the paper and suggests future research directions.

2. Related Work

2.1. Cooperative Search

In recent years, multi-UAV cooperative search strategies have attracted widespread attention, with numerous scholars conducting in-depth research from various perspectives. Fei et al. [25] proposed a robust cooperative flight strategy for multi-UAV systems operating under uncertain communication conditions, enhancing task performance in unreliable communication environments. Shen et al. [26] designed a digital twin-based distributed training framework that supports cooperative decision-making and dynamic adaptation of multiple UAVs in complex scenarios through deep integration of virtual simulation and physical systems. Meng et al. [27] introduced an evolutionary state estimation-based multi-strategy jellyfish search algorithm, significantly improving the path planning quality and cooperative efficiency of multi-UAV systems in unknown environments. Zhang et al. [28] focused on the cooperative reconnaissance of static targets using multiple UAVs, formulating the task as a multi-objective optimization model to achieve an effective balance between broad coverage and precise positioning, thereby enhancing the reliability of search outcomes. Yu et al. [29] addressed the cooperative search for dynamic targets with multiple UAVs and developed an improved optimization algorithm incorporating multiple gene types; experiments demonstrated that the approach increases the search efficiency of the UAV swarm. Hentou et al. [30] proposed a complete navigation and control system for the problem of efficient navigation in dynamic environments. This system effectively simulated human reasoning capabilities and enabled intelligent agents to navigate autonomously in dynamic environments. Although these studies have made significant progress in path planning, cooperative coverage, and target localization, most are based on the ideal assumption of global observability and do not adequately address the issue of partial observability caused by sensor limitations in real-world environments. Particularly in complex low-altitude urban settings, factors such as constrained UAV mobility, obstacle occlusion, and the high maneuverability of dynamic targets pose substantial challenges to cooperative search. It is evident that current methods still exhibit notable limitations in handling partial observations and dynamic decision-making, highlighting the urgent need for more adaptive and reasoning-capable cooperative decision-making frameworks.

2.2. MADRL

MADRL has demonstrated remarkable perception and decision-making capabilities in complex environments, achieving significant progress in various domains such as path planning [14], algorithm configuration [31], and intelligent decision-making [32]. Without relying heavily on prior knowledge, MADRL enables agents to autonomously learn cooperative strategies through continuous interaction with the environment, maximizing long-term cumulative rewards even in highly uncertain settings [33,34]. In UAV-related applications, Xing et al. [14] proposed an improved deep reinforcement learning algorithm for multi-UAV trajectory planning in environments with unknown obstacles. Their experiments show that the algorithm effectively identifies optimal policies that maximize expected rewards. Ming et al. [32] developed a deep reinforcement learning-based online operator selection framework, significantly enhancing the performance of multi-objective evolutionary algorithms in complex optimization problems. Wu et al. [35] investigated cooperative search in multi-layer aerial computing networks and introduced a MADRL method combining parameter sharing and action masking. This approach maximizes target discoveries and area coverage while minimizing search uncertainty, thereby improving overall system utility. Fu et al. [17] addressed multi-UAV cooperative pursuit using a decomposed MADDPG algorithm, alleviating credit assignment challenges in collaboration through value decomposition. Zhang et al. [10] proposed a multi-agent collaborative bidirectional coordination and target detection algorithm that maintains strong recognition and tracking performance under partial observability. Du et al. [36] studied multi-UAV pursuit evasion in 2D urban airspace under communication constraints, offering valuable insights for cooperative control in low-bandwidth environments. Mateescu et al. [37] utilized a simulation testing environment incorporating the digital twin of the NAO robot and a parallel learning process to accelerate the progress of machine learning.

Based on the above analysis, although MADRL shows considerable potential in multi-UAV cooperative applications, several limitations remain. Many algorithms rely on simplistic Gaussian distributions for policy sampling, which struggle to represent complex multi-modal decision behaviors. Furthermore, conventional MLP architectures are often inadequate for capturing intricate spatial–topological relationships among agents, obstacles, and targets, thereby limiting the generalization and reasoning capabilities of these methods in partially observable settings. In light of the distinctive challenges inherent in cooperative search tasks, we propose an improved MADRL algorithm designed to enhance decision-making effectiveness and system adaptability in low-altitude partially observable environments for multi-UAV cooperative search.

3. System Model and Problem Formulation

In this section, we describe the background of the research problem and the kinematic model of the UAVs, which serves as the control model for this paper, followed by an emphasis on the core challenge: the limited circular field-of-view constraint under low-altitude environments and cooperative control. On this basis, we provide a mathematical formulation of the task objectives and various constraints.

3.1. Scenario Description

Considering the altitude restrictions in urban low-altitude airspace, it is reasonable to assume that all UAVs operate at the same flight level. Accordingly, this study models the problem in a bounded two-dimensional urban airspace containing multiple static obstacles and $N$ homogeneous Search UAVs (hereinafter referred to as “searchers”), whose task is to cooperatively search for and surveil a moving evading vehicle (hereinafter referred to as the “evader”). Each searcher is equipped with a limited circular field of view—the evader’s position is perceived only when it enters the field of view of any searcher. Once the evader moves out of all searchers’ visual ranges, its position information is immediately lost. Furthermore, all UAVs are required to possess obstacle avoidance capabilities and must operate strictly within a predefined task area. Figure 1 illustrates the described scenario. In this setting, the evader’s velocity $v_e$ is greater than the velocity $v_i$ of the searchers, and its evasion strategy is non-deterministic, which collectively heightens the complexity of the search task. To simplify system modeling, the following assumptions are made:

(1) Communication among searchers is always reliable. Once the evader is detected by any searcher, its position is instantaneously shared with all other searchers, with no communication delay or localization error considered.

(2) Each searcher can accurately obtain its own state information (including position, velocity, and heading angle).

(3) The task is considered successful if the searchers continuously surveil the evader for a predefined duration $T_{\mathrm{s}}$.

(4) The influence of UAV attitude variation on the field of view is negligible during surveillance; therefore, the sensing area is simplified as a fixed circular region, ignoring any shape deformation.

3.2. UAV Kinematic and Circular Sensing Range

In the two-dimensional plane, the positions of the $i\text{-}\mathrm{th}$ searcher and the evader at time $t$ are denoted as ${\mathrm{p}}_i^t=(x_i^t,y_i^t)$ and ${\mathrm{p}}_{\mathrm{e}}^t=(x_{\mathrm{e}}^t,y_{\mathrm{e}}^t)$, respectively, where $i\in \{1,2,\dots ,N\}$, and $(x^t,y^t)$ represents the two-dimensional coordinates of the agent at time $t$. Meanwhile, the velocity and heading angle of the UAV at time $t$ are denoted as $v^t\mathrm{and} {\theta }^t$, respectively, with the specific kinematic relationship illustrated in Figure 2. The kinematic behavior of the UAV can thus be described by the following dynamic equations:

$$\left\{\begin{array}{c}{\dot{v}}^t={a_v}^t\\ {\dot{\theta }}^t={a_{\theta }}^t\\ {\dot{x}}^t=v^t\cos {\theta }^t\\ {\dot{y}}^t=v^t\sin {\theta }^t\end{array}\right.$$

(1)

where $a_{\mathrm{v}}^t$ and $a_{\theta }^t$ represent the acceleration and angular velocity of the UAV at time $t$, respectively, both of which are to be determined via MADRL. Due to the maneuvering constraints of the UAV, $a_{\mathrm{v}}^t$ and $a_{\theta }^t$ are subject to the following constraints:

$$-a_{v\max }<a_{\mathrm{v}}^t<a_{v\max }$$

(2)

$$-a_{\theta \max }<a_{\theta }^t<a_{\theta \max }$$

(3)

where $a_{v\max }$ and $a_{\theta \max }$ denote the maximum allowable linear acceleration and angular acceleration, respectively.

Each searcher is equipped with an onboard visual sensor, the perception range of which is modeled as a circular region with a fixed radius. All searchers are assumed to have identical sensing capabilities. Therefore, the field of view of a searcher is defined as follows:

$$V=\left\{p|{‖p-p_i^t‖}_2\le R\right\}$$

(4)

where $V$ denotes the field of view of a searcher, $p$ represents a coordinate within the UAV’s visual range, and $R$ is the radius of the sensing area. The visible region changes dynamically as the searcher moves, thereby enabling cooperative search of the target. This scenario is transformed into a two-dimensional layout as shown in Figure 3. To facilitate environmental modeling and reward mechanism design, the task area is discretized into a uniform grid, with records maintained for both explored and repeatedly explored regions.

3.3. Task Constraints

It is assumed that all searchers operate within a confined airspace denoted as $\mathfrak{R}$. The set of $m$ obstacles located above the flight altitude of the searchers is defined as $\mathit{B}$. Given that high-rise buildings dominate in realistic environments, all obstacles are modeled as cuboids, which project as rectangles in the top-view 2D plane, with their longer and shorter edges parallel to the $\mathrm{x}$ and $\mathrm{y}$ axes. For any obstacle $\mathrm{b}$, it is represented by the coordinates of its diagonal vertices ${\mathit{p}}_{\mathit{bi}}=[[x_{\mathrm{b}i},y_{\mathrm{b}i}],[x_{\mathrm{b}i}^{\prime },y_{\mathrm{b}i}^{\prime }]]$. Thus, $\mathit{B}=\{{\mathit{p}}_{\mathit{b}\mathbf{1}},{\mathit{p}}_{\mathit{b}\mathbf{2}},\dots ,{\mathit{p}}_{\mathit{b}\mathit{m}}\}$. The total area occupied by all obstacles is denoted as ${\mathfrak{R}}_{\mathrm{B}}$, which is a subset of $\mathfrak{R}$. All obstacles are fixed within the task area and impede UAV motion.

It is assumed that all searchers start from fixed initial positions, while the evader’s initial position is randomized, and none of them may be located within any obstacle region. The objective of the searchers is to detect the evader within the shortest possible time and maintain continuous surveillance of the evader for a specified duration, denoted as $T_{\mathrm{s}}$. Task success is defined by the following expressions:

$${\mathit{p}}_{\mathit{e}}^{\mathit{t}}\in V$$

(5)

$$T_{\mathrm{c}}=T_{\mathrm{t}}+T_s$$

(6)

where $T_{\mathrm{c}}$ represents the total task completion time, and $T_{\mathrm{t}}$ denotes the time elapsed before continuous observation is achieved.

Each searcher $i$ selects an appropriate action ${\mathit{u}}_{\mathit{i}}^{\mathit{t}}=[a_{\mathrm{v}}^t,a_{\theta }^t]$ based on its current local observation ${o_i}^t$ to search for the evader. Thus, the policy of searcher $i$ at time $t$ can be expressed as ${\pi }_i^t(u_i^t|o_i^t)$. We further define the joint policy of all searchers at time $t$ as ${\prod }_j^t$, such that ${\pi }_{\mathrm{j}}^t=[{\pi }_1^t,{\pi }_2^t,\dots {\pi }_N^t]\in {\prod }_j^t$. Ultimately, this cooperative search problem can be formulated as the following constrained optimization problem:

$$\arg \underset{{\mathsf{\Pi }}_j^t}{\min }\mathbb{E}\left[T_{\mathrm{c}}|{\Pi }_j^t={{\pi }_{\mathrm{j}}}^t,t=1,2,\dots T_{\mathrm{c}}\right]$$

(7)

$$\mathrm{s}.\mathrm{t}.\left|x_i^t-x_b\right|\ge l_b/2,\forall i\in N,\forall b\in B,\forall t\in \{1,2,\dots ,T_{\mathrm{c}}\}$$

(8)

$$\left|y_i^t-y_b\right|\ge w_b/2,\forall i\in N,\forall b\in B,\forall t\in \{1,2,\dots ,T_{\mathrm{c}}\}$$

(9)

$${‖p_i^t-p_j^t‖}_2\ge {\delta }_s,\forall i,j\in N,i\ne j,\forall t\in \{1,2,\dots ,T_{\mathrm{c}}\}$$

(10)

$$p_i^t=p_i^{t-1}+\Delta t{u}_i^{t-1},\forall i\in N,\forall t\in \{1,2,\dots ,T_{\mathrm{c}}\}$$

(11)

$$p_i^t\in \Omega ,i\in P,p_i^t\in \Omega ,\forall i\in N,\forall t\in \{1,2,\dots ,T_{\mathrm{c}}\}$$

(12)

$$0\le T_{\mathrm{c}}\le T_{\max }$$

(13)

where Equation (7) formulates the minimization of the total task time as the optimization objective for the searchers. In Equations (8) and (9), $l_b$ and $w_b$ denote the length and width of the obstacles, respectively, constraining the searchers to avoid collisions with all obstacles during flight. Equation (10) specifies the minimum safe distance that must be maintained among multiple agents to prevent collisions during cooperative operations. Equation (11) describes the state transition process in which each searcher, based on the kinematic model, executes an action ${\mathit{u}}_{\mathit{i}}^{\mathit{t}}$ at discrete time steps to update its position. Equation (12) restricts all agents to operate within the predefined task area $\mathfrak{R}$, ensuring no agent exceeds these bounds. Equation (13) defines the maximum allowable time steps $T_{\max }$; if the task is not completed within this limit, it is considered a failure.

4. Methodology

In this section, we elaborate on the strategy employed for multi-UAV cooperative search and formulate the corresponding POMDP. Building upon this formulation, we propose the NGASAC algorithm and provide a detailed description of its structural design and training procedure.

4.1. Introduction to Phased Search Strategy

Multi-UAV cooperative search for maneuvering targets presents several challenges. In real-world scenarios, the searching party often lacks prior information about the target, necessitating extensive cooperative exploration of the task area by multiple UAVs. Obstacle occlusion and the high maneuverability of the target further complicate sustained tracking. Even if a UAV detects the target, its typically lower speed compared to the target makes stable tracking difficult by a single agent; effective continuous surveillance requires collaboration among multiple UAVs.

Based on whether the target is within the field of view of any UAV and whether it has been detected previously, the search strategy is divided into three distinct operational modes: Cooperative Search, Tracking and Surveillance, and Focused Search.

(a) Cooperative Search Strategy: When the target has not been detected by any UAV, all searchers execute a cooperative search strategy aimed at maximizing area coverage and reducing uncertain regions.

(b) Tracking and Surveillance Strategy: Once any UAV detects the target, all searchers share this information and enter the tracking and surveillance state. In this phase, each UAV maneuvers to approach the target and maintain it within their collective field of view.

(c) Focused Search Strategy: If the target moves outside the field of view of all searchers, the system estimates a region of interest based on its last known position $p_{lost}=(x_{lost},y_{lost})$ and time $t_{lost}$. This region is a circular area centered at $p_{lost}$, with a radius that expands dynamically according to the maximum target speed $v_{e\max }$ and the time elapsed $(t-t_{lost})$. In this mode, multi-UAV concentrates its efforts on this suspected area for intensive search. If the target is re-detected, the strategy reverts to the tracking and surveillance mode; if the target is not found within a predetermined number of steps, the strategy switches back to the global cooperative search mode. The logic of the above “Cooperative Search—Tracking and Surveillance—Focused Search” strategy is illustrated in Figure 4. By dynamically responding to changes in target status, this approach enables adaptive allocation of search resources, thereby improving both search efficiency and task success rate.

4.2. POMDP

POMDP provides a formal framework for sequential decision-making in uncertain environments and is widely used in modeling multi-agent systems. It is defined by the tuple: $\mathrm{M}=\{S, U, P, r, O, N, \gamma \}$. which includes the state space $S$, action space $U$, state transition function $P$, reward function $r$, observation space $O$, number of agents $N$, and discount factor $\gamma$. Each agent is regarded as an independent entity that makes deterministic or stochastic decisions at each time step based on current information. The components are detailed below as applied to this problem.

4.2.1. State Space

A state encompasses the status of all searchers at time, expressed as:

$$s^t=[p_1^t,{\theta }_1^t,v_1^t,{\eta }_1^t\dots ,p_N^t,{\theta }_N^t,v_N^t,{\eta }_N^t]$$

(14)

where includes the coordinates, heading angle, velocity, and obstacle perception information of each agent. Obstacle information is acquired via onboard radar with a maximum detection range of. When approaching an obstacle, radar waves in that direction are partially blocked, reducing the effective return. Each searcher calculates the ratio of the actual returned wavelength to the maximum range as obstacle perception information:

$${\eta }_i^t={\displaystyle \frac{{\lambda }_i^t}{{\lambda }_{\max }}}$$

(15)

4.2.2. Action Space

This paper adopts a continuous action space. The action of a single agent at $t$ time is defined as: $u_i^t=[a_v^t,a_{\theta }^t]\in U$. This two-dimensional action represents the linear acceleration and angular acceleration of the agent. The joint action of all agents at $t$ time is defined as: $u_j^t=[u_1^t,u_2^t,\dots ,u_N^t]$, which is the output of the searchers’ policy network.

4.2.3. Local Observation Space

The local observation of a searcher at time $t$ is denoted as $o_i^t\in O_j^t$, where $O_j^t$ represents the joint observation space. Due to limited sensing range, the environment is partially observable. The observation space of each searcher consists of the following three components:

(a) Self-Information: Includes the searcher’s own coordinates $p_i^t$, velocity $v_i^t$, heading angle ${\theta }_i^t$ and obstacle perception information ${{\eta }_i}^t$.

(b) Teammate Information: Includes relative coordinates $\Delta p_{ij}^t$, distance $d_{ij}^t$, relative heading angle ${\theta }_{ij}^t$, and relative bearing ${\rho }_{ij}^t$, where $j\in \{1,2,\dots ,N\}$ and $i\ne j$.

(c) Evader Information: The information regarding the evader varies depending on the search phase, as detailed below:

Cooperative Search Phase: The target is not within the field of view of any searcher, and its state information is unobservable.

Tracking and Surveillance Phase: The target remains within the field of view of at least one searcher. Thus, the evader information includes relative coordinates $\Delta p_{i\mathrm{e}}^t$, distance $d_{i\mathrm{e}}^t$, and relative bearing ${\rho }_{i\mathrm{e}}^t$. These values are computed based on the evader’s position ${\mathrm{p}}_{\mathrm{e}}^t=(x_{\mathrm{e}}^t,y_{\mathrm{e}}^t)$.

Focused Search Phase: The target was previously detected but is currently lost. Searchers navigate toward the last known position of the evader, denoted as $\beta$. The information includes relative coordinates $\Delta p_{i\beta }^t$, distance $d_{i\beta }^t$ and relative bearing ${\rho }_{i\beta }^t$, all computed from the last known position ${\mathrm{p}}_{\beta }^t=(x_{\beta }^t,y_{\beta }^t)$.

4.2.4. Reward Function

Given the inherently cooperative nature of the multi-agent search task, this paper adopts a team-based reward mechanism. Relying solely on sparse rewards upon target discovery would lead to training difficulties and slow policy convergence. Therefore, we design corresponding reward functions for different search phases to effectively guide the agents’ behaviors.

During the Cooperative Search Phase, the reward function contains no target-related information and is designed to encourage exploration of unknown areas. It is defined as follows:

$$r_{s1}^t=S_{\mathrm{n}}^t/C_1$$

(16)

where $r_{s1}^t$ is the reward for exploring new areas, $S_{\mathrm{n}}^t$ is the area newly discovered at time $t$, and $C_1$ is a constant. Equation (16) implies that the searcher receives a reward proportional to the area of newly explored regions.

Concurrently, to reduce resource waste caused by overlapping fields of view and to encourage dispersed search efforts, the following reward is defined:

$$r_{s2}^t=\left\{\begin{array}{c}{S}_c^t/C_2,S_c^t\ge S_c^{t-1}\\ -S_c^t/C_2,S_c^t<S_c^{t-1}\end{array}\right.$$

(17)

where $r_{s2}^t$ is the field-of-view reward, $S_c^t$ is the total area within the field of view at time $t$, and $C_2$ is a constant. Equation (17) rewards the agent when the total visible area increases and penalizes it otherwise.

Thus, the total reward function during the cooperative search phase is:

$$r_s^t=r_{s1}^t+r_{s2}^t$$

(18)

During the Tracking and Surveillance Phase, since the target has been detected, the searchers should approach the evader as closely as possible to maintain surveillance. Hence, the distance between the searchers and the target becomes a key guiding factor. The distance reward is designed as:

$$r_{\mathrm{p}1}^t=\left\{\begin{array}{l}{\displaystyle \sum _{i=1}^N}\left(C_3/d_{ie}^t\right),\hspace{1em}{d}_{ie}^t<d_{ie}^{t-1}\\ -{\displaystyle \sum _{i=1}^N}\left(C_3/d_{ie}^t\right),\hspace{1em}{d}_{ie}^t\ge d_{ie}^{t-1}\end{array}\right.$$

(19)

where $r_{p1}^t$ is the distance reward, $d_{ie}^t$ is the distance between searcher $i$ and the evader at time $t$, and $C_3$ is a constant. Equation (19) encourages the searchers to continuously reduce their distance to the target.

Additionally, to promote more effective movement toward the target, a directional reward term is introduced, based on the consistency between the agent’s current heading angle ${\theta }_i^t$ and the relative bearing angle ${\rho }_{i\mathrm{e}}^t$:

$$r_{\mathrm{p}2}^t=\sum _{i=1}^N{C}_4\cos \left({\theta }_i^t-{\rho }_{i\mathrm{e}}^t\right)$$

(20)

where $r_{\mathrm{p}2}^t$ is the directional reward and $C_4$ is a constant. Equation (20) is maximized when the agent is heading directly toward the target.

Therefore, the total reward function during the Tracking and Surveillance Phase is defined as:

$$r_{\mathrm{p}}^t=r_{\mathrm{p}1}^t+r_{\mathrm{p}2}^t$$

(21)

During the Focused Search Phase, searchers are required to quickly reach the last known position of the target, denoted as $\beta$. The reward design in this phase is similar to that of the Tracking and Surveillance Phase. The distance reward is defined as:

$$r_{\mathrm{c}1}^t=\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^N}\left(C_4/d_{i\beta }^t\right),\hspace{1em}{d}_{i\beta }^t\ge d_{i\beta }^{t-1}\\ -{\displaystyle \sum _{i=1}^N}\left(C_4/d_{i\beta }^t\right),\hspace{1em}{d}_{i\beta }^t<d_{i\beta }^{t-1}\end{array}\right.$$

(22)

where $r_{\mathrm{c}1}^t$ denotes the distance reward, $C_4$ is a constant, and $d_{i\beta }^t$ represents the distance between searcher $i$ and the last known target position $\beta$ at time $t$.

Additionally, to encourage intensive search within the suspected region, an area coverage reward is introduced:

$$r_{\mathrm{c}2}^t=\left\{\begin{array}{c}{C}_5\times S_{\mathrm{s}}^t/[\pi \times {(t_{lost}\times v_{e\max })}^2],S_{\mathrm{s}}^t\ge S_{\mathrm{s}}^{t-1}\\ -C_5\times S_{\mathrm{s}}^t/[\pi \times {(t_{lost}\times v_{e\max })}^2],S_{\mathrm{s}}^t<S_{\mathrm{s}}^{t-1}\end{array}\right.$$

(23)

where $r_{\mathrm{c}2}^t$ is the reward for covering the suspected region, $C_5$ is a constant, $S_{\mathrm{s}}^t$ is the area within the suspected region that has been covered by the searchers up to time $t$, $t_{lost}$ is the time elapsed since the target was lost, and $v_{e\max }$ is the maximum speed of the evader. Equation (23) motivates the agents to expand the searched area within the region where the target is likely to be located.

Thus, the total reward function during the Focused Search Phase is:

$$r_{\mathrm{c}}^t=r_{\mathrm{c}1}^t+r_{\mathrm{c}2}^t$$

(24)

Another critical consideration is that searchers must avoid collisions with other agents and obstacles in the dynamic environment in real time. To this end, a collision warning penalty term is introduced:

$$r_w^t=\sum _{i=1}^N\min (C_6\times ({\lambda }_i^t-{\delta }_w),0)$$

(25)

where $r_w^t$ denotes the collision warning penalty, $C_6$ is a constant, ${\lambda }_i^t$ represents the length of the radar return at time $t$, and ${\delta }_w$ is the safety distance. Equation (25) takes effect when ${\lambda }_i^t<{\delta }_w$, with the penalty increasing as the agent approaches the obstacle, thereby encouraging the maintenance of a safe distance.

Thus, the total reward for all searchers is the sum of the phase-specific rewards and the collision warning penalty. To effectively guide the behavior of the agents, a high positive reward $r_{success}$ is provided upon successful completion of the task, while a large penalty $r_{fall}$ is imposed in cases of failure such as collisions, boundary violations, or timeout. This design incentivizes the agents to complete the search task efficiently while ensuring safety.

4.3. NGASAC

The NGASAC algorithm is built upon the Soft Actor-Critic (SAC) framework and incorporates several key enhancements tailored for multi-agent partially observable environments. First, we integrate NL with the Actor network to enhance the expressiveness of the policy and improve the flexibility of agent actions. Second, a MHGAT is incorporated into the Critic network to better capture spatial relationships among key entities in the environment, thereby accelerating convergence and improving final performance. Additionally, batch normalization (BN) and learning rate decay are employed to standardize input distributions, accelerate training, and enhance convergence stability. The algorithm adopts a CTDE architecture, and its core components are described below.

4.3.1. Normalizing Flow-Based Actor Network

In this work, the Actor network serves as the policy network for each agent, responsible for outputting a probability distribution over actions and generating continuous actions through sampling. Compared to networks that output deterministic actions, probabilistic policies enhance exploratory behavior. However, conventional policy networks typically output simple Gaussian distributions. Due to the unimodal nature of Gaussian distributions, it is difficult to accurately approximate complex optimal action distributions, thereby limiting the expressiveness of the policy. To overcome this limitation, we introduce NL, which enables the Actor network to learn more complex, multimodal action distributions while maintaining re-parameterizable sampling and differentiability. NL use a series of invertible transformations to map a base Gaussian distribution to a more complex distribution, achieving a “flow” of probability density via the change of variables theorem, thereby producing more expressive distributional forms.

The structure of the normalizing flow-based Actor network is illustrated in Figure 5. The input is the local observation $o_i$ of each agent. This is passed through a MLP to extract high-dimensional features, followed by two fully connected layers ${\mathrm{FC}}_i^u$ and ${\mathrm{FC}}_i^{\sigma }$ that output the mean $u_i$ and log standard deviation $\log {\sigma }_i$ of the Gaussian distribution, respectively. The activation function used in the MLP is the rectified linear unit (ReLU).

The above process can be expressed by Equation (26):

$$\left\{\begin{array}{c}{u}_i=F{C}_i^u(ML{P}_i(o_i))\\ \log {\sigma }_i=F{C}_i^{\sigma }(ML{P}_i(o_i))\end{array}\right.$$

(26)

Given $u_i$ and $\log {\sigma }_i$, a Gaussian distribution is obtained: $u_{i,0}\sim p_{i,0}=\mathcal{N}(u_i,{\sigma }_i^2)$, where $u_{i,0}$ is an action sampled from this initial Gaussian distribution. Then $u_{i,0}$ is fed into the first flow transformation layer, generating a new action $u_{i,1}=f_1(u_{i,0})$, which can be regarded as sampled from a new probability distribution $p_{i,1}$. This distribution $p_{i,1}$ is derived from the original distribution $p_{i,0}$ through an invertible transformation. We employ a transformation known as Planar Flow [22], which offers the advantage of low computational cost:

$$f\left(u\right)=u+{\mathcal{M}}_{z}G\left({\mathcal{M}}_w^{T}u+{\mathcal{M}}_b\right)$$

(27)

where ${\mathcal{M}}_z,{\mathcal{M}}_w\in {\mathbb{R}}^{D_u}$, ${\mathcal{M}}_b\in \mathbb{R}$ are learnable parameters, $D_u$ is the dimensionality of action $u$, and $G(·)$ is a differentiable smooth non-linear activation function; in this paper, the Tanh function is used. For the above transformation, the Jacobian determinant is computed as follows:

$$\left\{\begin{array}{c}\epsilon (u)=G^{\prime }\left({\mathcal{M}}_w^{T}u+{\mathcal{M}}_b\right){\mathcal{M}}_w\\ \det \left|{\displaystyle \frac{\partial f}{\partial u}}\right|=\left|\det \left(I+u\epsilon {(u)}^T\right)\right|=\left|1+u^T\epsilon (u)\right|\end{array}\right.$$

(28)

Subsequently, $u_{i,1}$ serves as the input to the next flow layer. Following the same procedure through $F$ flow layers, the final action $u_{i,end}$ is obtained, which can be expressed as:

$$u_{i,\mathrm{end}}=f_F\circ f_{F-1}\circ \dots \circ f_2\circ f_1(u_{i,0})$$

(29)

The parameters ${\mathcal{M}}_z,{\mathcal{M}}_w,{\mathcal{M}}_b$ of each flow layer are distinct. These parameters are generated by three fully connected layers $F{C}_i^Z,F{C}_i^W,F{C}_i^b$, which take as input the observation $o_i$ processed through $ML{P}_i$. This can be expressed as follows:

$${[{\mathcal{M}}_{z,1},{\mathcal{M}}_{z,2},\dots ,{\mathcal{M}}_{z,F}]}_i=F{C}_i^z\left(ML{P}_i(o_i)\right)$$

(30)

$${[{\mathcal{M}}_{w,1},{\mathcal{M}}_{w,2},\dots ,{\mathcal{M}}_{w,F}]}_i=F{C}_i^w\left(ML{P}_i(o_i)\right)$$

(31)

$${[{\mathcal{M}}_{b,1},{\mathcal{M}}_{b,2},\dots ,{\mathcal{M}}_{b,F}]}_i=F{C}_i^b\left(ML{P}_i(o_i)\right)$$

(32)

After $F$ invertible linear mappings, $u_{i,end}$ is sampled from a complex and potentially multimodal probability distribution. The fully connected layers $F{C}_i^Z,F{C}_i^W,F{C}_i^b$ are trained to learn flow transformation parameters adapted to the task requirements, enabling the agent to flexibly generate action policies suitable for complex environments.

4.3.2. Multi-Head Graph Attention Critic Network

In this study, the spatial topological relationships among searchers and between searchers and the environment are crucial for cooperative decision-making. For instance, during the cooperative search phase, UAVs must disperse to maximize coverage while maintaining a safe distance from obstacles. In the tracking and surveillance phase, they must rapidly approach the evader, with decisions heavily dependent on relative spatial relationships. To effectively capture such structural information, this paper introduces MHGAT, specifically constructing an Obstacle-oriented MHGAT (MHGAT-O) and a Target-oriented MHGAT (MHGAT-T) to model the spatial dependencies between searchers–obstacles and searchers–targets, respectively.

Specifically, during the cooperative search phase, we construct a spatial topology graph comprising each searcher and its $\mathcal{K}$ nearest obstacles. During the tracking and surveillance phase, a graph is built with all searchers and the evader as nodes. MHGAT-O and MHGAT-T process these different types of graph structures, and their input features also differ.

In MHGAT-O, the node set includes the searchers and their nearest obstacles. The input node features are denoted as ${\mathrm{H}}_O=[h{o}_1,h{o}_2,\dots ,h{o}_{N+N\mathcal{K}}]$. When $h{o}_i$ represents a searcher, it includes the searcher’s coordinates $p_i$ and heading angle ${\theta }_i$. When $h{o}_i$ represents an obstacle, it contains the coordinates of the obstacle’s center, with remaining dimensions padded by 0.

In the MHGAT-T, the node set includes all searchers and the evader. The input node features are denoted as ${\mathrm{H}}_T=[h{t}_1,h{t}_2,\dots ,h{t}_{N+1}]$, where each node feature has the same dimensionality. For example, $h{t}_i$ includes the current agent’s coordinates $p_i$, heading angle, velocity $v_i$, and obstacle information ${\eta }_i$.

The architecture of the Multi-Head Graph Attention Critic network is illustrated in Figure 6. During the graph attention (GAT) operation, let the input feature of a node be $h\in {\mathbb{R}}^{D_h}$, and the output feature be $h^{\prime }\in {\mathbb{R}}^{D_{h\prime }}$. The GAT employs a learnable linear transformation weight matrix $W\in {\mathbb{R}}^{D_{h^{\prime }}\times D_h}$. Node features entering the network are transformed by $W$ into higher-dimensional features, which are then concatenated. The attention coefficients are computed using a weight vector $\mathbf{a}$ and the LeakyReLU activation function. Taking nodes $i$ and $j$ as an example, the attention coefficient $e_{ij}$ is calculated as follows:

$$e_{ij}=LeakyReLU\left(a\left[W{h}_i\parallel W{h}_j\right]\right)$$

(33)

where $\mathbf{a}\in {\mathbb{R}}^{2D_{h^{\prime }}}$ is a trainable vector, and $\parallel$ denotes the concatenation operation. The attention coefficient $e_{ij}$ indicates the importance of node $j^{\prime }s$ features to node $i$.

To facilitate comparison of the importance of all neighboring nodes to node $i$, the attention coefficients are normalized using the $softmax$ function. For nodes $i$ and $j$, the normalized attention coefficient $A_{ij}$ is calculated as follows:

$$A_{ij}=softmax\left(e_{ij}\right)={\displaystyle \frac{\exp \left(e_{ij}\right)}{{\displaystyle \sum _{k\in {\mathcal{A}}_i}}\exp \left(e_{ik}\right)}}$$

(34)

where ${\mathcal{A}}_i$ denotes the set of all neighboring nodes of node $i$. After obtaining the normalized attention coefficients, a linear combination of the corresponding features is computed to form the output feature for each node. To capture diverse features and enhance spatial perception, a multi-head graph attention mechanism is employed. Each graph attention layer uses a distinct head parameter $\mathbf{a}$, and the outputs of multiple attention heads are concatenated to produce the final output of the multi-head graph attention layer:

$$h_i^{\prime }=|{|}_{m=1}^{\mathcal{M}}{\displaystyle \sum _{j\in {\mathcal{A}}_i}{A}_{ij}^m{W}^m{h}_j}$$

(35)

where $||$ denotes the concatenation operation, $A_{ij}^m$ represents the normalized attention coefficient between nodes $j$ and $i$ computed by the $m-th$ attention head, and $W^m$ is the learnable weight matrix of the $m-th$ head. Thus, the output node features of MHGAT-O are denoted as $H_O^{\prime }=[h{o}_1^{\prime },h{o}_2^{\prime },\dots h{o}_{N+N\mathcal{K}}^{\prime }]$, and those of MHGAT-T as $H_T^{\prime }=[h{t}_1^{\prime },h{t}_2^{\prime },\dots h{t}_{N+1}^{\prime }]$. Subsequently, $H_O^{\prime }$ and $H_T^{\prime }$ are concatenated and passed through a $Dropout$ (layer, which randomly sets a small proportion of values to zero to prevent overfitting. An ELU activation function is applied to obtain the final feature representation $H^{\ast }$:

$$H^{\ast }=ELU(Dropout({H_0}^{\prime }\parallel {H_T}^{\prime }))$$

(36)

The multi-head graph attention Critic network evaluates the decisions of each agent based on the joint observations and joint actions of all agents. Therefore, the concatenated vector of the joint observation $O_j^t$ and joint action $u_j^t$ is first processed by ${\mathrm{MLP}}_{q1}$-a two-layer fully connected network with ReLU activation—to produce an output feature vector $ou$. This process can be expressed as:

$$ou=ML{P}_{q1}\left(O_j^t\parallel u_j^t\right)$$

(37)

Subsequently, the concatenated vector of $ou$ and $H^{\ast }$ is fed into ${\mathrm{MLP}}_{q2}$, which consists of two fully connected layers with ReLU activation. The final output is the action $q$-value. This process can be expressed as:

$$q=ML{P}_{q2}\left(ou\parallel H^{\ast }\right)$$

(38)

The multi-head graph attention Critic network relies not only on the joint observations and actions of all agents but also emphasizes the graph-structured spatial features among agents. This enhances the Critic network’s ability to accurately assess the current state, provide more reasonable evaluations, and guide all agents toward more effective actions.

4.3.3. Training Algorithm

The NGASAC algorithm is built upon the maximum entropy reinforcement learning framework. Compared to traditional reinforcement learning methods, it encourages the policy network to maintain a certain level of exploration during training. The ultimate goal is for each agent to learn a policy ${\pi }_i^t$ such that the joint policy ${\prod }_j^t$ maximizes the sum of expected rewards and entropy, formally defined as:

$$r{\prod }_j^{t\ast }=\underset{{\prod }_j^t}{\arg \max }{\mathbb{E}}_{({\mathrm{s}}^t,{\mathrm{u}}_j^t)\sim {\mathcal{T}}_{{\prod }_j^t}}[{\displaystyle \sum _{t=0}^{+\infty }{\gamma }^t}(r(S^t={\mathbf{s}}^t,U^t={\mathbf{u}}^t)+\alpha \mathcal{H}({\prod }_j^t(\cdot |S^t={\mathbf{s}}^t))]$$

(39)

$$\mathcal{H}({\prod }_j^t(\cdot |S^t={\mathbf{s}}^t)\stackrel{\Delta }{=}{\mathbb{E}}_{{\mathbf{u}}_j^t\sim {\prod }_j^t}[-\log {\prod }_j^t(U^t={\mathbf{u}}_j^t|S^t={\mathbf{s}}^t)]$$

(40)

where ${\mathcal{T}}_{{\prod }_j^t}$ denotes the trajectory distribution under the joint policy ${\prod }_j^t$, $\mathcal{H}({\prod }_j^t(\cdot |S^t={\mathbf{s}}^t))$ represents the entropy of the joint policy ${\prod }_j^t$ at state ${\mathbf{s}}^t$, and $\alpha$ is a temperature coefficient that balances the importance of reward and entropy.

In this framework, the state-value function is defined as:

$$V\left(S^t=s^t\right)={\mathbb{E}}_{u_j^t\sim {\pi }_j^t}[Q(S^t=s^t,U^t=u_j^t)-\alpha \log {\pi }_j^t(U^t=u_j^t|S^t=s^t)]$$

(41)

and the soft Q-function is defined as:

$$Q\left(S^t=s^t,U^t=u_j^t\right)=r\left(S^t=s^t,U^t=u_j^t\right)+\gamma {\mathbb{E}}_{s^{t+1}\sim \mathcal{T}}\left[V\left(S^{t+1}=s^{t+1}\right)\right]$$

(42)

The soft Q-network, parameterized by $\omega$, serves as the Critic network in this algorithm. It is trained by minimizing the following Bellman error:

$$J_Q(\omega )={\mathbb{E}}_{(s^t,u_j^t)\sim \mathcal{D}}[{(Q_{\omega }(S^t=s^t,U^t=u_j^t)-r(S^t=s^t,U^t=u_j^t)-\gamma {\mathbb{E}}_{s^{t+1}\sim \mathcal{T}}[V_{\overline{\omega }}(S^{t+1}=s^{t+1})])}^2$$

(43)

where $\mathcal{D}$ denotes the experience replay buffer, and $\overline{\omega }$ represents the parameters of the target Q-network. The use of a target network helps mitigate overestimation issues. At the beginning of training, the target Q-network is a mirror of the original soft Q-network, sharing the same architecture and parameters. The target Q-network is updated according to the following rule:

$$\overline{\omega }\leftarrow \tau \times \omega +(1-\tau )\times \overline{\omega }$$

(44)

where $\tau$ is the mixing coefficient. The update of $\overline{\omega }$ follows a soft update strategy. Equation (44) indicates that at each step, the target network parameters $\overline{\omega }$ are updated by blending the parameters of the soft Q-network and the current target parameters. The policy of each agent is determined by the policy network parameters $\varphi$, which are optimized by minimizing the following loss:

$$J_{\pi }(\varphi )={\mathbb{E}}_{(s^t,o^t)\sim \mathcal{D}}[{\mathbb{E}}_{u^t\sim {\pi }_{\varphi }}[\alpha \log {\pi }_{\varphi }(U^t=u^t|O^t=o^t)-Q_{\theta }(S^t=s^t,U^t=u_j^t)]$$

(45)

The temperature coefficient $\alpha$ is automatically adjusted during training by minimizing the following loss:

$$J(\alpha )={\mathbb{E}}_{s^{\prime }\sim \mathcal{D}}\left[-\alpha \log {\pi }_j^t\left(U^t=u_j^t|S^t=s^t\right)-\alpha \overline{\mathcal{H}}\right]$$

(46)

where $\overline{\mathcal{H}}$ is a predefined target entropy.

Thus, the Critic network, Actor network, and temperature parameter $\alpha$ are optimized at each training step according to Equations (43), (45) and (46), with learning rates $l{r}_Q$, $l{r}_{\pi }$ and $l{r}_{\alpha }$, respectively. The pseudo-code of the NGASAC algorithm is provided in Algorithm 1.

Algorithm 1: NGASAC Algorithm

Input: Search space and initial positions of all agents Output: Policy networks for all agents 1: Initialize policy networks ${\pi }_{{\varphi }_1},{\pi }_{{\varphi }_2},\dots ,{\pi }_{{\varphi }_N}$, $Q$-value networks $Q_{{\omega }_1},Q_{{\omega }_2}$, and target Q-value networks $Q_{{\overline{\omega }}_1},Q_{{\overline{\omega }}_2}$, with ${\overline{\omega }}_1\leftarrow {\omega }_1,{\overline{\omega }}_2\leftarrow {\omega }_2$ 2: Initialize replay buffer $\mathcal{D}$ with capacity $\mathcal{Z}:\mathcal{D}\leftarrow \varnothing$ 3: for episode = 1: E do 4:   for t = 1: $T_{\max }$ do 5:  Obtain global state $s^t$ and joint observation ${\mathbf{o}}_j^t$ 6:  for i = 1: N do 7:   Extract individual observation ${\mathbf{o}}_i^t$ from ${\mathbf{o}}_j^t$ 8:   Select action ${\mathbf{u}}_i^t$ according to policy network ${\pi }_{\varphi i}(\cdot |{\mathbf{o}}_i^t)$ 9:   end for 10:   Execute joint action ${\mathbf{u}}_j^t$ 11:   Receive reward $r^t$,next global state $s^{t+1}$, and next joint observation ${\mathbf{o}}_j^{t+1}$ 12:   Store transition $({\mathbf{o}}_j^t,s^t,{\mathbf{u}}_j^t,r^t,o_j^{t+1},s^{t+1})$ in $\mathcal{D}$ 13:   Sample a random batch from $\mathcal{D}$ 14:   Update Q-value networks by minimizing loss in Equation (43): ${\omega }_k\leftarrow {\omega }_k-l{r}_Q{\stackrel{⌢}{\nabla }}_{{\omega }_k}{J}_Q({\omega }_k)$, for $k\in \left\{1,2\right\}$ 15:   Update each policy network by minimizing loss in Equation (45): ${\phi }_i\leftarrow {\phi }_i-l{r}_{\pi }{\stackrel{⌢}{\nabla }}_{{\phi }_i}{J}_{\pi }({\phi }_i)$, for $i\in \left\{1,\dots ,N\right\}$ 16:   Update temperature coefficient by minimizing loss in Equation (46): $\alpha \leftarrow \alpha -l{r}_{\alpha }{\stackrel{⌢}{\nabla }}_{\alpha }J(\alpha )$ 17:   Update target networks via soft update (Equation (44)) 18: end for 19: end for

4.3.4. Algorithm Complexity Analysis

Actor Network: Let the input dimension of the Actor network be $M_{in}$, the output dimension be 2, the hidden layer dimension be $M_h$, and the number of hidden layers be $L$. The computational complexity of the Actor network is: $O(M_{in}{M}_h+L{M_h}^2+2M_h)$. Additionally, the normalizing flow transformation introduces extra computation. Three separate single-layer networks are used to generate the flow parameters, with an input dimension of $M_h$, number of flows $F$, and UAV action dimension $M_a$. The output layer dimensions are $M_{a}F,M_{a}F$ and $F$ respectively. Thus, the computational complexity for generating the flow parameters is: $O(2M_h{M}_{a}F+M_{h}F)$. The final action is computed via matrix multiplication, with a complexity of: $O(2M_{a}F)$. Ignoring constant and lower-order terms, the total computational complexity of the normalizing flow-based Actor network is: $O_{actor}(M_{in}{M}_h+L{M_h}^2+M_h{M}_{a}F)$.

Critic Network: Assume the input dimension of the Critic’s ${\mathrm{MLP}}_{q1}$ is $M_{ia}$, the hidden layer dimension is $M_h$, and the output dimension is $M_{q1}$. The computational complexity of ${\mathrm{MLP}}_{q1}$ is $O(M_{ia}{M}_h+M_h{M}_{q1})$. Suppose the GAT network has $\mathcal{J}$ nodes, input node feature dimension $M_{ni}$, and output node feature dimension $M_{no}$, Let the number of edges in the graph be $\mathit{\epsilon }$. The complexity for node feature mapping is: $O(\mathcal{J}{M}_{ni}{M}_{no})$, and the complexity for computing attention coefficients via edge mapping is: $O(2\mathit{\epsilon }{M}_{no})$. Combining both, the computational complexity for a single-head GAT is: $O(\mathcal{J}{M}_{ni}{M}_{no}+2\mathit{\epsilon }{M}_{no})$. Ignoring differences in input dimensions between GAT-O and GAT-T and omitting constants, the complexity for a $K$-head MHGAT is: $O(K{M}_{no}(\mathcal{J}{M}_{ni}+\mathit{\epsilon }))$. The outputs of the two parts are concatenated and used as input to ${\mathrm{MLP}}_{q2}$, which has an input dimension of $M_{q1}+K\mathcal{J}{M}_{no}$, an output dimension of 1, a hidden layer dimension of $M_h$, and $L$ hidden layers. The computational complexity of ${\mathrm{MLP}}_{q2}$ is: $O((M_{q1}+K\mathcal{J}{M}_{no})M_h+L{M_h}^2+M_h)$. Neglecting constants and lower-order terms, the overall computational complexity of the multi-head graph attention Critic network is: $O_{critic}(M_h(M_{ia}+L{M}_h+M_{q1})+K{M}_{no}(\mathcal{J}(M_{ni}+M_h)+\mathit{\epsilon }))$.

Distributed Execution: During distributed execution of the algorithm, only the Actor network is used for computation. Let the total number of iterations be $T$ and the number of UAVs be $N$. The total computational complexity in the distributed execution phase is: $O_{de}=TN{O}_{actor}$.

Centralized Training: During the centralized training phase, both the Actor and Critic networks are utilized for computation. Let the total number of iterations be $T$, the number of UAVs be $N$, and the batch size be $D$. Due to the use of twin Q-networks, each training step involves $2N$ forward passes through the Actor network and $4+2N$ through the Critic network. Ignoring constant and lower-order terms, the simplified computational complexity is: $O_{ct}=TDN(O_{actor}+O_{critic})$.

5. Simulations

In this section, we conduct simulations to validate the effectiveness of the proposed NGASAC algorithm for multi-UAV cooperative search in partially observable low-altitude environments. To thoroughly evaluate its performance, we compare it against three classical MADRL algorithms using multiple representative metrics. Furthermore, we examine the generalization capability of NGASAC across different environmental settings and perform ablation studies to analyze the contribution of each key module.

5.1. Simulation Environment Setup

(a) Airspace Environment: We simulate a low-altitude urban environment, as illustrated in Figure 7. the flight space is confined to a region of [0,1000] m × [0,1000] m. The environment contains 10 square obstacles of identical size (70 m side length), with edges aligned parallel to the x- and y-axes.

(b) UAV Configuration: The system comprises three search UAVs. The initial positions of the searchers are fixed at [80,80], [80,100] and [100,80], while the initial position of the evader is randomly generated within the flight space outside obstacle areas. UAV parameters are set with reference to existing literature [38,39,40]; specific values are listed in

Table 1. Agent Configuration Parameters.

Name	Symbol	Value	Unit
Angular Velocity	$a_{\theta }^t$	$-\pi /3–\pi /3$	rad/s
Evader Velocity	$v_e$	12	m/s
Searcher Velocity	$v_i$	9–11	m/s
Searcher Acceleration	$a_{\mathrm{v}}^t$	−1–1	m/s2
Searcher Sensing Radius	$\mathrm{R}$	200	m
Minimum Safe Distance	${\delta }_s$	1	m
Maximum Radar Range	${\lambda }_{\max }$	100	m
Obstacle Warning Distance	${\delta }_w$	25	m
Required Monitoring Duration	$T_{\mathrm{s}}$	10	s

. The maximum task time is set to 140 s. The task terminates early if a searcher exits the flight space, collides with an obstacle or another agent, or successfully detects the evader.

5.2. Evasion Strategies

In the simulations, we design two distinct evasion strategies for the evader, as follows:

Random Evasion Strategy: The evader does not perceive the searchers’ information and moves randomly within the environment at maximum speed.

Evasive Escape Strategy: When the evader is within the field of view of any searcher, it obtains the searchers’ positions and actively moves in the direction opposite to the nearest searcher to maximize its escape probability.

In both strategies, the evader is equipped with obstacle avoidance capability. When the distance to an obstacle is less than 25 m, the evader proactively maneuvers to avoid collision. The same rule is applied to prevent it from escaping the task airspace boundaries.

5.3. Comparative Simulations

The hyperparameters of the NGASAC algorithm and the reward function are summarized in

Table 2. NGASAC Hyperparameter Settings.

Parameter	Value	Parameter	Value
Training steps	$5\times {10}^5$	Replay buffer size	$5\times {10}^5$
Batch size	256	Soft update coefficient	0.01
Discount factor	0.95	Dropout rate	0.01
Optimizer	Adam	LeakyReLU parameter	0.2
Number of attention heads	2 and 2	Constant $C_1$	$1\times {10}^4$
Number of flow layers	4	Constant $C_2$	$1.5\times {10}^5$
Hidden layer dimension	64	Constant $C_3$	100
Actor network learning rate	$1\times {10}^{-4}$	Constant $C_4$	0.25
Critic network learning rate	$1\times {10}^{-3}$	Constant $C_5$	1.2
Entropy learning rate	$1\times {10}^{-3}$	Constant $C_6$	0.02
Success reward $r_{success}$	10	Failure penalty $r_{fall}$	−10

. The compared algorithms include MASAC, MAPPO, and MADDPG. All simulations were conducted on a platform equipped with an i7-14700K processor, an RTX-4070Ti GPU, and 32 GB of RAM. During training, when the number of experiences in the replay buffer is less than the batch size, the searchers execute random actions. Experiences collected during this random exploration phase are retained. Once the buffer reaches the batch size, the model begins updating. Under both evasion strategies, the algorithm is evaluated every $1\times {10}^4$ training steps. Each evaluation consists of 50 episodes. In addition to cumulative reward, five evaluation metrics are defined based on the problem characteristics to compare the performance of different algorithms:

Success Rate: The proportion of tasks in which the searchers successfully maintain continuous surveillance of the target. A higher value indicates better policy performance.

Task Time: The average time required to complete a task. If a task fails (due to collision or exiting the airspace), the time for that episode is recorded as the maximum task time (140 s). A shorter task time reflects higher algorithmic efficiency.

Collision Departure Rate: The proportion of tasks in which a searcher collides or exits the airspace. A higher value indicates poorer safety and stability of the algorithm.

Search Time: The average time from the start of the task until the evader is first detected. A lower value indicates stronger cooperative search capability of the UAV group.

Disappearance Time: The average time required to re-detect the evader after it leaves the field of view of all searchers. If the target is lost multiple times, the durations are accumulated. This metric reflects the algorithm’s focused search capability after target loss. A shorter disappearance time indicates a more effective search strategy.

To further analyze the training dynamics of the algorithms, we tracked the changes in reward and success rate during training for all four algorithms, as shown in Figure 8 and Figure 9.

As observed from the reward curves (Figure 8), NGASAC consistently achieves higher reward values across different evaluation steps, with an overall trend of steady increase as training progresses, indicating strong reward optimization capability and policy stability. In contrast, the reward growth of MADDPG and MAPPO is relatively slow or exhibits greater volatility, suggesting lower learning efficiency or stability in this complex environment. As for MASAC, while it demonstrates better initial learning dynamics than MADDPG and MAPPO, its final reward and convergence stability are still surpassed by our proposed NGASAC. From the success rate curves (Figure 9), it can be seen that NGASAC generally attains a higher success rate compared to the other algorithms and remains at a high level after stabilization, demonstrating excellent performance in task completion. Although MASAC shows some improvement in the early stages, its final success rate remains lower than that of NGASAC. The success rates of MAPPO and MADDPG improve only marginally, indicating limitations in policy exploration or generalization. By integrating observations from both figures, it can be concluded that NGASAC significantly outperforms the comparison algorithm in both reward acquisition and task success rate, demonstrating superior optimization performance and convergence stability. Through the NF and MHGAT modules, it reduces low-quality experiences to achieve more effective exploration and learning, enhances the agent’s understanding of spatial relationships, better realizes relationship generalization, and allows the agent to make more efficient and complex actions, thereby breaking away from simple strategies and being applicable to multi-UAV cooperative search tasks in low-altitude partially observable environments.

The trained policy network was evaluated over 1000 episodes. The performance metrics are summarized in

Table 3. Simulation Results of Evaluation Metrics.

Evasion Strategy	Random Evasion				Evasive Escape
Algorithm	NGASAC	MASAC	MAPPO	MADDPG	NGASAC	MASAC	MAPPO	MADDPG
Reward	50.38 ± 2.30	47.80 ± 4.14	47.22 ± 3.82	47.69 ± 4.35	52.03 ± 4.93	48.20 ± 6.88	47.90 ± 4.06	48.04 ± 5.63
Success Rate (%)	84.80	81.50	82.00	79.10	77.40	71.60	69.50	69.30
Task Time (s)	70.39 ± 6.30	75.91 ± 5.74	79.04 ± 5.05	83.38 ± 4.99	95.19 ± 8.49	109.74 ± 9.03	115.37 ± 8.65	121.29 ± 7.37
Collision Departure Rate (%)	1.90	4.20	3.10	5.30	6.00	10.50	9.90	12.60
Search Time (s)	49.04 ± 7.36	54.72 ± 6.70	55.31 ± 6.37	55.09 ± 7.60	50.92 ± 8.58	57.18 ± 7.99	67.18 ± 6.22	63.28 ± 7.83
Disappearance (s)	8.03 ± 2.36	7.50 ± 2.44	8.91 ± 2.09	10.12 ± 3.81	14.82 ± 5.39	15.05 ± 4.40	14.38 ± 5.28	16.60 ± 6.35

(best results highlighted in bold). For intuitive comparison, line charts of the reward, success rate, task time, and collision departure rate under both evasion strategies are shown in Figure 10. Under the random evasion strategy, NGASAC outperformed all baseline algorithms in reward, success rate, task time, collision departure rate, and search time. It achieved an average reward of 50.38, significantly higher (by at least 2 points) than other algorithms, with a small standard deviation indicating high stability. The success rate reached 84.8%, and the collision departure rate was only 1.90%, clearly superior to even the next best algorithm, MAPPO (success rate: 82.00%, collision departure rate: 3.10%). NGASAC also achieved the shortest task time (70.39 s) and search time (49.04 s), demonstrating its ability to locate and monitor the target more quickly. In terms of disappearance time, NGASAC (8.03 s) performed on par with MASAC (7.50 s), indicating comparable capability in focused search after target loss. Under the evasive escape strategy, NGASAC maintained a significant advantage. It achieved a reward of 52.03, substantially higher than other algorithms (around 48). Although overall performance declined due to the target’s evasive behavior, NGASAC still achieved the highest success rate (77.40%) and the lowest collision departure rate (6.00%). While its task time (95.19 s) and search time (50.92 s) increased compared to the random evasion scenario, they remained significantly better than those of other algorithms. The disappearance time for NGASAC (14.82 s) showed no significant difference from MAPPO (14.38 s), further confirming its strong focused search capability. This indirectly indicates that the NGASAC algorithm has learned not to rush blindly to the next location, but rather a more balanced and strategic approach. When the target is lost, the intelligent system will comprehensively consider more factors, such as maintaining a safe distance from obstacles and teammates, and strategically positioning itself to block the potential escape routes of the evader, rather than merely pursuing its last known location. This kind of intelligent and collaborative behavior, although it may take a little more time to execute, can lead to a more robust long-term monitoring effect and a significant reduction in collision rates. Based on the above analysis, NGASAC demonstrates excellent stability, search efficiency, and obstacle avoidance performance across different target strategies, validating its effectiveness and superiority for multi-UAV cooperative search tasks in partially observable low-altitude environments.

Rapid search and continuous surveillance of the evader represent the core challenges of this task. To more clearly compare the performance of each algorithm in terms of task time, search time, and disappearance time, the average metrics under both evasion strategies are shown in Figure 11. The results indicate that under both strategies, the NGASAC algorithm consistently achieves the shortest task time and search time, demonstrating its ability to efficiently locate the evader and complete the monitoring task. The MASAC algorithm performs second best, while MAPPO and MADDPG exhibit the longest task times. Under the random evasion strategy, NGASAC’s disappearance time is only 0.53 s longer than that of MASAC. Under the evasive strategy, its disappearance time is only 0.44 s longer than that of MAPPO. This indicates that NGASAC can effectively learn and execute a focused search strategy after target loss, promptly re-examining suspicious regions to reacquire the evader. In summary, the analysis demonstrates that the NGASAC algorithm effectively learns strategies such as cooperative search, tracking and surveillance, and focused search, and achieves flexible switching among these strategies, thereby accomplishing the search and surveilling mission with high efficiency.

Figure 12 presents the simulation results under two evasion strategies: (a) random evasion and (b) evasive escape. The searcher policies in both experimental groups were obtained after 5 × 10⁵ training steps.

Under the random evasion strategy, the evader starts at [600, 870], and its movement is independent of the searchers’ positions. The three searchers depart from their predefined initial positions. During the cooperative search phase, they spread out to expand perceptual coverage, reduce field-of-view overlap, and efficiently explore unknown regions. The trajectories indicate that the searchers possess robust obstacle avoidance capabilities and can navigate autonomously within the task area. Searcher 2 first detects the evader, and this positional information is shared with other agents via a communication mechanism. Subsequently, Searcher 1 and Searcher 2 adjust their courses toward the target and initiate tracking. The green trajectories indicate that the searchers enter a tracking and surveillance state, maintaining continuous surveillance of the target until the preset threshold $T_{\mathrm{s}}$ is reached, marking successful task completion. Under the evasive escape strategy, the evader starts at [700, 900] and can access the searchers’ positional information within its field of view, enabling active evasion. During the cooperative search phase, the searchers still adopt a dispersed strategy, effectively avoiding obstacles and covering the task area. In this experiment, green trajectories represent periods when the target is detected by the searchers, while yellow trajectories indicate paths taken during focused search operations after the target is lost. Searcher 3 first detects the evader but loses visual contact due to the target’s evasive maneuvering. All searchers then converge to search the suspicious region around the target’s last known position. Eventually, Searcher 2 successfully reacquires the target. Although the evader attempts to escape again, it fails to break surveillance within 10 s, and the task is completed successfully. It is important to note that the present study, as a foundational investigation, does not account for external disturbances such as wind or internal uncertainties like sensor noise and actuator dynamics. These factors, while critical for real-world application, are left for future investigation to first conclusively establish the baseline performance of the proposed framework in a controlled setting.

5.4. Generalization Simulations

To validate the generalization capability of the NGASAC algorithm, we constructed five types of simulation environments with distinct characteristics. In all environments, the evader employs the evasive escape strategy. The neural network model with the highest success rate during training was selected for generalization testing. The specific environmental configurations are as follows:

(1) Environments with varying initial searcher positions: In the original training, the searchers’ starting positions were fixed. To evaluate the impact of initial position distribution on algorithm performance, we designed a set of environments with different initial configuration layouts, as illustrated in Figure 13a.

(2) Environments with varying obstacle quantity and layout: The original environment has a fixed number and shape of obstacles. To investigate the impact of obstacle density and geometric layout on the adaptability of the algorithm, two new obstacle configurations were designed: one is a high-density obstacle environment, and the other contains rectangular obstacles of random sizes, as shown in Figure 13b.

(3) Environments with different numbers of searchers: To analyze the influence of the number of agents on the cooperative strategy, an additional searcher was introduced, forming an experimental environment with four searchers, as illustrated in Figure 13d.

(4) New task environments: To examine the generalization ability of the policy network under changes in task structure, a new task was designed: in this scenario, three static targets are deployed. A target is considered successfully detected once it is found by any searcher. The objective is to discover as many targets as possible, as shown in Figure 13e.

To comprehensively evaluate the generalization performance of the NGASAC algorithm, 100 repeated simulations are conducted for each new scenario. The evaluation metric results for Scenarios S0 to S5 are summarized in

Table 4. Generalization experiment results.

	S0	S1	S2	S3	S4	S5
Reward	52.03	53.85	48.37	51.82	54.9	N/A
Success Rate (%)	77.40	81	69	75	88	90
Task Time (s)	95.19	88.40	112.74	101.57	74.60	92.49
Collision Departure Rate (%)	6	7	15	11	9	8
Search Time (s)	50.92	49.37	57.06	53.91	41.49	N/A
Disappearance (s)	14.82	18.26	30.11	26.5	12.88	N/A

. Note that for S5, only three metrics were considered: success rate, task time, and collision departure rate. S0 represents the original baseline scenario.

From the results, the distribution of initial positions (S1) has no significant impact on policy performance; however, a more dispersed initial layout helps improve search efficiency, increasing the success rate to 81% and reducing the task time to 88.40 s, while the collision departure rate remains largely stable. Regarding obstacle variations, the dense obstacle environment (S2) increases search difficulty to some extent, leading to longer task times and a higher collision rate, though the algorithm still maintains relatively good overall performance. Environments with differently shaped obstacles (S3) show some mitigation in success rate compared to S2, reaching 75%, but task time increases to 101.57 s, indicating that NGASAC retains a certain adaptability to unfamiliar obstacle configurations. When the number of searchers increases to four (S4), cooperative efficiency improves significantly—the success rate rises to 88%, and task time decreases substantially to 74.6 s, demonstrating the algorithm’s ability to effectively utilize additional agents to enhance performance. However, the collision departure rate increases slightly, which may be related to the increased complexity of cooperative maneuvering. In the new task scenario (S5), the algorithm exhibits strong transfer capability, achieving a success rate as high as 90%, indicating that the trained policy possesses good generalization and task extensibility.

5.5. Ablation Studies

To further validate the effectiveness of key components in the NGASAC algorithm, we designed systematic ablation simulations, including the following three variants:

Ablation 1: Remove the NF; the policy network directly samples actions from a Gaussian distribution.

Ablation 2: Remove the MHGAT; the Critic network estimates action values using only an MLP to process joint observations and actions.

SAC Variant: Remove both NL and the multi-head attention mechanism, equivalent to the standard SAC algorithm.

Simulations were conducted in an environment where the evader adopts an evasive strategy. Each algorithm was evaluated based on four metrics: reward, success rate, task time, and collision departure rate.

The experimental results are shown in

Table 5. Ablation Experiment Results.

	NGASAC	Ablation 1	Ablation 2	MASAC
Reward	52.03	51.44	51.08	48.2
Success Rate (%)	77.4	74.9	74	71.6
Task Time (s)	95.19	104.62	100.93	109.74
Collision Departure Rate (%)	6	8.4	8.9	10.5

and Figure 14. NGASAC achieves the best performance across all four evaluation metrics. Comparing Ablated 1 and Ablated 2, it is evident that removing the MAGAT mechanism has a more significant impact on performance, indicating that this mechanism plays a critical role in capturing the evader by enhancing spatial coordination among agents. Specifically: Compared to Ablated 1, NGASAC—due to the incorporation of the NL layer—achieves a 0.59 increase in reward, a 2.5% improvement in success rate, a reduction of 9.43 s in task time, and a 2.4% decrease in collision departure rate. Compared to Ablated2, the introduction of the MHGAT mechanism leads to a 0.95 increase in reward, a 3.4% improvement in success rate, a 5.74 s reduction in task time, and a 2.9% decrease in collision departure rate. Compared to the MASAC algorithm, NGASAC—by integrating both modules—achieves a 3.83 increase in reward, a 5.8% improvement in success rate, a 14.55 s reduction in task time, and a 4.5% decrease in collision departure rate. In summary, both the NL layer and the MHGAT mechanism contribute significantly to the performance improvement of the algorithm. They operate in a complementary manner, collectively ensuring the outstanding performance of NGASAC in multi-agent cooperative search tasks.

6. Conclusions and Future Work

This paper addresses the problem of multi-UAV cooperative search from a top-down perspective in Partially Observable Low-Altitude Environments. This scenario is highly relevant to real-world applications such as UAVs equipped with visual sensors searching for unauthorized ground targets and has the potential to be extended to complex environments (e.g., cooperative monitoring of marine organisms by multiple underwater vehicles). To tackle this task, we propose NGASAC, a MADRL algorithm that incorporates NF and MHGAT mechanisms on the basis of the MASAC framework, enhancing both model expressiveness and cooperative decision-making performance. The NF layer overcomes the limited representational capacity of traditional Gaussian distributions, improving the flexibility of action sampling. The multi-head attention mechanism effectively captures spatial relationships and dependencies among agents, providing more accurate value estimation for joint actions and thereby facilitating policy optimization. Simulations demonstrate that NGASAC outperforms MASAC, MAPPO, and MADDPG across multiple evaluation metrics, including reward, success rate, task time, and collision departure rate. It also exhibits strong generalization capabilities in new tasks, large-scale agent systems, and complex environments. In future work, we plan to introduce more real-world constraints, such as communication limitations among multiple UAVs, localization errors, and individual agent failures. We will also further explore the adaptability of the algorithm to three-dimensional spatial search tasks. Additionally, the NGASAC algorithm can be extended to other complex multi-agent cooperative scenarios, such as UAV formation control and collaborative navigation in challenging environments. At the same time, an important future direction is to strictly incorporate the influence of disturbances, and to study robust reinforcement learning methods or adaptive control techniques in order to achieve online compensation for continuous disturbances.