Multi-UAV Cooperative Searching and Tracking for Moving Targets Based on Multi-Agent Reinforcement Learning

Abstract

In this paper, we propose a distributed multi-agent reinforcement learning (MARL) method to learn cooperative searching and tracking policies for multiple unmanned aerial vehicles (UAVs) with limited sensing range and communication ability. Firstly, we describe the system model for multi-UAV cooperative searching and tracking for moving targets and consider average observation rate and average exploration rate as the metrics. Moreover, we propose the information update and fusion mechanisms to enhance environment perception ability of the multi-UAV system. Then, the details of our method are demonstrated, including observation and action space representation, reward function design and training framework based on multi-agent proximal policy optimization (MAPPO). The simulation results have shown that our method has well convergence performance and outperforms other baseline algorithms in terms of average observation rate and average exploration rate.

1. Introduction

Recently, the technology of UAV has advanced remarkablely and the cooperation of multiple UAVs has also been a research hotspot [1,2]. By working closely together, multiple UAVs can demonstrate superior intelligence, coordination, and autonomy [3,4]. At the same time, cooperative target searching and tracking has also become a significant application field for UAVs [5]. The aim of multi-UAV system in searching mission is twofold: searching undetected targets and monitoring detected targets. But the cooperation for multiple UAVs is a very challenging problem. Firstly, during the searching and tracking process, the situation is continuously changing, such as the locations of the moving targets, communication among UAVs, etc. It is essential to achieve real-time path planning in an unpredictable and ever-changing environment. Secondly, as the UAVs’ quantity rises, the joint action for UAVs becomes increasingly computationally complex.

Traditional methods for addressing the issue of target searching and tracking can be categorized into two groups. The first type is planning theory, such as geometric approach [6] and region-based approach [7]. The primary focus of these methods lies in addressing the issue of zoning search or formation search, both of which are typically appropriate for uncomplicated settings. But the search path for UAVs need to be redesigned once environment changes. The second type is rooted in the optimization theory [8,9]. The goal of these methods is to optimize target searching and tracking, aiming for a maximum coverage rate or maximum observation rate. Nevertheless, the computational complexity escalates significantly as the problem scale expands. In addition, their reliance on centralized architecture restricts the utilization under severe conditions.

Reinforcement learning (RL) focuses on training agents to make intelligent decisions through their interactions with the environment [10]. It has found extensive application in a variety of industries, such as robotics [11] and game playing [12]. Furthermore, multi-agent reinforcement learning (MARL) has also devoloped repaidly recently as an extension of reinforcement learning. It focuses on learning in environments with multiple agents that interact with each other. In MARL, the agents can have different objectives and may either collaborate or compete with each other. It is anticipated that MARL will be a key approach in resolving the problem of multi-UAV collaboration.

This paper proposes a distributed MARL method to learn target searching and tracking policies of multi-UAV system. Firstly, we discribe the system model of multi-UAV cooperative searching and tracking for moving targets. Moreover, the information update and fusion mechanism is proposed to enhance environment perception ability of the UAVs. Then, the details of our method are demonstrated, including observation and action space representation, reward function design and training framework based on MAPPO. The main contributions of our paper can be summarized as follows:

• The system model of multi-UAV cooperative searching and tracking for moving targets is constructed, which is extended from decentralized partially observable markov decision process. By optimizing both the average observation rate of moving targets and the average exploration rate of mission area, UAVs can maintain a constant observation over perceived targets and exploration for unknown environment;
• A novel information update and fusion mechanism is proposed to enhance environment perception ability of the multi-UAV system. In our model, each UAV keeps its individual cognitive information about the mission region to guide its action in a fully distributed decision-making approach. UAVs can achieve better understanding of the environment and better cooperatioin via information update and fusion;
• A distributed MARL method is proposed to learn cooperative searching and tracking policies for multi-UAV system.The reward function and observation space are newly designed, considering both target searching and region exploration. The method has also been proven effective through simulation analysis.

The remainder of the paper is organized as follows. Section 2 introduces related works about target searching and MARL. Section 3 describes the mission model of multi-UAV cooperative searching for moving targets. Section 4 details the information update and fusion mechanisms for multi-UAV system. In Section 5, the proposed method based on MARL is demonstrated. In Section 6, simulation results and discussion are presented. Finally, Section 7 concludes the paper.

2. Related Works

2.1. Target Searching and Tracking

Search theory originated from Koopman’s research during World War II [13]. As a branch of search theory, target searching and tracking aims to find more unknown targets and track the perceived targets using limited resources through cooperation among multiple agents [14]. A widely used approach in the search problem is to divide mission area into cells and link each cell with a target existence probability to construct a probability map of entire area [15]. Force vectors method is also commonly used to solve the problem. In this method, control of multiple agents is achieved by adding up the force vectors that are attractive to nearby targets and repulsive to nearby agents [16,17]. In [18], the authors constructed the issue as a Multi-Objective Optimization (MOO) problem and propose a MOO algorithm to solve it. In [19], the authors proposed constraint programming (CP) to address target search problem for multiple UAVs.

However, it’s hard for the traditional planning and optimization methods to pursue optimal solution when problem scale is large. To reduce computational complexity, swarm intelligence algorithm is proposed to address the target searching and tracking issue. In [20], the authors proposed a method called motion-encoded particle swarm optimization (MPSO) to locate mobile targets using UAVs. Zhen et al. proposed an intelligent collaborative mission planning method for multiple UAVs to search and locate time-sensitive moving objects in an unpredictable dynamic setting, utilizing a combination of artificial potential field and ant colony optimization (HAPF-ACO) technique [21]. In [22], the authors presented a dynamic discrete pigeon-inspired optimization (PIO) algorithm to solve search-attack mission planning problem for UAVs. Hayat et al. presented a multi-objective optimization approach based on genetic algorithm (GA) to allocate tasks for UAVs [23].

In light of the recent advancements in RL, certain studies have also endeavored to employ RL techniques to address the issue of target searching and tracking. Wang et al. presented a centralized RL method for UAV target searching and tracking, assuming that global state and actions of other UAVs are accessible to each individual UAV [24]. Yan et al. formulate the multi-object tracking issue as a motion planning problem and use deep reinforcement learning (DRL) to train a shared policy across all agents [25]. In [26], the authors presented the experience sharing Reciprocal Reward Multi-Agent Actor-Critic (MAAC-R) algorithm to learn cooperative tracking policy for UAV swarm. Shen et al. considered cooperative search for stationary targets by UAVs and proposed DNQMIX algorithm to solve the problem [27]. The comparison between the existing works are listed in

Table 1. Comparison between the existing works.

Reference	Method	Sensor Model	Communication Range	Target
[16]	A-CMOMMT	Deterministic	Limited	Moving
[17]	P-CMOMMT	Deterministic	Limited	Moving
[18]	MOO	Probabilistic	Unlimited	Moving
[19]	CP	Deterministic	Unlimited	Moving
[20]	PSO	Probabilistic	Unlimited	Moving
[21]	ACO	Probabilistic	Limited	Moving
[22]	PIO	Probabilistic	Limited	Static
[23]	GA	Deterministic	Limited	Static
[24]	RL	Probabilistic	Unlimited	Moving
[25]	DRL	Deterministic	Limited	Moving
[26]	MARL	Deterministic	Limited	Moving
[27]	MARL	Probabilistic	Limited	Static

.

2.2. Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) combines multi-agent systems (MAS) and reinforcement learning. It focuses on scenarios where multiple agents, each with their own set of actions and objectives, interact in the shared environment and learn how to make optimal decisions through reinforcement learning techniques [28]. An intuitive thought to solve MARL problem is to treat every agent separately, taking into account the other agents as part of the environment, as in independent Q-Learning (IQL) algorithm [29], independent actor-critic (IAC) algorithm [30] and independent proximal policy optimization algorithm (IPPO) [31]. However, the nonstationarity and partial observability of the multi-agent system in independent learning (IL) may result in learning instabilities or suboptimal performance observed in practice. A common approach to address nonstationarity of the environment is using a fully observable critic, such as multi-agent deep deterministic policy gradient (MADDPG) [32]. It utilizes the architecture of “centralized learning and decentralized execution” (CTDE) [33] and uses a joint critic to train policy network. In [34], the authors presented Multiple Actor Attention-Critic (MAAC) algorithm that can efficiently scale up with agents’ quantity. As a variant of proximal policy optimization (PPO), MAPPO is one of the state-of-the-art MARL algorithms [35]. The algorithm also adopts CTDE architecture and has high learning efficiency with limited computing resource.

Another MARL approach is to use value function factorization. In [36], the authors proposed value decomposition networks (VDN) to break down the value function and measure the effect of each agent on joint reward. Rashid et al. examines the identical issue as VDN and introduced the QMIX algorithm [37] which surpasses VDN. QMIX incorporates additional information from the global state to improve the solution quality. In [38], the authors proposed another value decomposition method value-decomposition actor-critic (VDAC). VDAC can use advantage actor critic training framework to efficiently gather samples.

3. Problem Formulation

The system model of cooperative searching and tracking problem for multiple UAVs is constructed in the section. We first propose several assumptions to simplify the scenario. Then the formulation of the problem is presented.

3.1. Scenario Description

We concentrate on the issue of multi-UAV cooperative searching and tracking for moving targets, as shown in Figure 1. The number and location of targets is unknown and the UAVs must therefore adopt a certain strategy to optimize the search and track process. To simplify the scenario and focus on the main problem, several assumptions are presented as follows:

• We assume that multiple homogeneous fixed-wing UAVs are used to search multiple unknown moving targets. UAVs’ flight is confined to a two-dimensional plane at a specific altitude and UAVs prevent collisions by stratifying the altitude. Targets wander randomly within the mission region;
• As shown in Figure 1, UAVs utilize the sensor to detect the targets below them. When a target falls in detection range of the sensor, UAVs can detect the target. But UAVs cannot distinguish the identities or indices of targets;
• Each UAV in our model possesses an adequate amount of memory and processing capacity. And UAVs cooperate with each other in a distributed framework. Each UAV can share information with neighboring UAVs in communication range and decides independently with its own cognitive information.

3.2. System Model

Environment model: The mission region $\mathbb{O}$ is regarded as a bounded two-dimensional rectangular plane and uniformly discretized into $L_x\times L_y$ square cells, which can be denoted as:

$$\mathbb{O}=\left\{G_{x,y}\mid x\in \left\{1,2,\dots ,L_x\right\},y\in \left\{1,2,\dots ,L_y\right\}\right\},$$

(1)

where $L_x$ and $L_y$ represent the number of cells in the rows and columns of mission region, respectively. In our model, we use $G_{x,y}$ to represent each cell and the coordinate of cell $G_{x,y}$ is indicated as ${\mathit{g}}_{x,y}=(x,y)$, where $x\in \left\{1,2,\dots ,L_x\right\},y\in \left\{1,2,\dots ,L_y\right\}$.

Target model: Let $V=\left\{V_1,V_2,V_3\dots ,V_{N_V}\right\}$ be the set of $N_V$ moving targets. And we use ${\mathsf{\nu }}_{k,t}=(x,y)$ to denote the coordinate of the target $V_k(1\le k\le N_V)$ at timestep $t(1\le t\le T)$. The mission period is divided into T discrete timesteps in our model. UAVs and targets make movement decisions only at the beginning of each timestep. At each timestep, each target can only take up a single cell. and the target presence state of each cell is denoted as ${\theta }_{x,y}$. There are two possible values of ${\theta }_{x,y}$, namely ${\theta }_{x,y}=1$ means that a target is located in the cell $G_{x,y}$ and ${\theta }_{x,y}=0$ means that there is no target. And it is possible to have more than one target in a single cell.

UAV model: The multi-UAV system $U=\left\{U_1,U_2,U_3\dots ,U_{N_U}\right\}$ consists of $N_U$ homogeneous fixed-wing UAVs. The location of UAV can be determined by its projection onto the mission region $\mathbb{O}$ and the coordinate of UAV $U_i(1\le i\le N_U)$ at timestep $t(1\le t\le T)$ is represented as ${\mathsf{\mu }}_{i,t}=(x,y)$. The UAV can choose to move into one of its eight neighbor cells or just stay at its current cell at each timestep.

Each UAV is equipped with a sensor and the UAV $U_i$ can only detect cells within its sensing area ${\mathbb{S}}_{i,t}$ at timestep t with sensing radius $R_s$ (A cell is regarded to be completely covered if its center is covered), where

$${\mathbb{S}}_{i,t}=\left\{G_{x,y}\in \mathbb{O}\mid \parallel {\mathsf{\mu }}_{i,t}-{\mathit{g}}_{x,y}\parallel \le R_s\right\}.$$

(2)

The detection result of UAV $U_i$ at timestep t for the cell $G_{x,y}$ is represented as $Z_{i,t}^{x,y}$. It has two possible values, namely $Z_{i,t}^{x,y}=1$ means that a target is detected in the cell and $Z_{i,t}^{x,y}=0$ means that no target is detected. The sensor of UAV can only sense the presence of target but cannot distinguish the identity. Due to the imperfect detection ability of sensors, the detection model’s conditional probability of positive and negative properties is constructed as follows:

$$P\left(Z_{i,t}^{x,y}|{\theta }_{x,y}\right)=\left\{\begin{array}{c}P\left(Z_{i,t}^{x,y}=1|{\theta }_{x,y}=1\right)=P_D\hfill \\ P\left(Z_{i,t}^{x,y}=0|{\theta }_{x,y}=1\right)=1-P_D\hfill \\ P\left(Z_{i,t}^{x,y}=1|{\theta }_{x,y}=0\right)=P_F\hfill \\ P\left(Z_{i,t}^{x,y}=0|{\theta }_{x,y}=0\right)=1-P_F\hfill \end{array}\right.,$$

(3)

where $P_D$ and $P_F$ denote the true detection probability and false alarm probability respectively.

Each UAV is limited to interacting with its neighboring UAVs within the communication range. In our model, we only take into account the restricted communication range, disregarding any communication delays, bandwidth restrictions or interruptions. The group of neighbors for UAV $U_i$ at timestep t is denoted as

$${\mathbb{N}}_{i,t}=\left\{U_j\in U\mid \parallel {\mathsf{\mu }}_{i,t}-{\mathsf{\mu }}_{j,t}\parallel \le R_c\right\},$$

(4)

where $R_c$ represents the communication range. And we use $d_{i,t}=\left|{\mathbb{N}}_{i,t}\right|$ to denote the number of neighbors of UAV $U_i$ at timestep t (a UAV is assumed to be a neighbor of itself).

Metrics: Target $V_k$ is observed when it is within the sensing area of at least one UAV, which can be denoted as

$$O_{k,t}=\left\{\begin{array}{c}1,\mathrm{if}\exists U_i,\parallel {\mathsf{\nu }}_{k,t}-{\mathsf{\mu }}_{i,t}\parallel \le R_s\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.,$$

(5)

where $O_{k,t}$ is the observation state of target $V_k$ at timestep t. The observation rate ${\alpha }_k$ of target $V_k$ can be denoted as

$${\alpha }_k=\frac{1}{T}\sum _{t=1}^T{O}_{k,t},$$

(6)

which denotes the proportion of the time under surveillance to total mission time. Therefore, one of the objectives for UAVs is to maximize the average observation rate $\overline{\alpha }$ of $N_V$ targets, which can be represented as follows:

$$\overline{\alpha }=\frac{1}{N_V}\sum _{k=1}^{N_V}{\alpha }_k.$$

(7)

In addition, since the targets’ quantity is unknown, the multi-UAV system must persistently search the mission area to detect new targets. Therefore, another objective of the multi-UAV system is to maximize the exploration rate $\beta$ of mission area, which can be represented as

$$\beta =\frac{1}{T}\frac{1}{L_x{L}_y}\sum _{x=1}^{L_x}\sum _{y=1}^{L_y}{F}_{x,y},$$

(8)

where $F_{x,y}$ indicates the latest observed time for the cell $G_{x,y}$. The maximum value of $\beta$ is 1, which represents that all the cells in mission area are being monitored during the whole mission. However it cannot be achieved since the maximum sensing area of the UAVs is less than the mission area, i.e., $\underset{U_i\in U}{\cup }{\mathbb{S}}_{i,t}<\mathbb{O}$. In the scenario of cooperative searching and tracking for moving targets, the multi-UAV system needs to balance between monitoring perceived targets and exploring mission area [25].

4. Information Update and Fusion

In our model, each UAV keeps its individual cognitive information about the mission region to guide its action in a fully distributed decision-making approach. There are three individual cognitive information maps, i.e., target probability map, environment search status map and UAV location map. In this section, we will introduce these three maps and their information update and fusion mechanisms.

4.1. Target Probability Map

In the multi-UAV system, each UAV $U_i$ keeps the target probability map ${\mathcal{P}}_{i,t}$ of the whole mission region, which can be represented as:

$${\mathcal{P}}_{i,t}={\left[{\mathcal{P}}_{i,t}^{x,y}\right]}_{L_x\times L_y}.$$

(9)

${\mathcal{P}}_{i,t}^{x,y}\in [0,1]$ indicates target existence probability in cell $G_{x,y}$ at timestep k in the understanding of UAV $U_i$. Initially, ${\mathcal{P}}_{i,0}^{x,y}=0.5$, which means that there is no prior information about the distribution of targets.

Fundamentally, every UAV should possess the capability to autonomously modify its target probability map solely relying on its own observations, without the need for information exchange with neighbors. This would ensure that the multi-UAV system remains robust to any disruption in communication. Therefore, we will first introduce the target probability update by observations for each individual UAV without cooperation. Bayesian rule [39,40] is most popular technique to update the probability map based on observations, which is represented as

$$\begin{array}{cc}\hfill {\mathcal{P}}_{i,t}^{x,y}& =P\left({\theta }_{x,y}=1\mid Z_{i,t}^{x,y}\right)\hfill \\ \hfill & =\frac{P\left(Z_{i,t}^{x,y}\mid {\theta }_{x,y}=1\right){\mathcal{P}}_{i,t-1}^{x,y}}{P\left(Z_{i,t}^{x,y}\mid {\theta }_{x,y}=1\right){\mathcal{P}}_{i,t-1}^{x,y}+P\left(Z_{i,t}^{x,y}\mid {\theta }_{x,y}=0\right)\left(1-{\mathcal{P}}_{i,t-1}^{x,y}\right)}\hfill \\ \hfill & =\left\{\begin{array}{cc}\frac{P_D{\mathcal{P}}_{i,t-1}^{x,y}}{P_D{\mathcal{P}}_{i,t-1}^{x,y}+P_F\left(1-{\mathcal{P}}_{i,t-1}^{x,y}\right)},\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=1\hfill \\ \frac{(1-P_D){\mathcal{P}}_{i,t-1}^{x,y}}{(1-P_D){\mathcal{P}}_{i,t-1}^{x,y}+(1-P_F)\left(1-{\mathcal{P}}_{i,t-1}^{x,y}\right)},\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=0\hfill \\ {\mathcal{P}}_{i,t-1}^{x,y},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(10)

It is evident that the probability map update does not follow a linear pattern. To reduce computational complexity, a linear update method is introduced, which is initially suggested in [41]. Firstly, Equation (10) is equivalent to

$$\frac{1}{{\mathcal{P}}_{i,t}^{x,y}}-1=\left\{\begin{array}{cc}\frac{P_F}{P_D}\left(\frac{1}{{\mathcal{P}}_{i,t-1}^{x,y}}-1\right),\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=1\hfill \\ \frac{1-P_F}{1-P_D}\left(\frac{1}{{\mathcal{P}}_{i,t-1}^{x,y}}-1\right),\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=0\hfill \\ \frac{1}{{\mathcal{P}}_{i,t-1}^{x,y}}-1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(11)

A nonlinear conversion of ${\mathcal{P}}_{i,t}^{x,y}$ is adopted:

$$Q_{i,t}^{x,y}\triangleq ln\left(\frac{1}{{\mathcal{P}}_{i,t}^{x,y}}-1\right).$$

(12)

Then, Equation (11) can be rewritten as:

$$Q_{i,t}^{x,y}=Q_{i,t-1}^{x,y}+{\Delta }_{i,t}^{x,y},$$

(13)

where

$${\Delta }_{i,t}^{x,y}=\left\{\begin{array}{cc}ln\frac{P_F}{P_D},\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=1\hfill \\ ln\frac{1-P_F}{1-P_D},\hfill & \mathrm{if}{Z}_{i,t}^{x,y}=0\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(14)

Compared to the Bayesian method in Equation (10), the update in Equation (13), which is a linear function of $Q_{i,t}^{x,y}$, is more efficient in calculation.

When multiple UAVs are deployed, each UAV can broadcast the cognitive information to its neighbors for information fusion and cooperation, which will result in better performance. The UAV starts by examining the cells in its sensing area and then passing the observations to its neighboring UAVs. Following the receipt of observations from other UAVs, $Q_{i,t}^{x,y}$ undergoes the following update:

$$R_{i,t}^{x,y}=\eta Q_{i,t-1}^{x,y}+\sum _{j\in {\mathbb{N}}_{i,t}}{\Delta }_{j,t}^{x,y},$$

(15)

where $\eta \in (0,1)$ is the information decaying factor. Then, each UAV $U_i$ transmits $R_{i,t}^{x,y}$ of the whole area to the neighbors for information fusion, which is represented by:

$$Q_{i,t}^{x,y}=\sum _{j\in {\mathbb{N}}_{i,t}}{w}_{i,j,t}{R}_{j,t}^{x,y},$$

(16)

where

$$w_{i,j,t}=\left\{\begin{array}{cc}1-\left(\left(d_{i,t}-1\right)/N_U\right),\hfill & \mathrm{if}j=i\hfill \\ 1/N_U,\hfill & \mathrm{if}j\ne i\mathrm{and}j\in {\mathbb{N}}_{i,t}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(17)

4.2. Environment Search Status Map

The environment search status map that records the latest detected time for cells can be indicated as follows:

$${\mathcal{E}}_{i,t}={\left[{\mathcal{E}}_{i,t}^{x,y}\right]}_{L_x\times L_y},$$

(18)

where ${\mathcal{E}}_{i,t}^{x,y}$ denotes the latest observed time of cell $G_{x,y}$ at timestep k in the understanding of UAV $U_i$. And the initial value is set to 0, i.e., ${\mathcal{E}}_{i,0}^{x,y}=0$.

The environment search status map ${\mathcal{E}}_{i,t}$ is updated in two stages. The map ${\mathcal{E}}_{i,t}$ is first updated by the individual detection of UAV $U_i$ at each timestep t, which can be represented as

$${\mathcal{E}}_{i,t}^{x,y}=t,\mathrm{if}{G}_{x,y}\in {\mathbb{S}}_{i,t}.$$

(19)

Then, the environment search status maps from the neighboring UAVs can be integrated. The information fusion mechanism of the environment search status map can be denoted as follows:

$${\mathcal{E}}_{i,t}^{x,y}={\mathcal{E}}_{j,t}^{x,y},\mathrm{if}{\mathcal{E}}_{j,t}^{x,y}>{\mathcal{E}}_{i,t}^{x,y}\mathrm{and}{U}_j\in {\mathbb{N}}_{i,t}.$$

(20)

4.3. UAV Location Map

The UAV location map records the positions of UAVs and can be represented as follows:

$${\mathcal{L}}_{i,t}={\left[{\mathcal{L}}_{i,t}^{x,y}\right]}_{L_x\times L_y},$$

(21)

where ${\mathcal{L}}_{i,t}$ indicates whether a UAV is in the cell $G_{x,y}$ at timestep t in the understanding of UAV $U_i$. ${\mathcal{L}}_{i,t}$ is defined as a binary variable, namely ${\mathcal{L}}_{i,t}=1$ means that there is a UAV in the cell and ${\mathcal{L}}_{i,t}=0$ means that there is no UAV.

The UAV location map ${\mathcal{L}}_{i,t}$ is updated in two stages. The map ${\mathcal{L}}_{i,t}$ of UAV $U_i$ is first updated by the location information of its neighbors within its communicaiton range at each timestep t, which can be represented as

$${\mathcal{L}}_{i,t}^{x,y}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathsf{\mu }}_{j,t}=(x,y)\mathrm{and}{U}_j\in {\mathbb{N}}_{i,0}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(22)

Then, the UAV location maps from the neighboring UAVs can be integrated. The information fusion mechanism of the UAV location map can be represented as follows:

$${\mathcal{L}}_{i,t}^{x,y}={\mathcal{L}}_{j,t}^{x,y},\mathrm{if}{\mathcal{L}}_{j,t}^{x,y}>{\mathcal{L}}_{i,t}^{x,y}\mathrm{and}{U}_j\in {\mathbb{N}}_{i,t}.$$

(23)

5. Proposed Method

5.1. Preliminaries

5.1.1. Decentralized Partially Observable Markov Decision Process

The cooperative searching and tracking problem for multiple UAVs can be modeled as a decentralized partially observable Markov decision process (Dec-POMDP). At every timestep t, each agent in the system make its decision based on the local observation (including perception and communication). And then the environment execute all agents’ joint action to update the immediate rewards and the next global state.

Assuming there are n agents, the Dec-POMDP can be described as follows (the time subscript t is omitted for convenience and the joint variables over all agents are bold) [42]:

$$(N,S,\mathit{A},T,\mathit{R},\mathit{O},Z,\gamma ),$$

(24)

where:

• N denotes the set of n agents;
• S denotes the global state space and $s\in S$ denotes the current and specific state;
• $\mathit{A}:\{A^{\left(1\right)},A^{\left(2\right)},\cdots A^{\left(n\right)}\}$ denotes all agents’ joint action space and $a^{\left(i\right)}\in A^{\left(i\right)}$ is the action of the i-th agent;
• $T:P\left(s^{\prime }\right|s,\mathit{a})\to [\mathtt{0},1]$ is the state trasition probability function from state s to next state $s^{\prime }$ given joint action $\mathit{a}:\{a^{\left(1\right)},a^{\left(2\right)},\cdots a^{\left(n\right)}\}$;
• $\mathit{R}(s,\mathit{a}):\{r^{\left(1\right)},r^{\left(2\right)},\cdots ,r^{\left(n\right)}\}$ denotes the joint reward function by executing the joint action $\mathit{a}$ given state s;
• $\mathit{O}:\{O^{\left(1\right)},O^{\left(2\right)},\cdots O^{\left(n\right)}\}$ represents all agents’ joint observation space, and $o^{\left(i\right)}\in O^{\left(i\right)}$ denotes the local observation of the i-th agent;
• $Z:o^{\left(i\right)}=Z(s,i)$ is the local observation function of the i-th agent given the global state s;
• $\gamma \in [0,1]$ denotes the constant discount factor.

5.1.2. Multi-Agent Proximal Policy Optimization

The performance of single agent reinforcement learning algorithms are usually unsatisfactory in multi-agent system because each agent is influenced by environment and other agents at the same time. Therefore, it is necessary to use MARL to train the agents in Dec-POMDP. As a variant of proximal policy optimization (PPO) specialized for multi-agent settings, Multi-Agent Proximal Policy Optimization (MAPPO) is one of the state-of-the-art MARL algorithms [35]. The algorithm adopts centralized training with decentralized execution (CTDE) architecture and has high learning efficiency with limited computing resource. In our paper, we adopt MAPPO as the learning method for multi-UAV cooperative searching and tracking problem.

PPO is an on-policy reinforcement learning algorithm based on Actor-Critic framework. It originates from Trust Region Policy Optimization (TRPO) and addressing the challenge posed by policy gradient algorithm’s sensitivity to step size and the difficulty in determining an appropriate step size. The objective of RL is to learn the optimal policy ${\pi }^{*}$ to maximize the cumulative discounted reward $R_t$:

$$R_t=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+{\gamma }^3{r}_{t+4}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1},$$

(25)

where $\gamma \in [0,1]$ is the discount factor. The policy $\pi$ can be measured by the state-value function $V^{\pi }\left(s\right)$ and the action-value function $Q^{\pi }(s,a)$. State-value function $V^{\pi }\left(s\right)$ refers to the expected cumulative discounted reward by following the policy $\pi$ under state s:

$$V^{\pi }\left(s_t\right)={\mathbb{E}}_{\pi }\left[R_t\mid s_t\right]={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{k+1}\mid s_t\right].$$

(26)

And action-value function $Q^{\pi }(s,a)$ refers to the expected reward by following the policy $\pi$ under state s and action a:

$$Q^{\pi }(s_t,a_t)={\mathbb{E}}_{\pi }\left[\begin{array}{c}{R}_t\mid s_t,a_t\end{array}\right]={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{k+1}\mid s_t,a_t\right].$$

(27)

To improve the learning efficiency and enhance the stability of training process, the advantage function $A^{\pi }(s,a)$ is introduced:

$$A^{\pi }(s_t,a_t)=Q^{\pi }(s_t,a_t)-V^{\pi }\left(s_t\right).$$

(28)

The advantage function $A^{\pi }(s,a)$ gives the advantage of taking a given action over following the policy. Define ${\hat{A}}_t$ as the estimate of advantage function $A^{\pi }(s,a)$ at timestep t:

$${\hat{A}}_t=-V\left(s_t\right)+r_t+\gamma r_{t+1}+\dots +{\gamma }^{T-t+1}{r}_{T-1}+{\gamma }^{T-t}V\left(s_T\right),$$

(29)

where T is the trajectory length. Policy gradient method is an approach to solve reinforcement learning problem by directly optimizing the policy. The loss function of policy gradient method used in PPO is denoted as

$$L^{CLIP}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min(r_t\left(\theta \right){\hat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-\epsilon ,1+\epsilon ){\hat{A}}_t)\right],$$

(30)

where $r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_d}\left(a_t\right|s_t)}$ represents the probability ratio. The clip function in Equation (30) is used to restrict $r_t\left(\theta \right)$ to the interval $[1-\epsilon ,1+\epsilon ]$.

Yu et al. [35] proposed MAPPO algorithm for POMDP based on PPO method. The method proposed in our paper uses MAPPO to train two independent networks, a policy network ${\pi }_{\theta }$ with parameter $\theta$ and a value network $V_{\varphi }$ with parameter $\varphi$, which can be shared among all homogeneous agents. The policy network ${\pi }_{\theta }$ maps the observation $o_t$ of the agent to the discrete action space. And the value network $V_{\varphi }$ maps the state $s_t$ of environment to current value. For multiple agents, the loss function $L^{CLIP}\left(\theta \right)$ is rewritten as

$$L^{CLIP}\left(\theta \right)=\frac{1}{BN}\sum _{i=1}^B\sum _{k=1}^N\min (r_{\theta ,i}^{\left(k\right)}{A}_i^{\left(k\right)},\mathrm{clip}(r_{\theta ,i}^{\left(k\right)},1-\epsilon ,1+\epsilon )A_i^{\left(k\right)}),$$

(31)

where B refers to the batch size and N refers to the agents’ quantity. In Equation (31), $r_{\theta ,i}^{\left(k\right)}=\frac{{\pi }_{\theta }\left(a_i^{\left(k\right)}\mid s_i^k\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_i^{\left(k\right)}\mid s_i^k\right)}$ is the probability ratio in multi-agent system and $A_i^{\left(k\right)}$ is the advantage function. Correspondingly, the loss function $L^{CLIP}\left(\varphi \right)$ of the value network $V_{\varphi }$ is rewritten as

$$L^{CLIP}\left(\varphi \right)=\frac{1}{BN}\sum _{i=1}^B\sum _{k=1}^{N}max[{(V_{\varphi }\left(s_i^{\left(k\right)}\right)-{\widehat{R}}_i)}^2,{(\mathrm{clip}(V_{\varphi }\left(s_i^{\left(k\right)}\right),V_{{\varphi }_{old}}\left(s_i^{\left(k\right)}\right)-\epsilon ,V_{{\varphi }_{old}}\left(s_i^{\left(k\right)}\right)+\epsilon )-{\widehat{R}}_i)}^2].$$

(32)

5.2. Observation and Action Space Representation

For the cooperative searching and tracking problem in our paper, each UAV in the multi-UAV system keeps its individual cognitive information about the mission region to guide its action in a fully distributed decision-making approach. However, if the UAV utilizes the cognitive data of the entire mission area, an excessive amount of incorrect information can lead to confusion in the agent’s decision-making. Thus, assuming that each UAV has a field of observation, it will only utilize the cognitive information within the restricted scope of observation. By decreasing the input dimension of policy network, the training speed can be enhanced, and the fixed input dimension can also enhance generalization to mission regions of any size.

Define the observation field as a square area (with side length $L_o$) centered in the UAV’s current cell. At each timestep, the UAV extract the portion of the local cognitive information about the mission region as input of the policy network. The observation of UAV $U_i$ at timestep t is composed of three parts:

$$o_{i,t}=[p_{i,t},e_{i,t},l_{i,t}],$$

(33)

• Target Probability Information $p_{i,t}$: It is extracted from target probability map ${\mathcal{P}}_{i,t}$ and the target existence probability out of mission region is assumed 0;
• Environment Search Status Information $e_{i,t}$: It is extracted from environment search status map ${\mathcal{E}}_{i,t}$ and ${\mathcal{E}}_{i,t}^{x,y}$ should be divided by t for normalization. The environment search status information out of the mission region is assumed 1, which means that there is no need to explore;
• UAV Location Information $l_{i,t}$: It is extracted from UAV location map ${\mathcal{L}}_{i,t}$. And the value of ${\mathcal{L}}_{i,t}^{x,y}$ out of the mission region is assumed 0, which means that there is no UAV.

At each timestep, UAVs have at most nine available actions based on their current position. The action space comprises a collection of target cells encircling the UAV, meaning that each UAV has the option to remain in its present cell or transition into any of its eight adjacent cells. In the event that an action causes the UAV to exit the mission region during the subsequent timestep, it will be excluded from the list of potential actions.

In our paper, policy network ${\pi }_{\theta }$ and value network $V_{\varphi }$ are constructed to process observation or state inputs. Their outputs are the probability distribution over available actions and estimate of the expected value, respectively. The state input to value network $V_{\theta }$ also contains the same three parts: target location information, environment search status information and UAV location information as the observation $o_{i,t}$ mentioned above. But the state input represents the global information of environment rather than local observation of a specific agent.

There are two kinds of network architectures used in our method: one utilizes CNN (convolutional neural network) and fully connected layers to process information, while the other one incorporates RNN (recurrent neural network) in addition to that. The network architecture of recurrent-MAPPO is shown in Figure 2. Policy and value network first use CNN to process the input information. Then the output is flattened to obtain a one-dimensional feature vector, followed by two fully-connected layers and a gated recurrent unit (GRU) network. The usage of RNN is aimed at mining useful information hidden in historical observations. Finally, policy network uses softmax function to obtain the selection probabilities for each action and value network outputs the estimate of the expected value. We also construct the network architecture without RNN to train the agent in our paper. It omits the GRU layer while keeping the rest of the network architecture unchanged.

5.3. Reward Function Design

To encourage UAVs to search and track more targets, UAVs need to balance between tracking the already perceived targets and exploring the mission region. Besides, they should avoid overlapping the sensing range to maximize the total detection area. Rewards are used to shape expectations in reinforcement learning. Therefore, it is critical to devise reasonable rewards for the multi-UAV system. The reward function proposed in our paper consists of three rewards:

$$r_{i,t}=r_{i,t}^1+r_{i,t}^2+r_{i,t}^3,$$

(34)

where $r_{i,t}$ is the reward for UAV $U_i$ at timestep t.

Tracking reward: We use $r_{i,t}^1$ to encourage UAVs to track the discovered targets. It is commonly believed that the closer the UAV is to a target, the better the UAV can track the target. Thus, the reward for UAV $U_i$ to track target $V_k$ at timestep t is as follows:

$$r_{i,t}^1\left(V_k\right)=\left\{\begin{array}{cc}1+\left(R_s-\parallel {\mathsf{\mu }}_{i,t}-{\mathsf{\nu }}_{k,t}\parallel \right)/R_s,\hfill & \parallel {\mathsf{\mu }}_{i,t}-{\mathsf{\nu }}_{k,t}\parallel ⩽R_s\hfill \\ 0,\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(35)

And the reward of UAV $U_i$ at timestep t for tracking multiple targets can be calculated as $r_{i,t}^1={\omega }_1{\sum }_{k=1}^{N_V}{r}_{i,t}^1\left(V_k\right)$, where ${\omega }_1$ is a positive coefficient.

Exploring reward: The reward $r_{i,t}^2$ encourages UAVs to explore the mission region and can be represented as follows:

$$r_{i,t}^2={\omega }_2({\beta }_{i,t}-{\beta }_{i,t-1}),$$

(36)

where ${\omega }_2$ is a positive coefficient. In Equation (36), ${\beta }_{i,t}$ denotes the local exploration rate of the mission area of UAV $U_i$ at timestep t and ${\beta }_{i,t}$ can be denoted as follows:

$${\beta }_{i,t}=\frac{1}{t}\frac{1}{L_x{L}_y}\sum _{x=1}^{L_x}\sum _{y=1}^{L_y}{\mathcal{E}}_{i,t}^{x,y},$$

(37)

which can be calculated using its local environment search status map ${\mathcal{E}}_{i,t}$.

Overlapping punishment: The reward $r_{i,t}^3$ is designed as a punishment of UAV $U_i$ for approaching UAV $U_j$ too close, which can be represented as follows:

$$r_{i,t}^3\left(U_j\right)=\left\{\begin{array}{cc}-\exp \left(\left(2R_s-∥{\mathsf{\mu }}_{i,t}-{\mathsf{\mu }}_{j,t}∥\right)/\left(2R_s\right)\right),\hfill & ∥{\mathsf{\mu }}_{i,t}-{\mathsf{\mu }}_{j,t}∥⩽2R_s\mathrm{and}i\ne j\hfill \\ 0,\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(38)

Therefore, the punishment of UAV $U_i$ at timestep t for approaching multiple UAVs too close can be represented as $r_{i,t}^3={\omega }_3{\sum }_{j=1}^{N_U}{r}_{i,t}^3\left(U_j\right)$, where ${\omega }_3$ is a positive coefficient.

5.4. Training Framework Based on MAPPO

MAPPO algorithm is used to train the agents in our paper, which adopts CTDE architecture. The framework of MAPPO is shown in Figure 3. In training stage, parallel training environments generate global state $s_t$. Then observation model O and reward function R respectively generate global observation ${\mathit{o}}_{\mathit{t}}$ and global reward ${\mathit{r}}_{\mathit{t}}$. The policy network of UAV generates joint action ${\mathit{a}}_{\mathit{t}}$ based on observation ${\mathit{o}}_{\mathit{t}}$. Finally, the updated global state $s_{t+1}$ is obtained by feeding joint action ${\mathit{a}}_{\mathit{t}}$ into the training environment, thereby completing one interaction loop. During each interaction, intermediate data [$s_t$, ${\mathit{o}}_{\mathit{t}}$, ${\mathit{a}}_{\mathit{t}}$, ${\mathit{r}}_{\mathit{t}}$, $s_{t+1}$, ${\mathit{o}}_{\mathit{t}+1}$] is stored in the replay buffer. Once the accumulated experience data attains a predetermined amount, the gradients are calculated and the weights of policy and value network are updated.

In testing stage, the observation model O takes global state $s_t$ as input and outputs global observations ${\mathit{o}}_{\mathit{t}}$ for each UAV. The UAVs then generate joint actions ${\mathit{a}}_{\mathit{t}}$ based on ${\mathit{o}}_{\mathit{t}}$. Finally, the testing environment updates the global state based on ${\mathit{a}}_{\mathit{t}}$, completing the interaction. The pseudocode of MAPPO algorithm is shown in Algorithm 1.

Algorithm 1: MAPPO

6. Simulation Results

6.1. Setting Up

The settings of simulation will be first introduced before conducting simulation and analysis. In our simulation environment, multi-UAV system will search and track moving targets according to a certain strategy. And the performance of various algorithms will be measured by testing a variety of problem scenarios with varying scales. All the experiments are carried out on a PC with Windows 10 operating system, AMD 3.8 GHz CPU, 16 GB internal memory, and GeForce RTX 3070 GPU. Moreover, the simulation programmes are developed based on Python 3.10 and PyTorch.

It is assumed that each cell has a side length of 1, which serves as the basis for all distance metrics. The positions of UAVs and targets are randomly initialized for each problem instance. The maximum speed of UAV is set to 1 cell per timestep and the maximum speed of target is set to 0.5 cell per timestep. We will set different sizes of mission region, different number of UAVs and targets for multiple problem instances of different scales. The other simulation parameters are listed in

Table 2. Simulation parameters.

Parameters	Value
Total misssion time steps (T)	200
Detection probability ($P_D$)	0.9
False alarm probability ($P_F$)	0.1
Sensing range ($R_s$)	3
Communication range ($R_c$)	6
Range of observation field ($L_o$)	7
Information decaying factor ($\eta$)	0.1
${\omega }_1$	2
${\omega }_2$	1
${\omega }_3$	0.5

.

6.2. Convergence Analysis

The convergence performance of MAPPO and IPPO with different network architectures is compared in the subsection. The learning curves which represent the episode rewards, average observation rate $\alpha$ and average exploration rate $\beta$ with respect to training timesteps are respectively depicted in Figure 4. The algorithms using basic network architectures are referred to as MAPPO and IPPO, while those utilizing network with GRU module are labeled as rMAPPO and rIPPO. We train each method for five times with ${10}^5$ timesteps. The mean is represented by dark lines and the standard deviation over five different random seeds is represented by the shaded area. The results of Figure 4 are obtained from the cooperative search mission of 5 UAVs for 7 moving targets in an area of 20 × 20 cells. And all the hyperparameters used in training process are listed in

Table 3. Hyperparameters used in training process.

Parameters	Value
Number of steps to execute (E)	${10}^5$
Batch size (B)	16
Learning rate ($L_r$)	$5\times {10}^{-4}$
Discount factor ($\gamma$)	0.99
Clip factor ($\epsilon$)	0.2
Optimizer	Adam

.

As depicted in Figure 4a, MAPPO and IPPO have shown better performance. The episode rewards obtained by MAPPO and IPPO converge to about 110, while rMAPPO and rIPPO converge to about 80. Moreover, we can see that the convergence of MAPPO and rMAPPO is faster and more stable compared to IPPO and rIPPO. As depicted in Figure 4b, the average observation rate $\alpha$ of MAPPO and IPPO converge to about 0.28, while rMAPPO and rIPPO converge to about 0.27. In Figure 4c, the average exploration rate $\beta$ of MAPPO and IPPO converge to about 0.81, while rMAPPO and rIPPO converge to about 0.75.

6.3. Performance Analysis

Figure 4 indicates that the convergence performance of basic network architecture is better than recurrent network architecture. On one hand, RNN is too complex to the problem, which will lead to overfitting. On the other hand, the information update and fusion mechanism proposed in our method already preserves historical information, making the role of recurrent neural network redundant. We can also see that the convergence of MAPPO training algorithm is faster and more stable compared to IPPO. The MAPPO algorithm using CTDE architecture can effectively utilize global information to achieve better and more stable convergence in the training process. Different from MAPPO, as a fully decentralized method, IPPO algorithm directly applies a single-agent RL algorithm for each individual agent without considering changes of other agents. Thus, the environment is non-stationary to the agents and convergence performance cannot be guaranteed.

In this subsection, we use heatmap to visualize the cognitive information maps that incorporate all the detection information at different timesteps and analyze the effectiveness of our method. Furthermore, the performance of MAPPO with different sensing and communication capabilities is analyzed. The testing environment contains 5 UAVs and 7 targets in an area of 20 × 20 cells.

Figure 5 shows the target probability map and environment search status map at different timesteps. In target probability map, the degree of yellow in the cell is directly linked to the likelihood of the target’s presence. In environment search status map, the proximity of the cell’s color to dark red directly correlates with a decrease in the duration since the previous detection on the cell. Here we do not present the UAV location map but directly mark the locations of UAVs and targets on the other two maps. From Figure 5a,c, it can be seen that only two targets are under observation at timestep 30 while five targets have already been tracked at timestep 180. In target probability map, target existence probability of the cells under observation where targets are located is 1. Target existence probability of other cells that have been detected but do not contain targets approaches 0. And probability of undetected cells is 0.5, which means that no information is available about the presence of targets. The target probability map proposed in our paper contains information about distribution of targets, which can effectively guide UAVs to track and monitor the moving targets. In Figure 5b,d, we can see that multi-UAV system has only detected part of the mission region at timestep 30 while almost the entire area has been explored at timestep 180. The environment search status map contains search status information of the mission region. It can not only guide the UAVs to explore undetected areas but also promote revisiting of area that has not been explored for a while.

Next, as shown in Figure 6, average episode rewards, average observation rate $\alpha$ and average exploration rate $\beta$ versus sensing range $R_s$ under different communication conditions are tested. The results with communication are depicted by blue lines, denoted as “Com”, while the results without communication are represented by red lines, denoted as “NoCom”. The testing environment contains 5 UAVs and 7 targets in an area of 20 × 20 cells. Each data point is an average of 10 tests. It can be found that with the increase in sensing range $R_s$, average episode rewards, average observation rate $\alpha$ and average exploration rate $\beta$ all increase. Larger sensing range enables multi-UAV to detect larger area and acquire more information, leading to better performance. Compared with the result without communication, we can see that the performance of multi-UAV system is better when UAVs communicate with each other. On one hand, UAVs can achieve better understanding of the environment through information sharing and fusion. On the other hand, UAVs can achieve better cooperation via communication.

6.4. Comparison with Other Methods

This subsection compares MAPPO with A-CMOMMT, ACO and Random algorithms to examine the effectiveness of our method. The results of the comparison were obtained from scenarios of different scales: 20 × 20 cells with 10 targets, 30 × 30 cells with 15 targets and 40 × 40 cells with 20 targets. Each data point is an average of 10 tests. Moreover, the performance results of MAPPO are obtained using the trained model with 5 UAVs, 7 targets and 20 × 20 cells. An introduction to the comparison algorithm is as follows:

• A-CMOMMT: A-CMOMMT [16] is a traditional method to solve the target searching and tracking problem. In this method, control of multiple agents is based on a combination of force vectors that are attractive for nearby targets and repulsive for nearby agents;
• ACO: As a swarm intelligence algorithm, ant colony optimization (ACO) is also applied in the target searching and tracking problem [21,43]. The pheromone used to initialize all the cells in this approach is identical. The pheromone of cells encountered by UAVs will undergo vaporization at every time step. The pheromone map, target existence probability and UAVs’ locations will be included in the heuristic information to guide UAVs’ decisions;
• Random: At each timestep, the agents randomly select an action from the potential candidates.

The performance comparison of MAPPO, A-CMOMMT, ACO and Random algorithms under different number of UAVs from scenarios of different scales is depicted in Figure 7. Figure 7a–c show the results of the average observation rate $\alpha$ versus the UAVs’ quantity for different scenarios. We can see that with the increase in the UAVs’ quantity, the average observation rate $\alpha$ increases across all algorithms and the performance of MAPPO, A-CMOMMT and ACO far surpass that of Random. We can see that our MAPPO algorithm has the highest average observation rate $\alpha$ in most cases. A-CMOMMT outperforms ACO by a small margin in the environment with 10 targets and 20 × 20 cells and 15 targets and 30 × 30 cells. But ACO performs better in the environment with 20 targets and 40 × 40 cells. ACO has better performance when the scale of the problem is large, but the difference between A-CMOMMT and ACO is not much overall. Figure 7d–f display the correlation between the average exploration rate $\beta$ and the quantity of UAVs. in different scenarios. We can see that the Random algorithm has the highest average exploration rate $\beta$ and the performance of MAPPO is second only to Random. And we can find that the average exploration rate $\beta$ of A-CMOMMT and ACO is far inferior to that of Random and MAPPO. The average exploration rate $\beta$ in our paper is used to examine the effectiveness of algorithms in exploring the mission region. And a higher value of the indicator also means that the observation of all targets is done in a relatively uniform manner. The standard deviation of observation rate across all targets in the environment with 10 targets and 20 × 20 cells is listed in

Table 4. Standard deviation of observation rate of targets.

	4 UAVs	6 UAVs	8 UAVs	10 UAVs
MAPPO	0.143	0.174	0.176	0.185
A-CMOMMT	0.228	0.237	0.231	0.239
ACO	0.226	0.254	0.220	0.193
Random	0.119	0.152	0.148	0.168

. As shown in Table 4, with better exploring performance, MAPPO can observe the targets relatively uniformly. In summary, the performance comparison illustrates that the proposed method MAPPO performs better in target searching and tracking problem than the traditional algorithm A-CMOMMT and swarm intelligence algorithm ACO.

7. Conclusions

We focus on the target searching and tracking problem and propose a distributed MARL method to learn the cooperative policies for the multi-UAV system in the paper. We first construct the system model of multi-UAV cooperative searching and tracking problem for multiple UAVs and consider average observation rate and average exploration rate as the metrics. Moreover, we propose the information update and fusion mechanisms to enhance environment perception ability of the multi-UAV system. Then, the details of our method are demonstrated, including observation and action space representation, reward function design and training framework based on MAPPO. The simulation results have shown that our method has well convergence performance and outperforms comparative algorithms in terms of average observation rate and average exploration rate. In the future, we will delve deeper into the collaborative policies of heterogeneous UAVs when it comes to target searching and tracking, and assess the practicality of the suggested approach for real-world UAVs.