Factored Multi-Agent Soft Actor-Critic for Cooperative Multi-Target Tracking of UAV Swarms

Abstract

In recent years, significant progress has been made in the multi-target tracking (MTT) of unmanned aerial vehicle (UAV) swarms. Most existing MTT approaches rely on the ideal assumption of a pre-set target trajectory. However, in practice, the trajectory of a moving target cannot be known by the UAV in advance, which poses a great challenge for realizing real-time tracking. Meanwhile, state-of-the-art multi-agent value-based methods have achieved significant progress for cooperative tasks. In contrast, multi-agent actor-critic (MAAC) methods face high variance and credit assignment issues. To address the aforementioned issues, this paper proposes a learning-based factored multi-agent soft actor-critic (FMASAC) scheme under the maximum entropy framework, where the UAV swarm is able to learn cooperative MTT in an unknown environment. This method introduces the idea of value decomposition into the MAAC setting to reduce the variance in policy updates and learn efficient credit assignment. Moreover, to further increase the detection tracking coverage of a UAV swarm, a spatial entropy reward (SER), inspired by the spatial entropy concept, is proposed in this scheme. Experiments demonstrated that the FMASAC can significantly improve the cooperative MTT performance of a UAV swarm, and it outperforms existing baselines in terms of the mean reward and tracking success rates. Additionally, the proposed scheme scales more successfully as the number of UAVs and targets increases.

1. Introduction

Multi-target tracking (MTT) with an unmanned aerial vehicle (UAV) swarm has attracted widespread research interest due to its important role in highly mobile networks [1,2,3,4]. In the MTT task [5,6], numerous small UAVs are deployed in a distributed manner across the task area to search for unknown vehicle user targets and track the perceived targets cooperatively. This technique addresses the requirement of tracking moving targets for periods of time by taking advantage of the flexibility, low cost, and high mobility of UAVs. A key challenge in MTT is how to coordinate multiple UAVs to track diverse unknown targets accurately and consecutively [1].

MTT with UAV swarm is essentially a multi-agent cooperative decision-making optimization problem. A large number of MTT studies based on intelligent optimization algorithms have been published in recent decades; studied methods include particle swarm optimization [7], multi-objective optimization [8], and cyclic stochastic optimization algorithms [9,10]. Traditional intelligent optimization algorithms can find the optimal or sub-optimal solution of the objective function through a parallel heuristic search. Such algorithms are gradient-free and have a strong optimization performance in complex non-convex objective functions. However, these algorithms also require online searches to solve the problem, which do not meet the real-time calculation requirements. Moreover, the optimization speed is greatly reduced as the problem’s scale grows. Therefore, these methods still present great difficulties in practical applications.

With the great progress of reinforcement learning (RL) and multi-agent reinforcement learning (MARL), some scholars have also attempted to use RL to solve the MTT problem [11,12,13,14]. Examples include the mixed noises deep deterministic policy gradient (MN-DDPG) [11] and the reciprocal reward multi-agent actor–critic (MAAC-R) [14]. RL and MARL utilize the agent to interact with the environment by trial and error to allow the UAV to learn and track the target. However, it is not easy to cooperatively realize the multi-target tracking of a UAV swarm using MARL. Generally, cooperative MARL adopts a centralized training with decentralized execution (CTDE) paradigms, which suffer from the global action–value function, whose complexity grows exponentially with the number of agents [15]. Value decomposition enables CTDE to be used in value-based MARL [16,17,18,19,20,21], but policy-based MARL or multi-agent actor-critic (MAAC) [22,23,24,25,26,27] frameworks that learn through continuous actions create high variance. This problem occurs because the exploration or sub-optimality of other agents’ policies might propagate through the centralized critic and aggravate the training of other agents. Therefore, determining how to realize efficient cooperation for multiple agents remains a significant challenge.

In addition, reward design plays a key role in RL settings. A reasonable and effective reward can significantly improve MTT performance. Information entropy is a measure of the uncertainty of the entropy. Spatial entropy is a measure of the spatial coverage [28], which is important for target searching and tracking tasks. Under this backdrop, using spatial entropy to guide the UAV swarm to search and track targets might improve the cooperation behavior between UAVs.

In this paper, we propose a cooperative MARL scheme called factored multi-agent soft actor-critic (FMASAC) to achieve MTT for a UAV swarm. First, we describe the MTT problem and formulize this multi-agent cooperative decision-making problem as a decentralized partially observable Markov decision process (Dec-POMDP) model. Then, inspired by the idea of value decomposition, we improve the vanilla multi-agent soft actor-critic (MASAC) by factorizing the centralized critic as the composition of the individual critic and using factored soft policy iterations to optimize the policy. On this basis, the FMASAC scheme using spatial entropy reward (SER) is proposed to learn and solve the MTT tasks of the UAV swarm. Finally, simulation experiments are conducted to test the effectiveness, generalization, and scalability of the proposed method. The contributions of this paper are summarized as follows:

• We formulate a UAV swarm-based MTT problem within a Dec-POMDP setting. The UAVs are partially observable, can only perceive the targets within the observation range, and can communicate with the neighboring UAVs. A recurrent neural network (RNN) is added to the actor-critic network to gather the historical information from the hidden state of the RNN, which solves the problem of incomplete information caused by partial observations. To increase the detection coverage and boost the tracking efficiency, inspired by the concept of spatial entropy, we design a shaping-reward SER. In addition, safe distance constraints are considered in the reward function to avoid collisions.
• MARL is used to solve the MTT optimization problem, which does not need a pre-set target trajectory and can learn to track targets in an unknown environment. Here, the CTDE paradigm is adopted, where global observation–action history can be accessed during centralized training, and trained policies are executed conditioned only on local information in a decentralized way. Moreover, the trained model can generalize to an unknown and dynamic changing environment.
• We propose using the FMASAC algorithm, which adopts an entropy maximization MARL for greater exploration and introduces the idea of value decomposition. This algorithm effectively combines the advantages of the value decomposition and MASAC methods, which reduces the variance in policy updates, achieves efficient credit assignment [29], and enables the scalable learning of a centralized critic in Dec-POMDP.

The rest of this paper is organized as follows. Section 2 summarizes related work about the target tracking problem. Section 3 provides a preliminary outline of Dec-POMDP and MASAC. Section 4 formulates the UAV kinematic model and the cooperative MTT optimization model. Section 5 details the proposed FMASAC-based MTT methods. Experimental results and analyses are presented in Section 6. Finally, Section 7 concludes this paper and envisages future work. For the sake of clarity, all the symbols and notations used in this paper are summarized in

Table 1. Summary of notations.

Symbol	Definition
$n,m$	The numbers of the UAVs and targets.
$i,j,k$	The indexes of each UAV, each neighbor, and each target.
$(x_i^{},y_i^{}),v_i,{\phi }_i^{}$	The position, velocity, and heading of UAV $i$.
$a_{\max },a_i^t$	The maximum heading angular rate and the action of UAV $i$.
$N_a,n_a$	The UAV’s cardinality of the discrete action space, and the corresponding index of its discrete action.
$d_o,d_c$	The maximum observation distance and maximum communication distance.
$d_{i,j},d_{i,k},d_{i,O}$	The distance between UAV $i$ and UAV $j$, target $k$, and scenario center $O$.
$o_{i,k}=(x_k^T,y_k^T,v_k^{Tx},v_k^{Ty})$	The UAV $i$’s observation information about target $k$.
$c_{i,j}=(x_j^{},y_j^{},v_j^x,v_j^y,a_j^{t-1})$	The UAV $i$’s communication information from neighbor $j$.
$o_i=(x_i,y_i,{\phi }_i,o_{i,k},c_{i,j})$	The UAV $i$’s local observation information.
$s$	The global state of the environment.

, and the definition of the default symbols in MARL is omitted.

2. Related Work

In Section 2, we review some recent publications on target tracking from two perspectives: traditional optimization methods and reinforcement learning methods. A brief introduction of these two optimization methods is provided in

Table 2. The common optimization methods for target tracking.

Methods	Advantage	Limitation
Traditional optimization	Powerful non-convex optimization and convergence performance; robust.	Computationally expensive; poor real-time performance of online searching; limitations in large-scale variable problems.
Reinforcement learning	Implicit modeling unknown environment; data-driven; offline training and online decision framework; near real-time solving speed.	Learning inefficiency; credit assignment; poor interpretation; scalability.

.

2.1. Traditional Optimization Methods

Target tracking is a classic optimization problem. Therefore, most published works are based on traditional optimization algorithms. Pitre et al. [7] described a joint search and tracking mission for UAVs and presented an objective function that integrates target detection, target tracking, and UAV survivability. A modified particle swarm optimization algorithm was proposed to optimize the detection and tracking trajectory. Jilkov et al. [8] developed a multiple objective optimization algorithm to optimize the search and tracking path under multiple objectives for one or multiple UAVs in uncertain environments. A cyclic stochastic optimization algorithm was adopted in [9] to solve the target tracking problem, which demonstrated that multiple agents can cooperatively search a region and track all sensed targets. More related works can be found in [10].

As we can conclude from the above studies, traditional optimization methods formulate the target tracking task as an optimization problem to solve. Despite powerful optimization and convergence performance, these methods require an online search for an unknown and dynamic changing environment, which is computationally expensive for onboard computers and can reduce the real-time performance of target tracking.

2.2. Reinforcement Learning Methods

With the tremendous advances in RL in recent years, some studies have tried to adopt RL to solve the target tracking problem. Li et al. [11] proposed an MN-DDPG-based online path-planning approach, which can achieve the perception of the target and good self-adaptive flight control for UAVs in the tasks related to maneuvering target tracking. Wang et al. [12] developed a centralized-RL-based cooperative target searching and tracking method for UAV fleets that handles the non-stationarity in multi-agent learning. Rosello et al. [13] formulated the MTT problem as a typical motion planning problem and used a MARL method to learn the tracking task. Zhou et al. [14] designed a MAAC-R algorithm using the maximum reciprocal reward to enable multiple UAVs to autonomously track multiple moving targets according to only the past and current position of the UAVs and the target. Notably, the learned policy can directly scale to other complex scenarios without re-training.

For the RL method, agents learn to track through repeated interactions with the environment and other agents and also learn potential cooperative relationships between the agents. Compared with traditional optimization methods, RL can implicitly encode the complex cooperative tracking rules into the neural networks to realize flexible and real-time tracking in an unknown environment through an offline training and online decision framework. However, existing MARL methods have encountered the issues of learning inefficiency, credit assignment, and scalability.

2.3. UAV Swarm Communication

Moreover, communication is the cornerstone of the UAV swarm’s ability to transmit information and achieve cooperation [30,31]. To better realize cooperative target tracking with UAV swarms, many methods has been proposed, including traditional optimization methods [32], learning-based methods [33], etc. More detailed communication examples, such as communication protocol, communication structure, communication object, and communication timing, can be found in Ref. [31].

In summary, different from the above works, our study realizes cooperative and scalable MTT for UAV swarms using a learning-based factored MASAC method. This method requires only the positional and velocity information and the local observation and communication information of the UAVs and targets, and it does not need to acquire the target motion pattern in advance. Moreover, the improved CTDE paradigm enables the efficient cooperative learning and the trained UAV model to be deployed in a distributed manner, with the real-time tracking of moving targets in unknown and dynamic changing environments.

3. Preliminary Analysis

In this section, we give the preliminary evaluation of Dec-POMDP and MASAC, which is the basis of subsequent modeling.

3.1. Dec-POMDP

Here, we consider the UAV swarm cooperative MTT task in which each UAV interacts with the environment only according to its local information (including observation and communication information) to achieve the same team goal. This task can be modeled as a Dec-POMDP $G=\langle I,S,A,P,R,\Omega ,O,n,\gamma \rangle$ [34,35], where $I$ is the set of agents, $S$ is the global state of the environment, $n$ is the total number of agents, and $\gamma \in [0,1]$ is the discount factor. Due to partial observability, at every timestep, each agent $i$ perceives a local observation $o_i\in \Omega$ from the observation function $O(s,i)$ and selects an action $a_i\in A$, forming a joint action $\mathit{a}\in A^n$ leading to the next state $s^{\prime }$, according to the state transition function $P(s^{\prime }|s,\mathit{a})$ and a shared team reward $r=R(s,\mathit{a})$ [26]. Each agent learns a deterministic policy ${\mu }_{{\theta }_i}({\tau }_i)$ or a stochastic policy ${\pi }_{{\theta }_i}(a_i|{\tau }_i)$, conditioned only on the local observation-action history ${\tau }_i$. The joint stochastic policy ${\pi }_{\mathit{\theta }}=\langle {\pi }_{{\theta }_1},\cdots ,{\pi }_{{\theta }_n}\rangle$ induces a joint action-value function $Q_{tot}^{}(\mathit{\tau },\mathit{a})={\mathbb{E}}_{s_{0:\infty },{\mathit{a}}_{0:\infty }}[{\displaystyle {\sum }_{t=0}^{\infty }{\gamma }^{t}R(s_t,{\mathit{a}}_t)}|s_0=s,{\mathit{a}}_0=\mathit{a},{\pi }_{\mathit{\theta }}]$. Similarly, the joint deterministic policy ${\mu }_{\mathit{\theta }}=\langle {\mu }_{{\theta }_1},\cdots ,{\mu }_{{\theta }_n}\rangle$ induces a joint action value function $Q_{tot}^{}(\mathit{\tau },\mathit{a})$.

3.2. MASAC

Soft Actor-Critic (SAC) is an off-policy, actor-critic algorithm in maximum entropy reinforcement learning [36], which simultaneously maximizes the expected return and the entropy of the policy to encourage exploration. SAC optimizes the critic through temporal-difference (TD) learning by minimizing the loss function:

$${ℒ}_Q(\varphi )={\mathbb{E}}_{(s,a,r,s^{\prime })~D}[{(Q_{\varphi }(s,a)-y)}^2]$$

(1)

where $y=r(s,a)+\gamma (Q_{\overline{\varphi }}(s^{\prime },{\pi }_{\overline{\theta }}(a^{\prime }|s^{\prime }))-\alpha \log ({\pi }_{\overline{\theta }}(a^{\prime }|s^{\prime })))$. Here, $D$ is the replay buffer of past experiences, and $\varphi$, $\overline{\varphi }$, and $\overline{\theta }$ are the parameters of the critic, target critic, and target actor, respectively. Notably, SAC adopts a soft action-value function by incorporating an entropy term. The policy is learned by the gradient ascent optimizer:

$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{s~D,a~\pi }[{\nabla }_{\theta }\log ({\pi }_{\theta }(a|s))(\alpha \log ({\pi }_{\theta }(a|s))-Q_{\varphi }(s,a))]$$

(2)

MASAC is an extension of SAC to multi-agent learning that adopts the CTDE paradigm to learn stochastic policies in both discrete and continuous action spaces. All critics are optimized together to minimize the joint loss function through parameter sharing:

$${ℒ}_Q(\varphi )={\displaystyle {\sum }_{i=1}^n{\mathbb{E}}_{(\tau ,a,r,{\tau }^{\prime })~D}[{(Q_i^{\varphi }(\mathit{\tau },\mathit{a})-y_i)}^2]}$$

(3)

where $y_i=r_i+\gamma (Q_i^{\overline{\varphi }}({\mathit{\tau }}^{\prime },{a^{\prime }}_1,\dots ,{a^{\prime }}_n{|}_{{\pi }_{{\overline{\theta }}_i}({a^{\prime }}_i|{{\tau }^{\prime }}_i)})-\alpha \log ({\pi }_{{\overline{\theta }}_i}({a^{\prime }}_i|{{\tau }^{\prime }}_i)))$. Here, $Q_i^{\varphi }$ is the centralized action-value function for agent $i$. The individual policies are updated by gradient ascent with the following loss:

$${\nabla }_{{\theta }_i}J({\pi }_{\theta })={\mathbb{E}}_{\tau ~D,a~\pi }[{\nabla }_{{\theta }_i}\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))(\alpha \log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))-Q_i^{\varphi }(\mathit{\tau },\mathit{a}))]$$

(4)

where all actions are sampled from all agents’ current policies to prevent overgeneralization [24]. However, MASAC cannot scale easily to the large joint action spaces inherent in multi-agent settings, as the joint action space grows exponentially with the number of agents.

4. Problem Formulation

In this section, we first establish a UAV kinematic model and then formulate the UAV-swarm-based MTT task as a Dec-POMDP, as illustrated in Figure 1. UAV swarms are deployed in a task area to search and track the target cooperatively. Each UAV needs to keep a minimum safe distance from others to avoid collision. Additionally, each UAV can be modeled as an agent whose observation and communication ranges are limited. Finally, SER is introduced to help increase the search and tracking coverage.

4.1. UAV Kinematic Model

In an MTT task, the UAV always flies at a fixed altitude [31]. Without a loss of generality, we adopt a two-dimensional kinematic model as follows:

$$\{\begin{array}{l}{x}_i^{t+1}=x_i^t+v_i\cos {\phi }_i^t\Delta t, 0\le x_i^t\le x_{\max }\\ y_i^{t+1}=y_i^t+v_i\sin {\phi }_i^t\Delta t,\hspace{1em}0\le y_i^t\le y_{\max }\\ {\phi }_i^{t+1}={\phi }_i^t+a_i^t\Delta t,\hspace{1em}\hspace{1em} -a_{\max }\le a_i^t\le a_{\max }\end{array}$$

(5)

where $(x_i^t,y_i^t)$, $v_i$, and ${\phi }_i^t$ are the position, velocity, and heading angle of UAV $i$ at time step $t$, respectively. Here, $v_i$ is a constant for simplicity, and $a_i^t$ is the heading angular rate whose maximum value is $a_{\max }$. Therefore, each UAV can be defined by its position and heading $(x_i^{},y_i^{},{\phi }_i^{})$.

4.2. Dec-POMDP Modeling

The Dec-POMDP model includes the design of the observation space, action space, and reward function, which are vital for agents to learn to cooperate.

4.2.1. Observation Space

Considering the limits of the observation and communication range of UAV in practice, we suppose that the UAV’s maximum observation range is $d_o$, and the maximum communication range is $d_c$, as depicted in Figure 2. If the distance between UAV $i$ and $j$ is less than $d_c$, the UAVs can communicate and share information with each other to promote cooperation. The communication information received by UAV $i$ can be denoted as $c_{i,j}=(x_j^{},y_j^{},v_j^x,v_j^y,a_j^{t-1})$, where $(v_j^x,v_j^y)$ is the velocity of UAV $j$ in direction $(x,y)$, and $a_j^{t-1}$ is the UAV’s last action. Similarly, if the distance between UAV $i$ and target $k$ is less than $d_o$, the UAV can perceive the target and acquire the target’s position and velocity information $o_{i,k}=(x_k^T,y_k^T,v_k^{Tx},v_k^{Ty})$. Therefore, the observation space of the UAV is composed of its own position, heading, local observation, and communication information, which is denoted as $o_i=(x_i,y_i,{\phi }_i,o_{i,k},c_{i,j})$.

4.2.2. Action Space

The action space of the UAV consists of the heading angle, according to Section 4.1. To promote learning efficiency, the heading is discretized into $N_a$ discrete values [14], which can be denoted as

$$a=\frac{2n_a-N_a-1}{N_a-1}{\dot{\phi }}_{\max },n_a\in [1,N_a]$$

(6)

4.2.3. Reward Function

In RL settings, the reward function is used to guide agent learning. Therefore, the design of the reward function is particularly critical and can greatly affect the learning speed and final performance of the agent.

In an MTT task, each UAV needs learn to search and track the target, while keeping the target in its observation range as much as possible. In addition, the UAVs should avoid collisions or flying out of the scenario boarder. Therefore, the reward consists of three parts: the tracking reward $r_i^{\mathrm{tracking}}$, the collision penalty $r_i^{\mathrm{collision}}$, and the boundary penalty $r_i^{\mathrm{bound}}$.

The tracking reward can be calculated by

$$r_i^{\mathrm{tracking}}=\underset{k}{\max }(r_{i,k}^{\mathrm{tracking}})$$

(7)

$$r_{i,k}^{\mathrm{tracking}}=\{\begin{array}{l}(d_o-d_{i,k})/d_o+1, d_{i,k}\le d_o\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} d_{i,k}>d_o\end{array}$$

(8)

where $d_{i,k}$ is the distance between UAV $i$ and target $k$. When UAV $i$ detects multiple targets, it only concentrates on tracking the nearest target and obtains the reward $\underset{k}{\max }(r_{i,k}^{\mathrm{tracking}})$.

The collision penalty can be calculated by

$$r_i^{\mathrm{collision}}=\underset{j}{\min }(r_{i,j}^{\mathrm{collision}})$$

(9)

$$r_{i,j}^{\mathrm{collision}}=\{\begin{array}{l}(d_{i,j}-d_s)/d_s-0.5, d_{i,j}\le d_s\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} d_{i,j}>d_s\end{array}$$

(10)

where $d_{i,j}$ is the distance between UAV $i$ and $j$, and $d_s$ is the safe distance needed between UAVs to avoid collision. If $d_{i,j}\le d_s$, the UAV will receive a negative collision penalty to improve safety.

The boundary penalty can be calculated by

$$r_i^{\mathrm{bound}}=\{\begin{array}{l}1-{\mathrm{e}}^{(d_{i,O}-d^{\mathrm{bound}}/2)}, d_{i,O}>d^{\mathrm{bound}}/2\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} d_{i,O}\le d^{\mathrm{bound}}/2\end{array}$$

(11)

where $d_{i,O}$ is the distance between UAV $i$ and scenario center $O$, and $d^{\mathrm{bound}}$ is the scenario boundary length. If UAV $i$ flies out of the boundaries, that UAV will be punished for ineffective searching and tracking.

Because the UAV swarm is cooperative, each UAV shares a global team reward, which can be represented by $r={\displaystyle {\sum }_{i=1}^n(r_i^{\mathrm{tracking}}+r_i^{\mathrm{collision}}+r_i^{\mathrm{bound}})}$.

4.3. Spatial Entropy Reward

Due to insufficient exploration, there are always some targets left without UAVs to track for the MTT task, which means becoming trapped in the local optima. To address this issue, one straightforward idea is to encourage the UAVs to keep a certain distance between each other to increase the detection coverage of the UAV swarm. Inspired by the spatial entropy concept [28], SER is proposed to enlarge the detection coverage of the UAV swarm as much as possible to increase the target detection probability.

First, we provide the definition of spatial entropy. Considering a two-dimensional polar coordinate space $(\rho ,\phi )$, the origin is the position of a UAV. Thus, each position is defined by polar coordinates, where $\rho$ is the distance from the pole, and $\phi$ is the relative angle. Then, the spatial density function is

$$p(\rho ,\phi )=K{e}^{-\lambda \rho }$$

(12)

where $\lambda$ is a parameter related to the mean distance to the pole, and $K$ is a normalizing constant that ensures the following:

$${\displaystyle {\int }_0^{2\pi }{\displaystyle {\int }_0^{\overline{R}}p(\rho ,\phi )\rho d\rho d\phi }}=1$$

(13)

where $\overline{R}$ is the radius. Thus, according to Equations (12) and (13), we obtain

$$K=\frac{\lambda }{2\pi (\frac{1}{\lambda }(1-e^{-\lambda \overline{R}})-e^{-\lambda \overline{R}}\overline{R})}$$

(14)

Then, similar to information entropy, spatial entropy ${\mathscr{H}}_{SE}$ is defined as

$$\begin{array}{l}{\mathscr{H}}_{SE}=-{\displaystyle {\int }_0^{2\pi }{\displaystyle {\int }_0^{\overline{R}}p(\rho ,\phi )\ln p(\rho ,\phi )\rho d\rho d\phi }}\hfill \\ \hspace{1em} =-2\pi K{\displaystyle {\int }_0^{\overline{R}}(\ln K-\lambda \rho )\rho e^{-\lambda \rho }d\rho }\hfill \\ \hspace{1em} =-2\pi K\ln K{\displaystyle {\int }_0^{\overline{R}}\rho e^{-\lambda \rho }d\rho }+2\pi K\lambda {\displaystyle {\int }_0^{\overline{R}}{\rho }^2{e}^{-\lambda \rho }d\rho }\hfill \\ \hspace{1em} =-\ln \lambda +\ln 2\pi +\ln (\frac{1}{\lambda }(1-e^{-\lambda \overline{R}})-e^{-\lambda \overline{R}}\overline{R})+ \frac{\frac{2}{\lambda }-\lambda e^{-\lambda \overline{R}}({\overline{R}}^2+\frac{2\overline{R}}{{\lambda }^2}+\frac{2}{\lambda })}{\frac{1}{\lambda }(1-e^{-\lambda \overline{R}})-e^{-\lambda \overline{R}}\overline{R}}\hfill \end{array}$$

(15)

The evaluation clearly shows that ${\mathscr{H}}_{SE}$ increases monotonously with $\overline{R}$. Therefore, we can control the size of $\overline{R}$ by adjusting the entropy:

$$d_{SE}={\mathscr{H}}_{SE}^{-1}({\mathscr{H}}_1)$$

(16)

where ${\mathscr{H}}_1$ is the expected minimum spatial entropy, and $d_{SE}$ is the equivalent spatial entropy distance.

Based on the definitions of ${\mathscr{H}}_{SE}$ and $d_{SE}$, we propose a concept called SER $r^{SE}$, which can be calculated as follows:

$$r^{SE}=\frac{{\overline{d}}_n-d_s}{d_{SE}-d_s}$$

(17)

where ${\overline{d}}_n=\frac{2}{n(n-1)}{\displaystyle {\sum }_{i\ne j}^n{d}_{i,j}}$ is the average distance of the UAV swarm within the communication range. If ${\overline{d}}_n$ > $d_{SE}$, then $r^{SE}>1$. Therefore, $r^{SE}$ encourages UAVs to keep as far away as possible to increase detection coverage during the search process.

Thus, the final reward function is shaped as

$$r={\displaystyle {\sum }_{i=1}^n(r_i^{\mathrm{tracking}}+r_i^{\mathrm{collision}}+r_i^{\mathrm{boundary}})+\kappa r^{SE}}$$

(18)

where $\kappa =\max (1-n_e/1000,0)$, and $n_e$ is the current episode number. $\kappa$ linearly anneals from one to zero after 1000 episodes to reduce the influence of $r^{SE}$ for tracking performance after a target is searched.

Finally, the objective function of this MTT optimization problem is

$$\underset{{\pi }_{\theta }}{\max }\mathbb{E}[{\displaystyle {\sum }_{t=0}^T{r}_t+\alpha \mathscr{H}({\pi }_{\theta })}]$$

(19)

where $\mathscr{H}({\pi }_{\theta })$ is the entropy of the policy. Next, we will solve this problem in the next section by using a new MARL algorithm.

5. Method

In Section 5, we propose a new algorithm called FMASAC, which incorporates the idea of value decomposition into MASAC. To address the issues of high variance and credit assignment in the cooperative MAAC framework, the centralized critic is factored to assign credit for each agent to improve its policy and thus increase the rewards. Then, factored soft policy iteration is presented to realize off-policy learning for the multi-agent stochastic policy gradient. Finally, the procedure of the FMASAC algorithm is given.

5.1. Learning a Centralized but Factored Critic

Learning a centralized critic conditioning the joint observation and action can be impractical and difficult due to the growth in the number of agents. Therefore, here, we employ the value decomposition in the MASAC framework to enable the scalable learning of a centralized critic in Dec-POMDP. We factor the centralized critic as a linear weighted combination of individual critics across agents:

$$Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)=g_{\omega }(s,{\{Q_i^{{\varphi }_i}({\tau }_i,a_i)\}}_{i=1}^n)$$

(20)

where $\varphi$ and ${\varphi }_i$ are the parameters of the centralized critic $Q_{tot}^{\varphi ,\omega }$ and the agent-wise utilities $Q_i^{{\varphi }_i}$, respectively. Here, $g_{\omega }$ is a linear monotonic function parametrized as a mixing network without the activation function, and $s$ is the global state of the environment. Among these, the linear weighted function increases the expressivity of the network, which combines individual agent utilities into a centralized critic or joint action-value function. Additionally, monotonicity ensures the individual global maximum (IGM) shown in Equation (21), i.e., consistency between the $\arg \max$ of the joint action-value function and the $\arg \max$ of the individual action-value function [37]:

$$\arg \underset{\mathit{a}}{\max }{Q}_{tot}=(\arg \underset{a_1}{\max }{Q}_1,\dots ,\arg \underset{a_n}{\max }{Q}_n)$$

(21)

To evaluate the policy, the critic network is optimized by minimizing the following TD loss:

$${ℒ}_Q(\varphi ,\omega )={\mathbb{E}}_D[{(Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)-y_{tot})}^2]$$

(22)

$$y_{tot}=r+\gamma (Q_{tot}^{\overline{\varphi },\overline{\omega }}({\mathit{\tau }}^{\prime },{a^{\prime }}_1,\dots ,{a^{\prime }}_n{|}_{{\pi }_{{\overline{\theta }}_i}({a^{\prime }}_i|{{\tau }^{\prime }}_i)},s^{\prime })-\alpha \log ({\pi }_{\overline{\mathit{\theta }}}({\mathit{a}}^{\prime }|{\mathit{\tau }}^{\prime })))$$

(23)

$${\pi }_{\overline{\mathit{\theta }}}({\mathit{a}}^{\prime }|{\mathit{\tau }}^{\prime })={\displaystyle {\prod }_{i=1}^n{\pi }_{{\overline{\theta }}_i}({a^{\prime }}_i|{{\tau }^{\prime }}_i)}$$

(24)

where Equation (23) is the target critic. $\overline{\varphi }$, $\overline{\mathit{\theta }}$, and $\overline{\omega }$ are the parameters of the target critic, actors, and mixing network, respectively. Among these, $\overline{\mathit{\theta }}$ represents the joint target actors, which are similar to those in [37].

5.2. Factored Soft Policy Iteration

Then, we leverage the factored critic to train actors in an end-to-end manner to realize decentralized execution. Here, similar to Equation (2), we propose a soft policy iteration with factored maximum entropy trained as follows:

$$\begin{array}{l}ℒ({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}[\alpha \log ({\pi }_{\mathit{\theta }}(\mathit{a}|\mathit{\tau }))-Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)]\hfill \\ \hspace{1em}\hspace{1em} =-g_{\omega }(s,{\mathbb{E}}_{{\pi }_{\theta }}[Q_i^{{\varphi }_i}({\tau }_i,a_i)-\alpha \log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))])\hfill \end{array}$$

(25)

where $g_{\omega }$ is a one-layer monotonic mixing network the same as that in Equation (20). The bias is generated from input $s$. Then, we derive the bias in detail. We adopt the aristocrat utility to assign credits [38], providing the following:

$$\begin{array}{l}{g}_{\omega }(s,{\mathbb{E}}_{{\pi }_{\theta }}[Q_i^{{\varphi }_i}({\tau }_i,a_i)-{\alpha }_i\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))])\\ \hspace{1em}\hspace{1em} ={\displaystyle \sum _i[{\omega }_i(s){\mathbb{E}}_{{\pi }_{\theta }}[Q_i^{{\varphi }_i}({\tau }_i,a_i)]+b(s)-}{\displaystyle \sum _i[{\omega }_i(s){\mathbb{E}}_{{\pi }_{\theta }}[{\alpha }_i\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))]}\\ \hspace{1em}\hspace{1em} \hspace{1em}({\displaystyle \sum _i[{\omega }_i(s){\mathbb{E}}_{{\pi }_{\theta }}[Q_i^{{\varphi }_i}({\tau }_i,a_i)]+b(s)=}{\mathbb{E}}_{{\pi }_{\theta }}[Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)])\\ \hspace{1em}\hspace{1em} ={\mathbb{E}}_{{\pi }_{\theta }}[Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)]-{\displaystyle \sum _i{\mathbb{E}}_{{\pi }_{\theta }}[{\omega }_i(s){\alpha }_i\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))]}\\ \hspace{1em}\hspace{1em} \hspace{1em} (\mathrm{let} \mathit{\alpha }={\omega }_i(s){\alpha }_i)\\ \hspace{1em}\hspace{1em} ={\mathbb{E}}_{{\pi }_{\theta }}[Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)]-{\displaystyle \sum _i{\mathbb{E}}_{{\pi }_{\theta }}[\mathit{\alpha }\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))]}\\ \hspace{1em}\hspace{1em} \hspace{1em} (\mathrm{let} {\pi }_{\mathit{\theta }}=\prod {\pi }_{{\theta }_i},\mathrm{then} {\displaystyle \sum _i{\mathbb{E}}_{{\pi }_{\theta }}\mathit{\alpha }\log ({\pi }_{{\theta }_i}(a_i|{\tau }_i))}={\mathbb{E}}_{{\pi }_{\theta }}\mathit{\alpha }\log ({\pi }_{\mathit{\theta }}(\mathit{a}|\mathit{\tau })))\\ \hspace{1em}\hspace{1em} ={\mathbb{E}}_{{\pi }_{\theta }}[Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)]-{\mathbb{E}}_{{\pi }_{\theta }}[\mathit{\alpha }\log ({\pi }_{\mathit{\theta }}(\mathit{a}|\mathit{\tau }))]\\ \hspace{1em}\hspace{1em} =-{\mathbb{E}}_{{\pi }_{\theta }}[\mathit{\alpha }\log ({\pi }_{\mathit{\theta }}(\mathit{a}|\mathit{\tau }))-Q_{tot}^{\varphi ,\omega }(\mathit{\tau },\mathit{a},s)]\end{array}$$

which then complies with the original SAC policy update.

Similar to the individual temperature parameter automatic adjustment in [36], we apply an automatically adjusted global temperature parameter by considering this adjustment as a constrained optimization problem. We optimize the temperature with the following loss:

$$J(\alpha )={\mathbb{E}}_{a_t\sim {\pi }_t}[-{\alpha }_i\log ({\pi }_{\mathit{\theta }}({\mathit{a}}_t|{\mathit{\tau }}_t))-\mathit{\alpha }{\mathscr{H}}_0]$$

(26)

where ${\mathscr{H}}_0$ is a minimum entropy constraint to guarantee $-\log ({\pi }_{\mathit{\theta }}({\mathit{a}}_t|{\mathit{\tau }}_t)>{\mathscr{H}}_0$, which ensures a minimum randomness of the policy. Additionally, $\alpha$ will decrease if the policy satisfies the entropy constraint; otherwise, $\alpha$ will be increased to enforce the policy to explore. Therefore, the entropy constraint encourages exploration and ensures that the policy convergence is optimal.

5.3. FMASAC-Based MTT of the UAV Swarm

Thus, we propose a cooperative multi-target tracking method with the UAV swarm based on the FMASAC algorithm. The overall algorithm architecture is depicted in Figure 3.

For each agent, there is an actor network that selects action $a_i^t$ conditioning based on its local observations $o_i^t=(x_i^t,y_i^t,{\phi }_i^t,o_{i,k}^t,c_{i,j}^t)$ and last time action $a_i^{t-1}$. There is also one critic network that estimates the $Q$ value of its observation-action $({\tau }_i,a_i)$, which are combined into the joint action-value function $Q_{tot}^{\varphi ,\omega }$ via a linear monotonic mixing function. $Q_{tot}^{\varphi ,\omega }$ is optimized via TD learning, according to Equation (22). Then, each optimized $Q_i^{{\varphi }_i}$ is used to help update each actor network via the factored soft policy iteration, according to Equation (25).

The detailed algorithm procedure is summarized as Algorithm 1, which can be divided into three phases: the initialize phase, the experience collection phase, and the network update phase. First, all parameters are initialized for learning. Then, in the experience collection phase, all UAV agents interact with the environment according to their policies to collect and store experiences. Actor and critic networks are optimized in the network update phase to enable each UAV to learn to cooperate and track the target. After training, each UAV can search and track the target autonomously based on its learned policy, and the UAV swarm can implement cooperative multi-target tracking behavior in an unknown environment.

Algorithm 1: Factored multi-agent soft actor-critic (FMASAC)

# Initialize phase # 1: Initialize critic networks $Q_i^{{\varphi }_i}$, actor networks ${\pi }_{{\theta }_i}$, and mixing network $g_{\omega }$ with random parameters $\varphi$, ${\theta }_i$, $\omega$ 2: Initialize target networks: $\overline{\varphi }=\varphi$, ${\overline{\theta }}_i={\theta }_i$, $\overline{w}=w$ 3: Initialize a replay buffer $D$ 4: for episode = 1 to max_train_episodes $N_e$ do 5:       Reset environment # Experience collection phase # 6:       for $t$ = 0 to max_episode_steps $N_t$ do 7:    For each agent $i$, take action $a_i\sim {\pi }_i(·|{\tau }_i)$ 8:    Execute joint action $\mathit{a}$ 9:    Observe observation ${\mathit{\tau }}^{\prime }$, and reward $r$, done, info 10:        Store $(\mathit{\tau },\mathit{a},r,{\mathit{\tau }}^{\prime })$ in replay buffer $D$ 11:      end for # Actor and critic network update phase # 12:      for $t$ = 1 to $T$ do 13:    Sample a minibatch trajectory from $D$ 14:    Calculate ${ℒ}_Q(\varphi ,\omega )$ 15:    Update critic networks and mixing network w.r.t. Equation (22) 16:      Update decentralized policies using the gradients ${\nabla }_{{\theta }_i}J({\pi }_{\theta })$ w.r.t. Equation (25) 17:    Update temperature parameter $\alpha$ w.r.t. Equation (26) 18:    if update target network then 19:     $\overline{\varphi }=\varphi$, ${\overline{\theta }}_i={\theta }_i$, $\overline{w}=w$ 20:    end if 21:      end for 22:end for 23:Return π

6. Experiments

In this section, we evaluate the proposed FMASAC algorithm using simulation experiments and compare it with three popular multi-agent actor-critic algorithms. The effectiveness, generalization, and scalability of the FMASAC algorithm are then validated. The simulation results show that the proposed algorithm can realize cooperative multi-target tracking while avoiding collision; the results outperform the baselines in terms of the mean reward and tracking success rate.

6.1. Parameter Setup

For this study, we developed a two-dimensional UAV swarm multi-target tracking simulation environment, which is developed based on commonly used tools, including Linux (Ubuntu 20.04) or MacOS, Pycharm, and Multi-Agent Particle Environment [22]. The user interface can be customized and modified. The number of UAVs and targets and the map size are variable and can be modified according to demand. The positions of the UAVs and targets are initialized randomly for each episode. The environmental parameters are configured as shown in

Table 3. Environmental parameters.

Entity	Physical Meaning	Notation	Value
Environment	Size		2000 m × 2000 m
	Scenario boundary length	$d^{\mathrm{bound}}$	2000 m
UAV	Observation range	$d_o$	200 m
	Communication range	$d_c$	800 m
	Safe distance	$d_s$	400 m
	Speed	$v_i$	60 m/s
	Maximum heading angular rate	$a_{\max }$	$\pi$/6 rad/s
Target	Speed	$v_k^T$	40 m/s

.

Next, we test the performance of the proposed FMASAC algorithm in the scenario with three UAVs and three targets and compare it with three baselines: the MASAC [39], multi-agent proximal policy optimization (MAPPO) [40], and multi-agent deep deterministic policy gradient (MADDPG) [22] algorithms. The algorithm hyperparameter settings are shown in

Table 4. Hyperparameter settings.

Hyperparameter	Value
Episode limit	100
Max step	1,005,000
Buffer size	5000
Minibatch size	64
Target update interval	200
Actor learning rate	0.0001
Critic learning rate	0.0005
TD lambda	0.6
Gamma	0.99
Minimum policy entropy	−1
Equivalent spatial entropy distance	500

, and the hyperparameters of the baselines are fine-tuned according to the information in [41]. For the sake of fairness, each algorithm is run with five random seeds, and the results are shown with 95% confidence intervals.

6.2. Effectiveness

First, each algorithm was trained for 1,005,000 steps until reaching convergence. To intuitively understand whether the UAVs learned how to track the targets, the tracking process of each UAV in the FMASAC algorithm is visualized (Figure 4). Figure 4 shows that each UAV learned to autonomously and cooperatively search and track a target.

Next, we plotted the mean reward and relative distances between the UAVs and the UAV target and the tracking success rate during training to measure the performance of the four algorithms. Successful tracking is determined by more than 70 tracking steps in an episode. The tracking success rate is defined as the ratio of the number of successful tracking times to the total tracking times every 80 episodes. Figure 5 shows the mean reward learning curve of the four algorithms.

Figure 5 shows that the performance of FMASAC is better than that of the other three baseline algorithms in terms of the final mean reward and convergence speed, which means that the factored critic and soft policy iterations are efficient. Compared to the vanilla MASAC algorithm, our FMASAC algorithm behaves more stably and has a lower policy variance, leading to improved performance, while the MASAC suffers from policy collapse. It can be inferred that the MASAC satisfies the conflicts in the policy update, as greedily maximizing the team reward may exacerbate the credit assignment issue, and the exploration or sub-optimality of some agents’ policies might aggravate the training of other agents through the centralized critic. Therefore, learning the factored critic implicitly realizes multi-agent credit assignment and efficient learning because the individual critic assigns credit information for each agent to improve its policy.

To further demonstrate the effectiveness and contribution of each core component in FMASAC, we conducted ablation studies as shown in Figure 6 to compare FMASAC, FMASAC using a fixed alpha named FMASAC_Fixed_α, and FMASAC without SER, named FMASAC_NoSER.

It can be seen from Figure 6 that the performance of the full version FMASAC is better than that of FMASAC_Fixed_α and the no $r^{SE}$ version FMASAC_NoSER. This result illustrates that automatic temperature adjustment parameter $\alpha$ can improve policy performance by balancing exploration and exploitation. Notably, SER significantly increases the stability and final performance of the policy by encouraging the expansion of the detection coverage, which objectively verifies the importance and effectiveness of SER.

Next, we analyze the relative distances between the UAVs and the UAV-target and tracking success rates during training for this tracking task. The results are shown in Figure 7.

As illustrated in Figure 7a, the relative distances between each UAV and the target are less than the UAV observation range after 0.3 million training steps, which means that each UAV can search and track a target successfully. Figure 7b shows that the relative distances between UAVs are larger than the UAV–UAV safe distance after training. Additionally, the UAVs learn to avoid collisions while tracking, and the distances between UAVs are smaller than the UAV communication range to better communicate mutual information. The most significant indicator in the MTT task, the tracking success rate, is depicted in Figure 7c. Here, the tracking success rate of each UAV during training reaches 1 after about 0.3 million steps. Therefore, three UAVs learned to autonomously search and track targets while avoiding collisions, which demonstrates that our proposed FMASAC algorithm is effective in the MTT task.

6.3. Generalization

Next, we load the trained policy model (actor network) without further fine-tuning to test the generalization of the model in unknown and dynamic changing environments. Here, all positions of the UAVs and targets are randomly initialized. The test results of one episode are shown in Figure 8.

As can be seen from Figure 8a, each UAV can quickly track a target after 10 steps in an unknown environment because the UAV policy model has seen hundreds and thousands of scenarios during training and has memorized the neural network parameters. Meanwhile, as shown in Figure 8b, the relative distances between UAVs are larger than the safe distance, while maintaining effective communication. In conclusion, the learned policy model can directly generalize to an unknown environment without fine-tuning or re-training, which offers efficient and real-time results.

Figure 9 shows the relative distances and test tracking success rates of 100 tests. We can see from the violin plot of relative distances that the UAV–target distances are almost less than the UAV observation range. This result indicates that every target is tracked by the trained UAV model, even in an unknown scenario. Moreover, the tracking success rate for each UAV during testing is greater than 98%, which indicates that the trained policy can be generalized to unknown scenarios.

6.4. Scalability

To further test the scalability of the proposed FMASAC algorithm in large-scale UAV swarm scenarios, we also performed experiments on five UAVs and five targets, as well as ten UAVs and ten targets. Figure 10 shows the change trend of the average tracked targets and tracking success rates of different algorithms as the number of UAVs increases. The detailed results of the different algorithms are shown in

Table 5. Comparison of results in different scenarios.

Scenarios	Map Size (km)	Indicators	FMASAC	MASAC	MAPPO	MADDPG
3 UAVs and 3 targets	2 × 2	Mean reward	172.63	144.54	160.51	54.28
		Reward standard deviation	1.11	18.84	1.26	8.07
		Average tracked targets	3	2.44	2.96	1.05
		Tracking success rate	100%	81.25%	98.75%	35%
5 UAVs and 5 targets	5 × 5	Mean reward	315.23	289.72	308.88	124.36
		Reward standard deviation	1.32	16.95	1.45	9.87
		Average tracked targets	4.88	4.19	4.81	1.56
		Tracking success rate	97.5%	83.75%	96.25%	31.25%
10 UAVs and 10 targets	10 × 10	Mean reward	611.97	457.42	543.83	277.19
		Reward standard deviation	1.64	23.63	1.58	13.42
		Average tracked targets	9.5	7.38	8.63	4.13
		Tracking success rate	95%	73.75%	86.25%	41.25%

.

It can be seen from Table 5 that the proposed FMASAC algorithm has the highest mean reward and the lowest standard deviation in almost three scenarios. Additionally, the average tracked targets and tracking success rates during training were better than those of the other baselines. Therefore, the performance of FMASAC is better than that of the other baselines in the three scenarios. This result verified that the factored critic and soft policy iteration reduced the complexity of the joint action-value function and enabled the scalable learning of CTDE paradigms by combining individual utilities conditioned on local information. Therefore, we can conclude that the FMASAC algorithm scales better when the number of agents and joint action spaces increase.

7. Conclusions

This paper studied the problem of multi-target tracking with a UAV swarm in an unknown environment. An end-to-end MARL scheme named FMASAC was proposed to enable UAV swarm cooperative multi-target tracking in a distributed manner. By learning a centralized-but-factored critic and factored soft policy iteration, the algorithm learning efficiency was improved, and the variance in policy updates was reduced. SER increased the detection coverage of the UAV swarm and improved cooperative performance in the task. The simulation results show that the proposed FMASAC algorithm can realize cooperative multi-target tracking and significantly outperform the three popular MARL baselines. Additionally, FMASAC can generalize and scale to unknown and large-scale scenarios.

In the future, we will continue to study the cooperative multi-target tracking method for heterogeneous nonlinear swarms with individual rewards in unknown environments [42,43]. In addition, more complicated real-world scenarios and accurate UAV/HFV models will be considered [44].