Multi-UAV Path Planning and Following Based on Multi-Agent Reinforcement Learning

Abstract

Dedicated to meeting the growing demand for multi-agent collaboration in complex scenarios, this paper introduces a parameter-sharing off-policy multi-agent path planning and the following approach. Current multi-agent path planning predominantly relies on grid-based maps, whereas our proposed approach utilizes laser scan data as input, providing a closer simulation of real-world applications. In this approach, the unmanned aerial vehicle (UAV) uses the soft actor–critic (SAC) algorithm as a planner and trains its policy to converge. This policy enables end-to-end processing of laser scan data, guiding the UAV to avoid obstacles and reach the goal. At the same time, the planner incorporates paths generated by a sampling-based method as following points. The following points are continuously updated as the UAV progresses. Multi-UAV path planning tasks are facilitated, and policy convergence is accelerated through sharing experiences among agents. To address the challenge of UAVs that are initially stationary and overly cautious near the goal, a reward function is designed to encourage UAV movement. Additionally, a multi-UAV simulation environment is established to simulate real-world UAV scenarios to support training and validation of the proposed approach. The simulation results highlight the effectiveness of the presented approach in both the training process and task performance. The presented algorithm achieves an 80% success rate to guarantee that three UAVs reach the goal points.

1. Introduction

The autonomous capabilities of unmanned aerial vehicles (UAVs) have been improved dramatically in recent years, because of the fast development of sensor technology, communication and control technique [1]. At present, UAVs are widely used in many fields, including industry, agriculture, search and rescue, environmental monitoring and transportation [2]. However, a single UAV can only accomplish some restricted missions, the reason being that a single UAV has limited endurance, payload capacity and working range. For the problems in complex scenarios, it is necessary for multiple UAVs to work together. Multiple UAVs can cooperate with each other, in order to provide strong robustness, fault tolerance and flexibility.

To realize the autonomous operation of multiple UAVs, many related technologies need to be developed, including collaborative navigation, planning and control. Multi-UAV path planning and the following techniques are very important, and ensure that multiple UAVs do not collide with obstacles while moving within the work area. A great multi-UAV path planning and the following algorithm enable each UAV to safely and efficiently navigate from its starting position to its destination. Other related applications, such as multi-robot cooperation, also require multi-agent path planning and following technique [3]. Path planning is an NP-hard problem [4]. Multi-UAV path planning builds upon the foundations laid by single-UAV path planning. Numerous research efforts have addressed the challenges of single UAV path planning, employing classical techniques, heuristics, and machine learning approaches. Mechali et al. [5] proposed an enhanced rapidly exploring random tree (RRT) algorithm for UAV, incorporating a technique to smooth and rectify the resultant path. This refinement aims to minimize energy consumption during the flight. Chen et al. [6] applied the A* algorithm to UAV path planning under the constraints of bearing the least danger and consuming the least fuel. Wu et al. [7] introduced an enhanced deep Q-network model by integrating the lazy training method to optimize UAV path planning.

Multi-agent path planning and the following problem have attracted the attention of many researchers [8,9,10]. In [11], a randomized search method with hill climbing was proposed, which minimizes the overall path lengths of the multi-robot system. In [12], a guided iterative prioritized planning algorithm providing the shortest time cost was derived for addressing the path planning problem of the multiple mobile agents. Based on traditional RRT, a decentralized multi-agent RRT algorithm was developed in [13], and the complex environmental constraint was considered at the same time. Intelligent optimization algorithms have also been applied to the multi-agent path planning and the following problem. In [14], an innovative artificial potential field algorithm was applied to get all feasible paths in a discrete gridded environment; then, an enhanced genetic algorithm was developed to find the optimal path. In [15], the unknown obstacle-rich scenes were considered, and a decentralized algorithm for multi-robot autonomous navigation was derived. The path planning and formation control problem for multi-UAV systems was studied in [16], and an improved artificial potential function-based path planning approach was proposed. In [17], based on distributed model predictive control, a distributed real-time search path planning method for a multi-UAV system was proposed. The avoiding collision requirement was considered in [18], and the multi-agent path planning problem in a road network was investigated. To deal with the multi-robot transporting problem, an integrated task assignment and path planning algorithm was proposed in [19]. In [20], a multi-strategy fusion differential evolution algorithm was proposed for the path planning problem of UAVs in a complex environment. In [21], a novel multi-UAV distributed online cooperation coverage path planning method was proposed, which was utilized to minimize task completion time. The multi-agent pathfinding problem was addressed in [22], and a prioritized safe interval path planning method with continuous time conflicts was proposed.

In recent years, the deep reinforcement learning (DRL) method and multi-agent deep reinforcement learning method [23] have been applied to deal with task allocation, link selection or path planning problems of the multi-agent system and autonomous vehicles [24]. In [25], the optimal links discovery and selection problem of the multi-UAV system was addressed through DRL, in which the continuous action spaces were sliced into a number of tractable fractions. A multi-agent mutual sampling method was proposed in [25], in order to stabilize and expedite the training procedure. The multi-UAV path planning problem can be modeled as a multi-agent reinforcement learning problem. However, independent Q-learning introduces the non-stationarity problem. In [26], a multi-agent variant of importance sampling was developed, in order to solve the non-stationarity problem to some extent. To realize coordinated control of the multi-UAV system, a distributed reinforcement learning approach was proposed in [27], which showed robust communication channel impairments. In [28], in order to deal with complex interaction and time-varying communication topology problems, an attention-enhanced reinforcement learning method was proposed for multi-agent cooperation. In [29], a virtual learning objective was introduced, and a cooperative reinforcement learning method with two layers was proposed for multi-agent systems. Deep reinforcement learning was applied in [30] to address the decentralized multi-agent pursuing issue. The multi-agent active search problem was considered in [31], and a Poisson point process formulation was introduced; then, the active search problem was turned into a reinforcement learning problem. In [32], reinforcement and imitation learning methods were combined, and a novel framework for multi-agent path finding was proposed. The safe multi-agent reinforcement learning method for multi-robot control was investigated in [33], where the neighbor’s safety constraints were considered to guarantee safe team behaviors.

Previous studies have primarily based their research on grid-based maps, while our approach innovates by using laser scan data as input. This choice enables a more accurate simulation of real-world applications. Unlike previous algorithms, our approach is more lightweight and does not include a communication mechanism. This design choice is deliberate, as our experiment is situated in a setting with continuous actions, and agents rely solely on laser scan data for observations. In this realistic scenario, sharing laser scan data among agents does not contribute to policy convergence or decision making. Consequently, our approach avoids integrating a communication mechanism to avoid additional computational costs. In this paper, we present a multi-UAV path planning and following approach based on multi-agent reinforcement learning. The UAV employs the SAC algorithm as a planner, training the policy to converge. The policy enables end-to-end processing of laser scan data, guiding the UAV to avoid obstacles and reach the goal. Other approaches often rely on traditional numerical optimization techniques that require the UAV’s environment to be first constructed using SLAM before path planning can be performed. Simultaneously, the planner utilizes paths generated through a sampling-based method as following points. As the UAV progresses, the following points are continually updated. Multi-UAV path planning tasks are facilitated, and policy convergence is accelerated through shared experiences among agents. The effectiveness of this approach is verified in a randomized map environment. The main contributions of this paper include:

• Establishing a multi-UAV simulation environment to train and validate the proposed approach. It involves individual interactions between agents and the environment, avoiding joint action and state spaces. This prevents exponential parameter growth with an increasing number of agents and maintains linearity.
• Proposing a parameter-sharing off-policy multi-agent path planning and following an approach that leverages interaction data from all agents. Experimental simulation results validate its effectiveness.
• Designing a reward function that encourages UAV movement, addressing the problem of UAVs remaining stationary in the initial phase and exhibiting overly cautious and minimal actions when approaching the goal.

The structure of this paper is as follows: Section 2 provides a detailed explanation of the parameter-sharing multi-agent path planning and the following approach. Section 3 provides details on simulation experiments and performance analysis. Finally, conclusions are summarized and future research is also noted in Section 4.

Part of the material has been used in our conference paper [34], which proposed a path planning and following method for a single quadrotor UAV. We believe that this paper is a sufficient extension of our previous work [34]. We extended our previous algorithm to a multi-UAV scenario based on multi-agent reinforcement learning, and completely new training and simulation results are shown in this paper.

2. Approach

In this chapter, we combine the SAC algorithm with a sampling-based path planning algorithm adaptively informed trees (AIT*) [35] to generate a path for the following. The path is partitioned into the following points, which are incorporated into the design of the state and reward functions, guiding the UAV to reach the goal more efficiently and effectively. Additionally, we extend the application to multiple agents, enabling multi-UAV path planning in task scenarios. Inspired by shared experience actor–critic (SEAC) [36] for multi-agent reinforcement learning, we expedite exploration, learning, and policy convergence by sharing experiences among agents.

2.1. Soft Actor–Critic

DRL is a critical machine learning tool primarily focusing on addressing sequential decision-making problems. Its learning process can be described as Markov decision process (MDP) [37]. The agent’s behavior is determined by the policy $\pi (·|\mathit{s}),$ which is stochastic and generates a vector consisting of the probabilities associated with selecting each possible action. At each time step t, the agent is in a given state ${\mathit{s}}_t$. ${\mathit{a}}_t$ is performed by the agent at time step t, which obeys the output distribution of policy $\pi$ in state ${\mathit{s}}_t$. Subsequently the agent is given a new state ${\mathit{s}}_{t+1}$ and receives an reward signal $r({\mathit{s}}_t,{\mathit{a}}_t)$. During interactions, the agent receives immediate rewards, using them to evaluate its actions. In [38], Yang compared the SAC algorithm with deep deterministic policy gradients (DDPG) [39] and twin-delayed deep deterministic (TD3) [40] algorithms in the quadrotor obstacle avoidance task. Simulation results show that SAC outperforms DDPG and TD3 in terms of stability and performance. Thus, we choose SAC to optimize the policy network $\pi$. SAC is an off-policy actor–critic algorithm based on the maximum entropy RL framework. The learning goal is to maximize the following objective function:

$$J\left(\pi \right)=\sum _{t=0}^T{\mathbb{E}}_{({\mathit{s}}_t,{\mathit{a}}_t)\sim {\rho }^{\pi }}\left[r({\mathit{s}}_t,{\mathit{a}}_t)+\alpha H\left(\pi (·|{\mathit{s}}_t)\right)\right],$$

(1)

where ${\rho }^{\pi }$ denotes state-action marginals of the trajectory distribution induced by a policy $\pi (\mathit{s},\mathit{a}),$ and the temperature parameter $\alpha$ controls the importance of entropy relative to the reward. Choosing the optimal $\alpha$ is non-trivial and must be tuned for each task. Rather than manually setting the $\alpha$, we automate the process by formulating different maximum entropy reinforcement learning goals, where entropy is considered a constraint [41]. We set the initial value of $\alpha$ to 0.2, which tends to decrease toward 0 as training progresses, as shown in Figure 1. $H\left(\pi \right(·|\mathit{s}\left)\right)$ is the policy entropy, which measures the dispersion of the output action distribution under a policy $\pi$ in the state $\mathit{s}$, and is computed as $H\left(\pi \left(·|\mathit{s}\right)\right)={\mathbb{E}}_{\mathit{a}\sim \pi (·|\mathit{s})}\left[-log\pi \left(\mathit{a}\right|\mathit{s})\right]$.

The maximum entropy term is a critical component [42]. The primary purpose is not to discard any useful action. It promotes exploration, prevents local optima, and enhances robustness in the agent’s behavior. It is well-suited for addressing the problems of model-free DRL, including high sample complexity and unstable convergence properties.

The state-value function $V\left({\mathit{s}}_t\right)$ and the action-value function $Q({\mathit{s}}_t,{\mathit{a}}_t)$ are used to evaluate the quality of the policy $\pi$. In the continuous state setting, we can consider a parameterized state value function $V_{\psi }\left({\mathit{s}}_t\right)$, soft Q-function $Q_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)$, and a tractable policy ${\pi }_{\varphi }({\mathit{s}}_t,{\mathit{a}}_t)$. The parameters of these networks are $\psi$, $\theta$ and $\varphi$, respectively. For instance, the value functions can be modeled as expressive neural networks, and the policy as a Gaussian with mean and covariance given by neural networks [43]. In practice, including a distinct function approximator $V_{\overline{\psi }}\left({\mathit{s}}_t\right)$ for the soft value has the dual benefit of enhancing training stability and facilitating concurrent training alongside the other networks.

The soft state-value function is trained to minimize the objective function [43]:

$$J_V\left(\psi \right)={\mathbb{E}}_{{\mathit{s}}_t\sim \mathcal{D}}\left[\frac{1}{2}{(V_{\psi }\left({\mathit{s}}_t\right)-{\mathbb{E}}_{{\mathit{a}}_t\sim {\pi }_{\varphi }}[Q_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)-log{\pi }_{\varphi }\left({\mathit{a}}_t\right|{\mathit{s}}_t)])}^2\right],$$

(2)

where $\mathcal{D}$ is a replay buffer. The gradient of Equation (2) is estimated with an unbiased estimator and used for optimizing itself,

$${\widehat{\nabla }}_{\psi }{J}_V\left(\psi \right)={\nabla }_{\psi }{V}_{\psi }\left({\mathit{s}}_t\right)(V_{\psi }\left({\mathit{s}}_t\right)-Q_{\theta }\left({\mathit{s}}_t,{\mathit{a}}_t\right)+log{\pi }_{\varphi }\left(\mathit{a}\right|\mathit{s})).$$

(3)

The soft Q-function is trained to minimize the objective function [43]:

$$J_Q\left(\theta \right)={\mathbb{E}}_{\left({\mathit{s}}_t,{\mathit{a}}_t\right)\sim \mathcal{D}}\left[\frac{1}{2}{(Q_{\theta }\left({\mathit{s}}_t,{\mathit{a}}_t\right)-\widehat{Q}({\mathit{s}}_t,{\mathit{a}}_t))}^2\right],$$

(4)

with

$$\widehat{Q}({\mathit{s}}_t,{\mathit{a}}_t)=r({\mathit{s}}_t,{\mathit{a}}_t)+\gamma {\mathbb{E}}_{{\mathit{s}}_{t+1}\sim p}\left[V_{\overline{\psi }}\left({\mathit{s}}_{t+1}\right)\right].$$

(5)

Again, the gradient of Equation (4) is:

$${\widehat{\nabla }}_{\theta }{J}_Q\left(\theta \right)={\nabla }_{\theta }{Q}_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)(Q_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)-r({\mathit{s}}_t,{\mathit{a}}_t)-\gamma V_{\overline{\psi }}\left({\mathit{s}}_{t+1}\right)).$$

(6)

SAC algorithm employs the Kullback–Leibler (KL) divergence [44] to measure the policy’s similarity before and after parameter updates. The actor updates the policy network by minimizing the following objective function:

$$J_{\pi }\left(\varphi \right)={\mathbb{E}}_{{\mathit{s}}_t\sim \mathcal{D}}\left[D_{KL}\left({\pi }_{\varphi }(·|{\mathit{s}}_t)\mid \frac{\exp \left(Q_{\theta }({\mathit{s}}_t,·)\right)}{Z_{\theta }\left({\mathit{s}}_t\right)}\right)\right].$$

(7)

The partition function $Z_{\theta }(·)$ serves to normalize the distribution. While it is usually intractable, it has no impact on the gradient concerning the new policy and can, therefore, be disregarded [43].

Finally, multiple consecutive updates of all the networks can be performed within a single step of environmental sampling. $\lambda$ is a learning rate and can be set to different values according to different network parameters. These learning rates and the weighting factor $\tau$ used for updating the target soft state-value network are hyperparameters that need to be tuned. A simplified pseudo-code for SAC is illustrated in Algorithm 1.

Algorithm 1 Soft actor–critic

1: Initialize parameters vectors $\psi , \overline{\psi }, \theta , \varphi$;   2: for each iteration do   3:     for each environment step do   4:         ${\mathit{a}}_t\sim {\pi }_{\varphi }\left({\mathit{a}}_t\right|{\mathit{s}}_t)$   5:         ${\mathit{s}}_{t+1}\sim p\left({\mathit{s}}_{t+1}\right|{\mathit{s}}_t,{\mathit{a}}_t)$   6:         $\mathcal{D}\leftarrow \mathcal{D}\cup \{({\mathit{s}}_t,{\mathit{a}}_t,r({\mathit{s}}_t,{\mathit{a}}_t),{\mathit{s}}_{t+1})$ }   7:     end for   8:     for each gradient step do   9:         $\psi \leftarrow \psi -\lambda {\widehat{\nabla }}_{\psi }{J}_V\left(\psi \right)$ 10:         ${\theta }_i\leftarrow {\theta }_i-\lambda {\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right)$ for $i\in 1,2$ 11:         $\widehat{\varphi }\leftarrow \varphi -\lambda {\widehat{\nabla }}_{\varphi }{J}_{\pi }\left(\varphi \right)$ 12:         $\overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi }$ 13:     end for 14: end for

2.2. Network Structure

The network structure of the hybrid path planning and the following algorithm is shown in Figure 2. The neural network architecture for the SAC algorithm consists of both actor and critic networks. The actor network comprises two fully connected(FC) layers with 256 units each, responsible for action generation. The critic network, represented by the Q-net, features two identical Q-networks, Q1 and Q2. These Q-networks are multi-layered, with 256 units in the first two FC layers and an output layer for Q-value estimation. These networks work together to evaluate actions generated by the actor for reinforcement learning tasks.

The state space consists of laser data, the UAV’s velocity, position relative to the goal, and a following point. For safety and practical physical reasons [45,46], we limit the absolute values of the UAV’s forward and angular velocities to within 1 m/s. At the same time, we aim to minimize acceleration to ensure that the UAV can smoothly adjust its orientation and perform maneuvers. Note that in our approach we mainly consider kinematic aspects to control the motion of the UAV. Compared to [38], we incorporate the following point into the state. As mentioned earlier, sampling-based path planning algorithms are well suited for obtaining a predefined path that the DRL-based planner can follow. AIT* [35] outperforms existing sampling-based algorithms when addressing the tested abstract problems. It finds an initial solution quickly and converges to the optimum in an anytime manner. Therefore, AIT* is chosen for generating the predefined path. We present an innovative hybrid approach for path planning and following, integrating SAC-based path planning with path-following techniques. Initially, a path is generated using AIT*, which then serves as the following points for the planner. This approach guides the agent to move more efficiently toward the goal, with the following points continuously updated in real time.

2.3. Multi-Agent Framework

Due to the homogeneity of tasks that multiple agents need to perform, we can extend this algorithm to multi-agent scenarios. A straightforward approach, such as independent Q-learning (IQL) [47], is to treat the other agents as part of the environment and allow each agent to learn its policy independently. However, this approach leads to uncooperative behavior among agents, and the interference between different agents’ policies introduces uncertainty regarding the convergence of their policies during training.

Introducing a communication mechanism can enhance IQL and foster cooperation among agents [48,49]. In scenarios with grid-based maps and discrete action spaces, agents can communicate to exchange locally observed state maps, leading to a better understanding of the environment and decision making. In contrast, our experiment involves a more realistic setting with continuous actions, where agents’ observations consist only of laser scan data. Sharing these data among agents provides no assistance in policy convergence and decision-making. Therefore, our algorithm does not introduce a communication mechanism to avoid additional computational costs. Agents treat each other as part of the environment. When another agent enters the laser range of a given agent, it can immediately perceive this through laser data. By avoiding explicit communication mechanisms, our algorithm presents a more computationally efficient and lightweight solution, providing practical insights for real-world multi-agent systems.

To facilitate the convergence of different agent policies, we enable efficient learning via experience sharing. Unlike IQL, we introduce an experience replay mechanism to train and promote the convergence of agent policies. A step transition $({\mathit{s}}_t,{\mathit{a}}_t,r_t,{\mathit{s}}_{t+1},done)$ is stored in the replay buffer, serving as experience. Here, $done$ is a boolean value indicating whether the current step signifies the end of an episode. The replay buffer, a key component, is designed to store the experiences of individual agents. This experience-sharing mechanism assumes an environment where local policy gradients of agents provide valuable learning directions for all agents. Essentially, this implies that agents can benefit from the experiences of their peers, even in scenarios where their targets and behavior policies are different. This collaborative learning approach fosters a collective intelligence among agents, promoting adaptability and efficiency in navigating complex environments. Previous research [49] suggests that utilizing the shortest path distances from all agents to each agent’s goal significantly enhances goal-oriented learning in partially observable environments. We leverage changes in the shortest distances and constraints on path-following points to guide not only the achievement of single-agent goals but also to improve collision avoidance cooperation between agents. The experience replay mechanism effectively utilizes training data from all agents, providing useful policy gradients for all. Additionally, we perform importance sampling, allowing agents to selectively choose samples from previous experiences rather than uniformly random sampling. One of the benefits of this mechanism is breaking the temporal correlation among data, enabling the model to adapt better to different situations. We define the sampling probability for the i-th transition as follows [38]:

$$P_i=\frac{p_i^{\beta }}{{\sum }_k{p}_k^{\beta }},$$

(8)

where the parameter $\beta$ serves as a factor regulating the impact of priority on sampling, $\beta \in [0,1]$. A larger $\beta$ tends to favor sampling based on priority, while a smaller $\beta$ leans toward uniform sampling. Specifically, Tom et al. suggest prioritizing the replay of transitions with higher expected learning progress, determined by the magnitude of their temporal difference (TD) error [50]. The priority $p_i$ of the i-th transition is calculated by:

$$p_i={\delta }_i^2+\eta ,$$

(9)

where $\delta$ denotes the TD error of the Q-function network and $\eta$ represents a small positive number, ensuring that all transitions have a non-zero probability of being sampled. In summary, we illustrate the model architecture of our proposed algorithm based on the multi-agent soft actor–critic framework in Figure 3.

2.4. Reward Function Design

A reward function $r_{total}$ has been designed to facilitate path following. $r_{total}$ is composed of the following elements:

$$r_{total}=r_{dis}+r_{follow}+r_{crash}+r_{free}+r_{step}+r_{move},$$

(10)

where $r_{dis}$ denotes the distance reward, $r_{follow}$ denotes the following reward, $r_{crash}$ denotes the collision reward, $r_{free}$ denotes the free space reward, and $r_{step}$ denotes the step reward. $r_{move}$ denotes the move reward. In the current setup, the reward function includes multiple sub-functions without explicit weights, incorporating a form of intelligence that avoids manual tuning of weight parameters. While the experiments show good results without explicit weights, users can customize the weights according to specific preferences. The expression for the distance reward $r_{dis}$ is:

$$r_{dis}=\left\{\begin{array}{cc}{r}_{arrival}\hfill & \mathrm{if}{d}_g<d_{gmin}\\ \Delta d_g\hfill & \mathrm{otherwise},\end{array}\right.$$

(11)

where $d_g$ represents the distance between the UAV and the goal. When $d_g$ is less than the threshold $d_{gmin}$, it is considered as reaching the goal, and a reward value $r_{arrival}$ is obtained. Otherwise, a reward value related to the distance change within this time step is obtained. The formula for following reward $r_{follow}$ is:

$$r_{follow}=-d_{P_{follow}},$$

(12)

where $d_{P_{follow}}$ is the distance from the UAV to $P_{follow}$, which is the following point closest to the UAV along the predefined path. Note that $r_{dis}$ depends on the changing value of the distance, while $r_{follow}$ depends directly on the distance itself. This approach helps maintain the UAV’s proximity to the path and focuses its interaction with the environment in the vicinity of the path. Consequently, this strategy yields a greater number of high-quality interaction sequences, ultimately enhancing the policy’s performance. The formula for the collision reward $r_{crash}$ is:

$$r_{crash}=-exp(-\frac{d_{ro}}{\tilde{r}}),$$

(13)

where $d_{ro}$ is the distance between the UAV and the nearest obstacle. $\tilde{r}$ is a hyperparameter, and $\tilde{r}\in {\mathbb{R}}^{+}$. As a UAV approaches obstacles, its laser scan detects shorter distances, resulting in reduced feedback rewards from environmental interactions. This mechanism guides the UAV to maintain a distance from obstacles, effectively avoiding collisions. To enhance the UAV’s avoidance of obstacles, the free space reward $r_{free}$ is defined as:

$$r_{free}=-\sum _{i}exp(-\frac{(d_i-d_{min})}{(d_{max}-d_{min})\overline{r}}),$$

(14)

where $d_i$ represents the i-th data of the LiDAR, and $d_{max}$ and $d_{min}$ represent its maximum and minimum values, respectively. $\overline{r}$ is a hyperparameter, and $\overline{r}\in {\mathbb{R}}^{+}$. The value of $r_{free}$ diminishes only when the UAV is in close proximity to an obstacle, and it approaches zero rapidly as the UAV moves far from the obstacle. This motivates the UAV to explore narrow spaces. To further promote swift target attainment, we introduce a step reward $r_{step}=-T$. As time T increases, $r_{step}$ becomes more punitive. During the experimental process, we observed that the UAVs tended to remain stationary in the initial stages and exhibited cautious behavior with minimal movement when approaching the goal. To address these issues, a reward function was designed to encourage UAV mobility. The move reward $r_{move}$ is defined as:

$$r_{move}=d_{move},$$

(15)

where $d_{move}$ represents the distance moved from the old state to the new state. Introducing a move reward can incentivize the agent to advance, thereby further reducing the episode time.

3. Experiment and Result

In this section, a simulation environment is established based on the robot operating system (ROS) and Rviz (a visualization tool). We use it to simulate multiple quadrotor UAVs performing a multi-agent path planning task. The proposed path planning and the following algorithm are trained and tested in the simulation environment. We will compare the proposed method with traditional methods and analyze the performance of the proposed method. In all the simulation results in this section, ‘baseline’ represents the SAC-based path planning method proposed in [38]. The ‘hybrid’ method denotes the path planning algorithm which is extended from our previous work by adding the path following point. The complete path planning and following approach proposed in this paper is represented as ‘hybrid-move’.

3.1. Simulation Environment

Based on ROS and Rviz, the simulation environment is established, as Figure 4 shows. In this simulation environment, the kinetic model is based on ROS, and the visualization tool is Rviz. The density of obstacles in the environment can be set arbitrarily. The obstacle is a cylinder with a preset radius. Obstacles can overlap with each other to form different shapes, in order to better simulate the complex obstacles in the real world.

In the simulation environment, the number of multiple UAVs is linearly expanded. Every UAV interacts with its surroundings through LiDAR one by one, and every UAV treats other UAVs as part of the environment. In this way, joint action space and state space are not used, so the related parameters increase linearly instead of exponentially when the number of UAVs increases. This feature is beneficial to the scalability and overall performance of the algorithm. The UAV characteristics in the simulation environment are shown in

Table 1. The UAV characteristics in the simulation environment.

Parameter	Value
Maximum Linear Velocity	1.0 m/s
Maximum Angular Velocity	1.0 rad/s
LiDAR Angular Range	$2.094$ rad to −2.094 rad
Maximum Range of Laser	3.0 m
Minimum Range of Laser	0.15 m
Number of Laser Beams	40
Flight Height	0.5 m

. It describes UAV velocity limits, LiDAR-related information, and flight height. This velocity limit is considered appropriate for the effective demonstration of simulation experiments and is also practical for potential real-world applications.

In our simulations, the experiment space is set to $20\mathrm{m}\times 20\mathrm{m}$, and the distance from the start to the end is $12\mathrm{m}$. The number of UAVs is three, and the density of the obstacle is set between $5\%$ and $15\%$.

3.2. Results and Discussion

In the training process, Figure 5 and Figure 6 show the episode reward and episode time, respectively. We also present crucial data in a tabular format depicting episode time of Figure 6, as shown in

Table 2. Episode time (seconds) of the three methods in different episodes.

Episode	Baseline (s)	Hybrid (s)	Hybrid-Move (s)
1	65.33	61.98	55.80
200	53.62	45.59	40.21
400	47.37	36.73	35.49
600	40.13	36.40	35.36
800	41.91	35.34	32.68
1000	37.83	35.57	32.74

.

As shown in Figure 5, all three methods complete the learning quickly and converge through 200 episodes. The episode reward rose quickly and then stabilized. As the agents interact with the environment, the policy is continuously updated and eventually converges. From Figure 5, we can also conclude that the episode reward of ‘baseline’ method is larger than that of ‘hybrid’ and ‘hybrid-move’ methods in the final training results. This is due to the fact that $r_{follow}$ is set to negative. The final episode reward of ‘hybrid-move’ method is larger than that of ‘hybrid’ method, because that a ‘move’ reward function with positive value is added in the proposed algorithm. The ‘move’ reward function encourages the movement of the UAV, which prevents task failure due to trapping in the local optimal solution or a long task duration.

Figure 6 and Table 2 illustrate the progression of episode times during training, which gradually decreased and finally stabilized. At the beginning of training, all of the three learning methods have not converged, and the UAVs might crash or fly randomly to explore. As long as there is a UAV that has not crashed or reached its goal point, the episode time keeps growing, so at the beginning of training, the episode time is very long. With the convergence of agent policy, UAVs can get to the goal point better and faster, and the episode time also decreases. From Figure 6, we can conclude that the proposed method ‘hybrid-move’ has the minimum episode time compared with ‘baseline’ and ‘hybrid’ methods, and the proposed method has the best performance.

A typical simulated flight process is shown in Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. The yellow curve represents the predefined path. The dotted line with blue, green, and red colors denote the detection range of LiDAR mounted on different UAVs. The UAV senses surrounding obstacles and other UAVs through laser scan data. The DRL-based planner takes the state composed of laser scan data, the UAV’s velocity and reference path information as input, and outputs velocity information to guide the UAV to the goal point. Obstacle avoidance among UAVs is demonstrated in another scenario, as shown in Figure 13. The UAVs adeptly detect obstacles in their vicinity using LiDAR data, concurrently being attuned to the presence of other UAVs.

To further test the performance of the algorithm, we ran 1000 rounds of tests and presented statistical results. Our simulation experiment employed three UAVs and counted the success rate of each UAV reaching the goal point. The results of each experiment can be divided into 0 UAVs, 1 UAV, 2 UAVs or 3 UAVs successfully reaching their goal points. We ran 1000 rounds of tests with our trained model, and the results are displayed in

Table 3. Simulation result: Data on the number of successful UAV arrivals of the three methods in 1000 rounds.

Methods	0 UAV	1 UAV	2 UAVs	3 UAVs
baseline	0	39	256	705
hybrid	0	34	244	722
hybrid-move	1	10	160	829

and Figure 14. Note that due to the random generation of environmental obstacles, extreme cases may occur, such as obstacles that coincide with or are too close to the start or goal points. These obstacles can prevent any UAV from reaching the goal point. While the probability of encountering such extreme situations is low, we acknowledge their potential and clarify that they do not significantly affect the overall success rate assessment of the algorithm. We can obtain that compared with the other two methods (about $70\%$ success rate), the proposed algorithm achieves $80\%$ success rate to guarantee that 3 UAVs reach the goal points. The experimental results confirm that adding path-following points and encouraging the move’s reward function has a significant improvement in the success rate of the task.

4. Conclusions

This paper introduces a parameter-sharing off-policy multi-agent path planning and the following approach. We establish a multi-UAV simulation environment to train and validate the proposed approach, and a parameter-sharing off-policy multi-agent path planning and the following approach is proposed. Furthermore, we design a reward function that encourages UAV movement, addressing the problem of UAVs remaining stationary in the initial phase and exhibiting overly cautious and minimal actions when approaching the goal. Future work will focus on applying the proposed algorithm to real UAVs, with a particular emphasis on integrating safety constraints to ensure the robust and secure deployment of the system in practical scenarios.