Cooperative Path Planning for Autonomous UAV Swarms Using MASAC-CA Algorithm

Abstract

Cooperative path planning for unmanned aerial vehicle (UAV) swarms has attracted considerable research attention, yet it remains challenging in complex, uncertain environments. To tackle this problem, we model the cooperative path planning task as a heterogeneous decentralized Markov Decision Process (MDP), emphasizing the symmetry-inspired role assignment between leader and wingmen UAVs, which ensures balanced and coordinated behaviors in dynamic settings. We address the problem using a Multi-Agent Soft Actor-Critic (MASAC) framework enhanced with a symmetry-aware reward mechanism designed to optimize multiple cooperative objectives: simultaneous arrival, formation topology preservation, dynamic obstacle avoidance, trajectory smoothness, and inter-agent collision avoidance. This design promotes behavioral symmetry among agents, enhancing both coordination efficiency and system robustness. Simulation results demonstrate that our method achieves efficient swarm coordination and reliable obstacle avoidance. Quantitative evaluations show that our MASAC-CA algorithm achieves a Mission Success Rate (MSR) of 99.0% with 2–5 wingmen, representing approximately 13% improvement over baseline MASAC, while maintaining Formation Keeping Rates (FKR) of 59.68–26.29% across different swarm sizes. The method also reduces collisions to near zero in cluttered environments while keeping flight duration, path length, and energy consumption at levels comparable to baseline algorithms. Finally, the proposed model’s robustness and effectiveness are validated in complex uncertain environments, underscoring the value of symmetry principles in multi-agent system design.

1. Introduction

With the development of robotics, micro-electro-mechanical systems (MEMSs), and artificial intelligence technologies, unmanned aerial vehicle (UAV) systems have been extensively deployed in environmental monitoring, data collecting, service delivery, and emergence communication. Significant limitations are observed in single-UAV systems during target tracking, where evasive maneuvers tend to cause mission failure due to target loss or UAV destruction. Multi-UAV cooperative path planning, utilizing distributed decision-making and multi-objective optimization [1], can significantly enhance mission robustness and operational efficiency. At its core lies the multi-UAV cooperative decision-making and optimization process [2]. This process must integrate mission requirements, inter-UAV collaborative relationships [3], and obstacle avoidance constraints to generate flight trajectories that achieve spatiotemporal and task-level coordination.

To tackle the multi-UAV cooperative path planning problem, this study proposes the MASAC-CA algorithm (Multi-Agent Soft Actor-Critic with Cooperative Arrival), built upon the Multi-Agent Soft Actor-Critic (MASAC) [4,5] framework. The algorithm models the path planning problem as a heterogeneous Markov Decision Process (MDP) [6]. This heterogeneity introduces a purposeful asymmetry in agent roles and state representations, which in turn enables the emergence of highly symmetric and coordinated collective behaviors, such as formation keeping and simultaneous arrival. Through task cooperativity analysis and the design of refined reward functions, it achieves multi-objective coordination encompassing simultaneous arrival, formation topology preservation, dynamic obstacle avoidance, trajectory smoothness, and inter-agent collision avoidance. The reward mechanism is designed with a symmetry-aware principle, ensuring that the incentives for coordination are balanced across different agents, promoting behavioral symmetry where needed (e.g., velocity matching) while respecting their functional asymmetries (e.g., leader-initiated coordination). By integrating core elements such as approach speed, flight time coordination, and arrival timing, this design establishes a systematic guidance mechanism for cooperative policy. The main contributions of this paper are as follows:

• We propose a hierarchically structured reward mechanism specifically designed for precision simultaneous arrival missions. This architecture systematically integrates sparse terminal rewards with dense intermediate incentives, incorporating a post-arrival holding mechanism for the leader UAV to prevent formation disintegration. The framework combines stagnation penalties, temporal efficiency incentives, and collision avoidance penalties to achieve multi-objective cooperative optimization. Notably, the integration of formation maintenance rewards and obstacle avoidance penalties establishes a dual-layer collision prevention system, significantly enhancing operational safety while preserving formation integrity throughout mission execution.
• We introduce a novel heterogeneous Markov Decision Process formulation that addresses Multi-UAV coordination challenges in simultaneous arrival scenarios [6]. The framework employs distinct state representations: the leader UAV’s state space incorporates a mission completion flag, enabling real-time terminal condition awareness, while wingmen states include the leader UAV’s positional coordinates [7]. This design establishes the leader’s arrival status and position as essential state information, with the completion flag serving as a global coordination signal [8]. The architecture ensures continuous positional tracking by wingmen throughout all mission phases, including operation after the leader reaches the target, creating state-level cooperative coupling that accurately models the leader–follower formation and eliminates formation instability risks inherent in conventional homogeneous approaches.
• Our methodology rigorously implements the Centralized Training with Decentralized Execution (CTDE) [9] paradigm to balance coordination efficiency with operational autonomy. During training, we employ centralized integration of agent experiences for joint policy optimization, preventing behavioral conflicts and suboptimal solutions [10]. During execution, each agent operates independently using locally observable states with decentralized policy networks, ensuring system robustness, real-time responsiveness, and resilience to communication latency [11]. This approach enables effective cooperative behavior while maintaining the operational independence required for dynamic environments [12]. The overall framework is illustrated in Figure 1.

2. Related Work

In Section 2, we survey recent advances in path planning through two methodological paradigms: traditional optimization methods and reinforcement learning methods.

2.1. Traditional Path Planning Methods

Current UAV path planning approaches exhibit a heavy reliance on precise mathematical models of the environment [13]. Algorithms such as Dijkstra determine optimal paths through exhaustive node traversal, while A* enhances search efficiency via a heuristic evaluation function [14]. D*, conversely, facilitates dynamic path updates [15]. However, their applicability in complex environments is severely hampered by the “curse of dimensionality”, resulting in prohibitive computational complexity or in-tractability [16]. Heuristic techniques, including Particle Swarm Optimization (PSO) [17] and Ant Colony Optimization (ACO) [18], effectively circumvent this dimensionality curse, offering advantages such as rapid planning speed and robust global exploration capabilities [19]. Nevertheless, their effectiveness is predominantly confined to static environment [20], rendering them inadequate for dynamic scenarios characterized by unknown global states and stringent real-time planning requirements [21].

2.2. Reinforcement Learning Methods

Deep Reinforcement Learning (DRL), which integrates perceptual capabilities with decision-making, presents a transformative approach to dynamic path planning. By formulating the problem as a sequential decision optimization task and adopting an “offline training, online inference” paradigm, DRL enables the learning of optimal agent policies and facilitates real-time planning in complex dynamic environments, demonstrating notable adaptability and transferability. Key single-agent DRL contributions include the Double Deep Q-Network (DDQN) algorithm [22], augmented with Kalman filtering for efficient UAV tracking of maneuvering targets; the MN-DDPG method [23], utilizing hybrid noise to enhance exploration efficiency [24]; the Twin Delayed DDPG (TD3) algorithm [25], addressing challenges in continuous state and action spaces; the Imitation-enhanced Dueling Double Deep Q-Network (ID3QN) algorithm [26], balancing exploration-exploitation via an ε-imitation strategy; and the Soft Actor-Critic (SAC) algorithm [27], tackling underwater robot localization and tracking. Critically, however, these single-agent DRL algorithms lack the capacity for effective multi-UAV cooperative planning.

Extending to multi-agent systems, the decoupled Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework [28] has been applied to UAV swarm autonomous tracking and obstacle avoidance. This framework optimizes cooperative decision-making under the CTDE principle, mitigating issues of suboptimality and centralization dominance. Nevertheless, constrained by the inherent exploration limitations of its underlying DDPG algorithm, MADDPG suffers from low sample learning efficiency.

Beyond MADDPG, other multi-agent reinforcement learning paradigms have attracted considerable research interest. The Multi-Agent Proximal Policy Optimization (MAPPO) algorithm [29], built on on-policy learning and a clipped objective function, exhibits excellent training stability in cooperative multi-agent settings. However, its on-policy nature leads to inherently lower sample efficiency, hindering its practical deployment in computationally demanding UAV simulation tasks.

In contrast, value decomposition methods such as QMIX [30] have demonstrated outstanding performance in discrete decision-making domains like StarCraft. By enforcing a monotonicity constraint to aggregate individual Q-values into a global Q-value, these methods achieve efficient decentralized execution and implicit credit assignment. However, this structural constraint, while enforcing a form of functional symmetry in value decomposition, limits the policy representational capacity, as it cannot model cooperative scenarios requiring an agent to take a locally suboptimal action for greater global benefit—a form of strategic asymmetry. Moreover, QMIX is fundamentally designed for discrete action spaces, rendering it difficult to apply directly to UAV path planning problems that require fine-grained continuous control.

To address these gaps, we introduce the MASAC-CA algorithm, founded on three core technical contributions deeply rooted in the principles of symmetry and asymmetry for effective coordination. First, it utilizes a heterogeneous MDP with role-aware states, creating an asymmetric information structure (including the leader’s terminal flag) that is essential for orchestrating symmetric outcomes like simultaneous arrival. This design ensures that the asymmetric roles are leveraged to achieve symmetric spatiotemporal coordination. Second, it leverages the MASAC framework to combine the sample efficiency of off-policy learning with innate support for continuous control, facilitated by an improved CTDE paradigm. The CTDE paradigm itself embodies a symmetric-asymmetric duality: asymmetric information access during training versus symmetric execution capabilities during deployment. Third, it employs a maximum entropy objective to boost exploration beyond monotonicity limitations, encouraging a more symmetric exploration of the action space for all agents. Together, these features enable robust multi-objective coordination encompassing simultaneous arrival, formation keeping, and collision/obstacle avoidance, proving particularly effective for cooperative UAV path planning in complex uncertain environments by skillfully managing the interplay between symmetry and asymmetry.

3. Problem Formulation

3.1. Problem Statement for UAV Swarms Cooperative Path Planning

As illustrated in Figure 2, a swarm of $N$ heterogeneous UAVs (using $N=3$ as an example) forms a formation characterized by one leader UAV and multiple wingman UAVs. The swarm must dynamically maneuver around randomly distributed obstacles (including jamming zones and No-Fly Zones (NFZs)) within the environment. They are required to establish the formation mid-flight and ultimately reach a randomly designated goal location. The UAVs utilize satellite data to pre-determine the approximate goal position. During flight, they continuously sense the external environment via onboard sensors to avoid hazardous areas, fulfilling their mission while ensuring operational safety. The UAVs possess distinct roles: one serves as the leader UAV, while the others function as wingman UAVs, with differing performance capabilities between types.

There is exactly one leader UAV. Its primary task is to navigate around obstacles to reach the goal location. The wingman UAVs, which may comprise multiple UAVs, must continuously follow the leader UAV while maintaining formation and avoiding obstacles. They must reach the goal within a short time delta relative to the leader. Wingmen may sacrifice themselves to protect the leader if necessary.

In summary, the tasks required for cooperative path planning in UAV swarms include:

• The leader UAV must avoid obstacles (including jamming zones and NFZs) to prevent collisions that could damage the UAV while ensuring it does not enter airspace restricted to UAV overflight or interference zones.
• The leader UAV and the wingman UAVs are considered to have arrived simultaneously if they reach the target location within a time difference of less than ten time steps.
• The wingmen should stay as much as possible at a safe distance from the leader UAV to maintain formation.

The task is considered successful if the leader UAV reaches the target location within the specified time while avoiding obstacles, and the wingman UAVs avoids obstacles as much as possible, with all UAVs arriving simultaneously within ten time steps.

Mission failure is defined under the following conditions:

• The leader UAV collides with an obstacle resulting in destruction, or enters an NFZ or jamming zone.
• The leader UAV fails to reach the goal location within the stipulated timeframe, indicating inadequate planning.

3.2. UAV Kinematic Model

In the cooperative path planning of a UAV swarm, the kinematic models of all UAVs are identical. The kinematic equation constraints can be expressed as:

$$\left\{\begin{array}{l}{\dot{x}}_i=v_i\cos {\psi }_i\\ {\dot{y}}_i=v_i\sin {\psi }_i\\ {\dot{\psi }}_i={\omega }_i\\ {\dot{v}}_i=a_i\end{array}\right.$$

(1)

where $(x_i,y_i)$ are the horizontal and vertical coordinates of the UAV, ${\psi }_i$ is the heading angle, $v_i$ denotes the velocity magnitude, ${\omega }_i$ represents the angular velocity, and $u_i$ is the acceleration $(i=1,2\dots ,N-1)$.

The discrete dynamic equation constraints for the UAV swarm from time $t$ to $t+1$ are:

$$\left\{\begin{array}{l}{x}_i^{t+1}=x_i^t+v_i^t\cos {\psi }_i^t\Delta t\\ y_i^{t+1}=y_i^t+v_i^t\sin {\psi }_i^t\Delta t\\ v_i^{t+1}=v_i^t+a_i^t\Delta t\\ {\psi }_i^{t+1}={\psi }_i^t+{\omega }_i^t\Delta t\end{array}\right.$$

(2)

The state of each UAV satisfies the following constraints:

$$\left\{\begin{array}{l}{x}_{i,\min }\le x_i^t\le x_{i,\max }\\ y_{i,\min }\le y_i^t\le y_{i,\max }\\ v_{i,\min }\le v_i^t\le v_{i,\max }\\ {\psi }_{i,\min }\le {\psi }_i^t\le {\psi }_{i,\max }\\ u_{i,\min }\le u_i^t\le u_{i,\max }\\ {\omega }_{i,\min }\le {\omega }_i^t\le {\omega }_{i,\max }\\ \sqrt{{(x_i^t-x_O)}^2+{(y_i^t-y_O)}^2}\ge R_O\end{array}\right.$$

(3)

where $(x_O,y_O)$ are the horizontal and vertical coordinates of the obstacle, $R_O$ is the radius of the obstacle.

3.3. MDP Modeling of UAV Swarm Cooperative Path Planning

MDP is an ideal model for sequential decision-making under environmental conditions [31,32,33,34,35]. In this paper, MDP is used to describe the decision model of UAV swarm path planning, which can be defined as a five-tuple $<S,A,P,R,\gamma >.$

(1)

State Set

$S$ is the state set, $S=\left\{s_0,s_1,\right.\dots ,s_{k-1},\dots ,\left.s_{N-1}\right\}$, where $s_{k-1}$ represents the state observation information of the k-th UAV ($1\le k\le N$), and $N$ is the total number of UAVs. This paper addresses cooperative path planning for UAV swarms. Each UAV’s observation comprises two parts: the first part consists of its intrinsic attributes including coordinates, heading angle, and speed magnitude, which are symmetric across all agents; the second part comprises type-dependent observational state information—for the leader, this includes the goal’s coordinates, obstacle coordinates, obstacle flag, and goal-reached flag; for wingmen, this includes coordinates of the leader, obstacle, and goal. This design introduces a controlled asymmetry in the observation space, which is crucial for facilitating role-specific decision-making that ultimately leads to symmetric collective outcomes, such as maintaining a symmetric formation geometry around the leader.

For the leader:

$$\begin{array}{l}{s}_0=[x_0,y_0,{\psi }_0,v_0,x_G,y_G,x_O,y_O,o_f,win]\\ o_f=\left\{\begin{array}{l}1,d_O\le 2R_O\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.\\ d_O=\sqrt{{(x_0-x_O)}^2+{(y_0-y_O)}^2}\\ win=\left\{\begin{array}{l}1,d_G\le D_1\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(4)

$\left(x_0,y_0\right)$ denotes the leader’s horizontal and vertical coordinates, with a heading angle of ${\psi }_0$ and a speed magnitude of $v_0$. $\left(x_{\mathrm{G}},y_{\mathrm{G}}\right)$ denotes the goal location’s horizontal and vertical coordinates, while $\left(x_O,y_O\right)$ denotes the obstacle’s horizontal and vertical coordinates. $d_O$ represents the distance between the UAV and the obstacle. $R_O$ is the radius of the obstacle. The obstacle flag $o_f$ is set to 1 if $d_O$ is less than twice the obstacle’s radius; otherwise, it is 0. $d_G$ is the distance between the UAV and the goal location. $D_1$ is the target arrival determination threshold. The target reached flag win is set to 1 if $d_G$ is less than $D_1$; otherwise, it is 0.

For wingmen:

$$\begin{array}{l}{s}_k=[x_k,y_k,{\psi }_k,v_k,x_G,y_G,x_O,y_O,x_0,y_0]\\ k=1,2,\dots ,N-1\end{array}$$

(5)

$\left(x_0,y_0\right)$ denotes the leader’s horizontal and vertical coordinates, while $\left(x_k,y_k\right)$ represents the coordinates of the k-th wingman. Each respective wingman has a heading angle of ${\psi }_i$ and speed magnitude $v_i$.

(2)

Action Set

$A$ is the action set, $A=\left\{a_0,a_1,\right.\dots ,a_{k-1},\dots ,\left.a_{N-1}\right\}$, where $a_{k-1}$ represents the action of the k-th UAV. UAV maneuver decisions are implemented by determining appropriate acceleration and angular velocity to achieve desired speed and heading angle. The joint action space for the k-th UAV is $a_{k-1}={\left\{u_{k-1},{\omega }_{k-1}\right\}}^T$, where $u_{k-1}$ denotes the k-th UAV’s acceleration and ${\omega }_{k-1}$ denotes its angular velocity.

(3)

State Transition Probability

$P(s\prime |s,a)$ denotes the probability of a UAV transitioning from state $s$ to state $s\prime$ after taking action $a$.

(4)

Discount Factor

The discount factor is a core hyperparameter in the MDP framework that modulates the agent’s temporal bias (myopic vs. far-sighted) and guarantees mathematical tractability of the problem. Its value lies in the interval [0, 1].

(5)

Reward Set

To satisfy cooperative path planning requirements for UAV swarms, the reward function incorporates proximity speed and approximated flight time terms. Separate reward functions are designed for two UAV types:

For the leader UAV:

(a)

Boundary penalty $r_{edge}$.

As shown in Equation (6), $dist\_d$ represents the distances from the UAV to each of the four boundaries, $d_G$ is the distance between the UAV and the target location, $S$ is the safety boundary threshold (which triggers a penalty when the UAV’s distance to any boundary is less than this threshold, with the penalty increasing as the UAV gets closer to the boundary; otherwise, no penalty occurs), $C$ is the boundary penalty coefficient, $S_1$ is the initial safety boundary threshold, and $C_1$ is the initial boundary penalty coefficient. When the UAV enters within the target approach threshold $D_2$, both the safety boundary threshold and the boundary penalty coefficient are reduced by half; this relaxation of boundary penalty constraints when the UAV is near the target prevents the UAV from being overly cautious about boundary collisions and facilitates closer approach to the target.

$$\begin{array}{l}{r}_{edge}={\displaystyle \sum \max \left(0,\frac{S-dist\_d}{S}\right) \ast C};\\ C=\left\{\begin{array}{l}0.5C_1,d_G<D_2\hfill \\ C_1, \mathrm{otherwise}\hfill \end{array}\right.\\ S=\left\{\begin{array}{l}0.5S_1,d_G<D_2\hfill \\ S_1,\mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(6)

(b)

Obstacle avoidance penalty $r_{obs}$

$d_O$ denotes the distance between the UAV and an obstacle, and $R_O$ represents the radius of the obstacle. A linear penalty is applied when the UAV’s distance to the obstacle is less than three times the obstacle’s radius but greater than one times its radius ($R_O\le d_O\le 3R_O$), where the penalty increases progressively as the UAV approaches the obstacle, providing incremental avoidance guidance. If the distance falls below the obstacle’s radius ($d_O<R_O$), the UAV is destroyed and receives a collision penalty $P_1$, compelling it to learn obstacle avoidance. Otherwise, no obstacle penalty is applied.

$$r_{obs}=\left\{\begin{array}{ll}{P}_1,\hfill & d_O<R_O\hfill \\ {\displaystyle \frac{d_O-60}{400}},& R_O\le d_O\le 3R_O\hfill \\ 0,& \mathrm{otherwise}\end{array}\right.$$

(7)

(c)

Goal reward $r_{goal}$

$d_G$ denotes the distance between the leader UAV and the target point, $D_1$ is the target arrival determination threshold, and $C_2$ is the heading deviation penalty coefficient. When the leader UAV’s distance to the target point is less than a fixed threshold ($d_G\le D_1$), the leader reaches the goal and receives reward $P$ (target achievement reward). As shown in Figure 3a, ${\phi }_0$ represents the angular difference between the leader’s heading ${\psi }_0$ and the bearing angle ${\theta }_0$ from the leader to the target point. This encourages the leader UAV to fly toward the target direction.

$$r_{goal}=\left\{\begin{array}{ll}P,\hfill & \hfill d_G\le D_1\\ {\phi }_0\times C_2,\hfill & \mathrm{otherwise}\end{array}\right.$$

(8)

$${\phi }_0={\psi }_0-{\theta }_0$$

(9)

(d)

Formation distance reward $r_{follow}$

Provided that the leader UAV has not crashed or reached the target location, $\overrightarrow{L}$ denotes the position of the leader UAV, $\overrightarrow{P_i}$ represents the current position of the i-th wingman UAV, ${\alpha }_i$ is its ideal angular position, $\overrightarrow{Q_i}$ is the ideal position for this wingman, and $d_i$ is the distance between the wingman’s current position and its ideal position. $D$ is the ideal formation distance between the wingman and leader, $\mathsf{\sigma }$ is the formation distance tolerance threshold, and $C_3$ is the formation penalty coefficient. As illustrated in Figure 3b, when the positional deviation $d_i$ between the wingman’s actual and ideal positions is within the tolerance threshold ($0\le d_i\le \sigma$), the reward value decreases gradually as increases; when the deviation exceeds the threshold ($d_i>\sigma$), the reward decreases sharply with increasing $d_i$; maximum reward is attained when the wingman’s actual and ideal positions coincide, thereby guiding wingman to maintain formation with the leader while avoiding inter-agent collisions.

$$\begin{array}{l}\overrightarrow{L}=[x_0,y_0],\overrightarrow{P_i}=[x_i,y_i]\\ {\alpha }_i=(i-1)\times {\displaystyle \frac{2\pi }{N}}\\ \overrightarrow{Q_i}=\overrightarrow{L }+D\cdot \left[\cos {\alpha }_i, \sin {\alpha }_i\right]\\ d_i=‖\overrightarrow{P_i}-\overrightarrow{Q_i}‖(i=1,\dots ,N-1)\\ r_{follow}=\left\{\begin{array}{ll}{C}_3{\displaystyle \sum _{i=1}^{N-1}{e}^{-\frac{{d_i}^2}{2{\sigma }^2}}},\hfill & o_f=0\wedge win=0\\ 0,\hfill & \mathrm{otherwise}\end{array}\right.\end{array}.$$

(10)

(e)

Stagnation penalty $r_{stagnation}$

A stagnation penalty $r_{stagnation}$ is subsequently designed. $\overrightarrow{L}$ denotes the leader UAV’s position at the current timestep, $\overrightarrow{L^{\prime }}$ represents its position at the previous timestep, $\Delta$ is the distance between these positions, $N_S$ is the stagnation step count, and $N_{\max }$ is the stagnation determination displacement threshold. $C_4$ serves as the stagnation penalty weight coefficient. When $\Delta <N_{\max }$, $N_S$ increments by 1; when $\Delta \ge N_{\max }$, $N_S$ decrements by 1 until reaching 0. The stagnation penalty reward is proportional to $N_S$ to prevent the leader UAV from remaining stationary.

$$\begin{array}{l}\overrightarrow{L}=(x_0,y_0),\Delta =‖\overrightarrow{L}-\overrightarrow{L^{\prime }}‖\\ N_S=\left\{\begin{array}{ll}{N}_S+1,\hfill & \Delta <N_{\max }\\ \max (0,N_S-1),\hfill & \Delta \ge N_{\max }\end{array}\right.\\ r_{stagnation}=\left\{\begin{array}{ll}-C_4\times N_S,\hfill & \Delta <N_{\max }\\ 0,\hfill & \mathrm{otherwise}\end{array}\right.\end{array}$$

(11)

(f)

Time penalty $r_{time\_penalty}$

This component encourages rapid decision-making by imposing a penalty of $C_5$ per timestep (where $C_5$ is the time penalty coefficient) when the leader UAV has neither crashed nor reached the target location.

$$r_{time\_penalty}=\left\{\begin{array}{ll}{C}_5,\hfill & d_G\le D_1\wedge d_O\ge R_O\\ 0,\hfill & \hfill \mathrm{otherwise}\end{array}\right.$$

(12)

(g)

Speed coordination reward $r_{speed}$

$l_i$ denotes the distance from the i-th wingman to the leader UAV, $\Delta v_i$ is the speed difference between the wingman and leader, $D_3$ is the formation coordination distance threshold, and $C_6$ is the speed difference tolerance. When $l_i$ remains within distance $D_3$ and $\Delta v_i$ is less than $C_6$, the speed coordination reward increments by 1, ensuring wingmen match the leader’s speed.

$$\begin{array}{l}{l}_i=‖(x_i,y_i)-(x_0,y_0)‖\\ \Delta v_i={‖v_i-v_0‖}_2\\ r_{speed}=r_{speed}+{\displaystyle \sum _{i=1}^{N-1}H(x)}\\ H(x)=\left\{\begin{array}{ll}1,\hfill & 0\le l_i\le D_3\Lambda \Delta v_i<C_6\\ 0,\hfill & \hfill \mathrm{otherwise}\end{array}\right.\end{array}$$

(13)

(h)

Temporal coordination reward $r_{reachstep}$

$t_{f,i}$ denotes the timestep when the i-th wingman reaches the target, $t_{lead}$ represents the timestep when the leader reaches the target, $\Delta t_i$ is the arrival time difference between wingman and leader, and $T_{\max }$ is the formation time coordination tolerance. When $\Delta t_i<T_{\max }$, a linearly decreasing reward is applied where smaller time differences yield higher rewards, guiding leader and wingmen to achieve simultaneous arrival at target.

$$\begin{array}{l}\Delta t_i=|t_{f,i}-t_{lead}|\\ r_{reachstep}[i]=\left\{\begin{array}{ll}200-20\Delta t_i,\hfill & \Delta t_i<T_{\max }\\ 0,\hfill & \mathrm{otherwise}\end{array}\right.\\ r_{reachstep}={\displaystyle \sum _{i=1}^{N-1}{r}_{reachstep}[i]}\end{array}$$

(14)

(i)

Terminal achievement reward $r_{ter\min ation}$

Case 1: All wingmen reach the target within 10 timesteps after the leader’s arrival. Case 2: Leader reaches within 1000 timesteps but wingmen fail to arrive within 1010 timesteps. Case 3: Leader fails to reach within the designated 1000 timesteps. Here, $P_2$ is the base reward for full formation coordination success, $P_3$ is the base reward for leader’s core mission completion, and $t$ represents elapsed timesteps. This incentivizes rapid task accomplishment by both leader and wingmen.

$$r_{ter\min ation}=\left\{\begin{array}{cc}{P}_2-t,& \hfill \mathrm{if}\mathrm{case}1\\ P_3-t,& \mathrm{elif}\mathrm{case}2\\ 0,& \hfill \mathrm{else}\mathrm{case}3\end{array}\right.$$

(15)

The reward function for the leader UAV is defined as: $R_L={\omega }_{L,1}\times r_{edge}+{\omega }_{L,2}\times r_{obs}$$+{\omega }_{L,3}\times r_{goal}+{\omega }_{L,4}\times r_{follow}+{\omega }_{L,5}\times r_{stagnation}+{\omega }_{L,6}\times r_{time\_penalty}+{\omega }_{L,7}\times r_{speed}+{\omega }_{L,8}\times r_{reachstep}$$+{\omega }_{L,9}\times r_{ter\min ation},$ Here, ${\omega }_{L,1}$ to ${\omega }_{L,9}$ represent the weights assigned to each reward component.

For wingmen, the boundary penalty ($r_{edge}$), obstacle avoidance penalty ($r_{obs}$), goal achievement reward ($r_{goal}$), formation distance reward ($r_{follow}$), stagnation penalty ($r_{stagnation}$), time penalty ($r_{time\_penalty}$), speed coordination reward ($r_{speed}$), and arrival time difference reward ($r_{reachstep}$) mirror those of the leader but with adjusted parameters ($C_2$ = 5, $C_3$ = 20, $C_4$ = 1, $C_5$ = −0.5). The formation distance, speed coordination, and arrival time difference rewards are specific to the k-th wingman relative to the leader (i.e., i = k in Equations (9), (12) and (13)). Additionally, the remaining flight time difference reward ($r_{\Delta tgo}$ [36]) is introduced. As shown in Figure 3c, $v_k$ denotes the speed magnitude of the k-th wingman ($v_0$ for the leader), ${\phi }_k$ represents the angular difference between the wingman’s heading angle ${\psi }_k$ and the bearing angle ${\theta }_k$ from the wingman to the target point. $V_{Ck}$ is the closing velocity component of the k-th wingman along the line-of-sight to the target point ($V_{C0}$ for the leader). The term $tg{o}_k$ approximates the remaining flight time for the k-th wingman ($tg{o}_0$ for the leader), and $C_7$ is the remaining flight time difference penalty coefficient. The difference in approximated remaining flight time between wingman k and the leader is $\Delta tg{o}_k$. A larger time difference incurs a higher penalty, guiding the leader and wingmen to reach the target simultaneously.

$$\begin{array}{l}{V}_{Ck}=v_k\times \cos {\phi }_k\\ tg{o}_k={\displaystyle \frac{V_{Ck}}{{‖\left(x_i,y_i\right)-\left(x_G,y_G\right)‖}_2}}\\ \Delta tg{o}_k=tg{o}_k-tg{o}_0\\ r_{\Delta tgo}=-C_7\times \Delta tg{o}_k\end{array}$$

(16)

The reward function for the k-th wingman is: $R_{F,k}={\omega }_{F,1}\times r_{edge,k}+{\omega }_{F,2}\times r_{obs,k}+{\omega }_{F,3}$$\times r_{goal,k}+{\omega }_{F,4}\times r_{follow,k}+{\omega }_{F,5}\times r_{stagnation,k}+{\omega }_{F,6}\times r_{time\_penalty,k}+{\omega }_{F,7}\times r_{speed,k}+{\omega }_{F,8}\times r_{reachstep,k}$$+{\omega }_{F,9}\times r_{\Delta tgo,k}$, where ${\omega }_{F,1}$ to ${\omega }_{F,9}$ denote the corresponding weights.

The overall reward set $R$ comprises the leader’s reward $R_L$ and the rewards for the $N-1$ wingmen $R_{F, k}$, expressed as: $R=\left[R_L,R_{F,1},\right.R_{F,2},\left.\dots ,R_{F,N-1}\right]$.

The weights assigned to sub-rewards are determined based on their relative importance to the overall mission accomplishment. The core principle guiding the weight selection process is to ensure that the agent prioritizes learning critical survival behaviors (e.g., obstacle avoidance) and primary mission objectives (e.g., synchronized arrival) over secondary goals (e.g., perfect formation maintenance). Initial weight values were heuristically assigned according to this priority and were subsequently empirically fine-tuned through ablation studies. The learning process demonstrates significant sensitivity to weight balancing. For instance, excessively increasing the weight for formation distance reward could lead the agent to over-prioritize precise positioning, potentially compromising goal attainment or even resulting in collisions. Conversely, if the weight for the synchronized arrival reward is set too low, the agent may successfully avoid obstacles but fail to coordinate arrival times.

4. Method

4.1. MASAC-CA Algorithm

The core of our approach is the MASAC-CA algorithm, which formulates the UAV swarm path planning problem within a Reinforcement Learning (RL) paradigm guided by symmetry principles. The overall RL concept, modeled as a heterogeneous Markov Decision Process (MDP), is depicted in Figure 4. The heterogeneity in the MDP is not arbitrary; it is a deliberate asymmetric design to enforce symmetry in the group’s spatiotemporal execution. The leader-wingman role asymmetry, reflected in their state representations and reward functions, serves as the foundation for achieving symmetric formation flight and simultaneous arrival. The MASAC-CA algorithm adopts a centralized training with decentralized execution framework, where both the leader UAV and each wingman UAV operate an independent Soft Actor-Critic (SAC) algorithm. This CTDE architecture itself exhibits a form of procedural symmetry: during execution, all agents symmetrically utilize their local policies, while the asymmetric coordination knowledge is distilled into these policies during the centralized training phase. The specific architectural details and training procedure of the MASAC-CA algorithm are presented in Figure 5 and described in the following subsections. During training, the Actor (policy network) takes the state information $s_k\left(t\right)$ of agent $k$ at time $t$ as input and outputs the parameters of the action probability distribution, which governs the agent’s execution of action $a_k\left(t\right)$. The physical environment executes the collective action set A (comprising all agents’ actions), yielding the state set $S(t+1)$ at $t+1$ and the reward set $R\left(t\right)$ at $t$. Subsequently, the current state set $S\left(t\right)$, action set $A\left(t\right)$, reward set $R\left(t\right)$, and next state set $S(t+1)$ are stored in the experience replay buffer. Finally, a batch of data is randomly sampled from the buffer, and agent $k$ uses the concatenated state-action information $(s_k,a_k)$ as input to its Critic network to guide the training of the value network.

4.2. Algorithm Update Procedure

For agent $k$, the update processes for its Actor network, Critic network, and entropy network are similar to the traditional SAC algorithm. The policy network is approximated by an Actor network with weight parameters $\theta$, while the value network is approximated by a Critic network with weight parameters $\omega$. This study employs the Adaptive Moment Estimation (Adam) algorithm to optimize and update the MASAC-CA network parameters. $B$ denotes the experience replay buffer. Loss calculation requires averaging over a batch of samples drawn from the buffer, reflecting the average quality of the sampled data. As shown in Figure 5, data $\left(R\right(t),A\left(t),S\right(t),S(t+1\left)\right)$ sampled from the experience replay buffer is used to update the Q Critic networks. Based on the optimal Bellman equation, $U{\left(t\right)}^{\left(q\right)}=r(t)+\gamma \cdot v(S(t+1))$ serves as the estimated true value of state $S\left(t\right)$. The predicted value estimate of state $S\left(t\right)$ is given by $q_i\left(S\right(t),A\left(t\right);{\omega }^{\left(i\right)})$$\left(i=0,1\right)$ evaluated at the taken $A\left(t\right)$. Finally, the MSE (Mean Squared Error) is used as the loss function, where the average is computed over a batch of data drawn from the buffer. This trains the neural networks Q0 Critic and Q1 Critic.

$$L_{MSE}={\displaystyle \frac{{\displaystyle {\sum }_B{\left[q_i\right(S\left(t),A\right(t);{\omega }^{\left(i\right)})-U{\left(t\right)}^{\left(q\right)}]}^2}}{\left|B\right|}}$$

(17)

Similarly, data $\left(R\right(t),A\left(t),S\right(t),S(t+1\left)\right)$ sampled from the buffer is used to update the V Critic network. The estimated true value of state $S\left(t\right)$ is given by $U{\left(t\right)}^{\left(v\right)}$, which incorporates an entropy term. The predicted value estimate is provided by the V Critic’s output $v\left(S\right(t);{\omega }^{\left(i\right)})$. The MSE loss function is then used to train the V Critic neural network.

$$\begin{array}{l}U{\left(t\right)}^{\left(v\right)}=E_{A^{\prime }\left(t\right)~\pi (·|S\left(t);\theta \right)}\left[\underset{i=0,1}{\min }{q}_i\left(S\right(t),\right.\left.A^{\prime }\left(t);{\omega }^{\left(i\right)}\right)-\alpha \ln \pi \left(A^{\prime }\right(t\left)\right|S\left(t);\theta \right)\right]\\ L_{MSE}={\displaystyle \frac{{\displaystyle {\sum }_B{\left[v\right(S\left(t);{\omega }^{\left(v\right)}\right)-U{\left(t\right)}^{\left(v\right)}]}^2}}{\left|B\right|}}\end{array}$$

(18)

The target V Critic network parameters, denoted by $\overline{\omega }$, are softly updated using the following equation, where $\tau$ is the soft update rate [37].

$$\overline{\omega }\leftarrow (1-\tau )\cdot \overline{\omega }+\tau \cdot \omega$$

(19)

The loss for training the Actor network is given by the following expression:

$$L=-{\displaystyle \frac{1}{\left|B\right|}}{\displaystyle {\sum }_B{E}_{A^{\prime }(t)\sim \mathsf{\pi }(·|S\left(t);\theta \right)}}[q_0(S(t),A^{\prime }(t\left)\right)-\mathsf{\alpha }\ln \mathsf{\pi }(A^{\prime }(t\left)\right|S(t);\theta \left)\right]$$

(20)

This loss is used for gradient descent training of the Actor network, optimized via the Adam algorithm. $A^{\prime }\left(t\right)$ denotes a newly sampled action under state $S\left(t\right)$ according to the current policy $\pi$ of the Actor network—not drawn from the experience replay buffer. $\alpha$ is the entropy reward coefficient, determining the importance of the entropy term $\ln \mathsf{\pi }(A^{\prime }\left(t\left)\right|S\right(t);\theta )$; a higher $\alpha$ indicates greater importance. This work employs Ornstein–Uhlenbeck (OU) noise, which generates temporally correlated exploration, as the exploration noise for the policy network. The complete training procedure of MASAC-CA is summarized in Algorithm 1.

Algorithm 1: MASAC-CA training procedure

Multi-Agent Soft Actor-Critic with Cooperative Arrival (MASAC-CA)

Input: Number of agents $N$, learning rates ${\eta }_{\theta },{\eta }_{\omega },{\eta }_{\alpha }$, soft update rate $\tau$, discount factor $\gamma$, replay buffer size $B$, batch size $M$

Output: Trained policy networks ${\pi }_k$ for all agents $k=1,\dots ,N$

1. Initialize:

Actor networks ${\pi }_k$ with parameters ${\theta }_k$ for each agent $k$

Critic networks $Q_{0,k},Q_{1,k}$ with parameters ${\omega }_{0,k},{\omega }_{1,k}$

Value network $V_k$ with parameters ${\omega }_{V,k}$

Target value network $V_k^{\prime }$ with parameters ${\omega }_{V,k}^{\prime }\leftarrow {\omega }_{V,k}$

Replay buffer $\mathcal{D}$ with capacity $B$

Entropy coefficient $\alpha$

2. for episode = 1 to $E_{\max }$ do

3.  Initialize environment, obtain initial heterogeneous state $S\left(0\right)$

4.  for $t=0$ to $T-1$ do

5.  for each agent $k$ do

6.   Sample action $a_k\left(t\right)\sim {\pi }_k\left(\cdot \mid s_k\left(t\right);{\theta }_k\right)$

7.  end for

8.  Execute joint action $A\left(t\right)=\left[a_1\left(t\right),\dots ,a_N\left(t\right)\right]$

9.  Observe comprehensive reward $R\left(t\right)$ and next heterogeneous state $S\left(t+1\right)$

10.   Store transition $\left(S\left(t\right),A\left(t\right),R\left(t\right),S\left(t+1\right)\right)$ in $\mathcal{D}$

11.  if $\mid \mathcal{D}\mid \ge M$ then

12.   Sample minibatch $\left\{\left(S,A,R,S^{\prime }\right)\right\}\sim \mathcal{D}$ of size $M$

13.  //Update Critic networks

14.   for each agent $k$ do

15.     $U_k^{\left(q\right)}=r_k+\gamma \cdot V_k^{\prime }(S^{\prime };{\omega }_{V,k}^{\prime })$ # Equation (17)

16.    Update ${\omega }_{i,k}$ by minimizing:

$L\left({\omega }_{i,k}\right)=\mathbb{E}([Q_{i,k}\left(S,A;{\omega }_{i,k}\right)-U_k^{\left(q\right)}{)}^2,i=0,1$ # Equation (17)

17.  end for

18. //Update Value networks

19.    for each agent $k$ do

20.    Sample $A^{\prime }\sim \pi \left(S;\theta \right)$

21.    $U_k^{\left(v\right)}=\mathbb{E}[\underset{i=0,1}{\min }{Q}_{i,k}\left(S,A^{\prime }\right)-\alpha \log \pi \left(A^{\prime }\mid S;\theta \right)]$ # Equation (18)

22.    Update ${\omega }_{V,k}$ by minimizing:

$L\left({\omega }_{V,k}\right)=\mathbb{E}[(V_k\left(S;{\omega }_{V,k}\right)-U_k^{\left(v\right)}{)}^2]$ # Equation (18)

23.    end for

24. //Update Actor networks

25.    for each agent $k$ do

26.    Update ${\theta }_k$ by minimizing:

$L\left({\theta }_k\right)=-\mathbb{E}\left[Q_{0,k}\left(S,A^{\prime }\right)-\alpha \log \pi \left(A^{\prime }\mid S;\theta \right)\right]$ # Equation (20)

27.    end for

28. //Soft update target networks

29.   for each agent $k$ do

30.   ${\omega }_{V,k}^{\prime }\leftarrow \left(1-\tau \right){\omega }_{V,k}^{\prime }+\tau {\omega }_{V,k}$# Equation (19)

31.   end for

32.   end if

33.  end for

34. end for

5. Experiments

5.1. Simulation Environment and Algorithm Parameter Settings

This study designs a reinforcement learning interaction environment for UAV path planning. The simulation platform was developed in Python within the PyCharm (Professional 2025.1.1) IDE. The core algorithm was implemented using the PyTorch 2.8.1 deep learning framework, and the environment was built following the OpenAI Gym interface standards for standardized agent-environment interactions. Numerical computation and visualization were handled using NumPy 1.23.3, Matplotlib 3.5.1, and PyGame 2.1.2. At each episode, the initial states of the UAV swarm, goal position and obstacle position are fully randomly initialized [5]. Since the UAVs in this paper are primarily intended for air-to-ground operations, their motion is simplified to a single altitude, conforming to kinematic equations in two-dimensional space. Consequently, the following assumptions are made:

(1)

The real-world three-dimensional motion of UAVs is simplified into a two-dimensional form.

(2)

The position information of the goal location and obstacles is assumed known, acquired by ground-based radar and communicated to the UAVs.

The cooperative path planning simulation environment for the unmanned swarm is defined as a 10,000 m × 8000 m two-dimensional plane. This custom environment features a built-in physics engine that computes state transitions based on the UAV kinematic model (Equations (1)–(3)) and handles collision detection. The composite reward function described in Section 3.3 is computed at each step. There are two distinct types of UAVs: the leader UAV, with a speed range of 100~200 m/s, acceleration control input range of −3~3 m/s², and angular velocity range of −0.6~0.6 rad/s; and the wingman, which possesses higher maneuverability than the leader one to facilitate maintaining formation. Its speed range is 100~500 m/s, acceleration range is −8~8 m/s², and angular velocity range is −1.2~1.2 rad/s. The planning task is deemed successful, ending the episode, if both the leader and wingman UAVs arrive within the circular area of the goal location within a short time difference. Conversely, the planning task fails and the episode ends if the leader UAV enters the circular area of any obstacle zone, which signifies a collision.

Table 1. Parameters and hyperparameters of MASAC-CA.

Parameter	Hyperparameter	Actor Network	Critic Network
Actor learning rate: $1 \times {10}^{-3}$	Maxstep: 1000	Input: 10	Input:10 × N + 2
Critic learning rate: $3 \times {10}^{-3}$	Batch size: 128	Hidden: (256, 256)	Hidden: (256, 256)
Entropy learning rate: $3 \times {10}^{-4}$	Maxepisode: 800	Output: 2	Output: 1
Soft update rate: $1 \times {10}^{-2}$	Replay buffer: 20,000	Activation: relu, tanh	Activation: relu
Discount factor: 0.9	Optimizer: adam	Weights: N(0, 0.1)	Weights: N(0, 0.1)

lists all parameters and hyperparameters used in the MASAC-CA algorithm. The parameters for the OU noise are set to ${\mathsf{\sigma }}_{\mathrm{OU}}=0.1$, ${\mathsf{\theta }}_{\mathrm{OU}}= 0.1$, ${\mathrm{dt}}_{OU}=0.01$, and OU noise is only added during the first 20 episodes. As shown in Algorithm 1, the actor network is a fully connected layer with a structure of 10-256-256-2, utilizing relu and tanh activation functions. The neural network’s initial weights are randomly sampled from a normal distribution with a mean of 0 and a variance of 0.1, with a learning rate of $1\times {10}^{-3}$. The learning rates for the actor, critic, and entropy networks were chosen based on established practices in SAC literature and preliminary hyperparameter tuning. Specifically, the actor employs a lower learning rate to ensure stable policy updates. The critic network takes the augmented information of state and action as input; the number of input layer nodes is 10 × N + 2, and it uses only the relu activation function. Its initial weights are also randomly sampled from a normal distribution (mean 0, variance 0.1), with a learning rate of $3\times {10}^{-3}$. A higher learning rate for the critic facilitates rapid convergence of value estimates, providing a stable foundation for the actor’s updates. The entropy network lacks a fully connected layer structure; its log entropy weight is automatically adjusted during training, with a learning rate of $3\times {10}^{-4}$. The smallest learning rate is used for the entropy network to ensure smooth and stable adjustment of the exploration-exploitation trade-off.

Table 2. MASAC-CA reward function parameters settings.

Reward Function Parameters	Parameter Symbol	Parameter Description	Leader Value	Wingman Value
Environment Interaction Parameters	$d_O$	Obstacle radius	20 m
	$D_1$	Target arrival determination threshold	40 m
	$D_2$	Target proximity threshold	50 m
	$S_1$	Basic safety boundary threshold	20 m
Penalty Coefficients	$P_1$	Collision penalty	−500
	$C_1$	Basic boundary penalty coefficient	−50
	$C_2$	Heading deviation penalty coefficient	10	5
	$C_3$	Formation penalty coefficient	300	400
	$C_4$	Stagnation penalty weight coefficient	0.1	1
	$C_5$	Time penalty coefficient	−0.6	−0.5
	$C_7$	Remaining flight time difference penalty coefficient	——	100
Formation Coordination Parameters	$D$	Ideal formation distance	30 m
	$\sigma$	Formation distance tolerance threshold	15 m
	$D_3$	Formation coordination distance threshold	50 m
	$C_6$	Velocity difference tolerance	2 m/s
	$T_{\max }$	Formation coordination time threshold	10 s
	$N_{\max }$	Stagnation determination displacement threshold	5 m
Reward Parameters	$P$	Target achievement reward	2000	1000
	$P_2$	Full formation coordination reward	1500	——
	$P_3$	Core mission achievement reward	1000	——

lists the parameter settings of the reward functions introduced in Section 3.3.

5.2. Training Process Comparison

During the training phase, the time step ∆t is set to 1, employing a dual-UAV formation cooperative training architecture with a dynamically generated random obstacle and a randomly generated goal point. When a mission fails due to collision with the obstacle, boundary violation, or timeout, the current training episode is immediately reset while maintaining a fixed total of 800 training episodes. To ensure statistical reliability and address performance variability concerns, we conducted multiple independent training runs (n = 50) and present the mean reward curves with standard deviation bands for MASAC-CA in Figure 6. This statistical validation demonstrates the algorithm’s consistency across different random seeds, with tight confidence intervals confirming the reproducibility of our results. As depicted in Figure 7, under the guidance of our meticulously designed reward function for simultaneous arrival tasks and heterogeneous Markov Decision Process (MDP) modeling, the resulting cooperative control strategy demonstrates wingmen’s ability to maintain stable preset formation distances while following the leader throughout all flight phases—including initial high-maneuverability turns, mid-term cruise segments, and final precision approach turns. By accumulating and normalizing per-step agent rewards across episodes, we compared the leader sub-reward, wingman sub-reward, and total reward curves of our MASAC-CA algorithm against baseline MASAC, as well as two additional baselines: Random Strategy and MADDPG. Unlike MASAC’s volatile normalized rewards (plunging from 0.89 to 0.45 due to frequent leader collisions with obstacles causing premature termination), the highly unstable rewards of Random Strategy (exhibiting severe fluctuations and persistently low values across episodes), and the unstable performance of MADDPG’s wingman sub-rewards (indicating its difficulty in maintaining a fixed formation topology over the long term), MASAC-CA’s spatiotemporal coordination rewards yield: wingman sub-rewards stabilizing after 100 episodes with minor fluctuations, while leader sub-rewards remain consistently stable post-100 episodes (confirming reliable obstacle avoidance and timely target arrival). This strategy thus ensures successful swarm cooperative path planning—obstacle avoidance, target arrival, and formation maintenance—by strategically managing wingman risks, enabling both leader and wingmen to consecutively reach targets within minimal time differentials.

5.3. Testing Process Comparison

To evaluate the performance of the MASAC-CA algorithm, multiple Monte Carlo tests were conducted on the trained cooperative path planning decisions of the UAV swarm, as presented in

Table 3. Performance evaluation indicators for collaborative path planning algorithm of UAV cluster.

Metric Name	Symbol	Formula
Mission Success Rate	$J_{\mathrm{MSR}}$	$J_{\mathrm{MSR}}={\displaystyle \frac{n_{\mathrm{C}}}{n_{\mathrm{T}}}}$
Formation Keeping Rate	$J_{\mathrm{FKR}}$	$J_{\mathrm{FKR}}={\displaystyle \frac{t_{\mathrm{C}}}{t_{\mathrm{T}}}}$
Total Flight Duration	$J_{\mathrm{T}}$	$J_{\mathrm{T}}=t_f$
Flight Trajectory Length	$J_{\mathrm{S}}$	$J_{\mathrm{S}}={\displaystyle {\int }_0^{t_f}v}dt$
Flight Energy Consumption	$J_{\mathrm{C}}$	$J_{\mathrm{C}}={\displaystyle {\int }_0^{t_f}\left(\left|a\right|+\left|\omega \right|\right)}dt$
Number of Collisions	$J_{\mathrm{C}\mathrm{N}}$	$J_{\mathrm{CN}}={\displaystyle \sum _{k=1}^{n_{\mathrm{T}}}{n}_{cn}}$
Abruptness	$J_{\mathrm{A}}$	$J_{\mathrm{A}}={\displaystyle \sum _{t=1}^{t_f}\left\{\begin{array}{ll}0,\hfill & if\left|\Delta {\epsilon }_t\right|<\frac{\pi }{6}\\ 1,\hfill & \mathrm{otherwise}\end{array}\right.}$

, using mission success rate (MSR, where $n_{\mathrm{C}}$ is the number of successful episodes with both leader and wingmen reaching the goal and $n_{\mathrm{T}}$ is the total test episodes), formation keeping rate (FKR, measuring the proportion of time steps $t_C$ maintaining formation versus total steps $t_{\mathrm{T}}$), total flight duration (T, with $t_f$ being the terminal time until collision or goal arrival), flight trajectory length (S, integrating velocity $v$ over flight duration), flight energy consumption (C, integrating acceleration $a$ and angular velocity $\omega$ over flight duration), collision number (CN, tallying collisions $n_{cn}$ across episodes), and abruptness (A, quantifying heading angle changes $\Delta {\epsilon }_t$ exceeding ${\displaystyle \frac{\pi }{6}}$) as performance metrics [38].

5.3.1. Wingman Quantity Analysis

For clear observation of flight trajectories, the time step is set to 0.1 during the testing phase. Utilizing the trained neural network weights from Section 3.2, experiments were conducted with the number of wingman UAV expanded to 2, 3, and 4. Performance metrics—including Mission Success Rate (MSR), Formation Keeping Rate (FKR), total flight duration, trajectory length, and energy consumption—were evaluated through 100 Monte Carlo tests (results summarized in

Table 4. The performance metrics of MASAC and MASAC-CA algorithms for varying numbers of UAVs.

Number	Algorithm	${\mathit{J}}_{\mathbf{MSR}} (\%)$	${\mathit{J}}_{\mathbf{F}\mathbf{K}\mathbf{R}}\left(\%\right)$	${\mathit{J}}_{\mathbf{T}}$	${\mathit{J}}_{\mathbf{S}}$	${\mathit{J}}_{\mathbf{C}}$
2	MASAC	87.0	22.83	203.93	116.63	229.45
	MASAC-CA	99.0	59.68	253.81	142.69	282.85
	Random Strategy	0.0	7.10	999.0	510.05	999.83
	MADDPG	0.0	5.55	215.01	122.86	184.78
3	MASAC	83.0	10.77	195.05	111.78	219.55
	MASAC-CA	99.0	40.91	253.52	140.62	283.34
	Random Strategy	0.0	1.54	999.0	501.34	999.66
	MADDPG	0.0	1.11	218.67	123.55	187.64
4	MASAC	83.0	1.75	200.34	111.67	229.99
	MASAC-CA	99.0	33.81	253.31	140.34	279.19
	Random Strategy	0.0	0.35	999.0	495.85	1002.41
	MADDPG	97.0	1.96	209.99	121.99	180.68
5	MASAC	87.0	1.57	214.61	119.63	242.24
	MASAC-CA	99.0	26.29	257.27	137.99	287.19
	Random Strategy	0.0	0.15	999.0	499.76	1001.76
	MADDPG	0.0	0.13	218.46	124.45	188.63

). While MASAC-CA performs comparably to MASAC in total flight duration, trajectory length, and energy consumption, it achieves about 13% improvement in MSR (approaching 100%) and significantly enhanced FKR. This effectively resolves critical challenges in swarm cooperative path planning, demonstrating the algorithm’s superiority. In contrast, both the Random Strategy and the MADDPG algorithm perform poorly due to their inability to maintain formation and achieve coordinated arrival at the target.

5.3.2. Cooperative Behavior Analysis

Figure 8 illustrates the temporal evolution of heading angle, velocity, and inter-agent distance between the leader and the k-th wingman during a test episode. Through comparative analysis of the four algorithms, distinct differences in their cooperative control capabilities become evident. In terms of heading angle and velocity coordination, MASAC-CA (solid red lines) demonstrates superior performance, achieving near-perfect alignment in both heading angles and velocities for both UAVs after 75 timesteps. In contrast, MASAC (dashed blue lines) fails to achieve velocity convergence between the two UAVs, while MADDPG (dash-dotted purple lines) exhibits significant disparities in both velocity and heading angle. This phenomenon arises because the reward functions of both MASAC and MADDPG inadequately address the temporal consistency requirements for formation maintenance, lacking effective cooperative guidance mechanisms.

Regarding inter-agent distance control, the algorithmic differences become more pronounced. The random policy (dotted gray lines) occasionally achieves velocity and heading alignment but maintains an excessive separation exceeding 100 m, rendering effective formation maintenance impossible. This reflects the inherent inefficiency of random exploration in complex environments—completely lacking learning capability and formation awareness, the inter-agent distances remain purely stochastic, preventing meaningful cooperation. MADDPG exhibits more hazardous behavior: its distance curve initially decreases sharply to 2 m before growing continuously without stabilization, indicating not only poor formation keeping but also high collision risk. This instability stems from MADDPG’s lack of explicit leader-wingman role coordination mechanisms, resulting in behavioral inconsistencies. MASAC’s distance behavior also proves problematic: the curve initially decreases to approximately 3 m (indicating high collision risk) before overshooting and increasing. This non-monotonic behavior results from flawed reward design—providing constant distance rewards within a 50-m threshold without precise guidance toward the ideal range, causing initial aggressive approach followed by subsequent overshoot.

Conversely, MASAC-CA exhibits exceptional performance in distance control: it smoothly reduces distance within the first 25 timesteps before stabilizing within the safe 30–40 m range. This optimal performance benefits from its Gaussian-based reward design, which peaks at the ideal distance (30 m), enabling smooth and stable convergence. The results demonstrate that under MASAC-CA, wingmen effectively converge toward the leader while maintaining safe distances, achieving synchronized formation movement with significantly enhanced cooperative formation-keeping capability. This reflects the successful enforcement of spatial symmetry in the swarm’s configuration, a direct result of the asymmetric role assignment and symmetric incentive mechanism in our reward design—markedly superior to MADDPG’s unstable performance and the completely uncoordinated behavior of the random policy.

5.3.3. Algorithm Robustness Analysis

To evaluate the model’s adaptability in complex uncertain environments, a scenario comprising ten randomized obstacles, a randomized goal, one leader UAV and three wingmen was designed (Figure 9). The distinct colors of the obstacles are used to differentiate them and enhance visual clarity. Robustness was verified through perturbations applied to the goal position during frames 80–180 (simulating positioning errors or environmental interference). During training, episodes terminated upon collision or goal arrival, whereas during validation, episodes concluded only upon goal arrival. Simulations demonstrate the model maintains formation and achieves coordinated goal arrival under complex uncertainties. Statistical results in Figure 10 confirm MASAC-CA’s superiority over MASAC and random policy [4] across all metrics (particularly in formation keeping and collision prevention). As depicted in Figure 11 (∆t = 1), collision counts and abruptness statistics over 100 episodes reveal: with increasing obstacle density, MASAC-CA maintains near-constant collision counts and abruptness while MASAC exhibits linear escalation. This superior collision avoidance (CN) performance is attributed to our core designs: an intelligent reward function with explicit safety penalties, a heterogeneous state design enabling real-time leader–follower coordination, and the robust CTDE framework. These elements collectively empower the swarm with enhanced collaborative obstacle avoidance capabilities. The results prove MASAC-CA’s significantly higher flight stability and safety.

6. Conclusions

In this paper, a cooperative path planning method for UAV swarms, termed MASAC-CA, was proposed and evaluated. The approach builds on a heterogeneous MDP formulation with agent-specific state representations—including a terminal flag for the leader and leader-referenced coordinates for wingmen—and a composite reward function that incentivizes simultaneous arrival while satisfying formation, safety, and spatial constraints. The core design philosophy leverages strategic asymmetry in roles and information structures to engender robust symmetries in the collective behavior, such as stable formation topology and synchronized arrival. This symmetry–asymmetry duality is central to the method’s effectiveness. The complex reward scheme is designed to enhance training stability and convergence relative to baseline algorithms. Simulation results demonstrate that MASAC-CA improves mission success rates by ~13% (reaching near 100%), sustains safe formation distance (30–40 m), reduces collisions to near zero, and exhibits robustness in cluttered environments. Moreover, the method maintains high performance in environments with multiple random obstacles, confirming its robustness.

In the future, we will extend the framework to 3D dynamic target pursuit scenarios, which involves addressing several key challenges: expanding the state space to include altitude and 3D orientation, extending the action space for 3D motion control, adapting the reward function for 3D formation keeping (using Euclidean distance) and obstacle avoidance (considering spherical obstacles), and developing strategies to mitigate the increased complexity and curse of dimensionality. The inherent spatial symmetries (e.g., rotational symmetry in formations) and asymmetries (e.g., in obstacle distributions) of 3D environments will be explicitly considered in the state and reward design. While this study validates the algorithm using a fixed formation topology, the proposed heterogeneous MDP and reward architecture provide a foundation that can be extended to support dynamic swarm topologies adapted to different mission phases or environmental constraints. This could involve studying morphological symmetry transitions, where the swarm dynamically reconfigures its symmetric pattern (e.g., from a linear line to a V-shape) in response to environmental cues. Additionally, to enhance practical applicability, we will systematically investigate the impact of realistic communication impairments, such as packet loss and delays, on system performance. This will involve modeling these constraints within the training environment and developing communication-aware policies to improve the robustness of MASAC-CA in real-world wireless network conditions. Furthermore, to bridge the gap between simulation and real-world deployment, future research will focus on the integration of onboard sensors and real-time data fusion techniques, enabling the system to operate autonomously in GPS-denied and communication-limited environments. The heterogeneous MDP architecture and CTDE paradigm validated in this 2D study provide a solid foundation for these extensions. We will subsequently promote its practical implementation in real UAV swarm platforms operating in complex and uncertain environments.