MASP: Scalable Graph-Based Planning Towards Multi-UAV Navigation

Abstract

This work investigates multi-UAV navigation tasks where multiple drones need to reach initially unassigned goals in a limited time. Reinforcement learning (RL) has recently become a popular approach for such tasks. However, RL struggles with low sample efficiency when directly exploring (nearly) optimal policies in a large exploration space, especially with an increased number of drones (e.g., 10+ drones) or in complex environments (e.g., a 3D quadrotor simulator). To address these challenges, this paper proposes Multi-UAV Scalable Graph-based Planner (MASP), a goal-conditioned hierarchical planner that reduces space complexity by decomposing the large exploration space into multiple goal-conditioned subspaces. MASP consists of a high-level policy that optimizes goal assignment and a low-level policy that promotes goal navigation. MASP uses a graph-based representation and introduces an attention-based mechanism as well as a group division mechanism to enhance cooperation between drones and adaptability to varying team sizes. The results demonstrate that MASP outperforms RL and planning-based baselines in task and execution efficiency. Compared to planning-based competitors, MASP improves task efficiency by over 27.92% in a 3D continuous quadrotor environment with 20 drones.

1. Introduction

Navigation is a fundamental task for drones, with broad applications in areas such as object detection and tracking [1,2], logistics and transportation [3,4], and disaster rescue [5,6]. This work focuses on a multi-UAV navigation problem, where multiple drones simultaneously navigate toward a set of initially unassigned goals. In this case, goal assignment and navigation can be jointly optimized, leading to more efficient and effective multi-UAV navigation.

Planning-based approaches have been widely applied in multi-UAV navigation tasks [7,8,9]. However, these methods struggle with complex coordination strategies and require intricate hyperparameter tuning for each specific scenario. Moreover, frequent re-planning at each decision step can be computationally expensive and time-consuming. Conversely, reinforcement learning (RL) demonstrates strong representational capabilities for complex strategies while maintaining low inference overhead once policies are well-trained [10,11]. However, directly learning an end-to-end policy [12,13] in a large exploration space suffers from low sample efficiency, which is more severe as the number of drones or the environmental complexity increases. Therefore, existing methods [14,15] mainly focus on simple scenarios with a few drones.

This paper proposes an RL-based hierarchical framework, Multi-UAV Scalable Graph-based Planner (MASP), to solve this challenge. MASP decomposes a large exploration space into multiple goal-conditioned subspaces by integrating a high-level policy that assigns goals to drones at each high-level step and a low-level policy that navigates drones toward the designated goals at each timestep. The reward function for the high-level policy, Goal Matcher, is designed to encourage each drone to select a goal distinct from those of other drones while minimizing the overall assignment cost. For the low-level policy, Coordinated Action Executor, the reward function promotes efficient navigation and collision avoidance. To ensure scalability to arbitrary numbers of drones, the drones and the goals are represented as two dynamic graphs, where nodes can expand as needed. Goal Matcher leverages an attention-based mechanism over these graphs to optimize goal assignment, as the attention mechanism is well-suited for solving long and variable-length sequences [16]. Recent studies [17,18,19] have also demonstrated its effectiveness in modeling interactions among varying numbers of agents. Coordinated Action Executor employs a group division mechanism that extracts features from only a small subset of drones at a time, significantly reducing the input dimension of drones’ state information while maintaining effective cooperation between drones during action execution.

MASP is compared with planning-based methods and RL-based competitors in two continuous testbeds: multi-agent particle environments (MPE) [20], where drones are abstracted as two-dimensional (2D) particles, and a complex three-dimensional (3D) quadrotor environment (OmniDrones) [21]. Empirical results demonstrate that MASP significantly outperforms its competitors in task and execution efficiency. MASP is applicable to tasks that require effective goal assignment and navigation. It serves as a policy enabling UAVs to rapidly and continuously make decisions based on current states. MASP can also be extended to vision-based applications by integrating it with a visual perception module. More information can be found at the website: https://sites.google.com/view/masp-drones (accessed on 20 June 2025).

The main contributions of this paper include:

• Proposing a hierarchical multi-UAV navigation planner, Multi-UAV Scalable Graph-based Planner (MASP), to decompose a large exploration space into multiple goal-conditioned subspaces, enhancing sample efficiency in a large exploration space.
• Representing drones and goals as graphs and applying an attention-based mechanism and a group division mechanism to ensure scalability to arbitrary numbers of drones and promote effective cooperation.
• Compared to planning-based competitors, enhancing task efficiency by over 19.12% in MPE with 50 drones and 27.92% in OmniDrones with 20 drones (referred to Section 5.5.1), and achieving at least a 47.87% enhancement across varying team sizes (referred to Section 5.5.2).

2. Related Work

2.1. Multi-UAV Navigation

Multi-UAV navigation [10,14,22,23] is a fundamental cooperative task. Existing approaches can be broadly categorized into two types: (1) navigation toward preassigned goals and (2) navigation toward a set of initially unassigned goals. This paper focuses on the latter, as jointly optimizing goal assignment and navigation improves task efficiency. As the number of drones increases, the exploration space grows significantly, making it challenging to find a (nearly) optimal navigation strategy. Some works address this challenge for universal use: Ref. [24] employs imitation learning to constrain the exploration space based on the expert trajectories, while DARL1N [25] restricts the exploration space by only perceiving adjacent neighbors. For multi-UAV navigation, [26,27] adopts independent learning with parameter sharing to cope with the large joint exploration space issue. However, these approaches neglect global information and may diminish the final performance. This work proposes a hierarchical framework that explicitly divides the process into two stages: goal assignment and goal navigation, which is based on prior knowledge of multi-UAV navigation. This method breaks down the exploration space into multiple goal-conditioned subspaces, enabling the search for (nearly) globally optimal strategies in each subspace.

For goal assignment, the Hungarian algorithm [28] uses a weighted bipartite graph to find a minimum-cost match. [29] allocates goals to minimize the longest individual trajectory. However, this method suffers from high computational overhead, making it unsuitable for large-scale multi-UAV systems. M* [30] mitigates this issue by reassigning targets only when potential collisions are detected, but it struggles when multiple drones simultaneously require reassignment. For goal navigation, effectively modeling drone interactions is crucial for collision avoidance and cooperative behavior. Recent works leverage transformer-based architectures [27,31,32] and graph-based methods [14,22] to capture these interactions in dynamic drone populations. Inspired by these advances, a graph-based method is introduced that adapts to large-scale and dynamic drone populations.

2.2. Goal-Conditioned HRL

Goal-conditioned hierarchical reinforcement learning (HRL) is effective across various tasks [33,34], typically using a high-level policy to select subgoals and a low-level policy to predict actions to achieve them. However, existing approaches [35,36,37] struggle in multi-target scenarios, as they fail to establish clear connections between subgoals and multiple targets. HTAMP [38] addresses this challenge for general multi-agent tasks, where the high-level policy directly assigns tasks to agents, and the low-level policy yields actions based on the assigned tasks. This hierarchical paradigm can be extended to multi-UAV scenarios [14,39,40], decomposing the whole navigation task into two stages: goal assignment and goal navigation. Inspired by [14,38], goal-conditioned HRL is employed for multi-UAV navigation.

2.3. Graph Neural Networks

Graph neural networks (GNNs) [41] excel at modeling complex relationships among diverse entities and have been widely applied in various domains, including social networks [42] and molecular structure [43]. In multi-UAV systems [44,45,46], GNNs are effective in capturing interactions among drones. DRGN [22] obtains extra spatial information through graph-convolution-based inter-UAV communication. [47] introduces a two-stream graph multi-agent proximal policy optimization (two-stream GMAPPO) algorithm that models UAVs and other entities as graph-structured data, aggregates the node information, and extracts relevant features. In this paper, MASP adopts graph-based representations to model the relationships between drones and goals.

3. Problem Formulation

This paper considers a multi-UAV navigation task where N drones reach M initially unassigned goals with B obstacles. The objective is to minimize the execution time required for goal assignment and goal navigation. This problem could be formulated as a goal-conditioned Markov Decision Process (MDP) $M=⟨\mathcal{N},\mathcal{G},S,A,P,R,\gamma ⟩$. $\mathcal{N}$ is the set of N drones. $\mathcal{G}$ is a goal set. $S=S_1\times S_2\times \dots \times S_N$ is the joint state space. $A=A_1\times A_2\times \dots \times A_N$ is the joint action space. P is the transition function. R is the reward function. $\gamma$ is the discount factor. Under MDP, the objective is transformed into maximizing the cumulative reward for efficient navigation.

4. Methodology

4.1. Overview

As the joint state and action spaces grow exponentially with the number of drones, multi-UAV navigation tasks face a critical challenge of low sample efficiency in policy learning. To address this challenge, this paper proposes Multi-UAV Scalable Graph-based Planner (MASP), a goal-conditioned HRL framework. MASP consists of a high-level policy ${\pi }_{{\theta }_h}^h\left(g\right|s)$ for goal assignment and a low-level policy ${\pi }_{{\theta }_l}^l\left(a\right|s,g)$ for goal navigation. Both policies are parameterized by neural networks with parameters ${\theta }_h$ and ${\theta }_l$, respectively. As each drone perceives an assigned goal rather than all possible goals, the hierarchical framework decomposes the state and action space into multiple goal-conditioned subspaces, effectively improving sample efficiency.

This work adopts the centralized training with decentralized execution (CTDE) paradigm, where each drone is equipped with its own MASP, ensuring scalability to an arbitrary number of drones. Take drone k as an example. The high-level policy, referred as Goal Matcher (GM), assigns a goal, $g_{t_h}\sim {\pi }_{{\theta }_h}^h\left(g\right|s)\in \mathcal{G}$, to drone k at every high-level step $t_h$. To achieve scalability in the number of drones, the drones and the goals are modeled as two dynamically expanding graphs. Attention-based mechanisms [48] are then applied to compute matching scores between drone k and goals in these graphs, effectively extracting meaningful features from the variable-sized and high-dimensional graph representation. Once drone k receives its assigned goal, the low-level policy, termed Coordinated Action Executor (CAE), outputs its environmental action at each timestep t, denoted as $a_t\sim {\pi }_{{\theta }_l}^l\left(a\right|s,g)\in A$. CAE proposes a group division mechanism to extract relationships among varying drone numbers while reducing the input dimension by focusing only on drones within the same group at a time. Notably, this mechanism is not applicable to GM, as goal assignment requires a global matching across all drones and goals. Incorporating the extracted drone relationships and goal positional information, drone k navigates toward its designated goal while maintaining cooperative behavior.

4.2. The High-Level Policy: Goal Matcher

The Workflow. The high-level policy focuses on goal assignment, a well-known NP-hard maximum matching problem [29,49]. To solve this challenge, this work introduces an RL-based solution, Goal Matcher, which operates in a discrete action space consisting of goal candidates. Its state space includes the positional information of all drones and goals. To ensure scalable representation, MASP constructs two dynamically expandable fully connected graphs: the drone graph, $G_A(X,E)$, and the goal graph, $G_T(X,E)$. The node set $X\in {\mathbb{R}}^{\{N,M\}\times L}$ represents drones or goals, where each node feature encodes positional information with L dimensions. All edge connections are initially set to 1.

As illustrated in Figure 1, a Self-Encoder is used to update node features in $G_A$ or $G_T$, by capturing the spatial relationship among drones or goals. Inspired by the effectiveness of self-attention mechanisms in handling long and variable-length sequences [16,50], GM applies self-attention [48] over graphs to compute $X_{attn}\in {\mathbb{R}}^{\{N,M\}\times L}$, in $G_A$ or $G_T$. The updated node features $Y\in {\mathbb{R}}^{\{N,M\}\times L}$ are obtained by adding $X_{attn}$ to X, allowing each node to encode spatial information with its neighbors. Then, GM introduces a Cross-Encoder to capture the relationship between drone k and goals, computing a matching score for goal assignment. The navigation distance is a key factor in multi-UAV navigation. Since obstacle avoidance is handled by the low-level policy, the simple L2 distance between drone k and each goal is adopted as the navigation distance, denoted as $D\in {\mathbb{R}}^{M\times 1}$. The features ${Y_A}^k$, $Y_T$, and D are passed through a multi-layer perceptron (MLP) layer to extract spatial relationships between drone k and goals. A softmax operation is then applied to the MLP output, yielding the matching score, $S^k\in {\mathbb{R}}^{M\times 1}$. This matching score represents the probability of assigning each goal to drone k. Finally, drone k selects the goal with the highest matching score at each high-level step. More details are provided on the website.

Reward Design. The reward for drone k, $R_{GM}^k$ is formulated as follows:

$$R_{GM}^k=\left\{\begin{array}{ll}0,& g_m^k==g_h^k;\\ -(1+{\displaystyle \frac{N_{repeat}}{N}}),& g_m^k\ne g_h^{k}and{N}_{repeat}>0;\\ -(1-{\displaystyle \frac{D_h}{D_m}}),& g_m^k\ne g_h^{k}and{N}_{repeat}=0.\end{array}\right.$$

(1)

To encourage selecting a goal that differs from other drones, a repeated penalty is introduced, where $N_{repeat}$ signifies the number of other drones assigned the same goal as drone k. Additionally, a distance-based reward is employed to minimize the total travel distance. Specifically, the Hungarian algorithm [28] is used as an expert. $g_m^k$ represents the predicted goal from GM for drone k, and $g_h^k$ denotes the goal assigned by the Hungarian algorithm. $D_m$ denotes the total distance cost from the predicted goals to agents, and $D_h$ denotes the total distance cost from the Hungarian-assigned goals to agents. If MASP has a lower distance cost than the Hungarian algorithm, $R_{GM}^k$ is positive, incentivizing GM to learn a better goal assignment strategy.

4.3. The Low-Level Policy: Coordinated Action Executor

Coordinated Action Executor takes the goal assigned by GM and navigates drone k toward this goal while minimizing collisions. In CAE, the action space consists of the environmental actions available to drone k, while the state space includes the locations and velocities of all drones, along with the location of the designated goal.

Group Division. During goal navigation, it is essential to extract drone relationships to ensure cooperation and collision avoidance. However, as the number of drones increases, directly using the state information of all drones results in high input dimensionality, making training inefficient. To solve this, MASP introduces a group division mechanism by focusing only on drones within the same group at a time. Each group is assumed to contain $N_g$ drones. As illustrated in Figure 2, all drones, excluding drone k, are first divided into groups of $(N_g-1)$ drones. If the drones cannot be evenly divided, some drones are randomly assigned to multiple groups to ensure group formation. Then, drone k is added to each group repeatedly, maintaining connections with all drones across groups.

The Workflow. CAE partitions drones into groups and represents each group as a fully connected subgraph, $G_S(X,E)$. In each subgraph, the node set $X\in {\mathbb{R}}^{N_g\times L}$ encodes the positions and velocities of the drones, while the edges E are set to 1. Notably, each subgraph is newly constructed and distinct from the graphs used in GM. To capture drone relationships efficiently, CAE introduces Group Information Fusion that applies graph convolutional networks (GCNs) to each subgraph and then aggregates the node features of drone k across all subgraphs using a mean operation over the graph–number dimension. This process yields an updated node feature of drone k, denoted as ${Y_S}^k\in {\mathbb{R}}^{1\times L}$, which encodes the relationships between drone k and all other drones. Next, an MLP layer is fed into the positional embedding of the designated goal $T^k\in {\mathbb{R}}^{1\times L}$ and the updated node feature of drone k, extracting the relationship between drone k and its assigned goal. Finally, based on the MLP output, $H^k\in {\mathbb{R}}^{1\times D_A}$, where $D_A$ represents the action dimension, CAE generates an environmental action to guide drone k toward its goal. More details are provided on the website.

Reward Design. The reward, $R_{CAE}^k$, for drone k in CAE encourages goal-conditioned navigation while minimizing collisions. $R_{CAE}^k$ is a linear combination of three components:

• A complete bonus for successfully reaching the assigned goal, $R_b$, which is 1 if agent k reaches its assigned goal, otherwise 0.
• A distance penalty for task efficiency, $R_d$, which is the negative L2 distance between agent k and its assigned goal.
• A collision penalty for collision avoidance, $R_c$, computed based on the number of collisions at each time step.

The formulation is as follows:

$$R_{CAE}^k=\alpha R_b^k+\beta R_d^k+\gamma R_c^k,$$

(2)

where $\alpha$, $\beta$ and $\gamma$ are the coefficients of $R_b^k$, $R_d^k$ and $R_c^k$, respectively.

4.4. Multi-UAV Scalable Graph-Based Planner Training

MASP, following the goal-conditioned MARL framework, trains two policy networks for Goal Matcher and Coordinated Action Executor separately, which are optimized by maximizing the accumulated reward in the entire episode via RL. Both GM and CAE are optimized using multi-agent proximal policy optimization (MAPPO) [12], an extension of proximal policy optimization (PPO) [51] specifically designed for multi-agent settings.

As shown in Algorithm 1, MASP takes in the drone graph $G_A$, the goal graph $G_T$, and the joint state s of all agents. Then, it produces the final policy ${\pi }_{{\theta }_h}^h$ for GM and ${\pi }_{{\theta }_l}^l$ for CAE. First, MASP initializes the number of agents along with several training parameters, including maximal steps, the high-level step, parameters in GM, parameters in CAE, GM reply buffer $D_h$, and CAE reply buffer $D_l$ (Line 1). GM assigns the goals to each agent k every high-level step (Lines 7–10) and receives the reward $r_{t_h}^k$ and the subsequent state $s_{t_h+1}^k$, where $t_h$ represents the step-count in $D_h$ (Line 11 and 12). Thereafter, ${\theta }_h$ for the policy network ${\pi }_{{\theta }_h}^h$ and ${\varphi }_h$ for the value network $V_{{\varphi }_h}^h$ in GM are updated. Regarding CAE, it outputs the action $a_{t_l}^k$ for each agent k, and stores a group of data in $D_l$ at each timestep (Line 15–18). Similarly, ${\theta }_l$ for the policy network ${\pi }_{{\theta }_l}^l$, and ${\varphi }_l$ for the value network $V_{{\varphi }_l}^l$ are updated in CAE.

Algorithm 1 Training Procedure of MASP

Require:  The drone graph $G_A$ and the goal graph $G_T$. The joint state s of all agents. Ensure:  Final policy ${\pi }_{{\theta }_h}^h$ for Goal Matcher and ${\pi }_{{\theta }_l}^l$ for Coordinated Action Executor. 1:   Initialize: agents number N, maximal steps in an episode $T_e$, high-level step $T_h$, params in GM ${\theta }_h$, ${\varphi }_h$, params in CAE ${\theta }_l$, ${\varphi }_l$, GM buffer $D_h$, CAE buffer $D_l$. 2:   while ${\theta }_h$ and ${\theta }_l$ not converges do 3:      Reset environment. 4:      Initialize: the step-count $t_h\leftarrow 0$ in $D_h$, the step-count $t_l\leftarrow 0$ in $D_l$. 5:      while $t_l<T_e$ and not terminal do 6:         if $t_h\%T_h==0$ then 7:             Update $G_A$ and $G_T$. 8:             Set $s_{t_h}$ as $G_A$ and $G_T$. 9:             Calculate ${\pi }_{{\theta }_h}^h\left(g_{t_h}\right|s_{t_h})$ and $V_{{\theta }_h}^h\left(s_{t_h}\right)$. 10:        Perform $g_{t_h}^k\sim {\pi }_{{\theta }_h}^h\left(g_{t_h}^k\right|s_{t_h}^k)$ for each agent k. 11:        Receive $r_{t_h}^k$ and $s_{t_h+1}^k$ for each agent k. 12:        Store $(s_{t_h}^k,g_{t_h}^k,{\pi }_{{\theta }_h}^h\left(g_{t_h^k}\right|s_{t_h}^k),V_{{\varphi }_h}^h\left(s_{t_h}\right),r_{t_h}^k,s_{t_h+1})$ in $D_h$ for each agent k. 13:        $t_h\leftarrow t_h+1$ 14:       end if 15:       Calculate ${\pi }_{{\theta }_l}^l\left(a_{t_l}\right|s_{t_l},g_{t_h})$ and $V_{{\varphi }_l}^l\left(s_{t_l}\right)$. 16:       Perform $a_{t_l}^k\sim {\pi }_{{\theta }_l}^l\left(a_{t_l}^k\right|s_{t_l}^k,g_{t_h}^k)$ for each agent k. 17:       Receive $r_{t_l}^k$ and $s_{t_l+1}^k$ for each agent k. 18:       Store $(s_{t_l}^k,a_{t_l}^k,{\pi }_{{\theta }_l}^l\left(a_{t_l}^k\right|s_{t_l}^k,g_{t_h}^k),V_{{\varphi }_l}^l\left(s_{t_l}^k\right),r_{t_l}^k,s_{t_l+1}^k)$ in $D_l$. 19:       $t_l\leftarrow t_l+1$ 20:    end while 21:    Perform update of ${\theta }_h$, ${\varphi }_h$, ${\theta }_l$ and ${\varphi }_l$. 22: end while

5. Experiments

5.1. Testbeds

To assess the effectiveness of this approach, two continuous testbeds are selected as shown in Figure 3: a 2D environment, MPE [20], and a 3D environment, OmniDrones [21], with a substantial number of drones.

MPE is a classical 2D environment where drones are abstracted as particles to enable fast verification of this method. The available environmental actions are discrete, including Up, Down, Left, and Right. Each action corresponds to a force with a fixed value in a specific direction. Therefore, the movement of the drones is continuous. The collision between drones causes a disruptive bounce-off, reducing navigation efficiency. The experiment is conducted with randomized spawn locations for N drones, N landmarks, and B obstacles. The drones, the obstacles, and the landmarks are circular with a radius of 0.1 m, 0.1 m, and 0.05 m, respectively. Experiments consider $N\in \{5,20,50\}$ and $B\in \{5,10,30\}$ on the maps of 4 m², 64 m², and 400 m², with the horizons of the environmental steps of 18, 45, and 90, respectively.

OmniDrones is an efficient and flexible 3D simulator for drone control, incorporating precise aerodynamic models and increasing the exploration space. The continuous action space of each drone is the throttle for each motor. Collisions directly lead to crashes and task failures, increasing task difficulty. The drones are modeled based on [52], which offers very high-fidelity simulation of sensors, actuators, and built-in controllers. This enables more efficient sim-to-real transfer of control policies. Specifically, the drones used are hummingbird models, each with a mass of 0.716 kg and an arm length of 0.17 m. Each drone is equipped with four motors. The moments of inertia are 0.007 kg·m² for the xx and yy axes, 0.012 kg·m² for the zz axis, and zero for all other components. The thrust coefficient is $8.54858\times {10}^{-6}$, and the maximum rotational velocity is 838 revolutions per second (rps). The landmarks are virtual balls of 0.05 m radius. Experiments consider $N\in \{5,20\}$ drones, where the spawn locations for both drones and landmarks are randomly distributed on the maps sized at 36 m² and 256 m², respectively. In addition, the horizons of the environmental steps are 300 for 5 drones and 500 for 20 drones.

5.2. Implementation Details

Each RL training is performed over three random seeds for a fair comparison. Each evaluation score is expressed in the format of “mean (standard deviation)”, averaged over a total of 300 testing episodes, i.e., 100 episodes per random seed. The high-level step is set to 3 timesteps, and the low-level step is set to 1 timestep. Experiments consider $N_g=3$ and $L=32$ (The sensitivity analysis is detailed in Section 5.8).

5.3. Evaluation Metrics

There are six statistical metrics to capture different characteristics of the navigation strategy.

• Success Rate (SR): This metric measures the average ratio of goals reached by the drones to the total goals per episode.
• Steps: This metric represents the average timesteps required to reach 100% Success Rate per episode.
• Collisions: This metric denotes the average number of collisions per step.
• Execution Time: This metric represents the time required for evaluation, measured in milliseconds (ms).
• Memory Usage: This metric represents the memory size required for evaluation, measured in Gigabyte (GB).
• Network Parameters: This metric refers to the number of network parameters needed for each RL-based method.

Steps is taken as the primary metric for measuring task efficiency. In tables, the value in bold signifies that this method earns the highest performance among all competitors in terms of this metric. The backslash in Steps denotes that the drones can not reach the target 100% Success Rate in any episode. An upward arrow (↑) signifies that higher values indicate better performance, while a downward arrow (↓) indicates that lower values are preferable.

5.4. Baselines

MASP is compared with three representative planning-based approaches (ORCA, RRT*, Voronoi) and four prominent RL-based solutions (MAPPO, DARL1N, HTAMP, MAGE-X). In planning-based approaches, the Hungarian algorithm, denoted (H), is utilized for goal assignment.

• ORCA [7]: ORCA is an obstacle avoidance algorithm that excels in multi-UAV scenarios. It predicts the movements of surrounding obstacles and other drones, and then infers a collision-free velocity for each drone.
• RRT* [8]: RRT* is a sample-based planning algorithm that builds upon the RRT algorithm [53]. It first finds a feasible path by sampling points in the scenario and then iteratively refines the path to achieve an asymptotically optimal solution.
• Voronoi [9]: A Voronoi diagram comprises a set of continuous polygons, with a vertical bisector of lines connecting two adjacent points. By partitioning the map into Voronoi units, a safe path can be planned from the starting point to the destination.
• MAPPO [12]: MAPPO is a straightforward extension of PPO in the multi-agent setting, where each agent is equipped with a policy with a shared set of parameters. MAPPO is updated based on the aggregated trajectories from all drones. Additionally, an attention mechanism is applied to enhance the model’s performance.
• (H)MAPPO [12]: (H)MAPPO adopts a hierarchical framework, with the Hungarian algorithm as the high-level policy for goal assignment and MAPPO as the low-level policy for goal achievement.
• DARL1N [25]: DARL1N is under an independent decision-making setting for large-scale agent scenarios. It breaks the curse of dimensionality by restricting the agent interactions to one-hop neighborhoods. This method is extended to multi-UAV navigation tasks.
• HTAMP [38]: This is a hierarchical method for task and motion planning via reinforcement learning, integrating high-level task generation with low-level action execution.
• MAGE-X [14]: This is a hierarchical approach applied in multi-UAV navigation tasks. It first centrally allocates target goals to the drones at the beginning of the episode and then utilizes GNNs to construct a subgraph only with important neighbors for higher cooperation.

5.5. Main Results

5.5.1. Training with a Fixed Team Size

MPE. The training curves is represented in Figure 4 and the evaluation performance is represented in

Table 1. Performance of MASP, planning-based baselines and RL-based baselines with $N=5,20,50$ drones in MPE.

Agents	Metrics	(H)ORCA	(H)RRT*	(H)Voronoi	(H)MAPPO	MAPPO	DARL1N	HTAMP	MAGE-X	MASP
N = 5	Steps↓	16.38(0.53)	7.58(1.49)	8.86(1.27)	8.04(1.53)	8.69(1.24)	∖	∖	14.75(0.60)	7.54(1.37)
	SR↑	0.72(0.01)	0.99(0.01)	0.98(0.01)	1.00(0.00)	1.00(0.00)	0.74(0.02)	0.65(0.05)	1.00(0.00)	1.00(0.00)
	Collisions↓	0.57(0.03)	3.41(0.14)	3.44(0.25)	0.42(0.02)	1.32(0.38)	2.80(0.53)	1.63(0.43)	0.45(0.02)	0.47(0.01)
N = 20	Steps↓	34.28(3.22)	26.86(2.27)	28.72(2.68)	27.43(2.49)	∖	∖	∖	∖	24.41(2.66)
	SR↑	0.95(0.03)	0.99(0.01)	1.00(0.00)	1.00(0.00)	0.71(0.02)	0.62(0.03)	0.05(0.01)	0.75(0.02)	1.00(0.00)
	Collisions↓	0.49(0.04)	1.69(0.05)	1.80(0.03)	0.75(0.05)	2.53(1.02)	3.22(1.23)	1.54(0.05)	1.78(0.14)	0.14(0.01)
N = 50	Steps↓	60.66(4.76)	58.68(3.89)	57.88(5.43)	51.88(4.06)	∖	∖	∖	∖	47.46(5.93)
	SR↑	0.97(0.01)	0.99(0.01)	0.99(0.01)	0.99(0.01)	0.01(0.01)	0.49(0.02)	0.02(0.01)	0.23(0.01)	0.99(0.01)
	Collisions↓	0.30(0.02)	1.39(0.14)	1.31(0.42)	0.42(0.75)	0.78(0.26)	1.14(0.34)	1.35(0.21)	1.69(0.37)	0.23(0.01)

. MASP outperforms all methods with the least training data. Its advantage in Steps, Success Rate, and Collisions becomes more evident as drone numbers increase. Especially in scenarios with 50 drones, MASP achieves a nearly 100% Success Rate, while other RL baselines fail to complete the task in any episode. Among RL baselines, except (H)MAPPO, DARL1N shows the best performance at $N=50$ with a 49% Success Rate, while it loses its advantage at $N=5$. This implies that when the scenario only has a few drones, DARL1N may lead to the absence of one-hop neighbors, thereby being harmful to the drones’ decisions. MAPPO has only a 0.01% Success Rate at $N=50$, showing that a single policy has difficulty in being optimized in a large exploration space. (H)MAPPO has comparable performance with MASP but requires 8.52% more Steps at $N=50$. MAGE-X has a 23% Success Rate at $N=50$, suggesting its MLP-based goal assignment encounters challenges in preventing collisions between a large number of drones. Although HTAMP is also in a hierarchical manner, its MLP backbone fails to perceive the relationship between drones and targets, exhibiting inferior performance.

Regarding planning-based methods, (H)RRT* and (H)Voronoi have comparable performance and require approximately 19.12% more Steps than MASP at $N=50$. This indicates that although (H)RRT* and (H)Voronoi can handle obstacle-rich scenarios, their cooperation strategies remain less effective than MASP. (H)ORCA performs the worst, requiring at least 21.76% more Steps than MASP, highlighting its difficulty in finding shorter navigational paths.

OmniDrones. The training performance is represented in Figure 5 and the evaluation results are represented in

Table 2. Performance of MASP, planning-based baselines and RL-based baselines with $N=5,20$ drones in OmniDrones.

Agents	Metrics	(H)ORCA	(H)RRT*	(H)Voronoi	(H)MAPPO	MAPPO	DARL1N	HTAMP	MAGE-X	MASP
N = 5	Steps↓	260.10(10.02)	236.10(9.47)	173.89(12.91)	185.34(7.14)	∖	∖	∖	231.26(19.32)	138.35(13.57)
	SR↑	0.98(0.03)	0.97(0.01)	0.99(0.02)	1.00(0.00)	0.50(0.09)	0.68(0.02)	0.32(0.08)	0.99(0.01)	1.00(0.00)
	Collisions↓	0.00(0.00)	0.01(0.01)	0.02(0.01)	0.01(0.01)	0.01(0.01)	0.01(0.01)	0.01(0.01)	0.01(0.01)	0.00(0.00)
N = 20	Steps↓	449.10(14.86)	438.23(11.02)	437.43(14.24)	335.74(13.07)	∖	∖	∖	∖	315.31(14.52)
	SR↑	0.97(0.03)	0.97(0.01)	0.93(0.01)	0.96(0.01)	0.15(0.03)	0.15(0.01)	0.14(0.03)	0.14(0.03)	0.97(0.01)
	Collisions↓	0.01(0.03)	0.03(0.03)	0.01(0.03)	0.01(0.01)	0.03(0.02)	0.05(0.01)	0.04(0.02)	0.08(0.02)	0.01(0.01)

. MASP is superior to all competitors, achieving a 100% Success Rate at $N=5$ and 96% at $N=20$ in this complex 3D environment. Most RL baselines, except MAGE-X and (H)MAPPO, achieve only around 50% Success Rate at $N=5$, failing to address the navigation problem in complex exploration space. MAGE-X drops to a 15% Success Rate at $N=20$. This reveals that its MLP-based goal assignment struggles with complex environments with increased drone numbers.

In contrast to RL baselines, most planning-based baselines achieve over 90% Success Rate. This is primarily owing to the Hungarian algorithm, ensuring that drones are assigned different goals. For Steps metric, the planning-based baselines require at least 27.92% more Steps than MASP at $N=20$. Among planning-based baselines, (H)Voronoi performs best at $N=5$ but shows comparable performance at $N=20$. This indicates that its partitioning strategy struggles to figure out efficient navigation paths in complex 3D environments as the number of drones increases.

5.5.2. Varying Team Sizes Within an Episode

Due to unstable communication or drone loss, the experiments further consider scenarios where the team size decreases during an episode. With more goals than drones, some drones may need to first reach some goals and then adjust their plans to navigate toward the remaining goals. The Success Rate is defined as the ratio of the goals reached during the navigation to the total number of goals. “$N_1\Rightarrow N_2$” is used to denote that an episode starts with $N_1$ drones and switches to $N_2$ after one-third of the total timesteps. The number of goals is $N_1$. Since MASP is trained with a fixed team size, this setup presents a zero-shot generalization challenge. Experiments adopt MASP model trained with $N_1$ drones. As RL baselines lack generalization in both network architecture and evaluation performance, MASP is only compared with planning-based methods. In addition, the Hungarian algorithm is originally designed for scenarios with equal numbers of drones and goals; therefore, it is applied on a subset of unreached goals that matches the drone count at each high-level step.

As shown in

Table 3. Performance of MASP and planning-based baselines with a varying team size in MPE.

Agents	Metrics	(H)ORCA	(H)RRT*	(H)Voronoi	MASP
$20\Rightarrow 10$	Steps↓	43.64(3.86)	40.68(3.37)	40.18(3.41)	22.38(3.04)
	SR↑	0.89(0.01)	0.92(0.01)	0.93(0.01)	1.00(0.00)
	Collisions↓	0.18(0.02)	0.72(0.14)	0.86(0.13)	0.16(0.03)
$50\Rightarrow 20$	Steps↓	∖	89.18(4.88)	88.73(4.27)	46.49(4.49)
	SR↑	0.86(0.01)	0.91(0.01)	0.91(0.02)	1.00(0.00)
	Collisions↓	0.15(0.03)	0.56(0.05)	0.67(0.03)	0.15(0.01)

, the advantage of MASP becomes more apparent in the setting of varying drone numbers. In the $50\Rightarrow 20$ scenario in MPE, MASP consumes around 47.87% fewer Steps than (H)RRT* and (H)Voronoi.

Table 4. Performance of MASP and planning-based baselines with a varying team size in OmniDrones.

Agents	Metrics	(H)ORCA	(H)RRT*	(H)Voronoi	MASP
$20\Rightarrow 10$	Steps↓	∖	∖	490.35(15.63)	426.73(18.50)
	SR↑	0.68(0.04)	0.61(0.04)	0.85(0.03)	0.94(0.01)
	Collisions↓	0.00(0.00)	0.01(0.01)	0.01(0.01)	0.01(0.01)

presents the evaluation results for zero-shot generalization in OmniDrones. The planning-based baselines fail to reach a 100% Success Rate in any episode, whereas MASP maintains an average Success Rate of 94%. This highlights the superior flexibility and adaptability of the goal assignment strategy compared to the Hungarian algorithm.

5.6. Robustness Evaluation in Noisy and Realistic Environments

To evaluate MASP’s applicability to real drones, experiments are conducted in environments with sensor noise and time delay. Concretely, experiments are conducted on Omnidrones with 5 and 20 drones, as well as with varying drone numbers from 20 to 10. The position of each drone is perturbed by uniform noise in the range [−0.1 m, 0.1 m], and a time delay of two timesteps is introduced. As RL-based baselines show inferior performance compared to planning-based competitors in Omnidrones, MASP is only compared with planning-based methods in this setting. MASP is trained and tested in these scenarios, with randomized initial states.

The evaluation results are presented in

Table 5. Robustness evaluation in Noisy and realistic environments of MASP and planning-based baselines with $N=5,20$ drones and a varying team size in OmniDrones.

Agents	Metrics	(H)ORCA	(H)RRT*	(H)Voronoi	MASP
$N=5$	Steps↓	228.05(14.52)	269.20(13.78)	203.65(10.71)	146.38(10.83)
	SR↑	0.94(0.02)	0.87(0.03)	0.93(0.02)	0.98(0.02)
	Collisions↓	0.01(0.01)	0.02(0.01)	0.01(0.01)	0.01(0.01)
$N=20$	Steps↓	408.10(10.40)	484.50(16.95)	453.35(13.38)	334.62(11.53)
	SR↑	0.98(0.02)	0.92(0.02)	0.91(0.03)	0.96(0.01)
	Collisions↓	0.01(0.01)	0.01(0.01)	0.01(0.01)	0.01(0.01)
$20\Rightarrow 10$	Steps↓	∖	∖	∖	439.62(13.02)
	SR↑	0.75(0.02)	0.57(0.02)	0.85(0.02)	0.90(0.01)
	Collisions↓	0.02(0.01)	0.01(0.01)	0.01(0.01)	0.01(0.01)

. Although sensor noise and time delay affect the overall performance of all methods, MASP consistently outperforms planning-based baselines, requiring at least 28.12% fewer Steps with 5 drones and 18.01% fewer Steps with 20 drones. In scenarios with varying drone numbers, planning-based baselines fail to complete the navigation in any episode, leaving Steps metric without value. For Success Rate, MASP achieves 90%, compared to 57∼85% for planning-based baselines. It indicates that the reward design for MASP promotes collision avoidance, enabling it to better handle positional disturbances. Additionally, the goal assignment of MASP encourages diverse target allocation, further mitigating the effects of noise and delay.

5.7. Computational Cost

Computational cost experiments in terms of time and memory are conducted in MPE with 100 drones to verify MASP’s feasibility for large-scale UAV scenarios. As shown in

Table 6. Performance of MASP on memory usage and execution time with N = 100 drones in MPE.

Agents	Metrics	GM	CAE	MASP
N = 100	Memory Usage↓	1.28	0.72	2
	Execution Time↓	11(1)	7(1)	18(2)

, a single inference of MASP requires approximately 2 GB memory size, comprising 1.28 GB for GM and 0.72 GB for CAE. GM uses an attention-based graph mechanism that considers all agent and goal positions for global matching, while CAE employs a group division mechanism. Therefore, the memory usage of GM is relatively larger than that of CAE. Notably, 2 GB memory usage is far less than 8∼16 GB capacity of Jetson Xavier NX, commonly used in real drones. For inference time, experiments consider the time that MASP costs for inferring an action at a time. MASP takes around 18 ms, comprising 11 ms for GM and 7 ms for CAE. This is affordable for a normal drone at a speed of 0∼5 m/s.

As most baselines struggle with 100-drone scenarios, execution time comparisons between MASP and the baselines are conducted in MPE with 50 drones. In

Table 7. Average performance of MASP and baselines on the execution time.

Methods	ORCA	RRT*	Voronoi	MAPPO	DARL1N	HTAMP	MAGE-X	MASP
Execution Time↓	39(6)	456(10)	44(5)	13(4)	16(3)	14(3)	20(5)	14(2)

, MASP is at least 3× more efficient than planning-based competitors. Compared to the best planning-based competitor, (H)RRT*, MASP’s computation time is 32× faster than it. While MASP has a comparable execution efficiency to RL-based methods, it outperforms them in task performance.

5.8. Sensitivity Analysis

Group Size.

Table 8. Performance of $N_g$ = 2, 3, and 5. P denotes required network parameters when $N_g$ = 2.

Agents	Metrics	2	3	5
N = 20	Steps↓	28.85(2.12)	24.41(2.66)	23.78(2.35)
	SR↑	1.00(0.00)	1.00(0.00)	1.00(0.00)
	Collisions↓	0.15(0.01)	0.14(0.01)	0.14(0.01)
	Network Parameters↓	P	~2P	~5P

shows a sensitivity analysis of the group size, $N_g$, for group division in MPE with 20 drones. When $N_g$ is too small, it hinders capturing the correlations between drones. A larger $N_g$ improves the performance but increases required network parameters (i.g., the results of metric ‘Network Parameters’ in Table 8), depreciating the training efficiency. The performance improvement of MASP is negligible when $N_g$ exceeds 3. Thus, $N_g=3$ is selected to balance the final performance and the training efficiency.

The High-level Step. Evaluation performance is reported in

Table 9. Performance of High-level Steps = 1, 3, and 10.

Agents	Metrics	1	3	10
N = 20	Steps↓	24.23(2.11)	24.41(2.66)	29.59(2.43)
	SR↑	1.00(0.00)	1.00(0.00)	1.00(0.00)
	Collisions↓	0.12(0.01)	0.14(0.01)	0.16(0.01)
	Execution Time↓	45(7)	13(4)	4(2)

to assess the sensitivity to high-level steps in MPE with 20 drones. Fewer high-level steps lead to more frequent goal assignment adjustments, improving the evaluation performance. However, it also increases execution time. The results indicate that the performance remains comparable when high-level steps are fewer than 3. To balance evaluation performance and execution time, the high-level step is set to 3.

Feature Embeddings. The evaluation results in

Table 10. Performance of L = 8, 32, and 64. P denotes required network parameters when L = 8.

Agents	Metrics	8	32	64
N = 20	Steps↓	28.66(2.73)	24.41(2.66)	24.82(2.00)
	SR↑	1.00(0.00)	1.00(0.00)	1.00(0.00)
	Collisions↓	0.18(0.01)	0.14(0.01)	0.13(0.01)
	Network Parameters↓	P	~4P	~9P

analyze the sensitivity of feature embedding dimensions, L, in MPE with 20 drones. As L decreases, training efficiency improves due to fewer required network parameters (i.g., the results of the metric ‘Network Parameters’ in Table 10). However, lower-dimensional embeddings may struggle to capture the complex features of drones and goals. The results suggest that choosing $L=32$ achieves a balance between evaluation performance and training efficiency.

5.9. Ablation Studies

To illustrate the effectiveness of each component of MASP, five variants of the method are evaluated in MPE with 20 drones:

• MASP w. RG: GM is replaced by random sampling, assigning each drone a random yet distinct goal.
• GM w.o. Graph: An MLP layer is adopted as an alternative to GM for assigning goals to drones at each high-level step. For drone k, the MLP layer takes in the concatenated position information of all drones and goals, with the position information of drone k listed first.
• CAE w.o. Graph: An MLP layer is employed as an alternative to CAE to capture the correlation between drones and goals. The input for drone k comprises the position of all drones and the assigned goal for drone k.
• MASP w. GraphSAGE: GM and Group Information Fusion module in CAE are replaced with GraphSAGE [54]. GraphSAGE generates embeddings by sampling and aggregating features from a node’s local neighborhood.
• MASP w. GATv2: GM and Group Information Fusion module in CAE are replaced with GATv2 [55]. GATv2 is a dynamic graph attention variant of GAT [56].

To demonstrate the performance of MASP and its variants, the training curves are reported in Figure 6 and the evaluation results are shown in

Table 11. Performance of MASP and RL variants with $N=20$ drones in MPE.

Agents	Metrics	MASP w. RG	GM w.o. Graph	CAE w.o. Graph	MASP w. GraphSAGE	MASP w. GATv2	MASP
N = 20	Steps↓	35.91(2.73)	∖	28.25(3.21)	38.97(3.22)	28.17(2.40)	24.41(2.66)
	SR↑	1.00(0.00)	0.04(0.01)	1.00(0.00)	0.85(0.04)	0.98(0.03)	1.00(0.00)
	Collisions↓	0.25(0.02)	3.32(0.02)	0.17(0.01)	1.26(0.03)	0.83(0.02)	0.14(0.01)

. MASP outperforms its counterparts with a 100% Success Rate and the fewest Steps. GM w.o. Graph degrades the most, indicating difficulties in perceiving the correlations between drones and goals without the graph-based Self-Encoder and Cross-Encoder. MASP w. RG consistently assigns different goals to drones and only trains the low-level policy, leading to the fastest training convergence. However, as shown in the evaluation performance in Table 11, it requires 32.02% more Steps due to random goal assignment. CAE w.o. Graph is slightly inferior to MASP with slower training convergence and 13.59% more Steps in the evaluation. This suggests that GCN in CAE better captures the relationships between drones and assigned goals than the MLP layer. MASP w. GATv2 and MASP show comparable training performance, but MASP achieves 13.35% fewer Steps during evaluation. Although MASP w. GATv2 is able to perceive all agents with different attentions, it lacks a specific design like MASP that first models the interaction of nodes in the same graph and then the interaction of nodes in different graphs. MASP w. GraphSAGE reduces computational complexity by only interacting with a fixed number of neighbors. However, due to its limited perception, MASP w. GraphSAGE may fail to achieve global goal assignment, sometimes assigning multiple agents the same goal. Therefore, it requires 37.36% more Steps than MASP.

5.10. Strategies Analysis

As depicted in Figure 7, the learned strategies of DARL1N and MASP are visualized to showcase high cooperation among MASP drones. In some cases, drones initially select the same goal (e.g., the drone in the red hexagon and the drone in the yellow hexagon). In Figure 7a, two DARL1N drones fail to adjust their strategies and finally stay at the same goal. In contrast, Figure 7b illustrates that when the red MASP drone notices the yellow MASP drone moving toward the same goal, it quickly adjusts its strategy to navigate toward another unreached goal. This cooperation strategy allows MASP drones to occupy different goals, achieving a 100% Success Rate.

6. Conclusions

This paper proposes a goal-conditioned hierarchical planner, Multi-UAV Scalable Graph-based Planner, to improve data efficiency and cooperation in navigation tasks with a varying and substantial number of drones. Thorough experiments demonstrate that MASP achieves higher task and execution efficiency than planning-based baselines and RL competitors in MPE and Omnidrones. Concretely, MASP increases execution efficiency by at least 3× (see Section 5.7). It also enhances task efficiency by over 19.12% in MPE with 50 drones and 27.92% in OmniDrones with 20 drones (see Section 5.5.1), achieving at least a 47.87% enhancement across varying team sizes (see Section 5.5.2).

7. Limitation

Although this method proposes a scalable and promising way for varying numbers of agents in multi-agent navigation, there still remains some limitations. First, this work focuses on multi-agent navigation where agents move toward initially unassigned goals. Leveraging prior knowledge, the task is naturally decomposed into goal assignment and goal navigation. To this end, a hierarchical framework is proposed with a high-level policy for assigning goals and a low-level policy for navigating to them. This intuitive decomposition avoids significant suboptimality. It is acknowledged that not all tasks can be heuristically decomposed based on prior knowledge. In complex multi-agent scenarios, tasks may lack clearly defined subtasks for each agent. A possible solution is to learn latent subtask representations [57,58], allowing the high-level policy to assign them for coordinated execution. Second, this approach assumes that every agent can receive global pose. In real deployments, communication among agents may meet challenges of delayed or missing data. This results in task failure or reduced task efficiency. To mitigate this, MASP needs an adaptive bandwidth allocator [59], and integrates a robust state estimator [60]. Finally, this method currently lacks vision-based mapping and dynamic perception modules. MASP intends to integrate lightweight vision sensors and adopt incremental vision-based mapping techniques [61,62].