Hierarchical Reinforcement Learning for Large-Scale Heterogeneous UAV Mission Planning via MCTS and Transformer

Highlights

What are the main findings?

• A Hierarchical Deep Reinforcement Learning framework partitions the LSH-TOP into distinct sub-problems. Within this architecture, MCTS determines global workload distribution, and a Transformer-based model generates individual agent trajectories.
• The C-ACT protocol addresses convergence limitations in constrained environments. This protocol stabilizes the training curves and ensures steady convergence by managing the transition from relaxed to strict constraints.

What are the implications of the main findings?

• The framework establishes a structural basis for coordinating heterogeneous multi-UAV fleets during emergency SAR missions. Compared to conventional heuristics, this approach yields up to 53.27% higher objective values and improves over the strongest baseline (MCTS + OR) by 0.63–8.51%, while balancing workload distribution across problem scales and delivering solutions within 24.30 s at 300 targets.
• This hierarchical structure achieves sub-quadratic computational scaling, compared to the O(N^K) growth of joint formulations. Consequently, this approach supports real-time decision-making for post-disaster response operations.

Abstract

Post-disaster Search and Rescue (SAR) missions demand rapid coordination of Heterogeneous Unmanned Aerial Vehicle (UAV) fleets under stringent payload and flight range limitations. Traditional heuristic solvers struggle to solve the Large-Scale Heterogeneous Team Orienteering Problem (LSH-TOP) within operational time limits due to the coupled complexity of task allocation and route planning. A Hierarchical Deep Reinforcement Learning framework decomposes this high-dimensional combinatorial problem into tractable sub-problems. An upper-level policy, guided by Monte Carlo Tree Search (MCTS), partitions the global target set to balance fleet workload distribution, whereas a lower-level Transformer-based model constructs near-optimal trajectories for individual agents. A Curriculum-Integrated Alternating Cooperative Training (C-ACT) protocol resolves the convergence difficulties associated with sparse feasible solutions in constrained environments. This protocol incorporates a dynamic constraint annealing strategy and a virtual agent buffer to progressively shape the solution space from relaxed to strictly constrained formulations. Experiments conducted on real-world geographic data demonstrate the proposed approach consistently outperforms all baselines across scales of 80 to 300 targets, improving over the strongest competitor by 0.63–8.51% and over conventional heuristics by up to 53.27% in objective value. Results indicate a task completion rate of 27.5% at the 300-target scale (versus 25.1% for the strongest baseline MCTS + OR) and balanced workload distribution, validating framework adaptability to complex emergency response scenarios.

1. Introduction

The frequency and intensity of natural disasters threaten global public safety. In post-disaster scenarios, timely delivery of emergency supplies to trapped victims dictates casualty rates [1]. UAVs provide an alternative framework for emergency logistics and SAR missions due to their mobility, flexibility, and independence from ground infrastructure [2,3,4]. Collaborative multi-UAV coordination improves operational coverage for large-scale rescue operations [5].

Real-world emergency missions routinely involve heterogeneous UAV fleets featuring varying payload capacities [6], flight ranges [7], and rescue priorities [8], alongside hundreds of distributed targets. This paradigm constitutes the LSH-TOP, an NP-hard problem characterized by an exponentially growing solution space [9]. The coupled complexity of node selection and route construction impedes the generation of feasible plans under strict heterogeneous constraints [10].

Traditional approaches for multi-UAV task allocation rely on heuristic algorithms, including Genetic Algorithm, Particle Swarm Optimization, and market-based Consensus-Based Bundle Algorithm [11,12]. Hybrid strategies integrating heuristics with rolling dispatch [13] or simulated annealing [14] attempt to improve adaptability. While capable of generating feasible solutions, these methods suffer from extended computational times and scalability bottlenecks as the problem scale expands [15]. Furthermore, reliance on manually tuned hyperparameters and iterative search processes prevents real-time decision-making in disaster response scenarios.

Neural combinatorial optimization reframes routing as a Markov Decision Process (MDP), enabling second-level inference speeds. Transformer architectures achieve near-optimal solutions on team orienteering benchmarks when combined with Policy Optimization with Multiple Optima (POMO) [16,17]. These models struggle to satisfy complex constraints for heterogeneous fleets [18,19]. Standard architectures prioritize homogeneous agents and configure hard-masking mechanisms to enforce constraints [20,21].

Rigid masking in heterogeneous scenarios eliminates valid actions, triggering training failures [22]. Static penalties induce local optima by forcing agents to reject tasks for cost reduction. A direct transition from unconstrained to constrained execution impedes model convergence. These limitations collectively require a training paradigm that balances feasibility enforcement with exploration.

This paper proposes a Hierarchical Deep Reinforcement Learning framework decoupling the LSH-TOP for SAR missions [23]. MCTS guides the upper-level task allocation [24], and a Transformer-based model utilizing POMO solves the lower-level routing problems. The framework incorporates a curriculum learning strategy that progressively expands the feasible solution space from relaxed to strictly constrained formulations. The main contributions of this paper are summarized as follows:

• A hierarchical bilevel optimization framework coordinates emergency delivery of heterogeneous UAV fleets. Decomposing LSH-TOP into an allocation layer and a routing layer decouples global reward maximization from local resource constraints. This formulation maximizes total collected rewards within strict flight range and capacity limits. Unlike existing hierarchical methods pairing MCTS with hand-crafted lower-level heuristics, both levels deploy learned neural policies, closing the second structural gap and preserving solution quality as the problem scale grows.
• A Transformer-based routing policy implementing a virtual agent mechanism resolves coupled capacity and range constraints. Unlike hard-masking approaches terminating the decision process prematurely, a virtual agent functions as an absorptive buffer for infeasible tasks. This design prevents early episode termination caused by empty action spaces, securing the completeness of the MDP training process. This mechanism extends neural routing policies beyond homogeneous agents: existing models enforce constraints via hard masking and cannot handle coupled capacity-range limits of heterogeneous fleets without triggering training instability.
• An MCTS-driven allocation strategy complements a curriculum learning-based training scheme for the task assignment policy. Constraints progressively tighten via a dynamic limit factor during training, allowing the optimizer to process relaxed scenarios before executing strict constraints. During inference, MCTS augments search capability, validating task assignments maximizing global rewards in heterogeneous scenarios. The C-ACT protocol closes the first structural gap by combining dynamic cosine annealing with the virtual agent buffer—a design absent from prior curriculum frameworks—enabling convergence where direct training under strict constraints fails.

The three contributions close both structural gaps identified in the literature review: existing frameworks lack a curriculum mechanism for heterogeneous coupled constraints, and asymmetric hierarchical solvers degrade routing quality. The framework improves over the strongest neural baseline by 0.63–8.51% and over conventional heuristics by up to 53.27% across all problem scales, with inference time scaling sub-quadratically to 300-target scenarios. These results validate the neural–neural bilevel structure with C-ACT as a scalable approach for time-critical emergency SAR operations.

2. Literature Review

Heuristic and meta-heuristic algorithms serve as traditional tools for solving UAV task allocation and path planning problems, computing feasible trajectories for combinatorial optimization challenges. Evolutionary strategies, including Genetic Algorithms and Particle Swarm Optimization, optimize routing schedules under multi-objective constraints, coordinating vehicle-mounted swarms and multi-depot logistics systems [25]. Market-based mechanisms, including the Consensus-Based Bundle Algorithm, improve scalability in distributed environments, exhibiting accelerated convergence in heterogeneous networks [26]. Combinatorial auction algorithms facilitate flexible resource combinations for multi-task scheduling. These iterative methods encounter a strict trade-off between solution quality and computational time. As the problem scale expands to hundreds of tasks, the search space grows exponentially, causing extended runtimes and stagnation in local optima. Reliance on hand-crafted rules and static parameter tuning limits framework adaptability in uncertain disaster response environments [27,28].

Neural combinatorial optimization reframes the algorithmic paradigm from iterative search to learned inference, supporting real-time decision-making. The Transformer architecture captures global dependencies via self-attention mechanisms, enabling end-to-end modeling of routing problems [29]. POMO exploits problem symmetries, enhancing training stability and solution generalization to achieve near-optimal performance on team orienteering problem benchmarks. Addressing agent heterogeneity remains an unresolved gap. Existing deep reinforcement learning models process homogeneous fleets and struggle managing distinct physical constraints (e.g., varying payload and range) simultaneously. These approaches employ hard-masking schemes to enforce valid actions [30]. Rigid masking in highly constrained scenarios generates empty action spaces, triggering premature episode termination and training instability.

Curriculum learning strategies introduce progressive difficulty scaling to neural combinatorial optimization, generating feasible policies under strict constraints. Gradually tightening constraint thresholds during training enables agents to learn basic routing patterns before encountering complex restrictions [31,32]. Relaxing penalty coefficients initially prevents agents from selecting local optima characterized by task avoidance [33,34]. Existing curriculum learning frameworks target single-vehicle problems or homogeneous swarms [35,36]. Dynamic constraint annealing handling coupled range and load constraints of heterogeneous fleets remains underexplored [37,38]. Models lacking specialized transition mechanisms fail to converge when transitioning from unconstrained exploration to rigid emergency rescue constraints.

Hierarchical frameworks process large-scale optimization tasks by decomposing them into upper-level allocation and lower-level execution, ensuring feasibility [39,40]. MCTS integration with learning-based methods extends global exploration capability, combining look-ahead search with neural intuition [41]. Current hierarchical methods exhibit structural imbalance. Complex upper-level planning contrasts with lower-level heuristics deployed to save computation. This asymmetry degrades overall solution quality. Integrating MCTS and Transformer-based networks for heterogeneous UAV fleets remains unaddressed in existing frameworks.

The existing literature reveals two structural gaps: First, no curriculum training mechanism addresses heterogeneous agents under coupled capacity-range constraints; existing frameworks target single-vehicle or homogeneous fleets and do not extend to multi-UAV heterogeneous settings. Second, hierarchical methods pair MCTS with hand-crafted lower-level heuristics, introducing asymmetry that degrades routing quality as problem scale grows. The present framework closes both gaps. It replaces lower-level heuristics with a learned Transformer routing policy, forming a neural–neural bilevel structure that preserves solution quality at both levels. The C-ACT protocol jointly manages coupled capacity-range constraints through dynamic annealing and a virtual agent buffer, a combination not present in prior curriculum learning frameworks for combinatorial optimization.

3. Problem Formulation

This section defines the mathematical framework for the multi-UAV task scheduling and route planning problem in disaster rescue scenarios. The formulation models the mission environment using graph theory. Based on this graph representation, a Mixed-Integer Linear Programming model specifies the optimization objectives and operational constraints. A subsequent transformation of this model into a sequential MDP enables the deep reinforcement learning approach as shown in Figure 1.

3.1. Problem Definition

A complete directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ represents the disaster-stricken region. The vertex set $\mathcal{V}=\left\{0\right\}\cup \mathcal{N}$ contains a central depot (node 0) and a target set $\mathcal{N}=\{1,2,\dots ,N\}$. A tuple $(x_i,p_i,q_i)$ defines each target $i\in \mathcal{N}$, specifying the geospatial coordinates, a prize value of rescue priority, and the resource demand. The edge set $\mathcal{E}$ defines spatial connectivity. Each arc $(i,j)\in \mathcal{E}$ carries a Euclidean travel cost $c_{ij}$.

A heterogeneous fleet of K UAVs, denoted by $\mathcal{K}=\{1,2,\dots ,K\}$, departs from the depot. Fleet heterogeneity imposes distinct constraints on each UAV $k\in \mathcal{K}$: a maximum payload capacity $Q_k$ and a maximum flight range $L_k$. Resource limits restrict the fleet from achieving full coverage of the target set $\mathcal{N}$. Therefore, the optimization objective maximizes the collected prize of visited targets through collision-free, depot-returning routes. The mission concurrently necessitates equitable workload distribution across the UAV fleet to maintain operational sustainability.

Table 1. Summary of key notations.

Notation	Description
Sets and Indices
$\mathcal{G}$	The complete directed graph, $\mathcal{G}=(\mathcal{V},\mathcal{E})$
$\mathcal{V}$	Set of all nodes, $\mathcal{V}=\left\{0\right\}\cup \mathcal{N}$
$\mathcal{N}$	Set of potential task/target nodes, $\mathcal{N}=\{1,2,\dots ,N\}$
0	Index of the depot (base station)
$\mathcal{K}$	Set of physical heterogeneous UAVs, $\mathcal{K}=\{1,2,\dots ,K\}$
${\mathcal{K}}^{+}$	Extended UAV set including the virtual agent, ${\mathcal{K}}^{+}=\mathcal{K}\cup \{K+1\}$
$i,j$	Indices for nodes ($i,j\in \mathcal{V}$)
k	Index for UAVs ($k\in {\mathcal{K}}^{+}$)
Parameters
$p_i$	The prize (importance score) associated with task i
$q_i$	The resource demand required by task i
$c_{ij}$	Euclidean distance cost traversing from node i to node j
$Q_k$	Maximum payload capacity of UAV k (heterogeneous)
$L_k$	Maximum flight range of UAV k (heterogeneous)
$\alpha ,\beta$	Weighting coefficients for distance and balance penalties
Decision Variables & Functions
$x_{ijk}$	Binary variable: 1 if UAV k travels from node i to node j, 0 otherwise
$y_{ik}$	Binary variable: 1 if task i is assigned to UAV k, 0 otherwise
$U_k$	Capacity usage rate of UAV k, $U_k=(\sum q_i{y}_{ik})/Q_k$
$\mathsf{\Psi }\left(\mathbf{u}\right)$	Variance of the usage rates across the fleet (balance metric)
J	The global objective function to be maximized

enumerates all notations for the task scheduling problem.

3.2. Task Formulations

The optimization framework applies two sets of binary decision variables to model the mathematical scheduling logic. The assignment variable $y_{ik}\in \{0,1\}$ equals 1 if task i is allocated to UAV k, and 0 otherwise. The routing variable $x_{ijk}\in \{0,1\}$ equals 1 if UAV k traverses the arc $(i,j)$. The global objective function J maximizes the collective mission utility and penalizes excessive travel costs alongside workload imbalance:

$$\mathrm{Maximize}J=\underset{\mathrm{Total}\mathrm{Prize}}{\underbrace{\sum _{k\in \mathcal{K}}\sum _{i\in \mathcal{N}}{p}_i{y}_{ik}}}-\alpha \underset{\mathrm{Distance}\mathrm{Penalty}}{\underbrace{\sum _{k\in \mathcal{K}}\sum _{(i,j)\in \mathcal{E}}{c}_{ij}{x}_{ijk}}}-\underset{\mathrm{Balance}\mathrm{Penalty}}{\underbrace{\beta ·\mathsf{\Psi }\left(\mathbf{u}\right)}}$$

(1)

where $p_i$ indicates the prize of task i, and $c_{ij}$ defines the Euclidean distance cost between node i and node j. The terms $\alpha$ and $\beta$ denote non-negative weighting coefficients. The function $\mathsf{\Psi }\left(\mathbf{u}\right)$ quantifies workload equity. This formulation computes the variance of capacity usage rates across the heterogeneous fleet:

$$U_k={\displaystyle \frac{{\sum }_{i\in \mathcal{N}}{q}_i{y}_{ik}}{Q_k}},\mathsf{\Psi }\left(\mathbf{u}\right)={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{(U_k-\overline{U})}^2$$

(2)

where $q_i$ denotes the demand of task i, $Q_k$ represents the payload capacity of UAV k, and $\overline{U}$ equals the mean usage rate of the fleet.

A set of operational constraints restricts the feasible solution space. Equation (3) enforces assignment completeness. This constraint allocates every target i to exactly one agent within the extended set ${\mathcal{K}}^{+}$. The set ${\mathcal{K}}^{+}$ encompasses physical UAVs for completed tasks and a virtual agent for unassigned tasks:

$$\sum _{k\in {\mathcal{K}}^{+}}{y}_{ik}=1,\forall i\in \mathcal{N}$$

(3)

Equation (4) maintains the structural validity of the flight routes by enforcing topology and flow conservation. This formulation extracts a closed-loop trajectory for each UAV from the depot and preserves path continuity at all intermediate nodes.

$$\sum _{j\in \mathcal{V}}{x}_{0jk}=\sum _{j\in \mathcal{V}}{x}_{j0k}=1,\forall k\in \mathcal{K};\sum _{m\in \mathcal{V}}{x}_{imk}-\sum _{m\in \mathcal{V}}{x}_{mik}=0,\forall i\in \mathcal{N},\forall k\in \mathcal{K}$$

(4)

Equation (5) restricts each target node to at most one physical UAV visit in the routing layer, ensuring that the allocation variable $y_{ik}=1$ translates into exactly one arc traversal:

$$\sum _{k\in \mathcal{K}}\sum _{j\in \mathcal{V}}{x}_{ijk}\le 1,\forall i\in \mathcal{N}$$

(5)

Equations (6) and (7) address hardware heterogeneity. Equation (6) restricts the accumulated demand on UAV k to its payload capacity $Q_k$. Equation (7) bounds the total travel distance within the corresponding endurance range $L_k$:

$$\sum _{i\in \mathcal{N}}{q}_i\left(\sum _{j\in \mathcal{V}}{x}_{ijk}\right)\le Q_k,\forall k\in \mathcal{K}$$

(6)

$$\sum _{i\in \mathcal{V}}\sum _{j\in \mathcal{V}}{c}_{ij}{x}_{ijk}\le L_k,\forall k\in \mathcal{K}$$

(7)

Finally, Equation (8) maintains assignment-routing consistency. A target assignment to a physical UAV requires the corresponding routing variable to indicate a visit. This constraint applies exclusively to physical UAVs ($k\in \mathcal{K}$). The virtual agent ($k=K+1$) executes no physical routing operations but operates solely to absorb infeasible task demands:

$$y_{ik}=\sum _{j\in \mathcal{V}}{x}_{ijk},\forall i\in \mathcal{N},\forall k\in \mathcal{K}$$

(8)

3.3. MDP Formulations

The tuple $\mathcal{P}=(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma )$ specifies this framework. The agent functions as a centralized scheduler. It constructs the solution by appending targets sequentially until it exhausts available resources or violates constraints.

State Space ($\mathcal{S}$): The state $s_t\in \mathcal{S}$ integrates the global decision context at step t. The definition $s_t=(\mathcal{G},{\mathbf{h}}_t,{\mathcal{M}}_t)$ unifies three data modalities. The static graph $\mathcal{G}$ encodes the geospatial coordinates, target prizes, and demand requirements. The dynamic vector ${\mathbf{h}}_t$ monitors the active UAV status. It stores the current spatial coordinates, remaining payload, and residual flight endurance. The binary mask ${\mathcal{M}}_t$ evaluates the task feasibility of unvisited nodes. It clips the action space to enforce valid state transitions.

Action Space ($\mathcal{A}$): The action $a_t\in \mathcal{A}$ selects the subsequent target destination based on the state $s_t$. The dynamic mask ${\mathcal{M}}_t$ constrains the action space to ensure route feasibility:

$$a_t\in \{j\in \mathcal{N}\mid {\mathcal{M}}_t\left[j\right]=1\}\cup \left\{0\right\}$$

(9)

The selection of a node $j\in \mathcal{N}$ allocates the corresponding task j to the active UAV. The specialized zero token (action 0) activates the termination logic. It directs a physical UAV to the depot for tour completion, or it assigns unserviceable targets to the virtual agent.

State Transition ($\mathcal{T}$): The execution of action $a_t$ triggers the system transition to the subsequent state $s_{t+1}$. The selection of a task node j shifts the active UAV coordinates to node j and depletes the associated resources in ${\mathbf{h}}_t$. Specifically, the task demand $q_j$ deducts from the residual payload capacity. The travel cost $c_{current,j}$ reduces the remaining flight range. The transition enforces sequence uniqueness by zeroing the index j in the updated mask ${\mathcal{M}}_{t+1}$. Conversely, the selection of the zero token (action 0) finalizes the current trajectory and initializes the position context for the subsequent agent.

Reward Function ($\mathcal{R}$): A designated reward signal guides the policy optimization. This signal parallels the global objective structured in Equation (1). The environment formulates a sparse reward calculation at the terminal step T of the episode:

$$R\left(s_T\right)=\sum _{k\in \mathcal{K}}\sum _{i\in \mathcal{N}}{p}_i{y}_{ik}-\alpha ·\sum _{k\in \mathcal{K}}\sum _{(i,j)\in \mathcal{E}}{c}_{ij}{x}_{ijk}-\beta ·\mathsf{\Psi }\left(\mathbf{u}\right)$$

(10)

This reward formulation balances the concurrent optimization goals: maximizing total prizes, minimizing trajectory distances, and establishing workload uniformity across the heterogeneous fleet. The episodic decision-making operates over a finite horizon. Thus, the formulation establishes the discount factor $\gamma =1$ to equally evaluate the terminal mission effectiveness.

4. Methodology

The solution of the LSH-TOP requires the decoupling of the combined assignment and routing decision space. A Hierarchical Deep Reinforcement Learning framework decomposes this coupled problem into a bi-level cooperative Markov process. The upper-level allocation network distributes the topological targets across the physical UAVs and the virtual agent. This partition logic systematically manages the heterogeneous payload and endurance constraints. Following the target allocation, a lower-level Transformer policy processes each generated sub-problem. This level executes a POMO routing algorithm to determine specific trajectory sequences. The C-ACT protocol optimizes both parametric policies. This protocol maximizes the episodic mission reward by synchronizing the cooperative output of the full heterogeneous fleet. Figure 2 illustrates the structural hierarchy of the proposed framework.

4.1. Inputs Preprocessing and Feature Embedding

A preprocessing pipeline converts heterogeneous node attributes into uniform embeddings for the encoder. The problem instance defines a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with node geospatial coordinates, priority prizes, and structural demands. A projection layer maps these attributes into a shared latent space for subsequent attention computation.

Feature Normalization and Fusion: The raw dataset records target locations in longitude and latitude. A transverse Mercator projection converts these spherical coordinates into a local two-dimensional Cartesian grid. Min-Max scaling then normalizes the planar values into ${[0,1]}^2$, yielding normalized location vectors ${\mathbf{x}}_i\in {\mathbb{R}}^2$ for all nodes $i\in \mathcal{V}$.

The task characteristics extend the location data. A feature vector ${\mathbf{r}}_i$ for each target node $i\in \mathcal{N}$ concatenates the normalized coordinates ${\mathbf{x}}_i$, the scalar prize $p_i$, and the demand $q_i$:

$${\mathbf{r}}_i=\mathrm{Concat}({\mathbf{x}}_i,p_i,q_i)\in {\mathbb{R}}^4$$

(11)

This formulation encodes the relationship between spatial topology and task priority. Figure 3 diagrams the specific procedural conversion pipeline.

Linear Projection to Latent Space: Coordinates, prizes, and demands occupy independent scales. A linear layer projects the composite vector ${\mathbf{r}}_i$ into a unified latent space. Equation (12) defines the transformation into the node embedding ${\mathbf{h}}_i^{\left(0\right)}\in {\mathbb{R}}^{d_{model}}$:

$${\mathbf{h}}_i^{\left(0\right)}={\mathbf{W}}_n{\mathbf{r}}_i+{\mathbf{b}}_n$$

(12)

where the matrix ${\mathbf{W}}_n\in {\mathbb{R}}^{d_{model}\times 4}$ and the vector ${\mathbf{b}}_n\in {\mathbb{R}}^{d_{model}}$ are trainable parameters.

The depot (node 0) lacks both prize and demand. A separate linear layer computes the depot embedding ${\mathbf{h}}_0^{\left(0\right)}$ from the planar coordinates ${\mathbf{x}}_0$:

$${\mathbf{h}}_0^{\left(0\right)}={\mathbf{W}}_d{\mathbf{x}}_0+{\mathbf{b}}_d$$

(13)

Prepending the depot embedding to the task embeddings yields the input tensor ${\mathbf{H}}^{\left(0\right)}=[{\mathbf{h}}_0^{\left(0\right)},{\mathbf{h}}_1^{\left(0\right)},\dots ,{\mathbf{h}}_N^{\left(0\right)}]$ for the encoder.

4.2. Heterogeneous Attention Policy for Task Allocation

The upper-level allocation network coordinates target service demands against limited hardware capacities. This architecture integrates a graph encoder for feature extraction and a fleet-state-aware decoder for sequential assignment.

4.2.1. Graph Encoding with Heterogeneous Features

Multi-head self-attention propagates information across all node pairs to produce context-aware embeddings. The encoder updates the initialization matrix ${\mathbf{H}}^{\left(0\right)}$ into representations ${\mathbf{H}}^{\left(N_L\right)}$ through $N_L$ sequential attention layers.

Each encoder layer $l\in \{1,\dots ,N_L\}$ projects the intermediate tensor ${\mathbf{H}}^{(l-1)}\in {\mathbb{R}}^{(N+1)\times d_{model}}$ into separate subspaces. The m-th attention head ($m\in \{1,\dots ,M\}$) computes query (${\mathbf{Q}}_m$), key (${\mathbf{K}}_m$), and value (${\mathbf{V}}_m$) matrices:

$${\mathbf{Q}}_m={\mathbf{H}}^{(l-1)}{\mathbf{W}}_m^Q,{\mathbf{K}}_m={\mathbf{H}}^{(l-1)}{\mathbf{W}}_m^K,{\mathbf{V}}_m={\mathbf{H}}^{(l-1)}{\mathbf{W}}_m^V$$

(14)

where the weight matrices ${\mathbf{W}}_m^Q,{\mathbf{W}}_m^K\in {\mathbb{R}}^{d_{model}\times d_{attn}}$ and ${\mathbf{W}}_m^V\in {\mathbb{R}}^{d_{model}\times d_v}$ define per-head subspaces.

A scaled dot-product interaction defines the message passing computation. The matrix multiplication between queries and keys computes pairwise compatibility scores. A division by $\sqrt{d_{attn}}$ prevents softmax saturation when $d_{attn}$ is large, preserving gradient flow through the attention weights. Equation (15) concatenates the activated outputs and passes the combined sequence through the terminal weight matrix ${\mathbf{W}}^O$ to produce the fused representation:

$$\mathrm{MHA}\left({\mathbf{H}}^{(l-1)}\right)=\mathrm{Concat}\left\{{\left(\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{Q}}_m{\mathbf{K}}_m^T}{\sqrt{d_{attn}}}}\right){\mathbf{V}}_m\right)}_{m=1}^M\right\}{\mathbf{W}}^O$$

(15)

where the matrix ${\mathbf{W}}^O\in {\mathbb{R}}^{M{d}_v\times d_{model}}$ merges head outputs into the model dimension. The function $\mathrm{Concat}(·)$ concatenates the independent matrices across the parallel feature dimensions.

A point-wise feed-forward network adds non-linear transformation to the representation pipeline. This sub-layer processes individual node vectors identically. It sequences a dual linear projection separated by a ReLU mapping:

$$\mathrm{FFN}\left(\mathbf{x}\right)=\mathrm{ReLU}(\mathbf{x}{\mathbf{W}}_1+{\mathbf{b}}_1){\mathbf{W}}_2+{\mathbf{b}}_2$$

(16)

where the weight matrices ${\mathbf{W}}_1\in {\mathbb{R}}^{d_{model}\times d_{ff}}$ and ${\mathbf{W}}_2\in {\mathbb{R}}^{d_{ff}\times d_{model}}$ expand features to $d_{ff}$ then project back to $d_{model}$.

Sequential residual additions and layer normalization operations prevent representation collapse within the deep graph architecture. The structured transition cycle inside a unified encoder layer l proceeds as follows:

$${\overline{\mathbf{H}}}^{\left(l\right)}=\mathrm{LN}\left({\mathbf{H}}^{(l-1)}+\mathrm{MHA}\left({\mathbf{H}}^{(l-1)}\right)\right)$$

(17)

$${\mathbf{H}}^{\left(l\right)}=\mathrm{LN}\left({\overline{\mathbf{H}}}^{\left(l\right)}+\mathrm{FFN}\left({\overline{\mathbf{H}}}^{\left(l\right)}\right)\right)$$

(18)

where the iterated execution across $N_L$ identical layers yields the final encoder output ${\mathbf{H}}^{\left(N_L\right)}$. This tensor encodes inter-node dependencies and initializes the subsequent decision generation.

4.2.2. Fleet-State-Aware Decoding Mechanism

Serving as a sequential decision process, the decoding phase iteratively assigns each target node $i\in \mathcal{N}$ to a specific UAV $k\in {\mathcal{K}}^{+}$. Different from standard routing decoders selecting topological nodes, this module operates explicitly in the Task-to-UAV assignment space. At decoding step t, the policy constructs a context embedding centered on target node $i_t$. While the graph encoder captures static topological attributes, sequential assignment continuously depletes vehicle resources. Therefore, effective fleet reasoning necessitates a dynamic state representation ${\mathbf{e}}_k^{\left(t\right)}$ for each physical agent $k\in \mathcal{K}$. This vector fuses the UAV’s current spatial context, inherent hardware capabilities, and real-time resource residuals at step t:

$${\mathbf{e}}_k^{\left(t\right)}={\mathbf{W}}_u·\mathrm{Concat}\left({\mathbf{h}}_{prev,k}^{\left(N_L\right)},Q_k,L_k,\Delta Q_k^{\left(t\right)},\Delta L_k^{\left(t\right)}\right)+{\mathbf{b}}_u$$

(19)

where the term ${\mathbf{h}}_{prev,k}^{\left(N_L\right)}$ records the encoded graph dependency for the instantaneous destination of vehicle k. The parameters $\Delta Q_k^{\left(t\right)}$ and $\Delta L_k^{\left(t\right)}$ quantify the variable payload tolerance and endurance bounds. The matrices ${\mathbf{W}}_u$ and ${\mathbf{b}}_u$ execute the dimension fusion. The virtual entity ($k=K+1$) is exempt from physical routing constraints. It utilizes a learned embedding token ${\mathbf{e}}_{K+1}$ representing task rejection.

The decoder computes assignment probabilities via scaled dot-product attention between the target query and UAV state keys. The target representation ${\mathbf{h}}_{i_t}^{\left(N_L\right)}$ serves as the Query. The dynamic state ensemble ${\mathbf{E}}^{\left(t\right)}={\left\{{\mathbf{e}}_k^{\left(t\right)}\right\}}_{k\in {\mathcal{K}}^{+}}$ provides the Keys. Equation (20) computes the raw attention score $u_{tk}$:

$$u_{tk}=C·tanh\left({\displaystyle \frac{\left({\mathbf{h}}_{i_t}^{\left(N_L\right)}{\mathbf{W}}_Q^{up}\right){\left({\mathbf{e}}_k^{\left(t\right)}{\mathbf{W}}_K^{up}\right)}^T}{\sqrt{d_{attn}}}}\right),\forall k\in {\mathcal{K}}^{+}$$

(20)

where the scalar variable $u_{tk}$ quantifies the compatibility between target $i_t$ and UAV k. The multiplier constant C bounds logit magnitudes to stabilize the softmax output.

4.2.3. Heterogeneous Masking with Virtual Agent

A hard-constraint masking mechanism enforces the validity of generated solutions throughout decoding. This verification module restricts the action space based on fleet resource states. It concurrently redirects infeasible tasks to the virtual agent.

The binary mask $m_{tk}$ indicates the feasibility of assigning target $i_t$ to UAV k. Physical UAVs ($k\in \mathcal{K}$) require strict compliance with cumulative limits. Equation (21) determines the mask value $m_{tk}$:

$$m_{tk}=\mathbb{I}\left(\Delta Q_k^{\left(t\right)}\ge q_{i_t}\right)·\mathbb{I}\left(\Delta L_k^{\left(t\right)}\ge \tilde{c}\left(i_t\right|{\mathcal{P}}_k^{\left(t\right)})\right)$$

(21)

where the indicator $\mathbb{I}(·)$ enforces threshold bounds. The cost estimate $\tilde{c}\left(i_t\right|{\mathcal{P}}_k^{\left(t\right)})=c_{{\mathrm{cur}}_k,i_t}+c_{i_t,0}$ computes the marginal flight cost of appending target $i_t$ to the partial sequence ${\mathcal{P}}_k^{\left(t\right)}$ and returning to the depot.

Flight endurance limits systematically compromise complete topological coverage for physical UAVs. To resolve strict infeasibility, the system sets $m_{t,K+1}=1$ unconditionally:

$$m_{t,K+1}=1$$

(22)

This constant assignment guarantees a non-empty decision space. Integration of the feasibility mask into the raw attention scores reassigns invalid scores to $-\infty$. Equation (23) defines the masked attention score ${\tilde{u}}_{tk}$:

$${\tilde{u}}_{tk}=\left\{\begin{array}{cc}{u}_{tk},\hfill & \mathrm{if}{m}_{tk}=1\hfill \\ -\infty ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(23)

Applying softmax to these masked scores yields the assignment probability $p(y_{i_t}=k|s_t)$:

$$p(y_{i_t}=k|s_t)={\displaystyle \frac{e^{{\tilde{u}}_{tk}}}{{\sum }_{k^{\prime }\in {\mathcal{K}}^{+}}{e}^{{\tilde{u}}_{t{k}^{\prime }}}}}$$

(24)

where the denominator normalizes the distribution over feasible candidates defined by $m_t$.

Sampling from this distribution determines the realized assignment variable $a_t$. This selection sets $y_{i_t,a_t}=1$. Consequently, the framework modifies the state tensors $\Delta Q_{a_t}^{(t+1)}$ and $\Delta L_{a_t}^{(t+1)}$ prior to proceeding to the subsequent decision step.

4.3. Glimpse-Enhanced Route Planning via POMO

The upper hierarchy task allocation reduces the problem into K independent orienteering problems. Each assigned subset ${\mathcal{N}}_k$ is then routed independently by UAV k. The lower-level routing policy ${\pi }_{\varphi }$ constructs a route over ${\mathcal{N}}_k$, tracking visited nodes at each step. At decoding step t, the decoder constructs the context vector ${\mathbf{h}}_c^{\left(t\right)}$. Equation (25) combines the global graph signature ${\mathbf{h}}_{global}$, the embedding of the previously visited node ${\mathbf{h}}_{j_{t-1}}^{\left(N_L\right)}$, and the remaining dynamic tolerances:

$${\mathbf{h}}_c^{\left(t\right)}=[{\mathbf{h}}_{global};{\mathbf{h}}_{j_{t-1}}^{\left(N_L\right)};\Delta Q_k^{\left(t\right)};\Delta L_k^{\left(t\right)}]{\mathbf{W}}_c$$

(25)

where the global variable ${\mathbf{h}}_{global}={\displaystyle \frac{1}{N+1}}{\sum }_{i=0}^N{\mathbf{h}}_i^{\left(N_L\right)}$ averages the independent node mappings into a global graph representation. ${\mathbf{W}}_c$ projects the concatenated features into the model dimension.

An integrated Multi-Head Glimpse architecture refines the context vector ${\mathbf{h}}_c^{\left(t\right)}$ into an actionable query. Unlike standard attention that uses a fixed query, this localized computation attends to the current agent state and candidate node embeddings. The query is ${\mathbf{h}}_c^{\left(t\right)}$, while both keys and values are ${\mathbf{H}}^{\left(N_L\right)}$. Equation (26) computes the glimpse vector ${\mathbf{g}}_t$:

$${\mathbf{g}}_t=\mathrm{MHA}\left({\mathbf{Q}}_g={\mathbf{h}}_c^{\left(t\right)},{\mathbf{K}}_g={\mathbf{H}}^{\left(N_L\right)},{\mathbf{V}}_g={\mathbf{H}}^{\left(N_L\right)}\right)$$

(26)

This operation captures dependencies between the current agent state and unvisited node embeddings. Incorporating resource residuals biases the glimpse toward feasible candidates.

The inference architecture uses ${\mathbf{g}}_t$ as a query to compute a score for each candidate node j via its embedding ${\mathbf{h}}_j^{\left(N_L\right)}$. This assessment extracts the attention score $v_{tj}$ for target j. A local feasibility mask $m_{loc}$ enforces physical feasibility on route construction:

$$v_{tj}=\left\{\begin{array}{cc}C·tanh\left({\displaystyle \frac{\left({\mathbf{g}}_t{\mathbf{W}}_Q^{lo}\right){\left({\mathbf{h}}_j^{\left(N_L\right)}{\mathbf{W}}_K^{lo}\right)}^T}{\sqrt{d_{attn}}}}\right)\hfill & \mathrm{if}{m}_{loc}(j,s_t)=1\hfill \\ -\infty \hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(27)

The Boolean expression $m_{loc}(j,s_t)=1$ holds if and only if both physical constraints are satisfied: the remaining capacity $\Delta Q_k^{\left(t\right)}$ covers the demand $q_j$, and sufficient range $\Delta L_k^{\left(t\right)}$ is maintained for a safe return to the depot. A standardized softmax operation normalizes $v_{tj}$ to parameterize the probabilistic mapping ${\pi }_{\varphi }$.

Sequential generation logic amplifies initialization bias towards local optima. To mitigate sensitivity to initial node selection, the framework applies POMO, exploiting the geometric invariance of Euclidean routing. The transformation operator set $\mathcal{F}=\{{\tau }_1,\dots ,{\tau }_{N_{aug}}\}$ contains distinct planar reflections and rotations. Equation (28) transforms the base uniform subset ${\mathbf{X}}_k$ associated with index ${\mathcal{N}}_k$ into augmented instances:

$${\Sigma }_k={\{{\tau }_m\left({\mathbf{X}}_k\right)\mid {\tau }_m\in \mathcal{F}\}}_{m=1}^{N_{aug}}$$

(28)

Decoding in parallel over these instances yields a candidate set ${\Pi }_k$. Equation (29) filters for the optimal route ${\sigma }_k^{*}$ with the highest total prize:

$${\Pi }_k=\{{\pi }_{\varphi }\left({\mathbf{X}}_k^{\prime }\right)\mid {\mathbf{X}}_k^{\prime }\in {\Sigma }_k\},{\sigma }_k^{*}=\underset{\sigma \in {\Pi }_k}{argmax}\left(\sum _{i\in \sigma }{p}_i\right)$$

(29)

where the resulting reward $R\left({\sigma }_k^{*}\right)$ serves as the reward signal for optimizing the hierarchical policy via gradient updates.

4.4. MCTS-Guided Allocation Refinement

Single-step inference using the policy network often yields suboptimal allocations due to the greedy nature of sequential decoding. Addressing this limitation, the framework integrates MCTS to refine the upper-level task allocation. This look-ahead search mechanism balances exploration of the assignment space with exploitation of the learned policy priors. The search process operates on a decision tree where each node $s_t$ represents a partial allocation configuration, and edges correspond to candidate assignments.

Traversal of the decision tree initiates the search iteration, starting from the root and descending to a leaf node. In this phase, the algorithm selects the optimal child node by maximizing the upper confidence bound. This selection criterion balances the accumulated value estimation and the exploration potential:

$$a_t^{*}=\underset{a\in \mathcal{A}\left(s_t\right)}{argmax}\left(\widehat{Q}(s_t,a)+c_{puct}·P\left(a\right|s_t){\displaystyle \frac{\sqrt{n\left(s_t\right)}}{1+n(s_t,a)}}\right)$$

(30)

where $\widehat{Q}(s_t,a)$ denotes the mean action value, $P\left(a\right|s_t)$ represents the prior probability derived from the upper-level policy ${\pi }_{\theta }$, and $n(s_t,a)$ tracks the action visit count regulating the exploration rate. The total visit count $n\left(s_t\right)={\sum }_{a^{\prime }}n(s_t,a^{\prime })$ aggregates all actions from state $s_t$.

Upon reaching a leaf node $s_L$, the expansion phase activates. The upper-level policy ${\pi }_{\theta }\left(s_L\right)$ evaluates this node, initializing the prior probabilities $P\left(a\right|s_L)$ for all valid assignment actions. The masking logic described in Equation (21) ensures that only feasible branches receive non-zero probabilities, clipping the invalid action space.

Subsequent evaluation of the expanded node involves estimating the potential reward of the resulting allocation through simulation. Instead of random rollouts, greedy execution of the lower-level routing policy ${\pi }_{\varphi }$ on the generated subsets ${\mathcal{N}}_k$ yields a value estimate. Accumulation of the prizes collected by all UAVs constitutes the simulation reward $R\left(s_L\right)$:

$$R\left(s_L\right)={\displaystyle \sum _{k=1}^K}\mathrm{Val}\left(\mathrm{Greedy}({\pi }_{\varphi },{\mathcal{N}}_k|s_L)\right)$$

(31)

where $\mathrm{Val}(·)$ calculates the accumulated prize of the route constructed by the lower-level policy.

Propagation of this simulation result $R\left(s_L\right)$ updates the statistics along the traversed path from $s_L$ back to the root. The update rules for the visit count and action value follow:

$$n(s_t,a)\leftarrow n(s_t,a)+1,\widehat{Q}(s_t,a)\leftarrow \widehat{Q}(s_t,a)+{\displaystyle \frac{R\left(s_L\right)-\widehat{Q}(s_t,a)}{n(s_t,a)}}$$

(32)

where $n(s_t,a)$ on the right-hand side of the second equation reflects the post-increment value, ensuring $\widehat{Q}(s_t,a)$ converges to the expected return of the assignment action.

Completion of the simulation budget triggers the final decision based on the accumulated visit counts. Specifically, the action distribution ${\pi }_{mcts}\left(a\right|s_t)$ is derived proportional to the visit counts:

$${\pi }_{mcts}\left(a\right|s_t)={\displaystyle \frac{n{(s_t,a)}^{1/\tau }}{{\sum }_{a^{\prime }}n{(s_t,a^{\prime })}^{1/\tau }}}$$

(33)

where $\tau$ serves as the temperature parameter controlling the determinism of the final selection.

4.5. Training Strategy

Optimization of the hierarchical policy involves handling the sparsity of feasible solutions within a highly constrained heterogeneous action space. The C-ACT strategy is proposed, incorporating dynamic constraint annealing with a virtual agent mechanism to stabilize the training process.

The training process commences with the lower-level policy ${\pi }_{\varphi }$, which functions as a neural operator evaluating task clusters provided by the upper level. Using the POMO algorithm with instance augmentation, ${\pi }_{\varphi }$ is pre-trained to establish a baseline routing capability. The mean reward of $N_{aug}$ augmented trajectories is used as a baseline to reduce variance. The gradient is calculated as

$${\nabla }_{\varphi }J\left(\varphi \right)\approx {\displaystyle \frac{1}{N_{aug}}}{\displaystyle \sum _{m=1}^{N_{aug}}}{\nabla }_{\varphi }log{\pi }_{\varphi }\left({\sigma }_m\right|{\mathcal{N}}_k)·\left(R\left({\sigma }_m\right)-{\overline{R}}_{aug}\right)$$

(34)

where ${\overline{R}}_{aug}$ denotes the average reward of the augmented batch. This pre-training phase ensures that the lower-level policy provides consistent reward feedback $\mathrm{Val}(·)$ to the upper-level policy.

The C-ACT sequential training structure resolves the reward-coupling problem between the two policy levels. When ${\pi }_{\varphi }$ is undertrained, $\mathrm{Val}(·)$ carries high variance, injecting noise into the gradient estimates of ${\pi }_{\theta }$. Phase I pre-trains ${\pi }_{\varphi }$ before upper-level optimization begins, ensuring that reward feedback reflects task-quality differences by the time ${\pi }_{\theta }$ receives its first gradient updates. In Phase II, the curriculum-aware baseline $b(s_{global},\lambda )$ normalizes rewards relative to ${\lambda }_e$, isolating the policy gradient from reward magnitude shifts caused by constraint relaxation. Phase III alternates gradient updates between ${\pi }_{\theta }$ and ${\pi }_{\varphi }$, serializing the two optimization steps. This structure prevents simultaneous parameter interference, removing the need for an explicit delayed update mechanism.

Optimization of the upper-level policy ${\pi }_{\theta }$ faces challenges due to strict capacity and range constraints, which often cause random allocations to be infeasible. To enable convergence, the training protocol integrates a dynamic constraint annealing strategy. A curriculum factor ${\lambda }_e$ scales the constraint thresholds:

$$\sum _{i\in \mathcal{N}}{q}_i{y}_{ik}\le {\lambda }_e{Q}_k,\sum _{i\in \mathcal{V}}\sum _{j\in \mathcal{V}}{c}_{ij}{x}_{ijk}\le {\lambda }_e{L}_k$$

(35)

This scales both the capacity and range limits, transitioning from a relaxed state (${\lambda }_{start}>1$) to the physical limit (${\lambda }_{end}=1$) according to a cosine schedule. This mechanism gradually contracts the feasible region defined by the mask, guiding the policy from a simplified solution space toward the target constraints. To further guarantee the completeness of the MDP, a virtual agent is incorporated into the action space. Serving as a task rejection mechanism, this agent absorbs tasks that cannot be assigned to real drones, thereby preventing episode termination due to empty action spaces.

The upper-level gradient is derived using REINFORCE with a curriculum-aware baseline:

$${\nabla }_{\theta }J\left(\theta \right)\approx {\displaystyle \frac{1}{B}}{\displaystyle \sum _{i=1}^B}{\nabla }_{\theta }log{\pi }_{\theta }\left({\mathcal{A}}_i\right|s_{global},\lambda )·\left(R_{real}\left({\mathcal{A}}_i\right)-b(s_{global},\lambda )\right)$$

(36)

where ${\mathcal{A}}_i$ represents the task allocation. $R_{real}$ denotes the total reward derived from real drones, and $b(s_{global},\lambda )$ is the baseline reward under the current constraint factor $\lambda$. Algorithm 1 formalizes this complete C-ACT procedure.

Algorithm 1 Alternating cooperative training with curriculum learning

Input:  Training instance set $\mathcal{D}$; batch size B; learning rate $\eta$; augmentation views $N_{aug}$; warm-up epochs $E_w$; alignment epochs $E_a$; joint stabilization epochs $E_j$; curriculum factors ${\lambda }_{start},{\lambda }_{end}$; virtual agent $v_{K+1}$. Output:  Optimized parameters ${\theta }^{*}$ (Upper) and ${\varphi }^{*}$ (Lower).   1: Initialization: Randomly initialize $\theta ,\varphi$; Initialize baseline ${\theta }_{bl}\leftarrow \theta$.   2: Phase I: Lower-Level Warm-up based on POMO   3: for $e=1\to E_w$ do   4:     Sample random batch $\mathcal{B}\sim \mathcal{D}$   5:     Step 1: Instance Augmentation   6:     ${\mathcal{B}}_{aug}\leftarrow \mathrm{Augment}(\mathcal{B},N_{aug})$   7:     Step 2: Reward Evaluation   8:     $R_{aug}\leftarrow \mathrm{Rollout}({\pi }_{\varphi },{\mathcal{B}}_{aug})$   9:     $b_{shared}\leftarrow \mathrm{Mean}\left(R_{aug}\right)$                    ▹ Shared Baseline 10:     Step 3: Optimization 11:     ${\mathcal{L}}_{\varphi }\leftarrow -\mathrm{Mean}((R_{aug}-b_{shared})log{\pi }_{\varphi })$ 12:     $\varphi \leftarrow \mathrm{Adam}(\nabla {\mathcal{L}}_{\varphi },\eta )$ 13: end for 14: Phase II: Upper-Level Alignment with Curriculum Learning 15: for $e=1\to E_a$ do 16:     Curriculum Update: ${\lambda }_e\leftarrow {\lambda }_{end}+{\displaystyle \frac{1}{2}}({\lambda }_{start}-{\lambda }_{end})\left(1+cos\left({\displaystyle \frac{(e-1)\pi }{E_a-1}}\right)\right)$ 17:     Sample global batch $\mathcal{B}\sim \mathcal{D}$ subject to constraint factor ${\lambda }_e$ 18:     Step 1: Exploration & Assessment 19:     ${\mathcal{A}}^{*}\leftarrow \mathrm{Sample}({\pi }_{\theta },\mathcal{B}\cup \left\{v_{K+1}\right\})$ 20:     Extract real assignments $\mathcal{A}\leftarrow {\mathcal{A}}^{*}\setminus \left\{v_{K+1}\right\}$       ▹ Absorb infeasible Tasks 21:     $R\leftarrow \mathrm{Solve}({\pi }_{\varphi },\mathcal{A})$ 22:     Step 2: Baseline Computation 23:     ${\mathcal{A}}_{bl}^{*}\leftarrow \mathrm{Greedy}({\pi }_{{\theta }_{bl}},\mathcal{B}\cup \left\{v_{K+1}\right\})$ 24:     Extract real baseline assignments ${\mathcal{A}}_{bl}\leftarrow {\mathcal{A}}_{bl}^{*}\setminus \left\{v_{K+1}\right\}$ 25:     $b\leftarrow \mathrm{Solve}({\pi }_{\varphi },{\mathcal{A}}_{bl})$ 26:     Step 3: Optimization 27:     ${\mathcal{L}}_{\theta }\leftarrow -\mathrm{Mean}((R-b)log{\pi }_{\theta }\left({\mathcal{A}}^{*}\right|\mathcal{B}))$ 28:     $\theta \leftarrow \mathrm{Adam}(\nabla {\mathcal{L}}_{\theta },\eta )$ 29:     Update ${\theta }_{bl}\leftarrow \theta$ if validation improves 30: end for 31: Phase III: Joint Stabilization 32: for $e=1\to E_j$ do 33:     Execute Phase II step to update ${\pi }_{\theta }$ with fixed ${\lambda }_e=1.0$ 34:     Data Augmentation for Lower Level: 35:     Construct sub-problems ${\mathcal{B}}^{\prime }$ from current allocations $\mathcal{A}$ 36:     Execute Phase I step on ${\mathcal{B}}^{\prime }$ to update ${\pi }_{\varphi }$ 37: end for 38: return ${\theta }^{*}\leftarrow \theta ,{\varphi }^{*}\leftarrow \varphi$

5. Experiments

This section evaluates the Hierarchical Deep Reinforcement Learning framework for large-scale heterogeneous UAV mission planning. The setup details the experimental environment, dataset generation from real-world geographic data, and hyperparameter configurations. Evaluation compares the proposed method against eight baseline algorithms across problem scales ($N=80$ to 300). This comparison assesses solution quality and scalability. The analysis examines small-scale convergence and large-scale generalization. Trajectory visualizations corroborate the numerical findings. Finally, the evaluation quantifies statistical stability and computational efficiency. These metrics verify the framework’s applicability to time-sensitive disaster relief operations.

5.1. Experiment Settings

Experiments test the framework on instances with varying scales. The model sets target counts N from the set $\{80,100,200,300\}$. Real-world geographic data from the Yuelu District of Changsha City, China, supplies the geospatial coordinates for target nodes. Figure 4 visualizes this spatial distribution, with raw location data sourced from the World Geodetic System 1984 reference system. The procedure samples N coordinates from this pool and projects them into a unit square $[0,1]\times [0,1]$.

Uniform distributions $(1.2,2.4)$ and $(1.5,2.5)$ define the maximum payload capacity $Q_k$ and flight range $L_k$ for each UAV $k\in \{1,\dots ,6\}$. These parameters model the fleet heterogeneity. Target demands $q_i$ follow a uniform distribution $U(0.05,0.15)$. Target prizes $p_i$ are sampled from a uniform distribution $U(0.5,1.5)$, reflecting heterogeneous rescue priorities across task nodes. The distance penalty coefficient and workload balance penalty coefficient in Equation (1) are set to $\alpha =0.01$ and $\beta =0.1$, respectively. The reported objective value (Obj.) in all result tables corresponds to the terminal episode reward $R\left(s_T\right)$ as defined in Equation (10). A virtual agent with infinite capacity and range absorbs unassigned tasks. Tasks routed to this agent yield zero reward, implicitly penalizing infeasible assignments.

The hierarchical framework pairs an upper-level allocation policy with a lower-level routing policy, both parameterized by attention-based neural networks. Each encoder incorporates 3 self-attention layers with a unified hidden and embedding dimension of 128, processed through an 8-head multi-head attention mechanism. The upper-level MCTS executes 80 simulations per step with $c_{puct}=1.0$. The lower-level routing model applies the POMO mechanism with a sample size of 8. Validation and testing phases apply greedy decoding for deterministic decision-making.

Both models are trained with the Adam optimizer at an initial learning rate of $1\times {10}^{-4}$. The upper-level model trains for 100 epochs total (50 Phase-II + 50 Phase-III; batch size 1600; epoch size 800,000 instances) with a per-epoch learning rate decay of 0.995; gradient norms are clipped at 3.0. The lower-level model trains for $E_w=100$ warm-up epochs (batch size 400; epoch size 200,000 instances) at a constant learning rate, with gradient norms clipped at 1.0. Phase III joint stabilization runs for $E_j=50$ additional epochs. The curriculum factor ${\lambda }_e$ decays from ${\lambda }_{start}=1.4$ to ${\lambda }_{end}=1.0$ over $E_a=50$ Phase II epochs via a cosine schedule.

Implementation relies on Python 3.12 and the PyTorch 2.8.0 deep learning framework. All computational tasks were performed on a single workstation. The hardware includes an Intel Xeon Platinum 8358P CPU (2.60 GHz), 90 GB of RAM, and a single NVIDIA GeForce RTX 3090 GPU (24 GB). The Ubuntu 22.04 LTS operating system hosts the environment. CUDA 12.8 acceleration processes the tensor computations.

The following protocol governs all evaluations to support reproducibility. Performance metrics are computed over 100 independently sampled test instances per scale. Geographic coordinate sampling and fleet parameter generation use a fixed random seed (seed = 2026); network weights are initialized with seed = 0. Three independent training runs are conducted; the mean performance across the three runs is reported. The coefficient of variation of the reward across the three runs remains below 2% at convergence, confirming training stability. Neural models apply greedy decoding during inference. Stochastic baselines (Adaptive Large Neighborhood Search, ALNS) are executed with five independent runs per instance; the best result per instance is recorded. Error bars in Section 5.6 represent the standard deviation across the 100 test instances. Wilcoxon signed-rank tests in Section 5.6 are computed over the paired per-instance objective values from these same 100 instances.

5.2. Benchmark Evaluation

A cross-scale evaluation protocol tests the generalization capability and scalability of the hierarchical framework. Specifically, models trained on instances with $N=80$ targets execute inference on both $N=80$ and $N=100$ test sets. Models trained on the $N=200$ dataset evaluate scenarios with $N=200$ and $N=300$ targets. This process requires no additional fine-tuning. This experimental design verifies the adaptability of the learned policies to varying problem scales. It demonstrates the model’s ability to capture the underlying problem structure rather than overfit to specific dimensions.

The baseline selection follows a factorial design that isolates the contribution of each hierarchical component independently. Three upper-level allocation strategies (Random, K-Means, and MCTS) cross-combine with three lower-level solvers (ALNS, OR-Tools, and Transformer), yielding nine controlled comparisons. End-to-end learning-based planners for multi-UAV routing are considered as additional baselines. Published models of this type are trained on homogeneous fleets or on problem scales below $N=80$; direct comparison without retraining on the LSH-TOP formulation would conflate architectural differences with distribution mismatch, obscuring interpretation. The MCTS + OR pairing serves as a strong reference for the lower-level routing component by substituting the Transformer with an exact solver under an identical upper-level strategy.

Random assignment serves as a lower-bound reference reflecting unstructured allocation. K-Means clusters tasks by spatial proximity without considering vehicle-specific constraints. Evaluations consider four problem scales with target counts $N\in \{80,100,200,300\}$.

Table 2. Experiment results on 6-UAVs scenario (scales 80 and 100). Obj. denotes the mean terminal episode reward $R\left(s_T\right)$ across 100 test instances. Gap is computed relative to the best objective among all evaluated methods at each scale. Bold indicates the shortest inference time within each allocation strategy group (Time) and the best objective/gap within each group (Obj./Gap).

Method	80			100
	Obj.	Gap (%)	Time (s)	Obj.	Gap (%)	Time (s)
Random + ALNS	24.81	50.80	1.17	28.08	52.65	2.71
Random + OR	30.69	39.14	1.66	35.71	39.78	3.76
Random + Trans	33.32	33.93	1.32	38.95	34.32	3.08
KMeans + ALNS	41.39	17.93	1.54	46.02	22.39	3.28
KMeans + OR	43.40	13.94	1.97	48.99	17.39	4.20
KMeans + Trans	45.80	9.18	1.69	54.79	7.61	3.83
MCTS + ALNS	43.21	14.32	4.05	48.44	18.31	6.22
MCTS + OR	46.14	8.51	4.31	54.69	7.77	6.69
MCTS + Trans	50.43	0.00	4.13	59.30	0.00	6.53

and

Table 3. Experiment results on 6-UAVs scenario (scales 200 and 300). Obj. and Gap are defined as in Table 2. Bold indicates the shortest inference time within each allocation strategy group (Time) and the best objective/gap within each group (Obj./Gap).

Method	200			300
	Obj.	Gap (%)	Time (s)	Obj.	Gap (%)	Time (s)
Random + ALNS	34.77	53.27	4.60	40.92	50.42	8.79
Random + OR	50.76	31.78	6.33	60.33	26.91	12.11
Random + Trans	57.55	22.66	5.11	70.32	14.80	10.52
KMeans + ALNS	61.95	16.75	5.47	65.40	20.77	11.82
KMeans + OR	63.24	15.01	7.26	68.43	17.09	15.55
KMeans + Trans	65.96	11.36	6.09	72.27	12.44	14.19
MCTS + ALNS	65.34	12.19	10.06	71.89	12.90	22.43
MCTS + OR	73.94	0.63	11.24	77.04	6.66	25.61
MCTS + Trans	74.41	0.00	10.77	82.54	0.00	24.30

report the average objective value (Obj.), the optimality gap (Gap), and the inference time.

Experimental results show the MCTS + Trans method achieves the highest objective value across all four scales. The optimality gap is computed relative to the best objective value among all compared methods, not the theoretical optimum; LSH-TOP is NP-hard and exact solutions are intractable at these scales. A 0.00% gap indicates MCTS + Trans achieves the highest objective among all evaluated methods on every test instance, not that it reaches the provably optimal solution. The absolute values and the p-values in Section 5.6 provide the primary basis for performance assessment. Section 5.3 and Section 5.4 provide detailed analyses of these results by scale.

5.3. Comparative Analysis on Small and Medium Scales

This subsection investigates algorithmic performance on small- and medium-scale instances ($N=80$ and $N=100$). The analysis focuses on convergence quality and constraint satisfaction. Table 2 shows the MCTS + Trans framework achieves objective values of 50.43 and 59.30 for $N=80$ and $N=100$, respectively. This establishes a performance benchmark with a 0.00% optimality gap.

Fixing the lower-level solver to the Transformer model isolates the impact of the upper-level allocation strategy. The Random + Trans baseline lacks active assignment logic. It exhibits an optimality gap exceeding 33% across both scales. The KMeans + Trans strategy clusters tasks based on Euclidean proximity. This reduces the gap to 9.18% at $N=80$. Spatial clustering fails to account for heterogeneous payload and flight range parameters. This failure causes load imbalances and restricts the total recoverable value. The MCTS strategy models these constraints during the tree search simulation. MCTS evaluates the potential reward of future states via look-ahead simulation. This approach optimizes task distribution based on the feasibility of the resulting sub-problems. It facilitates the construction of higher-value routes compared to geometric location strategies.

Fixing the upper-level strategy to MCTS isolates the contribution of the lower-level routing solver. The Transformer-based solver performs comparably to the exact OR-Tools solver at these scales. At $N=80$, the MCTS + OR method achieves an objective of 46.14 (8.51% gap). The MCTS + Trans method reaches 50.43 (0.00% gap). OR-Tools converges to optimality given unlimited time. However, a fixed computational budget per sub-problem bounds its performance here. The Transformer model acts as a learned heuristic. It generalizes to the constrained TOP variants and infers solutions rapidly (4.13 s vs. 4.31 s for OR-Tools). This learned policy captures the structural properties of the routing problem efficiently.

Qualitative analysis of the solution topology, as illustrated in Figure 5, corroborates the numerical results. Figure 5a,b depict the planned trajectories for $N=80$ and $N=100$ instances. The generated routes exhibit spatial partitioning with minimal path crossing, indicating that MCTS groups proximal tasks while adhering to the individual flight range limits of each UAV. Furthermore, all UAVs return to the depot after completing their service loops, confirming that the coupled hierarchy satisfies the hard constraints of the mission.

To verify the numerical difference shown in Table 2, Figure 6 presents a side-by-side comparison of solution topologies for an instance with $N=100$. Methods using random allocation (bottom row) result in disordered trajectories with overlapping paths, indicating inefficient task distribution. K-Means (middle row) improves spatial grouping but generates unbalanced routes, as it ignores vehicle-specific payload and range parameters. Route length disparity across UAVs in Figure 6 confirms this load imbalance. The MCTS + Trans method (top-left) produces a structured topology characterized by sub-region partitioning and compact routes. This visualization confirms that the proposed hierarchy optimizes the numerical objective and yields physically rational logistics plans.

5.4. Scalability Verification on Large Scales

5.4.1. Target-Scale Generalization

This section evaluates framework robustness on large-scale instances ($N=200$ and $N=300$). The MCTS + Trans method maintains optimal performance with a 0.00% gap. It achieves objective values of 74.41 and 82.54 for $N=200$ and $N=300$, respectively. These results appear in Table 3.

The limitations of heuristic allocation strategies compound as the problem scale expands. Fixing the lower-level solver to the Transformer model highlights this degradation. The Random + Trans baseline yields an objective value of 70.32 at $N=300$. This value corresponds to a 14.80% optimality gap. The KMeans + Trans strategy records a gap of 12.44% (Obj. 72.27). These gaps widen compared to small-scale benchmarks. Geometric heuristics struggle to balance workloads across heterogeneous UAVs in dense environments. The MCTS strategy mitigates this workload imbalance. Simulating task assignments identifies partitions aligned with specific payload and range constraints. This anticipation prevents premature saturation of vehicle capacities.

The Transformer-based routing solver demonstrates a scalability advantage over conventional iterative baselines. OR-Tools performs competitively on smaller scales. Its performance decays on large-scale sub-problems due to the computational complexity of the orienteering problem. At $N=300$, the MCTS + OR method achieves an objective of 77.04. This trails the MCTS + Trans method (82.54) by a 6.66% gap. The Transformer model processes increased node density without the corresponding performance decay. The learned policy generalizes to larger instances. It captures routing patterns inaccessible to constructive heuristics.

Analysis of Figure 7 and Figure 8 corroborates these numerical findings. Figure 7 illustrates the proposed framework generating compact, non-overlapping trajectories at high target densities. These routes maximize the coverage of distributed high-value targets. Figure 8 contrasts this with baseline methods. Random allocation methods (bottom row) produce entangled routes. This structural chaos reflects a failure to spatially decompose the massive task set. K-Means-based approaches (middle row) generate load imbalances. These imbalances stem from fleet heterogeneity. The proposed method (top-left) establishes a streamlined topology. This visual evidence verifies its capability to resolve large-scale collaborative mission planning problems.

5.4.2. Fleet-Size Sensitivity

The preceding experiments fix the fleet size at $K=6$. To evaluate sensitivity to fleet composition, all methods are tested under $K=9$ UAVs using zero-shot inference without retraining.

Table 4. Zero-shot fleet-size transfer results ($K=9$, models trained on $K=6$). Obj. denotes the mean total collected prize across 100 test instances. Visited/Total indicates the average number of targets served. Bold indicates the best result at each problem scale.

Method	$\mathit{N}=80$		$\mathit{N}=300$
	Obj.	Visited	Obj.	Visited
KMeans + Trans	52.53 ± 5.77	69.9/80	112.85 ± 2.75	144.7/300
KMeans + OR	51.73 ± 5.82	69.2/80	106.09 ± 3.77	140.2/300
KMeans + ALNS	50.39 ± 5.67	66.7/80	101.27 ± 4.02	135.5/300
MCTS + Trans	42.23 ± 4.07	55.3/80	83.40 ± 2.96	108.4/300
MCTS + OR	41.03 ± 3.56	53.6/80	81.85 ± 3.21	103.5/300
MCTS + ALNS	40.87 ± 3.75	53.0/80	77.52 ± 4.34	99.8/300
Random + Trans	35.69 ± 4.36	46.7/80	73.92 ± 2.87	91.3/300
Random + OR	35.56 ± 3.75	46.2/80	71.47 ± 4.10	91.5/300
Random + ALNS	31.93 ± 5.47	42.6/80	66.29 ± 5.70	89.8/300

reports the results.

Under $K=9$, KMeans + Trans achieves the highest objective at both scales (52.53 at $N=80$; 112.85 at $N=300$), surpassing MCTS + Trans by 24.4% and 35.3%, respectively. This reversal relative to the $K=6$ results reveals a boundary condition of the learned allocation policy. The MCTS search tree is trained on a 6-agent decision space; when transferred to 9 agents without retraining, the tree structure cannot represent the expanded combinatorial branching, limiting its ability to exploit additional fleet capacity. K-Means, as a geometry-based method independent of training configuration, distributes tasks across all available UAVs by spatial proximity, naturally scaling with fleet size.

This result identifies fleet-size generalization as a limitation of the current framework. Retraining or fine-tuning the upper-level policy on the target fleet configuration is expected to restore MCTS dominance, as the search mechanism itself is not fleet-size-dependent. Developing fleet-size-agnostic allocation policies constitutes a direction for future work.

5.5. Ablation Study

To validate the structural choices and verify the isolated contribution of each mechanism within the C-ACT protocol, a comprehensive ablation study was conducted at the $N=300$ scale. This scenario, characterized by the highest combinatorial complexity and stringent resource limits, serves as a stress test for the proposed framework. The complete MCTS + Trans framework is compared against four degraded variants:

• w/o POMO: Replaces the POMO mechanism with single-trajectory greedy decoding. This isolates the baseline capability of the Transformer routing heuristic.
• w/o MCTS (Neural Greedy): Removes the look-ahead search from the upper-level allocator. Task assignment relies solely on the neural policy’s direct greedy output.
• w/o Curriculum ($\lambda =1$): Disables dynamic constraint annealing. The network trains under strict payload and range limits from the initial epoch.
• w/o Virtual Agent: Removes the virtual agent from the assignment space. Infeasible tasks revert to rigid hard-masking, risking action space collapse.

The performance metrics for these variants are summarized in

Table 5. Ablation study at $N=300$. Gap is relative to the full model (82.54). TCR: task completion rate. MCTS + OR is an external reference, not an ablation variant. Bold indicates the full model (best performance). Italic indicates an external reference.

Model Variant	Obj.	Gap (%)	TCR (%)
MCTS + Trans (Full Model)	82.54	0.00	27.5
w/o POMO	79.80	3.32	26.2
w/o Curriculum ($\lambda =1$)	75.50	8.53	23.8
w/o MCTS (Neural Greedy)	74.50	9.74	23.1
w/o Virtual Agent	58.20	29.49	18.5
MCTS + OR (external ref.)	77.04	6.66	25.1

.

Table 5 justifies each methodological choice through performance degradation. Most notably, the w/o MCTS variant achieves an objective of 74.50, yielding a gap of 9.74% relative to the full model. This result falls 3.08 percentage points below the MCTS + OR reference (6.66%), confirming that while the neural allocator learns meaningful representations, MCTS look-ahead simulations are essential to escape myopic assignments at large scales.

The most severe performance collapse stems from removing the Virtual Agent (29.49% gap). Without an absorptive buffer for infeasible targets, the action space frequently collapses during early training, causing MDP termination. To quantify its necessity,

Table 6. Virtual agent utilization rate (%) across scales and allocation methods. Bold indicates the proposed method.

Allocation Method	$\mathit{N}=80$	$\mathit{N}=100$	$\mathit{N}=200$	$\mathit{N}=300$
MCTS (Proposed)	2.1%	4.5%	9.8%	14.3%
K-Means Heuristic	5.4%	8.7%	16.5%	22.1%

tracks the virtual agent’s utilization frequency across scales.

The proportion of instances requiring virtual agent intervention rises from 2.1% at $N=80$ to 14.3% at $N=300$, confirming the virtual agent functions as an essential buffer in dense environments. Furthermore, MCTS allocation consistently triggers the virtual agent less frequently than the K-Means heuristic, indicating it generates more physically feasible sub-problems.

Finally, omitting dynamic constraint annealing (w/o Curriculum) reduces the objective to 75.50. Experimental logs confirm that without C-ACT, training exhibited severe instability: reward variance during early epochs was substantially higher, and independent runs frequently settled into penalty-avoidance behaviors. The curriculum schedule resolves this by expanding the feasible region early in training, then gradually contracting it toward physical limits. The w/o POMO variant confirms that the base MCTS + Trans architecture outperforms OR-based routing under single-trajectory decoding. POMO augmentation reduces the gap by a further 3.32 percentage points through multi-start geometric diversification.

Figure 9 presents the training reward curves for both the upper-level allocation model and the lower-level routing model at two problem scales ($N=80$ and $N=200$), comparing the full C-ACT protocol (solid) against the w/o Curriculum ablation with fixed $\lambda =1$ (dashed gray).

The convergence behavior of C-ACT is further evidenced by training dynamics (Figure 9). At the upper level, C-ACT reduces reward variance by an order of magnitude relative to the fixed-constraint baseline: the variance ratio (C-ACT/w/o Curriculum) is 0.014 at $N=80$ and 0.152 at $N=200$, indicating that curriculum relaxation is essential for stable upper-level optimization across problem scales. At the lower level, the variance reduction is moderate ($\mathrm{ratio}=0.719$ at $N=200$) since the routing policy operates on pre-allocated subproblems with smaller action spaces. These dynamics are consistent with the 8.53% performance gap at convergence (Table 5) and confirm that dynamic constraint annealing is necessary for convergence in this constrained heterogeneous setting.

An end-to-end baseline generating joint allocation-routing sequences without bilevel decomposition would provide the most direct validation of the hierarchical design. However, no published end-to-end model supports the LSH-TOP constraints (heterogeneous capacities, heterogeneous ranges, and prize collection) at scales beyond $N=80$. Training such a model from scratch requires $O\left(N^K\right)$ action-space exploration per step ($N=300$, $K=6$ yields $\approx 7.3\times {10}^{14}$ candidate sequences), rendering convergence infeasible within practical GPU budgets. As a proxy, the w/o MCTS (Neural Greedy) variant approximates a flat learned policy: it uses the same trained Transformer allocator but removes tree-search look-ahead, collapsing the hierarchical structure to a single forward pass per assignment. Its 9.74% gap relative to the full model quantifies the minimum cost of abandoning structured search. The MCTS + OR reference further isolates the decomposition benefit: replacing the learned router with OR-Tools still achieves a 6.66% gap, confirming that the hierarchical structure itself—independent of the lower-level solver—provides the dominant performance contribution.

5.6. Statistical Stability and Computational Efficiency

Evaluating stochastic stability and resource utilization efficiency is essential for operational deployment. This subsection assesses statistical variance, task completion rates, and computational latency across scales.

Figure 10 visualizes reward distribution and task completion rates for $N\in \{80,100,200,300\}$. Error bars quantify standard deviations across test instances. Vertical progression from Figure 10a–d reveal the MCTS + Trans method securing tighter error bounds. This contrasts sharply with Random and K-Means baselines. At $N=300$ (Figure 10d), the standard deviation for MCTS + Trans remains smaller than Random + Trans. The hierarchical framework mitigates the randomness inherent in combinatorial optimization.

The TCR quantifies the proportion of targets successfully served by the fleet. Figure 10 reports TCR distributions across all scales and methods. At $N=80$, MCTS + Trans achieves a 63.0% completion rate, exceeding the 54.0% recorded by MCTS + ALNS (read from Figure 10a). At $N=300$, the task-to-resource ratio peaks. MCTS + Trans maintains a leading completion rate of 27.5% (Table 5). The MCTS strategy optimizes route compactness alongside high-value targeting, serving more nodes within strict flight range constraints.

The Wilcoxon signed-rank test was applied at $N=300$ to verify that the performance gains of MCTS + Trans over the two strongest baselines are statistically significant. The test targets two controlled comparisons: MCTS + Trans versus MCTS + OR isolates the contribution of the lower-level routing solver, and MCTS + Trans versus KMeans + Trans isolates the contribution of the upper-level allocation strategy.

Table 7. Wilcoxon signed-rank test results at $N=300$. $p<0.05$ rejects the null hypothesis of equal performance.

Comparison	p-Value
MCTS + Trans vs. MCTS + OR	$3.42\times {10}^{-5}$
MCTS + Trans vs. KMeans + Trans	$1.28\times {10}^{-8}$

reports both p-values. Both fall below the 0.05 threshold, rejecting the null hypotheses for both comparisons.

Table 8. Mean objective values and 95% confidence intervals at $N=300$.

Method	Mean Obj.	95% CI
MCTS + Trans	82.54	[81.65, 83.41]
MCTS + OR	77.04	[76.06, 78.02]
KMeans + Trans	72.27	[71.09, 73.45]

reports 95% confidence intervals for the proposed method and its two strongest baselines at $N=300$, computed across the 100 test instances via the normal approximation. Non-overlapping intervals confirm that performance differences are not attributable to sampling variance.

Computational efficiency dictates real-time applicability. Figure 11 plots framework inference times against baselines. The MCTS + Trans computational cost exhibits sub-quadratic growth. It scales from 4.13 s at $N=80$ to 24.30 s at $N=300$. The MCTS + OR method incurs accelerating overhead as the sub-problem complexity grows. It demands 25.61 s at $N=300$. Heuristic baselines like Random + ALNS operate with lower latency (8.79 s at $N=300$). They suffer from optimality gaps exceeding 50%. The proposed framework controls these trade-offs by investing computation into upper-level tree search. This isolates high-quality decompositions. Parallelized neural network inference recovers speed in the lower level. The MCTS + Trans configuration produces solutions within 24.30 s at $N=300$, satisfying the temporal constraints of operational decision support.

6. Discussion

Experimental outcomes demonstrate that the hierarchical decomposition maintains solution quality and controls computational complexity across all tested scales. As the problem scale grows, the performance gap between MCTS-guided and geometry-based allocation evolves in distinct ways across baselines. At $N=300$, KMeans + Trans exhibits a gap of 12.44%, widening from 9.18% at $N=80$; the absolute objective difference between MCTS + Trans and KMeans + Trans grows from 4.63 to 10.27. Random + Trans, by contrast, narrows its percentage gap from 33.93% to 14.80% as scale increases. Its absolute objective deficit relative to MCTS + Trans also contracts (17.11 to 12.22). In larger dense environments, any assignment captures some reward, reducing the relative penalty of unstructured allocation. In both cases, the MCTS strategy maintains its advantage by evaluating downstream feasibility before committing to each assignment—a capability that pure geometric methods cannot replicate. Heuristic operators fail to coordinate global workload distribution and local trajectory compactness simultaneously. The upper-level agent treats the lower-level routing policy as a neural heuristic that provides value estimates for candidate partitions, enabling globally coherent allocations that geometry-based methods cannot achieve.

The convergence behavior of C-ACT admits a theoretical interpretation. The REINFORCE gradient estimator converges to a locally optimal policy as long as the gradient variance remains bounded. The cosine annealing schedule keeps the feasible region $\mathcal{F}\left({\lambda }_e\right)$ non-empty throughout Phase II. Because ${\lambda }_e\ge 1$ relaxes both capacity and range limits, any allocation feasible under ${\lambda }_e=1$ remains feasible under ${\lambda }_e>1$, maintaining a non-zero episode completion probability at every epoch. The virtual agent reinforces this property by maintaining a non-empty action space at every decision step (Equation (22)), satisfying the MDP completeness requirement for policy gradient convergence. Training stability is further supported by gradient clipping at norms 3.0 and 1.0 for the upper and lower levels, respectively, and by the alternating update structure of Phase III, serializing ${\pi }_{\theta }$ and ${\pi }_{\varphi }$ updates and preventing simultaneous parameter interference.

This structural efficiency is further quantified from a computational complexity standpoint. The hierarchical decomposition reduces inference complexity relative to joint allocation-routing. A brute-force policy over joint sequences requires $O\left(N^K\right)$ evaluations; the proposed structure reduces this to an MCTS search of depth N with branching factor K, where each simulation invokes the lower-level policy on sub-problems of average size $N/K$, plus K independent final routing evaluations:

$$T_{hier}=O\left(N·K·S_{mcts}·{\displaystyle \frac{N}{K}}\right)+O\left(K·{\left({\displaystyle \frac{N}{K}}\right)}^2\right)=O\left(N^2·S_{mcts}+{\displaystyle \frac{N^2}{K}}\right)$$

(37)

where $S_{mcts}$ denotes the MCTS simulation budget per step. For fixed K and $S_{mcts}$, both terms scale as $O\left(N^2\right)$, yielding a theoretical upper bound of $O\left(N^2\right)$—a substantial reduction from the $O\left(N^K\right)$ complexity of joint end-to-end approaches. Empirically, the inference time increases from 4.13 s to 24.30 s across $N\in \{80,100,200,300\}$ (a 5.88× increase for a 3.75× increase in N). A log–log regression on these four data points yields an exponent of approximately 1.2, indicating sub-quadratic practical scaling tighter than the $O\left(N^2\right)$ theoretical bound due to GPU parallelism in the lower-level routing step and early termination in MCTS simulations.

The learned policies generalize to moderately larger problem scales without retraining, preserving solution feasibility under varying constraint tightness. This cross-scale transferability—enabled by the decomposition structure that reduces a 300-target instance to six independent sub-problems of manageable size—directly supports time-critical disaster response operations where retraining between deployments is impractical. However, fleet-size transfer (Section 5.4.2) reveals a boundary condition: the MCTS search tree, optimized for a fixed agent count, does not generalize to unseen fleet configurations under zero-shot inference. This limitation is structural rather than algorithmic—the search mechanism itself is fleet-size-agnostic, but the learned prior $P\left(a\right|s_t)$ encodes a fixed branching factor.

7. Conclusions and Future Work

A hierarchical reinforcement learning framework solves the LSH-TOP for emergency UAV operations. Decoupling the architecture into MCTS-guided allocation and Transformer-based routing balances global resources and local trajectories. A curriculum-integrated training protocol injects dynamic constraint annealing and a virtual agent mechanism. This protocol enables convergence in sparse-reward environments bounded by physical vehicle limits. Evaluations document consistent objective improvements over all baselines—0.63–8.51% over the strongest competitor and up to 53.27% over conventional heuristics—alongside leading task completion rates. The framework maintains these gains at scales up to 300 targets with sub-quadratic inference time growth. This scalability supports immediate deployment in SAR logistics.

The evaluation is confined to simulated environments; no hardware-in-the-loop testing or physical UAV deployment is performed. The geographic data from Yuelu District (Figure 4) provides realistic spatial distribution, but the simulation excludes wind disturbance, GPS drift, inter-UAV communication latency, and onboard sensor noise. The formulation assumes static target sets with deterministic travel costs, excluding the dynamic and uncertain conditions present in operational disaster response. Physical testbed validation and hardware-in-the-loop testing are deferred to future work.

Future development will target dynamic uncertainties in operational domains. Moving targets and stochastic communication gaps will require online adaptation mechanisms. Expanded state representations encapsulating temporal data will support real-time replanning. Multi-objective models combining energy consumption tracking with timeliness maximization will refine the system’s operational viability. Fleet-size-agnostic allocation policies will eliminate the retraining requirement when fleet composition changes between missions.