An Adaptive Scheduling Algorithm Integrating Hierarchical Reinforcement Learning and Semi-Markov Decision Processes

Abstract

Coordinating multiple unmanned aerial vehicle (UAV) systems under strict energy and temporal constraints remains a complex scheduling problem. Existing reinforcement learning methods typically rely on fixed-time-step modeling, which struggles to accommodate flight actions of varying durations and often leads to temporal mismatches between task planning and physical execution. To address this limitation, we propose an Adaptive Hierarchical Semi-Markov Decision Process (AH-SMDP) framework. This architecture decouples task allocation from execution by modeling variable-length actions via an SMDP. An event-driven synchronization mechanism is introduced to align the swarm’s decision-making rhythm with actual task completion times. Additionally, a state-aware reward formulation and a dynamic action space pruning strategy are designed to help UAVs balance energy efficiency with deadline compliance. Simulation results in multi-constraint environments demonstrate that the AH-SMDP framework effectively improves scheduling performance compared to standard MAPPO and PPO algorithms. Under the evaluated experimental settings, the proposed method yields improvements of approximately 30% in average task completion rate, 40% in energy reduction, and 60% in convergence stability. Ablation studies further suggest that this integrated framework offers a viable and effective approach for multi-UAV scheduling.

1. Introduction

Systems comprising multiple unmanned aerial vehicles (UAVs) are increasingly utilized in applications such as smart logistics, emergency response, and environmental monitoring. While single UAVs are limited in operational scope, cooperative multi-UAV swarms can expand coverage and enhance execution efficiency [1,2]. However, in practical deployments, these systems must often manage complex task sequences under limited battery capacities and strict temporal constraints. This operational reality necessitates a trade-off between the collective performance of the swarm and the energy endurance of individual drones. Consequently, developing effective cooperative decision-making strategies under such multiple constraints remains a significant challenge in multi-agent system research [3].

In this study, the “scheduling” problem is defined as the dynamic allocation of spatially distributed macro-tasks to individual UAVs, coupled with the execution of micro-actions (e.g., variable-duration flight maneuvers and charging behaviors). The primary purpose of this scheduling is to maximize the swarm’s global mission completion rate within specified temporal constraints, while autonomously minimizing total energy consumption and avoiding individual battery depletion.

Despite these defined objectives, achieving effective scheduling in dynamic environments highlights several limitations in existing methodologies. To address these challenges, this study focuses on the following three specific research problems (RPs):

1.

RP1 (Temporal Modeling Mismatch): How can a modeling framework be established to integrate high-level discrete task allocation with low-level, variable-duration physical flight execution to mitigate temporal misalignment?

2.

RP2 (Asynchronous Swarm Execution): How can the decision-making process be synchronized across the swarm to maintain cooperation when individual UAVs complete their assigned actions at varying times?

3.

RP3 (Dynamic Constraint Adaptation): How can the scheduling policy balance competing objectives (e.g., task timeliness versus energy conservation) given dynamic drone states and operational boundaries?

From a modeling perspective, tasks and flight actions in multi-UAV scheduling inherently require variable execution times. As a result, traditional fixed-time-step models often struggle to accurately represent the physical execution process. Relying on a single timescale can lead to a temporal mismatch between high-level planning and low-level control. A viable approach to address this limitation is to decouple the decision-making process into a hierarchical structure: an upper layer for macro-task allocation and a lower layer for micro-action execution, thereby establishing a dual-timescale framework. During mission execution, individual UAV states—such as remaining battery capacity, task progress, and potential delays—evolve dynamically and affect subsequent decisions. The interactions among these variable-length actions further increase environmental non-stationarity. Additionally, competing objectives, including task completion, energy conservation, and temporal constraints, complicate the application of standard unified optimization methods in these scenarios [4,5].

Current research addresses these challenges primarily through heuristic optimization and multi-agent reinforcement learning (MARL). While heuristic methods can yield near-optimal solutions in static or relatively stable environments, their computational overhead often limits their applicability in real-time operations. MARL approaches, conversely, offer enhanced adaptability through continuous learning. However, standard MARL frameworks generally rely on Markov Decision Processes (MDPs) with fixed time-steps. This formulation can struggle to accurately represent actions of varying durations. Consequently, this approach may lead to a temporal gap between high-level planning and physical execution, potentially introducing instability during multi-agent cooperation. Furthermore, many existing methods employ static reward functions. These fixed mechanisms typically lack the flexibility to accommodate dynamic state changes, which can restrict performance in tightly constrained environments.

To address these limitations, this paper investigates the multi-UAV scheduling problem using temporal modeling and adaptive decision-making. By incorporating a Semi-Markov Decision Process (SMDP), variable-duration actions are modeled and integrated into a hierarchical architecture. This structure decouples high-level task allocation from low-level flight execution. Instead of traditional fixed-interval updates, the framework utilizes an event-driven synchronization mechanism. A new global decision step is triggered when UAVs complete their assigned operations, thereby alleviating the temporal mismatch caused by asynchronous execution. Furthermore, a state-aware regulation mechanism is designed that combines dynamic reward weighting with action-space pruning. This approach enables the system to dynamically balance energy consumption and temporal constraints. Together, these components constitute the Adaptive Hierarchical Semi-Markov Decision Process (AH-SMDP) framework for swarm scheduling in tightly constrained environments.

The primary contributions of this paper are summarized as follows:

1.

Dual-Timescale Modeling for Variable-Duration Actions: A hierarchical SMDP framework is developed to mitigate the limitations of fixed time-step models. By decoupling task allocation from physical execution, this approach facilitates cooperative optimization across two distinct timescales, aligning more closely with actual flight dynamics.

2.

Event-Driven Temporal Alignment: An event-triggered synchronization mechanism is designed to accommodate the asynchronous execution of multiple UAVs. By utilizing task completion as the trigger for global updates, this mechanism coordinates the decision-making steps of the swarm. This approach helps mitigate the environmental non-stationarity typically introduced by uncoordinated actions, thereby promoting stable cooperation.

3.

State-Aware Adaptive Regulation: To manage the trade-off between temporal constraints and energy consumption, an adaptive regulation mechanism is introduced. By combining dynamic reward weighting with an action space pruning strategy, the policy adjusts its optimization focus across different mission phases. This mechanism improves overall scheduling efficiency while operating within defined safety boundaries.

2. Related Work

Existing research on multi-UAV cooperative scheduling can be broadly categorized into heuristic optimization, multi-agent reinforcement learning (MARL), and hierarchical decision-making with temporal modeling. This section reviews these methodologies, with a particular focus on their applicability and limitations in dynamic, multi-constraint environments.

2.1. Traditional Resource Scheduling and Heuristic Optimization

Early efforts in multi-UAV task allocation frequently relied on swarm intelligence and heuristic search techniques. To address temporal constraints, Yan et al. [6] introduced an enhanced particle swarm optimization (PSO) algorithm that incorporates a golden-sine strategy to balance exploration and exploitation. While this approach can be effective for smaller configurations, the computational overhead of such heuristic methods often scales poorly with the number of agents, limiting their applicability for real-time scheduling in large-scale deployments.

Distributed auction mechanisms have been proposed to enhance system scalability. For instance, Gao et al. [7] utilized a conditional probability-based method to resolve assignment conflicts via a distributed consensus protocol. Building on auction frameworks, Ye et al. [8] adapted the consensus-based bundle auction (CBBA-TCC) to manage temporal coupling constraints and mitigate deadlocks in heterogeneous clusters. Despite these structural improvements, auction-based and heuristic planners typically assume static or quasi-static conditions. In highly uncertain environments where execution times fluctuate or disturbances occur, the requirement for continuous re-planning can introduce substantial communication and computational overhead. Consequently, reliance on static assumptions or predefined planning mechanisms can restrict the adaptability of these methods in highly dynamic, tightly constrained scenarios.

Beyond heuristic search and distributed auctions, resource scheduling has been extensively studied through analytical allocation schemes. To optimize resource utilization in communication networks, Tu [9] proposed a traditional non-learning-based scheme for efficient resource utilization in multi-flow wireless multicasting transmissions. By theoretically analyzing flow scheduling and channel aggregation policies, this method aims to maximize concurrent transmissions based on dynamic channel states. Recent studies have advanced UAV resource management from multiple perspectives. For instance, Zeng and Zhang [10] employed continuous trajectory optimization to reduce energy consumption during UAV communications. Additionally, Tu [11] proposed resource-efficient seamless transitions to sustain performance in dynamic multi-hop UAV multicasting. While these approaches primarily address communication routing and continuous kinematic planning, the proposed AH-SMDP framework extends optimization to the discrete-continuous hybrid scheduling of multi-task swarm operations. Generally, these classical analytical approaches rely on rigorous mathematical programming—such as convex optimization, integer linear programming, or Lyapunov optimization—to derive optimal schedules under the assumption of known or quasi-static environmental parameters.

However, applying these conventional analytical paradigms directly to multi-UAV cooperative operations can introduce computational bottlenecks, thereby motivating the adoption of reinforcement learning (RL). Traditional mathematical models typically rely on explicit state-transition models and may struggle to accommodate highly dynamic, non-stationary environmental feedback in real time. Specifically, in scenarios involving variable-duration flight maneuvers and unpredictable energy degradation, recalculating exact analytical solutions at every time step can impose a prohibitive computational burden. RL offers a viable paradigm to address this intractability. By shifting the computational overhead of solving complex constraints to an offline training phase, RL enables UAVs to execute rapid, adaptive scheduling decisions online, significantly reducing the reliance on static environmental assumptions and predefined mathematical bounds.

2.2. Multi-Agent Reinforcement Learning Cooperative Methods

In recent years, multi-agent reinforcement learning (MARL) utilizing the centralized training with decentralized execution (CTDE) paradigm has emerged as a widely adopted approach for dynamic scheduling. To address the credit assignment challenge in cooperative settings, Rashid et al. [12] developed the QMIX framework, which enforces global monotonicity via non-linear value decomposition. Extending these concepts to UAV operations, Yang et al. [13] formulated the cooperative problem as a Dec-POMDP and applied an adapted MAPPO algorithm to navigate partially observable edge-computing environments.

While MARL offers considerable adaptability, standard formulations typically rely on Markov Decision Processes (MDPs) structured around fixed time-steps. This conventional modeling assumes uniform action durations, which can struggle to accommodate the varying temporal requirements of actual flight maneuvers—ranging from rapid obstacle avoidance to extended cruising. When applied to complex task sequences, such temporal simplifications can lead to a misalignment between high-level planning and physical execution. This discrepancy can exacerbate environmental non-stationarity during asynchronous multi-agent interactions, potentially hindering policy convergence and overall cooperative stability.

2.3. Hierarchical Decision-Making and Semi-Markov Temporal Modeling

Temporal abstraction offers a pathway to address the limitations of fixed-step modeling. Sutton et al. [14] established the foundational options framework to formally represent variable-duration actions. Extending this concept to UAV navigation, Hurst et al. [15] utilized the Semi-Markov Decision Process (SMDP) to link state transitions directly to physical time costs, effectively mitigating timescale distortion. Adopting a structural approach, Liu et al. [16] developed an option-based hierarchical deep reinforcement learning (OHDRL) strategy that decouples micro-level flight control from high-level scheduling, thereby improving system energy management. Despite these modeling advancements, coordinating asynchronous multi-agent execution remains challenging. Existing hierarchical frameworks often lack an event-triggered alignment mechanism, complicating state synchronization across a UAV swarm. Furthermore, these methods typically rely on static reward formulations. This rigid design can limit the system’s ability to dynamically adapt when agents encounter critical operational boundaries, such as depleted battery capacity or strict temporal constraints.

2.4. Gap Analysis

In summary, existing studies on multi-UAV cooperative scheduling generally exhibit the following limitations:

1.

Timescale Integration: Many current models lack a unified framework to concurrently process macro-level task assignment and micro-level flight execution across different timescales.

2.

Execution Misalignment: The absence of synchronization mechanisms for variable-duration actions can lead to temporal mismatches between high-level planning and physical execution.

3.

Rigid Reward Structures: Reliance on static reward formulations restricts the system’s ability to dynamically balance temporal constraints and energy consumption in fluctuating environments.

To address these limitations, this study proposes the Adaptive Hierarchical Semi-Markov Decision Process (AH-SMDP). By integrating an event-triggered synchronization mechanism and a state-aware reward strategy, this framework is designed to improve multi-UAV scheduling performance under defined energy and temporal constraints.

3. Proposed Method

3.1. Methodology Overview

To achieve the primary goal of maximizing the global mission completion rate while autonomously minimizing energy consumption under defined operational and temporal constraints, we propose the Adaptive Hierarchical Semi-Markov Decision Process (AH-SMDP) framework. Prior to presenting the mathematical formulations, this subsection provides a high-level overview of how the designed components interconnect to form a unified optimization process.

As illustrated in Figure 1, the AH-SMDP framework adopts a “macro-decision and micro-execution” architecture. It is composed of four core components designed to address the research problems (RPs) outlined in the Introduction:

1.

Macro-Level Task Allocation (Addressing RP1): As depicted in the upper section of Figure 1, acting as the central coordinator, this layer utilizes a QMIX network to process the global swarm state and assign macro-tasks to individual UAVs at discrete decision steps.

2.

Micro-Level Variable-Duration Execution (Addressing RP1): Corresponding to the lower execution block in Figure 1, upon receiving task assignments, individual UAVs execute local flight actions. By modeling this underlying execution as an SMDP, the framework represents the variable-time costs associated with physical flight maneuvers (illustrated by ${\tau }_1,{\tau }_2,{\tau }_3$).

3.

Event-Driven Temporal Alignment Mechanism (Addressing RP2): Illustrated by the dashed feedback loop on the right side of Figure 1, this component serves as the temporal bridge between the macro and micro layers. Rather than updating at fixed intervals, it utilizes task completion events as triggers. It pauses the macro-level global update until the UAV with the longest execution time completes its assigned action ($\Delta T_K=max\left(t_i\right)$), thereby coordinating the swarm’s decision steps and mitigating asynchronous mismatches.

4.

State-Aware Adaptive Regulation & Action Pruning (Addressing RP3): Conceptually embedded within the micro-level action execution shown at the bottom of Figure 1, this mechanism operates to maintain operational safety boundaries. It monitors each UAV’s remaining battery capacity and temporal constraints, adaptively adjusting reward weights and masking high-risk flight paths to significantly reduce the likelihood of individual battery depletion during execution.

Initially, the macro-level allocates tasks via a top-down mapping operator. UAVs subsequently execute these assignments at the micro-level over variable durations, operating within the safety parameters maintained by the adaptive regulation mechanism. Upon reaching the event-driven temporal boundary, the actual time and energy costs incurred at the micro-level are aggregated into a cumulative return via a bottom-up feedback operator. This return is then used to update the global macro policy. Together, these interconnected components facilitate a coherent transition from high-level planning to low-level execution.

3.2. Hierarchical SMDP-Based Dual-Timescale Framework Modeling

At the macro-level, the system utilizes the global state $S_K$ (including UAV positions, remaining battery capacities, task specifications, and temporal constraints) to generate a joint action $A_K$ via a QMIX network, facilitating multi-UAV task assignment. This layer operates in discrete decision steps, primarily handling global coordination and resource allocation.

At the micro-level, each UAV receives its allocation and independently executes the corresponding task within its local environment. Unlike traditional approaches based on fixed time-steps, the execution process is formulated as an SMDP to accommodate variable-duration actions. This representation aligns more closely with the temporal dynamics of physical flight and control processes.

To manage hierarchical interactions, the framework incorporates a bidirectional information exchange mechanism. The macro-level translates global task assignments into local objectives for individual UAVs via a downward mapping operator ($M_{down}$). Concurrently, the cumulative rewards generated during micro-level execution are aggregated and fed back to the macro-level via an upward feedback operator ($M_{up}$) to support subsequent policy updates.

Furthermore, the framework operates under the centralized training with decentralized execution (CTDE) paradigm: the global state is accessible during the training phase, whereas individual UAVs rely on local observations for independent decision-making during the execution phase. It is assumed that the system can access critical status updates, such as remaining battery capacity and task progression, from onboard sensors.

3.2.1. Variable Definitions

To support reproducibility and state-space verification,

Table 1. Detailed definition of the hierarchical state and action spaces in AH-SMDP.

Layer	Variable Type	Symbol	Dimension	Physical Meaning and Description
Macro-	Global State	$S_K$	${\mathbb{R}}^{N\times 3}$	Includes UAV coordinates $P_{UAV}$, energy levels $E_{UAV}$, task coordinates $L_{Task}$, and deadlines $D_{Task}$.
Allocation Layer	Joint Action	$A_K$	${\mathbb{Z}}^{N\times 1}$	Discrete task assignment indices, $a_{i,K}\in \{1,2,\dots ,M\}$, representing the ID of the task or charging station assigned to UAV i.
Micro-	Local State	$s_{i,t}$	${\mathbb{R}}^{1\times 3}$	Includes relative distance $\Delta p_{i,t}$, real-time energy level $e_{i,t}$, local wind disturbance $v_{wind,i,t}$, and task urgency $t_{urgency}$.
Execution Layer	Local Action	$u_{i,t}$	$[-\pi ,\pi ]$	Includes continuous heading angle ${\theta }_{i,t}$, flight velocity $v_{i,t}$, and discrete suspend/charging mode bit $m_{charge}$.

summarizes the formal definitions of the variables across different system layers. Specifically, ${\tau }_{i,K}$ denotes the action duration, representing the number of physical time-steps elapsed during execution under discrete-time modeling. Consequently, the actual physical time cost is calculated as ${\tau }_{i,K}·\Delta t$, where $\Delta t$ represents the single-step control period. The composite action $u_{i,t}=\{{\theta }_{i,t},v_{i,t},m_{i,t}\}$ incorporates continuous control variables (e.g., heading angle $\theta$ and flight velocity v) alongside discrete mode variables (e.g., hovering or charging status m).

3.2.2. Mathematical Definition of Cross-Layer Interaction Mechanisms

To manage information exchange across hierarchical layers, this framework defines top-down and bottom-up mapping operators. Top-Down Mapping Operator ($M_{\mathrm{down}}$): The macro-level action $A_K$ is mapped to local target features for the underlying SMDP via an objective mapping function. The initial state $s_{i,0}$ of micro-agent i is formulated as follows:

$$s_{i,0}={\mathrm{proj}}_i\left(S_K\right)\oplus \Phi \left(a_{i,K}\right)$$

(1)

where ${\mathrm{proj}}_i\left(S_K\right)$ denotes the localized information extracted from the global state for agent i, and $\Phi \left(a_{i,K}\right)$ represents the intention feature vector. This operator translates macro-level task assignments into micro-level control objectives, guiding the low-level policy during local execution. Bottom-Up Feedback Operator ($M_{\mathrm{up}}$): The actual execution time ${\tau }_{i,K}$ and energy consumption incurred at the lower level are aggregated to compute the cumulative return for the macro-level. The cumulative return $R_{i,K}$ acquired by UAV i during the execution of macro-action $a_{i,K}$ is calculated as follows:

$$R_{i,K}={\displaystyle \sum _{t=0}^{{\tau }_{i,K}-1}}{\gamma }^t{r}_{lower}(s_{i,t},u_{i,t})+{\gamma }^{{\tau }_{i,K}}P\left(s_{i,{\tau }_{i,K}}\right)$$

(2)

where $r_{lower}$ is a state-aware multi-objective immediate feedback function that evaluates factors such as micro-level energy consumption, temporal constraints, and safety boundary penalties (detailed in Section 3.5). This return serves as an individual value input for computing the global value function $Q_{tot}$, which is modeled by the QMIX network (detailed in Section 3.3). Consequently, this operator quantifies micro-level execution outcomes into macro-level returns, establishing a closed-loop temporal optimization process.

3.2.3. Event-Driven Temporal Alignment Mechanism

Because the actual physical duration ${\tau }_{i,K}$ of actions at the micro-execution layer varies across UAVs, we introduce an event-triggered mechanism to facilitate multi-agent state synchronization [17]. The macro-level step size $\Delta T_K$ is dynamically anchored to the UAV requiring the longest execution time in the current decision round, defined as $\Delta T_K={max}_i\left({\tau }_{i,K}\right)$. An update of the global state $S_{K+1}$ is triggered upon the completion of this most time-consuming action. This synchronization mechanism helps align the swarm at a unified temporal boundary, thereby mitigating the environmental non-stationarity and state inconsistencies typically introduced by asynchronous execution.

Under these definitions, the macro-level state transition can be formalized as a joint transition probability distribution $P(S_{K+1},\Delta T_K\mid S_K,A_K)$. Jointly determined by the underlying physical environmental dynamics and the micro-level control policy, this distribution characterizes the state evolution of macro-decisions given variable-duration actions. Because this transition depends explicitly on the action duration in addition to the current state and action, it satisfies the mathematical definition of a Semi-Markov Decision Process (the proof is provided in Appendix A). For UAVs that complete their tasks prior to $\Delta T_K$, the lower level triggers a hovering or standby command to maintain operational positioning. The energy and time costs incurred during this standby state are evaluated via the immediate reward function $r_{lower}$, which are subsequently incorporated as penalty terms within the global policy’s optimization process. If the execution time of a single UAV exceeds the maximum tolerance threshold $T_{max}$ (i.e., $t-t_K\ge T_{max}$), the system interrupts the current round and initiates task reallocation. Thus, the proposed modeling framework establishes a coherent cross-layer optimization process across both temporal and structural dimensions. Building upon this foundation, the subsequent section details the multi-agent value decomposition network at the macro-level.

3.3. Macro-Level: Global Cooperative Decision-Making Based on Monotonic Value Decomposition

Following the formulation of the hierarchical framework, the macro-allocation layer determines the optimal task assignment policy within the joint action space by leveraging the cumulative returns $R_{i,K}$ provided by the micro-execution layer. Given that the dimensionality of the joint action space scales exponentially with the number of UAVs, we employ the QMIX value decomposition architecture, which operates under the centralized training with decentralized execution (CTDE) paradigm [18]. The topological structure of this architecture is illustrated in Figure 2.

3.3.1. Local Utility and Global Value Decomposition

During the decentralized execution phase, each UAV i maintains an independent local value network. Because local observations provide only partial information at each time step and cannot fully capture task execution progress or historical state dependencies, this paper integrates a Gated Recurrent Unit (GRU) to temporally encode the sequence of local observations. This process approximates the hidden state $h_{i,K}$, thereby enhancing the stability of local value estimation in partially observable environments. Based on $h_{i,K}$, the network outputs the individual utility value $Q_i(h_{i,K},a_{i,K})$ for a given macro-instruction $a_{i,K}$.

To achieve global cooperation during the offline training phase, the system integrates these local utility values through a non-linear mixing network to approximate the global joint value function $Q_{\mathrm{tot}}(S_K,A_K)$. To ensure that the combination of greedy actions from individual agents during decentralized execution is equivalent to the global optimal action, the mixing network must satisfy the structural monotonicity constraint, defined as:

$${\displaystyle \frac{\partial Q_{\mathrm{tot}}}{\partial Q_i}}\ge 0,\forall i\in \{1,2,\dots ,N\}$$

(3)

This constraint theoretically ensures that the global optimal action can be derived by maximizing the local utility functions of all agents independently. To implement this condition, the weights of the mixing network are generated by a hypernetwork conditioned on the global state $S_K$. Furthermore, non-negative constraints—such as an absolute value or softplus transformation—are applied to the mixing network weights to maintain the monotonicity requirement.

3.3.2. Value Network Update Incorporating Temporal Characteristics

Unlike traditional fixed time-step models, macro-state evolution in this framework explicitly depends on the duration of micro-actions. In the centralized training phase, the system utilizes an experience replay buffer $\mathcal{D}$ to optimize the network parameters $\theta$ by minimizing a temporal difference (TD) loss function $L\left(\theta \right)$ that incorporates SMDP characteristics, expressed as

$$L\left(\theta \right)={\mathbb{E}}_{\mathbb{D}}\left[{\left(y_K-Q_{\mathrm{tot}}(S_K,A_K;\theta )\right)}^2\right]$$

(4)

where the target value $y_K$ extends the SMDP Bellman equation within the value decomposition framework, calculated as follows:

$$y_K={\displaystyle \sum _{i=1}^N}{R}_{i,K}+{\gamma }^{\Delta T_K}\underset{A_{K+1}}{max}{Q}_{\mathrm{tot}}(S_{K+1},A_{K+1};{\theta }^{-})$$

(5)

Here, ${\sum }_{i=1}^N{R}_{i,K}$ represents the global joint return for the current macro-step. This aggregation method assumes that individual returns are approximately decomposable at the macro level, meaning the global cooperative reward can be represented as a weighted combination of the execution costs and benefits across all UAVs. In this task-scheduling context, because task completion rate, energy consumption, and timeout penalties can be independently evaluated at the individual level and linearly superimposed at the system level, this assumption provides a practical engineering approximation.

Additionally, ${\gamma }^{\Delta T_K}$ denotes the dynamic discount factor for variable-duration actions. This formulation applies a heavier discount penalty to decisions that require extended execution times, encouraging the policy to balance task rewards with temporal efficiency. This discount mechanism not only affects the value estimation but also backpropagates to the policy network via the TD error, thereby explicitly incorporating a temporal constraint into the parameter optimization process. Together, these mechanisms help mitigate modeling deviations arising from temporal misalignments between macro-level planning and micro-level execution. Building upon this macro-level cooperative architecture, the subsequent section details the adaptive reward mechanism designed for the micro-execution layer.

3.4. Micro-Level: SMDP Execution Modeling Incorporating Temporal Characteristics

Upon receiving the macro-instruction $a_{i,K}$ and conditional objectives via the intention projection operator $M_{down}$, the micro-execution layer performs action optimization within its local physical space. Given the variance in the durations of different micro-actions (e.g., rapid obstacle avoidance versus extended cross-regional flight), standard fixed-step MDPs can lead to an underestimation of physical time costs. The proposed approach mitigates the time-scale distortion introduced by forcibly discretizing variable-duration actions into fixed time steps. Therefore, this paper formulates the underlying single-agent control process as a local Semi-Markov Decision Process (SMDP) [19,20]. For any given UAV i, its local control process is formalized as a 5-tuple $\langle S_i,U_i,P_i,R_i,F_i\rangle$. Here, $S_i$ and $U_i$ denote the local state and action spaces defined in Section 3.2.1. The term $P_i(s_{i,t+\tau }\mid s_{i,t},u_{i,t})$ represents the environmental-state transition probability, capturing the cumulative evolution of the environment induced by physical dynamics during the execution of the action. Concurrently, a temporal distribution $F_i(\tau \mid s_{i,t},u_{i,t})$ is introduced to describe the probability of action $u_{i,t}$ lasting for a duration of $\tau$ given state $s_{i,t}$. This distribution is determined by the physical environmental dynamics and is empirically estimated through sampled trajectories in a model-free manner during algorithm implementation. Under this SMDP framework, the optimization objective for the micro-agent is to maximize the expected local cumulative discounted return. Because the action $u_{i,t}$ is executed continuously for $\tau$ steps, the local action-value function $Q_{\mathrm{lower}}(s_{i,t},u_{i,t})$ is extended via the Bellman optimality equation over a variable time domain as follows:

$$Q_{\mathrm{lower}}(s_{i,t},u_{i,t})={\mathbb{E}}_{\tau ,s^{\prime }}\left[{\displaystyle \sum _{k=0}^{\tau -1}}{\gamma }^k{r}_{\mathrm{lower}}(s_{i,t+k},u_{i,t})+{\gamma }^{\tau }\underset{u^{\prime }}{max}{Q}_{\mathrm{lower}}(s_{i,t+\tau },u^{\prime })\right]$$

(6)

In this equation, the summation term evaluates the continuous accumulation of immediate rewards during the execution of action $u_{i,t}$. Furthermore, ${\gamma }^{\tau }$ acts as a dynamic time-discount factor, encouraging the policy to favor time-efficient behaviors during the optimization process.

The local SMDP process triggers a state transition and truncation when any of the following conditions are met: (1) the macro-task objective is achieved; (2) safety boundary modes, such as hovering or emergency charging, are triggered; or (3) the individual execution time reaches the tolerance threshold $T_{max}$. Upon truncation, the cumulative results of this interval are transformed into the execution cost $R_{i,K}$, which is evaluated by the macro-layer via the bottom-up operator $M_{up}$. During implementation, the micro-level policy is approximated using a value-based deep reinforcement learning method, enabling experience reuse through parameter sharing. Consequently, micro-level outcomes are integrated into the macro-layer via cumulative returns, facilitating cross-layer consistency in temporal modeling and decision optimization.

3.5. Micro-Level: State-Aware Adaptive Penalty and Safety Bounds

Following the formulation of the micro-level SMDP value equation, the convergence direction of the underlying policy ${\pi }_{\mathrm{lower}}$ is primarily guided by the reward function. When confronting dynamic physical boundaries, such as battery depletion and strict temporal constraints, conventional static-weight reward mechanisms can struggle to maintain operational safety under extreme conditions. Consequently, this study develops a state-aware multi-objective adaptive penalty function [21] and introduces an action-masking mechanism based on state-space constraints to serve as an operational safety measure.

3.5.1. Distributed Multi-Objective Adaptive Penalty Function

To mitigate non-convergent exploration potentially induced by positive reward loops, this approach explicitly separates single-step execution costs from terminal task rewards. The foundational incentive for macro-tasks, denoted as $R_{base}$, along with critical boundary penalties, is encapsulated within the terminal state return $P\left(s_{i,{\tau }_{i,K}}\right)$ defined in Section 3.2.2. Conversely, during the micro-execution period at time step t, the immediate feedback signal $r_{\mathrm{lower}}(s_{i,t},u_{i,t})$ evaluates the continuous costs incurred by physical actions, formulated as

$$r_{\mathrm{lower}}(s_{i,t},u_{i,t})=-\left[{\omega }_e\left(s_{i,t}\right)·c_{\mathrm{energy}}\left(u_{i,t}\right)+{\omega }_t\left(s_{i,t}\right)·c_{\mathrm{time}}\right]$$

(7)

where $c_{\mathrm{energy}}(·)$ and $c_{\mathrm{time}}$ denote the estimated energy consumption rate and the base time cost within a single control period, respectively. Both terms are normalized to a range of $[0,1]$ via a maximum-value mapping to reduce the impact of numerical scale discrepancies on gradient updates. The time cost term represents the immediate temporal expenditure of local actions, whereas the discount factor ${\gamma }^{\tau }$ adjusts the present value of long-term returns. Operating across local and global timescales, respectively, these two elements are designed to mitigate modeling redundancy. To facilitate a dynamic trade-off between competing objectives across varying operational phases, the energy efficiency weight ${\omega }_e$ and the temporal weight ${\omega }_t$ are structured as continuously differentiable state-mapping functions. Given the non-linear discharge characteristics of lithium batteries, the energy efficiency weight is formulated as an exponential decay function of the remaining battery ratio $E_{\mathrm{ratio}}\in (0,1]$, expressed as follows:

$${\omega }_e\left(s_{i,t}\right)={\omega }_{e,\min }+({\omega }_{e,\max }-{\omega }_{e,\min })exp(-{\kappa }_e·E_{\mathrm{ratio}})$$

(8)

This formulation approximates the non-linear performance degradation of the battery in low-power states while maintaining gradient continuity. As the battery capacity decreases, ${\omega }_e\left(s_{i,t}\right)$ increases, thereby encouraging the policy to prioritize energy-conserving behaviors. To manage strict temporal constraints, this approach utilizes a relative temporal consumption rate, defined as ${\rho }_{\mathrm{time}}=t_{\mathrm{elapsed}}/T_{\mathrm{deadline}}$, and employs a Sigmoid function to structure the penalty curve. This relationship is formulated as follows:

$${\omega }_t\left(s_{i,t}\right)={\omega }_{t,\min }+{\displaystyle \frac{{\omega }_{t,\max }-{\omega }_{t,\min }}{1+exp(-{\kappa }_t({\rho }_{\mathrm{time}}-{\rho }_{\mathrm{threshold}}))}}$$

(9)

These mapping parameters are determined via grid search. The dynamic evolution of these multi-objective adaptive weights is illustrated in Figure 3.

This joint mechanism allows the system to accommodate exploratory behaviors aimed at discovering improved cooperative strategies during the initial mission phases, provided that battery capacity and temporal margins are sufficient. Conversely, as the system approaches critical battery thresholds or temporal limits, the corresponding penalty weights increase rapidly according to their respective exponential and sigmoid formulations. This dynamic response encourages the UAV policy to transition toward conservative operational modes, prioritizing energy conservation and timely execution.

3.5.2. Dynamic Action Space Pruning Under Safety Constraints

To address inefficient exploration caused by gradient delays near critical boundaries (e.g., battery depletion), we introduce a dynamic action masking operator [22] alongside the adaptive reward formulation. By explicitly nullifying the probabilities of unsafe actions, this operator restricts the search space to a safe domain, thereby accelerating convergence under strict constraints. Given the original action space $U_i$, a binary mask vector $M\left(s_{i,t}\right)\in {\{0,1\}}^{|U_i|}$ is generated based on local states and safety thresholds (Figure 4, Algorithm 1). The network’s output distribution $\pi$ is then renormalized as follows:

$${\pi }^{\prime }\left(u\right|s)={\displaystyle \frac{\pi \left(u\right|s)·M(u,s)}{{\sum }_{u^{\prime }}\pi \left(u^{\prime }\right|s)·M(u^{\prime },s)}}$$

(10)

Applying this mask directly to the output ensures that updates to the local value function $Q_{\mathrm{lower}}$ are strictly confined to the feasible subspace. Consequently, the dynamic contraction of this safety space is structured into three progressive tiers:

Algorithm 1 Execution Decision Logic based on State-Aware Action Masking

Require:  Local state $s_{i,t}$ of UAV i at time t (including remaining battery ratio $E_{\mathrm{ratio}}$, relative deadline consumption rate ${\rho }_{\mathrm{time}}$), adaptive energy penalty weight ${\omega }_e\left(s_{i,t}\right)$, predefined battery safety thresholds $E_{\mathrm{critical}}$ and $E_{\mathrm{low}}$. Ensure:  Micro-execution action $u_{i,t}$. 1: // Phase 1: Safety Shield Masking (Survival First) 2: if $E_{\mathrm{ratio}}\le E_{\mathrm{critical}}$ then 3:    $u_{i,t}\leftarrow$ Recharge                // Action mask M collapses, only recharge permitted 4: else if $E_{\mathrm{ratio}}\le E_{\mathrm{low}}$ and ${\omega }_e\left(s_{i,t}\right)>{\omega }_{e,\mathrm{threshold}}$ then 5:    $u_{i,t}\leftarrow$ Recharge                               // Action space shrinks under high penalty 6:    // Phase 2: Deadline Avoidance Masking (Time Priority) 7: else if ${\rho }_{\mathrm{time}}>0.9$ then 8:    $u_{i,t}\leftarrow$ Fast_Path                // Facing severe timeout risk, masking high-duration 9:    // Phase 3: Standard SMDP Policy Optimization under Normal Conditions 10: else 11:    $u_{i,t}\leftarrow \arg {\max }_{u\in U_i}{Q}_{\mathrm{lower}}(s_{i,t},u)$                      // Optimize within pruned space 12: end if 13: return $u_{i,t}$

Building upon this mechanism, the dynamic safety-constrained action masking process operates across three progressive stages:

Emergency Safety Constraint: If the remaining battery ratio $E_{\mathrm{ratio}}\le E_{\mathrm{critical}}$, the system identifies a critical state of energy depletion. The action space is strictly restricted to charging behaviors, prioritizing the operational survival of the UAV platform.

Temporal Boundary Constraint: If the temporal constraint indicator ${\rho }_{\mathrm{time}}>0.9$, the system initiates a temporal warning state. By masking highly time-consuming actions (e.g., extended exploration), the policy is directed to favor rapid execution paths.

Optimization within the Feasible Region: When neither of the specified physical or temporal boundaries is triggered, the UAV performs standard optimization within the unmasked action subspace. This allows the system to maximize task returns while operating within established safety margins.

These operational stages are triggered by continuous state variables, facilitating a gradual policy adjustment during execution and mitigating decision oscillations near operational boundaries. Furthermore, to prevent the complete masking of the action space, the mechanism is configured to retain at least one viable action under any given state, thereby preserving decision continuity. By restricting high-risk exploratory behaviors via explicit state-space constraints, this approach can effectively accelerate the convergence of the swarm policy in highly constrained environments.

3.6. Complexity and Theoretical Consistency Analysis

Following the formulation of the micro-level adaptive penalty and dynamic action masking mechanisms, this section examines their theoretical compatibility with the macro-level QMIX value decomposition architecture. Furthermore, it analyzes the computational and communication overhead of the AH-SMDP framework.

3.6.1. IGM Theoretical Consistency Under Constrained Action Spaces

To facilitate the convergence stability of the multi-agent cooperative policy, the AH-SMDP framework should satisfy the Individual–Global–Max (IGM) condition, implying that the optimality of the global joint action aligns with the composition of individual optimal actions. Incorporating the dynamic weights ${\omega }_e$, ${\omega }_t$, and the mask vector $M\left(s_{i,t}\right)$, we establish the following assumption: the local reward function $r_{\mathrm{lower}}$ is separable with respect to action dependencies, formulated as

$$r_{\mathrm{lower}}(s,u)=-\left[{\omega }_e·c_{\mathrm{energy}}\left(u\right)+{\omega }_t·c_{\mathrm{time}}\left(u\right)\right]$$

(11)

where the weighting coefficients ${\omega }_e$ and ${\omega }_t$ depend solely on the state s and remain independent of the action u. The action mask $M\left(s\right)$ is determined by environmental physical boundaries and temporal constraints, remaining independent of value network parameter updates. At any decision state, the restricted feasible action set maintains $|U_i|\ge 1$.

Theorem 1 (IGM Consistency under Constrained Action Spaces). 

Under the specified assumptions, the value mapping generated by the micro-level adaptive mechanism preserves monotonicity. Consequently, the IGM condition remains applicable for the optimal control problem defined over the constrained joint action space $U^N$, indicating that this constrained formulation shares an equivalent optimality structure with the original problem within the feasible domain.

Proof. 

First, we establish the monotonic mapping of the local value function. For any fixed state $s_{i,t}$, the weights ${\omega }_e$ and ${\omega }_t$ remain constant according to the established assumptions. Consequently, there exists a monotonically increasing transformation $f(·)$ such that $Q_{\mathrm{lower}}^{\prime }(s,u)=f\left(Q_{\mathrm{lower}}(s,u)\right)$. This transformation arises from applying state-dependent proportional scaling and translation operations to the original reward function, while maintaining a monotonic mapping for each state. Given that the transformation f is independent of the specific action u, this mapping preserves the partial ordering of actions, formulated as

$$\arg \underset{u\in U_i}{\max }{Q}_{\mathrm{lower}}^{\prime }(s,u)=\arg \underset{u\in U_i}{\max }{Q}_{\mathrm{lower}}(s,u)$$

(12)

Second, we analyze the consistency of the subspace contraction. The macro-level mixing network satisfies ${\displaystyle \frac{\partial Q_{\mathrm{tot}}}{\partial Q_i}}\ge 0$ via non-negative weight constraints. Because this monotonicity holds over any subset of the original action space $U^N$, when the optimization domain is projected from $U^N$ to the feasible subspace ${\prod }_{i=1}^N{U}_{\mathrm{safe},i}$, the global optimal joint action can still be derived by combining the local optimal actions. This decomposition is expressed as

$$\arg \underset{\mathbf{u}\in U^N}{\max }{Q}_{\mathrm{tot}}(\mathbf{s},\mathbf{u})=\left[\begin{array}{c}\arg {\max }_{u_1\in U_1}{Q}_1(s_1,u_1)\\ ⋮\\ \arg {\max }_{u_N\in U_N}{Q}_N(s_N,u_N)\end{array}\right]$$

(13)

Finally, although the masking mechanism limits the available action set and indirectly influences the sampling distribution of the policy, it does not directly participate in the parameter gradient computation. Therefore, it facilitates stable policy convergence within the feasible domain manifold without disrupting the structural monotonicity of the framework. This concludes the proof. □

3.6.2. Computational and Communication Complexity Analysis

1.

Time Complexity

During the decentralized execution phase, each UAV performs local inference at every physical time step t. Let $d_s$ denote the local observation dimension, $d_h$ the GRU hidden layer dimension, and $|U_i|$ the output action space size. The time complexity of a single decision step is dominated by matrix operations, approximated as $\mathcal{O}(d_s{d}_h+d_h^2+|U_i\left|d_h\right)$. Assuming parallel computation across the swarm, the total computational workload scales linearly with respect to the number of agents, yielding an overall complexity of $\mathcal{O}\left(N\right(d_s{d}_h+d_h^2+|U_i|d_h\left)\right)$. This linear scaling facilitates real-time control in highly dynamic environments.

2.

Communication Complexity

Compared to centralized approaches or methods relying on complex graph structures—where communication overhead typically ranges from $\mathcal{O}\left(N\right)$ to $\mathcal{O}\left(N^2\right)$—the AH-SMDP framework avoids continuous state broadcasting among UAVs during the micro-execution phase. The swarm limits state synchronization and instruction dispatch to $\mathcal{O}\left(N\right)$ operations, occurring only at the event-triggered temporal alignment boundaries (i.e., at the $\Delta T_K$ nodes). This structure restricts communication requirements to linear complexity at discrete intervals, thereby improving system applicability in communication-constrained scenarios.

4. Experiments and Results

To evaluate the effectiveness of the AH-SMDP framework in multi-UAV cooperative scheduling and to examine the alignment between empirical performance and theoretical derivations, this section presents a series of simulation experiments. Evaluations were conducted across five independent random seeds to account for environmental stochasticity and to assess the algorithm’s generalization capabilities. The results are reported as mean values with standard deviation bands, illustrating convergence stability and statistical consistency. The empirical findings indicate that the proposed framework maintains consistent performance across the evaluated randomized scenarios.

4.1. Experimental Setup and Baselines

The simulation environment models a multi-UAV cooperative scheduling task within an urban setting, as depicted in Figure 5. Tests were conducted across three representative scenarios, recording four primary evaluation metrics: makespan, task completion rate, energy consumption, and failure rate.

1.

Baseline Scenario: Characterized by sufficient resources and the absence of strict temporal constraints. This scenario primarily evaluates the fundamental scheduling performance of the algorithms under ideal conditions.

2.

Temporally Constrained Scenario: Imposes strict task deadlines to assess the adaptability of the algorithms under critical temporal limits.

3.

Multi-Constraint Scenario: Incorporates coupled operational constraints, such as battery degradation and charging station congestion, to examine the capacity of the algorithms to maintain operational safety.

4.1.1. Baseline Algorithms and Mechanisms

The baselines selected for this study encompass representative paradigms, including centralized, decentralized, temporal-aware, and heuristic optimization methods. We compare the AH-SMDP framework against MAPPO [23], which employs the Centralized Training with Decentralized Execution (CTDE) paradigm to train a centralized value function based on global states while updating decentralized policy networks at fixed time steps. This comparison assesses how the proposed framework mitigates the temporal mismatches typically observed in conventional MARL. To represent standard independent learning, PPO is utilized as a baseline [24,25]. In PPO, each agent optimizes its policy using only local observations without global state sharing, serving to evaluate the impact of macro-level coordination in the hierarchical design. Furthermore, an SMDP variant is included as a structural baseline [26]. By removing hierarchical cooperation and retaining only variable-duration action modeling, this variant isolates the performance gains attributable to the dual-timescale hierarchical strategy. Finally, a Genetic Algorithm (GA) is introduced as a heuristic reference. As a population-based search method for static sequences, GA provides a benchmark for illustrating the limitations of static planning in dynamic environments, compared to the proposed adaptive approach. Detailed parameter configurations are provided in Appendix B.

4.1.2. Evaluation Metrics and Justifications

To evaluate the scheduling policies against the research problems defined in Section 2, we explicitly quantify the scheduling performance of the proposed framework using the following metrics:

• Makespan ($\mathit{M}$): This metric evaluates the temporal efficiency of the swarm, defined as the maximum physical completion time among all N UAVs:

  $$M=\underset{i\in \{1,\dots ,N\}}{max}\left\{t_i^{\mathrm{end}}\right\}$$

  (14)

  where $t_i^{\mathrm{end}}$ represents the time step at which UAV i completes its final assigned task. A lower makespan indicates higher efficiency in event-driven temporal alignment.
• Task Completion Rate ($\mathit{CR}$): This metric assesses the framework’s ability to satisfy strict temporal constraints, defined as the ratio of successfully completed tasks to the total number of generated tasks:

  $$CR=\left({\displaystyle \frac{N_{\mathrm{completed}}}{N_{\mathrm{total}}}}\right)\times 100\%$$

  (15)

  where $N_{\mathrm{completed}}$ is the number of tasks finished before their respective deadlines.
• Total Energy Consumption (${\mathit{E}}_{\mathrm{total}}$): This metric aggregates the physical energy expended by the swarm over the entire mission duration:

  $$E_{\mathrm{total}}={\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{t=0}^{T_{\max }}}{e}_i\left(t\right)$$

  (16)

  where $e_i\left(t\right)$ denotes the instantaneous energy consumed by UAV i at time step t, encompassing both flight maneuvers and hover/standby states.
• Failure Rate ($\mathit{FR}$): The failure rate quantifies the frequency of task terminations caused by boundary violations:

  $$FR=\left({\displaystyle \frac{N_{\mathrm{failed}}}{N_{\mathrm{total}}}}\right)\times 100\%$$

  (17)

  where $N_{\mathrm{failed}}$ represents the number of tasks aborted due to operational constraint violations, such as battery depletion or critical deadline breaches.

4.2. Experimental Results and Analysis

The empirical results are illustrated in Figure 6 and Figure 7. Under highly constrained scenarios, the AH-SMDP framework maintains consistent performance, demonstrating a strategic trade-off that prioritizes operational safety.

In the Baseline Scenario, MAPPO achieved a makespan of 13.5 min, whereas the AH-SMDP framework recorded 17.8 min. This difference reflects the strategic trade-off inherent in safety-aware optimization: scheduling efficiency is partially compromised to prioritize operational safety. Because the adaptive weights ${\omega }_e$ and ${\omega }_t$ maintain a foundational penalty even when resources are sufficient, the policy favors slightly longer, safer navigation paths to avert potential constraint violations. This conservative approach establishes the foundation for maintaining high task completion rates in more constrained environments.

Under the Temporally Constrained Scenario, the task completion rates for MAPPO and PPO decreased to 70.2% and 60.5%, respectively. In contrast, AH-SMDP maintained a completion rate of 98.7% (±0.6%). This performance advantage is primarily attributable to the combined effects of the temporal alignment mechanism (Section 3.2.3) and the constraint-aware action masking (Section 3.5.2). By structurally addressing the decision-execution misalignment caused by variable-duration actions, the framework effectively handles strict temporal limits.

In the Multi-Constraint Scenario, the AH-SMDP framework recorded zero task failures across all evaluated test runs. This operational safety is largely driven by the dynamic action masking mechanism (Section 3.5.2): as the remaining battery capacity approaches the critical threshold $E_{\mathrm{critical}}$, the mask vector $M\left(s_{i,t}\right)$ restricts the selection of unsafe execution paths. However, an observed limitation is that frequent triggering of the masking mechanism by coupled physical constraints can induce highly conservative behavior, potentially leading to a degradation in global scheduling efficiency under severe edge-case conditions.

4.3. Scalability Analysis in Large-Scale Swarms

To assess the scalability of the framework, the Multi-Constraint Scenario was expanded by scaling the number of UAVs (N) from 10 to 50. Task density was increased proportionally to maintain a consistent environmental load. The comparative results concerning task completion rate, total energy consumption, and average computation time are presented in Figure 8.

As the joint action space expands with the swarm size, baseline methods exhibit varying degrees of performance degradation. As illustrated in Figure 8A, at $N=50$, independent learning (PPO) and heuristic methods (GA) fail to navigate the coupled constraints, with task completion rates dropping below 5%. While MAPPO achieves optimal completion rates at smaller scales ($N\le 30$), its performance declines to 84.0% at $N=50$. More importantly, it incurs a high total energy consumption of 2228 units, suggesting inefficient exploration and severe local congestion within the expanded state space (Figure 8B).

In contrast, the AH-SMDP framework structures the expanded state space through its macro-micro decoupling architecture. Although its completion rate at smaller scales is marginally lower than MAPPO due to its conservative energy-saving policy, it achieves a comparable completion rate (84.0%) at $N=50$ while reducing total energy consumption by approximately 29% (1590 units). Furthermore, as shown in Figure 8C, while independent PPO exhibits the lowest inference time due to its lack of global coordination, its inability to satisfy task constraints renders it ineffective. The dynamic action masking mechanism enables the average inference time of AH-SMDP to remain in the sub-millisecond range (∼0.48 ms), aligning with the theoretical linear complexity trend ($\mathcal{O}\left(N\right)$). Conversely, the heuristic approach (GA) exhibits an exponential increase in computational delay. These findings suggest that the AH-SMDP framework possesses robust scalability and effective energy management capabilities for large-scale UAV deployments.

4.4. Ablation Study of Core Mechanisms

To isolate the contributions of individual modules within the AH-SMDP framework and analyze their interactions, this section presents a comparative evaluation using three model variants. The evaluated variants include: H-SMDP (omitting the adaptive reward mechanism), A-SMDP (excluding the hierarchical architecture), and AH-MDP (replacing the event-driven temporal alignment with a traditional fixed-step decision process). The empirical results and performance comparisons for these configurations are detailed in Figure 9 and

Table 2. Quantitative analysis of the ablation study for AH-SMDP and its variants in the extreme multi-resource scenario.

Model Variant	Removed Module Description	Makespan (min)	Makespan Degrad. Ratio	Comp. Rate (%)	Total Energy Consum. (Units)	Conv. Variance (${\mathit{\sigma }}^2$)	Variance Deter. Ratio
AH-SMDP (Ours)	Complete dual-layer adaptive model	17.8	-	100%	100	12.4	-
H-SMDP (w/o A)	Removed adaptive reward mechanism	24.5	+37.6%	85%	185	18.2	+46.7%
A-SMDP (w/o H)	Removed macro-level QMIX hierarchy	42.8	+140.4%	65%	240	45.6	+267.7%
AH-MDP (w/o SMDP)	Degraded to standard discrete MDP	38.5	+116.2%	70%	280	94.5	+662.1%

Note: The data presented above are derived from the post-convergence means across five independent random-seed experiments. The degradation ratios are calculated using the performance of AH-SMDP as the baseline.

.

First, this study examines the impact of temporal processing on system stability. By omitting the event-triggered mechanism and reverting the framework to a traditional fixed-step formulation (AH-MDP), the overall scheduling performance experiences a notable decline. Because the physical execution times of different UAV actions vary significantly, forcibly discretizing these actions into fixed intervals introduces temporal asynchrony among the agents. This misalignment results in a decision-execution mismatch. Quantitatively, the training fluctuation variance of AH-MDP increases to 94.5 (a more than six-fold increase compared to the full model), and the total energy consumption rises to 280 units. These results indicate that synchronizing the decision rhythm via event-driven temporal alignment facilitates the reduction of multi-agent coordination errors and preserves system stability.

Regarding architectural design, the A-SMDP variant omits the upper-level macro-allocation network, requiring UAVs to optimize actions directly within the full global state space. Without macro-level coordination, the agents exhibit an increased tendency for resource contention and operational conflicts, as illustrated in the radar chart in Figure 9. Quantitatively, the makespan extends to 42.8 min (a 140.4% increase), and the lack of coordinated operations reduces the task completion rate to 65%. This comparison demonstrates the structural advantages of the hierarchical architecture: decomposing the global scheduling problem into manageable local tasks effectively mitigates congestion and resource contention during multi-UAV cooperation.

At the micro-execution level, the H-SMDP variant omits the state-aware adaptive reward mechanism. Consequently, the system struggles to adapt to constrained conditions, such as energy depletion. Empirical data indicate that the total energy consumption of this variant exceeds that of the full model by approximately 85% (reaching 185 units), alongside a notable reduction in the task completion rate. Theoretically, the dynamic action masking and adaptive reward mechanisms operate synergistically to establish a structural safety bound. Action masking serves as a foundational constraint by strictly restricting hazardous selections (e.g., preventing long-distance exploration under critical battery levels). Concurrently, within the unmasked feasible domain, the adaptive reward formulation encourages the UAVs to dynamically balance execution speed and energy conservation. These integrated components are critical for facilitating the reliable operation of the UAV swarm across diverse physical constraints.Building upon the performance advantages established in the preceding sections, further examination of the framework’s sensitivity to hyperparameter variations and scale expansion is warranted.

4.5. Sensitivity and Scalability Analysis

The stability and scalability of the AH-SMDP framework were evaluated through a sensitivity analysis of the energy efficiency decay coefficient (${\kappa }_e$) and the temporal-constraint inflection threshold (${\rho }_{\mathrm{threshold}}$). Figure 10 illustrates the impact of these parameters on the task completion rate and total energy consumption for swarm scales of $N=10$ and $N=50$.

The analysis of ${\kappa }_e$ indicates a general trend where higher values correlate with more conservative energy policies, aligning with the exponential decay mechanism established in Equation (8). For smaller swarms ($N=10$), this conservatism consistently reduces both total energy consumption and the task completion rate. For larger swarms ($N=50$), the system maintains the completion rate more steadily across moderate values (${\kappa }_e\le 8$). However, a localized fluctuation in energy consumption is observed at ${\kappa }_e=8$ for $N=50$. This increase can be attributed to intensified local congestion and uncoordinated detours when a large number of agents simultaneously adopt highly conservative flight strategies. The empirical data suggest that the default setting (${\kappa }_e=5.0$) achieves a practical trade-off between energy conservation and task completion across the evaluated scales.

In parallel, evaluating the temporal constraint inflection threshold (${\rho }_{\mathrm{threshold}}$) indicates that the Sigmoid mapping in Equation 9 regulates the decision rhythm. When ${\rho }_{\mathrm{threshold}}$ exceeds the default value of 0.6, the swarm exhibits a delayed response to approaching temporal limits. While this delay causes a steady decrease in completion rates for $N=10$, the larger swarm ($N=50$) effectively sacrifices energy efficiency to maintain a relatively stable completion rate, triggering a sharp increase in total energy consumption. This phenomenon highlights an underlying scalability dynamic: in high-density environments, delayed urgency responses exacerbate cascading conflicts and redundant collision-avoidance maneuvers, thereby increasing the collective energy cost. Consequently, these results indicate that maintaining ${\rho }_{\mathrm{threshold}}$ at 0.6 helps mitigate the impact of scale variance, supporting the framework’s applicability in constrained scheduling environments.

4.6. Training Stability and Convergence Analysis

Figure 11 and Figure 12 illustrate the learning behaviors of the evaluated algorithms. The empirical results indicate that standard reinforcement learning methods exhibit limitations when processing actions of variable duration. For instance, although MAPPO demonstrates rapid improvement during the initial training phase, its reward curve experiences significant fluctuations in the middle and late stages. This instability primarily arises because traditional fixed-step models struggle to accurately capture the actual physical time consumed by different actions. Consequently, this discrepancy introduces temporal misalignment during policy evaluation, complicating the accurate attribution of rewards to effective actions. This observation aligns with the credit assignment challenges of conventional deep reinforcement learning in variable-duration trajectory optimization, as noted in recent literature [27].

In contrast, the AH-SMDP framework demonstrates steady performance improvements during the middle and late stages of training. Because the hierarchical architecture requires initial interaction overhead to align the macro and micro policies, the early convergence process is relatively gradual. However, after approximately 200 episodes, the system enters a stable convergence phase. This stability is primarily attributable to the event-driven temporal alignment mechanism. By standardizing the decision rhythm of each agent, this mechanism mitigates the temporal inconsistencies caused by asynchronous execution, thereby facilitating the stability of global cooperation.

Observing the standard deviation bands in Figure 12, the performance fluctuations of the AH-SMDP framework remain well-constrained. This reflects the structural role of the dynamic action-masking mechanism during the exploration phase. By restricting high-risk selections that could lead to task failure, this mechanism directs policy updates exclusively within the unmasked feasible region. This empirical observation aligns with the theoretical derivations regarding convergence stability presented in Section 3.6.1. Concurrently, mitigating invalid exploration refines the optimization trajectory, preventing the performance degradation typically associated with frequently violating constraint boundaries.

From an overall convergence perspective, the AH-SMDP framework prioritizes long-term systemic stability over the absolute efficiency of individual task executions. Although its makespan slightly trails that of MAPPO in the Baseline Scenario, its training process exhibits notably reduced variance. The empirical data indicate that the adaptive reward mechanism encourages the policy to select conservative, reliable trajectories as it approaches physical limits. This strategic trade-off mitigates the performance degradation commonly observed in baseline algorithms upon violating safety constraints, demonstrating that the proposed method maintains consistent performance in highly constrained environments.

5. Conclusions and Future Work

To address the cooperative scheduling challenges of large-scale UAV swarms operating under coupled spatio-temporal constraints, this study proposes the AH-SMDP framework. The performance of this framework has been evaluated through theoretical analysis and empirical simulations across multiple scenarios. The primary findings are summarized as follows:

1.

Impact of the Hierarchical Architecture and Temporal Alignment: The proposed “macro-allocation, micro-execution” hierarchical architecture decouples task assignment from underlying execution, thereby mitigating the dimensionality of the joint action space. Building upon this structure, the integration of semi-Markov modeling with event-driven temporal alignment structurally synchronizes variable-duration physical actions with discrete decision steps. This alignment mitigates the environmental non-stationarity typically induced by asynchronous multi-agent interactions.

2.

Impact of Constraint Awareness and Resource Management: Empirical data indicate a complementary relationship between the state-aware adaptive reward and the dynamic action masking mechanisms. Through physical boundary restrictions, action masking restricts high-risk explorations under critical operating conditions. Concurrently, within the unmasked feasible domain, the adaptive reward function encourages the policy to maintain a dynamic balance between task timeliness and energy conservation. In multi-constraint scenarios, this combined strategy facilitates operational safety while demonstrating improved energy efficiency compared to baseline algorithms.

3.

Alignment of Theoretical Compatibility and Performance Trade-offs: The convergence trajectories observed in the simulation environments align with the theoretical derivations presented in Section 3.6.1. This supports the premise that policy optimization conducted within a restricted action subspace preserves the monotonicity structure of the value decomposition network. Furthermore, although the integration of multi-objective safety constraints incurs a partial efficiency trade-off in the Baseline Scenario, this strategic approach improves the system’s capacity to maintain operational safety under highly constrained scheduling conditions, illustrating a necessary trade-off to enhance overall swarm adaptability.

While the AH-SMDP framework demonstrates performance stability in highly constrained environments, several limitations warrant further investigation. Future work will incorporate practical operational factors, including communication delays, packet losses, and perception noise. By integrating graph-based methodologies, subsequent research aims to address challenges in decentralized cooperation and credit assignment under partially observable conditions. Furthermore, to mitigate the current reliance on manually configured hyperparameter thresholds, future studies will explore meta-learning-based adaptive regulation mechanisms. This approach would enable the framework to dynamically adjust constraint intensities based on varying task distributions, thereby facilitating a more resilient trade-off between operational safety and global scheduling efficiency.