Joint Trajectory Design and Resource Allocation for QoS-Aware Emergency Data Collection in UAV-Assisted WPCNs: A Hierarchical DRL Approach

Abstract

Unmanned aerial vehicle (UAV)-assisted wireless-powered communication networks (WPCNs) have emerged as a promising solution for energy-constrained Industrial Internet of Things systems, where ground sensor nodes are often deployed in harsh and hard-to-reach environments. However, efficient UAV-assisted data collection remains challenging due to limited UAV onboard energy, realistic propulsion consumption, and varying quality-of-service requirements of industrial nodes. This paper investigates an energy efficiency maximization problem in a UAV-assisted WPCN by jointly optimizing the UAV trajectory, hovering altitude, and hybrid TDMA/NOMA resource allocation. To solve the resulting high-dimensional and highly coupled problem, we propose a deep reinforcement learning-driven Hierarchical Energy-Efficient Data Collection scheme, which we name DRL-HEEC. Specifically, a Double Deep Q-Network is employed at the upper level to optimize the UAV trajectory with altitude state inheritance, while a lower-level optimization engine based on the Dinkelbach method, block coordinate descent, and successive convex approximation is developed to handle heterogeneous resource allocation. Simulation results show that the proposed DRL-HEEC scheme outperforms other baselines. In particular, DRL-HEEC improves the system energy efficiency by approximately $9\%$ to $12\%$ compared to other reinforcement learning-based algorithms while ensuring QoS satisfaction on the part of emergency nodes.

1. Introduction

The rapid proliferation of the Industrial Internet of Things (IIoT) has driven a paradigm shift in modern industrial automation, enabling the large-scale deployment of interconnected sensors in harsh, remote, and expansive environments such as smart mines, automated ports, and large-scale manufacturing plants [1]. In these scenarios, continuous environmental monitoring, equipment diagnostics, and emergency anomaly reporting are essential for ensuring production safety and operational efficiency. However, industrial sensor nodes are often deployed in hazardous or hard-to-reach areas, which makes frequent battery replacement impractical or even infeasible. To overcome this energy bottleneck, the integration of unmanned aerial vehicles (UAVs) and wireless-powered communication networks (WPCNs) has emerged as a promising solution [2]. Benefiting from their flexible three-dimensional (3D) mobility and adjustable operating altitude, UAVs can serve as mobile hybrid access points (HAPs) to establish favorable line-of-sight (LoS) links with ground industrial nodes. In this way, UAV-mounted HAPs can proactively broadcast radio-frequency energy to charge ground nodes and subsequently collect their sensory data.

Despite these advantages, the practical deployment of UAV-assisted WPCNs in IIoT environments still faces significant challenges due to differing service requirements and tightly coupled physical constraints [3]. On the one hand, industrial sensing data are inherently heterogeneous. Critical alarm nodes usually require ultra-reliable transmission and strict quality-of-service (QoS) guarantees with specific throughput thresholds, whereas regular monitoring nodes mainly require best-effort data uploading. Traditional orthogonal multiple access (OMA) schemes, such as time-division multiple access (TDMA), suffer from low spectral efficiency when serving massive regular nodes. In contrast, pure non-orthogonal multiple access (NOMA) schemes may have difficulty providing deterministic QoS guarantees for critical emergency traffic. To address this trade-off, a hybrid approach can be utilized in which emergency nodes use exclusive channels (TDMA) for guaranteed QoS and regular nodes share the remaining spectrum via NOMA. On the other hand, the onboard energy capacity of UAVs is intrinsically limited. Most existing studies adopt simplified two-dimensional (2D) horizontal trajectory models with fixed flight altitudes, thereby neglecting the realistic aerodynamic propulsion energy consumption caused by 3D maneuvers. In complex industrial environments, the UAV needs to dynamically adjust its 3D hovering altitude to adapt to the varying LoS probability and channel conditions [4], resulting in a highly coupled optimization problem involving spatial trajectory planning, altitude control, and wireless resource allocation.

Solving this multidimensional optimization problem is particularly challenging because of the strong coupling between macroscopic UAV trajectory planning and microscopic physical-layer resource allocation. Such continuous state inheritance introduces strong temporal dependency among consecutive visiting decisions and breaks the independence among local optimization subproblems. As a result, traditional mathematical optimization methods such as block coordinate descent (BCD) and successive convex approximation (SCA) may suffer from prohibitive computational complexity, and can hardly guarantee global optimality for the overall coupled problem. Conversely, directly applying an end-to-end deep reinforcement learning (DRL) approach to the entire problem may lead to the curse of dimensionality. Moreover, a purely data-driven neural network may fail to strictly satisfy rigid analytical constraints [5] such as deterministic QoS thresholds and non-convex successive interference cancellation (SIC) feasibility conditions, especially during the exploration stage. Therefore, an effective optimization paradigm is required to decouple global UAV trajectory planning from local power and time resource scheduling while preserving the feasibility of varied service constraints.

Motivated by the above challenges (as shown in Figure 1), this paper investigates the energy efficiency (EE) maximization problem in a UAV-assisted WPCN by jointly optimizing the UAV trajectory, hovering altitude, and NOMA/TDMA resource allocation.

To decouple the spatial–temporal decision variables and handle the high-dimensional system state, we propose a DRL-driven Hierarchical Energy-Efficient Collection scheme, referred to as DRL-HEEC. The proposed scheme adopts a two-tier offline-to-online decision mechanism to jointly address the 3D state inheritance of UAV movement and heterogeneous resource scheduling among nodes. Specifically, a hierarchical DRL framework is developed in which the upper layer employs a Double Deep Q-Network (DDQN) to determine the global UAV trajectory based on inherited physical states and the lower layer solves the intra-cell resource allocation problem by combining analytical optimization techniques. The main contributions of this paper are summarized as follows:

• We propose the hierarchical DRL-HEEC scheme, which structurally decouples macroscopic UAV trajectory planning from microscopic wireless resource allocation. Within this architecture, an upper-layer DDQN agent dynamically determines the global UAV trajectory as a discrete spatial routing problem over hexagonal sub-cells according to the inherited UAV physical states, while a lower-layer optimization engine optimizes intra-cell time scheduling, access mode selection, and hovering altitude. This hierarchical design effectively reduces the dimensionality of the original coupled problem, improving both convergence performance and execution efficiency.
• We integrate a rotary-wing aerodynamic propulsion energy model to characterize the realistic energy consumption of UAVs during 3D flight maneuvers. To capture the temporal dependency caused by altitude variation, we introduce a discretized altitude state inheritance mechanism in the Markov decision process (MDP). Specifically, the terminating altitude of the UAV in the previous grid cell is treated as part of the initial state for the subsequent grid cell, enabling the DDQN agent to learn EE-aware trajectory decisions under practical 3D mobility constraints.
• To support heterogeneous QoS requirements, we formulate a lower-layer joint resource allocation problem for hybrid multiple access (HMA) transmission. Strictly orthogonal TDMA time slots are assigned to critical alarm nodes in order to guarantee their throughput requirements, while regular monitoring nodes are served through NOMA to improve spectral efficiency. By exploiting the Dinkelbach method together with BCD-SCA techniques, the proposed resource allocation engine jointly optimizes time allocation, SIC decoding order, and transmission scheduling, thereby improving the sum rate of regular nodes while satisfying the QoS constraints of critical nodes. Crucially, the optimized objective value derived from this lower-layer analytical engine is subsequently fed back as a deterministic reward signal to guide the upper-layer DRL trajectory exploration.
• Extensive simulation results demonstrate that the proposed DRL-HEEC scheme outperforms other baselines. The results show that the proposed scheme effectively improves global energy efficiency, especially under limited UAV battery capacity and dense node deployment. In addition, DRL-HEEC maintains QoS satisfaction for critical alarm nodes, verifying its effectiveness and robustness in complex WPCNs.

The rest of this paper is organized as follows: Section 2 reviews the related work; Section 3 describes the system model; Section 4 formulates the lower-level optimization problem and develops the corresponding solution method; Section 5 presents the upper-level reinforcement learning algorithm; Section 6 presents and discusses the simulation results; finally, Section 7 concludes the paper with a summary and discussion of future work.

2. Related Work

UAV-assisted communication networks have been widely studied due to their flexible deployment and mobility-enabled performance gains. However, their optimization is challenging because UAV trajectory, channel conditions, energy consumption, and resource allocation are highly coupled [6,7,8]. Existing works mainly address these issues through trajectory optimization and artificial intelligence-based methods, as reviewed in the following subsections.

2.1. Trajectory Optimization in UAV-Assisted Communication

In UAV-assisted communication systems, trajectory optimization is crucial for enhancing system-wide data collection performance [9,10]. To address this, the recent literature has extensively investigated various trajectory planning methodologies coupled with resource allocation strategies [11,12,13]. For instance, Zhang et al. [14] investigated the joint design of 3D UAV trajectory and time allocation for aerial data collection in hybrid-powered NOMA-IoT networks, seeking to maximize fair network throughput. Cao et al. [15] proposed a novel reinforcement learning-based heuristic algorithm that utilizes an attention mechanism to extract dynamic network topologies and delay priorities, enabling online UAV trajectory optimization to jointly minimize energy consumption and average sensor delay. Focusing on total energy efficiency, Li et al. [16] and Zhang et al. [17] developed a joint resource allocation and 3D trajectory optimization scheme for UAV-enabled networks. By actively manipulating the UAV’s altitude and horizontal position to improve channel gain differentiation, their strategy was able to achieve superior NOMA decoding performance compared to rigid fixed-altitude flight patterns. Zuo et al. [18] formulated a mixed-integer nonlinear programming (MINLP) problem to maximize the energy efficiency of fixed-wing UAVs by optimizing their 3D trajectories within multi-obstacle environments. Liu et al. [19] minimized the average age of information (AoI) metric by jointly optimizing the sensor node transmission power, island clustering, and UAV flight trajectory. In [20,21], the authors explored multi-UAV enabled wireless networks in which trajectory and communication were jointly optimized to maximize the minimum throughput. Despite these significant advancements, a critical limitation persists in the current literature, as existing studies either simplify the problem by assuming a constant UAV flight altitude or fail to consider whether altitude affects communication links.

2.2. Applications of Artificial Intelligence in UAV Communications

The integration of artificial intelligence into UAV communications is profoundly reshaping the architecture and performance of future wireless networks, encompassing full-stack optimization from physical-layer signal processing [22] to network-layer resource allocation [23]. In [24,25], the authors utilized deep reinforcement learning algorithms to jointly optimize the 3D flight trajectories and transmitting powers of UAVs, significantly reducing energy consumption and mitigating co-channel interference while satisfying QoS constraints. Wang et al. [26] investigated resource allocation in UAV-D2D networks by proposing a scalable heterogeneous multi-agent mean-field actor–critic framework. Karegar et al. [27] proposed a deep Q-learning-based fuzzy UAV path optimization method for software-defined wireless sensor networks; their approach effectively minimized system energy consumption while significantly enhancing communication bit rates and packet delivery performance. To ensure threshold AoI for UAV-assisted mobile crowd sensing under limited energy supplies, Wang et al. [28] proposed a decentralized multi-agent deep reinforcement learning framework, which they termed DRL-UCS(${\mathrm{AoI}}_{th}$). It incorporates a transformer-enhanced distributed architecture and an adaptive intrinsic reward mechanism to maximize data collection while minimizing AoI violation ratios. Xiao et al. [29] proposed a proximal policy optimization (PPO)-based UAV-EH scheme with action space cropping to optimize meta-surface allocation, effectively enhancing energy harvesting and extending UAV endurance. Alam and Moh [30] proposed a joint trajectory control, frequency allocation, and routing (JTFR) algorithm that maximizes link utility by considering link stability, queuing delay, and residual energy. Their approach integrates an adaptive distributed multi-agent deep deterministic policy gradient approach with swarm behavior to obtain the optimal solution. Artificial intelligence and especially deep reinforcement learning have shown great potential in UAV-assisted communication networks by enabling adaptive trajectory planning and resource allocation; however, practical deployment is still hindered by the curse of dimensionality, high training complexity, large sample requirements, limited onboard computation capability, and the sim-to-real performance gap.

Motivated by these limitations, this work investigates energy-efficient and QoS-aware UAV-assisted communication by considering differing QoS requirements of different nodes and altitude-dependent air-to-ground links. We develop a two-layer decoupled optimization framework that separates trajectory planning from communication resource optimization, aiming to improve energy efficiency while reducing problem complexity and preserving their coupling effects. Specifically, the maximized objective value from the lower layer serves as the exact reward signal guiding the upper layer’s spatial exploration. Unlike purely decoupled benchmarks that optimize these aspects in isolation, our framework reduces problem complexity while explicitly preserving the spatiotemporal trade-offs required to maximize overall EE.

3. System Model

3.1. Network Model

In this paper, we consider a mission scenario where a mobile UAV is dispatched to collect data across a WPCN. As illustrated in Figure 2, the overall network consists of a stationary HAP, the mobile UAV, and N sensor nodes distributed over the target region.

The HAP, located at the coordinates $(x_1,y_1,H_0)$, is primarily responsible for charging the UAV and receiving the data collected by the UAV, where $H_0$ denotes the altitude of the HAP above ground level, which is also set as the initial flight altitude of the UAV. To model the practical scenario, we partition the target area into M identical regular hexagonal grid cells, denoted by the set $\mathcal{M}=\{1,2,\dots ,M\}$. For any $m\in \mathcal{M}$, the center coordinates of the m-th grid cell are represented as $(x_m,y_m,0)$. Let $N_m$ denote the number of sensor nodes within the m-th hexagonal grid cell. Specifically, the total number of sensor nodes N satisfies

$$\sum _{m\in \mathcal{M}}{N}_m=N.$$

(1)

The i-th sensor node in grid cell m is denoted by $S_{i,m}$, with its coordinates represented as $s_{i,m}=(x_{i,m},y_{i,m},0)$, where $i\in \{1,2,\dots ,N_m\}$. These sensor nodes are equipped with energy harvesting (EH) modules. Operating under the harvest-then-transmit protocol, the nodes first harvest the radio frequency (RF) energy radiated by the UAV and use the accumulated energy to transmit their data back to the UAV. It is worth noting that the data collected by the nodes is categorized into regular data and emergency data; each node has a probability $P_e$ of generating emergency data (e.g., anomaly monitoring data and critical data). Consequently, emergency data should be granted prioritized transmission so as to satisfy a predefined minimum throughput threshold.

In this network architecture, the UAV plays the dual role of both mobile energy transmitter (MET) and mobile data collector (MDC). Specifically, the UAV flies horizontally along a dynamic trajectory to the center of another grid cell in order to execute energy broadcasting and data collection. Once the UAV hovers at the center of a grid cell, it first executes the wireless energy transfer (WET) phase, broadcasting RF energy to all nodes within this grid cell in the downlink. During the wireless information transfer (WIT) phase in the uplink, it collects the data transmitted via these nodes by employing the HMA strategy (as detailed in Section 3.5). To further enhance system performance, the hovering altitude of the UAV within each grid cell can be dynamically adjusted. Let $H_m$ denote the hovering altitude of the UAV when it hovers at the center of grid cell m. The adjustable altitude is bounded by

$$H_{\min }\le H_m\le H_{\max },$$

(2)

where $H_{\min }$ and $H_{\max }$ respectively represent the minimum and maximum safe flight altitudes of the UAV.

Furthermore, all operational energy of the UAV is drawn from its equipped battery with a capacity of $C_{\mathrm{uav}}$ (in J). To safely sustain the multi-tour operation, the residual energy $C_{\mathrm{res}}$ must strictly satisfy the condition

$$C_{\mathrm{res}}\ge C_{\mathrm{th}},$$

(3)

where $C_{\mathrm{th}}$ is a predefined energy threshold guaranteeing a safe return flight. Once $C_{\mathrm{res}}$ drops to this threshold, the UAV will finish its current data collection and return to the HAP in the next step. After being fully recharged at the HAP, the UAV will resume its uncompleted data collection tasks.

3.2. Proposed Scheme Overview

To achieve data collection that balances the QoS for emergency data and the global energy efficiency in large-scale wireless sensor networks, this paper proposes a DRL-HEEC scheme. In this subsection, we first present the overall operational workflow of our proposed scheme.

As illustrated in Figure 3, by adopting a two-tier optimization architecture, the proposed scheme decouples the highly complex joint trajectory and resource allocation problem into upper-level UAV trajectory planning and lower-level local resource allocation. We define each recurring operational cycle—where the UAV flies horizontally to a designated grid cell and subsequently adjusts to an optimal altitude to execute local tasks—as a “single step”. The execution workflow and mechanism of the scheme are detailed as follows:

(1)

Upper-Level UAV Trajectory Planning

At the upper level, the UAV is responsible for horizontal traversing flights among multiple hexagonal target grid cells. To find the optimal visiting sequence that maximizes the global system energy efficiency, we model this routing process as an MDP and employ a DRL algorithm for autonomous solving. Prior to the execution of each single step, the agent evaluates the UAV’s residual energy to ensure that the next planned action satisfies the safe-return energy threshold ($C_{\mathrm{th}}$). If the energy falls below this warning line, the UAV must return to the HAP for recharging first before resuming the uncompleted tasks.

(2)

Lower-Level Local Resource Allocation

When the UAV horizontally arrives above the center of a target grid cell, the lower-level local optimization mechanism is triggered for the current step. In this phase, the UAV executes a harvest-then-transmit workflow:

• Wireless Energy Transfer: The UAV first broadcasts RF energy to all ground nodes in the downlink region.
• Wireless Information Transfer: Following the WET phase, the sensor nodes adopt an HMA scheme for channel access to execute WIT. For emergency nodes carrying critical information (e.g., anomaly monitoring), the system allocates an exclusive orthogonal time slot (TDMA) to prioritize the QoS of their data transmission. For the remaining regular nodes, the system utilizes NOMA technology to concurrently receive data on the same time-frequency block, significantly enhancing the local throughput.
• Local Joint Optimization: During the aforementioned interactions, the lower-level algorithm incorporating the probabilistic LoS channel model optimizes the UAV’s hovering altitude ($H_m$) and time allocation strategy in the current grid cell. This maximizes the local EE while satisfying the QoS requirements of emergency nodes.

Through the tight collaboration between the upper and lower layers, the DRL-HEEC scheme prioritizes emergency data transmission while optimizing underlying resource allocation. DRL-HEEC drives the intelligent evolution of the UAV’s global trajectory from the bottom up, ultimately realizing energy efficiency maximization throughout the entire lifecycle of the data collection mission.

3.3. Channel Model

Within the considered WPCN, the communication links between the UAV and the ground nodes are susceptible to blockages caused by environmental obstacles such as foliage or terrain variations. Consequently, these links comprise a mixture of LoS and non-line-of-sight (NLoS) propagation paths. To precisely capture this heterogeneous propagation environment, we adopt the probabilistic air-to-ground (A2G) channel model presented in [31,32], which explicitly integrates the occurrence probabilities of both LoS and NLoS links.

Based on the discrete hovering trajectory established above, the UAV communicates with the nodes exclusively while hovering at the grid cell centers. When the UAV hovers at the center of the m-th grid cell with an altitude of $H_m$, its 3D coordinates are denoted by $q_m=(x_m,y_m,H_m)$. Consequently, the Euclidean distance $d_{i,m}$ between the UAV and the i-th sensor node $S_{i,m}$ ($i\in \{1,2,\dots ,N_m\}$) located in grid cell m is expressed as

$$d_{i,m}=\parallel q_m-s_{i,m}\parallel =\sqrt{{(x_m-x_{i,m})}^2+{(y_m-y_{i,m})}^2+H_m^2}.$$

(4)

The LoS probability is determined by the elevation angle (in degrees), which is defined as the angle formed between the horizontal plane and the line connecting the UAV to the ground node. The elevation angle ${\theta }_{i,m}$ with respect to the node $S_{i,m}$ can be calculated as

$${\theta }_{i,m}={\displaystyle \frac{180}{\pi }}\arcsin \left({\displaystyle \frac{H_m}{d_{i,m}}}\right).$$

(5)

As in [31,33], the probability of the existence of an LoS link, denoted as $P_{\mathrm{LoS},i,m}$, is given by

$$P_{\mathrm{LoS},i,m}={\displaystyle \frac{1}{1+a\exp (-b[{\theta }_{i,m}-a])}},$$

(6)

where a and b are environment-dependent constants [34] (e.g., for urban or suburban areas).

Thus, the average channel power gain $h_{i,m}$ between the UAV and the node $S_{i,m}$ is formulated as the expectation over both line-of-sight and non-line-of-sight links:

$$h_{i,m}=P_{\mathrm{LoS},i,m}{h}_{\mathrm{LoS},i,m}+(1-P_{\mathrm{LoS},i,m})h_{\mathrm{NLoS},i,m}$$

(7)

where $h_{\mathrm{LoS},i,m}$ denotes the channel power gain of the LoS link, which follows the free-space path loss model given by

$$h_{\mathrm{LoS},i,m}={\beta }_0{d}_{i,m}^{-\delta }.$$

(8)

Here, ${\beta }_0$ represents the channel power gain at a reference distance of 1 m and $\delta$ denotes the path loss exponent. Furthermore, the channel power gain of the NLoS link, denoted by $h_{\mathrm{NLoS},i,m}$, is given by

$$h_{\mathrm{NLoS},i,m}=\kappa h_{\mathrm{LoS},i,m},$$

(9)

where $\kappa \in (0,1)$ is the additional attenuation factor specific to the NLoS link.

3.4. Energy Model

In the considered UAV-assisted WPCN, the sustainability of the entire system is governed by the energy dynamics of both the aerial and ground terminals. We systematically analyze the energy model from two perspectives, the UAV and the ground nodes. For the UAV, its operational lifecycle is fundamentally bounded by the physical capacity of its onboard battery. Its total energy consumption consists of two main components: the mechanical propulsion energy required for spatial flying and hovering, and the communication energy consumed for downlink RF wireless charging. Conversely, for the ground nodes, their energy lifecycles operate strictly on a harvest-then-transmit protocol; they harvest energy from the UAV’s RF broadcasting and consume it for uplink data transmission. To provide a fine-grained analysis, we detail the specific energy consumption and harvesting models for each operational step as follows.

3.4.1. UAV Energy Consumption Model

Since the flight altitude of the UAV is dynamically adjustable, the propulsion energy is driven by horizontal flying, vertical ascent/descent, and hovering. Suppose that the UAV flies horizontally at a constant speed V, requiring a horizontal flying power $P_{\mathrm{fly}}$. According to the rotary-wing energy theory in [35], $P_{\mathrm{fly}}$ is precisely modeled as

$$P_{\mathrm{fly}}=P_0\left(1+{\displaystyle \frac{3V^2}{U_{\mathrm{tip}}^2}}\right)+P_i{\left(\sqrt{1+{\displaystyle \frac{V^4}{4v_0^4}}}-{\displaystyle \frac{V^2}{2v_0^2}}\right)}^{1/2}+{\displaystyle \frac{1}{2}}{d}_0\rho sA{V}^3,$$

(10)

where $P_0$ and $P_i$ denote the blade profile power and induced power in hover, respectively, $U_{\mathrm{tip}}$ is the tip speed of the rotor blade, $v_0$ is the mean rotor induced velocity, $d_0$ and $\rho$ denote the fuselage drag ratio and air density, and s and A represent the rotor solidity and rotor disk area. When the flight speed $V=0$, this formula reduces to the hovering power, denoted as $P_{\mathrm{hov}}$, which is given by

$$P_{\mathrm{hov}}=P_0+P_i.$$

(11)

Concurrently, the UAV ascends at a constant velocity $V_{\mathrm{asc}}$ and descends at $V_{\mathrm{des}}$. Considering the weight of the UAV and motor characteristics, the corresponding ascent power $P_{\mathrm{asc}}$ and descent power $P_{\mathrm{des}}$ are defined in relation to the aforementioned hovering power $P_{\mathrm{hov}}$ as

$$P_{\mathrm{des}}={\eta }_{\mathrm{des}}{P}_{\mathrm{hov}},$$

(12)

$$P_{\mathrm{asc}}={\eta }_{\mathrm{asc}}{P}_{\mathrm{hov}},$$

(13)

where ${\eta }_{\mathrm{des}}$ and ${\eta }_{\mathrm{asc}}$ represent the power scaling coefficients for descending and ascending, respectively, which are determined by the UAV’s weight and motor efficiency.

3.4.2. Single-Step Energy Consumption

During the mission execution cycle, the UAV executes a sequence of operational steps. A single step k involves moving from the previous way-point $v_{k-1}$ to the target grid cell $v_k$, adjusting the altitude to the optimal hovering height, and performing local tasks. Here, $v_k\in \mathcal{M}$ explicitly denotes the specific grid cell m visited at step k. Based on this discrete trajectory, the 2D horizontal flying distance $d\left[k\right]$ and the vertical altitude difference $\mathrm{\Delta }H\left[k\right]$ at step k are defined as follows:

$$d\left[k\right]=\sqrt{{(x_{v_k}-x_{v_{k-1}})}^2+{(y_{v_k}-y_{v_{k-1}})}^2},$$

(14)

$$\mathrm{\Delta }H\left[k\right]=H_{v_k}-H_{v_{k-1}}.$$

(15)

The total energy consumption of the UAV for executing the single step k, denoted as $E\left[k\right]$, consists of three main components: the movement energy $E_{\mathrm{fly}}\left[k\right]$, the hovering propulsion energy $E_{\mathrm{hov}}\left[k\right]$, and the RF transmission energy $E_{\mathrm{tx}}\left[k\right]$, that is,

$$E\left[k\right]=E_{\mathrm{fly}}\left[k\right]+E_{\mathrm{hov}}\left[k\right]+E_{\mathrm{tx}}\left[k\right].$$

(16)

Specifically, $E_{\mathrm{fly}}\left[k\right]$ accounts for both the horizontal flight at a constant speed V and the vertical altitude adjustment, which is formulated as

$$E_{\mathrm{fly}}\left[k\right]=P_{\mathrm{fly}}{\displaystyle \frac{d\left[k\right]}{V}}+{\displaystyle \frac{P_{\mathrm{asc}}}{V_{\mathrm{asc}}}}\max (0,\mathrm{\Delta }H\left[k\right])+{\displaystyle \frac{P_{\mathrm{des}}}{V_{\mathrm{des}}}}\max (0,-\mathrm{\Delta }H\left[k\right]).$$

(17)

Upon reaching the hovering position at grid cell $v_k$, the UAV initiates the WET phase by broadcasting RF energy with a constant transmission power $P_{\mathrm{tx}}$ for a duration of ${\tau }_{v_k}^E$, then enters the WIT phase to collect data for a duration of ${\tau }_{v_k}^D$. Consequently, the hovering propulsion energy $E_{\mathrm{hov}}\left[k\right]$ and the RF charging energy $E_{\mathrm{tx}}\left[k\right]$ are respectively given by

$$E_{\mathrm{hov}}\left[k\right]=P_{\mathrm{hov}}{\tau }_{v_k},$$

(18)

$$E_{\mathrm{tx}}\left[k\right]=P_{\mathrm{tx}}{\tau }_{v_k}^E,$$

(19)

where ${\tau }_{v_k}$ denotes the hovering duration in grid cell $v_k$, satisfying ${\tau }_{v_k}={\tau }_{v_k}^{\mathrm{E}}+{\tau }_{v_k}^{\mathrm{D}}$.

Note that if step k is the required return step to the HAP, then the UAV only performs physical movement without executing wireless power or information transfer tasks.

3.4.3. Ground Node Energy Harvesting Model

In the downlink WET phase, the UAV acts as a wireless energy transmitter broadcasting RF energy to the ground nodes. Specifically, when the UAV hovers at the center of the target grid cell m at step k (i.e., $m=v_k$), it transmits with a constant power $P_{\mathrm{tx}}$ for a dedicated energy broadcasting duration, denoted by ${\tau }_{v_k}^E$.

Accordingly, the RF power received by the i-th sensor node $S_{i,m}$ ($i\in \{1,2,\dots ,N_m\}$) located in this target grid cell m can be expressed as

$$P_{\mathrm{e},i,m}=P_{\mathrm{tx}}{h}_{i,m},$$

(20)

where $h_{i,m}$ is the channel power gain derived previously.

To quantify the wireless energy harvesting process at the ground nodes, the widely adopted linear EH model is employed in this section [36]. Under this assumption, the harvested energy is strictly proportional to both the received RF power and the charging duration. Therefore, the total energy harvested by node $S_{i,m}$ during this WET phase, denoted as $E_{i,m}^{\mathrm{EH}}$, is mathematically modeled as

$$E_{i,m}^{\mathrm{EH}}=\xi P_{\mathrm{e},i,m}{\tau }_{v_k}^E,$$

(21)

where $\xi \in (0,1)$ denotes the wireless energy conversion efficiency of the ground sensor node.

3.5. Hybrid Multiple Access Data Transmission Model

During the WIT phase while the UAV is hovering, ground nodes suffer from a “double penalty” caused by the near–far effect [37]: nodes located farther from the grid cell center harvest significantly less energy in the downlink WET phase, yet experience more severe channel attenuation during the uplink transmission. This creates a critical vulnerability: if such a distant edge node generates emergency data, its QoS is highly likely to be compromised; more specifically, if a pure NOMA protocol were employed, the extremely weak signal from this distant node would be deeply submerged in co-channel interference. Such a signal would be relegated to the bottom of the SIC decoding order, making it impossible to satisfy the strict minimum throughput threshold required for critical information.

To fundamentally address the aforementioned fairness bottleneck and QoS urgency issue, we propose a hybrid TDMA-NOMA protocol, termed the HMA strategy. Consistent with previous definitions, let m denote the target grid cell visited at step k (i.e., $m=v_k$) and let ${\tau }_m^D$ (equivalent to ${\tau }_{v_k}^D$) be the total WIT duration in this grid cell. Given that each ground node independently generates emergency data with a probability $P_e$, multiple priority nodes may coexist within the same grid cell. Therefore, we let ${\mathcal{P}}_m$ denote the set of high-priority emergency nodes in grid cell m, while the set of remaining regular nodes is denoted by ${\mathcal{N}}_m^{\prime }={\mathcal{N}}_m\setminus {\mathcal{P}}_m$.

As visually illustrated in Figure 4, the total uplink WIT time ${\tau }_m^D$ is partitioned into two consecutive sub-slots: a dedicated TDMA slot ${\tau }_m^{\mathrm{T}}$ for the emergency nodes, and a shared NOMA slot ${\tau }_m^{\mathrm{N}}$ for all other regular nodes, satisfying

$${\tau }_m^D={\tau }_m^{\mathrm{T}}+{\tau }_m^{\mathrm{N}}.$$

(22)

Specifically, for the sake of simplicity and fairness among critical data, the dedicated TDMA phase ${\tau }_m^{\mathrm{T}}$ is equally divided into individual orthogonal sub-slots for each emergency node $S_{i,m}\in {\mathcal{P}}_m$. Therefore, the allocated transmission duration for each emergency node is ${\displaystyle \frac{{\tau }_m^{\mathrm{T}}}{|{\mathcal{P}}_m|}}$, where $|{\mathcal{P}}_m|$ denotes the total number of emergency nodes in grid cell m. Note that if no emergency node exists in the current grid cell (i.e., $|{\mathcal{P}}_m|=0$), the TDMA phase is skipped entirely, yielding ${\tau }_m^{\mathrm{T}}=0$.

Operating under the harvest-then-transmit protocol, the ground nodes fully utilize their harvested energy to maximize the uplink throughput. Assuming $|{\mathcal{P}}_m|>0$, the constant uplink transmission power for a given node $S_{i,m}$ is calculated as follows:

$$P_{i,m}=\left\{\begin{array}{cc}{\displaystyle \frac{E_{i,m}^{\mathrm{EH}}\left|{\mathcal{P}}_m\right|}{{\tau }_m^{\mathrm{T}}}},\hfill & \mathrm{if}{S}_{i,m}\in {\mathcal{P}}_m,\hfill \\ {\displaystyle \frac{E_{i,m}^{\mathrm{EH}}}{{\tau }_m^{\mathrm{N}}}},\hfill & \mathrm{if}{S}_{i,m}\in {\mathcal{N}}_m^{\prime }.\hfill \end{array}\right.$$

(23)

3.5.1. Phase 1: Emergency Nodes TDMA Transmission

During the TDMA phase, each emergency node $S_{i,m}\in {\mathcal{P}}_m$ sequentially performs exclusive orthogonal transmission within its equally allocated sub-slot. The signal-to-noise ratio (SNR) ${\gamma }_{i,m}^{\mathrm{E}}$ at the UAV receiver and the corresponding data throughput $R_{i,m}^{\mathrm{E}}$ uploaded by emergency node $S_{i,m}$ are given by

$${\gamma }_{i,m}^{\mathrm{E}}={\displaystyle \frac{P_{i,m}{h}_{i,m}}{{\sigma }^2}},\forall S_{i,m}\in {\mathcal{P}}_m,$$

(24)

$$R_{i,m}^{\mathrm{E}}={\displaystyle \frac{{\tau }_m^{\mathrm{T}}}{|{\mathcal{P}}_m|}}B{\log }_2(1+{\gamma }_{i,m}^{\mathrm{E}}),\forall S_{i,m}\in {\mathcal{P}}_m,$$

(25)

where B is the system bandwidth and ${\sigma }^2$ represents the additive white Gaussian noise (AWGN) power.

3.5.2. Phase 2: Regular Nodes NOMA Transmission

During the subsequent shared phase ${\tau }_m^{\mathrm{N}}$, all regular nodes $S_{i,m}\in {\mathcal{N}}_m^{\prime }$ transmit their data simultaneously over the same frequency band using the NOMA protocol. To decouple the superimposed signals, the UAV receiver employs the SIC technique. To maximize the NOMA uplink sum-throughput, the UAV decodes the signals in descending order of their channel gains. Specifically, suppose thaat the regular nodes in ${\mathcal{N}}_m^{\prime }$ are sorted such that their channel gains satisfy $h_{1,m}\ge h_{2,m}\ge \cdots \ge h_{|{\mathcal{N}}_m^{\prime }|,m}$. According to the SIC principle, when decoding the signal of the i-th regular node $S_{i,m}$, interference originates solely from the remaining nodes that have not yet been decoded (i.e., those with comparatively weaker channel conditions). Thus, the signal-to-interference-plus-noise ratio (SINR) ${\gamma }_{i,m}^{\mathrm{N}}$ and the achievable data throughput $R_{i,m}^{\mathrm{N}}$ for regular node $S_{i,m}$ are formulated as follows:

$${\gamma }_{i,m}^{\mathrm{N}}={\displaystyle \frac{P_{i,m}{h}_{i,m}}{{\sum }_{l=i+1}^{|{\mathcal{N}}_m^{\prime }|}{P}_{l,m}{h}_{l,m}+{\sigma }^2}},\forall S_{i,m}\in {\mathcal{N}}_m^{\prime },$$

(26)

$$R_{i,m}^{\mathrm{N}}={\tau }_m^{\mathrm{N}}B{\log }_2(1+{\gamma }_{i,m}^{\mathrm{N}}),\forall S_{i,m}\in {\mathcal{N}}_m^{\prime }.$$

(27)

4. Single-Step Energy Efficiency Optimization

As outlined in the system overview, directly solving the deeply coupled global multi-tour optimization problem is computationally intractable. To address this, we adopt a two-tier decoupling architecture. This section focuses strictly on the local optimization executed at each hovering phase. Specifically, we first formulate the single-step energy efficiency optimization problem, then develop an efficient algorithm to obtain its optimal solution.

4.1. Problem Formulation

Suppose that at a given step k, the UAV has been dispatched from the previous waypoint $v_{k-1}$ to a target grid cell m (i.e., $v_k=m$) by the upper-level algorithm. The core objective of this lower tier is to jointly optimize the UAV’s continuous hovering altitude $H_m$ and the time allocation strategy ${\mathit{\tau }}_m=\{{\tau }_m^E,{\tau }_m^{\mathrm{T}},{\tau }_m^{\mathrm{N}}\}$ within this grid cell. This local problem aims to maximize the single-step EE ${\mathrm{\Phi }}_k$, taking into account not only the local operations but also the physical flying energy expended to reach the grid cell. Based on our previous definitions, the single-step optimization problem (P1) is formulated as follows:

$$\begin{array}{cc}\left(\mathrm{P}1\right):\underset{H_m,{\mathit{\tau }}_m}{\max }\hfill & {\mathrm{\Phi }}_k={\displaystyle \frac{{\sum }_{S_{i,m}\in {\mathcal{P}}_m}{R}_{i,m}^{\mathrm{E}}+{\sum }_{S_{i,m}\in {\mathcal{N}}_m^{\prime }}{R}_{i,m}^{\mathrm{N}}}{E\left[k\right]}}\hfill \end{array}$$

(28a)

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.\hfill & \mathrm{C}1:R_{i,m}^{\mathrm{E}}\ge R_{\mathrm{th}},\forall S_{i,m}\in {\mathcal{P}}_m,\hfill \end{array}$$

(28b)

$$\begin{array}{cc}\hfill & \mathrm{C}2:H_{\min }\le H_m\le H_{\max },\hfill \end{array}$$

(28c)

$$\begin{array}{cc}\hfill & \mathrm{C}3:{\tau }_m^{\mathrm{E}},{\tau }_m^{\mathrm{T}},{\tau }_m^{\mathrm{N}}\ge 0,\hfill \end{array}$$

(28d)

$$\begin{array}{cc}\hfill & \mathrm{C}4:P_{i,m}\le P_{\max },\forall S_{i,m}\in {\mathcal{N}}_m.\hfill \end{array}$$

(28e)

In this model, the objective function (28a) evaluates the ratio of the total collected data (the sum of throughput from both emergency nodes and regular nodes) to the total single-step energy consumption $E\left[k\right]$. Constraint C1 provides a QoS guarantee threshold $R_{\mathrm{th}}$ for all emergency nodes. Constraints C2 and C3 define the safe flight altitude range and the non-negativity of time variables, respectively. Constraint C4 represents the node’s transmit power constraint, while $p_{\max }$ denotes the maximum transmission power of the node. Note that the global UAV battery capacity limit is not included as a hard constraint in this local optimization; instead, it is inherently managed by the upper-level DRL, where the UAV will be forced to return to the HAP if its residual energy falls below a critical safety threshold (i.e., $C_{th}$).

4.2. Dinkelbach-Based Fractional Transformation

For notational brevity in the subsequent derivations, we first define the total collected data throughput in grid cell m as $R_m^{\mathrm{sum}}\triangleq {\sum }_{S_{i,m}\in {\mathcal{P}}_m}{R}_{i,m}^{\mathrm{E}}+{\sum }_{S_{i,m}\in {\mathcal{N}}_m^{\prime }}{R}_{i,m}^{\mathrm{N}}$. The objective of (P1) is a highly non-convex fractional function. Introducing an auxiliary parameter $q\ge 0$, based on Dinkelbach’s theorem, the original problem is transformed into an equivalent subtractive form, denoted as (P2):

$$\begin{array}{cc}\left(\mathrm{P}2\right):\underset{H_m,{\mathit{\tau }}_m}{\max }\hfill & \mathcal{F}\left(q\right)=R_m^{\mathrm{sum}}(H_m,{\mathit{\tau }}_m)-qE\left[k\right](H_m,{\mathit{\tau }}_m)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & \mathrm{C}1,\mathrm{C}2,\mathrm{C}3,\mathrm{C}4.\hfill \end{array}$$

(29)

The optimal parameter $q^{*}$ represents the global optimal energy efficiency if and only if $\mathcal{F}\left(q^{*}\right)=0$. To find $q^{*}$, an iterative procedure is employed. At the n-th outer iteration with a given $q^{\left(n\right)}$, we solve (P2) to obtain the optimal variables $(H_m^{\left(n\right)},{\mathit{\tau }}_m^{\left(n\right)})$. Then, the parameter is updated as $q^{(n+1)}={\displaystyle \frac{R_m^{\mathrm{sum}}(H_m^{\left(n\right)},{\mathit{\tau }}_m^{\left(n\right)})}{E\left[k\right](H_m^{\left(n\right)},{\mathit{\tau }}_m^{\left(n\right)})}}$ until convergence.

4.3. BCD-SCA Decoupled Solution Algorithm

Despite eliminating the fractional structure, the altitude $H_m$ and time allocation ${\mathit{\tau }}_m$ remain deeply coupled in (P2). To guarantee strict convergence, we adopt the block coordinate descent (BCD) framework in the inner loop to decouple the problem into two alternating subproblems.

4.3.1. Optimizing ${\mathit{\tau }}_m$ with Fixed $H_m$

With altitude fixed, the channel gains become constants. Let ${\mathrm{\Lambda }}_l$ denote the compound energy harvesting coefficient. The emergency node throughput $R_{i,m}^{\mathrm{E}}$ inherently exhibits a perspective function form, rendering it jointly concave with respect to time variables. However, the NOMA throughput $R_{i,m}^{\mathrm{N}}$ contains non-convex co-channel interference. We leverage the perspective function to reconstruct it into a strict difference of concave (DC) form:

$$R_{i,m}^{\mathrm{N}}=\underset{{\widehat{f}}_{i,m}\left({\mathit{\tau }}_m\right)}{\underbrace{{\tau }_m^{\mathrm{N}}B{\log }_2\left(1+{\displaystyle \frac{{\sum }_{l=i}^{|{\mathcal{N}}_m^{\prime }|}{\mathrm{\Lambda }}_l{\tau }_m^{\mathrm{E}}}{{\sigma }^2{\tau }_m^{\mathrm{N}}}}\right)}}-\underset{{\widehat{g}}_{i,m}\left({\mathit{\tau }}_m\right)}{\underbrace{{\tau }_m^{\mathrm{N}}B{\log }_2\left(1+{\displaystyle \frac{{\sum }_{l=i+1}^{|{\mathcal{N}}_m^{\prime }|}{\mathrm{\Lambda }}_l{\tau }_m^{\mathrm{E}}}{{\sigma }^2{\tau }_m^{\mathrm{N}}}}\right)}}.$$

(30)

At the r-th SCA iteration point ${\mathit{\tau }}_m^{\left(r\right)}$, first-order Taylor expansion is applied to the strictly concave function ${\widehat{g}}_{i,m}$ in order to obtain its global affine upper bound:

$${\tilde{g}}_{i,m}^{\mathrm{ub}}\left({\mathit{\tau }}_m\right)={\widehat{g}}_{i,m}\left({\mathit{\tau }}_m^{\left(r\right)}\right)+{\nabla }_{{\tau }_m^{\mathrm{E}}}{\widehat{g}}_{i,m}^{\left(r\right)}({\tau }_m^{\mathrm{E}}-{\tau }_m^{\mathrm{E}\left(r\right)})+{\nabla }_{{\tau }_m^{\mathrm{N}}}{\widehat{g}}_{i,m}^{\left(r\right)}({\tau }_m^{\mathrm{N}}-{\tau }_m^{\mathrm{N}\left(r\right)}).$$

(31)

The precise partial derivatives are derived as follows:

$${\nabla }_{{\tau }_m^{\mathrm{E}}}{\widehat{g}}_{i,m}^{\left(r\right)}={\displaystyle \frac{B}{\ln 2}}{\displaystyle \frac{I_i}{{\sigma }^2+I_i\frac{{\tau }_m^{\mathrm{E}\left(r\right)}}{{\tau }_m^{\mathrm{N}\left(r\right)}}}},$$

(32)

$${\nabla }_{{\tau }_m^{\mathrm{N}}}{\widehat{g}}_{i,m}^{\left(r\right)}=B{\log }_2\left(1+{\displaystyle \frac{I_i{\tau }_m^{\mathrm{E}\left(r\right)}}{{\sigma }^2{\tau }_m^{\mathrm{N}\left(r\right)}}}\right)-{\displaystyle \frac{B}{\ln 2}}{\displaystyle \frac{I_i\frac{{\tau }_m^{\mathrm{E}\left(r\right)}}{{\tau }_m^{\mathrm{N}\left(r\right)}}}{{\sigma }^2+I_i\frac{{\tau }_m^{\mathrm{E}\left(r\right)}}{{\tau }_m^{\mathrm{N}\left(r\right)}}}},$$

(33)

where $I_i={\sum }_{l=i+1}^{|{\mathcal{N}}_m^{\prime }|}{\mathrm{\Lambda }}_l$. By replacing ${\widehat{g}}_{i,m}$ with ${\tilde{g}}_{i,m}^{\mathrm{ub}}$, the NOMA throughput is lower-bounded by a concave function ${\tilde{R}}_{i,m}^{\mathrm{lb}}={\widehat{f}}_{i,m}-{\tilde{g}}_{i,m}^{\mathrm{ub}}$. Thus, Subproblem 1 at the r-th iteration is approximated as the following standard convex problem:

$$\begin{array}{cc}\left(\mathrm{P}2.1\right):\underset{{\mathit{\tau }}_m}{\max }\hfill & \sum _{S_{i,m}\in {\mathcal{P}}_m}{R}_{i,m}^{\mathrm{E}}\left({\mathit{\tau }}_m\right)+\sum _{S_{i,m}\in {\mathcal{N}}_m^{\prime }}{\tilde{R}}_{i,m}^{\mathrm{lb}}\left({\mathit{\tau }}_m\right)-qE\left[k\right]\left({\mathit{\tau }}_m\right)\hfill \end{array}$$

(34a)

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.\hfill & \mathrm{C}1,\mathrm{C}3,\mathrm{C}4.\hfill \end{array}$$

(34b)

4.3.2. Optimizing $H_m$ with Fixed ${\mathit{\tau }}_m$

With time fixed, the energy consumption $E\left[k\right]$ is convex with respect to altitude due to the convexity-preserving max operations in the UAV propulsion model. The throughput $R\left(H_m\right)$ is rigorously proven to be convex with respect to $H_m$ within the physical altitude range ($H_m\ge H_{\min }$). Utilizing SCA, the global concave lower bound for the throughput of any node $S_{i,m}$ (including emergency nodes in C1) is derived via Taylor expansion at the local point $H_m^{\left(r\right)}$:

$${\tilde{R}}_{i,m}^{\mathrm{lb}}\left(H_m\right)=R_{i,m}\left(H_m^{\left(r\right)}\right)+{\nabla }_{H_m}{R}_{i,m}\left(H_m^{\left(r\right)}\right)·(H_m-H_m^{\left(r\right)}).$$

(35)

For any regular node $S_{i,m}\in {\mathcal{N}}_m^{\prime }$, we define the conditionally fixed power factor as $S_i\left(H_m\right)={\sum }_{l=i}^{|{\mathcal{N}}_m^{\prime }|}{\displaystyle \frac{C_l^{\left(r\right)}}{H_m^2+r_{l,m}^2}}$, where the composite numerator $C_l^{\left(r\right)}$ perfectly absorbs the fixed time variables, transmit power, and fixed LoS probability $P_{\mathrm{LoS}}\left(H_m^{\left(r\right)}\right)$ from the current iteration point. Under this decoupling strategy, the effective partial derivative of the regular node throughput is chained as

$${\nabla }_{H_m}{R}_{i,m}^{\mathrm{N}}\left(H_m^{\left(r\right)}\right)={\displaystyle \frac{{\tau }_m^{\mathrm{N}}B}{\ln 2}}\left[{\displaystyle \frac{S_i^{\prime }\left(H_m^{\left(r\right)}\right)}{{\sigma }^2{\tau }_m^{\mathrm{N}}+S_i\left(H_m^{\left(r\right)}\right)}}-{\displaystyle \frac{S_{i+1}^{\prime }\left(H_m^{\left(r\right)}\right)}{{\sigma }^2{\tau }_m^{\mathrm{N}}+S_{i+1}\left(H_m^{\left(r\right)}\right)}}\right],$$

(36)

where $S_i^{\prime }\left(H_m\right)=\sum {\displaystyle \frac{-2C_l^{\left(r\right)}{H}_m}{{(H_m^2+r_{l,m}^2)}^2}}$. By substituting the non-convex throughput expressions with their concave lower bounds, Subproblem 2 at the r-th iteration is transformed into

$$\begin{array}{cc}\left(\mathrm{P}2.2\right):\underset{H_m}{\max }\hfill & \sum _{S_{i,m}\in {\mathcal{P}}_m}{\tilde{R}}_{i,m}^{\mathrm{E},\mathrm{lb}}\left(H_m\right)+\sum _{S_{i,m}\in {\mathcal{N}}_m^{\prime }}{\tilde{R}}_{i,m}^{\mathrm{N},\mathrm{lb}}\left(H_m\right)-qE\left[k\right]\left(H_m\right)\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.\hfill & {\tilde{R}}_{i,m}^{\mathrm{E},\mathrm{lb}}\left(H_m\right)\ge R_{\mathrm{th}},\forall S_{i,m}\in {\mathcal{P}}_m,\hfill \end{array}$$

(37b)

$$\begin{array}{c}\mathrm{C}2,\mathrm{C}4.\hfill \end{array}$$

(37c)

This approximated subproblem is strictly convex and can be efficiently solved by standard convex optimization solvers such as the MATLAB R2021b-based CVX toolbox.

Through alternating iterations between (P2.1) and (P2.2), the inner BCD-SCA algorithm strictly converges to a locally optimal strategy $(H_m^{*},{\mathit{\tau }}_m^{*})$ for a given q. Consequently, the outer Dinkelbach loop updates q until convergence, successfully yielding the maximum single-step energy efficiency ${\mathrm{\Phi }}_k^{*}$. This optimal analytical value seamlessly serves as the “immediate reward” for the UAV’s global trajectory action, providing mathematically exact feedback for the upper-level RL environment detailed in the following section.

5. Upper-Level Global Trajectory Design via DDQN

While the lower-level local optimization maximizes the single-step energy efficiency, the global trajectory heavily depends on the sequential visitation order. Since the UAV’s current flight decision strictly affects its remaining energy and available future paths, this sequential decision-making process is formulated as an MDP and solved using the DDQN algorithm.

5.1. MDP Modeling

The MDP is defined by a tuple $\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma \rangle$, capturing the dynamic interaction between the UAV agent and the communication environment.

(1)

State Space $\mathcal{S}$: At each step k, the state vector encompass the UAV’s physical status and the global mission progress, defined as $s_k=\left[l_k,C_{\mathrm{res}}\left[k\right],{\mathbf{U}}_k\right]$. Here, $l_k=(v_k,h_k)$ is a tuple denoting the UAV’s spatial position at step k, where $v_k\in \mathcal{M}$ represents the location index ($v_k=1$ corresponds to the HAP, while the remaining indices indicate the target service grid cells) and $h_k$ indicates the hovering altitude at step k. Furthermore, $C_{\mathrm{res}}\left[k\right]$ is the residual energy at step k and ${\mathbf{U}}_k=[u_{1,k},\dots ,u_{M,k}]$ is a binary mask indicating the unserved status of target grid cells at step k (where $u_{m,k}=1$ means unvisited).

(2)

Action Space $\mathcal{A}$: To simulate realistic flight kinematics while providing configurable trajectory flexibility, the UAV’s mobility is governed by a maximum allowed flight distance per step. Specifically, when hovering above a target grid cell, the UAV can transit to any grid cell within an n-tier hexagonal neighborhood (i.e., up to n hops away, where $n\in \{1,2,3,\dots \}$) or directly return to the HAP (i.e., $v_k=1$). However, once the UAV is at the HAP (e.g., after recharging), it can be dispatched to any grid cell to resume data collection. Thus, the dynamically available action space $\mathcal{A}\left(s_k\right)$ is formulated as

$$\mathcal{A}\left(s_k\right)=\left\{\begin{array}{cc}{\mathcal{N}}_{\mathrm{hex}}^{\left(n\right)}\left(v_k\right)\cup \left\{1\right\},\hfill & \mathrm{if}{v}_k\ne 1,\hfill \\ \mathcal{M},\hfill & \mathrm{if}{v}_k=1,\hfill \end{array}\right.$$

(38)

where ${\mathcal{N}}_{\mathrm{hex}}^{\left(n\right)}\left(v_k\right)$ denotes the set of adjacent neighboring grid cells up to n geometric layers from grid cell $v_k$ and $\mathcal{M}$ encompasses all location indices.

(3)

Reward Function $\mathcal{R}$: The immediate reward $r_k$ is the critical bridge coupling the upper and lower tiers. It dynamically evaluates the selected action $a_k$:

$$r_k=\left\{\begin{array}{cc}\beta {\mathrm{\Phi }}_k^{*},\hfill & \mathrm{if}{a}_k\in \mathcal{M}\mathrm{and}{u}_{a_k,k}=1,\hfill \\ -\omega ,\hfill & \mathrm{if}{a}_k\in \mathcal{M}\mathrm{and}{u}_{a_k,k}=0,\hfill \\ R_{\mathrm{c}},\hfill & \mathrm{if}\mathrm{all}\mathrm{target}\mathrm{grid}\mathrm{cells}\mathrm{are}\mathrm{served},\hfill \\ -\psi ,\hfill & \mathrm{if}\mathrm{energy}\mathrm{depletion}\mathrm{occurs}.\hfill \end{array}\right.$$

(39)

Specifically, visiting an unserved grid cell yields the scaled optimal local energy efficiency $\beta {\mathrm{\Phi }}_k^{*}$, where $\beta$ acts as a normalization factor to balance the reward magnitude. Conversely, traversing an already-visited grid cell or prematurely returning to the HAP yields a mild penalty $-\omega$. This minor penalty fundamentally discourages redundant flights that consume energy without throughput gains, yet is kept intentionally small because traversing a visited grid cell is sometimes geometrically inevitable for transiting to distant unserved areas. $R_{\mathrm{c}}$ is a positive completion bonus awarded when all target areas are successfully visited, and $\psi$ is a massive penalty for crashing due to energy exhaustion.

5.2. DDQN Algorithm

Given the high-dimensional hybrid state space and the dynamically masked discrete action space, we employ DDQN, an advanced value-based reinforcement learning framework that effectively mitigates the overestimation bias inherent in standard DQN.

The framework maintains two neural networks: an online Q-network $Q_{\theta }(s,a)$ with weights $\theta$ to select actions, and a target Q-network $Q_{{\theta }^{-}}(s,a)$ with weights ${\theta }^{-}$ to evaluate the selected actions. To strictly prevent the UAV from selecting physically unreachable grid cells or executing actions that lead to inevitable energy depletion, an action mask is applied to the network’s output layer. This mask forces the Q-values of invalid actions to $-\infty$, ensuring they are never selected during the $ϵ$-greedy exploration and exploitation phases.

During training, whenever the UAV executes a valid action, the environment pauses to run the lower-level solver, computes ${\mathrm{\Phi }}_k^{*}$, and feeds it back. The transition tuple $(s_k,a_k,r_k,s_{k+1})$ is then stored in an experience replay buffer $\mathcal{D}$. The online network is updated by sampling mini-batches from $\mathcal{D}$ and minimizing the mean squared error (MSE) loss:

$$L\left(\theta \right)={\mathbb{E}}_{(s_k,a_k,r_k,s_{k+1})\sim \mathcal{D}}\left[{\left(Y_k-Q_{\theta }(s_k,a_k)\right)}^2\right]$$

(40)

where $Y_k$ is the DDQN target, defined as

$$Y_k=r_k+\gamma Q_{{\theta }^{-}}\left(s_{k+1},\arg \underset{a^{\prime }}{\max }{Q}_{\theta }(s_{k+1},a^{\prime })\right),$$

(41)

with $\gamma \in [0,1]$ denoting the discount factor. By decoupling action selection (via $\theta$) from target evaluation (via ${\theta }^{-}$), DDQN prevents the propagation of overly optimistic value estimates. Through iterative interactions, decoupled dual-tier updates, and periodic synchronization of ${\theta }^{-}$ with $\theta$, the algorithm converges to a globally optimal energy-efficient trajectory and resource management policy.

Algorithm 1 outlines the proposed joint upper-lower tier optimization framework, where a DDQN agent manages global trajectory planning and action masking while an inner Dinkelbach-BCD-SCA method optimizes local altitude and resource allocation whenever an unserved target grid cell is visited. Specifically, starting from the HAP with a full battery $C_{\mathrm{uav}}$, the UAV iteratively selects valid actions $a_k\in \mathcal{A}\left(s_k\right)$ that account for physical energy consumption $E\left[k\right]$ and recharging mechanics at $v_k=1$, receiving immediate rewards $r_k$ scaled by the optimal local energy efficiency $\beta {\mathrm{\Phi }}_k^{*}$ for new service visits, global completion bonuses $R_{\mathrm{c}}$, crash penalties $-\psi$, or mild penalties for grid cell revisiting $-\omega$, while experience replay and periodic target network updates with frequency $\mathcal{C}$ ensure stable neural network training and convergence.

Algorithm 1 Joint Upper-Lower Tier Optimization via DDQN

Require:  Set of all grid cells $\mathcal{M}$, Battery capacity $C_{\mathrm{uav}}$, Replay buffer $\mathcal{D}$, Target update frequency $\mathcal{C}$, Learning rate ${\alpha }_0$, Discount factor $\gamma$. Ensure:  Optimal global flight policy and local resource allocation $\{H_m^{*},{\mathit{\tau }}_m^{*}\}$.   1: Initialize Online Q-network $Q_{\theta }$ and Target Q-network $Q_{{\theta }^{-}}$ with identical weights ${\theta }^{-}=\theta$.   2: for each training episode do   3:       Reset environment: $v_0=1$ (HAP), $C_{\mathrm{res}}\left[0\right]=C_{\mathrm{uav}}$, ${\mathbf{U}}_0=\mathbf{1}$.   4:       for each decision step $k=0,1,2,\dots$ do   5:             Obtain state $s_k=[l_k,C_{\mathrm{res}}\left[k\right],{\mathbf{U}}_k]$ and valid action space $\mathcal{A}\left(s_k\right)$.   6:             Select action $a_k\in \mathcal{A}\left(s_k\right)$ using $ϵ$-greedy policy based on masked $Q_{\theta }(s_k,a)$.   7:             Execute $a_k$: update location $v_{k+1}=a_k$ and calculate energy consumption, yielding $C_{\mathrm{res}}[k+1]=C_{\mathrm{res}}\left[k\right]-E\left[k\right]$.   8:             if $v_k\ne 1$ and $a_k==1$ then   9:                  Recharge battery: $C_{\mathrm{res}}[k+1]=C_{\mathrm{uav}}$. 10:             end if 11:             if $a_k\in \mathcal{M}$ and $u_{a_k,k}==1$ then 12:                  Inner Optimization: Execute Dinkelbach-BCD-SCA algorithm: 13:                  while Dinkelbach iteration n not converged do 14:                        while BCD-SCA iteration r not converged do 15:                              Update ${\mathit{\tau }}_m$ by solving (P2.1) via CVX. 16:                              Update $H_m$ by solving (P2.2) via CVX. 17:                        end while 18:                        Update $q^{(n+1)}=R_m^{\mathrm{sum}}/E\left[k\right]$. 19:                  end while 20:                  Set local EE ${\mathrm{\Phi }}_k^{*}=q^{*}$, assign reward $r_k=\beta {\mathrm{\Phi }}_k^{*}$, update $u_{a_k,k+1}=0$. 21:             else if all target grid cells are served (${\mathbf{U}}_{k+1}==\mathbf{0}$) then 22:                  Assign completion bonus $r_k=R_{\mathrm{c}}$ and terminate episode. 23:             else if $C_{\mathrm{res}}[k+1]\le C_{\mathrm{th}}$ then 24:                  Assign crash penalty $r_k=-\psi$ and terminate episode. 25:             else 26:                  Assign step penalty $r_k=-\omega$. 27:             end if 28:             Transition to $s_{k+1}$, store experience $(s_k,a_k,r_k,s_{k+1})$ in $\mathcal{D}$. 29:             Sample a random mini-batch from $\mathcal{D}$. 30:             Calculate DDQN target: $Y_j=r_j+\gamma Q_{{\theta }^{-}}\left(s_{j+1},\arg {\max }_{a^{\prime }}{Q}_{\theta }(s_{j+1},a^{\prime })\right)$. 31:             Perform gradient descent on Loss: $L\left(\theta \right)={\displaystyle \frac{1}{N}}\sum {(Y_j-Q_{\theta }(s_j,a_j))}^2$. 32:             if step $k\left(\mathrm{mod}\mathcal{C}\right)==0$ then 33:                  Update target network weights: ${\theta }^{-}\leftarrow \theta$. 34:             end if 35:       end for 36: end for

5.3. Computational Complexity Analysis

The overall per-step computational complexity of Algorithm 1 during the training phase is expressed as $\mathcal{O}\left(N_1+M_1{N}_1+{\mathbb{I}}_{\mathrm{cell}}·I_{\mathrm{dink}}{I}_{\mathrm{sca}}{N}_2^3\right)$, where $N_1$ is the neural network computation cost and $M_1$ represents the mini-batch size for experience replay. The indicator ${\mathbb{I}}_{\mathrm{cell}}\in \{0,1\}$ denotes whether the UAV visits a target grid cell. When triggered, the lower-layer optimization requires $I_{\mathrm{dink}}$ Dinkelbach iterations and $I_{\mathrm{sca}}$ BCD-SCA iterations, with $N_2$ being the dimension of variables.

Furthermore, the proposed algorithm exhibits favorable computational efficiency [38] and practical viability. Although this nested structure introduces significant overhead during offline training, our decoupled framework allows the lower-layer resource allocation to be pre-computed offline as an efficient lookup table. Consequently, the online execution avoids heavy real-time convex optimization, relying solely on lightweight network forward passes to ensure high efficiency for practical UAV deployments.

6. Simulation Results and Analysis

To evaluate the effectiveness and superiority of the proposed dual-tier DRL-HEEC framework, extensive numerical simulations are conducted using MATLAB. The local convex optimization subproblems are solved utilizing the CVX 2.2 toolbox. All simulations are executed on a desktop computer configured with a 12th Gen Intel(R) Core(TM) i7-12700KF CPU operating at 3.60 GHz, 32 GB of memory, and an NVIDIA GeForce RTX 3070 Ti GPU.

6.1. Simulation Setup

We consider a multi-cell target region partitioned into several regular hexagonal grid cells, as shown in Figure 5. The side length of each hexagonal grid cell is set to $L=50$ m. Within each grid cell, N ground nodes are randomly distributed, with each node having a probability $P_e$ of being an emergency node. The UAV is dispatched from the HAP (grid cell 1) to execute data collection tasks at other nodes, with its initial flight altitude set to $H_0=15$ m. Following [35] and [19], the propulsion dynamics parameters of the UAV are specified as follows: the blade profile power in hover $P_0=79.86\mathrm{W}$, the induced power in hover $P_i=88.63\mathrm{W}$, the tip speed of rotor blade $U_{\mathrm{tip}}=120\mathrm{m}/\mathrm{s}$, the mean rotor induced velocity in hover $v_0=4.03\mathrm{m}/\mathrm{s}$, the fuselage drag ratio $d_0=0.6$, the air density $\rho =1.225{\mathrm{kg}/\mathrm{m}}^3$, the rotor solidity $s=0.05$, and the rotor disc area $A=0.503{\mathrm{m}}^2$. Furthermore, the horizontal flight speed $V=15\mathrm{m}/\mathrm{s}$, the vertical ascending and descending speeds are $V_{\mathrm{asc}}=V_{\mathrm{des}}=3\mathrm{m}/\mathrm{s}$, and the corresponding power correction factors are ${\eta }_{\mathrm{asc}}=1.1$ and ${\eta }_{\mathrm{des}}=0.9$, respectively. The remaining parameters are summarized in

Table 1. Simulation parameters.

Parameter	Value	Parameter	Value
Minimum altitude $H_{\min }$	5 m	Maximum altitude $H_{\max }$	50 m
LoS probability parameter a	$9.61$	LoS probability parameter b	$0.16$
Reference channel gain ${\beta }_0$	${10}^{-3}$	Path loss exponent $\delta$	$2.2$
NLoS attenuation factor $\kappa$	$0.2$	System bandwidth B	1 MHz
Noise power ${\sigma }^2$	${10}^{-13}$ W	UAV transmit power $P_{\mathrm{tx}}$	5 W
Energy conversion efficiency $\xi$	$0.6$	Minimum throughput $R_{\mathrm{th}}$	100 kbits
UAV battery capacity $C_{\mathrm{uav}}$	20 kJ	Safe return threshold $C_{\mathrm{th}}$	$2.5$ kJ
Node maximum transmit power $P_{max}$	1 W	Crash penalties $\psi$	100
Mission complete reward $R_c$	50	Revisit penalty $\omega$	1
Cell hover time $\tau$	5 s	Gradient decay factor	$0.9$
Normalization factor $\beta$	${10}^{-4}$	Target update frequency $\mathcal{C}$	200
Discount factor $\gamma$	$0.99$	Replay buffer capacity	50,000
Mini-batch size	128	Minimum replay buffer size	1000
Initial exploration rate ${ϵ}_{\mathrm{start}}$	$1.0$	Final exploration rate ${ϵ}_{\mathrm{end}}$	$0.05$
Exploration decay steps	30,000	Hidden layer size	256
Squared gradient decay factor	$0.999$	Learning rate ${\alpha }_0$	${10}^{-4}$
Maximum training episodes	2000

.

6.2. Baseline Schemes

To evaluate the effectiveness of the proposed DRL-HEEC scheme, we compare it with three categories of baseline schemes, including deep Q-network (DQN)-based schemes, PPO-based schemes, and a greedy algorithm (GA)-based scheme. The details of these baselines are described as follows.

• DQN-based schemes [27]: The UAV trajectory planning is performed by a conventional DQN algorithm. In DQN-Opt, the UAV trajectory is determined by the DQN algorithm while the remaining components, including the optimal hovering altitude and lower-level resource allocation, are optimized using the same optimization framework as in the proposed scheme. In DQN-Fixed, UAV trajectory planning is performed by the DQN algorithm, while the UAV flies at a fixed altitude of 15 m during data collection. For the lower-level transmission strategy, all ground nodes are served using NOMA-based data transmission.
• PPO-based schemes [29]: The UAV trajectory planning is performed by the PPO algorithm. In PPO-Opt, the UAV visiting trajectory is determined by the PPO algorithm, while the optimal hovering altitude and lower-level resource allocation are solved using the same optimization framework as in the proposed scheme. In PPO-Fixed, UAV trajectory planning is conducted by the PPO algorithm, while the UAV maintains a fixed flight altitude of 15 m. In the lower-level resource allocation, TDMA is adopted for data transmission among ground nodes.
• GA-based scheme: In the GA scheme, both UAV trajectory planning and resource allocation are performed based on a greedy strategy. Specifically, at each decision step, the UAV selects the next visiting grid cell according to the locally best immediate reward, such as the shortest distance, largest amount of collected data, or highest instantaneous energy efficiency. The corresponding resource allocation is also determined greedily without considering long-term system performance.

6.3. Performance Comparison and Discussion

As shown in Figure 6, the proposed DRL-HEEC scheme consistently achieves a significantly higher and more stable cumulative reward than DQN and PPO. While the upper-level action space (e.g., UAV trajectory selection) remains relatively stable, expanding the node scale to $N=15$ significantly intensifies the complexity of underlying local resource allocation and data collection constraints. Under such dense and complex communication environments, traditional algorithms such as DQN and PPO often struggle to coordinate the tight coupling between trajectory control and lower-layer allocation, leading to performance degradation or suboptimal rewards. In contrast, DRL-HEEC effectively handles this heightened complexity through its robust reward design and coordination mechanism, maintaining superior reward performance across different scales.

As illustrated in Figure 7, the proposed DRL-HEEC achieves the shortest total flight distance of 4212.85 m, outperforming DQN (4270.03 m), PPO (4339.64 m), and the randomized greedy baseline (4576.24 m). The greedy approach suffers from myopic decisions, leading to severe detours and local optima, while traditional DRL benchmarks such as DQN and PPO face limitations in value estimation or policy updates. On the other hand, the proposed DRL-HEEC framework effectively captures network dynamics through its optimized reward mechanism; specifically, the incorporation of dynamic action masking proactively prunes invalid spatial transitions, eliminating the redundant exploration steps that inflate trajectory lengths in unconstrained benchmarks. Furthermore, the double DQN architecture effectively mitigates upward overestimation bias, ensuring stable value convergence and preventing the agent from adopting suboptimal routing policies. Consequently, DRL-HEEC generates a smoother and more globally optimal path, which translates directly into significant energy savings and enhanced operational efficiency for the UAV system.

To evaluate the performance of the proposed scheme across varying node scales, we generated topologies with different numbers of nodes ($N\in \{5,6,\dots ,15\}$) and conducted ten randomized trials for each, yielding a total of $10\times 10\times 11$ experimental results. By averaging these results, we determined the energy efficiency of various schemes at different node scales, as depicted in Figure 8. It is evident that our DRL-HEEC consistently maintains the highest energy efficiency. Meanwhile, DQN and DDQN, which integrate the underlying optimization algorithm proposed in this paper, also achieve commendable energy efficiency, further validating the effectiveness and universality of the proposed optimization mechanism. As the number of nodes increases, higher network complexity causes energy efficiency fluctuations in most schemes. Nevertheless, DRL-HEEC maintains robust and superior performance. This excellent scalability fundamentally stems from our two-layer decoupled architecture, which separates UAV trajectory planning from local resource allocation. This design effectively prevents the exponential explosion of the state action space as the network expands. Consequently, DRL-HEEC makes adaptive decisions without being overwhelmed by dimensionality, whereas traditional tightly coupled baselines suffer severe performance degradation in large-scale scenarios.

In addition, we compare the ratio of emergency nodes satisfying the QoS requirements to the total number of emergency nodes under different emergency data generation probabilities, i.e., $P_e\in \{0.1,0.2,0.3\}$, as shown in Figure 9. It can be clearly observed that the QoS requirements of emergency nodes are well satisfied under the proposed DRL-HEEC, DQN, and PPO schemes. In contrast, for the schemes employing a fixed flight altitude and a single TDMA- or NOMA-based access mechanism, an increasing number of emergency nodes fail to meet their QoS requirements as $(P_e$ increases. This observation further indicates that the QoS satisfaction of emergency nodes is less affected by the UAV trajectory, but is more closely related to the underlying resource allocation algorithm. Therefore, an effective resource allocation mechanism plays a critical role in guaranteeing the QoS performance of emergency services, especially when the probability of emergency data generation becomes higher.

Finally, we investigate the impact of the UAV battery capacity on energy efficiency, where the number of nodes in each grid cell is fixed at $N=10$. The corresponding results are presented in Figure 10. It can be observed that the energy efficiency decreases significantly when the UAV battery capacity is relatively small. This is mainly because the UAV has to return to the HAP more frequently for recharging, which introduces additional flight energy consumption and reduces the effective service time. Nevertheless, the proposed DRL-HEEC scheme still achieves the highest energy efficiency among all compared methods. This performance gain can be attributed to the proposed bi-level mechanism, which jointly improves both UAV trajectory planning and resource allocation. As a result, even under limited battery capacity, the proposed scheme can utilize the available energy more efficiently and maintain superior energy-efficiency performance. Compared with the other reinforcement learning-based algorithms, our proposed DRL-HEEC improves energy efficiency by approximately $9\%$ to $12\%$.

7. Conclusions and Future Work

This paper has studied an energy-efficient UAV-assisted data collection scheme for WPCN with both normal and emergency traffic. To maximize system energy efficiency while satisfying the QoS requirements of emergency nodes, we propose a DRL-based hierarchical energy-efficient control scheme, which we name DRL-HEEC. The proposed DRL-HEEC jointly optimizes UAV trajectory planning, charging decisions, access mode selection, and resource allocation through a bi-level decision-making structure. By coordinating upper-level UAV mobility control and lower-level TDMA/NOMA-based resource allocation, DRL-HEEC can effectively adapt to dynamic network states and improve energy utilization. Extensive simulation results verify the superiority of the proposed scheme over benchmark methods. Our proposed DRL-HEEC achieves the highest energy efficiency under different numbers of nodes, emergency data generation probabilities, and UAV battery capacities. Moreover, it can maintain a high QoS satisfaction ratio for emergency nodes even when the emergency traffic load increases. Compared with other reinforcement learning-based algorithms, the proposed scheme improves the energy efficiency by approximately $9\%$ to $12\%$, demonstrating the effectiveness of the proposed hierarchical joint optimization framework.

In future work, we will extend the proposed framework to multi-UAV cooperative scenarios in which UAV coordination, collision avoidance, and distributed resource allocation need to be addressed. In addition, more realistic system models such as imperfect channel state information, dynamic obstacles, and mobile IoT nodes, will be considered. Finally, the design of lightweight reinforcement learning algorithms will be explored in order to reduce computational complexity and facilitate real-time implementation in practical UAV-assisted WPCNs.