Parallel Task Offloading and Trajectory Optimization for UAV-Assisted Mobile Edge Computing via Hierarchical Reinforcement Learning

Abstract

With the rapid growth of IoT data and increasing demands for low-latency computation, UAV-assisted mobile edge computing (MEC) offers a flexible solution to overcome the limitations of fixed MEC servers. To better reflect concurrent service scenarios, this paper innovatively develops a multi-channel task modeling method, enabling UAVs to simultaneously select multiple users and offload tasks via separate channels, thereby breaking the constraints of traditional sequential assumptions. To address task conflicts and resource waste caused by sequential service assumptions, this paper proposes a hierarchical reinforcement learning-based trajectory and offloading decision-making framework (H-TAOD) for UAVs with multi-channel parallel processing. The framework decouples UAV trajectory planning and task offloading into two sub-tasks optimized via appropriate reinforcement learning methods. An invalid action masking (IAM) mechanism is introduced to avoid channel conflicts. Simulation results verify the superiority of H-TAOD in reward, delay, and convergence.

1. Introduction

With the rapid development of mobile communication technologies and the widespread use of smart devices, diverse network services, and applications have emerged, placing increasing demands on the quality of network services and latency. Mobile edge computing (MEC) has been proposed as a critical complement to cloud computing by deploying computing and storage resources at the edge of mobile networks to provide ultra-low-latency and high-bandwidth services.

In this context, computation-intensive and latency-sensitive mobile applications—such as augmented reality (AR), autonomous driving, and remote healthcare—are becoming increasingly prevalent, placing stringent requirements on response time and quality of service (QoS) [1]. MEC addresses these challenges by reducing communication delay and offloading burdens on the core network, while also enhancing user privacy protection [2].

To further improve MEC performance in dynamic or infrastructure-limited scenarios, unmanned aerial vehicles (UAVs) have been widely adopted due to their high mobility and deployment flexibility. UAVs can act as flying edge servers or relay nodes, capable of offloading computation tasks, collecting data, and facilitating communication [3,4]. As a result, UAV-assisted MEC systems have received growing attention in both academia and industry. For instance, Zhang Jie et al. [5] adopted Lyapunov optimization to jointly optimize UAV trajectories and offloading decisions under stochastic task arrivals and dynamic channel conditions. Lyapunov optimization, rooted in control theory, provides a powerful framework for stabilizing dynamic systems and has been widely applied to real-time decision-making in mobile edge computing. Wan Sheng et al. [6] proposed a three-layer data processing architecture using deep reinforcement learning and dynamic resource control. Additionally, Wang Liang et al. [7] developed a Q-learning-based algorithm for large-scale user association and task allocation in multi-UAV networks.

Recent works have also explored the integration of advanced learning methods into MEC optimization. For example, Asheralieva and Niyato [8] employed hierarchical game theory and reinforcement learning for multi-provider task sharing, and Wang Liang et al. [9] introduced a deep Q-network (DQN) to optimize UAV trajectories in dynamic environments. Other studies have employed deep deterministic policy gradient (DDPG) [10], double DQN (DDQN) [11,12], twin delayed DDPG (TD3) [13], and graph neural networks [14] to address various challenges in task offloading, energy efficiency, and fairness.

Moreover, hierarchical [15,16] and multi-agent reinforcement learning [17,18] have been employed to tackle collaborative optimization problems in UAV networks. Geng Yu et al. [19] and Zhou Hang et al. [20] proposed hierarchical and multi-agent RL algorithms to enhance relay selection and data transmission, respectively. Yao Ziqing et al. [21] incorporated graph attention mechanisms to improve task scheduling and service caching in edge digital twin systems.

As deep learning (DL) models become integral to modern MEC services, their increasing inference complexity and model sizes impose stringent demands on caching strategies and resource scheduling mechanisms. To address this, Xu Jing et al. [22] and Fang Cheng et al. [23] investigated task offloading and caching optimization under non-orthogonal multiple access (NOMA) scenarios, aiming to enhance throughput without compromising energy efficiency. Their work emphasized the importance of coordinated caching and transmission strategies in scenarios with heterogeneous quality-of-service (QoS) requirements. Building on this, Lin Ning et al. [24] introduced a dual-slot scheduling mechanism that jointly handles task offloading and caching placement. By decoupling the decision-making processes across two time slots, the proposed approach significantly improves system responsiveness and task completion latency. Xu Yuntao et al. [25] further extended the caching optimization paradigm by proposing a hybrid caching allocation strategy based on the soft actor–critic (SAC) algorithm, enabling the system to adaptively allocate caching resources in response to diverse user application demands and dynamic environmental conditions.

In parallel, several studies have explored the joint optimization of trajectory control and resource management from a multi-objective perspective. Wang Zhiheng et al. [26] proposed a UAV-assisted MEC architecture wherein UAV trajectories and resource allocation are jointly optimized through a dynamic programming framework. The proposed scheme enhances service efficiency by adapting UAV flight paths in accordance with computing and communication demands. Sun Guojie et al. [27] designed a comprehensive optimization framework that balances delay, energy consumption, and task completion rate using a multi-objective evolutionary algorithm. This approach ensures more reliable task execution across various operational constraints. Cui Jia et al. [28] focused on the integration of power-aware networks and UAV-MEC systems, presenting a joint trajectory and resource allocation strategy that dynamically adapts to variations in network topology and workload distribution. Hui Meng et al. [29] addressed the impact of UAV deployment height on system performance. By introducing a multivariable optimization model, their work simultaneously considers offloading ratio design and altitude control, achieving near-optimal performance bounds in diverse mission scenarios.

In addition, research in the fields of mobile edge computing and UAV modeling is increasingly moving toward more practical scenarios. Existing studies have advanced system modeling and optimization methods from various perspectives. For instance, Premsankar and Ghaddar [30] proposed an energy-efficient service placement strategy for latency-sensitive applications, Ko et al. [31] introduced a joint computation offloading and service caching mechanism considering personalized service preferences, and Hortelano et al. [32] provided a comprehensive survey on reinforcement learning-based computation offloading approaches in edge computing systems. Furthermore, Hayal et al. [33] focused on the practical modeling of hovering UAV-assisted free-space optical (FSO) communication systems, investigating performance under pointing errors and atmospheric turbulence, which offers valuable insights for enhancing the reliability of UAV communication and mobile edge computing systems.

For this purpose, although significant progress has been made, most existing studies assume that UAVs can only serve one user within a single time slot, overlooking their potential for parallel task processing. This assumption tends to cause service conflicts and resource wastage in multi-user high-concurrency scenarios, thereby limiting the overall performance of UAV-assisted mobile edge computing (UAV-MEC) systems.

To fully exploit the UAVs’ capability for concurrent task execution, this paper proposes a novel parallel task offloading and trajectory optimization framework for multi-channel UAV-assisted MEC systems. A practical system model supporting concurrent task processing is developed, and a hierarchical reinforcement learning (HRL)-based optimization approach is introduced to jointly optimize task offloading and UAV trajectory control.

In the proposed framework, the optimization problem is decoupled into two sub-tasks corresponding to two hierarchical layers: the lower layer handles task offloading and resource allocation, while the upper layer focuses on trajectory control. At the lower layer, a discrete soft actor–critic algorithm with invalid action masking (discrete SAC-IAM) is proposed to dynamically filter infeasible task-channel assignments, thereby improving decision stability. Specifically, the lower layer minimizes user device waiting time by performing multi-channel task selection and resource scheduling, ensuring efficient task dispatching. Meanwhile, the upper layer adopts a continuous SAC algorithm to control UAV trajectories, aiming to further reduce system delay while maintaining high service efficiency.

Through the collaborative design of the two layers, the proposed method enhances the training efficiency and decision flexibility of the UAV-MEC system. Simulation results demonstrate that the proposed approach outperforms existing baseline methods in terms of task delay, total accumulated reward, and convergence speed, validating its effectiveness and practicality for parallel task offloading and trajectory optimization in complex scenarios.

2. System Model

To support parallel task offloading and trajectory scheduling in multi-user scenarios, this section builds a UAV-assisted mobile edge computing (MEC) system model with multi-channel concurrent service capability. The UAV serves as an aerial edge node that can simultaneously offload tasks for multiple user equipment (UE) in each time slot. The model considers key factors such as transmission rates, computation limits, task queuing, and UAV mobility and energy constraints, aiming to capture the coordination between offloading and trajectory planning in complex environments. The main variables and parameters involved in the modeling are summarized in

Table 1. List of main notations.

Notation	Description
System Model
N, $\mathcal{N}$	Number and set of user equipments (UEs)
T, I, $\tau$, $\mathcal{I}$	Total task duration, number of slots, slot length, and set of slots
$D_{n,i}$	Task data size (Mbits) generated by UE n at slot i
$F_{n,i}$	Required CPU cycles for UE n’s task at slot i
s	CPU cycles required per unit data
K	Maximum number of tasks the UAV can process in parallel per slot
$D_{max}$	Maximum data volume (Mbits) the UAV can process per slot
$(X_i,Y_i)$	2D coordinate of the UAV in slot i
H	Fixed altitude of the UAV
$V_i$, ${\alpha }_i$	UAV speed and heading angle in slot i
$V_{x,i}$, $V_{y,i}$	UAV velocity components in x and y direction at slot i
$P_0$	Blade profile power in flight energy model
$U_{\mathrm{tip}}$	Rotor blade tip speed
$P_i$	Induced power in hover
$V_0$	Mean rotor induced velocity in hover
$d_0$	Fuselage drag ratio
$\rho$	Air density
S	Rotor disk area
A	UAV frame area
$t^{\mathrm{fly}}$	UAV flight duration within each slot
$(X_n,Y_n)$	2D position of UE n
$R_{n,i}$	Horizontal distance between UAV and UE n at slot i
$r_{n,i}$	Transmission rate from UE n to UAV at slot i
B, $p_n$	Bandwidth and transmit power of UE n
$f(R_{n,i},H)$	Path loss function including altitude and distance effects
$z_{n,i}$	Binary offloading decision variable for UE n at slot i
${\mathcal{Z}}_i$	Set of offloading decisions in slot i
$f^{\mathrm{uav}}$, $f^{\mathrm{UE}}$	CPU frequencies of UAV and UE, respectively,
$t_{n,i}^{\mathrm{tx}}$	Transmission delay for UE n’s task at slot i
$t_{n,i}^{\mathrm{uav}}$	Computation delay on UAV for UE n’s task at slot i
$t_{n,i}^{\mathrm{loc}}$	Local computation delay at UE n
$t_{n,i}$	Total delay of UE n at slot i
$P_i^f$	UAV flight power at speed $V_i$
$E_i^{\mathrm{fly}}$	UAV flight energy in slot i
$p_{\mathrm{hov}}$	UAV hovering power
$E_i^{\mathrm{hov}}$	Hovering energy in slot i
$\kappa$, $\beta$	Constants in UAV computation energy model
${\delta }_i$	CPU usage indicator in slot i
$E_i^{\mathrm{cmp}}$	UAV computation energy in slot i
$E_i^{\mathrm{total}}$	Total UAV energy consumption in slot i
$B_i$	Remaining UAV battery after slot i
$B_{min}$	Minimum allowed UAV battery level
Problem Formulation and Optimization
$(V_i,{\alpha }_i)$	UAV’s control variables: speed and heading
$\left\{z_{n,i}\right\}$	Offloading decision set across all slots
Objective	$min{\sum }_{i=1}^I{\sum }_{n=1}^N{t}_{n,i}$ (Minimize total user delay)
(C1)	Total offloaded data must not exceed $D_{max}$ in any slot, Formula (5)
(C2)	Number of offloaded tasks $\le K$ in each slot, Formula (6)
(C3)	UAV position constraints within $[0,L_{max}]$, Formula (18)
(C4)	UAV speed and heading constraints, Formula (19)
(C5)	UAV battery dynamics across slots, Formula (20)
(C6)	UAV battery level must stay above $B_{min}$, Formula (16)

.

First, an overview of the overall system process is presented. As illustrated in Figure 1, a schematic diagram of the UAV-assisted mobile edge computing system is shown. At the beginning, all users generate tasks and send offloading requests. The UAV receives these requests and determines the optimal decision. Subsequently, the system proceeds with decision execution, task transmission, offloading, and result feedback. If there are still remaining tasks, this cycle is repeated iteratively. The detailed system modeling is introduced as follows.

2.1. Basic Definitions

As illustrated in Figure 2, N UEs are randomly distributed in a square region of side length $L_{max}$, denoted by the set $\mathcal{N}$. A single UAV flies at a fixed altitude H throughout the task execution period T, which is discretized into I equal time slots with duration $\tau =T/I$, indexed by $\mathcal{I}$.

In each slot $i\in \mathcal{I}$, each UE $n\in \mathcal{N}$ may generate a task of size $D_{n,i}$ requiring $F_{n,i}=s·D_{n,i}$ CPU cycles. The UAV can handle up to K tasks and a total of $D_{max}$ Mbits per slot.

The UAV’s 2D position in slot i is $(X_i,Y_i)$, with heading angle ${\alpha }_i$ and speed $V_i$. Its movement within each slot duration $t^{\mathrm{fly}}$ follows:

$$V_{x,i}=V_{i}cos\left({\alpha }_i\right),V_{y,i}=V_{i}sin\left({\alpha }_i\right),$$

(1)

$$X_{i+1}=X_i+V_{x,i}·t^{\mathrm{fly}},Y_{i+1}=Y_i+V_{y,i}·t^{\mathrm{fly}}.$$

(2)

The horizontal distance from the UAV to UE n is given by:

$$R_{n,i}=\sqrt{{(X_i-X_n)}^2+{(Y_i-Y_n)}^2}.$$

(3)

The transmission rate is modeled as:

$$r_{n,i}=B{log}_2\left(1+{\displaystyle \frac{p_n}{f(R_{n,i},H)}}\right).$$

(4)

Offloading Decision

The offloading indicator $z_{n,i}\in \{0,1\}$ represents whether UE n offloads its task at slot i. The UAV’s per-slot resource constraints are:

$$\begin{array}{cc}\hfill \sum _{n=1}^N{z}_{n,i}{D}_{n,i}& \le D_{max},\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill \sum _{n=1}^N{z}_{n,i}& \le K.\hfill \end{array}$$

(6)

2.2. Computation Model

If a task is offloaded, the transmission and computation delays are calculated as:

$$\begin{array}{cc}\hfill t_{n,i}^{\mathrm{tx}}& ={\displaystyle \frac{z_{n,i}{D}_{n,i}}{r_{n,i}}},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill t_{n,i}^{\mathrm{uav}}& ={\displaystyle \frac{z_{n,i}{F}_{n,i}}{f^{\mathrm{uav}}}}.\hfill \end{array}$$

(8)

If the task is processed locally by the UE, the local delay is:

$$t_{n,i}^{\mathrm{loc}}={\displaystyle \frac{(1-z_{n,i})F_{n,i}}{f^{\mathrm{UE}}}}.$$

(9)

The total task execution delay for UE n in slot i is then:

$$t_{n,i}=max\left\{t_{n,i}^{\mathrm{tx}}+t_{n,i}^{\mathrm{uav}},t_{n,i}^{\mathrm{loc}}\right\}.$$

(10)

UAV Energy Consumption

The UAV’s energy consumption in each slot comprises flight, hovering, and computation energy:

Flight energy: Based on an aerodynamic model, the UAV flight power is given by:

$$P_i^f=P_0\left(1+{\displaystyle \frac{3V_i^2}{U_{\mathrm{tip}}^2}}\right)+P_i\left({\displaystyle \frac{V_0}{V_i}}\right)+{\displaystyle \frac{1}{2}}{d}_0\rho SA{V}_i^3,$$

(11)

where $P_0$ is the blade profile power, $U_{\mathrm{tip}}$ is the rotor tip speed, $P_i$ is the induced power in hover, $V_0$ is the mean rotor induced velocity in hover, $d_0$ is the fuselage drag ratio, $\rho$ is the air density, S is the rotor disk area, and A is the UAV’s frame area.

Then the flight energy during slot i is:

$$E_i^{\mathrm{fly}}=P_i^f·t^{\mathrm{fly}}.$$

(12)

Hovering energy:

$$E_i^{\mathrm{hov}}=p_{\mathrm{hov}}·(\tau -t^{\mathrm{fly}}).$$

(13)

Computation energy:

$$E_i^{\mathrm{cmp}}=\kappa ·{\left(f^{\mathrm{uav}}\right)}^{\beta }·{\delta }_i.$$

(14)

Total energy consumption:

$$E_i^{\mathrm{total}}=E_i^{\mathrm{fly}}+E_i^{\mathrm{hov}}+E_i^{\mathrm{cmp}}.$$

(15)

The UAV battery level evolves as:

$$B_{i+1}=B_i-E_i^{\mathrm{total}},B_i\ge B_{min}.$$

(16)

2.3. Objective Function

The overall objective is to minimize the total user delay over all time slots and users:

$$\underset{(X_i,Y_i,V_i,{\alpha }_i),z_{n,i}}{min}\sum _{i=1}^I\sum _{n=1}^N{t}_{n,i},$$

(17)

subject to the Constraints (5) and (6) and the following:

$$X_i,Y_i\in [0,L_{max}],$$

(18)

$$0\le V_i\le V_{max},0\le {\alpha }_i<2\pi ,$$

(19)

$$B_{i+1}=B_i-E_i^{\mathrm{total}}.$$

(20)

Battery Constraint (16) is enforced as well to ensure the UAV always operates above the safety level $B_{min}$.

In summary, this section models a multi-channel UAV-assisted MEC system, where key practical constraints such as the maximum task load per slot and the parallel channel concurrency limit are jointly considered. The system formulation integrates the optimization of UAV trajectory, task offloading decisions, and energy consumption management, with the objective of minimizing total user delay under mobility, communication, and energy constraints.

3. Hierarchical Optimization via SAC and IAM-Enhanced Discrete SAC

To efficiently solve the above joint optimization problem with multiple constraints and strong coupling, conventional optimization methods often encounter challenges in modeling complexity, poor generalization, and high computational overhead, especially in high-dimensional and mixed-action spaces. Given that UAV control involves a continuous action space, while user task scheduling belongs to a discrete decision space, this section adopts a hierarchical reinforcement learning (HRL) framework to decouple and collaboratively optimize the two sub-tasks.

Specifically, in the high-level policy, a soft actor–critic (SAC) agent is employed to handle the continuous control of UAV trajectory and energy management, enabling dynamic planning of UAV speed and heading. At each decision point along the trajectory, a low-level policy is designed based on a discrete SAC algorithm enhanced with an invalid action masking (IAM) mechanism, which effectively filters out infeasible actions and improves the efficiency and quality of service selection decisions in large discrete action spaces.

This hierarchical learning structure significantly enhances policy convergence and training stability, while improving the system’s adaptability and scalability in highly dynamic and constrained environments.

3.1. Hierarchical Decomposition of the Optimization Problem

At the beginning of each time slot i, the UAV determines a new position to move to, aiming to achieve satisfactory user-latency performance during this slot. After moving to the new position, the UAV remains static for the remainder of the time slot and makes offloading decisions to further reduce the total user latency. These two decision steps alternate until i reaches I.

Given the difference in action characteristics (continuous movement vs. discrete offloading), the overall optimization is split into two sub-problems:

• Location Optimization (${\mathcal{P}}_{\mathrm{loc}}$): At the beginning of time slot i, determine the UAV’s new position $(x_{i+1},y_{i+1})$ under constraints (16), (18)–(20). This affects the distances to users and thus the transmission rate, indirectly influencing the offloading performance within the slot. Formally:

  $$\underset{\left\{(x_t,y_t)\right\}}{min}\sum _{i=1}^I\sum _{n=1}^N{t}_{n,i},\mathrm{s}.\mathrm{t}.\left(16\right),\left(18\right)–\left(20\right).$$

  (21)
• Offloading Optimization (${\mathcal{P}}_{\mathrm{off}}$): Given the UAV’s position within a time slot, decide which users to serve (i.e., which tasks to offload) to minimize the total user latency:

  $$\underset{\left\{a_{n,t}\right\}}{min}\sum _{i=1}^I\sum _{n=1}^N{t}_{n,i},\mathrm{s}.\mathrm{t}.\left(5\right)–\left(6\right).$$

  (22)

By separating the problem into these two layers, we can effectively handle the high-dimensional joint optimization of continuous and discrete decisions.

3.2. Hierarchical Reinforcement Learning Framework

Before jointly training both decision layers, we first focus on the offloading decision layer in isolation. Specifically, in each training episode, we randomly generate user positions, task demands, and an initial UAV location. The UAV remains static during the episode, and the offloading layer is trained to learn optimal user selection strategies that minimize total latency under random spatial and task conditions.

Figure 3 illustrates the overall architecture of our hierarchical reinforcement learning framework, which decomposes the optimization into two coordinated decision layers:

• Location Optimization Layer: Observes the environment state and outputs a continuous action representing the UAV’s next position (or equivalently, its velocity direction ${\theta }_t$ and speed $V_t$). Once the UAV reaches the new position, control is passed to the lower layer.
• Offloading Decision Layer: Given the UAV’s current location, it determines which users to serve within the time slot using discrete actions, while ensuring the parallel channel and capacity constraints are respected.

The two layers share portions of the environment state (e.g., battery levels, user locations, task demands), but are trained separately, each optimizing its respective sub-objective. This design reduces the overall complexity of joint optimization and improves training efficiency.

3.3. Implementation of the Offloading Decision Layer

The offloading decision layer aims to select a subset of users to be served in each time slot, given the UAV’s current location. This layer is modeled as a Markov decision process (MDP), and we employ a discrete variant of the soft actor–critic (SAC) algorithm enhanced with an invalid action masking (IAM) mechanism, which explicitly filters infeasible actions to improve training stability and efficiency.

3.3.1. State Space, Action Space, and Reward Design

State space. Let ${\mathbf{s}}_t^{\mathrm{off}}$ denote the state at time slot t, including the following:

• UAV position $(x_t,y_t)$;
• Relative user positions: ${\{(x_n-x_t),(y_n-y_t)\}}_{n=1}^N$;
• Task demand of each user $C_{n,t}$;
• User availability indicator (selectable or not);
• Current UAV battery level $B_t$, remaining channel count $K^{\prime }\le K$, and remaining memory (if applicable).

Action space. At each step, the agent selects one user to serve. The action space is ${\mathcal{A}}^{\mathrm{off}}=\{0,1,\dots ,N\}$, where $a=0$ indicates no user is served due to resource constraints.

Reward function. The reward is defined as the negative of total latency:

$$r_t^{\mathrm{off}}=-\sum _{n=1}^N{t}_{n,t}.$$

(23)

3.3.2. Discrete-SAC with Invalid Action Masking

The SAC algorithm maximizes a reward–entropy objective. In discrete action spaces, the policy ${\pi }_{\varphi }\left(a\right|s)$ is modeled as a categorical distribution via the Softmax function. To enhance decision reliability under resource constraints, the invalid action masking (IAM) technique is incorporated into key SAC components.

Maximum entropy objective.

$$J\left(\pi \right)=\sum _t{\mathbb{E}}_{({\mathbf{s}}_t,a_t)\sim \pi }\left[R({\mathbf{s}}_t,a_t)+\alpha \mathcal{H}\left(\pi (·|{\mathbf{s}}_t)\right)\right]$$

(24)

Q-function update.

$$y=r+\gamma \sum _{a^{\prime }}{\pi }_{\varphi }\left(a^{\prime }\right|{\mathbf{s}}^{\prime })\left[\underset{i}{min}{Q}_{{\theta }_i}({\mathbf{s}}^{\prime },a^{\prime })-\alpha log{\pi }_{\varphi }\left(a^{\prime }\right|{\mathbf{s}}^{\prime })\right]$$

(25)

Soft value function.

$$V_{\psi }\left(\mathbf{s}\right)=\sum _a{\pi }_{\varphi }\left(a\right|\mathbf{s})\left[\underset{i}{min}{Q}_{{\theta }_i}(\mathbf{s},a)-\alpha log{\pi }_{\varphi }\left(a\right|\mathbf{s})\right]$$

(26)

Policy update.

$$J\left({\pi }_{\varphi }\right)={\mathbb{E}}_{\mathbf{s}}\sum _a{\pi }_{\varphi }\left(a\right|\mathbf{s})\left[\alpha log{\pi }_{\varphi }\left(a\right|\mathbf{s})-\underset{i}{min}{Q}_{{\theta }_i}(\mathbf{s},a)\right]$$

(27)

Temperature update.

$$J\left(\alpha \right)={\mathbb{E}}_{\mathbf{s},a\sim {\pi }_{\varphi }}\left[-\alpha \left(log{\pi }_{\varphi }\left(a\right|\mathbf{s})+{\mathcal{H}}_{\mathrm{target}}\right)\right]$$

(28)

3.3.3. Invalid Action Masking (IAM)

We define a binary mask $m\left(\mathbf{s}\right)$:

$$m_a\left(\mathbf{s}\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{action}a\mathrm{is}\mathrm{valid},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(29)

Masked soft value function:

$$V_{\psi }\left(\mathbf{s}\right)=\sum _a{m}_a\left(\mathbf{s}\right){\pi }_{\varphi }\left(a\right|\mathbf{s})\left[\underset{i}{min}{Q}_{{\theta }_i}(\mathbf{s},a)-\alpha log{\pi }_{\varphi }\left(a\right|\mathbf{s})\right]$$

(30)

Masked policy update:

$$J\left({\pi }_{\varphi }\right)={\mathbb{E}}_{\mathbf{s}}\sum _a{m}_a\left(\mathbf{s}\right){\pi }_{\varphi }\left(a\right|\mathbf{s})\left[\alpha log{\pi }_{\varphi }\left(a\right|\mathbf{s})-\underset{i}{min}{Q}_{{\theta }_i}(\mathbf{s},a)\right]$$

(31)

Masked temperature update:

$$J\left(\alpha \right)={\mathbb{E}}_{\mathbf{s},a\sim {\pi }_{\varphi }}\left[-\alpha m_a\left(\mathbf{s}\right)\left(log{\pi }_{\varphi }\left(a\right|\mathbf{s})+{\mathcal{H}}_{\mathrm{target}}\right)\right]$$

(32)

3.3.4. Iterative Offloading Decision Strategy

To avoid infeasible actions in large discrete spaces, the proposed method uses an iterative offloading strategy with IAM. Instead of generating all offloading decisions in one shot, the policy selects users one by one, updating the mask $m\left(\mathbf{s}\right)$ after each step to exclude already-selected or invalid users. This process continues until UAV resources are fully utilized or capacity limits are reached.

3.3.5. Offloading Decision Layer Training Pseudocode

As illustrated in Figure 4, the diagram presents the training workflow of the lower-layer network. The UAV interacts with the environment to obtain the current state $s_t$, which is fed into the policy network along with the corresponding invalid action mask $M\left(s_t\right)$ to generate a valid action $a_t$. This process is repeated iteratively until the UAV’s resources are fully utilized. The selected action is then executed, receiving an immediate reward, and the transition tuple $\{s_t,a_t,r_t,s_{t+1},M\left(s_t\right)\}$ is stored in the replay buffer.

Once the buffer has accumulated sufficient experience samples, the training process is triggered. Specifically, a mini-batch of samples is drawn from the buffer to compute the target value using the target Q-networks. The temporal difference error between the target and current Q-values is used to calculate the loss for updating the Q-networks. Meanwhile, the policy network is optimized using policy gradients derived from the minimum Q-value and the entropy regularization term. The entire training process follows the standard discrete soft actor–critic (SAC) update procedure, enhanced with invalid action masking (IAM) to ensure learning occurs strictly within the valid action space. This integration significantly improves training efficiency and policy stability under discrete constraints.

The offloading decision layer is trained first. The training pseudocode is Algorithm 1:

Algorithm 1 Offloading decision with IAM-SAC.

1: Initialize: $Q_{{\theta }_1},Q_{{\theta }_2}$, policy network ${\pi }_{\varphi }$, value network $V_{\psi }$, temperature $\alpha$, replay buffer $\mathcal{D}$  2: Set $m_a\left(\mathbf{s}\right)$ to mask invalid actions  3: for episode = 1 to max_episodes do  4:       Randomly initialize UAV and user positions, task demands  5:       Observe initial state ${\mathbf{s}}_0^{\mathrm{off}}$  6:       for $t=0$ to $T-1$ do  7:             Determine valid actions via $m_a\left({\mathbf{s}}_t\right)$  8:             Sample action $a_t\sim {\pi }_{\varphi }$ (excluding invalid actions)  9:             Execute $a_t$, observe reward $r_t$ and next state ${\mathbf{s}}_{t+1}$ 10:             Store transition $({\mathbf{s}}_t,a_t,r_t,{\mathbf{s}}_{t+1})$ in $\mathcal{D}$ 11:             Update Q-networks and policy via masked SAC update rules 12:       end for 13: end for

3.4. Implementation of the Location Optimization Layer

The goal of the location optimization layer is to determine the UAV’s continuous movement at the beginning of each time slot, in order to minimize the total task latency through better positioning. Once the movement decision is executed, the offloading layer operates for the remainder of the time slot.

To simplify the action representation and make it more interpretable, we parameterize the UAV’s movement as two components: the speed in the x- and y-axes, denoted as $V_{x,t}$ and $V_{y,t}$, respectively. The UAV then updates its position accordingly.

3.4.1. State Space, Action Space, and Reward Design

State space. Let ${\mathbf{s}}_t^{\mathrm{loc}}$ denote the state observed by the location optimization layer at time t. It includes:

• UAV current position $(x_t,y_t)$;
• Positions of all users relative to the UAV;
• Task demand and status (waiting/served) of each user;
• UAV status: battery level $B_t$, available channels, memory, etc.

Action space. The output of this layer is a continuous 2D velocity vector:

$${\mathbf{a}}_t=(V_{x,t},V_{y,t})\in {\mathbb{R}}^2.$$

This controls the UAV’s movement at the beginning of each time slot.

Reward function. After the UAV moves, the pre-trained offloading decision policy is invoked to determine latency, which is used as the reward:

$$r_t^{\mathrm{loc}}=-\sum _{n=1}^N{t}_{n,t}-{\lambda }_{\mathrm{bound}}·\mathbb{I}\left(\mathrm{out}-\mathrm{of}-\mathrm{bound}\right),$$

(33)

where $\mathbb{I}(·)$ is an indicator function for out-of-bound violations.

3.4.2. Training via Continuous Soft Actor–Critic (SAC)

The action space is continuous, so the standard SAC algorithm is applied using reparameterization. Compared to discrete SAC, continuous SAC samples from a Gaussian distribution and supports backpropagation.

Soft Q-function update.

$$J\left(Q_{{\theta }_i}\right)={\mathbb{E}}_{(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime })}\left[{\left(Q_{{\theta }_i}(\mathbf{s},\mathbf{a})-y\right)}^2\right],$$

(34)

with target:

$$y=r+\gamma \left[\underset{j=1,2}{min}{Q}_{{\theta }_j}({\mathbf{s}}^{\prime },{\tilde{\mathbf{a}}}^{\prime })-\alpha log{\pi }_{\varphi }\left({\tilde{\mathbf{a}}}^{\prime }\right|{\mathbf{s}}^{\prime })\right],$$

(35)

where ${\tilde{\mathbf{a}}}^{\prime }\sim {\pi }_{\varphi }(·|{\mathbf{s}}^{\prime })$ via reparameterization.

Policy update.

$$J\left({\pi }_{\varphi }\right)={\mathbb{E}}_{\mathbf{s}\sim \mathcal{D},ϵ\sim \mathcal{N}(0,1)}\left[\alpha log{\pi }_{\varphi }\left(\tilde{\mathbf{a}}\right|\mathbf{s})-\underset{i=1,2}{min}{Q}_{{\theta }_i}(\mathbf{s},\tilde{\mathbf{a}})\right],$$

(36)

where $\tilde{\mathbf{a}}={\mu }_{\varphi }\left(\mathbf{s}\right)+{\sigma }_{\varphi }\left(\mathbf{s}\right)·ϵ$.

Temperature update.

$$J\left(\alpha \right)={\mathbb{E}}_{\mathbf{a}\sim {\pi }_{\varphi }}\left[-\alpha \left(log{\pi }_{\varphi }\left(\mathbf{a}\right|\mathbf{s})+{\mathcal{H}}_{\mathrm{target}}\right)\right].$$

(37)

3.4.3. Location Layer Training Pseudocode

The SAC-based training process used in this section follows a similar procedure to the algorithm presented in the previous section, with the key difference being the removal of the invalid action masking (IAM) mechanism. As this adjustment simplifies the structure, a detailed description is omitted here. The training of the upper-layer network is built upon the foundation of the lower-layer network. Before initiating the upper-layer training, a brief pre-training phase is conducted for the lower layer to provide a reasonable initialization. This pre-training does not require full convergence. Subsequently, during the training of the upper-layer policy, the lower-layer network continues to be updated in parallel. This joint optimization strategy enables the coordination between both layers to better achieve the overall task objective.

The training pseudocode is as Algorithm 2:

Algorithm 2 Location Optimization with SAC.

1: Initialize: $Q_{{\theta }_1},Q_{{\theta }_2}$, policy network ${\pi }_{\varphi }$ (parameterized by ${\mu }_{\varphi }$, ${\sigma }_{\varphi }$), value network $V_{\psi }$, temperature $\alpha$, replay buffer $\mathcal{D}$  2: for episode = 1 to max_episodes do  3:       Randomly initialize UAV and user positions, task demands  4:       Observe initial state ${\mathbf{s}}_0^{\mathrm{loc}}$  5:       for $t=0$ to $T-1$ do  6:             Sample action ${\mathbf{a}}_t=(V_{x,t},V_{y,t})$ from ${\pi }_{\varphi }\left({\mathbf{s}}_t\right)$ via reparameterization  7:             Update UAV position based on velocity  8:             Use pre-trained offloading model to compute latency  9:             Compute reward $r_t$, observe next state ${\mathbf{s}}_{t+1}$ 10:             Store transition $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1})$ in $\mathcal{D}$ 11:             Update Q-networks and policy via SAC update rules 12:       end for 13: end for

Through this hierarchical structure, the location optimization layer and the offloading decision layer are trained to focus on distinct sub-objectives (continuous UAV movement vs. discrete offloading decisions). In particular, the offloading layer with invalid action masking (IAM) ensures that only feasible user selections are considered, while the upper layer learns optimal positioning to support such selections. Simulation results in the next section demonstrate the performance gains achieved by this coordinated learning scheme.

4. Simulation Results

To evaluate the performance of the proposed hierarchical soft actor–critic framework (H-SAC) for multi-channel UAV-assisted MEC systems, a series of comparative experiments were conducted against baseline methods, including standard SAC and DDPG. This section presents the experimental setup, performance metrics, and a detailed comparison in terms of convergence, user latency, UAV trajectory planning, and task scheduling.

4.1. Experimental Setup

All simulations were implemented in a Python 3.6 environment using TensorFlow 1.15. The UAV operates within a square area of 100 m × 100 m, with N = 10 user devices randomly distributed. The UAV starts from position (10, 10). All relevant parameters are summarized in

Table 2. Simulation parameters.

Symbol	Value	Description
$t^{fly}$	$1\mathrm{s}$	Time UAV fly length
K	$\{2,4\}$	Number of parallel channels
$D_{max}$	$\{4.6,9.2\}\mathrm{Mbits}$	UAV memory limit
N	10	Number of UEs
s	1000	CPU cycles per bit
$l_{max}$	$100\mathrm{m}$	Area side length
H	$50\mathrm{m}$	UAV altitude
B	$1\mathrm{MHz}$	Channel bandwidth
$P_n$	$0.1\mathrm{W}$	UE transmit power
${\sigma }^2$	${10}^{-13}\mathrm{W}$	Noise power
$f_n^{\mathrm{UE}}$	$0.6\mathrm{GHz}$	UE CPU frequency
$f_m^{\mathrm{UAV}}$	$3\mathrm{GHz}$	UAV CPU frequency
$k_n$	5	Power exponent in CPU energy model
$g_0$	$-50\mathrm{dB}$	Reference channel gain
$v_{x,max}$	$4\mathrm{m}$	Max x-axis velocity
$v_{y,max}$	$4\mathrm{m}$	Max y-axis velocity
$v_n$	3	Number of tasks per UE
$p_h$	$100\mathrm{W}$	UAV hovering power
$e_{max}$	$500\mathrm{kJ}$	UAV battery capacity
T	8	Time slot length

.

In both the continuous SAC and discrete SAC implementations, the actor and critic networks are designed as multi-layer perceptrons (MLPs) with three fully connected hidden layers consisting of 256, 128, and 64 neurons, respectively. All hidden layers use the ReLU activation function.

For the continuous SAC module applied to UAV trajectory control, the actor network outputs the mean and log standard deviation of a Gaussian distribution, which are then passed through a Tanh function to produce bounded continuous actions. The critic network estimates the Q-value of continuous state–action pairs.

In the discrete SAC module used for offloading decision-making, the actor network outputs a categorical probability distribution over all candidate actions via a Softmax layer. The critic network outputs Q-values for each discrete action given the current state. To handle resource constraints and action feasibility, an invalid action masking (IAM) mechanism is integrated into both the policy and value learning processes, ensuring that only valid actions are considered during training.

Three algorithms are evaluated:

• H-SAC (Proposed): A hierarchical RL framework with a continuous SAC for trajectory planning and a discrete SAC with IAM for offloading.
• SAC: A unified single-layer SAC that jointly outputs movement (continuous) and offloading decisions (discretized).
• DDPG: A single-layer DDPG using continuous outputs for both movement and offloading.

4.2. Convergence Analysis

Figure 5 and Figure 6 show the reward convergence under two different parallel channel settings ($K=2$ and $K=4$).

When $K=2$, all algorithms exhibit similar convergence trends. H-SAC achieves a slightly higher final reward but the improvement is modest due to the limited parallelism. In contrast, for $K=4$, H-SAC demonstrates faster convergence and higher final rewards, showing a clear advantage in leveraging multi-channel capacity via coordinated trajectory and offloading decisions.

4.3. Latency Performance

Figure 7 and Figure 8 illustrate the average latency across training episodes under different channel settings.

With $K=2$, all methods converge to comparable latency levels, with H-SAC maintaining a slight edge. When $K=4$, H-SAC substantially outperforms the baselines, achieving significantly lower average latency due to improved resource utilization and conflict avoidance.

4.4. Trajectory and Offloading Comparison

To further highlight the behavioral differences between the two methods, Figure 9 and Figure 10 visualize the UAV’s trajectory and its offloading behavior after training convergence under $K=4$.

Each figure consists of a sequence of six snapshots selected from the simulation window spanning time slots $T=0$ to $T=24$, with one snapshot approximately every four time slots. Here, T denotes the index of the decision-making time slot. The UAV’s motion trajectory and the offloaded users at each time step are visualized to reflect the spatio-temporal decision pattern of the agents. (In the figures, the solid blue lines represent the UAV’s movement trajectory, while the dashed blue lines indicate the UAV’s user selection decisions. Red circles denote the positions of the users, and blue squares mark the UAV’s discrete movement points).

Trajectory Planning:

• H-SAC: The UAV rapidly navigates toward task-dense regions and adapts its flight direction strategically at each time slot. The resulting path is smooth and purposeful, minimizing unnecessary deviations.
• SAC: The UAV’s trajectory is less efficient, with more erratic and indirect movements. It takes longer to approach areas with high task demands, and the path exhibits more abrupt changes.

Offloading Efficiency:

• H-SAC: The UAV consistently offloads tasks to the maximum allowed number of users per slot (K), with no task conflicts. The IAM-enhanced decision layer ensures valid and non-redundant selections, maximizing channel utilization.
• SAC: Offloading behavior is occasionally suboptimal. Some users may be selected multiple times or channels may be left unused due to uncoordinated decisions, leading to resource underutilization and longer delays.

These results confirm that the hierarchical structure in H-SAC facilitates both more effective trajectory planning and more efficient multi-user offloading in high-concurrency scenarios.

5. Conclusions and Future Work

This paper proposes a parallel task offloading and trajectory optimization approach for multi-channel UAV-assisted mobile edge computing (MEC) systems. Focusing on the multi-channel task offloading and path planning problem in single-UAV MEC scenarios, a system model supporting concurrent task execution is developed (see Section 2), and a hierarchical reinforcement learning (HRL)-based optimization framework, namely H-SAC, is introduced (see Section 3).The overall optimization problem is decomposed into two sub-tasks: continuous action space (trajectory control) and discrete action space (task selection), which are optimized by a continuous soft actor–critic and a discrete SAC enhanced with invalid action masking (IAM), respectively. Extensive simulation results (see Section 4) validate that the proposed method achieves superior performance compared to conventional single-layer reinforcement learning algorithms, in terms of reducing user latency, improving resource utilization, and enhancing training stability.

Future research could further extend the applicability of the proposed H-SAC framework to more realistic and complex environments. For instance, incorporating user mobility, non-ideal wireless channels, and environmental disturbances would enhance the system’s adaptability to real-world deployment conditions. In terms of optimization methods, adopting multi-objective reinforcement learning strategies could enable joint optimization of trajectory planning and task offloading. Additionally, introducing higher-fidelity UAV energy consumption models would contribute to improving the realism of system modeling. In large-scale user scenarios, it is necessary to design scalable and efficient scheduling mechanisms to effectively handle hundreds of concurrent task requests.