Edge-Intelligence-Powered Joint Computation Offloading and Unmanned Aerial Vehicle Trajectory Optimization Strategy

Abstract

UAV-based air-ground integrated networks offer a significant benefit in terms of providing ubiquitous communications and computing services for Internet of Things (IoT) devices. With the empowerment of edge intelligence (EI) technology, they can efficiently deploy various intelligent IoT applications. However, the trajectory of UAVs can significantly affect the quality of service (QoS) and resource optimization decisions. Joint computation offloading and UAV trajectory optimization bring many challenges, including coupled decision variables, information uncertainty, and long-term queue delay constraints. Therefore, this paper introduces an air-ground integrated architecture with EI and proposes a TD3-based joint computation offloading and UAV trajectory optimization (TCOTO) algorithm. Specifically, we use the principle of the TD3 algorithm to transform the original problem into a cumulative reward maximization problem in deep reinforcement learning (DRL) to obtain the UAV trajectory and offloading strategy. Additionally, the Lyapunov framework is used to convert the original long-term optimization problem into a deterministic short-term time-slot problem to ensure the long-term stability of the UAV queue. Based on the simulation results, it can be concluded that our novel TD3-based algorithm effectively solves the joint computation offloading and UAV trajectory optimization problems. The proposed algorithm improves the performance of the system energy efficiency by 3.77%, 22.90%, and 67.62%, respectively, compared to the other three benchmark schemes.

1. Introduction

The booming deployment of new Internet of Things (IoT) applications, including virtual reality, face recognition, environmental monitoring, traffic control, and security management (as cited in [1,2,3]), has triggered an explosive growth of data traffic at the edge [4]. These computation-intensive applications impose stringent requirements on the computational and communication capabilities of IoT edge devices [5]. However, the gap between the limited computational resources of IoT devices and the escalating demands of computational tasks has resulted in a deterioration of quality of service (QoS), ultimately hindering the effective deployment and performance of IoT applications. In addition, real-time data processing tasks generated by massive machine terminals, pattern recognition, and decision recommendation require artificial intelligence (AI) technology as a powerful service paradigm to assist in solution. Therefore, a new edge intelligence (EI) [6] paradigm combining AI and mobile edge computing (MEC) technologies will be a pivotal force in promoting the evolution of 6G IoT networks. This paradigm can propel learning capabilities from remote centers to the fringes of the network.

With the support of EI technology, intelligent IoT devices can be deployed in more fields to expand vertical application scenarios, such as intelligent transportation, intelligent industry, intelligent healthcare, and intelligent agriculture [7]. These EI-powered IoT devices can realize efficient sensing, transmission, and processing of massive amounts of information, extract valuable information and features from them in order to achieve more accurate and more intelligent decision-making and control, and, thus, provide better and more efficient support for IoT applications and services [8]. However, terrestrial networks cannot achieve ultra-large-scale global connections and ubiquitous communications, primarily due to constraints posed by the radio spectrum resources, geographical coverage limitations, and operational costs [9]. Considering the demands for ultra-reliable and low-latency transmission in areas that lack terrestrial communication network coverage (such as remote mountainous and oceanic areas) and in the areas where network capacity is insufficient to support the dynamic and rapid increase of IoT services (hotspot areas or rural), airborne devices due to their inherent mobility and flexibility become a promising economic solution well suited in the above areas [10].

The airborne platforms include high-altitude platform stations (HAPs) and low-altitude unmanned aerial vehicles (UAVs), which are the key components of the air-ground integrated networks. UAVs equipped with MEC can take advantage of their high mobility and controllability, quickly flying near the target devices, thereby effectively expanding the coverage of the communication network. Despite the aforementioned advantages of UAVs, their limited computational resources and battery capacity pose significant challenges in sustaining long-term service provision to IoT devices. To meet this challenge, HAPs are introduced as a supplement for the limitations of UAVs [11]. HAPs can be deployed at high altitudes for a long time to provide stable communication services in larger areas. Consequently, the collaborative effort of UAVs and HAPs becomes imperative to deliver computing services efficiently. Based on the actual demands of computing tasks in different environments and scenarios, UAVs, HAPs, or fixed ground infrastructures flexibly support intelligent IoT applications via ground air links [12].

However, introducing UAVs/HAPs with EI technology brings new challenges in low-carbon sustainable resource optimization and device deployment optimization. In addition, the trajectory of UAVs can significantly affect the quality of service (QoS), exacerbating the difficulty of resource optimization. Hence, there is a pressing need to intensify research efforts focused on the joint computation offloading and UAVs’ trajectory optimization strategies within the context of EI-powered air-ground integrated networks. However, most of the current works [13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31] have some limitations, as shown in

Table 1. Comparison of different optimization algorithms.

	Air Network Considers HAP	Consider the Ground Network	Consider Long-Term Constraints	Optimization Using Traditional Methods	Optimization Using DRL Methods
[13]	-	-	-	-	✓
[14]	-	-	-	✓	-
[15]	-	✓	-	-	✓
[16,17]	-	-	-	-	✓
[18,19,20]	-	✓	-	-	✓
[22]	-	✓	-	-	✓
[23]	-	-	-	-	✓
[24]	-	✓	-	-	✓
[25]	-	-	-	-	✓
[26]	-	-	✓	-	✓
[27]	-	-	-	-	✓
[28]	-	✓	-	-	✓
[29,30]	-	-	-	-	✓
[31]	-	✓	-	-	✓

, and they can summarised as follows: (1) the proposed optimization strategies cannot solve the problems in the integrated air-ground architecture, (2) the system model does not take into account the long-term stability of the queue, and (3) convex optimization algorithms are not appropriate for rapidly changing environments.

In order to address the above challenges, the air-ground integrated architecture with EI is introduced. It combines MEC and AI technology to provide faster and more accurate acquisition, transmission, and massive data processing. Subsequently, we propose a joint optimization problem that can maintain the long-term stability of the UAV queueing system. By applying the Lyapunov framework, we transform complex long-term optimization problems into manageable deterministic short-term time-slot models. Ultimately, we devise an algorithm that concurrently addresses computation offloading and UAV trajectory planning. To tackle the original intractable problem, we adopt the cutting-edge TD3 DRL algorithm, recasting it into a framework that maximizes cumulative rewards. Strategically designing the reward mechanism facilitates the neural network’s evolution towards superior decision-making capabilities, ultimately yielding the optimal strategies. The core contributions of this study can be concisely outlined as follows:

• We first propose an integrated network architecture incorporating UAVs, HAPs, and terrestrial communication infrastructures to support collecting, transmitting, and processing extensive IoT data. This is achieved through the strategic and flexible deployment of UAVs with EI. Based on this architecture design, UAVs can work with HAPs and remote base stations (BSs) to simultaneously process computation tasks of intelligent IoT applications.
• We define the joint computation offloading and UAVs trajectory optimization problem as a stochastic mixed-integer nonlinear programming (MINLP) problem. Considering the undeniable impact of queuing delay, we take the long-term constraints of queuing delay into problem modeling. Then, we transform the MINLP problem into a computational offloading problem and a UAV trajectory optimization problem for each time slot by using the Lyapunov optimization theory.
• We propose the TD3-based joint computation offloading and UAV trajectory optimization (TCOTO) algorithm to address the above challenge. Specifically, we consider the high coupling of task offloading and trajectory optimization strategies and transform this optimization into an MDP-based reinforcement learning problem. The UAV autonomously makes trajectory and offloading decisions based on the observed dynamic environment, such as UAV position, queue backlog, and channel state. Then, TD3 generates continuous actions to avoid overestimating the Q-value and improve training stability.

The structure of the paper is organized as follows: Section 2 reviews the related work; Section 3 outlines the system model. Section 4 gives the problem formulation and transformations. Section 5 details the approach used to address the issue. Section 6 includes simulation and performance evaluation. Moreover, Section 7 concludes the paper.

2. Related Works

2.1. The Architecture of Networks

Both references [13,14] employed UAVs, incorporating MEC servers for service delivery to ground users. Meanwhile, the author addressed the optimization of system energy efficiency through jointly optimizing UAV trajectories and user offloading decisions. Papers [15,16,17,18] added multiple UAVs to handle intensive computational tasks in order to alleviate the computational pressure of a single UAV. Differently, paper [19] introduced charging stations as its energy supplement and then proposed a priority-based DRL method for trajectory planning and charging scheduling of UAVs to maximize their energy efficiency. Furthermore, ref. [20] also factored in BSs within the network infrastructure to serve users. It arranges multiple UAVs near BSs to tackle delay-critical tasks promptly. For delay-resilient tasks, UAVs serve as intermediaries, forwarding data to BSs to conserve their limited resources. However, the previously mentioned study confined its network to one or multiple UAVs focused on data gathering and local processing, excluding BSs or HAPs. However, considering the limited computing resources of UAVs, such a network architecture cannot meet the computational demands of massive data in the IoT. Although multiple UAVs can alleviate the computational pressure, they require additional deployment costs. To solve this problem, we propose an integrated network architecture that avoids the degradation of service performance of UAVs due to insufficient computational resources through the mutual collaboration between ground and sky networks.

2.2. The Long-Term Constraints of Queue

Considering the sensitivity of future emerging services to service quality, the long-term performance of network systems will have an undeniable impact on the Smart IoT [21]. However, most of the current studies predominantly focus on short-term system performance, e.g., the paper [22] provided different service features based on different task types of users. The authors proposed an optimally embedded DRL scheme to maximize energy efficiency and service fairness. The paper [23] combined deep deterministic policy gradient (DDPG) and dueling DQN (DDQN) algorithms to deal with discrete and continuous hybrid action spaces, which were used to solve the ground equipment offloading decision and flight trajectory, respectively. Also, the algorithms proposed in the paper optimize the time allocated to the ground equipment to enhance UAV energy efficiency. The paper [24] tackled service distribution, trajectory refinement, and obstacle circumvention in a multi-UAV supported offloading framework. The authors were able to minimize user offloading delays and improve UAV energy efficiency by combining DDPG with a potential game to enable UAVs to avoid obstacles. Since the arrival of computing tasks is uncertain, it is impractical for the above literature to consider only short-term performance optimization. Recently, some studies have considered long-term performance optimization. For example, the paper [25] used the DDPG algorithm to jointly optimize UAV trajectories and task offloading strategy, thereby minimizing edge computing network latency. The paper [26] introduced a triple-learner DRL algorithm to enhance energy efficiency by simultaneously optimizing UAV trajectory planning and energy scheduling. However, these studies did not consider the impact of long-term queues on the system, so this paper introduces queuing theory and models to model and analyze the long-term queues of the system. By analyzing the system’s queuing characteristics and performance indicators, intelligent agents can better understand the impact of queues on system performance, thereby obtaining better trajectory planning and unloading task decisions.

2.3. Optimization Technique

Unlike traditional optimization methods, DRL can handle decision-making problems in dynamic scenarios without prior information, significantly simplifying problem-solving. The paper [27] introduced a Q-learning approach to concurrently optimize UAV trajectories and offloading ratios, minimizing MEC system delays for IoT devices servicing. The paper [28] embedded cognitive radios into UAVs to form a cognitive UAV-assisted network and then established backhaul links by collecting idle spectrum. The authors proposed a DDQN-based solution for the trajectory optimization, time allocation, and frequency band selection problems. This enables cognitive UAVs to make the best decisions autonomously and improve traffic offloading efficiency. The Q-learning and DDQN algorithms in the above papers performed well in relatively simple environments, but are unsuitable for problems with large action spaces. The DDPG approach in [29] efficiently tackled high-dimensional action spaces in intricate environments, yet encountered an issue with Q-value overestimation. TD3, an advanced DDPG refinement, effectively addresses this challenge, demonstrating superior performance compared to DDPG, and has garnered widespread application across various research domains. For example, the paper [30] constructed a cloud-side cooperative MEC system aimed at delivering premium user services. Then, the authors proposed a TD3 optimization algorithm to minimize the objective. The paper [31] proposed a video secure offloading scheme based on TD3, which saves system energy while ensuring system latency and video frame analysis precision. Then, through a large number of experiments, it was proven that the algorithm proposed is superior to comparative algorithms such as DDPG in terms of energy consumption and latency. Considering the diverse nature and long-term constraints of the network architecture presented in this study, relying on algorithms like DDPG could result in suboptimal performance and slow convergence. Therefore, the TD3 algorithm is needed to solve such high-dimensional optimization problems.

3. System Model

As shown in Figure 1, we propose an integrated system consisting of massive IoT devices, a UAV, an HAP, and a BS. The IoT devices generate a large amount of data for environment sensing, traffic control, or safety management tasks. We denote IoT devices set as $\kappa =\{1,2,\dots ,k,\dots ,K\}$, where k is the k-th IoT device. The UAVs fly over IoT devices to collect and process data further with the MEC services equipped on them. Given the limitations in power and computing resources of UAVs, computation tasks involving collected sensing data can be offloaded to remote BSs and HAPs to achieve improved delay and power performance. The UAV’s mission duration T is evenly divided into N time slots, represented by the set $\mathcal{T}=\{1,\dots ,t,\dots ,N\}$. $t\in \mathcal{T}$ is the t-th time slot, and its duration length is $\tau$. We assume that the UAV maintains a static position throughout each time slot of duration $\tau$.

The location of the k-th IoT device is ${\mathbf{p}}_k=\{x_k^{\mathrm{S}},y_k^{\mathrm{S}}\}$, and the vertical axis coordinates are all zero. The UAV operates at a constant altitude $H^{\mathrm{UAV}}$ and maximum speed $V_{max}^{\mathrm{UAV}}$ to collect data generated at the IoT devices. The initial point of the UAV’s mission is ${\mathbf{q}}_0^{\mathrm{UAV}}$ and the destination of the mission is ${\mathbf{q}}_T^{\mathrm{UAV}}$. During time slot t, the horizontal coordinates are given by $\begin{array}{c}{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)=\left\{x^{\mathrm{UAV}}\left(t\right),y^{\mathrm{UAV}}\left(t\right)\right\}\end{array}$, and the flight direction ${\mathbf{d}}^{\mathrm{UAV}}\left(t\right)$ satisfies $\left|{\mathbf{d}}^{\mathrm{UAV}}\left(t\right)\right|=1$. Similar to [32], the system environment (${\mathbf{q}}^{\mathrm{UAV}}\left(t\right)$, ${\mathbf{d}}^{\mathrm{UAV}}\left(t\right)$, and channel state information (CSI)) remain stable in a time slot. The positions and flight directions of the UAV in different time slots are related as follows:

$${\mathbf{q}}^{\mathrm{UAV}}\left(t+1\right)-{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)=V^{\mathrm{UAV}}\left(t\right)\tau {\mathbf{d}}^{\mathrm{UAV}}\left(t\right),$$

(1)

where $V^{\mathrm{UAV}}$ denotes the the UAV’s speed.

3.1. Communications Model

3.1.1. Data Collection

During data collection from IoT devices by the UAV, communication involves both line-of-sight (LoS) and non-line-of-sight paths (NLoS). The horizontal distance between the device k and the UAV is calculated as

$$∥{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)-{\mathbf{p}}_k\left(t\right)∥=\sqrt{(x^{\mathrm{UAV}}\left(t\right)-x_k^{\mathrm{S}}{)}^2+{(y^{\mathrm{UAV}}\left(t\right)-y_k^{\mathrm{S}})}^2}.$$

(2)

Then, the path-loss can be calculated by the following equation:

$$L_k\left(t\right)=20{log}_{10}\left({\displaystyle \frac{4\pi f_c\sqrt{{\left(H^{\mathrm{UAV}}\right)}^2+{∥{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)-{\mathbf{p}}_k\left(t\right)∥}^2}}{\mathrm{c}}}\right)+p_{k,t}^{\mathrm{LoS}}{\eta }_{k,t}^{\mathrm{LoS}}+(1-p_{k,t}^{\mathrm{LoS}}){\eta }_{k,t}^{\mathrm{NLoS}},$$

(3)

where c denotes the speed of light, $∥\mathbf{m}∥$ is the Euclidean paradigm of vector $\mathbf{m}$, and $f_c$ represents the communication carrier frequency. ${\eta }_{k,t}^{\mathrm{LoS}}$ and ${\eta }_{k,t}^{\mathrm{NLoS}}$ indicate the extra losses for LoS and NLoS links due to free-space propagation, respectively. $P_{k,t}^{\mathrm{LoS}}$ represents the probability of the LoS connection, which is given by

$$p_{\mathrm{k},t}^{\mathrm{LoS}}={\displaystyle \frac{1}{1+{\varpi }_{1}exp\left\{-{\varpi }_2[arctan\left(\frac{H^{\mathrm{UAV}}}{∥{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)-{\mathbf{p}}_k\left(t\right)∥}\right)-{\varpi }_1]\right\}}},$$

(4)

where $({\varpi }_1,{\varpi }_2,{\eta }_{\mathrm{k},t}^{\mathrm{LoS}},{\eta }_{\mathrm{k},t}^{\mathrm{NLoS}})$ are selected according to the environment [33].

Data collection rate from the k-th Iot device to the UAV is calculated as

$$R_k\left(t\right)=b_k\left(t\right)B^{\mathrm{UAV}}{log}_2\left(1+{\displaystyle \frac{P_k{10}^{-L_{k,t}/10}}{{\sigma }^2}}\right).$$

(5)

We presume that the UAV employs the frequency division multiple access (FDMA) technique; $b_k\left(t\right)$ denotes the fraction of channel bandwidth assigned by the UAV to IoT device k, $B^{\mathrm{UAV}}$ is the total bandwidth available to be allocated by the UAV, ${\sigma }^2$ is the power of the noise, and $P_k$ is the transmitting power of the edge device.

3.1.2. Computation Offloading to HAP

Due to the advantage of airborne platforms in high altitudes, the link between the UAV and the HAP is considered as LoS propagation, and its channel gain can be expressed as

$$h^{\mathrm{HAP}}={\displaystyle \frac{h_0}{{∥{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)-{\mathbf{p}}^{\mathrm{HAP}}∥}^2+{(H^{\mathrm{HAP}}-H^{\mathrm{UAV}})}^2}},$$

(6)

where $h_0$ represents the channel gain of LoS propagation at a reference distance of $d_0=1\mathrm{m}$, and ${\mathbf{p}}^{\mathrm{HAP}}$ is the horizontal position of the HAP, which is considered to be fixed in the air at a height of $H^{\mathrm{HAP}}$.

Then, the data rate of the communication link between the UAV and the HAP is formulated as

$$R^{\mathrm{HAP}}\left(t\right)=B^{\mathrm{HAP}}log\left(1+{\displaystyle \frac{h^{\mathrm{HAP}}{P}_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)}{{\sigma }^2}}\right),$$

(7)

where $B^{\mathrm{HAP}}$ and $P_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)$ denote the bandwidth and power used in computational offloading to the HAP, respectively. The amount of computational offloading from the UAV to the HAP can be calculated by $r^{\mathrm{HAP}}\left(t\right)=\tau R^{\mathrm{HAP}}\left(t\right)$.

3.1.3. Computation Offloading to BS

Similarly, the channel gain to the BS can be expressed as

$$h^{\mathrm{BS}}={\displaystyle \frac{h_0}{|{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)-{\mathbf{p}}^{\mathrm{BS}}{|}^2+{(H^{\mathrm{BS}}-H^{\mathrm{UAV}})}^2}},$$

(8)

where ${\mathbf{p}}^{\mathrm{BS}}$ is the coordinates of the BS on the ground and $H^{\mathrm{BS}}$ is the height of the BS.

Therefore, the communication rate between ground BS is given by

$$R^{\mathrm{BS}}\left(t\right)=B^{\mathrm{BS}}log\left(1+{\displaystyle \frac{h^{\mathrm{BS}}{P}_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)}{{\sigma }^2}}\right),$$

(9)

where $B^{\mathrm{BS}}$ and $P_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)$ denote the bandwidth and power used in computational offloading to the BS, respectively. The computational offloading to the BS can be calculated as $r^{\mathrm{BS}}\left(t\right)=\tau R^{\mathrm{BS}}\left(t\right)$.

According to the above model, the amount of computational offloading $r\left(t\right)$ by the UAV can be determined. $r\left(t\right)$ can be calculated as

$$r\left(t\right)=x\left(t\right)·r^{\mathrm{HAP}}\left(t\right)+(1-x\left(t\right))·r^{\mathrm{BS}}\left(t\right),$$

(10)

where $x\left(t\right)\in \left\{0,1\right\}$ is the offloading decision of the UAV, and when $x\left(t\right)=1$, it means that the UAV is offloading the computational tasks to the HAP, and vice versa, to the BS.

3.2. Queue Model

The UAV has a large buffer queue for storing collected data. The queue backlog $Q^{\mathrm{UAV}}\left(t\right)$ is calculated by

$$Q^{\mathrm{UAV}}\left(t+1\right)=max\left\{Q^{\mathrm{UAV}}\left(t\right)-Z^{\mathrm{UAV}}\left(t\right),0\right\}+A^{\mathrm{UAV}}\left(t\right),$$

(11)

where $A^{\mathrm{UAV}}\left(t\right)={\sum }_{k=0}^K{R}_k\left(t\right)\tau$ denotes the data collected by the UAV and can be processed from the next time slot. Itsignifies the overall volume of data exiting the UAV’s buffer queue. $l^{\mathrm{UAV}}\left(t\right)$ is the amount of data processed by the UAV, and it is calculated by $l^{\mathrm{UAV}}\left(t\right)=\tau f^{\mathrm{UAV}}\left(t\right)/L^{\mathrm{UAV}}$. $f^{\mathrm{UAV}}\left(t\right)$ represents the frequency of the CPU used by the UAV for local computation, and $L^{\mathrm{UAV}}$ is the computational density of the UAV, which is the number of CPU cycles needed to process a single bit.

The computation resource capacities and abilities of the BS and HAP are much more larger than that of UAVs; thereby, the data can be processed in time without causing data backlogs. The CPU frequency of the HAP is calculated by $f^{\mathrm{HAP}}\left(t\right)=r^{\mathrm{HAP}}\left(t\right)·L^{\mathrm{HAP}}/\tau$, where $L^{\mathrm{HAP}}$ denotes the computational density of HAP. Similarly, the CPU frequency of the BS is calculated by $f^{\mathrm{BS}}\left(t\right)=r^{\mathrm{BS}}\left(t\right)·L^{\mathrm{BS}}/\tau$, where $L^{\mathrm{BS}}$ denotes the computational density of the BS.

3.3. Energy Consumption Model

The total energy consumption $E_{\mathrm{total}}\left(t\right)$ accounts for the energy costs of the UAV, HAP, and BS. Consequently, $E_{\mathrm{total}}\left(t\right)$ can be expressed as

$$E_{\mathrm{total}}\left(t\right)=E_{\mathrm{UAV}}\left(t\right)+E_{\mathrm{HAP}}\left(t\right)+E_{\mathrm{BS}}\left(t\right).$$

(12)

3.3.1. UAV Energy Consumption

The total UAV energy consumption is calculated as

$$E_{\mathrm{UAV}}\left(t\right)=E_{\mathrm{com}}^{\mathrm{UAV}}\left(t\right)+E_{\mathrm{off}}^{\mathrm{UAV}}\left(t\right)+E_{\mathrm{fly}}^{\mathrm{UAV}}\left(t\right),$$

(13)

where the energy of processing data on the UAV is $E_{\mathrm{com}}^{\mathrm{UAV}}\left(t\right)=P_{\mathrm{com}}^{\mathrm{UAV}}\left(t\right)\tau$. According to the circuit theory, $P_{\mathrm{com}}^{\mathrm{UAV}}\left(t\right)={\kappa }^{\mathrm{UAV}}{\left(f^{\mathrm{UAV}}\left(t\right)\right)}^3$, ${\kappa }^{\mathrm{UAV}}$ is the size of the effective switching capacitance of the CPU.

The energy consumption of the data is offloaded to the HAP or BS and can be obtained by $E_{\mathrm{off}}^{\mathrm{UAV}}\left(t\right)=x\left(t\right)·E_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)+(1-x\left(t\right))·E_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)$, where the transmission energy required for offloading computational tasks to the HAP is $E_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)=P_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)\tau$, and to the BS is $E_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)=P_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)\tau$.

The flight energy of UAV movement is calculated according to the equation $E_{\mathrm{fly}}^{\mathrm{UAV}}\left(t\right)=P_{\mathrm{fly}}^{\mathrm{UAV}}\left(t\right)\tau$. Similarly to [34,35,36], $P_{\mathrm{fly}}^{\mathrm{UAV}}\left(t\right)$ is the propulsion power and its calculation formula is

$$\begin{array}{cc}\hfill P_{\mathrm{fly}}^{\mathrm{UAV}}\left(V^{\mathrm{UAV}}\left(t\right)\right)\left(t\right)& ={\mathrm{C}}_1(1+3{\displaystyle \frac{{\left(V^{\mathrm{UAV}}\left(t\right)\right)}^2}{{\left(V_{\mathrm{tip}}\right)}^2}})\hfill \\ \hfill & +{\mathrm{C}}_2\sqrt{\sqrt{{\mathrm{C}}_3+{\displaystyle \frac{{\left(V^{\mathrm{UAV}}\left(t\right)\right)}^4}{4}}-{\displaystyle \frac{{\left(V^{\mathrm{UAV}}\left(t\right)\right)}^2}{2}}}+{\mathrm{C}}_4{\left(V^{\mathrm{UAV}}\left(t\right)\right)}^3},\hfill \end{array}$$

(14)

where $V^{\mathrm{UAV}}\left(t\right)$ is the UAV’s velocity, $V_{\mathrm{tip}}$ denotes the tip velocity of the rotor blade, and ${\mathrm{C}}_1$, ${\mathrm{C}}_2$, ${\mathrm{C}}_3$, and ${\mathrm{C}}_4$ are constants depending on the weight of the UAV and its aerodynamic parameters.

3.3.2. HAP Energy Consumption

The HAP only considers the energy consumption $E_{\mathrm{com}}^{\mathrm{HAP}}\left(t\right)$ for computing data, and it is calculated by $E_{\mathrm{com}}^{\mathrm{HAP}}\left(t\right)=P_{\mathrm{com}}^{\mathrm{HAP}}\left(t\right)\tau$. The computational power of HAP is given as

$$P_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right)={\kappa }^{\mathrm{HAP}}{\left(f^{\mathrm{HAP}}\left(t\right)\right)}^3,$$

(15)

where ${\kappa }^{\mathrm{HAP}}$ is the capacitance at the HAP, and $f^{\mathrm{HAP}}\left(t\right)$ denotes the CPU frequency of the HAP used in processing the offloaded tasks, respectively.

3.3.3. BS Energy Consumption

Similar to the HAP, the energy consumption of the BS is calculated by $E_{\mathrm{com}}^{\mathrm{BS}}\left(t\right)=P_{\mathrm{com}}^{\mathrm{BS}}\left(t\right)\tau$. The computational power of BS is given as

$$P_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)={\kappa }^{\mathrm{BS}}{\left(f^{\mathrm{BS}}\left(t\right)\right)}^3,$$

(16)

where ${\kappa }^{\mathrm{BS}}$ is the capacitance at the BS, and $f^{\mathrm{BS}}\left(t\right)$ denotes the CPU frequency of the BS used in processing the offloaded tasks, respectively.

4. Problem Formulation and Transformation

4.1. Problem Formulation

Considering the demands for ultra-reliable communication and seamless coverage for IoT applications in the areas of lack of terrestrial communication network coverage or network capacity is insufficient, the integrated network is proposed in this paper to meet the above challenge. However, the limited computation resources and battery capacity of airborne devices pose a significant challenge in providing services to IoT devices for extended periods. Therefore, the objective is to optimize the system’s long-term average energy efficiency by comprehensively considering bandwidth allocation for UAV data collection, UAV trajectory planning, offloading decisions, and computational resource allocation in constraints of queue stability, communication, and computation resources. The data processed consist of both the data handled locally by the UAV and the data offloaded to the HAP or BS, i.e., $C_{\mathrm{total}}\left(t\right)=l^{\mathrm{UAV}}\left(t\right)+r\left(t\right)$.

The optimization issue $\mathbf{P}\mathbf{1}$ is defined as follows:

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{1}:& \underset{\mathbf{Z}}{max}\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=1}^T{\displaystyle \frac{{\omega }_c{C}_{\mathrm{total}}\left(t\right)}{{\omega }_e{E}_{\mathrm{total}}\left(t\right)}},\hfill \\ \hfill \mathrm{s}.\mathrm{t}.C_1:& 0\le {\mathrm{b}}_k\left(t\right)\le 1\forall k,t,\hfill \\ \hfill C_2:& \sum _{k=1}^K{\mathrm{b}}_k\left(t\right)\le 1\forall t,\hfill \\ \hfill C_3:& {\mathbf{q}}^{\mathrm{UAV}}\left(0\right)={\mathbf{q}}_0^{\mathrm{UAV}},\hfill \\ \hfill C_4:& {\mathbf{q}}^{\mathrm{UAV}}\left(T\right)={\mathbf{q}}_T^{\mathrm{UAV}},\hfill \\ \hfill C_5:& c_d\le {\mathbf{q}}^{\mathrm{UAV}}\left(t\right)\le c_u\forall t,\hfill \\ \hfill C_6:& {\mathbf{q}}^{\mathrm{UAV}}(t+1)-{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)\le V_{max}^{\mathrm{UAV}}{\mathbf{d}}^{\mathrm{UAV}}\left(t\right)\tau \forall t,\hfill \\ \hfill C_7:& f_t^{\mathrm{UAV}}\le f_{max}^{\mathrm{UAV}},\hfill \\ \hfill C_8:& P_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right),P_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)\le P_{\max }^{\mathrm{off}},\hfill \\ \hfill C_9:& \underset{\mathrm{t}\to \infty }{lim}{\displaystyle \frac{\left[Q^{\mathrm{UAV}}\left(t\right)\right]}{t}}=0,\hfill \end{array}$$

(17)

where $\mathbf{Z}=\{b_k\left(t\right),{\mathbf{q}}^{\mathrm{UAV}}\left(t\right),x\left(t\right),r\left(t\right),f^{\mathrm{UAV}}\left(t\right)\}$ denotes the optimization variables. $C_1$ and $C_2$ are constraints on the bandwidth allocation ratio, ensuring that the total bandwidth used for data collection does not exceed the total bandwidth $B^{\mathrm{UAV}}$. $C_3$ and $C_4$ give the initial point and final point of the task. $C_5$ and $C_6$ give the boundaries and maximum speed of the UAV trajectory, where $c_d$ and $c_u$ are the upper and lower boundaries, respectively. $C_7$ and $C_8$ denote the UAV’s computation frequency and transmit power constraints. $C_9$ means that the UAV keeps the task queue steady by ensuring that all arriving computation tasks are completed in a finite time. ${\omega }_c$ and ${\omega }_e$ are the processing weight and energy consumption weight, respectively, indicating whether the UAV favors processing more data or saving more energy in completing the task.

4.2. Problem Transformation by Lyapunov Optimization

Lyapunov optimization involves modeling a queue to address constraints within stochastic optimization problems. This approach decouples the problem from the time dimension by breaking down the operations necessary to satisfy long-term constraints into individual time slices. Furthermore, the queue’s size is factored into the algorithm. By doing so, the system can achieve a stable state, facilitating the optimization objective [37]. Consequently, Lyapunov optimization theory can be used to convert problem $\mathbf{P}\mathbf{1}$. Firstly, the Lyapunov function is defined as

$$L\left(Q^{\mathrm{UAV}}\left(t\right)\right)={\displaystyle \frac{1}{2}}{\left(Q^{\mathrm{UAV}}\left(t\right)\right)}^2.$$

(18)

Then, the Lyapunov drift is characterized by the expected change in the value of $\Delta \mathrm{L}\left(Q^{\mathrm{UAV}}\left(t\right)\right)$:

$$\Delta L\left(Q^{\mathrm{UAV}}\left(t\right)\right)=\mathbb{E}\left[L\left(Q^{\mathrm{UAV}}\left(t+1\right)\right)-L\left(Q^{\mathrm{UAV}}\left(t\right)\right)|Q^{\mathrm{UAV}}\left(t\right)\right].$$

(19)

To optimize energy efficiency while maintaining queue stability, the Lyapunov drift with penalty is given by the sum of the expected Lyapunov drift and the average energy efficiency weighted by the parameter V. Thus, the Lyapunov drift plus penalty function is

$$\Lambda L\left(Q^{\mathrm{UAV}}\left(t\right)\right)\triangleq \Delta \mathrm{L}\left(Q^{\mathrm{UAV}}\left(t\right)\right)-V\mathbb{E}\left[\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=1}^T{\displaystyle \frac{{\omega }_c{C}_{\mathrm{total}}\left(t\right)}{{\omega }_e{E}_{\mathrm{total}}\left(t\right)}}|Q^{\mathrm{UAV}}\left(t\right)\right],$$

(20)

where $V>0$ is the weight coefficient for balancing the stability of the mission queue and the optimization objective. Similar to [38], with the rise of V, the system’s constraint on the queue length becomes less, and, hence, the queue backlog will increase.

The UAV queue can be obtained by

$$\begin{array}{cc}\hfill {\left(Q^{\mathrm{UAV}}(t+1)\right)}^2& ={\left(\max \{Q^{\mathrm{UAV}}\left(t\right)-Z^{\mathrm{UAV}}\left(t\right)\}\right)}^2+A^{\mathrm{UAV}}{\left(t\right)}^2\hfill \\ \hfill & +2A^{\mathrm{UAV}}{\left(t\right)}^2\max \{Q^{\mathrm{UAV}}\left(t\right)-Z^{\mathrm{UAV}}\left(t\right)\}\hfill \\ \hfill & \le {\left(Q^{\mathrm{UAV}}\left(t\right)-Z^{\mathrm{UAV}}\left(t\right)\right)}^2+A^{\mathrm{UAV}}{\left(t\right)}^2+2A^{\mathrm{UAV}}\left(t\right)Q^{\mathrm{UAV}}\left(t\right)\hfill \\ \hfill & =Q^{\mathrm{UAV}}{\left(t\right)}^2+A^{\mathrm{UAV}}{\left(t\right)}^2+Z^{\mathrm{UAV}}{\left(t\right)}^2\hfill \\ \hfill & -2Q^{\mathrm{UAV}}\left(t\right)\left(Z^{\mathrm{UAV}}\left(t\right)-A^{\mathrm{UAV}}\left(t\right)\right),\hfill \end{array}$$

(21)

which is transformed to give

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{2}}\left[{\left(Q^{\mathrm{UAV}}(t+1)\right)}^2-Q^{UAV}{\left(t\right)}^2\right]& \le {\displaystyle \frac{1}{2}}[Z^{\mathrm{UAV}}{\left(t\right)}^2+A^{\mathrm{UAV}}{\left(t\right)}^2]\hfill \\ \hfill & -Q^{\mathrm{UAV}}\left(t\right)\left(Z^{UAV}\left(t\right)-A^{\mathrm{UAV}}\left(t\right)\right).\hfill \end{array}$$

(22)

Based on the above equation, we can obtain

$$\begin{array}{cccc}\hfill & \Lambda L(Q^{\mathrm{UAV}}\left(t)\right)\hfill & \hfill & \le {\displaystyle \frac{1}{2}}[Z^{\mathrm{UAV}}{\left(t\right)}^2+Q^{\mathrm{UAV}}{\left(t\right)}^2]\hfill \\ & \hfill & & -Q^{\mathrm{UAV}}\left(t\right)\left(Z^{\mathrm{UAV}}\left(t\right)-A^{\mathrm{UAV}}\left(t\right)\right)\hfill \\ & \hfill & & -V\mathbb{E}\left[\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=1}^T{\displaystyle \frac{{\omega }_c{C}_{\mathrm{total}}\left(t\right)}{{\omega }_e{E}_{\mathrm{total}}\left(t\right)}}|Q^{\mathrm{UAV}}\left(t\right)\right]\hfill \\ & \hfill & & \le C-Q^{\mathrm{UAV}}\left(t\right)\left(Z^{\mathrm{UAV}}\left(t\right)-A^{\mathrm{UAV}}\left(t\right)\right)\hfill \\ & \hfill & & -V\mathbb{E}\left[\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=1}^T{\displaystyle \frac{{\omega }_c{C}_{\mathrm{total}}\left(t\right)}{{\omega }_e{E}_{\mathrm{total}}\left(t\right)}}|Q^{\mathrm{UAV}}\left(t\right)\right],\hfill \end{array}$$

(23)

where C is a positive constant.

The deterministic problem formulation for each time slot is expressed as

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{2}:& \underset{\mathbf{b}\left(t\right),\mathbf{q}\left(t\right),\mathbf{r}\left(t\right),\mathbf{f}\left(t\right)}{max}F\left({\mathbf{X}}_t\right),\hfill \\ \hfill \mathrm{s}.\mathrm{t}.C_1:& 0\le {\mathrm{b}}_k\left(t\right)\le 1\forall k,t,\hfill \\ \hfill C_2:& \sum _{k=1}^K{\mathrm{b}}_k\left(t\right)\le 1\forall t,\hfill \\ \hfill C_3:& {\mathbf{q}}^{\mathrm{UAV}}\left(0\right)={\mathbf{q}}_0^{\mathrm{UAV}},\hfill \\ \hfill C_4:& {\mathbf{q}}^{\mathrm{UAV}}\left(T\right)={\mathbf{q}}_T^{\mathrm{UAV}},\hfill \\ \hfill C_5:& c_d\le {\mathbf{q}}^{\mathrm{UAV}}\left(t\right)\le c_u\forall t,\hfill \\ \hfill C_6:& {\mathbf{q}}^{\mathrm{UAV}}(t+1)-{\mathbf{q}}^{\mathrm{UAV}}\left(t\right)\le V_{max}^{\mathrm{UAV}}{\mathbf{d}}^{\mathrm{UAV}}\left(t\right)\tau \forall t,\hfill \\ \hfill C_7:& f{\left(t\right)}^{\mathrm{UAV}}\le f_{\max }^{\mathrm{UAV}},\hfill \\ \hfill C_8:& P_{\mathrm{off}}^{\mathrm{HAP}}\left(t\right),P_{\mathrm{off}}^{\mathrm{BS}}\left(t\right)\le P_{\max }^{\mathrm{off}},\hfill \end{array}$$

(24)

where the constraints $C_1$–$C_8$ are the same as in $\mathbf{P}\mathbf{1}$. The objective function is expressed as

$$F\left({\mathbf{X}}_t\right)=Q^{\mathrm{UAV}}\left(t\right)[Z_k\left(t\right)-\tau \sum _{k=0}^K{R}_k\left(t\right)]+V\left({\displaystyle \frac{{\omega }_c{C}_{\mathrm{total}}\left(t\right)}{{\omega }_e{E}_{\mathrm{total}}\left(t\right)}}\right)$$

(25)

To solve $\mathbf{P}\mathbf{2}$, the approach relies solely on the current queue status, without requiring any knowledge of future task arrivals or wireless channel conditions. This characteristic qualifies it as an online optimization method. The following section will present a TD3-based online optimization algorithm designed to efficiently address $\mathbf{P}\mathbf{2}$.

5. TD3-Based Joint Computation Offloading and UAV Trajectory Optimization Algorithm

The problem $\mathbf{P}\mathbf{2}$, derived from Lyapunov optimization, is inherently NP-hard, featuring high dimensionality, nonlinearity, and mixed integers, which poses challenges for traditional optimization methods to solve. Additionally, the fluctuating wireless communication environment and resources contribute to its dynamic nature, which renders it particularly suitable for resolution using DRL methods [31]. To solve this NP-hard problem, we propose an algorithm based on TD3. TD3 is an enhanced version of the DDPG algorithm that alleviates the issue of overestimating the Q-function of DDPG on the Q-value, thereby achieving improved performance [39]. To tackle the issue of bias overestimation, TD3 incorporates three essential techniques: (1) clipped double Q-learning, (2) target policy smoothing regularization, and (3) delayed policy updates [40]. The following section offers a brief overview of the TD3 algorithm framework in relation to these techniques.

5.1. TD3 Framework

The TD3 algorithm comprises six DNNs: an Actor policy network with parameter $\varphi$, two Critic value networks with parameters ${\theta }_1$ and ${\theta }_2$, along with their corresponding Target Actor target policy network with parameter $\varphi \prime$, and Target Critic target value networks with parameters ${\theta }_1^{\prime }$ and ${\theta }_2^{\prime }$ [41]. Refer to Figure 2 for an illustration.

5.1.1. Experience Generation

Based on the current state $\mathbf{s}\left(t\right)$, the agent derives actions $\mathbf{a}\left(t\right)$ through the policy network ${\pi }_{\varphi }$. These actions can be expressed as

$$\mathbf{a}\left(t\right)={\pi }_{\varphi }\left(\mathbf{s}\left(t\right)\right)+\psi ,\psi \sim clip(\mathcal{C}\mathcal{N}(0,\epsilon ),-c,c),$$

(26)

where $\psi$ denotes a random Gaussian noise with a mean of zero and variance $\epsilon$. $clip(•)$ denotes the boundary trimming function, which limits the noise value to a specific range. The added noise helps to avoid falling into local optima and improves exploration efficiency [42]. Initially, a higher variance $\epsilon$ is employed to capture a broader array of experience samples. Over time, this variance is systematically decreased to facilitate stable convergence of the algorithm [43]. The agent engages with the environment by executing the noisy action $\mathbf{a}\left(t\right)$ and subsequently receives the reward $R\left(t\right)$ and observes the next state $\mathbf{s}(t+1))$. This interaction forms an an experience tuple $(\mathbf{s}\left(t\right),\mathbf{a}\left(t\right),R(t),\mathbf{s}(t+1))$, serving as the training dataset for the online networks. The experience replay mechanism is a vital component of reinforcement learning, as it aids in breaking the correlation between experience data and enhances the algorithm’s convergence performance [44].

5.1.2. Training and Updating of Critic Network

When the replay buffer $\mathbb{R}$ exceeds its capacity, a small random batch of tuples $\mathbb{I}$ is sampled for training the online network. The two critic networks are updated by minimizing the temporal difference (TD) error [45]. For each tuple in the sampled batch, the Q-values are approximated by the critic networks as $Q_{{\theta }_1}(\mathbf{s}\left(t\right),\mathbf{a}(t+1))$ and $Q_{{\theta }_2}(\mathbf{s}\left(t\right),\mathbf{a}(t+1))$, respectively. The target Q-value is

$$y_i=R\left(t\right)+\gamma min\left\{Q_{{\theta }_{\mathrm{i}}^{\prime }}(\mathbf{s}\left(t+1\right),\mathbf{a}\left(t+1\right)){|}_{\mathbf{a}\left(t+1\right)={\pi }_{\varphi \prime }\left(\mathbf{s}\left(t+1\right)\right)+\psi }\right\},i\in \{1,2\},$$

(27)

where $\gamma \in [0,1]$ is the discount factor. To address the Q-value overestimation issue, the smallest Q-value from the two “twin” Critic networks is selected and used to update the target Q-value. This approach helps mitigate the problem of Q-value overestimation [46].

Based on the estimated and target Q-values, the loss functions of the two evaluated Critic networks are expressed as

$$L\left({\theta }_i\right)\approx {\displaystyle \frac{1}{\left|\left.\mathbb{I}\right|\right.}}\sum _{j\in \mathbb{I}}{(y_j-Q_{{\theta }_i}({\mathbf{s}}_j,{\mathbf{a}}_j))}^2,i\in \{1,2\}.$$

(28)

Then, the policy gradient of the loss function is expressed as

$${\nabla }_{{\theta }_{\mathrm{i}}}L\left({\theta }_i\right)=-{\displaystyle \frac{2}{\mathbb{I}}}\sum _{j\in \mathbb{I}}(y_j-Q_{{\theta }_i}({\mathbf{s}}_j,{\mathbf{a}}_j))\times {\nabla }_{{\theta }_i}{Q}_{{\theta }_i}({\mathbf{s}}_j,{\mathbf{a}}_j),\forall i=1,2.$$

(29)

Lastly, we can update the parameter ${\theta }_i$ as follows:

$${\theta }_i\leftarrow {\theta }_i-{\eta }_{\theta }{\nabla }_{{\theta }_i}L\left({\theta }_i\right),\forall i=1,2,$$

(30)

where ${\eta }_{\theta }$ is the learning rate.

5.1.3. Training and Updating of Actor Network

The agent’s goal is to optimize the total rewards, while the Q-value measures the expected long-term rewards for taking specific actions in the current state. Throughout the entire training process, the Actor network aims to generate actions $\mathbf{a}\left(t\right)$ that maximize these Q-values, ultimately maximizing the cumulative reward. Consequently, the loss functions are expressed as

$$J\left(\varphi \right)=\mathbf{E}\left[Q_{{\theta }_i}(\mathbf{s}\left(t\right),\mathbf{a}\left(t\right))\right]{|}_{\mathbf{a}\left(t\right)={\pi }_{\varphi }\left(\mathbf{s}\left(t\right)\right)}.$$

(31)

The parameter $\varphi$ is updated via a deterministic policy gradient, which is denoted by the following formula:

$${\nabla }_{\varphi }J\left(\varphi \right)={\displaystyle \frac{1}{\left|\left.\mathbb{I}\right|\right.}}\sum _{j\in \mathbb{I}}\left[{\nabla }_{\varphi }{\pi }_{\varphi }\left({\mathbf{s}}_j\right){\nabla }_{{\mathbf{a}}_j}{Q}_{{\theta }_i}({\mathbf{s}}_j,{\mathbf{a}}_j){|}_{{\mathbf{a}}_j={\pi }_{\varphi }\left({\mathbf{s}}_j\right)}\right].$$

(32)

TD3 uses the strategy of delayed Actor network update, which updates after a fixed step size. Therefore, the equation for the parameter $\varphi$ update is as follows:

$$\varphi \leftarrow \varphi +{\eta }_{\varphi }{\nabla }_{\varphi }J\left(\varphi \right),$$

(33)

where ${\eta }_{\varphi }$ is the learning rate.

5.1.4. Updating of Target Network

Finally, the target Actor and Critic networks are updated utilizing the soft update method, and are expressed as follows:

$${\theta }_i^{\prime }\leftarrow {\epsilon }_{\theta }{\theta }_i+(1-{\epsilon }_{\theta }){\theta }_i^{\prime },i\in 1,2,$$

(34)

$${\varphi }_i^{\prime }\leftarrow {\epsilon }_{\varphi }\varphi +(1-{\epsilon }_{\varphi }){\varphi }_i^{\prime },$$

(35)

where ${\epsilon }_{\theta }$ and ${\epsilon }_{\varphi }$ are the soft update rates.

5.2. TD3-Based Joint Computation Offloading and UAV Trajectory Optimization Algorithm Design

In order to efficiently address the problem $\mathbf{P}\mathbf{2}$, we introduce the TCOTO algorithm, which leverages the advanced DRL model TD3 for problem-solving. During the training process, the UAV functions as an agent, with the TCOTO network and experience buffer implemented on the UAV’s edge server. The buffer gathers and stores information about the agent’s interactions with the environment. Using this historical data, the TCOTO network refines its strategies to guide the UAV in making optimal decisions. Once the Actor network has been thoroughly trained, it can directly provide strategies for computation offloading and UAV trajectory optimization based on the current environmental conditions, continuing until the task is finished. Consequently, the basic elements of TD3 will be discussed in detail in the following sections.

5.2.1. State Space

The state space encompasses all potential states. In our proposed architecture, the state space can be represented as

$$\mathbf{s}\left(t\right)=({\mathbf{q}}^{\mathrm{UAV}}\left(t\right),{\mathbf{p}}_1,...,{\mathbf{p}}_k,Q^{\mathrm{UAV}}\left(t\right)),$$

(36)

where ${\mathbf{q}}^{\mathrm{UAV}}\left(t\right)$ denotes the 2D coordinates of the UAV in a given area, ${\mathbf{p}}_k$ denotes the 2D coordinates of the IoT devices, and $Q^{\mathrm{UAV}}\left(t\right)$ is the length of the UAV buffer queue, respectively.

5.2.2. Action

In order to solve the computational offloading and UAV trajectory optimization problems, the agent’s actions are defined as follows:

$$\mathbf{a}\left(t\right)=\left(b_k\left(t\right),V^{\mathrm{UAV}}\left(t\right),{\mathbf{d}}^{\mathrm{UAV}}\left(t\right),x\left(t\right),R\left(t\right),f^{\mathrm{UAV}}\left(t\right)\right),$$

(37)

where $b_k\left(t\right)\in [0,1],k\in \kappa$ represents the portion of bandwidth assigned to the UAV for gathering sensor data; $V^{\mathrm{UAV}}\left(t\right)\in \left[0,V_{max}^{\mathrm{UAV}}\right]$ is the UAV’s speed and ${\mathbf{d}}^{\mathrm{UAV}}\left(t\right)\in [0,2\pi ]$ is the angle of flight, which determine the UAV’s position at the next time ${\mathbf{q}}^{\mathrm{UAV}}(t+1)$; $x\left(t\right)$ is the offloading decision of the UAV; $R\left(t\right)$ is the amount of data the UAV offloads to the HAP or BS; $f^{\mathrm{UAV}}\left(t\right)\in \left[0,f_{max}^{\mathrm{UAV}}\right]$ is the CPU frequency used by the UAV to process data. Since the values in the action space are continuous variables, the joint problem is a continuous control problem.

5.2.3. Action

At each time slot, upon executing actions $\mathbf{a}\left(t\right)$, the agent receives an instantaneous reward from the environment. Since the goal of this paper is to optimize long-term energy efficiency, the reward function can be expressed as

$$R\left(t\right)=F\left({\mathbf{X}}_t\right)$$

(38)

In addition to satisfying the objective function, the reward needs to satisfy some additional reward and penalty mechanisms to meet the constraints:

• Penalty for UAV flight trajectory outside of boundaries: UAVs may fly out of the set boundary ranges while performing computation and communication tasks. Since the IoT devices are within the boundary range, UAVs out of boundaries will waste resources. To alleviate this problem, we have designed a penalty mechanism that defines a penalty function based on the distance the UAV exceeds the boundary, which can be expressed as

  $$R\left(t\right)=R\left(t\right)-{\omega }_f·\sqrt{{\left(x_f^{\mathrm{UAV}}\left(t\right)\right)}^2,{\left(y_f^{\mathrm{UAV}}\left(t\right)\right)}^2},$$

  (39)

  where ${\omega }_f$ denotes the penalty coefficient for UAV outside of boundaries. $X_f^{\mathrm{UAV}}\left(t\right)$ and $y_f^{\mathrm{UAV}}\left(t\right)$ denote the distances at which the UAV exceeds the x-axial and y-axial boundaries in the time slot $t+1$, respectively.

  $$X_f^{\mathrm{UAV}}\left(t\right)=\max \left(\left|x^{\mathrm{UAV}}\left(t\right)-{\displaystyle \frac{c_d+c_u}{2}}\right|-{\displaystyle \frac{c_u-c_d}{2}},0\right),$$

  (40)

  $$Y_f^{\mathrm{UAV}}\left(t\right)=\max \left(\left|y^{\mathrm{UAV}}\left(t\right)-{\displaystyle \frac{c_d+c_u}{2}}\right|-{\displaystyle \frac{c_u-c_d}{2}},0\right).$$

  (41)
• Reward for the UAV to reach the final position: the reward represents the “attractiveness” of the final point to the UAV, encouraging it to reach the final point within the allowed time. The additional reward is defined as

  $$R\left(t\right)=R\left(t\right)-{\omega }_t\times (T_{t+1}^{\mathrm{remain}}-T_{t+1}^{\min })$$

  (42)

  where $T_{t+1}^{\mathrm{remain}}$ is the remaining available time of the mission period, and $T_{t+1}^{min}$ is the minimum time taken required for the UAV to reach the final point in time slot $t+1$. $T_{t+1}^{min}$ can be expressed as

  $$T_{t+1}^{min}={\displaystyle \frac{∥{\mathbf{q}}^{\mathrm{UAV}}(t+1)-{\mathbf{q}}_T^{\mathrm{UAV}}∥}{V_{\max }^{\mathrm{UAV}}}}.$$

  (43)
  This indicates that the earlier the UAV approaches the final position, the more rewards it will receive. ${\omega }_t$ is the priority weight for the UAV to reach the final position before the end of the mission.

The detailed flow of the TCOTO algorithm is shown in Algorithm 1.

Algorithm 1 TCOTO

Input:  ${\mathbf{q}}^{\mathrm{UAV}}\left(t\right),{\mathbf{p}}_1,\dots ,{\mathbf{p}}_k,Q^{\mathrm{UAV}}\left(t\right)$ Output:  $b_k\left(t\right),V^{\mathrm{UAV}}\left(t\right),{\mathbf{d}}^{\mathrm{UAV}}\left(t\right),x\left(t\right),R\left(t\right),f^{\mathrm{UAV}}\left(t\right)$   1: Set the initial values for the network parameters $\left\{\varphi ,\varphi \prime ,{\theta }_1,{\theta }_2,{\theta }_1^{\prime },{\theta }_2^{\prime }\right\}$.   2: Set the experience buffer memory_size and specify the batch size for training batch size.   3: for $episode$ = 1, 2, …, Max_Episode do   4:     Initialize the environment.   5:     for $t=1,2,\dots ,T$ do   6:           (1) Select actions $\mathbf{a}\left(t\right)$ at the state $\mathbf{s}\left(t\right)$.   7:           (2) Execute $\mathbf{a}\left(t\right)$, obtain the reward $R\left(t\right)$ and the next state $\mathbf{s}(t+1)$.   8:           (3) Store the tuple into buffer.   9:           (4) Randomly select batch size data from buffer. 10:           (5) Calculate the TD target using Equation (27). 11:           (6) Obtain the loss function using Equation (28) and update the Critic network           parameters ${\theta }_1$ and ${\theta }_2$. 12:           if delayed_update = True then 13:                Calculate the loss function of the Actor network and update the parameter                $\varphi$ using Equation (33). 14:                Use soft updates to synchronize the parameters ${\theta }_1^{\prime }$ and ${\theta }_1^{\prime }$ of the Critic target                network. 15:         end if 16:     end for 17: end for

6. Simulation and Performance Evaluation

This section starts by outlining the simulation environment and the parameter settings. We then evaluate the performance of the proposed TCOTO algorithm against several benchmark algorithms and conduct a detailed analysis of the optimal strategies. Finally, we examine in detail the impact of different system parameters on performance metrics.

6.1. Parameter Setting

We analyze a network setup featuring a UAV and eight IoT devices, which are randomly placed within a remote square area measuring 1000 $\mathrm{m}$ ∗ 1000 $\mathrm{m}$. The UAV operates at an altitude of $H^{\mathrm{UAV}}=100$$\mathrm{m}$. The total mission duration is set to 100 $\mathrm{s}$, with $\tau =0.1\mathrm{s}$. The other parameters are shown in

Table 2. Simulation parameters.

Parameter	Value
UAV flight boundaries $c_d$, $c_u$	0 m, 1000 m
Additive loss ${\eta }_{k,t}^{\mathrm{LoS}}$, ${\eta }_{k,t}^{\mathrm{NLoS}}$	20
Carrier frequency $f_c$	0.1 GHz
Noise power ${\sigma }^2$	−144 dBm
Environment parameters ${\varpi }_1$, ${\varpi }_2$	4.88, 0.43
Transmit power of IoT devices $P_k$	0.4 W
Bandwidth $B^{\mathrm{UAV}}$, $B^{\mathrm{BS}}$, $B^{\mathrm{HAP}}$	1 MHz, 10 MHz, 10 MHz
Computational density $L^{\mathrm{UAV}}$, $L^{\mathrm{HAP}}$	737.5 cycles/bit
Effective switched capacitance ${\kappa }^{\mathrm{UAV}}$, ${\kappa }^{\mathrm{HAP}}$	${10}^{-28}{\mathrm{Ws}}^3/{\mathrm{cycle}}^3$
Maximum frequency of UAV CPU $f_{\max }^{\mathrm{UAV}}$	10 GHz
UAV related parameters ${\mathrm{C}}_1$, ${\mathrm{C}}_2$, ${\mathrm{C}}_3$, ${\mathrm{C}}_4$, $V_{\mathrm{tip}}$	80, 22, 263.4, 0.0092, 120 m/s
The speed of light c	$3\times {10}^8\mathrm{m}/\mathrm{s}$
Positions of HAP and BS ${\mathbf{p}}^{\mathrm{HAP}}$, ${\mathbf{p}}^{\mathrm{BS}}$	[500, 500, 1000], [1500, 0, 0]
Target network soft update rate ${\epsilon }_{\theta }$, ${\epsilon }_{\varphi }$	0.1, 0.1
Learning rate ${\eta }_{\varphi }$, ${\eta }_{\theta }$	0.0001, 0.0002
Discount factor $\gamma$	0.999
Actions noise variance $\epsilon$	0.01

.

The simulation experiments are run on the Python 3.6 development environment and TensorFlow deep learning framework. The TCOTO algorithm deploys an Actor network with four fully connected hidden layers. The actions are output through the hyperbolic tangent function and the output dimensions of 600, 300, 200, and 10. The Critic network features three dense hidden layers. The initial layer processes the input states and actions, producing a 600-dimensional output after applying the weight matrix. The other hidden layers output dimensions of 300 and 10, respectively, and the fully connected layer outputs a Q-value.

We adopt the following four benchmark algorithms with the proposed algorithm together to evaluate the joint optimization schemes.

• DQN-based Joint Computation Offloading and UAV Trajectory Optimization (DQN-COTO) Algorithm: DQN is a widely recognized DRL algorithm noted for its effectiveness in problems involving finite action and state spaces. For addressing continuous space challenges, the action space is divided into multiple distinct levels, allowing the agent to choose from these predefined options.
• AC-based Joint Computation Offloading and UAV Trajectory Optimization (AC-COTO) Algorithm: The AC algorithm is an algorithm based on value functions and random strategies. When choosing an action, it is not entirely certain, but rather based on a certain probability distribution.
• DDPG-based Joint Computation Offloading and UAV Trajectory Optimization (DDPG-COTO) Algorithm: DDPG is a well-established DRL technique designed for continuous spaces, built on the Actor-Critic (AC) framework. It differs from the AC method by using a deterministic policy, where the Actor network produces specific action values directly. Furthermore, DDPG introduces two target networks and employs soft parameter updates to boost the stability and convergence of the learning process.
• TCOTO algorithm (without HAP): To demonstrate the superiority of AGIN scenarios, we also set up the TCOTO algorithm in scenarios where HAP is not involved in data computation. This algorithm follows the same implementation as the proposed one, but the UAV has only two options: process data locally or offload them to a distant BS.

6.2. Performance Analysis

Figure 3 shows the average reward regarding four joint optimization algorithms in each iteration based on the Lyapunov drift weight coefficient $V=250$. The average reward after convergence of the proposed method is improved by 4.54%, 79.66%, and 249.13% compared to the DDPG-CTO, AC-CTO, and DQN-COTO algorithms, respectively. It can be observed that, at the start of the training phase, the average reward increases as the number of episodes grows. This is because the agent has learned to choose appropriate strategies to adapt to dynamic environments, such as computation offloading and UAV trajectories. After the intelligent agent further learns reward and punishment constraints in exploration, the proposed algorithm will gradually converge, and the final average reward will fluctuate around its convergence value. The proposed TCOTO algorithm ultimately achieves a higher average reward, which means it has an advantage over the other three algorithms. During the training process, the DDPG-COTO algorithm can obtain higher rewards compared to the TCOTO algorithm because TD3 introduces delayed policy updates during updates, which can avoid premature convergence of the policy to local optima. Moreover, our TCOTO algorithm demonstrates greater stability after convergence. This is because TD3 employs two Critic networks to address the overestimation issues seen in DDPG, thereby reducing Q-value estimation errors. Due to the limited number of discrete values in the action space of DQN, there are few strategies available for the agent to choose from. Moreover, DQN does not incorporate action exploration during training, which can easily lead to convergence.

Figure 4 shows the average energy efficiency over time slots. The average energy efficiency is obtained by dividing the amount of data processed and the energy consumption of the whole system (including UAV, HAP, and BS) at each time slot. It is evident that both the proposed algorithm and the four benchmark algorithms can achieve stable energy efficiency benefits. However, the energy efficiency of the proposed approach is increased by 3.77%, 22.90%, and 67.62% compared to the DDPG-COTO, AC-COTO, and DQN-COTO algorithms, respectively. This is because the proposed algorithm improves stability and efficiency by the three techniques introduced in Section 5, thus avoiding local optimums. Hence, the TCOTO algorithm achieves higher rewards in the training and convergence phases. UAV plans reasonable trajectories while ensuring queue stability, and making energy-saving offloading decisions for collected data, enabling the system to process more data with minimal energy consumption and avoiding resource waste. Furthermore, the energy efficiency of the TCOTO algorithm within the proposed architecture surpasses that of the system architecture without HAP. This superiority arises because the UAV can make suitable offloading decisions tailored to different environmental conditions in each time slot, thereby reducing energy consumption. This further highlights the advantages of the integrated architecture, showing that the TCOTO algorithm delivers the highest performance.

Figure 5 investigates the impact of parameter V on the average energy efficiency of the system. From the figure, it can be seen that when V is small (i.e., $235<V\le 245$), energy efficiency increases rapidly as V increases. When $245<V\le 250$, energy efficiency remains relatively stable. When V is large (i.e., $250<V\le 260$), energy efficiency decreases sharply as V increases. After determining the iteration count and other parameters, the selection of parameter V will significantly affect the performance of our proposed algorithm in terms of system energy efficiency. According to Equation (20) in Section 4, we can know that as V increases, the system’s constraints on the queue will decrease, leading to an increase in queue backlog. Therefore, when $235<V\le 245$, as V increases, the system’s constraint on the queue will become smaller, and at this point, the UAV will tend to collect more data. When the amount of data collected does not exceed its processing limit, the UAV can process more data at the same flight cost, thereby increasing the energy efficiency of the system. When the collected data approach the limit that the UAV can withstand (i.e., $245<V\le 250$), the growth rate of system energy efficiency tends to stabilize. But when the collected data exceed the limit that the UAV can withstand (i.e., $250<V\le 260$), the UAV requires additional energy cost to calculate or offload tasks, leading to reduced energy efficiency. With the ongoing increase in data volume, the UAV’s energy efficiency experiences a sharp decline. Therefore, we need to find a suitable value for parameter V to maintain the UAV queue at a reasonable length, thereby maximizing the system’s energy efficiency.

Figure 6 shows the trajectories of UAV and randomly distributed location IoT devices. Figure 6a represents the 3D trajectories of the UAV, and Figure 6b shows the corresponding 2D trajectories. It can be seen that the UAV can correctly reach the destination to complete the data collection task of IoT devices based on all joint optimization algorithms except for the DQN-COTO, because the limited discrete actions of the DQN-COTO algorithm are not sufficient to support the UAV in making reasonable trajectory planning. In addition, the flight path determined by the proposed algorithm aligns with expectations, as the UAV naturally moves nearer to IoT devices. This approach effectively fulfills the data collection needs from these devices while also minimizing the energy consumption associated with offloading computing tasks. This is because the proposed algorithm understands the reward and penalty constraints well enough to plan better trajectories.

Figure 7 shows the variation of average queue backlog $Q^{\mathrm{UAV}}\left(t\right)$ with time slots after the algorithm’s reward convergence. After the queue stabilizes, it is clear that the task backlog under the proposed algorithm is significantly less than that observed with the DDPG-COTO, AC-COTO, and DQN-COTO algorithms. This reduction in task backlog occurs because the TCOTO algorithm allows the UAV to adaptively modify its task offloading strategies and flight paths in response to real-time task queue conditions, enabling it to make more efficient decisions and minimize task accumulation. Since the DQN-COTO algorithm is prone to falling into local optimums, it is significantly worse than the other three algorithms at maintaining queue stability. In addition, the queue of the proposed algorithm can remain stable, while the queue of the benchmark algorithm fluctuates greatly. This is because the TCOTO algorithm has good performance in computation offloading and UAV trajectory strategy, and the data collected by the UAV can be processed quickly without causing a serious queue backlog.

7. Conclusions

This paper investigated the optimization problem of joint computation offloading and UAV trajectory planning in EI-powered air-ground integrated networks. This study focus was on enhancing the long-term average energy efficiency while maintaining the stability of the UAV’s long-term queue. The proposed online algorithm integrates Lyapunov optimization and DRL methods, converting the initial long-term optimization challenge into a manageable short-term problem within each time slot. It then dynamically refines the computation offloading and UAV trajectory strategies using an effective DRL approach. Simulation results show that our proposed algorithm performs well in making decisions based on the current environment, thus significantly improving the energy efficiency of the system. The proposed algorithm improved the performance of the system energy efficiency by 3.77%, 22.90%, and 67.62%, respectively, compared to the other three benchmark schemes. In our future work, we plan to explore the maximization of energy efficiency in multi-UAV scenarios by leveraging a multi-agent DRL algorithm.