Joint Computation Offloading and Trajectory Optimization for Edge Computing UAV: A KNN-DDPG Algorithm

Abstract

Unmanned aerial vehicles (UAVs) are widely used to improve the coverage and communication quality of wireless networks and assist mobile edge computing (MEC) due to their flexible deployments. However, the UAV-assisted MEC systems also face challenges in terms of computation offloading and trajectory planning in the dynamic environment. This paper employs deep reinforcement learning to jointly optimize the computation offloading and trajectory planning for UAV-assisted MEC system. Specifically, this paper investigates a general scenario where multiple pieces of user equipment (UE) offload tasks to a UAV equipped with a MEC server to collaborate on a complex job. By fully considering UAV and UE movement, computation offloading ratio, and blocked relations, a joint computation offloading and trajectory optimization problem is formulated to minimize the maximum computational delay. Due to the non-convex nature of the problem, it is converted into a Markov decision process, and solved by the deep deterministic policy gradient (DDPG) algorithm. To enhance the exploration capability and stability of DDPG, the K-nearest neighbor (KNN) algorithm is employed, namely KNN-DDPG. Moreover, the prioritized experience replay algorithm, where the constant learning rate is replaced by the decaying learning rate, is utilized to enhance the converge. To validate the effectiveness and superiority of the proposed algorithm, KNN-DDPG is compared with the benchmark DDPG algorithm. Simulation results demonstrate that KNN-DDPG can converge and achieve 3.23% delay reduction compared to DDPG.

1. Introduction

The rapid growth of Industry 4.0 has fueled the revolution in Industrial Internet of Things (IIoT) services, such as cloud robots in innovative manufacturing environments [1,2]. With the growing demand for computation-intensive and latency-sensitive workloads, IIoT user equipment (UE) such as mobile robots struggle to meet the stringent quality of service requirements. Traditional cloud computing concentrates computation and storage resources in data centers far away from UE, imposing extensive delay and bandwidth consumption during data transmission [3]. Mobile edge computing (MEC) focuses on moving computing resources to the edge of the network to provide high-performance, low-latency services to UE. Therefore, it efficiently handles computation-intensive and latency-sensitive tasks [4].

Previous works mainly focus on fixed base stations (BSs) to provide computing services. For example, Zhao et al. maximized the system utility through offloading computation and resource allocation optimization in cloud-assisted MEC systems [5]. However, in some practical applications (e.g., communication disruption due to natural disasters), deploying traditional terrestrial BSs cannot provide sufficient service range. Thus, the mobility and autonomy of unmanned aerial vehicles (UAVs) become one of the best ways. Currently, UAVs are expected to play a critical role in future wireless communication systems [6]. Indeed, UAVs can serve as relay BSs or aerial BSs in situations where communications are obstructed, thereby enhancing the network coverage in areas that are otherwise inaccessible or experiencing high traffic [7].

UAV-assisted MEC systems have great potential in providing IIoT services, e.g., industry, military, logistics, aerial photography, and smart cities [8,9]. Currently, UAVs can be used as MEC servers in scenarios where UAVs have large battery and computation power, or in scenarios where there is no area of terrestrial MEC systems due to natural disasters. Moreover, UAVs can be used as repeaters to connect two pieces of UE whose communication channels are blocked. This new setup can help UE with limited local resources to access computation resources [10]. With the development of MEC systems, an increasing number of researchers employ deep reinforcement learning (DRL) algorithms to optimize computation offloading and/or trajectory planning for UAV-assisted MEC systems [11,12,13,14,15,16]. Although deep deterministic policy gradient (DDPG) is suitable for continuous control scenarios, it suffers from inefficient exploration, which may lead to limited state space utilization and thus tends to converge to locally optimal solutions [17].

Motivated by the above deficiencies, this paper improves on the DDPG algorithm by utilizing the K-nearest neighbor (KNN) algorithm and priority experience replay (PER) for computation offloading and trajectory optimization UAV-assisted MEC systems. Specifically, KNN searches for high-dimensional data quickly and approximately, and when combined with the K-dimension tree, it reduces the computational complexity. Based on KNN, the states in reinforcement learning (RL) are categorized, and the reward function is incorporated according to the number of different classifications, thus exploring low-frequency states. In addition to this, this paper prioritizes the sampling of important samples in the empirical playback phase of RL, i.e., the PER algorithm, which enhances the ability of DDPG algorithm to explore effectively with a small number of samples. Instead of the traditional constant learning rate during training, the learning rate decay is utilized for training to improve the stability of the algorithm.

The contributions of this paper can be summarized as follows:

• Considering the UAV-assisted MEC system in the presence of obstacle blocked, the joint computation offloading and trajectory optimization (JCOT) problem under a dynamic channel is modeled. Under energy constraints, the JCOT problem is solved by jointly optimizing UE scheduling, computation offloading ratio of UE, UAV trajectory to minimize the maximum computational delay.
• For the JCOT problem, the corresponding Markov decision process (MDP) is designed and formulated, and the KNN-DDPG algorithm is proposed based on the constructed problem. The developed program improves the exploration capabilities of the DDPG in two ways. It enhances the exploration breadth of algorithm by utilizing categorical counting based on the KNN algorithm. Additionally, it improves the algorithm depth of exploration for important data by utilizing sampling based on the PER algorithm.
• Extensive simulations validate that the proposed algorithm improves DDPG under different parameters and communication conditions, and is more stable than other baseline algorithms.

The rest of the paper is organized as follows. Section 2 introduces the recent works on UAV-assisted MEC system. Section 3 presents the system model of UAV-assisted MEC system and the optimization problem. Section 4 introduces the KNN-DDPG algorithm. Section 5 illustrates the simulation results, and makes comparisons with current baseline algorithm. Finally, Section 6 concludes this work. For easy reading, all abbreviations throughout the paper are listed in

Table 1. Summary of abbreviations.

Full Name	Abbreviation
Base station	BS
Deep reinforcement learning	DRL
Deep deterministic policy gradient	DDPG
Industrial Internet of Things	IIoT
Joint computation offloading and trajectory optimization	JCOT
K-nearest neighbor	KNN
Line-of-sight	LoS
Markov decision process	MDP
Mobile edge computing	MEC
Prioritized experience replay	PER
Reinforcement learning	RL
Temporal difference	TD
Unmanned aerial vehicle	UAV
User equipment	UE

.

2. Related Work

2.1. Literature Summary

Existing works on UAV-assisted MEC systems can be classified as the following three types, according to their optimization objectives and the proposed algorithms.

• Minimize energy consumption for the entire system or mobile UE. Wang et al. proposed a collaborative resource optimization problem to minimize the total energy consumption of the system by jointly optimizing computation offloading, UAV flight trajectory and edge computing resource allocation [18]. However, this strategy does not effectively exploit the advantages of UAVs in terms of flexibility, mobility and ease of deployment. Wang et al. minimized the total UAV energy consumption through joint partitioning and UAV trajectory scheduling to extend UAV flight time and network lifetime [19].
• Minimize task completion time. Nath et al. studied a multi-user cache-assisted MEC system with stochastic task arrivals and proposed a DDPG-based dynamic scheduling strategy to jointly optimize dynamic caching, computation offloading, and resource allocation in the system [20]. Gao et al. developed a deep reinforcement learning-based approach for trajectory optimization and resource allocation, which intelligently manages the time allocation of TDs to maximize safe computation capacity using Deep Q-Learning. This method considers constraints such as time, UAV motion, minimum computation capacity, and data stability [21]. However, obstacle blocking in practical applications was not considered.
• Balancing energy consumption and latency. Zeng et al. employed method of Lyapunov to transform the long-term optimization problem into two deterministic online optimization subproblems, which were solved iteratively to balance the trade-off between energy consumption and queue stability for users with heterogeneous demands [22]. Their approach minimized both the energy consumption and completion time of UAVs. Zhang et al. considered random UE data arrivals to minimize the long-term average weighted system energy while ensuring queue stability and adhering to UAV trajectory constraints [23,24]. However, their method does not take into account the movement of UE and the entire trajectory of the UAV from the initial position to the destination is recomputed for each time slot, which increases the computational complexity.

The above works are summarized in

Table 2. Summary of related works.

Reference	Optimization Objective	Optimization Variables	Algorithms
[18]	Minimize total system energy consumption	Computation offloading; UAV flight trajectory; Resource allocation	Multi-agent delayed deterministic policy gradient; Karush–Kuhn–Tucker conditions
[19]	Minimize total system energy consumption	Regional division; UAV trajectory	Semi-discrete optimal transmission problems; Determine the shortest route
[20]	Minimize task completion time	Dynamic caching; Computation offloading; Resource allocation	DDPG
[21]	Minimize task completion time	Trajectory optimization; Resource allocation	Deep Q-Learning
[22]	Balancing energy consumption and latency	Energy consumption; Queue stability for users	Lyapunov; Successive convex approximation
[23]	Balancing energy consumption and latency	Average weighted energy consumption of UE and UAV	Lyapunov

.

2.2. Motivation and Contribution

Most existing UAV-on-ground channel models rely on deterministic line-of-sight connections to depict free-space path loss, yet this approach often falls short in urban or suburban settings [25]. Furthermore, the communication dynamics in UAV-assisted systems can change rapidly, and many mission offloading challenges are non-convex, making it difficult for conventional optimization methods to handle them effectively. Additionally, while machine learning techniques are often used to address various challenges in UAV-assisted MEC systems, the complexity of data processing and analysis rises as the size of the data increases, which can lead to delays in decision making. On the other hand, DRL tackles these decision-making hurdles by leveraging its capacity to learn and adapt through real-time environmental data. This paper explore a UAV-assisted MEC system, while considering constraints such as UAV energy limitations, UE and UAV movement, UE task size, and blocked conditions. The JCOT problem is modeled as an MDP since the action space in this system is mostly high-dimensional continuous and the problem in this paper is non-convex. Finally, the KNN-DDPG algorithm jointly optimizes UE scheduling, the UAV trajectory, and the computation offloading ratio in order to enhance the realism of the decision problem. The JCOT problem aims to minimize the maximum computational delay.

3. System Model and Problem Formulation

As depicted in Figure 1, a UAV-assisted MEC system comprises N UE and a UAV equipped with a MEC server. The local computing power of UE is limited, but with low-latency computation tasks. In contrast, UAVs with MEC server have strong computation power, and are in proximity to UE to provide MEC services. Thus, UE offloads part of the computation tasks to UAVs for edge computing via 5G ultra reliable ultra latency communication (URLLC).

3.1. Communication Model

The entire communication period T of the system is divided into time slots $t\in \left\{1,2,\dots ,T\right\}$ of equal size [26]. In each time slot, the UAV can only provide communication and computation services to a single UE [27].

The set of $n\in \left\{1,2,\dots ,N\right\}$ UE in time slot t is denoted as $U\left(t\right)=U_1\left(t\right),U_2\left(t\right),\dots ,U_N\left(t\right)$. The coordinates of $U{E}_n$ at time slot t are $Q_{ue}\left(t\right)={[X_n\left(t\right),Y_n\left(t\right)]}^T$. The UAV transmits data for $U{E}_n$ in this time slot, and the position of the UAV flight at a fixed altitude H in the horizontal plane is defined as $Q_{uav}\left(t\right)={[X\left(t\right),Y\left(t\right)]}^T$, where the UAV is flying at a fixed altitude H. The coordinates are updated after the UAV has flown to a new hovering position of $Q_{uav}(t+1)={[X(t+1),Y(t+1)]}^T$. The UAV hovers over the location and communicates with one of the UE. In a time slot, the horizontal coordinates of the UAV motion can be expressed as

$$X(t+1)=X\left(t\right)+t_{fly}v\left(t\right)cos\theta \left(t\right),$$

(1)

$$Y(t+1)=Y\left(t\right)+t_{fly}v\left(t\right)sin\theta \left(t\right).$$

(2)

The new hovering position of the UAV is expressed as

$$Q_{uav}(t+1)={\left[X\left(t\right)+t_{fly}v\left(t\right)cos\beta \left(t\right),Y\left(t\right)+t_{fly}v\left(t\right)sin\theta \left(t\right)\right]}^T,$$

(3)

where $t_{fly}$ denotes the flight time of the UAV, $v\left(t\right)\in [0,v_{max}]$ is the flight speed of the UAV, $\theta \left(t\right)\in [0,2\pi ]$ is the angle of flight of the UAV in the horizontal plane projection with respect to the x-axis in the horizontal direction. At the time slot t, the distance between the UE and the UAV can be expressed as [28]

$$d=\sqrt{{\left(X\left(t\right)-X_n\left(t\right)\right)}^2+{\left(Y\left(t\right)-Y_n\left(t\right)\right)}^2+H^2}.$$

(4)

UAVs are commonly airborne, and carry communications equipment. Additionally, they have the advantage of a higher altitude position, which helps establish line-of-sight (LoS) communication links, resulting in more stable and efficient communications. Due to the fact that UAVs are much higher than ground UE, the LoS channel of the UAV communication link is more prominent than other channel losses [29]. The LoS communication links channel gain between the UAV and $U{E}_n$ can be expressed as

$$h_{up}\left(t\right)={\beta }_0{d}_{uu}^{-2}\left(t\right)={\displaystyle \frac{{\beta }_0}{H^2+\left|\right|Q_{uav}\left(t\right)-Q_{ue}\left(t\right)\left|\right|}},$$

(5)

where ${\beta }_0$ denotes the channel gain of 1m and $d_{uu}^{-2}\left(t\right)$ is the distance between the UAV and the UE at the time slot t. Due to the severe impact of obstruction by environmental obstacles present in the system environment (e.g., trees or buildings), the wireless transmission rate can be expressed as

$$r_n\left(t\right)=B{log}_2\left(1+{\displaystyle \frac{P_{up}{h}_{up}\left(t\right)}{{\delta }^2+f_n\left(t\right)P_{NLoS}}}\right),$$

(6)

where B denotes the communication bandwidth, $P_{up}$ denotes the uplink transmit power, $f_n\left(t\right)$ indicates the presence or absence of obstacles at time slot t (0 means no blocked and 1 means blocked), and ${\mathit{P}}_{NLoS}$ is the transmission loss.

3.2. Computation Model

Unlike the edge computing of terrestrial MEC system, the edge computing of UAV-assisted MEC system involves three processes: computation offloading, local computing, and UAV flight. The proposed UAV includes partial offloading of UE tasks within each time slot. Let ${\alpha }_n\left(t\right)\in [0,1]$ indicate the computation offloading ratio from n-th UE to UAV, while the remaining computation tasks are calculated locally. In time slot t, the local computational delay and energy consumption can be expressed as

$$T_{Local}\left(t\right)={\displaystyle \frac{\left(1-{\alpha }_n\left(t\right)\right)D_n\left(t\right)C_{ue}}{f_{ue}}},$$

(7)

$$E_{Local}\left(t\right)=\mathit{k}{f}_{ue}^2\left(1-{\alpha }_n\left(t\right)\right)D_n\left(t\right)C_{ue},$$

(8)

where $D_n\left(t\right)$ denotes the computation size of the n-th UE, $C_{ue}$ is the CPU cycles required to process each unit byte, $\mathit{k}={10}^{-27}$ is the impact factor of chip architecture on CPU processing, and $f_{ue}$ denotes the computing capability of the UE [30]. Due to the small delay in returning results from the MEC server in the UAV, this paper ignores its effect. The computational delay of edge computing is expressed as

$$T_{up,n}\left(t\right)={\displaystyle \frac{{\alpha }_n\left(t\right)D_n\left(t\right)}{r_n\left(t\right)}}.$$

(9)

The energy consumption for the UAV flight can be expressed as

$$E_{fly}\left(t\right)={\displaystyle \frac{1}{2}}{M}_{uav}{t}_{fly}{\left|\right|v\left(t\right)\left|\right|}^2,$$

(10)

where $M_{uav}$ is related to the payload of the UAV, $t_{fly}$ denotes the flight time of UAV [12].

The computational delay and energy can be expressed as

$$T_{uav,n}\left(t\right)={\displaystyle \frac{{\alpha }_n\left(t\right)D_n\left(t\right)C_{uav}}{f_{uav}}},$$

(11)

$$E_{uav,n}\left(t\right)={\displaystyle \frac{{\alpha }_n\left(t\right)D_n\left(t\right)C_{ue}}{f_{uav}}},$$

(12)

where $f_{uav}$ denotes the computing capability of UAV.

3.3. Problem Formulation

To optimize the computation offloading and trajectory planning, we formulate the JCOT problem as follows:

$$\underset{R_n\left(t\right),{\alpha }_n\left(t\right),Q_{uav}(t+1),Q_{ue}(t+1)}{min}\sum _{t=1}^T\sum _{n=1}^N{R}_n\left(t\right)max\left\{T_{local}\left(t\right),T_{up}\left(t\right)+T_{uav}\left(t\right)\right\}$$

(13)

such that

$$R_n\left(t\right)\in \left\{0,1\right\},\forall n\in \left\{1,2,\dots ,N\right\},t\in \left\{1,2,\dots ,T\right\},$$

(14)

$$\sum _{n=1}^N{R}_n\left(t\right)=1,\forall t,$$

(15)

$$Q_{uav}\left(t\right)\in \left\{X\left(t\right),Y\left(t\right)\left|X\right(t)\in [0,L],Y(t)\in [0,W]\right\},\forall t,$$

(16)

$$Q_{ue}\left(t\right)\in \left\{X_{ue}\left(t\right),Y_{ue}\left(t\right)|X_{ue}\left(t\right)\in [0,L],Y_{ue}\left(t\right)\in [0,W]\right\},\forall t,n,$$

(17)

$$b_n\left(t\right)\in \left\{0,1\right\},\forall t,n,$$

(18)

$$0\le {\alpha }_n\left(t\right)\le 1,\forall t,n,$$

(19)

$$\sum _{n=1}^N\left(E_{fly}\left(t\right)+E_{uav}\left(t\right)\right)\le E,\forall t,$$

(20)

$$\sum _{t=1}^T\sum _{n=1}^N{D}_n\left(t\right){\alpha }_n\left(t\right)=D,\forall t,n,$$

(21)

where $R_n\left(t\right)\in \{0,1\}$ indicates whether UE is scheduled. Constraints (14) and (15) ensure that the UAV can only communicate with one UE in each time slot. Constraints (16) and (17) denote the constraint ranges of the UAV and the UE. Constraint (18) denotes the blocked situation when the UAV and the UE communicate in time slot t. Constraint (19) constrains the range of values for the computation offloading ratio. Constraint (20) ensures that the energy consumed by the UAV throughout the system does not exceed the maximum battery capacity. Constraint (21) indicates that all tasks need to be due in the entire time period.

4. The Proposed KNN-DDPG Algorithm

The above optimization problem is mixed-integer non-linear programming. Thus, solving this problem by relying on traditional optimization methods is challenging. Therefore, a DRL-based approach is used to solve the problem.

4.1. MDP Modeling

4.1.1. State

The state space at the time slot t is expressed as

$$S_t=\left\{E_{battery},U\left(t\right),Q_{uav},D_{remain}\left(t\right),D\left(t\right),b_n\left(t\right)\right\},$$

(22)

where $E_{battery}$ denotes the remaining energy of the UAV battery in the current time slot, $U\left(t\right)$ is the location of UE, which a UAV selects in the t-th time slot, $Q_{uav}$ denotes the location information of UAV, and $D_{remain}\left(t\right)$ is the size of remaining tasks the system must complete in the whole period. $D\left(t\right)$ is the size of the randomly generated task in the current time slot, and $b_n\left(t\right)$ is the blocking relationship between UE. For $t=1$, $E_{battery}=E$, $D_{remain}\left(t\right)=D$.

4.1.2. Action

An agent optimizes the flight angle of the UAV in the continuous motion space by selecting UE to be serviced based on the observed environment and the current system state, including the UE to be served $\theta \left(t\right)$, flight speed $v\left(t\right)$, and the computation offloading ratio $\alpha \left(t\right)$. At the time slot t, the action space is represented as

$$A_t=\left\{n\left(t\right),\theta \left(t\right),v\left(t\right),\alpha \left(t\right)\right\}.$$

(23)

It should be noted that the agent selects a variable action $n\left(t\right)\in [0,N]$ if it needs to be decentralized, i.e., if $n\left(t\right)=0$, then $n^{\prime }=1$. If $n\left(t\right)\ne 0$, $n^{\prime }=\lceil n\left(t\right)\rceil$, where $\lceil .\rceil$ is an upward rounding symbol.

4.1.3. Reward

The total system reward is the sum of the rewards corresponding to the number of steps the UAV flight. The UAV can obtain an action for each step during the flight while flying to a new position and a UE for data transfer. The maximum delay of edge computing and local computing is minimized under the constraint of UAV energy consumption. The target delay function, as well as the reward function, is expressed as

$${\tau }_{delay}=\sum _{n=1}^{N}max\left\{T_{local}\left(t\right),T_{up}\left(t\right)+T_{uav}\left(t\right)\right\},$$

(24)

$$r_n=r(s_n,a_n)=-{\tau }_{delay}.$$

(25)

when $n=n^{\prime }$, $R_n\left(t\right)$ = 1. Otherwise, $R_n\left(t\right)$ = 0. When action $a_n$ is performed, the smaller the computational delay, the larger the reward.

4.2. KNN-DDPG Solution

Figure 2 depicts the KNN-DDPG framework.

The KNN algorithm searches for similar states to speed up the training process of the DDPG algorithm. This approach can utilize the existing empirical data to guide the exploration of the intelligent body by finding similar states in the state space. This guidance improves training efficiency and allows the intelligent body to learn good strategies faster. This modification optimizes the training efficiency and speed. Indeed, accelerating training by searching for similar states reduces the sample complexity of DDPG algorithm, leading to more efficient use of the training data, faster convergence, and improved strategy stability. The main idea is that after classifying the state space according to KNN, counts are counted for different classifications during the training process and count rewards are formed. Combining this reward with the training reward motivates the exploration of historical states. After completing the counting process, the counting results are used for the count reward function of KNN-DDPG, expressed as

$$r_{count}=\beta {\displaystyle \frac{d_{count}}{m}},$$

(26)

where m is the total number of currently categorized states, and a weighted decay factor $\beta$ that decays with the number of iterations is added in order to explore states with a low number of historical samples. $\beta$ is defined as:

$$\beta =\omega {\gamma }^n,$$

(27)

where $\omega$ is the set of weights, $\gamma$ is the decay factor, and n is the number of iterations. This process prevents the divergence caused by the increasing $r_{count}$ as the number of iterations increases, thus ensuring the final convergence of the algorithm. Finally, the final total reward is obtained by adding up $r_{count}$ using custom weights ${\omega }_r$.

$$r_{total}=r_n+{\omega }_r{r}_{count}$$

(28)

In DDPG, the strategy function acts as an “actor”, generating actions based on the current state of the continuous space. The Q-value function acts as a “critic”, evaluating the performance of the actor by estimating the value of the actions taken, thus guiding the actor to improve its subsequent actions. Figure 2 highlights that DDPG uses two different DNNs to approximate the actor network $\mu \left(s\right|{\theta }^{\mu })$, i.e., the Q-value function, and the critic function $Q(s,a|{\theta }^Q)$, i.e., the Q-value function. The first is learned and improved for final use, and the other is used to calculate the temporal difference (TD) error. In addition, both the actor and the critic networks include a target network with the same structure as the respective primary networks, improving clarity and maintaining consistency in the description of the target networks. The actor network ${\mu }^{\prime }$ involves the parameter ${\theta }^{{\mu }^{\prime }}$ and the critic target network $Q^{\prime }$, parameter ${\theta }^{Q^{\prime }}$. The parameters of the critic network are updated by minimizing the loss value (mean square error loss), and the loss function when the Critic network is updated as

$$L={\displaystyle \frac{1}{N}}\sum _{i=1}^t{\left(y_i-Q(s_i,a_i|{\theta }^Q)\right)}^2,$$

(29)

where

$$y_i=r_i+\gamma Q\left(s_{i+1},\mu \left(s_{i+1}\right)|{\theta }^Q\right).$$

(30)

The maximum discount incentive for DDPG is

$$R_i=\sum _{i=t}^{\infty }{\gamma }^{i-t}r(s_i,a_i),$$

(31)

where $\gamma \in [0,1]$ is the discount factor, and $r(s_i,a_i)$ is the immediate reward at the i-th time slot.

In traditional DDPG, samples can be randomly extracted in the experience pool M during training. However, this overlooks the significance of individual experiences, meaning that some may hold more learning potential than others. To tackle this shortcoming, PER comes into play, assigning priorities to experiences by evaluating their TD error—essentially gauging the disparity between predicted and target Q-values within DDPG (Equation (32)).

$$\sigma =y-Q(s,a)=R+\gamma Q^{\prime }\left(s^{\prime },{\mu }^{\prime }\left(s^{\prime }\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }}\right)-Q(s,a)$$

(32)

In this paper, rank based prioritization PER is used for prioritization as it has better stability as compared to proportional prioritization. Calculate the sampling probability according to the ranking $rank\left(i\right)$ of the TD errors $p_i$:

$$p_i={\displaystyle \frac{1}{rank\left(i\right)}},$$

(33)

After executing the algorithm, the agent observes the next state $s_{i+1}$ and the immediate reward $r_i$ and stores it in M, which at this point has the highest priority $p_t=1$. After K transitions in M, the actor target network ${\mu }^{\prime }$ outputs the action ${\mu }^{\prime }\left(s_{j+1}\right)$ to the critic target network $Q^{\prime }$, and the critic network calculates the target value $y_i$ based on Equation (30). The TD-error ${\sigma }_j$ is calculated, and the transition priority is updated as $p_j\leftarrow \left|{\sigma }_j\right|$.

Then, minimize the loss function by updating the critic network Q. The actor network provides the gradient of the small batch of actions to the critic network for the gradient of action ${\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_j,a=\mu \left(s_j\right)}$. The optimizer can be exported through its optimizer ${\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu }){|}_{s=s_j}$. The actor network can be updated based on Equation (34).

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{K}}\sum _j\left[{\nabla }_{a}Q(s,a|{\theta }^Q){{|}_{s=s_j,a=\mu \left(s_j\right|{\theta }^{\mu })}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|}_{s=s_j}\right]$$

(34)

Finally, the DDPG agent soft updates the commenter target network and the actor target network using a small constant $\iota$, which can be expressed as:

$${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q+(1-\iota ){\theta }^{Q^{\prime }},$$

(35)

$${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }+(1-\iota ){\theta }^{{\mu }^{\prime }}.$$

(36)

The pseudo-code of the KNN-DDPG algorithm is shown in Algorithm 1.

Algorithm 1: The proposed KNN-DDPG algorithm

5. Simulations

In this section, the effectiveness and superiority of the proposed KNN-DDPG algorithm is validated through numerical simulations. First, the simulation parameter settings are presented. The performance of the KNN-DDPG algorithm is then validated under different parameters and conditions, and compared with the benchmark DDPG algorithm.

5.1. Simulation Setting

The simulations is executed based on Python 3.7 and the TensorFlow 2 framework on the Anaconda platform. We consider a square region that has $N=4$ UE randomly placed in a 100 m × 100 m area to simulate the system environment on the Anaconda platform. The remaining parameters are listed in

Table 3. Simulation parameters.

Parameter Name	Parameter	Default Value
Communication cycle	T	320 s
Number of time slots	I	40
Flight area	X, Y, H	100 m
UAV weight	G	9.65 kg
Transmission loss	$P_{NLoS}$	−80 dB
Noise power of the receiver	${\theta }^2$	−100 dB
Transmission bandwidth	B	1 MHz
Maximum UAV flight speed	$v_{max}$	50 m/s
UAV flight time	$t_{fly}$	1 s
UE movement speed	$v_{ue}$	1 m/s
Battery capacity	E	500 KJ
Transmission power	P	0.1 W
CPU cycles	s	1000 cycles/bit
UE computing capability	$f_{ue}$	0.6 GHz
UAV computing capability	$f_{uav}$	1.2 GHz

[31].

5.2. Result Analysis

Figure 3 demonstrates the delay variation with the number of training episodes for different algorithms. KNN-DDPG demonstrates the best convergence, as it converges quickly, reaches the lowest delay in the early stage of training (within 200 rounds), and shows excellent stability in subsequent training. In contrast, DDPG also shows some convergence, but is slower and more volatile, demonstrating inferior final convergence than KNN-DDPG.

Figure 4 compares the algorithms with different task sizes. Obviously, the length of the box (i.e., the size of the interquartile range) is usually shorter, which validates the stability of KNN-DDPG, as shorter interquartile range imply less fluctuation in the delay values and more consistent performance. At small task sizes, such as medium quality video streaming, real-time conferencing (60 Mbits and 80 Mbits), the median delay of KNN-DDPG is below or close to DDPG, and its delay variation range is small. At medium task sizes, such as high-quality video streaming, basic virtual reality experiences (100 Mbits), the median delay of KNN-DDPG, while slightly higher, is still below or close to DDPG, and its range of data fluctuations is small. At larger task sizes, such as ultra-high-definition video streaming, and high-speed real-time data analysis tasks (120 Mbits and 140 MBits), although the median delay of KNN-DDPG increases, its delay is more tightly distributed, and it exhibits higher stability compared to the DDPG algorithm. KNN-DDPG has less performance fluctuation at all task sizes, especially at small-to-medium task sizes. This stability makes KNN-DDPG more suitable for application scenarios that require stable performance. The delay performance of KNN-DDPG remains stable when the task size increases. Although the performance decreases slightly, it has no significant signs of performance degradation, demonstrating strong robustness.

In summary, KNN-DDPG shows more stability and superiority under most task sizes, especially in scenarios that require low delay and high stability, making KNN-DDPG more competitive. Even under larger task loads, it still maintains a certain performance advantage and stability, making it a preferred solution to cope with various loads in edge computing environments.

To demonstrate the improvement of the algorithm in terms of exploration capability, the following experiment compares the DDPG and KNN-DDPG algorithms during training. The corresponding results are presented in Figure 5, where the color of each point reflects how often the intelligent agent samples the state quantity. Specifically, DDPG utilizes existing strategies in high-power and short-range areas with limited exploration in other areas, resulting in large fluctuations in offload rates. This reliance on specific states hinders rapid adaptation to new situations, suggesting that DDPG may be unable to explore effective strategies in regions with large state changes. In contrast, KNN-DDPG effectively balances exploration and utilization through KNN and prioritized empirical replay, dynamically adapting to strategies that explore new states while ensuring high utilization, thus maintaining a stable offloading ratio. The state space coverage is more uniform, adapts well to different battery power, distance, and offloading ratios, and maintains a high offloading ratio, even at low power or distance. These features indicate that KNN-DDPG has better balance and stability under different state conditions. In conclusion, DDPG struggles with complex or rapidly changing environments, leading to poor adaptability and suboptimal offloading ratios in several states. However, KNN-DDPG demonstrates superior environmental adaptability, finding effective strategies in diverse state spaces and maintaining stable performance, thereby demonstrating deeper exploration and optimized decision-making.

As presented in Figure 6, as the amount of UE increases, each algorithm shows delayed growth. The growth rate and the final delay of DDPG are significantly higher than that of the KNN-DDPG algorithm in the multiuser task environment. When the number of users increased from 6 to 8, the DDPG algorithm delay increased by 5.8% and the KNN-DDPG algorithm increased by 3.74%. Although the KNN-DDPG algorithm at a constant learning rate is less stable than the KNN-DDPG algorithm, the final experimental results are similar. Herein, the constant learning rate for the actor network and the critical network of both KNN-DDPG and DDPG algorithms are set to 0.01 and 0.02, respectively.

Figure 7 compares the impact of different discount factors $\gamma$ on the convergence performance of the proposed algorithm. For $\gamma =0.1$, the delay distribution of KNN-DDPG is relatively centralized, but its median is higher than that of DDPG. However, KNN-DDPG has a narrower range of latencies and a shorter box (interquartile range), suggesting the performance is more stable at this discount factor. On the other hand, DDPG has more fluctuating delay performance and outliers (points well above the box), suggesting that its performance is less stable than that of KNN-DDPG under this condition. For $\gamma =0.001$, more centralized and stable performance of KNN-DDPG compared to DDPG. The box plot shows a narrower range and less variation in the median. Despite some outliers, there are fewer fluctuations in delay overall. The delay performance of DDPG still fluctuates. For $\gamma =0.001$, stability is improved relative to the $\gamma =0.1$ case. For $\gamma =0.00001$, KNN-DDPG has the most stable delay performance. All delay values are concentrated in a narrow range with very short boxes and no significant outliers, indicating that KNN-DDPG is consistent and stable in this condition. In contrast, DDPG has significantly fewer delay fluctuations at this discount factor. However, some outliers are present, showing that its improved performance is not as stable as KNN-DDPG.

Figure 8 compares the KNN-DDPG algorithm using decaying learning rate and fixed learning rate approach, inferring that when no learning rate decay is applied, the algorithm converges slower in the early stages with higher and more fluctuating delay and performs poorly, especially in the first 200 rounds. However, when the learning rate decay is applied, the algorithm quickly reduces the delay and stabilizes after about 200 rounds. This result suggests that learning rate decay improves the convergence speed and stability of KNN-DDPG, allowing it to find a better strategy more quickly and keep the delay fluctuations small in later rounds.

6. Conclusions

This paper addressed the challenge of computation offloading and trajectory planning for UAV-assisted MEC system, where a UAV serves multiple pieces of UE. Correspondingly, the JCOT problem was formulated to optimize UE scheduling, computation offloading ratio and UAV trajectory. To solve this problem, the KNN-DDPG algorithm with PER was proposed and validated by simulations. The results demonstrated that KNN-DDPG achieve 3.23% lower delay than DDPG. Future works will extend to more complex scenarios that the MEC systems involving multiple UAVs to enhance computation offloading and edge computing.