Uplink Throughput Maximization in UAV-Aided Mobile Networks: A DQN-Based Trajectory Planning Method

Abstract

This paper focuses on the unmanned aerial vehicles (UAVs)-aided mobile networks, where multiple ground mobile users (GMUs) desire to upload data to a UAV. In order to maximize the total amount of data that can be uploaded, we formulate an optimization problem to maximize the uplink throughput by optimizing the UAV’s trajectory, under the constraints of the available energy of the UAV and the quality of service (QoS) of GMUs. To solve the non-convex problem, we propose a deep Q-network (DQN)-based method, in which we employ the iterative updating process and the Experience Relay (ER) method to reduce the negative effects sequence correlation on the training results, and the $\epsilon$-greedy method is applied to balance the exploration and exploitation, for achieving the better estimations of the environment and also taking better actions. Different from previous works, the mobility of the GMUs is taken into account in this work, which is more general and closer to practice. Simulation results show that the proposed DQN-based method outperforms a traditional Q-Learning-based one in terms of both convergence and network throughput. Moreover, the larger battery capacity the UAV has, the higher uplink throughput can be achieved.

1. Introduction

Recently, due to their cost-effectiveness, mobility, and operational flexibility, unmanned aerial vehicles (UAVs) have been applied in a variety of areas such as surveillance, surveying, commodities delivery, searching and rescuing, as well as emergency wireless coverage. By serving as airborne relays, data collectors, and aerial base stations (ABSs), UAVs can be deployed in wireless networks to assist the wireless information transmissions. Compared to ground base stations, UAV is more flexible, which is capable of establishing a good Line of Sight (LoS) link to construct stronger communication channels for the ground mobile users (GMUs) [1]. Consequently, the data rate of the wireless network can be greatly increased by the help of UAVs. Actually, when a UAV acts as an ABS, it can fly to and fro to provide flexible coverage services. However, due to the limited resources, including available energy and time, the UAV’s trajectory should be carefully planned to meet different goals, such as maximizing the sum data rate, minimizing the total energy consumption or minimizing the communication delay [2,3].

It is noticed that the most existing works have only considered the scenarios with static users. That is, in their works, the users’ mobility was not taken into account. When GMUs move, the channels between GMUs and the UAV may change quickly over time. In this case, the existing methods cannot capture the channel varying features cause by users’ mobility, which may be inefficient when they are applied to the scenarios with mobile users. To fill the gap, this paper studies the trajectory planning for the UAV-assisted networks with multiple mobile users, where a set of GMUs randomly move and desire to offload data to a UAV. The GMUs access the UAV in a time division multiple access manner to avoid inter-user interference. Due to the random mobility of GMUs, the state space is quite large. We present a DQN-based method to train the agent acted by the UAV. The contributions are summarized as follows:

• We formulate an optimization problem to maximize the uplink throughput by optimizing the UAV’s trajectory, under the constraints of the UAV’s available energy and the QoS requirements of GMUs. To efficiently characterize GMU’s mobility, the Gauss–Markov mobility model (GM) is employed;
• To solve the formulated problem, we propose a DQN-based UAV trajectory optimization method in which the reward function is designed based on the GMU offload gain and the energy consumption penalty, and the $\epsilon$-greedy method is employed to balance the exploration and exploitation;
• Simulation results show that the proposed DQN-based method outperforms traditional Q-Learning-based one in terms of convergence and network throughput, and the larger battery capacity the UAV has, the higher uplink throughput that can be achieved. In addition, compared with other specific trajectories, the DQN-based method can increase the total offload data amount of the system by more than about 19%.

This paper is structured as follows: Section 2 presents an overview of related previous work. Section 3 presents the system model and the optimization objectives of the considered UAV-assisted network. Section 4 presents the deep reinforcement learning nomenclature and theory and applies this theory to the current delivery problem, presenting the DQN-based solution. In Section 5, final simulation results are reported and discussed. Finally, Section 1 concludes the article.

The following Table 1 describes the significance of various abbreviations and acronyms used throughout the thesis. The page on which each one is defined or first used is also given.

Table 1. Abbreviation table.

Abbreviation	Meaning	Page
$\mathrm{ABS}$	Aerial base station	1
$\mathrm{DQN}$	Deep Q-network	2
$\mathrm{ER}$	Experience relay	7
$\mathrm{FPAPs}$	Fixed perceptual access points	3
$\mathrm{GMU}$	Ground mobile user	1
$\mathrm{LoS}$	Line of Sight	1
$\mathrm{QoS}$	Quality of service	6
$\mathrm{UAV}$	Unmanned aerial vehicle	1

2. Related Work

Thus far, different optimization methods have been applied to plan the UAVs’ trajectories. In [4], a UAV trajectory planning algorithm was proposed by employing the block coordinate descent and successive convex optimization techniques to maximize the minimum data rate of all ground users. In [5], graph theory and convex optimization were applied to find the approximate trajectory solution to minimize the UAV’s mission completion time. In [6], a sub-gradient method was applied to optimize the power consumption of a UAV system while ensuring the minimum data rate of IoT. In [7], particle swarm optimization and genetics algorithm were applied to minimize the transmit power required by the UAV subject to the users’ minimum data rate. In [8], graph theory and kernel K-means as well as the convex optimization method were applied to minimize the the sensor nodes’ maximal age of information (AoI) subject to the limited energy capacity.

However, in aforementioned works, only conventional optimization methods were employed and iterative algorithms were designed, which may be with high computational complexity, especially in the large-scale network scenarios or in a fast time-varying environment. Recently, with the remarkable results shown by reinforcement learning in decision-making tasks, scholars have started to employ reinforcement learning to plan the UAV trajectory.

Compared to traditional methods, the reinforcement learning method has many advantages, such as high adaptability to the environment and real-time interaction with environment. Therefore, the deepQ-network (DQN) has attracted increasing attention in UAV’s trajectory planning [9,10,11,12,13]. In [10], a DQN-based method was proposed to minimize the average data buffer length and maximize the residual battery level of the system, by planning the UAV’s trajectory. In [11], the DQN was employed to solve a priority-oriented trajectory planning problem to satisfy the latency tolerance of the network. In [12], a DQN-based method was proposed to maximize the quality of service based on the freshness of data. In [13], a DQN-based scheme was designed to maximize the uplink throughput of the UAV-assisted emergency communication networks.

The following Table 2 summarizes aforementioned existing works by methods used and optimization objectives.

Table 2. Related work table.

Methods	Optimization Objectives
Block coordinate descent and successive convex optimization method	Maximize the minimum data rate of all ground users [4].
Graph theory and convex optimization	Minimize the UAV’s mission completion time [5]; Minimize the the sensor nodes’ maximal age of information subject to the limited energy capacity [8].
Sub-gradient method	Minimize the power consumption of a UAV system while ensuring the minimum data rate of IoT [6].
Particle swarm optimization and genetics method	Minimize the transmit power required by the UAV subject to the users’ minimum data rate [7].
DQN-based method	Minimize the average data buffer length [9]; Maximize the residual battery level of the system [10]; Minimize the delay of the network [11]; Maximize the quality of service based on the freshness of data [12]; Maximize the uplink throughput [13]; Maximize the mean opinion score for users [14].

3. System Model

The system model of the considered UAV-assisted network is shown in Figure 1, which consists of an energy-limited UAV, K GMUs and M fixed perceptual access points (FPAPs). The UAV works as an ABS and provides computing services to the GMUs, communicating with the served GMU, computing the data uploaded by GMUs, and also returning calculation results to GMUs. Within the UAV’s coverage, all of the GMUs can be served, but due to the limited processing capacity, it is assumed that the UAV can only serve one GMU at a time. The total available energy of the UAV is denoted by $E_0$. Let the operational period of the UAV be N seconds. Within N, all energy $E_0$ can be used.

Figure 1. UAV service model for ground mobile user equipment.

To operate the system, the time is discretized and N is split into T time slots, so the slot with the time duration of

$$\delta =N/T.$$

(1)

When T is sufficiently large, $\delta$ is very small. Compared to the UAV that can move at high speed, GMUs are moving relatively slowly. Thus, in each $\delta$, the location of the GMU can be roughly regarded as static, and the UAV is able to fly as well as hover to serve the selected GMU by taking the FPAPs as moving references. We also assume that, in time slot t, the UAV hovers above the m-th FPAP, it is able to receive as well as complete the offloaded task ${\mu }_k\left(t\right)$ of the k-th GMU, which is associated with the FPAP. The UAV’s location at time slot t is denoted by

$$l_m^{\mathrm{UAV}}\left(t\right)=\left[x_m^{\mathrm{UAV}}\left(t\right),y_m^{\mathrm{UAV}}\left(t\right),H\right]$$

(2)

where $m\in \{1,2,\dots ,M\},t\in \{1,2,\dots ,T\}$, and H is hovering height of the UAV. The K GMUs are distributed randomly, and the k-th GMU is denoted by $k\in \{1,2,\dots ,K\}$. The UAV’s remaining energy at time slot t is $q\left(t\right)\in \left[0,E_0\right]$.

3.1. User Mobile Model

As mentioned previously, the locations of all GMUs can be roughly regarded as static in each time slot t. Based on the Gauss–Markov mobile model [15], the update for the velocity $v_k\left(t\right)$ and the angle ${\theta }_k\left(t\right)$ of the k-th GMU at the time slot t can be expressed by

$$\left\{\begin{array}{c}\hfill v_k\left(t\right)={\kappa }_1{v}_k(t-1)+\left(1-{\kappa }_1\right)\overline{v}+\sqrt{1-{\kappa }_1^2}{\mathrm{\Phi }}_k,\\ \hfill {\theta }_k\left(t\right)={\kappa }_2{\theta }_k(t-1)+\left(1-{\kappa }_2\right){\overline{\theta }}_k+\sqrt{1-{\kappa }_2^2}{\mathrm{\Psi }}_k,\end{array}\right.$$

(3)

where $0\le {\kappa }_1,{\kappa }_2\le 1$ denote the weights of the previous state, and $\overline{v}$ indicates the average velocity of all GMUs. ${\overline{\theta }}_k$ is the average angle of the k-th GMU. ${\mathrm{\Phi }}_k$ and ${\mathrm{\Psi }}_k$ of the k-th GMU characterize the randomness of GMU’s mobility, both obeying independent Gaussian distribution, but having separate numerical characteristics $\left({\overline{\xi }}_{v_k},{\varsigma }_{v_k}^2\right)$ and $\left({\overline{\xi }}_{{\theta }_k},{\varsigma }_{{\theta }_k}^2\right)$. The location of the k-th GMU at time slot t is denoted as

$$l_k^{\mathrm{GMU}}\left(t\right)=\left[x_k^{\mathrm{GMU}}\left(t\right),y_k^{\mathrm{GMU}}\left(t\right),0\right].$$

(4)

Following (3), one has that

$$\left\{\begin{array}{c}\hfill x_k^{\mathrm{GMU}}\left(t\right)=x_k^{\mathrm{GMU}}(t-1)+v_k(t-1)cos\left({\theta }_k(t-1)\right){\Delta }_{t,t-1},\\ \hfill y_k^{\mathrm{GMU}}\left(t\right)=y_k^{\mathrm{GMU}}(t-1)+v_k(t-1)sin\left({\theta }_k(t-1)\right){\Delta }_{t,t-1},\end{array}\right.$$

(5)

3.2. UAV Energy Consumption Model

The energy consumed by the UAV consists of communication energy consumption and propulsion energy consumption. Due to the communication energy consumption being relatively small, we mainly consider propulsion energy consumption of the UAV. According to [16], for a rotary-wing UAV flying with speed V, the propulsion power consumption can be calculated by

$$P\left(V\right)=P_0\left(1+\frac{3V^2}{U_{\mathrm{tip}}^2}\right)+P_1{\left(\sqrt{1+\frac{V^4}{4v_0^4}}-\frac{V^2}{2v_0^2}\right)}^{1/2}+\frac{1}{2}{d}_0\rho sA{V}^3,$$

(6)

where $P_0$ and $P_1$ represent the blade profile power and induced power in hovering status, respectively, $U_{tip}$ is the tip speed of the rotor blade of the UAV, $v_0$ is the mean rotor induced velocity of the UAV in hovering, $d_0$ is the fuselage drag ratio, $\rho$ is the density of air, s is the rotor solidity, and A denotes rotor disc area. All the relevant parameters are explained in details in Table 3. In particular, when UAV is hovering, the speed of UAV is 0, the hover power is $P\left(0\right)$. Conversely, when the UAV is flying, the flight power is $P\left(V\right)$. In each slot, the UAV will fly to the m-th FPAP and hover over it until the offloaded task of the k-th GMU is completed. Thus, the energy consumption generated of the t-th time slot can be calculated by

$$\mathcal{E}\left(t\right)=P\left(V\right)T_f\left(t\right)+P\left(0\right)T_h\left(t\right),$$

(7)

where $T_f\left(t\right)$ is the flying time and $T_h\left(t\right)$ is the hovering time. Suppose at time slot t that the UAV is hovering over the m-th FPAP. The channel power gain $c_{k,m}\left(t\right)$ of the line of sight (LoS) link between the UAV and the k-th GMU is given by

$$\begin{array}{c}\hfill c_{k,m}\left(t\right)=\frac{{\rho }_0}{\sqrt{H^2+{\left[x_m^{\mathrm{UAV}}\left(t\right)-x_k^{\mathrm{GMU}}\left(t\right)\right]}^2+{\left[y_m^{\mathrm{UAV}}\left(t\right)-y_k^{\mathrm{GMU}}\left(t\right)\right]}^2}},\end{array}$$

(8)

where ${\rho }_0$ denotes the path loss per meter, and H is hovering height of the UAV. Following (8), the uplink data rate $R_{k,m}$ from the k-th GMU to the UAV at t is given by

$$R_{k,m}\left(t\right)=B{log}_2\left(1+\frac{P_t{c}_{k,m}\left(t\right)}{{\sigma }^2}\right),$$

(9)

where B is channel bandwidth, $P_t$ is the transmit power of the GMU, and ${\sigma }^2$ is the Gaussian noise power received by the UAV. The hovering time of the UAV at the t-th time slot can be calculated by

$$T_h\left(t\right)=\frac{{\mu }_k\left(t\right)}{R_{k,m}\left(t\right)},$$

(10)

where ${\mu }_k\left(t\right)$ is the amount of data that can be offloaded by the k-th GMU in the t-th time slot.

Table 3. Simulation parameter setting.

Parameter	Meaning	Value
$\mathrm{H}$	Flight height of the UAV	100 m
$\mathrm{V}$	Flight velocity of UAV	20 m/s
$P_0$	Blade profile power	79.9 W
$P_1$	Induced power	88.6 W
$U_{tip}$	Tip speed of the rotor blade of the UAV	120 m/s
$v_0$	The mean rotor induced velocity of the UAV	9 m/s
$d_0$	Fuselage drag ratio	0.48
$\rho$	Density of air	1.225 kg/m${}^3$
s	Rotor solidity	0.002
A	Rotor disc area	0.503 m${}^2$
${\rho }_0$	Path loss for a distance of 1 m	−50 dB
${\mathrm{P}}_{\mathrm{t}}$	User transmission power	0.1 W
${\sigma }^2$	Noise	−130 dB

The remaining energy of the UAV at time slot $(t+1)$ is given by

$$q(t+1)=q\left(t\right)-\mathcal{E}\left(t\right).$$

(11)

3.3. Problem Formulation

The GMU employs a full offload mode. That is, all data of the associated GMU are offloaded to the UAV and then calculated. The binary variable $b_k\left(t\right)\in \left\{0,1\right\}$ is defined to indicate whether the UAV receives offloading data from the k-th GMU during the t-th time slot. If $b_k\left(t\right)=1$, the UAV receives offloaded data from the k-th GMU; otherwise, there is no offloaded data from the k-th GMU to the UAV. Because, in each time slot, the UAV is only allowed to serve one GMU, it therefore satisfies that

$${\sum }_{k=1}^K{b}_k\left(t\right)=1,\forall t$$

(12)

Thus, the number of data bits offloaded from the k-th GMU to the UAV in t time slots is given by

$${\mu }_k\left(t\right)=\left({\sum }_{i=1}^t{b}_k\left(i\right)R_{k,m}\left(i\right)\right)\delta$$

(13)

In order to meet the quality of service (QoS) requirements for GMU k, ${\mu }_k\left(T\right)$ should satisfy

$${\mu }_k\left(T\right)\ge D_k,\forall k$$

(14)

where the constant $D_k$ is the minimum number of offloading data bits to guarantee the QoS of the GMU.The total number of data bits offloaded by all GMUs to the UAV is given by

$$\widehat{\mu }={\sum }_{k=1}^K{\mu }_k\left(t\right)$$

(15)

The total energy consumption of the UAV must be limited by $E_0$ during T time slots, which is expressed by

$${\sum }_{t=1}^T\mathcal{E}\left(t\right)\le E_0.$$

(16)

Let $\mathcal{L}=\left[x_m^{\mathrm{UAV}}\left(t\right),y_m^{\mathrm{UAV}}\left(t\right),H\right]$ be the location vector of the UAV, and let $\mathcal{B}=\left[b_k\left(t\right)\right]$ denote the GMU schedule indication. In order to maximize the total amount of the data that can be uploaded, under the constraints of the available energy of the UAV and the QoS of GMUs by planning the UAV’s trajectory, we formulate the optimization problem as

$$\begin{array}{cc}\hfill \mathrm{P}:& \underset{\mathcal{L},\mathcal{B}}{\max }\widehat{\mu }\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\sum }_{k=1}^K{b}_k\left(t\right)=1,\forall t,\hfill \\ \hfill & {\mu }_k\left(T\right)\ge D_k,\forall k,\hfill \\ \hfill & {\sum }_{t=1}^T\mathcal{E}\left(t\right)\le E_0.\hfill \end{array}$$

(17)

4. The DQN-Based Method for UAV Trajectory Planning

4.1. Reinforcement Learning and DQN

Reinforcement Learning is the science of decision-making. It is about learning the optimal behavior from interactions in order to achieve a goal. The learner and decision maker is called agent, while the things it interacts with is called environment.

Figure 2 illustrates the agent–environment interaction. The interaction occurs at each of t discrete time steps. At each time step, the agent observes the environment, receives a state $S\left(t\right)$ from the state space S and chooses an action $A\left(t\right)$ from the action space $\mathcal{A}$. Through action $A\left(t\right)$, the agent and the environment interact once, which makes the agent obtain a numerical reward $R\left(t\right)$ from the environment as a result of the previous action. In addition, then the agent comes into a new step $S(t+1)$.

Figure 2. The agent–environment interaction in reinforcement learning.

Reinforcement Learning methods achieve a goal usually by choosing a policy that can maximize the total amount of reward, called return $g\left(t\right)$:

$$g\left(t\right)\doteq R\left(t\right)+\gamma R(t+1)+{\gamma }^{2}R(t+2)+\dots =\sum _{k=0}^{\infty }{\gamma }^{k}R(t+k),$$

(18)

where $\gamma \in [0,1]$ is the discount factor. The discount factor $\gamma$ determine the impact of future returns. A factor of 0 causes the agent to only consider current rewards, whereas factor approaching 1 causes it to strive for high returns in the long term. The fundamental ideas behind reinforcement learning are well-introduced in surveys of reinforcement learning and optimal control [17].

Q-learning is one of the well-known reinforcement learning methods that can be employed with no knowledge of the environment. However, this method cannot be applied to an infinite continuous state space due to the limitation of Q-table. In 2015, DeepMind [18] was successful in combining Q-learning with neural networks and named the method DQN. The structure of DQN is shown in Figure 3. The authors applied a deep neural network to approximate the optimal action-value function, which represents the maximum of the total of rewards, and then to choose the policy.

Figure 3. DQN structure.

Furthermore, there were two key ideas in the DQN method. The first idea was to match the action-values to target values that were only periodically updated, thereby reducing correlations with the target. The second one applied a mechanism named experience replay in which random batch selection of samples can remove correlation in the observations of states and improve data distribution.

As is known, due to GMUs’ mobility, the channels between GMUs and the UAV may change quickly over time. In this case, the UAV has to select a suitable GMU to serve based on the changing channels. If the traditional Q-Learning method is adopted, it may fail to plan the UAV’s trajectory because of an oversized Q-table or excessive runtime. To accommodate the larger state space, the DQN that employs deep neural networks to estimate the optimal action value function $Q\left(s\left(t\right),a\left(t\right)\right)$ is adopted.

4.2. The DQN-Based Method for UAV Trajectory Planning

In order to improve the convergence rate and training effect, the DQN-based method we proposed applies the iterative updating process and the experience relay (ER) method to reduce the negative effects sequence correlation on the training results. In addition, it also applies the $\epsilon$-greedy method to balance the exploration and exploitation, for achieving the better estimations of the environment and also taking better actions. To plan trajectories of the UAV, the agent, action space, state space and reward function in the DQN-based method are defined as follows, where the UAV is regarded as an agent in the DQN, and its chosen strategy can be denoted as $\pi$.

4.3. Action Space

The UAV chooses a GMU at each time slot and flies to hover over the FPAP associated with that GMU to serve it. The action space $\mathcal{A}$ contains two kinds of actions, given by

$$\begin{array}{c}\mathcal{A}=\left\{a_{k,m}\left(t\right)\mid a_{k,m}\left(t\right)=\left\{a_k^{\mathrm{GMU}}\left(t\right),a_m^{\mathrm{FPAP}}\left(t\right)\right\}\right.,\\ k\in \{1,2,\dots ,K\},m\in \{1,2,\dots ,M\},\\ t\in \{1,2,\dots ,\mathrm{T}\}\}\end{array}$$

(19)

where $a_k^{\mathrm{GMU}}\left(t\right)$ indicates that the UAV chooses the k-th GMU in the t-th time slot, and $a_m^{\mathrm{FPAP}}\left(t\right)$ indicates that the UAV flies to the m-th FPAP at the t-th time slot. In the DQN-based method, action selection by an agent is accompanied by exploration and exploitation. In this paper, during choosing actions, we employ the $\epsilon$-greedy method to find a balance between exploration and exploitation, which introduces a parameter $\epsilon$ with a small value. The agent chooses the action with the largest value of action value function with probability $(1-\epsilon )$, and explores the environment with probability $\epsilon$. Thus, the agent has an $\epsilon$ probability of stepping out of the local optimum and finding a global optimum that is closer to the actual one. The $\epsilon -greedy$ method is given by

$$\pi (a\mid s)=\left\{\begin{array}{c}1-\epsilon +\frac{\epsilon }{\left|A\right(s\left)\right|}a=argmaxQ(s,a)\\ \frac{\epsilon }{\left|A\right(s\left)\right|}a\ne argmaxQ(s,a)\end{array}\right.$$

(20)

where $\left|A\left(s\right)\right|$ is the total number of the agent actions in state s. When the action a maximizes the action value function, the probability of choosing action a is $1-\epsilon +\frac{\epsilon }{\left|A\left(s\right)\right|}$, and the probability of exploring is $\frac{\epsilon }{\left|A\left(s\right)\right|}$ instead.

4.4. State

The state space of system is denoted as

$$\begin{array}{cc}\hfill S=& \left\{s\left(t\right)\mid s\left(t\right)=\left\{l_k^{\mathrm{GMU}}\left(t\right),l_m^{\mathrm{UAV}}\left(t\right),c_{k,m}\left(t\right),q\left(t\right),{\mu }_k\left(t\right)\right\}\right.\hfill \\ \hfill & k\in \{1,2,\dots ,K\},m\in \{1,2,\dots ,M\},\hfill \\ & t\in \{1,2,\dots ,T\}\}\hfill \end{array}$$

(21)

where $l_k^{\mathrm{GMU}}\left(t\right)$, $l_m^{\mathrm{UAV}}$, $c_{k,m}\left(t\right)$, ${\mu }_k\left(t\right)$, and $q\left(t\right)$ are previously given by (2), (4), (8), (11) and (14).

4.5. Reward

The UAV works as an ABS to communicate with the GMU to fulfill its computational tasks. When the GMU offloads the data, the system needs to provide some reward to encourage the UAV to serve this GMU, referring to [19]. Thus, the relationship between the offloading data ${\mu }_k\left(t\right)$ and the gain ${U(\varphi \mu }_k\left(t\right))$ is expressed in terms of sigmoid function, i.e.,

$$U\left(\varphi {\mu }_k\left(t\right)\right)=1-exp\left[-\frac{{\left(\varphi {\mu }_k\left(t\right)\right)}^{\eta }}{\varphi {\mu }_k\left(t\right)+\beta }\right]$$

(22)

where the constant $\varphi$ is the scaling factor of ${\mu }_k\left(t\right)$, and $\eta$ and $\beta$ are used to adjust the efficiencies of ${U(\mu }_k\left(t\right))$. The comparison of the ${U(\mu }_k\left(t\right))$ is shown in Figure 4. The system gain first increases rapidly as ${\mu }_k\left(t\right)$ rises and then becomes stable when ${\mu }_k\left(t\right)$ is large enough. According to Figure 4, in order to improve the training, we choose $\eta =2$ and $\beta =10$ as the best parameter pairs. Because, compared with other parameter pairs, it has a smoother curve and better numerical features. In addition, in terms of (12), the UAV is prevented from serving any single GMU for a long time and then ignoring other GMUs, thus ensuring the QoS of GMUs.

Figure 4. System income versus the data size with different $\eta$ and $\beta$.

The UAV is rewarded when it receives and processes offloading data transmitted by the GMU. However, at the same time, it also consumes energy due to flying and hovering. The system reward in time slot t can be obtained at the beginning of the next time slot $(t+1)$, so we design the system reward as

$$r(t+1)=U\left(\varphi {\mu }_k\left(t\right)\right)-\psi \mathcal{E}\left(t\right),$$

(23)

where $\psi$ is used to normalize $\mathcal{E}\left(t\right)$ and unify the units of $\mathcal{E}\left(t\right)$ and ${U(\varphi \mu }_k\left(t\right))$.

This reward function employs energy consumption as the penalty term and ${U(\varphi \mu }_k\left(t\right))$ as the reward term. It allows the UAV to maximize the amount of data that can be offloaded by GMUs via learning to complete more offloading computational tasks at relatively low energy consumption. The trajectory planning of the UAV is achieved based on the optimal GMU node selected by the reward function.

To correctly enhance system performance, the average gain g per training epoch is applied to demonstrate the average network throughput, which is given by

$$g={\sum }_{t=1}^{T}r(t+1)/T,t\in \{1,\dots ,T\},$$

(24)

To improve the QoS of GMUs, the average network reward is maximized. We aim to finding the optimal strategy ${\pi }^{*}$ based on the maximum average reward in each epoch. Thus, we maximize the average long-term network reward and increase the probability that the strategy in learning chooses an action depending on the current state $S\left(t\right)$ instead.

The state-action value function is given by

$$Q\left(s\left(t\right),a\left(t\right)\right)=\mathbb{E}\left[{\sum }_{i=t}^{T-1}\omega r(i+1)\mid s\left(i\right),a\left(i\right)\right]$$

(25)

where $\omega \in \left[0,1\right]$ is the discount factor, and $r(i+1)$ is immediate reward based on the state-action pair $(s_i,a_i)$ at the t-th time slot. This value function $Q\left(s\right(t),a(t\left)\right)$ is employed to evaluate the actions chosen by the UAV according to the state at time slot t. The DQN approximates the Q value by using two DNNs with different parameters, respectively, $\theta$ and ${\theta }^{-}$. One of the DNNs is a prediction network that takes the current state–action pair $\left(s\right(t),a(t\left)\right)$ as input, and the output is the predicted value $Q_{\mathrm{predicted}}^{\mathrm{DQN}}(s\left(t\right),a\left(t\right);\theta )$. The other is the target network that takes the next state $s(t+1)$ as input, and the output is the maximum Q value of the next state–action pair, so the target value of $\left(s\right(t),a(t\left)\right)$ is given by

$$Q_{\mathrm{target}}^{\mathrm{DQN}}\left(s\left(t\right),a\left(t\right);{\theta }^{-}\right)=r(t+1)+\omega \underset{a\prime }{max}Q\left(s(t+1),a^{\prime };{\theta }^{-}\right)$$

(26)

where the parameter $a^{\prime }$ is candidate action for the next state.

4.6. Algorithm Framework

For clarity, the overall training framework of the proposed DQN-based algorithm is summarized in Algorithm 1.

Algorithm 1: The DQN-based trajectory planning algorithm.

Input: Markov decision process $(S,\mathcal{A},P,R,{\rm Y})$, replay storage M, number of      cycles N, explore probability $\epsilon \in (0,1)$, deep learning networks Q, number      of steps to update the target network $\zeta$, and learning rate ${\alpha }_r$. Output: The optimal strategy $\mathcal{L}$. for$j=0$ to Ndo    According to $Q_{\theta }(S\left(j\right),\mathcal{A}\left(j\right))=\max {\mathrm{Q}}_{\theta }(\mathrm{s}\left(\mathrm{t}\right),\mathrm{a})$, choose any action in action     space $\mathcal{A}$ with the probability $\epsilon$, and choose $\mathcal{A}\left(j\right)$ with probability $(1-\epsilon )$.    Execute action $\mathcal{A}\left(j\right)$ and observe reward $R\left(j\right)$ from the environment to obtain     the next state $S(j+1)$.    Store $\left(S\left(j\right),\mathcal{A}\left(j\right),R\left(j\right),S(j+1)\right)$ from the environmental transfer process into     replay storage M.    Sample from a small batch of individual samples $\left(s\left(t\right),a\left(t\right),r\left(t\right),s(t+1)\right)$ in     replay storage M.    Calculate the target value $Y_t=r\left(t\right)+\gamma \underset{a\in A}{\max }{Q}_{\theta }\left(s\left(t\right)\mid a\right)$ in each conversion     process $\left(s\left(t\right),a\left(t\right),r\left(t\right),s(t+1)\right)$.    Updates parameters of the Q-network.    Execute gradient descent algorithm.     $\theta \leftarrow \theta -{\alpha }_r\frac{1}{n}{\sum }_{t\in \left[n\right]}\left[Y_t-Q_{\theta }\left(s\left(t\right),a\left(t\right)\right)\right]·{\nabla }_{\theta }{Q}_{\theta }\left(s\left(t\right),a\left(t\right)\right)$    Updates to the target network: after $\zeta$ steps, updates. ${\theta }^{-}\leftarrow \theta$ end return The optimal strategy $\mathcal{L}$;

5. Numerical Results and Discussion

We simulate an uplink UAV-assisted wireless network with a 500 m × 500 m region, with 5 GMUs and 25 FPAPs randomly distributed in the region. The simulation parameters are summarized in Table 3.

We first simulate loss function $J\left(\varphi \right)$ with different learning rates as shown in Figure 5. It can be seen that, when the learning rate $\alpha \le 0.05$, the loss function $J\left(\varphi \right)$ converges to near zero. Moreover, with $\alpha =0.01$, it converges with about 100 epochs, and the loss function curve becomes smooth. Thus, we select $\alpha =0.01$ as the learning rate in the subsequent simulations.

Figure 5. The loss functions with different learning rates.

Figure 6 shows the variation of the loss value versus the training steps. It is observed that the loss value decreases with oscillating before 200 epochs and converges to near zero with about 250 epochs. This indicates that the proposed method is able to converge well.

Figure 6. The loss versus the training steps.

Figure 7 compares the proposed DQN-based method with the traditional Q-learning based one. It shows that the proposed DQN-based method achieves much better performance in terms of average return. Moreover, it is seen that the traditional Q-Learning method stops running when the number of episodes is more than 500 while our proposed method can continue to run after 500 episodes, with a stable average return. This is because the Q-learning method stores a large amount of states and actions in the Q-table, which consumes a lot of memory, and the limited memory capacity terminates its running. In Figure 7, the average returns with different battery levels are also presented. Battery level means the available energy of the UAV, where battery level 1 is 100 KJ, battery level 2 is 200 KJ and battery level 3 is 300 KJ. It is seen that the higher the battery level, the more average return gain that is achieved by our presented DQN-based method, compared to the Q-learning based one.

Figure 7. Comparison of the proposed DQN based method with a traditional Q-learning method.

Figure 8 discusses the average throughput achieved by our presented method with different battery levels. Consistent with Figure 7, it is seen that the higher the battery level that UAV has, the longer the UAV runs, the larger number of GMUs that can be served, and the larger the average system throughput that can be achieved. Moreover, with battery level 1 and level 2, the average system converges at about 250 episodes, and, with battery level 3, the average system converges at about 500 episodes.

Figure 8. Average throughput versus Episodes.

In addition, we demonstrate the performance of the proposed method by comparing it with other UAV trajectories. Figure 9 plots some different UAV trajectories. In the figure, the red, green, blue, brown and orange curves portray the movement trajectories of five GMUs over a period of time, respectively, and 25 grey points denote the locations of 25 FPAPs. Different UAV trajectories are drawn in red straight lines. It is worth noting that, in the final subfigure, the trajectory of the UAV is hovering over the central FPAPs providing services to GMUs until the battery runs out.

Figure 9. Different UAV trajectories; (a) triangle trajectory 1; (b) triangle trajectory 2; (c) rectangle trajectory; (d) constant hovering.

Figure 10 and Figure 11 show the performance comparison versus different trajectories. Figure 10 shows the total run time and the total amount of offloaded data of the limited power UAV under different trajectories. We can see that the UAV running on the trajectory planned by the proposed method can not only run longer than other trajectories, but can also obtain a larger amount of offload data.

Figure 10. Total offload data versus time.

Figure 11. Total offload data versus different trajectories.

Figure 11 visualizes the total offload data amount of different trajectories, and the proposed method plans the trajectory with a larger offload data amount. Compared with other trajectories, the proposed method can increase the total offload data amount of the system by more than about 19%.

6. Conclusions

This paper studied the trajectory planning in the UAV-assisted mobile networks. Due to the mobility of the GMUs, the state space was quite large, and we proposed a DQN-based method. The simulation results show that the proposed DQN-based method outperforms a traditional Q-Learning-based one in terms of convergence and network throughput, and is able to maximize the system uplink throughput. According to our work, it can help power-limited UAV to make better decisions in real-time varying scenarios to achieve greater system throughput. In future work, further improvements can be made, such as using the hierarchical DQN or the deep deterministic policy gradient to make strategies and also attempt to research trajectory planning of multi-assisted UAV. Moreover, in our work, the flight height of the UAV is fixed, but in some high-low scenarios, it is also an interesting direction to research the 3D trajectory planning of the UAV.