Joint Optimization of Resource Utilization, Latency and UAV Trajectory in the Power Information Acquisition System

Abstract

In order to reduce the peak–valley difference of the power grid load, reasonably arrange users’ electricity consumption time and realize the intelligent management of the power grid, we construct a user electricity consumption information acquisition system based on unmanned aerial vehicles (UAVs) by using a sensor network. In order to improve the service quality of the system and reduce the system delay, this paper comprehensively considers the factors that affect the user’s electricity consumption information collection system, such as the UAV trajectory, the unloading decision of the data receiving point and so on. Therefore, this paper puts forward an effective iterative optimization algorithm for joint UAV trajectory and unloading decisions based on a deep Q network (DQN), in order to obtain the optimal UAV trajectory and unloading decision design, acquire the optimal solution to minimize the time delay of the monitoring system and maximize the service quality of the user electricity information collection system, thus ensuring the stable operation of the user electricity information collection system. In this paper, different complexity algorithms are used to solve this problem. Compared with the greedy algorithm, the proposed algorithm, CDQN, improves the system service quality by approximately $2\%$ and reduces the system delay by approximately $16\%$, so that the user’s electricity consumption information can be analyzed and processed faster.

1. Introduction

In power grid system management, the power grid load is often unbalanced. The question of how to balance the power grid load and reduce the gap between the peaks and valleys of the power grid load is of great research significance. Peak-shaving and valley-filling technology is applied in the power grid management system. The peak-shaving and valley-filling technology arranges the power generation and time reasonably according to the user’s electricity consumption and law and balances the power grid load. Therefore, the question of how to obtain the user’s electricity consumption information is very important. The existing user information collection technology can easily produce local optimal solutions, which makes the monitoring results inaccurate. A real-time monitoring method for power grid users’ energy consumption data based on the Internet of Things has been proposed [1]. In industrial production, the consumption of electric energy is very large. Most enterprises have the problem of repeated monitoring of energy consumption, so the optimization of energy consumption monitoring points is of great significance, and monitoring points are selected according to the fluctuation coefficients of energy efficiency [2]. In order to overcome the complexity of sensor equipment in data acquisition, transmission, storage and analysis, and to achieve the purpose of condition monitoring, estimation and control, a load monitoring system for chemical enterprises has been proposed to collect energy consumption data and analyze energy consumption, and an Elman neural network based on the sparrow search algorithm was proposed to predict the change and distribution trends of electricity consumption in the future production cycles of enterprises [3]. The above literature summarizes the latest applications in the field of sensing and state acquisition in modern industry.

Unmanned aerial vehicles (UAVs) have the advantages of high flexibility, low costs and easy implementation, so a UAV auxiliary network is considered an attractive solution with seamless coverage and high disaster tolerance [4]. The limited coverage of Mobile Edge Computing (MEC) needs to explore cooperation with UAV. UAVs can be combined with wireless sensor networks, and a certain model architecture can be built for it to complete the key tasks of computing in the future [5]. A dRA framework based on a dual depth Q network has been proposed, which can maximize the energy efficiency (EE) and total network throughput in a UAV auxiliary ground network [6]. A method to analyze the coverage probability of a UAV-assisted cellular network with incomplete beam alignment has been proposed, the influence of beam alignment error on coverage probability was studied, and the expression of coverage probability under line-of-sight and non-line-of-sight conditions was derived [7]. Considering the constraints of anti-collision and communication interference between UAVs, a joint optimization problem of aircraft communication scheduling, UAV power allocation and UAV trajectory optimization was proposed to maximize system throughput [8]. By considering the interference between the ground base station and the UAV, a UAV trajectory optimization problem was proposed to maximize the sum rate of edge users [9].

The question of how to reduce the complexity of model construction, plan the flight trajectory of UAVs and reduce flight energy consumption is a problem worthy of consideration. Therefore, many systems first optimize the trajectory of the UAV, and then consider other issues to reduce the complexity of the objective function. The communication area is divided into multiple sectors and assigned to each UAV, and then the order of communication areas that UAVs pass through is determined and the number of UAVs that maximize the time that UAVs stay in the communication area is deduced [10]. A simple circular UAV trajectory was proposed. Under this trajectory, the UAV’s flight radius and speed are jointly optimized to maximize energy efficiency [11]. In order to prolong the running time of unmanned aerial vehicles (UAVs) and the related network life, the problems of joint area division and UAV trajectory scheduling optimization were put forward [12]. By optimizing the trajectory of the UAV, the task completion time of the UAV is minimized, and, at the same time, the connection quality constraint of the UAV link specified by the minimum received signal-to-noise ratio target is obeyed [13]. A cooperative scheme of unmanned aerial vehicles (UAVs) was proposed, i.e., heuristic UAV replacement, which allows UAV relays to work individually one by one to maximize end-to-end throughput [14].

Based on the above description, this paper designs a secure communication system to collect users’ electricity consumption information. The system collects users’ electricity consumption information through sensors and sends it to the data receiving point, which sums up the received information and decides whether to upload the information to the UAV for analysis or local analysis. The frequency division multiple access (FDMA) method is used to connect data acquisition points with unmanned aerial vehicles. Therefore, the novelty and contributions of this paper can be highlighted in the following aspects.

(1) This paper analyzes the factors that affect the stable operation of the user’s electricity consumption information acquisition system, such as the unloading decision, UAV running trajectory and so on. In order to reduce the system delay and improve the service quality of the user electricity information acquisition system, a joint UAV trajectory and unloading decision algorithm (CDQN) is proposed. The optimal UAV trajectory and unloading decision design are obtained.

(2) In this paper, a multi-constraint hybrid programming for the joint optimization problem is proposed, and an effective iterative algorithm based on DQN is proposed, so as to obtain a good solution to reduce the system delay and maximize the system service quality, thus ensuring the stable operation of the user electricity information collection system.

(3) The unloading decision is a 0-1 integer optimization problem. Different complexity methods are proposed to solve this problem, and their performance differences are compared through simulation, so that they can be selected according to actual needs.

The paper is organized as follows. In Section 2, we provide a summary of the related literature. In Section 3, we outline the system model. Section 4 discusses the formulation and solution of the related problems. In Section 5, we analyze the system through simulation. Lastly, in Section 6, we present a detailed analysis and discussion of the entire article.

2. Related Work

In order to further improve the service quality of the system, an optimization algorithm can be used to obtain the optimal solution of the problem. For complex optimization problems, convex optimization can solve the optimal solutions of most problems, and for non-convex problems, it can still be solved by convex optimization algorithms, so convex optimization algorithms are still widely used. The UAV path is discretized into n line segments. Because this problem is not convex in the original form, successive convex approximation (SCA) is used to solve the joint problem of UAVs in trajectory design, task unloading and caching [15]. Large-scale and non-convex task scheduling problems are decoupled into a main problem and three subproblems. Among them, the task unloading of the main unloading problem is transformed into a typical knapsack problem to solve, and the sub-problem of computing resource allocation is solved by the Lagrange multiplier method [16]. Spatial and temporal variables are decoupled and solved hierarchically. In order to avoid collision, the lower layer solves spatial variables and uses ready-made convex optimization to deal with the strong sensitivity of polynomial coefficients and constraints. The upper layer optimizes the time variables, and the gradient of the time distribution is used to optimize the time distribution together with the gradient descent method [17]. Joint service caching, task unloading, communication and computing resource allocation and UAV layout optimization are established. This problem is a mixed integer nonlinear programming problem, which is decomposed into four sub-problems by block coordinate descent, and then the approximate optimal solution is obtained by an iterative algorithm of successive convex approximation [18]. A secure communication scheme of a non-orthogonal multiple access UAV-MEC system for flight eavesdroppers is proposed. According to the channel coefficient, transmission power, CPU calculation frequency, local calculation and UAV trajectory, successive convex approximation and block coordinate descent are used to solve this problem [19].

In order to deal with these problems better, evolutionary algorithms can be used to solve complex optimization problems, including particle swarm optimization, genetic algorithm, the ant colony algorithm and so on. Under the constraints of a deadline, waiting time and energy, an effective heuristic method of task assignment is proposed to generate feasible flights for each UAV. Finally, the Pareto-based index is introduced to evaluate the performance of the comparison algorithm [20]. A two-level joint optimization method has been proposed. In the outer layer, the particle swarm optimization algorithm combined with a genetic algorithm operator (PSO-GA) is used to optimize UAV deployment, aiming at minimizing the average task response time by jointly optimizing UAV deployment and calculating unloading [21]. A parallel particle swarm optimization algorithm based on a graphics processor was proposed to balance the cluster size and determine the shortest path along these clusters, while minimizing the flight time and energy consumption of the UAV [22]. A heuristic algorithm was proposed to minimize energy consumption and task execution delays, which consists of task allocation, differential evolution assistance and non-dominated sorting steps [23]. A heuristic algorithm based on particle swarm optimization (PSO) is proposed to jointly optimize task scheduling and UAV flight paths [24]. A large-scale path loss fuzzy C-means algorithm was proposed to predict the optimal positions of the UAVs [25].

In order to obtain the optimal solution faster and better, we can often yield unexpected results by simplifying the optimization problem and solving it. Under the constraints of the average energy consumption of UAVs and the stability of the data queue, the problem was expressed as multi-stage stochastic optimization, and the multi-stage stochastic problem was transformed into a deterministic problem per slot with much fewer optimization variables by Lyapunov optimization [26]. An iterative subchannel allocation and speed optimization algorithm was proposed to jointly solve the subchannel allocation and UAV speed optimization problem [27]. A new method based on rolling optimization (RHO) was proposed, which greatly reduces the number of optimization variables and the computational complexity for each problem [28]. Binary constraints of mixed integer non-convex optimization problems were transformed into a series of equivalent equality constraints, and a penalty-based algorithm was proposed to obtain suboptimal solutions [29]. A clonal selection algorithm was proposed, which clones the rechargeable UAV into several non-rechargeable virtual UAVs and searches the trajectory for each virtual UAV to reduce the computational complexity of mixed integer linear programming [30].

3. System Model

As shown in Figure 1, in order to better implement the technology of peak shaving and valley filling, this paper constructs a power grid energy consumption monitoring system, so as to monitor the power consumption of users in real time. The power grid energy consumption monitoring system is realized by the sensor network, which deploys multiple sensor nodes to monitor users’ voltage, current, power consumption time and other information. The sensor nodes transmit the collected user information to the ground data receiving point, and the ground data receiving point uploads the user information to realize real-time data processing and analysis, so as to make better use of the peak-shaving and valley-filling technology, reasonably arrange users’ power consumption time, reduce the power grid load trough, fill the power grid load trough and balance the power grid load.

The power grid energy consumption monitoring system constructed in this paper includes m different data receiving points and unmanned aerial vehicles, and the data receiving point set record is $\mathcal{M}=\{1,\cdots ,m,\cdots ,M\}$. The sensor transmits the collected user information to the ground data receiving point, which receives and sorts out the received user information. The UAV starts from the starting point, flies along a certain trajectory and is connected with the ground users during the flight. The ground users consider whether to upload the local information to the UAV for calculation or to calculate locally. In particular, the power grid energy consumption monitoring system constructed in this paper focuses on the collection of users’ electricity consumption information, ignoring the process of transmitting users’ information from sensor nodes to ground data receiving points, focusing on the process of collecting information from data receiving points, analyzing and calculating data and introducing edge computing technology assisted by drones to realize real-time data analysis and preprocessing. Considering the heterogeneity of computing resources and communication resources, we assume that the communication bandwidth of UAV is B. In time slot t, the unloading decision from data receiving point m to the UAV is expressed as ${\rho }_t^{m,u}\in \left\{0,1\right\}$. When ${\rho }_t^{m,u}=1$, the data receiving point m chooses to upload data to the UAV, and if ${\rho }_t^{m,u}=0$, the data receiving point chooses to process them locally.

3.1. Communication Model

The coordinate of the m-th user is expressed as $\left(x_m,y_m\right)$; before the UAV communicates with the data acquisition point, the UAV keeps flying at a fixed height h from the ground during the flight, and the coordinates of the UAV are expressed as $\left(X_t,Y_t\right)$, so the distance between the m-th data acquisition points and the UAV can be expressed as

$$d_t^{m,u}=H^2+{\left(X_t-x_m\right)}^2+{\left(Y_t-y_m\right)}^2$$

(1)

For the air-to-ground channel, it is assumed that there is nothing in the line-of-sight (LoS) path from the UAV to the ground. In addition, due to the long distance between the ground data receiving points, the interference between the ground data receiving points can be ignored. Therefore, it can be reasonably assumed that the channel from the UAV to the ground data receiving point is the LoS channel. Using a simple channel model, the channel gain is controlled by the LoS link, so the channel power gain between the m-th data acquisition point and the UAV can be written as

$$h_t^{m,u}=\frac{|g_0{|}^2}{{\left(d_t^{m,u}\right)}^2}$$

(2)

where $d_t^{m,u}$ represents the distance between the drone and the data acquisition points m in the time slot t. $g_0$ is the reference channel gain, and the reference distance is 1 m, which obeys the complex Gaussian distribution, i.e., $g_0\zeta \left(0,1\right)$.

In the task unloading stage, the UAV adopts the directional antenna method and orthogonal frequency division multiplexing technology to reduce the interference between the data acquisition points and UAV. $h_t^{m,u}$ is the channel gain of data acquisition point m and the UAV. According to the Shannon formula theorem, the achievable uplink transmission rate of data acquisition point m and the UAV can be expressed as

$$\begin{array}{cc}\hfill r_t^{m,u}& =B{log}_2\left(1+\frac{p^m{h}_t^{m,u}}{{\sigma }^2}\right)\hfill \\ & =B{log}_2\left(1+\frac{p^m{\left|g^0\right|}^2}{{\sigma }^2\left(H^2+{\left(X_t-x_m\right)}^2+{\left(Y_t-y_m\right)}^2\right)}\right)\hfill \end{array}$$

(3)

where B is the communication bandwidth of the UAV, $p^m$ is the uplink transmission power of data acquisition point m and ${\sigma }^2$ is the power spectral density of additive Gaussian white noise.

3.2. Time Delay Model

For each data acquisition point $m\in \mathcal{M}$, there is a task amount $N_t^m=\left\{D_t^m,C_t^m\right\}$ that needs to be completed in each time gap, where $D_t^m$ is the data amount of the task generated by the data acquisition point, and $C_t^m$ is the number of CPU revolutions required to calculate the 1-bit task.

For each time slot t, the task amount $D_t^m$ can be arbitrarily divided into local calculation and UAV-side calculation. It is worth noting that in the construction of the system model in this paper, the processing of power grid user information is emphasized, and the timely processing of information data is very important for the stable operation of the system. Therefore, this paper considers the local processing delay required to process information and the transmission delay of uploading information from the data receiving point to the UAV.

When the local processing of the data receiving point is relatively idle, the collected user information can be calculated on the local side according to the principle of proximity—at this time, ${\rho }_t^{m,u}=0$. In order to effectively use the local time delay of the data acquisition point, the time delay of data acquisition point m in local task calculation is as follows:

$$T_{loc}^m=\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}$$

(4)

where $f_t^m$ is the frequency of local calculation of the data acquisition point.

When the local computing is busy, if the user information is still unloaded to the local computing, it will take a long time to allow the data acquisition point to establish communication with the UAV and then upload a certain amount of tasks to the UAV for calculation. A UAV assisted by a sensor network can alleviate the conflict between devices, and, at this time, ${\rho }_t^{m,u}=1$. In time slot T, the communication delay in uploading some tasks to the UAV at data acquisition point m can be written as

$$\begin{array}{cc}\hfill T_{t,up}^{m,u}& =\frac{{\rho }_t^{m,u}{D}_t^m}{r_t^{m,u}}\hfill \\ & =\frac{{\rho }_t^{m,u}{D}_t^m}{Blog\left(1+\frac{p^m{\left|g^0\right|}^2}{{\sigma }^2\left(H^2+{\left(X_t-x_m\right)}^2+{\left(Y_t-y_m\right)}^2\right)}\right)}\hfill \end{array}$$

(5)

where $r_t^{m,u}$ indicates the transmission rate of user m in the t time slot.

3.3. System Service Quality Model

In order to improve the service quality of the whole user electricity information collection system and reduce the system delay, this paper puts forward a system service quality model. First, the data acquisition point is connected with the UAV, and then the data acquisition point decides whether to unload the data to the UAV for data analysis and processing. In particular, the position of each data acquisition point is fixed, while the trajectory of the UAV is variable, and the service quality of the system is mainly determined by the transmission rate between the UAV and data acquisition point. The greater the transmission rate, the more data the UAV will process and analyze, and the higher the service quality of the user electricity information collection system. In order to measure the service quality of the system, we use the $\Phi \left(·\right)$ function as a measure. The $\Phi \left(·\right)$ function maps the rate to the utility function of the system quality of service, which is defined as follows:

$$\sum _m\Phi \left(r_t^{m,u}\right)$$

(6)

where the service quality of the system is the sum of the service quality of all data acquisition points, and $\Phi \left(·\right)$ reflects the influence of data acquisition point m on the service quality of the system. The greater $r_t^{m,u}$ is, the higher the service quality of the system is.

4. Problem Formulation and Proposed Solution

The core aim of this paper is to reduce the system delay and improve system service quality. Therefore, this paper considers the optimization problems that affect the system delay and service quality, including the trajectory of the UAV and the unloading decision. The data acquisition point collects and sorts out the information transmitted by the sensor and then uploads the information to the drone for analysis and processing. On this basis, a multi-constraint hybrid optimization problem based on the UAV trajectory and unloading decision is proposed in this paper. The convex optimization algorithm is used to solve the optimization problem of the unloading decision, and a DQN is used to solve the optimization problem of the UAV trajectory. Through many explorations of the agent, until the trajectory and unloading decision of the UAV are fixed, the objective function value of the problem P converges, and finally the trajectory and unloading decision of the UAV are obtained. Therefore, through the above scheme, we can state that a high-quality suboptimal solution has been obtained.

4.1. Problem Building

In this part, we construct the joint unloading decision and the optimization problem of the UAV’s trajectory to minimize the system delay and maximize the service quality of the system to complete all the tasks of the UAV. Note that the unloading decision of all data acquisition points is $\mathsf{\rho }$, and the trajectory of the UAV is $\left(\mathit{X},\mathit{Y}\right)$; then, the optimization problem is as follows:

$$\begin{array}{cc}\hfill \mathit{P}:\underset{\left(\mathit{X},\mathit{Y}\right)}{max}& \sum _t\sum _m\mathsf{\Phi }\left({\mathrm{r}}_{\mathrm{t}}^{m,u}\right)-\mathrm{w}\left({\mathrm{T}}_{\mathrm{t},\mathrm{loc}}^{\mathrm{m}}+{\mathrm{T}}_{\mathrm{t},\mathrm{up}}^{\mathrm{m},\mathrm{u}}\right)\hfill \\ \hfill & =\underset{\left(\mathit{X},\mathit{Y}\right),\mathsf{\rho }}{max}\sum _t\sum _m\mathsf{\Phi }\left({\mathrm{r}}_{\mathrm{t}}^{\mathrm{s}}\right)-\mathrm{w}\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}-w\frac{{\rho }_t^{m,u}{D}_t^m}{r_t^{m,u}}\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\mathrm{s}.\mathrm{t}.\hfill & \mathrm{C}1:\hfill & \rho \in \left\{0,1\right\}\hfill \\ \hspace{1em}\hfill & C2:\hfill & \left({\mathrm{X}}_0,{\mathrm{Y}}_0\right)={\mathrm{q}}_0,\left({\mathrm{X}}_{\mathrm{T}},{\mathrm{Y}}_{\mathrm{T}}\right)={\mathrm{q}}_{\mathrm{T}}\hfill \end{array}$$

(8)

w is the proportion of system delay in the objective function. From the problem $\mathit{P}$, we can conclude that the higher the transmission rate between the data acquisition point and the UAV, the higher the service quality of the system. For the unloading decision, the lower the unloading decision, the longer the local calculation time, and the higher the unloading decision, the longer the uploading task time of the data collection point. Constraint (1) restricts the data acquisition point to choose whether to unload the task or not to unload it to the UAV; Constraint (2) restricts the starting and ending positions of the UAV.

From the above, the problem $\mathit{P}$ is a mixed optimization problem and a non-convex optimization problem, because the problem P contains a non-concave objective function. In particular, the problem of UAV trajectory optimization is a non-convex optimization problem, which cannot be solved directly by standard convex optimization technology.

4.2. Trajectory Optimization of UAV

When solving problem $\mathit{P}$, in order to improve the service quality of the system, it is necessary to optimize the trajectory of the UAV according to the state of the UAV at each moment. The following equation relates to the trajectory optimization of the UAV:

$$\begin{gathered} \begin{array}{cc}\hfill \mathit{P}\mathbf{1}:\underset{\left(\mathit{X},\mathit{Y}\right)}{max}& \sum _t\sum _m\mathsf{\Phi }\left({\mathrm{r}}_{\mathrm{t}}^{m,u}\right)-w\left({\mathrm{T}}_{\mathrm{t},\mathrm{loc}}^{\mathrm{m}}+{\mathrm{T}}_{\mathrm{t},\mathrm{up}}^{\mathrm{m},\mathrm{u}}\right)\hfill \\ \hfill & =\underset{\left(\mathit{X},\mathit{Y}\right)}{max}\sum _t\sum _m\mathsf{\Phi }\left({\mathrm{r}}_{\mathrm{t}}^{m,u}\right)-w\left(\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}+\frac{{\rho }_t^{m,u}{D}_t^m}{r_t^{m,u}}\right)\hfill \\ \hfill & =\underset{\left(\mathit{X},\mathit{Y}\right)}{max}\sum _t\sum _m\Phi \left(r_t^{m,u}\right)-w\frac{{\rho }_t^{m,u}{D}_t^m}{B{log}_2\left(1+\frac{p^m{\left|g_0\right|}^2}{{\sigma }^2\left(H^2+{\left(X_t-x_m\right)}^2+{\left(Y_t-y_m\right)}^2\right)}\right)}\hfill \\ \hfill & -w\sum _t\sum _m\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}\hfill \end{array} \\ s.t.\left({\mathrm{X}}_0,{\mathrm{Y}}_0\right)={\mathrm{q}}_0,\left({\mathrm{X}}_{\mathrm{T}},{\mathrm{Y}}_{\mathrm{T}}\right)={\mathrm{q}}_{\mathrm{T}} \end{gathered}$$

(9)

By combining the above equations with the basic knowledge of convex optimization, it can be deduced that the problem P1 is a non-convex optimization problem and cannot be solved by traditional convex optimization techniques. It can be solved by transforming the non-convex problem into a convex problem or by using deep reinforcement learning.

In the following, we will first give the definition of Markov decision making based on UAV path planning and optimize the motion direction of the UAV at every time gap n. By a tuple $\left\{s,a,\tau ,r,s_{\_}\right\}$, s is the system state, a is the movement action and r is a reward function. Therefore, the MDP problem can be defined as follows.

(1) s: In time slot n, the system state is defined as the coordinate set of the UAV at the current moment $\left(X_t,Y_t\right)$.

(2) a: In time slot n, the UAV can move in one direction, which is mainly represented by u, d, h, e.

(3) $s_{\_}$: In time slot n + 1, the next state of the system is defined as the UAV coordinate set at the next moment $\left(X_{t+1},Y_{t+1}\right)$.

(4) r: In order to solve the problem P, we use the target value of the problem P as the reward function R, which is defined as

$$r=\mathsf{\Phi }\left({\mathrm{r}}_{\mathrm{t}}^{\mathrm{s}}\right)-\mathrm{w}\left({\mathrm{T}}_{\mathrm{t},\mathrm{loc}}^{\mathrm{m}}+{\mathrm{T}}_{\mathrm{t},\mathrm{up}}^{\mathrm{m},\mathrm{u}}\right)$$

(10)

In order to speed up the convergence of the algorithm, when the UAV receives data, the flight energy consumption of the UAV is used as the penalty function, and the penalty function is as follows:

$$\phi \left(a\right)=\beta e_t$$

(11)

Because of the randomness of agents, it is difficult to model the state transition probability $\tau$. Because the action space and state space in the MDP problem are discrete, we use the DQN method to make decisions and divide the Q value into the state value and action value, which avoids overestimating the Q value and further improves the learning performance.

4.3. Unloading Decision Optimization

When the UAV reaches a position, the unloading decision between the data acquisition point and the UAV is optimized, and the optimization formula is as follows:

$$\begin{gathered} \begin{array}{cc}\hfill \mathrm{P}2:\underset{\mathsf{\rho }}{min}& \sum _t\sum _m\mathrm{w}\left({\mathrm{T}}_{\mathrm{t},\mathrm{loc}}^{\mathrm{m}}+{\mathrm{T}}_{\mathrm{t},\mathrm{up}}^{\mathrm{m},\mathrm{u}}\right)\hfill \\ \hfill & =\underset{\mathsf{\rho }}{min}\sum _t\sum _m\mathrm{w}\left(\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}+\frac{{\rho }_t^{m,u}{D}_t^m}{r_t^{m,u}}\right)\hfill \end{array} \\ s.t.{\rho }_t^{m,u}\in \left\{0,1\right\} \end{gathered}$$

(12)

The optimization problem of unloading decision $\mathsf{\rho }$ mentioned above is an integer optimization problem. The unloading decision can be relaxed and changed from an integer optimization problem of 01 to a continuous optimization problem between 0 and 1. After relaxing the unloading decision $\mathsf{\rho }$, the following equation can be obtained:

$$\begin{gathered} \begin{array}{cc}\hfill \mathrm{P}2:\underset{\mathsf{\rho }}{min}& \sum _t\sum _m\mathrm{w}\left({\mathrm{T}}_{\mathrm{t},\mathrm{loc}}^{\mathrm{m}}+{\mathrm{T}}_{\mathrm{t},\mathrm{up}}^{\mathrm{m},\mathrm{u}}\right)\hfill \\ \hfill & =\underset{\mathsf{\rho }}{min}\sum _t\sum _m\mathrm{w}\left(\frac{\left(1-{\rho }_t^{m,u}\right)D_t^m{C}_t^m}{f_t^m}+\frac{{\rho }_t^{m,u}{D}_t^m}{r_t^{m,u}}\right)\hfill \end{array} \\ s.t.\rho \in \left[0,1\right] \end{gathered}$$

(13)

It can be concluded that the optimization problem of the unloading decision is convex and can be solved by a convex optimization algorithm.

In the process of learning, the agent iterates the unloading decision and the trajectory of the UAV until it finds a high-quality suboptimal solution.

We propose an iterative algorithm with low complexity, which aims at minimizing the system delay and maximizing the service quality of the system. Firstly, the trajectory of the UAV is randomly initialized to obtain the position of the UAV, and the unloading decision is solved by the convex optimization algorithm. Then, the solution is obtained by calculating the system delay and the system service quality from the obtained unloading decision. After repeated iterations of the agent, until the unloading decision and the trajectory of the UAV do not change, and the target value converges to the predefined accuracy, we can state that we have found a suboptimal solution to this problem. The specific algorithm process is shown in Algorithm 1.

Algorithm 1 Convex optimization and DQN (CDQN)

1: Initialize UAV status, user unloading decision.   2: repeat   3:    repeat   4:      Initialize UAV starting point $\left(X_0,Y_0\right)$ and $t=0$.   5:      Convex optimizer solves the user unloading decision ${\rho }_t$ by Equation (13).   6:      Obtain the next position of the UAV $\left(X_{t+1},Y_{t+1}\right)$ and the current reward $r_{t+1}$.   7:      Convex optimizer solves the user unloading decision ${\rho }_{t+1}$ by Equation (10).   8:    until The drone reaches the finish line.   9:    The agent returns to the starting point for training and learning. 10: until The target no longer changes.

5. Analysis of Simulation Results

The simulation results show that the proposed algorithm is effective. In the simulation, we assume a sensor-based user electricity information acquisition system, in which the positions of data acquisition points are randomly deployed in a certain range. The CDQN algorithm proposed in this paper is simulated and compared with the FD, relax and random algorithms.

5.1. Simulation Parameters

The simulation parameters are shown in

Table 1. Simulation parameters.

Variable	Parameter	Value
w	Proportional parameter of time delay	10
S	Distribution area	480 × 240
$f_t^m$	Frequency of local calculation of data acquisition point	1 × 10${}^8$
C	Local computing resources	500
$D_t^m$	User data volume	3∽6
$p^m$	Data transmission power	20 mW

below.

5.2. Simulation Result Analysis

Figure 2 shows the convergence of the algorithm proposed in this paper. As can be seen from the figure, from the beginning, the convergence curve of the algorithm proposed in this paper rises with the increase in the iteration times. After a period of iteration, the UAV trajectory and unloading decision will not change, and then the algorithm will gradually converge. From the convergence curve of the algorithm, it can be seen that the CDQN algorithm proposed in this paper converges after 200 iterations, so we can conclude that the CDQN algorithm proposed in this paper can also obtain high-quality approximate suboptimal solutions. Because the solution obtained by reinforcement learning is often not the optimal solution, there is a certain gap between the results obtained in this paper and the optimal solution, but the CDQN algorithm proposed in this paper also obviously improves the performance.

Figure 3 shows the target values of the system under different algorithms. For UAVs, under the constraint of the UAV trajectory, they will always reduce the transmission delay as much as possible and improve the service quality of the system, so UAVs will choose data receiving points close to the maximum communication rate as much as possible. As can be seen from the figure, under the same constraint conditions, the target value of the CDQN algorithm is the largest, followed by the relax algorithm, and there is little difference between the FD algorithm and random algorithm. For the relax algorithm, in the process of optimizing the unloading decision, the unloading decision is relaxed, and the unloading decision is changed from 0 or 1 to any value between 0 and 1, so the data access point will upload data to the UAV for calculation according to a certain unloading ratio, and the remaining data will be kept for calculation at the local end. The CDQN algorithm proposed in this paper iteratively solves the unloading decision and UAV trajectory many times until the obtained results remain unchanged. The random algorithm optimizes the problem and does not comprehensively consider the unloading decision, so the data uploaded to the drone are uncertain. Therefore, the CDQN algorithm can reduce the system delay as much as possible, thus greatly improving the system service quality.

Figure 4 shows the system quality of service under different algorithms. The service quality of the CDQN algorithm system is the highest, and it is not notably different from the FD algorithm proposed in this paper, because the FD algorithm uses the UAV trajectory based on the CDQN algorithm, and the UAV trajectory is related to the transmission rate between the UAV and the data acquisition point. The better the UAV trajectory, the higher the service quality of the system, so the service quality of the CDQN algorithm system is not notably different from the FD algorithm. The relax algorithm and random algorithm use a fixed UAV trajectory. It can be seen that in the CDQN algorithm proposed in this paper, the performance of the UAV trajectory obtained by the DQN algorithm is better. The CDQN algorithm proposed in this paper optimizes the trajectory and unloading decision of the UAV as a whole, maximizes the service quality of the system, reduces the system delay and optimizes the objective function value of the problem. Therefore, it can be concluded that the CDQN proposed in this paper improves the performance of the system and is more accurate, which is very important for the realization of the solution.

Figure 5 shows the influence of different algorithms on the system delay. The system delay of the CDQN algorithm proposed in this paper is the smallest, followed by the relax algorithm, and there is little difference between the FD algorithm and random algorithm. The CDQN algorithm proposed in this paper relaxes the unloading decision, changes the unloading decision from 0 or 1 to any value between 0 and 1 and then uses the convex optimization algorithm to find the optimal unloading decision. The FD algorithm adopts the path planned by the CDQN algorithm proposed in this paper and uses the greedy algorithm to solve the unloading decision problem of the data acquisition points, so as to shorten the time of uploading information from the data acquisition points to the UAV as much as possible, which makes the unloading decision unable to balance the transmission delay and calculation delay, resulting in a further delay increase. The relax algorithm relaxes the constraint of the unloading decision problem, so the system delay of the relax algorithm is lower than that of the FD algorithm and random algorithm, but the CDQN algorithm has a lower system delay. The random algorithm is random and unstable. It can be seen that the CDQN algorithm proposed in this paper is more accurate and stable than the other algorithms. It can be seen from the above that the CDQN algorithm proposed in this paper effectively reduces the transmission delay of the system.

6. Conclusions

This paper mainly considers a user electricity information collection system, and the intelligent management of the user electricity information collection system needs to process and analyze the data uploaded by the data receiving point in time. In order to improve the service quality of the system and reduce the system delay, this paper uses a UAV auxiliary system to analyze and process the data. For the target problem constructed in this paper, the DQN algorithm and convex optimization algorithm are iteratively combined to solve the joint optimization problem, in which DQN solves the trajectory problem of the UAV and the convex optimization algorithm solves the unloading decision problem of the data acquisition points. The simulation results show that compared with other algorithms, the CDQN algorithm can effectively improve the system service quality and reduce the system delay, thus improving the sensor utilization. In future work, we will strive to further improve the accuracy of the algorithm and reduce its complexity in order to improve the algorithm. In addition, a more realistic background will be considered in the optimization problem.