UAV-Assisted Privacy-Preserving Online Computation Offloading for Internet of Things

Abstract

Unmanned aerial vehicle (UAV) plays a more and more important role in Internet of Things (IoT) for remote sensing and device interconnecting. Due to the limitation of computing capacity and energy, the UAV cannot handle complex tasks. Recently, computation offloading provides a promising way for the UAV to handle complex tasks by deep reinforcement learning (DRL)-based methods. However, existing DRL-based computation offloading methods merely protect usage pattern privacy and location privacy. In this paper, we consider a new privacy issue in UAV-assisted IoT, namely computation offloading preference leakage, which lacks through study. To cope with this issue, we propose a novel privacy-preserving online computation offloading method for UAV-assisted IoT. Our method integrates the differential privacy mechanism into deep reinforcement learning (DRL), which can protect UAV’s offloading preference. We provide the formal analysis on security and utility loss of our method. Extensive real-world experiments are conducted. Results demonstrate that, compared with baseline methods, our method can learn cost-efficient computation offloading policy without preference leakage and a priori knowledge of the wireless channel model.

1. Introduction

With the rapid development of unmanned aerial vehicles (UAVs), they are applied in the various applications, such as data collection and remote sensing among Internet of Things (IoT) sensors [1]. Although the benefits of high mobility, swift deployment, and low economic cost, large-scale application of UAV is limited by the computation capacity and energy. Recently, computation offloading is regarded as a promising solution for enabling UAV-assisted IoT to process huge data produced by IoT sensors [2,3].

Existing computation offloading methods for IoT focus on two main categories, i.e., one-shot optimization methods [4,5] and DRL-based methods [6,7]. Compared with one-shot optimization methods, the DRL-based methods can assist devices to learn computation offloading policy with higher energy efficiency and low time delay. Besides this benefit, DRL-based methods allow the devices to learn computation offloading policy without a priori knowledge of wireless channel model, which can be applied to solve the wireless channel dynamic between the UAV and IoT sensors [8].

Although the benefits of applying DRL into computation offloading, the vulnerabilities in DRL can be exploited by adversaries to interfere the UAV with learning policy [9], which hinders it from being applied to the real world. Figure 1 provides a case of computation offloading preference leakage over UAV-assisted IoT. The adversary misleads the UAV to offload tasks to malicious BSs by inversing the RL algorithm based on the observations of the offloading decision and the transmission radio link status.

1.1. Related Works and Challenges

(1) DRL-based Computation Offloading in UAV-Assisted MEC Networks: Zhou et al. [10] formulated the computing task scheduling problem as a constrained Markov decision process (CMDP) and solved it by proposing a novel risk-sensitive DRL method, where the UAV’s energy consumption violation is defined as the risk metric. Liu et al. [11] modified the vanilla Q-Learning algorithm to maximize the profit of the UAV under the constant cruising path. Recently, multi-UAV scenario also has been taken into consideration. Wei et al. [12] proposed a distributed DRL-based method that is enhanced by prioritized experience replay (PER), denoted as DDRL-PER. In their work, the proposed DDRL-PER method is adopted to solve the computation offloading problem over multi-UAV MEC network. Zhu et al. [13] decomposed the complex task and proposed a DRL-based method for the task offloading over UAV group, which optimizes the policy under the constraints of energy, dynamic network state. Seid et al. [14] designed a collaborative learning framework for computation offloading and resource allocation over multi-UAV-assisted IoT, where the UAV group was divided into multiple clusters. Then, the distributed deep deterministic policy gradient (DDPG) algorithms was adopted to solve the multi-cluster computation offloading problem. Sacco et al. [15] also proposed a multi-agent DRL-based method to solve the multi-tasks offloading problem over multi-UAV networks. Moreover, Gao et al. [16] combined game theory with DRL to solve the joint optimization of task offloading and multi-UAV trajectories.

Challenge 1: Existing DRL-based methods for computation offloading over UAV-assisted IoT take protecting computation offloading preference into consideration.

(2) Privacy Preserving in MEC: Considering the privacy preserving has not been widely researched over UAV-assisted MEC networks, we extend our review to the architectures of MEC networks that are not limited to UAV-assisted MEC networks. The privacy issues related to DRL-based offloading method were firstly investigated by He et al. [17]. The privacy constraints were generated in the value function, and the computation offloading policy was learned within several training episodes. Then, the authors adopted Q-learning method to solve the private problem. Furthermore, an active suppression method was proposed in Reference [18], which can prevent adversaries from eavesdropping. However, this method needs to modify the hardware, such as the antenna of mobile devices, in order to effectively suppress adversaries with different signal types. In addition, this active suppression method leads to excessive hardware modification costs when it is applied on a large scale UAV-assisted IoT. To improve the privacy-preserving learning efficiency of computation offloading policy, Min et al. [19] utilized the transfer learning and Dyna architecture to accelerate the learning speed, while He et al. [20] proposed the RL- and DRL-based methods based on generic framework of Lyapunov optimization, which is resistant to user presence inference attack. Existing works holds the assumption that there is no malicious BSs; thus, they mainly generate private constraints in value function to preserve the privacy. However, malicious BSs cannot be completely avoided in the real world. Once the value function is obtained by adversaries, the adversaries can compromise the mobile devices to offload computation tasks to malicious BSs.

Challenge 2: Existing privacy-preserving works that are designed for the DRL-based computation cannot prevent the malicious BSs from inferring the value function of the DRL algorithm.

1.2. Contributions

To solve the aforementioned privacy issues in existing works, we propose a differential privacy (DP)-based deep Q-learning (DP-DQL) method to solve the computation offloading preference leakage issue over UAV-assisted IoT. Our contributions can be summarized as follows.

• We investigate a new privacy leakage issue within the online computation offloading over UAV-assisted IoT, namely computation offloading preference leakage.
• We propose a differential privacy-based deep Q-learning (DP-DQL) method to protect computation offloading preference over UAV-assisted IoT. In the proposed DP-DQL method, the DQL is adopted as the basic framework for efficiently learning computation offloading policy without the a priori knowledge of the wireless channel model. Then, a generated Gaussian noise is generated in the policy updating process of DQL, which can protect the computation offloading preference. Finally, the learning speed of DP-DQL is accelerated by the PER technique [21] by replying the experience with high temporal-difference error.
• We provide theoretical analysis for the differential privacy guarantee and utility loss. Then, the convergence, privacy protection, and cost efficiency of our method is demonstrated by extensive real-world experiments. The results show that our method can help UAV learn the cost-efficient computation offloading policy with the differential privacy guarantee.

The rest of this paper is organized as follows. Section 2 gives the necessary background, system model and problem formulation, details of the proposed method and theoretical analysis. Then, we design and conduct the experiments in Section 3. We discuss the impact of key parameters on the convergence and the limitations of the proposed method in Section 4. Finally, we conclude this paper in Section 5.

2. Materials and Methods

In this section, we first provide the background techniques. Then, we describe the system model and formulate the research problem of this paper. Finally, we show the proposed DP-DQL method and give the theoretical analysis in terms of privacy guarantee and utility loss.

2.1. Background Techniques

2.1.1. Differential Privacy

Differential privacy [22,23] establishes a strong standard for privacy guarantees in knowledge transfer, which aims to disable data analysis algorithms from distinguishing between two neighboring inputs. The key definitions are provided in the following.

Definition 1.

For any two neighboring inputs $z,z{}^{\prime }\in \mathcal{B}$ and subset of outputs $\mathcal{D}\subseteq \mathcal{E}$, the ($\alpha ,\mathcal{𝓎}$)-differential privacy can be guaranteed once the mechanism $\mathcal{C}:\mathcal{B}\to \mathcal{D}$ satisfies the inequality

$${\displaystyle \frac{1}{e^{\alpha }}}\mathbb{P}(\mathcal{C}\left(z\right)\in \mathcal{D})⩽\mathbb{P}(\mathcal{C}\left(z{}^{\prime }\right)\in \mathcal{D})+\mathcal{𝓎}.$$

(1)

The definition of output’s global sensitivity is shown as follows.

Definition 2.

Given $\forall z,z{}^{\prime }\in \mathcal{B}$ as neighboring inputs, the output’s global sensitivity can be computed as

$${\mathrm{\Gamma }}_{\mathcal{C}}={sup}_{z,z{}^{\prime }\in \mathcal{B}}\parallel \mathcal{C}\left(z\right)-\mathcal{C}\left(z{}^{\prime }\right)\parallel ,$$

(2)

where $\parallel ·\parallel$ represents the norm function in $\mathcal{E}$.

2.1.2. Deep Q-Learning

Deep Q-learning [24] leverages the deep neural network to approximate the value function, which aims to find a policy ${\Pi }^{*}(·)$ that can minimize the Bellman error as follows.

$${\displaystyle \frac{1}{2}}{(Q^{\Pi }(s_t,a_t)-\mathbb{E}[u_t+\tau {max}_{a{}^{\prime }}{Q}^{\Pi }(s{}^{\prime },a{}^{\prime })])}^2.$$

(3)

2.2. System Model and Problem Formulation

2.2.1. System Model

Figure 1 shows a UAV-assisted MEC system for IoT, which contains N fixed base stations (BSs), denoted as a set $\mathcal{N}=\{1,2,...,N\}$ and a UAV. $(x_n,y_n,0)$ is the coordinate of BS n, and $(x,y,h)$ is the coordinate of the UAV, where h indicates the flight height of UAV. Referring to Reference [25], UAV adopts Wi-Fi or LTE technology to communicate with BS and smart factories. At each time slot t, a computation task $T_t$ (e.g., pattern recognition) is collected by the UAV from a smart factory, where $T_t\in \mathcal{T}$. The task $T_t$ is described as $T_t=(D_t,C_t)$, where $D_t$ is the maximum execution time, and $C_t$ is the bits of task $T_t$. Morevoer, every $\xi$ CPU cycles can process a bit in the task.

Due to the binary computation offloading is the special case of partial computation offloading [26,27], we investigate partial offloading in this paper for generality. To process a task, the UAV needs to decide how much of a task should be offload to BSs. Formally, $s_t$ represents the offloaded proportion of a task $T_t$. To improve the performance of computation offloading decision, the UAV can adjust the offloaded proportion of a task. We define the CPU frequency of the BS and UAV as $f^n$ and f, respectively. We assume that the each BS has the same CPU frequency, which can avoid some of fallacies of computing during BS parallel processes tasks [28]. Hence, the cost model is shown in the following. The time spent for locally processing a task $P_t^L$ is

$$P_t^L={\displaystyle \frac{(1-s_t)C_t\xi }{f}}.$$

(4)

The local energy consumption $E_t^L$ is

$$E_t^L=(1-s_t)C_t{\left(f\right)}^2\xi \beta ,$$

(5)

where the $\beta$ is a coefficient related to the CPU architecture [29].

The cost on offloading task to BS n consists of two parts, i.e., the time cost and the energy cost. The time $P_t^O$ for offloading and processing task is

$$P_t^O={\displaystyle \frac{s_t{C}_t}{r_t^n}}+{\displaystyle \frac{s_t{C}_t\xi }{f^n}},$$

(6)

where the $C_t{s}_t/r_t^n$ is the time for transmitting the task to BSs, and $\xi C_t{s}_t/f^n$ represents the processing time in the BS.

Based on References [30,31], the energy consumption on transmitting a task to BS n depends on the transmit power $EP$, which can be shown as

$$E_t^O=EP{\displaystyle \frac{s_t{C}_t}{r_t^n}}.$$

(7)

In this paper, we assume that the BSs have sufficient energy. This is reasonable that fixed BSs are usually deployed in the area that has wired power supplied by grid. Hence, the energy consumption on the BS is not considered in this paper. For convenience, the major notations used in this paper are summarized in

Table 1. List of major notations.

Notations	Descriptions
n	The index of BSs
N	The number of BSs
$\mathcal{N}$	The set of BSs
$x_n,y_n$	The x-coordinate and y-coordinate of BS n
$x,y,h$	The x-coordinate, y-coordinate, and height of UAV
t	Time index
$f^n,f$	The CPU frequency of n-th BS
f	The CPU frequency of the UAV
$T_t,\mathcal{T}$	The computation task in time slot t and the set of computation task
H	The reset factor of the DP-DQL
$P_t^O,E_t^O$	The time and energy consumption at time slot t in BSs
$C_t,D_t$	The bits and maximum execution time of task $T_t$
$P_t^L,E_t^L$	The consumed time and energy to locally process task at time slot t
$EP$	The transmit power of transmitting a bit from BS n to the UAV
$a_t,s_t,u_t$	The action, state, and reward of the DP-DQL in t-th time slot
$TP,V$	The number of training episode and the maximum learning steps within a training episode
$\tau ,\gamma$	The discount factor and learning rate of the proposed DP-DQL
$A,\mathcal{Z}$	The mini-batch size and the replay buffer
$\xi$	The bits which be processed during a CPU cycle
$r_t^n$	The radio link transmission rate between the UAV and BS n
$\Psi$	The balance factor of the DP-DQL

.

2.2.2. Threat Model and Privacy Issue

In this paper, we consider a new privacy leakage issue over UAV-assisted IoT, namely computation offloading preference leakage, which is shown in Figure 1. In this paper, we assume that the adversary knows the inputs and formats of UAV’s computation offloading policy in advance. It is reasonable to make the assumption because:

• The BSs can provide customized services for UAV based on the formats of UAV’s computation offloading policy and the inputs of the policy.
• Once the adversary induce the BS, it can monitor the inputs and formats of UAV’s computation offloading policy.

In accordance with above assumptions, the adversaries monitor UAV’s computation offloading decision and the input of computation offloading policy, e.g., radio link transmission rate between UAV and BSs. The adversaries utilize an algorithm, such as inverse reinforcement learning algorithm, to infer the UAV’s computation offloading preference based on monitoring results. Furthermore, the adversaries construct specific inputs for UAV’s computation offloading policy, e.g., improving radio link transmission rate with the help of malicious BSs, which can mislead UAV to offload computation tasks onto the malicious BSs.

2.2.3. Design Goals

To solve above privacy issues, our proposed method should reach the goals as follows.

• Differential privacy guarantee: DP-DQL method should provide $(\alpha ,\mathcal{𝓎})$-differential privacy for UAV during learning process so that the value function of the UAV’s computation offloading policy will not be inferred by the adversaries based on the system state and offloading decision.
• Minor utility loss: DP-DQL method should guarantee that, compared with the traditional DQL method, the performance of the DP-DQL method will not be significantly degraded by adding the differential privacy mechanism.

2.2.4. Problem Formulation

As claimed in Section 2.1.2, we formulate the problem of privacy-preserving computation offloading in UAV-assisted MEC network for IoT as the Markov decision process (MDP), which is defined as a tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$.

 (1)

System state: The system state is the offloaded proportion. Formally, $s_t\in \mathcal{S}$ ranges from 0 to 1.

 (2)

Action space: The UAV adjusts the offloaded proportion of a task by increasing or decreasing from 0 to 0.25. Formally, $a_t\in [0,0.25]$.

 (3)

Reward function: The weighted average of energy and time costs is adopted as the reward function, which is given as follows:

$$u_t=-\eta (E_t^O+E_t^L+P_t^L+P_t^O),$$

(8)

where $\eta$ is the normalization function. To meet the differential private requirements, the value domain of the reward function is constrained from 0 to 1. The proof will be given in Section 2.4.1.

2.3. DP-Based Deep Q-Learning for Computation Offloading

In this section, we firstly give an overview of the DP-DQL. Then, the details for the DP-DQL are provided.

2.3.1. Overview

The steps of online learning are shown in Figure 2, which consists of four stages.

• Initialization: Initializing the parameters used in DP-DQL approach.
• Exploring: The UAV executes offloading action and obtains reward from the environment.
• Generating differential disturbance: The UAV generates the specific Gaussian noise to prevent the computation offloading preference leakage.
• PER-based policy updating: The UAV updates computation offloading policy with the help of PER technique.

Algorithm 1 shows the details of the DP-DQL method, and its description is given as follows.

Algorithm 1 Differential Privacy-based Deep Q-Learning for computation offloading method

1: Initialize the parameters of DP-DQL method; 2: for$j\in [1,TP]$do 3:     Reset the environment; 4:     Reset differential dict $l(·)$; 5:     for $t\in [0,V-1]$ do 6:      Conduct the action $a_t=arg{max}_{a}l\left(s_t\right)+{\Pi }_{\zeta }(s_t,a)$; 7:      Reach to the state $s_{t+1}$ and get a reward $u_t$; 8:      Compute the maximum priority ${\mathcal{𝓏}}_t$ via Equation (9) and store it with transition; 9:      if $t\equiv 0modA$ then 10:          for $i\in [1,A]$ do 11:           Generating differential disturbance ${\delta }_t$; 12:           Sample transition via Equation (13); 13:           Compute importance-sampling weight via Equation (14); 14:           Compute TD-error via Equation (9); 15:           Update the priority of the transition; 16:           Compute accumulated policy gradient $\psi$ by Equation (15); 17:          end for 18:          Softly updating; 19:      end if 20:     end for 21: end for

2.3.2. Initialization (Lines 1–4)

The online policy $\Pi (·)$ and target policy $\Pi {}^{\prime }(·)$ are initialized with random weights $\zeta$ and $\zeta {}^{\prime }$ (Lines 1), where the target policy $\Pi {}^{\prime }(·)$ is used to slow the updating rate of online policy $\Pi (·)$ and, hence, improve the stability of the algorithm. The environment is constructed for learning computation offloading policy. Then, the differential dict $l(·)$ is initialized and reset to $NULL$ every $TP/H$ training episodes. (Line 4). The differential dict $l(·)$ is defined as a Dictionary structure, where the key size and value size are set to $\left|\mathcal{A}\right|$ and 2, respectively.

2.3.3. Exploring (Lines 5–8)

If a task $T_t$ is collected by UAV at time slot t, the UAV makes offloading decision by online computation offloading policy $\Pi \left(s_t\right)$ under differential disturbance $l\left(s_t\right)$ (Line 6). After receiving the reward $u_t$ (Line 6), the UAV obtains a new state $s_{t+1}$ (Line 7). Based on old state $s_t$, action $a_t$, new state $s_{t+1}$, and reward $u_t$, the UAV constructs a transition. Then, the UAV compute the priority ${\mathcal{𝓏}}_t$ and store it in replay buffer $\mathcal{Z}$ with the transition (Line 8). Based on Reference [21], the priority ${\mathcal{𝓏}}_t$ can be computed as follows:

$${\mathcal{𝓏}}_t=|{\Pi }_{\zeta }(s_t,a_t)-\Theta |.$$

(9)

The $\Theta$ in Equation (9) is given as follows:

$$\Theta =\left\{\begin{array}{ccc}\hfill & u_t& ift==V\hfill \\ \hfill & \Pi {}^{\prime }(s_{t+1},a{{}^{\prime }}_{t+1})\tau +u_t& t<V\hfill \end{array}\right.,$$

(10)

where $a{{}^{\prime }}_{t+1}=\Pi {}^{\prime }\left(s_{t+1}\right)$.

2.3.4. Generating Differential Disturbance (Lines 9–11)

Once replay buffer is filled with the transitions (Line 9), a mini-batch of the transitions will be sampled from replay buffer to update online and target policies (Line 10). Gaussian process ${\delta }_t\sim \mathcal{Y}({\upsilon }_t,{\varphi }_t)$ generates a differential disturbance $y_t$ for each action $a\in \mathcal{A}$ (Line 11). The differential disturbance ${\delta }_t$ is appended to differential dict $l\left(s_t\right)\leftarrow {\delta }_t$, then $l(·)$ is sorted. The ${\upsilon }_t$ and ${\varphi }_t$ are given below based on [32]:

$${\upsilon }_t={\displaystyle \frac{(e^{\Phi ϵ}-e^{-\Phi ϵ})l\left(s_{t-1}\right)+(e^{\Phi \Omega }-e^{-\Phi \Omega })l\left(s_{t+1}\right)}{e^{\Phi \Lambda }-e^{-\Phi \Lambda }}},$$

(11)

$${\varphi }_t=1-[{\displaystyle \frac{(e^{\Phi ϵ}-e^{-\Phi ϵ})e^{\Phi ϵ})+(e^{\Phi \Omega }-e^{-\Phi \Omega })e^{\Phi \Omega })}{e^{\Phi \Lambda }-e^{-\Phi \Lambda }}}],$$

(12)

where $\Phi ={(4(1+\Psi )/A)}^{-1}\gamma$, $\Psi$ represents the balance factor, $ϵ={\parallel s_t-s_{t-1}\parallel }_2$, $\Lambda ={\parallel s_{t+1}-s_{t-1}\parallel }_2$, and $\Omega ={\parallel s_{t+1}-s_t\parallel }_2$.

2.3.5. PER-Based Policy Updating (Lines 12–21)

In this stage, the accumulated policy gradient $\psi$ is calculated based on PER technique. Specifically, the sampling probability of a transition is computed by (Line 12)

$$\mathbb{P}\left(i\right)={\displaystyle \frac{{\mathcal{𝓏}}_i^{{\theta }_1}}{{\sum }_{\mathcal{ℊ}}{\mathcal{𝓏}}_{\mathcal{ℊ}}^{{\theta }_1}}},$$

(13)

where the $\mathcal{ℊ}$ is the index of a transition. Then, the function of importance sampling weight (Line 13) is to decrease bias referred to Reference [21], which is given as follows:

$${{\rm Y}}_i={\displaystyle \frac{{\left(\right|\mathcal{Z}\left|p\mathbb{P}\left(i\right)\right)}^{-{\theta }_2}}{{max}_A{{\rm Y}}_i}},$$

(14)

where $\mathbb{P}(·)$ is the sampled probability, $\left|\mathcal{Z}\right|$ represents the replay buffer size, ${\theta }_2$ is used to determine how much priority affects the sampling probability, and ${{\rm Y}}_i$ is an importance factor of transition. Hence, the TD-error is calculated via Equation (9) (Lines 14), and the absolute TD-error value is adopted to update the ith transition priority (Lines 15). Finally, the accumulated policy gradient $\psi$ is computed based on a chain rule in Reference [33] as shown below (Line 16):

$$\psi \leftarrow \psi +l\left(s_i\right)+{{\rm Y}}_i{\nabla }_{\zeta }\Pi (s_i,a_i){\mathcal{𝓏}}_t,$$

(15)

where ${\nabla }_{\zeta }\Pi (s_i,a_i)$ is the gradient of the online policy for vector $(s_n,a_n)$. Then, the online policy $\Pi (·)$ is softly update as follows:

$${\zeta }^{\Pi }=\zeta +\psi \omega ,$$

(16)

where $\omega$ is a soft update coefficient (Line 18). Then, set $\psi \leftarrow 0$. Finally, the target policy $\Pi {}^{\prime }(·)$ is updated (Line 18).

2.4. Theoretical Analysis

2.4.1. Differential Privacy Guarantee

To analyze the differential privacy guarantee of the DP-DQL method under the adversary model in Section 2.2.2, we firstly provide a necessary theorem.

Theorem 1.

Given the sample path k from Gaussian process $\mathcal{F}(0,{\rho }^{2}K)$, ${max}_{\mu \in [0,1]}k\left(\right)$ exists with high probability. For each $w>0$ in Sobolev space $G^1$, we have

$$\mathbb{P}(w+8.68\sqrt{\Phi }\rho ⩽{max}_{⪵\in [0,1]}k(⪵))⩽{\displaystyle \frac{1}{e^{w^2/2}}},$$

(17)

where the $\Phi =\frac{1}{4\gamma (1+\Psi )/A}$.

Proof. 

Firstly, we define some necessary notations. For a sample path k, we define $k_{0p}=\{k\left({ℯ}_0\right),k\left({ℯ}_2\right),\dots ,k\left({ℯ}_{2p}\right)\}$ and $k_{1p}=\{k\left({ℯ}_1\right),k\left({ℯ}_3\right),\dots ,k\left({ℯ}_{2p-1}\right)\}$, where $p⩾1$, ${ℯ}_{\mathcal{𝒾}}=\mathcal{𝒾}/2p$, $(\mathcal{𝒾}=0,\dots ,2p)$.

Then, we consider the base case that the sample path k contains two elements, i.e., $k_2=k\left(0\right),k\left(1\right)$. Therefore, the expectation $\mathbb{E}$ is

$$\begin{array}{cc}\hfill \mathbb{E}(max{k}_2)& ={\displaystyle \frac{1}{2}}\mathbb{E}\left(\right|k\left(0\right)-k\left(1\right)\left|\right)\hfill \\ & =\sqrt{{\displaystyle \frac{(1-e^{-\Phi })}{\pi }}}\rho \hfill \\ & ⩽\sqrt{{\displaystyle \frac{\Phi }{\pi }}}\rho ,\hfill \end{array}$$

(18)

which is based on $k\left(0\right)-k\left(1\right)\sim \mathcal{F}(0,2(1-e^{-\Phi }){\rho }^2)$.

Finally, we consider the case of $\left|k\right(⪵\left)\right|>2$. Specifically, we aim to bound the expectation $\mathbb{E}(max{k}_p)$ for all $p>1$. Referring to Chernoff bound, we have

$$e^{\mathcal{𝓉}\mathbb{E}({max}_{\mathcal{𝒾}}{\mathcal{𝒿}}_{\mathcal{𝒾}})}⩽\mathbb{E}(e^{\mathcal{𝓉}{max}_{\mathcal{𝒾}}{\mathcal{𝒿}}_{\mathcal{𝒾}}})⩽p{e}^{{\mathcal{𝓉}}^2{\rho }^2/2},$$

(19)

where ${\mathcal{𝒿}}_{\mathcal{𝒾}}$ is the p independent Gaussian random variables based on $\mathcal{F}(0,{\rho }_{\mathcal{𝒿}}^2)$. Let $\mathcal{𝓉}=\sqrt{2lnp}/{\rho }_{\mathcal{𝒿}}$, we have ${max}_{\mathcal{𝒾}}{\mathcal{𝒿}}_{\mathcal{𝒾}}⩽\sqrt{2lnp}/{\rho }_{\mathcal{𝒿}}$.

Let ${\mathcal{𝓂}}_p=\mathbb{E}(max{k}_{2^{p}0})$. Due to $k_{2^{p}0}\subset k_{2^{p+1}0}$, the series ${\mathcal{𝓂}}_p$ is non-decreasing. Then, we derive the upper bound of ${\mathcal{𝓂}}_{p+1}-{\mathcal{𝓂}}_p$. Referring to Reference [32], we have $\exists \mathcal{𝒿},\mathbb{E}(max(0,max{k}_{2^{p}1}-k_{2^{p}0}))⩽\mathbb{E}(max(0,\mathcal{𝒿}))$. The bound of $\mathbb{E}(max(0,\mathcal{𝒿}\left)\right)$ is given as follows.

$$\begin{array}{cc}\hfill e^{{\displaystyle \frac{\mathbb{E}(\mathcal{𝒿},0)}{\sqrt{\Phi /2^p}\rho }}}& ⩽\mathbb{E}(e^{{\displaystyle \frac{\mathbb{E}(\mathcal{𝒿},0)}{\sqrt{\Phi /2^p}\rho }}})\hfill \\ & ⩽\mathbb{E}(max(e^{{\displaystyle \frac{\mathcal{𝒿}}{\sqrt{\Phi /2^p}\rho }}},1))\hfill \\ & ⩽\mathbb{E}(e^{1+{\displaystyle \frac{\mathcal{𝒿}}{\sqrt{\Phi /2^p}\rho }}})\hfill \\ & ⩽e^{\sqrt{p}+1}+1.\hfill \end{array}$$

(20)

Hence,

$$\mathbb{E}(max(\mathcal{𝒿},0))⩽(\sqrt{p}+1+{\displaystyle \frac{1}{e^{\sqrt{p}+1}}})\sqrt{{\displaystyle \frac{\Phi }{2^p}}}\rho .$$

(21)

Finally, we have

$${\mathcal{𝓂}}_{p+1}-{\mathcal{𝓂}}_p⩽(\sqrt{p}+1+{\displaystyle \frac{1}{e^{\sqrt{p}+1}}})\sqrt{{\displaystyle \frac{\Phi }{2^p}}}\rho .$$

(22)

Based on induction, we can get $\forall p$,

$${\mathcal{𝓂}}_p⩽\sqrt{{\displaystyle \frac{\Phi }{\pi }}}\rho +\sum _{\mathcal{𝒾}}^{\infty }(\sqrt{\mathcal{𝒾}}+1+{\displaystyle \frac{1}{e^{\sqrt{\mathcal{𝒾}}+1}}})\sqrt{{\displaystyle \frac{\Phi }{2^{\mathcal{𝒾}}}}}\rho <8.68\sqrt{\Phi }\rho .$$

(23)

Referring to Reference [34], $\mathbb{E}(maxk)$ shares the same upper bound of ${\mathcal{𝓂}}_p$ almost surely when k is continuous with probability one. Hence, Theorem 1 follows. □

Theorem 2.

The proposed DP-DQL method ensures the ($\alpha ,\mathcal{𝓎}+H{e}^{-{(2\Psi -8.68\sqrt{\Phi }\rho )}^2/2}$)-differential privacy once neighboring rewards ${\parallel u{}^{\prime }-u\parallel }_{\infty }⩽1$, if

$$\rho <{\displaystyle \frac{\Psi }{8.68\sqrt{\Phi }}},$$

(24)

and

$$\rho ⩾1.41\sigma \sqrt{ln{\displaystyle \frac{1.25}{\mathcal{𝓎}}}ln({\displaystyle \frac{\alpha }{\mathcal{𝓎}}}+e){\displaystyle \frac{V}{A}}},$$

(25)

where $\Phi ={(4{(A)}^{-1}\gamma (1+\Psi ))}^{-1}$, $\sigma ={\displaystyle \frac{X}{A}}\sqrt{(8\gamma (1+\Psi ))(A+16\gamma (1+\Psi ))}$, X represents the Lipschitz constant, A is the mini-batch size, and V is the training episodes.

Proof. 

To prove this theorem, we firstly prove that the neighboring value functions cannot be distinguished with differential privacy guarantee in one policy updating step. Then, we extend conclusion to multiple policy updating steps. Referring to Reference [35], we have

$${\parallel {\Pi }^{*}(·)-\Pi (·)\parallel }_{\infty }⩽{\displaystyle \frac{\gamma X(l(s_{t+1}-l\left(s_t\right))+2)}{A}},$$

(26)

where $\Pi (·)$ is the value function from u and ${\Pi }^{*}(·)$ is the value function from $u^{*}$. Note that ${\parallel u^{*}-u\parallel }_{\infty }⩽1$. The inequality $|\Pi (·)-c|⩽2\gamma X(1+\Psi )/A$ holds with at least probability of $1-e^{-{(2\Psi -8.68\sqrt{\Phi }\rho )}^2/2}$, where ${\Pi }^{\#}(·)$ is the neighboring value function of $\Pi (·)$, according to Theorem 1. Then, for each ${\parallel u^{*}-u\parallel }_{\infty }⩽1$, ${\parallel {\Pi }^{*}(·)-\Pi (·)\parallel }_{\infty }⩽4\gamma X(1+\Psi )/A$ holds based on triangle inequality with the same ${\Pi }^{\#}(·)$. Let $k={\Pi }^{*}(·)-\Pi (·)$, we can obtain Equation (27) in the Sobolev space.

$${\parallel k\parallel }_G^2⩽(1+\frac{\Phi }{2}){({\displaystyle \frac{4\gamma X(1+\Psi )}{A}})}^2+{\displaystyle \frac{X^2}{2\Phi }}.$$

(27)

Let $\Phi =B/(4\gamma X(1+\Psi ))$, and the Equation (27) can be rewritten as

$${\parallel k\parallel }_G^2⩽{\displaystyle \frac{16X^2{\gamma }^2{(1+\Psi )}^2+4\gamma A{X}^2(1+\Psi )}{A^2}}.$$

(28)

Referring to Reference [36], we can get

$$\mathbb{P}[\underset{u,u{}^{\prime }}{max}\parallel u{}^{\prime }-u\parallel ⩽\alpha ]>1-(\mathcal{𝓎}+H{e}^{-{\displaystyle \frac{{(2\Psi -8.68\sqrt{\Phi }\rho )}^2}{2}}}),$$

(29)

by adding Gaussian disturbance $l\sim \mathcal{F}(0,{\rho }^{2}K)$ to $\Pi (·)$ within a policy updating step on the basis of Equation (25). This conclusion can be generalized to multiple policy updating steps by the theorem in Reference [37]. Hence, Theorem 2 is proved. □

2.4.2. Minor Utility Loss

Before giving the final proof of utility loss, we show a necessary theorem and its proof.

Theorem 3.

Assume that ${\mathcal{U}}_a^{\#}$ is the optimal result of the inequality constraint problem

$$\begin{array}{cc}\hfill \underset{{\mathcal{U}}_0,{\mathcal{U}}_1,s,{\mathcal{U}}_n}{maximize}& \sum _a{\mathcal{U}}_a^{T}u{{}^{\prime }}_a\hfill \\ \hfill s.t.& \sum _a{\mathit{e}}^T{\mathcal{U}}_a⩽{\displaystyle \frac{\left|\mathcal{S}\right|}{1-\tau }},\hfill \\ \hfill & {\mathcal{U}}_a⩾0\hfill \\ \hfill & \sum _a(\mathbb{I}-\tau {\chi }_a^T){\mathcal{U}}_a=\mathit{e},\hfill \end{array}$$

(30)

and we can obtain

$$\mathbb{E}[\sum _a{\mathcal{U}}_a^{\#T}{u}_a]⩾\sum _a{\mathcal{U}}_a^{*T}{u}_a-{\displaystyle \frac{2\sqrt{2}\left|\mathcal{S}\right|\rho }{\sqrt{\pi }(1-\tau )}}.$$

(31)

Proof. 

Given $u_a^{\#}=u_a+{\delta }_a$, we can get Equation (32) based on the strong duality and non-negativity in (♠) and (▪), respectively.

$$\begin{array}{cc}\hfill \mathbb{E}[\sum _a{\mathcal{U}}_a^{\#T}{u}_a]& =\mathbb{E}[\sum _a{\mathcal{U}}_a^{\#T}(u_a^{\#}-{\delta }_a)]\hfill \\ \hfill & ⩾\mathbb{E}[\sum _a{\mathcal{U}}_a^{*T}(u_a^{\#}-\sum _a{\mathcal{U}}_a^{\#T}{\delta }_a)]\hfill \\ \hfill & =\mathbb{E}[\sum _a{\mathcal{U}}_a^{*T}(u_a+{\delta }_a)-\sum _a{\mathcal{U}}_a^{\#T}{\delta }_a]\hfill \\ \hfill & \stackrel{♠}{=}\sum _a{\mathcal{U}}_a^{*T}{u}_a+\mathbb{E}[\sum _a{({\mathcal{U}}_a^{*}-{\mathcal{U}}_a^{\#})}^T{\delta }_a\hfill \\ \hfill & \stackrel{\blacksquare }{⩾}\sum _a{\mathcal{U}}_a^{*T}{u}_a-{\displaystyle \frac{2}{\left|\mathcal{S}\right|(1-\tau )}}\mathbb{E}[\sum _a{\parallel {\delta }_a\parallel }_1]\hfill \\ \hfill & =\sum _a{\mathcal{U}}_a^{*T}{u}_a-{\displaystyle \frac{2\sqrt{2}\rho \left|\mathcal{S}\right|}{\sqrt{(1-\tau )\pi }}}.\hfill \end{array}$$

(32)

□

Finally, we prove the convergence of the proposed method through Theorem 4.

Theorem 4.

Compared with vanilla DQL, the utility loss of the our DP-DQL method tends to be 0 even under the worst case, where $H=1$.

Proof. 

We have Equation (33) by solving Equation (30)

$$\mathbb{E}[\sum _a{u}_a{\mathcal{U}}_a^{\#T}]⩾\sum _a{u}_a{\mathcal{U}}_a^{*T}-{\displaystyle \frac{2\sqrt{2}\rho \left|\mathcal{S}\right|}{\sqrt{\pi }(1-\tau )}},$$

(33)

according to Theorem 3. Further, according to Reference [32], the equation $\mathbb{E}[u^{\#}{e}^T]=\mathbb{E}[{\sum }_a{u}_a{\mathcal{U}}_a^{\#T}]$ holds. Based on the strong duality, we have ${\sum }_a{u}_a{\mathcal{U}}_a^{*T}=u^{*}{e}^T$. As $\mathbb{E}{[\parallel u^{*}-u^{\#}\parallel ]}_1={\mathcal{U}}^T{u}^{*}-\mathbb{E}[e^T{u}^{\#}]$. Considering state space is infinity, the upper bound of $\mathbb{E}{[\parallel u^{*}-u^{\#}\parallel ]}_1$ tends to be zero. Moreover, Reference [24] guarantees the convergence of vanilla DQL. Therefore, our DP-DQL method achieves minor utility loss compared with vanilla DQL and converges within finite training episodes. □

3. Results

In this section, we design four experiments to evaluate the convergence, privacy protection and cost efficiency of the DP-DQL method.

3.1. Experiment Settings

Scenario: In this paper, the UAV flies around the area at a constant height to collect data and make computation offloading decision based on its offloading policy. The device for locally processing task is a Raspberry Pi 3B+, which is adopted as the airborne computer. Figure 3a shows the architecture of UAV used for experiments, and the flying area is shown in Figure 3b. In this paper, we firstly randomly deploy three laptops in Figure 3b to represent the BSs. Then, we deploy three actual rodeside units (RSU) to evaluate the variation of results. Because the computing power of actual RSU is similar to that of Raspberry Pi [38,39], the actual RSU is represented by the Raspberry Pi 4.

Parameter settings: The radio link transmission rate from n-th BS to the UAV is $r_t^n\in$ {2 Mb/s, 6 Mb/s, 10 Mb/s}. At each time slot t, the values of data size $C_t$ for the task $T_t$ is $C_t\in$ {20 Mb, 40 Mb, 60 Mb}, and each task should be processed within $D_t=3$ s. During task processing, each bit needs $\xi =1000$ CPU cycles to process [40]. As defined in Reference [41], the transmit power form BSs to the UAV $EP$ is $0.2$ W. The proposed DP-DQL method is implemented by Pytorch 1.1 and Python 3.6. We adopt a four-layer fully-connected feedforward neural network to implement the online and target policies. The learning rate $\gamma$ is set to 0.001, while the discounted factor $\tau$ is set to 0.999. The replay buffer size $\left|\mathcal{Z}\right|$ is 1024. The size of mini-batch A is 128. During online learning, there are $TP=100$ training episodes and $V=50$ learning steps in every training episode. For convenience, the all values of parameters are summarized in

Table 2. Parameter values.

Parameter	Value	Parameter	Value
h	5 m	$C_t$	{20, 40, 60} Mb
f	$1\times {10}^9$ cycles/s	$f^n$	$3\times {10}^9$ cycles/s
$D_t$	3 s	$\xi$	1000
$r_t^n$	{2, 6, 10} Mb/s	$\beta$	$1\times {10}^{-11}$
$EP$	0.2 W	$TP$	100
V	50	N	3
$\gamma$	0.001	$\tau$	0.999
${\theta }_1$	0.5	${\theta }_2$	0.5
$\left|\mathcal{Z}\right|$	1024	A	128
${\theta }_1$	0.5	${\theta }_2$	0.5

.

3.2. Baseline Methods

We evaluate the efficiency of our DP-DQL method by comparing it with two baseline methods:

• Greedy: This method has been widely adopted as a baseline method, where all tasks are fully offloaded to the BSs.
• Deep Q-learning with non-differentially-private mechanism (DQL-non-DP) [19]: We adopt a model-free method designed for healthcare IoT network [19] and adjust it according to the system state space of this paper. This method can learn the cost-efficient computation offloading policy and serve as the baseline of cost efficiency for the DP-DQL method. The DQL-non-DP method shares the same hyperparameters with the DP-DQL method in the following experiments.

3.3. The Convergence of the DP-DQL Method

In this paper, the proposed DP-DQL method is modified at the action selection step in exploring and the accumulated policy gradient computing step in PER-based policy updating with the aim of protecting computation offloading preference. However, these two modifications may affect the learning performance of the proposed method. To evaluate the potential effect, we vary the $\sigma$ to test the impact of the modifications on the learning performance of the proposed method. According to Theorem 2, once the other hyperparameters and experiment parameters are determined, $\sigma$ will be an important parameter in the DP-DQL method to determine privacy level. In this paper, we set $\sigma \in \{0,0.2,0.4,0.6,0.8\}$. Note that $\sigma =0$ is a special case that privacy-preserving mechanism is not applied. Figure 4 shows the results. The results indicate that, with $\sigma$ increasing, DP-DQL method needs more training episodes to approximate the learning performance than the non-privacy case. It raises a problem that what value of $\sigma$ should we choose to achieve the best learning performance while preserving computation offloading preference. From Figure 4, it can be seen that the DP-DQL method allows UAV to learn the a stability computation offloading policy within 20 TPs, 20 TPs, and 34 TPs under the case of $\sigma =0$, $\sigma =0.2$, and $\sigma =0.4$, respectively. When $\sigma >0.4$, the DP-DQL method will continue to oscillate with no sign of convergence. Hence, we can see that DP-DQL method performs well when $\sigma ⩽0.4$.

3.4. The Privacy Protection of the DP-DQL Method

According to the threat model in Section 2.2.2, the adversary tries to increase the similarity of the distributions of the vanilla value function and recovered value function in the state space $\mathcal{S}$. In this paper, we adopt the t-test to quantitatively evaluate the above similarity. To conduct the t-test, we firstly make a set of hypotheses, including ${\mathcal{K}}_0$ and ${\mathcal{K}}_{\mathcal{W}}$. Note that the subscript 0 is the null hypothesis, and $\mathcal{W}$ is the alternative hypothesis.

• ${\mathcal{K}}_0$: The distributions of the vanilla value function is the same as that of recovered value function in the state space $\mathcal{S}$.
• ${\mathcal{K}}_{\mathcal{W}}$: The distributions of the vanilla value function is not the same as that of recovered value function in the state space $\mathcal{S}$.

Referring to Section 3.3, we set the value function with $\sigma =0$ as the vanilla value function, while the value function with $\sigma =0$, $\sigma =0.2$, $\sigma =0.4$, $\sigma =0.6$, and $\sigma =0.8$ is set as recovered value function. We randomly generate the twenty pairs ($r_t^n$, $C_t$), and input them to the vanilla value function and recovered value function, respectively. Then, we obtain two sets of values. Finally, we calculate the p-value of two sets.

Table 3. The results of t-test.

	$\sigma$ values	0	0.2	0.4	0.6	0.8
	p-values	0.197	1.23 × 10${}^{-15}$	5.73 × 10${}^{-75}$	7.54 × 10${}^{-64}$	4.25 × 10${}^{-32}$

shows the results. It can be seen that the p-value is less than 0.001 in most cases, except the case of $\sigma -0$. Hence, the null hypothesis ${\mathcal{K}}_{\mathcal{W}}$ is accepted with strong evidence. It indicates that the value function of proposed DP-DRL method cannot be recovered by inverse RL.

3.5. The Cost Efficiency of the DP-DQL Method

In this experiment, our aim is to evaluate how much the privacy preserving mechanism in the proposed method affects its cost efficiency, compared to the baseline methods. We adopt the weighted average of the cost $P_t^L,E_t^L,P_t^O,$ and $E_t^O$ as the comparing metric, which is calculated based on Equation (8). Note that, due to the different ranges of the cost $P_t^L,E_t^L,P_t^O,$ and $E_t^O$, they are normalized as Equation (8). According to Section 3.3, we set $\sigma =0.2$ and $\sigma =0.4$ in the experiments. To avoid statistical deviation, we perform each experiment with 10 random seeds.

Figure 5 shows the influence of the radio transmission rate $r_t^n$ on the cost efficiency of DP-DQL and baseline methods. The radio transmission rate $r_t^n$ is set to be 2 Mb/s, 6 Mb/s, and 10 Mb/s, respectively, while the bits of a task is $C_t$ = 40 Mb. Compred with the baseline method, i.e., DQL-non-DP method, we can see that DP-DQL method has little cost efficiency reduction. For instance, we select the DP-DQL method with $\sigma =0.2$ to compare with DQL-non-DP method. When two methods converge, the average cost of DP-DQL method is 15%, 18%, and 20% less than that of DQL-non-DP method in the case of $r_t^n=2$ Mb/s, $r_t^n=6$ Mb/s, and $r_t^n=10$ Mb/s. Moreover, we observe that the proposed DP-DQL method requires more training episodes, which varies from 20 TPs to 35 TPs, to achieve the similar cost than DQL-non-DP method. The extra training episodes used by the proposed DP-DQL method indicate the tradeoff between privacy and cost. Furthermore, we can find that, with the increase of radio transmission rate $r_t^n$, the Greedy method can achieve better the cost efficiency of DP-DQL method in the cases of $r_t^n=6$ Mb/s and 10 Mb/s. The reason is that promising wireless channel status reduces the transmitting cost. However, compared with DP-DQL method, the Greedy method cannot preserve computation offloading preference.

Moreover, the cost efficiency of DP-DQL is evaluated under different task bits $C_t$, compared with baseline methods. The bits of a task $C_t$ is set to be 20 Mb, 40 Mb, and 60 Mb, while the radio transmission rate $r_t^n$ is 10 Mb/s. The results in Figure 6 show that DP-DQL method and DQL-non-DP method outperform the Greedy method in most cases, except the early learning stage of the case of $C_t=20$ Mb. The largest improvement of rewards is in the case of $C_t$ = 60 Mb, which is 260%. We can see that the difference in rewards between the DP-DQL method and DQL-non-DP method varies relatively little with $C_t$. The maximum difference is only 12%, indicating that the proposed method is not overly affected by privacy-preserving mechanisms in terms of cost efficiency and can learn a cost-efficient computation offloading policy. The reason is that offloading all of a task to the BS will not cause too much cost in transmission time when the size of a task is small. With the size of a task increases, transmission time becomes the bottleneck of the cost efficiency of the Greedy method.

3.6. The Performance of the DP-DQL Method Deployed in a Realistic Scenario

Through changing the laptops to the actual RSUs in the scenario, we re-examine the performance of the proposed method and DQL-non-DRL method in terms of cost efficiency. The parameters are set the same as Section 3.5. Figure 7 and Figure 8 show the cost efficiency of the proposed method and baseline methods under different transmission rate $r_t^n$ and bits of a task $C_t$, respectively. It can be seen that the proposed method can still converge with finite TPs. However, by pairwise comparing Figure 5 with Figure 7, and Figure 6 with Figure 8, we can see that the experiments with actual RSUs increase the cost. The reason is that the weaker CPU computing power of the actual RSU increases the time cost and eventually leads to an increase in the total cost. Specifically, based on the assumption that the BS energy consumption is not considered in Section 2.2.1, the energy cost does not change when the actual RSUs are used to replace the laptops, which is still the local energy cost $E_t^L$ plus the transmission energy cost $E_t^O$. However, the time $P_t^O$ for offloading and processing will increase due to the weaker CPU computing power of the actual RSUs. To further verify our point, setting $\sigma =0.4$, Figure 9 shows the proportion of time cost in the total cost. It can be seen that the proportion of time cost to the total cost increases after replacing with actual RSUs. Hence, changing laptops to the actual RSUs will increase the time cost and eventually lead to an increase in the total cost.

4. Discussion

In this section, we firstly discuss the impact of the parameter value on the learning performance of the proposed method. Then, we discuss the limitations of the proposed method.

4.1. Impact of the Key Parameters on the Convergence of DP-DQL Method

As shown in Line 4, Algorithm 1, the reset factor H is the updating frequency of the differential dict $l(·)$, which can affect the convergence of the proposed method. In this experiment, we evaluate the influence of value selection of reset factor H on DP-DQL performance by setting the $\sigma =0.2$. Figure 10 shows the results. In the figure, it can be seen that the average reward is not influenced by the value of reset factor H.

4.2. Limitations and Future Works

The proposed method supports only one UAV to offload the task to single BS at one time slot. This can cover most of the existing daily application scenarios, such as grid inspection, remote sensing, etc. However, the limited endurance of a single UAV limits its ability to perform more complex tasks. Multi-UAV collaboration provides a viable idea, but the proposed method is not able to support privacy-preserving computation offloading in multi-UAV-assisted IoT scenarios. In the future, techniques, such as distributed reinforcement learning and local differential privacy, offer potential solutions to the above needs.

5. Conclusions

In this paper, we propose a differential privacy-based deep Q-learning method for computation offloading over UAV-assisted IoT, which can protect UAV’s computation offloading preference. The formal analysis shows that the proposed DP-DQL method meets the design goals, i.e., differential privacy guarantee and minor utility loss. Furthermore, we evaluate the convergence and privacy of DP-DQL method by the real-world experiment. The results indicate that our DP-DQL method can achieve long-term energy performance under the privacy guarantee, compared with baseline methods. In the future, we will further investigate various privacy issues on DRL-based computation offloading methods.