UAVAssisted PrivacyPreserving Online Computation Offloading for Internet of Things
Abstract
:1. Introduction
1.1. Related Works and Challenges
1.2. Contributions
 We investigate a new privacy leakage issue within the online computation offloading over UAVassisted IoT, namely computation offloading preference leakage.
 We propose a differential privacybased deep Qlearning (DPDQL) method to protect computation offloading preference over UAVassisted IoT. In the proposed DPDQL method, the DQL is adopted as the basic framework for efficiently learning computation offloading policy without the a priori knowledge of the wireless channel model. Then, a generated Gaussian noise is generated in the policy updating process of DQL, which can protect the computation offloading preference. Finally, the learning speed of DPDQL is accelerated by the PER technique [21] by replying the experience with high temporaldifference error.
 We provide theoretical analysis for the differential privacy guarantee and utility loss. Then, the convergence, privacy protection, and cost efficiency of our method is demonstrated by extensive realworld experiments. The results show that our method can help UAV learn the costefficient computation offloading policy with the differential privacy guarantee.
2. Materials and Methods
2.1. Background Techniques
2.1.1. Differential Privacy
2.1.2. Deep QLearning
2.2. System Model and Problem Formulation
2.2.1. System Model
2.2.2. Threat Model and Privacy Issue
 The BSs can provide customized services for UAV based on the formats of UAV’s computation offloading policy and the inputs of the policy.
 Once the adversary induce the BS, it can monitor the inputs and formats of UAV’s computation offloading policy.
2.2.3. Design Goals
 Differential privacy guarantee: DPDQL method should provide $(\alpha ,\mathcal{\U0001d4ce})$differential privacy for UAV during learning process so that the value function of the UAV’s computation offloading policy will not be inferred by the adversaries based on the system state and offloading decision.
 Minor utility loss: DPDQL method should guarantee that, compared with the traditional DQL method, the performance of the DPDQL method will not be significantly degraded by adding the differential privacy mechanism.
2.2.4. Problem Formulation
 (1)
 System state: The system state is the offloaded proportion. Formally, ${s}_{t}\in \mathcal{S}$ ranges from 0 to 1.
 (2)
 Action space: The UAV adjusts the offloaded proportion of a task by increasing or decreasing from 0 to 0.25. Formally, ${a}_{t}\in [0,0.25]$.
 (3)
 Reward function: The weighted average of energy and time costs is adopted as the reward function, which is given as follows:
2.3. DPBased Deep QLearning for Computation Offloading
2.3.1. Overview
 Initialization: Initializing the parameters used in DPDQL approach.
 Exploring: The UAV executes offloading action and obtains reward from the environment.
 Generating differential disturbance: The UAV generates the specific Gaussian noise to prevent the computation offloading preference leakage.
 PERbased policy updating: The UAV updates computation offloading policy with the help of PER technique.
Algorithm 1 Differential Privacybased Deep QLearning for computation offloading method 

2.3.2. Initialization (Lines 1–4)
2.3.3. Exploring (Lines 5–8)
2.3.4. Generating Differential Disturbance (Lines 9–11)
2.3.5. PERBased Policy Updating (Lines 12–21)
2.4. Theoretical Analysis
2.4.1. Differential Privacy Guarantee
2.4.2. Minor Utility Loss
3. Results
3.1. Experiment Settings
3.2. Baseline Methods
 Greedy: This method has been widely adopted as a baseline method, where all tasks are fully offloaded to the BSs.
 Deep Qlearning with nondifferentiallyprivate mechanism (DQLnonDP) [19]: We adopt a modelfree method designed for healthcare IoT network [19] and adjust it according to the system state space of this paper. This method can learn the costefficient computation offloading policy and serve as the baseline of cost efficiency for the DPDQL method. The DQLnonDP method shares the same hyperparameters with the DPDQL method in the following experiments.
3.3. The Convergence of the DPDQL Method
3.4. The Privacy Protection of the DPDQL Method
 ${\mathcal{K}}_{0}$: The distributions of the vanilla value function is the same as that of recovered value function in the state space $\mathcal{S}$.
 ${\mathcal{K}}_{\mathcal{W}}$: The distributions of the vanilla value function is not the same as that of recovered value function in the state space $\mathcal{S}$.
3.5. The Cost Efficiency of the DPDQL Method
3.6. The Performance of the DPDQL Method Deployed in a Realistic Scenario
4. Discussion
4.1. Impact of the Key Parameters on the Convergence of DPDQL Method
4.2. Limitations and Future Works
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
 FragaLamas, P.; Ramos, L.; MondéjarGuerra, V.M.; FernándezCaramés, T.M. A Review on IoT Deep Learning UAV Systems for Autonomous Obstacle Detection and Collision Avoidance. Remote Sens. 2019, 11, 2144. [Google Scholar] [CrossRef] [Green Version]
 Qiu, T.; Chi, J.; Zhou, X.; Ning, Z.; Atiquzzaman, M.; Wu, D.O. Edge Computing in Industrial Internet of Things: Architecture, Advances and Challenges. IEEE Commun. Surv. Tutor. 2020, 22, 2462–2488. [Google Scholar] [CrossRef]
 Zhou, F.; Hu, R.Q.; Li, Z.; Wang, Y. Mobile Edge Computing in Unmanned Aerial Vehicle Networks. IEEE Wirel. Commun. 2020, 27, 140–146. [Google Scholar] [CrossRef] [Green Version]
 Hong, Z.; Chen, W.; Huang, H.; Guo, S.; Zheng, Z. MultiHop Cooperative Computation Offloading for Industrial IoTEdgeCloud Computing Environments. IEEE Trans. Parallel Distrib. Syst. 2019, 30, 2759–2774. [Google Scholar] [CrossRef]
 Chen, W.; Zhang, Z.; Hong, Z.; Chen, C.; Wu, J.; Maharjan, S.; Zheng, Z.; Zhang, Y. Cooperative and Distributed Computation Offloading for BlockchainEmpowered Industrial Internet of Things. IEEE Internet Things J. 2019, 6, 8433–8446. [Google Scholar] [CrossRef]
 Dai, Y.; Zhang, K.; Maharjan, S.; Zhang, Y. Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks. IEEE Trans. Ind. Inform. 2021, 17, 4968–4977. [Google Scholar] [CrossRef]
 Ren, Y.; Sun, Y.; Peng, M. Deep Reinforcement Learning Based Computation Offloading in Fog Enabled Industrial Internet of Things. IEEE Trans. Ind. Inform. 2021, 17, 4978–4987. [Google Scholar] [CrossRef]
 Yang, L.; Li, M.; Si, P.; Yang, R.; Sun, E.; Zhang, Y. EnergyEfficient Resource Allocation for BlockchainEnabled Industrial Internet of Things with Deep Reinforcement Learning. IEEE Internet Things J. 2021, 8, 2318–2329. [Google Scholar] [CrossRef]
 Pan, X.; Wang, W.; Zhang, X.; Li, B.; Yi, J.; Song, D. How You Act Tells a Lot: PrivacyLeaking Attack on Deep Reinforcement Learning. In Proceedings of the International Foundation for Autonomous Agents and Multiagent Systems, AAMAS 2019, Montreal, QC, Canada, 13–17 May 2019; pp. 368–376. [Google Scholar]
 Zhou, C.; Wu, W.; He, H.; Yang, P.; Lyu, F.; Cheng, N.; Shen, X. Deep Reinforcement Learning for DelayOriented IoT Task Scheduling in SAGIN. IEEE Trans. Wirel. Commun. 2021, 20, 911–925. [Google Scholar] [CrossRef]
 Liu, P.; He, H.; Fu, T.; Lu, H.; Alelaiwi, A.; Wasi, M.W.I. Task offloading optimization of cruising UAV with fixed trajectory. Comput. Netw. 2021, 199, 108397. [Google Scholar] [CrossRef]
 Wei, D.; Ma, J.; Luo, L.; Wang, Y.; He, L.; Li, X. Computation offloading over multiUAV MEC network: A distributed deep reinforcement learning approach. Comput. Netw. 2021, 199, 108439. [Google Scholar] [CrossRef]
 Zhu, S.; Gui, L.; Zhao, D.; Cheng, N.; Zhang, Q.; Lang, X. LearningBased Computation Offloading Approaches in UAVsAssisted Edge Computing. IEEE Trans. Veh. Technol. 2021, 70, 928–944. [Google Scholar] [CrossRef]
 Seid, A.M.; Boateng, G.O.; Anokye, S.; Kwantwi, T.; Sun, G.; Liu, G. Collaborative Computation Offloading and Resource Allocation in MultiUAVAssisted IoT Networks: A Deep Reinforcement Learning Approach. IEEE Internet Things J. 2021, 8, 12203–12218. [Google Scholar] [CrossRef]
 Sacco, A.; Esposito, F.; Marchetto, G.; Montuschi, P. Sustainable Task Offloading in UAV Networks via MultiAgent Reinforcement Learning. IEEE Trans. Veh. Technol. 2021, 70, 5003–5015. [Google Scholar] [CrossRef]
 Gao, A.; Wang, Q.; Chen, K.; Liang, W. MultiUAV Assisted Offloading Optimization: A Game Combined Reinforcement Learning Approach. IEEE Commun. Lett. 2021, 25, 2629–2633. [Google Scholar] [CrossRef]
 He, X.; Liu, J.; Jin, R.; Dai, H. PrivacyAware Offloading in MobileEdge Computing. In Proceedings of the 2017 IEEE Global Communications Conference (GLOBECOM 2017), Singapore, 4–8 December 2017; pp. 1–6. [Google Scholar]
 He, X.; Jin, R.; Dai, H. PhysicalLayer Assisted PrivacyPreserving Offloading in MobileEdge Computing. In Proceedings of the 2019 IEEE International Conference on Communications (ICC 2019), Shanghai, China, 20–24 May 2019; pp. 1–6. [Google Scholar]
 Min, M.; Wan, X.; Xiao, L.; Chen, Y.; Xia, M.; Wu, D.; Dai, H. LearningBased PrivacyAware Offloading for Healthcare IoT with Energy Harvesting. IEEE Internet Things J. 2019, 6, 4307–4316. [Google Scholar] [CrossRef]
 He, X.; Jin, R.; Dai, H. Deep PDSLearning for PrivacyAware Offloading in MECEnabled IoT. IEEE Internet Things J. 2019, 6, 4547–4555. [Google Scholar] [CrossRef]
 Schaul, T.; Quan, J.; Antonoglou, I.; Silver, D. Prioritized Experience Replay. In Proceedings of the 4th International Conference on Learning Representations, (ICLR 2016), San Juan, Puerto Rico, 2–4 May 2016. [Google Scholar]
 Dwork, C.; McSherry, F.; Nissim, K.; Smith, A.D. Calibrating Noise to Sensitivity in Private Data Analysis. Lecture Notes in Computer Science. In Proceedings of the Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, 4–7 March 2006; Springer: Berlin/Heidelberg, Germany, 2006; Volume 3876, pp. 265–284. [Google Scholar]
 Dwork, C.; Roth, A. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci. 2014, 9, 211–407. [Google Scholar] [CrossRef]
 Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.A.; Fidjeland, A.; Ostrovski, G.; et al. Humanlevel control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
 Messous, M.A.; Senouci, S.; Sedjelmaci, H.; Cherkaoui, S. A Game Theory Based Efficient Computation Offloading in an UAV Network. IEEE Trans. Veh. Technol. 2019, 68, 4964–4974. [Google Scholar] [CrossRef]
 Li, K.; Tao, M.; Chen, Z. Exploiting Computation Replication for Mobile Edge Computing: A Fundamental ComputationCommunication Tradeoff Study. IEEE Trans. Wirel. Commun. 2020, 19, 4563–4578. [Google Scholar] [CrossRef] [Green Version]
 Zhou, F.; Hu, R.Q. Computation Efficiency Maximization in WirelessPowered Mobile Edge Computing Networks. IEEE Trans. Wirel. Commun. 2020, 19, 3170–3184. [Google Scholar] [CrossRef] [Green Version]
 Tiwari, N.; Bellur, U.; Sarkar, S.; Indrawan, M. CPU Frequency Tuning to Improve Energy Efficiency of MapReduce Systems. In Proceedings of the 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS), Wuhan, China, 13–16 December 2016; pp. 1015–1022. [Google Scholar]
 He, X.; Jin, R.; Dai, H. Peace: PrivacyPreserving and CostEfficient Task Offloading for MobileEdge Computing. IEEE Trans. Wirel. Commun. 2020, 19, 1814–1824. [Google Scholar] [CrossRef]
 Min, M.; Xiao, L.; Chen, Y.; Cheng, P.; Wu, D.; Zhuang, W. LearningBased Computation Offloading for IoT Devices with Energy Harvesting. IEEE Trans. Veh. Technol. 2019, 68, 1930–1941. [Google Scholar] [CrossRef] [Green Version]
 Chen, X.; Jiao, L.; Li, W.; Fu, X. Efficient MultiUser Computation Offloading for MobileEdge Cloud Computing. IEEE/ACM Trans. Netw. 2016, 24, 2795–2808. [Google Scholar] [CrossRef] [Green Version]
 Wang, B.; Hegde, N. PrivacyPreserving QLearning with Functional Noise in Continuous Spaces. arXiv 2019, arXiv:1901.10634. [Google Scholar]
 Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; Riedmiller, M.A. Deterministic Policy Gradient Algorithms. In Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 21–26 June 2014; Volume 32, pp. 387–395. [Google Scholar]
 MacKay, D.J. Introduction to Gaussian processes. NATO ASI Ser. F Comput. Syst. Sci. 1998, 168, 133–166. [Google Scholar]
 Wei, D.; Xi, N.; Ma, J.; Li, J. Protecting Your Offloading Preference: Privacyaware Online Computation Offloading in Mobile Blockchain. In Proceedings of the 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS), Tokyo, Japan, 25–28 June 2021; pp. 1–10. [Google Scholar]
 Hall, R.; Rinaldo, A.; Wasserman, L.A. Differential privacy for functions and functional data. J. Mach. Learn. Res. 2013, 14, 703–727. [Google Scholar]
 Kairouz, P.; Oh, S.; Viswanath, P. The Composition Theorem for Differential Privacy. IEEE Trans. Inf. Theory 2017, 63, 4037–4049. [Google Scholar] [CrossRef]
 Patel, A.; Shah, N.; Limbasiya, T.; Das, D. VehicleChain: Blockchainbased Vehicular Data Transmission Scheme for Smart City. In Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 6–9 October 2019; pp. 661–667. [Google Scholar]
 Sakuraba, A.; Sato, G.; Uchida, N.; Shibata, Y. Performance Evaluation of Improved V2X Wireless Communication Based on Gigabit WLAN. Lecture Notes in Networks and Systems. In Proceedings of the International Conference on Broadband and Wireless Computing, Communication and Applications, Yonago, Japan, 28–30 October 2020; Springer: Cham, Switzerland, 2020; Volume 159, pp. 131–142. [Google Scholar]
 You, C.; Huang, K.; Chae, H.; Kim, B. EnergyEfficient Resource Allocation for MobileEdge Computation Offloading. IEEE Trans. Wirel. Commun. 2017, 16, 1397–1411. [Google Scholar] [CrossRef]
 Cheng, X.; Lyu, F.; Quan, W.; Zhou, C.; He, H.; Shi, W.; Shen, X. Space/AerialAssisted Computing Offloading for IoT Applications: A LearningBased Approach. IEEE J. Sel. Areas Commun. 2019, 37, 1117–1129. [Google Scholar] [CrossRef]
Notations  Descriptions 

n  The index of BSs 
N  The number of BSs 
$\mathcal{N}$  The set of BSs 
${x}_{n},{y}_{n}$  The xcoordinate and ycoordinate of BS n 
$x,y,h$  The xcoordinate, ycoordinate, and height of UAV 
t  Time index 
${f}^{n},f$  The CPU frequency of nth BS 
f  The CPU frequency of the UAV 
${T}_{t},\mathcal{T}$  The computation task in time slot t and the set of computation task 
H  The reset factor of the DPDQL 
${P}_{t}^{O},{E}_{t}^{O}$  The time and energy consumption at time slot t in BSs 
${C}_{t},{D}_{t}$  The bits and maximum execution time of task ${T}_{t}$ 
${P}_{t}^{L},{E}_{t}^{L}$  The consumed time and energy to locally process task at time slot t 
$EP$  The transmit power of transmitting a bit from BS n to the UAV 
${a}_{t},{s}_{t},{u}_{t}$  The action, state, and reward of the DPDQL in tth time slot 
$TP,V$  The number of training episode and the maximum learning steps within a training episode 
$\tau ,\gamma $  The discount factor and learning rate of the proposed DPDQL 
$A,\mathcal{Z}$  The minibatch size and the replay buffer 
$\xi $  The bits which be processed during a CPU cycle 
${r}_{t}^{n}$  The radio link transmission rate between the UAV and BS n 
$\Psi $  The balance factor of the DPDQL 
Parameter  Value  Parameter  Value 

h  5 m  ${C}_{t}$  {20, 40, 60} Mb 
f  $1\times {10}^{9}$ cycles/s  ${f}^{n}$  $3\times {10}^{9}$ cycles/s 
${D}_{t}$  3 s  $\xi $  1000 
${r}_{t}^{n}$  {2, 6, 10} Mb/s  $\beta $  $1\times {10}^{11}$ 
$EP$  0.2 W  $TP$  100 
V  50  N  3 
$\gamma $  0.001  $\tau $  0.999 
${\theta}_{1}$  0.5  ${\theta}_{2}$  0.5 
$\left\mathcal{Z}\right$  1024  A  128 
${\theta}_{1}$  0.5  ${\theta}_{2}$  0.5 
$\sigma $ values  0  0.2  0.4  0.6  0.8  
pvalues  0.197  1.23 × 10${}^{15}$  5.73 × 10${}^{75}$  7.54 × 10${}^{64}$  4.25 × 10${}^{32}$ 
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. 
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wei, D.; Xi, N.; Ma, J.; He, L. UAVAssisted PrivacyPreserving Online Computation Offloading for Internet of Things. Remote Sens. 2021, 13, 4853. https://doi.org/10.3390/rs13234853
Wei D, Xi N, Ma J, He L. UAVAssisted PrivacyPreserving Online Computation Offloading for Internet of Things. Remote Sensing. 2021; 13(23):4853. https://doi.org/10.3390/rs13234853
Chicago/Turabian StyleWei, Dawei, Ning Xi, Jianfeng Ma, and Lei He. 2021. "UAVAssisted PrivacyPreserving Online Computation Offloading for Internet of Things" Remote Sensing 13, no. 23: 4853. https://doi.org/10.3390/rs13234853