Adaptive Real-Time Offloading Decision-Making for Mobile Edges: Deep Reinforcement Learning Framework and Simulation Results
Abstract
:1. Introduction
2. Background and Related Work
2.1. Deep Learning and Deep Reinforcement Learning Foundations
2.2. Related Work: Reinforcement Learning for Mobile Edge Computing
3. System Model
3.1. Reference System Model and Formulation
- State {TD capacity, StandBy Q capacity, ES capacity}
- Action {0, 1}
3.2. Offloading Decision: Trade-Off between Energy Consumption and Delay
4. Proposed DQN-Based Offloading Decision Algorithm
4.1. Deep Reinforcement Learning
4.2. DQN-Based Offloading and Compression Decision
Algorithm 1 DQN-based offloading decision | |
1: | Initialize replay memory D to capacity N |
2: | Initialize action-value function Q with random weights |
3: | Initialize target action-value function with weights = |
4: | for episode = 1, M do |
5: | Initialize sequence = and preprocessed sequence = |
6: | fort = 1, T do |
7: | With probability selects a random action |
8: | Otherwise select = |
9: | Execute action in emulator and observe reward , and image |
10: | Set = * + * + with weight parameter |
11: | Set = , , and process = |
12: | Store transition in D |
13: | Sample random minibatch of transitions from D |
14: | if Episode terminates at step then |
15: | |
16: | else |
17: | |
18: | end if |
19: | Compute a gradient descent step on with respect to the network parameters |
20: | Every C steps reset =Q |
21: | end for |
22: | end for |
Algorithm 2 Threshold-based compression decision |
if task is decided to offload then |
2: if then |
compression is not necessary, store the data in the Q |
4: else |
compress and offload the compressed task to edge server |
6: end if |
else if task is decided to process locally then |
8: if then |
perform the task at terminal device without any negative reward |
10: else |
perform the task at terminal device |
12: a negative reward () is added |
end if |
14: end if |
5. Performance Evaluation
5.1. Simulation Setting
5.2. Simulation Results and Discussions
6. Concluding Remarks and Future Work
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Kim, J.; Molisch, A.F. Fast Millimeter-Wave Beam Training with Receive Beamforming. J. Commun. Netw. 2014, 16, 512–522. [Google Scholar] [CrossRef]
- Kwon, D.; Kim, S.-W.; Kim, J.; Mohaisen, A. Interference-Aware Adaptive Beam Alignment for Hyper-Dense IEEE 802.11ax Internet-of-Things Networks. Sensors 2018, 18, 3364. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kim, J.; Caire, G.; Molisch, A.F. Quality-Aware Streaming and Scheduling for Device-to-Device Video Delivery. IEEE/ACM Trans. Netw. 2016, 24, 2319–2331. [Google Scholar] [CrossRef]
- Dao, N.-N.; Vu, D.-N.; Na, W.; Kim, J.; Cho, S. SGCO: Stabilized Green Crosshaul Orchestration for Dense IoT Offloading Services. IEEE J. Sel. Areas Commun. 2018, 36, 2538–2548. [Google Scholar] [CrossRef]
- Choi, M.; No, A.; Ji, M.; Kim, J. Markov Decision Policies for Dynamic Video Delivery in Wireless Caching Networks. IEEE Trans. Wirel. Commun. 2019, 18, 5705–5718. [Google Scholar] [CrossRef] [Green Version]
- Kwon, D.; Kim, J. Multi-Agent Deep Reinforcement Learning for Cooperative Connected Vehicles. In Proceedings of the IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA, 9–13 December 2019. [Google Scholar]
- Kwon, D.; Kim, J. Optimal Trajectory Learning for UAV-BS Video Provisioning System: A Deep Reinforcement Learning Approach. In Proceedings of the IEEE International Conference on Information Networking (ICOIN), Kuala Lumpur, Malaysia, 9–11 January 2019. [Google Scholar]
- Wu, H. Multi-Objective Decision-Making for Mobile Cloud Offloading: A Survey. IEEE Access 2018, 6, 3962–3976. [Google Scholar] [CrossRef]
- Wu, H.; Sun, Y.; Wolter, K. Energy-Efficient Decision Making for Mobile Cloud Offloading. IEEE Trans. Cloud Comput. 2020, 2020, 1–15. [Google Scholar] [CrossRef]
- Kim, B.; Min, H.; Heo, J.; Jung, J. Dynamic Computation Offloading Scheme for Drone-based Surveillance Systems. Sensors 2018, 18, 2982. [Google Scholar] [CrossRef] [Green Version]
- Huang, L.; Feng, X.; Zhang, C.; Qian, L.; Wu, Y. Deep Reinforcement Learning-based Joint Task Offloading and Bandwidth Allocation for Multi-User Mobile Edge Computing. Digit. Commun. Netw. 2019, 5, 10–17. [Google Scholar] [CrossRef]
- Akherfi, K.; Gerndt, M.; Harroud, H. Mobile Cloud Computing for Computation Offloading: Issues and Challenges. Appl. Comput. Inform. 2018, 14, 1–16. [Google Scholar] [CrossRef]
- Park, S.; Kim, J.; Kwon, D.; Shin, M.; Kim, J. Joint Offloading and Streaming in Mobile Edges: A Deep Reinforcement Learning Approach. In Proceedings of the IEEE VTS Asia Pacific Wireless Communications Symposium (APWCS), Singapore, 28–30 August 2019. [Google Scholar]
- Shin, M.; Choi, D.-H.; Kim, J. Cooperative Management for PV/ESS-Enabled Electric-Vehicle Charging Stations: A Multi-Agent Deep Reinforcement Learning Approach. IEEE Trans. Ind. Inform. 2020, 16, 3493–3503. [Google Scholar] [CrossRef]
- Feng, L.; Obayshi, M.; Kuremoto, T.; Kobayashi, K. An Intelligent Control System Construction using High-Level Time Petri Net and Reinforcement Learning. In Proceedings of the IEEE International Conference on Control, Automation and Systems (ICCAS), Gyeonggi-do, Korea, 27–30 October 2010. [Google Scholar]
- Bhagat, S.; Banerjee, H.; Tse, Z.T.H.; Ren, H. Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges. Robotics 2019, 8, 4. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Y.; Sun, P.; Yin, Y.; Lin, L.; Wang, X. Human-Like Autonomous Vehicle Speed Control by Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 26–30 June 2018. [Google Scholar]
- Xu, X.; Zuo, L.; Li, X.; Qian, L.; Ren, J.; Sunm, Z. A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways. IEEE Trans. Syst. Man Cybern. Syst. 2018. [Google Scholar] [CrossRef]
- Zheng, R.; Liu, C.; Guo, Q. A Decision-Making Method for Autonomous Vehicles based on Simulation and Reinforcement Learning. In Proceedings of the International Conference on Machine Learning and Cybernetics (ICMLC), Tianjin, China, 14–17 July 2013. [Google Scholar]
- Sqryn, M.; Sharma, A.; Parkar, D.; Shrimal, M. Distributed Deep Reinforcement Learning on the Cloud for Autonomous Driving. In Proceedings of the IEEE/ACM International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS), Gothenburg, Sweden, 28 May 2018. [Google Scholar]
- Shin, M.; Kim, J. Randomized Adversarial Imitation Learning for Autonomous Driving. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Macau, China, 10–16 August 2019. [Google Scholar]
- Shin, M.; Kim, J. Adversarial Imitation Learning via Random Search. In Proceedings of the International Joint Conference on Meural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019; pp. 1–8. [Google Scholar]
- Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; et al. Mastering the Game of Go without Human Knowledge. Nature 2017, 550, 354–359. [Google Scholar] [CrossRef] [PubMed]
- Shi, W.; Cao, J.; Zhang, Q.; Li, Y.; Xu, L. Edge Computing: Vision and Challenges. IEEE Internet Things J. 2016, 3, 637–646. [Google Scholar] [CrossRef]
- Chen, X.; Jiao, L.; Li, X.; Fu, X. Efficient Multi-User Computation Offloading for Mobile-Edge Cloud Computing. IEEE/ACM Trans. Netw. 2016, 24, 2795–2808. [Google Scholar] [CrossRef] [Green Version]
- Munoz, O.; Iserte, A.P.; Vidal, J.; Molina, M. Energy-Latency Trande-off for Multiuser Wireless Computation Offloading. In Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC) Workshop on Cloud Technologies and Energy Efficiency in Mobile Communication Networks, Istanbul, Turkey, 6–9 April 2014. [Google Scholar]
- Wang, J.; Zhao, L.; Liu, J.; Kato, N. Smart Resource Allocation for Mobile Edge Computing: A Deep Reinforcement Learning Approach. IEEE Trans. Emerg. Top. Comput. 2020. [Google Scholar] [CrossRef]
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 1998. [Google Scholar]
- Sutton, R.S.; McAllester, D.A.; Singh, S.P.; Mansour, Y. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), Denver, CO, USA, 1 January 2000; pp. 1057–1063. [Google Scholar]
- Zhu, L.; He, Y.; Yu, F.R.; Ning, B.; Tang, T.; Zhao, N. Communication-based Train Control System Performance Optimization using Deep Reinforcement Learning. IEEE Trans. Veh. Technol. 2017, 66, 10705–10717. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Petersen, S. Human-Level Control through Deep Reinforcement Learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef]
- Nair, A.; Srinivasan, P.; Blackwell, S.; Alcicek, C.; Fearon, R.; Maria, A.; Panneershelvam, V.; Suleyman, M.; Beattie, C.; Petersen, S.; et al. Massively Parallel Methods for Deep Reinforcement Learning. arXiv 2015, arXiv:preprint/1507.04296. [Google Scholar]
- Kober, J.; Bagnell, J.A.; Peters, J. Reinforcement Learning in Robotics: A Survey. Int. J. Robot. Res. 2013, 32, 1238–1274. [Google Scholar] [CrossRef] [Green Version]
- Arulkumaran, K.; Deisenroth, M.P.; Brundage, M.; Bharath, A.A. Deep Reinforcement Learning: A Brief Survey. IEEE Signal Process. Mag. 2017, 34, 26–38. [Google Scholar] [CrossRef] [Green Version]
- Bozorgchenani, A.; Tarchi, D.; Corazza, G.E. An Energy and Delay-Efficient Partial Offloading Technique for Fog Computing Architectures. In Proceedings of the IEEE Global Communications Conference (GLOBECOM), Singapore, 4–8 December 2017; pp. 1–6. [Google Scholar]
- Feng, W.; Yang, C.; Zhou, X. Multi-User and Multi-Task Offloading Decision Algorithms Based on Imbalanced Edge Cloud. IEEE Access 2019, 7, 95970–95977. [Google Scholar] [CrossRef]
Parameter | Value |
---|---|
Processing delay at terminal | 1.0 |
Processing delay at edge server | 0.08 |
Transmission delay (uploading) | 0.25 |
Transmission delay (downloading) | 0.05 |
Processing energy at terminal | 1.0 |
Processing energy at edge server | 0.05 |
Transmission energy (uploading) | 0.15 |
Transmission energy (downloading) | 0.05 |
DQN Based | Local Computing | Full Offloading | |
---|---|---|---|
Energy consumption | 52.448 | 315.830 | 71.845 |
Execution delay | 153.320 | 563.830 | 122.374 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Park, S.; Kwon, D.; Kim, J.; Lee, Y.K.; Cho, S. Adaptive Real-Time Offloading Decision-Making for Mobile Edges: Deep Reinforcement Learning Framework and Simulation Results. Appl. Sci. 2020, 10, 1663. https://doi.org/10.3390/app10051663
Park S, Kwon D, Kim J, Lee YK, Cho S. Adaptive Real-Time Offloading Decision-Making for Mobile Edges: Deep Reinforcement Learning Framework and Simulation Results. Applied Sciences. 2020; 10(5):1663. https://doi.org/10.3390/app10051663
Chicago/Turabian StylePark, Soohyun, Dohyun Kwon, Joongheon Kim, Youn Kyu Lee, and Sungrae Cho. 2020. "Adaptive Real-Time Offloading Decision-Making for Mobile Edges: Deep Reinforcement Learning Framework and Simulation Results" Applied Sciences 10, no. 5: 1663. https://doi.org/10.3390/app10051663
APA StylePark, S., Kwon, D., Kim, J., Lee, Y. K., & Cho, S. (2020). Adaptive Real-Time Offloading Decision-Making for Mobile Edges: Deep Reinforcement Learning Framework and Simulation Results. Applied Sciences, 10(5), 1663. https://doi.org/10.3390/app10051663