DRL-Based Dependent Task Offloading Strategies with Multi-Server Collaboration in Multi-Access Edge Computing
Abstract
:1. Introduction
- We design an MDP to accurately model the dependent task offloading problem with well-designed state space, action space, and reward function. Aiming at the above problems, we propose an intelligent computing offloading algorithm based on discrete SAC, which can adapt to the dynamic network environment and has the advantages of high stability and high sample utilization. The SACDTO strategy with the maximum entropy objective has a higher exploration ability to learn a continuous action space.
- We extend the application scenario to multiple edge servers and study the computational offloading problem of various mobile users offloading tasks to multiple MEC servers through the base station. As multiple servers are deployed around mobile users, task offloading is no longer a simple binary offloading decision, and there are not only two processing methods: local execution or edge offloading. Instead, specific consideration should be given to whether task offloading should be performed and the number of the MEC server responsible for a task’s execution after offloading.
- We use the collaboration between the cloud, edge, and terminal to realize the training of a neural network with the massive computing power of cloud, and the trained scheduler can offload tasks to MEC servers for calculation.
- We transform the original DAG into a series of embeddings that contain task profiles and dependency information for the application, which are transformed into inputs to the neural network. In addition, we conduct plenty of simulation experiments using synthetic DAG, analyze the convergence curves of each strategy, and discuss the time delay and energy consumption when the number of tasks and the communication to calculation ratio are different, which correspond to the characteristics of real applications. The results show that the proposed method outperforms other comparison algorithms in dynamic MEC scenarios.
2. Related Work
3. System Model and Problem Formulation
3.1. MEC Architecture
3.2. System Model
3.3. Problem Formulation
4. SACDTO Strategy
4.1. SAC Algorithm for Discrete Actions
4.2. The Task Offloading Model
4.2.1. State Space
4.2.2. Action Space
4.2.3. Reward Function
4.3. SACDTO Implementation
Algorithm 1: SACDTO | |
Input: Episode, environment, batch size, replay buffer size | |
1 | Initialize the networks: Qθ1 →R|A|,Qθ2 →R|A|,πϕ→[0, 1]|A| |
Initialize the target network: | |
Initialize the target network weights: Initialize the replay buffer: | |
2 | foreach episodedo |
3 | Obtain state St from environment; |
4 | while not done: |
5 | Allocate computing resources to each MEC server; |
6 | Determine the current task to work on based on the priority list; |
7 | at~πϕ (at|st); |
8 | Update the remaining computing resources for each MEC server according to at; |
9 | st+1 ~ p (st+1|st,at); |
10 | ; |
11 | if current episode % learning interval step==0 then |
12 | Sample a random minibatch of samples from D to calculate the target values; |
13 | Update the Q-function parameters ; |
14 | Update policy weights ; |
15 | Update temperature ; |
16 | Update target network weights ; |
Output: θ1, θ2, ϕ |
5. Performance Evaluation
5.1. Baseline Approaches
- Proximal Policy Optimization-based Task Offloading (PPOTO): PPO is an improved algorithm of Policy Gradient, but still has the disadvantage of low sampling efficiency;
- Dueling Double Deep-Q Network-based Task Offloading (D3QNTO): Combines the advantages of Dueling DQN and Double DQN, and improves the training algorithm and model structure of DQN;
- Random Task Offloading (RTO): Computation tasks are offloaded randomly;
- All Local (ALOC): All computing tasks are executed locally.
5.2. Simulation and Results
6. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Yan, M.; Li, W.; Chan, C.A.; Bian, S.; Chih-Lin, I.; Gygax, A.F. PECS: Towards personalized edge caching for future service-centric networks. China Commun. 2019, 16, 93–106. [Google Scholar] [CrossRef]
- Wang, B.; Song, Y.; Cao, J.; Cui, X.; Zhang, L. Improving Task Scheduling with Parallelism Awareness in Heterogeneous Computational Environments. Future Gener. Comput. Systems. 2019, 94, 419–429. [Google Scholar] [CrossRef]
- Liu, S.; Cheng, P.; Chen, Z.; Xiang, W.; Vucetic, B.; Li, Y. Contextual User-Centric Task Offloading for Mobile Edge Computing in Ultra-Dense Network. IEEE Trans. Mob. Computing 2022. [Google Scholar] [CrossRef]
- Fan, X.; Cui, T.; Cao, C.; Chen, Q.; Kwak, K.S. Minimum-Cost Offloading for Collaborative Task Execution of MEC-Assisted Platooning. Sensors 2019, 19, 847. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, W.; Wen, Y. Energy-Efficient Task Execution for Application as a General Topology in Mobile Cloud Computing. IEEE Trans. Cloud Comput. 2015, 6, 708–719. [Google Scholar] [CrossRef]
- Mao, N.; Chen, Y.; Guizani, M.; Lee, G.M. Graph Mapping Offloading Model Based On Deep Reinforcement Learning with Dependent Task. In Proceedings of the 2021 International Wireless Communications and Mobile Computing (IWCMC), Harbin, China, 28 June–2 July 2021; pp. 21–28. [Google Scholar]
- Chen, J.; Yang, Y.; Wang, C.; Zhang, H.; Qiu, C.; Wang, X. Multitask Offloading Strategy Optimization Based on Directed Acyclic Graphs for Edge Computing. IEEE Internet Things J. 2022, 9, 9367–9378. [Google Scholar] [CrossRef]
- Leng, L.; Li, J.; Shi, H.; Zhu, Y.A. Graph convolutional network-based reinforcement learning for tasks offloading in multi-access edge computing. Multimed Tools 2021, 80, 29163–29175. [Google Scholar] [CrossRef]
- Guan, X.; Lv, T.; Lin, Z.; Huang, P.; Zeng, J. D2D-Assisted Multi-User Cooperative Partial Offloading in MEC Based on Deep Reinforcement Learning. Sensors 2022, 22, 7004. [Google Scholar] [CrossRef]
- Huynh, L.N.T.; Pham, Q.-V.; Pham, X.-Q.; Nguyen, T.D.T.; Hossain, M.D.; Huh, E.-N. Efficient Computation Offloading in Multi-Tier Multi-Access Edge Computing Systems: A Particle Swarm Optimization Approach. Appl. Sci. 2020, 10, 203. [Google Scholar] [CrossRef] [Green Version]
- Ke, H.; Wang, H.; Sun, H. Multi-Agent Deep Reinforcement Learning-Based Partial Task Offloading and Resource Allocation in Edge Computing Environment. Electronics 2022, 11, 2394. [Google Scholar] [CrossRef]
- Tang, M.; Wong, V.W.S. Deep Reinforcement Learning for Task Offloading in Mobile Edge Computing Systems. IEEE Trans. Mob. Comput. 2022, 21, 1985–1997. [Google Scholar] [CrossRef]
- Hu, H.; Wu, D.; Zhou, F.; Jin, S.; Hu, R.Q. Dynamic Task Offloading in MEC-Enabled IoT Networks: A Hybrid DDPG-D3QN Approach. In Proceedings of the 2021 IEEE Global Communications Conference (GLOBECOM), Madrid, Spain, 7–11 December 2021; pp. 1–6. [Google Scholar]
- Liu, K.-H.; Hsu, Y.-H.; Lin, W.-N.; Liao, W. Fine-Grained Offloading for Multi-Access Edge Computing with Actor-Critic Federated Learning. In Proceedings of the 2021 IEEE Wireless Communications and Networking Conference (WCNC), Nanjing, China, 3 March–1 April 2021; pp. 1–6. [Google Scholar]
- Liu, J.; Lin, F.; Liu, K.; Zhao, Y.; Li, J. Research on Multi-Terminal’s AC Offloading Scheme and Multi-Server’s AC Selection Scheme in IoT. Entropy 2022, 24, 1357. [Google Scholar] [CrossRef]
- Li, S.; Hu, S.; Du, Y. Deep Reinforcement Learning and Game Theory for Computation Offloading in Dynamic Edge Computing Markets. IEEE Access 2021, 9, 121456–121466. [Google Scholar] [CrossRef]
- Wang, J.; Hu, J.; Min, G.; Zhan, W.; Ni, Q.; Georgalas, N. Computation Offloading in Multi-Access Edge Computing Using a Deep Sequential Model Based on Reinforcement Learning. IEEE Commun. Mag. 2019, 57, 64–69. [Google Scholar] [CrossRef] [Green Version]
- Chen, X.; Liu, G. Federated Deep Reinforcement Learning-Based Task Offloading and Resource Allocation for Smart Cities in a Mobile Edge Network. Sensors 2022, 22, 4738. [Google Scholar] [CrossRef]
- Dang, X.; Su, L.; Hao, Z.; Shang, X. Dynamic Offloading Method for Mobile Edge Computing of Internet of Vehicles Based on Multi-Vehicle Users and Multi-MEC Servers. Electronics 2022, 11, 2326. [Google Scholar] [CrossRef]
- Sun, C.; Wu, X.; Li, X.; Fan, Q.; Wen, J.; Leung, V.C.M. Cooperative Computation Offloading for Multi-Access Edge Computing in 6G Mobile Networks via Soft Actor Critic. IEEE Trans. Netw. Sci. Eng. 2021. [Google Scholar] [CrossRef]
- Liu, P.; Ge, S.; Zhou, X.; Zhang, C.; Li, K. Soft Actor-Critic-Based DAG Tasks Offloading in Multi-Access Edge Computing with Inter-User Cooperation. In Algorithms and Architectures for Parallel Processing; Lai, Y., Wang, T., Jiang, M., Xu, G., Liang, W., Castiglione, A., Eds.; ICA3PP 2021; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2022; Volume 13157. [Google Scholar] [CrossRef]
- Wang, B.; Wang, C.H.; Huang, W.W.; Song, Y.; Qin, X.Y. Security-aware task scheduling with deadline constraints on heterogeneous hybrid clouds. J. Parallel Distrib. Comput. 2021, 153, 15–28. [Google Scholar] [CrossRef]
- Wang, B.; Cheng, J.; Cao, J.; Wang, C.; Huang, W. Integer particle swarm optimization based task scheduling for device-edge-cloud cooperative computing to improve SLA satisfaction. Peer J. Comput. Sci. 2022, 8, e893. [Google Scholar] [CrossRef]
- He, W.; Gao, L.; Luo, J. A Multi-Layer Offloading Framework for Dependency-Aware Tasks in MEC. In Proceedings of the ICC 2021-IEEE International Conference on Communications, Montreal, QC, Canada, 14–18 June 2021; pp. 1–6. [Google Scholar]
- Chen, L.; Wu, J.; Zhang, J.; Dai, H.-N.; Long, X.; Yao, M. Dependency-Aware Computation Offloading for Mobile Edge Computing with Edge-Cloud Cooperation. IEEE Trans. Cloud Comput. 2022, 10, 2451–2468. [Google Scholar] [CrossRef]
- Long, J.; Luo, Y.; Zhu, X.; Luo, E.; Huang, M. Computation offloading through mobile vehicles in IoT-edge-cloud network. J. Wirel. Com. Netw. 2020, 244, 1–21. [Google Scholar] [CrossRef]
- Dai, F.; Liu, G.; Mo, Q.; Xu, W.; Huang, B. Task offloading for vehicular edge computing with edge-cloud cooperation. World Wide Web 2022, 25, 1999–2017. [Google Scholar] [CrossRef]
- Developing software for multi-access edge computing. ETSI White Pap. 2019, 20, 1–38.
- Yan, M.; Li, S.; Chan, C.A.; Shen, Y.; Yu, Y. Mobility Prediction Using a Weighted Markov Model Based on Mobile User Classification. Sensors 2021, 21, 1740. [Google Scholar] [CrossRef] [PubMed]
- Zou, J.; Hao, T.; Yu, C.; Jin, H. A3c-do: A regional resource scheduling framework based on deep reinforcement learning in edge scenario. IEEE Trans. Comput. 2021, 70, 228–239. [Google Scholar] [CrossRef]
- Arabnejad, H.; Barbosa, J.G. List scheduling algorithm for heterogeneous systems by an optimistic cost table. IEEE Trans. Parallel Distrib. Syst. 2013, 25, 682–694. [Google Scholar] [CrossRef]
Approaches | Advantages | Disadvantages |
---|---|---|
Convex relaxation approaches [9] or heuristic local search approaches [4,10] | Closer to the optimal offloading solution. | It is easy to fall into the local optima, and it is necessary to re-solve the optimization problem when the external environment changes. |
DQN-based approaches [9,10,11,12,13] | Suitable for dynamic environments. | When the number of wireless devices grows exponentially, these approaches are expensive. |
PPO-based approaches [16,17] | Good effect, and it can realize discrete control and continuous control. | The sample efficiency is low and requires a large number of samples, which is not suitable for actual application scenarios. |
DDPG-based approaches [18,19] | High efficiency. | The explorability, stability, and robustness of this method are not good enough. |
Notations | Definition |
---|---|
π(·|·) | Offloading policy |
The number of clock cycles required to process each bit of data | |
D | Replay buffer |
Psend, Prec | The transmitted and received power |
ω | The parameter that determines the send or receive rate |
O1:i | Set of offloading decisions from task v1 to vi |
ξ | Directed acyclic graph |
β1, β2 | The weight coefficient of energy consumption and delay ratio |
θ1, θ2, ϕ | Parameters of SACDTO |
α | The temperature parameter |
ai | The offloading action of task vi |
M | MEC server collection |
Tiloc, Tis | The local and MEC server computation delay of the task vi |
fms | The CPU clock speed of the MEC server numbered m |
filoc | The CPU clock speed of the mobile device where task vi is located |
Ri(ω) | Sending rate or receiving rate |
CmMEC | Total computing resources of the MEC server numbered m |
di | The data size of task vi |
Ci | The total number of clock cycles required by task vi |
Finish_Ti | The finish time of task vi |
Finish_Tiloc | Local finish time of task vi |
Finish_Titrans(ω) | The finish time of upload and download task vi |
Finish_Tpre | The finish time of the previous task |
Finish_Ti,mser | The finish time of task vi on the MEC server numbered m |
Avail_Tiloc | The CPU idle time of the earliest available task on the local processor |
Avail_Titrans(ω) | The earliest available time of the transmission link |
Avail_Ti,mser | The earliest available CPU idle time of the MEC server |
Parameters | Value |
---|---|
Replay memory size | 5000 |
Optimizer | Adam |
Learning rate for Actor and Critic networks | 0.001 |
Minibatch size | 128 |
Discounted factor for reward | 0.99 |
Delayed update factor | 0.995 |
Initial value of temperature parameter | 0.2 |
The learning rate of temperature parameter | 0.001 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Peng, B.; Li, T.; Chen, Y. DRL-Based Dependent Task Offloading Strategies with Multi-Server Collaboration in Multi-Access Edge Computing. Appl. Sci. 2023, 13, 191. https://doi.org/10.3390/app13010191
Peng B, Li T, Chen Y. DRL-Based Dependent Task Offloading Strategies with Multi-Server Collaboration in Multi-Access Edge Computing. Applied Sciences. 2023; 13(1):191. https://doi.org/10.3390/app13010191
Chicago/Turabian StylePeng, Biying, Taoshen Li, and Yan Chen. 2023. "DRL-Based Dependent Task Offloading Strategies with Multi-Server Collaboration in Multi-Access Edge Computing" Applied Sciences 13, no. 1: 191. https://doi.org/10.3390/app13010191
APA StylePeng, B., Li, T., & Chen, Y. (2023). DRL-Based Dependent Task Offloading Strategies with Multi-Server Collaboration in Multi-Access Edge Computing. Applied Sciences, 13(1), 191. https://doi.org/10.3390/app13010191