An Improved Adaptive Service Function Chain Mapping Method Based on Deep Reinforcement Learning
Abstract
:1. Introduction
1.1. Global Technology Status
1.2. Related Works
1.3. Motivation and Contribution
- (1)
- We analyse that the existing SFC mapping methods are difficult to deal with the balance between mapping rate and effective service cost when facing dynamic SFC requests, and propose an adaptive SFC mapping method that jointly optimizes mapping rate and effective service cost.
- (2)
- The SFC mapping problem is decomposed into the SFCR mapping problem and the VNF re-orchestrating problem, meanwhile an improved SFCR mapping algorithm based on DDPG and a VNF reorchestration algorithm based on historical mapping data are proposed, respectively.
- (3)
- The pdh network structure topology in SDNlib is used as the experimental topology, and the change between the mapping rate and the effective service cost of ISM-DRL under different weight trends is verified.
2. Modelling of SFC Mapping
2.1. Mapping Model
2.2. Problem Description
2.2.1. SFCR Mapping
2.2.2. VNF Orchestration
3. SFC Adaptive Mapping Scheme
3.1. ISM-DRL Framework
3.2. SFCR Mapping Algorithm
- (1)
- State space
- (2)
- Action space
- (3)
- Reward space
Algorithm 1: The improved SFCR mapping algorithm based on DDPG |
1: Input: network topology , Network service function request 2: Output: mapping topology 3: Random initialize parameter ,, , and D 4: For episode=1, T do 5: Initialize mapping topology , state and update parameter 6: For t in do 7: Select action according to the current policy 8: Update the mapping topology 9: Obtain and 10: Store transition in D 11: Sample a random mini batch of from D 12: For in do 13: 14: 15: 16: 17: 18: End for 19: End for 20: End for |
3.3. VNF Re-orchestration Algorithm
Algorithm 2: VNF re-orchestration algorithm |
1: Input: network topology 2: Output: Updated 3: Obtain the first H samples from D to form 4: If : Initalize ; Break 5: Initalize ,,, 6: Calculate , , and of various VNFs at different time slots t according to Equations (12)–(15) 7: For 1, M do 8: Average request rate: 9: Average usage rate: 10: Average number of deployments: 11: Average number of inactivations: 12: End for 13: For 1, M do 14: If : 15: Uninstall 20% of this VNF 16: If : 17: Install 20% of this VNF 18: If : 19: Activate 10% of this VNF 20: If : 21: Dormant 10% of this VNF 22: End for 23: Updating network VNF deployments: |
3.4. Time Complexity
4. Experimental Evaluation
4.1. Experimental Environment and Parameter Configuration
4.2. Experimental Comparison
4.2.1. Average Effective Service Cost Rate
4.2.2. Average Mapping Rate
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
SFC | Service Function Chain |
SFCR | Service Function Chain Request |
NFV | Network Function Virtualization |
VNF | Virtual Network Function |
VNFI | Virtual Network Function Instance |
DRL | Deep Reinforcement Learning |
DDPG | Deep Deterministic Policy Gradient |
References
- Sun, G.; Xu, Z.; Yu, H.; Chang, V. Dynamic network function provisioning to enable network in box for industrial applications. IEEE Trans. Ind. Inform. 2020, 17, 7155–7164. [Google Scholar] [CrossRef]
- Mei, C.; Liu, J.; Li, J.; Zhang, L.; Shao, M. 5G network slices embedding with sharable virtual network functions. J. Commun. Netw. 2020, 22, 415–427. [Google Scholar] [CrossRef]
- Fang, L.; Zhang, X.; Sood, K.; Wang, Y.; Yu, S. Reliability-aware virtual network function placement in carrier networks. J. Netw. Comput. Appl. 2020, 154, 102536. [Google Scholar] [CrossRef]
- Qiu, H.; Tang, H.; You, W. Online Service Function Chain Deployment Method Based on Deep Q Network. J. Electron. Inf. Technol. 2021, 43, 3122–3130. [Google Scholar]
- Herrera, J.G.; Botero, J.F. Tabu Search For Service Function Chain Composition In NFV. IEEE Lat. Am. Trans. 2021, 19, 17–25. [Google Scholar] [CrossRef]
- Han, X.; Meng, X.; Yu, Z.; Zhai, D. A Dynamic Adjustment Method of Service Function Chain Resource Configuration. KSII Trans. Internet Inf. Syst. 2021, 15, 2783–2804. [Google Scholar]
- Zhang, D.; Zheng, Z.; Lin, X.; Chen, X.; Wu, C. Dynamic backup sharing scheme of service function chains in NFV. China Commun. 2022, 19, 178–190. [Google Scholar] [CrossRef]
- Wei, S.; Zhou, J.; Chen, S. Delay-Aware Multipath Parallel SFC Orchestration. IEEE Access 2022, 10, 120035–120055. [Google Scholar] [CrossRef]
- Xu, S.; Liao, B.; Hu, B.; Han, C.; Yang, C.; Wang, Z.; Xiong, A. A reliability-and-energy-balanced service function chain mapping and migration method for Internet of Things. IEEE Access 2020, 8, 168196–168209. [Google Scholar] [CrossRef]
- Yaghoubpour, F.; Bakhshi, B.; Seifi, F. End-to-end delay guaranteed Service Function Chain deployment: A multi-level mapping approach. Comput. Commun. 2022, 194, 433–445. [Google Scholar] [CrossRef]
- Li, G.; Feng, B.; Zhou, H.; Zhang, Y.; Sood, K.; Yu, S. Adaptive service function chaining mappings in 5G using deep Q-learning. Comput. Commun. 2020, 152, 305–315. [Google Scholar] [CrossRef]
- Fu, X.; Yu, F.R.; Wang, J.; Qi, Q.; Liao, J. Dynamic service function chain embedding for NFV-enabled IoT: A deep reinforcement learning approach. IEEE Trans. Wirel. Commun. 2019, 19, 507–519. [Google Scholar] [CrossRef]
- Li, W.; Wu, H.; Jiang, C.; Jia, P.; Li, N.; Lin, P. Service Chain Mapping Algorithm Based on Reinforcement Learning. In Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC), Limassol, Cyprus, 15–19 June 2020; pp. 800–805. [Google Scholar]
- Tang, L.; He, L.; Tan, Q.; Chen, Q. Virtual Network Function Migration Optimization Algorithm Based on Deep Deterministic Policy Gradient. J. Electron. Inf. Technol. 2021, 43, 404. [Google Scholar]
- Ouamri, M.A.; Barb, G.; Singh, D.; Alexa, F. Load Balancing Optimization in Software-Defined Wide Area Networking (SD-WAN) using Deep Reinforcement Learning. In Proceedings of the 2022 International Symposium on Electronics and Telecommunications (ISETC), Timisoara, Romania, 10–11 November 2022; pp. 1–6. [Google Scholar] [CrossRef]
- Filali, A.; Mlika, Z.; Cherkaoui, S.; Kobbane, A. Preemptive SDN Load Balancing With Machine Learning for Delay Sensitive Applications. IEEE Trans. Veh. Technol. 2020, 69, 15947–15963. [Google Scholar] [CrossRef]
- Wang, T.; Fan, Q.; Li, X.; Zhang, X.; Xiong, Q.; Fu, S.; Gao, M. DRL-SFCP: Adaptive Service Function Chains Placement with Deep Reinforcement Learning. In Proceedings of the ICC 2021—IEEE International Conference on Communications, Montreal, QC, Canada, 14–23 June 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Tam, P.; Math, S.; Kim, S. Priority-Aware Resource Management for Adaptive Service Function Chaining in Real-Time Intelligent IoT Services. Electronics 2022, 11, 2976. [Google Scholar] [CrossRef]
- Xu, S.; Li, Y.; Guo, S.; Lei, C.; Liu, D.; Qiu, X. Cloud–Edge Collaborative SFC Mapping for Industrial IoT Using Deep Reinforcement Learning. IEEE Trans. Ind. Inform. 2022, 18, 4158–4168. [Google Scholar] [CrossRef]
- Li, J.; Shi, W.; Wu, H.; Zhang, S.; Shen, X. Cost-Aware Dynamic SFC Mapping and Scheduling in SDN/NFV-Enabled Space–Air–Ground-Integrated Networks for Internet of Vehicles. IEEE Internet Things J. 2022, 9, 5824–5838. [Google Scholar] [CrossRef]
- Xiao, Y.; Zhang, Q.; Liu, F.; Wang, J.; Zhao, M.; Zhang, Z.; Zhang, J. NFVdeep: Adaptive Online Service Function Chain Deployment with Deep Reinforcement Learning. In Proceedings of the International Symposium on Quality of Service, IWQoS’19, Phoenix, AZ, USA,, 24–25 June 2019; Association for Computing Machinery: New York, NY, USA, 2019. [Google Scholar]
- Yue, Y.; Cheng, B.; Liu, X.; Wang, M.; Li, B.; Chen, J. Resource optimization and delay guarantee virtual network function placement for mapping SFC requests in cloud networks. IEEE Trans. Netw. Serv. Manag. 2021, 18, 1508–1523. [Google Scholar] [CrossRef]
- Muzahid, A.J.M.; Kamarulzaman, S.F.; Rahman, M.A.; Alenezi, A.H. Deep Reinforcement Learning-Based Driving Strategy for Avoidance of Chain Collisions and Its Safety Efficiency Analysis in Autonomous Vehicles. IEEE Access 2022, 10, 43303–43319. [Google Scholar] [CrossRef]
Optimising Costs | Optimised SFCR Mapping | VNF Reorchestration | Method | |
---|---|---|---|---|
This paper | ✓ | ✓ | ✓ | DDPG |
Literature [12] | × | ✓ | × | DQN |
Literature [14] | ✓ | × | × | DDPG |
Parameter | Description |
---|---|
The underlying physical network topology | |
N | Physical service nodes |
L | Physical link |
L | Physical link |
VNFI set of physical service nodes | |
The current remaining CPU resources and memory resources | |
The current remaining bandwidth resource | |
A set of network service function request | |
The fth SFCR | |
The CPU and memory resource requirements required by VNF | |
The bandwidth resources required by each virtual link | |
The service mapping graph | |
T | Time period and Training steps |
Experimental Parameters | Parameter Value |
---|---|
Physical Node CPU and Memory Resources | Random distribution between [200, 300] |
Physical Link Resources | Random distribution between [200, 300] |
Number of VNFIs deployed per physical node | Uniform distribution between [2, 5] |
Variety of virtual network functions | M = 5 kinds |
SFCR Composition | 2∼5 kinds of network functions |
Time Period | 10 s |
Number of SFCRs arrived in the time period | Poisson distribution with an average of 5 |
Service time per request d | Random distribution between [10, 50] |
CPU and memory resources required for normal operation of VNF | Random distribution between [10, 20] |
Virtual link bandwidth normal operation requirements | Random distribution between [10, 20] |
Experimental Parameters | Parameter Value |
---|---|
Training steps T | 5000 |
Learning rate of actor/critic | 0.002/0.001 |
Target network parameter update rate | 0.001 |
Size of the experience replay pool D | 5000 |
Discount factor | 0.7 |
Greedy | 0.01 |
Raining batch size | 128 |
Reward weight parameter | 0–1 |
ISM-DRL | DDPG | DQN | |
---|---|---|---|
66.23% | 55.72% | 45.38% | |
75.89% | 64.72% | 53.49% | |
84.86% | 70.96% | 60.56% |
ISM-DRL | DDPG | DQN | |
---|---|---|---|
75.76% | 60.67% | 42.94% | |
62.83% | 49.75% | 40.11% | |
57.68% | 46.45% | 31.41% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Huang, W.; Li, S.; Wang, S.; Li, H. An Improved Adaptive Service Function Chain Mapping Method Based on Deep Reinforcement Learning. Electronics 2023, 12, 1307. https://doi.org/10.3390/electronics12061307
Huang W, Li S, Wang S, Li H. An Improved Adaptive Service Function Chain Mapping Method Based on Deep Reinforcement Learning. Electronics. 2023; 12(6):1307. https://doi.org/10.3390/electronics12061307
Chicago/Turabian StyleHuang, Wanwei, Song Li, Sunan Wang, and Hui Li. 2023. "An Improved Adaptive Service Function Chain Mapping Method Based on Deep Reinforcement Learning" Electronics 12, no. 6: 1307. https://doi.org/10.3390/electronics12061307
APA StyleHuang, W., Li, S., Wang, S., & Li, H. (2023). An Improved Adaptive Service Function Chain Mapping Method Based on Deep Reinforcement Learning. Electronics, 12(6), 1307. https://doi.org/10.3390/electronics12061307