DDPG-Based Computation Offloading Strategy for Maritime UAV
Abstract
1. Introduction
2. Related Works
3. System Model
3.1. Communication Model
3.2. Computational Model
- (1)
- Local computing model
- (2)
- Edge Computing Model
3.3. Problem Description
4. Research on DDPG-Based Strategy for Joint Computation Offloading and Resource Allocation in Maritime UAV Networks
4.1. Markov Decision Process
- State space: The state space in time slot can be expressed as follows:
- (2)
- Reward function: the intelligent body improves the computing offloading policy through the learning process, so the reward function is maximized MIoT system by minimizing the latency of the UAV-assisted, and the opposite of the processing latency of the time slot is the reward function, which can be expressed as follows:
4.2. DDPG Algorithm Framework
4.3. DDPG-Based Computing Offloading and Resource Allocation Algorithm for Maritime UAV
Algorithm 1 Calculation Offloading and Resource Allocation Optimization Algorithm for Offshore UAV Based on DDPG |
|
5. Analysis of Simulation Results
5.1. Simulation Settings
5.2. Performance Analysis
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Li, K.; Wang, X.; Ni, Q.; Huang, M. Entropy-Based Reinforcement Learning for Computation Offloading Service in Software-Defined Multi-Access Edge Computing. Future Gener. Comput. Syst. 2022, 136, 241–251. [Google Scholar] [CrossRef]
- Lin, Z.; Chen, X.; Chen, P. Energy Harvesting Space-Air-Sea Integrated Networks for MEC-Enabled Maritime Internet of Things. China Commun. 2022, 19, 47–57. [Google Scholar] [CrossRef]
- Borkar, V.S.; Choudhary, S.; Gupta, V.K.; Kasbekar, G.S. Scheduling in Wireless Networks with Spatial Reuse of Spectrum as Restless Bandits. Perform. Eval. 2021, 149–150, 102208. [Google Scholar] [CrossRef]
- Niu, Z. TANGO: Traffic-Aware Network Planning and Green Operation. IEEE Wirel. Commun. 2011, 18, 25–29. [Google Scholar] [CrossRef]
- Wang, J.; Ge, Y. A Radio Frequency Energy Harvesting-Based Multihop Clustering Routing Protocol for Cognitive Radio Sensor Networks. IEEE Sens. J. 2022, 22, 7142–7156. [Google Scholar] [CrossRef]
- Xiao, Z.; Chen, Y.; Jiang, H.; Hu, Z.; Lui, J.C.S.; Min, G.; Dustdar, S. Resource Management in UAV-Assisted MEC: State-of-the-Art and Open Challenges. Wirel. Netw. 2022, 28, 3305–3322. [Google Scholar] [CrossRef]
- Yan, C.; Fu, L.; Zhang, J.; Wang, J. A Comprehensive Survey on UAV Communication Channel Modeling. IEEE Access 2019, 7, 107769–107792. [Google Scholar] [CrossRef]
- Akhtar, M.W.; Saeed, N. UAVs-Enabled Maritime Communications: UAVs-Enabled Maritime Communications: Opportunities and Challenges. IEEE Syst. Man Cybern. Mag. 2023, 9, 2–8. [Google Scholar] [CrossRef]
- Jiao, X.; Chen, Y.; Chen, Y.; Wu, X.; Guo, S.; Zhu, W.; Lou, W. SIC-Enabled Intelligent Online Task Concurrent Offloading for Wireless Powered MEC. IEEE Internet Things J. 2024, 11, 22684–22696. [Google Scholar] [CrossRef]
- Nomikos, N.; Gkonis, P.K.; Bithas, P.S.; Trakadas, P. A Survey on UAV-Aided Maritime Communications: Deployment Considerations, Applications, and Future Challenges. IEEE Open J. Commun. Soc. 2023, 4, 56–78. [Google Scholar] [CrossRef]
- Qiu, T.; Zhao, Z.; Zhang, T.; Chen, C.; Chen, C.L.P. Underwater Internet of Things in Smart Ocean: System Architecture and Open Issues. IEEE Trans. Ind. Inform. 2020, 16, 4297–4307. [Google Scholar] [CrossRef]
- Wei, Z.; He, R.; Li, Y.; Song, C. DRL-Based Computation Offloading and Resource Allocation in Green MEC-Enabled Maritime-IoT Networks. Electronics 2023, 12, 4967. [Google Scholar] [CrossRef]
- Cao, S.; Chen, S.; Chen, H.; Zhang, H.; Zhan, Z.; Zhang, W. HCOME: Research on Hybrid Computation Offloading Strategy for MEC Based on DDPG. Electronics 2023, 12, 562. [Google Scholar] [CrossRef]
- Zhang, J.; Guo, H.; Liu, J.; Zhang, Y. Task Offloading in Vehicular Edge Computing Networks: A Load-Balancing Solution. IEEE Trans. Veh. Technol. 2020, 69, 2092–2104. [Google Scholar] [CrossRef]
- Zhang, K.; Leng, S.; He, Y.; Maharjan, S.; Zhang, Y. Mobile Edge Computing and Networking for Green and Low-Latency Internet of Things. IEEE Commun. Mag. 2018, 56, 39–45. [Google Scholar] [CrossRef]
- Wang, J.; Hu, J.; Min, G.; Zomaya, A.Y.; Georgalas, N. Fast Adaptive Task Offloading in Edge Computing Based on Meta Reinforcement Learning. IEEE Trans. Parallel Distrib. Syst. 2021, 32, 242–253. [Google Scholar] [CrossRef]
- Yang, T.; Feng, H.; Gao, S.; Jiang, Z.; Qin, M.; Cheng, N.; Bai, L. Two-Stage Offloading Optimization for Energy–Latency Tradeoff with Mobile Edge Computing in Maritime Internet of Things. IEEE Internet Things J. 2020, 7, 5954–5963. [Google Scholar] [CrossRef]
- Su, X.; Xue, H.; Zhou, Y.; Zhu, J. Research on Computing Offloading Method for Maritime Observation Monitoring Sensor Network. J. Commun. 2021, 42, 149–163. [Google Scholar]
- Jiang, F.; Wang, K.; Dong, L.; Pan, C.; Xu, W.; Yang, K. Deep-Learning-Based Joint Resource Scheduling Algorithms for Hybrid MEC Networks. IEEE Internet Things J. 2020, 7, 6252–6265. [Google Scholar] [CrossRef]
- Wang, R.; Jiang, X.; Zhou, Y.; Li, Z.; Wu, D.; Tang, T.; Fedotov, A.; Badenko, V. Multi-Agent Reinforcement Learning for Edge Information Sharing in Vehicular Networks. Digit. Commun. Netw. 2022, 8, 267–277. [Google Scholar] [CrossRef]
- Lee, Y.H.; Dong, F.; Meng, Y.S. Near Sea-Surface Mobile Radiowave Propagation at 5 GHz: Measurements and Modeling. Radioengineering 2014, 23, 824–830. [Google Scholar] [CrossRef]
- Hasselt, H.V.; Guez, A.; Silver, D. Deep Reinforcement Learning with Double Q-learning. Computer Ence. arXiv 2015, arXiv:1509.06461. [Google Scholar] [CrossRef]
- Wang, Z.; Freitas, N.D.; Lanctot, M. Dueling Network Architectures for Deep Reinforcement Learning. JMLR.org. arXiv 2015, arXiv:1511.06581. [Google Scholar] [CrossRef]
Parameter | Value | Parameter | Value |
---|---|---|---|
Number of MIoT devices | 4 | ||
Length and width of sea surface area | 100 m | 1000 cycles/bit | |
100 m | 0.1 W | ||
9.65 kg | −50 dB | ||
50 m/s | 1 MHz | ||
400 s | −100 dBm | ||
Number of time slots | 40 | 0.6 GHz | |
500 KJ | 1.2 GHz | ||
5 KJ | 0.5 s |
Parameter | Value |
---|---|
Number of neural network layers | 3 |
Neurons in fully connected layers | [400,300,10] |
Actor network learning rate | 0.001 |
Critic network learning rate | 0.002 |
Discount factor | 0.001 |
Exploration rate | 0.01 |
Mini-batch size | 64 |
Total episodes | 1000 |
Time steps per episode | 40 |
Experience buffer size | 10,000 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhao, Z.; Xu, Y.; Yu, Q. DDPG-Based Computation Offloading Strategy for Maritime UAV. Electronics 2025, 14, 3376. https://doi.org/10.3390/electronics14173376
Zhao Z, Xu Y, Yu Q. DDPG-Based Computation Offloading Strategy for Maritime UAV. Electronics. 2025; 14(17):3376. https://doi.org/10.3390/electronics14173376
Chicago/Turabian StyleZhao, Ziyue, Yanli Xu, and Qianlian Yu. 2025. "DDPG-Based Computation Offloading Strategy for Maritime UAV" Electronics 14, no. 17: 3376. https://doi.org/10.3390/electronics14173376
APA StyleZhao, Z., Xu, Y., & Yu, Q. (2025). DDPG-Based Computation Offloading Strategy for Maritime UAV. Electronics, 14(17), 3376. https://doi.org/10.3390/electronics14173376