Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic
Abstract
:1. Introduction
- The optimal adaptive control problem of VSG is transformed into an RL task, thereby obviating the necessity for intricate mathematical models and expert knowledge. Subsequently, the state-of-the-art SAC algorithm is employed to train the agent, enabling it to discover the optimal strategy.
- The traditional optimization objective of VSG focuses solely on mitigating active power and frequency fluctuations, overlooking the optimization of the system’s transient response time. In the reward function design, this paper introduces an adjustment time component, motivating the agent to refine the strategy further for enhanced system performance.
2. System Model and Problem Formulation
2.1. VSG Control Principle
2.2. Objective Function
2.3. Constraint Conditions
3. Transformation and Solution
3.1. Markov Decision Model
3.1.1. State Space
3.1.2. Action Space
3.1.3. Reward Function
3.2. Principle of Soft Actor Critic
4. Simulation Results
4.1. Comparison of Different Reward Functions
4.2. Comparison of Different RL Algorithms
4.3. Case Studies
4.3.1. Active Power Reference Disturbance
4.3.2. Load Disturbance
4.3.3. Grid Frequency Disturbance
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Kryonidis, G.C.; Kontis, E.O.; Chrysochos, A.I.; Oureilidis, K.O.; Demoulias, C.S.; Papagiannis, G.K. Power flow of islanded AC microgrids: Revisited. IEEE Trans. Smart Grid 2018, 9, 3903–3905. [Google Scholar] [CrossRef]
- Fang, J.; Li, H.; Tang, Y.; Blaabjerg, F. Distributed power system virtual inertia implemented by grid-connected power converters. IEEE Trans. Power Electron. 2017, 33, 8488–8499. [Google Scholar] [CrossRef]
- She, B.; Li, F.; Cui, H.; Wang, J.; Min, L.; Oboreh-Snapps, O.; Bo, R. Decentralized and Coordinated Vf Control for Islanded Microgrids Considering DER Inadequacy and Demand Control. IEEE Trans. Energy Convers. 2023, 38, 1868–1880. [Google Scholar] [CrossRef]
- Rehman, W.U.; Moeini, A.; Oboreh-Snapps, O.; Bo, R.; Kimball, J. Deadband voltage control and power buffering for extreme fast charging station. In Proceedings of the 2021 IEEE Madrid PowerTech, Madrid, Spain, 28 June–2 July 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 1–6. [Google Scholar]
- Chen, S.; Sun, Y.; Hou, X.; Han, H.; Fu, S.; Su, M. Quantitative Parameters Design of VSG Oriented to Transient Synchronization Stability. IEEE Trans. Power Syst. 2023, 38, 4978–4981. [Google Scholar] [CrossRef]
- Chen, M.; Zhou, D.; Blaabjerg, F. High penetration of inverter-based power sources with VSG control impact on electromechanical oscillation of power system. Int. J. Electr. Power Energy Syst. 2022, 142, 108370. [Google Scholar] [CrossRef]
- Li, J.; Wen, B.; Wang, H. Adaptive virtual inertia control strategy of VSG for micro-grid based on improved bang-bang control strategy. IEEE Access 2019, 7, 39509–39514. [Google Scholar] [CrossRef]
- Hou, X.; Sun, Y.; Zhang, X.; Lu, J.; Wang, P.; Guerrero, J.M. Improvement of frequency regulation in VSG-based AC microgrid via adaptive virtual inertia. IEEE Trans. Power Electron. 2019, 35, 1589–1602. [Google Scholar] [CrossRef]
- Shi, T.; Sun, J.; Han, X.; Tang, C. Research on adaptive optimal control strategy of virtual synchronous generator inertia and damping parameters. IET Power Electron. 2023, 17, 121–133. [Google Scholar] [CrossRef]
- Yin, Q.; Ma, H.; Shan, Y. VSG Fuzzy Control Strategy of Energy Storage in Grid-connected Mode. In Proceedings of the 2022 Asia Power and Electrical Technology Conference (APET), Shanghai, China, 11–13 November 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 191–197. [Google Scholar]
- He, P.; Li, Z.; Jin, H.; Zhao, C.; Fan, J.; Wu, X. An adaptive VSG control strategy of battery energy storage system for power system frequency stability enhancement. Int. J. Electr. Power Energy Syst. 2023, 149, 109039. [Google Scholar] [CrossRef]
- Pournazarian, B.; Sangrody, R.; Lehtonen, M.; Gharehpetian, G.B.; Pouresmaeil, E. Simultaneous optimization of virtual synchronous generators parameters and virtual impedances in islanded microgrids. IEEE Trans. Smart Grid 2022, 13, 4202–4217. [Google Scholar] [CrossRef]
- Wu, Y.S.; Liao, J.T.; Yang, H.T. Parameter Optimization of Virtual Synchronous Generator Control Applied in Energy Storage and Photovoltaic Systems for an Island Microgrid. In Proceedings of the 2023 IEEE Power & Energy Society General Meeting (PESGM), Orlando, FL, USA, 16–20 July 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1–5. [Google Scholar]
- Saleh, A.; Omran, W.A.; Hasanien, H.M.; Tostado-Véliz, M.; Alkuhayli, A.; Jurado, F. Manta ray foraging optimization for the virtual inertia control of islanded microgrids including renewable energy sources. Sustainability 2022, 14, 4189. [Google Scholar] [CrossRef]
- Faragalla, A.; Abdel-Rahim, O.; Orabi, M.; Abdelhameed, E.H. Enhanced virtual inertia control for microgrids with high-penetration renewables based on whale optimization. Energies 2022, 15, 9254. [Google Scholar] [CrossRef]
- Wang, H.-N.; Liu, N.; Zhang, Y.-Y.; Feng, D.-W.; Huang, F.; Li, D.-S.; Zhang, Y.-M. Deep reinforcement learning: A survey. Front. Inf. Technol. Electron. Eng. 2020, 21, 1726–1744. [Google Scholar] [CrossRef]
- Shakya, A.K.; Pillai, G.; Chakrabarty, S. Reinforcement Learning Algorithms: A brief survey. Expert Syst. Appl. 2023, 231, 120495. [Google Scholar] [CrossRef]
- Annaswamy, A.M. Adaptive Control and Intersections with Reinforcement Learning. Annu. Rev. Control Robot. Auton. Syst. 2023, 6, 65–93. [Google Scholar] [CrossRef]
- Zhang, K.; Zhang, C.; Xu, Z.; Ye, S.; Liu, Q.; Lu, Z. A Virtual Synchronous Generator Control Strategy with Q-Learning to Damp Low Frequency Oscillation. In Proceedings of the 2020 Asia Energy and Electrical Engineering Symposium (AEEES), Chengdu, China, 29–31 May 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 111–115. [Google Scholar]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
- Wu, W.; Guo, F.; Ni, Q.; Liu, X.; Qiu, L.; Fang, Y. Deep Q-Network based Adaptive Robustness Parameters for Virtual Synchronous Generator. In Proceedings of the 2022 IEEE Transportation Electrification Conference and Expo, Asia-Pacific (ITEC Asia-Pacific), Haining, China, 28–31 October 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 1–4. [Google Scholar]
- Li, Y.; Gao, W.; Huang, S.; Wang, R.; Yan, W.; Gevorgian, V.; Gao, D.W. Data-driven optimal control strategy for virtual synchronous generator via deep reinforcement learning approach. J. Mod. Power Syst. Clean Energy 2021, 9, 919–929. [Google Scholar] [CrossRef]
- Park, M.; Lee, S.Y.; Hong, J.S.; Kwon, N.K. Deep Deterministic Policy Gradient-Based Autonomous Driving for Mobile Robots in Sparse Reward Environments. Sensors 2022, 22, 9574. [Google Scholar] [CrossRef]
- Tsai, J.; Chang, C.C.; Li, T. Autonomous Driving Control Based on the Technique of Semantic Segmentation. Sensors 2023, 23, 895. [Google Scholar] [CrossRef]
- Zheng, K.; Jia, X.; Chi, K.; Liu, X. DDPG-based joint time and energy management in ambient backscatter-assisted hybrid underlay CRNs. IEEE Trans. Commun. 2022, 71, 441–456. [Google Scholar] [CrossRef]
- Fujimoto, S.; Hoof, H.; Meger, D. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; PMLR: Stockholm, Sweden, 2018; pp. 1587–1596. [Google Scholar]
- Li, M.; Shu, S.; Wang, Y.; Yu, P.; Liu, Y.; Zhang, Z.; Hu, W.; Blaabjerg, F. Analysis and improvement of large-disturbance stability for grid-connected VSG based on output impedance optimization. IEEE Trans. Power Electron. 2022, 37, 9807–9826. [Google Scholar] [CrossRef]
- Chen, S.; Sun, Y.; Han, H.; Fu, S.; Luo, S.; Shi, G. A modified VSG control scheme with virtual resistance to enhance both small-signal stability and transient synchronization stability. IEEE Trans. Power Electron. 2023, 38, 6005–6014. [Google Scholar] [CrossRef]
- BS EN 50438:2007; Requirements for the Connection of Micro-Generators in Parallel with Public Low-Voltage Distribution Networks. European Committee for Electrotechnical Standardization: Brussels, Belgium, 2007.
- Yao, F.; Zhao, J.; Li, X.; Mao, L.; Qu, K. RBF neural network based virtual synchronous generator control with improved frequency stability. IEEE Trans. Ind. Informatics 2020, 17, 4014–4024. [Google Scholar] [CrossRef]
- Wu, D.; Lei, Y.; He, M.; Zhang, C.; Ji, L. Deep reinforcement learning-based path control and optimization for unmanned ships. Wirel. Commun. Mob. Comput. 2022, 2022, 7135043. [Google Scholar] [CrossRef]
- Pérez-Gil, Ó.; Barea, R.; López-Guillén, E.; Bergasa, L.M.; Gomez-Huelamo, C.; Gutiérrez, R.; Diaz-Diaz, A. Deep reinforcement learning based control for Autonomous Vehicles in CARLA. Multimed. Tools Appl. 2022, 81, 3553–3576. [Google Scholar] [CrossRef]
- Haarnoja, T.; Zhou, A.; Abbeel, P.; Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; PMLR: Stockholm, Sweden, 2018; pp. 1861–1870. [Google Scholar]
- Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.; Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; et al. Soft actor-critic algorithms and applications. arXiv 2018, arXiv:1812.05905. [Google Scholar]
- Chen, P.; Pei, J.; Lu, W.; Li, M. A deep reinforcement learning based method for real-time path planning and dynamic obstacle avoidance. Neurocomputing 2022, 497, 64–75. [Google Scholar] [CrossRef]
- Pan, J.; Huang, J.; Cheng, G.; Zeng, Y. Reinforcement learning for automatic quadrilateral mesh generation: A soft actor–critic approach. Neural Netw. 2023, 157, 288–304. [Google Scholar] [CrossRef]
- Wu, J.; Wei, Z.; Li, W.; Wang, Y.; Li, Y.; Sauer, D.U. Battery thermal-and health-constrained energy management for hybrid electric bus based on soft actor-critic DRL algorithm. IEEE Trans. Ind. Inform. 2020, 17, 3751–3761. [Google Scholar] [CrossRef]
Parameters | Value | Parameters | Value |
---|---|---|---|
BatchSize | 128 | ||
ReplayBuffer | 100,000 | ||
0.001 | |||
0.001 | |||
0.001 | |||
0.99 |
Methods | |||
---|---|---|---|
Fixed | 18.7 | 0.35 | 0.30 |
Adaptive | 8.9 | 0.25 | 0.26 |
Fuzzy | 6.2 | 0.27 | 0.25 |
DDPG | 5.1 | 0.24 | 0.20 |
SAC | 0.0 | 0.21 | 0.12 |
Methods | |||
---|---|---|---|
Fixed | 19.2 | 0.18 | 0.30 |
Adaptive | 5.8 | 0.14 | 0.26 |
Fuzzy | 9.2 | 0.15 | 0.25 |
DDPG | 5.5 | 0.12 | 0.19 |
SAC | 0.0 | 0.11 | 0.12 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Lu, C.; Zhuan, X. Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic. Sensors 2024, 24, 2035. https://doi.org/10.3390/s24072035
Lu C, Zhuan X. Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic. Sensors. 2024; 24(7):2035. https://doi.org/10.3390/s24072035
Chicago/Turabian StyleLu, Chuang, and Xiangtao Zhuan. 2024. "Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic" Sensors 24, no. 7: 2035. https://doi.org/10.3390/s24072035
APA StyleLu, C., & Zhuan, X. (2024). Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic. Sensors, 24(7), 2035. https://doi.org/10.3390/s24072035