High-Order Grid-Connected Filter Design Based on Reinforcement Learning
Abstract
:1. Introduction
- The proposed method can simultaneously optimize the structure and parameters of the filter.
- By using RL, the proposed method can automatically design filters with certain specifications.
- By using the proposed method, the performances of the designed higher-order grid-connected filters are greatly improved.
2. Learning Architecture
- Action:
- 2.
- State:
Algorithm 1. RL-based LTLCL filter design algorithm |
1: Initialize highest score 2: for i = 1 to M do 3: Initialize replay buffer and 4: for j = 1 to N do 5: Receive observation state 6: Compute for every step t 7: Compute for every step t and add to 8: if then 9: 10: end if 11: Store transition (; ; ) in 12: end if 13: for j = 1 to K do 14: Compute the optimization target: 15: 16: Update the parameters θ of the networks with by ADAMW 17: end for 18: end for |
- 3.
- Reward:
3. LTLCL-Type Grid-Connected Filters and Electrical Constraints
3.1. LTLCL-Type Grid-Connected Filter
3.2. System Configuration
3.3. Electrical Constraints
- (1)
- The reactive power does not exceed 5% of the rated power, so the sum of capacitors is limited by
- (2)
- The voltage drop across the total series inductors does not exceed 10% of the rated voltage rms. Therefore, the total series inductance value is limited byThe reward is constructed as follows:
- (3)
- The inverter-side inductance determines the maximum peak-to-peak current ripple. The choice has been made to enable support for ripple current of up to 60% of the rated current. The constraint is as follows:This constraint can be used to determine the lower and upper bounds of .
- (4)
- The designed filter consists of trap branches. The zero-impedance paths formed by the trap branches can effectively attenuate harmonic components at multiple switching frequencies. Therefore, it is necessary to ascertain whether the harmonic component at ( + 1) is within 0.3% of the fundamental current. As shown in (8), the harmonic amplitude of the output voltage can be calculated, as follows:In this paper, when + 1 is odd, the investigation is limited to harmonic amplitudes for k = ±2, ±4; when + 1 is even, consideration is restricted to harmonic amplitudes for k = ±1, ±3. Therefore, the maximized harmonic components around ( + 1) can be described as:The constraint can be derived as:denotes the reward when the constraint (16) is satisfied and is a constant. In this paper, equals 10.
- (5)
- To avoid system instability caused by harmonic resonance in the high-frequency band and low-frequency band of the filter, its characteristic resonant frequency needs to be limited between 10 and 0.5. With the same total capacitance, the first resonant frequency of the LTLCL filter is approximately the resonant frequency of the LCL filter. Therefore, the resonant frequency of the LCL filter can be used to approximate the first resonant frequency of the LTLCL filter. The first resonant frequency can be derived as:Hence, is constructed as follows:is also a constant larger than 0. In this paper, equals 10.
- (6)
- also includes a limit on the total capacitance, i.e., the smaller the total capacity, the higher the . In addition to the capacitance, the objective is to minimize the total inductance. Therefore, with respect to the total inductance, a penalty term can be constructed as follows:
- (7)
- To limit the order of the filter, a penalty term related to the order is therefore introduced. Let the coefficient be , which in this case equals 0.5. Then the reward is constructed as follows:
4. Verification
4.1. Train Result
4.2. Solution Comparsion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Heydt, G.T. The Next Generation of Power Distribution Systems. IEEE Trans. Smart Grid 2010, 1, 225–235. [Google Scholar] [CrossRef]
- Erika, T.; Holmes, D.G. Grid current regulation of a three-phase voltage source inverter with an LCL input filter. IEEE Trans. Power Electron. 2003, 18, 888–895. [Google Scholar] [CrossRef]
- Islam, M.; Afrin, N.; Mekhilef, S. Efficient Single Phase Transformerless Inverter for Grid-Tied PVG System with Reactive Power Control. IEEE Trans. Sustain. Energy 2016, 7, 1205–1215. [Google Scholar] [CrossRef]
- Liserre, M.; Blaabjerg, F.; Hansen, S. Design and control of an LCL-filter-based three-phase active rectifier. IEEE Trans. Ind. Appl. 2005, 41, 1281–1291. [Google Scholar] [CrossRef]
- Wu, X.; Li, X.; Yuan, X.; Geng, Y. Grid Harmonics Suppression Scheme for LCL-Type Grid-Connected Inverters Based on Output Admittance Revision. IEEE Trans. Sustain. Energy 2015, 6, 411–421. [Google Scholar] [CrossRef]
- Poongothai, C.; Vasudevan, K. Design of LCL Filter for Grid-Interfaced PV System Based on Cost Minimization. IEEE Trans. Ind. Appl. 2019, 55, 584–592. [Google Scholar] [CrossRef]
- Wu, W.; He, Y.; Tang, T.; Blaabjerg, F. A New Design Method for the Passive Damped LCL and LLCL Filter-Based Single-Phase Grid-Tied Inverter. IEEE Trans. Ind. Electron. 2013, 60, 4339–4350. [Google Scholar] [CrossRef]
- Wu, W.; He, Y.; Blaabjerg, F. An LLCL Power Filter for Single-Phase Grid-Tied Inverter. IEEE Trans. Power Electron. 2012, 27, 782–789. [Google Scholar] [CrossRef]
- Xu, D.; Wang, F.; Ruan, Y.; Mao, H.; Zhang, W.; Yang, Y. Topology deduction and analysis of grid-interfacing filters. Diangong Jishu Xuebao/Trans. China Electrotech. Soc. 2015, 30, 15–25. [Google Scholar]
- Xu, J.; Yang, J.; Ye, J.; Zhang, Z.; Shen, A. An LTCL Filter for Three-Phase Grid-Connected Converters. IEEE Trans. Power Electron. 2014, 29, 4322–4338. [Google Scholar] [CrossRef]
- Zhang, Z.; He, C.; Ye, J.; Xu, J.; Pan, L. Switching ripple suppressor design of the grid-connected inverters: A perspective of many-objective optimization with constraints handling. Swarm Evol. Comput. 2019, 44, 293–303. [Google Scholar] [CrossRef]
- Solatialkaran, D.; Zare, F.; Saha, T.K.; Sharma, R. A Novel Approach in Filter Design for Grid-Connected Inverters Used in Renewable Energy Systems. IEEE Trans. Sustain. Energy 2020, 11, 154–164. [Google Scholar] [CrossRef]
- Wang, X.; Ruan, X.; Liu, S.; Tse, C.K. Full Feedforward of Grid Voltage for Grid-Connected Inverter with LCL Filter to Suppress Current Distortion Due to Grid Voltage Harmonics. IEEE Trans. Power Electron. 2010, 25, 3119–3127. [Google Scholar] [CrossRef]
- Jiao, Y.; Lee, F.C. LCL Filter Design and Inductor Current Ripple Analysis for a Three-Level NPC Grid Interface Converter. IEEE Trans. Power Electron. 2015, 30, 4659–4668. [Google Scholar] [CrossRef]
- Beres, R.N.; Wang, X.; Blaabjerg, F.; Liserre, M.; Bak, C.L. Optimal Design of High-Order Passive-Damped Filters for Grid-Connected Applications. IEEE Trans. Power Electron. 2016, 31, 2083–2098. [Google Scholar] [CrossRef]
- Cai, Y.; He, Y.; Zhou, H.; Liu, J. Design Method of LCL Filter for Grid-Connected Inverter Based on Particle Swarm Optimization and Screening Method. IEEE Trans. Power Electron. 2021, 36, 10097–10113. [Google Scholar] [CrossRef]
- Liserre, M.; Dell’Aquila, A.; Blaabjerg, F. Genetic algorithm-based design of the active damping for an LCL-filter three-phase active rectifier. IEEE Trans. Power Electron. 2004, 19, 76–86. [Google Scholar] [CrossRef]
- Cho, J.H.; Kim, D.-H.; Virikova, M.; Sinak, P. Design of LCL filter using hybrid intelligent optimization for photovoltaic system. In Ubiquitous Computing and Multimedia Applications—Second International Conference, UCMA 2011, Proceedings, 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 90–97. [Google Scholar] [CrossRef]
- Yan, W.; Dong, M.; Li, L.; Liang, R.; Xu, C. Filter Design for Single-Phase Grid-Connected Inverter Based on Reinforcement Learning. In Proceedings of the 2022 IEEE 17th Conference on Industrial Electronics and Applications (ICIEA), Chengdu, China, 16–19 December 2022; pp. 261–266. [Google Scholar] [CrossRef]
- Mirhoseini, A.; Goldie, A.; Yazgan, M.; Jiang, J.W.; Songhori, E.; Wang, S.; Lee, Y.-J.; Johnson, E.; Pathak, O.; Nazi, A.; et al. A graph placement methodology for fast chip design. Nature 2021, 594, 207–212. [Google Scholar] [CrossRef] [PubMed]
- Cao, W.; Benosman, M.; Zhang, X.; Ma, R. Domain Knowledge-Based Automated Analog Circuit Design with Deep Reinforcement Learning. arXiv 2022, arXiv:2202.13185. [Google Scholar]
- Dong, M.; Liang, R.; Yang, J.; Xu, C.; Song, D.; Wan, J. Topology Derivation of Multiport DC–DC Converters Based on Reinforcement Learning. IEEE Trans. Power Electron. 2023, 38, 5055–5064. [Google Scholar] [CrossRef]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
- Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019; International Conference on Learning Representations, ICLR: New Orleans, LA, USA, 2019. [Google Scholar]
- Khan, D.; Qais, M.; Sami, I.; Hu, P.; Zhu, K.; Abdelaziz, A.Y. Optimal LCL-filter design for a single-phase grid-connected inverter using metaheuristic algorithms. Comput. Electr. Eng. 2023, 110, 108857. [Google Scholar] [CrossRef]
Parameter | Value | Parameter | Value |
---|---|---|---|
Batch Size | 64 | Learning Rata | 0.0003 |
Buffer Size | 20,000 | Discount Factor | 0.99 |
Actor hidden layer size | [64,64] | Critic hidden layer size | [64,64] |
Parameter | Value |
---|---|
220 V | |
5000 W | |
400 V | |
100π rad/s | |
20,000π rad/s | |
11 A |
Components (x) | |||
---|---|---|---|
0.76 | 2.20 | 0.015 | |
0.092 | 9.20 | 0.092 | |
0.11 | 5.48 | 0.54 | |
- |
2 | 3 | 4 | 5 | ||
---|---|---|---|---|---|
Parameter | |||||
(mH) | 2.08 | 1.38 | 1.08 | 0.98 | |
(mH) | 0.46 | 0.28 | 0.27 | 0.20 | |
(mH) | 0.28 | 0.27 | 0.18 | 0.28 | |
(F) | 2.04 | 0.59 | 0.81 | 0.70 | |
(F) | 0.86 | 0.92 | 1.02 | 0.70 | |
(mH) | 0.29 | 0.27 | 0.25 | 0.36 | |
(F) | 1.24 | 0.65 | 0.22 | 0.49 | |
(mH) | 0.05 | 0.098 | 0.29 | 0.13 | |
(F) | - | 1.45 | 0.65 | 0.49 | |
(mH) | - | 0.019 | 0.044 | 0.058 | |
(F) | - | - | 0.65 | 0.92 | |
(mH) | - | - | 0.024 | 0.017 | |
(F) | - | - | - | 0.92 | |
(mH) | - | - | - | 0.011 | |
(mH) | 3.15 | 2.32 | 2.14 | 2.04 | |
(F) | 4.14 | 3.60 | 3.34 | 4.20 |
Method | Proposed | NSGA-II | SMS-EMOA |
---|---|---|---|
(mH) | 1.08 | 1.63 | 1.82 |
(mH) | 0.27 | 0.85 | 1.00 |
(mH) | 0.18 | 0.39 | 0.77 |
F) | 0.81 | 0.32 | 2.41 |
F) | 1.02 | 1.22 | 1.12 |
(mH) | 0.25 | 0.20 | 0.22 |
F) | 0.22 | 1.72 | 0.43 |
(mH) | 0.29 | 0.037 | 0.15 |
F) | 0.65 | 1.46 | 1.05 |
(mH) | 0.044 | 0.019 | 0.027 |
F) | 0.65 | 0.68 | 0.46 |
(mH) | 0.024 | 0.023 | 0.035 |
(mH) | 2.14 | 3.20 | 4.00 |
F) | 3.34 0.48 | 5.40 0.55 | 5.48 0.54 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liao, L.; Liu, X.; Zhou, J.; Yan, W.; Dong, M. High-Order Grid-Connected Filter Design Based on Reinforcement Learning. Energies 2025, 18, 586. https://doi.org/10.3390/en18030586
Liao L, Liu X, Zhou J, Yan W, Dong M. High-Order Grid-Connected Filter Design Based on Reinforcement Learning. Energies. 2025; 18(3):586. https://doi.org/10.3390/en18030586
Chicago/Turabian StyleLiao, Liqing, Xiangyang Liu, Jingyang Zhou, Wenrui Yan, and Mi Dong. 2025. "High-Order Grid-Connected Filter Design Based on Reinforcement Learning" Energies 18, no. 3: 586. https://doi.org/10.3390/en18030586
APA StyleLiao, L., Liu, X., Zhou, J., Yan, W., & Dong, M. (2025). High-Order Grid-Connected Filter Design Based on Reinforcement Learning. Energies, 18(3), 586. https://doi.org/10.3390/en18030586