High-Order Grid-Connected Filter Design Based on Reinforcement Learning

Abstract

In grid-connected inverter systems, grid-connected filters can effectively eliminate harmonics. High-order filters perform better than conventional filters in eliminating harmonics and can reduce costs. For high-order filters, the use of multi-objective optimization algorithms for parameter optimization presupposes that the circuit structure must be known. To realize the design of the filter structure and related circuit parameters that meet the requirements of the grid-connected inverter system during the design process, this paper proposes a reinforcement learning (RL) method for designing higher-order filters. Our approach combines key domain knowledge with the characteristics of structural changes to obtain some constraints, which are then processed to obtain reward and are incorporated into RL strategy learning to determine the optimal structure and corresponding circuit parameters. The proposed method realizes the simultaneous design of parameters and structures in filter design, which greatly improves the efficiency of filter design. Simulation results for the corresponding grid-connected system setup show that the grid-connected filter designed by our method demonstrates a good performance in terms of filter dimension, harmonic rejection, and total harmonic distortion.

1. Introduction

Grid-connected inverters are crucial in sustainable energy generation systems [1]. However, the use of pulse width modulation (PWM) injects a large amount of harmonics into the grid-connected inverter system [2]. Therefore, it is usually necessary to insert a passive low-pass filter between the inverter and the grid to eliminate the effect of higher harmonics on the system [3].

Specifically, LCL-type filters are widely used because of their excellent high-frequency attenuation performance and the simplicity of their design method [4,5]. However, such filters usually require larger inductors, which not only leads to higher costs, but also causes larger voltage drops [6].

Recently, to obtain higher harmonic attenuation and lower inductor cost, LC series resonant branches have been widely used in the design of grid-connected filters [7]. The LLCL filter circuit contains a series resonant branch [8], which particularly attenuates the current ripple component at the switching frequency and lowers the total inductance. However, its attenuation in the high frequency band is only −20 dB/decade. Topological derivation of filters up to the fifth order reveals that the addition of LC resonant branches to the LCL filter provides better harmonic rejection [9]. The LCL filter attenuates at −40 dB/decade in the high-frequency band. Hence, higher-order grid-connected filters based on LCL and LC resonant branches are becoming popular. The LC resonant branch is also called a trap branch. The inductor–trap–capacitor–inductor (LTCL) filter can attenuate the current ripple components at the multiples of switching frequencies and guarantee −60 dB/decade attenuation in the high-frequency band [10]. The number of trap branches (n) of the LTCL filter is not fixed. To reduce the capacity of individual inductors, LTLCL filters are proposed based on the LTCL structure [11].

Filter parameter design is also an important aspect. Conventional methods are based on expert experience for dynamic tuning of parameters [12]. Expert design knowledge comes mainly from the constraints of some grid-connected standards and from trial-and-error experience. In other words, the parameter range is roughly determined according to the constraints of grid-connected current harmonics [13], current ripple [14], reactive power [15], etc., and then the parameters are tuned step by step. Based on the existing domain knowledge, some advanced methods are used to automate the parameter design. LCL parameter optimization methods based on particle swarm algorithm (PSO) and genetic algorithm (GA) are proposed in [16,17], respectively, where the designed objective functions include total inductance, total harmonic distortion (THD), etc. A clone-selection algorithm is used to optimize the parameters of the LCL filter [18]. Higher-order filter parameters such as LTLCL can also be obtained using a multi-objective optimization algorithm [11]. However, the parameter optimization for LTLCL presupposes that the topology must be known. In practice, optimization of the filter topology takes precedence over parameter optimization.

Artificial intelligence techniques are now widely used in the field of circuit design and power electronics topology design [19], and in particular, reinforcement learning (RL) is popular among designers. A step-by-step chip layout method is proposed based on reinforcement learning [20]. An analog circuit design method based on GCN-RL is proposed in [21]. In the field of power electronics, researchers have also used reinforcement learning to do topology derivation for DC–DC converters [22]. In summary, RL is flexible in the design process, so RL studies for circuits can also be used for filter structure and parameter optimization.

To further explore the potential of RL in high-order filter design, this paper focuses on the expandable LTLCL high-order filter, which also features a large search space due to its variable structure. The major contributions of this article are summarized as follows:

• The proposed method can simultaneously optimize the structure and parameters of the filter.
• By using RL, the proposed method can automatically design filters with certain specifications.
• By using the proposed method, the performances of the designed higher-order grid-connected filters are greatly improved.

The rest of this paper is organized as follows. The proposed learning architecture for filter design is presented in Section 2. Section 3 describes the specific characteristics and electrical design constraints of the LTLCL filter. In Section 4, the learning results are illustrated, and the highest-scoring filter circuit is verified by simulation and experimental results. Finally, Section 5 concludes the paper.

2. Learning Architecture

As shown in Figure 1, the proposed learning architecture is based on RL. In this framework, the design of grid-connected filters can be regarded as a step-by-step design process. The RL agent changes the circuit structure and parameters through actions based on the circuit state, and obtains the reward given by the evaluation program. The reward from changing the circuit is equal to 0 for all steps except the last one, which consists of various electrical constraints on the filter. Through multiple iterations, the agent improves the quality of its decisions and eventually learns the best strategy for maximizing the reward.

In this paper, the LTLCL-type filter is the object of study, of which structure is as shown in Figure 2. The number of trap branches of LTLCL is equal to $n_{\mathit{structure}}$. Then, including fixed components, 2$n_{\mathit{structure}}$ + 4 components need to be determined for a complete filter design.

In our learning framework, the action $a_t$, state $s_t,$ and reward $r_t$ are defined as follows:

• Action:

In this paper, the action consists of two parts. At the beginning of a complete filter design, a0 determines the number of trap branches, i.e., $n_{\mathit{structure}}$. Component parameters are determined by each subsequent step of the action at until a complete circuit is obtained. Before designing the filter, the lower and upper bounds [$X_{\mathit{min}}$, $X_{\mathit{max}}$] for the parameter x will be approximately determined, followed by the utilization of a discrete action space to adjust the component parameters. Then, the equation for the variation of parameter x is defined as follows:

$$\mathrm{x}={\mathrm{X}}_{\min }+{\mathrm{a}}_{\mathrm{t}}\times \Delta$$

(1)

where $\Delta$ is the step size of the parameter change, and $a_t$ takes values in the range [0, $X_{\mathit{max}}-{\displaystyle \frac{X_{\mathit{min}}}{\Delta }}$].

2.

State:

A complete circuit requires the determination of the number of trap branches $n_{\mathit{structure}}$ and 2$n_{\mathit{structure}}$ + 4 component parameters. This means that the state will contain two parts, one part is a valid dimension for determining the circuit structure and parameters, and the other part is considered as an invalid dimension. Therefore, the dimension of the state should be set to a larger value $D_{\mathit{state}}$.

Algorithm 1 is the key to the proposed RL-based LTLCL filter design. It initializes necessary variables like the highest score and buffers. In each episode, it gets the state, computes relevant values, updates the highest score if needed, and stores transitions. After all episodes in an epoch, it calculates the optimization target and uses ADAMW to update network parameters. Through iterative training across epochs, it enables the agent to learn the optimal filter design strategy considering electrical constraints and the reward mechanism.

Algorithm 1. RL-based LTLCL filter design algorithm

Input:  Number of epochs M, number of episodes sampled per epoch N, number of network updates per epoch K, other hyperparameters of the model and the ADAMW optimizer Output:  Highest scoring filter  1: Initialize highest score ${sc}_{max}$  2: for i = 1 to M do  3:  Initialize replay buffer $\mathcal{P}$ and $\mathcal{G}$  4:  for j = 1 to N do  5:   Receive observation state $S_t$  6:   Compute ${\hat{A}}_t$ for every step t  7:   Compute $G_t$ for every step t and add to $\mathcal{G}$  8:   if $r_t>0$ then  9:     ${\mathit{sc}}_{\mathit{max}}=\mathit{max}({\mathit{sc}}_{\mathit{max}},r_t)$  10:    end if  11:    Store transition ($s_t$; $A_t$; $r_t$) in $\mathcal{P}$  12:   end if  13:   for j = 1 to K do  14:    Compute the optimization target:  15:    $$\begin{gathered} J={\mathrm{E}}_{\left({\hat{A}}_t,s_t,r_{\pi }\right)~\mathrm{P},G_t~G}[\mathrm{E}\left[\mathit{min}\left(r_{\pi }{\hat{A}}_t,\mathit{clip}\left(r_{\pi },1-ϵ,1+ϵ\right){\hat{A}}_t\right)\right]-c_1\mathrm{E}[(V(s_t)- \\ {G}_t{)}^2]+c_{2}H\left(\pi \right(•\left)\right|s_t)] \end{gathered}$$  16:    Update the parameters θ of the networks with ${\nabla }_{\theta }J$ by ADAMW  17:   end for  18: end for

3.

Reward:

The reward is an important indicator used to measure the performance of the designed filter. The reward is always equal to 0 until the end of an episode. When the state is complete, it is then passed into the circuit evaluation program to obtain sc. The reward defined as follows:

$${\mathrm{r}}_{\mathrm{t}}=\left\{\begin{array}{c}0,\hspace{1em}\hspace{1em}\mathrm{t}\ne {\mathrm{D}}_{\mathrm{state}}\\ \mathrm{s}\mathrm{c},\hspace{1em}\hspace{1em}\mathrm{t}={\mathrm{D}}_{\mathrm{state}}\end{array}\right.$$

(2)

where sc is the evaluation score. The specific electrical constraints and the construction of the reward can be seen in Equation (21).

In this paper, the Proximal Policy Optimization (PPO) [23] algorithm is employed for training the agent. PPO uses a surrogate objective function to optimize the policy network (Actor) and value function (Critic). The actor network represents the policy and outputs the probability of taking each action in a given state. The critic network estimates the value function and outputs the expected reward for a given state. PPO uses the actor to collect experience data from the environment. It then utilizes this experience to train both the actor and critic networks. The goal of PPO is to maximize the optimization target J defined as:

$$\mathrm{J}=\mathrm{E}[{\mathrm{J}}^{\mathrm{act}}-{\mathrm{c}}_1\mathrm{E}[{\left(\mathrm{V}\right({\mathrm{s}}_{\mathrm{t}})-{\mathrm{G}}_{\mathrm{t}})}^2\left]{\mathrm{c}}_2\mathrm{H}\left(\mathsf{\pi }\right(•\left)\right|{\mathrm{s}}_{\mathrm{t}})\right]$$

(3)

$$J^{act}=\mathbb{E}\left[\min \left({\mathrm{r}}_{\mathsf{\pi }}{\hat{\mathrm{A}}}_{\mathrm{t}},\mathrm{clip}\left({\mathrm{r}}_{\mathsf{\pi }},1-\mathsf{ϵ},1+\mathsf{ϵ}\right){\hat{\mathrm{A}}}_{\mathrm{t}}\right)\right]$$

(4)

where $c_1$, $c_2,$ and $ϵ$ are hyperparameters, $H\left(\pi \right(•\left)\right|s_t)$ represents the entropy regularization term of the policy distribution, and ${\hat{A}}_t$ is the estimated advantage function defined as:

$${\hat{\mathrm{A}}}_{\mathrm{t}}=\sum _{\mathrm{i}=0}^{\infty }{\left(\mathsf{\gamma }\mathsf{\lambda }\right)}^{\mathrm{i}}[{\mathrm{r}}_{\mathrm{t}}+\mathsf{\gamma }{\mathrm{V}}_{\mathrm{old}}\left({\mathrm{s}}_{\mathrm{t}+\mathrm{i}+1}\right)-{\mathrm{V}}_{\mathrm{old}}\left({\mathrm{s}}_{\mathrm{t}+\mathrm{i}}\right)]$$

(5)

where $\gamma$ and $\lambda$ are hyperparameters, the policy is $\pi \left(a_t\right|s_t)$, $r_{\pi }$ is equal to ${\displaystyle \frac{\pi \left(a_t\right|s_t)}{{\pi }_{\mathit{old}}\left(a_t\right|s_t)}}$, and $G_t$ is the accumulated discounted reward. $J^{\mathit{act}}$ represents the expected return, which is the long-term cumulative reward that the policy expects to obtain after taking an action in a certain state, and $V_{\mathit{old}}\left(s_{t+i+1}\right)$ is the value estimate of the future state by the old Critic network.

The RL-based filter design algorithm is as shown in Algorithm 1. In each epoch, N times $A_t$ and $G_t$ are calculated, then the highest score is recorded along with the corresponding circuit. The algorithm then updates the network hyperparameters by computing the optimization objective using ADAMW [24].

3. LTLCL-Type Grid-Connected Filters and Electrical Constraints

3.1. LTLCL-Type Grid-Connected Filter

The LTLCL filter topology is as shown in Figure 2, where $L_1$ is the inverter-side inductor, $L_2$ is the splitting grid-side inductor, and $L_3$ is the gride-side inductor. The topology consists of $n_{\mathit{structure}}$ parallel LC series resonant circuits, where the i-th branch involves the resonant capacitor $C_i$ and the resonant inductor $L_{\mathit{fi}}$.

LTLCL filter has better attenuation of higher harmonics. The LTLCL filter transfer function $i_g\left(s\right)/u_i\left(s\right)$ can be derived as:

$$\begin{array}{c}{\mathrm{G}}_{{\mathrm{u}}_{\mathrm{i}}-{\mathrm{i}}_{\mathrm{g}}}\left(\mathrm{s}\right)={\displaystyle \frac{{\mathrm{i}}_{\mathrm{g}}\left(\mathrm{s}\right)}{{\mathrm{u}}_{\mathrm{i}}\left(\mathrm{s}\right)}}={\displaystyle \frac{\mathrm{Z}}{{\mathsf{\alpha }}_1{\mathrm{s}}^2+{\mathsf{\alpha }}_2{\mathrm{s}}^4+\mathrm{Z}[{\mathsf{\alpha }}_3\mathrm{s}+{\mathsf{\alpha }}_4{\mathrm{s}}^3]}}\hfill \\ \mathrm{Z}={\displaystyle \frac{1}{\frac{1}{{\mathrm{L}}_{\mathrm{f}1}\mathrm{s}+\frac{1}{{\mathrm{C}}_1\mathrm{s}}}+\dots +\frac{1}{{\mathrm{L}}_{\mathrm{f}{\mathrm{n}}_{\mathrm{structure}}}\mathrm{s}+\frac{1}{{\mathrm{C}}_{{\mathrm{n}}_{\mathrm{structure}}}\mathrm{s}}}}}\hfill \\ {\mathsf{\alpha }}_1={\mathrm{L}}_1\left({\mathrm{L}}_2+{\mathrm{L}}_3\right)\hfill \\ {\mathsf{\alpha }}_2={\mathrm{L}}_1{\mathrm{L}}_2{\mathrm{L}}_3{\mathrm{C}}_{\mathrm{f}}\hfill \\ {\mathsf{\alpha }}_3={\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3\hfill \\ {\mathsf{\alpha }}_4={\mathrm{L}}_3{\mathrm{C}}_{\mathrm{f}}({\mathrm{L}}_1+{\mathrm{L}}_2)\hfill \end{array}$$

(6)

Figure 3 shows the Bode diagrams of LTLCL with two trap branches ($n_{\mathit{structure}}$ = 2), where ${\omega }_{s\omega }$ is the switching angular frequency. Since the resonant frequency of the trap branch is a multiplicative switching frequency, when Ci is determined, $L_{\mathit{fi}}$ can be calculated by

$${\mathrm{L}}_{\mathrm{f}\mathrm{i}}={\displaystyle \frac{1}{{\mathrm{i}}^2{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}^2{\mathrm{C}}_{\mathrm{i}}}}$$

(7)

Thus, the number of component parameters designed becomes $n_{\mathit{structure}}$ + 4.

3.2. System Configuration

Figure 4 shows the three-phase grid-connected inverter. $V_{\mathit{dc}}$ is the DC link voltage. In this paper, the sinusoidal PWM (SPWM) is used. The a-phase bridge output voltage can be calculated as:

$$\begin{array}{c}\hspace{1em}{\mathrm{u}}_{\mathrm{a}\mathrm{N}}\left(\mathrm{t}\right)={\displaystyle \frac{{\mathrm{M}}_{\mathrm{r}}{\mathrm{V}}_{\mathrm{dc}}}{2}}\sin {\mathsf{\omega }}_0\mathrm{t}\hfill \\ +{\displaystyle \frac{2{\mathrm{V}}_{\mathrm{dc}}}{\mathsf{\pi }}}{\displaystyle \sum _{\mathrm{h}=1,3\dots }^{\infty }}{\displaystyle \sum _{\mathrm{k}=0,\pm 2\dots }^{\infty }}{\displaystyle \frac{4}{3}}{\displaystyle \frac{{\mathrm{J}}_{\mathrm{k}}\left(\frac{\mathrm{h}{\mathrm{M}}_{\mathrm{r}}\mathsf{\pi }}{2}\right)}{\mathrm{h}}}\sin {\displaystyle \frac{\mathrm{h}\mathsf{\pi }}{2}}{\sin }^2{\displaystyle \frac{\mathrm{k}\mathsf{\pi }}{3}}\cos (\mathrm{h}{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\mathrm{t}+\mathrm{k}{\mathsf{\omega }}_0\mathrm{t})\hfill \\ +{\displaystyle \frac{2{\mathrm{V}}_{\mathrm{dc}}}{\mathsf{\pi }}}{\displaystyle \sum _{\mathrm{h}=2,4\dots }^{\infty }}{\displaystyle \sum _{\mathrm{k}=0,\pm 1\dots }^{\infty }}{\displaystyle \frac{4}{3}}{\displaystyle \frac{{\mathrm{J}}_{\mathrm{k}}\left(\frac{\mathrm{h}{\mathrm{M}}_{\mathrm{r}}\mathsf{\pi }}{2}\right)}{\mathrm{h}}}\cos {\displaystyle \frac{\mathrm{h}\mathsf{\pi }}{2}}{\sin }^2{\displaystyle \frac{\mathrm{k}\mathsf{\pi }}{3}}\sin (\mathrm{h}{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\mathrm{t}+\mathrm{k}{\mathsf{\omega }}_0\mathrm{t})\hfill \end{array}$$

(8)

where $M_r$ is the modulation ratio, ${\omega }_0$ is the modulation frequency, ${\omega }_{s\omega }$ is the carrier frequency, and J is the Bessel function.

For odd multiples (h = 1, 3, 5, …) of the carrier frequency harmonic nearby, $\mathit{cos}{\displaystyle \frac{h\pi }{2}}$ = 0; for even multiples (h = 2, 4, 6, …) of the carrier switching frequency, $\mathit{sin}{\displaystyle \frac{h\pi }{2}}$ = 0. Therefore, the harmonic components of the bridge arm output voltage only appear at frequencies where h + k is odd.

3.3. Electrical Constraints

(1)

The reactive power does not exceed 5% of the rated power, so the sum of capacitors is limited by

$$\sum _{\mathrm{i}=1}^{{\mathrm{n}}_{\mathrm{structure}}}{\mathrm{C}}_{\mathrm{i}}+{\mathrm{C}}_{\mathrm{f}}\le {\displaystyle \frac{5\%·{\mathrm{P}}_{\mathrm{rated}}}{{\mathrm{V}}_{\mathrm{g}}^2{\mathsf{\omega }}_0}}$$

(9)

where $P_{\mathit{rated}}$ is the rated power. Based on the above inequality, the reward $r_{\mathit{circuit}\_1}$ is constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_1}={\displaystyle \frac{5\%·{\mathrm{P}}_{\mathrm{rated}}}{{\mathrm{V}}_{\mathrm{g}}^2{\mathsf{\omega }}_0}}-\sum _{\mathrm{i}=1}^{{\mathrm{n}}_{\mathrm{structure}}}{\mathrm{C}}_{\mathrm{i}}-{\mathrm{C}}_{\mathrm{f}}$$

(10)

(2)

The voltage drop across the total series inductors does not exceed 10% of the rated voltage rms. Therefore, the total series inductance value is limited by

$${\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3\le {\displaystyle \frac{10\%·{\mathrm{V}}_{\mathrm{g}}^2}{{\mathrm{P}}_{\mathrm{rated}}{\mathsf{\omega }}_0}}$$

(11)

The reward $r_{\mathit{circuit}\_2}$ is constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_2}={\displaystyle \frac{10\%·{\mathrm{V}}_{\mathrm{g}}^2}{{\mathrm{P}}_{\mathrm{rated}}{\mathsf{\omega }}_0}}-({\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3)$$

(12)

(3)

The inverter-side inductance $L_1$ determines the maximum peak-to-peak current ripple. The choice has been made to enable support for ripple current of up to 60% of the rated current. The constraint is as follows:

$$20\%\le {\displaystyle \frac{2\mathsf{\pi }{\mathrm{V}}_{\mathrm{dc}}}{8{\mathrm{L}}_1{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}{\mathrm{I}}_{\mathrm{ref}}}}\le 60\%$$

(13)

This constraint can be used to determine the lower and upper bounds of $L_1$.

(4)

The designed filter consists of $n_{\mathit{structure}}$ trap branches. The zero-impedance paths formed by the trap branches can effectively attenuate harmonic components at multiple switching frequencies. Therefore, it is necessary to ascertain whether the harmonic component at ($n_{\mathit{structure}}$ + 1)$f_{s\omega }$ is within 0.3% of the fundamental current. As shown in (8), the harmonic amplitude $A_{\mathit{nk}}$ of the output voltage can be calculated, as follows:

$${\mathrm{A}}_{\mathrm{nk}}=\left\{\begin{array}{c}\left|{\displaystyle \frac{8{\mathrm{V}}_{\mathrm{dc}}}{3\mathsf{\pi }}}{\displaystyle \frac{{\mathrm{J}}_{\mathrm{k}}\left(\frac{\mathrm{n}{\mathrm{M}}_{\mathrm{r}}\mathsf{\pi }}{2}\right)}{\mathrm{n}}}\sin {\displaystyle \frac{\mathrm{n}\mathsf{\pi }}{2}}{\sin }^2{\displaystyle \frac{\mathrm{k}\mathsf{\pi }}{3}}\right|,\hspace{1em}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{o}\mathrm{d}\mathrm{d}\\ \left|{\displaystyle \frac{8{\mathrm{V}}_{\mathrm{dc}}}{3\mathsf{\pi }}}{\displaystyle \frac{{\mathrm{J}}_{\mathrm{k}}\left(\frac{\mathrm{n}{\mathrm{M}}_{\mathrm{r}}\mathsf{\pi }}{2}\right)}{\mathrm{n}}}\cos {\displaystyle \frac{\mathrm{n}\mathsf{\pi }}{2}}{\sin }^2{\displaystyle \frac{\mathrm{k}\mathsf{\pi }}{3}}\right|,\hspace{1em}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\end{array}\right.$$

(14)

In this paper, when $n_{\mathit{structure}}$ + 1 is odd, the investigation is limited to harmonic amplitudes for k = ±2, ±4; when $n_{\mathit{structure}}$ + 1 is even, consideration is restricted to harmonic amplitudes for k = ±1, ±3. Therefore, the maximized harmonic components around ($n_{\mathit{structure}}$ + 1)$f_{s\omega }$ can be described as:

$${\mathrm{A}}_{\max }=\max \left({\mathrm{A}}_{\left({\mathrm{n}}_{\mathrm{structure}}+1\right)\mathrm{k}}\right)$$

(15)

The constraint can be derived as:

$${\displaystyle \frac{{\mathrm{A}}_{\max }\left|{\mathrm{G}}_{{\mathrm{u}}_{\mathrm{i}}\text{-}{\mathrm{i}}_{\mathrm{g}}}\right(\mathrm{j}\left({\mathrm{n}}_{\mathrm{structure}}+1\right){\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\left)\right|}{{\mathrm{I}}_{\mathrm{ref}}}}\le 0.3\%$$

(16)

where $I_{\mathit{ref}}$ is the amplitude of the reference grid current. The reward $r_{\mathit{circuit}\_3}$ is constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_3}=\left\{\begin{array}{c}{\mathrm{C}}_1,\hspace{1em}\mathrm{i}\mathrm{f}\mathrm{s}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{f}\mathrm{y}\left(16\right)\\ -{\mathrm{C}}_1,\hspace{1em}\hspace{1em}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$$

(17)

$C_1$ denotes the reward when the constraint (16) is satisfied and is a constant. In this paper, $C_1$ equals 10.

(5)

To avoid system instability caused by harmonic resonance in the high-frequency band and low-frequency band of the filter, its characteristic resonant frequency needs to be limited between 10${\omega }_0$ and 0.5${\omega }_{s\omega }$. With the same total capacitance, the first resonant frequency of the LTLCL filter is approximately the resonant frequency of the LCL filter. Therefore, the resonant frequency of the LCL filter can be used to approximate the first resonant frequency of the LTLCL filter. The first resonant frequency ${\omega }_r$ can be derived as:

$${\mathsf{\omega }}_{\mathrm{r}}\approx \sqrt{{\displaystyle \frac{{\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3}{{\mathrm{L}}_1({\mathrm{L}}_1+{\mathrm{L}}_1\left)\right({\sum }_{\mathrm{i}=1}^{{\mathrm{n}}_{\mathrm{structure}}}{\mathrm{C}}_{\mathrm{i}}+{\mathrm{C}}_{\mathrm{f}})}}}$$

(18)

Hence, $r_{\mathit{circuit}\_4}$ is constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_4}=\left\{\begin{array}{l}{\mathrm{C}}_2,\hspace{1em}\mathrm{i}\mathrm{f}10{\mathsf{\omega }}_0\le {\mathsf{\omega }}_{\mathrm{r}}\le 0.5{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\\ -{\mathrm{C}}_2,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$$

(19)

$C_2$ is also a constant larger than 0. In this paper, $C_2$ equals 10.

(6)

$r_{\mathit{circuit}\_1}$ also includes a limit on the total capacitance, i.e., the smaller the total capacity, the higher the $r_{\mathit{circuit}\_1}$. In addition to the capacitance, the objective is to minimize the total inductance. Therefore, with respect to the total inductance, a penalty term can be constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_5}=-\left(\sum _{\mathrm{i}=1}^{{\mathrm{n}}_{\mathrm{structure}}}{\mathrm{L}}_{\mathrm{fi}}+{\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3\right)$$

(20)

(7)

To limit the order of the filter, a penalty term related to the order is therefore introduced. Let the coefficient be $k_{\mathit{penalty}}$, which in this case equals 0.5. Then the reward $r_{\mathit{circuit}\_6}$ is constructed as follows:

$${\mathrm{r}}_{\mathrm{circuit}\_6}=-{\mathrm{k}}_{penalty}\times {\mathrm{n}}_{\mathrm{structure}}$$

(21)

In summary, the total score sc obtained from the circuit evaluation program is derived as:

$$\mathrm{sc}=\sum _{\mathrm{i}=1}^6{\mathsf{\omega }}_{\mathrm{i}}\times {\mathrm{r}}_{\mathrm{circuit}\_\mathrm{i}}$$

(22)

where ${\omega }_i$ are the weighting factors introduced to balance order of magnitude of $r_{\mathit{circuit}\_i}$.

4. Verification

The multi-objective optimization algorithm can be used to optimize the parameters of the grid-connected filter, as seen in some related studies. For instance, in the literature [25], they focused on optimizing the LCL filter using four metaheuristic algorithms, namely the Whale Optimization Algorithm, the Circle Search Algorithm, the Particle Swarm Optimization, and the Gray Wolf Optimization. Their approach aimed to minimize the total harmonic distortion (THD) and the error between the reference and real grid current, considering the LCL filter’s parameters, the damping coefficient of the capacitor current feedback active damping (CCF–AD) method, and the gains of the proportional resonant (PR) controller. However, the dimension of its decision vector varies with $n_{\mathit{structure}}$; when $n_{\mathit{structure}}$ changes, it must be re-optimized.

In our method, when the maximum value of $n_{\mathit{structure}}$ is determined, the dimension of state does not change as $n_{\mathit{structure}}$ changes, so only one exploration is needed. During the process, in addition to finding the highest-scoring filter, the highest-scoring circuit parameters are recorded for each of the four cases. This allows comparing not only the global optimal case but also the local optimal case.

The training program runs on a computer with AMD Ryzen 7 2700 CPU, Nvidia RTX2080 GPU, and 16 GB of RAM. The configurations of the parameters of the RL algorithm are as listed in

Table 1. Parameters of the RL algorithm.

Parameter	Value	Parameter	Value
Batch Size	64	Learning Rata	0.0003
Buffer Size	20,000	Discount Factor	0.99
Actor hidden layer size	[64,64]	Critic hidden layer size	[64,64]

.

In this validation, the maximum value of $n_{\mathit{structure}}$ was set to 5. The approximate block diagram of the simulation is as shown in Figure 4. Simulation setting is as shown in

Table 2. Simulation setting.

Parameter	Value
$\mathrm{Grid}\mathrm{voltage}{V}_g$	220 V
$\mathrm{Rated}\mathrm{power}{P}_{\mathit{rated}}$	5000 W
$\mathrm{DC}\text{-}\mathrm{link}\mathrm{voltage}{V}_{dc}$	400 V
$\mathrm{Fundamental}\mathrm{frequency}{\omega }_0$	100π rad/s
$\mathrm{Switching}\mathrm{frequency}{\omega }_{s\omega }$	20,000π rad/s
$\mathrm{Rated}\mathrm{reference}\mathrm{peak}\mathrm{current}{I}_{\mathit{ref}}$	11 A

. According to the simulation settings, the upper and lower limits of the inductor-capacitor can be obtained as shown in

Table 3. Upper and lower limits of component parameters.

Components (x)	${\mathit{X}}_{\mathit{min}}$	${\mathit{X}}_{\mathit{max}}$	$\mathbf{\Delta }$
$L_1 \left(\mathrm{mH}\right)$	0.76	2.20	0.015
${L_2,L}_3 \left(\mathrm{mH}\right)$	0.092	9.20	0.092
$C_k,C_f \left(\mathsf{\mu }\mathrm{F}\right)$	0.11	5.48	0.54
$L_{\mathit{fk}}$	${\displaystyle \frac{1}{k^2{\omega }_{s\omega }^2{C}_k}}$	${\displaystyle \frac{1}{k^2{\omega }_{s\omega }^2{C}_k}}$	-

. The lower and upper bounds for parameters are determined based on grid-connected requirements. For LTLCL filter components, these bounds are set considering electrical constraints. Initial values are randomly chosen within this range for RL training start.

4.1. Train Result

Figure 5 shows variation of reward (i.e., mean episode reward) and best score in training. Figure 5a shows that the reward converges after about 2 million steps. According to the designed constraints, the value of $n_{\mathit{structure}}$ eventually converges to the case of $n_{\mathit{structure}}$ = 4. According to Figure 5b, the best score also converges to a fixed value. The circuit structure and the waveform diagram of the grid current $i_g$ obtained from the simulation are shown in Figure 6.

To verify that the best case in the same evaluation system is $n_{\mathit{structure}}=4$, a separate search is conducted for the cases where ns $n_{\mathit{structure}}=2,3,5$, followed by the identification of the best-scoring circuits. The parameters of the grid-connected filter corresponding to the highest score and optimal parameters in other cases are shown in

Table 4. The filter parameters corresponding to the highest score.

	${\mathbf{r}}_{\mathbf{structure}}$	2	3	4	5
Parameter
$L_1$ (mH)		2.08	1.38	1.08	0.98
$L_2$ (mH)		0.46	0.28	0.27	0.20
$L_3$ (mH)		0.28	0.27	0.18	0.28
$C_f$ ($\mathsf{\mu }$F)		2.04	0.59	0.81	0.70
$C_1$ ($\mathsf{\mu }$F)		0.86	0.92	1.02	0.70
$L_{f1}$ (mH)		0.29	0.27	0.25	0.36
$C_2$ ($\mathsf{\mu }$F)		1.24	0.65	0.22	0.49
$L_{f2}$ (mH)		0.05	0.098	0.29	0.13
$C_3$ ($\mathsf{\mu }$F)		-	1.45	0.65	0.49
$L_{f3}$ (mH)		-	0.019	0.044	0.058
$C_4$ ($\mathsf{\mu }$F)		-	-	0.65	0.92
$L_{f4}$ (mH)		-	-	0.024	0.017
$C_5$ ($\mathsf{\mu }$F)		-	-	-	0.92
$L_{f5}$ (mH)		-	-	-	0.011
$L_{\mathit{total}}$(mH)		3.15	2.32	2.14	2.04
$C_{\mathit{total}}$ ($\mathsf{\mu }$F)		4.14	3.60	3.34	4.20

. To balance the quality of filtering and the cost of the circuit, the total capacity of inductors and capacitors should be set to a reasonable value. According to Table 4, The case of $n_{\mathit{structure}}=4$ requires a smaller total inductance and total capacitance, which complies with the design objectives. According to constraint (9), the upper limit of the total capacitance is 5.48 µF, and the designed filters meet this constraint.

Total harmonic distortion rate (THD) is an important indicator of grid-side current $i_g$ quality. Simulations are done for each of the four filters individually, recording the current $i_g$ and doing FFT analysis on them to calculate their THDs. By FFT analysis, it can be found that when $n_{\mathit{structure}}=4$, the THD is at minimum and the harmonic amplitudes are all less than 0.3% of the fundamental current amplitude. It can also be seen from Figure 7 that the higher the $n_{\mathit{structure}}$, the greater the attenuation of the current harmonics at the multiplicative switching frequency, but the trapped branch introduces positive resonance peaks, thus causing oscillations at some frequencies and thus increasing the THD. Since current harmonics are mainly concentrated at multiples of the switching frequencies, positive resonance points should be avoided near these frequencies as much as possible.

4.2. Solution Comparsion

To verify whether the filter parameters designed by RL are optimal, two multi-objective optimization algorithms, namely NSGA-II and SMS-EMOA, are selected for conducting parameter search in the case of $n_{\mathit{structure}}=4$. Subsequently, simulations are performed, and the results are compared. According to the Equations (6), (9), (11), (13), (16), and (18), the objective function and constraints are constructed as follows:

$${\mathrm{f}}_{{\mathrm{L}}_{\mathrm{total}}}=\sum _{\mathrm{i}=1}^4{\mathrm{L}}_{\mathrm{fi}}+{\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3$$

(23)

$${\mathrm{f}}_{{\mathrm{C}}_{\mathrm{total}}}=\sum _{\mathrm{i}=1}^4{\mathrm{C}}_{\mathrm{k}}+{\mathrm{C}}_{\mathrm{f}}$$

(24)

$${\mathrm{f}}_{\mathrm{i}}=20\lg \left|{\mathrm{G}}_{{\mathrm{u}}_{\mathrm{i}}\text{-}{\mathrm{i}}_{\mathrm{g}}}\right(\mathrm{I}•\mathrm{j}{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\left)\right|$$

(25)

$$\min \mathrm{F}=({\mathrm{f}}_{{\mathrm{L}}_{\mathrm{total}}},{\mathrm{f}}_{{\mathrm{C}}_{\mathrm{total}}},{\mathrm{f}}_1,{\mathrm{f}}_2,\dots ,{\mathrm{f}}_4)$$

(26)

$$\mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}{\sum }_{\mathrm{i}=1}^4{\mathrm{C}}_{\mathrm{i}}+{\mathrm{C}}_{\mathrm{f}}\le {\displaystyle \frac{5\%·{\mathrm{P}}_{\mathrm{rated}}}{{\mathrm{V}}_{\mathrm{g}}^2{\mathsf{\omega }}_0}}\\ {\mathrm{L}}_1+{\mathrm{L}}_2+{\mathrm{L}}_3\le {\displaystyle \frac{10\%·{\mathrm{V}}_{\mathrm{g}}^2}{{\mathrm{P}}_{\mathrm{rated}}{\mathsf{\omega }}_0}}\\ 20\%\le {\displaystyle \frac{2\mathsf{\pi }{\mathrm{V}}_{\mathrm{dc}}}{8{\mathrm{L}}_1{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}{\mathrm{I}}_{\mathrm{ref}}}}\le 60\%\\ {\displaystyle \frac{{\mathrm{A}}_{\max }\left|{\mathrm{G}}_{{\mathrm{u}}_{\mathrm{i}}\text{-}{\mathrm{i}}_{\mathrm{g}}}\right(\mathrm{j}\left(\mathrm{n}+1\right){\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\left)\right|}{{\mathrm{I}}_{\mathrm{ref}}}}\le 0.3\%\\ 10{\mathsf{\omega }}_0\le {\mathsf{\omega }}_{\mathrm{r}}\le 0.5{\mathsf{\omega }}_{\mathrm{s}\mathsf{\omega }}\end{array}\right.$$

(27)

Equation (23) serves for computing the total inductance of the filter, whereas Equation (24) is dedicated to calculate the total capacitance. Equation (25) primarily functions to assess the attenuation level of the filter concerning harmonics at specific frequencies. Additionally, Equation (26) represents a comprehensive objective function that combines multiple factors, including total inductance, total capacitance, and harmonic attenuation, across various frequencies. Finally, Equation (27) constitutes a set of constraints designed to confine the value range of filter parameters, thereby guaranteeing the practical feasibility of the designed filter.

A multi-objective optimization algorithm is employed to acquire a set of feasible solutions, followed by the utilization of the scoring Equation (21) to identify the parameters with the highest scores. Figure 8 shows the feasible solutions and the final decision vectors for both algorithms (NSGA-II and SMS EMOA).

In summary, the parameters of the filter designed by RL and the filter designed by the optimization algorithm are shown in

Table 5. Optimal parameters obtained by different methods.

Method	Proposed	NSGA-II	SMS-EMOA
$L_1$ (mH)	1.08	1.63	1.82
$L_2$ (mH)	0.27	0.85	1.00
$L_3$ (mH)	0.18	0.39	0.77
$C_f$$(\mathsf{\mu }$F)	0.81	0.32	2.41
$C_1$$(\mathsf{\mu }$F)	1.02	1.22	1.12
$L_{f1}$ (mH)	0.25	0.20	0.22
$C_2$$(\mathsf{\mu }$F)	0.22	1.72	0.43
$L_{f2}$ (mH)	0.29	0.037	0.15
$C_3$$(\mathsf{\mu }$F)	0.65	1.46	1.05
$L_{f3}$ (mH)	0.044	0.019	0.027
$C_4$$(\mathsf{\mu }$F)	0.65	0.68	0.46
$L_{f4}$ (mH)	0.024	0.023	0.035
$L_{\mathit{total}}$(mH)	2.14	3.20	4.00
$C_{\mathit{total}}$$(\mathsf{\mu }$F) $THD(\%)$	3.34 0.48	5.40 0.55	5.48 0.54

. The filters obtained by NSGA-II and SMS EMOA algorithms are simulated, and FFT analysis of the grid current $i_g$ is performed as shown in Figure 9. By comparing Figure 7c, Figure 9a, and Figure 9b, it can be seen that the filter designed by RL has a smaller THD value. Combined with Table 5, although the filters designed by the multi-objective optimization algorithms have slight good filtering, its total capacity of inductor–capacitor is higher, which will increase the design cost.

Our proposed method allows not only to optimize the structure, i.e., the filter topology is not fixed, but also to optimize the component parameters. The reinforcement learning approach is more suitable for designing high-order grid-connected filters than the complexity of multi-objective optimization algorithm construction.

5. Conclusions

This paper presents a reinforcement learning approach for designing high-order filters, which can adapt to different application scenarios by varying the structure and parameters of the LTLCL filter. This approach is the first to achieve simultaneous optimization of filter topology and component values, and it can explore the optimal circuit from a global perspective.

For the proposed method, a comparison is made between the search results in the scenario of varying structure and those in the case of fixed structure. According to the simulation setup, the optimal structure obtained by our method is the case where the number of branches is four. The optimization results achieved by the multi-objective optimization algorithm in the case $n_{\mathit{structure}}=4$ are compared. The filter designed by RL performs well in terms of filtering performance and cost reduction.

In the future, the proposed method can be used to design more complex filter circuits.