Optimal Current Balance Control of Three-Level Inverter under Grid Voltage Unbalance: An Adaptive Dynamic Programming Approach

Abstract

When the grid voltage is unbalanced, the positive and negative sequence components in the grid voltage cause grid current to be disordered. Under current balance control, proportional integral (PI) closed-loop control will increase the grid currents instantaneously, which affects the safety and reliability of the inverter operation, and PI parameters are difficult to select without the complete system mathematical model. This paper introduces an adaptive dynamic programming (ADP) approach to solve this problem. The best state feedback controller for the system is obtained by driving the ADP by the value iteration ($VI$) algorithm without the need for an accurate mathematical model. In the simulations, the ADP approach can improve the dynamic performance of the system, the current increase can be suppressed when the grid voltage is unbalanced, and the harmonic rate of output currents is reduced.

1. Introduction

1.1. Motivation and Incitement

In a power electronic converter, the inverter can feed back the power to the grid and obtain a sinusoidal current waveform in the grid. Multi-level inverters are widely used in high-voltage and high-power fields compared to conventional inverters [1,2]. Among them, the three-level inverter is widely used because of its simple topology and low cost.

The proportional integral (PI) (note: PI is different from the following policy iterations ($PI$)) control principle is simple and easy to implement, but no error tracking can be achieved for the amount of communication. Some scholars have proposed grid feed-forward PI control. Although the influence of the grid voltage on the incoming current is eliminated, the complexity of the system is increased [3]. The work in [4] proposed a closed-loop control method for inner loop proportional control and external loop network side inductor current quasi-ratio resonance control. Although this method can effectively suppress static error, it lacks real-time performance; especially the network voltage is heavily distorted. When the dynamic response is slow, it is easy to cause system instability. The work in [5] proposed a control strategy for quasi-ratio resonance plus grid feed-forward. Although the control effect was improved, the system complexity increased.

Operational control strategy under three-phase conditions in grid-connected inverter unbalanced power grids should also be considered. It is necessary to consider how to obtain the three-phase balanced current under the condition of an unbalanced three-phase power grid. The software phase-locked loop based on the decoupling of the double-synchronous coordinate system has higher steady-state accuracy, but it relies on phase feedback. Once the phase of the grid is abrupt, there is a large overshoot in the transition process [6]. The work in [7,8] installed a low-pass filter between traditional phase-locked loops. This method can cause defects such as phase angle shift and slow response. The work in [9] introduced droop control of grid voltage unbalance. The work in [10] introduced the robust $H_{\infty }$ control of the autonomous microgrid. In the existing references, all work was basically based on a complete mathematical model.

Therefore, this paper takes the three-level inverter as the research object and proposes an adaptive dynamic programming method to make the three-phase current balance in the case of three-phase unbalance quickly. This method is an optimal algorithm that attempts to maximize or minimize performance metrics under certain constraints. Dynamic programming is essentially the optimal decision process, but it is difficult to solve dynamic programming due to dimensionality disasters [11,12]. In order to solve this problem, ADP has made a lot of progress [13,14,15,16,17,18]. Currently, there are two algorithms for solving ADP, which are policy iterations ($PI$) and value iteration ($VI$) [19,20,21,22]. The main difference between the two methods is that $VI$ is usually implemented offline and $PI$ is usually implemented online [23].

The linear optimal output regulation problem (loorp) can improve system transient performance [24,25,26]. Under certain conditions, System trace output can be converted to adjustment output [27]. For the discrete control system, $x(k+1)=F\left(x\right(k),u(k\left)\right)$. In general, for finite time optimal control, its performance indicators are usually written as $J={\sum }_k^{\infty }(x^T\left(k\right)Qx\left(k\right)+u^T\left(k\right)Ru\left(k\right))$. However, for infinite time, this performance indicator function is no longer applicable, since the control track $u\left(k\right)$ depends on the change of the tracking track $\eta \left(k\right)$. By calculating J, it is possible to get infinite results. We can define the performance indicator function for time-varying optimal tracking control and write it as $J={\sum }_k^{\infty }(△x^T\left(k\right)Q△x\left(k\right)+△u^T\left(k\right)R△u\left(k\right))$ [28,29]. We need to solve the riccati equation to find the optimal controller to maximize or minimize the performance index function [30]. However, in that work, the proposed control strategy was based on the complete mathematical model of the system. This method is not suitable when the system is unknown or unmodeled. The work in [31] was an optimal control strategy without the complete system mathematical model. A data-driven approach through input and output data was designed.

1.2. Contribution and Paper Structure

In past works, current control loops were controlled by PI control. The PI controller: $v_d=(k_P+\frac{k_I}{s})(i_d^{*}-i_d)-wL{i}_q^{*}-e_d$. $k_P$ and $k_I$ are closely related to the system model. However, in many cases, we are unable to get all the models of the system. The P and I parameters are difficult to select. In this paper, the ADP method was used instead of PI control. The ADP closed-loop control achieved good performance without knowing the complete mathematical model of the system.

In this paper, the inverter output voltage was used as the control variable and the grid current as the state variable. The grid voltage is the external measurable variable of the system, and the optimal control is obtained by minimizing some secondary performance indicators. The closed loop system of the strategy has good transient performance. This paper mainly contributes as follows: The first part introduces the modeling of the inverter on the $dq$ coordinate system [32]. The second part reviews the linear quadratic regulator (LQR) problem, derives the system error equation, sets the performance index function, and rescues the riccati equation. We then show the tracking of the extended state equation of the problem through a linear system to express the information of the system and design the VI method to find the optimal control trajectory. The third part introduces the main results. The fourth part gives the relevant conclusions.

2. Three-Level Inverter

2.1. Mathematical Modeling in the Coordinate System

The main circuit diagram of the three-level inverter is shown in Figure 1.

Each phase consisted of four power switches. Take the Phase A switch as an example. When $T_{11}$ and $T_{12}$ are turned on at the same time, the output level is $\frac{V_{dc}}{2}$, denoted as: $S_a=1$. When $T_{12}$ and $T_{13}$ are turned on at the same time, the output level is zero, denoted as: $S_a=0$. When $T_{13}$ and $T_{14}$ are turned on at the same time, the output level is $-\frac{V_{dc}}{2}$, denoted as: $S_a=-1$. The voltage of the output phase voltage relative to the point O of the power supply can be represented by the DC side voltage $V_{dc}$ and the switching function $S_k$:

$$u_{ko}=\frac{V_{dc}}{2}{S}_k$$

(1)

We can get the output voltage of the inverter:

$$u_k=(S_k-\frac{1}{3}\sum _{k=a,b,c}{S}_k)V_{dc}$$

(2)

According to $Kirchhof{f}^{\prime }s$ law of voltage and current:

$$L\frac{d{i}_k}{dt}+r{i}_k+e_k=u_k$$

(3)

Since the inverter exhibited strong coupling, it was necessary to perform decoupling analysis. This paper uses 3/2 transform and $2s/2r$ transform. The transformation matrix is as follows:

$$\begin{array}{c}\hfill C_{abc/\alpha \beta }=\frac{2}{3}\left[\begin{array}{ccc}1& -\frac{1}{2}& -\frac{1}{2}\\ 0& -\frac{\sqrt{3}}{2}& -\frac{\sqrt{3}}{2}\end{array}\right]\end{array}$$

$$\begin{array}{c}\hfill T_{\alpha \beta /dq}=\left[\begin{array}{cc}cos\phi & sin\phi \\ -sin\phi & cos\phi \end{array}\right]\end{array}$$

where $\phi$ is the angle between the d- and $\alpha$-axis.

According to the transformation matrix, we can get the dynamic mathematical model under the $dq$ coordinate system.

$$\left\{\begin{array}{c}L\frac{d{i}_d}{dt}+r{i}_d+e_d=wL{i}_q+v_d\hfill \\ L\frac{d{i}_q}{dt}+r{i}_q+e_q=-wL{i}_d+v_q\hfill \end{array}\right.$$

(4)

where w is the angular frequency.

2.2. Mathematical Modeling under Grid Voltage Unbalance

The grid voltage e consists of positive sequence $e^{+}$, negative sequence $e^{-}$, and zero sequence component $e^0$ with the grid voltage unbalanced. For a neutral-free three-phase system, the influence of the zero sequence voltage is often ignored, i.e.,

$$\begin{array}{c}\hfill e=e^{+}+e^{-}\end{array}$$

In the coordinate system, the grid voltage vector can be expressed as:

$$\begin{array}{c}\hfill e=e^{jwt}{e}^{+}+e^{-jwt}{e}^{-}\end{array}$$

For the same reason:

$$\begin{array}{c}\hfill i=e^{jwt}{i}^{+}+e^{-jwt}{i}^{-}\end{array}$$

$$\begin{array}{c}\hfill v=e^{jwt}{v}^{+}+e^{-jwt}{v}^{-}\end{array}$$

(5)

By (4) and the transformation matrix, we can obtain:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{e}_{dq}^{+}=v_{dq}^{+}+L\frac{d{i}_{dq}^{+}}{dt}+r{i}_{dq}^{+}-jwL{i}_{dq}^{+}\hfill \\ e_{dq}^{-}=v_{dq}^{-}+L\frac{d{i}_{dq}^{-}}{dt}+r{i}_{dq}^{-}-jwL{i}_{dq}^{-}\hfill \end{array}\right.\end{array}$$

(6)

Positive and negative sequence quantities are decomposed as follows on the $dq$ coordinate axis:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{e}_{dq}^{+}=e_d^{+}+j{e}_q^{+}\hfill \\ e_{dq}^{-}=e_d^{-}+j{e}_q^{-}\hfill \end{array}\right.\end{array}$$

(7)

By (6) and (7), we have:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{e}_d^{+}=v_d^{+}+L\frac{d{i}_d^{+}}{dt}+r{i}_d^{+}+wL{i}_q^{+}\hfill \\ e_q^{+}=v_q^{+}+L\frac{d{i}_q^{+}}{dt}+r{i}_q^{+}-wL{i}_d^{+}\hfill \\ e_d^{-}=v_d^{-}+L\frac{d{i}_d^{-}}{dt}+r{i}_d^{-}+wL{i}_q^{-}\hfill \\ e_q^{-}=v_q^{-}+L\frac{d{i}_q^{-}}{dt}+r{i}_q^{-}-wL{i}_d^{-}\hfill \end{array}\right.\end{array}$$

(8)

By (8), we can obtain:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\dot{x}=Ax+Bu+D\sigma \hfill \\ y=Cx\hfill \\ \dot{x^{\prime }}=A^{\prime }{x}^{\prime }+B^{\prime }{u}^{\prime }+D^{\prime }{\sigma }^{\prime }\hfill \\ y^{\prime }=C^{\prime }{x}^{\prime }\hfill \end{array}\right.\end{array}$$

(9)

where $x=[i_d^{+};i_q^{+}]$, $x^{\prime }=[i_d^{-};i_q^{-}]$, $u=[u_d^{+};u_q^{+}]$, $u^{\prime }=[u_d^{-};u_q^{-}]$, $\sigma =[e_d^{+};e_q^{+}]$, ${\sigma }^{\prime }=[e_d^{-};e_q^{-}]$, $A=A^{\prime }=\left[\begin{array}{cc}-\frac{r}{L}& -w\\ w& -\frac{r}{L}\end{array}\right]$, $B=B^{\prime }=\left[\begin{array}{cc}-\frac{1}{L}& 0\\ 0& -\frac{1}{L}\end{array}\right]$, $D=D^{\prime }=\left[\begin{array}{cc}\frac{1}{L}& 0\\ 0& \frac{1}{L}\end{array}\right]$, and $C=C^{\prime }=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$.

Equation (9) is discretized in the following form:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}x(k+1)=G\left(T\right)x\left(k\right)+H\left(T\right)u\left(k\right)+M\left(T\right)\sigma \left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right.\end{array}$$

(10)

where T is the sampling period, $G\left(T\right)=TA+I$, $H\left(T\right)=TB$, and $M\left(T\right)=TD$.

3. Linear Optimal Output Adjustment Problem

3.1. Problem Description

We consider the following system:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}x(k+1)=Ax\left(k\right)+Bu\left(k\right)+D\sigma \left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right.\end{array}$$

(11)

where $x\left(k\right)\in {\mathbb{R}}^n$ is the state variable of this system. $u\left(k\right)\in {\mathbb{R}}^m$ is the control input. $\sigma \left(k\right)\in {\mathbb{R}}^m$ is the external disturbance of the measurable system, and $y\left(k\right)\in {\mathbb{R}}^n$ is the system output. $A\in {\mathbb{R}}^{n\times n}$, $B\in {\mathbb{R}}^{n\times m}$,$D\in {\mathbb{R}}^{n\times m}$, and $C\in {\mathbb{R}}^{n\times n}$ are certain constant matrices. We need to find an optimal control strategy u, so that the system (11) output tracking $\eta \left(k\right)\in {\mathbb{R}}^n$.

Assumption 1.

For controllable u, the system (11) is controllable. The inverter controllability proved in the Appendix A;

Assumption 2.

There is a one-to-one correspondence between $x\left(k\right)$ and $\eta \left(k\right)$ ($\eta \left(k\right)$ is the ideal state).

We write the ideal control system as follows:

$$\eta (k+1)=A\eta \left(k\right)+B{u}^{*}\left(k\right)+D\sigma \left(k\right)$$

(12)

where $\eta \left(k\right)$ is the ideal state, $u^{*}\left(k\right)$ is the ideal control, and $\sigma \left(k\right)$ is defined as above.

The ideal control input can be written as:

$$u^{*}\left(k\right)=B^{-1}(\eta (k+1)-A\eta \left(k\right)-D\sigma \left(k\right))$$

(13)

Then, from (11) and (12), we can get the error system:

$$e(k+1)=Ae\left(k\right)+Bv\left(k\right)$$

(14)

which minimizes the following performance index:

$$J_{e\left(k\right)}=\sum _k^{\infty }(e^T\left(k\right)Qe\left(k\right)+v^T\left(k\right)Rv\left(k\right))$$

(15)

where $Q=Q^T\ge 0$, $R=R^T\ge 0$, $e\left(k\right)=x\left(k\right)-\eta \left(k\right)$ is the output error of the system. $v\left(k\right)=u\left(k\right)-u^{*}\left(k\right)$ is the input error of the system.

According to the above mathematical transformation. The problem of System (11) changes to the optimal adjustment problem of System (14). The optimal controller is:

$$u\left(k\right)=u^{*}\left(k\right)+v\left(k\right)$$

(16)

3.2. Linear Quadratic Regulator

According to the linear optimal theory, the feedback controller can be written as:

$$v\left(k\right)=-Ke\left(k\right)$$

(17)

Since in the LQR problem, each value is under the action of a positive definite symmetric matrix, the performance index can be written as a quadratic form:

$$J_{e\left(k\right)}=e^T\left(k\right)Pe\left(k\right)$$

(18)

where $P^{*}={\left(P^{*}\right)}^T>0$ is the only solution for the discrete time algebra riccati equation$\left(ARE\right)$ below:

$$P=A^{T}PA+Q-A^{T}PB{(R+B^{T}PB)}^{-1}{B}^{T}PA$$

(19)

Then, in this setup, the optimal feedback control gain is:

$$K={(R+B^{T}PB)}^{-1}{B}^{T}PA$$

(20)

Then, by (14) and (17), we have:

$$\begin{array}{c}\hfill e\left(k\right)=(A-BK)e(k-1)={(A-BK)}^{k}e\left(0\right)\end{array}$$

(21)

For the error system, $k\to \infty$, $e\left(k\right)\to 0$.

We note that P in Equation (19) is non-linear, and the exact value is usually difficult to find. Below, we use the iterative method to find the approximate solution of P. The specific steps are as follows.

Theorem 1.

The VI Algorithm 1 is able to approximate the solution to (19) with assured convergence. Consider $P_j$ and $K_{j+1}$ defined in (22) and (23). Then, for all $j=0,1,\cdots$:

(1) 

$A-B{K}_j$ satisfies $|{\lambda }_i|<1$;

(2) 

$j\to \infty$, $K_j\to K^{*}$, and $P_j\to P^{*}$.

The proof is in Appendix C.

Algorithm 1 VI algorithm [23].

Let $K_0$ be any stabilizing feedback gain matrix ($|{\lambda }_i|<1$, and ${\lambda }_i$ are eigenvalues of the matrix $A-B{K}_0$. The proof is in Appendix B). Repeat the following steps for $j=0,1,\cdots$ and $\tau >0$ Solve Equation (19): $$\begin{array}{c}\hfill P_j=A^T{P}_{j}A+Q-K_j^T(R+B^T{P}_{j}B)K_j\end{array}$$ (22) Update control gain by: $$\begin{array}{c}\hfill K_{j+1}={(R+B^T{P}_{j}B)}^{-1}{B}^T{P}_{j}A\end{array}$$ (23) Until $|P_{j+1}-P_j|\le \tau$

However, this optimal controller is completely dependent on the model and requires a perfect understanding of the model. In many cases, we are unable to get all the models of the system. In the next section, we will show how to use the VI method to design the controller.

3.3. Solve ADP by the VI Method

In this section, we design the VI method to find the optimal controller without an accurate system model. We consider the error system (14). The dynamics can be written over the time range $[k-q,k]$ as the unfolded state and the output equation. The new equation is:

$$\begin{array}{c}\hfill e\left(k\right)=A^{q}e(k-q)+B_q\tilde{v}(k-1,k-q)\\ \hfill \tilde{e}(k-1,k-q)=C_{q}e(k-q)+T_q\tilde{v}(k-1,k-q)\end{array}$$

(24)

where:

$$B_q=\left[\begin{array}{ccc}B& AB& \cdots A^{q-1}B\end{array}\right]$$

$$C_q={\left[\begin{array}{ccc}{A}^{q-1}& A^{q-2}& \cdots I\end{array}\right]}^T$$

$$\tilde{e}(k-1,k-q)=\left[\begin{array}{c}e(k-1)\\ e(k-2)\\ ⋮\\ e(k-q)\end{array}\right],\tilde{v}(k-1,k-q)=\left[\begin{array}{c}v(k-1)\\ v(k-2)\\ ⋮\\ v(k-q)\end{array}\right]$$

and:

$$T_q=\left[\begin{array}{ccc}0& B& AB\cdots A^{q-2}B\\ 0& 0& AB\cdots A^{q-3}B\\ ⋮& ⋮& \ddots \\ 0& 0& 0\cdots 0\end{array}\right]$$

By (24), we can obtain:

$$e\left(k\right)=Mz\left(k\right)=\left[\begin{array}{cc}{M}_e& M_v\end{array}\right]z\left(k\right)$$

(25)

with $M=[A^q{C}_q^{+},B_q-A^q{C}_q^{+}{T}_q]$, $C_q^{+}$ is the generalized inverse matrix of $C_q$, and $z\left(k\right)=\left[\begin{array}{c}\tilde{e}(k-1,k-q)\\ \tilde{v}(k-1,k-q)\end{array}\right]$.

Let us assume $E=Q+A^{T}PA$, $F=A^{T}PB$, and $G=R+B^{T}PB$. Then, $K=G^{-1}{F}^T$ and $G^T=G$. By (19), we can obtain:

$$e^T\left(k\right)Pe\left(k\right)=e^T\left(k\right)(E-F{G}^{-1}{F}^T)e\left(k\right)$$

(26)

and:

$$\begin{array}{c}\hfill e^T\left(k\right)F{G}^{-1}{F}^{T}e\left(k\right)=-v^T\left(k\right)F^{T}e\left(k\right)\\ \hfill =-e^T\left(k\right)Fv\left(k\right)\\ \hfill =v^T\left(k\right)Gv\left(k\right)\\ \hfill =-v^T\left(k\right)F^{T}e\left(k\right)-e^T\left(k\right)Fv\left(k\right)-v^T\left(k\right)Gv\left(k\right)\end{array}$$

(27)

By (26) and (27), we have:

$$\begin{array}{c}\hfill e^T\left(k\right)Pe\left(k\right)={\left[\begin{array}{c}e\left(k\right)\\ v\left(k\right)\end{array}\right]}^T\left[\begin{array}{cc}E& F\\ F^T& G\end{array}\right]\left[\begin{array}{c}e\left(k\right)\\ v\left(k\right)\end{array}\right]\end{array}$$

By (25), we have:

$$\begin{array}{c}\hfill e^T\left(k\right)Pe\left(k\right)={\left[\begin{array}{c}z\left(k\right)\\ v\left(k\right)\end{array}\right]}^T\left[\begin{array}{cc}{E}^{\prime }& F^{\prime }\\ F^{\prime T}& G^{\prime }\end{array}\right]\left[\begin{array}{c}z\left(k\right)\\ v\left(k\right)\end{array}\right]\end{array}$$

where $E^{\prime }=M^{T}EM$, $F^{\prime }=M^{T}F$, and $G^{\prime }=G$.

By (15) and (18), we can obtain:

$$\begin{array}{c}\hfill e^T\left(k\right)Pe\left(k\right)=e^T\left(k\right)Qe\left(k\right)+v^T\left(k\right)Rv\left(k\right)+e^T(k+1)Pe(k+1)\end{array}$$

(28)

and:

$$\begin{array}{c}\hfill e^T(k+1)Pe(k+1)={\left[\begin{array}{c}z(k+1)\\ v(k+1)\end{array}\right]}^T\left[\begin{array}{cc}{E}^{\prime }& F^{\prime }\\ F^{\prime T}& G^{\prime }\end{array}\right]\left[\begin{array}{c}z(k+1)\\ v(k+1)\end{array}\right]\\ \hfill ={\left[\begin{array}{c}z(k+1)\\ -K^{\prime }z(k+1)\end{array}\right]}^T\left[\begin{array}{cc}{E}^{\prime }& F^{\prime }\\ F^{\prime T}& G^{\prime }\end{array}\right]\left[\begin{array}{c}z(k+1)\\ -K^{\prime }z(k+1)\end{array}\right]\end{array}$$

where $v(k+1)=-Ke\left(k\right)=-KMz\left(k\right)$. Then, $K^{\prime }=KM=G^{\prime }{F}^{\prime T}$.

Change Equation (19) to the following equation:

$$\begin{array}{c}\hfill {\left[\begin{array}{c}z\left(k\right)\\ v\left(k\right)\end{array}\right]}^T\left[\begin{array}{cc}{E}^{\prime }& F^{\prime }\\ F^{\prime T}& G^{\prime }\end{array}\right]\left[\begin{array}{c}z\left(k\right)\\ v\left(k\right)\end{array}\right]-{\left[\begin{array}{c}z(k+1)\\ -K^{\prime }z(k+1)\end{array}\right]}^T\left[\begin{array}{cc}{E}^{\prime }& F^{\prime }\\ F^{\prime T}& G^{\prime }\end{array}\right]\left[\begin{array}{c}z(k+1)\\ -K^{\prime }z(k+1)\end{array}\right]\\ \hfill =e^T\left(k\right)Qe\left(k\right)+v^T\left(k\right)Rv\left(k\right)\end{array}$$

From the previous, we know that p is only related to $M_e$, $M_v$, and $P_j$. ${lim}_{j\to \infty }{P}_j=P^{*}$. $P^{*}$ is only related to the system characteristics and has nothing to do with the input and output of the system. For different control signals $v\left(k\right)$, the $\mathcal{P}$ value is determined. When $j\to \infty$, $E_j^{\prime }\to E_{*}^{\prime }$, $F_j^{\prime }\to F_{*}^{\prime }$, $G_j^{\prime }\to G_{*}^{\prime }$, and $K_j^{\prime }\to K_{*}^{\prime }$.

For a symmetric matrix $P\in {\mathbb{R}}^{m\times m}$ and a column vector $v\in {\mathbb{R}}^n$, we assume $vecs\left(P\right)={[p_{11},2p_{12},\cdots ,2p_{m-1,m},p_{mm}]}^T\in {\mathbb{R}}^{\frac{1}{2}m(m+1)}$ and $vecv\left(v\right)={[v_1^2,v_1{v}_2,\cdots ,v_1{v}_n,v_2^2,v_2{v}_3,\cdots ,v_{n-1}{v}_n,v_n^2]}^T\in {\mathbb{R}}^{\frac{1}{2}n(n+1)}$. Given a positive integer $k_0$ and $q<k_0$ and a sufficiently large positive integer s and given an arbitrary control policy $v\left(k\right)$ on $[0,k_0+s]$, let:

$$\begin{array}{c}\hfill {\psi }^j\left(k\right)=e^T\left(k\right)Qe\left(k\right)+v^T\left(k\right)Rv\left(k\right)+vecv{\left(\alpha \left(k\right)\right)}^{T}vecs\left(\left[\begin{array}{cc}{E}_j^{\prime }& F_j^{\prime }\\ F_j^{\prime T}& G_j^{\prime }\end{array}\right]\right)\end{array}$$

and:

$$\Theta =\left[\begin{array}{c}vecv{\left(\beta \left(k_0\right)\right)}^T\\ vecv{\left(\beta (k_0+1)\right)}^T\\ \cdots \\ vecv{\left(\beta (k_0+s)\right)}^T\end{array}\right]$$

where $\alpha \left(k\right)=[z(k+1),-K^{\prime }z(k+1)]$ and $\beta \left(k\right)=\left[z\right(k),v(k\left)\right]$. For $j=0,1,2\cdots$, we define:

$${\Psi }^j={[{\psi }^j\left(k_0\right),{\psi }^j(k_0+1),\cdots ,{\psi }^j(k_0+s)]}^T$$

(29)

Then:

$$\begin{array}{c}\hfill \Theta vecs\left(\left[\begin{array}{cc}{E}_{j+1}^{\prime }& F_{j+1}^{\prime }\\ F_{j+1}^{\prime T}& G_{j+1}^{\prime }\end{array}\right]\right)={\Psi }^j\end{array}$$

(30)

Remark 1.

As $j=0$, matrices $P_0=0$ and $K_0=0$ indicate that ${\psi }^0\left(k\right)=e^T\left(k\right)Qe\left(k\right)+v^T\left(k\right)Rv\left(k\right)+e^T(k+1)Qe(k+1)$. Therefore, when implementing Algorithm 1, we do not need the initial values $K_0^{\prime }$ and ${\mathrm{Y}}_0$. Then, the following properties hold [33]:

(1) 

$j\to \infty$,${\mathrm{Y}}_j\to {\mathrm{Y}}^{*}$,

(2) 

$j\to \infty$,$K_j^{\prime }\to K_{*}^{\prime }$,

(3) 

$k\to \infty$,$e\left(k\right)\to 0$.

Hence, Algorithm 2 converges, and the closed-loop system is stable. The VI-based measurement feedback ADP flowchart is shown in Figure 2.

Algorithm 2ADP algorithm by the VI method.

1. Select an error $\tau >0$ and $j=0$. 2. Calculate $K_j^{\prime }$: $$\begin{array}{c}\hfill \Theta vecs\left({\mathrm{Y}}_{j+1}\right)={\Psi }^j\\ \hfill K_{j+1}^{\prime }={({\mathrm{Y}}_{j+1}^{22})}^{-1}{\mathrm{Y}}_{j+1}^{21}\end{array}$$ (31) where ${\mathrm{Y}}_{j+1}=\left[\begin{array}{cc}{E}_{j+1}^{\prime }& F_{j+1}^{\prime }\\ F_{j+1}^{\prime T}& G_{j+1}^{\prime }\end{array}\right]$ 3. Until $|{\mathrm{Y}}_{j+1}-{\mathrm{Y}}_j|<\tau$

Get the approximated optimal control policy:

$$v\left(k\right)=-K_{*}^{\prime }z\left(k\right)$$

(32)

Theorem 2.

The original system optimal controller is:

$$u\left(k\right)=v\left(k\right)+u^{*}\left(k\right)$$

(33)

where $u^{*}\left(k\right)=-{\overline{K^{\prime }}}_{*}\overline{z}\left(k\right)$ and $\overline{z}\left(k\right)=\left[\begin{array}{c}\tilde{\eta }(k-1,k-q)\\ \widetilde{u^{*}}(k-1,k-q)\\ \tilde{\sigma }(k-1,k-q)\end{array}\right]$. The proof is in Appendix D.

The optimal feedback controller is:

$$u\left(k\right)=-K_{*}^{\prime }z\left(k\right)-{\overline{K^{\prime }}}_{*}\overline{z}\left(k\right)$$

(34)

4. Simulations

In this paper, $r=0.1\Omega$, $L=1.5$ mH, $V_{dc}=$ 1000 V, $Ts={10}^{-5}$ s, and $Q=R=eye\left(3\right)$. Given $i_d^{*}=10$, $i_q^{*}=0$, and $q=1$, the inverter filter structure diagram and the comparison of the two control methods are shown in Figure 3. The inverter structure and $dq$ component separation are shown in Figure 3a.

PI control is shown in Figure 3b. The PI regulation decoupling equation is:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{v}_d=(k_P+\frac{k_I}{s})(i_d^{*}-i_d)-wL{i}_q^{*}-e_d\hfill \\ v_q=(k_P+\frac{k_I}{s})(i_q^{*}-i_q)+wL{i}_d^{*}-e_q\hfill \end{array}\right.\end{array}$$

(35)

where $K_p=0.02$ and $K_I=500$ [3]. ADP control is shown in Figure 3c. Calculate v and $u^{*}$ according to the current error and the command current, respectively. In Figure 3c, we can obtain ADP by Algorithm 1:

$$\begin{array}{c}\hfill v_{dq}\left(k\right)=-K_{*}^{\prime }\left[\begin{array}{c}{e}_{12}(k-1)\\ v_{12}(k-1)\end{array}\right]-{\overline{K^{\prime }}}_{*}\left[\begin{array}{c}{i}_{dq}^{*}(k-1)\\ u_{12}^{*}(k-1)\\ e_{dq}(k-1)\end{array}\right]\end{array}$$

(36)

where:

$$K_{*}^{\prime }=\left[\begin{array}{cccc}-2.2762& 0.0144& 0.0153& -0.0000\\ 0.0144& -2.2762& 0.0000& 0.0153\end{array}\right]$$

and:

$${\overline{K^{\prime }}}_{*}=\left[\begin{array}{cccccc}-2.2762& 0.0144& 0.0153& -0.0000& -0.0153& 0.0000\\ 0.0144& -2.2762& 0.0000& 0.0153& -0.0000& -0.0153\end{array}\right]$$

4.1. Grid Voltage Balance

Grid voltage is shown in Figure 4a. It can be seen that the grid voltage is three-phase symmetrical with an amplitude of 310 V.

The grid current of PI closed-loop control and ADP closed-loop control is shown in Figure 4b,c, respectively. The figure shows in the case of grid voltage balance that the grid current tracking command value when the system reaches stability. From Figure 4b, we can see that the grid current reaches the steady state value at t = 0.005 s. From Figure 4c, we can see that the grid current reaches the steady state value at t = 0.035 s. This shows that the grid current of ADP closed-loop approach had a faster response speed than the PI closed-loop approach.

4.2. Grid Voltage Unbalance

The grid voltage unbalance is shown in Figure 5a; at t = 0.3 s, the amplitude of the phase A voltage dropped by 80%.

The grid current of the PI closed-loop approach and the ADP closed-loop approach is shown in Figure 5b,c, respectively. As shown in Figure 5b, after the Phase A voltage dropped, the Phase A current rose sharply, and the highest amplitude reached 100 A. The Phase B and C currents also rose and finally stabilized at about 60 A. The current increased several times compared to the normal operation, which can damage the entire system. As shown in Figure 5c, after the Phase A voltage dropped, the phase A current saw almost no change, and the Phase B and C currents were reduced.

4.3. Grid Current Balance Control

The grid voltage unbalance had positive and negative sequence components. Therefore, the positive and negative sequence components should be controlled separately. To suppress the negative sequence components, the current command is:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{i}_d^{*+}=10\hfill \\ i_q^{*+}=0\hfill \\ i_d^{*-}=0\hfill \\ i_q^{*-}=0\hfill \end{array}\right.\end{array}$$

(37)

By (9) and (10), positive and negative sequence component control had the same mathematical model. Therefore, the positive and negative sequences had the same control loop. By (35), the positive and negative sequence PI adjustment decoupling equations are:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{v}_d^{+}=(k_P+\frac{k_I}{s})(i_d^{+\ast }-i_d^{+})-wL{i}_q^{+\ast }-e_d^{+}\hfill \\ v_q^{+}=(k_P+\frac{k_I}{s})(i_q^{\ast +}-i_q^{+})+wL{i}_d^{+\ast }-e_q^{+}\hfill \end{array}\right.\end{array}$$

(38)

and:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{v}_d^{-}=(k_P+\frac{k_I}{s})(i_d^{\ast -}-i_d^{-})-wL{i}_q^{-\ast }-e_d^{-}\hfill \\ v_q^{-}=(k_P+\frac{k_I}{s})(i_q^{\ast -}-i_q^{-})+wL{i}_d^{-\ast }-e_q^{-}\hfill \end{array}\right.\end{array}$$

(39)

Then, negative and positive sequences had the same control process. By (5) and (6), the output voltage of the inverter is:

$$\begin{array}{c}\hfill v_{dq}=e^{jwt}{v}_{dq}^{+}+e^{-jwt}{v}_{dq}^{-}\end{array}$$

(40)

In this paper, the positive and negative sequence components were separated by modulus detection. By (6), in the $\alpha \beta$ coordinate system, the grid voltage positive sequence vector modulo $|e^{+}|$, the angular frequency w rotated counterclockwise, the grid voltage negative sequence vector modulo $|e^{-}|$, and angular frequency w rotated clockwise are shown in Figure 6.

Both had the same angular velocity and the opposite direction of rotation. $\left|e\right|$ was the largest when two vectors were in the same direction, and $\left|e\right|$ was the smallest when in the opposite direction.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}max\left|e\right|=|e^{+}|+|e^{-}|\hfill \\ min\left|e\right|=|e^{+}|-|e^{-}|\hfill \end{array}\right.\end{array}$$

(41)

The modulus of the positive and negative sequence components is:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}|e^{+}|=\frac{1}{2}\left(max\right|e|+min\left|e\right|)\hfill \\ |e^{-}|=\frac{1}{2}\left(max\right|e|-min\left|e\right|)\hfill \end{array}\right.\end{array}$$

(42)

We can get the positive and negative order $dq$ components respectively:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{e}_d^{+}=\left|e^{+}\right|cos\phi \hfill \\ e_q^{+}=\left|e^{+}\right|sin\phi \hfill \end{array}\right.\end{array}$$

and:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{e}_d^{-}=\left|e^{-}\right|cos\phi \hfill \\ e_q^{-}=\left|e^{-}\right|sin\phi \hfill \end{array}\right.\end{array}$$

The grid voltage unbalance is shown in Figure 7a; at t = 0.3 s, the amplitude of the Phase A voltage dropped by 80%.

The grid voltage $dq$ axis positive and negative sequence components are shown in Figure 7b.

The grid current of PI closed-loop control and ADP closed-loop control is shown in Figure 7c,d, respectively. As shown in Figure 7c, after the Phase A voltage dropped, the grid currents increased instantaneously, and the maximum was 40 A. After a brief rise, the current gradually approached stability, and when t = 0.345 s, the grid currents were stable. As shown in Figure 7d, after the Phase A voltage dropped, the grid current had almost no fluctuations in this process, keeping the command value. From this, we can see that the balanced current strategy using ADP closed-loop control not only could suppress the sharp rise of current, but also made the current respond quickly to the steady state value. The current overshoot of two control methods is shown in

Table 1. Overshoot of grid currents.

Control Method	Phase A Current	Phase B Current	Phase C Current
PI closed loop control	<300%	300%	−300%
ADP closed loop control	No overshoot	No overshoot	No overshoot

. Under PI closed-loop control, $Overshoot\left(A\right)<\frac{40-10}{10}=300\%$, $Overshoot\left(B\right)=\frac{40-10}{10}=300\%,$ and $Overshoot\left(C\right)<\frac{-40+10}{10}=-300\%$. Under the ADP closed-loop control, the grid currents were not overshot.

4.4. Grid Current Balance Control of Command Power

By [34], the command current calculation formula is:

$$i_{\alpha \beta }^{*}=\frac{2}{3}\frac{1}{e^{+}-e^{-}}[(p^{*}-j{q}^{*})e^{+}{e}^{jwt}-(p^{*}+j{q}^{*})e^{-}{e}^{-jwt}]$$

(43)

In this paper, we assumed $p^{*}=6000W$ and $q^{*}=2000W$. The command current $\alpha$ and $\beta$ components are shown in Figure 8a. The grid currents of PI closed-loop control and ADP closed-loop control are shown in Figure 8b,c, respectively. We can see that both control methods stabilized the current. Two control methods’ Phase A current comparison is shown in Figure 8d. We can see that the current of the ADP closed-loop control method was smoother at steady state. The $THD$ of grid currents are shown in

Table 2. THD of grid currents.

Control Method	Phase A Current	Phase B Current	Phase C Current
PI closed-loop control	2.04%	1.87%	1.97%
ADP closed-loop control	1.30%	1.29%	1.29%

. We can see that the three-phase grid currents $TH{D}_{PI}$ were $2.04\%$, $1.87\%$, and $1.97\%$, and $TH{D}_{ADP}$ were $1.30\%$, $1.29\%$, $1.29\%$. This shows that the ADP closed-loop control method had better steady state performance.

4.5. Discussions

The main purpose of this paper is to introduce the application of adaptive dynamic programming in grid-connected inverters. The simulation experiment was performed in the MATLAB/Simulink environment. Although the verification was for the Phase A voltage drop of $80\%$, this can be verified for other situations.

5. Conclusions

In this paper, an ADP algorithm was proposed instead of the original PI algorithm. The optimal tracking problem was solved without the complete system mathematical model. In the control strategy proposed in this paper, the grid currents can track the command currents. Compared with PI closed-loop control, ADP closed-loop control reduces current oscillation and improved response speed.

In this paper, we only considered current as the indicator. However, in the actual microgrid system, there are many indicators that affect system performance. In the hierarchical droop-based control strategies [35,36], we can set different performance indicators under different requirements, such as $▵P$, $▵Q$ ($▵P=P-P^{*}$, and P and $P^{*}$ represent output active power and command active power, respectively. $▵Q=Q-Q^{*}$, and Q and $Q^{*}$ represent output reactive power and command reactive power, respectively). In island mode, we can set $▵w$ and $▵U$($▵w=w-w^{*}$, and w and $w^{*}$ represent the output angular frequency and the given angular frequency, respectively. $▵U=U-U^{*}$, and U and $U^{*}$ represent the output voltage and the given voltage, respectively) as performance indicators. Then, we can set the performance indicator function according to the selected indicators and find the optimal controller by the ADP approach. This reduces system oscillations and improves performance.