Improved Performance for PMSM Sensorless Control Based on the LADRC Controller, ESO-Type Observer, DO-Type Observer, and RL-TD3 Agent

Abstract

Starting from the fact that in sensorless control systems of the Permanent Magnet Synchronous Motor (PMSM), the load torque can have short and significant variations, this paper presents the sensorless control of a PMSM based on a Linear Adaptive Disturbance Rejection Controller (LADRC) type controller. Essentially, the successful operation of the LADRC controller to achieve PMSM rotor speed control performance depends on a good estimation of the disturbances acting on the system. Traditionally, an Extended State Observer (ESO) is used to make such an estimate. In this paper, it is proposed to use a Disturbance Observer (DO) to estimate the external disturbances, and after their rejection, the LADRC controller ensures an equivalent global behavior of the control system with an ideal double integrator, thus increasing ease in achieving the desired control performance. Control structures and Matlab/Simulink implementation of the PMSM sensorless control system based on the LADRC controller with an ESO-/DO-type observer are presented, as is its use in tandem with a Reinforcement Learning Twin-Delayed Deep Deterministic Policy Gradient (RL-TD3) specially trained agent that provides correction signals for more accurate estimation of external disturbances and hence improved control performance. To optimize the gain value of the DO-type observer, a computational intelligence algorithm such as the Ant Colony Algorithm (ACO) is used. Qualitatively superior performance is achieved by using LADRC with the RL-TD3 agent control structure in terms of parametric robustness, response time, and steady-state error. In addition, by calculating the fractal dimension (DF) of the controlled signal and the PMSM rotor speed, it is found that the higher the DF, the better the performance of the control system. The validation of the superiority of the proposed control structures is carried out by means of numerical simulations in the Matlab/Simulink environment.

1. Introduction

Typically, the overall control strategy of a PMSM is of the FOC type [1,2,3], which includes two cascaded control loops, the inner loop providing current control and the outer loop providing PMSM rotor speed control. Due to the multitude of applications and domains, among which we can mention robotics, computer peripherals, the aerospace industry, and electric drives [4,5], in which PMSM is used, the controllers implemented in the global FOC strategy have also evolved from simple to complex, i.e., from classical PI-type controllers [6,7] to adaptive [8,9], predictive [10], or robust [11] type controllers [12,13].

Also, by using state estimators and, in particular, by estimating the PMSM rotor speed, sensorless control systems are obtained that have the advantage of providing high system reliability by eliminating additional hardware structures such as speed sensors. Common state estimators include the Luenberger Observer [14,15], Model Reference Adaptive System (MRAS) [16,17], and Sliding Mode Observer (SMO) [18,19], for the case where the deterministic approach is chosen, and for the case where the stochastic approach is chosen, estimators of the Kalman filter type are used [20,21].

In the case of PMSM rotor speed control, the load torque can be considered a disturbance acting on the system, and in the case of using a suitable ESO-type observer [22,23,24,25], after a good estimation of the disturbance, the use of a LADRC-type controller [26,27,28,29] ensures the operation of the whole control system as equivalent to an ideal double integrator, which means an easy adjustment to obtain the desired performance of the control system. In this respect, ESO-type observer synthesis is essential. In terms of estimating the disturbances acting on the system, this paper proposes the use of a DO-type observer [30,31,32,33]. It is also proposed to use an RL-TD3 agent [11,34,35,36], which, after the training period, will provide correction signals to the ESO-/DO-type observer to estimate the external disturbances as accurately as possible. In this way, the LADRC controller used together with the proposed control structure will ensure superior performance of the PMSM control system. Typically, PI-type controllers provide good performance in a linearized area, but in the case of large torque variations, the response times increase and steady-state error occurs. The control structure proposed in this paper provides good control performance in the case of load torque variation as well as in the case of parametric variation.

The operating equations and structures of the Matlab/Simulink implementation of the main blocks of the proposed sensorless control system based on LADRC, an ESO-/DO-type observer, and a properly trained RL-TD3 agent are presented.

The main contributions to this article are as follows:

• Synthesis of a sensorless PMSM sensorless control system using a LADRC controller based on an ESO-type observer.
• Synthesis of a sensorless PMSM sensorless control system using a LADRC controller based on a DO-type observer.
• Optimization of the gain for DO-type observers using a computational intelligence ACO algorithm.
• Utilizing the sensorless performance of the PMSM control system by using a LADRC controller based on an ESO-/DO-type observer in combination with an RL-TD3 agent that provides correction signals to the ESO-/DO-type observer after the training period to achieve the most accurate estimation of disturbances and superior control performance.
• Comparative presentation of the performance of the proposed sensorless control systems by comparing quality indicators such as response time, error, speed ripple, and DF [37,38] of the PMSM rotor speed signal.

The rest of the article is organized as follows: Section 2 presents the proposed sensorless control structure of PMSM. The LADRC control with an ESO-type observer of PMSM is presented in Section 3, while the DO-type observer is presented in Section 4. The Matlab/Simulink implementation of the improved sensorless control of PMSM based on LADRC with an ESO-type observer/DO-type observer and RL-TD3 agent is presented in Section 5, and the numerical simulation is presented in Section 6. Section 7 presents some conclusions and suggestions for future approaches.

2. Proposed Structures for Sensorless Control of PMSM

The commonly used overall PMSM control strategy is based on a FOC structure with cascaded PI controllers performing current control in the inner loop and PMSM rotor speed control in the outer loop. Figure 1 shows the block diagram of the FOC-type control structure for the PMSM control system.

The notations used in relation (1) are the usual ones, and they describe the operation of the PMSM in the d-q rotating reference frame [2,3,25]. Therefore, the following notations are used for the variables: the stator voltages and currents, u_d, u_q, and i_d, i_q; the stator inductances and resistances, L_d, L_q, and R_d, R_q; and the stator resistance, R_s. In the usual case, L_d = L_q, and R_d = R_q = R_s. Other parameters used in describing the PMSM operating equations are: the flux linkage (λ₀), the number of pole pairs (n_p), the rotor moment inertia combined with load moment inertia (J), the load torque (T_L), the viscous friction coefficient (B), the PMSM rotor electrical angle (θ_e), and the PMSM rotor speed (ω).

$$\{\begin{array}{l}\frac{d{i}_d}{dt}=-\frac{R_s}{L_d}{i}_d+\frac{L_q}{L_d}{n}_p\omega i_q+\frac{1}{L_d}{u}_d\\ \frac{d{i}_q}{dt}=-\frac{R_s}{L_q}{i}_q-\frac{L_d}{L_q}{n}_p\omega i_d-\frac{{\lambda }_0}{L_q}{n}_p\omega +\frac{1}{L_q}{u}_q\\ \frac{d\omega }{dt}=\frac{3}{2}\frac{n_p}{J}\left({\lambda }_0{i}_q+\left(L_d-L_q\right) i_d{i}_q\right)-\frac{1}{J}{T}_L-\frac{B}{J}\omega \\ \frac{d{\theta }_e}{dt}=n_p\omega \end{array}$$

(1)

The system shown in (1) can be rewritten in the form given in system (2) [2,25]:

$$\left[\begin{array}{c}{\dot{i}}_d\\ {\dot{i}}_q\\ \dot{\omega }\end{array}\right]=\left[\begin{array}{c}-\frac{R_s}{L_q}{i}_d+n_p\omega i_q\\ -\frac{R_s}{L_q}{i}_q-n_p\omega i_d-\frac{{\lambda }_0}{L_q}{n}_p\omega \\ \frac{3}{2}\frac{n_p{\lambda }_0}{J}{i}_q-\frac{B}{J}\omega \end{array}\right]+\left[\begin{array}{cc}\frac{1}{L_q}& 0\\ 0& \frac{1}{L_q}\\ 0& 0\end{array}\right]\left[\begin{array}{c}{u}_d\\ u_q\end{array}\right]+\left[\begin{array}{c}0\\ 0\\ -\frac{T_L}{J}\end{array}\right]$$

(2)

In the following sections, a LADRC controller with an ESO-type observer control structure is proposed. The replacement of the ESO-type observer with the DO-type observer is also proposed. Since the LADRC control is based on the estimation of the disturbance acting on the system with the help of an ESO-type observer, and after eliminating this term by the LADRC control law and achieving a double integrator behavior, it is proposed to use an RL-TD3 agent that, after the training period, will provide correction signals for a better estimation of the disturbance acting on the system.

Thus, Figure 2 shows the diagram of the proposed sensorless control system for PMSM based on the LADRC controller with an ESO-type observer and RL-TD3 agent. Similarly, Figure 3 shows the diagram for the proposed PMSM sensorless control system based on the LADRC controller with a DO-type observer and RL-TD3 agent.

3. LADRC-Type Control for PMSM

LADRC controllers can be used to control a PMSM where the load torque T_L (considered an external disturbance) may have significant amplitude variations. For this purpose, a state observer of the ESO type is used to estimate both the speed of the PMSM rotor and the external disturbance. In this way, the command provided by the LADRC controller will cancel the effect of external disturbances and give the control system behavior equivalent to an integrator-type system, thus achieving superior PMSM control performance. It is concluded that the main problem in this section is the synthesis of an ESO-type observer that provides the most accurate PMSM rotor speed and external disturbance.

In the general case, we can start from a differential equation of order n given by a relation of the form (3) [25,30].

$$y^{(n)}(t)=f\left(y(t),\dot{y}(t),\dots ,y^{(n-1)}(t),d(t),t\right)+bu(t)$$

(3)

Thus, Equation (3) describes a monovariable nonlinear system affected by perturbations. Let l be the derivative of the system output y by y^(l), the system input is u, and the disturbance acting on the system is denoted by d.

After using the notations: $x_1=y$, $x_2=\dot{y},\dots ,x_n=y^{(n-1)}$, we can rewrite the system given in relation (3) as the following system:

$$\{\begin{array}{c}{\dot{x}}_i=x_{i+1},i=1,\dots ,n-1\\ {\dot{x}}_n=f\left(x_1,x_2,\dots ,x_n,d,t\right)+bu\end{array}$$

(4)

An additional state is added in the form of the system (5).

$$\{\begin{array}{c}{x}_{n+1}=f\left(x_1,x_2,\dots ,x_n,d,t\right)\\ {\dot{x}}_{n+1}=h(t)\end{array}$$

(5)

The form of the function h(t) can be obtained from system (5) as follows:

$$h(t)=\dot{f}\left(x_1,x_2,\dots ,x_n,d,t\right)$$

(6)

The disturbances acting on the system are included in the variable f.

Starting from these elements, the synthesis of an ESO-type observer can be written as follows [25,30]:

$$\{\begin{array}{c}{\dot{\widehat{x}}}_i={\widehat{x}}_{i+1}+{\beta }_i(y-{\widehat{x}}_1),i=1,\dots ,n\\ {\dot{\widehat{x}}}_{n+1}={\beta }_{n+1}(y-{\widehat{x}}_1)\end{array}$$

(7)

The ESO-type observer system described by relation (7) will provide an estimate of the initial system states plus an estimate of the f-term incorporating the perturbations.

Adapting the above relations to a second-order system yields relation (8), where the system input is u, the system output is y, the external disturbance is w, and b is considered a tuning parameter in the LADRC controller structure [26,29].

$$\ddot{y}=f\left(\dot{y},y,w,t\right)+bu$$

(8)

In relation (8), the notations are as follows: $x_1=y$, $x_2=\dot{y}$, and $x_3=f$. f denotes an extended state of the system that includes disturbances acting on the system and parametric uncertainties.

An ESO-type observer can be described by the following system:

$$\{\begin{array}{l}e=z_1-y\\ {\dot{z}}_1=z_2+L_{1}e\\ {\dot{z}}_2=z_3+L_{2}fal(e,{\alpha }_1,{\delta }_1)+bu\\ {\dot{z}}_3=-L_{3}fal(e,{\alpha }_2,{\delta }_2)\end{array}$$

(9)

In system (9), the difference between the actual and estimated output of the system is denoted by e, and z_i denotes the estimates of the states $x_i,i=1,2,3$. In addition, L_i denotes the gain factors (amplifications) of the observer.

Define the nonlinear function fal(e,α,δ), which is defined by the relation (10), where $\alpha \in (0,1)$ and the nonlinearity of the function are characterized by a small number δ.

$$fal(e,\alpha ,\delta )=\{\begin{array}{l}\frac{e}{{\delta }^{\alpha -1}}, \left|e\right|\le \delta \\ {\left|e\right|}^{\alpha }sign(e), \left|e\right|>\delta \end{array}$$

(10)

From this, the LADRC controller provides the control input u given by the following equations:

$$\{\begin{array}{l}{e}_1=r-z_1\\ e_2=\dot{r}-z_2\\ u_0=K_{p}fal(e,{\alpha }_1,{\delta }_1)+K_{d}fal(e,{\alpha }_2,{\delta }_2)\\ u=u_0-\frac{z_3}{b}\end{array}$$

(11)

where K_p and K_d represent the tuning parameters of the LADRC controller, and the reference is denoted by r (in the case of PMSM equations of motion $\dot{r}={\omega }_{ref}$).

By linearizing Equations (10) and (11), the linear forms given by relations (12) and (13) are obtained, and a graphical description of the LADRC control structure of a PMSM is given in Figure 4. The ESO-type observer (third order) block diagram for rotor speed PMSM estimation is shown in Figure 5.

$$\{\begin{array}{l}e=y-z_1\\ {\dot{z}}_1=z_2+L_{1}e\\ {\dot{z}}_2=z_3+L_{2}e+bu\\ {\dot{z}}_3=L_{3}e\end{array}$$

(12)

$$\{\begin{array}{l}{e}_1=r-z_1\\ e_2=\dot{r}-z_2\\ u_0=K_p{e}_1+K_d{e}_2\\ u=u_0-\frac{z_3}{b}\end{array}$$

(13)

Assuming in relation (8) $h=\dot{f}$, where f has a bounded increase, the equivalent linear system can be obtained in the form given in (14).

$$\{\begin{array}{l}\dot{x}=Ax+Bu+Eh\\ y=Cx\end{array}$$

(14)

where: $A=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right]$, $B=\left[\begin{array}{c}0\\ b\\ 0\end{array}\right]$, $C=\left[\begin{array}{ccc}1& 0& 0\end{array}\right]$, and $E=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$.

In the case of a second-order system, an ESO-type observer of order 3 is obtained, which can be described by the system (15).

$$\{\begin{array}{l}\dot{z}=Az+Bu+L(x_1-z_1)\\ \widehat{y}=Cz\end{array}$$

(15)

In system (15), the vector of gain factors is given by:

$$L=\left[\begin{array}{c}{L}_1\\ L_2\\ L_3\end{array}\right]$$

(16)

The frequency band of the ESO-type observer is denoted by ω₀. The poles of the ESO-type observer are assigned to −ω₀, and thus from relation (17), it can be obtained the vector L from relation (16) in the form of relation (18).

$$\lambda (s)=s^3+L_1{s}^2+L_{2}s+L_3={\left(s+{\omega }_0\right)}^3$$

(17)

$$L=\left[\begin{array}{c}{L}_1\\ L_2\\ L_3\end{array}\right]=\left[\begin{array}{c}3{\omega }_0\\ 3{\omega }_0^2\\ {\omega }_0^3\end{array}\right]$$

(18)

In relation (19), the u command represents the output of the LADRC controller.

The signal u₀ is a control signal provided by the ideal dynamic system of types (20) and (21), in which an exact rejection of disturbances z₃ = f occurs.

$$u=\frac{u_0-z_3}{b}$$

(19)

$$\ddot{y}=f-z_3+u_0$$

(20)

$$\ddot{y}=u_0$$

(21)

Thus, the behavior of the PMSM control system in which the controller is LADRC-type and the external disturbance-type signal is estimated using an ESO-type observer is equivalent to an ideal system (20) and (21). Note that Equation (21) is of ideal double integrator, and the command signal u₀ can be chosen in the following form:

$$u_0=K_p(r-z_1)+K_d(\dot{r}-z_2)$$

(22)

Note that the u₀ command signal is equivalent to the signal provided by a PD (proportional derivative) type controller.

The frequency band of the LADRC controller is denoted by ω_c. The poles of this controller are assigned to −ω_c in relation (23) [30].

$$\lambda (s)=s^2+K_{d}s+K_p={\left(s+{\omega }_c\right)}^2$$

(23)

$$K_p={\omega }_c^2$$

(24)

$$K_d=2{\omega }_c$$

(25)

At the end of this section a simplified version of the PMSM sensorless control system is presented using an ESO-type second-order observer and a first-order LADRC controller.

From the description of the PMSM equations in Section 2, it can be seen that they can be reduced to a differential equation of first order, as described by the relation (26).

$$\frac{d\omega }{dt}=\frac{1}{J}\left(\frac{3}{2}{n}_p{\psi }_f{i}_q-T_L-B\omega \right)$$

(26)

In relation (26), the generalized disturbance f can be written in the form given in relation (27).

$$f=-\frac{B}{J}\omega -\frac{T_L}{J}$$

(27)

By replacing relation (27) with relation (26), the following expression is obtained:

$$\frac{d\omega }{dt}=f+b{i}_q$$

(28)

It is noted $x_1=\omega$, $x_2=f$, and $\dot{f}=h$. Based on this, the PMSM system can be described as a second-order system:

$$\{\begin{array}{l}\dot{x}=Ax+Bu+Eh\\ y=Cx\end{array}$$

(29)

where: $A=\left[\begin{array}{cc}0& 1\\ 0& 0\end{array}\right]$, $B=\left[\begin{array}{c}b\\ 0\end{array}\right]$, $C=\left[\begin{array}{cc}1& 0\end{array}\right]$, and $E=\left[\begin{array}{c}0\\ 1\end{array}\right]$.

One can synthesize an ESO-type observer of second order as in Equations (30) and (31), where the PMSM rotor speed estimate is given by z₁ and the generalized disturbance estimate f is given by z₂.

$$\{\begin{array}{l}\dot{z}=Az+Bu+L(x_1-z_1)\\ y=Cz\end{array}$$

(30)

$$L=\left[\begin{array}{c}2{\omega }_0\\ {\omega }_0^2\end{array}\right]$$

(31)

In relation (31), the vector of gain factors is obtained by assigning the poles to −ω₀ by identifying the coefficients of a second-order polynomial as follows:

$$\lambda (s)=s^2+L_{1}s+L_2={\left(s+{\omega }_0\right)}^2$$

(32)

Based on the above, the control law is given in the following form:

$$u=i_q=\frac{u_0-z_2}{b}$$

(33)

The equivalent dynamical system in the form of an integrator is given by the relation (34).

$$\frac{d\widehat{\omega }}{dt}=u_0$$

(34)

It can be seen that the control law u₀ is given by relation (35) in the form of a proportional controller.

$$u_0=K_p\left(r-\widehat{\omega }\right)$$

(35)

In relation (35), r is the PMSM rotor speed reference, K_p = ω_c², and a graphical description of the LADRC control structure of a PMSM is shown in Figure 6. The ESO-type observer (2nd order) block diagram for rotor speed PMSM estimation is shown in Figure 7.

4. DO-Type Observer

Let us consider the nonlinear system in the relation (36):

$$\{\begin{array}{l}\dot{x}=f(x)+g_1(x)u+g_2(x)d\\ y=h(x)\end{array}$$

(36)

where $x\in {\Re }^n$ is the state, $u\in {\Re }^m$ is the control input, $y\in {\Re }^s$ is the output vector, and disturbance is $d\in {\Re }^q$.

Given that f(x), g₁(x), g₂(x), and h(x) are smooth functions in x, the challenge is to implement an observer to estimate the disturbance in the system (36). According to [30], the observer structure is defined in the relation (26).

$$\{\begin{array}{l}\dot{p}=l(x)g_2(x)p-l(x)\left[g_2(x)q(x)+f(x)+g_1(x)u\right]\\ \widehat{d}=p+q(x)\end{array}$$

(37)

where $p\in {\Re }^q$ is the state of the nonlinear observer, and g(x) is a function to be designed.

l(x) denotes the observer’s gain when expressed by the relation (38).

$$l(x)=\frac{\partial q(x)}{\partial x}$$

(38)

Given $e_d=d-\widehat{d}$, it is shown in [30] that the observer described in relation (37) reconstructs the asymptotic perturbation d under the condition that the gain l(x) is chosen such that Equation (39) is asymptotically stable.

$${\dot{e}}_d=-l(x)g_2(x)e_d$$

(39)

Starting from the PMSM equations described in Section 2, they can be rewritten in the d-q reference frame as in relations (40) and (41). Note that the notations are the usual ones, but when writing relation (40), the terms containing the external perturbation are highlighted.

$$\{\begin{array}{l}{\dot{x}}_1=x_2\\ {\dot{x}}_2=k_1{x}_3-k_2{x}_2-k_3\\ {\dot{x}}_3=-g_1{x}_3-g_2{x}_2{x}_4-g_3{x}_4+g_4{u}_q+f_1\\ {\dot{x}}_4=-g_1{x}_4+g_2{x}_2{x}_3+g_4{u}_d+f_2\end{array}$$

(40)

where:

$$\{\begin{array}{l}{x}_1={\theta }_e; x_2={\omega }_e; x_3=i_q; x_4=i_d\\ g_1=\frac{R_s}{L}; g_2=n_p; g_3=\frac{n_p{\lambda }_0}{L}; g_4=\frac{1}{L}\\ k_1=\frac{3n_p{\lambda }_0}{2J}; k_2=\frac{B}{J}; k_3=\frac{T_L}{J}\end{array}$$

(41)

In relation (41), parametric variations can be considered, resulting in relations (42) and (43).

$${\dot{x}}_2=k_1{x}_3-k_2{x}_2-k_3=\left(k_{1n}+\Delta k_1\right)x_3-\left(k_{2n}+\Delta k_2\right)x_2-\left(k_{3n}+\Delta k_3\right)=k_{1n}{x}_3-k_{2n}{x}_2+d$$

(42)

$$d=\Delta k_1{x}_3-\Delta k_2{x}_2-k_3$$

(43)

where k_1n, k_2n, and k_3n represent the nominal parameters, and the parametric variations are denoted by Δk₁, Δk₂, and Δk₃.

The term that includes disturbances caused by parametric variations is denoted by d.

Assuming that these parametric variations have bounded values, an observer of the type given in relation (37), but in a linearized form, can be defined as in relation (44).

$$\{\begin{array}{l}\widehat{d}=p+l{x}_2\\ \dot{p}=-lp-l\left(l{x}_2-k_{2n}{x}_2+k_{1n}{x}_3\right)\end{array}$$

(44)

In relation (44), as in the general nonlinear version, $\widehat{d}$ is the estimate of d, and p is the observer state. Let l be the gain of the observer, which is considered constant with respect to the general form given in relation (38), and let x be the state of the initial system. Figure 8 shows the DO-type observer block diagram.

The estimation error is assumed to be bounded according to the relation (45).

$$\{\begin{array}{l}{e}_d=\widehat{d}-d\\ \left|e_d\right|\le \xi \end{array}$$

(45)

The expressions in relation (46) are obtained by calculating the derivative of the estimation error in relation (45).

$${\dot{e}}_d=\dot{\widehat{d}}-\dot{d}=\dot{p}+l{\dot{x}}_2-\dot{d}=-lp-l\left(l{x}_2-k_{2n}{x}_2+k_{1n}{x}_3\right)+l\left(k_{1n}{x}_3-k_{2n}{x}_2+d\right)$$

(46)

In relation (46), $\dot{d}$ is calculated from relation (43) taking into account expressions (40)–(42). Based on this, we obtain the following relationship:

$${\dot{e}}_d=-l\left(p+l{x}_2\right)+ld=-l\left(\widehat{d}-d\right)=-l{e}_d$$

(47)

From relation (47), we can conclude that the observer proposed in relation (44) is asymptotically stable for l > 0. By choosing a Lyapunov function $V=\frac{1}{2}{e}_d^2$ and calculating its derivative $\dot{V}=e_d{\dot{e}}_d$, using Equation (47), one obtains the strict negativity of the derivative $\dot{V}$, which implies the stability of the DO-type observer.

In particular, by using a DO-type observer and choosing Δk₁ and Δk₂ equal to zero, a simplified equation can be obtained as in relation (48), from which the load torque T_L can be estimated.

$$\frac{d}{dt}\omega =\frac{1}{J}\left(\frac{3}{2}{n}_p{\lambda }_0{i}_q-T_L-B\omega \right)$$

(48)

The state vector is selected, where $x_1=\theta$,${\dot{x}}_1=x_2=\omega$, perturbation $d=-\frac{T_L}{J}$ and $K_T=\frac{3}{2}\frac{n_p{\lambda }_0}{J}$ Equation (48) can be written in the following form:

$${\dot{x}}_2=-\frac{B}{J}x+K_T{i}_q+d$$

(49)

To estimate the PMSM load torque T_L, the observer described in Equation (50) is used.

$$\{\begin{array}{l}\widehat{d}=p+l{x}_2\\ \dot{p}=-l \left(\frac{B}{J}x+K_T{i}_q+\widehat{d}\right)\end{array}$$

(50)

Figure 9 shows the DO-type observer block diagram for rotor speed PMSM estimation. The estimated quantities are the rotor position θ, the rotor speed ω, and the d term, which includes the uncertainties resulting from the parametric variation in the relationship (43).

Note that in Figure 9, given the estimated d-term ($\widehat{d}$), by adding the term $-\frac{B}{J}\omega$, we can obtain exactly the term denoted by z₃, which represents the generalized perturbation denoted by f in relation (27), which characterizes the ESO-type observer. Considering this specification, Figure 10 proposes the block diagram of the LADRC-type PMSM speed control using the DO-type observer, where the estimated quantities z₁, z₂, and z₃ are estimated using the DO-type observer instead of the ESO-type observer.

5. Matlab/Simulink Implementation of Improved Sensorless Control of PMSM Based on LADRC with ESO-Type Observer/DO-Type Observer and RL-TD3 Agent

The Matlab/Simulink implementations in this section are based on the description in the previous sections, and the nominal parameters of the PMSM used are shown in

Table 1. PMSM nominal parameters used in numerical simulation.

Parameter	Value	Unit
Stator resistance—Rs	2.875	Ω
Inductances on the d-q rotating reference frame—Ld and Lq	0.0085	H
Combined inertia of rotor and load—J	0.008	kg·m2
Combined viscous friction of rotor and load—B	0.005	N·m·s/rad
Flux induced by the permanent magnets of the rotor in the stator phases—λ0	0.175	Wb
PMSM Pole pairs number—nP	4	-

.

Figure 11 shows the block diagram of the Matlab/Simulink implementation for PMSM control based on the LADRC controller and ESO-type observer.

To improve the performance of the LADRC-type controller based on an ESO-type observer for disturbance estimation, as shown in system (15), an RL-TD3 agent can be used (Figure 12).

The implementation of an RL-TD3 agent first requires a training phase, during which the RL-TD3 agent [11,36], based on the signals collected from the system and by maximizing a reward that can be considered an integral performance criterion, will provide a correction signal for the current reference i_qref. Next, the inner current control loops are those used in the FOC control strategy and can be implemented as ON/OFF controllers with hysteresis for fast response. The usual expression of reward is given in the relation (51). The first four terms refer to the actual error due to the d-q reference frame terms, the i_d and i_q currents, and the speed and position of the PMSM rotor ω and θ relative to their references. The last term $u_{t-1}^j$ in the relation (51) contains the actions that were generated at previous times.

Figure 13 shows the evolution of the performance of the RL-TD3-type agent for improvement of LADRC with an ESO-type observer for PMSM sensorless control.

$$R_{RL-TD3}=-\left(5{\omega }_{error}^2+5{\theta }_{error}^2+5i_{derror}^2+5i_{qerror}^2+0.1{\displaystyle \sum _j{\left(u_{t-1}^j\right)}^2}\right)$$

(51)

For the implementation of the proposal presented at the end of Section 4, where the ESO-type observer is replaced by a DO-type observer described by the system (50), Figure 14 shows the block diagram of the Matlab/Simulink implementation for PMSM sensorless control based on the LADRC controller and DO-type observer.

Similar to the way presented for the LADRC controller improvement using an RL-TD3 agent when using an ESO-type observer, a similar structure as shown in Figure 15 can be used when using a DO-type observer. By using the same strategies and reward given by relation (51) in the case of using LADRC based on an ESO-type observer as in the case of using a DO-type observer, Figure 15 shows the block diagram of the Matlab/Simulink implementation for PMSM control based on a LADRC controller and a DO-type observer using the RL-TD3 agent. Furthermore, Figure 16 shows the evolution of the performance of the RL-TD3 type agent for the improvement of LADRC with the DO-type observer for PMSM sensorless control.

To optimize the value of the parameter l in system (50), which represents the gain of the DO-type observer, a computational intelligence algorithm, namely ACO [37], is used. Figure 17 shows the performance of the ACO-type algorithm for optimization of the DO-type observer gain. From Figure 17, it can be seen that after 100 iterations, the best cost indicator has a convergent evolution, which is a clue to the success of the training phase of the ACO algorithm. After optimization, the obtained value of parameter l is 191.

6. Numerical Simulations

This section presents numerical simulations for a PMSM whose parameters are presented in Section 5, and the controllers used are of type PI and LADRC, respectively, as presented in the previous sections.

The PI-type controller ensures good performance of the PMSM rotor speed control system when the PMSM load torque T_L is close to the nominal value. Therefore, Figure 18 shows the evolution of the PMSM rotor speed, electromagnetic torque T_e, stator currents i_a, i_b, i_c, and i_d and i_q currents in the d-q reference frame. The PMSM rotor speed reference sequence is as follows: ω_ref = [1000, 1200, 1400, 900] rpm for a load torque T_L = 0.5 Nm. Figure 18a shows the evolution of the parameters of PMSM control system based on PI speed controller; Figure 18b shows the evolution of the parameters of PMSM sensorless control system based on LADRC controller with ESO-type observer; Figure 18c shows the evolution of the parameters of PMSM sensorless control system based on LADRC controller with DO-type observer; and Figure 18d shows the comparison between the performance of PMSM control system based on classical type PI controller, PMSM sensorless control system based on LADRC controller with ESO-type observer, and PMSM sensorless control system based on LADRC controller with DO-type observer, respectively. It can be seen that the difference in performance between the three control systems is minimal. The numerical values of the main parameters of the LADRC controller based on an ESO-type observer are the following: ω_c = 100 rad/s and ω₀ = 200 rad/s.

By increasing the load torque T_L to 2 Nm, it can be seen in Figure 19 that the PMSM control system based on the classical PI controller manages to stabilize the system, but with a steady-state error and a longer response time compared to the case where the load torque is greater than 0.5 Nm. However, it can be seen that the control performance remains the same when using the LADRC controller with an ESO-type observer.

Figure 20 shows the time evolution of the PMSM sensorless control system parameters using the LADRC controller with an ESO-type observer for a speed reference ω_ref = 1000 rpm and a load torque T_L = 4 Nm. It is observed that the PMSM sensorless control system based on the LADRC controller with an ESO-type observer achieves very good control performance even though the load torque has increased greatly. The explanation, as presented in Section 3, is that by estimating the load torque as accurately as possible, the control system compensates for the effect of this variable, which can be considered a disturbance, and the overall behavior of the control system approaches that of a linear type, which maintains its performance even over a wide range of parameter variations. For example, in Figure 21, the performance of the control system is maintained for a 100% variation of the J parameter and T_L = 4 Nm with uniformly distributed noise in the case of using an ESO-type observer for the sensorless approach. The analysis in Figure 21 shows the parametric robustness of the LADRC controller with an ESO-type observer, with a corresponding increase in response time from 30.19 ms to 38.69 ms.

Figure 22 shows the time evolution of the PMSM sensorless control system parameters using the LADRC controller with a DO-type observer for a speed reference ω_ref = 1000 rpm and a load torque T_L = 4 Nm. Furthermore, in order to demonstrate the parametric robustness of the sensorless PMSM control system under the conditions presented above regarding parametric robustness and in the case of the sensorless approach using the LADRC controller with a DO-type obsever in Figure 23, the evolution of the control system’s quantities of interest is shown. It can be seen that in terms of the response time of the velocity signal, there is an increase from 29.59 ms to 37.51 ms. Therefore, both the parametric robustness of the LADRC controller with a DO-type observer and the superiority of the DO-type observer over the ESO-type observer can be concluded.

As shown in Section 4, the estimation of the load torque, considered a disturbance parameter, can also be performed with a DO-type observer integrated in the LADRC structure, and the results of the PMSM sensorless control system are also very good, similar to those obtained when using an ESO-type observer.

Using the diagrams proposed in the previous sections for the use of an RL-TD3 agent to improve the performance of the PMSM rotor speed control system, Figure 24 and Figure 25 show the response of the PMSM sensorless control system based on the LADRC controller with an embedded ESO-type observer (Figure 24) and DO-type observer (Figure 25).

Figure 26 shows the comparative evolution of the PMSM speed under the conditions where the controller used for the PMSM sensorless control system is one of the following: LADRC controller with ESO-type observer; LADRC controller with DO-type observer; LADRC controller with ESO-type observer and RL-TD3 agent; and LADRC controller with DO-type observer and RL-TD3 agent. It can be seen that the LADRC controller with the DO observer and RL-TD3 agent has the best performance in terms of PMSM rotor speed control.

To compare the performance of the proposed control systems, the DF calculation was also taken into account. It can be specified that the box-counting method is used to calculate DF according to [11,38,39].

Thus, Figure 27, Figure 28, Figure 29 and Figure 30 show the graphical representation for DF of the PMSM rotor speed signal for each of the proposed control systems based on combinations of the LADRC controller with an ESO-/DO-type observer and an RL-TD3 agent. Again, when using PMSM rotor speed sensorless control systems, it can be seen that the higher the performance of the control system, the higher the DF of the PMSM rotor speed signal, as shown in

Table 2. Performances of proposed controllers for PMSM sensorless type control systems.

Controller Type for PMSM Speed Control	Response Time [ms]	Rotor Speed Ripple [rpm]	DF of PMSM Rotor Speed Signal
LADRC Controller with ESO-type Observer	30.19	119.79	0.94588 +/− 0.11768
LADRC Controller with DO-type Observer	29.59	118.56	0.94696 +/− 0.16951
LADRC Controller with ESO-type Observer and RL-TD3 Agent	28.56	117.42	0.94751 +/− 0.11721
LADRC Controller with DO-type Observer and RL-TD3 Agent	28.49	116.38	0.94827 +/− 0.11746

.

While the first part of this section demonstrated the superiority of the LADRC controller over the classical PI controller through numerical simulations, Table 2 summarizes the comparative performance of the LADRC controller based on the ESO-type observer and the DO-type observer. Similar performance is observed with a slight superiority when using the DO-type observer. An increase in the performance of the LADRC controller based on the ESO-/DO-type observer is achieved by using an RL-TD3 agent.

Table 2 shows the performance improvements when using an RL-TD3 agent due to the 1-milisecond reduction in response time but also due to the reduction in rotor speed ripple.

The quality index used is the speed ripple, which is classically calculated according to the following relation: ${\omega }_{rip}=\sqrt{\left(1/N\right){\displaystyle \sum _{i=1}^N{\left({\omega }_{estim}(i)-{\omega }_{ref}(i)\right)}^2}}$ [36]. In this relation, N represents the number of samples, ω_estim is the estimated PMSM rotor speed using an ESO-type observer or DO-type observer, and ω_ref is the reference rotor speed. In this case, it is observed that most of the accumulated value is given by the transient regime, where the step signal is 1000 rpm and lasts about 8–10% of the simulation duration. The geometric interpretation of the ripple value is given by the area bounded by the reference speed and the estimated PMSM rotor speed.

The following quality indicators are used to compare the performance of the control systems presented: response time, rotor speed ripple, and DF of the PMSM rotor speed signal. It can be concluded that the use of an RL-TD3 agent improves the performance of the LADRC controller in that the RL-TD3 agent will provide correction signals that ensure a more accurate estimation of the disturbance acting on the system, but the LADRC controller will still ensure a global behavior like that of a double integrator by using a PD control law.

From Table 2, it can be seen the performance improvement of both the LADRC controller with an ESO-type observer and the LADRC controller with a DO-type observer by using the RL-TD3 agent. Thus, there is a decrease in response time by 5.4% and 3.7%, respectively, and in PMSM rotor speed by 1.98% and 1.94%, respectively. Regarding the direct comparison between the control types LADRC controller with ESO-type observer and LADRC controller with DO-type observer, both in the basic versions and when used in tandem with an RL-TD3 agent, there is a decrease in response time of 1.99% and 0.3%, respectively, and in terms of PMSM rotor speed of 0.2% and 0.1%, respectively. Comparing the performance of the PMSM control system based on LADRC controller with ESO-type observer in the basic version with the performance of the PMSM control system based on LADRC controller with DO-type observer plus RL-TD3 agent shows a decrease in response time of 5.63% and, regarding the PMSM rotor speed ripple, a decrease of 2.85%. Thus, it can be concluded that in order to obtain the most accurate disturbance rejection, the use of an RL-TD3 agent is proposed, both in combination with an ESO-type observer and with a DO-type observer.

7. Conclusions

This paper presents the global FOC control structure of a PMSM in which the outer rotor speed control loop is implemented using a LADRC controller. Typically, the operation of the LADRC controller requires rejection of the disturbance acting on the system, which is estimated using an ESO-type observer. Similar performance for a LADRC controller can be obtained by replacing the ESO-type observer with a DO-type observer. A computational intelligence algorithm—ACO—is used to optimize the gain value of the DO-type observer. A presentation of the operating equations, control structures, and their implementation in the Matlab/Simulink development environment is provided. In order to obtain the most accurate disturbance rejection, the use of an RL-TD3 agent is proposed, both in combination with an ESO-type observer and with a DO-type observer. Comparative results obtained from numerical simulations are presented for cases where the controllers are PI, LADRC, or LADRC in tandem with an RL-TD3 agent. Superior performance is achieved when an RL-TD3 is used due to the correction signals provided for the most accurate estimation of disturbances, even in the case of parametric variations within wide limits. Furthermore, it is also concluded in this case that the DO-type observer is superior compared to the ESO-type observer.

Future work proposes real-time implementation and optimization of the control structure by modifying the RL-TD3 agent and the cumulative reward, along with optimization of the other control parameters using a wide range of computational intelligence algorithms.