Robust Control of An Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning

Abstract

This paper is primarily focused on the robust control of an inverted pendulum system based on policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

1. Introduction

In the last decade, there has been increased interest in the robust control of the inverted pendulum system (IPS) owing to its high potential in testing a variety of advanced control algorithms. Robust control is widely used in power electronics, flight control, motion control, network control, and IPSs, in addition to other fields [1,2]. Research on the robust control of an IPS has provided advantageous results in recent years. An inverted pendulum is an experimental device that has insufficient drive, absolute instability, and uncertainty. It has become an excellent benchmark in the field of automatic control over the last few decades as it provides better explanations for model-based nonlinear control techniques and is a typical experimental platform for verifying classical and modern control theories.

Although the earliest research on IPSs can be traced back to 1908 [3], there is almost no literature on this subject between 1908 and 1960. In 1960, a number of tall, slender structures survived the Chilean earthquake, while structures that appeared more stable were severely damaged. Therefore, some scholars conducted more in-depth research to obtain a suitable explanation [4]. A pendulum structure under the effect of an earthquake was modeled as a base and rigid block system, and block overturning was studied by applying a horizontal acceleration, sinusoidal pulses, and seismic-type excitations to the system. It was observed that there is an unexpected scaling effect that makes the large block more stable than the small block among two geometrically similar blocks. Furthermore, tall blocks exhibit greater stability during earthquakes when exposed to horizontal forces. Since then, with the development of modern control theory, various control methods have been applied to different types of IPSs, such as proportional–integral–derivative control, cloud model control, fuzzy control, sliding mode control, and neural network control methods [5,6,7]. These methods provide different ideas for the control of IPSs.

As is known, the IPS is an uncertain system, and the uncertainty of its model is naturally within the scope of consideration. The aim of the robust control of an IPS is to find a controller capable of addressing system uncertainties. When the system is disturbed by uncertainty, robust control laws can stabilize the system. Because it is difficult to directly solve the robust control problem, some scholars transformed the robust control problem into an optimal control problem. In [8], the authors proposed a robust optimal control method for linear systems with matching uncertainty. However, the situation where the uncertainty does not meet matching conditions has not been considered. Lin et al. [9,10] conducted research on the robust optimal control of uncertain systems by adjusting the value of the weighting matrix and solving an algebraic Riccati equation (ARE) to obtain robust control laws. Zhang et al. [11] presented a unified framework for studying robust optimal control problems with adjustable uncertainty sets. Wang et al. [12] developed a novel adaptive critical learning approach for robust optimal control of a class of uncertain affine nonlinear systems with matching uncertainties. And the data-based adaptive critical designs were developed to solve the Hamilton–Jacobi–Bellman (HJB) equation corresponding to the transformed optimal control problem.

In fact, the pioneering methods for solving optimal control problems mainly include dynamic programming [13] and maximum principles [14]. With the dynamic programming method, solving the HJB equation yields optimal control of the system. As for the optimal control problem of a linear system with a quadratic performance index, irrespective of whether it is a continuous system or a discrete system, it finally comes down to solving an ARE. However, when the dimension of the state vector or control input vector in the dynamic system is large, the so-called “curse of dimensionality” appears when the dynamic programming method is used to solve the optimal control problem [15]. To overcome this weakness, some scholars have used the reinforcement learning (RL) policy to solve the optimal control problem [16,17].

When RL was initially used for system control, it was primarily focused on discrete-time systems or discretized continuous-time systems in research on problems such as the billiard game problem [18], scheduling problem [19], and robot navigation problem [20]. Furthermore, the application of RL algorithms to continuous-time and continuous-state systems was initially extended by Doya et al. [21]. They used the known system model to learn the optimal control policy. In the context of control engineering, RL and adaptive dynamic programming link traditional optimal control methods to adaptive control methods [22,23,24]. Vrabie et al. [25] used the RL algorithm to solve the optimal control problem of the continuous time system. In the case of the linear system, system data are collected, and the solution of the HJB equation is obtained via online policy iteration (PI) using the least squares method. Xu et al. [26,27] proposed an RL algorithm based on linear continuous-time systems to solve the robust control and robust tracking problems through online PI. The algorithm takes into consideration the uncertainty in the system’s state and input matrices and improves the method for solving robust control.

The IPS demonstrates a positive impact in the validation of RL algorithms. There are some literatures on RL to solve the control problem of inverted pendulum systems. Bates [28] harnessed GPUs to quickly train a simulation of an inverted pendulum to balance itself. Israilov et al. [29] used two model-free RL algorithms to control targets and proposed a general framework to reproduce successful experiments and simulations based on the inverted pendulum. In addition, there are still many studies of this kind, for example [30,31,32]. However, the results of these studies focused more on reducing the time to reach equilibrium, without fully considering uncertainty in the system. We attempt to solve the robust control problem of an uncertain IPS using RL algorithms. Only the input and output data need to be collected when using the RL control algorithm and the state matrix of the nominal system need not be known. This study lays out a theoretical foundation for the wide application of the RL control algorithm in engineering systems. The main contributions of this study are as follows.

(1) The state-space model of the IPS with uncertainty is established and a robust optimal control method is applied to the IPS model. By constructing an appropriate performance index, the optimal control method is used to design a robust control law for an uncertain IPS.

(2) A PI algorithm in the RL has been designed to realize the robust optimal control of an IPS. The use of the RL algorithm to solve the robust control problem of IPS does not require knowing the state matrix, only collecting input and output data. The application of the RL for solving the control problem of an IPS has significance for its potential application in practical engineering.

The organization of this paper is as follows. In Section 2, the state-space equation of the IPS and a linearization model are established. The robust control and RL algorithm for linearizing the IPS are presented in Section 3 and Section 4. In Section 5, we establish the nonlinear state-space model of the IPS and propose the corresponding RL algorithm. The RL algorithm is then verified via a simulation in Section 6. Finally, we summarize the work of this paper and potential future research directions.

2. Model Formulation

In this section, we established a physical model of a first-order linear IPS according to Newton’s second law. By selecting appropriate state variables, the state-space model with uncertainty is derived.

2.1. Modeling of Inverted Pendulum System

The inverted pendulum experimental device comprises a pendulum and a trolley [33]. Its structure is presented in Figure 1. The encoder is a photoelectric rotary one, and the motor is a direct-current servo motor. For a detailed information on the experimental platform, see [34]. Moreover, its simplified physical model is presented in Figure 2, which mainly includes the pendulum and trolley. In Figure 2, owing to the interaction between the trolley and pendulum, the trolley is subjected to a force $F_3$ from the pendulum, which acts in the lower left direction. Furthermore, the pendulum is subjected to a force $F_4$ from the trolley, which acts in the upper right direction. In addition, the pendulum and trolley are also subjected to other forces, as shown in Figure 3 and Figure 4, respectively. The trolley is driven by a motor to perform horizontal movements on the guide rail. In Figure 3, the trolley is subjected to the force $F_1$ from the motor and gravity. $F_2$ represents the resistance between the trolley and guide rail. Furthermore, $N_1$ and $P_1$ are the two components of force $F_3$. In Figure 4, the pendulum is subjected to gravity $G=m_{1}g$, and $N_2$ and $P_2$ are the two components of force $F_4$.

To facilitate subsequent calculations, we define the parameter of the first-order IPS, as shown in

Table 1. IPS parameter symbols.

Parameter	Unit	Significance
$m_1$	kg	Mass of the trolley
$m_2$	kg	Mass of the pendulum
L	m	Half the length of the pendulum
z	N/m/s	Friction coefficient between the trolley and guide rail
x	m	Displacement of the trolley
$\theta$	rad	Angle from the upright position
I	kg·m2	Moment of inertia of pendulum

. The time parameter symbol $\left(t\right)$ is omitted, which indicates that x represents $x\left(t\right)$. In Figure 3, according to Newton’s second Law, the trolley satisfies the following equation in the horizontal direction:

$$\begin{array}{c}\hfill F_1-N_1-F_2=m_2\ddot{x}\end{array}$$

(1)

We assume that the resistance is proportional to the speed of the trolley. Therefore, $F_2=z\dot{x}$, z is the proportional coefficient. Moreover, in Figure 4, the pendulum satisfies the following equation in the horizontal direction:

$$\begin{array}{c}\hfill N_2=m_1\frac{d^2}{d{t}^2}(x-L\sin \theta )=m_1\ddot{x}+m_{1}L{\dot{\theta }}^2\sin \theta -m_{1}L\ddot{\theta }\cos \theta \end{array}$$

(2)

Considering that $N_1=N_2$ in Figure 2, and on substituting (2) into (1), we obtain

$$\begin{array}{c}\hfill F_1=(m_1+m_2)\ddot{x}+z\dot{x}+m_{1}L{\dot{\theta }}^2\sin \theta -m_{1}L\ddot{\theta }\cos \theta \end{array}$$

(3)

Next, in Figure 4, we analyze the resultant force in the vertical direction of the pendulum, and the following equation can be obtained.

$$\begin{array}{c}\hfill P_2-m_{1}g=m_1\frac{d^2}{d{t}^2}\left(\cos \theta \right)=-m_{1}L{\dot{\theta }}^2\cos \theta -m_{1}L\ddot{\theta }\sin \theta \end{array}$$

(4)

The component force of $N_2$ in the direction perpendicular to the pendulum is

$$\begin{array}{c}\hfill N_2\cos \theta =m_1\frac{d^2}{d{t}^2}(x-L\sin \theta )\cos \theta =m_1\ddot{x}\cos \theta +m_{1}L{\dot{\theta }}^2\sin \theta cos\theta -m_{1}L\ddot{\theta }{\cos }^2\theta \end{array}$$

(5)

Based on the torque balance, we can obtain the following equation

$$\begin{array}{c}\hfill I\ddot{\theta }=P_{2}L\sin \theta +N_{2}L\cos \theta \end{array}$$

(6)

where I is the moment of inertia of the pendulum. On substituting Equations (4) and (5) into Equation (6),

$$\begin{array}{c}\hfill (I+m_1{L}^2)\ddot{\theta }-m_{1}gL\sin \theta =m_{1}L\ddot{x}\cos \theta \end{array}$$

(7)

Thus far, Equations (3) and (7) constitute the dynamic model of the IPS. Moreover, it can be assumed that the rotation angle of the pendulum is very small, that is, $\theta <<1\left(radian\right)$. Therefore, it can be approximated that

$$\begin{array}{c}\hfill sin\theta \approx \theta ,cos\theta \approx 1\end{array}$$

Therefore, it follows from Equations (3) and (7),

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{F}_1=(m_1+m_2)\ddot{x}+z\dot{x}+m_{1}L{\dot{\theta }}^2\theta -m_{1}L\ddot{\theta }\hfill \\ (I+m_1{L}^2)\ddot{\theta }-m_{1}gL\theta =m_{1}L\ddot{x}\hfill \end{array}\right.\end{array}$$

(8)

2.2. State-Space Model with Uncertainty

In Section 2.1, we established the dynamic model of the IPS as shown in Equation (8). Next, we will derive the state-space model of the IPS.

As the rotation angle of the pendulum $\theta$ is very small, it can be approximated that $\dot{\theta }\approx 0$, ${\dot{\theta }}^2\approx 0$. It follows from (8) that

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{F}_1=(m_1+m_2)\ddot{x}+z\dot{x}-m_{1}L\ddot{\theta }\hfill \\ (I+m_1{L}^2)\ddot{\theta }-m_{1}gL\theta =m_{1}L\ddot{x}\hfill \end{array}\right.\end{array}$$

(9)

Equation (9) is the linearized dynamic model of the system. The first equation comes from (1)–(3), which is the equilibrium force equation of the system in the horizontal direction. The second equation comes from (4)–(7), which is the equilibrium force equation of the system in the vertical direction. The state variables of the system can be defined as

$$\begin{array}{c}\hfill x_1=x,x_2=\dot{x},x_3=\theta ,x_4=\dot{\theta }.\end{array}$$

Therefore, the following state-space equation can be derived.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}\dot{x_1}=x_2\hfill \\ \dot{x_2}=\frac{-(I+m_1{L}^2)z}{I(m_1+m_2)+m_1{m}_2{L}^2}{x}_2+\frac{m_1^{2}g{L}^2}{I(m_1+m_2)+m_1{m}_2{L}^2}{x}_3+\frac{I+m_1{L}^2}{I(m_1+m_2)+m_1{m}_2{L}^2}u\hfill \\ \dot{x_3}=x_4\hfill \\ \dot{x_4}=\frac{-m_{1}Lz}{I(m_1+m_2)+m_1{m}_2{L}^2}{x}_2+\frac{m_{1}gL(m_1+m_2)}{I(m_1+m_2)+m_1{m}_2{L}^2}{x}_3+\frac{m_{1}L}{I(m_1+m_2)+m_1{m}_2{L}^2}u\hfill \end{array}\right.\end{array}\end{array}$$

(10)

where u represents the force $F_3$ from the motor. Using $W=I(m_1+m_2)+m_1{m}_2{L}^2$, Equation (10) can be written as

$$\begin{array}{c}\hfill \dot{x}=Ax\left(t\right)+Bu\left(t\right)\end{array}$$

where

$$\begin{array}{c}\hfill A=\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& \frac{-(I+m_1{L}^2)z}{W}& \frac{m_1^{2}g{L}^2}{W}& 0\\ 0& 0& 0& 1\\ 0& \frac{-m_{1}Lz}{W}& \frac{m_{1}gL(m_1+m_2)}{W}& 0\end{array}\right],B=\left[\begin{array}{c}0\\ \frac{I+m_1{L}^2}{W}\\ 0\\ \frac{m_{1}L}{W}\end{array}\right]\end{array}$$

However, the accurate model of the IPS is difficult to obtain, and all its parameters have uncertainties. In this paper, the friction coefficient z between the trolley and guide rail is selected as an uncertain parameter. The numerical values of the other parameters in Table 1 are known, where $m_1=0.109,m_2=1.096,L=0.25,$ and $I=0.0034$. Therefore, the state-space model of the uncertain IPS can be abbreviated as

$$\begin{array}{c}\hfill \dot{x}=A\left(z\right)x+Bu\end{array}$$

(11)

where

$$\begin{array}{ccc}\hfill A\left(z\right)& =& \left[\begin{array}{cccc}0& 1& 0& 0\\ 0& -0.8832z& 0.6293& 0\\ 0& 0& 0& 1\\ 0& -2.3566z& 27.8285& 0\end{array}\right],B=\left[\begin{array}{c}0\\ 0.8832\\ 0\\ 2.3566\end{array}\right]\hfill \end{array}$$

Here we choose $z_0=0.1$ as the nominal value and denote the nominal matrix of the system as $A\left(z_0\right)$. Therefore, the nominal system corresponding to the uncertain system (11) is

$$\begin{array}{c}\hfill \dot{x}=A\left(z_0\right)x+Bu\end{array}$$

(12)

where

$$\begin{array}{ccc}\hfill A\left(z_0\right)& =& \left[\begin{array}{cccc}0& 1& 0& 0\\ 0& -0.0883& 0.6293& 0\\ 0& 0& 0& 1\\ 0& -0.2357& 27.8285& 0\end{array}\right]\hfill \end{array}$$

3. Robust Control of Uncertain Linear System

This section mainly presents the robust optimal control methods for the uncertain IPS modeled in the previous section by selecting the appropriate performance index function and solving an ARE to construct the robust control law. When the uncertain parameters of the system change within a certain range, this robust control law can cause the system to become asymptotically stable.

The following lemmas are proposed to prove the main results of this paper.

Lemma 1. 

The nominal system (12) corresponding to system (11) is stabilizeable.

Proof. 

For the four-dimensional continuous time-invariant system presented in system (12), the controllability matrix is constructed as

$$\begin{array}{c}\hfill G=\left[BA\left(z_0\right)BA{\left(z_0\right)}^{2}BA{\left(z_0\right)}^{3}B\right]\end{array}$$

Therefore, we have

$$\begin{array}{c}\hfill rank\left(G\right)=rank\left[\begin{array}{cccc}0& 0.8832& -0.0780& 1.4899\\ 0.8832& -0.0780& 1.4899& -0.2626\\ 0& 2.3566& -0.2082& 65.5990\\ 2.3566& -0.2082& 65.5990& -6.1442\end{array}\right]=4\end{array}$$

Therefore, system (12) is completely controllable, which means that the system can be stabilized. This completes the proof.    □

Lemma 2. 

There is an $m\times n$ matrix $\delta \left(z\right)$, such that the system matrices $A\left(z\right)$ and $A\left(z_0\right)$ satisfy the following matched condition.

$$\begin{array}{c}\hfill A\left(z\right)-A\left(z_0\right)=B\delta \left(z\right)\end{array}$$

(13)

Proof. 

$$\begin{array}{c}\hfill A\left(z\right)-A\left(z_0\right)=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0.8832(0.1-z)& 0& 0\\ 0& 0& 0& 0\\ 0& 2.3566(0.1-z)& 0& 0\end{array}\right]=\left[\begin{array}{c}0\\ 0.8832\\ 0\\ 2.3566\end{array}\right]\delta \left(z\right)=B\delta \left(z\right)\end{array}$$

(14)

where

$$\begin{array}{c}\hfill \delta \left(z\right)=\left[\begin{array}{cccc}0& 0.1-z& 0& 0\end{array}\right]\end{array}$$

(15)

This completes the proof.    □

Lemma 3. 

For any $z\in [0,1]$, there exists a positive semidefinite matrix F, such that $\delta \left(z\right)$ satisfies

$$\begin{array}{c}\hfill \delta {\left(z\right)}^T\delta \left(z\right)\le F\end{array}$$

(16)

where $F\ge 0$.

Proof. 

According to Lemma 2, we can obtain

$$\begin{array}{c}\hfill \delta {\left(z\right)}^T\delta \left(z\right)=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& {(0.1-z)}^2& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right]\le \left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0.81& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right]=F\end{array}$$

(17)

This completes the proof.    □

For nominal system (12), we construct the following ARE.

$$\begin{array}{c}\hfill SA\left(z_0\right)+A{\left(z_0\right)}^{T}S+M-SB{B}^{T}S=0\end{array}$$

(18)

where $M=F+I$. According to the above three lemmas and ARE (18), we propose the following theorem.

Theorem 1. 

Let us suppose that S is a symmetric positive definite solution to ARE (18). Then, for all uncertainties $z\in [0,1]$, the feedback control $u=Kx$, $K=-B^{T}S$ can make system (11) asymptotically stable.

Proof. 

We define the following Lyapunov function.

$$\begin{array}{c}\hfill V\left(x\right)=x^{T}Sx\end{array}$$

(19)

We set $u=Kx$ and take the time derivative of Lyapunov Function (19) along system (11). We can then obtain

$$\dot{V}\left(x\right)=x^T\left[{\left(A\left(z\right)\right)}^T+K^T{B}^T\right]Sx+x^{T}S\left[\left(A\left(z\right)\right)+BK\right]x$$

According to Lemma 2, we can obtain

$$\begin{array}{cc}\hfill \dot{V}\left(x\right)=& x^T\left[{\left(A\left(z_0\right)+B\delta \left(z\right)\right)}^T+K^T{B}^T\right]Sx+x^{T}S\left[\left(A\left(z_0\right)+B\delta \left(z\right)\right)+BK\right]x\hfill \\ \hfill =& x^T[A{\left(z_0\right)}^{T}S+SA\left(z_0\right)+\delta {\left(z\right)}^T{B}^{T}S]x+x^{T}SB\delta \left(z\right)x+2x^{T}SBKx\hfill \end{array}$$

On substituting ARE (18) into the above equation, we obtain

$$\begin{array}{cc}\hfill \dot{V}\left(x\right)=& -x^{T}Mx+x^{T}SB{B}^{T}Sx+x^T\delta {\left(z\right)}^T{B}^{T}Sx+x^{T}SB\delta \left(z\right)x+2x^{T}SBKx\hfill \end{array}$$

because $K=-B^{T}S$,

$$\begin{array}{cc}\hfill \dot{V}\left(x\right)=& -x^{T}Mx+x^T{K}^{T}Kx-x^T\delta {\left(z\right)}^{T}Kx+x^{T}SB\delta \left(z\right)x+2x^{T}SBKx\hfill \end{array}$$

As

$$\begin{array}{cc}\hfill & -x^T{K}^{T}Kx-2x^T{K}^T\delta \left(z\right)x=-x^T{\left(K+\delta \left(z\right)\right)}^T\left(K+\delta \left(z\right)\right)x+x^T\delta {\left(z\right)}^T\delta \left(z\right)x\le x^T\delta {\left(z\right)}^T\delta \left(z\right)x\hfill \end{array}$$

we can obtain

$$\begin{array}{cc}\hfill \dot{V}\left(x\right)=& -x^{T}Mx+x^T{K}^{T}Kx-x^T\delta {\left(z\right)}^{T}Kx+x^{T}SB\delta \left(z\right)x+2x^{T}SBKx\hfill \\ \hfill =& -x^{T}Mx-x^T{K}^{T}Kx-2x^T\delta {\left(z\right)}^{T}Kx\hfill \\ \hfill =& -x^{T}Mx-x^T{\left(K+\delta \left(z\right)\right)}^T\left(K+\delta \left(z\right)\right)x+x^T\delta {\left(z\right)}^T\delta \left(z\right)x\hfill \\ \hfill \le & -x^{T}Mx+x^T\delta {\left(z\right)}^T\delta \left(z\right)x\hfill \\ \hfill \le & -x^T(M-F)x\hfill \\ \hfill \le & -x^{T}x\hfill \end{array}$$

Therefore,

$$\begin{array}{c}\hfill \dot{V}\left(x\right)=0x=0\\ \hfill \dot{V}\left(x\right)\le 0x\ne 0\end{array}$$

According to the Lyapunov stability theorem [35], the uncertain system (11) is asymptotically stable. Theorem 1 has thus been proved.    □

4. RL Algorithm for Robust Optimal Control

In this section, we propose an RL algorithm for solving the robust control problem of an IPS through online PI. According to ARE (18), the following optimal control problem is constructed. For the nominal system,

$$\begin{array}{c}\hfill \dot{x}=A\left(z_0\right)x+Bu\end{array}$$

we find a control u, such that the following performance index reaches a minimum.

$$\begin{array}{c}\hfill J={\int }_t^{\infty }[x^{T}Mx+u^{T}u]dt\end{array}$$

(20)

where $M=F+I>0$. For any initial time t, the optimal cost can be written as

$$\begin{array}{cc}\hfill V\left[x\right(t\left)\right]& ={\int }_t^{\infty }\left(x^{T}Mx+u^{T}u\right)dt\hfill \\ \hfill & ={\int }_t^{t+\delta t}\left(x^{T}Mx+u^{T}u\right)dt+{\int }_{t+\delta t}^{\infty }\left(x^{T}Mx+u^{T}u\right)dt\hfill \\ \hfill & ={\int }_t^{t+\delta t}\left(x^{T}Mx+u^{T}u\right)dt+V\left[x(t+\delta t)\right]\hfill \end{array}$$

From Lyapunov Function (19), we obtain

$$\begin{array}{cc}\hfill & x{\left(t\right)}^{T}Sx\left(t\right)={\int }_t^{t+\delta t}\left(x^{T}Mx+u^{T}u\right)dt+x{(t+\delta t)}^{T}Sx(t+\delta t)\hfill \end{array}$$

(21)

where S is the solution to ARE (18). We propose the following RL algorithm for solving a robust controller.

In Algorithm 1, by providing an initial stabilizing control law, repeated iterations are performed between steps 3 and 4 until convergence. We can then obtain the robust control gain K of system (11).

Algorithm 1 RL Algorithm for Uncertain Linear IPS

(1) $M=F+I$ is computed. (2) An initial stabilization control gain $K_0$ is selected. (3) Policy evaluation: $S_i$ is solved using the equation $x^T\left(t\right)S_{i}x\left(t\right)={\int }_t^{t+\delta t}{x}^T(M+K_i^T{K}_i)xdt+x^T(t+\delta t)S_{i}x(t+\delta t)$. (4) Policy improvement: $K_{i+1}=-B^T{S}_i$. (5) We set $i=i+1$, and steps 3 and 4 are repeated until $\parallel S_{i+1}-S_i\parallel \le ϵ$, where $ϵ>0$ is a small constant.

Remark 1. 

Step 3 in Algorithm 1 is the policy evaluation, and step 4 is the policy improvement. Equivalently, the solving of the equation in step 3 is actually solving a least squares problem. In the integral interval, if sufficient data are obtained in the system, the least square method can be used to solve $S_i$.

Next, we prove the convergence of Algorithm 1. However, it is necessary to prove the following Lemma first.

Lemma 4. 

On assuming that the matrix $A\left(z_0\right)+B{K}_i$ is stable, solving the matrix $S_i$ from step 3 of Algorithm 1 becomes equivalent to solving the following equation.

$$S(A\left(z_0\right)+B{K}_i)+{(A\left(z_0\right)+B{K}_i)}^{T}S+M+K_i^T{K}_i=0$$

(22)

Proof. 

We rewrite the equation of step 3 in Algorithm 1 as follows

$$\begin{array}{c}\hfill \underset{\delta t\to 0}{lim}\frac{x^T(t+\delta t)S_{i}x(t+\delta t)-x^T\left(t\right)S_{i}x\left(t\right)}{\delta t}+\underset{\delta t\to 0}{lim}\frac{{\int }_t^{t+\delta t}{x}^T(M+K_i^T{K}_i)xdt}{\delta t}=0\end{array}$$

(23)

According to the definition of the derivative, it can be observed that the first term of Equation (23) is the derivative of $x^T\left(t\right)S_{i}x\left(t\right)$ with respect to time t. We thus obtain

$$\begin{array}{c}\hfill \frac{d\left(x^T\left(t\right)S_{i}x\left(t\right)\right)}{dt}+\underset{\delta t\to 0}{lim}\frac{d}{d\delta t}{\int }_t^{t+\delta t}{x}^T(M+K_i^T{K}_i)xdt=0\end{array}$$

(24)

Further re-arranging Equation (24) yields

$$\begin{array}{c}\hfill x^T[S(A\left(z_0\right)+B{K}_i)+{(A\left(z_0\right)+B{K}_i)}^{T}S+M+K_i^T{K}_i]x=0\end{array}$$

(25)

which means that (22) is established. Next, we reverse the process.

Along the stable system $\dot{x}=(A+B{K}_i)x$, the time derivative of the Lyapunov function $V_i\left(x\right)=x^T{S}_{i}x$ is calculated. We can then obtain

$$\begin{array}{cc}\hfill {\dot{V}}_i\left(x\right)& =\frac{d{x}^T\left(t\right)S_{i}x\left(t\right)}{dt}=x^T{(A\left(z_0\right)+B{K}_i)}^T{S}_{i}x+x^T{S}_i(A\left(z_0\right)+B{K}_i)x\hfill \end{array}$$

(26)

On integrating both sides of the Equation (26) in the interval $(t,\delta t)$, we obtain

$$\begin{array}{c}\hfill x^T(t+\delta t)S_{i}x(t+\delta t)-x^T\left(t\right)S_{i}x\left(t\right)={\int }_t^{t+\delta t}{x}^T(M+K_t^T{K}_i)xdt\end{array}$$

This completes the proof.

According to the existing conclusions [36], iterative relations (22) and step 3 of Algorithm 1 converge to form the solution of ARE (18).    □

Remark 2. 

The behavior of the control is evaluated using a cost function, which is similar to the reward in RL. The agent corresponds to the controller in optimal control, and the control process corresponds to the environmental model in RL. In control engineering, maximizing rewards is equivalent to minimizing the cost function, so the ultimate goal of the controller is to develop an optimal control policy by learning.

5. Robust Control of Nonlinear IPS

In this section, a nonlinear state-space model of the IPS is established. Moreover, we construct a suitable auxiliary system and corresponding performance index. The problem of the robust control of the IPS is then transformed into the optimal control problem of the auxiliary system. We finally propose the corresponding RL algorithm.

5.1. Nonlinear State-Space Representation of IPS

Based on the uncertain linear inverted pendulum model (11) established in Section 2.1, we consider the following uncertain nonlinear system.

$$\begin{array}{c}\hfill \dot{x}=A\left(z\right)x\left(t\right)+Bu\left(t\right)+F(z,x)\end{array}$$

(27)

where $F(z,x)$ represents the nonlinear perturbation of the system and can be used to represent various nonlinearity factors in the system. Based on the modeling process in Section 2 and [8], it is assumed that

$$\begin{array}{c}\hfill F(z,x)=\left[\begin{array}{c}0\\ \frac{-(I+m_1{L}^2)z}{W}(\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}-1)x_2\\ 0\\ \frac{-m_{1}Lz}{W}(\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}-1)x_2\end{array}\right]\end{array}$$

where the parameters I, $m_1$, L, and W are the same as those in (10). On rewriting system (27), we obtain

$$\begin{array}{c}\hfill \dot{x}=\overline{A}x+Bu+\overline{F}(z,x)\end{array}$$

(28)

where

$$\begin{array}{c}\hfill \begin{array}{cc}& \overline{A}=\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& \frac{m_1^{2}g{L}^2}{W}& 0\\ 0& 0& 0& 1\\ 0& 0& \frac{m_{1}gL(m_1+m_2)}{W}& 0\end{array}\right],\overline{F}(z,x)=\left[\begin{array}{c}0\\ \frac{-(I+m_1{L}^2)z}{W}(\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2})x_2\\ 0\\ \frac{-m_{1}Lz}{W}(\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2})x_2\end{array}\right]\hfill \end{array}\end{array}$$

On substituting the parameter values into system (28), we can obtain

$$\begin{array}{c}\hfill \begin{array}{cc}& \overline{A}=\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& 0.6293& 0\\ 0& 0& 0& 1\\ 0& 0& 27.8285& 0\end{array}\right],B=\left[\begin{array}{c}0\\ 0.8832\\ 0\\ 2.3566\end{array}\right],\overline{F}(z,x)=\left[\begin{array}{c}0\\ -0.8832z{x}_2\\ 0\\ -2.3566z{x}_2\end{array}\right][\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}]\hfill \end{array}\end{array}$$

5.2. Robust Control of Nonlinear IPS

To obtain the robust control law for an uncertain nonlinear IPS, we propose the following two lemmas.

Lemma 5. 

There exists an uncertain function $G(z,x)$ such that $\overline{F}(z,x)$ can be decomposed into the following form.

$$\begin{array}{c}\hfill \overline{F}(z,x)=BG(z,x)\end{array}$$

Proof. 

$$\begin{array}{cc}\hfill \overline{F}(z,x)=& \left[\begin{array}{c}0\\ \frac{-(I+m_1{L}^2)z}{N}{x}_2\\ 0\\ \frac{-m_{1}Lz}{N}{x}_2\end{array}\right][\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}]\hfill \\ \hfill =& \left[\begin{array}{c}0\\ \frac{I+m_1{L}^2}{N}\\ 0\\ \frac{m_{1}L}{N}\end{array}\right](-z{x}_2[\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}])=BG(z,x)\hfill \end{array}$$

where $G(z,x)=-z{x}_2[\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}]$. This completes the proof.    □

Lemma 6. 

There exists an upper bound function $f_{max}\left(x\right)$ such that $G(z,x)$ satisfies

$$\begin{array}{c}\hfill \mid G(z,x\mid \le f_{max}\left(x\right)\end{array}$$

(29)

Proof. 

$$\begin{array}{cc}\hfill \mid G(z,x)\mid & =\mid -z{x}_2[\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}]\mid \hfill \\ \hfill & =\mid z{x}_2\cos (x_1{x}_2+x_3{x}_4)+z(0.5x_1+2x_3-4x_4)\mid \hfill \\ \hfill & \le \mid x_2\cos (x_1{x}_2+x_3{x}_4)+z(0.5x_1+2x_3-4x_4)\mid \hfill \\ \hfill & \le \mid x_2+0.5x_1+2x_3-4x_4\mid \hfill \\ \hfill & =f_{max}\left(x\right)\hfill \end{array}$$

This completes the proof.    □

We construct the optimal control problems for a nominal system.

$$\begin{array}{c}\hfill \dot{x}=\overline{A}x\left(t\right)+Bu\left(t\right)\end{array}$$

(30)

We determine a controller u, which minimizes the following performance index

$$\begin{array}{c}\hfill J(x_0,u)={\int }_0^{\infty }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt\end{array}$$

(31)

Based on the performance index Function (31), the cost function to the admissible control policy $u\left(x\right)$ is

$$\begin{array}{c}\hfill V\left(x\right)={\int }_t^{\infty }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt\end{array}$$

(32)

We define $\nabla V$ as the gradient of $V\left(x\right)$ with respect to x. Finding differentiation on both sides of (32) with respect to t yields the following Bellman equation.

$$\begin{array}{c}\hfill f_{max}^2\left(x\right)+x^{T}x+u^{T}u+\nabla V^T[\overline{A}x+Bu]=0\end{array}$$

(33)

Then, we define the following Hamiltonian function.

$$H(x,u,\nabla V)=f_{max}^2\left(x\right)+x^{T}x+u^{T}u+\nabla V^T[\overline{A}x+Bu]$$

(34)

On assuming that the minimum exists and is unique, the optimal control function for the given problem is then obtained as

$$\begin{array}{c}\hfill u_{opt}=-\frac{1}{2}{B}^T\nabla V_{opt}\end{array}$$

(35)

On substituting Equation (35) into Equation (33), the HJB equation that satisfies the optimal function $V_{opt}\left(x\right)$ can be obtained as

$$\begin{array}{c}\hfill f_{max}^2\left(x\right)+x^{T}x+\nabla V_{opt}^T\overline{A}x-\frac{1}{4}\nabla V_{opt}^{T}B{B}^T\nabla V_{opt}=0\end{array}$$

(36)

with the initial condition $V_{opt}\left(0\right)=0$.

On solving the optimal function $V_{opt}\left(x\right)$ from Equation (36), the solution of the optimal control problem can be obtained. The solution of the robust control problem can then be obtained. The following theorem shows that the optimal control $u_{opt}=-\frac{1}{2}{B}^T\nabla V_{opt}$ is a robust controller for a nonlinear IPS.

Theorem 2. 

On considering the nominal system (30) with the performance index (31) and assuming that solution $V_{opt}\left(x\right)$ of the HJB Equation (36) exists, the optimal control law (35) can then globally stabilize the IPS (28).

Proof. 

We select $V_{opt}\left(x\right)$ as the Lyapunov function. On considering the performance index function (31), $V_{opt}\left(x\right)\ge 0$ is in evidence, and $V_{opt}\left(0\right)=0$. Solving the derivative of $V_{opt}\left(x\right)$ with respect to t along system (28) yields

$$\begin{array}{cc}\hfill \frac{d{V}_{opt}}{dt}=& \nabla V_{opt}^T[\overline{A}x+F(z,x)]-\frac{1}{2}\nabla V_{opt}^{T}B{B}^T\nabla V\hfill \end{array}$$

According to Lemma 5, it follows that

$$\begin{array}{cc}\hfill \frac{d{V}_{opt}}{dt}=& \nabla V_{opt}^T\overline{A}x+\nabla V_{opt}^{T}BG(z,x)-\frac{1}{2}\nabla V_{opt}^{T}B{B}^T\nabla V\hfill \end{array}$$

(37)

According to HJB Equation (36), we can obtain

$$\begin{array}{c}\hfill \nabla V_{opt}^T\overline{A}x=-f_{max}^2\left(x\right)-x^{T}x+\frac{1}{4}\nabla V_{opt}^{T}B{B}^T\nabla V_{opt}\end{array}$$

(38)

On substituting Equation (38) into Equation (37), we obtain

$$\begin{array}{cc}\hfill \frac{d{V}_{opt}}{dt}=& -f_{max}^2\left(x\right)-x^{T}x+\frac{1}{4}\nabla V_{opt}^{T}B{B}^T\nabla V_{opt}+\nabla V_{opt}^{T}BG(z,x)-\frac{1}{2}\nabla V_{opt}^{T}B{B}^T\nabla V\hfill \\ \hfill =& -f_{max}^2\left(x\right)-x^{T}x-\frac{1}{4}\nabla V_{opt}^{T}B{B}^T\nabla V_{opt}+\nabla V_{opt}^{T}BG(z,x)\hfill \end{array}$$

(39)

From Equation (39), we can obtain

$$\begin{array}{cc}\hfill \frac{d{V}_{opt}}{dt}=& -f_{max}^2\left(x\right)-x^{T}x-\frac{1}{4}[\nabla V_{opt}^{T}B{B}^T\nabla V_{opt}-4\nabla V_{opt}^{T}BG(z,x)+4G^T(z,x)G(z,x)]+G^T(z,x)G(z,x)\hfill \\ \hfill =& -x^{T}x+G^T(z,x)G(z,x)-f_{max}^2\left(x\right)-\frac{1}{4}{H}^T(z,x)H(z,x)\hfill \\ \hfill \le & -x^{T}x\hfill \end{array}$$

(40)

where $H(z,x)=B^T\nabla V_{opt}-2G(z,x)$. According to the Lyapunov stability criterion, the optimal controller (35) can asymptotically stabilize the uncertain nonlinear IPS (28) for all the allowable uncertainties. Therefore, for a constant $p>0$, there exists a neighborhood $\mathcal{N}=\{x:\parallel x\parallel <p\}$ near the origin, so that, if $x\left(t\right)\in \mathcal{N}$, then $x\to 0$ when $t\to \infty$. However, $x\left(t\right)$ cannot remain outside the domain $\mathcal{N}$ forever, or else $\parallel x\left(t\right)\parallel \ge p$ for all $t>0$, which implies that

$$\begin{array}{cc}\hfill V_{opt}\left[x\left(t\right)\right]-V_{opt}\left[x\left(0\right)\right]& ={\int }_0^t{\dot{V}}_{opt}\left(x\left(\tau \right)\right)d\tau \hfill \\ \hfill & \le {\int }_0^t-x^{T}xd\tau \hfill \\ \hfill & \le {\int }_0^{\tau }-p^{2}d\tau \hfill \\ \hfill & =-p^{2}t\hfill \end{array}$$

Let $t\to \infty$, then

$$\begin{array}{c}\hfill V_{opt}\left[x\left(t\right)\right]\le V_{opt}\left[x\left(0\right)\right]-p^{2}t\to -\infty \end{array}$$

This completes the proof.    □

5.3. RL Algorithm for Nonlinear IPS

For a nonlinear IPS, we consider the optimal control problems (30) and (31). For any admissible control, the cost function corresponding to the optimal control problem can be expressed as

$$\begin{array}{cc}\hfill V\left[x\right(t\left)\right]=& {\int }_t^{\infty }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt\hfill \\ \hfill =& {\int }_t^{t+\varphi }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt+{\int }_{t+\varphi }^{\infty }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt\hfill \end{array}$$

where $\varphi >0$ is an arbitrarily selected constant. We can then obtain the integral reinforcement relation satisfied by the cost function

$$\begin{array}{cc}\hfill V\left[x\right(t\left)\right]=& {\int }_t^{t+\varphi }[f_{max}^2\left(x\right)+x^{T}x+u^{T}u]dt+V\left[x(t+\varphi )\right]\hfill \end{array}$$

(41)

According to the integral-based reinforcement relations (41) and the optimal controller (35), the RL algorithm for the robust control of the nonlinear IPS is as follows.

In Algorithm 2, by providing an initial stabilizing control law, the algorithm iterates repeatedly between steps 3 and 4 until convergence. We can then obtain the robust control gain u of system (28).

Algorithm 2 RL Algorithm of Uncertain Nonlinear IPS

(1) A non-negative function $f_{max}\left(x\right)$ is selected. (2) An initial stabilization control law $u_0\left(x\right)$ is selected. (3) Policy evaluation: the $V_i\left(x\right)$ from $V_i\left[x\left(t\right)\right]={\int }_t^{t+\varphi }[f_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)]dt+V_i\left[x(t+\varphi )\right]$ is solved. (4) Policy improvement: $u_{i+1}\left(x\right)=-\frac{1}{2}{B}^T\nabla V_i$. (5) $i=i+1$ is set and steps 3 and 4 are repeated until $\parallel V_{i+1}-V_i\parallel \le ϵ$, where $ϵ>0$ is a small constant.

Next, we prove the convergence of Algorithm 2. The following conclusion provides an equivalent form of the integral strengthening relation in step 3.

Lemma 7. 

On assuming that $u_i\left(x\right)$ is the stabilization control function of the nominal system (30), solving the cost function $V_i\left(x\right)$ from the equation in step 3 in Algorithm 2 can be equivalent to solving Equation (42).

$$\begin{array}{cc}\hfill & f_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)+\nabla V_i[\overline{A}x+B{u}_i]=0\hfill \end{array}$$

(42)

Proof. 

On dividing both sides of the equation in step 3 by $\varphi$ and taking the limit, we obtain

$$\begin{array}{c}\hfill \underset{\varphi \to 0}{lim}\frac{V_{i}x(t+\varphi )-V_{i}x\left(t\right)}{\varphi }+\underset{\varphi \to 0}{lim}\frac{{\int }_t^{t+\varphi }[f_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)]dt}{\varphi }=0\end{array}$$

Based on the definition of the function limit and L’ Hopital’s rule, we obtain

$$\begin{array}{c}\hfill \frac{d{V}_{i}x\left(t\right)}{dt}+\underset{\varphi \to 0}{lim}\frac{d}{d\varphi }{\int }_t^{t+\varphi }[f_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)]dt=0\end{array}$$

Therefore,

$$\begin{array}{c}\hfill f_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)+\nabla V_i[\overline{A}x+B{u}_i]=0\end{array}$$

However, along the stable system $\dot{x}=\overline{A}x+B{u}_i$, finding the derivative of $V_i\left(x\right)$ with respect to t yields

$$\begin{array}{c}\hfill \frac{d}{dt}\left(V_i\left(x\right)\right)=\nabla V_i(\overline{A}x+B{u}_i)\end{array}$$

On integrating both sides of the above equation from t to $t+\varphi$, we obtain

$$\begin{array}{c}\hfill V_i\left[x(t+\varphi )\right]-V_i\left[x\left(t\right)\right]={\int }_t^{t+\varphi }\nabla V_i(\overline{A}x+B{u}_i)d\tau \end{array}$$

Then, from (42), we obtain

$$\begin{array}{c}\hfill V_i\left[x\left(t\right)\right]={\int }_t^{t+\varphi }{f}_{max}^2\left(x\right)+x^{T}x+u_i^T\left(x\right)u_i\left(x\right)d\tau +V_i\left[x(t+\varphi )\right]\end{array}$$

The above equation is consistent with the third step of Algorithm 2. This completes the proof. □

According to the conclusions of [25,37], if the initial control policy $u_0\left(x\right)$ can stabilize the system, the control policy taken using the optimal control Function (35) and Equation (42) also can stabilize the system. Furthermore, the iteratively calculated cost function sequence converges to the optimal cost function. From Lemma 7, we know that Equation (42) and the equation of step 3 are equivalent. Therefore, the iterative relationship between steps 3 and 4 in Algorithm 2 converges on the optimal control and optimal cost functions.

6. Numerical Simulation Results

This section includes two simulation examples to illustrate the practical applicability of the theoretical results in the robust control of the uncertain IPS.

6.1. Example 1

Considering system (11), whose state-space model can be referenced in [34], our objective is to obtain a robust control u such that it is stable. Based on Lemmas 1–3, the weighting matrix M is selected as

$$\begin{array}{c}\hfill M=F+I=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1.81& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\end{array}$$

We present the initial stability control law

$$\begin{array}{c}\hfill u_0=[1.09004.1230-24.8908-6.7726]x\end{array}$$

The initial state of the nominal system is selected as $x_0={\left[0111\right]}^T$. The time-step size for the collecting system status and input information is set as 0.01 s. Algorithm 1 converges after six iterations, and the $S_d$ matrix and control gain $K_d$ converge to the following optimal solutions:

$$\begin{array}{c}\hfill S_d=\left[\begin{array}{cccc}2.4465& 2.0822& -6.2489& -1.2066\\ 2.0822& 4.3346& -14.0702& -2.7082\\ -6.2489& -14.0702& 100.8262& 18.9646\\ -1.2066& -2.7082& 18.9646& 3.6646\end{array}\right]\end{array}$$

(43)

and

$$\begin{array}{c}\hfill K_d=[1.00442.5538-32.2652-6.2440]\end{array}$$

(44)

There are 10 independent numerical samples in the matrix $S_d$. These 10 numerical samples are collected in each iteration to address the least squares problem. The evolution of the control signal u is presented in Figure 5. Figure 6 illustrates the iterative convergence process of the S matrix, where $S(i,j)$ represents the element lying at the intersection of the i-th row and the j-th column in the symmetric matrix S, where $i=1,2,3,4$, $j=1,2,3,4$.

The ARE (18) is solved directly by Matlab, the S matrix and optimal feedback K are obtained as follows.

$$\begin{array}{c}\hfill S=\left[\begin{array}{cccc}2.4455& 2.0802& -6.2326& -1.2039\\ 2.0802& 4.3307& -14.0381& -2.7032\\ -6.2326& -14.0381& 100.5677& 18.9228\\ -1.2039& -2.7032& 18.9228& 3.6579\end{array}\right]\end{array}$$

(45)

$$\begin{array}{c}\hfill K=[1.0002.5455-32.1952-6.2327]\end{array}$$

(46)

As is apparent, the results from the two methods are very similar. Figure 7 presents the closed-loop trajectory of system (11). Figure 7a–d represent the closed-loop system trajectories for uncertain parameters $z=0.1,0.4,0.7,1.0$, respectively. It is easy to observe that the system is stable, which means that the controller is valid

Table 2. Characteristic root of system (11) when z takes different values.

z	$\mathit{\lambda }1$	$\mathit{\lambda }2$	$\mathit{\lambda }3$	$\mathit{\lambda }4$
0.1	−6.60	−4.23	−0.85 + 0.32i	−0.85 − 0.32i
0.2	−6.73	−4.33	−0.78 + 0.43i	−0.78 − 0.43i
0.3	−6.86	−4.41	−0.71 + 0.50i	−0.71 − 0.50i
0.4	−7.00	−4.48	−0.65 + 0.55i	−0.65 − 0.55i
0.5	−7.14	−4.54	−0.60 + 0.59i	−0.60 − 0.59i
0.6	−7.28	−4.59	−0.55 + 0.62i	−0.55 − 0.62i
0.7	−7.42	−4.63	−0.50 + 0.65i	−0.50 − 0.65i
0.8	−7.56	−4.67	−0.46 + 0.67i	−0.46 − 0.67i
0.9	−7.70	−4.70	−0.42 + 0.68i	−0.42 − 0.68i
1.0	−7.84	−4.73	−0.38 + 0.69i	−0.38 − 0.69i

displays the respective partial eigenvalues of the system (11) with $u=Kx$ under varying values of z. From Table 2, we can observe that the eigenvalues of the closed-loop system all have negative real parts. Thus, the uncertain linear system (11) with robust control $u=Kx$ is asymptotically stable for all $0\le z\le 1$.

6.2. Example 2

Let us consider the nonlinear IPS (28). According to Lemma 5, system (28) can be rewritten as

$$\begin{array}{c}\hfill \dot{x}=\overline{A}x+Bu+BG(z,x)\end{array}$$

(47)

The optimal control problem for the IPS is as follows: for nominal system (30), we find an optimal control u such that the performance index (31) achieves a minimum.

According to Lemma 6, we obtain

$$\begin{array}{c}\hfill \mid G(z,x)\mid =\mid -z{x}_2[\cos (x_1{x}_2+x_3{x}_4)+\frac{0.5x_1+2x_3-4x_4}{x_2}]\mid \le |x_2+0.5x_1+2x_3-4x_4|=f_{max}\left(x\right)\end{array}$$

then

$$\begin{array}{c}\hfill f_{max}^2\left(x\right)={(x_2+0.5x_1+2x_3-4x_4)}^2=x^T\left[\begin{array}{cccc}0.25& 0.5& 1& -2\\ 0.5& 1& 2& -4\\ 1& 2& 4& -8\\ -2& -4& -8& 16\end{array}\right]x\end{array}$$

According to performance index (31), the weight matrix M is selected as

$$\begin{array}{c}\hfill M=\left[\begin{array}{cccc}1.25& 0.5& 1& -2\\ 0.5& 2& 2& -4\\ 1& 2& 5& -8\\ -2& -4& -8& 17\end{array}\right]\end{array}$$

Based on Algorithm 2, we give the initial control policy

$$\begin{array}{c}\hfill u_0=[1.09004.1230-24.8908-6.7726]x\end{array}$$

The initial state of the system is selected as $x_0={\left[0111\right]}^T$. Algorithm 2 converges after six iterations, and the $S_d$ matrix and control gain $K_d$ converge to the following optimal solutions.

$$\begin{array}{c}\hfill S_d=\left[\begin{array}{cccc}2.4325& 2.4398& -6.2874& -1.3888\\ 2.4398& 5.0469& -14.0539& -3.0045\\ -6.2874& -14.0539& 130.4535& 18.9735\\ -1.3888& -3.0045& 18.9735& 4.2715\end{array}\right]\end{array}$$

(48)

and

$$\begin{array}{c}\hfill K_d=[1.11802.6229-32.3006-7.4126]\end{array}$$

(49)

The evolution of the control signal u is presented in Figure 8. Figure 9 presents the convergence process of the $S_d$ matrix.

We also selected z = 0.1, 0.4, 0.7, 1.0. Figure 10 presents the closed-loop trajectory of system (28). Figure 10a–d represent the closed-loop system trajectories for uncertain parameters z = 0.1, 0.4, 0.7, and 1.0, respectively. It is easy to observe that the system is stable, which means that the controller is valid.

7. Conclusions

In this paper, the robust control problem of a first-order IPS is studied. The linearization and nonlinear state-space representation are established, and an RL algorithm for the robust control of the IPS is proposed. The controller of the uncertain system is obtained using the method of online PI. The results thus obtained show that the error between the controller obtained using the RL algorithm and by directly solving ARE is very small. Moreover, the algorithm can provide a controller that meets the requirements without the nominal matrix A of the system being known, only collecting input and output data. This improves the current state at which the robust control of the IPS relies excessively on the nominal matrix. In future research, we intend to take into consideration that the input matrix of the system also has uncertainty and extend the RL algorithm to more general systems.