Data-Driven Optimal Preview Repetitive Control of Linear Discrete-Time Systems

Abstract

This paper investigates the problem of data-driven optimal preview repetitive control of linear discrete-time systems. Firstly, by integrating prior information into the preview time domain, an augmented state-space system is established. Secondly, the original output tracking problem is mathematically reconstructed and transformed into the optimization problem form of a linear quadratic tracking (LQR). Furthermore, a Q-function-based iterative algorithm is designed to dynamically calculate the optimal tracking control gain based solely on online measurable data. This method has a dual-breakthrough feature: it neither requires prior knowledge of system dynamics nor provides an initial stable controller. Finally, the superiority of the proposed scheme is verified through numerical simulation experiments.

1. Introduction

In practical control system applications, there is often a situation where the system tracks a specific periodic signal. Repetitive control (RC) is a high-precision control method that tracks a reference signal with a known period [1]. Preview control (PC) is a control method that improves the performance of closed-loop systems by utilizing known future information from target signals or interference signals [2]. The combination of PC and RC, known as preview repetitive control [3,4] (PRC), was first developed in the early 1990s. It utilizes both the internal model compensation mechanism of repetitive learning and the compensation of preview information; thus, it can significantly improve the control performance of closed-loop systems.

Recently, a great deal of research has been devoted to the theory and applications of PRC, and various structures and algorithms have been devised [5,6,7,8]. In [5], a design method of discrete-time sliding-mode preview repetitive servo systems was proposed. Using a parameter-dependent Lyapunov function method and linear matrix inequality (LMI) techniques, a robust guaranteed-cost PRC law was proposed in [6]. Based on the optimal control theory, the problem of optimal PRC for continuous-time linear systems was investigated in [7] by solving the algebraic Riccati equation (ARE). Using a two-dimensional model approach,  observer-based PRC for uncertain discrete-time systems was studied in [8]. In [9], a discrete-time PRC scheme was proposed for linear discrete-time periodic systems. To compensate for unknown external disturbances, the problem of PRC with equivalent-input disturbance (EID) for uncertain continuous-time systems was presented in [10]. The problem of Padé approximation-based optimal PRC with EID was proposed in [11]. In [12], the PRC for discrete-time linear parameter-varying systems with a time-varying delay was investigated. However, the above results depend on a perfect understanding of the system model.

In model driven-based control, a mathematical model is first established and the controller is then designed based on this model [13,14]. The inherent problem of system modeling is the motivation for research in the field of data-driven control (DDC) technology [15,16]. An approach of designing an optimal preview output tracking controller using online measurable data and a Q-function-based iteration algorithm was developed [17]. A data-driven framework to control a complex network optimally and without any knowledge of the network dynamics was develop in [18]. The fault-tolerant consensus control problem for a general class of linear continuous-time multi-agent systems (MASs) was considered in [19]. In [20], the problem of the decoupled eata-based optimal control policy for a nonlinear stochastic systems was addressed. In [21], a data-based iteration learning control for multiphase batch processes was investigated. A data-based approach to linear quadratic tracking (LQT) control design was presented in [22]. In [23], a novel adaptive dynamic programming-based learning algorithm was proposed by only accessing the input and output data to solve the LQT problem. In [24], a robust data-driven finite-horizon linear quadratic regulator control problem of unknown linear time-invariant systems was addressed. DDC-based $H_{\infty }$ optimal tracking control for systems in the presence of external disturbance was presented in  [25,26].

Note that the mechanism model with system matrices and order needs to be known in existing PC/RC studies. Actually, many practical plants are hard to model, particularly for the complicated modern control systems and complex networks [27]. On the other hand, the above studies do not reflect the research interests in data-driven PC and RC compensation information. These observations inspire our current study. In this paper, we investigate the problem of data-driven preview repetitive control (DD-PRC) of linear discrete-time systems. The main contributions of this paper are summarized as follows: (i) By taking advantage of the preview information of the desired tracking signal as well as the periodic learning ability of repetitive control, the original control problem is converted to an LQT one under an augmented state-space model. (ii) In order to solve the DDC-based LQT problem, a Q-function-based iterative algorithm is then designed to dynamically calculate the optimal tracking control gain instead of solving the ARE.

The rest of this paper is organized follows. Section 2 gives the problem formulation. The data-driven PRC law design method is given in Section 3. In Section 4, the numerical simulation is presented to demonstrate the validity of the design method. Finally, the conclusions are drawn in Section 5.

2. Problem Formulation

Consider a discrete-time linear system given by

$$\left\{\begin{array}{c}x(k+1)=Ax\left(k\right)+Bu\left(k\right),\hfill \\ y\left(k\right)=Cx\left(k\right),\hfill \end{array}\right.$$

(1)

where $x\left(k\right)\in {\mathbb{R}}^n$ is the system state, $y\left(k\right)\in {\mathbb{R}}^m$ is the measurement output, and $u\left(k\right)\in {\mathbb{R}}^p$ is the control input. $A,B$, and C are the corresponding appropriate dimensional matrices.

The configuration of the data-driven PRC system is shown in Figure 1, where $G\left(z\right)$ is the controlled plant as described in System (1). $r\left(k\right)\in {\mathbb{R}}^m$ denotes the constant reference trajectory signal with a period of L. $C_R\left(z\right)$ is the basic repetitive control block, and $v\left(k\right)$ is the output of the repetitive controller and is given by

$$v\left(k\right)=\left\{\begin{array}{cc}e\left(k\right),\hfill & 0\le k<L,\hfill \\ v(k-L)+e\left(k\right),\hfill & k\ge L,\hfill \end{array}\right.$$

(2)

where $e\left(k\right)=r\left(k\right)-y\left(k\right)$ is the tracking error.

The following assumptions are needed for System (1).

Assumption 1. 

Assume that $A,B$, and C satisfy both $(A,B)$ controllable and $(A,C)$ observable.

Assumption 2. 

The preview length of the reference trajectory signal $r\left(k\right)$ is limited to M ($M<L$). That is, at the current moment k, the current value of $r\left(k\right)$ and the M-step future values $r\left(k\right),r(k+1),\dots ,r(k+M)$ are available. Additionally, it is assumed that the future values beyond the reference trajectory forward range L are set to 0, i.e.,  $r(k+L)=0$.

Remark 1. 

Note that Assumption 1 is a necessary assumption for the optimal state feedback control, while Assumption 2 is a standard working hypothesis introduced to facilitate the PC mathematical development [2,3,4]. It indicates that the reference signal’s values are known in the actual system for a certain time period, during which the reference signal exceeds the preview length, and the impact is very small.

The objective of this paper is to design a DD-PRC law $u\left(k\right)$ with the form of

$$u\left(k\right)=\sum _{j=0}^{L-1}{K}_{vj}v(k-j)+K_{x}x\left(k\right)+\sum _{j=0}^M{K}_{rj}r(k+j),$$

(3)

where $K_{vj}$ and $K_{rj}$ are the gains of the repetitive controller and the preview compensator, respectively, and $K_x$ is the gain of the feedback controller, such that the system output $y\left(k\right)$ tracks the reference signal $r\left(k\right)$ and the quadratic cost function

$$J=\sum _{k=0}^{\infty }\left[e^T\left(k\right)Q_{e}e\left(k\right)+\Delta u^T\left(k\right)R\Delta u\left(k\right)\right]$$

(4)

is minimized subject to system (1), where $Q_e$ and R are symmetric positive definite and semi-positive definite symmetric matrices, respectively, and $\Delta u\left(k\right)=u\left(k\right)-u(k-L).$

Now, define the following L-order difference operator:

$$\left\{\begin{array}{c}\Delta x\left(k\right)=x\left(k\right)-x(k-L),\hfill \\ \Delta y\left(k\right)=y\left(k\right)-y(k-L),\hfill \\ \Delta u\left(k\right)=u\left(k\right)-u(k-L),\hfill \\ \Delta r\left(k\right)=r\left(k\right)-r(k-L).\hfill \end{array}\right.$$

(5)

Substituting (5) into System (1) yields

$$\left\{\begin{array}{c}\hspace{1em}\Delta x(k+1)=A\Delta x\left(k\right)+B\Delta u\left(k\right),\\ e(k+1)=\Delta r(k+1)-CA\Delta x\left(k\right)-CB\Delta u\left(k\right)\\ +e(k+1-L).\end{array}\right.$$

(6)

Introducing the state vector

$${\overline{x}}_d\left(k\right)={\left[e\left(k\right),e(k-1),\dots ,e(k-L+1),\Delta x\left(k\right)\right]}^T,$$

(7)

Therefore,

$$\left\{\begin{array}{c}{\overline{x}}_d(k+1)=\overline{A}{\overline{x}}_d\left(k\right)+D_r\Delta r(k+1)+D\Delta u\left(k\right),\\ \hspace{1em}\hspace{1em} \Delta y\left(k\right)=C_1{\overline{x}}_d\left(k\right),\end{array}\right.$$

(8)

where

$$\begin{array}{ccc}& & \overline{A}=\left[\begin{array}{ccccc}0& \cdots & 0& I& -CA\\ I& \cdots & 0& 0& 0\\ ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& \cdots & I& 0& 0\\ 0& \cdots & 0& 0& A\end{array}\right],\hfill \\ & & D={\left[\begin{array}{ccccc}-CB& 0& \cdots & 0& B\end{array}\right]}^{\mathrm{T}},\hfill \\ & & D_r={\left[\begin{array}{ccccc}I& 0& \cdots & 0& 0\end{array}\right]}^{\mathrm{T}},C_1=\left[\begin{array}{cccc}0& \cdots & 0& C\end{array}\right].\hfill \end{array}$$

Since M-step future values $r\left(k\right),r(k+1),\dots ,r(k+M)$ are available, a new $m(M+1)\times 1$ state vector $X_r\left(k\right)$ can be defined as

$$X_r\left(k\right)={\left[\begin{array}{cccc}\Delta r\left(k\right)& \Delta r(k+1)& \cdots & \Delta r(k+M)\end{array}\right]}^{\mathrm{T}}.$$

(9)

Combining Equations (8) and (9) yields

$$\overline{X}(k+1)={\Phi }_x\overline{X}\left(k\right)+D_x\Delta u\left(k\right),$$

(10)

where

$$\begin{array}{ccc}& & \overline{X}\left(k\right)=\left[\begin{array}{c}{\overline{x}}_d\left(k\right)\\ X_r\left(k\right)\end{array}\right]\in {\mathbb{R}}^{n+L+m(M+1)},\hfill \\ & & D_x=\left[\begin{array}{c}D\\ 0\end{array}\right]\in {\mathbb{R}}^{[n+L+m(M+1\left)\right]\times p},\hfill \\ & & {\Phi }_x=\left[\begin{array}{cc}\overline{A}& D_{xr}\\ 0& A_r\end{array}\right]\in {\mathbb{R}}^{[n+L+m(M+1\left)\right]\times [n+L+m(M+1\left)\right]},\hfill \\ & & D_{xr}=\left[\begin{array}{ccccc}0\hfill & D_r\hfill & 0\hfill & \cdots \hfill & 0\hfill \end{array}\right]\in {\mathbb{R}}^{(n+L)\times m(M+1)},\hfill \\ & & A_r=\left(\begin{array}{ccccc}0& I& 0& \cdots & 0\\ 0& \ddots & \ddots & \ddots & ⋮\\ ⋮& & \ddots & \ddots & 0\\ 0& \cdots & \cdots & 0& I\\ 0& \cdots & \cdots & 0& 0\end{array}\right)\in {\mathbb{R}}^{m(M+1)\times m(M+1)}.\hfill \end{array}$$

With these definitions, the performance index (4) can be rewritten as

$$\overline{J}=\sum _{k=0}^{\infty }\left[{\overline{X}}^{\mathrm{T}}\left(k\right)Q\overline{X}\left(k\right)+\Delta u^{\mathrm{T}}\left(k\right)R\Delta u\left(k\right)\right]$$

(11)

where, $Q=diag\left[\begin{array}{ccc}{Q}_e\hfill & \cdots \hfill & 0\hfill \end{array}\right]$.

Thus, the control problem associated with Discrete System (1) under Performance Index (4) is converted into an optimal LQT control problem of the Augmented System (10), subject to Performance Index (11).

According to (2) and (13), we can get

$$\begin{array}{ccc}\hfill \Delta u\left(k\right)& =& \sum _{j=0}^{L-1}{K}_{vj}e(k-j)+K_x\Delta x\left(k\right)\hfill \\ & +& \sum _{j=0}^M{K}_{rj}\Delta r(k+j)\hfill \\ & =& -K_1{\overline{x}}_d\left(k\right)-K_2{X}_r\left(k\right)\hfill \\ & =& -K\overline{X}\left(k\right),\hfill \end{array}$$

(12)

where

$$\begin{array}{ccc}& & K_1=-[K_{v0},K_{v1},\cdots ,K_{v(L-1)},K_x],\hfill \\ & & K_2=-[K_{r0},K_{r1},\cdots ,K_{rM}],\hfill \\ & & K=-[K_1,K_2].\hfill \end{array}$$

Hence, if the optimal controller $\Delta u\left(k\right)=-K\overline{X}\left(k\right)$ subject to (10) and (11) is solved, then the DD-PRC law (13) will be derived. In fact, using (5) and (12), we can find that

$$u\left(k\right)=\sum _{j=0}^{L-1}{K}_{vj}v(k-j)+K_{x}x\left(k\right)+\sum _{j=0}^M{K}_{rj}r(k+j).$$

(13)

It can be seen that the above control law $u\left(k\right)$ consists of three terms. The first term represents RC, which is used to eliminate the steady-state tracking error. The second term describes the state-feedback control, which is used to guarantee the stability of closed-loop system. The third term denotes the preview action, which is used to improve the transient response of systems. The combination of these three terms can achieve the satisfactory tracking performance.

3. Data-Driven PRC Law Design

It has been shown in [28] that the optimal controller can be obtained if we can find a unique positive definite solution of the following algebraic Riccati equation (ARE):

$$P={\Phi }_x^{T}P{\Phi }_x+Q-{\Phi }_x^{T}P{D}_x\left(R+D_x^{T}P{D}_x\right)D_x^{T}P{\Phi }_x.$$

(14)

Then, we can obtain the optimal controller:

$$\Delta u\left(k\right)=-K\overline{X}\left(k\right),$$

(15)

where

$$K={\left(R+D_x^{T}P{D}_x\right)}^{-1}{D}_x^{T}P{\Phi }_x.$$

(16)

This problem can be solved using the iterative algorithm [17,28] to approximate the solution of ARE (14), which is given as Algorithm 1 for comparison in the next section.

Algorithm 1 Iterative algorithm for solving K

1.  Set $K_0$ and $j=0$, where j denotes iteration index. 2.  Solve the following equation for $P_{j+1}$, $$\begin{array}{ccc}\hfill P_{j+1}& =& {\Phi }_x^T{P}_j{\Phi }_x+Q\hfill \\ & -& {\Phi }_x^T{P}_j{D}_x\left(R+D_x^T{P}_j{D}_x\right)D_x^T{P}_j{\Phi }_x.\hfill \end{array}$$ (17) 3.  Calculate $K_{j+1}$ by $$K_{j+1}={\left(R+D_x^T{P}_{j+1}{D}_x\right)}^{-1}{D}_x^T{P}_{j+1}{\Phi }_x.$$ (18) 4.  Set $j=j+1$ and repeat step 2–3 until $∥P_{j+1}-P_j∥\le \epsilon$ with a small constant $\epsilon$$(\epsilon >0)$. Output $P_{j+1}$ as the solution of ARE (14).

Note that the above iterative algorithm, Algorithm 1, is essentially model-based. In this section, we will develop a model-free control approach by using online input and output data.

For the LQT problem, we define the following cost function:

$$V\left(\overline{X}\left(k\right)\right)=\sum _{k=0}^{\infty }\left[{\overline{X}}^T\left(k\right)Q\overline{X}\left(k\right)+\Delta u^T\left(k\right)R\Delta u\left(k\right)\right],$$

(19)

whose optimal value can be parameterized as $V\left(\overline{X}\left(k\right)\right)={\overline{X}}^T\left(k\right)P\overline{X}\left(k\right).$

Using the well-known Bellman equation, we get

$$\begin{array}{cc}\hfill V^{*}\left(\overline{X}\left(k\right)\right)& =\mathcal{R}(\overline{X}\left(k\right),\Delta u\left(k\right))+V\left(\overline{X}(k+1)\right)\hfill \\ \hfill & =\mathcal{R}(\overline{X}\left(k\right),\Delta u\left(k\right))\hfill \\ \hfill & +V^{*}({\Phi }_x\overline{X}\left(k\right)+D_x\Delta u\left(k\right)),\hfill \end{array}$$

(20)

where $\mathcal{R}(.)={\overline{X}}^T\left(k\right)Q\overline{X}\left(k\right)+\Delta u^T\left(k\right)R\Delta u\left(k\right)$ can be viewed as the gain from reinforcement learning, and the corresponding $V\left(\overline{X}(k+1)\right)$ function can be viewed as the cost function.

Define a Q-function, also known as action value function $Q(\overline{X}\left(k\right),u\left(k\right))$, given by

$$Q(\overline{X}\left(k\right),\Delta u\left(k\right))=\mathcal{R}(\overline{X}\left(k\right),\Delta u\left(k\right))+V\left(\overline{X}(k+1)\right).$$

(21)

From (21), it can be observed that this Q-function describes the performance associated with the state–action pair $(\overline{X}\left(k\right),\Delta u\left(k\right)).$

For the proposed optimal controller $u\left(k\right)$, it can be derived from Equations (20) and (21) that $V\left(\overline{X}\left(k\right)\right)=Q(\overline{X}\left(k\right),\Delta u\left(k\right))$ and therefore the Q-function also has a corresponding Bellman formation:

$$\begin{array}{ccc}\hfill Q(\overline{X}\left(k\right),\Delta u\left(k\right))& =& \mathcal{R}(\overline{X}\left(k\right),\Delta u\left(k\right))\hfill \\ & +& Q(\overline{X}(k+1),\Delta u(k+1)).\hfill \end{array}$$

(22)

From the optimality of Bellman’s formula, the optimal value function $V^{*}\left(\overline{X}\left(k\right)\right)$ and the optimal controller $\Delta u^{*}\left(k\right)$ is given by

$$V^{*}\left(\overline{X}\left(K\right)\right)=\underset{\Delta u\left(k\right)}{min}{Q}^{*}(\overline{X}\left(k\right),\Delta u\left(k\right))$$

(23)

$$u^{*}\left(k\right)=arg\underset{\Delta u\left(k\right)}{min}{Q}^{*}(\overline{X}\left(k\right),\Delta u\left(k\right)).$$

(24)

It follows from the Q-function (21) that

$$\begin{array}{cc}\hfill & Q^{*}(\overline{X}\left(k\right),\Delta u\left(k\right))\hfill \\ \hfill =& {\overline{X}}^T\left(k\right)Q_a\overline{X}\left(k\right)+u^T\left(k\right)R\Delta u\left(k\right)\hfill \\ \hfill & +{\left[{\Phi }_x\overline{X}\left(k\right)+D_x\Delta u\left(k\right)\right]}^{T}P\left[{\Phi }_x\overline{X}\left(k\right)+D_x\Delta u\left(k\right)\right]\hfill \\ \hfill =& {\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]}^{T}G\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right],\hfill \end{array}$$

(25)

where

$$\begin{array}{cc}\hfill G& =\left[\begin{array}{cc}{G}_{11}\hfill & G_{12}\hfill \\ G_{12}^T\hfill & G_{22}\hfill \end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{cc}{Q}_a+{\Phi }_x^{T}P{\Phi }_x& {\Phi }_x^{T}P{D}_x\\ D_x^{T}P{\Phi }_x& R+D_x^{T}P{D}_x\end{array}\right].\hfill \end{array}$$

(26)

By solving ${\displaystyle \frac{\partial Q^{*}(\overline{X}\left(k\right),\Delta u\left(k\right))}{\partial \Delta u\left(k\right)}}=0$, we can obtain

$$\Delta u^{*}\left(k\right)=-K\overline{X}\left(k\right)=-G_{22}^{-1}{G}_{12}^T\overline{X}\left(k\right)$$

(27)

If P is the only positive definite solution obtained from the ARE (14), then $K=G_{22}^{-1}{G}_{12}^T$, which is equal to the gain of the optimal LQT control.

The Q-learning scheme generates the sequences $Q^j(\overline{X}\left(k\right),\Delta u\left(k\right))$, and $\Delta u\left(k\right)$ can converge to $Q^{*}(\overline{X}\left(k\right),\Delta u\left(k\right))$, $\Delta u^{*}\left(k\right)$, respectively, which has been proven in [29]. It is worth pointing out that this Q-learning scheme requires a priori information of system dynamics and an initially stabilizing control gain. To overcome this drawback, in the following section, based on the Q-function value, a data-driven based algorithm will be developed to obtain the optimal control gain.

To implement the iterative algorithm, the iterative Q-function $Q^j(\overline{X},\Delta u)$ is denoted as

$$Q^j(\overline{X},u)={\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]}^T{G}^j\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right],$$

(28)

where

$$G^j=\left[\begin{array}{cc}{G}_{11}^j& G_{12}^j\\ {\left(G_{12}^j\right)}^T& G_{22}^j\end{array}\right].$$

(29)

It follows from (27), (28), and (29) that

$$\Delta u^j=-K_j\overline{X}\left(k\right)=-{\left(G_{22}^j\right)}^{-1}{\left(G_{12}^j\right)}^T\overline{X}\left(k\right).$$

(30)

Using (17) and (28), we have

$$\begin{array}{cc}\hfill & {\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]}^T{\Pi }^{j+1}\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]\hfill \\ \hfill =& {\left[\begin{array}{c}\overline{X}(k+1)\\ -K_j\overline{X}(k+1)\end{array}\right]}^T{\Pi }^j\left[\begin{array}{c}\overline{X}(k+1)\\ -K_j\overline{X}(k+1)\end{array}\right]\hfill \\ \hfill & +{\overline{X}}^T\left(k\right)Q_a\overline{X}\left(k\right)+\Delta u^T\left(k\right)R\Delta u\left(k\right),\hfill \end{array}$$

(31)

where

$${\Pi }^j=\left[\begin{array}{cc}{Q}_a+{\Phi }_x^T{P}_j{\Phi }_x& {\Phi }_x^T{P}_j{D}_x\\ D_x^T{P}_j{\Phi }_x& R+D_x^T{P}_j{D}_x\end{array}\right]=G^j.$$

(32)

In the following, for computational simplicity, the Kronecker product is introduced to convert $G^j$ into a vector representation. Note that

$$a^{T}Wb=(b^T\otimes a^T)\mathrm{vec}\left(W\right),$$

(33)

where $a\in {\mathbb{R}}^{1\times m},W\in {\mathbb{R}}^{m\times m},b\in {\mathbb{R}}^{m\times 1}$, ⊗ is the Kronecker product, and $\mathrm{vec}\left(W\right)$ is the vector formed by stacking the columns of matrix $W.$ Therefore, it follows from (31) that

$${\rho }_j(X,\Delta u)vec\left(G^{j+1}\right)={\pi }_j$$

(34)

where

$$\begin{array}{cc}\hfill & {\rho }_j(X,\Delta u)={\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]}^T\otimes {\left[\begin{array}{c}\overline{X}\left(k\right)\\ \Delta u\left(k\right)\end{array}\right]}^T\hfill \\ \hfill & {\pi }_j=\left\{{\left[\begin{array}{c}\overline{X}(k+1)\\ -K_j\overline{X}(k+1)\end{array}\right]}^T\otimes {\left[\begin{array}{c}\overline{X}(k+1)\\ -K_j\overline{X}(k+1)\end{array}\right]}^T\right\}\hfill \\ \hfill & \times vec\left(G^j\right)+\left({\overline{X}}^T\left(k\right)\otimes {\overline{X}}^T\left(k\right)\right)vec\left(Q_a\right)\hfill \\ \hfill & +\left(\Delta u^T\left(k\right)\otimes \Delta u{\left(k\right)}^T\right)vec\left(R\right).\hfill \end{array}$$

(35)

For a given positive integer N, denote $\Xi =[{\rho }_1^T,{\rho }_2^T,\cdots ,{\rho }_N^T]$ and $\Theta =[{\pi }_1^T,{\pi }_2^T,\cdots ,{\pi }_N^T]$. Then by collecting sample data, we have

$${\Xi }^T\mathrm{vec}\left(G^{j+1}\right)=\Theta .$$

(36)

By using the least squares method, unknown parameter vectors can be calculated by

$$\mathrm{vec}\left(G^{j+1}\right)={(\Xi {\Xi }^T)}^{-1}\Theta .$$

(37)

Remark 2. 

Note that the matrix Ξ is used in each iteration, so the measured data can be fully utilized. Thus, we should collect enough data so that the rank of Ξ is equal to the unknown independent parameter possessed by the vector G, ensuring that Equation (37) has a unique solution. The rank condition is bound up with the persistence of excitation or exploratory signals, which ensures that enough information about system dynamics is collected.

On the other hand, the convergence of the least squares solution depends on the convexity of the problem and the initial conditions of the iterative algorithm. When the least squares problem is convex, the iterative method usually converges to the optimal solution. If it is a non-convex problem, there may be convergence difficulties. By adding constraints, such as adjusting the threshold and the maximum number of iterations, the convergence of the non-convex problem can be improved.

Now, a data-driven algorithm is developed for solving K in (30), referred to as Algorithm 2.   

Algorithm 2 Data-driven algorithm for solving K

1.  Set $K_0$ and $j=0$, where j denotes iteration index. 2.  Calculate $G^{j+1}$ by (37). 3.  Controller update $$\begin{array}{ccc}& & \Delta u{\left(k\right)}^{j+1}=-K^{j+1}\overline{X}\left(k\right),\hfill \\ & & K_{j+1}={\left(G_{22}^{j+1}\right)}^{-1}{\left(G_{12}^{j+1}\right)}^T.\hfill \end{array}$$ 4.  Set $j=j+1$ and repeat step 2–3 until $∥G^j-G^{j-1}∥\le \epsilon$ with a small constant $\epsilon$$(\epsilon >0)$. Output $K_{j+1}$ as the optimal controller gain.

Remark 3. 

Algorithm 2 is a DDC method. The algorithm is implemented using past input, output, and preview reference data measured by the system, which does not require any priori knowledge of the system dynamics.

Remark 4. 

Algorithm 2 is a DD-PRC algorithm for linear discrete-time systems. According to the data-driven robust fault-tolerant control studied in [15], Algorithm 2 can be extended to nonlinear systems by using the Markov parameter sequence identification method.

4. Simulation Results

In this section, the effectiveness of the proposed algorithm is verified by a simulation example.

Consider the following linear discrete system:

$$\left\{\begin{array}{c}x(k+1)=\left[\begin{array}{cc}0.6& 0.1\\ 0.45& 0.3\end{array}\right]x\left(k\right)+\left[\begin{array}{c}0.3\\ 0.2\end{array}\right]u\left(k\right),\hfill \\ y\left(k\right)=\left[12\right]x\left(k\right),\hfill \end{array}\right.$$

(38)

where $x\left(0\right)={[1,-1]}^T$. Assume that the reference signal with period $L=5$ is given by

$$\begin{array}{c}\hfill r\left(k\right)=sin\left({\displaystyle \frac{2\pi }{5}}k\right)+0.25sin\left({\displaystyle \frac{4\pi }{5}}k\right)+0.5sin\left({\displaystyle \frac{6\pi }{5}}k\right).\end{array}$$

The performance index is denoted as (4) with $Q=5$ and $R=1$. Set the preview length to $M=3$.

By some calculation, we can find that $(A,B)$ is controllable and $(A,C)$ is observable. By solving the ARE (14), we obtain the optimal control gain:

$$\begin{array}{c}\hfill K^{*}=[0.23100.0050-0.0637-0.2180-1.1284\\ \hfill 1.51941.27770-1.1284-0.2180-0.0637].\end{array}$$

Set $K_0=0,$$\epsilon ={10}^{-4}.$ Using Algorithm 1 to solve for K, at $j=6$th iteration, we get the optimal control gain $K_6^1=K^{*}$. While using Algorithm 2 to solve for K, at $j=6$th iteration, we get

$$\begin{array}{c}\hfill K_6^2=[0.23110.0051-0.0636-0.2179-1.1283\\ \hfill 1.51921.27750-1.1283-0.2178-0.0637].\end{array}$$

Altough the $K_6^1$ obtained by Algorithm 1 is more accurate, it depends on the prior information of system matrices. On account of the inevitability of parametric variations or unmodeled dynamics, it is usually hard to obtain the perfect model of a physical system. In the next section, we will use $K_6^2$ to perform the simulation.

The simulation results are given in Figure 2, Figure 3 and Figure 4. Figure 2 shows the norm of the difference between the controller gain K and the optimal tracking controller gain $K^{*}$ during the iteration process. Figure 3 shows the system output $y\left(k\right)$ and the reference signal $r\left(k\right)$ under the data-driven PRC law (13) at $j=6$th iteration. The tracking error is depicted in Figure 4. It can be seen from these Figures that the data-based algorithm proposed in this paper is capable of realizing tracking control tasks successfully.

To test the robustness, we compared the simulation results between our approach and the Q-learning method proposed in [24]. Add a disturbance signal $d\left(k\right)=sin\left({\displaystyle \frac{2\pi }{5}}k\right)$ to the input signal of the control system. The simulation result is shown in Figure 5. It can be seen that both the outputs can track the reference signal accurately. However, DD-PRC can more effectively reduce the tracking error. The reason is the combination of preview and repetitive learning.

Furthermore, Gaussian white noise is also added to the disturbance term $d\left(k\right)$. Noise with varying variances is selected, and 50 Monte Carlo experiments are conducted for each variance level. This process yields the mean and standard deviation (SD) of the sum square error (SSE) for the two controllers under different noise intensities, as presented in

Table 1. The comparison of SSE with different controller.

Controller	SSE	SD of SSE
Q-learning method [24]	1.9875	0.0354
DD-PRC	1.5846	0.0338

. As is evident from Table 1, we can conclude that the DD-PRC method has superior anti-interference ability compared to the Q-learning method proposed in [24].

5. Conclusions

In this paper, a data-based PRC algorithm for addressing the tracking control problem of linear discrete-time systems is designed. The algorithm transforms the system’s optimal tracking problem into a linear quadratic regulation problem. A data-driven approach is proposed without relying on system dynamics, enabling the resolution of LQT using solely measured input, output, and reference trajectory data. Finally, MATLAB R2022a simulation results demonstrate that the proposed data-driven PRC algorithm exhibits satisfactory tracking performance.