Adaptive Finite-Time-Based Neural Optimal Control of Time-Delayed Wheeled Mobile Robotics Systems

Abstract

For nonlinear systems with uncertain state time delays, an adaptive neural optimal tracking control method based on finite time is designed. With the help of the appropriate LKFs, the time-delay problem is handled. A novel nonquadratic Hamilton–Jacobi–Bellman (HJB) function is defined, where finite time is selected as the upper limit of integration. This function contains information on the state time delay, while also maintaining the basic information. To meet specific requirements, the integral reinforcement learning method is employed to solve the ideal HJB function. Then, a tracking controller is designed to ensure finite-time convergence and optimization of the controlled system. This involves the evaluation and execution of gradient descent updates of neural network weights based on a reinforcement learning architecture. The semi-global practical finite-time stability of the controlled system and the finite-time convergence of the tracking error are guaranteed.

1. Introduction

Adaptive intelligent control algorithms have developed rapidly with the advancement of intelligent approximation technology, especially in neural networks (NN) and fuzzy logic systems (FLS), and have achieved a series of excellent research results [1,2,3,4,5,6,7,8,9]. This has also significantly motivated many scholars to explore the adaptive control algorithm, laying a solid foundation for using the corresponding control theory algorithm in the field of practical engineering applications.

Considering that control and decision-making problems are essentially optimization problems, and optimal control plays a key role in engineering applications, the research on intelligent majorization control algorithms in this paper has a certain role in promoting practical engineering applications. In view of the importance of optimal control, many scholars have conducted extensive research on optimal control algorithms and have obtained certain achievements, mainly including two optimization methods [10,11], adaptive dynamic programming (ADP) methods [12] and reinforcement learning (RL) methods [13].

The ADP approach could realize the online approximation of the optimal target by the recursive numerical method without relying on the control algorithm of the model [14,15,16,17,18,19]. Using NNs, the performance function, designed control laws, and the uncertain part of the nonlinear system could be approximated, which helps solve the HJB function; then, the optimal stability is guaranteed. Similar to the learning mechanism of mammals, the reinforcement learning mechanism aims to regulate both the critic and action adaptive laws in order to control the long-term interaction cost of the environment. The action NNs could modify the action laws, while the critic NNs reduce the virtual energy of the long-term storage function. Thanks to the interoperability of the operating mechanism, refs. [20,21,22,23,24] have made outstanding contributions to online optimization control and model-free optimization control.

Although previous ADP-based methods perform well for non-linear systems without time delays, achieving the ideal control effect on time-delayed non-linear systems is often challenging. Therefore, research on this topic has generated interest among experts and scholars and has achieved preliminary results. However, the time delay in the form of nonlinear interference is a major obstacle to applications of control theory algorithms. Some scholars have paid attention to this and have achieved certain results. Regarding existing methods, there are two main forms of system delay: state and input [25].

State time delays are mainly found in intricate engineering systems, for example, wheeled mobile robot (WMR) systems and chemical engineering, which are hysteresis induced by internal propagation of signals during system motion. With assistance from the Lyapunov–Krasovskii functional (LKF) [25,26,27], the influence caused by the state time delay is overcome, and superior control algorithms are designed.

Due to the contradiction between the convergence characteristics of existing optimization algorithms in infinite time and the fast convergence requirements of actual engineering systems, it has greatly inhibited the practical application and promotion of intelligent optimization algorithms. Therefore, in recent years, some scholar award research has focused on the study of convergence speed and convergence domain equilibrium. The existing breakthrough theoretical research results on infinite convergence algorithms [28,29,30,31,32,33] have promoted the research process of finite-time convergence control algorithms to a certain extent and also reflect the necessity of studying the algorithm from the side. At the same time, they also point out the key and difficult issues faced by finite-time convergence control research.

To meet the finite-time or finite-horizon domain convergence characteristics of actual engineering requirements, some scholars have begun relevant research. For nonlinear discrete systems, researchers use the ADP-based approach to solve the finite-time domain convergence problem [34,35], which greatly stimulates the authors’ research passion for finite-time convergence optimization control algorithms.

Different from finite-horizon convergence, besides guaranteeing the time domain of system convergence, finite-time convergence also increases the speed and accuracy of system convergence. However, the existing research is not perfect and is still in its infancy, but some studies with excellent performance have been obtained [36,37,38,39,40,41,42,43,44,45]. Up to now, the finite-time optimization algorithm, considering both convergence speed and convergence precision as well as considering energy consumption, is basically absent. Therefore, based on the previous research, this paper not only considers the state delay, but also considers the input delay, and uses the ADP method to effectively resolve the finite-time optimal tracking control problem of the controlled target.

An adaptive finite-time online optimal tracking control method based on neural networks is designed for uncertain nonlinear systems with state time delays. Firstly, the initial nonlinear system is extended to an augmentation system, which contains tracking error and target expectation information, and a novel discounted performance function is presented. Secondly, a Hamiltonian function is constructed, and the appropriate LKFs are used to resolve the problem of state delay. Then, for the solution of the ideal HJB function, this paper introduces the method of integral reinforcement learning (IRL). Finally, by designing the optimal control strategy and optimizing the control adaptive law, the semi-global practical finite-time stability (SGPFS) lemma, not only is the influence of time delays eliminated, but the stability of uncertain nonlinear systems is guaranteed. The main innovative work includes:

(1)

The time-delay effect is incorporated into the strategy design process to address the finite-time convergence issues.

(2)

The problem caused by the state time delay is solved simultaneously in the optimal control process.

(3)

The optimal control policy guarantees that the target control system achieves optimal control within a finite time.

2. System Description and Preliminaries

Considering the state time-delayed nonlinear system as

$$\dot{\beta }\left(t\right)=p\left(t\right)\beta \left(t\right)+h\left(\beta \left(t-t_1\right)\right)+g\left(t\right)u\left(t\right)+\omega \left(t\right)$$

(1)

where the delayed dynamics $h\left(\beta \left(t-t_1\right)\right)$ is one known function vector with an unknown time delay $t_1$. For the sake of simplicity in subsequent expressions, except for the hysteresis term $\beta \left(t-t_1\right)$, $t$ and other variables are omitted. $g\left(t\right)$ denotes the input function, $p\left(t\right)$ denotes the state function, $u\left(t\right)$ denotes the system control input, and $\omega \left(t\right)$ denotes the external perturbation function.

Considering the state and the input time delays in system (1), the appropriate LKF is introduced to deal with the state time-delay problem, respectively. And according to Remark 1 in [26], only when the delayed dynamics $\alpha \left(t\right)$ are known, one can obtain $h\left({\beta }_1\right)-h\left({\beta }_2\right)={\left.\partial h\left(\beta \right)/\partial \beta \right|}_{\beta ={\beta }^{\kappa }}\left({\beta }_1-{\beta }_2\right)$ with ${\beta }^{\kappa }=\kappa {\beta }_1+\left(1-\kappa \right){\beta }_2$ and $0<\kappa <1$.

The following scientific assumptions are made, and corresponding lemmas are given to ensure that the subsequent design process achieves the expected control objectives.

Assumption 1.

Both function $p\left(t\right)$ and $g\left(t\right)$ are continuously differentiable. For the time-delay function $p\left(\cdot \right)$, its Jacobi matrix $\partial p\left(\beta \right)/\partial \beta$ satisfies the Lipchitz condition $‖\partial p\left(\beta \right)/\partial \beta ‖\le \eta$ with $\eta \ge 0$.

Assumption 2.

The boundedness of the unknown input transfer function $g\left(t\right)$ can be obtained as $\underset{\_}{g}<g\le \overline{g}$. Similarly, ${\sigma }_{\min }\le ‖\sigma \left(\cdot \right)‖\le {\sigma }_{\max }$; ${\phi }_{\min }\le ‖\phi \left(\cdot \right)‖\le {\phi }_{\max }$ can be used to present the boundedness of the activation functions in hidden layers of NNs $\phi \left(\cdot \right)$ and the functional approximation error $\sigma \left(\cdot \right)$.

Lemma 1

([44]). For any states $y_i\in R$,  $i=1,2,\dots ,m$, if the positive constant satisfies $0<q<1$, we have

$${\left({\displaystyle \sum _{i=1}^m\left|y_i\right|}\right)}^q\le {{\displaystyle \sum _{i=1}^m\left|y_i\right|}}^q\le m^{1-q}{\left({\displaystyle \sum _{i=1}^m\left|y_i\right|}\right)}^q$$

(2)

Lemma 2

([39]). For the nonlinear system $\dot{x}=f\left(x\right)$, if (3) holds,

$$\dot{L}\left(x\right)\le -\iota L^b\left(x\right)+\sigma ,t\ge 0$$

(3)

where $L\left(x\right)$ is a smooth positive definite function, $\iota >0$, $0<b<1$, $\sigma >0$, one can further obtain that the nonlinear system $\dot{x}=f\left(x\right)$ is SGPFS.

In this paper, by designing an adaptive NN-based optimal controller $u\left(t\right)$ such that $\beta \left(t\right)$, the output of the system could track ${\beta }_d\left(k\right)$ well in a finite time. The two main types of neural networks used in this paper include critic neural networks and action neural networks. The critic neural network is used for the estimation of the long-term utility function, while the action neural network is used to ensure the stability of the system and the solution of the optimal control inputs of the system.

3. Controller Design and Stability Analysis

This section is divided into subheadings, which provide a concise and precise description of the experimental results and their interpretation, as well as the experimental conclusions that can be drawn. Depicted in Figure 1, in this section, we design an optimal controller, which ensures the optimal control of the system and converges within a finite time. By transforming the initial system into an augmented system, which tracks errors and targets expected information, a novel discounted performance function is presented. Furthermore, a Hamiltonian function is constructed, and the time-delay problem will be solved by using the appropriate LKFs. Then, by introducing the IRL method to the Hamiltonian function, a finite-time optimal tracking controller based on neural networks is designed. Finally, the adaptive law of the appropriate evaluator and the adaptive law of the action NN are designed; the target system’s SGPFS can be ensured.

Figure 1. Finite time convergence adaptive optimal tracking control algorithm structure drawing.

3.1. System Transformation

Considering the nonstrict nonlinear system (1), we developed a controller using a neural network to enable the system to follow the desired trajectory. Firstly, the tracking error system can be design as

$$z\left(t\right)=\beta \left(t\right)-{\beta }_d\left(t\right)$$

(4)

Then, find the (4) derivative, and we can obtain

$$\dot{z}=p\left(t\right)\beta \left(t\right)+h\left(\beta \left(t-t_1\right)\right)+g\left(t\right)u\left(t\right)+\omega \left(t\right)\left(t\right)-{\dot{\beta }}_d$$

(5)

Assumption 3.

The target-given trajectory ${\beta }_d$ with the initial state as ${\beta }_d\left(0\right)=0$ is bounded, and ${\dot{\beta }}_d\left(t\right)$ can be rewritten into the form of (6) by a command generator function that satisfies the Lipschitz continuity property.

$${\dot{\beta }}_d\left(t\right)=l\left({\beta }_d\left(t\right)\right)$$

(6)

The algorithm is expected to adopt a new type of discounted performance function, which includes both tracking error terms and expected trajectories and time-delay terms. Therefore, we constructed the following widening system.

$$\dot{\psi }\left(t\right)=F\left(\psi \left(t\right)\right)+H\left(\psi \left(t-t_1\right)\right)+G\left(t\right)u\left(t\right)+W\left(t\right)$$

(7)

where $\psi \left(t\right)=\left[z\left(t\right),{\beta }_d\left(t\right)\right]$, $F\left(\psi \left(t\right)\right)=\left[\begin{array}{c}{F}_1\left(t\right)\left(z+{\beta }_d\right)-l\left({\beta }_d\right)\\ l\left({\beta }_d\right)\end{array}\right]$, $H\left(\psi \left(t-t_1\right)\right)=\left[\begin{array}{c}{H}_1\left(z\left(t-t_1\right)+{\beta }_d\left(t-t_1\right)\right)\\ 0\end{array}\right]$, $G\left(t\right)=\left[\begin{array}{c}{G}_1\left(t\right)\\ 0\end{array}\right]$, $W\left(t\right)=\left[\begin{array}{c}{D}_1\left(t\right)\\ 0\end{array}\right]$.

Furthermore, the novel discounted performance function is

$$L_1\left(t\right)={\displaystyle {\int }_t^{t+t_0}{e}^{-\chi \left(\tau -t\right)}\left({\Gamma }^{T}Q\Gamma +U\left(u\right)\right)}d\tau$$

(8)

where $\Gamma ={\left[\psi \left(t\right),\psi \left(t-t_1\right)\right]}^T$, $\chi$ is the discount factor, with $\chi >0$ is a constant, and $Q=\left[\begin{array}{cc}{Q}_1& 0\\ 0& Q_2\end{array}\right]$, where $Q_i$ is a matrix that is positive definite, and $t_1$ satisfies $t\ge t_1$, and the semi-global uniform convergence in (7) can be ensured with $t\ge t_1$.

Based on [46,47,48,49,50,51,52], and taking the input constraints into consideration, the nonquadratic functional is proposed as

$$U\left(u\right)=2{\displaystyle {\int }_0^u{\left(\epsilon {\hspace{0.17em}\tanh }^{-1}\left(v/\epsilon \right)\right)}^{T}R{d}_v}$$

(9)

where $\epsilon$ is the saturation input, and $R=diag\left(r_1,r_2\right)$, $U\left(u\right)$ is a non-quadratic matrix.

3.2. Virtual Control

In this part, based on the Hamiltonian function, which is established based on the discounted performance function, the virtual optimal controller $u^{*}\left(t\right)$ will be designed.

To obtain the tracking Bellman equation, we used the Leibniz rule and (9) to obtain

$${\dot{L}}_1=\chi L_1-\left(1-e^{-\chi t_0}\right)\left[{\Gamma }^{T}Q\Gamma -2{\displaystyle {\int }_0^u{\left({\epsilon }_0{\hspace{0.17em}\tanh }^{-1}\left({v/\epsilon }_0\right)\right)}^{T}R{d}_v}\right]$$

(10)

Then, we moved the right-hand side of Equation (10) to the left-hand side of the equation and substituted it into Equation (8) to finally obtain Equation (11)

$$\begin{gathered} V=\left(1-e^{-\chi t_0}\right)\left[{\Gamma }^{T}Q\Gamma +2{\displaystyle {\int }_0^u{\left(\epsilon {\hspace{0.17em}\tanh }^{-1}\left(v/\epsilon \right)\right)}^{T}R{d}_v}\right] \\ -\chi L_1+\frac{\partial L_1}{\partial \Gamma }\left(F\left(\psi \right)+Gu+H\left(\psi \left(t-t_1\right)\right)+W\right)=0 \end{gathered}$$

(11)

In addition, we designed the optimal cost function in (12) from

$$L_1^{*}\left(t\right)=\underset{u}{\min }{\displaystyle {\int }_t^{t+t_0}{e}^{-\chi \left(\tau -t\right)}\left({\Gamma }^{T}Q\Gamma +U\left(u\right)\right)}d\tau$$

(12)

the following conditions should be guaranteed

$$\begin{gathered} V^{*}=\left(1-e^{-\chi t_0}\right)\left[{\Gamma }^{T}Q\Gamma +2{\displaystyle {\int }_0^{u^{*}}{\left(\epsilon {\hspace{0.17em}\tanh }^{-1}\left(v/\epsilon \right)\right)}^{T}R{d}_v}\right] \\ -\chi L_1+\frac{\partial L_1^{*}}{\partial \Gamma }\left(F\left(\psi \right)+G{u}^{*}+H\left(\psi \left(t-t_1\right)\right)+W\right)=0. \end{gathered}$$

(13)

Based on (11) and [53], and the finite-time convergence theory [34], the optimal control input defined as

$$u=-\epsilon \hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\frac{\epsilon {\mathrm{r}}^{-1}{G}^T}{2\left(1-e^{-\chi t_0}\right)}\frac{\partial L_1}{\partial \mathsf{\Psi }}\right).$$

(14)

According to (13) and [44,53], the ideal optimal control input is abbreviated as

$$u^{*}=-\epsilon \hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \mathsf{\Psi }}\right)$$

(15)

where $l_1=\frac{R^{-1}}{2\epsilon }$ is a constant greater than zero and $\Xi =\frac{{\mathrm{r}}^{-1}{G}^T}{2\epsilon \left(1-e^{-\chi t_0}\right)}$.

Then, together with (14), (9) can be written as the following form

$$\begin{gathered} U\left(u^{*}\right)=2{\epsilon }^2{\left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right)}^T\hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right) \\ +{\epsilon }^2{R}_0\ln \left(E_0-{\tanh }^2\left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right)\right) \end{gathered}$$

(16)

where $R_0=\left[R_1,\dots ,R_m\right]$ and $E_0=\underset{m}{\underbrace{\left[1,\dots ,1\right]}}$.

The Hamiltonian function can be written as the following form

$$\begin{gathered} V^{*}=-\chi L_1^{*}+\left(1-e^{-\chi t_0}\right){\psi }^{T}Q\psi +2{\epsilon }^2\left(1-e^{-\chi t_0}\right)l_1{\psi }^{2b-1}\hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right) \\ +\frac{\partial L_1^{*}}{\partial \Gamma }\left(F\left(\psi \right)+H\left(\psi \left(t-t_1\right)\right)+W\right) \\ +{\epsilon }^2{R}_0\ln \left(E_0-{\tanh }^2\left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right)\right)=0 \end{gathered}$$

(17)

Furthermore, (17) can be written as

$$\begin{gathered} \begin{array}{cc}\hfill V^{*}& =2{\epsilon }^2\left(1-e^{-\chi t_0}\right)l_1{\psi }^{2b-1}\hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right)\hfill \\ & +\frac{\partial L_1^{*}}{\partial \Gamma }\left[F\left(\psi \right)+H\left(\psi \right)+W-\left(H\left(\psi \right)-H\left(\psi \left(t-t_1\right)\right)\right)\right]\hfill \end{array} \\ -\chi L_1^{*}+\left(1-e^{-\chi t_0}\right){\Gamma }^{T}Q\Gamma +{\epsilon }^2{R}_0\ln \left(E_0-{\tanh }^2\left(l_1{\psi }^{2b-1}+\Xi \frac{\partial L_1^{*}}{\partial \Gamma }\right)\right)=0 \end{gathered}$$

(18)

To deal with the challenges brought by online tracking control, the optimal value $L_1^{*}$ should be solved using (17). Furthermore, the optimal control policy $u(L_1^{*})$ is shown in (14).

3.3. State Time Delay

Choosing the appropriate LKFs solves the problem caused by the state time delay, which laid the foundation for the application of the IRL algorithm.

According to Assumption 2 in [36] and Remark 5 in [15], the IRL method can be used to solve the $L_1^{*}$, only when the function ${\Theta }_1\left(t\right)=F\left(\psi \right)+H\left(\psi \right)$, ${\Theta }_2\left(t\right)=H\left(\psi \right)-H\left(\psi \left(t-t_1\right)\right)$ and $G$ satisfy that

$$‖F\left(\psi \right)+H\left(\psi \right)‖\le b_1‖\psi ‖$$

(19)

$$‖H\left(\psi \right)-H\left(\psi \left(t-t_1\right)\right)‖\le {\displaystyle \sum _{\theta =1}^n{b}_{2,\theta }‖\psi \left(t-\theta \Delta t\right)‖}$$

(20)

$$‖G‖\le b_3$$

(21)

where $b_1$, $b_{2,\theta }$, and $b_3$ are positive constants, with $0<\theta \le n$ and $\Delta t=t_1/n$.

Considering the function $F\left(\psi \right)$ and the known function $H\left(\psi \right)$ satisfy the Lipchitz condition, and Assumption 2, (19) and (21) can be guaranteed. However, the state time delay $t_1$ is uncertain, and the boundedness of (20) cannot be obtained. In addition, because of the uncertain state time delay, the uncertain function $H\left(\psi \left(t-t_1\right)\right)$ cannot be approximated using NN.

In order to better complete the controller design, the problem caused by the state time delay will be handled first. Defining the new function ${\Theta }_2\left(t\right)$ as

$${\Theta }_2\left(t\right)=H\left(\psi \right)-H\left(\psi \left(t-t_1\right)\right)$$

(22)

and (22) can be written in the following form

$$\begin{gathered} \begin{array}{cc}\hfill {\Theta }_2\left(t\right)=& H\left(\psi \right)-H\left(\psi \left(t-\Delta t\right)\right)+H\left(\psi \left(t-\Delta t\right)\right)\hfill \\ & -H\left(\psi \left(t-2\Delta t\right)\right)+H\left(\psi \left(t-2\Delta t\right)\right)-H\left(\psi \left(t-3\Delta t\right)\right)+\dots \hfill \end{array} \\ -H\left(\psi \left(t-\theta \Delta t\right)\right)+H\left(\psi \left(t-\left(\theta +1\right)\Delta t\right)\right)-\dots \\ +H\left(\psi \left(t-\left(n-1\right)\Delta t\right)\right)-H\left(\psi \left(t-t_1\right)\right) \end{gathered}$$

(23)

where $\Delta t=t_1/n$, and both $i$ and $n$ are positive integers.

By Assumption 1, the mean-value theorem is introduced to $H\left(\psi \right)$. Therefore, one obtains

$$\Delta \psi \left(t-\Delta t\right)=H\left(\psi \left(t\right)\right)-H\left(\psi \left(t-\Delta t\right)\right)={\left.\frac{\partial H\left(\psi \right)}{\partial \psi }\right|}_{\psi ={\psi }^{\eta }}\left(\psi \left(t\right)-\psi \left(t-\Delta t\right)\right)$$

(24)

where ${\psi }^{\eta }=\eta \psi \left(t\right)+\left(1-\eta \right)\psi \left(t-\Delta t\right)$, $0<\eta <1$.

The error function caused by $\Delta t$ can be obtained as

$$\Delta \psi \left(t-\left(\theta +1\right)\Delta t\right)={\left.\frac{\partial H\left(\psi \right)}{\partial \psi }\right|}_{\psi ={\psi }^{\eta }}\left(\psi \left(t-\theta \Delta t\right)-\psi \left(t-\left(\theta +1\right)\Delta t\right)\right)$$

(25)

Defining the augmented system states as

$$\Delta \overline{\psi }=\left[\Delta \psi \left(t\right),\dots ,\Delta \psi \left(t-\theta \Delta t\right)\dots ,\Delta \psi \left(t-t_1\right)\right]$$

(26)

then, we can write system (24) as follows

$$\Delta \overline{\dot{\psi }}\left(t\right)=\mathsf{\Pi }\left(\Delta \overline{\psi }\left(t\right)\right)$$

(27)

To guarantee that system (27) is uniformly ultimately bounded (UUB), the following lemma is proposed.

Lemma 3.

If the dimension of the state vector matches that of the function $\mathsf{\Pi }\left(\Delta \psi \right)$, where $\mathsf{\Pi }\left(\Delta \psi \left(0\right)\right)=0$.

$$\Delta \dot{\psi }\left(t\right)=\mathsf{\Pi }\left(\Delta \psi \left(t\right)\right)$$

(28)

converges to a compact set exponentially, where the Lyapunov function satisfies

$$c_1{‖\Delta \psi \left(t\right)‖}^2\le L_0\left(\Delta \psi \left(t\right)\right)\le c_2{‖\Delta \psi \left(t\right)‖}^2+c_3$$

(29)

$${\dot{L}}_0\left(\Delta \psi \left(t\right)\right)\le -c_4{‖\Delta \psi \left(t\right)‖}^2+c_5$$

(30)

where $c_i>0$, $\theta =1,2,\dots ,5$. In addition, the uniformly ultimately boundedness of system (28) can be guaranteed.

Proof. 

Inspired by the research in [26], the following proof process is given. Defining the initial state of (28) as $\Delta \dot{\psi }\left(0\right)=\Delta \psi \left(t\right)$, we obtain

$$\left\{\begin{array}{l}\Delta \dot{\psi }\left(t\right)=\mathsf{\Pi }\left(\Delta \dot{\psi }\left(t-\Delta t\right)\right)=\mathsf{\Pi }\left(\Delta \psi \left(t\right)\right)\\ \Delta \dot{\psi }\left(t-\Delta t\right)=\mathsf{\Pi }\left(\Delta \dot{\psi }\left(t-2\Delta t\right)\right)={\mathsf{\Pi }}^2\left(\Delta \psi \left(t\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}⋮\\ \Delta \dot{\psi }\left(t-\theta \Delta t\right)=\mathsf{\Pi }\left(\Delta \dot{\psi }\left(t-\left(n-\theta -1\right)\Delta t\right)\right)={\mathsf{\Pi }}^{\theta +1}\left(\Delta \psi \left(t\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}⋮\\ \Delta \dot{\psi }\left(t-\left(n-1\right)\Delta t\right)=\mathsf{\Pi }\left(\Delta \dot{\psi }\left(t-t_1\right)\right)={\mathsf{\Pi }}^n\left(\Delta \psi \left(t\right)\right)\end{array}\right.$$

(31)

If the system exponentially converges to a compact set, then

$$‖{\mathsf{\Pi }}^{\theta }\left(\Delta \psi \left(t\right)\right)‖\le a_1{a}_2^{\theta }‖\Delta \psi \left(t\right)‖+a_3$$

(32)

where positive constants satisfy $a_1>0$, $0<a_2<1$, and $a_3\ge 0$.

Furthermore, the Lyapunov function is

$$\Delta L_0={\displaystyle \sum _{i=0}^{n-1}{\left[{\mathsf{\Pi }}^{\theta }\left(\Delta \psi \left(t\right)\right)\right]}^T\left[{\mathsf{\Pi }}^{\theta }\left(\Delta \psi \left(t\right)\right)\right]}.$$

(33)

Submitting (32) to (33), we obtain

$$\Delta L_0\le 2a_1^2{\displaystyle \sum _{i=0}^{n-1}{a}_2^{2\theta }}{‖\Delta \psi \left(t\right)‖}^2+2n{a}_3^2\le 2a_1^2{\displaystyle \sum _{i=0}^{n-1}{‖\Delta \psi \left(t\right)‖}^2}+2n{a}_3^2.$$

(34)

Considering the fact that $0<a_2<1$, we obtain

$$c_1{\displaystyle \sum _{i=0}^{n-1}{‖\Delta \psi \left(t\right)‖}^2}\le \Delta L_0\left(\Delta \psi \left(t\right)\right)\le c_2{\displaystyle \sum _{i=0}^{n-1}{‖\Delta \psi \left(t\right)‖}^2}+c_3$$

(35)

where $c_1=1$, $c_2=2a_1^2\frac{1-a_2^{2n}}{1-a_2^2}$, $c_3=2n{a}_3^2$.

Moreover, we can obtain

$$\begin{gathered} \Delta {\dot{L}}_0\left(\Delta \psi \left(t\right)\right)=\frac{1}{\Delta t}\left[\Delta L_0\left(\Delta \psi \left(t+\Delta t\right)\right)-\Delta L_0\left(\Delta \psi \left(t\right)\right)\right] \\ =\frac{1}{\Delta t}\left[{\displaystyle \sum _{i=1}^n{\left[{\mathsf{\Pi }}^i\left(\Delta \psi \left(t\right)\right)\right]}^2}-{\displaystyle \sum _{i=0}^{n-1}{\left[{\mathsf{\Pi }}^i\left(\Delta \psi \left(t\right)\right)\right]}^2}\right] \\ \le \frac{1}{\Delta t}\left[{\left[a_1{a}_2^n‖\Delta \psi \left(t\right)‖+a_3\right]}^2-{‖\Delta \psi \left(t\right)‖}^2\right] \\ \le -\left(1-2a_1^2{a}_2^{2n}\right)\frac{1}{\Delta t}{‖\Delta \psi \left(t\right)‖}^2+2a_3^2\frac{1}{\Delta t} \\ \le -c_4{‖\Delta \psi \left(t\right)‖}^2+c_5 \end{gathered}$$

(36)

where $c_4=\left(1-2a_1^2{a}_2^{2n}\right)/\Delta t$ and $c_5=2a_3^2/\Delta t$. If a sufficiently small-time interval $\Delta t$ is chosen, $n$ will be sufficiently large to ensure that $c_4$ is a positive constant.

Based on, (35) and (36), one has

$$\begin{array}{cc}\hfill \Delta {\dot{L}}_0\left(\Delta \psi \left(t\right)\right)& \le -\left(1-2a_1^2{a}_2^{2n}\right){‖\Delta \psi \left(t\right)‖}^2+2a_3^2\hfill \\ & \le -\frac{\left(1-2a_1^2{a}_2^{2n}\right)\left(1-a_2^2\right)}{2a_1^2\left(1-a_2^{2n}\right)}{L}_0\left(\Delta \psi \left(t\right)\right)\hfill \\ & +\frac{2\left(1-a_2^2\right)\left(1-2a_1^2{a}_2^{2n}\right)n{a}_3^2}{2a_1^2\left(1-a_2^{2n}\right)}+2a_3^2\hfill \\ & =-b_1\Delta L_0\left(\Delta \psi \left(t\right)\right)+b_2\hfill \end{array}$$

(37)

where

$$b_1=\frac{\left(1-2a_1^2{a}_2^{2n}\right)\left(1-a_2^2\right)}{2a_1^2\left(1-a_2^{2n}\right)}$$

(38)

$$b_2=\frac{2\left(1-a_2^2\right)\left(1-2a_1^2{a}_2^{2n}\right)n{a}_3^2}{2a_1^2\left(1-a_2^{2n}\right)}+2a_3^2$$

(39)

Furthermore, we can obtain the Lyapunov function (23), and guarantee the UUB of system (23), which is composed by n subsystems similar to (28)

$${\dot{L}}_0\left(\Delta \psi \left(t\right)\right)={\displaystyle \sum _{i=1}^n\Delta {\dot{L}}_0\left(\Delta \psi \left(t-\theta \Delta t\right)\right)}$$

(40)

Substituting (36) for (40), one has

$${\dot{L}}_0\left(\Delta \psi \left(t\right)\right)\le -c_4{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}+c_5$$

(41)

Similarly, one has

$${\dot{L}}_0\left(\Delta \psi \left(t\right)\right)\le -b_1{L}_0\left(\Delta \psi \left(t\right)\right)+b_2$$

(42)

when $n$ and $a_1$ are selected large enough, the ultimate boundedness of $L_0\left(\Delta \psi \left(t\right)\right)$ can be assured for any initial condition $L_0\left(\Delta \psi \left(t-t_1\right)\right)$ within a bounded set, guaranteeing the UUB of system states are guaranteed.

The proof is competed. □

3.4. Critic NN and Value Function Approximation

In summary, the boundedness of (19)–(21) can be obtained. The IRL method will be extended to the solution of $L_1^{*}$ in the following section.

When the IRL interval is choose as $T>0$, (8) can be written as the following form

$$L_1\left(t-T\right)=\left(1-e^{-\chi t_0}\right){\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(\tau -\left(t-T\right)\right)}\left({\Gamma }^{T}Q\Gamma +U\left(u\right)\right)}d\tau +e^{-\chi T}{L}_1\left(t\right)$$

(43)

Assuming that (8) is a continuous smooth function, $L_1$ and its gradient $\partial L_1/\partial \psi$ are approximated as

$$L_1={\omega }_c^T{\phi }_c\left(\Gamma \right)+{\epsilon }_c$$

(44)

$$\frac{\partial L_1}{\partial \Gamma }={\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }$$

(45)

where ${\omega }_c\in {ℝ}^{l_c}$ is the constant-target online estimate parameter vector, in which $l_c$ is the quantity of neurons within the neural network, and ${\phi }_c$ and ${\epsilon }_c$ are the activation function of the critic NN and approximate error, respectively.

Assumption 4.

The boundedness of the activation function and assessment of the error of the critic NN and their gradient can be obtained as $‖{\phi }_c‖\le {\overline{\phi }}_c$, and $‖{\epsilon }_c‖\le {\overline{\epsilon }}_c$, $‖\partial {\phi }_c/\partial \mathsf{\Psi }‖\le {\overline{\epsilon }}_{c,0}$ and $‖\partial {\epsilon }_c/\partial \mathsf{\Psi }‖\le {\overline{\epsilon }}_{c,0}$, respectively.

When the IRL interval is $T>0$, the Bellman equation induced in the critic NN estimated value, can be expressed as

$$z_B={\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(\tau -t+T\right)}\left(1-e^{-\chi t_0}\right)\left[{\Gamma }^{T}Q\Gamma +U\left(u\right)\right]}d\tau +{\omega }_c^T\Delta {\phi }_c$$

(46)

where

$$\Delta {\phi }_c=e^{-\chi T}{\phi }_c\left(\Gamma \left(t\right)\right)-{\phi }_c\left(\Gamma \left(t-T\right)\right)$$

(47)

The constraint on (46) can be derived based on Assumption 4, i.e., $‖z_B‖\le {\overline{z}}_B$.

To derive the approximate tracking Bellman function, the approximation of the neural network is evaluated to obtain

$${\widehat{L}}_1={\widehat{\omega }}_c^T{\phi }_c\left(\Gamma \right)$$

(48)

where ${\widehat{\omega }}_c$ is the estimation of the critic law ${\omega }_c$.

Therefore, the estimation of (46) is

$$z_B=R\left(t\right)+{\widehat{\omega }}_c^T\Delta {\phi }_c$$

(49)

where the reinforcement learning reward is denoted by

$$r\left(t\right)={\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(\tau -t+T\right)}\left(1-e^{-\chi t_0}\right)\left[{\Gamma }^{T}Q\Gamma +\widehat{U}\left(u\right)\right]}d\tau .$$

(50)

To reduce the approximation error, we give a function in (51) from

$$Z_B\left(t\right)=1/2z_B^2\left(t\right).$$

(51)

We can obtain the following expression using the Gradient descent method.

$${\dot{\widehat{\omega }}}_c={\alpha }_c\frac{\Delta {\phi }_c\Delta {\phi }_c^T}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}{z}_B$$

(52)

where ${\alpha }_c$ represents the learning rate of the critic neural network.

Considering ${\tilde{\omega }}_c={\omega }_c-{\widehat{\omega }}_c$, (46) and (49), we have

$$z_B={\tilde{\omega }}_c^T\Delta {\phi }_c+{\epsilon }_B$$

(53)

$${\dot{\tilde{\omega }}}_c=-{\alpha }_c\frac{\Delta {\phi }_c\Delta {\phi }_c^T}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}{\tilde{\omega }}_c+{\alpha }_c\frac{\Delta {\phi }_c\Delta {\phi }_c^T}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}{\epsilon }_B$$

(54)

3.5. Action NN and Controller Design

According to (45), the optimal control input, i.e., (14) is as follows

$$u=-\epsilon \hspace{0.17em}\tanh \left[l_1{\psi }^{2b-1}+\Xi \left({\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }\right)\right]$$

(55)

To solve the issue in the tracking HJB induced $\frac{\partial {\epsilon }_c}{\partial \mathsf{\Psi }}$, we obtain

$$\begin{gathered} {\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(v-t+T\right)}{\dot{\phi }}_{c}dv}={\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(v-t+T\right)}\frac{\partial {\phi }_c}{\partial \Gamma }}\left({\Theta }_1+{\Theta }_2+Gu+W\right)dv \\ =\Delta {\phi }_c+\chi {\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(v-t-T\right)}{\phi }_c}dv \end{gathered}$$

(56)

In addition, we obtain

$$\Delta {\phi }_c={\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(v-t+T\right)}\left(\frac{\partial {\phi }_c}{\partial \Gamma }\left({\Theta }_1+{\Theta }_2+Gu+W\right)-\chi {\phi }_c\right)}dv$$

(57)

Equation (16) becomes

$$\begin{gathered} U\left(u\right)={\epsilon }^2{R}_0\ln \left\{E_0-{\tanh }^2\left[l_1{\psi }^{2b-1}+\Xi \left({\omega }_c^T\frac{\partial {\phi }_c}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }\right)\right]\right\} \\ -2\epsilon \left(l_1{\psi }^{2b-1}+\Xi \left({\omega }_c^T\frac{\partial {\phi }_c}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }\right)\right)u \end{gathered}$$

(58)

Then, (46) can be rewritten as

$$\begin{array}{l}{\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(\tau -t+T\right)}\left\{\left(1-e^{-\chi t_1}\right){\Gamma }^{T}Q\Gamma -\chi {\omega }_c^T{\phi }_c\right.}\\ +2{\epsilon }^2\left(1-e^{-\chi t_1}\right)l_1{\psi }^{2b-1}\hspace{0.17em}\tanh \left[l_1{\psi }^{2b-1}+\Xi {\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right]+{\epsilon }_{HJB}\\ +{\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\left({\Theta }_1+{\Theta }_2+W\right)\left.+{\epsilon }^2{R}_0\ln \left\{E_0-{\tanh }^2\left[l_1{\psi }^{2b-1}+\Xi \left({\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)\right]\right\}\right\}dv=0\end{array}$$

(59)

where

$$\begin{gathered} {\epsilon }_{HJB}=e^{-\chi \left(\tau -t+T\right)}\left\{-{\epsilon }^2{R}_0\ln \left[E_0-{\tanh }^2\left(l_1{\psi }^{2b-1}+\Xi {\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)\right]\right.-2{\epsilon }^2\left(1-e^{-\chi t_0}\right)l_1{\psi }^{2b-1} \\ \hspace{1em}\hspace{1em}\times \tanh \left[l_1{\psi }^{2b-1}+\Xi {\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right]+2{\epsilon }^2\left(1-e^{-\chi t_0}\right)l_1{\psi }^{2b-1}\hspace{0.17em}\tanh \left[l_1{\psi }^{2b-1}+\Xi \left({\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }\right)\right] \\ \left.-{\omega }_c^T\frac{\partial {\epsilon }_c}{\partial \Gamma }\left({\Theta }_1+{\Theta }_2+W\right)+{\epsilon }^2{R}_0\ln \left[E_0-{\tanh }^2\left(l_1{\psi }^{2b-1}+\Xi \left({\omega }_c\frac{\partial {\phi }_c^T}{\partial \Gamma }+\frac{\partial {\epsilon }_c}{\partial \Gamma }\right)\right)\right]\right\} \end{gathered}$$

(60)

To obtain the limitation of the HJB proximity error, we can use the boundary proximity error. In addition, when NN is selected, the construction cannot be changed. We can only solve this problem by uncertain weights of NN.

Approximating the control input (55) by critic NN, we have

$$u_1=-\epsilon \hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)$$

(61)

where ${\widehat{\omega }}_c$ is the estimated value of ${\omega }_c$.

However, the function of (61) is only to estimate the current critical NN weight, which fails to keep the system (1) stable. Hence, to guarantee the stability of the system and solve the optimal control strategy, we introduce another NN as the action NN.

$${\widehat{u}}_1=-\epsilon \hspace{0.17em}\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_a\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)$$

(62)

where ${\widehat{\omega }}_a$ represents the weight vector of the action neural network, denoting the present evaluation value of ${\omega }_c$.

Then, the interval IRL Bellman equation error is estimated as

$${\widehat{z}}_B={\displaystyle {\int }_{t-T}^t{e}^{-\chi \left(\tau -t+T\right)}\left[{\Gamma }^{T}Q\Gamma +\widehat{U}\left(u\right)\right]}d\tau +{\widehat{\omega }}_c^T\Delta {\phi }_c$$

(63)

where

$$\widehat{U}\left(u\right)=2{\displaystyle {\int }_0^{\widehat{u}}{\left(\epsilon {\hspace{0.17em}\tanh }^{-1}\left(v/\epsilon \right)\right)}^{T}R{d}_v}$$

(64)

Therefore, (52) can be rewritten as

$${\dot{\widehat{\omega }}}_c={\alpha }_c\frac{\Delta {\phi }_c\Delta {\phi }_c^T}{{‖1+\Delta {\phi }_c^T\Delta \phi ‖}^2}{\widehat{z}}_B$$

(65)

defining the input assessment error as

$$z_u={\widehat{u}}_1-u_1=\epsilon \left(\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_a\frac{\partial {\phi }_a^T}{\partial \Gamma }\right)-\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)\right)$$

(66)

To minimize (66), we use the following formula

$$Z_u\left(t\right)=z_u^T\left(t\right)R{z}_u\left(t\right).$$

(67)

We can utilize the gradient descent method to derive the following equation

$${\dot{\widehat{\omega }}}_a=-{\alpha }_a\left[{\Xi }^{\prime }\frac{\partial {\phi }_a^T}{\partial \Gamma }{\mathrm{sech}}^2\left(l_1{\psi }^{2b-1}+\Xi \frac{\partial {\phi }_a^T}{\partial \Gamma }{\widehat{\omega }}_a\right)z_u+\eta {\widehat{\omega }}_a\right]$$

(68)

where ${\Xi }^{\prime }=R\Xi$, and $\eta$ is a positive design variable.

3.6. Stability Analysis

According to the proposed lemmas and assumptions, the following theorem is given to analyze the effectiveness of the proposed algorithm.

Theorem 1.

Based on the definition in [43], Lemma 1–4, Assumptions 1–4, and the design of the proposed control policy (62) and the control laws, (65) and (68), the proposed optimal tracking control algorithm ensures that the partially uncertain nonlinear system (1) is SGPFS.

Proof of Theorem 1.

The candidate function of the Lyapunov function is designed as

$$L\left(k\right)=L_0\left(k\right)+L_1\left(k\right)+L_2\left(k\right)+L_3\left(k\right)+L_4\left(k\right)$$

(69)

where ${\dot{L}}_0\left(\Delta \psi \left(t\right)\right)\le -c_4{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}+c_5$ and $L_1\left(k\right)$ is given in (8) as the optimal value function. Then, the provided expressions are applicable:

$$L_2={\displaystyle \sum _{i=1}^n{\left(\Delta \psi \left(t-\theta \Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-\theta \Delta t\right)\right)}$$

(70)

$$L_3=1/{\alpha }_c{\tilde{\omega }}_c^T{\tilde{\omega }}_c$$

(71)

$$L_4=1/{\alpha }_a{\tilde{\omega }}_a^T{\tilde{\omega }}_a$$

(72)

Based on (24) and (31), the first derivative of $L_2$ can be given as

$$\begin{gathered} {\dot{L}}_2=2{\displaystyle \sum _{j=1}^n\Delta \psi \left(t-i\Delta t\right)Q_3\Delta \dot{\psi }\left(t-i\Delta t\right)} \\ =2{\left(\Delta \psi \left(t-n\Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t\right)\right)+\dots +{\left(\Delta \psi \left(t-\Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-\left(n-1\right)\Delta t\right)\right) \\ +{\left(\Delta \psi \left(t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-n\Delta t\right)\right) \end{gathered}$$

(73)

Using Young’s inequality $\pm a^T{b}_0\le \mu a^{T}a/2+{b_0}^T{b}_0/\left(2\mu \right)$, one has

$$\begin{gathered} \begin{array}{cc}\hfill {\dot{L}}_2& \le \mu {\left(\Delta \psi \left(t-n\Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-n\Delta t\right)\right)+\frac{1}{\mu }{\left(\Delta \psi \left(t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t\right)\right)+\dots \hfill \\ & +\mu {\left(\Delta \psi \left(t-\Delta t\right)\right)}^T\hfill \end{array} \\ \begin{array}{l}\hspace{1em}\times Q_3\left(\Delta \psi \left(t-\Delta t\right)\right)+\frac{1}{\mu }{\left(\Delta \psi \left(t-\left(n-1\right)\Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-\left(n-1\right)\Delta t\right)\right)\\ \hspace{1em}+\mu {\left(\Delta \psi \left(t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t\right)\right)\end{array} \\ +\frac{1}{\mu }{\left(\Delta \psi \left(t-n\Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-n\Delta t\right)\right) \end{gathered}$$

(74)

$${\dot{L}}_2\le c_0{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}$$

(75)

where $c_0=\left(\mu +1/\mu \right)Q_3$.

Based on (44), the first derivative of $L_3$ is

$${\dot{L}}_3={\widehat{\omega }}_c^T\frac{\partial \phi }{\partial \Gamma }\left({\Theta }_1+{\Theta }_2+Gu+W\right)$$

(76)

Considering (57), with the IRL interval chosen small enough, we have ${\rho }_1{\phi }_c={\phi }_c\left(t-T\right)$, ${\rho }_1=1\pm {\rho }_0$, ${\rho }_1\in \mathrm{U}\left(1,{\rho }_0\right)$, and ${\rho }_0$ is a sufficiently small positive constant.

$$\left(\chi +\frac{1}{T}\right)\phi -\frac{{\phi }_c\left(t-T\right)}{T{e}^{-\chi T}}\approx \frac{\partial \phi }{\partial \Gamma }\left({\Theta }_1+{\Theta }_2+Gu+W\right)$$

(77)

Then, (76) can be written as

$$\begin{gathered} {\dot{L}}_3=\left(\chi +\frac{1}{T}\right){\widehat{\omega }}_c^T\phi -\frac{1}{T{e}^{-\chi T}}{\widehat{\omega }}_c^T{\phi }_c\left(t-T\right) \\ \le \left({\left(\chi +\frac{1}{T}\right)}^2+\frac{1}{T^2{e}^{-2\chi T}}\right){\left({\widehat{\omega }}_c^T\phi \right)}^T{\widehat{\omega }}_c^T{\phi }_c \end{gathered}$$

(78)

Based on (54) we have

$${\epsilon }_B=\frac{T}{t_1}{\widehat{\omega }}_c^T{\phi }_c+{\widehat{\omega }}_c^T\left(e^{-\chi T}{\phi }_c-{\phi }_c\left(t-T\right)\right)$$

(79)

Then, the approximate of (54) is

$$\begin{gathered} {\dot{\tilde{\omega }}}_c=-{\alpha }_c\frac{\Delta \phi \Delta {\phi }^T}{{‖1+\Delta {\phi }^T\Delta \phi ‖}^2}{\tilde{\omega }}_c \\ +{\alpha }_c\frac{\Delta \phi }{{‖1+\Delta {\phi }^T\Delta \phi ‖}^2}\left(\frac{T}{t_1}{\widehat{\omega }}_c^T{\phi }_c+{\widehat{\omega }}_c^T\left(e^{-\chi T}{\phi }_c-{\phi }_c\left(t-T\right)\right)\right) \end{gathered}$$

(80)

And, for the first difference of (71)

$$\begin{gathered} {\dot{L}}_3=-\frac{\Delta {\phi }_c\Delta {\phi }_c^T}{{‖1+\Delta {\phi }_c^T\Delta \phi ‖}^2}{\tilde{\omega }}_c^T{\tilde{\omega }}_c \\ +\left({\omega }_c^T-{\widehat{\omega }}_c^T\right)\frac{\Delta {\phi }_c}{{‖1+\Delta {\phi }_c^T\Delta \phi ‖}^2}\left(\frac{T}{t_1}{\widehat{\omega }}_c^T{\phi }_c+{\widehat{\omega }}_c^T\left(e^{-\chi T}{\phi }_c-{\phi }_c\left(t-T\right)\right)\right) \end{gathered}$$

(81)

Using Cauchy’s mean value theorem, (81) changed into

$$\begin{gathered} {\dot{L}}_3=\frac{1}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}-\left[{‖\Delta \phi ‖}^2{\tilde{\omega }}_c^T{\tilde{\omega }}_c^T+{\left({\omega }_c^T\Delta {\phi }_c\right)}^T{\omega }_c^T\Delta {\phi }_c\right. \\ \left.-\frac{1}{2}\left(\frac{T}{t_1}\left(e^{-\chi T}-{\rho }_1\right)-\frac{T^2}{t_1^2}\right){\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c\right] \end{gathered}$$

(82)

Based on (68) and (66), we have

$$\begin{gathered} {\dot{\tilde{\omega }}}_a={\alpha }_a\left[{\Xi }^{\prime }\frac{\partial {\phi }_a}{\partial \Gamma }{\mathrm{sech}}^2\left(l_1{\psi }^{2b-1}+{\Xi }^T\frac{\partial {\phi }_c^T}{\partial \Gamma }{\widehat{\omega }}_a\right)\times \right. \\ \left(\tanh \left(\Xi {\widehat{\omega }}_a\frac{\partial {\phi }_a^T}{\partial \Gamma }+l_1{\psi }^{2b-1}\right)-\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)\right)\left.-\eta {\tilde{\omega }}_a+\eta {\omega }_c\right] \end{gathered}$$

(83)

Then, the first derivative of $L_3$ is

$${\dot{L}}_4=-\eta {\tilde{\omega }}_a^T{\tilde{\omega }}_a+{\tilde{\omega }}_a^T{J}_0$$

(84)

where

$$\begin{gathered} J_0={\Xi }^{\prime }\frac{\partial {\phi }_a}{\partial \Gamma }{\mathrm{sech}}^2\left(l_1{\psi }^{2b-1}+{\Xi }^T\frac{\partial {\phi }_a^T}{\partial \Gamma }{\widehat{\omega }}_a\right)\left[\left(\tanh \left(\Xi {\widehat{\omega }}_a\frac{\partial {\phi }_a^T}{\partial \Gamma }+l_1{\psi }^{2b-1}\right)\right.\right. \\ \left.\left.-\tanh \left(l_1{\psi }^{2b-1}+\Xi {\widehat{\omega }}_c\frac{\partial {\phi }_c^T}{\partial \Gamma }\right)\right)+\eta {\omega }_c\right] \end{gathered}$$

(85)

By using Cauchy’s mean value theorem, we have

$${\dot{L}}_4\le -\left(\eta -1/\left(2{\alpha }_a\right)\right){\tilde{\omega }}_a^T{\tilde{\omega }}_a+\frac{{\alpha }_a}{2}{J}_0^2$$

(86)

Above all, the first difference of (69)

$$\dot{L}\le -c_1{\left({\widehat{\omega }}_c^T\phi \right)}^T{\widehat{\omega }}_c^T\phi -c_2{\tilde{\omega }}_c^T{\tilde{\omega }}_c-c_3{\tilde{\omega }}_a^T{\tilde{\omega }}_a-c_6{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}+J^2$$

(87)

where $c_{\theta }>0$, $\theta =1,2,3$, $c_1=\left(\frac{\frac{T}{t_1}\left(e^{-\chi T}-{\rho }_1\right)-\frac{T^2}{t_1^2}}{2{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}-{\left(\chi +\frac{1}{T}\right)}^2+\frac{1}{T^2{e}^{-2\chi T}}\right)$, $c_2=\frac{{‖\Delta {\phi }_c‖}^2}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}$, $c_3=\left(\eta -\frac{1}{2{\alpha }_a}\right)$, $c_6=c_4-c_0$ and $J^2=\frac{{\left({\omega }_c^T\Delta {\phi }_c\right)}^T{\omega }_c^T\Delta {\phi }_c}{{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2}+c_5+\frac{{\alpha }_a}{2}{J}_0^2$.

To make the finite-time convergence, we deal with the equation and add and subtract several terms on the right side

$$\begin{gathered} \begin{array}{cc}\hfill \dot{L}& \le -c_1{\left({\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c\right)}^{\beta }+c_1{\left({\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c\right)}^{\beta }\hfill \\ & -c_1{\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c-c_2{\left({\tilde{\omega }}_c^T{\tilde{\omega }}_c\right)}^{\beta }+c_2{\left({\tilde{\omega }}_c^T{\tilde{\omega }}_c\right)}^{\beta }\end{array} \\ -c_2{\tilde{\omega }}_c^T{\tilde{\omega }}_c-c_3{\left({\tilde{\omega }}_a^T{\tilde{\omega }}_a\right)}^{\beta }+c_3{\left({\tilde{\omega }}_a^T{\tilde{\omega }}_a\right)}^{\beta }-c_3{\tilde{\omega }}_a^T{\tilde{\omega }}_a-c_6{\left({\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}\right)}^{\beta } \\ +c_6{\left({\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}\right)}^{\beta }-c_6{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}+J^2 \end{gathered}$$

(88)

To make the system (1) stable, and the finite time, we consider Lemma 1. Therefore, the constant must be greater than zero.

$${\alpha }_a>\frac{1}{2\eta }$$

(89)

and there are the following formulas:

$$2{‖1+\Delta {\phi }^T\Delta \phi ‖}^2\left({\left(\chi +\frac{1}{T}\right)}^2-\frac{1}{T^2{e}^{-2\gamma T}}\right)t_1^2-T\left(e^{-\chi T}-{\rho }_1\right)t_1+T^2<0$$

(90)

with

$$e^{-\chi T}>{\rho }_1$$

(91)

$$T\chi +1>\frac{1}{e^{-\chi T}}$$

(92)

then

$$t_1>\frac{e^{-\chi T}-{\rho }_1+\sqrt{{\left(e^{-\chi T}-{\rho }_1\right)}^2-8{‖1+\Delta {\phi }_c^T\Delta {\phi }_c‖}^2\left({\left(\chi +\frac{1}{T}\right)}^2-\frac{1}{T^2{e}^{-2\chi T}}\right)}}{2{‖+1\Delta {\phi }_c^T\Delta {\phi }_c‖}^2\left({\left(\chi +\frac{1}{T}\right)}^2-\frac{1}{T^2{e}^{-2\chi T}}\right)}T.$$

(93)

Taking Lemma 1 to $c_1{\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c$, $c_2{\tilde{\omega }}_c^T{\tilde{\omega }}_c$ and $c_3{\tilde{\omega }}_a^T{\tilde{\omega }}_a$, as $x=1$, and $y=c_1{\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c$, or $y=c_2{\tilde{\omega }}_c^T{\tilde{\omega }}_c$, or $y=c_3{\tilde{\omega }}_a^T{\tilde{\omega }}_a$, or $y=c_6{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}$ with ${\mu }_1=b$, ${\mu }_2=1-b$ and $l={\left(1-b\right)}^{\frac{1-b}{b}}$. Then, we have

$$c_1{\left({\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c\right)}^b\le bl+c_1{\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c$$

(94)

$$c_2{\left({\tilde{\omega }}_c^T{\tilde{\omega }}_c\right)}^b\le bl+c_2{\tilde{\omega }}_c^T{\tilde{\omega }}_c$$

(95)

$$c_3{\left({\tilde{\omega }}_a^T{\tilde{\omega }}_a\right)}^b\le bl+c_3{\tilde{\omega }}_a^T{\tilde{\omega }}_a$$

(96)

$$c_6{\left({\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}\right)}^b\le bl+c_6{\displaystyle \sum _{i=1}^n{‖\Delta \psi \left(t-\theta \Delta t\right)‖}^2}.$$

(97)

Considering (94)–(97), (88) can be rewritten as

$$\begin{gathered} \begin{array}{cc}\hfill \dot{L}& \le -c_1{\left({\left({\widehat{\omega }}_c^T{\phi }_c\right)}^T{\widehat{\omega }}_c^T{\phi }_c\right)}^b-c_2{\alpha }_c^{-b}{\left({\tilde{\omega }}_c^T{\alpha }_c^{-1}{\tilde{\omega }}_c\right)}^b\hfill \\ & -c_6{Q}_3^{-1}{\left({\displaystyle \sum _{i=1}^n{\left(\Delta \psi \left(t-\theta \Delta t\right)\right)}^T{Q}_3\left(\Delta \psi \left(t-\theta \Delta t\right)\right)}\right)}^b\hfill \end{array} \\ -c_3{\alpha }_a^{-b}{\left({\tilde{\omega }}_a^T{\alpha }_a^{-1}{\tilde{\omega }}_a\right)}^b+4bl+J^2. \end{gathered}$$

(98)

Inspired by reference [45], we have $\dot{\widehat{x}}\left(t\right)=-\chi \widehat{x}\left(t\right)+\kappa \upsilon \left(t\right)$, if $\upsilon \left(t\right)>0$, $\forall t>t_0$, $\dot{\widehat{x}}>0$, $x\left(t\right)>0$, $\forall t>t_0$.

$$\dot{L}\le -c{L}^b+\pi$$

(99)

where

$$c=\min \left(c_1,c_2{\alpha }_c^{-b},-c_3{\alpha }_a^{-b},c_6{Q}_3^{-1}\right)$$

(100)

$$\pi =4bl+J^2.$$

(101)

Based on (89)–(93), (99)–(101), and the lemma in [39], the boundedness of all singles in the closed loop nonstrict system is SGPFS for $\forall t>t_1$.

Furthermore, by using (62), we obtain the optimal control strategy, which guarantees that the target nonlinear system system’s state and the stability of input delays are maintained, and that the tracking error converges to a sufficiently small neighborhood around zero.

The proof is completed. □

4. Results of Simulation Example

The WMR system [54] in Figure 2 illustrates the effectiveness of the proposed algorithm.

$$\left\{\begin{array}{l}m\dot{v}\cos {\beta }_w-mv{\dot{\beta }}_w\sin {\beta }_w+m{d}_w{\dot{\varphi }}^2=F_{DP}^1+F_{DP}^2-f_{DP}\\ -mg\sin \theta \cos o\\ I\dot{\omega }=-F_{DP}^1{d}_1+F_{DP}^2{d}_1-{\tau }_R\end{array}\right.$$

(102)

where $m$ represent the robot mass, and $I$ denotes the rotational inertia around the motion center. ${\beta }_w$ is the angle between the robot speed and the $x_m$ axis, and $o$ is the angle of inclination of the ground environment where the robot is located.

Figure 2. The structure of the wheeled mobile robot.

Then, rewrite (102) as a vector form

$$\mathit{M}\dot{\mathit{v}}+\mathit{V}\mathit{v}+\mathit{G}=\mathit{B}\left(\mathit{\tau }-{\mathit{T}}_{De}\right)-{\mathit{F}}_R$$

(103)

where, according to [55], we have $\mathit{M}=\left[\begin{array}{cc}m\cos {\beta }_w& 0\\ 0& I\end{array}\right]$, $\mathit{V}=\left[\begin{array}{cc}-m{\dot{\beta }}_w\sin {\beta }_w& m{d}_2\dot{\varphi }\\ 0& 0\end{array}\right]$, $\mathit{v}=\left[\begin{array}{l}v\\ \omega \end{array}\right]$, ${\mathit{F}}_R=\left[\begin{array}{c}{f}_{DP}\\ {\tau }_R\end{array}\right]$, $\mathit{\tau }=\left[\begin{array}{l}{\tau }_1\\ {\tau }_2\end{array}\right]$, $\mathit{G}=\left[\begin{array}{c}mg\sin \theta \cos o\\ 0\end{array}\right]$, $\mathit{B}=\frac{1}{r_s}\left[\begin{array}{cc}1& 1\\ -d_1& d_2\end{array}\right]$, ${\mathit{T}}_{De}=\left[\begin{array}{c}\left\{R{C}_{01}^s+k_{R{C}_1}^s{s}_1+A_{R{C}_1}^s\cos \left(n_L{\vartheta }_2-{\xi }_{01}^s\right)\right\}{F}_N^1\\ \left\{R{C}_{02}^s+k_{R{C}_2}^s{s}_2+A_{R{C}_2}^s\cos \left(n_L{\vartheta }_2-{\xi }_{02}^s\right)\right\}{F}_N^2\end{array}\right]$.

Considering the symmetry of the quality matrix and incorporating the time delay in the state, the state-space form can be used to represent the dynamic system of WMR.

$$\dot{\mathit{v}}\left(t\right)=f\left(t\right)\mathit{v}\left(t\right)+h\left(\mathit{v}\left(t-t_1\right)\right)+g\left(t\right)u+\omega \left(t\right)$$

(104)

where $f\left(t\right)=-{\mathit{M}}^{-1}\mathit{V}$, $g\left(t\right)={\mathit{M}}^{-1}\mathit{B}$ is an unknown function, and $\omega \left(t\right)={\mathit{M}}^{-1}\left(\mathit{B}{\mathit{T}}_{De}+{\mathit{F}}_R-\mathit{G}\right)$ denotes the resistance torque with the same effect and unknown resistance.

The desired trajectories indicate the forward and steering angular velocity, are given as $x_{d,1}=1.2+0.5\sin \left(0.05t\right)$ and $x_{d,2}=0.5\cos \left(0.05t\right)$.

Depending on the actual WMR system, the initial values are ${\beta }_w\left(0\right)={\left[0,0\right]}^T$, $rand\left(1,4\right)$ and $rand\left(1,4\right)$, and ${\alpha }_c=0.13$, ${\alpha }_a=0.12$, $\lambda =0.05$, $\gamma =0.10$ and $R=1$, $Q=\left[1,0;0,1\right]$ in this simulation. Then, the following simulation results are presented.

With the state time delay handled by appropriate LKFs, the impact of delay is successfully suppressed. From Figure 3 and Figure 4, we can obtain that the tracking performance of the proposed algorithm has good tracking performance.

Figure 3. Tracking trajectories of the states.

Figure 4. Tracking errors.

In addition, the adaptive update of the critic and the action can be reflected in Figure 5 and Figure 6, which ensures the boundedness of the adaptive law. Moreover, the tracking trajectory of the WMR is shown in Figure 7. According to the process, the signal in the wheeled mobile robotic system is SGPFS. Compared with previous work [56], with similar control effects, this paper additionally considers finite-time control and the final simulation results achieve finite-time convergence, reflecting the control advantages of the proposed algorithm.

Figure 5. The adaptive laws of the action NNs.

Figure 6. The adaptive laws of the critic NNs.

Figure 7. Tracking trajectories of the position.

5. Conclusions

A finite-time adaptive online optimization tracking control algorithm was suggested for nonlinear systems incorporating state time delays. By using appropriate LKFs, the issue arising from time delays in both state and input variables has been resolved. Then, a novel nonquadratic HJB function was defined, where finite time was selected as the upper limit of integration, which contains information of the state time delay on the premise of containing the basic information. With the premise of meeting specific requirements, the ideal HJB function was solved by using the IRL method. Furthermore, the SGPFS was guaranteed with the definition of the optimal control policy and the update of the adaptations of the critic and action NNs.