Reinforcement-Learning-Based Tracking Control with Fixed-Time Prescribed Performance for Reusable Launch Vehicle under Input Constraints

Abstract

This paper proposes a novel reinforcement learning (RL)-based tracking control scheme with fixed-time prescribed performance for a reusable launch vehicle subject to parametric uncertainties, external disturbances, and input constraints. First, a fixed-time prescribed performance function is employed to restrain attitude tracking errors, and an equivalent unconstrained system is derived via an error transformation technique. Then, a hyperbolic tangent function is incorporated into the optimal performance index of the unconstrained system to tackle the input constraints. Subsequently, an actor-critic RL framework with super-twisting-like sliding mode control is constructed to establish a practical solution for the optimal control problem. Benefiting from the proposed scheme, the robustness of the RL-based controller against unknown dynamics is enhanced, and the control performance can be qualitatively prearranged by users. Theoretical analysis shows that the attitude tracking errors converge to a preset region within a preassigned fixed time, and the weight estimation errors of the actor-critic networks are uniformly ultimately bounded. Finally, comparative numerical simulation results are provided to illustrate the effectiveness and improved performance of the proposed control scheme.

1. Introduction

Recent years have witnessed an increasing demand for reliable and economical access to space. Reusable launch vehicles (RLV), as a cost-effective means of undertaking space missions, are attracting more and more attention from researchers [1]. A dynamic model of RLV provides strong non-linear and coupling characteristics due to the complex flight environment of the re-entry phase. External disturbances, uncertain structural and aerodynamic parameters, and input constraints inevitably exist during real flight, having a significant impact on the attitude control system. In this context, attitude control for RLV is a challenging topic and has elicited widespread interest. Various control methodologies, such as adaptive control [2], dynamic inversion control [3], robust control [4], sliding mode control [5,6], and neural network (NN) control [7,8], have been applied over the past decades. Nevertheless, there is still scope to develop an optimal control approach for RLV suffering from complicated non-linear dynamics, parametric uncertainties and limited inputs.

From a mathematical point of view, the Hamilton–Jacobi–Bellman (HJB) function and its solution are required to be established to solve the optimal control problems. However, it is difficult to derive an analytical solution from the HJB function for non-linear continuous-time systems. Given this, a reinforcement learning (RL) scheme with an actor-critic (AC) structure was initially created by Werbos [9], whereby a critic network was exploited to approximate the value function, and an actor network was deployed to obtain the optimal control policy. Informed by Werbos’ contribution, Vamvoudakis et al. developed an online AC algorithm to solve the continuous-time infinite horizon optimal control problem [10]. He et al. proposed a novel online learning and optimization structure by incorporating a reference network into the AC structure [11]. Ma et al. devised a learning-based adaptive sliding mode control scheme for a tethered space robot with limited inputs [12]. Although the above control strategies have provided excellent results in terms of optimal control, the existing problem concerns the need for accurate dynamic modeling [13]. Given the parametric uncertainties and unknown disturbances, it is, in practice, difficult to exactly determine the system dynamics for RLV, limiting the methods’ applicability. Therefore, a further problem exists in that the aforementioned methods’ robustness must be enhanced for practical systems with unknown dynamics. Fan et al. combined (ISMC) with a reinforcement learning control scheme for non-linear systems with partially unknown dynamics [14]. However, the input constraints were not considered, and the ISMC used may lead to unexpected oscillation [15]. Zhang et al. developed a learning-based $H_{\infty }$ tracking control scheme for which the uncertainties and input constraints were considered [16]. Nevertheless, the control design is conservative, and the iterative algorithm is rather complicated.

It is of note that previous RL-based control methods only establish asymptotic or finite-time convergence [17]. Therefore, the upper bound of the convergence time is uncontrollable, and the transient and steady-state performance, namely, the maximum overshoot and the steady accuracy, cannot be quantitatively prearranged by users. As a promising solution to this problem, the prescribed performance control (PPC) method created by Bechlioulis et al. has attracted widespread attention [18]. The salient feature of PPC is that users can quantitatively pre-arrange both the transient performance and the steady tracking error. In [19], a novel PPC scheme combined with a command filter was proposed for a quadrotor unmanned aerial vehicle subject to error constraints. In [20], an NN-based adaptive non-affine tracking controller was devised for an air-breathing hypersonic vehicle with guaranteed prescribed performance. In [21], a data-driven PPC scheme was developed for an unmanned surface vehicle with unknown dynamics. However, the conventional PPC approach only ensures that the system states converge to the preset region as time tends to infinity [22], leading to an unsatisfactory solution for time-limited problems, such as the re-entry mission. Moreover, the input constraints and the optimality are not comprehensively considered in the conventional PPC paradigm. Therefore, the expected performance index cannot be consistently guaranteed and optimized for RLV suffering from poor aerodynamic maneuverability and limited control torques.

Motivated by the foregoing considerations, a novel RL-based tracking controller, with fixed-time prescribed performance for RLV subject to parameter uncertainties, external disturbances and input constraints, is investigated. The main contributions and characteristics of the proposed method can be summarized as follows.

• An online RL-based, nearly optimal, controller with limited inputs is developed by synthesizing the AC structure and the hyperbolic tangent performance index. In addition, the robustness of the learning-based controller is strengthened by incorporating a super-twisting-like sliding mode control.
• Compared with a previous learning-based controller described in [10,11,12], in which the system dynamics are required to be known exactly, the proposed control scheme only requires the input-output data pairs of RLV, such that the system dynamics can be completely unknown.
• In contrast to existing RL-based control schemes with asymptotic or finite-time convergence [10,11,12,17,21], the proposed control scheme can ensure that the tracking errors converge to a preset region within a preassigned fixed time. Moreover, the prescribed transient and steady-state performance can be guaranteed.
• Comparative numerical simulation investigations show that the proposed method can provide improved performance in terms of the transient response, and steady accuracy with less control effort.

2. Problem Statement and Preliminaries

2.1. Problem Statement

Following [2], the control-oriented model of the rigid-body RLV is given as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\dot{\mathrm{\Theta }}=\mathit{R}\mathit{\omega }\hfill \\ \mathit{I}\dot{\mathit{\omega }}=-\mathrm{\Omega }\mathit{I}\mathit{\omega }+\mathit{M}+\Delta \mathit{D}\hfill \end{array}\right.\end{array}$$

(1)

where $\mathrm{\Theta }={[\alpha ,\beta ,\sigma ]}^{\top }$ represents the attitude angle vector, $\mathit{\omega }={[p,q,r]}^{\top }$ denotes the angular rate vector, $\mathit{M}={[M_x,M_y,M_z]}^{\top }$ is the control input vector, and $M_x,M_y,M_z$ are limited to the interval $[-\overline{M},\overline{M}]$, $\Delta \mathit{D}=\Delta {\mathit{D}}_a+\Delta {\mathit{D}}_e$ is the unknown disturbance vector, $\Delta {\mathit{D}}_a$ is the aerodynamic torque vector, and $\Delta {\mathit{D}}_e$ is the external disturbance vector. $\Delta {\mathit{D}}_a$ can be formulated as:

$$\begin{array}{c}\hfill \Delta {\mathit{D}}_a=\left[\begin{array}{c}\left(\frac{m_x^p{L}_{r}p}{V}+m_x^{\sigma }\sigma \right)q_V{S}_r{L}_r\\ \left(\frac{m_y^r{L}_{r}r}{V}+m_y^{\beta }\beta \right)q_V{S}_r{L}_r\\ \left(\frac{m_z^q{L}_{r}q}{V}+m_z^{\alpha }\alpha \right)q_V{S}_r{L}_r\end{array}\right]\end{array}$$

(2)

where $m_x^p$, $m_y^r$ and $m_z^q$ represent the damping moment coefficients, $m_x^{\sigma }$, $m_y^{\beta }$ and $m_z^{\alpha }$ are the static stability moment coefficients, V is the velocity, $q_V$ is the dynamic pressure, and $S_r$ and $L_r$ are the cross-sectional area and the reference length of RLV, respectively.

The skew-symmetric matrix $\mathrm{\Omega }$, the inertia matrix $\mathit{I}$, and the coordinate transformation matrix $\mathit{R}$ are defined by

$$\begin{array}{c}\hfill \begin{array}{c}\mathrm{\Omega }=\left[\begin{array}{ccc}0& -r& q\\ r& 0& -p\\ -q& p& 0\end{array}\right],\mathit{I}=\left[\begin{array}{ccc}{I}_{xx}& 0& -I_{xz}\\ 0& I_{yy}& 0\\ -I_{zx}& 0& I_{zz}\end{array}\right],\hfill \\ \mathit{R}=\left[\begin{array}{ccc}-cos\alpha tan\beta & 1& -sin\alpha tan\beta \\ sin\alpha & 0& -cos\alpha \\ -cos\alpha cos\beta & -sin\beta & -sin\alpha cos\beta \end{array}\right].\hfill \end{array}\end{array}$$

(3)

Defining the guidance command vector ${\mathrm{\Theta }}_d={\left[{\alpha }_c,{\beta }_c,{\sigma }_c\right]}^{\top }$, the attitude tracking error vector ${\mathit{e}}_1=\mathrm{\Theta }-{\mathrm{\Theta }}_d={[e_{1\alpha },e_{1\beta },e_{1\sigma }]}^{\top }$, and the angular rate tracking error vector ${\mathit{e}}_2=\mathit{R}\mathit{\omega }-{\dot{\mathrm{\Theta }}}_d$, the tracking error dynamics are given as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{\mathit{e}}}_1={\mathit{e}}_2\hfill \\ {\dot{\mathit{e}}}_2={\mathit{B}}_1\mathit{M}+\Delta {\mathit{D}}_1\hfill \end{array}\right.\end{array}$$

(4)

where ${\mathit{B}}_1=\mathit{R}{\mathit{I}}^{-1}$ is the control matrix, $\Delta {\mathit{D}}_1=-\mathit{R}{\mathit{I}}^{-1}\mathrm{\Omega }\mathit{I}\mathit{\omega }-{\ddot{\mathrm{\Theta }}}_c+\dot{\mathit{R}}\mathit{\omega }+\mathit{R}{\mathit{I}}^{-1}\Delta \mathit{D}$ denotes the lumped disturbance vector.

Assumption A1

([2]). $\Delta {\mathit{D}}_1$ is bounded by $\left|\right|\Delta {\mathit{D}}_1\left|\right|\le D_m$.

Assumption A2

([2]). During the re-entry phase, $\beta \ne \pm 90\mathrm{deg}$, thus $\mathit{R}$ is always invertible.

Control Objective: According to the tracking error dynamics (4), the control objective of this paper can be summarized as developing an RL-based optimal control scheme with guaranteed fixed-time prescribed performance such that the attitude tracking errors $e_{1i}(i=\alpha ,\beta ,\sigma )$ can converge to a preset region within a preassigned fixed time.

2.2. Preliminaries

Lemma 1

([23]). Considering a non-linear function $\alpha \left(x\right)=\ln [1-{\tanh }^2\left(\mathit{x}\right)]$, the following equation

$$\begin{array}{c}\hfill \alpha \left(x\right)=\ln \left(4\right)-2x\mathrm{sign}\left(x\right)+{\kappa }_{\alpha }\end{array}$$

(5)

always holds, where ${\kappa }_{\alpha }$ is bounded by a real positive constant.

Lemma 2

([24]). Considering the following fixed-time prescribed performance function (FTPPF)

$$\begin{array}{c}\hfill \rho \left(t\right)=\left\{\begin{array}{c}\left({\rho }_0-{\rho }_{\infty }\right)\left[\frac{sin\left(2\pi t/T\right)}{2\pi }-\frac{t}{T}\right]+{\rho }_0,0\le t\le T\hfill \\ {\rho }_{\infty },t>T\hfill \end{array}\right.\end{array}$$

(6)

where ${\rho }_0>{\rho }_{\infty }>0$ represent the initial and terminal values of FTPPF, respectively. $T>0$ is the preassigned convergence time. It can be concluded that $\rho \left(t\right)$ is a positive, non-increasing and ${𝒞}^2$ continuous function with $\rho \left(0\right)={\rho }_0$, $\rho \left(T\right)={\rho }_{\infty }$ and $\dot{\rho }\left(T\right)=\ddot{\rho }\left(T\right)=0$.

Lemma 3

([25]). For any $\mu >0$, the following inequality holds

$$\begin{array}{c}\hfill 0\le \left|x\right|-xtanh\left(\frac{x}{\mu }\right)\le k_p\mu \end{array}$$

(7)

where $k_p$ satisfies $k_p=e^{-(k_p+1)}$ (i.e., $k_p=0.2785$).

3. Controller Design

3.1. Prescribed Performance Constraint

In this subsection, the following constraint is formulated to restrain the attitude tracking errors $e_{1i}(i=\alpha ,\beta ,\sigma )$ within the FTPPF

$$\begin{array}{c}\hfill -{\rho }_i\left(t\right)<e_{1i}\left(t\right)<{\rho }_i\left(t\right)\end{array}$$

(8)

where ${\rho }_i\left(t\right)$ is defined in Lemma 2. Subsequently, the equivalent unconstrained error variables ${\eta }_{1i}(i=\alpha ,\beta ,\sigma )$ can be derived via the error transformation method [18]

$$\begin{array}{c}\hfill {\eta }_{1i}=\frac{1}{2}\ln \frac{1+z_i}{1-z_i}\end{array}$$

(9)

with $z_i=e_{1i}/{\rho }_i$. Taking the first and second-order time derivatives of ${\eta }_{1i}$ yields

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{\eta }}_{1i}={\xi }_i{\eta }_{2i}\hfill \\ {\dot{\eta }}_{2i}={\ddot{e}}_{2i}-{\mathsf{\Lambda }}_i\hfill \end{array}\right.\end{array}$$

(10)

with ${\eta }_{2i}=e_{2i}-\frac{{\dot{\rho }}_i}{{\rho }_i}{e}_{1i}$, ${\xi }_i=\frac{1}{2{\rho }_i}\left(\frac{1}{z_i+1}-\frac{1}{z_i-1}\right)\ge \frac{1}{{\rho }_{i\infty }}$ and ${\mathsf{\Lambda }}_i=\frac{{\rho }_i{\ddot{\rho }}_i{e}_{1i}+{\rho }_i{\dot{\rho }}_i{e}_{2i}-{\dot{\rho }}_i^2{e}_{1i}}{{\rho }_i^2}$. Substituting (4) into (10), one obtains

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{\mathit{\eta }}}_1=\mathsf{\xi }{\mathit{\eta }}_2\hfill \\ {\dot{\mathit{\eta }}}_2={\mathit{B}}_1\mathit{M}+{\mathit{D}}_1-\mathsf{\Lambda }\hfill \end{array}\right.\end{array}$$

(11)

where $\mathsf{\xi }=\mathbb{D}\left({\xi }_i\right)$, ${\mathit{\eta }}_2=\mathbb{C}\left({\eta }_{2i}\right)$ and $\mathsf{\Lambda }=\mathbb{C}\left({\mathsf{\Lambda }}_i\right)$, and the symbols $\mathbb{C}\left(·\right)$ and $\mathbb{D}\left(·\right)$ represent the diagonal matrix and the column vector, respectively.

3.2. Reinforcement Learning-Based Control Design

Firstly, the following sliding variable $\mathit{s}$ is defined as:

$$\begin{array}{c}\hfill \mathit{s}={\mathit{\eta }}_2+\mathit{c}{\mathit{\eta }}_1\end{array}$$

(12)

where $\mathit{c}$ is a positive diagonal matrix. Taking the first time derivative of $\mathit{s}$ along (11) yields

$$\begin{array}{c}\hfill \dot{\mathit{s}}=\mathsf{\Xi }+{\mathit{B}}_1\mathit{M}+{\mathit{D}}_1\end{array}$$

(13)

where $\mathsf{\Xi }=\mathit{c}{\dot{\mathit{\eta }}}_1-\mathsf{\Lambda }$. In order to achieve a satisfactory tracking performance, a traditional super-twisting controller based on (12) and (13) is developed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathit{M}=-{\mathit{B}}_1^{-1}\mathrm{\Gamma },\mathrm{\Gamma }=\mathsf{\Xi }+\mathbb{C}\left[{\lambda }_{1i}{\mathrm{sig}}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d,\end{array}\end{array}$$

(14)

where ${\mathrm{sig}}^{1/2}\left(s_i\right)={|s_i|}^{1/2}\mathrm{sign}\left(s_i\right)$, ${\dot{\mathit{M}}}_d=-\mathbb{D}\left({\lambda }_{2i}\right)\mathrm{sign}\left(s_i\right)$, and ${\lambda }_{1i}$ and ${\lambda }_{2i}$ are positive constants. Nevertheless, the feasibility of (14) may not be guaranteed with limited control inputs, and the time-fuel performance index cannot be approximately optimized. To this end, an online RL-based, nearly optimal, controller is proposed by integrating the AC structure into the super-twisting controller for a comprehensive solution to the issue mentioned above.

Before elaborating the detailed design procedure, it is assumed that there exists a group of admissible control strategies [10]

$$\begin{array}{c}\hfill \mathit{M}={\mathit{M}}_o-{\mathit{B}}_1^{-1}\left\{\mathbb{C}\left[{\lambda }_{1i}{\mathrm{sig}}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d\right\}\in {\mathrm{\Omega }}_u.\end{array}$$

(15)

Moreover, $\mathit{M}$ can achieve the control objective with the following time-fuel performance index being satisfied

$$\begin{array}{c}\hfill V\left(\mathit{s}\right)={\int }_t^{\infty }\left\{V_s\left(\tau \right)+J\left[{\mathit{M}}_o\left(\tau \right)\right]\right\}\mathrm{d}\tau \end{array}$$

(16)

where $V_s={\mathit{s}}^{\top }\mathit{Q}\mathit{s}$, $\mathit{Q}$ is a positive diagonal matrix, and $J\left({\mathit{M}}_o\right)$ is chosen as [26]:

$$\begin{array}{c}\hfill J\left({\mathit{M}}_o\right)=2{\int }_{\mathbf{0}}^{{\mathit{M}}_o}\left[\lambda {\tanh }^{-1}{\left(\frac{\mathit{v}}{\lambda }\right)}^{\top }\mathit{\varpi }\right]\mathrm{d}\mathit{v}\end{array}$$

(17)

where $\mathit{\varpi }$ is selected as a positive diagonal matrix, $\mathit{v}$ represents the variable of integration, and the upper and lower bounds of $\mathit{v}$ are ${\mathit{M}}_o$ and $\mathbf{0}$, respectively. The Lyapunov equation of (16) can be calculated as:

$$\begin{array}{c}\hfill V_s\left(t\right)+J\left[{\mathit{M}}_o\left(t\right)\right]+\nabla V_s^{\top }\dot{\mathit{s}}=0\end{array}$$

(18)

where $\nabla V_s=\partial V\left(\mathit{s}\right)/\partial \mathit{s}$. The optimal value function is defined as:

$$\begin{array}{c}\hfill V^{*}\left(\mathit{s}\right)=\underset{{\mathit{M}}_o}{\min }{\int }_t^{\infty }\left\{V_s\left(\tau \right)+J\left[{\mathit{M}}_o\left(\tau \right)\right]\right\}\mathrm{d}\tau ,\end{array}$$

(19)

and the corresponding HJB equation can be formulated as:

$$\begin{array}{c}\hfill \underset{{\mathit{M}}_o}{\min }\left\{V_s+J\left({\mathit{M}}_o\right)+\nabla V_s^{*\top }\dot{\mathit{s}}\right\}=0,\end{array}$$

(20)

where $\nabla V_s^{*}=\partial V^{*}\left(\mathit{s}\right)/\partial \mathit{s}$.

Equation (20) is equivalent to

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathit{M}}_o^{*}}\left[V_s+J\left({\mathit{M}}_o^{*}\right)+\nabla V_s^{*\top }\dot{\mathit{s}}\right]=0.\end{array}$$

(21)

Solving the partial derivative of (21) yields the following optimal control strategy

$$\begin{array}{c}\hfill \begin{array}{c}{\mathit{M}}_o^{*}=-\lambda \tanh \left({\overline{\mathit{M}}}_o^{*}\right),{\overline{\mathit{M}}}_o^{*}=\frac{1}{2\lambda }{\mathit{\varpi }}^{-1}{\mathit{B}}_1^{\top }\nabla V_s^{*}.\hfill \end{array}\end{array}$$

(22)

Substituting ${\mathit{M}}_o^{*}$ into (17), one obtains

$$\begin{array}{c}\hfill \begin{array}{c}J\left({\mathit{M}}_o^{*}\right)=\lambda \nabla V_s^{*\top }{\mathit{B}}_1\tanh \left({\overline{\mathit{M}}}_o^{*}\right)+{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\overline{\mathit{M}}}_o^{*}\right)\right]\hfill \\ =-\nabla V_s^{*\top }{\mathit{B}}_1{\mathit{M}}_o^{*}+{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\overline{\mathit{M}}}_o^{*}\right)\right],\hfill \end{array}\end{array}$$

(23)

where ${\mathbf{1}}_c$ is a column vector with all elements being one, and $\overline{\mathit{\varpi }}$ is a row vector generated by the elements on the main diagonal of $\mathit{\varpi }$. Combining (20) and (23), it can be derived that

$$\begin{array}{c}\hfill V_s+\nabla V_s^{*\top }\dot{\mathit{s}}+{\lambda }^2\overline{\mathit{\varpi }}\ln [{\mathbf{1}}_c-{\tanh }^2\left({\overline{\mathit{M}}}_o^{*}\right)]=0.\end{array}$$

(24)

The nearly optimal control ${\mathit{M}}_o^{*}$ can be obtained by solving (24) if $\nabla V_s^{*}$ is available. Inspired by the NN-based control scheme, the following approximation of $V^{*}\left(\mathit{s}\right)$ can be established

$$\begin{array}{c}\hfill V^{*}\left(\mathit{s}\right)={\mathit{W}}^{*\top }\mathit{\sigma }\left[\mathit{s}\left(t\right)\right]+ϵ\left[\mathit{s}\left(t\right)\right]\end{array}$$

(25)

where ${\mathit{W}}^{*}$ is the optimal weight vector, $\mathit{\sigma }\left[\mathit{s}\left(t\right)\right]$ is the base vector, and $ϵ\left[\mathit{s}\left(t\right)\right]$ is the approximation error of NN. Subsequently, the gradient of $V^{*}\left(s\right)$ is

$$\begin{array}{c}\hfill \nabla V_s^{*}={\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s\left[\mathit{s}\left(t\right)\right]+\nabla {\mathsf{ϵ}}_s\left[\mathit{s}\left(t\right)\right]\end{array}$$

(26)

where $\nabla {\mathit{\sigma }}_s=\partial \mathit{\sigma }/\partial \mathit{s}$, $\nabla {\mathsf{ϵ}}_s=\partial \mathsf{ϵ}/\partial \mathit{s}$. In light of the universal approximation property of NN for smooth functions on prescribed compact sets, the approximation errors $ϵ\left[\mathit{s}\left(t\right)\right]$ and $\nabla {\mathsf{ϵ}}_s\left[\mathit{s}\left(t\right)\right]$ are bounded with a finite dimension of $\mathit{\sigma }\left[\mathit{s}\left(t\right)\right]$ [27]. Moreover, it is assumed that $\left|\right|{\mathit{W}}^{*}\left|\right|$, $\mathit{\sigma }\left[\mathit{s}\left(t\right)\right]$ and $\nabla {\mathit{\sigma }}_s\left[\mathit{s}\left(t\right)\right]$ are bounded [13,14,16,17,21].

Recalling (22), the NN-based nearly optimal control law can be formulated as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathit{M}={\widehat{\mathit{M}}}_o^{*}-{\mathit{B}}_1^{-1}\left\{\mathbb{C}\left[{\lambda }_{1i}{\mathrm{sig}}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d\right\}\hfill \\ {\widehat{\mathit{M}}}_o^{*}=-\lambda \tanh \left[\frac{1}{2\lambda }{\mathit{\varpi }}^{-1}{\mathit{B}}_1^{\top }\left(\nabla {\mathit{\sigma }}_s^{\top }{\mathit{W}}^{*}+\nabla {\mathsf{ϵ}}_s\right)\right]\hfill \end{array}\right..\end{array}$$

(27)

Defining the Bellman error as [14]:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill B_{ϵ}={\lambda }^2\overline{\mathit{\varpi }}& \left\{\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\overline{\mathit{M}}}_o^{*}\right)\right]\left.-\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]\right\}\right.\hfill \end{array}\end{array}$$

(28)

with ${\widehat{\overline{\mathit{M}}}}_o^{*}=\frac{1}{2\lambda }{\mathit{\varpi }}^{-1}{\mathit{B}}_1^{\top }\nabla {\mathit{\sigma }}_s^{\top }{\mathit{W}}^{*}$, (20) can be rewritten as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill V_s& +{\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s\mathrm{\Gamma }+{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]+{ϵ}_H=0\hfill \end{array}\end{array}$$

(29)

where ${ϵ}_H=\nabla {\mathsf{ϵ}}_s^{\top }\mathrm{\Gamma }+B_{ϵ}+{\mathit{W}}^{*}\nabla {\mathit{\sigma }}_s{\mathit{D}}_2$ is the bounded HJB error [10].

In this paper, the optimal weight ${\mathit{W}}^{*}$ is generated by the online RL scheme with the AC structure. In this context, the nearly optimal control policy is formulated as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathit{M}={\widehat{\mathit{M}}}_a^{*}-{\mathit{B}}_1^{-1}\left\{\mathbb{C}\left[{\lambda }_{1i}{\mathrm{sig}}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d\right\}\hfill \\ {\widehat{\mathit{M}}}_a^{*}=-\lambda \tanh \left({\widehat{\overline{\mathit{M}}}}_a^{*}\right),{\widehat{\overline{\mathit{M}}}}_a^{*}=\frac{1}{2\lambda }{\mathit{\varpi }}^{-1}{\mathit{B}}_1^{\top }\nabla {\mathit{\sigma }}_s^{\top }{\widehat{\mathit{W}}}_a\hfill \end{array}\right.\end{array}$$

(30)

where ${\widehat{\mathit{W}}}_a$ is the weight of the actor network. Moreover, the performance index (16) can be estimated as:

$$\begin{array}{c}\hfill {\widehat{V}}_s={\widehat{\mathit{W}}}_c^{\top }{\mathit{\sigma }}_s\left[\mathit{s}\left(t\right)\right]\end{array}$$

(31)

where ${\widehat{\mathit{W}}}_c$ is the weight of the critic network. The adaptation laws for ${\widehat{\mathit{W}}}_c$ and ${\widehat{\mathit{W}}}_a$ are designed as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\dot{\widehat{\mathit{W}}}}_c=-{\mathit{A}}_1\left\{\overline{\mathit{\theta }}\left[{\mathit{\theta }}^{\top }{\widehat{\mathit{W}}}_c+V_s+J\left({\widehat{\mathit{M}}}_a^{*}\right)\right]+p_c\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\right\},\end{array}\end{array}$$

(32)

$$\begin{array}{c}\hfill {\dot{\widehat{\mathit{W}}}}_a=-{\mathit{A}}_2\left[p_a\left({\widehat{\mathit{W}}}_a-{\widehat{\mathit{W}}}_c\right)-\frac{\lambda p_a}{p_c}\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\mathrm{\Psi }{\overline{\mathit{\theta }}}^{\top }{\widehat{\mathit{W}}}_c\right],\end{array}$$

(33)

where $\mathit{\theta }=\nabla {\mathit{\sigma }}_s\left(\mathrm{\Gamma }+{\mathit{B}}_1{\widehat{\mathit{M}}}_a^{*}\right)$, $\overline{\mathit{\theta }}=\mathit{\theta }/{\left({\mathit{\theta }}^{\top }\mathit{\theta }+1\right)}^2$; ${\mathit{A}}_1$ and ${\mathit{A}}_2$ are positive diagonal matrices; $\mathrm{\Psi }=tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)-tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}/\kappa \right)$; $p_c$, $p_a$ and $\kappa$ are positive real constants. The projection operator $\mathrm{Proj}\left(·\right)$ is imposed on (33) to guarantee that ${\widehat{\mathit{W}}}_a$ is bounded [28]. It is assumed that ${\mathit{\theta }}_1=\mathit{\theta }/\left({\mathit{\theta }}^{\top }\mathit{\theta }+1\right)$ is persistently excitating (PE) [29].

The proposed control scheme is illustrated by a block diagram in Figure 1.

Figure 1. The block diagram of the proposed control scheme.

3.3. Stability Analysis

Theorem 1.

Considering the control-oriented RLV model (1) with input constraints, if the initial conditions satisfies $-{\rho }_{i0}<e_{1i}\left(0\right)<{\rho }_{i0}$, the nearly optimal control policy is chosen as (30) with the weight update laws (32) and (33), then the following results can be obtained:

• the sliding variable $\mathit{s}$, the weight estimation errors ${\tilde{\mathit{W}}}_a$ and ${\tilde{\mathit{W}}}_c$ are uniformly ultimately bounded (UUB);
• the attitude tracking errors $e_{1i}$ uniformly obey the fixed-time performance envelops in (8).

Proof of Theorem 1

Consider the Lyapunov function candidate as follows:

$$\begin{array}{c}\hfill V=V\left(\mathit{s}\right)+\frac{1}{2}{\tilde{\mathit{W}}}_c^{\top }{\mathit{A}}_1^{-1}{\tilde{\mathit{W}}}_c+\frac{p_c}{2p_a}{\tilde{\mathit{W}}}_a^{\top }{\mathit{A}}_2^{-1}{\tilde{\mathit{W}}}_a\end{array}$$

(34)

where ${\tilde{\mathit{W}}}_c={\mathit{W}}^{*}-{\widehat{\mathit{W}}}_c$ and ${\tilde{\mathit{W}}}_a={\mathit{W}}^{*}-{\widehat{\mathit{W}}}_a$. Taking the first time derivative of V yields

$$\begin{array}{c}\hfill \dot{V}=\dot{V}\left(\mathit{s}\right)+{\dot{V}}_c+{\dot{V}}_a\end{array}$$

(35)

with ${\dot{V}}_c={\tilde{\mathit{W}}}_c^{\top }{\mathit{A}}_1^{-1}{\dot{\tilde{\mathit{W}}}}_c$, ${\dot{V}}_a=\frac{p_c}{p_a}{\tilde{\mathit{W}}}_a^{\top }{\mathit{A}}_2^{-1}{\dot{\tilde{\mathit{W}}}}_a$, and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \dot{V}\left(\mathit{s}\right)=\nabla V_s^{\top }\left(\mathrm{\Gamma }+{\mathit{D}}_1+{\mathit{B}}_1{\widehat{\mathit{M}}}_a^{*}\right).\end{array}\end{array}$$

(36)

By invoking the HJB equation (29), $\nabla V_s^{\top }\mathsf{\Xi }$ can be rewritten as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \nabla V_s^{\top }\mathsf{\Xi }& =-V_s-{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]+\nabla V_s^{\top }\left\{\mathbb{C}\left[{\lambda }_{1i}{\mathrm{sig}}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d\right\}-{\epsilon }_e\hfill \end{array}\end{array}$$

(37)

where ${\epsilon }_e=B_{ϵ}+\nabla V_s^{\top }{\mathit{D}}_1$. Substituting (37) into (36) yields

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \dot{V}\left(\mathit{s}\right)& =-V_s-{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]-\lambda \nabla V_s^{*\top }{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)-B_{ϵ}.\hfill \end{array}\end{array}$$

(38)

Furthermore, according to (17), it can be deduced that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]& =J\left({\underline{\mathit{M}}}_o^{*}\right)+\nabla V_s^{*\top }{\mathit{B}}_1{\underline{\mathit{M}}}_o^{*}\hfill \end{array}\end{array}$$

(39)

with ${\underline{\mathit{M}}}_o^{*}=-\lambda tanh\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)$. Equation (38) can be rewritten as:

$$\begin{array}{c}\begin{array}{c}\dot{V}\left(\mathit{s}\right)=-V_s-J\left({\underline{\mathit{M}}}_o^{*}\right)-\nabla V_s^{*\top }{\mathit{B}}_1{\underline{\mathit{M}}}_o^{*}-B_{ϵ}-\lambda \left({\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s+\nabla {\mathsf{ϵ}}_s^{\top }\right){\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\hfill \\ \hspace{1em}\hspace{1em}=-V_s-J\left({\underline{\mathit{M}}}_o^{*}\right)+{\epsilon }_v-\lambda \left({\widehat{\mathit{W}}}_a^{\top }+{\tilde{\mathit{W}}}_a^{\top }\right)\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\hfill \end{array}\hfill \end{array}$$

(40)

where ${\epsilon }_v=-B_{\epsilon }+\nabla V_s^{*\top }{\mathit{B}}_1{\underline{\mathit{M}}}_o^{*}-\nabla {\mathsf{ϵ}}_s^{\top }{\mathit{B}}_1{\widehat{\mathit{M}}}_a^{*}$. Noting that $tanh\left(·\right)$ is an odd function, it can be concluded that $\lambda {\widehat{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)>0$. Moreover, it is indicated that $J\left({\underline{\mathit{M}}}_o^{*}\right)>0$ from the definition in (17). Based on the above discussions, (40) can be simplified as:

$$\begin{array}{c}\hfill {\dot{V}}_s\left(\mathit{s}\right)\le -V_s-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+{\epsilon }_v.\end{array}$$

(41)

Subsequently, incorporating (32) with ${\dot{V}}_c$ yields

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\dot{V}}_c& =-{\tilde{\mathit{W}}}_c^{\top }{\mathrm{\Gamma }}_1{\dot{\widehat{\mathit{W}}}}_c={\tilde{\mathit{W}}}_c^{\top }\left\{\overline{\mathit{\theta }}\left[{\mathit{\theta }}^{\top }{\widehat{\mathit{W}}}_c+V_s+J\left({\widehat{\mathit{M}}}_a^{*}\right)\right]+p_c\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\right\}.\hfill \end{array}\end{array}$$

(42)

Recalling (29) and the definition of $\mathit{\theta }$, (42) can be rearranged as:

$$\begin{array}{c}\hfill \begin{array}{c}{\dot{V}}_c={\tilde{\mathit{W}}}_c^{\top }\left\{\overline{\mathit{\theta }}[{\mathit{\theta }}^{\top }{\widehat{\mathit{W}}}_c-{\mathit{\theta }}^{\top }{\mathit{W}}^{*}+{\mathit{\theta }}^{\top }{\mathit{W}}^{*}-{ϵ}_H-{\lambda }^2\overline{\mathit{\varpi }}\ln [{\mathbf{1}}_c-{\tanh }^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)]\right.\hfill \\ \left.-{\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s\mathrm{\Gamma }+J\left({\widehat{\mathit{M}}}_a^{*}\right)]\right\}+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\hfill \\ ={\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}\left\{-{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+{\left(\nabla {\mathit{\sigma }}_s{\mathit{B}}_1{\widehat{\mathit{M}}}_a^{*}\right)}^{\top }{\mathit{W}}^{*}-{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{tanh}^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]-{ϵ}_H\right.\hfill \\ +\lambda {\widehat{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+\left.{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{tanh}^2\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\right]\right\}+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\hfill \\ ={\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}\left\{-{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)-{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{tanh}^2\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]-{ϵ}_H\right.\hfill \\ \left.+{\lambda }^2\overline{\mathit{\varpi }}\ln \left[{\mathbf{1}}_c-{tanh}^2\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\right]\right\}+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right).\hfill \end{array}\end{array}$$

(43)

With the aid of Lemma 1 and Lemma 3, (43) can be further simplified as:

$$\begin{array}{c}\hfill \begin{array}{c}{\dot{V}}_c={\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}\left[-{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)-{ϵ}_H-\lambda {\mathit{B}}_1^{\top }\nabla {\mathit{\sigma }}_s^{\top }{\widehat{\mathit{W}}}_a\mathrm{sign}\right.\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\hfill \\ \left.+\lambda {\mathit{B}}_1^{\top }\nabla {\mathit{\sigma }}_s^{\top }{\mathit{W}}^{*}\mathrm{sign}\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)+{\lambda }^2\overline{\mathit{\varpi }}\left({\mathsf{\epsilon }}_a-{\mathsf{\epsilon }}_{a^{*}}\right)\right]+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\hfill \\ ={\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}\left[-{ϵ}_H+{\lambda }^2\overline{\mathit{\varpi }}\left({\mathsf{\epsilon }}_a-{\mathsf{\epsilon }}_{a^{*}}\right)+\lambda {\mathit{B}}_1^{\top }\nabla {\mathit{\sigma }}_s^{\top }{\mathit{W}}^{*}\mathrm{sign}\left({\widehat{\overline{\mathit{M}}}}_o^{*}\right)\right]\hfill \\ -{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}\left\{\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+\lambda {\widehat{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\left[tanh\left(\frac{{\widehat{\overline{\mathit{M}}}}_a^{*}}{\kappa }\right)+{\kappa }_1\right]\right\}\hfill \\ -{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\hfill \end{array}\end{array}$$

(44)

where $\left|\right|{\mathsf{\epsilon }}_a\left|\right|$, $\left|\right|{\mathsf{\epsilon }}_{a^{*}}\left|\right|$ and $|{\kappa }_1|$ are bounded. For ease of notation, two bounded vectors ${\aleph }_1$ and ${\aleph }_2$ are defined as:

$$\begin{array}{c}\hfill \begin{array}{c}{\aleph }_1=\left\{\lambda \nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left(\frac{{\widehat{\overline{\mathit{M}}}}_a^{*}}{\kappa }\right)-\lambda \nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)\right\}{\overline{\mathit{\theta }}}^{\top }{\mathit{W}}^{*},\hfill \\ {\aleph }_2=\lambda {\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\mathrm{sign}\left({\widehat{\overline{\mathit{W}}}}_o^{*}\right)-\lambda {\mathit{W}}^{*\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\left[tanh\left(\frac{{\widehat{\overline{\mathit{M}}}}_a^{*}}{\kappa }\right)+{\kappa }_1\right]\hfill \\ -{ϵ}_H+{\lambda }^2\overline{\mathit{\varpi }}\left({\mathsf{\epsilon }}_a-{\mathsf{\epsilon }}_{a^{*}}\right),\hfill \end{array}\end{array}$$

(45)

then (44) can be further sorted into the following structure

$$\begin{array}{c}\hfill \begin{array}{c}{\dot{V}}_c=-{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)+{\tilde{\mathit{W}}}_c\overline{\mathit{\theta }}{\aleph }_2+{\tilde{\mathit{W}}}_a^{\top }{\aleph }_1+\lambda {\kappa }_1{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\hfill \\ +\lambda {\widehat{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\mathrm{\Psi }.\hfill \end{array}\end{array}$$

(46)

Substituting (33), (41), and (46) into (35) yields

$$\begin{array}{c}\hfill \begin{array}{c}\dot{V}\le -V_s-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+{\epsilon }_{-}{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+p_c{\tilde{\mathit{W}}}_c^{\top }\left({\widehat{\mathit{W}}}_c-{\widehat{\mathit{W}}}_a\right)\hfill \\ +{\tilde{\mathit{W}}}_c\overline{\mathit{\theta }}{\aleph }_2+{\tilde{\mathit{W}}}_a^{\top }{\aleph }_1+\lambda {\kappa }_1{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1-\lambda {\widehat{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\hfill \\ +\lambda {\widehat{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\mathrm{\Psi }+{\tilde{\mathit{W}}}_a^{\top }\left[p_c\left({\widehat{\mathit{W}}}_a-{\widehat{\mathit{W}}}_c\right)-\lambda \nabla {\mathit{\sigma }}_s{\mathit{B}}_1\mathrm{\Psi }{\overline{\mathit{\theta }}}^{\top }{\widehat{\mathit{W}}}_c\right]\hfill \\ \le -V_s-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+{\epsilon }_v-{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+p_c{\left({\tilde{\mathit{W}}}_a-{\tilde{\mathit{W}}}_c\right)}^{\top }\left({\tilde{\mathit{W}}}_a-{\tilde{\mathit{W}}}_c\right)\hfill \\ +{\tilde{\mathit{W}}}_c\overline{\mathit{\theta }}{\aleph }_2+{\tilde{\mathit{W}}}_a^{\top }{\aleph }_1+\lambda {\kappa }_1{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1\hfill \\ \le -V_s-{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\mathit{\theta }}^{\top }{\tilde{\mathit{W}}}_c+{\epsilon }_{v1}+{\tilde{\mathit{W}}}_c^{\top }\overline{\mathit{\theta }}{\aleph }_3\hfill \end{array}\end{array}$$

(47)

where ${\epsilon }_{v1}={\epsilon }_v-\lambda {\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_{1}tanh\left({\widehat{\overline{\mathit{M}}}}_a^{*}\right)+{\tilde{\mathit{W}}}_a^{\top }{\aleph }_1$ and ${\aleph }_3={\aleph }_2+\lambda {\kappa }_1{\tilde{\mathit{W}}}_a^{\top }\nabla {\mathit{\sigma }}_s{\mathit{B}}_1$.

With the definition of the generalized vector $\mathbf{G}={\left[\mathit{s},{\tilde{\mathit{W}}}_c^{\top }{\mathit{\theta }}_1\right]}^{\top }$, the inequality (47) can be rewritten as:

$$\begin{array}{c}\hfill \dot{V}\le -{\mathbf{G}}^{\top }\mathrm{\Pi }\mathbf{G}+{\mathbf{G}}^{\top }{\mathrm{\Pi }}_{\theta }+{\epsilon }_{v1}\end{array}$$

(48)

where $\mathrm{\Pi }=\mathrm{diag}\left(\mathit{Q},\mathit{I}\right)$, ${\mathrm{\Pi }}_{\theta }={\left[\mathbf{0},{\aleph }_3/\left({\mathit{\theta }}^{\top }\mathit{\theta }+1\right)\right]}^{\top }$. Given that ${\mathrm{\Pi }}_{\theta }$, ${\widehat{\mathit{W}}}_a$ are bounded vectors, it can be concluded that the ${\tilde{\mathit{W}}}_a$ is a bounded vector and $\left|\right|{\mathrm{\Pi }}_{\theta }\left|\right|\le {\overline{\mathrm{\Pi }}}_{\theta }$ and $|{\epsilon }_{v1}|\le {\overline{\epsilon }}_{v1}$. Therefore, $\dot{V}$ is negative if

$$\begin{array}{c}\hfill \left|\right|\mathbf{G}\left|\right|>\frac{{\overline{\mathrm{\Pi }}}_{\theta }\sqrt{{\overline{\mathrm{\Pi }}}_{\theta }^2+4{\lambda }_{\min }\left(\mathrm{\Pi }\right){\overline{\epsilon }}_{v1}}}{2{\lambda }_{\min }\left(\mathrm{\Pi }\right)},\end{array}$$

(49)

which implies that the sliding variable $\mathit{s}$ and ${\tilde{\mathit{W}}}_c^{\top }{\mathit{\theta }}_1$ are UUB. Furthermore, in view of the assumption that ${\mathit{\theta }}_1$ is PE, the weight estimation error ${\tilde{\mathit{W}}}_c$ is also UUB [14]. Once the sliding variable converges to the vicinity of origin, the gradient vector $\nabla \mathit{\sigma }\left[\mathit{s}\left(t\right)\right]$ together with ${\widehat{\mathit{M}}}_a^{*}$ will also be small enough. Defining $\mathit{d}=\mathsf{\Xi }+{\mathit{D}}_1+{\mathit{B}}_1{\widehat{\mathit{M}}}_a^{*}$, and assuming that there exists a small positive constant $\overline{d}$ satisfying $\left|\right|\mathit{D}\left|\right|\le \overline{d}$, the dynamic of the sliding variable can be represented as:

$$\left[\begin{array}{c}\dot{\mathit{s}}\\ {\dot{\mathit{M}}}_d\end{array}\right]=\left[\begin{array}{c}\mathit{d}-\mathbb{C}\left[{\lambda }_{1i}{sig}^{1/2}\left(s_i\right)\right]-{\mathit{M}}_d\\ -\mathbb{D}\left({\lambda }_{2i}\right)sign\left(s_i\right)\end{array}\right].$$

(50)

According to the deduction lines in [30], the sliding variable can reach a small neighborhood of origin; thus, the equivalent unconstrained errors ${\eta }_{1i}$ are bounded.

Recalling the error transformation method in (9), it can be derived that

$$\begin{array}{c}\hfill z_i=T\left({\eta }_{1i}\right)=\frac{exp\left({\eta }_{1i}\right)-exp\left(-{\eta }_{1i}\right)}{exp\left({\eta }_{1i}\right)+exp(-{\eta }_{1i})}.\end{array}$$

(51)

Note that $T\left({\eta }_{1i}\right)$ is a smooth, strictly increasing, function w.r.t. ${\eta }_{1i}$ and satisfies $\underset{{\eta }_{1i}\to -\infty }{\lim }T\left({\eta }_{1i}\right)=-1$, $\underset{{\eta }_{1i}\to \infty }{\lim }T\left({\eta }_{1i}\right)=1$; thus, it can be derived that $-1<z_i<1$ and $-{\rho }_i<e_{1i}<{\rho }_i$ hold true if ${\eta }_{1i}$ is bounded. Following the above analysis, it can be deduced that $e_{1i}$ never violates the fixed-time prescribed performance constraints in (8) for all $t>0$, and $|e_{1i}|<{\rho }_{i\infty }$ for $t>T$.

This completes the proof of Theorem 1. □

4. Numerical Simulations

In this section, numerical simulations for the re-entry phase of RLV are carried out to illustrate the effectiveness and improved performance of the proposed control scheme. The parameters of the RLV are based on [31]. The initial states of the re-entry phase are set as ${\alpha }_0=47\mathrm{deg}$, ${\beta }_0=-2\mathrm{deg}$, ${\sigma }_0=2\mathrm{deg}$, $p_0=0\mathrm{deg}/\mathrm{s}$, $q_0=0\mathrm{deg}/\mathrm{s}$ and $r_0=0\mathrm{deg}/\mathrm{s}$. The guidance commands are designed as ${\alpha }_c=(45-0.5t)\mathrm{deg}$, ${\beta }_c=0\mathrm{deg}$ and ${\sigma }_c=0\mathrm{deg}$. The uncertainties of the inertia parameters, the aerodynamic parameters and the air density are each set to +20% bias. The external disturbance $\Delta {\mathit{D}}_e$ is given as [32]:

$$\begin{array}{c}\hfill \Delta {\mathit{D}}_e=\left[\begin{array}{c}\left(0.5+cos\left(\pi t/2\right)+sin\left(\pi t/3\right)\right)\times p\\ \left(0.5+cos\left(\pi t/2\right)+sin\left(\pi t/4\right)\right)\times r\\ \left(0.5+cos\left(\pi t/3\right)+sin\left(\pi t/2\right)\right)\times q\end{array}\right]\times {10}^4\left(\mathrm{N}·\mathrm{m}\right).\end{array}$$

(52)

The saturation bound of the control torques is set as $\overline{M}=1\times {10}^5\left(\mathrm{N}·\mathrm{m}\right)$. The attitude tracking errors are required to be less than $0.5\mathrm{deg}$ for $t\ge 2\mathrm{s}$. Therefore, the parameters of the FTPPF are selected as ${\rho }_{0i}=\pi /36,{\rho }_{\infty i}=\pi /360$ and $T=2$. Comprehensively considering the satisfactory transient and steady-state performance, the parameters of the proposed control scheme are rigorously chosen as $\mathit{c}=2{\mathit{I}}_{3\times 3},\mathit{\varpi }={\mathit{I}}_{3\times 3},\mathit{Q}={\mathit{I}}_{3\times 3},{\lambda }_{1i}=2,{\lambda }_{2i}=0.1$. Inspired by the work of [12,14], the suitable basis vectors can be selected as polynomial combinations of the concerned state variables in the performance index (16). The base vectors of the actor and critic networks are adjusted in repeated trials to balance the approximation error and the computational burden. They are identically selected as:

$$\begin{array}{c}\hfill \mathit{\sigma }\left[\mathit{s}\left(t\right)\right]={\left[\frac{s_{\alpha }^4}{4},\frac{s_{\alpha }^3}{3},\frac{s_{\alpha }^2}{2},\frac{s_{\beta }^4}{4},\frac{s_{\beta }^3}{3},\frac{s_{\beta }^2}{2},\frac{s_{\sigma }^4}{4},\frac{s_{\sigma }^3}{3},\frac{s_{\sigma }^2}{2}\right]}^{\top }.\end{array}$$

(53)

The initial values of ${\widehat{\mathit{W}}}_a$ and ${\widehat{\mathit{W}}}_c$ are randomly generated in $\left(0,1\right)$. The user-defined parameters of the weight adaptation laws are designed as ${\mathit{A}}_1={\mathit{I}}_{9\times 9},{\mathit{A}}_2=0.1{\mathit{I}}_{9\times 9},p_c=0.001,p_a=1,\kappa =0.001$.

Furthermore, in order to demonstrate the superiority of the proposed control scheme, the RL-based finite-time control (RLFTC) with input constraints in [17], and the robust adaptive backstepping control (RABC) method in [33], are implemented. To provide a fair comparison, the time-fuel performance index of the RLFTC is identical to the proposed method, and the generation method for the initial values of the actor network and the critic network remains the same. The other user-defined parameters of the RLFTC are intensively selected as $T_1=0.1,\lambda =2/3,{\mathrm{\Gamma }}_c=0.2,{\mathrm{\Gamma }}_a=0.2,k_a=0.1$. The parameters of the RABC remain the same as that in [33]. The simulation results are given in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

Figure 2. Time histories of the attitude angles.

Figure 3. Time histories of the attitude tracking error $e_{1\alpha }$.

Figure 4. Time histories of the attitude tracking error $e_{1\beta }$.

Figure 5. Time histories of the attitude tracking error $e_{1\sigma }$.

Figure 6. Time histories of the angular rates.

Figure 7. Time histories of the sliding manifolds.

Figure 8. Time histories of the control torques.

Figure 9. Weights of the actor network.

Figure 10. Weights of the critic network.

The tracking performances of the attitude angles under three controllers are shown in Figure 2, Figure 3, Figure 4 and Figure 5. It can clearly be seen that the proposed control scheme can provide faster convergence and smaller overshoot in the presence of parametric uncertainties and external disturbances. The angular rates of RLV and the sliding manifolds are demonstrated in Figure 6 and Figure 7, which provide further evidence for the improved performance of the proposed control scheme. Moreover, by comparing the tracking performance with the control inputs illustrated in Figure 8, it can be observed that the proposed control scheme exhibits better transient and steady-state performance with limited control inputs. The evolution trajectories of ${\widehat{\mathit{W}}}_a$ and ${\widehat{\mathit{W}}}_c$ are depicted in Figure 9 and Figure 10, respectively. It can be readily found that ${\widehat{\mathit{W}}}_a$ and ${\widehat{\mathit{W}}}_c$ are convergent to the same values, which indicates that the ideal weight vector ${\mathit{W}}^{*}$ can be effectively estimated via the proposed adaptation law.

To make a clear comparison, the maximum overshoot and the adjustment time are introduced to evaluate the transient performance of these controllers. Moreover, the integral absolute control effort (IACE) index ${\int }_0^{30}\left(\right|M_i\left|\right)\mathrm{d}\tau ,(i=x,y,z)$ and the integral of the time and absolute error (ITAE) ${\int }_0^{30}\tau \left(\right|e_{1i})|\mathrm{d}\tau ,(i=\alpha ,\beta ,\sigma )$ index of three control schemes are calculated to evaluate the tracking accuracy and control effort. The performance indices of the three channels are summarized in Table 1, Table 2 and Table 3.

Table 1. Performance indexes of the $\alpha$-channel.

Performance Index	Proposed Method	RABC	RLFTC
Maximum Overshoot	/	20.5%	34.0%
Adjustment Time 1 (s)	3.9	5.6	4.3
ITAE index ($deg·{\mathrm{s}}^2$)	1.6407	43.2600	1.6962
IACE index ($N·\mathrm{m}·\mathrm{s}$)	$8.507\times {10}^5$	$8.893\times {10}^5$	$1.022\times {10}^6$

¹ The time when the attitude tracking error reaches to ±0.05 deg.

Table 2. Performance indexes of the $\beta$-channel.

Performance Index	Proposed Method	RABC	RLFTC
Maximum Overshoot	4.5%	19.0%	38.2%
Adjustment Time (s)	3.1	4.1	4.6
ITAE index ($deg·{\mathrm{s}}^2$)	0.7347	1.5192	1.1507
IACE index ($N·\mathrm{m}·\mathrm{s}$)	$1.8365\times {10}^5$	$2.0571\times {10}^5$	$4.155\times {10}^5$

Table 3. Performance indexes of the $\sigma$-channel.

Performance Index	Proposed Method	RABC	RLFTC
Maximum Overshoot	5.5%	17.7%	53.5%
Adjustment Time (s)	4.1	4.8	4.7
ITAE index ($deg·{\mathrm{s}}^2$)	0.8783	1.7853	1.7242
IACE index ($N·\mathrm{m}·\mathrm{s}$)	$4.6577\times {10}^3$	$5.0277\times {10}^3$	$1.8364\times {10}^4$

From the foregoing simulation results, it can be concluded that the proposed control scheme outperforms RLFTC and RABC in terms of transient performance, tracking accuracy and control effort. Furthermore, by synthesizing the AC structure and the fixed-time PPC paradigm, the proposed control scheme offers an online RL-based model-free solution for controlling RLV and other complex industrial systems.

5. Conclusions

In this paper, an innovative RL-based tracking control scheme with fixed-time prescribed performance has been proposed for RLV under parametric uncertainties, external disturbances, and input constraints. By resorting to the FTPPF, fixed-time performance envelopes have been imposed on the attitude tracking errors. Combined with the AC-based online RL structure and the super-twisting-like sliding mode control, the optimal control policy and the performance index have been learned recursively, and the robustness of the learning process has been further enhanced. Moreover, theoretical analysis has demonstrated that the attitude tracking error can converge to a preset region within a preassigned fixed time, and that the sliding variable, the weight estimation errors of the actor and critic networks are UUB. Comparative simulation results have verified the effectiveness and improved performance of the proposed control scheme. The angular rate constraints will be addressed in our future work, and the optimal control problem for the underactuated RLV will be specifically addressed. Experimental investigations, such as hardware-in-the-loop simulations, will be undertaken.