Finite-Time Adaptive Reinforcement Learning Control for a Class of Morphing Unmanned Aircraft with Mismatched Disturbances and Coupled Uncertainties

Highlights

What are the main findings?

• A novel RL-based adaptive finite-time control scheme for morphing unmanned aircraft is synthesized. It can address mismatched disturbances, coupled uncertainties, and non-affine characteristics, enabling the aircraft’s attitude to converge to the desired value within a finite time.
• The attitude dynamics of the morphing unmanned aircraft are described as a class of mismatched non-affine systems, which are more suitable for practical scenarios and simplify the analysis process compared to previous models.

What is the implication of the main finding?

• Applying reinforcement learning to morphing unmanned aircraft enhances its ability to handle uncertainties, and finite-time reinforcement learning helps limit the control convergence time, thus improving the trajectory-tracking control performance.
• The proposed scheme for RL-based robust adaptive flight control offers a method that can be extended to various aircraft control research fields.

Abstract

This paper proposes a finite-time adaptive reinforcement learning (RL) control law for a class of morphing unmanned aircraft with mismatched disturbances and coupled uncertainties. To handle the mismatched disturbances, an adaptive upper-bound estimator as well as the parameter adaptive laws have been proposed. Aiming at the coupled uncertainties, an RL-based online uncertainty estimator and a corresponding finite-time compensation control law have been developed. To deal with the non-affine characteristics, an auxiliary integral system has been introduced. By systematically integrating the aforementioned adaptive upper-bound estimators, finite-time control law, and the auxiliary signals, a novel RL-based adaptive finite-time control framework is constructed for morphing unmanned aircraft. Simulation results reveal the finite-time convergence and the advantages of the proposed method.

1. Introduction

Morphing unmanned aircraft can optimize their aerodynamics and performance by changing their shape to adapt to diverse flight scenarios [1]. It can be classified by morphing scale, location, gas flow, and implementation method [2]. Yu et al. [3] and Noordin et al. [4] developed PID-based UAV adaptive control methods, achieving satisfactory tracking performance. As the application scenarios increase, there is an urgent need to improve its modeling and control law design [5]. Guidance and control technology is crucial as it ensures stable flight in complex environments, enables real-time adjustment of flight paths and morphing strategies, and enhances overall performance and adaptability [6,7]. He et al. [8] proposed an integrated guidance and control method using backstepping and fixed-time sliding mode control. This method includes a morphing term and solves the guidance and control problems of high-speed morphing aircraft. It stabilizes the system and makes errors converge quickly, thus improving control performance. Zhang et al. [9] presented an event-triggered fixed-time sliding mode control, which enhances the aircraft’s robustness and adaptability under complex conditions. Abouheaf et al. [10] developed a machine-learning-based autonomous morphing control for flexible-wing morphing aircraft. This method shows better stability than conventional model-free adaptive control approaches. The literature [11,12] designs a feedback-linearization-based nonlinear command and stability system for variable aspect ratio morphing aircraft. This system improves flexibility and fuel efficiency. The literature [13] proposes an online actor–critic control for variable aspect-ratio and swept-morphing-wing aircraft, which solves the problem of control input frequency. Numerical simulations prove its effectiveness. The literature [14] proposes a preset-performance sliding-mode controller for high-speed morphing unmanned aircraft attitude control under strong uncertainty. Paired with a finite-time neural network disturbance observer, it enhances the system’s performance and adaptability to disturbances and parameter changes. These methods combine advanced control theories to solve the high-speed guidance and control problems of morphing unmanned aircraft.

In recent years, advanced adaptive control approaches have been extensively researched and applied in the realm of complex and uncertain systems [15,16,17]. Among them, model reference adaptive control (MRAC) is a prominent technique that enables real-time parameter adjustment. It achieves this by comparing the performance of the system with a pre-established reference model. The literature [18,19,20] puts forward an enhanced nonlinear dynamic inversion control method based on MRAC. This innovation strengthens the aircraft’s resilience against faults and external disturbances. It refines the accuracy of adaptive estimation and incorporates a low-pass filter to optimize the balance between dynamic performance and robustness. To design the controller better, it is crucial to elucidate the distinctions and relationships among parameter uncertainties, disturbance rejection, and fault-tolerant control. Parameter uncertainties can be caused by sensor biases, which introduce inaccuracies in the measurement of system states. Actuator failures, on the other hand, represent a severe type of disturbance. These failures can disrupt the normal operation of the aircraft and require effective fault-tolerant control strategies to ensure safe flight. To ensure robust performance, researchers focus on parameter uncertainties and disturbance rejection. For robustness enhancement, fuzzy adaptive control effectively strengthens system robustness under uncertain and fuzzy operating conditions. As detailed in the literature [21], a novel fixed-time adaptive generalized type-2 fuzzy logic control scheme was devised for hypersonic aircraft grappling with uncertainties. The efficacy of this control scheme is validated through simulations. Based on the observer, a robust flight controller for a disturbance-unmanned aerial vehicle system is developed [22]. In addition, in the literature [23], the control problem of nonuniform nonlinear systems with time-varying delays in both state and input has also been discussed. Adaptive sliding mode control has also made significant strides in enhancing system robustness and responsiveness. In reference [24], a dedicated control strategy for hypersonic aircraft has been proposed. It constructs a dynamic uncertainty model and estimates uncertainty bounds online to coordinate the robustness and responsiveness of the controller. During flight, morphing unmanned aircraft experience complex shape alterations and fluctuations in aerodynamic characteristics. Traditional control methods like MRAC or sliding mode control often can’t accurately capture these nonlinear dynamics. To address this, some scholars have studied the finite-time asynchronous control of singular fuzzy Markov jump systems [25]. Their research resulted in an adaptive event-triggered scheme, which reduces communication frequency and improves system efficiency. Additionally, Shi et al. [26] introduced a distributed adaptive event-triggered control strategy to solve the cooperative output regulation problem in heterogeneous linear multi-agent systems.

Nevertheless, abrupt changes in morphing unmanned aircraft, particularly shape-switching events, can surpass the predefined triggering conditions. This may result in overly frequent or missed trigger events, thereby introducing instability into the system. Adaptive neural network control emerges as a viable solution. It can manage nonlinear and high-dimensional systems, ensuring stability under unknown dynamics and external disturbances. Research [27,28] introduced a fault-tolerant control approach that combines an adaptive neural network with a nonlinear observer. This synergy improves the robustness and real-time decision-making capabilities of nonlinear systems. To optimize control performance, reinforcement learning has been integrated into the adaptive control framework, expanding its applicability. Reference [29] presented an adaptive model-free fault-tolerant control solution based on integral reinforcement learning for highly flexible aircraft with actuator failures. Reinforcement-learning-based methods, with their distinct advantages in handling nonlinearity, model-free scenarios, real-time learning, and multi-objective optimization, can surmount the control challenges of morphing unmanned aircraft in complex environments.

However, most existing design methods for proving the Lyapunov stability theory can only achieve asymptotic stability. That is, the system takes an infinite time to converge to the equilibrium point [30,31,32,33]. In many engineering applications, it is desired that the control objectives be achieved as soon as possible. Thus, finite-time control has emerged. Finite-time control is applicable to different types of systems. For example, references [34,35,36] proposed finite-time control methods for different types of nonlinear systems, ensuring the finite-time stability of the closed-loop systems. In the aerospace field, finite-time control is mainly used for spacecraft attitude tracking. For instance, reference [37] studied the vibration suppression and attitude tracking of flexible spacecraft under model uncertainties and external disturbances. It designs a multivariable finite-time control scheme for spacecraft attitude tracking based on a novel dynamic sliding dynamics and an adaptive disturbance observer (ADO). Reference [38] investigated the finite-time tracking control problem of a class of nonlinear systems and proposed a new finite-time command-filtered backstepping method. This method has the advantages of conventional command-filtered backstepping control and ensures finite-time convergence.

Research [39] designed a concurrent-learning adaptive finite-time control based on inertial parameter identification under external disturbances and applied it to aerospace attitude control. Reference [40] proposed a novel discrete-time fuzzy preselected performance control (PPC) method. By employing an indirect stabilization mechanism and a low-computation fuzzy approximation strategy, this approach achieves finite-time convergent control of unknown system dynamics without the need for complex model reconstruction. Furthermore, reference [41] developed a fixed-time pre-configured controller for electromechanical systems based on enhanced fuzzy neural approximation and backstepping design. Extending this research to hypersonic vehicles with elevator-stuck faults, the authors propose a fuzzy fault-tolerant control scheme that ensures appointed-time convergence [42]. Reference [43] studied the spacecraft formation flying system affected by external disturbances and parameter uncertainties; it designed a new improved, fast integral terminal sliding-mode control law, and proposed an adaptive tracking control for the spacecraft formation flying system.

Only a few studies focus on applying finite-time control to morphing unmanned aircraft. In the early stage, Wang et al. [44] proposed a smooth-switching state-feedback controller design method for the altitude-keeping and attitude-stability problems of morphing unmanned aircraft during continuous morphing. They establish a chained smooth-switching system model and derive sufficient conditions for finite-time boundedness and robustness. Subsequently, Cheng et al. [45] studied the asynchronous finite-time H∞ control problem of morphing unmanned aircraft with controller uncertainties. Considering the inherent packet dropouts of the system and controller uncertainties, they propose a non-fragile finite-time H∞ controller design method, and its effectiveness is verified by numerical examples. In recent years, some scholars have combined finite-time control with disturbance observers to improve the robustness of morphing unmanned aircraft [46,47].

Based on the above analysis, the application of reinforcement learning to the trajectory-tracking control of morphing unmanned aircraft has not been reported yet. In particular, the research on finite-time reinforcement learning in this field is lacking. Considering the strong coupling and uncertainties of morphing unmanned aircraft, the application of reinforcement learning can enhance its ability to deal with uncertainties. Meanwhile, finite-time reinforcement learning can help limit and shorten the system’s stabilization and control convergence time. Therefore, it is of great significance to improve the trajectory-tracking control performance of morphing unmanned aircraft, and this paper will conduct in-depth research on this.

Based on the above analysis, the main contributions of this paper are as follows:

• The attitude dynamics of the morphing unmanned aircraft are described as a class of mismatched non-affine systems, including matched and mismatched disturbances, non-affine input, and internal uncertainties. Compared to previous models [14,48,49], the proposed model is more applicable to practical scenarios and simplifies the analysis process.
• Compared with the literature [13], our work focuses on finite-time control, mismatched disturbances, and coupled uncertainties, while the literature [13] addresses non-affine control systems and control input frequency constraints. Different from the literature [27] on highly flexible aircraft, our method targets morphing unmanned aircraft, devises adaptive finite-time controllers, and ensures finite-time attitude convergence with better performance.
• This paper proposes a design framework for RL-based adaptive anti-disturbance flight control, offering a paradigm that can be extended to various aircraft.

The remainder of this paper is structured as follows. In Section 2, the problem is formulated, and preliminary knowledge is introduced. Section 3 presents the main findings. Section 4 provides simulation studies. Finally, Section 5 concludes the paper.

2. Problem Formulation and Preliminaries

2.1. Dynamic Model Description

The attitude dynamics model of a morphing unmanned aircraft can be formulated as follows [8,14]:

$$\begin{array}{l}\left[\begin{array}{c}\dot{\alpha }\\ \dot{\beta }\\ {\dot{\gamma }}_v\end{array}\right]=\left[\begin{array}{ccc}-\cos \alpha \tan \beta & \sin \alpha \tan \beta & 1\\ \sin \alpha & \cos \alpha & 0\\ \cos \alpha \sec \beta & -\sin \alpha \sec \beta & 0\end{array}\right]\left[\begin{array}{c}{\omega }_x\\ {\omega }_y\\ {\omega }_z\end{array}\right]\\ -\left[\begin{array}{c}\sec \beta \cos {\gamma }_v\\ \sin {\gamma }_v\\ -\tan \beta \cos {\gamma }_v\end{array}\right]\dot{\theta }-\left[\begin{array}{c}-\sec \beta \cos \theta \sin {\gamma }_v\\ \cos \theta \cos {\gamma }_v\\ -\sin \theta +\tan \beta \cos \theta \sin {\gamma }_v\end{array}\right]\dot{\sigma },\end{array}$$

(1)

where $\alpha$ is the angle of attack, $\beta$ is the sideslip angle, ${\gamma }_v$ is the tilt angle, $\theta$ is the velocity inclination angle, and $\sigma$ is the track yaw angle, and ${\omega }_x,{\omega }_y,{\omega }_z$ are the angular velocities of the roll, yaw, and pitch channels, respectively.

The longitudinal dynamics model of the aircraft is as follows:

$$\dot{\omega }={\left(I_b+{\displaystyle \sum _{j=L,r}{I}_{bj}}\right)}^{-1}\left(M_c-M_f-{\displaystyle \sum _{j=L,r}{m}_j}\left(r_{Om\to Oj}+r_{Oj\to c_j}\right)\times \left(v+\omega \times v\right)\right),$$

(2)

where $\omega ={[{\omega }_x,{\omega }_y,{\omega }_z]}^T$, $I_b$ denotes the moment of inertia of the aircraft body, $I_{bj}$ denotes the moment of inertia of the wing, $M_c$ represents the aerodynamic torque associated with control ability, $M_f$ is the additional moment caused by deformation, $r_{Om\to Oj}$ is the rotation vector of the wing relative to the fuselage in this system, $r_{Oj\to c_j}$ is the rotation vector of the wing relative to the fuselage under the wing mounting system, $v$ represents the matrix of velocity magnitudes, and $m_j$ is the mass of the aircraft wing, respectively. The detailed descriptions of some notations are as follows.

The $I_{bj}$ is expressed as:

$$\begin{array}{l}{I}_{bj}=-m_j{r}_{Om\to Oj}^{\times }\times r_{Om\to Oj}^{\times }-m_j{R}_{bj}{r}_{Oj\to c_j}^{\times }\times R_{bj}^T{r}_{Om\to Oj}^{\times }\\ -m_j{r}_{Om\to Oj}^{\times }{R}_{bj}{r}_{Oj\to c_j}^{\times }\times R_{bj}^T{r}_{Om\to Oj}^{\times }+R_{bj}{I}_j{R}_{bj}^T,\end{array}$$

(3)

where $r_{Oj\to c_j}^{\times }$ and $r_{Om\to Oj}^{\times }$ represent the cross-product vector matrices of $r_{Oj\to c_j}$ and $r_{Om\to Oj}$, respectively, $R_{bj}$ is a rotation matrix from the wing installation frame to the body coordinate system, and is expressed as follows:

$$\left\{\begin{array}{l}{R}_{bl}={\left[\begin{array}{c}{R}_y({90}^{°}+{\chi }_l)\end{array}\right]}^T\hfill \\ R_{br}={\left[\begin{array}{c}{R}_y(-{90}^{°}-{\chi }_r)\end{array}\right]}^T,\hfill \end{array}\right.$$

(4)

where $R_y$ is the basic rotation matrix of the $y$ axis, and $\chi$ is the caster angle.

The $M_{\mathrm{c}}$ can be given as follows:

$$M_{\mathrm{c}}=\left[\begin{array}{c}{M}_{cx}\\ M_{cy}\\ M_{cz}\end{array}\right]={\displaystyle \frac{1}{2}}\rho v^2{S}_0{L}_0\left[\begin{array}{c}{m}_{cx}\\ m_{cy}\\ m_{cz}\end{array}\right],$$

(5)

where $\rho$ represents the atmospheric density, $v$ is the velocity magnitude, $S_0$ represents the reference area, $L_0$ is the reference length, $m_{cx},m_{cy},m_{cz}$ represents the rotation moment coefficient, yaw moment coefficient, and pitch moment coefficient, respectively.

The determination of $m_{ci}$ can be presented by

$$m_{ci}=m_{ci}(\alpha ,\beta ,\chi ,{\delta }_x,{\delta }_y,{\delta }_z),\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i=x,y,z,$$

(6)

where ${\delta }_x$, ${\delta }_y$ and ${\delta }_z$ represent the roll angle, yaw angle, and pitch angle, respectively.

The $M_f$ can be given as follows:

$$\begin{array}{l}{M}_{\mathrm{f}}={\omega }_b\times \left(I_b{\omega }_b\right)+{\displaystyle \sum _{j=L,r}{I}_{bj}}{\dot{\omega }}_j+{\displaystyle \sum _{j=L,r}{m}_j}{r}_{Om\to Oj}\times \left[{\omega }_b\times \left({\omega }_b\times r_{Om\to Oj}\right)\right]\\ +{\displaystyle \sum _{j=L,r}{m}_j}{r}_{Oj\to c_j}\times \left[{\omega }_b\times \left({\omega }_b\times r_{O_m\to O_j}\right)\right]\\ +{\displaystyle \sum _{j=L,r}{m}_j}{r}_{O_m\to O_j}\times \left\{\left({\omega }_b+{\omega }_j\right)\times \left[\left({\omega }_b+{\omega }_j\right)\times r_{Oj\to c_j}\right]\right\},\end{array}$$

(7)

where $I_{bj}=R_{bj}{I}_b{R}_{bj}^T,$ ${\omega }_b$ represents the angular velocity around the center of mass $O_m$ of the aircraft in the body coordinate system, ${\omega }_j$ and ${\dot{\omega }}_j$ are the angular velocity and angular acceleration, respectively.

By defining the state variables $x_1=\left[\begin{array}{ccc}{x}_{11}& x_{12}& x_{13}\end{array}\right]={\left[\begin{array}{ccc}\alpha & \beta & {\gamma }_v\end{array}\right]}^T$ and $x_2={\left[\begin{array}{ccc}{x}_{21}& x_{22}& x_{23}\end{array}\right]}^T=$ ${\left[\begin{array}{ccc}{\omega }_x& {\omega }_y& {\omega }_z\end{array}\right]}^T$, control input $u={\left[\begin{array}{ccc}{\delta }_x& {\delta }_y& {\delta }_z\end{array}\right]}^T$, then (1) and (2) can be simplified in the following vector form with uncertain mismatched non-affine functions:

$$\begin{array}{l}{\dot{x}}_1\left(t\right)=f_1\left(x_1\left(t\right)\right)+\Delta f_1\left(x_1\left(t\right)\right)+g_1\left(x_1\left(t\right),x_2\left(t\right)\right)+d_1\left(t\right)\\ {\dot{x}}_2\left(t\right)=f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)+d_2\left(t\right),\end{array}$$

(8)

where $\Delta f_1\left(x_1\left(t\right)\right)$ and $\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)$ are the unknown structural uncertainties caused by the deformation of the aircraft, $d_1\left(t\right)$ and $d_2\left(t\right)$ represent the various unknown disturbances encountered during flight, $g_1\left(x_1\left(t\right),x_2\left(t\right)\right)$ and $g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)$ denotes the non-affine functions, some notations are detailed as:

$$\begin{array}{l}{f}_1\left(x_1\left(t\right)\right)=-{\left[\begin{array}{ccc}\sec \beta \cos {\gamma }_v& \sin {\gamma }_v& -\tan \beta \cos {\gamma }_v\end{array}\right]}^T\dot{\theta }\\ -{\left[\begin{array}{lll}-\sec \beta \cos \theta \sin {\gamma }_v\hfill & \cos \theta \cos {\gamma }_v\hfill & -\sin \theta +\tan \beta \cos \theta \sin {\gamma }_v\hfill \end{array}\right]}^T\dot{\sigma },\end{array}$$

(9)

$$\begin{array}{l}{f}_2\left(x_1\left(t\right),x_2\left(t\right)\right)=-{\left(I_b+{\displaystyle \sum _{j=L,r}{I}_{bj}}\right)}^{-1}{M}_{\mathrm{f}}\\ -{\left(I_b+{\displaystyle \sum _{j=L,r}{I}_{bj}}\right)}^{-1}\left({\displaystyle \sum _{j=L,r}{m}_j}\left(r_{Om\to Oj}+r_{Oj\to c_j}\right)\times \left(v+\omega \times v\right)\right).\end{array}$$

(10)

$$\begin{array}{l}{g}_1\left(x_1\left(t\right),x_2\left(t\right)\right)=\left[\begin{array}{ccc}-\cos \alpha \tan \beta & \sin \alpha \tan \beta & 1\\ \sin \alpha & \cos \alpha & 0\\ \cos \alpha \sec \beta & -\sin \alpha \sec \beta & 0\end{array}\right]\left[\begin{array}{c}{\omega }_x\\ {\omega }_y\\ {\omega }_z\end{array}\right],\\ g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)={\left(I_b+{\displaystyle \sum _{j=L,r}{I}_{bj}}\right)}^{-1}\left[\begin{array}{l}{m}_{cx}(\alpha ,\beta ,\chi ,{\delta }_x,{\delta }_y,{\delta }_z)\\ m_{cy}(\alpha ,\beta ,\chi ,{\delta }_x,{\delta }_y,{\delta }_z)\\ m_{cz}(\alpha ,\beta ,\chi ,{\delta }_x,{\delta }_y,{\delta }_z)\end{array}\right].\end{array}$$

(11)

Remark 1. 

In the research of morphing unmanned aircraft control system modeling, to conform to real-world engineering practices and simplify the analysis, specific symbols are defined. $\Delta f_1\left(x_1\left(t\right)\right)$ and $\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)$ stand for unknown structural uncertainties due to aircraft deformation, $d_1\left(t\right)$ and $d_2\left(t\right)$ denote various unknown flight disturbances. These flight disturbances encompass sensor biases that can lead to parameter uncertainties, as well as actuator failures, which are a particularly severe type of disturbance. Additionally, $g_1\left(x_1\left(t\right),x_2\left(t\right)\right)$ and $g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)$ represents non-affine functions. Furthermore, analysis of (9)–(11) shows that system (8) has significant non-affine features, posing great challenges to morphing unmanned aircraft control design.

Thereafter, when there is no risk of confusion, vectors may not be bolded, and function arguments may be omitted.

Based on the above analysis and simplification, the control-oriented morphing unmanned aircraft attitude model is obtained as an uncertain mismatched non-affine system as follows:

$$\begin{array}{l}{\dot{x}}_1\left(t\right)=f_1\left(x_1\left(t\right)\right)+\Delta f_1\left(x_1\left(t\right)\right)+g_1\left(x_1\left(t\right),x_2\left(t\right)\right)+d_1\left(t\right)\\ {\dot{x}}_2\left(t\right)=f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)+d_2\left(t\right)\\ y\left(t\right)=x_1\left(t\right),\end{array}$$

(12)

where $y\left(t\right)$ represents the system output vector.

To design an RL-based adaptive fault-tolerant controller for a morphing unmanned aircraft facing external disturbances, unknown dynamics, and non-affine inputs, this paper sets two control goals: (a) All the signals in the closed-loop system are semi-global practical finite-time stable. (b) The racking errors converge to a small neighborhood of origin in a finite time.

The following assumptions and lemmas are necessary to design the controller.

Assumption 1. 

To ensure the validity of the input, it is assumed that $\partial g_1/\partial x_1,$ $\partial g_1/\partial x_2,\partial g_2/\partial x_1,\partial g_2/\partial x_2,\partial g_2/\partial u$ are all invertible matrices, and it has [50,51]:

$$\lambda \left[{\displaystyle \frac{\partial g_i}{\partial x_1}}\right]\le \underset{\_}{\pi }, \lambda \left[{\displaystyle \frac{\partial g_i}{\partial x_2}}\right]\le \underset{\_}{\pi }, \lambda \left[{\displaystyle \frac{\partial g_2}{\partial u}}\right]\le \underset{\_}{\pi },i=1,2,$$

(13)

where $\lambda \left[\partial g_i/\partial x_1\right], \lambda \left[\partial g_i/\partial x_2\right]$ and $\lambda \left[\partial g_2/\partial u\right]$ represents the eigenvalue of the matrix $\partial g_i/\partial x_1,\partial g_i/\partial x_2$ and $\partial g_2/\partial u$, respectively.

Assumption 2. 

The desired trajectory $y_d$ and its first derivative are continuous and bounded [52].

Definition 1. 

If for all $ϵ(t_0)={ϵ}_0$, there exist $\mathcal{l}>0$ and a settling time $T(\mathcal{l},{ϵ}_0)<\infty$ such that $\Vert ϵ(t)\Vert <\mathcal{l}$ holds for all $t\ge t_0+T$, then the equilibrium $ϵ=0$ of the nonlinear system $\dot{ϵ}=f(ϵ)$ is semi-global practical finite-time stable [53].

Lemma 1. 

For any $ƛ>0$ and $x\in R$, there exists [49,53]:

$$0\le \left|x\right|-x\tanh \left({\displaystyle \frac{x}{ƛ}}\right)\le \kappa ƛ,$$

(14)

where $\kappa =0.2785$.

Lemma 2. 

Given any constants $\Im >0$, $0<l<1$, and $\wp >0$, consider system $\dot{ϵ}=f(ϵ)$ [54]. If there exists a smooth positive-definite function $V(\eta )$ such that:

$$\dot{V}(ϵ)\le -\Im V^l(ϵ)+\wp ,t\ge 0,$$

(15)

holds, then system $\dot{ϵ}=f(ϵ)$ is semi-global practical finite-time stable.

Lemma 3. 

For any real variables $p,q$ and any positive constants $a_1,a_2,\hslash$, the following inequality holds [55,56]:

$${\left|p\right|}^{a_1}{\left|q\right|}^{a_2}\le {\displaystyle \frac{a_1}{a_1+a_2}}\hslash {\left|p\right|}^{a_1+a_2}+{\displaystyle \frac{a_2}{a_1+a_2}}{\hslash }^{-{\displaystyle \frac{a_1}{a_2}}}{\left|q\right|}^{a_1+a_2}.$$

(16)

Lemma 4. 

For ${\mathsf{{\rm Y}}}_i\in R$ and $0<\gamma \le 1$, the following inequality holds [57]:

$${\left({\displaystyle \sum _{i=1}^n\left|{\mathsf{{\rm Y}}}_i\right|}\right)}^{\gamma }\le {{\displaystyle \sum _{i=1}^n\left|{\mathsf{{\rm Y}}}_i\right|}}^{\gamma }\le n^{1-\gamma }{\left({\displaystyle \sum _{i=1}^n\left|{\mathsf{{\rm Y}}}_i\right|}\right)}^{\gamma }.$$

(17)

2.2. Designs of Actor–Critic Neural Networks

Research shows that radial basis function neural networks (RBFNNs) can approximate any smooth continuous function $f\left(x\right)$ with any precision [58]. As a current research hotspot, the actor–critic framework enables the critic network to receive information from the environment and evaluate control performance through a cost function. Based on this evaluation, the actor network generates a control strategy for the actuator. Owing to the incorporation of reinforcement signals, the resulting control law exhibits faster convergence and reduced steady-state error. Based on this, an RL framework is developed in this study, using RBFNNs to approximate unknown nonlinear functions and penalty functions, namely:

$$f\left(x\right)=W_a^{\ast T}{\mathsf{\Phi }}_a\left(x\right)+{\epsilon }_a(x),\left|{\epsilon }_a\left(x\right)\right|\le {\overline{\epsilon }}_a,$$

(18)

where $x={\left[x_1,x_2,\dots x_n\right]}^T$ is the input vector of the RBFNN, $\epsilon \left(x\right)$ is the approximation error, $W_a^{\ast }$ is the optimized vector of the weight vector $W_a={\left[{\underset{\_}{\upsilon }}_1,{\underset{\_}{\upsilon }}_2,\dots ,{\underset{\_}{\upsilon }}_p\right]}^T$, and ${\mathsf{\Phi }}_a(x)={\left[{\mathsf{\Phi }}_1(x),\dots ,{\mathsf{\Phi }}_p(x)\right]}^T$ is the basis function vector, $p$ is the number of hidden nodes, ${\mathsf{\Phi }}_j(x)=e^{-{(x-{\mu }_j)}^2/(2{\sigma }_j{\phantom{,}}^2)},j=1,\dots ,p$ is the basis function, ${\mu }_j$ is the center of the basis function, ${\sigma }_j$ is the width of the Gaussian basis function, ${\epsilon }_a(x)$ is the approximation error, and ${\overline{\epsilon }}_a$ is a positive integer.

When designing the actor–critic neural networks, several important design principles of the weight adaptive law in reinforcement learning are considered to ensure the effectiveness and stability of the system.

To approximate the unknown nonlinearity $\Delta f_i\left({\overline{x}}_i\right)$ in (12), an actor neural network $W_{ai}^T{\mathsf{\Phi }}_{ai}\left({\overline{x}}_i\right)$ is introduced, which is expressed as follows:

$$\Delta f_i\left({\overline{x}}_i\right)=W_{ai}^T{\mathsf{\Phi }}_{ai}\left({\overline{x}}_i\right)+{\epsilon }_{ai},$$

(19)

where ${\overline{x}}_i={\left[x_1,\dots x_i\right]}^T\in R^i,i=1,2$, $W_{ai}$ is the ideal actor neural network weight.

Define the weight error of the actor neural network as ${\tilde{W}}_{ai}={\widehat{W}}_{ai}-W_{ai}$, then its approximation error is expressed as:

$$H_{ai}={\tilde{W}}_{ai}^T{\mathsf{\Phi }}_{ai}\left({\overline{x}}_i\right).$$

(20)

To improve the tracking performance, a new error function of the actor neural network is constructed based on the approximation error and the penalty function:

$$\begin{array}{l}{e}_{ai}=H_{ai}+{\Gamma }_i{\widehat{J}}_i,\\ E_{ai}={\displaystyle \frac{1}{2}}{e}_{ai}^T{e}_{ai},\end{array}$$

(21)

where ${\Gamma }_i$ is the gain to be chosen, and the goal of updating the network weights to minimize $E_{ai}$.

Based on the information gathered from the environment, an integral penalty function is constructed to generate RL signals:

$$J_i(t)={\displaystyle {\int }_{\tau }^{\infty }{q}_i}(t)dt,$$

(22)

where $q_i(t)=z_i^T{Q}_i{z}_i$.

The specific meaning of the variables will be given later. The penalty function can be approximated by a critic neural network:

$$\begin{array}{l}{J}_i=W_{ci}^T{\mathsf{\Phi }}_{ci}\left({\overline{x}}_i\right)+{\epsilon }_{ci},\\ {\widehat{J}}_i={\widehat{W}}_{ci}^T{\mathsf{\Phi }}_{ci}\left({\overline{x}}_i\right),\end{array}$$

(23)

where $W_{ci}$ is the ideal critic neural network weight.

The update rule for the weights of the actor–critic network is designed as follows:

$${\dot{\widehat{W}}}_{ai}=-{\tau }_i{\eta }_i{\widehat{W}}_{ai}-{\eta }_i{\mathsf{\Phi }}_{ai}\left({\overline{x}}_i\right)\left[{\mathsf{\Phi }}_{ai}^T{\widehat{W}}_{ai}+{\widehat{J}}_i{\Gamma }_i^T\right],$$

(24)

where ${\eta }_i>0,{\tau }_i>0$ are the parameters to be chosen.

Construct the residual mean square error function of the critic network:

$$\begin{array}{l}{e}_{ci}=q_i(t)+{\dot{\widehat{J}}}_i=q_i(t)+{\widehat{W}}_{ci}^T{\dot{\mathsf{\Phi }}}_{ci}\left({\overline{x}}_i\right),\\ E_{ci}={\displaystyle \frac{1}{2}}{e}_{ci}^T{e}_{ci}.\end{array}$$

(25)

The update rule for the weights of the critic network is derived using the gradient descent method as follows:

$${\dot{\widehat{W}}}_{ci}=-{\varpi }_i\left({\widehat{W}}_{ci}^T{\dot{\mathsf{\Phi }}}_{ci}+q_i(t)\right){\dot{\mathsf{\Phi }}}_{ci}-{\omega }_i{\varpi }_i{\widehat{W}}_{ci},$$

(26)

where ${\omega }_i$ and ${\varpi }_i$ are the positive design constants.

By applying these design principles, we can design an effective weight adaptive law for the actor–critic neural networks, which helps to improve the performance of the control system.

Remark 2. 

The $\widehat{J}$ term approximates the unknown value function, enabling the actor to consider future rewards. This has helped the actor learn an optimal long-term policy, as focusing only on immediate rewards would lead to sub-optimal behavior. The second correction term enhances the stability and convergence of the actor–critic algorithm. It can adjust the actor’s update step, preventing aggressive updates that could cause divergence or oscillation. Moreover, it accounts for environmental uncertainties, improving the system’s robust ability to handle unexpected situations.

Property 1. 

To satisfy the persistence excitation condition, the basis functions ${\mathsf{\Phi }}_{ci}$ and ${\mathsf{\Phi }}_{ai}$, and the derivatives ${\dot{\mathsf{\Phi }}}_{ci}$ and ${\dot{\mathsf{\Phi }}}_{ai}$ are assumed to satisfy $‖{\dot{\mathsf{\Phi }}}_{ai}‖\le {\mathsf{\Phi }}_{aim},‖{\dot{\mathsf{\Phi }}}_{ci}‖\le {\mathsf{\Phi }}_{cim},‖{\mathsf{\Phi }}_{ai}‖$ $\le {\mathsf{\Phi }}_{aiM},‖{\mathsf{\Phi }}_{ci}‖\le {\mathsf{\Phi }}_{ciM},i=1,2$. Meanwhile, the estimation errors of actor–critic learning and their derivative are bounded, that is $\left|{\epsilon }_{ci}\right|\le {\epsilon }_{ciM}$ and $\left|{\dot{\epsilon }}_{ci}\right|\le {\epsilon }_{cim}$ [51].

Remark 3. 

RBFNNs are selected over other function approximations for several well-founded reasons. Firstly, owing to their universal approximation property, RBFNNs are capable of accurately approximating uncertainties [52]. Secondly, for high-dimensional data, RBFNNs offer computational efficiency as their training process is relatively straightforward.

Remark 4. 

When determining the centers and widths, a trial-and-error technique is used to process the input data. The centers are set in high-density regions, and prior knowledge of the system’s critical operating points is incorporated. The widths are set according to the distances between the centers and fine-tuned on the validation set.

Remark 5. 

The penalty function (22) and update laws (24) and (26) are designed with the aim of ensuring the stability and convergence of the RL framework. The penalty function serves to penalize undesired system behaviors, and the update laws are derived from Lyapunov techniques. The detailed derivations of these aspects will be presented in the revised manuscript.

Remark 6. 

The actor–critic neural network and RBFNN architectures, learning mechanisms, and generalization abilities are designed to suit different application scenarios. The actor–critic is for dynamic scenarios, while RBFNN excels at local function approximation.

3. Main Results

This section details the design process of the RL-based adaptive fault-tolerant control method. First, by introducing an auxiliary integral term, the system is reformulated as an augmented affine system with the overall non-affine function treated as the control input. Next, a virtual control law is designed, which integrates the actor–critic network and the disturbance boundary estimator. The final control signal is obtained through recursive design, and the actual control input is generated through integration. The structure of the controller is shown in Figure 1.

3.1. The Design of the Augmented System

Through the auxiliary integration method, the augmented system is established as follows:

$$\begin{array}{l}{\dot{x}}_1\left(t\right)=f_1\left(x_1\left(t\right)\right)+\Delta f_1\left(x_1\left(t\right)\right)+g_1\left(x_1\left(t\right),x_2\left(t\right)\right)+d_1\left(t\right)\\ {\dot{x}}_2\left(t\right)=f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)+g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)+d_2\left(t\right)\\ \dot{u}=u_f\\ y\left(t\right)=x_1\left(t\right),\end{array}$$

(27)

where $d_1\left(t\right)$ denotes mismatched disturbance, $d_2\left(t\right)$ denotes matched disturbance, which are compensated by the Tanh function boundary estimator. Additionally, unknown nonlinear functions $\Delta f_1\left(x_1\left(t\right)\right),\Delta f_2\left(x_1\left(t\right),x_2\left(t\right)\right)$ are approximated using an actor–critic framework.

Different from the affine system, non-affine systems treat the non-affine function holistically as the subsystem input, with the tracking error formally defined as follows:

$$\begin{array}{l}{z}_1\left(t\right)=x_1\left(t\right)-y_d\\ z_2\left(t\right)=g_1\left(x_1\left(t\right),x_2\left(t\right)\right)-x_{1d}\left(t\right)\\ z_3\left(t\right)=g_2\left(x_1\left(t\right),x_2\left(t\right),u\left(t\right)\right)-x_{2d}\left(t\right),\end{array}$$

(28)

where $y_d$ is the desired output value of the system, $x_{1d}$ and $x_{2d}$, the constructed virtual control variables, will be presented below.

Let $x_{ic}$ denote the virtual control law to be designed for the i-th subsystem. To mitigate differential explosion, a first-order low-pass filter [58] is introduced as:

$${\delta }_i{\dot{x}}_{id}+x_{id}=x_{ic},x_{id}\left(0\right)=x_{ic}\left(0\right),i=1,2,$$

(29)

where $0<{\delta }_i<1$ denotes the filter time constant to be designed, through which both $x_{id}$ and its derivative can be obtained. The boundary layer error is defined as:

$$y_i=x_{id}-x_{ic},$$

(30)

where $x_{ic}$ denotes the filtered value of the virtual controller $x_{id}$.

Then we can obtain:

$${\dot{y}}_i={\dot{x}}_{id}-{\dot{x}}_{ic}=-{\displaystyle \frac{y_i}{{\delta }_i}}-{\dot{x}}_{ic}.$$

(31)

3.2. Controller Design

Step 1. From the first equality in Equation (28), it yields:

$$\begin{array}{l}{\dot{z}}_1\left(t\right)={\dot{x}}_1\left(t\right)-{\dot{y}}_d\\ \hspace{1em}=f_1\left(x_1\left(t\right)\right)+\Delta f_1\left(x_1\left(t\right)\right)+g_1\left(x_1\left(t\right),x_2\left(t\right)\right)+d_1\left(t\right)-{\dot{y}}_d\\ \hspace{1em}=f_1+z_2+y_1+x_{1c}+d_1+\Delta f_1-{\dot{y}}_d.\end{array}$$

(32)

Let $\Delta {\widehat{f}}_1$ denote the estimated value of the inner-loop nonlinearity $\Delta f_1$. According to Section 2.2, it can be approximated using an actor network, expressed as:

$$\Delta f_1=W_{a1}^T{\mathsf{\Phi }}_{a1}(x_1)+{\epsilon }_{a1},$$

(33)

where ${\epsilon }_{a1}$ is the error of the actor network.

Define $D_1={\sup }_{t\ge 0}‖d_1\left(t\right)+{\epsilon }_{a_1}‖$, thus, the virtual controller can be designed as:

$$x_{1c}=-k_1{\left(z_1^T{z}_1\right)}^{l-1}{z}_1-{\widehat{W}}_{a1}^T{\mathsf{\Phi }}_1\left(x_1\right)-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)-f_1+{\dot{y}}_d.$$

(34)

Let ${\widehat{D}}_1$ denote the estimated value of the mismatched disturbance. From Equations (32) and (34), we obtain:

$$z_1^T{\dot{z}}_1=-k_1{\left(z_1^T{z}_1\right)}^l+z_1^T\left(z_2+y_1\right)+z_1^T\left(d_1+{\epsilon }_{a1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)-z_1^T{\tilde{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right),$$

(35)

where ${\tilde{W}}_{a1}={\widehat{W}}_{a1}-W_{a1}$.

The integral penalty function of the control system is defined as: $J_1(t)={\displaystyle {\int }_{\tau }^{\infty }{q}_1(t)dt},$ where $q_1(t)=z_1{\phantom{,}}^T{Q}_1{z}_1$. Then, the actor–critic network weight updating law is designed as:

$${\dot{\widehat{W}}}_{a1} =-{\tau }_1{\eta }_1{\widehat{W}}_{a1}-{\eta }_1{\mathsf{\Phi }}_{a1}(x_1)[{\mathsf{\Phi }}_{a1}{\phantom{,}}^T{\widehat{W}}_{a1}+{\widehat{J}}_1{\Gamma }_1^T] .$$

(36)

The critic network weight updating law is derived via the gradient descent method as:

$${\dot{\widehat{W}}}_{c1}=-{\varpi }_1({\dot{\widehat{W}}}_{c1}^T{\dot{\mathsf{\Phi }}}_{c1}+q_1(t)){\dot{\mathsf{\Phi }}}_{c1}-{\omega }_1{\varpi }_1{\widehat{W}}_{c1}.$$

(37)

The updating law of ${\widehat{D}}_1$ is given as:

$${\dot{\widehat{D}}}_1={\tau }_{D_1}{z}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)-{\tau }_{D_1}{\gamma }_{D_1}{\widehat{D}}_1.$$

(38)

Step 2. Define $D_2={\sup }_{t\ge 0}‖d_2\left(t\right)+{\epsilon }_{a2}‖$, where ${\widehat{D}}_2$ is the estimated value. According to Equation (28), we have:

$$\begin{array}{l}{\dot{z}}_2={\dot{g}}_1-{\dot{x}}_{1d}\\ \hspace{1em}={\displaystyle \frac{\partial g_1}{\partial x_1}}{\dot{x}}_1+{\displaystyle \frac{\partial g_1}{\partial x_2}}{\dot{x}}_2-{\dot{x}}_{1d}\\ \hspace{1em}={\displaystyle \frac{\partial g_1}{\partial x_1}}\left(f_1+g_1+d_1+\Delta f_1\right)+{\displaystyle \frac{\partial g_1}{\partial x_2}}\left(f_2+\Delta f_2+z_3+y_2+x_{2c}+d_2\right)-{\dot{x}}_{1d}.\end{array}$$

(39)

Introduce the actor network to compensate $\Delta f_2\left(t,x_1,x_2\right)$:

$$\begin{array}{l}\Delta f_2\left(t,x_1,x_2\right)=W_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)+{\epsilon }_{a2},\\ \Delta {\widehat{f}}_2\left(t,x_1,x_2\right)={\widehat{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right).\end{array}$$

(40)

The designed virtual control law is:

$$x_{2c}={\left({\displaystyle \frac{\partial g_1}{\partial x_2}}\right)}^{-1}\left(\begin{array}{l}-k_2{\left(z_2^T{z}_2\right)}^{l-1}{z}_2-({\displaystyle \frac{1}{2}}+{\displaystyle \frac{5}{2}}{\underset{\_}{\pi }}^2)z_2-{\displaystyle \frac{\partial g_1}{\partial x_1}}\left(f_1+g_1+{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)+{\widehat{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right)\right)\\ -{\displaystyle \frac{\partial g_1}{\partial x_2}}\left({\widehat{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)+f_2+{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)+{\dot{x}}_{1d}\end{array}\right).$$

(41)

Substituting into Equation (39), it yields:

$$\begin{array}{l}{z}_2^T{\dot{z}}_2=-k_2{\left(z_2^T{z}_2\right)}^l+{\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)-{\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2{\tilde{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right) +{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\left(z_3+y_2\right)\\ \hspace{1em} -{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2{\tilde{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)+{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)\end{array}$$

(42)

Similarly, the actor–critic framework is designed with the penalty function defined as: $J_2(t)={\displaystyle {\int }_{\tau }^{\infty }{q}_2(t)dt}$ and $q_2(t)=z_2{\phantom{,}}^T{Q}_2{z}_2$.

Construct the critic network module effect error function and design the weight update law as:

$$\begin{array}{l}{\dot{\widehat{W}}}_{a2} =-{\displaystyle \frac{\partial g_1}{\partial x_2}}{\eta }_2{\mathsf{\Phi }}_{a2}(x_2)[{\displaystyle \frac{\partial g_1}{\partial x_2}}{\mathsf{\Phi }}_{a2}{\phantom{,}}^T{\widehat{W}}_{a2}+{\widehat{J}}_2{\Gamma }_2^T]-{\eta }_2{\tau }_2{\widehat{W}}_{a2},\\ {\dot{\widehat{W}}}_{c2}=-{\varpi }_2({\dot{\widehat{W}}}_{{\phantom{,}}_{c2}}^T{\dot{\mathsf{\Phi }}}_{c2}+q_2(t)){\dot{\mathsf{\Phi }}}_{c2}-{\omega }_2{\varpi }_2{\widehat{W}}_{c2}.\end{array}$$

(43)

The updating law of ${\widehat{D}}_2$ is given as:

$${\dot{\widehat{D}}}_2={\displaystyle \frac{\partial g_1}{\partial x_2}}{\tau }_{D_2}{z}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}·{\displaystyle \frac{z_2}{{\epsilon }_{D2}}}\right)-{\tau }_{D_2}{\gamma }_{D_2}{\widehat{D}}_2.$$

(44)

Step 3. From the third equation of Equation (28), let $u_f\left(t\right)=\dot{u}\left(t\right)$, it can be obtained:

$$\begin{array}{l}{\dot{z}}_3={\dot{g}}_2-{\dot{x}}_{2d}\\ \hspace{1em}={\displaystyle \frac{\partial g_2}{\partial x_1}}{\dot{x}}_1+{\displaystyle \frac{\partial g_2}{\partial x_2}}{\dot{x}}_2+{\displaystyle \frac{\partial g_2}{\partial u}}\dot{u}-{\dot{x}}_{2d}\\ \hspace{1em}={\displaystyle \frac{\partial g_2}{\partial x_1}}\left(f_1+g_1+d_1+\Delta f_1\right)+{\displaystyle \frac{\partial g_2}{\partial x_2}}\left(f_2+g_2+\Delta f_2+d_2\right)+{\displaystyle \frac{\partial g_2}{\partial u}}{u}_f-{\dot{x}}_{2d}.\end{array}$$

(45)

The virtual control signal is designed as:

$$u_f={\left({\displaystyle \frac{\partial g_2}{\partial u}}\right)}^{-1}\left(\begin{array}{l}-k_3{\left(z_3^T{z}_3\right)}^{l-1}{z}_3-({\displaystyle \frac{1}{2}}+2{\underset{\_}{\pi }}^2)z_3-{\displaystyle \frac{\partial g_2}{\partial x_1}}\left(f_1+g_1+{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)+{\widehat{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right)\right)\\ -{\displaystyle \frac{\partial g_2}{\partial x_2}}\left(f_2+g_2+{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)+{\widehat{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)\right)+{\dot{x}}_{2d}\end{array}\right).$$

(46)

We can obtain:

$$\begin{array}{c}{z}_3^T{\dot{z}}_3=-k_3{\left(z_3^T{z}_3\right)}^l+{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)-{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3{\tilde{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right)\\ +{\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}·{\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)-{\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3{\tilde{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right).\end{array}$$

(47)

3.3. Proof of Stability

Theorem 1. 

For a second-order non-affine nonlinear system that satisfies Assumptions 1–2 and Lemmas 1–4, if the virtual control signals, adaptive laws, and reinforcement learning (RL) update laws are designed as described in Section 3.1, the closed-loop system can achieve the control the objectives (a) and (b).

Proof. 

① Choose the overall candidate Lyapunov function as:

$$\begin{array}{l}V={\displaystyle \frac{1}{2}}{z}_1^T{z}_1+{\displaystyle \frac{1}{2}}{z}_2^T{z}_2+{\displaystyle \frac{1}{2}}{z}_3^T{z}_3+{\displaystyle \frac{1}{2}}{y}_1^T{y}_1+{\displaystyle \frac{1}{2}}{y}_2^T{y}_2+{\displaystyle \frac{1}{2}}Tr({\tilde{W}}_{a1}^T{\eta }_1^{-1}{\tilde{W}}_{a1})+{\displaystyle \frac{1}{2}}Tr({\tilde{W}}_{c1}^T{\varpi }_1^{-1}{\tilde{W}}_{c1})\\ +{\displaystyle \frac{1}{2}}Tr({\tilde{W}}_{a2}^T{\eta }_2^{-1}{\tilde{W}}_{a2})+{\displaystyle \frac{1}{2}}Tr({\tilde{W}}_{c2}^T{\varpi }_2^{-1}{\tilde{W}}_{c2})+{\displaystyle \frac{1}{2{\tau }_{D1}}}{\tilde{D}}_1^T{\tilde{D}}_1+{\displaystyle \frac{1}{2{\tau }_{D2}}}{\tilde{D}}_2^T{\tilde{D}}_2\end{array}$$

(48)

By taking its derivative, we can obtain:

$$\begin{array}{l}\dot{V}=z_1^T{\dot{z}}_1+z_2^T{\dot{z}}_2+z_3^T{\dot{z}}_3+y_1^T{\dot{y}}_1+y_2^T{\dot{y}}_2+{\displaystyle \frac{1}{{\tau }_{D_1}}}{\tilde{D}}_1^T{\dot{\widehat{D}}}_1+{\displaystyle \frac{1}{{\tau }_{D_2}}}{\tilde{D}}_2^T{\dot{\widehat{D}}}_2+Tr({\tilde{W}}_{a1}^T{\eta }_1^{-1}{\dot{\widehat{W}}}_{a1})\\ \hspace{1em}\hspace{1em}+Tr({\tilde{W}}_{c1}^T{\varpi }_1^{-1}{\dot{\widehat{W}}}_{c1})+Tr({\tilde{W}}_{a2}^T{\eta }_2^{-1}{\dot{\widehat{W}}}_{a1})+Tr({\tilde{W}}_{c2}^T{\varpi }_2^{-1}{\dot{\widehat{W}}}_{c2})\\ \hspace{1em}=-k_1{\left(z_1^T{z}_1\right)}^l-k_2{\left(z_2^T{z}_2\right)}^l-k_3{\left(z_3^T{z}_3\right)}^l-{\displaystyle \frac{3}{2}}{z}_1^T{z}_1-({\displaystyle \frac{1}{2}}+{\displaystyle \frac{5}{2}}{\underset{\_}{\pi }}^2)z_2^T{z}_2-({\displaystyle \frac{1}{2}}+2{\underset{\_}{\pi }}^2)z_3^T{z}_3-{\displaystyle \frac{y_1^T{y}_1}{{\delta }_1}}\\ \hspace{1em}-{\dot{x}}_{1c}^T{y}_1-{\displaystyle \frac{y_2^T{y}_2}{{\delta }_2}}-{\dot{x}}_{2c}^T{y}_2+z_1\left(z_2+y_1\right)+{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\left(z_3+y_2\right)+\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)z_1\\ \hspace{1em}+({\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3)\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)+\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}·{\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right){\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\\ \hspace{1em}+{\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}·{\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)-{\tilde{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right)(z_1+{\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3)\\ \hspace{1em}-{\tilde{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)({\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3)+{\displaystyle \frac{1}{{\tau }_{D_1}}}{\tilde{D}}_1^T{\dot{\widehat{D}}}_1+{\displaystyle \frac{1}{{\tau }_{D_2}}}{\tilde{D}}_2^T{\dot{\widehat{D}}}_2+Tr({\tilde{W}}_{a1}^T{\eta }_1^{-1}{\dot{\widehat{W}}}_{a1})\\ \hspace{1em}+Tr({\tilde{W}}_{c1}^T{\varpi }_1^{-1}{\dot{\widehat{W}}}_{c1})+Tr({\tilde{W}}_{a2}^T{\eta }_2^{-1}{\dot{\widehat{W}}}_{a2})+Tr({\tilde{W}}_{c2}^T{\varpi }_2^{-1}{\dot{\widehat{W}}}_{c2}).\end{array}$$

(49)

According to Lemma 3, we have:

$$\begin{array}{l}-{\dot{x}}_{1c}^T{y}_1\le {\displaystyle \frac{y_1^T{y}_1}{2{\delta }_1}}+{\displaystyle \frac{{\delta }_1{\left|{\dot{x}}_{1c}\right|}^2}{2}}, -{\dot{x}}_{2c}^T{y}_2\le {\displaystyle \frac{y_2^T{y}_2}{2{\delta }_2}}+{\displaystyle \frac{{\delta }_2{\left|{\dot{x}}_{2c}\right|}^2}{2}},\\ z_1^T\left(z_2+y_1\right)+{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\left(z_3+y_2\right)\le z_1^T{z}_1+\left({\displaystyle \frac{1}{2}}+{\underset{\_}{\pi }}^2\right)z_2^T{z}_2+{\displaystyle \frac{1}{2}}{z}_3^T{z}_3+{\displaystyle \frac{1}{2}}{y}_1^T{y}_1+{\displaystyle \frac{1}{2}}{y}_2^T{y}_2,\\ ({\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3)\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)\le {\displaystyle \frac{1}{2}}{\underset{\_}{\pi }}^2{z}_2^T{z}_2+{\displaystyle \frac{1}{2}}{\underset{\_}{\pi }}^2{z}_3^T{z}_3+{\displaystyle \frac{3}{2}}{\overline{\epsilon }}_{D_1}^2,\\ {\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)\le {\displaystyle \frac{1}{2}}{\underset{\_}{\pi }}^2{z}_3^T{z}_3+{\displaystyle \frac{1}{2}}{\overline{\epsilon }}_{D_2}^2,\\ -{\tilde{W}}_{a1}^T{\mathsf{\Phi }}_{a1}\left(x_1\right)(z_1+{\displaystyle \frac{\partial g_1}{\partial x_1}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_1}}{z}_3)\le {\displaystyle \frac{1}{2}}{z}_1^T{z}_1+{\displaystyle \frac{1}{2}}{\underset{\_}{\pi }}^2(z_2^T{z}_2+z_3^T{z}_3)+{\displaystyle \frac{3}{2}}{\mathsf{\Phi }}_{a1M}^{2}Tr\left({\tilde{W}}_{a1}{\phantom{,}}^T{\tilde{W}}_{a1}\right),\\ -{\tilde{W}}_{a2}^T{\mathsf{\Phi }}_{a2}\left(x_1,x_2\right)({\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2+{\displaystyle \frac{\partial g_2}{\partial x_2}}{z}_3)\le {\displaystyle \frac{1}{2}}{\underset{\_}{\pi }}^2(z_2^T{z}_2+z_3^T{z}_3)+{\mathsf{\Phi }}_{a2M}^{2}Tr\left({\tilde{W}}_{a2}{\phantom{,}}^T{\tilde{W}}_{a2}\right).\end{array}$$

(50)

It follows from Lemma 1 that:

$$\begin{array}{l}{z}_1\left(d_1+{\epsilon }_{a_1}-{\widehat{D}}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)\right)\le -{\tilde{D}}_1{z}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)+\kappa {\epsilon }_{D_1},\\ {\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\left(d_2+{\epsilon }_{a_2}-{\widehat{D}}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)\right)\le -{\tilde{D}}_2{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}\cdot {\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)+\kappa {\epsilon }_{D_2}.\end{array}$$

(51)

Combing Equations (38) and (44), we can obtain:

$$\begin{array}{l}{\displaystyle \frac{1}{{\tau }_{D_1}}}{\tilde{D}}_1^T{\dot{\widehat{D}}}_1={\tilde{D}}_1{z}_1\tanh \left({\displaystyle \frac{z_1}{{\epsilon }_{D_1}}}\right)-{\gamma }_{D_1}{\widehat{D}}_1{\tilde{D}}_1,\\ {\displaystyle \frac{1}{{\tau }_{D_2}}}{\tilde{D}}_2^T{\dot{\widehat{D}}}_2={\tilde{D}}_2{\displaystyle \frac{\partial g_1}{\partial x_2}}{z}_2\tanh \left({\displaystyle \frac{\partial g_1}{\partial x_2}}·{\displaystyle \frac{z_2}{{\epsilon }_{D_2}}}\right)-{\gamma }_{D_2}{\widehat{D}}_2{\tilde{D}}_2.\end{array}$$

(52)

Through transforming the inequalities, we derive the following relationship:

$$\begin{array}{l}-{\gamma }_{D_1}{\widehat{D}}_1^T{\tilde{D}}_1\le -{\displaystyle \frac{1}{2}}{\gamma }_{D_1}{\tilde{D}}_1^T{\tilde{D}}_1+{\displaystyle \frac{1}{2}}{\gamma }_{D_1}{D}_1^T{D}_1,\\ -{\gamma }_{D_2}{\widehat{D}}_2^T{\tilde{D}}_2\le -{\displaystyle \frac{1}{2}}{\gamma }_{D_2}{\tilde{D}}_2^T{\tilde{D}}_2+{\displaystyle \frac{1}{2}}{\gamma }_{D_2}{D}_2^T{D}_2,\\ Tr({\tilde{W}}_{a1}^T{\eta }_1^{-1}{\dot{\tilde{W}}}_{a1}^{\phantom{,}})+Tr({\tilde{W}}_{c1}^T{\varpi }_1^{-1}{\dot{\tilde{W}}}_{c1}^{\phantom{,}})\\ \le -{\displaystyle \frac{1}{2}}\left({\tau }_1-{\mathsf{\Phi }}_{a1M}^2\right)Tr\left({\tilde{W}}_{a1}{\phantom{,}}^T{\tilde{W}}_{a1}\right)-{\displaystyle \frac{1}{2}}\left({\omega }_1-{\mathsf{\Phi }}_{c1m}^2-{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right)Tr\left({\tilde{W}}_{c1}{\phantom{,}}^T{\tilde{W}}_{c1}\right)\\ +{\displaystyle \frac{1}{2}}\left({\tau }_1+{\mathsf{\Phi }}_{a1M}^2\right)Tr(W_{a1}{\phantom{,}}^T{W}_{a1})+{\displaystyle \frac{1}{2}}\left({\omega }_1+2{\mathsf{\Phi }}_{c1m}^2+{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right)Tr\left(W_{c1}^T{W}_{c1}\right)+{\displaystyle \frac{1}{2}}{\epsilon }_{c1m}^2,\\ Tr({\tilde{W}}_{{\phantom{,}}_{a2}}^T{\eta }_2^{-1}{\dot{\tilde{W}}}_{{\phantom{,}}_{a2}}^{\phantom{,}})+Tr({\tilde{W}}_{{\phantom{,}}_{c2}}^T{\varpi }_2^{-1}{\dot{\tilde{W}}}_{{\phantom{,}}_{c2}}^{\phantom{,}})\\ \le -{\displaystyle \frac{1}{2}}({\tau }_2-{\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2-2\pi {\mathsf{\Phi }}_{a2M}^2)Tr({\tilde{W}}_{a2}{\phantom{,}}^T{\tilde{W}}_{a2})-{\displaystyle \frac{1}{2}}({\omega }_2-{\mathsf{\Phi }}_{c2m}^2-\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2)Tr({\tilde{W}}_{{\phantom{,}}_{c2}}^T{\tilde{W}}_{c2})\\ +{\displaystyle \frac{1}{2}}({\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2+{\tau }_2)Tr\left(W_{a2}{\phantom{,}}^T{W}_{a2}\right)+{\displaystyle \frac{1}{2}}(\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2+{\omega }_2+2{\mathsf{\Phi }}_{c2m}^2)Tr(W_{{\phantom{,}}_{c2}}^T{W}_{c2})+{\displaystyle \frac{1}{2}}{\epsilon }_{c2m}^2.\end{array}$$

(53)

It can be sorted out:

$$\begin{array}{l}\dot{V}\le -k_1{\left(z_1^T{z}_1\right)}^l-k_2{\left(z_2^T{z}_2\right)}^l-k_3{\left(z_3^T{z}_3\right)}^l-({\displaystyle \frac{1}{2{\delta }_1}}-{\displaystyle \frac{1}{2}})y_1^T{y}_1-({\displaystyle \frac{1}{2{\delta }_2}}-{\displaystyle \frac{1}{2}})y_2^T{y}_2\\ -{\displaystyle \frac{1}{2}}{\gamma }_{D_1}{\tilde{D}}_1^T{\tilde{D}}_1-{\displaystyle \frac{1}{2}}{\gamma }_{D_2}{\tilde{D}}_2^T{\tilde{D}}_2-{\displaystyle \frac{1}{2}}\left({\tau }_1-4{\mathsf{\Phi }}_{a1M}^2\right)Tr\left({\tilde{W}}_{a1}{\phantom{,}}^T{\tilde{W}}_{a1}\right)-{\displaystyle \frac{1}{2}}\left({\omega }_1-{\mathsf{\Phi }}_{c1m}^2-{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right)Tr\left({\tilde{W}}_{c1}{\phantom{,}}^T{\tilde{W}}_{c1}\right)\\ -{\displaystyle \frac{1}{2}}({\tau }_2-{\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2-2\underset{\_}{\pi }{\mathsf{\Phi }}_{a2M}^2-2{\mathsf{\Phi }}_{a2M}^2)Tr({\tilde{W}}_{a2}{\phantom{,}}^T{\tilde{W}}_{a2})-{\displaystyle \frac{1}{2}}({\omega }_2-{\mathsf{\Phi }}_{c2m}^2-\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2)Tr({\tilde{W}}_{c2}^T{\tilde{W}}_{c2})\\ +\kappa {\epsilon }_{D_1}+{\displaystyle \frac{3}{2}}{\overline{\epsilon }}_{D_1}^2+\kappa {\epsilon }_{D_2}+{\displaystyle \frac{1}{2}}{\overline{\epsilon }}_{D_2}^2+{\displaystyle \frac{1}{2}}{\gamma }_{D_1}{D}_1^T{D}_1+{\displaystyle \frac{1}{2}}{\gamma }_{D_2}{D}_2^T{D}_2+{\displaystyle \frac{{\delta }_1{\left|{\dot{x}}_{1c}\right|}^2}{2}}+{\displaystyle \frac{{\delta }_2{\left|{\dot{x}}_{2c}\right|}^2}{2}}\\ +{\displaystyle \frac{1}{2}}\left({\tau }_1+{\mathsf{\Phi }}_{a1M}^2\right)Tr(W_{a1}{\phantom{,}}^T{W}_{a1})+{\displaystyle \frac{1}{2}}\left({\omega }_1+2{\mathsf{\Phi }}_{c1m}^2+{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right)Tr\left(W_{c1}^T{W}_{c1}\right)+{\displaystyle \frac{1}{2}}{\epsilon }_{c1m}^2\\ +{\displaystyle \frac{1}{2}}({\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2+{\tau }_2)Tr\left(W_{a2}{\phantom{,}}^T{W}_{a2}\right)+{\displaystyle \frac{1}{2}}(\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2+{\omega }_2+2{\mathsf{\Phi }}_{c2m}^2)Tr(W_{{\phantom{,}}_{c2}}^T{W}_{c2})+{\displaystyle \frac{1}{2}}{\epsilon }_{c2m}^2.\end{array}$$

(54)

By defining:

$$\overline{C}=\min \left\{\begin{array}{l}2k_1,2k_2,2k_3,({\displaystyle \frac{1}{{\delta }_1}}-1),({\displaystyle \frac{1}{{\delta }_2}}-1),{\tau }_{D_1}{\gamma }_{D_1},{\tau }_{D_2}{\gamma }_{D_2},\left({\tau }_1-4{\mathsf{\Phi }}_{a1M}^2\right),\left({\omega }_1-{\mathsf{\Phi }}_{c1m}^2-{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right),\\ ({\tau }_2-{\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2-2\underset{\_}{\pi }{\mathsf{\Phi }}_{a2M}^2-2{\mathsf{\Phi }}_{a2M}^2),({\omega }_2-{\mathsf{\Phi }}_{c2m}^2-\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2)\end{array}\right\},$$

(55)

$$\begin{array}{l}\epsilon =\kappa {\epsilon }_{D_1}+{\displaystyle \frac{3}{2}}{\overline{\epsilon }}_{D_1}^2+\kappa {\epsilon }_{D_2}+{\displaystyle \frac{1}{2}}{\overline{\epsilon }}_{D_2}^2+{\displaystyle \frac{1}{2}}{\gamma }_{D_1}{D}_1^T{D}_1+{\displaystyle \frac{1}{2}}{\gamma }_{D_2}{D}_2^T{D}_2+{\displaystyle \frac{{\delta }_1{\left|{\dot{x}}_{1c}\right|}^2}{2}}+{\displaystyle \frac{{\delta }_2{\left|{\dot{x}}_{2c}\right|}^2}{2}}\\ +{\displaystyle \frac{1}{2}}\left({\tau }_1+{\mathsf{\Phi }}_{a1M}^2\right)Tr(W_{a1}{\phantom{,}}^T{W}_{a1})+{\displaystyle \frac{1}{2}}\left({\omega }_1+2{\mathsf{\Phi }}_{c1m}^2+{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2\right)Tr\left(W_{c1}^T{W}_{c1}\right)+{\displaystyle \frac{1}{2}}{\epsilon }_{c1m}^2\\ +{\displaystyle \frac{1}{2}}({\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2+{\tau }_2)Tr\left(W_{a2}{\phantom{,}}^T{W}_{a2}\right)+{\displaystyle \frac{1}{2}}(\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2+{\omega }_2+2{\mathsf{\Phi }}_{c2m}^2)Tr(W_{c2}^T{W}_{c2})+{\displaystyle \frac{1}{2}}{\epsilon }_{c2m}^2.\end{array}$$

(56)

Then, Equation (54) can be rewritten as:

$$\begin{array}{l}\dot{V}\le -{\displaystyle \frac{1}{2}}\overline{C}{\left(z_1^T{z}_1\right)}^l-{\displaystyle \frac{1}{2}}\overline{C}{\left(z_2^T{z}_2\right)}^l-{\displaystyle \frac{1}{2}}\overline{C}{\left(z_3^T{z}_3\right)}^l-{\displaystyle \frac{1}{2}}\overline{C}{y}_1^T{y}_1-{\displaystyle \frac{1}{2}}\overline{C}{y}_2^T{y}_2\\ -{\displaystyle \frac{\overline{C}}{2{\tau }_{D1}}}{\tilde{D}}_1^T{\tilde{D}}_1-{\displaystyle \frac{\overline{C}}{2{\tau }_{D2}}}{\tilde{D}}_2^T{\tilde{D}}_2-{\displaystyle \frac{\overline{C}}{2{\eta }_1}}Tr\left({\tilde{W}}_{a1}{\phantom{,}}^T{\tilde{W}}_{a1}\right)-{\displaystyle \frac{\overline{C}}{2{\varpi }_1}}Tr\left({\tilde{W}}_{c1}{\phantom{,}}^T{\tilde{W}}_{c1}\right)\\ -{\displaystyle \frac{\overline{C}}{2{\eta }_1}}Tr({\tilde{W}}_{a2}{\phantom{,}}^T{\tilde{W}}_{a2})-{\displaystyle \frac{\overline{C}}{2{\varpi }_2}}Tr({\tilde{W}}_{c2}^T{\tilde{W}}_{c2})+\epsilon .\end{array}$$

(57)

According to Lemma, we obtain:

$$\begin{array}{l}{\left({\displaystyle \frac{1}{2}}{y}_i^T{y}_i\right)}^l\le \left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}+{\displaystyle \frac{1}{2}}{y}_i^T{y}_i,\\ {\left({\displaystyle \frac{1}{2{\tau }_{Di}}}{\tilde{D}}_i^T{\tilde{D}}_i\right)}^l\le \left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}+{\displaystyle \frac{1}{2{\tau }_{Di}}}{\tilde{D}}_i^T{\tilde{D}}_i,\\ {\left({\displaystyle \frac{1}{2{\eta }_i}}Tr\left({\tilde{W}}_{ai}{\phantom{,}}^T{\tilde{W}}_{ai}\right)\right)}^l\le \left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}+{\displaystyle \frac{1}{2{\eta }_i}}Tr\left({\tilde{W}}_{ai}{\phantom{,}}^T{\tilde{W}}_{ai}\right),\\ {\left({\displaystyle \frac{1}{2{\varpi }_i}}Tr\left({\tilde{W}}_{ci}{\phantom{,}}^T{\tilde{W}}_{ci}\right)\right)}^l\le \left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}+{\displaystyle \frac{1}{2{\varpi }_i}}Tr\left({\tilde{W}}_{ci}{\phantom{,}}^T{\tilde{W}}_{ci}\right).\end{array}$$

(58)

It follows from Lemma 4 that:

$$-\overline{C}\left({\displaystyle \frac{1}{2}}{\left(z_i^T{z}_i\right)}^l\right)\le -2^{l-1}\overline{C}{\left({\displaystyle \frac{1}{2}}{z}_i^T{z}_i\right)}^l.$$

(59)

Finally, we can obtain:

$$\begin{array}{l}\dot{V}\le -2^{l-1}\overline{C}{\left({\displaystyle \frac{1}{2}}{z}_1^T{z}_1\right)}^l-2^{l-1}\overline{C}{\left({\displaystyle \frac{1}{2}}{z}_2^T{z}_2\right)}^l-2^{l-1}\overline{C}{\left({\displaystyle \frac{1}{2}}{z}_3^T{z}_3\right)}^l-\overline{C}{\left({\displaystyle \frac{1}{2}}{y}_1^T{y}_1\right)}^l+9\overline{C}\left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}\\ -\overline{C}{\left({\displaystyle \frac{1}{2}}{y}_2^T{y}_2\right)}^l-\overline{C}{\left({\displaystyle \frac{1}{2}}{y}_3^T{y}_3\right)}^l-\overline{C}{\left({\displaystyle \frac{1}{2{\tau }_{D1}}}{\tilde{D}}_1^T{\tilde{D}}_1\right)}^l-\overline{C}{\left({\displaystyle \frac{1}{2{\tau }_{D2}}}{\tilde{D}}_2^T{\tilde{D}}_2\right)}^l\\ -\overline{C}{\left({\displaystyle \frac{1}{2{\eta }_1}}Tr\left({\tilde{W}}_{a1}{\phantom{,}}^T{\tilde{W}}_{a1}\right)\right)}^l-\overline{C}\left({\displaystyle \frac{1}{2{\eta }_2}}Tr\left({\tilde{W}}_{a2}{\phantom{,}}^T{\tilde{W}}_{a2}\right)\right)\\ -{\left({\displaystyle \frac{1}{2{\varpi }_2}}Tr\left({\tilde{W}}_{c2}{\phantom{,}}^T{\tilde{W}}_{c2}\right)\right)}^l+\epsilon .\end{array}$$

(60)

Let:

$$\begin{array}{l}C=\min \left\{2^{l-1}\overline{C},\overline{C}\right\}\\ D=\epsilon +9\overline{C}\left(1-l\right)l^{{\displaystyle \frac{l}{1-l}}}.\end{array}$$

(61)

Then, Equation (60) can be rewritten as:

$$\dot{V}\le -C{V}^l+D.$$

(62)

Based on Lemma 2, selecting the parameters $k_i>0,i=1,2,3,0<{\delta }_i<1,{\tau }_{D_i}>0,{\gamma }_{D_i}>0,{\tau }_1$ $>4{\mathsf{\Phi }}_{a1M}^2,{\omega }_1>{\mathsf{\Phi }}_{c1m}^2+{\mathsf{\Phi }}_{c1M}^2{\Gamma }_{1M}^2,{\tau }_2>{\underset{\_}{\pi }}^2{\mathsf{\Phi }}_{a2M}^2+2\underset{\_}{\pi }{\mathsf{\Phi }}_{a2M}^2+2{\mathsf{\Phi }}_{a2M}^2,{\omega }_2>{\mathsf{\Phi }}_{c2m}^2+\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2+\underset{\_}{\pi }{\mathsf{\Phi }}_{c2M}^2{\Gamma }_{2M}^2,$ $i=1,2$ to derive the explicit conditions for $\overline{C}>0$, we conclude that all the signals of the closed-loop system are semi-global practical finite-time stable.

② Given $T_r={\displaystyle \frac{1}{(1-l)\aleph C}}\left[V^{1-l}(0)-{\left({\displaystyle \frac{D}{(1-\aleph )C}}\right)}^{\Xi }\right]$, where $\Xi =(1-l)/l,\aleph \in (0,1)$. With the analysis in [52], it yields:

$$V\le {\left({\displaystyle \frac{D}{\left(1-\aleph \right)C}}\right)}^{1/l},\forall t\ge T_r.$$

(63)

Then, combining Equation (48), it can be obtained:

$$\left|z_1\left(t\right)\right|=\left|x_1\left(t\right)-y_d\right|<2{\left({\displaystyle \frac{D}{(1-\aleph )C}}\right)}^{{\displaystyle \frac{1}{2l}}},\forall t\ge T_r.$$

(64)

Thereby, it can be concluded that the tracking error converges to and remains in a small neighborhood of the desired signals for $\forall t\ge T_r$. □

Remark 7. 

The system tracking error $z_1\left(t\right)$enters and remains within a steady-state neighborhood determined by the parameters $D,C,\aleph ,l$, after a finite time $T_r$. The convergence process can be accelerated by appropriately increasing the value of $l$.

4. Simulation Study

In this section, numerical simulations are performed to confirm the effectiveness and advantages of the developed RL-based finite-time adaptive fault tolerant control method for a morphing unmanned aircraft.

The initial conditions are chosen as: ${\left[\begin{array}{ccc}\alpha & \beta & {\gamma }_v\end{array}\right]}^T={\left[\begin{array}{ccc}{20}^{\circ }& 5^{\circ }& -5^{\circ }\end{array}\right]}^T$, ${[\begin{array}{ccc}{\omega }_x& {\omega }_y& {\omega }_z\end{array}]}^T=$ ${\left[\begin{array}{ccc}0.1rad/s& 0.1rad/s& 0.1rad/s\end{array}\right]}^T$; The adaptive parameters and RL parameters are set as: $k_1=3$, $k_2=5$, $k_3=8$, ${\tau }_1={\tau }_2=8$, ${\eta }_1=10$, ${\eta }_2=1$, ${\varpi }_1=5$, ${\varpi }_2=0.5$, ${\omega }_1=0.5$, ${\omega }_2=0.1$; The disturbance compensation adaptative parameters are defined as: ${\epsilon }_{D_1}={\epsilon }_{D_2}=0.01$, ${\tau }_{D_1}={\tau }_{D_2}=10$, ${\gamma }_{D_1}={\gamma }_{D_2}$$=10$; Structural parameters of the morphing unmanned aircraft: $I_{bx}=20$, $I_{by}=I_b{\phantom{,}}_z=150$; The aircraft’s flight velocity is $V=600$ m/s, and its flight altitude is $H=15$ km. To ensure the proper implementation of the simulations, they are conducted under three different cases. The different parameters of the uncertainties $\Delta f_1,\Delta f_2$ and disturbances $d_1\left(t\right)$ and $d_2\left(t\right)$ expressed in model (27) are defined under three cases. The detailed specifications of these scenarios are presented in

Table 1. The different parameter values of the morphing unmanned aircraft under three cases.

Cases	$\mathbf{\Delta }{\mathit{f}}_{\mathbf{1}}$	${\mathit{d}}_{\mathbf{1}}\left(\mathit{t}\right)$	$\mathbf{\Delta }{\mathit{f}}_{\mathbf{2}}$	${\mathit{d}}_{\mathbf{2}}\left(\mathit{t}\right)$
Case 1	$\dot{\theta }={0.007}^{\circ }/s$	$\dot{\sigma }={0.01}^{\circ }/s,\theta =0^{\circ }$	$0.01I_b$	$0.01M_f+0.01I_b$
Case 2	$\dot{\theta }={0.01}^{\circ }/s$	$\dot{\sigma }={0.006}^{\circ }/s,\theta ={0.03}^{\circ }$	$0.008I_b$	$0.006M_f+0.01I_b$
Case 3	$\begin{array}{l}\dot{\theta }={0.005}^{°}\times \\ (1+0.1\sin (t))/s\end{array}$	$\begin{array}{l}\dot{\sigma }={0.008}^{°}\times (1+0.1\cos (t))/s,\\ \theta ={0.02}^{°}\end{array}$	$\begin{array}{l}0.004\times (1+\\ 0.1\sin (t))I_b\end{array}$	$\begin{array}{l}0.009M_f+0.01\times (1\\ +0.1\cos (t))I_f\end{array}$

.

The simulation results are given in Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6. Figure 2 demonstrates the trajectory tracking performance of the morphing unmanned aircraft’s attitude angles under coupled uncertainties. Obviously, the proposed RL-based adaptive controller achieves convergence within a 0.001-error bound around the desired trajectories. Despite an initial transient overshoot, the intelligent controller can quickly compensate for the uncertainties, exhibiting strong adaptability and robustness. The disturbance estimation and neural network weight estimation signals are shown in Figure 3, Figure 4 and Figure 5, and it can be seen that all signals are bounded. Figure 6 shows the control torque variation curve over time under Case 1.

To verify the advantages of the proposed reinforcement learning adaptive control (RLAC) method and its adaptability, such as speed and stability, the neural network adaptive control (NNAC) [59] method and the adaptive fault-tolerant control (AFTC) [60] method are selected for comparison. These existing adaptive control methods serve as benchmarks to highlight the unique features of the RLAC method. All controllers track identical reference trajectories under the same experimental conditions. The experimental results comprehensively compare the proposed RLAC method with two existing control approaches, AFTC and NNAC methods, are shown in Figure 7 and Figure 8.

Table 2. Tracking performance of three methods.

Method	Steady-State Error (Rad)	Convergence Time (s)
RLAC	0.0038, 0.0024, 0.00087	0.79, 0.66, 0.82
NNAC	0.007, 0.0023, 0.001	1.43, 4.19, 1.19
AFTC	>0.026, >0.025, >0.0026	5.81, >10, 1.9

presents the steady-state error and convergence time of the three methods, where convergence time is defined as the time required to complete 90% of the convergence process. In Figure 8, the tracking error is defined as the sum of the two norms of the errors $z_1$ and $z_2$, that is, $e={‖z_1‖}_2+{‖z_2‖}_2$, which clearly highlights the superiority of the proposed RLAC method.

The results demonstrate that under the same experimental conditions, the AFTC method has limitations. It has a relatively large steady-state error of about 0.25 and a convergence time exceeding 10 s, which means it struggles to precisely track the reference trajectory and reaches a stable state slowly, leading to inefficient system operation.

The NNAC method shows improvements. It can achieve a steady-state error of less than 0.0015, more accurate than the AFTC method. However, it suffers from significant overshoot, which may cause system instability. In contrast, the RLAC method demonstrates remarkable superiority. It has a steady-state error below 0.001, the fastest convergence speed, and can rapidly adapt to changing environments. In terms of tracking performance, it outperforms both AFTC and NNAC methods in speed and stability. Consequently, the RLAC method offers significant advantages over AFTC and NNAC methods, making it a more promising choice for practical control applications.

5. Conclusions

The RL-based adaptive fault-tolerant control method proposed in this paper effectively solves the control problem for a class of morphing unmanned aircraft under mismatched disturbances and coupled uncertainties. The aircraft’s uncertainties are modeled as a non-affine second-order nonlinear system, and the non-affine structure is handled by introducing an auxiliary integral system. The unknown functions are evaluated by the introduced RL algorithms, and the control actions are adjusted by the developed RL-based adaptive laws. The filtering errors and disturbances are compensated by adopting a disturbance boundary estimator. The RL-based adaptive fault-tolerant control method has been successfully designed by integrating RL, disturbance estimation, and finite-time theory. It is proven by the Lyapunov function that the system has finite-time stability and all signals are bounded. Numerical simulations verify the effectiveness and superiority of this method. In the future, this method is expected to be extended to aircraft models, considering various factors in the actual flight environment.