Fault-Tolerant Controller Design for Reusable Launch Vehicle

Abstract

A fault-tolerant controller design for reusable launch vehicles (RLVs) is discussed in this paper. The control precision of RLVs is very important, since it must be ensured that an RLV’s speed reaches zero while flying to the target point. More seriously, the rocket’s thrust system may suffer from faults, so the fault-tolerant control of RLVs is very important. The landing dynamic model of RLVs is very complex, and the thrust is coupled with time-varying states, which make the controller design of RLVs very difficult. Based on the specific control requirements of rocket landing, the control design problem is first transformed into a normal model in this paper. Then, considering potential thrust faults, an optimal fault-tolerant controller is designed using reinforcement learning. Considering sensor faults and actuator faults, this paper presents the corresponding fault-tolerant controller design method. Considering that the analytical problem of the proposed fault-tolerant controller is difficult to solve, this paper presents an approximation method for the analytical solution based on a neural network. The simulation results demonstrate that the proposed controller ensures the safe and stable landing of the rocket in both nominal and fault scenarios.

1. Introduction

With the global development of outer space, RLVs have become a prominent focus of research [1]. RLVs have significant advantages, including rapid deployment, large load carrying capacity, and reusability, and all of these advantages substantially reduce the cost of space transportation and exploration. However, to enable reuse ability, RLVs must perform vertical landings with high precision, placing stringent demands on their guidance and control systems [2,3].

In the vertical recovery of RLVs, the control force is provided by the rocket’s thrust. During the landing process, the control objective of RLVs is to ensure that they reach the target landing point smoothly while minimizing fuel consumption. In this phase, the only adjustable control input is the magnitude and direction of the thrust. However, the system must simultaneously control the rocket’s position to the target and reduce its velocity to zero, which means that there are six outputs to be controlled and three control inputs. This mismatch between the number of control inputs (thrust magnitude and direction) and controlled outputs (position and velocity states) makes the rocket landing problem a typical underactuated control problem. However, to ensure a smooth landing, both the position and velocity of the rocket must be simultaneously regulated. The recovery of launch vehicles represents a typical underactuated control problem.

The design of the optimal controller for an underactuated system is challenging [4,5]. To achieve this, a trajectory tracking control law based on the dynamic inversion method combined with an online trajectory update strategy is proposed in [6]; by using this method, effective reentry guidance tracking is achieved. Sliding mode dynamic surface control is employed to design a precise vertical recovery control strategy, ensuring high control accuracy during vertical recovery [7]. A two-point boundary value approach, formulating the rocket’s initial and desired states as boundary conditions to optimize the trajectory, is introduced in [8,9]; subsequently, an indirect method was used to design the rocket’s optimal flight path. However, the indirect method faces challenges in real-time implementation and convergence guarantees. A direct method for designing controllers for rocket recovery by solving convex optimization problems to achieve landing control is proposed in [10,11]. Nevertheless, both direct and indirect methods require accurate knowledge of the rocket’s nonlinear model and disturbances. In practice, due to the complex atmospheric environment and machining accuracy limitations, the RLV model inevitably contains uncertainties. Moreover, the rocket operates over a large envelope, and states of RLVs are coupled with each other, which leads to the complexity of the controller design of RLVs [12,13].

For the rocket landing control problem under uncertainties and model deviations, the indirect method is improved by neural networks [14,15]. A large set of rocket landing sample data is obtained by extensive training, and then neural networks are used to fit the data; thus a new empirical control law that achieves rocket landing control is derived. By applying reinforcement learning, a staged reward function is built [16]. The staged reward function is solved by Q-learning, and then a rocket landing controller is optimized under energy constraints. Ref. [17] analyzes the uncertain factors of RLVs and presents a design method for parameterized controllers.

The rocket landing control problem has attracted widespread attention. Researchers have designed landing controllers using methods such as direct approaches, indirect approaches, neural networks, and reinforcement learning. However, in practical applications, rocket thrust systems may experience faults, which significantly complicate landing control under fault conditions. The attitude FTC problem for faulty rocket systems is proposed, and a robust control method based on fixed-time observers is proposed [18]. An adaptive FTC technique for rocket attitude control systems is presented in [19]. The FTC problem of the attitude control system during rocket reentry is discussed in [20]. Nevertheless, fault-tolerant control in the rocket landing and recovery process remains relatively underexplored and lacks a comprehensive solution. In summary, although the rocket landing problem has received considerable attention, existing studies have yet to address fault-tolerant control in rocket landing guidance.

Based on the above discussion, this paper discusses the fault-tolerant landing controller designing of RLVs. Uncertainties such as inaccurate nonlinear rocket models and unknown external disturbances are taken into account. While considering these uncertainties and disturbances, neural networks are employed to construct the performance function and control law within the rocket controller, and a model-free fault-tolerant control design method is proposed. The contributions of this paper can be summarized as follows:

• A data-based FTC method for RLVs is proposed in this paper. The proposed method can realize the accurate control of RLVs without complete information on the RLV’s nonlinear model.
• Both the uncertainties and external interference of RLVs are considered in this paper. The proposed method can deal with not only the faults of RLVs but also their uncertainties and external interference.
• Both sensor faults and partial failure faults are considered and addressed simultaneously within the proposed FTC method.

2. Landing Model of Reusable Launch Vehicle

2.1. Nonlinear Landing Model of Reusable Launch Vehicle

The rocket landing problem can be described by the following nonlinear dynamic equations:

$$\left\{\begin{array}{l}\dot{r}=V\\ \dot{V}=g+{\displaystyle \frac{T+D}{m}}\\ \dot{m}={\displaystyle \frac{‖T‖}{V_{ex}}}\end{array}\right.$$

(1)

where $r$ represents the position vector of the rocket; $V$ represents the velocity vector; $m$ denotes the mass of the rocket; $\dot{m}$ represents the consumption rate of mass, that is, the consumption rate of fuel; and $g$ is the gravitational acceleration at the rocket’s location. $T$ denotes the thrust vector of the rocket, and $V_{ex}$ represents the rocket’s exhaust velocity, which is a constant. $D$ stands for the aerodynamic drag experienced by the rocket, which depends on the vehicle’s velocity and is calculated simultaneously using the following equation:

$$D=-{\displaystyle \frac{1}{2}}\rho V^2{S}_{ref}{C}_D$$

(2)

where $\rho$ is the atmospheric density, which depends on the rocket’s position; $S_{ref}$ is the aerodynamic reference area of the rocket; $C_D$ is the drag coefficient, which is related to the rocket’s Mach number.

To ensure control accuracy, $g$ is closely related to the rocket’s altitude. In practical applications, $g$ is calculated using the following formula:

$$g=-{\displaystyle \frac{\mu }{{‖R_E+r‖}^3}}\left(R_E+r\right)$$

(3)

where $\mu$ is the Earth’s gravitational constant, and $R_E$ is the vector from the Earth’s center to the surface.

2.2. Control Requirements

The control objective can be formulated as the following optimal control problem:

$$\begin{array}{l}\max m\\ subject to\\ \left\{\begin{array}{l}\dot{r}=V\\ \dot{V}=g+{\displaystyle \frac{T+D}{m}}\\ \dot{m}=-{\displaystyle \frac{‖T‖}{V_{ex}}}\\ r(t_0)=r_0,V(t_0)=V_0,m(t_0)=m_0\\ r(t_f)=0,V(t_f)=0\end{array}\right.\end{array}$$

(4)

where $t_0$ denotes the initial time, and $t_f$ represents the final landing time. $r_0, V_0, m_0$ correspond to the position, velocity, and mass at the initial time, respectively. $V_{ex}$ denotes the exhaust velocity. The rocket’s intrinsic constraints are taken into account; specifically, the thrust is bounded as $T_{\min }\le ‖T‖\le T_{\max }$, where $T_{\min }, T_{\max }$ are positive real numbers.

3. Model Transformation

By examining Equations (1)–(3) and considering the control requirements, the variables to be controlled include the position and velocity in three directions, as well as the rocket’s mass. However, the available control input is limited to the rocket’s thrust, so this problem is a typical underactuated control problem. To facilitate controller design, the model is transformed according to the control requirements.

3.1. Selection of Control Variables

To facilitate the controller design, the RLV’s position and velocity are decomposed in the North–East–Up coordinate frame. In this coordinate system, both the position and velocity of the rocket are represented as three-dimensional state variables. Meanwhile, the thrust of the rocket is also decomposed accordingly. Assuming that the rocket does not rotate in space, the thrust can be expressed as a three-dimensional control input vector.

$$\begin{array}{l}u={\left[\begin{array}{ccc}{u}_1,& u_2,& u_3\end{array}\right]}^T\\ \hspace{0.17em}={\left[\begin{array}{ccc}‖T‖\cos \alpha \sin \beta ,& ‖T‖\sin \alpha \sin \beta ,& ‖T‖\cos \beta \end{array}\right]}^T\end{array}$$

where $\alpha$ denotes the angle between the thrust vector and the east direction in the “north-east” plane, while $\beta$ represents the angle between the thrust vector and the “upward” (zenith) axis. Under this configuration, the constraints on the three input variables of the rocket are given as follows:

$$\left\{\begin{array}{l}{T}_{\min }\le ‖T‖\le T_{\max }\\ 0\le \alpha \le {\displaystyle \frac{\pi }{2}}\\ 0\le \beta \le {\displaystyle \frac{\pi }{2}}\end{array}\right.$$

3.2. Fully Actuated System

To facilitate controller design, the rocket landing model is transformed based on control requirements. Let $r=\sqrt{x^2+y^2+z^2}$ and $V=\sqrt{V_x^2+V_y^2+V_z^2}$, where $x,y,z$ denotes the position vector of the rocket expressed in the local East–North–Up coordinate system, and $V_x,V_y,V_z$ represents the velocity vector in the same coordinate frame. The nonlinear dynamics of the rocket can then be rewritten as follows:

$$\left\{\begin{array}{l}\dot{r}={\displaystyle \frac{x\dot{x}+y\dot{y}+z\dot{z}}{\sqrt{x^2+y^2+z^2}}}={\displaystyle \frac{x{V}_x+y{V}_y+z{V}_z}{\sqrt{x^2+y^2+z^2}}}\\ \dot{V}={\displaystyle \frac{V_x{\dot{V}}_x+V_y{\dot{V}}_y+V_z{\dot{V}}_z}{\sqrt{V_x^2+V_y^2+V_z^2}}}\\ \dot{m}=-{\displaystyle \frac{‖T‖}{V_{ex}}}\end{array}\right.$$

(5)

where

$$\begin{array}{l}{\dot{V}}_x={\displaystyle \frac{\frac{V_{x}D}{\sqrt{V_x^2+V_y^2+V_z^2}}}{m}}+{\displaystyle \frac{u_1}{m}}\\ {\dot{V}}_y={\displaystyle \frac{\frac{V_{y}D}{\sqrt{V_x^2+V_y^2+V_z^2}}}{m}}+{\displaystyle \frac{u_2}{m}}\\ {\dot{V}}_z=g+{\displaystyle \frac{\frac{V_{z}D}{\sqrt{V_x^2+V_y^2+V_z^2}}}{m}}+{\displaystyle \frac{u_3}{m}}\end{array}$$

At this stage, the rocket landing problem is transformed into a control problem with three inputs and three outputs, where the control objectives achieve $r=0$ and $V=0$ while maximizing the remaining fuel.

To address the above control problem, new output variables are selected as follows:

$$\left\{\begin{array}{l}{\chi }_1=r+\lambda \dot{r}\\ {\chi }_2=V\\ {\chi }_3=m\end{array}\right.$$

(6)

where $\lambda$ represents the coupling between position and velocity. The value of r is a positive number, and the specific value is determined by the importance of position control accuracy and speed control accuracy. Then the new equation can be expressed as follows:

$$\dot{\chi }=f(\chi )+g(\chi )u$$

(7)

The specific form of $f(\chi ),g(\chi )$ is determined by Equations (1), (2) and (5).

4. Fault Description of RLVs

The RLV’s thrust is controlled by an onboard computer. The thrust system itself may exhibit deviations. In particular, the erosion of the thrust system’s nozzle may occur, causing the rocket’s thrust vector to deviate from the desired direction; then the control accuracy of the rocket is affected. In this paper, two common fault modes are considered for the design of a fault-tolerant controller for the rocket.

4.1. Partial Failure Fault

The rocket propulsion system may produce a thrust lower than the expected value. This type of fault can be described as follows:

$${‖T‖}_{\mathrm{r}}=\eta (t){‖T‖}_{\mathrm{d}}$$

(8)

where ${‖T‖}_{\mathrm{d}}$ denotes the desired thrust of the rocket, i.e., the thrust required under fault-free conditions; ${‖T‖}_x$ represents the actual thrust that the rocket can provide; and $\eta (t)$ indicates the thrust efficiency of the rocket, with $0<\eta (t)\le 1$. In this scenario, the rocket experiences a partial failure fault. Considering the actual occurrence of faults in the rocket, the efficiency $\eta (t)$ is unknown. Under this fault mode, the rocket control problem can be transformed into the following:

$$\begin{array}{l}\max m\\ Subject to\\ \left\{\begin{array}{c}\dot{\chi }=f(\chi )+g(\chi )u\\ {‖T‖}_{\mathrm{r}}=\eta (t){‖T‖}_{\mathrm{d}}\\ 0<\eta (t)\le 1\\ 0\le \alpha \le {\displaystyle \frac{\pi }{2}}\\ 0\le \beta \le {\displaystyle \frac{\pi }{2}}\end{array}\right.\end{array}$$

(9)

4.2. Sensor Fault

The rocket’s thrust command is generated by converting the electrical signals received from the onboard computer. However, during signal transmission, signal attenuation may occur, causing the electrical signal received by the thrust generation system to be lower than the expected value. As a result, the rocket’s thrust will deviate. The thrust can be expressed as follows:

$${‖T‖}_{\mathrm{r}}={‖T‖}_{\mathrm{d}}-{‖T‖}_{\mathrm{f}}$$

(10)

where ${‖T‖}_{\mathrm{d}}$ denotes the desired thrust of the rocket, and ${‖T‖}_{\mathrm{f}}$ represents the thrust error caused by the sensor fault. ${‖T‖}_{\mathrm{f}}$ is a time-varying variable. This type of fault is defined as a sensor fault, with a critical threshold for thrust loss caused by the fault given by ${‖T‖}_{\mathrm{f}}\le \overline{T}$. ${‖T‖}_{\mathrm{r}}$ represents the actual thrust of the rocket. Under this fault mode, the rocket’s control problem can be formulated as follows:

$$\begin{array}{l}\max m\\ Subject to\\ \left\{\begin{array}{l}\dot{\chi }=f(\chi )+g(\chi )u\\ {‖T‖}_{\mathrm{r}}={‖T‖}_{\mathrm{d}}-{‖T‖}_{\mathrm{f}}\\ T_{\min }\le {‖T‖}_{\mathrm{d}}\le T_{\max }\\ {‖T‖}_{\mathrm{f}}\le \overline{T}\\ 0\le \alpha \le {\displaystyle \frac{\pi }{2}}\\ 0\le \beta \le {\displaystyle \frac{\pi }{2}}\end{array}\right.\end{array}$$

(11)

5. Design of Optimal Fault-Tolerant Controller

By analysis, the aforementioned FTC problem is a nonlinear time-varying control problem. Moreover, the inputs are not given in an explicit form, and there is coupling among the input variables; all of this makes the direct FTC design of RLVs challenging. To achieve optimal control for (9) and (11), a model-free optimal fault-tolerant controller is considered, and a data-driven online solution method based on reinforcement learning is employed to implement the optimal controller.

5.1. Optimal Fault-Tolerant Controller Design for Partial Actuator Failures

5.1.1. Optimal Fault-Tolerant Controller Design

Considering the partial failure model of the rocket described in Equation (8), the controllable inputs available from the rocket can be represented as follows:

$$\begin{array}{l}{u}_r={\left[\begin{array}{ccc}{u}_{1r},& u_{2r},& u_{3r}\end{array}\right]}^T\\ ={\left[\begin{array}{ccc}{‖T‖}_r\cos \alpha \sin \beta ,& {‖T‖}_r\sin \alpha \sin \beta ,& {‖T‖}_r\cos \beta \end{array}\right]}^T\end{array}$$

Since $\eta (t)$ and ${‖T‖}_r$ are unknown, the exact form of $u_r$ cannot be determined. Considering that the optimization objective of rocket landing is to minimize fuel consumption while ensuring a stable landing, the following performance index is selected as the optimization criterion:

$$\begin{array}{l}J(\chi (t),u(t))={\displaystyle {\int }_0^{\infty }(}{\chi }^T(\tau )Q\chi (\tau )+{u_r}^T(\tau )R{u}_r(\tau ))d\tau \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} ={\displaystyle {\int }_0^{\infty }\mathsf{\Theta }}\left(\chi (\tau ),u_r(\tau )\right)d\tau \end{array}$$

(12)

where $Q$ and $R$ are positive definite weighting matrices

$$\mathsf{\Theta }\left(\chi (\tau ),u(\tau )\right)={\chi }^T(\tau )Q\chi (\tau )+{u_r}^T(\tau )R{u}_r(\tau ).$$

Remark 1. 

Regarding the optimization criterion (9), the control objective is to ensure that the rocket reaches the designated landing point while minimizing fuel consumption. In performance index (12), by appropriately selecting the weighting matrices $Q$ and $R$, the optimization criterion (9) can be effectively optimized. Therefore, the performance index (12) is reasonable.

For the control system (9), combined with the performance index function established in (12), the following Hamilton–Jacobi–Bellman (HJB) function is defined:

$$\begin{array}{l}H(\chi (t),u_r(t),J(t))\\ =\mathsf{\Theta }(\chi (t),u_r(t))+\nabla J^T(\chi (t),u_r(t))\dot{\chi }(t)\\ ={\chi }^T(t)Q\chi (t)+{u_r}^T(t)R{u}_r(t)+\\ \hspace{1em}\nabla J^T(\chi (t),u_r(t))\left(f(\chi )+g(\chi )u_r\right)\end{array}$$

(13)

where $\nabla J^T(\chi (t),u_r(t))$ is the gradient of the performance index $J(\chi (t),u_r(t))$. According to optimal control theory, the required optimal controller can guarantee the minimization of the following performance index:

$$\begin{array}{l}{J}^{\ast }(\chi (\tau ),u_r(t))\\ =\underset{u\in \Omega }{\min }{\displaystyle {\int }_0^{\infty }\mathsf{\Theta }}(\chi (\tau ),u_r(\tau ))d\tau \\ ={\displaystyle {\int }_0^{\infty }\mathsf{\Theta }}(\chi (\tau ),u_r^{\ast }(\tau ))d\tau \end{array}$$

Under the action of this controller,

$$\begin{array}{l}H(\chi (\tau ),{u_r}^{\ast }(t),J^{\ast }(t))\\ ={\chi }^T(t)Q\chi (t)+u_r^{\ast T}(t)R{u_r}^{\ast }(t)+\\ \hspace{1em}\nabla J^{\ast T}(\chi (\tau ),u_r(t))\left(f(\chi )+g(\chi ){u_r}^{\ast }(t)\right)\\ =0\end{array}$$

To solve the above HJB equation, the partial derivatives should equal zero:

$${\displaystyle \frac{\partial H(\chi (\tau ),{u_r}^{\ast }(t),J(t))}{\partial {u_r}^{\ast }(t)}}=0$$

The optimal control input which can ensure that the performance index (12) reaches the minimum under given constraints ${u_r}^{\ast }(t)$ can be obtained

$${u_r}^{\ast }(t)=-{\displaystyle \frac{1}{2}}{R}^{-1}{g}^T(\chi )\nabla J(\chi (\tau ),u_r(t))$$

(14)

Substituting the optimal control input into Equation (13) yields

$$\begin{array}{l}H(\chi (\tau ),{u_r}^{\ast }(t),J^{\ast }(t))\\ ={\chi }^T(t)Q\chi (t)+\nabla J^{\ast T}(\chi (t),u_r(t))f(\chi )\\ \hspace{1em} -{\displaystyle \frac{1}{4}}\nabla J^{\ast T}(\chi (t),u_r(t))g(\chi )R^{-1}{g}^T(\chi )\nabla J^{\ast }(e(t),u_r(t))\\ =0\end{array}$$

By substituting the optimal control input into Equation (13), the optimal control law (14) is obtained, which serves as the optimal controller for the system under partial actuator failures. However, this optimal controller cannot be directly implemented since the gradient of the performance index function (12), denoted as $\nabla J(\chi (\tau ),u_r(t))$, cannot be solved analytically. To better solve the optimal control input, a policy iteration method from reinforcement learning is employed to compute the system’s optimal control.

The online solution of the optimal controller can be achieved through Algorithm 1.

Algorithm 1: Policy iterative algorithm of FTC

Initialization: Select $u\left(0\right),J\left(0\right),$

Step1: pass

$\begin{array}{l}\nabla J^{\left(k+1\right)T}(\chi (t),u(t))\left(f(x)+g(x)u\right)\\ +{\chi }^T(t)Q\chi (t)+u(t)Ru(t))\\ =0\end{array}$

Calculate $J^{\left(k+1\right)}(\chi (t),u(t))$

Step2: Updated Control Input

$u^{k+1}(t)=-{\displaystyle \frac{1}{2}}{R}^{-1}{g}^T(\chi )\nabla J^{\left(k\right)}(\chi (t),u(t))$

If $‖J^{\left(k+1\right)}(\chi (t),u(t))-J^{\left(k\right)}(\chi (t),u(t))‖\le \epsilon$, then stop the computation; otherwise, proceed to Step 1. Here, $\epsilon$ is a sufficiently small positive constant.

5.1.2. Online Solution Method for Fault-Tolerant Controller

For the nonlinear model (9) proposed in this paper, directly applying Algorithm 1 to iteratively compute the controller solution is highly challenging, since the model parameters are time-varying and the fault variable $\eta (t)$ is unknown. Under such circumstances, this paper proposes an online iterative solution method. Considering the performance index (12) and the controller (14), the following neural network is selected for online approximation:

$$\begin{array}{l}J(t)=W_J^T\phi (t)+e_J(t)\\ u(t)=W_u^T\psi (t)+e_v(t)\end{array}$$

(15)

where $W_J^T$ denotes the weight matrix of the performance evaluation neural network, and $\phi (t)$ represents its basis function; $W_u^T$ is the weight matrix of the controller neural network, and $\psi (t)$ denotes its basis function. The terms $e_J(t)$ and $e_v(t)$ represent the approximation errors.

Assuming that both the weights and basis functions in (15) are bounded, the rocket dynamics described in (12) can be rewritten under the conditions of the controller (14). The following can be obtained:

$$\dot{\chi }(t)=f(\chi )+g(\chi )W_u^T\psi (t)$$

(16)

The update law for the optimal control is given by the following:

$$\begin{array}{l}{J}^{k+1}(t+\Delta t)=-{\displaystyle {\int }_t^{t+\Delta t}\left[{\psi }^T(t)W_{u}R{W}_u^T\psi (t)\right.}\left.+{\chi }^T(\tau )Q\chi (\tau )\right]d\tau \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{0.17em}-{\displaystyle {\int }_t^{t+\Delta t}2}{\left(u^{k+1}(t)\right)}^T{R}_{\chi }d\tau +J^{k+1}(t)\\ u^{k+1}(t)=-{\displaystyle \frac{1}{2}}{R}^{-1}{g}^T(x)\nabla J^{k+1}(t)\end{array}$$

The HJB equation is as follows:

$$\begin{array}{l}H(\chi ,u,J)={\chi }^T(t)Q\chi (t)+\psi {(t)}^T{W}_u^{\ast }R{W}_u^{\ast T}\psi (t)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} +W_J^{\ast T}\nabla \phi (t)\left(f(\chi )+g(\chi )W_v^{\ast T}\psi (t)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=0\end{array}$$

where $W_u^{\ast }$ and $W_v^{\ast }$ represent the optimal weights of the chosen performance approximation neural network and the controller neural network, respectively. Since the optimal weights are unknown, let ${\widehat{W}}_v$ and ${\widehat{W}}_J$ denote the estimates of these optimal weights. Therefore,

$$\begin{array}{l}\widehat{J}(t)={\widehat{W}}_J^T\phi (t)\\ \widehat{u}(t)={\widehat{W}}_v^T\psi (t)\end{array}$$

(17)

Then the HJB equation is

$$\begin{array}{l}J(\chi ,u,J)={\chi }^T(t)Q\chi (t)+\psi {(t)}_u^T{\widehat{W}}_{v}R{\widehat{W}}_v^T\psi (t)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}+{\widehat{W}}_J^T\nabla \phi (t)\left(f(\chi )+g(\chi ){\widehat{W}}_v^T\psi (t)\right)+{\epsilon }_{HJI}\\ \hspace{1em}\hspace{1em}\hspace{1em}=0\end{array}$$

where ${\epsilon }_{HJI}$ denotes the estimated total error.

$$\begin{array}{l}{\epsilon }_{HJI}=\psi {(t)}^T{W}_{u}R{\chi }_u(t)+{\chi }^T(t)R_{\chi }(t)+\\ \hspace{1em}\hspace{1em}{\chi }_u^T(t)R{W}_u^T\psi (t)+\nabla J_u(t)\left(f(\chi )+g(\chi )W_u^T\psi (t)\right)\end{array}$$

The residual error of the system at this moment is

$$\begin{array}{l}\mathsf{\Upsilon }={\displaystyle {\int }_t^{t+\Delta t}\left[{\chi }^T(\tau )Q\chi (\tau )+\psi {(t)}^T{\widehat{W}}_u^{(k)}R{\widehat{W}}_u^{(k)T}\psi (\varsigma )\right]}d\tau \\ \hspace{1em}\hspace{1em}-{\displaystyle {\int }_t^{t+\Delta t}2}{\left({\widehat{W}}_u^{(k+1)T}\psi (t)\right)}^{T}R{\chi }_{1}d\tau \\ \hspace{1em}\hspace{1em}+{\widehat{W}}_J^{(k+1)T}\left(\phi (t+\Delta t)-\phi (t)\right)\\ \hspace{1em}={\displaystyle {\int }_t^{t+\Delta t}\left[{\chi }^T(\tau )Q\chi (\tau )+\psi {(t)}^T{\widehat{W}}_u^{(k)}R{\widehat{W}}_u^{(k)T}\psi (\varsigma )\right]}d\tau \\ \hspace{1em}\hspace{1em}-{\displaystyle {\int }_t^{t+\Delta t}2}\left({\chi }^{T}R\right)\otimes {\psi }^T(t)d\tau ·vec\left({\widehat{W}}_u^{(k+1)}\right)\\ \hspace{1em}\hspace{1em}+{\left(\phi (t+\Delta t)-\phi (t)\right)}^T{\widehat{W}}_J^{(k+1)}\end{array}$$

To better represent the residual error and facilitate subsequent derivations, a new weight vector is defined as follows:

$${\widehat{W}}^{(k+1)}=\left[\begin{array}{c}{\widehat{W}}_J^{(k+1)}\\ vec\left({\widehat{W}}_u^{(k+1)}\right)\end{array}\right]$$

(18)

The residual error can be expressed as

$$\mathsf{\Upsilon }={\overline{H}}_1^{(k)}+\overline{H}{\widehat{W}}^{(k+1)}$$

(19)

where

$$\begin{array}{l}{\overline{H}}_1^{(k)}=-{\displaystyle {\int }_t^{t+\Delta t}\left[{\chi }^T(\tau )Q\chi (\tau )+\psi {(t)}^T{\widehat{W}}_u^{(k)}R{\widehat{W}}_u^{(k)T}\psi (\varsigma )\right]}d\tau \\ \overline{H}={\left[\begin{array}{c}\left(\phi (t+\Delta t)-\phi (t)\right)\\ {\displaystyle {\int }_t^{t+\Delta t}2}\left({\chi }^{T}R\right)\otimes {\psi }^T(t)d\tau \end{array}\right]}^T\end{array}$$

To address the above issues, the following weight update law is designed:

$$\dot{\widehat{W}}=-\tau {\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}\mathsf{\Upsilon }$$

(20)

where $\tau$ denotes the adjustable learning rate.

Theorem 1.

For the RLV landing system (9) subject to partial actuator failures, to achieve the optimal performance index (12), the controller defined by Equation (15) is employed, and the update law for the controller and performance approximation network follows Equation (20). Under these conditions, the landing system remains stable and reaches the desired landing point.

Proof. 

Choose the Lyapunov function as follows:

$$V(t)=J(t)+{\displaystyle \frac{1}{\delta }}tr({\tilde{W}}^T\tilde{W})$$

where $\tilde{W} =W^{\ast }- \widehat{W}$; taking the derivative of the above equation yields the following:

$$\dot{V}(t)={\nabla }^{T}J(t)(f\left(\chi \right)+g\left(\chi \right){\widehat{W}}_u^T\psi (t))+{\displaystyle \frac{1}{2\delta }}tr({\tilde{W}}^T\tilde{W})$$

Based on Equation (12),

$$\begin{array}{l}\dot{V}(t)={\widehat{W}}_J^T\nabla \phi (t)\left(f(\chi )+g(\chi ){\widehat{W}}_v^T\psi (t)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}+e{V}_J(\varsigma )+{\displaystyle \frac{1}{2\delta }}{\overline{W}}^T{\tilde{W}}^T\end{array}$$

where $e{V}_J(\varsigma )={\epsilon }_J(\varsigma )$. Substituting this into the Hamiltonian function yields

$$\begin{array}{l}H(\chi ,u^{\ast },J^{\ast })={\chi }^T(t)Q\chi (t)+\psi {(\varsigma )}^T{W}_v^{\ast }R{W}_v^{\ast T}\psi (\varsigma )\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} +W_J^{\ast T}\nabla \phi (\varsigma )\left(f(\chi )+g(\chi )W_v^{\ast T}\psi (\varsigma )\right)+{\epsilon }_{HJI}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =0\end{array}$$

Based on the neural network weight formulation,

$$\begin{array}{l}{\widehat{W}}_J^T\nabla \phi (t)\left(f\left(\chi \right)+g\left(\chi \right){\widehat{W}}_v^T\psi (t)\right)\le \\ -{\chi }^T(t)Q\chi (t)-\psi {(\varsigma )}^T{W}_v^{\ast }R{W}_v^{\ast T}\psi (\varsigma )-{\epsilon }_{HJI}\end{array}$$

For Equation (18),

$$E=H_1^{\ast }-H_1+\overline{H}({\mathsf{\Psi }}^{\ast }-\widehat{\mathsf{\Psi }})$$

Let $e_{H_1}=H_1^{\ast }-H_1$ and $e_{H_1}={\displaystyle {\int }_t^{t+\Delta t}2}\psi {(t)}^T\mathsf{\Xi }\psi (\varsigma )d\tau =\left(2\psi {(t)}^T\mathsf{\Xi }\psi (\varsigma )\right)\Delta t$, where $\mathsf{\Xi }=\left[W_u^{\ast }R{W}_u^{\ast T}-{\widehat{W}}_u^{(k)}R{\widehat{W}}_u^{(k)T}\right]$; therefore $e_{H_1}\le d_1‖{\tilde{W}}_u^{(k)}‖+d_2$, and $d_1$ and $d_2$ are bounded normal variables.

For $tr({\tilde{W}}^T\widehat{W}),$

$$\begin{array}{l}{\displaystyle \frac{1}{2\delta }}tr({\tilde{W}}^T\widehat{W})=tr({\tilde{W}}^T{\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}\mathcal{Ϝ})\\ \le tr({\tilde{W}}^T{\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}\overline{H}({\mathsf{\Psi }}^{\ast }-\widehat{\mathsf{\Psi }})+e_{H_1})\\ =tr({\tilde{W}}^T{\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}\overline{H}{W}^{\ast }-{\tilde{W}}^T{\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}\overline{H}\widehat{\mathsf{\Psi }}\\ \hspace{1em}\hspace{1em} +{\tilde{W}}^T{\displaystyle \frac{\overline{H}}{{\left(\overline{H}{\overline{H}}^T+1\right)}^2}}{e}_{H_1})\\ \le {\displaystyle \frac{d_{\eta }}{\Delta t}}‖\tilde{W}‖\end{array}$$

Therefore, $\dot{V}(t)\le d_{\eta }‖\tilde{W}‖+d_1‖\tilde{W}‖+d_2$. According to the theory of bounded stability of the system [20], the system is boundedly stable. The proof is completed. □

Remark 2.

In the design of partially faulty controllers, thrust efficiency $\eta (t)$ is unknown. However, during system derivation, neural network approximation treats the input matrix and efficiency as a combined entity, thereby simultaneously addressing the uncertainties in both the input matrix and efficiency.

5.2. Design of Fault-Tolerant Controller for Sensor Failures

In Section 5.1, the optimal controller for the landing rocket under partial failure conditions was obtained. This section analyzes the fault-tolerant control problem of the rocket under sensor failure conditions.

The model of the rocket considering sensor failures, as established in Section 3.2, can represent the overall landing rocket system under this fault mode as follows:

$$\dot{\chi }=f(\chi )+g(\chi )(u-f_a)$$

(21)

where $f_a$ represents the thrust loss of the rocket caused by the fault. At this point, according to the performance function in Section 5.1, its derivative is taken.

$$\dot{V}(t)={\nabla }^{T}J(t)(f\left(\chi \right)+g\left(\chi \right){\widehat{W}}_u^T\psi (t)-g\left(\chi \right)f_a)$$

Based on the HJB equation, a further simplification of the above expression yields the following:

$$\begin{array}{l}{\widehat{W}}_J^T\nabla \phi (t)\left(f\left(\chi \right)+g\left(\chi \right){\widehat{W}}_v^T\psi (t)-f_a\right)\\ \le -e^T(t)Qe(t)-\psi {(\varsigma )}^T{W}_v^{\ast }R{W}_v^{\ast T}\psi (\varsigma )-{\epsilon }_{HJI}+{f_a}^{T}R{f}_a\end{array}$$

It can be seen that the designed fault-tolerant controller needs to compensate for the efficiency loss of the actuators caused by faults. Under the premise that the optimal controller for the nominal condition is known, the fault-tolerant controller for rocket landing is implemented as follows:

Theorem 2. 

For the RLV landing system described by (9), to satisfy the optimal performance criterion in (12), a controller as given in (22) is employed. The update laws for both the controller and the performance approximation network follow (20), while the adaptive estimation parameter update law for faults is given by (23). Under these conditions, the landing system remains stable and successfully reaches the desired landing point.

$$\begin{gathered} u(t)=u^{\ast }(t)+{\widehat{f}}_a \\ {u}^{\ast }(t)=W_u^T\psi (t) \end{gathered}$$

(22)

$${\dot{\widehat{f}}}_a=\sigma (2u{\ast }^{T}R+{‖x^{T}g(\chi )‖}_{\max }{)}^T$$

(23)

where ${‖·‖}_{\max }$ denotes the maximum value of the function.

Proof. 

Choose the Lyapunov function as follows:

$$V(t)=J(t)+{\displaystyle \frac{1}{\delta }}tr({\tilde{W}}^T\tilde{W})+{\displaystyle \frac{1}{\sigma }}{{\tilde{f}}_a}^T{\tilde{f}}_a$$

where ${\tilde{f}}_a =f_a -{\widehat{f}}_a$, and $\sigma$ is a constant. Taking the derivative of the above equation yields the following:

$$\begin{array}{l}\dot{V}(t)={\nabla }^{T}J(t)(f\left(\chi \right)+g\left(\chi \right){\widehat{W}}_u^T\psi (t)+{\widehat{f}}_a)\\ \hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle \frac{1}{2\delta }}tr({\tilde{W}}^T\tilde{W})-{\displaystyle \frac{1}{\sigma }}{{\dot{\widehat{f}}}_a}^T{\tilde{f}}_a\end{array}$$

Based on Equation (12), considering ${\nabla }^{T}J(t)={\widehat{W}}_J^T\nabla \phi (t)$,

$$\begin{array}{l}{\widehat{W}}_J^T\nabla \phi (t)\left(f\left(x\right)+g\left(x\right){\widehat{W}}_v^T\psi (t)-f_a\right)\\ \le -e^T(t)Qe(t)-\psi {(\varsigma )}^T{W}_v^{\ast }R{W}_v^{\ast T}\psi (\varsigma )+{f_a}^{T}R{f}_a-{\epsilon }_{HJI}\end{array}$$

Then, combined with the proof of Theorem 1,

$$\dot{V}(t)=d_{\eta }‖\tilde{W}‖+d_1‖\tilde{W}‖+d_2-{\displaystyle \frac{1}{\sigma }}\left(2u{\ast }^{T}R{{\dot{\widehat{f}}}_a}^T-x^{T}g(\chi ){{\dot{\widehat{f}}}_a}^T\right){\tilde{f}}_a$$

By combining this with the update law of ${\widehat{f}}_a$, the system is shown to be bounded and stable, thus completing the proof. □

6. Simulation Results

To verify the performance of the proposed method, it is applied to the landing control of a recoverable rocket. In the simulation, the rocket’s initial position is set to $\left[-3000 \mathrm{m}, 300 \mathrm{m}, 4000 \mathrm{m}\right]$, and the initial velocity is $\left[300 \mathrm{m}/\mathrm{s}, 30 \mathrm{m}/\mathrm{s}, -300 \mathrm{m}/\mathrm{s}\right]$. The mass of the rocket is 48,200 kg, and the rocket’s fault conditions are configured as follows:

(1) For partial failure faults, $\eta (t)=0.8$;

(2) For the sensor fault, ${‖T‖}_{\mathrm{f}}=\mathrm{20,000}(1-e^{-t})$ N.

The basis function selected in the controller design is $\phi (t)=\psi (t)=\left[\begin{array}{cccccc}x& y& z& V_x& V_y& V_z r V\end{array}\right]$. According to the controller designed in this paper, simulations were first conducted under fault-free conditions. The proposed FTC is marked as $u_f$, while a normal controller without considering the fault is marked as $u_{\mathrm{n}}$. The following three cases are considered in the simulation:

Case I: Only the partial failure fault is considered;

Case II: Only the sensor fault is considered;

Case III: Both the partial failure fault and the sensor fault are considered.

The simulation results for the above three cases are shown in the following figures.

Case I: Figure 1 represents the position variation curve, Figure 2 represents the velocity variation curve, Figure 3 shows the rocket’s thrust variation curve, and Figure 4 depicts the rocket’s mass variation curve. As shown in the figures, the designed FTC $u_f$ achieves the stable control of the rocket under fault-free conditions and ensures a stable landing.

Case II: Figure 5 represents the position variation curve, Figure 6 represents the velocity variation curve, Figure 7 shows the rocket’s thrust variation curve, and Figure 8 depicts the rocket’s mass variation curve. As shown in the figures, under sensor fault, the designed FTC $u_f$ achieves the stable control of the rocket under fault-free conditions and ensures a stable landing.

Case III: Figure 9 represents the position variation curve, Figure 10 represents the velocity variation curve, Figure 11 shows the rocket’s thrust variation curve, and Figure 12 depicts the rocket’s mass variation curve. As shown in the figures, the designed FTC $u_f$ achieves the stable control of the rocket under fault-free conditions and ensures a stable landing under both the partial failure fault and the sensor fault.

Based on the simulation results, it can be observed that when the system operates without faults, the designed controller is capable of achieving the safe landing control of the reusable launch vehicle. Furthermore, when faults occur, the designed controller can still ensure the safe landing control of the launch vehicle under fault conditions.

7. Conclusions

This paper addresses the landing problem of RLVs. By analyzing the landing rocket model, the typical underactuated control challenge of the rocket’s control system is identified. Subsequently, through model transformation, a controllable rocket landing model is derived. Considering the presence of uncertainties and external disturbances in the rocket model, a reinforcement learning-based controller is designed. Additionally, a fault-tolerant controller is developed to handle sensor faults in the rocket. The simulation results demonstrate that the proposed controllers ensure the safe and precise landing of the reusable launch vehicle under both fault-free and faulty conditions.