Event-Based Robust Reinforcement Learning Course Tracking for MSVs with Actuator Faults

Abstract

This paper investigates the robust course control for marine surface vehicles (MSVs) by using reinforcement learning and a dynamic event-triggered mechanism under rudder faults. A fault-tolerant optimized tracking control scheme is designed based on reinforcement learning. Firstly, a general actuator fault model is established. Secondly, internal and external disturbances as well as fault dynamics are aggregated into lumped uncertainties, and a fault observer is constructed to estimate and compensate the lumped uncertainty term. Then, a tracking cost function for the MSV system is constructed, and a reinforcement learning framework with a single critic network is used to approximate the value function and generate a learning-based compensation input. Finally, a dynamic event-triggered mechanism is implemented between the control layer and the execution layer to reduce the triggering frequency of actuators. Lyapunov stability theory is utilized to prove that all signals within the closed-loop system are uniformly ultimately bounded. Simulation results verify that the proposed control scheme achieves favorable overall performance.

1. Introduction

Marine surface vehicles (MSVs) are increasingly required to undertake navigation tasks with higher levels of autonomy in long-distance transportation, environmental monitoring, maritime patrol, and offshore operations [1,2,3]. Among the basic motion-control functions of an MSV, automatic course control directly determines whether the vessel can maintain or follow a prescribed heading command in the presence of continuously changing operating conditions. Ship-steering dynamics are characterized by large inertia, pronounced nonlinearities, uncertain hydrodynamic coefficients, and persistent environmental loads induced by wind, waves, and currents. Moreover, long-term operation of steering gears may introduce loss of effectiveness, bias, or other actuator abnormalities, which can further deteriorate course-tracking accuracy and even threaten navigational safety. Meanwhile, frequent rudder actions increase steering-gear activity, mechanical wear, and unnecessary control expenditure. Therefore, a course-control system should not only preserve tracking accuracy and robustness against uncertainties and actuator faults but also account for control performance and actuator utilization [4,5].

A variety of control techniques have been developed for ship autopilots. Owing to their transparent structure and convenient implementation, PID controllers and their variants remain attractive for ship-steering applications. Moving beyond linear feedback, adaptive and nonlinear control techniques have been introduced to cope with uncertain ship dynamics. Skjetne et al. [6] developed an adaptive maneuvering control framework and experimentally validated it using a model ship, demonstrating the capability of nonlinear adaptive control in marine applications. Zhang et al. [7] combined backstepping with neural networks for nonlinear ship course-keeping, where neural approximators and adaptive laws were employed to compensate for unknown nonlinearities and environmental disturbances. These studies substantially improve robustness and tracking performance. Nevertheless, most conventional control strategies are constructed primarily from stability and tracking-error considerations, while the compromise between tracking accuracy and control effort is not explicitly characterized by a performance criterion [8,9].

In the modern day, a wide range of optimization methods have been applied to ship systems [10,11,12,13]. Reinforcement learning (RL) provides an alternative route for nonlinear optimal control by approximating the solution of the Hamilton–Jacobi–Bellman (HJB) equation without requiring its explicit analytical solution [14,15]. Zhang et al. [16] subsequently developed a data-driven approximate optimal tracking framework in which the optimal tracking problem was addressed through RL. These developments stimulated the application of learning-based optimization to marine control systems. In particular, Wen et al. [17] incorporated optimization into the recursive backstepping procedure for vessel tracking and used actor-critic learning structures to approximate the optimal virtual and actual control policies. Chen et al. [18] further investigated adaptive optimal tracking of an underactuated surface vessel by integrating backstepping and actor-critic reinforcement learning in the presence of uncertain dynamics and external disturbances. Although the aforementioned methods verified the application potential of RL in improving tracking performance, adaptively adjusting the actor-critic network simultaneously introduces additional network weights and update rules, which increases structural complexity. More critically, if actuator performance degradation is taken into account, the difficulty of optimal heading control for MSVs will rise significantly.

Actuator faults constitute another important issue in autonomous marine control [19,20,21]. Zhu et al. [22] studied tracking control of MSVs subject to actuator faults, input saturation, parameter uncertainties, and environmental disturbances, and developed an appointed-time fault-tolerant tracking strategy. In addition to fault accommodation, reducing unnecessary control updates has become increasingly relevant because continuously refreshed control commands may result in excessive actuator activity. Event-triggered control addresses this issue by updating the transmitted control signal only when a prescribed condition is satisfied. Zhu et al. [23] developed an event-triggered adaptive neural fault-tolerant controller for underactuated MSVs with actuator faults and saturation, thereby reducing actuator execution while maintaining tracking stability. More recently, Shen et al. [24] combined ADP with a dynamic event-triggered mechanism for constrained optimal tracking of MSVs. These results provide important progress toward resource-efficient intelligent marine control. However, fault compensation, optimized course regulation, and dynamic event-triggered actuator operation are often treated separately or require relatively complicated learning architectures. In particular, simultaneously accounting for rudder faults, lumped internal and external uncertainties, optimal course-tracking performance, and actuator execution frequency using a compact single-critic learning architecture remains a problem worthy of further investigation. As summarized in Table 1, unlike representative existing methods that address only subsets of performance awareness, fault tolerance, disturbance rejection, and dynamic event-triggered implementation, the proposed scheme integrates all four capabilities within a unified framework, highlighting its methodological novelty and comprehensive control performance.

Table 1. Different control performance comparison.

Control Method	Performance-Aware	Fault-Tolerant	Disturbance Rejection	Dynamic ET
[17]	√
[14]	√		√
[19]		√	√	√
[23]		√		√
[24]	√			√
Proposed Method	√	√	√	√

√ indicates that the corresponding control method possesses the corresponding property.

Motivated by the above observations, this paper develops an event-based optimized course-tracking scheme for MSVs subject to actuator faults. The main contributions are summarized as follows.

• A fault-tolerant course-tracking controller is developed under the RL framework. In contrast to conventional actor-critic implementations requiring separate policy and value networks [14,16,17,18], only a critic network is employed to approximate the value function, from which the approximate optimal control input is obtained. The compensation policy is constructed from the estimated value-function gradient, avoiding the need for a separate actor network. This provides a compact learning structure for the proposed control scheme.
• A dynamic event-triggered mechanism is inserted between the control layer and the actuator execution layer. The continuously generated optimized command is therefore transmitted to the actuator only when the triggering condition is activated, reducing unnecessary rudder updates and execution frequency.

2. Problem Formulations and Preliminaries

2.1. Problem Formulation

The model of the MSV course-control system is shown as follows [25]:

$$\left\{\begin{array}{c}\dot{\phi }=r\hfill \\ \dot{r}=-a_{1}r-a_2{r}^3-a_3{r}^5+g{\delta }_f+\mathsf{\Delta }\hfill \end{array}\right.$$

(1)

where $\phi$ is the heading angle, and r is the yaw velocity. $g=K_0/T_0$ is the gain of the control input. $K_0$ is the ship turning index and $T_0$ is the following index. ${\delta }_f$ is the faulty rudder. $\mathsf{\Delta }$ is the unknown ocean disturbance affect on the rudder. $a_i$ is the unknown coefficient of r. ${\phi }_r$ is the desired signal.

2.2. Fault Rudder Model of the MSV

The fault model of the rudder can be written as [26]

$${\delta }_f=\kappa \left(t\right)\delta +\rho \left(t\right)$$

(2)

where $0\le \kappa \left(t\right)\le 1$ and $\rho \left(t\right)$ denote the rudder loss efficiency factor and bounded additive fault signal. It can be assumed that $\left|\dot{\kappa }\left(t\right)\right|\le {\kappa }_m$ is bounded and $\rho \left(t\right)$ is an unknown varying constant. Based on $\kappa \left(t\right)$ and $\rho \left(t\right)$, there are three conditions:

• Fault-free: When $\kappa \left(t\right)=1$ and $\rho \left(t\right)=0$, there exists ${\delta }_f=\delta$.
• Loss-of-effectiveness fault: When $0<\kappa \left(t\right)<1$ and $\rho \left(t\right)=0$, there exists ${\delta }_f=\kappa \left(t\right)\delta$.
• Bias fault: When $\rho \left(t\right)\ne 0$, there exists ${\delta }_f=\kappa \left(t\right)\delta +\rho \left(t\right)$.

Lemma 1.

For any given continuous function $\sigma \left(\chi \right)$ with $\sigma \left(0\right)=0$, defined in a compact set ${\mathsf{\Omega }}_{\chi }\subseteq \mathcal{R}$, $\sigma \left(\chi \right)$ can be approximated by the NN with arbitrary precision [17], namely,

$$\sigma \left(\chi \right)={\theta }^{\top }\vartheta \left(\chi \right)+ϵ$$

(3)

where θ denotes the weight vector of the neural networks. $\vartheta \left(\chi \right)$ is the basis function. ϵ is the bounded approximation error.

3. Event-Based RL Course-Tracking Design

3.1. Fault-Tolerant Tracking Control Design

Establish the following error surfaces as

$${\epsilon }_{\phi }=\phi -{\phi }_r$$

(4)

$${\epsilon }_r=r-\alpha$$

(5)

where $\alpha$ denotes the virtual optimal course signal, which can be defined as $\alpha ={\alpha }_a+{\alpha }_b$.

Step 1: Differentiating the first error surface (4) and using $r={\epsilon }_r+\alpha$ gives

$$\begin{array}{cc}\hfill {\dot{\epsilon }}_{\phi }& =\dot{\phi }-{\dot{\phi }}_r\hfill \\ \hfill & ={\epsilon }_r+{\alpha }_a+{\alpha }_b-{\dot{\phi }}_r\hfill \end{array}$$

(6)

Select the candidate for the Lyapunov function in the following:

$$V_{\phi }={\displaystyle \frac{1}{2}}{\epsilon }_{\phi }^2$$

(7)

Taking the time derivative of (7) gives

$${\dot{V}}_{\phi }={\epsilon }_{\phi }({\epsilon }_r+{\alpha }_a+{\alpha }_b-{\dot{\phi }}_r)$$

(8)

Design the virtual control as

$${\alpha }_a=-c_{\phi }{\epsilon }_{\phi }+{\dot{\phi }}_r$$

(9)

where $c_{\phi }$ is a positive constant to be designed.

Substituting (9) into (8) yields

$${\dot{V}}_{\phi }=-c_{\phi }{\epsilon }_{\phi }^2+{\epsilon }_{\phi }{\epsilon }_r+{\epsilon }_{\phi }{\alpha }_b.$$

(10)

This expression shows that the nominal virtual control ${\alpha }_a$ introduces the stabilizing term $-c_{\phi }{\epsilon }_{\phi }^2$, whereas ${\alpha }_b$ is retained as the RL-based compensation input.

Step 2: Taking the derivative of the second error surface (5) gives

$$\begin{array}{cc}\hfill {\dot{\epsilon }}_r& =\dot{r}-\dot{\alpha }\hfill \\ \hfill & =f\left(r\right)+g(\kappa \left(t\right)\delta +\rho \left(t\right))+\mathsf{\Delta }-\dot{\alpha }\hfill \\ \hfill & =\delta +\left[f\left(r\right)-f\left(\alpha \right)\right]+f\left(\alpha \right)+(g\kappa -1)\delta +g\rho +\mathsf{\Delta }-\dot{\alpha }\hfill \\ \hfill & =\delta +o\left(\gamma \right)+{\theta }_a^{\top }{\vartheta }_a\left(\alpha \right)+{ϵ}_a+(g\kappa -1)\delta +g\rho +\mathsf{\Delta }-\dot{\alpha }\hfill \end{array}$$

(11)

where $f\left(r\right)=-a_{1}r-a_2{r}^3-a_3{r}^5$. In the third line of (11), $f\left(\alpha \right)$ and $\delta$ are added and subtracted to separate the known input channel from the nonlinear uncertainty. Define $\gamma ={[r,\alpha ]}^{\top }$, $o\left(\gamma \right)=f\left(r\right)-f\left(\alpha \right)$, and $f\left(\alpha \right)={\theta }_a^{\top }{\vartheta }_a\left(\alpha \right)+{ϵ}_a$, where ${\theta }_a$ is the ideal NN weight vector, ${\vartheta }_a\left(\alpha \right)$ is the basis-function vector, and ${ϵ}_a$ is the bounded approximation error. The composite uncertainty is then defined as

$$\mathsf{\Theta }=\mathsf{\Delta }+(g\kappa -1)\delta +{ϵ}_a+g\rho$$

(12)

The lumped term $\mathsf{\Theta }$ is estimated online using the auxiliary observer state $\nu$, as follows:

$$\left\{\begin{array}{c}\nu =\widehat{\mathsf{\Theta }}-b_{1}r\hfill \\ \dot{\nu }=-b_1\nu -b_1\left(\widehat{f}\left(r\right)+\delta +b_{1}r\right)\hfill \end{array}\right.$$

(13)

where $b_1>0$ is the observer gain and $\widehat{f}\left(r\right)$ denotes the available NN estimate of $f\left(r\right)$. From the first equation in (13), the lumped-uncertainty estimate is reconstructed as $\widehat{\mathsf{\Theta }}=\nu +b_{1}r$.

The stability of the designed observer can be proved by [19]. Following the backstepping structure in Step 2, the stabilizing controller and its adaptation law are designed as

$${\delta }_a=-c_r{\epsilon }_r-{\epsilon }_{\phi }-{\widehat{\theta }}_a^{\top }{\vartheta }_a\left(\alpha \right)+\dot{\alpha }-\widehat{\mathsf{\Theta }}$$

(14)

$${\dot{\widehat{\theta }}}_a={\epsilon }_r{\vartheta }_a\left(\alpha \right)-c_{\theta }{\widehat{\theta }}_a$$

(15)

where $c_r>0$ and $c_{\theta }>0$ are design parameters. Define the NN weight error as ${\tilde{\theta }}_a={\theta }_a-{\widehat{\theta }}_a$.

Substituting $\delta ={\delta }_a+{\delta }_b$, (12), and (14) into (11) gives the explicit second-error dynamics

$${\dot{\epsilon }}_r=-c_r{\epsilon }_r-{\epsilon }_{\phi }+{\delta }_b+o\left(\gamma \right)+{\tilde{\theta }}_a^{\top }{\vartheta }_a\left(\alpha \right).$$

(16)

This equation makes explicit the stabilizing feedback terms, the RL input ${\delta }_b$, the NN weight error, and the residual nonlinear/observer errors.

Choose the augmented Lyapunov function

$$V_r={\displaystyle \frac{1}{2}}{\epsilon }_{\phi }^2+{\displaystyle \frac{1}{2}}{\epsilon }_r^2+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_a^{\top }{\tilde{\theta }}_a$$

(17)

Taking the derivative of (17), using the Step-1 result, and substituting the second-error dynamics above yields

$$\begin{array}{cc}\hfill {\dot{V}}_r=& -c_{\phi }{\epsilon }_{\phi }^2-c_r{\epsilon }_r^2+{\epsilon }_{\phi }{\alpha }_b+{\epsilon }_r{\delta }_b\hfill \\ \hfill & +{\epsilon }_{r}o\left(\gamma \right)+{\epsilon }_r{\tilde{\theta }}_a^{\top }{\vartheta }_a\left(\alpha \right)-{\tilde{\theta }}_a^{\top }{\dot{\widehat{\theta }}}_a.\hfill \end{array}$$

(18)

Using (15), the two terms containing ${\epsilon }_r{\tilde{\theta }}_a^{\top }{\vartheta }_a\left(\alpha \right)$ cancel. In addition,

$$c_{\theta }{\tilde{\theta }}_a^{\top }{\widehat{\theta }}_a=c_{\theta }{\tilde{\theta }}_a^{\top }({\theta }_a-{\tilde{\theta }}_a)\le {\displaystyle \frac{c_{\theta }}{2}}\parallel {\theta }_a{\parallel }^2-{\displaystyle \frac{c_{\theta }}{2}}{\parallel {\tilde{\theta }}_a\parallel }^2.$$

(19)

The derivative of $V_r$ therefore satisfies

$$\begin{array}{cc}\hfill {\dot{V}}_r& \le -c_{\phi }{\epsilon }_{\phi }^2-c_r{\epsilon }_r^2+{\displaystyle \frac{1}{2}}{c}_{\theta }\parallel {\theta }_a{\parallel }^2-{\displaystyle \frac{1}{2}}{c}_{\theta }{\parallel {\tilde{\theta }}_a\parallel }^2\hfill \\ \hfill & +{\mathcal{E}}^T\left[\begin{array}{c}0\\ \mathcal{O}\left(\gamma \right)\end{array}\right]+{\mathcal{E}}^T\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]{\mathcal{U}}_b\hfill \end{array}$$

(20)

where $\mathcal{E}={[{\epsilon }_{\phi },{\epsilon }_r]}^{\top }$ and ${\mathcal{U}}_b={[{\alpha }_b,{\delta }_b]}^{\top }$. Equation (20) clearly separates the negative feedback terms, the bounded residual term, and the RL compensation input. According to [26], the remaining task is to design ${\mathcal{U}}_b$ so that the error system is stabilized while the prescribed performance index is minimized.

3.2. RL-Based Compensation Design

Based on the residual and compensation terms identified in (20), the subsystem optimized by RL is written compactly as

$$\dot{\mathcal{E}}=F\left(\mathcal{E}\right)+{\mathcal{U}}_b$$

(21)

where $F\left(\mathcal{E}\right)={[0,\mathcal{O}\left(\gamma \right)]}^{\top }$ represents the bounded residual channel after the nominal backstepping terms have been introduced. The cost function is defined as

$$\mathcal{J}={\int }_t^{\infty }\left({\mathcal{E}}^{\top }\mathcal{Q}\mathcal{E}+{\mathcal{U}}_b^{\top }\mathcal{R}{\mathcal{U}}_b\right)dt$$

(22)

where $\mathcal{Q}$ and $\mathcal{R}$ are positive definite matrices.

For an admissible initial control ${\overline{\mathcal{U}}}_b$, the Hamiltonian function is defined as

$$\mathcal{H}={\mathcal{E}}^{\top }\mathcal{Q}\mathcal{E}+{\overline{\mathcal{U}}}_b^{\top }\mathcal{R}{\overline{\mathcal{U}}}_b+{\left({\nabla }_{\mathcal{E}}\mathcal{J}\right)}^{\top }\left(F\left(\mathcal{E}\right)+{\overline{\mathcal{U}}}_b\right)$$

(23)

where ${\nabla }_{\mathcal{E}}\mathcal{J}$ denotes the column gradient of $\mathcal{J}$ with respect to $\mathcal{E}$. Applying the stationary condition $\partial \mathcal{H}/\partial {\mathcal{U}}_b=0$ gives

$${\mathcal{U}}_b^{*}=-{\displaystyle \frac{1}{2}}{\mathcal{R}}^{-1}{\nabla }_{\mathcal{E}}{\mathcal{J}}^{*}$$

(24)

Substitution of (24) into the Hamiltonian yields the HJB equation

$$0={\mathcal{E}}^{\top }\mathcal{Q}\mathcal{E}+{\left({\nabla }_{\mathcal{E}}{\mathcal{J}}^{*}\right)}^{\top }F\left(\mathcal{E}\right)-{\displaystyle \frac{1}{4}}{\left({\nabla }_{\mathcal{E}}{\mathcal{J}}^{*}\right)}^{\top }{\mathcal{R}}^{-1}{\nabla }_{\mathcal{E}}{\mathcal{J}}^{*}$$

(25)

The NN is used to approximate the critic term

$$\begin{array}{cc}\hfill \mathcal{J}& ={\theta }_b^{\top }{\vartheta }_b\left(\mathcal{E}\right)+{ϵ}_b\hfill \\ \hfill & ={\left[\begin{array}{ccc}{\theta }_{b1}^{\top }& \dots & 0_{1\times n}\\ ⋮& \ddots & ⋮\\ 0_{1\times n}& \dots & {\theta }_{bn}^{\top }\end{array}\right]}^{\top }\left[\begin{array}{c}{\vartheta }_{b1}\\ ⋮\\ {\vartheta }_{bn}\end{array}\right]+\left[\begin{array}{c}{ϵ}_{b1}\\ ⋮\\ {ϵ}_{bn}\end{array}\right]\hfill \end{array}$$

(26)

where ${\theta }_b$ is the adaptive parameter of the critic term and ${ϵ}_b$ is the approximation error.

Hence, the approximations of the optimal term are shown as

$${\widehat{\mathcal{U}}}_b=-{\displaystyle \frac{1}{2}}{\mathcal{R}}^{-1}\nabla {\vartheta }_{\epsilon }^T{\widehat{\theta }}_b$$

(27)

with the update principle as

$$\begin{array}{cc}\hfill {\dot{\widehat{\theta }}}_b=-c_b& \left[\begin{array}{c}{(\nabla {\vartheta }_{b\epsilon }^{\top })}^{\top }\widehat{F}\left(\mathcal{E}\right)\\ -{\displaystyle \frac{1}{2}}{(\nabla {\vartheta }_{b\epsilon }^{\top })}^{\top }{\mathcal{R}}^{-1}(\nabla {\vartheta }_{b\epsilon }^{\top }){\widehat{\theta }}_b\end{array}\right]\hfill \\ \hfill & \times \left[\begin{array}{c}{\mathcal{E}}^{\top }Q\mathcal{E}+{\widehat{\theta }}_b^{\top }{(\nabla {\vartheta }_{b\epsilon }^{\top })}^{\top }\widehat{F}\left(\mathcal{E}\right)\\ -{\displaystyle \frac{1}{4}}{\widehat{\theta }}_b^{\top }{(\nabla {\vartheta }_{b\epsilon }^{\top })}^{\top }{\mathcal{R}}^{-1}(\nabla {\vartheta }_{b\epsilon }^{\top }){\widehat{\theta }}_b\end{array}\right]\hfill \end{array}$$

(28)

where $c_b$ is the NN learning rate parameter. The optimal fault-tolerant tracking design can be shown as $\delta ={\delta }_a+{\widehat{\delta }}_b$.

3.3. Event-Based Mechanism

To reduce excessive wear of the rudder and save communication resources in the control–actuator link, this paper proposes a event-based mechanism to enhance the effect and scope of optimal control. The proposed mechanism is expressed as follows:

$$\left\{\begin{array}{c}{E}_{\delta }\left(t\right)=\widehat{\delta }\left(t_k\right),\forall t\in [t_k,t_{k+1})\hfill \\ t_{k+1}=\inf \{t>t_k\mid \lambda \zeta \left(t\right)\le \left|e_{\delta }\left(t\right)\right|\}\hfill \\ \dot{\zeta }\left(t\right)=-\lambda \zeta \left(t\right)+\varpi \left|e_{\delta }\left(t\right)\right|,\hfill \end{array}\right.$$

(29)

where $e_{\delta }\left(t\right)=\delta \left(t\right)-\widehat{\delta }\left(t_k\right)$, $\zeta \left(0\right)>0$ is the initial value of the internal dynamic variable, $\lambda >0$ is the threshold attenuation coefficient, and $\varpi \ge 0$ is a design parameter. $E_{\delta }$ is the actuator input held between two consecutive triggering instants.

3.4. Stability Analysis

Theorem 1.

Consider the MSV course-tracking system subject to the actuator faults in (2). Under the fault-tolerant controller (14), the RL-based compensation controller (27), the critic updating law (28), and the dynamic event-triggered mechanism (29), suppose that the desired course signal and its required derivatives are bounded, the NN approximation errors are bounded, and the critic regressor satisfies the persistent excitation condition. Then, by appropriately selecting the controller gains and the critic learning gain, all signals in the resulting closed-loop system are uniformly ultimately bounded. Moreover, the triggering sequence generated by (29) has no finite accumulation point; therefore, Zeno behavior is excluded.

Proof. 

Choose the sum of the Lyapunov function candidate as

$$V_{all}=V_r+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_b^{\top }{\tilde{\theta }}_b$$

(30)

According to the error transformation and controller (14), the derivative of $V_r$ can be rearranged as

$$\begin{array}{cc}\hfill {\dot{V}}_r& \le -c_{\phi }{\epsilon }_{\phi }^2-c_r{\epsilon }_r^2-{\displaystyle \frac{1}{2}}{c}_{\theta }{∥{\tilde{\theta }}_a∥}^2+{\displaystyle \frac{1}{2}}{c}_{\theta }{∥{\theta }_a∥}^2\hfill \\ \hfill & +{\mathcal{E}}^{\top }\left[F\left(\mathcal{E}\right)+{\widehat{\mathcal{U}}}_b\right]+D_a\hfill \end{array}$$

(31)

$D_a$ collects the bounded NN approximation error, residual fault-observation error and other bounded uncertainty terms.

From the ideal cost function

$${\mathcal{J}}^{*}\left(\mathcal{E}\right)={\theta }_b^{*\top }{\vartheta }_b\left(\mathcal{E}\right)+{ϵ}_b\left(\mathcal{E}\right),$$

(32)

The ideal optimal compensation term can be obtained as

$${\mathcal{U}}_b^{*}=-{\displaystyle \frac{1}{2}}{\mathcal{R}}^{-1}{\left(\nabla {\vartheta }_b\right)}^{\top }{\theta }_b^{*}+{ϵ}_u$$

(33)

where ${ϵ}_u=-{\displaystyle \frac{1}{2}}{\mathcal{R}}^{-1}\nabla {ϵ}_b$ represents the error caused by the approximation of the critic function.

Consider the actual control input as (27). Therefore, it has

$${\widehat{\mathcal{U}}}_b-{\mathcal{U}}_b^{*}=-{\displaystyle \frac{1}{2}}{\mathcal{R}}^{-1}{(\nabla {\vartheta }_b)}^{\top }{\tilde{\theta }}_b-{ϵ}_u$$

(34)

Based on the universal approximation theorem of neural networks and reference [24], we can assume that $\parallel \nabla {\vartheta }_b\parallel \le {\overline{\vartheta }}_b$, $∥{ϵ}_b∥⩽{ϵ}_{bM}$, $∥{\theta }_b∥⩽{\theta }_{bM}$, and $\parallel {ϵ}_u\parallel \le {\overline{ϵ}}_u$.

Define

$$l_b={\displaystyle \frac{1}{2}}∥{\mathcal{R}}^{-1}∥{\overline{\vartheta }}_b$$

(35)

Then, we yield

$$∥{\widehat{\mathcal{U}}}_b-{\mathcal{U}}_b^{*}∥\le l_b∥{\tilde{\theta }}_b∥+{\overline{ϵ}}_u$$

(36)

Hence,

$${\mathcal{E}}^{\top }\left({\widehat{\mathcal{U}}}_b-{\mathcal{U}}_b^{*}\right)\le l_b∥{\tilde{\theta }}_b∥∥\mathcal{E}∥+{\overline{ϵ}}_u∥\mathcal{E}∥$$

(37)

Applying Young’s inequality gives, for arbitrary constants,

$$l_b∥{\tilde{\theta }}_b∥∥\mathcal{E}∥\le {\displaystyle \frac{1}{2}}{\eta }_1{∥\mathcal{E}∥}^2+{\displaystyle \frac{l_b^2}{2{\eta }_1}}{∥{\tilde{\theta }}_b∥}^2$$

(38)

and

$${\overline{ϵ}}_u\parallel \mathcal{E}\parallel \le {\displaystyle \frac{1}{2}}{\eta }_2{\parallel \mathcal{E}\parallel }^2+{\displaystyle \frac{1}{2{\eta }_2}}{\overline{ϵ}}_u^2$$

(39)

where ${\eta }_1, {\eta }_2>0$.

In summary, it can be obtained that

$$\begin{array}{cc}\hfill {\dot{V}}_{all}& \le -\left(c_0-{\displaystyle \frac{{\eta }_1+{\eta }_2}{2}}\right){\parallel \mathcal{E}\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{c_{\theta }}{2}}\parallel {\overline{\theta }}_a{\parallel }^2-{\displaystyle \frac{l_b^2}{2{\eta }_1}}{\parallel {\overline{\theta }}_b\parallel }^2+D_r\hfill \end{array}$$

(40)

where $D_r=D_a+{\displaystyle \frac{1}{2{\eta }_2}}{\overline{ϵ}}_u^2$.

The derivative bound in Equation (40) can equivalently be written in the transpose-times-vector form as

$$\begin{array}{cc}\hfill {\dot{V}}_{\mathrm{all}}\le & -\left(c_0-{\displaystyle \frac{{\eta }_1+{\eta }_2}{2}}\right){\mathcal{E}}^{\top }\mathcal{E}-{\displaystyle \frac{c_{\theta }}{2}}{\tilde{\theta }}_a^{\top }{\tilde{\theta }}_a-{\displaystyle \frac{l_b^2}{2{\eta }_1}}{\tilde{\theta }}_b^{\top }{\tilde{\theta }}_b+D_r.\hfill \end{array}$$

(41)

Since $V_{\mathrm{all}}={\displaystyle \frac{1}{2}}{\mathcal{E}}^{\top }\mathcal{E}+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_a^{\top }{\tilde{\theta }}_a+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_b^{\top }{\tilde{\theta }}_b$, the preceding inequality gives

$${\dot{V}}_{\mathrm{all}}\le -\lambda V_{\mathrm{all}}+D_r,$$

(42)

where $\lambda =\min \left\{2c_0-({\eta }_1+{\eta }_2),c_{\theta },{\displaystyle \frac{l_b^2}{{\eta }_1}}\right\}>0$.

Thus, Equation (40) can also be written as

$${\dot{V}}_{\mathrm{all}}\le -\lambda V_{\mathrm{all}}+D_r.$$

(43)

where $\lambda =\min \{2c_0-\left({\eta }_1+{\eta }_2\right),c_{\theta },{\displaystyle \frac{l_b^2}{{\eta }_1}}\}>0$.

Since the desired course signal and its derivatives are bounded, the system states, adaptive parameters, and optimized commands are bounded. Consequently, all signals in the closed-loop system are UUB.

It remains to prove that the triggering mechanism is free of Zeno behavior. From (29), $\varpi \ge 0$, and $|e_{\delta }\left(t\right)|\ge 0$, one has

$$\dot{\zeta }\left(t\right)\ge -\lambda \zeta \left(t\right),$$

(44)

which, together with $\zeta \left(0\right)>0$, yields

$$\zeta \left(t\right)\ge \zeta \left(0\right)e^{-\lambda t}>0.$$

(45)

According to the preceding boundedness result, the continuously generated rudder command $\delta \left(t\right)$ evolves in a compact set. Because the closed-loop vector field, the selected basis functions, and the desired signal are smooth with bounded required derivatives, there exists a constant ${\overline{d}}_{\delta }>0$ such that $|\dot{\delta }\left(t\right)|\le {\overline{d}}_{\delta }$. At each triggering instant, the newly computed command is transmitted to the actuator, so that $\widehat{\delta }\left(t_k\right)=\delta \left(t_k\right)$ and $e_{\delta }\left(t_k\right)=0$. Therefore, for all $t\in [t_k,t_{k+1})$,

$$\left|e_{\delta }\left(t\right)\right|=\left|{\int }_{t_k}^t\dot{\delta }\left(s\right)\mathrm{d}s\right|\le {\overline{d}}_{\delta }(t-t_k).$$

(46)

Suppose, by contradiction, that infinitely many triggering instants accumulate at a finite time $T_z$. For every $t\in [0,T_z]$, the dynamic threshold satisfies

$$\lambda \zeta \left(t\right)\ge \lambda \zeta \left(0\right)e^{-\lambda T_z}=:{\underline{\mu }}_z>0.$$

(47)

If a subsequent event occurs, the continuity of $e_{\delta }\left(t\right)$ and $\zeta \left(t\right)$ implies that the triggering boundary is reached at $t_{k+1}$, namely $|e_{\delta }\left(t_{k+1}^{-}\right)|=\lambda \zeta \left(t_{k+1}\right)$. It follows that

$$t_{k+1}-t_k\ge {\displaystyle \frac{{\underline{\mu }}_z}{{\overline{d}}_{\delta }}}>0.$$

(48)

Thus, all inter-event times before $T_z$ possess a strictly positive lower bound, contradicting the assumed finite accumulation of triggering instants. If no subsequent event occurs, Zeno behavior is trivially impossible. Hence, the proposed dynamic event-triggered control scheme is Zeno-free. □

4. Simulation Results and Discussion

4.1. Simulation Results

To validate the proposed fault-tolerant RL control strategy, numerical studies are conducted using the sea-going training vessel Yulong as the simulation platform, with its parameters found in [25].

In the simulations, the initial states are chosen as ${\psi }_0=5^{\circ }$ and $r_0=0.{01}^{\circ }/\mathrm{s}$; the observer state and parameter are designed as ${\nu }_0=0.1$ and $b_1=10$. For the optimal compensation term design, select $\mathcal{Q}=5$ and $\mathcal{R}=2$. The parameters are selected as $c_{\phi }=1.5$ and $c_r=5$. The initial adaptive parameter and weight are chosen randomly in $(0,1)$. The adaptive parameter is set as $c_{\theta }=0.02$ and the learning rate is $c_b=0.01$. Gaussian basis functions are selected as the basis functions of the neural network. In addition, we take into account the practical requirement that the rudder angle shall not exceed ±${35}^{\circ }$, and impose constraints on the control input.

To support persistent excitation (PE), the excitation signal added to the rudder-command channel is specified as

$$p\left(t\right)=s_e\left(t\right){\left[0.5\sin \left(0.05t\right)+0.3\sin \left(0.13t\right)+0.2\sin \left(0.23t\right)\right]}^{\circ }$$

(49)

with $s_e\left(t\right)=\left\{\begin{array}{c}[1-\cos (\pi t/10\left)\right]/2,0\le t<10\mathrm{s}\hfill \\ 1,t\ge 10\mathrm{s}\hfill \end{array}\right.$. The minimum value within $minp\left(t\right)$ in 0–1200 s of the computation window is $-0.{714}^{\circ }$. The eigenvalues, i.e., the window energy, can be expressed by the following formula:

$$G_p\left(t\right)={\int }_t^{t+T}{p}^2\left(\tau \right)\mathrm{d}\tau .$$

Window length is $T=250\mathrm{s}$; we can thus calculate ${\lambda }_{min}\left(G_p\left(t\right)\right)$.

We select autopilot tracking signals that reflect specific actual performance requirements as follows:

$${\ddot{\phi }}_m\left(t\right)+0.1{\dot{\phi }}_m\left(t\right)+0.0025{\phi }_m\left(t\right)=0.0025{\phi }_r\left(t\right).$$

(50)

Among them, ${\phi }_m$ represents the ideal system performance characterizing the heading; ${\phi }_r$ is the command input signal, whose value ranges from $0^{\circ }$ to ${30}^{\circ }$.

The external disturbance is chosen as

$$\mathsf{\Delta }=0.025\sin \left(0.05t\right)+0.015\cos \left(0.03t\right).$$

(51)

To verify the effectiveness of the algorithm proposed in this paper, two cases are used for specific demonstration. Among them, Case 1 and Case 2 demonstrate the performance under moderate magnitude faults and higher magnitude faults, respectively. This is shown as

$$\begin{array}{c}\hfill {\delta }_{f1}=\left\{\begin{array}{cc}\delta ,\hfill & 0<t\le 600 \mathrm{s}\hfill \\ 0.6\left|\sin \right(0.02t\left)\right|\times \delta +5\sin \left(0.1t\right),\hfill & t\ge 600 \mathrm{s}\hfill \end{array}\right.\end{array}$$

(52)

$$\begin{array}{c}\hfill {\delta }_{f2}=\left\{\begin{array}{cc}\delta ,\hfill & 0<t\le 600 \mathrm{s}\hfill \\ 0.4\left|\sin \right(0.02t\left)\right|\times \delta +7\sin \left(0.1t\right).\hfill & t\ge 600 \mathrm{s}\hfill \end{array}\right.\end{array}$$

(53)

It is particularly pointed out that in order to ensure the validity of the algorithm comparison, the parameters and initial values in these control experiments are designed to be basically consistent.

Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 and Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16 present the simulation results for Cases 1 and 2, respectively. In Case 1, the actual course closely follows the desired profile, while the yaw-rate and tracking-error responses exhibit only short transients near the command changes and remain close to zero during the steady portions. The constrained rudder input remains within the prescribed range, and the estimated lumped uncertainty nearly overlaps its actual value over the simulated interval. The event-triggering instants are nonuniform, and the accumulated number of updates is lower than that of the time-triggered implementation; meanwhile, the adaptive parameters and critic weights remain bounded. In Case 2, the course is also tracked over most of the simulation, but a visibly larger transient occurs at approximately 1000 s under the higher-magnitude fault. This transient is accompanied by larger excursions in the yaw rate, tracking error, rudder demand, and adaptive parameter, after which these responses decrease toward their previous levels. The observer still follows the dominant variation of the lumped uncertainty, and the event-triggered updates remain intermittent. Therefore, the two cases indicate bounded closed-loop responses and reduced update activity under the tested conditions, while the Case 2 results also show that the transient tracking accuracy deteriorates when the fault magnitude is increased. Figure 17 shows an overall inward-winding trend of the error trajectories toward a neighborhood of the origin in both cases, with larger intermediate excursions in Case 2.

Figure 1. Course-tracking performance of the RL fault-tolerant control scheme in Case 1.

Figure 2. Yaw-rate response of the MSV in Case 1.

Figure 3. Course-tracking errors in Case 1.

Figure 4. Rudder control input under the prescribed constraint in Case 1.

Figure 5. Lumped-uncertainty estimation performance of the observer in Case 1.

Figure 6. Performance of the dynamic event-triggered mechanism in Case 1.

Figure 7. Convergence behavior of the adaptive parameters in Case 1.

Figure 8. Convergence behavior of the critic-network weights in Case 1.

Figure 9. Course-tracking performance of the RL fault-tolerant control scheme in Case 2.

Figure 10. Yaw-rate response of the MSV in Case 2.

Figure 11. Course-tracking errors in Case 2.

Figure 12. Rudder control input under the prescribed constraint in Case 2.

Figure 13. Lumped-uncertainty estimation performance of the observer in Case 2.

Figure 14. Performance of the dynamic event-triggered mechanism in Case 2.

Figure 15. Convergence behavior of the adaptive parameters in Case 2.

Figure 16. Convergence behavior of the critic-network weights in Case 2.

Figure 17. Inward-spiral convergence of error surface in two cases. (a) Case 1; (b) Case 2.

The simulation results for Case 2 are presented in Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16, following the same order as the results for Case 1.

4.2. Discussion

Theoretical guarantees. The Lyapunov analysis establishes uniform ultimate boundedness of the closed-loop signals, while the event-triggered analysis excludes Zeno behavior by deriving a positive lower bound on the inter-event time. These are analytical results under the assumptions stated in this manuscript, including bounded signals, approximation errors, and sufficient control authority.

Numerical performance and trade-offs. The two simulation cases show that the course is tracked under the tested fault conditions, but also reveal a limitation: the higher-magnitude fault in Case 2 produces visibly larger transient excursions around 1000 s. The benchmark table makes the tracking–actuation trade-off more explicit. For example, the course RMSE of the proposed controller is 0.386° in Case 1 and 3.613° in Case 2, compared with 0.215° and 3.401°, respectively, for the time-triggered controller. Thus, in these simulations, the event-triggered implementation reduces update activity but does not achieve the time-triggered controller’s lower RMSE. The results also show favorable course RMSE and computation time relative to the reported method [14] under the same test settings (Table 2).

Table 2. Indicator comparison of different controllers.

Controller	Case 1			Case 2
	Proposed Controller	Time-Triggered Controller	[14]	Proposed Controller	Time-Triggered Controller	[14]
${\mathrm{RMSE}}_{\phi }$ (°)	0.386	0.215	0.505	3.613	3.401	7.324
${\mathrm{RMSE}}_{\theta }$ (°)	0.690	0.431	Not considered	5.922	4.647	Not considered
Time Consumption (s)	5.540	5.134	10.472	7.746	7.035	15.819
Triggered Times	678	12001	1121	710	12001	1338

Practical applicability. The evidence remains numerical and is based on the specified Yulong heading-dynamics model and simulation settings. This manuscript does not claim experimental validation or establish performance across different vessels, operating conditions, or sea environments. The limitations and future-work discussion identify hardware-in-the-loop and vessel experiments, sensitivity analysis, and more detailed actuator, sensor, communication, and computational constraints as necessary next steps.

Limitations. The present study is limited to theoretical analysis and numerical simulations using a single vessel heading-dynamics model. No hardware-in-the-loop, model-scale, or full-scale experimental validation was performed. The reported results therefore provide evidence of the controller behavior under the specified simulation conditions, rather than establishing its applicability across different vessel types, operating speeds, loading conditions, or sea environments. The parameter configuration was not subjected to a comprehensive sensitivity analysis, and the effects of broader variations in the controller gains, observer gain, learning parameters, network initialization, and triggering settings remain to be investigated. Although a rudder-angle constraint is included in the simulations, the effects of steering-gear rate limits, detailed actuator dynamics, sensor noise, communication delays, and real-time computational limitations are not explicitly validated. Furthermore, the analytical claims depend on the stated approximation, excitation, and fault assumptions. The present results do not establish comparable performance outside these assumptions, including complete actuator failure or fault variations beyond the assumed regularity and available control authority.

5. Conclusions

This paper developed an event-based robust course-tracking scheme for MSVs subject to uncertain nonlinear dynamics, environmental disturbances, and rudder faults. The internal uncertainty, external disturbance, and actuator-fault effects were collected into a lumped term and reconstructed by an observer. A tracking-oriented performance index and a single-critic RL structure were then used to generate the learning-based compensation input without introducing a separate actor network. In addition, a dynamic event-triggered mechanism was placed between the controller and rudder actuator so that the calculated command was transmitted only when the triggering condition was satisfied. Under the stated boundedness, approximation, excitation, and control-authority assumptions, the Lyapunov analysis established uniform ultimate boundedness of the closed-loop signals, and the derived positive lower bound on the inter-event time excluded Zeno behavior.

Future work will address the present limitations through systematic sensitivity analysis and parameter-selection studies; hardware-in-the-loop, model-scale, and full-scale experiments involving different vessel types, speeds, loading conditions, and sea environments; and the inclusion of steering-gear rate limits, detailed actuator dynamics, sensor noise, communication delays, and real-time computational constraints. Further research will also investigate less restrictive approximation and excitation conditions, fault scenarios approaching complete loss of actuator authority using suitable reconfiguration or redundancy, and direct experimental evaluation of energy consumption and mechanical wear.