ESO-Enhanced Actor–Critic Reinforcement Learning-Optimised Trajectory Tracking Control for 3-DOF Marine Vessels

Abstract

This paper develops an extended-state-observer (ESO)-enhanced actor–critic reinforcement learning (RL) scheme for the trajectory tracking control of 3-DOF marine vessels subject to uncertain hydrodynamics and environmental disturbances. A coordinate-consistent error construction is provided to obtain an exact strict-feedback second-order uncertain template. On this basis, an Hamilton–Jacobi–Bellman (HJB)-inspired optimised control structure is implemented: the critic approximates the optimal value-gradient and the actor generates the optimised control law. A key simplification is employed: rather than minimising the squared Bellman residual via complex gradients, we introduce an HJB-inspired actor–critic consistency regularisation through a weight-matching coupling. This yields computationally light online update laws and enables transparent Lyapunov-based stability analysis while not claiming exact HJB satisfaction or policy optimality. The ESO estimates lumped uncertainty and provides feedforward compensation, so the RL module learns only the observer residual. A composite Lyapunov analysis establishes the semi-global uniform ultimate boundedness of tracking errors and boundedness of all observer signals. Practical implementation with thruster allocation, explicit wind–wave–current disturbance shaping filters, and a theory-aligned ablation protocol are provided for reproducibility.

1. Introduction

The accurate trajectory tracking of marine surface vessels is a foundational capability for autonomous navigation, offshore operations, and safety-critical station-keeping tasks [1,2]. In practical deployments, tracking performance is challenged by a combination of (i) unmodelled and time-varying hydrodynamic effects, (ii) stochastic environmental loads induced by wind, waves, and currents, and (iii) strict actuation constraints arising from limited thruster authority and allocation feasibility. These factors can jointly induce large tracking errors, severe input saturation, and even qualitative trajectory distortion when the control law and the physical actuation model are not formulated in a coordinate-consistent manner. Thus, purely model-based designs may suffer from performance degradation when the nominal model is inaccurate, whereas aggressive robust designs may achieve stability at the expense of excessive control effort.

From a modelling perspective, the planar 3-degree-of-freedom (3-DOF) vessel dynamics constitute a coupled multi-input multi-output (MIMO) nonlinear system in which the kinematics are naturally expressed in the inertial frame while the generalised forces and moment are applied in the body-fixed frame. This inherent frame mismatch complicates both controller synthesis and theoretical analysis [3]. A common approach is to design tracking controllers via backstepping or strict-feedback transformations, which can provide systematic stability proofs when the system is cast into a suitable cascade form. However, when environmental disturbances and unmodeled dynamics are significant, purely model-based designs often become conservative to preserve robustness, or require increasingly complex adaptation laws [4]. Moreover, the presence of actuator bounds and allocation constraints can invalidate unconstrained stability arguments, since saturation introduces additional nonlinearities and may render a reference trajectory infeasible [5]. A prominent model-light route to robustness is active disturbance rejection control (ADRC), whose core component is the extended state observer (ESO) that estimates the total disturbance—including unmodeled dynamics and external perturbations—in real time [6,7,8]. Systematic tuning principles based on scaling and bandwidth parameterisation further popularised ADRC in engineering practice by explicitly linking observer and controller bandwidth to disturbance rejection and noise sensitivity [9,10]. In marine robotics, ESO-type disturbance rejection has been adopted for USV trajectory tracking to mitigate ocean environmental loads and parameter uncertainty, and recent studies have demonstrated its practicality in simulation and field experiments [11]. Nevertheless, ESO-based tracking designs are typically not optimal in an explicit sense; moreover, under hard input bounds, tuning purely for disturbance rejection may increase peak thrust demand and saturation rate, which can slow down recovery and distort the intended path.

Learning-based control methods, particularly reinforcement learning (RL), offer a complementary direction by improving performance through online policy optimisation [12,13]. Actor–critic structures are especially attractive for continuous-control problems, as they can approximate optimal policies and value functions without requiring an exact analytic solution of the Hamilton–Jacobi–Bellman (HJB) equation [14]. Nevertheless, vanilla actor–critic schemes may exhibit policy drift, sensitivity to stochastic disturbances, and degraded transient behavior when deployed on systems with severe uncertainty and hard actuation limits. In marine applications, these issues are amplified by persistent environmental loads and the coupling between translational and rotational motions [15]. Therefore, a key open problem is how to integrate learning with robust disturbance rejection in a way that (a) preserves coordinate consistency between theory and implementation, (b) remains stable under wind–wave–current disturbances, and (c) is feasible under realistic actuator constraints. RL—particularly, adaptive dynamic programming (ADP) and actor–critic structures—offers a complementary route by approximating the value function and policy to circumvent the analytic solution of the HJB equation. For continuous-time nonlinear systems, actor–critic learning connects directly to optimal control through policy iteration, temporal-difference learning, and Hamiltonian residual minimisation, enabling online performance improvement without requiring an explicit solution of the HJB partial differential equation [16]. In the marine domain, deep RL has been explored for path-following and trajectory tracking under uncertainty [17], and very recent work has started to incorporate input saturation characteristics into RL-based USV optimal tracking formulations [18]. However, pure RL controllers can be sensitive to unmodeled dynamics and external disturbances, and stability guarantees often become delicate when exploration, approximation errors, and actuator constraints coexist. Recent studies in other networked and cyber-physical domains also indicate a growing trend toward hybrid frameworks that combine structured optimisation components with learning-based adaptation for dynamic multi-objective decision-making [19]. This motivates a principled integration of disturbance observers with RL so that learning takes place around a robust baseline and focuses on optimal refinement rather than disturbance cancellation.

To highlight the motivation, we briefly compare this work with representative related approaches. Model-based robust tracking controllers provide stability guarantees but may become conservative under strong uncertainty and persistent disturbances, especially when actuator constraints are active. ESO- or ADRC-based designs offer practical disturbance rejection by estimating lumped uncertainties, but they do not explicitly address learning-oriented performance refinement. In contrast, many DRL frameworks in other domains such as multi-agent DRL and multi-DNN DRL for MEC or SD-WAN optimisation are primarily designed for slot-based decision problems, where DRL serves as the main decision engine for resource allocation and does not aim to provide Lyapunov-style closed-loop stability guarantees for continuous-time nonlinear dynamics. The present manuscript targets safety-critical continuous-time marine motion control and adopts a hybrid decomposition philosophy, while the ESO-based controller provides the stabilising robust baseline and the actor–critic component is incorporated as a Lyapunov-guided consistency regularisation and secondary online refinement mechanism. This comparison clarifies the research gap and motivates the proposed ESO-enhanced actor–critic framework. Motivated by these challenges, this paper develops a unified control framework that couples an ESO with an actor–critic RL module for the constrained trajectory tracking of 3-DOF vessels. The ESO is used to estimate and compensate lumped uncertainties, including wind–wave–current loads and unmodeled dynamics, thereby providing a robust baseline that stabilises the learning process. On top of this robustification, an actor–critic component refines the control policy toward improved performance such as reduced tracking error and control energy. To strengthen the mathematical consistency of learning, we introduce an HJB-consistency mechanism based on a computable residual metric and a weight-matching coupling between the actor and critic updates. This coupling aligns the learning dynamics with the optimality structure and mitigates uncontrolled weight growth and oscillatory behavior. Importantly, the overall design is formulated in a coordinate-consistent strict-feedback tracking structure, explicitly mapping inertial-frame tracking errors to physically applied body-fixed forces and moment, and incorporating actuator saturation and optional thruster allocation constraints to ensure feasibility.

The proposed framework is validated on a standard 3-DOF vessel benchmark under stochastic wind–wave–current disturbances. Beyond reporting tracking trajectories, we adopt reproducible evaluation metrics that capture both control performance and learning consistency, such as integrated tracking error, control energy, saturation rate, HJB residual, and actor–critic weight mismatch norms. In addition, systematic ablation studies are provided to quantify the complementary roles of robust disturbance estimation and learning-based optimisation.

The main methodological contributions of this paper are summarised as follows:

• We integrate ESO-based lumped-disturbance estimation with actor–critic policy improvement within a single closed-loop architecture suitable for 3-DOF vessel tracking. Compared with ESO-only and disturbance-sensitive RL-only designs, the proposed integration typically yields smaller tracking errors with lower control effort under environmental loads [20,21].
• Compared with standard TD-error-driven actor–critic schemes commonly used in the RL literature [22,23], the proposed weight-matching coupling is introduced as a Lyapunov-guided consistency regularisation within a control-oriented framework, which helps to suppress parameter drift and improves robustness under disturbances and actuator constraints.
• While most RL controllers operate as black boxes, this work guarantees SGUUB stability with a tracking error bound explicitly tied to the ESO residual. This provides a transparent mechanism for performance tuning linking observer bandwidth directly to control precision, which offers reliability that standard heuristic RL methods lack.

The remainder of the paper is organised as follows. Section 2 presents the vessel model, disturbance representation, and coordinate-consistent strict-feedback tracking error system. Section 3 develops the ESO design, the actor–critic learning law with HJB-consistency, and the overall constrained control synthesis. It provides the closed-loop stability analysis and establishes boundedness and ultimate tracking performance guarantees. Section 4 reports simulation results and ablation studies under wind–wave–current disturbances and actuator limits. Finally, Section 5 concludes the paper and outlines future research directions.

2. Vessel Model and Tracking Objective

Consider 3-DOF vessel dynamics [24]

$$\begin{array}{cc}\hfill \dot{\eta }& =R\left(\psi \right)\nu ,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill M\left(\nu \right)\dot{\nu }+C\left(\nu \right)\nu +D\left(\nu \right)\nu & =\tau +d\left(t\right),\hfill \end{array}$$

(2)

where $M\left(\nu \right)\succ 0$ is the inertia matrix, $C\left(\nu \right)$ is the coriolis term, $D\left(\nu \right)$ is the damping term, and $d\left(t\right)$ collects wind, wave, current, and unmodelled effects.

The control objective is designed as follows: given a smooth desired trajectory ${\eta }_d\left(t\right)$ with bounded derivatives, design a control input $\tau$ such that the tracking errors converge to a small neighbourhood of zero, while maintaining reasonable control effort and robustness against uncertainty and disturbances.

Define the earth-fixed tracking error

$${\eta }_1\triangleq \eta -{\eta }_d.$$

(3)

Define the earth-fixed strict-feedback velocity error

$${\eta }_2\triangleq R\left(\psi \right)\nu -{\dot{\eta }}_d.$$

(4)

Then, the strict-feedback first equation holds exactly:

$${\dot{\eta }}_1={\eta }_2.$$

(5)

Differentiate (4), then one has

$$\begin{array}{cc}\hfill {\dot{\eta }}_2& =\dot{R}\left(\psi \right)\nu +R\left(\psi \right)\dot{\nu }-{\ddot{\eta }}_d\hfill \\ \hfill & =\dot{R}\left(\psi \right)\nu +R\left(\psi \right)M^{-1}\left(\nu \right)\left(\tau +d-C\left(\nu \right)\nu -D\left(\nu \right)\nu \right)-{\ddot{\eta }}_d.\hfill \end{array}$$

(6)

Choose a constant nominal inertia matrix $M_0\succ 0$ and define a nominal feedforward term ${\tau }_{ff}\left(t\right)$:

$${\tau }_{ff}\left(t\right)\triangleq C_0\left({\nu }_d\right){\nu }_d+D_0\left({\nu }_d\right){\nu }_d+M_0{\dot{\nu }}_d,$$

(7)

where $(C_0,D_0)$ are nominal models. Implement

$$\tau =M_{0}u+{\tau }_{ff}\left(t\right),$$

(8)

with design input $u\in {\mathbb{R}}^3$. Substituting (8) into (6) yields

$${\dot{\eta }}_2=u+f(\chi ,t),$$

(9)

where $\chi$ collects measurable signals and the lumped uncertainty is

$$\begin{array}{cc}\hfill f(\chi ,t)\triangleq & \underset{\mathrm{input}\mathrm{channel}\mathrm{mismatch}}{\underbrace{\left(R\left(\psi \right)M^{-1}\left(\nu \right)M_0-I\right)u}}+\underset{\mathrm{rotation}-\mathrm{induced}\mathrm{coupling}}{\underbrace{\dot{R}\left(\psi \right)\nu }}\hfill \\ \hfill & +\underset{\mathrm{model}/\mathrm{ff}/\mathrm{env}\mathrm{mismatch}}{\underbrace{R\left(\psi \right)M^{-1}\left(\nu \right)\left({\tau }_{ff}-C\left(\nu \right)\nu -D\left(\nu \right)\nu +d\left(t\right)\right)-{\ddot{\eta }}_d}}.\hfill \end{array}$$

(10)

Remark 1. 

The lumped uncertainty $f(\chi ,t)$ in Equation (10) contains an input-coupled mismatch term that is linear in u. To clarify well-posedness, we decompose

$$f(\chi ,t)={\mathsf{\Delta }}_u\left(\chi \right)u+\overline{f}(\chi ,t),$$

(11)

where

$${\mathsf{\Delta }}_u\left(\chi \right)=R\left(\psi \right)M^{-1}\left(\nu \right)M_0-I,$$

(12)

and $\overline{f}(\chi ,t)$ collects the remaining rotation-induced coupling, model mismatch, and environmental disturbance terms. In implementation, the control input u is computed explicitly from measured and estimated signals and then mapped to the physical body frame with actuator saturation. Hence, the closed loop is a well-defined nonlinear ODE with saturation rather than an algebraic loop. We assume that $I+{\mathsf{\Delta }}_u\left(\chi \right)$ is uniformly nonsingular in the operating region, which is a mild small-gain-type condition ensuring the existence and uniqueness of solutions and preserving ESO convergence under the input-coupled uncertainty.

Thus, the second-order uncertain template is

$${\dot{\eta }}_1={\eta }_2,{\dot{\eta }}_2=u+f(\chi ,t).$$

(13)

Remark 2. 

The strict-feedback template in (13) is introduced for synthesis on actor–critic learning and observer-based compensation. In implementation, the virtual input u is mapped back to the physical body-frame force and moment through the kinematics and thruster allocation, so actuator constraints can be enforced without modifying the learning law.

Assumption 1. 

The lumped disturbance $d\left(t\right)$ is bounded and has a bounded derivative, i.e., $\parallel d\left(t\right)\parallel \le \overline{d}$ and $\parallel \dot{d}\left(t\right)\parallel \le {\overline{d}}_1$.

Assumption 2. 

The reference trajectory ${\eta }_d\left(t\right)$ is twice continuously differentiable and bounded, together with its derivatives, i.e., ${\eta }_d\left(t\right)$, ${\dot{\eta }}_d\left(t\right)$, and ${\ddot{\eta }}_d\left(t\right)$ are bounded.

Assumption 3. 

The regressor signal $S\left({\eta }_e\right)$ satisfies a finite excitation condition over a sliding window. There exist $T_w>0$ and $\gamma >0$ such that

$${\int }_{t-T_w}^{t}S\left({\eta }_e\right)S^{\top }\left({\eta }_e\right)d\tau ⪰\gamma I\forall t\ge T_w.$$

This assumption is used to strengthen mismatch convergence statements such as ${\widehat{W}}_a-{\widehat{W}}_c\to 0$.

Define the infinite-horizon quadratic performance index

$$J\left({\eta }_e\left(0\right)\right)={\int }_0^{\infty }\left({\eta }_e^T\left(\sigma \right){\eta }_e\left(\sigma \right)+u^T\left(\sigma \right)u\left(\sigma \right)\right)d\sigma ,$$

(14)

where ${\eta }_e={[{\eta }_1^T,{\eta }_2^T]}^T\in {\mathbb{R}}^6$.

The quadratic form in (14) reflects a standard tracking-to-regulation conversion: the stacked error state ${\eta }_e$ is penalised to enforce accurate tracking, while u is penalised to avoid excessive virtual control action. This optimal-control viewpoint provides a principled way to couple learning with a stabilising baseline in the presence of persistent environmental disturbances.

Let $J^{*}\left({\eta }_e\right)$ be the optimal value function. The Hamiltonian is

$$H({\eta }_e,u,J^{*})\triangleq {\eta }_e^T{\eta }_e+u^{T}u+{\left({\displaystyle \frac{d{J}^{*}}{d{\eta }_1}}\right)}^T{\dot{\eta }}_1+{\left({\displaystyle \frac{d{J}^{*}}{d{\eta }_2}}\right)}^T{\dot{\eta }}_2.$$

(15)

At the optimum, the HJB equation implies $H({\eta }_e,u,J^{*})=0$ and the optimal input satisfies the stationarity condition with respect to u. Since $J^{*}$ is unknown for the vessel tracking problem with uncertainty, we approximate it via a critic and compute an approximately optimal policy via actor.

Using (13), the stationary condition $\partial H/\partial u=0$ yields

$$u^{*}=-{\displaystyle \frac{1}{2}}{\displaystyle \frac{d{J}^{*}}{d{\eta }_2}}.$$

(16)

Treat $f(\chi ,t)$ in (13) as an extended state:

$$z_1={\eta }_2,z_2=f.$$

Then, ${\dot{z}}_1=u+z_2$, ${\dot{z}}_2=\dot{f}$. A second-order ESO is chosen as

$$\begin{array}{cc}\hfill {\dot{\widehat{z}}}_1& =u+{\widehat{z}}_2+{\beta }_1(z_1-{\widehat{z}}_1),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\dot{\widehat{z}}}_2& ={\beta }_2(z_1-{\widehat{z}}_1),\hfill \end{array}$$

(18)

with parameters ${\beta }_1,{\beta }_2>0$.

Use the compensated input

$$u={\widehat{u}}^{*}\left({\eta }_e\right)-\widehat{f},\widehat{f}\triangleq {\widehat{z}}_2.$$

(19)

Then, one obtains

$${\dot{\eta }}_1={\eta }_2,{\dot{\eta }}_2={\widehat{u}}^{*}\left({\eta }_e\right)+\mathsf{\Delta }f,\mathsf{\Delta }f\triangleq f-\widehat{f}.$$

(20)

Let $S\left({\eta }_e\right)\in {\mathbb{R}}^m$ be a bounded basis-function vector. Define critic and actor weights

$${\widehat{W}}_c\left(t\right)\in {\mathbb{R}}^{m\times 3},{\widehat{W}}_a\left(t\right)\in {\mathbb{R}}^{m\times 3},$$

so that ${\widehat{W}}_c^{T}S\left({\eta }_e\right)\in {\mathbb{R}}^3$ and ${\widehat{W}}_a^{T}S\left({\eta }_e\right)\in {\mathbb{R}}^3$.

Choose the critic approximation of the value-gradient

$${\displaystyle \frac{d{\widehat{J}}^{*}}{d{\eta }_2}}=2{\mathsf{\Gamma }}_1{\eta }_1+2{\mathsf{\Gamma }}_2{\eta }_2+{\widehat{W}}_c^{T}S\left({\eta }_e\right),$$

(21)

where ${\mathsf{\Gamma }}_1,{\mathsf{\Gamma }}_2\in {\mathbb{R}}^{3\times 3}$ are diagonal positive definite matrices.

Implement optimised nominal input

$${\widehat{u}}^{*}\left(\eta \right)=-{\mathsf{\Gamma }}_1{\eta }_1-{\mathsf{\Gamma }}_2{\eta }_2-{\displaystyle \frac{1}{2}}{\widehat{W}}_a^{T}S\left({\eta }_e\right).$$

(22)

Use online updates

$$\begin{array}{cc}\hfill {\dot{\widehat{W}}}_c& =-k_{c}S\left({\eta }_e\right)S^T\left({\eta }_e\right){\widehat{W}}_c,\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\dot{\widehat{W}}}_a& =-S\left({\eta }_e\right)S^T\left({\eta }_e\right)\left(k_a({\widehat{W}}_a-{\widehat{W}}_c)+k_c{\widehat{W}}_c\right),\hfill \end{array}$$

(24)

with design parameter $k_a>0,k_c>0$.

Remark 3. 

The update laws in Equations (23) and (24) are designed as a Lyapunov-guided actor–critic adaptation mechanism for the proposed ESO-based robust-control framework, rather than a standard TD Bellman-residual-gradient deep reinforcement learning update. This control-oriented design is adopted to maintain closed-loop boundedness and stable parameter adaptation under disturbances and actuator constraints. In this context, the actor–critic coupling term is used as a consistency regularisation term, instead of being interpreted as a standalone Bellman-optimal learning rule.

Using the HJB equation and its approximation, we define an optimality-related residual as

$$\epsilon \left(t\right)\triangleq H({\eta }_e,u,{\widehat{J}}^{*}).$$

(25)

In the present control-oriented design, this residual is introduced as a diagnostic indicator to interpret the subsequent stationarity and weight-matching construction under the adopted approximation. Since Equations (21) and (22) are designed as Lyapunov-guided consistency regularisation rather than TD-error-driven learning, we do not claim that $\epsilon \left(t\right)$ is explicitly minimised or driven to zero by the update law.

Impose

$${\displaystyle \frac{\partial H({\eta }_e,u,{\widehat{J}}^{*})}{\partial {\widehat{W}}_a}}={\displaystyle \frac{1}{2}}S\left({\eta }_e\right)S^T\left({\eta }_e\right)\left({\widehat{W}}_a-{\widehat{W}}_c\right)=\mathbf{0}.$$

(26)

Remark 4. 

The stationarity condition $\partial H/\partial {\widehat{W}}_a=\mathbf{0}$ is imposed as a tractable surrogate consistency condition for the parameterised policy with respect to the approximate Hamiltonian constructed using ${\widehat{J}}^{*}$. It provides a convenient direction to promote actor critic parameter consistency and facilitate Lyapunov analysis. It should be emphasised that this stationarity condition for the approximate Hamiltonian does not imply the pointwise satisfaction of the true HJB equation $H({\eta }_e,u^{*},J^{*})=0$ nor guarantee global optimality. In the present design, the coupled actor–critic updates in Equations (21) and (22) are adopted for Lyapunov-compatible online adaptation within the ESO-based robust control framework.

3. Stability Analysis

Assumption 4. 

The basis functions $S\left({\eta }_e\right)$ used in the actor–critic parameterisation are generated by Gaussian RBFs with fixed centers and widths; hence, $S\left({\eta }_e\right)$ is uniformly bounded in the operating region and there exists a constant $\overline{S}$ such that $|S\left({\eta }_e\right)|\le \overline{S}$ for all η in this region. Moreover, the actor and critic weights are assumed to evolve in compact sets and remain bounded, that is, there exist constants ${\overline{W}}_a$ and ${\overline{W}}_c$ such that $|{\widehat{W}}_a\left(t\right)|\le {\overline{W}}_a$ and $|{\widehat{W}}_c\left(t\right)|\le {\overline{W}}_c$ for all t. This boundedness is consistent with the coupled update structure and can also be enforced by standard projection-based constrained adaptation when needed.

This section establishes the boundedness of the overall ESO–actor–critic closed loop. The analysis proceeds in three steps: (i) show that the ESO yields a uniformly ultimately bounded estimation (UUB) error for the lumped uncertainty, (ii) show that the critic and actor weight dynamics remain bounded under the chosen update laws, and (iii) combine these properties into a composite Lyapunov argument that produces an explicit ultimate tracking-error bound. Denote errors $e_1=z_1-{\widehat{z}}_1$, $e_2=z_2-{\widehat{z}}_2=\mathsf{\Delta }f$. Then,

$$\begin{array}{cc}\hfill {\dot{e}}_1& =e_2-{\beta }_1{e}_1,\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\dot{e}}_2& =\dot{f}-{\beta }_2{e}_1.\hfill \end{array}$$

(28)

Lemma 1. 

Under Assumptions 1, 2 and 4 and under bounded closed-loop signals, there exists a constant ${\overline{f}}_1$ such that $|\dot{f}(\chi ,t)|\le {\overline{f}}_1$ for all t.

This follows because $f(\chi ,t)$ depends on bounded kinematic and dynamic states, bounded disturbances, and bounded control input, and its time derivative depends on bounded derivatives of these signals. In particular, the control input is bounded due to actuator saturation and the basis functions have bounded values in the operating region; hence, all terms contributing to $\dot{f}$ remain bounded, which yields the stated bound.

Let Lyapunov function candidate

$$V_{eso}={\displaystyle \frac{1}{2}}\parallel e_1{\parallel }^2+{\displaystyle \frac{1}{2{\beta }_2}}{\parallel e_2\parallel }^2.$$

(29)

Differentiating gives

$$\begin{array}{cc}\hfill {\dot{V}}_{eso}& =-{\beta }_1\parallel e_1{\parallel }^2+{\displaystyle \frac{1}{{\beta }_2}}{e}_2^T\dot{f}\le -{\beta }_1\parallel e_1{\parallel }^2+{\displaystyle \frac{1}{{\beta }_2}}\parallel e_2\parallel {\overline{f}}_1\hfill \\ \hfill & \le -{\beta }_1\parallel e_1{\parallel }^2+{\displaystyle \frac{{ϵ}_f}{2{\beta }_2}}{\parallel e_2\parallel }^2+{\displaystyle \frac{1}{2{ϵ}_f{\beta }_2}}{\overline{f}}_1^2,\hfill \end{array}$$

(30)

for any ${ϵ}_f>0$. Hence, $(e_1,e_2)$ are UUB. In particular, there exist finite constants ${\overline{\mathsf{\Delta }}}_f$ and $T_0$ such that $\parallel \mathsf{\Delta }f\left(t\right)\parallel = \parallel e_2\left(t\right)\parallel \le {\overline{\mathsf{\Delta }}}_f$ for all $t\ge T_0$.

Next, consider the parameter estimation dynamics induced by the coupled actor–critic updates. By standard arguments for gradient-type updates with a shared regressor $S\left({\eta }_e\right)$, the actor weight error $\tilde{W}$ satisfies $\dot{\tilde{W}}=-k_{a}S{S}^T\tilde{W}$ in the idealised approximation setting, which immediately implies a non-increasing quadratic storage function. Define $\tilde{W}\triangleq {\widehat{W}}_a-{\widehat{W}}_c$ and

$$P\left(t\right)=\mathrm{Tr}\left({\tilde{W}}^T\tilde{W}\right)={\parallel \tilde{W}\parallel }_F^2.$$

(31)

Specifically, we have

$$\dot{P}=-2k_a{\parallel S^T\tilde{W}\parallel }_F^2\le 0.$$

(32)

Also,

$${\displaystyle \frac{d}{dt}}\mathrm{Tr}\left({\widehat{W}}_c^T{\widehat{W}}_c\right)=-2k_c\mathrm{Tr}\left({\widehat{W}}_c^{T}S{S}^T{\widehat{W}}_c\right)\le 0,$$

so ${\widehat{W}}_c$ is bounded and non-increasing in norm. Therefore, ${\widehat{W}}_a=\tilde{W}+{\widehat{W}}_c$ is bounded:

$$\parallel {\widehat{W}}_a\left(t\right)\parallel \le {\overline{W}}_a<\infty ,\forall t\ge 0.$$

(33)

Lemma 2. 

Define $\tilde{W}\triangleq {\widehat{W}}_a-{\widehat{W}}_c$ and $P\left(t\right)=\mathrm{Tr}\left({\tilde{W}}^{\top }\tilde{W}\right)={\parallel \tilde{W}\parallel }_F^2.$ Under the coupled updates (21) and (22), the mismatch dynamics satisfies $\dot{\tilde{W}}=-k_{a}S\left({\eta }_e\right)S^{\top }\left({\eta }_e\right)\tilde{W}$, and hence $\dot{P}\left(t\right)$ satisfies (32). Therefore, $P\left(t\right)$ is non-increasing and $\tilde{W}\left(t\right)$ is bounded for all $t\ge 0$.

Remark 5. 

Lemma 1 shows that Equation (24), together with (21) and (22), enforces a monotonic decrease in the actor–critic mismatch energy P(t). This promotes parameter consistency under the adopted approximation and serves as a Lyapunov-guided consistency regularisation. However, ${\widehat{W}}_a\approx {\widehat{W}}_c$ alone does not imply a pointwise satisfaction of the HJB equation $H(\eta ,u^{*},J^{*})=0$ nor exact policy optimality. Hence, the stationarity and weight-matching construction are interpreted as an HJB-inspired surrogate rather than a proof of exact HJB optimality.

If the finite excitation condition in Assumption 3 holds, then Proposition 1 further implies $W_a-W_c$ converges to zero; otherwise, only boundedness and mismatch energy dissipation are guaranteed.

Proposition 1. 

Assume that there exist $T_w>0$ and $\gamma >0$ such that

$${\int }_{t-T_w}^{t}S\left(\eta \left(\tau \right)\right)S^{\top }\left(\eta \left(\tau \right)\right)d\tau ⪰\gamma I\forall t\ge T_w$$

Then, ${lim}_{t\to \infty }{\parallel {\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\parallel }_F=0$ and hence ${\widehat{W}}_a\left(t\right)\to {\widehat{W}}_c\left(t\right)$.

Choose a scalar $\alpha \in (0,1)$ and define

$$V_e^{★}\triangleq {\displaystyle \frac{1}{2}}{\parallel {\eta }_1\parallel }^2+\alpha {\eta }_1^T{\eta }_2+{\displaystyle \frac{1}{2}}{\parallel {\eta }_2\parallel }^2.$$

(34)

Using $|{\eta }_1^T{\eta }_2| \le {\displaystyle \frac{1}{2}}({\parallel {\eta }_1\parallel }^2+{\parallel {\eta }_2\parallel }^2)$, we obtain the global quadratic bounds

$${\displaystyle \frac{1-\alpha }{2}}\left({\parallel {\eta }_1\parallel }^2+{\parallel {\eta }_2\parallel }^2\right)\le V_e^{★}\le {\displaystyle \frac{1+\alpha }{2}}\left({\parallel {\eta }_1\parallel }^2+{\parallel {\eta }_2\parallel }^2\right).$$

(35)

Now, define the full Lyapunov candidate

$$V\triangleq V_e^{★}+{\displaystyle \frac{1}{2k_a}}P+V_{eso}.$$

(36)

Remark 6. 

The coupling parameter $\alpha \in (0,1)$ is introduced to shape the quadratic bounds of $V_e^{★}$ and to enlarge the feasibility margin when the ESO and learning transients coexist. In practice, a smaller α yields a more conservative but more robust bound, while a larger α can improve transient response at the expense of reduced robustness margin.

It is positive definite and radially unbounded in $({\eta }_1,{\eta }_2,\tilde{W},e_1,e_2)$ because $V_e^{★}$ is quadratically bounded by (35).

Differentiate (34), which yields

$$\begin{array}{cc}\hfill {\dot{V}}_e^{★}& ={\eta }_1^T{\dot{\eta }}_1+\alpha ({\dot{\eta }}_1^T{\eta }_2+{\eta }_1^T{\dot{\eta }}_2)+{\eta }_2^T{\dot{\eta }}_2\hfill \\ \hfill & ={\eta }_1^T{\eta }_2+\alpha ({\eta }_2^T{\eta }_2+{\eta }_1^T{\dot{\eta }}_2)+{\eta }_2^T{\dot{\eta }}_2\hfill \\ \hfill & ={\eta }_1^T{\eta }_2+\alpha {\parallel {\eta }_2\parallel }^2+{(\alpha {\eta }_1+{\eta }_2)}^T\left(-{\mathsf{\Gamma }}_1{\eta }_1-{\mathsf{\Gamma }}_2{\eta }_2-\frac{1}{2}{\widehat{W}}_a^{T}S+\mathsf{\Delta }f\right)\hfill \\ \hfill & =-\alpha {\eta }_1^T{\mathsf{\Gamma }}_1{\eta }_1-{\eta }_2^T{\mathsf{\Gamma }}_2{\eta }_2-{\eta }_2^T{\mathsf{\Gamma }}_1{\eta }_1-\alpha {\eta }_1^T{\mathsf{\Gamma }}_2{\eta }_2+\alpha {\parallel {\eta }_2\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}{(\alpha {\eta }_1+{\eta }_2)}^T{\widehat{W}}_a^{T}S+{(\alpha {\eta }_1+{\eta }_2)}^T\mathsf{\Delta }f.\hfill \end{array}$$

(37)

Use eigenvalue bounds:

$${\eta }_1^T{\mathsf{\Gamma }}_1{\eta }_1\ge {\lambda }_{min}\left({\mathsf{\Gamma }}_1\right)\parallel {\eta }_1{\parallel }^2,{\eta }_2^T{\mathsf{\Gamma }}_2{\eta }_2\ge {\lambda }_{min}\left({\mathsf{\Gamma }}_2\right){\parallel {\eta }_2\parallel }^2.$$

For cross terms, apply Young inequalities with tunable scalars ${\epsilon }_{12},{\epsilon }_{21}>0$:  

$$\begin{array}{cc}\hfill |{\eta }_2^T{\mathsf{\Gamma }}_1{\eta }_1|& \le \parallel {\mathsf{\Gamma }}_1\parallel \parallel {\eta }_2\parallel \parallel {\eta }_1\parallel \le {\displaystyle \frac{{\epsilon }_{21}}{2}}\parallel {\eta }_2{\parallel }^2+{\displaystyle \frac{\parallel {\mathsf{\Gamma }}_1{\parallel }^2}{2{\epsilon }_{21}}}{\parallel {\eta }_1\parallel }^2,\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill |\alpha {\eta }_1^T{\mathsf{\Gamma }}_2{\eta }_2|& \le \alpha \parallel {\mathsf{\Gamma }}_2\parallel \parallel {\eta }_1\parallel \parallel {\eta }_2\parallel \le {\displaystyle \frac{{\epsilon }_{12}}{2}}\parallel {\eta }_2{\parallel }^2+{\displaystyle \frac{{\alpha }^2{\parallel {\mathsf{\Gamma }}_2\parallel }^2}{2{\epsilon }_{12}}}{\parallel {\eta }_1\parallel }^2.\hfill \end{array}$$

(39)

Substituting (38) and (39) into (37) yields

$$\begin{array}{cc}\hfill {\dot{V}}_e^{★}& \le -\left(\alpha {\lambda }_{min}\left({\mathsf{\Gamma }}_1\right)-{\displaystyle \frac{\parallel {\mathsf{\Gamma }}_1{\parallel }^2}{2{\epsilon }_{21}}}-{\displaystyle \frac{{\alpha }^2{\parallel {\mathsf{\Gamma }}_2\parallel }^2}{2{\epsilon }_{12}}}\right){\parallel {\eta }_1\parallel }^2\hfill \\ \hfill & -\left({\lambda }_{min}\left({\mathsf{\Gamma }}_2\right)-\alpha -{\displaystyle \frac{{\epsilon }_{21}}{2}}-{\displaystyle \frac{{\epsilon }_{12}}{2}}\right){\parallel {\eta }_2\parallel }^2-{\displaystyle \frac{1}{2}}{(\alpha {\eta }_1+{\eta }_2)}^T{\widehat{W}}_a^{T}S+{(\alpha {\eta }_1+{\eta }_2)}^T\mathsf{\Delta }f.\hfill \end{array}$$

(40)

Define

$$c_1\triangleq \alpha {\lambda }_{min}\left({\mathsf{\Gamma }}_1\right)-{\displaystyle \frac{\parallel {\mathsf{\Gamma }}_1{\parallel }^2}{2{\epsilon }_{21}}}-{\displaystyle \frac{{\alpha }^2{\parallel {\mathsf{\Gamma }}_2\parallel }^2}{2{\epsilon }_{12}}},c_2\triangleq {\lambda }_{min}\left({\mathsf{\Gamma }}_2\right)-\alpha -{\displaystyle \frac{{\epsilon }_{21}}{2}}-{\displaystyle \frac{{\epsilon }_{12}}{2}}.$$

(41)

Choose ${\mathsf{\Gamma }}_1,{\mathsf{\Gamma }}_2$ and ${\epsilon }_{12},{\epsilon }_{21}$ such that $c_1>0$ and $c_2>0$.

Using Assumption 4 and (33) for any epsilon greater than zero, it follows that, for any ${ϵ}_a>0$,

$$\begin{array}{cc}\hfill \left|{\displaystyle \frac{1}{2}}{(\alpha {\eta }_1+{\eta }_2)}^T{\widehat{W}}_a^{T}S\right|& \le {\displaystyle \frac{1}{2}}\parallel \alpha {\eta }_1+{\eta }_2\parallel \parallel {\widehat{W}}_a\parallel \parallel S\parallel \le {\displaystyle \frac{1}{2}}(\alpha \parallel {\eta }_1\parallel +\parallel {\eta }_2\parallel ){\overline{W}}_a\overline{S}\hfill \\ \hfill & \le {\displaystyle \frac{{ϵ}_a}{2}}{\parallel {\eta }_2\parallel }^2+{\displaystyle \frac{{ϵ}_a}{2}}{\alpha }^2{\parallel {\eta }_1\parallel }^2+{\displaystyle \frac{{\overline{W}}_a^2{\overline{S}}^2}{2{ϵ}_a}}.\hfill \end{array}$$

(42)

For any ${ϵ}_d>0$, one obtains

$$\begin{array}{cc}\hfill {(\alpha {\eta }_1+{\eta }_2)}^T\mathsf{\Delta }f& \le \parallel \alpha {\eta }_1+{\eta }_2\parallel \parallel \mathsf{\Delta }f\parallel \le (\alpha \parallel {\eta }_1\parallel +\parallel {\eta }_2\parallel )\parallel \mathsf{\Delta }f\parallel \hfill \\ \hfill & \le {\displaystyle \frac{{ϵ}_d}{2}}\parallel {\eta }_2{\parallel }^2+{\displaystyle \frac{1}{2{ϵ}_d}}{\parallel \mathsf{\Delta }f\parallel }^2+{\displaystyle \frac{{ϵ}_d}{2}}{\alpha }^2\parallel {\eta }_1{\parallel }^2+{\displaystyle \frac{1}{2{ϵ}_d}}{\parallel \mathsf{\Delta }f\parallel }^2\hfill \\ \hfill & ={\displaystyle \frac{{ϵ}_d}{2}}\parallel {\eta }_2{\parallel }^2+{\displaystyle \frac{{ϵ}_d}{2}}{\alpha }^2\parallel {\eta }_1{\parallel }^2+{\displaystyle \frac{1}{{ϵ}_d}}{\parallel \mathsf{\Delta }f\parallel }^2.\hfill \end{array}$$

(43)

Substitute (42) and (43) into (40):

$$\begin{array}{cc}\hfill {\dot{V}}_e^{★}& \le -\left(c_1-{\displaystyle \frac{{ϵ}_a}{2}}{\alpha }^2-{\displaystyle \frac{{ϵ}_d}{2}}{\alpha }^2\right)\parallel {\eta }_1{\parallel }^2-\left(c_2-{\displaystyle \frac{{ϵ}_a}{2}}-{\displaystyle \frac{{ϵ}_d}{2}}\right){\parallel {\eta }_2\parallel }^2\hfill \\ \hfill & +\underset{\triangleq {\rho }_a}{\underbrace{{\displaystyle \frac{{\overline{W}}_a^2{\overline{S}}^2}{2{ϵ}_a}}}}+\underset{\triangleq {\rho }_d\left(t\right)}{\underbrace{{\displaystyle \frac{1}{{ϵ}_d}}{\parallel \mathsf{\Delta }f\parallel }^2}}.\hfill \end{array}$$

(44)

Define the tightened constants

$${\tilde{c}}_1\triangleq c_1-{\displaystyle \frac{{\alpha }^2}{2}}({ϵ}_a+{ϵ}_d),{\tilde{c}}_2\triangleq c_2-{\displaystyle \frac{1}{2}}({ϵ}_a+{ϵ}_d).$$

(45)

Choose ${ϵ}_a,{ϵ}_d$ sufficiently small so that ${\tilde{c}}_1>0$ and ${\tilde{c}}_2>0$.

From (32), ${\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{1}{2k_a}}P\right)\le 0$. Combine (44) with (30), which yields

$$\begin{array}{cc}\hfill \dot{V}& \le -{\tilde{c}}_1\parallel {\eta }_1{\parallel }^2-{\tilde{c}}_2\parallel {\eta }_2{\parallel }^2+{\rho }_a+{\displaystyle \frac{1}{{ϵ}_d}}{\parallel \mathsf{\Delta }f\parallel }^2-{\beta }_1\parallel e_1{\parallel }^2+{\displaystyle \frac{{ϵ}_f}{2{\beta }_2}}{\parallel \mathsf{\Delta }f\parallel }^2+{\displaystyle \frac{1}{2{ϵ}_f{\beta }_2}}{\overline{f}}_1^2\hfill \\ \hfill & \le -{\tilde{c}}_1\parallel {\eta }_1{\parallel }^2-{\tilde{c}}_2\parallel {\eta }_2{\parallel }^2-{\beta }_1\parallel e_1{\parallel }^2+{\rho }_a+\left({\displaystyle \frac{1}{{ϵ}_d}}+{\displaystyle \frac{{ϵ}_f}{2{\beta }_2}}\right){\parallel \mathsf{\Delta }f\parallel }^2+{\displaystyle \frac{1}{2{ϵ}_f{\beta }_2}}{\overline{f}}_1^2.\hfill \end{array}$$

(46)

For $t\ge T_0$, $\parallel \mathsf{\Delta }f\left(t\right)\parallel \le {\overline{\mathsf{\Delta }}}_f$ from the ESO UUB property. Therefore, for all $t\ge T_0$,

$$\dot{V}\le -\lambda \left(\parallel {\eta }_1{\parallel }^2+\parallel {\eta }_2{\parallel }^2+{\parallel e_1\parallel }^2\right)+\rho ,$$

(47)

where one can choose

$$\lambda \triangleq min\{{\tilde{c}}_1,{\tilde{c}}_2,{\beta }_1\},$$

(48)

and an explicit ultimate-bound constant

$$\rho \triangleq {\rho }_a+\left({\displaystyle \frac{1}{{ϵ}_d}}+{\displaystyle \frac{{ϵ}_f}{2{\beta }_2}}\right){\overline{\mathsf{\Delta }}}_f^2+{\displaystyle \frac{1}{2{ϵ}_f{\beta }_2}}{\overline{f}}_1^2={\displaystyle \frac{{\overline{W}}_a^2{\overline{S}}^2}{2{ϵ}_a}}+\left({\displaystyle \frac{1}{{ϵ}_d}}+{\displaystyle \frac{{ϵ}_f}{2{\beta }_2}}\right){\overline{\mathsf{\Delta }}}_f^2+{\displaystyle \frac{1}{2{ϵ}_f{\beta }_2}}{\overline{f}}_1^2.$$

(49)

Theorem 1. 

Assume that Assumptions 1, 2, and 4 hold; the controller and observer learning gains are selected such that ${\tilde{c}}_1>0$ and ${\tilde{c}}_2>0$ in (45). Let λ and ρ be defined in (48) and (49), and let $\mu \triangleq {\displaystyle \frac{2\lambda }{1+\alpha }}$. Then, all closed-loop signals $({\eta }_1,{\eta }_2,{\widehat{W}}_a,{\widehat{W}}_c,e_1,e_2)$ remain bounded. Moreover, for all $t\ge T_0$, the composite Lyapunov function satisfies the comparison solution (51), and the tracking errors are semi-globally uniformly ultimately bounded with the explicit ultimate bound (52).

Proof. 

For $t\ge T_0$, the ESO property yields $\parallel \mathsf{\Delta }f\left(t\right)\parallel \le {\overline{\mathsf{\Delta }}}_f$, so the dissipation inequality (47) holds with the constants $\lambda$ and $\rho$ in (48) and (49). The remaining steps follow by standard comparison arguments, as detailed below.

Using (35), we have

$$V\ge V_e^{★}\ge {\displaystyle \frac{1-\alpha }{2}}(\parallel {\eta }_1{\parallel }^2+\parallel {\eta }_2{\parallel }^2).$$

Hence, (47) implies the standard comparison inequality

$$\dot{V}\le -\underset{\triangleq \mu }{\underbrace{{\displaystyle \frac{2\lambda }{1+\alpha }}}}V+\rho ,\forall t\ge T_0,$$

(50)

because $V\le {\displaystyle \frac{1+\alpha }{2}}(\parallel {\eta }_1{\parallel }^2+\parallel {\eta }_2{\parallel }^2)+{\displaystyle \frac{1}{2k_a}}P+V_{eso}$ and the negative term in (47) dominates the state part; $\mu$ can be conservatively selected as $\mu ={\displaystyle \frac{2\lambda }{1+\alpha }}$.

Therefore,

$$V\left(t\right)\le \left(V\left(T_0\right)-{\displaystyle \frac{\rho }{\mu }}\right)e^{-\mu (t-T_0)}+{\displaystyle \frac{\rho }{\mu }},\forall t\ge T_0,$$

(51)

and

$$\underset{t\to \infty }{lim\; sup}\left(\parallel {\eta }_1{\left(t\right)\parallel }^2+{\parallel {\eta }_2\left(t\right)\parallel }^2\right)\le {\displaystyle \frac{2}{1-\alpha }}\underset{t\to \infty }{lim\; sup}V\left(t\right)\le {\displaystyle \frac{2}{1-\alpha }}·{\displaystyle \frac{\rho }{\mu }}={\displaystyle \frac{(1+\alpha )}{(1-\alpha )}}·{\displaystyle \frac{\rho }{\lambda }}.$$

(52)

Hence, the closed-loop tracking errors are SGUUB, and all signals $({\eta }_1,{\eta }_2,{\widehat{W}}_a,{\widehat{W}}_c,e_1,e_2)$ remain bounded.    □

Remark 7. 

To avoid confusion with Lyapunov-drift-based single-slot optimisation methods, we emphasise that the Lyapunov function used in this paper is introduced only for the stability analysis of the closed-loop ESO actor–critic system. The long-term control objective is still defined by the infinite-horizon cost in Equation (14), and the actor–critic module performs HJB-inspired online policy refinement for this objective. The Lyapunov function is then used to establish boundedness and the SGUUB of the tracking, observer, and learning dynamics under disturbances and actuator constraints.

Remark 8. 

The proposed ESO learning framework is fundamentally different from generic deep reinforcement learning methods for MEC optimisation [25]. The methods use deep reinforcement learning as the main decision engine for slot-based resource allocation, whereas our method targets continuous-time nonlinear motion control, where ESO provides the stabilising robust baseline and learning serves as a secondary performance-refinement mechanism.

Implementation algorithm for ESO-enhanced actor–critic RL-optimised 3-DOF trajectory tracking is demonstrated below (Algorithm 1).

Algorithm 1. ESO-enhanced actor–critic RL-optimised trajectory tracking (3-DOF).

1: Initialise: choose ${\mathsf{\Gamma }}_1,{\mathsf{\Gamma }}_2,k_a,k_c$; choose ESO bandwidth ${\omega }_o$ and set ${\beta }_1=2{\omega }_o$, ${\beta }_2={\omega }_o^2$; initialise ${\widehat{W}}_a\left(0\right),{\widehat{W}}_c\left(0\right),{\widehat{z}}_1\left(0\right),{\widehat{z}}_2\left(0\right)$.   2: for each control step t do   3:     Measure $(\eta ,\nu )$ and compute references $({\eta }_d,{\dot{\eta }}_d,{\ddot{\eta }}_d)$ and $({\nu }_d,{\dot{\nu }}_d)$.   4:     Compute errors: ${\eta }_1=\eta -{\eta }_d$, ${\eta }_2=R\left(\psi \right)\nu -{\dot{\eta }}_d$.   5:     Set $z_1={\eta }_2$; integrate ESO (17) and (18) to obtain $\widehat{f}={\widehat{z}}_2$.   6:     Compute basis vector $S\left({\eta }_e\right)$.   7:     Actor output: ${\widehat{u}}^{*}=-{\mathsf{\Gamma }}_1{\eta }_1-{\mathsf{\Gamma }}_2{\eta }_2-{\displaystyle \frac{1}{2}}{\widehat{W}}_a^{T}S\left({\eta }_e\right)$.   8:     Apply compensated equivalent input: $u={\widehat{u}}^{*}-\widehat{f}$.   9:     Update critic: integrate ${\dot{\widehat{W}}}_c=-k_{c}S\left({\eta }_e\right)S^T\left({\eta }_e\right){\widehat{W}}_c$. 10:     Update actor: integrate ${\dot{\widehat{W}}}_a=-S\left({\eta }_e\right)S^T\left({\eta }_e\right)\left(k_a({\widehat{W}}_a-{\widehat{W}}_c)+k_c{\widehat{W}}_c\right)$. 11:     Compute $\tau =M_{0}u+{\tau }_{ff}\left(t\right)$ and allocate thrusters. 12: end for

4. Simulation Studies

This section provides a reproducible simulation protocol and a theory-aligned ablation study. The uncertainty and the environmental loads are generated continuously by wind–wave–current shaping filters, which is consistent with Assumption 1. To avoid a trivial initial condition, the vessel starts from an offset position that does not lie on the desired circle, and both the start and terminal points are explicitly marked in the trajectory plots. Vessel model parameters are set as Cybership II parameters. The simulations were executed on software stack Matlab R2025a version.

A circular reference trajectory is designed as

$$\begin{array}{cc}\hfill x_d\left(t\right)& =Rcos\left(\omega t\right),\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill y_d\left(t\right)& =Rsin\left(\omega t\right),\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill {\psi }_d\left(t\right)& =\mathrm{atan}2({\dot{y}}_d\left(t\right),{\dot{x}}_d\left(t\right)),\hfill \end{array}$$

(55)

with $R=1.5\mathrm{m}$ and $\omega =2\pi /70\mathrm{rad}/\mathrm{s}$.

Generate environmental disturbance model $d\left(t\right)={[d_x\left(t\right),d_y\left(t\right),d_{\psi }\left(t\right)]}^T$ by wind–wave–current filters:

$$d\left(t\right)=d_c\left(t\right)+d_w\left(t\right)+d_{wv}\left(t\right).$$

(56)

Steady current-induced bias is exploited as

$$d_c\left(t\right)=\left[\begin{array}{c}{K}_c{V}_c^{2}cos{\chi }_c\\ K_c{V}_c^{2}sin{\chi }_c\\ K_{c\psi }{V}_c^2\end{array}\right].$$

(57)

The wind gust low-frequency filter is chosen as

$${\dot{\xi }}_w\left(t\right)=-{\omega }_w{\xi }_w\left(t\right)+{\sigma }_w{\omega }_{w}w\left(t\right),w\left(t\right)\sim \mathcal{N}(0,I_3),$$

(58)

$$d_w\left(t\right)=\mathrm{diag}({\kappa }_{wx},{\kappa }_{wy},{\kappa }_{w\psi }){\xi }_w\left(t\right).$$

(59)

Wave-frequency load is employed as a second-order filter. For each $\mathcal{l}\in \{x,y,\psi \}$,

$${\ddot{\xi }}_{\mathcal{l}}\left(t\right)+2{\zeta }_{\mathcal{l}}{\omega }_{\mathcal{l}}{\dot{\xi }}_{\mathcal{l}}\left(t\right)+{\omega }_{\mathcal{l}}^2{\xi }_{\mathcal{l}}\left(t\right)={\sigma }_{\mathcal{l}}{\omega }_{\mathcal{l}}^2{w}_{\mathcal{l}}\left(t\right),w_{\mathcal{l}}\left(t\right)\sim \mathcal{N}(0,1),$$

(60)

$$d_{wv}\left(t\right)=\left[\begin{array}{c}{\xi }_x\left(t\right)\\ {\xi }_y\left(t\right)\\ {\xi }_{\psi }\left(t\right)\end{array}\right].$$

(61)

Disturbance filter parameters are shown in

Table 1. Disturbance filter parameters.

Parameter	Meaning	Value
$V_c$	current magnitude	$0.2$ m/s
${\chi }_c$	current direction	0 rad
$K_c,K_{c\psi }$	current-to-load gains	chosen to match bias
${\omega }_w$	wind gust bandwidth	$0.2$ rad/s
${\sigma }_w$	wind RMS scaling	$1.0$
${\kappa }_{wx},{\kappa }_{wy},{\kappa }_{w\psi }$	wind load scaling	$(50,80,30)$
${\omega }_x,{\omega }_y,{\omega }_{\psi }$	wave dominant freq	$(1.0,1.0,0.8)$ rad/s
${\zeta }_x,{\zeta }_y,{\zeta }_{\psi }$	wave damping ratios	$(0.2,0.2,0.25)$
${\sigma }_x,{\sigma }_y,{\sigma }_{\psi }$	wave intensity	$(30,40,20)$

.

All compared controllers share the same plant, the same reference trajectory, and the same actuator constraints. Thus, differences in performance are attributable to the control law rather than feasibility handling. We compare an ablation study and four methods: (i) nominal feedback (no ESO/RL), $u=-{\mathsf{\Gamma }}_1{\eta }_1-{\mathsf{\Gamma }}_2{\eta }_2$, (ii) ESO-only, $u=-{\mathsf{\Gamma }}_1{\eta }_1-{\mathsf{\Gamma }}_2{\eta }_2-\widehat{f}$, (iii) RL-only, $u={\widehat{u}}^{*}\left(\eta \right)$ with (23) and (24), and $\widehat{f}\equiv 0$, and (iv) proposed control design. The following performance metrics are reported over $t\in [0,T]$: (i) RMS position error ${\mathrm{RMS}}_p=\sqrt{{\displaystyle \frac{1}{T}}{\int }_0^T\left({(x-x_d)}^2+{(y-y_d)}^2\right)dt}$, (ii) RMS yaw error ${\mathrm{RMS}}_{\psi }=\sqrt{{\displaystyle \frac{1}{T}}{\int }_0^T{(\psi -{\psi }_d)}^{2}dt}$, (iii) control energy $E_{\tau }={\int }_0^T{\tau }^T\tau dt$. Controller settings are shown in

Table 2. Controller and observer parameters.

Parameter	Meaning	Value
${\mathsf{\Gamma }}_1$	feedback gain on ${\eta }_1$	$2I_3$
${\mathsf{\Gamma }}_2$	feedback gain on ${\eta }_2$	$3I_3$
$k_a$	actor learning gain	2.0
$k_c$	critic learning gain	1.2
m	number of basis functions	25
$S\left({\eta }_e\right)$	basis type	Gaussian RBF
${\omega }_o$	ESO bandwidth	8–15 rad/s
${\beta }_1,{\beta }_2$	ESO gains	${\beta }_1=2{\omega }_o,{\beta }_2={\omega }_o^2$

.

The ablation results in

Table 3. Ablation results under wind–wave–current disturbances.

Method	RMS Pos (m)	IAE Pos (m·s)	RMS Yaw (rad)
Nominal (No ESO/RL)	0.0393	0.9935	0.0290
ESO-only	0.0383	0.8807	0.0273
RL-only	0.0393	0.9929	0.0290
ESO+RL (Proposed)	0.0382	0.8687	0.0270

indicate that the dominant robustness improvement in this representative case is provided by the ESO-based disturbance estimation, while the actor–critic component mainly serves as a Lyapunov-guided consistency regularisation and secondary refinement mechanism within the proposed control-oriented framework. Therefore, the numerical margin between ESO only and ESO plus actor–critic may be modest under the reported disturbance realisation, and we avoid interpreting it as a statistically significant gain without repeated-run dispersion measures.

Figure 1 shows the 2-D trajectory-tracking performance of the circular reference. All controllers converge to the vicinity of the desired orbit after a short transient caused by the initial offset. However, the proposed method exhibits the tightest overlap with the reference curve and the smallest visible deviation along the orbit, indicating superior steady-state disturbance rejection under persistent environmental loads. In particular, compared with the ESO-only and RL-only baselines, the proposed controller reduces residual drift and maintains a smaller tracking tube around the reference circle. It shows the overall trajectory-tracking behavior under the same disturbance realisation. All methods converge toward the reference trajectory after the initial offset, while the proposed ESO plus actor–critic scheme exhibits a tighter overlap with the desired path in the steady phase. This visual trend is consistent with the intention of using ESO to handle the dominant disturbance rejection and using the actor–critic coupling to provide a stable refinement around the robust baseline. To highlight the steady tracking regime, the zoomed view in Figure 2 shows that the proposed method achieves the smallest residual error level and mitigates long-term drift more effectively under the same disturbance realisation and actuator constraints. The proposed controller achieves the lowest steady tracking error and the fastest decay after the initial transient. It reports the time evolution of the position tracking error magnitude. The transient part reflects how fast the controller recovers from the initial offset, whereas the steady part reflects the residual tracking tube under persistent disturbances. The proposed method yields a smoother and lower steady error envelope in this representative case, indicating improved disturbance rejection and reduced oscillations in the tracking response. The heading tracking performance is shown in Figure 3. The proposed method maintains a smaller and smoother $e_{\psi }\left(t\right)$ response, which contributes to the reduced lateral deviation on the circular path and prevents error accumulation caused by heading misalignment. Note that a small negative steady value of the yaw tracking error in Figure 3 indicates a slight signed offset rather than instability, and such a residual bias is consistent with the SGUUB property under persistent disturbances and actuator constraints. Figure 4 plots the three-channel commanded input $\tau ={[{\tau }_x,{\tau }_y,{\tau }_{\psi }]}^{\top }$ for the proposed controller. The input signals exhibit a larger transient effort to recover from the initial offset, followed by a bounded and smooth steady regime. Importantly, the improved tracking accuracy of the proposed method is not achieved by persistent saturation or excessively aggressive actuation; instead, it results from the complementary compensation structure that combines disturbance estimation and learning-based residual. The control inputs generated by the proposed controller. The signals show a larger but short-lived transient effort to compensate the initial offset, followed by bounded and relatively smooth steady behavior. This indicates that the improved tracking observed in Figure 1, Figure 2 and Figure 3 is not obtained by persistent aggressive actuation, but by the combined effect of ESO disturbance compensation and the consistency regularisation in the actor–critic adaptation. To further quantify the visual result, the position error norm and the RMS index are reported in Figure 5 and Table 3. The proposed method achieves the lowest RMS position error while keeping the commanded inputs. The quantitative results in Table 3 report the tracking metrics for the representative disturbance realisation used in this study and the values are presented to illustrate qualitative performance trends rather than statistical significance. In the reported no-event wind–wave–current case, the proposed ESO+RL controller attains the smallest overall tracking error (${\mathrm{RMS}}_p=0.0382$ m and ${\mathrm{IAE}}_p=0.8687$ m·s). The proposed method achieves the best tracking performance among the compared methods in this representative case, while the margin over the ESO-only baseline is modest. Figure 5 summarises the tracking performance indicators for the compared methods in this representative disturbance case. The proposed method attains the smallest overall error level among the compared controllers, while the difference between ESO-only and ESO plus actor–critic remains modest. This supports the interpretation that ESO contributes the primary robustness gain, and the actor–critic coupling provides secondary refinement under the reported scenario. Figure 6 reports the evolution of actor and critic weight norm. Compared with RL-only, the proposed architecture yields a markedly smaller and more stationary critic weight norm, indicating that ESO and input-effectiveness adaptation reduce the residual uncertainty seen by the RL layer, thus improving learning stability and accelerating convergence. This shows the evolution of the actor and critic weight norms. The trajectories indicate that the coupled update law provides a bounded and stable parameter adaptation process, which is consistent with the Lyapunov-guided consistency regularisation interpretation of Equations (21) and (22). In this manuscript, we do not interpret these curves as evidence of TD error-driven optimal learning, but as evidence of stable online adaptation within the ESO-based robust control framework. Overall, the results in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 demonstrate that the proposed controller provides superior tracking accuracy under persistent disturbances in a no-event scenario, while maintaining feasible and bounded control inputs under identical actuator constraints.

The results indicate that the ESO module provides the dominant disturbance rejection capability in this scenario, and the actor–critic coupling mainly serves as a Lyapunov-guided consistency regularisation and online refinement component rather than the primary source of robustness. Therefore, the RL-only baseline can be close to the nominal controller and the additional benefit of ESO plus RL over ESO-only may be modest under the reported disturbance realisation.

5. Conclusions

In this study, a coordinate-consistent, ESO-enhanced actor–critic RL control scheme was developed to address the 3-DOF trajectory-tracking problem for marine vessels subjected to complex environmental disturbances. By leveraging an ESO with explicit reproducible disturbance shaping, the framework successfully decouples lumped uncertainties from the control law, thereby significantly reducing the learning burden on the neural network approximators. The proposed actor–critic architecture maintains computational efficiency via a Lyapunov-guided weight-matching consistency regularisation, supporting stable online refinement and real-time implementability; exact HJB optimality and policy optimality are not claimed under this simplified adaptation structure. Stability analysis demonstrates that the closed-loop system achieves SGUUB tracking, with the steady-state error bounds explicitly characterised by the ESO estimation residual. This integration of robust state estimation and adaptive learning provides a rigorous foundation for future enhancements, which will focus on incorporating actuator allocation constraints into the optimisation manifold and integrating formal safety filters to ensure operational resilience in constrained maritime environments.