Enhancing Robust Adaptive Dynamic Positioning of Full-Actuated Surface Vessels: Reinforcement Learning Approach for Unknown Hydrodynamics

Abstract

In this article, a robust adaptive dynamic position-control problem is addressed for full-actuated surface vessels under coupled uncertainties from unmodeled hydrodynamic effects and time-varying external disturbances. To obtain a high-performance dynamic position controller, a reinforcement learning (RL) weights law involving actor and critic networks is designed without knowledge of the model dynamics and the disturbance parameters. This can enhance the robustness of the closed-loop control system. Furthermore, dynamic surface control is integrated to diminish the design complexity resulting from the derivative of the kinematics, while ensuring semi-global uniformly ultimately bounded (SGUUB) stability through Lyapunov-based synthesis. Simulations are carried out to evaluate the superiority and feasibility of the proposed algorithm.

1. Introduction

In recent years, artificial intelligence has achieved an exploratory application in the field of marine engineering [1]. Some new approaches have been introduced into the autonomous control of marine vehicles for the high-efficiency implementation of engineering missions, such as path following, formation control, and dynamic positioning (DP), to name but a few. Among these tasks, especially, is the DP of fully actuated surface vehicles demanding high-precision operation under unpredictable external disturbances and model hydrodynamic parameters [2]. To this end, neural networks and a Takagi–Sugeno (T-S) fuzzy system [3] are utilized for addressing uncertainties [4], which involves a larger number of computational parameters. An explicit fact is that the RL strategy [5,6] should be further studied to enhance the control performance of the DP system [7], highlighting the need for more adaptive solutions.

In the literature, a large number of advanced DP control algorithms have been presented for fully actuated surface vehicles. By dynamically adjusting control parameters in real time, adaptive control methods [8,9] enhance the robustness and stability of systems to effectively address variations in system parameters and external disturbances [10]. In controller design using event-triggered approaches [11], computational resource utilization is optimized and operational efficiency improved by designing appropriate triggering conditions that reduce the update frequency of control signals, while maintaining system performance. Additionally, through the accurate modeling and compensation of nonlinear dead zones, dead-zone compensation control [12] is used to mitigate the impact of actuator nonlinearities within the system. Along similar lines, by predicting and compensating for communication and actuation delays, time-delay compensation control successfully enhances system’s reliability and response speed. Despite these advancements, the aforementioned methods do not achieve the expected results satisfactorily, and they still have significant limitations when dealing with uncertainties in the hydrodynamic model and time-varying external disturbances.

It should be pointed that the above-mentioned control strategies do not concern the strong nonlinear properties resulting from low-velocity operations, such as the complex hydrodynamic parameters between the hull and the ocean environment. Around these issues, although traditional nonlinear control approaches have achieved satisfactory performance in several applications, such as the controller design based on Lyapunov stability analysis, they still have limitations for complex, high-dimensional, and dynamical changes in nonlinear systems. To overcome this, with their strong approximation capabilities, neural networks are used in system modeling and control strategy design [13,14], which possess strong fault tolerance and robustness [15]. Similarly, through utilizing fuzzy rules to handle uncertainties, the fuzzy control approaches in [16] simplify complexity with linear systems. Currently, as an important branch of the artificial intelligence field, machine learning [17] has a broad application prospect. Furthermore, RL is an automatic learning approach [18] that interacts with the environment to adjust behavioral strategies through reward and punishment signals, as described in [19]. In particular, compared to traditional methods, where the system explored the simulated environment through trial and error, RL enables the direct acquisition of the optimal control strategy through interaction with the environment, without the need for a preexisting hydrodynamic model. It allows for real-time adaptation of the strategy through online learning, effectively handling time-varying disturbances and parameter uncertainties, and thereby enhancing positioning accuracy, energy efficiency, and resilience to interference [20]. Despite the fact that RL has been widely applied in various fields, it is largely untapped in DP. Through its adaptive learning ability, RL can optimize control strategies in real time in a dynamically changing environment, and it is particularly suitable for addressing challenges such as hydrodynamic uncertainty and external disturbances. The training efficiency and stability of the algorithm have been improved, and the dynamic response and disturbance suppression ability of the system have also been enhanced. By approximating solutions to the Hamilton–Jacobi–Bellman (HJB) equation, RL can address the nonlinearities inherent in DP systems [21], which offers a promising solution to their control challenges. RL provides a solution for DP systems that goes beyond traditional methods through model-free learning, online adaptation, complex disturbance suppression capabilities, and reduced training data requirements. Its unique advantages in scenarios of model uncertainty, time-varying disturbances, and multi-objective optimization make it an ideal choice for DP control in a highly dynamic marine environment.

Motivated by the above observations, this paper proposes an enhancing robust adaptive DP control algorithm for fully actuated surface vehicles by employing the A-C RL mechanism and dynamic surface control technique. The major contributions of this article are twofold:

(1)

Compared with conventional neural networks [13], the core innovation lies in a RL framework where actor–critic mechanisms are presented for the DP system: the actor network explicitly estimates multi-source model uncertainties, while the critic network dynamically optimizes the coupled interactions between positioning accuracy and environmental disturbances via a value function;

(2)

By constructing a gain-related adaptive law, a high-efficiency thrust allocation algorithm of the multi-actuators of the fully actuated vessels is proposed, where the complexity calculation can be avoided by employing the dynamic surface control technique. The proposed methodology significantly enhances operational autonomy while maintaining compatibility with standard marine control hardware platforms.

2. Problem for Formulation and Preliminaries

2.1. Preliminaries

Throughout this paper, $\mathrm{diag}\{y_1,\dots ,y_n\}$ denotes the main diagonal matrix, with the diagonal elements being $y_1,\dots ,y_n$. For a given matrix $\mathrm{X}=[x_{i,j}]\in {ℝ}^{m\times n}$ where ${ℝ}^{m\times n}$ represents the set of all $m\times n$ matrices consisting of real numbers, $m$ and $n$ represent the number of rows and columns of the matrix, respectively, ${‖\mathrm{X}‖}_F^2=\mathrm{tr}\left\{{\mathrm{X}}^{\mathrm{T}}\mathrm{X}\right\}={\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{j=1}^n{x}_{i,j}^2}}$. $\left|\cdot \right|$ indicates the matrices of the absolute operator of a scalar. $‖\cdot ‖$ is the Euclidean norm of a vector, and ${‖\cdot ‖}_F$ denotes the Frobenius norm and usually refers to the square of the Frobenius norm of the matrix, which is the sum of the squares of all elements. $\mathrm{tr}\left\{\mathrm{X}\right\}$ is the sum of the main diagonal elements of $\mathrm{X}$. In mathematics, the trace of a matrix is mathematically equivalent to the square of the Frobenius norm. $\left(\cdot \right)$ represents a certain variable or parameter; here, the “·” represents a placeholder indicating the name of a specific variable or parameter. $\widehat{\left(\cdot \right)}$ is the estimate of $\left(\cdot \right)$. The estimation error $\widetilde{\left(\cdot \right)}=\widehat{\left(\cdot \right)}-\left(\cdot \right)$.

2.2. Dynamic Model of Marine Vessels

The purpose of the DP of vessels at sea is to accurately control the vessel propulsion system, which can adjust the position of the vessel in real time to meet the high stability requirements of offshore operations and ensure the accuracy and stability of the working location. According to seakeeping and maneuvering theory, the model of the marine vessel for 3 degrees of freedom (DOFs) can be expressed as

$$\dot{\mathit{\eta }}=R\left(\psi \right)\mu$$

(1)

$$M\dot{\mu }+N\left(\mu \right)\mu =\tau +{\tau }_w$$

(2)

$$\tau =T\left(\beta \right)\kappa \left(n\right){\mu }_p$$

(3)

where $\eta ={\left[x,y,\psi \right]}^{\mathrm{T}}\in {ℝ}^3$ represents the position and heading angle vector in the geodetic coordinate system, $\mu ={\left[u,v,r\right]}^{\mathrm{T}}\in {ℝ}^3$ represents the velocity and rotational angular velocity vector in the hull coordinate system, and the rotational matrix between body-fixed and earth-fixed coordinate frames is expressed as Equation (4). Note that $R^{\mathrm{T}}\left(\psi \right)=R^{-1}\left(\psi \right)$, $‖R‖=1$.

$$R\left(\psi \right)=\left[\begin{array}{ccc}\cos \left(\psi \right)& \sin \left(\psi \right)& 0\\ -\sin \left(\psi \right)& \cos \left(\psi \right)& 0\\ 0& 0& 1\end{array}\right]$$

(4)

In Equation (2), $M\in {ℝ}^{3\times 3}$ is the inertial mass matrix, and $N\left(\mu \right)\mu$ are a nonlinear fluid mechanics function, which includes Coriolis forces and centripetal forces (moments), as well as nonlinear damping forces. $\tau ={\left[{\tau }_u,{\tau }_v,{\tau }_r\right]}^{\mathrm{T}}$ represents the controlling force and torque provided by the driving device, and in Equation (1) their detailed distribution models are given. ${\tau }_w={\left[{\tau }_{wu},{\tau }_{wv},{\tau }_{wr}\right]}^{\mathrm{T}}$ is a bounded disturbance caused by the disturbance of wind, wave, and current environment.

$$M=\left[\begin{array}{ccc}m-X_{\dot{u}}& 0& 0\\ 0& m-Y_{\dot{v}}& m{x}_G-Y_{\dot{r}}\\ 0& m{x}_G-Y_{\dot{r}}& I_z-N_{\dot{r}}\end{array}\right]$$

(5)

$$N\left(\mu \right)\mu =\left[\begin{array}{c}-X_u-X_{\left|u\right|u}\left|u\right|u+Y_{\dot{v}}v\left|r\right|+Y_{\dot{r}}rr\\ -X_{\dot{u}}ur-Y_{v}v-Y_{r}r-Y_{\left|v\right|v}\left|v\right|v-Y_{\left|v\right|r}\left|v\right|r\\ \left(X_{\dot{u}}-Y_{\dot{v}}\right)uv-Y_{\dot{r}}ur-N_{v}v-N_{r}r-N_{\left|v\right|v}\left|v\right|v-N_{\left|v\right|r}\left|v\right|r\end{array}\right]$$

(6)

$I_z$ represents the moment of inertia along the body-fixed $z_b$ axis. $Y_{\dot{v}}$, $Y_{\dot{r}}$, and $N_{\dot{r}}$ characterize added mass and inertia moment. Around $N\left(\mu \right)\mu ,X_u,X_{\left|u\right|u},Y_{\dot{v}}$, etc., are all hydrodynamic force derivatives.

In Equation (3), $T\left(\beta \right)\in {ℝ}^{3\times q}$ is the thrust allocation matrix. $q$ is the number of collocated thrusters. $\beta$ is the angle of the equivalent. $\kappa \left(n\right)=\mathrm{diag}\left\{{\kappa }_1\left(n_1\right),\dots ,{\kappa }_q\left(n_q\right)\right\}\in {ℝ}^{q\times q}$, is the unknown coefficient moment matrix that depends on the propeller rotation speed $n_i$, where $i=1,\dots ,q$. and ${\kappa }_i\left(n_i\right)$ is the elements of the $\kappa \left(n\right)$. In the actual use environment, the change of wake flow and the influence of hull movement make the performance of the propeller more difficult to predict and quantify. ${\mu }_p={\left[\left|p_1\right|p_1,\dots ,\left|p_q\right|p_q\right]}^{\mathrm{T}}$ is the actual controllable input, being the pitch ratio of thrusters $p_i\in \left[-1,1\right]$, where $i=1,\dots ,q$.

Assumption 1.

As the mass matrix, $M$ is positive, definite, and invertible. Due to the inherent port–starboard symmetry and approximate fore–aft symmetry of surface vessels, this assumption is typically satisfied.

Assumption 2.

In Equation (2), ${\tau }_w$ represents a bounded disturbance, i.e., there exists a positive constant vector ${\tau }_w={\left[{\tau }_{wu},{\tau }_{wv},{\tau }_{wr}\right]}^{\mathrm{T}}>0$. Note that ${\tau }_w$ is an unknown vector solely for the purposes of analysis.

Lemma 1.

In practical engineering, the thrust generated by a propulsor is always limited. Therefore, for a given vessel, the force coefficient of the propulsor is a constant that must satisfy the condition $0<\underset{¯}{{\kappa }_i}\le {\kappa }_i\left(n_i\right)\le \overline{{\kappa }_i}$, where $i=1,\dots ,q$. Both $\underset{¯}{{\kappa }_i}$ and $\overline{{\kappa }_i}$ are unknown and are used solely for stability analysis.

2.3. NN Function Approximation

In a control system, for any given nonlinear continuous function $f\left(x\right)$ satisfying the initial condition $f\left(0\right)=0$, the NN serves as an efficient approximator to model $f\left(x\right)$ [22]. From [23,24,25], the $f\left(x\right)$ is expressed approximately as

$$f\left(x\right)=W^{*\mathrm{T}}\varphi \left(x\right)+\epsilon \left(x\right)$$

(7)

where the matrix $W^{*\mathrm{T}}$ represents the ideal weight matrix; $\epsilon \left(x\right)$ denotes the approximation error; and $\varphi \left(x\right)$ represents the basis function vector, described by

$${\varphi }_i\left(x\right)={\displaystyle \frac{1}{\sqrt{2\pi }{\zeta }_i}}\exp \left(-{\displaystyle \frac{{\left(x-o_{ri}\right)}^{\mathrm{T}}\left(x-o_{ri}\right)}{{\zeta }_i^2}}\right),i=1,\dots ,l$$

(8)

with $o_{ri}\in {ℝ}^n$ representing the center vector and ${\zeta }_i$ denoting the width of the Gaussian function.

3. Robust Adaptive Neural Cooperative Controller

In this section, an RL approach based on the actor–critic framework is proposed to develop an adaptive control method [26] for the DP system of vessels. This method is capable of adapting to complex and dynamic marine environmental changes through continuous learning and the optimization of control policies. Under Assumptions 1 and 2, a robust neural adaptive control scheme is designed for the DP system, as described by Equations (1) and (2), utilizing dynamic surface control (DSC) techniques. The schematic diagram and detailed block diagram of the proposed DP system is illustrated in Figure 1 and Figure 2, providing a comprehensive overview of the system architecture.

3.1. Control Design

Let ${\eta }_d$ be the target position; the entire control synthesis consists of the following several steps. Define the position error vector as

$${\eta }_e=\eta -{\eta }_d$$

(9)

Taking the time derivative of Equation (9), one has

$${\dot{\eta }}_e=R\left(\psi \right)\mu -{\dot{\eta }}_d$$

(10)

Based on the position error dynamics Equation (10), the immediate control ${\alpha }_{\mu }\in {ℝ}^3$ can be directly selected as

$${\alpha }_{\mu }=R^{-1}\left(\psi \right)\left(-K_{\eta }{\eta }_e+{\dot{\eta }}_d\right)$$

(11)

where $K_{\eta }$ is a positive definite diagonal matrix. Obviously, the differential expression of ${\alpha }_{\mu }$ is very complicated and difficult to solve. In order to solve this problem, DSC technology [14,27] is applied, and the first-order filter is introduced in Equation (12) with the time constant matrix $t_{\mu }=\mathrm{diag}\left\{t_u,t_v,t_r\right\}$.

$${\alpha }_{\mu }-{\beta }_{\mu }=t_{\mu }{\dot{\beta }}_{\mu },{\alpha }_{\mu }\left(0\right)={\beta }_{\mu }\left(0\right)$$

(12)

The vector ${\beta }_{\mu }$ as the output is the reference signal for the velocity vector $\mu$. Meanwhile, ne can define the error vector as

$$q_{\mu }={\alpha }_{\mu }-{\beta }_{\mu }$$

(13)

and the velocity vector as

$${\mu }_e=\mu -{\beta }_{\mu }$$

(14)

The derivative of $q_{\mu }$ is obtained by Equations (13) and (14)

$$\begin{array}{ll}{\dot{q}}_{\mu }& ={\dot{\alpha }}_{\mu }-{\dot{\beta }}_{\mu }\\ & ={\Omega }_{\mu }\left(\psi ,r,{\eta }_e,{\dot{\eta }}_e\right)-t_{\mu }^{-1}{q}_{\mu }\end{array}$$

(15)

where ${\Omega }_{\mu }\left(\cdot \right)={\left[{\mathsf{\Omega }}_u\left(\cdot \right),{\mathsf{\Omega }}_v\left(\cdot \right),{\mathsf{\Omega }}_r\left(\cdot \right)\right]}^{\mathrm{T}}$ is a vector with its elements being bounded continuous functions. The position error derivative expression is further obtained from Equations (10) and (15) as

$${\dot{\eta }}_e=R\left(\psi \right)\left({\alpha }_{\mu }-q_{\mu }+{\mu }_e\right)-{\dot{\eta }}_d$$

(16)

The primary control variable ${\alpha }_{\mu p}={\left[{\alpha }_{up},{\alpha }_{vp},{\alpha }_{rp}\right]}^{\mathrm{T}}$ in the proposed design is chosen as Equation (22) for the desired thrusts term $T\left(\beta \right)\kappa \left(n\right){\mu }_p$.

Due to system complexity and dynamic environmental/operational conditions, the servo system governing propeller thrust faces substantial uncertainties, which include load variations, environmental disturbances, and fluctuations in system parameters [25], inducing variations in the thrust coefficient and compromising the stability of the propulsion system [28]. To address these challenges, a robust adaptive control strategy is proposed for adjusting controller parameters in real time to adapt to changes in propeller dynamics or the vessel’s state [29]. The estimation of $1/{\kappa }_i\left(n_i\right)$ is accomplished using ${\widehat{\lambda }}_k$ [30]. The control law, expressed as $p={\left[p_1,\dots ,p_q\right]}^{\mathrm{T}}$, is derived based on Equations (17) and (18), with the corresponding adaptive law detailed in Equation (19). This strategy ensures enhanced stability and operational efficiency under varying conditions.

$$p=\mathrm{sgn}\left({\mu }_p\right)\ast \sqrt{\left|{\mu }_p\right|}$$

(17)

$${\mu }_p=\mathrm{diag}\left\{{\widehat{\lambda }}_1,\dots ,{\widehat{\lambda }}_q\right\}{T}^{†}\left(\cdot \right){\alpha }_{\mu p}$$

(18)

$${\dot{\widehat{\lambda }}}_k={\chi }_k\left({\displaystyle \sum _{i=u,v,r}{\displaystyle \sum _{j=u,v,r}{T}_{k,j}^{†}{T}_{i,k}{i}_e{\alpha }_{jp}}}\right)-{\chi }_k\sigma k\left({\widehat{\lambda }}_k-{\lambda }_k\left(0\right)\right),k=1,\dots ,q$$

(19)

$${\alpha }_{\mu p}=-K_{\mu }{\mu }_e+{\widehat{W}}_a^{\mathrm{T}}{\varphi }_a+{\dot{\beta }}_{\mu }-R\left(\psi \right){\eta }_e$$

(20)

where $K_{\mu },{\chi }_k={\left[{\chi }_1,\dots ,{\chi }_q\right]}^{\mathrm{T}}$ and ${\sigma }_k={\left[{\sigma }_1,\dots ,{\sigma }_q\right]}^{\mathrm{T}}$ are the positive parameters; especially, ${\sigma }_k\left({\widehat{\lambda }}_k-{\widehat{\lambda }}_k\left(0\right)\right)$ could protect the estimation ${\widehat{\lambda }}_k$ from drifting divergence. $T^{†}$ denotes the pseudo inverse of $T$. According to the adaptive law Equation (20), the gain correlation coefficient is added to compensate for the influence of model uncertainty and interference [31].

3.2. Critic and Actor NN Design

This section introduces the RL [32] utilizing the critic and actor method to control the DP of vessels [33,34]. At the same time, in order to construct the Bellman error of nonlinear systems, the long-term strategy performance index function [35] is defined as follows:

$$J_c\left(t\right)={\displaystyle {\int }_t^{\infty }{\gamma }^{\frac{t-x}{\omega }}h\left({\eta }_e\right)}dx$$

(21)

where $\omega >0$ is integral reinforcement interval, and $\gamma$ is the discount rate that reduces the weight of this cost incurred further in the future. Based on the delay characteristics of the system, a smaller value (such as 0.9) was selected through the debugging of parameters in order to pay more attention to the long-term return. $J_c\left(t\right)={\left[J_{c1}\left(t\right),J_{c2}\left(t\right),J_{c3}\left(t\right)\right]}^{\mathrm{T}}$, which integrates future information that is not known at the current time related to the position error vector.

$$h_i=\left\{\begin{array}{c}0,\left|{\eta }_{ei}\right|\le C_i\\ 1,\left|{\eta }_{ei}\right|\ge C_i\end{array}\right.,i=1,2,3$$

(22)

where $C_i$ denotes a small positive threshold associated with tracking accuracy, typically set as 0.2, and $h_i$ and ${\eta }_{ei}$ express the $i^{\mathrm{th}}$ parameters of $h\left({\eta }_e\right)$ and ${\eta }_e$, respectively. Under the current control strategy, $h_i=1$ indicates a decrease in tracking performance and an increase in long-term performance indicators, while $h_i=0$ indicates the opposite, which reflects a significant tracking error under the current control strategy.

At the same time, the long-term strategy performance index function at time $t-\omega$ is defined as follows: $h_c\left(t\right)={\left[h_{c1}\left(t\right),h_{c2}\left(t\right),h_{c3}\left(t\right)\right]}^{\mathrm{T}}$, $h_c\left(t\right)={\displaystyle {\int }_{t-\omega }^t{\gamma }^{\frac{t-x}{\omega }}h\left({\eta }_e\right)}dx$.

However, for the highly nonlinear and coupled system described in Equation (3), the long-term utility function $J_c\left(t\right)$ incorporates future information that is unavailable at the current time, making it challenging to solve directly, even for linear systems. Only a limited class of nonlinear systems with specific functional designs and appropriate parameters can achieve the explicit evolution of $J_c\left(t\right)$. $J_c\left(t\right)$ is particularly difficult to solve. To address this issue, the critic NN, which is utilized as the approximation, is expressed as $J_c\left(t\right)\approx W_c^{*\mathrm{T}}\Delta {\varphi }_c\left(t\right)$ where $W_c^{*}$ is the ideal weight matrix of the critic NN, ${\eta }_e$ is the input vector of the critic NN, and ${\varphi }_c$ is the given critic NN vector. The actor NN operates based on the control law, driven by the critic NN’s evaluation of control performance [36].

From Equation (7), the actor NN $f\left(\mu \right)=W_a^{*}{\varphi }_a\left(\mu \right)+{\epsilon }_a\left(\mu \right)$ is designed. The update mechanism aims to maintain closed-loop stability and optimize the performance index $J_c\left(t\right)$. However, since $W_c^{*}$ $W_a^{*}$ is unknown, ${\widehat{W}}_c$ and ${\widehat{W}}_a$ are utilized to approximate $J_c\left(t\right)$ and $f\left(\mu \right)$ in real time, respectively.

As outlined in [37], the strategic utility function is constructed as ${\varpi }_a\left(t\right)={\widehat{W}}_a{\varphi }_a\left(\mu \right)+{\widehat{J}}_c\left(\mu \right)+{\mu }_e$, where ${\mu }_e$ is the input vector of the actor NN system at time $t$, ${\widehat{W}}_a$ represents the ideal weight matrix of the actor NN, and ${\epsilon }_a\left(\mu \right)$ is an NN approximation error. Meanwhile, the temporal difference error ${\varpi }_c$ is defined as ${\varpi }_c\left(t\right)={\widehat{W}}_c^{\mathrm{T}}\Delta {\varphi }_c\left(t\right)+h_c$, where $\Delta {\varphi }_c={\varphi }_c\left({\eta }_e\left(t\right)\right)-\gamma {\varphi }_c\left({\eta }_e\left(t-\omega \right)\right)$. Positive constants $K_{h_c}$, $K_{W_c}$, $K_{{\varphi }_c}$, $K_{{\epsilon }_c}$, $K_{W_a}$, $K_{{\varphi }_a}$ and $K_{{\epsilon }_a}$ exist such that ${‖h_c‖}_F\le K_{h_c}$, ${‖W_c^{\ast }‖}_F\le K_{W_c}$, ${‖{\varphi }_c\left({\eta }_e\right)‖}_F\le K_{{\varphi }_c}$, ${‖{\epsilon }_c\left({\eta }_e\right)‖}_F\le K_{{\epsilon }_c}$, ${‖\Delta {\varphi }_c‖}_F\le \left(1+\gamma \right)K_{{\varphi }_c}$, ${‖W_a^{\ast }‖}_F\le K_{W_a}$, ${‖{\varphi }_a\left(\mu \right)‖}_F\le K_{{\varphi }_a}$, ${‖{\epsilon }_a\left(\mu \right)‖}_F\le K_{{\epsilon }_a}$. Using a gradient descent algorithm, the adaptive control rate of critic and actor NN part are respectively defined as

$${\dot{\widehat{W}}}_{cj}=-{\gamma }_c{\displaystyle \frac{\partial \left({\varpi }_c^{\mathrm{T}}{\varpi }_c\right)}{{\widehat{W}}_{cj}}}=-{\gamma }_c\Delta {\varphi }_c\left(t\right){\left[{\widehat{W}}_{cj}^{\mathrm{T}}\Delta {\varphi }_c\left(t\right){\rho }_j+h_{ck}\right]}^{\mathrm{T}}$$

(23)

$${\dot{\widehat{W}}}_{ai}=-{\gamma }_a{\displaystyle \frac{\partial \left({\varpi }_a^{\mathrm{T}}{\varpi }_a\right)}{{\widehat{W}}_{ai}}}=-{\gamma }_a{\varphi }_a\left({\varphi }_a^{\mathrm{T}}{\widehat{W}}_{ai}{\rho }_i+{\mu }_e^{\mathrm{T}}+{\varphi }_c^{\mathrm{T}}{\widehat{W}}_{cj}{\rho }_j\right)$$

(24)

where ${\rho }_i$, ${\rho }_j$ are gain coefficients, ${\widehat{W}}_c={\left[{\widehat{W}}_{cx},{\widehat{W}}_{cy},{\widehat{W}}_{c\psi }\right]}^{\mathrm{T}}$, ${\widehat{W}}_a={\left[{\widehat{W}}_{au},{\widehat{W}}_{av},{\widehat{W}}_{ar}\right]}^{\mathrm{T}}$, $k=1,2,3$, $i=u,v,r$, and $j=x,y,\psi$. By repeatedly adjusting the size of the gain coefficient in the simulation, the stability and convergence during the training process are guaranteed.

To prevent one network form dominating while the other stagnates, the mitigation of uncoordinated updates between actor and critic NN is solved by the inclusion of gain coefficients. ${\gamma }_c=\mathrm{diag}\left({\gamma }_{cx},{\gamma }_{cy},{\gamma }_{c\psi }\right)>0$ and ${\gamma }_a=\mathrm{diag}\left({\gamma }_{au},{\gamma }_{av},{\gamma }_{ar}\right)>0$ represent the self-defined learning rate matrix. Then, one can define the weight error as ${\tilde{W}}_c={\widehat{W}}_c-W_c^{\ast }$ and ${\tilde{W}}_a={\widehat{W}}_a-W_a^{*}$. Neural networks play a crucial role in the training and performance of networks. To accelerate the training, adjust the network architecture to meet the requirements and ensure that in the actor–critic framework, the critic network can effectively estimate the value function, while the actor network can stably learn the optimal strategy.

Remark 1.

NNs are widely recognized as effective tools for the adaptive approximation of complex unknowns, such as unmodeled dynamics and uncertainties, within intelligent learning-based control frameworks. RL techniques, particularly those employing actor–critic architectures, are designed to optimize long-term rewards through a continuous interaction with online control performance. This performance is inherently influenced by both internal control strategies and external disturbances, which collectively shape the system’s behavior and adaptability. Through multiple repeated experiments, the parameters are evaluated, and the balance of each parameter is adjusted. Due to the complex design of the dynamic adjustment strategy, the hyperparameters need to be adjusted multiple times. Relying on empirical adjustment and repeated trial and error, the values of the parameters are finally determined in Equation (39).

3.3. Stability Analysis

Theorem 1.

Consider a closed-loop system composed of vessel dynamics, a robust neural control law, an actor–critic NN, and adaptive laws. Assume that the initial conditions are satisfied:

$${\eta }_e^{\mathrm{T}}\left(0\right){\eta }_e\left(0\right)+{\mu }_e^{\mathrm{T}}\left(0\right){\mu }_e\left(0\right)+q_e^{\mathrm{T}}\left(0\right)q_e\left(0\right)+{\tilde{W}}_a^{\mathrm{T}}\left(0\right){\gamma }_a^{-1}{\tilde{W}}_a\left(0\right)+{\tilde{W}}_c^{\mathrm{T}}\left(0\right){\gamma }_c^{-1}{\tilde{W}}_c\left(0\right)\le 0$$

and there exist adjustable parameters $K_{\eta },K_{\mu },t_{\mu },{\lambda }_k,{\chi }_k,{\gamma }_c,{\gamma }_a$. Then, the closed-loop system consisting of the vessel model Equations (1) and (2), the control law Equations (11) and (20), and the adaptive law Equation (19) is stable. The proposed adaptive NN controller, integrating RL, ensures the following closed-loop properties for all initial conditions:

(1)

Semi-Global Uniform Ultimate Boundedness: The state variables of the closed-loop system are semi-globally uniformly ultimately bounded;

(2)

Tightly Concentrated Errors: The tracking position error ${\eta }_e$, tracking velocity error ${\mu }_e$, and the NN weight errors ${\tilde{W}}_c$ and ${\tilde{W}}_c$ are kept within a tight region;

(3)

The optimal control strategy in RL is determined by approximating the ideal unknown weights. An estimator of actor and critic weights is designed to allow the NN to update recursively in parallel.

The proof is as follows: According to the characteristics of vessel DP and control design, the following Lyapunov function is used:

$$V={\displaystyle \frac{1}{2}}{\eta }_e^{\mathrm{T}}{\eta }_e+{\displaystyle \frac{1}{2}}{\mu }_e^{\mathrm{T}}M{\mu }_e+{\displaystyle \frac{1}{2}}{q}_e^{\mathrm{T}}{q}_e+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{k=1}^q\frac{\kappa \left(n_k\right){\tilde{\lambda }}_k^2}{{\chi }_k}}+{\displaystyle \frac{1}{2}}\mathrm{tr}\left({\gamma }_a^{-1}{\tilde{W}}_a^{\mathrm{T}}{\tilde{W}}_a\right)+{\displaystyle \frac{1}{2}}\mathrm{tr}\left({\gamma }_c^{-1}{\tilde{W}}_c^{\mathrm{T}}{\tilde{W}}_c\right)$$

(25)

The time derivative of Equation (25) is

$$\dot{V}={\eta }_e^{\mathrm{T}}{\dot{\eta }}_e+{\mu }_e^{\mathrm{T}}M{\dot{\mu }}_e+q_e^{\mathrm{T}}{\dot{q}}_e+{\displaystyle \sum _{k=1}^q\frac{\kappa \left(n_k\right){\tilde{\lambda }}_k{\dot{\widehat{\lambda }}}_k}{{\chi }_k}}+\mathrm{tr}{\left({\gamma }_a^{-1}{\tilde{W}}_a^{\mathrm{T}}\dot{\widehat{W}}\right)}_a+\mathrm{tr}\left({\gamma }_c^{-1}{\tilde{W}}_c^{\mathrm{T}}{\dot{\widehat{W}}}_c\right)$$

(26)

Similar to the former, we further have

$$\begin{array}{ll}{\eta }_e^{\mathrm{T}}{\dot{\eta }}_e& ={\eta }_e^{\mathrm{T}}\left(R\left(\psi \right)\mu -{\dot{\eta }}_d\right)\\ & \le -K_{\eta }{\eta }_e^{\mathrm{T}}{\eta }_e+{\eta }_e^{\mathrm{T}}R\left(\psi \right){\mu }_e+{‖{\eta }_e‖}_F^2+{\displaystyle \frac{1}{4}}{‖q_e‖}_F^2\end{array}$$

(27)

According to Equations (4) and (14) and Young’s inequality,

$${\mu }_e^{\mathrm{T}}M{\dot{\mu }}_e={\mu }_e^{\mathrm{T}}\left[\tau +{\tau }_{\omega }-N\left(\mu \right)\mu \right]-{\mu }_e^{\mathrm{T}}{q}_e{t}_{\mu }^{-1}M$$

(28)

where the $N\left(\mu \right)\mu -{\tau }_{\omega }$ term is treated with an $f\left(\mu \right)={\widehat{W}}_a^{\mathrm{T}}{\varphi }_a\left(\mu \right)+{\epsilon }_a\left(\mu \right)$ approximation. And according to Equation (28),

$$\begin{array}{ll}{\mu }_e^{\mathrm{T}}M{\dot{\mu }}_e& ={\mu }_e^{\mathrm{T}}\left[\tau -{\widehat{W}}_a^{\mathrm{T}}{\varphi }_a\left(\mu \right)-{\epsilon }_a\left(\mu \right)-q_e{t}_{\mu }^{-1}M\right]\\ & =-{\mu }_e^{\mathrm{T}}\left[K_{\mu }{\mu }_e+{\epsilon }_a\left(\mu \right)-T\kappa \left(\cdot \right){\tilde{\lambda }}_k{T}^{†}{\alpha }_{\mu p}\right]\\ & \le -\left(K_{\mu }-{\displaystyle \frac{1}{4}}I\right){\mu }_e^{\mathrm{T}}{\mu }_e+{\mu }_e^{\mathrm{T}}T\kappa \left(\cdot \right){\tilde{\lambda }}_k{T}^{†}{\alpha }_{\mu p}+{‖{\overline{\epsilon }}_a\left(\mu \right)‖}_F^2\end{array}$$

(29)

Moreover, we also have

$$\begin{array}{ll}{q}_{\mu }^{\mathrm{T}}{\dot{q}}_{\mu }& =q_{\mu }^{\mathrm{T}}\left({\dot{\alpha }}_{\mu }-{\dot{\beta }}_{\mu }\right)\\ & \le -q_{\mu }^{\mathrm{T}}{t}_{\mu }^{-1}{q}_{\mu }+{\displaystyle \sum _{i=u,v,r}\frac{K_{{\mathsf{\Omega }}_i}^2{q}_i^2}{2b}}+{\displaystyle \frac{3}{2}}\end{array}$$

(30)

Through the designed adaptive control law, Equation (19) is further obtained by

$${\displaystyle \sum _{k=1}^q\frac{\kappa \left(n_k\right){\tilde{\lambda }}_k{\dot{\widehat{\lambda }}}_k}{{\chi }_k}}={\displaystyle \sum _{k=1}^q\kappa \left(\cdot \right)}{\tilde{\lambda }}_k\left[{\displaystyle \sum _{i=u,v,r}{\displaystyle \sum _{j=u,v,r}{T}_{k,j}^{†}{T}_{i,k}{i}_e{\alpha }_{jp}}}-{\sigma }_k\left({\widehat{\lambda }}_k-{\dot{\widehat{\lambda }}}_k\left(0\right)\right)\right]$$

(31)

The first term cancels out the one of $T\kappa \left(\cdot \right){\tilde{\lambda }}_k{T}^{†}{\alpha }_{\mu p}{\mu }_e$, and the second term is determined by Young’s inequality:

$$T\kappa \left(\cdot \right){\tilde{\lambda }}_k{T}^{†}{\alpha }_{\mu p}{\mu }_e+{\displaystyle \sum _{k=1}^q\frac{\kappa \left(n_k\right){\tilde{\lambda }}_k{\dot{\widehat{\lambda }}}_k}{{\chi }_k}}\le -{\displaystyle \frac{1}{2}}{\displaystyle \sum _{k=1}^q\kappa \left(n_k\right){\sigma }_k}\left\{{\tilde{\lambda }}_k^2-\left[{\widehat{\lambda }}_k-{\widehat{\lambda }}_k\left(0\right)\right]\right\}$$

(32)

Finally, according to Equation (24), the error of RL weight is analyzed as

$$\mathrm{tr}\left({\gamma }_a^{-1}{\tilde{W}}_a^{\mathrm{T}}{\dot{\widehat{W}}}_a\right)\le -{\displaystyle \frac{1}{4}}{K}_{{\varphi }_a}^2\mathrm{tr}\left({\tilde{W}}_a^{\mathrm{T}}{\tilde{W}}_a\right)+{\mu }_e^{\mathrm{T}}{\mu }_e+K_{{\varphi }_c}^2\mathrm{tr}\left({\tilde{W}}_c^{\mathrm{T}}{\tilde{W}}_c\right)+K_{{\varphi }_a}^2{K}_{W_a}^2+K_{{\varphi }_c}^2{K}_{W_a}^2$$

(33)

$$\mathrm{tr}\left({\gamma }_c^{-1}{\tilde{W}}_c^{\mathrm{T}}{\dot{\widehat{W}}}_c\right)\le -{\displaystyle \frac{1}{2}}{\left(1+\gamma \right)}^2{K}_{{\varphi }_c}^2\mathrm{tr}\left({\tilde{W}}_c^{\mathrm{T}}{\tilde{W}}_c\right)+{\displaystyle \frac{1}{2}}{K}_{h_c}^2+{\left(1+\gamma \right)}^2{K}_{{\varphi }_c}^2{K}_{W_c}^2$$

(34)

To sum up, the final form is obtained by combining from Equation (27) to Equation (34):

$$\begin{array}{ll}\dot{V}& \le -\left(K_{\eta }-I\right){\eta }_e^{\mathrm{T}}{\eta }_e-\left(K_{\mu }-{\displaystyle \frac{5}{4}}I\right){\mu }_e^{\mathrm{T}}{\mu }_e-{\displaystyle \sum _{i=u,v,r}\left(t_{\mu }^{-1}-\frac{1}{4}-\frac{K_{{\mathsf{\Omega }}_i}^2}{2b}\right)}{q}_i^2\\ & -{\displaystyle \frac{1}{2}}{\displaystyle \sum _{k=1}^q\kappa \left(n_k\right){\sigma }_k{\tilde{\lambda }}_k^2}-{\displaystyle \frac{1}{4}}{K}_{{\varphi }_c}^2\left[{\left(1+\gamma \right)}^2-2\right]\mathrm{tr}\left({\tilde{W}}_c^{\mathrm{T}}{\tilde{W}}_c\right)-{\displaystyle \frac{1}{4}}{K}_{{\varphi }_a}^2\mathrm{tr}\left({\tilde{W}}_a^{\mathrm{T}}{\tilde{W}}_a\right)\\ & +{\displaystyle \frac{3}{2}}b+{\overline{\epsilon }}_a^2+{\displaystyle \frac{1}{2}}{{\displaystyle \sum _{k=1}^q\kappa \left(n_k\right){\sigma }_k\left[{\widehat{\lambda }}_k-{\widehat{\lambda }}_k\left(0\right)\right]}}^2+{\displaystyle \frac{1}{2}}{K}_{h_c}^2+K_{{\varphi }_a}^2{K}_{W_a}^2+K_{{\varphi }_c}^2{K}_{W_a}^2+{\left(1+\gamma \right)}^2{K}_{{\varphi }_c}^2{K}_{W_c}^2\\ & \le -\left({\lambda }_{\min }\left\{K_{\eta }\right\}-1\right){\eta }_e^{\mathrm{T}}{\eta }_e-\left({\lambda }_{\min }\left\{K_{\mu }\right\}-{\displaystyle \frac{5}{4}}\right){\mu }_e^{\mathrm{T}}{\mu }_e-{\displaystyle \sum _{i=u,v,r}\left(t_{\mu }^{-1}-\frac{1}{4}-\frac{K_{{\mathsf{\Omega }}_i}^2}{2b}\right)}{q}_i^2\\ & -{\displaystyle \frac{1}{2}}{\displaystyle \sum _{k=1}^q\kappa \left(n_k\right){\sigma }_k{\tilde{\lambda }}_k^2}-{\displaystyle \frac{1}{4}}{K}_{{\varphi }_c}^2\left[{\left(1+\gamma \right)}^2-2\right]\mathrm{tr}\left({\tilde{W}}_c^{\mathrm{T}}{\tilde{W}}_c\right)-{\displaystyle \frac{1}{4}}{K}_{{\varphi }_a}^2\mathrm{tr}\left({\tilde{W}}_a^{\mathrm{T}}{\tilde{W}}_a\right)+\varrho \end{array}$$

(35)

The design parameters are appropriately selected, satisfying

$$\begin{array}{l}{a}_1\le {\lambda }_{\min }\left\{K_{\eta }\right\}-1,a_2\le {\lambda }_{\min }\left\{K_{\mu }\right\}-{\displaystyle \frac{5}{4}},\\ a_3\le {\displaystyle \sum _{i=u,v,r}\left(t_{\mu }^{-1}-\frac{1}{4}-\frac{K_{{\mathsf{\Omega }}_i}^2}{2b}\right)},\\ a_4\le {\displaystyle \frac{1}{2}}{K}_{{\varphi }_c}^2\left[{\left(1+\gamma \right)}^2-2\right],a_5\le {\displaystyle \frac{1}{4}}{K}_{{\varphi }_a}^2\end{array}$$

(36)

where

$$\begin{array}{ll}a& =\min \left\{a_1,a_2,a_3,a_4,a_5\right\}\\ \varrho & ={\displaystyle \frac{3}{2}}b+{\overline{\epsilon }}_a^2+{\displaystyle \frac{1}{2}}{{\displaystyle \sum _{k=1}^q\kappa \left(n_k\right){\sigma }_k\left[{\widehat{\lambda }}_k-{\widehat{\lambda }}_k\left(0\right)\right]}}^2\\ & +{\displaystyle \frac{1}{2}}{K}_{h_c}^2+K_{{\varphi }_a}^2{K}_{W_a}^2+K_{{\varphi }_c}^2{K}_{W_a}^2+{\left(1+\gamma \right)}^2{K}_{{\varphi }_c}^2{K}_{W_c}^2\end{array}$$

(37)

By adjusting the following parameter matrix, all tracking errors of the closed-loop control system are converged to a compact set. Further, by adjusting the parameter $K_{\eta },K_{\mu },t_{\mu }^{-1},\gamma$, the following conditions can be obtained. Integrating two sides of Equation (35), one can obtain $V\left(t\right)\le \varrho /2a+\left(V\left(0\right)-\varrho /2a\right)\exp \left(-2at\right)$. The $V\left(t\right)$ can converge into $\varrho /2a$ with $t\to \infty$, and the bounded variable $\varrho$ can be small enough through turning the control parameters appropriately. Therefore, all the state variables in the closed-loop control system are SGUUB.

4. Numerical Simulation

In this section, two representative case studies are compared to illustrate the effectiveness and merits of the control scheme, i.e., a comparative experiment with the result in [38], and in order to verify the DP adaptive control method based on integral RL, MATLAB 2024a is used as the simulation platform to conduct experiments on the marine environment model. The DP system model used in the experiment includes a thruster, servo system, control system, and external disturbance model. And the environmental disturbances are as follows: the wind is at level 6, the number of wave frequency segments is 10, and the wave velocity is $10\times 1852/3600$. The nominal physical parameters [39] of the vessels are as follows in Equation (38):

$$\begin{array}{l}{X}_{\dot{u}}=-0.7212\times {10}^6,Y_{\dot{v}}=-3.6921\times {10}^6\\ Y_{\dot{r}}=-1.0234\times {10}^6,I_z-N_{\dot{r}}=3.7454\times {10}^9\\ X_u=5.0242\times {10}^4,Y_v=2.7229\times {10}^5\\ Y_r=-4.3933\times {10}^6,Y_{\left|v\right|v}=1.7860\times {10}^4\\ X_{\left|u\right|u}=1.0179\times {10}^3,Y_{\left|v\right|r}=-3.0068\times {10}^5\\ N_v=-4.3821\times {10}^6,N_r=4.1894\times {10}^8\\ N_{\left|v\right|v}=-2.4684\times {10}^5,N_{\left|v\right|r}=6.5759\times {10}^6\end{array}$$

(38)

With the initial states $\eta \left(0\right)={\left[10 \mathrm{m},10 \mathrm{m},140 \mathrm{deg}\right]}^{\mathrm{T}}$ and $\mu \left(0\right)={\left[10 \mathrm{m}/\mathrm{s},10 \mathrm{m}/\mathrm{s},140 \mathrm{deg}/\mathrm{s}\right]}^{\mathrm{T}}$, the desired position and heading are chosen as ${\eta }_d={\left[0 \mathrm{m},0 \mathrm{m},156 \mathrm{deg}\right]}^{\mathrm{T}}$. The design parameters utilized for the model are chosen as in Equation (39):

$$\begin{array}{l}{K}_{\eta }=\mathrm{diag}\left\{0.4,0.4,0.2\right\},K_{\mu }\mathrm{diag}\left\{0.4,0.4,0.2\right\},\\ {\chi }_k=\left[0.1,0.1,0.05,0.05,0.28,0.32,0.21\right],\\ {\sigma }_k=\left[3.2,3.2,4.1,3.5,\mathrm{1..8},1.2,3.1\right],\\ t_{\mu }=\mathrm{diag}\left\{0.01,0.01,0.01\right\},\\ {\gamma }_c=\mathrm{diag}\left\{0.01,0.01,0.01\right\},\\ {\gamma }_a=\mathrm{diag}\left\{0.01,0.01,0.01\right\},\\ {\rho }_x=230,{\rho }_y=220,{\rho }_{\psi }=200,\\ {\rho }_u=66,{\rho }_v=58,{\rho }_r=59,\\ C_x=0.3,C_y=0.3,C_{\psi }=0.3,\gamma =0.9,\omega =8\end{array}$$

(39)

The comparative experimental results are illustrated in Figure 3, Figure 4, Figure 5 and Figure 6. In Figure 3, with its trajectory exhibiting a narrower error range and smoother curve, the vessel’s position ultimately stabilizes within the desired attitude domain. As the simulation progresses, the system demonstrates a high accuracy, with the proposed algorithm outperforming in terms of control and positioning precision. The trajectory (the proposed curve) converges tightly within the desired error boundary. In contrast, the other curve shows persistent oscillations with a maximum deviation.

The comparison of specific position parameters $\left(x,y,\psi \right)$ in Figure 4 reveals that both algorithms attain fast convergence within the target region. However, the algorithm from [38], exhibits significant fluctuations in attitude and velocity errors, whereas the proposed algorithm maintains smoother variations, achieving a precise targeting of the desired attitude. Figure 5 presents a remarkable demonstration of control robustness, with the proposed algorithm reducing speed variations by an order of magnitude compared to the oscillatory behavior characteristic of conventional methods [38]. Under substantial external disturbances, the proposed algorithm reduces the speed fluctuations of the vessel to the target point more stably. Based on the adaptive control algorithm’s ability, the outcome highlights the RL optimizing the system’s response to disturbance, reducing the necessity for abrupt adjustments in propeller output to accommodate environmental changes. During the initial stage, there is a significant transient overshoot in [38], while the proposed algorithm has an overshoot of no more than 0.1 $\mathrm{m}/\mathrm{s}$. Safety improvement: The steady-state fluctuation range of $u$ and $v$ is reduced, indicating that the vessel is less susceptible to drift caused by wind and waves during positioning operations. Its low fluctuation and high-precision characteristics are particularly suitable for high-demand marine engineering scenarios. In the comparison in Figure 6, the proposed algorithm demonstrates smoother temporal variations in force and torque, minimizing sudden adjustments or fluctuations in control inputs. For DP systems requiring high precision and stability, the proposed method significantly enhances operational stability and control efficiency, thereby improving overall system performance metrics. The attenuation rate $a$ determines the speed at which the system state tends to be stable. It can be seen from the theoretical analysis that the convergence effect is good. In Figure 3 and Figure 4, both the velocity vector and the position vector have converged to zero in the domain near 100 s, and the dynamic indicators such as the overshoot and the adjustment time perform well. RL and adaptive control methods are adopted to further improve the convergence speed and enhance the transient response characteristics of the system. This scheme ensures that the system reaches a stable process quickly and smoothly, reduces over-rush and slows oscillation to make the oscillation amplitude smaller, and improves the stability of the system.

Figure 7 depicts the pitch rate of the control propeller, in the simulation experiments, with all values falling within reasonable ranges. The two subplots represent the pitch ratio and the azimuth angle of the azimuth thruster, respectively [30]. The azimuth angle of the propeller directly affects the anti-drift ability of the vessel by controlling the thrust direction of the thruster. Its dynamic adjustment can align the thrust vector in reverse with environmental disturbances (such as wind loads), significantly enhancing the positioning stability. Multiple thrusters are combined through differentiated azimuth angles to form a cooperative thrust vector field, achieving high-precision torque control. This azimuth–pitch joint control mechanism is optimized in real time through an adaptive algorithm. It can not only adjust the thrust phase in advance, according to the environmental interference prediction model, to avoid overdraft but also dynamically reconstruct the thrust distribution when the thruster fails. Figure 8 illustrates significant variations in the long-term utility function. Over time, the control system progressively minimizes the heading angle error, rendering changes negligible over time, thus achieving the required performance and stability. The primary objective of both networks is to minimize the utility function, optimizing the control strategy and reducing the impact of wind and wave variations to ensure high-precision positioning.

Figure 9 presents the parameters of the robust adaptive control, where the incorporation of the RL network significantly smooths their fluctuations, reflecting improved control performance. Subsequently, both position and velocity vector fluctuations remain minimal, maintaining precise positioning and robustness against disturbances in complex environments. Notably, due to the larger parameter space required for policy learning, the weight norm of the actor network is slightly larger than that of the critic network, reflecting its higher structural complexity. In contrast, the critic, which estimates the value function, typically operates within a less complex action space, leading to a smaller weight norm. This observation underscores the distinct structural and training demands of the two networks during the learning process. Figure 10 and Figure 11 present the cost function trends and norm of the current weights for the two networks. With more fluctuations in the early learning phase, the critics cost quantifies the error in estimating the long-term utility. The norm of the actor weights updates and focuses on optimizing the strategy to enhance future expected rewards, showing greater variation due to uncertainty in the training environment. The primary objective of the actor–critic RL method in tuning the action–critic NNs is to minimize the cost function $V_a={\varpi }_a^{\mathrm{T}}{\varpi }_a$ and $V_c={\varpi }_c^{\mathrm{T}}{\varpi }_c$, effectively tracking the target heading and maintaining small heading errors despite environmental disturbances. This method dynamically adjusts the control strategy based on historical data and environmental feedback, demonstrating a high accuracy in practical applications. The critic network evaluates the long-term value function and directs the actor to focus on actions that minimize cumulative future errors. The actor network learns a strategy that achieves a balance between tracking accuracy and energy efficiency. The proposed RL framework ensures adaptability to non-stationary disturbances, while avoiding the excessive punishment of control efforts. It is achieved through collaborative actor-driven strategy optimization and critic-guided value learning.

In Figure 12, in the area where the position and velocity error values rapidly decrease and then return to zero, even when the system is confronted with disturbances under harsh sea conditions, it can still rapidly converge and maintain stability and robustness.

Through the RL process, the agent learns the most energy-efficient control strategy while maintaining control precision, particularly in unstable sea conditions, where the system optimally manages propeller operations to avoid excessive adjustments. It could be heavily impacted by harsh sea conditions in [38], and its capacity to mitigate such disturbances was relatively weak. In contrast, the single parameter of the proposed algorithm can be finely adjusted, enabling the gradual stabilization of the critic network’s value estimation. With adequate training and sufficient interaction with the environment, the algorithm demonstrates improved performance.

5. Conclusions

Even though robust adaptive control boasts strong stability and the ability to handle uncertainties, when facing extreme conditions, especially sea condition changing rapidly, its judgement is slow, and its energy consumption is high. By combining RL with traditional robust control, it performs well in dynamic maritime environments, significantly reducing position and heading errors, while optimizing energy efficiency and ensuring real-time adaptability, which guarantees smaller error fluctuations and enhanced stability. Moreover, it addresses the problems of response delay and performance degradation in extreme conditions. This synergy improves control accuracy and stability, making it highly effective in challenging maritime operations.