Sliding Mode Control for Variable-Speed Trajectory Tracking of Underactuated Vessels with TD3 Algorithm Optimization

Abstract

An adaptive sliding mode controller (SMC) design with a reinforcement-learning parameter optimization method is proposed for variable-speed trajectory tracking control of underactuated vessels under scenarios involving model uncertainties and external environmental disturbances. First, considering the flexible control requirements of the vessel’s propulsion system, the desired navigation speed is designed to satisfy an S-curve acceleration and deceleration process. The rate of change of the trajectory parameters is derived. Second, to address the model uncertainties and external disturbances, an extended state observer (ESO) is designed to estimate the unknown bounded disturbances and to provide feedforward compensation. Moreover, an adaptive law is designed to estimate the upper bound of the unknown disturbances, ensuring system stability even in the presence of asymptotic observation errors. Finally, the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm is employed for real-time controller parameter tuning. Numerical simulation results demonstrate that the proposed method significantly improves the trajectory tracking accuracy and dynamic response speed of the underactuated vessel. Specifically, for a sinusoidal trajectory with an amplitude of 200 m and a frequency of 0.01, numerical results show that the proposed method achieves convergence of the longitudinal tracking error to zero, while the lateral tracking error remains stable within 1 m. For the circular trajectory with a radius of 300 m, the numerical results indicate that both the longitudinal and lateral tracking errors are stabilized within 1 m. Compared with the fixed-value sliding mode controller, the proposed method demonstrates superior trajectory tracking accuracy and smoother control performance.

1. Introduction

Driven by the significant advancements in artificial intelligence (AI) and machine learning (ML) technologies, underactuated vessels have shown considerable potential in applications such as marine surveying, hydro-meteorological observation, and transportation [1]. Compared with traditional fully actuated vessels, underactuated vessels are gaining attention due to their simplified structure, reduced energy consumption, and cost advantages. However, the reduction in the number of actuators (propellers or rudders) results in a decrease in the system’s controllable degrees of freedom (DOF), which makes motion control in complex marine environments more challenging. The motion control of underactuated vessels primarily includes trajectory tracking, dynamic positioning, and automatic berthing. Typically, these control methods are implemented on embedded system platforms; for example, the Robot Operating System (ROS). By executing control algorithms in real time on an embedded controller, both a fast response and high-precision control should be satisfied. Furthermore, the compact size and low power consumption of embedded devices provide optimal hardware support for the deployment of vessel control systems. With the rapid development and application of unmanned vessels, trajectory tracking control has become critical for enhancing the safety and efficiency of unmanned vessel operations in marine environments. Trajectory tracking requires the vessel to precisely follow a predetermined path and to reach the target position within the specified time frame. In practice, the vessel must adjust its rudder torque and propeller thrust to correct its position and heading in real time, compensating for deviations between the desired trajectory and actual trajectory caused by external disturbances such as wind, waves, and currents. Due to the nonlinear characteristics, large inertia effects, and model uncertainties of the underactuated vessel, the development of robust trajectory tracking control algorithms with a fast response, high accuracy, and effective disturbance rejection has become crucial for solving the motion control problems of underactuated vessels.

Many scholars have conducted studies on the control of trajectory tracking. Classical linear control methods, such as PID controllers [2,3,4,5,6] and the Linear Quadratic Regulator (LQR) [7,8,9,10], have proven effective in enhancing the tracking accuracy as well as in improving system stability and robustness. However, these methods exhibit limited adaptability for complex systems and typically rely on precise system models. In contrast, SMC exhibits superior robustness and adaptability in environments with severe nonlinearities and high disturbances [11,12,13,14,15]. Consequently, SMC has become an effective approach for achieving precise control of trajectory tracking.

Despite the numerous advantages of SMC, it also has certain limitations. For instance, high-frequency switching inherent in SMC can lead to sharp fluctuations in the control signal, which may negatively impact the actuators. Additionally, designing an appropriate sliding surface for complex systems often requires substantial theoretical and empirical foundations. Moreover, when the system state has not yet reached the sliding surface, the initial performance of SMC may be suboptimal, leading to less effective control. To address these challenges, extensive research has been dedicated to enhancing the performance and expanding the applicability of SMC. Mao et al. [16] addressed the challenges of input distribution uncertainty and system uncertainties in nonlinear systems by designing an observer to estimate and compensate for these uncertainties. They proposed an adaptive SMC to achieve stable control of nonlinear systems. Zhou et al. [17] focused on second-order nonlinear systems and designed a periodic delayed sliding mode surface, introducing a non-singular SMC to ensure that the system states converge to zero within a predetermined time. Wang et al. [18] tackled issues related to switching mechanisms, nonlinearity, and uncertainties in systems by proposing a switching fuzzy integral sliding mode surface. The effectiveness of this approach was demonstrated through theoretical analysis and numerical validation. Gandikota et al. [19] addressed nonlinearities and model mismatches by integrating adaptive mechanisms with super-twisting SMC, thereby enhancing system robustness and tracking performance. Numerous researchers have made significant improvements to SMC in complex nonlinear environments, leading to their widespread application in various engineering problems [20,21]. Similarly, SMC has also gained broad recognition in the field of vessel trajectory tracking control. Herman [22] addressed the trajectory tracking control problem for underactuated marine vehicles by designing a first-order integral sliding mode surface combined with a constant-speed reaching law. He proposed an adaptive integral SMC, achieving trajectory tracking for vehicles with symmetric inertia matrices. Zhu et al. [23] developed an exponential terminal SMC for underactuated unmanned surface vessels (USVs). Simulation results demonstrated that compared with a global integral SMC, the proposed exponential SMC exhibited superior convergence and robustness. Wang et al. [24] focused on the trajectory tracking problem in the horizontal plane of underactuated autonomous underwater vehicles (AUVs) and proposed a non-singular fast terminal SMC incorporating an exponential reaching law, which improved tracking speed and was validated through simulations. Xu et al. [25] introduced a non-singular terminal SMC for underactuated USVs. They designed a nonlinear ESO to estimate complex environmental disturbances, effectively addressing input saturation and enabling trajectory tracking control under unknown external disturbances. Wu et al. [26] proposed a dual-loop control structure for the three-dimensional trajectory tracking of underactuated AUVs. The inner-loop controller, based on non-singular integral terminal SMC, ensured rapid trajectory tracking and finite-time convergence. Lei et al. [27] presented an adaptive fast non-singular terminal SMC for underactuated vessel trajectory tracking, accelerating error convergence and avoiding singularity issues. They employed the dynamic surface control method to address the “differentiation explosion” problem, used an adaptive law to estimate the upper bound of unknown disturbances, and mitigated chattering by substituting the conventional sign function with a hyperbolic tangent function. Sun et al. [28] designed a dual-loop control system for trajectory tracking of underactuated USVs equipped with vector propulsion systems. They introduced a non-singular integral terminal SMC approach and approximated unknown disturbances using the minimal learning parameter theory, achieving precise control of the USVs. Zhang et al. [29] addressed finite-time trajectory tracking control for surface vessels affected by dynamic uncertainties and unknown time-varying disturbances. They used adaptive neural networks to approximate dynamic uncertainties and estimated error bounds through adaptive laws. Zhang et al. [30] tackled the trajectory tracking control problem for underactuated autonomous surface vessels subject to dynamic uncertainties and nonlinear external disturbances. They employed neural networks and low-frequency learning techniques to estimate and approximate system uncertainties effectively.

In summary, various improvements have been proposed by scholars to mitigate the chattering problem in SMC. These include the use of saturation functions, exponential approaching laws, and second-order sliding mode control, which effectively reduce the impact on actuators by smoothing the switching process. In terms of the sliding surface design, researchers typically combine Lyapunov stability theory with practical engineering experience to optimize the parameter settings of the sliding surface, thereby adapting it to the dynamic characteristics of different systems. Moreover, to enhance the performance of SMC during the initial approaching phase, some studies have integrated SMC with other control methods, such as adaptive control, fuzzy control, or deep reinforcement-learning algorithms. These hybrid control strategies significantly improve the response performance of SMC in complex environments. Therefore, this paper proposes a TD3-tuned SMC integrated with an ESO to estimate and feedforward compensate for unknown disturbances. Furthermore, existing research has mostly focused on constant-speed trajectory tracking. However, in practical vessel motion, the vessel’s speed often undergoes significant variations, especially during the automatic berthing process, where the speed changes are particularly pronounced. Endo [31] designed a corresponding berthing controller based on the typical steps of berthing (e.g., “full speed—half speed—slow speed—stop”) and established the relationship between vessel speed and propeller speed through a questionnaire survey. This approach was successfully applied in the controller design. In this context, this paper proposes a variable-speed trajectory tracking controller that meets the maneuvering requirements for port entry and exit, with the desired speed designed to follow an S-curve acceleration/deceleration profile.

This paper investigates the variable-speed trajectory tracking control problem for underactuated vessels, with the following main contributions: (1) In response to the maneuvering requirements during port entry and exit, a desired speed profile based on an S-curve acceleration/deceleration process is designed, and a variable-speed trajectory tracking controller is developed to address practical requirements; (2) An ESO is designed to estimate and compensate for unknown disturbances via feedforward control, and global system stability is analyzed using Lyapunov functions; (3) The TD3 algorithm is proposed to optimize the SMC parameters. By employing multiple networks and selecting the minimum value, the issue of value overestimation in reinforcement learning is addressed. Additionally, the delayed update strategy improves the stability of the actuator network, resulting in smoother parameter outputs from the SMC.

The remainder of this paper is organized as follows. Section 2 establishes a 3-DOF vessel model and defines the trajectory tracking problem. Section 3 utilizes an ESO to estimate and compensate for unknown disturbances through feedforward control and designs a SMC with TD3 algorithm optimization. Section 4 introduces the overall structure of the control system. Section 5 validates the effectiveness of the proposed algorithm through simulation results. Finally, Section 6 presents the conclusions.

2. Problem Formulation and Preliminaries

2.1. Problem Formulation

Before designing the trajectory tracking controller, this paper establishes an underactuated vessel’s 3-DOF control model. The model considers surge, sway, and yaw motions, providing the foundation for the subsequent control system design. The body-fixed coordinate system {b} and the inertial coordinate system {n} are shown in Figure 1, denoted as $O_b-x_b{y}_b{z}_b$ and $O_n-x_n{y}_n{z}_n$, respectively. The body-fixed coordinate system {b} typically has its origin at the vessel’s center of gravity, with $O_b{x}_b$ pointing along the vessel’s heading direction on the horizontal plane, $O_b{y}_b$ pointing to the vessel’s starboard side, perpendicular to the vessel’s longitudinal profile, and $O_b{z}_b$ pointing vertically upwards. The inertial coordinate system {n} typically has its origin at the earth’s center, with $O_n{x}_n$ pointing toward the north, $O_n{y}_n$ pointing toward the east, and $O_n{z}_n$ pointing toward the center of the Earth. The vessel’s control model is then given by Equations (1) and (2):

$$\left\{\begin{array}{l}\dot{x}=u\cos \psi -v\sin \psi \\ \dot{y}=u\sin \psi +v\cos \psi \\ \dot{\psi }=r\end{array}\right.$$

(1)

$$\left\{\begin{array}{l}\dot{u}={\displaystyle \frac{f_u\left(u,v,r\right)}{m_{11}}}+{\displaystyle \frac{m_{22}}{m_{11}}}vr+{\displaystyle \frac{{\tau }_u}{m_{11}}}+{\displaystyle \frac{{\tau }_{wu}}{m_{11}}}\\ \dot{v}={\displaystyle \frac{f_v\left(u,v,r\right)}{m_{22}}}-{\displaystyle \frac{m_{11}}{m_{22}}}ur+{\displaystyle \frac{{\tau }_{wv}}{m_{22}}}\\ \dot{r}={\displaystyle \frac{f_r\left(u,v,r\right)}{m_{33}}}+{\displaystyle \frac{{\tau }_r}{m_{33}}}+{\displaystyle \frac{{\tau }_{wr}}{m_{33}}}\end{array}\right.$$

(2)

Here, $u$, $v$, and $r$ denote the longitudinal linear velocity, lateral linear velocity, and yaw angular velocity, respectively, of the vessel in the {b} coordinate system. $x$, $y$, and $\psi$ denote the longitudinal coordinate, lateral coordinate, and yaw angle, respectively, of the vessel in the {n} coordinate system. ${\tau }_u$ and ${\tau }_r$ denote the longitudinal thrust from the vessel’s propeller and the rudder torque in the {b} coordinate system, respectively. ${\tau }_{wu}$, ${\tau }_{wv}$, and ${\tau }_{wr}$ denote the longitudinal interference force, lateral interference force, and yaw interference moment, respectively, acting on the vessel in the {b} coordinate system. $f_u\left(u,v,r\right)$, $f_v\left(u,v,r\right)$, and $f_r\left(u,v,r\right)$ denote the respective viscous forces acting on the vessel due to the fluid in the {b} coordinate system. $m_{11}$, $m_{22}$, and $m_{33}$ denote the total mass of the vessel, including the additional mass and inertia generated by the fluid’s inertial forces, respectively. Equation (1) denotes the kinematic model of the vessel, describing the relationship between the body-fixed coordinate system {b} and the inertial coordinate system {n}. Equation (2) denotes the dynamic model of the vessel, describing the forces acting on the vessel in the body-fixed coordinate system {b}.

Typically, underactuated vessels often use podded propulsion systems as their power devices. The forces generated by the pod system primarily consist of the propeller thrust $T_{pod}$ and the lateral force $F_N$, and their calculation formulas are given by Equation (3):

$$\begin{array}{l}{T}_{pod}=\rho n^2{D_p}^4{K}_T\left(J_p\right)\\ F_N=0.5\rho A_{pod}f\left(\Lambda \right){U_p}^2\sin {\alpha }_{pod}\end{array}$$

(3)

Here, $\rho$ denotes the seawater density. $n$ denotes the propeller rotational speed. $D_p$ denotes the propeller diameter. $K_T$ denotes the propeller thrust coefficient. $J_p$ denotes the propeller advance ratio. $A_{pod}$ denotes the effective area of the podded propulsor. $f\left(\Lambda \right)$ is a function of the pod pillar aspect ratio, where $\Lambda$ typically takes a value between 0.5 and 3.0. $U_p$ denotes the effective incoming flow speed at the pod. ${\alpha }_{pod}$ denotes the effective angle of attack at the pod. $x_p$ denotes the X-axis coordinate of the position of the pod propeller. When the pod rotation angle is $\delta$, the resulting longitudinal thrust and rudder torque are given by Equation (4):

$$\left\{\begin{array}{l}{\tau }_u=\left(1-t_p\right)T_{pod}\cos \delta -\left(1-t_R\right)F_N\sin \delta \\ {\tau }_r=-\left[\left(1-t_p\right)T_{pod}\sin \delta +\left(1+a_{Hpod}\right)F_N\cos \delta \right]L_{pod}\end{array}\right.$$

(4)

Here, $L_{pod}$ denotes the distance from the pod to the vessel’s center. $t_p$ denotes the thrust deduction coefficient. $t_R$ denotes the lateral deduction coefficient. ${\alpha }_{Hpod}$ denotes the correction coefficient for calculating the induced lateral force generated by the hull when the pod is steered.

Based on the control model, a sliding mode controller for trajectory tracking of the underactuated vessel is designed. Without loss of generality, the following assumptions are made:

Assumption 1. 

The vessel’s parameterized tracking trajectory $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$ is smooth and has first and second derivatives. Here, $\omega$ is a function of time t, and its variation law $\dot{\omega }\left(t\right)$ determines the desired speed for trajectory tracking.

Assumption 2. 

The external disturbances ${\tau }_{wu}$, ${\tau }_{wv}$, and ${\tau }_{wr}$ acting on the underactuated vessel are time-varying and bounded, with unknown upper bounds, and satisfy the following inequality:

$$\left|{\tau }_{wu}\right|\le {\tau }_{wu}^{*}, \left|{\tau }_{wv}\right|\le {\tau }_{wv}^{*}, \left|{\tau }_{wr}\right|\le {\tau }_{wr}^{*}$$

(5)

Here, ${\tau }_{wu}^{*}>0$, ${\tau }_{wv}^{*}>0$, and ${\tau }_{wr}^{*}>0$ denote the unknown upper bounds of the disturbances.

Therefore, the control objective of this paper is as follows: For the underactuated vessel control model, considering the unknown bounded disturbances from wind and waves, a trajectory tracking sliding mode controller is designed based on the LOS guidance law and an S-curve desired speed. This controller ensures that the vessel’s closed-loop system maintains ultimately uniformly bounded observation errors. Furthermore, the TD3 algorithm is employed for the online optimization of the SMC parameters, thereby enhancing the trajectory tracking performance.

2.2. Preliminaries

In the trajectory tracking control of underactuated vessels, the control objective is to adjust the propeller thrust and rudder torque to gradually bring the vessel closer to the desired trajectory and, ultimately, guide it along the predetermined path. The LOS guidance method, as a simple and efficient trajectory tracking strategy, has been widely applied in the field of vessel control. The core idea of the LOS guidance method is to steer the vessel’s heading toward the target, guiding it progressively along the predefined path. Among the various LOS methods, the look-ahead distance LOS (LADLOS) method is particularly favored for its strong adaptability and low requirements for path smoothness, making it widely used in trajectory tracking for vessels, vehicles, robots, and other mobile platforms. In the LADLOS method, the system dynamically calculates a look-ahead point on the target path based on the current position. The look-ahead point is located on the target path at a fixed look-ahead distance from the vessel’s current position.

This paper uses the LADLOS method to design the desired heading angle, as shown in Figure 2. The current position of the underactuated vessel is denoted as $\left(x,y\right)$, and the desired point is denoted as $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$, where $\omega$ is the parameter of the desired trajectory. The look-ahead point $\left(x_{los},y_{los}\right)$ lies on the tangent to the desired trajectory, and the distance between the look-ahead point and the desired point is $\Delta$. Based on the navigation principles of the LADLOS method, the desired heading angle of the vessel is given by Equation (6):

$${\chi }_{los}={\psi }_d+\arctan \left(-{\displaystyle \frac{y_e}{\Delta }}\right)$$

(6)

Here, ${\psi }_d$ denotes the angle of the desired trajectory, and $y_e$ denotes the lateral tracking error for trajectory tracking. Considering the effect of the vessel’s sway motion, a drift angle will be generated. Thus, the reference angle ${\psi }_r$ is given by Equation (7):

$${\psi }_r={\chi }_{los}-\beta ={\psi }_d+\arctan \left(-{\displaystyle \frac{y_e}{\Delta }}\right)-\beta$$

(7)

$$\beta =\arctan \left({\displaystyle \frac{v}{u}}\right)$$

(8)

The use of the drift angle $\beta$ to pre-correct the desired heading angle ${\chi }_{los}$ is a common approach in the LOS guidance law. However, during the simulation of variable-speed trajectory tracking in this paper, it is observed that under low-speed and high-disturbance conditions, the vessel’s longitudinal speed and lateral speed are similar in magnitude. The drift angle $\beta$ can easily lead to differential peak phenomena with rapid changes, causing the drift angle $\beta$ to dominate the reference angle ${\psi }_r$, resulting in system instability and making trajectory tracking difficult. The underlying issue lies in the fact that the drift angle $\beta$ is corrected based on speed rather than displacement. Since speed is the derivative of displacement, ${\psi }_r$ becomes highly sensitive to changes in the lateral speed at low speeds. To address this, a first-order inertia element is introduced to filter the drift angle $\beta$, with the discrete first-order inertial filter given by Equation (9):

$${\beta }_{n+1}^{\prime }={\displaystyle \frac{T}{T+\Delta t}}{\beta }_n^{\prime }+{\displaystyle \frac{\Delta t}{T+\Delta t}}{\beta }_{n+1}$$

(9)

In this context, ${\beta }_n^{\prime }$ and ${\beta }_{n+1}^{\prime }$ denote the filtered drift angle at the previous time step and current time step, respectively. ${\beta }_{n+1}$ is the raw drift angle calculated at the current time. $T$ denotes the time constant of the first-order inertia element. $\Delta \mathrm{t}$ denotes the sampling period of the discrete system. By applying the first-order inertial filter to smooth the drift angle, abrupt changes in the yaw angle are avoided, which prevents sharp variations in the rudder angle during heading control, thereby making the approach more consistent with engineering practice.

In the trajectory tracking of an underactuated vessel, as the desired point $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$ continuously advances, the moving speed depends on the rate of change of the parameters $\dot{\omega }\left(t\right)$. The trajectory tracking controller controls the vessel to track the desired point with speed U and yaw angle $\psi$. Once the vessel reaches the desired trajectory, it remains on the trajectory and moves forward at the desired speed. Therefore, the rate of change of the parameter $\dot{\omega }\left(t\right)$ not only determines the magnitude of the reference angle ${\psi }_r$ but also determines whether the vessel can track the desired trajectory at the desired speed.

Lemma 1. 

For any smooth parameterized tracking trajectory $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$, if the trajectory variation rate is chosen as $\dot{\omega }=\left(U+k_{\omega }{x}_e\right)/\zeta$, the vessel’s yaw angle $\psi$ will converge to the reference angle ${\psi }_r$, and the vessel will converge to the desired trajectory at the desired speed. Here, $\zeta =\sqrt{{x_d^{\prime }}^2+{y_d^{\prime }}^2}$, $U=\sqrt{u^2+v^2}$. $x_d^{\prime }$ and $y_d^{\prime }$ denote the first-order derivatives of the reference trajectory $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$ with respect to the parameter $\omega$. $k_{\omega }$ denotes a positive constant, and $x_e$ denotes the longitudinal tracking error.

Proof of Lemma 1. 

Let the desired position of the underactuated vessel be denoted as $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$. The longitudinal tracking error and lateral tracking error for trajectory tracking in the body coordinate system {b} are given by Equation (10):

$$\left[\begin{array}{c}{x}_e\\ y_e\end{array}\right]=\left[\begin{array}{cc}\cos \left({\psi }_d\right)& \sin \left({\psi }_d\right)\\ -\sin \left({\psi }_d\right)& \cos \left({\psi }_d\right)\end{array}\right]\left[\begin{array}{c}x-x_d\\ y-y_d\end{array}\right]$$

(10)

Here, $x_e$ and $y_e$ denote the longitudinal tracking error and the lateral tracking error in the {b} coordinate system. ${\psi }_d$ denotes the angle of the desire trajectory. Taking the derivatives of $x_e$ and $y_e$, we obtain:

$$\begin{array}{l}\left[\begin{array}{c}{\dot{x}}_e\\ {\dot{y}}_e\end{array}\right]=\left[\begin{array}{cc}\cos {\psi }_d& \sin {\psi }_d\\ -\sin {\psi }_d& \cos {\psi }_d\end{array}\right]\left[\begin{array}{c}\dot{x}-{\dot{x}}_d\\ \dot{y}-{\dot{y}}_d\end{array}\right]+\left[\begin{array}{cc}-{\dot{\psi }}_d\sin {\psi }_d& {\dot{\psi }}_d\cos {\psi }_d\\ -{\dot{\psi }}_d\cos {\psi }_d& -{\dot{\psi }}_d\sin {\psi }_d\end{array}\right]\left[\begin{array}{c}x-x_d\\ y-y_d\end{array}\right]\\ =\left[\begin{array}{c}u\cos \left(\psi -{\psi }_d\right)-v\sin \left(\psi -{\psi }_d\right)-\dot{\omega }\left(x_d^{\prime }\cos {\psi }_d+y_d^{\prime }\sin {\psi }_d\right)+{\dot{\psi }}_d{y}_e\\ u\sin \left(\psi -{\psi }_d\right)+v\cos \left(\psi -{\psi }_d\right)+\dot{\omega }\left(x_d^{\prime }\sin {\psi }_d-y_d^{\prime }\cos {\psi }_d\right)-{\dot{\psi }}_d{x}_e\end{array}\right]\\ =\left[\begin{array}{c}U\cos \left(\psi -{\psi }_d+\beta \right)-\dot{\omega }\sqrt{{x_d^{\prime }}^2+{y_d^{\prime }}^2}\cos \left({\psi }_d-{\psi }_d\right)+{\dot{\psi }}_d{y}_e\\ U\sin \left(\psi -{\psi }_d+\beta \right)+\dot{\omega }\sqrt{{x_d^{\prime }}^2+{y_d^{\prime }}^2}\sin \left({\psi }_d-{\psi }_d\right)-{\dot{\psi }}_d{x}_e\end{array}\right]\\ =\left[\begin{array}{c}U\cos \left(\psi -{\psi }_d+\beta \right)+{\dot{\psi }}_d{y}_e-\dot{\omega }\zeta \\ U\sin \left(\psi -{\psi }_d+\beta \right)-{\dot{\psi }}_d{x}_e\end{array}\right]\end{array}$$

(11)

Here, $\zeta =\sqrt{{x_d^{\prime }}^2+{y_d^{\prime }}^2}$, $U=\sqrt{u^2+v^2}$. The trajectory variation rate is defined as $\dot{\omega }=\left(U+k_{\omega }{x}_e\right)/\zeta$, with $k_{\omega }>U/\Delta$.

Select the Lyapunov function given by Equation (12):

$$V={\displaystyle \frac{1}{2}}{x}_e^2+{\displaystyle \frac{1}{2}}{y}_e^2$$

(12)

Taking the derivative of Equation (12), we obtain

$$\begin{array}{l}\dot{V}=x_e{\dot{x}}_e+y_e{\dot{y}}_e\\ =x_e\left(U\cos \left(\psi -{\psi }_d+\beta \right)+{\dot{\psi }}_d{y}_e-\dot{\omega }\rho \right)+y_e\left(U\sin \left(\psi -{\psi }_d+\beta \right)-{\dot{\psi }}_d{x}_e\right)\\ =U{x}_e\cos \left(\psi -{\psi }_d+\beta \right)-U{x}_e+U{y}_e\sin \left(\psi -{\psi }_d+\beta \right)-k_{\omega }{x_e}^2\\ =U{x}_e\left(\cos \left(\psi -{\psi }_d+\beta \right)-1\right)+U{y}_e\sin \left(\psi -{\psi }_d+\beta \right)-k_{\omega }{x_e}^2\end{array}$$

(13)

Define ${\psi }_e=\psi -{\psi }_r$, where the purpose of the heading control is to converge the vessel’s yaw angle $\psi$ to the reference angle ${\psi }_r$; hence, ${\psi }_e\to 0$. Combining with Equation (7), it can be derived that $\left(\psi -{\psi }_d+\beta \right)\to \arctan \left(-y_e/\Delta \right)$. Therefore,

$$\sin \left(\psi -{\psi }_d+\beta \right)=-{\displaystyle \frac{y_e}{\sqrt{{y_e}^2+{\Delta }^2}}}$$

(14)

Substituting Equation (14) into Equation (13), we obtain

$$\dot{V}=U{x}_e\left[\cos \left(\psi -{\psi }_d+\beta \right)-1\right]-{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-k_{\omega }{x_e}^2$$

(15)

Due to the properties of trigonometric functions, we obtain $\cos \left(\psi -{\psi }_d+\beta \right)-1<\left|\sin \left(\psi -{\psi }_d+\beta \right)\right|$. Thus,

$$\begin{array}{l}\dot{V}<U\left|x_e\right|\sin \left(\psi -{\psi }_d+\beta \right)-{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-k_{\omega }{x_e}^2\\ =-{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-k_{\omega }{x_e}^2+{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}\left|x_e\right|\left|y_e\right|\\ =-{\displaystyle \frac{1}{2}}{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-{\displaystyle \frac{1}{2}}{k}_{\omega }{x_e}^2-{\displaystyle \frac{1}{2}}\left[{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-2{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}\left|x_e\right|\left|y_e\right|+k_{\omega }{x_e}^2\right]\end{array}$$

(16)

Moreover, since $k_{\omega }>U/\Delta$, we obtain

$$\begin{array}{l}\dot{V}<-{\displaystyle \frac{1}{2}}{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-{\displaystyle \frac{1}{2}}{k}_{\omega }{x_e}^2-{\displaystyle \frac{1}{2}}\left[{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-2{\displaystyle \frac{\sqrt{U/\Delta }\sqrt{k_{\omega }}}{\sqrt{{y_e}^2/{\Delta }^2+1}}}\left|x_e\right|\left|y_e\right|+k_{\omega }{x_e}^2\right]\\ \le -{\displaystyle \frac{1}{2}}{\displaystyle \frac{U}{\sqrt{{y_e}^2+{\Delta }^2}}}{y_e}^2-{\displaystyle \frac{1}{2}}{k}_{\omega }{x_e}^2-{\displaystyle \frac{1}{2}}{\left[\sqrt{k_{\omega }}\left|x_e\right|-{\displaystyle \frac{\sqrt{U/\Delta }}{\sqrt{{y_e}^2/{\Delta }^2+1}}}\left|y_e\right|\right]}^2<0\end{array}$$

(17)

From the above, it can be concluded that $\dot{V}<0$, and $x_e$, $y_e$ asymptotically converge to 0. □

In the heading control process, as the lateral tracking error $y_e$ decreases, the vessel’s yaw angle $\psi$ approaches the reference angle ${\psi }_r$, ultimately aligning the vessel’s yaw angle $\psi$ with the angle of the desired trajectory ${\psi }_d$, thus achieving heading control. In the speed control process, as the longitudinal tracking error decreases, the speed of the desired point on the desired trajectory aligns with the vessel’s motion speed, meaning that the desired point and the vessel move forward at the same speed.

In the speed control process, the selection of the desired speed profile is critical for control performance. In dynamic and complex scenarios, such as in the precise operations of a vessel in port, the flexible working requirements of the motors necessitate smooth speed variations, a fast dynamic response, and system stability. In this context, the seven-segment S-curve acceleration/deceleration process has become the ideal choice [32,33]. By introducing continuous changes in the acceleration and jerk, this method not only effectively avoids the vibration issues caused by abrupt changes in acceleration in traditional methods but also significantly improves the smoothness of trajectory tracking and the control accuracy. Furthermore, the S-curve design provides flexible parameter adjustment capabilities, allowing the system to find the optimal balance between response speed and comfort based on the actual task requirements. This smooth-speed planning approach ensures the system’s robustness against environmental disturbances and mitigates the adverse effects of nonlinear dynamics on control performance. Therefore, the seven-segment S-curve acceleration/deceleration process, with its high dynamic performance, excellent smoothness, and strong engineering feasibility, has become the ideal method for generating the desired velocity in control systems, providing a stable and efficient input foundation for subsequent controller design.

The seven stages of the S-curve acceleration/deceleration process are as follows: jerk acceleration phase (T1), uniform acceleration phase (T2), deceleration acceleration phase (T3), constant velocity phase (T4), acceleration–deceleration phase (T5), uniform deceleration phase (T6), and jerk deceleration phase (T7), as shown in Figure 3. In this paper, the variable $J$ is introduced to represent the rate of change of acceleration.

$$J={\displaystyle \frac{da}{dt}}\left(J>0\right)$$

(18)

Based on the vessel’s propulsion characteristics, the maximum acceleration is denoted as $a_{\max }$, and a variable ${\tau }_k=t-t_k\left(k=1,\cdots ,7\right)$ is introduced to represent the operating time in the current stage. The operating time for each stage, namely T1, T2, T3, T5, T6, and T7, is $T$. The calculation formula for the S-curve acceleration process is given by Equation (19):

$$u_{d1}={\displaystyle \frac{1}{2}}J{{\tau }_1}^2, u_{d2}={\displaystyle \frac{1}{2}}J{T}^2+JT{\tau }_2, u_{d3}={\displaystyle \frac{3}{2}}J{T}^2+JT{\tau }_3-{\displaystyle \frac{1}{2}}J{{\tau }_3}^2$$

(19)

The calculation formula for the S-curve deceleration process is given by Equation (20):

$$u_{d5}=2J{T}^2-{\displaystyle \frac{1}{2}}J{{\tau }_5}^2, u_{d6}={\displaystyle \frac{3}{2}}J{T}^2-JT{\tau }_6, u_{d7}={\displaystyle \frac{1}{2}}J{T}^2-JT{\tau }_7+{\displaystyle \frac{1}{2}}J{{\tau }_7}^2$$

(20)

3. Adaptive Sliding Mode Controller

In Section 2, based on the underactuated vessel’s motion model, this paper employs the LADLOS method to compute the desired heading angle and designs a desired speed profile that follows the S-curve acceleration/deceleration process, laying the foundation for trajectory tracking control. However, in real maritime environments, vessel motion is inevitably affected by complex external disturbances, such as wind, waves, and currents, which exhibit uncertainty and dynamic characteristics. Therefore, in this section, the focus is on addressing the trajectory tracking control problem under the influence of unknown disturbances. To tackle these challenges, an ESO is first used to estimate unknown environmental disturbances. To ensure system stability during the progressive estimation process, even in the presence of observation errors, an adaptive control law is applied to estimate the upper bound of the unknown disturbances. Furthermore, the estimated disturbance results are used as feedforward compensation in the SMC, and an adaptive sliding mode control law based on the hyperbolic tangent function is designed to replace the traditional sign function. This approach smooths the control input and avoids high-frequency chattering. Finally, the TD3 algorithm is introduced for real-time tuning of the SMC parameters. This control strategy significantly enhances the system’s robustness and trajectory tracking performance.

3.1. Extended State Observer (ESO)

In the trajectory tracking of underactuated vessels, traditional control methods struggle to achieve sufficient accuracy in the state estimation and dynamic feedback due to external disturbances in the complex marine environment as well as the nonlinear characteristics and modeling uncertainties inherent in the vessel’s system. This results in degraded control performance. To address this issue, the ESO [34] is introduced as an effective auxiliary tool. The ESO is designed to estimate the system’s state variables and external disturbances in real time, thereby providing more precise feedback signals for the controller. The fundamental idea behind designing the ESO is to model the system’s dynamic characteristics and external disturbances as an “extended state”, which is then estimated by the observer. The uncertainties of the vessel and external environmental disturbances, such as wind and waves, are unified as unknown disturbance terms in the system. An ESO is used to observe the vessel’s states and unknown disturbances. Let $x_1=\upsilon$, $\upsilon =\left[u,v,r\right]$, $x_2=M^{-1}{\tau }_w$, $M^{-1}=diag[1/m_{11},1/m_{22},1/m_{33}]$, and ${\tau }_w=[{\tau }_{wu},{\tau }_{wv},{\tau }_{wr}]$. Considering the underactuated vessel model, the design of the ESO is given by Equation (21):

$$\left\{\begin{array}{l}{\dot{\widehat{x}}}_1=M^{-1}f+{\widehat{x}}_2+M^{-1}\tau -{\beta }_1\left({\widehat{x}}_1-x_1\right)\\ {\dot{\widehat{x}}}_2=-{\beta }_2\left({\widehat{x}}_1-x_1\right)\end{array}\right.$$

(21)

Here, $f=\left[f_u\left(u,v,r\right)+m_{22}vr,f_v\left(u,v,r\right)-m_{11}ur,f_r\left(u,v,r\right)\right]$, and $\tau =\left[{\tau }_u,0,{\tau }_r\right]$. ${\widehat{x}}_1$ denotes the observed value of speed, and ${\widehat{x}}_2$ denotes the observed value of $M^{-1}{\tau }_w$. ${\beta }_i\left(i=1,2\right)$ is the design positive constant, and it denotes the gain of the ESO. Typically, ${\beta }_1=2{\omega }_0,{\beta }_2={\omega }_0^2$. ${\omega }_0$ denotes the bandwidth, which is determined by the frequency of the disturbance signal to be observed.

Define the observation error as $e={\left[e_1,e_2\right]}^T$, where $e_1={\widehat{x}}_1-x_1$, and $e_2={\widehat{x}}_2-M^{-1}{\tau }_w$. Combining Equations (1), (2) and (21), the state equation for the observation error is given by Equation (22):

$$\begin{gathered} \dot{e}=Ae+B{M}^{-1}{\dot{\tau }}_w \\ A=\left[\begin{array}{cc}-{\beta }_1& 1\\ -{\beta }_2& 0\end{array}\right], B=\left[\begin{array}{c}0\\ -1\end{array}\right] \end{gathered}$$

(22)

Clearly, by solving the differential equation in Equation (22), we obtain

$$e\left(t\right)=e^{At}e\left(0\right)+{\displaystyle {\int }_0^t{e}^{A\left(t-\tau \right)}}{M}^{-1}{\dot{\tau }}_{w}d\tau B$$

(23)

From Assumption 2, it follows that $‖{\dot{\tau }}_w‖\le S$, where $S$ is a positive design constant. For the matrix $A$, there exists a Vandermonde matrix $N$ that satisfies the following relationship: $A=Ndiag\left(-{\beta }_1,-{\beta }_2\right)N^{-1}$. Therefore,

$$\begin{array}{l}‖e\left(t\right)‖\le ‖e^{At}‖‖e\left(0\right)‖+S‖M^{-1}‖‖{\displaystyle {\int }_0^t{e}^{A\left(t-\tau \right)}}d\tau ‖‖B‖\\ \le ‖Ndiag\left(e^{-{\omega }_{0}t},e^{-{\omega }_{0}t}\right)N^{-1}‖‖e\left(0\right)‖+S‖M^{-1}‖‖{\displaystyle {\int }_0^t{e}^{A\left(t-\tau \right)}}d\tau ‖‖B‖\\ \le ‖N‖‖N^{-1}‖‖e^{-{\omega }_{0}t}‖‖e\left(0\right)‖+S‖N‖‖N^{-1}‖‖M^{-1}‖‖{\displaystyle {\int }_0^t{e}^{-{\omega }_0\left(t-\tau \right)}}d\tau ‖‖B‖\\ \le ‖N‖‖N^{-1}‖\left(‖e^{-{\omega }_{0}t}‖‖e\left(0\right)‖+S{\displaystyle \frac{1}{{\omega }_0}}‖M^{-1}‖‖1-e^{-{\omega }_{0}t}‖‖B‖\right)\end{array}$$

(24)

From $‖B‖=1$, it is known that as $t\to \infty$, $‖1-e^{-{\omega }_{0}t}‖\to 1$. According to Equation (24), the convergence of $‖e\left(t\right)‖$ is related to the bandwidth value ${\omega }_0$. The larger the bandwidth, the faster the $‖e\left(t\right)‖$ convergence, and it will ultimately converge to the boundary value $\left(S/{\omega }_0\right)‖M^{-1}‖$. Therefore, the observation error of the disturbance estimated by the ESO is convergent and will eventually converge to the boundary value.

3.2. Heading and Speed Sliding Mode Control

The core idea of SMC is to design a sliding surface such that the system’s state converges to this surface and remains on it, thereby achieving the desired trajectory tracking. In this paper, the states of the underactuated vessel are defined as the speed error and heading error, with the calculation formulas given by Equation (25):

$$u_e=u-u_d, {\psi }_e=\psi -{\psi }_r$$

(25)

Design the sliding surfaces given by Equation (26) and Equation (27), respectively.

$$s_1=u_e+{\lambda }_1{\displaystyle \underset{0}{\overset{t}{\int }}{u}_{e}d\tau },{\lambda }_1>0$$

(26)

$$s_2={\dot{\psi }}_e+{\lambda }_2{\psi }_e=r-{\dot{\psi }}_r+{\lambda }_2{\psi }_e,{\lambda }_2>0$$

(27)

Considering the underactuated vessel model and taking the time derivative of Equations (26) and (27), the results are given by Equations (28) and (29).

$${\dot{s}}_1={\dot{u}}_e+{\lambda }_1{u}_e=\left({\displaystyle \frac{f_u}{m_{11}}}+{\displaystyle \frac{m_{22}}{m_{11}}}vr+{\displaystyle \frac{{\tau }_{wu}}{m_{11}}}+{\displaystyle \frac{{\tau }_u}{m_{11}}}\right)-{\dot{u}}_d+{\lambda }_1{u}_e$$

(28)

$${\dot{s}}_2=\dot{r}-{\ddot{\psi }}_r+{\lambda }_2{\dot{\psi }}_e={\displaystyle \frac{f_r}{m_{33}}}+{\displaystyle \frac{{\tau }_{wr}}{m_{33}}}+{\displaystyle \frac{{\tau }_r}{m_{33}}}-{\ddot{\psi }}_r+{\lambda }_2{\dot{\psi }}_e$$

(29)

Based on the SMC concept, the control laws are given by Equations (30) and (31).

$${\tau }_u=m_{11}{\dot{u}}_d-{\lambda }_1{m}_{11}{u}_e-f_u-m_{22}vr-{\widehat{\tau }}_{wu}-{\epsilon }_1\tanh \left(s_1\right)-k_1{s}_1$$

(30)

$${\tau }_r=m_{33}{\ddot{\psi }}_r-{\lambda }_2{m}_{33}{\dot{\psi }}_e-f_r-{\widehat{\tau }}_{wr}-{\epsilon }_2\tanh \left(s_2\right)-k_2{s}_2$$

(31)

Here, ${\epsilon }_1$, $k_1$, ${\epsilon }_2$, and $k_2$ are positive constants. In the equation, the $\tanh \left(\cdot \right)$ is used to replace the $sign\left(\cdot \right)$ in the traditional SMC.

Theorem 1. 

For the underactuated vessel motion model given by Equations (1) and (2), under the conditions specified in Assumptions 1 and 2, and given the desired trajectory  $\left(x_d\left(\omega \right),y_d\left(\omega \right)\right)$ and the desired speed profile, the SMC laws for thrust (Equation (30)) and steering torque (Equation (31)) are designed, and the boundary values of external environmental disturbances are estimated. By appropriately selecting the parameters ${\epsilon }_1$, ${\epsilon }_2$, $k_1$, $k_2$, ${\lambda }_1$, and ${\lambda }_2$, it can be guaranteed that the underactuated vessel will track the desired trajectory with the desired speed, and that all states of the trajectory tracking system are ultimately uniformly bounded.

Proof of Theorem 1. 

Construct the Lyapunov function given by Equation (32).

$$V_3={\displaystyle \frac{1}{2}}{m}_{11}{s_1}^2+{\displaystyle \frac{1}{2}}{m}_{33}{s_2}^2+{\displaystyle \frac{1}{2{\gamma }_1}}{{\tilde{\tau }}_{wu}^{*}}^2+{\displaystyle \frac{1}{2{\gamma }_2}}{{\tilde{\tau }}_{wr}^{*}}^2$$

(32)

Here, ${\widehat{\tau }}_{wu}^{*}$ and ${\widehat{\tau }}_{wr}^{*}$ denote the estimated values of ${\tau }_{wu}^{*}$ and ${\tau }_{wr}^{*}$, respectively. ${\tilde{\tau }}_{wu}^{*}={\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^{*}$ and ${\tilde{\tau }}_{wr}^{*}={\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^{*}$ denote the observation errors of the disturbance bounds of the vessel’s external disturbances. Taking the derivative of the Lyapunov function yields

$${\dot{V}}_3=m_{11}{s}_1{\dot{s}}_1+m_{33}{s}_2{\dot{s}}_2+{\displaystyle \frac{1}{{\gamma }_1}}{\tilde{\tau }}_{wu}^{*}{\dot{\tilde{\tau }}}_{wu}^{*}+{\displaystyle \frac{1}{{\gamma }_2}}{\tilde{\tau }}_{wr}^{*}{\dot{\tilde{\tau }}}_{wr}^{*}$$

(33)

Substituting Equations (28) and (29), we obtain

$${\dot{V}}_3=-k_1{s_1}^2-{\epsilon }_1{s}_1\tanh \left(s_1\right)-{\tilde{\tau }}_{wu}{s}_1-k_2{s_2}^2-{\epsilon }_2{s}_2\tanh \left(s_2\right)-{\tilde{\tau }}_{wr}{s}_2+{\displaystyle \frac{1}{{\gamma }_1}}{\tilde{\tau }}_{wu}^{*}{\dot{\widehat{\tau }}}_{wu}^{*}+{\displaystyle \frac{1}{{\gamma }_2}}{\tilde{\tau }}_{wr}^{*}{\dot{\widehat{\tau }}}_{wr}^{*}$$

(34)

Here, ${\tilde{\tau }}_{wu}={\widehat{\tau }}_{wu}-{\tau }_{wu}$, and ${\tilde{\tau }}_{wr}={\widehat{\tau }}_{wr}-{\tau }_{wr}$. The adaptive laws for designing ${\widehat{\tau }}_{wu}^{*}$ and ${\widehat{\tau }}_{wr}^{*}$ are formulated as follows [35]:

$$\left\{\begin{array}{l}{\dot{\widehat{\tau }}}_{wu}^{*}={\gamma }_1\left(s_1\varphi \left(s_1\right)-{\sigma }_1\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)\right)\\ {\dot{\widehat{\tau }}}_{wr}^{*}={\gamma }_2\left(s_2\varphi \left(s_2\right)-{\sigma }_2\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\right)\end{array}\right.$$

(35)

In Equation (35), ${\gamma }_1$, ${\gamma }_2$, ${\sigma }_1$, and ${\sigma }_2$ are positive design constants, while ${\sigma }_1$ and ${\sigma }_2$ are small. $\varphi \left(s_i\right)=\tanh \left(s_i/{\varsigma }_i\right)$, $i=1,2$. ${\varsigma }_i$ is a positive design constant. ${\tau }_{wu}^0$ and ${\tau }_{wr}^0$ denote the prior estimates of ${\widehat{\tau }}_{wu}^{*}$ and ${\widehat{\tau }}_{wr}^{*}$, respectively. Substituting Equation (35) into Equation (34), we obtain

$$\begin{array}{l}{\dot{V}}_3=-k_1{s_1}^2-{\epsilon }_1{s}_1\tanh \left(s_1\right)-{\tilde{\tau }}_{wu}{s}_1-k_2{s_2}^2-{\epsilon }_2{s}_2\tanh \left(s_2\right)-{\tilde{\tau }}_{wr}{s}_2\\ +{\tilde{\tau }}_{wu}^{*}\left(s_1\varphi \left(s_1\right)-{\sigma }_1\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)\right)+{\tilde{\tau }}_{wr}^{*}\left(s_2\varphi \left(s_2\right)-{\sigma }_2\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\right)\end{array}$$

(36)

Moreover, due to the properties of $\tanh \left(\cdot \right)$, it can be concluded that

$$\begin{array}{l}{\dot{V}}_3\le -k_1{s_1}^2-k_2{s_2}^2-{\epsilon }_1\left|s_1\right|-{\epsilon }_2\left|s_2\right|-{\tilde{\tau }}_{wu}^{*}\left|s_1\right|-{\tilde{\tau }}_{wr}^{*}\left|s_2\right|\\ +{\tilde{\tau }}_{wu}^{*}\left(s_1\varphi \left(s_1\right)-{\sigma }_1\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)\right)+{\tilde{\tau }}_{wr}^{*}\left(s_2\varphi \left(s_2\right)-{\sigma }_2\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\right)\\ =-k_1{s_1}^2-k_2{s_2}^2-{\epsilon }_1\left|s_1\right|-{\epsilon }_2\left|s_2\right|-{\tilde{\tau }}_{wu}^{*}\left(\left|s_1\right|-s_1\varphi \left(s_1\right)\right)-{\tilde{\tau }}_{wr}^{*}\left(\left|s_2\right|-s_2\varphi \left(s_2\right)\right)\\ +{\sigma }_1\left({\tau }_{wu}^{*}-{\widehat{\tau }}_{wu}^{*}\right)\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)+{\sigma }_2\left({\tau }_{wr}^{*}-{\widehat{\tau }}_{wr}^{*}\right)\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\end{array}$$

(37)

For any positive constants ${\varsigma }_i$, ${\varpi }_i\le 1\left(i=1,2\right)$, we obtain [36]

$$0\le \left|{\varpi }_i\right|-{\varpi }_i\varphi \left({\varpi }_i\right)=\left|{\varpi }_i\right|-\tanh \left({\varpi }_i/{\varsigma }_i\right)\le 0.2785{\varsigma }_i, i=1,2$$

(38)

Taking into account the inequality given by Equation (39),

$$\begin{array}{l}{\sigma }_1\left({\tau }_{wu}^{*}-{\widehat{\tau }}_{wu}^{*}\right)\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)\le -{\displaystyle \frac{{\sigma }_1}{2}}{\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^{*}\right)}^2+{\displaystyle \frac{{\sigma }_1}{2}}{\left({\tau }_{wu}^{*}-{\tau }_{wu}^0\right)}^2\\ {\sigma }_2\left({\tau }_{wr}^{*}-{\widehat{\tau }}_{wr}^{*}\right)\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\le -{\displaystyle \frac{{\sigma }_2}{2}}{\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^{*}\right)}^2+{\displaystyle \frac{{\sigma }_2}{2}}{\left({\tau }_{wr}^{*}-{\tau }_{wr}^0\right)}^2\end{array}$$

(39)

Substituting Equation (39) into Equation (37), we obtain

$$\begin{array}{l}{\dot{V}}_3\le -k_1{s_1}^2-k_2{s_2}^2-{\epsilon }_1\left|s_1\right|-{\epsilon }_2\left|s_2\right|-{\tilde{\tau }}_{wu}^{*}\left(\left|s_1\right|-s_1\varphi \left(s_1\right)\right)-{\tilde{\tau }}_{wr}^{*}\left(\left|s_2\right|-s_2\varphi \left(s_2\right)\right)\\ +{\sigma }_1\left({\tau }_{wu}^{*}-{\widehat{\tau }}_{wu}^{*}\right)\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^0\right)+{\sigma }_2\left({\tau }_{wr}^{*}-{\widehat{\tau }}_{wr}^{*}\right)\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^0\right)\\ \le -k_1{s_1}^2-k_2{s_2}^2-{\epsilon }_1\left|s_1\right|-{\epsilon }_2\left|s_2\right|-0.2785{\varsigma }_1{\tilde{\tau }}_{wu}^{*}-0.2785{\varsigma }_2{\tilde{\tau }}_{wr}^{*}\\ -{\displaystyle \frac{{\sigma }_1}{2}}{\left({\widehat{\tau }}_{wu}^{*}-{\tau }_{wu}^{*}\right)}^2+{\displaystyle \frac{{\sigma }_1}{2}}{\left({\tau }_{wu}^{*}-{\tau }_{wu}^0\right)}^2-{\displaystyle \frac{{\sigma }_2}{2}}{\left({\widehat{\tau }}_{wr}^{*}-{\tau }_{wr}^{*}\right)}^2+{\displaystyle \frac{{\sigma }_2}{2}}{\left({\tau }_{wr}^{*}-{\tau }_{wr}^0\right)}^2\\ \le -{\mu }_1\left({s_1}^2+{s_2}^2+\left|s_1\right|+\left|s_2\right|+{\tilde{\tau }}_{wu}^{*}+{\tilde{\tau }}_{wr}^{*}+{{\tilde{\tau }}_{wu}^{*}}^2+{{\tilde{\tau }}_{wr}^{*}}^2\right)+C\end{array}$$

(40)

Here,

$$\begin{array}{l}{\mu }_1=\min \left(k_1,k_2,{\epsilon }_1,{\epsilon }_2,0.2785{\varsigma }_1,0.2785{\varsigma }_2,{\displaystyle \frac{{\sigma }_1}{2}},{\displaystyle \frac{{\sigma }_2}{2}}\right)\\ C={\displaystyle \frac{{\sigma }_1}{2}}{\left({\tau }_{wu}^{*}-{\tau }_{wu}^0\right)}^2+{\displaystyle \frac{{\sigma }_2}{2}}\left({\tau }_{wr}^{*}-{\tau }_{wr}^0\right)\end{array}$$

(41)

From the above equation, it can be concluded that ${\dot{V}}_3<0$ when $‖{s_1}^2+{s_2}^2+\left|s_1\right|+\left|s_2\right|+{\tilde{\tau }}_{wu}^{*}+{\tilde{\tau }}_{wr}^{*}+{{\tilde{\tau }}_{wu}^{*}}^2+{{\tilde{\tau }}_{wr}^{*}}^2‖>C/{\mu }_1$. Thus, the system achieves stability. Therefore, the signals $s_1$, $s_2$, ${\tilde{\tau }}_{wu}^{*}$, and ${\tilde{\tau }}_{wr}^{*}$ in the closed-loop system are uniformly ultimately bounded. It can also be concluded that $u_e$ and ${\psi }_e$ are bounded. By combining Equation (17), it can be ensured that the vessel’s trajectory tracking error is uniformly ultimately bounded. □

3.3. TD3 Algorithm Parameter Optimization

TD3 is an efficient, policy-gradient-based algorithm in the field of deep reinforcement learning, specifically designed for continuous action space problems. The core of TD3 lies in the use of a deterministic policy and an actor–critic architecture to approximate the optimal policy. The primary objective is to learn the optimal policy through interactions with the environment, thereby maximizing the cumulative reward. The TD3 algorithm is an optimized version of the Deep Deterministic Policy Gradient (DDPG) algorithm. It utilizes two separate critic networks, and when computing the target value, the smaller of the two is chosen, effectively mitigating the overestimation issue found in the DDPG. Additionally, random noise is introduced into the action of the next state during target value computation, enhancing the accuracy of value estimation. Finally, the actor network is updated only after multiple updates to the critic network, ensuring stability in actor network training. The parameter optimization structure based on TD3 is shown in Figure 4.

The TD3 environment constructed in this paper consists of an underactuated vessel model and a sliding mode controller. The state space and action space are defined as follows: $\left\{s_1,s_2,s_3,s_4\in S|s_1=x_e,s_2=y_e,s_3={\psi }_e,s_4=u_e\right\}$ and $\left\{a_1,a_2\in A|a_1={\lambda }_1,a_2={\lambda }_2\right\}$. The state variables reflect the longitudinal tracking error, lateral tracking error, yaw tracking error, and speed tracking error. The action space denotes the parameters of the sliding mode controller that need to be optimized. The objective of trajectory tracking for the underactuated vessel is to minimize the error between the vessel’s state and the desired state. As the trajectory moves toward the target trajectory, the longitudinal tracking error, lateral tracking error, yaw tracking error, and speed tracking error should gradually approach zero. Therefore, the reward function is designed as follows:

$$r={\displaystyle \frac{1}{\left|s_1\right|+\left|s_2\right|+\left|s_3\right|+\left|s_4\right|+\varsigma }}$$

(42)

$\varsigma$ denotes a small positive constant, which is used to prevent the issue of infinite rewards when the state variables are all zero. The TD3 algorithm includes six neural networks, and the parameters of the neural networks are set as shown in

Table 1. Network structure.

Network	Hidden Layer	Neurons	Input	Output
Target actor network	3	256	2	4
Target critic network1	3	256	6	1
Target critic network2	3	256	6	1
Actor network	3	256	2	4
Critic network	3	256	6	1
Critic network	3	256	6	1

.

4. System Control Structure

To address the trajectory tracking problem of underactuated vessels, this paper proposes a sliding mode controller optimized by the TD3 algorithm. The overall control structure is shown in Figure 5. The system input is the desired trajectory $\left(x\left(\omega \right),y\left(\omega \right)\right)$ and speed $u_d$, while the outputs are the vessel’s actual state. The method involves constructing two control subsystems to achieve precise control of the speed and the yaw angle, thereby fulfilling the trajectory tracking control objective. The control system takes the speed tracking error and yaw tracking error as control state inputs and designs an integral sliding mode surface for speed control and a differential sliding mode surface for heading control. Based on the sliding mode surface feedback control strategy, the system outputs the longitudinal thrust and steering torque, ensuring that the longitudinal tracking error and the lateral tracking error gradually converge to achieve accurate tracking of the desired trajectory. To handle external and model uncertainties, an ESO is introduced to observe and compensate for the compound disturbances in real time, and an adaptive law is designed to estimate the upper bound of the unknown disturbances, ensuring system stability even in the presence of gradual observation errors. To further optimize the performance of the sliding mode controller, the TD3 algorithm is utilized to optimize the control parameters. The TD3 algorithm is an enhanced version of the DDPG algorithm, mitigating the value overestimation issue present in the DDPG, and employing a delayed update strategy to ensure smoothness of the control parameter outputs and the stability of system dynamic responses.

5. Simulation Results

To verify the effectiveness of the proposed method, a simulation is conducted on a recreational boat from Ningbo FlyShark Power Technology Co., Ltd, located in Ningbo, China. Considering the influence of hydrodynamic factors, the Inoue model [37] is introduced to calculate the viscous forces generated by the system, as shown in Equation (43). $X_{uu}$, $X_{vv}$, $X_{vr}$, $X_{rr}$, $Y_v$, $Y_r$, $Y_{vv}$, $Y_{vr}$, $Y_{rr}$, $N_v$, $N_r$, $N_{rr}$, $N_{vvr}$, and $N_{vrr}$ denote hydrodynamic coefficients, and in practical applications, their dimensionless forms are often calculated using empirical formulas. The Zhou Zhaoming formula [38] is used to calculate the additional inertial forces generated by the system, with the corresponding formula provided in Equation (44). $m_x$, $m_y$, and $J_{zz}$ denote the additional mass in the X and Y directions and additional moment of inertia in the Z direction, respectively. $I_{zz}$ denotes the moment of inertia about the Z-axis. The dimensionless parameters of the model and the hydrodynamic parameters obtained from the calculations are listed in

Table 2. Main data of the ship model.

Parameter	Value	Parameter	Value	Parameter	Value	Parameter	Value
Displacement ($m$)	1148 kg	$m_{11}$	1170.58	$Y_v^{\prime }$	−0.2316	${N_{rr}}^{\prime }$	−0.0496
Length ($L$)	7.201 m	$m_{22}$	1865.66	$Y_r^{\prime }$	0.06544	${N_{vvr}}^{\prime }$	−0.5987
Width ($B$)	2.507 m	$m_{33}$	8472.23	${Y_{vv}}^{\prime }$	−0.5490	${N_{vrr}}^{\prime }$	−0.0624
Draft ($d$)	0.3 m	$X_{uu}$	−413.58	${Y_{vr}}^{\prime }$	−0.2576
Block coefficient ($C_b$)	0.2068	$X_{vv}$	−5.27	${Y_{rr}}^{\prime }$	−0.0377
Wetted surface area ($S$)	8.07 ${\mathrm{m}}^2$	$X_{vr}$	−834.71	${N_v}^{\prime }$	−0.0833
Water Density ($\rho$)	1025 kg/${\mathrm{m}}^2$	$X_{rr}$	413.41	${N_r}^{\prime }$	−0.0380

, where the “dash” data represents the dimensionless data. The parameters for the TD3 algorithm are set as follows: $\gamma =0.99$, $\tau =0.005$, $\epsilon =0.2$, $c=0.5$, $d=2$.

$$\left\{\begin{array}{l}{f}_u\left(u,v,r\right)=X_{uu}{u}^2+X_{vv}{v}^2+X_{vr}vr+X_{rr}{\mathrm{r}}^2\\ f_v\left(u,v,r\right)=Y_{v}v+Y_{r}r+Y_{vv}v\left|v\right|+Y_{vr}v\left|r\right|+Y_{rr}r\left|r\right|\\ f_r\left(u,v,r\right)=N_{v}v+N_{r}r+N_{rr}r\left|r\right|+N_{vvr}{v}^{2}r+N_{vrr}v{r}^2\end{array}\right.$$

(43)

$$\left\{\begin{array}{ll}\hfill m_x=& {\displaystyle \frac{m}{100}}[0.398+11.97C_b\left(1+3.73{\displaystyle \frac{d}{B}}\right)-2.89C_b{\displaystyle \frac{L}{B}}\left(1+1.13{\displaystyle \frac{d}{B}}\right)\hfill \\ & +0.175C_b{\left({\displaystyle \frac{L}{B}}\right)}^2\left(1+0.541{\displaystyle \frac{d}{B}}\right)-1.107{\displaystyle \frac{L}{B}}{\displaystyle \frac{d}{B}}]\hfill \\ \hfill m_y=& m[0.882-0.54C_b\left(1-1.6{\displaystyle \frac{d}{B}}\right)-0.156{\displaystyle \frac{L}{B}}\left(1-0.673C_b\right)\hfill \\ & +0.826{\displaystyle \frac{d}{B}}{\displaystyle \frac{L}{B}}\left(1-0.678{\displaystyle \frac{d}{B}}\right)-0.638{\displaystyle \frac{d}{B}}{\displaystyle \frac{L}{B}}\left(1-0.669{\displaystyle \frac{d}{B}}\right)]\hfill \\ \hfill J_{zz}/I_{zz}=& 16{\left\{\left[33-76.85C_b\left(1-0.784C_b\right)+3.43{\displaystyle \frac{L}{B}}\left(1-0.63C_b\right)\right]/100\right\}}^2\hfill \end{array}\right.$$

(44)

5.1. Maneuverability Analysis

Simulation tests are conducted to evaluate the operability of the constructed model. The impacts of thrust and torque on the vessel’s motion state are tested, as these are key factors for achieving trajectory tracking control of an underactuated vessel.

Under the condition that both the initial velocity and torque are 0, the thrust ${\tau }_u$ is input as follows: 10 N, 50 N, 100 N, 200 N, 500 N, 1000 N. The simulation results show that as the thrust increases, the vessel’s forward speed $u$ also increases accordingly, as shown in Figure 6a. As the thrust increases, the forward speed of the vessel gradually approaches a stable value. Specifically, with thrust inputs of 10 N, 50 N, 100 N, 200 N, 500 N, and 1000 N, the vessel’s final forward velocities stabilize at 0.15 m/s, 0.25 m/s, 0.5 m/s, 1.1 m/s, and 1.55 m/s, respectively. Under the condition of an initial velocity of 0 and a thrust of 100 N, the thrust inputs are as follows: −100 N, −50 N, 50 N, and 100 N, with the simulation results shown in Figure 6b. Specifically, when the torque is ±50 N, the vessel’s turning radius is approximately 5 m, and when the torque is ±100 N, the turning radius is approximately 14 m. Moreover, the vessel’s rotation direction is opposite to the sign of the applied torque. When the torque is positive, the vessel rotates counterclockwise. When the torque is negative, the vessel rotates clockwise. The simulation results lead to the conclusion that the vessel’s motion control can be achieved by adjusting the magnitude of the thrust and torque. The established vessel model is consistent with the actual operational behavior of vessels, effectively simulating the vessel’s motion state under different control inputs and providing a reliable basis for trajectory tracking control simulations.

5.2. Sine Wave Trajectory Tracking Simulation

This section presents the trajectory tracking control simulations for a sine trajectory. The equation of the sine trajectory to be tracked is $y=200\sin \left(0.01x\right)$. Based on Assumption 2, the disturbance is set as shown in Equation (45):

$$\left\{\begin{array}{l}{\tau }_{wu}=100\sin \left(0.1t\right)\\ {\tau }_{wv}=100\sin \left(0.1t\right)\\ {\tau }_{wr}=100\sin \left(0.1t\right)\end{array}\right.$$

(45)

Let the initial position of the vessel be $\left(x,y,\psi \right)=\left(20m,50m,pi/4rad\right)$, and the initial velocity of the vessel be $\upsilon =\left(u,v,r\right)=\left(0,0,0\right)$. The action space range is set to ${\lambda }_1\in \left[0.1,10\right]$ and ${\lambda }_2\in \left[0.1,10\right]$, with the other parameters set to $k_1=k_2={\epsilon }_1={\epsilon }_2=k_{\omega }=1$. To verify the impact of the drift angle on the vessel’s heading control, a series of simulation experiments are conducted, and the results are shown in Figure 7. In the early stages of the vessel’s operation, the influence of the drift angle on the reference angle is significant. The maximum error between the reference angle calculated through drift angle filtering and the reference angle obtained by removing the drift angle can reach 0.4 rad. Furthermore, it can also be observed that when the trajectory curvature is large, the method of calculating the reference angle by removing the drift angle introduces errors compared with the method that calculates the reference angle through drift angle filtering. Therefore, it is particularly important to use filtering techniques to smooth the drift angle. A low-pass filter is used to smooth the drift angle, and the desired heading angle of the vessel varies smoothly. With the presence of the drift angle, higher control accuracy is achieved. In summary, the drift angle has a significant impact on the trajectory tracking performance of underactuated vessels, especially in the initial heading control phase. Therefore, it is necessary to use appropriate filtering techniques to process the drift angle, improving the vessel’s heading accuracy and trajectory tracking stability. Considering the drift angle filtering, the vessel’s trajectory tracking simulation results are shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. Figure 8a shows the sliding mode controller parameters obtained through the TD3 algorithm.

When the SMC parameters are set to ${\lambda }_1=1$ and ${\lambda }_2=1$, the actual operating trajectory of the SMC is shown by the orange curve in Figure 8b. The actual operating trajectory of the sliding mode controller optimized by the TD3 algorithm (SMC-TD3) in real time is shown by the red curve in Figure 8b. Comparing both trajectories with the desired trajectory, the results indicate that the SMC-TD3 performs better in tracking the desired trajectory. It is able to track the desired path more quickly and with smaller tracking errors. Figure 9 presents a comparison of the longitudinal tracking error and the lateral tracking error in the SMC and the SMC-TD3. It can be observed that the longitudinal tracking error of the SMC stabilizes within the range of [−3, 3], while the longitudinal tracking error of the SMC-TD3 converges to 0. This indicates that the SMC-TD3 achieves smaller longitudinal tracking errors. Both methods exhibit bounded lateral tracking errors. The lateral tracking error of the SMC stabilizes within the range of [−15, −12.5], whereas the lateral tracking error of the SMC-TD3 stabilizes within the range of [−1, 1], showing that the SMC-TD3 achieves smaller lateral tracking errors. Figure 10 shows a comparison of the speed tracking error and the yaw tracking error in the SMC and the SMC-TD3. The speed tracking errors of both methods converge to 0. Both methods also exhibit convergence of the yaw tracking error. The SMC’s yaw tracking error converges to −0.8, while the SMC-TD3’s yaw tracking error converges to 0. Comparing the two methods, the SMC-TD3 demonstrates superior overall control performance. Figure 11 illustrates the speed control performance of both the SMC and the SMC-TD3. Both methods are capable of achieving variable-speed tracking control for the underactuated vessel. Figure 12 displays the control force variations for both controllers. While both methods ensure smooth control of the underactuated vessel, the SMC-TD3 exhibits smoother control. Therefore, the SMC-TD3 is better suited to meet the practical engineering control requirements.

5.3. Circular Trajectory Tracking Simulation

This section describes the trajectory tracking control simulation performed for a circular trajectory. The equation of the circular trajectory to be tracked is given by Equation (46), and the external disturbance is set as shown in Equation (45):

$$\left\{\begin{array}{l}{x}_d=300\cos \left(0.005t\right)\\ y_d=300\sin \left(0.005t\right)\end{array}\right.$$

(46)

Let the initial position of the vessel be $\left(x,y,\psi \right)=\left(0m,0m,0rad\right)$, and the initial vessel speed be $\upsilon =\left(u,v,r\right)=\left(0,0,0\right)$ m/s. Set the action space as ${\lambda }_1\in \left[0.1,10\right]$ and, ${\lambda }_2\in \left[0.1,10\right]$, with the other parameters as $k_1=k_2={\epsilon }_1={\epsilon }_2=k_{\omega }=1$. The vessel’s trajectory tracking results are shown in Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17. The SMC parameters obtained through the TD3 algorithm are shown in Figure 13a.

When the sliding mode control parameters are set to ${\lambda }_1=1$ and ${\lambda }_2=1$, the actual operating trajectory of the SMC is shown by the orange line in Figure 13b. The actual operating trajectory of the SMC-TD3 is shown by the red line in Figure 13b. A comparison of the two trajectories with the desired trajectory shows that SMC-TD3 performs better in trajectory tracking, enabling faster convergence to the desired path. Figure 14 presents the comparison of the longitudinal tracking error and lateral tracking error in the SMC and the SMC-TD3. It can be observed that both methods lead to the longitudinal tracking error and lateral tracking error approaching zero. However, the SMC-TD3 exhibits a smoother tracking performance. Figure 15 shows a comparison of the speed tracking error and yaw tracking error in the SMC and the SMC-TD3. Both methods exhibit bounded speed tracking errors. The SMC’s speed tracking error converges to −0.1, while the SMC-TD3’s speed tracking error converges to 0. Both methods show bounded yaw tracking errors. The SMC’s yaw tracking error converges to 0, while the SMC-TD3’s yaw tracking error converges to −0.1. However, the SMC-TD3 demonstrates a faster dynamic response. Figure 16 illustrates the speed control performance of both the SMC and the SMC-TD3. Both methods are capable of achieving variable-speed tracking control for the underactuated vessel. Figure 17 presents the control force variations for the SMC and the SMC-TD3. It can be observed that the SMC-TD3 exhibits smoother control compared with the SMC method. Furthermore, the control force of the SMC-TD3 is smoother, better meeting the practical engineering control requirements.

6. Conclusions

This paper investigates the problem of variable-speed trajectory tracking control for underactuated vessels. First, considering the hydrodynamic effects on the underactuated ship, a 3-DOF motion model of the underactuated vessel is established. Next, based on the motion model, two sliding mode controllers are designed: a speed sliding mode controller and a heading sliding mode controller. For speed control, the desired speed is designed to follow an S-curve acceleration/deceleration process in accordance with the variable-speed tracking control requirements of the underactuated vessel. For heading control, the LADLOS method is used to dynamically calculate the desired heading angle of the underactuated vessel. To address external composite disturbances, an ESO is designed to observe and compensate for the disturbances in real-time. An adaptive law is also designed to estimate the error bounds, and the global stability of the system is proven using the Lyapunov method. Third, to further optimize the performance of the sliding mode controller, the TD3 algorithm is introduced for parameter tuning. Additionally, due to the dynamic characteristics of the underactuated vessel’s propulsion system, the phenomenon of sudden changes in the drift angle often occurs when the vessel starts, and this drift angle can affect the heading control. To address the drift angle issue, a first-order inertial element filter is introduced to smooth the drift angle data. Numerical simulation results demonstrate that the proposed method significantly improves the trajectory tracking accuracy and dynamic response speed of the underactuated vessel. Specifically, for the sinusoidal trajectory with an amplitude of 200 m and a frequency of 0.01, numerical results show that the proposed method achieves convergence of the longitudinal tracking error to zero, while the lateral tracking error remains stable within 1 m. For the circular trajectory with a radius of 300 m, the numerical results indicate that both the longitudinal and lateral tracking errors are stabilized within 1 m. Compared with the fixed-value sliding mode controller, the proposed method demonstrates superior trajectory tracking accuracy and smoother control performance.