Research on Path-Following Technology of a Single-Outboard-Motor Unmanned Surface Vehicle Based on Deep Reinforcement Learning and Model Predictive Control Algorithm

Abstract

Path following is one of the key technologies for unmanned surface vehicles (USVs). This paper proposes a path-tracking control method for a single-outboard-motor USV based on a Deep Deterministic Policy Gradient (DDPG) algorithm and model predictive control (MPC) algorithm. Initially, the motion model and outboard motor model of the USV are analyzed. Subsequently, simulation and real ship experiments provide a comprehensive performance comparison between the proposed DDPG-MPC method and the traditional ALOS-PID method. The results indicate that for straight path tracking, the DDPG-MPC algorithm achieves 37% and 21% reductions in the average cross error and heading angle error, respectively, compared to the ALOS-PID algorithm. The real ship experiments further validate the DDPG-MPC algorithm’s advantages in real-world environments. Specifically, under disturbances like wind, waves, and currents, the maximum cross error of the DDPG-MPC algorithm is one-third of the ALOS-PID algorithm. Additionally, the DDPG-MPC algorithm sustains a higher and more stable longitudinal velocity over extended periods, while the ALOS-PID algorithm shows greater instability and variability. Overall, the findings confirm the feasibility and effectiveness of the proposed approach, highlighting its potential for enhancing path-tracking control performance in single-outboard-motor USVs.

1. Introduction

Unmanned surface vehicles (USVs) are increasingly valued for their intelligence, maneuverability, and adaptability in various applications such as water patrols, hydrological surveys, maritime operations, rescue missions, and military usage [1,2,3,4]. Path following, a crucial technology for the autonomous navigation of USVs, focuses on maximizing efficiency and ensuring safety by maintaining adherence to a planned path within accepted error margins, as opposed to trajectory tracking, which involves strict timing at specific points [5,6,7]. Path-following technology principally involves path guidance and heading control components [8,9,10]. Among the many methods studied, line of sight (LOS) and vector field (VF) are prevalent [11,12]. These methods facilitate the control of underactuated vessels by translating the path-following challenge into a heading management task using well-elaborated guidance laws.

Several advancements have enhanced the LOS method’s efficacy. Wang et al. [13] introduced a Predictor-Based Fixed-Time LOS guidance law tailored for USVs navigating in unknown disturbances, featuring global fixed-time convergence that guarantees quick stabilization and robust performance, regardless of the initial conditions. Liu et al. [14] developed an adaptive LOS (ALOS) that finely adjusts to changes in sideslip, maintaining trajectory precision under varying conditions. Zhou et al. [15] enhanced this approach with their integral–differential LOS (IDLOS), which incorporates adaptive elements to speed convergence and minimize overshoot, optimizing response times and control precision. Further advancements came from Tong et al. [16], who proposed a finite-time, error-constrained LOS suitable for navigating curved paths under uncertain circumstances, expanding the versatility of LOS applications. Complementarily, Fossen et al. [17] crafted a nonlinear ALOS method that effectively counters environmental drift, such as wind and currents, ensuring uniform semi-global exponential stability (USGES) under linear trajectory conditions. In comparative analysis, ALOS guidance [16] stands out in dynamic environments due to its rapid adaptability and robust stability, while the ALOS law [17] offers finite-time convergence and cross-track error constraints, ideal for underactuated systems facing unpredictable challenges. On the other hand, Wang’s Predictor-Based Fixed-Time LOS [13] provides a resilient framework ensuring swift and precise control. Nonetheless, despite higher complexity and computational demands, these advanced LOS methods excel in scenarios requiring an immediate response and exceptional accuracy, with ALOS proving particularly effective in resource-limited settings. Hong et al. [18] refined LOS by integrating a variable “acceptance circle” radius to enhance flexibility during waypoint transitions, offering a significant improvement over traditional methods. In contrast, VF methods, developed initially for micro aerial vehicles, have been adapted for USVs [19,20,21], with comparative studies suggesting these methods provide excellent path adherence but may cause yaw instability [22]. Collectively, these adaptations balance robustness, convergence speed, and computational efficiency, pushing forward the development of unmanned ship path-following technology. However, the current research landscape in this domain is not without its limitations. The existing research and practical application of unmanned surface ship heading guidance laws based on the LOS method and the VF method have inherent defects [23]. The LOS method grapples with the challenge of striking a balance between the response time and steady-state error in forward distance settings [24]. Moreover, when transitioning between different desired paths, the convergence speed of the associated controller cannot be assured [25]. Conversely, the VF method is plagued by rudder oscillation in the horizontal direction [26].

Recent research [27,28,29,30,31] has newly incorporated deep reinforcement learning into USV path-following strategies, highlighting novel advancements in control methodologies. Zhao et al. [6] developed an enhanced deep Q-network (DQN) for the simplified control of three-degree-of-freedom USV models, unveiling significant potential for adaptive learning in maritime navigation scenarios. Zhong et al. [32] introduced a novel control approach utilizing a Deep Deterministic Policy Gradient (DDPG) algorithm designed around a composite state space and dynamic reward function, aimed at reducing the impacts of large inertia and state lag on RL agents’ training. Han et al. [33] has developed a system using distributed deep reinforcement learning and adaptive neural networks for straight-line path tracking and formation control in USVs, employing radial basis function neural networks (RBF NNs) to approximate the hydrodynamic forces and unknown external disturbances affecting the vehicles, demonstrating high precision in tracking control through simulation results. Furthermore, Zhu et al. [34] has proposed an intelligent path-tracking control strategy for underactuated USVs with unknown model parameters, combining LOS guidance and Q-learning to enhance path-following accuracy under uncertain dynamics. These advances underscore the growing versatility and challenge of applying deep reinforcement learning, showcasing its potential in achieving sophisticated maritime navigational tasks while highlighting inherent limitations such as computational complexity, data requirements, and response time to dynamic environmental changes.

However, a considerable part of the research still relies on differential twin propeller thrusters configured at the stern and cannot be directly converted to a single-outboard-motor setup [27,28,29,30,31], which highlights the need for specific solutions for different propulsion types. This paper introduces a novel path-following control method for a single-outboard-motor USV that combines deep reinforcement learning with model predictive control. This approach addresses the shortcomings of traditional LOS and VF approaches and is tailored to the specific propulsion characteristics of a single-outboard-motor USV. By implementing deep reinforcement learning for path tracking in a single-outboard-motor USV mission, we significantly improve the path-following capabilities. At the same time, our reinforcement learning method DDPG-MPC also improves the versatility of reinforcement learning-based path tracking in complex ocean environments, contributing innovation to autonomous USV path tracking. By employing this cutting-edge strategy, we extend the practical applicability of reinforcement learning to different operating conditions and improve the resilience and adaptability of USVs to a wider range of environmental challenges.

2. Kinematic Modeling

2.1. Kinematic Modeling of USV

The kinematic modeling of USVs involves multiple factors such as thruster forces and moments, viscous hydrodynamics and external disturbances, and has highly nonlinear and variable coupling characteristics, making it a very complex process [35]. It needs to balance two aspects. Firstly, in order to avoid the model being too complex and unable to guarantee real-time performance in subsequent control operations, reasonable assumptions and simplifications need to be made during modeling. The second is to avoid neglecting some key system characteristics of the model when simplifying it.

The kinematic modeling of USVs generally involves two coordinate systems, namely the world coordinate system $O{X}_w{Y}_w{Z}_w$ and the body-fixed coordinate system $O{X}_b{Y}_b{Z}_b$ [36], as shown in Figure 1. In this paper, the world coordinate system is set as the NED coordinate system. In this system, the six-degrees-of-freedom (DOF) motion vector of the USV can be represented as $\eta ={\left[x,y,z,\phi ,\theta ,\psi \right]}^T$, where $x$, $y$, and $z$ denote displacements along the $X_w$, $Y_w$, and $Z_w$ axes, respectively, and $\phi$, $\theta$, and $\psi$ represent the roll, pitch, and yaw angles, corresponding to rotations around the $X_w$, $Y_w$, and $Z_w$ axes.

In the body-fixed coordinate system, this paper defines the origin $O$ as the center of gravity of the USV, $X_b$ as the forward direction, $Y_b$ as the starboard direction, and $Z_b$ as the downward direction of the USV. The rotations around the $X_b$, $Y_b$, and $Z_b$ axes produce roll rate $p$, pitch rate $q$, and yaw rate $r$, respectively. The translations along the $X_b$, $Y_b$, and $Z_b$ axes generate surge speed $u$, sway speed $v$, and heave speed $w$. In the body-fixed coordinate system, the velocity of the USV can be represented by the vector $\upsilon ={\left[u,v,w,p,q,r\right]}^T$.

In the study of USV dynamics, more emphasis is placed on the planar motion state of the USV. Therefore, the motions in roll, pitch, and heave are generally ignored. As a result, the motion of the USV can be represented by a three-degree-of-freedom motion model based on Lagrangian mechanics, as shown in Equation (1).

$$M\dot{\upsilon }+C\left(\upsilon \right)\upsilon +D\left(\upsilon \right)\upsilon =\tau$$

(1)

where $M$ represents the inertia matrix, $C\left(\upsilon \right)$ represents the Coriolis and centripetal force matrix, and $D\left(\upsilon \right)$ represents the hydrodynamic damping matrix. The above terms can be calculated using the following formulas:

$$M=\left[\begin{array}{ccc}m-X_{\dot{u}}& 0& 0\\ 0& m-Y_{\dot{v}}& m{x}_g-Y_{\dot{r}}\\ 0& m{x}_g-Y_{\dot{r}}& I_z-N_{\dot{r}}\end{array}\right]$$

(2)

$$C\left(\upsilon \right)=\left[\begin{array}{ccc}0& 0& -\left(m-Y_{\dot{v}}\right)v-\left(m{x}_g-Y_{\dot{r}}\right)r\\ 0& 0& \left(m-X_{\dot{u}}\right)u\\ \left(m-Y_{\dot{v}}\right)v+\left(m{x}_g-Y_{\dot{r}}\right)r& -\left(m-X_{\dot{u}}\right)u& 0\end{array}\right]$$

(3)

$$D\left(\upsilon \right)=\left[\begin{array}{ccc}-X_u& 0& 0\\ 0& -Y_v& -Y_r\\ 0& -N_v& -N_r\end{array}\right]$$

(4)

where $X_{\dot{u}}$, $Y_{\dot{v}}$, $Y_{\dot{r}}$, $N_{\dot{r}}$, $X_u$, $Y_v$, $Y_r$, $N_v$, and $N_r$ are the hydrodynamic coefficients of the USV.

$\upsilon$ represents the three-degree-of-freedom velocity vector of the USV in the body-fixed coordinate system as shown in Equation (5).

$$\upsilon ={\left[\begin{array}{ccc}u& v& r\end{array}\right]}^T$$

(5)

$\tau$ represents the force exerted by the propeller on the USV as shown in Equation (6).

$$\tau ={\left[\begin{array}{ccc}{\tau }_u& {\tau }_v& {\tau }_r\end{array}\right]}^T$$

(6)

Among them, ${\tau }_u$ is the longitudinal thrust, ${\tau }_v$ is the lateral thrust, and ${\tau }_r$ is the yaw moment. These three quantities can be calculated using Equation (7).

$$\{\begin{array}{l}{\tau }_u=T\cos \delta \\ {\tau }_v=T\sin \delta \\ {\tau }_r=x_{l}T\sin \delta \end{array}$$

(7)

where $T$ represents the thrust of the outboard motor, $\delta$ represents the thrust angle of the outboard motor, and $x_l$ represents the distance from the rotation axis of the outboard motor to the center of rotation of the USV.

The USV propelled by a single outboard engine only has two control variables: the thrust angle and thrust of the outboard engine, but the USV needs to control the three degrees of freedom, which belongs to the underactuated system. Based on the underactuated characteristics of a single-outboard-motor USV, the following definitions are made in the paper [37,38]: $m_{11}=m-X_{\dot{u}}$, $m_{22}=m-Y_{\dot{v}}$, $m_{23}=m{x}_g-Y_{\dot{r}}$, $m_{32}=m{x}_g-N_{\dot{v}}$, $m_{33}=I_z-N_{\dot{r}}$, $d_{11}=-X_u$, $d_{22}=-Y_v$, $d_{23}=-Y_r$, $d_{32}=-N_v$, and $d_{33}=-N_r$. So, Equation (1) can be transformed into Equation (8).

$$\left[\begin{array}{c}{m}_{11}\dot{u}\\ m_{22}\dot{v}+m_{23}\dot{r}\\ m_{32}\dot{v}+m_{33}\dot{r}\end{array}\right]+\left[\begin{array}{c}{d}_{11}u-m_{22}vr-m_{23}{r}^2\\ d_{22}v-d_{23}r+m_{11}ur\\ m_{22}vu+m_{23}ru+d_{32}v-m_{11}uv+d_{33}r\end{array}\right]=\left[\begin{array}{c}{\tau }_u\\ 0\\ {\tau }_r\end{array}\right]$$

(8)

Assuming that the center of gravity of the hull coincides with the geometric center and that the USV is symmetric in the fore–aft and port–starboard directions, we have $x_g=0$, $Y_{\dot{r}}=0$, $N_{\dot{v}}=0$, $Y_r=0$, and $N_v=0$, allowing Equation (8) to be simplified to Equation (9).

$$\{\begin{array}{l}\dot{u}={\displaystyle \frac{m_{22}}{m_{11}}}vr-{\displaystyle \frac{d_{11}}{m_{11}}}u+{\displaystyle \frac{1}{m_{11}}}{\tau }_u\\ \dot{v}=-{\displaystyle \frac{m_{11}}{m_{22}}}ur-{\displaystyle \frac{d_{22}}{m_{22}}}v\\ \dot{r}={\displaystyle \frac{m_{11}-m_{22}}{m_{33}}}uv-{\displaystyle \frac{d_{33}}{m_{33}}}r+{\displaystyle \frac{1}{m_{33}}}{\tau }_r\end{array}$$

(9)

2.2. Kinematic Modeling of the Outboard Motor

As can be seen from Equation (7), the longitudinal thrust, lateral thrust, and steering moment exerted by the outboard motor on the USV are determined by the thrust $T$ and thrust angle $\delta$ of the outboard motor. Therefore, it is necessary to model the thrust model and servo model of the single outboard motor.

2.2.1. The Outboard Motor Model

The outboard motor has the advantage of being easy to retract, allowing it to be stowed away when an amphibious USV comes ashore, thereby avoiding interference with the ground. This makes it suitable as a propulsion system for amphibious USVs. The thrust generated by the outboard motor is determined by the propeller’s rotational speed, and its thrust mathematical model is given by Equation (10) [39]:

$$T=(1-t_P)\rho n^2{D}_P^4{K}_T(J_p)$$

(10)

where $\rho$ represents the seawater density, $n$ denotes the propeller rotational speed, and $D_P$ indicates the propeller diameter. Additionally, $t_P$ stands for the reduction coefficient, $K_T(J_p)$ represents the thrust coefficient, and $J_p$ indicates the advance coefficient. The calculation methods for these coefficients are described in detail below.

The reduction coefficient $t_P$ can be approximated using the Hallrot empirical formula [40], which is suitable for single-propeller propulsion systems, as shown in Equation (11).

$$t_P=0.001979{\displaystyle \frac{L}{\left(B-B{C}_{p1}\right)}}+1.0585C_{10}-0.000524-0.1418{\displaystyle \frac{D_P^2}{Bd}}+0.0015C_{stern}$$

(11)

where $L$ represents the waterline length, $B$ denotes the beam of the vessel, and $d$ indicates the draft depth. And $C_{p1}$ can be calculated using Equation (12).

$$C_{p1}=1.45C_P-0.315-0.225L_{cb}$$

(12)

where $C_P$ represents the prismatic coefficient and $L_{cb}$ denotes the position of the forward half-length center of buoyancy.

$C_{10}$ represents a coefficient related to the length-to-beam ratio, which can be calculated using Equation (13). $C_{stern}$ denotes a coefficient related to the stern shape of the vessel, which can be determined using Equation (14).

$$\{\begin{array}{cc}{C}_{10}=B/L& L/B>5.2\\ C_{10}=0.25-{\displaystyle \frac{0.003328402}{B/L-0.134615385}}& L/B\le 5.2\end{array}$$

(13)

$$\{\begin{array}{l}\begin{array}{cc}{C}_{stern}=-10& V-ShapedHull\end{array}\\ \begin{array}{cc}{C}_{stern}=0& ConventionalHull\end{array}\\ \begin{array}{cc}{C}_{stern}=+10& U-ShapedHullwithaHognerStern\end{array}\end{array}$$

(14)

The thrust coefficient $K_T$ can be calculated using multiple regression analysis [41], as illustrated in Equation (15).

$$K_T={\displaystyle \sum _{i=0}^{n1}{\displaystyle \sum _{j=0}^{n2}{A}_{ij}{\left(\frac{P}{D}\right)}^i{J}_P^j}}$$

(15)

where $J_P={\displaystyle \frac{\left(1-w_p\right)u}{n{D}_p}}$ represents the advance coefficient, and $w_p$ denotes the wake coefficient, which can be determined using the Bappmiller formula, as shown in Equation (16).

$$w_p=0.165C_b^x\sqrt{{\displaystyle \frac{\sqrt[3]{\nabla }}{D_p}}}-\Delta w$$

(16)

where $\nabla$ represents the displacement volume, $x$ is a coefficient related to the propeller installation position, as shown in Equation (17), and $\Delta w$ represents the correction value for the wake coefficient, which can be expressed using Equation (18).

$$\{\begin{array}{cc}x=1& PropellerPositionedontheCenterline\\ x=2& PropellerPositionedontheSide\end{array}$$

(17)

$$\Delta w=\{\begin{array}{ll}0.1\left(F_n-0.2\right),& F_n>0.2\\ 0,& F_n\le 0.2\end{array}$$

(18)

where $F_n=V/\sqrt{gL}$ is the Froude number and $g$ is the gravitational acceleration.

2.2.2. The Servo Model

The outboard engine servo model can be represented as a first-order inertial component, which aligns with the approach by M. T. Meziou, J. Ghommam, and N. Derbel in their paper [42], as shown in Equation (19).

$$\dot{\delta }=-{\displaystyle \frac{1}{T_d}}\delta +{\displaystyle \frac{1}{T_d}}{\delta }_d$$

(19)

where $T_d$ represents the time coefficient, $\delta$ denotes the current propulsion angle, and ${\delta }_d$ indicates the target propulsion angle.

3. Path-Following Control Algorithm

This paper presents a design strategy that divides the path-following control framework of a single-outboard-motor USV into two key components: path-following guidance law and heading control. To overcome the limitations of conventional algorithms, it introduces a novel method integrating DDPG with MPC, as illustrated in Figure 2. This method computes the desired heading angle ${\psi }_d$ from a pre-planned path and employs an MPC-based heading controller to adjust the USV’s heading angle, enabling effective path following.

3.1. Path-Following Guidance Law Based on DDPG Algorithm

3.1.1. Design of the Markov Decision Process

In reinforcement learning, constructing a Markov decision model is a key element. The algorithm’s efficiency and convergence heavily rely on accurately defining the state space, action space, and reward function [43]. Hence, to develop a high-performance guidance law for USV path following, it is crucial to design a suitable Markov decision model tailored to the specific requirements of single-outboard-motor USVs.

The state space is defined as follows:

$$S=[u,r,\beta ,\dot{\beta },\epsilon ,\dot{\epsilon },\Delta {\psi }_d^p]$$

(20)

where $u$ represents the longitudinal velocity of the USV, $r$ denotes the USV’s pitch rate, $\beta$ is the deviation between the USV’s heading angle and the desired path angle, $\epsilon$ is the cross error between the desired path and the USV’s real-time position, and $\Delta {\psi }_d^p$ is the action taken in the previous time step, which is the desired change in the heading angle of the USV.

In path-following control, greater emphasis is placed on the longitudinal velocity of the USV, while lateral drift velocity is often overlooked. To ensure the path-following guidance law can generate a real-time optimal desired heading angle based on the USV’s longitudinal velocity $u$, this study incorporates the longitudinal velocity into the state space. The state space includes the deviation $\beta$ between the current heading angle and the desired path angle, as well as the cross-track error between the desired path and the USV’s real-time position, reflecting the discrepancy between the actual and desired positions. Furthermore, to evaluate if the USV’s state changes align with the trend of reducing tracking errors, the state space also integrates the rate of change in the deviation between the heading and path angles $\dot{\beta }$, the rate of change in the cross-track error, and the action taken in the previous time step $\Delta {\psi }_d^p$.

The action space is defined as follows:

$$A=[\Delta {\psi }_d]$$

(21)

$\Delta {\psi }_d$ represents the change in the desired heading angle of the USV, and $\Delta {\psi }_d\in \left[-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}\right]$ is such that the USV’s desired heading angle ${\psi }_d$ can be calculated using formula ${\psi }_d=\psi +\Delta {\psi }_d$, where $\psi$ denotes the actual heading angle of the USV.

The reward function is defined as follows:

$$r=\{\begin{array}{l}{r}_{\epsilon }+r_{\beta }+r_{\Delta {\psi }_d}\left|\beta \right|<{\displaystyle \frac{\pi }{2}}\\ r_{\beta }\left|\beta \right|\ge {\displaystyle \frac{\pi }{2}}\end{array}$$

(22)

The definitions of each term are as follows:

$$r_{\epsilon }=e^{-k_{\epsilon }\left|\epsilon \right|}$$

(23)

$$r_{\beta }=e^{-k_{\beta }\left|\beta \right|}$$

(24)

$$r_{\Delta {\psi }_d}=e^{-k_{\Delta {\psi }_d}\left|\Delta {\psi }_d\right|}$$

(25)

where $k_{\epsilon }$, $k_{\beta }$, and $k_{\Delta {\psi }_d}$ are constants, calibrated through multiple experiments, and $k_{\epsilon }=0.7$, $k_{\beta }=4.82$, and $k_{\Delta {\psi }_d}=0.3$.

Equation (23) describes the reward function $r_{\epsilon }$ for the cross error $\epsilon$ between the actual position of the unmanned boat and the desired path. As shown in Figure 3a, the function exhibits the following characteristics: when the cross error $\epsilon$ is 0, the value $r_{\epsilon }$ reaches 1; as the value of $\left|\epsilon \right|$ increases, the reward value $r_{\epsilon }$ decreases rapidly. However, when the error $\left|\epsilon \right|$ increases to 8 m (approximately the length of a boat), the reward value $r_{\epsilon }$ is almost close to 0. This design encourages the agent to actively minimize the cross error during the interaction with the environment in the form of an exponential function, thereby achieving more accurate path tracking.

Equation (24) is the reward function $r_{\beta }$ related to the deviation $\beta$ between the USV’s heading angle and the desired path angle. The graph of this function is shown in Figure 3b. When the angle deviation $\beta$ is 0, the value of $r_{\beta }$ reaches its maximum of 1. To prevent the USV from heading away from the desired path, that is, when $\left|\beta \right|>{\displaystyle \frac{\pi }{2}}$ occurs, the value of $r_{\beta }$ remains near 0.

Equation (25) defines the reward function $r_{\Delta {\psi }_d}$ associated with changes in the USV’s desired heading angle $\Delta {\psi }_d$. Illustrated in Figure 3c, this function is carefully designed to minimize abrupt alterations in the desired heading direction. By doing so, it ensures a smooth and continuous adjustment process, enhancing the stability and precision of the USV’s navigation path. This approach ultimately contributes to more efficient and reliable path tracking.

Equation (22) is the final reward function $r$ after weighting the individual reward functions. When $\left|\beta \right|\ge {\displaystyle \frac{\pi }{2}}$ occurs, it indicates that the deviation between the USV’s heading angle and the desired path angle is large, and the heading angle should be adjusted as soon as possible to prevent the deviation between the USV’s actual position and the desired tracking path from further increasing. Therefore, $r=r_{\beta }$ is chosen; when $\left|\beta \right|<{\displaystyle \frac{\pi }{2}}$ is the case, all three reward functions are considered, and $r=r_{\epsilon }+r_{\Delta {\psi }_d}+r_{\beta }$ is chosen.

3.1.2. Implementation of the DDPG Algorithm

The DDPG algorithm is a type of deep reinforcement learning method built on the Actor–Critic framework [44]. Its structural principles are shown in Figure 4.

The DDPG agent consists of a policy network and a value network. The policy network, referred to as the Actor Network, takes the state vector $s_t$ as input and produces the action vector ${\mu }_{\theta }\left(s_t\right)$ following the action policy, with $\theta$ representing the weights of the Actor Network. Structurally, the Actor Network includes an input layer, two hidden layers, and an output layer. The input layer’s size aligns with the state vector’s dimension, while the output layer matches the action vector’s dimension. ReLU is used as the activation function in the hidden layers, and the output layer applies the tanh function to normalize the action values, which are then scaled to the actual action range.

The value network, known as the Critic Network, takes the state vector $s_t$ and the action vector $a_t$ generated by the Actor Network as inputs and produces the action value $Q_{\omega }(s_t,a_t)$, with $\omega$ representing the weights of the Critic Network. The state vector is processed through two hidden layers, while the action vector passes through one hidden layer. These outputs are then combined via tensor addition, followed by an additional hidden layer and an output layer. ReLU serves as the activation function across all hidden layers. The training process of the DDPG agent includes the following steps:

➀

Randomly initialize the Actor Network $\mu (s|\theta )$ and Critic Network $Q(s,a|\omega )$; initialize the target network’s actor ${\omega }^{\prime }\leftarrow \omega$ and ${\theta }^{\prime }\leftarrow \theta$; initialize the experience replay buffer $R$.

➁

Begin a new episode by initializing the USV’s state randomly in the environment as the initial state $s_1$. At each time step, the current state vector $s_t$ and action policy ${\mu }_{\theta }\left(s_t\right)$ are combined with exploration noise $N_t$ to produce the action vector $a_t={\mu }_{\theta }(s_t)+N_t$, which defines the USV’s desired heading angle. The heading controller then tracks this angle, and the environment provides the updated state vector $s_{t+1}$. Meanwhile, rewards are computed using environmental data, such as cross-track and heading angle errors. Interaction data $(s_t,a_t,r_t,s_{t+1})$ generated at each time step are saved in the experience replay buffer $R$.

➂

Randomly sample N interaction data from the experience replay buffer $R$ to update the networks. Update the Critic Network by minimizing the loss function $L={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_i{(y_i-Q_{\omega }(s_i,a_i))}^2}$, where $y_i=r_i+\gamma {Q^{\prime }}_{{\omega }^{\prime }}(s_{i+1},{{\mu }^{\prime }}_{{\theta }^{\prime }}(s_{i+1}))$ represents the target value output by the target Critic Network. Update the Actor Network based on the sampled gradients, with the gradient formula given by the following:

$${\nabla }_{\theta }\mu {|}_{s_t}\approx {\displaystyle \frac{1}{N}}{\displaystyle \sum _i{\nabla }_a{Q}_{\omega }(s,a){|}_{s=s_t,a=\mu (s_t)}{\nabla }_{\theta }{\mu }_{\theta }(s){|}_{s_t}}$$

(26)

➃

Perform soft updates on the parameters of the target Critic Network and target Actor Network, with the update formula as follows:

$$\{\begin{array}{l}{\omega }^{\prime }\leftarrow \tau \omega +(1-\tau ){\omega }^{\prime }\\ {\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }\end{array}$$

(27)

with $\tau \in \left(0,1\right)$.

➄

Return to step 2 to continue learning at the next time step. When reset conditions are met or learning exceeds the set time steps, start a new episode.

3.2. Heading Controller Design Based on MPC Algorithm

Assuming the longitudinal velocity $u$ of the USV remains constant, the linearized model of the USV can be written in state-space form, resulting in the system state-space equations:

$$\begin{array}{l}\dot{x}\left(t\right)=Ax\left(t\right)+B\delta (t)\\ y(t)=Cx(t)\end{array}$$

(28)

where $x={\left[\begin{array}{ccc}v& r& \psi \end{array}\right]}^T$ is the system state vector, $\delta$ is the input variable, i.e., the thrust angle, $A$ is the state transition matrix, $B$ is the control input matrix, $y={\left[\psi \right]}^T$ is the system output variable, and $C=\left[\begin{array}{ccc}0& 0& 1\end{array}\right]$ is the output matrix. Discretizing the continuous state-space equations yields the following:

$$\begin{array}{l}{x}_m\left(k+1\right)=A_m{x}_m\left(k\right)+B_m\delta \left(k\right)\\ y\left(k\right)=C_m{x}_m\left(k\right)\end{array}$$

(29)

where $A_m=\left(AT+I\right)$, $B_m=BT$, $C_m=C$, and $T$ represents the sampling period.

To address steady-state errors, incremental state-space equations are applied by substituting state variables and control inputs with their respective changes. Differentiating both sides of the state-space equations results in the following:

$$\begin{array}{l}{x}_m\left(k+1\right)-x_m\left(k\right)=A_m\left(x_m\left(k\right)-x_m\left(k-1\right)\right)+B_m\left(\delta \left(k\right)-\delta \left(k-1\right)\right)\\ y\left(k+1\right)-y\left(k\right)=C_m\left(x_m\left(k+1\right)-x_m\left(k\right)\right)\end{array}$$

(30)

Define $\Delta x_m\left(k+1\right)=x_m\left(k+1\right)-x_m\left(k\right)$, $\Delta \delta \left(k\right)=\delta \left(k\right)-\delta \left(k-1\right)$, and substitute them to obtain the incremental state-space equations:

$$\begin{array}{l}\left[\begin{array}{c}\Delta x_m\left(k+1\right)\\ y\left(k+1\right)\end{array}\right]=\left[\begin{array}{cc}{A}_m& 0\\ C_m{A}_m& 1\end{array}\right]\left[\begin{array}{c}\Delta x_m\left(k\right)\\ y\left(k\right)\end{array}\right]+\left[\begin{array}{c}{B}_m\\ C_m{B}_m\end{array}\right]\Delta \delta \left(k\right)\\ y\left(k\right)=\left[\begin{array}{cccc}0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}\Delta x_m\left(k\right)\\ y\left(k\right)\end{array}\right]\end{array}$$

(31)

The matrix form can be written as follows:

$$\begin{array}{l}x\left(k+1\right)=A_{k}x\left(k\right)+B_k\Delta \delta \left(k\right)\\ y\left(k\right)=C_{k}x\left(k\right)\end{array}$$

(32)

If the control input sequence over the prediction horizon is available, future system outputs can be estimated using the system model and current state. Let N denote the prediction horizon. The equations for system state prediction are as follows:

$$\{\begin{array}{l}x\left(k+1|k\right)=A_{k}x\left(k\right)+B_k\Delta \delta \left(k\right)\\ x\left(k+2|k\right)=A_k^{2}x\left(k\right)+A_k{B}_k\Delta \delta \left(k\right)+B_k\Delta \delta \left(k+1\right)\\ \dots \\ x\left(k+N|k\right)=A_k^{N}x\left(k\right)+A_k^{N-1}{B}_k\Delta \delta \left(k\right)+\dots +A_k{B}_k\Delta \delta \left(k+N-2\right)+B_k\Delta \delta \left(k+N-1\right)\end{array}$$

(33)

Substituting the output matrix $C_k$, the system output prediction equation is obtained as follows:

$${\overrightarrow{y}}_{k+1}=P{x}_k+H\Delta {\overrightarrow{u}}_k$$

(34)

where ${\overrightarrow{y}}_{k+1}$ represents the predicted system output sequence at the current time step, and ${\overrightarrow{u}}_k$ represents the current system input sequence.

$$P=\left[\begin{array}{c}{C}_k{A}_k\\ C_k{A}_k^2\\ ⋮\\ C_k{A}_k^n\end{array}\right]$$

(35)

$$H=\left[\begin{array}{cccc}{C}_k{B}_k& 0& \dots & 0\\ C_k{A}_k{B}_k& C_k{B}_k& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ C_k{A}_k^{n-1}{B}_k& C_k{A}_k^{n-2}{B}_k& \dots & C_k{B}_k\end{array}\right]$$

(36)

To achieve convergence of path-following errors while minimizing the variation in the thrust angle to reduce energy loss, the objective function for optimization is defined as follows:

$$\min J_k={\displaystyle \sum _{i=1}^N{\Vert y\left(k+i|k\right)-{\psi }_d\Vert }_Q^2}+{\displaystyle \sum _{i=1}^N{\Vert \Delta u\left(k+i-1|k\right)\Vert }_R^2}$$

(37)

where $N$ is the total number of optimization steps, $Q=\left(\begin{array}{ccc}{q}_1& \dots & q_N\end{array}\right)$ represents the output error weight matrix, and $R=\left(\begin{array}{ccc}{r}_1& \dots & r_N\end{array}\right)$ represents the input weight matrix.

In practice, there are thrust angle saturation limits and thrust angle rate saturation limits for the outboard motor. These constraints must be considered during the study. The two types of constraints can be expressed as follows:

$${\delta }_{\min }\le \delta \le {\delta }_{\max }$$

(38)

$$\Delta {\delta }_{\min }\le \Delta \delta \le \Delta {\delta }_{\max }$$

(39)

where ${\delta }_{\min }$ and ${\delta }_{\max }$ represent the minimum and maximum thrust angles of the outboard motor, while $\Delta {\delta }_{\min }$ and ${\delta }_{\max }$ represent the minimum and maximum thrust angle rates.

The optimization problem can be reformulated into a standard quadratic programming (QP) form, allowing the optimal thrust angle increment sequence to be determined. By adding the first element of this sequence to the previous thrust angle, the control input for the current time step is generated, and the process repeats. This approach, known as the rolling optimization strategy, enables the desired heading angle control based on MPC.

4. Experimental Results and Analysis

This paper conducts simulation and experimental research based on an independently developed amphibious USV propelled by a single outboard motor, as shown in Figure 5. The main technical parameters of the amphibious USV for experiment are shown in

Table 1. Main technical parameters of the amphibious USV for experiment.

Parameter	Value
Length overall	8 m
Extreme breadth	2.3 m
Load draft	0.55 m
Maximum speed	25 kn
Propeller diameter	$14^{″}$
Maximum propulsion angle	30°

, and the main technical parameters of the outboard motor are shown in Figure 6. The full-scale test is conducted in the harbor of Guangzhou Shipyard International Co., Ltd., in Nansha District, Guangzhou, China (latitude: about N 22.7023°, longitude: about E 113.6505°), with sea conditions ranging from level 1 to level 2, wave height of about 0.2 m, sea surface wind force of about level 1 to level 2, and clear weather.

This section outlines the training process of the proposed DDPG-based path guidance law and examines the training results. To validate the efficiency and performance of the single-outboard-motor USV path-following algorithm combining DDPG and MPC, the high-performing ALOS-PID algorithm is used for the simulation and real ship comparison experiments.

4.1. Agent Training and Results Analysis

The software environment for the agent experiment is based on the Windows 10 operating system with a program implemented in PyTorch. The hardware setup includes an AMD R7-5900X CPU and an NVIDIA GeForce RTX 3090Ti GPU. The hyperparameter settings of the training process of the DDPG agent are shown in

Table 2. Hyperparameter settings of the training process of the DDPG agent.

Parameter	Value
Number of neurons in the hidden layer	256
Optimizer	Adam
Network learning rate	0.0001
Soft update factor $\tau$	0.005
Reward discount factor $\gamma$	0.99
Batch size	256
Experience replay pool size	1 × 106

. Exploration noise $N_t$ selected Ornstein–Uhlenbeck noise with a mean of 0 and a variance of 0.25.

At the start of each training episode, a 400 m straight path is defined, with the USV’s initial position and heading angle randomly set within a 100 m radius of the path’s starting point. The USV propeller thrust is randomly assigned a value between [4000, 6000] and remains constant throughout the episode, enabling the trained path guidance model to accommodate various USV postures and speeds. A total of 1000 episodes are trained, each containing 1000 steps with a time interval of 0.05 s. Details of the MPC heading controller parameters are provided in

Table 3. Parameter settings of the MPC heading controller.

Parameter	Value
Predicted step length $N_p$	10
Dwell time	0.1 s
Propulsion angle constraints	$-{30}^{\circ }\le \delta \le {30}^{\circ }$
Propulsion angular velocity constraint	$-{10}^{\circ }/s\le \Delta \delta \le {10}^{\circ }/s$
Weight parameters $Q$	1
Weight parameters $R$	1

.

Figure 7 illustrates the cumulative reward curve for each episode during training. Initially, the cumulative reward remained steady at around 200. After approximately 50 training episodes, it rose rapidly and stabilized at about 900. Following another 100 episodes, the cumulative reward showed continuous growth, indicating that the agent was progressively learning a strategy to achieve higher rewards. This suggests that the path-following guidance law effectively reduces cross-tracking and heading angle errors, eventually stabilizing the cumulative reward at around 2700. This confirms the validity of the designed reward function, as it ensures convergence of the cumulative reward during training. Subsequently, the performance of the trained path-following guidance law is evaluated through comparative experiments.

4.2. Simulation Comparative Experiment

This study employs the nonlinear grey box model from Matlab’s system identification toolbox to determine the parameters of the USV’s three-degree-of-freedom model, resulting in Equations (40) and (41).

$$\{\begin{array}{l}\dot{u}=0.9294vr-0.4114u+0.0003{\tau }_u\\ \dot{v}=-0.8771ur-2.4674v\\ \dot{r}=-0.0533uv-4.2943r+0.0001{\tau }_r\end{array}$$

(40)

$$\dot{r}=-1.5753r+0.3932\delta$$

(41)

A comparative simulation experiment of USV path following is conducted for a single straight path. The simulation model of the comparative simulation experiment is shown in Figure 8. The experimental path settings and the initial posture parameters of the USV are shown in

Table 4. Parameter settings for path-following experiment of a single straight path.

Parameter	Value
Starting point of the path	(0, 0)
End point of the path	(0, 500)
Direction angle of the path	90°
Initial position of the USV	(40, 0)
Initial heading angle of the USV	0°

. The parameter settings of the ALOS-PID algorithm are shown in

Table 5. Parameter settings related to the ALOS-PID algorithm.

Parameter	Value
Front visual distance $\Delta$ (m)	20
The adaptive gain coefficient $\gamma$	0.0013
Observer gain coefficient $K_f$	0.2
Determine radius $r_s$ (m)	8
$K_p$	0.83
$K_i$	0
$K_d$	1.1

. The simulation experiment results are shown in Figure 9, where the blue line represents the results of the DDPG-MPC algorithm proposed in this paper, and the red line represents the results of the ALOS-PID algorithm used for comparison. Figure 9a shows the comparison of the path-following trajectories of the two methods, Figure 9b shows the comparison of the cross error during the path-following process of the two methods, and Figure 9c shows the comparison of the heading angle error of the two methods.

Figure 9 demonstrates that the ALOS-PID algorithm responds faster. Regarding the cross error, the ALOS-PID algorithm reduces the cross error to zero for the first time in approximately 15 s, while the DDPG-MPC algorithm achieves this in about 30 s. For the heading angle error, both algorithms take roughly the same time to reach zero initially. However, the ALOS-PID algorithm exhibits a larger overshoot compared to the DDPG-MPC algorithm. After the cross error reaches zero, the ALOS-PID algorithm’s cross error gradually increases, peaking at around 16 m, whereas the DDPG-MPC algorithm remains stable. Similarly, the heading angle error shows a comparable trend, with the ALOS-PID algorithm peaking at approximately 7.3° after the initial zero point. Additionally, the DDPG-MPC algorithm outperforms the ALOS-PID algorithm in steady-state performance, as the latter retains some steady-state error even after 50 s of tracking.

By analyzing the average errors of the sampling points for both algorithms throughout the path-following process, as presented in

Table 6. Comparison of average error of path following of a single straight path.

Algorithm	Average Cross Error (m)	Average Heading Angle Error (°)
ALOS-PID	7.871	14.0
DDPG-MPC (Ours)	4.955	11.1

, it is evident that the DDPG-MPC algorithm outperforms the ALOS-PID algorithm in terms of the average cross error and heading angle error. Specifically, the average cross error of the DDPG-MPC algorithm is approximately 37% lower than that of the ALOS-PID algorithm, while its average heading angle error is about 21% smaller.

In addition, this paper also conducted a comparative simulation experiment of USV path following of a multi-polyline path to compare and verify the performance of the algorithms at path turning points. The experimental path points and initial pose of the USV are shown in

Table 7. Parameter settings for path-following experiment of multi-polyline path.

Parameter	Value
Knuckle point of the path	(0, 0), (0, 300), (50, 300), (50, 0), (100, 0), (100, 300), (150, 300), (150, 0), (200, 0), (200, 300)
Initial position of the USV	(−30, −30)
Initial heading angle of the USV	0°

. And the simulation experiment results are shown in Figure 10.

It can be observed from Figure 10 that when the USV navigates along the longitudinal long straight path, the performance of the DDPG-MPC and ALOS-PID algorithms is basically consistent with the results of the single straight path simulation experiment. However, when the lateral short straight path segment switches to the longitudinal long straight path segment, the crossover error of the DDPG-MPC algorithm is slightly larger than that of the ALOS-PID algorithm.

Table 8. Comparison of average error of path following of multi-polyline path.

Agorithm	Average Cross Error (m)	Average Heading Angle Error (°)
ALOS-PID	4.801	10.8
DDPG-MPC (Ours)	4.394	11.3

compares the average errors of the two algorithms on the multi-fold path in detail. It is worth noting that the average crossover error of the DDPG-MPC algorithm is about 8.5% smaller than that of the ALOS-PID, but its average heading angle error is about 4.6% higher. This phenomenon can be attributed to the fact that the traditional ALOS-PID algorithm is prone to overshoot when dealing with right-angle turns. It quickly returns to the trajectory by adopting a larger angle, which performs better in some indicators, but causes subsequent trajectory oscillation. In contrast, the DDPG-MPC algorithm chooses a smoother adjustment strategy and uses a smaller angle to fit the path. Although it seems to have a larger error in related indicators, it effectively avoids the oscillation phenomenon and improves the overall path stability and tracking smoothness. This design achieves performance optimization and balance in a complex path environment.

Based on the above simulation results, it can be concluded that the DDPG-MPC algorithm outperforms the ALOS-PID algorithm in all aspects of single straight-line path following. When switching path inflection points in multi-polyline path following, although the cross error of the DDPG-MPC algorithm is slightly larger than that of the ALOS-PID algorithm, the above difference can be ignored in the overall path-following process. The above comparison results verify the effectiveness and superiority of the DDPG-MPC algorithm proposed in this paper in terms of overall performance.

4.3. Real Ship Comparative Experiment

After verifying the effectiveness of the algorithm through simulation comparative experiments, the proposed algorithm is applied to a real ship to carry out path-following real ship comparative experiments to further validate its performance in actual navigation. The experimental path points and the initial pose of the USV are shown in

Table 9. Parameter settings for real ship comparative experiment.

Parameter	Value
The latitude and longitude of the path point	(N 22.704023, E 113.644455), (N 22.700570, E 113.646854), (N 22.694336, E 113.648297), (N 22.690591, E 113.645698), (N 22.689223, E 113.648798)
Initial position of the USV	The latitude and longitude: (N 22.704600, E 113.644300)
Initial heading angle of the USV	20.0°

.

Figure 11 illustrates the comparison of path-following trajectories from the real ship experiment. The red line depicts the expected path, the blue line shows the trajectory of the ALOS-PID algorithm, and the yellow line represents the trajectory of the DDPG-MPC algorithm. The trajectory diagram indicates that, despite environmental interference such as wind, waves, and currents, both algorithms exhibit differences from the simulation results but can successfully complete the path-following task. Notably, the DDPG-MPC algorithm demonstrates superior overall path-following performance compared to the ALOS-PID algorithm.

Figure 12 compares the errors of the two algorithms during the experiment. The results show that the DDPG-MPC algorithm’s maximum cross error is around 10 m, while the ALOS-PID algorithm’s maximum cross error is approximately 30 m. As summarized in

Table 10. Comparison of average error of path following of real ship experiment.

Algorithm	Average Cross Error (m)	Average Heading Angle Error (°)
ALOS-PID	7.233	34.6
DDPG-MPC (Ours)	6.747	13.8

, the average cross error and heading angle error for each sampling point during the path-following process were computed. Both errors of the DDPG-MPC algorithm are lower than those of the ALOS-PID algorithm, with the average cross error decreased by 6.7% and the average heading angle error reduced by 60.0%.

The three velocity components of the USV in the real ship experiment are shown in Figure 13. It can be seen from the figure that the USV controlled by the DDPG-MPC algorithm can maintain a relatively high longitudinal speed for a long time, while the longitudinal speed of the USV controlled by the ALOS-PID algorithm is more unstable and varies relatively greatly.

5. Conclusions

This paper proposes a path-following control method for a single-outboard-motor USV, incorporating DDPG and MPC algorithms. The research includes modeling and analysis of the USV’s motion model and the outboard motor model. A comprehensive performance comparison with the traditional ALOS-PID method was conducted through simulations and real ship experiments, demonstrating the feasibility and effectiveness of the proposed approach.

In simulation experiments, whether for a single straight path or a multi-polyline path, the DDPG-MPC algorithm proposed in this study outperformed the ALOS-PID algorithm in the average cross error and heading angle error. Specifically, on the single straight path, the DDPG-MPC algorithm achieved an average cross error of 4.955 m, approximately 37% lower than the ALOS-PID algorithm’s 7.871 m. Regarding the heading angle error, the DDPG-MPC algorithm recorded an average of 11.1°, about 21% lower than the ALOS-PID algorithm’s 14.0°. Furthermore, the DDPG-MPC algorithm demonstrated superior stability and response time while effectively reducing the thruster’s energy consumption.

Real ship experiments confirmed the practical advantages of the DDPG-MPC algorithm. Despite external disturbances like wind, waves, and currents, the DDPG-MPC algorithm’s maximum cross error was approximately 10 m, compared to 30 m for the ALOS-PID algorithm. During path following, the DDPG-MPC algorithm achieved an average cross error of 6.747 m, 6.7% lower than the ALOS-PID algorithm’s 7.233 m. In terms of the heading angle error, the DDPG-MPC algorithm recorded an average of 13.8°, significantly lower than the 34.6° of the ALOS-PID algorithm, representing a 60% reduction. Moreover, the DDPG-MPC algorithm maintained a more stable and higher longitudinal velocity over a longer duration, whereas the ALOS-PID algorithm showed less stability and greater variability in longitudinal velocity.

Under real sea conditions, the DDPG-MPC algorithm demonstrated strong adaptability and robustness, effectively handling interference factors like wind, waves, and currents to accomplish path-following tasks. From the experimental results and comparative analysis, it can be inferred that the DDPG-MPC algorithm proposed in this study offers notable advantages in the overall performance of path following for single-outboard-motor USVs and holds significant potential for broader application and promotion.

Future research will aim to further optimize the algorithm to improve its real-time performance and robustness, making it applicable to more complex real-world navigation scenarios. These advancements are anticipated to deliver safer, more efficient, and stable path-following solutions for the autonomous navigation of single-outboard-motor USVs.