PID Controller Based on Improved DDPG for Trajectory Tracking Control of USV

Abstract

When navigating dynamic ocean environments characterized by significant wave and wind disturbances, USVs encounter time-varying external interferences and underactuated limitations. This results in reduced navigational stability and increased difficulty in trajectory tracking. Controllers based on deterministic models or non-adaptive control parameters often fail to achieve the desired performance. To enhance the adaptability of USV motion controllers, this paper proposes a trajectory tracking control algorithm that calculates PID control parameters using an improved Deep Deterministic Policy Gradient (DDPG) algorithm. Firstly, the maneuvering motion model and parameters for USVs are introduced, along with the guidance law for path tracking and the PID control algorithm. Secondly, a detailed explanation of the proposed method is provided, including the state, action, and reward settings for training the Reinforcement Learning (RL) model. Thirdly, the simulations of various algorithms, including the proposed controller, are presented and analyzed for comparison, demonstrating the superiority of the proposed algorithm. Finally, a maneuvering experiment under wave conditions was conducted in a marine tank using the proposed algorithm, proving its feasibility and effectiveness. This research contributes to the intelligent navigation of USVs in real ocean environments and facilitates the execution of subsequent specific tasks.

1. Introduction

Unmanned surface vehicles (USVs), also known as Autonomous Surface Vessels (ASVs), are capable of autonomous navigation and operations. They are often deployed in complex and harsh oceanic environments, which necessitate robust autonomous navigation capabilities. This autonomous navigation serves as the foundation for subsequent tasks such as collision avoidance, automatic docking, target tracking, and formation control. Among these capabilities, autonomous trajectory tracking is a critical aspect that highlights the USV’s navigational proficiency. When navigating dynamic ocean environments with significant wave and wind disturbances, USVs face time-varying external interferences and limitations due to their underactuated nature, leading to reduced navigational stability and increased difficulty in trajectory tracking. Over the years, researchers have relentlessly explored the trajectory tracking control mechanisms of USVs under wave and wind conditions.

Do K. D. [1] investigated the trajectory tracking control of USVs in the presence of wind and wave disturbances using the Serret–Frenet frame. An adaptive robust trajectory tracking controller for underactuated USVs was developed by designing nonlinear observers for the lateral velocity and yaw rate based on nonlinear backstepping and the Lyapunov stability theory [2].

Sun [3] designed an exponentially stable trajectory tracking controller for USVs based on the dynamic surface control theory. Aguiar A. P. [4] researched trajectory tracking control for USVs using switching control techniques, while Fahimi F. [5] proposed a robust sliding mode trajectory tracking control method. Harmouche M. [6] addressed the lack of speed measurement feedback in the trajectory tracking of USVs by proposing a control method based on observer technology and backstepping. Katayama H. [7] introduced a straight-line trajectory tracking control method for underactuated vessels with semi-global uniform asymptotic stability, utilizing nonlinear sampled-data control theory, state feedback, output feedback control techniques, and observer technology [8,9]. However, the aforementioned methods, including optimal control, feedback linearization, and backstepping, require precise modeling to achieve high control accuracy [10]. The motion model of a USV is affected by variables such as speed and load, making precise modeling challenging. Moreover, disturbances from wind, waves, and currents during navigation complicate the path-following control of USVs. Therefore, in real oceanic environments, control algorithms based on deterministic models or those with non-adaptive control parameters often fail to achieve the desired control performance.

The PID control algorithm remains dominant in ship autopilot systems because of its simplicity and reliability. However, when USVs experience substantial time-varying disturbances, the effectiveness of fixed-parameter PID algorithms is not satisfactory. Researchers have sought to enhance the effectiveness through improving the adaptability of PID parameters [11]. Most efforts are focused on combining PID with other theories such as fuzzy control and neural networks to achieve the adaptive tuning of PID parameters.

Adaptive PID controllers adjust PID parameters dynamically during the process of tracking a desired heading, significantly enhancing the algorithm’s dynamic response. However, due to model uncertainties and external disturbances, a discrepancy between the estimated and actual system outputs often exists. Hu Zhiqiang [12] proposed an online self-optimizing PID heading control algorithm, which facilitates the online adjustment of control parameters and exhibits robust performance and interference resistance.

Genetic algorithms (GAs), known for their stable global optimization capabilities, are frequently used for parameter tuning in various USV path controllers. Liu [13] has demonstrated the use of GA for the online tuning of PID parameters to implement adaptive PID control. However, these controllers face challenges such as prolonged parameter optimization times, which can impact the real-time applicability on actual vessels. Designing crossover and mutation operators within the optimization process can shorten the optimization time and enhance the algorithm’s real-time performance [14].

Fuzzy logic control translates expert knowledge into fuzzy rules; it can effectively address the impacts of model uncertainties and random disturbances on USV path-following control. In practical applications, fuzzy logic is often used for the parameter tuning of PID controllers and sliding mode controllers due to its rapid response and real-time performance [15,16]. Liu [15] has established fuzzy rules according to path point errors, heading errors, and error differentials to improve control smoothness. However, the accuracy of fuzzy controllers mainly depends on the complexity of the fuzzy rules, which is generally constructed based on expert knowledge and dynamic models, so complex rules may lead to computational challenges [17].

Peng Yan [18] researched the challenges faced by USV systems, such as large inertia, long time delays, nonlinearity, difficulty in precise modeling, and susceptibility to external disturbances like waves. The study revealed that traditional PID control often fails to achieve satisfactory trajectory tracking performance. Consequently, a PID cascade controller based on generalized predictive control (GPC-PID) was designed to separately control the heading and steering motions of USVs, providing enhanced resistance to external disturbances. Additionally, radial basis function (RBF) neural networks can approximate model uncertainties and external disturbances affecting PID parameters, thus improving the robustness and interference resistance of the controller. RBF neural networks are frequently employed to model the impact of internal and external disturbances on PID parameters.

Reinforcement Learning (RL) has been extensively applied in control systems [19]. Its capability to operate without precise mathematical models and its self-learning abilities make it particularly effective in addressing challenges related to model uncertainties and unknown disturbances in USV trajectory tracking control [20,21]. Neural networks were utilized within sliding mode controllers to tune controller parameters, with RL employed to evaluate the tuning efficacy [22]. This approach enabled the self-learning of neural network parameters, addressing the low model accuracy requirement of sliding mode control while mitigating its chattering issue.

Sun [3] explored trajectory tracking control for underactuated USVs under unknown disturbances using neural network control technologies. Another study [23] employed Q-learning for PID parameter tuning, demonstrating that this controller can effectively resist external disturbances and facilitate the motion control of mobile robots through experiments. Bertaska [24] developed a multi-mode switching controller using Q-learning, which intelligently switches controllers based on the environment and the operational state of the USV, enhancing control performance across different conditions.

Magalhaes [25] proposed a RL control method based on Q-learning that mimics fish swimming motions to control the fins and tail of an unmanned underwater vehicle (UUV) for trajectory tracking. Bian [26] employed neural network controllers for ship trajectory tracking control; however, such learning methods require the pre-acquisition of reliable ship navigation sample data. Deep Q Networks (DQNs), which incorporate experience replay and fixed Q-targets, significantly enhance the stability and expressiveness of complex RL problems [27]. However, DQNs remain constrained to discrete action or state spaces based on Q-learning, making them challenging to apply to the continuous control problem of USV motion under dynamic wave conditions, where low control precision can lead to chattering.

Wu [28] modeled the trajectory tracking problem as a Markov Decision Process (MDP) based on the maneuvering characteristics and control requirements of ships. The DDPG algorithm was used as the controller, and offline learning methods were applied for controller training. Simulation tests showed promising results, although the simulation environment lacked environmental disturbances. Woo [29] introduced a trajectory tracking controller based on Deep Reinforcement Learning (DRL), allowing USVs to learn navigation experiences during voyages. In repetitive trajectory tracking simulations, the control strategy was trained to enable adaptive interaction with the environment, achieving trajectory following. However, this method directly employed DRL as the control strategy rather than adaptive parameter matching, which lacks the mature performance of classical control algorithms. Additionally, it did not leverage the dynamic performance of USVs within the controller, but treated it as a black box, which demands high-quality training data. Without comprehensive training data, the control performance may not meet expectations.

The analysis above highlights several key issues in the current trajectory tracking control methods for USVs: Firstly, many existing algorithms fail to consider the impact of time-varying environmental disturbances. Secondly, when employing machine learning algorithms for training USV motion controllers, acquiring sample data poses significant challenges, and the quality of these training samples greatly influences the performance of machine learning-based controllers. Thirdly, when using RL algorithms such as Q-learning for controller training, these approaches typically use a black box method to compensate for time-varying and difficult-to-quantify environmental disturbances. However, the motion and control models established by these methods are based on discrete mathematical models with low precision. Fourthly, directly using RL as a controller, rather than for adaptive parameter tuning, fails to leverage the superior performance of classical control algorithms and relies entirely on black box processes, which preclude the utilization of USV dynamics within the controller.

To address these challenges, this paper proposes a trajectory tracking control algorithm for USVs that calculates PID control parameters based on an improved Deep Deterministic Policy Gradient (DDPG) algorithm. By integrating RL algorithms with classical control theory, this approach aims to enhance the adaptability of USV motion controllers. The subsequent sections of this paper are organized as follows: Section 2 introduces the maneuvering motion model and parameters for USVs, the guidance law for path tracking, and the PID control algorithm employed. Section 3 provides a detailed explanation of the proposed method for calculating PID parameters based on the improved DDPG algorithm and outlines the state, action, and reward settings for training the RL model. Section 4 presents the simulation results of various algorithms, demonstrating the superiority of the algorithm proposed in this study. In Section 5, maneuvering experiment in wave conditions was conducted in a marine tank using the proposed algorithm, proving its feasibility and effectiveness.

2. The Maneuvering Motion Model of USV and Path-Following Controller for USV

2.1. The Maneuvering Motion Model of USV

The 3-DOF Maneuvering Modeling Group (MMG) motion model of the USV is described as follows [30]:

$$\left\{\begin{array}{l}\dot{x}=ucos\phi -vsin\phi \\ \begin{array}{l}\dot{y}=usin\phi +vcos\phi \\ \dot{\phi }=r\\ \begin{array}{l}\dot{u}={\displaystyle \frac{1}{m+m_x}}\left[\left(m+m_y\right)rv+X_H+X_P+X_R\right]\\ \dot{v}={\displaystyle \frac{1}{m+m_y}}\left[-\left(m+m_x\right)rv+Y_H+Y_P+Y_R\right]\\ \dot{r}={\displaystyle \frac{1}{I_{zz}+J_{zz}}}\left(N_H+N_P+N_R\right)\end{array}\end{array}\end{array}\right.$$

(1)

where $X$, $Y$, and $N$ represent hydrodynamic forces and moments, while the subscripts $H$, $P$, and $R$ represent the hull, propeller, and rudder, respectively. $x$ and $y$ are the coordinates of the USV in the geodetic coordinate system, $\phi$ is the angle of the bow, $u$ represents the longitudinal velocity of the USV in the local coordinate system, $v$ is the lateral velocity, and $r$ denotes the angular velocity in the local coordinate system. $m_x$ is the additional longitudinal mass, $m_y$ is the additional transverse mass, and $I_{zz}$ is the inertia moment of yaw. $J_{zz}$ is the additional moment of inertia in yaw.

The model for calculating $X_{calm}$, $Y_{calm}$, $N_{calm}$ was established using the MMG model.

$$\left\{\begin{array}{l}{X}_{calm}=X_H+X_P+X_R\\ Y_{calm}=Y_H+Y_R\\ N_{calm}=N_H+N_R\end{array}\right.$$

(2)

The force acting on the hull is approximate to the hydrodynamic derivative, ${\mathrm{X}}_{\mathrm{H}}$, ${\mathrm{Y}}_{\mathrm{H}}$, ${\mathrm{N}}_{\mathrm{H}}$, and can be expressed as

$$\left\{\begin{array}{c}{X}_H=X_0+X_u\Delta u+X_{uu}{\left(\Delta u\right)}^2+X_{uuu}{\left(\Delta u\right)}^3+X_{vv}{v}^2+X_{rr}{r}^2+X_{vr}vr\\ Y_H=Y_{v}v+Y_{vvv}{v}^3+Y_{\dot{v}}\dot{v}+Y_{r}r+Y_{rrr}{r}^3+Y_{\dot{r}}\dot{r}+Y_{vrr}v{r}^2+Y_{vvr}{v}^{2}r\\ N_H=N_{v}v+N_{vvv}{v}^3+N_{\dot{v}}\dot{v}+N_{r}r+N_{rrr}{r}^3+N_{\dot{r}}\dot{r}+N_{vrr}v{r}^2+N_{vvr}{v}^{2}r\end{array}\right.$$

(3)

where $\Delta u=u-u_0$ is the speed difference of the ship relative to the initial ship speed and ${-X}_0$ is the resistance when the speed of ship is $u_0$. The geometric and physical models of the USV used in this study are depicted in Figure 1.

Table 1. Parameters of the USV.

Part	Items	Definition	Value
Hull	$L_{pp}\left[\mathrm{m}\right]$	Length between Perpendiculars	1.5
	$B\left[\mathrm{m}\right]$	Breadth of Waterline	0.444
	$d\left[\mathrm{m}\right]$	Depth	0.107
	$C_B$	Block Coefficient	0.395
	$x_G\left[\mathrm{m}\right]$	X-position of Gravity Center relative to Midship	−0.12
	$z_G\left[\mathrm{m}\right]$	Z-position of Gravity Center relative to Baseline	−0.3
	$I_z[\mathrm{k}\mathrm{g}·{\mathrm{m}}^2]$	Gyration about Z-axis	3.947
	$I_x[\mathrm{k}\mathrm{g}·{\mathrm{m}}^2]$	Gyration about X-axis	0.35
Propeller	$D_p \left[\mathrm{m}\right]$	Diameter	0.029
	$Z_P$	Number of Blades	5
Rudder	$A_R$ $\left[{\mathrm{m}}^2\right]$	Longitudinal Projection Area	0.001675
	$H_R \left[\mathrm{m}\right]$	Rudder Height	0.05

illustrates parameters of the USV.

Using Computational Fluid Dynamics (CFD) and regression analysis, the hydrodynamic derivatives for static water maneuvering have been obtained, as shown in

Table 2. Hydrodynamic derivatives.

Hydrodynamic Derivative	Value
$X_0^{\prime }$	−0.0026
$X_u^{\prime }$	−0.000700
$X_{\dot{u}}^{\prime }$	−0.000238
$X_{uu}^{\prime }$	0.002500
$X_{uuu}^{\prime }$	−0.001800
$X_{vv}^{\prime }$	0.002744
$X_{rr}^{\prime }$	−0.002807
$X_{vr}^{\prime }$	0.01247
$Y_v^{\prime }$	−0.01471
$Y_{\dot{v}}^{\prime }$	−0.007049
$Y_{vvv}^{\prime }$	0.112200
$N_v^{\prime }$	−0.006399
$N_{vvv}^{\prime }$	−0.006952
$Y_r^{\prime }$	0.003013
$Y_{\dot{r}}^{\prime }$	−0.000777
$Y_{rrr}^{\prime }$	−0.000467
$N_r^{\prime }$	−0.001708
$N_{\dot{v}}^{\prime }$	−0.000283
$N_{\dot{r}}^{\prime }$	−0.000419
$N_{rrr}^{\prime }$	−0.000261
$Y_{vvr}^{\prime }$	−0.005687
$Y_{vrr}^{\prime }$	−0.013020
$N_{vvr}^{\prime }$	−0.015600
$N_{vrr}^{\prime }$	−0.001047

.

The hydrodynamic interaction coefficients between the ship, propeller, and rudder are presented in

Table 3. Ship–propeller–rudder hydrodynamic interaction coefficients.

Interaction Coefficients	Definition	Value
${\alpha }_H$	Rudder Force Correction Factor	0.5
$t_p$	Thrust Reduction Coefficient	0.01
$w_{p0}$	Wake Fraction Coefficient	0.01
$\epsilon$	Propeller–Rudder Interaction Coefficient	0.52
$\kappa$		0.23
$t_R$		0.15
$\gamma$	Speed Coefficient	1.55
$K_T$	Open Water Propeller Thrust Coefficient	$0.5932-0.1971{\mathrm{J}}_{\mathrm{p}}-0.0481{\mathrm{J}}_{\mathrm{p}}^2$

.

These hydrodynamic coefficients are used to account for the interactions between the ship, propeller, and rudder when calculating maneuvering forces.

2.2. The LOS Guidance Rule and PID Controller

The LOS strategy aims to guide the USV to track a virtual target along the desired trajectory, minimizing the position and heading deviations between the USV’s actual location and the target, ultimately ensuring the USV reaches the intended path. The trajectory tracking strategy using LOS is illustrated in Figure 2.

In Figure 2, $XOY$ denotes the global coordinate system, $x{O}^{\prime }y$ denotes the local coordinate system, and the USV’s position is denoted as $\mathrm{P}$, with heading angle $\phi$, longitudinal velocity $u$, lateral velocity $v$, and yaw rate $r$. The target trajectory is segmented into several approximate straight-line segments, with the current segment being tracked lying between points ${\mathrm{P}}_{\mathrm{k}-1}$ and ${\mathrm{P}}_{\mathrm{k}}$. The lateral tracking error is denoted as $y_e$, the lookahead distance as Δ, and the desired heading angle as ${\psi }_{\mathrm{d}}$. During the curve tracking process, the sub-target along the trajectory needs to be switched. The switch occurs from ${\mathrm{P}}_{\mathrm{k}-1}$ to ${\mathrm{P}}_{\mathrm{k}}$ when $\overrightarrow{{\mathrm{P}}_{\mathrm{k}-1}\mathrm{P}}·\overrightarrow{{\mathrm{P}}_{\mathrm{k}-1}{\mathrm{P}}_{\mathrm{k}}}<0$.

The PID controller is a linear regulator that compares the desired heading angle ${\psi }_{\mathrm{d}}\left(k\right)$ with the actual heading angle $\phi \left(k\right)$ to form the heading angle deviation $e\left(k\right)$:

$$e\left(k\right)={\psi }_{\mathrm{d}}\left(k\right)-\phi \left(k\right)$$

(4)

The desired rudder angle can be expressed as Equation (5):

$$\delta \left(k\right)=K_{\mathrm{p}}e\left(k\right)+K_{\mathrm{i}}\sum _{i=0}^{k}e\left(i\right)dt-K_{\mathrm{d}}\left(e\left(k\right)-e\left(k-1\right)\right)/dt$$

(5)

Considering the integral saturation condition of the PID controller, the PD parameters are adjusted to ensure the USV quickly tend to the desired track and keep the USV navigating within the error range. Therefore, Equation (5) can also be expressed as Equation (6):

$$\delta =K_{\mathrm{p}}e+K_{\mathrm{d}}(e-e^{\prime })$$

(6)

where ${\mathrm{e}}^{\prime }$ is the error at the previous moment. The neural network is performed to produce the appropriate PD parameters.

3. PID Parameter Calculating Model Based on an Improved DDPG Algorithm

3.1. Algorithm Description

The tracking control process of USVs is a time-sequential decision-making problem characterized by MDPs. Thus, a USV can be regarded as an agent, with its motion control expressed in the form of $\{s,A,P(s^{\prime }|s, a), R\}$. Herein, the details are as follows:

a.

$P\left(s^{\prime }\right|s,a)$ is the state transition matrix and refers to the probability of transitioning to the next state ${\mathrm{s}}^{\prime }$ after applying action $\mathrm{a}$ in the current state $\mathrm{s}$.

b.

$s$, the state set encompasses all states and exhibits the Markov property; that is, future states depend only on the current state and are independent of past states.

c.

$A$, the action set, comprises all possible actions that the agent can select. State transitions depend not only on the environment but also on the agent’s ability to guide state transitions by selecting different actions.

d.

$R$, the reward function, maps states and actions to rewards, reflecting preferences for different states.

Executing actions transitions the system from an initial state to a terminal state, forming a trajectory $\tau$, represented as

$$\tau =(s_0,a_0,s_1,a_1,...s_t,a_t)$$

(7)

where $s_0$ is the initial state, $s_t$ is the terminal state, and $a_i$ is the action chosen at time step i.

The aim is to maximize the cumulative reward $R\left(\tau \right)$ along the trajectory, expressed as

$$R(\tau )={\sum }_{t=0}^T{\gamma }^t{r}_t$$

(8)

where $T$ is the terminal time, $r_t$ is the reward at time $t$, and $\gamma$ is the discount factor, with values ranging between 0 and 1.

To maximize the cumulative reward $R\left(\tau \right)$, the policy $\pi$ must be continuously optimized. The policy $\pi$ maps states $\mathrm{s}$ to actions $a$, and the agent chooses actions based on the observed state according to the policy. The strategy that maximizes the cumulative reward is called the optimal policy, denoted as ${\pi }^{*}$.

$${\pi }^{*}={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{\pi }{E}_{\tau ~\pi }\left[R\right(\tau \left)\right]$$

(9)

where $E$ is the expectation, $\tau ~\pi$ is the trajectory $\tau$ based on policy $\pi$.

The goal of RL is to find the optimal policy ${\pi }^{*}$.

$$Q^{*}(s,a)={\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathsf{\pi }}\left(E_{\tau ~\pi }\left[R\right(\tau \left)\right|s_t=s,a_t=a]\right)$$

(10)

$Q^{*}(s,a)$ denotes the optimal action value function, can be used to provide the best action, and can be recursively expressed based on the Bellman equation:

$$Q^{*}\left(s,a\right)=E_{s^{\prime }~P}\left[r\left(s,a,s^{\prime }\right)+{max}_{a^{\prime }}\left[\gamma Q^{*}\left(s^{\prime },a^{\prime }\right)\right]\right]$$

(11)

where $s^{\prime }~P$ denotes the next state derived from the environment transition probability $P$, and $a^{\prime }$ represents the subsequent action obtained from the policy $\pi$.

$$a^{*}\left(s\right)={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_a{Q}^{*}(s,a)$$

(12)

$a^{*}\left(s\right)$, the best actions, can be recursively expressed by the Bellman equation.

In policy-based methods, the policy is denoted as ${\pi }_{\theta }$, where $\theta$ represents a set of parameters constituting the policy. The goal is to maximize the expected return $J\left({\pi }_{\theta }\right)=E_{\tau ~{\pi }_{\theta }}\left[R\right(\tau \left)\right]$, and gradient ascent is employed to update the parameters $\theta$ and optimize the policy.

$${\theta }_{k+1}={\theta }_k+\alpha \nabla J\left({\pi }_{\theta }\right){|}_{{\theta }_k}$$

(13)

where $\nabla J\left({\pi }_{\theta }\right)$ is the policy gradient.

The improved DDPG algorithm in this study utilizes an Actor–Critic architecture, comprising four networks, detailed as follows:

(1)

Current Actor Network

The input to the current Actor is the state space $s$, and the output is the action $a$. In this study, the state space is defined as

$$s=[u,v,r,\phi ,e_{psi},y_e,d,{\alpha }_k,\dot{d},\dot{e_{psi}}]$$

(14)

where $u$ and $v$ are the longitudinal and lateral velocities, $r$ is the yaw rate, $\phi$ is the heading angle, $e_{psi}$ is the lateral deviation from the target trajectory, $d$ is the rudder angle, and ${\alpha }_k$ is the inclination angle of the target trajectory. The action $a$ is defined as

$$a=[k_p,k_d]$$

(15)

The objective of updating the Actor is to maximize the Q-value evaluated by the current Critic network. Thus, the gradient of the Actor is updated via backpropagation through the Critic’s gradient.

The Actor network architecture, as shown in Figure 3, consists of an input layer, two hidden layers, and an output layer. The input layer has a dimension of 10, the two hidden layers contain 400 and 300 neurons, and the output layer has a dimension of 2. To extract features effectively, and also to prevent gradient saturation and vanishing problems, ReLU is used in the hidden layers and Tanh is used in the output layer.

Figure 4 illustrated the relationship between the Actor network and USV motion controller. During training in the simulation system, the Actor outputs actions based on the input state through its neural network. These actions, representing PID control parameters, are utilized for USV motion control. The control deviation of the USV is used as input to obtain the USV control output variables, such as steering actions in heading control. Consequently, the USV executes steering actions, which are resolved through the rudder force model in the simulation environment, resulting in the USV’s subsequent motion posture.

(2)

Target Actor Network

The Target Actor network generates target actions $a^{\prime }$, which are fed into the Target Critic network. The structure of the Target Actor mirror that of the current Actor network. Its weights are obtained via soft updates from the current Actor’s weights using Polyak averaging:

$${\theta }_{target}\leftarrow {\tau \theta }_{current}+(1-\tau ){\theta }_{target}$$

(16)

where ${\theta }_{target}$ represents the target network parameters, ${\theta }_{current}$ is the current network parameters, and $\tau$ is a small coefficient. The arrow symbol used here is used to update ${\theta }_{target}$. This slow update process maintains the stability and relative independence of the target network, facilitating smoother changes that enhance algorithm stability and convergence.

(3)

Current Critic network

The current Critic network evaluates the value of action $a$ in state $s$. Training the current Critic network requires target values, which are computed using the Target Critic and Target Actor networks. The current Critic is a feedforward neural network; its structure is depicted in Figure 5. The input is a concatenation of the environmental state $\mathrm{s}$ and action $\mathrm{a}$, outputting the state-action value $Q(s,a)$. The hidden layers contain 400 and 300 neurons, and the output layer dimension is 1, representing the state-action value.

(4)

Target Value Network (Target Critic)

The Target Critic network calculates the target Q-value $Q^{\prime }(s,a)$ and, combined with the actual reward $r$, computes the TD error, which is used to update the weights of the current Critic network. The structure of the Target Critic network mirrors that of the current Critic but is used for target updates. Its weights are also obtained through soft updates from the current Critic’s weights. The input consists of the environmental state $s$ and action $a^{\prime }$, and the output is the state-action value $Q^{\prime }(s,a)$.

3.2. Ornstein–Uhlenbeck (OU) Noise

OU noise is a time-correlated stochastic process initially used to describe Brownian motion in physics. In RL, it is employed to generate smooth and orderly noise sequences that aid in exploring the action space. The mathematical expression for the OU process is

$${dx}_t=\theta (\mu -x_t)dt+\sigma d{W}_t$$

(17)

where $x_t$ is the value of the noise at time t, $\theta$ is the parameter controlling the speed of noise regression to the mean, $\mu$ is the long-term mean of the noise, $\sigma$ is the intensity of noise fluctuations, and $d{W}_t$ is the standard Brownian motion.

At each time step, the noise is updated according to the following OU process:

$$x_{t+1}=x_t+\theta (\mu -x_t)\Delta t+\sigma \sqrt{\Delta t}\mathcal{N}\left(\mathrm{0,1}\right)$$

(18)

where $\mathcal{N}\left(\mathrm{0,1}\right)$ is a normal distribution with mean 0 and variance 1. When executing the strategy, the generated OU noise is added to the actions derived from the deterministic strategy.

3.3. Binary Experience Pool Based on Adaptive Batching

To address the slow training speed of the DDPG algorithm, the experience pool is divided into a success experience pool and a failure experience pool. To eliminate correlations between data, an adaptive batch size function is designed, where $B$ experiences are sampled from each of the success and failure pools. New experience is gained from the environment (including the current state, action taken, reward received, next state, and termination status), and it is added to the buffer. During each current network training session, a random batch of experiences is sampled from the buffer, effectively breaking the temporal correlation between experiences and ensuring that training data are more independently and identically distributed.

$$\sigma =1+{\displaystyle \frac{n_e}{n_{max}}}$$

(19)

$$B=⌊\sigma n⌋$$

(20)

In this context, $n_e$ represents the number of training iterations, and $n_{max}$ denotes the total number of training iterations set.

The overall algorithmic process is as follows:

1.

The Actor generates a set of actions, to which OU noise is added.

2.

The agent obtains the next state based on the current action and inputs it into the reward function. The data from $(s_t,a_t,r,s_{t+1})$ is categorized into the successful experience pool or the failed experience pool based on success or failure.

3.

A sample of $\mathrm{n}$ experiences is drawn from the experience pool, and both $s_t$ and $s_{t+1}$ are input into the Actor network, while $(s_t,a_t,r,s_{t+1})$ are fed into the Critic network, where iterative updates are performed.

4.

The Target Actor network receives $s_t$ and $s_{t+1}$; inputs action $a$ and random noise into the agent, which interacts with the environment to obtain the next action ${\mathrm{a}}_{\mathrm{t}+1}$; and outputs it to the Target Critic network. The action values $Q$ are then received to update the network.

5.

The Target Critic network receives $s_{t+1}$ and $a_{t+1}$ and calculates the Q-values. These values are then combined with the rewards to compute the labels used for iterative network updates. See Figure 6.

3.4. Reward Definition and Analysis

Based on the maneuvering characteristics of USVs and the features of path tracking control, the reward function is defined as follows:

$$r_{psi}=\left\{\begin{array}{cc}0,& e_{psi}\le 0.1\\ -e_{psi}-0.1e_{psi\_last},& e_{psi}>0.1\end{array}\begin{array}{c}\\ \end{array}\right.$$

(21)

$$r_{y_e}=\left\{\begin{array}{cc}0,& y_e\le 0.1\\ -0.1,& y_e>0.1\end{array}\begin{array}{c}\\ \end{array}\right.$$

(22)

$$r=r_{y_{\mathrm{e}}}+r_{\mathrm{psi}}$$

(23)

The reward function comprehensively considers both the lateral deviation and heading angle deviation in trajectory tracking, with the expectation that both deviations remain minimal. According to the reward setting, if the heading angle deviation is less than 0.1, the reward is 0. Conversely, if the heading angle deviation exceeds 0.1, a negative reward is generated, with the magnitude of the negative reward positively correlated with the consecutive heading angle deviations. This design aims to use RL to reduce the heading angle deviation. For lateral deviation, the requirement is to keep it within 1 m. Within this range, the lateral deviation reward is 0, but if it exceeds 1 m, a negative reward of −0.1 is applied, aiming to reduce lateral deviation through RL.

Below is an analysis of typical scenarios based on this reward setting (see Figure 7):

(a)

$y_e>1 \left[\mathrm{m}\right]$, $e_{psi}>0.1 \left[\mathrm{r}\mathrm{a}\mathrm{d}\right]$: As illustrated in the scenario, both the lateral and heading angle deviations receive significant negative rewards (on the order of −0.1). During training, actions receiving nearly 0 rewards will be emphasized, guiding the USV to alter its heading angle toward the LOS target point and reduce lateral error. According to the reward setting, the priority between altering the motion direction toward the LOS target point and approaching the tracking trajectory to reduce lateral deviation is dynamically adjusted.

(b)

$y_e>1 \left[\mathrm{m}\right]$, $e_{psi}<0.1 \left[\mathrm{r}\mathrm{a}\mathrm{d}\right]$: In this scenario, the lateral deviation receives a significant negative reward (on the order of −0.1), while the negative reward for heading angle deviation is minor (close to 0). During training, actions with nearly 0 rewards are reinforced, guiding the USV to approach the tracking trajectory to reduce lateral error without excessively adjusting the heading angle, which could lead to increased lateral deviation and negative rewards for heading angle deviation. According to the reward setting, the priority between turning toward the LOS target point and approaching the tracking trajectory is dynamically adjusted.

(c)

$y_e<1 \left[\mathrm{m}\right]$, $e_{psi}>0.1 \left[\mathrm{r}\mathrm{a}\mathrm{d}\right]$: Here, the lateral deviation receives a minor negative reward (close to 0), while the heading angle deviation incurs a significant negative reward (on the order of −0.1). During training, actions yielding nearly 0 rewards are emphasized, guiding the USV to adjust its heading angle toward the LOS target to reduce the heading angle deviation. However, due to the inertia of USV motion and the narrow 1 m reward boundary for lateral deviation, if the heading angle deviation is substantial, the USV might overshoot the 1 m boundary during adjustment, resulting in oscillation around the tracking trajectory. To avoid this, the USV should not have excessive heading angle deviation when entering this scenario.

(d)

$y_e<1 \left[\mathrm{m}\right]$, $e_{psi}<0.1 \left[\mathrm{r}\mathrm{a}\mathrm{d}\right]$: In this case, both lateral and heading angle deviations receive minor negative rewards (close to 0). The USV’s trajectory tracking control tends to stabilize, meeting the tracking requirements and maintaining this condition despite wave disturbances. However, if the USV enters this scenario and maintains stability, there may exist a steady-state lateral error of less than 1 m. With 0 rewards for both lateral and heading angle deviations, this steady-state error might not be corrected. This issue can be addressed by dynamically changing the LOS target point, but this requires a longer training time.

4. Simulation Results and Analysis

To validate the effectiveness of the proposed improved DDPG-based PID control algorithm for USV trajectory tracking, this study conducts a comprehensive comparison with both an adaptive PID control algorithm and a standard DDPG-based PID control algorithm. The hardware used for model training includes a desktop computer equipped with an Intel Core i5-12700 CPU (Intel, Santa Clara, CA, USA) and 8 GB of RAM. The software environment consists of the Windows 11 operating system, TensorFlow version 2.1.19 (GPU), and Python version 3.9.1. During the adaptive navigation control, the RL parameter tuner was trained for a total of 1500 steps per episode, with a sampling time of 0.1 s. The desired trajectory was set as a straight line parallel to the y-axis, and a total of 2000 episodes were trained. Each episode terminated either upon reaching the specified number of training steps or upon arriving at the designated end point on the straight line.

The simulation environment was designed to include scenarios with wave disturbances. To assess the system’s stability and resilience against external disturbances, a transverse force of $[-0.2\times {10}^3,0.2\times {10}^3]$ [N] and a turning moment of $\left[-0.2\times {10}^3,0.2\times {10}^3\right]$ [N·m] were introduced as perturbations. This is modeled as $f\left(x\right)=0.2U\left(t\right)$, where $U\left(t\right)$ represents an independent random variable at each time point, following a uniformly distributed random function $U\left(t\right)~f\_u(-\mathrm{1,1})$, with $f\_u\left(\right)$ denoting the random distribution. The applied transverse disturbance force is depicted in Figure 8, while the turning disturbance moment is illustrated in Figure 9.

4.1. Comparative Analysis of Zigzag Trajectory Tracking

Figure 10 presents the results of Zigzag trajectory tracking under wave disturbance using adaptive PID algorithms, PID parameters calculated by the DDPG algorithm, and PID parameters determined by the improved DDPG algorithm. The trajectory path points are set as polylines between the coordinates (0, 0), (45, 45), (85, 5), and (125, 45). The figures indicate that, due to disturbances, all three algorithms exhibit significant overshoot in the initial stage. However, the USV is eventually able to adjust to the desired trajectory. During target point transitions, each algorithm demonstrates effective control performance, with lateral deviation errors fluctuating within acceptable limits and heading angle deviations rapidly diminishing until stabilization; the rudder angle changes are also smooth. Notably, both RL algorithms outperform the adaptive PID controller in managing lateral deviations and heading tracking. Moreover, the method employing PID parameters calculated using the improved DDPG algorithm results in a smoother desired heading angle, indicating superior tracking performance. Its steering curve is also more stable. See also

Table 4. Comparison of Zigzag trajectory tracking metrics.

Controller	Stable Mean Lateral Error/m	Relative Cross-Track Error	Mean Heading Angle Tracking Deviation/°
Improved DDPG and PID	0.109	0.245	3.601
DDPG and PID	0.388	0.874	3.952
Adaptive PID	0.985	2.218	6.584

.

4.2. Comparative Analysis of Curved Trajectory Tracking

Figure 11 illustrates the comparison of tracking a circular trajectory with a radius of 10 m under sea state 4 with a 90 degree wave, using the three different aforementioned algorithms for PID parameter calculating. It is evident that the USV can quickly adjust to the desired trajectory, demonstrating effective control performance. The lateral deviation errors fluctuate within acceptable limits, and the heading angle deviations rapidly decrease and stabilize. The changes in rudder angle are also smooth. Both RL algorithms outperform the adaptive PID controller in managing lateral deviation and heading tracking. When using the adaptive PID controller and the controller based on DDPG-calculated PID parameters, significant initial mistakes can lead to difficulties in quickly eliminating errors. Combined with the inertia of the USV and wave disturbances, this can result in oscillations. In contrast, the improved DDPG algorithm for PID control dynamically adjusts control parameters according to the trend in tracking error changes, reducing oscillations near the trajectory and achieving better control performance. The resulting desired heading angle and steering curve are smoother, indicating that this controller is highly adaptable and robust, providing superior tracking performance. See also

Table 5. Comparison of circle trajectory tracking metrics.

Controller	Stable Mean Lateral Error [m]	Relative Cross-Track Error	Mean Heading Angle Tracking Deviation [deg]
Improved DDPG and PID	0.402	0.905	1.98
DDPG and PID	0.568	1.279	2.56
Adaptive PID	0.652	1.468	2.98

.

5. Experimental Results and Analysis

It is challenging to conduct curve tracking experiments due to the wave maker located in the ship towing tank, which has a limited width. Therefore, this study conducted Zigzag trajectory tracking control experiments under wave conditions only. The positioning equipment used was an indoor UWB system, and the USV was equipped with a gyroscope for heading, a servo motor for propeller adjustment, and a rudder servo for steering. The main control board was an STM32, facilitating wireless serial communication with the host computer. The host computer sent commands to the model ship, and the USV transmitted real-time sensor data back to the host for display. Simultaneously, the host computer performed control calculations based on the collected data, utilizing the proposed improved DDPG algorithm for calculating PID parameters.

The experiment involved six irregular trajectory points. Initially, the propeller speed was adjusted in still water to achieve a USV speed of 0.8 [m/s], corresponding to a Froude number of 0.21. The USV’s heading was adjusted based on its real-time position relative to the first target point of the trajectory. Once the wave-making machine established a stable waveform with the desired amplitude and wavelength, intelligent control was activated on the host computer. The USV dynamically adjusted its rudder angle according to the algorithm, ensuring movement along the trajectory points. The aforementioned method was used to measure and record the motion data of the USV. The experimental process is depicted in Figure 12, and the six trajectory points along with the USV’s motion trajectory are shown in Figure 13.

The experiment demonstrated that, despite wave disturbances, the USV effectively tracked all trajectory points with a smooth path, achieving a maximum trajectory error of approximately 200 [mm]. This error is small and is primarily attributed to deviations in trajectory point acquisition caused by the UWB positioning accuracy and the resulting control deviations. Under the current hardware conditions and operating modes, the positioning accuracy was optimized using filtering techniques; however, it is still affected by equipment precision, UWB positioning methods, and electromagnetic interference from water waves and metal tracks in the tank. Despite these challenges, the experiment and error analysis indicate that the intelligent matching technique for control parameters based on the proposed method achieves an excellent control accuracy under existing conditions.

Figure 14 illustrates the real-time feedback of the rudder angle and the measured heading angle during the experiment. The changes in heading reveal five major trend adjustments required for path tracking, corresponding to the five segments in the actual trajectory. The heading changes were smooth, with a maximum rudder angle of 50 degrees at sharp turns along the target trajectory. The adaptive control yielded favorable results.

6. Conclusions

In this study, we present a novel algorithm that integrates DDPG with a PID controller to achieve path following in the presence of complex conditions, particularly wave interference. The algorithm begins by leveraging a 3-DOF MMG maneuvering motion model, which serves as the foundation for subsequent DDPG training. CFD simulations and regression analysis are employed to extract hydrodynamic derivatives and interaction coefficients necessary for precise motion prediction. We further detail the LOS guidance strategy, which directs the USV to follow a virtual target along the desired trajectory, with PID parameters adjusted dynamically via the DDPG framework.

The design of both the Actor and Critic networks is carefully structured, and to address the issue of slow training speeds in DDPG, we implement a dual-experience pool, separating the successful and failed experiences. Additionally, an adaptive batch size function is introduced to minimize data correlations, further enhancing the training efficiency. The reward function is rigorously formulated to account for both the lateral deviation and heading angle deviation, and its effectiveness is thoroughly examined under various operating conditions.

Simulations and experimental trials involving path following in Zigzag and turning maneuvers, conducted under wave disturbance, demonstrate the algorithm’s robust performance. The USV maintained superior tracking accuracy, even under continuously varying wave directions, highlighting the algorithm’s strong generalization capabilities and robustness.

Two key practical insights emerge from this work: First, the desired trajectory must align with the USV’s maneuvering limits. Trajectory points that exceed the USV’s steering capabilities, especially under full rudder conditions, will render the target trajectory unattainable in the presence of external disturbances. This not only compromises tracking performance but also hinders the Reinforcement Learning process. Second, while improving tracking accuracy, setting excessively small thresholds for error in the reward function can lead to instability in the neural network. Lowering the static error threshold makes successful experiences more difficult to achieve, thereby slowing the agent’s learning rate and significantly increasing the training time.