Path Planning for Automatic Berthing Using Ship-Maneuvering Simulation-Based Deep Reinforcement Learning

Abstract

Despite receiving much attention from researchers in the field of naval architecture and marine engineering since the early stages of modern shipbuilding, the berthing phase is still one of the biggest challenges in ship maneuvering due to the potential risks involved. Many algorithms have been proposed to solve this problem. This paper proposes a new approach with a path-planning algorithm for automatic berthing tasks using deep reinforcement learning (RL) based on a maneuvering simulation. Unlike the conventional path-planning algorithm using the control theory or an advanced algorithm using deep learning, a state-of-the-art path-planning algorithm based on reinforcement learning automatically learns, explores, and optimizes the path for berthing performance through trial and error. The results of performing the twin delayed deep deterministic policy gradient (TD3) combined with the maneuvering simulation show that the approach can be used to propose a feasible and safe path for high-performing automatic berthing tasks.

1. Introduction

Since the early stages of modern shipbuilding, much attention has been paid to automated methods of ship navigation, particularly with the continuous advancements in artificial intelligence (AI). As a result, the number of autonomous ships has rapidly grown. Autonomous ship navigation offers substantial advantages in terms of safety, efficiency, reliability, and environmental sustainability. By harnessing advanced technologies such as sensor systems, data analysis, and artificial intelligence and by reducing the risk of human error, autonomous navigation systems ensure the safety of ship operations. These systems operate consistently and reliably, unhindered by human limitations, resulting in more predictable performance and fewer accidents. Autonomous ships can process vast amounts of data, enabling informed decision making, collision avoidance, and adaptation to changing conditions [1]. However, automatic ship berthing remains an extremely complex task, particularly under low-speed conditions where the hydrodynamic forces acting on the ship are highly nonlinear [2]. Controlling the ship becomes challenging and necessitates the expertise of an experienced commander. Numerous researchers have conducted extensive studies on the principles and algorithms for automatic ship berthing. Researchers have developed various ship control algorithms based on control theories and maneuverability assumptions [3,4,5,6,7,8,9]. These approaches proved effective under defined berthing conditions before the existence of AI. The development of AI algorithms has propelled the creation of algorithms and methods that enhance ship control performance, improve safety, and greatly reduce accidents in the marine industry. Many supervised learning algorithms based on neural networks have exhibited promising results with a high success rate, as in [10,11,12,13]. Applying AI algorithms eliminates the need to clearly understand the mathematical model of ships. However, acquiring a substantial number of labeled training data can be time-consuming and costly.

Unlike the aforementioned methods where the training dataset is not necessary, reinforcement learning techniques, which constitute an area of machine learning, allow the ship to learn and optimize its berthing maneuvers through interactions with a simulated environment. The application of RL in the automatic berthing task has shown good results when the ship can automatically learn the strategy and optimize the control policy to move to the berthing point [14,15].

In this paper, initial development of a novel path-planning algorithm for autonomous ship berthing that uses the latest technique in reinforcement learning, called twin delayed deep deterministic policy gradient (TD3), is proposed. The TD3 algorithm was introduced by Scott Fujimoto (2018) and is specifically designed for continuous action spaces. TD3 is an extension of deep deterministic policy gradients (DDPGs) and aims to address certain challenges and improve the stability of learning in complex environments. TD3 exploration algorithms allow the agent to explore the environment and gain new experiences that optimize rewards through trial and error. It employs two distinct value function estimators, which mitigate the overestimation bias and stabilize the learning process. Leveraging these two critics, TD3 provides more accurate value estimates and facilitates better policy updates. High performance and stability compared to other algorithms in the field of RL were shown in [16]. In combination with the MMG model, the solutions for ship-maneuvering motion simulation that were proposed by a research group maneuvering modeling group (MMG) in 1977 [17] suggest a feasible path, resulting in faster convergence and improved accuracy.

This article is organized into five parts. The first section introduces previous studies conducted in this field. Section 2 presents the equation of motion for the ship based on the MMG model along with hydrodynamic and interaction coefficients. Section 3 outlines the path-planning algorithm based on deep reinforcement learning TD3. Section 4 showcases and discusses simulation results for two berthing cases. Finally, Section 5 concludes this research.

2. Mathematical Model

2.1. Coordinated System

This paper focuses on the motions of USVs in horizontal planes only. Thus, two coordinate systems were defined for a maneuvering ship based on the right-hand rule, as shown in Figure 1. The earth-fixed coordinate system is $Oxy$, where the origin is located on the water surface and the body-fixed coordinate system is $o{x}_b{y}_b$, where the origin is in the midship. The $x$ and $y$ axes point toward the ship’s bow and starboard, respectively. The heading angle $\psi$ represents the angle between the $x$ and $x_b$ axes.

2.2. Mathematical Model of USV

The motion equation with three degrees of freedom (3-DOF) is established based on Newton’s second law. In this paper, the berthing task assumed performing in the calm water, the disturbances from environments such as waves and wind in the port area are ignored due to the simplicity of the equation of motion. Thus, the heave, roll, and pitch motion are relatively small and not significantly affected in the equation of motion; thus, it can be neglected from the equation of motion. Furthermore, the low-speed condition made the 3-DOF motions sufficient to simulate the motion of the vehicle. Additionally, the main purpose of this paper is to focus on the path planning algorithm to generate the feasible path for the berthing task based on the maneuvering simulation. The application of the 3-DOF motion equations causes simplicity but retains characteristics of the system. The MMG model for the 3-DOF equation of motion suggested in [17] that the total ship force and moment can be divided into sub-components: the hull, thruster, and steering system. Thus, the 3-DOF motion equation is expressed as

$$\begin{array}{l}m\left(\dot{u}-vr-x_{b_G}{r}^2\right)=X_H+X_P+X_R\\ m\left(\dot{v}+ur+x_{b_G}\dot{r}\right)=Y_H+Y_P+Y_R\\ I_{zz}\dot{r}+m{x}_G\left(\dot{v}+ur\right)=N_H+N_P+N_R\end{array}$$

(1)

where $m$ is the mass of the ship; $x_{b_G}$ is the longitudinal position of the center of gravity of the ship; $u, v, r$ denote the surge, sway, and yaw velocities, respectively; the symbol “$\dot{}$” in the head of the variable denotes the derivative of the variable with respect to time; $I_{zz}$ is the moment representing the mass moment of inertia with respect to the z-axis; $X$, $Y$, and $N$ represent the surge force, lateral force, and yaw moments, respectively, at the midship; and the subscripts $H$, $P$, and $R$ denote the hull, rudder, and propeller, respectively.

Due to the operation conditions in the berthing phase, the hydrodynamic forces and moments around the midship acting on the ship hull were investigated at a low speed with a wide range of drift angles [2]. Thus, the equation of motion considers some of the high-order hydrodynamic coefficients and the hydrodynamic forces and moments caused by the hull, which are expressed as follows:

$$\begin{array}{l}{X}_H=X_{\dot{u}}\dot{u}+X_{u\left|u\right|}u\left|u\right|+X_{vv}{v}^2+X_{r\mathrm{r}}{r}^2+X_{v\mathrm{r}}v\mathrm{r}\\ {\mathrm{Y}}_H=Y_{\dot{v}}\dot{v}+Y_{\dot{r}}\dot{r}+Y_{v}v+Y_{vvv}{v}^3+Y_{vvvvv}{v}^5+Y_{r}r+Y_{r\left|r\right|}r\left|r\right|+Y_{vvr}{v}^{2}r+Y_{vrr}v{r}^2\\ N_H=N_{\dot{v}}\dot{v}+N_{\dot{r}}\dot{r}+N_{v}v+N_{uv}uv+N_{vvv}{v}^3+N_{uvvv}u{v}^3+N_{r}r+N_{r\left|r\right|}r\left|r\right|+N_{vvr}{v}^{2}r+N_{vrr}v{r}^2\end{array}$$

(2)

The hydrodynamic coefficients were described under the Taylor series expansion in terms of surge velocity, sway velocity, and yaw rate.

The ship model was equipped with twin-propeller and twin-rudder systems [18]. The thruster model is expressed as follows:

$$\begin{array}{l}{X}_P=\left(1-t\right)\rho D^4\left[{\left(n^P\right)}^2{K}_T^P{J}_P^P+{\left(n^S\right)}^2{K}_T^S{J}_P^S\right]\\ Y_P=0\\ N_P=y_P\left(1-t\right)\rho D_P^4\left[{\left(n^P\right)}^2{K}_T^P{J}_P^P-{\left(n^S\right)}^2{K}_T^S{J}_P^S\right]\end{array}$$

(3)

where $D$ is the propeller diameter; $n$ is the propeller revolutions per minute; $t$ denotes the thrust deduction factor; $y_P$ is the lateral position of the propeller from the centerline; and the superscripts $S$ and $P$ denote the side of the propeller (port and starboard).

The thrust coefficient $K_T$ is described as the function of the advanced ratio coefficient $J_P$ that was obtained through the propeller open water test

$$K_T=k_0+k_1{J}_P+k_2{J}_P^2$$

(4)

The parameter required for the estimation of thrust is given as

$$\begin{array}{l}{J}_P^{P,S}=\frac{u_P^{P,S}}{n^{P,S}{D}_P}\\ u_P^{P,S}=\left(1-w_P^{P,S}\right)u\\ w_P^{P,S}=w_{P0}\exp \left(-C_P{v^{\prime }}_P^2\right)\\ v_P=v+x_{P}r\\ C_P^P=C_P^{-} \mathrm{and} C_P^S=C_P^{+} \mathrm{when} {\beta }_P>0\\ C_P^P=C_P^{+} \mathrm{and} C_P^S=C_P^{-} \mathrm{when} {\beta }_P<0\end{array}$$

(5)

where the wake fraction at the propeller position $w_P$ was estimated using the wake fraction at the propeller in the straight motion $w_{P0}$; the geometrical inflow angle to the propeller position is denoted by ${\beta }_P$; $C_P^{+} \mathrm{and} C_P^{-}$ describe the wake-changing coefficients for plus and minus ${\beta }_P$ due to lateral motion; and $x_P$ denotes the longitudinal position from the midship.

Forces and moments due to the steering system for the twin rudder were calculated based on the normal force $F_N$ and are expressed as

$$\begin{array}{l}{X}_R=-\left(1-t_R\right)\left(F_N^P+F_N^S\right)\sin \delta \\ Y_R=\left(1+a_H\right)\left(F_N^P+F_N^S\right)\cos \delta \\ N_R=\left(x_R+a_H{x}_H\right)\left(F_N^P+F_N^S\right)\cos \delta -\\ \hspace{1em}\hspace{1em}\hspace{0.17em}{y}_R\left(1-t_R\right)\left(F_N^P-F_N^S\right)\sin \delta \end{array}$$

(6)

where the normal force acting on the rudder is described as follows (Equation (7)).

$$F_N^{P,S}=\frac{1}{2}\rho A_R{\left(U_R^{P,S}\right)}^2{f}_a\sin {\alpha }_R^{P,S}$$

(7)

The parameters required for estimating the rudder forces and moment during the maneuver are given as

$$\begin{array}{l}{U}_R^{P,S}=\sqrt{{\left(u_R^{P,S}\right)}^2+{\left(v_R^{P,S}\right)}^2}\\ f_a=\frac{6.13\Lambda }{\Lambda +2.25}\\ {\alpha }_R^{P,S}=\delta -{\tan }^{-1}\left(\frac{v_R^{P,S}}{u_R^{P,S}}\right)\\ u_R^{P,S}=\epsilon u_R^{P,S}\sqrt{\eta {\left[1+\kappa \left(\sqrt{1+\frac{8K_T^{P,S}}{\pi {\left(J_P^{P,S}\right)}^2}}-1\right)\right]}^2+\left(1-\eta \right)}\\ v_R^{P,S}={\gamma }_R^{P,S}\left(v+l_{R}r\right)\\ {\gamma }_R^P={\gamma }_R^{-} \mathrm{and} {\gamma }_R^S={\gamma }_R^{+} \mathrm{when} {\beta }_R>0\\ {\gamma }_R^P={\gamma }_R^{+} \mathrm{and} {\gamma }_R^S={\gamma }_R^{-} \mathrm{when} {\beta }_R<0\end{array}$$

(8)

where $F_N$ is the normal rudder force; $t_R$, $a_H$, and $x_H$ are the steering resistance deduction factor, the rudder increase factor, and the position of an additional lateral force component, respectively; $U_R$ is the resultant rudder inflow velocity; $f_a$ is the rudder lift gradient coefficient; $\Lambda$ is the rudder aspect ratio; ${\alpha }_R$ is the effective inflow angle to the rudder; $u_R$ and $v_R$ are longitudinal and lateral inflow velocity components to the rudder; $\epsilon$ is a ratio of a wake fraction at the propeller and rudder position; $\gamma$ is the flow straightening coefficient; and ${\beta }_R$ is the effective inflow angle to the rudder in the maneuvering motion.

2.3. Hydrodynamic and Interaction Coefficients

Previous studies were carried out by research groups at Changwon National University on hydrodynamic properties under operating conditions [19]. Experiments were conducted on the Korean autonomous surface ship (KASS) model, a ship model used in the project carried out by many universities and research institutes to develop autonomous ships. The main characteristics and the shape of the ship’s model are shown in

Table 1. Principal dimensions.

Item (Unit)	Value
Length perpendicular, $L_{pp}$ ($m$)	22.000
Breadth, $B$ ($m$)	6.000
Draft, $T$ ($m$)	1.250
Displacement Volume, $V$ ($m^3$)	86.681
Rudder area, $A_R$ ($m^2$)	0.518
Rudder span, $H_R$ ($m$)	0.900
Propeller diameter, $D_P$ ($m$)	0.950

and Figure 2 The cross-comparison of the results between the previous studies and [20] shows similarities in the results. Figure 3 shows the comparison results of the turning maneuverability at a rudder angle of 35 degrees at three and six knots. Similarly, in order to obtain the feasible path for the automatic berthing task, the high-accuracy equation of motion and hydrodynamic coefficients of the USV should be investigated carefully using the Korean autonomous surface ship (KASS) model. In this paper, the hydrodynamic coefficients were estimated using the captive model test at Changwon National University and compared with the CFD method as presented in [21]. The coefficients relative to only the surge velocity were estimated through the resistance test. The hydrodynamic coefficients related to surge and sway velocity for forces and moments were estimated through a static drift test. The hydrodynamic coefficients related to the yaw rate were estimated using the circular motion test and the hydrodynamic coefficients related to the combined effect of sway velocity and yaw rate were estimated using the combined circular motion with drift test. The added mass and interaction coefficients were selected from [21]. The summary of hydrodynamics and interaction coefficients are shown in

Table 2. Hydrodynamic force and moment coefficients.

$\mathbf{Hull} (\times 10^{-5}$)
$X_{\dot{u}}$	−81	$Y_{\dot{v}}$	−1034	$Y_{\dot{v}}$	64
$X_{uu}$	−627	$Y_{\dot{r}}$	−126	$Y_{\dot{r}}$	−33
$X_{vv}$	−407	$Y_v$	−2610	$N_v$	−130
$X_{rr}$	675	$Y_{vvv}$	−3530	$N_{uv}$	−513
$X_{vr}$	226	$Y_{vvvvv}$	3080	$N_{vvv}$	−2
		$Y_r$	390	$N_{uvvv}$	−138
		$Y_{r\left|r\right|}$	−47	$N_r$	−178
		$Y_{vrr}$	−2170	$N_{r\left|r\right|}$	−253
		$Y_{vvr}$	−3590	$N_{vrr}$	−420
				$N_{vvr}$	−1830

and

Table 3. Interaction coefficients.

Propeller and Rudder
$1-t_R$	0.934	$1+a_H$	0.702	${\gamma }_R^{+}$	0.342
$\epsilon$	0.960	$C_P^{+}$	−2.713	${\gamma }_R^{-}$	0.634
$\kappa$	0.695	$C_P^{-}$	11.211

.

2.4. Maneuverability

To define the problem, it is necessary to assess the maneuverability of the ship. Understanding the maneuverability makes it reasonable to determine where to start the automatic berthing process. The maneuvering simulation under low speed (1 knot) was conducted to investigate the ship-maneuvering characteristics in the port environment. Figure 4 shows the trajectory of the turning circle test at 35 degrees of a rudder angle at 1 knot. The simulation results show that this can easily turn into a range of approximately 3$L_{PP}$. Thus, the berthing area should be greater than three times that of $L_{PP}$ from the berthing point. The maneuvering characteristics are shown in

Table 4. Turning maneuverability characteristics.

	Starboard Turning	Port Turning
Advance ($L_{PP}$)	2.681	2.680
Transfer ($L_{PP}$)	1.114	1.115
Turning Radius ($L_{PP}$)	2.625	0.623
Tactical diameter ($L_{PP}$)	2.935	2.932

.

3. Path-Planning Approach

In the last few decades, significant advancements have been made in the field of artificial intelligence, particularly reinforcement learning, a subfield of machine learning, which falls under the machine learning category. Reinforcement learning involves training data by assigning rewards and punishments based on behavior and state. Unlike supervised and semi-supervised learning, reinforcement learning does not rely on pairs of input data or true results and it does not explicitly evaluate near-optimal actions as true or false. As a result, reinforcement learning offers a solution to tackle complex problems, including the control of robots, self-driving cars, and even applications in the aerospace industry. A noteworthy advancement in reinforcement learning is the introduction of a twin delayed deep deterministic policy gradient in 2018. TD3 is an effective model-free policy reinforcement learning method. The TD3 agent is an actor–critic reinforcement learning agent that optimizes the expected long-term reward. Specifically, TD3 builds upon the success of the deep deterministic policy gradient algorithm developed in 2016. DDPG remains highly regarded and successful in the continuous action space, finding extensive applications in fields such as robotics and self-driving systems.

However, like many algorithms, DDPG has its limitations, including instability and the need for fine-tuning hyperparameters for each task. Estimation errors gradually accumulate during training, leading to suboptimal local states, overestimation, or severe forgetfulness on the part of the agent. To address these issues, TD3 was developed with a focus on reducing the overestimation bias prevalent in previous reinforcement learning algorithms. This is achieved through the incorporation of three key features:

• The utilization of twin critic networks, which work in pairs.
• Delayed updates of the actor.
• Action noise regularization.

By implementing these features, TD3 aims to enhance the stability and performance of reinforcement learning algorithms, ultimately improving their applicability in various domains.

3.1. Conception

The path-planning algorithm in this paper was performed using the KASS model mentioned in Section 2.3. The port selected was the Busan port, whose geometry is shown in Figure 5. The objective of this result is to use the TD3 (the pseudocode as shown in Algorithm 1) for training the model that can generate the path for the berthing process. First, the TD3 algorithm trains the model with the combination of maneuvering simulations suggested by the MMG model. This approach allows for the integration of realistic ship motion dynamics into the training process. Then, this model is used to generate the desired path for the berthing task by inputting the state of the ship and predicting the control signal $n$ (propeller speed) and $\delta$ (rudder angle), based on the input ship state $s(x,y,\psi ,u,v, \mathrm{and} r)$. The concept of path planning for automatic berthing tasks is shown in Figure 6.

Algorithm 1: Pseudocode TD3 Algorithm
1	Initialize the critic network $Q_{\varphi 1},Q_{\varphi 2}$ and actor-network ${\mu }_{\theta }$ with random parameter ${\varphi }_1,{\varphi }_2,\theta$
2	Initialize the target parameter to the main parameter ${\theta }_{t\arg }\leftarrow \theta ,{\varphi }_{t\arg 1}\leftarrow {\varphi }_1,{\varphi }_{t\arg 2}\leftarrow {\varphi }_2$
3	For $t=0$ to $T-1$ do:
4	Observe the state $s$ of the environment and choose the action $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}a=clip\left({\mu }_{\theta }\left(s\right)+\epsilon ,a_{low},a_{high}\right) \mathrm{where} \epsilon ~N$
5	Execute action $a$ in the TD3 environment to observe the new state $s^{\prime }$, reward $r$, and done signal that gives the signal to stop training for this step.
6	Store training set $\left(s,a,r,s^{\prime },d\right)$ in replay buffer D
7	If $s^{\prime }$ taking to the goal point, reset the environment state
8	If it is time for an update: For $j$ in range (custom decided) do:
9	Randomly sample a batch of transitions $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}B=\left\{\left(s,a,r,s^{\prime },d\right)\right\} \mathrm{from} D$
10	Compute Target actions: $\begin{array}{l}{a}^{\prime }\left(s^{\prime }\right)=clip\left({\mu }_{\theta t\arg }\left(s^{\prime }\right)+clip(\epsilon ,-c,c),a_{low},a_{high}\right) \\ \mathrm{where} \epsilon ~\left(N,\sigma \right)\end{array}$
11	Compute targets: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}y\left(r,s^{\prime },d\right)=r+\gamma \left(1-d\right)\underset{i=1,2}{\min }{Q}_{\varphi t\arg i}\left(s^{\prime },a^{\prime }\left(s^{\prime }\right)\right)$
12	Update Q-functions using gradient descent: $\begin{array}{l}\hspace{1em} \hspace{1em} \hspace{1em} \hspace{1em} \hspace{1em} {\nabla }_{\varphi i}\frac{1}{\left|B\right|}{\displaystyle \sum _{\left(s,a,r,s\prime ,d\right)\in B}^{}{\left(Q_{\varphi i}\left(s,a\right)-y\left(r,s\prime ,d\right)\right)}^2 }\\ \hspace{1em} \hspace{1em} \hspace{1em} \hspace{1em} \hspace{1em} \hspace{1em} \mathrm{where} \mathrm{i}=1,2\end{array}$
13	If $j$ mod policy delay == 0 then: Update policy by the one-step deterministic policy gradient ascent using $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\nabla }_{\theta }\frac{1}{\left|B\right|}{\displaystyle \sum _{s\in B}{Q}_{\varphi 1}\left(s,{\mu }_{\theta }\left(s\right)\right)}$
14	Update target networks: $\begin{array}{l}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\varphi }_{t\arg .i}\leftarrow \rho {\varphi }_{t\arg .i}+\left(1-\rho \right){\varphi }_i \mathrm{where} \mathrm{i}=1,2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\theta }_{t\arg }\leftarrow \rho {\theta }_{t\arg }+\left(1-\rho \right)\theta \end{array}$
15	End if End for End if End until convergence

3.2. Setting for Reinforcement Learning

In this section, the parameters and variables for the TD3 algorithm to deal with the automatic berthing task were set as follows:

• Observation space and state: The observation space and state were defined as the set of physical velocity, position, and orientation. The state vector $s(x,y,\psi ,u,v, \mathrm{and} r)$ includes the position $x$,$y$ as the element. The orientation $\psi$ is the heading angle. The linear velocity is $u$,$v$ and the angular velocity is $r$;
• Action: The control action includes the control input of the thruster (revolution of propeller) and steering (rudder angle) system. The action signal is continuous in the range [−1, 1], where [−1, 1] = [−300, 100] rpm represents the thrust system and [−1, 1] where [−1, 1] = [−35, 35] is the degrees for the steering system.
• Reward function: This plays a crucial role in the design of a reinforcement learning application. It serves as a guide for the network training process and helps optimize the model’s performance throughout each episode. If the reward function does not accurately capture the objectives of the target task, the model may struggle to achieve desirable performance.

  $$\begin{array}{l}{r}_S=r_i-r_{i-1}\\ r=r_{Dist}+r_{LinVel}+r_{AngVel}+r_{Heading}\\ r_{Dist}=-100\sqrt{x^2+y^2}\\ r_{Heading}=-1000\left|{\psi }_{T\arg et}-\psi \right|\\ r_{LinVel}=-2000\sqrt{u^2+v^2}\\ r_{AngVel}=-1000\left|r\right|\end{array}$$

In this paper, based on the boundary state, the weights 100, 1000, 2000, and 1000 were assigned to the weight of distance, heading, linear, and angular velocity, respectively. The reward value was described as the sum of the reward in each time step. It received a positive value if the state variable changed to the required value and vice versa. Furthermore, for the faster convergence of rewards in the first stage of the berthing task, the distance is more prior to the target state than the speed and heading. Thus, the reward function of distance is multiplied by the distance coefficients. It is (1.1-Distance coefficients) in the case of the reward function of the heading, resultant velocity, and yaw rate. The reward coefficients are described in Figure 7.

• Environment: The environment receives the input as the control input and the state then returns the ship’s new state and the reward for this action. The environment function was built based on a maneuvering simulation that uses the MMG model as a mathematical simulation.
• Agent: The hyperparameters for the TD3 model were selected as follows: the number of hidden layers was set as two layers with 512 units for each. The learning rate for the actor and critic networks α and β was set to 0.0001. The discount factor $\gamma$ was 0.99. The soft update coefficient $\tau$ was 0.005. The batch size was 128. This training process was set to 20,000/50,000 steps for the warmup of the model with the exploration noise set in

  Table 5. Exploration rates.

  Item	Value (Exploration Rate ϵ)
  Step [0–5000]	0.5
  Step [5000–10,000]:	0.4
  Step [10,000–15,000]	0.3
  Step [15,000–20,000]	0.2
  Step [20,000–50,000]	0.1

  .

3.3. Boundary Conditions

The simulations were performed at Busan port, with the satellite capture as in Figure 7. The geometry of this port was simplified as Figure 8. Considering the geometry of this port, two cases of berthing were selected to investigate the path-plan-generating system.

Case 1: Parallel berthing task in a 30.5 × 202 m water area. In this case, the ship’s initial states were generated randomly, as described in

Table 6. Initial state values (case 1).

Item	Value
$x_0 \left(m\right)$	[−20, 20]
$y_0 \left(m\right)$	[−10, 10]
${\psi }_0 (°)$	[−5, 5]
$u_0 \left(knot\right)$	1
$v_0 \left(knot\right)$	0
$r_0 (°/s)$	0

. The berthing point was assumed, as shown in

Table 7. Target point (case 1).

Item	Value
$x \left(m\right)$	180 $\pm$ 2
$y \left(m\right)$	20 $\pm$ 0.5
$\psi (°)$	0 $\pm$ 3
$u \left(m\right)$	0 $\pm$ 0.1
$v \left(m\right)$	0 $\pm$ 0.05
$r (°/s)$	0 $\pm$ 1

. The berthing task can be considered a success if the ship state is in the range of the values described in Table 7. The boundary area is described as a Figure 8a and

Table 8. Boundary values (case 1).

Item	Value
$x \left(m\right)$	[−20, 182]
$y \left(m\right)$	[−10, 20.5]

.

Case 2: Perpendicular berthing task in a 42 × 170.5 m water area. In this case, the ship’s initial states were generated randomly, as described in

Table 9. Initial state values (case 2).

Item	Value
$x_0 \left(m\right)$	[−20, 20]
$y_0 \left(m\right)$	[−10, 10]
${\psi }_0 (°)$	[−5, 5]
$u_0 \left(knot\right)$	1
$v_0 \left(knot\right)$	0
$r_0 (°/s)$	0

. The berthing point was assumed, as shown in

Table 10. Target point (case 2).

Item	Value
$x \left(m\right)$	150 $\pm$ 0.5
$y \left(m\right)$	30 $\pm$ 2
$\psi (°)$	90 $\pm$ 3
$u (m/s)$	0 $\pm$ 0.1
$v (m/s)$	0 $\pm$ 0.05
$r (°/s)$	0 $\pm$ 1

. The berthing task can be considered a success if the ship state is in the range of the values described in Table 10. The boundary area is described as a Figure 8b and

Table 11. Boundary values (case 2).

Item	Value
$x \left(m\right)$	[−20, 150.5]
$y \left(m\right)$	[−10, 32]

.

4. Simulation Results and Discussion

The USV is considered to have successful berthing if it approaches the target berthing point with an error within the allowable range established in Section 3.3. The training process stops when the number of training episodes reaches 50,000.

Figure 9 and Figure 10 sequentially present a set of information that includes the trajectory, surge, sway, yaw rate, heading angle, and control input as the propeller revolution and rudder angle. The USV starts from the random state shown in Section 3.3. Under the automatic control of the TD3 model, the ship has successfully berthed at the target location. The ship states gradually change from the initial state to the required range. In particular, the combination with the adjusted reward function made the ship berthing process happen faster and more optimally. The time series of surge velocity was shown in the first phase of the berthing process. The ship’s speed was increased to reduce the distance to the berthing point. However, the sway velocity and yaw rate did not change much in this phase. In the last phase of the berthing process, the importance of surge velocity, yaw rate, and heading angle is much more important than the distance in the first phase. So, the change in this value seems more sudden to adapt to the required value of the berthing process.

Figure 11a,b shows the learning performance of TD3 case 1 with 50,000 episodes. Although the warmup episode number is 20,000, the average reward shows that the model seems to be successfully berthing and stable before the warmup phase is finished. This demonstrates that TD3 provides good reinforcement learning for the automatic berthing process. To account for the difference in penalty values affecting the training process, too high a penalty will cause the model to misjudge the states, for instance, in cases where the vessel has moved reasonably close to the berthing position. However, it then collides with the wall and receives a heavy point deduction, causing the model to judge that process as wrong and try actions other than that process. This makes the training process longer. These results show that the method proposed in this paper has a high performance and success rate with a low penalty value.

The simulation results show that the combination of TD3 and the maneuvering simulation proposes a powerful and accurate system for the automatic berthing process. Comparing the shape of the average reward to the results shown in [15] demonstrates that the stability of the TD3 algorithm is better than the older algorithm in the field of reinforcement learning. In particular, the method proposed in this paper is easier to apply because of the ability to learn, explore, and optimize the policy automatically. It can be used for another ship model if we know the ship’s hydrodynamic characteristics.

However, the limitations in this paper are evident. Firstly, the simplification of the port condition: due to the simplicity of the model, the effect of the disturbance was ignored in the simulation. This causes non-accuracy if there is wind or waves. The second limitation of this approach is the simplification of the obstacles and port geometry. It has a significant effect on the determination of the initial state and the berthing point. The presence of moving obstacles can make the training time increase significantly. So, the method proposed in this paper should only be used in the determined port and not for moving obstacles. Finally, this paper proposed the initial development of the path planning system. The results are performed based on maneuver simulation. The accuracy and performance of this approach in real-world operations need to be carefully considered and evaluated.

Although the approach in this paper uses the newest technique in reinforcement learning at present, the performance of this method should be investigated and compared carefully.

5. Conclusions and Remarks

In this study, the Korea autonomous surface ship (KASS) model was selected as the target ship to perform the training for path planning for the autonomous berthing task. A mathematical model and the hydrodynamics coefficients suggested in previous research conducted at Changwon National University provided an accurate model for solving the motion of a slow ship. By performing the path-planning algorithm based on the combination of TD3 and a maneuvering simulation, the automatic berthing task could be conducted with the automatic berthing problem and stable performance using reinforcement learning.

Even though the high performance of the path-planning system was shown, the complex environmental disturbance in the port area needs to be included in the model. It takes more time to train the model but is a necessary factor in the real situation. Additionally, several algorithms based on control theory must be considered for faster convergence.