Human-Navigable Ship-Handling Support Using Improved Deep Deterministic Policy Gradient for Survey Line Tracking

Abstract

This study presents a human-navigable ship-handling support system that employs artificial intelligence (AI) for survey line tracking. AI was developed using the Deep Deterministic Policy Gradient (DDPG), a type of deep reinforcement learning (DRL), and was evaluated through experiments conducted with a research vessel. The experiments revealed several issues inherent to DRL that required improvement. The first issue was the asymmetry observed in the policy learned through the DDPG. To address this, a learning approach that utilizes symmetric training data and symmetry-constrained actor and critic neural networks was proposed. The second issue was excessive steering during tracking maneuvers. To mitigate this, an objective function for actor learning that incorporates a cost term to suppress the magnitude of actions was proposed. The third issue was the frequent oscillation of actions. To resolve this, improved conditioning for action policy smoothing was introduced in the objective function to smooth actions appropriate to the situation. A subsequent experiment at sea was conducted to evaluate the improved AI-based ship-handling support system. As a result, precise path tracking performance with minimal operator discomfort and smooth control actions was achieved through manual ship handling guided by AI-generated instructions under actual sea conditions.

1. Introduction

Japan relies on other countries for most of its resources. With land resources becoming depleted, the focus is shifting toward marine resources. Numerous valuable resources, such as rare earth elements and methane hydrates, exist on the seabed in Japan’s exclusive economic zone. The development of such seabed resources requires surveying a wide sea area. Moreover, seabed investigations are essential for predicting natural hazards such as subduction zone earthquakes and submarine volcanic eruptions. Offshore wind power generation, which does not produce CO₂ emissions, has also attracted attention as a measure against global warming. Seabed surveys are indispensable for designing the mooring of floating offshore wind turbines. Consequently, ocean surveys are becoming increasingly important; however, they demand substantial time and cost due to the vast areas that must be surveyed. For this purpose, careful ship handling to precisely track survey lines over a long period of time is required. Therefore, it is necessary to develop human-navigable ship-handling instructions for seafarers to use as practical support.

Proportional–integral–differential (PID) control is commonly employed for line tracking maneuvers. A control output is calculated with constant gains for the value of the error, the sum of the time series of errors, and the changing rate of errors. However, it is challenging to apply PID control to ships with multiple actuators. It is also difficult to achieve precise control under dynamic variations in disturbances and tracking targets. This is because the gains are fixed and tuned to maximize the average performance in advance. In addition, survey line tracking control needs to anticipate a ship’s behavior and changes in the target line in advance. Early action is necessary because the inertial effect is large, and a ship does not turn as soon as steering begins. Such prediction-based control is difficult to achieve using PID control, so model predictive control (MPC) is usually used instead [1,2,3]. MPC is based on motion prediction; therefore, it can reduce overshoots and oscillations of the control variable. MPC outputs the optimized control output within a predicted horizon. However, the computational cost of this process is high; therefore, online control via MPC is difficult in cases where complex control targets and/or constraints are involved [4]. Furthermore, MPC obtains the control output using model predictions, so errors in the maneuvering model could degrade control performance. Due to difficulties in eliminating model errors, control schemes capable of compensating for discrepancies between numerical models and real-world dynamics have been studied [5,6,7,8]. The adaptability of control schemes to problems involving model errors and disturbances has been demonstrated in previous studies. Although these methods are well established in control theory, designing constraints that satisfy multiple objectives remains challenging. Human-navigable control requires the simultaneous consideration of tracking accuracy, control smoothness, robustness to noise, and real-time capability.

Artificial intelligence (AI) has advanced rapidly in recent years. In a previous study, a multilayer neural network was incorporated into an existing control law to estimate model uncertainty, and the performance was enhanced [9]. Another approach that uses AI technology is to develop a control law based directly on deep reinforcement learning (DRL). DQN, a pioneering method in DRL, received a higher score in Atari games than a human player [10], demonstrating the potential of AI to exceed human capabilities in specific tasks. Thereafter, many improved methods and algorithms for DRL have been studied and applied in various research fields, including line tracking problems [11,12,13,14]. These studies demonstrated that AI (a well-learned model using DRL) is effective for the development of autonomous line tracking control. DRL optimizes control policies by considering future vessel motion and target line changes through a vast trial-and-error simulation. The advantage of AI is that it can achieve more precise and flexible control compared to PID control; furthermore, online control can be achieved due to its superior speed compared to MPC and its robustness against model errors. Whereas MPC computes optimal control using a maneuvering model, DRL learns the optimal action directly from data; thus, it can adapt to system dynamics even when model errors exist, because the input data are updated as state feedback.

However, AI-based control systems suffer from certain limitations. One limitation is that the process of decision-making is not clear because its criterion is not designed by humans; rather, AI learns autonomously. Automatic control systems or navigation support systems are preferred as white boxes to increase user trust. Although explainable AI (XAI) has been studied to address this issue [15,16], existing methods cannot fully clarify the basis of decision-making. Another problem with AI is that outputs can be highly sensitive to small fluctuations in input data. Because the observation data of a ship are affected by the ship’s surge, sway, yaw, roll motion, and observation noise, AI’s perceived state oscillates frequently. Consequently, the control outputs change over time, even though the actual ship state does not change significantly. Such behavior can reduce ride comfort, damage cargo, strain steering gears, and reduce user confidence. Event-triggered control is a method used to reduce the frequency of output updates [17]. However, when the threshold is set to a large value in order to decrease the update frequency, the deviation at the moment of control updating becomes large. Consequently, it is difficult to simultaneously reduce both the frequency of output updates and the magnitude of the output change in a single update step. In addition, the timing of event occurrences is difficult for human operators to predict. Therefore, event-triggered control is not suitable for the development of human-navigable control systems.

Another issue is the sensitivity to observation noise, which is problematic for the robustness of AI systems. Furthermore, the control algorithms proposed in previous studies were validated only through numerical simulations; therefore, the performance of maneuvering control systems trained via numerical simulation in a real-world environment has not been clarified. Addressing these DRL-related issues is essential for practical deployment.

This study proposes a DRL-based control system for achieving precise and human-navigable survey line tracking using the improved Deep Deterministic Policy Gradient (DDPG). In this study, the term “human-navigable” is defined as the ability of a human operator to perform operations in the same manner as in normal manual operation while following AI’s instructions. To develop human-navigable AI, it is essential that the control instructions vary smoothly and at a frequency comparable with that of a human operator, that the resulting control actions are similar to those of a human operator, and that the control accuracy is equal to or greater than that achieved by a human operator. The features of the proposed scheme include symmetric maneuvering control, suppression of unnecessary excessive control, and infrequent and smooth control. Symmetric maneuvering control is achieved through a symmetry-constrained neural network architecture. Human-navigable, infrequent, and smooth control can be achieved by devising an objective learning function. These algorithms can contribute to the development of human-like autonomous controls and human-navigable ship-handling support. First, the effectiveness of each algorithm was verified through numerical experiments. Second, an onboard survey line tracking system incorporating all the algorithms was developed for the research vessel. Finally, the developed system was validated through experiments. The results confirmed that symmetric, smooth, and precise line tracking control can be achieved under real sea conditions.

2. Algorithm

2.1. Reinforcement Learning

Reinforcement learning (RL) consists of an agent, environment, and interactions between them. In the learning process, the agent observes state $s_t$ at time $t$. The agent selects an action $a_t$ according to a policy ${\pi }_{\theta }$. Then, the environment transitions to the next state $s_{t+1}$ and the agent receives a reward $r_{t+1 }$ from the environment. This sequence is repeated, and the agent updates its policy. This process is illustrated in Figure 1.

Figure 1. Interaction between environment and agent in reinforcement learning.

In RL, the agent learns a policy that maximizes the future cumulative reward through trial and error. The cumulative reward is described by Equation (1). When the rewards at future steps are summed, they are discounted by a factor γ to account for the uncertainty of future rewards. Actions that will be highly rewarded in the future are learned using cumulative rewards. By experiencing various states, the agent learns a policy that can handle various situations.

$$R_t=\sum {\gamma }^n{r}_{t+1+n}$$

(1)

2.2. Deep Deterministic Policy Gradient

The Deep Deterministic Policy Gradient (DDPG) [18] consists of two neural networks (NNs). It addresses continuous action space using actor and critic networks. The actor network predicts the optimal action $a$ from state $s$. The critic network predicts cumulative rewards from state $s$ and action $a$. The actor network and the critic network are denoted as $\mu \left(s|{\theta }^{\mu }\right)$ and $Q\left(s,a|{\theta }^Q\right)$, respectively. The parameters of the critic network are updated by minimizing the loss function shown in Equation (2). The loss is a measure of the discrepancy between the predicted and received rewards. The actor network parameters are updated to maximize the critic network’s output, as described in Equation (3). During learning, separate target networks with the same architecture as the actor and critic networks are used for action selection and cumulative reward estimation. The parameters of these networks are updated according to Equation (4). Here, ${\theta }^{\mu ’}$ and ${\theta }^{Q’}$ denote the parameters of the target actor and target critic networks, respectively.

$$L={\displaystyle \frac{1}{N}}\sum _i{\left(r_i+\gamma Q^{’}\left(s_{i+1},{\mu }^{’}\left(s_{i+1}|{\theta }^{{\mu }^{’}}\right)|{\theta }^{Q^{’}}\right)-Q\left(s_{i+1},a_{i+1}|{\theta }^Q\right)\right)}^2$$

(2)

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N}}\sum _i{\left.{\nabla }_{a}Q\left(s,a|{\theta }^Q\right)\right|}_{s=s_i,a=\mu \left(s_i\right)}{\left.{\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right)\right|}_{s_i}$$

(3)

$$\left\{\begin{array}{c}{\theta }^{{\mu }^{’}}\leftarrow {\left(1-\tau \right)\theta }^{{\mu }^{’}}+\tau {\theta }^{\mu }\\ {\theta }^{Q^{’}}\leftarrow {\left(1-\tau \right)\theta }^{Q^{’}}+\tau {\theta }^Q\end{array}\right.$$

(4)

3. Learning

3.1. Learning Environment

The maneuvering motion model of the research vessel TAKAMARU, operated by the Japan Fisheries Research and Education Agency, was developed to describe ship motion in a numerical environment. The equations of motion are given in Equation (5), and the coordinate system of the maneuvering model is shown in Figure 2. To represent the hydrodynamic force components, the so-called MMG model [19] was employed. In the MMG model, hydrodynamic forces are decomposed into components with respect to the ship hull, propeller, rudder, and their interaction. Accordingly, the total force acting on the vessel is expressed as the sum of these components, as shown in Equation (6). Here, the subscripts H, R, and P denote the hull, rudder, and propeller, respectively. The coefficients appearing in the expressions for each force component were determined through captive model experiments. In this study, the maneuvering coefficients were identified with a circular motion test, and the maneuvering motion model was tuned to reproduce the measured motions observed in actual ship experiments. Further details of the MMG model can be found in the literature [19]. A list of the defined symbols in Equations (5) and (6) is shown in Table 1.

$$\left\{\begin{array}{c}\left(m+m_x\right)\dot{u}-\left(m+m_y\right)v_{m}r-x_{G}m{r}^2=X\\ \left(m+m_y\right)\dot{v_m}+\left(m+m_x\right)ur+x_{G}m\dot{r}=Y\\ \left(I_{ZG}+x_G^{2}m+J_z\right)\dot{r}+x_{G}m\left(\dot{v_m}+ur\right)=N_m\end{array}\right.$$

(5)

$$\left\{\begin{array}{c}X=X_H+X_R+X_P\\ Y=Y_H+Y_R\\ N_m=N_H+N_R\end{array}\right.$$

(6)

Figure 2. Coordination of MMG model.

Table 1. List of symbols.

$m$	Ship’s mass
$m_x, m_y$	Added masses of x axis direction and y axis direction, respectively
$I_{ZG}$	Moment of inertia of ship around center of gravity
$J_Z$	Added moment of inertia
$u$	Surge velocity at center of gravity
$v_m$	Lateral velocity at midship
$r$	Yaw rate
$x_G$	Longitudinal coordinate of center of gravity of ship
$X,Y,N_m$	Surge force, lateral force, and yaw moment around midship, ignoring added mass components
$X_H,Y_H,N_H$	Surge force, lateral force, and yaw moment around midship acting on ship hull, ignoring added mass components
$X_P$	Surge force due to propeller
$X_R,Y_R,N_R$	Surge force, lateral force, and yaw moment around midship by steering

Target lines were generated randomly by combining straight lines and arcs. The ranges of the arc radius and central angle were 50–200 m and 45–315°. The range of the straight line was 300–500 m. An example of a generated target line is shown in Figure 3. The learning time step was set to 5 s, and each training episode consisted of 240 steps.

Figure 3. Example of trajectory generated in learning environment.

3.2. Observation Setting

A state at time $t$ consists of information about the ship’s motion and the target line. Ship information includes longitudinal velocity, lateral velocity, rate of turn, drift angle, and rudder angle, denoted as $u_t$, $v_t$, $r_t$, ${\beta }_t$, and ${\delta }_t$, respectively. The target line is discretized at intervals corresponding to the distance traveled by the ship during one time step, i.e., $V_{t}dt$. The relative coordinates of the target points are included in the representation of the state. The first target point $T_0$ is defined as the intersection of the target line and is perpendicular from the ship to the target line. The number of target points was set to 10 in the experiment, determined according to the time constant of the maneuvering model. The current and four preceding states were used as the input data. The definition of the state is given by Equation (7), and the definition of the target points is shown in Figure 4.

Figure 4. Definition of target points $T_0\dots T_5$. The first target point $T_0$ is located at $\left(T_x^0,T_y^0\right)$.

$$s_t≔\left[u_t,v_t,r_t,{\beta }_t,{\delta }_t,T_x^0,T_y^0,T_x^1,T_y^1,\dots ,T_x^9,T_y^9\right]$$

(7)

The action of the actor is represented by the continuous rudder angle, bounded within the range of −35–35°.

3.3. Reward Setting

Line tracking control is achieved by maintaining the ship’s course parallel to the target line. Thus, a reward is given when deviations in distance and angle occur. The reward function is defined using a Gaussian function. It is calculated using Equation (8), where $d_t\left[\mathrm{m}\right]$ and ${\theta }_t[°]$ denote the deviation distance and deviation angle, respectively. In addition, a reward of +0.1 is assigned when these deviations decrease from one step to the next, ensuring the explicit goodness of the actions. The parameters constituting the reward function, such as 25, 30, and others, were determined so that AI exhibited good performance in both simulations and full-scale ship experiments.

$$R_t=0.1\times \left(2\times {0.5}^{\left({\left(d_t/25\right)}^2+{\left({|\theta }_t|/30\right)}^2\right)}-1\right)$$

(8)

3.4. Numerical Validation

The structure of the actor network and critic network is shown in Table 2 and Table 3. Dropout layers with a rate of 0.2 are used after non-output layers. Activations are represented by the ReLU function in hidden layers, the linear function in the output of the actor network, and the softsign function in the output of the critic network. The learning hyperparameters are presented in Table 4.

Table 2. Structure of hidden layers of actor network.

Layer	Number of Nodes
Fully Connected	128
Fully Connected	512
Fully Connected	512
Fully Connected	128
Fully Connected	1

Table 3. Structure of hidden layers of critic network.

Layer	Number of Nodes
Fully Connected	128
Fully Connected	512
Fully Connected	512
Fully Connected	128
Fully Connected	1

Table 4. Hyperparameters.

Optimizer	Adam
Learning rate	0.01
Memory size	120,000
Batch size	2048

The developed AI system was first evaluated via numerical simulation. An example result of the simulation of line tracking is shown in Figure 5. In the figure, the triangles indicate the ship positions at time intervals of 120 s. The average deviation distance from the target line is 2.0 [m], while the length of the ship is 29.5 [m]. The developed AI system demonstrates high precision in line tracking. In the next section, the performance of the developed AI system is validated with an actual ship.

Figure 5. Simulated trajectory (red line) and target line (dashed line).

3.5. Experimental Validation

3.5.1. Experiment Environment

A full-scale experiment was conducted in Tateyama Bay, Japan. The research vessel TAKAMARU, shown in Figure 6, is capable of receiving rudder commands from an external PC; however, most existing ships cannot be directly controlled by external systems. Therefore, the developed AI system was implemented not as an automatic controller but as a ship-handling support system. Specifically, the AI-generated rudder commands were displayed on a monitor, and a crew member manually steered the vessel in accordance with these instructions.

Figure 6. Research vessel “TAKAMARU”.

3.5.2. Clarification of AI’s Intent

A major issue in AI-based control systems is the opacity of the decision-making process. To enhance transparency, the MMG model was used to generate predicted ship motions based on the AI’s rudder commands, which were visualized on a display screen. The predicted horizon exceeded 100 steps (one step is 5 s) and was updated at a rate of 1 Hz. The user interface (UI) developed for the experiment is shown in Figure 7. The red marker indicates the current position of the ship, the blue line indicates the predicted trajectory, and the black line indicates the target line. The current rudder angle and the rudder angle indicated by the AI system are displayed together. The steering is manually controlled such that the current rudder follows the command rudder. It is clear that the AI system intends to steer the ship to the right to track the target line. The UI helps the operator to understand the AI system’s intention in real time.

Figure 7. Display of user interface.

3.5.3. Ship Experiment

Survey lines were used as the tracking target for the ship experiment. An example of a survey line tracking trajectory in actual ocean-bottom exploration is shown in Figure 8. The vessel followed each survey line sequentially, transitioning from the end of one line to the beginning of the next via a trackable interpolation curve. In this study, circular arcs were employed to construct these interpolation curves. Since ships have large inertia and cannot change their heading instantaneously, a margin was introduced between the straight line and the arc to enable a gradual approach parallel to the line. The radius of the interpolation arc, the margin length, and the minimum turning radius are denoted by $R$, $m$, and $R_{min}$, respectively. If the distance between adjacent survey lines is greater than $2R_{min}$, the lines are expanded outward by a margin $m$ and connected using a semicircular arc, as shown in Figure 9a. In this case, the radius $R$ is set to half of the distance between the lines. When the distance between the lines is smaller, interpolation is performed, as illustrated in Figure 9b, where the radius $R$ is set to $R_{min}$. In this case, a single semicircular arc with radius $R_{min}$ cannot be placed between the lines. Therefore, the first circular arc is introduced to create sufficient spacing, followed by the second arc that connects to the next line. These circular arcs are arranged geometrically. Since the radius of the interpolation circular arcs is set according to the maneuverability, trackable interpolation is achieved. The perpendicular lines should be tracked according to the purpose of the survey. In this case, the interpolation shown in Figure 9c was used. Trackable interpolation is achieved by determining $R$ according to the maneuverability. Based on the maneuvering characteristics of the vessel, the parameters were set to $R_{min}$ = 100 m and m = 30 m.

Figure 8. Example of trajectories of line tracking for ocean-bottom survey. Different colors indicate tracking results obtained on different days [20].

Figure 9. Interpolation of survey lines.

3.5.4. Results of Experiment

The trajectory and time series of the rudder angle are shown in Figure 10 and Figure 11. The gray and blue lines in Figure 10 represent the target line and the actual trajectory, respectively. The triangles indicate the ship positions at time intervals of 120 s.

Figure 10. Trajectory of ship experiment. The dashed line indicates the target line.

Figure 11. Time series of rudder angle.

4. Developing a Human-Navigable AI System

4.1. Symmetry of AI Control

Control systems designed by humans, such as PID controllers, are symmetrical. In contrast, the symmetry of the AI system is not ensured, which could lead to user suspicion and distrust. The asymmetry of AI control arises from the randomness of the initial parameter values of NNs and the bias of the learning data. Incorporating symmetry into DRL not only yields symmetric control behavior but may also improve learning efficiency. Therefore, this study considers DRL using symmetric training data and symmetry-constrained neural networks. A symmetrical state $\overline{s_t}$ is defined in Equation (9). The symmetry is lateral in the relative coordinates.

$$\left\{\begin{array}{c}{s}_t≔\left[u_t,v_t,r_t,{\beta }_t,{\delta }_t,T_x^0,T_y^0,T_x^1,T_y^1,\dots ,T_x^9,T_y^9\right]\\ \overline{s_t}≔\left[u_t,{-v}_t,-r_t,-{\beta }_t,{-\delta }_t,-T_x^0,T_y^0,-T_x^1,T_y^1,\dots ,-T_x^9,T_y^9\right]\end{array}\right.$$

(9)

First, symmetry is introduced into the training data. When sampling minibatch data from the replay buffer, symmetric data are generated from the sampled states, and both the original and symmetric data are included in the minibatch. This process is illustrated in Figure 12.

Figure 12. Creation of symmetric minibatch data.

Next, symmetry is incorporated into the NN architecture. The outputs of the actor network from the symmetric data should have signs opposite to those of the original data. The outputs of the critic network from the symmetric data should have the same values as those of the original data. The NN structure incorporating these symmetry constraints is shown in Figure 13. For the actor network, the symmetric input data $\overline{s}$ are created from the original input data $s$. Common hidden layers are applied to both datasets. The outputs of the hidden layers are $f\left(s\right)$ and $f\left(\overline{s}\right)$. Finally, the difference between $f\left(s\right)$ and $f\left(\overline{s}\right)$ is calculated as the output of the actor network. The activation of the output is a softsign function according to the DDPG. When the input state changes from $s$ to $\overline{s}$, the output of the actor network changes from $softsign\left(f\left(s\right)-f\left(\overline{s}\right)\right)$ to $softsign\left(f\left(\overline{s}\right)-f\left(s\right)\right)$. Thus, an output value with the opposite sign is obtained. In the critic network, the subtraction layer in the actor network is changed to the addition layer to make the output data from the symmetric input the same. The final activation is a linear function.

Figure 13. Structure of neural network considering symmetry.

Symmetric actions can be achieved solely through the use of a symmetric actor network; therefore, a symmetric critic network is not strictly required. However, incorporating symmetry into the critic network may further improve learning efficiency. To examine the effectiveness of symmetry, the following three scenarios were evaluated:

• ① Symmetric minibatch data.
• ② A symmetric actor network.
• ③ A symmetric actor network and a symmetric critic network.

The learning environment and reward settings were the same as those described in the previous section. The network architectures without symmetry correspond to those in Table 2 and Table 3, whereas the symmetry-constrained network architectures are presented in Table 5 and Table 6. The hyperparameters were the same as those listed in Table 4.

Table 5. Structure of hidden layers of actor network considering symmetry.

Layer	Number of Nodes
Fully Connected	128
Fully Connected	512
Fully Connected	512
Fully Connected	1

Table 6. Structure of hidden layers of critic network considering symmetry.

Layer	Number of Nodes
Fully Connected	128
Fully Connected	512
Fully Connected	512
Fully Connected	1

Under conditions of symmetry, learning was performed, and the results were compared with those of the conventional DDPG. The episode rewards and the critic network’s loss values during training are shown in Figure 14 and Figure 15. The blue, orange, green, and red lines represent the results of the conventional DDPG, the symmetric actor network, the symmetric actor network and symmetric critic, and the symmetric minibatch data, respectively. Figure 14 shows that the episode rewards for the symmetry-based approaches are generally higher than those for the conventional DDPG. In addition, incorporating symmetry reduced the critic network’s loss. These results indicate that symmetry contributes to improved learning performance.

Figure 14. Transition of episode rewards (circle markers) and moving averages (lines).

Figure 15. Transition of mean absolute error (circle markers) and moving averages (lines).

The line tracking simulation results shown in Figure 16 confirm that high-precision tracking can be achieved. The results obtained using a symmetric actor network exhibit a symmetric time series of the rudder angle. In contrast, learning with symmetric data alone did not fully achieve symmetry; in particular, symmetry was not maintained between steps 150 and 200. Based on these results, the line tracking performance is improved most effectively when both the actor and critic networks incorporate symmetry.

Figure 16. Trajectory and rudder angle in survey line tracking using symmetric NN.

4.2. Limitation of Action Changes

According to the results of DDPG validation, the AI-generated rudder command was changed from −35° to 35°. Such abrupt control actions are not typically executed by human operators and impose significant load on the steering gear. The frequency of changing rudder commands should be limited considering the limits of the actuator, such as the changing rate of the rudder angle and the reduction in the inertial force, which results in anxiety for passengers. However, a learning method that accounts for the limit on the number of changing actions has not yet been proposed.

An initial attempt was made to incorporate a penalty for changes in the rudder angle during learning, but the outcomes were not satisfactory. Therefore, this study proposes a method to explicitly consider limits on action changes within the objective function.

In the DDPG, the parameters of the actor network are updated to maximize the objective function. The objective function considering the limits on action changes, $J_{{\pi }_{\theta }}^{\Delta \mathrm{a}}$, is defined by Equation (10), where the first term represents the cumulative reward, which is the standard objective in the conventional DDPG. The second term, ${\lambda }_{\Delta a}{L}_{\Delta a}$, represents the restriction of action changes $\Delta a$, and ${\lambda }_{\Delta a}$ is the strength of the restriction. The parameters are updated to minimize the number of changing actions. The strength of the second term depends on its value and the gradient. Therefore, the function should have a large gradient if the number of changing actions exceeds the limit. In this study, the Swish function [21] is employed for this purpose, as shown in Equation (10). The resulting shape of the objective function is shown in Figure 17.

$$\left\{\begin{array}{c}{J}_{{\pi }_{\theta }}^{\Delta \mathrm{a}}=J_{{\pi }_{\theta }}-{\lambda }_{\Delta a}{L}_{\Delta a}\\ L_{\Delta a}={\displaystyle \frac{swish\left(\beta \left(\Delta a-{\Delta a}_{max}\right)\right)+\Delta a-swish\left(-\beta \Delta a_{max}\right)}{\Delta a_{max}-swish\left(-\beta \Delta a_{max}\right)}}\\ \Delta a=\left|{\pi }_{\theta }\left(s_t\right)-{\pi }_{\theta }\left(s_{t+1}\right)\right|\end{array}\right.$$

(10)

Figure 17. Loss function with respect to degree of action. Red line indicates zero and the limit.

The learning process was performed under three conditions, no limit, a limit of 20°/step, and a limit of 40°/step, to validate the proposed method. The results are shown in Figure 18. According to these results, the deviation distances were comparable across all conditions.

Figure 18. Comparison of trajectory and rudder command. The dashed line in the trajectory figure indicates the target line.

The distribution of the number of changing actions is shown in Figure 19a–c, and the cumulative ratio of the degree of rudder change is shown in Figure 19d. According to Figure 19d, imposing a limit reduces the frequency of large rudder angle changes. In more than 95% of the steps, the magnitude of action changes remains below the specified limit. Although the proposed method does not perfectly enforce the action change constraint, it effectively suppresses excessive changes. Based on these results, the proposed method can be used to incorporate action change limits into the learning process. In this study, the restriction was implemented using the Swish function, although the ReLU function could also be used.

Figure 19. Probability density of degree of rudder change.

4.3. Situational Action Policy Smoothing

NNs are highly sensitive to small variations in input data. This issue is well known not only in autonomous control but also in image recognition. In real sea conditions, measured data are affected by ship motions (e.g., rolling), sensor errors, and observation noise. If an AI system responds excessively to such variations, rudder commands may fluctuate significantly even when the vessel is moving along a straight line.

A smoothing method for DRL, Conditioning for Action Policy Smoothing (CAPS) [22], has been proposed to address this issue. In CAPS, the objective function of the actor network, $J_{{\pi }_{\theta }}^{CAPS}$, is described by Equation (11). The second and third terms are the temporal and spatial smoothing terms, respectively, with ${\lambda }_T$ and ${\lambda }_S$ controlling the strength of these smoothing terms. $D_T$ and $D_S$ denote the distance measures. The actor parameters are updated to maximize the cumulative reward using the first term. The second term encourages temporal smoothness by minimizing the difference between consecutive actions, ${\pi }_{\theta }\left(s_t\right)$ and ${\pi }_{\theta }\left(s_{t+1}\right)$, as shown in the second line of Equation (11). The third term suppresses sensitivity to input noise by minimizing the difference between the action for the observed state, ${\pi }_{\theta }\left(s_t\right)$, and the action for the noise-perturbed state, ${\pi }_{\theta }\left({\mathrm{s}}_{\mathrm{t}}^{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}\right)$. The noise distribution was determined from the experimental data.

$$\left\{\begin{array}{c}{J}_{{\pi }_{\theta }}^{CAPS}=J_{{\pi }_{\theta }}-{\lambda }_T{L}_T-{\lambda }_S{L}_S\\ L_T=D_T\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left(s_{t+1}\right)\right)\\ L_S=D_S\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left({\mathrm{s}}_{\mathrm{t}}^{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}\right)\right)\end{array}\right.$$

(11)

The temporal term in CAPS acts as a constraint on changes in the action. Although this helps suppress meandering on straight paths by stabilizing the actor output, it also delays rudder adjustments when transitioning from straight segments to curves. In other words, CAPS does not consider changes in the observed state. To address this, a state-dependent restriction is proposed.

The improved objective function is described by Equation (12). The second term is the restriction on temporal action changes. Since the number of changing actions is divided by the number of changing states, the limit is smaller when the observed states change significantly. This improved method is called Situation Action Policy Smoothing (SAPS). $L_T^a$, $L_T^s$, and $L_S$ are the differences between consecutive actions, consecutive states, and a noisy state, respectively. ${\lambda }_T^a$, ${\lambda }_T^s$, and ${\lambda }_S$ are the strength of the per-step action constraint, the relaxation of the constraint in response to state variations, and the strength of the constraint against noise., respectively.

$$\left\{\begin{array}{c}{J}_{{\pi }_{\theta }}^{SAPS}=J_{{\pi }_{\theta }}-{\displaystyle \frac{{\lambda }_T^a{L}_T^a}{{\lambda }_T^s{L}_T^s+1}}-{\lambda }_S{L}_S\\ L_T^a=D_T^a\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left(s_{t+1}\right)\right)\\ L_T^s=D_T^a\left(s_t, s_{t+1}\right)\\ L_S=D_S\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left(\overline{s_t}\right)\right)\end{array}\right.$$

(12)

Learning was performed using CAPS and SAPS to validate the proposed method. They were compared at the same restrictive strength. The objective function parameters are listed in Table 7. The trajectories of the simulation results are shown in Figure 20. It is shown that SAPS can smoothly follow both straight and curved lines with high accuracy, while CAPS causes delayed rudder movement in curved sections. These results demonstrate that SAPS achieves situational smoothing with improved accuracy.

Table 7. Parameters of CAPS and SAPS.

CAPS	SAPS
${\lambda }_S=0.02$	${\lambda }_S=0.02$
${\lambda }_T=0.02$	${\lambda }_T^a=0.02$
-	${\lambda }_T^s=1.0$

Figure 20. Comparison of trajectory and rudder angle between CAPS (red) and SAPS (blue). The dashed line in the trajectory figure indicates the target line.

Next, the conventional DDPG and the DDPG with SAPS were compared. The results are presented in Figure 21. The average change in the rudder angle per step decreased from 16.5°/step in the conventional DDPG to 0.73°/step with SAPS. This indicates that the proposed smoothing method is highly effective.

Figure 21. Comparison of trajectory and rudder angle between DDPG (red) and SAPS (blue). The dashed line in the trajectory figure indicates the target line.

5. Validation Using Ship Experiment

5.1. Results

In the previous section, methods for improving the DDPG were described. The proposed objective function and its parameters are given by Equation (13) and in Table 8. The architectures of the actor and critic networks are the same as those presented in Table 5 and Table 6.

$$\left\{\begin{array}{c}{J}_{{\pi }_{\theta }}^{\mathrm{\prime }}=J_{{\pi }_{\theta }}-{\lambda }_S{L}_S-{\displaystyle \frac{{\lambda }_T^a{L}_T^a}{{\lambda }_T^S{L}_T^S+1}}-{\lambda }_{\Delta a}{L}_{\Delta a}\\ L_S=D_S\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left(s_t^{noise}\right)\right)\\ L_T^a=D_T^a\left({\pi }_{\theta }\left(s_t\right),{\pi }_{\theta }\left(s_{t+1}\right)\right)\\ L_T^S=D_T^S\left(s_t,s_{t+1}\right)\\ L_{\Delta a}={\displaystyle \frac{swish\left(\beta \left(\Delta a-{\Delta a}_{max}\right)\right)+\Delta a-swish\left(-\beta \Delta a_{max}\right)}{\Delta a_{max}-swish\left(-\beta \Delta a_{max}\right)}}\\ \Delta a=\left|{\pi }_{\theta }\left(s_{t+1}\right)-{\pi }_{\theta }\left(s_t\right)\right|\end{array}\right.$$

(13)

Table 8. Parameters of proposed objective function.

${\lambda }_S$	0.02
${\lambda }_T^a$	0.02
${\lambda }_T^s$	1.0
${\lambda }_{\Delta a}$	0.02
$\Delta a_{max}$	1.0

A line tracking AI system was developed incorporating the proposed improvements and was validated under real sea conditions. The AI system was validated using survey lines with three types of interpolation. The results are shown in Figure 22, Figure 23 and Figure 24. In the trajectory figures, the triangles indicate the ship positions at time intervals of 120 s. During these experiments, environmental disturbances were mild (average tidal current: 0.4 knots; average wind speed: 2.0 m/s). The average deviation distances were 3.6, 4.2, and 4.6 m, while the maximum deviations were 8.3, 11.8, and 17.5 m. Since the maximum deviations were less than half the vessel length, these results demonstrate that high-precision line tracking was achieved in actual sea conditions. Moreover, changes in the rudder movement were smooth, which is favorable for operator comfort and practical use.

Figure 22. Results of ship experiment in calm conditions (wide survey line).

Figure 23. Results of ship experiment in calm conditions (narrow survey line).

Figure 24. Results of ship experiment in calm conditions (perpendicular survey line).

Additional experiments were conducted to evaluate the system’s performance under stronger disturbances (average tidal current: 0.8 knots; average wind speed: 6.5 m/s). The results are shown in Figure 25, Figure 26 and Figure 27. The average deviation distances were 3.5, 1.8, and 1.8 m, and the maximum deviations were 13.0, 6.6, and 6.1 m. The rudder movements remained smooth. These results indicate that the proposed method achieves precise and stable line tracking even under strong disturbances.

Figure 25. Results of ship experiment under strong disturbance (wide survey line).

Figure 26. Results of ship experiment under strong disturbance (narrow survey line).

Figure 27. Results of ship experiment under strong disturbance (perpendicular survey line).

5.2. Discussion

This paper proposes a steering control support system based on an improved DDPG algorithm and compares its performance with that of the conventional DDPG method and human helmsman. The former serves as a baseline reinforcement learning (RL) method. The latter represents human steering, which is the most commonly employed approach in actual ship operations and can be regarded as the best baseline for validating the effectiveness of the proposed method.

The conventional DDPG shown in Figure 10 and Figure 11 and the improved DDPG were compared. The trajectories and time series of the rudder angle are shown in Figure 28 and Figure 29. The triangles indicate the ship positions at time intervals of 120 s. The average change in rudder angle was 4.508°/step in the DDPG and 0.355°/step in the improved DDPG. Since the average number of rudder changes was also reduced by approximately 8%, it can be concluded that the proposed method is effective for achieving smooth rudder control without compromising tracking accuracy.

Figure 28. Trajectories (blue lines) of DDPG (top panel) and improved DDPG (bottom panel), and the target line (dashed line).

Figure 29. Time series of rudder command of DDPG (top panel) and improved DDPG (bottom panel).

Experiments testing line tracking by humans without AI support were also conducted using the same survey lines. The trajectory and the time series of the rudder angle are shown in Figure 30 and Figure 31. The triangles indicate the ship positions at time intervals of 120 s. For a straight line, line tracking can be achieved by making the latitude of the ship the same as the latitude of the survey line. However, for a curved or diagonal line, the latitude and longitude of the ship must be the same as those of the survey line. It was difficult for humans to control both the latitude and longitude. In contrast, the developed AI system tracked the curved lines, and thus, we can conclude that it is effective for automating survey line tracking. The average changes in the rudder angle per step under AI and human operation were 0.51 [deg/s] and 0.50 [deg/s], respectively. The AI generated human-like smooth control actions while achieving higher tracking accuracy. These results confirm that the developed AI system can provide human-navigable ship-handling support.

Figure 30. Comparing trajectories under AI operation (top panel) and human operation (bottom panel). The dashed line indicates the target line.

Figure 31. Comparing time series of rudder angle under AI operation (top panel) and human operation (bottom panel).

The results of all experiments conducted under the same conditions as those in Figure 22 and Figure 23 are shown in Figure 32. The experiments were conducted four times for both the wide and narrow survey lines, with each trial plotted in a different color in the figures, and all trials were performed by the same helmsman. As shown in Figure 32, similar trajectories were obtained across all attempts. Although the helmsman’s control inputs varied between trials, the results indicate that the proposed AI system effectively supports the helmsman in achieving a consistent level of tracking precision.

Figure 32. Trajectories of all experiments for wide and narrow survey lines. The dashed line indicates the target line.

5.3. Comparison of Line Tracking Between AI-Supported Human Control and AI Control

In the ship experiments described in the previous section, the developed AI system was implemented as a support system that displayed rudder commands to the operator. This approach achieved high-precision line tracking performance. In this section, the results of fully automated AI control are presented and compared with those of AI-supported human control.

In the AI control experiment, rudder commands were transmitted directly from an external laptop to the vessel’s steering system, which executed the corresponding rudder angles automatically. The trajectories and time series of the rudder angle for both AI-supported helmsman control and fully automated AI control are shown in Figure 33. The average deviation distances from the target path under AI-supported helmsman control and AI control were 9.79 [m] and 10.08 [m], respectively. According to the results, high-precision line tracking was achieved in both scenarios, and it is clear that the developed AI system can give steering instructions that are feasible for a human operator to follow. Moreover, even if the operator’s steering response is slightly delayed, the difference in performance remains minimal because the AI-based steering assistance incorporates state feedback.

Figure 33. Comparison of trajectories between full AI control and helmsman control supported by AI. The dashed line in the trajectory figure indicates the target line.

In conclusion, the developed AI system is effective both as a human-navigable support system and as an autonomous controller. Given that most existing ships cannot be directly controlled by external computers, the ability of the system to operate in human-navigable support mode provides a significant practical advantage and facilitates its deployment in current maritime operations. This capability is particularly valuable during the transitional period before fully autonomous ships become feasible.

6. Conclusions

This study addressed the limitations of deep reinforcement learning (DRL) in achieving human-navigable AI-based ship handling support in survey line tracking. First, the symmetry of maneuvering control was guaranteed by introducing symmetry-constrained actor and critic networks. Second, excessive control actions were suppressed by incorporating a penalty term into the objective function using the Swish function. Third, situational smoothing of the rudder actions was achieved by introducing improved conditioning for action policy smoothing. Ship experiments were conducted at sea to validate the proposed methods. The experiments demonstrated that the proposed method achieved smooth and precise control with an appropriate update frequency under real sea conditions. The AI-generated rudder commands were also found to be interpretable and manageable for human helmsmen to follow. Furthermore, comparison with the AI control experiments showed that line tracking was similarly smooth and accurate, confirming the effectiveness of the system as a human-navigable ship handling support tool.

The proposed ship-handling support tool using improved DRL for line tracking was validated at sea under natural disturbances. Further investigation of its performance under more severe environmental conditions is required to assess its suitability for practical deployment. In addition, it is necessary to quantitatively evaluate the human-friendliness of the proposed method using appropriate indices, such as the helmsman’s tension level and workload. Furthermore, incorrect or faulty data may be present in real-world environments. Therefore, incorporating fault-tolerant control (FTC) mechanisms, such as those reported in [23], to enhance the robustness of the proposed AI system under practical operating conditions is an important direction for future work.