Robust Decision-Making for the Reactive Collision Avoidance of Autonomous Ships against Various Perception Sensor Noise Levels

Abstract

Autonomous ships are expected to extensively rely on perception sensors for situation awareness and safety during challenging operations, such as reactive collision avoidance. However, sensor noise is inevitable and its impact on end-to-end decision-making has not been addressed yet. This study aims to develop a methodology to enhance the robustness of decision-making for the reactive collision avoidance of autonomous ships against various perception sensor noise levels. A Gaussian-based noisy perception sensor is employed, where its noisy measurements and noise variance are incorporated into the decision-making as observations. A deep reinforcement learning agent is employed, which is trained in different noise variances. Robustness metrics that quantify the robustness of the agent’s decision-making are defined. A case study of a container ship using a LIDAR in a single static obstacle environment is investigated. Simulation results indicate sophisticated decision-making of the trained agent prioritising safety over efficiency when the noise variance is higher by conducting larger evasive manoeuvres. Sensitivity analysis indicates the criticality of the noise variance observation on the agent’s decision-making. Robustness is verified against noise variance up to 132% from its maximum trained value. Robustness is verified only up to 76% when the agent is trained without the noise variance observation with lack of its prior sophisticated decision-making. This study contributes towards the development of autonomous systems that can make safe and robust decisions under uncertainty.

1. Introduction

1.1. Background

The maritime sector is currently experiencing a paradigm shift towards the new “shipping 4.0” era, characterised by its increased connectivity, digitalisation, and autonomy [1]. The main driving factor of this transformation is the vision for a more sustainable future [2] propelled by the emergence of key enabling technologies, including the internet of things [3], cloud computing [4], big data analytics [5], blockchain [6], cyber-physical systems [7], digital twins [8], additive manufacturing [9], and augmented and virtual reality [10], as well as artificial intelligence (AI) [11]. Owning to such technologies, conventional ships have been gradually adopting various degrees of autonomy, forming a new class of maritime autonomous surface ships (MASSs) [12] with the premise of unlocking new levels of sustainability via reducing fuel consumption, optimising route planning, eliminating accidents related to erroneous human decision-making, and more [13].

One of the preconditions to achieve higher degrees of autonomy MASSs that do not require human intervention is the advancement of sensor technologies used as their primary source of perceiving the surrounding environment [14], known as perception sensors [15]. Perception sensors are exteroceptive that measure values originated outside of the controlled system to perceive the surrounding environment, while localisation sensors are proprioceptive that measure values within the system to localise it with respect to the perceived environment [16]. In safety-critical operations, like collision avoidance, the role of perception sensors is pivotal not only for sensing the surrounding obstacles, but also due to their chain effect on the quality of situation awareness, safety of decision-making, and effectiveness of actuators control [17], as presented in Figure 1. Among the various global to local collision avoidance approaches [18], reactive collision avoidance is the most local and perception sensor-dependent one, as it requires real-time decisions made at the point of sensing and at the lowest level of actuators control to achieve the highest possible degree of autonomy for real-world applications, where information is often limited, dynamic, and uncertain [19].

However, despite the current advancements of sensor technologies and unique features of each perception sensor [20],degradation of measurements due to sensor noise is inevitable [21]. Sensor noise refers to the random variation inherent in the measurements that can be attributed to a wide range of factors from intrinsic limitations, such as a low resolution or design imperfections, to extrinsic influences, such as atmospheric conditions or electromagnetic interference [22]. The adverse marine environment adds another layer of complexity considering the extreme temperatures and humidity levels, corrosive nature of sea water, and visibility impairing rain, fog, or glare [23]. This further exacerbates the uncertainty of the measurements, which can manifest as false detection, such as false positive or false negative [24], or even as complete detection failure of a collision-inducing obstacle [25]. Addressing sensor noise during reactive collision avoidance, where the temporal and spatial margins for error are minimal, is a pressing challenge that needs to be investigated prior to the full-scale implementation of MASSs.

1.2. Literature Review

Previous studies employed several algorithms to address the reactive collision avoidance of MASSs in different scenarios. Xu et al. [26] employed modified artificial potential field and velocity obstacle in scenarios with a single static obstacle to multiple static obstacles. Blindheim et al. [27] used model predictive control in scenarios with multiple static obstacles, considering dynamic risk pertaining to emergency situations, such as impaired thrusters, total blackout, or strong winds. Gao et al. [28] employed particle swarm optimisation and dynamic window (DW) in scenarios with a single dynamic obstacle, while complying to the International Regulations for Preventing Collisions at Sea (COLREGs) under head-on, crossing, and overtaking situations. Serigstad et al. [29] employed hybrid DW and a rapidly exploring random tree in scenarios with static and dynamic obstacles, while tracking a global path. Wang et al. [30] used a deep Q network (DQN) in scenarios with a single static obstacle or single to multiple dynamic obstacles with COLREGs compliance and good seamanship. Cheng et al. [31] used DQN in scenarios with multiple static obstacles and unknown environmental disturbances. However, a common limitation across these studies was the assumption of complete information of the navigating area and the obstacles without accounting for perception sensors.

Some studies addressed the above limitation by incorporating various perception sensors for obstacles detection. Zhou et al. [32], Meyer et al. [33], and Heiberg et al. [34] used general range finder sensors with multiple beams to measure the distances and relative angles to obstacles. Kim et al. [35], Gonzalez et al. [36], and Villa et al. [37] used light detection and rangings (LIDARs) to generate point clouds or clusters of the surrounding environment and measure the positions, distances, and relative angles to obstacles. Wang et al. [38] used radio detection and ranging (RADAR) to calculate the distance to the closest point of approach and the time to the closest point of approach. Song et al. [19] used automatic identification system for dynamic obstacles data. Peng et al. [39] used LIDAR and sound navigation and ranging to detect surface and underwater obstacles, respectively. However, these studies assumed ideal sensors without accounting for more realistic noisy measurements.

Some studies addressed the influence of noise on perception sensors using various noise filtering techniques. Han et al. [40] used an extended Kalman filter (EKF) for electro-optical and infrared camera, RADAR, and LIDAR measurements to predict the motion of obstacles. Han et al. [41] used EKF on RADAR measurements to predict the motion of obstacles. Stanislas et al. and Kim et al. [42] filtered RADAR and LIDAR measurements by experimentally tuning the intensity of basic RADAR settings and filtering out LIDAR measurements that exceeded a predefined threshold to estimate the state of obstacles. However, the primary focus of these studies was limited to filtering out sensor noise to enhance the situation awareness for the state estimation of the obstacles.

1.3. Aim & Contributions

The review of pertinent literature highlights a gap of investigating the direct impact of unfiltered noisy measurements of a perception sensor on the decision-making during safety-critical operations. Hence, this study aims to bridge this gap by proposing a methodology that enhances the robustness of decision-making for the reactive collision avoidance of a MASS against various perception sensor noise levels. The key contributions of this study are as follows.

• A novel approach that incorporates sensor noise for end-to-end decision-making pertaining to a deep reinforcement learning (DRL) agent is proposed, providing a way forwards towards a more effective integration of signal processing and decision-making techniques.
• An effective training framework for a DRL agent that enhances the robustness and sophistication of the agent’s decision-making against various sensor noise levels is proposed.
• A systematic way to analyse and interpret the decision-making of a trained DRL agent is presented.
• A systematic way to analyse the sensitivity of a trained DRL agent to inputs pertaining to sensor noise is presented.

1.4. Outline

The remainder of this study is structured as follows. Section 2 presents the proposed methodology and the subsequent phases involved. Section 3 presents the rationale and characteristics of the investigated case study. Section 4 presents the results and discussion. Finally, Section 5 presents the main findings, limitations, and outlook for future studies.

2. Methodology

The proposed methodology is developed into five subsequent phases, as presented in Figure 2, with each phase serving a specific objective as outlined below.

• Phase 1. Scenarios simulation: The objective is to simulate scenarios pertaining to the reactive collision avoidance of a MASS using a noisy perception sensor, by employing the main components as digital twins. The employed digital twins are the ship manoeuvrability, perception sensor and its noise model, and map.
• Phase 2. Decision-making problem formulation: The objective is to formulate the decision-making problem pertaining to the investigated scenarios as a Markov decision process, by identifying the rewards, states, and actions. The identified rewards are associated with path following, nominal navigation, actuator control, and collision avoidance objectives. The identified states are associated with variables related to the rewards and noise level of the perception sensor. The identified actions are associated with the actuator control.
• Phase 3. Agent training: The objective is to train a DRL agent that can make decisions over the formulated problem, by developing a training framework. The developed training framework considers the noise levels envelope of the perception sensor investigated during the agent’s training.
• Phase 4. Robustness quantification: The objective is to quantify the robustness of the trained agent’s decision-making against various perception sensor noise levels, by defining robustness metrics. The defined robustness metrics are associated with path following and collision avoidance.
• Phase 5. Robustness verification: The objective is to verify the robustness of the trained agent’s decision-making against various perception sensor noise levels, by investigating various scenarios. The investigated scenarios are within and outside the trained noise levels envelope of the perception sensor.

2.1. Scenarios Simulation

The objective of this phase is to simulate scenarios pertaining to the reactive collision avoidance of a MASS using a noisy perception sensor, by employing the main components as digital twins. Digital twins stand for high-fidelity models that enable real-time virtual simulations, while accurately mirroring the full-scale reality of their physical twins [43]. Digital twins are particularly important for the investigation of safety-critical operations in extreme scenarios [44], where risk on the property’s integrity, environmental protection, and human life is involved [45]. The digital twins of the main components, as delineated in this Section, are employed in MATLAB/Simulink 2023b, due to the customisation capabilities of a wide range of digital twin libraries, including the Navigation Toolbox [46], LIDAR Toolbox [47], and Robotics System Toolbox [48].

2.1.1. Ship Manoeuvrability

The digital twin of the own ship (OS) manoeuvrability is employed, which refers to the ability of a ship to maintain or change its motion states under the operation of actuators, such as propellers, rudders, and thrusters [49]. For digital twin applications, numerical simulations of ship manoeuvring can be employed using either system-based methods that use mathematical models to express the manoeuvring motion as an ordinary differential equation according the Newton’s second law [49] or computational fluid dynamics (CFD) methods that resolve the complex fluid-structure interactions by incorporating both viscous and rotational effects in the flow problem [50]. However, despite CFD simulations being considered complementary to the experimental methods due to their high-fidelity and growing computational power [51], their time-consuming aspect renders them impractical [52], especially when real-time or a significant number of simulations need to be conducted [53].

On the other hand, system-based methods are typically limited to three or four degrees of freedom (3DOF, 4DOF) [54] with their main advantage being the rapid computational time, despite their compromise on the accuracy for long term manoeuvring simulations [53]. For instance, a system-based ship manoeuvring simulation of 150 s can be executed in 50 s on a typical Intel Core i7 @3.2 GHz processor [53], whereas a CFD-based simulation of 19 s can take 225 h on an Intel Xeon E5-2680 v2 @2.8 GHz processor [55]. The two most widely used system-based models are the Abkowitz model [56], also known as the whole ship model, which expresses the hydrodynamic forces and moments acting on a ship as Taylor series, and the manoeuvring modelling group (MMG) model [57], also known as the modular model, which decomposes the forces and moments acting on different components, typically the ship hull, propeller, and rudder [58]. Compared to the Abkowitz model that considers the ship as a whole, the MMG model is based on first principles that carry explicit and clear physical meaning of each component including their interactions [59]. In addition, it has the capability to accommodate different actuators configurations, such as single-screw [60], twin-screw [61], and twin azimuth thrusters [62], integrate external disturbances, such as currents, waves, and wind, and be combined with other numerical, experimental, or empirical methods [59].

To describe the manoeuvring motions, two coordinate systems are typically used, which are the space-fixed coordinate system $o_0$–$x_0{y}_0{z}_0$ and the body-fixed coordinate system $o_{\mathrm{B}}$–$x_{\mathrm{B}}{y}_{\mathrm{B}}{z}_{\mathrm{B}}$ [63], as presented in Figure 3. The origin $o_{\mathrm{B}}$ is assigned at the midship instead at the centre of gravity G, when a fixed position is needed regardless of the load condition. The $x_0{y}_0$-plane and $x_{\mathrm{B}}{y}_{\mathrm{B}}$-plane coincide with the still-water surface plane, whereas the $x_{\mathrm{B}}$-axis is directed towards the bow and the $z_0$-axis and $z_{\mathrm{B}}$-axis are directed downwards. u and v denote the surge and sway velocities in the $x_{\mathrm{B}}$-direction and $y_{\mathrm{B}}$-direction, respectively. $\dot{\psi }$ denotes the yaw rate around $z_{\mathrm{B}}$-axis, where $\psi$ denotes the heading angle from $x_0$-axis to $x_{\mathrm{B}}$-axis. $\dot{\varphi }$ denotes the roll rate, where $\varphi$ denotes the roll angle. U denotes the resultant velocity as $U=\sqrt{u^2+v^2}$, and $\beta$ denotes the drift angle as $\beta ={tan}^{-1}(-v/u)$.

The motion equations of a 4DOF MMG that couples the surge, sway, yaw, and roll motions, assuming that the heave and pitch motions are negligible, is expressed as [64]:

$$\left\{\begin{array}{cc}\hfill (m+m_{x_{\mathrm{B}}})\dot{u}-(m+m_{y_{\mathrm{B}}})v\dot{\psi }& =X\hfill \\ \hfill (m+m_{y_{\mathrm{B}}})\dot{v}+(m+m_{x_{\mathrm{B}}})u\dot{\psi }+m_{y_{\mathrm{B}}}{\alpha }_{y_{\mathrm{B}}}\ddot{\psi }-m_{y_{\mathrm{B}}}{l}_{y_{\mathrm{B}}}\ddot{\varphi }& =Y\hfill \\ \hfill (I_{z_{\mathrm{B}}}+J_{z_{\mathrm{B}}})\ddot{\psi }+m_{y_{\mathrm{B}}}{\alpha }_{y_{\mathrm{B}}}\dot{v}& =N-x_{\mathrm{B},G}Y\hfill \\ \hfill (I_{x_{\mathrm{B}}}+J_{x_{\mathrm{B}}})\ddot{\varphi }-m_{y_{\mathrm{B}}}{l}_{y_{\mathrm{B}}}\dot{v}-m_{x_{\mathrm{B}}}{l}_{x_{\mathrm{B}}}u\dot{\psi }& =K-W\overline{GM}\varphi \hfill \end{array}\right.$$

(1)

where m, $m_{x_{\mathrm{B}}}$, and $m_{y_{\mathrm{B}}}$ denote the ship’s mass and the added mass in the $x_{\mathrm{B}}$-direction and $y_{\mathrm{B}}$-direction, respectively. $I_{x_{\mathrm{B}}}$, $I_{z_{\mathrm{B}}}$, $J_{x_{\mathrm{B}}}$, and $J_{z_{\mathrm{B}}}$ denote the moment of inertia about the $x_{\mathrm{B}}$-axis and $z_{\mathrm{B}}$-axis, and the added moment of inertia in their corresponding axes, respectively. $a_{y_{\mathrm{B}}}$ denotes the $x_{\mathrm{B}}$-coordinate of centre of $m_{y_{\mathrm{B}}}$, while $l_{x_{\mathrm{B}}}$ and $l_{y_{\mathrm{B}}}$ denote the $z_{\mathrm{B}}$-coordinates of the centres of $m_{x_{\mathrm{B}}}$ and $m_{y_{\mathrm{B}}}$, respectively. $x_{\mathrm{B},G}$ denotes the $x_{\mathrm{B}}$-coordinate of centre of gravity, W denotes the weight of water displaced by the ship hull, and $\overline{GM}$ denotes the metacentric height. X, Y, N, and K denote the forces in the $x_{\mathrm{B}}$-axis and $y_{\mathrm{B}}$-axis, yaw moment about the midship, and roll moment about the centre of gravity, respectively.

2.1.2. Perception Sensor

The digital twin of the perception sensor is employed considering its fundamental measurement principle and its noise model. During reactive collision avoidance, where the temporal and spatial margins for error are limited, perception sensors that provide distance measurements from collision-inducing obstacles become essential [65], such as stereo-cameras, RADARs, and LIDARs. Stereo-cameras passively capture electromagnetic waves using two pairs of monocular-cameras and provide the cheapest option when it comes to extracting distance measurements [14]. However, they require complex and heavy image processing techniques to extract such information [66] and exhibit the lowest accuracy depending on the illumination conditions [67].

RADARs and LIDARs actively emit electromagnetic waves in the microwave and infrared spectrum, respectively, and use the time-of-flight principle of the pulses reflected from the obstacles to directly measure the distances [68]. RADARs are suitable for long distance measurements of up to 10 NM and operation in all maritime environments [69]. However, the scanning frequency is low, approximately every 2–3 s, the measurements are limited only to 2 dimensions (2D), and the resolution and accuracy are low even at close distances, ranging from several to hundreds of m, due to the relatively wide beams [70,71].

In contrast, LIDARs exhibit the highest accuracy of distance and angular measurements due to their very narrow laser beams [15]. Scanning is conducted multiple times per s with each spin consisted of a few million measurements, allowing accurate distance, speed, position, and size estimation of the obstacles, as well as generation of a 3D map of the surrounding environment [70]. Typical LIDARs measure distances of up to a few hundreds m with some reaching up to a few km [69,72].

However, LIDARs are particularly susceptible to noise coming from the maritime environment, including sea water, rain, fog, and snow [70]. Water droplets in the atmosphere cause severe attenuation of the laser beams, absorb their energy, reduce the surface reflectivity of the obstacles, or cause pseudo-points from beams backscattered from the droplets that can be falsely detected as obstacles [73]. In addition, random noise can be produced by the sensor itself due to its intrinsic design limitations, discharges in the circuits, or during the conversion from analogue to digital signal [74]. Due to the many contributing factors that are hard to be known apriori [74], a common approach is to assume that the noise follows a Gaussian distribution [75], as presented in Figure 4. Hence, the time-of-flight equation of a LIDAR sensor with an added Gaussian noise model is expressed as [76]:

$$d_i=\frac{c{t}_i}{2}+ϵ$$

(2)

where $d_i$ denotes the distance of the ith LIDAR measurement, c denotes the speed of light, $t_i$ denotes the time of flight, and $ϵ$ denotes the added noise that follows a Gaussian distribution with 0 mean and variance ${\sigma }^2$. The total LIDAR measurements at each timestep t constitutes to the array $d_t=[d_{1,t},d_{2,t},\dots ]$.

2.1.3. Map

The digital twin of the map is employed, which stands for the spatial representation of the environment [77], such as the navigational area of the OS. Typical map representation methods are divided into topological maps, feature maps, and occupancy grid maps [78]. Topological maps use graphs, such as nodes and arcs, to represent the environment in the most compact way, but exhibit challenges in storing proximity information or modelling complex environments due to their simplicity [79]. Feature maps use parametric features, such as points and lines, to represent distinctive parts of the environment that are identifiable by the perception sensor, but they are not suitable for highly unstructured environments with no distinct geometries and when detailed navigation is needed due to its limited resolution [78].

Occupancy grid maps use grid of fixed resolution to represent the environment, where each cell stores a probability of being free or obstacle-occupied [80]. Their main advantages are their rich representation of the environment in both 2D and 3D, flexibility in balancing resolution with accuracy depending on the application requirements, and compatibility with noisy perception sensors, such as a LIDAR [81]. Thus, occupancy grid maps are commonly used to represent the navigational areas for path planning and collision avoidance applications [82], as presented in Figure 5.

2.2. Decision-Making Problem Formulation

The objective of this phase is to formulate the decision-making problem pertaining to the investigated scenarios. Specifically, this falls into the category of complex sequential decision-making problems under uncertainty, which are typically formulated using a mathematical framework known as the Markov decision process (MDP) [83]. In fact, many real-world decision-making problems can be modelled as an MDP due to its abstract and flexible framework [84]. The fundamental idea of this framework is to achieve a goal by learning from interactions, in which the learning decision-maker is known as the agent and everything else that interacts with it as the environment $\mathcal{E}$.

The interaction at each timestep t is boiled down into the agent receiving a state of the environment $S_t$ and selecting an action $A_t$ based on the received state that leads to the reception of a reward $R_{t+1}$ and new state $S_{t+1}$ [85], as presented in Figure 6. The sum of all rewards that the agent accumulates from timestep t and onwards through this interaction is known as the return $G_t$, which is expressed as:

$$G_t=R_t+R_{t+1}+R_{t+2}+\dots$$

(3)

However, considering the uncertainties of the future, the return is formulated as the expected discounted return, which is expressed as:

$$G_t=R_t+\gamma R_{t+1}+{\gamma }^2{R}_{t+2}+\dots =\sum _{i=t}^{\mathcal{T}}{\gamma }^{i-t}{R}_i$$

(4)

where $\gamma =[0,1]$ denotes the discount rate, i denotes the timestep $i=[0,\mathcal{T}]$, and $\mathcal{T}$ denotes the terminal timestep.

The overall goal of the agent is to learn a policy $\pi$ that maximises the expected return from its initial timestep $G_0$ by mapping from states to actions. However, considering that the policy can be stochastic, the return $G_0$ is formulated as an expected return J, which is expressed as:

$$J={\mathbb{E}}_{r_i,s_i\sim \mathcal{E},a_i\sim \pi }\left[G_0\right]$$

(5)

where, $s_i$, $a_i$, and $r_i$ denote the values of the state, action, and reward, respectively.

To learn such a policy, the expected return at each timestep t and onwards after taking the action $a_t$ in state $s_t$ following the policy $\pi$ needs to be learned, known as the action-value function $Q^{\pi }(s_t,a_t)$, which is expressed as:

$$Q^{\pi }(s_t,a_t)={\mathbb{E}}_{r_{i\ge t},s_{i>t}\sim \mathcal{E},a_{i>t}\sim \pi }[G_t\mid s_t,a_t]$$

(6)

The action-value function can be formulated in its recursive form using the Bellman equation, which is expressed as:

$$Q^{\pi }(s_t,a_t)={\mathbb{E}}_{r_t,s_{t+1}\sim \mathcal{E}}\left[r(s_t,a_t)+\gamma {\mathbb{E}}_{a_{t+1}\sim \pi }\left[Q^{\pi }(s_{t+1},a_{t+1})\right]\right]$$

(7)

Finally, the action-value function can be further simplified when a deterministic policy $\mu$ is used, which is expressed as:

$$Q^{\mu }(s_t,a_t)={\mathbb{E}}_{r_t,s_{t+1}\sim \mathcal{E}}[r(s_t,a_t)+\gamma Q^{\mu }(s_{t+1},\mu \left(s_{t+1}\right))]$$

(8)

Thus, the goal of the agent is to learn an optimal policy by accurately estimating the optimal action-value function at each timestep, thereby maximising the expected return.

2.2.1. Rewards

The rewards that the agent learns to maximise are defined as reward functions, which are identified according to path following, nominal navigation, actuator control, and collision avoidance objectives. The path following reward is expressed as:

$$R_{1,t}=\left\{\begin{array}{c}\hfill k_1+\frac{k_2}{e_{\mathrm{XT},t}^2+k_3}+\frac{k_4}{e_{\mathrm{H},t}^2+k_5+k_6\left|e_{\mathrm{XT},t}\right|},\mathrm{if}\left|e_{\mathrm{XT},t}\right|\le \mathrm{XTL}\\ \hfill \frac{k_2}{e_{\mathrm{XT},t}^2+k_3}+\frac{k_4}{e_{\mathrm{H},t}^2+k_5+k_6\left|e_{\mathrm{XT},t}\right|},\mathrm{if}\left|e_{\mathrm{XT},t}\right|>\mathrm{XTL}\end{array}\right.$$

(9)

where $k_1,k_2,k_3,k_4,k_5,k_6>0$ denote the coefficients, $e_{\mathrm{H}}$ denotes the heading error of the OS from the path as $e_{\mathrm{H}}={\psi }_{\mathrm{WP}}-\psi$ given the heading of the path ${\psi }_{\mathrm{WP}}=arctan(y_2-y_1,x_2-x_1)$ from two waypoints $(x_1,y_1)$ and $(x_2,y_2)$, $e_{\mathrm{XT}}$ denotes the cross-track error of the OS from the path as $e_{\mathrm{XT}}=-(x-x_1)sin\left({\psi }_{\mathrm{WP}}\right)+(y-y_1)cos\left({\psi }_{\mathrm{WP}}\right)$ [86], and XTL denotes the threshold for the path following criterion often refer to as the cross-track limit [87]. It can be inferred that the reward is maximised when the cross-track and heading errors of the OS from the path are minimised concurrently. Also, an additional constant reward $k_1$ is given when the OS follows the path within the given threshold.

The nominal navigation reward is expressed as:

$$R_{2,t}=k_7{v}_t^2+k_8{\dot{\psi }}_t^2+k_9{\dot{v}}_t^2+k_{10}{\ddot{\psi }}_t^2$$

(10)

where $k_7,k_8,k_9,k_{10}<0$ denote the coefficients. It can be inferred that the reward is maximised when the sway and yaw velocities and accelerations of the OS are minimised.

The actuator control reward is expressed as:

$$R_{3,t}=k_{11}{\delta }_{\mathrm{C},t}^2+k_{12}{\dot{\delta }}_{\mathrm{C},t}^2$$

(11)

where $k_{11},k_{12}<0$ denote the coefficients, ${\delta }_{\mathrm{C}}$ and ${\dot{\delta }}_{\mathrm{C}}$ denote the magnitude and rate of the action command executed by the agent. It can be inferred that the reward is maximised when the magnitude and the rate of the actuator are minimised.

The collision avoidance reward is expressed as:

$$R_{4,t}=k_{13}{d}_t+k_{14},\mathrm{if}{d}_t>d_{\min }$$

(12)

where $k_{13}>0$ and $k_{14}<0$ denote the coefficients, and $d_{\min }$ denotes the threshold for the collision criterion. It can be inferred that the reward is maximised when the distances of the OS from the obstacles are maximised.

The final reward function is expressed as:

$$R_t=R_{1,t}+R_{2,t}+R_{3,t}+R_{4,t}$$

(13)

2.2.2. States & Actions

The states that the agent receives from the environment are identified as the set of variables related to the rewards, known as observations, which constitutes the state space [85]. A low-dimensional state space that consists of a limited number of observations can lead to an incomplete understanding of the environment and sub-optimal decision-making, whereas a high-dimensional state space can exponentially increase the computational complexity and training time of the agent [88]. Thus, the state space is expressed as:

$$S_t=[{\psi }_t,u_t,{\dot{u}}_t,v_t,{\dot{v}}_t,r_t,{\dot{r}}_t,U_t,{\dot{U}}_t,e_{\mathrm{XT},t},{\dot{e}}_{\mathrm{XT},t},{\ddot{e}}_{\mathrm{XT},t},e_{\mathrm{H},t},{\dot{e}}_{\mathrm{H},t},{\ddot{e}}_{\mathrm{H},t},d_t,\mathcal{N}]$$

(14)

It is worth noting that despite noise being added on the perception sensor measurements observation d, an additional noise variance observation $\mathcal{N}$ is added to represent the value of the noise variance ${\sigma }^2$ of the perception sensor, as presented prior in Equation (2).

The actions that the agent executes to affect the state of the environment and maximise the rewards are identified as the set of all possible actions that the actuator can take, which constitutes the action space [85]. Thus, the action space is expressed as:

$$A_t={\delta }_{\mathrm{C},t}$$

(15)

2.3. Agent Training

The objective of this phase is to train an agent that can make decisions over the formulated problem, by developing a training framework. Recently, DRL methods have gained significant attention due to the major breakthroughs achieved in decision-making, by integrating the trial-and-error techniques of reinforcement learning and feature extraction capabilities of deep learning using artificial neural networks [89]. Specifically, DRL methods have surpassed previous limitations of addressing complex decision-making problems with high-dimensional state and action spaces [90] with significant end-to-end learning [91] at a human level [92,93]. The DRL training framework, as delineated in this Section, is developed in MATLAB/Simulink 2023b, due to the customisation capabilities of a wide range of machine learning libraries, including the Deep Learning Toolbox [94] and Reinforcement Learning Toolbox [95].

2.3.1. Deep Reinforcement Learning Agent

A DRL agent is employed, where some of the most commonly used algorithms are the trust region policy optimisation [96], deep deterministic policy gradient (DDPG) [97], proximal policy optimisation [98], soft actor-critic [99], and twin delayed DDPG (TD3) [100]. While each algorithm has its own advantages and disadvantages, choosing the most suitable one for each application is not a trivial effort. However, a comparative study evaluating the aforementioned algorithms for process control applications provides a good indication of the superior control performances of DDPG and TD3 algorithms [101]. In addition, DDPG algorithm is suitable for continuous and high-dimensional state and action spaces and challenging physical control problems with inertia and fine control of actions [102].

The DDPG algorithm is based on an actor–critic architecture, where the actor and the critic represent the policy $\mu (s\mid {\theta }^{\mu })$ and action-value function $Q(s,a\mid {\theta }^Q)$, respectively, using deep neural networks (DNNs) as function approximators parameterised by their respective weights ${\theta }^{\mu }$ and ${\theta }^Q$. The actor and critic are referred to as the main actor network and main critic network, respectively, that constitute the main network. The actor and critic learn off-policy by utilising a replay buffer $\mathcal{D}$ of finite size that stores the tuples $(s_t,a_t,r_t,s_{t+1})$ generated from the environment at each timestep t. Particularly, the actor and critic are updated by uniformly sampling a minibatch $\mathcal{B}$ of tuples $(s_i,a_i,r_i,s_{i+1})$ from the replay buffer to minimise the correlations between the samples, where $i=[1,b]$ and b denotes the size of the minibatch.

Based on the prior Equation (7), the targets for the learning of the critic is expressed as:

$$y_i=r_i+\gamma Q(s_{i+1},\mu (s_{i+1}\mid {\theta }^{\mu })\mid {\theta }^Q)$$

(16)

However, since the action-value function is updated while in the target, it leads to divergence in learning. To tackle this, DDPG algorithm employs a target network, which is a copy of the main network. Specifically, the target actor network ${\mu }^{\prime }(s\mid {\theta }^{{\mu }^{\prime }})$ and target critic network $Q^{\prime }(s,a\mid {\theta }^{Q^{\prime }})$ with their respective weights ${\theta }^{Q^{\prime }}$ and ${\theta }^{{\mu }^{\prime }}$ are used to give stable targets, which is expressed as:

$$y_i=r_i+\gamma Q^{\prime }(s_{i+1},{\mu }^{\prime }(s_{i+1}\mid {\theta }^{{\mu }^{\prime }})\mid {\theta }^{Q^{\prime }})$$

(17)

Thus, the main critic network learns by minimising the critic loss, which is expressed as:

$$L\left({\theta }^Q\right)=\frac{1}{N}\sum _{i=1}^N{(y_i-Q(s_i,a_i\mid {\theta }^Q))}^2$$

(18)

and by updating its weights in the direction of gradient descent that minimises the loss, which is expressed as:

$${\theta }^Q\leftarrow {\theta }^Q-a_Q{\nabla }_{{\theta }^Q}L$$

(19)

where $a_Q$ denotes the learning rate of the main critic network.

The main actor network learns by minimising the actor loss, which is expressed as:

$$J\left({\theta }^{\mu }\right)=-\frac{1}{N}\sum _{i=1}^{N}Q(s_i,\mu (s_i\mid {\theta }^{\mu })\mid {\theta }^Q)$$

(20)

and by updating its weights in the direction of gradient ascent that maximises the expected return, which is expressed as:

$${\theta }^{\mu }\leftarrow {\theta }^{\mu }+a_{\mu }{\nabla }_{{\theta }^{\mu }}J$$

(21)

where $a_{\mu }$ denotes the learning rate of the main actor network.

The target actor network and target critic network learn by updating their weights using soft target updates, which is expressed as:

$${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}$$

(22)

$${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$$

(23)

where $\tau \ll 1$ denotes the target smooth factor.

It is worth noting that every tuple of the replay buffer are normalised for effective learning, known as batch normalisation. Also, noise sampled from a noise process $\mathcal{W}$, such as an Ornstein-Uhlenbeck process, is added on the main actor network’s output to encourage exploration. A pseudo-code and schematic diagram of the algorithm are presented in Algorithm 1 and Figure 7.

Algorithm 1: Deep deterministic policy gradient

2.3.2. Training Framework

A training framework is developed considering the noise levels envelope of the perception sensor investigated during the agent’s training. Specifically, the noise variance ${\sigma }^2$ of the perception sensor are generated from uniformly distributed random numbers between a predefined minimum and maximum value, the range of which constitutes the noise levels envelope. A single noise variance is used during each episode until the termination of that episode, where an episodes stands for the sequence of timesteps of an episodic task from the beginning to the end [85]. It is worth noting that the reward functions $R_t$ and thresholds for the path following and collision criteria XTL and $d_{\min }$, respectively, remain the same, regardless of whether the noise variance is low or high.

Each episode is automatically terminated when the OS surpasses the boundaries of the map or when the timestep reaches the predefined terminal timestep $\mathcal{T}$. In addition, the episode is terminated when the collision criterion is triggered. The training performance of the agent is evaluated based on different training performance metrics, including the return per episode and the loss per step, where step refers to the update iteration. Particularly, the overall trends of these metrics towards a local maximum or local minimum and reduction of their variances are considered as good indicators of convergence and stability of training. Finally, the training is conducted in parallel using 30 cores of an Intel Xeon Platinum 8260 CPU processor for the episodes simulation and a NVIDIA Quadro RTX 8000 GPU processor for the update of the agent’s learnable parameters.

2.4. Robustness Quantification & Verification

The objective of the robustness quantification phase is to quantify the robustness of the trained agent’s decision-making against various perception sensor noise levels, by defining robustness metrics. Robustness stands for the magnitude of tolerability of a system against disturbances until a set requirement or threshold is violated [103,104]. A robustness metric provides a binary assessment of system’s robustness state based on the compliance or violation of the threshold. In addition, it quantifies the degree of robustness by measuring the available margin towards violating the threshold.

Two robustness metrics are defined pertaining to path following and collision avoidance $R{M}_{\times }$ and $R{M}_{\ast }$, respectively, which are expressed as:

$$R{M}_{\times }=\left|e_{\mathrm{XT},\times }\right|<{\mathrm{Threshold}}_{\times }$$

(24)

$$R{M}_{\ast }=d_{\ast }>{\mathrm{Threshold}}_{\ast }$$

(25)

where × and ∗ denote the timesteps of the first and minimum obstacle detection, respectively, ${\mathrm{Threshold}}_{\times }$ denotes the robustness threshold for path following, and ${\mathrm{Threshold}}_{\ast }$ denotes the robustness threshold for collision avoidance. It can be inferred that the robustness for path following requires cross-track error within a margin prior to the first obstacle detection, whereas the robustness for collision avoidance requires sufficient distance from the obstacles at the minimum obstacle detection. It is worth noting that the robustness thresholds are not required to be equal to the thresholds for path following and collision criteria XTL and $d_{\min }$, respectively, depending on the robustness requirements.

Finally, the objective of the robustness verification phase is to verify the robustness of trained agent’s decision-making, by investigating various scenarios. Specifically, scenarios within and outside the trained noise levels envelope are investigated, until either of the robustness thresholds for path following or collision avoidance are violated.

3. Case Study

The case study considers a single-screw high-speed container ship as the OS, known as the S-175 [64]. The S-175 is selected, as it is one of the benchmark hull forms used in manoeuvring studies, due to its well-defined hydrodynamic coefficients [105]. A 4DOF MMG model is used, as released in the Marine Systems Simulator Toolbox [106], that considers the none-negligible roll-coupling effect in high-speed container ships with low metacentric height [107], such as the S-175. In addition, the model considers no environmental disturbances and it is assumed to provide sufficient fidelity, as 3DOF is often sufficient in most manoeuvring studies [108].

A first-order linear differential equation model is used to simulate the rudder dynamics, where the deflection rate of the rudder $\dot{\delta }$ is defined as $\dot{\delta }=({\delta }_{\mathrm{C}}-\delta )/T_{\delta }$ given the commanded rudder angle ${\delta }_{\mathrm{C}}$, rudder angle $\delta$, and time constant $T_{\delta }$ that expresses the time delay due to the main servo in linear manners [109,110]. Two slew rates are used to saturate the maximum commanded rudder angle as $|{\delta }_{\mathrm{C}}|\le {\delta }_{\mathrm{C},\max }$ and the maximum deflection rate of the rudder as $|\dot{\delta }|\le {\dot{\delta }}_{\max }$ [111]. In addition, a constant nominal propeller revolution $n_{\mathrm{nom}}$ is considered to operate at 85% load of the candidate engine’s specified maximum continuous rating (SMCR). The main particulars of the OS are presented in

Table 1. Main particulars of the OS.

Particulars	Symbol	Value
Length	L	175 m
Beam	B	25.4 m
Draft	T	8.5 m
Depth	D	11.0 m
Displaced volume	∇	21,222 ${\mathrm{m}}^3$
Block coefficient	$c_B$	0.559
Maximum commanded rudder angle 1	${\delta }_{\mathrm{C},\max }$	10 deg
Maximum rudder deflection rate 2	${\dot{\delta }}_{\max }$	5 deg/s
Time constant 3	$T_{\delta }$	1 s
Nominal propeller revolution at 85% SMCR 4	$n_{\mathrm{nom}}$	99.5 rpm
Nominal speed	$U_{\mathrm{nom}}$	10.41 m/s

¹ It should not exceed 35 deg according to the International Convention for the Safety of Life at Sea (SOLAS), Regulation 29 [112]. A relatively small maximum commanded rudder angle is selected to reduce the roll effect [64]. ² It should exceed 2.32 deg/s according to the SOLAS, Regulation 29 [112]. ³ Typical values range between 1–3 s [113]. ⁴ For a candidate marine two-stroke engine, such as the 7S60ME-C10, that satisfies the power requirements of SMCR between 11,200–15,000 kW for a container ships with deadweight tonnage between 26,500–33,000 mt [114].

.

A 2D binary occupancy map is considered, assuming that in most surface-based applications the most critical information to be perceived is on the horizontal plane [115]. The size of the map is 5 × 5 km with resolution of 1 × 1 m, where each cell stores a binary value, 0 or 1, to represent a free or obstacle-occupied area, respectively. It is worth noting that collision with static obstacles remains a challenging issue as global databases of maritime accidents suggest that 21% of 513 investigated collision accidents originated from strikes with other ships that were not underway using their engines, such as being engaged in fishing, moored, or at anchor [116]. For this reason, the case study considers a single static obstacle on the map that the OS encounters during the following of a straight global path generated by two waypoints. The static obstacle is modelled as a circle of 100 m radius to arbitrarily approximate the shape and size of any mid-sized ship, similar to the OS size.

A 2D LIDAR is considered, which is set to scan at the midship with a maximum detecting range of 1341 m. The area that exceeds this range constitutes the sensor’s blind-spot zone, where no information can be derived from [117]. It is worth noting that the selected maximum detecting range is not arbitrary but it is comparative to the manoeuvrability of the OS. Specifically, assuming that the turning circle at the maximum rudder angle is a good indicator of the OS manoeuvrability [118], the ratio of the maximum detecting range to its maximum advance is 1.4. In addition, a forward scanning with 225 deg field of view is assumed to be sufficient among static obstacles, where an angular resolution of 5.63 deg constitutes to a total of 41 LIDAR measurements that guarantees the detection of any circular obstacle with radius greater than 65.8 m at its maximum detecting range.

A representation of investigated case study is presented in Figure 8. In each episode, the OS is always initiated at the first waypoint and at nominal navigation conditions heading towards the next waypoint. The obstacle is allocated on the path with its vertical position randomly generated between $y_{\mathrm{obs}}=[2000,3500]$ m. In addition, the noise variance of the LIDAR is randomly generated between ${\sigma }^2=[0,25]$ ${\mathrm{m}}^2$. The thresholds for the path following and collision criteria are set as $\mathrm{XTL}=B/2$ and $d_{\min }=B/2$, respectively. The robustness thresholds for path following and collision avoidance as set as ${\mathrm{Threshold}}_{\times }=\mathrm{XTL}$ and ${\mathrm{Threshold}}_{\ast }=d_{\min }$, respectively. A DDPG agent is employed, whose main hyperparameters pertaining to the network’s architecture are two hidden layers with 600 neurons each for the actor network and two hidden layers with 500 neurons each for the critic network constituting to a total of 396,001 and 281,001 learnable parameters including their weights, respectively. Finally, the simulation timestep and update iteration step of the agent are set to 1 s to be compatible with real-time applications.

4. Results & Discussion

4.1. Agent Training

The training performance of the ${\mathrm{agent}}_{\mathrm{A}}$ in terms of the return per episode, actor loss per step, and critic loss per step is presented in Figure 9. It is noted that the return convergences to a local maximum approximately from episode 4800, as indicated by its moving median, with further reduction of its variance from episode 5500, which denotes that the agent consistently maximises the return in each episode. The actor loss converges to a local minimum from step $2.25\times {10}^6$, as indicated by its moving median, which denotes that the actor converges to a stable policy. In contrast, the critic loss diverges from a local minimum from step $1.75\times {10}^6$, as indicated by its moving median, with a general increase of its variance due to exploding values, which suggests that the critic does not converge to a stable estimation of the expected return. However, despite this instability, the critic loss exhibits a downwards trend potentially towards a local minimum from step $2.3\times {10}^6$, as indicated by its moving median, which is in alignment to the overall convergence trends of the other training performance metrics.

The training is terminated at episode 5820, which constitutes to a total of 2,529,998 steps and approximately 44 h of training. The minimum and maximum values of the training performance metrics derived during the training and their respective moving medians at the end of the training are presented in

Table 2. Training performance metrics during the training of ${\mathrm{agent}}_{\mathrm{A}}$.

${\mathbf{Return}}_{\mathbf{A},\mathbf{min}}$	${\mathbf{Return}}_{\mathbf{A},\mathbf{max}}$	$\mathbf{Moving}{\mathbf{Median}}_{\mathbf{A}}$ 1
−67.42	749.72	633.84
$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{min}}$	$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{Median}}_{\mathbf{A}}$ 2
−344.62	−0.45	−288.36
$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{min}}$	$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{median}}_{\mathbf{A}}$ 2
$2.10\times {10}^{-3}$	3,750,911	19.39

¹ At episode 5820. Sliding window of 150 episodes. ² At episode 5820. Sliding window of 25,000 steps.

. The noise variances ${\sigma }^2$ and vertical positions $y_{\mathrm{obs}}$ generated randomly during the training in terms of their distributions are presented in Figure 10. The mean values of their distributions $\overline{{\sigma }^2}=12.40$ ${\mathrm{m}}^2$ and ${\overline{y}}_{\mathrm{obs}}=2745.70$ m denote that the agent is trained effectively from diverse scenarios of uniformly distributed random values.

The exploding values of the critic loss can be attributed to the poor observations normalisation or issues related to numerical differentiation. Specifically, some observations are found to explode during the training, such as the ${\ddot{e}}_{\mathrm{XT}}$, as presented in Figure 11. Retraining the ${\mathrm{agent}}_{\mathrm{A}}$, while using hard limiters on these observations, improves the convergence of the critic loss. Similar effects can be derived by adjusting the timestep, using a different solver, or employing other differentiation techniques, but potentially at the expense of longer simulation periods. However, this is presented only as a critical discussion on the effect of exploding observations on the training performance. The initial settings are maintained as the optimisation of the observations normalisation and numerical differentiation exceed the scope of this study.

4.2. Simulation Results

The trained ${\mathrm{agent}}_{\mathrm{A}}$ is simulated in various scenarios within the trained noise levels envelope, but for brevity, only the results pertaining to a single obstacle position on the middle of the path $x_{\mathrm{obs}}=2500$ m and $y_{\mathrm{obs}}=2500$ m and minimum and maximum noise variances ${\sigma }^2=0$ ${\mathrm{m}}^2$ and ${\sigma }^2=25$ ${\mathrm{m}}^2$, respectively, are presented. The agent’s decision-making in terms of the conducted manoeuvring in each scenario is presented in Figure 12 and in Video S1. The results are derived 3.57 times faster than the defined simulation timestep, which suggests the efficiency of employing the agent in real-time applications. Also, the same random seeds are used for the reproducibility of the generated noises in the LIDAR measurements and consistency of the results. Finally, considering that the timesteps × and ∗ refer to the first and minimum obstacle detections, respectively, each manoeuvring simulation is decomposed into three distinct stages as path following manoeuvre, evasive manoeuvre, and path recovery manoeuvre.

The returns that the agent accumulates are 695.72 when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and 708.32 when ${\sigma }^2=0$ ${\mathrm{m}}^2$, which suggest that the agent’s decision-making is different in each scenario. Specifically, the decomposition of each return to its reward functions, as presented prior in Equation (13), indicates that $R_1$ corresponds to 74.9% of the difference from ${\sigma }^2=25$ ${\mathrm{m}}^2$ to ${\sigma }^2=0$ ${\mathrm{m}}^2$, followed by the $R_4$, $R_2$, and $R_3$ that correspond to −39.9%, 35.4%, and 29.6%, respectively. This implies that the higher accumulated return when the noise variance is lower is due to the agent’s decision-making exploiting more the path following, nominal navigation, and actuator control rewards against the collision avoidance reward, which is manifested as smaller evasive manoeuvres. The opposite applies when the noise variance is higher resulting in larger evasive manoeuvres. This lends credence to the notion of the agent’s sophisticated decision-making, capable of prioritising safety over efficiency depending on the noise variance, which is desirable from safety perspective considering the increased uncertainties under higher noise variances and the limited temporal and spatial margins for error during reactive collision avoidance.

The actor’s policy in terms of the commanded rudder angle ${\delta }_{\mathrm{C}}$ per timestep t is presented in Figure 13. It is noted that the commanded rudder angles become more erratic when ${\sigma }^2=25$ ${\mathrm{m}}^2$, potentially due to the uncertainties of the noisy measurements. For instance, a LIDAR measurement less than its maximum detecting range clearly indicates a detection of an obstacle when ${\sigma }^2=0$ ${\mathrm{m}}^2$, whereas the distinction between noisy LIDAR measurements and detection of an obstacle becomes more uncertain when ${\sigma }^2=25$ ${\mathrm{m}}^2$. However, despite this uncertainty, the actor deflects the rudder angle to initiate an evasive manoeuvre only after the first obstacle detection, as indicated by the timesteps ×. This can be attributed to the great feature extraction capabilities of DNNs, enabling them to discern genuine obstacle detections amidst the noisy measurements.

In addition, it is noted that the actor’s policy is inherently different in each scenario, not only in terms of its erratic output. Specifically, the absolute area under the commanded rudder angles is increased by 25.2% when ${\sigma }^2=25$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=0$ ${\mathrm{m}}^2$, where the normalised area per timestep for the evasive manoeuvre corresponds to 55.3% of the difference, followed by the path following and path recovery manoeuvres that correspond to 28.7% and 16.0%, respectively. In addition, the magnitude of the commanded rudder angles at the minimum obstacle detections are 7.76 deg when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and 5.97 deg ${\sigma }^2=0$ ${\mathrm{m}}^2$, as indicated by the timesteps ∗. It can be derived that the prior finding on the agent’s sophisticated decision-making conducting different evasive manoeuvres depending on the noise variance is attributed to the difference between the commanded rudder angles, which is particularly evident during the evasive manoeuvres. This lends credence to the notion of the actor’s expressiveness, stemming from its policy’s ability to capture the complexity and richness across different noise variances and provide a different output, which is desirable considering the dynamic operating conditions affecting the stochastic nature of noise differently.

The critic’s estimation in terms of the expected return Q-value per return is presented in Figure 14. It is worth noting that despite the role of the critic’s estimation on the actor’s policy improvement taking place only during the training of the agent, the investigation of the Q-value can give meaningful insight to interpret the learned policy, as discussed herein. It is noted that the Q-values are more closely spread throughout the diagonal of the optimal estimation when ${\sigma }^2=0$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=25$ ${\mathrm{m}}^2$, which suggests that the critic makes more accurate estimations consistently when the noise variance is lower. Specifically, the respective errors between the Q-value and actual return at the first and minimum obstacle detections are −156.82 and −49.92, respectively, when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and −96.88 and −25.43, respectively, when ${\sigma }^2=0$ ${\mathrm{m}}^2$, as indicated by the timesteps × and ∗, respectively.

In addition, it is noted that the estimations are inherently different in each scenario, not only in terms of its accuracy. Specifically, the absolute area under the Q-values is reduced by −18.4% when ${\sigma }^2=25$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=0$ ${\mathrm{m}}^2$, where the normalised area per timestep for the path following manoeuvre corresponds to 57.2% of the difference, followed by the evasive manoeuvre and path recovery manoeuvre that correspond to 39.5% and 3.3%, respectively. The difference between the Q-values that is particularly evident during the path following manoeuvre even when no obstacle detection has occurred, suggests that the critic exhibits a general trend towards conservatism over optimism when the noise variance is higher. Considering that the reward functions remain unchanged regardless of the noise variance, the critic’s conservatism can be attributed to the uncertainties of triggering the collision criterion. For instance, given equal obstacle positions and evasive manoeuvres in two episodes during the agent’s training, the probability of the collision criterion being triggered is higher in the episode with the higher noise variance, which if triggered, leads to the automatic termination of the episode and consequently to the accumulation of a lower return.

To compensate that, the critic learns to underestimate its expected return when the noise variance is higher, which encourages the actor not to exploit the current policy but rather to explore. Exploitation of the policy corresponds to increased tendency towards maximising the return manifested as smaller evasive manoeuvres according to the prior finding on the agent’s decision-making over the rewards. Exploration of the policy corresponds to increased tendency towards exploring different commanded rudder angles manifested as larger evasive manoeuvres, which verifies the prior finding on the actor’s learned policy to output larger commanded rudder angles when the noise variance is higher. This lends credence to the notion of the critic’s adaptability, stemming from its ability to adjust its estimation of the expected return depending on the noise variance, which is desirable from safety perspective considering conservatism during higher collision risk.

4.3. Sensitivity Analysis

Sensitivity analysis of the trained ${\mathrm{agent}}_{\mathrm{A}}$’s actor and critic to noise variance is conducted, considering the prior findings suggesting its impact on their output. Specifically, the noise variance ${\sigma }^2$ affects the noise level of the LIDAR measurements observation d and the value of the noise variance observation $\mathcal{N}$, as presented prior in Equation (14). To investigate the sensitivity to each of these observations, they are decoupled by assigning two independent noise variances as ${\sigma }_d^2$ and ${\sigma }_{\mathcal{N}}^2$, respectively. The sensitivity is quantified by measuring the errors of the actor’s and critic’s outputs in terms of the commanded rudder angles and expected returns $e_{{\delta }_{\mathrm{C}}}$ and $e_Q$, respectively, using as baselines for the analysis their original outputs, as presented prior in Section 4.2.

The agent is simulated in various scenarios investigating different combinations of noise variances ${\sigma }_d^2=[0,25]$ ${\mathrm{m}}^2$ and ${\sigma }_{\mathcal{N}}^2=[0,25]$ ${\mathrm{m}}^2$, but for brevity only the results pertaining to their minimum and maximum values are discussed. The actor’s and critic’s outputs when ${\sigma }_d^2=25$ ${\mathrm{m}}^2$ and ${\sigma }_{\mathcal{N}}^2=0$ ${\mathrm{m}}^2$ and when ${\sigma }_d^2=0$ ${\mathrm{m}}^2$ and ${\sigma }_{\mathcal{N}}=25$ ${\mathrm{m}}^2$ are presented in Figure 15 and Figure 16. Qualitative evaluation suggests that their outputs resemble the original outputs, as presented prior in Figure 13 and Figure 14, where the similarities are found only when the noise variance ${\sigma }_{\mathcal{N}}^2$ is equal to the ${\sigma }^2$, regardless of the ${\sigma }_d^2$. The erratic output is introduced only when ${\sigma }_d^2=25$ ${\mathrm{m}}^2$, which suggests that the erratic output is intrinsic to the LIDAR measurements observation d and not to the noise variance observation $\mathcal{N}$.

The results of the sensitivity analysis in terms of the distributions of the actor’s and critic’s errors $e_{{\delta }_{\mathrm{C}}}$ and $e_Q$, respectively, are presented in

Table 3. Sensitivity analysis of the ${\mathrm{agent}}_{\mathrm{A}}$’s actor and critic to different noise variance combinations ${\sigma }_d^2$ and ${\sigma }_{\mathcal{N}}^2$.

${\mathit{\sigma }}^2$ $\left({\mathbf{m}}^2\right)$	${\mathit{\sigma }}_{\mathit{d}}^2$ $\left({\mathbf{m}}^2\right)$	${\mathit{\sigma }}_{\mathcal{N}}^2$ $\left({\mathbf{m}}^2\right)$	${\overline{\mathit{e}}}_{{\mathit{\delta }}_{\mathbf{C}}}$ $\left(\mathbf{deg}\right)$	${\mathit{\sigma }}_{{\mathit{e}}_{{\mathit{\delta }}_{\mathbf{C}}}}$ $\left(\mathbf{deg}\right)$	${\mathit{e}}_{{\mathit{\delta }}_{\mathbf{C}},\mathbf{min}}$ $\left(\mathbf{deg}\right)$	${\mathit{e}}_{{\mathit{\delta }}_{\mathbf{C}},\mathbf{max}}$ $\left(\mathbf{deg}\right)$
0	25	0	$-3.95\times {10}^3$	0.48	−5.47	2.60
0	0	25	$3.35\times {10}^3$	1.53	−13.60	7.54
25	25	0	$-7.23\times {10}^3$	1.61	−7.52	13.60
25	0	25	$1.08\times {10}^3$	0.36	−2.38	3.46
${\mathbf{\sigma }}^{\mathbf{2}}$ $\left({\mathbf{m}}^{\mathbf{2}}\right)$	${\mathbf{\sigma }}_{\mathit{d}}^{\mathbf{2}}$ $\left({\mathbf{m}}^{\mathbf{2}}\right)$	${\mathbf{\sigma }}_{\mathcal{N}}^{\mathbf{2}}$ $\left({\mathbf{m}}^{\mathbf{2}}\right)$	${\overline{\mathit{e}}}_{\mathit{Q}}$	${\mathbf{\sigma }}_{{\mathit{e}}_{\mathit{Q}}}$	${\mathit{e}}_{\mathit{Q},\mathbf{min}}$	${\mathit{e}}_{\mathit{Q},\mathbf{max}}$
0	25	0	4.26	16.41	−25.90	109.03
0	0	25	53.72	17.96	−26.14	80.98
25	25	0	−49.38	21.05	−87.22	59.03
25	0	25	0.03	2.37	−10.49	42.22

, Figure 17 and Figure 18. It is noted that the actor’s error $e_{{\delta }_{\mathrm{C}}}$ exhibits major difference on the standard deviation values ${\sigma }_{e_{{\delta }_{\mathrm{C}}}}$, which denote variability between the commanded rudder angles similar to the prior finding on the actor’s output. The critic’s error $e_Q$ exhibits major difference on the mean values ${\overline{e}}_Q$, which denote central tendency deviation between the expected returns similar to the prior findings on the critic’s output. Specifically, the differences are evident during the variations of the noise variance ${\sigma }_{\mathcal{N}}^2$ with the standard deviation and mean values of the actor’s and critic’s errors ${\sigma }_{e_{{\delta }_{\mathrm{C}}}}$ and ${\overline{e}}_Q$ differing up to 347.2% and −164,700.0%, respectively, compared to their respective values during the variations of the noise variance ${\sigma }_d^2$, which denotes that both the actor and critic are significantly more sensitive to the noise variance ${\sigma }_{\mathcal{N}}^2$. This lends credence to the notion that the noise variance observation $\mathcal{N}$ is the most critical one, to which the sophisticated decision-making of the agent, expressiveness of the actor’s policy, and adaptability of the critic’s estimation can be attributed. The practical implications of this notion is the importance of signal processing techniques not only in filtering out the noise, but also in accurately estimating the noise variance value from the noisy measurements in order to guarantee the expected decision-making from the agent. It provides a way forward towards a more effective and efficient integration between signal processing and decision-making techniques, which is pivotal to ensure safety of autonomous systems.

4.4. Robustness Verification

The robustness of the ${\mathrm{agent}}_{\mathrm{A}}$’s decision-making against noise variances in terms of the robustness metrics for path following and collision avoidance $R{M}_{\times }$ and $R{M}_{\ast }$, respectively, is presented in

Table 4. Robustness metrics of the ${\mathrm{agent}}_{\mathrm{A}}$ against noise variances ${\sigma }^2$.

${\mathit{\sigma }}^2$ $\left({\mathbf{m}}^2\right)$	${\mathit{RM}}_{\mathbf{\times }\mathbf{,}\mathit{A}}$ $\mathbf{\left(}\mathbf{m}\mathbf{\right)}$	${\mathit{RM}}_{\mathbf{\ast }\mathbf{,}\mathit{A}}$ $\mathbf{\left(}\mathbf{m}\mathbf{\right)}$
0	0.14	40.41
10	0.24	38.27
20	0.75	42.87
30	1.69	50.17
40	2.00	53.59
50	4.57	58.14
60	14.63	64.33

and Figure 19. Specifically, the robustness metrics are measured within and outside the trained noise variance envelope, until the robustness thresholds are violated. It is worth noting that the maximum cross-track errors $|e_{\mathrm{XT},\max }|$ are measured in the absence and presence of the obstacle as an indication of the expected maximum path deviation and maximum manoeuvre for the respective robustness metrics. It is noted that the robustness threshold for path following is compliant until when ${\sigma }^2=58$ ${\mathrm{m}}^2$, which is an increase of 132% from its maximum trained value. The agent’s decision-making in terms of the conducted manoeuvring when σ² = 0 m² and σ² = 58 m² is presented in Video S2. Specifically, the robustness metric for path following $R{M}_{\times }$ decreases approximately from ${\sigma }^2=40$ ${\mathrm{m}}^2$ due to early path deviation prior to obstacle detection. However, it is worth noting that the robustness thresholds for path following as $B/2$ is already a stringent requirement considering that typical XTLs are of a few NM [87].

The robustness threshold for collision avoidance is compliant even when the threshold for path following is violated. Specifically, the robustness metric for collision avoidance $R{M}_{\ast }$ exhibits an increasing trend, where it is increased by 15.5% when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and by 52.6% when ${\sigma }^2=58$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=0$ ${\mathrm{m}}^2$. It is worth noting that the increase of the robustness metric for collision avoidance is noted even prior to the early path deviation of the robustness metric for path following taking effect from ${\sigma }^2=40$ ${\mathrm{m}}^2$. It can be derived that the agent’s decision-making for path following and collision avoidance is robust against noise variances, due to the agent’s ability to generalise its sophisticated decision-making of prioritising safety over efficiency across higher noise variances. This lends credence to the notion that the noise variance observation $\mathcal{N}$ continuous to exhibit critical importance on the agent’s decision-making even beyond the trained noise level envelope.

4.5. Effectiveness of the Proposed Methodology

${\mathrm{Agent}}_{\mathrm{B}}$ that exhibits the same hyperparameter settings as the ${\mathrm{agent}}_{\mathrm{A}}$ is trained within the same training framework, but without the noise variance observation $\mathcal{N}$, to investigate the effectiveness of the proposed methodology. The training performance of the ${\mathrm{agent}}_{\mathrm{B}}$ in terms of the return per episode, actor loss per step, and critic loss per step is presented in Figure 20. The training of the ${\mathrm{agent}}_{\mathrm{B}}$ is terminated at episode 5970, where it exhibits similar convergence trends as the ${\mathrm{agent}}_{\mathrm{A}}$. The minimum and maximum values of the training performance metrics derived during the training and their respective moving medians are presented in

Table 5. Training performance metrics during the training of ${\mathrm{agent}}_{\mathrm{B}}$. The training performance metrics of the ${\mathrm{agent}}_{\mathrm{A}}$ are presented for comparison.

${\mathbf{Return}}_{\mathbf{A},\mathbf{min}}$	${\mathbf{Return}}_{\mathbf{A},\mathbf{max}}$	$\mathbf{Moving}{\mathbf{Median}}_{\mathbf{A}}$ 1	${\mathbf{Return}}_{\mathbf{B},\mathbf{min}}$	${\mathbf{Return}}_{\mathbf{B},\mathbf{max}}$	$\mathbf{Moving}{\mathbf{Median}}_{\mathbf{B}}$ 2
−67.42	749.72	633.84	−65.25	751.75	719.89
$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{min}}$	$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{median}}_{\mathbf{A}}$ 3	$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{B}\mathbf{,}\mathbf{min}}$	$\mathbf{Actor}{\mathbf{loss}}_{\mathbf{B}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{median}}_{\mathbf{B}}$ 4
−344.62	−0.45	−288.36	−296.91	−0.21	−281.76
$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{min}}$	$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{A}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{median}}_{\mathbf{A}}$ 3	$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{B}\mathbf{,}\mathbf{min}}$	$\mathbf{Critic}{\mathbf{loss}}_{\mathbf{B}\mathbf{,}\mathbf{max}}$	$\mathbf{Moving}{\mathbf{median}}_{\mathbf{B}}$ 4
$2.10\times {10}^{-3}$	3,750,911	19.39	$1.69\times {10}^{-3}$	651,757.60	18.97

¹ At episode 5820. Sliding window of 150 episodes. ² At episode 5970. Sliding window of 150 episodes. ³ At episode 5820. Sliding window of 25,000 steps. ⁴ At episode 5970. Sliding window of 25,000 steps.

.

The trained ${\mathrm{agent}}_{\mathrm{B}}$ is simulated in the same scenarios as the ${\mathrm{agent}}_{\mathrm{A}}$, but for brevity only the results pertaining to the minimum and maximum noise variances ${\sigma }^2=0$ ${\mathrm{m}}^2$ and ${\sigma }^2=25$ ${\mathrm{m}}^2$, respectively, are presented. The returns that the ${\mathrm{agent}}_{\mathrm{B}}$ accumulates are 770.68 when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and 771.46 when ${\sigma }^2=0$ ${\mathrm{m}}^2$, which suggest that the agent’s decision-making is not significantly different in each scenario. The actor’s policy in terms of the commanded rudder angle ${\delta }_{\mathrm{C}}$ per timestep t is presented in Figure 21. The absolute area under the commanded rudder angles is increased only by 9.8% when ${\sigma }^2=25$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=0$ ${\mathrm{m}}^2$ and the magnitude of the commanded rudder angles at the minimum obstacle detections are 9.71 deg when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and 9.70 deg when ${\sigma }^2=0$ ${\mathrm{m}}^2$, as indicated by the timesteps ∗, which suggest that the commanded rudder angles are not significantly different in each scenario, except for the erratic output noted when ${\sigma }^2=25$ ${\mathrm{m}}^2$.

The critic’s estimation in terms of the expected return Q-value per return is presented in Figure 22. The absolute area under the Q-values is reduced only by −0.1% when ${\sigma }^2=25$ ${\mathrm{m}}^2$ compared to when ${\sigma }^2=0$ ${\mathrm{m}}^2$ and the errors between the estimated Q-value and actual return at the first and minimum obstacle detections are −183.83 and −117.16, respectively, when ${\sigma }^2=25$ ${\mathrm{m}}^2$ and −187.78 and −116.74, respectively, when ${\sigma }^2=0$ ${\mathrm{m}}^2$, as indicated by the timesteps × and ∗, respectively, which suggest that the Q-values are not significantly different in each scenario.

The robustness of the ${\mathrm{agent}}_{\mathrm{B}}$’s decision-making against noise variances in terms of the robustness metrics for path following and collision avoidance $R{M}_{\times }$ and $R{M}_{\ast }$, respectively, is presented in

Table 6. Robustness metrics of the ${\mathrm{agent}}_{\mathrm{B}}$ against noise variances ${\sigma }^2$. Robustness metrics of the ${\mathrm{agent}}_{\mathrm{A}}$ are presented for comparison.

${\mathit{\sigma }}^2$$\left({\mathbf{m}}^2\right)$	${\mathit{RM}}_{\times ,\mathit{A}}$$\left(\mathbf{m}\right)$	${\mathit{RM}}_{\mathbf{\ast }\mathbf{,}\mathit{A}}$$\mathbf{\left(}\mathbf{m}\mathbf{\right)}$	${\mathit{\sigma }}^{\mathbf{2}}$  $\mathbf{\left(}{\mathbf{m}}^{\mathbf{2}}\mathbf{\right)}$	${\mathit{RM}}_{\mathbf{\times }\mathbf{,}\mathit{B}}$  $\mathbf{\left(}\mathbf{m}\mathbf{\right)}$	${\mathit{RM}}_{\mathbf{\ast }\mathbf{,}\mathit{B}}$  $\mathbf{\left(}\mathbf{m}\mathbf{\right)}$
0	0.14	40.41	0	0.75	31.79
10	0.24	38.27	10	0.78	23.70
20	0.75	42.87	20	0.60	19.60
30	1.69	50.17	30	0.91	16.99
40	2.00	53.59	40	0.55	14.53
50	4.57	58.14	50	0.65	11.44

and Figure 23. It is noted that the robustness threshold for path following is compliant even outside of the trained noise levels envelope. However, the robustness threshold for collision avoidance is violated when ${\sigma }^2=44$ m, which is an increase of only 76% from its maximum trained value. The agent_B’s decision-making in terms of the conducted manoeuvring when σ² = 0 m² and σ² = 44 m² is presented in Video S3. The maximum manoeuvres indicated by the maximum cross-track errors $|e_{\mathrm{XT},\max }|$ suggest that the ${\mathrm{agent}}_{\mathrm{B}}$’s decision-making across different noise variances remain the same, consequently degrading the robustness against noise variances, especially for collision avoidance. This lends credence to the notion that training the agent without the noise variance observation $\mathcal{N}$ comes at the expense of reduced sophisticated decision-making, lack of expressiveness of the actor’s policy and adaptability of the critic’s estimation, with worse robustness and generalisation against noise variances, highlighting the effectiveness of the proposed methodology.

5. Conclusions

In this study, a methodology was developed to enhance the robustness of decision-making for the reactive collision avoidance of autonomous ships against various perception sensor noise levels. Digital twins of the ship manoeuvrability, perception sensor, and map were employed. A Gaussian-based noise model was employed to simulate the noisy measurements of the perception sensor. The decision-making problem was formulated as a Markov decision process, where rewards pertaining to path following, nominal navigation, actuator control, and collision avoidance objectives were identified. The noisy measurements of the perception sensor and its noise variance were incorporated into the decision-making as observations. A training framework was developed pertaining to the trained noise levels envelope. Robustness metrics that quantify the robustness of the agent’s decision-making for path following and collision avoidance were defined, which were measured in various scenarios within and outside the trained noise levels envelope. A deep deterministic policy gradient agent was employed, where the actor was set to output the commanded rudder angles and the critic to estimate the expected return. A case study of a container ship using a light detection and ranging (LIDAR) in a single static obstacle environment was investigated. The main findings of this study are as follows.

• The trained agent exhibited enhanced sophisticated decision-making prioritising safety over efficiency when the noise variance was higher. Specifically, the decision-making of the agent over the rewards exploited less the path following, nominal navigation, and actuator control rewards over the collision avoidance reward, manifested as larger evasive manoeuvres.
• The actor’s policy exhibited enhanced expressiveness by outputting different commanded rudder angles depending on the noise variance. Specifically, major difference in the commanded rudder angles was evident during the evasive manoeuvre verifying the agent’s sophisticated decision-making. Also, the actor initiated an evasive manoeuvre by successfully discerning genuine obstacle detections amidst noisy measurements, which was attributed to the great feature extraction capabilities of deep neural networks.
• The critic’s estimation exhibited enhanced adaptability by adjusting its optimism and conservatism depending on the noise variance. Specifically, the critic learned to underestimate the expected return when the noise variance was higher verifying the actor’s policy expressiveness. The critic’s conservatism was attributed to the higher probability of triggering the collision criterion in higher noise variances.
• Sensitivity analysis indicated the criticality of the noise variance observation for the agent’s decision-making. Specifically, major differences in the actor’s and critic’s outputs were noted during the variations of the noise variance observation and not during the variations of the LIDAR measurements observation. However, the LIDAR measurements observation was found to be responsible for the erratic output.
• The robustness of the agent’s decision-making against noise variance was verified up to 132% from its maximum trained value. Specifically, the robustness for path following decreased with the increase of noise variance, but the robustness for collision avoidance increased by 52.6% from its initial value considering minimum noise variance. The increase of robustness was attributed to the agent’s ability to generalise its sophisticated decision-making across higher noise variances.
• The robustness of the agent’s decision-making against noise variance was verified only up to 76% from its maximum trained value, when trained without the noise variance observation. Specifically, the robustness for collision avoidance exhibited a decreasing trend with the increase of noise variances. This was attributed to the far less sophisticated decision-making, lack of expressiveness of the actor’s policy and adaptability of the critic’s estimation, with worse generalisation capabilities, highlighting the effectiveness of the proposed methodology.

The limitations of this study include the assumption of ideal environmental conditions with no disturbances and consideration of a single static obstacle. The noise model was also generic, whereas noise was only considered on the distance measurements and not on the angular measurements. Another limitation is pertaining to the heavy computation effort required for the training of deep reinforcement learning agents, considering the extensive time and expensive computation setup. Future studies entail the investigation of more realistic environments, including environmental disturbances and specific noise models pertaining to specific environmental conditions, data fusion of multiple perception sensors, and comparative analysis of alternative deep reinforcement learning algorithms. Nonetheless, this study contributes towards the development of autonomous systems that can make safe and robust decisions under uncertainty.