Three-Dimensional Autonomous Navigation of Unmanned Underwater Vehicle Based on Deep Reinforcement Learning and Adaptive Line-of-Sight Guidance

Abstract

Unmanned underwater vehicles (UUVs) face significant challenges in achieving safe and efficient autonomous navigation in complex marine environments due to uncertain perception, dynamic obstacles, and nonlinear coupled motion control. This study proposes a hierarchical autonomous navigation framework that integrates improved particle swarm optimization (PSO) for 3D global route planning, and a deep deterministic policy gradient (DDPG) algorithm enhanced by noisy networks and proportional prioritized experience replay (PPER) for local collision avoidance. To address dynamic sideslip and current-induced deviations during execution, a novel 3D adaptive line-of-sight (ALOS) guidance method is developed, which decouples nonlinear motion in horizontal and vertical planes and ensures robust tracking. The global planner incorporates a multi-objective cost function that considers yaw and pitch adjustments, while the improved PSO employs nonlinearly synchronized adaptive weights to enhance convergence and avoid local minima. For local avoidance, the proposed DDPG framework incorporates a memory-enhanced state–action representation, GRU-based temporal processing, and stratified sample replay to enhance learning stability and exploration. Simulation results indicate that the proposed method reduces route length by 5.96% and planning time by 82.9% compared to baseline algorithms in dynamic scenarios, it achieves an up to 11% higher success rate and 10% better efficiency than SAC and standard DDPG. The 3D ALOS controller outperforms existing guidance strategies under time-varying currents, ensuring smoother tracking and reduced actuator effort.

1. Introduction

When a UUV performs global route planning in a high-dimensional environment, it faces issues such as long calculation times and a single optimization target, which limits its ability to efficiently find routes in large sea areas. Improving the computational efficiency of global route planning and balancing the navigation distance, planning efficiency, and the feasibility of planning under a multi-objective optimization framework are core issues addressed in this study.

Based on global route planning, the UUV needs to perform local collision avoidance in observable environments under conditions of sparse rewards. The current related collision avoidance algorithms still have deficiencies in terms of reliability, real-time performance, generalization ability, and learning sample utilization of collision avoidance planning. They are unable to make effective decisions for collision avoidance planning with multiple dynamic obstacles, making it difficult for UUVs to make effective decisions when dealing with dynamic obstacles and uncertain environments. How to build an efficient local collision avoidance strategy and improve real-time obstacle avoidance capabilities is a key challenge for UUV intelligent navigation technology.

Deep Reinforcement Learning (DRL) has powerful self-learning capabilities. It does not require accurate modeling of the obstacle environment in which the UUV operates, nor does it heavily rely on sensor accuracy. Huang et al. [1] proposed the Double Deep Q Network (DQN) algorithm to optimize collision avoidance and navigation planning. The Double DQN algorithm decouples action selection and evaluation, obtaining the action index corresponding to the maximum Q value through the current network and inputting it into the target network to calculate the target Q value. The results show that overestimation is reduced, and the Double DQN algorithm can effectively process complex environmental information and perform optimal navigation planning. The application of value function DRL in autonomous collision avoidance has been confirmed in existing studies. Compared with value-based and model-free methods, policy gradient-based methods have great advantages in processing continuous actions. Xi [2] proposed an end-to-end perception-planning-execution method based on a two-layer deep deterministic policy gradient (DDPG) algorithm to address the problems of sparse rewards, single strategies, and poor environmental adaptability in UUV local motion planning tasks, in order to overcome the challenges of end-to-end methods that directly output control force in terms of training and learning. Wu et al. [3] proposed an end-to-end UUV motion framework based on the PPO algorithm. The framework directly takes raw sonar perception information as input and incorporates multiple objective factors, such as collision avoidance, collision, and speed, into the reward function. Experimental results show that the algorithm can enable a UUV to achieve effective collision avoidance in an unknown underwater environment full of obstacles. However, he did not consider the nonlinear characteristics of UUV kinematics and dynamics. To address this problem, Hou et al. [4] designed a DDPG collision avoidance algorithm in which two neural networks control the thruster and rudder angle position of the UUV, respectively. The reward function considers the distance from the obstacle to the sonar, as well as the difference between the current position and the previous position to the target point. Simulation results show that the UUV can plan a collision-free path in an unknown continuous environment. Sun et al. [5] demonstrated a collision avoidance planning method that integrates DDPG and Artificial Potential Field (APF), taking sensor information as input and outputting longitudinal velocity and bow angle. This method combines APF to design a reward function and uses a SumTree structure to store experience samples according to their importance, giving priority to extracting high-quality samples, and accelerating the convergence of the SumTree-DDPG algorithm. Li et al. [6] introduced a goal-oriented post-experience replay method to address the inherent problem of sparse rewards in UUV local motion planning tasks. Simulation results show that this method effectively improves the stability of training and improves sample efficiency. In addition, an additional APF method dynamic collision avoidance emergency protection mechanism, is designed for UUV. The combination of DRL and APF has the potential to improve the performance of intelligent agents in complex environments, but DRL and APF are both reactive collision avoidance algorithms. If they are combined, the UUV decision-making and planning will conflict, and a coordination mechanism needs to be designed. Tang [7] uses sonar data as the input of the network and uses the policy network output of the TD3 framework as the motion command of the UUV to complete the collision avoidance planning of the UUV in a 3D unknown environment. In the above papers, the interference of ocean currents is not considered, and the obstacles in the environment are regular. To address these problems, Huang [8] proposed a UUV reactive collision avoidance controller based on the Soft Actor–Critic (SAC) method, and designed the state space, action space, and multi-objective function of reactive collision avoidance. The simulation results show that this method can realize collision avoidance planning in sparse 3D static and dynamic environments.

From the above research results, we know that there are a large number of scholars studying collision avoidance planners based on the DRL algorithm, but only a few papers are studying samples in the learning process. During the model training process, the blind exploration of the DRL algorithm generates a large number of low-quality samples, which will reduce the utilization rate of effective sample data, thereby greatly reducing the learning efficiency. In addition, there is little research on the stability and generalization of the training model. Therefore, this also provides research value and significance for the work of this paper.

UUV autonomous navigation planning is closely related to route tracking. Navigation planning is crucial for ensuring that UUVs provide safe routes and avoid obstacles during autonomous missions, while route tracking is the process of controlling and executing the UUV’s planned routes. The quality of navigation planning directly affects the executability of route tracking; that is, high-quality planned routes should have good command smoothness and eliminate chattering, which is conducive to UUV operation execution.

For 2D route tracking problems, line-of-sight (LOS) is a commonly used method that can be used to guide a UUV to track along a preset path. The LOS method calculates the LOS angle between the current position of the UUV and the target waypoint and generates corresponding heading instructions to gradually converge to the target path, thereby completing route tracking [9]. Fossen and Pettersen et al. [10] found the source of tracking error and proved the algorithm’s strong convergence and robustness to disturbances. Although the proportional LOS guidance method is effective and popular, significant tracking errors may occur during route tracking when the UUV is subjected to drift forces caused by ocean currents. To overcome this difficulty, researchers have completed a lot of work to mitigate the effects of sideslip. In reference [11], accelerometers are used to measure longitudinal and lateral accelerations, and then the corresponding velocities are calculated by integrating these measurements. Reference [12] proposed an integral LOS guidance method to achieve straight-line route tracking, which can avoid the risk of integrator saturation [13]. Reference [14] derived an improved integral LOS guidance method for tracking on curved paths. Reference [15] proposed direct and indirect integral LOS guidance methods for ships exposed to time-varying ocean currents. The above references mitigate the effects of sideslip angle by adding integral terms to the LOS guidance method. Integral LOS can only handle constant sideslip angles. When following a curved path or time-varying ocean disturbances, the sideslip angle will also change due to the change in ocean current disturbances over time. Secondly, the stability is reduced due to the existence of phase lag superposition integral terms. Liu et al. [16] used LOS based on an extended state observer [17] for route tracking of underactuated vehicles when the sideslip angle changes over time. In the marine environment, the actual motion of a UUV is affected by complex nonlinear coupling relationships, involving multiple degrees of freedom and complex motion states. The LOS method has been widely used in two-dimensional route tracking, but there are still challenges in terms of time-varying ocean currents, sideslip angle changes, and 3D path planning. Research route tracking methods suitable for 3D environments to solve the path deviation problem caused by the nonlinear coupled motion of UUVs in 3D space, so as to improve the autonomous navigation capability of UUVs in complex marine environments.

When UUVs perform tasks in the ocean environment, they are inevitably affected by time-varying ocean currents and dynamic sideslip angles, which significantly increase the complexity of route tracking. Traditional guidance methods have poor tracking stability under time-varying flow field disturbances and environmental changes, making it difficult to achieve high-precision three-dimensional route tracking. How to improve navigation stability and autonomous control capabilities in a high-dimensional environment is an important direction that underwater autonomous navigation technology needs to break through.

The main contributions of this article can be summarized as follows:

• The multi-level state perturbation guidance mechanism and the noisy network with multi-scale parameter noise fusion solve the training oscillation problem caused by fixed-scale noise, and the sampling number penalty term and stratified priority sampling strategy are used to improve learning efficiency and maintain the generalization ability of the DDPG model.
• The horizontal and vertical front sight vectors are designed to decouple the control design of UUV nonlinear motion, and their stable convergence is proved by theory, which solves the problem that the integral LOS guidance method is difficult to directly track the three-dimensional planning waypoints under the condition of UUV attitude change caused by ocean current disturbance.

2. Construction of UUV Autonomous Navigation Planning Model

In order to analyze the problem of UUV underwater maneuvering motion control and obstacle detection model construction, the north–east–down (NED), UUV coordinate system, and forward-looking sonar coordinate system were established, as shown in Figure 1. $x$, $y$, and $z$ represent the displacement of the UUV relative to the three axes of the NED coordinate system. A vehicle coordinate system $o_b-x_b{y}_b{z}_b$ fixed to the center of mass of the UUV was established to describe the translation and rotation of the UUV. Among them, the axis $o{x}_b$ is located in the longitudinal section of the UUV, pointing from the center of mass of the UUV to its bow direction; the $o{y}_b$ axis is perpendicular to the longitudinal section of the UUV, with the starboard direction of the UUV as the positive direction; the $o{z}_b$ axis is located in the longitudinal symmetry plane of the UUV, perpendicular to the $o{x}_b$ axis, and points to the bottom of the UUV. Table 1 shows the UUV motion parameters.

Figure 1. Coordinate systems of UUV and sonar.

Table 1. UUV motion parameters and symbols.

DOF	Linear Velocity/Angular Velocity	Position/Euler Angle
Surge	$u$	$x$
Sway	$v$	$y$
Heave	$w$	$z$
Roll	$p$	$\phi$
Pitch	$q$	$\theta$
Yaw	$w$	$\Psi$

2.1. Kinematic and Dynamic Model of 3-DOF UUV

Considering the 3-DOF underactuated UUV kinematic of $u$, $q$ and $r$, spatial motions such as swaying, yawing, and pitching can be realized, providing a model simulation basis for the subsequent 3D route optimization of UUV autonomous navigation planning and the planned route tracking motion in 3D space [18]:

$$\left\{\begin{array}{c}\dot{x}=u\cos \Psi \cos \theta \hfill \\ \dot{y}=u\sin \Psi \cos \theta \hfill \\ \dot{z}=-u\sin \theta \hfill \\ \dot{\phi }=q\sin \theta \tan \theta +r\tan \theta \hfill \\ \dot{\theta }=q\hfill \\ \dot{\Psi }=r/\cos \theta , \theta \ne \pm {90}^{\circ }\hfill \end{array}\right.$$

(1)

The kinematics of the UUV should also satisfy the following constraints:

$$\left\{\begin{array}{l}-40°\le \theta \le 40°\\ 2\mathrm{kn}\le u\le 8\mathrm{kn}\\ -10°/\mathrm{s}\le q\le 10°/\mathrm{s}\\ -15°/\mathrm{s}\le r\le 15°/\mathrm{s}\\ -1\mathrm{kn}/\mathrm{s}\le \dot{u}\le 1\mathrm{kn}/\mathrm{s}\end{array}\right.$$

(2)

UUV 3-DOF dynamic model:

$$M\dot{V}+C(V)V+D(V)V=\tau$$

(3)

where $V={[u,q,r]}^T$ represents the UUV surge, pitch, and yaw.

Mass and added mass matrix $M$:

$$M=\left[\begin{array}{ccc}m-X_{\dot{u}}& 0& 0\\ 0& I_y-M_{\dot{q}}& 0\\ 0& 0& I_z-N_{\dot{r}}\end{array}\right]$$

(4)

where $X_{\dot{u}}$, $M_{\dot{q}}$ and $N_{\dot{r}}$ represent the equivalent additional inertia caused by the ocean current on the UUV’s motion along the three degrees of freedom of surge, pitch, and bow rotation, respectively, without considering the direct coupling of the 3-DOF in inertia.

Coriolis matrix $C(V)$:

$$C(V)=\left[\begin{array}{ccc}0& 0& -(m-X_{\dot{u}})(r+q)\\ 0& 0& 0\\ (m-X_{\dot{u}})(r+q)& 0& 0\end{array}\right]$$

(5)

where $(m-X_{\dot{u}})(r+q)$ is the inertial force term caused by motion coupling. A simplified asymmetric structure is adopted, and only the bow rotation and pitch cross terms caused by the change in surge velocity are retained.

Damping matrix $D(V)$:

$$D(V)=\left[\begin{array}{ccc}{X}_u+X_{\left|u\right|}\left|u\right|& 0& 0\\ 0& M_q+M_{\left|q\right|}\left|q\right|& 0\\ 0& 0& N_r+N_{\left|r\right|}\left|r\right|\end{array}\right]$$

(6)

In the formula, $X_u$, $M_q$ and $N_r$ represent the linear damping coefficients of the 3-DOF $u$, $q$ and $r$, respectively; $X_{\left|u\right|}$, $M_{\left|q\right|}$ and $N_{\left|r\right|}$ represent the nonlinear damping coefficients of the corresponding degrees of freedom, respectively; $\left|u\right|$, $\left|q\right|$ and $\left|r\right|$ take absolute values to reflect the physical property that nonlinear damping increases with increasing speed.

These control inputs are expressed in vector form as follows:

$$\tau ={\left[\begin{array}{ccc}{\tau }_u& {\tau }_q& {\tau }_r\end{array}\right]}^T$$

(7)

Among them, ${\tau }_u$ is the longitudinal thrust provided by the propeller, ${\tau }_q$ is the pitch moment provided by the horizontal rudder, and ${\tau }_r$ is the yaw turning moment provided by the rudder.

2.2. Three-Dimensional Global Environment Model

This study adopts a grid environment in the horizontal direction and discretizes the space in the vertical direction to form an environment representation suitable for navigation planning, as shown in Figure 2. $(x,y,z)$ is the coordinate of any point in 3D space, and the UUV planning space can be expressed as:

$$\{(x,y,z)|x_{\min }\le x\le x_{\max },y_{\min }\le y\le y_{\max },z_{\min }\le z\le z_{\max }\}$$

(8)

Among them, $x_{\min }$, $x_{\max }$, $y_{\min }$, $y_{\max }$, $z_{\min }$ and $z_{\max }$ are the value ranges of each dimension of UUV, respectively.

Figure 2. Schematic diagram of 3D space and slices.

When the UUV moves to point $(x,y,z)$ in 3D space, the altitude of the terrain corresponding to the UUV vertically is represented by $z_{xy}$ ($1<x<m$, $1<y<n$, $m$, $n$ are all positive integers). The two-dimensional matrix $V_{map}$ represents the environmental model of the UUV, and the index values $m$ and $n$ of $V_{map}$ represent the maximum boundaries of the three-dimensional ocean environment in the north and east directions. Therefore, each element in the matrix represents the altitude of the vertical terrain corresponding to the point:

$$V_{map}=\left[\begin{array}{cccc}{z}_{11}& z_{12}& \dots & z_{1n}\\ z_{21}& z_{22}& \dots & z_{2n}\\ \dots & \dots & \dots & \dots \\ z_{m1}& z_{m2}& \dots & z_{mn}\end{array}\right]$$

(9)

2.3. Time-Varying Ocean Current Disturbance Model

Assuming that in the NED coordinate system, the ocean current speed is $U_c$ and the direction is ${\Psi }_c$. The ocean current speed $U_c$ changes dynamically according to the first-order lag system as follows:

$$U_c(t+h)=e^{-h\cdot {\omega }_c}{U}_c(t)+(1-e^{-h\cdot {\omega }_c})U_{cd}$$

(10)

$$U_c(t)=U_c(t)+k\times \mathcal{N}(0,1)$$

(11)

where $U_c(t)$ is the current current speed; $U_{cd}$ is the target current speed; $h$ is the simulation step size; ${\omega }_c$ is the law of current speed change. $\mathcal{N}(0,1)$ is the normal distribution noise with a mean of 0 and a standard deviation of 1.

$${\Psi }_c(t+h)=e^{-h\cdot {\omega }_{\Psi }}{\Psi }_c(t)+(1-e^{-h\cdot {\omega }_{\Psi }}){\Psi }_{cd}$$

(12)

$${\Psi }_c(t)={\Psi }_c(t)+{\displaystyle \frac{\pi }{180}}\times {\displaystyle \frac{1}{20}}\times \mathcal{N}(0,1)$$

(13)

where ${\Psi }_c(t)$ is the current direction; ${\Psi }_{cd}$ is the target current direction; and ${\omega }_{\Psi }$ is the law of current direction change. Both $U_c$ and ${\Psi }_c$ are exponentially smooth changes, reflecting the existence of certain inertia and disturbance in natural current changes. At the same time, weak Gaussian white noise disturbances are superimposed to simulate small-scale uncertainties in the real ocean environment.

2.4. Noise Modeling and Characteristics Analysis of Forward-Looking Sonar Images

This paper uses the Oculus M750d multi-beam forward-looking sonar as a simulation detection prototype system, as shown in Figure 3. Its horizontal detection angle is 130° fan-shaped area, vertical angle is 20°, maximum range is 120 m, effective range is 0.1 m–120 m, operating frequency is 750 kHz, and the maximum update rate is 27 Hz. Its distance resolution is 1° and contains 512 beams.

Figure 3. Oculus M750d multibeam forward-looking sonar and sonar image.

In order to improve the effectiveness of subsequent image enhancement and target detection algorithms, it is necessary to systematically model and analyze the sources, characteristics, and statistical properties of noise in sonar images, as shown in Appendix A. Sonar image noise can be divided into three categories according to the noise source: self-noise, environmental noise and reverberation noise [19].

A multi-beam forward-looking sonar simulation model was created to simulate the underwater obstacle perception scene. The sonar horizontal opening angle was divided into a cluster every 10 degrees, and the minimum value among the distance values of the upper, middle and lower layers was selected as the corresponding detection distance $d_i$ of the sonar. The sonar measurement values of 512 beams were reduced to 13 dimensions $[d_0,d_1,d_2,\dots ,d_{12}]$, as shown in Figure 4.

Figure 4. Multibeam forward-looking sonar field of view detection modeling.

In order to accurately measure the position of obstacles in the global NED coordinate system, the multi-beam forward-looking sonar obtains the obstacle distance and angle information, and converts it into the NED coordinate system through coordinate transformation:

$$\left[\begin{array}{l}{x}_b^g\\ y_b^g\\ z_b^g\end{array}\right]=R\left(\mathrm{\Theta }\right)\left[\begin{array}{l}{x}_b^s+l\\ y_b^s\\ z_b^s\end{array}\right]+\left[\begin{array}{l}{x}_u^g\\ y_u^g\\ z_u^g\end{array}\right]$$

(14)

where $(x_b^g,y_b^g,z_b^g)$ is the coordinate of the obstacle in the NED coordinate system, $(x_b^s,y_b^s,z_b^s)$ is the coordinate of the obstacle measurement information converted to the UUV carrier coordinate system, $(x_u^g,y_u^g,z_u^g)$ is the coordinate of the UUV in the NED coordinate system, $l$ represents the distance from the sonar to the center of mass of the UUV, and the origin of the sonar coordinate is on the $x_o{x}_b$ axis of the UUV.

3. Autonomous Collision Avoidance Planning Method for UUV Based on Target Information Driven DDPG

3.1. Deep Deterministic Policy Gradients

We consider an AUV that interacts with the environment $E$ in discrete time steps. At time $t\in [1,T]$ the AUV takes an action $a_t$ according to the observation $s_t$ and receives a scalar reward $r_t$ (here, we assumed the environment is fully observed.) We model it as a Markov decision process with a state space $S$, action space $A$, state distribution $p(s_1)$, transition dynamics $p(s_{t+1}|s_t,a_t)$, and reward function $r(s_t,a_t)$. The sum of discounted future reward is defined as $R_t={\displaystyle {\sum }_{k=t}^{\infty }{\gamma }^{k-t}r(s_k,a_k)}$ with a discounting factor $\gamma \in (0,1)$. The AUV’s goal is to obtain a policy that maximizes the cumulative discounted reward $R_t$. We denote the discounted state distribution for a policy $\pi$ as ${\rho }^{\pi }$ and the action-value function $Q^{\pi }(s_t,a_t)={\mathrm{{\rm E}}}_{r_t,s_{t+1}\sim E, a_t\sim {\rho }^{\pi }}[R_t|s_t,a_t]$. The goal of UUV is to learn a series policy that maximizes the expected return $J={\mathrm{{\rm E}}}_{s_i,a_i\sim E,a_i\sim \pi }[R_1]$.

We use an off-policy approach to learn a deterministic target policy H from the trajectories generated by the random behavior policy $\pi (s,a)$. We average the state distribution of the behavior policy and transform the performance objective function into the value function of the target policy ${\mu }_{w^{\mu }}(s)$:

$$J_{\beta }({\mu }_{w^{\mu }})={\displaystyle {\int }_S{\rho }^{\beta }(s)V^{\mu }(s)ds={\displaystyle {\int }_S{\rho }^{\beta }(s)Q^{\mu }(s,{\mu }_{w^{\mu }}(s))ds}}$$

(15)

$${\nabla }_{w^{\mu }}{J}_{\beta }({\mu }_{w^{\mu }})\approx {\displaystyle {\int }_S{\rho }^{\beta }(s){\nabla }_{w^{\mu }}{\mu }_{w^{\mu }}(a|s)Q^{\mu }(s,a)ds}={\mathrm{{\rm E}}}_{s\sim {\rho }^{\beta }}[{\nabla }_{w^{\mu }}{\mu }_{w^{\mu }}(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{w^{\mu }}(s)}]$$

(16)

$${\delta }_t=r_t+\gamma Q^{w^Q}(s_{t+1},{\mu }_{w^{\mu }}(s_{t+1}))-Q^{w^Q}(s_t,a_t)$$

(17)

$$w_{t+1}^Q=w_t^Q+{\alpha }_{w^Q}{\delta }_t{\nabla }_{w^Q}{Q}^{w^Q}(s_t,a_t)$$

(18)

$$w_{t+1}^{\mu }=w_t^{\mu }+{\alpha }_{w^{\mu }}{\nabla }_{w^{\mu }}{\mu }_{w^{\mu }}(s_t){\nabla }_a{Q}^{w^Q}(s_t,a_t){|}_{a={\mu }_{w^{\mu }}(s)}$$

(19)

The validity of (16) does not depend on ${\nabla }_{w^{\mu }}{Q}^{{\mu }_{w^{\mu }}}(s,a)$, but it is approximated by a neural network. Replace the true value function $Q^{\mu }(s,a)$ with a differentiable value function $Q^w(s,a)$ in Formula (16) to derive an off-policy deterministic policy gradient update policy method. The Critic network estimates the true value $Q^w(s,a)\approx Q^{\mu }(s,a)$ from the trajectory generated by the policy $\beta (a|s)$ using an appropriate policy evaluation algorithm. In the following off-policy DDPG algorithm, the Critic network uses Q-learning updates to estimate the action value function.

3.2. Target Information Driven DDPG Algorithm Framework

(1)

State-space design with target information as input

In this paper, the Actor network input $s_t=[d_0,d_1,d_2,\dots ,d_{12},{\rho }_{ugd},{\rho }_{ug\Psi }]$ is expanded from a 14-dimensional vector to a 15-dimensional vector, which includes the processed sonar data $[d_0,d_1,d_2,\dots ,d_{12}]$, the distance ${\rho }_{ugd}=\mathrm{d}(p_{uuv}-p_{goal}{)}_t-\mathrm{d}(p_{uuv}-p_{goal}{)}_{t-1}$ between the current position of the UUV and the target point, and the deviation angle ${\rho }_{ug\Psi }=\Psi {(p_{uuv}-p_{goal})}_t-\Psi {(p_{uuv}-p_{goal})}_{t-1}$ between the UUV heading and the target point at two moments. The Critic network is 17-dimensional. In addition to the above inputs, it also includes the UUV’s strategy at the previous moment (including longitudinal accelerations and yaw angular velocity), which are normalized and used as the input of the network. Not only is the orientation information of the target point added, but also the strategy of the previous moment is introduced to reduce the probability of blind learning of the UUV in the case of sparse rewards, and make the UUV’s movement smoother.

(2)

Continuous action space design

DRL methods usually participate in exploration behavior by injecting noise into the action space. Adding noise to the action space is a common method to improve the performance of Dueling DQN. In the Q learning process, by applying the strategy, a certain proportion of random actions will be added when selecting actions. Different from the previous chapter, in this chapter, the continuous state quantities of the longitudinal acceleration and bow angular velocity of the UUV are used as the output strategy, and a noise network is used instead of adding noise to the strategy to improve the stability and generalization of the model.

(3)

DDPG network framework driven by target information

The improved DDPG algorithm network framework is shown in Figure 5. The algorithm obtains training data from the sample pool through the experience replay mechanism and uses the Critic network to calculate the Q value gradient information to guide the Actor network update, thereby continuously optimizing the policy network parameters. Compared with the DQN algorithm based on the value function, the DDPG algorithm shows higher stability in tasks in the continuous action space, higher policy optimization efficiency, and faster convergence speed, and the number of training steps required to reach the optimal solution is also significantly reduced. The network update in this paper uses the average reward of the round to eliminate the impact of the difference in the number of rounds on the evaluation results, making the evaluation results more stable and improving the stability of the model.

Figure 5. Target-oriented DDPG collision avoidance algorithm framework.

(4)

Comprehensive reward function

This paper defines a comprehensive reward function based on the environment in which the UUV is located. The comprehensive reward function designed in this section is expressed as four dynamic reward items:

$$R_t^1={\alpha }_d\ast \tanh (d_t/d_{sg})$$

(20)

where ${\alpha }_d$ represents the distance penalty factor, which is a negative number; $d_t$ is the distance between the UUV and the target point at time $t$, and $d_{sg}$ is the distance between the starting point and the target point.

$$R_t^2=\left\{\begin{array}{c}-2,\hspace{0.25em}\mathrm{if} d<(0.7\times d_{\max })\mathrm{and}-30°<{\theta }_{uo}<30°\hfill \\ -1,\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\mathrm{if} (0.7\times d_{\max })<d<d_{\max }\hfill \end{array}\right.$$

(21)

where $d$ is the distance between the UUV and the obstacle, $d_{\max }$ is the maximum detection distance of the sonar, and ${\theta }_{uo}$ is the angle between the UUV heading and the obstacle. If the UUV hits an obstacle or a boundary, then $R_t^3=-80$; if the UUV reaches the target point, then $R_t^4=50$.

The comprehensive reward function of this paper is the sum of $R_t^1$, $R_t^2$, $R_t^3$, $R_t^4$:

$$R_t=R_t^1+R_t^2+R_t^3+R_t^4$$

(22)

3.3. Design of DDPG Method with Noisy Network

DDPG uses a deterministic strategy. For complex environments in continuous space, it is often difficult to conduct sufficiently effective global exploration by only adding Gaussian noise or OU noise for exploration. Once it falls into a local optimum, it may lack higher-dimensional randomness to jump out of the local extreme point.

In DDPG, the policy network outputs continuous action values, and the square error $d(\pi ,\tilde{\pi })$ can be used directly to measure the difference between the output actions of two policies:

$$d(\pi ,\tilde{\pi })=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\mathrm{{\rm E}}}_s[{(\pi {(s)}_i-\tilde{\pi }{(s)}_i)}^2]}}$$

(23)

In the DDPG algorithm implementation, $N(0,{\sigma }^2)$ is used as the perturbation of the strategy, and the perturbation strategy distribution also obeys $\tilde{\pi }\sim \mathrm{\Gamma }(0,{\sigma }^2)$. Set the adaptive scaling factor to $\delta =\sigma$ and construct a single linear unit of the noise network:

$$y=w\cdot x+b$$

(24)

where $x\in {ℝ}^{p\ast 1}$, $y\in {ℝ}^{q\ast 1}$, $w\in {ℝ}^{p\ast q}$, $b\in {ℝ}^{q\ast 1}$. The idea of the noise network is to regard the parameter $w$ as a distribution rather than a series of values. The resampling method is used for $w\sim \mathrm{\Gamma }({\mu }^w,{\sigma }^w)$. First, a noise ${\zeta }^w\sim \mathrm{\Gamma }(0,I)$ is sampled, and the following is obtained:

$$w={\sigma }^w\odot {\zeta }^w+{\mu }^w$$

(25)

At this time, the parameter $w$ needs to learn the mean and variance. Compared with DDPG, the parameters that need to be learned become twice as many as the original. Similarly, the bias becomes $b={\sigma }^b\odot {\zeta }^b+{\mu }^b$, and the resulting linear unit becomes:

$$y=({\sigma }^w\odot {\zeta }^w+{\mu }^w)\cdot x+{\sigma }^b\odot {\zeta }^b+{\mu }^b$$

(26)

Here we take the standard DQN, $Q(s,a;w)$ as an example, and replace the Q network weight $w$ with the noisy weight parameters ${\sigma }^w$, ${\zeta }^w$, ${\mu }^w$, replace $b$ with ${\sigma }^b$, ${\zeta }^b$, ${\mu }^b$.Get the DQN with noisy network:

$${\tilde{y}}_i=r+\gamma \underset{a}{\max }\tilde{Q}(s_{t+1},a;{\sigma }^w,{\zeta }^w,{\mu }^w,{\sigma }^b,{\zeta }^b,{\mu }^b)$$

(27)

TD loss function $L({\sigma }^w,{\mu }^w,{\sigma }^b,{\mu }^b)$:

$$L({\sigma }^w,{\mu }^w,{\sigma }^b,{\mu }^b)={\displaystyle \frac{1}{2}}{[\tilde{Q}(s_t,a;{\sigma }^w,{\zeta }^w,{\mu }^w,{\sigma }^b,{\zeta }^b,{\mu }^b)-{\tilde{y}}_i]}^2$$

(28)

The training method of the noisy network is exactly the same as that of the standard neural network. Both use backpropagation to calculate the gradient and then use the gradient to update the neural parameters. The chain rule can be used to calculate the gradient of the loss function with respect to the parameters:

$${\displaystyle \frac{\partial L}{\partial {\sigma }^w}}={\displaystyle \frac{\partial L}{\partial y}}\cdot {\displaystyle \frac{\partial y}{\partial {\sigma }^w}} , {\displaystyle \frac{\partial L}{\partial {\mu }^w}}={\displaystyle \frac{\partial L}{\partial y}}\cdot {\displaystyle \frac{\partial y}{\partial {\mu }^w}} , {\displaystyle \frac{\partial L}{\partial {\sigma }^b}}={\displaystyle \frac{\partial L}{\partial y}}\cdot {\displaystyle \frac{\partial y}{\partial {\sigma }^b}} , {\displaystyle \frac{\partial L}{\partial {\mu }^b}}={\displaystyle \frac{\partial L}{\partial y}}\cdot {\displaystyle \frac{\partial y}{\partial {\mu }^b}}$$

(29)

$${\sigma }^w\leftarrow {\sigma }^w+\alpha {\displaystyle \frac{\partial L}{\partial {\sigma }^w}} , {\mu }^w\leftarrow {\mu }^w+\alpha {\displaystyle \frac{\partial L}{\partial {\mu }^w}} , {\sigma }^b\leftarrow {\sigma }^b+\alpha {\displaystyle \frac{\partial L}{\partial {\sigma }^b}} , {\mu }^b\leftarrow {\mu }^b+\alpha {\displaystyle \frac{\partial L}{\partial {\mu }^b}}$$

(30)

In order to further improve the robustness, strategy generalization ability and training stability of UUV’s autonomous collision avoidance in complex dynamic ocean environments, this paper proposes an improved noise network mechanism based on the introduction of noise network in the original DDPG structure, combined with a GRU module, which systematically improves the responsiveness of the collision avoidance strategy to dynamic changes in the environment.

(1)

Design of multi-level state disturbance guidance mechanism

Different from the traditional noise network method that introduces fixed distributed disturbances at the parameter layer, this study constructs a state disturbance guidance network, which uses the temporal dynamic change in the current observation state to adjust the noise intensity and realize adaptive exploration driven by environmental uncertainty perception. Construct the disturbance weight function ${\sigma }_i$:

$${\sigma }_i={\sigma }_0\cdot (1+\kappa \cdot {\mathrm{Var}}_t(s_{t-n,t}))$$

(31)

In the formula, ${\sigma }_0$ is the initial disturbance intensity; $\kappa$ is the adjustment coefficient; and ${\mathrm{Var}}_t(s_{t-n,t})$ is the variance of the state change in the $n$-step time window. The parameter noise amplitude in the Actor network is dynamically adjusted according to the disturbance weight to achieve the state perception ability of more aggressive strategy exploration when the environment changes more drastically.

(2)

Multi-scale parameter noise fusion structure

In order to avoid the training oscillation problem caused by fixed-scale noise, a multi-scale noise fusion module is designed to introduce independent noise of different scales into each actor and critic hidden layer to form an inter-layer noise regularized gradient flow. The hidden state is extracted by GRU and sent to the noise fusion network to improve the memory of historical decision trajectories and cope with dynamic obstacle strategy patterns. Gaussian noise $N(0,{\sigma }_i^2)$ of different scales is embedded in each layer and flexibly fused:

$$h_i^{noisy}={\alpha }_i\cdot (W_{i}x+b_i+{\epsilon }_i)+(1-{\alpha }_i)\cdot h_i$$

(32)

where ${\epsilon }_i\sim N(0,{\sigma }_i^2)$ represents the Gaussian noise introduced by the $i\mathrm{th}$ layer; ${\alpha }_i$ represents the learnable noise fusion coefficient of the $i\mathrm{th}$ layer; $W_i$ and $b_i$ represent the weight and bias of the $i\mathrm{th}$ layer; $h_i$ represents the original forward feature of the $i\mathrm{th}$ layer; and $h_i^{noisy}$ represents the output feature after fusion of the noise.

3.4. Design of Experience Pool Sample Sorting Method Based on Proportional Priority

In order to improve the utilization efficiency of training samples and the speed of strategy convergence, a prioritized experience replay (PPER) mechanism based on temporal difference error (TD error) is introduced. This mechanism quantifies the contribution of each experience to learning, uses TD error as a metric for experience priority, and prioritizes sampling of high TD error samples for training, thereby improving the use value and learning efficiency of samples.

However, although traditional PPER improves sample utilization, it also faces the following problems: ① Scanning the entire experience pool at a high frequency to update the priority increases the computational time; ② It is sensitive to random reward signals or function approximation errors, which can easily introduce training instability; ③ Excessive concentration on a few high TD error samples may cause the strategy to fall into local optimality or even overfitting, reducing the generalization ability of the model [20].

(1)

Introducing the Sampling Times Penalty

In order to solve the above problems, this paper introduces a sampling number penalty term, which is between pure greedy priority and uniform random sampling. The $i$th sampling probability $P(i)$ of the transition state is defined as:

$$P(i)={\displaystyle \frac{{(\left|p_i\right|+\epsilon )}^{\alpha }/{(n_i+1)}^{\beta }}{{\displaystyle {\sum }_k{(\left|p_k\right|+\epsilon )}^k/{(n_k+1)}^{\beta }}}}$$

(33)

Among them, $p(i)$ represents the priority of the $i\mathrm{th}$ experience, expressed as TD error; the exponent $\alpha$ determines the priority, $\alpha =0$ corresponds to uniform sampling; $|p(i)|+\epsilon$ is used to ensure that each experience has a certain priority, and $\epsilon$ is a very small number; $n_i$ represents the number of times the $i\mathrm{th}$ experience is sampled; $\beta$ controls the penalty intensity of the sampling frequency, which is often takes in the range of (0.4~0.6). This mechanism measures the importance of each experience based on its TD error and uses it as the basis for the sampling probability. The larger the TD error, the more valuable the experience is to the current strategy optimization, and therefore, the higher the probability of being sampled.

(2)

Stratified Prioritized Sampling

In the process of UUV collision avoidance planning model training, due to the large number of samples, the model is prone to overfitting risks. In addition, there are common problems in the UUV environment, such as sparse reward signals and drastic changes in the UUV bow angle. The traditional experience replay mechanism is difficult to effectively improve learning efficiency and maintain the generalization ability of the model.

To alleviate the above problems, the samples in the experience pool are divided into several priority intervals (divided into five levels according to percentiles) according to their TD error size, and random sampling is performed in each interval according to the set sampling weight. This strategy ensures that high-error samples are sampled first while avoiding excessive concentration of samples in intervals with large TD errors, thereby reducing the risk of the model falling into local optimality or overfitting, and enhancing sample diversity and overall training stability.

This paper proposes a DDPG collision avoidance planning algorithm based on sampling number penalty and stratified sampling strategy to improve priority experience replay, namely, proportional priority experience replay.

This form is more robust to outliers, and the Algorithm 1 pseudo-code is as follows:

Algorithm 1. DDPG Algorithm with TD Error Ratio Priority Experience Replay

Input: Minimum batch $k$, step length $\eta$, experience cycle $K$ and size $N$, importance index $\alpha$, $\beta$, total time $T$. Initialize experience pool $D$, $\Delta =0$, $p_1=1$ Select $a_0\sim {\pi }_w(s_0)$ according to the state quantity $s_0$ for $t=1 to T do$   Obtain the observation $s_t,r_{t+1},s_{t+1}$ from the environment and save the state transition tuple $(s_t,a_t,r_{t+1},s_{t+1})$ in the experience pool with the largest state priority $p_t={\max }_{i<t}{p}_i$.   if $t\equiv 0$ mod $K$ then     for $j=1$ to $k$ do     Sampling state transitions from the experience pool $j\sim P(j)=\frac{p_j^{\alpha }}{{\displaystyle {\sum }_i{p}_i^{\alpha }}}$     Calculating sampling importance weights ${\overline{w}}_j=\frac{{(N\cdot P(j))}^{-\beta }}{{\max }_i{w}_i}$.     Calculating TD Error ${\delta }_j=r_j+{\gamma }_j{Q}_{t\arg et}(s_j,\arg {\max }_{a}Q(s_j,a))-Q(s_{j-1},a_{j-1})$     Update state transition importance probability $p_j\leftarrow \left|{\delta }_j\right|$     Cumulative weight change $\Delta \leftarrow \Delta +{\overline{w}}_j\cdot {\delta }_j\cdot {\nabla }_{w}Q(s_{j-1},a_{j-1})$     end for     Update network weights $w\leftarrow w+\eta \cdot \Delta$, Reset $\Delta =0$     Copy the weights to the Q target network after a period of time $w_{t\arg et}\leftarrow w$     end if     Select action $a_t\sim {\pi }_w(s_t)$ end for

3.5. Simulation Verification and Analysis

This paper compares the path planning effects of DQN, Dueling DQN, DDPG, SAC, and the proposed algorithm in static obstacle environments. The evaluation indicators include average path length and planning stability. The DDPG algorithm training parameters are shown in Table 2.

Table 2. Hyperparameter settings for DDPG model training.

Parameters	Value
Minimum batch size	64
Replay experience pool size	100,000
Experience playback history length	10
Target network update frequency	8
Reward discount factor	0.98
Learning rate	0.0002
Learning rate discount factor	0.2
Exploration rate initial value	1
Minimum exploration rate	0.1
Optimizer	Adam
Activation Function	ReLu
Start update round	20
Actor NN structure	[160, 154]
Critic NN structure	[160, 154]
Network initialization	Xavier
Policy update delay	2
Mutual Entropy Regularization Coefficient	0.006

Figure 6 shows the environment obstacles constructed using a randomly generated method. Each collision avoidance planning algorithm was trained for 300 rounds. As shown in the figure, the UUV’s start point is located at [50 m, 45 m] and the target point is located at [520 m, 585 m]. The shortest distance between the two points is 715.9 m. The DDPG (no target information driven) algorithm uses a fully connected neural network. A comparison of Figure 6a,d shows that without proper target information driving the strategy, effective convergence is difficult. Once the target location or corresponding reward mechanism is clearly defined in DDPG, combined with a recurrent neural network to memorize historical states, navigation efficiency will be significantly improved.

Figure 6. Comparison of the impact of having and not having goal-oriented information on the performance of planning algorithms.

Table 3 shows that DDPG methods using either LSTM or GRU outperform the DDPG collision avoidance planning algorithm without target information in terms of training time, number of early rounds to reach the target, and number of path steps. DDPG-GRU performs best, achieving the best results in all four metrics: shortest training time, earliest target arrival, shortest path, and best success rate. This demonstrates that GRU is more effective in processing time series data for this task. DDPG without target information significantly lags behind in both convergence speed and path efficiency, demonstrating that incorporating target information and leveraging historical state memory are crucial for improving policy learning in complex environments.

Table 3. Training performance of four different DRL methods.

Methods	Training Time (h)	Earliest Arrival Round Number	Shortest Path Length (m)	Success Rate (%)
DDPG	9.6	118	1093.5	56.7%
Dueling DQN-LSTM	6.8	67	1036.2	70.7%
DDPG-LSTM	5.2	53	1042.8	78.3%
DDPG-GRU	4.4	34	850.2	88.7%

Figure 7 shows that after each algorithm successfully reaches the target for the first time, the reward value rises rapidly, indicating a significant improvement in policy performance. The cumulative returns of all algorithms gradually increase, reflecting the continuous exploration and optimization of the policy. The DDPG-target-free algorithm only incorporates obstacle distance information and lacks target information. In contrast, the algorithm with a memory-enhanced contextual value network and target information achieves higher cumulative returns, demonstrating that historical state and target information are crucial for the UUV collision avoidance model. Although Dueling DQN-LSTM reduces Q-value overestimation, its final performance is less stable than the DDPG-LSTM and DDPG-GRU algorithms, and its success rate is lower.

Figure 7. Reward function change curves of different algorithms.

The effect of the noise network on the algorithm is verified in the environment of Figure 8. The UUV start point (yellow circle) is located at [380 m, 20 m], and the target point (red circle) is located at [135 m, 575 m]. The straight-line distance between the two points is 606.7 m.

Figure 8. Trajectory comparison of adding noise network.

Table 4 shows that all collision avoidance algorithms can avoid dense static obstacles. The DDPG algorithm has the longest path, while the DDPG + Noise Network algorithm has the shortest path. The DDPG + Noise Network algorithm has the smallest cumulative heading adjustment angle, resulting in the smoothest path. Compared to the Dueling DQN, DQN + Noise Network, and DDPG algorithms, the cumulative adjustment angle is reduced by 63.0%, 63.2%, and 40.3%, respectively.

Table 4. Comparison of planning algorithm path length and cumulative turning angle.

Methods	Path Length (m)	$\mathbf{Total} \mathbf{Adjustment} \mathbf{Angle}$(°)
Dueling DQN	819.0	1681.4
DQN + NoisyNetwork	753.3	1688.1
DDPG	958.9	1041.8
DDPG + NoisyNetwork	721.8	621.5

Adding noise to the network weights to achieve parameter noise allows the UUV to produce richer behavioral performance. Adding regularized noise to each layer can increase the stability of training, but it may cause changes in the distribution of activation values of the previous and next layers. Although methods such as evolutionary strategies use parameter perturbations, they lose temporal structure information during training and require more learning samples. In this chapter, the same noise is added to each layer, but the noise between layers does not affect each other.

Figure 9 is a test of the performance of different DRL algorithms in a static environment. The positions of the starting point and the target point remain unchanged. The size and distribution range of each obstacle are randomly generated. Each algorithm is run 100 times (only one round is shown in the figure).

Figure 9. Run 100 times to test the success rate of different algorithms.

As can be seen from the data in Table 5, the traditional DQN based on a discrete action space and its improved Dueling DQN algorithm performed poorly in terms of average path length and standard deviation, reaching 1494.0 m, 45.5 m and 1451.0 m, 42.0 m, respectively, indicating that they have problems of large path volatility and insufficient strategy accuracy in complex dynamic environments. In comparison, the DDPG series of algorithms using continuous action space modeling has better overall performance. Among them, although the original DDPG algorithm is slightly inferior to some of the comparison algorithms (1452.7 m) in average path length, its standard deviation is only 24.1 m, showing strong path stability.

Table 5. Comparison of average path length and standard deviation of different algorithms.

Methods	Average Path Length (m)	Standard Deviation (m)
DQN	1494.0	45.5
Dueling DQN	1451.0	42.0
DDPG + NoisyNetwork	1382.3	37.9
DDPG	1452.7	24.1
SAC	1378.6	26.6
DDPG + PPER	1346.5	27.7
DDPG + NoisyNetwork + PPER	1330.4	21.3

In Figure 10, the DQN algorithm has the lowest success rate, and the SAC algorithm has a higher success rate than the DDPG algorithm that has only one improvement method; the success rate of the DDPG algorithm with a noisy network is 14%, 11%, and 2% higher than that of DQN, Dueling DQN, and the DDPG algorithm without a noisy network, respectively, indicating the effectiveness of the noisy network; the success rate of the DDPG algorithm with a noisy network and proportional priority experience replay is 25%, 21%, and 5% higher than that of DQN, Dueling DQN, and SAC algorithms, respectively, indicating that the introduction of a noisy network and proportional priority experience replay greatly improves the performance of the DDPG algorithm.

Figure 10. Success rates of different algorithms.

The complex simulation environment in Figure 11 contains seven irregular static obstacles of unknown islands and six dynamic obstacles. The UUV starting point (yellow circle) is located at [70 m, 990 m], the target point (red circle) is located at [800 m, 135 m], and the shortest distance between the two points is 1124.4 m. The speed of each dynamic obstacle is randomly generated between 1 m/s and 2 m/s, and its direction of movement is also randomly generated.

Figure 11. Trajectory diagram of the improved DDPG collision avoidance planning algorithm at different times in complex environments.

Figure 11 shows that the UUV adopted a safe collision avoidance strategy based on the DDPG + NoiseNetwork + PPER algorithm at three different times, avoiding the risk of collision between the UUV and different obstacles. At t = 460 s, the UUV is about to meet the dynamic obstacle 5 moving to the lower left. In order to avoid the collision, the UUV deflects to the left and moves in the same direction as the obstacle, and then the two move to the upper left together. When t = 600 s, after the obstacle avoidance action is completed, the UUV quickly adjusts its heading and moves towards the target point again. This shows that the proposed improved DDPG algorithm can perceive the movement of obstacles in real time in a complex dynamic environment and make smooth and coherent collision avoidance decisions.

In summary, the DDPG + NoisyNetwork + PPER model proposed in this paper has become the most advantageous path planning strategy at present with the comprehensive performance of shortest path length and smallest fluctuation. This method significantly improves stability while ensuring the efficiency of UUV path.

4. Comprehensive Simulation Verification of UUV Planning, Route Tracking and Autonomous Navigation

In order to achieve seamless connection from route planning to execution, UUV needs to have route tracking capability to ensure that it can accurately navigate along the planned route. In complex flow fields, UUVs cause dynamic sideslip angles due to factors such as hydrodynamic nonlinearity, asymmetric flow disturbances, control lag and incomplete observation. Its unpredictability and control limitations make it difficult to completely eliminate it, which seriously affects the path tracking accuracy and navigation stability. To address this problem, this paper constructs horizontal and vertical front sight vectors, and on this basis proposes a 3D adaptive line-of-sight (ALOS) guidance algorithm to ensure the feasibility of the 3D navigation planning results of UUVs. The simulation verifies the effectiveness of the UUV kinematic model in tracking the planned route under current disturbances, ensuring the feasibility of the UUV collision avoidance planning results.

4.1. Design of 3D ALOS Guidance Method Based on Horizontal and Vertical Front Sight Vectors

As shown in Figure 12, a third coordinate system {P} is introduced when implementing three-dimensional planning route tracking. The axis of the coordinate system {P} is parallel to the introduced path. In the figure, ${\theta }_h={\tan }^{-1}(y_e^p/{\Delta }_h)$ and ${\theta }_v={\tan }^{-1}(z_e^p/{\Delta }_v)$ are two waypoints $(x_i,y_i,z_i)$ and $(x_{i+1},y_{i+1},z_{i+1})$ output by the UUV collision avoidance planning in the NED coordinate system. The origin of the tangential coordinate system {P} of the current path is located at $(x_i,y_i,z_i)$, and its horizontal axis points to the next waypoint $(x_{i+1},y_{i+1},z_{i+1})$. ${\pi }_h$ is the angle of rotation of the UUV’s NED tracking error around the Z axis, and ${\pi }_v$ is the angle of rotation of the UUV around the Y axis.

Figure 12. UUV 3D ALOS guidance diagram.

According to the conversion relationship between the coordinate system in Section 2 and the {P} coordinate system, the longitudinal–lateral–vertical tracking error of the UUV in the {P} coordinate system is obtained:

$$\left[\begin{array}{c}{x}_e^p\\ y_e^p\\ z_e^p\end{array}\right]=R_{Y,{\pi }_v}^T{R}_{Z,{\pi }_h}^T\left(\left[\begin{array}{c}x\\ y\\ z\end{array}\right]-\left[\begin{array}{c}{x}_i\\ y_i\\ z_i\end{array}\right]\right)$$

(34)

Rotation matrix:

$$R_{Y,{\pi }_v}^T=\left[\begin{array}{ccc}\cos ({\pi }_v)& 0& \sin ({\pi }_v)\\ 0& 1& 0\\ -\sin ({\pi }_v)& 0& \cos ({\pi }_v)\end{array}\right]$$

(35)

$$R_{Z,{\pi }_h}^T=\left[\begin{array}{ccc}\cos ({\pi }_h)& -\sin ({\pi }_h)& 0\\ \sin ({\pi }_h)& \cos ({\pi }_h)& 0\\ 0& 0& 1\end{array}\right]$$

(36)

where

$${\pi }_h=\mathrm{a}\tan 2(y_{i+1}-y_i,x_{i+1}-x_i)$$

(37)

$${\pi }_v=\mathrm{a}\tan 2(-(z_{i+1}-z_n),\sqrt{{(x_{i+1}-x_i)}^2+{(y_{i+1}-y_i)}^2})$$

(38)

For a surge velocity of $0<u_{\min }<u<u_{\max }$, the differential equation of motion for the rate of change in position of the UUV in the NED coordinates is expressed as:

$$\begin{array}{l}\dot{x}=U_h\cos (\Psi +{\beta }_c)\\ \dot{y}=U_h\sin (\Psi +{\beta }_c)\\ \dot{z}=-U_v\cos (\theta -{\alpha }_c)\end{array}$$

(39)

where ${\alpha }_c={\tan }^{-1}({\displaystyle \frac{v\sin (\varphi )+w\cos (\varphi )}{u}})$, ${\beta }_c={\tan }^{-1}({\displaystyle \frac{v\cos (\varphi )-w\sin (\varphi )}{U_v\cos (\theta -{\alpha }_c)}})$, $U_h$ and $U_v$ are the velocity components in the horizontal and vertical directions, respectively, and are given by:

$$U_v=u\sqrt{1+{\tan }^2({\alpha }_c)}$$

(40)

$$U_h=U_v\cos (\theta -{\alpha }_c)\sqrt{1+{\tan }^2({\beta }_c)}$$

(41)

Desired heading and pitch angles:

$${\Psi }_d={\pi }_h-{\widehat{\beta }}_c-{\tan }^{-1}(y_e^p/{\Delta }_h)$$

(42)

$${\theta }_d={\pi }_c+{\widehat{\alpha }}_c+{\tan }^{-1}(z_e^p/{\Delta }_v)$$

(43)

Under the influence of ocean current disturbance, the UUV’s longitudinal velocity, sway velocity, and bow angle will change due to external environmental interference, and the phase angles ${\alpha }_c$ and ${\beta }_c$ will also change accordingly, which will cause the horizontal front sight component $U_h$ and the vertical front sight component $U_v$ determined by Formulas (40) and (41) to also change. In order to simplify the complexity of tracking, the concept of relative velocity is used to simulate ocean current disturbance. In the process of straight-line path tracking, the following assumptions are made:

Assumption 1.

During the path following process, ${\alpha }_c$and${\beta }_c$are fixed values, that is${\dot{\alpha }}_c=0$and${\dot{\beta }}_c=0$.

The assumption is satisfied when the UUV follows a straight path at a constant speed and is considered to meet the assumption even if there is a slight speed change under time-varying environmental disturbances.

The tracking error differential equations along three directions are:

$$\left[\begin{array}{c}{\dot{x}}_e^p\\ {\dot{y}}_e^p\\ {\dot{z}}_e^p\end{array}\right]=R_{Y,{\pi }_v}^T{R}_{Z,{\pi }_h}^T\left[\begin{array}{c}{U}_h\cos (\Psi +{\beta }_c)\\ U_h\sin (\Psi +{\beta }_c)\\ -U_v\cos (\theta -{\alpha }_c)\end{array}\right]$$

(44)

Expanding the last two lines of the above equation corresponds to the lateral and vertical tracking errors:

$${\dot{y}}_e^p=U_h\sin (\Psi +{\beta }_c-{\pi }_h)$$

(45)

$${\dot{z}}_e^p=U_h\sin ({\pi }_v)\cos (\Psi +{\beta }_c-{\pi }_h)-U_v\cos ({\pi }_v)\sin (\theta -{\alpha }_c)$$

(46)

Rewrite the vertical path differential Formula (46) into a function form of P to realize the design of the depth controller. Substitute Formula (41) into Formula (46) to obtain:

$${\dot{z}}_e^p=U_v\cos (\theta -{\alpha }_c)\sqrt{1+{\tan }^2({\beta }_c)}\sin ({\pi }_v)\cos (\Psi +{\beta }_c-{\pi }_h)-U_v\cos ({\pi }_v)\sin (\theta -{\alpha }_c)$$

(47)

Using the trigonometric transformation equation of the sum and difference angles, we can obtain:

$$\begin{array}{ll}{\dot{z}}_e^p& =U_v\sin ({\pi }_v)\cos (\theta -{\alpha }_c)\sqrt{1+{\tan }^2({\beta }_c)}\cos (\Psi +{\beta }_c-{\pi }_h)\\ & -U_v\sin ({\pi }_h)\cos (\theta -{\alpha }_c)-U_v\sin (\theta -{\alpha }_c-{\pi }_v)\\ & = -U_v\sin (\theta -{\alpha }_c-{\pi }_v)+U_v\sin ({\pi }_v)\cos (\theta -{\alpha }_c)[\sqrt{1+{\tan }^2({\beta }_c)}\cos (\Psi +{\beta }_c-{\pi }_h)+1]\end{array}$$

(48)

Then,

$${\dot{z}}_e^p=-U_v\sin (\theta -{\alpha }_c-{\pi }_v)+{\displaystyle \frac{U_h\sin ({\pi }_v)}{\sqrt{1+{\tan }^2({\beta }_c)}}}[\sqrt{1+{\tan }^2({\beta }_c)}\cos (\Psi +{\beta }_c-{\pi }_h)+1]$$

(49)

Guidance rate ALOS for 3D path following:

$${\Psi }_d={\pi }_h-{\widehat{\beta }}_c-{\tan }^{-1}({\displaystyle \frac{y_e^p}{{\Delta }_h}})$$

(50)

$${\dot{\widehat{\beta }}}_c={\gamma }_h{\displaystyle \frac{{\Delta }_h}{\sqrt{{\Delta }_h^2+{(y_e^p)}^2}}}{y}_e^p$$

(51)

$${\theta }_d={\pi }_v+{\widehat{\alpha }}_c+{\tan }^{-1}({\displaystyle \frac{z_e^p}{{\Delta }_v}})$$

(52)

$${\dot{\widehat{\alpha }}}_c={\gamma }_v{\displaystyle \frac{{\Delta }_v}{\sqrt{{\Delta }_v^2+{(z_e^p)}^2}}}{z}_e^p$$

(53)

For Formulas (45) and (48), the motion of the horizontal and vertical planes is decoupled. ${\Delta }_h>0$ and ${\Delta }_v>0$ are the self-specified forward distances, and ${\gamma }_h>0$, ${\gamma }_v>0$ are the adaptive gain. ${\widehat{\alpha }}_c$, ${\widehat{\beta }}_c$ is the estimated value of ${\alpha }_c$, ${\beta }_c$, respectively. The stability analysis assumes that the UUV can obtain perfect path tracking $\Psi ={\Psi }_d$ and $\theta ={\theta }_d$ by turning the heading and the deep autopilot. The parameter mapping method is used to enhance the robustness of the ALOS guidance rate:

$${\dot{\widehat{\alpha }}}_c={\gamma }_v{\displaystyle \frac{{\Delta }_v}{\sqrt{{\Delta }_v^2+{(z_e^p)}^2}}}\mathrm{Pr}({\widehat{\alpha }}_c,z_e^p)$$

(54)

$${\dot{\widehat{\alpha }}}_c={\gamma }_v{\displaystyle \frac{{\Delta }_v}{\sqrt{{\Delta }_v^2+{(z_e^p)}^2}}}\mathrm{Pr}({\widehat{\alpha }}_c,z_e^p)$$

(55)

The parameter estimates are compact sets $\left|{\widehat{\alpha }}_c\right|\le M_{\widehat{\theta }}$, $\left|{\widehat{\beta }}_c\right|\le M_{\widehat{\theta }}$, $M_{\widehat{\theta }}=M_{\theta }+\epsilon$, $\epsilon$ is a very small positive number, making $M_{\widehat{\theta }}>M_{\theta }$ true,

$$\mathrm{Pr}(\widehat{\theta },\tau )=\left\{\begin{array}{l}(1-c(\widehat{\theta }))\tau \hspace{0.25em}\hspace{0.25em}\mathrm{if}\hspace{0.25em}\hspace{0.25em}\left|\widehat{\theta }\right|>M_{\theta },{\widehat{\theta }}^T\tau >0\\ \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\tau \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\mathrm{else}\end{array}\right.$$

(56)

where $c(\widehat{\theta })=\min \{1,({\widehat{\theta }}^2-M_{\theta }^2)/(M_{\widehat{\theta }}^2-M_{\theta }^2)\}$ is a special form of parameter mapping, which is modified to ensure semi-global stability.

$${\dot{y}}_e^p=U_h\sin ({\tilde{\beta }}_c-{\tan }^{-1}({\displaystyle \frac{y_e^p}{{\Delta }_h}}))$$

(57)

$${\dot{z}}_e^p=U_v\sin ({\tilde{\alpha }}_c-{\tan }^{-1}({\displaystyle \frac{z_e^p}{{\Delta }_v}}))+g(t,y_e^p,{\tilde{\beta }}_c)$$

(58)

where ${\tilde{\alpha }}_c={\alpha }_c-{\widehat{\alpha }}_c$, ${\tilde{\beta }}_c={\beta }_c-{\widehat{\beta }}_c$.

$$g(t,y_e^p,{\tilde{\beta }}_c)={\displaystyle \frac{U_h\sin ({\pi }_v)}{\sqrt{1+{\tan }^2({\beta }_c)}}}[\sqrt{1+{\tan }^2({\beta }_c)}\cos ({\tilde{\beta }}_c-{\tan }^{-1}({\displaystyle \frac{y_e^p}{{\Delta }_h}}))-1]$$

(59)

Using the trigonometric transformation equations of sum and difference angles:

$${\dot{y}}_e^p=-U_h\sin ({\tan }^{-1}({\displaystyle \frac{y_e^p}{{\Delta }_h}}))\cos ({\tilde{\beta }}_c)+U_h\cos ({\tan }^{-1}({\displaystyle \frac{y_e^p}{{\Delta }_h}}))\sin ({\tilde{\beta }}_c)$$

(60)

$${\dot{z}}_e^p=-U_v\sin ({\tan }^{-1}({\displaystyle \frac{z_e^p}{{\Delta }_v}}))\cos ({\tilde{\alpha }}_c)+U_v\cos ({\tan }^{-1}({\displaystyle \frac{z_e^p}{{\Delta }_v}}))\sin ({\tilde{\alpha }}_c)+g(t,y_e^p,{\tilde{\beta }}_c)$$

(61)

Using $\sin ({\tan }^{-1}(x/d))=x/\sqrt{d^2+x^2}$ and $\cos ({\tan }^{-1}(x/d))=d/\sqrt{d^2+x^2}$, the above equation is transformed into a nonlinear cascade system:

$${\displaystyle {\sum }_1:\left\{\begin{array}{c}{\dot{z}}_e^p=-\frac{U_v{\Delta }_v}{\sqrt{{({\Delta }_v)}^2+{(z_e^p)}^2}}(\cos ({\tilde{\alpha }}_c)\frac{z_e^p}{{\Delta }_v}-\sin ({\tilde{\alpha }}_c))+g(t,y_e^p,{\tilde{\beta }}_c)\\ {\dot{\tilde{\alpha }}}_c=-{\gamma }_v\frac{{\Delta }_v}{\sqrt{{({\Delta }_v)}^2+{(z_e^p)}^2}}\mathrm{Pr}({\widehat{\alpha }}_c,z_e^p)\end{array}\right.}$$

(62)

$${\displaystyle {\sum }_2:\left\{\begin{array}{c}{\dot{y}}_e^p=-\frac{U_h{\Delta }_h}{\sqrt{{({\Delta }_h)}^2+{(y_e^p)}^2}}(\cos ({\tilde{\beta }}_c)\frac{y_e^p}{{\Delta }_h}-\sin ({\tilde{\beta }}_c))\\ {\dot{\tilde{\beta }}}_c=-{\gamma }_h\frac{{\Delta }_h}{\sqrt{{({\Delta }_h)}^2+{(y_e^p)}^2}}\mathrm{Pr}({\widehat{\beta }}_c,y_e^p)\end{array}\right.}$$

(63)

From Assumption 1, we can see that ${\dot{\tilde{\alpha }}}_c=-{\dot{\widehat{\alpha }}}_c$, ${\dot{\tilde{\beta }}}_c=-{\dot{\widehat{\beta }}}_c$, $U_h$, $U_v$, $g(t,y_e^p,{\tilde{\beta }}_c)$ are time-varying, and the ${\Sigma }_1$ and ${\Sigma }_2$ subsystems are non-autonomous systems. In some cases, it is advantageous to make the look-ahead distances ${\Delta }_h$, ${\Delta }_v$ also time-varying. Assumption 1 is valid, and the stability of the cascade c and v is guaranteed by Lemmas 1, 2, and Theorem 1.

Lemma 1.

If the perturbation $g(t,y_e^p,{\tilde{\beta }}_c)\equiv 0$is of initial value $(z_e^p,{\tilde{\alpha }}_c)=(0,0)$, then the cascade system ${\Sigma }_1$is uniformly semiglobal exponentially stable.

Lemma 2.

If the initial value $(y_e^p,{\tilde{\beta }}_c)=(0,0)$, then the cascade system ${\Sigma }_2$is uniform semiglobal exponential stable (USGES).

Theorem 1.

Consider a three-dimensional path-following system for a UUV guided by the adaptive line-of-sight (ALOS) method. The system dynamics are decoupled into a horizontal guidance subsystem and a vertical guidance subsystem as follows:

$${\dot{\tilde{y}}}_e=-{\gamma }_h{\tilde{y}}_e+g_1(t)$$

(64)

$${\dot{\tilde{z}}}_e=-{\gamma }_v{\tilde{z}}_e+h({\tilde{y}}_e)+g_2(t)$$

(65)

where ${\gamma }_h,{\gamma }_v>0$ are control gains; $g_1(t)$, $g_2(t)$ are bounded external disturbances satisfying $\left|g_i(t)\right|\le {\overline{g}}_i,i=1,2$; $h({\tilde{y}}_e)$ is a continuous function that satisfies the Lipschitz condition: $\left|h({\tilde{y}}_e)\right|\le L\left|{\tilde{y}}_e\right|$ for some $L>0$.

Then, under suitable selection of ${\gamma }_h$ and ${\gamma }_v$, the origin of the cascaded system ${\Sigma }_1-{\Sigma }_2$ is Uniformly Semiglobally Exponentially Stable (USGES) with respect to bounded disturbances.

Proof. 

(1)

Lyapunov Stability of the Horizontal Subsystem

Define the Lyapunov function candidate:

$$V_1({\tilde{y}}_e)={\displaystyle \frac{1}{2}}{\tilde{y}}_e^2$$

(66)

Taking the derivative along the trajectory of the system yields:

$${\dot{V}}_1={\tilde{y}}_e{\dot{\tilde{y}}}_e=-{\gamma }_h{\tilde{y}}_e^2+{\tilde{y}}_e{g}_1(t)\le -{\gamma }_h{\tilde{y}}_e^2+\left|{\tilde{y}}_e\right|{\overline{g}}_1$$

(67)

By Young’s inequality:

$${\dot{V}}_1\le -{\gamma }_h{\tilde{y}}_e^2+{\displaystyle \frac{{\tilde{y}}_e^2}{2{\gamma }_h}}+{\displaystyle \frac{{\gamma }_h{\overline{g}}_1^2}{2}}=-({\gamma }_h-{\displaystyle \frac{1}{2{\gamma }_h}}){\tilde{y}}_e^2+{\displaystyle \frac{{\gamma }_h{\overline{g}}_1^2}{2}}$$

(68)

Therefore, the subsystem is input-to-state stable (ISS) and exponentially stable when ${\gamma }_h>{\displaystyle \frac{1}{\sqrt{2}}}$.

(2)

Lyapunov Stability of the Vertical Subsystem

$$V_2({\tilde{z}}_e)={\displaystyle \frac{1}{2}}{z}_e^2$$

(69)

$${\dot{V}}_2={\tilde{z}}_e{\dot{\tilde{z}}}_e=-{\gamma }_v{\tilde{z}}_e^2+{\tilde{z}}_{e}h({\tilde{y}}_e)+{\tilde{z}}_e{g}_2(t)$$

(70)

Using the Lipschitz condition and Young’s inequality:

$$\left|{\tilde{z}}_{e}h({\tilde{y}}_e)\right|\le L\left|{\tilde{y}}_e\right|\left|{\tilde{z}}_e\right|\le {\displaystyle \frac{L}{2\epsilon }}{\tilde{y}}_e^2+{\displaystyle \frac{L\epsilon }{2}}{\tilde{z}}_e^2$$

(71)

Similarly, ${\tilde{y}}_2{d}_2(t)\le {\displaystyle \frac{1}{2\delta }}{\overline{g}}_2^2+{\displaystyle \frac{\delta }{2}}{\tilde{y}}_2^2$, $\epsilon =\delta ={\displaystyle \frac{{\gamma }_v}{2L+1}}$, thus:

$${\dot{V}}_2\le -({\gamma }_v-{\displaystyle \frac{L\epsilon +\delta }{2}}){\tilde{z}}_e^2+{\displaystyle \frac{L}{2\epsilon }}{\tilde{y}}_e^2+{\displaystyle \frac{1}{2\delta }}{\overline{g}}_2^2$$

(72)

Let $\epsilon$, $\delta$ be chosen such that:

$${\gamma }_v>{\displaystyle \frac{L\epsilon +\delta }{2}}\Rightarrow \exists {\alpha }_2>0$$

(73)

Then:

$${\dot{V}}_2\le -{\alpha }_2{\tilde{z}}_e^2+\beta {\tilde{y}}_e^2+C$$

(74)

with $\beta ={\displaystyle \frac{L}{2\epsilon }}$, $C={\displaystyle \frac{1}{2\delta }}{\overline{g}}_2^2$.

(3)

Composite Lyapunov Function

Define the total Lyapunov function:

$$V=V_1+V_2={\displaystyle \frac{1}{2}}{\tilde{y}}_e^2+{\displaystyle \frac{1}{2}}{\tilde{z}}_e^2$$

(75)

$$\dot{V}\le -({\alpha }_1-\beta ){\tilde{y}}_e^2-{\alpha }_2{\tilde{z}}_e^2+{\overline{g}}^2+C$$

(76)

If the gains ${\gamma }_h$, ${\gamma }_v$ are selected so that ${\alpha }_1>\beta$, the system satisfies “limited input disturbance and exponential convergence of state”, that is, it satisfies the USGES condition. □

4.2. UUV Autonomous Navigation Planning and Route Tracking Capability Simulation Verification

The simulation environment shown in Figure 13 simulates the scenario of a UUV searching for targets in narrow waters under variable depth conditions. The UUV starts from the yellow square starting point and heads for the yellow pentagonal target point. During navigation, the UUV will encounter multiple prohibited areas, including light-colored spherical static obstacles and dark-colored dynamic obstacles (such as minefields, enemy moving UUVs, sonar detection areas, etc.), as shown in the figure. These obstacles pose a threat to the safe navigation of the UUV. The simulation parameter settings for variable depth environments are shown in Table 6. The experiment compares the autonomous collision avoidance performance of PSO combined with five algorithms, DWA, DQN, Dueling DQN, SAC, and DDPG, in a dynamic environment to evaluate the effectiveness and stability of these algorithms in dealing with environmental uncertainty.

Figure 13. UUV collision avoidance planning trajectory diagram of different planning algorithms under variable depth conditions.

Table 6. Setting of simulation parameters for variable depth environment.

Parameter		Value
UUV	Starting point location	[500 m, 80 m, 540 m]
	Target point location	[150 m, 720 m, 500 m]
	Initial state $\begin{array}{l}(u_0,v_0,w_0), \\ (p_0,q_0,r_0), \\ (x_0,y_0,z_0)\end{array}$	$[0,0,0]$
	Initial Euler angles $(\varphi ,\theta ,\Psi )$	$[0,0,\frac{\pi }{2}]$
Dynamic obstacle initial position coordinates	1	[580 m, 300 m, 410 m]
	2	[180 m, 450 m, 330 m]
	3	[450 m, 600 m, 330 m]
	4	[410 m, 790 m, 400 m]
	5	[210 m, 780 m, 390 m]
Sonar	Detection false alarm rate	2%
	Missed detection rate	2%
	Range observation noise standard deviation	2 m

In the planning results of different algorithms shown in Figure 14, the vertical axis marks 1, 2, 3, 4, and 5 correspond to the initial distances of the five dynamic obstacles relative to the UUV in Figure 10. The performance of each algorithm in the figure can be evaluated by the change in the distance between the UUV and the dynamic obstacle over time. The longer the distance, the better the obstacle avoidance effect. The figure shows that the distance between the algorithm proposed in this paper and the dynamic obstacle is greater than 20 m, indicating that the UUV successfully avoided collisions with dynamic obstacles in all simulations of the autonomous collision avoidance planning algorithm.

Figure 14. Distance variation curves between UUV and dynamic obstacles under different planning algorithms under variable depth conditions.

Table 7 summarizes the path length, planning time, and minimum distance to dynamic obstacles of different algorithms. The results show that the DDPG algorithm generates the shortest path, while the DQN algorithm has the shortest average single-step planning time, that is, the output speed of the control command is the fastest. In contrast, the planning time of PSO combined with DWA is the longest, mainly because in a three-dimensional dynamic environment, the improved PSO requires multiple iterations to optimize the optimal global route, and the DWA algorithm with curvature constraints is also time-consuming each time it predicts the trajectory. In addition, the DWA algorithm with curvature constraints also consumes a lot of time to calculate the predicted trajectory each time. PSO combined with DWA, DQN, and DDPG are similar in the minimum distance to a certain dynamic obstacle. They can all effectively avoid obstacles and ensure the safe navigation of UUVs in complex environments. Relatively speaking, although the path lengths of Dueling DQN and SAC are greater than those of DDPG, their minimum collision avoidance distances with dynamic obstacles are larger, which also reflects a certain degree of safety redundancy.

Table 7. Path length, planning time, and minimum collision avoidance distance of different planning algorithms under variable depth conditions.

Methods	Route Length (m)	Average Single-Step Planning Time (ms)	Minimum Collision Avoidance Distance (m)
PSO + DWA	1035.7	3549.4	28.2
DQN	1027.9	36.5	29.5
Dueling DQN	1065.6	39.3	72.6
SAC	1084.4	36.7	71.8
DDPG	1018.7	37.9	28.4

In order to verify whether the UUV waypoints generated by the planning algorithm are executable, the UUV waypoints in Figure 15 are input into the 3-DOF model of the UUV to observe the change curves of each state quantity. When $t<200 \mathrm{s}$, the ocean current disturbance is $U_c=0.2 \mathrm{m}/\mathrm{s}$, ${\Psi }_c={120}^{°}$; when $100 \mathrm{s}\le \mathrm{t}<300 \mathrm{s}$, $U_c=0.2 \mathrm{m}/\mathrm{s}$; ${\Psi }_c$ gradually increases from ${120}^{°}$ to ${140}^{°}$ and then ${\Psi }_c$ fluctuates in a small range.; when $\mathrm{t}>300 \mathrm{s}$, $U_c$ gradually increases from $0.2 \mathrm{m}/\mathrm{s}$ to $0.65 \mathrm{m}/\mathrm{s}$, and then fluctuates around $0.65 \mathrm{m}/\mathrm{s}$.

Figure 15. Ocean current speed and direction change curve.

Combining Figure 13, Figure 14 and Figure 16, we can see that the trajectory of each algorithm runs relatively smoothly in the initial stage; when encountering dynamic obstacles 3 and 4, the DWA algorithm, DQN algorithm, and Dueling algorithm made a strategy to make the track bend more, especially the waypoints planned by the DWA algorithm and DQN algorithm, the UUV route tracking control system cannot complete effective tracking. However, the waypoints of the Dueling DQN algorithm can be realized by the UUV route tracking control system. The trajectories of the SAC algorithm and the DDPG algorithm are relatively stable, the overall tracking effect is good, and they show strong dynamic adaptability.

Figure 16. The waypoints of UUV path tracking with different planning algorithms under ocean current disturbance.

Figure 17 shows that in the presence of ocean current interference, the proposed three-dimensional ALOS guidance algorithm has relatively stable speed control in the longitudinal direction, indicating good control capability of the UUV longitudinal speed; all attitude quantities remain stable before 300 s, and the roll angular velocity and pitch angular velocity are close to 0 between 100 s and 300 s; between 300 s and 500 s, the collision avoidance behavior of each collision avoidance planning algorithm against dynamic obstacles causes fluctuations in attitude quantities. Combined with Figure 16, it can be seen that the paths of SAC and the proposed DDPG algorithm are smoother, and the collision avoidance strategy is better. It can achieve stable and continuous heading adjustment, which is conducive to improving the overall tracking quality.

Figure 17. UUV motion state change curves of different planning algorithms under variable depth conditions.

Figure 18 shows the changes in the angle of attack $\alpha$ and sideslip angle $\beta$ during the navigation process of the UUV under different planning algorithms. The angle of attack of each planning algorithm is positive at the initial moment, indicating that the UUV is in the climbing stage. This is consistent with the path trajectory of Figure 16, indicating that the pitch attitude control effect is good. The sideslip angle of the UUV at the initial moment is due to the large angle between the bow direction of the UUV and the direction of the ocean current at the beginning. Under the joint action of the three-dimensional ALOS algorithm and the controller, the sideslip angle decreases rapidly. The sideslip angle of each algorithm is controlled within ±10°. The results show that the three-dimensional ALOS track tracking method combined with the collision avoidance strategy can effectively improve the attitude stability of the UUV in a time-varying ocean current environment.

Figure 18. UUV attack angle and sideslip angle curves of different planning algorithms under variable depth conditions.

Figure 19 shows the pitch angle and bow angle tracking errors of each algorithm. In the variable depth collision avoidance planning and three-dimensional route tracking tasks, the DDPG algorithm performs best in pitch angle, bow angle, and position error indicators, showing good route tracking accuracy. The SAC algorithm also has high attitude and trajectory control capabilities and is suitable for autonomous navigation in dynamic and complex environments. DQN and Dueling DQN have certain deficiencies in attitude control accuracy and trajectory smoothness. As shown in Figure 16, the traditional improved PSO + DWA and DQN algorithms cannot reach the target point safely, and it is difficult to complete obstacle avoidance planning in highly dynamic scenes.

Figure 19. UUV pitching angle and heading angle tracking error curves under different algorithms.

Table 8 shows the comparison of the RMS of pitch angle tracking and bow angle tracking of different algorithms in a variable depth environment. The SAC algorithm has the smallest RMSE for pitch angle tracking; the DDPG algorithm has the smallest RMSE for bow angle tracking, which is 12.8%, 17.1%, 21.7%, and 9.4% lower than the DWA, DQN, Dueling DQN, and SAC algorithms, respectively; the DDPG algorithm has the smallest RMSE for position tracking, which is 23.8%, 15.04%, 14.2%, and 11.1% lower than the DWA, DQN, Dueling DQN, and SAC algorithms, respectively. Overall, DDPG shows comprehensive advantages in robustness, accuracy, and track smoothness, verifying its superiority in complex 3D dynamic environments.

Table 8. Comparison of tracking root mean square error of different algorithms in variable depth environment.

Methods	$\mathbf{RMSE} \mathbf{of} \mathbf{Pitch} \mathbf{Angle} \mathbf{Tracking}$(°)	$\mathbf{RMSE} \mathbf{of} \mathbf{Heading} \mathbf{Angle} \mathbf{Tracking}$ (°)	Position Tracking RMSE (m)
Improved PSO + DWA	8.8	28.9	12.6
DQN	7.3	30.4	11.3
Dueling DQN	3.6	32.2	11.2
SAC	2.4	27.8	10.8
DDPG	2.8	25.2	9.6

In the UUV actuator change curves of Figure 20, different planning algorithms, DDPG and SAC, have the smallest fluctuations in the changes in horizontal rudder angle and tail rudder angle, and the control is smoother; the thruster speed of each method can be maintained stably, ensuring the UUV speed stability and track tracking continuity.

Figure 20. Change curve of UUV actuator under different planning algorithms under variable depth conditions.

From the above simulation results, it can be seen that the improved DDPG algorithm in this paper shows better tracking performance, especially in the bow angle tracking error and position tracking error, which are better than other DRL methods. It is suitable for UUV autonomous navigation tasks with high requirements for route tracking accuracy and stability. The change curve of the actuator shows that the control is smoother, and the algorithm also meets strong adaptability. In addition, compared with the improved DWA algorithm, the collision avoidance planning time of the DDPG algorithm, SAC algorithm, and improved DWA algorithm is shortened by more than 95%, which proves that the DRL algorithm has strong real-time performance when doing collision avoidance planning. The distance between the algorithm proposed in this paper and multiple dynamic obstacles is greater than 20 m, indicating that the UUV successfully avoids collision with dynamic obstacles under the planning of all autonomous collision avoidance algorithms.

5. Conclusions

This paper proposes a hierarchical autonomous navigation framework for unmanned underwater vehicles (UUVs), combining global route planning based on an Improved particle swarm optimization (PSO) algorithm, local collision avoidance using a deep deterministic policy gradient (DDPG) method enhanced by noisy networks and proportional priority experience replay (PPER), and robust three-dimensional path tracking using an adaptive line-of-sight (ALOS) guidance strategy.

In summary, the proposed method demonstrates strong capability for efficient and safe UUV navigation in scenarios involving sparse rewards, limited perception, and nonlinear motion coupling. The framework effectively bridges global planning, local avoidance, and route tracking into a coherent autonomous navigation strategy.

Future research work on this topic includes:

When facing multiple dynamic obstacles, the UUV’s collision avoidance planning ability is not as good as the DWA algorithm with curvature constraints. In the future, it is planned to add the ability to perceive dynamic obstacles to the model, use the extended Kalman filter (EKF) for online filtering and trajectory prediction of dynamic obstacles, integrate the EKF output into the state input of DRL, and introduce the reward function of dynamic obstacles. Use the LSTM/GRU network to help the UUV remember the movement trend of obstacles in the past few steps to improve the collision avoidance prediction ability.