Research on Obstacle Avoidance Planning for UUV Based on A3C Algorithm

Abstract

Deep reinforcement learning is an artificial intelligence technology that combines deep learning and reinforcement learning and has been widely applied in multiple fields. As a type of deep reinforcement learning algorithm, the A3C (Asynchronous Advantage Actor-Critic) algorithm can effectively utilize computer resources and improve training efficiency by synchronously training Actor-Critic in multiple threads. Inspired by the excellent performance of the A3C algorithm, this paper uses the A3C algorithm to solve the UUV (Unmanned Underwater Vehicle) collision avoidance planning problem in unknown environments. This collision avoidance planning algorithm can have the ability to plan in real-time while ensuring a shorter path length, and the output action space can meet the kinematic constraints of UUVs. In response to the problem of UUV collision avoidance planning, this paper designs the state space, action space, and reward function. The simulation results show that the A3C collision avoidance planning algorithm can guide a UUV to avoid obstacles and reach the preset target point. The path planned by this algorithm meets the heading constraints of the UUV, and the planning time is short, which can meet the requirements of real-time planning.

1. Introduction

The UUV has strong civilian and military value due to its autonomous navigation ability, which can obtain information on ocean and seabed environments and detect and identify moving targets [1]. The capability of obstacle avoidance planning is an important influencing factor in determining whether UUVs can independently complete preset tasks [2]. Autonomous obstacle avoidance in UUVs refers to the UUV detecting surrounding obstacles and collecting their status information, and planning its paths according to a certain algorithm to avoid obstacles and ultimately reach the preset target point [3]. Deep reinforcement learning algorithms have good decision-making ability and can learn independently, which is used to solve continuous decision-making problems. So Deep reinforcement learning algorithms are very suitable for solving obstacle avoidance planning problems [4].

Deep reinforcement learning is one of the most focused directions in the field of artificial intelligence in recent years, which combines the perceptual ability of deep learning with the decision-making ability of reinforcement learning. It can directly control the behavior of agents through high-dimensional perceptual input learning, providing ideas for solving perceptual decision-making problems in complex systems [5]. In recent years, research on deep reinforcement learning algorithms has mainly focused on the DQN (Deep Q-network) algorithm and its related improvements. Mnih et al., from DeepMind, propose the DQN algorithm [6], through which the agent can learn to play video games by simply obtaining raw pixels from images [7]. Mnih introduces a separate Q-function network and introduces iterative updates into DQN to reduce the correlation between the target value and the current value, proposing an improved version of DQN, namely Nature DQN [8]. Wang et al. propose the Dueling DQN model, which separates state values and action dominance values, enabling better integration of network architecture and RL algorithms [9]. Hasselt proposes using adaptive normalization learning to address the issue of not changing the scale of the approximation function during the learning process in DQN, which affects the quality of the algorithm in different game applications by cutting the feedback to a predetermined range [10]. Lillicrap proposes the DDPG (Deep Deterministic Policy Gradient) algorithm, which is a reinforcement learning algorithm of the Actor-Critic framework that combines the policy gradient algorithm and DQN algorithm, and can update the policy network in a single step through an off-policy method to achieve the maximum total reward [11]. Mnih et al. propose the A3C algorithm, which introduces the technique of distributed parallel training based on the classic Actor-Critic framework [12]. The parallel threads of the A3C algorithm can explore different actions and effectively improve learning efficiency.

There are many types of traditional collision avoidance planning algorithms, such as the Dynamic Window method [13], Rapidly-Exploring Random Tree [14], and the artificial potential field method [15], but these lack the ability to learn and have poor adaptability to the environment. Heuristic search algorithms commonly used to solve collision avoidance planning problems include the genetic algorithm [16], Particle Swarm Optimization Algorithm [17], and Ant Colony Algorithm [18]. It is difficult to achieve real-time planning for collision avoidance planning based on heuristic search algorithms. The robot needs to replan the next path point for each time step they take, while heuristic search algorithms require a longer time for each planning, making it difficult to conduct real-time planning in the environment. And some heuristic search algorithms are prone to falling into local optima, resulting in the robot being unable to reach the target point [19]. Some scholars use neural network algorithms to solve path planning problems. Changjian Lin et al. propose an improved recurrent neural network for UUV online obstacle avoidance [20]. This algorithm obtains shorter paths, uses less energy through their actuators, and is insensitive to noise. However, the collision avoidance strategy learned by this algorithm is based on expert data and lacks adaptability to the environment. The quality of expert data has a significant impact on collision avoidance effectiveness. Once the expert knowledge is unreasonable, the collision avoidance strategies learned by the neural network are less likely to achieve good results.

The collision avoidance planning policy based on deep reinforcement learning is autonomously learned by the agent, and the training dataset comes from the interaction between the agent and the environment. Reasonable setting of the reward function can achieve good collision avoidance effects. The collision avoidance planning algorithm based on deep reinforcement learning has good real-time performance and can meet the heading constraints of robots. Prashant Bhopale proposed a modified Q-learning obstacle avoidance algorithm for UUVs [21]. This method reduces the chances of a collision with obstacles, but the action space of this method is discrete, which may result in the resulting path not being optimal. Jiawei Wang proposed the APF-DQN collision avoidance planning algorithm [22]. This method introduces direction reward to solve the problem of sparse reward in the DQN algorithm, but the action space of this method is discrete, and the obstacle environment is relatively simple. Janani Bodaragama proposed a collision avoidance planning algorithm based on the Random Network Distillation algorithm [23]. However, the action space in the text is discrete, with an angular velocity of 1.5 rad set, which is very detrimental to the control of actual robots.

As a type of deep reinforcement learning algorithm, the A3C (Asynchronous Advantage Actor-Critic) algorithm can effectively utilize computer resources and improve training efficiency by synchronously training Actor-Critic in multiple threads. The A3C algorithm can effectively solve problems in the field of mobile robots. Yoko Sasaki proposed the A3C Motion Learning algorithm for an autonomous mobile robot [24]. The experimental results indicate that the robot can acquire short-term, simple collision avoidance motion. However, the research object in the article is omnidirectional mobile robots, which reduces the difficulty of designing collision avoidance algorithms. Z. Zhou proposed a path planning algorithm based on deep reinforcement learning, aiming at the demand for flexibility and real-time performance in an unknown aquatorium [25]. This method discretizes the angular velocity, which is not conducive to USV control and may result in suboptimal paths.

In order to improve the real-time performance of collision avoidance planning algorithms and ensure that the planned path meets the kinematic constraints of UUVs, this article will use the A3C algorithm to solve the collision avoidance planning problem of UUVs in unknown environments. This algorithm adopts a continuous action space and restricts the heading angle of UUVs.

The structure of this article is as follows: Section 2 describes the basic principles of deep reinforcement learning and the kinematic model of the UUV. Section 3 designs a UUV collision avoidance planning algorithm based on A3C. Section 4 conducts simulation experiments in different obstacle environments. The conclusion is given in Section 5.

2. Materials

This section mainly introduces the kinematic modeling of the UUV and the basic principles of the A3C algorithm.

2.1. UUV Model

This article takes the UUV as the research object, with a total length of 4.5 m, a total width of 1.1 m, and a height of 0.6 m. In order to study the motion law of the UUV and determine its underwater motion position information, it is necessary to first establish a suitable coordinate system and analyze the force and maneuverability of the UUV under this coordinate system. Usually, the two basic coordinate systems for UUV motion modeling are fixed coordinate systems and motion coordinate systems, as shown in Figure 1.

For the convenience of research, this article simplifies the six-degree-of-freedom motion model of UUVs, only considering the motion of the underactuated UUV in the longitudinal, transverse, and bow directions. Therefore, the assumptions are made:

(1)

Neglecting the influence of third-order and higher-order hydrodynamic coefficients on the UUV;

(2)

Neglecting the influence of roll, pitch, and heave movements of the UUV on horizontal motion.

The movement of the UUV is mainly carried out in three degrees of freedom: longitudinal, transverse, and bow. This article defines the vector $\eta ={\left[\begin{array}{ccc}x& y& \psi \end{array}\right]}^T$ to represent the generalized position information of the UUV in a fixed coordinate system, The vector $v={\left[\begin{array}{ccc}u& v& r\end{array}\right]}^T$ represents the generalized velocity of the UUV, where $u$ is the longitudinal velocity of the UUV, $v$ is the lateral velocity of the UUV, and $r$ is the raw rate. The schematic diagram of the horizontal movement of the UUV is shown in Figure 2, where $v_t=\sqrt{u^2+v^2}$ is the UUV’s synthesis speed and $\beta =\arctan (v/u)$ is the drift angle.

In the absence of interference, the kinematic model of the UUV in the horizontal plane is as follows:

$$\left\{\begin{array}{c}\dot{x}=u\cos (\psi )-v\sin (\psi )\\ \dot{y}=u\sin (\psi )+v\cos (\psi )\\ \dot{\psi }=r\end{array}\right.$$

(1)

The corresponding dynamic equations are as follows:

$$\left\{\begin{array}{c}\dot{u}=\frac{F}{m_u}-\frac{d_u}{m_u}\\ \dot{v}=-\frac{m_{ur}ur}{m_v}-\frac{d_v}{m_v}\\ \dot{r}=\frac{T}{m_r}-\frac{d_r}{m_r}\end{array}\right.$$

(2)

where $m_u=m-X_{\dot{u}}$, $m_v=m-Y_{\dot{v}}$, $m_r=I_z-N_{\dot{r}}$, $m_{ur}=m-Y_r$, $m_r=I_z-N_{\dot{r}}$, $m_{ur}=m-Y_r$, $d_u=-X_{uu}{u}^2-X_{vv}{v}^2$, $d_v=-Y_{v}uv-Y_{v\left|v\right|}v\left|v\right|$, $d_r=-N_{v}uv-N_{v\left|v\right|}v\left|v\right|-N_{r}ur$. $m$ is the mass of the UUV, with a value of 2234.5 kg; $I_z$ is the mass moment of inertia with regard to the Z axis of the body-fixed frame; $X_{\{\cdot \}}$, $Y_{\{\cdot \}}$ and $N_{\{\cdot \}}$ are the hydrodynamic coefficients; and $F$ and $T$ are the longitudinal thrust and turning moment of the UUV, respectively.

The INFANTE AUV model is the research object of this article [26]. The mass of the UUV is 2234.5 kg and the relevant hydrodynamic coefficients are shown in the

Table 1. Hydrodynamic coefficient.

hydrodynamic parameter	$X_{\dot{u}}$/$Kg$	$Y_{\dot{v}}$/$Kg$	$N_{\dot{r}}$/$N\cdot m^2$	$X_{uu}$/$Kg\cdot m^{-1}$	$X_{vv}$/$Kg\cdot m^{-1}$	$Y_v$/$Kg\cdot m^{-1}$
parameter value	−142	−1715	−1350	−35.4	−128.4	−346
hydrodynamic parameter	$Y_{v\left|v\right|}$/$Kg\cdot m^{-1}$	$Y_r$/$Kg$	$N_r$/$Kg\cdot m$	$N_{v\left|v\right|}$/$Kg$	$I_z$/$N\cdot m^2$	$N_v$/$Kg$
parameter value	−667	435	−1427	443	2000	−686

.

When controlling the UUV, the traditional PID control algorithm was used, with heading angle $\psi$ and longitudinal velocity $u$ as the controlled variables. Taking the expected longitudinal speed $u_{com}=1 \mathrm{m}/\mathrm{s}$ and the expected heading angle ${\psi }_{com}=0.5 \mathrm{rad}$ as an example, the PID control effect is shown in the Figure 3.

As shown in the above figure, the PID controller can adjust the controlled quantity to the given value within 10 s.

2.2. A3C

2.2.1. Reinforcement Learning

The principle of reinforcement learning is shown in Figure 4. When an agent completes a task, it first generates action $A_t$, which will have an impact on the environment, causing the agent to reach a new state $S_t$. At this point, the agent will receive an immediate reward $R_t$ from the environment. If repeated like this, the continuous interaction between the intelligent agent and the environment will generate a large amount of data, including status, rewards, and actions. The reinforcement learning algorithm uses the generated data to modify its own action strategy. The agent interacts with the environment again based on the new strategy, generating a new round of data, and improving its own strategy again using the new round of data. After multiple iterations, the strategy gradually converges, and the agent can learn the optimal strategy to complete the corresponding task.

In a broad sense, reinforcement learning can be considered a sequential decision problem, with the goal of finding the decision sequence that maximizes the expected cumulative return. Sequential decision problems have a lot of content, and, specifically, they should be incorporated into the framework of the Markov Decision Process (MDP) [27].

The goal of reinforcement learning is to find a strategy that can achieve the maximum expected cumulative return. The strategy can be understood as a mapping relationship from state $S_t$ to action $A_t$, which means that the probability of action in the state can be determined based on the strategy. The strategy can usually be represented by the symbol $\pi$.

$$\pi (a|s)=P[A_t=a|S_t=s]$$

(3)

The overall process of the MDP is summarized as follows: set the initial state of the intelligent agent to $s_0$; obtain the current action $a_0$ according to the designed strategy $\pi (a|s)$; then obtain the state $s_1$ at the next moment based on the probability of state transition $P(s_{t+1}|s_t,a_t)$; at the same time, the intelligent experience receives the reward feedback $r_1$ from environment. The entire process continues to loop until the final state $s_T$, ultimately resulting in a trajectory sequence $\tau =(s_0,a_0,s_1,\dots ,s_T)$. After obtaining the complete trajectory sequence, the cumulative return can be calculated, which is $R={\displaystyle \sum _{t=0}^{T-1}{\gamma }^t{r}_t}$. The expected cumulative return is shown in the following equation:

$$E_R=E[{\displaystyle \sum _{t=0}^{T-1}{\gamma }^t{r}_t}]$$

(4)

The strategy that the agent ultimately seeks is the one that maximizes the expected cumulative return:

$${\pi }^{*}=\underset{\pi }{\max }{E}^{\pi }[{\displaystyle \sum _{t=0}^{T-1}{\gamma }^t{r}_t}]$$

(5)

2.2.2. A3C

Deep learning belongs to the connectionist school, which mainly focuses on intelligent perception and optimization objectives are mostly continuous functions. Reinforcement learning is an intelligent decision-making approach that belongs to the behaviorist school of thought, with optimization objectives being discrete functions. Although deep learning and reinforcement learning both belong to machine learning, reinforcement learning has certain advantages compared to supervised and unsupervised learning in deep learning [28]. The deep reinforcement learning method combines the powerful perception ability of deep learning with the decision-making ability of reinforcement learning. This algorithm is more intelligent and can solve more complex tasks, making it a more human-like artificial intelligence method [29].

As a type of deep reinforcement learning, the A3C algorithm can effectively utilize computer resources and improve training efficiency by synchronously training Actor-Critic in multiple threads. The A3C algorithm is a reinforcement learning algorithm based on the Actor-Critic framework [30]. The Actor-Critic framework can be divided into two different systems, Actor and Critic, which can be replaced by different neural networks [31]. The Critic can be used to learn the reward and punishment mechanism. After learning, the Actor uses the strategy function to generate actions according to the state. Then, the parameter values of the policy function are updated according to the knowledge learned by the Critic, and the Critic network updates its own parameters while evaluating the Actor. The schematic diagram of the Actor-Critic framework is shown in Figure 5.

The Actor also can be called strategy $\pi$. The Actor can be parameterized directly, represented by $\theta$ here. Then the action $a$ of the agent can be obtained by sampling the strategy $\pi$, which can be expressed in the following form:

$$a\sim \pi (s,a,\theta )$$

(6)

In the Actor-Critic framework, in order to evaluate the performance of the agent, the concept of Q value in reinforcement learning can be used. At this point, the role of the Critic is to fit the Q value and evaluate the score of the actor’s behavior, which can be approximated by using a neural network. The gradient expression of the Actor is as follows:

$$\nabla {\overline{R}}_{\theta }=\frac{1}{m}{\displaystyle \sum _{n=1}^m{\displaystyle \sum _{t=1}^{T_n}{Q}^{{\pi }_{\theta }}(s_t^n,a_t^n)\nabla \log p_{\theta }(a_t^n|s_t^n)}}$$

(7)

In order to update the strategy by the Actors, it is necessary for the Critic to fit the Q value. The updated method of the Critic is achieved through the Temporal Difference (TD) error, and its loss function is as follows:

$$Loss=\frac{1}{m}{\displaystyle \sum _{n=1}^m{\displaystyle \sum _{t=1}^{T_n}{(r_t^n+\underset{a_{t+1}^{\pi }}{\max }{Q}^{{\pi }_{\theta }}(s_{t+1}^n,a_{t+1}^n)-Q^{{\pi }_{\theta }}(s_t^n,a_t^n))}^2}}$$

(8)

The A3C is a new lightweight parallel algorithm. In order to break the correlation of data, an asynchronous method was adopted, which can ensure that the data is not generated synchronously. The A3C [32] framework sets up a Global Network (Global Neural Network Model) at the top level of the algorithm. The Global Neural Network Model is an Actor-Critic structure, mainly used to collect the experience learned by each sub Actor-Critic model and update its parameters for the sub Actor-Critic. The sub Actor-Critic model can be seen as a local model under multi-threading, where each sub thread corresponds to a set of algorithm models and training environments. Each sub thread does not interfere with each other, and in order to make the model converge faster, some method can be used to make the training environment of each sub thread different, so that the sub threads can interact with different environments as much as possible, learn different experiences, and accelerate the learning of the global model. After interacting with the environment, the sub thread updates the learned experience to the global neural network. The structure diagram of the A3C algorithm is shown in Figure 6.

The A3C algorithm process is shown in Algorithm 1:

Algorithm 1 A3C

// Assume global shared parameter vectors $\theta$ and ${\theta }_v$ and global shared counter T = 0

// Assume thread-specific parameter vectors ${\theta }^{\prime }$ and ${\theta }_v^{\prime }$

Initialize thread step counter t $\leftarrow$ 1

repeat

Reset gradients: $d\theta \leftarrow 0$ and $d{\theta }_v\leftarrow 0$.

Synchronize thread-specific parameters ${\theta }^{\prime }=\theta$ and ${\theta }_v^{\prime }={\theta }_v$

$t_{start}=t$

Get state $s_t$

repeat

Perform $a_t$ according to policy $\pi (a_t|s_t;{\theta }^{\prime })$

Receive reward $r_t$ and new state $s_{t+1}$

$t\leftarrow t+1$

$T\leftarrow T+1$

until terminal $s_t$ or $t-t_{start}==t_{\max }$

$R=\left\{\begin{array}{cc}\hspace{1em}\hspace{1em}0& for ter\min al s_t\hfill \\ V(s_t,{\theta }_v^{\prime })& for non-terminal s_t\hfill \end{array}\right.$

for $i\in \left\{t-1,\dots ,t_{start}\right\}$ do

$R\leftarrow r_i+\gamma R$

Accumulate gradients wrt ${\theta }^{\prime }$:$d\theta \leftarrow d\theta +{\nabla }_{{\theta }^{\prime }}\log \pi (a_i|s_i;{\theta }^{\prime })(R-V(s_i;{\theta }_v^{\prime }))$

Accumulate gradients wrt ${\theta }_v^{\prime }$:$d{\theta }_v\leftarrow d{\theta }_v+\partial {(R-V(s_i;{\theta }_v^{\prime }))}^2/\partial {\theta }_v^{\prime }$

end for

Perform asynchronous update of $\theta$ using $d\theta$ and of ${\theta }_v$ using d${\theta }_v$

until $T>T_{max}$

3. The A3C Collision Avoidance Planning Algorithm

In order to achieve autonomous obstacle avoidance for the UUV, this paper uses the A3C reinforcement learning algorithm. This section introduces the selection of state space, the design of reward function, the construction of the environment, and the design of action space.

3.1. State Space

The state information includes observation information obtained from the environment and target point information, which has a significant impact on the final effect. The UUV carries the SeaBat 8125-H forward-looking sonar model from RESON company, which can detect a horizontal 120° fan-shaped area with a vertical opening angle of 17°. The state information selected in this article includes the distance between the UUV and the obstacle, the angle at which the robot’s heading deviates from the obstacle, the distance between the UUV and the target point, and the angle at which the robot’s heading deviates from the target. Because the obstacle data returned by the forward-looking sonar have a higher dimension, and the A3C algorithm needs to spend more time learning the data. In reactive collision avoidance planning algorithms, the high-dimensional obstacle data are not necessary, so obstacle data are simplified through partitioning. The 120° fan-shaped area is divided at intervals of 10°. The expressions of distance ${\rho }_t$ between the UUV and target point, and the angle ${\theta }_t$ at which the robot’s heading deviates from the target are as follows.

$${\rho }_t=\sqrt{{\left(x_r-x_g\right)}^2+{\left(y_r-y_g\right)}^2}$$

(9)

$${\theta }_t={\theta }_r-\arctan \frac{y_g-y_r}{x_g-x_r}$$

(10)

where $(x_r,y_y)$ are the coordinates of the UUV in the global coordinate system, and $(x_g,y_g)$ are the coordinates of the target point in the global coordinate system.

3.2. Action Space

The action strategy selection used in this section is designed as a continuous action space. If the action space includes the linear velocity and angular velocity, which are both continuous, the deep reinforcement learning algorithm is difficult to converge. Therefore, the A3C collision avoidance planning algorithm keeps the linear velocity unchanged and only changes the heading of the UUV, that is, the action space includes the angular velocity item. Considering the actual situation, the angular velocity of the UUV is limited between $-5^{\circ }/s$ and $5^{\circ }/s$.

3.3. Reward Function

The design of the reward function is of utmost importance in reinforcement learning, and the quality of reinforcement learning training results is closely related to the reward function. The purpose of reinforcement learning is to obtain the maximum long-term cumulative reward. The reward function evaluates each action based on its interaction with the environment, so the quality of the reward function affects the quality of the final strategy.

The reward function settings in this section are as follows:

$$R_{}=R_r+R_d{R}_c+R_s$$

(11)

$R_r$ represents the reward received by the agent when it reaches the final target point; $R_c$ represents the reward received by the agent when colliding with obstacles in the environment; $R_d=k_{dis}\cdot di{s}_{obs}$, and $di{s}_{obs}$ indicate the distance between the UUV and the nearest obstacle, and $k_{dis}$ is the obstacle reward coefficient. The main purpose of the $R_d$ is to successfully avoid obstacles. $R_s=k_{step}\cdot step\_count$ represents the penalty of the reward function on the number of steps and can ensure that the number of steps for the UUV to reach the target point is minimized as much as possible, $k_{step}$ is the penalty coefficient for the number of steps, and $step\_count$ is the current cumulative number of steps.

4. Experiments and Results

The experiments are conducted to verify the A3C collision avoidance planning algorithm. The experiments take the UUV model in Section 2 as the research object. The obstacle environments include dense irregular obstacle environments and narrow passage obstacle environments. The performance of the algorithms is evaluated from the aspects of path length, average solution time, and whether the target point has been reached.

In order to demonstrate the advantages of the algorithm in this article, this section uses the artificial potential field method and genetic algorithm as comparisons. In the experiment, the longitudinal velocity $u$ of the UUV remains fixed at 1 m/s, and only the heading angle of the UUV is changed to achieve obstacle avoidance. The experiments use the PID controller to control the heading angle and the longitudinal speed of the UUV.

The simulation results are as follows:

Figure 7 shows the reward function curve, from which it can be seen that after a certain number of iterations, the reward function converges, proving the feasibility of the algorithm proposed in this article. In the Figure 8, it can be seen that the A3C collision avoidance planning algorithm can plan a safe and collision-free path in a dense and irregular obstacle environment, while the APF algorithm cannot adapt to such obstacle environments due to its lack of learning ability to the environment. In Figure 9, it can be seen that three algorithms can successfully reach the target point in narrow channel obstacle environments. However, due to the addition of turning restrictions, the A3C algorithm has a relatively smooth change in heading angle and trajectory, while the APF algorithm and GA algorithm do not have turning restrictions, resulting in a significant difference between the expected heading angle obtained and the current heading angle. On the one hand, it puts higher requirements on the UUV controller, and on the other hand, it will cause the UUV to perform a significant turning action, which puts higher requirements on the performance of the UUV.

Table 2. The performance in the dense irregular obstacle environment.

Algorithm	Path Length/m	Execution Time/ms	Whether the Target Point Has Been Reached
A3C	545	1.2	√
APF	/	1.1	×
GA	546	311	√

and

Table 3. The performance in the narrow passage obstacle environment.

Algorithm	Path Length/m	Execution Time/ms	Whether the Target Point Has Been Reached
A3C	503	1.3	√
APF	506	1.3	√
GA	505	310	√

show the performance comparison of three algorithms. From the perspective of path length, the three algorithms are equally matched. From the perspective of average execution time, the execution time of the A3C algorithm and APF algorithm is similar, while the execution time of the GA algorithm is longer.

5. Conclusions

This article proposes the A3C local collision avoidance planning algorithm for UUVs, which maintains the fixed longitudinal velocity of the UUV and only changes its heading. For the problem of collision avoidance planning, this article designs state space, action space, and reward function. In order to smooth the collision avoidance trajectory of the UUV and reduce the difficulty of controlling the UUV, this article applies yaw restrictions to the final output of the algorithm, controlling the angular velocity between $-5^{\circ }/\mathrm{s}$ and $5^{\circ }/\mathrm{s}$. In order to verify the effectiveness of the algorithm proposed in this article, simulation experiments were conducted and compared with the artificial potential field method and genetic algorithm. The simulation results show that the A3C collision avoidance planning algorithm can plan smooth collision avoidance paths in dense and irregular obstacle environments, as well as narrow channel obstacle environments. The execution time of the A3C collision avoidance planning algorithm is in milliseconds, and the planned yaw rate is small, which can ensure the smoothness of the trajectory.