Deep Reinforcement Learning-Based Path Planning for Multi-Arm Manipulators with Periodically Moving Obstacles

Abstract

In the workspace of robot manipulators in practice, it is common that there are both static and periodic moving obstacles. Existing results in the literature have been focusing mainly on the static obstacles. This paper is concerned with multi-arm manipulators with periodically moving obstacles. Due to the high-dimensional property and the moving obstacles, existing results suffer from finding the optimal path for given arbitrary starting and goal points. To solve the path planning problem, this paper presents a SAC-based (Soft actor–critic) path planning algorithm for multi-arm manipulators with periodically moving obstacles. In particular, the deep neural networks in the SAC are designed such that they utilize the position information of the moving obstacles over the past finite time horizon. In addition, the hindsight experience replay (HER) technique is employed to use the training data efficiently. In order to show the performance of the proposed SAC-based path planning, both simulation and experiment results using open manipulators are given.

1. Introduction

In the fourth industrial revolution, the operation of an autonomous multi-robot in a complicated workspace has been an important challenge for modern smart factories [1]. It is important to replace the human workforce with robots, to collaborate with robots, and to deploy robots in an efficient manner [2,3]. This paper presents a deep reinforcement learning-based path planning algorithm to deal with periodically moving obstacles.

1.1. Background and Motivation

In the autonomous robot framework, there are three fundamental concepts such as perception, planning, and control [4]. The perception aims to obtain information about the environment using various sensors. The planning schedules a sequence of valid configurations for the robot arms or wheels in order to perform a given task. When the path achieving the goal is given, the controller adjusts actuators so that, for example, the configurations of the robot manipulators follow the planned path as close as possible.

Although the robot manipulator operation is quite diverse in the workspace, most operations can simply be interpreted as moving the robot from a starting point to a goal point. Hence, it is utmost essential for the manipulator to move without collision with any obstacles in the workspace. In practice, there are typically two kinds of obstacles: static and periodically moving obstacles. In the case of static obstacles, their locations do not change in the middle of the robot operation, and most existing path planning methods focus on the static obstacles. For the robot operation with the static obstacles, it is a common situation in practice that human experts design the path for the robot manipulators in order to perform a given task. Recent research effort is directed to make this procedure in an automatic manner. In other words, the human experts’ path planning is replaced with various automatic algorithms [5]. The other is the periodically moving obstacle. For the purpose of dealing with the moving obstacles, this paper considers the situation where every dynamic object in the workspace moves by plan. Consequentially, it is assumed in this paper that there are no unexpected obstacles, and that all dynamic obstacles are moving periodically. In the case of industrial robots in a real factory, this is the case. Moreover, since one robot manipulator is viewed as an obstacle to the other robots, and the entire operation on a factory is usually batch production, many obstacles can be regarded as periodically moving obstacles if they are not static obstacles. Since existing path planning algorithms deal with such a moving obstacle using any methods for static obstacles, the results are conservative. In other words, to deal with the moving obstacles, those algorithms view the whole area where the obstacle moves as one big artificial static obstacle, which is inevitably conservative.

In robot path planning research, the sampling-based algorithm is representative [6]. In order to compute the path connecting the initial and goal points, the sampling-based algorithm samples the nodes from the collision-free space and connects those nodes. Hence, the method relies on how to sample the nodes. In the case of multi-arm manipulators, the problem dimension increases quadratically as the number of arms increases, which makes it difficult for the planner to find an optimal path. Moreover, if the obstacle moves, it is not easy to use the sampling-based algorithms. Existing path planning algorithms, especially sample-based approaches, work well for a single robot manipulator. However, if the target manipulator is a multi-arm manipulator with moving obstacles, the path planning problem becomes difficult since the problem dimension is high and it is not trivial to sample the configuration space.

Hence, the focus of this paper is placed on devising a deep reinforcement learning-based path planning algorithm which can generate collision-free paths when the multi-arm manipulator has both static and periodically moving obstacles.

1.2. Related Work

Fast marching method (FMM) [7], probabilistic road map (PRM) method [8], and rapid exploring random trees (RRT) method  [9,10,11] are representatives of sampling-based algorithms. In PRM, the shortest path is computed from the graph made by the sampled points using the Dijkstra algorithm  [12,13]. Artificial potential field methods are also popularly used to design the path planning algorithm which leads the robot manipulator from the starting point to the goal point  [14,15]. In the derivation, the gradient of the potential function plays a key role in computing the direction of the optimal path [16]. Since it is based on the gradient descent method, it can suffer from trapping in the local minimum, which makes it difficult to be applied to high-dimensional problems [17]. In addition, artificial intelligence like reinforcement learning is also widely used to design collision-avoiding path planning  [18]. In [19], Real-Time RRT* (RT-RRT*) is proposed to solve the path planning problem with a dynamic obstacle in a two-dimensional environment.

This paper improves the result in [5,20] in such a way that the deep reinforcement learning-based path planning for multi-arm manipulators with both static and periodically moving obstacles is proposed using the SAC algorithm with hindsight experience replay (HER).

1.3. Proposed Method

This paper proposes a deep reinforcement learning-based path planning algorithm for multi-arm manipulators with periodically moving obstacles. In particular, in order to deal with the high-dimensional property of the multi-arm manipulators, the SAC-based path planning algorithm is devised. Since the SAC-based algorithm uses an entropy term in its objective function, it can find the optimal solution of the high-dimensional problem, which makes the SAC-based algorithm outperform the existing results. Moreover, for the purpose of handling periodically moving obstacles in the path planning, the neural networks in the SAC are designed such that the position information of the obstacles over the past finite time horizon is used as an input to the neural networks together with the joint information of the robot. Note that the SAC employs five neural networks in order to estimate the value function and optimal policy. Hence, it is crucial to design the neural networks appropriately depending on the given problem. After the proposed SAC-based path planning is trained offline for various starting and goal points in the workspace, it can find the optimal path for arbitrary starting and goal points online by computing the forward path of the trained actor-network, which means that the proposed method is computationally cheap even for the high-dimensional problems. Both simulation and experiment results using real robot manipulators show the efficiency of the proposed method.

2. Background Concept and Problem Modeling

2.1. Path Planning for Robot Manipulator and Configuration Space

In the path planning problem for a robot manipulator, the configuration is defined as a position representation of the robot in the workspace. In other words, the configuration space $\mathbf{Q}$ (also called joint space in a robot manipulator) is a set of all possible joint angles of the robot manipulator [3,21,22]. To be specific, $\mathbf{Q}$ is defined as a subset of n-dimensional vector space ${\mathbb{R}}^n$, where n is the number of the joints of the robot manipulator. Because of this reason, the position of the robot can be represented as point $\overline{q}\in {\mathbb{R}}^n$ that has n angle values (${\overline{q}}_1,...,{\overline{q}}_n$), where each value indicates the joint angle of the robot. The set $\mathbf{Q}$ consists of two subsets: the first subset is the collision-free space ${\mathbf{Q}}_{\mathrm{free}}$ that is comprised of all possible configurations, and in which the robot does not collide with any obstacles or itself. The second subset is called the collision space ${\mathbf{Q}}_{\mathrm{collide}}$ which is the complement of ${\mathbf{Q}}_{\mathrm{free}}$ in $\mathbf{Q}$ and the subset of $\mathbf{Q}$. In ${\mathbf{Q}}_{\mathrm{collide}}$, the robot arm manipulator collides with obstacles or itself [21,23]. The robot manipulator does not collide with any obstacles or itself if its joint angles belong to ${\mathbf{Q}}_{\mathrm{free}}$. However, if there is a collision between the robot manipulator and obstacles or itself, the values of the joint angles must be in ${\mathbf{Q}}_{\mathrm{collide}}$.

In the sampling-based method, the discrete representation of ${\mathbf{Q}}_{\mathrm{free}}$ is obtained by random sampling in ${\mathbf{Q}}_{\mathrm{free}}$. The valid path is defined by a connected line between the sampling points. Such sampled points and paths can establish a graph. Because of this construction, the nodes in the graph describe possible configurations of the robot manipulators. In addition, the collision-free paths between any two nodes are represented by connected edges in the graph. For the notation of the path planning problem, let ${\overline{q}}_t\in \mathbf{Q}\in {\mathbb{R}}^n$ mean the values of the joint angles of the manipulator at the tth iteration, and T describe the maximum number of iterations in the algorithm. Thus, the algorithm needs to find the path within T iterations for a given starting configuration ${\overline{q}}_{\mathrm{init}}\in {\mathbf{Q}}_{\mathrm{free}}$ and goal configuration ${\overline{q}}_{\mathrm{goal}}\in {\mathbf{Q}}_{\mathrm{free}}$ .

Based on this concept, when ${\overline{q}}_{\mathrm{init}}\in {\mathbf{Q}}_{\mathrm{free}}$ and ${\overline{q}}_{\mathrm{goal}}\in {\mathbf{Q}}_{\mathrm{free}}$ are acquired, the algorithm computes a valid continuous and shortest path linking ${\overline{q}}_{\mathrm{init}}$ and ${\overline{q}}_{\mathrm{goal}}$. The path is defined as a sequence of the state ${\overline{q}}_t$ such that ${\overline{q}}_0={\overline{q}}_{\mathrm{init}},\cdots ,{\overline{q}}_{T_1}={\overline{q}}_{\mathrm{goal}}$ where $T_1\le T$. In addition, the sequence $\{{\overline{q}}_0,\cdots ,{\overline{q}}_{T_1}\}$ and the line segment connecting any two neighbors state ${\overline{q}}_t$ and ${\overline{q}}_{t+1}$ from $\{{\overline{q}}_0,\cdots ,{\overline{q}}_{T_1}\}$ have to belong to ${\mathbf{Q}}_{\mathrm{free}}$.

2.2. Collision Detection in Workspace Using the Oriented Bounding Box (OBB)

The definition of workspace $\mathbf{W}$ is a space where an actual robot operates. In many cases including the robot manipulator, $\mathbf{W}$ space is in three-dimensional Euclidean space ${\mathbb{R}}^3$. To confirm that ${\overline{q}}_t\in {\mathbf{Q}}_{\mathrm{free}}$, collision detection methods have to be applied in workspace $\mathbf{W}$ [21,23]. To this end, the oriented bounding boxes (OBB) [24] is employed for collision detection. In OBB, both the robot arm and obstacles are modeled as a box. Because the modeling is implemented in OBB, the collision checking between 2 boxes only has 15 cases: three faces of the first box, three faces of the second box, and nine edge combinations between the first box and second box [25]. Based on two box checking, the higher level checking between one robot and one obstacle can be implemented as a repeated two OBB checking [26].

2.3. Reinforcement Learning

As a standard solution to the MDP problem, the fundamental method in the reinforcement learning algorithm uses an optimization-based method to maximize the sum of the reward signal by letting the agent interact with a stochastic environment and choose the action over sequences of discrete-time steps [27]. The MDP is defined by the tuple $\{S,A,P,r,\gamma \}$, where S denotes the set of the state, A the set of action, P the transition probability, $r(s,a)$ the reward function, and $\gamma$ the discount factor [28]. The transition probability $P\left(s^{\prime }\right|s,a)$ represents the probability of moving the current state s to the next state $s^{\prime }$ when action $a\in A$ is applied. Regarding the agent, the distribution of action a for the given state s from the environment is implied by the policy that is denoted by $\pi \left(a\right|s)$. At each time step t, the agent selects action $a_t\in A$ based on the policy $\pi :S\to A$ and state $s_t\in S$, and applies it to the environment. After that, the environment returns both the next state $s_{t+1}\in S$ and reward $r_{t+1}\in \mathbb{R}$ according to the transition probability $P:S\times A\to \mathcal{P}\left(S\right)$, where $\mathcal{P}\left(S\right)$ means that the probability of the next state and the reward function $r:S\times A\to \mathbb{R}$. By repeating this procedure, the learning process updates the policy until it reaches the optimal policy, defined by ${\pi }^{*}$, which maximizes the expected return $E_{\pi }\left[{\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}\right]$. In order to acquire the optimal policy, the optimal value function (i.e., estimate of the maximum return) is approximated by value-based methods like deep Q-network (DQN) as function approximation [29]. The other way to get the optimal policy is using the policy gradient methods to compute the optimal policy directly from the agent experience. Representative methods of the policy gradient are REINFORCE, actor–critic method, deterministic policy gradient (DPG), deep DPG (DDPG), asynchronous advantage actor–critic (A3C), trust region policy optimization (TRPO), maximum a posteriori policy optimization (MPO), and distributed distributional DDPG (D4PG) [30,31,32,33,34]. In the reinforcement learning, the training performance mostly depends on the set of samples of state $s_t$, action $a_t$, reward $r_{t+1}$, and next state $s_{t+1}$. In addition to the various reinforcement learning algorithms, efforts are directed at devising techniques to utilize the samples (called episode) effectively for better learning, for example replay memory [29] and HER [35]. In this paper, for the purpose of tackling the path planning problem for the multi-arm manipulators with periodically moving obstacles, a soft actor–critic (SAC)[36] algorithm is employed.

3. Soft Actor–Critic Based Path Planning for Periodically Moving Obstacles

In this section, in order to design a SAC-based path planning, the MDP for the problem is defined, and the proposed deep learning network used in the SAC is presented. Although the considered MDP is quite similar to that in [20], since both deal with multi-arm manipulators, the MDP is explained here in a self-contained manner.

3.1. Path Planning for the Multi-Arm Manipulator and Augmented Configuration Space

The multi-arm manipulator under consideration consists of m identical robot-arm manipulators that have n number of joints. Let $q_t^i\in {\mathbb{R}}^n$ determine the joint angles of the ith robot-arm at the tth iteration in the proposed SAC-based method, and $q_{t,j}^i\in \mathbb{R}$ denotes the jth component of $q_t^i\in {\mathbb{R}}^n$, i.e., the value of the jth joint angle of the ith robot-arm.

In the path planning algorithm for the multi-arm manipulator, if the configuration space is defined for each robot, it is difficult to define ${\mathbf{Q}}_{\mathrm{free}}$ and ${\mathbf{Q}}_{\mathrm{collide}}$ during the robot operation since ${\mathbf{Q}}_{\mathrm{free}}$ for each arm is affected by the other arms, i.e., one robot serves as an obstacle to the others in the workspace. Taking this into account, it is convenient to define one big configuration space by augmenting the configuration space of each manipulator. This corresponds to regarding the multi-arm manipulator as one virtual manipulator. The dimension of the augmented configuration space is $nm$. Hence, the state of the virtual manipulator is represented by

$$q_t=\left[\begin{array}{c}{q}_t^1\\ q_t^2\\ ⋮\\ q_t^m\end{array}\right]\in {\mathbb{R}}^{nm},q_t^i=\left[\begin{array}{c}{q}_{t,1}^i\\ q_{t,2}^i\\ ⋮\\ q_{t,n}^i\end{array}\right]\in {\mathbb{R}}^n,i\in \{1,2,\cdots ,m\}.$$

(1)

Therefore, the resulting augmented configuration space is described by ${\mathbf{Q}}^a:={\mathbf{Q}}_{\mathrm{free}}^a\bigcup {\mathbf{Q}}_{\mathrm{collide}}^a$ where

$${\mathbf{Q}}_{\mathrm{free}}^a=\underset{m\mathrm{times}}{\underbrace{{\mathbf{Q}}_{\mathrm{free}}\times \cdots \times {\mathbf{Q}}_{\mathrm{free}}}}\subset {\mathbb{R}}^{nm}\mathrm{and}{\mathbf{Q}}_{\mathrm{collide}}^a=\underset{m\mathrm{times}}{\underbrace{{\mathbf{Q}}_{\mathrm{collide}}\times \cdots \times {\mathbf{Q}}_{\mathrm{collide}}}}\subset {\mathbb{R}}^{nm},$$

${\mathbf{Q}}_{\mathrm{free}}^a$ defines the collision-free space for $q_t$ and ${\mathbf{Q}}_{\mathrm{collide}}^a$ denotes the corresponding collision space. Finally, the path planning problem of the multi-arm manipulator can be redefined as the path planning for the (virtual) single-arm manipulator with a given starting point $q_0\in {\mathbf{Q}}_{\mathrm{free}}^a$ and goal point $q_{\mathrm{goal}}\in {\mathbf{Q}}_{\mathrm{free}}^a$ being

$$q_0=\left[\begin{array}{c}{q}_{\mathrm{init}}^1\\ q_{\mathrm{init}}^2\\ ⋮\\ q_{\mathrm{init}}^m\end{array}\right],q_{\mathrm{goal}}=\left[\begin{array}{c}{q}_{\mathrm{goal}}^1\\ q_{\mathrm{goal}}^2\\ ⋮\\ q_{\mathrm{goal}}^m\end{array}\right],$$

where $q_{\mathrm{init}}^i$ and $q_{\mathrm{goal}}^i$ denote the starting and goal point of the ith robot-arm.

3.2. Multi-Arm Manipulator Markov Decision Process Considering Periodically Moving Obstacle

In order to apply deep reinforcement learning to the path planning algorithm for the multi-arm manipulator considering a periodically moving obstacle, a multi-arm manipulator MDP (MAMMDP${}_{\mathrm{mo}}$), i.e., the tuple $\{S,A,P,r,\gamma \}$, needs to be defined. For a better understanding, the diagram of MAMMDP${}_{\mathrm{mo}}$ is depicted in Figure 1.

In principle, MAMMDP${}_{\mathrm{mo}}$ is similar to MAMMDP in [20]. The only difference is that the state in MAMMDP${}_{\mathrm{mo}}$ is defined such that the information on the moving obstacle is related to the state. For this purpose, it has to be determined which information on the obstacle is used in defining MAMMDP${}_{\mathrm{mo}}$. In this paper, the location of the obstacle in the workspace for the past $n_d$ sampling times is used where $n_d>0$ denotes the window size. Namely, the following information on the obstacle is used:

$$D_t=\left[\begin{array}{c}{d}_t\\ d_{t-1}\\ ⋮\\ d_{t-(n_d-1)}\end{array}\right]\in {\mathbb{R}}^{3·n_d},d_t=\left[\begin{array}{c}{x}_t^d\\ y_t^d\\ z_t^d\end{array}\right]\in {\mathbb{R}}^3,$$

(2)

where $d_t$ represents the location of the obstacle in the workspace at the sampling time t. This $n_d$ can be viewed as the number of the features obtained from the obstacle. In addition to this time window, it is also assumed that $D_t$ is available only when the obstacle belongs to a predefined region such as a neighborhood of the multi-arm manipulator. When the manipulator does not belong to the region, $D_t$ is not available to the MAMMDP${}_{\mathrm{mo}}$. With this in mind, the current state $s_t\in {\mathbb{R}}^{nm}$ is defined as $s_t=(q_t\parallel D_t)$. In other words, the state contains the information of not only the current configuration of the multi-arm manipulator but also $D_t$ (i.e., obstacle location over the past $n_d$ sampling times).

The action is calculated by $a_t=f(e_t,s_t)$, where $e_t$ follows the Gaussian distribution $\mathcal{N}(0,{\sigma }_t)$ with ${\sigma }_t$ being the variance, and $f(·,·)$ generates a stochastic action using noise $e_t$ and state $s_t$. As a matter of fact, function $f(·)$ serves as a sampler. Thus, the action is sampled from a probability distribution and variation of the configuration. Then, the next configuration is calculated as the sum of the current configuration and the action such as ${\widehat{q}}_{t+1}=q_t+\alpha a_t+{ϵ}_e$, where tuning parameter $\alpha$ is defined as the geometric mean of the maximum variations of each joint and ${ϵ}_e$ is an environmental noise, where ${ϵ}_e\sim \mathcal{N}(0,{\sigma }_e)$. When ${\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{collide}}^a$ happens, the next configuration becomes the previous configuration meaning that it stays at the current configuration. The following summarizes how the configuration is updated:

$$q_{t+1}=\left\{\begin{array}{cc}{\widehat{q}}_{t+1},\hfill & \mathrm{if}{\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{free}}^a,\hfill \\ q_t,\hfill & \mathrm{if}{\widehat{q}}_{t+1}\notin {\mathbf{Q}}_{\mathrm{free}}^a.\hfill \end{array}\right.$$

Using this, the next state is denoted as $s_{t+1}=(q_{t+1}\parallel D_{t+1})$, where $D_{t+1}$ is the updated information of the obstacle position. Applying the action and obtaining the state and reward are repeated until the next configuration reaches the goal point $q_{\mathrm{goal}}$. Owing to the practical limitation like numerical errors, instead of $q_{t+1}=q_{\mathrm{goal}}$, the relaxed condition $\parallel q_{t+1}-q_{\mathrm{goal}}\parallel \le \eta ·\alpha$ is employed for the termination in an implementation where $\parallel ·\parallel$ defines the norm of a vector and tuning parameter $\eta$ satisfies $0<\eta <1$.

If the next configuration is not the goal point, the corresponding reward is $-1$. However, when the agent meets the terminal condition, the corresponding reward becomes 0. If the agent cannot reach the goal until the maximum number of iteration, the total reward is given by $-T$. On the other case, if the iteration ends at a certain iteration $T_1<T$, the total reward amounts to $-(T_1-1)$. This reward function $r(s_t,a_t)$ is summarized as follows:

$$r_{t+1}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}|q_{t+1}-q_{\mathrm{goal}}|\le \eta ·\alpha \hfill \\ -1,\hfill & \mathrm{if}{q}_{t+1}\in {\mathbf{Q}}_{\mathrm{collide}}^a\hfill \\ -1,\hfill & \mathrm{if}{q}_{t+1}\in {\mathbf{Q}}_{\mathrm{free}}^a\hfill \end{array}\right.$$

(3)

In view of the goal of reinforcement learning, the objective of the proposed algorithm is to find the optimal policy that maximizes the total reward. If the iteration can finish before reaching T, the sum of reward is more than $-T$ such as $-(T_1-1)>-T$. By trying to maximize the total reward in training process, the agent finds the shortest path.

In the next subsection, it is presented how to update the policy in the middle of training using deep neural networks.

3.3. Soft Actor–Critic (SAC) Based Path Planning Considering Periodically Moving Obstacles

Since our MAMMDP${}_{\mathrm{mo}}$ has a continuous action value, there are several reinforcement learning methods for continuous action MDP such as those in [32,36,37]. Soft actor–critic (SAC), especially, is the best known for its performance for the MDP with the continuous action and high-dimensional state. SAC is based on maximum entropy reinforcement learning that maximizes not only the expected sum of rewards but also the entropy of the policy of action [36,38].

When the policy $\pi$ is updated by maximizing only the total reward, the center of the distribution of the policy is formed around a specific action. However, by maximizing policy entropy together with the reward, the SAC method can make the distribution spread more and encourage the agent to explore a wider range of the actions. The soft state-value function and soft Q-value function are defined as follows:

$$V\left(s_t\right)={\mathbb{E}}_{a_t\sim \pi }[Q(s_t,a_t)-\beta log\pi \left(a_t\right|s_t)],$$

(4)

$$Q(s_t,a_t)=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim p}\left[V\left(s_{t+1}\right)\right],$$

(5)

where $\beta$ denotes the temperature parameter of the entropy function. The soft state value function $V\left(s_t\right)$ defined in (4) is the expected sum of the entropy augmented reward for the given current state $s_t$. In addition, the soft Q-value function or soft action-value function $Q(s_t,a_t)$ are defined in (5) and is the expected sum of augmented reward for a given pair of state $s_t$ and action $a_t$.

For the purpose of estimating the soft value and Q-value functions, and finding the optimal policy, the SAC-based algorithm consists of five neural networks: two for the soft value function, two for the soft Q-functions, and one for the policy. The proposed SAC-based path planning utilizes the structure described in Figure 2. The structure is quite similar to that in [20] except for the information of the moving obstacle. As the proposed path planning has to take the periodically moving obstacle into account, it has to be decided properly how to use the obstacle information $D_t$ for the path planning. Since information $D_t$ has to be considered when the policy is trained at a given state in order for the manipulator to enable to avoid the obstacle, information $D_t$ is also used as the input to the network for the policy. In addition, since the policy network is affected by the value functions, as a result, all the neural networks for the proposed SAC-based path planning have to have information $D_t$ as an input.

Figure 3 illustrates how the deep neural network (DNN) estimates the state value function. The DNNs N1 and N2 are parameterized by $\psi$ and $\overline{\psi }$. The network parameterized by $\overline{\psi }$ is the target network which makes the training stable and enhances the learning performance of DNNs N3 and N4 for estimating the Q-value function. The inputs to DNN N1 are the configuration $q_t$, goal state $q_{\mathrm{goal}}$, and $D_t$. In order to train DNN N1, the following objective function and its gradient are used:

$$J_V\left(\psi \right)={\mathbb{E}}_{s_t}\left[\frac{1}{2}{(V_{\psi }\left(s_t\right)-{\mathbb{E}}_{a_t}[\underset{k=1,2}{min}{Q}_{{\theta }_k}(s_t,a_t)-\beta log{\pi }_{\varphi }\left(a_t\right|s_t)])}^2\right],$$

(6)

where the minimum of the soft Q-function is determined between $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$, which are the output of the DNN N3 and N4. It is reported that the use of such a minimum makes it possible to avoid overestimation of Q-value [37]. The parameterized policy ${\pi }_{\varphi }$ comes from DNN N5. To find the optimal parameter $\psi$ of DNN N1, the network N1 for the soft state value is trained by minimizing $J_V\left(\psi \right)$ using $\nabla J_V$. After updating the parameter $\psi$ using the gradient descent, the target parameter $\overline{\psi }$ is updated according to $\overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi }$ at each training step where $\tau \in [0,1]$.

In order to estimate the action value function, two DNNs are again used as depicted in Figure 4. The input to N3 and N4 are the same as those of N1 and N2. The network parameters ${\theta }_1$ and ${\theta }_2$ are trained by minimizing the following objective function:

$$J_Q\left({\theta }_{k=1,2}\right)={\mathbb{E}}_{s_t,a_t}\left[\frac{1}{2}{(Q_{{\theta }_{k=1,2}}(s_t,a_t)-(r(s_t,a_t)+V_{\overline{\psi }}\left(s_{t+1}\right)))}^2\right],$$

(7)

where $V_{\overline{\psi }}$ comes from DNN N2, and the double Q-value method is applied as it is suggested in [36,37,39]. The minimum Q-function is also used in the DNN for the objective function (8).

Figure 5 describes the input and output of DNN N5. Similar to the other DNNs, the network parameter $\varphi$ is trained by minimizing the objective function in the following:

$$J_{\pi }\left(\varphi \right)={\mathbb{E}}_{s_t}[log{\pi }_{\varphi }\left(a_t\right|s_t)-\underset{k=1,2}{min}{Q}_{{\theta }_k}(s_t,a_t)],$$

(8)

where $Q_{{\theta }_k}$ is provided by N3 and N4. In fact, this function is a kind of the Kullback–Leibler (KL) divergence between the policy and Q-value [36]. The output of DNN N5 is $f_{\varphi }(e_t;s_t)$, and the action is sampled from $f_{\varphi }(e_t;s_t)$ such that $a_t=f_{\varphi }(e_t;s_t)$.

Since the SAC is a type of the off-policy actor–critic method [36,40], the training gathers transition tuples $(s_t,a_t,s_{t+1},r(s_t,a_t))$ in every action step and stores them in experience replay memory $\mathcal{D}$ [29]. At the start of the training, a collection of these tuples is sampled from the reply memory and used to calculate the expectation in the objective functions. In addition to the SAC, Hindsight Experience Replay (HER) is used to enhance the sample efficiency  [20,35]. In summary, the proposed SAC-based path planning algorithm is presented in the form of the pseudo-code in Algorithm 1.

Algorithm 1 Proposed SAC-based path planning algorithm for multi-arm manipulators.

1: Define MAMMDP and the current state $s_t=(q_t\parallel D_t)$ and the initial and goal point $q_{\mathrm{init}}$ and $q_{\mathrm{goal}}$ 2: Initialize network parameters $\psi ,{\theta }_{1,2},\varphi$ 3: Initialize the parameter values of the target network $\overline{\psi }\leftarrow \psi$ 4: Initialize global replay memory $\mathcal{D}$ 5: for  $e=1$ to M  do 6:     Initialize local buffer $\mathcal{L}$                                        ▹ Memory for an episode 7:     for $t=0$ to$T-1$ do 8:         Randomly choose obstacle position $D_t$ and the goal and initial positions $q_{\mathrm{goal}},q_{\mathrm{init}}\in {\mathbf{Q}}_{\mathrm{free}}^a$ 9:         $a_t=f_{\varphi }({ϵ}_t,s_t\left|\right|q_{\mathrm{goal}}),s_t=(q_t\parallel D_t),{ϵ}_t\sim \mathcal{N}(0,{\sigma }_t)$ 10:         ${\widehat{q}}_{t+1}=q_t+\alpha ·a_t+{ϵ}_e,{ϵ}_e\sim \mathcal{N}(0,{\sigma }_e)$ 11:         if ${\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{free}}^a$ then                                        ▹ Get next state and reward 12:            $q_{t+1}\leftarrow {\widehat{q}}_{t+1}$ 13:            $r_{t+1}=-1$ 14:         else if ${\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{collide}}^a$ then 15:            $q_{t+1}\leftarrow q_t$ 16:            $r_{t+1}=-1$ 17:         else if $|q_{t+1}-q_{\mathrm{goal}}|\le \eta ·\alpha$ then 18:            $r_{t+1}=0$ 19:            Terminate due to goal arrival 20:         end if 21:         Store the transition $\left(s_t\right||q_{\mathrm{goal}},a_t,r_{t+1},s_{t+1}|\left|q_{\mathrm{goal}}\right)$ in $\mathcal{D},\mathcal{L}$                           ▹ Parameters update 22:         Sample mini-batch of m transitions $\left(s_l\right||q_{\mathrm{goal}},a_l,r_{l+1},s_{l+1}|\left|q_{\mathrm{goal}}\right)$ from $\mathcal{D}$ 23:         $J_V\left(\psi \right)={\mathbb{E}}_{s_l}[\frac{1}{2}(V_{\psi }(s_l\left|\right|q_{\mathrm{goal}})-{\mathbb{E}}_{a_l}[{min}_{k=1,2}{Q}_{{\theta }_k}(s_l\left|\right|q_{\mathrm{goal}},a_l)-\beta log{\pi }_{\varphi }(a_l|s_l\left|\right|q_{\mathrm{goal}}{\left)\right])}^2]$ 24:         $J_Q\left({\theta }_{k=1,2}\right)={\mathbb{E}}_{s_l,a_l}[\frac{1}{2}(Q_{{\theta }_{k=1,2}}(s_l\left|\right|q_{\mathrm{goal}},a_l)-(r_{l+1}+V_{\overline{\psi }}(s_{l+1}\left|\right|q_{\mathrm{goal}}{\left)\right))}^2]$ 25:         $J_{\pi }\left(\varphi \right)={\mathbb{E}}_{s_l,a_l}[\beta log{\pi }_{\varphi }(a_l|s_l\left|\right|q_{\mathrm{goal}})-{min}_{k=1,2}{Q}_{{\theta }_k}(s_l\left|\right|q_{\mathrm{goal}},a_l\left)\right]$ 26:         Each network parameter $\psi ,{\theta }_{1,2},\varphi$ is updated by gradient descent 27:         using ${\nabla }_{\psi }{J}_V\left(\psi \right),{\nabla }_{{\theta }_1}{J}_Q\left({\theta }_1\right),{\nabla }_{{\theta }_2}{J}_Q\left({\theta }_2\right),{\nabla }_{\varphi }{J}_{\pi }\left(\varphi \right)$ 28:         Update state value target $\overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi }$ 29:     end for 30:     if $q_T\ne q_{\mathrm{goal}}$ then                                                ▹ HER 31:         Set additional goal $q_{\mathrm{goal}}^{\prime }\in \{q_1,q_2,\cdots ,q_T\}$ 32:         for $t=0$ to$T-1$ do 33:            Sample a transition $\left(s_t\right||q_{\mathrm{goal}},a_t,r_t,s_{t+1}|\left|q_{\mathrm{goal}}\right)$ from $\mathcal{L}$ 34:            if $|q_{t+1}-q_{\mathrm{goal}}^{\prime }|\le \eta ·\alpha$ then 35:                $r_{t+1}^{\prime }=0$ 36:            else  $r_{t+1}^{\prime }=-1$ 37:            end if 38:            Store the transition $\left(s_t\right||q_{\mathrm{goal}}^{\prime },a_t,r_{t+1}^{\prime },s_{t+1}|\left|q_{\mathrm{goal}}^{\prime }\right)$ in $\mathcal{D}$ 39:         end for 40:     end if 41: end for

4. Case Study: Two 3DOF Manipulators with a Periodically Moving Obstacle on a Line

This section presents how to implement the proposed SAC-based path training for the multi-arm manipulator with a periodically moving obstacle, and shows the result using not only simulation but also experiment.

4.1. Simulation

For the implementation of the proposed SAC-based path planning algorithm, two 3DOF open-manipulators are used. The detailed information about the manipulator can be seen at http://en.robotis.com/model/page.php?co_id=prd_openmanipulator (accessed on 14 March 2021). The entire workspace is 70 cm × 120cm × 100 cm. For the moving obstacle, a hexagon is assumed to move along a line segment back and forth periodically in the workspace, the length of the line segment is 40 cm, and the average speed of the moving obstacle is 9 cm/sec. In the simulation, it is always possible for the agent to know the location information of the obstacle. For the training, $n_d$ is set to 5 meaning that the location of the obstacle over the past five sampling times are given to the networks. Moreover, a square prism is located in the workspace as a static obstacle. The two manipulators are located such that the intersection of the workspace of both is not empty. The parameters of the robot manipulators and environment are summarized in

Table 1. Parameters of the 3DOF manipulator and environment.

Name	Value	Notation	Unit
The number of manipulator joints	3	n	pc. (Piece)
The number of manipulators	2	m	pc. (Piece)
Joint maximum	(140, −45, 150, 140, −45, 150)		${}^{\circ }$ (Degree)
Joint minimum	(−140, −180, 45, −140, −180, 45)		${}^{\circ }$ (Degree)
Dimension of ${\mathbf{Q}}_{\mathrm{free}}^a$	6	$n·m$	Dimension
The number of dynamic obstacle	1		pc. (Piece)
The number of features	5	$n_d$	Feature
The dynamic obstacle axis	y

.

The hyper-parameters of the SAC-based path planning are presented in

Table 2. Tuning parameters for the designed SAC with HER.

Name	Value	Notation	Unit
Policy network size	$12\times 800\times 500\times 400\times 400\times 300\times 6$	$\varphi$	Node
Soft Q network size	$18\times 800\times 500\times 400\times 400\times 300\times 1$	${\theta }_{1,2}$	Node
Soft value network size	$12\times 800\times 500\times 400\times 400\times 300\times 1$	$\psi ,\overline{\psi }$	Node
Learning rate	0.0001
Replay memory size	${10}^6$	$\mathcal{D}$	Buffer
Episode maximum step	100	T	Step
Soft value target copy rate	0.005	$\tau$
Mini batch size	512	m	Batch
Environment noise deviation	0.002	${ϵ}_e$
Action step size	0.3813	$\alpha$
Goal boundary	0.2	$\eta$
Discount factor	0.98	$\gamma$
Entropy temperature parameter	0.2	$\beta$

.

Figure 6 show the workspace under consideration. In Figure 6a, obstacle 1 moves back and forth along the line segment defined by $D_{\mathrm{ob}}=\left\{(x,y,z)\right|x=150\mathrm{mm},-25\mathrm{mm}\le y\le 375\mathrm{mm},\mathrm{and}z=300\mathrm{mm}\}$. The obstacle 2 depicts the static obstacle. It is assumed that the manipulator 1 can detect the obstacle only when it is in $\{x=150\mathrm{mm},0\mathrm{mm}\le y\le 175\mathrm{mm},\mathrm{and}z=300\mathrm{mm}\}$, and the manipulator 2 can do only when it is in $\{x=150\mathrm{mm},175\mathrm{mm}\le y\le 350\mathrm{mm},\mathrm{and}z=300\mathrm{mm}\}$. Namely, the manipulators can detect the moving obstacle only when it is in a predefined area. In this simulation, the location of the moving obstacle is directly extracted as the input of the network. Due to that, the network can detect the obstacle location based on the input value of the network. Figure 6b describes how the PRM deals with the moving obstacle. As seen in Figure 6b, the PRM views entire $D_{\mathrm{ob}}$ as a static obstacle to deal with the moving obstacle.

In the learning process, the five neural networks in the proposed SAC-based path planning are trained by 70,000 episodes using GPU Nvidia GeForce RTX 2080Ti and CPU Intel i7-9700F (https://www.nvidia.com, https://www.intel.com (accessed on 14 March 2021). For the training, it took 36 h and the proposed algorithm was implemented using Tensorflow 2.1. In each episode, the agent explores and learns the environment based on the previous experience that was stored in the memory. To visualize the success of the agent, the success ratio from every 10 episodes is calculated and recorded. Figure 7 depicts the success ratio from 70,000 episodes. In Figure 7, the green chart describes the actual success ratio. However, for better visualization, the moving average of the success ratio is calculated and shown as the dark green line. At the end of the training, the success ratio reaches 81.76 %.

Not only the success ratio but also the reward value is monitored during the training. Because the agent receives −1 for the reward at every step if the next configuration is not the goal, the total reward value can be interpreted as the required total time step for the agent to complete the task. The total reward of every episode is shown in Figure 8. The light blue line tells the total reward in each episode and the dark blue line describes the moving average of the total reward. From the repeated learning, similar convergent results on the success ratio and total reward are obtained. In view of Figure 7 and Figure 8, it can be concluded that the agent is well trained, which means the agent can find the optimal path for arbitrary starting and goal configurations.

In view of the learning results, the success rate of the training is not satisfactory if $n_d$ is set to a value less than 5. On the other hand, when it is higher than 5, the success rates of most learning reach a similar level to that with $n_d=$ 5. This is why $n_d$ is set to 5. To support this, Figure 9 presents the learning performance depending on various values of $n_d$.

When the training is over, the actor-network is used to generate the optimal path for an arbitrary starting and goal configurations. Figure 10 shows an example of initial and goal configurations.

Figure 11 shows how the trained actor generates the path by interacting MAMMDP${}_{\mathrm{mo}}$ and measuring the location of the moving obstacle.

To be specific, ($q_{\mathrm{goal}},q_0,D_0$) is given to the trained actor–network in the beginning and the actor–network calculates the variations of each joint denoted by $f_{\varphi }(s_t\left|\right|q_{\mathrm{goal}})$ in Figure 11. Then, each joint changes its angle by the variations and the resulting new configuration $q_1$ is generated. This is one iteration. In the second iteration, ($q_{\mathrm{goal}},q_1,D_1$) is injected into the trained actor–network with the updated obstacle information $D_1$, and the goal configuration is reached by repeating the same procedure.

One testing result is shown in Figure 12. In Figure 12, the areas in dark green denote ${\mathbf{Q}}_{\mathrm{collide}}$. In this result, it is important to note that the path generated by the proposed SCA-based planning can penetrate $D_{\mathrm{ob}}$ without collision if necessary. On the other hand, the paths by other methods go around $D_{\mathrm{ob}}$. This shows that the proposed method generates indeed an efficient path when there is a moving obstacle. For comparison, the paths computed by PRM and TD3 are also presented. Hence, there are three generated paths in Figure 12. The red lines are the resulted paths by the PRM method (10,000 sampled points graph), the blue lines by TD3 with HER, and the green lines by the proposed method path. As seen in Figure 12, the proposed SAC-based method generates the smoothest and shortest path.

To analyze quantitatively, 100 simulations with the arbitrary initial point and goal point are carried out using PRM, TD3 with HER, and the proposed method simultaneously. Then, the lengths of all 100 generated paths are given in Figure 13. As seen in Figure 13, the proposed SAC-based path planning generates the shortest path. Since, inevitably, PRM and TD3 with HER deal with the moving obstacle in a conservative manner, the result is quite natural actually. In

Table 3. Comparison of the proposed result with existing methods.

Method	Average Path Cost	Cost Percentage
PRM	4.1070	100%
SAC	3.4866	85.14%
TD3	4.2330	100.03%

, the optimal path by the proposed method is shorter by 14.86% than the path by PRM. If the path by the SAC with HER is compared with the path from TD3 with HER, the path by the proposed method is 17.40% better than that by TD3.

4.2. Experiment

In order to see if the proposed SAC-based path planning is implementable in real manipulators, it is tested using two open manipulators. Figure 14 shows the real experimental setup. In real implementation, a camera is used to detect the location of the moving obstacle and installed on the top of the workspace. The location of the moving obstacle is sampled every 0.35 s using the camera. The simulation and experimental results are displayed on the webpage https://sites.google.com/site/cdslweb/publication/dynamicsacpath (accessed on 14 March 2021). In view of the experimental results, it is confirmed that the proposed SAC-based path planning works in real multi-arm manipulators effectively.

5. Conclusions

This paper presents a deep reinforcement learning-based path planning algorithm for multi-arm manipulators with both static and periodically moving obstacles. To this end, the recently developed soft actor–critic (SAC) is employed for the deep reinforcement learning since the SAC can compute the optimal solution for the high-dimensional problem and the path planning problem for the multi-arm manipulators is essentially high dimensional. To be specific, the Markov decision process is properly defined for the path planning problem, and the input and output structure of the neural networks in the SAC is designed such that the information of the moving obstacles plays an important role in the training of the neural networks. In both the simulation and experimental results, it is confirmed that the proposed SAC-based path planning generates shorter and smoother paths compared with existing results when there is a periodically moving obstacle.

The role of the neural networks in the SAC is to estimate the future location of the moving obstacles. If the obstacle moves periodically, it might be possible to estimate the future location better using other deep learning algorithms tailored to estimation of the periodic signals like Recurrent Neural Networks (RNN). Hence, possible future research includes how to improve the estimation of the moving obstacle using deep learning algorithms for better path planning.