Deep Reinforcement Learning-Based Enhancement of Robotic Arm Target-Reaching Performance

Abstract

This work investigates the implementation of the Deep Deterministic Policy Gradient (DDPG) algorithm to enhance the target-reaching capability of the seven degree-of-freedom (7-DoF) Franka Pandarobotic arm. A simulated environment is established by employing OpenAI Gym, PyBullet, and Panda Gym. After 100,000 training time steps, the DDPG algorithm attains a success rate of 100% and an average reward of −1.8. The actor loss and critic loss values are 0.0846 and 0.00486, respectively, indicating improved decision-making and accurate value function estimations. The simulation results demonstrate the efficiency of DDPG in improving robotic arm performance, highlighting its potential for application to improve robotic arm manipulation.

1. Introduction

Enhancing the precision of robotic arm manipulation remains a core research area towards the aim of achieving full autonomy of robots in various sectors such as industrial manufacturing and assembly processes. given the maturity of image recognition and vision systems, the next logical progression in this field is to achieve complete autonomy of robotic manipulators through the use of machine learning (ML), artificial neural networks (ANNs), and artificial intelligence (AI) as a whole [1,2]. Numerous attempts have been made to create intelligent robots that can take tasks and execute them accordingly [3,4,5,6,7,8,9,10].

Although the development of a system with intelligence close to that of humans is still a long way off, robots that can perform specialized autonomous activities such as intelligent facial emotion recognition [11], flying in natural and man-made environments [12], driving a vehicle [13], swimming [14], carrying boxes and materials over different terrains [15], and picking up and placing objects [16,17] have already been realized.

However, some challenges must be overcome in order to achieve this goal. For instance, the mapping complexity from Cartesian space to the joint space of a robotic arm increases with the number of joints and linkages possessed by the manipulator. This is problematic, as the tasks assigned to a robotic arm are in Cartesian space, whereas the commands (velocity or torque) are in joint space [18,19]. Therefore, if full autonomy of robotic manipulators is the objective, then resolving the target-reaching problem represents one of the most crucial factors that must be addressed.

As described in [20,21], reinforcement learning is a type of machine learning that aims to maximize the outcome of a given system by using a dynamic and autonomous trial-and-error approach. It shares a similar objective with human intelligence, which is characterized by the ability to perceive and retain information as knowledge to be used for environment-adaptive behaviors. Trial-and-error search and delayed rewards are fundamental aspects of reinforcement leaning, which allows the learning strategy to interact with the environment by performing actions and discovering rewards [22]. Through this approach, software agents and machines can automatically select the most effective course of action to take in a given circumstance, resulting in improved performance. Reinforcement learning offers a framework and set of tools for designing sophisticated and complex robotic behaviors [23,24]. In contrast, the challenges presented by robotic issues serve as motivation, impact, and confirmation of advances in reinforcement learning. Multiple previous works on the implementation of reinforcement learning in the field of robotics depict this fact [25,26,27,28,29].

In addition to the above-mentioned studies, several different approaches to robotic manipulation have been investigated, including model-based reinforcement learning [30], supervised learning-based approaches [31], and classical control-based techniques. Although these techniques have proven effective in various settings, they often come with limitations in terms of flexibility, sample efficiency, or generalization to unseen tasks. In contrast, the proposed method makes use of reinforcement learning to improve robotic manipulators’ capacity for self-adaptation to changing surroundings. Instead of depending only on preset models or rules, reinforcement learning allows robots to learn from their experiences and improve their performance. This paper further examines the performance of our approach compared to existing reinforcement learning methods in terms of efficiency, stability, and real-world applicability.

Recent advances in RL-based robotic manipulation have focused on improving sampling efficiency, stability, and real-world generalization. The Soft Actor–Critic (SAC) algorithm is a model-free deep reinforcement learning algorithm capable of handling continuous action spaces, and has shown strong adaptability in domains such as energy-efficient building control, where it achieved a 9.7% reduction in energy costs while maintaining thermal comfort [32]. Similarly, Twin Delayed DDPG (TD3) has been improved through priority experience replay and transfer learning, significantly improving learning efficiency in mobile robot path planning, as demonstrated by increasing success rates by 16.6% and reducing training times by 23.5% [33].

Although model-based approaches such as PILCO [34] offer superior sample efficiency, they often struggle with high-dimensional control problems such as robotic manipulation. In contrast, policy-gradient methods such as PPO and DDPG remain widely used thanks to their direct interaction-based learning capabilities. However, challenges such as hyperparameter sensitivity and the exploration–exploitation tradeoff continue to persist, representing critical areas of research.

The key contributions of this work are as follows:

• Implementation of the deep deterministic policy gradient algorithm in a simulated environment to improve the target-reaching performance of a robotic arm.
• Integration of reinforcement learning with OpenAI Gym, PyBullet, and Panda Gym to create a flexible simulation framework.
• Comparison of the DDPG (off-policy) and Proximal Policy Optimization (PPO) (on-policy) algorithms to assess their efficiency, stability, and performance in robotic manipulation tasks.
• Evaluation of the performance and training behavior of the robotic agent using the DDPG and PPO algorithms in the context of a simulated target-reaching task.

This paper evaluates both algorithms based on their learning efficiency and applicability in the context of robotic manipulation simulations.

2. Modeling of Robotic Arm

Direct Kinematic Model of Robotic Arm

The rotation matrices in the Denavit–Hartenberg (DH) coordinate frame represent the rotations on the X and Z axes. The rotation matrices for these axes are respectively provided as follows:

$$R_x=\left[\begin{array}{ccc}1& 0& 0\\ 0& C{\delta }_i& -S{\delta }_i\\ 0& S{\delta }_i& C{\delta }_i\end{array}\right]$$

(1)

and

$$R_z=\left[\begin{array}{ccc}C{\varphi }_i& -S{\varphi }_i& 0\\ S{\varphi }_i& C{\varphi }_i& 0\\ 0& 0& 1\end{array}\right].$$

(2)

The homogeneous transformation matrix ($T_{i-1}^i$) that accounts for rotation and translation is provided as follows:

$$\begin{array}{ll}{T}_{i-1}^i& =\left[\begin{array}{cc}{R}_{i-1}^i& p_{i-1}^i\\ 0& 1\end{array}\right]\\ & ={\mathrm{Rot}}_x\left({\varphi }_i\right)·{\mathrm{trans}}_x\left(d_i\right)·{\mathrm{trans}}_z\left({\Delta }_i\right)·{\mathrm{Rot}}_z\left(a_i\right)\end{array}$$

$$\Rightarrow \begin{array}{l}{T}_{i-1}^i=\left[\begin{array}{cccc}C{\varphi }_i& -S{\varphi }_{i}C{\delta }_i& S{\varphi }_{i}S{\delta }_i& a_{i}C{\varphi }_i\\ S{\varphi }_i& C{\varphi }_{i}C{\delta }_i& -C{\varphi }_{i}S{\delta }_i& a_{i}S{\varphi }_i\\ 0& S{\delta }_i& C{\delta }_i& d_i\\ 0& 0& 0& 1\end{array}\right]\end{array}$$

(3)

where:

• The rotation matrix ($R_{i-1}^i$) represents the orientation of the i-th frame relative to the $(i-1)$-th frame.
• $P_{i-1}^i$ represents the center of the link frame with components ($P_x$, $P_y$, and $P_z$).

Denavit–Hartenberg (DH) Axis Representation

The four DH parameters that describe the translation and rotation relationship between two consecutive coordinate frames are as follows:

• d: Distance between the current frame and the previous frame along the Z-axis,
• ($\varphi$): Angle between the X-axis of the previous frame and the X-axis of the current frame about the previous z-axis,
• a: Distance between the Z-axes of the current and previous frames.
• $\delta$: offset of the previous frame from the current frame along the Z-axis of the current frame.

The Denavit–Hartenberg (DH) parameters for the Franka Panda robot shown in Figure 1 are provided in

Table 1. Denavit–Hartenberg axis representation.

Joint	a (m)	d (m)	$\mathit{\delta }$ (Rad)	$\mathit{\varphi }$ (Rad)
1	0	0.333	0	${\varphi }_1$
2	0	0	−90	${\varphi }_2$
3	0	0.316	90	${\varphi }_3$
4	0.0825	0	90	${\varphi }_4$
5	−0.0825	0.384	−90	${\varphi }_5$
6	0	0	90	${\varphi }_6$
7	0.088	0	90	${\varphi }_7$
Flange	0	0.107	0	0

. From the above DH parameters and based on the homogeneous transformation matrix (3), the transformation matrix for the Franka Panda robot is derived as shown below.

$$T_{i-1}^i=\left[\begin{array}{cccc}C{\varphi }_i& -S{\varphi }_{i}C{\delta }_i& S{\varphi }_{i}S{\delta }_i& a_{i}C{\varphi }_i\\ S{\varphi }_i& C{\varphi }_{i}C{\delta }_i& -C{\varphi }_{i}S{\delta }_i& a_{i}S{\varphi }_i\\ 0& S{\delta }_i& C{\delta }_i& d_i\\ 0& 0& 0& 1\end{array}\right]$$

$$T_0^1=\left[\begin{array}{cccc}C{\varphi }_1& -S{\varphi }_1& 0& 0\\ S{\varphi }_1& C{\varphi }_1& 0& 0\\ 0& 0& 1& 0.33\\ 0& 0& 0& 1\end{array}\right]$$

(4)

$$T_1^2=\left[\begin{array}{cccc}C{\varphi }_2& 0& -S{\varphi }_2& 0\\ S{\varphi }_2& 0& C{\varphi }_2& 0\\ 0& -1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(5)

$$T_2^3=\left[\begin{array}{cccc}C{\varphi }_3& 0& S{\varphi }_3& 0\\ S{\varphi }_3& 0& -C{\varphi }_3& 0\\ 0& 1& 1& 0.316\\ 0& 0& 0& 1\end{array}\right]$$

(6)

$$T_3^4=\left[\begin{array}{cccc}C{\varphi }_4& 0& S{\varphi }_4& 0.0825C{\varphi }_4\\ S{\varphi }_4& 0& -C{\varphi }_4& 0.0825S{\varphi }_4\\ 0& 1& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

(7)

$$T_4^5=\left[\begin{array}{cccc}C{\varphi }_5& 0& -S{\varphi }_5& 0.0825C{\varphi }_5\\ S{\varphi }_5& 0& C{\varphi }_5& 0.0825S{\varphi }_5\\ 0& -1& 0& 0.384\\ 0& 0& 0& 1\end{array}\right]$$

(8)

$$T_5^6=\left[\begin{array}{cccc}C{\varphi }_6& 0& S{\varphi }_6& 0\\ S{\varphi }_6& 0& -S{\varphi }_6& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(9)

$$T_6^7=\left[\begin{array}{cccc}C{\varphi }_7& 0& S{\varphi }_7& 0.088C{\varphi }_7\\ S{\varphi }_7& 0& -C{\varphi }_7& 0.088S{\varphi }_7\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(10)

$$T_7^0=T_0^1·T_1^2·T_2^3·T_3^4·T_4^5·T_5^6·T_6^7$$

(11)

$$T_7^0=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& p_x\\ r_{21}& r_{22}& r_{23}& p_y\\ r_{31}& r_{32}& r_{33}& p_z\\ 0& 0& 0& 1\end{array}\right]$$

(12)

The orientation and position of the end effector are respectively provided by

$$\begin{array}{ll}{R}_{11}& =-S{\varphi }_{1}S{\varphi }_{3}C{\varphi }_2+C{\varphi }_{1}C{\varphi }_{3}C{\varphi }_4\end{array}$$

(13)

$$\begin{array}{ll}{R}_{12}& =-S{\varphi }_{1}C{\varphi }_{2}C{\varphi }_{3}C{\varphi }_4-S{\varphi }_{3}C{\varphi }_1\end{array}$$

(14)

$$\begin{array}{ll}{R}_{13}& =S{\varphi }_{2}C{\varphi }_1\end{array}$$

(15)

$$\begin{array}{ll}{R}_{21}& =S{\varphi }_{1}C{\varphi }_{3}C{\varphi }_4+S{\varphi }_{3}C{\varphi }_{1}C{\varphi }_2\end{array}$$

(16)

$$\begin{array}{ll}{R}_{22}& =-S{\varphi }_{1}S{\varphi }_{3}C{\varphi }_2+C{\varphi }_{1}C{\varphi }_{3}C{\varphi }_4\end{array}$$

(17)

$$\begin{array}{ll}{R}_{23}& =-S{\varphi }_{2}S{\varphi }_1\end{array}$$

(18)

$$\begin{array}{ll}{R}_{31}& =-S{\varphi }_{2}C{\varphi }_{3}C{\varphi }_4\end{array}$$

(19)

$$\begin{array}{ll}{R}_{32}& =S{\varphi }_{2}S{\varphi }_{3}C{\varphi }_4\end{array}$$

(20)

$$\begin{array}{ll}{R}_{33}& =C{\varphi }_2\end{array}$$

(21)

and

$$\begin{array}{ll}{p}_x=& -0.107S{\varphi }_{2}C{\varphi }_1+0.088C{\varphi }_1\\ & +0.384\left(-S{\varphi }_{1}S{\varphi }_{3}C{\varphi }_2+C{\varphi }_{1}C{\varphi }_{3}C{\varphi }_4\right)\\ & +0.316C{\varphi }_{1}C{\varphi }_2+0.333C{\varphi }_1\end{array}$$

(22)

$$\begin{array}{ll}{p}_y=& -0.107S{\varphi }_{2}S{\varphi }_1+0.088S{\varphi }_1\\ & +0.384\left(S{\varphi }_{1}C{\varphi }_{3}C{\varphi }_4+S{\varphi }_{3}C{\varphi }_{1}C{\varphi }_2\right)\\ & +0.316S{\varphi }_{1}C{\varphi }_2+0.333S{\varphi }_1\end{array}$$

(23)

$$\begin{array}{ll}{p}_z=& 0.107C{\varphi }_2\\ & +0.384S{\varphi }_{2}C{\varphi }_{3}C{\varphi }_4\\ & +0.316S{\varphi }_2+0.333S{\varphi }_2.\end{array}$$

(24)

3. Deep Reinforcement Learning Algorithm Design

3.1. Policy Gradient Algorithm

The policy gradient theorem states that the expected return to the policy parameters can be calculated as the sum of the action value function $q^{\pi }(s,a)$ multiplied by the policy function ${\pi }_{\varrho }(s,a)$, summed over all states s and actions a and weighted by the stationary distribution of states $d^{\pi }\left(s\right)$:

$$\begin{array}{l}O\left(\varrho \right)={\displaystyle \sum _s}{d}^{\pi }\left(s\right)\sum _a{Q}^{\pi }(s,a){\pi }_{\varrho }(s,a)\end{array}$$

(25)

where:

• Objective function ($J_{\varrho }$): This represents the expected cumulative reward obtained by following the policy ${\pi }_{\varrho }$ in the given environment. The objective function is optimized by adjusting the parameter $\varrho$ to maximize the expected cumulative reward.
• Discounted state distribution ($d^{\pi }\left(s\right)$): This represents the probability of being in a particular state s under the policy $\pi$; mathematically,

  $$\begin{array}{l}{d}^{\pi }\left(s\right)=\underset{t\to \infty }{lim}P(s_t=s|s_0,{\pi }_{\varrho }),\end{array}$$

  (26)

  where $s_t=s$ when starting from $s_0$ and following policy ${\pi }_{\varrho }$ for t time steps.
• Action–value function ($Q^{\pi }(s,a)$): This represents the expected cumulative reward obtained by taking action a in state s and following policy $\pi$ thereafter.
• Policy function(${\pi }_{\varrho }(s,a)$): This represents the probability of taking action a in state s under the parameterized policy $\varrho$.

The policy gradient theorem helps to solve this problem by providing a formula for the gradient of the expected return to the policy parameters. This formula involves the stationary distribution of the Markov chain, and is provided by

$$\begin{array}{l}{\nabla }_{\varrho }{J}_{\varrho }={\displaystyle \sum _s}{d}^{\pi }\left(s\right){\displaystyle \sum _a}{q}^{\pi }(s,a){\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s),\end{array}$$

(27)

where $q^{\pi }(s,a)$ is the state–action value function for policy ${\pi }_{\varrho }$:

$$\begin{array}{l}\propto {\displaystyle \sum _{s\in S}}{d}^{\pi }\left(s\right){\displaystyle \sum _{a\in A}}{Q}^{\pi }(s,a){\nabla }_{\varrho }{\pi }_{\varrho }(s,a).\end{array}$$

(28)

3.1.1. Derivation of the Policy Gradient Theorem

The steps of the derivation using the derivative product rule of Equation (29) are as follows:

$$\begin{array}{l}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)={\nabla }_{\varrho }\left[{\displaystyle \sum _{a\in A}}{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\right],\end{array}$$

(29)

using the derivative product rule

$$\begin{array}{l}={\displaystyle \sum _{a\in A}}({\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)+{\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{Q}^{\pi }(s,a)).\end{array}$$

Step 1: Apply the derivative product rule:

$$\begin{array}{ll}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)& ={\nabla }_{\varrho }{\displaystyle \sum _{a\in A}}\left({\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\right).\end{array}$$

Step 2: Distribute the derivative operator inside the summation:

$$\begin{array}{ll}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)& ={\displaystyle \sum _{a\in A}}{\nabla }_{\varrho }\left({\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\right).\end{array}$$

Step 3: Apply the chain rule to differentiate the product of ${\pi }_{\varrho }\left(a\right|s)$ and $Q^{\pi }(s,a)$ from $\varrho$:

$$\begin{array}{ll}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)& ={\displaystyle \sum _{a\in A}}\left({\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)+{\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{Q}^{\pi }(s,a)\right).\end{array}$$

Step 4: Simplify the expression by rearranging the terms.

After applying the derivative product rule, the derivative of the state–value function $V^{\pi }\left(s\right)$ to the policy parameter $\varrho$ is

$$\begin{array}{l}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)={\displaystyle \sum _{a\in A}}\left({\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)+{\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{Q}^{\pi }(s,a)\right).\end{array}$$

(30)

We can extend $Q^{\pi }(s,a)$ by incorporating the value of the future state. This can be done by considering the state–action pair $(s,a)$ and summing over all possible future states $s^{\prime }$ and corresponding rewards r:

$$\begin{array}{ll}=& {\displaystyle \sum _{a\in A}}[{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\\ & +{\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{\displaystyle \sum _{s^{\prime },r}}P(s^{\prime },r|s,a)(r+V^{\pi }\left(s^{\prime }\right))].\end{array}$$

(31)

Because $P(s^{\prime },r|s,a)$ and r are not functions of $\varrho$, the derivative operator ${\nabla }_{\varrho }$ can be moved inside the summation over $s^{\prime },r$ without affecting these terms:

$$\begin{array}{ll}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)=& {\displaystyle \sum _{a\in A}}[{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\\ & +{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime },r}}P(s^{\prime },r|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)].\end{array}$$

(32)

Next, we can observe that ${\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)$ is the derivative of the state–value function from the policy parameter $\varrho$ at state $s^{\prime }$. This can be rewritten as ${\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)={\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right){\sum }_{s^{\prime }}P\left(s^{\prime }\right|s,a)$. Here, $P\left(s^{\prime }\right|s,a)$ represents the probability of transitioning to state $s^{\prime }$ given the current state–action pair $(s,a)$. We can now make the following substitution:

$$\begin{array}{l}={\displaystyle {\displaystyle \sum _{a\in A}}}\left[{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)+{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime },r}}P(s^{\prime },r|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\right]\end{array}$$

$$\begin{array}{l}={\displaystyle \sum _{a\in A}}\left[{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)+{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime }}}P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\right]\end{array}$$

where $P\left(s^{\prime }\right|s,a)={\sum }_{r}P\left(s^{\prime }\right|s,a)$.

Now, we have

$$\begin{array}{l}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)={\displaystyle \sum _{a\in A}}[{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\\ & +{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime }}}P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)].\end{array}$$

(33)

We can consider the following visit sequence and identify the probability of changing state s to state x using policy ${\pi }_{\varrho }$ after k steps as $\rho (s\to x,k)$:

$$s\stackrel{a\sim {\pi }_{\varrho }(.|s)}{\to }{s}^{\prime }\stackrel{a\sim {\pi }_{\varrho }(.|s^{\prime })}{\to }{s}^{″}\stackrel{a\sim {\pi }_{\varrho }(.|s^{″})}{\to }\cdots$$

• For $k=0:\rho (s\to s,k=0)=1$, and
• For $k=1$, we can consider every action that might be taken and add up the probabilities of reaching the desired state:

  $$\begin{array}{l}\rho (s\to x,k=1)={\displaystyle \sum _a}{\pi }_{\varrho }\left(a\right|s)P\left(s^{\prime }\right|s,a).\end{array}$$

  (34)
• The goal is to move from s to x after $k+1$ steps by following ${\pi }_{\varrho }$. The agent can first move from s to intermediate state $s^{\prime }(s^{\prime }\in S)$ before proceeding to the final state x in the last steps after k stages. This allows us to recursively update the visitation probability as

  $$\begin{array}{l}\rho (s\to x,k+1)={\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s\to s^{\prime },k)\rho (s^{\prime }\to x,1).\end{array}$$

  (35)

After dealing with the probability $\rho (s\to x,k)$ for transitioning from state s to state x after a certain number of steps k, the next step is to drive a recursive formulation for ${\nabla }_{\varrho }{V}^{\pi }\left(s\right)$.

To accomplish this, a function $\varphi \left(s\right)$ is introduced and defined as follows:

$$\begin{array}{l}\varphi \left(s\right)={\displaystyle \sum _{a\in A}}{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\end{array}$$

(36)

where $\Phi \left(s\right)$ represents the sum of the gradients of the policy ${\pi }_{\varrho }$ to $\varrho$, weighted by the corresponding action–value function $Q^{\pi }(s,a)$.

To simplify,

$$\begin{array}{l}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)=\Phi \left(s\right)+\left[{\displaystyle \sum _{a\in A}}{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime }}}P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\right]\end{array}$$

(37)

$$\begin{array}{l}=\varphi \left(s\right)+{\displaystyle \sum _a}{\pi }_{\varrho }\left(a\right|s){\displaystyle \sum _{s^{\prime }}}P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\end{array}$$

(38)

$$\begin{array}{l}=\Phi \left(s\right)+{\displaystyle \sum _{s^{\prime }}}{\displaystyle \sum _{a\in A}}{\pi }_{\varrho }\left(a\right|s)P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\end{array}$$

(39)

$$\begin{array}{l}=\Phi \left(s\right)+{\displaystyle \sum _{s^{\prime }}}{\displaystyle \sum _{a\in A}}{\pi }_{\varrho }\left(a\right|s)P\left(s^{\prime }\right|s,a){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\end{array}$$

(40)

$$\begin{array}{l}=\Phi \left(s\right)+{\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s\to s^{\prime },1){\nabla }_{\varrho }{V}^{\pi }\left(s^{\prime }\right)\end{array}$$

(41)

$$\begin{array}{ll}& =\Phi \left(s\right)+{\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s\to s^{\prime },1)\left[\varphi \left(s^{\prime }\right)+{\displaystyle \sum _{s^{″}}}{\rho }^{\pi }(s^{\prime }\to s^{″},1){\nabla }_{\varrho }{V}^{\pi }\left(s^{″}\right)\right].\end{array}$$

Considering $s^{\prime }$ as the middle point for $s^{\prime }\to s^{″}$,

$$\begin{array}{ll}=& \Phi \left(s\right)+\left[{\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s\to s^{\prime },1)\varphi \left(s\right)\right]\\ & +\left[{\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s^{\prime }\to s^{″},2){\nabla }_{\varrho }{V}^{\pi }\left(s^{″}\right)\right].\end{array}$$

(42)

The expression for ${\nabla }_{\varrho }{V}^{\pi }\left(s\right)$ can be unrolled as follows:

$$\begin{array}{ll}=& \Phi \left(s\right)+\left[{\displaystyle \sum _{s^{\prime }}}{\rho }^{\pi }(s\to s^{\prime },1)\Phi \left(s^{\prime }\right)\right]\\ & +\left[{\displaystyle \sum _{s^{″}}}{\rho }^{\pi }(s\to s^{″},2)\Phi \left(s^{″}\right)\right]\\ & +\left[{\displaystyle \sum _{s^{‴}}}{\rho }^{\pi }(s\to s^{‴},3){\nabla }_{\varrho }{V}^{\pi }\left(s^{‴}\right)\right]\end{array}$$

(43)

$$\begin{array}{l}{\nabla }_{\varrho }{V}^{\pi }\left(s\right)={\displaystyle \sum _{x\in S}}{\displaystyle \sum _{k=0}}{\rho }^{\pi }(s\to x,k)\Phi \left(x\right).\end{array}$$

(44)

Eliminating the derivatives of the Q-value function ${\nabla }_{\varrho }{Q}^{\pi }\left(s\right)$ and inserting objective function $O\left(\varrho \right)$, starting from state $S_0$,

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)={\nabla }_{\varrho }{V}^{\pi }\left(S_0\right),\end{array}$$

(45)

$$\begin{array}{l}={\displaystyle \sum _s}{\displaystyle \sum _{k=0}}{\rho }^{\pi }(s\to x,k)\varphi \left(s\right).\end{array}$$

(46)

Letting $\eta \left(s\right)={\sum }_s{\rho }^{\pi }(s,\to x,k)$,

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)={\displaystyle \sum _s}\eta \left(s\right)\varphi \left(s\right).\end{array}$$

(47)

Substituting $\eta \left(s\right)={\sum }_s{\rho }^{\pi }(s,\to x,k)$ into Equation (47),

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)={\displaystyle \sum _s}\eta \left(s\right)\varphi \left(s\right).\end{array}$$

(48)

We can normalize $\eta \left(s\right)$, $s\in S$ to be the probability distribution, as shown in

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)=\left({\displaystyle \sum _s}\eta \left(s\right)\right){\displaystyle \sum _s}\left[{\displaystyle \frac{\eta \left(s\right)}{{\sum }_s\eta \left(s\right)}}\varphi \left(s\right)\right].\end{array}$$

(49)

Because ${\sum }_s\eta \left(s\right)$ is constant, the gradient of the objective function is proportional to the normalized $\eta \left(s\right)$ and $\varphi \left(s\right)$:

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)\propto {\displaystyle \sum _s}\left[{\displaystyle \frac{\eta \left(s\right)}{{\sum }_s\eta \left(s\right)}}\varphi \left(s\right)\right]\end{array}$$

(50)

where $d^{\pi }\left(s\right)={\displaystyle \frac{\eta \left(s\right)}{{\sum }_s\eta \left(s\right)}}$ is a stationary distribution.

In the episodic case, the constant proportionality ${\sum }_s\eta \left(s\right)$ is the average length of an episode, while in the continuing case it is 1 [36]:

$$\begin{array}{l}={\displaystyle \sum _s}{d}^{\pi }\left(s\right){\displaystyle \sum _{a\in A}}{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)Q^{\pi }(s,a)\end{array}$$

(51)

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)\propto {\displaystyle \sum _{s\in S}}{d}^{\pi }\left(s\right){\displaystyle \sum _{a\in A}}{Q}^{\pi }(s,a){\nabla }_{\varrho }\pi \left(\varrho \right)\left(a\right|s)\end{array}$$

(52)

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)={\displaystyle \sum _{s\in S}}{d}^{\pi }\left(s\right){\displaystyle \frac{{\nabla }_{\varrho }{\pi }_{(}\varrho )\left(a\right|s)}{{\pi }_{(}\varrho )\left(a\right|s)}}\end{array}$$

(53)

$$\begin{array}{l}={\displaystyle \sum _{s\in S}}{d}^{\pi }\left(s\right){\displaystyle \frac{{\nabla }_{\varrho }{\pi }_{(}\varrho )\left(a\right|s)}{{\pi }_{(}\varrho )\left(a\right|s)}}\end{array}$$

(54)

$$\begin{array}{l}{\nabla }_{\varrho }O\left(\varrho \right)=E_{\pi }\{Q^{\pi }(s,a){\nabla }_{\varrho }ln\left({\pi }_{\varrho }\left(a\right|s)\right)\}\end{array}$$

(55)

where $E_{\pi }\mathrm{refers}\mathrm{to}{E}_{s\sim d_{\pi },a\sim {\pi }_{\varrho }}$ when the distribution of the states and actions follows policy ${\pi }_{\varrho }$ (on-policy).

3.1.2. Off-Policy Gradient Algorithms

Because DDPG is an off-policy gradient algorithm, we first discuss this type of policy gradient algorithm in more detail.

The behavior policy for collecting samples is known and labeled as $\alpha \left(a\right|s)$. The objective function sums up the reward over the state distribution defined by this behavior policy:

$$O\left(\varrho \right)={\displaystyle \sum _{s\in \mathcal{S}}}{d}^{\alpha }\left(s\right){\displaystyle \sum _{a\in \mathcal{A}}}{Q}^{\pi }(s,a){\pi }_{\varrho }\left(a\right|s)$$

$$={\mathbb{E}}_{s\sim d^{\alpha }}\left[{\displaystyle \sum _{a\in \mathcal{A}}}{Q}^{\pi }(s,a){\pi }_{\varrho }\left(a\right|s)\right]$$

where $d^{\alpha }\left(s\right)$ is the stationary distribution of behavior policy $\alpha$, as $d^{\alpha }\left(s\right)={lim}_{t\to \infty }P(S_t=s|S_0,\alpha )$ and $Q^{\pi }$ is the action–value function estimated about the target policy $\pi$. Given that the training observations are sampled by $a\sim \alpha \left(a\right|s)$, we can rewrite the gradient as

$$\begin{array}{ll}{\nabla }_{\varrho }O\left(\varrho \right)& ={\nabla }_{\varrho }{\mathbb{E}}_{s\sim d^{\alpha }}\left[{\displaystyle \sum _{a\in \mathcal{A}}}{Q}^{\pi }(s,a){\pi }_{\varrho }\left(a\right|s)\right]\end{array}$$

(56)

by using the derivative product rule

$$\begin{array}{l}={\mathbb{E}}_{s\sim d^{\alpha }}\left[{\displaystyle \sum _{a\in \mathcal{A}}}\left(Q^{\pi }(s,a){\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)+{\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{Q}^{\pi }(s,a)\right)\right]\end{array}$$

(57)

and ignoring ${\pi }_{\varrho }\left(a\right|s){\nabla }_{\varrho }{Q}^{\pi }(s,a)$:

$$\begin{array}{l}\stackrel{\left(i\right)}{\approx }{\mathbb{E}}_{s\sim d^{\alpha }}\left[{\displaystyle \sum _{a\in \mathcal{A}}}{Q}^{\pi }(s,a){\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)\right]\end{array}$$

(58)

$$\begin{array}{l}={\mathbb{E}}_{s\sim d^{\alpha }}\left[{\displaystyle \sum _{a\in \mathcal{A}}}\alpha \left(a\right|s){\displaystyle \frac{{\pi }_{\varrho }\left(a\right|s)}{\alpha \left(a\right|s)}}{Q}^{\pi }(s,a){\displaystyle \frac{{\nabla }_{\varrho }{\pi }_{\varrho }\left(a\right|s)}{{\pi }_{\varrho }\left(a\right|s)}}\right]\end{array}$$

(59)

$$\begin{array}{l}={\mathbb{E}}_{\alpha }\left[{\displaystyle \frac{{\pi }_{\varrho }\left(a\right|s)}{\alpha \left(a\right|s)}}{Q}^{\pi }(s,a){\nabla }_{\varrho }ln{\pi }_{\varrho }\left(a\right|s)\right]\end{array}$$

(60)

where $\left[{\displaystyle \frac{{\pi }_{\varrho }(a\mid s)}{\alpha (a\mid s)}}\right]$ is the importance weight. Because $Q^{\pi }$ is a function of the target policy, and consequently a function of the policy parameter $\varrho$, the derivative ${\nabla }_{\varrho }{Q}^{\pi }(s,a)$ must also be computed using the product rule. However, in practice it is challenging to compute ${\nabla }_{\varrho }{Q}^{\pi }(s,a)$ directly. Fortunately, policy improvement can still be guaranteed by approximating the gradient and ignoring the gradient of $Q^{\pi }$, which achieves eventual convergence to the true local minimum.

In summary, when applying the policy gradient in the off-policy setting, it can be adjusted using a weighted sum, where the weight is the ratio of the target policy to the behavior policy, $\left[{\displaystyle \frac{{\pi }_{\varrho }\left(a\right|s)}{\alpha \left(a\right|s)}}\right]$ [36].

3.2. Deterministic Policy Gradient (DPG)

The policy function $\pi (·|s)$ is typically represented as a probability distribution over actions (A) based on the current state, making it inherently stochastic. However, in the case of the DPG, the policy is modeled as a deterministic decision, denoted as $a=\mu \left(s\right)$. Instead of selecting actions probabilistically, the DPG directly maps states to specific actions without uncertainty. Let

• ${\rho }_0\left(s\right)$ be the initial distribution over states,
• ${\rho }^{\mu }(s\to s^{\prime },k)$ be the density of visitation probability in state $s^{\prime }$ when starting from state s after moving k steps by policy $\mu$, and
• ${\rho }^{\mu }\left(s^{\prime }\right)$ be the discounted state distribution, defined as
  ${\rho }^{\mu }\left(s^{\prime }\right)={\int }_{\mathcal{S}}{\sum }_{k=1}^{\infty }{\gamma }^{k-1}{\rho }_0\left(s\right){\rho }^{\mu }(s\to s^{\prime },k)d_s$.

Then, the objective function to be optimized is

$$\begin{array}{l}O\left(\varrho \right)={\int }_{\mathcal{S}}{\rho }^{\mu }\left(s\right)Q(s,{\mu }_{\varrho }\left(s\right))ds.\end{array}$$

(61)

According to the chain rule, we first take the gradient of Q with respect to action a, then take the gradient of the deterministic policy function $\mu$ with respect to $\varrho$:

$$\begin{array}{ll}{\nabla }_{\varrho }O\left(\varrho \right)& ={\int }_{\mathcal{S}}{\rho }^{\mu }\left(s\right){\nabla }_a{Q}^{\mu }(s,a){\nabla }_{\varrho }{\mu }_{\varrho }{\left(s\right)|}_{a={\mu }_{\varrho }\left(s\right)}ds,\end{array}$$

(62)

$$\begin{array}{ll}& ={\mathbb{E}}_{s\sim {\rho }^{\mu }}{\left[{\nabla }_a{Q}^{\mu }(s,a){\nabla }_{\varrho }{\mu }_{\varrho }\left(s\right)\right|}_{a={\mu }_{\varrho }\left(s\right)}].\end{array}$$

(63)

3.3. Deep Deterministic Policy Gradient (DDPG)

By combining DQN and DPG, DDPG leverages the power of deep neural networks to handle high-dimensional state spaces and complex action spaces, making it suitable for a wide range of reinforcement learning tasks. The original DQN works in discrete space; DDPG extends it to continuous space with the actor–critic framework while learning a deterministic policy. To improve exploration, an exploration policy ${\mu }^{\prime }$ is constructed by adding noise $\mathcal{N}$:

$$\begin{array}{l}{\mu }^{\prime }\left(s\right)={\mu }_{\varrho }\left(s\right)+\mathcal{N}.\end{array}$$

(64)

Moreover, the DDPG algorithm integrates a sophisticated technique known as soft updates, also called conservative policy iteration, to update the parameters of both the actor and critic networks. This revised methodology utilizes a small parameter, denoted $\tau$, which is much smaller than 1 ($\tau \ll 1$).

The soft update equation is formulated as follows:

$$\begin{array}{l}{\varrho }^{\prime }\leftarrow \tau \varrho +(1-\tau ){\varrho }^{\prime }.\end{array}$$

(65)

Unlike the approach employed in the DQN, this guarantees that the target network values are gradually altered over time when the target network remains static for a fixed period.

As shown in Algorithm 1, the DDPG algorithm follows an actor-critic framework where the critic network estimates the action-value function, while the actor network learns a deterministic policy. The training process begins with initializing the networks and reply buffer. During each episode, actions are selected using the current policy with added exploration noise, and the resulting transition is stored in the reply buffer. The critic is updated by minimizing the loss function, and the actor is optimized using the policy gradient. Furthermore, the target networks for both the actor and the critic are updated using soft updates to ensure stability in learning [37].

Algorithm 1: Deep Deterministic Policy Gradient [37]

4. Results and Discussion

4.1. Simulation Environment and Training Setup

The deep reinforcement learning agent was trained using Python on a Jupyter Notebook using a Linux Ubuntu 20.04 operating system. The training process spanned approximately 5.415 h on a system with an Intel graphics card, 8 GB of RAM, and a 1.9 GHz processor.

To simulate the target reaching task of the Franka Panda robotic arm, we utilized the PandaReach-v2 environment from the panda_gym library, which is built on PyBullet. This physics-based environment enables realistic interactions for reinforcement learning.

To train the Deep Deterministic Policy Gradient (DDPG) algorithm, Stable Baselines3, an optimized deep-reinforcement learning library was used. The training process spanned 100,000 time steps, allowing the robotic agent to learn an optimal policy for precise target reaching. The integration of OpenAI Gym, PyBullet, and Panda Gym provided an efficient and flexible simulation framework to evaluate reinforcement learning techniques in robotic manipulation.

4.2. Hyperparameter Selection and Initial Search

The parameters listed in

Table 2. Parameters and Hyperparameters.

Parameter	Value
Policy	MultiInputPolicy
Replay buffer class	HerReplayBuffer
Verbose	1
Gamma	0.95
Tau ($\tau$)	0.005
Batch size	512, 1024, 2048
Buffer size	100,000
Replay buffer kwargs	rb kwargs
Learning rate	1e-3, 2e-4
Action noise	Normal action noise
Policy kwargs	Policy kwargs
Tensorboard log	Log path

were selected for this work.

4.2.1. Batch Size Comparison

Upon completion of the training process, it was observed that there was a minimal disparity in the success rate (Figure 2) and the cumulative reward (Figure 3) between the different batch sizes. Furthermore, the decrease in critic loss values (Figure 4) and actor loss values (Figure 5) indicated an improvement in the ability of both the actor and critic networks to approximate the optimal policy and value functions. Although the success rate and cumulative reward were similar, the enhanced convergence demonstrated by the lower losses with a batch size of 2048 suggests a more efficient learning process and a potentially higher quality of the policies learned within this setting. However, as Figure 6 shows, the training speed varied widely with batch sizes, with 512 being the fastest, 1024 second, and 2048 last. Despite its slower training speed, a batch size of 2048 was used because it achieved better stability and convergence. Larger batch sizes generally reduce gradient variance, leading to a smoother optimization process and improved overall model quality.

4.2.2. Learning Rate Comparison

After the training process, the observations shown in Figure 7 and Figure 8 revealed a minimal disparity in the cumulative reward and the success rate achieved between the two learning rates. However, the learning rate of 2e-4 shows a slightly better success rate compared to the learning rate of 1e-3. In contrast, when using the learning rate of 1e-3, a notable decrease in both actor loss (Figure 9) and critic loss (Figure 10) was observed, indicating improved policy and value estimation. Despite comparable cumulative reward and success rates, the reduced losses with the learning rate of 1e-3 signified enhanced convergence and a potentially more efficient learning process, suggesting that the agent may have acquired higher-quality learned policies.

4.3. Selection of Optimal Hyperparameters and Extended Training of DDPG Agent

After comparing the hyperparameters and performing a detailed analysis of the associated results, we selected the hyperparameters with the most promising performance as shown in

Table 3. Selected parameters and hyperparameters.

Parameter	Value
Policy	MultiInputPolicy
Replay buffer class	HerReplayBuffer
Verbose	1
Gamma	0.95
Tau ($\tau$)	0.005
Batch size	2048
Buffer size	100,000
Replay buffer kwargs	rb kwargs
Learning rate	1e-3
Action noise	Normal action noise
Policy kwargs	Policy kwargs

. Following this, an extended training phase was initiated that included 100,000 time steps as shown in

Table 4. Training metrics at time steps of 200 and at $\mathrm{100,000}$.

Category	Value	Category	Value
rollout/		rollout/
Episode length	50
Episode mean reward	−49.2	Episode mean reward	−1.8
Success rate	0	Success rate	1
time/		time/
Episodes	4	Episodes	2000
FPS	18	FPS	5
Time elapsed	10	Time elapsed	19,505
Total timesteps	200	Total time steps	100,000
train/		train/
Actor loss	0.625	Actor loss	0.0846
Critic loss	0.401	Critic loss	0.00486
Learning rate	0.001	Learning rate	0.001
Number of updates	50	Number of updates	99,850

. This extended training phase served as the fundamental training stage, which is elaborated in the following section.

4.3.1. Improvement in Cumulative Reward and Success Rate

The mean episode reward (Figure 11) improved from −49.2 in the first loop to −1.8 in the last loop, while the success rate (Figure 12) increased from 0 in the first loop to 1 in the last loop.

4.3.2. Frames per Second (FPS)

The training speed (Figure 13) decreased from 18 frames per second (FPS) in the first loop to 5 FPS in the last loop.

4.3.3. Improvement in Actor and Critic Losses

The actor loss (Figure 14) decreased from 0.625 in the first loop to 0.0846 in the last loop, while the critic loss (Figure 15) decreased from 0.401 in the first loop to 0.00486 in the last loop.

4.4. Comparing DDPG and PPO: Off-Policy vs. On-Policy Reinforcement Learning Algorithms

We trained the Proximal Policy Optimization (PPO) on-policy reinforcement learning algorithm and compared its performance with the Deep Deterministic Policy Gradient (DDPG) off-policy algorithm. The results are shown in Figure 16 and Figure 17. As shown in Figure 16, the cumulative reward achieved by DDPG was $-1.8$, while the cumulative reward obtained by PPO was $-50$. The results of this comparison indicate that DDPG shows better performance in this particular scenario than PPO in terms of cumulative reward.

5. Conclusions

In this study, we applied the Deep Deterministic Policy Gradient (DDPG) algorithm to train a robotic arm manipulator, specifically the Franka Panda robotic arm, for a target-reaching task. The objective of this task was to enable the robotic arm to accurately reach a designated target position. The DDPG algorithm was chosen because of its effectiveness in continuous control tasks and its ability to learn policies with high-dimensional action spaces. Using a combination of deep neural networks and an actor–critic architecture, the DDPG algorithm was able to approximate the optimal policy for the robotic arm. When comparing the performance of PPO and DDPG algorithms after training for 100,000 time steps, PPO achieved a mean episode reward of $-50$ indicating that the agent struggled to achieve positive rewards on average. Despite training at a relatively fast speed of $561FPS$, these results suggest that the PPO algorithm faced challenges in finding successful strategies for the given task.

However, the DDPG algorithm demonstrated superior performance, with a mean episode reward of $-1.8$. It achieved a success rate of 1, indicating consistent success in achieving the desired outcomes. Despite a slower training speed of $5FPS$, the DDPG algorithm showcased the ability to learn and improve its policy effectively over time. Based on these results, DDPG outperformed PPO in terms of cumulative reward and success rate in the given scenario.

From the above results, it can be concluded that PPO performs worse than DDPG, mainly because of the way each algorithm learns. PPO is a policy-based method, which means that it learns only from its most recent interactions and ignores past experiences. Although this promotes stability, it makes learning less effective because the agent constantly needs new data to improve. However, DDPG is an off-policy algorithm, meaning that it stores past experiences in a replay buffer and continues to learn them. This makes training more sample-efficient, which is especially useful for complex tasks such as robotic arm control.

Another key difference is how the two algorithms update their policies. PPO uses a clipping mechanism to prevent large policy changes, which helps with stability but can slow down learning, especially in high-dimensional action spaces. Meanwhile, DDPG uses deterministic updates, allowing it to fine-tune actions more precisely.

Although DDPG had a much slower training speed, it ultimately learned better strategies, leading to a higher success rate and better rewards compared to PPO.