Deep Deterministic Policy Gradient-Based Parameter Adaptation for Synchronous Sliding-Mode Control with Time-Delay Estimation in Dual-Arm Robot Manipulators Under System Uncertainties

Abstract

This paper presents a synchronous sliding-mode control with time-delay estimation (SSMC-TDE)-based adaptive control framework for coordinated motion control of dual-arm robotic manipulators operating under system uncertainties. The baseline SSMC-TDE scheme is constructed using synchronization and cross-coupling errors to ensure precise coordinated motion among robot joints, while sliding-mode control effectively handles strong nonlinearities, and the time-delay estimation technique approximates lumped uncertainties arising from external disturbances, modeling errors, and payload variations. The stability of the closed-loop system is rigorously analyzed and guaranteed using the Lyapunov theory. To overcome performance degradation caused by manually tuned control gains, a deep reinforcement learning-assisted parameter adaptation mechanism is integrated into the SSMC-TDE structure. Specifically, a Deep Deterministic Policy Gradient (DDPG) algorithm is employed to adapt selected control gains online through a reward function designed to simultaneously enhance motion synchronization and reduce trajectory-tracking errors, while preserving the stability properties of the underlying controller. Simulation studies are conducted within a co-simulation framework integrating MATLAB/Simulink and ROS/Gazebo for a dual-arm robotic platform. Quantitative evaluations based on the root mean square error (RMSE) of trajectory-tracking and synchronization errors across all six joints demonstrate that, averaged over both scenarios, the proposed DDPG-assisted SSMC-TDE achieves an overall RMSE reduction of 35.52% and 99.3% compared with conventional SSMC and SSMC-TDE controllers, respectively, confirming its superior performance and robustness under system uncertainties.

1. Introduction

Dual-arm robot manipulators have become an indispensable technology in advanced robotic systems, enabling collaborative operations such as cooperative transportation, precision assembly, and the dexterous manipulation of heavy, deformable, or fragile objects in industrial, service-oriented, and safety-critical environments [1,2]. Their ability to emulate human bimanual coordination allows two manipulators to jointly regulate motion, force, and interaction dynamics, thereby achieving superior dexterity and robustness compared with single-arm systems [3,4]. Despite these advantages, achieving high-performance coordinated control remains challenging due to the nonlinear and strongly coupled nature of dual-arm dynamics, where Coriolis and centrifugal effects, gravitational forces, and unmodeled interaction phenomena introduce significant uncertainties [1,5]. Moreover, practical disturbances such as payload variations, joint friction, sensor noise, and external perturbations further degrade synchronization accuracy and tracking precision [6].

To mitigate these difficulties, a broad spectrum of synchronous coordination strategies has been investigated. Impedance and admittance controllers offer compliant interaction but often suffer from reduced accuracy under fast or highly nonlinear motions [7,8,9]. Adaptive and passivity-based approaches enhance robustness to parametric uncertainties but typically depend on restrictive excitation conditions or conservative energy constraints that limit operational flexibility [9,10]. In contrast, sliding-mode control has proven to be an effective nonlinear control strategy and has been successfully implemented in various practical engineering systems, such as grid-connected power electronic converters and motor drive applications, demonstrating strong robustness against uncertainties and external disturbances [11]. Nevertheless, sliding-mode-based synchronous control provides strong disturbance rejection but remains sensitive to fixed design parameters, often resulting in pronounced chattering or degraded transient behavior when operating across varying environmental and interaction conditions [12]. Consequently, ensuring fast convergence, high synchronization fidelity, and strong robustness in dual-arm systems remains an open challenge.

Synchronous sliding-mode control (SSMC) has been widely adopted because it directly incorporates cross-coupling error structures that enforce coordinated behavior between the two manipulators, leading to asymptotic convergence of both trajectory-tracking and synchronization errors [13]. When combined with time-delay estimation (TDE), the SSMC-TDE framework reconstructs lumped uncertainties from delayed measurements, reducing reliance on accurate dynamic modeling and increasing resilience to unknown disturbances and unmodeled dynamics [14,15]. Lyapunov-based analysis has rigorously established that the incorporation of TDE preserves closed-loop stability under bounded approximation errors, ensuring robust synchronization even in the presence of substantial uncertainties [16,17]. Nevertheless, the overall performance of SSMC-TDE highly depends on the designer-selected sliding structure, which governs convergence characteristics, error coupling behavior, and chattering attenuation. Inadequate selection of this structure may impair tracking quality, weaken synchronization, or cause excessive actuator effort issues that are further exacerbated under time-varying disturbances, friction changes, and unpredictable interaction dynamics [18,19]. Manual parameterization, while often sufficient in simulation studies, lacks adaptability and cannot guarantee consistent performance across diverse real-world scenarios [20,21]. These limitations highlight the need for an autonomous online adaptation mechanism capable of continuously refining controller behavior in response to evolving system conditions.

In recent years, deep reinforcement learning (DRL) has demonstrated strong potential for enhancing adaptability in robotic control. Unlike manually tuned controllers, DRL algorithms such as DDPG can autonomously adjust control parameters through continuous interaction with the environment. DRL has been successfully applied to sliding-mode-based quadrotor control [22], robot manipulation and adaptive interaction learning [23], optimization of PID-type controllers under nonlinear disturbances [24], and adaptive sliding-mode gain tuning for multibody robotic systems [25]. These works consistently show that DRL improves robustness, tracking accuracy, and adaptability beyond what conventional manual parameterization can achieve.

Recently, fixed-time and predefined-time control strategies, including adaptive neural network-based fixed-time control [26,27] and prescribed performance-based reliable control [28,29], have also been proposed to guarantee convergence within a fixed or predefined time bound that is independent of initial conditions. These methods ensure that tracking errors evolve within prescribed performance limits, even in the presence of uncertainties, disturbances, and actuator constraints. However, such approaches typically rely on predefined convergence-time specifications, prescribed performance functions, and relatively complex controller architectures involving neural networks, terminal sliding modes, or barrier Lyapunov functions. As a result, their flexibility may be limited when system dynamics, interaction conditions, or task requirements vary significantly.

In parallel, advanced disturbance rejection and learning-based control strategies have also been investigated for uncertain nonlinear robotic systems [30,31,32,33]. State-filtered disturbance rejection control methods achieve effective disturbance attenuation by explicitly estimating and compensating disturbances through filter-based observers; however, their performance is closely tied to filter design and measurement quality, and the resulting controller complexity increases with system dimensionality. Multilayer neuroadaptive reinforcement learning approaches based on actor–critic architectures embed neural networks directly into the control loop to approximate unknown dynamics or control policies [34,35,36], which may complicate stability analysis and require additional excitation or boundedness assumptions.

In contrast to the above methods, this study adopts a stability-preserving, learning-assisted control framework in which SSMC-TDE serves as a robust baseline controller, while DRL is employed solely as an auxiliary mechanism for online parameter adaptation [37,38]. This separation allows Lyapunov-based stability guarantees to be retained while leveraging the learning capability of DRL to enhance synchronization performance, transient response, and control smoothness under complex uncertainties, without introducing additional disturbance observers or neural approximators into the control law.

According to the above discussion, this paper proposes a DRL-enhanced control framework that integrates the DDPG algorithm into a stability-guaranteed SSMC-TDE architecture. Leveraging an actor–critic structure with target networks, the DDPG agent autonomously adjusts critical controller parameters in real time, enabling the system to maintain high synchronization accuracy, improve tracking precision, and reduce chattering, all without requiring prior knowledge of the robot dynamics. A task-specific reward function is formulated to jointly promote robust coordination, transient performance, and smooth interaction. The main contributions of this work are threefold:

(1)

A DRL-assisted parameter adaptation framework integrated with a baseline SSMC–TDE controller, enabling online performance-oriented gain adjustment for synchronous dual-arm robotic manipulation under significant uncertainties;

(2)

Lyapunov-based stability analysis demonstrating that the integration of the DRL adaptation layer preserves the stability properties of the underlying SSMC-TDE control structure;

(3)

Comprehensive co-simulation studies validating improved robustness, convergence performance, and synchronization accuracy compared with fixed-parameter baseline controllers.

The remainder of the paper is organized as follows: Section 2 formulates the problem and describes the dual-arm system dynamics; Section 3 details the SSMC-TDE controller and its stability; Section 4 presents the DRL-based adaptation using DDPG; Section 5 reports simulation results; and Section 6 concludes the paper.

2. Dynamic Modeling and Problem Formulation

2.1. Dual-Arm Robot Dynamics

The dynamics equations of the $i$-th arm $\left(i=1,2\right)$ of the dual-arm (see Figure 1) in the joint space are expressed as follows [1,5,39]:

$$M_i\left(q_i\right){\ddot{q}}_i+C_i\left(q_i,{\dot{q}}_i\right){\dot{q}}_i+g_i\left(q_i\right)+d_i={\tau }_i$$

(1)

where $M_i\left(q_i\right)\in {ℝ}^{3\times 3}$ is the positive definite symmetric inertia matrix of the $i$-th arm. $C_i\left(q_i,{\dot{q}}_i\right)\in {ℝ}^{3\times 3}$ is the Coriolis and centrifugal force matrix. $g_i\left(q_i\right)\in {ℝ}^{3\times 1}$ denotes the gravitational vector acting on the $i$-th arm. $d_i\in {ℝ}^{3\times 1}$ is the vector representing uncertainties, including parameter variations and friction. ${\tau }_i={\left[{\tau }_{1i},{\tau }_{2i},{\tau }_{3i}\right]}^T$ is the vector of joint torques generated at the joints of the $i$-th arm. The vectors $q_i$, ${\dot{q}}_i$, and ${\ddot{q}}_i$ denote, respectively, the joint positions, velocities, and accelerations of the joints of the $i$-th arm, with ${\left(·\right)}_i={\left[{(·)}_{1i},{(·)}_{2i},{(·)}_{3i}\right]}^T$. From Equation (1), the dynamic model of the dual-arm system can be rewritten as:

$$M\left(q\right)\ddot{q}+C\left(q,\dot{q}\right)\dot{q}+g\left(q\right)+d=\tau$$

(2)

where $M\left(q\right)=blkdiag\left(M_1\left(q_1\right),M_2\left(q_2\right)\right)\in {ℝ}^{6\times 6}$ is the combined inertia matrix of the two arms; similarly $C\left(q,\dot{q}\right)=blkdiag\left(C_1\left(q_1,{\dot{q}}_1\right),C_2\left(q_2,{\dot{q}}_2\right)\right)\in {ℝ}^{6\times 6}$; $g\left(q\right)={\left[g_1{\left(q_1\right)}^T,g_2{\left(q_2\right)}^T\right]}^T\in {ℝ}^{6\times 1}$; $d={\left[d_1^T,d_2^T\right]}^T\in {ℝ}^{6\times 1}$; $\tau ={\left[{\tau }_1^T,{\tau }_2^T\right]}^T\in {ℝ}^{6\times 1}$; $q={\left[q_1^T,q_2^T\right]}^T={\left[q_{11},q_{21},q_{31},q_{12},q_{22},q_{32}\right]}^T$; $\dot{q}={\left[{\dot{q}}_1^T,{\dot{q}}_2^T\right]}^T={\left[{\dot{q}}_{11},{\dot{q}}_{21},{\dot{q}}_{31},{\dot{q}}_{12},{\dot{q}}_{22},{\dot{q}}_{32}\right]}^T$; and $\ddot{q}={\left[{\ddot{q}}_1^T,{\ddot{q}}_2^T\right]}^T={\left[{\ddot{q}}_{11},{\ddot{q}}_{21},{\ddot{q}}_{31},{\ddot{q}}_{12},{\ddot{q}}_{22},{\ddot{q}}_{32}\right]}^T$.

Property 1:

$M\left(q\right)$ is a positive definite matrix and always invertible for $\forall q$.

Property 2:

$\left(M\left(q\right)-2C\left(q,\dot{q}\right)\right)$ is a skew-symmetric matrix that satisfies $x^T\left({\dot{M}}_D\left(q_D\right)-2C_D\left({\dot{q}}_D,q_D\right)\right)x=0$.

The dynamic Equation (2) is derived based on the Euler–Lagrange method. However, owing to discrepancies between the mathematical model and the actual system, it can be expressed as follows:

$$M_0\left(q\right)\ddot{q}+C_0\left(q,\dot{q}\right)\dot{q}+g_0\left(q\right)+\Delta M\left(q\right)\ddot{q}+\Delta C\left(q,\dot{q}\right)\dot{q}+\Delta g\left(g\right)+d=\tau$$

(3)

where $M_0\left(q\right),C_0\left(q,\dot{q}\right)$ and $G_0\left(q\right)$ represent the nominal dynamic components, which are derived from the Euler–Lagrange formulation based on known robot parameters; and unmodeled components $\Delta M\left(q\right),\Delta C\left(q,\dot{q}\right)$ and $\Delta G\left(q\right)$ represent the unmodeled dynamic components.

2.2. Synchronous Coordination Error

Consider a dual-arm robotic system performing a cooperative manipulation task involving a heavy object. Let the desired joint trajectories of the two arms be $x_d=\left[{q_{1d}}^T,{q_{2d}}^T\right]\in {ℝ}^{6\times 1}$ and $x=\left[{q_1}^T,{q_2}^T\right]\in {ℝ}^{6\times 1}$ be the vector of actual joint trajectories measured from the robot’s joints. The joint tracking error vector $e={\left[e_1^T,e_2^T\right]}^T\in {ℝ}^{6\times 1}$ is defined as:

$$e=x-x_d$$

(4)

To determine the capability for synchronous motion between the two arms, the synchronization error vector $e_s={\left[e_{s_1}^T,e_{s_2}^T\right]}^T\in {ℝ}^{6\times 1}$ is defined as:

$$e_s=Te$$

(5)

where $T=\left[I_{3\times 3},-I_{3\times 3};-I_{3\times 3},I_{3\times 3}\right]\in {ℝ}^{6\times 6}$ is the synchronization matrix, and $I\in {ℝ}^{3\times 3}$ denotes the identity matrix. The cross-coupling error vector $e_c\in {ℝ}^{6\times 1}$ is defined from the tracking error $e$ and the synchronization error $e_s$:

$$e_c=e+\alpha e_s=\left(I+\alpha T\right)e$$

(6)

where $\alpha =diag\left({\alpha }_1,\dots ,{\alpha }_6\right)\in {ℝ}^{6\times 6}$ is a positive definite diagonal matrix, and when $\alpha$ is chosen appropriately $\left(0<{\alpha }_i\right)$, the matrix $\left(I+\alpha T\right)$ becomes positive definite and full rank, i.e., $\left|\left(I+\alpha T\right)\right|\ne 0$. In this case, if the cross-coupling error $e_c\to 0$, then tracking error $e\to 0$, according to (6). This further supports the synchronization error $e_s\to 0$ according to (5), meaning that $e_1=e_2$, thereby achieving synchronous motion between the two arms. In summary, the control objective of synchronous motion between the two arms is achieved when the cross-coupling error, $e_c$, asymptotically converges to 0.

2.3. Root Mean Square Error

To evaluate the quality of the tracking errors as well as the effectiveness of the trained agent, the RMSE [40] criterion is employed, which is defined as follows:

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\left(S_i-O_i\right)}^2}}$$

(7)

where $n$ denotes the total number of samples. $S_i$ represents the $i$-th setpoint value, and $O_i$ corresponds to the system output (feedback) at the $i$-th sample.

3. Synchronous Sliding-Mode Control with Time-Delay Estimation

3.1. Control Design

Figure 2 depicts the structure of the SSMC-TDE for coordinated motion control of a dual-arm robot. The controller integrates synchronous (cross-coupling) control, which enforces coordination between the two manipulators, sliding-mode control to address strong nonlinearities, and TDE to compensate for lumped uncertainties caused by modeling errors, external disturbances, and payload variations. This integrated control framework ensures robust and precise coordinated performance for dual-arm robotic systems. The detailed design procedure of the SSMC-TDE controller is presented below.

Define a diagonal positive definite matrix $\overline{M}\in R^{6\times 6}$. The dynamic Equation (3) of the dual-arm robot can then be written as:

$$M_0\left(q\right)\ddot{q}+C_0\left(q,\dot{q}\right)\dot{q}+g_0\left(q\right)+\overline{M}\ddot{q}+\left[\Delta M\left(q\right)-\overline{M}\right]\ddot{q}+\Delta C\left(q,\dot{q}\right)\dot{q}+\Delta g\left(q\right)+d=\tau$$

(8)

Let $f\left(q,\dot{q},\ddot{q}\right)$ be the nominal dynamic components of the robot and $\Delta \left(q,\dot{q},\ddot{q}\right)$ be the lumped uncertainty components of the model. Equation (8) is written concisely as follows:

$$\overline{M}\ddot{q}+f\left(q,\dot{q},\ddot{q}\right)+\Delta \left(q,\dot{q},\ddot{q}\right)=\tau$$

(9)

Assume that the signals $\Delta \left(q,\dot{q},\ddot{q}\right)$ in (9) are continuously measured over a sampling interval of sufficiently small duration $L$ (typically the sensor sampling period). Thus, the actual value of $\Delta \left(q,\dot{q},\ddot{q}\right)$ at time $t$ can be approximated $\Delta \left(q,\dot{q},\ddot{q}\right)$ at time $t-L$:

$$\Delta {\left(q,\dot{q},\ddot{q}\right)}_{\left(t\right)}\cong \Delta {\left(q,\dot{q},\ddot{q}\right)}_{\left(t-L\right)}$$

(10)

Equations (9) and (10), with $\widehat{\Delta }\left(q,\dot{q},\ddot{q}\right)$ denoting the time-delay estimation (TDE) value, yield:

$$\widehat{\Delta }{\left(q,\dot{q},\ddot{q}\right)}_{\left(t\right)}\cong \Delta {\left(q,\dot{q},\ddot{q}\right)}_{\left(t-L\right)}={\tau }_{\left(t-L\right)}-\overline{M}{\ddot{q}}_{\left(t-L\right)}-f{\left(q,\dot{q},\ddot{q}\right)}_{\left(t-L\right)}$$

(11)

where ${\ddot{q}}_{\left(t-L\right)}$ is computed using the following finite-difference approximation:

$${\ddot{q}}_{t-L}={\displaystyle \frac{q_{\left(t\right)}-2q_{\left(t-L\right)}+q_{\left(t-2L\right)}}{L^2}}$$

(12)

Assumption 1.

When the value $L$ is sufficient, the TDE approximation error is bounded by a constant $\epsilon$:

$${‖\Delta \left(q,\dot{q},\ddot{q}\right)-\widehat{\Delta }\left(q,\dot{q},\ddot{q}\right)‖}_{\infty }\le \epsilon$$

(13)

Based on (8) and (9), let $x_1=q,x_2=\dot{q},$ and $x_d=q_d$, the dynamic model of the robot manipulator is expressed in state–space form as follows:

$$\left\{\begin{array}{c}{\dot{x}}_1=x_2\hfill \\ {\dot{x}}_2={\tilde{M}}^{-1}\left[u-C_0\left(x_1,x_2\right)x_2-g_0\left(x_1\right)-\Delta \left(x_1,x_2,{\dot{x}}_2\right)\right]\hfill \\ y=x_1\hfill \end{array}\right.$$

(14)

where $\tilde{M}=M_0\left(x_1\right)+\overline{M}$, $x_1={\left[{q_{i1}}^T,{q_{i2}}^T\right]}^T\in R^{6\times 1},i=1,2,3$, and $u=\tau$ is the control input.

The sliding variable $s\in {ℝ}^{6\times 1}$ is chosen based on the cross-coupling error $e_c$ as:

$$\begin{array}{c}s={\dot{e}}_c+\lambda e_c\hfill \\ =\left(I+\alpha T\right)\left(\dot{e}+\lambda e\right)\hfill \end{array}$$

(15)

where $\lambda =diag\left(\left[{\lambda }_1,\dots ,{\lambda }_6\right]\right)\in {ℝ}^{6\times 6}$ is a positive definite diagonal matrix. The derivative of the sliding variable (15) is computed as:

$$\begin{array}{c}\dot{s}={\ddot{e}}_c+\lambda {\dot{e}}_c\hfill \\ =\left(I+\alpha T\right)\left({\dot{x}}_2-{\ddot{x}}_d+\lambda \dot{e}\right)\hfill \end{array}$$

(16)

Substituting ${\dot{x}}_2$ from (14) into (16) results in:

$$\dot{s}=\left(I+\alpha T\right)\left[{\tilde{M}}^{-1}\left[u-C_0\left(x_1,x_2\right)x_2-g_0\left(x_1\right)-\Delta \left(x_1,x_2,{\dot{x}}_2\right)\right]-{\ddot{x}}_d+\lambda \dot{e}\right]$$

(17)

The control law of the synchronous sliding mode controller is designed as:

$$u_{SSMC}=u_{eq}+u_r$$

(18)

By choosing $\dot{s}=-\left(I+\alpha T\right){\tilde{M}}^{-1}Ks$ when the system uncertainties are neglected ($\Delta \in R^{6\times 1}$ is assumed to be zero), the equivalent control input $u_{eq}$ is determined as:

$$u_{eq}=\tilde{M}\left({\ddot{x}}_d-\lambda \dot{e}\right)+C_0\left(x_1,x_2\right)x_2+g_0\left(x_1\right)-Ks$$

(19)

The robust control component $u_r$ is defined as [31,41,42]:

$$u_r=-\eta \mathrm{sgn}\left(s\right)$$

(20)

where $\eta \in {ℝ}^{6\times 6}$ is a positive diagonal matrix satisfying $\eta \ge {‖d‖}_{\infty }$, which ensures the robustness of the system in the presence of disturbances.

Remark 1.

Since the sign function is inherently discontinuous, chattering naturally occurs when the sliding surface $s$ crosses zero, which can potentially damage the mechanical actuators in practice. To mitigate this issue in simulations, the sign function in Equation (20) has been replaced with a continuous saturation function [43,44], effectively smoothing the switching behavior and reducing high-frequency oscillations in the control input.

Combining (19) and (20), the control law in (18) becomes:

$$u_{SSMC}=\tilde{M}\left({\ddot{x}}_d-\lambda \dot{e}\right)+C_0\left(x_1,x_2\right)x_2+g_0\left(x_1\right)-Ks-\eta \mathrm{sgn}\left(s\right)$$

(21)

To estimate the uncertain components $\Delta \left(x_1,x_2,{\dot{x}}_2\right)$ acting on the system, the TDE method is employed as in (11). Therefore, the synchronous sliding-mode control law combined with TDE is expressed as:

$$u_T=u_{SSMC}+u_{TDE}$$

(22)

Based on (21) and (11), the SSMC-TDE control law in (18) is obtained as:

$$u_T=\tilde{M}\left({\ddot{x}}_d-\lambda \dot{e}\right)+C_0\left(x_1,x_2\right)x_2+g_0\left(x_1\right)-Ks-\eta \mathrm{sgn}\left(s\right)+\widehat{\Delta }\left(x_1,x_2,{\dot{x}}_2\right)$$

(23)

Substituting (23) into (17) results in:

$$\dot{s}={\tilde{M}}^{-1}\left(I+\alpha T\right)\left(\widehat{\Delta }-\Delta -Ks-\eta \mathrm{sgn}\left(s\right)\right)$$

(24)

3.2. Stability Analysis

A Lyapunov function is selected as:

$$V={\displaystyle \frac{1}{2}}{s}^{T}s$$

(25)

The time derivative of (25) is:

$$\dot{V}=s^T\dot{s}$$

(26)

Replacing (24) with (26) yields:

$$\dot{V}=s^T\left[{\tilde{M}}^{-1}\left(I+\alpha T\right)\left(\widehat{\Delta }-\Delta -Ks-\eta \mathrm{sgn}\left(s\right)\right)\right]$$

(27)

From Assumption 1, it follows that:

$$\dot{V}=-s^T\left[{\tilde{M}}^{-1}\left(I+\alpha T\right)\left(Ks+\eta \mathrm{sgn}\left(s\right)-\epsilon \right)\right]$$

(28)

Remark 2.

Since the matrix $\tilde{M}=M_0\left(x_1\right)+\overline{M}>0$, and the synchronization coefficient vector $\alpha$ is chosen to be strictly positive, the matrix is $\left(I +\alpha T\right)>0$. In addition, the gain matrix $K$ is positive definite, implying that $-s^T\left[{\tilde{M}}^{-1}\left(I+\alpha T\right)Ks\right]\le 0,\forall s$. To ensure that (28) is strictly negative, the following condition must be satisfied:

$$\eta >\left|\epsilon \right|$$

(29)

If the time-delay estimation error $\epsilon$ is bounded within a finite region, the stability condition (29) guarantees that the time derivative of the Lyapunov function remains negative semi-definite. Consequently, the closed-loop system preserves Lyapunov stability, and the tracking errors are ensured to remain uniformly bounded within a neighborhood of the origin.

4. DRL-Based Online Parameter Adaptation (DDPG)

4.1. Online Parameter Adaptive with SSMC-TDE

4.1.1. State, Action, and Reward Design

The DDPG algorithm [45] is employed to achieve online adaptation of the control gains in the SSMC–TDE controller for the dual-arm robot. As an actor–critic, off-policy algorithm, DDPG interacts with the system during training, using the robot states and tracking errors as observations, and control gains as continuous actions. The training process is guided by a reward function designed to penalize tracking errors, synchronization errors, and excessive control effort. The overall framework is illustrated in Figure 3. Due to its off-policy nature, DDPG enables policy updates using previously collected samples, allowing the training process to be conducted offline without requiring an explicit model of robot dynamics.

To formulate the reinforcement learning problem, the interaction between the DDPG agent and the SSMC–TDE-controlled dual-arm robot is defined in terms of the state, action, and reward. The state vector is constructed to capture both tracking and synchronization performance and is defined as:

$$s_o={\left[e,e_s\right]}^T\in R^{18\times 1}$$

(30)

The action generated by the DDPG agent corresponds to the continuous control gain parameters of the SSMC–TDE controller and is expressed as:

$$a={\left[{\lambda }_n,{\eta }_n,K_n,{\overline{M}}_n,{\alpha }_n\right]}^T\in R^{15\times 1}$$

(31)

where ${\lambda }_n,{\eta }_n$ and $K_n$ are updated to adjust the corresponding sliding-mode control gains, while ${\alpha }_n$ is used to update the synchronization gain $\alpha$ and ${\overline{M}}_n$ is employed to update the gain $\overline{M}$ of the TDE module.

To achieve accurate trajectory tracking, improved synchronization, and controlled input magnitude in the dual-arm system, the reward function is designed to penalize tracking errors, synchronization errors, and control effort:

$$r={\displaystyle \sum _{i=1}^6\left(-w_e\left|e\left(i\right)\right|-w_s\left|e_s\left(i\right)\right|-w_u\left|u\left(i\right)\right|\right)}$$

(32)

where $e\left(i\right)$ and $e_s\left(i\right)$ denote the trajectory-tracking error and synchronization error, respectively. $u\left(i\right)$ represents the control input magnitude. The weighting coefficients $w_e,w_s$ and $w_u\in {ℝ}^{6\times 6}$ are selected to balance tracking accuracy, synchronization performance, control effort, and smoothness.

The continuous actions generated by the DDPG agent are normalized and constrained within a compact interval to ensure stability-preserving adaptation (see the update control gains block in Figure 3). Specifically, each action component is first saturated within $\left[-1,1\right]$, and the corresponding control gains $k_i\left(t\right)$ are updated using a bounded multiplicative adaptation law based on predefined nominal values:

$$k_i\left(t\right)=k_{i0}\left(1+\beta a_i\left(t\right)\right)$$

(33)

where $k_{i0}$ denotes the initial stabilizing gain obtained from the baseline SSMC–TDE controller, $\beta \in \left(0,1\right)$ is a scaling factor, and $a_i\left(t\right)$ is the normalized output of the actor network. As a result, the adapted gains satisfy the following bounds:

$$\left(1-\beta \right)k_{i0}\le k_i\left(t\right)\le \left(1+\beta \right)k_{i0}$$

(34)

which guarantees that all control parameters remain strictly positive and bounded at all times.

After the training phase, only the trained actor network is retained and deployed online as an RL-based gain tuning module, while the critic network is discarded. As shown in Figure 4, the trained actor network adaptively adjusts the SSMC-TDE control gains in real time based on the current system states, thereby enhancing robustness and adaptability during execution.

4.1.2. DDPG Training Setup in MATLAB Simulink

The DDPG algorithm is an off-policy actor–critic method. Accordingly, the tuning of the SSMC–TDE controller parameters using DDPG is conducted offline in the MATLAB/Simulink environment, as illustrated in Figure 5. The RL Agent block is implemented using the Reinforcement Learning Toolbox, where the training environment is configured with the hyperparameters summarized in

Table 1. Hyperparameters of the DDPG algorithm for agent training.

Define	Symbol	Tuning Parameter
Sample time	$T_S$	0.05
Smooth factor	$\phi$	0.001
Reward discount factor	$\gamma$	0.99
Learning rate for actor	${\alpha }_a$	0.0001
Learning rate for critic	${\alpha }_c$	0.001
Minibatch size	$N_b$	128
Experience buffer length	R	${10}^6$

and the actor and critic networks are initialized with the architectures shown in Figure 6 and Figure 7, respectively. Based on the observed system states, joint tracking errors of the dual-arm robot, and the accumulated reward following each action, the agent generates continuous actions to update the control gain parameters of the SSMC-TDE controller.

The training process is performed on a workstation equipped with an Intel Core i7-10750H CPU (2.6 GHz), 16 GB RAM, and an NVIDIA GeForce GTX 1650 Ti GPU. A circular reference trajectory is defined within the workspace of the dual-arm robot, as described by (35), to enable the agent to learn appropriate responses to external disturbances. Each training episode is simulated for 20 s with a sampling period of 0.01 s, using a fixed-step solver configuration and the ode4 (Runge–Kutta) integration method:

$${\mathrm{X}}_m^d\left(t\right)=\left[\begin{array}{l}{x}_m^d\left(t\right)\hfill \\ y_m^d\left(t\right)\hfill \\ {\gamma }_m^d\left(t\right)\hfill \end{array}\right]=\left[\begin{array}{l}0.08\cos \left(0.2\pi t+{\displaystyle \frac{\pi }{2}}\right)\hfill \\ 0.08\sin \left(0.2\pi t+{\displaystyle \frac{\pi }{2}}\right)+0.2\hfill \\ {\displaystyle \frac{\pi }{9}}\sin \left(0.2\pi t\right)\hfill \end{array}\right]$$

(35)

To enhance the robustness of the learned policy, different disturbance scenarios are randomly applied during the training process. Specifically, an external force of 20 N is applied along the y-direction to joint 2 of Arm 1 from 10 s to 10.5 s, while a force of −20 N is simultaneously applied to joint 2 of Arm 2. Alternatively, a varying payload disturbance is introduced, where a payload of 1 kg is applied to joint 2 of both robot arms during the first 10 s and increased to 3 kg from 10 s to 20 s. In addition, model uncertainties are emulated through friction effects, including viscous friction and Coulomb friction, which are modeled as follows:

$${\tau }_{\mathrm{fric}}\left(t\right)=F_v\dot{q}+F_c\mathrm{sign}\left(\dot{q}\right)$$

(36)

Different friction parameters are employed depending on the disturbance type. When external forces are applied, the viscous and Coulomb friction vectors are set to ${\mathbf{F}}_{\mathbf{v}}=diag\left(\left[\mathrm{0.05,0.05,0.2,0.05,0.05,0.2}\right]\right)$ and ${\mathbf{F}}_{\mathbf{c}}=diag\left(\left[\mathrm{0.3,0.2,0.1,0.3,0.2,0.1}\right]\right)$ respectively. When payload variations are introduced, the corresponding friction coefficients are assigned as ${\mathbf{F}}_{\mathbf{v}}=diag\left(\left[\mathrm{0.05,0.05,0.0}.\mathrm{5,0.05,0.05,0.05}\right]\right)$ and ${\mathbf{F}}_{\mathbf{c}}=diag\left(\left[\mathrm{0.25,0.15,0.05,0.25,0.15,0.05}\right]\right)$.

4.2. DDPG Framework

In this subsection, the DDPG algorithm adopted for training the proposed RL-based gain tuning module is described in detail in Figure 8. The formulation of the actor–critic architecture, the learning process, and the overall training procedure are presented to clarify how the control gain parameters of the SSMC–TDE controller are optimized.

DDPG is an off-policy actor–critic reinforcement learning algorithm designed for continuous action spaces. It employs two neural networks: an actor network $\mu \left(\delta | {\varphi }^{\mu }\right)$, which represents a deterministic policy mapping the system states $\delta$ to a continuous action, and a critic network $Q\left(\delta ,a | {\varphi }^Q\right)$, which approximates the action–value function. In this study, the system state is constructed to reflect the tracking and coordination performance of the dual-arm robot, while the action corresponds to the continuous control gain parameters of the SSMC–TDE controller. During training, the critic network is updated by minimizing the temporal-difference (TD) error between the predicted Q-value and the target value. For a sampled transition $\left({\delta }_i,a_i,r_i,{\delta }_{i+1}\right)$ from the replay buffer, the target value is computed as:

$$y_i=r_i+\gamma Q^{\prime }\left({\delta }_{i+1},{\mu }^{\prime }\left({\delta }_{i+1} | {\varphi }^{u\prime }\right) | {\varphi }^{Q\prime }\right)$$

(37)

where $Q^{\prime }$ and $\mu \prime$ denote the target critic and actor networks, respectively, and $\gamma \in \left(0,1\right)$ is a discount factor. The critic parameters ${\varphi }^Q$ are updated by minimizing the loss function:

$$L_f={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\left(y_i-Q\left({\delta }_i,a_i | {\varphi }^Q\right)\right)}^2}$$

(38)

The actor network is trained to maximize the expected return by following the deterministic policy gradient. The gradient of the performance objective with respect to the actor parameters is approximated as:

$${\nabla }_{{\varphi }^{\mu }}J\approx {\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\nabla }_{a}Q\left(\delta ,a | {\varphi }^Q\right) {|}_{\delta ={\delta }_i,a=\mu \left({\delta }_i\right)}\nabla {\varphi }^{\mu }\mu \left(\delta | {\varphi }^{\mu }\right) {|}_{\delta ={\delta }_i}}$$

(39)

Through this update mechanism, the actor learns to generate control gain parameters that improve the overall tracking and synchronization performance of the dual-arm robot. To enhance training stability and data efficiency, an experience replay buffer is employed to store previously observed transitions. Mini-batches are randomly sampled from the replay buffer during training, which helps break temporal correlations between consecutive samples. In addition, target networks are introduced for both the actor and the critic to stabilize learning. The target network parameters are updated using a soft update scheme given by:

$$\begin{array}{l}{\varphi }^{Q\prime }\leftarrow \nu {\varphi }^Q+\left(1-\nu \right){\varphi }^{Q\prime }\\ {\varphi }^{\mu \prime }\leftarrow \nu {\varphi }^{\mu }+\left(1-\nu \right){\varphi }^{\mu \prime }\end{array}$$

(40)

with $\nu \in \left(0,1\right)$ as a smoothing factor.

Exploration during training is achieved by adding a stochastic noise process $\mathcal{N}$ to the actor output, enabling sufficient exploration of the continuous action space. The complete training procedure of the DDPG algorithm is summarized in Algorithm 1, including network initialization, experience collection, parameter updates, and target network soft updates. After convergence, the trained actor network is retained and deployed online to adapt the SSMC-TDE control gains in real time.

Algorithm 1: Deep Deterministic Policy Gradient algorithm

Initialize the Actor $\mu$ and Critic $Q$ networks along with their corresponding target networks ${\mu }^{\prime }$ and $Q^{\prime }$, and create an experience replay buffer. for i = 1, episode_max do   Initialize a random noise process for action exploration   Receive initial state ${\delta }_1$   for j = 1, J do   Select action $a_j=\mu \left({\delta }_j | {\varphi }^{\mu }\right)+N$ based on the current policy with added exploration noise.   Execute action $a_j$ and observe reward $r_j$ and observe new state ${\delta }_{j+1}$.   Store transition $\left({\delta }_j,a_j,r_j,{\delta }_{j+1}\right)$ in $R$.   Sample a random minibatch of $N_b$ transitions from $R$.   Set $y_j$ as Equation (37).   Update the critic by minimizing the loss function $L_f$ given in (38).   Update the actor policy using the sampled policy gradient defined in (39).   Update the target networks expressed in (40).   end end

5. Simulation and Co-Simulation Results

5.1. Simulation Environment

A co-simulation framework between MATLAB/Simulink and Gazebo is established, as illustrated in Figure 9, to support agent training and to validate the agent’s adaptability under various system uncertainties. The co-simulation environment consists of a Windows-based physical computer running MATLAB/Simulink with the Robotics System Toolbox and the Gazebo Co-Simulation package, together with a VMware Workstation virtual machine running Ubuntu with ROS and Gazebo installed. The overall procedure for constructing the co-simulation environment for the dual-arm robot is shown in Figure 10. The virtual machine performs the 3D simulation of the dual-arm robot in Gazebo, as depicted in Figure 11, where the physical environment such as external forces, payload variations, and friction is realistically emulated. Meanwhile, the physical machine is responsible for generating reference trajectories, executing the control algorithms, and collecting feedback data from Gazebo (see Figure 12).

Figure 13 illustrates the training performance of the agent using the DDPG algorithm within a co-simulation framework integrating MATLAB/Simulink and Gazebo. The results indicate that after approximately 300 training episodes, the average reward converges to a steady value of about −250. Based on this outcome, the trained actor network is extracted and employed as an RL-based gain tuner to adaptively adjust the parameters of the SSMC-TDE controller in response to environmental disturbances during coordinated motion tasks of the dual-arm robotic system. To validate the effectiveness of the proposed approach, two simulation scenarios are conducted using the trained agent to track predefined trajectories (see

Table 2. Summary of simulation scenarios and verification objectives.

Scenario	Uncertainty/Disturbance	Description	Verification Purpose
Scenario 1	External force disturbance	±10 N applied to joint 21 and 22 (8–9 s)	Robustness to external disturbances
Scenario 2	Payload variation	Payload increases from 0.5 kg (0–8 s) → 1 kg (8–15 s) → 1.5 kg (15–20 s)	Adaptability to time-varying payloads

).

During the simulation phase, the friction effects are modeled using the same parameters as those used during training. In addition, the reference trajectory, simulation duration, and sampling time are kept identical to the training setup to ensure a fair and consistent evaluation environment. The control gain parameters of the SSMC and SSMC-TDE controllers employed in the simulations are summarized in

Table 3. The control coefficients of the two controllers SSMC and SSMCTDE.

Controller	Control Coefficient
SSMC	$\begin{array}{l}\lambda =diag\left(\left[10,18,40,10,18,40\right]\right);\\ K=diag\left(\left[0.3,0.12,0.12,0.3,0.12,0.12\right]\right);\\ \eta =diag\left(\left[0.15,0.15,0.05,0.15,0.15,0.05\right]\right);\\ \alpha =diag\left(\left[0.4,0.4,0.4,0.4,0.4,0.4\right]\right).\end{array}$
SSMC-TDE	$\begin{array}{l}\lambda =diag\left(\left[10,18,40,10,18,40\right]\right);\\ K=diag\left(\left[0.3,0.12,0.12,0.3,0.12,0.12\right]\right);\\ \eta =diag\left(\left[0.15,0.15,0.05,0.15,0.15,0.05\right]\right);\\ \alpha =diag\left(\left[0.4,0.4,0.4,0.4,0.4,0.4\right]\right);\\ \overline{M}=diag\left(\left[{10}^{-4},{10}^{-4},{10}^{-4},{10}^{-4},{10}^{-4},{10}^{-4}\right]\right).\end{array}$

. Finally, all simulation results, including comparative evaluations between the proposed controller and the conventional SSMC and SSMC-TDE controllers with fixed control gains, are recorded and presented in the subsequent Figures and Tables.

Remark 3.

In order to improve tracking accuracy, enhance synchronization among the joints, and counteract the effects of nonlinearities such as model uncertainties, external disturbances, and friction, the parameters of the SSMC controller are selected through a trial-and-error procedure subject to the Lyapunov stability conditions presented in Appendix A. The coefficient matrices $\lambda$ and $K$ are chosen in a manner analogous to the proportional and derivative gains of a conventional PD controller. The matrix $\eta$ is gradually tuned from small to large values based on the resulting tracking performance and the oscillation amplitude of the control signal. Meanwhile, the synchronization gain matrix $\alpha$ is selected such that the synchronization error vector $e_s$ converges to a neighborhood of zero. To ensure a fair comparison, the parameters of the SSMC-TDE controller are chosen to be identical to those of the SSMC controller. In addition, the diagonal elements of the matrix $\overline{M}$ are selected to be sufficiently small compared to unity. This choice is motivated by the fact that $\overline{M}$ directly affects the estimation capability of the TDE scheme for compensating nonlinear system dynamics in the control input. If $\overline{M}$ is chosen too large, the system may exhibit oscillatory behavior and potentially become unstable. Finally, for the proposed RLSSMC-TDE approach, the controller parameters are adaptively tuned via reinforcement learning using the DDPG algorithm, with the initial coefficient matrices selected to be identical to those of the SSMC-TDE controller.

5.2. Simulation Results and Discussions

5.2.1. Scenario 1

Figure 14 illustrates the joint tracking errors of both arms of the dual-arm robot. The black dashed line, blue dashed line, and red dashed line represent the responses of the conventional SSMC, the SSMC-TDE controller, and the proposed RLSSMC-TDE controller, respectively. As observed in the Figure, compared with the conventional SSMC, both SSMC-TDE and the proposed RLSSMC-TDE achieve significantly smaller tracking errors, which can be attributed to the time-delay estimation mechanism that compensates for lumped uncertainties arising from unmodeled dynamics and external disturbances. This confirms the effectiveness of TDE in enhancing baseline tracking performance. More importantly, when external forces are applied from 8 s to 9 s to joint 2 of Arm 1 (see Figure 14b) and Arm 2 (see Figure 14e), the benefit of reinforcement learning-based gain adaptation becomes evident. Unlike the fixed-gain SSMC-TDE, the RLSSMC-TDE dynamically adjusts its control gains in response to sudden disturbance-induced error variations, resulting in faster error attenuation and reduced peak tracking deviations. This adaptive behavior explains why the proposed controller consistently achieves lower RMSE values, as reported in

Table 4. Root mean square tracking error at each joint of the dual-arm robot.

Controllers	Joint 11	Joint 21	Joint 31	Joint 12	Joint 22	Joint 32
SSMC	0.0234	0.0586	0.1071	0.0518	0.098	0.2268
SSMC-TDE	0.001	0.0016	0.0028	0.0011	0.0018	0.0032
Proposed method	0.0006	0.001	0.0018	0.0007	0.0014	0.0018

, despite the already strong baseline performance of SSMC-TDE. In addition, the synchronization performance is further illustrated in Figure 15, where opposite external forces are applied to the two arms. Although all controllers restore synchronization after the disturbance, the RLSSMC-TDE maintains smaller transient synchronization errors, indicating enhanced coordination robustness under asymmetric disturbances. The corresponding control inputs in Figure 16 reflect the inherent switching behavior of sliding-mode control, while the smoother error responses achieved by the proposed method result from adaptive gain regulation rather than aggressive suppression of the switching action. The compensation signals in Figure 17 further demonstrate that TDE effectively reconstructs the lumped uncertainties, allowing the learning-based adaptation to focus on performance refinement rather than disturbance rejection.

Table 4 presents the root mean square tracking errors of the end-effector trajectories of the dual-arm robot. The numerical results clearly indicate that the proposed controller achieves superior control performance, improved trajectory-tracking accuracy, and enhanced synchronization capability compared with the conventional SSMC and SSMC-TDE methods.

5.2.2. Scenario 2

The joint trajectory-tracking errors of the dual-arm robot in Scenario 2 are shown in Figure 18. Compared with the conventional SSMC and fixed-gain SSMC-TDE controllers, the proposed RLSSMC-TDE controller achieves improved tracking accuracy, particularly in instances when payload changes occur, as illustrated in Figure 18b,c. This behavior indicates that the reinforcement learning-based gain adaptation enables the controller to respond effectively to abrupt variations in payload-induced dynamics, which cannot be adequately compensated by manually tuned control gains. The influence of time-varying payloads on synchronization performance is depicted in Figure 19. Although all three controllers preserve synchronization in steady states, the proposed RLSSMC-TDE consistently exhibits smaller transient synchronization errors during payload transitions. This improvement demonstrates enhanced robustness of the proposed approach against asymmetric and time-varying uncertainties affecting the dual-arm system. Similar to Scenario 1, the corresponding control inputs and time-delay estimation compensation signals are presented in Figure 20 and Figure 21, respectively. The control inputs retain the characteristic switching behavior of sliding-mode control, while the compensation signals confirm the effectiveness of the TDE mechanism in reconstructing lumped uncertainties. The quantitative RMSE results summarized in

Table 5. The root mean square tracking error at each joint of the robot.

Controllers	Joint 11	Joint 21	Joint 31	Joint 12	Joint 22	Joint 32
SSMC	0.0699	0.5788	0.1078	0.0864	0.5874	0.2277
SSMC-TDE	0.0007	0.0018	0.0027	0.0012	0.002	0.0029
Proposed method	0.0004	0.0012	0.0018	0.0007	0.0014	0.0019

further support these observations, showing consistent performance improvements across all joints. Overall, these results validate the effectiveness and robustness of the proposed RLSSMC-TDE controller for dual-arm robotic systems operating under model uncertainties, friction effects, external disturbances, and time-varying payloads.

6. Conclusions

This paper presents a DRL-enhanced SSMC-TDE framework for dual-arm robot manipulators operating under significant uncertainties. By integrating the DDPG algorithm, the controller autonomously adapts the sliding surface and robust gains online, thereby improving synchronization accuracy, trajectory-tracking performance, and chattering suppression across a wide range of operating conditions. Quantitative simulation results demonstrate that, compared with conventional SSMC-TDE and SSMC schemes, the proposed method reduces the average RMSE of trajectory-tracking and synchronization errors by 35.52% and 99.3%, respectively, across all six joints in two representative scenarios. Lyapunov-based analysis guarantees the stability of the closed-loop system under nonlinear dynamics, payload variations, and external disturbances. Furthermore, MATLAB/Simulink-ROS/Gazebo co-simulations confirm smoother control actions and reduced control effort.

Future work will focus on hardware validation, multi-modal sensing integration, extensions to multi-arm systems, and exploration of alternative DRL algorithms and digital-twin-based training to enhance real-world performance. In addition, inspired by recent studies on imperfect dynamical systems [46,47], future research will investigate whether certain nonlinear imperfections can be systematically exploited to improve adaptability, robustness, or cooperative behavior in multi-robot manipulation tasks.