Centralized Reinforcement Learning for Shared-Target Tracking of Dual Soft Robotic Manipulators with B-Spline-Parameterized Distributed Actuation

Abstract

Coordinated control of multiple soft robotic manipulators is challenging because continuous deformation, distributed actuation, and nonlinear dynamics increase both modeling and policy-learning complexity. This paper proposes a simulation-based reinforcement learning framework for coordinated shared-target tracking using two soft manipulators. Each arm is modeled as a three-dimensional shearable Cosserat rod in PyElastica. A centralized Markov decision process integrates the states of both manipulators and the motion of the shared target. To reduce action dimensionality while retaining distributed actuation, the normal and binormal torque profiles are parameterized by low-dimensional B-spline control points. A cooperative reward penalizes the tracking errors of both arms and encourages simultaneous convergence, while a Twin Delayed Deep Deterministic Policy Gradient algorithm is used for policy training. The policy is trained with bounded random target motion and evaluated on circular, rectangular, and figure-eight trajectories. Simulation results show that both end-effectors track targets with different motion characteristics, achieving mean spatial errors of 0.043–0.065 m. The framework maintains stable tracking under smooth motion and recovers rapidly near sharp turns. These results demonstrate the feasibility of coordinated shared-target tracking for dual soft manipulators and provide a foundation for collision-aware coordinated tracking and simulation-to-real transfer.

1. Introduction

Soft robotic manipulators have attracted considerable attention because their inherent compliance, continuous deformation, and structural adaptability enable safe interaction with humans, delicate objects, and unstructured environments. Many soft manipulators are inspired by biological muscular hydrostats, particularly octopus arms, which can bend, elongate, shorten, and vary their stiffness without relying on rigid skeletal structures [1,2]. Similar biological principles have been transferred to pneumatically actuated continuum manipulators to achieve intrinsic compliance, hyper-redundant motion, and distributed deformation [3]. These characteristics make soft manipulators promising for medical assistance, human–robot interaction, inspection, and manipulation in constrained or uncertain environments. However, the same morphological compliance that provides these advantages also introduces substantial challenges in modeling and control. Soft manipulators exhibit large deformation, nonlinear material behavior, distributed actuation, dynamic coupling, and high-dimensional configuration spaces, making accurate and computationally efficient control difficult.

Existing control methods for soft robotic manipulators can generally be classified into model-based, data-driven, and reinforcement learning-based approaches. Model-based methods establish analytical or numerical descriptions of manipulator kinematics and dynamics and subsequently design controllers based on these models. For example, piecewise constant curvature or piecewise constant strain formulations approximate a continuously deformable body using a finite number of sections [4,5]. Finite-element deformation-space formulations provide geometrically exact representations of continuum deformation but generally require more intensive numerical computation [6]. Cosserat rod theory offers a more general continuum-mechanics formulation capable of describing bending, torsion, shear, extension, inertia, and external loading within a unified framework [7,8]. Variable-curvature models have also been developed to account for external loads while maintaining sufficient computational efficiency for feedforward control [9]. Nevertheless, model-based control usually requires simplifying assumptions, accurate material and damping parameters, and repeated numerical integration. These requirements may limit its real-time applicability, particularly when a soft manipulator undergoes complex three-dimensional dynamic deformation.

Data-driven methods provide an alternative by learning the input–state or input–output relationships of soft robotic systems directly from measured data [10,11]. Koopman operator-based methods, for example, have been used to construct explicit control-oriented representations of nonlinear soft robot dynamics, allowing model predictive control to be implemented using approximately linear lifted models [12,13]. More recently, physics-based Koopman models have been combined with data-driven residual models to improve data efficiency and prediction accuracy while retaining a structure suitable for real-time control [14]. Although data-driven models can reduce dependence on explicit mechanical derivations, their performance is strongly influenced by the quantity, diversity, and quality of the training data. Models trained over a limited operating range may therefore exhibit degraded accuracy when the robot configuration, payload, material properties, or task trajectories differ from those represented in the training dataset.

Reinforcement learning provides another control paradigm in which a policy is learned through repeated interactions with an environment [15,16]. Instead of explicitly solving an inverse model at every control step, reinforcement learning directly optimizes the control policy according to a task-oriented reward function. Model-based policy learning has been used for closed-loop dynamic control of pneumatically actuated soft manipulators under variable actuation frequencies and external loads [17]. Model-free reinforcement learning has also been applied to the reaching control of soft continuum arms in both simulation and physical experiments [18]. Adaptive reinforcement learning controllers have subsequently been developed to improve reaching accuracy and compensate for stochastic behavior, model discrepancies, and actuator damage [19]. Deep reinforcement learning has further been extended to contact-rich pose and force control, including dynamic pushing tasks with simulation-to-real transfer [20].

Physics-based simulation is particularly important for reinforcement learning of soft robots because direct policy training on physical platforms can require extensive data collection and may cause premature material degradation. The Elastica framework provides a Cosserat-rod-based environment capable of simulating bending, torsion, shear, and extension and has been integrated with several continuous-control reinforcement learning algorithms, including Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), Proximal Policy Optimization (PPO), Soft Actor–Critic (SAC), and Trust Region Policy Optimization (TRPO) [21]. Among these algorithms, TD3 reduces value overestimation through clipped double-Q learning, delayed policy updates, and target policy smoothing, making it suitable for continuous-action control problems [22]. Nevertheless, reinforcement learning for soft robots still involves challenges related to computational cost, reward design, training stability, sample efficiency, and policy transfer from simulation to physical systems.

Most existing reinforcement learning studies on soft robotic manipulators have focused on single-arm reaching, trajectory tracking, obstacle avoidance, or object interaction [17,18,19,23]. Although single-arm benchmarks are valuable for evaluating control strategies, they do not fully represent tasks in which multiple deformable manipulators operate within a shared workspace. Compared with single-arm control, dual-arm soft robotic control requires a policy to process the states of two continuously deformable bodies, allocate actuation commands to both manipulators, coordinate their end-effector responses, and avoid unstable or unsafe configurations. The problem is therefore not equivalent to simply duplicating a single-arm controller.

Several studies have begun to explore dual-arm soft and continuum robotic systems. Wang and Xu developed a pneumatically actuated dual-soft-manipulator platform and used binocular visual servoing to perform a collaborative assembly task [24]. Jiang et al. proposed a multi-agent TD3 method for coordinated control of dual continuum manipulators in a space-capture scenario, incorporating inter-arm collision avoidance and actuator saturation constraints [25]. More recently, a bioinspired mobile robot equipped with two soft arms and tactile fingertips was developed for adaptive obstacle avoidance and object manipulation [26]. These studies demonstrate the potential of multi-arm soft and continuum robots, but systematic investigations combining nonlinear Cosserat-rod dynamics, low-dimensional distributed actuation, and centralized reinforcement learning for dual soft manipulators remain limited. It is also important to distinguish coordinated shared-target tracking from full coordinated tracking. In cooperative grasping, transportation, or assembly, multiple manipulators must regulate contact forces, maintain object-level geometric constraints, and coordinate their relative poses. These requirements introduce contact dynamics, force distribution, collision avoidance, and physical sensing. As an initial step toward such tasks, the present study considers a non-contact shared-target tracking problem in which two soft robotic manipulators are required to simultaneously follow the same moving target. Although this task does not yet include object grasping or force regulation, it captures several fundamental components of multi-arm soft robotic control, including joint state observation, simultaneous distributed action generation, multi-arm tracking evaluation, and coordinated convergence toward a shared task objective.

The advantages of robotic compliance are particularly important in delicate biological applications, including nondestructive sampling of fragile deep-reef organisms [27] and minimally invasive interaction with living plant tissues [28]. Within the scope of the present non-contact tracking task, a representative application is coordinated multi-view inspection, in which two soft manipulators carrying cameras or other non-contact sensors maintain complementary viewpoints of the same moving region without grasping or touching it. This task can also be regarded as a pre-contact coordination stage preceding dual-arm grasping or manipulation.

To address this problem, this paper develops a simulation-based reinforcement learning framework for the coordinated tracking of dual soft robotic manipulators. Two three-dimensional shearable soft rods are constructed using the PyElastica simulation engine based on Cosserat rod theory. A centralized Markov decision process is formulated using compact observations that incorporate the down-sampled configurations of both manipulators, their end-effector motion states, and the position and velocity information of the target. Instead of independently commanding every discretized rod element, the distributed actuation torque along each manipulator is represented using a small number of B-spline control points. This parameterization reduces the policy action dimension and generates spatially continuous actuation profiles that are compatible with the continuous deformation characteristics of soft structures. A reward function is designed to penalize the tracking errors of both manipulators and provide additional incentives when the two end-effectors simultaneously enter predefined target neighborhoods. The resulting policy is trained using the TD3 algorithm.

The main contributions of this study are summarized as follows:

• A physics-based dual-soft-manipulator reinforcement learning environment is established using two three-dimensional shearable Cosserat rods. The environment provides joint state observations, distributed actuation interfaces, and a standardized interaction process for centralized policy training.
• A low-dimensional action parameterization strategy is incorporated in which the spatially distributed normal and binormal actuation torques of both manipulators are represented using B-spline control points. This formulation avoids direct element-wise actuation while preserving continuously varying deformation capability.
• A centralized reinforcement learning formulation is introduced for coordinated shared-target tracking. The policy jointly processes the states of both manipulators, simultaneously generates their distributed actuation commands, and is optimized using a reward function that accounts for the tracking performance of both end-effectors and their simultaneous convergence.
• A simulation evaluation protocol is established using bounded random target motion for policy training and circular, rectangular, and figure-eight trajectories for testing. These trajectories represent smooth constant-curvature motion, piecewise-linear motion with sharp direction changes, and continuously varying-curvature motion, respectively, thereby enabling the tracking behavior of the two manipulators to be examined under different target-motion characteristics.

The present study focuses on establishing the simulation framework and evaluating the feasibility of centralized dual-arm soft robotic tracking. Inter-arm contact, collision-aware policy optimization, object-level coordinated tracking, and simulation-to-real transfer are not addressed in the current work and will be investigated in future studies.

2. Methods

To tackle the coordinated shared-target tracking problem for dual soft manipulators, this section presents the proposed reinforcement learning control framework. The implementation extends the open-source single-arm 3D target-tracking environment Elastica-RL-control [21] by adding a second Cosserat rod, joint dual-arm observations, a combined 24-dimensional action space, and a reward function constructed from the states of both end effectors. The resultant framework integrates physics-based Cosserat-rod simulation with centralized reinforcement learning. For convenience, the principal symbols and abbreviations used throughout this paper are summarized in Appendix A.

2.1. Cosserat Rod-Based Soft Arm Modeling

In this paper, each soft arm is modeled as a slender continuum, with Cosserat rod theory employed to provide a mechanical description. The Cosserat rod model comprehensively captures the bending, torsion, shear, and extensional deformations of the soft continuum—along with the associated inertial and damping effects—within a unified dynamic framework.

As illustrated by the mechanical-geometric parameterization in Figure 1, within a global fixed coordinate system (World frame, with orthonormal basis vectors $e_1,e_2,e_3$), the configuration of the soft arm at any given time is fully determined by a time-evolving centerline $r(s,t)\in {ℝ}^3$, where $s\in [0,L]$ represents the material arc-length coordinate along the undeformed rod and $t$ represents time. To accurately describe the spatial orientation of the cross-section, a local material coordinate system (with orthonormal basis vectors defined as $d_1,d_2,d_3$) is attached to every point along the centerline. The transformation between the global coordinate system and the local coordinate system is defined by a rotation matrix $Q(s,t)$.

By differentiating the centerline position and local orientation, the various strain characteristics of the soft arm can be rigorously quantified. The local stretching factor is defined as $e=\left|{𝜕}_{s}r\right|$. When the rod undergoes no shear deformation, the local basis vector $d_3$ remains aligned with the tangent to the centerline; however, in the presence of shear and stretching deformations, the shear strain vector can be expressed as $\sigma =Q{𝜕}_{s}r-d_3$. Furthermore, the 3D bending and torsional deformations of the soft arm are described by the curvature vector $\kappa$, satisfying ${𝜕}_s{d}_j=\kappa \times d_j(j=1,2,3)$.

Based on the aforementioned kinematic parameters, the complete spatial dynamic behavior of the Cosserat rod is governed by the following two sets of partial differential equations, representing the conservation of linear momentum and angular momentum, respectively:

$$\rho A{𝜕}_t^{2}r={𝜕}_s\left({\displaystyle \frac{Q^{T}S\sigma }{e}}\right)+ef,$$

(1)

$${\displaystyle \frac{\rho I}{e}}{𝜕}_t\omega ={𝜕}_s\left({\displaystyle \frac{B\kappa }{e^3}}\right)+{\displaystyle \frac{\kappa \times B\kappa }{e^3}}+\left(Q{\displaystyle \frac{{𝜕}_{s}r}{e}}\times S\sigma \right)+\left(\rho I{\displaystyle \frac{\omega }{e}}\right)\times \omega +{\displaystyle \frac{\rho I\omega }{e^2}}{𝜕}_{t}e+ec,$$

(2)

In the aforementioned equations, $\rho$ represents the density of the soft material, $A$ is the cross-sectional area, and $I$ is the cross-sectional moment of inertia matrix; $S$ and $B$ represent the shear stiffness and bending stiffness matrices of the rod, respectively; $\omega$ is the angular velocity in the local coordinate system (satisfying ${𝜕}_t{d}_j=\omega \times d_j$), and $\mathrm{f}$ and $\mathrm{c}$ represent the densities of the external resultant force and resultant moment applied per unit arc length, respectively. It is particularly worth noting that the actuation torque output by the reinforcement learning strategy is applied continuously and in a distributed manner across the entire soft arm through the term $\mathrm{c}$ in the equations.

The dual-arm simulation environment is implemented using PyElastica, an open-source numerical framework for Cosserat-rod dynamics. Two parallel three-dimensional shearable rods are modeled. Within the adopted formulation, the simulator represents bending, torsion, shear, extension, inertia, damping, and externally applied distributed moments and provides the state transitions used for policy training. The rod centerline positions and orientations are obtained through forward dynamic integration rather than through an inverse-kinematics solver. However, the model parameters have not been calibrated against a specific physical dual-arm platform. Therefore, the reported results should be interpreted as simulation-based evidence rather than hardware validation.

2.2. Dual-Arm Shared-Target Tracking Formulation

The task considered in this study is shared-target tracking by two soft robotic arms. Let $p_{\mathrm{tip}}^{\left(1\right)}\left(t\right)$ and $p_{\mathrm{tip}}^{\left(2\right)}\left(t\right)$ denote the tip positions of the first and second arm, and let $p_g\left(t\right)$ denote the target position. The instantaneous tracking errors are defined as $e_i(t)={‖p_{\mathrm{tip}}^{(i)}(t)-p_g(t)‖}_2,\hspace{0.33em}\hspace{0.33em}i\in \{1,2\}$. The objective is not only to minimize each individual error but also to encourage both arms to reach and maintain proximity to the same target within the same episode.

This shared-target tracking formulation is intended to isolate coordinated motion and synchronization. It intentionally excludes object grasping, manipulator–target contact, contact-force regulation, inter-arm contact, and hardware sensing. The target sphere serves only as a moving positional reference and does not physically interact with either manipulator. Consequently, no manipulator–target contact force is generated or recorded in the present simulations. The task should therefore be interpreted as a non-contact coordination benchmark or as a pre-contact stage preceding cooperative manipulation.

2.3. MDP Formulation and Action Parameterization

The centralized dual-arm control problem is formulated as a Markov decision process (MDP), $\mathcal{M}=\left(\mathcal{S},\mathcal{A},P,R,\gamma \right)$, where $\mathcal{S}$ is the observation space, $\mathcal{A}$ is the action space, $P$ is the transition induced by the simulator, $R$ is the reward function, and $\gamma$ is the discount factor used by the RL algorithm. At each control step, the policy receives the current observation and selects an action, and the simulator advances for a fixed number of physical integration steps before returning the next observation and reward.

The observation is designed to expose compact but task-relevant information rather than the full finite-dimensional rod state. In the current implementation, the observation contains down-sampled nodal positions of both arms, the velocity norms and directions of the two arm tips, and the position, velocity norm, and velocity direction of the target. This can be summarized as $s_t=\left[s_t^{\left(1\right)},s_t^{\left(2\right)},x_t^g,v_t^g\right]$, where $s_t^{\left(i\right)}$ is the compact state of arm $i$, and $x_t^g$ and $v_t^g$ represent target motion information. The current observation dimension is 81 for the selected discretization and task configuration. Specifically, each manipulator is discretized into 20 rod elements and therefore contains 21 nodal positions. The nodal positions are sampled at an interval of two nodes, including the two endpoints, resulting in 11 sampled nodes for each manipulator. Each arm contributes 33 position components, corresponding to the three-dimensional coordinates of the 11 sampled nodes, together with one end-effector velocity magnitude and three components of the normalized end-effector velocity direction. The shared target contributes three position components, one velocity magnitude, and three components of its normalized velocity direction. Consequently, the observation dimension is

$$\dim (\mathcal{S})=2\left(11\times 3+1+3\right)+\left(3+1+3\right)=81$$

(3)

Thus, each manipulator contributes 37 state variables, while the moving target contributes seven state variables. The 81-dimensional observation is directly available from the simulator. In a prospective physical implementation, the sampled centerline positions could be reconstructed using multi-view vision, optical markers, or distributed shape sensors such as fiber Bragg grating sensors. The end-effector velocities could be obtained through finite differences or state estimation, whereas the target position and velocity could be measured using external or onboard vision. Sensor noise, estimation errors, latency, and occlusion are not modeled in the present study. Therefore, the simulated observation and control rates should not be interpreted as demonstrated hardware sensing rates.

For the present three-dimensional bending task, the action is defined as $a_t=\left[u_n^{\left(1\right)},u_b^{\left(1\right)},u_n^{\left(2\right)},u_b^{\left(2\right)}\right]\in {\left[-1,1\right]}^{24}$, where $u_n$ and $u_b$ denote the normal and binormal actuation control points for each arm. Each direction uses six B-spline control points. These low-dimensional control points are converted into smooth distributed torque profiles along the corresponding arm. This action parameterization avoids directly commanding every rod element while still allowing spatially varying deformation along the arm body. Tangential actuation can be enabled in the same interface for higher-dimensional tasks but is not used in the current three-dimensional bending setting.

In this study, the B-spline mapping is employed as a low-dimensional action-to-torque parameterization within the dual-arm reinforcement learning framework. Although the B-spline basis itself is not newly developed, its integration with the centralized policy enables spatially continuous distributed actuation while avoiding direct element-wise torque commands.

Specifically, the material arc-length coordinate is first normalized as

$$\xi ={\displaystyle \frac{s}{L}},\hspace{0.33em}\xi \in [0,1],$$

(4)

For each manipulator $\mathrm{i}$ and actuation direction $d\in \{n,b\}$, the policy generates six normalized interpolation values $u_{d,j}^{(i)}(t)\in [-1,1],\hspace{0.33em}j=1,\dots ,6$. Two additional endpoint values are imposed as zero because no actuation moment is generated at the two ends of the rod. The complete interpolation data are therefore defined as

$${\overline{u}}_{d,j}^{(i)}(t)=\left\{\begin{array}{ll}0,\hfill & j=0 \mathrm{or} j=7,\hfill \\ u_{d,j}^{(i)}(t),\hspace{1em}\hfill & j=1,\dots ,6,\hfill \end{array}\right.\hspace{0.33em}{\xi }_j={\displaystyle \frac{j}{7}}.,$$

(5)

A cubic interpolating B-spline with the default not-a-knot boundary condition is constructed through the eight equally spaced data points:

$$B_d^{(i)}(\xi ,t)={\mathcal{I}}_3\left({\left\{{\xi }_j,{\overline{u}}_{d,j}^{(i)}(t)\right\}}_{j=0}^7\right),$$

(6)

where ${\mathcal{I}}_3(\cdot )$ denotes cubic spline interpolation. The dimensional distributed moment is obtained by scaling the interpolated profile:

$${\tau }_d^{(i)}(\xi ,t)={\alpha }_d{B}_d^{(i)}(\xi ,t),\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{\alpha }_n={\alpha }_b=75\hspace{0.33em}\mathrm{N}\mathrm{m}.,$$

(7)

The value of 75 N·m is the common scaling factor used for the normal and binormal directions. It corresponds to a nominal range of $[-75,75]\hspace{0.33em}\mathrm{N}\cdot \mathrm{m}$ at the prescribed interpolation locations. Because the cubic interpolated profile is not subjected to post-interpolation clipping, this value should not be interpreted as a strict hard bound on every evaluated point along the rod. The maximum moments observed during policy evaluation are therefore reported separately from the nominal scaling factor.

The spatial actuation moment applied to manipulator $i$ is consequently expressed as

$${\tau }_{\mathrm{act}}^{(i)}(s,t)={\tau }_n^{(i)}(\xi ,t)d_1^{(i)}(s,t)+{\tau }_b^{(i)}(\xi ,t)d_2^{(i)}(s,t),$$

(8)

where $d_1^{\left(i\right)}$ and $d_2^{\left(i\right)}$ denote the local normal and binormal material-frame directions of the $i$-th manipulator. Tangential actuation along $d_3^{\left(i\right)}$ is not considered in the present tracking task.

For numerical implementation, the continuous B-spline torque profiles are evaluated at the centers of the discretized rod elements. For a manipulator divided into $n_e$ elements, the normalized coordinate of the $k$-th element is given by

$${\xi }_k={\displaystyle \frac{k-\frac{1}{2}}{n_e}},\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}k=1,\dots ,n_e,$$

(9)

The evaluated torque values are then applied to the corresponding rod elements as external actuation moments in PyElastica. During each policy-control interval, the B-spline control points remain unchanged, while the physics simulator advances for a fixed number of integration steps. At the next control step, the policy generates a new set of control points and the distributed actuation profiles are reconstructed accordingly.

This formulation reduces the policy action dimension to 24, corresponding to six control points for each of the two actuation directions of both manipulators. At the same time, the local support and continuity properties of the cubic B-spline basis prevent abrupt spatial changes between adjacent rod elements and generate continuously varying distributed actuation profiles.

The distributed normal and binormal moments used in this study should be interpreted as an idealized actuation representation rather than as direct commands to a specific physical actuator. A possible physical realization could employ multiple independently driven antagonistic tendon pairs, pneumatic chambers, or other segment-wise bending actuators distributed along each arm. However, the present model does not include actuator dynamics, bandwidth limitations, saturation, hysteresis, or the mapping from individual actuator inputs to the continuous moment field. Consequently, the physical actuation mechanism and its achievable torque range would need to be identified and validated for a specific hardware platform.

2.4. Joint Tracking Reward and Termination Design

The reward function is formulated to account for both individual tracking accuracy and cooperative convergence toward the shared target. Let

$$e_i(t)=\parallel p_{\mathrm{ee}}^{\left(i\right)}(t)-p_{tar}(t){\parallel }_2,i=1,2,$$

(10)

denote the Euclidean tracking error between the end-effector of arm $i$ and the target at time $t$. The distance-based reward is defined as

$$R_d\left(t\right)=-{\lambda }_1{e}_1{\left(t\right)}^2-{\lambda }_2{e}_2{\left(t\right)}^2$$

(11)

where ${\lambda }_1$ and ${\lambda }_2$ are positive weighting coefficients. In this study, both weights are set to one. This term penalizes large tracking errors of either arm and encourages both end-effectors to remain close to the shared target.

To further promote cooperative convergence, sparse bonuses are assigned according to a set of nested target neighborhoods. Individual bonuses are given when either arm enters a predefined neighborhood, while an additional cooperative bonus is assigned only when both arms simultaneously enter the strictest neighborhood. The bonus term is written as

$$\begin{array}{ll}{R}_b(t)\hfill & ={\beta }_1\mathbb{I}\{e_1(t)<{ϵ}_1\}+{\beta }_1\mathbb{I}\{e_2(t)<{ϵ}_1\}\hfill \\ ⁠\hfill & +{\beta }_2\mathbb{I}\{e_1(t)<{ϵ}_2\}+{\beta }_2\mathbb{I}\{e_2(t)<{ϵ}_2\}\hfill \\ ⁠\hfill & +{\beta }_3\mathbb{I}\{e_1(t)<{ϵ}_3\wedge e_2(t)<{ϵ}_3\},\hfill \end{array}$$

(12)

where $\mathbb{I}\{\cdot \}$ is the indicator function, and ${ϵ}_1>{ϵ}_2>{ϵ}_3$ define nested target neighborhoods with increasingly strict tracking requirements. This design discourages the policy from relying on only one arm and encourages both end-effectors to approach the target cooperatively.

The total reward used for policy optimization is then defined as

$$R(t)=R_d(t)+R_b(t)+R_{\mathrm{invalid}}(t),$$

(13)

where

$$R_{\mathrm{invalid}}(t)=\left\{\begin{array}{ll}{r}_{\mathrm{invalid}}, \hfill & \mathrm{if} \mathrm{an} \mathrm{invalid} \mathrm{state} \mathrm{is} \mathrm{detected}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(14)

An invalid state is detected when a NaN value occurs in the nodal position array of either manipulator. In this case, the episode is immediately terminated and $r_{\mathrm{invalid}}=-1000$ is assigned.

In normal simulation states, the reward is therefore given by the sum of the distance penalty and the sparse bonus, namely $R(t)=R_d(t)+R_b(t)+R_{\mathrm{invalid}}(t)$.

The framework can further incorporate a safety-oriented inter-arm collision penalty when required. Let

$$d_{12}^{\min }(t)={\min }_{m,n}{‖p_m^{(1)}(t)-p_n^{(2)}(t)‖}_2$$

(15)

denote the minimum distance between discrete nodes on the two arms, where $p_m^{\left(1\right)}(t)$ and $p_n^{\left(2\right)}(t)$ are the positions of node $m$ on the first arm and node $n$ on the second arm, respectively. A collision or near-collision event can be defined when

$$d_{12}^{\min }(t)<{\rho }_1+{\rho }_2+\delta$$

(16)

where ${\rho }_1$ and ${\rho }_2$ are local arm radius and $\delta$ is a prescribed safety margin. A corresponding penalty $r_{col}=-{\lambda }_{col}$ can then be added to the reward. This term is included as an extensible safety formulation, but it is not activated or experimentally evaluated in the current results.

The parameter values used in the reward function are summarized in

Table 1. Parameters used in the reward function.

Parameter	Value	Unit
${\lambda }_1$	1.0	-
${\lambda }_2$	1.0	-
${ϵ}_1$	0.1	M
${ϵ}_2$	0.075	M
${ϵ}_3$	0.05	M
${\beta }_1$	0.5	-
${\beta }_2$	1.0	-
${\beta }_3$	1.5	-
$r_{\mathrm{invalid}}$	−1000	-
${\rho }_1$	0.05	M
${\rho }_2$	0.05	M
$\delta$	0.01	M
${\lambda }_{\mathrm{col}}$	10.0	-
$r_{\mathrm{col}}$	−10.0	-

. Equal distance weights are assigned to the two manipulators because they have identical mechanical models and pursue the same tracking objective. The threshold distances of 0.10, 0.075, and 0.05 m correspond to 2, 1.5, and 1 times the target radius, respectively, and define nested coarse, intermediate, and strict tracking regions. The associated bonuses of 0.5, 1.0, and 1.5 progressively emphasize tracking precision and simultaneous convergence of both end effectors. These values were selected empirically to retain dense distance-based guidance while avoiding a purely sparse-reward formulation.

2.5. Policy Training and Evaluation Protocol

The controller is implemented as a centralized, model-free, off-policy actor–critic policy using Twin Delayed Deep Deterministic Policy Gradient (TD3). At each control step, a single actor network receives the joint 81-dimensional observation of the two manipulators and generates their combined 24-dimensional continuous action. During training, two independent critic networks estimate the state–action values to reduce value overestimation, while delayed actor updates and target-policy smoothing are used to improve training stability. During evaluation, the trained deterministic actor directly generates the actions without online model identification or optimization. The environment also supports DDPG and PPO for algorithm comparison.

Each episode lasts 10 s of simulated time. The physics integration time step is 0.0002 s, and the action is updated every 10 integration steps, yielding approximately 5000 policy interaction steps per episode. Six control points are used per arm and per actuation direction.

The TD3 policy receives the 81-dimensional joint observation and generates a 24-dimensional continuous action vector corresponding to the B-spline actuation control points of the two manipulators. The actor network directly maps the current observation to the normalized action, while two independent critic networks estimate the corresponding state–action values. The network architecture and principal optimization hyperparameters used for TD3 training are summarized in

Table 2. Network architecture and training hyperparameters of the TD3 policy.

Hyperparameter	Value
Policy network	MLP policy
Total training timesteps	$1.0\times {10}^7$
Random seed	0
Replay buffer size	$2.0\times {10}^6$
Learning starts	$5.0\times {10}^4$
Learning rate	$3.0\times {10}^{-4}$
Discount factor	0.99
Soft update coefficient	0.005
Batch size	128
Train frequency	100
Gradient steps	100
Policy delay	2
Target policy noise	0.2
Target noise clip	0.5
Action noise	None
Random exploration	0.0
Verbose	1

. Unless otherwise specified, these hyperparameters remain unchanged throughout the training process and the subsequent trajectory-tracking evaluations.

The simulation and reinforcement-learning implementation was developed in Python 3.7 using PyElastica 0.1.0, Stable-Baselines 2.10.2, TensorFlow 1.15.0, NumPy 1.21.6, and SciPy 1.7.3. All training and evaluation runs were performed on a workstation equipped with an Intel(R) Core(TM) i7-13650HX CPU, an NVIDIA GeForce RTX 4070 Laptop GPU, and 16 GB of system memory under Ubuntu 20.04. Training the TD3 policy for 10⁷ environment interaction steps required approximately 8.5 h. During training, each 10 s of simulated physical time required an average of approximately 15.3 s of wall-clock time, corresponding to an effective real-time factor of approximately 0.65.

The tracking performance of each soft manipulator is evaluated using the axis-wise root mean square error (RMSE), axis-wise mean absolute error (MAE), instantaneous spatial error, mean spatial error, and three-dimensional RMSE. For the $i$-th manipulator, the tracking error along each Cartesian direction is defined as

$$e_q^{\left(i\right)}(t)=p_{tip,q}^{\left(i\right)}\left(t\right)-p_{g,q}\left(t\right),q\in \left\{x,y,z\right\}$$

(17)

$p_{tip,q}^{\left(i\right)}\left(t\right)$ and $p_{g,q}\left(t\right)$ denote the end-effector position of manipulator $i$ and the target position along direction $q$, respectively. The axis-wise RMSE and MAE are calculated as

$$\begin{array}{l}{\mathrm{RMSE}}_q^{\left(i\right)}=\sqrt{{\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T{e}_q^{\left(i\right)}}{\left(t\right)}^2}\\ MA{E}_q^{\left(i\right)}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T\left|e_q^{\left(i\right)}\left(t\right)\right|}\end{array}$$

(18)

The instantaneous spatial tracking error is defined as

$$e_{sp}^{\left(i\right)}\left(t\right)={‖p_{tip}^{\left(i\right)}\left(t\right)-p_g^{⁠}\left(t\right)‖}_2$$

(19)

Accordingly, the mean spatial error and the three-dimensional RMSE are calculated as

$$\begin{array}{l}{\overline{e}}_{sp}^{\left(i\right)}\left(t\right)={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T\left|e_{sp}^{\left(i\right)}\left(t\right)\right|},\\ RMS{E}_{3d}^{\left(i\right)}=\sqrt{{\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T{e}_{sp}^{\left(i\right)}}{\left(t\right)}^2}\end{array}$$

(20)

All metrics are calculated over the same evaluation interval and are reported separately for the two manipulators. The axis-wise MAE represents the average magnitude of the tracking deviation along each Cartesian direction and is less sensitive to isolated large errors. The axis-wise RMSE assigns greater weight to large deviations through squared errors. The instantaneous spatial error is the Euclidean distance between the end-effector and the target, while the mean spatial error is its temporal average. The three-dimensional RMSE is calculated from the squared Euclidean tracking distance over all $T$ evaluation samples. All position-error metrics are reported in meters, and $T$ denotes the total number of samples in the evaluation interval.

3. Simulation Experiment

3.1. Simulation Parameters

The simulation environment uses PyElastica to instantiate two initially straight and parallel shearable Cosserat rods. The fixed bases of the first and second manipulators are located at

$$p_b^{(1)}={\left[0,0,0\right]}^{\mathrm{T}}\mathrm{m},\hspace{0.33em}{p}_b^{(2)}={\left[0,0,0.2\right]}^{\mathrm{T}}\mathrm{m}$$

(21)

respectively. The resulting base separation is therefore 0.2 m along the global $z$-direction. Both manipulators are initially aligned with the positive global $y$-direction. Their initial normal direction is aligned with the positive global z-direction, while the corresponding binormal direction is aligned with the positive global x-direction. The proximal end of each rod is fixed, whereas the distal end remains unconstrained.

Both manipulators use identical structural and material parameters. Each rod has an undeformed length of 1.0 m and is discretized into 20 elements. A uniform circular cross-section with a radius of 0.05 m is adopted along the entire rod. The material density is $1000\hspace{0.33em}\mathrm{kg}/{\mathrm{m}}^3$, the Young’s modulus is 10 MPa, the Poisson ratio is 0.5, and the dissipation coefficient is 30.

The environment is configured as a three-dimensional bending problem with $\dim =3.0$. Therefore, distributed actuation is enabled in the local normal and binormal directions, whereas tangential actuation is disabled. Six B-spline control points are assigned to each actuation direction of each manipulator. Accordingly, the centralized policy produces 24 action variables at each control step.

The equations of motion are integrated using the Position Verlet scheme with a physical time step of 0.0002 s. A policy action is maintained for 10 physical integration steps, corresponding to a control interval of

$$\Delta t_c=10\Delta t=2.0\times {10}^{-3}\hspace{0.33em}\mathrm{s}$$

(22)

or an effective control frequency of 500 Hz. Each simulation episode lasts 10 s and therefore contains approximately 5000 policy interaction steps.

The shared target is represented by a rigid sphere with a radius of 0.05 m and a density of $1000\hspace{0.33em}\mathrm{kg}/{\mathrm{m}}^3$. Its translational motion is determined by the selected task mode. In the present study, the sphere serves as a moving positional reference, and physical contact between the sphere and the two manipulators is not considered.

No gravitational body force is applied to either manipulator in the present simulations. The reported results therefore correspond to an idealized gravity-free environment or to a system with perfect gravity compensation. This assumption isolates the coordinated tracking behavior from gravitational sag. In a physical implementation, gravity would introduce configuration-dependent deformation and additional holding torque and could consequently increase the steady-state tracking error. Explicit gravity modeling and compensation are left for future hardware-oriented studies.

Figure 2 illustrates the simulation environment for the dual-arm soft robot built based on the parameter configuration described in this section. Two soft arms with identical structural parameters are symmetrically arranged with their bases laterally separated, fixed at the proximal end and free at the distal end; the shared target sphere is initially located within the common reachable workspace of both arms.

3.2. Training Performance and Algorithm Compatibility

To examine the compatibility of the proposed dual-arm environment with different continuous-control reinforcement-learning algorithms, PPO, DDPG, and TD3 were evaluated under the same observation space, action parameterization, reward function, target-motion setting, and total number of environment interaction steps. Each algorithm was independently trained using three random seeds (0, 1, and 2). The purpose of this comparison is primarily to verify that the proposed environment can support different policy-optimization mechanisms and to provide a preliminary basis for selecting the controller used in the subsequent tracking evaluations, rather than to establish a general performance ranking among reinforcement-learning algorithms.

For each algorithm, the episode return was recorded throughout the training process. The episode return is defined as the cumulative reward obtained during one episode:

$$G_k={\displaystyle \sum _{t=0}^{T_k-1}{R}_t},$$

(23)

where $G_k$ denotes the return of the k-th episode, $R_t$ is the reward obtained at control step t, and $T_k$ is the number of control steps in the episode.

Figure 3 presents the training–return curves of PPO, DDPG, and TD3 in the proposed dual-soft-manipulator environment. The horizontal axis represents the cumulative number of environment interaction steps, while the vertical axis represents the episode return. For each algorithm, the solid curve denotes the mean return over three random seeds, and the shaded region represents ±1 sample standard deviation. The same smoothing procedure was applied to all algorithms for visualization.

For each random seed, the final-stage return was calculated as the average episode return over the final 10% of the training timesteps. The reported values represent the mean ± sample standard deviation across three random seeds.

Figure 3 shows that all three algorithms completed the training process and achieved substantially higher returns than their initial policies, confirming the compatibility of the proposed environment with different continuous-control algorithms. PPO and DDPG exhibited relatively rapid improvements during the early training stage, after which their mean returns remained comparatively stable. TD3 improved more gradually but achieved a higher mean return during the later training stage. As summarized in

Table 3. Final-stage episode returns over three random seeds.

Algorithm	Final-Stage Episode Return (×103)
PPO	$9.71\pm 6.20$
DDPG	$8.78\pm 4.68$
TD3	$13.54\pm 3.65$

, the final-stage returns of PPO, DDPG, and TD3 were $9.71\pm 6.20$, $8.78\pm 4.68$, and $13.54\pm 3.65\times {10}^3$, respectively. Under the adopted environment and hyperparameter settings, TD3 therefore achieved the highest mean final-stage return and the lowest observed standard deviation among the three evaluated algorithms. Based on this training behavior and its suitability for continuous action spaces, TD3 was selected as the controller for the subsequent trajectory-tracking evaluations. Nevertheless, because only three random seeds and one set of algorithm-specific hyperparameters were considered, these results should not be interpreted as establishing the general superiority of TD3 over PPO or DDPG.

The following experiments therefore focus on evaluating the tracking behavior of the TD3 controller under different target trajectories, while the PPO and DDPG results are retained to demonstrate algorithm compatibility and provide a multi-seed comparison of training behavior.

3.3. Dual-Arm Shared-Target Tracking

To verify whether the trained strategy can drive the end effectors of two soft arms towards a common moving target, we conducted trajectory tracking experiments. During training, the target performed bounded random motion within the shared workspace with a linear velocity set to 0.3 m/s.

In the strategy validation phase, the experiment specifically selected circular, rectangular, and figure-eight trajectories as tasks, corresponding to three typical working conditions: regular smooth motion with constant curvature, piecewise-linear motion with sharp turns, and gradual turning motion with continuously changing curvature. This systematically verified the steady-state tracking accuracy, dynamic response capability to sudden commands, and cooperative motion synchronization of the soft arms under different path characteristics.

3.4. Tracking Error Analysis

To quantitatively evaluate the coordinated tracking performance of the two manipulators, the axis-wise RMSE and MAE are used to characterize the tracking errors in the x, y, and z-directions, while the instantaneous spatial error, mean spatial error, and three-dimensional RMSE are used to evaluate the overall end-effector tracking performance.

The experiment selected three typical target trajectories—circle, rectangle, and figure-eight—to conduct tests, corresponding to three typical working conditions: constant curvature smooth motion, right-angle abrupt change motion, and continuous variable curvature motion, respectively.

Figure 4 illustrates the 3D tracking comparison results of the dual-arm end effectors on a circular target trajectory. The target moves in a uniform circular motion with an angular velocity of ${\displaystyle \frac{2\pi }{10}}\hspace{0.33em}\mathrm{rad}/\mathrm{s}$ along a radius of 0.3 m, exhibiting constant curvature and smooth motion. This target representation method provides a simple and controllable way to evaluate whether the ends of two soft arms can coordinately move towards the same moving target. From the overall trajectory, both robotic arms closely track the target sphere’s path, with high trajectory overlap and no significant global lag or large deviation. This intuitively verifies the effectiveness of the proposed framework in cooperative tracking under continuous smooth motion scenarios.

As can be seen from the curve variation pattern in Figure 5, the two arms exhibit a small transient deviation at the initial stage of the mission and then converge rapidly and enter the steady-state fluctuation range. This initial deviation arises because both manipulators start from straight and stationary configurations, whereas the target is initially offset from the two end effectors and begins moving at the start of the evaluation. The policy must first generate sufficient distributed bending moments to deform and accelerate the compliant rods against their elastic stiffness, inertia, and damping. The observed deviation is therefore primarily an initialization transient rather than an indication of unstable tracking. Because the reference remains continuously time-varying rather than representing a step input, classical step-response indices such as rise time, settling time, percentage overshoot, and under-/critical-/over-damped classifications are not directly applicable to the present tracking task. Quantitatively, the spatial average errors of soft arm 1 and soft arm 2 are 0.065 m and 0.043 m, respectively. Since the curvature of the circular trajectory is constant, the errors in each direction of the two arms show periodic fluctuation characteristics that match the motion cycle, and the error amplitude and fluctuation pattern of the two arms are basically consistent, reflecting good dual-end coordination and synchronization.

Figure 6 shows the 3D tracking comparison results of the dual-arm end effectors on a rectangular target trajectory. This trajectory includes multiple right-angle turns to verify the dynamic response capability of the strategy to stepping motion commands and trajectories with abrupt changes in motion direction. The target was set to move at a constant linear velocity of 0.2 m/s along a rectangle 0.6 m long and 0.4 m wide. From the overall trajectory profile, both arms can completely reproduce the shape of the rectangular path; within the straight sections, the dual-arm end effectors closely follow the target, exhibiting high tracking accuracy; at the right-angle turns, there is a small overshoot and hysteresis, which is consistent with the inertial response characteristics brought about by the large deformation of the soft robotic arm.

As shown in Figure 7, during the straight-line running segment of the trajectory, the errors in all directions of both arms can be stably maintained at a low level; whenever passing through a right-angle inflection point, the error in the corresponding motion direction shows an instantaneous peak, the total error increases synchronously, and then quickly falls back to the steady-state range. Quantitatively, the spatial average errors of soft arm 1 and soft arm 2 are 0.054 m and 0.05 m, respectively. This result indicates that the proposed strategy has good tracking and recovery capability for sudden commands, and the peak error and convergence speed of the two arms are basically matched, showing stable cooperative consistency.

Figure 8 shows the 3D tracking comparison results of the dual-arm end effectors on a figure-eight target trajectory. This trajectory is a closed parametric trajectory formed by the superposition of two sinusoidal motions of different frequencies in the x-z plane. The target completes one sinusoidal reciprocating motion in the x-direction and two sinusoidal reciprocating motions in the z-direction. Unlike the circular trajectory with constant curvature and the rectangular trajectory with abrupt changes in motion direction, the figure-eight trajectory exhibits continuously varying curvature and non-uniform target speed. At the same time, the target velocity is non-constant and varies approximately between 0.1 m/s and 0.3 m/s along the trajectory to verify the adaptive tracking capability of the framework for complex turning paths. From the perspective of trajectory fitting, both robotic arm end effectors can smoothly track the continuous turning motion of the target without obvious trajectory distortion or stuttering, and the overall tracking process is smooth and natural.

Figure 9 shows that, due to the continuous changes in trajectory curvature and target speed, the error curve exhibits a gradually fluctuating characteristic, without any abrupt peaks. Throughout the process, the error levels of the two robotic arms are basically consistent. As observed from the error curves in Figure 9d, the spatial average errors of soft arm 1 and soft arm 2 are 0.062 m and 0.053 m, respectively, with only a slight increase in error at the curve with the largest curvature, but still within a controllable range. This demonstrates that the proposed framework exhibits stable tracking performance and good generalization ability for continuously varying curvature motion.

To further evaluate the benefit of the centralized formulation, an additional benchmark was conducted using two independently controlled single-arm TD3 policies adapted from the original Case 1 environment. Each independent policy received only the state of its corresponding manipulator and the target state and generated the actuation commands for that manipulator without access to the state or action of the other arm. The mechanical parameters, B-spline action scaling, target trajectories, simulation duration, and error calculation method were kept consistent with those used in the centralized dual-arm evaluations. Because explicit mechanical coupling and inter-arm contact are not included in the present simulation, the two independent policies were evaluated separately under identical target trajectories, and their results were combined to represent the independent-control baseline.

As shown in

Table 4. Comparison of tracking RMSE between centralized TD3 and independently controlled single-arm TD3 policies.

Trajectory	Controller	Arm 1 RMSE (m)	Arm 2 RMSE (m)
Circular	Centralized TD3	0.065	0.043
	Independent single-arm TD3	0.054	0.059
Rectangular	Centralized TD3	0.054	0.050
	Independent single-arm TD3	0.067	0.064
Figure-eight	Centralized TD3	0.062	0.053
	Independent single-arm TD3	0.070	0.066

, both control configurations successfully completed the three target-tracking tasks. For the circular trajectory, the independent-control baseline achieved a lower RMSE for Arm 1 (0.054 m versus 0.065 m), whereas the centralized controller achieved a lower RMSE for Arm 2 (0.043 m versus 0.059 m). Thus, the circular trajectory produced an arm-dependent result rather than a uniform advantage for either configuration. For the rectangular trajectory, the centralized controller reduced the RMSE from 0.067 to 0.054 m for Arm 1 and from 0.064 to 0.050 m for Arm 2. For the figure-eight trajectory, the centralized controller similarly achieved lower RMSE values for both arms, with values of 0.062 and 0.053 m, compared with 0.070 and 0.066 m for the independently controlled policies. The advantage of the centralized formulation was therefore more evident for the rectangular and figure-eight trajectories, whereas the simpler circular trajectory showed mixed per-arm results.

These results show that independently controlled single-arm policies remain capable of completing the local target-tracking task. However, each independent policy makes decisions using only its own manipulator state and the target state and therefore cannot condition its action on the instantaneous state of the other manipulator. In contrast, the centralized controller jointly processes the states of both manipulators and generates their actions under a shared tracking objective. The lower RMSE values obtained for both manipulators under the rectangular and figure-eight trajectories suggest that the joint information structure becomes more beneficial as the target motion involves sharp directional changes or continuously varying curvature. Moreover, the centralized formulation provides a unified basis for incorporating relative-configuration constraints and object-level cooperative objectives in future extensions, although explicit inter-arm contact and collision avoidance are not experimentally evaluated in the present study.

3.5. Actuation Moment Analysis

To further characterize the actuation requirements of the learned controller, the distributed actuation moments were recorded during the 10-s figure-eight tracking evaluation. For each manipulator, the normal and binormal moment distributions were sampled at 603 time instants, with each record containing the moments applied to the 20 rod elements. To compactly represent the spatial distribution, the spatial root-mean-square (RMS) moment was calculated at each time instant as

$${\tau }_{d,\mathrm{RMS}}^{(i)}(t)=\sqrt{{\displaystyle \frac{1}{N_e}}{\displaystyle \sum _{e=1}^{N_e}{\left[{\tau }_{d,e}^{(i)}(t)\right]}^2}},$$

(24)

where $d\in \{n,b\}$ denotes the normal or binormal direction, $i\in \{1,2\}$ denotes the manipulator, and $N_e=20$ is the number of rod elements.

As shown in Figure 10, the spatial RMS moments of both manipulators remain within comparable ranges throughout the figure-eight tracking task, without systematic growth or divergence. The maximum absolute distributed moments, evaluated over all time instants and rod elements, were 138.9 and 122.4 N·m in the normal direction for Manipulators 1 and 2, respectively, and 119.4 and 121.7 N·m in the binormal direction. Therefore, the overall maximum normal and binormal moments were 138.9 and 121.7 N·m, respectively. These peak values are higher than the nominal scaling coefficient of 75 N·m because the coefficient is applied to the B-spline control values rather than imposed as a hard saturation limit on the interpolated moment distribution. The cubic interpolation may consequently produce local values outside the range of its control values.

4. Discussion

The simulation results provide evidence for the feasibility of the proposed centralized dual-arm reinforcement-learning framework for non-contact shared-target tracking. Under the circular trajectory, both manipulators maintained bounded tracking errors with periodic variations corresponding to the target motion. For the rectangular trajectory, transient increases in tracking error occurred near the right-angle turns, followed by recovery to the steady tracking range. Under the figure-eight trajectory, stable tracking was maintained despite the continuously varying curvature and target speed. These results show that the trained policy can accommodate several representative target-motion patterns, although the present evaluation does not establish generalization beyond the considered workspace and trajectory types.

The multi-seed comparison further demonstrates that the proposed environment is compatible with different continuous-control reinforcement-learning algorithms. PPO and DDPG exhibited relatively rapid improvements during the early stage of training, whereas TD3 improved more gradually and achieved a higher mean return during the final stage. TD3 also exhibited the lowest observed final-stage variability among the evaluated algorithms. Nevertheless, the comparison was conducted using only three random seeds and one set of algorithm-specific hyperparameters. The results should therefore be interpreted as supporting the selection of TD3 for the subsequent tracking evaluations rather than establishing its general superiority over PPO and DDPG.

The comparison with independently controlled single-arm TD3 policies provides further insight into the role of the centralized formulation. For the circular trajectory, the two control configurations produced mixed arm-dependent results, without a uniform advantage for either controller. For the rectangular and figure-eight trajectories, however, the centralized controller achieved lower tracking errors for both manipulators. This result suggests that the joint observation becomes more useful when the target motion contains sharp directional changes or continuously varying curvature. Each independent policy selects its action using only the state of its corresponding manipulator and the target, whereas the centralized policy can condition the actions of both manipulators on their joint instantaneous state. The centralized structure therefore provides a more suitable basis for introducing relative-configuration constraints, collision-aware objectives, and object-level cooperative tasks in future extensions.

The B-spline action parameterization reduces the dimensionality of the policy output while producing spatially continuous normal and binormal actuation profiles along both manipulators. The recorded moment histories remained bounded throughout the evaluated trajectory, although the maximum interpolated moments exceeded the nominal scaling coefficient applied to the B-spline control values. This occurs because the scaling coefficient defines the range of the interpolation values rather than a hard saturation limit on every point of the reconstructed moment profile. The reported moment histories consequently characterize the actuation demands of the learned policy within the simulation, but they should not be interpreted as directly achievable commands for a particular physical actuator.

Several limitations remain. First, the mechanical parameters have not been calibrated against a physical dual-arm platform, and the simulations do not include actuator dynamics, sensor noise, estimation errors, communication delays, material variability, or assembly errors. Second, gravity and gravity-compensation errors are neglected, which may result in more favorable tracking performance than would be obtained on a physical platform. Third, although an extensible inter-arm safety formulation is presented, explicit contact modeling and collision avoidance are not activated or experimentally evaluated. Fourth, the current task is limited to non-contact shared-target tracking and does not include contact-rich cooperative operations such as grasping, handling, or force regulation. Finally, the algorithm comparison remains limited in terms of random seeds and hyperparameter configurations. These limitations define the scope of the present conclusions and motivate further simulation and physical validation.

5. Conclusions

This study developed a centralized reinforcement-learning framework and a physics-based simulation environment for non-contact shared-target tracking by two soft robotic manipulators. The environment integrates two Cosserat-rod models in PyElastica, an 81-dimensional joint observation, a 24-dimensional B-spline-parameterized action, and a reward function that accounts for the tracking performance of both end effectors. The framework was evaluated using circular, rectangular, and figure-eight target trajectories and was compatible with PPO, DDPG, and TD3. Across three random seeds, TD3 achieved the highest mean final-stage episode return among the evaluated algorithms. The independently controlled single-arm policies remained capable of completing the tracking tasks, but the centralized controller produced lower RMSE values for both manipulators under the rectangular and figure-eight trajectories, while the circular trajectory produced mixed arm-dependent results. The recorded moment histories further quantified the actuation demands of the learned policy, with overall maximum absolute normal and binormal moments of 138.9 and 121.7 N·m, respectively.

These findings support the centralized formulation as a practical simulation-level basis for studying coordinated control of multiple soft manipulators, while avoiding claims of direct readiness for physical implementation. Future work will incorporate actuator and sensor models, gravity, parameter uncertainty, and domain randomization to evaluate robustness and simulation-to-physical transfer. Explicit inter-arm contact modeling and collision-aware objectives will also be introduced, followed by extensions to contact-rich cooperative tasks such as dual-arm grasping, object handling, and force regulation. Physical implementation will ultimately be required to validate the achievable actuation range, sensing requirements, and real-world tracking performance of the proposed framework.