GT-TD3: A Kinematics-Aware Graph-Transformer Framework for Stable Trajectory Tracking of High-Degree-of-Freedom (DOF) Manipulators

Abstract

Accurate trajectory tracking of redundant manipulators is difficult because the controller must simultaneously model local couplings between adjacent joints and global dependencies across the whole kinematic chain. Existing reinforcement learning methods typically employ multilayer perceptrons, which do not explicitly exploit manipulator structure and therefore show limited stability and representation ability in high-dimensional continuous control tasks. This paper proposes GT-TD3, a Graph Transformer-enhanced-Twin Delayed Deep Deterministic Policy Gradient framework, for redundant manipulator trajectory tracking. The proposed actor first converts the raw system state into joint-level node features and uses a graph neural network to extract local kinematic coupling information. A Transformer is then employed to capture long-range dependencies among joints. To strengthen the use of structural priors, topology- and distance-related bias terms are incorporated into the attention mechanism, enabling the network to encode manipulator structure during global feature learning. Experiments on a 7-DoF KUKA iiwa manipulator in PyBullet demonstrate that GT-TD3 outperforms MLP, pure GNN, and pure Transformer baselines in tracking performance. The proposed method achieves more stable training, faster convergence, and smoother and more accurate end-effector motion. The results show that the integration of local graph modeling and structure-aware global attention provides an effective solution for high-precision trajectory tracking of redundant manipulators.

1. Introduction

As robotics technology rapidly advances, the precise control of end-effectors has emerged as a primary research focus, directly determining task efficiency and safety in high-risk applications [1,2,3,4]. This requirement is particularly evident in offshore oil drilling. Here, robotic arms are increasingly deployed for pipe handling where tasks demanding stringent trajectory precision to prevent collisions, alignment errors, and equipment damage within confined, easily perturbed spaces. Similar precision constraints apply to automated assembly and the dismantling of nuclear facilities, where even marginal deviations risk structural failure [5]. Furthermore, enabling redundant and continuum robots to navigate unstructured environments requires flexible path planning combined with dynamic obstacle avoidance [6]. Overcoming nonlinear dynamics and external disturbances to meet these strict criteria remains a substantial challenge.

Traditional control strategies, such as proportional-integral-derivative (PID) [7] and computed torque control, typically depend on exact mathematical models. Consequently, their performance degrades rapidly in the presence of modeling errors. Deep reinforcement learning (DRL) introduces a more adaptable alternative for manipulator control. Frameworks ranging from value-based approaches like DQN [8] and Double Q-learning [9] to actor–critic architectures like DDPG and TD3 [10] learn control policies directly through environmental interaction [11]. Despite their adaptability, standard DRL methods struggle with high-dimensional continuous control. While recent studies have successfully integrated neural network control [12], CNN-based predictive models [13], and variable impedance control [14] into robotic tasks, conventional multilayer perceptron (MLP) policies exhibit inherent limitations. The complex structures of high-degree-of-freedom systems generate expansive state and action spaces that easily overwhelm MLPs. More importantly, standard MLPs fail to exploit the inherent structural relationships along the robotic kinematic chain. Even though algorithms like TD3 enhance training stability via a twin-critic mechanism [10], their inability to explicitly encode joint connectivity restricts both learning efficiency and ultimate control performance.

To address these representation limits, neural architectures incorporating structural information offer a promising alternative. Graph neural networks (GNNs), for instance, naturally align with the interconnected joint structure of robotic arms by modeling node relationships through localized message passing [15,16,17]. Conversely, Transformers excel at capturing long-range dependencies via self-attention, demonstrating utility in navigation and obstacle avoidance [18]. Relying solely on standard Transformers, however, is suboptimal for manipulator control. Because the vanilla self-attention mechanism operates without joint topology or kinematic distance priors, the network must relearn these physical constraints from scratch. This observation suggests the need for an attention bias mechanism rooted in structural priors, effectively forcing the global dependency modeling process to respect the physical kinematic chain.

Motivated by this insight, we propose a structure-aware control framework named the Graph Transformer-Twin Delayed Deep Deterministic Policy Gradient (GT-TD3). Our approach integrates both GNN and Transformer modules directly into the TD3 actor architecture. Specifically, the GNN extracts local kinematic coupling features [19] while the Transformer, augmented with a kinematic-aware attention bias, models global joint dependencies [20]. By embedding topology and distance priors into the attention calculation via a biased attention mechanism, GT-TD3 significantly enhances the state representation capability without altering the underlying off-policy training dynamics of standard TD3. We evaluated this framework on a 7-DoF KUKA LBR iiwa manipulator within the PyBullet physics engine. Compared to standard MLP, pure GNN, and pure Transformer baselines, experimental results demonstrate that GT-TD3 achieves substantially higher algorithmic stability, faster convergence rates, and executes smoother, higher-precision end-effector trajectories.

2. Related Work

2.1. From Traditional Control to Deep Reinforcement Learning

Traditional methods, ranging from proportional-integral-derivative (PID) and computed torque control [21] to sampling-based planners such as A* [7] and rapidly exploring random trees (RRT) [22], provide a solid foundation for robotic motion generation and control. However, their practical performance often depends on accurate analytical models and careful parameter tuning. In real-world applications, modeling errors, actuator uncertainties, sensor noise, and external disturbances can substantially reduce their effectiveness, especially in unstructured or dynamically changing environments. These challenges become more pronounced in high-dimensional robotic systems, where joint coupling and control complexity make conventional design increasingly difficult.

To address these limitations, deep reinforcement learning (DRL) has emerged as a promising alternative for continuous robotic control. By learning policies directly from interaction data, DRL offers stronger adaptability in nonlinear and high-dimensional settings. In particular, end-to-end actor–critic methods such as DDPG, TD3 [10], and soft actor–critic (SAC) have achieved strong results in dexterous manipulation and navigation tasks [11].

2.2. Structured Neural Architectures in Robotic Control

To overcome the representation bottlenecks of standard multilayer perceptrons (MLPs), researchers have increasingly integrated structural priors into neural architectures. Graph neural networks (GNNs) exemplify this shift, naturally accommodating the chain-like topology of robotic manipulators by explicitly embedding joint relationships [16,23]. Frameworks such as NerveNet [24] and graph policy gradients [3], alongside various graph-based motion planners [17,25,26], successfully leverage localized message passing for modular robot control.

Nevertheless, GNNs exhibit inherent limitations when scaled to high-degree-of-freedom systems. Because they depend exclusively on local message passing, transferring information between distant nodes—such as from the robot base to the end-effector—requires numerous propagation steps. As theoretically analyzed by Alon and Yahav [19], this prolonged routing often induces over-smoothing and significantly impedes global information exchange.

Conversely, Transformers excel at modeling long-range dependencies, utilizing self-attention mechanisms to establish direct connections between distant states across various robotic applications [18,26]. Yet, applying vanilla Transformers to robotic control introduces a different set of challenges. Standard implementations treat joint states as unconstrained sequences, essentially discarding the robot’s physical topology. As highlighted in the recent literature [20], stripping away these inherent constraints forces the network to redundantly relearn fundamental kinematics from scratch, thereby degrading learning efficiency. Furthermore, the absence of explicit structural priors compromises local feature perception, frequently triggering training instability during continuous control [27]. To reconcile these conflicting paradigms, our framework equips the Transformer with a kinematic-aware attention bias module. By encoding joint topology, relative configurations, and physical distances directly into the attention matrix, the network is compelled to respect the manipulator’s kinematic realities during global dependency modeling, thereby maximizing structural utilization and mitigating inefficient exploration.

2.3. Position of Our Work

While the application of neural networks to robotic inverse kinematics (IK) is well-documented, ranging from early foundational explorations by Tejomurtula and Kak [28] to more recent validations of approximation accuracy and training stability by Gao [29] and Cagigas-Muñiz [30], our research diverges fundamentally from simply reaffirming these capabilities. Instead, the specific novelty and improvements of this paper manifest across three distinct dimensions.

First, whereas the traditional literature predominantly formulates IK as a static mapping problem, we reframe it as a continuous, dynamic control challenge. By continuously outputting joint velocity commands based on instantaneous states and goal vectors, our approach is inherently optimized for the fluid trajectory tracking demands of a 7-DoF redundant manipulator.

Second, rather than relying on unstructured dense layers to approximate kinematic relationships, we synthesized a hybrid GNN–Transformer architecture. By natively incorporating link distances, joint bounds, and physical symmetries into the network’s computational graph, the policy actively leverages the exact structural fingerprint of the robotic arm itself.

Finally, instead of evaluating success purely via isolated inverse solution errors, our framework simultaneously addresses end-effector precision, aggregate trajectory deviations, and action smoothness constraints. Consequently, our proposed reinforcement learning paradigm moves beyond computing mere reachable waypoints, delivering execution-ready, structure-integrated continuous control that is significantly more stable for real-world deployment.

3. Methods

3.1. Overall Framework of GT-TD3

Our study addresses the challenge of target-conditioned trajectory tracking control by employing a 7-degree-of-freedom KUKA iiwa robotic arm within the PyBullet simulation environment. At the initialization of each episode, the environment randomly samples a reachable Cartesian target point, prompting the system to generate a straight reference trajectory that spans from the end-effector’s initial position directly to this goal. To standardize the execution phase, the temporal length of this trajectory is strictly fixed at 200 control steps. Consequently, the fundamental learning objective of the policy is to formulate continuous joint velocity commands. These control signals must not only propel the end-effector accurately toward the target point, but they must also closely track the reference path while ensuring overall kinematic smoothness.

In this work, we developed GT-TD3 on top of the standard TD3 framework specifically to tackle high-degree-of-freedom robotic arm control. To keep training stable, we exclusively enhanced the actor network’s architecture, leaving the critic branch with its traditional twin Q-network setup. The core intuition behind GT-TD3 is as follows: instead of feeding the network a raw, flat state vector, we first reconstruct it into joint-level nodes. A GNN then extracts the localized coupling relationships between adjacent joints. After that, a Transformer equipped with a kinematic-aware structural bias steps in to model the long-range global dependencies. Finally, a gated fusion module seamlessly blends these local and global features to output the continuous control action.

Rather than directly combining a GNN and a Transformer into TD3, the proposed actor is constructed through four sequential steps. First, we encode the raw environmental state into seven individual joint nodes. Second, we drive local message passing across these nodes based strictly on the robot’s serial kinematic chain. Third, we map these graph-processed features into the Transformer space, injecting a kinematic-aware attention bias right on top of the standard sinusoidal positional encoding. Fourth, a gated fusion and readout module computes the actor’s final action. This architecture enables the policy network to jointly exploit local topological structure and long-range inter-joint dependencies. The overall architecture of the proposed GT-TD3 framework is illustrated in Figure 1. Figure 1 summarizes the main components of the proposed framework, which are described in detail in the following sections.

3.2. TD3-Based Learning Framework

This method is built on the TD3 framework, but the main modification is made on the actor side, where the policy function ${\pi }_{\theta }\left(s_t\right)$ is redesigned as a GNN–Transformer network that explicitly follows the joint structure of the manipulator. The critic branch remains unchanged.

As shown in Figure 2, the overall framework still follows the asymmetric actor–critic design of TD3. In other words, only the policy network is structurally enhanced.

More specifically, the method employs two independent critics, denoted as $Q_{{\varphi }_1}(s_t,a_t)$ and $Q_{{\varphi }_2}(s_t,a_t)$. Both critics use standard multilayer perceptrons to model the concatenated state-action pair. The detailed network architecture is shown in Figure 3. This keeps the value branch simple. It also helps maintain stable training and provides the actor with value gradients that are less affected by estimation bias.

Given a transition sample $(s_t,a_t,r_t,s_{t+1},d_t)$, where $d_t\in \{0, 1\}$ is the terminal flag, the non-terminal mask is defined as $m_t=1-d_t$. The target value in TD3 is then written as:

$$y_t=r_t+\gamma m_t\underset{k\in \{1,2\}}{\mathrm{m}\mathrm{i}\mathrm{n}}{Q}_{{\varphi }_k^{\prime }}(s_{t+1},{\tilde{a}}_{t+1})$$

(1)

where the next action is generated by target policy smoothing:

$${\tilde{a}}_{t+1}=\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}({\pi }_{{\theta }^{\prime }}(s_{t+1})+ϵ, -a_{\mathrm{m}\mathrm{a}\mathrm{x}}, a_{\mathrm{m}\mathrm{a}\mathrm{x}})$$

(2)

$$ϵ=\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}(\mathcal{N}(0,{\sigma }^{2}I), -c, c)$$

(3)

Here, ${\theta }^{\prime }$ and ${\varphi }_k^{\prime }$ denote the target parameters of the actor and the two critics, respectively. The symbols $\sigma$ and $c$ represent the standard deviation of the policy noise and the clipping threshold of that noise. This formulation follows the standard TD3 setting.

Like the original TD3, both critics are trained to regress toward the same clipped double-Q target. The loss function, however, is slightly different. Instead of mean squared error, this implementation uses the Smooth L1 loss:

$$L_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{c}}=\mathrm{S}\mathrm{m}\mathrm{o}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{L}1(Q_{{\varphi }_k}(s_t,a_t), y_t),k\in \{1,2\}$$

(4)

This choice improves robustness when the TD error contains occasional large values and makes critic training less sensitive to outliers. The two critics are updated with separate optimizers, and their target networks are softly updated after each training iteration. The actor, in contrast, follows the delayed update mechanism of TD3, which means that it is updated once every $d$ critic updates.

The actor objective in this implementation is defined as:

$$L_{\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}}=-{\mathbb{E}}_{s_t\sim \mathcal{D}}[Q_{{\varphi }_1}(s_t,{\pi }_{\theta }(s_t))]+\lambda {\mathbb{E}}_{s_t\sim \mathcal{D}}[\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}({\pi }_{\theta }(s_t{)}^2)]$$

(5)

Here, $\lambda$ is 0.001. The first term is the standard deterministic policy gradient objective. The second term is an action-magnitude regularizer. This additional regularization term helps alleviate the weak-gradient problem that can appear when the $tanh$ output approaches saturation. This, therefore, makes policy optimization more stable in practice.

After each actor update, the actor target network is also softly updated. In addition, the gradient norms of both the actor and the critics are clipped to an upper bound of 1.0. This further improves the training stability. Therefore, the contribution of this method does not come from changing the off-policy learning rule itself. The novelty lies in the actor design. More specifically, the main contribution is the combination of structured state representation and hierarchical feature modeling on the policy side. This is where the method differs from standard TD3. By placing the structural prior only in the actor, the method keeps the critic simple, stable, and easy to optimize. At the same time, it avoids the extra instability that graph modules or attention layers may introduce into Q-value estimation. The overall framework of our improved GT-TD3 algorithm is presented in Algorithm 1.

Algorithm 1. Training Procedure of GT-TD3

Input: actor ${\pi }_{\theta }$, twin critics $Q_{{\varphi }_1},Q_{{\varphi }_2}$, target networks ${\pi }_{{\theta }^{\prime }}$, $Q_{{\varphi }_1^{\prime }}$, $Q_{{\varphi }_2^{\prime }}$, replay buffer $\mathcal{D}$, total steps $T$, warm-up steps $T_0$, policy delay $d$, discount factor $\gamma$, soft update rate $\tau$, target noise $\sigma$, clipping threshold $c$ Initialize target networks: ${\theta }^{\prime }\leftarrow \theta$, ${\varphi }_1^{\prime }\leftarrow {\varphi }_1$, ${\varphi }_2^{\prime }\leftarrow {\varphi }_2$

3.3. Joint-Based State Encoding

To better exploit the kinematic structure of the 7-DoF serial manipulator, the raw 20-dimensional environment state is first reorganized into seven joint-level nodes. The raw state is defined as:

$$s_t=\left[q_t,{\dot{q}}_t,p_{ee,t},g_t\right]$$

(6)

where $q_t\in {\mathbb{R}}^7$ denotes the joint positions, ${\dot{q}}_t\in {\mathbb{R}}^7$ denotes the joint velocities, $p_{ee,t}\in {\mathbb{R}}^3$ denotes the current end-effector position, and $g_t\in {\mathbb{R}}^3$ denotes the goal position.

The encoder first constructs several goal-related quantities. These terms provide a compact description of the current task.

$$v_t=g_t-p_{ee,t}$$

(7)

$${\rho }_t=\parallel v_t{\parallel }_2$$

(8)

$${\widehat{g}}_t={\displaystyle \frac{v_t}{\mathrm{m}\mathrm{a}\mathrm{x}({\rho }_t,\epsilon )}}$$

(9)

Here, ${\rho }_t$ is the Euclidean distance from the end effector to the goal, ${\widehat{g}}_t$ is the unit direction vector pointing toward the goal, and $\epsilon$ is a small constant introduced to avoid division by zero.

The scale-dependent quantities are then normalized. This keeps the input range more stable during training. Specifically, joint velocities are normalized by the maximum velocity bound of 1.5 rad/s, while the goal distance is normalized by the workspace scale of 1.2 m. In addition, a cumulative joint-angle feature is introduced to describe the accumulated pose from the base to the current joint:

$$q_{i,t}^{\mathrm{c}\mathrm{u}\mathrm{m}}={\displaystyle \frac{\sum _{k=1}^i{q}_{k,t}}{7\times 3.0}}$$

(10)

Under the default setting, each joint node is assigned a 6-dimensional hand-crafted feature vector:

$${\tilde{x}}_{i,t}=\left[q_{i,t},{\displaystyle \frac{{\dot{q}}_{i,t}}{1.5}},q_{i,t}^{\mathrm{c}\mathrm{u}\mathrm{m}},{\displaystyle \frac{{\rho }_t}{1.2}},{\widehat{g}}_{x,t},{\widehat{g}}_{y,t}\right]$$

(11)

One detail should be clarified here. The default implementation does not use the full three-dimensional direction vector $\left({\widehat{g}}_x,{\widehat{g}}_y,{\widehat{g}}_z\right)$. Instead, only ${\widehat{g}}_x$ and ${\widehat{g}}_y$ are retained in the node feature. This choice mainly serves to control the node dimension, and it is also reasonable because ${\widehat{g}}_t$ is a unit vector, so part of the information in ${\widehat{g}}_z$ can still be reflected indirectly through the other components.

To help the network distinguish different physical joints, a 1-dimensional learnable joint identity embedding $e_i$ is further added to each node. It is concatenated with the hand-crafted feature as:

$$x_{i,t}=[{\tilde{x}}_{i,t},e_i]$$

(12)

As a result, the actual input dimension of each node is 7 under the default setting. Six dimensions come from the hand-crafted state features, and the last dimension comes from the learnable joint identity embedding. The final joint-node sequence is written as:

$$X_t=[x_{1,t},x_{2,t},\dots ,x_{7,t}{]}^{\mathrm{\top }}\in {\mathbb{R}}^{7\times 7}$$

(13)

This encoding preserves information at three different levels. It is not just a simple feature split. First, $q_{i,t}$ and ${\dot{q}}_{i,t}$ describe the instantaneous state of each joint. Second, $q_{i,t}^{\mathrm{c}\mathrm{u}\mathrm{m}}$ reflects the accumulated pose from the base to joint $i$, which helps characterize the overall manipulator configuration. Third, ${\rho }_t$, ${\widehat{g}}_{x,t}$, and ${\widehat{g}}_{y,t}$ broadcast task-related goal information to every joint node. In this way, both the later graph module and the Transformer module can perceive the current control objective from local and global perspectives.

3.4. Local Dependency Modeling Through Gated Graph Aggregation

After joint-level encoding, the seven nodes are connected according to the serial kinematic topology of the manipulator. This gives the model an explicit structural prior. Let $A\in {\mathbb{R}}^{7\times 7}$ denote the adjacency matrix with self-loops, and let row-wise normalization be applied to $A$. With this design, each joint preserves its own feature during propagation while also receiving messages from its upstream and downstream neighbors.

Before graph aggregation, the node features are first projected into a unified hidden space using a two-layer node-wise MLP:

$$h_i^{\left(0\right)}={\mathrm{M}\mathrm{L}\mathrm{P}}_{\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}}(x_{i,t})$$

(14)

The default hidden dimension is $d_g=64$. This projection prepares the node features for subsequent local interaction.

A two-layer gated graph aggregation module is then used to capture local dependencies. For the $\mathcal{l}$-th layer, the update is defined as:

$$m_i^{\left(\mathcal{l}\right)}=W_m^{\left(\mathcal{l}\right)}\sum _j{A}_{ij}{h}_j^{\left(\mathcal{l}\right)}$$

(15)

$$z_i^{\left(\mathcal{l}\right)}=\sigma \left(W_z^{\left(\mathcal{l}\right)}[ h_i^{\left(\mathcal{l}\right)}\parallel m_i^{\left(\mathcal{l}\right)} ]\right)$$

(16)

$$h_i^{\left(\mathcal{l}+1\right)}=\mathrm{L}\mathrm{N}\left(z_i^{\left(\mathcal{l}\right)}\odot h_i^{\left(\mathcal{l}\right)}+\left(1−z_i^{\left(\mathcal{l}\right)}\right)\odot m_i^{\left(\mathcal{l}\right)}\right)$$

(17)

Here, $h_i^{\left(\mathcal{l}\right)}$ denotes the hidden feature of the $i$-th joint node at layer $\mathcal{l}$, $m_i^{\left(\mathcal{l}\right)}$ denotes the aggregated neighborhood message, and $z_i^{\left(\mathcal{l}\right)}$ denotes the element-wise gating vector. The function $\sigma (\cdot )$ is the sigmoid activation, and $\mathrm{L}\mathrm{N}(\cdot )$ denotes layer normalization. The learnable matrices $W_m^{\left(\mathcal{l}\right)}$ and $W_z^{\left(\mathcal{l}\right)}$ are used for neighborhood message mapping and gate generation, respectively. The detailed structure of this local dependency modeling module is depicted in Figure 4.

This design allows each joint to adaptively balance two sources of information. One comes from the joint itself, and the other comes from its local kinematic neighborhood. When neighboring information is more informative, the model can place more weight on $m_i^{\left(\mathcal{l}\right)}$. When the current joint state is more critical, it can preserve more of $h_i^{\left(\mathcal{l}\right)}$. This mechanism is especially suitable for the KUKA iiwa, which is a typical serial manipulator with strong local coupling between adjacent joints.

3.5. Kinematic-Aware Transformer with Sinusoidal Positional Encoding

After the local graph aggregation stage, the model obtains joint-level local representations:

$$H_t^G=[h_1^G,\dots ,h_7^G{]}^{\mathrm{\top }}\in {\mathbb{R}}^{7\times d_g}$$

(18)

These features are not fed into the Transformer directly. To make them suitable as Transformer tokens, they are first projected into a shared hidden space through a linear mapping:

$$Z_t^{\left(0\right)}=H_t^G{W}_{in}+b_{in}\in {\mathbb{R}}^{7\times d}$$

(19)

where the default hidden dimension is $d=128$.

One point should be clarified here. The Transformer does not take the raw state as input. Instead, it receives the joint representations that have already been processed by the GNN. The two modules are therefore connected in series rather than in parallel. The model first captures local dependencies and then models global interactions. Before entering the first self-attention layer, standard sinusoidal positional encoding is added to the token sequence:

$${\tilde{Z}}_t^{\left(0\right)}=Z_t^{\left(0\right)}+E_{pos}$$

(20)

where $E_{pos}\in {\mathbb{R}}^{7\times d}$ denotes the sinusoidal positional encoding matrix.

This step preserves the order information of the joint sequence. It also helps the Transformer distinguish different joint positions within the chain. Under the default setting, the Transformer encoder contains two self-attention layers and four attention heads. It adopts a Pre-LayerNorm architecture and uses a feed-forward network with a fourfold expansion ratio. The internal structure of the kinematics-aware Transformer encoder is presented in Figure 5.

Unlike physically grounded kinematics-aware formulations such as Sheng et al. (2024) [31], the mechanism used here does not explicitly model manipulator dynamics or Jacobian constraints. Instead, it introduces a kinematics-aware structural bias into the self-attention computation by using joint topology, path distance, and joint-range priors derived from the serial manipulator. The structural prior matrix is defined as:

$$P=C-D+S$$

(21)

where $D$ is the normalized path-distance matrix, $C$ is the motion-range coupling matrix, and $S$ is the symmetry reward matrix. Among these three terms, $D$ describes how far two joints are from each other along the real kinematic chain. It is not a simple index distance. For the KUKA iiwa 7, the adjacent joint-link distances are set as:

$$l=[132.9, 87.8, 151.4, 90.6, 122.3, 141.1, 80.4] \mathrm{mm}$$

and the corresponding joint motion ranges are:

$$r=[170°, 120°, 170°, 120°, 170°, 120°, 175°]$$

For any joint pair $\left(i, j\right)$, the normalized path distance is defined as:

$$D_{ij}={\displaystyle \frac{\sum _{k=\mathrm{m}\mathrm{i}\mathrm{n}(i,j)}^{\mathrm{m}\mathrm{a}\mathrm{x}(i,j)-1}{l}_k}{\underset{u,v}{\mathrm{m}\mathrm{a}\mathrm{x}}\sum _{k=\mathrm{m}\mathrm{i}\mathrm{n}(u,v)}^{\mathrm{m}\mathrm{a}\mathrm{x}(u,v)-1}{l}_k}},D_{ij}\in [\mathrm{0,1}]$$

(22)

This term penalizes attention between joints that are farther apart along the manipulator chain. In this way, the model can retain a clearer sense of kinematic structure during global interaction.

The motion-range coupling term is defined as:

$$C_{ij}={\displaystyle \frac{r_i{r}_j}{\underset{u,v}{\mathrm{m}\mathrm{a}\mathrm{x}}(r_u{r}_v)}},i\ne j$$

(23)

with $C_{ii}=0$.

This term assigns larger values to joint pairs with broader motion capability. It reflects the obvious intuition that such joints are more likely to play a stronger role in coordinated movement. In addition, the KUKA iiwa 7 shows an alternating pattern of large-range and small-range joints. To describe this pattern, a simple symmetry reward is introduced:

$$S_{ij}=\left\{\begin{array}{cc}0.1,& \mathrm{if} r_i \mathrm{and} r_j \mathrm{belong} \mathrm{to} \mathrm{the} \mathrm{same} \mathrm{kinematic} \mathrm{category}\\ 0,& \mathrm{otherwise}\end{array}\right.$$

(24)

The final structural prior is therefore written as:

$$P_{ij}=C_{ij}-D_{ij}+S_{ij}$$

(25)

Based on this prior, the bias term for the $h$-th attention head is defined as:

$$B_h={\alpha }_{h}P+\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}({\mathsf{\Delta }}_h)$$

(26)

where ${\alpha }_h$ is a learnable head-level scaling factor and ${\mathsf{\Delta }}_h\in {\mathbb{R}}^{7\times 7}$ is a learnable residual bias matrix.

The attention of the $h$-th head is then computed as:

$${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_h(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Q_h{K}_h^{\mathrm{\top }}}{\sqrt{d_h}}}+B_h\right)V_h$$

(27)

where $Q_h$, $K_h$, and $V_h$ are obtained by linear projections of the current layer input.

This design makes the attention mechanism more likely to focus on structurally related joint pairs. At the same time, $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}({\mathsf{\Delta }}_h)$ keeps a learnable correction term, so the model does not rely only on fixed prior information. For this reason, the module is described here as kinematic-aware rather than dynamic-aware. The added bias reflects manipulator structure, but it does not explicitly model dynamic equations, torques, or inertial effects. Besides the attention bias, the model also introduces node-level hierarchical weights based on joint motion ranges:

$$w_i=0.6+0.6\cdot {\displaystyle \frac{r_i-r_{\mathrm{m}\mathrm{i}\mathrm{n}}}{r_{\mathrm{m}\mathrm{a}\mathrm{x}}-r_{\mathrm{m}\mathrm{i}\mathrm{n}}}},w_i\in [\mathrm{0.6,1.2}]$$

(28)

These weights are not applied to the softmax attention probabilities and are only used to rescale the node outputs after attention aggregation. If the attention output of node $i$ is denoted as $o_i$, the reweighted output is written as:

$$o_i^{\prime }=w_i{o}_i$$

(29)

This operation gives relatively more emphasis to joints with larger motion ranges. It is a simple design, but it helps reflect structural differences across joints at the feature level. One final point should be noted. In the current implementation, the structural prior parameters are built from fixed constants extracted from the KUKA iiwa URDF.

However, the proposed method itself is not tied to one specific robot platform, and its design is inherently generalizable to different serial manipulators. For any serial manipulator, the same construction procedure can be followed as long as the adjacent joint distances $l_k$ and joint motion ranges $r_i$ are available from the URDF file, DH parameters, or manufacturer specifications. Based on this information, the kinematics-aware prior matrix can be constructed in the same manner and then incorporated into the attention computation of the Transformer encoder. If the target robot does not exhibit an obvious symmetric pattern, the symmetry term can simply be set to $S=0$. In that case, the structural bias can still be effectively formed using the topology-based distance term and the joint-range-related term. This design gives the proposed method both structural expressiveness and practical flexibility when adapting to different robotic systems. Thus far, the configurations of the graph topology, GNN encoder, Transformer specifications, and activation function have been presented, with detailed settings shown in

Table 1. Detailed configuration of the GT-TD3 network architecture, including the graph topology, GNN encoder, transformer specifications, and activation.

Module	Parameter	Value	Configuration Details
Graph Topology	Nodes/Edges	$\mathrm{N}=7$	1-hop chain graph with self-loops
	Configured dim	${\mathrm{d}}_{\mathrm{node}}=6$	$\left[q_{i,t},{\displaystyle \frac{{\dot{q}}_{i,t}}{1.5}},q_{i,t}^{\mathrm{c}\mathrm{u}\mathrm{m}},{\displaystyle \frac{{\rho }_t}{1.2}},{\widehat{g}}_{x,t},{\widehat{g}}_{y,t}\right]$
GNN Encoder	Layers	2	Gated message passing with LayerNorm
	Hidden dimension	64	Row-normalized adjacency
	Readout	Mean + Max	Global pooling
Transformer	Encoder layers	2	KAPE-based encoder
	Attention heads	4	Multi-head self-attention
	Embedding dim	${\mathrm{d}}_{\mathrm{model}}=128$	Latent feature space
	FNN dimension	${\mathrm{d}}_{\mathrm{ff}}=256$	Expansion ratio = 4
Activation	Hidden/Output	LeakyReLU/ELU/Tanh	Implementation-consistent

.

3.6. Cross-Scale Feature Fusion and Action Output Head

After the graph module, the network captures local kinematic features at the joint level. After the Transformer module, it obtains global features that describe long-range dependencies across joints. These two kinds of information are not equivalent. They should not simply overwrite each other. For this reason, the model does not directly replace the GNN representation with the Transformer output. Instead, it performs a gated cross-scale fusion in a shared hidden space, so that local structure and global context can be preserved at the same time.

More specifically, let $g_i\in {\mathbb{R}}^d$ denote the local feature obtained by linearly projecting the GNN output, and let $t_i\in {\mathbb{R}}^d$ denote the global feature produced by the Transformer encoder. Each joint therefore has two feature views. One is local. The other is global. For the $i$-th joint node, the fusion gate is defined as:

$${\beta }_i=\sigma (W_c[g_i\parallel t_i]+b_c)$$

(30)

and the fused feature is written as:

$$f_i=\mathrm{L}\mathrm{N}({\beta }_i\odot g_i+(1-{\beta }_i)\odot t_i)$$

(31)

Here, ${\beta }_i$ is an element-wise gating vector. It controls how much information should be taken from the local branch and how much should be taken from the global branch.

This design allows the fusion process to adapt across joints and states. When the motion of one joint depends more strongly on nearby kinematic constraints, the model can keep more of $g_i$. When long-range coupling becomes more important, it can place more weight on $t_i$. The fused node set is then written as:

$$F=\{f_i{\}}_{i=1}^7$$

(32)

At this stage, the network has already integrated both local and global information. A compact readout is still needed. Rather than flattening all node features directly, the model uses two parallel pooling paths, namely the mean pooling and max pooling:

$$f_{\mathrm{global}}=[\mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}(F)\parallel \mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}(F)]$$

(33)

This readout is more compact than direct flattening. It also provides a more stable global summary of the joint set.

Mean pooling captures the average trend of all fused node features. Max pooling, in contrast, highlights the most salient responses among the joints. By combining the two, the model can preserve both overall information and strong local activations. This makes the final representation more suitable for action generation. The deterministic action is finally produced by a lightweight MLP:

$$a_t=a_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(\mathrm{M}\mathrm{L}\mathrm{P}(f_{\mathrm{global}}))$$

(34)

In the default implementation, this action head contains one hidden layer with 256 units and uses the ELU activation function. The final output is a 7-dimensional joint velocity command. The outer $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(\cdot )$ and the scaling factor $a_{\mathrm{m}\mathrm{a}\mathrm{x}}$ explicitly constrain the action range.

As a result, the actor output can be directly used as the target joint-velocity control signal in the simulation environment. Overall, this output head is intentionally simple. The main modeling burden is placed on the structured encoder and the cross-scale fusion module, while the final action projection is kept lightweight and stable.

3.7. Reward Function

This study uses a dense reward function to encourage target approaching, trajectory tracking, successful arrival, and smooth control. The total reward is defined as:

$$r_t=0.3 r_{\mathrm{goal}}+0.4 r_{\mathrm{track}}+0.3 r_{\mathrm{success}}$$

(35)

The goal-reaching term is:

$$r_{\mathrm{g}\mathrm{o}\mathrm{a}\mathrm{l}}=-\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(d_g)+\mathrm{e}\mathrm{x}\mathrm{p}(-10d_g)$$

(36)

$$d_g={\parallel p_{ee}-p_{target}\parallel }_2$$

(37)

The tracking term is:

$$r_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{k}}=\mathrm{e}\mathrm{x}\mathrm{p}\left(-5{\parallel P_{\mathrm{e}\mathrm{e}}-P_{\mathrm{r}\mathrm{e}\mathrm{f}}(t)\parallel }_2\right)$$

(38)

where $P_{\mathrm{ref}}(t)$ is the reference point at the current step. The success reward is:

$$r_{\mathrm{s}\mathrm{u}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}}=I(d_g<ϵ)$$

(39)

where $ϵ$ is the target threshold.

This formulation emphasizes tracking accuracy while still considering final target convergence, task completion, and action continuity. The reward function in this study is mainly designed empirically, although it follows well-established reward-shaping principles in robot reinforcement learning [32,33]. This still remains a limitation of the present work.

4. Experiments and Results

4.1. Experimental Setup

For our primary experimental platform, we selected the 7-degree-of-freedom KUKA LBR iiwa 14 R820 manipulator. The simulation environment was constructed using the PyBullet physics engine, which provides a highly realistic framework for simulating rigid-body dynamics and complex collision behaviors. Within this environment, we executed the point-to-point trajectory tracking task (illustrated in Figure 6), leveraging an NVIDIA GeForce RTX 5070 GPU to accelerate the deep reinforcement learning training process.

The system’s state space, denoted as $s_t\in {\mathbb{R}}^{20}$, is composed of four distinct components: joint positions, joint velocities, the instantaneous Cartesian coordinates of the end-effector, and the Cartesian position of the target point. During the state encoding phase, the model dynamically derives supplementary features, such as goal direction and Euclidean distance, by calculating the spatial difference between the current end-effector location and the target. Conversely, the action space, $a_t\in {\mathbb{R}}^7$ directly outputs continuous velocity commands for each of the seven joints. We define one episode as successful when the end effector reaches the target within 200 simulation steps and the position error satisfies ${\parallel p_{ee}-p_{target}\parallel }_2<5 \mathrm{c}\mathrm{m}$.

To ensure statistical reliability, each model was evaluated across five random seeds (0, 1, 42, 123, and 2025), with the main text reporting the aggregated results of five primary seeds. Consequently, the reported learning curves and quantitative metrics accurately reflect policy performance across diverse initializations. Furthermore, we enforced strictly fair evaluation protocols across all baseline models by keeping the parameter scales and training conditions identical. The sole distinguishing factor among the tested policies is their underlying backbone network architecture, with all models sharing the standardized hyperparameters detailed in

Table 2. Hyperparameters and optimization settings used for training the deep reinforcement learning policies.

Parameter	Symbol	Value	Description
Total Timesteps	${\mathrm{T}}_{\mathit{total}}$	500,000	Maximum training timesteps
Warm-up Steps	${\mathrm{T}}_{\mathit{start}}$	25,000	Random exploration phase
Min-batch Size	$\mathrm{B}$	512	Increased for stability
Replay Buffer Size	$\mathrm{R}$	500,000	Extended memory capacity
Learning Rate	$\mathsf{\alpha }$	1 × 10−5	Unified for Actor/Critic
Discount Factor	$\mathsf{\gamma }$	0.99	Future reward weight
Soft Update Rate	$\mathsf{\tau }$	0.003	Target network update
Policy Delay	$\mathrm{d}$	2	Delayed Actor updates
Exploration Noise	$\mathsf{ϵ}$	0.1	Action noise (Gaussian)
Policy Smooth Noise	$\mathsf{\sigma }$	0.2	Target policy noise
Noise Clip	$\mathrm{c}$	0.5	Noise clipping range

.

4.2. Learning Dynamics and Convergence Analysis

To rigorously evaluate both learning efficiency and the quality of the learned policies, we monitored eight distinct metrics throughout the training process. The evaluation was divided into two primary analytical dimensions: Figure 7 illustrates the core task-level performance, while Figure 8 details the metrics associated with trajectory fidelity and operational safety.

4.2.1. Task Performance and Sample Efficiency

As illustrated by the task-level performance metrics in Figure 7, GT-TD3 significantly outperformed all baseline methods in both sample efficiency and final asymptotic performance. This superiority is clearly reflected in the learning curves for cumulative reward (Figure 7a) and success rate (Figure 7b). Specifically, our proposed framework not only converged faster but also yielded a substantially higher cumulative reward than the other three configurations. Furthermore, it rapidly achieved and sustained a dominant success rate after 350 k training steps. In stark contrast, the standard MLP baseline suffered from severe fluctuations throughout the entire training process. While the pure GNN and pure Transformer architectures managed to secure higher average success rates than the MLP, they still exhibited pronounced oscillatory behavior during learning.

Secondary evaluation metrics further corroborate these findings. The time-to-success curve (Figure 7c) demonstrates that GT-TD3 eventually converges to the lowest required temporal steps, indicating that the fused model executes the tracking task with greater speed and decisive action. Finally, the evaluation minimum distance metric (Figure 7d) confirms this steady convergence; GT-TD3 produced the smallest final error variance, proving that the hybrid structure guides the end-effector to the target zone with superior stability and precision.

4.2.2. Trajectory Fidelity and Kinematic Optimization

Beyond mere task completion, we rigorously evaluated the kinematic quality of the generated trajectories, with the quantitative findings detailed in Figure 8. In terms of tracking accuracy, both the Root Mean Square Error (RMSE) (Figure 8a) and end-point error (Figure 8c) demonstrated that GT-TD3 yields significantly smoother and narrower error curves. Ultimately, the fused architecture successfully converges to the lowest possible error threshold, whereas the pure GNN and Transformer baselines stagnate at notably higher error margins. This stark contrast underscores the absolute necessity of feature fusion for achieving precise position control.

From a safety perspective, the maximum deviation metric (Figure 8b) reveals that our proposed method adheres to much tighter spatial constraints along the entire reference path. Furthermore, an analysis of path length (Figure 8d) highlights the superior operational efficiency of GT-TD3. While alternative models successfully reach the target region, their generated paths are frequently circuitous and winding. Conversely, GT-TD3 consistently converges to the shortest valid trajectory.

Collectively, these evaluations substantiate that the asymmetric fusion structure extends well beyond simply reaching a spatial goal; it inherently facilitates the generation of motion trajectories that are fundamentally more accurate, safer, and highly efficient.

4.3. Test Results Analysis

The comprehensive test results across all evaluated models, as displayed in Figure 9 and Figure 10, demonstrate that GT-TD3 largely mirrors the favorable behavioral trends initially observed during the training phase. With the minor exception of the minimum distance and end-point error metrics, where our framework was marginally outperformed by one specific baseline, GT-TD3 maintains a definitive evaluative advantage.

Broadly speaking, the proposed architecture not only delivered superior performance across the majority of the testing criteria but also exhibited significantly reduced performance fluctuations. These outcomes collectively validate that GT-TD3 substantially elevates the baseline trajectory tracking capabilities of the robotic manipulator while simultaneously ensuring remarkable model stability and robustness across multiple independent testing iterations.

4.4. Stability Analysis

This study uses Figure 11 and Figure 12 to analyze the stability of different models under disturbance conditions. This study denotes the end-effector tracking error as $e_k$. This study uses $V(e_k)= \parallel e_k{\parallel }^2$ as the candidate function in the stability analysis. This study further computes $\mathsf{\Delta }{V}_k=V(e_{k+1})-V(e_k)$. This study uses the negative dV ratio to denote the proportion of time steps with $\mathsf{\Delta }{V}_k<0$. This metric reflects whether the system error energy keeps decreasing over time. This study also uses the final tracking error, recovery step, and stability success rate as supporting metrics. Figure 11 shows how these metrics change under different levels of initial disturbance. Figure 12 shows the distributions of these metrics under the maximum disturbance, $\sigma =0.25 \mathrm{rad}$. These two sets of results describe model stability from two angles: one shows the overall trend and the other shows the behavior under strong disturbance.

As illustrated in Figure 11, the stability of all evaluated models predictably deteriorates as the magnitude of the initial disturbance increases. For the majority of policies, this heightened control difficulty manifests directly as inflated final tracking errors and correspondingly prolonged recovery steps. Despite this general trend of degradation across the baseline architectures, GT-TD3 distinguishes itself by sustaining a relatively high negative dV ratio across almost the entire perturbation spectrum. By consistently securing lower final errors while demanding fewer recovery steps, our proposed framework demonstrates superior robustness over the full range of disturbance conditions.

To dissect policy performance under the most extreme conditions, Figure 12 isolates the detailed outcomes at the maximum disturbance level. Within this visualization, the embedded boxplots mark the medians and interquartile ranges, while the varying widths of the violin plots explicitly capture the concentration of the outcome distributions. Individual trial results, overlaid as scatter points, further reveal that GT-TD3 yields not only superior median performance but also a highly concentrated data distribution. This tight clustering signifies that the fused architecture behaves with exceptional steadiness under intense perturbations, thereby minimizing the probability of generating overtly unstable control sequences. Conversely, although baseline models occasionally manage to recover in isolated trials, their broadly scattered outcome distributions betray a profound sensitivity to severe external disturbances.

Ultimately, the continuous trajectory evaluations in the line plots and the discrete extreme-case distributions in the violin plots mutually reinforce the same core conclusion. Synthesizing these analyses, it is evident that GT-TD3 not only elevates average task execution but also provides empirical evidence of improved stability.

4.5. Trajectory Tracking Evaluation

To provide a more granular assessment of trajectory quality, Figure 13 illustrates four representative end-effector tracking tasks. By visually contrasting the actual executed trajectories (solid lines) against their corresponding references (dashed lines), we can intuitively gauge tracking accuracy, motion smoothness, and convergence behavior. For a comprehensive comparison, the figure displays three randomly initialized runs for each evaluated policy.

The light blue trajectories, denoting the GT-TD3 control outcomes, showcase the most superior tracking proficiency among all candidates. The end-effector adheres tightly to the intended path with negligible lateral deviation and entirely fluid speed transitions, free from abrupt command jumps. More crucially, this policy achieves stable terminal convergence, effectively sidestepping the severe overshoot phenomena that frequently plague high-degree-of-freedom robotic control. This exceptional stability validates the premise of our hybrid architecture. By concurrently modeling localized structural couplings and overarching global dependencies, GT-TD3 orchestrates a far more coordinated control behavior across the entire operational workspace.

Conversely, the dark blue trajectories generated by the pure graph neural network (GNN) baseline expose the inherent limitations of relying solely on topological methodologies. While adept at preserving localized joint connections, the GNN policy suffers from pronounced S-shaped deviations during the early and intermediate stages of motion. This observation aligns with established theoretical findings: standard message-passing mechanisms possess restricted receptive fields, severely handicapping the model’s capacity to resolve long-range geometric relationships in the absence of a global context [19]. Although continuous feedback eventually allows the policy to correct these detours, the resulting path is marred by redundant turning maneuvers, yielding a cumulative control error substantially higher than that of our fused model.

The pure Transformer baseline (red trajectories) maintains overarching motion continuity, yet is plagued by a conspicuous control lag. This latency forces the end-effector into sluggish reactions during high-curvature maneuvers, precipitating visible spatial tracking errors. Such behavior substantiates recent critiques within the field. Because standard Transformers lack explicit kinematic structural priors despite their prowess in sequence modeling, their localized feature perception becomes alarmingly inefficient during continuous control, ultimately impeding response times [20,28]. Furthermore, the presence of mild oscillations near the final target highlights a secondary weakness: relying exclusively on self-attention mechanisms proves inadequate for the high-frequency fine-tuning actions required during terminal stabilization.

Finally, the yellow trajectories tracking the multilayer perceptron (MLP) baseline exhibit the most degraded performance. The motion is characterized by extreme jitter, severe mid-route deviations, and intense oscillatory behavior as the episode progresses. The model is trapped in a loop of constant directional overcorrections, a chaotic pattern that intensifies as the end-effector approaches its target. This vividly illustrates the classic dilemma faced by unstructured reinforcement learning policies deployed on high-degree-of-freedom hardware: they simply collapse under the weight of high-dimensional state spaces [34]. Bereft of both the structural priors inherent to GNNs and the overarching dependency frameworks provided by Transformers, the MLP cannot adequately filter state noise. Consequently, its heavy reliance on direct state mapping dooms it to an inescapable cycle of violent oscillation and reactive correction.

5. Discussion

Our experiments clearly show that relying on a single neural network architecture just is not enough to capture both the local and global dependencies needed for complex manipulator control. For example, because the MLP baseline uses a simple direct global mapping, it tends to suffer from continuous fluctuations and constant mid-course corrections. The GNN, on the other hand, is great at looking at neighboring joints, but it hits a wall when trying to understand long-range connections. Meanwhile, the pure Transformer is excellent at seeing the big picture globally, but it does not really understand the physical structure of the robotic arm out of the box.

This is exactly why GT-TD3 works so well, it brings together the best of both worlds. By blending local structural awareness with global dependency tracking, it creates a much smoother and more coordinated control process. What this really highlights is how important physical, structural priors are in continuous control tasks. A robotic arm is not just a sequence of numbers; it has a rigid physical chain and strict joint constraints. If we do not give the network a ““heads-up”” about these physical rules, it must learn them all from scratch, which makes training unnecessarily hard. Our structure-aware attention bias essentially bakes these rules directly into the model, making trajectory tracking much more stable.

Of course, this approach is not without its trade-offs. The structured design does add some training overhead compared to the simpler baselines. For instance, under identical hardware settings, training the MLP baseline took approximately 1.25 h, whereas the GT-TD3 model required 1.98 h. Though it introduced additional spatio-temporal complexity, we found the extra time to be completely manageable. There are also a few limitations to keep in mind. So far, we have only tested this on a 7-DoF KUKA arm in PyBullet, focusing mainly on simple point-to-point straight-line tracking. We have also not fed the model more complex physical dynamics like torque or contact forces yet, nor have we integrated real-time visual perception systems. Therefore, it definitely needs more testing in messier, real-world scenarios.

Still, we think that this method is very close to being ready for real hardware. In a physical setup, all the system really needs is real-time data on joint positions, velocities, and the end-effector pose, alongside the target path. Since the core of our method just relies on standard robot specs (like joint connections and limits), adapting it to a different manipulator should be pretty straightforward. In a physical setup, the deployment path follows a clear pipeline: state acquisition, target input, policy inference, velocity control output, safety-constrained execution. Looking ahead, we plan to try this out on different robot platforms and more complicated paths. We will also be looking into adding parameter randomization and noise injection to help bridge the sim-to-real gap and get this working reliably in the real world.

6. Conclusions

In this paper, we introduced GT-TD3, a reinforcement learning framework built specifically to handle the tricky task of trajectory tracking for redundant robotic arms. We essentially combined a graph neural network (GNN) and a Transformer inside the actor network of an asymmetric actor–critic setup. The GNN handles the local joint connections, while the Transformer looks at the global picture. By adding structural biases—like kinematic chain distances, joint limits, and symmetry features—we successfully fed the robot’s physical realities directly into the attention mechanism.

When we tested it on a 7-DoF KUKA arm, the results were promising. GT-TD3 outperformed the standard MLP, pure GNN, and pure Transformer models. The data showed that our method is not just more stable; it learns faster and produces much smoother, more precise end-effector movements. Ultimately, it demonstrates that combining local structure with global awareness is a highly robust way to solve high-precision robotic tracking problems. Moving forward, we plan to test this framework on robots with entirely different body structures and push it to handle even more complex real-world tasks.