Inverse Kinematics-Augmented Sign Language: A Simulation-Based Framework for Scalable Deep Gesture Recognition

Abstract

In this work, we introduce IK-AUG, a unified algorithmic framework for kinematics-driven data augmentation tailored to sign language recognition (SLR). Departing from traditional augmentation techniques that operate at the pixel or feature level, our method integrates inverse kinematics (IK) and virtual simulation to synthesize anatomically valid gesture sequences within a structured 3D environment. The proposed system begins with sparse 3D keypoints extracted via a pose estimator and projects them into a virtual coordinate space. A differentiable IK solver based on forward-and-backward constrained optimization is then employed to reconstruct biomechanically plausible joint trajectories. To emulate natural signer variability and enhance data richness, we define a set of parametric perturbation operators spanning spatial displacement, depth modulation, and solver sensitivity control. These operators are embedded into a generative loop that transforms each original gesture sample into a diverse sequence cluster, forming a high-fidelity augmentation corpus. We benchmark our method across five deep sequence models (CNN3D, TCN, Transformer, Informer, and Sparse Transformer) and observe consistent improvements in accuracy and convergence. Notably, Informer achieves 94.1% validation accuracy with IK-AUG enhanced training, underscoring the framework’s efficacy. These results suggest that algorithmic augmentation via kinematic modeling offers a scalable, annotation free pathway for improving SLR systems and lays the foundation for future integration with multi-sensor inputs in hybrid recognition pipelines.

1. Introduction

Sign language serves as a primary mode of communication for millions of individuals with hearing or speech impairments [1,2]. Bridging the communication gap between signers and non-signers has long been a central challenge in inclusive human–computer interaction [3,4]. With the rise of deep learning and computer vision, there has been considerable progress in automatic Sign Language Recognition (SLR), where gesture sequences are mapped to textual or spoken representations [5,6,7]. Yet despite recent advances, existing SLR systems often fall short in real-world settings, primarily due to a lack of scalable, diverse, and structurally consistent training data [8].

Most current datasets rely on constrained acquisition environments and limited signer diversity, which in turn restricts the generalization capabilities of trained models [9,10]. Moreover, traditional data augmentation techniques such as image-level transformations or synthetic recombination fail to capture the biomechanical realism and temporal coherence intrinsic to human gestures [11,12]. As a result, models trained on such data often struggle with signer variability, camera-view inconsistencies, and gesture ambiguity in practical applications [13].

To address these challenges, we propose a unified algorithmic framework, Inverse Kinematics-Augmented Gesture Generator (IK-AUG), which systematically synthesizes biomechanically plausible sign language sequences through kinematic modeling and structured simulation. Instead of relying on pixel-level transformations or stochastic recombination, IK-AUG formulates sign augmentation as a pose-to-motion reconstruction problem, where sparse 3D landmarks are lifted into full joint trajectories using an inverse kinematics solver embedded in a virtual environment.

The core of IK-AUG integrates a differentiable kinematic engine based on forward-and-backward constraint propagation with a set of multi-modal perturbation operators parameterized by spatial noise, depth offsets, and solver sensitivity modulation. These operators act as domain-specific augmentation functions that preserve semantic intent while injecting controlled variability across anatomical and dynamic dimensions.

To enable comprehensive evaluation, we instantiate IK-AUG within a Unity-based simulation pipeline and generate an enriched dataset from limited raw gestures. The synthesized samples are then used to train five representative temporal recognition models: CNN3D [14], Temporal Convolutional Network (TCN) [15], Transformer [16], Informer [17], and Sparse Transformer [18], revealing consistent accuracy improvements and faster convergence. Notably, attention-based models exhibit the most significant gains, validating the framework’s ability to enhance generalization across spatiotemporal architectures.

By transforming raw motion capture into high-fidelity gesture distributions, our algorithm not only mitigates data scarcity but also establishes a scalable, annotation-free augmentation paradigm for future gesture-based recognition systems.

The key contributions of this work are summarized as follows:

• We propose IK-AUG, a unified algorithmic framework that leverages inverse kinematics within a virtual simulation environment to generate anatomically and temporally coherent sign language sequences from sparse 3D inputs.
• We design a set of differentiable perturbation operators including depth modulation, spatial displacement, and solver sensitivity variation that systematically inject realistic variability while preserving semantic consistency, forming a domain-specific augmentation module.
• We conduct comprehensive empirical evaluations across five deep sequence models (CNN3D, TCN, Transformer, Informer, Sparse Transformer), demonstrating that IK-AUG consistently enhances convergence efficiency and classification performance, particularly in attention-based architectures.
• We establish a foundation for future hybrid gesture learning systems by outlining a scalable pathway to integrate multi-sensor modalities (e.g., inertial and haptic glove data) into a unified simulation–supervision loop.

Collectively, this work situates itself at the algorithmic intersection of inverse kinematics, simulation-based data generation, and spatiotemporal learning, offering a modular and extensible paradigm for gesture-centric recognition tasks.

2. Related Work

2.1. Virtual and Immersive Environments for Sign Language Learning and Recognition

Virtual environments have increasingly been employed to facilitate interactive learning and recognition of sign language. Several works have explored immersive VR platforms that enable real-time interaction between learners and signing avatars, allowing for effective acquisition of vocabulary and grammar through motion tracking and feedback loops [19]. Leap Motion-based systems further provide low-latency hand tracking from egocentric views, supporting real-time gesture classification with lightweight hardware [20,21]. However, these efforts primarily focus on interaction and recognition, whereas our method leverages virtual environments not just for learning but also for structured gesture data synthesis filling a crucial gap in generalizable data generation.

2.2. Inverse Kinematics for Motion Reconstruction

Inverse kinematics (IK) is widely adopted in gesture animation and full-body motion reconstruction, serving as a vital bridge between sparse positional input and anatomically consistent pose synthesis. Prior works have integrated IK with sign language notation to animate avatars from structured linguistic forms [22], while others reconstruct realistic full-body movements from sparse tracker data using learned IK models [23]. Our approach similarly employs IK but uniquely incorporates it into a data augmentation framework, ensuring that each synthesized gesture not only satisfies anatomical constraints but is also training-ready for downstream recognition models.

2.3. Data Augmentation for Sign Language Recognition

The challenge of data scarcity in sign language recognition (SLR) has prompted various augmentation strategies, including landmark transformation, image-space perturbation, and contextual recombination. For instance, landmark-based systems have reduced input dimensionality while improving generalization in in-the-wild scenarios [24]. Hybrid image augmentation involving background, lighting, and geometric shifts has also proven effective in improving model robustness under real-world conditions [25]. Similarly, studies using face swapping and affine transformations have achieved substantial accuracy improvements across top-k metrics [26]. Compared with these methods, our IK-based augmentation operates in 3D semantic space, allowing controlled, biomechanically constrained gesture variation beyond pixel-level manipulation.

2.4. Skeleton-Based and Adversarial Augmentation Techniques

Skeleton-based representations are increasingly utilized in SLR due to their interpretability and compactness. Previous work has proposed generating 3D skeletons from monocular videos and refining them through IK for animation and recognition tasks [27]. Meanwhile, adversarial learning-based augmentation frameworks like AVSN have shown that jointly optimizing augmentation and recognition can uncover model vulnerabilities and enhance generalization [28]. Our pipeline similarly adopts a skeleton-centric view, but emphasizes forward simulation with perturbation using sensitivity-controlled kinematic modifications to produce robust gesture sequences.

2.5. Multimodal and Glove-Enhanced Recognition Systems

Recent advancements in wearable technology have enabled multimodal gesture recognition via smart gloves equipped with triboelectric, inertial, and flexion sensors. These systems demonstrate the feasibility of recognizing full sentences in sign language and enabling bidirectional communication in virtual environments [29]. Despite their high fidelity, such systems face practical limitations due to hardware cost and user accessibility. In contrast, our method remains purely vision-based, while offering future extensibility to glove input for hybrid learning and supervision.

3. Methodology and Algorithms

3.1. Theoretical Foundation

3.1.1. Virtual Environments for Structured Simulation and Data Augmentation

Virtual environments have emerged as an indispensable tool in data-centric artificial intelligence, enabling researchers to decouple learning-based model development from the limitations of physical data acquisition. By leveraging physically grounded simulation engines and programmable control interfaces, modern virtual platforms allow for the synthesis of perceptually realistic, structurally consistent, and semantically controllable datasets across a wide range of modalities. In high-stakes domains such as robotics, embodied AI, and human-centric vision, this synthetic-to-real pipeline has fundamentally redefined how datasets are generated, curated, and scaled.

From a systems perspective, virtual environments offer a generative prior over interaction spaces. They permit precise specification of agent morphology, environmental constraints, temporal control, and multimodal sensory feedback. This level of control is particularly beneficial when designing data regimes that demand statistical diversity while retaining structural coherence, e.g., varying motion trajectories while preserving physical feasibility, or sampling camera perspectives without violating temporal causality. Such controllability is difficult to achieve in naturalistic data collection pipelines, where sensor noise, occlusion, and annotation ambiguity often introduce unquantifiable biases.

Virtual data augmentation is not merely a proxy for “more data”, but a mechanism for bias shaping and generalization conditioning; it enables targeted perturbations of scene structure, motion context, or agent–object relationships in ways that real-world sampling cannot support at scale. This property is increasingly central in domains where distribution shift, edge cases, or long-tail behaviors dominate downstream model performance.

Virtual environments have transitioned from visualization tools to algorithmic infrastructure providing not only photorealistic assets but also structured, verifiable, and fully controllable simulation scaffolds that form the foundation of modern synthetic data pipelines.

3.1.2. Inverse Kinematics (IK) for Motion Modeling

Inverse kinematics (IK) is a fundamental technique in robotics and motion synthesis that focuses on determining a set of joint configurations that achieve a desired end-effector position or trajectory. In contrast to forward kinematics where the pose of a skeleton is calculated by successively applying joint transformations from root to end-effector IK solves the inverse problem: given the target positions of certain parts (e.g., fingertip, hand, foot), it computes the internal joint angles that satisfy kinematic constraints while maintaining anatomical plausibility.

Mathematically, IK is often formulated as a constrained nonlinear optimization problem. Let $\mathit{\theta }\in {\mathbb{R}}^n$ represent the vector of joint parameters (e.g., angles or positions), and let $f\left(\mathit{\theta }\right)\in {\mathbb{R}}^3$ be the forward kinematics function that maps joint parameters to the Cartesian position of an end-effector. Given a desired target position ${\mathit{x}}_{\mathrm{target}}\in {\mathbb{R}}^3$, IK seeks a solution to the following system:

$$f\left(\mathit{\theta }\right)={\mathit{x}}_{\mathrm{target}}.$$

(1)

Due to redundancy or nonlinearity in articulated systems, an exact solution may not exist or may not be unique. Thus, the problem is commonly relaxed to minimizing the squared positional error:

$$\underset{\mathit{\theta }}{min}{∥f\left(\mathit{\theta }\right)-{\mathit{x}}_{\mathrm{target}}∥}^2,$$

(2)

subject to joint limits:

$${\mathit{\theta }}_{min}\le \mathit{\theta }\le {\mathit{\theta }}_{max}.$$

(3)

In practice, iterative methods are used to approximate solutions. One widely used approach involves the Jacobian matrix $J\left(\mathit{\theta }\right)\in {\mathbb{R}}^{3\times n}$, defined as:

$$J\left(\mathit{\theta }\right)={\displaystyle \frac{\partial f\left(\mathit{\theta }\right)}{\partial \mathit{\theta }}}.$$

(4)

A linear approximation of the end-effector motion is given by:

$$\Delta \mathit{x}\approx J\left(\mathit{\theta }\right)\Delta \mathit{\theta },$$

(5)

and the corresponding joint update can be computed via the Jacobian pseudoinverse:

$$\Delta \mathit{\theta }=J{\left(\mathit{\theta }\right)}^{+}\Delta \mathit{x},$$

(6)

where $J^{+}$ denotes the Moore-Penrose pseudoinverse of the Jacobian.

Alternative real-time solvers such as Cyclic Coordinate Descent (CCD) optimize each joint sequentially and are computationally efficient for low-DOF systems like fingers.

In the domain of motion modeling and gesture synthesis, IK ensures physical realism and structural coherence. For instance, fingertip trajectories captured from video can be converted into full-hand joint configurations using IK, even when some joints are occluded or missing. This enables realistic hand articulation from sparse data.

Moreover, IK systems are often integrated with animation engines or physics-based simulators to dynamically adjust to constraints such as obstacle avoidance or balance preservation during complex transitions.

In summary, IK serves as a bridge between sparse high-level control signals and full-body anatomically plausible movements. Its integration is crucial in gesture reconstruction, animation blending, and physics-informed motion generation pipelines.

3.1.3. Deep Learning Architectures for Sign Language Recognition

Deep learning models have become foundational to the task of sign language recognition (SLR), as they offer powerful mechanisms for extracting and modeling spatiotemporal patterns from high-dimensional motion sequences. Unlike static image classification, SLR requires understanding structured motion over time, often involving complex hand articulation, temporal dependencies, and multimodal cues. To address these challenges, both convolutional and attention-based neural architectures have been extensively adopted.

Convolutional approaches such as 3D Convolutional Neural Networks (CNN3D) capture spatiotemporal dependencies by applying convolutional filters across both spatial and temporal dimensions. This enables them to learn short-range motion patterns effectively while maintaining spatial coherence, making them suitable for recognizing locally structured gestures. In contrast, self-attention-based architectures like Transformers model global temporal relationships through attention mechanisms, allowing them to capture long-range dependencies without relying on recurrence or local convolutions. Their ability to process variable-length inputs and attend to both past and future frames makes them particularly effective for recognizing continuous, coarticulated gesture sequences where temporal context is critical.

The role of these architectures in SLR extends beyond prediction; they serve as analytical tools for evaluating temporal consistency, identifying gesture boundaries, and quantifying the discriminability of learned representations. By learning to map raw motion data into structured latent spaces, deep models enable systematic assessment of gesture classes and can serve as a diagnostic mechanism for evaluating the quality and diversity of training datasets.

3.2. Proposed Augmentation Framework via Kinematic Gesture Simulation

3.2.1. Overview of the Synthetic–Real Gesture Loop

To address the limitations of existing sign language datasets in terms of anatomical fidelity, inter-signer variability, and scalability, we propose IK-AUG a structured kinematic augmentation framework that bridges real-world motion acquisition and virtual gesture synthesis. As shown in Figure 1, IK-AUG is a multi-stage pipeline that combines inverse kinematics-based hand reconstruction, biomechanically constrained animation, and perturbation-driven augmentation to produce diverse and semantically faithful gesture sequences for downstream sign language recognition (SLR) tasks.

As detailed in Algorithm 1, the process begins with extracting 3D keypoints K from raw sign language video V using a vision-based estimator $\mathcal{E}$ (e.g., MediaPipe). These keypoints are spatially normalized and mapped into a virtual coordinate system via a transformation module $\mathcal{T}$, which includes noise filtering, frame interpolation, and re-orientation. A forward-and-backward iterative IK solver $\mathcal{I}$ reconstructs complete skeletal hand poses S from sparse landmarks, ensuring biomechanical plausibility. The resulting animation is rendered on a digital hand model within Unity3D (version 2022.3.40f1; Unity Technologies, San Francisco, CA, USA), forming a baseline gesture trajectory.

To simulate natural motion variability, a multi-modal perturbation operator ${\mathcal{P}}_{\Theta }$ is applied, where $\Theta =\{{ϵ}_{xy},{ϵ}_z,{\lambda }_{IK}\}$ defines parameter ranges for spatial jitter, depth displacement, and solver sensitivity, respectively. Each configuration generates a perturbed variant ${\overrightarrow{D}}_i$ that is exported as both video and coordinate data. Optionally, tactile and kinematic information from multi-sensor gloves $\mathcal{G}$ can be integrated to further enrich motion fidelity and robustness. The output dataset ${\mathcal{D}}_{aug}$ forms a synthetic–real manifold of augmented gesture sequences suitable for deep learning-based recognition and translation systems.

Algorithm 1 IK-AUG: Inverse Kinematic-Augmented Gesture Synthesis Framework

Require: Raw video sequence V, pose estimator $\mathcal{E}$, transformation module $\mathcal{T}$, IK solver $\mathcal{I}$,       perturbation config $\Theta =\{{ϵ}_{xy},{ϵ}_z,{\lambda }_{IK}\}$, number of augmentations N

Ensure: Augmented dataset ${\mathcal{D}}_{aug}=\{({\overrightarrow{D}}_1,{\mathcal{C}}_1),...,({\overrightarrow{D}}_N,{\mathcal{C}}_N)\}$

1:  $K\leftarrow \mathcal{E}\left(V\right)$                                                                 ▹ Extract 3D keypoints from raw video

2:   $K^{\prime }\leftarrow \mathcal{T}\left(K\right)$            ▹ Transform to virtual space: normalization, filtering, re-alignment

3:   $S\leftarrow \mathcal{I}\left(K^{\prime }\right)$                                  ▹ Apply IK solver to reconstruct skeletal pose sequence

4:   Initialize ${\mathcal{D}}_{aug}\leftarrow \varnothing$

5:   for$i=1$ to N do

6:         Sample perturbation config $\{{ϵ}_{xy}^{\left(i\right)},{ϵ}_z^{\left(i\right)},{\lambda }_{IK}^{\left(i\right)}\}\sim \Theta$

7:          $S^{\left(i\right)}\leftarrow \mathcal{P}(S;{ϵ}_{xy}^{\left(i\right)},{ϵ}_z^{\left(i\right)},{\lambda }_{IK}^{\left(i\right)})$                 ▹ Apply spatial, depth, and kinematic variation

8:          ${\overrightarrow{D}}_i\leftarrow \mathrm{RenderAnimation}\left(S^{\left(i\right)}\right)$

9:          ${\mathcal{C}}_i\leftarrow \mathrm{ExportCoordinates}\left(S^{\left(i\right)}\right)$

10:          ${\mathcal{D}}_{aug}\leftarrow {\mathcal{D}}_{aug}\cup \left\{({\overrightarrow{D}}_i,{\mathcal{C}}_i)\right\}$

11:   end for

12:   return ${\mathcal{D}}_{aug}$

3.2.2. Inverse Kinematics Solver for Hand Pose Reconstruction

To reconstruct hand gestures that conform to biomechanical constraints from keypoint position data, this work introduces an inverse kinematics (IK) solver within a virtual environment. The objective is to infer joint rotations of each finger based on the spatial positions of the fingertips, enabling the digital hand model to produce anatomically plausible postures.

In our system, the IK module takes the augmented keypoints transformed into the virtual coordinate system as input, and drives a 3D digital hand model as the target. Through iterative optimization, joint angles are computed to animate the skeleton and generate complete hand gestures.

We employ the Finger Rig module from the Final IK plugin to construct an independent three-segment kinematic chain (MCP–PIP–DIP) for each finger. The inverse kinematics solving process, as detailed in Algorithm 2, is based on the Forward And Backward Reaching Inverse Kinematics (FABRIK) algorithm. FABRIK approximates the fingertip’s spatial target through iterative forward and backward propagation steps.

Specifically, in each iteration, the backward phase adjusts joint positions from the fingertip to the root (DIP → PIP → MCP), preserving fixed bone lengths while progressively moving the chain toward the desired configuration. The subsequent forward phase restores the root position (MCP) and propagates the joint updates outward, ensuring physical plausibility and spatial continuity. In cases where the target lies beyond the maximum reach of the chain, the solver enters an emergency stretch mode, aligning all segments linearly toward the unreachable target to preserve anatomical consistency.

Finally, local joint rotations are derived from the resulting directional vectors and applied to the skeletal animation system to generate smooth and continuous hand gestures. This solver architecture effectively balances geometric constraints with real-time performance, providing fast convergence, numerical stability, and seamless integration into data-driven gesture synthesis pipelines. It is particularly well-suited for high-precision sign language reconstruction and target-controlled animation tasks.

Algorithm 2 Finger Rig-Based IK Solver

Input: Target position $T\in {\mathbb{R}}^3$, joint positions $\{J_0,J_1,J_2\}$ (MCP → PIP → DIP), segment        lengths $\{L_0,L_1\}$, convergence threshold $\epsilon$

Output: Joint rotations $\{{\theta }_0,{\theta }_1,{\theta }_2\}$

1:  if$\parallel J_0-T\parallel >L_0+L_1$then

2:        // Target unreachable: fully extend finger

3:        $\mathrm{Direction}\leftarrow \mathrm{normalize}(T-J_0)$

4:         $J_1\leftarrow J_0+L_0·\mathrm{Direction}$

5:         $J_2\leftarrow J_1+L_1·\mathrm{Direction}$

6:   else

7:         repeat

8:               // Backward pass

9:               $J_2\leftarrow T$

10:               $J_1\leftarrow J_2+L_1·\mathrm{normalize}(J_1-J_2)$

11:               $J_0\leftarrow J_1+L_0·\mathrm{normalize}(J_0-J_1)$

12:               // Forward pass (optional if root fixed)

13:               $J_0\leftarrow \mathrm{fixed}\mathrm{root}\mathrm{position}$

14:               $J_1\leftarrow J_0+L_0·\mathrm{normalize}(J_1-J_0)$

15:               $J_2\leftarrow J_1+L_1·\mathrm{normalize}(J_2-J_1)$

16:         until $\parallel J_2-T\parallel <\epsilon$

17:   end if

18:   Compute joint angles $\{{\theta }_0,{\theta }_1,{\theta }_2\}$ from final $\{J_0,J_1,J_2\}$

19:   Apply joint rotations to digital hand model

3.2.3. Sign Language Diversity Enhancement Strategy Driven by Multimodal Perturbation

Following the accurate reconstruction of hand gestures through inverse kinematics (IK), we introduce a multimodal perturbation strategy to systematically enhance the diversity and robustness of synthesized sign language sequences within a virtual environment. As illustrated in Figure 2, this stage applies structure-preserving perturbations across spatial, kinematic, and visual dimensions to emulate the natural variability and observational noise encountered in real-world gesture execution.

The process begins with IK-based joint angle estimation from video-derived keypoints, which are mapped onto a 3D digital hand model to reconstruct precise finger trajectories. To further enhance fidelity beyond local articulation, a body animation module generates synchronized upper-body motion, including arms and torso, ensuring that the resulting digital gestures reflect both fine-grained control and holistic motion patterns consistent with natural sign communication.

To introduce controllable variation into the synthesized gestures, we design three complementary perturbation mechanisms:

• Depth Offset: Simulates variations in camera-to-subject distance by perturbing the depth plane of the virtual viewpoint.
• 2D Spatial Perturbation: Introduces localized shifts in the image plane (X-Y), mimicking tracking noise, occlusions, or signer-dependent expressive differences.
• IK Sensitivity Control: Adjusts solver hyperparameters such as target weight and damping, enabling variability in articulation flexibility and gesture expressiveness.

All perturbations are governed by a sensitivity parameter $\epsilon$, which ensures semantic consistency while introducing naturalistic motion deviation. From a single original gesture instance $D_1$, this strategy generates a diverse sequence set $D_1\to D_n$, collectively forming the Augmented Gesture Sequences shown in Figure 3.

To further illustrate the effect of the augmentation pipeline, a supplementary video is provided showing the real gesture sequence and its IK-AUG-generated variants in motion. The video demonstrates how the perturbation components preserve semantic integrity while introducing realistic variability. It is available at: https://youtu.be/hawctx3g2fE (accessed on 8 July 2025).

4. Experiments

4.1. Experimental Setup

All experiments were conducted on a workstation running Ubuntu 21.04, equipped with an NVIDIA RTX 3090 GPU and 32 GB of RAM. The implementation was based on PyTorch 1.11.0 and Python 3.10.

The dataset used in this study consists of ten manually collected sign gestures, captured from a frontal view. Raw hand landmark data was extracted using MediaPipe, and subsequently converted into structured CSV files. Each sample in the dataset is represented as a sequence of 100 frames, with 21 hand keypoints per frame, and each keypoint described by its 3D spatial coordinates (x, y, z). Thus, the final input format for all models is standardized as a tensor of shape [100, 21, 3].

All models were trained exclusively on these structured 3D pose keypoints extracted from video using the MediaPipe Hands framework. No image frames, rendered videos, or skeletal animations were used during training. This ensures that the input data is purely geometric and fully standardized across all experiments.

This setup enables a consistent pipeline for training and evaluating various gesture recognition models under controlled conditions.

All models were trained using specific hyperparameters tuned via the Optuna optimization framework. Each model architecture CNN3D, TCN, Transformer, Informer, and Sparse Transformer was independently optimized using cross-validated search over learning rate, batch size, dropout rate, hidden dimensions, and weight decay. The optimization process was conducted within a fixed computational budget to ensure fairness. The final hyperparameter configurations, selected based on best validation accuracy, are summarized in

Table 1. Model -specific training hyperparameters used in all experiments.

Model	Batch Size	Learning Rate	Weight Decay	Dropout	Hidden Dim	Notes
CNN3D	16	$3.15\times {10}^{-4}$	$1.79\times {10}^{-5}$	0.395	320
TCN	32	$1.48\times {10}^{-4}$	$5.21\times {10}^{-5}$	0.266	256	Kernel size = 3, 6 layers
Transformer	32	$4.60\times {10}^{-4}$	$8.55\times {10}^{-5}$	0.102	32	2 layers, 8 heads
Informer	32	$9.83\times {10}^{-5}$	$7.92\times {10}^{-3}$	0.422	256	Kernel size = 0, 4 layers
Sparse Transformer	16	$7.79\times {10}^{-4}$	$2.97\times {10}^{-4}$	0.458	384

. These values were held constant across all training runs involving raw data, rotation-augmented data, and IK-AUG data to ensure a controlled and unbiased comparison across augmentation strategies.

4.2. Experimental Results

To evaluate the effectiveness of the proposed data augmentation strategy, we first trained two representative baseline models CNN3D and Transformer on both the raw dataset and the augmented dataset for 500 epochs. The training and validation accuracy curves are shown in Figure 4.

The CNN3D model trained on the raw data exhibits signs of overfitting, where the training accuracy continues to increase while the validation accuracy fluctuates and fails to steadily improve. In contrast, as shown on the right, the CNN3D model trained on the augmented data achieves consistently higher validation accuracy with reduced fluctuation, indicating better generalization and more stable convergence.

For the Transformer model, the benefit of augmentation is even more pronounced. When trained on the raw dataset, the Transformer fails to reach satisfactory accuracy within 500 epochs and suffers from underfitting. However, when trained on the augmented dataset, the model quickly converges to a high level of both training and validation accuracy, demonstrating that the augmented data not only improves overall performance but also enables more effective model fitting within the same training budget.

These results confirm that the proposed augmentation approach enhances both convergence speed and final accuracy, especially for deeper architectures such as Transformers.

To provide a comprehensive evaluation of model performance on the augmented gesture dataset, we trained five representative architectures CNN3D, TCN, Transformer, Informer, and Sparse Transformer for 1000 epochs. The training trajectories and final accuracy outcomes are presented in Figure 5, respectively.

As illustrated in Figure 5, all models demonstrate effective learning and converge successfully on the augmented dataset. CNN3D and TCN exhibit fast initial convergence, reaching high accuracy in the early training phase. Transformer-based models, including Informer and Sparse Transformer, present a more progressive and stable learning process, characterized by consistent improvement across epochs and smoother validation curves.

Figure 5 complements these findings by comparing the final training and validation accuracy. All models achieve high overall accuracy, confirming the suitability of the augmented data across architectural paradigms. Notably, Transformer-based models attain a particularly strong balance between training and validation performance, with Informer and Sparse Transformer both reaching 94.1% validation accuracy. In addition, CNN3D achieves the highest validation accuracy of 96.5%, further validating the effectiveness of the augmentation strategy on convolutional architectures.

Overall, these results highlight the versatility and strength of the augmented dataset in supporting both classical and modern architectures. In particular, Transformer variants exhibit highly consistent convergence and competitive accuracy, suggesting that the augmentation pipeline enables them to fully leverage their representational capacity for gesture modeling.

To quantify the effectiveness of the proposed IK-based data augmentation strategy, we conducted controlled experiments by training all five models CNN3D, TCN, Transformer, Informer, and Sparse Transformer with and without augmentation. The results are presented in Figure 6.

Figure 6 shows the validation accuracy comparison under both settings. For all models, introducing IK-based augmentation consistently leads to higher validation accuracy, confirming the general benefit of the method. Notably, the improvement is most prominent in Transformer-based architectures. The vanilla Transformer sees its validation accuracy increase from 87.1% to 93.2%, while Informer and Sparse Transformer both improve by more than 4.7%.

To further highlight the contribution of each model, Figure visualizes the proportion of total accuracy gain attributable to each architecture. The Transformer model accounts for the largest share (0.061), followed closely by Informer (0.058) and Sparse Transformer (0.047). These observations demonstrate that the augmentation strategy not only improves model performance uniformly, but also enables attention-based models to realize their full potential on complex gesture sequences.

Together, these results underscore the robustness and scalability of our augmentation pipeline across different model paradigms.

To further demonstrate the advantage of our IK-based augmentation strategy, we established a baseline using a conventional augmentation technique, specifically, 2D random rotation. Each model was trained with pose sequences augmented by randomly rotating the hand keypoints within ±10° in the image plane. As shown in Figure 7, the models trained with IK-AUG consistently outperform those trained with random rotation across all five architectures. These results confirm that IK-AUG provides stronger semantic and structural regularization compared to geometric-only augmentation techniques.

To further examine the consistency and robustness of IK-based augmentation across different training budgets, we extended the evaluation to include three epoch settings: 500, 1000, and 2000. The results are summarized in Figure 8.

As shown in Figure 8, all five models demonstrate consistent improvements in validation accuracy after applying the augmentation method, regardless of the number of training epochs. Transformer-based architectures, particularly Informer and Sparse Transformer, not only exhibit high baseline performance but also maintain stable and notable gains at all epoch levels. For example, Informer achieves gains of 0.040, 0.058, and 0.053 across the 500, 1000, and 2000 epoch settings, respectively.

To better understand the influence of training duration on augmentation effectiveness, Figure 6b plots the accuracy gains over epochs for each model. Most models show either a stable or upward trend in gain magnitude as training progresses, with Transformer achieving the largest gain at 1000 epochs (+0.070), and Informer sustaining gains above 0.05 throughout. This suggests that our augmentation strategy scales well with increased training and can be fully leveraged by attention-based architectures.

Figure 8 compares the final validation accuracy on the augmented dataset alone, without reference to raw data performance. The results demonstrate that accuracy steadily improves with longer training for nearly all models, reaffirming the compatibility of the augmented data with extended training. Notably, all models surpass 93% validation accuracy at 2000 epochs, with Transformer-based models consistently ranking at the top.

Collectively, these findings validate that the IK-based augmentation not only yields immediate performance boosts, but also maintains its effectiveness over prolonged training periods, particularly when combined with Transformer-family architectures.

To further validate the impact of IK-based augmentation on test-time performance, we conducted a comparative analysis across all models using the test set. As summarized in

Table 2. Test set classification accuracy (%) for all models trained on original vs. augmented datasets.

Model	Original Data Accuracy	Augmented Data Accuracy
CNN3D	85%	90%
TCN	83%	89%
Transformer	90%	92%
Informer	91%	94%
Sparse Transformer	89%	92%

, all five architectures exhibit consistent improvements in final test accuracy after being trained on the augmented dataset. CNN3D and TCN achieve relative gains of +5% and +6%, respectively, while Transformer-based models Transformer, Informer, and Sparse Transformer show marginal but stable improvements of +2% to +3%, further confirming the compatibility of the synthesized gestures with generalization tasks.

Additionally, Figure 9 presents the confusion matrices of the Transformer model trained with and without augmentation. The augmented model (right) demonstrates stronger diagonal dominance and reduced inter-class confusion compared to the baseline (left), highlighting the improved discriminative ability and classification robustness introduced by the synthetic diversity.

These results emphasize that the proposed augmentation framework not only boosts training efficiency and validation performance but also translates into measurable generalization gains on unseen data, making it a practical and scalable solution for gesture-based recognition systems.

4.3. Evaluation

To quantitatively assess the effectiveness of the proposed augmentation framework, we adopted a series of evaluation metrics and comparative strategies centered on model generalization, classification accuracy, and inter-class discriminability.

The primary evaluation metric is top-1 classification accuracy on a held-out test set, ensuring comparability across models and datasets. All models were trained under identical settings with and without augmentation to isolate the contribution of synthetic data. For further validation, we utilized confusion matrix analysis to visualize class-level performance and to identify reductions in misclassification errors, particularly for semantically or visually similar gestures.

In addition, we computed relative accuracy gains across multiple architectures and training budgets to evaluate consistency and scalability. The inclusion of temporal models (e.g., TCN) and attention-based architectures (e.g., Transformer, Informer) provides a broad perspective on how different inductive biases interact with the augmented data.

This evaluation protocol enables us to not only confirm the overall effectiveness of the augmentation strategy, but also to uncover architecture-specific benefits and gesture-specific improvements. These findings are further analyzed in the subsequent discussion section.

4.4. Discussion

The experimental findings substantiate the algorithmic merit of the proposed IK-AUG framework across diverse model architectures and training configurations. Notably, inverse kinematics-driven augmentation consistently enhances both convergence stability and generalization capacity particularly in Transformer-based models, which are otherwise susceptible to overfitting and data inefficiency in low-resource regimes. By synthesizing biomechanically constrained yet variationally rich gesture sequences, IK-AUG effectively expands the data manifold while preserving the semantic integrity of motion classes, thereby serving as an implicit regularizer in temporal representation learning.

Temporal convolutional architectures such as TCN demonstrate robust performance under this augmentation regime, exhibiting efficient convergence and favorable trade-offs between model complexity and recognition accuracy. In contrast, attention-centric models like Sparse Transformer attain high asymptotic accuracy but display greater sensitivity to hyperparameter configurations and training dynamics, suggesting that they are more reliant on augmentation-induced structural priors to capture gesture regularity.

Despite these improvements, confusion matrix analysis reveals persistent misclassification in semantically adjacent gesture classes. This indicates that further disambiguation may require the integration of complementary modalities, e.g., depth maps, surface cues, or multiview projections, as well as temporal alignment mechanisms that better capture inter-frame dependencies and gesture onset–offset transitions.

In addition, we recognize that the current study is constrained by the limited scale of the dataset, which includes 10 gesture classes recorded in a controlled environment. This limitation may restrict the generalization of the observed findings to more diverse, real-world signing scenarios. We plan to address this by scaling up the dataset in future work, incorporating a broader vocabulary, more signer diversity, and varied contextual conditions. Such efforts will enable a more comprehensive evaluation of the framework’s robustness and applicability.

Although IK-AUG integrates multiple perturbation modules namely spatial jitter, depth modulation, and solver sensitivity control these components are intrinsically coupled within a unified simulation loop that operates on inverse kinematics-reconstructed trajectories. As such, removing individual components would compromise the biomechanical plausibility or semantic consistency of the synthesized gestures. In future work, we plan to develop a modular variant of the pipeline to facilitate interpretable component-wise evaluation and principled ablation analysis.

Overall, the results validate IK-AUG not merely as a data expansion tool, but as a principled algorithmic mechanism for generating anatomically faithful, semantically grounded, and training-efficient gesture corpora. These insights underscore the importance of geometry-aware augmentation in deep sequence modeling, and lay a foundation for hybrid simulation–sensor pipelines in future sign language understanding systems.

5. Conclusions

This work introduces IK-AUG, a principled augmentation framework that unifies inverse kinematics, simulation-based gesture modeling, and parametric perturbation into a coherent algorithmic process for sign language recognition. By transforming sparse keypoint trajectories into anatomically valid motion sequences within a controllable virtual environment, IK-AUG addresses key limitations of existing gesture datasets in scalability, structural diversity, and biomechanical fidelity.

Through the integration of 3D pose estimation, forward–backward IK solvers, and multi-modal augmentation operators parameterized by spatial, depth, and kinematic factors, the framework yields gesture variants that span a rich data manifold while preserving semantic consistency. Experimental evaluations across multiple deep temporal architectures validate the efficacy of IK-AUG in enhancing classification accuracy, reducing overfitting, and improving class-level discriminability particularly for high-capacity attention-based models such as Transformers. Moreover, the augmentation benefits persist across varied training budgets, confirming the robustness and transferability of the synthetic–real gesture loop.

Beyond empirical gains, this research contributes a generalized computational abstraction of gesture synthesis, whereby virtual motion priors can be encoded, perturbed, and operationalized for model training. The framework thus serves not merely as a data augmentation technique, but as a reusable generative substrate for simulation-driven learning paradigms.

Looking ahead, the integration of multi-sensor glove streams featuring inertial, haptic, and flexion feedback offers a compelling pathway toward hybrid supervision regimes, enabling the joint modeling of physical and virtual motion distributions. Additionally, advancing cross-domain adaptation techniques may allow IK-AUG-generated sequences to serve as pretraining anchors for real-world SLR, thereby narrowing the sim-to-real gap through structured algorithmic alignment.

While the IK-AUG framework demonstrates consistent improvements across diverse model architectures, it is important to acknowledge the current study’s limitations. The experimental dataset comprises 10 gesture classes collected under constrained conditions, which may limit the generalizability of the findings to broader, real-world scenarios. To address this, future work will focus on scaling up the dataset to include a more extensive vocabulary, diverse signers, and varied environmental contexts. We also plan to evaluate the framework on large-scale benchmarks such as RWTH-PHOENIX-Weather and How2Sign, and explore the integration of multimodal inputs to support hybrid simulation–sensor learning pipelines.

In total, IK-AUG exemplifies the algorithmic fusion of kinematic reasoning, generative simulation, and data-driven learning laying the foundation for a new class of geometry-aware, sensor-informed learning frameworks situated at the intersection of embodied AI and multimodal human–computer interaction.