Hand Kinematic Model Construction Based on Tracking Landmarks

Abstract

Visual body-tracking techniques have seen widespread adoption in applications such as motion analysis, human–machine interaction, tele-robotics and extended reality (XR). These systems typically provide 2D landmark coordinates corresponding to key limb positions. However, to construct a meaningful 3D kinematic model for body joint reconstruction, a mapping must be established between these visual landmarks and the underlying joint parameters of individual body parts. This paper presents a method for constructing a 3D kinematic model of the human hand using calibrated 2D landmark-tracking data augmented with depth information. The proposed approach builds a hierarchical model in which the palm serves as the root coordinate frame, and finger landmarks are used to compute both forward and inverse kinematic solutions. Through step-by-step examples, we demonstrate how measured hand landmark coordinates are used to define the palm reference frame and solve for joint angles for each finger. These solutions are then used in a visualization framework to qualitatively assess the accuracy of the reconstructed hand motion. As a future work, the proposed model offers a foundation for model-based hand kinematic estimation and has utility in scenarios involving occlusion or missing data. In such cases, the hierarchical structure and kinematic solutions can be used as generative priors in an optimization framework to estimate unobserved landmark positions and joint configurations. The novelty of this work lies in its model-based approach using real sensor data, without relying on wearable devices or synthetic assumptions. Although current validation is qualitative, the framework provides a foundation for future robust estimation under occlusion or sensor noise. It may also serve as a generative prior for optimization-based methods and be quantitatively compared with joint measurements from wearable motion-capture systems.

1. Introduction

Within the past decades, hand tracking through various ambient sensing modalities has been gaining a considerable attention given its potential applications in many areas [1,2,3,4]. One of the immediate applications of such tracking and recognition is through the development of human–machine interaction, virtual reality, and tele-robotics. In addition, through increased precision in sensing modalities such as RGB-D sensing and through advances in deep-learning tracking models, it is now possible to establish a more accurate description of tracking hand landmarks defined through its various key anatomical locations. However, discerning the information regarding these spatial landmark data and constructing an accurate kinematic hand model faces several fundamental challenges. First, the human hand exhibits kinematic redundancy, with many degrees of freedom (DoF) leading to multiple possible configurations for the same fingertip position. Second, singularities may arise in certain joint configurations, complicating inverse kinematics. Third, anatomical variability across individuals can affect model generalizability. Our method addresses these by employing a hierarchical model structure that reduces redundancy through localized transformations, mitigating singularities through stable axis assignments, and leveraging real measurement data to accommodate anatomical differences. This approach enables practical estimation and reconstruction of realistic hand poses in diverse conditions.

1.1. Literature Review

1.1.1. Non-Invasive Hand-Tracking Systems

The sensing of hand and limb motion is a critical area of study with diverse applications, including human–computer interaction (HCI), virtual reality interfaces, and the monitoring of athletic performance. Existing commercial sensor systems often pose challenges due to their invasive nature, requiring users to wear specialized equipment, such as gloves or markers. This has created a need for alternative, non-invasive approaches.

One such effort is DigitEyes by Rehg et al. [5], a vision-based hand-tracking system that leverages a kinematic model to extract line and point features from grayscale images of unmarked hands. It enables real-time tracking for applications like 3D mouse interaction. Further advancing grayscale-based tracking, Rehg et al. [6] introduced a method capable of recovering the state of a 27-degrees-of-freedom (DoF) hand model using only standard grayscale video input. These early studies highlight the complexity of accurately modeling highly articulated mechanisms such as the human hand, which involve a vast state space and intricate visual patterns.

1.1.2. Depth-Based and Anatomically Constrained Models

The introduction of depth sensors has significantly improved the robustness and precision of 3D hand pose estimation. However, a persistent challenge is ensuring that predictions comply with anatomical constraints. Addressing this, Isailovic et al. [7] proposed an anatomical filter that accepts 3D tracker outputs and corrects the resulting 26-DoF hand vectors using biomechanical limitations, thereby enhancing realism.

Alternative sensing modalities have also been explored. For example, Li et al. [8] presented Aili, a table-lamp system that reconstructs hand skeletons without cameras or wearables. Using LED panels and low-cost photodiodes, it derives 2D binary blockage maps to infer hand position and shape, demonstrating a seamless blend of sensing and everyday utility.

Other studies focus on marker-based validation. Cereatti et al. [9] developed a multi-camera system with 24 surface markers to reconstruct finger kinematics in real time. Their method relies on a rigid-body model with 22 DoFs and includes automatic calibration for joint axes and rotation centers. This system has been validated on thumb flexion, grasping, and pointing tasks.

1.1.3. Kinematic Modeling from Anatomical Landmarks

The importance of anatomical accuracy extends beyond human motion. For instance, Haufe et al. [10] introduced a detailed kinematic model of Drosophila legs using anatomical landmarks such as condyles, which better capture the natural orientation of oblique joint axes. In the context of human-hand tracking, Ji and Yang [11] proposed a hierarchical topology-based method. Their approach defines global palm orientation and local finger-joint connectivity, then uses angle-based features with a regression forest to estimate 3D joint positions.

Pena et al. [12] contributed a simulation framework for manipulation and grasping based on a 25-DoF skeletal hand model. Their model incorporates palm arching and additional wrist/carpometacarpal joint DoFs, enhancing anatomical fidelity. They define joint relationships using Denavit–Hartenberg parameters, allowing for workspace estimation and detailed motion simulation.

1.1.4. Recent Advances in Adaptive and Probabilistic Modeling

Recent research emphasizes anatomical constraints, adaptiveness, and probabilistic modeling in hand kinematics. Zimmermann and Brox [13] introduced a hybrid learning model integrating biomechanical priors for real-time 3D pose estimation. Lapresa, Zollo and Cordella [14] developed a subject-specific adaptation framework that updates kinematic parameters online. Xu and Lee [15] formulated a probabilistic estimation approach that accounts for occlusions and enforces anatomical plausibility via joint limit constraints. These works illustrate a growing shift toward anatomically informed, subject-adaptive models—principles that also motivate the framework proposed in this paper.

1.1.5. MediaPipe-Based Tracking and Limitations

MediaPipe-based solutions have gained traction due to their efficiency and wide deployment. Amprimo et al. [2] validated both the standard Google MediaPipe Hand (GMH) model and a depth-enhanced variant (GMH-D), showing improved spatial accuracy in clinical hand assessments when compared against motion-capture ground truth. Similarly, Pfisterer et al. [16] demonstrated that fusing MediaPipe landmarks with RGB-D data from RealSense sensors could improve gesture recognition in human–robot interaction scenarios.

While these systems effectively enhance landmark accuracy through depth integration, they largely treat the tracked keypoints as direct pose inputs without explicitly modeling the underlying kinematics. In contrast, our approach introduces a fully articulated hierarchical kinematic model. We define local coordinate frames for each joint, solve both forward and inverse kinematics based on spatial geometry, and reconstruct hand postures using Unity3D (version 2023.2.20f1) for visual validation. This framework offers high interpretability, facilitates simulation, and supports future extensions under noisy or incomplete observations.

2. Background Material

2.1. Sensor Specifications

To ensure accurate 3D measurements of hand landmarks, we employed the Intel RealSense D435i RGB-D camera (Intel Corporation, Santa Clara, CA, USA) as our sensing device. The D435i integrates a high-resolution RGB sensor (1920 × 1080 at 30 frames per s) with an active infrared stereo depth sensor (1280 × 720 at 30 frames per s). The depth sensing operates within a range of 0.2 to 10 m, with a stereo baseline of 50 mm and field of view of approximately $87\times {58}^{\circ }$. The sensor provides synchronized RGB and depth frames, enabling frame-by-frame fusion of 2D image coordinates and calibrated depth values. In our experimental setup, the RealSense SDK v2.56.1 was used to extract aligned RGB and depth frames. All depth measurements were calibrated in millimeters and subsequently converted into 3D spatial coordinates using the camera’s intrinsic parameters, as detailed in Appendix A.

2.2. MediaPipe

MediaPipe Hand [17] (Google AI Edge Portal) is a hand-tracking framework that provides a solution for detecting and estimating the 2D hand landmarks in video streams using a single RGB camera. The MediaPipe Hand model tracks 21 distinct hand landmarks, which is achieved through a pipeline that integrates hand detection, keypoint localization, and pose estimation. The pixel coordinates in the RGB image for each hand landmark can be expressed as a pair of values or in vector form:

$$p_i=(u_i,v_i)=\left[\begin{array}{c}{u}_i\\ v_i\end{array}\right],\mathrm{for}i=0,\dots ,20,$$

(1)

here, $u_i$ and $v_i$ represent the pixel coordinates of the ith landmark with respect to the image buffer frame. These landmark coordinates can then be calibrated in order to associate the representative pixel coordinates to the physical coordinate with respect to the sensor frame.

In order to associate additional coordinate measure to the existing 2D measurement, MediaPipe also offers an estimate of the physical distance of the landmarks with respect to the sensor frame. For example, Figure 1 illustrates three distinct tracking frames of a continuous hand movement sequence between calibrated the near to far distances with respect to the sensor.

As can be seen in Figure 2a, the estimated depth values of the tracked landmarks provided by MediaPipe algorithm are not consistent. To remedy this inconsistency, we have utilized an RGB-D sensor which can offer a calibrated depth or distance of the measured landmarks with respect to the sensor frame. Comparing the results of Figure 2b with Figure 2a, the advantages of utilizing an additional sensing modality which can be fused with the information obtained from the MediaPipe tracking algorithm can be observed (See Appendix A). In addition, since MediaPipe’s depth estimation is not calibrated to real-world physical units, this renders it ineffective for fulfilling the objectives of this paper.

This integration replaces the relative depth estimates from the MediaPipe Hand model with the actual depth values obtained from the Intel RealSense RGB-Depth sensor (Intel Corporation, Santa Clara, CA, USA). The 2D pixel coordinates $(u_i,v_i)$, initially estimated by MediaPipe, along with their corresponding depth values $d(u_i,v_i)$ captured by the depth sensor, are converted into real-world 3D spatial coordinates using the camera’s intrinsic parameters—specifically, the focal lengths $f_x$, $f_y$, and the principal point offsets $c_x$, $c_y$ (see Appendix A for further details on aligning the RGB image pixels with the depth buffer). Notably, $(u_i,v_i)$ in the RGB and depth buffers may differ and require calibration). This transformation enables the calculation of the accurate real-world 3D spatial coordinates ${\mathbf{d}}_i=(x_i,y_i,z_i)$ and the ith hand landmark, with respect to the sensor. The conversion can be expressed mathematically as:

$$\begin{array}{cc}\hfill x_i& ={\displaystyle \frac{(u_i-c_x)·d(u_i,v_i)}{f_x}}\hfill \\ \hfill y_i& ={\displaystyle \frac{(v_i-c_y)·d(u_i,v_i)}{f_y}}\hfill \\ \hfill z_i& =d(u_i,v_i),i=0,\dots ,20\hfill \end{array}$$

(2)

In the next section, we present a method on how to construct the kinematic model of the hand using the spatial information of these tracked landmarks. The overall kinematic model comprises the local root coordinate frame associated with the palm describing the position and transformation of the hand with respect to the world coordinate frame, i.e., the sensor coordinate, and a collection of coordinate frames and an associated transformation matrix describing the kinematic model of the finger with respect to the palm-coordinate frame.

3. Hand Kinematic Model Definitions

In this section, we present an approach for constructing the kinematic model of the hand based on the spatial landmarks’ tracking information. We first define the kinematic world coordinate frame, according to which all the coordinates of the hand landmarks are measured. The definition of the origin of the local hand coordinate frame based on the landmark definitions of MediaPipe is presented in Section 3.2, followed by definitions of hand kinematic parameters presented in Section 3.3.

3.1. The World-Frame Definition

When we are defining a model for a moving hand within a three-dimensional space, two primary parameters are necessary for describing the state of the hand at any given time, namely position and orientation. The description of these parameters is inherently relative, as both the position and orientation of an object such as the whole hand is always referenced with respect to the origin and three pre-defined orthogonal axes of a fixed universal (world) coordinate frame, also referred to as the global frame.

By convention, the world frame, denoted by $\left\{\mathbf{W}\right\}$, is usually attached to a non-moving object, thereby enabling all hand movements to be described with respect to it. In this paper, the sensor frame, located at the center of the sensor, is adopted as the world frame, as is shown in Figure 3.

We denote the unit vectors representing the three principal directions of the world frame $\left\{\mathbf{W}\right\}$ as ${\widehat{\mathbf{X}}}_w$, ${\widehat{\mathbf{Y}}}_w$ and ${\widehat{\mathbf{Z}}}_w$. To align with the 2D image-frame systems, where pixel coordinates originate from the upper-left corner, axes ${\widehat{\mathbf{X}}}_w$ and ${\widehat{\mathbf{Y}}}_w$ are oriented to the right and downwards, respectively. This alignment simplifies the process of mapping 2D image pixels to 3D real-world coordinates using physically calibrated measurable quantities (see Appendix A). Finally, using the right-hand rule, we establish that the ${\widehat{\mathbf{Z}}}_w$ axis points from the sensor toward the objects, which aligns with the depth data measurement, representing the distance between the hand and the sensor.

3.2. Definitions of Hand Hierarchical Coordinate Frames

In the world coordinate frame $\left\{\mathbf{W}\right\}$, the hand’s position can be located, and further the rotational joint angle of each finger joint can be described by defining local (or relative) coordinate frames. The wrist frame $\left\{\mathbf{0}\right\}$, which is affixed to the wrist of the hand with origin defined at the wrist landmark is define relative to the world frame. While the definition of this relative coordinate frame is arbitrary, as shown in Figure 4, we define the ${\widehat{\mathbf{Y}}}_0$ of the wrist frame to be aligned with the general direction of finger flexion; the unit vector ${\widehat{\mathbf{X}}}_0$ is oriented from the wrist towards the root of the middle finger, while ${\widehat{\mathbf{Z}}}_0$ is orthogonal to the palm plane defined by the other axes, resulting in a direction that points towards the back of the hand by assuming that the front of the hand faces the sensor.

The kinematic model of the hand is based on 21 landmarks, including 1 root (the wrist) and 20 links (phalanges) across the five fingers. Each link is assigned a corresponding local frame, allowing for a description of the position and rotation of each link.

This model employs a hierarchical structure of local frames corresponding to anatomical joints. As shown in Figure 4, the first layer includes local frames $\left\{\mathbf{1}\right\}$, $\left\{\mathbf{5}\right\}$, $\left\{\mathbf{9}\right\}$, $\left\{\mathbf{13}\right\}$, and $\left\{\mathbf{17}\right\}$ at the metacarpophalangeal (MCP) joints, defined with respect to the root frame $\left\{\mathbf{0}\right\}$. The second layer consists of local frames $\left\{\mathbf{2}\right\}$, $\left\{\mathbf{6}\right\}$, $\left\{\mathbf{10}\right\}$, $\left\{\mathbf{14}\right\}$, and $\left\{\mathbf{18}\right\}$ at the proximal interphalangeal (PIP) joints, defined relative to the first layer. Similarly, frames $\left\{\mathbf{3}\right\}$, $\left\{\mathbf{7}\right\}$, $\left\{\mathbf{11}\right\}$, $\left\{\mathbf{15}\right\}$, and $\left\{\mathbf{19}\right\}$ at the distal interphalangeal (DIP) joints form the third layer. Finally, the fourth layer comprises frames $\left\{\mathbf{4}\right\}$, $\left\{\mathbf{8}\right\}$, $\left\{\mathbf{12}\right\}$, $\left\{\mathbf{16}\right\}$, and $\left\{\mathbf{20}\right\}$ at the fingertips.

Each metacarpophalangeal (MCP) joint, or finger root, possesses two degrees of freedom (DOF), enabling it to flex and extend (bend up and down) as well as abduct and adduct (move side to side). When the root of a finger is fixed, and the finger is bent up and down, the proximal interphalangeal (PIP) and distal interphalangeal (DIP) joints each exhibit one DOF. Consequently, the workspace (the area the fingertip can reach) of a finger is a two-dimensional plane. Each of the five fingers can thus be modeled as a kinematic chain comprising three links connected by three revolute joints, with parallel rotation axes. The axes $\widehat{\mathbf{Y}}$ of the local frames for each finger align with the axes of the revolute joints, pointing to the page, as illustrated in Figure 5. The axes $\widehat{\mathbf{X}}$ for each of the three joints orient towards the origin of the subsequent frame. For example, along the index finger, ${\widehat{\mathbf{X}}}_5$ points to the origin of local frame $\left\{\mathbf{6}\right\}$, ${\widehat{\mathbf{X}}}_6$ to the origin of local frame $\left\{\mathbf{7}\right\}$, and ${\widehat{\mathbf{X}}}_7$ to the origin of local frame $\left\{\mathbf{8}\right\}$. Note that since $\left\{\mathbf{8}\right\}$ is located at fingertip and has no DOF, ${\widehat{\mathbf{X}}}_8$ aligns with ${\widehat{\mathbf{X}}}_7$. The axes $\widehat{\mathbf{Z}}$ of the local frames are assigned using the right-hand rule. This layered approach effectively models the intricate kinematic structure of the human hand. Although the present approach primarily includes grasping and natural resting poses, the model is capable of representing a wider range of joint configurations.

3.3. Hand Kinematic Parameters

Kinematic modeling involves a mathematical framework to represent the motion properties of a system such as a robot or a human hand, without considering the forces that cause the movements. It provides a way to describe, for example, the positions and orientations of the interconnected segments or links within the structure relative to each other and fixed world frame. This model is essential, for example, in understanding and predicting how joint movements translate into overall hand movement.

In this context, forward kinematics refers to the process of calculating the positions and orientations of the fingertips and intermediate links of the hand relative to the wrist and the world frame, given known joint angles and the fixed kinematic parameters of the hand. This computation starts from the base frame (e.g., the wrist) and propagates outward through the hand kinematic chain. On the other hand, inverse kinematics focuses on determining the joint angles needed to achieve a specific kinematic configuration of the hand.

In this subsection, we define the parameters of the kinematic model for the palm and fingers. The next section presents methods to resolve these kinematic model parameters for the hand using data collected from the MediaPipe Hand model and the RGB-D sensor.

Palm Parameters:In this paper, we represent the model of the palm as a planar structure in three-dimensional space. We assume that the wrist and the roots of the five fingers lie on a single plane, specifically the xy-plane of the local frame $\left\{\mathbf{0}\right\}$. A set of 16 parameters are defined, where the initial six parameters are time-varying while the five length and five angle parameters are assumed to be constant:

• Position Parameters: The position vector ${\mathbf{d}}_0^w={(x_0,y_0,z_0)}^T$ specifies the distance of translation between the origins of the wrist frame $\left\{\mathbf{0}\right\}$ and the world frame $\left\{\mathbf{W}\right\}$ along ${\widehat{\mathbf{X}}}_w$, ${\widehat{\mathbf{Y}}}_w$ and ${\widehat{\mathbf{Z}}}_w$, respectively.
• Orientation Parameters: The orientation of the local frame $\left\{\mathbf{0}\right\}$ is defined by the Euler angles $\{{\gamma }_0,{\beta }_0,{\alpha }_0\}$; these angles describe the sequential rotations around axes ${\widehat{\mathbf{Z}}}_0$, ${\widehat{\mathbf{Y}}}_0$, and ${\widehat{\mathbf{X}}}_0$.
• Base Length Parameters: The lengths $\{l_{0,1},l_{0,5},l_{0,9},l_{0,13},l_{0,17}\}$ represent the fixed distances between the origin of the local frame $\left\{\mathbf{0}\right\}$ and the origins of local frames $\left\{\mathbf{1}\right\}$, $\left\{\mathbf{5}\right\}$, $\left\{\mathbf{9}\right\}$, $\left\{\mathbf{13}\right\}$, and $\left\{\mathbf{17}\right\}$. These measurements, in centimeters, are spatial relationships between the wrist and finger roots.
• Angle Parameters: The angles $\{{\theta }_1,{\theta }_5,{\theta }_9,{\theta }_{13},{\theta }_{17}\}$ describe the rotational offset between the ${\widehat{\mathbf{X}}}_0$ axis and the axes ${\widehat{\mathbf{X}}}_1$, ${\widehat{\mathbf{X}}}_5$, ${\widehat{\mathbf{X}}}_9$, ${\widehat{\mathbf{X}}}_{13}$, and ${\widehat{\mathbf{X}}}_{17}$, respectively. These angles, measured in radians, are pivotal for capturing the hand’s natural articulation around the ${\widehat{\mathbf{Z}}}_0$ axis.

Parameters of Fingers: As shown in Figure 5, each finger requires three length parameters (as constraints) and four angle parameters (between 0 to 90 degrees) to define its pose, amounting to a total of 35 parameters for all five fingers:

• Base-Length Parameters: A set of 15 parameters determines the fixed lengths of the three links $l_{i,i+1}$, $l_{i+1,i+2}$ and $l_{i+2,i+3}$, within each finger, where $i=1,5,9,13,17$:

  $$\{(l_{1,2},l_{2,3},l_{3,4}),(l_{5,6},l_{6,7},l_{7,8}),(l_{9,10},l_{10,11},l_{11,12}),(l_{13,14},l_{14,15},l_{15,16}),(l_{17,18},l_{18,19},l_{19,20})\}$$
• Angle Parameters for Finger Roots: A set of 10 parameters describes five pairs of rotation angles ${\beta }_i$ and ${\gamma }_i$ around axes ${\widehat{\mathbf{Y}}}_i$ and ${\widehat{\mathbf{Z}}}_i$ for the root joints of each finger, where $i=1,5,9,13,17$:

  $$\{({\beta }_1,{\gamma }_1),({\beta }_5,{\gamma }_5),({\beta }_9,{\gamma }_9),({\beta }_{13},{\gamma }_{13}),({\beta }_{17},{\gamma }_{17})\}$$
• Angle Parameters for Other Joints: A set of 10 parameters describes five pairs of rotation angles ${\beta }_{i+1}$ and ${\beta }_{i+2}$ for the PIP and DIP joints around corresponding ${\widehat{\mathbf{Y}}}_i$ of each finger, $i=1,5,9,13,17$:

  $$\{({\beta }_2,{\beta }_3),({\beta }_6,{\beta }_7),({\beta }_{10},{\beta }_{11}),({\beta }_{14},{\beta }_{15}),({\beta }_{19},{\beta }_{20})\}$$

4. Hand Hierarchical Transformations

To effectively track human-hand movements and reconstruct its corresponding kinematic representation for graphical reconstruction, it is essential to represent the position and orientation of each joint’s local coordinate frame in 3D space and the corresponding joint parameters of each finger. We introduce a systematic approach for resolving these parameters through transformations between adjacent coordinate frames. The input parameters are spatial coordinates of 21 hand landmarks provided by the MediaPipe through RGB-D sensor, which are position parameters of each origin of the local frame $\left\{\mathbf{i}\right\}$ with respect to world frame $\left\{\mathbf{W}\right\}$, denoted as vector ${\mathbf{d}}_i^w={(x_i,y_i,z_i)}^T$ for $i=0,1,\dots ,20$, (Section 3.1).

Table 1. Measured 3D spatial coordinates ${\mathbf{d}}_i^w$ of 21 hand landmarks (in meters).

	Joints	MCP (1st Layer)	PIP (2nd Layer)	DIP (3rd Layer)	Tip (4th Layer)
Finger
Wrist		${\mathbf{d}}_0^w$
Thumb		${\mathbf{d}}_1^w$	${\mathbf{d}}_2^w$	${\mathbf{d}}_3^w$	${\mathbf{d}}_4^w$
Index		${\mathbf{d}}_5^w$	${\mathbf{d}}_6^w$	${\mathbf{d}}_7^w$	${\mathbf{d}}_8^w$
Middle		${\mathbf{d}}_9^w$	${\mathbf{d}}_{10}^w$	${\mathbf{d}}_{11}^w$	${\mathbf{d}}_{12}^w$
Ring		${\mathbf{d}}_{13}^w$	${\mathbf{d}}_{14}^w$	${\mathbf{d}}_{15}^w$	${\mathbf{d}}_{16}^w$
Little		${\mathbf{d}}_{17}^w$	${\mathbf{d}}_{18}^w$	${\mathbf{d}}_{19}^w$	${\mathbf{d}}_{20}^w$

shows the representation of these measured 3D spatial coordinates for each joint of the thumb, index, middle, ring, and little fingers.

Figure 6 shows the 2D hand-tracking data and the corresponding raw 3D calibrated measured data reconstruction. On the left, the RGB image shows a hand in a front-facing position, where key hand landmarks are tracked using the MediaPipe framework. These landmarks are accurately projected onto the image plane, ensuring comprehensive coverage of all joint locations. On the right, we visualize the 3D point cloud of these landmarks using depth data from an RGB-D sensor. The depth values associated with each landmark are mapped to their respective x and y coordinates derived from the MediaPipe are mapped to the depth buffer (Appendix A).

Table 2. An measured example of 3D spatial coordinates ${\mathbf{d}}_i^w$ of 21 hand landmarks.

	Joints	MCP (1st Layer)	PIP (2nd Layer)	DIP (3rd Layer)	Tip (4th Layer)
Finger
Wrist		(−0.21, −0.11, 0.54)
Thumb		(−0.19, −0.15, 0.53 )	(−0.17, −0.18, 0.54)	(−0.15, −0.20, 0.55)	(−0.14, −0.22, 0.56)
Index		(−0.13, −0.16, 0.56)	(−0.10, −0.16, 0.56)	(−0.08, −0.16, 0.55)	(−0.06, −0.16, 0.55)
Middle		(−0.13, −0.14, 0.57)	(−0.09, −0.14, 0.57)	(−0.07, −0.14, 0.56)	(−0.05, −0.14, 0.55)
Ring		(−0.13, −0.12, 0.57)	(−0.10, −0.12, 0.57)	(−0.07, −0.12, 0.56)	(−0.06, −0.12, 0.56)
Little		(−0.14, −0.10, 0.57)	(−0.11, −0.10, 0.57)	(−0.09, −0.10, 0.57)	(−0.07, −0.10, 0.57)

provides the 3D hand landmark data ${\mathbf{d}}_i^w$ in meters relative to the world frame, where the origin is aligned with the RGB-D sensor. The positive x-axis extends to the right, the positive y-axis points downward, and the z-axis represents the depth from the sensor.

It is important to note that the fixed-length parameters used in this study are derived from the physical measurements of a single individual. While this enables precise modeling for a given user, the approach does not capture variability in hand sizes or joint constraints across the population. As such, unrealistic joint configurations may arise if this model is applied directly to users with different hand geometries. Future work will explore methods to generalize the model by incorporating user-specific calibration procedures or employing learning-based techniques to estimate anatomical parameters from motion data.

In the following section, we present a numerical example illustrating the process of resolving parameters for hierarchical transformations in the index finger after each transformation definition.

4.1. Resolution of Palm-Coordinate Frame

In this subsection, we present the method for computing the palm’s orientation and position which is used as the hand local coordinate system at $\left\{\mathbf{0}\right\}$.

Position Parameters:The position vector ${\mathbf{d}}_0^w={(x_0,y_0,z_0)}^T$, directly measured from Mediapipe and the RGB-D sensor, specifies the distance of translation between the origins of the wrist frame $\left\{\mathbf{0}\right\}$ and the world frame $\left\{\mathbf{W}\right\}$ along ${\widehat{\mathbf{X}}}_w$, ${\widehat{\mathbf{Y}}}_w$, and ${\widehat{\mathbf{Z}}}_w$, respectively. For example, the position vector:

$${\mathbf{d}}_0^w=(-0.2113,-0.1128,0.5380)$$

(3)

indicates that the wrist landmark is located 0.21 m to the left, 0.11 m above the sensor, and 0.54 m away from the sensor, as measured by the MediaPipe through calibrated RGB-D measures.

Orientation Parameters: In the kinematic model of the hand, the orientation of the palm is with respect to the world frame is parameterized by three degrees of freedom (DOF). Let ${\mathbf{R}}_x\left({\alpha }_0\right)$, ${\mathbf{R}}_y\left({\beta }_0\right)$, and ${\mathbf{R}}_z\left({\gamma }_0\right)$ represent rotations about axes ${\widehat{\mathbf{X}}}_0$, ${\widehat{\mathbf{Y}}}_0$, and ${\widehat{\mathbf{Z}}}_0$, respectively, where ${\alpha }_0$, ${\beta }_0$, and ${\gamma }_0$ denote the corresponding rotation angles. Given the set of Euler angles $\{{\alpha }_0,{\beta }_0,{\gamma }_0\}$, the orientation matrix of the wrist frame $\left\{\mathbf{0}\right\}$ can be computed by converting the Euler angles to a rotation matrix:

$$\begin{array}{cc}\hfill {\mathbf{R}}_0^w& ={\mathbf{R}}_z\left({\gamma }_0\right){\mathbf{R}}_y\left({\beta }_0\right){\mathbf{R}}_x\left({\alpha }_0\right)\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_0& -sin{\gamma }_0& 0\\ sin{\gamma }_0& cos{\gamma }_0& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}cos{\beta }_0& 0& sin{\beta }_0\\ 0& 1& 0\\ -sin{\beta }_0& 0& cos{\beta }_0\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& cos{\alpha }_0& -sin{\alpha }_0\\ 0& sin{\alpha }_0& cos{\alpha }_0\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_{0}cos{\beta }_0& cos{\gamma }_{0}sin{\beta }_{0}sin{\alpha }_0-sin{\gamma }_{0}cos{\alpha }_0& cos{\gamma }_{0}sin{\beta }_{0}cos{\alpha }_0+sin{\gamma }_{0}sin{\alpha }_0\\ sin{\gamma }_{0}cos{\beta }_0& sin{\gamma }_{0}sin{\beta }_{0}sin{\alpha }_0+cos{\gamma }_{0}cos{\alpha }_0& sin{\gamma }_{0}sin{\beta }_{0}sin{\alpha }_0-cos{\gamma }_{0}sin{\alpha }_0\\ -sin{\beta }_0& cos{\beta }_{0}sin{\alpha }_0& cos{\beta }_{0}cos{\alpha }_0\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}{\widehat{\mathbf{X}}}_0^w& {\widehat{\mathbf{Y}}}_0^w& {\widehat{\mathbf{Z}}}_0^w\end{array}\right]\hfill \end{array}$$

(4)

where each column of ${\mathbf{R}}_0^w$ denotes the principal directions of the wrist frame $\left\{\mathbf{0}\right\}$ in terms of the world frame $\left\{\mathbf{W}\right\}$. This matrix encapsulates the wrist’s orientation in the global coordinate system.

To resolve the palm’s orientation parameters, we first define the ${\widehat{\mathbf{Y}}}_0$ of the wrist frame $\left\{\mathbf{0}\right\}$ as the normalized vector between landmark 5 (base of the index finger) and landmark 13 (base of the ring finger). These two landmarks provide a consistent reference indicator for of the construction of the palm’s orientation matrix, or:

$${\widehat{\mathbf{Y}}}_0^w={\displaystyle \frac{{\mathbf{d}}_{13}^w-{\mathbf{d}}_5^w}{\parallel {\mathbf{d}}_{13}^w-{\mathbf{d}}_5^w\parallel }}.$$

(5)

Next, we define the ${\widehat{\mathbf{Z}}}_0$ as the normal vector to the above defined palm plane. We define this vector through selection of two vectors: one from landmark 0 (wrist) to landmark 9 (base of the middle finger) and ${\widehat{\mathbf{Y}}}_0$ as:

$${\widehat{\mathbf{Z}}}_0^w={\displaystyle \frac{{\mathbf{d}}_9^w-{\mathbf{d}}_0^w}{\parallel {\mathbf{d}}_9^w-{\mathbf{d}}_0^w\parallel }}\times {\widehat{\mathbf{Y}}}_0^w.$$

(6)

Finally, the unit vector ${\widehat{\mathbf{X}}}_0$ is determined using the right-hand rule, ensuring a consistent and orthogonal coordinate construction:

$${\widehat{\mathbf{X}}}_0^w={\widehat{\mathbf{Y}}}_0^w\times {\widehat{\mathbf{Z}}}_0^w,$$

(7)

Given the description of ${\mathbf{R}}_0^w$ in terms of the palm landmarks, the parametrization of the hand orientation with respect to the world frame can be described using three Euler angles $({\alpha }_0,{\beta }_0,{\gamma }_0)$, which can be obtained through inverse calculation as:

$$\begin{array}{cc}\hfill {\beta }_0& =arcsin(-{\widehat{X}}_{0,z}^w)\hfill \\ \hfill {\gamma }_0& =arctan({\widehat{X}}_{0,y}^w,{\widehat{X}}_{0,x}^w)\hfill \\ \hfill {\alpha }_0& =arctan({\widehat{Y}}_{0,z}^w,{\widehat{Z}}_{0,z}^w)\hfill \end{array}$$

(8)

Given the position vector of the hand wrist as ${\mathbf{d}}_0^w={(x_0,y_0,z_0)}^T$, the position and orientation of frame are determined. We represent the final pose matrix of the palm using a homogeneous transformation matrix:

$${\mathbf{T}}_0^w=\left[\begin{array}{cc}{\mathbf{R}}_0^w& {\mathbf{d}}_0^w\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]$$

(9)

Follow Equations (5)–(7), the principal unit axes of the wrist frame $\left\{\mathbf{0}\right\}$ can be determined as:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{Y}}}_0^w& ={\displaystyle \frac{{\mathbf{d}}_{13}^w-{\mathbf{d}}_5^w}{\parallel {\mathbf{d}}_{13}^w-{\mathbf{d}}_5^w\parallel }}\hfill \\ \hfill & ={\displaystyle \frac{(0.0017,0.0380,0.0090)}{\parallel (0.0017,0.0380,0.0090)\parallel }}\hfill \\ \hfill & =(0.0427,0.9720,0.2310)\hfill \\ \hfill {\widehat{\mathbf{Z}}}_0^w& ={\displaystyle \frac{{\mathbf{d}}_9^w-{\mathbf{d}}_0^w}{\parallel {\mathbf{d}}_9^w-{\mathbf{d}}_0^w\parallel }}\times {\widehat{\mathbf{Y}}}_0^w.\hfill \\ \hfill & ={\displaystyle \frac{(0.0837,-0.0235,0.0320)}{\parallel (0.0837,-0.0235,0.0320)\parallel }}\times (0.0427,0.9720,0.2310)\hfill \\ \hfill & =(-0.3963,-0.1957,0.8970)\hfill \\ \hfill {\widehat{\mathbf{X}}}_0^w& ={\widehat{\mathbf{Y}}}_0^w\times {\widehat{\mathbf{Z}}}_0^w\hfill \\ \hfill & =(0.0427,0.9720,0.2310)\times (-0.3963,-0.1957,0.8970)\hfill \\ \hfill & =(0.9171,-0.1299,0.3768)\hfill \end{array}$$

(10)

Given the above resolved palm-coordinate frame from the landmark data, we can construct the homogeneous transformation matrix ${\mathbf{T}}_0^w$ of the palm-coordinate frame located at landmark $\mathbf{0}$ with respect to the world frame as:

$${\mathbf{T}}_0^w=\left[\begin{array}{cccc}0.9171& 0.0427& -0.3963& -0.2113\\ -0.1299& 0.9720& -0.1957& -0.1128\\ 0.3768& 0.2310& 0.8970& 0.5380\\ 0& 0& 0& 1\end{array}\right]$$

(11)

The Euler angles for the wrist frame can be resolved using Equation (8) as follows (Figure 7):

$$\begin{array}{cc}\hfill {\gamma }_0& =arctan\left({\displaystyle \frac{{\widehat{X}}_{0,y}^w}{{\widehat{X}}_{0,x}^w}}\right)=-0.14\mathrm{radians}=-{8.06}^{\circ }\hfill \\ \hfill {\beta }_0& =arcsin(-{\widehat{X}}_{0,z}^w)=-0.39\mathrm{radians}=-{22.14}^{\circ }\hfill \\ \hfill {\alpha }_0& =arctan\left({\displaystyle \frac{{\widehat{Y}}_{0,z}^w}{{\widehat{Z}}_{0,z}^w}}\right)=0.25\mathrm{radians}={14.44}^{\circ }\hfill \end{array}$$

(12)

When comparing these values to Figure 3, the negative value of ${\gamma }_0$ can be attributed to the slight counter-clockwise rotation of the hand around ${\widehat{\mathbf{Z}}}_w$. This can be verified by comparing the y-coordinates of landmarks 0 and 9, where $d_{0,y}^w=-0.11$ and $d_{9,y}^w=-0.14$, indicating an upward shift. Similarly, the negative value of ${\beta }_0$ corresponds to a slight counter-clockwise rotation around ${\widehat{\mathbf{Y}}}_w$ (pointing downward), which is corroborated by comparing the depth values of landmarks 0 and 9: $d_{0,z}^w=0.54$ and $d_{9,z}^w=0.57$. Additionally, by checking the depth values of the index-finger root ($d_{5,z}^w=0.55$) and little-finger root ($d_{17,z}^w=0.57$), we conclude that the hand exhibits a slight clockwise rotation around the ${\widehat{\mathbf{X}}}_w$, as indicated by the positive value of the corresponding angle.

Base-Length Parameters: The base lengths $\{l_{0,1},l_{0,5},l_{0,9},l_{0,13},l_{0,17}\}$ represent the distances between the origin of the local frame $\left\{\mathbf{0}\right\}$ and the origins of local frames $\left\{\mathbf{1}\right\}$, $\left\{\mathbf{5}\right\}$, $\left\{\mathbf{9}\right\}$, $\left\{\mathbf{13}\right\}$, and $\left\{\mathbf{17}\right\}$ at the first layer. These measurements, in centimeters, are computed by:

$$l_{0,i}=\parallel {\mathbf{d}}_i^w-{\mathbf{d}}_0^w\parallel ,i=1,5,9,13,17.$$

(13)

Using Equation (13), we compute the distances between the wrist and each of the finger-base landmarks. These lengths are critical for defining the spatial relationships between the wrist and finger joints, which are fundamental for kinematic modeling and inverse kinematic calculations. The results are expressed both in meters and centimeters for clarity:

$$\begin{array}{cc}\hfill l_{0,1}& =\parallel {\mathbf{d}}_1^w-{\mathbf{d}}_0^w\parallel =0.045\mathrm{m}=4.5\mathrm{cm}\hfill \\ \hfill l_{0,5}& =\parallel {\mathbf{d}}_5^w-{\mathbf{d}}_0^w\parallel =0.095\mathrm{m}=9.5\mathrm{cm}\hfill \\ \hfill l_{0,9}& =\parallel {\mathbf{d}}_9^w-{\mathbf{d}}_0^w\parallel =0.093\mathrm{m}=9.3\mathrm{cm}\hfill \\ \hfill l_{0,13}& =\parallel {\mathbf{d}}_{13}^w-{\mathbf{d}}_0^w\parallel =0.089\mathrm{m}=8.9\mathrm{cm}\hfill \\ \hfill l_{0,17}& =\parallel {\mathbf{d}}_{17}^w-{\mathbf{d}}_0^w\parallel =0.086\mathrm{m}=8.6\mathrm{cm}\hfill \end{array}$$

(14)

These values quantify the distances from the wrist landmark ${\mathbf{d}}_0^w$ to the base joints of the thumb (${\mathbf{d}}_1^w$), index finger (${\mathbf{d}}_5^w$), middle finger (${\mathbf{d}}_9^w$), ring finger (${\mathbf{d}}_{13}^w$), and little finger (${\mathbf{d}}_{17}^w$).

Angle Parameters: The angles $\{{\theta }_1,{\theta }_5,{\theta }_9,{\theta }_{13},{\theta }_{17}\}$ describe the rotational offset between the ${\widehat{\mathbf{X}}}_0$ axis and the axes ${\widehat{\mathbf{X}}}_1$, ${\widehat{\mathbf{X}}}_5$, ${\widehat{\mathbf{X}}}_9$, ${\widehat{\mathbf{X}}}_{13}$, and ${\widehat{\mathbf{X}}}_{17}$, which correspond to the bases of the thumb, index, middle, ring, and little fingers, respectively. Since we assume that local frames of the first layer for each finger, $\left\{\mathbf{1}\right\}$, $\left\{\mathbf{5}\right\}$, $\left\{\mathbf{9}\right\}$, $\left\{\mathbf{13}\right\}$, and $\left\{\mathbf{17}\right\}$ lie within the $xy$-plane of the wrist frame $\left\{\mathbf{0}\right\}$, the relative positions of these frames with respect to $\left\{\mathbf{0}\right\}$, denoted by ${\mathbf{d}}_i^0$, can be expressed as:

$$\begin{array}{cc}\hfill {\mathbf{d}}_i^0& =\left[\begin{array}{ccc}cos{\theta }_i& -sin{\theta }_i& 0\\ sin{\theta }_i& cos{\theta }_i& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{l}_{0,i}\\ 0\\ 0\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{c}{l}_{0,i}cos{\theta }_i\\ l_{0,i}sin{\theta }_i\\ 0\end{array}\right],i=1,5,9,13,17,\hfill \end{array}$$

(15)

Here, $l_{0,i}$ is computed by Equation (13) and ${\theta }_i$ is the angle that describes the rotation required to align each finger’s base frame with the wrist frame. To compute the angles ${\theta }_i$, we can use Equation (15) to match the observed positions of the finger-base landmarks relative to the wrist in the world frame. Specifically, given the known measured coordinates of landmarks ${\mathbf{d}}_i^w$ in the world frame $\left\{\mathbf{W}\right\}$, we can map these coordinates into the local wrist frame $\left\{\mathbf{0}\right\}$. This transformation is achieved by applying the inverse of the transformation matrix ${\mathbf{T}}_0^w$, which encodes the position and orientation of the wrist frame relative to the world frame:

$${\mathbf{d}}_i^0={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_i^w,i=1,2,\dots ,20.$$

(16)

In the local wrist frame, the transformed positions ${\mathbf{d}}_i^0$ are parameterized as given in Equation (15). We can solve for ${\theta }_i$ as:

$${\theta }_i=\mathrm{atan}2(d_{i,y}^0,d_{i,x}^0)$$

(17)

where $d_{i,x}^0$ and $d_{i,y}^0$ are the x- and y-components of the transformed vector ${\mathbf{d}}_i^0$.

Given Equations (16) and (17), we calculate the angular parameters for the fingers relative to the wrist frame (Figure 8). By transforming the world coordinates of each finger base into the wrist coordinate frame, we can compute the corresponding angle parameters as follows:

$$\begin{array}{cc}\hfill {\mathbf{d}}_1^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_1^w=(0.0213,-0.0380,0)\hfill \\ \hfill {\theta }_1& =arctan(d_{1,y}^0,d_{1,x}^0)=-1.06\mathrm{radians}=-{60.68}^{\circ }\hfill \\ \hfill {\mathbf{d}}_5^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_5^w=(0.0892,-0.0327,0)\hfill \\ \hfill {\theta }_5& =arctan(d_{5,y}^0,d_{5,x}^0)=-0.35\mathrm{radians}=-{20.14}^{\circ }\hfill \\ \hfill {\mathbf{d}}_9^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_9^w=(0.0919,-0.0119,0)\hfill \\ \hfill {\theta }_9& =arctan(d_{9,y}^0,d_{9,x}^0)=-0.13\mathrm{radians}=-{7.38}^{\circ }\hfill \\ \hfill {\mathbf{d}}_{13}^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_{13}^w=(0.0892,0.0062,0)\hfill \\ \hfill {\theta }_{13}& =arctan(d_{13,y}^0,d_{13,x}^0)=0.07\mathrm{radians}={4.03}^{\circ }\hfill \\ \hfill {\mathbf{d}}_{17}^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_{17}^w=(0.0815,0.0262,0)\hfill \\ \hfill {\theta }_{17}& =arctan(d_{17,y}^0,d_{17,x}^0)=0.31\mathrm{radians}={17.81}^{\circ }\hfill \end{array}$$

(18)

4.2. Resolving Layered Parameters of Fingers in Hierarchical Transformations

In the kinematic model of the hand as introduced in previous section, the metacarpophalangeal (MCP), proximal interphalangeal (PIP), and distal interphalangeal (DIP) joints are organized within a hierarchical structure. Each joint’s local coordinate frame is derived from its parent joint’s frame, establishing a consistent chain of transformations from the base of the finger to the fingertip. This section provides a comprehensive explanation of how the parameters for these hierarchical transformations are resolved, detailing how each joint’s movement is propagated along the kinematic chain to accurately model finger rotations.

First Layer (at MCP Joints): Each MCP joint has two DOF, enabling it to bend up and down and move side to side; five pairs of rotation angles ${\beta }_i$ and ${\gamma }_i$ around axes ${\widehat{\mathbf{Y}}}_i$ and ${\widehat{\mathbf{Z}}}_i$ for the root joints of each finger are defined in the previous section. Thus, the orientation of local frames at MCP joints with respect to wrist frame $\left\{\mathbf{0}\right\}$ can be expressed as:

$$\begin{array}{cc}\hfill {\mathbf{R}}_i^0& ={\mathbf{R}}_z\left({\gamma }_i\right){\mathbf{R}}_y\left({\beta }_i\right)\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_i& -sin{\gamma }_i& 0\\ sin{\gamma }_i& cos{\gamma }_i& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}cos{\beta }_i& 0& sin{\beta }_i\\ 0& 1& 0\\ -sin{\beta }_i& 0& cos{\beta }_i\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_{i}cos{\beta }_i& -sin{\gamma }_i& cos{\gamma }_{i}sin{\beta }_i\\ sin{\gamma }_{i}cos{\beta }_i& cos{\gamma }_i& sin{\gamma }_{i}sin{\beta }_i\\ -sin{\beta }_i& 0& cos{\beta }_i\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}{\widehat{\mathbf{X}}}_i^0& {\widehat{\mathbf{Y}}}_i^0& {\widehat{\mathbf{Z}}}_i^0\end{array}\right],i=1,5,9,13,17\hfill \end{array}$$

(19)

To compute the angles ${\gamma }_i$ and ${\beta }_i$ based on the rotation matrix ${\mathbf{R}}_i^0=\left[\begin{array}{ccc}{\widehat{\mathbf{X}}}_i^0& {\widehat{\mathbf{Y}}}_i^0& {\widehat{\mathbf{Z}}}_i^0\end{array}\right]$, we can utilize the geometric relationship between these axes and the wrist frame’s axes. Along each of the five fingers, the axes $\widehat{\mathbf{X}}$ for each of the three joints orient towards the origin of the next joint’s frame; therefore

$${\widehat{\mathbf{X}}}_i^0={\displaystyle \frac{{\mathbf{d}}_{i+1}^0-{\mathbf{d}}_i^0}{\parallel {\mathbf{d}}_{i+1}^0-{\mathbf{d}}_i^0\parallel }},i=1,5,9,13,17$$

(20)

Based on the kinematic model we defined, the axes ${\widehat{\mathbf{Y}}}_i$ of the local frames for each finger align with the axes of the revolute joints, which are perpendicular to both the corresponding ${\widehat{\mathbf{X}}}_i$ and ${\widehat{\mathbf{Z}}}_0$, the z-axis of the wrist. By computing the cross product of ${\widehat{\mathbf{Z}}}_0$ and ${\widehat{\mathbf{X}}}_i^0$, we get

$${\widehat{\mathbf{Y}}}_i^0={\displaystyle \frac{{\widehat{\mathbf{Z}}}_0\times {\widehat{\mathbf{X}}}_i^0}{\parallel {\widehat{\mathbf{Z}}}_0\times {\widehat{\mathbf{X}}}_i^0\parallel }},i=1,5,9,13,17$$

(21)

Finally, the unit axes ${\widehat{\mathbf{Z}}}_i^0$ of the local frames at the first layer are calculated as the cross product of the corresponding ${\widehat{\mathbf{X}}}_i^0$ and ${\widehat{\mathbf{Y}}}_i^0$:

$${\widehat{\mathbf{Z}}}_i^0={\widehat{\mathbf{X}}}_i^0\times {\widehat{\mathbf{Y}}}_i^0,i=1,5,9,13,17$$

(22)

By integrating with the transformed position vectors ${\mathbf{d}}_i^0$, we obtain the transformations at the first layer:

$${\mathbf{T}}_i^0=\left[\begin{array}{cc}{\mathbf{R}}_i^0& {\mathbf{d}}_i^0\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right],i=1,5,9,13,17.$$

(23)

${\beta }_i$ and ${\gamma }_i$ at the first layer can be extracted from the element in the third row, third column and the second row, second column of the rotation matrix ${\mathbf{R}}_i^0$ in Equation (19):

$$\begin{array}{cc}\hfill {\beta }_i& =arccos\left({\widehat{Z}}_{i,z}^0\right)\hfill \\ \hfill {\gamma }_i& =arccos\left({\widehat{Y}}_{i,y}^0\right),i=1,5,9,13,17\hfill \end{array}$$

(24)

Transformation to the First Layer of the Measured Index Finger: Based on Equations (19)–(24), the three principal unit vectors for the local frame $\left\{\mathbf{5}\right\}$ at the root of the index finger can be derived as follows:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{X}}}_5^0& ={\displaystyle \frac{{\mathbf{d}}_6^0-{\mathbf{d}}_5^0}{\parallel {\mathbf{d}}_6^0-{\mathbf{d}}_5^0\parallel }}={\displaystyle \frac{(0.0300,-0.0048,-0.0137)}{\parallel (0.0300,-0.0048,-0.0137)\parallel }}=(0.9006,-0.1439,-0.4102)\hfill \\ \hfill {\widehat{\mathbf{Y}}}_5^0& ={\displaystyle \frac{{\widehat{\mathbf{Z}}}_0\times {\widehat{\mathbf{X}}}_5^0}{\parallel {\widehat{\mathbf{Z}}}_0\times {\widehat{\mathbf{X}}}_5^0\parallel }}={\displaystyle \frac{(0.1439,0.9006,0)}{\parallel (0.1439,0.9006,0)\parallel }}=(0.1577,-0.9875,0)\hfill \\ \hfill {\widehat{\mathbf{Z}}}_5^0& ={\widehat{\mathbf{X}}}_5^0\times {\widehat{\mathbf{Y}}}_5^0=(0.4050,-0.0647,0.9120)\hfill \end{array}$$

(25)

We then form the rotation matrix ${\mathbf{R}}_5^0$ using these unit vectors:

$$\begin{array}{cc}\hfill {\mathbf{R}}_5^0& =\left[\begin{array}{ccc}{\widehat{\mathbf{X}}}_5^0& {\widehat{\mathbf{Y}}}_5^0& {\widehat{\mathbf{Z}}}_5^0\end{array}\right]=\left[\begin{array}{ccc}0.9006& 0.1577& 0.4050\\ -0.1439& 0.9875& -0.0647\\ -0.4102& 0& 0.9120\end{array}\right]\hfill \\ \hfill & ={\mathbf{R}}_z\left({\gamma }_5\right){\mathbf{R}}_y\left({\beta }_5\right)\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_5& -sin{\gamma }_5& 0\\ sin{\gamma }_5& cos{\gamma }_5& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}cos{\beta }_5& 0& sin{\beta }_5\\ 0& 1& 0\\ -sin{\beta }_5& 0& cos{\beta }_5\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\gamma }_{5}cos{\beta }_5& -sin{\gamma }_5& cos{\gamma }_{5}sin{\beta }_5\\ sin{\gamma }_{i}cos{\beta }_5& cos{\gamma }_5& sin{\gamma }_{5}sin{\beta }_5\\ -sin{\beta }_5& 0& cos{\beta }_5\end{array}\right]\hfill \end{array}$$

(26)

The angle parameters at the index finger MCP joint can now be calculated as:

$$\begin{array}{cc}\hfill {\beta }_5& =arccos\left({\widehat{Z}}_{5,z}^0\right)=arccos\left(0.9120\right)=0.4227\mathrm{radians}={24.22}^{\circ }\hfill \\ \hfill {\gamma }_5& =arccos\left({\widehat{Y}}_{5,y}^0\right)=arccos\left(0.9875\right)=0.1583\mathrm{radians}={9.07}^{\circ }\hfill \end{array}$$

(27)

As shown in Figure 9, the local frame $\left\{\mathbf{5}\right\}$, corresponding to the index finger MCP joint, can be described by a rotation relative to the wrist frame $\left\{\mathbf{0}\right\}$ first by ${9.07}^{\circ }$ about ${\widehat{\mathbf{Z}}}_0$ and then by ${24.22}^{\circ }$ about ${\widehat{\mathbf{Y}}}_0$. Additionally, there is a translation of $0.0892$ m along ${\widehat{\mathbf{X}}}_0$ and $-0.0327$ m along ${\widehat{\mathbf{Y}}}_0$. The transformation matrix ${\mathbf{T}}_5^0$ that defines the relationship between the index finger root and the wrist frame can be expressed as:

$${\mathbf{T}}_5^0=\left[\begin{array}{cc}{\mathbf{R}}_5^0& {\mathbf{d}}_5^0\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]=\left[\begin{array}{cccc}0.9006& 0.1577& 0.4050& 0.0892\\ -0.1439& 0.9875& -0.0647& -0.0327\\ -0.4102& 0& 0.9120& 0\\ 0& 0& 0& 1\end{array}\right]$$

(28)

This transformation provides the full description of the index finger’s MCP joint in relation to the wrist frame, accounting for both rotational and translational movements.

Second and Third Layer (at PIP and DIP Joints): Each of the five fingers can be modeled as a kinematic chain consisting of three links connected by three revolute joints, with parallel rotation axes. The PIP and DIP joints, which form the second and third layers of this chain, each possess a single DOF. Consequently, the rotation of each joint in these layers can be fully described by a single rotational parameter ${\beta }_j$ for the PIP joint and ${\beta }_{j+1}$ for the DIP joint, where $j=2,6,10,14,18$. These angles represent the flexion-extension motions at the respective joints and are defined as the angle between ${\widehat{\mathbf{X}}}_j$ and ${\widehat{\mathbf{X}}}_{j-1}$ for the PIP joint and between ${\widehat{\mathbf{X}}}_{j+1}$ and ${\widehat{\mathbf{X}}}_j$ for the DIP joint.

Since all finger movements are considered locally within each kinematic chain, relative to the base frames of each finger root (i.e., $\left\{\mathbf{1}\right\}$, $\left\{\mathbf{5}\right\}$, $\left\{\mathbf{9}\right\}$, $\left\{\mathbf{13}\right\}$, and $\left\{\mathbf{17}\right\}$), the rotation angles will be computed based on the relative orientations of the joints within the preceding joint’s frame:

$$\begin{array}{cc}\hfill cos{\beta }_j& ={\displaystyle \frac{{\widehat{\mathbf{X}}}_j^{j-1}·{\widehat{\mathbf{X}}}_{j-1}^{j-1}}{\parallel {\widehat{\mathbf{X}}}_j^{j-1}\parallel \parallel {\widehat{\mathbf{X}}}_{j-1}^{j-1}\parallel }},j=2,6,10,14,18\hfill \\ \hfill & ={\displaystyle \frac{({\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1})·({\mathbf{d}}_j^{j-1}-{\mathbf{d}}_{j-1}^{j-1})}{\parallel {\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1}\parallel \parallel {\mathbf{d}}_j^{j-1}-{\mathbf{d}}_{j-1}^{j-1}\parallel }}\hfill \\ \hfill {\beta }_j& =arccos{\displaystyle \frac{({\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1})·({\mathbf{d}}_j^{j-1}-{\mathbf{d}}_{j-1}^{j-1})}{\parallel {\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1}\parallel \parallel {\mathbf{d}}_j^{j-1}-{\mathbf{d}}_{j-1}^{j-1}\parallel }}\hfill \end{array}$$

(29)

where

$$\begin{array}{cc}\hfill {\mathbf{d}}_j^{j-1}& ={\left({\mathbf{T}}_{j-1}^0\right)}^{-1}{\mathbf{d}}_j^0\hfill \\ \hfill {\mathbf{d}}_{j+1}^{j-1}& ={\left({\mathbf{T}}_{j-1}^0\right)}^{-1}{\mathbf{d}}_{j+1}^0\hfill \end{array}$$

(30)

Similarly, for the angle parameters at the third layer:

$$\begin{array}{cc}\hfill cos{\beta }_{j+1}& ={\displaystyle \frac{{\widehat{\mathbf{X}}}_{j+1}^{j-1}·{\widehat{\mathbf{X}}}_j^{j-1}}{\parallel {\widehat{\mathbf{X}}}_{j+1}^{j-1}\parallel \parallel {\widehat{\mathbf{X}}}_j^{j-1}\parallel }},j=2,6,10,14,18\hfill \\ \hfill & ={\displaystyle \frac{({\mathbf{d}}_{j+2}^{j-1}-{\mathbf{d}}_{j+1}^{j-1})·({\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1})}{\parallel {\mathbf{d}}_{j+2}^{j-1}-{\mathbf{d}}_{j+1}^{j-1}\parallel \parallel {\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1}\parallel }}\hfill \\ \hfill {\beta }_{j+1}& =arccos{\displaystyle \frac{({\mathbf{d}}_{j+2}^{j-1}-{\mathbf{d}}_{j+1}^{j-1})·({\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1})}{\parallel {\mathbf{d}}_{j+2}^{j-1}-{\mathbf{d}}_{j+1}^{j-1}\parallel \parallel {\mathbf{d}}_{j+1}^{j-1}-{\mathbf{d}}_j^{j-1}\parallel }}\hfill \end{array}$$

(31)

where

$${\mathbf{d}}_{j+2}^{j-1}={\left({\mathbf{T}}_{j-1}^0\right)}^{-1}{\mathbf{d}}_{j+2}^0$$

(32)

The orientation matrix of the local frame at joint j relative to its parent joint $j-1$ is given by:

$$\begin{array}{cc}\hfill {\mathbf{R}}_j^{j-1}& ={\mathbf{R}}_y\left({\beta }_j\right),j=2,6,10,14,18\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\beta }_j& 0& sin{\beta }_j\\ 0& 1& 0\\ -sin{\beta }_j& 0& cos{\beta }_j\end{array}\right]\hfill \end{array}$$

(33)

Similarly, the orientation matrix of the local frame at joint $j+1$ (DIP joint) relative to joint j (PIP joint) is:

$$\begin{array}{cc}\hfill {\mathbf{R}}_{j+1}^j& ={\mathbf{R}}_y\left({\beta }_{j+1}\right),j=2,6,10,14,18\hfill \\ \hfill & =\left[\begin{array}{ccc}cos{\beta }_{j+1}& 0& sin{\beta }_{j+1}\\ 0& 1& 0\\ -sin{\beta }_{j+1}& 0& cos{\beta }_{j+1}\end{array}\right]\hfill \end{array}$$

(34)

By integrating the translation vectors, the relative positions of the joints can be expressed as:

$${\mathbf{d}}_j^{j-1}=\left[\begin{array}{c}{l}_{j-1,j}\\ 0\\ 0\end{array}\right],{\mathbf{d}}_{j+1}^j=\left[\begin{array}{c}{l}_{j,j+1}\\ 0\\ 0\end{array}\right],j=2,6,10,14,18.$$

(35)

The complete transformations at the second layer (PIP joint) and the third layer (DIP joint) are then obtained as follows:

$${\mathbf{T}}_j^{j-1}=\left[\begin{array}{cc}{\mathbf{R}}_j^{j-1}& {\mathbf{d}}_j^{j-1}\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right],{\mathbf{T}}_{j+1}^j=\left[\begin{array}{cc}{\mathbf{R}}_{j+1}^j& {\mathbf{d}}_{j+1}^j\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right],j=2,6,10,14,18.$$

(36)

Transformations to the Second and Third Layer of the Measured Index Finger: Since all finger movements are modeled locally within each kinematic chain, relative to the base frames of each finger root, the rotation angles are computed based on the relative orientations of consecutive joints within the preceding joint’s frame. This hierarchical transformation approach ensures consistent rotational relationships between joints as they propagate along the kinematic chain.

First, the position vectors ${\mathbf{d}}_6^5$, ${\mathbf{d}}_7^5$ and ${\mathbf{d}}_8^5$ are calculated by transforming the parent wrist frame $\left\{\mathbf{0}\right\}$ into the local frame of the base joint (the MCP joint of the index finger, $\left\{\mathbf{5}\right\}$):

$$\begin{array}{cc}\hfill {\mathbf{d}}_6^0& ={\left({\mathbf{T}}_0^w\right)}^{-1}{\mathbf{d}}_6^w=(0.0892,-0.0327,0)\hfill \\ \hfill {\mathbf{d}}_6^5& ={\left({\mathbf{T}}_5^0\right)}^{-1}{\mathbf{d}}_6^0=(0.0333,0,0)\hfill \\ \hfill {\mathbf{d}}_7^5& ={\left({\mathbf{T}}_5^0\right)}^{-1}{\mathbf{d}}_7^0=(0.0520,0.0016,-0.0077)\hfill \\ \hfill {\mathbf{d}}_8^5& ={\left({\mathbf{T}}_5^0\right)}^{-1}{\mathbf{d}}_8^0=(0.0679,0.0037,-0.0114).\hfill \end{array}$$

(37)

To compute the rotation angle at the second layer (PIP joint), we perform a scalar product of the position vectors relative to the local frames. The angle ${\beta }_6$ is derived as follows:

$$\begin{array}{cc}\hfill cos{\beta }_6& ={\displaystyle \frac{{\widehat{\mathbf{X}}}_6^5·{\widehat{\mathbf{X}}}_5^5}{\parallel {\widehat{\mathbf{X}}}_6^5\parallel \parallel {\widehat{\mathbf{X}}}_5^5\parallel }}\hfill \\ \hfill & ={\displaystyle \frac{({\mathbf{d}}_7^5-{\mathbf{d}}_6^5)·({\mathbf{d}}_6^5-{\mathbf{d}}_5^5)}{\parallel {\mathbf{d}}_7^5-{\mathbf{d}}_6^5\parallel \parallel {\mathbf{d}}_6^5-{\mathbf{d}}_5^5\parallel }}\hfill \\ \hfill {\beta }_6& =arccos{\displaystyle \frac{(0.0187,0,-0.0077)·(0.0333,0,0)}{\parallel (0.0187,0,-0.0077)\parallel \parallel (0.0333,0,0)\parallel }}\hfill \\ \hfill & =0.3877\mathrm{radians}={22.22}^{\circ }\hfill \end{array}$$

(38)

For the angle parameters at the third layer (DIP joint), the rotation angle ${\beta }_7$ is computed similarly to the previous layer. Using the relative orientations of the joints, we can express ${\beta }_7$ as follows:

$$\begin{array}{cc}\hfill cos{\beta }_7& ={\displaystyle \frac{{\widehat{\mathbf{X}}}_7^5·{\widehat{\mathbf{X}}}_6^5}{\parallel {\widehat{\mathbf{X}}}_7^5\parallel \parallel {\widehat{\mathbf{X}}}_6^5\parallel }}\hfill \\ \hfill & ={\displaystyle \frac{({\mathbf{d}}_8^5-{\mathbf{d}}_7^5)·({\mathbf{d}}_7^5-{\mathbf{d}}_6^5)}{\parallel {\mathbf{d}}_8^5-{\mathbf{d}}_7^5\parallel \parallel {\mathbf{d}}_7^5-{\mathbf{d}}_6^5\parallel }}\hfill \\ \hfill {\beta }_7& =arccos{\displaystyle \frac{(0.0158,0,-0.0038)·(0.0187,0,-0.0077)}{\parallel (0.0158,0,-0.0038)\parallel \parallel (0.0187,0,-0.0077)\parallel }}\hfill \\ \hfill & =0.1531\mathrm{radians}={8.77}^{\circ }\hfill \end{array}$$

(39)

Here, each of ${\beta }_6={22.22}^{\circ }$ and ${\beta }_7={8.77}^{\circ }$ represents the flexion at the PIP and DIP joint of the index finger, as shown in Figure 10.

The orientation of the local frame $\left\{\mathbf{6}\right\}$ at the second joint (PIP joint) of the index finger relative to its parent joint (MCP joint) is represented by the matrix, which is defined by the rotation angle ${\beta }_6$ around the local ${\widehat{\mathbf{Y}}}_6$ -axis:

$$\begin{array}{cc}\hfill {\mathbf{R}}_6^5& ={\mathbf{R}}_y\left({\beta }_6\right)=\left[\begin{array}{ccc}cos{\beta }_6& 0& sin{\beta }_6\\ 0& 1& 0\\ -sin{\beta }_6& 0& cos{\beta }_6\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}0.9258& 0& 0.3781\\ 0& 1& 0\\ -0.3781& 0& 0.9258\end{array}\right]\hfill \end{array}$$

(40)

Similarly, the orientation matrix for the local frame $\left\{\mathbf{7}\right\}$ at the third joint (DIP joint) relative to the second joint (PIP joint) is:

$$\begin{array}{cc}\hfill {\mathbf{R}}_7^6& ={\mathbf{R}}_y\left({\beta }_7\right)=\left[\begin{array}{ccc}cos{\beta }_7& 0& sin{\beta }_7\\ 0& 1& 0\\ -sin{\beta }_7& 0& cos{\beta }_7\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}0.9883& 0& 0.1525\\ 0& 1& 0\\ -0.1525& 0& 0.9883\end{array}\right]\hfill \end{array}$$

(41)

The translational displacements between consecutive joints are defined by the following translation vectors:

$${\mathbf{d}}_6^5=\left[\begin{array}{c}{l}_{5,6}\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}0.0333\\ 0\\ 0\end{array}\right],{\mathbf{d}}_7^6=\left[\begin{array}{c}{l}_{6,7}\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}0.0202\\ 0\\ 0\end{array}\right]$$

(42)

The complete transformation matrices at the second (PIP joint) and third layers (DIP joint) are constructed by combining the rotation matrices with the corresponding translation vectors. The transformation matrix from the MCP joint to the PIP joint is given by:

$${\mathbf{T}}_6^5=\left[\begin{array}{cc}{\mathbf{R}}_6^5& {\mathbf{d}}_6^5\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]$$

(43)

Similarly, the transformation matrix from the PIP joint to the DIP joint is:

$${\mathbf{T}}_7^6=\left[\begin{array}{cc}{\mathbf{R}}_7^6& {\mathbf{d}}_7^6\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]$$

(44)

Finally, note that the frame $\left\{\mathbf{8}\right\}$ is located at the fingertip and, by design, has no degrees of freedom (DOF). As a result, the transformation matrix from joint $\left\{\mathbf{7}\right\}$ (DIP joint) to the fingertip $\left\{\mathbf{8}\right\}$ is a simple identity matrix with a translation vector representing the fixed length between joint $\left\{\mathbf{7}\right\}$ and the fingertip $\left\{\mathbf{8}\right\}$. This is given by:

$${\mathbf{T}}_8^7=\left[\begin{array}{cccc}1& 0& 0& l_{7,8}\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

(45)

Since there is no rotation or additional degrees of freedom at the fingertip, this transformation matrix simply translates the local frame by the fixed length $l_{7,8}$, which represents the distance between the DIP joint and the fingertip. This final transformation completes the kinematic chain of the index finger, preserving the hierarchical structure of the transformations from the MCP joint to the fingertip.

5. Kinematic of Hand for Graphical Reconstruction in Unity

After resolving the kinematic parameters of the hand model, we implemented a reconstruction of a simulated human hand in Unity. Unity’s advanced rendering capabilities enable realistic, interactive visualizations, which are essential for validating the accuracy of both the hand model and the reconstruction process. By reconstructing the hand’s movement from the kinematic data, we can analyze the accuracy of joint movements, finger articulation, and interactions with objects in a virtual environment. In this section, we detail the process of implementing a rigged hand model in Unity, the definition of local coordinate frames, and the conversion of kinematic model parameters into Unity’s coordinate system.

5.1. Unity Environment and Rigged Hand Model

Unity version 2022.3.5f1 (Long-Term Support) was used to implement and visualize the reconstructed kinematic hand model. A rigged avatar hand model with 16 joints (one wrist joint and three joints for each of the five fingers) was imported from the Unity Asset Store. Each joint in the model is controlled via local coordinate frames and arranged hierarchically to reflect the anatomical structure of the human hand.

Scripts written in C# were developed to update joint rotations in real time using computed kinematic parameters. The Unity camera was aligned to mimic the depth sensor’s viewpoint, ensuring that tracked and reconstructed hand movements were spatially consistent. Due to Unity’s use of a left-handed coordinate system, we explicitly inverted the y-axis and transformed all rotational parameters accordingly to match the right-handed sensor coordinate system defined in Section 3.

5.2. Left-Handed Frame in Unity

To graphically visualize hand movements, we utilize a rigged hand model in Unity, which replicates the actual texture and motion of the human hand. As shown in Figure 11, the hand model contains 16 movable joints, including the wrist and three joints for each of the five fingers $(1+3\times 5=16)$, excluding the fingertips.

In the rigged hand model, each joint is represented by a local coordinate frame ${\left\{\mathbf{U}\right\}}_{\mathbf{i}}$, for $i=0,1,\dots ,15$ that is oriented relative to the parent’s coordinate frame. The wrist frame ${\left\{\mathbf{U}\right\}}_{\mathbf{i}}$, being the root joint, serves as the origin for the hand’s coordinate system. Each subsequent joint has its own local frame, which is updated based on the joint’s current orientation and position.

In Unity, a camera object defines the viewer’s perspective, capturing the scene and rendering 3D objects. In our setup, we define the camera’s coordinate frame as $\left\{\mathbf{U}\right\}$, which corresponds to the depth sensor’s frame in the real-world setup. This camera object is positioned in Unity to mimic the physical sensor’s viewpoint, ensuring the virtual hand model moves in sync with real-world hand movements.

As stated previously, the world frame, denoted as $\left\{\mathbf{W}\right\}$, is established at the center of the depth sensor and defined by three orthogonal axes: ${\widehat{\mathbf{X}}}_w$, ${\widehat{\mathbf{Y}}}_w$, and ${\widehat{\mathbf{Z}}}_w$. In this setup, ${\widehat{\mathbf{X}}}_w$ points to the right, ${\widehat{\mathbf{Y}}}_w$ points downward, and ${\widehat{\mathbf{Z}}}_w$ extends outward from the sensor toward the objects being tracked.

However, Unity uses a left-handed coordinate system for its camera object, where ${\widehat{\mathbf{X}}}_u$ and ${\widehat{\mathbf{Z}}}_u$ in the camera frame $\left\{\mathbf{U}\right\}$ align with the ${\widehat{\mathbf{X}}}_w$ and ${\widehat{\mathbf{Z}}}_w$ of the world frame, respectively. The key difference is the orientation of the Y-axis: in Unity’s camera frame ${\widehat{\mathbf{Y}}}_u$, it points upward, opposite to the downward-pointing ${\widehat{\mathbf{Y}}}_w$ in the world frame. This inversion must be accounted for when transforming the hand model’s parameters to ensure that the visualized hand movements in Unity accurately reflect the real-world hand tracked by the depth sensor.

5.3. Transformation from Hand Kinematic Model to Rigged Hand

Once the joint angles have been resolved in the kinematic model using the hierarchical structure of the hand, they need to be mapped to the corresponding joint angles in Unity.

As shown in Figure 12, the local coordinate frames in the rigged hand model and the hand kinematic model are highly consistent in their definitions. Both models use a hierarchical structure, and the orientation of the x- and y-axes across the 16 joints is the same. However, a notable difference exists between the two models: the kinematic model utilizes a right-handed coordinate system, while the rigged hand model in Unity adopts a left-handed coordinate system. As a result, the z-axes in the two systems point in opposite directions. In the rigged hand model, the z-axis points perpendicular to the palm, facing toward the camera, whereas in the kinematic model, the z-axis points away from the hand and the camera.

Thus, in order to properly reconstruct hand movements in Unity, the parameters from the kinematic model must undergo a transformation. This transformation involves two key steps: coordinate transformation and local frame transformation.

Coordinate Transformation: If a point $\mathbf{p}$ has coordinates ${\mathbf{p}}_r=({\mathbf{p}}_x,{\mathbf{p}}_y,{\mathbf{p}}_z)$ in the right-handed coordinate system ${\mathbf{O}}_r$, and we define a left-handed coordinate system ${\mathbf{O}}_l$ at the same origin with the same x- and y-axes but with the z-axis pointing in the opposite direction, the coordinates of the point $\mathbf{p}$ in the left-handed coordinate system will be ${\mathbf{p}}_l=({\mathbf{p}}_x,{\mathbf{p}}_y,-{\mathbf{p}}_z)$. In matrix form, this transformation can be expressed as:

$${\mathbf{p}}_l=\left[\begin{array}{c}{\mathbf{p}}_x\\ {\mathbf{p}}_y\\ -{\mathbf{p}}_z\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{c}{\mathbf{p}}_x\\ {\mathbf{p}}_y\\ {\mathbf{p}}_z\end{array}\right]={\mathbf{R}}_{r\to l}{\mathbf{p}}_r$$

(46)

where the transformation matrix to switch from a right-hand system to a left-hand system is as follows:

$${\mathbf{R}}_{r\to l}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]$$

(47)

This transformation accounts for the difference in the handedness of the two coordinate systems, ensuring that the joint positions in Unity’s left-handed frame correctly correspond to the real-world positions tracked by the right-handed kinematic model.

Local Frame Transformation: Since the rigged hand and the kinematic hand model differ in their local coordinate frame conventions, the rotation matrices for each joint must also be transformed to reflect this change. Consider a local coordinate frame $\left\{\mathbf{O}\right\}$ in the right-handed system ${\mathbf{O}}_r$, where its orientation is represented by the rotation matrix ${\mathbf{R}}_r$. At the origin of this right-handed coordinate system ${\mathbf{O}}_r$, we can define a corresponding left-handed coordinate system ${\mathbf{O}}_l$ where the x- and y-axes remain the same, but the z-axis points in the opposite direction. We now seek to compute the orientation matrix ${\mathbf{R}}_l$ for the local coordinate frame $\left\{\mathbf{O}\right\}$ in this left-handed system.

Let ${\mathbf{p}}_r$ represent an arbitrary point in the right-handed system. After applying the rotation matrix, the transformed point in ${\mathbf{O}}_r$ is:

$${\mathbf{p}}_r^{\prime }={\mathbf{R}}_r{\mathbf{p}}_r$$

(48)

In the left-handed coordinate system ${\mathbf{O}}_l$, the corresponding points before and after rotation are represented by:

$${\mathbf{p}}_l={\mathbf{R}}_{r\to l}{\mathbf{p}}_r,{\mathbf{p}}_l^{\prime }={\mathbf{R}}_{r\to l}{\mathbf{p}}_r^{\prime }$$

(49)

Therefore, the transformed point ${\mathbf{p}}_l^{\prime }$ in the left-handed system can be expressed as:

$${\mathbf{p}}_l^{\prime }={\mathbf{R}}_{r\to l}{\mathbf{p}}_r^{\prime }={\mathbf{R}}_{r\to l}{\mathbf{R}}_r{\mathbf{p}}_r={\mathbf{R}}_{r\to l}{\mathbf{R}}_r{\mathbf{R}}_{r\to l}^{-1}{\mathbf{p}}_l$$

(50)

Thus, the orientation matrix of the local coordinate frame $\mathbf{O}$ in the left-handed system ${\mathbf{O}}_l$ is:

$${\mathbf{R}}_l={\mathbf{R}}_{r\to l}{\mathbf{R}}_r{\mathbf{R}}_{r\to l}^{-1}$$

(51)

Since the matrix ${\mathbf{R}}_{r\to l}$ represents a reflection operation, and reflections are involuntary transformations (applying them twice returns the original state). When we multiply ${\mathbf{R}}_{r\to l}$ by itself:

$${\mathbf{R}}_{r\to l}·{\mathbf{R}}_{r\to l}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]·\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]=\mathbf{I}$$

(52)

It follows that:

$${\mathbf{R}}_{r\to l}={\mathbf{R}}_{r\to l}^{-1}$$

(53)

In summary, to transform the hand kinematic model’s local coordinate frames to the rigged hand’s left-handed coordinate system in Unity, the orientation matrix for each of the 16 joints must be transformed using Equation (51). This ensures that all joint rotations and movements are accurately mapped from the real-world right-handed system to Unity’s left-handed system. For example, using the rotational transformation matrices ${\mathbf{R}}_5^0$, ${\mathbf{R}}_6^5$, ${\mathbf{R}}_7^6$ defined in the previous section using measured data, we can compute their corresponding rotation matrix in left-handed coordinate system:

$$\begin{array}{cc}\hfill {\mathbf{R}}_{5l}^0& ={\mathbf{R}}_{r\to l}{\mathbf{R}}_{5r}^0{\mathbf{R}}_{r\to l}^{-1}\hfill \\ \hfill & =\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{ccc}0.9006& 0.1577& 0.4050\\ -0.1439& 0.9875& -0.0647\\ -0.4102& 0& 0.9120\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}0.9006& 0.1577& -0.4050\\ -0.1439& 0.9875& 0.0647\\ 0.4102& 0& 0.9120\end{array}\right]\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill {\mathbf{R}}_{6l}^5& ={\mathbf{R}}_{r\to l}{\mathbf{R}}_{6r}^5{\mathbf{R}}_{r\to l}^{-1}\hfill \\ \hfill & =\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{ccc}0.9258& 0& 0.3781\\ 0& 1& 0\\ -0.3781& 0& 0.9258\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}0.9258& 0& -0.3781\\ 0& 1& 0\\ 0.3781& 0& 0.9258\end{array}\right]\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill {\mathbf{R}}_{7l}^6& ={\mathbf{R}}_{r\to l}{\mathbf{R}}_{7r}^6{\mathbf{R}}_{r\to l}^{-1}\hfill \\ \hfill & =\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{ccc}0.9883& 0& 0.1525\\ 0& 1& 0\\ -0.1525& 0& 0.9883\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}0.9883& 0& -0.1525\\ 0& 1& 0\\ 0.1525& 0& 0.9883\end{array}\right]\hfill \end{array}$$

(56)

5.4. Quaternions Representation for Joint Rotations

In Unity, rotations are most effectively represented using quaternions. Quaternions offer a robust method to represent rotations without suffering from gimbal lock, which can occur when using Euler angles. A quaternion is defined by four components: $(w,x,y,z)$, where w represents the scalar part, and $(x,y,z)$ forms the vector part, indicating the axis of rotation.

Given a rotation matrix obtained from the kinematic model, the corresponding quaternion is derived as follows:

$$\mathbf{q}=\left(w,x,y,z\right)=\left(cos\left({\displaystyle \frac{\theta }{2}}\right),u_{x}sin\left({\displaystyle \frac{\theta }{2}}\right),u_{y}sin\left({\displaystyle \frac{\theta }{2}}\right),u_{z}sin\left({\displaystyle \frac{\theta }{2}}\right)\right)$$

(57)

where $\theta$ is the rotation angle, and $\overrightarrow{u}=(u_x,u_y,u_z)$ is the normalized axis of rotation. This representation ensures that all joint rotations are smooth and efficient for computational purposes in Unity.

To convert a 3 × 3 rotation matrix into a quaternion, the components of the quaternion are calculated from the matrix elements [18]. The matrix is of the form:

$$\left[\begin{array}{ccc}{m}_{00}& m_{01}& m_{02}\\ m_{10}& m_{11}& m_{12}\\ m_{20}& m_{21}& m_{22}\end{array}\right]$$

(58)

and the quaternion components are computed as:

$$\begin{array}{cc}\hfill q_w& ={\displaystyle \frac{\sqrt{max(0,1+m_{00}+m_{11}+m_{22})}}{2}}\hfill \\ \hfill q_x& ={\displaystyle \frac{\sqrt{max(0,1+m_{00}-m_{11}-m_{22})}}{2}}\hfill \\ \hfill q_y& ={\displaystyle \frac{\sqrt{max(0,1-m_{00}+m_{11}-m_{22})}}{2}}\hfill \\ \hfill q_z& ={\displaystyle \frac{\sqrt{max(0,1-m_{00}-m_{11}+m_{22})}}{2}}\hfill \end{array}$$

(59)

Finally, the signs of $q_x$, $q_y$, and $q_z$ are adjusted based on the off-diagonal elements of the rotation matrix to ensure consistency with the direction of rotation:

$$\begin{array}{cc}\hfill q_x& =q_x·sign(q_x·(m_{21}-m_{12}))\hfill \\ \hfill q_y& =q_y·sign(q_y·(m_{02}-m_{20}))\hfill \\ \hfill q_z& =q_z·sign(q_z·(m_{10}-m_{01}))\hfill \end{array}$$

(60)

Therefore,

$${\mathbf{R}}_{5l}^0=\left[\begin{array}{ccc}0.9006& 0.1577& -0.4050\\ -0.1439& 0.9875& 0.0647\\ 0.4102& 0& 0.9120\end{array}\right]$$

(61)

produces

$$\begin{array}{cc}\hfill q_{5w}& ={\displaystyle \frac{\sqrt{max(0,1+0.9006+0.9875+0.9120)}}{2}}=0.9747\hfill \\ \hfill q_{5x}& ={\displaystyle \frac{\sqrt{max(0,1+0.9006-0.9875-0.9120)}}{2}}=0.0166\hfill \\ \hfill q_{5y}& ={\displaystyle \frac{\sqrt{max(0,1-0.9006+0.9875-0.9120)}}{2}}=0.2091\hfill \\ \hfill q_{5z}& ={\displaystyle \frac{\sqrt{max(0,1-0.9006-0.9875+0.9120)}}{2}}=0.0773\hfill \end{array}$$

(62)

and

$$\begin{array}{cc}\hfill q_{5x}& =q_{5x}·sign(q_{5x}·(0-0.0647))=-0.0166\hfill \\ \hfill q_{5y}& =q_{5y}·sign(q_{5y}·(-0.4050-0.4102))=-0.2091\hfill \\ \hfill q_{5z}& =q_{5z}·sign(q_{5z}·(-0.1439-0.1577))=-0.0773\hfill \end{array}$$

(63)

Finally,

$$\begin{array}{cc}\hfill {\mathbf{q}}_5& =(0.9747,-0.0166,-0.2091,-0.0773)\hfill \\ \hfill {\mathbf{q}}_6& =(0.9813,0,-0.1927,0)\hfill \\ \hfill {\mathbf{q}}_7& =(0.9971,0,-0.0765,0)\hfill \end{array}$$

(64)

While the current implementation assumes full visibility of all 21 landmarks, we acknowledge that occlusions—either from self-shadowing or external objects—can impair direct observation of certain joint positions. The proposed kinematic model, by establishing a hierarchical and constraint-based structure, allows for integration with future estimation methods such as Bayesian networks or optimization schemes to infer occluded landmarks. Addressing such occlusion scenarios is a natural extension of this work.

6. Discussions and Conclusions

6.1. Discussions

This paper presents various steps in developing a kinematic model of a dexterous hand using hand-pose-tracking data augmented with depth measures. The modeling method and the corresponding kinematic solutions were demonstrated through various measured data in resolving a hand kinematic parameters from captured RGB-D data. The 21 landmarks detected by MediaPipe were identified using calibrated RGB images where the corresponding depth values were measured (Figure 13). However, it was observed that as the hand moves, the data tends to become noisy. For instance, when the hand moves along a trajectory from the top left to the bottom right of the image, while MediaPipe continues to detect the 21 landmarks consistently, the associated depth values can contain outliers or even return invalid zero values. These discrepancies in the depth data can arise from factors such as occlusions, rapid hand movements, or limitations in the sensor’s precision.

As an example, Figure 14 illustrates the process of the hand approaching a cup in three consecutive stages. In Figure 14a, the last two joints of the index finger show irregular depth data: one is recorded as 0, and the other as a far-off value (likely the distance to the white wall behind the hand, instead of the finger itself). In Figure 14b, the corresponding reconstruction in Unity shows the misalignment, where the index finger in the virtual hand model differs from the actual position of the real index finger. In Figure 14c, depth outliers are detected in both the thumb and the ring finger, resulting in erroneous positioning. The Unity reconstruction in Figure 14d reflects this discrepancy. In Figure 14e, as the fingertips of four fingers are slightly occluded by the cup’s handle, their depth values are recorded as 0. This further contributes to inaccuracies in the Unity reconstruction, as shown in Figure 14f, where the fingers deviate from their true positions due to the occlusion-induced depth errors.

To address such issues in proper kinematic resolutions, the use of prior knowledge, such as fixed user’s finger lengths, and the use of inverse kinematic model as constraints can be used to better estimate missing and erroneous landmarks. This can offer an approach to calculate the best angular parameters for reconstructing accurate hand movements in Unity, even when sensor data is incomplete or noisy.

6.2. Conclusions and Future Work

This paper presents a structured methodology for constructing a 3D kinematic model of the human hand based on spatially calibrated landmark data. The proposed model establishes a hierarchical coordinate system rooted at the palm and resolves both forward and inverse kinematic parameters for each finger joint. These results are integrated into a graphical reconstruction framework in Unity, enabling qualitative validation through simulated hand visualization. The main contributions of this work include the formulation of a hierarchical kinematic model driven by real-world sensor data, the derivation of joint parameters through geometric transformations, and the demonstration of pose reconstruction using a visualization platform.

While the current study focuses on single-subject modeling and offline data, future research will expand the framework in several directions [19,20]. We plan to incorporate wearable motion-capture devices to obtain ground-truth joint measurements, which will allow for a quantitative evaluation of the model’s accuracy. Comparative studies with state-of-the-art hand-pose estimation techniques will also be conducted to benchmark performance under various conditions, including sensor noise and occlusion. Additionally, we aim to extend the system for real-time tracking and modeling across different users, capturing dynamic hand interactions more effectively. To enhance robustness, we intend to integrate model-based estimation with data-driven methods such as neural networks or Bayesian inference frameworks. Finally, we envision applications of the proposed model in rehabilitation monitoring, and teleoperation systems, where accurate and interpretable hand motion representation is crucial. These extensions will further strengthen the generalizability and practical relevance of the presented approach.