#### 2.1. Finger Model

Finger motion limitations can be described by a kinematic model based on anatomical data from the structure of the hand. All fingers except the thumb are divided into three phalanges, in order of distance from the base: proximal, middle, and distal, connected to each other with joints.

The interphalangeal joints each have one degree of freedom, and in simple cases, all finger movements are assumed to only include flexion and extension, and thus lie in the flexion plane, as shown in

Figure 5. As such, the finger can be modeled using a simplified kinematic model in the form of a flat three-link chain, corresponding to the three phalanges. Sources [

23,

24,

25,

26,

27,

28,

29] validated this model as a fine first approximation.

Let us consider our model as a system of three rigid links (which we will also refer to as phalanges), interconnected by uniaxial hinges. Their rotation axes coincide and they are orthogonal to the axes of the phalanges. In addition, one of the phalanges, with its free end, is attached through a similar hinge to the metacarpus. The position and orientation of the metacarpus are considered to be constantly known from optical tracking. As discussed above, the phalanges are always located in the same flection plane.

This model used several orthonormal coordinate systems: the base system tied to the metacarpus and local systems linked to each of the phalanges and sensors (

Figure 6). Their basic triple coordinates are denoted as

${e}_{c}=\{x,y,z\}$ for the metacarpus system and

${e}_{k}=\{{x}_{k},{y}_{k},{z}_{k}\}$ for the local systems. Here,

k is the phalange number, calculated from the attachment to the metacarpus. The beginning of the coordinate system of the

${k}^{\mathrm{th}}$ phalange is the

${k}^{\mathrm{th}}$ joint, and the beginning of the base coordinate system is the zero metacarpus joint.

The vectors x and ${x}_{i}$ are all aligned with each other and the axes of rotation of the hinges. Vectors ${y}_{i}$ are each directed along the axis of their corresponding phalange, and vectors z, ${z}_{i}$ complement the others to form an orthogonal right-handed coordinate system in ${\mathbb{R}}^{3}$.

The initial position of all coordinate systems was considered to be such that the base triples of all local systems coincided with the global base triple. For the ${i}^{\mathrm{th}}$ phalange, we can define the angle ${\theta}_{i}$ between the vectors y and ${y}_{i}$, where the positive direction of rotation is considered to be clockwise rotation around the x axis. The angles of rotation of the hinges are: ${\phi}_{i}={\theta}_{i}-{\theta}_{i-1}$ (here ${\theta}_{-1}$ is 0).

For each phalange, we define a vector ${\overrightarrow{r}}_{i}={l}_{i}{y}_{i}$, where ${l}_{i}$ is the length of the phalange.

We placed sensors on Phalanges 1 and 2. We assumed that the instrumental coordinate systems coincided with those of their corresponding phalanges, ${e}_{{i}_{k}}$, where sensor number k was placed on the ${i}_{k}^{\mathrm{th}}$ phalange, at a distance of ${p}_{k}$ from its proximal end. The position of the sensor relative to the proximal end of its phalange can be described by the radius vector ${\overrightarrow{h}}_{k}={p}_{k}{\overrightarrow{y}}_{{i}_{k}}={\alpha}_{k}{\overrightarrow{r}}_{{i}_{k}}$ (where ${\alpha}_{k}=\frac{{p}_{k}}{{l}_{{i}_{k}}}$).

The relative position of the ends of each phalange can be in turn described by the vector

${\overrightarrow{r}}_{i}={l}_{i}{\{0,cos{\theta}_{i},sin{\theta}_{i}\}}^{\top}={R}_{i}{\{0,{l}_{i},0\}}^{\top}$, where:

is the rotation matrix corresponding to a rotation by the angle of

${\theta}_{i}$ around the

x axis.

Through the summation of these radius vectors, the finger’s configuration was entirely determined by either of the triples of angles: ${\theta}_{0\dots 2}$ or ${\phi}_{0\dots 2}$. The problem of estimating the configuration of the finger was thus equivalent to the problem of estimating the set of angles ${\phi}_{0},{\phi}_{1},{\phi}_{2}$.

Let us consider the finger in motion, with the angles

${\theta}_{i}\left(t\right)$, angular speeds

${\omega}_{i}\left(t\right)={\dot{\theta}}_{i}\left(t\right)$, and angular accelerations

${\dot{\omega}}_{i}\left(t\right)$ known for all joints, as well as the acceleration of the zero joint relative to the global inertial coordinate system (

Figure 7).

Denote

${\overrightarrow{s}}_{k}$ as the radius vector of the

${k}^{\mathrm{th}}$ sensor in the base coordinate system of the model, then:

The acceleration of the sensor in an inertial coordinate system is represented as:

Here, $\dot{\overrightarrow{\omega}}=\dot{\omega}{\overrightarrow{x}}_{i}$ is a vector representation of the angular acceleration.

To calculate the modeled sensor’s readings, we needed to subtract the value just found from the acceleration due to gravity

$\overrightarrow{g}$ and also calculate the representation of the resulting vector in the instrumental coordinate system of the sensor. Hereon, we assumed that the sensors’ coordinate systems coincided with the local coordinate axes and that their sensitivity scale factor was calibrated. With this, the readings of the

${k}^{\mathrm{th}}$ sensor equate to:

#### 2.2. Simple Switching Tracking Algorithm

Tracking the human hand movements has important specifics with regards to the object being tracked. The wide range of possible hand movements significantly complicates the task. Goal-directed hand movements are similar in structure to eye movements [

30]. Therefore, hand movement tracking can apply the oculography motion detection approach [

31]. In the following paragraphs, we will formulate a criterion for switching between several types of tracking, similar to how it is done in eye tracking tasks.

Let us assume that the exact location and orientation of the metacarpus at each moment in time are known from the optical tracking data. For most simple grasping movements, we can take as the first approximation that the angle in the distal joint always equals the angle in the proximal joint [

32].

#### 2.2.1. Algorithm of The Position Estimation of The Single Phalange

Let us divide all finger motions into two distinct classes of “slow” and “fast” motion, which we define as follows:

Slow motion is the movement of the finger during which the acceleration of all of its elements relative to the wrist is negligible compared to the acceleration due to gravity. All other movement is classified as fast motion.

Fast motions are characterized by large angular velocities, phalange accelerations, and a rather short duration because of the fact that the extent of finger movement is limited. Slow motions, on the other hand, have a significantly longer total duration than fast ones and are are typically represented by maintaining a fixed finger configuration.

This leads to the idea of using two different estimation algorithms for different motion classes and switching between them when moving from one class to another.

#### 2.2.2. Slow Motion Estimation

We can consider a slow motion using the kinematic model described above.

Let us define the local acceleration of a sensor as:

From Equations (

4) and (

3), we get:

Since we know the movement of the wrist, we also know the representation $f=\{{f}_{1},{f}_{2},{f}_{3}\}$ of the vector ${\overrightarrow{g}}_{c}={\overrightarrow{a}}_{0}-\overrightarrow{g}$ in the base coordinate system of the model ${e}_{c}$.

By definition, ${f}_{j}=\langle {e}_{c\phantom{\rule{3.33333pt}{0ex}}j},\phantom{\rule{3.33333pt}{0ex}}{\overrightarrow{g}}_{c}\rangle $, and ${\overrightarrow{g}}_{c}={\displaystyle \sum _{j=1}^{3}}{f}_{j}{\overrightarrow{e}}_{k}$.

We similarly define the representation ${f}^{\prime}$ of the acceleration of a sensor in the basis ${e}^{\prime}={\{{\overrightarrow{{e}^{\prime}}}_{1},{\overrightarrow{{e}^{\prime}}}_{2},{\overrightarrow{{e}^{\prime}}}_{3}\}}^{\top}$ of the instrumental coordinate system of the sensor: ${f}_{j}^{\prime}=\langle {{e}^{\prime}}_{j},\phantom{\rule{3.33333pt}{0ex}}{\overrightarrow{g}}_{c}\rangle $

As such,

${\theta}_{i}$ can be estimated via the function

$\mathrm{atan}2({f}_{3}^{\prime},{f}_{2}^{\prime})$, defined as follows:

#### 2.2.3. Fast Movement Estimation

Let us consider now a fast motion. According to our definition, it is a transition between periods of slow motion, where the time of this movement is much less than one second. If we assume that at the beginning of the motion, the finger configuration is known to the system with acceptable accuracy, then we can calculate the orientation of the phalange by integrating the measured angular velocities based on the known initial position:

where

$\mathtt{int}\left(i\right)$ is a function performing a step of a numerical integration algorithm.

The accumulated error due to integration can later be corrected during the following phase of slow motion.

#### 2.2.4. Errors in The Estimation Algorithms for Slow and Fast Motion

Let us define the errors:

We will now estimate the deviation magnitude of the slow movement estimation algorithm. Suppose that angular velocities do not exceed

${\omega}_{max}$, angular accelerations do not exceed

${\dot{\omega}}_{max}$, and phalanges are no longer than

${l}_{max}$. Therefore, the magnitude of the accelerations is limited by:

Thus, the value

$\overrightarrow{\mu}$ from (

5) corresponds to:

Now, consider how the deviation of orientation estimation

$\Delta {\theta}_{g}$ depends on

$\overrightarrow{g}$ and

$\overrightarrow{\mu}$. Since a flat model is used, the algorithm only considers the components of accelerations within the flexion plane

$yz$. Taking into account that for small angles

$\mathrm{tan}\left(x\right)\approx x$ and that the maximum estimation error is attained when

$\overrightarrow{\mu}\perp \overrightarrow{g}$, we can estimate the error magnitude for phalange

k using (6):

From (7), it follows that for identical finger movements, the accuracy of the estimate from observing the vector $\overrightarrow{g}$ diminishes as the magnitude of the in-plane component of $\overrightarrow{g}$ decreases. Ultimately, if the flexion plane is horizontal, the estimation error can become arbitrarily large, and the estimate carries no actual information. In this case, the only available way to determine orientation is through integration of AVS readings.

The integration error, on the other hand, grows linearly with time: $|\Delta {\theta}_{\omega}\left(t\right)|\approx |\Delta \omega |t$, where $|\Delta \omega |$ characterizes the average AVS deviation.

#### 2.2.5. Switching Algorithm

For the proposed algorithm to perform optimally, the switching criterion has to minimize the general deviation of the overall estimate of the phalange’s orientation.

Consider some possible arbitrary movements. The deviation of the slow motion estimation algorithm is then a time dependent function:

In turn, the total error during fast motion estimation has the form:

where

${t}_{\mathrm{s}}$ is the time of the last switch to integration. In the worst case, error accumulation is going in the same direction as the previous deviation of the slow motion algorithm, giving us an upper bound:

We divide time into discrete intervals

${t}_{1}\dots {t}_{n}$ and introduce an estimate of the error at the

ith time moment:

The estimates ${\tilde{\Delta \theta}}_{g}\left(t\right)$ and ${\tilde{\Delta \theta}}_{\omega}\left(t\right)$ are defined according to (7) and (8). For each time interval, we choose the algorithm that minimizes the estimate of the total error $\tilde{\Delta \theta}$. From (8), it follows that for each moment, an integration step would yield an error of no more than $\Delta \theta \left({t}_{k+1}\right){|}_{\omega}=\Delta \theta \left({t}_{k}\right)+({t}_{k+1}-{t}_{k})\xb7|\Delta \omega |$, independent of ${t}_{\mathrm{s}}$. This value can be directly compared with $\Delta {\theta}_{g}\left({t}_{i}\right)$.

Thus, the final estimation algorithm is as follows:

- 1.
receive sensor readings $\widehat{a},\widehat{\omega}$;

- 2.
calculate ${\Delta \theta}_{g}\left({t}_{i}\right)=\frac{|\tilde{\mu}|\left(\widehat{\omega}\right)}{|{\overrightarrow{g}}_{xy}|}$ and $\Delta \theta \left({t}_{i}\right){|}_{\omega}=\Delta \theta \left({t}_{i-1}\right)+({t}_{i}-{t}_{i-1})\xb7|\Delta \omega |$;

- 3.
choose the estimation mode:

if currently in fast motion mode and ${\Delta \theta}_{g}\left({t}_{i}\right)<\Delta \theta \left({t}_{i}\right){|}_{\omega}$, then switch to slow motion mode, and assign $\Delta \theta \left({t}_{i}\right):={\Delta \theta}_{g}\left({t}_{i}\right)$;

if currently in slow motion mode and $\Delta \theta \left({t}_{i}\right){|}_{\omega}<{\Delta \theta}_{g}\left({t}_{i}\right)$, then switch to fast motion mode, taking $\tilde{\theta}\left({t}_{i-1}\right)$ as the initial estimate, and assign $\Delta \theta \left({t}_{i}\right):=\Delta \theta \left({t}_{i}\right){|}_{\omega}$.

- 4.
Calculate a new orientation estimate $\tilde{\theta}$ using the currently selected estimation algorithm.

#### 2.3. Madgwick Filter Modification

The Madgwick filter algorithm suggested in [

20] is used for the restoration of body orientation according to the readings of microelectromechanical sensors. Usually, the filter is presented in two modifications. The first modification can be used for INS (Inertial Navigation Systems), which consist of only a three-axis accelerometer and AVS. The second is applied to INS that also contain a three-axis magnetometer. A three-axis magnetometer measures the Earth’s magnetic field vector together with local magnetic distortions. This complicates its use in rooms with VR equipment, metal structures, and other objects that cause large distortions in the magnetic field.

We propose a modified Madgwick filter, taking into account the features of the kinematic model of a finger. Instead of the magnetic field induction vector, we can take the normal axis of the flexion plane as the second correction vector. This modification will always work correctly except in the case of the co-directionality of the correction vectors, which, due to its rarity, we can neglect.

#### 2.3.1. Finger Rotation Estimation

Let us describe our proposed modified Madgwick filter. Hereafter, we will use the quaternion apparatus to represent rotations.

In this section, the sign ⊗ shall denote the operation of quaternion multiplication. A tilde over a variable $\tilde{\xb7}$ denotes the estimate of the corresponding quantity, and a circumflex $\widehat{\xb7}$ denotes its measured value. The subscript before a variable indicates the target coordinate system. A superscript indicates the coordinate system with respect to which the variable is specified. E and ${S}_{k}$ denote, respectively, the global coordinate system tied to the Earth and the instrumental coordinate system of sensor k.

In particular, we introduce the quaternion ${}_{{S}_{k}}^{E}\tilde{q}$ to describe the estimate of the sensor’s orientation relative to the Earth, and the vectors ${}^{{S}_{k}}\widehat{\overrightarrow{f}}$ to denote acceleration and angular velocity measurements in the sensor’s coordinate system.

From:

and having the readings of the sensors and the previous orientation estimate

${}_{E}^{{S}_{k}}{\tilde{q}}_{t-1}$ (which is initially taken from the optical tracking data), we can get an estimate

${}_{E}^{{S}_{k}}{\tilde{q}}_{\omega ,t}$ of the sensor’s orientation relative to the ground:

When constructing the orientation filter, it is assumed that the accelerometer will measure only acceleration due to gravity, and we know the plane of motion from the readings of the optical system on the metacarpus.

Let us calculate another estimate by solving a problem of numerical optimization for the desired quaternion, in which as the initial approximation, we take a previous estimate

${}_{E}^{{S}_{k}}{\tilde{q}}_{t-1}$, and as the cost function, we take the measure of the accuracy of vector alignment achieved by the desired rotation:

where

J is the cost function,

${}^{{S}_{k}}\widehat{\overrightarrow{f}}$ are the accelerometer measurements in the coordinate system of the sensor,

${}^{E}\overrightarrow{g}$ is a known gravity vector in the global coordinate system, and

${}^{E}{\overrightarrow{g}}_{k}$ is a vector obtained from (2) using current sensor data and past orientation estimates

${}_{E}^{{S}_{k}}{\tilde{q}}_{t-1}$ in the global coordinate system.

The problem is solved by the gradient descent method. The only possible solution is chosen, taking into account the normal to the flexion plane, known from the optical tracking data. The estimate

${}_{E}^{{S}_{k}}{\tilde{q}}_{\nabla ,t}$ of the sensor orientation relative to the ground is obtained:

The optimal value

${\mu}_{t}$ depends on the rotation speed and can be calculated based on the readings of the angular velocity sensors [

20]:

#### 2.3.2. Combining Filter Algorithm

Obtaining estimates

${}_{E}^{{S}_{k}}{\tilde{q}}_{\omega ,t}$ from the angular velocity and

${}_{E}^{{S}_{k}}{\tilde{q}}_{\nabla ,t}$ from the observations of known vectors, we can determine the joint estimate as a linear combination with weights

${\gamma}_{t}$,

$(1-{\gamma}_{t})$, similar to the classic Madgwick filter:

Given that both estimates

${}_{E}^{{S}_{k}}{\tilde{q}}_{\omega ,t}$ and

${}_{E}^{{S}_{k}}{\tilde{q}}_{\nabla ,t}$ are obtained from the previous general estimate

${}_{E}^{{S}_{k}}{\tilde{q}}_{t-1}$ by adding small terms to it

${}_{E}^{{S}_{k}}{\dot{q}}_{\omega ,t}\Delta t$ and

$-{\alpha \left|\right|}_{E}^{{S}_{k}}{\dot{q}}_{\omega ,t}\left|\right|\Delta t\frac{\nabla J}{\left|\right|\nabla J\left|\right|}$, we get:

According to the article [

20], the parameter

${\gamma}_{t}$ can be considered small, and by replacing

${\gamma}_{t}=\frac{\beta \Delta t}{{\mu}_{t}}$, where

$\beta $ is a small number, we can simplify the expression (16):

Additionally, since the described operations do not guarantee the preservation of the unit norm of the quaternion, the resulting estimate must be normalized:

The expressions (15), (18), and (19) define the final form of the filter.

It is possible to use a non-constant value for parameter $\beta $, changing it depending on the current motion, decreasing by large values with spurious accelerations $\mu $. This can further improve filter accuracy by reducing the impact of accelerometer errors on estimation during fast movements.