An Interactive Digital-Twin Model for Virtual Reality Environments to Train in the Use of a Sensorized Upper-Limb Prosthesis

Abstract

In recent years, the boost in the development of hardware and software resources for building virtual reality environments has fuelled the development of tools to support training in different disciplines. The purpose of this work is to discuss a complete methodology and the supporting algorithms to develop a virtual reality environment to train the use of a sensorized upper-limb prosthesis targeted at amputees. The environment is based on the definition of a digital twin of a virtual prosthesis, able to communicate with the sensors worn by the user and reproduce its dynamic behaviour and the interaction with virtual objects. Several training tasks are developed according to standards, including the Southampton Hand Assessment Procedure, and the usability of the entire system is evaluated, too.

1. Introduction

The human hand is a masterpiece of working capabilities, and so researchers have been struggling to replace the lost functionalities of arm (and/or forearm) amputees by means of more and more sophisticated sensorized prostheses [1,2,3]. Allowing even a partial recovery of full functionalities can significantly increase the quality of amputees’ lives. Accordingly, current upper-limb prostheses, particularly those including hand segments, can be articulated by interpreting myoelectric signals gathered on the amputated limb [4,5,6,7], allowing some form of grasping and manipulation.

However, in general, the higher the degree of functionality of the prosthesis, the greater the training time and efforts required for the amputee to be confident with the electromechanical arm [8,9]. In particular, three main aspects can be underlined: the user’s need to become familiar with an “unnatural appendix”, also from a psychological point of view; the dull and annoying training time needed from the user; and the correct recognition of the user’s intentions to correctly drive the prosthetic hand. Accordingly, a key rule can be played by preventive training activities to be carried out in safe environments, which virtual environments can act as [10,11,12,13], allowing the identification of possible functional customisations before the actual physical prosthesis is completed and mounted onto the amputated limb.

Within this frame and scientific background, the use of virtual reality and the building of digital twins can be very helpful tools for improving training, even for the use of complex and advanced prostheses. On the other hand, as far as we know, examples of their full integration are still missing. For this reason, we propose a new, ad hoc-developed, interactive training methodology for amputees to become confident with a sensorized upper-limb prosthesis, based on the advantages offered by virtual reality environments and simulation numerical models, by combining computer graphics, multibody simulation, computer vision, sensors, and virtual reality.

The main goal is to integrate different computer-aided methodologies to build a comprehensive digital twin to be integrated into a virtual reality environment. This environment will contain a series of standardized exercises aimed at reducing the training time required to use a sensorized prosthesis. Although the procedure is developed and tested specifically for the iHannes prosthesis, made by the INAIL Italian Centre for Prosthetics and the Italian Institute of Technology (IIT) [14,15], the proposed algorithms are generally applicable to any type of sensorized upper-limb prosthesis.

The main novelty of this investigation is the integration of different methodologies into a self-contained approach and an immersive virtual environment (a fully functional digital twin). The combined use of advanced simulation models; a sequential impulse solver; and digital twins with real-time gesture classification, object interaction, realistic deformation of the skin, and quantitative virtual training exercises is suitable for a comprehensive virtual training environment. We believe that, apart from this specific implementation, this approach can be used for other prosthesis training procedures.

This work can be split into three main parts: the sensorized upper-limb prosthesis with myoelectric inputs; the digital replica of the complex system (namely, the digital twin) of the prosthesis, focusing on the algorithms, numerical models, and implementation procedures; and the training tests and usability concerns. The specific background and state of the art of each integrated methodology are included in Section 3 and split into each digital-twin component.

2. The Sensorized Prosthesis

In general terms, an upper-limb prosthesis, with a built-in socket acting as an interface between the amputated anatomical segment and an anthropomorphic structure, aims to replicate as many natural amputated limb functionalities as possible [16,17]. Sensorized prostheses can include (epidermal or implanted) sensors to collect the myoelectric signals of the amputated limb, in combination with actuators to morph different configurations of the anthropomorphic hand [18]. The gathered myoelectric signals are processed by classification algorithms to identify which posture the subject’s neuromuscular system is expressing. Different strategies can be applied for training the classification system, including machine-learning approaches [19], to recognize the intentional gesture, such as the index and thumb grip, the three-digital grip, and the fist. Downstream of the classifier, the resulting signal is to actuate the prosthesis, accordingly, as previously pointed out in other works [20].

3. Digital Twin and Physical Twin Integration

The core of the proposed methodology is based on realizing a digital twin (DT) of the sensorized upper-limb prosthesis, namely its digital replica [21], which differs from a virtual prototype in three main aspects:

• The DT communicates (preferably two-way) with its physical counterpart, namely the physical twin (PT) (exchanges information with real sensors);
• The DT increases the information of its PT through mathematical models, providing real-time augmented information;
• The DT is conceived to flank the PT throughout its life cycle.

DTs can easily interface with virtual and/or augmented reality environments, with immersive video technologies to facilitate (even immersive and highly realistic) communication with the user [22]. DTs have been adopted in different fields, from maintenance to product and process design, and, in recent years, researchers have been considering their adoption for training purposes [23], particularly in medicine [24,25,26]. Within this frame, we are focused on improving sensorized prosthesis dexterity to assess the strategic, but often disregarded, aspect of their long-term adoptability [8,9], moreover allowing customization and optimization for classifiers, electronics, and mechanical parts of the ensemble, before “wearing” the actual device. The virtual reality approach immerses the user in an environment where the prosthesis is highlighted, allows verification that the interfaced electronics correctly interpret his/her intent, and allows a series of manipulation and dexterity exercises to be performed as many times as is desired by the user.

Figure 1 shows a scheme of the DT-PT three communication paths (COM paths).

The first path delivers real-time information related to the arm positions in space with respect to the user’s head, in order to correctly collimate the virtual prosthesis on the upper limb. The second path regards the myoelectric sensors’ communication for the assessment of the hand grasping poses. The directions of both the first and second communication paths are from the physical to the digital twin, whilst the third path goes in the opposite direction to deliver visual feedback to the user. A data processing unit is in the middle of the three paths with the aim to synchronize information, ensure the real-time rendering and realism of the scene and, therefore, achieve training success.

To this aim, from the hardware side, we adopted the VIVE PRO virtual reality system (by HTC Corporation, New Taipei City, Taiwan) consisting of a head-mounted display, fixed infrared lighthouses, and crown-shaped trackers. On the helmet and on the trackers, there is a series of photodiodes that are activated by the lighthouses, and differences in illumination allow for determining their spatial position and attitude. From the software side, we developed the digital twin and the virtual scene using the UNITY platform (by Unity Technologies, San Francisco, CA, USA).

During training, the user is asked to wear the helmet and the two trackers using armbands (Figure 2). One tracker is firmly fixed on the arm, and the other on the remaining part of the forearm. Moreover, the user is asked to put on the myoelectric sensors (pads) and connect the whole system to the data processing workstation. The DT gathers information from the sensors of the PT and updates the immersive virtual scene using a specific procedure and computational algorithm (described in the next subsection). In the virtual environment, the virtual arm representing the sensorized prosthesis is placed according to the physical sensors. The virtual scenario is completed by other virtual objects classified as interactable entities that can come in contact and interact with the virtual hand. A set of instructions is also present in the virtual scene to guide the user to complete the training tasks.

3.1. Virtual Prosthesis Dynamic Model and Trackers Interface

The role of the dynamic model is to accurately reproduce the motion dynamics of the virtual prosthesis, starting from the data of the trackers and the epidermal myoelectric sensors. The whole model can be broken down into two functional subassemblies. The first aims to replicate the dynamics of the arm and forearm on which the prosthesis is attached through an inverse dynamics model starting from the information acquired from the trackers. The second subassembly is needed to manage the movements of the grasping fingers (Figure 3).

The first subassembly consists of four rigid bodies. The first two, the main ones, are representative of the arm and forearm. They are connected by a spherical joint that replicates the functionality of the elbow in the distal portion of the arm. Although, from an anatomical point of view, the elbow joint does not have a simple spherical joint behaviour but a more complex elasto-kinematic characteristic [27], this assumption simplifies the complexity of the constraining equations without affecting the model’s reliability. This approach can easily be adapted to the presence of the trackers for controlling the relative motion between body segments.

The first subassembly then includes two other dummy (fictitious) bodies that are necessary to replicate the presence of trackers and efficiently transfer the information coming from them. These dummy bodies are not connected to the rest of the system by kinematic pairs but only by two elastic bushing elements. This solution has been chosen to obtain a reliable and computationally efficient model. This choice is often used as a smart compromise in overabundant systems [28]. In fact, the direct transfer of information from the trackers to the respective arm and forearm bodies (for example, by means of constraint equations that impose the equality of positions and attitudes) leads to an overabundant system of equations (too many constraining equations with respect to the degrees of freedom). This produces complications in the integration phase, and the solution of the dynamic system may fail or be extremely slow.

Considering that a body in a three-dimensional space has 6 degrees of freedom, the number of degrees of freedom of a system composed of two rigid bodies connected by a spherical joint is equal to 9, while the constraint equations from the trackers are 2 × 6 = 12. This leads to an overabundant system (12 > 9). Using bushing elements instead of kinematic constraints does not alter the number of the system’s number of degrees of freedom, which remains 9. From a mathematical point of view, a bushing element between two reference systems belonging to two different bodies can be written as

$$\left\{F\right\}=\left[K\right]\left\{\Delta \right\}$$

(1)

where $\left\{F\right\}$ is the vector of the generalised forces, $\left[K\right]$ is a 6 × 6 stiffness matrix, and $\left\{\Delta \right\}$ is the vector of the generalised (linear and angular) displacements between the two reference frames. In other words, a bushing element behaves as a multidimensional spring force. By imposing a sufficiently high value on all the elements of the $\left[K\right]$ matrix, the bushing element behaves in a similar way to a fixed constraint, ensuring the correct connection between the bodies. However, the intrinsic penalty approach of the bushing element allows the tolerance of slight measurement errors, tracker movements on anatomical segments, skin/cloth artefacts, and slight calibration errors without a significant impact on the solution of the equations.

The second subassembly is composed of 20 rigid bodies that schematise the various phalanges of the fingers (four bodies for each of the fingers). In this case, we connected the various segments using spherical joints. The second subassembly is a group of rigid bodies that, despite having an anatomical conformation, are not present in reality due to the presence of the amputated limb. Therefore, in this case, it is not included in the inverse dynamics problem. It is only used to simulate the movement of the prosthesis fingers and their interactions with objects in grasping and manipulation exercises. The movement of these rigid bodies is not controlled by trackers or other physical devices but is imposed by the interpretation of the myoelectric signals using a neural network classifier.

From a practical point of view, for each of the classifiable gestures, a series of time laws of the relative angles between the adjacent anatomical segments of the fingers is provided. They are imposed by means of driving constraints when the classifier recognises a pose with a sufficient degree of confidence. The speed at which these movements take place is calibrated according to the performance of the actual prosthesis in order to increase the sense of realism and proper delay. If the classifier does not return any posture update, the second subassembly actually behaves as a single rigid body because all the relative driving constraints enforce a stationary motion (zero relative velocities).

The two subassemblies are then connected to each other by a fixed constraint between the distal end of the forearm and the proximal end of the palm. Overall, the whole dynamic virtual arm system has

• 24 rigid bodies (4 in the first subassembly and 20 in the second subassembly);
• 17 kinematic constraints with 54 scalar equations (1 spherical joint between the arm and the forearm, 1 fixed constraint between the forearm and the palm, and 15 spherical joints between adjacent phalanges of the hand);
• 57 motion constraints (6 scalar equations for each of the trackers and 3 scalar equations to control the relative rotation for each of the spherical constraints between adjacent phalanges);
• 2 bushing elements (one for each of the trackers).

The dynamic model is built and solved according to the rigid multibody dynamics method, not using the global formulation [29] but adapting the sequential impulse formulation [30,31]. Studies in the literature have shown that this formulation is particularly suitable for solving dynamics problems in real-time and with interactivity requirements such as virtual and augmented reality environments [32]. The sequential impulse formulation is based on a two-step algorithm:

• At the beginning, the equations of motion are solved by neglecting the kinematic constraints (free-body equations, considering only the external and internal forces applied to the bodies);
• Subsequently, a series of impulses is applied to all the bodies, one at a time, sequentially and iteratively, to update their motion kinematics, fulfilling the constraint equations within a specific tolerance.

Considering a dynamic system where $\left[M\right]$ is the mass matrix, $\left\{q\right\}$ is the vector of the generalised coordinates, $\left\{\psi \right\}$ is the vector of the constraint equations, and $\left\{F_e\right\}$ is the vector of the external forces and torques, in the first step of the formulation, the equations are solved neglecting the constraints:

$$\left[M\right]{\left\{\ddot{q}\right\}}_{approx}=\left\{F_e\right\}$$

(2)

where ${\left\{\ddot{q}\right\}}_{approx}$ is the vector of the approximated accelerations.

Approximated velocities and positions can be calculated as linear approximations from accelerations:

$$\begin{array}{l}{\left\{\dot{q}\right\}}_{approx}=h\cdot {\left\{\ddot{q}\right\}}_{approx}\\ {\left\{q\right\}}_{approx}=h\cdot {\left\{\dot{q}\right\}}_{approx}\end{array}$$

(3)

where $h$ is the integration time step, assumed as fixed.

Corrective impulses ${\left\{P\right\}}_{constraint}$ generate variations according to Newton’s law of impulses:

$$\left[M\right]\left({\left\{\dot{q}\right\}}_{corrected}-{\left\{\dot{q}\right\}}_{approx}\right)={\left\{P\right\}}_{constraint}$$

(4)

where ${\left\{\dot{q}\right\}}_{corrected}$ is the vector of the corrected velocities that can be computed from Equation (4) as:

$${\left\{\dot{q}\right\}}_{corrected}={\left\{\dot{q}\right\}}_{approx}+{\left[M\right]}^{-1}{\left\{P\right\}}_{constraint}$$

(5)

Considering that impulses are exerted to satisfy constraints, it will result in the following:

$${\left\{P\right\}}_{constraint}={\left[{\psi }_q\right]}^T\left\{\delta \right\}$$

(6)

where $\left[{\psi }_q\right]$ is the Jacobian matrix of the constraint equations and $\left\{\delta \right\}$ is the vector of the Lagrange multipliers associated with the impulses.

Considering the time derivatives of the constraint equations, we obtain:

$$\begin{array}{l}\left\{\psi \right\}=\left\{0\right\}\Rightarrow \frac{\mathrm{d}\left\{\psi \right\}}{\mathrm{d}t}=\left[{\psi }_q\right]\left\{\dot{q}\right\}+\left\{{\psi }_t\right\}=\left\{0\right\}\\ \left[{\psi }_q\right]\left\{{\dot{q}}_{corrected}\right\}+\left\{{\psi }_t\right\}=\left\{0\right\}\end{array}$$

(7)

Substituting the corrected accelerations in Equation (7):

$$\left[{\psi }_q\right]\left(\left\{{\dot{q}}_{approx}\right\}+{\left[M\right]}^{-1}{\left[{\psi }_q\right]}^T\left\{\delta \right\}\right)+\left\{{\psi }_t\right\}=\left\{0\right\}$$

(8)

Solving for Lagrange multipliers of the impulses:

$$\left\{\delta \right\}={\left(\left[{\psi }_q\right]{\left[M\right]}^{-1}{\left[{\psi }_q\right]}^T\right)}^{-1}\left(\left[{\psi }_q\right]{\left\{\dot{q}\right\}}_{approx}+\left\{{\psi }_t\right\}\right)$$

(9)

and consequently, we find the impulses to be applied:

$${\left\{P\right\}}_{constraint}={\left[{\psi }_q\right]}^T\left\{\delta \right\}$$

(10)

Correcting the accelerations and fulfilling the constraints requires a few iterations by repeating the algorithm.

The sequential impulse formulation is also particularly suitable for managing contacts, which are necessary to implement the general forms of interaction with virtual objects in the immersive scene. In particular, the collisions may be treated as penalty constraints. The generic not-overlapping condition between two bodies, $C_n$ can be written as:

$$C_n=\left(P_A-P_B\right)\cdot n_A\ge 0$$

(11)

where $P_A$ and $P_B$ are the closest points of two bodies that can come in contact and $n_A$ is the normal unit vector of the body shape at $P_A$.

Equation (11) can be written at the velocity level as:

$$\frac{d{C}_n}{dt}={\dot{C}}_n=\frac{d\left(P_A-P_B\right)}{dt}\cdot n_A+\left(P_A-P_B\right)\cdot \frac{d{n}_A}{dt}\ge 0$$

(12)

The Baumgarte scheme [33] can be used to push the bodies apart when they overlap. The velocity constraint is augmented with a feedback term proportional to the penetration depth:

$${\dot{C}}_n+\beta C_n=0$$

(13)

where $\beta$ is a tunable scalar parameter which governs the speed of the penetration resolution. Equation (13) is then appended to the vector $\left\{\psi \right\}$ and treated as a generic constraint.

Therefore, the complete procedure from the acquisition of the signals to the solution of the digital twin model to the integration with the virtual reality environment can be summarised in Figure 4.

In summary, the proposed procedure starts with the acquisition of the signals from the trackers and the myoelectric pads. From the former, the information about the position and attitude of the trackers is extracted, and from the latter, the electrical potential differences are extracted and then transferred to the classifier. The classifier interprets the patterns and returns the most likely grasping posture. With this posture, the driving constraints on the spherical joints are updated to move the phalanges accordingly. The data of the position and the attitude of the trackers and of the driving constraints on the phalanges are then transferred to the sequential impulse solver that integrates the equations. Once resolved, the updated positions and attitudes of all rigid bodies of the model are known.

The arm/hand mesh is congruently deformed starting from the position of the rigid bodies that work as a skeleton in the skinning mesh approach, a common strategy in computer graphics [34,35] and which is discussed in the next subsection of the paper. The updated arm model, as well as the position of all the other virtual bodies in the training scene, are then updated in the virtual reality environment. Knowing the position of the polygons in the arm mesh, it is possible to calculate the interpenetrations between the human virtual arm and all the other virtual objects in the scene, which are then returned to the solver for the next integration step. The entire algorithm must be completed within a single visualisation frame to ensure synchronism between the real scene and the virtual model, in order to have the DT properly collimated with the PT.

3.2. Arm and Hand Mesh Skinning

Mesh skinning is a technique used in computer graphics to animate and deform a 3D model [36]. The basic idea is to use a skeleton, or a hierarchical set of interconnected bones, to represent the underlying structure of a character. The actual 3D model (mesh) is then deformed based on the movement of these bones, allowing for realistic and dynamic character animations. When a bone moves, the mesh nodes are more or less attracted depending on the applied weight. If a bone has a weight 1 on a mesh node, it means that the node rigidly moves with the bone (it has the same spatial transformation). If a bone has a weight 0 on a mesh node, it means that the node is not influenced by the movement of the bone. A weight between 0 and 1 means that the actual movement of the node is blended between the movements of two or more bones. By using mesh skinning, the deformation of the mesh can be smooth and very realistic.

In the DT, the arm, forearm, and hand multibody model work as the skeleton of bones, and the anatomical shape works as the deformable mesh (Figure 5). The entire virtual deformable arm is a mesh with 4446 faces and 2506 nodes. This choice is a good compromise between accuracy and computational effort. Each bone has a weight on the nodes of the mesh, and some examples of these weights are reported in Figure 6. It can be seen, as a general strategy, that the weight of the nodes between two joints is 1 and then decreases in the areas around the joints themselves where the actual deformations occur.

3.3. Implementation of Stable Grasping Actions

Although the contact interaction among all the virtual objects in the environment is always present to mimic the sense of realism, the grasping of the objects must be enforced using a more stable connection. In fact, rigid bodies whose kinematics is driven by external sources (such as the arm and the forearm) may produce excessive contact impulses that make it difficult to reach a stable grasping due to the rapid motion increments. For this reason, the grasping condition is enhanced using the grasping active feature/object active feature GAF/OAF methodology [37]. According to this methodology, a series of reference systems is assigned to the hand (GAF) and the bodies (OAF) that can be grasped (Figure 7). These reference systems work as communicators. The main steps of the methodology are:

• when the user expresses the intention of grasping using a specific pose, the corresponding GAF is activated;
• the relative position and attitude between the activated GAF and all the OAFs on the different objects in the scene are checked;
• if the check produces a positive match (GAF and OAF are close and aligned), the grasping is confirmed, a constraint condition between the two reference systems is enforced, and the vector $\left\{\psi \right\}$ is updated accordingly;
• if the check produces a negative match (GAF and OAF are far and/or misaligned), the grasping is cancelled and the integration goes on without modifying the equations.

Further specific details of the methodology can be found in the seminal paper [37].

4. Training Tasks

The training of the prosthesis is realized by:

• Updating the meshed virtual arm driven by the user movements, according to the classification outputs from myoelectric signal data, as for the actual prosthesis;
• Implementing a series of grasping and manipulation exercises to improve the dexterity, according to the Southampton Hand Assessment Procedure (SHAP) [38], with the implementation of a virtual replica of the operations (named the VR-SHAP test).

The training tasks are based on the recognition of four main hand gestures, which are the power grip (grasp an object with all fingers), the three-finger grip (grasp with thumb, index and middle finger only), the pointing finger, and the flat hand, as shown in Figure 8.

In particular, the SHAP is useful for evaluating both impaired natural hands and prostheses. It involves a timed assessment consisting of 12 abstract object tasks and 14 daily living activities. The functionality profiles, derived from the timed tasks, work as metrics for assessing a subject’s (dis)ability, by providing an objective appraisal of the subject’s functional level, with the advantage of quantifying impairments in a single test rather than using relative improvement measures. Moreover, the SHAP can assess the effectiveness of a prosthetic end-effector and its controller by focusing on the user’s performance and can allow for tracking patients’ rehabilitation progress. This approach helps to highlight functional differences between several prosthetic devices and appropriate control schemes, while providing a contextual measure and comparison of hand function.

On the basis of the aforementioned four main hand poses (recognized by the classifier), we implemented six compatible tasks naming the procedure the Virtual Reality Southampton Hand Assessment Procedure (VR-SHAP). The procedures in VR-SHAP (Figure 9) are divided into three difficulty levels, as discussed in the following.

In the first level of difficulty, the user is asked to grasp an object, such as a bottle, a ball, or a plate, and place it on a base overcoming an obstacle (a barrier). The bottle and the ball are grasped through the power grip (Figure 10), the plate through the three-finger grip.

The second level of difficulty involves the rotation of a handle and the movement of a cylindrical slider along a guide up to a final target. The task involves the elbow rotation and the power grip pose. For the handle (Figure 11), the rotation is considered accomplished when the angle between the handle rotation axis and a reference direction is less than 5 degrees.

The third level of difficulty consists of inserting small objects (buttons) into a box (Figure 12). This task is classified with a higher level of difficulty because, due to the reduced size of the objects to be manipulated, greater precision and skill are required by the user.

The start and finish of all the tasks are managed by pressing a start or stop button using the pointing finger pose, and the execution time is used to evaluate the performance. The hand/objects grasping is implemented by means of the aforementioned OAF/GAF methodology for a stable grip, whilst the simple interaction is based on 3D contacts detected by means of mesh colliders adopting the corresponding equations (Equation (13)) of the sequential impulse solver.

The adopted VR-SHAP assessment is scored according to the index of functionality (IOF) algorithm [38], by considering the square sum of all the scores of the performed tasks and, ranging from zero (poor performance) to 100 (excellent performance). According to this approach, the overall score $d$ of the tests can be assessed with the formula:

$$d=\sqrt{{\displaystyle \sum _{i=1}^k{z}_i^2}}$$

(14)

where $z_i$ is the score of the $i$-th task and $k$ is the number of executed tasks. The score of each task is computed using:

$$z_i=\frac{x_i-{\tilde{x}}_i}{s_i}$$

(15)

where $x_i$ is the time to complete the $i$-th task, ${\tilde{x}}_i$ is the average time to complete the $i$-th task of the reference sample, and $s_i$ is the standard deviation of the time measured for the $i$-th task of the reference sample;

The average and standard deviation of times, for evaluating Equations (14) and (15), are reported in the next section, together with the results on usability in

Table 1. Average and standard deviation time for the six tasks of VR-SHAP.

Task	Average Time [s]	Standard Deviation [s]	Level of Difficulty
Bottle	8.4	2.4	Level 1
Ball	7.2	3.2	Level 1
Plate	6.8	2.4	Level 1
Handle	6.0	4.0	Level 2
Slider	19.2	4.0	Level 2
Buttons	17.6	4.0	Level 3

.

5. Usability Assessment Results

At the beginning of the test, the users were trained in the overall procedure (how to wear the trackers, the myoelectric sensors, and the head-mounted display, and how to manage the virtual reality environment), both by means of oral instructions and graphical guidelines. Then, the users were asked to repeat the overall procedure five times, so as to gain in familiarity with the system.

A preliminary usability analysis of the developed virtual environment was performed by means of a questionnaire collecting the degree of impact and the difficulties in understanding the guidelines, setup procedure, and scene interaction (also in terms of real-time aspects, excluding the classification performances), statistically executed on thirty-six users, evenly gender distributed, and 20–50 years of age (mean value 30 years, standard deviation 8.5 years). These users were asked to assign a score of 0 to 10 (from very bad/difficult to very good/easy condition). According to the results reported in Figure 13, the usability analysis reports a relatively high level of scores, testifying to an adequate implementation of the DT simulative model and communication interfaces.

Moreover, users were asked to execute the VR-SHAP tasks following the procedure described in the previous section. The results in terms of the average time and standard deviation are reported in Table 1. These values are then used for evaluating the VR-SHAP test according to Equations (14) and (15).

At the end of all of the tests, the users were asked to rate the level of difficulty encountered in performing the tasks and provide reasons for their ratings.

As we could expect (Figure 14), the first-level tasks show a high easiness rating, due to there being fewer constraints for the implementation of the OAF method. Despite the higher difficulty of the third-level tasks (due to the reduced size of the objects to grasp), the easiness rating was higher than that of the second-level tasks. We argue that this can be related to the reduced number of degrees of freedom of the virtual prosthesis wrist which makes the reaching of large rotations more difficult.

To score the clarity of guidelines and the general impact with virtual reality (related to the sensations experienced throughout the immersion in the virtual environment), users were asked to rate the (dis)comfort of their experience. According to the results (Figure 15), the guidelines were considered very clear (the application being as independent as possible from the operator), with the sickness directly correlated with the users’ age. In general, the immersive experience was well tolerated by all the users without excessive discomfort.

Regarding the setup, a higher difficulty was reported in the positioning of the second tracker (that may interfere with the myoelectric sensors’ mounting). Moreover, some users initially reported some issues with the task involving handle rotation, so that we facilitated the self-alignment of the handle with the virtual hand in running, reducing the torsional stiffness of the spring attached to the handle. Some movements and interactions were also improved and optimized to reduce the gap between the DT and the PT. Finally, the initial discomfort perceived by some users at the end of the virtual experience was reduced by softening the colours used in the virtual environment.

6. Conclusions

We report a complete virtual environment based on digital twin resources, including hardware and software issues, to support the amputee’s training of an upper-limb sensorized prosthesis. The hardware consists of the HTC Vive Pro system and the software is built in-house using the UNITY development platform. The digital twin is developed using multibody dynamics techniques and the sequential impulse solver formulation that allows for a robust, accurate, and reliable solution of the equation of motion, including contact dynamics.

The overall system was tested with users performing several training tasks including the gesture intent recognition and reproducing most of the SHAP test assessment methodology. According to the usability tests, the training tasks appear easy to use, and very limited pre-training and setting are needed. The dexterity of the user, measured according to the test KPIs, increases over time with practice, confirming the procedure’s efficiency. The usability analysis highlighted the good results (even in this initial phase), suggesting possible improvements both for the configuration phase of the sensors and in the implementation of the interactive environment.

In general, this study demonstrates the validity of training tools developed using virtual reality, interactive environments, and digital twins for medical rehabilitation purposes. Future improvements of the digital twin can regard force feedback systems to enforce correspondence with reality with a fully two-way communication between the physical and digital twins. The research will be completed by an extensive experimental campaign to evaluate the medium- and long-term effectiveness of using the proposed virtual environment as a training aid in comparison with the standard procedure. At present, we have developed a single digital twin, which, as mentioned in the introduction, adapts to the iHannes prosthesis produced by the INAIL prosthetic centre. However, we believe that the approach and the virtual environment can be fully generalizable by simply updating the geometrical models and the mathematical conditions for different poses and grips.