A Pre-Fitting Mixed Reality System for Myosignals Evaluation and Optimal Control Training for Hand Prostheses

Abstract

Nowadays, upper limb prosthesis acceptance remains low due to ineffective control techniques and the relative training methods. However, an engaging training method applied during early prosthetic fitting seems to have a positive impact on acceptance. To this aim, in this paper we present a Mixed Reality (MR) pre-fitting training system based on Microsoft HoloLens2, designed for users of the Hannes prosthetic hand. The system allows individuals with varying stump morphologies to control a holographic Hannes hand through a low-latency interface, replicating the physical architecture and motion of the real device in an immersive, portable environment. This pilot study presents the preliminary evaluation of the MR framework conducted with two limb difference participants (one naïve user and one experienced user in myoelectric prosthesis control) performing a novel bimanual Target Achievement Control (TAC) test. Their performance in controlling the real Hannes device was subsequently compared to a reference group of eight transradial limb difference individuals. Both MR-trained participants showed faster learning and better adaptation to the real prosthesis than the reference group. These preliminary findings suggest that the proposed MR framework, combined with an MR bimanual TAC test, may improve learning and user experience, ultimately contributing to lower prosthesis abandonment rates.

1. Introduction

Despite the advancements of the last decade in the field of upper-limb prosthetics from both mechatronic and control perspectives, no significant improvements in the overall abandonment rate have been reported. Recently, a survey on prosthetic usage has been conducted to evaluate user acceptance in the context of current technological developments [1]. Unlike the previous review on prosthesis abandonment rates [2], participants were recruited not only from rehabilitation centers but also from the community, providing a broader representation of prosthesis use in real-world settings [1]. A questionnaire was administered to 68 traumatic upper-limb amputees, of whom 25 responded (93% of the prosthesis users were myoelectric prosthesis users). The study reported a rejection rate of 44%, with no significant difference in prosthesis acceptance over the past decade [1].

Typical powered prostheses are controlled by muscle activity of the two antagonist muscles, flexor and extensor of the wrist, generally recorded by surface electromyographic (EMG) sensors, as described in [2]: myoelectric control. The muscle contraction level modulates the speed or strength of the grasping, implementing a proportional control [3]. Although different reasons can be identified as main sources for the still high abandonment rate, the most relevant are the effectiveness of myoelectric prosthetic control and prosthesis control training, together with the lack of comfort and weight of the device [1,4].

Therefore, targeted and personalized prosthetic training may have a positive impact on device acceptance. In particular, the quality of the training seems to be one of the most relevant factors, together with early prosthetic fittings: prosthetic training within the first six months after amputation (before amputees get used to performing most of the everyday activities with one hand) leads to a much higher acceptance [1,5].

As a consequence, highlighting the importance of muscle training throughout all phases of rehabilitation, from pre-prosthesis to post-prosthesis receiving, is crucial for enhancing usability and user acceptance of the final device [6]. Irrespective of the prosthesis control method, users must possess the ability to selectively activate specific muscles and modulate their activation for proportional control over degrees of freedom. Typically, this involves executing movements distinct from those used before amputation to achieve desired actions [7]. Consequently, evaluating myo-signals becomes a fundamental step for an early myoelectric prosthetic fitting. This evaluation includes identifying residual antagonist muscles, determining optimal electrode positions, and setting amplifier gains for precise socket manufacture and easy control [8].

Researchers have explored integrating technology to aid pre-prosthetic training for individuals with amputations or congenital limb deficiencies. Various solutions, including myoelectric signal visualization and virtual prosthesis control on screens, have been presented over the years [7,8,9]. Notably, modern technologies such as advanced VR headsets [10,11] have enabled the development of immersive applications, allowing individuals with upper-limb differences to simulate virtual prosthesis control in engaging environments [12]. Studies indicate that immersive VR applications enhance the sense of ownership over a virtual limb, positively influencing prosthesis acceptance and satisfaction [13].

The literature delves into innovative approaches in prosthetic training, particularly leveraging Augmented Reality (AR). A noteworthy solution is the game-based training tool [14], which offers an intuitive method for myoelectric prosthetic muscle training. Users experience a real-time mirrored view with a virtual arm superimposed on their residual limb, controlled by muscle activity and aided by fiducial markers.

With the advent of head-mounted display devices (HMDs) [15], virtual environments became increasingly immersive by providing a head-tracked first-person view, allowing the viewpoint of the user to be dynamically updated according to head movements. This immersive perspective has been shown to facilitate embodiment and enable more precise interactions with virtual objects [16]. The advent of HMDs has facilitated the development of Mixed Reality (MR) training systems for upper-limb prosthesis control that project an augmented virtual prosthesis controlled through muscle activation and enable users to interact with holograms using gestures or voice commands [17,18,19]. The immersive potential of MR is highlighted in [17], where participants are free to manipulate holographic objects with their healthy hand and with a holographic arm projected on the stump to treat phantom limb pain.

MR also holds considerable potential for providing engaging and effective prosthetic training. Advanced prosthetic control has been investigated using novel MR-based training systems [18,19], in which a virtual arm model is projected onto the arm of the user through an HMD equipped with a stereoscopic camera. The virtual arm is controlled through EMG signals acquired by a Myo Armband placed on the residual limb of the participant. It is demonstrated that prosthesis control skills can be transferred from the MR environment to real scenarios [19].

Despite promising advances in MR-based prosthetic training, some limitations remain. In particular, similar MR prosthetic training setups introduce latency in visualization, with an average control delay of 250 ms [18,19]. Such delays can impart an unnatural sense of control over the artificial limb, potentially affecting the user experience. The maximum amount of time that can be used by the controller for data collection and analysis to maximize classification accuracy without affecting prosthesis performance is called the optimal delay threshold. It was demonstrated that the optimal control delay threshold is 100 ms: delays exceeding 100 ms can result in a consistent decrease in prosthesis performance [20]. Moreover, the design of the MR systems limits the prosthetic training exclusively to distal transradial amputees that meet specific anatomical requirements [13,21]. This constraint arises from the need for the residual limb to be sufficiently long to allow the camera to maintain a clear view of the marker used to perform virtual prosthesis and stump alignment, while also being sufficiently thick to fit within the adjustable circumference range of the commercial EMG armband. For the Myo Armband, the adjustable circumference range is approximately 19–34 cm [22]. Additionally, MR prosthetic training systems were mainly tested on able-bodied participants and preliminarily evaluated with a single transradial limb-difference individual. The potential influence of prior experience in myoelectric prosthesis control of the participant on the outcomes could not be assessed. Importantly, the absence of a control group of participants with upper-limb differences limits the ability to determine whether skills acquired through MR-based training can transfer to real-world prosthesis control.

The present work introduces an MR upper-limb prosthetic training system designed to address critical limitations identified in previous literature. The proposed system achieves a control latency below the optimal threshold reported for prosthetic performance, reducing the temporal discrepancy between myoelectric commands and the visualization of the virtual prosthesis. Moreover, the virtual prosthesis of the MR system shares the same hardware and control architecture embedded in the real prosthetic device used in the real-world setting: the Hannes hand prosthesis [23]. In addition, our architecture reduces the anatomical constraints imposed by previous systems, as it does not require the residual limb to meet specific length and circumference requirements for camera-based marker tracking and EMG armband placement. This potentially broadens the applicability of MR-based training to users with different residual limb anatomies.

Unlike previous studies that did not assess the influence of prior myoelectric experience, our preliminary evaluation involved transradial limb-difference participants with different experience in myoelectric prosthesis control. Finally, while previous MR-based training systems were predominantly evaluated with able-bodied subjects, the present study involved only upper-limb-different individuals who completed a full clinical protocol with the Hannes prosthesis. This provides a more realistic preliminary assessment of the efficacy of the proposed MR training system to control a real prosthesis in real scenarios.

To preliminarily validate the proposed system, we developed an extended version of the original Target Achievement Control (TAC) test as part of the Hannes pre-fitting process. The TAC test is a real-time virtual assessment of myoelectric prosthesis control in which the user attempts to move a virtual prosthesis from a predefined starting position to a specified target posture and maintain it within a defined tolerance for a predetermined dwell time. The test evaluates the ability of the user to accurately and efficiently control the degrees of freedom of the virtual prosthesis [24]. In this work, we introduce a novel version of the TAC test in which the target posture is represented by a 3D hologram and needs to be reached both with the real healthy hand of the participant and with the EMG-controlled virtual Hannes prosthesis. This test is seamlessly integrated into the MR environment, employing a custom-made setup and an MR framework tailored for Microsoft Hololens2 [10].

While the sample size of the involved population limits statistical conclusions, this study aims to overcome key limitations of prior works and demonstrate the potential benefits of the proposed MR platform as a portable and effective preparatory training for people approaching a new prosthesis.

2. Materials and Methods

In this section, the overall Hannes system and HoloApp architecture, as well as a use-case validation method, are described. Firstly, in Section 2.1, the overall architecture is presented in detail, focusing on the integration between the MR platform and the Hannes prosthetic device. Subsequently, in Section 2.2, the different communication methods between the hardware components of the overall system are analyzed to evaluate latency. Additionally, Section 2.3 highlights the HoloApp developed for the immersive prosthetic pre-fitting process (muscular evaluation and control training). Finally, in Section 2.4, our novel bimanual TAC test is presented as a user case study for the preliminary validation of the system.

2.1. Overall Architecture

A Mixed Reality system has been developed employing a HoloLens 2 device manufactured by Microsoft (Redmond, WA, USA). The device can be worn by the user, overlapping holograms onto the real environment. Further, the system can implement a prosthesis hologram that reproduces the real behavior of the Hannes prosthetic system [23]. According to that, the user can control the holographic prosthesis using muscle contraction in the same manner as a commercial prosthesis. This proposed solution helps fully customize, train and validate the Hannes system and its controller, with a realistic relationship between the real environment and the virtual one, with the aim of realizing advanced assessment protocols capable of promoting the learning process of new users and the integration between user and prosthesis. The overall system is described in Figure 1 and consists of three main components:

Figure 1. (a) Mixed Reality overall system; (b) Schematic representation of the communication between the components of the overall MR system: the PC makes the bridge between the user EMG data and the holographic representation in the Hololens app.

• Hannes prosthesis system;
• The virtual environment;
• Hololens app.

The Hannes prosthesis system is composed of a custom-made socket that represents the physical connection interface between the user and the prosthesis. The socket contains a battery pack, a set of two electromyographic sensors (EMG, 13E200 MyoBock Electrodes), and an EMG processing board (EMG-Master) [23]. The two EMG sensors, respectively placed in the forearm flexor and extensor muscles, are used to detect the residual muscular activity at the stump level, i.e., the muscular contractions of the forearm flexor and extensor muscles. The muscular activity was processed by the EMG-Master to generate control signals to actuate the Hannes prosthesis. Such a board can record up to 6 EMG sensors at 300 Hz and generate control signals to move up to 3 degrees of freedom (DoFs) of the Hannes hand [25]: hand aperture, wrist rotation, and wrist flexion/extension. It is worth noticing that, for the architecture proposed by this work, the EMG-Master can communicate with a Host PC via Bluetooth to send the control commands for controlling the virtual hand in the virtual environment according to the muscular activity.

A VR framework was developed using the Unity development suite in C#. This application runs on a Host PC (Dell precision 3571, Intel i7, 16 Gb Ram, Windows 10), where the virtual representation of Hannes is controlled with the EMG activity of the residual limb of the patient, replicating exactly the proportional control of the real prosthesis. At the same time, the host PC sends the velocity references received from the EMG Master to Microsoft Hololens2.

The VR framework allows the clinician to perform EMG signal evaluation and control parameter tuning. Parameter tuning for EMG prosthesis control refers to the process of adjusting and optimizing the control parameters of an EMG-based prosthetic system to achieve reliable, accurate, and intuitive control. It typically represents one of the first stages after receiving a myoelectric prosthesis. The modified control parameters are instantly sent to the EMG Master and saved in the EEPROM memory so that the patient can immediately test the comfort of controlling the real Hannes hand and the virtual hand.

Such a virtual solution can evaluate the level of controllability of the device reached by the user, performing a Target Achievement Control test. This test was already employed in different works to evaluate the performance of users on-screen and control algorithms to manage multiple DoFs. Therefore, the on-screen test does not allow replication of a real scenario where the user does not perceive the size of the device and the third dimension of the hand configuration. So, to overcome this limitation, the Hololens app was developed in a Mixed Reality environment.

The Hololens app (HoloApp) is an application realized using the Unity development suite in C#. This application runs on a Hololens2 device, reproducing the augmented representation of Hannes’ prosthesis in the user’s view. The prosthesis is projected as a 3D hologram, and it moves coherently with the virtual hand and the real hand prosthesis. Such a solution allows the user to perform any kind of test in a realistic condition.

2.2. Device Communication

The EMG-Master board performs A/D conversion of EMG signals at 300 Hz and then transmits data to the host PC via Bluetooth communication [26] using two BGX13P modules (one embedded on the EMG-Master board and one on a USB dongle connected to the PC), making a bridge between the devices. The Bluetooth communication between the EMG-Master and the host PC is bilateral: (i) the EMG data, the prosthesis measurement, and the References are sent from the EMG-Master to the host PC; (ii) the parameters customized to the user are sent from the host PC to the EMG-Master to be stored on board EEPROM during the tuning phase. At the same time, the host PC runs a Unity-based application that receives and parses the data coming from Bluetooth. In that application, the virtual hand moves on the screen, replicating the same movements of the real hand at 200 Hz. Then, the application on the PC communicates with the Hololens2 device wirelessly using a router as a bridge. In that case, the Hololens2 runs the Unity-based HoloApp, and the host PC sends control commands through a UDP communication protocol in order to move the holographic hand.

The most important technical requirement for this application is the latency: it needs to be below the perceived delay of real prosthesis control [20].

The total latency of the proposed system is the sum of: EMG-Master acquisition time, Bluetooth data packet transmission time from EMG-Master to host PC, Unity PC application frame interval, UDP data packet transmission time from host PC to Hololens2 device and Unity Hololens application frame interval (Table 1).

Table 1. Latency of each component of the setup.

Latencies	T [ms]
EMGMaster EMG acquisition	<3.3
BT data transmission	5
Unity PC app frame interval	16.6
UDP data transmission	<15
Unity Hololens app frame interval	16.6
Total	<56.5

Several tests on the BGX13P module used as a USB dongle on the host PC were conducted to find the streaming data transmission time from the EMG-Master to the host PC. It was found that, for transmissions of continuous flows of characters with a baud rate of 115,200 bps (board default value), the characteristic flight time of the data packet is 5 ms.

Regarding the communication between the PC and Hololens, real-time EMG signal data are transmitted from the host PC (server) to the MR device (client) using UDP sockets. To estimate the transmission time of the data packets, an echo response of each data packet received from the host PC was implemented on Hololens: the holographic glasses send the data back to the server immediately after having received them. The UDP transmission time from the host PC to Hololens is logically lower than the time interval between the server transmission and reception of the same data. The UDP transmission-reception times were collected across 3 days, performing 3 TAC tests lasting one minute for each day. The mean of the times collected was calculated. It was found that the mean UDP transmission and reception time during real-time EMG signal data streaming to Hololens is equal to 15 ms. The summarization of the different delay components obtains an overall System Latency < 56.5 ms, which is under the optimal control delay threshold of 100 ms [20] with respect to similar MR prosthetic training work [18,19].

2.3. HoloApp

The amputee can perform muscular evaluation and train the control parameters of the future Hannes device thanks to the HoloApp. The application consists of three main phases: (i) prosthesis projection; (ii) parameter tuning; and (iii) Assessment. A specific case of assessment will be described in Section 2.4.

2.3.1. Prosthesis Projection

Since AR is the superposition of virtual objects in the real environment, it can provide individuals with real-time interactions, removing the need to use mental imagery. Similar to AR applications that enhance the realistic experience of virtually trying on their products to aid consumers during decision-making [27], it is possible to create AR solutions in which users digitally wear the Hannes prosthesis during the pre-fitting process.

First, the firmware projects on the flat surface in front of the user the holograms of two boxes. Each of them contains the components of the future real prosthesis: the socket and the Hannes hand. One of the boxes contains the components of the right Hannes prosthesis, and the other contains the components of a left Hannes prosthesis. The user needs to touch the box corresponding to his/her amputation to open its contents. The holograms of the socket and the Hannes hand can be grabbed and manipulated (translated and rotated) with the healthy hand. Before controlling the holographic prosthesis, the patient needs to wear it, replicating and mimicking the real fitting phase of the physical device (Figure 2). The user must:

Figure 2. HoloApp procedure to select and virtually wear the prosthesis in the residual limb: (a) choose between right hand and left hand, (b) grab the socket and place it on the residual limb, (c) grab the hand and place it over the socket, (d) push the plug button on the socket to turn on the prosthesis.

• Choose between right hand and left hand;
• Grab the socket and place it on the residual limb;
• Grab the hand and place it over the socket;
• Push the plug button on the socket to turn on the prosthesis.

The hologram of the socket is fixed to the position given by the user, who can always grab it and place it again. Unlike other solutions [17,18,21], the residual limb position is not constantly estimated with external devices. We preferred computational efficiency and the possibility to use the firmware with any kind of upper limb amputation over the hologram following the stump when moved. In the evaluation pre-fitting phase, the patient does not have to get used to the weight of the prosthesis (like the post-fitting phases), so the procedure can be performed with the residual limb comfortably leaned on the table. When the holographic prosthesis is turned on, the firmware starts to wait for the command data from the host PC that animates the hand coherently.

2.3.2. Parameter Tuning

The clinician performs the EMG signal evaluation and starts tuning the control parameters. In particular, the tuned parameters involved the minimum signal amplitude (threshold) to generate a movement and the associated mapping of speed for proportional control (gain). These two parameters are fundamental to allow the amputee to control the holographic Hannes hand with fine motion. The clinician changes these values with a slider and sends the new parameters to Hololens through UDP so that the user can control the holographic hand with the new settings and test the resulting motion. The host PC also sends the new parameter values to the EMG-Master through Bluetooth: the values are saved on the EEPROM of the board.

2.4. A Case Study: A Novel MR Bimanual TAC Test

Once the user achieves accurate and fine control with the selected parameters, it is possible to train the participant to activate muscles selectively and to modulate their activation for proportional control with the proposed MR bimanual TAC test. Our novel TAC test was inspired by the virtual TAC test presented in [28], which consists of a virtual solid arm that can be controlled by the user with EMG signals, and a second virtual semi-transparent hand which shows the target configuration. Unlike this study [28], both the virtual EMG-controlled hand and the virtual semi-transparent target hand of our TAC test are 3D holograms and are placed in the real environment. Moreover, since bimanual augmented reality exercises have been studied to treat Phantom Limb Pain (especially for amputees during early prosthetic fitting) [17], the proposed TAC test is composed of bimanual trials. In particular, the test is composed of 3 sessions of 21 trials each, and a single trial consists of the following steps, represented in Figure 3:

Figure 3. MR bimanual Target Achievement Control Test. In step A the healthy hand of the user reaches the virtual target; in step B the virtual target disappears, and the user can control the virtual Hannes to reach the healthy hand configuration; in step C the subject reaches the correct configuration during the available time. The task is completed, and a novel virtual target will appear in a new configuration.

• The semi-transparent holographic target hand is projected over the healthy hand of the subject. The user is free to move the hand: the hologram will follow its position and orientation in space. The target position is randomized while ensuring that the same position is not repeated consecutively.
• The user needs to reach the target configuration with the healthy hand with a predefined tolerance. The configuration error needs to be kept lower than 5% for 1 s consecutively. After that, the semi-transparent hand disappears.
• The user has a limited amount of time to reach the target configuration with a predefined tolerance with the Hannes hand hologram. The single trial must be considered successful if the configuration error is kept lower than 5% for 3 s consecutively. The single trial must be considered failed if the target configuration is not reached within 20 s or if the configuration of the healthy hand changes in the process.

A custom algorithm has been developed so that the healthy hand posture could be comparable with that of the holographic Hannes hand, since their motion is different. The biggest difference in the motion is that the distal phalanges are not actuated in the Hannes prosthesis, so they cannot be considered in the evaluation of the posture. For this reason, to get the posture of the healthy hand, we decided to detect the index-middle joint and the thumb proximal joint and measured the distance between them (D). Since every person has a different maximum opening and a maximum closing position of the hand, each user must perform a fast calibration prior to the TAC test. The calibration phase consists of performing 5 opening/closing gestures with the healthy hand to find the maximum distance (Dmax) and minimum distance (Dmin) between the index middle joint and the thumb proximal joint of the user.

Regarding the semi-transparent hand and the Hannes hand hologram, the opening/closing references (AC) range from 0 (maximum opening) to 1 (maximum closing). In step 1, the reference value for AC movement of the semi-transparent hand is randomly generated, while in step 3 the references are continually sent from the host PC. To find out if the healthy hand and the semi-transparent hand (step 1) or the holographic Hannes hand (step 3) have the same posture with a tolerance of 5% degrees, the following condition needs to be verified:

AC − 0.05 < 1 − (D − Dmin)/(Dmax − Dmin) < AC + 0.05,

(1)

The empirically identified tolerance threshold is 0.05.

2.4.1. Subjects

Two upper limb different adults who never used a Hannes prosthesis before voluntarily participated in the experiment, performing the MR training and evaluation with the proposed MR bimanual TAC test. The participants were a naïve subject with transradial congenital limb deficiency (female, 50 years old), who had never used a myoelectric prosthesis, and an experienced user (pro user) in myoelectric control with a transradial amputation (male, 60 years old). After this pre-fitting training, a real Hannes hand was given to the two participants to perform some clinically validated functional tests. To validate the efficacy of such an MR protocol in improving the controllability of Hannes, the performances of the two participants were compared with those of a reference group of eight new Hannes prosthesis users (N = 8; 1 female; 7 males; 51.22 ± 6.31 years old) with previous experience in EMG control, who followed the same protocol as the tested subjects but without the use of MR training. The experimental protocol (CP-PP3AS1/1-03) was approved by the AVEC (Area Vasta Emilia Centro) Ethics Committee, and the subjects signed an informed consent form to perform all the following tests. It is important to acknowledge that the recruitment of participants with limb differences represents a significant challenge in clinical studies. In this work, this challenge was further increased by the additional requirement that participants had no prior experience in controlling the Hannes prosthesis.

2.4.2. Experimental Protocol

The experimental protocol is followed by the two subjects described in Section 2.4.1 and consists of the steps described in Section 2.3. In this scenario, the two participants wear the Hololens device and start the app to perform the experiment during the pre-fitting phase. In the first step, the subjects need to wear the virtual prosthesis. After this step, the experimenter sets the EMG parameters (threshold and gain) to guarantee the user suitable controllability of the virtual hand. Then, the subjects have a single training session of 21 trials to learn how the MR bimanual TAC test works and set the proper time limit for the real experimental phase. The experiment phase consists of a total of 3 sessions of 21 bimanual TAC test trials each. We will call this part of the experimental protocol “V-1”. During the V-1 phase, the error over time (between the controlled hand and target hand) was collected to perform the offline analysis of the performance of the naïve subject and have the comparison with the pro user.

After the V-1, a real Hannes device was given for the first time to the two subjects to perform 3 clinically validated functional tests: 3 sessions of the Minnesota Dexterity Test (MMDT) and the Southampton Hand Assessment Procedure (SHAP), during which the execution times were measured, and 3 sessions of the Box and Blocks test (BBT), in which the number of moved boxes was collected. We will call this part of the experimental protocol “T-1”. The T-1 functional test session was also performed by all the other pro users who, like the two participants that performed the V-1 experimental phase, had never used the Hannes prosthesis before.

The absence of a real-world Hannes baseline before the V-1 MR training could limit the extent to which improvements in real-world prosthesis management can be directly attributed to the proposed MR system. The effect of the MR training was assessed through a second virtual training phase and subsequent practical Hannes functional tests conducted a month after the V-1 and the T-1. The second virtual training phase consists of another MR bimanual TAC test, and it was performed by the two subjects that participated in the V-1. We will call this second virtual experimental phase “V0”. Like the V-1, the V0 consists of 3 sessions of 21 trials each.

After the V0, the two subjects performed all the functional tests again. We will call this part of the experimental protocol “T0”. The T0 functional tests session was also performed by all the other pro users that did not perform the virtual experimental sessions (V-1 and V0).

During T-1 and T0, performance data were collected to perform the offline analysis to progressively compare the ability in using Hannes prosthesis between the participants of the virtual experimental sessions and the reference group.

2.4.3. Data Analysis

Success rate and position error have been obtained for each subject from the measurements collected during each session of the V-1 and V0 bimanual TAC tests. Success rate is defined as the ratio between the number of successful trials and the total number of trials. The position error is defined as the absolute difference between the target hand aperture/closure and the controlled hand aperture/closure at the end of each trial.

The data analysis during the sessions of the bimanual TAC test (V-1 and V0) aimed to compare the performance trends in controlling a virtual Hannes prosthesis with EMG signals between a naïve subject in myoelectric hand control and a pro user.

Following each bimanual TAC test, the two subjects completed two questionnaires to provide insights into the subjective experience of the proposed Mixed Reality platform: the System Usability Scale (SUS) for measuring the usability of the system, and the NASA Task Load Index (TLX) for assessing the subject workload.

Regarding the T-1 and T0 hand function tests, the MMDT and SHAP average execution times, along with the average BBT score of the two subjects and the reference group, were obtained.

The data analysis during the three functional tests (T-1 and T0) aimed to compare the performance trends in controlling a real Hannes prosthesis between the two subjects that underwent the virtual sessions (V-1 and V0) and the reference group who did not receive the virtual sessions.

3. Results

The bar graphs of Figure 4 show the success rate and the average position error of the naïve subject compared to the pro user across the three sessions of the MR bimanual TAC test before the Hannes fitting phase (V-1). In particular, the position error is expressed as the ratio between the position error and the maximum position error; data are reported as mean ± SD across the 21 trials of each session. Both success rate and position error are expressed as a percentage.

Figure 4. Performance of the naïve (blue) and the pro user (red) subjects across the MR bimanual TAC test before the Hannes fitting phase (V-1). (a) The success rate across the sessions; (b) The position error reported as mean ± SD across the sessions.

The mean value of the success rate across the three sessions is 52% for the naïve subject and 61.6% for the pro user, while the mean value of the position error is 8.1 ± 13.4% for the naïve subject and 9.9 ± 16.2% for the pro user. Results show a greater increase in the success rate of the naïve subject: in the last session (S3), the naïve subject achieved a success rate of 75.8%, which was higher than the 61.6% observed for the pro user, whose performance remains approximately stable during the test.

After the pre-fitting phase, the naïve subject also achieved better results in performing the 3 functional tests during the T-1 experimental phase compared to the pro user (except for SHAP power grasp), as shown in Table 2.

Table 2. Results obtained from all the subjects during functional tests after the pre-fitting phase T-1. The values expressed in the Reference Group column represent the mean value of the results of each functional test calculated across the eight members of the reference group.

Functional Test	Naïve	Pro User	Reference Group
MMDT (s)	223	244.7	219.41
BBT (n)	17.7	11.7	14.90
SHAP–spherical (%)	64	57	55.13
SHAP–tripod (%)	35	12	24.75
SHAP–power (%)	34	42	36.38
SHAP–lateral (%)	50	30	38.67
SHAP–tip (%)	15	12	12.75
SHAP–extension (%)	63	25	48.63
SHAP–IoF (%)	43	34	40

Table 2 also indicates that the results of the naïve subject in performing the BBT and the SHAP test are better compared to those of the other pro users who did not use the Mixed Reality setup (except for SHAP power grasp), while the MMDT performances are similar. The naïve subject achieved similar performances in the MMDT compared to the pro users. Contrary to the pro user, who used the MR setup, the naïve subject outperformed in all functional tests (except for SHAP power grasps).

The bar graphs of Figure 5 show the success rate and the average position error of the naïve subject compared to the pro user across the three sessions of the MR bimanual TAC test after the Hannes fitting phase (V0). The mean value of the success rate across the three sessions was 86.8% for the naïve subject and 70.9% for the pro user, while the mean position error was 3 ± 4.7% for the naïve subject and 7.6 ± 14.2% for the pro user. Like the pre-fitting phase (V-1), results show a greater increase in the success rate of the naïve subject, along with an evident decrease in the number of trials failed: in the last session (S3), the naïve subject achieved a success rate of 100%, which was higher than the 76% observed for the pro user, whose performance remains approximately stable during the test. The naïve subject also achieved an overall better performance in the functional tests during the T0 experimental phase compared to the pro user.

Figure 5. Performance of naïve (blue) and pro user (red) subjects across the 3 sessions of the MR bimanual TAC test after the Hannes fitting phase (V0). (a) The success rate across the sessions; (b) The position error reported as mean ± SD across the sessions.

Table 3 shows results of the naïve subject in performing the MMDT, and most of the SHAP grasp types were performed better by the naïve subject, while the BBT average score is consistent for all subjects. Moreover, the naïve subject attains an average better performance in all the functional tests compared to the pro users who did not use the MR setup (except for SHAP tripod and tip grasps). Unlike T-1, the results of all functional tests of the pro user who performed V0 are close to the results of the other pro users (except for SHAP spherical grasp), indicating a faster learning curve in using the Hannes prosthesis.

Table 3. Results obtained from all the subjects during functional tests after the T0 phase. The values expressed in the Reference Group column represent the mean value of the results of each functional test calculated across the eight members of the reference group.

Functional Test	Naïve	Pro User	Reference Group
MMDT (s)	185	203	192.45
BBT (n)	17	17.7	16.85
SHAP–spherical (%)	77	37	71
SHAP–tripod (%)	28	43	41.88
SHAP–power (%)	51	58	49.63
SHAP–lateral (%)	56	42	46.75
SHAP–tip (%)	26	20	27.25
SHAP–extension (%)	74	54	64.50
SHAP–IoF (%)	53	48	50.88

The subjective experience with the proposed MR platform is shown in Figure 6. The SUS test indicates that the subjects appreciated the usability of the system: they found the application nice and quite easy to use during both V-1 and V0. The NASA TLX questionnaires show that the mental load was the predominant workload during both V-1 and V0. The virtual TAC tests were not physically demanding and did not cause much frustration. Both subjects experienced more workload during V0 compared to V-1, especially the naïve subject.

Figure 6. Questionnaire results of Naïve (blue) and Pro-user (red) subjects after V-1 (light colors), and after V0 (dark colors): (a) SUS questionnaire results, (b) NASA LTX first part questionnaire results, (c) NASA LTX second part questionnaire results. The “X” indicates the same result between V-1 and V0 for the specific question.

4. Discussion

In this work, we proposed an architecture of an MR training system. This system enables an upper limb deficient person with any kind of residual limb morphology to control a hologram Hannes device with a latency under the optimal control threshold for prostheses [20]. The objective was to demonstrate the potential benefits of the proposed MR platform as a portable and effective preparatory training before the fitting phase of a real prosthesis. To achieve that, we investigated the learning phase in controlling Hannes prosthesis of a group of 10 upper limb deficient subjects that completed a full clinical trial with the device. Two subjects in the group enrolled for this study also completed the myoelectric signals evaluation and training with the proposed MR system. The long-term goal of this approach is to potentially reduce prosthesis rejection by improving signal evaluation and control training.

The results of the V-1 pre-fitting experimental phase reveal that an upper limb deficient subject who has never used a myoelectric prosthesis faces more difficulties in performing the initial sessions of the MR bimanual TAC test compared to a myoelectric prosthesis pro user. However, the naïve subject achieves better performance by the end of the test, showing a significant improvement in controlling the Hannes system compared to the pro user. This improvement may be attributed to the fact that the pro user required more time to modify his control strategy, having been trained with different myoelectric signals. In contrast, the naïve subject started training with completely new EMG control signals. These results are further confirmed after the Hannes fitting phase. During the T-1 practical session with the real prosthesis, the naïve subject demonstrates better performance in almost all the functional tests compared to the pro user, especially in tasks requiring fine myoelectric control like the MMDT and BBT. These preliminary results suggest that the proposed MR platform may have potential as an early evaluation and training method for naïve subjects and that previous experience with a different myoelectric control strategy may influence the adaptation process. These observations suggest that combining traditional methods with the proposed MR system may have the potential for the naïve subject to outperform the pro user in almost all the functional tests to support and possibly accelerate the learning process for new myoelectric prosthesis users.

The virtual experimental phase V0 and the functional tests in the T0 phase were conducted to investigate the learning phase in mastering Hannes prosthesis control signals over time. The results of the V0 phase show a significant improvement of the naïve subject in controlling the Hannes virtual prosthesis during the MR bimanual TAC test compared to the pro user. Once again, the difficulty of the pro user in changing his control strategy is likely the cause. These results are reiterated in the T0 practical session with the real prosthesis, where the naïve subject outperforms the pro user in almost all the functional tests. This observation may indicate the importance of early training for naïve users. However, despite lower performance in functional tests, the pro user demonstrates a much faster learning curve in controlling the Hannes real prosthesis compared to the reference group that did not use the MR platform, particularly in tasks requiring fine myoelectric control like the MMDT and BBT.

These findings suggest that the proposed MR platform appears to be a feasible and useful system for signal evaluation and early training in new users of real prosthetic devices, like the Hannes prosthesis. Nevertheless, the limited sample size prevents drawing definitive conclusions about its effectiveness.

5. Conclusions

We introduced a novel Mixed Reality platform designed to support the training of upper limb deficient people approaching a new prosthetic device. The proposed system provides a portable and realistic training environment that can accommodate users with different residual limb morphologies and levels of prior myoelectric experience. Tested with both a naïve user and an experienced user in myoelectric control, preliminary results suggest a potential benefit of MR-based pre-fitting training for adaptation to the Hannes prosthesis. Furthermore, the participants reported positive experiences with the MR-based training, suggesting that the approach may provide an engaging training environment. Although larger studies are needed to validate these preliminary observations, the proposed MR pre-fitting approach may represent a valuable complementary tool within rehabilitation workflows, potentially enhancing user experience and supporting long-term prosthesis adoption.