Tactile Myography: An Off-Line Assessment of Able-Bodied Subjects and One Upper-Limb Amputee

Abstract

Human-machine interfaces to control prosthetic devices still suffer from scarce dexterity and low reliability; for this reason, the community of assistive robotics is exploring novel solutions to the problem of myocontrol. In this work, we present experimental results pointing in the direction that one such method, namely Tactile Myography (TMG), can improve the situation. In particular, we use a shape-conformable high-resolution tactile bracelet wrapped around the forearm/residual limb to discriminate several wrist and finger activations performed by able-bodied subjects and a trans-radial amputee. Several combinations of features/classifiers were tested to discriminate among the activations. The balanced accuracy obtained by the best classifier/feature combination was on average 89.15% (able-bodied subjects) and 88.72% (amputated subject); when considering wrist activations only, the results were on average 98.44% for the able-bodied subjects and 98.72% for the amputee. The results obtained from the amputee were comparable to those obtained by the able-bodied subjects. This suggests that TMG is a viable technique for myoprosthetic control, either as a replacement of or as a companion to traditional surface electromyography.

1. Introduction

Upper-limb amputations are a serious impediment in a world where the large majority of daily tasks are operated by hands. In spite of the remarkable body of research conducted in the scientific communities of assistive robotics, sensors, machine learning and biomedical engineering in the past couple of decades, we are still far from the ideal solution to provide a reliable form of control of upper-limb prostheses. According to comprehensive reviews [1,2,3], rejection rates reach disastrous levels for all kinds of upper-limb prosthetic devices (body-powered, self-powered and/or passive devices). Reported mean rejection rates are on average one third for pediatric and one fourth for adult patients, with about 1900 traumatic upper-limb amputations per year in Europe and maintaining an estimated total population of 94,000 upper-limb amputees [4]. At the same time, notice that one third of upper-limb amputees still does use a passive prosthesis [5] and that at least one recent study [6] makes a decisive point for body-powered devices against self-powered ones. For a somewhat contradictory survey, see, e.g., [7].

The main reasons for this impasse include the lack of dexterity and long reaction times of the prosthetic device and, most crucially, poor reliability of the Human-Machine Interface (HMI) used to control it [1,2,3,4,8]. In practice, the standard prosthetic user of a self-powered prosthesis is still unable to hold an object exactly the way and for the time (s)he wants.

Given the state of the art of myoelectric control, even an expensive multi-fingered hand prosthesis such as Touch Bionics’s i-LIMB Quantum [9] can provide little more than a body-powered one-degree-of-freedom (1-DOF) gripper. The appropriateness of such expensive solutions, as opposed to less expensive and less dexterous, but more reliable body-powered 1-DOF grippers definitely comes into question.

The community is therefore looking for new solutions to improve the reliability of myocontrol; in particular, multi-modal sensing techniques to non-invasively gather the user’s intent [10,11,12] are being explored. One such idea is that of detecting the deformations that occur on the surface of the forearm/residual limb, denoting muscular activity, and in turn representing intended wrist, hand and finger activations. From now on, we will denote as activation any action voluntarily performed by a human subject, which is also the desired target of a hand/wrist prosthesis; for instance, wrist flexion implies the activation of some of the muscles in the forearm and should correspond to the flexion of the prosthetic wrist. This informal definition has the advantage that it abstracts away from whether it is a muscle isometric contraction or the result of a movement and from whether it is being performed by able-bodied or by amputated subjects. Amputees in particular can still perform muscle contractions, which obviously lead to no movement and no generation of torque. Pioneering work by, among others, Craelius [13] has already shown that relevant information can indeed be obtained by embedding a small number of force sensors in a silicon sleeve or a semi-rigid cuff. A straightforward extension to this idea is that of employing tactile sensing, which is a high-density form of force sensing; the resulting technique is called Tactile Myography (TMG), analogous to surface Electromyography (sEMG).

In the experiment we hereby report about, we applied TMG to 10 able-bodied participants, as well as to a trans-radial amputee, and assessed its effectiveness to discriminate (classify) several wrist, hand and finger activations. We developed a shape-conformable bracelet using our custom high-density tactile sensors [14]. The bracelet can be strapped around the forearm or the residual limb. During wrist, hand and finger activations, our bracelet yields a high-resolution image of the related forearm/residual limb deformations, which cause pressure changes that are captured by the tactile sensors. During two experiments, we associated the features extracted from such images with the images from a visual stimulus and fed them to classification algorithms. In the first experiment, ten able-bodied subjects were instructed to track the stimulus, which showed repeated, simple wrist, hand and finger activations. In the second experiment, a very similar protocol was requested from a left trans-radial amputated subject.

The experimental results reveal that the classification accuracy is on average 89.15% for the able-bodied subjects and 88.72% for the amputee for finger and wrist movements; such values are in line with, or superior to, what can be found in previous literature in similar experiments (see, e.g., [15]). The results obtained by the amputee are comparable to those obtained by the able-bodied subjects, a surprising fact given that the subject had been amputated eight years before the experiment and, during this time, has been using no prostheses at all.

Related Work

The need for novel sensors to improve the accuracy and stability of myoelectric control has been advocated multiple times (e.g., in [11,12]) as a substitute and/or complement to the more traditional technique of sEMG. There is no agreement so far as to which technique or combination of techniques yields the best overall results (Guo and colleagues, as well as Ravindra and Castellini, already discussed this topic [15,16]). There is a drive in the community to hold on to surface techniques, since invasive methods such as needle EMG and direct connection to the nerves are still not practical for clinical use [4]. One such idea, and a very practical one given its alleged ease of production, low cost and high effectiveness (so far just in the laboratories) consists of detecting the changes in the volume and position of the muscles (bulging) while human subjects move and exert forces. Early experiments using pneumatics to detect muscle bulges date back to as early as 1966 [17]. From about 2000 on, this very idea has been developed with various names and slightly different kinds of sensors and housings: in [13,18], pressure vector decoding or residual kinetic imaging is defined, employing myopneumatic (pressure) sensors embedded in a silicone sleeve or in a bracelet; in [19,20], Force Myography (FMG) or surface muscle pressure is introduced, which exploits the same principle, but using very affordable Force-Sensing-Resistors (FSRs). The same research group has recently applied FMG to gait control [20] and brain injury rehabilitation [21]. Gait control was also realized in this way in an earlier paper by Lukowicz et al. [22]. FMG has been further extensively developed, tested and compared against sEMG and ultrasound imaging [15], assessed when fused with sEMG [23], and tested on amputees [24] with good-to-excellent classification accuracy. All in all, detection of muscle bulging using a small number of force or pressure sensors is well established in controlled conditions.

We define Tactile Myography (TMG) as the extension of FMG to high density (high resolution). The most interesting examples of TMG, both realized via a resistive pressure-sensing approach, are found in [25] and in [26]. Radmand et al. [26] propose a $14\times 9$ tactile array with a spatial resolution of 10 mm embedded in a rigid plastic socket and demonstrate excellent classification rates on able-bodied subjects. Furthermore, they collect data for several different body postures, an issue which is usually detrimental to any machine-learning-based myocontrol approach. (Notice that in Radmand et al.’s work, what we here denote as TMG is called HD-FMG for High-Density FMG. In this paper, we rather stick to the acronym TMG to denote this approach.) In the work by Kõiva et al. [25], which can be considered an early, preliminary version of this very paper, values of root-mean-squared error are obtained by TMG when regression is applied, which are in line with related literature employing FMG, sEMG and ultrasound imaging. TMG should show the same higher quality and stability of signals with respect to sEMG that FMG enjoys [15,23], with the advantage of potentially providing more information thanks to the higher resolution.

If we consider the output of a tactile sensor array as an image (i.e., each sensor output represents the intensity of a pixel), image processing algorithms can be applied to extract discriminative information from the pressure map. For example, Scale-Invariant Feature Transform (SIFT) is a well-established method designed for object recognition from images [27] that has been implemented in tactile-based object shape recognition [28]. Gradient-based features extracted from Regions of Interest (ROI) and Histograms of Oriented Gradients (HOG) have been utilized to discriminate finger positions from ultrasound images [29,30,31] and could potentially be applied to sensor arrays. In all these cases, the extracted features were used in combination with a classifier or a regression method to model the intent of the subject. Another approach is to learn the features and perform classification directly from the raw pressure maps, without handcrafted feature design. In this context, deep learning algorithms are becoming increasingly useful given their classification performance on image processing [32,33].

2. Materials and Methods

2.1. Experimental Setup

Two similar experiments were performed: the first involved the able-bodied subjects, whereas the second involved the amputated subject. The same experimental setup, described in detail in the following sections, was used for both experiments (see Figure 1). The setup was intentionally maintained as simple as possible, and in the end, it only consisted of the aforementioned tactile bracelet and of a monitor on which a visual stimulus was displayed.

2.1.1. Tactile Bracelet

The tactile bracelet (Figure 2) is composed of up to 10 tactile sensor boards and a single main board, which collects the tactile data from the sensor boards and optionally provides motion tracking capabilities via embedded inertial-measurement-unit. To enable optimal covering of different arm circumferences, the amount of connected sensor boards is variable by design.

Each tactile sensor is based on a resistive working principle in which the interface resistivity between two surfaces changes according to the applied load. This is achieved using conductive tracks as electrodes and conductive foam or rubber as the sensor material, which is a technique first introduced by Weiss and Wörn in [34]. Figure 2b illustrates this basic working principle and depicts the three parts that contribute to the sum of the final sensor resistance, $R_t$: the variable contact interface resistances, $R_{s1}$ and $R_{s2}$, and a constant sensor material volume resistance, $R_v$. In its simplest form, the tactile sensor electrodes can be produced by using a common printed circuit board (PCB). A number of possible sensor materials can be considered, such as elastomer foam with added carbon particles (as used in typical electrostatic discharge (ESD) packaging foam), conductive fabrics and conductive rubber. For improved respiration of the skin and for optimal sensor characteristics, in the bracelet, we opted for porous conductive foam. In the first experiment, 6 mm-thick polyurethane foam (Warmbier 4451.W [35]) was used; in the second, we opted for thinner (3 mm), but stiffer and thus more robust polyolefin foam (Polyform PE PF554 [36]).

Each sensor board in the tactile bracelet had 4 × 8 tactile pixels (taxels) in a 5-mm grid that were sampled using two 16-channel 12-bit resolution analog-to-digital converters (ADCs) in parallel (Figure 3). The digitalized data were provided internally on an Serial Peripheral Interface (SPI) bus. With a board width of 20 mm, an adult can typically fit all 10 sensor boards around the circumference of the arm, resulting in a high resolution tactile image of 10 × 4 × 8 = 320 taxels. The data of the bracelet is streamed out over USB using virtual serial port communication (USB-CDC). The tactile data from the bracelet were captured at 80 frames per second. The host software rearranges the stream of incoming taxel data and can visualize it as a two-dimensional tactile array. A wide double-sided hook-and-loop band was selected for mounting the sensor and data collector boards around an arm. This made quick individual sensor positioning and overall bracelet circumference and tension adjustment possible, while simultaneously providing a sturdy attachment. Appropriate fasteners were designed and 3D printed to attach the printed circuit boards to the hook-and-loop band.

2.1.2. Visual Stimulus

Each participant was asked to try to mimic a sequence of wrist/hand/finger activations performed by a realistic 3D hand model on a computer monitor. The model anatomically matches a human hand and enforces independent flexion of each finger, as well as the activation of the three DOFs of the wrist, with values normalized between 0 and 1. Each DOF activation rose from 0 to 1 and back with a sinusoidal pattern, realistically simulating the flexion of fingers, the rotation of the thumb, the flexion of the wrist, and so on (for our purposes, positive and negative activations of the wrist were treated as separate activations, e.g., the wrist flexion would be treated as a different activation from the wrist extension, since different muscle groups are required to perform them).

The values of the DOF activations of the visual stimulus were used as the ground truth against which the tactile features were matched. In our study, we only used static hand positions. Data collected during the transition and adaptation phases between one activation and the following were not considered in the analysis. This is an instance of the so-called realistic setup or on-off goal-directed stimulus, already successfully employed, e.g., in [30,37]. This method has the drawback of potentially reducing the precision in the prediction of the intended activations due to the delay required by the subject to adapt; nevertheless, it is an accepted way to associate an intended activation with a specific input signal pattern; in the case of amputees, it actually is the only possible way, since amputees cannot produce a reliable ground truth in principle.

Figure 4a depicts the timing of the visual stimulus, as well as the data recording.

2.2. Participants

Ten able-bodied participants (two left-handed and eight right handed, $27.3\pm 5.66$ years old) and one trans-radial left hand male amputee were recruited for this study. The amputee was 68 years old and underwent surgery in 2007 following a work-related incident involving explosive materials. He reported no usage of any prosthesis whatsoever since the amputation and stated that he could distinctly move each phantom finger without phantom-limb pain. The residual limb was about 20.5 cm long and was in good condition (no skin rash, no abnormal redness, no reported pain or exfoliation). All participants were informed, both in writing and orally, about the procedure and possible risks and gave written informed consent before each experiment began. The experiments were performed according to the World Medical Association (WMA) Declaration of Helsinki and were preliminarily approved by the Work Council of the German Aerospace Center (DLR).

2.3. Experimental Protocol

Each able-bodied participant would sit comfortably in front of the monitor showing the visual stimulus. The experiment consisted of ten repetitions of a sequence of six activations each, namely rotation (i.e., opposition) of the thumb, flexion of the index finger, flexion of the little finger, flexion of the wrist, extension of the wrist and supination of the wrist (Figure 4b). This specific choice of activations was motivated by the following three factors: (a) to experiment with activations that are routinely encountered in daily living; and at the same time, (b) to keep the duration of the experiment reasonably short for both able-bodied and amputated participants; and to select (c) the activations that could correspond to the DOFs available in current commercial multi-fingered prosthetic hands that can connect with prosthetic wrists. In order to produce activations that would require a reasonable amount of force/torque, the participants were instructed to press with their fingers on the bare table (thumb rotation, index flexion and little finger flexion) and to simply flex, extend and supinate the wrist to their limit with the arm lifted from the table. The preliminary results of this experiment, as well as more details on the experimental setup are described in a previous publication [25].

The amputee followed a similar experimental protocol. He was introduced to the experiment, seated comfortably in front of the monitor as shown in Figure 1 and was similarly asked to mimic the visual stimulus. The protocol consisted of three trials, each consisting of, in turn, five repetitions of the same sequence of activations as described previously for the able-bodied participants (Figure 4b). The first trial was performed with the residual limb, i.e., the bracelet was placed on the participant’s residual limb, and he was asked to try to perform the activations seen on the screen with his residual limb; the second trial was performed with his intact limb and was therefore a shortened version of the protocol for able-bodied subjects; lastly, the third trial was again performed with the residual limb. This protocol is motivated, on the one hand, by the need to keep the amputee’s protocol as similar as possible to that of the able-bodied subjects, in order to provide comparable results; on the other hand, we wanted to check if any learning effect would appear between the first and the third trial. The complete experiment lasted about 30 min.

2.4. Data Analysis

Tactile data were acquired from the tactile bracelet at a sampling rate of 80 samples per second; the values of the visual stimulus were synchronized by linearly interpolating the timestamps of the respective data channels. The data from the tactile sensors were filtered with a 1st-order Butterworth bandpass filter with cutoff frequencies at 0.01 and 1 Hz to remove high-frequency disturbances, heart rate and signal drift due to memory effects of the foam. Figure 4c shows a typical pattern obtained for each movement required in the experimental protocol (average of all movements for one able-bodied subject).

Different methods were applied for feature extraction, including Harris corner extraction [38], the structural similarity index [39] on bicubic interpolated data and Region of Interest (RoI) gradients [30,40,41], which yielded the highest classification accuracies in a preliminary round of experiments. As opposed to the RoIs used in [30,40], which were round-shaped and overlapped one another by about 10%, in this case, due to the lower resolution of the tactile data with respect to the ultrasound images used in those papers, we adopted a simpler strategy, defining each RoI as a non-overlapping 4 × 4 taxel square. Then, like in the aforementioned papers, for each RoI, we computed three parameters of interest $\alpha ,\beta$ and $\gamma$ linearly approximating the taxel intensities. (More in detail, for each RoI i, ${\alpha }_i$ and ${\beta }_i$ represent the mean intensity gradient along the x and y axis, while ${\gamma }_i$ is an intensity offset. Further details can be found in [30].) The feature extraction method was uniform for all subjects and, once again, in line with previous references, was not targeted at any anatomical feature. This reduces the preparation time of the experiment.

A portion of the collected data was then reduced to three dimensions using principal component analysis (PCA) and visualized for qualitative assessment (see Figure 5, where samples collected during each stimulated movement are visualized in different colors; notice that in some cases, a sample set is not visible since its cluster gets overshadowed by other ones; that is the case of thumb rotation in Figure 5e for instance). The more separated the clusters of data appear in three dimensions, the better are the classification results; in particular, wrist movements are supposed to be more distinguishable from one another in the feature space than when also considering finger movements.

For comparison, graphs in the first column (Figure 5a,c,e) show the full set of movements, while those in the second column (Figure 5b,d,f) show only the wrist movements. Figure 5a,b show data gathered from the able-bodied subject with the most separable classes (Fisher’s separateness index); Figure 5c,d show those obtained from the able-bodied subject least separable classes; and Figure 5e,f show those obtained from the amputee during the third trial.

The qualitative examination of these graphs indicated that the wrist movements appeared well separated in all cases (even for the worst cases) and that the finger movements tended to cluster worse than the wrist movements. Furthermore, the data collected from the amputee during the third trial were hardly distinguishable from those of the able-bodied subjects.

Given this analysis, we chose to use two very simple classification methods, namely a k-Nearest-Neighbors classifier (k-NN) using the Euclidean distance and the Nearest-Cluster-Centroid classifier (NCC) using both the Euclidean and the Mahalanobis distances. The choice of k-NN and NCC was substantiated by their simplicity (in comparison to Artificial Neural Networks (ANN)) and fast training, which can be easily implemented for on-line analysis. For the k-NN classifier, we chose $k=1$, which had the highest accuracy results across k-values between 1 and 10. The classes to be discriminated were either the seven movements (thumb rotation, index flexion, flexion of the little finger, wrist flexion, wrist extension, wrist supination and rest) or the four wrist movements (wrist flexion, wrist extension, wrist supination and rest). Since we had 20 RoIs on each tactile image and three features were extracted from each RoI, each image sample was represented by a 60-dimensional feature vector.

For the first experiment, only the last five repetitions of the movement sequence were used for further analysis (the first prototype of the tactile bracelet with a soft foam was more susceptible to the memory effect of the foam). The three selected classifiers (k-NN, NCC with Euclidean distance and NCC with Mahalanobis distance) were trained using leave-one-repetition-out cross-validation for each subject. The training set was composed of four repetitions and tested on the last repetition. As the resting position was performed more often than the other movements, the mean and standard deviation of the balanced accuracy were computed for each subject.

In the second experiment, the same classifiers used in the first experiment were applied on the three trials performed by the amputee. For each trial, leave-one-repetition-out cross-validation was applied, using 4 repetitions as the training set and one as the testing set. The mean and standard deviation of the balanced accuracy were calculated for each trial.

3. Results

3.1. Experiment #1 (Able-Bodied Subjects)

Figure 6 shows in the first row the classification accuracy obtained in the first experiment. Figure 6a shows that the k-NN and the NCC with Euclidean distance outperformed the NCC with Mahalanobis distance in the classification of all movements for all able-bodied subjects. Only in three cases (Subjects 6, 7 and 10) did the k-NN yielded lower accuracies than the NCC with Euclidean distance. The accuracy of the k-NN varied between 70.40% and 100% with a mean accuracy of 89.15% over all subjects. To know if the difference in the performance of the classifiers is significant, Kruskal–Wallis one-way analysis of variance was performed as the mean accuracies of the different classifiers are not normally distributed. A significant difference was obtained between the three classifiers (H(30) = 12.26, $p=0.0022$). For further analysis, a multiple comparison test was performed showing that the accuracy means over all able-bodied subjects using NCC with Mahalanobis distance are significantly different than the accuracy means over all subjects using k-NN or NCC with Euclidean distance. However, there is no significant difference between the results using k-NN and NCC with Euclidean distance.

The classification of wrist movements, including the resting position, showed high accuracies overall (Figure 6b). The k-NN seems to perform better than the other classifiers, obtaining an accuracy between 95.13% and 100%. The mean accuracy over all subjects was 98.44% for k-NN, 97.77% for NCC with Euclidean distance and 94.53% for NCC with Mahalanobis distance. However, the Kruskal–Wallis one-way analysis of variance showed in this case that there is no significant difference in the performance of the different classifiers (H(30) = 2.41, $p=0.3$).

3.2. Experiment #2 (Amputee)

The second row of Figure 6 shows the classification accuracy obtained in the second experiment, i.e., on the data obtained from the amputee. Figure 6c shows that the highest accuracies applied to all movements were obtained in the third trial, which was performed with the amputated arm. The mean accuracy of the k-NN was 84.37% in Trial 1 and 84.40% in Trial 2. The same classifier achieved 93.07% mean accuracy in the third trial.

Similarly to the results obtained from the able-bodied subjects, the classification of the wrist movements was more accurate (Figure 6d). In this case, the NCC with Mahalanobis distance performed better in the first trial, but worse than the other classifiers considered in Trials 2 and 3. The mean accuracy results of the k-NN for the intact arm in Trial 2 were 100% and 99.18% for the amputated arm in Trial 3. For this experiment, no statistical analysis was performed as we only collected data from one amputee.

4. Discussion

In this paper, we have proposed a further investigation into the usage of Tactile Myography (TMG) for dexterous myocontrol, as a substitute or a companion to surface EMG. A few remarks are necessary.

4.1. Tactile Myography for Myocontrol

Myocontrol is the usage of bodily signals related to muscle activation to control assistive or rehabilitation devices, prostheses being one of the paradigmatic examples [10]. As happens with machine learning (ML) in general, ML-based myocontrol will work reliably only if repeatable and distinct signal patterns for each desired action can be determined [42]. To this aim, the limitations of sEMG are well known in the scientific community (see, e.g., [12]): besides being prone to problems affecting the physical nature of the sensors (e.g., electrode displacement and lift-off), electromyography suffers from the changes in the signal, which can be observed whenever muscle fatigue appears and/or due to sweating, which alters the conductivity of the skin.

All in all, a question arises whether TMG is a valid replacement or a companion to sEMG. It seems reasonable to claim that detecting deformations of the residual limb via pressure (using TMG and its low-resolution counterpart, FMG) is not affected by conductivity of the skin or by fatigue, both inherent problems to the very nature of electromyography [20,23]. The other problems remain on the table, bound to the inevitable necessity that pressure/force sensors stay in place and in contact with the skin. Besides this, force/pressure sensing enables the detection of contact forces, accelerations and orientations, as well as the deformations in the subject’s body, which are due to skeletal structures other than muscles (e.g., tendons). Such factors could be problematic for the detection of activation patterns, but could also represent further valuable information to be exploited.

A thorough study comparing sEMG and TMG/FMG both from the point of view of the signals generated and as a means for myocontrol, including the usage of either/both techniques in real life, has yet to appear. Nevertheless, there are strong hints pointing at the feasibility of this marriage: the potential is great. In particular, in [23], the signals of FMG and sEMG were qualitatively and quantitatively compared, showing clearly a higher stability in time (less oscillatory behavior), and that FMG signals are better separated in the input space. Furthermore, in [24], an extensive study on four trans-radial amputees was performed, showing that eight FSR sensors suffice to reach classification accuracy values that are comparable with the standard found in the literature for sEMG. Notice, anyway, that both aforementioned studies employ FMG rather than TMG, that is, they rely on a few force-sensing resistors scattered on the forearm of able-bodied subjects or embedded inside a socket for amputees. On the other hand, as far as proper TMG is concerned, Radmand et al. [26] have already shown that TMG “footprints” or “images” can be used to classify a relatively high number of hand and wrist activations with remarkable precision. Their experiment was performed on 10 able-bodied subjects, using a tactile sensor that has roughly half the density as the one we present, and is rigid.

A direct comparison of sEMG and TMG in conditions similar to those described here is not available at the moment as far as we know; but in [43], such a comparison/combination of techniques during an online task was carried out, in which TMG was shown to outperform sEMG.

4.2. What This Study Shows

To the best of our knowledge, this is the first study in which TMG (as opposed to FMG) has been applied to an amputee, albeit for a short time and in lab-controlled conditions. The results however show that high rates of classification can be obtained; moreover, in very similar conditions from a pool of able-bodied subjects, our study showed that the amputee’s results were comparable to those of the able-bodied subjects. In particular, the tactile bracelet we have presented, having 320 taxels with a spatial resolution of 5 mm, was adapted to the forearm of 10 able-bodied subjects, as well as to the residual limb of a left trans-radial amputee. It is worthwhile to note that we employed no signal windowing at all: we used the signals as they came out of the device, with a first order bandpass filter; therefore, the delay between the data acquisition and the feature extraction was negligible. On top of this, the optimal combination of features and classifier is the simplest and fastest one, which lets us claim that an online prediction would work with a barely noticeable delay between the intent of the subject and the prediction itself. Of course, this is the subject of future assessment as we have no online study so far.

Notice that the experiment we have reported about in this paper cannot yet indicate how well TMG would work in practice. The accuracy results obtained here might be insufficient; even values around 98% would in some cases not be enough [6]. Moreover, the experimental protocol only involved standard movements (i.e., no daily-living activity tasks), offline analysis and no additional weight on the forearm when performing the movements, as would be the case while grasping. All tests were performed on five repetitions only of the selected actions, which is hardly a good measure of their variability in practice; this could also have influenced the choice of the best feature/classifier combination.

Engaged in a simple, repetitive experiment involving muscle activations, both able-bodied subjects and a trans-radial amputee produced corresponding tactile patterns, which led to a simple classifier to obtain high accuracy percentages, ranging from 70%–100% across all trials. These values are in line with, or even superior to, those found in the literature while using other techniques, among which is the traditional choice for myocontrol, sEMG (see, e.g., [15] for an offline comparison among such human-machine interfaces). In an excellent recent survey by Fang et al. [44], one sees that classification results obtained from offline data gathered from a similar population of amputated subjects (that is, trans-radials) hover in the best cases from 90% to close to 100%. Such numbers are as well in agreement with the already mentioned articles by Cho et al. [24] and Radmand et al. [26]. A remark is necessary here though: one must be careful with such results, since it is well known that off-line abstract performance rarely translates to practical usability [45]. See the Future Work subsection for more about this.

We were able to distinguish single-finger activations and even discern thumb rotation, which is even more interesting, since the muscles operating this activation are inside the hand. We speculate that synergistic activations of non-intrinsic muscles might be responsible for this surprising result; in the amputated subject, spontaneous re-innervation could also play a role. Let us remark that each residual limb is unique, therefore high-resolution approaches such as TMG could definitely help identify and explore the spots with the highest remaining activity.

The comparison of features and classification methods we performed show that the simplest features and classifier works best. This is also visually clear if one inspects the PCA-reduced patterns of Figure 5. Not surprisingly, wrist movements were easier to classify, whereas introducing finger movements decreased the classification accuracy. The results also show that the principle enforced by the tactile bracelet works to a good extent in the case of an amputated subject. To the best of our knowledge, this is the first attempt to assess the usefulness and feasibility of TMG including an amputated subject. Moreover, the amputated subject’s patterns are qualitatively indistinguishable from those produced by the able-bodied subjects (Figure 5). The results suggest a learning effect (Figure 6 second row, improving from about 84%–92%) in the case of the amputated subject.

4.3. Final Remarks and Future Work

In this work, we have been able to test only one amputated subject. We then put forward two remarks about his results, without any pretense to statistical validity. On the one hand, our subject had not been exposed to any such experiment before and used no prosthesis whatsoever since the operation, which happened eight years before the experiment. Given the long time of no usage of the muscles in the residual limb and the fact that he was not familiar with prosthetic devices, we find his performance surprising. On the other hand, he claimed that he could feel and move his phantom fingers, which might have helped him produce the required distinct movements. An analysis on more amputated subjects, together with a screening of their physiological characteristics, must be performed in order to get a clearer picture.

Regarding TMG in itself, a long-term evaluation is necessary to gather conclusive information about the durability of the proposed tactile sensor system and the effects of sweating on the material over longer time periods. TMG, which is high-resolution force sensing, provides many signals gathered from a relatively small surface and may at first sight seem highly redundant; but one must imagine, in the final setup, that TMG would be applied on a residual limb whose muscle morphology is, in general, unknown. That is where, we believe, the high resolution of TMG could help gain a better accuracy and reliability with respect to FMG. Of course in that case, the control system must be apt to work seamlessly with highly dimensional samples.

We argue that even more importantly, it is necessary to develop a novel way of assessing the feasibility of an approach, be it mono- or multi-modal, which takes into account the complexity of the situations an amputee encounters in real life. Each situation potentially induces changes in the signals recorded by the interface, which must be taken into account [46]. Therefore, a specific training-and-testing protocol must be devised, through which the effectiveness and usefulness of novel approaches can be seriously assessed. Given this requirement, our main line of future research goes in the direction of determining the optimal set of sensors through which to realize an ever-improving myocontrol.