2.3. Experimental Protocol
Each able-bodied participant would sit comfortably in front of the monitor showing the visual stimulus. The experiment consisted of ten repetitions of a sequence of six activations each, namely rotation (i.e., opposition) of the thumb, flexion of the index finger, flexion of the little finger, flexion of the wrist, extension of the wrist and supination of the wrist (Figure 4
b). This specific choice of activations was motivated by the following three factors: (a) to experiment with activations that are routinely encountered in daily living; and at the same time, (b) to keep the duration of the experiment reasonably short for both able-bodied and amputated participants; and to select (c) the activations that could correspond to the DOFs available in current commercial multi-fingered prosthetic hands that can connect with prosthetic wrists. In order to produce activations that would require a reasonable amount of force/torque, the participants were instructed to press with their fingers on the bare table (thumb rotation, index flexion and little finger flexion) and to simply flex, extend and supinate the wrist to their limit with the arm lifted from the table. The preliminary results of this experiment, as well as more details on the experimental setup are described in a previous publication [25
The amputee followed a similar experimental protocol. He was introduced to the experiment, seated comfortably in front of the monitor as shown in Figure 1
and was similarly asked to mimic the visual stimulus. The protocol consisted of three trials, each consisting of, in turn, five repetitions of the same sequence of activations as described previously for the able-bodied participants (Figure 4
b). The first trial was performed with the residual limb, i.e., the bracelet was placed on the participant’s residual limb, and he was asked to try to perform the activations seen on the screen with his residual limb; the second trial was performed with his intact limb and was therefore a shortened version of the protocol for able-bodied subjects; lastly, the third trial was again performed with the residual limb. This protocol is motivated, on the one hand, by the need to keep the amputee’s protocol as similar as possible to that of the able-bodied subjects, in order to provide comparable results; on the other hand, we wanted to check if any learning effect would appear between the first and the third trial. The complete experiment lasted about 30 min.
2.4. Data Analysis
Tactile data were acquired from the tactile bracelet at a sampling rate of 80 samples per second; the values of the visual stimulus were synchronized by linearly interpolating the timestamps of the respective data channels. The data from the tactile sensors were filtered with a 1st-order Butterworth bandpass filter with cutoff frequencies at 0.01 and 1 Hz to remove high-frequency disturbances, heart rate and signal drift due to memory effects of the foam. Figure 4
c shows a typical pattern obtained for each movement required in the experimental protocol (average of all movements for one able-bodied subject).
Different methods were applied for feature extraction, including Harris corner extraction [38
], the structural similarity index [39
] on bicubic interpolated data and Region of Interest (RoI) gradients [30
], which yielded the highest classification accuracies in a preliminary round of experiments. As opposed to the RoIs used in [30
], which were round-shaped and overlapped one another by about 10%, in this case, due to the lower resolution of the tactile data with respect to the ultrasound images used in those papers, we adopted a simpler strategy, defining each RoI as a non-overlapping 4 × 4 taxel square. Then, like in the aforementioned papers, for each RoI, we computed three parameters of interest
linearly approximating the taxel intensities. (More in detail, for each RoI i
represent the mean intensity gradient along the x
is an intensity offset. Further details can be found in [30
].) The feature extraction method was uniform for all subjects and, once again, in line with previous references, was not targeted at any anatomical feature. This reduces the preparation time of the experiment.
A portion of the collected data was then reduced to three dimensions using principal component analysis (PCA) and visualized for qualitative assessment (see Figure 5
, where samples collected during each stimulated movement are visualized in different colors; notice that in some cases, a sample set is not visible since its cluster gets overshadowed by other ones; that is the case of thumb rotation in Figure 5
e for instance). The more separated the clusters of data appear in three dimensions, the better are the classification results; in particular, wrist movements are supposed to be more distinguishable from one another in the feature space than when also considering finger movements.
For comparison, graphs in the first column (Figure 5
a,c,e) show the full set of movements, while those in the second column (Figure 5
b,d,f) show only the wrist movements. Figure 5
a,b show data gathered from the able-bodied subject with the most separable classes (Fisher’s separateness index); Figure 5
c,d show those obtained from the able-bodied subject least separable classes; and Figure 5
e,f show those obtained from the amputee during the third trial.
The qualitative examination of these graphs indicated that the wrist movements appeared well separated in all cases (even for the worst cases) and that the finger movements tended to cluster worse than the wrist movements. Furthermore, the data collected from the amputee during the third trial were hardly distinguishable from those of the able-bodied subjects.
Given this analysis, we chose to use two very simple classification methods, namely a k-Nearest-Neighbors classifier (k-NN) using the Euclidean distance and the Nearest-Cluster-Centroid classifier (NCC) using both the Euclidean and the Mahalanobis distances. The choice of k-NN and NCC was substantiated by their simplicity (in comparison to Artificial Neural Networks (ANN)) and fast training, which can be easily implemented for on-line analysis. For the k-NN classifier, we chose , which had the highest accuracy results across k-values between 1 and 10. The classes to be discriminated were either the seven movements (thumb rotation, index flexion, flexion of the little finger, wrist flexion, wrist extension, wrist supination and rest) or the four wrist movements (wrist flexion, wrist extension, wrist supination and rest). Since we had 20 RoIs on each tactile image and three features were extracted from each RoI, each image sample was represented by a 60-dimensional feature vector.
For the first experiment, only the last five repetitions of the movement sequence were used for further analysis (the first prototype of the tactile bracelet with a soft foam was more susceptible to the memory effect of the foam). The three selected classifiers (k-NN, NCC with Euclidean distance and NCC with Mahalanobis distance) were trained using leave-one-repetition-out cross-validation for each subject. The training set was composed of four repetitions and tested on the last repetition. As the resting position was performed more often than the other movements, the mean and standard deviation of the balanced accuracy were computed for each subject.
In the second experiment, the same classifiers used in the first experiment were applied on the three trials performed by the amputee. For each trial, leave-one-repetition-out cross-validation was applied, using 4 repetitions as the training set and one as the testing set. The mean and standard deviation of the balanced accuracy were calculated for each trial.