A Self-Learning and Adaptive Control Scheme for Phantom Prosthesis Control Using Combined Neuromuscular and Brain-Wave Bio-Signals ^†

Abstract

The control scheme in a myoelectric prosthesis includes a pattern recognition section whose task is to decode an input signal, produce a respective actuation signal and drive the motors in the prosthesis limb towards the completion of the user’s intended gesture motion. The pattern recognition architecture works with a classifier which is typically trained and calibrated offline with a supervised learning framework. This method involves the training of classifiers which form part of the pattern recognition scheme, but also induces additional and often undesired lead time in the prosthesis design phase. In this study, a three-phase identification framework is formulated to design a control architecture capable of self-learning patterns from bio-signal inputs from electromyography (neuromuscular) and electroencephalography (brain wave) biosensors, for a transhumeral amputee case study. The results show that the designed self-learning framework can help reduce lead time in prosthesis control interface customisation, and can also be extended as an adaptive control scheme to minimise the performance degradation of the prosthesis controller.

1. Introduction

Bionic prosthesis arms, also known as a myoelectric prosthesis, are a kind of functional-based prosthesis arm which has been seen to be the closest alternative to a biological limb [1]. The myoelectric prosthesis works with a control system whose role is, given a physiological signal acquired by a biosensor, to decode a user motion intent, and from this, apply an actuation signal to make the myoelectric limb perform a selected gesture motion [1]. As per Fougner et al. [2], a number of control schemes have been trialled for the control of the myoelectric limb, and the pattern recognition scheme has been viewed as the favoured control method due to the level of intuitiveness which it affords its users. An artificial intelligence classifier is typically employed as part of this exercise and is usually calibrated using the favoured supervised learning framework, which is based on an iterative optimisation sequence where an algorithm learns the best internal model configuration that allows it to best identify labelled examples of the dataset for which it is being calibrated [3]. The shortcomings of the supervised learning framework include the lag-time induced by the reliance on an ‘expert in loop’ for tasks such as labelling of the training set, which requires more expert knowledge depending on the format and size of the training data, in addition to associated computational training times [3].

As part of efforts towards overcoming this limitation posed by the classifier learning framework, in this study, we propose an automated three-phase identification method which is used towards the design of a self-learning prosthesis controller. Another key issue faced by myoelectric prosthesis users is the classifier degradation, and thus, inaccurate gesture intent decoding due to a range of factors such as electrode shifts, changes in stump, sensor-skin impedance variation and sensor drifts [4]. This issue forms part of what has caused myoelectric prosthesis users to begin to abandon their prosthesis limbs as per reported surveys [4]. It is foreseen that the self-learning control framework, if applied at various intervals, can be used to recalibrate the prosthesis controller when necessary, and help form an adaptive prosthesis controller, thereby mitigating the classifier degradation problem. Thus, in this study, we investigate the design of a multi-stage self-learning prosthesis controller capable of functioning in an automated fashion and reducing downtime associated with the calibration of the controller while serving as an adaptive strategy to retune the classifier within the controller to minimise classifier degradation.

2. Materials and Methods

This paper utilises the dataset acquired by Li et al. [5] using electromyography (EMG) and electroencephalography (EEG) sensors. This section gives a brief overview of the bio-sensors used, and the data collection procedure, followed by the designed prosthesis controller framework [5].

2.1. Biosensors

2.1.1. Electromyography (EMG)

EMG signals are superimposed electrical signal representations of action potentials from motor neurons which are dependent on the physiological and anatomical properties of an individual [3,5]. Using dipole theory, an EMG signal can be mathematically modelled as a continuous extracellular action potential from a multiple-source dipole, as seen in equation 1 [6]:

$${\varphi }_e\left(t\right)=-\frac{a^2.{\sigma }_i}{4.{\sigma }_e}.{{\displaystyle \int }}_{-\infty }^{+\infty }\frac{\partial IAP\left(x,t\right)}{\partial x}.a_x^{-}.\frac{\partial }{\partial x}\left(\frac{1}{r\left(x\right)}\right)dx$$

(1)

where ${\varphi }_e$ is the time-varying extracellular potential, ${\sigma }_e$ is the conductivity of the extracellular medium,${\sigma }_i$ is the intracellular conductivity, $a$ is the radius of the fiber, $t$ is time, $r$ is the distance of the source excitation to the recording sensor, $x$ is a point in space within the fibre element, $a_x^{-}$ is the length of the anatomical fiber and $\frac{\partial IAP}{\partial x}$ is the dipole strength at a point along the fiber axis.

-

EMG Sensors

The EMG instrumentation used for data acquisition by Li et al. [5] was the Refa 128 high-density electrodes by TMS International BV, Netherlands, with 32 electrode channels [3]. The acquisition electronics comprised a bandpass filter in the 10–500 Hz frequency range, 24-bit resolution and a sample rate of 1024 Hz [3].

2.1.2. Electroencephalography (EEG)

The passive variant of EEG which was used for data collection by Li et al. [5] is based on the recording of electrical signals emanating from the brain using surface electrodes, where the recorded EEG signals represent brain-based potential signals from various regions of activations [7]. On a closer scale, these EEG signals occur from the neuronal firing of billions of pyramid-like cells within the skull of a human being [7]. Using a combination of dipole theory, and assuming the forward EEG problem, a measured potential of an EEG signal can be formulated as follows [7]:

For a multilayer head model with concentric spheres L whose radius spans 0 < r₁ < r₂… < r_L with anisotropic conductivities ${\sigma }_1,\dots .{\sigma }_L$, the electric potential u measured at point $x$ can be expressed as Equation (2) [7]:

$$u\left(r_s,q,x\right)=\frac{\left|\left|q\right|\right|}{4\pi {\sigma }_L{r}_L^2}={{\displaystyle \sum }}_{n=1}^{\infty }\frac{2n+1}{n}{\left(\frac{r_s}{r_L}\right)}^{n-1}{f}_n\left[ncos\propto P_n\left(cos\gamma \right)+cos\beta sin\propto P_n^1\left(cos\gamma \right)\right]$$

(2)

where s is the dipole source located within a proximity of sphere of radius r_s of moment q; the boundary sphere is r_L; ${\sigma }_L$ is the anisotropic conductivity within the boundary sub-domain of L; $f_n$ is the EEG measurement for the nth element in the infinite set; $\propto$ is the angle between the point S and measurement point x; $\gamma$ is the angle between two planar vectors pairs of S and q, and S and x; and $P_n$ and $P_n^1$ represent the Legendre polynomial coefficients of the series.

-

EEG Sensors

Li et al. [5] used the 64-sensor EEG channel EasyCap, Herrsching, Germany, with the Al-AgCl electrodes and Neuroscan system version 4.3. The EEG signals were band-passed using filters designed with the frequency range of 0.05–100 Hz and acquired at a sampling rate of 1024 Hz [5].

2.2. Data Collection and Demographic Information

The dataset acquired by Li et al. [5] involved a simultaneous acquisition of EMG and EEG from amputated subjects whom had lost their limb due to trauma. A single amputee’s data set has been used for the work presented as part of this paper. The transhumeral amputee subject was 49 years old with a left-side amputation, three years post-amputation due to trauma, and a stump length (measured from shoulder blade downwards) of 20 cm. Ethical approval was granted for the study by the Institutional Review Board of Shenzhen Institutes of Advanced Technology, with a unique reference number of SIAT-IRB-150515-H0077.

The Hand Open (HO), Hand Close (HC) gestures, which represent two of the key gesture sets relevant for a functional prosthesis, are the gestures used for part of the prosthesis control exercises presented in this work [5]. A total of 32 EMG electrode channels were distributed around the stump and deltoid of the subject, while the EEG cap comprising 64 electrodes was worn on his head. A sum of 10 repetitions was done for each gesture set with a break between acquisition runs to minimise the effect of fatigue on the data collection process; an image of the acquisition setup can be seen in Figure 1.

2.3. Self-Learning Controller Framework

This section describes the steps and architecture of the self-learning controller which, after the acquisition of a biosignal, includes separate electrode channel selection, and the automated learning consisting of Feature Extraction, Signal Fusion, Dimensionality Reduction, Labelling and Motion Intent Classification.

-

Electrode Channel Selection: The electrode channel selection was carried out separately as part of the framework being described in this paper. This aspect was done by Li et al. [5], and involved the use of the Sequential Forward Selection (SFS) algorithm which is a variant of a greedy search algorithm, as can be seen in Equation (3):

$$\mathrm{Acc}\left(S_k+x^{*}\right) =\mathrm{argmax} \mathrm{Acc}\left(S_k+x_j^{*}\right), \mathrm{k}\in \{1,2,\dots .\mathrm{n}\}$$

(3)

where Acc represents classification accuracy, $S_k$ is a selected electrode, $x_j^{*}$ is an electrode channel and jth element in the set k. Using this framework, the 10 electrodes from both the EMG and EEG biosensors were identified by Li et al. [5]. These electrodes formed the signal channels used for the signal processing stage and represent a first-stage dimensionality reduction process.

2.3.1. Feature Extraction and Feature Vector Fusion

A total of four time-domain features were extracted from the signals of both the EMG and EEG as proposed by Li et al., as described in Equations (4)–(7) [3,5]:

$$\mathrm{Mean} \mathrm{absolute} \mathrm{value} \left(\mathrm{MAV}\right)=\frac{1}{N}{{\displaystyle \sum }}_{n=1}^N\left|x_n\right|$$

(4)

where N = number of samples and $x_n$ is the nth sample of the EMG signal.

$$\mathrm{Waveform} \mathrm{Length} \left(\mathrm{WL}\right)={{\displaystyle \sum }}_{n=2}^N\left|x_n-x_{n-1}\right|$$

(5)

$$\mathrm{Zero} \mathrm{Crossing} \left(\mathrm{ZC}\right)={{\displaystyle \sum }}_{n=1}^{N}sgn\left(-x_i{x}_{i+1}\right)Sgn\left(x\right)=\{\begin{array}{c}1, x>0\\ 0, otherwise\end{array}$$

(6)

$$\mathrm{Number} \mathrm{of} \mathrm{slope} \mathrm{sign} \mathrm{changes} \left(\mathrm{SSC}\right)={{\displaystyle \sum }}_{n=2}^{N}f[\left[\left(x_n-x_{n-1}\right).\left(x_n-x_{n+1}\right)\right]]Sgn\left(x\right)=\{\begin{array}{c}1, x>0\\ 0, otherwise\end{array}$$

(7)

A threshold of 0.1 mV was used for the calculations of the ZC and SSC features, and following the extraction of the features, respective feature vectors were formed for both the EMG and EEG, and then fused together to construct an EMG-EEG feature vector. The 10 repetitions from 10 SFS electrodes for the two gestures performed, followed by the four features extracted, amounted to a total of 800 data points used for the classification process in the case of the EMG/EEG only, and 1600 for the fused EMG-EEG biosensing modules.

2.3.2. Dimensionality Reduction

Dimensionality reduction is a useful approach that involves the reduction of the size of the feature vector, which in turn, helps minimise the computation time and remove noise and redundancies from the dataset [8]. Principal component analysis (PCA) is a linear dimensionality reduction technique used to reduce the dimension of data while preserving its structure and minimising loss of information, predominantly by creating new variables that maximise variance [8]. This approach obtains a number of principal components (PCs) by solving an eigenvector and eigenvalue problem without the requirement of prior information or class labels, thus making it an unsupervised data analysis method [9].

Using the computationally efficient covariance method, the PCA can be calculated as follows, assuming a data vector $X$= $x_1,x_2\dots ..x_N$:

-

Mean centring of the data to produce a standardised vector B

-

Calculation of the covariance matrix $\mathit{C}=\frac{1}{n-1}\mathit{B}\ast \mathit{B}$

where $C$ is the covariance matrix, * is the conjugate transpose operator, and $n-1$ is used in this case due to Bessel’s correction factor used to negate the effect of bias on sample variance.

-

Calculation of the eigenvalues and eigenvectors of the covariance matrix which produces a diagonal of the covariance matrix $\mathit{C}$, which can be formulated as ${\mathit{V}}^{-1}\mathit{C}\mathit{V}=\mathit{D}$, where $\mathit{D}$ represents the eigenvalues of the covariance matrix and $\mathit{V}$ is the matrix of the right-side eigenvalues.

-

Arrange eigenvalues and eigenvectors in descending order and calculate the energy $E_j={{\displaystyle \sum }}_{k=1}^N{D}_{kk}$ for all columns in the feature vector.

-

Truncate the eigenvectors whilst ensuring that 90% of the cumulative energy is preserved, and project the feature vector in a new basis $\mathit{G}=\mathit{B}\ast \mathit{W}$, where the columns in G represent PC’s $1, 2,\dots .n$.

In this work, the first two PC’s were selected as they accounted for 95% of the information in the data, thus allowing a dimensionality reduction from 4 × 200 to 2 × 200 for the EMG-Only and EEG-Only, while most notably dropping the dimension from 8 × 200 to 2 × 200 for the case of the signal fusion of EMG-EEG.

2.3.3. Iterative Clustering and Motion Intent Classification

The final stage involves the labelling and forming of clusters as a means of distinguishing between various phantom motion intent signals used to actuate respective hand gestures in the prosthesis limb. For this section, two unsupervised classification methods were used as follows:

-

K-means: Is a form of iterative clustering algorithm where the data are segmented into K different classes using a centroid mean and Euclidean distance metric. During iterations, the algorithm aims to maximise the distance between classes, and sorts data points into their respective classes by their proximity to assigned clusters within Euclidean space [10]. This approach uses the expectation-maximisation (E-M) framework assuming a random initialisation: The E step involves the assignments of clusters using ${{\displaystyle \sum }}_{i=1}^m{{\displaystyle \sum }}_{k=1}^{K}argmi{n}_j{\left|\left|x^i-{\mu }_k\right|\right|}^2$, where $x^i$ is a data point and ${\mu }_k$ is the centroid mean; while the M step involves the recalculation of the class centroid using the expression ${\mu }_k=\frac{{{\displaystyle \sum }}_{i=1}^m{w}_{ik}{x}^i}{{{\displaystyle \sum }}_{i=1}^m{w}_{ik}}$, where $w_{ik}$ is a binary metric used to indicate whether or not a data point belongs in a certain class [10]. Due to the random cluster centroid initialisation, running the K-means algorithm at different times could yield different results; thus, a model selection phase has been included where the model selected was that which produced the lowest error for the performance index J defined in equation 8 after five separate runs of the algorithm. The number of clusters was defined a priori from the number of gesture motions performed.

$$J=|\left(Number of motion repetitions performed*Number of electrode channels\right)-{{\displaystyle \sum }}_{i=1}^m{x}_k^i|$$

(8)

where $x_k^i$ is a data point assigned to a specific class k

-

Gaussian mixture model (GMM): Working with a Gaussian assumption, the GMM is a probabilistic framework which is an extension of the K-means algorithm, with the GMM providing flexibility between a hard clustering option which sorts the data into a solitary class, while the soft clustering allows for data to belong to more than one class [11]. The GMM model can be described and parametrised as containing a mixture proportion, mean and covariance. A multidimensional model of the GMM framework can be seen in Equations (9) and (10):

$$p\left(\overrightarrow{x}\right)={\displaystyle \sum }_{i=1}^K{\pi }_{i}N(\overrightarrow{x}|\overrightarrow{{\mu }_i},{{{\displaystyle \sum }}^{ }}_i)$$

(9)

$$N\left(x|{\mu }_i,{{{\displaystyle \sum }}^{ }}_i\right)=\frac{1}{\surd {\left(2\pi \right)}^K|{{{\displaystyle \sum }}^{ }}_i|}\exp (-\frac{1}{2}{\left(\overrightarrow{x}-\overrightarrow{{\mu }_i}\right)}^T{{{\displaystyle \sum }}^{ }}_i-1\left(\overrightarrow{x}-{\overrightarrow{\mu }}_i\right))$$

(10)

$${\displaystyle \sum }_{i=1}^K{\pi }_i=1$$

(11)

where $\overrightarrow{x}$ is a datapoint,${\mu }_i$ is the mean, ${{{\displaystyle \sum }}^{ }}_i$ is the covariance, $N$ symbolises a Gaussian distribution, k is the number of mixture components, and ${\pi }_i$ are the various component weights with a normalisation constraint as shown in equation 11 to ensure the total probability sums up to 1. The learning model for the GMM is also based on the E-M framework where the model parameters are estimated iteratively with the maximum likelihood estimation approach [11]. The hard clustering option was utilised in this work and, as with the K-means algorithm, the number of clusters and mixture components were defined a priori, and included a model selection phase as mentioned with the performance index defined in equation 8 used for the selection process. The algorithm was run with a full covariance option and converged within 15 iterations on average.

A diagrammatic flow of the self-learning control framework can be seen in Figure 2.

3. Results

3.1. Intent Decoding

As a way of testing the intent decoding capability of models, a separate unseen dataset of the feature vector after the PCA transformation was put aside, used to test the models and referred to as the Hold-Out dataset. The results of the various cluster assignment runs from the Hold-Out can be seen in

Table 1. Results of the cluster assignment exercises and model accuracy validation using a Hold-Out test set (where highlighted figures represent the selected models on which the Hold-Out test was validated).

	GMM-EMG Only	K-Means-EMG Only	GMM-EEG Only	K-Means-EEG Only	GMM-EMG-EEG	K-Means- EMG-EEG
Cluster Model 1 Accuracy	83%	81%	64%	63%	68%	83%
Cluster Model 2 Accuracy	99%	81%	64%	58%	98%	83%
Cluster Model 3 Accuracy	99%	81%	64%	58%	98%	83%
Cluster Model 4 Accuracy	99%	81%	64%	58%	98%	83%
Clustering Model 5 Accuracy	99%	81%	64%	58%	70%	83%
Hold-Out Test Accuracy	100%	80%	90%	60%	100%	80%

, where the accuracy has been computed as the Classification Accuracy = $\frac{{{\displaystyle \sum }}^{ }Number of correctly clustered samples}{{{\displaystyle \sum }}^{ }Total number of samples}$ expressed as a percentage.

From the results in Table 1, it can be seen that the designed self-learning framework is capable of automatedly learning the patterns in the data, and thus, the necessary intent decoding task was achieved with high accuracy. From the results, it can be seen that the best accuracy for the Hold-Out testing was jointly between the EMG-Only and EMG-EEG. For the case of the EEG, there was a reduction in the test accuracy by a factor of 10% for the GMM and 20% for the K-means. This is thought to be due to EEG being a biosensor which requires a high number of electrode channels when compared with EMG, and due to this, provided a slightly lower test accuracy. In the case of the signal fusion of EMG-EEG, the test accuracy provided an identical result for both classifiers, as was seen in the EMG only, although it is expected that the fused biosignals would be more robust to uncertainties that could cause misclassification of motion intent signals. Observing the results from both classifiers, it can be seen that GMM outperforms the K-means for every configuration of the bio-signal input. The reason for this is thought to be due to the K-means clustering assumption that the data are spherical, whereas these kinds of bio-signals have been seen to be nonlinear and favour nonlinear class separation. The GMM also takes into account sample covariance as part of the clustering process, as can be seen in equation 10, thereby allowing for a more informed formation of clusters – with the trade-off for this being additional computational requirements relative to K-means, which works purely with Euclidean distance.

Extension towards an Adaptive Control Framework

Due to factors mentioned in Section 1, external factors could make redundant trained class boundaries, and as a result, cause classifier degradation; this warrants the need for classifier relearning to adapt to the dynamic changes which may be causing performance degradation [4]. Researchers in this area, such as Samuel et al., Chen et al., and Asogbon et al. [12,13,14], have proposed extensions to various classifier architectures to allow online adaptation of classifier decision boundaries, as necessary [4]. Here, we propose an extension of the presented self-learning control framework as a means of an adaptive framework for cluster decision cluster reformation with real-time data from current anatomical and acquisition electronic states. As a means of controller adaptation, the self-learning process can be prompted either as an interval-based adaptive solution defined by a user-defined timeframe where the self-learning process is initiated to adapt the control architecture, or after a sequence of misclassifications occurs. Further research is required to investigate and formalise this notion.

4. Conclusions

In this paper, a self-learning controller framework capable of learning patterns from input data in an automated fashion and decoding phantom motion intents has been proposed. The framework comprises a pre-processing stage involving the SFS followed by a three-phase self-learning which includes feature extraction, dimensionality reduction and iterative clustering. The benefits of this can reduce the lag-time associated with the favoured supervised learning method used in current myoelectric prosthesis control architectures, and add a further layer of autonomy to the prosthesis limb. This framework can also be expanded to help mitigate classifier degradation by being a user-prompted relearning framework capable of relearning the class boundary variations in the data resulting from physiological changes and drifts in the acquisition electronics in the prosthesis arm. Subsequent work is now required to validate this approach on a broader number of transhumeral amputees, followed by an optimisation exercise to observe how many unique motion intents can form part of the framework while maintaining an acceptable intent decoding accuracy.