Robust Bio-Signal Based Control of an Intelligent Wheelchair

Abstract

In this paper, an adaptive human-machine interaction (HMI) method that is based on surface electromyography (sEMG) signals is proposed for the hands-free control of an intelligent wheelchair. sEMG signals generated by the facial movements are obtained by a convenient dry electrodes sensing device. After the signals features are extracted from the autoregressive model, control data samples are updated and trained by an incremental online learning algorithm in real-time. Experimental results show that the proposed method can significantly improve the classification accuracy and training speed. Moreover, this method can effectively reduce the influence of muscle fatigue during a long time operation of sEMG-based HMI.

1. Introduction

The deployment of electric powered wheelchairs (EPWs) has been increased rapidly for better quality of life for handicapped and elderly people over the last 20 years [1]. However, most of these EPWs are traditionally controlled by joysticks, which are not suitable for people with severe physical disabilities like spinal cord injury or hemiplegia. Therefore, alternative HMI methods for wheelchair operation have been studied, such as voice recognition [2], eye moment [3], head gesture [4] and biopotential signals interface [5,6,7,8]. Bio-potential signals such as Electrooculargraphic (EOG), electromyographic (EMG) and electroencephalographic (EEG) are also deployed to control such kind of smart service robot [9,10,11]. Compared to other signals, EMG signal is more easily obtained and has less signal interference, especially for the weak groups, such as patients with upper limb amputation and elderly people for which other methods are inconvenient for their daily life.

The EMG signal, which is sampled from the skin’s surface, can directly reflect the movement intention of human body and could be very suitable to serve as the controller. Englehart and Hudgins investigated a continuous segmentation or continuous classification by using different analysis window lengths [12]. Moon used a preset threshold value method to distinct the scapulae muscle signal at fairly high recognition rates [13]. Hiroki Tamura et al. obtained s-EMG signals from facial movements and six patterns were recognized to control the wheelchair movements [14]. Lai Wei combined forehead sEMG signals and color face image information to control the smart robot [5]. However, it suffers from poor reliability and robustness in long-term operations, which were mainly caused by muscle fatigue. These may arise not only because of peripheral changes at the level of the muscle, but also because the central nervous system fails to drive the motoneurons adequately [15]. Usually, the amplitude of the sEMG signals from muscles we used without fatigue is lower than 400 μV. However, after fatigue occurs in muscles, there are significant rises to 600 μV, which can be seen Figure 1. Qin and Mitsuhiro investigate a torque estimation method for muscle fatigue tracking, using stimulus evoked electromyography in the context of a functional electrical stimulation rehabilitation system [16]. However, how to choose the best signal evaluation parameter according to specific muscle condition and method of reducing the influence of muscle fatigue is still not very clear, and has been discussed during recent decades [17,18].

Figure 1. The amplitude of surface electromyography (sEMG) signals.

Figure 1. The amplitude of surface electromyography (sEMG) signals.

To investigate the real time and long time performance of the proposed system in this paper, forehead sEMG signals are first obtained and analyzed by using three dry electrodes sensor. Then, a four-order autoregressive model parameter algorithm is deployed to acquire data compression and extract four coefficients of the signals. An incremental online support vector machine (SVM) is employed to classify sEMG patterns that have the systematic parameters corresponding to the changes of the signals during the running time. Furthermore, the improved incremental training algorithm is presented and applied on the HMI system.

2. The Research Objects and Methods

The sEMG signal controlled intelligent wheelchair should provide a high degree of intuitive and dexterous control, and offer a convenient and high level of performance for disabled people. The human face has plenty of facial muscles to make expressions. The frontalis muscle, located on the front of the head, is one of the largest muscles which can provide continual and distinct, recognizable signals. The main components of the system include sEMG signals collected by electrodes placed on the skin over a user’s forehead muscle, electrodes accompanied by miniature pre-amplifiers to differentiate small signals of interest, signal features detected after amplified, filtered, digitized via standard EMG instruments, and finally transferred to a wheelchair controller. Five subjects voluntarily participated in the experiment and have independent consciousness without facial disease, deformity or trauma. In addition, there was no muscle fatigue before the experiments were performed.

The experimental system consists of an intelligent wheelchair with two independently driven motors, a laptop computer for human-machine interface, and a data acquisition device. A Cyberlink or NIA system is used to acquire the sEMG signals of the subjects. The interface unit has an amplifier with a gain of 50,000, and a filter system provides a band pass filter with a bandwidth of 0.5 to 45 Hz. The weak bioelectricity signals from forehead surface skin are adopted from the user by wearing a sensory headband. The interface box processes the signals and connects to a computer through a serial or USB port. Figure 2 shows the headband devices we used in the experimental system.

Figure 2. The headband EMG device in the human-machine interaction (HMI) system.

Figure 2. The headband EMG device in the human-machine interaction (HMI) system.

The muscle movements are defined to be control signals for a HMI system, namely single jaw click (SJC) and double jaw click (DJC). They are generated by the subjects contracting the masseter muscle and buccinator muscle with a jaw clenching and chewing-like movement. Every subject uses a designed sEMG pattern based HMI system to control an intelligent wheelchair, and the control signals are recorded at the same time. There are five control states designed for the wheelchair, namely: “Forward”, “Backward”, “Left”, “Right”, and “Stop”. Note that the first four states are scanned in turn. Users can use a single jaw click movement to switch the direction to whereas they want. If the wheelchair is in the control state of “Forward”, “Backward”, “Left”, or “Right”, a double jaw click movement will be used to switch the wheelchair to the “Stop” state. The complete structure of the HMI system is depicted in Figure 3.

Figure 3. The diagram of the proposed HMI system.

Figure 3. The diagram of the proposed HMI system.

3. Feature Extraction

Myoelectric control systems can be divided into two groups. In the first group, the desired classes of functions are discriminated from signal patterns by classifiers, and the variety of functions depends directly on classification performance. In the second group, recognition-based controllers, which are mainly constructed on threshold control and/or finite state machines, are merely output limited and pre-defined control commands based on a sequence of input signal patterns [19]. The latter gives a direct sense sometimes but has a non-adaptive shortage. After the forehead sEMG signals are obtained by using the dry electrodes sensor, an autoregressive model parameter algorithm is deployed to extract the signal feature.

sEMG signals can be considered as the output signals of a linear sequence with zero mean white noise. The white noise reflects the randomness of signals, while the deterministic part (the order and parameters of the model) reflects the predictability of the process. The four-order AR model is chosen to analyze the sEMG signals; the mathematical expression is as follows:

(1)

where x(n) is the current sampling value of sEMG signals. w(n) is the current incentive value, i.e., the residual of the white noise. p is the order of the model. a_k is the kth coefficient of the AR model.

The meaning of Equation (1) is that x(n) is generated by the linear combination of several past values x(n - k) and the current incentive value w(n). The model function is defined as follows:

(2)

For the AR model, the selection of p is critical. If the order is too low, like a high-order curve fitted with a low-order curve, it will produce a smooth result; if the order is too high, like a low-order curve fitted with a high-order curve, it will produce a dramatic change and vibration. Many experiments show that the analysis and recognition results of sEMG signals are the best when the order of AR model is 4, so the four-order AR coefficient vector A = [a1, a2, a3, a4] is the input to the SVM algorithm. Table 1 shows four groups of all four-order AR coefficients of Single jaw click and Double jaw click. The AR coefficients corresponding to two kinds of facial movements have obvious differences, so the variations of coefficients can be used as the basis for judging the changes of movement patterns. Meanwhile, it verifies whether the AR coefficients are reasonable as sEMG features.

Table 1. AR coefficients of Single jaw click and Double jaw click.

		1	2	3	4
Single jaw click	a1	−1.6459	−1.4644	−0.84630	−1.6434
	a2	0.77630	1.0654	−0.44390	0.72990
	a3	−0.15580	−1.1692	−0.10020	−0.11680
	a4	0.039300	0.57750	0.40900	0.048200
Double jaw click	a1	−0.96300	−0.99530	−1.2525	−0.98050
	a2	−0.22440	−0.38880	0.12190	−0.32100
	a3	0.087300	0.088600	0.31680	0.22390
	a4	0.12780	0.31810	−0.16790	0.097000

Table 1. AR coefficients of Single jaw click and Double jaw click.

		1	2	3	4
Single jaw click	a1	−1.6459	−1.4644	−0.84630	−1.6434
	a2	0.77630	1.0654	−0.44390	0.72990
	a3	−0.15580	−1.1692	−0.10020	−0.11680
	a4	0.039300	0.57750	0.40900	0.048200
Double jaw click	a1	−0.96300	−0.99530	−1.2525	−0.98050
	a2	−0.22440	−0.38880	0.12190	−0.32100
	a3	0.087300	0.088600	0.31680	0.22390
	a4	0.12780	0.31810	−0.16790	0.097000

Since EMG is a random signal, its power spectral density (PSD) should be estimated prior to calculating the time-course of the spectral variables. The mean (MNF) and median (MDF) spectral frequencies, which provide some basic information about signal spectral, have been the most popular such variables both in academic studies and in clinical practice, owing to their relevance to underlying physiological processes that control fatigue [20,21]. To measure muscle fatigue, the widely used Fast Fourier transform technique (FFT) is utilized to analyze mean frequency of sEMG.

(3)

(4)

where P_i is the ith line of the power spectrum; f_i is the frequency variable; and M is the highest harmonic considered.

MNF and MDF were calculated as previously described [22]. Statistical analysis is applied to find meaningful differences of sEMG from different subjects during various periods. The five subjects are required to perform repetitive movements and the average MNF and MDF calculated by before muscle fatigue, run 20 min, 40 min, and 60 min is showed in Figure 4. Each period lasted 20 s. Both MNF and MDF will decrease with time when muscle fatigue takes place. The MNF and MDF of forehead muscle decreased 10.8%, 17.2% separately while the average err fluctuation was almost unchanged after 60 min muscle contractions. It is experimentally proven that the manifestation of fatigue in myoelectric signals is significant during long-term muscular activities.

Figure 4. Mean (MNF) and median (MDF) shifts during different periods. Each period lasted 20 s.

Figure 4. Mean (MNF) and median (MDF) shifts during different periods. Each period lasted 20 s.

4. Adaptive Incremental Online SVM

sEMG patterns gradually change during long durations of muscle activities due to muscle fatigue, so the threshold method will lose its effectiveness. As we know, a SVM classifier constructs an optimal separating hyperplane in a high dimension feature space of training data that is mapped using a nonlinear kernel function [23,24]. The key ideas of SVM are: getting a maximized classification margin by solving dual optimization problem, and then obtaining an optimal separating hyperplane.

If the data is linearly separable, the decision function is:

(5)

where {x_i, y_i}, i = 1,…, l, are the training samples, y_i is the class label of the training sample x_i, l is the number of the training samples, b is the offset.

When the data is nonlinearly separable, the original data is mapped into a high dimensional feature space by kernel function K(x_i, x), in which the mapped data is linearly separable. Then, the decision function is as follows:

(6)

(7)

According to the Karush-Kuhn-Tucke (KKT) condition [25], there is only one solution for the optimization problem based on the quadratic optimization theory, and α = [α₁, α₂,…, α_l] is the optimal solution for the dual problem. When it satisfies the KKT condition, the problem is divided into the following three situations:

(8)

where α is Lagrangian dual, c is an upper bound for samples that lie on the wrong side of the hyperplane. Nonzero constant α_i is the sample corresponding to support vector. Samples are located on the classification interval 0 < α_i < c. When α_i = c, samples lie in the classification interval.

Incremental support vector machines are instrumental in practical applications of online learning. The purpose of the incremental algorithm is to add the new sample data to the three sets and to delete the redundant data, while maintaining all other samples that still belong to the three sets. In the classical SVM, the hyperplane in m dimensions can be shown as w^T∙x + b = 0. In our updating samples method, only the samples that are closest to the current boundary are chosen to train the classifier. There is a maximum distance of all update samples to boundary, which is defined as 2δ.

(9)

If the new added training sample violates the KKT conditions, the Lagrange multiplier α that is obtained from the previous sample will not be the optimum solution. The proposed improved online incremental training algorithm with the updating method is as follows. The standard SVM is used to initialize the classification and a training set is obtained. The next step is choosing the most informative patterns that have the best possibility of becoming the support vectors in the in-put sEMG sequence, and checking whether these samples meet the KKT conditions. If not, try the next data until KKT conditions are satisfied. Then construct samples including the update data, and train samples online to form a new classifier. The process repeats until there is no in-put sEMG data.

5. Experiment Results

Traditional SVM, such as LIBSVM, employed a radial basis function kernel to classify data, with the same parameter (C = 1 and γ = 0.5) [26]. The data is normalized and limited to the range of −1 to 1. 800 samples are randomly selected from 2,000 pre-collected signals as a training set. After that, choose 800 samples in the rest samples randomly as a testing set. When the result is compared with the method given above, the classification accuracy is shown in Table 2. The training time was generated at 300 samples.

Table 2. Comparison real time performance of the LIBSVM and the Incremental online support vector machine (SVM).

	LIBSVM	Incremental online SVM
Single jaw click	73%	90%	recognition rate
	92 ms	58 ms	training time
Double jaw click	81%	93%	recognition rate
	103 ms	62 ms	training time

Table 2. Comparison real time performance of the LIBSVM and the Incremental online support vector machine (SVM).

	LIBSVM	Incremental online SVM
Single jaw click	73%	90%	recognition rate
	92 ms	58 ms	training time
Double jaw click	81%	93%	recognition rate
	103 ms	62 ms	training time

Note that five subjects in the experiment are required to control the intelligent wheelchair running throughout a specified path in the lab. A map of the experiment environment with the designed path is given in Figure 5. The size of the map is 8,000 × 6,000 mm and the actual size of the wheelchair is 1,000 × 700 mm. “Forward”, “Backward”, “Left”, “Right”, and “Stop”: we tried to generate an average for each control state over two experiment days. For each experiment, the five subjects are required to control the intelligent wheelchair for 90 min without a rest.

Figure 5. The path designed for experiment.

Figure 5. The path designed for experiment.

Figure 6 shows the average time consumptions of five subjects in repetitive experiments during deferent periods. The average consuming time is 103 s for one round at the beginning, but the average time increased to 154 s at the last period when muscle fatigue occurred. After using the new method, the average time at the last period could decrease to 110 s. Table 3 shows the average recognition rate of five subjects controlling the wheelchair using the proposed HMI system with the two online SVM in different periods. The recognition of jaw click, or what is called “single jaw click” and “double jaw click”, is the most important factor of the experiment.

Figure 6. The five subjects’ average time consumptions of repetitive experiments during different periods.

Figure 6. The five subjects’ average time consumptions of repetitive experiments during different periods.

Table 3. Comparison long time performance of the LIBSVM and the Incremental online SVM.

Subjects	Recognition rate for five state using LIBSVM			Recognition rate for five state using incremental online SVM in seconds
	0–20 min	20–40 min	40–60 min	0–20 min	20–40 min	40–60 min
A	86.5	80.1	73.8	92.6	88.4	86.7
B	88.7	81.6	73.3	92.8	89.5	87.4
C	83.1	79.2	70.9	86.7	82.8	79.3
D	90.4	85.5	79.2	93.6	90.1	88.3
E	80.8	75.3	70.1	84.9	81.6	78.4
Average	85.9	80.34	73.46	90.12	86.48	84.02

Table 3. Comparison long time performance of the LIBSVM and the Incremental online SVM.

Subjects	Recognition rate for five state using LIBSVM			Recognition rate for five state using incremental online SVM in seconds
	0–20 min	20–40 min	40–60 min	0–20 min	20–40 min	40–60 min
A	86.5	80.1	73.8	92.6	88.4	86.7
B	88.7	81.6	73.3	92.8	89.5	87.4
C	83.1	79.2	70.9	86.7	82.8	79.3
D	90.4	85.5	79.2	93.6	90.1	88.3
E	80.8	75.3	70.1	84.9	81.6	78.4
Average	85.9	80.34	73.46	90.12	86.48	84.02

Experimental results indicate that the performance of the sEMG based HMI system with LIBSVM gradually declines as time increases with the occurrence of muscle fatigue. The subjects control the intelligent wheelchair by this system using more time in different periods, as shown in Table 3. Also, one should note that, although the threshold EMG method has a pretty high recognition rate before muscle fatigue, the performance after that declines quickly. The incremental online SVM proposed showed good properties in both convergence and optimality.

6. Conclusions

Muscle fatigue can negatively influence the performance of human-machine interaction using sEMG signals. This paper presents a novel sEMG based HMI system for the hands-free control of an intelligent wheelchair both in real time and in long time operations. The system is used to control the simple movements of the intelligent wheelchair, i.e., forward, turn-left, turn-right, backward and stop. The experimental results show that the proposed HMI is easy to operate in real-time and has a high recognition rate. It could be deployed in many aspects of real world applications, such as helping the elderly and disabled, rehabilitation, and so on.

The speed and accuracy of pattern recognition of forehead sEMG signals are to be further improved in our next stage of research. By improving the incremental online SVM algorithm, we hope to reduce the false recognition rate, and obtain a fast response time. In addition, we will investigate how to effectively reduce the noisy signals from facial movements when a user is talking and looking around in the practical application. Moreover, Electrooculography (EOG) signals will be combined with EMG signals to generate the control signals for wheelchair motion control so that the control method is more flexible and robust.

Finally, it is worth remarking that Brain Computer Interface (BCI) based on the Electrooculography (EEG) signals have been used to control the electric-powered wheelchair during the last decade [17,18]. However, these EEG based HMIs are not performing well in the real world since they are non-stationary and low in signal to noise ratio. This was mainly caused by the subject’s brain conditions or dynamically changing environments. Therefore, it is necessary to carefully design signal processing algorithms for feature extraction, classification and adaptation. Alternatively, EEG signals could be fused with EMG and EOG signals for accurate human intension detection in real time.