Improving sEMG-Based Hand Gesture Recognition through Optimizing Parameters and Sliding Voting Classifiers

Abstract

In this paper, we present a preliminary study that proposes to improve surface electromyography (sEMG)-based hand gesture recognition through optimizing parameters and sliding voting classifiers. Targeting the high-performing myoelectric control system, the traditional methods for hand gesture recognition still need to further improve the classification accuracy and utilization rate for sEMG signals. Therefore, the proposed method first optimizes parameters to reduce redundant information by selecting the proper values for the window length, the overlapping rate, the number of channels, and the features of sEMG signals. In addition, the random forest (RF) classifier is an advanced classifier for sEMG-based hand gesture recognition. To further improve classification performance, this paper proposes a sliding voting random forest (SVRF) classifier which can reduce potential pseudo decisions made by the RF classifier. Finally, experiments were conducted using two sEMG datasets, named DB2 and DB4, from the NinaPro database, as well as self-collected data. The results illustrate a certain improvement in classification accuracy based on the optimized values for window length, overlapping rate, number of channels, and features of sEMG signals. And the SVRF classifier can significantly improve performance with higher accuracy compared with the traditional linear discriminate analysis (LDA), k-nearest neighbors (KNN), support vector machine (SVM), and RF classifiers.

1. Introduction

In the field of myoelectric control systems, hand gesture recognition can convert a user’s hand movements into control instructions, which is one of the important ways to achieve natural and accurate communication and interaction between humans and machines [1]. In the field of human motion signal detection, force myography (FMG), surface electromyography (sEMG), and mechanomyography(MMG) are the three most commonly used and important biological signals for detecting muscle movement. Compared with FMG and MMG, sEMG signals are easier to collect and process in hand gesture recognition. Due to the fact that sEMG signals are generally generated 30–150 ms ahead of hand movements, sEMG-based hand gesture recognition methods are widely used in intelligent prosthetics, industrial robots, human–machine interaction, and other fields and exhibit noninvasive and low-cost characteristics [2].

The process of sEMG-based hand gesture recognition mainly includes three steps: data segmentation, feature extraction, and pattern classification [3]. Data segmentation is the process of grouping sEMG data by signal windowing. The window length of segmentation needs to meet the requirements that balance the effects of real-time control and classification error [4,5]. In a study on hand gesture recognition, Englehart et al. reported that the classification performance of the linear discriminant analysis (LDA) classifier was optimal with a window length of 256 ms and a window increment of 16 ms (or an overlapping rate of 93.75%) when four hand movements were classified by using self-collected sEMG data from 12 subjects [6]. Huang et al. reported that the classification performance of the Gaussian mixture model classifier was optimal with a window length of 256 ms and a window increment of 32 ms (or an overlapping rate of 87.5%) when six hand movements were classified by using self-collected sEMG data from 12 subjects [7]. Oskoei et al. reported that the classification performance of the support vector machine (SVM) classifier was optimal with a window length of 200 ms and a window increment of 50 ms (or an overlapping rate of 75%) when six hand movements were classified by using self-collected sEMG data from 11 subjects [8]. Smith et al. reported that the range of window lengths for achieving the best classification performance was 150–250 ms when seven hand movements were classified using self-collected sEMG data from 13 subjects [9]. Robinson et al. reported that the classification performance of the random forest (RF) classifier was optimal with a window length of 256 ms and a window increment of 10 ms (or an overlapping rate of 96%) when 17 hand movements were classified by using the sEMG data from the first 11 subjects of the DB2 dataset from the Ninapro database [10]. Hassan et al. reported that the SVM classifier had the best classification performance with a window length of 240 ms and a window increment of 120 ms (or an overlapping rate of 50%) when seven hand movements were classified by using self-collected sEMG data from six subjects [11]. The above studies indicate that the values chosen for the window length and overlapping rate have a certain degree of impact on the performance of classifiers. However, the selection of window length and overlapping rate values is quite unreasonable in some experiments. For example, in references [7,10,11], the window length or overlapping rate was directly set to a fixed value. In reference [8], different window lengths were processed in different ways. For window lengths less than 200 ms, disjoint segmentation was used, while for window lengths greater than 200 ms, overlapped segmentation was utilized with a window increment of 200 ms. Furthermore, when the processing time for a window length of data is used instead of the overlapping rate, there is significant variation between different studies; for instance, the processing time for a window length of data as the corresponding window increment and a fixed window increment of 25 ms were used in references [6,9], respectively. This may result in a higher overlapping rate corresponding to a larger number of windows and features in subsequent processing, with the classification accuracy increasing with the increase in window length. However, common methods of feature extraction generally use a fixed window length to extract features of sEMG signals with overlapping or nonoverlapping adjacent windows [12,13].

In the past few years, deep learning methods (DLMs) have become a research hotspot in fields such as sEMG-based hand gesture recognition [14,15,16]. DLMs have achieved much higher accuracy than traditional machine learning methods (MLMs). Compared with traditional MLMs, DLMs use more data to achieve better scaling and do not require the step of feature extraction. However, for smaller datasets, traditional MLMs are often superior to DLMs. To achieve high performance, DLMs require a very large dataset. For many applications, such large datasets are not easy to obtain, and their assembly is costly and time-consuming. Deep networks require high-end GPUs to train within a reasonable time frame with a large amount of data. These GPUs are very expensive, but it is not feasible in practice to achieve high performance without training deep networks through them. Traditional MLMs only require a decent CPU to train well, without the need for the best hardware. Due to the direct feature extraction involved in traditional MLMs, their algorithms are easy to interpret and understand. However, deep networks are “black boxes”, and even current researchers cannot fully understand their “internal” aspects. Due to the lack of a theoretical basis, hyperparameter tuning and network design are also considerable challenges. Therefore, although DLMs have such high performance, the performance of traditional MLMs is better than that of DLMs in some specific cases. For sEMG-based hand gesture recognition, the traditional MLMs, including LDA [6], k-nearest neighbor (KNN) [17], SVM [18], and RF [19] models, are usually chosen, and the latter study has reported that the RF classifier achieves the highest classification accuracy using the same datasets from the Ninapro database [19].

Given the above analysis, in order to further improve the accuracy of hand gesture recognition, this paper proposes an improved scheme through optimizing parameters and sliding voting classifiers. The proposed method mainly uses the sliding window method for time-domain feature extraction, while the proper values for the window length, the overlapping rate, the number of channels, and the features of sEMG signals are selected to reduce redundant information by optimizing parameters. In addition, we propose a time-series sliding voting strategy to process the classification results of the RF classifier to reduce potential pseudo decisions in the decision-making flow, which can enhance the classification performance of the RF classifier. The experimental results for different classifiers show that the proposed sliding voting RF (SVRF) classifier can effectively improve classification accuracy, and the average accuracy rate is improved by more than 2%.

2. Methods for sEMG-Based Hand Gesture Recognition

2.1. Data Segmentation

An example of an sEMG signal is shown in Figure 1. In myoelectric control systems, the acquired sEMG signals present a form of serial data stream. As shown in Figure 2, the sliding window method is used to segment sEMG signals with an overlapping strategy, where $T_{swin}$ represents the sliding window length, $T_{vwin}$ represents the voting window length, $T_{inc}$ represents the window increment, and $\tau$ represents the data processing time of each sliding window.

In general, for real-time control applications, the segmentation length for processing data should not exceed 300 ms of controller delay acceptable to the users [4,5]. Therefore, when $\tau$ is usually less than 20 ms [4,5], $T_{vwin}$ is set to 280 ms in this study. Moreover, $T_{inc}$ will result in an overlapping rate, which is expressed as follows:

$$overlap=\frac{T_{swin}-T_{inc}}{T_{swin}}\times 100\%$$

(1)

2.2. Feature Extraction

According to the studies in the literatures, sEMG signals often contain a mixture of very-low-frequency (close to DC) and high-frequency interference components, while the effective spectrum of sEMG signals is distributed between 10 and 500 Hz. Therefore, sEMG signals detected from SMT (surface mount technology) electrodes need to undergo signal conditioning processes, such as high-pass filtering (DC isolation processing), high magnification, and low-pass filtering (filtering of high-frequency interference components) [3]. Moreover, feature extraction is used to highlight important information in sEMG signals and reduce the impact of noise and other interference components, which is beneficial for improving the classification accuracy of sEMG-based hand gesture recognition [20,21].

The features of sEMG signals mainly include time-domain, frequency-domain, and time–frequency-domain features. To reduce the processing time of feature extraction and improve the robustness of hand gesture recognition, this study only considers time-domain features of sEMG signals [22]. The four time-domain features, including mean absolute value (MAV), slope sign change (SSC), waveform length (WL), and zero crossing (ZC), have been proven to be suitable for hand gesture recognition [12]. So, feature extraction was performed on each sliding window shown in Figure 2, with the calculation of MAV, SSC, WL, and ZC as features according to the following:

$$MAV=\frac{1}{N}{\displaystyle \sum _{n=1}^N\left|x_n\right|}$$

(2)

$$\begin{array}{l}SSC={\displaystyle \sum _{n=2}^{N}f\left[\left(x_n-x_{n-1}\right)\times \left(x_n-x_{n+1}\right)\right]}\\ f(x)=\{\begin{array}{l}1,x\ge 0\\ 0,x<0\end{array}\end{array}$$

(3)

$$WL={\displaystyle \sum _{n=1}^N\left|x_{n+1}-x_n\right|}$$

(4)

$$\begin{array}{l}ZC={\displaystyle \sum _{n=1}^{N-1}\left[\mathrm{sgn}\left(x_n\times x_{n+1}\right)\cap \left(x_n\times x_{n+1}\right)\ge 0\right]}\\ \mathrm{sgn}(x)=\{\begin{array}{l}1,x\ge 0\\ 0,x<0\end{array}\end{array}$$

(5)

where $x_n$ is the amplitude of sEMG signals at the $nth$ data point; $n=1,2,\cdots \cdots ,N$; and $N$ is the number of data points in the sliding window ($T_{swin}$).

Figure 3a shows the sEMG signals of eight channels with thumbs up from the Ninapro database, and Figure 3b shows the calculated MAV values for each sliding window based on the sEMG signals in Figure 3a (with a sliding window length of 200 ms and an overlapping rate of 90%). Figure 4a shows the sEMG signals of eight channels with extended index and middle fingers and flexion of the other fingers from the Ninapro database, and Figure 4b shows the calculated MAV values for each sliding window based on the sEMG signals in Figure 4a (with a sliding window length of 200 ms and an overlapping rate of 90%). From these figures, it can be seen that the trend of the MAV values of the sEMG signals is more pronounced than that of the raw sEMG signals, and the dimensionality of the sEMG signals is also reduced. Therefore, classification accuracy can be improved by inputting feature vectors into a classifier for hand gesture recognition.

In a hand gesture recognition system, feature extraction is always directly applied to the entire sEMG data segment of the selected channel [3]. In order to improve classification accuracy, the popular methods adopted are: (1) increasing the number of sEMG signal channels; (2) extracting more features of sEMG signals. Therefore, the number and types of features extracted in these methods will increase, which will impose a burden on the system’s software and hardware resources, seriously affecting the system’s operational efficiency.

2.3. Pattern Classification

2.3.1. Random Forest Classifier

RF uses a random approach to combine multiple decision trees into a forest, with the final classification result determined by the outcomes of these decision trees. The flowchart of an RF classifier is shown in Figure 5. Independent decision trees are used in RF, which are formed by randomly chosen variables and built by an algorithm. The RF classifier integrates multiple decision trees using the ensemble learning idea to achieve the effect of multiple weak classifiers forming a strong classifier.

The RF classifier uses two random processes: one is the random selection of a training subset of the decision tree, and the other is the random selection of a specified number of features as nodes in the process of generating decision trees. And the bootstrap method is commonly used to obtain multiple random and independent training subsets. The decision tree classifier is constructed from the training subset and is not pruned during the training process. Finally, multiple decision tree classifiers construct a random forest model. Each decision tree classifier ultimately makes a decision, and the final classification result of the RF is determined through voting.

During the classification process, the possibility of multiple classifiers producing incorrect results is greatly reduced compared to using only a single classifier. Therefore, RF classifiers have been widely used in classification tasks. The RF classifier requires pre-set parameters, such as the number of trees, the number of splits, and the depth of the RF, which were set to 50, 24, and 5, respectively, in this study. These values were selected after analyzing the impact of changing these parameters on the performance of the RF classifier.

2.3.2. Time-Series Sliding Voting Strategy

The processing method of the decision flow using the time-series sliding voting strategy is shown in Figure 6, where the sliding window is used to obtain the classification results as the RF classifier’s outcome. From the front to the end of a voting window, successive decisions are obtained using the sliding window method. The final classification result can be obtained as the SVRF classifier’s outcome by using a time-series sliding voting strategy. The total number of votes for a voting window ($T_{vwin}$) is expressed as follows:

$$votes=\frac{T_{vwin}-T_{swin}}{T_{inc}}+1$$

(6)

If the number of decision a is $m_1$, the number of decision b is $m_2$, the number of decision c is $m_3$,$\cdots \cdots$, and $m_1>m_2>m_3>\cdots \cdots$, then the final classification result is a. As shown in Figure 6, the decision flow is obtained by using the features of each sliding window ($T_{swin}$) of the sEMG signal for classification. The classifier’s outcome for the first voting window ($T_{vwin}$) is: a, a, b, a, a, c, a. Here, the number of decision a is 5, the number of decision b is 1, and the number of decision c is 1, so the final classification result is a. Next, the classifier’s outcome for the second voting window ($T_{vwin}$) is: a, b, a, a, c, a, a. Here, the number of decision a is 5, the number of decision b is 1, and the number of decision c is 1, so the final classification result is a, and so on.

The proposed scheme for sEMG-based hand gesture recognition in this paper is shown in Figure 7, which mainly includes inputting the dataset, dividing the training set and testing set, extracting features using the sliding window method, normalization of feature values, classification of the RF classifier, and making the final decision using the time-series sliding voting strategy.

3. Results and Discussion

3.1. Testing Using the Ninapro Dataset

3.1.1. Experimental Protocol

The experiments used two datasets with the same collection protocol from the Ninapro database named DB2 and DB4. The sEMG data were acquired at a sampling rate of 2 kHz. The DB2 dataset includes six repetitions of 49 hand movements from 40 intact subjects, and the DB4 dataset includes six repetitions of 52 hand movements from 10 intact subjects. But the tests in this paper only selected 40 hand movements from the DB2 and DB4 datasets, as shown in Figure 8. The sEMG data of 17 hand movements were set as A group data (Exercise B in the DB2 and DB4 datasets), and the sEMG data of the other 23 hand movements were set as B group data (Exercise C in the DB2 and DB4 datasets). Therefore, the A group data had the following size: $17\times (40+10)\times 6=5100$, and the B group data had the following size: $23\times (40+10)\times 6=6900$. Consequently, the entire dataset with a size of 12,000 was composed of A group and B group data, which were divided into two subsets: a training subset with a size of 10,000 and a testing subset with a size of 2000.

3.1.2. Optimizing Parameters

An increase in the number of sEMG features not only degrades classification performance, it also increases the complexity of the classifier. Therefore, selecting appropriate parameters, such as the window length, the overlapping rate, the number of channels, and the features of sEMG signals, is an effective process for eliminating redundant and irrelevant features. To improve sEMG-based hand gesture recognition, selection of appropriate parameters ought to be carefully considered. In this study, the optimization of parameters was carried out by evaluating the effect of parameters on classification accuracy using the selected DB2 dataset.

The sliding window length and overlapping rate have a certain impact on the performance of the classifier. Here, in order to evaluate their effect on classification accuracy, we set the sliding window lengths and overlapping rates to 50 ms, 100 ms, 150 ms, 200 ms, 250 ms, 300 ms, 400 ms, 500, and 600 ms and 10%, 30%, 50%, 70%, and 90%, respectively. The classification results for the RF classifier using different sliding window lengths and overlapping rates are shown in Figure 9.

From Figure 9, it can be seen that the overlapping rate has a significant impact on the performance of the classifier. As a result, a decrease in the overlapping rate will lead to a decrease in the number of sliding windows and classifier decisions, resulting in a decrease in classification accuracy. So, the overlapping rate was set to 90% in this paper.

In Figure 9, in the front of each curve within the range of 50–100 ms, increasing the sliding window length can improve the performance of the classifier. Due to insufficient information, the final classification accuracy for the window length of 50–100 ms is lower than that for the window length of 100–250 ms. In the latter half of each curve, as the sliding window length increases, each curve shows a decrease. The decreases are most significant for curves with low overlapping rates, reaching as low as 58.41%.

When different sliding window lengths and a fixed overlapping rate (set to 90%) were used to observe their impact on the performance of the classifier, the classification accuracy curve showed an upward trend as the sliding window length increased, as shown in Figure 10.

In addition, processing times with sliding window lengths of 100–300 ms (with an overlapping rate of 90%) were tested using 300 ms sEMG data. And the test results are shown in

Table 1. Processing time of 300 ms data using different sliding window lengths (with an overlapping rate of 90%).

Time	Sliding Window Length (ms)
	100	150	200	250	300
Processing time of feature extraction process (ms)	35.4	27.8	20.2	12.6	5.1
Processing time of classification process (ms)	491.8	324.7	241.1	191.2	157.5
Total processing time (ms)	527.2	352.5	261.3	203.8	162.6

. The program was run on a 64-bit Windows 10 system with a 1.90 GHz USA Intel Core i7-8650U CPU and 16.00 GB of Korean Samsung RAM. Considering that the acceptable controller delay for users cannot exceed 300 ms, a sliding window length of 200 ms was finally chosen for the real-time control applications.

The number of data channels and features also have a certain impact on the performance of the classifier. In the following tests, sEMG signals from the first 8 channels (out of a total of 12 channels) in the DB2 and DB4 datasets were used to extract four time-domain features. Obviously, there may be redundant information in these channels. To reduce the calculation burden and eliminate the redundant information, the following two tests were conducted.

The first test was performed using four time-domain features: MAV, WL, ZC, and SSC, in which eight channels of sEMG data were used without changing other parameters. The classification results obtained using only a single feature of the sEMG data are shown in Figure 11. It is evident that the classification accuracy when using MAV and WL features alone is much higher than that using ZC and SSC features, indicating that MAV and WL are significant features of sEMG signals. Therefore, MAV and WL features were selected for subsequent analysis.

The second test was performed using the first eight channels of the sEMG data with the restriction to MAV and WL features. The classification results obtained using only a single channel of sEMG data are shown in Figure 12. It is obvious that the classification accuracy of any sEMG data channel is less than 20%, so it was decided to choose the full set of channels to achieve higher classification accuracy.

3.1.3. Classification Performance Evaluation

After the above steps, all the parameters in the proposed method had been determined. Using A group and B group data from the DB2 dataset for classification testing, the tests of hand gesture recognition were performed by two methods: the RF and SVRF classifiers. The test results are shown in Figure 13, and it can be clearly observed that the performance of the SVRF classifier is better than that of the RF classifier.

Next, the stability of the proposed method was verified by a 6-fold cross-validation test using A group data from the DB2 dataset. The classification results are shown in Figure 14. Figure 14a shows an average classification accuracy of 93.88% by the 6-fold cross-validation test, with a variation range of 93.88% ± 3.47%. Figure 14b shows a confusion matrix for the proposed method determined by the 6-fold cross-validation test, with an average classification accuracy of 91.32% and the lowest classification accuracy being 80% for different hand movements. In Figure 14b, the values in the black rectangles represent the average classification accuracies, and each row of values represent the deviation rate of the classification. The values in the lighter-colored rectangles on both sides of the diagonal indicate a smaller deviation in classification. The results show that the proposed method has a certain degree of stability.

In addition, the test was conducted on 17 hand movements in the A group data and 23 hand movements in the B group data from the DB2 and DB4 datasets. The classification results for the proposed method are shown in Figure 15. Figure 15 clearly shows that the proposed method obtained different classification results for the A group and B group data. The average classification accuracies obtained using the A group data are 93.88% and 92.58%, and those obtained using the B group data are 90.86% and 85.59%.

Finally, the classification accuracy of the proposed method was evaluated by comparison with the traditional LDA, KNN, SVM, and RF classifiers using the DB2 and DB4 datasets. The average classification accuracies can be found in Figure 16. For the DB2 dataset, the SVRF classifier achieves 92.14% accuracy, which is higher than that of the LDA (77.72%), KNN (80.26%), SVM (85.72%), and RF (89.74%) classifiers. Additionally, the SVRF classifier increases the accuracy of the RF classifier from 89.74% to 92.14%. For the DB4 dataset, the SVRF classifier achieves 88.56% accuracy, which is higher than that of the LDA (75.42%), KNN (78.76%), SVM (83.02%), and RF (87.76%) classifiers. Additionally, the SVRF classifier increases the accuracy of the RF classifier from 87.76% to 88.56%. So, when using the DB2 and DB4 datasets to classify hand gestures, the SVRF classifier has a better performance compared with the traditional methods.

3.2. Testing Using Self-Collected Data

In order to validate the proposed method for practical applications, a simple two-channel sEMG data acquisition system was constructed [23,24]. In this paper, the electrodes used were commercial AgCl electrodes. And the experimental muscles were the extensor digitorum and the extensor pollicis longus muscles of the forearms of the subjects, which were the placement positions for the electrodes shown in Figure 17. When placing electrodes, they must be aligned with the direction of muscle fibers, and the reference electrode should be placed in the area with the least number of forearm muscles.

The sEMG data were first acquired using a front-end acquisition circuit based on the differential amplification method with a sampling frequency of 1000 Hz. Then, an STM32 microcontroller sent data to the host computer through Bluetooth wireless communication. Finally, the host computer saved the raw data in text format for subsequent processing.

During the acquisition, five intact subjects (two males and three females) of similar age (20 to 22 years old) were asked to repeat the six hand movements shown in Figure 18, which can be considered basic hand movements: (a) cylindrical grasp, (b) tip, (c) hook, (d) palmar, (e) spherical, and (f) lateral. Each movement repetition lasted 6 s and was followed by 6 s of rest, and each movement was repeated 30 times. So, a total of 180 two-channel sEMG data were collected for each subject, resulting in a total of $5\times 6\times 30=900$ raw sEMG data.

The proposed method was verified based on self-collected data. The classification results for the LDA and SVRF classifiers are illustrated in Figure 19, where the parameters were kept consistent.

Considering the average classification accuracies, the SVRF classifier achieved an accuracy of 94.23% against an accuracy of 76.67% for the LDA classifier. Although the number of subjects in the collected data was relatively small, these results still confirm that the proposed method has better performance and potential practical applications.

4. Conclusions

In this paper, a method is proposed to improve sEMG-based hand gesture recognition through optimizing parameters and using an SVRF classifier. First, we investigate the effect of parameters on sEMG-based hand gesture recognition, which mainly include the window length, the overlapping rate, the number of channels, and the features of sEMG signals. The proper values for these parameters were selected after analyzing the impact of changing these parameters on the performance of the classifier. The test results indicate that the classification accuracy of the proposed method can be improved with the optimized values. In addition, the proposed method mainly uses the sliding window method for time-domain feature extraction to improve classification performance. Furthermore, we evaluate the classification accuracies of different classifiers using the DB2 and DB4 datasets from the Ninapro database, as well as self-collected data. Compared with the traditional LDA, KNN, SVM, and RF classifiers, we find that the proposed SVRF classifier not only has stable performance but also higher classification accuracy for sEMG-based hand gesture recognition. In the future, if the proposed method can be applied in practical applications, a more refined scheme should be implemented with more subjects and hand movements.