Lower-limb prostheses greatly improve amputees' locomotive ability in performing daily activities. Most of the commercially available lower-limb prostheses are energetically passive. Though they are widely used, they have inevitable deficiencies such as more metabolic power consumption and asymmetrical gait pattern . Thus, studies on powered lower-limb prostheses are gaining more and more attention [1–6]. Compared with passive lower-limb prostheses, powered prostheses can provide necessary power in certain periods to make the locomotion more natural, but they need more complex control strategies [7–12] and more locomotion information from amputees. As control approaches for different locomotion modes are different, powered prostheses should “know” amputees' locomotion intents in advance, so as to select appropriate parameters and realize safe and fluent control. Therefore, the prediction of amputees' locomotion intents plays an important role for the control of powered lower-limb prostheses.
Some methods have been proposed for lower-limb locomotion mode recognition using signals measured from mechanical sensors [13–15] and bioelectric signals such as electromyography (EMG) signals [16–20]. Varol et al. collected signals of joint angles and angular velocities of the knee and ankle, socket sagittal plane moment, foot forces of heel and ball, and realized the recognition of three locomotion modes (standing, sitting and walking) and transitions between them . However, half a second delay existed for locomotion intent recognition. Young et al. proposed a recognition system capable of performing transitions between walking on level ground, ramps and stairs for a powered prosthesis, using onboard mechanical sensors . However, the overall recognition accuracy was only 93.9%. Huang et al. used EMG signals measured from lower-limb and signals of a 6-DOF load cell to classify six locomotion modes and five locomotion transitions . However, the method was only tested offline, and recognition performance of long-time use was not reported. The evaluation of long-time use is important for myoelectric decoding algorithms, because they are inherently subject to decay in recognition accuracy over time , which is caused by the variation of EMG signals due to electrode conductivity changes, electrophysiological changes, spatial changes, user changes and potentially other factors .
In principle, fusion of multi-sensor data provides significant advantages over single source data . Therefore, sensor fusion is widely used in various fields such as robotics [24,25], medical applications , target tracking [27–29], person identity verification , human activity recognition [31,32] and so on. More and more intent prediction systems for powered lower-limb prosthesis control also developed with multi-sensor fusion [14,18]. Single kind of sensors only collect part of the locomotion information, which may be insufficient. With the fusion of different kinds of sensor signals, more useful information are obtained and recognition performance can therefore be improved. Huang et al. compared the recognition result of using only EMG signals with that using EMG-mechanical fusion, and validated that the performance with multi-sensor fusion was significantly better .
In this paper, we propose a locomotion intent prediction system based on a new way of multi-sensor fusion, and systematically evaluate its promise for clinical application. Signals of three inertial measurement units (IMUs) and two custom made pressure insoles are measured. These two kinds of sensors have been widely used for human movement measurement and detection [33–35]. Many works have been done on signal processing technologies for these sensors, and some methods have been proposed to deal with signal drift and interference . In this research, with linear discriminant analysis (LDA) classifier, time-domain feature set and a two-level recognition strategy, the system can successfully recognize six locomotion modes and ten locomotion transitions. We also evaluated the performance of long-time use for this system on seven able-bodied subjects, which was essential for clinical application. No deteriorating tendency of recognition performance was observed after long-term use, and only a small data amount was needed for system training. These properties demonstrate the potential of applying the system for the control of powered lower-limb prostheses in daily life.
2. Measurement System
2.1. Sensor Placement on Human Body
To measure as much useful locomotion information as possible, the positions of sensors were carefully selected (Figure 1). Three IMUs (SparkFun Electronics Inc., Boulder, CO, USA) were placed on the thigh (IMU module2), the shank (IMU module1), and the foot of the measured leg (FP module1), respectively. They were used to measure angles of the mentioned limbs in the sagittal plane and frontal plane, and accelerations along two perpendicular axes in the sagittal plane. In addition, two custom made pressure insoles were placed in shoes of both sides to detect gait events and record foot pressure information during stance periods. Foot pressure sampling circuits were placed on the shank for the unmeasured leg (FP module2) and the back of the shoe for the measured leg (FP module1). There was a control circuit on the waist to control the data sequence. A lithium battery (10.8 V) was used to provide power for the sensors.
2.2. Inertial Measurement Unit
The IMU board is built with an accelerometer, a gyroscope and a magnetometer (Figure 2c). The accelerometer is a digital microchip named ADXL345 (Analog Devices Inc., Norwood, MA, USA) with three-axis measurement. The gyroscope is a three-axis micro-electromechanical systems (MEMS) based integrated circuit (ITG-3200, InvenSense Inc., San Jose, CA, USA) and the magnetometer is a surface-mount, multi-chip module designed for low-field magnetic sensing with a digital interface (HMC5883L, Honeywell International Inc., Morristown, NJ, USA). ATMEGA328 (Atmel Corp., San Jose, CA, USA) is used as the micro control unit (MCU) of the IMU board for data collection. The raw data measured with sensor chips are transmitted to the MCU via inter-integrated circuit (IIC) bus. To eliminate signal drift and noise, the direction-cosine-matrix (DCM) method is used. Data of the gyroscope are used for estimating the orientation of the IMU board. The magnetometer and the accelerometer are used to complement errors of the orientation. Output data of the IMU board include the pitch angle, the roll angle and accelerations in two axes, containing reliable and useful locomotion information.
2.3. Foot Pressure Insoles
The foot pressure insoles were self-made . FlexiForce A401 (Tekscan Inc., South Boston, MA, USA) sensors were placed on the bottom of the insole to record foot pressure signals. The FlexiForce A401 sensor is a kind of force sensitive resistors which is inverse proportion to the force exerted on it. The sensors are selected for the following reasons. First, the resolution of the sensors is high enough to record the changes of foot pressure during stance periods. Second, the sensors are thin and small, making it comfortable to wear the equipped insoles. Third, the measuring range and the useful lifespan can meet the requirement for locomotion mode recognition. To collect more useful information of foot pressure during walking, positions of the sensors were carefully selected . As is shown in Figure 2b, the four key positions are the big toe, the first metatarsal, the fourth metatarsal, and the heel. The position selection is similar to previous studies [38,39].
The foot pressure sampling circuits are designed to measure the foot pressure of both feet (Figure 2d). For each circuit, an inverting amplifier is built using LMV324 (STMicroelectronics Inc., Geneva, Switzerland) to convert the foot pressure to linear dependent voltage. Then the signals are input to the STM32F103 (STMicroelectronics Inc., Geneva, Switzerland) for analog to digital converting (ADC). For the sampling circuit on the measured leg, an IMU module is also embedded to measure movement information of the foot.
2.4. Data Transmission
The control circuit controls the data sequence of the sensor modules. In each sample interval, the control circuit collects data from each sensor module and then transmitted them to the receiver circuit with wireless module. The communication between the control circuit and the sensor modules is implemented with recommended standard 485 bus (Figure 2a). Sampling rates for all sensors are 100 Hz. In other words, all sensor data of each sample has to be sent to the receiver circuit in 10 ms. In the designed system, the data sequence is controlled with the polling method. In this method, each sensor module is assigned with an ID. Each sampling interval (10 ms) is separated to several time slices. During each slice, the control circuit broadcasts the command data packet with an ID and waits for the response. Then the sensor module with the same ID sends out the data. This method guarantees the unobstructed communication on the bus. The wireless module on the control circuit is built based on nRF24L01 (Nordic semiconductor Inc., Oslo, Norway). The wireless module sends the data out in each sample interval. Cyclic Redundancy Checking and automatic retransmission method is used to reduce error rate.
3.1. Subjects and Experiment Protocol
Seven able-bodied subjects participated in this research and provided written and informed consent. They had an average age of 24.1 (±0.5) years, an average height of 1.71 (±0.02) m and an average weight of 72.0 (±2.5) kg.
The experiment consisted of two sections. The first section included 20 experiment pairs. Each pair included one experiment trial with the measured leg first walking (marked as Set-A) and one trial with the unmeasured leg first walking (marked as Set-B). Experiment of the first section was taken continuously and five-minute rests were allowed every five pairs. The second section also included 20 experiment pairs. It started immediately after the first section. This section lasted for five hours and one experiment pair was tested every 15 min. In each experiment trial, different locomotion tasks were performed continuously in predefined order, so as to collect data of targeted locomotion transitions. The subjects stood still at the beginning, and then took level-ground walking, stair ascent, level-ground walking, ramp descent, level-ground walking, stand, turning back, stand, level-ground walking, ramp ascent, level-ground walking, stair descent, level-ground walking, stand, turning back and stand in turn. The staircase had 4 stairs and each one was 75 cm in width, 40 cm in depth and 15 cm in height. The inclination angle of the 2.1-meter long ramp was 16.5°.
Be similar with , the standing mode defined in this research included static standing (i.e., standing still) and dynamic standing (turning back with small-step walking). For both static and dynamic standing, the powered prosthesis could work as a passive one because little energy was needed. In summary, six kinds of locomotion modes (level-ground walking (W), stair ascent (SA), stair descent (SD), ramp ascent (RA), ramp descent (RD) and standing (S)) and ten kinds of locomotion transitions (S→W, W→SA, SA→W, W→RD, RD→W, W→S, W→RA, RA→W, W→SD and SD→W) were tested. Note that transitions of S→W and W→S occurred twice in the experiment trial.
3.2.1. Data for Classifier Training
As the measured leg played different roles in Set-A trials and Set-B trials, especially for locomotion transitions (in Set-A trials, the measured leg performed as the leading leg for transitions of S→W, W→SA, SA→W, W→RD, RD→W, and W→RA; in Set-B trials, the measured leg performed as the following leg for these transitions), there might be some differences between the data measured in the two kinds of experiment trials. In daily life, the leading leg for a locomotion transition is randomly selected (i.e., either leg may perform as the leading leg). To make the classifier suitable for both situations, the data measured in Set-A and Set-B were used together for classifier training. The necessity of this training approach was verified by comparing recognition results with different combinations of training data and testing data.
3.2.2. Data Segmentation and Labeling
To realize real-time locomotion mode recognition, we used overlapped sliding windows for data segmentation. In this case, some analysis windows contained data of two different modes when locomotion transitions occurred. If more than half of the analysis window was before the boundary of the two locomotion modes, the window was labeled as the former mode, otherwise it was labeled as the latter one. In this research, boundaries between two contiguous locomotion modes are defined as follows. For transitions from stand to other locomotion modes, the boundary was the moment when either foot left the ground. For transitions from other locomotion modes to stand, the boundary was the moment when the swing leg contacted the ground before standing still. For the rest locomotion transitions, the boundary was defined as the middle of swing period of the leg which first left the previous terrains. Window increment was 10 ms in this research.
The size of analysis window might be a factor influencing recognition performance. Larger analysis windows contained more information and might increase recognition accuracy, but also increased computation burden and might cause larger time delay for locomotion intent prediction. To systematically investigate the impact of window size on recognition, we varied window size from 100 ms to 200 ms and compared recognition performances.
3.2.3. Two-Level Recognition Strategy
Signals of pressure insoles and IMUs are quasi-cyclic and varied with gait phases. To decrease variances of feature values within class and improve recognition performance, a two-level recognition strategy was utilized (Figure 3). First of all, the gait cycle was segmented into several continuous phases which could be detected with sensory signals; each phase had a corresponding classifier. It is worth emphasizing that the phases should be detected reliably for all the locomotion modes. In the first level of recognition, current gait phase was determined using sensory signals. And in the second level, the recognition result was obtained with the classifier of current gait phase. In our previous research , four nonadjacent phases with predefined sizes were defined, which did not meet the requirement for continuous recognition. In this study, we solved the problem by dividing the gait cycle into four adjacent phases: initial double-limb stance (DS1), single-limb stance (SS), terminal double-limb stance (DS2) and swing (SW). They could be detected with signals measured from both foot pressure insoles. For analysis windows containing signals of two adjacent phases, if more than half of the data belong to the first phase, the analysis window was labeled as the former phase; otherwise it was labeled as the latter phase. Six time-domain feature values (maximum, minimum, mean value, waveform length , standard deviation and root mean square) were calculated for signal channels of pressure insoles as well as IMUs. Feature values of the two kinds of sensor signals were combined together to generate the feature set (Figure 3). Linear discriminant analysis (LDA) classifier, which was similar as the one used in , was selected as the classifier for all the four phases.
Majority voting is a widely used post-processing approach , which utilizes classification results of multiple adjacent analysis windows to produce more accurate recognition decisions. However, potentials exist to improve this approach. In the original majority voting, each decision has the same weight value. This is not efficient when part of the decisions for voting are unreliable. We modified this method by adding weight value for each decision, which was determined by the posterior probability of the recognized decision with LDA classifier. Weight value of the i-th decision for majority voting was defined as
Voting value of Mode-j was calculated by
The final decision was determined by
In this research, the number of decisions for voting was N = 5, thresholds for weight value calculation and final decision determination were p0 = 0.5 and γ = 0.75, respectively.
3.3. Performance Evaluation
To make reasonable evaluations of recognition performance, we divided the locomotion period into steady locomotion periods (i.e., no locomotion transitions) and locomotion transition periods. Locomotion transition periods were defined as follows. For transitions from stand to other locomotion modes, the locomotion transition period began when either foot left the ground and ended at the beginning of the next SS phase. For transitions from other locomotion modes to stand, the transition period began when the swing leg left the ground and ended when the swing leg contacted the ground before standing still. The transition period lasted for the length of a swing phase. For transitions between other locomotion modes, the transition period began when the swing leg left the ground just before transition and ended at the beginning of SS phase immediately after transition. The rest periods of the experiment trial were steady locomotion periods. Since the selection of the leading leg influenced the segmentation of locomotion transition periods, recognition performances of Set-A trials and Set-B trials should be evaluated separately. To reliably evaluate the overall performance of the system, we used leave-one-out-cross-validation (LOOCV) for result calculation and results of Set-A trials and Set-B trials were averaged.
3.3.1. Steady Locomotion Periods
The average classification accuracy (CA) for steady locomotion periods was calculated by
As certain modes tended to be more frequently misclassified as some other modes, we constructed the confusion matrix to quantify the error distribution:
3.3.2. Locomotion Transition Periods
To evaluate whether transition detections were made in time, the critical moment was defined. For transitions from other locomotion modes to stand, the critical moment was the time when the swing leg (either the measured leg or the unmeasured leg) contacted the ground before standing still. For the other locomotion transitions, the critical moment was the beginning of initial double stance (i.e., foot-contact) for the measured foot. A correct locomotion transition detection should follow two rules. First, more than 30 consecutive correct recognition decisions were made. Second, after the first condition was satisfied, no false decisions occurred in the remaining transition period. The moment of the first correct decision was marked as tpre. Prediction time was calculated by
It is worth emphasizing that it's not a mistaken transition prediction when false recognitions were made, as long as they were corrected within the transition period. Considering prediction time as well as the number of missed detections, we calculated the adjusted prediction time (APD) to evaluate the performance of transition detection in the experiment trial. APD was defined as
3.3.3. Long-Time Use and Training Amounts
Apart from safety and powerful functions, long-time use and easy training are also important goals of developing a locomotion intent prediction system for prosthesis control. Complex training process and repeated training make it difficult for the user to accept the prosthesis. To evaluate the performance of our recognition system in long-time use, we used all the data measured in the first section of experiment for classifier training and tested the recognition performance with data measured some time (15 to 300 min) after training in the second section of experiment. Whether recognition performance deteriorated significantly over time can be illustrated by comparing average recognition accuracy and adjusted prediction time of each experiment pair. In addition, we used data of different number (1 to 20) of experiment pairs measured in the first section of the experiment to train the system and compared corresponding recognition results with data measured in the second section of the experiment. The necessary number of training pairs was determined when recognition results didn't improve significantly any more as the number of training pairs increased.
4.1. Data for Classifier Training
Recognition accuracies in steady locomotion periods with different combinations of training and testing data were compared. We found classifiers trained with data of only Set-A trials or with data of only Set-B trials were insufficient to work appropriately for both situations (Table 1). When data of Set-A trials were used for classifier training, average testing accuracy of Set-A trials was 5.81% higher than that of Set-B trials. The difference was statically significant (p < 0.001, one-way repeated measures ANOVA). Similarly, testing accuracy of Set-B trials was 5.63% higher than that of Set-A trials when classifiers were trained with data of Set-B trials, and the difference was also statistically significant (p < 0.005). When classifiers were trained with data of Set-A trials and Set-B trials together, overall recognition accuracy was significantly higher than that when data of only Set-A trials (p < 0.001) or data of only Set-B trials (p < 0.005) were used for training. The results indicated that data of Set-A trials and Set-B trials should be used together for classifier training.
4.2. Influence of Window Size
Window size influenced recognition accuracy in steady locomotion periods and prediction time in locomotion transition periods. The performances tended to be better as window size increased (Figure 4). However, larger analysis window increased computation complexity. Therefore, we should make a trade-off between recognition performance and computation complexity. One-way repeated measures ANOVAs were performed to analysis the influence of window size on recognition performance. Window size varied from 100 ms to 200 ms with 10 ms interval. The main effects were significant for both recognition accuracy and adjusted prediction time (p < 0.001). Pairwise comparisons between successive sizes yielded significant effect until 150 ms for recognition accuracy and 110 ms for adjusted prediction time. Therefore, we selected 150 ms as the optimal window size.
4.3. Modified Post-Processing Approach
The modified post-processing approach performed better than the original majority voting (Figure 5). The recognition error decreased by 0.07% and the difference was statistically significant (p < 0.05). The adjusted prediction time increased by 146 ms and the difference was also statistically significant (p < 0.001). The number of decisions for the original majority voting was selected as 15, which could meet the requirement for recognition accuracy as well as prediction time. Results of both recognition errors and prediction time improved. Though recognition error decreased by only 0.07%, considering recognition errors for modified post-processing approach and the original majority voting were very low (0.29% and 0.36%, respectively), this decrease was relatively obvious. More importantly, these improvements were statically significant. It further demonstrated the modified post-processing approach was more efficient than the original majority voting.
4.4. Recognition Accuracy in Steady Locomotion Periods
Satisfactory recognition accuracy was obtained in steady locomotion periods (Table 2). Average recognition accuracies were 99.50% ± 0.15%, 99.49% ± 0.12%, 99.92% ± 0.04% and 99.59% ± 0.13% for DS1, SS, DS2 and SW, respectively. Most of the recognition errors were caused by the confusion of stair ascent and ramp ascent in DS1 and SW, walk misclassified as ramp descent in DS1 and stair ascent misclassified as walk in SS. The lowest recognition accuracy occurred in the recognition of ramp ascent. However, the accuracy was still as high as 98.90%. It indicated that the system could perform excellently for all the locomotion modes during the whole gait cycle.
4.5. Recognition Performance in Locomotion Transition Periods
Average prediction times and numbers of missed detections of every transition in the experiment trial were calculated (Table 3). Obvious differences of transition detection results between Set-A trials and Set-B trials were observed. As mentioned in Methods, the measured leg performed as different roles for transitions in Set-A trials and Set-B trials. The prediction time of transition was much larger with the unmeasured leg leading than that with the measured leg leading. Though detections for transitions of W→RD and RD→W in Set-A trials, during which the measured leg first transited to new terrains, could not be made before critical moments reliably, the other transitions could all be detected before critical moments. In addition, no missed detections was observed for all the trials of the seven able-bodied subjects.
4.6. Performance of Long-time Use
We evaluated the performance of the intent prediction system when it worked for a long time (up to 5 h) without retraining (Figure 6). The classifiers were trained with the data measured in the first section of the experiment. We used LOOCV to calculate training results of the system (recognition accuracy was 99.71% and adjusted prediction time was 4882 ms), as red lines shown in Figure 6. Average recognition accuracies over the seven subjects varied from 99.05% to 99.74% and average adjusted prediction times varied from 4451 ms to 4976 ms after training, as blue curves shown in Figure 6. Though overall recognition performances after training were a little worse than those of training results, they were still acceptable. In addition, deteriorating tendencies were not observed, unlike the EMG based recognition algorithms.
4.7. Influence of Training Amounts
Recognition performances with different numbers of training pairs were compared (Figure 7). To avoid the same data being used for training as well as testing, we used the data measured in the first section of the experiment for classifier training and all the data measured in the second section for testing. As we could see from Figure 7, satisfactory recognition performance could be obtained with small number of training pairs, and the performance didn't improve significantly as training amounts increased. We performed one-way repeated measures ANOVAs to analysis the influence of training amounts. Pairwise comparisons between successive numbers yielded significant effect until 3 for recognition accuracy and 2 for prediction time. However, recognition accuracy still increased by more than 0.1% as training amounts increased until the number of training pairs reached 6. Therefore, we concluded that data of more than 6 experiment pairs could meet the requirement for classifier training.
4.8. Online Test
To further validate the performance of the locomotion intent prediction system, we also tested it online. Recognition results of Set-A experiment trial and Set-B experiment trial were shown in Figure 8, respectively. The system was trained with data of only 6 experiment pairs and performed well for most of the experiment periods. For the Set-A experiment trial, false recognitions occurred during transitions of W→SA and RA→W, which were caused by the confusion of stair ascent and ramp ascent. However, the periods of false recognition were very short, and recognition stream soon turned to be correct before critical times for transitions. Transition of RD→W was detected 60 ms after critical moment, and the other transitions were predicted in advance. For the Set-B experiment trial, false recognitions occurred during transitions of W→RD and SD→W, which were caused by the confusion of stair descent and ramp descent. But the periods of false recognition were also very short, and recognition stream soon turned to be correct before critical times for transitions. All transitions were successfully predicted in advance. The overall online test performances were in consistence with the offline evaluation results.
Note that a video is included in the supplemental material, which recorded online test experiments of a Set-A trial and a Set-B trial.
We developed a promising locomotion intent prediction system for the control of powered lower-limb prostheses. With the fusion of IMU signals and foot pressure signals at the feature level, the system realized real-time continuous recognition of locomotion modes with high accuracy and could even detect transitions between different locomotion modes in advance for most of the transitions. In addition, with small amount of training data, the system could perform satisfactory and reliable work for a long time without retraining. These results demonstrated the prospect of realizing safe and fluent control for powered lower-limb prostheses with this locomotion intent prediction system.
The locomotion intent prediction system is promising to improve the performance of powered lower-limb prosthesis for the following reasons. First, the system performs well for frequently occurred locomotion modes and locomotion transitions in daily life. To evaluate whether this system can meet the requirement of daily activities, we considered more locomotion modes and locomotion transitions than previous studies [14,15,18]. More importantly, overall recognition accuracy (99.71%) and the performance of transition detection (no missed detection) are excellent. For most of the locomotion transitions, the system can detect them in advance, which leaves more time for the powered prosthesis to react to terrain changes. Though transitions of W→RD and RD→W in Set-A trials could not be detected before critical moments reliably, the impacts on powered prosthesis control are not significant, because control performance also depends on other factors such as mechanical property and control strategy of the powered prosthesis. Some powered lower-limb prostheses tend to be passive for a period of time after foot contact (i.e., the critical moment), during which no motor control is needed [1,2,11]. Therefore, transition detections with small time delays can be accepted.
Second, the system can meet the requirement of naturally selection of leading leg for locomotion transitions. The goal of powered lower-limb prosthesis research is enabling amputees to walk with normal gaits. Therefore, the system is designed with the same consideration. In the experiment, two legs should naturally alternate during walking in different terrains and recognition performances of both Set-A and Set-B trials should be acceptable. We used data of Set-A trials and data of Set-B trials together for classifier training. Recognition accuracies in steady locomotion periods are above 99% for both Set-A and Set-B trials (Table 1). However, as less time is left for transition detection when the measured leg first transited to a new locomotion mode, it is more difficult to make an accurate detection (Table 3). We report transition detection results of both Set-A and Set-B trials, and validate they are acceptable for both situations.
Third, the system is easy trained and capable of long-time use without retraining. As we can see from Figure 7, data of 6 or more experiment pairs (i.e., 12 experiment trials) are enough to train a reliable system. In this research, the measured time of each experiment trial is 40 s. Therefore, the training process only takes about 15 min, including data processing and rest time. More importantly, the performance of the system is close to the training results even after five hours (Figure 6). Decaying tendency of recognition performance which occurred in EMG based recognition systems  was not observed. Thus, amputees don't need to worry about cumbersome system training and possible frequent retraining.
Additionally, the system is easy to integrate with a powered lower-limb prosthesis in hardware as well as in control strategy. To our knowledge, some previous studies have implemented recognition algorithms in embedded systems [44,45]. And as signals measured from IMUs have clear physical meanings, they have already been integrated with some prosthetic systems and used for prosthesis control [10,46,47]. In addition, some of the existing powered lower-limb prostheses are controlled with finite-state-machine [1,2,7,11]. But most of them required users to manually switch the control mode, which is inconvenient for practical application. The proposed recognition system can be used to detect current locomotion mode, and automatically select the appropriate control mode.
Our intent prediction system show great potential for powered lower-limb prosthesis control. However, deficiencies and improvement spaces exist for the two-level recognition strategy. First of all, gait phases are determined using foot-contact information of both feet. However, it is cumbersome to wear an extra pressure insole on the unmeasured side, which may be a limitation for clinical applications. We will find new ways to segment the gait cycle with sensor signals measured from only the measured leg by adding extra sensors, such as a load cell integrated with the prosthesis . Second, it is not necessary for all the phases to use the same classifier, feature set, window size and other parameters for classification and post-processing, because signals of different phases have their own characteristics. For example, IMU signals vary a lot in SW phase, while has relatively small variance in SS phase. We will perform parameter optimization individually for each phase, and it could make the locomotion intent prediction system more efficient. In the future, we will test the system on amputee subjects with powered lower-limb prostheses worn and evaluate the performance of prosthesis control.
This work was supported by the National Natural Science Foundation of China (No. 61005082, 61020106005), Beijing Nova Program (No. Z141101001814001), PKU-Biomedical Engineering Joint Seed Grant 2014 and the 985 Project of Peking University (No. 3J0865600).
Conflict of Interest
The authors declare no conflict of interest.
- Au, S.; Weber, J.; Herr, H. Powered ankle-foot prosthesis improves walking metabolic economy. IEEE Trans. Robot. 2009, 25, 51–66. [Google Scholar]
- Sup, F.; Varol, H.; Mitchell, J.; Withrow, T.; Goldfarb, M. Preliminary evaluations of a self-contained anthropomorphic transfemoral prosthesis. IEEE/ASME Trans. Mechatron. 2009, 14, 667–676. [Google Scholar]
- Hitt, J.; Sugar, T.; Holgate, M.; Bellmann, R.; Hollander, K. Robotic transtibial prosthesis with biomechanical energy regeneration. Ind. Robot. 2009, 36, 441–447. [Google Scholar]
- Versluys, R.; Desomer, A.; Lenaerts, G.; van Damme, M.; Bey, P.; Van der Perre, G.; Peeraer, L.; Lefeber, D. A pneumatically powered below-knee prosthesis: Design specifications and first experiments with an amputee. Proceedings of the 2nd IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Scottsdale, AZ, USA, 19–22 October 2008; pp. 372–377.
- Bellman, R.; Holgate, M.; Sugar, T. SPARKy 3: Design of an active robotic ankle prosthesis with two actuated degrees of freedom using regenerative kinetics. Proceedings of the 2nd IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Scottsdale, AZ, USA, 19–22 October 2008; pp. 511–516.
- Zhu, J.; Wang, Q.; Wang, L. On the design of a powered transtibial prosthesis with stiffness adaptable ankle and toe joints. IEEE Trans. Ind. Electron. 2014, 61, 4797–4807. [Google Scholar]
- Zlatnik, D.; Steiner, B.; Schweitzer, G. Finite-state control of a trans-femoral (TF) prosthesis. IEEE Trans. Control Syst. Technol. 2002, 10, 408–420. [Google Scholar]
- Martinez-Villalpando, E.C.; Herr, H. Agonist-antagonist active knee prosthesis: A preliminary study in level-ground walking. J. Rehabil. Res. Dev. 2009, 46, 361–373. [Google Scholar]
- Eilenberg, M.F.; Geyer, H.; Herr, H. Control of a powered ankle-foot prosthesis based on a neuromuscular model. IEEE Trans. Neural. Syst. Rehabil. Eng. 2010, 18, 164–173. [Google Scholar]
- Sup, F.; Varol, H.A.; Goldfarb, M. Upslope walking with a powered knee and ankle prosthesis: Initial results with an amputee subject. IEEE Trans. Neural. Syst. Rehabil. Eng. 2011, 19, 71–78. [Google Scholar]
- Yuan, K.; Zhu, J.; Wang, Q.; Wang, L. Finite-state control of powered below-knee prosthesis with ankle and toe. Proceedings of the 18th IFAC World Congress, Milano, Italy, 28 August–2 September 2011; pp. 2865–2870.
- Jiménez-Fabián, R.; Verlinden, O. Review of control algorithms for robotic ankle systems in lower-limb orthoses, prostheses, and exoskeletons. Med. Eng. Phys. 2012, 34, 397–408. [Google Scholar]
- Varol, H.A.; Sup, F.; Goldfarb, M. Powered sit-to-stand and assistive stand-to-sit framework for a powered transfemoral prosthesis. Proceedings of the IEEE International Conference on Rehabilitation Robotics, Kyoto, Japan, 23–26 June 2009; pp. 645–651.
- Varol, H.A.; Sup, F.; Goldfarb, M. Multiclass real-time intent recognition of a powered lower limb prosthesis. IEEE Trans. Biomed. Eng. 2010, 57, 542–551. [Google Scholar]
- Young, A.J.; Simon, A.M.; Hargrove, L.J. A training method for locomotion mode prediction using powered lower limb prostheses. IEEE Trans. Neural. Syst. Rehabil. Eng. 2014, 22, 671–677. [Google Scholar]
- Peeraer, L.; Aeyels, B.; Van Der Perre, G. Development of EMG based mode and intent recognition algorithms for a computer-controlled above-knee prosthesis. J. Biomed. Eng. 1990, 12, 178–182. [Google Scholar]
- Huang, H.; Kuiken, T.A.; Lipschutz, R.D. A strategy for identifying locomotion modes using surface electromyography. IEEE Trans. Biomed. Eng. 2009, 56, 65–72. [Google Scholar]
- Huang, H.; Zhang, F.; Hargrove, L.; Dou, Z.; Rogers, D.; Englehart, K. Continuous locomotion mode identification for prosthetic legs based on neuromuscular-mechanical fusion. IEEE Trans. Biomed. Eng. 2011, 58, 2867–2875. [Google Scholar]
- Hargrove, L.J.; Simon, A.M.; Young, A.J.; Lipschutz, R.D.; Finucane, S.B.; Smith, D.G.; Kuiken, T.A. Robotic leg control with EMG decoding in an amputee with nerve transfers. N. Engl. J. Med. 2013, 369, 1237–1242. [Google Scholar]
- Miller, J.D.; Beazer, M.S.; Hahn, M.E. Myoelectric walking mode classification for transtibial amputees. IEEE Trans. Biomed. Eng. 2013, 60, 2745–2750. [Google Scholar]
- Jain, S.; Singhal, G.; Smith, R.J.; Kaliki, R.; Thakor, N. Improving long term myoelectric decoding, using an adaptive classifier with label correction. Proceedings of the 4th IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, Rome, Italy, 24–27 June 2012; pp. 532–537.
- Sensinger, J.W.; Lock, B.A.; Kuiken, T.A. Adaptive pattern recognition of myoelectric signals: Exploration of conceptual framework and practical algorithms. IEEE Trans. Neural. Syst. Rehabil. Eng. 2009, 17, 270–278. [Google Scholar]
- Khaleghi, B.; Khamis, A.; Karray, F.O.; Razavi, S.N. Multisensor data fusion: A review of the state-of-the-art. Inf. Fusion 2013, 14, 28–44. [Google Scholar]
- Luo, R.C.; Su, K.L. Autonomous fire-detection system using adaptive sensory fusion for intelligent security robot. IEEE/ASME Trans. Mechatron. 2007, 12, 274–281. [Google Scholar]
- Marín, L.; Vallés, M.; Soriano, Á.; Valera, Á.; Albertos, P. Multi sensor fusion framework for indoor-outdoor localization of limited resource mobile robots. Sensors 2013, 13, 14133–14160. [Google Scholar]
- Ren, H.; Kazanzides, P. Investigation of attitude tracking using an integrated inertial and magnetic navigation system for hand-held surgical instruments. IEEE/ASME Trans. Mechatron. 2012, 17, 210–217. [Google Scholar]
- Smith, D.; Singh, S. Approaches to multisensor data fusion in target tracking: A survey. IEEE Trans. Knowl. Data Eng. 2006, 18, 1696–1710. [Google Scholar]
- Musleh, B.; García, F.; Otamendi, J.; Armingol, J.M.; De la Escalera, A. Identifying and tracking pedestrians based on sensor fusion and motion stability predictions. Sensors 2010, 10, 8028–8053. [Google Scholar]
- Fotiadis, E.P.; Garzón, M.; Barrientos, A. Human detection from a mobile robot using fusion of laser and vision information. Sensors 2013, 13, 11603–11635. [Google Scholar]
- Ben-Yacoub, S.; Abdeljaoued, Y.; Mayoraz, E. Fusion of face and speech data for person identity verification. IEEE Trans. Neural Netw. 1999, 10, 1065–1074. [Google Scholar]
- Banos, O.; Damas, M.; Pomares, H.; Rojas, I. On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 2012, 12, 8039–8054. [Google Scholar]
- Guiry, J.J.; van de Ven, P.; Nelson, J. Multi-sensor fusion for enhanced contextual awareness of everyday activities with ubiquitous devices. Sensors 2014, 14, 5687–5701. [Google Scholar]
- Kotiadis, D.; Hermens, H.J.; Veltink, P.H. Inertial gait phase detection for control of a drop foot stimulator inertial sensing for gait phase detection. Med. Eng. Phys. 2010, 32, 287–297. [Google Scholar]
- Giansanti, D.; Dozza, A.; Chiari, L. Energetic assessment of trunk postural modifications induced by a wearable audio-biofeedback system. Med. Eng. Phys. 2009, 31, 48–54. [Google Scholar]
- Saitoa, M.; Nakajimaa, K.; Takanoe, C.; Ohtaa, Y.; Sugimotob, C.; Ezoeb, R.; Sasakib, K.; Hosakab, H.; Ifukubec, T.; Inof, S.; et al. An in-shoe device to measure plantar pressure during daily human activity. Med. Eng. Phys. 2011, 33, 638–645. [Google Scholar]
- Yun, X.; Bachmann, E.R. Design, implementation, and experimental results of a quaternion-based Kalman filter for human body motion tracking. IEEE Trans. Robot. 2006, 22, 1216–1277. [Google Scholar]
- Wang, X.; Wang, Q.; Zheng, E.; Wei, K.; Wang, L. A wearable plantar pressure measurement system: Design specifications and first experiments with an amputee. Proceedings of the 12th International Conference of Intelligent Autonomous Systems, Jeju, South Korea, 26–29 June 2012; pp. 273–281.
- Bamberg, S.J.M.; Benbasat, A.Y.; Scarborough, D.M.; Krebs, D.E.; Paradiso, J.A. Gait analysis using a shoe-integrated wireless sensor system. IEEE Trans. Inform. Techn. Biomed. 2008, 12, 413–423. [Google Scholar]
- Senanayake, C.M.; Senanayake, S.M.N.A. Evaluation of gait parameters for gait phase detection during walking. Proceedings of 2010 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Salt Lake, UT, USA, 5–7 September 2010; pp. 127–132.
- Chen, B.; Zheng, E.; Fan, X.; Liang, T.; Wang, Q.; Wei, K.; Wang, L. Locomotion mode classification using a wearable capacitive sensing system. IEEE Trans. Neural. Syst. Rehabil. Eng. 2013, 21, 744–755. [Google Scholar]
- Hudgins, B.; Parker, P.; Scott, R.N. A new strategy for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 1993, 40, 82–94. [Google Scholar]
- Huang, H.; Zhou, P.; Li, G.; Kuiken, T.A. An analysis of EMG electrode configuration for targeted muscle reinnervation based neural machine interface. IEEE Trans. Neural. Syst. Rehabil. Eng. 2008, 16, 37–45. [Google Scholar]
- Englehart, K.; Hudgins, B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 2003, 50, 848–854. [Google Scholar]
- Zhang, X.; Huang, H.; Yang, Q. Implementing an FPGA system for real-time intent recognition for prosthetic legs. Proceedings of the 49th Annual Design Automation Conference, San Francisco, CA, USA, 3–7 June 2012; pp. 169–175.
- Hernandez, R.; Yang, Q.; Huang, H.; Zhang, F.; Zhang, X. Design and implementation of a low power mobile CPU based embedded system for artificial leg control. Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Osaka, Japan, 3–7 July 2013; pp. 5769–5772.
- Herr, H.M.; Grabowski, A.M. Bionic ankle-foot prosthesis normalizes walking gait for persons with leg amputation. Proc. R. Soc. B 2012, 279, 457–464. [Google Scholar]
- Wang, Q.; Yuan, K.; Zhu, J.; Wang, L. Finite-state control of a robotic transtibial prosthesis with motor-driven nonlinear damping behaviors for level ground walking. Proceedings of the 13th International Workshop on Advanced Motion Control, Yokohama, Japan; 2014; pp. 155–160. [Google Scholar]
- Zhang, F.; Di Santo, W.; Ren, J.; Dou, Z.; Yang, Q.; Huang, H. A novel CPS system for evaluating a neural-machine interface for artificial legs. Proceedings of 2011 IEEE/ACM International Conference on Cyber-Physical Systems, Chicago, IL, USA, 12–14 April 2011; pp. 67–76.
|Training||Set-A||Set-B||Set-A & Set-B|
|Set-A||99.26 ± 0.21||93.45 ± 0.68||96.35 ± 0.38|
|Set-B||93.61 ± 1.26||99.24 ± 0.36||96.43 ± 0.76|
|Set-A & Set-B||99.37 ± 0.14||99.64 ± 0.19||99.51 ± 0.13|
|Phase||Targeted Mode||Stand||Walk||Stair Ascent||Stair Descent||Ramp Ascent||Ramp Descent|
|DS1||Stand||99.96 ± 0.03||0.00 ± 0.00||0.00 0.00||0.00 ± 0.00||0.03 ± 0.03||0.00 ± 0.00|
|Walk||0.00 ± 0.00||98.99 ± 0.29||0.00 ± 0.00||0.20 ± 0.13||0.02 ± 0.02||0.78 ± 0.27|
|Stair Ascent||0.00 ± 0.00||0.00 ± 0.00||99.53 ± 0.18||0.00 ± 0.00||0.47 ± 0.18||0.00 ± 0.00|
|Stair Descent||0.10 ± 0.10||0.30 ± 0.21||0.00 ± 0.00||99.50 ± 0.24||0.00 ± 0.00||0.09 ± 0.07|
|Ramp Ascent||0.00 ± 0.00||0.01 ± 0.01||0.84 ± 0.24||0.00 ± 0.00||99.15 ± 0.18||0.00 ± 0.00|
|Ramp Descent||0.00 ± 0.00||0.10 ± 0.05||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||99.90 ± 0.05|
|SS||Stand||99.76 ± 0.07||0.24 ± 0.07||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Walk||0.22 ± 0.19||99.35 ± 0.16||0.15 ± 0.05||0.24 ± 0.05||0.00 ± 0.00||0.04 ± 0.02|
|Stair Ascent||0.23 ± 0.23||0.61 ± 0.37||99.13 ± 0.40||0.02 ± 0.02||0.00 ± 0.00||0.00 ± 0.00|
|Stair Descent||0.38 ± 0.30||0.25 ± 0.17||0.02 ± 0.02||99.17 ± 0.49||0.00 ± 0.00||0.18 ± 0.18|
|Ramp Ascent||0.19 ± 0.19||0.06 ± 0.04||0.00 ± 0.00||0.00 ± 0.00||99.75 ± 0.23||0.00 ± 0.00|
|Ramp Descent||0.20 ± 0.20||0.01 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||99.79 ± 0.20|
|DS2||Stand||99.97 ± 0.03||0.03 ± 0.03||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Walk||0.00 ± 0.00||99.92 ± 0.06||0.08 ± 0.06||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Stair Ascent||0.00 ± 0.00||0.11 ± 0.07||99.89 ± 0.07||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Stair Descent||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||100.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Ramp Ascent||0.00 ± 0.00||0.06 ± 0.04||0.14 ± 0.09||0.06 ± 0.06||99.74 ± 0.15||0.00 ± 0.00|
|Ramp Descent||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||100.00 ± 0.00|
|SW||Stand||99.87 ± 0.05||0.13 ± 0.05||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00|
|Walk||0.00 ± 0.00||99.81 ± 0.12||0.00 ± 0.00||0.00 ± 0.00||0.00 ± 0.00||0.19 ± 0.12|
|Stair Ascent||0.00 ± 0.00||0.00 ± 0.00||99.75 ± 0.12||0.00 ± 0.00||0.25 ± 0.12||0.00 ± 0.00|
|Stair Descent||0.00 ± 0.00||0.17 ± 0.11||0.00 ± 0.00||99.60 ± 0.21||0.16 ± 0.11||0.07 ± 0.05|
|Ramp Ascent||0.00 ± 0.00||0.00 ± 0.00||1.10 ± 0.34||0.00 ± 0.00||98.90 ± 0.34||0.00 ± 0.00|
|Ramp Descent||0.00 ± 0.00||0.18 ± 0.18||0.00 ± 0.00||0.21 ± 0.11||0.00 ± 0.00||99.61 ± 0.18|
|Set-A Trials||Set-B Trials|
|Transitions||Prediction Time (ms)||Number of Missed Detections||Prediction Time (ms)||Number of Missed Detections|
|S→W||451 ± 10||0||1035 ± 21||0|
|W→AS||222 ± 14||0||628 ± 20||0|
|AS→W||204 ± 9||0||732 ± 29||0|
|W→DR||27 ± 33||0||508 ± 19||0|
|DR→W||−120 ± 13||0||284 ± 35||0|
|W→S||256 ± 19||0||110 ± 16||0|
|S→W||396 ± 17||0||891 ± 21||0|
|W→AR||116 ± 17||0||447 ± 19||0|
|AR→W||822 ± 28||0||270 ± 13||0|
|W→DS||887 ± 34||0||210 ± 14||0|
|DS→W||886 ± 37||0||117 ± 13||0|
|W→S||241 ± 23||0||145 ± 17||0|
Note: A negative value of prediction time indicates the decision is made after the critical moment.
© 2014 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license ( http://creativecommons.org/licenses/by/3.0/).