A New Time–Frequency Feature Extraction Method for Action Detection on Artificial Knee by Fractional Fourier Transform

With the aim of designing an action detection method on artificial knee, a new time–frequency feature extraction method was proposed. The inertial data were extracted periodically using the microelectromechanical systems (MEMS) inertial measurement unit (IMU) on the prosthesis, and the features were extracted from the inertial data after fractional Fourier transform (FRFT). Then, a feature vector composed of eight features was constructed. The transformation results of these features after FRFT with different orders were analyzed, and the dimensions of the feature vector were reduced. The classification effects of different features and different orders are analyzed, according to which order and feature of each sub-classifier were designed. Finally, according to the experiment with the prototype, the method proposed above can reduce the requirements of hardware calculation and has a better classification effect. The accuracies of each sub-classifier are 95.05%, 95.38%, 91.43%, and 89.39%, respectively; the precisions are 78.43%, 98.36%, 98.36%, and 93.41%, respectively; and the recalls are 100%, 93.26%, 86.96%, and 86.68%, respectively.


Introduction
The knee joint is an important supporting joint of the lower limbs. For a knee joint injury that seriously impedes the movement of the lower limbs, the best treatment is to replace the joint with an artificial joint. Among patients, a large number need amputation, and prosthesis is an important guarantee for the physical and physiological recovery of amputees. In the lower limb prosthesis, the artificial knee joint is the core component and it can help the human body to achieve supported standing and flexible walking so that patients show good gait [1].
Before the 1990s, the movement of prosthetics could not keep pace with the speed of amputees and road conditions, and the movement lacked stability [2]. With the continuous development of bionic intelligent knee prosthesis technology, there are many intelligent knee prostheses, which can achieve dynamic adjustment, real control of gait, and other functions. The intelligent lower limb usually refers to the unpowered intelligent lower limb. The flexion and extension movement of the prosthetic knee joint is driven by the leg stump. The intelligent control system only adjusts the size of the damping torque of the knee joint [3]. For the knee joint of the unpowered intelligent lower limb prosthesis that controls gait by adjusting the damping torque, the accuracy of damping adjustment directly affects the comfort, convenience and safety of the knee joint in question. Generally, the damping adjustment is based on the recognition of the current action state. Among the existing action detection methods of intelligent prosthetics, methods that detect the motion frequency or the position of the piston of a cylinder and the pressure at the bottom of the foot have long lag times and low accuracy. Methods that extract the features of acceleration or angular velocity from the time domain have few features from which to choose, hence they are susceptible to interference and low robustness. So, an accurate, fast, low-cost and stable action detection method is needed.
The action detection often needs to go through the process of data collection, data preprocessing, feature extraction, the establishment of an action detection model, and the use of the model for action detection [4]. Currently, sensors used to collect data can be divided into two major categories of wearable and nonwearable. The nonwearable methods include motion detection methods based on radar [5], and machine vision [6]. The wearable methods include motion detection methods based on accelerometers [7], accelerometers and gyroscopes [8], accelerometers and other sensors [9], and myoelectricity [10]. The nonwearable sensors are not suitable for human motion recognition on prosthesis due to their severe constraints of scene and high cost. In the wearable methods, myoelectricity is inconvenient to wear on a daily basis due to the need to install electrodes in multiple places on the wearer's legs. The microelectromechanical systems (MEMS) inertial measurement unit (IMU) integrated with accelerometers, gyroscopes and barometers can be integrated into the control circuit and does not require additional operation. So, it is more suitable for human action data acquisition on commercial products.
For feature extraction, the preprocessed data are usually divided into a series of windows by the sliding window mechanism of fixed length first-in, first-out, and the feature values are extracted by a series of methods. Aziz et al., for example, calculated the mean and variance of data in each window to construct 18 dimensional features [11]; Pierleoni et al. discovered that the RMS (Root Mean Square) of acceleration was no less than 2.5 g when the impact process of motion occurs [12]; Cheng et al. judged the state of motion according to an amplitude of acceleration higher than a given threshold [13].
For the establishment of an action detection model, the commonly used methods are the threshold method and the machine learning method. The threshold method determines motion state by comparing the extracted features with a setting threshold; generally, it can be divided into two types: a single state recognition and a multiple state recognition. The machine learning method regards action detection as a typical classification problem and constructs a motion detection model based on the training set composed of various motion data. Typical machine learning algorithms such as support vector machine (SVM) [11,[14][15][16], decision tree (DT), Naive Bayes [17], deep learning [18], artificial neural network (ANN) [19], and K nearest neighbor (K-NN) [20] can be used for model construction. The preferable effective algorithms include convolutional neural network (CNN), I support vector machine (1SVM), CNN+1SVM [21], hidden Markova model [22], K-NN and ANN [23].
However, in most action detection methods the machine learning method can achieve higher precision, but it needs lots of features and is a complicated detection model with a slow detection speed and high calculate cost. The threshold method is simple in calculation and fast in speed, which is very suitable for long-term operation on micro-wearable devices with limited resources. However, the selection of its threshold value often depends on the simulated motion data, and there are few features from which to choose. When the distribution of test data and training data is greatly different, it will seriously affect the detection accuracy of the model, so it lacks certain applicability. Therefore, a detection method that can provide sufficient features for selection, at the same time as satisfying high computing speed and low model complexity, is needed. Considering that human action is a time-varying movement, and we cannot only look for features in the time domain or frequency domain, we came up with the idea of using the fractional Fourier transform (FRFT) in order to provide sufficient features from the fractional domain, and also in order to obtain suitable features for low power consumption wearable devices.
Fractional Fourier transform (FRFT) is a representation method on the fractional-order Fourier domain by the signal on the time-frequency plane after the coordinate axis rotates counterclockwise around the origin at any angle. It is a time-frequency analysis method and a generalized Fourier transform. Fractional Fourier transform has many properties that traditional Fourier transform does not have, and is widely applied in scientific research and engineering technology [24]. In this paper, firstly the effect of FRFT is analyzed, and the results of transformation with the same action in different orders, or different actions in the same order, are compared. Then, eight features are selected from the commonly used features to construct a feature vector space. The variation of each feature after FRFT in different orders is analyzed. Then, the dimension of the feature vector is further reduced, and the results of different features after FRFT in different orders are analyzed, according to which we design a classifier. Finally, a prototype of an intelligent knee joint is designed, and the method is verified by experiments.

Fractional Fourier Transform (FRFT)
In the field of signal processing, the traditional Fourier transform is a mature and widely used mathematical tool. The fractional order Fourier transform (FRFT) is proposed in the form of pure mathematics by V. Namias from the view of feature and feature function [25]. Then, researchers proposed the concept of FRFT from an optical point of view. It can be proved that these definitions are completely equivalent [26]. FRFT is first applied to optical signal processing because it can be implemented by simple optical devices. In recent years, several rapid algorithms for FRFT have been discovered, so that FRFT has received attention in multiple areas of signal processing.

Definition
Generally, the p-order fractional Fourier transform of a function f p (u) can be expressed as: f p (u) or F p f (u), where, F p f (u) can be interpreted as the operator F p on the function f (u) whose result is in the domain u.
Except for comparison with the time domain or the frequency domain, to explain the reason that choose FRFT to extract features, Li Chao derived the relationship between the amplitude mean A in the time domain and the amplitude B in the fractional domain by the timewidth-bandwidth in theory [28].
From the fraction domain sampling theorem, the relationship between the fraction domain bandwidth Bu and the frequency domain bandwidth B is: where α is the rotation angle of the fraction domain.
The timewidth and bandwidth of the fraction domain signal are defined as follows: where u a is the frequency of the fractional domain and x(u a ) is the FRFT of x(t).
From the uncertainty principle of the signal frequency domain: where ∆t is the timewidth and ∆u is the bandwidth. The uncertainty principle of the fractional domain can be derived from the three Equations (4)- (6): From Parseval, the energy of the signal in the time domain is the same as that in the fractional domain: where E is the total energy in the time domain, and E α is the total energy in fraction domain. Furthermore: Combining Equations (7)-(9), it can be observed that the signal's amplitude average A in the time domain and the B of the signal show a non-linear change, and it is related to the changed angle in the fractional domain [29,30].
Therefore, a sequence of values with no significant difference in amplitude in time domain can be converted into a fractional sequence with significant differences in amplitude by choosing an appropriate FRFT order [28].

Discrete FRFT
Equation (2) is the calculation method in the continuous domain. Such continuous transformation cannot be calculated in practice, and it is usually necessary to carry out numerical calculation by sampling and interpolating continuous signals. According to the Shannon interpolation equation and numerical integration operation, the discrete calculation equation of FRFT is shown in Equation (10): After sorting, it is shown as follows: where E t (u k ) = exp(− jπtu 2 k ), t = tan(α/2), s = − csc(α) and x k = u k = k/2∆. ∆ is the dimension normalized time domain or frequency domain scale.

Transform Effect
According to the theory above, the inertial data collected by MEMS can be processed by FRFT. In contrast to the time domain and the frequency domain, the fractional domain has a better diversity and can provide more optional features. Figures 1 and 2 are the results of the same data after FRFT in different orders. Figure 3a is the results of the same class data after FRFT in the same order. Figure 3b is the results of different data after FRFT in the same order. It can be observed that the results of the same data after FRFT in different orders show significant differences, and the results of different data after FRFT in the same order also show differences. Therefore, different actions can be distinguished by selecting an appropriate order and an appropriate feature.
Micromachines 2019, 10, x FOR PEER REVIEW 5 of 19  is the dimension normalized time domain or frequency domain scale.

Transform Effect
According to the theory above, the inertial data collected by MEMS can be processed by FRFT. In contrast to the time domain and the frequency domain, the fractional domain has a better diversity and can provide more optional features. Figures 1 and 2 are the results of the same data after FRFT in different orders. Figure 3a is the results of the same class data after FRFT in the same order. Figure 3b is the results of different data after FRFT in the same order. It can be observed that the results of the same data after FRFT in different orders show significant differences, and the results of different data after FRFT in the same order also show differences. Therefore, different actions can be distinguished by selecting an appropriate order and an appropriate feature.   Micromachines 2019, 10, x FOR PEER REVIEW 5 of 19  is the dimension normalized time domain or frequency domain scale.

Transform Effect
According to the theory above, the inertial data collected by MEMS can be processed by FRFT. In contrast to the time domain and the frequency domain, the fractional domain has a better diversity and can provide more optional features. Figures 1 and 2 are the results of the same data after FRFT in different orders. Figure 3a is the results of the same class data after FRFT in the same order. Figure 3b is the results of different data after FRFT in the same order. It can be observed that the results of the same data after FRFT in different orders show significant differences, and the results of different data after FRFT in the same order also show differences. Therefore, different actions can be distinguished by selecting an appropriate order and an appropriate feature.

Feature
After the performance of FRFT on the collected data, the appropriate features should be extracted and then classified according to the features. The commonly used features are shown in Table 1. However, not all the features are appropriate for the data in MEMS IMU, besides, too many features will lead to excessive computing costs, time costs, storage costs, and reduce the endurance of the device. However, too few features will not reach enough accuracy to classify different actions. Therefore, it is necessary to select appropriate features for action recognition.

Feature
After the performance of FRFT on the collected data, the appropriate features should be extracted and then classified according to the features. The commonly used features are shown in Table 1. However, not all the features are appropriate for the data in MEMS IMU, besides, too many features will lead to excessive computing costs, time costs, storage costs, and reduce the endurance of the device. However, too few features will not reach enough accuracy to classify different actions. Therefore, it is necessary to select appropriate features for action recognition. Table 1. The commonly used features and the calculation method.

Features Calculation Method
Maximum The number of peaks in signal (pksNum) Mean of Peaks pksMean = i pks i n In this paper, the extreme difference range, standard deviation std, variance var, interquartile range IQR, mean mean, mean of peaks pksMean, and number of peaks pksNum were selected to form the f eature_vector: f eature_vector = [range, std, var, rms, IQR, mean, pksMean, PksNum]  Tables 2-5 are the mean, minimum standard deviation and corresponding order of standard deviation of each feature of each action. It can be observed that, except for rms, all the other features change significantly, but this does not mean that rms cannot be used for classification. Besides, it can be observed that it might be useful to classify walk and dwstairs by rms and mean, to classify upstairs and dwstairs by range and pksNum, and to classify walk, dwstairs and run by std and pksNum. upstairs, and dwstairs (downstairs), in different orders. Figures 5,7,9 and 11 are the standard deviation and mean of eight features of each action. Tables 2-5 are the mean, minimum standard deviation and corresponding order of standard deviation of each feature of each action. It can be observed that, except for rms , all the other features change significantly, but this does not mean that rms cannot be used for classification. Besides, it can be observed that it might be useful to classify walk and dwstairs by rms and mean , to classify upstairs and dwstairs by range and pksNum , and to classify walk, dwstairs and run by std and pksNum .     Tables 2-5 are the mean, minimum standard deviation and corresponding order of standard deviation of each feature of each action. It can be observed that, except for rms , all the other features change significantly, but this does not mean that rms cannot be used for classification. Besides, it can be observed that it might be useful to classify walk and dwstairs by rms and mean , to classify upstairs and dwstairs by range and pksNum , and to classify walk, dwstairs and run by std and pksNum .                 Figure  12 shows the effect of distinguishing two actions by two kinds of features. It can be observed that to classify walk and dwstairs, the combination of pksNum and std is better than the combination of range and pksMean , and the order = 0.67 is better than order = 0.20. To classify upstairs and dwstairs, the combination of range and pksNum is better than the combination of rms and IQR , and the order = 0.64 is better than order = 0.20. To classify walk and upstairs, the combination of rms and mean is better than the combination of std and IQR , and the order = 0.71 is better than order = 0.20. To separate run from the other actions, the combination of rms and mean is better than the combination of range and IQR , and it is better when order = 0.75. Figure 13 shows the effect of classification with two features extracted directly in the time domain without FRFT. It can be observed that the effect was not good for classification.  Different features have different classification effects for the inertia data in MEMS IMU. Figure 12 shows the effect of distinguishing two actions by two kinds of features. It can be observed that to classify walk and dwstairs, the combination of pksNum and std is better than the combination of range and pksMean, and the order = 0.67 is better than order = 0.20. To classify upstairs and dwstairs, the combination of range and pksNum is better than the combination of rms and IQR, and the order = 0.64 is better than order = 0.20. To classify walk and upstairs, the combination of rms and mean is better than the combination of std and IQR, and the order = 0.71 is better than order = 0.20. To separate run from the other actions, the combination of rms and mean is better than the combination of range and IQR, and it is better when order = 0.75. Figure 13 shows the effect of classification with two features extracted directly in the time domain without FRFT. It can be observed that the effect was not good for classification.

Classifier.
According to the conclusion above, different features and different orders have different effects for classification. The optimal order and feature of each action are different. A binary classifier was designed which classifies by changing the order and feature value of each sub-classifier. Only one action is separated at a time, and then all motion gestures are recognized by multiple classifications. The structure of the classifier is shown in Figure 14.

Classifier
According to the conclusion above, different features and different orders have different effects for classification. The optimal order and feature of each action are different. A binary classifier was designed which classifies by changing the order and feature value of each sub-classifier. Only one action is separated at a time, and then all motion gestures are recognized by multiple classifications. The structure of the classifier is shown in Figure 14.  Figure 14. The structure and process of the classifier. As different orders and feature vectors are needed to classify walk, upstairs and dwstairs, one can distinguish walk and dwstairs first, and then separate upstairs from walk and dwstairs, respectively, then separate upstairs from walk and dwstairs, respectively.

Experiment Design
According to the conclusion above, an experiment for verification is designed. The information of the subject is shown in Table 6. To protect the privacy of amputees, the photos of amputees will not be shown here. The artificial knee used in the experiment is shown in Figure 15, the MEMS IMU used in the experiment is shown in Figure 16, and the main specifications of the devices are given in Table 7. Figure 14. The structure and process of the classifier. As different orders and feature vectors are needed to classify walk, upstairs and dwstairs, one can distinguish walk and dwstairs first, and then separate upstairs from walk and dwstairs, respectively, then separate upstairs from walk and dwstairs, respectively.

Experiment Design
According to the conclusion above, an experiment for verification is designed. The information of the subject is shown in Table 6. To protect the privacy of amputees, the photos of amputees will not be shown here. The artificial knee used in the experiment is shown in Figure 15, the MEMS IMU used in the experiment is shown in Figure 16, and the main specifications of the devices are given in Table 7.  Figure 14. The structure and process of the classifier. As different orders and feature vectors are needed to classify walk, upstairs and dwstairs, one can distinguish walk and dwstairs first, and then separate upstairs from walk and dwstairs, respectively, then separate upstairs from walk and dwstairs, respectively.

Experiment Design
According to the conclusion above, an experiment for verification is designed. The information of the subject is shown in Table 6. To protect the privacy of amputees, the photos of amputees will not be shown here. The artificial knee used in the experiment is shown in Figure 15, the MEMS IMU used in the experiment is shown in Figure 16, and the main specifications of the devices are given in Table 7.

Results and Analysis
After the experimental data collection, the same processing method is used to extract the feature vector through FRFT after dividing the cycle, and the classifier designed above is used for classification, the results are shown in Figure 17 and the performance of the sub-classifier is detailed in Table 8. The comparison of the accuracy in fractional domain and time domain is in Table 9. It can be seen that the effect of fractional domain is better than the effect of time domain.

Conclusions
With the aim of designing an action detection method on artificial knee, a new time-frequency feature extraction method is proposed. This method is targeted at four common actions of the artificial knee wearer, and extracted features from the inertia data are measured by MEMS IMU, using fractional Fourier transform (FRFT) to magnify the diversity of features. FRFT is employed to extract the appropriate feature vectors and construct a feature vector composed of eight features.
By analyzing the results of these features after FRFT in different orders, it was discovered that it may have a good effect to classify walk and dwstairs by rms and mean, to classify upstairs and dwstairs by range and pksNum, and to classify walk, dwstairs and run by std and pksNum.
The results of the classification with different features and orders are also analyzed. It was discovered that in order to classify walk and dwstairs, the combination of pksNum and std is better than the combination of range and pksMean, and it is better when order = 0.67 than order = 0.20. To classify upstairs and dwstairs, the combination of range and pksNum is better than the combination of rms and IQR, and it is better when order = 0.64 than order = 0.20. To classify walk and upstairs, the combination of rms and mean is better than the combination of std and IQR, and it is better when order = 0.71 than order = 0.20. To separate run from the other actions, the combination of rms and mean is better than the combination of range and IQR, and it is better when order = 0.75.
Finally, the verification experiment is designed with the artificial lower limb knee prototype. The results show that the method that we proposed has a good classification performance while also reducing the requirements of hardware calculation. The accuracies of each sub-classifier are 95.05%, 95.38%, 91.43%, and 89.39%, respectively; the precisions are 78.43%, 98.36%, 98.36%, and 93.41%, respectively; and the recalls are 100%, 93.26%, 86.96%, and 86.68%, respectively.