# A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Background and Literature Review

## 3. Overview of the Proposed Method

## 4. Proposed Method

#### 4.1. Recognition of Atomic Activities

#### 4.2. Recognition of Composite Activities

#### 4.2.1. Handcrafted Features

- Maximum: Let X is the feature vector. The $Max\left(X\right)$ function finds and returns the largest feature value ${x}_{i}\in X$.
- Minimum: The $Min\left(X\right)$ function finds and returns the smallest feature value ${x}_{i}\in X$.
- Average: For N number of feature values, the average returns the center value of feature vector X. That is,$$Average\left(X\right)=\mu =\frac{{\sum}_{i=1}^{n}{x}_{i}}{N}.$$
- Standard Deviation: It describes the amount of disparity in feature vector $X=\{{x}_{1},{x}_{2}\dots {x}_{N}\}$ and can be computed using the following formulation:$$Stdev\left(X\right)=\sigma =\sqrt{\frac{1}{N}\sum _{i=1}^{N}{({x}_{i}-\mu )}^{2}}.$$
- Zero Crossing: It is used to estimate the difference between a rapid and slow movement of activity [61] and can be calculated by estimating how often the signal value crosses zero in either direction.
- Percentiles: Percentile defines a number where a certain percentage of scores fall below that number. That is, the pth percentile is a value such that, at most, $(100\times p)\%$ of the measurements fall below than this value, and $100\times (1-p)\%$ of the measurements fall above this value. For instance, the 25th percentile means that this value is bigger than 25 values and smaller than 75 feature values. The 25th percentile is also called the first quartile. The 50th percentile is generally the median, and the 75th percentile is also called the third quartile.
- Interquartile Range: The difference between the third and first quartiles is known as the interquartile range.
- Skewness: It calculates the asymmetry of the probability distribution of data about its mean and can be calculated as:$$Sk=\frac{1}{N{\sigma}^{3}}\sum _{i=1}^{n}{({x}_{i}-\mu )}^{3}.$$
- Kurtosis: It is the measure that how heavily the tails of distribution differ from the tails of a normal distribution. The higher value of kurtosis corresponds to the greater extremity of deviations which refer to outliers [62]. Mathematically, it can be computed as:$$Kr=\frac{1}{N{\sigma}^{4}}\sum _{i=1}^{n}{({x}_{i}-\mu )}^{4}.$$
- Auto-correlation: It measures the degree of similarity between a given time series data and a lagged version of itself over the successive time intervals. That is, it depicts the notch of similarity between a current feature value and its earlier values [63], and it can be computed as:$${r}_{k}=\frac{{\sum}_{i=1}^{N-k}({x}_{i}-\mu )({x}_{i+k}-\mu )}{{\sum}_{i=1}^{N}{({x}_{i}-\mu )}^{2}}.$$
- Order mean values: They are computed from the arranged set (increasing order) of values. That is, the first-order mean is the simple smallest sample value ${x}_{1}$ in an arranged feature set X, the second-order defines the second smallest value ${x}_{2}$, and so forth [64].
- Norm values: They are used to estimate the distance of a feature vector from its origin. We used two different measures: ${L}_{1}$-norm (also known as Manhattan distance) and ${L}_{2}$-norm (also known as Euclidean norm) [65].
- Spectral energy: We recall that several sensors are used in the recorded data to analyze human activities, and they can be considered as a function whose amplitude is changing over time. We used Fourier transformation to transform the time-based signal to its frequency spectrum, and spectral energy formulation is employed to calculate the signal energy distribution across the frequency. It measures the sum of squared amplitude of frequency content $\u03dd\left(n\right)$. That is,$${S}_{E}=\sum _{i=1}^{N}\u03dd{\left(n\right)}^{2}.$$It can also be computed using normalized frequency spectra. That is,$$\widehat{\u03dd}\left(n\right)=\frac{\u03dd\left(n\right)}{{\sum}_{i=1}^{N}\u03dd\left(n\right)}.$$After the normalization process, the Equation (8) can be described as,$$N{S}_{E}=\sum _{i=1}^{N}\widehat{\u03dd}{\left(n\right)}^{2}.$$
- Spectral entropy: It is based on the concept of the Shannon entropy and is used to measure the spectral of the signal distribution in terms of frequency. The mathematical formulation of spectral entropy can be described as:$${S}_{EN}=-\sum _{i=1}^{N}\widehat{\u03dd}\left(n\right)\times log\widehat{\u03dd}\left(n\right).$$

#### 4.2.2. Subspace Pooling

- Step 1: The aforementioned subspace pooling techniques are applied on original multidimensional data (i.e., the recognition score of the atomic activities).
- Step 2: Eigenvectors of $n\times n$ dimensions are extracted using SVD and PCA, where $n=18\times 61=1098$.
- Step 3: The absolute sum of every single column of eigenvectors with dimension $n\times 1$ is computed.
- Step 4: The sorting algorithm is applied on the sum of eigenvectors with respect to the index to assess the importance of every feature (obtained after subspace pooling technique). The feature with the highest sum is considered the most important feature.
- Step 5: The original data (i.e., atomic scores) is arranged with respect to the sorted sum.

#### 4.3. Classification

- Support Vector Machine (SVM) [73] is a simple and well-known supervised machine learning algorithm to solve the problem of classification. SVM first maps the training instances into high dimensional space and then extracts a decision boundary (e.g., hyperplane) between the instances of different classes based on the principle of maximizing the margin (i.e., the distance between the hyperplane and the nearest data point in each class is maximized). The training instances which are very close to the class boundary are known as support vectors. The training process aims to find such a hyperplane that should be in the middle of positive and negative instances, and the distance of hyperplane with the nearest positive and negative instances should be maximized [3,74]. We used simple LibLinear SVM [73], and the optimal value or hyperparameter/regularization parameter C is selected empirically within the range of (${2}^{-6}$– ${2}^{6}$).
- RF [75] is an ensemble learning algorithm that uses multiple individual learners and fuses the results. In particular, it comprises the collection of decision trees with a random subset of data, and the outputs of all the decision trees are then combined to create the final decision (i.e., recognition). The regularization parameter n is selected empirically in the range of 600–2000 with the increment size of 200.
- The Hidden Markov Model (HMM) [76] has also been employed as a classifier to recognize the human actions [77,78] using a few unobservable (i.e., hidden) states, and sequences whose behavior depends on the hidden states [79]. The model is usually constructed using a set of states with a stochastic matrix storing the transition information between each state. The elements of such a matrix hold probabilities of states over time known as transition probabilities. Every state in HMM is associated with a probability density function (PDF) which helps in determining that every state emits a sequence over time known as observation sequence. The main objective of HMM is to learn the behavior of hidden states based on observable sequences [80].
- The idea of ensemble classifier has also attracted an increasing amount of attention from the research community due to its effectiveness in the field of pattern recognition [81]. Ensemble classifier aims that the recognition technique not to depend on a decision of a single classification model, rather the final decision is based on the combination of several individual classifier [82]. It can be built by training several individual classifiers (also known as base learners) and combining their results/estimations by voting (either weighted or unweighted). That is, the estimations of all classification algorithms are combined so that votes with the highest number (i.e., maximum occurrences) are considered as the final ensemble prediction [83]. It is quite a known fact that ensemble classifiers give more accurate results than the other individual classifiers (base learners) from which they have been built.

## 5. Experiments and Results

#### 5.1. Experimental Setting

- k-fold cross-validation (CV): Training and testing data split on basis of k folds. We set the value of k = 3.
- Hold-out cross-validation: We used the data of 3 subjects for training and the data of the remaining 3 subjects for testing purposes, iteratively.
- Leave-one-out cross-validation: The data of 5 subjects are used for training, and the data of the remaining 1 subject is used for testing purposes, iteratively.

#### 5.1.1. Using Handcrafted Features

#### 5.1.2. Using Subspace Pooling Technique-Based Features

#### 5.1.3. Using Optimal Feature Selection

- First, we applied SVD and PCA separately and the matrix of eigenvectors (1098-dimensional) is extracted.
- Second, the sum of absolute values of every column of eigenvectors matrix is calculated, arranged in descending order with respect to its index number, and stored in a separate matrix.
- Third, original combined data was arranged with respect to the sorted sum of absolute eigenvectors.
- Fourth, the dimensions were reduced by iteratively selecting the small sets of features keeping in view the variance of eigenvectors.
- Finally, the selected set of features were evaluated using SVM and RF.

#### 5.1.4. Using HMM

#### 5.1.5. Using Ensemble Classifier

#### 5.2. Discussion

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- De-La-Hoz-Franco, E.; Ariza-Colpas, P.; Quero, J.M.; Espinilla, M. Sensor-based datasets for human activity recognition–a systematic review of literature. IEEE Access
**2018**, 6, 59192–59210. [Google Scholar] [CrossRef] - Urwyler, P.; Rampa, L.; Stucki, R.; Büchler, M.; Müri, R.; Mosimann, U.P.; Nef, T. Recognition of activities of daily living in healthy subjects using two ad-hoc classifiers. Biomed. Eng. Online
**2015**, 14, 54. [Google Scholar] [CrossRef] [PubMed][Green Version] - Khan, M.H. Human Activity Analysis in Visual Surveillance and Healthcare; Logos Verlag Berlin GmbH: Berlin, Germany, 2018; Volume 45. [Google Scholar]
- Li, F.; Shirahama, K.; Nisar, M.A.; Köping, L.; Grzegorzek, M. Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors
**2018**, 18, 679. [Google Scholar] [CrossRef][Green Version] - Ke, S.R.; Thuc, H.L.U.; Lee, Y.J.; Hwang, J.N.; Yoo, J.H.; Choi, K.H. A review on video-based human activity recognition. Computers
**2013**, 2, 88–131. [Google Scholar] [CrossRef] - Lara, O.D.; Labrador, M.A. A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor.
**2012**, 15, 1192–1209. [Google Scholar] [CrossRef] - Hammerla, N.Y.; Halloran, S.; Plötz, T. Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv
**2016**, arXiv:1604.08880. [Google Scholar] - Radu, V.; Lane, N.D.; Bhattacharya, S.; Mascolo, C.; Marina, M.K.; Kawsar, F. Towards multimodal deep learning for activity recognition on mobile devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, Heidelberg, Germany, 12–16 September 2016; pp. 185–188. [Google Scholar]
- Koutroumbas, K.; Theodoridis, S. Pattern Recognition; Academic Press: Cambridge, MA, USA, 2008. [Google Scholar]
- Peng, X.; Wang, L.; Wang, X.; Qiao, Y. Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice. Comput. Vis. Image Underst.
**2016**, 150, 109–125. [Google Scholar] [CrossRef][Green Version] - Cai, J.; Luo, J.; Wang, S.; Yang, S. Feature selection in machine learning: A new perspective. Neurocomputing
**2018**, 300, 70–79. [Google Scholar] [CrossRef] - Wang, H.; Kläser, A.; Schmid, C.; Liu, C.L. Dense trajectories and motion boundary descriptors for action recognition. Int. J. Comput. Vis.
**2013**, 103, 60–79. [Google Scholar] [CrossRef][Green Version] - Durgesh, K.S.; Lekha, B. Data classification using support vector machine. J. Theor. Appl. Inf. Technol.
**2010**, 12, 1–7. [Google Scholar] - Nurhanim, K.; Elamvazuthi, I.; Izhar, L.; Ganesan, T. Classification of human activity based on smartphone inertial sensor using support vector machine. In Proceedings of the 2017 IEEE 3rd International Symposium in Robotics and Manufacturing Automation (ROMA), Kuala Lumpur, Malaysia, 19–21 September 2017; pp. 1–5. [Google Scholar]
- Nisar, M.A.; Shirahama, K.; Li, F.; Huang, X.; Grzegorzek, M. Rank Pooling Approach for Wearable Sensor-Based ADLs Recognition. Sensors
**2020**, 20, 3463. [Google Scholar] [CrossRef] - Aggarwal, J.K.; Ryoo, M.S. Human activity analysis: A review. ACM Comput. Surv. (CSUR)
**2011**, 43, 1–43. [Google Scholar] [CrossRef] - Bulling, A.; Blanke, U.; Schiele, B. A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. (CSUR)
**2014**, 46, 1–33. [Google Scholar] [CrossRef] - Logan, B.; Healey, J.; Philipose, M.; Tapia, E.M.; Intille, S. A long-term evaluation of sensing modalities for activity recognition. In Proceedings of the International conference on Ubiquitous Computing 2007, Innsbruck, Austria, 16–19 September 2007; pp. 483–500. [Google Scholar]
- Rohrbach, M.; Rohrbach, A.; Regneri, M.; Amin, S.; Andriluka, M.; Pinkal, M.; Schiele, B. Recognizing fine-grained and composite activities using hand-centric features and script data. Int. J. Comput. Vis.
**2016**, 119, 346–373. [Google Scholar] [CrossRef][Green Version] - Blanke, U.; Schiele, B. Remember and transfer what you have learned-recognizing composite activities based on activity spotting. In Proceedings of the International Symposium on Wearable Computers (ISWC) 2010, Seoul, Korea, 10–13 October 2010; pp. 1–8. [Google Scholar]
- Rai, A.; Yan, Z.; Chakraborty, D.; Wijaya, T.K.; Aberer, K. Mining complex activities in the wild via a single smartphone accelerometer. In Proceedings of the Sixth International Workshop on Knowledge Discovery From Sensor Data, Beijing, China, 12 August 2012; pp. 43–51. [Google Scholar]
- Bharti, P.; De, D.; Chellappan, S.; Das, S.K. HuMAn: Complex activity recognition with multi-modal multi-positional body sensing. IEEE Trans. Mob. Comput.
**2018**, 18, 857–870. [Google Scholar] [CrossRef] - Nguyen, L.T.; Zeng, M.; Tague, P.; Zhang, J. Recognizing new activities with limited training data. In Proceedings of the 2015 ACM International Symposium on Wearable Computers, Osaka, Japan, 9–11 September 2015; pp. 67–74. [Google Scholar]
- Nair, H.; Tan, C.; Zeng, M.; Mengshoel, O.J.; Shen, J.P. AttriNet: Learning mid-level features for human activity recognition with deep belief networks. In Proceedings of the Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, London, UK, 11–13 September 2019; pp. 510–517. [Google Scholar]
- Vrigkas, M.; Nikou, C.; Kakadiaris, I.A. A review of human activity recognition methods. Front. Robot. AI
**2015**, 2, 28. [Google Scholar] [CrossRef][Green Version] - Attal, F.; Mohammed, S.; Dedabrishvili, M.; Chamroukhi, F.; Oukhellou, L.; Amirat, Y. Physical human activity recognition using wearable sensors. Sensors
**2015**, 15, 31314–31338. [Google Scholar] [CrossRef] [PubMed][Green Version] - Jiang, W.; Yin, Z. Human activity recognition using wearable sensors by deep convolutional neural networks. In Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia, 26–30 October 2015; pp. 1307–1310. [Google Scholar]
- Zhang, M.; Sawchuk, A.A. USC-HAD: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA, 5–8 September 2012; pp. 1036–1043. [Google Scholar]
- Lawal, I.A.; Bano, S. Deep human activity recognition using wearable sensors. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, Rhodes, Greece, 1–9 June 2019; pp. 45–48. [Google Scholar]
- Shirahama, K.; Grzegorzek, M. On the generality of codebook approach for sensor-based human activity recognition. Electronics
**2017**, 6, 44. [Google Scholar] [CrossRef][Green Version] - Javed, A.R.; Sarwar, M.U.; Khan, S.; Iwendi, C.; Mittal, M.; Kumar, N. Analyzing the effectiveness and contribution of each axis of tri-axial accelerometer sensor for accurate activity recognition. Sensors
**2020**, 20, 2216. [Google Scholar] [CrossRef][Green Version] - Khan, M.H.; Farid, M.S.; Grzegorzek, M. A generic codebook based approach for gait recognition. Multimed. Tools Appl.
**2019**, 78, 35689–35712. [Google Scholar] [CrossRef] - Rangarajan, L.; Veerabhadrappa. Bi-level dimensionality reduction methods using feature selection and feature extraction. Int. J. Comput. Appl.
**2010**, 4, 33–38. [Google Scholar] - Zebari, R.; Abdulazeez, A.; Zeebaree, D.; Zebari, D.; Saeed, J. A Comprehensive Review of Dimensionality Reduction Techniques for Feature Selection and Feature Extraction. J. Appl. Sci. Technol. Trends
**2020**, 1, 56–70. [Google Scholar] [CrossRef] - Li, M.; Wang, H.; Yang, L.; Liang, Y.; Shang, Z.; Wan, H. Fast hybrid dimensionality reduction method for classification based on feature selection and grouped feature extraction. Expert Syst. Appl.
**2020**, 150, 113277. [Google Scholar] [CrossRef] - Shi, Q.; Luo, H.; Han, J. Subspace Pooling Based Temporal Features Extraction For Audio Event Recognition. In Proceedings of the Interspeech 2019, Graz, Austria, 15–19 September 2019; pp. 3850–3854. [Google Scholar]
- Zhang, S.; Zhang, Q.; Wei, X.; Wang, P.; Jiao, B.; Zhang, Y. Person Re-identification in Aerial Imagery. arXiv
**2019**, arXiv:1908.05024. [Google Scholar] [CrossRef][Green Version] - Wei, X.; Zhang, Y.; Gong, Y.; Zheng, N. Kernelized subspace pooling for deep local descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 18–23 June 2018; pp. 1867–1875. [Google Scholar]
- Guyon, I.; Elisseeff, A. An introduction to variable and feature selection. J. Mach. Learn. Res.
**2003**, 3, 1157–1182. [Google Scholar] - Salem, N.; Hussein, S. Data dimensional reduction and principal components analysis. Procedia Comput. Sci.
**2019**, 163, 292–299. [Google Scholar] [CrossRef] - Sadou, B.; Lahoulou, A.; Bouden, T.; Avila, A.R.; Falk, T.H.; Akhtar, Z. Blind Image Quality Assessment Using Singular Value Decomposition Based Dominant Eigenvectors for Feature Selection. In Proceedings of the 5th International Conference on Signal and Image Processing (SIPRO’19), Toronto, ON, Canada, 13–14 July 2019; pp. 233–242. [Google Scholar]
- D’Addabbo, A.; Papale, M.; Di Paolo, S.; Magaldi, S.; Colella, R.; d’Onofrio, V.; Di Palma, A.; Ranieri, E.; Gesualdo, L.; Ancona, N. SVD based feature selection and sample classification of proteomic data. In Proceedings of the International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, Zagreb, Croatia, 3–5 September 2008; pp. 556–563. [Google Scholar]
- Song, F.; Guo, Z.; Mei, D. Feature selection using principal component analysis. In Proceedings of the 2010 International Conference on System Science, Engineering Design and Manufacturing Informatization, Yichang, China, 12–14 November 2010; Volume 1, pp. 27–30. [Google Scholar]
- Malhi, A.; Gao, R.X. PCA-based feature selection scheme for machine defect classification. IEEE Trans. Instrum. Meas.
**2004**, 53, 1517–1525. [Google Scholar] [CrossRef] - Yuce, B.; Mastrocinque, E.; Packianather, M.S.; Pham, D.; Lambiase, A.; Fruggiero, F. Neural network design and feature selection using principal component analysis and Taguchi method for identifying wood veneer defects. Prod. Manuf. Res.
**2014**, 2, 291–308. [Google Scholar] [CrossRef] - Gulzar, Z.; Leema, A.A.; Malaserene, I. Human Activity Analysis using Machine Learning Classification Techniques. Int. J. Innov. Technol. Explor. Eng. (IJITEE)
**2019**. [Google Scholar] [CrossRef] - Khan, M.H.; Schneider, M.; Farid, M.S.; Grzegorzek, M. Detection of infantile movement disorders in video data using deformable part-based model. Sensors
**2018**, 18, 3202. [Google Scholar] [CrossRef][Green Version] - Yazdansepas, D.; Niazi, A.H.; Gay, J.L.; Maier, F.W.; Ramaswamy, L.; Rasheed, K.; Buman, M.P. A multi-featured approach for wearable sensor-based human activity recognition. In Proceedings of the 2016 IEEE International Conference on Healthcare Informatics (ICHI), Chicago, IL, USA, 4–7 October 2016; pp. 423–431. [Google Scholar]
- Sánchez, D.; Tentori, M.; Favela, J. Activity recognition for the smart hospital. IEEE Intell. Syst.
**2008**, 23, 50–57. [Google Scholar] [CrossRef][Green Version] - Piyathilaka, L.; Kodagoda, S. Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features. In Proceedings of the 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), Melbourne, Australia, 19–21 June 2013; pp. 567–572. [Google Scholar]
- Cuntoor, N.P.; Yegnanarayana, B.; Chellappa, R. Interpretation of state sequences in HMM for activity representation. In Proceedings of the (ICASSP’05)—IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, PA, USA, 18–23 March 2005; Volume 2, p. ii-709. [Google Scholar]
- Pietrzykowski, M.; Sałabun, W. Applications of Hidden Markov Model: State-of-the-art. Int. J. Comput. Technol. Appl.
**2014**, 5, 1384–1391. [Google Scholar] - Khan, M.H.; Farid, M.S.; Grzegorzek, M. A non-linear view transformations model for cross-view gait recognition. Neurocomputing
**2020**, 402, 100–111. [Google Scholar] [CrossRef] - Opitz, D.; Maclin, R. Popular ensemble methods: An empirical study. J. Artif. Intell. Res.
**1999**, 11, 169–198. [Google Scholar] [CrossRef] - Mishra, S.K. A review of ensemble technique for improving majority voting for classifier. Int. J.
**2013**, 3, 177–180. [Google Scholar] - Shen, H.B.; Chou, K.C. Ensemble classifier for protein fold pattern recognition. Bioinformatics
**2006**, 22, 1717–1722. [Google Scholar] [CrossRef] - Khan, M.H.; Farid, M.S.; Grzegorzek, M. Using a generic model for codebook-based gait recognition algorithms. In Proceedings of the 2018 International Workshop on Biometrics and Forensics (IWBF), Sassari, Italy, 7–8 June 2018; pp. 1–7. [Google Scholar]
- Wang, L. Support Vector Machines: Theory and Applications; Springer Science & Business Media: New York, NY, USA, 2005; Volume 177. [Google Scholar]
- Khan, M.H.; Farid, M.S.; Grzegorzek, M. Spatiotemporal features of human motion for gait recognition. Signal Image Video Process.
**2019**, 13, 369–377. [Google Scholar] [CrossRef] - Rank Pooling Approach for Wearable Sensor-Based ADLs Recognition. Available online: https://www.info.kindai.ac.jp/~shirahama/rank_pooling (accessed on 1 January 2021).
- Cook, D.J.; Krishnan, N.C. Activity Learning: Discovering, Recognizing, and Predicting Human Behavior From Sensor Data; John Wiley & Sons: Hoboken, NJ, USA, 2015. [Google Scholar]
- Esmael, B.; Arnaout, A.; Fruhwirth, R.K.; Thonhauser, G. A statistical feature-based approach for operations recognition in drilling time series. Int. J. Comput. Inf. Syst. Ind. Manag. Appl.
**2015**, 5, 454–461. [Google Scholar] - Box, G.E.; Jenkins, G.M. Time Series Analysis: Forecasting and Control San Francisco; Wiley: Hoboken, NJ, USA, 1976. [Google Scholar]
- Order Statistics: Simple Definition, Examples—Statistics How to. Available online: https://www.statisticshowto.com/order-statistics/ (accessed on 15 January 2021).
- How Statistical Norms Improve Modeling|by Madeline Schiappa|Towards Data Science. Available online: https://towardsdatascience.com/norms-penalties-and-multitask-learning-2f1db5f97c1f (accessed on 15 January 2021).
- Zhai, H.; Zhang, H.; Xu, X.; Zhang, L.; Li, P. Kernel sparse subspace clustering with a spatial max pooling operation for hyperspectral remote sensing data interpretation. Remote Sens.
**2017**, 9, 335. [Google Scholar] [CrossRef][Green Version] - Fernando, B.; Habrard, A.; Sebban, M.; Tuytelaars, T. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 2960–2967. [Google Scholar]
- Klema, V.; Laub, A. The singular value decomposition: Its computation and some applications. IEEE Trans. Autom. Control
**1980**, 25, 164–176. [Google Scholar] [CrossRef][Green Version] - Van Loan, C.F.; Golub, G.H. Matrix Computations; Johns Hopkins University Press: Baltimore, MD, USA, 1983. [Google Scholar]
- Hotelling, H. Analysis of a complex of statistical variables into principal components. J. Educ. Psychol.
**1933**, 24, 417. [Google Scholar] [CrossRef] - Yam, Y.; Baranyi, P.; Yang, C.T. Reduction of fuzzy rule base via singular value decomposition. IEEE Trans. Fuzzy Syst.
**1999**, 7, 120–132. [Google Scholar] - Bolón-Canedo, V.; Sánchez-Maroño, N.; Alonso-Betanzos, A. A review of feature selection methods on synthetic data. Knowl. Inf. Syst.
**2013**, 34, 483–519. [Google Scholar] [CrossRef] - Fan, R.E.; Chang, K.W.; Hsieh, C.J.; Wang, X.R.; Lin, C.J. LIBLINEAR: A library for large linear classification. J. Mach. Learn. Res.
**2008**, 9, 1871–1874. [Google Scholar] - Khan, M.H.; Helsper, J.; Boukhers, Z.; Grzegorzek, M. Automatic recognition of movement patterns in the vojta-therapy using RGB-D data. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 1235–1239. [Google Scholar]
- Breiman, L. Bagging predictors. Mach. Learn.
**1996**, 24, 123–140. [Google Scholar] [CrossRef][Green Version] - Rabiner, L.R. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE
**1989**, 77, 257–286. [Google Scholar] [CrossRef] - Kellokumpu, V.; Pietikäinen, M.; Heikkilä, J. Human Activity Recognition Using Sequences of Postures. In Proceedings of the IAPR Conference on Machine Vision Application 2005, Tsukuba Science City, Japan, 16–18 May 2005; pp. 570–573. [Google Scholar]
- Kolekar, M.H.; Dash, D.P. Hidden markov model based human activity recognition using shape and optical flow based features. In Proceedings of the 2016 IEEE Region 10 Conference (TENCON), Singapore, 22–25 November 2016; pp. 393–397. [Google Scholar]
- Stikic, M.; Huynh, T.; Van Laerhoven, K.; Schiele, B. ADL recognition based on the combination of RFID and accelerometer sensing. In Proceedings of the 2008 Second International Conference on Pervasive Computing Technologies for Healthcare, Tampere, Finland, 30 January–1 February 2008; pp. 258–263. [Google Scholar]
- Melnikoff, S.J.; Quigley, S.F.; Russell, M.J. Implementing a hidden Markov model speech recognition system in programmable logic. In Proceedings of the International Conference on Field Programmable Logic and Applications, Belfast, UK, 27–29 August 2001; pp. 81–90. [Google Scholar]
- Rodriguez, J.J.; Kuncheva, L.I.; Alonso, C.J. Rotation forest: A new classifier ensemble method. IEEE Trans. Pattern Anal. Mach. Intell.
**2006**, 28, 1619–1630. [Google Scholar] [CrossRef] - Kittler, J.; Hatef, M.; Duin, R.P.; Matas, J. On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell.
**1998**, 20, 226–239. [Google Scholar] [CrossRef][Green Version] - Onan, A.; Korukoğlu, S.; Bulut, H. A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification. Expert Syst. Appl.
**2016**, 62, 1–16. [Google Scholar] [CrossRef] - LG G5 Smart Phones. Available online: https://www.lg.com/us/g5-phones/ (accessed on 29 March 2021).
- HUAWEI: SmartWatches. Available online: https://consumer.huawei.com/en/wearables/ (accessed on 29 March 2021).
- JINS MEME: Eyewear that Sees Your EVERYDAY. Available online: https://jins-meme.com/en/ (accessed on 29 March 2021).

**Figure 1.**Hierarchical model to recognize human activities. In the first step, the atomic activities are detected from the original sensory data, and their recognition scores are obtained, which are used to recognize the composite activities in the second step.

**Figure 3.**The depiction of codebook-based feature extraction process: (

**a**) Codebook construction by grouping the similar subsequences using k-means clustering algorithm. The center of each group is set as “codeword”. (

**b**) Features are computed on each of the subsequences by assigning them to the most similar codeword [4].

**Figure 5.**Activity recording setup using three wearable devices. The movement data of smart glasses and watch is initially sent to smartphone via Bluetooth connection and later all the recorded data sent to a home-gateway using Wi-Fi connection [15].

**Figure 6.**The composite activity recognition using handcrafted features with Support Vector Machine (SVM) and Random Forest (RF). (

**a**) K-fold cross-validation, (

**b**) hold-out cross-validation, and (

**c**) leave-one-out cross-validation. In k-fold cross-validation, the training and testing data is split based on k folds. We set the value of k = 3. In hold-out cross-validation, the data of 3 subjects are used for training, and the data of the remaining 3 subjects is used for testing purposes, iteratively. In leave-one-out cross-validation, the data of 5 subjects are used for training, and the data of the remaining 1 subject is used for testing purposes, iteratively.

**Figure 7.**The composite activity recognition using features derived from the subspace pooling technique. The features are classified using Support Vector Machine (SVM) and Random Forest (RF). (

**a**) K-fold cross-validation; (

**b**) hold-out cross-validation.

**Figure 8.**The composite activity recognition using features derived from the subspace pooling technique. The features are classified using Support Vector Machine (SVM) and Random Forest (RF). (

**a**) Leave-one-out cross-validation: subspace pooling using singular value decomposition (SVD), and (

**b**) leave-one-out cross-validation: subspace pooling using principal component analysis (PCA).

Maximum | Skewness |

Minimum | Kurtosis |

Average | Auto-correlation |

Standard-deviation | First-Order Mean (FOM) |

Zero crossing | Norm of FOM |

Percentile 20 | Second-order mean (SOM) |

Percentile 50 | Norm of SOM |

Percentile 80 | Spectral energy |

Interquartile | Spectral entropy |

**Table 2.**Details of composite activities in the CogAge dataset [15]. The left hand represents the number of instances in which the activities are mainly performed using the left hand only, whereas, in the both hands setting, the activities are performed using both hands.

Total subjects: | 6 |

Total activities: | 7 |

Activities: | Brushing teeth |

Cleaning room | |

Handling medications | |

Preparing food | |

Styling hair | |

Using telephone | |

Washing hands | |

Feature representation: | 61-dimensional |

Total number of instances: | 752 |

Left hand instances: | 471 |

Right hand instances: | 281 |

**Table 3.**The composite activity recognition using a set of optimal features derived from the subspace pooling technique. The feature are classified with k-fold (where $k=3$) cross-validation. The average recognition accuracy with k-fold cross-validation is also presented.

SVD | PCA | |||
---|---|---|---|---|

All Hands | Left Hand | All Hands | Left Hand | |

Accuracy (SVM) | 47.04% | 53.00% | 47.04% | 53.00% |

Accuracy (RF) | 46.00% | 53.00% | 46.00% | 52.00% |

**Table 4.**The composite activity recognition using a set of optimal features derived from the subspace pooling technique. The features are classified with hold-out cross-validation. The average recognition accuracy with hold-out cross-validation is also presented.

SVD | PCA | |||
---|---|---|---|---|

All Hands | Left Hand | All Hands | Left Hand | |

Accuracy (SVM) | 51.40% | 54.01% | 51.40% | 54.67% |

Accuracy (RF) | 53.12% | 55.99% | 53.37% | 55.99% |

**Table 5.**The composite activity recognition using a set of optimal features derived from the subspace pooling technique. The features are classified with leave-one-out cross-validation. The average recognition accuracy with leave-one-out cross-validation is also presented.

SVD | PCA | |||
---|---|---|---|---|

All Hands | Left Hand | All Hands | Left Hand | |

Accuracy (SVM) | 64.93% | 60.61% | 58.31% | 60.33% |

Accuracy (RF) | 61.53% | 62.50% | 62.05% | 63.00% |

**Table 6.**The comparative analysis of all the feature extraction techniques along with the classification algorithms. In all the experiments of leave-one-out cross-validation, the average recognition accuracies are reported here.

Handcrafted feature extraction | |
---|---|

Leave-one-out CV + SVM | 48.9% |

Leave-one-out CV + RF | 79% |

Subspace pooling | |

PCA + Leave-one-out CV + RF | 62.8% |

Optimal feature selection | |

SVD+ Leave-one-out CV + SVM | 62.8% |

HMM | |

Leave-one-out CV | 64% |

Method | Baseline | Proposed |
---|---|---|

Hidden Markov Model + Holdout CV | 51.20% [15] | 51.2% |

Hidden Markov Model + Leave-one-out CV | 61.01% [15] | 64% |

**Table 8.**The comparison of the proposed handcrafted features with different state-of-the-art techniques.

Proposed Handcrafted Features | |||||
---|---|---|---|---|---|

SVM | RF | ||||

k-Fold | Hold-Out | Leave-One-Out | k-Fold | Hold-Out | Leave-One-Out |

42% | 35% | 48.9% | 66% | 72% | 79% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Amjad, F.; Khan, M.H.; Nisar, M.A.; Farid, M.S.; Grzegorzek, M.
A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data. *Sensors* **2021**, *21*, 2368.
https://doi.org/10.3390/s21072368

**AMA Style**

Amjad F, Khan MH, Nisar MA, Farid MS, Grzegorzek M.
A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data. *Sensors*. 2021; 21(7):2368.
https://doi.org/10.3390/s21072368

**Chicago/Turabian Style**

Amjad, Fatima, Muhammad Hassan Khan, Muhammad Adeel Nisar, Muhammad Shahid Farid, and Marcin Grzegorzek.
2021. "A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data" *Sensors* 21, no. 7: 2368.
https://doi.org/10.3390/s21072368