Classification of Human Motion Data Based on Inertial Measurement Units in Sports: A Scoping Review

Abstract

Inertial measurement units (IMU) are widely used in sports applications to digitise human motion by measuring acceleration and rotational velocity in three-dimensional space. A common machine learning problem is the classification of human motion primitives from IMU data. In order to investigate the classification methods used in the existing literature and to analyse whether and how the time-dependent data structure is considered in the classification process of motion data analysis in sports, a scoping review was conducted. Based on a keyword search, articles from 2010 to 2021 were extracted, and 93 articles were relevant for data extraction. Over- and undersampling of data and data augmentation techniques were rarely used. The classification methods applied can be divided into three main branches: classic machine learning and deep learning models, threshold-based approaches, and dynamic time warping. The most often applied algorithms were support vector machines (SVM), followed by neural networks and k-nearest neighbours. In comparative works, when more than one classifier was applied, random forests, neural networks, boosting models and SVM were found to be the methods that achieved the highest accuracy. If the time-dependent data structure was taken into account, it was incorporated either within the models, for example, by using long-short-term memory models or within the feature calculation step by using rolling windows with an overlap, which was the most common method of considering the time dependency of the IMU data.

1. Introduction

Human motion analysis is applied in various application areas such as athletic performance, medical diagnostics, surveillance, human–machine user interface, or video-conferencing, etc. [1]. Depending on the activity of interest, different sensor set-ups are used to capture and digitise human motion. Ref. [2] differentiate the systems into direct and indirect ones. Indirect systems are vision-based and are represented, for example, by video or optoelectronic techniques. A popular example of a direct system widely used in sports are inertial measurement units (IMU). They typically measure acceleration and angular velocity in three dimensions. The result of a measurement are multivariate time series.

In general, the analysis of human motion data can be split into five steps, which are described as activity recognition chain (ARC) [3]. Figure 1 visualises a typical ARC. As shown, the first step is the collection of the raw IMU sensor data. The second step consists of the pre-processing, such as filtering or removal of outliers. Step three is the segmentation phase, in which the raw time series is divided into segments, so-called motion or action primitives. These simple, atomic movements can be combined sequentially to form a complex behavior [4]. In the action hierarchy of [5], action primitives are the lowest unit in the hierarchy from which actions are formed. Step four of the ARC is the calculation of features for each segment, which could be time or frequency domain features, and step five is the classification phase. As for the classification step of the ARC, there are different methods that can be roughly divided into traditional and deep learning models according to [6]. There are already various reviews about the application of machine and deep learning techniques for human motion analysis. Ref. [7] have already analysed the application of machine learning algorithms in the field of sport-specific movement recognition in a systematic review for the period of January 2000 to May 2018. The authors analysed in-field automatic sport-specific movement detection and recognition with IMU and vision-based data. Ref. [8] published a survey on activity detection and classification based on wearable sensors, where they focused on the sensor specifications, such as sensor type, number of sensors used, sensor location, type of activities detected, processing type etc. Ref. [6] concentrated in their survey on the analysis of accelerometer data for human activity recognition. In addition, there exist various surveys for the analysis of deep learning and machine learning methods for general activity detection based on sensor data (see e.g., ref. [9] or [10]).

Figure 1. Activity recognition chain (adapted from [3]).

In general, time series classification is a challenging task [11], and several classification algorithms have been developed for both univariate and multivariate data [11,12,13]. The methods used for time series classification consider the temporal structure differently. E.g., long short-term memory models (LSTM), which are a type of recurrent neural network, learn the order dependencies in sequential data [14]. In contrast, another common deep neural network architecture for time series classification, the multilayer perceptron (MLP), processes time series elements independently from each other [11].

The aim of this work is to identify the classification methods used for human motion analysis in sports in the form of a scoping review and to analyse whether and how the time series structure of the data is considered. Compared to [7], we focus exclusively on classification tasks and IMU data, do not only include machine or deep learning algorithms, and also include in-lab studies. We mainly discuss the fifth step of the ARC and investigate the methodological and statistical specifications of human motion data classification in sports across all disciplines. The accuracy of classification models depends on many different aspects, such as the classification task, the number of sensors, or the validation procedure, and it is difficult to compare the different disciplines. Although model performance is somewhat related to model quality, we would like to emphasize that it is not the goal of this work to analyse the performance of the models in detail. The focus of this work is to give an overview of which methods have already been used for the classification of human motion primitives in sports and whether and how the time dependencies of the IMU data have been taken into account in the classification processes.

To explore the existing methods used for classification of human motion in sports, a scoping review was conducted, as this review type aims to map key concepts and identify gaps in research [15]. In comparison to systematic reviews that focus on clearly defined research questions and also analyse the quality of the articles, scoping reviews generally answer broader research questions and are thus used to identify further in-depth analyses or reviews [16].

The analysis is mainly based on the description for scoping reviews of [17] which is based on the PRISMA extension for scoping reviews [18] and followed a review protocol for scoping reviews (PRISMA-ScR).

1.1. Data Bases, Key Words and Inclusion Criteria

Three databases that include journals as well as conference proceedings were searched: Scopus and Web of Science to cover interdisciplinary research, as well as IEEE Xplore that covers articles and conference proceedings from, among others, bioengineering, computing and processing or signal processing and analysis. The first database search was performed in July 2020, where journals and conference proceedings were extracted. To update the analysis, another search was performed in March 2022 to further include journal articles from August 2020 to December 2021.

We adapted the keyword string of [7] by focusing on general (individual) motion classification in sports, but did not concentrate on outcome predictions of matches and games.

The following keyword string was used for Web of Science and IEEE Xplore:

((sensor OR accelerometer OR gyroscope OR inertial measurement OR imu) AND (sport* OR prevention) AND (prediction OR classification OR recognition) AND (movement OR motion OR kinematic*) NOT review)

and the following keyword string was used for Scopus:

((sensor OR accelerometer OR gyroscope OR inertial measurement OR imu) AND (sport* OR prevention) AND (prediction OR classification OR recognition) AND (movement OR motion OR kinematic*) AND NOT review).

Articles in which the classification process was not clearly described and in which no active human movements were classified were not considered. In addition, articles that used image or video data were excluded. Further, articles that focused on the general classification of activities into major activities such as lying, sitting, or walking were excluded, as were articles that were not written in English. An overview of the inclusion and exclusion criteria is provided in Table 1. Because some of the papers met more than one exclusion criterion, the sum is greater than the total number of excluded papers.

Table 1. Inclusion and exclusion criteria.

Inclusion	Exclusion
2010–2021	outside 2010–2021
English	full text not in English
classification method is clearly described	no clear description, no classification
active human movement	not human movement
IMU sensor data	image/video data
specific sports activity/motion primitive	general activity classification, fall detection etc.

The scoping review mainly analysed the classification methods applied in the existing literature and examined the following research questions:

• Data: Which data sources were used? How large was the sample size? Were resampling techniques/data augmentation techniques applied?
• Preparation: Was dimension reduction applied? How was the data set split into training and validation data?
• Methods: Which methods were used? Do these methods take into account dependencies between the segments/single movements?
• Validation: Which metrics were used for performance evaluation?

1.2. Study Selection

Keyword searches of the three databases retrieved 779 articles covering the period from 2010 to July 2020 and an additional 266 articles with the search in 2022 to include journal articles from August 2020 to December 2021. After screening the titles and abstracts, 190 articles remained for detailed full-text analysis. Of these 190 articles, 93 were relevant for inclusion. For quality assurance, each full text was screened twice and data extraction were performed by two authors (once by C.K. and once by W.K., S.K., V.V., H.R., S.M., or S.B.) with an online survey tool (LimeSurvey; LimeSurvey GmbH, Hamburg, Germany) to extract the information systematically. Forty-three articles were excluded because the classification task was not included at all or was not one of the focuses of the article, 28 articles were not related to sports or addressed a different application area, 20 articles did not use IMU data, ten did not clearly describe the analysis, were posters, or were not found, and five were conference papers that were preliminary work for an included journal article that included the same methods or an extension of an existing work without any new classification method. Finally, two papers were excluded because the full text was not in English. Figure 2 provides an overview of the inclusion process.

Figure 2. Flow diagram of the review.

Table 2 lists all articles, including year, discipline, the dependent variable, the classes that are classified, and methods.

Table 2. Overview of the articles included (incl. sports discipline, dependent variable, classes and method).

Authors	Year	Discipline	Dependent Variable	Classes	Methods
		Classification of a motion primitive
Acikmese et al. [19]	2017	basketball	basketball exercise type	forward–backward dribbling, left–right dribbling, regular dribbling, two hands dribbling, shooting and layup	SVM
Ahmadi et al. [20]	2015	activities during sports training session	type of activity	agility cuts, walking, sprinting, jogging, box jumps, and football free kicks	KNN, radial basis function, NB, RF
Alobaid et al. [21]	2018	soccer	soccer activities	shooting the ball, passing the ball, heading the ball, running and dribbling	NB, KNN, RF, SVM, DT, multinomial logistic regression, logistic model tree, multilayer perceptron
Amerineni et al. [22]	2021	taekwondo and boxing	kicking and boxing strikes	18 boxing punches and 24 taekwondo kicks	DTW, CNN, fusion models
Anand et al. [23]	2017	swing sports (tennis, golf etc.)	types of shots	tennis: forehand topspin, forehand slice, backhand topspin, backhand slice, serve; badminton: serve, clear, drop, smash; squash: forehand, backhand, serve	CNN, bi-directional LSTM
Anik et al. [24]	2016	badminton	badminton move	smash, serve, backhand, forehand, return	root mean square values, KNN, SVM
Aung et al. [25]	2013	walking	foot strike	heel strike, toe off, no-event	GMM
Bauza et al. [26]	2012	tennis	tennis swing patterns	forehand, backhand, volley, smash, service	DTW
Büthe et al. [27]	2016	tennis	tennis shot strokes + tennis steps	shot: forehand topspin, forehand slice, backhand topspin, backhand slice, smash; steps: shot step, side step	SVM, longest common subsequence
Charvátová et al. [28]	2021	cycling	downhill or uphill	downhill or uphill	two-layer neural network
Dasgupta et al. [29]	2018	walking	cognitive load	load (yes/no)	logistic regression, support vector machine, random forest, and learning vector quantisation
Gellaerts et al. [30]	2018	ski mountaineering	ski mountaineering activity	kickturn, skin on, skin off, backpack	threshold-based
Giandolini et al. [31]	2014	running	foot strike	rearfoot, midfoot, forefoot	threshold-based
Groh et al. [32]	2016	freestyle snowboarding	grind tricks and air tricks	Grinds: 50–50, BS-Boardslide, FS-Boardslide; Airs: Method, BS-180, FS-360	NB, C4.5, KNN, SVM with a radial basis kernel
Groh et al. [33]	2017	skateboarding	trick performed	ollie, nollie, kickflip, heelflip, pop shove-it BS, pop shove-it FS, 360-shove-it BS, varialflip, hardflip, double kickflip, 360-flip, bail	NB, RF, linear SVM, SVM with a radial-basis kernel, KNN
Hachaj and Ogiela [34]	2018	karate	karate kicks	frontal kick, knee strike, round kick and side kick	HMM
Hachaj et al. [35]	2017	karate	karate techniques	28 different karate techniques (e.g., different types of stances, block, kicks or punches)	DTW
Haider et al. [36]	2019	volleyball	volleyball specific actions	action (forearm pass, one hand pass, overhead pass, serve, smash, underhand pass, underhand serve, block), non-action	DT, KNN, NB, LDA, SVM
Helten et al. [37]	2011	trampoline jumps	jump classes	13 different jump classes (e.g., Barani, half twist, pike jump)	motion templates
Hendry et al. [38]	2020	ballet	movement type	3 levels: Level 1: jump vs. leg lift; Level 2: 3 jump types/3 direction of leg lift; Level 3: 5 laterality (landing leg)/6 laterality lifted leg	CNN
Hoettinger et al. [39]	2016	surfing	surfing on a wave	wave or not-wave	SVM, HMM
Holatka et al. [40]	2019	volleyball	setting technique	different movement setting labels	deep convolutional neural and LSTM recurrent network
Hollaus et al. [41]	2020	American football	drop or catch	catch or drop	CNN
Hu et al. [42]	2020	basketball	basketball activity	shooting, passing, lay-ups, dribbling	DTW
Huang et al. [43]	2010	walking and running	walking vs. running	walking vs. running	Bayesian network classifier
Huang et al. [44]	2019	football	football movements	walking, dribbling and stepover	neural network binary classifiers, sequence mapping
Jang et al. [45]	2018	cross-country skiing	cross-country skiing techniques	8 different classical or skating styles	CNN-LSTM, KNN
Jensen et al. [46]	2013	swimming	swimming styles	butterfly, backstroke, breaststroke, and freestyle	classifier based on linear regression
Jensen et al. [47]	2016	swimming	swimming styles	butterfly, backstroke, breaststroke, freestyle, rest, turn	AdaBoost, logistic regression, DT, SVM
Jeong et al. [48]	2019	physical workout	physical workouts	pull up, row-barbell, bench press, dips, squat, deadlift, and military press	CNN, LSTM, SVM, multilayer perceptron, 2D CNN
Kos et al. [49]	2016	tennis	tennis stroke	serve, hit, bounce, net, and null	threshold-based
Labintsev et al. [50]	2021	karate	punch class	Yun Tsuki, Mawashi Tsuki, Age Tsuki, Uraken, no punch	CNN
Ladha et al. [51]	2013	climbing	climbing vs. non-climbing	climbing vs. non-climbing	KNN, DT, logistic regression
Li et al. [52]	2021	Baduanjin (Chinese sport)	motion class + quality	8 standard motions	KNN, SVM, NB, logistic regression, DT, ANN, 1D-CNN
Lisca et al. [53]	2021	soccer (goalkeeper)	motion classes	dive or not dive + 27 motion classes	XGBoost, (ANN, SVM, quadratic discriminant analysis for comparison)
Ma et al. [54]	2018	basketball	basic basketball movements	9 kinds of basic basketball movement	two-layer feedforward network
Ma et al. [55]	2018	resistance training	low-intensity resistance band activity	tummy rotation, straight arm pull, cross and pull, leg press	CNN
Malawski and Kwolek [56]	2018	fencing	type of dynamic action	4 lunge types, step forward, step backward	DTW, SVM, RF
Matsuyama et al. [57]	2021	dancing	dance figures	13 types of dance figures	LSTM
McGrath et al. [58]	2019	cricket	bowl or non-bowl	bowl or non-bowl	RF, linear SVM, polynomial SVM, neural network, gradient boosting (XGB) algorithms
Mo and Zeng [59]	2019	running	running gait pattern	forefoot strike or rearfoot strike	threshold-based
Nguyen et al. [60]	2015	basketball	basketball activity	walking, running, jogging, pivot, shoots from different locations, layups, sliding, sprinting, undefined	SVM
Nishizaki and Makino [61]	2019	tennis	tennis swing	player ID	LSTM
Nurwanto et al. [62]	2016	light sport exercise	light sport activity exercise	push up, sit up, squat jump	KNN with DTW
Ogasawara et al. [63]	2021	archery	shooting detection	shooting detector (yes/no)	DT
Ooi and Gouwanda [64]	2021	badminton	stroke type	backhand cross court net, backhand lift, forehand drop, forehand lift, forehand lob, forehand smash	NN
Peng et al. [65]	2018	volleyball	volleyball jumping pattern	12 target class motions	neural network
Preatoni et al. [66]	2020	functional fitness exercises	fitness drills	4 popular functional fitness drills	KNN, SVM
Qi et al. [67]	2019	gym physical activity	gym activity	first: free weight; non-free weight; free weight: bench press, deadlifts, squats; non-free weight: walking, running, sitting	first: SVM, second: NN, (DT, KNN, HMM for comparison)
Qiu et al. [68]	2021	kayak	rowing cycle	left blade entry, left blade pull stage, left exit stage, left recovery stage, right blade entry, right pull stage, right exit stage, the right recovery stage	SVM, KNN, DT, ensemble learning
Rawashdeh et al. [69]	2016	baseball, volleyball	throw, serve, null class	throw, serve, null class	two-stage approach: first, rule, second decision tree
Reilly et al. [70]	2021	soccer	change of direction	changes of direction	logistic model tree, RF
Rindal et al. [71]	2018	cross-country skiing	sub-technique	8 cross-country sub-techniques	NN
Sha et al. [72]	2021	table tennis	hits and misses	hits and misses	SVM, DT, LDA, KNN, NB
Shi et al. [73]	2018	basketball	basketball motion	dribble, jump, turning around	k-means
Silva et al. [74]	2019	boccia	boccia bowl throw	gesture A, gesture B	SVM with a DTW kernel; KNN, linear SVM, RBF SVM, DT, RF, NB, Gaussian process
Stoeve et al. [75]	2021	soccer	ball contact	null, pass, shot	SVM, CNN, LSTM, convLSTM
Strohrmann et al. [76]	2013	running	arm carriage during running	arms parallel driving forward, arms aiming at symmetry line, arms crossing symmetry line	NB, KNN, SVM, logistic regression
Sun et al. [77]	2021	volleyball	action type	volleyball actions	SVM
Tabrizi et al. [78]	2020	table tennis	stroke type	topspin, push and basic	LSTM, 2D-CNN, RFB SVM
Takata et al. [79]	2019	kendo	basic strike-thrust activities and striking positions	4 general types: Men, Tsuki, Do, Kote; further 8 detailed types	RF, SVM, KNN, NN
Tjønnås et al. [80]	2019	cross-country skiing	sub-techniques	6 techniques	decision-function-based
Vales-Alonso et al. [81]	2015	volleyball	movement	jump detection	KNN
Wang et al. [82]	2020	swimming	swimming phase	swimming phase	SVM, NB, DT, KNN, logistic regression, hybrid NN/HMM model
Wang et al. [83]	2019	swimming	swimming style	butterfly, breaststroke, freestyle and back crawl	HMM
Wang et al. [84]	2015	ball games	discipline	badminton, tennis or ping-pong	SVM
Weiping et al. [85]	2017	tennis	tennis stroke type	forehand, backhand, serve	threshold-based
Wu et al. [86]	2019	yoga	yoga posture	18 common yoga postures (in stand, lean and lie categories)	two-stage: back-propagation artificial neural network + fuzzy C-means
Xia et al. [87]	2020	racket sport	badminton and table tennis	4 types of table tennis (service, stroke, spin, picking-up), 4 types of badminton (service, drive, smash, picking-up), walking	multilayer hybrid clustering
Zago et al. [88]	2019	general running (soccer players in experiment)	turn direction	left or right	LDA, SVM, boosted tree, ANN
Zhang et al. [89]	2018	gym exercises	exercise type/exercise set-up	different push-ups, and barbell curls movement	random forest
Zhang et al. [90]	2019	table tennis	ping pong basic movements	forehand attack, forehand drive, forehand chop, forehand flick, backhand control, backhand drive, backhand chop, backhand flick
Zhang et al. [91]	2018	tennis	tennis stroke type	forehand topspin, forehand slice, backhand topspin, backhand slice, serve	LSTM
Zhang et al. [92]	2021	football	football actions	static, walking, jogging, juggling, pass, trapping, tackle, shoot	DT, SVM
Zhao et al. [93]	2019	tennis	tennis stroke style	serve, ground-stroke, volley	RF, NB, AdaBoost, SVM, DT
Zhao and Chen [94]	2020	basketball	basketball posture/action	dribbling, passing, catching, shooting	SVM, back-propagation neural network
Zhao et al. [95]	2021	gymnastics	gymnastics motion	stretching, chest, body rotation, kicking, body side, whole-body exercise	SVM, KNN, NB, DT
Zheng et al. [96]	2019	swimming	stroke phase	propulsion, glide, recovery	KNN
		Classification of a quality related outcome
Brock et al. [97]	2017	ski jumping	ski jumping motion style error	error jump/non-error jump	SVM, DTW
Brock and Ohgi [98]	2017	ski jumping	ski jumping errors	error jump/non-error jump (for 9 jump categories)	CNN, SVM, HMM
Ebert et al. [99]	2017	bodyweight exercises - fitness	quality of movement	1 (very good) to 5 (very bad)	RF, DT, SVM, NB, Auto-WEKA
Jensen et al. [100]	2012	golf	experience level	experienced vs. unexperienced player	AdaBoost, Fisher-LDA, KNN, NB, SVM-linear, SVM-rbf
Jung et al. [101]	2020	walking	group	semiathlete, ordinary, deformity	DCNN
Kianifar et al. [102]	2016	squat	squat quality	good, (moderate), poor	SVM, linear multinomal logistic regression, DT
Liang et al. [103]	2017	chest and hip rotation/trunk movement	execution quality	good, poor	SVM
Liu et al. [104]	2020	running	performance level	competitive, recreational, or novice	gradient boosting decision tree, CNN, MLP
Liu et al. [105]	2020	canoeing	proficiency level	coach, novice	SVM, Logistic Regression, DT, XGBoost
Malik et al. [106]	2014	walking	gait patterns	normal, average, poor	adaptive fuzzy logic based classification module
Mitchell et al. [107]	2015	running	running type; synchronicity of running steps	symmetrical or asymmetrical; synchronous vs. asynchronous	RF, NB, Radial Basis Function Network
Rose et al. [108]	2021	baseball	throw identification and quality	low, medium, high	quadratic SVM
Sanusi et al. [109]	2021	table tennis	stroke quality	correct, incorrect	LSTM
Viyanon et al. [110]	2016	table tennis	quality	in, out, hitnet	DT
Zhao et al. [111]	2016	archery	archer’s release quality	good, dead, pluck	RF

DT: decision tree, SVM: support vector machine; RF: random forest; boosting: models that are based on boosting algorithms; KNN: k-nearest-neighbours; NB: naive Bayes; GMM: Gaussian mixture models; DTW: dynamic time warping; HMM: hidden Markov model; Log R.: logistic regression; own dev.: own developed method; LDA: linear discriminant analysis; CNN: convolutional neural network; LSTM: long short-term memory.

1.3. Data Items and Synthesis

For each included article we extracted and analysed information on the article type, publication year, discipline, data source, IMU type, position of the sensor, the classification method, the method how temporal dependency was considered, sample size, resampling techniques, data set split, dimension reduction, number of classified labels, the performance metrics, and the classification results. The extracted data were then aggregated to be displayed via tables and summarised based on a narrative synthesis. The included disciplines are further displayed with a Word Cloud, generated with the software R [112].

2. Results

Analysing the 93 articles that were included in the review showed that classification of IMU data was applied in a lot of different sports disciplines: individual sports, such as running, ski mountaineering or surfing, as well as teams sports, such as football, basketball or volleyball. Classification of motion can be found in both summer sports (e.g., surfing) and winter sports (e.g., cross-country skiing) disciplines. The disciplines that were analysed most frequently were tennis and workout training/exercises (seven contributions each), followed by basketball and soccer (six contribution each). Figure 3 shows the different sports disciplines ordered by their frequencies where similar terms were grouped (e.g., all fitness exercises and workouts were grouped together).

Figure 3. Disciplines that were analysed in the included articles.

The dependent variable classified was either an activity related to sports, a motion primitive, such as the tennis stroke type or swimming style, or a quality related to the outcomes, such as good or bad movement quality. See Table 2 for the disciplines and the dependent variable for each of the included articles, split first into those articles that analysed a motion type, and second, those that analysed a quality-related outcome.

2.1. Data Sources and Sample

All articles analysed captured acceleration, and 71 also used gyroscopes (angular velocity) in data collection. Thirty-five papers used IMU that included magnetometers, although not all of them used this data. In addition to IMU sensors, some papers used other data sources, such as GPS [28,70,80] or barometric pressure sensors (e.g., [30,60,79,80]). Ref. [57] also included 3D pose data, ref. [28] combined acceleration data with GPS and heart rate data, and Ref. [109] combined and compared smartphone IMU data with data from the Microsoft Kinect.

In most papers (79), the IMU type was a wearable or specialised hardware, five analysed IMU data collected via smartwatches or activity trackers, and seven with smartphones or iPods. Further, Ref. [73] used insole sensors, and some papers did not specify the sensor type [36,42]. IMU sensors were either attached directly to the body or to clothing, e.g., belts, shoes, or gloves, or they were mounted on sports equipment (14). For some summer sports, particularly those involving rackets (tennis) or clubs (golf), and for some winter sports (skiing), IMU were often attached directly to the sports equipment (see Table A1 in the Appendix A for an overview of the type and position of IMU).

The number of subjects ranged from 1 [54,65,109,111] to 836 [75] subjects (mean = $21.7$, median = 10, sd = $88.8$) from whom data were collected. However, not all authors described the number of subjects. Oversampling techniques to increase sample size were used by [40,46,53,65,71]. Ref. [63] prepared random additional negative samples to ensure a balanced training set, ref. [102] also ensured a balanced data set, and [70,75] down-sampled the majority class. In addition, the general challenge of unbalanced data sets was mentioned by some authors in the text (e.g., [25,67,78]). In the work of [96], the training data set (five stroke cycles) was much smaller than the test data set (more than 60 cycles). In the study by [80], the authors used data collected during another study. In addition, Ref. [25] used different data sets, one of which was based on simulated data. The effect of sample size on classification performance was investigated by [47], who tried different sample and window sizes, as well as different time intervals and sizes of training sets. Additionally, [107] analysed different combinations and numbers of sensors.

2.2. Preparation

Dimension reduction methods were applied in 34 works [19,21,23,25,29,34,35,37,39,47,51,54,58,65,67,68,69,70,72,73,76,77,78,82,87,94,95,98,99,100,102,103,105,106] and ranged from visual analysis to wrapper functions. In 12 works [21,34,39,51,67,68,77,82,87,94,102,105], principal component analysis (PCA) was used, seven applied correlation calculations to identify redundant features [21,23,29,58,65,72,103] and 25 [19,21,23,25,29,35,37,47,54,65,69,70,72,73,76,77,78,83,95,98,99,100,103,105,106] works applied another dimension reduction method. Some authors tried more than one dimension reduction method, such as [102] who used 18 different feature selection techniques. Ref. [56] evaluated each feature separately and then performed a validation using the feature fusion of all feature sets. Ref. [34] tried two different versions of data sets, one with PCA (five features), one without. Furthermore, refs. [42,94] tried different versions and numbers of input features and compared the performance of the classifiers.

Table 3 gives a detailed overview of the papers where feature reduction was described and the details of both the full and the reduced feature set were given. The table states the number of features of the full data set and the number of features of the reduced data set.

Table 3. Details of the full and reduced feature set.

Refs.	Number of Features before Reduction	Number of Features after Reduction
[19]	153	28
[21]	90	15/25
[23]	approx. 2000	300–500, which were further reduced
[34]	8	5
[35]	16	12
[52]	306	96
[58]	282	223
[65]	32	24
[66]	108/216	20
[67]	88	36
[68]	23	4
[69]	81	8
[70]	25	16 (for RF)
[73]	24	3
[78]	840	52 (for RBF-SVM)
[87]	120	1
[94]	72	30
[95]	84	11
[99]		information was compressed by a ratio of 1:109
[102]	210	2 classes: 7; 3 classes: 4
[103]	97	45, 50, 46
[104]	132	6
[106]	6	4

2.3. Classification

2.3.1. Data Set Split

To validate the trained classification models, the sample was split into several sets. In 33 papers, the sample was split on a subject-wise basis, so that the algorithm was trained with different subjects than those included in the validation data set. Further, in six works, the classifiers were trained with some samples of a person and tested with the remaining samples of that person, and in 65 works, the data set split into training, testing, and validation was either random or not clear. In ten papers, different versions of data set splits were tried. Table 4 shows the type of data set split in each article. Further, to analyse the details of the data set split, Table A4 in the Appendix A gives the details on the split into training, validation, and test data set for each paper. As seen in Table A4, the split varies from simple splits into training and test data sets, to cross-validation approaches and leave-one-subject-out cross-validation procedures.

Table 4. Type of data set split into training, validation and test data set.

Split Type	Article	Total Articles
Subject-wise	[20,23,25,26,27,29,32,34,35,36,38,39,45,46,47,53,56,57,60,62,63,66,71,73,79,82,87,100,101,102,105,106,110]	33
Person-dependent	[26,27,56,60,91,96]	6
Random split/not clear	[19,21,22,24,28,29,30,31,33,35,37,39,40,41,42,43,44,48,49,50,51,52,53,54,55,58,59,61,64,65,66,67,68,69,70,71,72,74,75,76,77,78,79,81,82,83,84,85,86,88,89,90,92,93,94,95,97,98,99,102,103,107,108,109,111]	65
Different versions	[29,35,39,43,71,79,82,91,102,104]	10

2.3.2. Classification Methods

The classification methods can be generally divided into three main branches: first, classic machine learning methods, such as decision trees, k-nearest neighbours (KNN), or naive Bayes (NB) classifiers, which were applied in 78 papers. Second, seven works developed threshold or correlation-based algorithms to classify the motion data into predefined groups. Third, dynamic time warping (DTW) clustering approaches were used in seven works. Table A2 in the Appendix A shows a detailed overview of the methods used in each article. In addition to these three main approaches, the authors’ own developed methods were applied in seven works. Refs. [73,86,87] applied cluster-based approaches. Some authors used hybrid or two-stage approaches. Ref. [32] classified, in their skateboard classification, first grid or air and then the final trick in the second stage. Ref. [86] applied a back-propagation artificial neural network in the first stage, and in the second, fuzzy C-means. Ref. [69] combined a rule to first spot a motion gesture of interest (first stage) with a decision tree for activity classification (second stage). Ref. [82] classified one of four styles in the first stage and continued with a more detailed classification in the second stage. The number of classes varied from binary classification [29,39,41,43,51,53,58,59,63,70,74,81,88,98,100,103,107,109] to 28 labels in [35]. Ref. [53] applied binary classification to identify dives or no-dives of goalkeepers as well as multi-class classification to identify motion classes. Ref. [21] also tested an ensemble approach where they combined random forests, a support vector machine (SVM) and KNN with a voting approach. The authors also included personal characteristics in their classification models. Ref. [110] calculated two separated models for forehand and backhand strokes and Ref. [67] proposed a two-layer recognition framework.

A detailed analysis of the classification methods (see Table 5 and Figure 4) revealed that SVM was the method most frequently used in 42 articles, followed by any type of neural networks (39) and KNN (23). Out of the 39 papers that used any type of neural network (NN), 10 applied an LSTM. Sixteen works used a random forest, 19 a decision tree, and eight models were based on boosting algorithms. Seventeen applied NB classifiers, 11 logistic regression, seven DTW, and six linear discriminant analysis. Nearly half (42; 45%) of the works only used one classification method; the others compared more than one where the total range goes from one to eight different classifiers.

Table 5. Classification methods applied in the included articles.

SVM	NN	KNN	RF	DT	NB	log. reg.	Boosting	HMM	Other
42	39	23	16	19	17	11	8	6	26

SVM: support vector machine; KNN: k-nearest neighbours; NN: neural network; RF: random forest; DT: decision tree; NB: naive Bayes; log. reg.: logistic regression; boosting: models based on boosting algorithms; HMM: hidden Markov model.

Figure 4. The number of applied methods in the literature. (SVM: support vector machine; NN: neural network; KNN: k-nearest neighbours; DT: decision tree; NB: naive Bayes; RF: random forest; log. reg.: logistic regression; boosting: models based on boosting algorithms; DTW: dynamic time warping; own dev: own developed method; HMM: hidden Markov model; LDA: linear discriminant analysis; GMM: Gaussian mixture models).

Moreover, Figure 5 shows the number of applied algorithms per sports discipline. Interestingly, in some disciplines such as ski jumping, cross-country skiing, ballet/dancing, or archery, very few different algorithms are used. For the categories fitness, yoga, bodyweight, exercises, and badminton, baseball, cricket, golf, tennis, table tennis, on the other hand, almost the entire arsenal of machine learning algorithms is used in the literature.

Figure 5. The number of applied methods per discipline. (SVM: support vector machine; NN: neural network; KNN: k-nearest neighbours; DT: decision tree; NB: naive Bayes; RF: random forest; log. reg.: logistic regression; boosting: models based on boosting algorithms; DTW: dynamic time warping; own dev: own developed method; HMM: hidden Markov model; LDA: linear discriminant analysis; GMM: Gaussian mixture models).

2.3.3. Dependency Structure

The temporal information of the IMU data was included in different ways in the articles analysed. Table A3 in the Appendix A shows a detailed overview of the methods applied in each article. On the one hand, six papers applied hidden Markov models (HMM) and ten papers used LSTM or convolutional neural networks (CNN)-LSTM, which can handle temporal data. Further, ten works used DTW or a longest common subsequence algorithm, 12 used CNN, and one applied sequence mapping [27]. On the other hand, 23 articles used rolling windows with an overlap in the feature calculation process, and thus incorporated the time-dependent data structure in a more indirect way by these overlapping windows. Ref. [66] tried different windows and window overlapping sizes and in addition, the effect of subsets of the five IMUs. In 42 works, other classification methods were applied, no details were provided, or time dependence was not incorporated.

Table 6 and Figure 6 give an overview of the different ways dependency was handled in the papers. Since some papers computed the features using rolling windows with an overlap and applied classification methods that take into account temporal information, the sum is greater than 93.

Table 6. Methods of how temporal information was considered.

HMM	LSTM/CNN-LSTM	CNN	Sequence Mapping	DTW/LCSS	Overlap	Other Classification Method/No Details
6	10	12	1	10	23	42

HMM: hidden Markov model; LSTM: long-short term memory network; CNN: convolutional neural network; DTW: dynamic time warping; LCSS: longest common subsequence; overlap: overlap in the feature calculation.

Figure 6. The number of applied methods to consider temporal information. (overlap: overlap in the feature calculation; CNN: convolutional neural network; LSTM: long-short term memory network; DTW: dynamic time warping; LCSS: longest common subsequence; HMM: hidden Markov model; seq: sequence mapping).

2.4. Validation

The classifiers were validated using different methods. Table A4 in the Appendix A and Figure 7 give a detailed overview of the validation methods applied in each article. Accuracy was the most often used performance metric (78), followed by the confusion matrix (60). In total, 37 papers reported recall, 32 precision, 22 F1 score, 10 specificity, and 7 receiver operating characteristic (ROC) curves or area under the curve (AUC). In addition, six papers also considered computation time and cost. One reported Cohen’s Kappa and one paper reported Matthews correlation coefficient.

Figure 7. The number of validation methods in the literature. (Acc: Accuracy; Conf. M.: Confusion Matrix; Rec.: Recall; Pr.: Precision; F1: F1 Score; ROC: Receiver Operating Characteristic; AUC: Area Under the Curve; Comp: Computational).

2.5. Classification Performance

The classification accuracies ranged up to 100% [35,88,105] in the analysed work. Due to the different classification tasks for the distinct disciplines, the different experimental set-ups (including different numbers of subjects and trials, different numbers and positions of sensors etc.), and the different validation procedures (e.g., cross-validation, leave-one-subject-out validation etc.), a direct comparison between the different works was not carried out.

However, to emphasize a better understanding of the performance of machine learning algorithms, we compared the results of the studies where multiple machine learning approaches were compared with each other. This analysis showed that the best model differs between the works and also within the works, depending on, e.g., different validation procedures or different training data sets. In [100], the best model varied between the different types of features. While NB performed best when all features were used, KNN performed best in a leave-one-subject-out feature selection. In [102], the best performing model depended on the number of classes to classify. Ref. [56] found differences between person dependent and person independent classification tasks. While DTW performed better in the person-dependent case, SVM performed better in the person-independent case [56]. Ref. [58] compared random forests, linear SVM, polynomial SVM, NN and a gradient boosting algorithm and found that no model was better than the others. Ref. [66] compared different classifiers with different window sizes and found that the SVM with a cubic kernel with 600 ms windows and a 10% overlap performed best. In 15 out of the 36 times (41.7%) where SVMs were used in comparative works, SVMs achieved the highest accuracy. Random forest achieved the highest accuracy values in 8 out of 14 (57.1%) works, neural networks in 10 out of 20 (50.0%), and boosting approaches in 4 out of 8 (50.0%) works where more than one classifier was compared. Table A2 in the Appendix A marks the best performing method with the highest classification accuracy for each work that applied more than one classifier.

3. Discussion

The aim of this scoping review was to to present an overview of the methods used in the literature to classify human motion data based on IMU data, to analyse whether and how the time-dependent data structure of IMU sensors is taken into account in the classification step of the ARC, and further to identify research areas that need further detailed investigation, e.g., in the form of a systematic review. Due to the variety of disciplines and classification tasks, an aggregation of the results was not possible. However, in conducting the scoping review, data extraction also proved to be difficult because of the different disciplines involved and sometimes because of inconsistencies in reporting the data analysis process. For example, sample size is reported in a variety of ways, sometimes as total sample (e.g., steps), per subject, or number of trials (e.g., runs). Furthermore, the included features are not always clearly described and the number of features used is not always comprehensible. The different division of the data into training, test, and validation sets is also not always clearly stated which might be due to page limits. However, for direct comparisons and the identification of the best performing classifiers for certain research questions, this information would be needed.

The analysis revealed that classification methods have been applied to a variety of disciplines, where tennis, soccer, and basketball were the three most often analysed sports. Human motion data was mainly classified via deep learning, classic machine learning, DTW, or threshold-based approaches. For each of these methods, different pros and cons can be identified and the approaches differ in their algorithmic complexity, model accuracies, and degree of interpretability. While deep learning approaches show a high level of accuracy, they are less easy to interpret than rule-based learners, linear or logistic regression, and decision trees that show high interpretability but less model accuracy [113]. The SVM, the most often applied approach in the analysed works, was the classification method in $45.2\%$ of the papers, and shows a high generalisation capability but has weaknesses with regard to algorithmic complexity and the training time in large data sets, the challenge of correct classification in imbalanced data sets, and the development of classifiers for multi-class problems [114]. DTW, which has been successfully applied in various works in the area of biosignal processing, takes nonlinearities into account [115]. On the other hand, DTW is time consuming due to the calculation of the similarity distance for each pair of sequences [116]. LSTM and HMM are also popular tools for sequential and time series data [14,117].

In general, time-dependence of the IMU data was, if it was considered, incorporated in different ways: either directly within the methods applied, such as LSTM, or in the feature calculation step by incorporating time-dependency explicitly using rolling windows with an overlap.

Due to the difference in data collection, experimental set-up, and classification tasks, it is difficult to compare the classification performances across the works. In general, the analysis revealed that there is no single best-performing method, as analysing the classification methods compared within works revealed the best classifiers varied between the works and also depended on the training data sets/different number of features or validation procedures. Overall, deep learning models, random forests, boosting approaches and, SVMs mostly showed higher classification accuracy than decision trees, KNN, and NB in direct comparison. However, more complex models, such as LSTM, do not always achieve higher classification accuracy, as [48] found that the one-dimensional CNN performed best, compared to the SVM, multilayer perceptron, LSTM, and other deep convolutional neural networks. In this context, it also has to been mentioned that nearly half of the works, $45\%$, only described one classification method and thus lacked a comparison of different classification methods and approaches. Identifying the best working method is therefore not possible in these cases. Future work should concentrate on comparative works and, e.g., compare machine learning approaches with the researchers’ own developed threshold-classification approaches, or methods where the time series structure of the IMU data is considered with methods where it is not considered.

In most of the works, the data was either split random/record-wise into training, validation, or test data, or the split was not clear. However, the use of a record-wise split has been criticised, especially for clinical outcome predictions [118]. Thirty-three papers validated the models via a subject-wise split and some even tried different split, and thus validation, versions to analyse whether the models perform well and are generalisable to new subjects. Further, in around $1/3$ of the papers, feature reduction techniques were applied, although the number of reduced features was not always stated.

When analysing the four most frequently analysed sports disciplines that were analysed in more than five papers (tennis, soccer, and basketball), it can be shown that in soccer, 4/6 ($66.67\%$) of the papers applied a neural network approach [21,44,53,75] or an SVM [21,53,75,92]. While in basketball, $50\%$ of the works applied an SVM [19,60,94], no clear pattern can be identified for the classification methods when classifying tennis activities. Furthermore, when it comes to analysing the methods for incorporating time-dependence in the three sports disciplines, no sport-specific trend can be found. Future work could thus systematically compare and analyse classification methods in one specific sports discipline to investigate which of the different classification approaches performs best in certain circumstances in the form of a systematic review. These systematic reviews could also include quality assessments of the included works to analyse whether the lack of clarity is correlated with the journal citation index.

A further question could be to investigate the trade-off between model complexity, real-time processing requirements, and model accuracy. The generalisability of the developed models to different populations (in the sports domain, e.g., amateur, recreational, or professional athletes) could be investigated in future research and transfer learning approaches could be tested. Future research could also systematically analyse the effect of feature selection/dimension reduction for specific sports disciplines.

In addition, few articles mentioned the challenge of balance/imbalanced data sets and over- and undersampling of data, and data augmentation techniques were rarely used. Therefore, analysing the impact of unbalanced data sets and applying different resampling or data augmentation methods, such as the synthetic minority oversampling technique [119] or adaptive synthetic [120] etc., could be an interesting research question for future work. Hybrid, stacking, and ensemble methods were also rarely used and could be explored in future work. One of the papers included in this review used additional simulated data. The use and application of different data augmentation techniques for time series classification (see, e.g., [121]) to simulate human motion data could also be a future research question.

Future work should investigate the relationship between the choice of discipline and how it affects data requirements and algorithm performance. It is plausible to assume that the dynamics of a sport’s movement patterns raise the required signal quality in terms of sample rate and resolution. Likewise, it is unknown how the volume (i.e., the number of recorded motion primitives) of the collected sensor data affects classification performance. In summary, we suggest that future work investigates the interdependence between (expected) movement dynamics, sensor configuration, and algorithm design. This would help to build a systematic body of knowledge for sensor selection in sports science research (see [122]). Future work could also analyse how correct or incorrect segmentation of data into human motion primitives affects the ARC, in particular the classification step.

4. Conclusions

The main goal of this review was to investigate the methods used for classifying human motion primitives based on IMU data and to analyse whether and how the time series structure of the IMU data is taken into account in the classification process. The full text analysis showed that different approaches are used for classification, from machine learning and deep learning methods to self-developed threshold-based approaches and DTW. Direct comparisons of models to identify best-performing approaches proved to be difficult due to the different experimental set-ups and often vague descriptions of the classification processes, such as feature generation or data set split. The most often applied algorithms were SVM, followed by neural networks and KNN. When more than one classifier was compared in the works, random forests, neural networks, boosting models, and SVM achieved the highest accuracy values. Thus, we find that SVM and neural networks are both among the most commonly used methods and among the best-performing methods when multiple methods were compared.

The time-dependent data structure, when taken into account, was incorporated in different ways: either within the methods themselves, such as using LSTM models or DTW approaches, or within the feature calculation step by using rolling windows with an overlap, which was the method most often used to consider time dependencies.

The choice of classifier for future applications depends mainly on the task and discipline. The requirements and application goals (e.g., real-time requirements, computation time) may also play a role in selecting the appropriate classifier for future human motion classification projects.