Next Article in Journal
Sensory Processing Disorders in Children and Adolescents: Taking Stock of Assessment and Novel Therapeutic Tools
Previous Article in Journal
Contingent Social Interaction Does Not Prevent Habituation towards Playback of Pro-Social 50-kHz Calls: Behavioral Responses and Brain Activation Patterns
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Brain Sciences
Volume 12
Issue 11
10.3390/brainsci12111477
brainsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Detection of Fall Risk in Multiple Sclerosis by Gait Analysis—An Innovative Approach Using Feature Selection Ensemble and Machine Learning Algorithms

by
Paula Schumann
1,†,
Maria Scholz
2,†,
Katrin Trentzsch
2,
Thurid Jochim
1,
Grzegorz Śliwiński
1,
Hagen Malberg
1 and
Tjalf Ziemssen
2,*
1
Institute of Biomedical Engineering, TU Dresden, Fetscherstr. 29, 01307 Dresden, Germany
2
Center of Clinical Neuroscience, Neurological Clinic, University Hospital Carl Gustav Carus, TU Dresden, Fetscherstr. 74, 01307 Dresden, Germany
*
Author to whom correspondence should be addressed.
†
These authors have contributed equally to this work.
Brain Sci. 2022, 12(11), 1477; https://doi.org/10.3390/brainsci12111477
Submission received: 10 October 2022 / Revised: 24 October 2022 / Accepted: 26 October 2022 / Published: 31 October 2022
(This article belongs to the Section Sensory and Motor Neuroscience)
Browse Figures
Versions Notes

Abstract

:
One of the common causes of falls in people with Multiple Sclerosis (pwMS) is walking impairment. Therefore, assessment of gait is of importance in MS. Gait analysis and fall detection can take place in the clinical context using a wide variety of available methods. However, combining these methods while using machine learning algorithms for detecting falls has not been performed. Our objective was to determine the most relevant method for determining fall risk by analyzing eleven different gait data sets with machine learning algorithms. In addition, we examined the most important features of fall detection. A new feature selection ensemble (FS-Ensemble) and four classification models (Gaussian Naive Bayes, Decision Tree, k-Nearest Neighbor, Support Vector Machine) were used. The FS-Ensemble consisted of four filter methods: Chi-square test, information gain, Minimum Redundancy Maximum Relevance and RelieF. Various thresholds (50%, 25% and 10%) and combination methods (Union, Union 2, Union 3 and Intersection) were examined. Patient-reported outcomes using specialized walking questionnaires such as the 12-item Multiple Sclerosis Walking Scale (MSWS-12) and the Early Mobility Impairment Questionnaire (EMIQ) achieved the best performances with an F1 score of 0.54 for detecting falls. A combination of selected features of MSWS-12 and EMIQ, including the estimation of walking, running and stair climbing ability, the subjective effort as well as necessary concentration and walking fluency during walking, the frequency of stumbling and the indication of avoidance of social activity achieved the best recall of 75%. The Gaussian Naive Bayes was the best classification model for detecting falls with almost all data sets. FS-Ensemble improved the classification models and is an appropriate technique for reducing data sets with a large number of features. Future research on other risk factors, such as fear of falling, could provide further insights.

1. Introduction

People with Multiple Sclerosis (pwMS) are often at a high risk of falling due to many neurological abnormalities associated with the disease progression [1]. There are several risk factors in pwMS for falling, such as a high Expanded Disability Status Scale (EDSS), disease activity, sensory or cognitive impairments, impaired perception resulting from visual impairment and bladder incontinence as well as gait abnormalities and fear of falling [2,3,4,5,6,7,8,9,10]. Among these risk factors, gait abnormalities are considered to be one of the most common impairments in pwMS [11,12].
There are different possibilities to assess gait impairment, for example, rater-based, sensor-based and patient-based methods. Rater-based methods are objective and give an overview of different functional areas. Sensor-based methods are standardized non-human-dependent measures that allow multidimensional viewing of gait. Patient self-assessment plays an important role in measuring gait changes in daily life and, therefore, better reflects the health status [13]. Procedures applying cognitive tasks during walking are able to detect more sensitive gait abnormalities since patients with increased cognitive impairment need more attention while walking in order to compensate for further disorders [14]. Methods that measure balance, as well as methods for early identification of mobility impairments, already reveal changes that are not yet evident in normal walking [15,16,17].
Gait parameters can be correlated with falls. A review reported that pwMS who fall have a higher overall disability stage, a worse balance compared to non-fallers, a lower walking speed and a shorter stride length [8]. Early detection of functional decline could therefore prevent falls and injuries. Most previous studies used statistical hypothesis testing to identify patients at risk of falling. Retrospective fall status classification commonly helps to determine outcomes that are sufficiently sensitive to distinguish people who have fallen from those who have not [18]. However, due to the large number of outcome variables, it is difficult to compare results across investigations.
Machine learning algorithms allow for discovering new patterns that were hidden beneath the complexity of the data. Several studies have retrospectively analyzed fall detection with machine learning with a focus on the elderly population [19,20,21,22,23,24,25,26] and people with neurological disorders [27,28,29]. Different conventional algorithms were trained in these studies, among which Naive Bayes, Support Vector Machines, Decision Tree and k-Nearest Neighbor achieved high performances [19,20,21,22,24,25,27]. However, most of the studies considered only wearable sensor-based parameters or used a small cohort of patients. Deep learning methods applied in two further studies outperformed conventional machine learning algorithms, on the contrary [23,28], but need thousands of samples to analyze efficiently [30,31]. Research with patient-reported outcomes (PRO) and combining different assessment systems is still lacking.
Even a small amount of noisy data can further hide patterns from machine learning algorithms. To boost the detectability, feature selection is one of the most important processing steps [19,21,26,32]. Several techniques are available, each one with its own strengths and weaknesses. Because no best method has been generalized so far [33,34,35], feature selection ensembles have been applied [36].
In our study, we incorporate different sensor methods, clinically accepted patient-reported measures and neurologist-administered assessments for detecting fallers using machine learning-based techniques. The aim of our work is to identify the most suitable methods for fall detection. We investigate a feature selection ensemble to improve the classification performance. We hypothesize that methods that detect gait abnormalities more sensitively and measure early gait impairments would be better suited for fall detection.

2. Materials and Methods

2.1. Study Design

A sample of 1240 pwMS was recruited in the Multiple Sclerosis Center at the University Hospital Dresden from November 2020 until September 2021. Patients performed gait analyses according to the Dresden protocol for Multidimensional Walking Assessment (DMWA) [37] and answered the question of how often they have fallen in the last thirty days. The main inclusion criterion was the available response to the fall question. Patients without a clinical diagnosis of MS, pregnant patients and patients who were unable to walk, either with or without assistive devices, were excluded from the study. Written informed consent was obtained from the individuals for the publication of any potentially identifiable images or data included in this article. The studies involving human participants were reviewed and approved by the Ethics Committee at Technische Universität Dresden. Approval number: BO-EK-320062021.

2.2. Data Sets

In the present study, data were collected during the routine visit to our clinic by research nurses with the multidimensional gait analysis according to the DMWA protocol [37]. The multidimensional gait analysis comprises a systematic recording of human gait at different speeds under differentiated tasks with standardized test methods and additional balance testing while standing.
Subjects were tested with the GAITRite (CIR-Systems Inc., Franklin, NJ, USA) system. The GAITRite is an instrumental gait analysis system for recording spatiotemporal gait parameters. It consists of a portable electronic walking mat equipped with pressure sensors, which measure foot pressure and graphically display and quantify the gait pattern by means of processing software [38]. Subjects had to walk twice, passing the 8-meter walkway at their self-selected walking speed. This measurement resulted in two data sets. The first data set included items of normal walking (GR_N data set), and the second one included items of walking while answering questions (GR_D data set).
Another system used for measuring balance and spatiotemporal gait parameters is the Mobility Lab (APDM Inc., Portland, OR, USA) [39,40]. Six individual sensor units were attached to the patient’s wrists, ankles, sternum and lower back for recording the patient while walking. Furthermore, patients performed normal (ML_N data set) and dual-task (ML_D data set) walking over an 8-meter walkway for recording walking quality. For testing postural sway, the Romberg stance was performed with open (ML_S_EO data set) and closed eyes (ML_S_EC data set) in a firm stance with the feet hip-width parallel and the hands supported on the hips.
The widespread EDSS was used to assess the neurostatus of different functional systems (visual, brainstem, pyramidal, cerebellar, sensory, bowel bladder, cognition/fatigue, ambulation) [41]. The EDSS was assessed by a certified expert and depended on the rater’s subjective judgment [42]. Its subcategories of functional systems and total score formed the EDSS data set.
Two further assessments are the Timed 25-Foot Walk Test (T25-FW) and the 2-Minute Walk Test (2MWT). The most widely used and robust T25-FW was applied to determine walking speed over a distance of 7.62 m recorded with a stopwatch by a nurse [43]. Walking endurance was evaluated using the 2MWT by continuously recording the distance walked for 2 minutes using an odometer [44]. These two parameters, all functional scores as well as the EDSS score, were summarized in the Basic data set.
The patient’s subjective perception of increasing walking impairment was assessed with the 12-item Multiple Sclerosis Walking Scale (MSWS-12) (MSWS-12 data set) in which patients rate their limitations in walking ability due to MS during the past two weeks with a Likert-Scale (not at all, a little, moderately, quite a lot, extremely) [45]. The perception of already low impairment was measured with the 9-item Early Mobility Impairment Questionnaire (EMIQ) (EMIQ data set) by which patients describe their ability to walk in the past 30 days with a Likert-Scale (none of the time, some of the time, most of the time, all of the time) [17].
All 428 parameters for each data set are summarized in Table 1. In addition to these ten data sets, an 11th data set (All data set) was formed with all features.

2.3. Data Preprocessing

Data sets must be preprocessed before being applied to classification methods. Metric features were present in the data sets GR_N, GR_D, ML_N, ML_D, ML_S_EO and ML_S_EC. In case of missing values, features were imputed with median values. The MSWS-12 and the EMIQ data set are questionnaires and consist of ordinal features. These features were imputed with mode values in case of missing values. The EDSS data set consists of ordinal features. The EDSS score is determined using the functional scores (visual, brainstem, pyramidal, cerebellar, sensory, bowel/bladder, cognition/fatigue, ambulation). For missing values, we used knowledge-based imputation [41], or if not possible, we imputed with the mode value from patients with the same final EDSS score. The Basic data set consists of metric features and ordinal features of the EDSS data set. The metric features were also imputed with median values. All ordinal features have a natural order. Thus, we assumed these features as quasi-metric. Missing values of ordinal features were handled in the same way as the EDSS data set. All features were standardized before applying the classification models.

2.4. Feature Selection Ensemble

We wanted to investigate which of these eleven data sets (including the All data set) are suitable for fall detection using machine learning algorithms and explore the most important features of each data set. For this purpose, a feature selection ensemble (FS-Ensemble) was developed. Feature selection ensembles have successfully been applied in various publications [46,47,48,49]. The objective of our FS-Ensemble was to remove irrelevant and noisy features and thus increase the classification performance. The methodology is illustrated in Figure 1.
An ensemble benefits from different metrics used by each filter method to rate the features by relevance [50]. Therefore, we used four filter methods: Chi-square test (Chi-Square), information gain (InfoGain), Minimum Redundancy Maximum Relevance (MRMR) and RelieF. These methods are based on different metrics (statistical, information, mutual information, distance), which makes them suitable for an ensemble [46]. We limited the feature selection methods to filters because these methods are independent of a classification model in contrast to wrapper and embedded methods. As a result, these methods do not tend to overfit and will produce more general results than wrapper methods [32]. The filter methods Chi-Square, MRMR and RelieF were performed using MATLAB R2021b [51]. The filter method InfoGain was performed using WEKA 3.8 [52].
All four filter methods generated a ranked list of features. A threshold was applied to select a subset of features of each filter method (filter subsets). Then, a combination method combined the four filter subsets. In the literature, this approach showed a better performance than first applying a combination method and, subsequently, a threshold method [46,47].
Three thresholds and four combination methods were investigated according to a study by Seijo-Pardo et al. [46]. The thresholds were set to 50%, 25% or 10% and determined the number of features in each filter subset. The following combination methods were applied to merge the subsets:
  • Union: A feature was selected for the final subset if this feature was present in at least one of the filter subsets.
  • Union 2: A feature was selected in the final subset if this feature was in at least two of the filter subsets.
  • Union 3: A feature was selected in the final subset if this feature was in at least three of the filter subsets.
  • Intersection: A feature was selected in the final subset if this feature was in all of the filter subsets.
The final subset created in each process contains all input features used for the classification models. The combination methods were performed using Python version 3.8.8.

2.5. Classification Models

Four classification methods were applied: Gaussian Naive Bayes, Decision Tree, k-Nearest Neighbor and Support Vector Machine. The Classification and Regression Tree algorithm (CART algorithm) was used to build up the Decision Tree. The Radial Basis Function (RBF) kernel was used for training a Support Vector Machine. The parameters of the Decision Tree, the k-Nearest Neighbor and the Support Vector Machine were optimized using grid search. The ranges of the hyperparameters are shown in Table 2. A stratified 5-fold cross-validation with F1 score as evaluation score was implemented. F1 score is more suitable than the accuracy in the case of imbalanced data [53,54].
F1 score, recall, precision, specificity and Cohen´s kappa were calculated to evaluate and compare the different models based on the hyperparameter optimization using 5-fold cross-validation. The stratified 5-fold cross-validation was repeated 10 times to reduce bias when splitting the data into the folds.
A permutation test was performed on each model to verify how the results compare to random guessing [55]. The test performed 1000 permutations. Each permutation consists of data points with randomly chosen labels. This removed the dependency between features and labels. The p-value of this test was calculated to estimate whether the predictions were better than random guessing. Machine learning classification and evaluation were performed using Python version 3.8.8 and scikit-learn version 0.18.1 [56].

3. Results

3.1. Subject’s Characteristics

We had 1240 pwMS participate in the study; 911 were female (73.5%) and 329 males (26.5%) aged 17 to 80 (mean ± SD = 44.8 ± 12.2). Further, 172 (13.9%) pwMS stated having fallen. Thereby, age (p < 0.001, t-test) was a significant factor, as well as the type of therapy: no therapy, first-line therapy or second-line therapy (p < 0.001, Chi-square test). Participants reported a mean disease duration since diagnosis of 12.96 (SD = 57.68) years and a median EDSS of 3.0 (interquartile range IQR = 2.0). The median EDSS score was 2.0 (IQR = 2.5) for non-fallers and 5.0 (IQR = 3.0) for fallers. Table 3 summarizes the characteristics of the study participants.
Not all data sets were available for each patient due to mobility impairments, system errors or missing self-reports.

3.2. Feature Selection Ensemble and Classification

Four classification models were used to determine the most suitable method for fall detection. For a better overview, we focused on the best classification results for each data set. PROs (MSWS-12 and EMIQ) achieved the best F1 score (F1 = 0.54 ± 0.00). The All data set showed an almost equally good performance (F1 = 0.53 ± 0.00). In these three data sets, recall was higher than precision. Rater-based methods had a better F1 score (F1 = 0.48 ± 0.00 and F1 = 0.47 ± 0.01) than sensor-based methods. Among the sensor-based methods, the measurements with cognitive tasks showed worse F1 values (F1 = 0.43 ± 0.01 and F1 = 0.42 ± 0.00) than measurements during normal walking (F1 = 0.39 ± 0.00 and F1 = 0.38 ± 0.01). Balance tests showed the worst results (F1 = 0.30 ± 0.01 and F1 = 0.29 ± 0.02). Gaussian Naive Bayes was the best classification model for all data sets except for ML_S_EO. The k-Nearest Neighbor was the best model (F1 = 0.29 ± 0.02) for this method. However, the performance of this model was poorest in comparison to the other performances of the Gaussian Naive Bayes. ML_S_EO and ML_S_EC achieved the worst results with the Support Vector Machine (F1 = 0.20 ± 0.01 and F1 = 0.19 ± 0.02). All of the best-performing classification models of each data set generated a highly significant value (p ≤ 0.001) in the permutation test.
The best results for the F1 score for each classification model are shown in Figure 2. The overall best performances for each data set are shown in Table 4. If two results showed the same F1 score for a data set, the result with fewer features was chosen.
The optimized parameters for each classification model are shown in Appendix A (Table A1). Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, and Table A12 in Appendix B show the classification results for all eleven data sets. The FS-Ensemble improved the classification performance in most cases or at least did not lead to any fallbacks. There was no threshold or combination method that always worked best, but often a restrictive selection improved the performance.
When using the data set with all 428 features (All data set), all models showed a better F1 score when reducing the number of features. The Gaussian Naive Bayes outperformed the other classification models in each experiment. Additionally, the method showed a consistently lower standard deviation. Decision Tree achieved the worst performances in most cases. The best performance for the Gaussian Naive Bayes classification model was achieved by reducing to 9 features (99% reduction), for the Decision Tree when limiting the number of features to 9 (99% reduction), for the Support Vector Machine when restricting to 25 (94% reduction) features and for the k-Nearest Neighbor when using 44 features (90% reduction).
Figure 3 shows the classification performances of the four classification models using all data (All data set) depending on the number of features.
After determining the best method for each data set, we considered the list of features selected in each method. Table 5 shows the final feature subsets that achieved the best performance according to Table 4.
Although only features from EMIQ and MSWS-12 were selected by the filter methods, the All data set achieved the highest recall (74.0%). Q1_EMIQ, Q4_EMIQ, Q8_EMIQ and Q12_MSWS-12 were selected in both final subsets: using all data and the individual data set. For detecting falls with the Gaussian Naive Bayes when using the All data set, the features addressing the patient´s judgment of walking, running and stair climbing ability, subjective effort as well as necessary concentration and walking fluency during walking, frequency of stumbling and indication of avoidance of social activities were used. For rater-based data sets (Basic and EDSS), all given features were used to achieve the best performance. GAITRite data sets concentrated on step count, velocity and step and stride length for the left and right leg, whereas under dual-task conditions, only step and stride length were used. The final feature subset of the Mobility Lab normal walking data set used for fall detection consisted of the percentage amount of the individual stride phases, the gait speed and the stride length of the right leg. Among the dual-task conditions, the number of used features was reduced to the stride length of the left leg, the percentage amount of the Terminal Double Support phase of the right leg and the degree of the toe-off angle of both legs. During balance testing with eyes open, the jerk trajectory and degree of the 95% Ellipse rotation were the best features for detecting falls. For balance testing with eyes closed, mostly all features were used. The final feature subset of the MSWS-12 data set included, just like the All data set, the concentration and estimation of the balance and needed walking aid at home. The final feature subset of the EMIQ data set included, just like the All data set, the concentration, frequency of stumbling and indication of avoidance of social activities. Furthermore, the indication of avoidance of sports activities was used.

4. Discussion

Our objective was to investigate which of the eleven data sets is suitable to detect falls and the most important features for this purpose. A new feature selection ensemble (FS-Ensemble) and four classification models were applied. The Questionnaires (MSWS-12 and EMIQ) achieved the best performance (F1 = 0.54 ± 0.00/0.01) and seemed most suitable for fall detection when using the Gaussian Naive Bayes. Thus, we are able to confirm our hypothesis that patients’ self-assessments can better reflect health status than other methods. PROs can provide more in-depth insights into a specific domain, are thought to be more sensitive to changes in health status, and questions may seem more relevant and, therefore, more acceptable to patients [13,57,58]. This statement is supported by the composition of the feature subset of the All data set, which consists exclusively of questionnaire parameters. A recall of 74% for fall detection can be attained if using a selection of features of MSWS-12 and EMIQ.
Contrary to the assumption that balance parameters are more suitable in detecting falls, based upon a more sensitive recording of gait alterations compared to typical gait data [15,16], our results show that data sets with balance parameters (ML_S_EO, ML_S_EC) are less able to detect falls in pwMS (F1 = 0.30 ± 0.01 and F1 = 0.29 ± 0.02) instead of PROs and rater-based items with the used ML methods. Including further machine learning methods could still improve their F1 score.
Rater-based methods showed the second-best performance after patient-based methods. One reason could be the examination of different bodily functions. Research on fall risk factors in inpatients identified twenty different intrinsic and extrinsic factors [59]. This suggests that including multiple factors, as collected with the EDSS, leads to a better result than using only sensor-based data. The performances were not improved when including the T25-FW and the 2MWT in addition to the EDSS score and the functional scores (Basic data set). Thus, these methods bring no added benefit but instead increase the effort of data collection. In this case, we do not recommend this combination of tests and EDSS scores when detecting falls with machine learning.
Sensor-based systems such as the GAITRite system or the Mobility Lab show the best F1 scores with the Gaussian Naive Bayes but seem to be unsuitable for fall detection compared to the questionnaires and rater-based methods. The systems are only occasionally used in routine clinical practice, and clinicians only consider a few parameters in the assessment. An automatic analysis of the results of these sensor systems would be a benefit for the users. First, this could save personnel capacities, and second, an evaluation of all available parameters would be possible. Therefore, in our opinion, further research with the sensor-based system used is promising. Additionally, extending the database can improve performance by collecting more sensor-based data from fallen patients.
The Gaussian Naive Bayes outperformed the other classification methods and is suitable for handling these different data sets (Figure 2). Thus, we recommend using the Gaussian Naive Bayes for detecting falls in pwMS. The fewer features included in the calculations, the better the detection of falls (Figure 3).
Knowing that perceived fragility in the wake of fear of falling can lead to falls, we recommend including this factor in future studies. Fear of falling can reduce self-confidence, worsen physical health, including mobility, due to social withdrawal and physical inactivity, and subsequently lead to falls [8]. Therefore, some studies defined fear of falling as a risk factor for fall detection [60,61]. Thus, detecting fear of falling and appropriate treatment can reduce the risk of falling [8].
Our study is not without limitations. Self-reported questionnaires, as well as the question “How often have you fallen in the last thirty days?", always collect retrospective information, so a recall bias could impact the reporting of falls. A basic human need is independence. As reporting of falls indicates a lack of independence, MS patients may under-report their number of falls [10,62]. Bias could be reduced by recording falls more accurately via a diary, in which patients note falls daily. Furthermore, the use of longitudinal data would be interesting to predict falls.
Missing values are a major problem of studies, as they affect the significance of results. A natural selection of patients who were unable to complete gait tests due to the severity of their disease limits the validity of our results. Cognitive and balance tasks are more challenging since patients with increased cognitive impairment need more attention while walking in order to compensate for further disorders [14]. This may result in missing data in more severely impaired individuals and could explain the poor performance of the sensor-based data sets in our study (Figure 2) compared to other studies [14,15,16,17].
The data show significant differences between patients who fall and those who do not fall in relation to their age and how their MS disease is treated. To achieve better comparability of the data, a propensity score matching procedure would be beneficial. Due to the high number of patents, we included in our study (N = 1240), we have dispensed with this procedure.
Furthermore, the answer options for the questionnaires were coded from one to four or five. The questionnaire items own a natural order (ordinal features). Therefore, the items were assumed as quasimetric features in the preprocessing step to process all features equally and to use the same classification models. Thus, the results must be considered with caution. The numeric distances between the coded items could be unequal. Therefore, the results do not have to correspond to reality in this case, but this assumption is a common preprocessing step in statistics and machine learning [63,64].
Test sets were not used in our study due to the small data size. Therefore, the results could have a positive bias. We addressed this problem by only using filter methods in contrast to other studies [46,48,65]. Filter methods are more general than other methods and do not tend to overfit [35,66]. In addition, a stratified 5-fold cross-validation was used for the grid search and performance evaluation. This validation method was often used for small data sizes and to make the results more generally valid [67]. However, the split of the folds in the train and validation set is a significant issue for calculating and interpreting the validation score since the score could be obtained by chance [68]. Thus, the cross-validation was repeated 10 times for performance evaluation and preceded by a permutation test. These methods are sufficient to overcome the missing test sets.
The data sets in this study were imbalanced due to the nature of fall detection. The used machine learning methods are not suitable for handling these imbalanced data sets [69]. This problem can be reduced with feature selection to delete redundant and irrelevant features. Thus, the methods could achieve better performance. Our FS-Ensemble consisted of four filter methods. These filter methods are suitable to ensure diversity for the ensemble [36,55], but another ensemble configuration could increase the diversity and, thus, improve the classification performance. Seijo-Pardo et al. used an ensemble that consists of four filter methods and two embedded methods [46]. They summarized that the ensemble works better than the individual feature selection methods. Moghimi et al. investigated filter, wrapper and embedded methods as an ensemble [70]. A combination of each one of these selection methods achieved the best classification performance.
The FS-Ensemble failed when the data set already consisted of very few features (e.g., Basic, EDSS, EMIQ and MSWS12). The nature of our feature selection method allows for the selection of few or even only one feature. This feature size would be insufficient for some of the machine learning algorithms in this study. Most machine learning methods are suitable for detecting patterns by considering feature dependencies. This benefit disappears when using a few or just one feature. Furthermore, the applied permutation test failed in some of these cases (p > 0.001). This means that the classification results occurred randomly and confirms that few or one features in a machine learning model are insufficient for providing valid results. Since some classification models can handle a single feature within the data set, the same method was applied throughout all machine learning algorithms for completeness to detect the most suitable one. For this purpose, the FS-Ensemble was an appropriate method. But we recommend using a higher threshold and the union combination method on data sets with few features.

5. Conclusions

We investigated eleven data sets derived from widely used gait assessment systems and PRO questionaries for mobility assessment to evaluate their predictive power for the detection of falling in pwMS. We can confirm that patients’ self-assessments can better reflect health status than other methods. Therefore, easy to use questionnaires (MSWS-12 and EMIQ) can provide a highly cost-effective tool for day-to-day clinical praxis when assisted by a sufficient trained machine learning algorithm. These kinds of algorithms could be implemented in AI-based medical software with very little effort compared to sensor-based assessment systems. Nevertheless, sensor data allows additional in-depth insights into the specific impairment as the root cause of the fall risk and possible starting point for a successful treatment.
The FS-Ensemble is appropriate to improve the classification performance using data sets with more than 30 features. Further research is needed to investigate whether classification performance can be improved and generalized with more data from fallen patients. Additionally, the inclusion of more risk factors of falling, such as fear of falling, must be investigated for preventing falls.

Author Contributions

Conceptualization, P.S., M.S., K.T., T.J. and T.Z.; methodology, P.S. and M.S.; software, P.S.; validation, T.J. and K.T.; formal analysis, P.S. and M.S.; investigation, K.T.; resources, T.Z., H.M. and G.Ś.; data curation, K.T., P.S. and M.S.; writing—original draft preparation, P.S. and M.S.; writing—review and editing, P.S., M.S., T.J., K.T. and T.Z.; visualization, P.S. and M.S.; supervision, T.Z. and H.M.; project administration, T.Z. and G.Ś. All authors have read and agreed to the published version of the manuscript.

Funding

The APC was funded by M-ERA.Net project MBrace and is co-financed with tax funds on the basis of the budget passed by the Saxon State Parliament.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee at the Dresden University of Technology (protocol code: BO-EK-320062021 and date of approval: 02.08.2021).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on reasonable request from the corresponding author.

Conflicts of Interest

T.Z. received personal compensation from Biogen, Bayer, Celgene, Novartis, Roche, Sanofi and Teva for consulting services and additional financial support for the research activities from Bayer, BAT, Biogen, Novartis, Teva and Sanofi. M.S. received travel grants from Teva. P.S., K.T., T.J., G.S. and H.M. declare no conflicts of interest.

Appendix A

Table
Table A1. Results of the hyperparameter optimization for each data set; Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); Euclidean (Eucl.); Manhattan (Manh.). Expanded Disability Status Scale (EDSS); Basic data set = 25 Foot Walk Test + 2-minute walk test + EDSS; All data set = All features of the Basic data set, the GAITRite System data sets, the Mobility Lab Gait data sets, the Twelve Item Multiple Sclerosis Walking Scale data set and the Early Mobility Impairment Questionnaire data set.
Table A1. Results of the hyperparameter optimization for each data set; Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); Euclidean (Eucl.); Manhattan (Manh.). Expanded Disability Status Scale (EDSS); Basic data set = 25 Foot Walk Test + 2-minute walk test + EDSS; All data set = All features of the Basic data set, the GAITRite System data sets, the Mobility Lab Gait data sets, the Twelve Item Multiple Sclerosis Walking Scale data set and the Early Mobility Impairment Questionnaire data set.
ParametersBaselineThreshold 10%Threshold 25%Threshold 50%
UnionUnion2Union3Inter-sectionUnionUnion2Union3Inter-sectionUnionUnion2Union3Inter-section
All data set
DTCriterion:EntropyEntropyGiniEntropyGiniGiniEntropyGiniEntropyGiniEntropyGiniEntropy
Maximum depth:88464610778329
Minimum:1015611291610695221513
kNNWeights:DistanceDistanceDistanceDistanceUniformDistanceDistanceDistanceDistanceDistanceUniformUniformUniform
Distance metric:Eucl.Eucl.Eucl.Eucl.Manh.Eucl.Eucl.Eucl.Manh.Eucl.Eucl.Eucl.Eucl.
No. neighbors:210610324444333
SVMRegularization C:10101010010010010010010100100100100
Kernel coefficient γ:0.0010.010.010.010.10.0010.0010.010.10.0010.0010.0010.01
Basic data set
DTCriterion:GiniEntropyGiniGini-GiniEntropyGini-GiniGiniEntropyGini
Maximum depth:7755-465-710102
Minimum:201455-21195-20955
kNNWeights:DistanceUniformUniformUniform-DistanceUniformUniform-DistanceDistanceDistanceUniform
Distance metric:Manh.Eucl.Eucl.Eucl.-Manh.Eucl.Eucl.-Manh.Eucl.Manh.Eucl.
No. neighbors:2333-353-2229
SVMRegularization C:10010011-1001001-1001001000.1
Kernel coefficient γ:0.1111-0.111-0.10.110.0001
EDSS data set
DTCriterion:GiniEntropyGiniGini-GiniEntropyGini-GiniGiniEntropyGini
Maximum depth:6755-865-6682
Minimum:211455-12195-212175
kNNWeights:DistanceUniformUniformUniform UniformUniformUniform-DistanceDistanceDistanceUniform
Distance metric:Eucl.Eucl.Eucl.Eucl.-Eucl.Manh.Eucl.-Eucl.Eucl.Eucl.Eucl.
No. neighbors:2333-533-2339
SVMRegularization C:10010011-1001001-100101000.1
Kernel coefficient γ:0.1111-111-0.1110.0001
GR_N data set
DTCriterion:EntropyGiniEntropyEntropy-EntropyGiniGiniEntropyEntropyGiniEntropyGini
Maximum depth:1071010-9958101095
Minimum:11979-1117301112141524
kNNWeights:DistanceUniformDistanceDistance-DistanceDistanceDistanceDistanceDistanceDistanceDistanceDistance
Distance metric:Manh.Manh.Eucl.Manh.-Eucl.Manh.Manh.Manh.Manh.Manh.Manh.Manh.
No. neighbors:2323-22232222
SVMRegularization C:10100100100-100100100100100100100100
Kernel coefficient γ:0.010.0111-0.010.01110.010.010.11
GR_D data set
DTCriterion:EntropyGiniEntropyEntropy-EntropyEntropyEntropyGiniEntropyEntropyGiniGini
Maximum depth:891010-691010101089
Minimum:17995-22595216108
kNNWeights:UniformDistanceUniformDistance-DistanceDistanceUniformuniformDistanceDistanceDistanceUniform
Distance metric:Manh.Manh.Eucl.Eucl.-Eucl.Manh.Manh.Eucl.Manh.Eucl.Eucl.Manh.
No. neighbors:5222-52332223
SVMRegularization C:1010100100-10010100100100100100100
Kernel coefficient γ:0.010.10.11-0.010.1110.010.010.010.1
ML_N data set
DTCriterion:EntropyGiniGiniGini-GiniGiniGiniEntropyGiniGiniGiniEntropy
Maximum depth:6249-22575293
Minimum:811188-143013141111730
kNNWeights:DistanceDistanceDistanceUniform-DistanceDistanceUniformDistanceDistanceDistanceDistanceDistance
Distance metric:Manh.Eucl.Eucl.Eucl.-Manh.Eucl.Eucl.Manh.Eucl.Eucl.Eucl.Eucl.
No. neighbors:2223-22322222
SVMRegularization C:101001001-100100100100100100100100
Kernel coefficient γ:0.010.010.11-0.010.10.110.010.010.010.1
ML_D data set
DTCriterion:EntropyGiniGiniEntropyGiniGiniEntropyGiniGiniEntropyEntropyGiniEntropy
Maximum depth:666109106665567
Minimum:1551771051416915151511
kNNWeights:DistanceDistanceDistanceDistanceDistanceDistanceDistanceDistanceDistanceDistanceDistanceDistanceUniform
Distance metric:Eucl.Manh.Manh.Manh.Eucl.Eucl.Manh.Manh.Eucl.Eucl.Eucl.Manh.Manh.
No. neighbors:2422422222223
SVMRegularization C:100101001001010010100100100100100100
Kernel coefficient γ:0.010.10.00010.000110.010.10.110.010.010.010.1
ML_S_EO data set
DTCriterion:GiniEntropyGiniEntropy-GiniGiniEntropy-EntropyEntropyGini-
Maximum depth:108106-699-7109-
Minimum:16677-17147-14107-
kNNWeights:DistanceDistanceDistanceUniform-DistanceDistanceDistance-DistanceDistanceDistance-
Distance metric:Manh.Eucl.Eucl.Eucl.-Manh.Eucl.Eucl.-Manh.Eucl.Manh.-
No. neighbors:2329-624-422-
SVMRegularization C:10010100100-100100100-10100100-
Kernel coefficient γ:0.1111-0.111-0.10.11-
ML_S_EC data set
DTCriterion:EntropyGiniGini--EntropyGiniGini-GiniGiniGiniGini
Maximum depth:955--773-5779
Minimum:7721--7621-2010155
kNNWeights:UniformDistanceDistance--DistanceDistanceUniform-DistanceDistanceUniformDistance
Distance metric:Eucl.Eucl.Manh.--Eucl.Manh.Eucl.-Eucl.Eucl.Manh.Manh.
No. neighbors:324--223-3232
SVMRegularization C:10010010--10010010-10010010010
Kernel coefficient γ:0.10.11--0.111-0.10.111
MSWS-12 data set
DTCriterion:EntropyEntropyGiniGini-EntropyEntropyGiniGiniEntropyGiniGiniGini
Maximum depth:7522-46224432
Minimum:241455-75557555
kNNWeights:UniformUniformUniformUniform-UniformUniformUniformUniformUniformDistanceDistanceUniform
Distance metric:Manh.Eucl.Eucl.Eucl.-Manh.Manh.Eucl.Eucl.Eucl.Eucl.Manh.Eucl.
No. neighbors:3333-393371033
SVMRegularization C:10010.10.1-1001000.10.1100100100.1
Kernel coefficient γ:0.110.00010.0001-0.10.10.00010.00010.10.110.0001
EMIQ data set
DTCriterion:GiniGiniGiniGini-EntropyGiniGini-giniginiGinigini
Maximum depth:2433-763-2223
Minimum:72755-1355-715155
kNNWeights:DistanceUniformUniformUniform-UniformUniformUniform-UniformUniformUniformDistance
Distance metric:Manh.Eucl.Eucl.Eucl.-Eucl.Eucl.Eucl.-Manh.Manh.Manh.Eucl.
No. neighbors:7799-359-9995
SVMRegularization C:1010011-10101-10101010
Kernel coefficient γ:0.10.111-111-0.1111

Appendix B

Table
Table A2. Performance of the four classification models on detection fall using all data; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A2. Performance of the four classification models on detection fall using all data; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-4280.44 ± 0.0150.9 ± 1.038.1 ± 0.886.9 ± 0.30.33 ± 0.010.001
50%Union3450.44 ± 0.0150.7 ± 0.838.4 ± 0.787.1 ± 0.30.33 ± 0.010.001
Union 22460.45 ± 0.0151.8 ± 0.640.1 ± 0.787.7 ± 0.30.35 ± 0.010.001
Union 31910.45 ± 0.0051.6 ± 0.839.9 ± 0.487.7 ± 0.20.35 ± 0.010.001
Intersection740.43 ± 0.0053.6 ± 0.736.2 ± 0.485.1 ± 0.20.32 ± 0.010.001
25%Union2240.45 ± 0.0054.4 ± 0.738.6 ± 0.386.3 ± 0.20.35 ± 0.000.001
Union 21160.48 ± 0.0159.1 ± 0.640.1 ± 0.686.0 ± 0.30.38 ± 0.010.001
Union 3710.49 ± 0.0063.6 ± 0.639.7 ± 0.284.7 ± 0.10.39 ± 0.000.001
Intersection170.53 ± 0.0074.4 ± 0.441.2 ± 0.283.2 ± 0.10.43 ± 0.000.001
10%Union940.48 ± 0.0063.6 ± 0.638.9 ± 0.484.2 ± 0.20.38 ± 0.010.001
Union 2440.53 ± 0.0172.9 ± 0.841.5 ± 0.483.7 ± 0.20.43 ± 0.010.001
Union 3250.52 ± 0.0073.8 ± 0.540.4 ± 0.382.8 ± 0.20.42 ± 0.000.001
Intersection90.53 ± 0.0074.0 ± 0.041.0 ± 0.383.1 ± 0.20.43 ± 0.000.001
DT100%-4280.30 ± 0.0228.9 ± 2.030.6 ± 3.189.5 ± 1.40.19 ± 0.030.001
50%Union3450.30 ± 0.0528.4 ± 5.131.7 ± 4.190.3 ± 1.00.20 ± 0.050.001
Union 22460.27 ± 0.0420.9 ± 4.038.9 ± 1.494.8 ± 1.00.19 ± 0.030.001
Union 31910.28 ± 0.1022.4 ± 10.740.8 ± 3.695.0 ± 2.20.21 ± 0.080.001
Intersection740.32 ± 0.0427.2 ± 3.937.6 ± 3.792.8 ± 1.00.23 ± 0.040.001
25%Union2240.29 ± 0.0225.2 ± 2.735.4 ± 2.992.7 ± 1.20.20 ± 0.020.001
Union 21160.32 ± 0.0230.2 ± 2.734.5 ± 3.190.9 ± 1.20.22 ± 0.030.001
Union 3710.33 ± 0.0329.1 ± 3.537.4 ± 2.992.2 ± 1.00.24 ± 0.030.001
Intersection170.33 ± 0.0428.3 ± 4.141.3 ± 3.893.6 ± 1.10.25 ± 0.040.001
10%Union940.33 ± 0.0429.8 ± 5.138.5 ± 3.692.5 ± 0.90.25 ± 0.040.001
Union 2440.32 ± 0.0226.1 ± 2.443.1 ± 2.894.5 ± 0.60.25 ± 0.020.001
Union 3250.30 ± 0.0524.9 ± 5.739.1 ± 3.293.9 ± 1.30.22 ± 0.040.001
Intersection90.36 ± 0.0327.7 ± 2.750.5 ± 3.195.7 ± 0.30.29 ± 0.030.001
kNN100%-4280.33 ± 0.0232.3 ± 2.234.1 ± 1.390.1 ± 0.50.23 ± 0.020.001
50%Union3450.32 ± 0.0325.5 ± 2.641.5 ± 3.094.3 ± 0.60.24 ± 0.030.001
Union 22460.36 ± 0.0329.8 ± 3.245.2 ± 4.694.2 ± 0.80.28 ± 0.040.001
Union 31910.37 ± 0.0331.7 ± 3.343.6 ± 3.793.5 ± 0.80.28 ± 0.040.001
Intersection740.34 ± 0.0229.1 ± 2.341.8 ± 2.193.6 ± 0.60.26 ± 0.020.001
25%Union2240.34 ± 0.0233.1 ± 2.334.9 ± 1.590.2 ± 0.60.24 ± 0.020.001
Union 21160.38 ± 0.0333.5 ± 2.743.5 ± 3.493.1 ± 0.70.29 ± 0.030.001
Union 3710.40 ± 0.0137.5 ± 1.243.6 ± 1.792.3 ± 0.50.32 ± 0.010.001
Intersection170.35 ± 0.0332.0 ± 3.138.7 ± 2.292.0 ± 0.50.26 ± 0.030.001
10%Union940.40 ± 0.0231.7 ± 2.252.5 ± 3.195.4 ± 0.40.33 ± 0.030.001
Union 2440.43 ± 0.0239.4 ± 2.848.1 ± 1.893.3 ± 0.30.35 ± 0.030.001
Union 3250.38 ± 0.0232.1 ± 2.347.8 ± 2.594.4 ± 0.40.31 ± 0.020.001
Intersection90.35 ± 0.0332.1 ± 2.538.9 ± 2.792.0 ± 0.40.26 ± 0.030.001
SVM100%-4280.26 ± 0.0220.4 ± 1.935.5 ± 2.794.1 ± 0.60.18 ± 0.020.001
50%Union3450.29 ± 0.0427.1 ± 4.832.1 ± 4.291.0 ± 0.90.19 ± 0.050.001
Union 22460.31 ± 0.0228.0 ± 2.436.0 ± 2.692.1 ± 0.60.22 ± 0.030.001
Union 31910.34 ± 0.0228.9 ± 2.241.0 ± 2.893.4 ± 0.50.25 ± 0.030.001
Intersection740.35 ± 0.0332.3 ± 3.138.6 ± 2.891.9 ± 0.60.26 ± 0.030.001
25%Union2240.33 ± 0.0231.3 ± 2.735.7 ± 2.391.1 ± 0.50.24 ± 0.030.001
Union 21160.36 ± 0.0330.2 ± 2.246.2 ± 3.994.4 ± 0.70.29 ± 0.030.001
Union 3710.38 ± 0.0234.6 ± 2.741.9 ± 1.892.4 ± 0.30.29 ± 0.020.001
Intersection170.33 ± 0.0228.3 ± 1.939.9 ± 3.193.2 ± 0.60.25 ± 0.030.001
10%Union940.39 ± 0.0332.1 ± 2.948.4 ± 4.294.6 ± 0.50.31 ± 0.040.001
Union 2440.41 ± 0.0233.3 ± 2.351.9 ± 3.295.1 ± 0.60.33 ± 0.030.001
Union 3250.42 ± 0.0437.5 ± 4.348.2 ± 4.293.6 ± 0.70.34 ± 0.050.001
Intersection90.33 ± 0.0230.9 ± 2.335.9 ± 1.491.3 ± 0.40.24 ± 0.020.001
Table
Table A3. Performance of the four classification models on detection fall using Basic data set; basic data set exists of the 25 Foot Walk Test + 2-minute walk test + Expanded Disability Status Scale; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A3. Performance of the four classification models on detection fall using Basic data set; basic data set exists of the 25 Foot Walk Test + 2-minute walk test + Expanded Disability Status Scale; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-110.47 ± 0.0157.6 ± 0.939.4 ± 0.485.7 ± 0.20.36 ± 0.010.001
50%Union110.47 ± 0.0157.6 ± 0.939.4 ± 0.485.7 ± 0.20.36 ± 0.010.001
Union 280.43 ± 0.0149.2 ± 0.838.1 ± 0.587.1 ± 0.10.32 ± 0.010.001
Union 340.43 ± 0.0146.9 ± 0.739.1 ± 0.488.2 ± 0.10.32 ± 0.010.001
Intersection10.05 ± 0.002.9 ± 0.042.0 ± 1.299.4 ± 0.00.04 ± 0.000.001
25%Union70.46 ± 0.0057.1 ± 0.537.9 ± 0.384.9 ± 0.10.35 ± 0.000.001
Union 240.45 ± 0.0048.4 ± 0.642.4 ± 0.489.4 ± 0.10.36 ± 0.010.001
Union 310.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Intersection-------
10%Union20.38 ± 0.0033.1 ± 0.044.7 ± 0.393.4 ± 0.10.30 ± 0.000.001
Union 210.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Union 310.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Intersection-------
DT100%-110.27 ± 0.0320.4 ± 3.542.0 ± 1.895.5 ± 0.70.20 ± 0.030.001
50%Union110.27 ± 0.0320.4 ± 3.542.0 ± 1.895.5 ± 0.70.20 ± 0.030.001
Union 280.29 ± 0.0222.3 ± 2.141.7 ± 4.094.9 ± 0.70.21 ± 0.030.001
Union 340.29 ± 0.0421.7 ± 3.244.5 ± 4.795.6 ± 0.80.22 ± 0.040.001
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.000.995
25%Union70.23 ± 0.0315.2 ± 2.644.0 ± 5.796.9 ± 0.50.17 ± 0.030.001
Union 240.18 ± 0.0412.2 ± 3.138.4 ± 4.196.9 ± 0.50.13 ± 0.030.001
Union 310.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Intersection-------
10%Union20.20 ± 0.0313.5 ± 2.339.8 ± 3.696.7 ± 0.60.14 ± 0.030.001
Union 210.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Union 310.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Intersection-------
kNN100%-110.26 ± 0.0225.2 ± 2.126.0 ± 2.088.4 ± 0.50.14 ± 0.020.001
50%Union110.26 ± 0.0225.2 ± 2.126.0 ± 2.088.4 ± 0.50.14 ± 0.020.001
Union 280.25 ± 0.0125.8 ± 1.825.1 ± 1.387.6 ± 0.70.13 ± 0.020.001
Union 340.33 ± 0.0131.9 ± 1.334.2 ± 1.590.1 ± 0.50.23 ± 0.020.001
Intersection10.07 ± 0.084.7 ± 5.221.1 ± 14.998.4 ± 1.70.04 ± 0.050.001
25%Union70.28 ± 0.0223.4 ± 1.534.5 ± 2.492.8 ± 0.60.19 ± 0.020.001
Union 240.24 ± 0.0218.3 ± 1.937.1 ± 3.695.0 ± 0.60.17 ± 0.020.001
Union 310.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Intersection-------
10%Union20.26 ± 0.0321.0 ± 3.634.8 ± 7.493.1 ± 2.80.17 ± 0.040.001
Union 210.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Union 310.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Intersection-------
SVM100%-110.26 ± 0.0224.8 ± 2.327.3 ± 2.289.3 ± 0.60.15 ± 0.020.001
50%Union110.26 ± 0.0224.8 ± 2.327.3 ± 2.289.3 ± 0.60.15 ± 0.020.001
Union 280.21 ± 0.0215.5 ± 1.534.2 ± 3.495.2 ± 0.40.14 ± 0.020.001
Union 340.29 ± 0.0322.7 ± 3.041.9 ± 3.594.9 ± 0.40.22 ± 0.030.001
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
25%Union70.26 ± 0.0219.8 ± 2.236.9 ± 1.894.6 ± 0.40.18 ± 0.020.001
Union 240.20 ± 0.0214.0 ± 1.836.8 ± 3.596.1 ± 0.60.14 ± 0.020.001
Union 310.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Intersection-------
10%Union20.17 ± 0.0211.7 ± 1.934.4 ± 3.196.4 ± 0.60.11 ± 0.020.001
Union 210.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Union 310.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Intersection-------
Table
Table A4. Performance of the four classification models on detection fall using Expanded Disability Status Scale data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A4. Performance of the four classification models on detection fall using Expanded Disability Status Scale data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-90.48 ± 0.0064.6 ± 0.338.2 ± 0.283.1 ± 0.10.37 ± 0.000.001
50%Union90.48 ± 0.0064.6 ± 0.338.2 ± 0.283.1 ± 0.10.37 ± 0.000.001
Union 260.45 ± 0.0056.8 ± 0.537.2 ± 0.584.5 ± 0.30.34 ± 0.010.001
Union 340.44 ± 0.0150.5 ± 1.239.3 ± 0.887.4 ± 0.40.34 ± 0.010.001
Intersection10.05 ± 0.002.9 ± 0.042.0 ± 1.299.4 ± 0.00.04 ± 0.000.001
25%Union40.44 ± 0.0043.5 ± 0.343.6 ± 0.390.9 ± 0.00.35 ± 0.000.001
Union 230.43 ± 0.0042.4 ± 0.043.3 ± 0.191.0 ± 0.00.34 ± 0.000.001
Union 310.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Intersection-------
10%Union20.38 ± 0.0033.1 ± 0.044.7 ± 0.393.4 ± 0.10.30 ± 0.000.001
Union 210.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Union 310.37 ± 0.0032.0 ± 0.042.6 ± 0.093.1 ± 0.00.28 ± 0.000.001
Intersection-------
DT100%-90.24 ± 0.0316.5 ± 2.141.4 ± 5.096.2 ± 0.60.17 ± 0.030.001
50%Union90.24 ± 0.0316.5 ± 2.141.4 ± 5.096.2 ± 0.60.17 ± 0.030.001
Union 260.25 ± 0.0217.6 ± 2.143.8 ± 3.096.3 ± 0.50.19 ± 0.020.001
Union 340.24 ± 0.0214.3 ± 1.938.7 ± 2.296.3 ± 0.70.14 ± 0.020.001
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.000.995
25%Union40.20 ± 0.0313.3 ± 2.543.3 ± 4.697.2 ± 0.50.15 ± 0.030.001
Union 230.20 ± 0.0413.1 ± 3.240.6 ± 3.897.0 ± 0.40.14 ± 0.040.001
Union 310.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Intersection-------
10%Union20.20 ± 0.0313.5 ± 2.339.8 ± 3.696.7 ± 0.60.14 ± 0.030.001
Union 210.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Union 310.16 ± 0.0310.0 ± 2.142.5 ± 4.397.8 ± 0.40.11 ± 0.030.001
Intersection-------
kNN100%-90.28 ± 0.0227.5 ± 1.629.2 ± 1.689.2 ± 0.50.17 ± 0.020.001
50%Union90.28 ± 0.0227.5 ± 1.629.2 ± 1.689.2 ± 0.50.17 ± 0.020.001
Union 260.28 ± 0.0324.8 ± 3.232.6 ± 3.991.7 ± 0.90.18 ± 0.040.001
Union 340.28 ± 0.0323.1 ± 3.136.4 ± 3.593.5 ± 0.70.20 ± 0.030.001
Intersection10.07 ± 0.084.7 ± 5.221.1 ± 14.998.4 ± 1.70.04 ± 0.050.001
25%Union40.23 ± 0.0217.3 ± 1.936.5 ± 2.795.1 ± 0.30.16 ± 0.020.001
Union 230.25 ± 0.0219.9 ± 1.733.8 ± 3.893.6 ± 0.90.16 ± 0.030.001
Union 310.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Intersection-------
10%Union20.26 ± 0.0321.0 ± 3.634.8 ± 7.493.1 ± 2.80.17 ± 0.040.001
Union 210.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Union 310.24 ± 0.0621.7 ± 1.032.1 ± 7.391.7 ± 4.90.15 ± 0.040.001
Intersection-------
SVM100%-90.27 ± 0.0224.1 ± 2.030.5 ± 2.891.1 ± 0.60.17 ± 0.030.001
50%Union90.27 ± 0.0224.1 ± 2.030.5 ± 2.891.1 ± 0.60.17 ± 0.030.001
Union 260.24 ± 0.0218.9 ± 2.033.7 ± 2.794.0 ± 0.50.16 ± 0.020.001
Union 340.26 ± 0.0319.4 ± 2.239.5 ± 4.095.2 ± 0.40.19 ± 0.030.001
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
25%Union40.20 ± 0.0214.5 ± 1.332.7 ± 2.295.2 ± 0.40.13 ± 0.020.001
Union 230.18 ± 0.0312.7 ± 2.334.3 ± 3.896.1 ± 0.50.12 ± 0.030.001
Union 310.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Intersection-------
10%Union20.17 ± 0.0211.7 ± 1.934.4 ± 3.196.4 ± 0.60.11 ± 0.020.001
Union 210.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Union 310.08 ± 0.034.4 ± 1.737.8 ± 4.198.9 ± 0.30.05 ± 0.020.001
Intersection-------
Table
Table A5. Performance of the four classification models on detection fall using GAITRite System normal walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A5. Performance of the four classification models on detection fall using GAITRite System normal walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-820.35 ± 0.0128.7 ± 0.943.3 ± 0.994.0 ± 0.20.26 ± 0.010.001
50%Union750.34 ± 0.0127.7 ± 0.842.4 ± 1.093.9 ± 0.20.25 ± 0.010.001
Union 2520.35 ± 0.0129.9 ± 0.843.2 ± 1.093.6 ± 0.20.27 ± 0.010.001
Union 3300.36 ± 0.0131.3 ± 0.642.6 ± 0.793.2 ± 0.20.28 ± 0.010.001
Intersection70.42 ± 0.0045.4 ± 0.638.4 ± 0.388.3 ± 0.10.31 ± 0.000.001
25%Union430.35 ± 0.0129.1 ± 0.742.5 ± 0.793.7 ± 0.10.26 ± 0.010.001
Union 2250.38 ± 0.0135.1 ± 0.641.6 ± 0.692.1 ± 0.20.29 ± 0.010.001
Union 3130.40 ± 0.0141.0 ± 0.738.2 ± 0.989.3 ± 0.30.29 ± 0.010.001
Intersection30.39 ± 0.0037.7 ± 0.440.9 ± 0.491.2 ± 0.10.30 ± 0.000.001
10%Union200.40 ± 0.0140.7 ± 1.539.9 ± 1.190.1 ± 0.20.31 ± 0.010.001
Union 280.42 ± 0.0046.8 ± 0.638.0 ± 0.487.7 ± 0.20.31 ± 0.010.001
Union 340.41 ± 0.0142.3 ± 0.640.2 ± 0.589.9 ± 0.10.31 ± 0.010.001
Intersection-------
DT100%-820.24 ± 0.0321.3 ± 3.228.1 ± 2.991.3 ± 0.60.14 ± 0.030.001
50%Union750.25 ± 0.0321.4 ± 3.029.6 ± 2.491.9 ± 0.70.15 ± 0.030.001
Union 2520.22 ± 0.0217.3 ± 2.629.3 ± 2.493.3 ± 1.00.13 ± 0 020.001
Union 3300.22 ± 0.0217.1 ± 2.531.2 ± 1.893.9 ± 0.80.14 ± 0.020.001
Intersection70.26 ± 0.0318.8 ± 2.743.7 ± 3.796.1 ± 0.60.19 ± 0.030.001
25%Union430.23 ± 0.0320.4 ± 2.227.3 ± 4.091.1 ± 1.30.13 ± 0.030.001
Union 2250.22 ± 0.0217.0 ± 1.831.9 ± 3.694.1 ± 0.90.14 ± 0.020.001
Union 3130.23 ± 0.0416.4 ± 3.242.1 ± 4.696.4 ± 0.60.17 ± 0.040.001
Intersection30.25 ± 0.0318.2 ± 2.840.1 ± 4.695.5 ± 1.00.18 ± 0.030.001
10%Union200.21 ± 0.0217.1 ± 2.528.4 ± 2.893.0 ± 1.10.12 ± 0.020.001
Union 280.25 ± 0.0320.1 ± 3.233.0 ± 4.493.4 ± 1.00.16 ± 0.040.001
Union 340.25 ± 0.0319.2 ± 3.235.2 ± 2.694.3 ± 0.70.17 ± 0.030.001
Intersection-------
kNN100%-820.20 ± 0.0119.6 ± 0.920.9 ± 1.388.0 ± 0.70.08 ± 0.010.015
50%Union750.20 ± 0.0219.5 ± 1.821.5 ± 1.588.6 ± 0.40.08 ± 0.020.010
Union 2520.26 ± 0.0124.3 ± 1.527.1 ± 1.589.4 ± 0.70.14 ± 0.020.001
Union 3300.24 ± 0.0223.6 ± 1.824.8 ± 1.988.5 ± 0.50.12 ± 0.020.001
Intersection70.26 ± 0.0225.1 ± 1.626.3 ± 1.788.7 ± 0.30.14 ± 0.020.002
25%Union430.20 ± 0.0119.3 ± 1.221.5 ± 1.088.7 ± 0.40.08 ± 0.010.005
Union 2250.25 ± 0.0224.7 ± 1.725.4 ± 1.888.3 ± 0.80.13 ± 0.020.001
Union 3130.22 ± 0.0220.7 ± 1.922.8 ± 1.488.8 ± 0.50.10 ± 0.020.001
Intersection30.27 ± 0.0221.6 ± 1.836.3 ± 2.593.9 ± 0.60.19 ± 0.020.001
10%Union200.25 ± 0.0219.3 ± 1.434.0 ± 2.093.9 ± 0.40.16 ± 0.020.001
Union 280.25 ± 0.0224.0 ± 2.425.2 ± 2.288.6 ± 0.60.13 ± 0.030.001
Union 340.30 ± 0.0224.2 ± 1.939.5 ± 2.494.1 ± 0.40.22 ± 0.020.001
Intersection-------
SVM100%-820.21 ± 0.0313.9 ± 2.441.5 ± 5.296.9 ± 0.30.15 ± 0.030.001
50%Union750.24 ± 0.0219.0 ± 1.833.3 ± 1.693.9 ± 0.40.16 ± 0.020.001
Union 2520.26 ± 0.0318.5 ± 2.343.5 ± 4.296.1 ± 0.40.19 ± 0.030.001
Union 3300.18 ± 0.0213.2 ± 2.126.9 ± 3.394.2 ± 0.60.09 ± 0.020.001
Intersection70.20 ± 0.0314.8 ± 2.233.6 ± 5.295.3 ± 0.60.13 ± 0.030.001
25%Union430.26 ± 0.0418.5 ± 2.942.4 ± 4.996.0 ± 0.40.19 ± 0.040.001
Union 2250.18 ± 0.0211.5 ± 1.347.2 ± 3.697.9 ± 0.10.14 ± 0.020.001
Union 3130.17 ± 0.0115.7 ± 1.418.9 ± 1.289.1 ± 0.80.05 ± 0.010.093
Intersection30.16 ± 0.029.8 ± 1.545.2 ± 5.398.1 ± 0.30.12 ± 0.020.001
10%Union200.15 ± 0.039.5 ± 1.939.5 ± 6.697.7 ± 0.40.10 ± 0.030.001
Union 280.20 ± 0.0216.4 ± 1.825.3 ± 2.192.2 ± 0.80.10 ± 0.020.002
Union 340.19 ± 0.0211.9 ± 1.644.9 ± 5.197.7 ± 0.20.14 ± 0.020.001
Intersection-------
Table
Table A6. Performance of the four classification models on detection fall using GAITRite System dual task walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A6. Performance of the four classification models on detection fall using GAITRite System dual task walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-820.33 ± 0.0133.2 ± 1.032.2 ± 1.089.1 ± 0.40.22 ± 0.010.001
50%Union670.34 ± 0.0134.4 ± 0.833.5 ± 0.789.3 ± 0.20.23 ± 0.010.001
Union 2480.34 ± 0.0035.5 ± 0.632.8 ± 0.488.6 ± 0.20.23 ± 0.000.001
Union 3330.33 ± 0.0135.1 ± 0.932.0 ± 0.588.4 ± 0.20.23 ± 0.010.001
Intersection160.37 ± 0.0046.1 ± 0.730.7 ± 0.383.8 ± 0.20.25 ± 0.000.001
25%Union440.34 ± 0.0136.2 ± 0.831.9 ± 0.588.0 ± 0.20.23 ± 0.010.001
Union 2250.36 ± 0.0041.9 ± 0.431.7 ± 0.385.9 ± 0.20.24 ± 0.000.001
Union 3110.38 ± 0.0044.5 ± 0.732.7 ± 0.485.7 ± 0.20.26 ± 0.010.001
Intersection40.38 ± 0.0141.0 ± 0.835.7 ± 0.488.5 ± 0.10.28 ± 0.010.001
10%Union200.36 ± 0.0141.0 ± 0.732.0 ± 0.686.4 ± 0.20.24 ± 0 010.001
Union 2100.38 ± 0.0043.8 ± 0.534.0 ± 0.486.7 ± 0.20.27 ± 0.000.001
Union 320.31 ± 0.0026.0 ± 0.539.1 ± 0.593.7 ± 0.10.23 ± 0.010.001
Intersection-------
DT100%-820.24 ± 0.0519.8 ± 4.929.7 ± 4.292.8 ± 0.80.15 ± 0.050.001
50%Union670.22 ± 0.0317.1 ± 2.929.9 ± 3.793.7 ± 1.00.13 ± 0.030.001
Union 2480.20 ± 0.0318.0 ± 2.823.7 ± 3.890.8 ± 1.70.10 ± 0.030.001
Union 3330.22 ± 0.0218.0 ± 2.229.8 ± 3.393.3 ± 1.10.14 ± 0.020.001
Intersection160.18 ± 0.0214.0 ± 2.224.9 ± 2.993.4 ± 0 80.09 ± 0.020.001
25%Union440.18 ± 0.0412.8 ± 3.131.7 ± 4.995.6 ± 1.30.11 ± 0.030.001
Union 2250.23 ± 0.0219.9 ± 2.227.4 ± 2.691.7 ± 1.00.13 ± 0.020.001
Union 3110.21 ± 0.0217.1 ± 2.227.3 ± 3.192.9 ± 0.90.12 ± 0.020.001
Intersection40.23 ± 0.0318.4 ± 2.529.7 ± 3.693.2 ± 0.70.14 ± 0.030.001
10%Union200.24 ± 0.0420.1 ± 4.029.2 ± 4.992.4 ± 1.00.14 ± 0.040.001
Union 2100.23 ± 0.0218.7 ± 2.129.0 ± 3.092.8 ± 1.00.13 ± 0.020.001
Union 320.22 ± 0.0317.8 ± 2.330.7 ± 3.293.7 ± 0.70.14 ± 0.030.001
Intersection-------
kNN100%-820.20 ± 0.0313.2 ± 2.142.1 ± 4.197.2 ± 0.20.15 ± 0.030.001
50%Union670.21 ± 0.0320.9 ± 2.921.9 ± 2.588.4 ± 0.40.09 ± 0.030.001
Union 2480.25 ± 0.0224.1 ± 2.626.4 ± 1.989.5 ± 0.70.14 ± 0.020.001
Union 3330.26 ± 0.0225.4 ± 2.026.4 ± 1.688.9 ± 0.50.15 ± 0 020.001
Intersection160.21 ± 0.0114.5 ± 1.235.6 ± 2.395.9 ± 0.40.14 ± 0.010.001
25%Union440.25 ± 0.0218.5 ± 1.740.4 ± 3.495.7 ± 0.50.18 ± 0.020.001
Union 2250.22 ± 0.0220.7 ± 2.222.9 ± 1.989.1 ± 0.60.10 ± 0 020.001
Union 3110.25 ± 0.0218.8 ± 1.535.6 ± 2.794.7 ± 0.40.17 ± 0.020.001
Intersection40.18 ± 0.0214.3 ± 2.225.8 ± 2.993.6 ± 0.50.10 ± 0.030.001
10%Union200.26 ± 0.0324.8 ± 3.228.1 ± 2.590.1 ± 0.40.16 ± 0.030.001
Union 2100.28 ± 0.0228.6 ± 2.027.1 ± 1.288.0 ± 0.40.16 ± 0.020.001
Union 320.27 ± 0.0225.9 ± 2.527.3 ± 2.289.2 ± 0.50.15 ± 0.030.001
Intersection-------
SVM100%-820.20 ± 0.0216.3 ± 2.226.0 ± 2.492.8 ± 0.60.11 ± 0.020.001
50%Union670.22 ± 0.0217.7 ± 2.528.9 ± 2.593.2 ± 0.50.13 ± 0.020.001
Union 2480.23 ± 0.0316.0 ± 2.040.4 ± 4.696.3 ± 0.30.16 ± 0.030.001
Union 3330.19 ± 0.0212.1 ± 1.542.5 ± 6.997.4 ± 0.60.14 ± 0.030.001
Intersection160.20 ± 0.0315.1 ± 2.128.0 ± 4.193.9 ± 0.50.11 ± 0.030.008
25%Union440.24 ± 0.0217.4 ± 1.837.3 ± 2.395.4 ± 0.40.17 ± 0.020.001
Union 2250.21 ± 0.0214.5 ± 1.936.3 ± 3.496.0 ± 0.50.14 ± 0.020.001
Union 3110.16 ± 0.0115.1 ± 1.317.7 ± 1.089.0 ± 0.80.04 ± 0.010.061
Intersection40.13 ± 0.017.8 ± 1.036.1 ± 3.497.8 ± 0.20.08 ± 0.010.001
10%Union200.24 ± 0.0216.8 ± 1.741.0 ± 2.796.2 ± 0.50.17 ± 0.020.001
Union 2100.21 ± 0.0214.6 ± 1.537.3 ± 2.296.2 ± 0.30.14 ± 0.020.001
Union 320.14 ± 0.028.4 ± 1.547.6 ± 4.198.6 ± 0.20.11 ± 0.020.001
Intersection-------
Table
Table A7. Performance of the four classification models on detection fall using Mobility Lab Gait normal walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A7. Performance of the four classification models on detection fall using Mobility Lab Gait normal walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-840.41 ± 0.0043.1 ± 0.738.2 ± 0.388.6 ± 0.20.30 ± 0.000.001
50%Union720.41 ± 0.0143.5 ± 0.938.7 ± 0.888.8 ± 0.20.31 ± 0.010.001
Union 2480.40 ± 0.0141.0 ± 0.938.3 ± 0.689.2 ± 0.20.29 ± 0.010.001
Union 3360.39 ± 0.0140.4 ± 0.838.7 ± 0.489.6 ± 0.10.29 ± 0.010.001
Intersection120.41 ± 0.0044.3 ± 0.438.8 ± 0.688.6 ± 0.20.31 ± 0.010.001
25%Union500.40 ± 0.0142.4 ± 0.938.5 ± 0.688.9 ± 0.30.30 ± 0.010.001
Union 2210.41 ± 0.0142.4 ± 1.039.8 ± 0.789.5 ± 0.20.31 ± 0.010.001
Union 3110.42 ± 0.0044.8 ± 0.439.5 ± 0.388.8 ± 0.10.32 ± 0.000.001
Intersection20.36 ± 0.0132.2 ± 1.139.6 ± 1.192.0 ± 0.10.26 ± 0.010.001
10%Union220.42 ± 0.0146.0 ± 0.738.9 ± 0.888.2 ± 0.30.32 ± 0.010.001
Union 280.43 ± 0.0142.9 ± 0.643.1 ± 0.790.8 ± 0.20.34 ± 0.010.001
Union 320.37 ± 0.0133.6 ± 0.842.4 ± 0.692.6 ± 0.10.29 ± 0.010.001
Intersection-------
DT100%-840.22 ± 0.0417.1 ± 3.329.7 ± 4.693.4 ± 1.20.13 ± 0.040.001
50%Union720.25 ± 0.0319.2 ± 2.334.4 ± 3.494.0 ± 0.60.16 ± 0.030.001
Union 2480.23 ± 0.0517.3 ± 5.236.8 ± 3.095.2 ± 1.00.16 ± 0.040.001
Union 3360.24 ± 0.0321.4 ± 3.026.8 ± 3.390.4 ± 1.30.13 ± 0.030.001
Intersection120.23 ± 0.0415.6 ± 3.642.5 ± 6.196.6 ± 0.60.16 ± 0.040.001
25%Union500.19 ± 0.0513.5 ± 4.436.5 ± 4.796.2 ± 1.20.13 ± 0.040.001
Union 2210.23 ± 0.0316.8 ± 3.339.7 ± 2.695.8 ± 0.80.17 ± 0.030.001
Union 3110.26 ± 0.0319.6 ± 2.937.2 ± 3.294.6 ± 0.50.18 ± 0.030.001
Intersection20.23 ± 0.0216.0 ± 1.343.1 ± 3.496.5 ± 0.60.17 ± 0.020.001
10%Union220.23 ± 0.0717.4 ± 7.137.6 ± 5.595.4 ± 1.50.16 ± 0.060.001
Union 280.24 ± 0.0316.4 ± 2.642.5 ± 3.996.4 ± 0.50.17 ± 0.030.001
Union 320.24 ± 0.0317.0 ± 2.243.1 ± 3.596.3 ± 0.50.18 ± 0.020.001
Intersection-------
kNN100%-840.22 ± 0.0221.6 ± 2.822.8 ± 2.188.2 ± 0.60.10 ± 0.030.001
50%Union720.25 ± 0.0325.0 ± 3.024.5 ± 2.587.4 ± 0.70.12 ± 0.030.006
Union 2480.26 ± 0.0126.3 ± 1.726.5 ± 1.488.1 ± 0.40.14 ± 0.020.001
Union 3360.26 ± 0 0125.2 ± 1.226.3 ± 1.488.5 ± 0.70.14 ± 0.010.023
Intersection120.22 ± 0.0221.3 ± 2.423.0 ± 2.188.4 ± 0.50.10 ± 0.020.005
25%Union500.24 ± 0.0222.0 ± 1.827.1 ± 1.490.4 ± 0.40.13 ± 0.020.002
Union 2210.21 ± 0.0120.5 ± 1.321.9 ± 1.388.1 ± 0.60.09 ± 0.010.021
Union 3110.21 ± 0.0215.6 ± 1.932.2 ± 3.994.6 ± 0.50.13 ± 0.030.001
Intersection20.21 ± 0.0221.0 ± 2.521.5 ± 2.087.5 ± 0.70.09 ± 0.020.019
10%Union220.31 ± 0.0230.1 ± 1.932.2 ± 2.189.6 ± 0.80.20 ± 0.020.001
Union 280.26 ± 0.0325.1 ± 3.128.1 ± 2.589.5 ± 0.60.15 ± 0.030.001
Union 320.21 ± 0.0317.1 ± 2.628.9 ± 3.593.2 ± 0.60.12 ± 0.030.001
Intersection-------
SVM100%-840.17 ± 0.0211.8 ± 1.529.2 ± 2.295.4 ± 0.40.09 ± 0.020.001
50%Union720.20 ± 0.0217.3 ± 1.624.2 ± 2.391.1 ± 0.80.10 ± 0.020.001
Union 2480.27 ± 0.0121.1 ± 1.336.9 ± 2.594.1 ± 0.60.18 ± 0.020.001
Union 3360.22 ± 0.0215.5 ± 1.936.4 ± 3.595.6 ± 0.30.15 ± 0.030.001
Intersection120.22 ± 0.0218.7 ± 2.526.2 ± 2.891.4 ± 1.00.11 ± 0.030.001
25%Union500.23 ± 0.0420.8 ± 3.427.1 ± 3.690.9 ± 0.50.13 ± 0.040.001
Union 2210.21 ± 0.0217.9 ± 2.324.6 ± 1.791.1 ± 0.50.10 ± 0.020.002
Union 3110.27 ± 0.0221.3 ± 1.936.9 ± 2.394.1 ± 0.60.19 ± 0.020.001
Intersection20.19 ± 0.0211.8 ± 1.655.5 ± 4.398.5 ± 0.30.15 ± 0.020.001
10%Union220.28 ± 0.0220.1 ± 1.946.9 ± 4.196.3 ± 0.60.22 ± 0.020.001
Union 280.17 ± 0.0311.2 ± 2.236.2 ± 4.096.8 ± 0.30.11 ± 0.030.001
Union 320.06 ± 0.023.1 ± 0.944.5 ± 7.199.3 ± 0.30.04 ± 0.010.001
Intersection-------
Table
Table A8. Performance of the four classification models on detection fall using Mobility Lab Gait dual task walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A8. Performance of the four classification models on detection fall using Mobility Lab Gait dual task walking data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-840.36 ± 0.0144.9 ± 0.930.2 ± 0.583.6 ± 0.30.24 ± 0.010.001
50%Union610.37 ± 0.0045.5 ± 0.531.2 ± 0.484.1 ± 0.10.25 ± 0.000.001
Union 2450.35 ± 0.0043.1 ± 0.429.9 ± 0.483.9 ± 0.30.23 ± 0.000.001
Union 3390.36 ± 0.0143.7 ± 0.730.1 ± 0.583.9 ± 0.30.23 ± 0.010.001
Intersection230.36 ± 0.0042.5 ± 0.631.5 ± 0.485.3 ± 0.20.24 ± 0.010.001
25%Union440.36 ± 0.0044.2 ± 0.530.9 ± 0.584.3 ± 0.30.24 ± 0.010.001
Union 2230.38 ± 0.0044.9 ± 0.532.8 ± 0.385.4 ± 0.20.26 ± 0.000.001
Union 3130.39 ± 0.0145.3 ± 0.633.7 ± 0.585.9 ± 0.20.27 ± 0.010.001
Intersection40.39 ± 0.0040.7 ± 0.637.4 ± 0.589.2 ± 0.10.29 ± 0.010.001
10%Union210.37 ± 0.0143.6 ± 0.831.6 ± 0.585.0 ± 0.20.25 ± 0.010.001
Union 270.37 ± 0.0042.1 ± 0.533.2 ± 0.486.6 ± 0.10.26 ± 0.000.001
Union 330.38 ± 0.0136.0 ± 0.739.4 ± 0.691.2 ± 0.10.28 ± 0.010.001
Intersection10.20 ± 0.0112.7 ± 0.742.8 ± 2.497.3 ± 0.20.14 ± 0.010.001
DT100%-840.21 ± 0.0316.9 ± 2.426.9 ± 4.192.6 ± 1.00.11 ± 0.030.001
50%Union610.18 ± 0.0413.9 ± 3.526.9 ± 4.694.0 ± 1.30.10 ± 0.040.001
Union 2450.20 ± 0.0315.6 ± 2.830.3 ± 3.794.3 ± 1.00.12 ± 0.030.001
Union 3390.21 ± 0.0216.5 ± 2.429.5 ± 4.293.6 ± 1.50.12 ± 0.030.001
Intersection230.21 ± 0.0516.8 ± 5.228.6 ± 5.493.3 ± 1.70.12 ± 0.050.001
25%Union440.21 ± 0.0320.3 ± 4.022.3 ± 2.988.8 ± 1.40.09 ± 0.030.001
Union 2230.23 ± 0.0317.4 ± 2.834.0 ± 5.994.5 ± 1.20.15 ± 0.040.001
Union 3130.20 ± 0.0215.0 ± 2.032.2 ± 3.195.0 ± 0.70.13 ± 0.020.001
Intersection40.21 ± 0.0415.9 ± 3.331.6 ± 5.294.5 ± 0.90.13 ± 0.040.001
10%Union210.22 ± 0.0317.8 ± 3.828.1 ± 2.792.8 ± 1.10.12 ± 0.030.001
Union 270.25 ± 0.0318.2 ± 2.838.0 ± 4.595.3 ± 0.70.17 ± 0.030.001
Union 330.28 ± 0.0322.4 ± 3.036.5 ± 3.593.8 ± 0.40.19 ± 0.030.001
Intersection10.18 ± 0.0311.8 ± 2.635.7 ± 5.296.6 ± 0.60.12 ± 0.030.001
kNN100%-840.18 ± 0.0217.4 ± 2.018.8 ± 1.888.1 ± 0.90.06 ± 0.020.045
50%Union610.20 ± 0.0218.3 ± 2.821.6 ± 2.089.5 ± 0.70.08 ± 0.020.009
Union 2450.26 ± 0.0225.3 ± 2.226.4 ± 2.088.8 ± 0.90.14 ± 0.020.001
Union 3390.18 ± 0.0117.4 ± 1.419.5 ± 1.488.6 ± 0.70.06 ± 0.010.007
Intersection230.22 ± 0.0216.5 ± 1.833.4 ± 4.094.7 ± 0.70.14 ± 0.030.001
25%Union440.25 ± 0.0224.3 ± 2.225.9 ± 1.789.0 ± 0.40.14 ± 0.020.001
Union 2230.21 ± 0.0219.1 ± 2.522.9 ± 2.389.8 ± 0.40.10 ± 0.030.006
Union 3130.20 ± 0.0218.6 ± 1.522.1 ± 1.889.6 ± 0.50.09 ± 0.020.028
Intersection40.21 ± 0.0219.8 ± 1.923.5 ± 2.089.8 ± 0.60.10 ± 0.020.003
10%Union210.24 ± 0.0218.5 ± 1.635.6 ± 3.594.7 ± 0.50.16 ± 0.020.001
Union 270.22 ± 0.0120.4 ± 1.522.9 ± 1.589.1 ± 0.50.10 ± 0.020.038
Union 330.24 ± 0.0223.1 ± 2.325.4 ± 2.589.2 ± 0.90.13 ± 0.030.001
Intersection10.19 ± 0 0316.8 ± 2.622.4 ± 3.590.8 ± 0.70.08 ± 0.030.005
SVM100%-840.19 ± 0.0215.0 ± 1.824.9 ± 3.192.8 ± 1.00.09 ± 0.020.001
50%Union610.25 ± 0.0221.9 ± 2.328.6 ± 1.891.3 ± 0 60.15 ± 0.020.001
Union 2450.22 ± 0.0318.5 ± 2.527.4 ± 3.392.2 ± 0.60.12 ± 0.030.001
Union 3390.21 ± 0.0315.9 ± 2.329.2 ± 3.193.9 ± 0.40.12 ± 0.030.002
Intersection230.23 ± 0.0320.4 ± 2.525.6 ± 2.690.6 ± 0.40.12 ± 0.030.001
25%Union440.26 ± 0.0222.5 ± 2.029.7 ± 2.991.5 ± 0.60.16 ± 0.030.001
Union 2230.19 ± 0.0213.5 ± 1.730.2 ± 3.895.0 ± 0.40.11 ± 0.020.001
Union 3130.21 ± 0.0217.3 ± 2.428.1 ± 1.993.0 ± 0.80.12 ± 0.020.001
Intersection40.21 ± 0.0218.9 ± 1.924.1 ± 2.290.6 ± 0.80.10 ± 0.020.002
10%Union210.22 ± 0.0317.2 ± 2.331.5 ± 3.694.1 ± 0.50.14 ± 0.030.001
Union 270.19 ± 0.0312.7 ± 2.035.4 ± 4.696.3 ± 0.50.12 ± 0.030.001
Union 330.20 ± 0.0314.6 ± 2.730.0 ± 3.494.6 ± 0.50.12 ± 0.030.015
Intersection10.03 ± 0 021.6 ± 1.137.0 ± 16.999.6 ± 0.10.02 ± 0.020.024
Table
Table A9. Performance of the four classification models on detection fall using Mobility Lab Romberg stance with open eyes data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A9. Performance of the four classification models on detection fall using Mobility Lab Romberg stance with open eyes data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-320.26 ± 0.0120.3 ± 0.836.6 ± 1.894.4 ± 0.20.18 ± 0.010.001
50%Union310.26 ± 0.0120.2 ± 0.936.5 ± 1.694.4 ± 0.20.18 ± 0.010.001
Union 2220.26 ± 0.0219.3 ± 1.338.8 ± 2.195.2 ± 0.20.18 ± 0.020.001
Union 3110.22 ± 0.0115.7 ± 1.038.5 ± 1.696.0 ± 0.20.16 ± 0.010.001
Intersection-------
25%Union210.24 ± 0.0117.5 ± 0.736.5 ± 1.395.2 ± 0.20.16 ± 0.010.001
Union 290.21 ± 0.0114.1 ± 0.738.6 ± 1.396.4 ± 0.20.14 ± 0.010.001
Union 320.18 ± 0.0111.0 ± 0.742.8 ± 1.797.7 ± 0.10.13 ± 0.010.001
Intersection-------
10%Union80.21 ± 0.0114.8 ± 0.838.9 ± 1.996.3 ± 0.20.15 ± 0.010.001
Union 230.22 ± 0.0115.5 ± 1.040.9 ± 1.296.4 ± 0.20.16 ± 0.010.001
Union 310.17 ± 0.0111.0 ± 0.641.9 ± 1.997.6 ± 0.10.12 ± 0.010.001
Intersection-------
DT100%-320.21 ± 0.0416.4 ± 3.931.3 ± 4.194.3 ± 0.90.13 ± 0.040.001
50%Union310.20 ± 0.0415.8 ± 3.229.5 ± 4.593.9 ± 1.10.12 ± 0.040.001
Union 2220.21 ± 0.0317.0 ± 2.826.9 ± 4.092.7 ± 0.60.11 ± 0.040.001
Union 3110.22 ± 0.0218.6 ± 1.827.3 ± 2.392.1 ± 0.50.12 ± 0.020.001
Intersection-------
25%Union210.22 ± 0.0315.8 ± 2.534.4 ± 3.795.2 ± 0.80.14 ± 0.030.001
Union 290.21 ± 0.0215.5 ± 1.833.5 ± 3.595.0 ± 1.00.14 ± 0.020.001
Union 320.25 ± 0.0319.4 ± 2.934.7 ± 3.194.2 ± 0.70.17 ± 0.030.001
Intersection-------
10%Union80.21 ± 0.0217.5 ± 1.927.2 ± 2.192.5 ± 1.00.12 ± 0.020.001
Union 230.23 ± 0.0217.5 ± 1.733.4 ± 4.294.4 ± 1.00.15 ± 0.020.001
Union 310.24 ± 0.0215.7 ± 1.752.2 ± 4.297.7 ± 0.50.19 ± 0.020.001
Intersection-------
kNN100%-320.23 ± 0.0124.2 ± 0.922.0 ± 1.286.4 ± 0.60.10 ± 0.010.001
50%Union310.24 ± 0.0318.1 ± 2.234.8 ± 3.294.6 ± 0.40.16 ± 0.030.001
Union 2220.24 ± 0.0123.0 ± 1.524.8 ± 1.288.9 ± 0.60.12 ± 0.010.001
Union 3110.25 ± 0.0325.0 ± 2.824.4 ± 2.387.7 ± 0.50.13 ± 0.030.001
Intersection-------
25%Union210.20 ± 0.0213.4 ± 1.438.8 ± 3.596.6 ± 0.30.14 ± 0.020.001
Union 290.27 ± 0.0226.7 ± 2.226.4 ± 2.088.2 ± 0.40.15 ± 0.020.001
Union 320.29 ± 0.0224.8 ± 1.535.0 ± 1.692.7 ± 0.30.20 ± 0.020.001
Intersection-------
10%Union80.25 ± 0.0219.1 ± 1.934.7 ± 2.894.3 ± 0.40.16 ± 0.020.001
Union 230.22 ± 0.0121.9 ± 1.122.2 ± 1.587.8 ± 0.60.10 ± 0.020.002
Union 310.23 ± 0.0314.9 ± 1.953.4 ± 4.297.9 ± 0.20.18 ± 0.030.001
Intersection-------
SVM100%-320.18 ± 0.0215.6 ± 1.620.4 ± 2.390.2 ± 0.60.07 ± 0.020.057
50%Union310.14 ± 0.029.6 ± 1.328.8 ± 3.896.2 ± 0.40.08 ± 0.020.002
Union 2220.20 ± 0.0113.5 ± 1.035.8 ± 4.496.1 ± 0.60.13 ± 0.020.001
Union 3110.19 ± 0.0313.5 ± 2.733.5 ± 3.395.8 ± 0.30.12 ± 0.030.001
Intersection-------
25%Union210.18 ± 0.0115.8 ± 1.321.8 ± 1.790.9 ± 0.90.08 ± 0.010.007
Union 290.18 ± 0.0313.8 ± 2.425.5 ± 4.393.6 ± 0.60.09 ± 0.030.001
Union 320.13 ± 0.037.6 ± 1.839.9 ± 7.298.2 ± 0.30.09 ± 0.030.005
Intersection-------
10%Union80.18 ± 0.0213.8 ± 1.727.9 ± 3.594.3 ± 0.60.10 ± 0.020.001
Union 230.09 ± 0.025.2 ± 1.527.7 ± 5.297.9 ± 0.30.05 ± 0.020.006
Union 310.09 ± 0.024.9 ± 1.366.5 ± 12.999.6 ± 0.40.07 ± 0.020.001
Intersection-------
Table
Table A10. Performance of the four classification models on detection fall using Mobility Lab Romberg stance with closed eyes (data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A10. Performance of the four classification models on detection fall using Mobility Lab Romberg stance with closed eyes (data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-320.27 ± 0.0121.5 ± 0.837.7 ± 1.294.4 ± 0.10.19 ± 0.010.001
50%Union300.28 ± 0.0121.7 ± 0.938.6 ± 1.194.6 ± 0.10.20 ± 0.010.001
Union 2220.27 ± 0.0120.1 ± 0.939.9 ± 1.495.2 ± 0.10.19 ± 0.010.001
Union 3100.26 ± 0.0118.5 ± 0.344.3 ± 2.196.3 ± 0.30.20 ± 0.010.001
Intersection20.29 ± 0.0020.9 ± 0.445.3 ± 0.496.0 ± 0.10.22 ± 0.000.001
25%Union220.30 ± 0.0123.4 ± 0.940.8 ± 1.394.6 ± 0.20.22 ± 0.010.001
Union 290.26 ± 0.0118.5 ± 0.746.3 ± 2.296.6 ± 0.30.20 ± 0.010.001
Union 310.26 ± 0.0118.5 ± 0.344.3 ± 2.196.3 ± 0.30.20 ± 0.010.001
Intersection-------
10%Union100.26 ± 0.0119.1 ± 0.741.6 ± 1.295.8 ± 0.10.19 ± 0.010.001
Union 220.12 ± 0.017.2 ± 0.746.8 ± 2.698.7 ± 0.10.09 ± 0.010.001
Union 3-------
Intersection-------
DT100%-320.21 ± 0.0418.1 ± 4.224.6 ± 2.791.3 ± 1.00.11 ± 0.030.001
50%Union300.22 ± 0.0415.4 ± 3.537.2 ± 4.895.9 ± 0.70.15 ± 0.040.001
Union 2220.22 ± 0.0417.0 ± 3.431.6 ± 3.894.2 ± 0.80.14 ± 0.030.001
Union 3100.23 ± 0.0416.5 ± 3.538.1 ± 5.395.8 ± 0.70.16 ± 0.040.001
Intersection20.20 ± 0.0215.1 ± 2.130.2 ± 2.694.5 ± 0.70.12 ± 0.020.001
25%Union220.18 ± 0.0413.8 ± 3.625.6 ± 4.193.8 ± 0.90.09 ± 0.040.001
Union 290.20 ± 0.0315.4 ± 2.728.8 ± 3.694.0 ± 0.50.12 ± 0.030.001
Union 310.23 ± 0.0416.5 ± 3.538.1 ± 5.395.8 ± 0.70.16 ± 0.040.001
Intersection-------
10%Union100.19 ± 0.0413.6 ± 3.034.2 ± 5.395.9 ± 0.80.13 ± 0.040.001
Union 220.24 ± 0.0216.6 ± 1.343.9 ± 2.696.6 ± 0.30.18 ± 0.020.001
Union 3-------
Intersection-------
kNN100%-320.17 ± 0.0212.9 ± 1.624.2 ± 2.893.6 ± 0.30.08 ± 0.020.001
50%Union300.16 ± 0.0212.0 ± 1.423.4 ± 2.393.8 ± 0.40.07 ± 0.020.001
Union 2220.20 ± 0.0220.5 ± 2.419.8 ± 1.786.9 ± 0.70.07 ± 0.020.002
Union 3100.21 ± 0.0216.6 ± 2.028.8 ± 2.393.5 ± 0.40.12 ± 0.020.001
Intersection20.22 ± 0.0222.2 ± 2.321.5 ± 2.287.2 ± 0.70.09 ± 0.030.006
25%Union220.24 ± 0.0222.7 ± 1.724.5 ± 1.789.0 ± 0.40.12 ± 0.020.002
Union 290.23 ± 0.0222.2 ± 2.024.5 ± 1.789.2 ± 0.40.12 ± 0 020.001
Union 310.21 ± 0.0216.6 ± 2.028.8 ± 2.393.5 ± 0.40.12 ± 0.020.005
Intersection-------
10%Union100.24 ± 0.0124.0 ± 1.324.8 ± 1.088.5 ± 0.50.13 ± 0.010.001
Union 220.22 ± 0.0317.3 ± 2.229.7 ± 4.293.5 ± 0.60.13 ± 0.030.001
Union 3-------
Intersection-------
SVM100%-320.18 ± 0.0216.0 ± 1.820.8 ± 2.390.4 ± 0.70.07 ± 0.020.002
50%Union300.18 ± 0.0115.6 ± 1.520.6 ± 1.790.5 ± 0.60.07 ± 0.020.007
Union 2220.19 ± 0.0214.1 ± 1.629.8 ± 3.194.7 ± 0.60.11 ± 0.020.001
Union 3100.15 ± 0.0210.2 ± 1.227.4 ± 2.795.7 ± 0.30.08 ± 0.020.001
Intersection20.10 ± 0.015.6 ± 0.837.0 ± 2.298.5 ± 0.10.06 ± 0.010.001
25%Union220.18 ± 0.0215.8 ± 2.121.0 ± 2.590.6 ± 0.60.07 ± 0.020.002
Union 290.19 ± 0.0216.0 ± 1.624.5 ± 2.392.2 ± 0.60.10 ± 0.020.001
Union 310.15 ± 0.0210.2 ± 1.227.4 ± 2.795.7 ± 0.30.08 ± 0.020.001
Intersection-------
10%Union100.18 ± 0.0313.2 ± 1.930.0 ± 4.395.1 ± 0.40.11 ± 0.030.001
Union 220.10 ± 0.025.6 ± 0.950.6 ± 5.899.1 ± 0.20.08 ± 0.010.001
Union 3-------
Intersection-------
Table
Table A11. Performance of the four classification models on detection fall using Twelve Item Multiple Sclerosis Walking Scale data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A11. Performance of the four classification models on detection fall using Twelve Item Multiple Sclerosis Walking Scale data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-120.50 ± 0.0070.9 ± 0.439.1 ± 0.282.3 ± 0.10.40 ± 0.000.001
50%Union100.51 ± 0.0071.4 ± 0.339.5 ± 0.182.5 ± 0.10.40 ± 0.000.001
Union 280.52 ± 0.0070.8 ± 0.441.0 ± 0.283.6 ± 0.10.42 ± 0.000.001
Union 350.52 ± 0.0067.5 ± 0.542.3 ± 0.385.2 ± 0.20.42 ± 0.000.001
Intersection10.30 ± 0.0523.8 ± 5.544.2 ± 2.795.1 ± 1.40.23 ± 0.040.001
25%Union70.53 ± 0.0071.6 ± 0.441.8 ± 0.284.0 ± 0.10.43 ± 0.000.001
Union 230.54 ± 0.0065.9 ± 0.045.4 ± 0.187.3 ± 0.00.45 ± 0.000.001
Union 310.30 ± 0.0523.8 ± 5.544.2 ± 2.795.1 ± 1.40.23 ± 0.040.001
Intersection10.30 ± 0.0523.8 ± 5.544.2 ± 2.795.1 ± 1.40.23 ± 0.040.001
10%Union20.48 ± 0.0050.2 ± 0.346.2 ± 0.290.6 ± 0.10.39 ± 0.000.001
Union 210.30 ± 0.0523.8 ± 5.544.2 ± 2.795.1 ± 1.40.23 ± 0.040.001
Union 310.30 ± 0.0523.8 ± 5.544.2 ± 2.795.1 ± 1.40.23 ± 0.040.001
Intersection-------
DT100%-120.33 ± 0.0324.4 ± 1.949.8 ± 6.396.0 ± 1.00.26 ± 0.030.001
50%Union100.30 ± 0.0723.3 ± 6.845.2 ± 5.095.5 ± 1.30.23 ± 0.060.001
Union 280.33 ± 0.0426.6 ± 4.942.8 ± 3.294.3 ± 1.10.25 ± 0.040.001
Union 350.39 ± 0.0431.1 ± 4.350.9 ± 3.895.2 ± 0.60.31 ± 0.040.001
Intersection10.01 ± 0.020.5 ± 1.16.3 ± 13.599.8 ± 0.30.01 ± 0.011.000
25%Union70.33 ± 0.0826.7 ± 8.543.6 ± 4.294.6 ± 1.60.25 ± 0.070.001
Union 230.35 ± 0.0328.3 ± 3.845.8 ± 2.694.6 ± 0.70.27 ± 0.030.001
Union 310.01 ± 0.020.5 ± 1.16.3 ± 13.599.8 ± 0.30.01 ± 0.011.000
Intersection10.01 ± 0.020.5 ± 1.16.3 ± 13.599.8 ± 0.30.01 ± 0.011.000
10%Union20.22 ± 0.0414.1 ± 3.155.7 ± 7.198.1 ± 0.90.18 ± 0.030.001
Union 210.01 ± 0.020.5 ± 1.16.3 ± 13.599.8 ± 0.30.01 ± 0.011.000
Union 310.01 ± 0.020.5 ± 1.16.3 ± 13.599.8 ± 0.30.01 ± 0.011.000
Intersection-------
kNN100%-120.37 ± 0.0133.1 ± 1.441.5 ± 2.092.5 ± 0.60.28 ± 0.020.001
50%Union100.38 ± 0.0232.1 ± 1.646.9 ± 3.394.1 ± 0.80.30 ± 0.020.001
Union 280.35 ± 0.0229.1 ± 1.945.1 ± 1.894.3 ± 0.40.27 ± 0.020.001
Union 350.31 ± 0.0326.1 ± 2.936.9 ± 2.692.8 ± 0.70.22 ± 0.030.001
Intersection10.30 ± 0.0724.8 ± 7.841.1 ± 5.593.8 ± 3.00.22 ± 0.050.001
25%Union70.37 ± 0.0332.3 ± 2.542.6 ± 3.393.0 ± 0.50.28 ± 0.030.001
Union 230.38 ± 0.0231.2 ± 2.147.6 ± 2.494.5 ± 0.40.30 ± 0.020.001
Union 310.30 ± 0.0724.8 ± 7.841.1 ± 5.593.8 ± 3.00.22 ± 0.050.001
Intersection10.30 ± 0.0724.8 ± 7.841.1 ± 5.593.8 ± 3.00.22 ± 0.050.001
10%Union20.37 ± 0.0632.9 ± 8.243.6 ± 5.893.1 ± 1.90.29 ± 0.060.001
Union 210.30 ± 0.0724.8 ± 7.841.1 ± 5.593.8 ± 3.00.22 ± 0.050.001
Union 310.30 ± 0.0724.8 ± 7.841.1 ± 5.593.8 ± 3.00.22 ± 0.050.001
Intersection-------
SVM100%-120.36 ± 0.0234.2 ± 2.137.2 ± 2.190.7 ± 0.70.26 ± 0.020.001
50%Union100.37 ± 0.0235.9 ± 2.538.0 ± 2.090.6 ± 0.40.27 ± 0.020.001
Union 280.37 ± 0.0235.7 ± 2.138.7 ± 1.890.9 ± 0.70.27 ± 0.020.001
Union 350.30 ± 0.0324.1 ± 2.639.7 ± 2.794.1 ± 0.40.22 ± 0.030.001
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
25%Union70.37 ± 0.0233.8 ± 2.642.0 ± 2.492.5 ± 0.50.29 ± 0.030.001
Union 230.29 ± 0.0421.4 ± 3.845.7 ± 4.695.9 ± 0.40.22 ± 0.040.001
Union 310.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
Intersection10.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
10%Union20.24 ± 0.0215.1 ± 1.457.6 ± 6.198.2 ± 0.40.19 ± 0.020.001
Union 210.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
Union 310.00 ± 0.000.0 ± 0.00.0 ± 0.01.0 ± 0.00.00 ± 0.001.000
Intersection-------
Table
Table A12. Performance of the four classification models on detection fall using Early Mobility Impairment Questionnaire data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Table A12. Performance of the four classification models on detection fall using Early Mobility Impairment Questionnaire data set; Gaussian Naive Bayes (GNB); Decision Tree (DT); k-Nearest Neighbor (kNN); Support Vector Machine (SVM); p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
GNB100%-90.54 ± 0.0371.3 ± 0.343.1 ± 0.384.8 ± 0.20.44 ± 0.000.001
50%Union80.54 ± 0.0070.2 ± 0.244.1 ± 0.385.7 ± 0.10.45 ± 0.000.001
Union 250.54 ± 0.0066.0 ± 0.745.7 ± 0.587.3 ± 0.20.45 ± 0.010.001
Union 350.54 ± 0.0066.0 ± 0.745.7 ± 0.587.3 ± 0.20.45 ± 0.010.001
Intersection20.50 ± 0.0055.6 ± 0.745.7 ± 0.089.4 ± 0.10.41 ± 0.000.001
25%Union40.54 ± 0.0161.0 ± 1.448.0 ± 0.589.4 ± 0.20.45 ± 0.010.001
Union 230.53 ± 0.0059.9 ± 1.148.1 ± 0.389.6 ± 0.30.45 ± 0.000.001
Union 310.45 ± 0.0040.1 ± 0.051.5 ± 0.093.9 ± 0.00.37 ± 0.000.001
Intersection-------
10%Union20.48 ± 0.0150.2 ± 0.845.3 ± 0.690.2 ± 0.10.39 ± 0.010.001
Union 210.45 ± 0.0040.1 ± 0.051.5 ± 0.093.9 ± 0.00.37 ± 0.000.001
Union 310.45 ± 0.0040.1 ± 0.051.5 ± 0.093.9 ± 0.00.37 ± 0.000.001
Intersection-------
DT100%-90.35 ± 0.0627.4 ± 6.249.0 ± 4.495.5 ± 0.70.28 ± 0.060.001
50%Union80.35 ± 0.0627.4 ± 6.249.1 ± 4.595.5 ± 0.80.28 ± 0.060.001
Union 250.35 ± 0.0726.3 ± 6.654.9 ± 3.796.5 ± 0.90.29 ± 0.060.001
Union 350.35 ± 0.0726.3 ± 6.654.9 ± 3.796.5 ± 0.90.29 ± 0.060.001
Intersection20.02 ± 0.041.0 ± 2.15.7 ± 12.099.5 ± 0.80.01 ± 0.020.001
25%Union40.32 ± 0.0423.7 ± 3.648.5 ± 2.296.0 ± 0.40.25 ± 0.030.001
Union 230.31 ± 0.0322.2 ± 2.849.8 ± 2.796.4 ± 0.20.24 ± 0.030.001
Union 310.39 ± 0.0029.9 ± 0.654.6 ± 1.896.0 ± 0.40.32 ± 0.000.001
Intersection-------
10%Union20.31 ± 0.0322.3 ± 2.551.4 ± 2.896.6 ± 0.20.25 ± 0.030.001
Union 210.39 ± 0.0029.9 ± 0.654.6 ± 1.896.0 ± 0.40.32 ± 0.000.001
Union 310.39 ± 0.0029.9 ± 0.654.6 ± 1.896.0 ± 0.40.32 ± 0.000.001
Intersection-------
kNN100%-90.35 ± 0.0329.0 ± 2.444.5 ± 2.794.2 ± 0.40.27 ± 0.030.001
50%Union80.36 ± 0.0328.0 ± 2.149.5 ± 3.995.4 ± 0.50.29 ± 0.030.001
Union 250.35 ± 0.0227.7 ± 2.748.8 ± 2.495.3 ± 0.50.28 ± 0.020.001
Union 350.35 ± 0.0227.7 ± 2.748.8 ± 2.495.3 ± 0.50.28 ± 0.020.001
Intersection20.29 ± 0.0622.9 ± 5.940.2 ± 3.694.6 ± 1.20.21 ± 0.050.001
25%Union40.37 ± 0.0331.3 ± 3.645.8 ± 4.494.0 ± 1.20.29 ± 0.030.001
Union 230.39 ± 0.0332.6 ± 4.548.1 ± 3.994.2 ± 1.40.31 ± 0.030.001
Union 310.27 ± 0.0619.1 ± 5.951.1 ± 8.296.7 ± 2.00.21 ± 0.050.001
Intersection-------
10%Union20.31 ± 0.0424.5 ± 4.243.5 ± 5.094.8 ± 1.00.24 ± 0.040.001
Union 210.27 ± 0.0619.1 ± 5.951.1 ± 8.296.7 ± 2.00.21 ± 0.050.001
Union 310.27 ± 0.0619.1 ± 5.951.1 ± 8.296.7 ± 2.00.21 ± 0.050.001
Intersection-------
SVM100%-90.38 ± 0.0330.4 ± 2.851.1 ± 3.495.3 ± 0.50.31 ± 0.030.001
50%Union80.39 ± 0.0331.4 ± 2.852.9 ± 2.795.5 ± 0.30.32 ± 0.030.001
Union 250.37 ± 0.0230.2 ± 2.249.0 ± 2.794.9 ± 0.40.30 ± 0.020.001
Union 350.37 ± 0.0230.2 ± 2.249.0 ± 2.794.9 ± 0.40.30 ± 0.020.001
Intersection20.04 ± 0.042.0 ± 2.114.5 ± 13.098.9 ± 0.90.01 ± 0.020.032
25%Union40.32 ± 0.0224.5 ± 1.846.2 ± 3.795.4 ± 0.50.25 ± 0.030.001
Union 230.34 ± 0.0326.2 ± 2.848.9 ± 2.395.6 ± 0.40.27 ± 0.030.001
Union 310.38 ± 0.0329.0 ± 2.954.1 ± 2.196.0 ± 0.40.31 ± 0.030.001
Intersection-------
10%Union20.39 ± 0.0329.8 ± 3.154.7 ± 2.096.0 ± 0.20.32 ± 0.030.001
Union 210.38 ± 0.0329.0 ± 2.954.1 ± 2.196.0 ± 0.40.31 ± 0.030.001
Union 310.38 ± 0.0329.0 ± 2.954.1 ± 2.196.0 ± 0.40.31 ± 0.030.001
Intersection-------

References

  1. Morrison, S.; Rynders, C.A.; Sosnoff, J.J. Deficits in medio-lateral balance control and the implications for falls in individuals with multiple sclerosis. Gait Posture 2016, 49, 148–154. [Google Scholar] [CrossRef] [PubMed]
  2. Allali, G.; Laidet, M.; Armand, S.; Elsworth-Edelsten, C.; Assal, F.; Lalive, P.H. Stride time variability as a marker for higher level of gait control in multiple sclerosis: Its association with fear of falling. J. Neural Transm. 2016, 123, 595–599. [Google Scholar] [CrossRef] [PubMed]
  3. Kalron, A.; Frid, L.; Gurevich, M. Concern about falling is associated with step length in persons with multiple sclerosis. Eur. J. Phys. Rehabil. Med. 2015, 51, 197–205. [Google Scholar]
  4. Nilsagard, Y.; Gunn, H.; Freeman, J.; Hoang, P.; Lord, S.; Mazumder, R.; Cameron, M. Falls in people with MS—An individual data meta-analysis from studies from Australia, Sweden, United Kingdom and the United States. Mult. Scler. J. 2015, 21, 92–100. [Google Scholar] [CrossRef] [Green Version]
  5. Nilsagård, Y.; Lundholm, C.; Denison, E.; Gunnarsson, L.G. Predicting accidental falls in people with multiple sclerosis—A longitudinal study. Clin. Rehabil. 2009, 23, 259–269. [Google Scholar] [CrossRef]
  6. Finlayson, M.L.; Peterson, E.W.; Cho, C.C. Risk Factors for Falling among People Aged 45 to 90 Years with Multiple Sclerosis. Arch. Phys. Med. Rehabil. 2006, 87, 1274–1279. [Google Scholar] [CrossRef] [PubMed]
  7. Kasser, S.L.; Jacobs, J.V.; Foley, J.T.; Cardinal, B.J.; Maddalozzo, G.F. A prospective evaluation of balance, gait, and strength to predict falling in women with multiple sclerosis. Arch. Phys. Med. Rehabil. 2011, 92, 1840–1846. [Google Scholar] [CrossRef]
  8. Scholz, M.; Haase, R.; Trentzsch, K.; Weidemann, M.L.; Ziemssen, T. Fear of falling and falls in people with multiple sclerosis: A literature review. Mult. Scler. Relat. Disord. 2021, 47, 102609. [Google Scholar] [CrossRef]
  9. Cameron, M.H.; Nilsagard, Y. Balance, gait, and falls in multiple sclerosis. In Handbook of Clinical Neurology; Elsevier: Amsterdam, The Netherlands, 2018; Volume 159, pp. 237–250. [Google Scholar]
  10. Kalron, A.; Allali, G. Gait and cognitive impairments in multiple sclerosis: The specific contribution of falls and fear of falling. J. Neural Transm. 2017, 124, 1407–1416. [Google Scholar] [CrossRef]
  11. Panitch, H.; Applebee, A. Treatment of walking impairment in multiple sclerosis: An unmet need for a disease-specific disability. Expert Opin. Pharmacother. 2011, 12, 1511–1521. [Google Scholar] [CrossRef]
  12. Zwibel, H.L. Contribution of impaired mobility and general symptoms to the burden of multiple sclerosis. Adv. Ther. 2009, 26, 1043–1057. [Google Scholar] [CrossRef] [PubMed]
  13. D’Amico, E.; Haase, R.; Ziemssen, T. Review: Patient-reported outcomes in multiple sclerosis care. Mult. Scler. Relat. Disord. 2019, 33, 61–66. [Google Scholar] [CrossRef]
  14. Fritz, N.E.; Kloos, A.D.; Kegelmeyer, D.A.; Kaur, P.; Nichols-Larsen, D.S. Supplementary motor area connectivity and dual-task walking variability in multiple sclerosis. J. Neurol. Sci. 2019, 396, 159–164. [Google Scholar] [CrossRef] [PubMed]
  15. Martin, C.L.; Phillips, B.A.; Kilpatrick, T.J.; Butzkueven, H.; Tubridy, N.; McDonald, E.; Galea, M.P. Gait and balance impairment in early multiple sclerosis in the absence of clinical disability. Mult. Scler. 2006, 12, 620–628. [Google Scholar] [CrossRef] [PubMed]
  16. Pau, M.; Mandaresu, S.; Pilloni, G.; Porta, M.; Coghe, G.; Marrosu, M.G.; Cocco, E. Smoothness of gait detects early alterations of walking in persons with multiple sclerosis without disability. Gait Posture 2017, 58, 307–309. [Google Scholar] [CrossRef]
  17. Ziemssen, T.; Phillips, G.; Shah, R.; Mathias, A.; Foley, C.; Coon, C.; Sen, R.; Lee, A.; Agarwal, S. Development of the multiple sclerosis (MS) early mobility impairment questionnaire (EMIQ). J. Neurol. 2016, 263, 1969–1983. [Google Scholar] [CrossRef]
  18. Torchio, A.; Corrini, C.; Anastasi, D.; Parelli, R.; Meotti, M.; Spedicato, A.; Groppo, E.; D’Arma, A.; Grosso, C.; Montesano, A.; et al. Identification of modified dynamic gait index cutoff scores for assessing fall risk in people with Parkinson disease, stroke and multiple sclerosis. Gait Posture 2022, 91, 1–6. [Google Scholar] [CrossRef]
  19. Drover, D.; Howcroft, J.; Kofman, J.; Lemaire, E. Faller Classification in Older Adults Using Wearable Sensors Based on Turn and Straight-Walking Accelerometer-Based Features. Sensors 2017, 17, 1321. [Google Scholar] [CrossRef]
  20. Palmerini, L.; Klenk, J.; Becker, C.; Chiari, L. Accelerometer-Based Fall Detection Using Machine Learning: Training and Testing on Real-World Falls. Sensors 2020, 20, 6479. [Google Scholar] [CrossRef]
  21. Silva, C.A.; García−Bermúdez, R.; Casilari, E. Features Selection for Fall Detection Systems Based on Machine Learning and Accelerometer Signals. In Lecture Notes in Computer Science (including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer Science and Business Media Deutschland GmbH: Berlin, Germany, 2021; Volume 12862, pp. 380–391. [Google Scholar]
  22. Kelly, D.; Condell, J.; Gillespie, J.; Munoz Esquivel, K.; Barton, J.; Tedesco, S.; Nordstrom, A.; Åkerlund Larsson, M.; Alamäki, A. Improved screening of fall risk using free-living based accelerometer data. J. Biomed. Inform. 2022, 131, 104116. [Google Scholar] [CrossRef]
  23. Chandak, A.; Chaturvedi, N. Dhiraj Machine-Learning-Based Human Fall Detection Using Contact- and Noncontact-Based Sensors. Comput. Intell. Neurosci. 2022, 2022, 9626170. [Google Scholar] [CrossRef] [PubMed]
  24. Zurbuchen, N.; Wilde, A.; Bruegger, P. A Machine Learning Multi-Class Approach for Fall Detection Systems Based on Wearable Sensors with a Study on Sampling Rates Selection. Sensors 2021, 21, 938. [Google Scholar] [CrossRef] [PubMed]
  25. Liu, K.C.; Hsieh, C.Y.; Hsu, S.J.P.; Chan, C.T. Impact of Sampling Rate on Wearable-Based Fall Detection Systems Based on Machine Learning Models. IEEE Sens. J. 2018, 18, 9882–9890. [Google Scholar] [CrossRef]
  26. Martins, L.M.; Ribeiro, N.F.; Soares, F.; Santos, C.P. Inertial Data-Based AI Approaches for ADL and Fall Recognition. Sensors 2022, 22, 4028. [Google Scholar] [CrossRef]
  27. Rehman, R.Z.U.; Zhou, Y.; Din, S.D.; Alcock, L.; Hansen, C.; Guan, Y.; Hortobágyi, T.; Maetzler, W.; Rochester, L.; Lamoth, C.J.C. Gait Analysis with Wearables Can Accurately Classify Fallers from Non-Fallers: A Step toward Better Management of Neurological Disorders. Sensors 2020, 20, 6992. [Google Scholar] [CrossRef]
  28. Meyer, B.M.; Tulipani, L.J.; Gurchiek, R.D.; Allen, D.A.; Adamowicz, L.; Larie, D.; Solomon, A.J.; Cheney, N.; McGinnis, R.S. Wearables and Deep Learning Classify Fall Risk from Gait in Multiple Sclerosis. IEEE J. Biomed. Health Inform. 2021, 25, 1824–1831. [Google Scholar] [CrossRef]
  29. Piryonesi, S.M.; Rostampour, S.; Piryonesi, S.A. Predicting falls and injuries in people with multiple sclerosis using machine learning algorithms. Mult. Scler. Relat. Disord. 2021, 49, 102740. [Google Scholar] [CrossRef]
  30. Ravi, D.; Wong, C.; Deligianni, F.; Berthelot, M.; Andreu-Perez, J.; Lo, B.; Yang, G.Z. Deep Learning for Health Informatics. IEEE J. Biomed. Health Informatics 2017, 21, 4–21. [Google Scholar] [CrossRef] [Green Version]
  31. Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M.C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016, 316, 2402–2410. [Google Scholar] [CrossRef]
  32. Chandrashekar, G.; Sahin, F. A survey on feature selection methods. Comput. Electr. Eng. 2014, 40, 16–28. [Google Scholar] [CrossRef]
  33. Bommert, A.; Sun, X.; Bischl, B.; Rahnenführer, J.; Lang, M. Benchmark for filter methods for feature selection in high-dimensional classification data. Comput. Stat. Data Anal. 2020, 143, 106839. [Google Scholar] [CrossRef]
  34. Bolón-Canedo, V.; Sánchez-Maroño, N.; Alonso-Betanzos, A. A review of feature selection methods on synthetic data. Knowl. Inf. Syst. 2013, 34, 483–519. [Google Scholar] [CrossRef]
  35. Liu, H.; Yu, L. Toward integrating feature selection algorithms for classification and clustering. IEEE Trans. Knowl. Data Eng. 2005, 17, 491–502. [Google Scholar] [CrossRef] [Green Version]
  36. Bolón-Canedo, V.; Alonso-Betanzos, A. Ensembles for feature selection: A review and future trends. Inf. Fusion 2019, 52, 1–12. [Google Scholar] [CrossRef]
  37. Trentzsch, K.; Weidemann, M.L.; Torp, C.; Inojosa, H.; Scholz, M.; Haase, R.; Schriefer, D.; Akgün, K.; Ziemssen, T. The Dresden Protocol for Multidimensional Walking Assessment (DMWA) in Clinical Practice. Front. Neurosci. 2020, 14, 582046. [Google Scholar] [CrossRef]
  38. Electronic Gaitr. GAITRite Electronic Walkway Technical Reference. Available online: https://www.procarebv.nl/wp-content/uploads/2017/01/Technische-aspecten-GAITrite-Walkway-System.pdf (accessed on 5 June 2021).
  39. APDM—Wearable Technologies. Available online: https://www.apdm.com/ (accessed on 4 October 2018).
  40. Mancini, M.; King, L.; Salarian, A.; Holmstrom, L.; McNames, J.; Horak, F.B. Mobility Lab to Assess Balance and Gait with Synchronized Body-worn Sensors. J. Bioeng. Biomed. Sci. 2013, 12 (Suppl. 1), 7. [Google Scholar] [CrossRef]
  41. Kurtzke, J.F. Rating neurologic impairment in multiple sclerosis: An expanded disability status scale (EDSS). Neurology 1983, 33, 1444–1452. [Google Scholar] [CrossRef] [Green Version]
  42. Inojosa, H.; Schriefer, D.; Ziemssen, T. Clinical outcome measures in multiple sclerosis: A review. Autoimmun. Rev. 2020, 19, 102512. [Google Scholar] [CrossRef]
  43. Cutter, G.R.; Baier, M.L.; Rudick, R.A.; Cookfair, D.L.; Fischer, J.S.; Petkau, J.; Syndulko, K.; Weinshenker, B.G.; Antel, J.P.; Confavreux, C.; et al. Development of a multiple sclerosis functional composite as a clinical trial outcome measure. Brain 1999, 122, 871–882. [Google Scholar] [CrossRef]
  44. Rossier, P.; Wade, D.T. Validity and reliability comparison of 4 mobility measures in patients presenting with neurologic impairment. Arch. Phys. Med. Rehabil. 2001, 82, 9–13. [Google Scholar] [CrossRef]
  45. Hobart, J.C.; Riazi, A.; Lamping, D.L.; Fitzpatrick, R.; Thompson, A.J. Measuring the impact of MS on walking ability: The 12-item MS Walking Scale (MSWS-12). Neurology 2003, 60, 31–36. [Google Scholar] [CrossRef]
  46. Seijo-Pardo, B.; Bolón-Canedo, V.; Alonso-Betanzos, A. On developing an automatic threshold applied to feature selection ensembles. Inf. Fusion 2019, 45, 227–245. [Google Scholar] [CrossRef]
  47. Ali, S.I.; Bilal, H.S.M.; Hussain, M.; Hussain, J.; Satti, F.A.; Hussain, M.; Park, G.H.; Chung, T.; Lee, S. Ensemble Feature Ranking for Cost-Based Non-Overlapping Groups: A Case Study of Chronic Kidney Disease Diagnosis in Developing Countries. IEEE Access 2020, 8, 215623–215648. [Google Scholar] [CrossRef]
  48. Bolón-Canedo, V.; Sánchez-Maroño, N.; Alonso-Betanzos, A. An ensemble of filters and classifiers for microarray data classification. Pattern Recognit. 2012, 45, 531–539. [Google Scholar] [CrossRef]
  49. Tsymbal, A.; Pechenizkiy, M.; Cunningham, P. Diversity in search strategies for ensemble feature selection. Inf. Fusion 2005, 6, 83–98. [Google Scholar] [CrossRef]
  50. Sagi, O.; Rokach, L. Ensemble learning: A survey. WIREs Data Min. Knowl. Discov. 2018, 8, e1249. [Google Scholar] [CrossRef]
  51. The MathWorks, Inc. The Statistics and Machine Learning ToolboxTM: User’s Guide v12.2; The MathWorks, Inc.: Natick, MA, USA; Available online: https://de.mathworks.com/help/pdf_doc/stats/stats.pdf (accessed on 6 March 2022).
  52. Frank, E.; Hall, M.A.; Witten, I.H. The WEKA Workbench. Online Appendix for “Data Mining: Practical Machine Learning Tools and Techniques”, 4th ed.; Morgan Kaufmann: Burlington, MA, USA, 2016. [Google Scholar]
  53. He, H.; Garcia, E.A. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 2009, 21, 1263–1284. [Google Scholar] [CrossRef]
  54. Kotsiantis, S.; Kanellopoulos, D.; Pintelas, P. Handling imbalanced datasets: A review. GESTS Int. Trans. Comput. Sci. Eng. 2006, 30, 25–36. [Google Scholar]
  55. Ojala, M.; Garriga, G.C. Permutation Tests for Studying Classifier Performance Markus Ojala. J. Mach. Learn. Res. 2010, 11, 1833–1863. [Google Scholar]
  56. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michael, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  57. Scholz, M.; Haase, R.; Trentzsch, K.; Stölzer-hutsch, H.; Ziemssen, T. Improving Digital Patient Care: Lessons Learned from Patient-Reported and Expert-Reported Experience Measures for the Clinical Practice of Multidimensional Walking Assessment. Brain Sci. 2021, 11, 786. [Google Scholar] [CrossRef]
  58. Nowinski, C.J.; Miller, D.M.; Cella, D. Evolution of Patient-Reported Outcomes and Their Role in Multiple Sclerosis Clinical Trials. Neurotherapeutics 2017, 14, 934–944. [Google Scholar] [CrossRef]
  59. Callis, N. Falls prevention: Identification of predictive fall risk factors. Appl. Nurs. Res. 2016, 29, 53–58. [Google Scholar] [CrossRef]
  60. Subramaniam, S.; Faisal, A.I.; Deen, M.J. Wearable Sensor Systems for Fall Risk Assessment: A Review. Front. Digit. Health 2022, 4, 921506. [Google Scholar] [CrossRef]
  61. Howcroft, J.; Kofman, J.; Lemaire, E.D. Prospective Fall-Risk Prediction Models for Older Adults Based on Wearable Sensors. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 25, 1812–1820. [Google Scholar] [CrossRef]
  62. Kalron, A.; Achiron, A. Postural control, falls and fear of falling in people with multiple sclerosis without mobility aids. J. Neurol. Sci. 2013, 335, 186–190. [Google Scholar] [CrossRef]
  63. Roma, P.; Monaro, M.; Muzi, L.; Colasanti, M.; Ricci, E.; Biondi, S.; Napoli, C.; Ferracuti, S.; Mazza, C. How to Improve Compliance with Protective Health Measures during the COVID-19 Outbreak: Testing a Moderated Mediation Model and Machine Learning Algorithms. Int. J. Environ. Res. Public Health 2020, 17, 7252. [Google Scholar] [CrossRef]
  64. Oztekin, A.; Delen, D.; Turkyilmaz, A.; Zaim, S. A machine learning-based usability evaluation method for eLearning systems. Decis. Support Syst. 2013, 56, 63–73. [Google Scholar] [CrossRef]
  65. Saeys, Y.; Abeel, T.; Van De Peer, Y. Robust feature selection using ensemble feature selection techniques. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Antwerp, Belgium, 15–19 September 2008; Volume 5212, pp. 313–325. [Google Scholar] [CrossRef] [Green Version]
  66. Saeys, Y.; Inza, I.; Larranaga, P. A review of feature selection techniques in bioinformatics. Bioinformatics 2007, 23, 2507–2517. [Google Scholar] [CrossRef] [Green Version]
  67. Duan, K.; Keerthi, S.S.; Poo, A.N. Evaluation of simple performance measures for tuning SVM hyperparameters. Neurocomputing 2003, 51, 41–59. [Google Scholar] [CrossRef]
  68. Wolpert, D.H. On Overfitting Avoidance as Bias; The Santa Fe Institute: Santa Fe, NM, USA, 1993. [Google Scholar]
  69. Sun, Y.; Wong, A.K.C.; Kamel, M.S. Classification of imbalanced data: A review. Int. J. Pattern Recognit. Artif. Intell. 2009, 23, 687–719. [Google Scholar] [CrossRef]
  70. Moghimi, A.; Yang, C.; Marchetto, P.M. Ensemble Feature Selection for Plant Phenotyping: A Journey from Hyperspectral to Multispectral Imaging. IEEE Access 2018, 6, 56870–56884. [Google Scholar] [CrossRef]
Brainsci 12 01477 g001 550
Figure 1. Workflow of the feature selection ensemble and the classification methods. Chi-square test (Chi-Square), Information gain (InfoGain), Minimum Redundancy Maximum Relevance (MRMR) and RelieF rank the features. Then, a threshold reduced this ranked list to a feature subset. A combination method combined these four feature subsets to a final feature subset for classification. Three different thresholds were investigated: 50%, 25%, 10%. Four different combination methods were investigated: Union, Union 2, Union 3, Intersection.
Figure 1. Workflow of the feature selection ensemble and the classification methods. Chi-square test (Chi-Square), Information gain (InfoGain), Minimum Redundancy Maximum Relevance (MRMR) and RelieF rank the features. Then, a threshold reduced this ranked list to a feature subset. A combination method combined these four feature subsets to a final feature subset for classification. Three different thresholds were investigated: 50%, 25%, 10%. Four different combination methods were investigated: Union, Union 2, Union 3, Intersection.
Brainsci 12 01477 g001
Brainsci 12 01477 g002 550
Figure 2. Best Performance (F1 score) of the four classification models on the detection of fall using different data sets. The error bars present the mean and standard deviation across 5-fold cross-validation repetition. All data set = All features of the following data sets; Basic data set = 25 Foot Walk Test + 2-minute walk test + EDSS; Expanded Disability Status Scale (EDSS); GAITRite System normal (GR_N data set) and dual-task (GR_D data set) walking; Mobility Lab Gait normal (ML_N data set) and dual-task (ML_D data set) walking; Mobility Lab Romberg stance with open (ML_S_EO data set) and closed eyes (ML_S_EC data set); Twelve Item Multiple Sclerosis Walking Scale (MSWS-12); Early Mobility Impairment Questionnaire (EMIQ).
Figure 2. Best Performance (F1 score) of the four classification models on the detection of fall using different data sets. The error bars present the mean and standard deviation across 5-fold cross-validation repetition. All data set = All features of the following data sets; Basic data set = 25 Foot Walk Test + 2-minute walk test + EDSS; Expanded Disability Status Scale (EDSS); GAITRite System normal (GR_N data set) and dual-task (GR_D data set) walking; Mobility Lab Gait normal (ML_N data set) and dual-task (ML_D data set) walking; Mobility Lab Romberg stance with open (ML_S_EO data set) and closed eyes (ML_S_EC data set); Twelve Item Multiple Sclerosis Walking Scale (MSWS-12); Early Mobility Impairment Questionnaire (EMIQ).
Brainsci 12 01477 g002
Brainsci 12 01477 g003 550
Figure 3. Classification performance (F1 score) of the four models on All data set depending on the number of features. The error bars present the mean and standard deviation across 5-fold cross-validation repetition. A dotted line maps the F1 score trend with increasing feature integer values. No assumption is made for the performance of feature subsets that lie between the data points. All data set = 25 Foot Walk Test + 2-minute walk test + Expanded Disability Status Scale + GAITRite System data sets + Mobility Lab Gait data sets + 12 item Multiple Sclerosis Walking Scale data set + Early Mobility Impairment Questionnaire data set.
Figure 3. Classification performance (F1 score) of the four models on All data set depending on the number of features. The error bars present the mean and standard deviation across 5-fold cross-validation repetition. A dotted line maps the F1 score trend with increasing feature integer values. No assumption is made for the performance of feature subsets that lie between the data points. All data set = 25 Foot Walk Test + 2-minute walk test + Expanded Disability Status Scale + GAITRite System data sets + Mobility Lab Gait data sets + 12 item Multiple Sclerosis Walking Scale data set + Early Mobility Impairment Questionnaire data set.
Brainsci 12 01477 g003
Table
Table 1. Mobility and clinical parameters recorded by the physician or nurse, self-reported gait parameters by patients as well as gait parameters recorded by gait analysis systems.
Table 1. Mobility and clinical parameters recorded by the physician or nurse, self-reported gait parameters by patients as well as gait parameters recorded by gait analysis systems.
Data SetNo.Features
Basic data set11Walking speed (T25-FW), walking endurance (2MWT), EDSS subcategories of functional systems (visual, brainstem, pyramidal, cerebellar, sensory, bowel bladder, cognition/fatigue, ambulation) and total score
EDSS data set9EDSS subcategories of functional systems (visual, brainstem, pyramidal, cerebellar, sensory, bowel bladder, cognition/fatigue, ambulation) and total score
GAITRite System (GR_N and GR_D data set)82Ambulation Time, Cadence, Cycle Time, Walking Distance, Foot Length, Foot Width, Functional Ambulation Profile, Leg Length, Step Count, Step Length, Step Time, Stride Length, Stride Time, Stride Velocity, walking velocity, various parameters of the individual gait phases [38]
Mobility Lab Gait (ML_N and ML_D data set)84Duration, Cadence, Circumduction, Foot Strike Angle, Gait Cycle Duration, Gait Speed, Lateral Step Variability, Stance duration, Step Duration, Stride Length, Toe Off Angle, Coronal Range of Motion, Sagittal Range of Motion, Transverse Range of Motion, Turns Angle, Turns Duration, Steps in Turn, Turn Velocity, Arm Range of Motion, Arm Swing Velocity [39]
Mobility Lab Postural Sway (ML_S_EO and ML_S_EC data set)32Ellipse Axis, Ellipse Rotation, Range of sway in the sagittal and coronal plane, anterior-posterior and mediolateral trunk fluctuation range, Sway Velocity [39]
MSWS-12 data set12Self report: Limitation of walking ability, Limitation of balance, Limitation of walking distance, Effort for walking, Need of walking aids, Limitation of walking velocity [45]
EMIQ data set
In the past 30 days, how often have you…		9Self report: Walking ability without/during cognitive tasks, Need of walking aid, Limitation of walking velocity, Limitation of household, social and physical activity [17]
All data set428All features of the Basic data set, the GAITRite System data sets, the Mobility Lab Gait data sets, the MSWS-12 data set and the EMIQ data set
25 Foot Walk Test (T25-FW), 2-minute walk test (2MWT), Expanded Disability Status Scale (EDSS); Twelve Item Multiple Sclerosis Walking Scale (MSWS-12); Early Mobility Impairment Questionnaire (EMIQ). GAITRite System normal (GR_N data set) and dual-task (GR_D data set) walking; Mobility Lab Gait normal (ML_N data set) and dual-task (ML_D data set) walking; Mobility Lab Romberg stance with open (ML_S_EO data set) and closed eyes (ML_S_EC data set).
Table
Table 2. Range of hyperparameters optimized with grid search.
Table 2. Range of hyperparameters optimized with grid search.
MethodHyperparameterMinMaxStep SizeScale
Decision TreeCriterion: ‘gini’ or ‘entropy’----
Maximum depth2101linear
Minimum samples at a leaf node5301linear
k-Nearest NeighborWeights: ‘uniform’ or ‘distance’----
Distance metric: ‘euclidean’ or ‘manhattan’----
Numbers of neighbors2101linear
Support Vector MachineRegularization C0.110010logarithmic
Kernel coefficient gamma10−5110logarithmic
Table
Table 3. Demographic and clinical characteristics of the patients (N = 1240). First-line therapies subsume the disease-modifying drugs azathioprine, Glatirameracetate, Dimethylfumarat, Interferons and Teriflunomide. Second-line therapies subsume the high-efficacy MS treatments Alemtuzumab, Cladribine, S1P Modulators, Natalizumab, Anti-CD20 treatments.
Table 3. Demographic and clinical characteristics of the patients (N = 1240). First-line therapies subsume the disease-modifying drugs azathioprine, Glatirameracetate, Dimethylfumarat, Interferons and Teriflunomide. Second-line therapies subsume the high-efficacy MS treatments Alemtuzumab, Cladribine, S1P Modulators, Natalizumab, Anti-CD20 treatments.
VariableParticipants, No. (%)
GenderFemale911 (73.5)
Male329 (26.5)
MS subtypeRelapsing-remitting1091 (88.0)
Primary progressive83 (6.7)
Secondary progressive65 (5.2)
MS subtype still unclear1 (0.1)
Immunomodulatory TherapiesFirst-line therapies357 (28.8)
Second-line therapies629 (50.7)
No therapy254 (20.5)
Multiple Sclerosis (MS).
Table
Table 4. Performances of the best classification model per data set on detecting falls. The values are presented as mean ± standard deviation across 5-fold cross-validation repetition. p-value via permutation test.
Table 4. Performances of the best classification model per data set on detecting falls. The values are presented as mean ± standard deviation across 5-fold cross-validation repetition. p-value via permutation test.
Thresh-OldCombination MethodNo. FeaturesF1Recall (%)Precision (%)Specificity (%)Kappap
All data set (n = 929)
GNB10%Intersection90.53 ± 0.0074.0 ± 0.041.0 ± 0.383.1 ± 0.20.43 ± 0.000.001
Basic data set (n = 1237)
GNB100%-110.47 ± 0.0157.6 ± 0.939.4 ± 0.485.7 ± 0.20.36 ± 0.010.001
EDSS data set (n = 1237)
GNB100%-90.48 ± 0.0064.6 ± 0.338.2 ± 0.283.1 ± 0.10.37 ± 0.000.001
GR_N data set (n = 1212)
GNB50%Intersection70.42 ± 0.0045.4 ± 0.638.4 ± 0.388.3 ± 0.10.31 ± 0.000.001
GR_D data set (n = 1213)
GNB25%Intersection40.38 ± 0.0141.0 ± 0.835.7 ± 0.488.5 ± 0.10.28 ± 0.010.001
ML_N data set (n = 1006)
GNB10%Union 280.43 ± 0.0142.9 ± 0.643.1 ± 0.790.8 ± 0.20.34 ± 0.010.001
ML_D data set (n = 971)
GNB25%Intersection40.39 ± 0.0040.7 ± 0.637.4 ± 0.589.2 ± 0.10.29 ± 0.010.001
ML_S_EO data set (n = 1201)
kNN25%Union 320.29 ± 0.0224.8 ± 1.535.0 ± 1.692.7 ± 0.30.20 ± 0.020.001
ML_S_EC data set (n = 1190)
GNB25%Union220.30 ± 0.0123.4 ± 0.940.8 ± 1.394.6 ± 0.20.22 ± 0.010.001
MSWS-12 data set (n = 1227)
GNB25%Union 230.54 ± 0.0065.9 ± 0.045.4 ± 0.187.3 ± 0.00.45 ± 0.000.001
EMIQ data set (n = 1239)
GNB25%Union40.54 ± 0.0161.0 ± 1.448.0 ± 0.589.4 ± 0.20.45 ± 0.010.001
Gaussian Naive Bayes (GNB); k-Nearest Neighbor (kNN). All data set = All features of the following data sets; Basic data set = 25 Foot Walk Test + 2-minute walk test + EDSS; Expanded Disability Status Scale (EDSS); GAITRite System normal (GR_N data set) and dual-task (GR_D data set) walking; Mobility Lab Gait normal (ML_N data set) and dual-task (ML_D data set) walking; Mobility Lab Romberg stance with open (ML_S_EO data set) and closed eyes (ML_S_EC data set); Twelve Item Multiple Sclerosis Walking Scale (MSWS-12); Early Mobility Impairment Questionnaire (EMIQ).
Table
Table 5. The final feature subsets achieve the best performance according to Table 4. The features, highlighted in bold, were chosen as the most relevant features in both the All data set and the individual data sets. The unit of each gait parameter is shown in parentheses. The parameters have been recorded and merged for the left and right sides (L/R). The parameters are presented as mean [mean] or standard deviation [std].
Table 5. The final feature subsets achieve the best performance according to Table 4. The features, highlighted in bold, were chosen as the most relevant features in both the All data set and the individual data sets. The unit of each gait parameter is shown in parentheses. The parameters have been recorded and merged for the left and right sides (L/R). The parameters are presented as mean [mean] or standard deviation [std].
FeaturesUsed No./Total No.
All data set
GNBQ1_EMIQ, Q4_EMIQ, Q8_EMIQ, Q1_MSWS-12, Q2_MSWS-12, Q3_MSWS-12, Q7_MSWS-12, Q11_MSWS-12, Q12_MSWS-129/428
Basic data set
GNBT25-FW, 2MWT, EDSS_Ambulation, EDSS_Bowel_Bladder, EDSS_Brainstem, EDSS_Cerebellar, EDSS_cognition/fatigue, EDSS_Pyramidal, EDSS_Score, EDSS_Sensory, EDSS_Visual11/11
EDSS data set
GNBEDSS_Ambulation, EDSS_Bowel_Bladder, EDSS_Brainstem, EDSS_Cerebellar, EDSS_cognition/fatigue, EDSS_Pyramidal, EDSS_Score, EDSS_Sensory, EDSS_Visual9/9
GR_N data set
GNBStep Count, Step Length (cm) L, Step Length (cm) R, Stride Length (cm) L, Stride Length (cm) R, Stride Velocity Right, Velocity7/82
GR_D data set
GNBStep Extremity (ratio) L, Step Length (cm) L, Step Length (cm) R, Stride Length (cm) R4/82
ML_N data set
GNBLower Limb – Double Support L (%GCT) [mean], Lower Limb – Gait Speed L (m/s) [mean], Lower Limb – Gait Speed R (m/s) [mean], Lower Limb – Single Limb Support L (%GCT) [mean], Lower Limb – Stance R (%GCT) [mean], Lower Limb – Swing R (%GCT) [mean], Lower Limb – Terminal Double Support R (%GCT) [std], Stride Length R (m) [mean]8/84
ML_D data set
GNBLower Limb – Stride Length R (m) [mean], Lower Limb – Terminal Double Support R (%GCT) [mean], Lower Limb – Toe Off Angle L (degrees) [mean], Lower Limb – Toe Off Angle R (degrees) [mean]4/84
ML_S_EO data set
kNNAcc – Jerk (Sagittal) (m2/s5); Angles – 95% Ellipse Axis 1 Radius (degrees)2/32
ML_S_EC data set
GNBAcc – 95% Ellipse Axis 1 Radius (m/s2), Acc – 95% Ellipse Rotation (radians), Acc – Centroidal Frequency (Hz), Acc – Centroidal Frequency (Coronal) (Hz), Acc – Centroidal Frequency (Sagittal) (Hz), Acc – Frequency Dispersi on (AD), Acc – Frequency Dispersion (Coronal) (AD), Acc – Frequency Dispersion (Sagittal) (AD), Acc – Jerk (m2/s5), Acc – Jerk (Coronal) (m2/s5), Acc – Jerk (Sagittal) (m2/s5), Acc – Mean Velocity (Coronal) (m/s), Acc – Path Length (m/s2), Acc – Path Length (Coronal) (m/s2), Acc – Path Length (Sagittal) (m/s2), Acc – Range (m/s2), Acc – Range (Coronal) (m/s2), Acc – RMS Sway (Coronal) (m/s2), Acc – RMS Sway (Sagittal) (m/s2), Angles – 95% Ellipse Axis 1 Radius (degrees), Angles – 95% Ellipse Rotation (radians), Angles – RMS Sway (Coronal) (degrees)22/32
MSWS-12 data set
GNBQ5_MSWS-12, Q8_MSWS-12, Q12_MSWS-123/12
EMIQ data set
GNBQ1_EMIQ, Q4_EMIQ, Q8_EMIQ, Q9_EMIQ4/9
Gait cycle time (GCT); Acceleration (Acc); root mean square (RMS); Q() = number of questions; All data set = All features of the following data sets; Basic data set = 25 Foot Walk Test (T25-FW) + 2-minute walk test (2MWT) + EDSS; Expanded Disability Status Scale (EDSS); GAITRite System normal (GR_N data set) and dual-task (GR_D data set) walking; Mobility Lab Gait normal (ML_N data set) and dual-task (ML_D data set) walking; Mobility Lab Romberg stance with open (ML_S_EO data set) and closed eyes (ML_S_EC data set); Twelve Item Multiple Sclerosis Walking Scale (MSWS-12); Early Mobility Impairment Questionnaire (EMIQ).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Schumann, P.; Scholz, M.; Trentzsch, K.; Jochim, T.; Śliwiński, G.; Malberg, H.; Ziemssen, T. Detection of Fall Risk in Multiple Sclerosis by Gait Analysis—An Innovative Approach Using Feature Selection Ensemble and Machine Learning Algorithms. Brain Sci. 2022, 12, 1477. https://doi.org/10.3390/brainsci12111477

AMA Style

Schumann P, Scholz M, Trentzsch K, Jochim T, Śliwiński G, Malberg H, Ziemssen T. Detection of Fall Risk in Multiple Sclerosis by Gait Analysis—An Innovative Approach Using Feature Selection Ensemble and Machine Learning Algorithms. Brain Sciences. 2022; 12(11):1477. https://doi.org/10.3390/brainsci12111477

Chicago/Turabian Style

Schumann, Paula, Maria Scholz, Katrin Trentzsch, Thurid Jochim, Grzegorz Śliwiński, Hagen Malberg, and Tjalf Ziemssen. 2022. "Detection of Fall Risk in Multiple Sclerosis by Gait Analysis—An Innovative Approach Using Feature Selection Ensemble and Machine Learning Algorithms" Brain Sciences 12, no. 11: 1477. https://doi.org/10.3390/brainsci12111477

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics