Detection of Walking Features Using Mobile Health and Deep Learning

Abstract

This study identifies seven human subjects’ walking features by training a deep learning model with sensor data. Using the proposed Mobile Health Application developed for collecting sensor data from an Android device, we collected data from human subjects with a history of mild traumatic brain injury. The sensors measure acceleration in m/s² with respect to: the X, Y, and Z directions using an accelerometer, the rate of rotation around a spatial axis with a gyroscope, and nine parameters of a rotation vector with rotation vector components along the X, Y, Z axes using a rotation vector software-based sensor. We made a deep learning model using Tensorflow and Keras to identify the walking features of the seven subjects. The data are classified into the following categories: Accelerometer (X, Y, Z); Gyroscope (X, Y, Z); Rotation (X, Y, Z); Rotation vector (nine parameters); and a combination of the preceding categories. Each dataset was then used for training and testing the accuracy of the deep learning model. According to the Keras evaluation function, the deep learning model trained with Rotation vector data shows 99.5% accuracy for classifying walking characteristics of subjects. In addition, the ability of the model to accurately classify the characteristics of subjects’ walking with all datasets combined is 99.9%.

1. Introduction

Because the human body is composed of many different biomechanical systems (e.g., multiple joints, diverse sensorimotor responses, muscular and skeletal interconnectivity, etc.), individuals may have an inherently unstable biomechanical structure. For individuals with a traumatic brain injury (TBI), the brain injury may further exacerbate problems with this unstable biomechanical structure. Mild types of TBI or mild traumatic brain injury (mTBI) also contribute to several body balance problems because unspecified areas in the injured brain may negatively influence the center of gravity (COG) of the human body. In fact, several individuals with even a history of mTBI experience long-term difficulties in sensorimotor functions, including visual function or eye movements and muscle imbalance, which may cause limited mobility and impaired body balance [1,2,3,4,5]. Therefore, it is important to understand the challenges associated with maintaining daily balance, such as walking balance, among those with a history of mTBI. Previous research indicates that mTBIs increase the range of COG [6,7], thus making walking different from that of individuals without mTBI. This COG is related to walking balance [8].

Previous studies in human balance and related health problems have used a variety of measurement tools, such as force plates for postural stability testing [6], motion capture systems for gait analysis [9], visual motor equipment to measure eye movements [10], or body sensors to measure gait patterns [8,11]. However, the major limitation of these tools is that they require affected individuals to visit a clinical or research laboratory for assessment. Therefore, it is not suitable to measure human body balance for mobile health (mHealth).

Mobile health (mHealth) methodologies [11,12] are inexpensive and widely available, and are used for monitoring gait disturbances associated with chronic diseases [13]. Additionally, mHealth is a nice tool for providing real-time assessment and receiving feedback because it works on a smartphone. Real-time assessment and feedback [14] can both detect and prevent further functional deterioration, as well as save affected individuals multiple trips to clinics [15,16].

In this study, smartphone sensors collected data on leisurely walking for subsequent analysis using deep learning. Data for six mTBI subjects and one non-TBI subject were collected using a mHealth application that accesses two hardware sensors and one software sensor on smartphone devices that were worn by the subjects. On-board hardware recorded acceleration in m/s², with the accelerometer recording acceleration in X, Y, and Z directions, and a gyroscope measuring acceleration with respect to rotation around a spatial axis. In addition, nine parameters associated with a rotation vector and rotation vector component along the X, Y, and Z axes were recorded using rotation vector software-based sensors previously installed on the Android devices. The subjects walked about 220-feet round trips. Walking data for the seven subjects were imported into a deep learning model originally built with TensorFlow [17] and Keras [18] in R, and subsequently classified in order to improve accuracy of the deep learning model. Model training with accelerometer data yielded a 50.5% accuracy in classifying subjects’ walking. Deep learning model training with Gyroscope data presented an accuracy of 48.3%. The accuracy of the model trained with the rotation vector component along the X, Y, and Z axis data was 77.1%. Model training with the nine parameters of rotation vector was 99.5%. Model training with all data sets yielded a 99.9% accuracy in predicting the classification of the subject’s walking.

2. Experiment Methods

2.1. Mobile Health Application

The mHealth application was developed to measure and record acceleration data and rotational data in real time using an Android smartphone’s motion sensors [19,20]. Figure 1a shows the mHealth application in use during data collection. The application was developed for use with Android mobile platforms, software development kits (SDKs) greater than 21 [21], using Android Studio, The mHealth application was installed on the Samsung Galaxy S8 with the Android 7.0 mobile operating system.

The smartphone measures rotation and acceleration through an angle around an axis (X, Y, Z) as shown in Figure 1a. The mHealth application collects rotation data and acceleration data in 10 ms intervals using the on-board accelerometer, gyroscope, and rotation vector software-based sensor.

The data are then saved to an SQLite database and CSV (comma-separated values) files stored on the smartphone. The application (Figure 1b) also collects subjects’ background information prior to real-time data collection, including any head, back, or leg pain; prior concussion experience; and gender, race, height, and weight.

2.2. Data Collection

A total of seven subjects were tested and their walking data were collected by the mHealth application.

Table 1. Human subjects’ general information.

ID	Age	Race	Gender	# of mTBI Injuries	Year of Last Injury	Weight	Height
A01	23	Asian	Female	1	2019	127	5′1″
A02	35	Asian	Male	5	2018	148	5′4″
A04	20	Asian	Female	1	2015	117	4′11″
C20	32	White	Female	3	2015	155	5′4″
C21	19	White	Female	4	2019	132	5′8″
C17	21	White	Female	1	2015	115	5′1″
CS03	24	White	Female	0	0	115	5′2″

describes their background information, including age, race, gender, number of mTBI injuries, year of last injury, weight, and height. All of the subjects’ ages ranged from 19 to 35 years. Among six subjects who reported a history of mTBI, three subjects (50%) had more than one mTBI and two of which were injured within the past year. One subject did not have any history of mTBI.

During the experiment, each subject wore a smartphone in the pocket of a waistband located at the center of the front body, as shown in Figure 2a. The smartphone screen was placed facing the direction of travel with its top turned to the right side of the body. The subjects walked along a straight, linear path at a comfortable pace in an indoor hallway of a building. The subjects were also instructed to walk leisurely to an initial destination ca. 220 feet away from their current location and return to the starting location as shown in Figure 2b.

Upon return, each subject walked a total of ca. 440 feet round trip. The mHealth application was activated upon the subject’s touching the “begin” button on the phone screen at the beginning of the trial, and the data collection was terminated in the mHealth application when the subject returned to the starting location.

3. Analysis and Experimental Results

We analyzed subjects’ acceleration in m/s² for the rate of rotations around spatial X, Y, Z axes; the nine parameters of rotation vector; and the rotation vector component along the X, Y, Z axes.

Table 2. Statistical information of A01 subject’s data.

ID	Sensor	Min	Max	Mean	Med	Var	SD	Q1	Q2	Q3	Q4
A01	accX	4.111	17.638	9.459	8.731	4.848	2.202	4.111	7.844	8.731	10.873
	accY	−5.259	5.494	−0.276	−0.309	2.207	1.486	−5.259	−1.286	−0.309	0.692
	accZ	−6.954	7.104	−0.174	0.230	3.122	1.767	−6.954	−1.101	0.230	1.105
	gyroX	−1.185	2.714	0.014	0.042	0.190	0.436	−1.185	−0.315	0.042	0.344
	gyroY	−1.260	1.510	0.002	−0.033	0.094	0.306	−1.260	−0.213	−0.033	0.181
	gyroZ	−1.115	1.229	0.002	−0.008	0.103	0.322	−1.115	−0.209	−0.008	0.200
	rotX	−0.756	0.749	0.033	−0.412	0.260	0.510	−0.756	−0.466	−0.412	0.552
	rotY	−0.600	0.011	−0.493	−0.493	0.004	0.060	−0.600	−0.544	−0.493	−0.444
	rotZ	−0.662	0.672	0.056	−0.393	0.251	0.501	−0.662	−0.433	−0.393	0.567

shows the resulting statistical analysis from one of the subjects in terms of acceleration (accX, accY accZ); the rate of rotations as measured by the on-board gyroscope (gyroX, gyroY, gyro Z); and the rotation vector component assessed by the rotation vector software-based sensor (rotX, rotY, rotZ). Table 2 shows the minimum value, maximum value, mean, median, variance (var), standard deviation (SD), and Quartiles (Q). For example, A01 has a higher acceleration speed downward from its current location (X axis in Figure 2a) than upwards (−X axis in Figure 2a) due to the effect of gravity. In addition, although subjects intended to walk in a straight line (as indicated by the Z axis in Figure 2a), some subjects proceeded to deviate left (−Y axis in Figure 2a) or right (Y axis in Figure 2a) from the target path. All the subjects’ statistical information on walking data are displayed in Appendix A and Appendix B and show a scatter plot of rotation X, Y, Z with the first 30 s of walking data.

Data corresponding to the first 10 s (1000 rows) and the last 10 s (1000 rows) during each subject’s data collection session were removed to account for the start- and stop-button activation within the mHealth application. The data were then analyzed by the deep learning algorithms originally developed within Tensorflow and Keras in R, as shown in Figure 3. The model uses one input layer, one output layer, and three hidden layers. The output layer uses softmax activation [22] function and the inner and hidden layers use the Rectified Linear Units (relu) [23] function for activation. Figure 4 shows the detailed information of each layer. Both the input layer and hidden layers contain 256 arbitrarily decided neurons and the output layer contains 7 neurons for classifying seven subjects. The dropout rate [24] is 50% for both the input and hidden layers to avoid overfitting from the training data set (x_data_train, y_data_train). For initializing the weight of neurons, the model uses the Glorot normal initializer (also called Xavier normal initializer) [25].

Figure 3b shows the compilation of the deep learning model. The model is compiled with the categorical cross entropy loss function [17,18] to classify seven categories, and the Adam algorithm [26] to find a minimized cost of the model. For training of the model, we used the ID of subjects as a label (y_data) and the time series data (accX, accY, accZ, gyroX, gyroY, gyroZ, rotX, rotY, rotZ, and the nine parameters for the rotation vector) as input data (x_data). The ratio of input (x_data_train)–output (y_data_train) training data set, to input (x_data_test)–output (y_data_test) test data set is seven to three. So, approximately 100,000 input rows of data (x_data_train)–output (y_data_train) are used for training and approximately 43,000 input rows of data (x_data_test)–output (y_data_test) are used to test the model. Model training used the training data set (x_data_train, y_data_train). The training data set was used 300 times (epoch) to train the deep learning model as shown in Figure 3c and Figure 5 shows the loss and accuracy of the model during training, where X-axis of the graph represents epoch time and Y-axis represents a loss value by optimizer and accuracy of evaluation. Improving the accuracy of the deep learning model was facilitated with five different combinations of data sets including acceleration (Acc X, Y, Z) in the X, Y, and Z directions for the data set corresponding to the gyroscope (data set titled, Gyro X, Y, Z); the rotation vector component along the X, Y, Z axis (Rot X, Y, Z) data set; the nine parameters of the rotation vector (Rot vector—9); and the previously stated datasets combined. The trained deep learning model used Acc data (Figure 5a) and Gyro data (Figure 5b) to gradually minimize loss and increased the accuracy of the model up to 50 epochs. However, models of more than 100 epochs experienced few significant gains in terms of greater accuracy and minimization of loss.

The deep learning model trained with the Rot data (Figure 5c), the Rotation vector data (Figure 5d) and the dataset composed of all existing data sets combined (Figure 5e), each experienced significant decreases in the loss and similarly significant increases in model accuracy. Rotation vector (Figure 5d) and the combination of all data sets (Figure 5e) achieved up to 100% accuracy during the training of the deep learning model. Accuracy for the Rot (Figure 5c) increased to approximately 80% during model training. Among the multiple data sets, the deep learning model presented the highest performance when trained with the Rot XYZ (Figure 5d) and with all datasets combined (Figure 5e).

Outcomes associated with model training were evaluated with the test data set (x_data_test, y_data_test) using Keras code (Figure 3d).

Table 3. Loss and accuracy of the model.

Model Data Type	Loss	Accuracy
Acc X, Y, Z	1.291	0.505 (50.5%)
Gyro X, Y, Z	1.316	0.483 (48.3%)
Rot X, Y, Z	0.540	0.771 (77.1%)
Rot vector—9	0.014	0.995 (99.5%)
All data set	0.010	0.999 (99.9%)

shows the result of this evaluation. Model training with acceleration (Acc X, Y, Z) in X, Y, and Z directions data sets indicates a 50.5% accuracy in classifying subjects’ walking characteristics using the test data (x_data_test). The model trained with Gyroscope data (Gyro X, Y, Z) presents an accuracy rate of 48.3%. The model trained with the rotation vector component along the X, Y, Z axis data set (Rot X, Y, Z) reveals 77.1 % accuracy. Rot XYZ is calculated from the nine parameters associated with the rotational vector. Thus, the trained model with Rot XYZ presents a higher accuracy than the accuracies derived from the accelerometer-related, and gyroscope-related datasets. The model trained with the nine parameters of rotation vector (Rot vector—9) displays accuracy of 99.5%. The model trained with all datasets shows 99.9% accuracy in matching the subject’s unique walking features with the corresponding correct subject (y_data_test) using test data (x_data_test). According to evaluation of the model, using nine parameters of rotation vector data was most effective in training the deep learning model for classifying the walking characteristics of the seven subjects.

4. Conclusions and Future Work

We analyzed the walking characteristics of six mTBI subjects and one non-mTBI subject to identify their walking features by training the deep learning model with multiple data sets. We collected the walking data from the subjects using the mHealth application originally developed for measuring acceleration in m/s² in the X, Y, and Z directions from the accelerometer; the rate of rotations around a spatial axis of gyroscope hardware; nine parameters of rotation vectors; and rotation vector components along the X, Y, Z axis using rotation vector software-based sensor on the Android device. The sensor data were subsequently used to train a deep learning model developed with Tensorflow and Keras in R. According to model evaluations, using nine parameters of rotational vectors was the most effective dataset in training the deep learning model to accurately identify the walking characteristics of the seven subjects (95.5% accuracy).

With clinical data of mTBI and non-mTBI subjects, we will make a deep learning model that can predict (or be matched with) different levels of post-injury symptom severity on the Glasgow coma scale [27]. It is our hope that this model and the mHealth application together will contribute to the early detection of acute decline in sensorimotor function after mTBI based on the patient walking data. Additionally, the mHealth application will be useful to examine and screen the clinical symptom progress during post-mTBI recovery [28].