Smartphone Motion Mode Recognition ^†

Abstract

The possibility of using mobile devices, such as smartphones, for locating a person indoor is becoming more attractive for many applications. Among them are health care and safety services, commercial and emergency applications. One of the approaches to find the smartphone position is known as Pedestrian Dead Reckoning (PDR). PDR relies on the smartphone low-cost sensors, such as accelerometers, gyroscopes, barometer and magnetometers. An appropriate calibration phase to find the step length algorithm gains is required before PDR can be applied. These gains are very sensitive to the user and smartphone mode. In this research, we employ machine learning classifications algorithms in order to recognize and classify the pedestrian and smartphone modes. A methodology of training on a single user and testing on multiple users is proposed and experimentally evaluated Results show successes in classifying the user and smart phone modes.

1. Introduction

The possibility of using mobile devices (such as smartphones) for locating a person is becoming more and more attractive for many applications. Among them are health care services, commercial applications, emergency applications and safety services as [1]. While in outdoors, the positioning of a person by its smartphone is usually based on Global Navigation Satellite Systems (GNSS) [2]. However, in indoor environments the availability of satellite signals cannot be guaranteed and GNSS based services can be highly degraded or totally denied.

In such situations, one of the approaches to find the position of the smartphone is known as Pedestrian Dead Reckoning (PDR) [3,4]. PDR may rely on the smartphone low-cost sensors such as accelerometers, gyroscopes and magnetometers. In general, PDR uses the accelerometers to detect the pedestrian steps and then estimate the step length. Next, the heading is obtained from the gyroscopes and/or magnetometer. Given the pedestrian initial conditions and by using the current heading and step length size, the current pedestrian position can be found. An appropriate calibration phase to find the step length algorithm gains is required before PDR can be applied. These gains are very sensitive to the user and smartphone mode as recent papers such as [5], showed that by recognizing the mode of the smartphone (handheld, in a pocket, texting and etc) and/or the pedestrian (walking, running, elevator and etc.) [6] (and an comprehensive survey paper [7]) the accuracy of PDR algorithms can greatly be improved.

In this research, we employ machine learning classifications algorithms in order to recognize and classify the smartphone modes. A methodology of training on a single user and testing on multiple users is proposed and experimentally evaluated Results show successes in classifying the user and smart phone modes.

The rest of the paper is organized as follows: Section 2 describes the methodology and strategy used for the mode recognition process. Section 3 presents the experimental setup and results and Section 4 gives the conclusions.

2. Methodology

The overview of the classification process for mode recognition is illustrated in Figure 1. In the data acquisition phase, the data required for the training and prediction steps is collected using the smartphone sensors. In the classification phase, the data is being preprocessed (noise reduction, outliers rejection and etc.) and relevant features are extracted. Utilizing those features a classification model is chosen after processing the training data. The classification model is then used on the collected test data to perform mode recognition.

2.1. Strategy

In this research we use a single user with a single phone for data collection required in the training process. Of course, collecting data from multiple users and multiple phones would probably make the classifier more robust, yet we focus here on single phone and single user data collection. The collected data is based on the accelerometers and gyros raw data. Other smartphone sensors such as magnetometer, barometer, light sensor, sound-meter and etc are not used. The accelerometer and gyro raw data are a vector of specific force and a vector of the angular velocity, respectively. Given these vectors, their magnitude is calculated. Features are extracted based solely on the magnitudes of the specific force and angular velocity vectors. The specific force and angular rate vector components where not used because they are sensitivity to the smartphone orientation in the person hand or pocket.

The magnitude based features are used in the training process which outputs the best classifier to be used in the prediction phase. The prediction phase input is a set of collected data from multiple users and multiple phones. The output of the prediction step is a measure of the accuracy of the chosen classifier to recognize the user and smartphone mode. This research strategy is illustrated in Figure 2.

The data was collected during four smartphone modes: (1) pocket; (2) swing; (3) texting and (4) talking while the user is walking in normal or fast walking speed.

2.2. Smartphone Sensors

We use only the inertial sensors of the smartphone, that is accelerometers and gyros for our analysis. The three-orthogonal accelerometers measures the specific force vector

$$\mathbf{f}={\left[f_x{f}_y{f}_z\right]}^T$$

(1)

and the three-orthogonal gyroscopes measure the angular rate vector

$$\mathbf{\omega }={\left[{\mathbf{\omega }}_x{\mathbf{\omega }}_y{\mathbf{\omega }}_z\right]}^T$$

(2)

both without an external reference [8,9]. The smartphone accelerometers and gyros are Micro-Electrical-Micro-Mechanical (MEMS) based sensors.

Loosely speaking, the basic working principle of a MEMS accelerometer is described using a proof mass [2,10]. Consider, a proof mass which is free to move with respect to the accelerometer case along the accelerometer’s sensitive axis, restrained by springs. When an accelerating force along the sensitive axis is applied to the case, it will move with respect to the mass until the acceleration of the mass due to the asymmetric forces exerted by the springs matches the acceleration of the case due to the externally applied force. The resultant position of the mass with respect to the case is proportional to the acceleration applied to the case. Thus, by measuring the position of the mass, the applied acceleration is found.

MEMS gyros working principle can be described using a vibratory beam [2,10]. Consider, a vibratory beam element that is driven to undergo simple harmonic motion. Application of angular rate perpendicular to the motion of the beam gives rise to Coriolis acceleration along the axis perpendicular to both the driven vibration and the projection of the angular rate vector. Measuring the Coriolis acceleration enables the extraction of the applied angular rate.

2.3. Feature Extraction

On each working window (as will be defined in the following section) two types of features are used: (1) statistical features and (2) time-domain features. All features were calculated on the magnitude of the specific force vector

$$f_m=\sqrt{f_x^2+f_y^2+f_z^2}$$

(3)

and the magnitude of the angular rate

$${\omega }_m=\sqrt{{\omega }_x^2+{\omega }_y^2+{\omega }_z^2}$$

(4)

2.3.1. Statistical Features

• Mean. The mean of a signal.
• Median. The median is the middle value separating the higher half of a data sample from the lower half.
• Standard deviation. The square root of the variance (measure of the spread of data around the mean).
• Average absolute difference. Measure of the spread of data around its mean, taking the absolute difference between values and the mean.
• Interquartile range (iqr). It is the difference between 75th percentile and 25th percentile of the data where percentile of $Y\%$ is the value separating the higher 100-$Y\%$ of a data sample from the lower $Y\%$ of the data.
• Skewness. A measure of the asymmetry of the probability distribution of a signal.
• Kurtosis. A measure of the ‘tailedness’ of the probability distribution of a signal.
• Signal energy. The sum of the squares of signal values.
• Signal magnitude area. The sum of absolute values of a signal.
• Max. The maximum value in the window of the signal.
• Min. The minimum value in the window of the signal.
• Amplitude. The absolute difference between the maximum value and minimum value.

2.3.2. Time-Domain Features

• Number of peaks. The count of the number of maximum points within the desired window of the signal where the maximum points should be above a predefined value and located after w samples from the last maximum point.

2.3.3. Cross Sensor Features

• Gyro-Accelerometer Correlation. Is the cross-correlation coefficient between the gyro and acceleration sensors.
• Gyro-Accelerometer Maximum. The multiplication result of the gyro and acceleration maximum values.
• Gyro-Accelerometer Standard Deviation. The multiplication result of the gyro and acceleration standard deviation values.

3. Experimental Results and Discussion

3.1. Experimental Setup

The acceleration and gyro data was collected in a sampling rate of 50 Hz. After the collection the corresponding magnitudes were evaluated. An outliers rejection algorithm was applied to remove samples which are over 3 standard deviations from the signal. On the remaining data a sliding window with length of 128 samples ($2.5$ s) was applied with an overlapping of 127 samples.

The data was collected from a single smartphone and from a single user during four smartphone modes: (1) pocket; (2) swing; (3) texting and (4) talking while the user is walking in normal or fast walking speed. The total number of windows in each mode is presented in

Table 1. Number of windows used for each smartphone mode

Mode	Number of Windows Training	Number of Windows Test
Pocket	38,492	6320
Swing	60,812	17,523
Talking	27,697	7348
Texting	27,434	13,655

for the training and test datasets. The test database was collected from six persons - five men and one woman with different smartphones.

3.2. The Learning Process

We examined four types of machine-learning classifying algorithms [11,12]: (1) multi-class Support vector machine (SVM); (2) Random Forest (RF); (3) K-nearest neighbor (KNN) and (4) Multi-layer Perceptron classier (MLP). Accuracy was chosen as the performance measure in the presented analysis. Accuracy is the measure of the proportion of all cases which have been correctly classified out from the total cases. The accuracy of each classifier on the test dataset is given in

Table 2. Accuracy Results

Classifier	Accuracy (%)
MLP	86.2
SVM	84.1
KNN	82.7
RF	86.7

. All classifier obtained an accuracy above $82.5\%$ in particular RF achieved an accuracy of $86.7\%$.

Focusing on RF, the confusion matrix is illustrated in Figure 3. There, each column is the predicted motion mode as labeled at the bottom of the column while the true mode is labeled at the beginning of each row. The taking mode was best recognized with $96\%$ accuracy while the swing mode was the worst recognized with $79\%$ accuracy. It appears that the most challenging case was to distinguish the swing mode from texting and pocket modes.

Feature importance, based on RF classier, is shown in Figure 4. The most dominant feature is the gyro amplitude (ampgyro) followed by accelerometer amplitude (ampforce) and the amplitude multiplication between the two (AmpAccGyro). Other dominant features where the signal energy of the gyro (enygyro) and acceleromter (enygforce) and the number of peaks of the gyro and accelerometer (peaksgyro, peaksgforce).

4. Conclusions

In this research we proposed and demonstrated a methodology for recognizing smartphone mode. Experimental results showed an accuracy of $86.7\%$ in the mode recognition process. Such a mode recognition approach can help improve the performance of PDR algorithms.