Machines Perceive Emotions: Identifying Affective States from Human Gait Using On-Body Smart Devices

Abstract

Emotions are a crucial part of our daily lives, and they are defined as an organism’s complex reaction to significant objects or events, which include subjective and physiological components. Human emotion recognition has a variety of commercial applications, including intelligent automobile systems, affect-sensitive systems for customer service and contact centres, and the entertainment sector. In this work, we present a novel deep neural network of the Convolutional Neural Network - Bidirectional Gated Recurrent Unit (CNN-RNN) that can classify six basic emotions with an accuracy of above 95%. The deep model was trained on human gait data captured with body-mounted inertial sensors. We also proposed a reduction in the input space by utilizing 1D magnitudes of 3D accelerations and 3D angular velocities ($\widehat{ma{g}_a}$, $\widehat{ma{g}_{\omega }}$), which not only minimizes the computational complexity but also yields better classification accuracies. We compared the performance of the proposed model with existing methodologies and observed that the model outperforms the state-of-the-art.

1. Introduction

Humans have access to a wide range of verbal and nonverbal cues from nature with which to communicate their emotions. Our emotions greatly influence our social relationships. Emotions are complex mental states that comprise subjective experience, pathological and physiologic responses, and other factors [1,2]. They are consistent responses to events based on some stimuli, and the intensity of response varies from human to human. Human Emotion Recognition (HER) has several exciting application areas such as human–computer interaction to better understand and respond to human-centric events, affect-sensitive systems for autism therapy, chat-bots in call centres, in the entertainment industry, and so on. Such applications can potentially alter the way we interact with computers in our daily lives [3,4,5].

Humans can express emotions both verbally and non-verbally. The expression of verbal emotions is limited and typically indicated by known words, idioms, or phrases, whereas nonverbal emotions can be shown more diversely. Nonverbal emotions can be revealed through physiological indications, facial expressions, speech, gestures, etc. For many decades, psychologists and scientists have extensively explored the field of human sentiment analysis to understand human emotions [6,7,8,9]. On the one hand, psychologists probe human emotions from the perspective of social psychology as “emotions naturally emerge out of the core principles of evolutionary psychology” [10]. However, scientists from other fields, such as the medical sciences, hardware design engineering, computer science, etc., investigate emotions to comprehend how the brain functions, to create new, improved hardware, and to create intelligent systems to assess, forecast, and identify human affective states.

With the rapid advancement of consumer electronics, it is now feasible to capture human emotions using various technologies (including smart devices). This includes, but is not limited to, vision sensors to capture facial expressions or record human gait in the form of videos [11,12], speech-based emotions recognition [13,14,15], electroencephalograms (EEG) to record the electrical activity of the brain [16,17], and wearable inertial sensors to capture human gait data under different affective states [16]. Facial expressions, being the most prominent artefact of intimate emotions, have been extensively explored [18]. Although facial expression-based emotions prediction has improved significantly over time, environment and occlusion-invariant facial expression-based emotions recognition systems are still very challenging, especially given the COVID-19 situation where it is recommended to wear face masks. Moreover, recording facial data raises serious privacy concerns and hence cannot be used in practical applications. Furthermore, facial expressions are not trustworthy indicators of emotion since they may be manipulated and are not generated as a result of a particular feeling [19].

Wearable inertial sensors, such as smart fitness bands, smartwatches, smart in-soles, etc., have turned into a key enabling technology for a wide variety of applications. They are low cost, do not impose any environmental constraints, and have been widely used in human gait analysis, healthcare and health monitoring, and sports activities [20,21,22,23,24]. The inertial gait data collected via wearable gadgets have been analysed to better understand human motion from a variety of perspectives such as estimating soft-biometrics such as gender, age, and height [25,26], predicting emotions [19,27,28], classification of terrains and ground surfaces [29], human activity recognition [30,31], cognitive impairment and fall detection [32,33,34], and even person re-identification [35,36].

Body posture and movement are the most independent forms of expressing human emotions, as opposed to facial expressions, which may be altered and hence faked. Much research suggests that the kinematics of walking can convey emotional states [19,37,38,39,40,41]. However, most prior research either deals with a restricted range of emotions or has marginally poor classification accuracies and hence cannot be employed in practical applications [27,42,43].

The work at hand presents a CNN-RNN-based deep learning model, which can predict emotions from human gait using low-level kinematic signals. To train the model, we used an inertial-based emotions dataset collected in one of our previous studies [19]. An autobiographical memories paradigm was employed for emotional stimuli to collect the emotional data. A chest-mounted smartphone with on-board inertial sensors (tri-axial accelerometer and tri-axial gyroscope) was used to capture gait data from 40 healthy volunteers for six fundamental human emotions, including anger, happy, disgust, sad, fear, and surprise. Subjects were instructed to recollect a prior significant incident and walk straight twice for 10-meters back and forth for each emotion. This resulted in a 40-m walk for each mood.

During the design of the deep model, a key factor was reducing the model’s complexity to speed up the learning process. To this end, we proposed the computation of the magnitude of 3D accelerations ($\widehat{ma{g}_a}$) from the raw signals in the pre-processing step. This reduces the input space from 3D to 1D. Similarly, we compute the magnitude of 3D angular velocities ($\widehat{ma{g}_{\omega }}$) from the raw signals, which reduces the input space from 3D to 1D. The proposed model was separately trained and validated with $\widehat{ma{g}_a}$ and $\widehat{ma{g}_{\omega }}$. For the sake of comparison, the proposed model was validated with a 3D input space (3D accelerations and 3D angular velocities) as well as a 6D input space (6D accelerations and angular velocities). During experimentation, it was observed that the reduction of input size to 1D ($\widehat{ma{g}_a}$ or $\widehat{ma{g}_{\omega }}$) significantly reduces the computation time as compared to 3D or 6D input sizes. Furthermore, the proposed methodology also outperforms existing state-of-the-art approaches. The following are the significant contributions of this work:

• We present a CNN-RNN deep architecture (inspired by InceptionResNet CNN and BiGRU models) where we proposed a novel approach of dense connections employing 1 × 1 convolutions from raw input to the BiGRU model (see Section 4);
• We proposed a reduction in the input space by utilising 1D magnitudes of 3D accelerations and 3D angular velocities ($\widehat{ma{g}_a}$, $\widehat{ma{g}_{\omega }}$), which result in lower processing requirements as well as better results (Section 4 and Section 5);
• The proposed deep model was trained and evaluated with different input sizes (1D–6D) for the sake of comparison and the best results are reported (Section 5);
• The proposed model was compared with the existing approaches and it was observed that the proposed model outperforms the state-of-the-art approaches (Section 6.1).

The rest of the article is organised as follows: Section 2 discusses related literature, Section 3 describes the dataset used in this study; Section 4 explains the methodology including pre-processing, the proposed model, the training procedure, and the model’s complexity; Section 5 presents the results computed with different input sizes, and Section 6 concludes the article.

2. Literature Review

Emotion recognition based on inertial sensor data and machine or deep learning entails collecting data on bodily movements and gestures associated with various emotions using sensors such as accelerometers and gyroscopes. These data are subsequently analysed by AI systems to precisely detect an individual’s emotional state.

There are various advantages of using inertial sensors and AI for emotion recognition over traditional approaches such as facial expression analysis or self-reporting. They can give a more objective and accurate assessment of emotions, even when people are attempting to conceal their emotions.

Accurate assessment of the emotional state of a user is vital for improving machine–user interactions. Smart wearable devices are the right choice for emotional estimation because of ease of use, mobility, privacy, and the ubiquity of IMUs in these devices. In this context, Zhang et al. [27] investigated the use of smart wristbands with built-in accelerometers to identify three human emotions: neutral, angry, and happy. A population of 123 subjects was used to collect emotional data and traditional machine-learning classifiers were used for training and validation. They reported the best classification accuracy of 81.2%. The dataset proposed in this study includes only the data of Chinese subjects and data on only three emotions were collected.

Piskioulis et al. [37] described a method for emotion recognition based on non-intrusive mobile sensing employing accelerometer and gyroscope sensor data supplied by cellphones. To monitor the players’ sensor data while playing the game, an Android OS sensor log app was employed, and a prototype gaming software was created. Data from 40 participants were analysed and used to train classifiers for two emotions: happiness and impatience. The validation investigation found that the accuracy for joy is 87.90% and the accuracy for frustration is 89.45%. This study only deals with two types of emotions.

Reyana et al. [38] designed a system for emotion recognition based on physiological inputs. Body sensors such as muscle pressure sensors, heartbeat sensors, accelerometers, and capacitive sensors were used to gather signals. There were four classes: happy, sad, furious, and neutral. The suggested approach achieved the following percentages of accuracy for emotional states: happy 80%, sad 70%, angry 90%, and neutral 100%. The study considered only three basic emotions.

Quiroz et al. [42] performed emotion recognition based on smart bands inertial sensors data. The dataset collected was based on 50 volunteers. The volunteers were given audio-visual stimuli to enter a state and then were requested to walk. The presented study only dealt with two classes—happy and sad. The accuracy achieved was 75%.

Gravina and Li used sensor and feature-level fusion techniques based on body-worn inertial sensors attached to users’ wrists and a pressure detection module put on the seat to construct a system that identifies and monitors in-seat actions [44]. They concentrated on four major emotional activities: frustration, sadness, curiosity, and happiness. Their findings revealed that integrating temporal and frequency-domain data from all sensors resulted in outstanding classification accuracies (about 99.0%).

A complete system for ambient assisted living was proposed by Costa et al. [8] to alter the environment of the living based on the emotions of the elderly. They developed a smart band to measure electrodermal activity, temperature, heart rate, and mobility. They proposed a neural network consisting of fully connected layers which were trained on 18 features to classify eight different emotions, i.e., happiness, contempt, anger, disgust, fear, sadness, neutral, and surprise. They reported 20% of misclassification cases. The study proposed collecting data on eight emotions but made use of multiple sensors including heart rate and temperature sensors which are usually not available in smart-wearable devices.

The study by Cui et al. [43] employed a smartphone’s onboard tri-axial accelerometer attached to the wrist and ankle of the subjects to collect data for three emotions: happy, neutral, and anger. They extracted 114 features using principal component analysis (PCA). Different classical machine learning classifiers were trained and they found that the support vector machine (SVM) performed best. They also provided a comparison of emotional identification using wrist versus ankle data. They reported the best classification accuracy of 85% and also showed that the emotions prediction from the ankle sensor is much higher than from the wrist sensor.

Hashmi et al. [19] collected emotion data from smartphones’ onboard IMUs for six different emotions: anger, sad, happy, fear, disgust, and surprise. They demonstrated a manually-built feature set that comprised elements from the time, frequency, and wavelet domains. They concluded that time, frequency, and wavelet domain characteristics may be employed to train various supervised learning models. The best classification accuracy for all six emotions is stated to be 86.45%.

In the majority of the studies above, the number of classes is limited excluding the study [19] which deals with the case of classifying six activities.

3. Emotions Dataset

The current study was an extended work of one of our previous studies [19], where extensive data collection sessions were conducted to record emotional data. A total of 40 healthy subjects participated in the data collection session (male-to-female ratio of 26:14, an average age of 25.2 ± 5.9 (years), an average height of 171.6 ± 8.4 (cm)). A chest-mounted smartphone with a built-in tri-axial accelerometer and tri-axial gyroscope was used to collect inertial data. Six basic emotions were selected as proposed by Ekman and Friesen [45] and include “happy”, “fear”, “sad”, “disgust”, “anger” and “surprise”. For emotional stimuli, an autobiographical memories paradigm was employed, where the subjects were asked to recall a past relevant emotional event before the start of each trial. The subjects walked naturally and no special instructions were given in this regard. The participants recalled a relevant event for each emotion and walked straight for 40 m. The total number of samples in the dataset is 839,800. The class distribution of different emotions is shown in Figure 1a. It can be observed from the class distribution that the dataset is not fully balanced. Figure 2 depicts the data collection process.

Hot vs. Cold Emotions Mapping

A hot emotion is associated with a high degree of interest, passion, or activity along with a high level of arousal, from the biological context, when dopamine—a type of neurotransmitter—is released and blood pumps through the muscles. Hot emotions make us feel hotter and more active, and our skin may get red, exacerbating the feeling and look of a higher temperature. On the contrary, when the arousal level is low, humans experience cold emotions. They could have a more cognitive experience than an emotional experience. While cold, we may remain calm and in control of ourselves. We might also be bored, flat, or sluggish [46,47,48].

We categorized the emotional data into hot and cold emotion classes resulting in a binary classification problem.

Table 1. Mapping of six basic emotions to hot and cold classes.

Category	Emotions
Hot Emotions	Surprise, Anger, Happy
Cold Emotions	Fear, Sad, Disgust

maps all six emotions into two classes. The hot emotions include angry, surprise, happy, whereas the cold emotions include fear, sad, disgust. Fear is an emotion which is associated with both high arousal levels (e.g., a sudden encounter with a harmful object) and low arousal levels (e.g., a concern/worry of being unable to achieve a goal). Since an autobiographical memories paradigm was employed in the data collection, we have placed the fear emotion under the cold category. The class distribution for generated hot and cold classes is presented in Figure 1b (hot: 46.3%, cold: 53.7%).

4. Methodology

In this section, we will discuss the proposed methodology. We will start with thepPre-processing (Section 4.1) where we describe the computation of the magnitudes of 3D accelerations and 3D angular velocities. Then, the proposed deep architecture is explained (Section 4.2). Afterwards, the model training process is described (Section 4.3). The complete flow of the suggested approach is depicted in Figure 3.

4.1. Pre-Processing

A critical goal of the proposed work was minimizing computational complexity, making it more suitable for Internet of Things (IoT) applications and real-time systems. One way to minimize the system’s complexity is to reduce the input size [49]. The main advantage of 1D magnitudes is the removal of the sensor’s orientation effect. Such an effect is noticeable when the sensor is attached to a joint with higher degrees of freedom, e.g., a wrist-mounted smartwatch or smart band. In this context, we proposed computing a 1D magnitude from 3D accelerations and 3D angular velocities, i.e., reducing the input size from 3D to 1D. The magnitude of the three-dimensional accelerations was calculated as follows. Let $a_x$, $a_y$, $a_z$ be 3D accelerations, then the magnitude of the 3D accelerations, $\widehat{ma{g}_a}$, is given as:

$$\begin{array}{c}\hfill \widehat{ma{g}_a}=\sqrt{(a_x^2+a_y^2+a_z^2)}-9.8.\end{array}$$

(1)

The gravitational constant, 9.8, was subtracted from the magnitude to remove the effect of gravity.

Similarly, let ${\omega }_x$, ${\omega }_y$, ${\omega }_z$ be 3D angular velocities, then the magnitude of the 3D angular velocities, $\widehat{ma{g}_{\omega }}$, is given as:

$$\begin{array}{c}\hfill \widehat{ma{g}_{\omega }}=\sqrt{({\omega }_x^2+{\omega }_y^2+{\omega }_z^2)}.\end{array}$$

(2)

The $\widehat{ma{g}_a}$ and $\widehat{ma{g}_{\omega }}$ were separately fed into the deep learning model for training and validation.

4.2. The Architecture

The proposed architecture for classifying human emotions consists of the Convolutional Neural Network – Recurrent Neural Network (CNN-RNN) model. The CNN-RNN models were first proposed for image-captioning applications [50], where the CNN models are used to extract visual features from images followed by the recurrent units such as the gated recurrent unit (GRU) [51] or long short term memory (LSTM) [52] to generate captions. The model we proposed comprises two InceptionResNet [53] inspired CNN modules built on a bidirectional RNN [54], consisting of a bidirectional-GRU with 128 units (see Figure 3). The feature map created by the CNN module was concatenated with the raw signals through a connection of a 1 × 1 convolutional layer. To keep the spatial dimensions identical, we used 1 × 1 convolution operations. A total of 64, 1 × 1 kernels were applied to the raw sensor data. This concatenation was inspired by DenseNet [55], which is an image classification model. The benefit of InceptionNet [56] is that it has variable-sized kernels that enable it to detect features of different sizes. Similarly, the ResNet [57], or having a residual connection, lets the model learn the identity; thus, the model is capable of bypassing a module if it does not contribute to classification. For the sequential data, RNNs perform better; therefore, the feature map generated by the CNN modules is given to the bidirectional GRU. The motivation behind introducing the dense connection was that, if CNNs are of no benefit, the raw sensor data are still processed by RNNs resulting in a better classification accuracy. To the best of our knowledge, this type of dense connection, where a feature map generated by CNNs is concatenated with the raw signals, has never before been experimented with for any kind of classification problem. In our experimentation, we found that this significantly improves the performance of the entire model. Figure 3 shows the overall proposed architecture and presents the InceptionResNet modules that are incorporated into the model.

The proposed model has five different kernels, i.e., 1 × 1, 1 × 3, 1 × 5, 1 × 7, and 1 × 9. The input is convolved with 64, 1 × 1 kernels before applying 10 kernels of sizes 1 × 3, 1 × 5, 1 × 7, and 1 × 9. This reduces the overall complexity of the model as pointed out in the existing literature [58]. We tested it by adding different-sized kernels sequentially, which improves the performance significantly. The selection of the number of kernels and their spatial dimensions were based on empirical results (see Section 5.3). Moreover, the addition of a max-pool also improves performance. The feature map generated by the kernels and max-pool was then concatenated. The final output was then passed to 64 kernels of 1 × 1 convolutions to change the spatial dimensions. The input feature map was also passed through 64, 1 × 1 kernels, and feature maps generated from both were then added. The overall architecture has three such modules in it, which were selected by empirical analysis.

The batch normalization and dropout were set to 0.25, and were applied after each module and to the feature map generated by the Bi-GRU units. This significantly reduces the training time. The feature vector generated by Bi-GRU was then reshaped and flattened before passing it to a fully connected layer of 512 neurons. The number of neurons in the dense layer was also selected empirically. A dropout of 0.25 was then applied for the sake of regularization.

4.3. Training Process

For all of the investigations, Google Colaboratory’s runtime GPUs were used. The experiment was carried out entirely with Python using Keras 2.4.3 and Tensorflow 2.4.0 to build the model. Seaborn and Sklearn were used to visualize and compute the confusion matrices and performance reports. A 10-fold cross-validation technique was employed for validation. The batch size was set to 64, and the epoch count was set at 120. The number of epochs was determined empirically by observing the accuracy and loss graphs for training and testing. To minimize loss, the neural network weights and the learning rate should be adjusted. We learned how to adjust the weights or learning rate of our neural network using an optimizer. We experimented with several optimizers, including “Adam”, “Momentum”, “RMSprob”, and others, and found that “Adam” performed the best in comparison with another optimizer.

5. Results

We trained, validated, and tested the aforementioned deep model with the emotions dataset collected in one of our previous studies (as explained in Section 3). The results are presented in the form of confusion matrices, accuracy, precision, recall, and F1-score. Since the data are unbalanced, the accuracy becomes a less accurate performance measure so the remaining assessment parameters were computed to realistically analyze the model’s behaviour. Researchers from the domain of gait analysis also emphasize and strongly advocate using these assessment parameters for the evaluation of unbalanced datasets [31,59,60,61,62,63]. A 10-fold cross-validation technique was employed for validation and was used to evaluate the models; the best results are reported.

The results section is divided into five subsections, namely multi-emotion classification, hot vs. cold emotion classification, the effect of segmentation window size (Section 5.3), accuracy and loss curves (Section 5.4), and complexity of the model (Section 5.5).

5.1. Multi-Emotion Classification

This subsection discusses the multi-emotion classification results computed from the Magnitude of 3D accelerations (Section 5.1.1), Magnitude of 3D angular velocities (Section 5.1.2), 3D accelerations (Section 5.1.3), 3D angular velocities (Section 5.1.4), and 6D accelerations and angular velocities (Section 5.1.5). The results are reported as confusion matrices, F1-Score, precision, and recall.

5.1.1. Magnitude of 3D Accelerations

The main objective of this research activity was the classification of emotions using a deep model with reduced complexity. This was achieved by reducing the input feature space by computing the 1D magnitude of 3D accelerations. Figure 4a shows the confusion matrix computed with a 1D magnitude of 3D accelerations. The average classification accuracy of 95.23% was achieved for six basic emotions. Furthermore, the F1-score, recall, and precision remained at 95% (Figure 5). A significant improvement of 8.5% in classification accuracy was observed in comparison with the results reported by Hashami et al., [19], where the best accuracy of 86.45% was reported.

5.1.2. Magnitude of 3D Angular Velocities

The performance with the magnitude of 3D angular velocities is not comparable with any of the other cases, especially the previous case, i.e., the magnitude of 3D accelerations which has the same computational complexity. The average classification accuracy achieved for this case was 50.15% as shown in

Table 2. Performance comparison between different input sizes used in the proposed model. The best results are produced with a magnitude of 3D accelerations. All of the values are in percentage (%).

Cases	Accuracy	F1-Score	Precision	Recall
$\widehat{ma{g}_a^w}$ (1D)	95.23	95	95	95
$\widehat{ma{g}_{\omega }^w}$ (1D)	50.15	50	52	50
($a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$) (6D)	95.01	95	95	95
($a_x^w,a_y^w,a_z^w)$ (3D)	94.56	95	95	95
(${\omega }_x^w,{\omega }_y^w,{\omega }_z^w)$ (3D)	92.58	92	94	93

, and the confusion matrix is given in Figure 4b. The F1 scores for sad disgust, fear, happy, surprise, and anger remained at 63%, 51%, 38%, 48%, 50%, and 55%, respectively. A comparison with other cases is shown in Figure 5.

5.1.3. 3D Accelerations

The performance for this case is comparable with the first case, i.e., the magnitude of 3D accelerations; however, the complexity of the model is increased here mainly due to the change in input size to 3D. A comparison with other cases is given in Table 2. It is observable that the accuracy for this case is 94.56%. Figure 6a shows the confusion matrix and minimal confusion is seen between different classes. The highest levels of confusion are present between anger and disgust, i.e., 3.7%.

5.1.4. 3D Angular Velocities

Contrary to the magnitude of the 3D angular velocities, the use of 3D angular velocities results in much better classification accuracies. We have achieved 92.58% classification accuracy for this particular case. The F1 scores for sad, disgust, fear, happy, surprise, and anger are 98%, 82%, 95%, 97%, 88%, and 97%, respectively. Figure 6b shows the confusion matrix and most of the confusion is observable between disgust and fear, i.e., 5.9%.

5.1.5. 6D Acceleration and Angular Velocities

This case used the highest input size, i.e., 6D accelerations and angular velocities, and hence it is the computationally least efficient case. Since all of the 6D components were fed into the proposed deep model, as expected it produced a comparable performance. The average classification accuracy remained at 95% (similar to the first case, i.e., the 1D magnitude of 3D accelerations). The F1-Score, precision, and recall remained at 95% as well. The best case for classification for all six emotions in the competitive study [19] had a classification accuracy of 86.45%. This was the case for the 200 top contributing features and the classification model used was support vector machine (SVM). The confusion matrix for their study is presented in Figure 7b for the sake of this case, as given in Figure 7a.

5.2. Hot vs. Cold Emotion Classification

This section presents the results of binary emotion classification (hot vs cold emotions) for the case of the magnitude of 3D accelerations ($\widehat{ma{g}_a^w}$) only. It is observable from the previously presented multi-emotion classification results that the magnitude of 3D accelerations yields the best results in terms of performance and complexity. A similar trend was observed in the binary classification case and a classification accuracy of 98.08% is noted. The performance report is given in Figure 8 and as a heatmap. The F1-score, precision, and recall remained at 98%. The confusion matrix is given in Figure 8b. The loss and accuracy curves are presented in Figure 9. It can be seen that the model fits the data well and no over-fitting is seen.

5.3. Effect of Segmentation Window Size

The size of the signal’s segmentation window is a critical element in the case of inertial sensors-based classification. The importance of this has been highlighted in [35,64]. The selection of window sizes and step sizes was inspired by [35], where they used the segmentation size of 200 with the same stepping size, i.e., 200 resulting in 0% overlap in two segments. The motivation was to align the step size of 200 samples with the walking frequency of a normal human gait. For our case, we experimented with different segmentation window sizes and step sizes, and the results are given in

Table 3. Comparison of different segmentation window sizes and step sizes and their effect on test accuracies. A window size of 256 and a step size of 16 produce the best test accuracies.

Window	Step	Overlap between	Test
Size	Size	Consecutive Samples (%)	Accuracy (%)
200	100	50	52.95
400	100	75	62.44
256	16	93.75	95

. The optimal values are 256 for the window size and 16 for the step size. The experiment for the selection of these parameters was performed only for the case of the magnitude of 3D accelerations.

5.4. Accuracy and Loss Curves

Figure 10 depicts the training and testing accuracy and loss curves computed using the magnitude of 3D accelerations (1D input size). The training and testing accuracy and loss curves converge decently, indicating a satisfactory fit and ruling out over-fitting.

5.5. Complexity of the Model

Table 4. Comparison of trainable and non-trainable parameters for different cases of multi-emotion classification.

Cases	Trainable	Non-Trainable
	Parameters	Parameters
$a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$ (6D)	758,160	896
$a_x^w,a_y^w,a_z^w$ (3D)	756,624	896
${\omega }_x^w,{\omega }_y^w,{\omega }_z^w$ (3D)	756,624	896
Magnitude of 3D Angular Velocities (1D)	755,600	896
Magnitude of 3D Accelerations (1D)	755,600	896

shows a comparison of the complexity of the models trained on different input features, in terms of trainable and non-trainable parameters. Similarly, Figure 11 compares the complexity of models trained on different input features in terms of trainable and non-trainable parameters. It is evident that the reduction in input space (i.e., 1D input size) requires lesser trainable parameters and, hence, it reduces the complexity of the model by minimizing model training time.

Table 5. Computation time required by the proposed model for the training and inference under different sizes of input signals.

Input	Training Time (ms)	Inference Time (ms)
$a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$ (6D)	286,227	385
$a_x^w,a_y^w,a_z^w$ (3D)	269,682	229
$\widehat{ma{g}_a^w}$ (1D)	147,615	67

and Figure 12 present the computation time required by the proposed model for the training and inference under different sizes of input signals, i.e., 1D ($\widehat{ma{g}_a^w}$), 3D ($a_x^w,a_y^w,a_z^w$), and 6D ($a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$). With a 1D input size ($\widehat{ma{g}_a^w}$), the model was trained in 147,615 ms, whereas the inference took only 67 ms. When the input size was 3D ($a_x^w,a_y^w,a_z^w$), the model took 269,682 ms for training and 229 ms for inference. With a 6D input size ($a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$), the training time of the model was 286,227 ms and inference time was 385ms. In sum, the training and inference time required by the proposed 1D input size is faster than 3D and 6D by a factor of about 3.4 and 6.4, respectively. With these findings, it can be concluded that the 1D input size, being efficient and lightweight, is appropriate for the real-time classification of human emotions using wearable inertial sensor data. The experiment was conducted on a single Google Colab instance, equipped with a Tesla T4 GPU, two CPU cores, and 12.68 GB of RAM.

6. Conclusions

We presented a unique CNN-RNN deep learning model for classifying human emotional states using body-mounted inertial sensors. Our experiments have demonstrated that decreasing the input space not only reduces the model’s complexity but also gives the best classification accuracies. The input space can be reduced by computing the magnitude of 3D accelerations or 3D angular velocities. Our results show a classification accuracy above 95% with the magnitude of 3D accelerations ($\widehat{ma{g}_a}$), i.e., 1D input signal. For the sake of comparison, we also trained and validated the proposed deep model with 3D input signals (3D accelerations, 3D angular velocities) and 6D input signals (6D accelerations and angular velocities). The results were computed on a multi-emotions inertial dataset collected in one of our previous studies, which covers six basic emotions, i.e., disgust, happy, anger, surprise, sad, and fear.

6.1. Comparison with Existing Approaches

A detailed comparison with the existing studies is given in

Table 6. Performance comparison of the proposed work with previous studies.

Sr. #	Ref.	Year	Sensor	Emotions Classified	Model Type	Accuracy (%)
1	Piskioulis et al., [37]	2021	Smartphone	Enjoyment and Frustration	LR, DT, LR, SVM, MLP	Enjoyment (87.90), Frustration (89.45)
2	Reyana et al., [38]	2021	Capacitive sensor, Accelerometer	Neutral, Angry	ML	Neutral (100), Angry (90)
			Heartbeat, Muscle pressure	Happy, Sad	ML	Happy (80), Sad (70)
3	Hashmi et al., [19]	2020	Smartphone	Surprise, Happy, Sad, Anger, Fear, Disgust	SVM, RF	86.45
4	Quiroz et al., [42]	2017	Smartwatch	Happy, Sad	LR, RF	75
5	Cui et al., [43]	2016	Smartphone	Happy, Anger, Neutral	SVM, DT, MLP, RT, RF	80
6	Zhang et al., [27]	2016	Smart Bracelet	Happy, Anger, Neutral	SVM, DT, RT, RF	81.2
7	Proposed Approach	2023	Smartphone	Surprise, Happy, Sad, Anger, Fear, Disgust	$a_x^w,a_y^w,a_z^w,{\omega }_x^w,{\omega }_y^w,{\omega }_z^w$	95.01
					$a_x^w,a_y^w,a_z^w$	94.56
					$\widehat{ma{g}_a^w}$	95.23
					${\omega }_x^w,{\omega }_y^w,{\omega }_z^w$	92.58

. It can be observed that the most related work is [19], in which they proposed a set of manually-crafted features which are used to train different supervised learning models. We have used the same dataset to train our proposed deep learning model and achieved the best classification accuracy of 95% using the 1D input signal.

6.2. Limitations and Future Work Directions

We essentially worked on six basic emotions–disgust, happy, anger, surprise, sad, fear. However, a variety of micro-expressions such as pain, pleasure, joy, etc., exist. Extending the dataset with a wider range of emotions is an important future direction, which will help in developing better deep models to estimate and understand human emotions on the fly. Similarly, the demographic diversity of the subject is another important direction of future work to analyze emotions from a demographic perspective. Additionally, for practical applications, developing a real-time, edge-based emotions prediction system developed over the node (e.g., a smartwatch) is another crucial aspect to be worked on in the future.