Sensors

Editorial

7 pages, 204 KiB

Open AccessEditorial

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

by Kyandoghere Kyamakya, Fadi Al-Machot, Ahmad Haj Mosa, Hamid Bouchachia, Jean Chamberlain Chedjou and Antoine Bagula

Sensors 2021, 21(7), 2273; https://doi.org/10.3390/s21072273 - 24 Mar 2021

Cited by 9 | Viewed by 3614

Abstract

Intelligent sociotechnical systems are gaining momentum in today’s information-rich society, where different technologies are used to collect data from such systems and mine this data to make useful insights about our daily activities [...] Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

Research

Jump to: Editorial, Review, Other

15 pages, 2447 KiB

Open AccessArticle

Enhancing Mouth-Based Emotion Recognition Using Transfer Learning

by Valentina Franzoni, Giulio Biondi, Damiano Perri and Osvaldo Gervasi

Sensors 2020, 20(18), 5222; https://doi.org/10.3390/s20185222 - 13 Sep 2020

Cited by 30 | Viewed by 4180

Abstract

This work concludes the first study on mouth-based emotion recognition while adopting a transfer learning approach. Transfer learning results are paramount for mouth-based emotion emotion recognition, because few datasets are available, and most of them include emotional expressions simulated by actors, instead of [...] Read more.

This work concludes the first study on mouth-based emotion recognition while adopting a transfer learning approach. Transfer learning results are paramount for mouth-based emotion emotion recognition, because few datasets are available, and most of them include emotional expressions simulated by actors, instead of adopting real-world categorisation. Using transfer learning, we can use fewer training data than training a whole network from scratch, and thus more efficiently fine-tune the network with emotional data and improve the convolutional neural network’s performance accuracy in the desired domain. The proposed approach aims at improving emotion recognition dynamically, taking into account not only new scenarios but also modified situations to the initial training phase, because the image of the mouth can be available even when the whole face is visible only in an unfavourable perspective. Typical applications include automated supervision of bedridden critical patients in a healthcare management environment, and portable applications supporting disabled users having difficulties in seeing or recognising facial emotions. This achievement takes advantage of previous preliminary works on mouth-based emotion recognition using deep-learning, and has the further benefit of having been tested and compared to a set of other networks using an extensive dataset for face-based emotion recognition, well known in the literature. The accuracy of mouth-based emotion recognition was also compared to the corresponding full-face emotion recognition; we found that the loss in accuracy is mostly compensated by consistent performance in the visual emotion recognition domain. We can, therefore, state that our method proves the importance of mouth detection in the complex process of emotion recognition. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

16 pages, 377 KiB

Open AccessArticle

Arousal Detection in Elderly People from Electrodermal Activity Using Musical Stimuli

by Almudena Bartolomé-Tomás, Roberto Sánchez-Reolid, Alicia Fernández-Sotos, José Miguel Latorre and Antonio Fernández-Caballero

Sensors 2020, 20(17), 4788; https://doi.org/10.3390/s20174788 - 25 Aug 2020

Cited by 16 | Viewed by 3184

Abstract

The detection of emotions is fundamental in many areas related to health and well-being. This paper presents the identification of the level of arousal in older people by monitoring their electrodermal activity (EDA) through a commercial device. The objective was to recognize arousal [...] Read more.

The detection of emotions is fundamental in many areas related to health and well-being. This paper presents the identification of the level of arousal in older people by monitoring their electrodermal activity (EDA) through a commercial device. The objective was to recognize arousal changes to create future therapies that help them to improve their mood, contributing to reduce possible situations of depression and anxiety. To this end, some elderly people in the region of Murcia were exposed to listening to various musical genres (flamenco, Spanish folklore, Cuban genre and rock/jazz) that they heard in their youth. Using methods based on the process of deconvolution of the EDA signal, two different studies were carried out. The first, of a purely statistical nature, was based on the search for statistically significant differences for a series of temporal, morphological, statistical and frequency features of the processed signals. It was found that Flamenco and Spanish Folklore presented the highest number of statistically significant parameters. In the second study, a wide range of classifiers was used to analyze the possible correlations between the detection of the EDA-based arousal level compared to the participants’ responses to the level of arousal subjectively felt. In this case, it was obtained that the best classifiers are support vector machines, with 87% accuracy for flamenco and 83.1% for Spanish Folklore, followed by K-nearest neighbors with 81.4% and 81.5% for Flamenco and Spanish Folklore again. These results reinforce the notion of familiarity with a musical genre on emotional induction. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

17 pages, 266 KiB

Open AccessArticle

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

by Patrícia Bota, Chen Wang, Ana Fred and Hugo Silva

Sensors 2020, 20(17), 4723; https://doi.org/10.3390/s20174723 - 21 Aug 2020

Cited by 39 | Viewed by 5299

Abstract

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of [...] Read more.

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure A1

24 pages, 12579 KiB

Open AccessArticle

StressFoot: Uncovering the Potential of the Foot for Acute Stress Sensing in Sitting Posture

by Don Samitha Elvitigala, Denys J. C. Matthies and Suranga Nanayakkara

Sensors 2020, 20(10), 2882; https://doi.org/10.3390/s20102882 - 19 May 2020

Cited by 16 | Viewed by 5506

Abstract

Stress is a naturally occurring psychological response and identifiable by several body signs. We propose a novel way to discriminate acute stress and relaxation, using movement and posture characteristics of the foot. Based on data collected from 23 participants performing tasks that induced [...] Read more.

Stress is a naturally occurring psychological response and identifiable by several body signs. We propose a novel way to discriminate acute stress and relaxation, using movement and posture characteristics of the foot. Based on data collected from 23 participants performing tasks that induced stress and relaxation, we developed several machine learning models to construct the validity of our method. We tested our models in another study with 11 additional participants. The results demonstrated replicability with an overall accuracy of 87%. To also demonstrate external validity, we conducted a field study with 10 participants, performing their usual everyday office tasks over a working day. The results showed substantial robustness. We describe ten significant features in detail to enable an easy replication of our models. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

21 pages, 3358 KiB

Open AccessArticle

Facial Expression Recognition Based on Weighted-Cluster Loss and Deep Transfer Learning Using a Highly Imbalanced Dataset

by Quan T. Ngo and Seokhoon Yoon

Sensors 2020, 20(9), 2639; https://doi.org/10.3390/s20092639 - 5 May 2020

Cited by 35 | Viewed by 5668

Abstract

Facial expression recognition (FER) is a challenging problem in the fields of pattern recognition and computer vision. The recent success of convolutional neural networks (CNNs) in object detection and object segmentation tasks has shown promise in building an automatic deep CNN-based FER model. [...] Read more.

Facial expression recognition (FER) is a challenging problem in the fields of pattern recognition and computer vision. The recent success of convolutional neural networks (CNNs) in object detection and object segmentation tasks has shown promise in building an automatic deep CNN-based FER model. However, in real-world scenarios, performance degrades dramatically owing to the great diversity of factors unrelated to facial expressions, and due to a lack of training data and an intrinsic imbalance in the existing facial emotion datasets. To tackle these problems, this paper not only applies deep transfer learning techniques, but also proposes a novel loss function called weighted-cluster loss, which is used during the fine-tuning phase. Specifically, the weighted-cluster loss function simultaneously improves the intra-class compactness and the inter-class separability by learning a class center for each emotion class. It also takes the imbalance in a facial expression dataset into account by giving each emotion class a weight based on its proportion of the total number of images. In addition, a recent, successful deep CNN architecture, pre-trained in the task of face identification with the VGGFace2 database from the Visual Geometry Group at Oxford University, is employed and fine-tuned using the proposed loss function to recognize eight basic facial emotions from the AffectNet database of facial expression, valence, and arousal computing in the wild. Experiments on an AffectNet real-world facial dataset demonstrate that our method outperforms the baseline CNN models that use either weighted-softmax loss or center loss. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

33 pages, 7654 KiB

Open AccessArticle

The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

by Dilana Hazer-Rau, Sascha Meudt, Andreas Daucher, Jennifer Spohrs, Holger Hoffmann, Friedhelm Schwenker and Harald C. Traue

Sensors 2020, 20(8), 2308; https://doi.org/10.3390/s20082308 - 17 Apr 2020

Cited by 23 | Viewed by 4874

Abstract

In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional [...] Read more.

In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional and cognitive load states. It consists of six experimental sequences, inducing Interest, Overload, Normal, Easy, Underload, and Frustration. Each sequence is followed by subjective feedbacks to validate the induction, a respiration baseline to level off the physiological reactions, and a summary of results. Further, prior to the experiment, three questionnaires related to emotion regulation (ERQ), emotional control (TEIQue-SF), and personality traits (TIPI) were collected from each subject to evaluate the stability of the induction paradigm. Based on this HCI scenario, the University of Ulm Multimodal Affective Corpus (uulmMAC), consisting of two homogenous samples of 60 participants and 100 recording sessions was generated. We recorded 16 sensor modalities including 4 × video, 3 × audio, and 7 × biophysiological, depth, and pose streams. Further, additional labels and annotations were also collected. After recording, all data were post-processed and checked for technical and signal quality, resulting in the final uulmMAC dataset of 57 subjects and 95 recording sessions. The evaluation of the reported subjective feedbacks shows significant differences between the sequences, well consistent with the induced states, and the analysis of the questionnaires shows stable results. In summary, our uulmMAC database is a valuable contribution for the field of affective computing and multimodal data analysis: Acquired in a mobile interactive scenario close to real HCI, it consists of a large number of subjects and allows transtemporal investigations. Validated via subjective feedbacks and checked for quality issues, it can be used for affective computing and machine learning applications. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

15 pages, 2936 KiB

Open AccessArticle

EEG Based Classification of Long-Term Stress Using Psychological Labeling

by Sanay Muhammad Umar Saeed, Syed Muhammad Anwar, Humaira Khalid, Muhammad Majid and Ulas Bagci

Sensors 2020, 20(7), 1886; https://doi.org/10.3390/s20071886 - 29 Mar 2020

Cited by 71 | Viewed by 7500

Abstract

Stress research is a rapidly emerging area in the field of electroencephalography (EEG) signal processing. The use of EEG as an objective measure for cost effective and personalized stress management becomes important in situations like the nonavailability of mental health facilities. In this [...] Read more.

Stress research is a rapidly emerging area in the field of electroencephalography (EEG) signal processing. The use of EEG as an objective measure for cost effective and personalized stress management becomes important in situations like the nonavailability of mental health facilities. In this study, long-term stress was classified with machine learning algorithms using resting state EEG signal recordings. The labeling for the stress and control groups was performed using two currently accepted clinical practices: (i) the perceived stress scale score and (ii) expert evaluation. The frequency domain features were extracted from five-channel EEG recordings in addition to the frontal and temporal alpha and beta asymmetries. The alpha asymmetry was computed from four channels and used as a feature. Feature selection was also performed to identify statistically significant features for both stress and control groups (via t-test). We found that support vector machine was best suited to classify long-term human stress when used with alpha asymmetry as a feature. It was observed that the expert evaluation-based labeling method had improved the classification accuracy by up to 85.20%. Based on these results, it is concluded that alpha asymmetry may be used as a potential bio-marker for stress classification, when labels are assigned using expert evaluation. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

19 pages, 1289 KiB

Open AccessArticle

Two-Stream Attention Network for Pain Recognition from Video Sequences

by Patrick Thiam, Hans A. Kestler and Friedhelm Schwenker

Sensors 2020, 20(3), 839; https://doi.org/10.3390/s20030839 - 4 Feb 2020

Cited by 28 | Viewed by 5489

Abstract

Several approaches have been proposed for the analysis of pain-related facial expressions. These approaches range from common classification architectures based on a set of carefully designed handcrafted features, to deep neural networks characterised by an autonomous extraction of relevant facial descriptors and simultaneous [...] Read more.

Several approaches have been proposed for the analysis of pain-related facial expressions. These approaches range from common classification architectures based on a set of carefully designed handcrafted features, to deep neural networks characterised by an autonomous extraction of relevant facial descriptors and simultaneous optimisation of a classification architecture. In the current work, an end-to-end approach based on attention networks for the analysis and recognition of pain-related facial expressions is proposed. The method combines both spatial and temporal aspects of facial expressions through a weighted aggregation of attention-based neural networks’ outputs, based on sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs). Each input stream is fed into a specific attention network consisting of a Convolutional Neural Network (CNN) coupled to a Bidirectional Long Short-Term Memory (BiLSTM) Recurrent Neural Network (RNN). An attention mechanism generates a single weighted representation of each input stream (MHI sequence and OFI sequence), which is subsequently used to perform specific classification tasks. Simultaneously, a weighted aggregation of the classification scores specific to each input stream is performed to generate a final classification output. The assessment conducted on both the BioVid Heat Pain Database (Part A) and SenseEmotion Database points at the relevance of the proposed approach, as its classification performance is on par with state-of-the-art classification approaches proposed in the literature. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Graphical abstract

20 pages, 1320 KiB

Open AccessArticle

How Laboratory Experiments Can Be Exploited for Monitoring Stress in the Wild: A Bridge Between Laboratory and Daily Life

by Yekta Said Can, Dilara Gokay, Dilruba Reyyan Kılıç, Deniz Ekiz, Niaz Chalabianloo and Cem Ersoy

Sensors 2020, 20(3), 838; https://doi.org/10.3390/s20030838 - 4 Feb 2020

Cited by 35 | Viewed by 7107

Abstract

Chronic stress leads to poor well-being, and it has effects on life quality and health. Society may have significant benefits from an automatic daily life stress detection system using unobtrusive wearable devices using physiological signals. However, the performance of these systems is not [...] Read more.

Chronic stress leads to poor well-being, and it has effects on life quality and health. Society may have significant benefits from an automatic daily life stress detection system using unobtrusive wearable devices using physiological signals. However, the performance of these systems is not sufficiently accurate when they are used in unrestricted daily life compared to the systems tested in controlled real-life and laboratory conditions. To test our stress level detection system that preprocesses noisy physiological signals, extracts features, and applies machine learning classification techniques, we used a laboratory experiment and ecological momentary assessment based data collection with smartwatches in daily life. We investigated the effect of different labeling techniques and different training and test environments. In the laboratory environments, we had more controlled situations, and we could validate the perceived stress from self-reports. When machine learning models were trained in the laboratory instead of training them with the data coming from daily life, the accuracy of the system when tested in daily life improved significantly. The subjectivity effect coming from the self-reports in daily life could be eliminated. Our system obtained higher stress level detection accuracy results compared to most of the previous daily life studies. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

27 pages, 1226 KiB

Open AccessArticle

Activity Recognition Using Wearable Physiological Measurements: Selection of Features from a Comprehensive Literature Study

by Inma Mohino-Herranz, Roberto Gil-Pita, Manuel Rosa-Zurera and Fernando Seoane

Sensors 2019, 19(24), 5524; https://doi.org/10.3390/s19245524 - 13 Dec 2019

Cited by 21 | Viewed by 4067

Abstract

Activity and emotion recognition based on physiological signal processing in health care applications is a relevant research field, with promising future and relevant applications, such as health at work or preventive care. This paper carries out a deep analysis of features proposed to [...] Read more.

Activity and emotion recognition based on physiological signal processing in health care applications is a relevant research field, with promising future and relevant applications, such as health at work or preventive care. This paper carries out a deep analysis of features proposed to extract information from the electrocardiogram, thoracic electrical bioimpedance, and electrodermal activity signals. The activities analyzed are: neutral, emotional, mental and physical. A total number of 533 features are tested for activity recognition, performing a comprehensive study taking into consideration the prediction accuracy, feature calculation, window length, and type of classifier. Feature selection to know the most relevant features from the complete set is implemented using a genetic algorithm, with a different number of features. This study has allowed us to determine the best number of features to obtain a good error probability avoiding over-fitting, and the best subset of features among those proposed in the literature. The lowest error probability that is obtained is 22.2%, with 40 features, a least squares error classifier, and 40 s window length. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

25 pages, 5852 KiB

Open AccessArticle

Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments

by Válber César Cavalcanti Roza and Octavian Adrian Postolache

Sensors 2019, 19(24), 5516; https://doi.org/10.3390/s19245516 - 13 Dec 2019

Cited by 18 | Viewed by 7219

Abstract

The present work tries to fill part of the gap regarding the pilots’ emotions and their bio-reactions during some flight procedures such as, takeoff, climbing, cruising, descent, initial approach, final approach and landing. A sensing architecture and a set of experiments were developed, [...] Read more.

The present work tries to fill part of the gap regarding the pilots’ emotions and their bio-reactions during some flight procedures such as, takeoff, climbing, cruising, descent, initial approach, final approach and landing. A sensing architecture and a set of experiments were developed, associating it to several simulated flights (

N_{f l i g h t s} = 13

) using the Microsoft Flight Simulator Steam Edition (FSX-SE). The approach was carried out with eight beginner users on the flight simulator (

N_{p i l o t s} = 8

). It is shown that it is possible to recognize emotions from different pilots in flight, combining their present and previous emotions. The cardiac system based on Heart Rate (HR), Galvanic Skin Response (GSR) and Electroencephalography (EEG), were used to extract emotions, as well as the intensities of emotions detected from the pilot face. We also considered five main emotions: happy, sad, angry, surprise and scared. The emotion recognition is based on Artificial Neural Networks and Deep Learning techniques. The Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) were the main methods used to measure the quality of the regression output models. The tests of the produced output models showed that the lowest recognition errors were reached when all data were considered or when the GSR datasets were omitted from the model training. It also showed that the emotion surprised was the easiest to recognize, having a mean RMSE of 0.13 and mean MAE of 0.01; while the emotion sad was the hardest to recognize, having a mean RMSE of 0.82 and mean MAE of 0.08. When we considered only the higher emotion intensities by time, the most matches accuracies were between 55% and 100%. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

21 pages, 2482 KiB

Open AccessArticle

Dilated Skip Convolution for Facial Landmark Detection

by Seyha Chim, Jin-Gu Lee and Ho-Hyun Park

Sensors 2019, 19(24), 5350; https://doi.org/10.3390/s19245350 - 4 Dec 2019

Cited by 11 | Viewed by 4727

Abstract

Facial landmark detection has gained enormous interest for face-related applications due to its success in facial analysis tasks such as facial recognition, cartoon generation, face tracking and facial expression analysis. Many studies have been proposed and implemented to deal with the challenging problems [...] Read more.

Facial landmark detection has gained enormous interest for face-related applications due to its success in facial analysis tasks such as facial recognition, cartoon generation, face tracking and facial expression analysis. Many studies have been proposed and implemented to deal with the challenging problems of localizing facial landmarks from given images, including large appearance variations and partial occlusion. Studies have differed in the way they use the facial appearances and shape information of input images. In our work, we consider facial information within both global and local contexts. We aim to obtain local pixel-level accuracy for local-context information in the first stage and integrate this with knowledge of spatial relationships between each key point in a whole image for global-context information in the second stage. Thus, the pipeline of our architecture consists of two main components: (1) a deep network for local-context subnet that generates detection heatmaps via fully convolutional DenseNets with additional kernel convolution filters and (2) a dilated skip convolution subnet—a combination of dilated convolutions and skip-connections networks—that are in charge of robustly refining the local appearance heatmaps. Through this proposed architecture, we demonstrate that our approach achieves state-of-the-art performance on challenging datasets—including LFPW, HELEN, 300W and AFLW2000-3D—by leveraging fully convolutional DenseNets, skip-connections and dilated convolution architecture without further post-processing. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

20 pages, 4443 KiB

Open AccessArticle

An Exploration of Machine Learning Methods for Robust Boredom Classification Using EEG and GSR Data

by Jungryul Seo, Teemu H. Laine and Kyung-Ah Sohn

Sensors 2019, 19(20), 4561; https://doi.org/10.3390/s19204561 - 20 Oct 2019

Cited by 25 | Viewed by 5646

Abstract

In recent years, affective computing has been actively researched to provide a higher level of emotion-awareness. Numerous studies have been conducted to detect the user’s emotions from physiological data. Among a myriad of target emotions, boredom, in particular, has been suggested to cause [...] Read more.

In recent years, affective computing has been actively researched to provide a higher level of emotion-awareness. Numerous studies have been conducted to detect the user’s emotions from physiological data. Among a myriad of target emotions, boredom, in particular, has been suggested to cause not only medical issues but also challenges in various facets of daily life. However, to the best of our knowledge, no previous studies have used electroencephalography (EEG) and galvanic skin response (GSR) together for boredom classification, although these data have potential features for emotion classification. To investigate the combined effect of these features on boredom classification, we collected EEG and GSR data from 28 participants using off-the-shelf sensors. During data acquisition, we used a set of stimuli comprising a video clip designed to elicit boredom and two other video clips of entertaining content. The collected samples were labeled based on the participants’ questionnaire-based testimonies on experienced boredom levels. Using the collected data, we initially trained 30 models with 19 machine learning algorithms and selected the top three candidate classifiers. After tuning the hyperparameters, we validated the final models through 1000 iterations of 10-fold cross validation to increase the robustness of the test results. Our results indicated that a Multilayer Perceptron model performed the best with a mean accuracy of 79.98% (AUC: 0.781). It also revealed the correlation between boredom and the combined features of EEG and GSR. These results can be useful for building accurate affective computing systems and understanding the physiological properties of boredom. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

25 pages, 8813 KiB

Open AccessArticle

Wearables and the Quantified Self: Systematic Benchmarking of Physiological Sensors

by Günther Sagl, Bernd Resch, Andreas Petutschnig, Kalliopi Kyriakou, Michael Liedlgruber and Frank H. Wilhelm

Sensors 2019, 19(20), 4448; https://doi.org/10.3390/s19204448 - 14 Oct 2019

Cited by 14 | Viewed by 6230

Abstract

Wearable sensors are increasingly used in research, as well as for personal and private purposes. A variety of scientific studies are based on physiological measurements from such rather low-cost wearables. That said, how accurate are such measurements compared to measurements from well-calibrated, high-quality [...] Read more.

Wearable sensors are increasingly used in research, as well as for personal and private purposes. A variety of scientific studies are based on physiological measurements from such rather low-cost wearables. That said, how accurate are such measurements compared to measurements from well-calibrated, high-quality laboratory equipment used in psychological and medical research? The answer to this question, undoubtedly impacts the reliability of a study’s results. In this paper, we demonstrate an approach to quantify the accuracy of low-cost wearables in comparison to high-quality laboratory sensors. We therefore developed a benchmark framework for physiological sensors that covers the entire workflow from sensor data acquisition to the computation and interpretation of diverse correlation and similarity metrics. We evaluated this framework based on a study with 18 participants. Each participant was equipped with one high-quality laboratory sensor and two wearables. These three sensors simultaneously measured the physiological parameters such as heart rate and galvanic skin response, while the participant was cycling on an ergometer following a predefined routine. The results of our benchmarking show that cardiovascular parameters (heart rate, inter-beat interval, heart rate variability) yield very high correlations and similarities. Measurement of galvanic skin response, which is a more delicate undertaking, resulted in lower, but still reasonable correlations and similarities. We conclude that the benchmarked wearables provide physiological measurements such as heart rate and inter-beat interval with an accuracy close to that of the professional high-end sensor, but the accuracy varies more for other parameters, such as galvanic skin response. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Graphical abstract

18 pages, 542 KiB

Open AccessArticle

Ambulatory and Laboratory Stress Detection Based on Raw Electrocardiogram Signals Using a Convolutional Neural Network

by Hyun-Myung Cho, Heesu Park, Suh-Yeon Dong and Inchan Youn

Sensors 2019, 19(20), 4408; https://doi.org/10.3390/s19204408 - 11 Oct 2019

Cited by 29 | Viewed by 3671

Abstract

The goals of this study are the suggestion of a better classification method for detecting stressed states based on raw electrocardiogram (ECG) data and a method for training a deep neural network (DNN) with a smaller data set. We suggest an end-to-end architecture [...] Read more.

The goals of this study are the suggestion of a better classification method for detecting stressed states based on raw electrocardiogram (ECG) data and a method for training a deep neural network (DNN) with a smaller data set. We suggest an end-to-end architecture to detect stress using raw ECGs. The architecture consists of successive stages that contain convolutional layers. In this study, two kinds of data sets are used to train and validate the model: A driving data set and a mental arithmetic data set, which smaller than the driving data set. We apply a transfer learning method to train a model with a small data set. The proposed model shows better performance, based on receiver operating curves, than conventional methods. Compared with other DNN methods using raw ECGs, the proposed model improves the accuracy from 87.39% to 90.19%. The transfer learning method improves accuracy by 12.01% and 10.06% when 10 s and 60 s of ECG signals, respectively, are used in the model. In conclusion, our model outperforms previous models using raw ECGs from a small data set and, so, we believe that our model can significantly contribute to mobile healthcare for stress management in daily life. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

16 pages, 2302 KiB

Open AccessArticle

A Wearable In-Ear EEG Device for Emotion Monitoring

by Chanavit Athavipach, Setha Pan-ngum and Pasin Israsena

Sensors 2019, 19(18), 4014; https://doi.org/10.3390/s19184014 - 17 Sep 2019

Cited by 66 | Viewed by 16140

Abstract

For future healthcare applications, which are increasingly moving towards out-of-hospital or home-based caring models, the ability to remotely and continuously monitor patients’ conditions effectively are imperative. Among others, emotional state is one of the conditions that could be of interest to doctors or [...] Read more.

For future healthcare applications, which are increasingly moving towards out-of-hospital or home-based caring models, the ability to remotely and continuously monitor patients’ conditions effectively are imperative. Among others, emotional state is one of the conditions that could be of interest to doctors or caregivers. This paper discusses a preliminary study to develop a wearable device that is a low cost, single channel, dry contact, in-ear EEG suitable for non-intrusive monitoring. All aspects of the designs, engineering, and experimenting by applying machine learning for emotion classification, are covered. Based on the valence and arousal emotion model, the device is able to classify basic emotion with 71.07% accuracy (valence), 72.89% accuracy (arousal), and 53.72% (all four emotions). The results are comparable to those measured from the more conventional EEG headsets at T7 and T8 scalp positions. These results, together with its earphone-like wearability, suggest its potential usage especially for future healthcare applications, such as home-based or tele-monitoring systems as intended. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

16 pages, 1404 KiB

Open AccessArticle

Deep ECG-Respiration Network (DeepER Net) for Recognizing Mental Stress

by Wonju Seo, Namho Kim, Sehyeon Kim, Chanhee Lee and Sung-Min Park

Sensors 2019, 19(13), 3021; https://doi.org/10.3390/s19133021 - 9 Jul 2019

Cited by 48 | Viewed by 7492

Abstract

Unmanaged long-term mental stress in the workplace can lead to serious health problems and reduced productivity. To prevent this, it is important to recognize and relieve mental stress in a timely manner. Here, we propose a novel stress detection algorithm based on end-to-end [...] Read more.

Unmanaged long-term mental stress in the workplace can lead to serious health problems and reduced productivity. To prevent this, it is important to recognize and relieve mental stress in a timely manner. Here, we propose a novel stress detection algorithm based on end-to-end deep learning using multiple physiological signals, such as electrocardiogram (ECG) and respiration (RESP) signal. To mimic workplace stress in our experiments, we used Stroop and math tasks as stressors, with each stressor being followed by a relaxation task. Herein, we recruited 18 subjects and measured both ECG and RESP signals using Zephyr BioHarness 3.0. After five-fold cross validation, the proposed network performed well, with an average accuracy of 83.9%, an average F1 score of 0.81, and an average area under the receiver operating characteristic (ROC) curve (AUC) of 0.92, demonstrating its superiority over conventional machine learning models. Furthermore, by visualizing the activation of the trained network’s neurons, we found that they were activated by specific ECG and RESP patterns. In conclusion, we successfully validated the feasibility of end-to-end deep learning using multiple physiological signals for recognition of mental stress in the workplace. We believe that this is a promising approach that will help to improve the quality of life of people suffering from long-term work-related mental stress. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

15 pages, 426 KiB

Open AccessArticle

Combining Inter-Subject Modeling with a Subject-Based Data Transformation to Improve Affect Recognition from EEG Signals

by Miguel Arevalillo-Herráez, Maximo Cobos, Sandra Roger and Miguel García-Pineda

Sensors 2019, 19(13), 2999; https://doi.org/10.3390/s19132999 - 8 Jul 2019

Cited by 32 | Viewed by 4426

Abstract

Existing correlations between features extracted from Electroencephalography (EEG) signals and emotional aspects have motivated the development of a diversity of EEG-based affect detection methods. Both intra-subject and inter-subject approaches have been used in this context. Intra-subject approaches generally suffer from the small sample [...] Read more.

Existing correlations between features extracted from Electroencephalography (EEG) signals and emotional aspects have motivated the development of a diversity of EEG-based affect detection methods. Both intra-subject and inter-subject approaches have been used in this context. Intra-subject approaches generally suffer from the small sample problem, and require the collection of exhaustive data for each new user before the detection system is usable. On the contrary, inter-subject models do not account for the personality and physiological influence of how the individual is feeling and expressing emotions. In this paper, we analyze both modeling approaches, using three public repositories. The results show that the subject’s influence on the EEG signals is substantially higher than that of the emotion and hence it is necessary to account for the subject’s influence on the EEG signals. To do this, we propose a data transformation that seamlessly integrates individual traits into an inter-subject approach, improving classification results. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

24 pages, 3408 KiB

Open AccessArticle

Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction

by Christiane Goulart, Carlos Valadão, Denis Delisle-Rodriguez, Douglas Funayama, Alvaro Favarato, Guilherme Baldo, Vinícius Binotte, Eliete Caldeira and Teodiano Bastos-Filho

Sensors 2019, 19(13), 2844; https://doi.org/10.3390/s19132844 - 26 Jun 2019

Cited by 42 | Viewed by 5211

Abstract

Child-Robot Interaction (CRI) has become increasingly addressed in research and applications. This work proposes a system for emotion recognition in children, recording facial images by both visual (RGB—red, green and blue) and Infrared Thermal Imaging (IRTI) cameras. For this purpose, the Viola-Jones algorithm [...] Read more.

Child-Robot Interaction (CRI) has become increasingly addressed in research and applications. This work proposes a system for emotion recognition in children, recording facial images by both visual (RGB—red, green and blue) and Infrared Thermal Imaging (IRTI) cameras. For this purpose, the Viola-Jones algorithm is used on color images to detect facial regions of interest (ROIs), which are transferred to the thermal camera plane by multiplying a homography matrix obtained through the calibration process of the camera system. As a novelty, we propose to compute the error probability for each ROI located over thermal images, using a reference frame manually marked by a trained expert, in order to choose that ROI better placed according to the expert criteria. Then, this selected ROI is used to relocate the other ROIs, increasing the concordance with respect to the reference manual annotations. Afterwards, other methods for feature extraction, dimensionality reduction through Principal Component Analysis (PCA) and pattern classification by Linear Discriminant Analysis (LDA) are applied to infer emotions. The results show that our approach for ROI locations may track facial landmarks with significant low errors with respect to the traditional Viola-Jones algorithm. These ROIs have shown to be relevant for recognition of five emotions, specifically disgust, fear, happiness, sadness, and surprise, with our recognition system based on PCA and LDA achieving mean accuracy (ACC) and Kappa values of 85.75% and 81.84%, respectively. As a second stage, the proposed recognition system was trained with a dataset of thermal images, collected on 28 typically developing children, in order to infer one of five basic emotions (disgust, fear, happiness, sadness, and surprise) during a child-robot interaction. The results show that our system can be integrated to a social robot to infer child emotions during a child-robot interaction. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

16 pages, 1768 KiB

Open AccessArticle

Identifying Traffic Context Using Driving Stress: A Longitudinal Preliminary Case Study

by Olga Vl. Bitkina, Jungyoon Kim, Jangwoon Park, Jaehyun Park and Hyun K. Kim

Sensors 2019, 19(9), 2152; https://doi.org/10.3390/s19092152 - 9 May 2019

Cited by 51 | Viewed by 6586

Abstract

Many previous studies have identified that physiological responses of a driver are significantly associated with driving stress. However, research is limited to identifying the effects of traffic conditions (low vs. high traffic) and road types (highway vs. city) on driving stress. The objective [...] Read more.

Many previous studies have identified that physiological responses of a driver are significantly associated with driving stress. However, research is limited to identifying the effects of traffic conditions (low vs. high traffic) and road types (highway vs. city) on driving stress. The objective of this study is to quantify the relationship between driving stress and traffic conditions, and driving stress and road types, respectively. In this study, electrodermal activity (EDA) signals for a male driver were collected in real road driving conditions for 60 min a day for 21 days. To classify the levels of driving stress (low vs. high), two separate models were developed by incorporating the statistical features of the EDA signals, one for traffic conditions and the other for road types. Both models were based on the application of EDA features with the logistic regression analysis. City driving turned out to be more stressful than highway driving. Traffic conditions, defined as traffic jam also significantly affected the stress level of the driver, when using the criteria of the vehicle speed of 40 km/h and standard deviation of the speed of 20 km/h. Relevance to industry: The classification results of the two models indicate that the traffic conditions and the road types are important features for driving stress and its related applications. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

24 pages, 1650 KiB

Open AccessArticle

Recognition of Emotion Intensities Using Machine Learning Algorithms: A Comparative Study

by Dhwani Mehta, Mohammad Faridul Haque Siddiqui and Ahmad Y. Javaid

Sensors 2019, 19(8), 1897; https://doi.org/10.3390/s19081897 - 21 Apr 2019

Cited by 47 | Viewed by 7953

Abstract

Over the past two decades, automatic facial emotion recognition has received enormous attention. This is due to the increase in the need for behavioral biometric systems and human–machine interaction where the facial emotion recognition and the intensity of emotion play vital roles. The [...] Read more.

Over the past two decades, automatic facial emotion recognition has received enormous attention. This is due to the increase in the need for behavioral biometric systems and human–machine interaction where the facial emotion recognition and the intensity of emotion play vital roles. The existing works usually do not encode the intensity of the observed facial emotion and even less involve modeling the multi-class facial behavior data jointly. Our work involves recognizing the emotion along with the respective intensities of those emotions. The algorithms used in this comparative study are Gabor filters, a Histogram of Oriented Gradients (HOG), and Local Binary Pattern (LBP) for feature extraction. For classification, we have used Support Vector Machine (SVM), Random Forest (RF), and Nearest Neighbor Algorithm (kNN). This attains emotion recognition and intensity estimation of each recognized emotion. This is a comparative study of classifiers used for facial emotion recognition along with the intensity estimation of those emotions for databases. The results verified that the comparative study could be further used in real-time behavioral facial emotion and intensity of emotion recognition. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

14 pages, 478 KiB

Open AccessArticle

A Deep-Learning Model for Subject-Independent Human Emotion Recognition Using Electrodermal Activity Sensors

by Fadi Al Machot, Ali Elmachot, Mouhannad Ali, Elyan Al Machot and Kyandoghere Kyamakya

Sensors 2019, 19(7), 1659; https://doi.org/10.3390/s19071659 - 7 Apr 2019

Cited by 79 | Viewed by 7994

Abstract

One of the main objectives of Active and Assisted Living (AAL) environments is to ensure that elderly and/or disabled people perform/live well in their immediate environments; this can be monitored by among others the recognition of emotions based on non-highly intrusive sensors such [...] Read more.

One of the main objectives of Active and Assisted Living (AAL) environments is to ensure that elderly and/or disabled people perform/live well in their immediate environments; this can be monitored by among others the recognition of emotions based on non-highly intrusive sensors such as Electrodermal Activity (EDA) sensors. However, designing a learning system or building a machine-learning model to recognize human emotions while training the system on a specific group of persons and testing the system on a totally a new group of persons is still a serious challenge in the field, as it is possible that the second testing group of persons may have different emotion patterns. Accordingly, the purpose of this paper is to contribute to the field of human emotion recognition by proposing a Convolutional Neural Network (CNN) architecture which ensures promising robustness-related results for both subject-dependent and subject-independent human emotion recognition. The CNN model has been trained using a grid search technique which is a model hyperparameter optimization technique to fine-tune the parameters of the proposed CNN architecture. The overall concept’s performance is validated and stress-tested by using MAHNOB and DEAP datasets. The results demonstrate a promising robustness improvement regarding various evaluation metrics. We could increase the accuracy for subject-independent classification to 78% and 82% for MAHNOB and DEAP respectively and to 81% and 85% subject-dependent classification for MAHNOB and DEAP respectively (4 classes/labels). The work shows clearly that while using solely the non-intrusive EDA sensors a robust classification of human emotion is possible even without involving additional/other physiological signals. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

Review

Jump to: Editorial, Research, Other

26 pages, 1595 KiB

Open AccessEditor’s ChoiceReview

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

by Javier Marín-Morales, Carmen Llinares, Jaime Guixeres and Mariano Alcañiz

Sensors 2020, 20(18), 5163; https://doi.org/10.3390/s20185163 - 10 Sep 2020

Cited by 119 | Viewed by 15446

Abstract

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which [...] Read more.

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

36 pages, 2749 KiB

Open AccessReview

EEG-Based BCI Emotion Recognition: A Survey

by Edgar P. Torres, Edgar A. Torres, Myriam Hernández-Álvarez and Sang Guun Yoo

Sensors 2020, 20(18), 5083; https://doi.org/10.3390/s20185083 - 7 Sep 2020

Cited by 161 | Viewed by 13788

Abstract

Affecting computing is an artificial intelligence area of study that recognizes, interprets, processes, and simulates human affects. The user’s emotional states can be sensed through electroencephalography (EEG)-based Brain Computer Interfaces (BCI) devices. Research in emotion recognition using these tools is a rapidly growing [...] Read more.

Affecting computing is an artificial intelligence area of study that recognizes, interprets, processes, and simulates human affects. The user’s emotional states can be sensed through electroencephalography (EEG)-based Brain Computer Interfaces (BCI) devices. Research in emotion recognition using these tools is a rapidly growing field with multiple inter-disciplinary applications. This article performs a survey of the pertinent scientific literature from 2015 to 2020. It presents trends and a comparative analysis of algorithm applications in new implementations from a computer science perspective. Our survey gives an overview of datasets, emotion elicitation methods, feature extraction and selection, classification algorithms, and performance evaluation. Lastly, we provide insights for future developments. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

Other

Jump to: Editorial, Research, Review

12 pages, 771 KiB

Open AccessLetter

Comparison of Regression and Classification Models for User-Independent and Personal Stress Detection

by Pekka Siirtola and Juha Röning

Sensors 2020, 20(16), 4402; https://doi.org/10.3390/s20164402 - 7 Aug 2020

Cited by 22 | Viewed by 5288

Abstract

In this article, regression and classification models are compared for stress detection. Both personal and user-independent models are experimented. The article is based on publicly open dataset called AffectiveROAD, which contains data gathered using Empatica E4 sensor and unlike most of the other [...] Read more.

In this article, regression and classification models are compared for stress detection. Both personal and user-independent models are experimented. The article is based on publicly open dataset called AffectiveROAD, which contains data gathered using Empatica E4 sensor and unlike most of the other stress detection datasets, it contains continuous target variables. The used classification model is Random Forest and the regression model is Bagged tree based ensemble. Based on experiments, regression models outperform classification models, when classifying observations as stressed or not-stressed. The best user-independent results are obtained using a combination of blood volume pulse and skin temperature features, and using these the average balanced accuracy was 74.1% with classification model and 82.3% using regression model. In addition, regression models can be used to estimate the level of the stress. Moreover, the results based on models trained using personal data are not encouraging showing that biosignals have a lot of variation not only between the study subjects but also between the session gathered from the same person. On the other hand, it is shown that with subject-wise feature selection for user-independent model, it is possible to improve recognition models more than by using personal training data to build personal models. In fact, it is shown that with subject-wise feature selection, the average detection rate can be improved as much as 4%-units, and it is especially useful to reduce the variance in the recognition rates between the study subjects. Full article

(This article belongs to the Special Issue Emotion and Stress Recognition Related Sensors and Machine Learning Technologies)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

Share This Special Issue

Special Issue Editors

Special Issue Information

Benefits of Publishing in a Special Issue

Published Papers (26 papers)

Editorial

Research

Review

Other

Further Information

Guidelines

MDPI Initiatives

Follow MDPI