Next Article in Journal
Performance Evaluation of an Integrated Fuzzy-Based Driving-Support System for Real-Time Risk Management in VANETs
Next Article in Special Issue
Recognition of Typical Locomotion Activities Based on the Sensor Data of a Smartphone in Pocket or Hand
Previous Article in Journal
Motion Capture Data Analysis in the Instantaneous Frequency-Domain Using Hilbert-Huang Transform
Previous Article in Special Issue
Hybrid System of Emotion Evaluation in Physiotherapeutic Procedures
Article

Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

1
Department of Computational Intelligence, Wrocław University of Science and Technology, 50-370 Wrocław, Poland
2
Faculty of Computer Science and Management, Wrocław University of Science and Technology, 50-370 Wrocław, Poland
3
Jožef Stefan Institute, 1000 Ljubljana, Slovenia
4
Jožef Stefan Postgraduate School, 1000 Ljubljana, Slovenia
*
Author to whom correspondence should be addressed.
Sensors 2020, 20(22), 6535; https://doi.org/10.3390/s20226535
Received: 2 October 2020 / Revised: 1 November 2020 / Accepted: 6 November 2020 / Published: 16 November 2020
(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)
To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors. View Full-Text
Keywords: deep learning; multimodal deep learning; end-to-end machine learning; affect recognition; emotion recognition; stress detection; wearables; physiological signals; personal sensors deep learning; multimodal deep learning; end-to-end machine learning; affect recognition; emotion recognition; stress detection; wearables; physiological signals; personal sensors
Show Figures

Figure 1

MDPI and ACS Style

Dzieżyc, M.; Gjoreski, M.; Kazienko, P.; Saganowski, S.; Gams, M. Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data. Sensors 2020, 20, 6535. https://doi.org/10.3390/s20226535

AMA Style

Dzieżyc M, Gjoreski M, Kazienko P, Saganowski S, Gams M. Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data. Sensors. 2020; 20(22):6535. https://doi.org/10.3390/s20226535

Chicago/Turabian Style

Dzieżyc, Maciej, Martin Gjoreski, Przemysław Kazienko, Stanisław Saganowski, and Matjaž Gams. 2020. "Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data" Sensors 20, no. 22: 6535. https://doi.org/10.3390/s20226535

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop