Next Article in Journal
Combining Radar and Optical Sensor Data to Measure Player Value in Baseball
Next Article in Special Issue
Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models
Previous Article in Journal
Monocular Depth Estimation with Joint Attention Feature Distillation and Wavelet-Based Loss Function
Article

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

1
Multimedia Computing Group, Delft University of Technology, 2600AA Delft, The Netherlands
2
Centrum Wiskunde & Informatica (CWI), 1098XG Amsterdam, The Netherlands
3
Future Media and Convergence Institute, Xinhuanet & State Key Laboratory of Media Convergence Production Technology and Systems, Xinhua News Agency, Beijing 100000, China
*
Authors to whom correspondence should be addressed.
Sensors 2021, 21(1), 52; https://doi.org/10.3390/s21010052
Received: 3 December 2020 / Revised: 19 December 2020 / Accepted: 21 December 2020 / Published: 24 December 2020
(This article belongs to the Special Issue Sensor Based Multi-Modal Emotion Recognition)
Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: 76.37% and 74.03% for V-A on CASE, and 70.29% and 68.15% for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neutral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance. View Full-Text
Keywords: emotion recognition; video; physiological signals; machine learning emotion recognition; video; physiological signals; machine learning
Show Figures

Figure 1

MDPI and ACS Style

Zhang, T.; El Ali, A.; Wang, C.; Hanjalic, A.; Cesar, P. CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors. Sensors 2021, 21, 52. https://doi.org/10.3390/s21010052

AMA Style

Zhang T, El Ali A, Wang C, Hanjalic A, Cesar P. CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors. Sensors. 2021; 21(1):52. https://doi.org/10.3390/s21010052

Chicago/Turabian Style

Zhang, Tianyi, Abdallah El Ali, Chen Wang, Alan Hanjalic, and Pablo Cesar. 2021. "CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors" Sensors 21, no. 1: 52. https://doi.org/10.3390/s21010052

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop