Next Article in Journal
Open Database for Accurate Upper-Limb Intent Detection Using Electromyography and Reliable Extreme Learning Machines
Next Article in Special Issue
The Effectiveness of Depth Data in Liveness Face Authentication Using 3D Sensor Cameras
Previous Article in Journal
Recent Advances in Brillouin Optical Time Domain Reflectometry
Previous Article in Special Issue
Three-D Wide Faces (3DWF): Facial Landmark Detection and 3D Reconstruction over a New RGB–D Multi-Camera Dataset
Article Menu
Issue 8 (April-2) cover image

Export Article

Open AccessReview

A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data

School of Information Technology, Deakin University, Burwood, VIC 3125, Australia
Data61, CSIRO, Battery Point, TAS 7004, Australia
Swinburne Data Science Research Institute, Swinburne University of Technology, Hawthorn, VIC 3122, Australia
Author to whom correspondence should be addressed.
Sensors 2019, 19(8), 1863;
Received: 28 February 2019 / Revised: 15 April 2019 / Accepted: 15 April 2019 / Published: 18 April 2019
(This article belongs to the Special Issue Sensor Applications on Face Analysis)
PDF [2200 KB, uploaded 25 April 2019]


Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system. View Full-Text
Keywords: facial expression recognition (FER); multimodal sensor data; emotional expression recognition; spontaneous expression; real-world conditions facial expression recognition (FER); multimodal sensor data; emotional expression recognition; spontaneous expression; real-world conditions

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

Share & Cite This Article

MDPI and ACS Style

Samadiani, N.; Huang, G.; Cai, B.; Luo, W.; Chi, C.-H.; Xiang, Y.; He, J. A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data. Sensors 2019, 19, 1863.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Sensors EISSN 1424-8220 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top