Dynamic Facial Expression Recognition by Concatenation of Raw, Semi-Raw, and Distance Features ^†

Abstract

We propose a method for dynamic facial expression recognition that integrates three complementary feature streams from video sequences: (1) raw texture features extracted with EfficientNet-B0, (2) deep geometric features from face mesh representations (semi-raw, EfficientNet-B0), and (3) explicit geometric features derived from facial landmark distances. After refinement with Neighborhood Component Analysis (NCA), features are concatenated and fed to Bi-LSTM modeling temporal dynamics. The method achieved 58.25% (UAR) and 58.40% (WAR) on CREMA-D, and it achieved 82.81% (UAR) and 82.99% (WAR) on RAVDESS. The Bi-LSTM contains 8.68 M parameters, while the EfficientNet-B0 feature extractors add approximately 4 M parameters.

1. Introduction

This work introduces a dynamic facial expression recognition approach designed to address the objectivity and scalability issues present in human-based monitoring within controlled environments. The proposed method serves as an auxiliary metric that can support telemedicine by providing additional cues in remote patient assessment, assist psychology by offering quantitative emotional indicators, and aid legal fields through complementary behavioral analysis.

Several methods have been proposed for emotion classification from audiovisual signals. For instance, an architecture based on I3D and SoundNet has been employed, where the extracted features are processed by an LSTM trained with Deep Metric Learning (DML), and K-Nearest Neighbor (KNN) is used as the final classifier [1]. Similarly, another approach combines VGG-M and VGGish, together with Positional Encoding (PE) and a Multi-Head Self-Attention (MHSA) mechanism [2]. In addition, other proposed methods are summarized. A limitation in benchmarking computational efficiency is the frequent omission of parameter counts in the state-of-the-art literature. Nevertheless, given that these models are often derived from base architectures containing more than 200 million parameters, it is evident that they operate at a significantly higher computational complexity than the method proposed in this work.

Therefore, the following methodological pipeline is proposed:

• A uniform set of 100 frames per video is selected, corresponding to the average interval of facial expression presentation across both datasets.
• A multi-stream feature extraction approach is implemented for each frame, encompassing the following: (i) the original video processed with EfficientNet-B0 (raw features); (ii) the video with FaceMesh landmarks processed with EfficientNet-B0 (semi-raw features); and (iii) specific facial landmark distances derived from FaceMesh applied to the original video (distance features). A total of 8 M parameters are used in this stage by the two CNNs.
• To address computational constraints, dimensionality reduction is applied. The raw and semi-raw feature vectors (originally 1280 dimensions per frame) are reduced to 20 dimensions using Neighborhood Component Analysis (NCA). Furthermore, a selected subset of specific landmarks is used for the distance-based features, resulting in a feature vector of size 26.
• A lightweight Bidirectional LSTM (Bi-LSTM) network is employed for the final classification on both datasets. The performance of this parameter-efficient architecture is benchmarked against state-of-the-art methods, where it is expected to demonstrate competitive performance. The Bi-LSTM classifier operates with only 8.68 M parameters.

As shown in Figure 1, the block diagram is structured into two main stages. The preprocessing stage involves operations such as facial detection, cropping, resizing and alignment. Subsequently, the feature extraction stage generates three distinct feature vectors: raw features (20 dimensions), semi-raw features (20 dimensions), and geometric features. The geometric features are derived by calculating the Euclidean distances between facial landmarks, which are then normalized to ensure inter-sample consistency, resulting in a 26-dimensional vector. The concatenation of these vectors yields a unified representation of 66 dimensions per frame.

Figure 1. Block diagram of the proposed processing pipeline.

2. Materials and Method

The models were evaluated using the data sets detailed in Table 1, two public emotion datasets were used in this study: CREMA-D (University of Southern California, Los Angeles, CA, USA) and RAVDESS (Ryerson University, Toronto, ON, Canada).

Table 1. This table shows the amount of data and that 80% of the training and 20% of testing of the total were used for each data set.

Class	# Videos (RAVDESS)	# Videos (CREMA-D)
Angry	192	1271
Calm	192	--
Disgust	192	1271
Fearful	192	1271
Happy	192	1271
Neutral	96	1087
Sad	192	1271
Surprise	192	--
Total	1440	7442

As shown in Table 2, the Bi-LSTM classification hyperparameters remained consistent across experiments, except for the number of epochs, which was adjusted specifically for each dataset.

Table 2. The configuration employed the Adam optimizer with a learning rate of $1\times {10}^{-5}$ and Cross-Entropy Loss. The training was conducted with a batch size of 128 for 30 epochs on the CREMA-D dataset, and a batch size of 32 for 55 epochs on the RAVDESS dataset.

Model	Input	Hidden Neurons	Hidden Layers	Dropout Rate
Bi-LSTM	66	512	2	0.2

3. Results

As shown in Table 3 and Table 4, the proposed method achieves competitive results despite its significantly lower computational complexity. It is crucial to highlight that our model requires substantially fewer parameters than the state-of-the-art approaches included in the comparison. Furthermore, a key advantage of our framework is that it relies exclusively on visual signals, whereas other competitive methods are multimodal, and utilize both image and audio data.

Table 3. Performance evaluation of state-of-the-art methods using Weighted Average Recall (WAR) and Unweighted Average Recall (UAR) on the CREMA-D dataset.

Method	Modality	WAR	UAR
Ghaleb et al. [1]	Audio + Video	66.80	-
Lei et al. [2]	Video	64.76	64.68
Ghaleb et al. [3]	Video	51.70	-
	Audio + Video	67.20	-
Proposed	Video	58.40	58.25

Table 4. Performance evaluation of state-of-the-art methods using Weighted Average Recall (WAR) and Unweighted Average Recall (UAR) on the RAVDESS dataset.

Method	Modality	WAR	UAR
Ghaleb et al. [1]	Audio + Video	60.50	-
Ghaleb et al. [3]	Video	58.20	-
	Audio + Video	76.30	-
Sun et al. [4]	Video	75.01	75.15
Proposed	Video	83.00	82.81

Table 5 and Table 6 present the results of incrementally concatenating the feature vectors, showing the contribution of each addition to the overall performance.

Table 5. The following results quantify the individual contribution of each feature set’s dimensionality to the performance on the CREMA-D [5] database, utilizing the fixed hyperparameters detailed in the previous section.

Method	WAR	UAR
Raw features	51.34	51.42
(Raw + distances) features	53.83	53.86
(Raw + distances + semi-raw) features	58.40	58.25

Table 6. The following results quantify the individual contribution of each feature set’s dimensionality to the performance on the RAVDESS [6] database, utilizing the fixed hyperparameters detailed in the previous section.

Method	WAR	UAR
Raw features	70.49	70.39
(Raw + distances) features	75.35	75.28
(Raw + distances + semi-raw) features	83.00	82.81