Analysis and Comparison of Machine Learning-Based Facial Expression Recognition Algorithms

Abstract

With the rapid development of artificial intelligence technology, facial expression recognition (FER) has gained increasingly widespread applications in digital human generation, humanoid robotics, mental health, and human–computer dialogue. Typical FER algorithms based on machine learning have been widely studied over the past few decades, which motivated our survey. In this study, we have surveyed the state of the art in FER across two categories: traditional machine learning-based (ML-based) and deep learning-based (DL-based) approaches. Each category is analyzed based on six subcategories. Then, twelve methods, including four ML-based models and eight DL-based models, are compared to evaluate FER performance across four datasets. The experimental results show that in validation sets, the average accuracy of HOG-SVM is 50.12%, which is the best performance for the four ML-based methods; in contrast, Poster has an average accuracy of 75.98%, which is the best result obtained among the eight DL-based methods. The most difficult expression to recognize is contempt, with recognition accuracies of 10.00% and 40.06% for ML-based and DL-based methods, respectively. The accuracy of the ML-based method for identifying neutral expression is the highest at 35.25%; the DL-based method has the highest accuracy in identifying surprise at 69.56%. From the theoretical analysis and comparative experimental results of existing methods, we can see that FER faces challenges, including inaccurate recognition in complex environments and unbalanced data categories, highlighting several future research directions, especially those involving the latest applications of digital humans and large language models.

1. Introduction

With the rapid development of artificial intelligence technology, digital humans (meta-humans) have been applied to news anchors, virtual e-commerce live streaming, AI-assisted teaching, and humanoid robots. Generating appropriate expressions for digital humans and robots will improve their visual impact. Therefore, the recognition, analysis, and generation of digital human facial expressions are topics worthy of study. The recognition and analysis of facial expressions provide the technical basis for facial expression generation.

Facial expression is one of the most intuitive external manifestations of human emotions and psychological states, playing a vital role in human communication throughout history. Facial expression recognition (FER) is an important research direction in computer vision and pattern recognition, aiming to equip computer systems with the ability to interpret the emotional information in human facial expressions. In the field of human–computer interaction, it can enable computer systems to perceive users’ emotions more naturally and intelligently, thereby optimizing the interactive experience. For example, intelligent customer service can adjust the response strategy according to users’ expressions and enhance quality of service [1]. In medical health, it can assist doctors in diagnosing patients’ mental states and help with mental health assessment and disease monitoring [2]. In security monitoring, it can provide timely warnings of potentially dangerous behaviors by identifying abnormal expressions, ensuring public safety. In recent years, the generation of digital humans and the widespread use of humanoid robots have also made facial expression recognition applicable in a large number of scenarios.

Since the 20th century, research on facial expression recognition has been widely conducted. Early on, due to limitations in computer performance and image processing technology, research on facial expression recognition focused on developing basic theory and exploring simple approaches. With the rapid development of computer technology, especially the iterative advancement of image processing, pattern recognition, machine learning (ML), and related technologies, FER research has gradually shifted from traditional methods based on prior knowledge to deep learning (DL) models. Traditional methods often use geometric features, such as the relative position and contour shape of facial organs, or texture features, such as local binary pattern (LBP) and its variants, to extract the features of expression, and then combine support vector machine (SVM) and adaptive boosting (AdaBoost) classifiers to realize expression classification [3]. With the development of deep learning, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their derived models have been widely used. CNNs’ powerful image feature extraction capabilities allow them to automatically learn multi-level features from expression images [4]; RNNs can effectively process expression sequence data and capture dynamic changes in expression over time [5]. In addition, some hybrid learning (HL) methods integrating multiple features or models have emerged in the field of deep learning technology, such as cascaded networks and multimodal models [6], aiming to fully leverage the advantages of different technologies and improve the performance of expression recognition technologies [7]. The general history of facial expression recognition is shown in Figure 1.

Figure 1 shows the development timeline of FER, which can be summarized into the following key stages:

i.

Theory and data foundation (1971–2000): In 1971, Ekman and Friesen [8] proposed the basic emotion theory applicable across cultures, before establishing the facial motion coding system (FACS) in 1978, defining action units (AUs), which laid the theoretical foundation for subsequent algorithm annotation. To support data-driven research, the Jaffe dataset was released in 1998, and the CK dataset was released in 2000 [9], establishing an early standard for expression recognition in controlled environments.

ii.

Machine learning and Feature Engineering (2009–2012): This period focused on the effectiveness of manual features. In 2009, Shan et al. [10] demonstrated that combining local binary patterns (LBPs) with SVMs achieved a breakthrough performance on a Cohn–Kanade dataset. In 2010, the release of the extended CK dataset [11] provided more complete annotations for action units and emotion, and these factors remain important benchmarks.

iii.

Deep learning and field challenges (2013–2019): The release of FER2013 in 2013 marked a shift in research focus to large-scale “in the wild” data. Subsequently, Mollahosseini et al. [12] successfully applied the concept and other deep networks to FER. In 2017, the release of AffectNet and RAF-DB further developed the field to address the complexity of real-world expressions (e.g., complex emotions). Self-supervised learning (such as SimCLR) also began to affect the FER pre-training paradigm in around 2019.

iv.

Transformer and multimodal integration (2021-present): Since 2021, vision transformers (VITs) have been systematically introduced into FER to capture global dependencies. The EmoSet released in 2023 represents a trend toward larger-scale, richer visual emotion data. The current research frontier is gradually shifting toward multimodal fusion (images, voices, and physiological signals) and exploring privacy-preserving technologies.

Although FER technology has made significant progress, it still faces many significant challenges in achieving wider applications. One problem is the changeable real environment. Dramatic changes in lighting conditions, facial gestures from different angles, and partial occlusion caused by wearing masks and glasses will seriously compromise the accuracy of expression recognition [13]. Another challenge is the richness and diversity of human expressions. In addition to basic expressions such as happiness, surprise and sadness, there are many subtle and fuzzy composite expressions, which are difficult to identify [14] accurately; the differences in the facial structure and muscle movement patterns of different individuals, as well as the diversity of expression and understanding in individuals from different cultural backgrounds, all add difficulty to the construction of a unified and accurate expression recognition model [15]. In addition, reducing the model’s parameter count and floating-point operations is a challenge. The lightweight model can be better deployed on some edge devices, reducing deployment costs [16]. Therefore, in the absence of a unified standard method, in this study, we survey the existing methods to help researchers further promote the development of expression recognition technology, as well as to select appropriate methods according to different needs. Unlike most existing reviews [17,18,19,20,21] that focus on text analysis, i.e., comparing data, conclusions, method names, ideas, and performance indicators, we add a comprehensive experimental comparison. The similarities and differences between our review and these existing reviews are listed in

Table 1. A comparison of our survey with 5 survey references.

Ref.	Year	Technique Review	Datasets	Future Directions	Comp Exp.
[17]	2023	FER steps: detection, feature extraction, augmentation	11	Mapping of the various elements involved in fulfilling the task of facial expression recognition in different studies	No
[18]	2024	emotion theories; recognition pipeline; traditional methods; DL-based; applications	7	Cross-cultural; real-time emotion recognition; multimodal emotion; explainable AI	No
[19]	2024	evolution timeline; pro-deep learning with pipeline; DL-based	21	Human identification mechanism; accelerated model; multiple modalities; thermal infrared imaging; model reduction; high performance on “in-the-wild” dataset	No
[20]	2024	FER basics; FER applications; IoT; Cloud computing	0	Compound emotions; cross cultural models; multi-pose FER; enhancing dataset; noise labels	No
[21]	2025	ML-Based; DL-Based; HL-based	7	Provide more inclusive and diverse datasets; improve precision; mitigate bias; unsupervised and self-supervised learning methods.	No
Ours	--	ML-Based; DL-Based. Both reviewed with 6 subcategories	4	Dataset enhancement; high-accuracy model; combining with a large language model; cross-dataset applications	Yes

Note: “Datasets” represents the number of datasets reviewed in each reference; “Comp exp.” is an abbreviation of comparative experiment.

. As shown in the table, the highlights of this study include a detailed literature review, algorithmic comparisons across multiple datasets, and suggestions for future research directions using large language models (LLMs).

The main contributions of this study include the following three points:

• We review some typical facial expression recognition methods by category, based on the theoretical ideas and modeling tools of the algorithms. The selection criteria for a reference to be included in this review are twofold. The first is that it selects at least one representative method from each category or subcategory. The second is that it selects a reference that proposed a method that was significant or showed significant technological progress.
• We compare the performances of 12 FER algorithms based on 4 outdoor facial expression datasets. Comparisons of experiments using the same dataset on the same computer are lacking in many existing review articles.
• We propose potential future research directions in the field of FER, especially by combining LLMs and digital human technology, based on summarizing the gap between the existing methods and the future demand of technology.

The structure of this paper is as follows: The second section introduces the standard preprocessing technology used for facial expression data. In the third section, the existing expression recognition methods are reviewed, including traditional feature extraction methods and deep learning methods. In the fourth section, several typical FER methods are summarized, and comparative experiments are conducted to reveal the performances of these methods. Finally, the fifth section summarizes the conclusion and points out future research directions.

2. Preprocessing

In FER, data preprocessing can improve model performance. Due to the poor quality of public facial expression datasets, problems such as face occlusion, non-uniform lighting (too dark or too bright), and blurred or low-resolution images often occur, significantly affecting the accuracy of feature extraction and classification. Therefore, it is usually necessary to preprocess the original image to reduce noise interference and improve data consistency. Here, we present several image preprocessing methods used in facial expression recognition.

Face detection and alignment are used to locate the face and align the positions of facial features (such as the eyes and mouth corners) [22]. Zhang et al. [23] proposed a multiple-task cascaded convolutional network (MTCNN) for face detection and alignment. This method achieves robust detection across multiple poses and complex lighting conditions by refining candidate boxes iteratively and combining face detection and keypoint localization. Wang et al. [24] introduced the image inpainting and the structural similarity index measure (SSIM) method to address the occlusion problem in expression images. They used completion technology based on structural constraints to fill the occluded area and ensured the overall consistency of the repaired image and the original face through structural similarity.

Normalization and scale adjustment are used to normalize the input image. After normalization, these input images are resized to a fixed size and normalized to a uniform gray range (e.g., [0,1] or a standard normal distribution), thereby avoiding training instability caused by resolution differences or inconsistent pixel value ranges. Cootes et al. [25] used the active appearance model (AAM) to realize the geometric normalization of the face. By jointly optimizing the shape and texture models, this method aligns different face images to a unified reference coordinate system, enabling comparisons of changes in expression in a structured space.

Illumination and contrast enhancement can be implemented using histogram equalization, gamma correction, or adaptive brightness adjustment, thereby alleviating feature distortion caused by changes in illumination intensity. Pizer et al. [26] used adaptive histogram equalization to improve image visual quality by enhancing local contrast. This method can maintain good texture details across different lighting conditions, thereby alleviating the interference of lighting differences on expression feature extraction.

Image enhancement and denoising can be achieved using smoothing and super-resolution reconstruction. Tomasi and Manduchi [27] proposed a bilateral filtering method combining spatial-domain and pixel-intensity-domain weights to remove noise while preserving image edges. When applied to facial images, this method can effectively reduce blur and noise and improve the identifiability of facial texture features.

In general, reasonable preprocessing can not only effectively alleviate adverse factors such as occlusion, illumination, and blur but also enhance the discriminability of feature expression to a certain extent, thereby improving the facial expression recognition performance of subsequent traditional machine learning or deep learning models.

3. Facial Expression Recognition Method

This section reviews the FER methods developed and applied in the past 20 years. These methods can be divided into two categories according to whether it is necessary to actively extract features: methods based on prior features (machine learning-based methods) and methods based on deep learning (deep learning-based methods). The former type relies on researchers to design features (such as facial texture and organ position) and extract features based on prior knowledge to recognize facial expressions, such as SVM algorithm, which offers fast speed and strong interpretability when constructing classifiers. However, its effect depends on the characteristics of the data and the design, and its generalizability is generally weak. The latter type is based on deep learning and can automatically learn features, such as a convolutional neural network, offering high recognition performance but weak interpretability and a high parameter learning cost. The main progress in the development of the FER methods is outlined in the following subsections. Although the models selected overlap with the above survey references, our survey covers both classic manual features and the latest architectures from 2024 to 2025, providing an effective technical supplement to the existing survey literature.

3.1. Methods Based on Prior Features

The prior features listed here refer to the statistics constructed by researchers based on specific theories, mainly including wavelet transform (WT), local binary patterns (LBPs), histogram of oriented gradients (HOG), the gray-level co-occurrence matrix (GLCM), local phase quantization (LPQ), and scale-invariant feature transform (SIFT). The methods are described in detail below. Figure 2 shows the general traditional machine learning method flow for classification tasks.

3.1.1. Wavelet Transform-Based Method

Wavelet transform has a good effect in terms of extracting the texture features of facial expressions, so it is also used in expression recognition. For example, using Gabor wavelets, a key complex function of the method is

$$\begin{array}{c}G\left(x,y;\lambda ,\theta ,\psi ,\sigma ,\gamma \right)=\exp \left(-{\displaystyle \frac{x{\mathrm{’}}^2+{\gamma }^{2}y{\mathrm{’}}^2}{2{\sigma }^2}}\right)\cdot \exp \left(i\left(2\pi {\displaystyle \frac{x\mathrm{’}}{\lambda }}+\psi \right)\right)\end{array}$$

(1)

where $x,y$ are the coordinates of pixel points; $\lambda$ is the wavelength of the sine wave, where the smaller the value, the finer the detection; $\theta$ is the direction of the filter; and $\psi$ is the phase offset. When $\psi$ = 0, it represents the even symmetric filter, and when $\psi ={\displaystyle \frac{\pi }{2}}$, it represents the odd symmetric filter; $\sigma$ is the standard deviation of Gaussian function, which controls the size of the filter window, while $\gamma$ is the spatial aspect ratio.

We searched for articles from 2000 to 2024 on the “IEEE Xplore” platform using the keyword “facial expression recognition” and obtained 15,924 results. Then, we added the keyword “Gabor” to the advanced search function and obtained 1107 results. This shows that the Gabor filter plays a role in expression recognition due to its excellent texture analysis capabilities across multiple scales and orientations. In 2008, Bashyal et al. proposed a method of facial expression recognition using Gabor wavelet and learning vector quantization [28]. The method noted that the two-dimensional Gabor function is a plane wave with wave factor $k$, which is limited by a Gaussian envelope function with a relative width $\sigma$:

$$\psi \left(k,x\right)={\displaystyle \frac{k^2}{{\sigma }^2}}\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\displaystyle \frac{k^2{x}^2}{2{\sigma }^2}}\right)\left[\exp \left(\mathrm{i}k·x\right)-\exp \left(-{\displaystyle \frac{{\sigma }^2}{2}}\right)\right]$$

(2)

where $\sigma$ is set to $\pi$ when the image has a resolution of 256 × 256. A set of discrete Gabor kernels is used, which are the spatial frequencies $k={\displaystyle \frac{\pi }{4}},{\displaystyle \frac{\pi }{8}},{\displaystyle \frac{\pi }{16}}$ and six different directions from 0° to 180° (every 30°). A filter group consisting of 18 filters is obtained. The filter group is applied to 34 target points of the image to obtain the feature vector. Then, principal component analysis (PCA) is used to reduce the dimension. Finally, classification is performed using learning vector quantization.

Zhou et al. [29] proposed a quaternion wavelet transformer (QWTR) in 2025. This method is oriented toward facial expression recognition in a complex field environment. It aims to address the decline in recognition accuracy caused by factors such as illumination changes, head pose deviations, and occlusions. This method combines quaternion mathematical theory with transformer structure, and the overall framework is composed of three parts: quaternion histogram equalization (QHE) is used to enhance image brightness and contrast and maintain color structure; quaternion wavelet transform-based feature selection (QWT-FS) extracts high-quality emotional features by decomposing and filtering the most relevant wavelet coefficients; quaternion-valued transformer (QVT) integrates a quaternion CNN and a multi-head attention transformer module to jointly model facial emotional information from local and global levels. The experiment was carried out on four typical FER in-the-wild datasets (RAF-DB, AffectNet, SFEW, and ExpW). QWTR surpassed the current state-of-the-art model across many evaluation metrics, demonstrating excellent robustness and generalization. Particularly in challenging scenes such as illumination changes, skin color differences, occlusions, and negative postures, the model maintains stable, efficient recognition performance.

3.1.2. Local Binary Pattern-Based Method

Local binary pattern (LBP) is also used to extract texture features. The idea behind this method is to describe the local texture features of an image by comparing the gray values of each pixel with those of its adjacent pixels. Let the central pixel be $\left(x_c,y_c\right)$ and its corresponding gray value be $g_c$. The gray value of the $P$ adjacent pixels of the pixel is $g_i\left(i=\mathrm{1,2},\dots ,P\right)$:

$$\begin{array}{c}{s}_i=\left\{\begin{array}{c}1,g_i>g_c\hfill \\ 0,ortherwise\end{array}\right.\end{array}$$

(3)

It is referred to as the 0–1 encoding of the ith-nearest-neighbor pixel. Then, it arranges the 0–1 codes of the $P$ adjacent pixels in clockwise or counterclockwise order to obtain a binary code and convert it to decimal to obtain the LBP value of the central pixel. At present, many improved versions of the traditional LBP method have been developed, such as circular LBP and rotation-invariant LBP. In facial expression recognition, the face can be divided into multiple regions, LBP histograms are created on each region, and then histograms are concatenated to form feature vectors.

Benniu et al. [30] combined LBP (local binary pattern) and ORB (Oriented FAST and Rotated BRIEF) features for facial expression recognition on the CK+, JAFFE, and MMI databases, demonstrating that this feature extraction method was effective. ORB comprises four steps: fast detection of key points, Harris corner response filtering, calculation of key point directions, and a brief description of the image around each key point. First, let $p$ be a candidate corner, $t$ be the intensity threshold set in advance, and $p_i(i=\mathrm{1,2},\dots ,n)$ be $n$ consecutive pixels around $p$. $g_p$ is the gray value of point $p$. Similarly, $g_{p_i}$ is the gray value of point $p_i$. If $\left|g_{p_i}-g_p\right|>t$ for all $p_i$ values, then point $p$ is a corner. Given the possibility of overlapping corners, a non-maximum suppression strategy is adopted. Next, the main direction of the feature point, that is, the corner point, is calculated. Then, the region around the key points is described using the brief descriptor. Next, select several pairs of pixels around the feature points, compare their gray values, and express the results with 0–1. Finally, a binary code is formed.

3.1.3. Directional Gradient Histogram-Based Method

The histogram of oriented gradients (HOG) is often used to describe the local characteristics of an image by counting the distribution of gradient directions in a local region.

Firstly, the image is preprocessed to convert the color image into a gray image. Then, gamma correction is performed, that is, a power transformation is performed on each pixel value. Let $I(x,y)$ be the value of the pixel $(x,y)$. After gamma transformation, the new pixel value is $I’(x,y)={I(x,y)}^{\gamma }$. Then, the Sobel operator is used to calculate the gradients $G_x$ and $G_y$ of the image in the horizontal and vertical directions. Then, the gradient amplitude $G(x,y)$ is obtained by $G\left(x,y\right)=\sqrt{{G_x}^2+{G_y}^2}$. According to $\theta =\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}{\displaystyle \frac{G_y}{G_x}}$, calculate the gradient direction. The image can be divided into several patches, usually $8\times 8$ or $16\times 16$ pixels. In each patch, the gradient direction of each pixel is calculated and divided into nine intervals, either 0° to 180° or 0° to 360°. For each pixel, the gradient direction is used to determine the corresponding interval, and the gradient amplitude is added to it; finally, a nine-dimensional feature vector is formed. Pierluigi Carcagnì et al. [31] verified the practicability of the HOG feature extraction method and discussed how to adjust HOG parameters to make it more suitable for the facial expression recognition task.

3.1.4. Gray-Level Co-Occurrence Matrix-Based Method

For any pixel (x, y) in an image and another point $(x+a, y+b)$ with a certain distance from it, let the gray value pairs of these points be $(g_1,g_2)$. Assuming that the gray values are divided into $k$ intervals, there are $k^2$ intervals of such gray value pairs. Let $(x, y)$ traverse the entire image, count the number of occurrences of each gray value pair, and arrange it into a square matrix. The number of occurrences is normalized to the probability of the occurrence of each gray value pair. Such a square matrix is called a gray-level co-occurrence matrix. Let $(a, b)$ take a different combination, and then the gray-level co-occurrence relationships of pixel pairs at different distances and directions can be counted.

In 2014, Murty et al. [32] proposed a facial expression recognition method based on the fusion of the gray-level co-occurrence matrix (GLCM) and distinctive local binary pattern (DLBP) based on a $3\times 3$ first-order compressed image (FCI), which is used to accurately recognize seven kinds of expressions neutral, happy, sad, surprised, angry, disgusted, and fearful. The method consists of three steps: Firstly, $5\times 5$ subimages are compressed into $3\times 3$ subimages to retain key information. Secondly, two DLBPs (the sum of the upper-triangle LBP, SUTLBP, and the sum of the lower-triangle LBP, SLTLBP) are extracted from the triangular pattern of the upper and lower parts of the $3\times 3$ subimage. Finally, GLCM is constructed based on DLBP, and characteristic parameters such as contrast, uniformity, energy and correlation are calculated for recognition. This method overcomes the generalization problem caused by the unpredictable distribution of facial images in real-world environments, a limitation of traditional statistical methods (such as PCA and LDA), and addresses the limitations of traditional LBP due to lighting, thereby improving classification performance by combining structural and statistical features. On the database containing 213 female facial expression images, the k-nearest neighbor classifier (k = 1) achieved an accuracy of 96.67%, significantly better than those of the two-tier perceptron architecture developed by Zhengyou Zhang [33] and the expression analysis method developed by De la Torre F et al. [34], and verified its effectiveness.

3.1.5. Local Phase Quantization-Based Method

By analyzing the phase information of the local region, detailed features of the image can be captured. Compared with the direct use of pixel values or other amplitude-based feature descriptors, phase information is more robust to illumination changes and image blur. This is because illumination changes mainly affect the image amplitude, rather than its phase. Local Phase Quantization (LPQ) is based on quantizing the phase information of the local Fourier transform. The specific calculation process is as follows: For a fixed-size window, the pixels within it are subjected to the Short-Time Fourier Transform (STFT). Let the image be $f\left(x\right)$, the pixel be $x$, and its neighborhood be $N_x$. Then, the local Fourier transform of pixel $x$ is

$$F\left(u,x\right)=\sum _{y\in N_x}f\left(y\right).\omega \left(y-x\right).e^{-j2\pi u^{T}y}$$

(4)

where $\omega \left(y-x\right)$ is the weighted window parameter, which is the commonly used Gaussian window, and $u$ is the frequency vector. In most cases, four frequencies, recorded as $u_1,u_2,u_3,u_4$, are selected. Let

$$\begin{array}{c}{F}_x=\left[\begin{array}{c}F\left(u_1,x\right),F\left(u_2,x\right),F\left(u_3,x\right),F\left(u_4,x\right)\end{array}\right]\end{array}$$

(5)

For each frequency point $u_k$, the symbols of the real and imaginary parts of the Fourier transform are calculated. Let

$$\begin{array}{c}{q}_j=\left\{\begin{array}{c}1 if{ g}_j\ge 0\\ 0 else\hfill \end{array}\right.\end{array}$$

(6)

where $g_j$ is the j-th term of the vector $G\left(x\right)=\left[\mathrm{R}\mathrm{e}\right\{F_x\},\mathrm{I}\mathrm{m}\{F_x\left\}\right]$. Thus, an eight-bit binary number is obtained and converted into a decimal number. This decimal number is the LPQ value of pixel $x$. The image is then divided into several regions. The LPQ values for each region are calculated, and a histogram of the statistics is generated. Finally, the LPQ feature of the image is obtained by connecting the histograms of each region. At present, LPQ features have also been improved and varied.

Jiawaili et al. proposed a new facial expression recognition algorithm based on rotation-invariant local phase quantization (RI-LPQ) and sparse representation [35]. RI-LPQ was used to extract expression features, combined with the sparse representation-based classification (SRC) method. The test image was represented as a linear combination of training images, and the expression was determined using sparse representation residual analysis. Compared with 2DPCA+SVM, LDA+SVM, and other algorithms, the new method achieves better performance and recognizes facial expression images with different occlusions.

Nikan et al. proposed a face recognition method combining multi-scale LPQ and multi-resolution LBP [36]. First, the image is preprocessed to obtain a representation that is insensitive to illumination changes. Then, the image is divided into small sub-blocks, and LPQ and LBP features are extracted from them. By combining score-level and decision-level features, recognition performance under severe illumination changes is improved.

3.1.6. Method Based on SIFT

SIFT is a classic local feature extraction algorithm that was proposed by Lowe in 1999 [37] and further improved in 2004 [38]. Its core advantage lies in its robustness to image scaling, rotation, and illumination changes, as well as its good adaptability to angle changes and noise. In this method, the image is convolved with Gaussian kernels of different scales to generate a series of blurred images, and the differences between adjacent blurred images are calculated to obtain difference-of-Gaussian (DoG) images. In these DoG images, the local maxima or minima present within a 26-neighborhood area around each pixel at the current scale and adjacent scales are searched for candidate key points. Then, the gradient direction histogram in the neighborhood of the key points is calculated. These histograms are combined into a 128-dimensional feature vector and normalized. The SIFT feature is widely used in facial expression recognition. For example, Berretti et al. [39] proposed an expression recognition method based on a 3D facial model, achieving a recognition rate of 78.43% on the BU-3DFE dataset by automatically detecting and combining key points and SIFT descriptors using an SVM classifier. In addition, Shi et al. [40] proposed an improved SIFT algorithm to enhance the robustness of expression recognition under different lighting and pose conditions through shape decomposition and feature point constraint.

3.2. Deep Learning-Based Method

The deep learning-based method generally does not require researchers to design statistical features and directly uses the original image as input. Researchers mainly focus on the model architecture. The following are nine typical convolutional neural network models for facial expression recognition. Figure 3 shows the general process of expression recognition using a deep learning method.

3.2.1. Facial Expression Recognition Based on Traditional CNN

In recent years, various classic CNN architectures (such as VGGNet, ResNet, and Inception) have been introduced into the field of expression recognition, leading to significant performance improvement. In 2023, Luo et al. [41] proposed an improved CNN model by optimizing the convolution activation function and the classification layer loss function. Using the convolution activation function, the authors comprehensively consider the pixels of the facial expression image, the convolution operation times, and the pooling operation times and attain highly integrated feature extraction. Using the classification loss function, the weights and bias values of the convolution layer, pooling layer, and classification layer are defined, enhancing the flexibility of the model. In the experimental part, three gradient descent algorithms (e.g., Full Gradient Descent, FGD; Stochastic Gradient Descent, SGD; Mini-batch Gradient Descent, MGD) and six traditional models (LBP+SVM, CSLBP/RILPQ+SVM, DeeID, Deep Face, LeNet-5, and the improved CNN model) are proposed. The experimental results show that the improved algorithm achieves 87% classification and 88% recognition accuracy, outperforming other comparison methods. In 2024, Shehada et al. [42] proposed two methods to improve facial expression recognition performance. The first method combines negative emotions (anger, fear, sadness, disgust) into a single category to simplify the classification task. The second method introduces diverse datasets. In terms of training and testing, the authors used 80% of the FER-2013, IMDB, AffectNet, and RAF-DB dataset samples for training; 20% of the FER-2013 samples for validation; and all CK+ samples for testing, achieving 90.9% and 94.7% accuracy on the FER-2013 and CK+ datasets, respectively.

However, challenges remain in distinguishing similar negative expressions, such as anger and disgust. In 2024, Adiga et al. [43] proposed that, based on image preprocessing, a multi-layer CNN be used to automatically extract facial features, such as edges, textures, and shapes. During training, the probability of each emotion is computed using the softmax activation function, and the model is optimized with the Adam optimizer or the SGD optimizer using the cross-entropy loss function. The experimental results show that the model achieves high accuracy and generalization on the FER-2013 dataset, especially for happy, sad, surprised, and other emotions.

3.2.2. Residual Network-Based Method

Deep residual learning addresses the degradation problem during deep neural network training. This method uses the core strategy of residual mapping and shortcut connection. In expression recognition, ResNet is often used as a benchmark model for comparison with other methods. At the same time, ResNet is often used as a part of many neural networks to increase the depth of the network, enabling the model to achieve better results.

Istiqomah A A et al. [44] used the pre-trained ResNet-50 network as the feature extractor in the expression recognition task and introduced transfer learning to improve the performance of the model; the Kaggle public facial expression dataset was used, with 80% used for training and 20% used for testing, and edge detection and region detection were used in preprocessing, with an accuracy of 99.49% and an F1 score of 99.60%. Jiahang Li [45] systematically compared the performances of different depth variants of ResNet (ResNet-18, 50, 101, and 152) and the attention-enhancement variant SE-ResNet. They conducted experiments on the FER-2013 dataset and used image enhancement techniques (e.g., random cropping, horizontal flipping, and brightness and contrast adjustments) to improve the model’s generalization. The results showed that SE-ResNet-34 with medium depth achieved the best performance, with an accuracy of 70.80%, better than those of SE-ResNet-50 and SE-ResNet-101 with deep depth. Roy et al. [46] proposed ResEmoteNet, which uses a CNN to extract hierarchical features, employs squeeze-and-excitation (SE) blocks to enhance key facial feature channels selectively, and combines residual blocks to alleviate the problem of deep network gradient disappearance and improve the learning of complex features. On the open-source datasets FER-2013, RAF-DB, AffectNet-7, and ExpW, the accuracy of ResEmoteNet [46] was 79.79%, 94.76%, 72.39%, and 75.67%, respectively, surpassing the accuracies of APViT, S2D, and other methods.

3.2.3. Attention Mechanism-Based Method

In facial expression recognition, an attention mechanism can help the model to focus on highly correlated facial regions, such as eyes, mouth corners, and eyebrows, especially under complex conditions such as subtle expressions, facial occlusion, or lighting changes, thereby improving robustness. This mechanism can be implemented using two methods: the first method is embedding traditional convolutional neural network modules (such as SE and CBAM) to enhance feature expression; the second method uses a full-attention architecture (such as a transformer) to completely replace convolutional operations. Using the former method, channel attention mainly focuses on “which feature channels are more important”, while the latter method focuses on “which positions in the image are more critical”. This mechanism improves the model’s discrimination ability and effectively alleviates feature redundancy.

Considering the illumination change, occlusion, micro-expression, and other factors present during facial expression recognition, Shiwei et al. [47] proposed a fine-grained method of global and local feature fusion, which achieved good classification performance through multi-stage preprocessing, such as super-resolution processing, illumination and shadow processing, and texture enhancement, before using a double-branch convolutional neural network to fuse global and local features.

Because the fuzzy annotation in the dataset will affect the performance of the model, Shu Liu et al. [48] proposed solving FER tasks through the label distribution learning paradigm and developed a two-branch adaptive distribution fusion (ADA-DF) framework, which uses the attention module to give high weights to the determined samples and low weights to the uncertain samples. The model achieved high accuracy on RAF-DB, AffectNet, and SFEW.

Amine Bohi et al. [49] proposed using ConvNeXt as the basic framework to enhance the robustness of the model with respect to scale, rotation, and translation changes and integrated an SE module to model the dependency between channels, highlight key features, and add self-attention regularization to enhance the balance and discrimination of feature expressions by balancing the attention weight. Ablation experiments showed that each module of EmoNext [50] (STN, SE, and SA regular terms) achieves significant performance improvement.

In 2025, Cao et al. [51] proposed a fuzzy perception recognition framework, COA, to address the issue of “wrong labels” and “fuzzy labels” in the sample. The COA framework introduces the emotion extraction and expression description modules. The former extracts rich semantic features from multi-scale and local attention mechanisms, respectively, via the coupled flow structure; the latter generates tag pairs by accurately predicting the probability, accurately describes the multiple emotional attributes of fuzzy samples, and better guides the model to fit mixed features during training. In the study, experiments were conducted on five datasets: RAF-DB, FERPlus, AffectNet, SFEW, and CAER-S. The results show that COA has significant advantages and robustness in dealing with a high proportion of fuzzy samples.

In 2024, Ngwe et al. [52] addressed the challenge of FER under natural and complex conditions. They proposed the lightweight network Patt-Lite, which integrates a patch extraction block and an attention classifier based on the MobileNetV1 architecture to improve recognition performance. Experiments on CK+, RAF-DB, FER-2013, FERPlus, and other datasets showed that Patt-Lite achieves 100% accuracy on CK+ and 95.05%, 92.50%, and 95.55% accuracy on RAF-DB, FER-2013, and FERPlus, respectively; these outcomes are significantly better than those of existing lightweight methods and some transformer models. The parameter is only 1.10 M, demonstrating the method’s robustness under complex conditions.

3.2.4. Transformer-Based Method

The original transformer model is a deep learning architecture based on a self-attention mechanism proposed by Vaswani et al. [53] in 2017. Unlike the traditional RNNs or CNNs, the transformer completely discards the cyclic structure and convolutional operations and relies solely on the self-attention mechanism to model dependencies between elements at different positions in the input sequence. It can simultaneously focus on global and local information in the sequence, significantly improving the model’s ability to understand context semantics. For facial expression recognition, the transformer-based model can effectively capture the subtle changes in key areas of the face (such as eyes, mouth, and eyebrows) and their correlation, focus on the discriminant features related to expression through the multi-level attention mechanism, and alleviate the influence of interference factors such as light and posture change.

In recent years, researchers have further improved the accuracy and robustness of the model in expression recognition by improving the input representation of the transformer (such as introducing facial key point information) and optimizing the attention mechanism (such as combining local attention and global attention), making it an important research direction in this field. In 2024, Raman et al. [54] proposed a multimodal facial expression recognition (MMFER) framework based on a hierarchical cross-attention graph convolutional network (HCA-GCN), which improves recognition accuracy and robustness by fusing multimodal data such as visual images, audio signals, and contextual information. Firstly, a CNN, a Short-Time Fourier Transform (STFT) and a pre-trained transformer are used to extract modal features. Then, a cross-modal graph structure is constructed by layering graph convolutions. First, the internal features of the single mode are refined, and then the dynamic correlation between modes is captured. Finally, the dynamic weighted features of the layers are fused via cross-attention, and the resulting features are classified using softmax.

In 2025, Zhou et al. [55] proposed a high-performance facial expression recognition model, HPC, including a high-dynamic-range preprocessing module (HDR preprocessing module), a parallel attention vision transformer structure (parallel attention vision transformer), and a co-expression head. During the preprocessing stage, the HDR preprocessing module optimizes the input image through local contrast and detail enhancement technology to enhance the adaptability of the model to light and detail changes. During the feature processing stage, the parallel attention visual transformer structure uses the multi-head self-attention mechanism encoder to capture and process the multi-scale facial expression features effectively and realizes the detailed analysis of subtle expression differences. In the feature fusion stage, the cooperative expression head uses the cooperative expression mechanism to efficiently process and optimize the features of different expression states.

In 2024, Liu et al. [56] proposed a framework based on human expression-sensitive cues (HESPs) to capture subtle expression dynamics by improving the clip model, addressing the challenge of clip models being too complex to recognize unknown expression categories in open-set video facial expression recognition (OV-FER). The framework includes three modules: the text prompt module introduces the learnable emotion template and negative text prompt to optimize the expression ability of the text encoder for known/unknown emotions; the visual cue module uses the class activation map to locate the expression sensitive area, combining with the time global pool to aggregate the video sequence characteristics; and the open-set multi-task learning scheme enhances the ability of cross-modal feature alignment and unknown category generalization through known category loss, negative representation loss, and global prediction loss.

In 2025, Mao et al. [57] proposed a lightweight, improved version of Poster++ to address the high computational cost of the Poster [58] model in FER, achieving improved efficiency and performance by optimizing the dual-flow design, cross-fusion mechanism, and multi-scale feature extraction. The model removes the image from the original dual-flow structure and routes it only to the landmark branch, while retaining the landmark-to-image branch to focus on key expression features. At the same time, the window-based Multi-Head Cross Self-Attention (W-MCSA) mechanism replaces traditional cross attention, reducing computational complexity from quadratic to linear and enabling feature fusion by directly extracting multi-scale features from the backbone network and combining them with the two-layer transformer module.

3.2.5. Graph Structure-Based Method

In recent years, GNNs have directly modeled non-European spatial data, making them particularly suitable for processing graph-structured data composed of facial key points, providing a new idea for expression recognition. The graph-based expression recognition process comprises three steps:

(1)

Construct the face image structure: Hundreds of facial key points are extracted using face key point detection methods (such as Dlib, MediaPipe, or OpenFace), and each key point is used as a graph node. Then, the adjacency matrix is generated using k-nearest neighbors (KNNs), full connectivity, or prior knowledge of facial anatomical structure, so that the relationship between nodes reflects local facial geometry.

(2)

Graph neural network modeling: After obtaining the graph structure, the graph convolutional network (GCN) or its improved version is used to extract the features of the graph data. Node features (such as keypoint coordinates, texture features, or motion features) are updated via the graph convolutional layer to transmit and aggregate information between nodes. This enables the model to capture interactions among local facial regions, thereby enhancing its ability to model complex expressions.

(3)

Classification and identification: After several layers of graph convolution, node features are usually transformed into a graph-level representation through global pooling. Finally, the fully connected layer and softmax classifier are used to output specific expression categories.

Many studies have demonstrated the effectiveness of the graph-based expression recognition method. For example, the FERGCN model proposed by Liao et al. [59] uses the face’s key points as nodes, constructs an adjacency matrix to capture the face’s geometric relationships, and performs feature propagation via a graph convolutional network. This method can capture the dependence between local areas of the face, especially in highly sensitive regions such as the eyebrows, eyes, nose, and mouth corners. Liu et al. [60] proposed a video expression recognition method based on graph convolution. Firstly, CNNs are used to extract local features from different regions of the face (e.g., eyes, mouth, and eyebrows), which are modeled as graph nodes. The edges between nodes connect region features, and GCN learns the interactions between regions.

Furthermore, to capture the temporal dynamics of expression, this method combines an LSTM to model temporal features and enables joint learning of space and time. Antoniadis et al. [61] proposed an emotion GCN model to address the significant co-occurrence and semantic dependence among expressions. This method constructs a graph in the label space, treats different emotion categories as nodes, and establishes edges based on statistical co-occurrence or semantic relationships, enabling GCN to explicitly model dependencies between categories. In addition, the method combines two expression modeling methods—classification and regression—and further improves the model’s generalization ability through multi-task learning.

3.2.6. RNN-Based Approach

An RNN is a type of deep learning model designed for processing sequential data. Its core feature is that, by introducing cyclic connections, the model can retain historical input information and apply it to the current time calculation, thereby naturally capturing temporal dependence in the sequence. RNNs are often used to process video sequences, dynamic images, and other information that contain a time dimension. For facial expression recognition, especially dynamic expression recognition, RNNs can effectively model the evolution of expression over time, capture the transition characteristics from neutral expression to target expression, and determine the temporal correlation of the movement of various regions of the face (such as the speed of rising corners of the mouth, the amplitude of frowning, etc.). For example, by inputting the video frame sequence into an RNN, the model can learn the dynamic patterns of different expressions on the time axis and then improve the recognition ability of complex expressions (such as micro expressions) or fuzzy expressions.

Furthermore, traditional RNNs suffer from gradient disappearance or gradient explosion, making it challenging to handle long-term sequence dependence. By introducing input, forgetting, and output gates to control the inflow and outflow of information, LSTM can effectively alleviate the gradient problem and better capture the characteristics of long-term time series. Gated Recurrent Unit (GRU) simplifies LSTM gating and reduces computational complexity while maintaining performance. These improved models have been widely used in facial expression recognition, especially when combined with dynamic information such as facial keypoint sequences and optical flow, thereby significantly improving the model’s sensitivity to temporal changes and recognition stability. Güler et al. [62] proposed a multimodal emotion recognition framework integrating electroencephalogram (EEG) signals and facial expressions to address the limitations of single-modality approaches. They used deep learning models such as GRUs, LSTMs, and transformers to explore the synergistic effect of multi-source data on emotion classification. This method also needs to solve the problem of modal dimension mismatch. In 2025, Yuanlun Xie et al. [63] proposed an end-to-end low-illumination FER framework (LL-FER), which realizes the collaborative optimization of image enhancement and expression recognition by jointly training the low-illumination enhancement network (LLENet) and the FER network.

4. Experimental Comparison

Although existing reviews cover a wide range of methods, most rely on data from original studies for comparison, lacking a unified experimental environment. The core contribution of this study is the establishment of a unified benchmark framework that rigorously compares 12 methods across different technical paradigms within the same hardware environment, data partitioning, and hyperparameter settings.

The selection of the 12 assessed models is based on three criteria. The first criterion is the historical baseline. We have chosen traditional manual feature extraction methods such as LBP, GLCM, and HOG. Although they are no longer SOTA, as classic representatives of the ‘pre-deep learning era’, they provide important baselines for benchmark testing, are representative, and reflect the evolutionary representativeness of FER technology. Based on this criterion, four traditional machine learning methods, namely GLCM-SVM [32], HOG-SVM [64], LBP-SVM [30], and LPQ-SRC [35], are selected.

The second criterion is paradigm coverage. The chosen method should cover the mainstream architectures of the deep learning era to ensure the breadth of method evaluation. According to this criterion, eight DL-based models, namely ADA-DF [48], CNN-SIFT [65], EmoNext [50], FERGCN [59], GCN [66], ResEmoteNet [46], ResNet [67], and Poster (Pyramid Cross-Fusion Transformer) [58], are selected. Among them, ResNet represents the classic convolutional neural network architecture. Poster and EmoNext are the most advanced attention-based methods currently available. FERGCN is a structured modeling approach for non-Euclidean data. ADA-DF is a specialized method for solving the problems of “class imbalance” and “label ambiguity”.

The third criterion is reproducibility. To ensure the fairness of this benchmark, we have selected models that provide official open-source code, thereby ensuring the reliability of the experimental results. This enables us to reproduce experiments with new datasets, rather than relying solely on the metric values reported in the original references.

According to the three criteria, 12 models are selected, and their publication years, method characteristics, and environmental configurations are listed in

Table 2. A comparison of the main information of 12 FER methods.

ID	Method	Year	Method Characteristics	Category
1	GLCM-SVM [32]	2014	Improved the LBP texture feature and the GLCM statistical feature.	ML-based
2	HOG-SVM [64]	2019	HOG feature combined with the support vector machine classifier.	ML-based
3	LBP-SVM [30]	2021	Combined LBP and ORB features.	ML-based
4	LPQ-SRC [35]	2012	LPQ feature combined with the sparse representation classifier.	ML-based
5	ADA-DF [48]	2023	Double-branch feature extraction combined with the adaptive distribution fusion mechanism.	DL-based
6	CNN-SIFT [65]	2017	Combined deep–shallow feature fusion, CNN and SIFT hybrid features, local invariance, and deep semantics.	DL-based
7	EmoNext [50]	2023	Combined ConvNeXt-based optimization and a tailored training strategy.	DL-based
8	FERGCN [59]	2022	Combined graph convolution modeling, facial topology representation, and local dependency capture.	DL-based
9	GCN [66]	2020	Gabor–CNN fusion, with directional and frequency texture extraction and fast feature extraction.	DL-based
10	ResEmoteNet [46]	2024	Residual structure integration and improved loss design.	DL-based
11	ResNet [67]	2016	Residual connections and alleviated gradient vanishing.	DL-based
12	Poster [58]	2023	Double-branch structure, landmark–image cross-fusion, and pyramid multi-scale fusion.	DL-based

.

4.1. Experimental Data

The comparative experiment was conducted using four datasets: AffectNet [68], FERPlus [69], FER-2013 [70], and RAF-DB [71]. The first two datasets both contain images of eight different facial expressions; the last two datasets both contain images with seven different facial expressions.

4.1.1. AffectNet

AffectNet (2017) [68] is one of the largest and most challenging facial expression recognition datasets currently in use, and it is used to support research on expression recognition, emotion analysis, and multidimensional emotion modeling. It was built by the research team of Ohio State University, who collected more than 1 million facial images from the Internet. These images were automatically crawled using a search engine with emotional keywords and manually labeled. AffectNet provides two annotation methods: The first method uses seven (or eight) categories of basic emotional labels, including neutral, happy, sad, angry, surprised, disgusted, fear and contempt. The second method annotates the continuous emotion dimension, that is, each image has the corresponding value of happiness and arousal, which can be used for regression tasks and emotion distribution modeling. About 450,000 images in the dataset have human-labeled labels, and more than 500,000 have automatically generated labels. AffectNet reflects the complexity of the “real world” in terms of emotional category imbalance, posture change, expression subtlety, etc. It is one of the most important benchmark datasets for evaluating deep learning expression recognition models.

4.1.2. FERPlus

FERPlus (face expression recognition plus 2016) [69] is an enhanced version of Microsoft Research relabeling based on the FER-2013 “wild” expression dataset produced in 2016: it retains the original 35,887 48 × 48 gray face images and the original training/public/private triad structure, but introduces the voting results of 10 crowdsourcing taggers for each image, as well as gives the count distributions for eight emotions (neutral, happy, surprised, sad, angry, disgusted, fear, contempt) and two auxiliary tags (“unknown” and “non face”); this ensures that either a single tag can be obtained with a majority vote or multi-tag or soft tag learning can be directly carried out using the probability distribution. Despite challenging low-resolution, real-world conditions, FERPlus has become a standard benchmark for studying noise-robust learning, expression distribution modeling, and depth network robustness.

4.1.3. FER-2013

FER-2013 (Facial Expression Recognition 2013) [70] is a widely used facial expression recognition dataset, which Kaggle initially released in the 2013 “challenges in representation learning” competition. The dataset contains 35,887 grayscale face images, each with 48 × 48 pixels. Its data were collected from Google image searches and cover seven basic emotions: anger, disgust, fear, happiness, sadness, surprise, and neutrality. The images are all face regions subjected to automatic detection and alignment. Although there is a certain degree of noise and annotation error, its large scale, standard division (28,709 training sets, 3589 verification sets, 3589 test sets), and wide use make it one of the most classic benchmark datasets in the field of expression recognition. FER-2013 is often used for training and evaluating deep learning models, such as CNNs, ResNets, and VGGs. It plays an important role in robustness research for expression recognition, emotion classification, multi-task learning, and related areas.

4.1.4. RAF-DB

RAF-DB (Real World Effective Faces Database 2017) [71] was proposed by Shan and Deng in the IEEE Transactions on Image Processing in 2017. It contains about 30,000 “in the wild” face images. Through crowdsourcing, each image is labeled by about 40 taggers on average, thus ensuring high reliability. The dataset includes 7 basic expressions (such as happy, sad, angry, etc.) and 12 complex expressions, and it provides face-keypoint information. It is widely used for facial expression recognition, complex emotion modeling, and robust learning and is an important benchmark dataset in the field.

Our experimental datasets were downloaded from Kaggle.

Table 3. Image information of the four expression datasets.

ID	Dataset	Number of Categories	Year	Number of Samples	Resolution
1	AffectNet	8	2017	450,000	Unfixed
2	FERPlus	8	2016	35,887	$112\times 112$
3	FER-2013	7	2013	35,887	$48\times 48$
4	RAF-DB	7	2017	15,339	$100\times 100$

provides a summary of the original dataset, and

Table 4. The number of images in each expression category in the four datasets.

Expression	AffectNet	FERPlus	FER-2013	RAF-DB
Anger	5500	9322	4953	3204
Contempt	4250	9030	0	0
Disgust	4303	9021	547	867
Fear	5500	9098	5121	2460
Happy	5500	9929	8989	355
Neutral	5500	12,994	6198	1619
Sad	5500	9449	6077	877
Surprise	5500	9450	4002	5957

provides a summary of the number of various expressions in the datasets used in our experiment.

4.2. Implementation Details

In our experiments, we conducted analyses across two distinct computational setups tailored to model complexity. For traditional machine learning methods (GLCM-SVM, HOG-SVM, LBP-SVM, LPQ-SRC), we utilized a local workstation powered by an Intel(R) Core(TM) i9-14900HX CPU @ 2.20 GHz with 16 GB RAM. Due to the complexity of deep learning models and the large amounts of training data, they were trained on remote high-performance servers equipped with dedicated accelerators. Specifically, ADA-DF, EmoNext, ResEmoteNet, ResNet, and Poster were trained on a single NVIDIA RTX 4060 GPU (CUDA 12.4, PyTorch 2.5.1), while CNN-SIFT, FERGCN, and GCN were executed on a single NVIDIA Tesla V100 GPU (CUDA 12.8/9.0.176, PyTorch 2.8.0/1.1.0). The software environment spanned Python versions 3.7 to 3.11, ensuring compatibility with the original implementations.

The experimental settings of the four traditional methods are as follows:

• GLCM-SVM: For the GLCM-SVM method, we implemented a hybrid feature extraction pipeline designed to capture texture statistics from quantized patterns. Input images were first transformed into Uniform LBP maps ($R=1, P=8$) and subsequently quantized into 15 discrete levels (DLBP). On these quantized maps, GLCMs were computed using three distances ($d \in \{1, 2, 3\}$) and four orientations ($0$°, 45°, 90°, 135°). Six statistical descriptors—Contrast, Dissimilarity, Homogeneity, ASM, Energy, and Correlation—were extracted and aggregated via their mean and standard deviation to form the final feature vector. The classification was performed using a linear SVM with a regularization parameter $C=1.0$ and a maximum of 5000 iterations, utilizing a random 80/20 train–test split strategy.
• HOG-SVM: The HOG-SVM approach utilized the histogram of oriented gradients descriptor implemented via OpenCV to capture local shape and edge information. Input images were resized to $128 \times 128$ pixels. We configured the HOG descriptor with a cell size of $8 \times 8$ pixels, a block size of $16 \times 16$ pixels, and a block stride of $8 \times 8$ pixels. Gradient orientations were discretized into nine bins. Given the high dimensionality of the resulting feature vector, we employed an SVM with a Radial Basis Function (RBF) kernel. The regularization parameter $C$ was set to 10, and the kernel coefficient $\gamma$ was set to ‘scale’. Similarly to the GLCM method, the dataset was split randomly, with 80% for training and 20% for validation.
• LBP-SVM: For the LBP-SVM method, we adopted a feature fusion strategy to combine global texture patterns with local keypoint descriptors. Texture features were extracted using uniform LBP ($R=1, P=8$) and represented as a normalized histogram. Simultaneously, ORB descriptors were computed with a maximum limit of 200 keypoints, which were then aggregated using mean pooling. These two feature sets were concatenated to form the final representation. Classification was conducted using a linear SVM ($C=1.0, max\_iter=5000$). Unlike the randomized splits used in other baseline methods, this experiment strictly followed the pre-defined training–validation splits provided by the respective datasets, with 80% for training and 20% for validation.
• LPQ-SRC: The LPQ-SRC method employed local phase quantization features, which are known for their robustness to blur and illumination variations. The LPQ descriptor was configured with a local window size of $3 \times 3$, a frequency estimation parameter $\alpha =1.0$, and a correlation coefficient $\rho =0.90$. The resulting feature codes were accumulated into a 256-dimensional histogram and normalized using a standard scaler. For classification, we utilized a sparse representation-based classifier (SRC). The overcomplete dictionary was constructed using all available training samples. The sparse coding problem was solved using the Orthogonal Matching Pursuit (OMP) algorithm with a sparsity constraint of $K=30$ non-zero coefficients, and the final class label was determined based on the minimum reconstruction residual.

The training superparameters of the deep learning models are listed in

Table 5. Comparison of 4 DL-based FER methods.

ID	Method	Optimizer	Learning Rate	Batch-Size	Epochs
5	ADA-DF [48]	Adam	0.001	64	100
6	CNN-SIFT [65]	Adam	0.001	64	100
7	EmoNext [50]	AdamW	0.0001	32	100
8	FERGCN [59]	Adam	0.001	32	100
9	GCN [66]	SGD	0.005	64	200
10	ResEmoteNet [46]	SGD	0.001	16	100
11	ResNet [67]	Adam	0.00025	32	100
12	Poster [58]	SAM	0.00004	32	200

Note: The model ID is the same as that in Table 4.

. We optimized the solver configuration for different architectures: most models (ADA-DF, FERGCN, ResNet) use the Adam optimizer, while GCN and ResEmoteNet use SGD. Poster uses a more advanced SAM (sharpness-aware minimization) optimizer to improve its generalization ability. The learning rate falls within $[4\times {10}^{-5},5\times {10}^{-3}]$ according to the convergence characteristics of the model. All models were trained for 100 or 200 epochs, and the batch size was set to 16, 32, or 64 to accommodate the video memory limit.

4.3. Analysis of Experimental Results

The following two sections compare the 12 FER methods listed in Table 2 in terms of performance and classification effect. The comparative experiment is based on the four datasets of AffectNet, FERPlus, FER-2013, and RAF-DB introduced in Section 4.1.

4.3.1. Performance Comparison of Various Models

The experimental results of four traditional methods and eight deep learning methods across four natural datasets are listed in

Table 6. The training and verification accuracy of 12 models on four datasets.

Method	Train					Val
	AffectNet	FERPlus	FER2013	RAF-DB	Ave.	AffectNet	FERPlus	FER2013	RAF-DB	Ave.
GLCM-SVM [32]	12.70	16.28	14.82	37.13	20.23	12.12	16.49	13.94	38.25	20.20
HOG-SVM [64]	99.45	99.63	99.78	100.00	99.72	34.88	51.18	47.23	67.17	50.12
LBP-SVM [30]	18.71	18.34	25.22	38.53	25.20	20.00	17.52	24.65	38.66	25.21
LPQ-SRC [35]	13.40	13.92	15.28	18.66	15.32	12.38	13.52	14.97	17.41	14.57
ADA-DF [48]	64.18	96.19	75.2	92.65	82.06	57.25	71.56	65.37	99.51	73.42
CNN-SIFT [65]	53.98	82.70	80.62	93.58	77.72	34.38	57.00	52.58	65.50	52.37
EmoNext [50]	59.34	90.21	73.05	75.25	74.46	56.75	76.03	75.68	82.87	72.83
FERGCN [59]	28.20	31.06	36.49	51.27	36.76	29.63	31.21	36.77	52.53	37.54
GCN [66]	70.11	96.09	94.40	94.30	88.73	59.25	68.42	69.38	85.03	70.52
ResEmoteNet [46]	86.79	98.57	98.42	98.85	95.66	50.88	63.12	58.90	78.75	62.91
ResNet [67]	58.14	86.73	76.14	92.46	78.37	33.62	47.69	39.26	57.75	44.58
Poster [58]	97.28	99.01	99.22	99.77	98.82	62.88	76.15	74.15	90.74	75.98

Note: The bold number indicates the maximum value of the column (optimal), while the underlined number represents the second largest value in the column (suboptimal).

. In this table, we can see that the depth model (ADA-DF [48], EmoNext [50], GCN [66], ResNet [67], ResEmoteNet [46], and Poster [58]) can generally achieve high accuracy on the training set. Some methods such as AffectNet and RAF-DB can even exceed 95% while also maintaining strong performance on the validation set, especially on the “in the wild” datasets, and the validation accuracy of the depth model is significantly higher than those of the traditional methods, showing good generalization ability and robustness to complex environments (such as different lighting, posture, and occlusion). In contrast, the traditional prior feature methods (CNN-SIFT [65], GLCM-SVM [32], HOG-SVM [64], LBP-SVM [30] and LPQ-SRC [35]) perform pretty well on the training set. However, the accuracy on the validation set is generally low, especially on large-scale, real-world images, which are difficult to adapt to expression diversity and label noise, leading to significant performance degradation. In addition, it is worth noting that a deep learning model must not follow the assumption of the more complex, the better. Although ResEmoteNet [46] has a large parameter count, its performance across multiple datasets is not the best, and it exhibits some overfitting. Moreover, although the traditional HOG-SVM [64] method performs nearly perfectly on the training set, its performance on the verification set is clearly insufficient, indicating its limited generalization ability.

In contrast, the ADA-DF [48] model achieves better recognition performance while maintaining low complexity through its double-branch structure, thereby verifying the effectiveness of constructing the loss function based on the actual label distribution. Particularly on the RAF-DB dataset, the model’s recognition accuracy is 89.44%, demonstrating strong generalization. However, there is still room for improvement in its performance on other datasets. The GCN [66] model achieves good results across all expression recognition categories. Moreover, its performance is stable. The effectiveness of the Gabor feature in expression recognition is verified.

In expression recognition, in addition to accuracy, the number of parameters (trainable params) and computational complexity (MACS) of deep learning models are important indicators for evaluating their lightweight degree. Despite the high computational efficiency and low cost of traditional machine learning methods, the classification effect is poor; this study focuses on comparing the structural complexity of eight deep learning models: ADA-DF [48], CNN-SIFT [65], EmoNext [50], FERGCN [59], GCN [66], ResEmoteNet [46], Poster [58] and standard ResNet [67].

Table 7. Parameter complexes of eight deep learning-based models.

Model	Trainable Parameters	MACs
ADA-DF [48]	22.05 M	1.72 G
CNN-SIFT [65]	7.48 M	7.48 M
EmoNext [50]	30.57 M	4.57 G
FERGCN [59]	4.872 K	0.593 M
GCN [66]	4.438 M	10.74 M
Poster [58]	71.8 M	3.95 G
ResEmoteNet [46]	80.24 M	4.35 G
ResNet [67]	57.91 K	193.3 M

Note: The bold number indicates the best one among the column.

shows significant differences in the number of parameters and calculations across different facial expression recognition models. High-performance deep networks such as ADA-DF [48], EmoNext [50], Poster [58], and ResEmoteNet [46] have tens of millions of parameters and G-level MACs. These models have strong expression capabilities and are suitable for training and reasoning on high-performance GPUs. The parameter count of the traditional convolutional network, ResNet [67], is relatively small. However, due to the computational characteristics of the convolutional operation, its MAC count reaches 193.3 M, which still imposes a significant computational burden. The graph convolutional models GCN [66] and FERGCN [59] exhibit lightweight characteristics, especially FERGCN [59], which has only 4.872 K parameters and 0.593 M MACs, making it very suitable for resource-constrained scenarios. CNN-SIFT [65], as a method combining a priori features with a shallow network, is at a medium level in terms of the number of parameters and calculations, taking into account specific performance and efficiency.

We take eight kinds of expressions as examples. The input image size for EmoNext and Poster is 3 × 224 × 224 pixels. In contrast, the GCN model uses an input image size of 1 × 90 × 90 pixels. Other models have an input image size of 3 × 120 × 120 pixels. Additionally, the parameters and floating-point operations of each model are compared.

Overall, there is an apparent trade-off between the capacity and computational cost of facial expression recognition models. A deep network with a high number of parameters and high MACs can provide stronger learning ability, but the cost of training and deployment is high; lightweight models such as FERGCN [59] occupy very low computing resources and are suitable for real-time or embedded applications, but their limited capacity may affect their performance in complex scenarios. Models representing a combination of the two, such as ResNet [67] and CNN-SIFT [65], strike a balance between performance and efficiency. Therefore, in practical applications, it is necessary to select appropriate models according to task complexity, computing resources, and real-time requirements.

4.3.2. Analysis of Identification of Each Category

We analyze the recognition accuracy of each model in each dataset. The recognition accuracy of a category is equal to the ratio of the number of correctly predicted pictures in the category to the total number of pictures in the category. The classification accuracies of each model on the validation sets of the four datasets are listed in

Table 8. The classification accuracies of each model on the validation set of AffectNet.

Model	Neutral	Happy	Sad	Surprise	Fear	Disgust	Anger	Contempt	Ave
GLCM-SVM [32]	0.18	0.12	0.09	0.16	0.05	0.12	0.12	0.13	0.12
HOG-SVM [64]	0.87	0.52	0.23	0.28	0.30	0.24	0.25	0.10	0.35
LBP-SVM [30]	0.11	0.41	0.22	0.24	0.21	0.00	0.40	0.01	0.20
LPQ-SRC [35]	0.05	0.08	0.00	0.04	0.83	0.02	0.02	0.02	0.13
ADA-DF [48]	0.62	0.78	0.48	0.57	0.62	0.56	0.52	0.43	0.57
CNN-SIFT [65]	0.36	0.37	0.27	0.19	0.15	0.01	0.23	0.05	0.20
EmoNext [50]	0.48	0.79	0.58	0.51	0.62	0.49	0.51	0.56	0.57
FERGCN [59]	0.17	0.74	0.17	0.44	0.25	0.02	0.40	0.06	0.28
GCN [66]	0.50	0.74	0.69	0.63	0.63	0.47	0.57	0.51	0.59
Poster [58]	0.56	0.63	0.55	0.67	0.68	0.70	0.64	0.65	0.64
ResEmoteNet [46]	0.37	0.74	0.44	0.61	0.47	0.53	0.49	0.42	0.51
ResNet [67]	0.28	0.51	0.30	0.34	0.31	0.26	0.36	0.33	0.34

Note: The bold number indicates the maximum value of the column (optimal), while the underlined number represents the second largest value in the column (suboptimal).

,

Table 9. The classification accuracies of each model on the validation set of FERPlus.

Model	Neutral	Happy	Sad	Surprise	Fear	Disgust	Anger	Contempt	Ave
GLCM-SVM [32]	0.22	0.12	0.13	0.09	0.14	0.14	0.11	0.14	0.14
HOG-SVM [64]	0.93	0.63	0.36	0.63	0.38	0.38	0.42	0.22	0.49
LBP-SVM [30]	0.50	0.06	0.03	0.24	0.04	0.12	0.14	0.17	0.16
LPQ-SRC [35]	0.02	0.04	0.22	0.32	0.03	0.03	0.35	0.01	0.13
ADA-DF [48]	0.82	0.91	0.72	0.80	0.83	0.59	0.78	0.47	0.74
CNN-SIFT [65]	0.86	0.59	0.50	0.65	0.52	0.49	0.52	0.35	0.56
EmoNext [50]	0.84	0.91	0.72	0.86	0.75	0.63	0.78	0.57	0.76
FERGCN [59]	0.18	0.68	0.03	0.49	0.31	0.21	0.17	0.36	0.30
GCN [66]	0.86	0.88	0.74	0.86	0.58	0.47	0.77	0.24	0.68
Poster [58]	0.76	0.91	0.60	0.78	0.89	0.85	0.62	0.81	0.78
ResEmoteNet [46]	0.80	0.86	0.64	0.79	0.59	0.57	0.63	0.28	0.65
ResNet [67]	0.64	0.62	0.37	0.58	0.51	0.47	0.40	0.32	0.49

Note: The bold number indicates the maximum value of the column (optimal), while the underlined number represents the second largest value in the column (suboptimal).

,

Table 10. The classification accuracies of each model on the validation set of FER-2013.

Model	Neutral	Happy	Sad	Surprise	Fear	Disgust	Anger	Ave
GLCM-SVM [32]	0.18	0.27	0.14	0.14	0.07	0.19	0.10	0.16
HOG-SVM [64]	0.88	0.64	0.19	0.53	0.23	0.33	0.21	0.43
LBP-SVM [30]	0.04	0.89	0.08	0.01	0.00	0.00	0.01	0.15
LPQ-SRC [35]	0.65	0.08	0.10	0.09	0.07	0.03	0.01	0.15
ADA-DF [48]	0.69	0.87	0.54	0.70	0.39	0.17	0.64	0.57
CNN-SIFT [65]	0.07	0.00	0.06	0.30	0.23	0.22	0.23	0.16
EmoNext [50]	0.74	0.95	0.58	0.82	0.68	0.64	0.69	0.73
FERGCN [59]	0.25	0.74	0.39	0.45	0.05	0.00	0.07	0.28
GCN [66]	0.65	0.88	0.59	0.78	0.52	0.52	0.66	0.66
Poster [58]	0.69	0.87	0.62	0.82	0.67	0.81	0.72	0.74
ResEmoteNet [46]	0.58	0.81	0.49	0.74	0.45	0.48	0.48	0.58
ResNet [67]	0.32	0.67	0.32	0.54	0.25	0.17	0.33	0.37

Note: The bold number indicates the maximum value of the column (optimal), while the underlined number represents the second largest value in the column (suboptimal).

and

Table 11. The classification accuracies of each model on the validation set of RAF-DB.

Model	Neutral	Happy	Sad	Surprise	Fear	Disgust	Anger	Ave
GLCM-SVM [32]	0.05	0.11	0.00	0.84	0.11	0.09	0.13	0.19
HOG-SVM [64]	0.90	0.25	0.08	0.89	0.49	0.44	0.56	0.52
LBP-SVM [30]	0.00	0.00	0.00	0.95	0.02	0.00	0.08	0.15
LPQ-SRC [35]	0.06	0.09	0.22	0.03	0.36	0.01	0.18	0.14
ADA-DF [48]	1.00	0.95	0.99	1.00	1.00	1.00	0.99	0.99
CNN-SIFT [65]	0.55	0.29	0.16	0.85	0.56	0.40	0.62	0.49
EmoNext [50]	0.84	0.43	0.63	0.93	0.81	0.64	0.79	0.72
FERGCN [59]	0.29	0.00	0.00	0.87	0.08	0.11	0.66	0.29
GCN [66]	0.83	0.62	0.57	0.94	0.78	0.76	0.87	0.77
Poster [58]	0.90	0.86	0.80	0.96	0.91	0.88	0.85	0.88
ResEmoteNet [46]	0.85	0.46	0.47	0.93	0.68	0.66	0.76	0.69
ResNet [67]	0.50	0.16	0.03	0.86	0.38	0.24	0.57	0.39

Note: The bold number indicates the maximum value of the column (optimal), while the underlined number represents the second largest value in the column (suboptimal).

.

Table 8, Table 9, Table 10 and Table 11 show that the deep learning method is clearly superior to the traditional prior-feature method in facial expression recognition as a whole. The depth model, represented by ResNet [67], ADA-DF [48], EmoNext [50], ResEmoteNet [46], Poster [58] and GCN [66], can achieve high recognition accuracy in most expression categories, especially in common expression categories such as happy, neutral, and surprise, which shows that the depth model has significant advantages in feature extraction and complex pattern modeling. In particular, the ADA-DF [48], Poster [58], and GCN [66] models achieve higher than 90% accuracy in some categories, reflecting their strong adaptability to high-quality annotations and grayscale images, as well as their ability to capture the fuzziness and diversity of expressions. From a global perspective, in “in the wild” datasets such as AffectNet and RAF-DB, the depth model can still maintain high performance. Recognition of some small sample categories, such as sadness, anger, and contempt, is slightly low. However, the overall accuracy is still significantly higher than those of traditional methods, reflecting the robustness of the depth method across complex scenes, varying lighting conditions, pose changes, and occlusions.

In contrast, traditional prior-feature methods (such as LBP-SVM [30], HOG-SVM [64], GLCM-SVM [32], CNN-SIFT [65], etc.) may achieve a certain level of accuracy on the training set. However, they are unstable on the verification set and have limited adaptability to rare expression categories and complex environments. The accuracy levels of some methods in a few categories are even below 20%, indicating that they struggle to capture the diversity and subtle changes in facial expressions fully.

Table 12. The average accuracies of the 12 methods in recognizing different facial expressions.

Models	Neutral	Happy	Sad	Surprise	Fear	Disgust	Anger	Contempt
ML-based	0.3525	0.2694	0.1306	0.3425	0.2081	0.1338	0.1931	0.1000
DL-based	0.5891	0.6631	0.4700	0.6956	0.5359	0.4694	0.5725	0.4006

Note: The bold number indicates the maximum value of the column (optimal).

shows the average accuracies of both the ML-based and DL-based methods in recognizing different facial expressions. The average accuracy is calculated as a simple average of the data in Table 8, Table 9, Table 10 and Table 11. In Table 12, we can see that the most difficult expression to recognize is contempt, with recognition accuracies of 10.00% and 40.06% for ML-based and DL-based methods, respectively. The accuracy of the ML-based method for identifying neutral is the highest at 35.25%; the DL-based method has the highest accuracy for identifying surprise, with 69.56%.

Overall, these results show that deep feature extraction and structured modeling methods can more comprehensively and effectively represent facial expression features, which is the mainstream and the most effective technical approach in current facial expression recognition research, and they also provide a solid foundation for further research on compound expression recognition, multimodal emotion understanding, and robustness improvement.

4.4. Discussion

Our experiment reveals a significant phenomenon: traditional machine learning methods (such as HOG-SVM and LBP-SVM) experience severe performance decline on natural scene datasets such as AffectNet and RAF-DB. The fundamental reason is the limitation of the feature extraction mechanism. Traditional descriptors are highly dependent on strict pixel alignment and local texture consistency. However, in the field, uncontrollable illumination, pose variations, and occlusion distort the distribution of these rigid features. In contrast, deep learning models (such as ResNet and Poster) extract high-level semantic features with spatial invariance via hierarchical convolutions or an attention mechanism, thereby adapting to complex environmental interference.

In addition, through in-depth analysis of accuracy differences across various categories (see Table 8, Table 9, Table 10 and Table 11), we observed that all methods performed better on RAF-DB in the “happy” and “neutral” categories than in the “sad” and “disgust” categories. This performance deviation directly reflects the dataset’s long-tail distribution (as shown in Table 4). Due to the scarcity of minority samples, the model tends to overfit the majority during training to reduce the global loss, resulting in a fuzzy decision boundary for the minority. This shows that simply improving the model architecture cannot completely solve this problem, and future research must be combined with data resampling or cost-sensitive learning strategies.

5. Conclusions and Future Work

In this study, we surveyed facial expression recognition models from recent decades that employ both machine learning and deep learning methods. Machine learning-based methods were reviewed across six subcategories: wavelet transform, LBP, HOG, GLCM, LPQ, and SIFT. Deep learning-based methods were reviewed across six subcategories: CNNs, ResNets, attention mechanisms, transformers, graph structures, and RNNs. In this way, the technical details of various methods over the past few years can be seen at a glance, a feature that sets this survey apart from many existing surveys.

We examined four datasets and compared twelve models through a series of experiments. Our comparative experiments demonstrate the following: (1) The results indicate that deep learning-based methods outperform traditional machine learning methods on the validation sets. (2) Models with the highest number of parameters or floating-point operations (MACs) do not always perform the best. For example, although ResEmoteNet has the largest number of parameters (80.24 M), its accuracy on RAF-DB is lower than that of Poster with fewer parameters (71.8 M). This discovery emphasizes an important insight: efficient architecture design, such as effective feature fusion or attention mechanisms, is more critical than simply increasing network depth or parameter size. (3) Class imbalance remains the most significant bottleneck restricting the improvement of FER accuracy. All evaluated models, regardless of their architecture, exhibit significant performance degradation in categories with few samples. This indicates that the majority of classes largely dominate the current performance improvement.

Based on the literature review and the results of our comparative experiments, future research can be further developed in the following areas:

For dataset enhancement, in view of the widespread category imbalance in facial expression recognition tasks, we should strengthen the modeling and enhancement of a small number of emotion samples; at the same time, it is necessary to consider the complex factors in the real scene, such as blur, occlusion (e.g., faces with masks), illumination change, etc., to test the robustness of a model. In addition, facial expressions are related to ethnic and cultural habits. The observer’s cultural background will significantly adjust their perception and judgment of expression [72], so it is necessary to establish culturally relevant facial expression datasets to improve the specificity of expression recognition. Cross-cultural datasets can also be established to examine the generalization of an FER method.

Regarding model accuracy, although existing models have achieved good performance in expression recognition, some fuzzy expressions (as shown in Figure 4) are still not recognized accurately, indicating that they require further improvement.

Noting the outstanding performances of large language models in assisting scientific research over the past two years, exploring efficient feature-extraction and fusion mechanisms for FER with large language models is a promising direction for the coming years. In addition, because the emotional information conveyed by the face is limited, we can develop a multimodal emotion recognition model that combines it with other types of data containing emotional information to improve recognition accuracy.

In the training strategy, we can leverage multi-task learning, transfer learning, and label distribution-based modeling further to improve a model’s cross-dataset and cross-scene generalization performance and enhance its practicability and adaptability.

In terms of application prospects, with the rapid development of digital human technology and humanoid robots, facial expression recognition, as one of the key technologies for natural human–computer interaction, will offer broad application value across related fields. For example, in intelligent education, expression recognition can be used to monitor learners’ emotional states in real time. In medical and health care, it can provide new means for the auxiliary diagnosis of emotional disorders and pain intensity [73,74]. In intelligent customer service and human–computer interaction, it helps to improve the naturalness and affinity of the system. Finally, in safety monitoring and driving assistance, it can provide support for abnormal emotion or fatigue detection.