Facial Expression Recognition Integrating Multi-Stage Feature Sparse Constraints and Key Region Graph Learning

Abstract

Current Facial expression recognition methods typically extract facial features indiscriminately, incorporating expression-irrelevant information that compromises recognition accuracy. To overcome this, we propose Multi-stage Feature Sparse Constraints (MFSC), a novel model that integrates a Multi-scale Attention-based Sparse Window Selection (MSAWS) mechanism with key region graph learning. Notably, MFSC operates without dependence on pre-extracted facial landmarks, enabling more flexible deployment. The MSAWS mechanism progressively filters redundant features through multi-stage sparse attention, adaptively selecting the most discriminative facial patches. The selected tokens are structured into a dynamic graph to model regional relationships via graph neural networks (GNNs). Critically, our framework further introduces a global-guided fusion module, which effectively integrates fine-grained local features from an IR50 backbone with the global topological features from the GNN through cross-attention. This integration enables complementary strengths, where local details are enhanced by global semantic context. Comprehensive experiments on RAF-DB, FER2013, and AffectNet-7 datasets demonstrate MFSC’s superior performance, achieving state-of-the-art accuracy of 92.31%, 76.21%, and 67.35%, respectively. These results validate the effectiveness of our approach in focusing computational resources on expression-salient regions while maintaining a lightweight and efficient architecture.

1. Introduction

In recent years, deep learning techniques have significantly advanced the development of knowledge graphs [1,2], recommender systems [3,4,5,6], computer vision [7,8,9], and autonomous driving [10]. Among them, convolutional neural networks (CNNs) have demonstrated strong performance in feature extraction due to their local receptive fields and hierarchical structures. Classical CNN models such as Visual Geometry Group Network (VGGNet) and Residual Network (ResNet) have been extensively adopted in facial expression recognition (FER) task, effectively capturing both low-level texture features and high-level semantic information from static images. To better adapt to the needs of expression recognition, researchers have proposed various improved CNN architectures, including attention-enhanced CNN, adaptive weight adjustment modules, multi-branch structures, and multi-scale fusion networks. These methods have achieved notable performance improvements on public FER datasets [11,12,13]. However, while CNNs are effective in extracting local features, their limited receptive field makes it challenging to model global contextual information effectively. To address this limitation, many researchers have introduced the Transformer architecture into visual tasks. Models like Vision Transformer (ViT) leverage self-attention mechanisms to capture global contextual dependencies and have demonstrated superior performance in tasks such as image classification and segmentation [14,15,16]. Considering the non-Euclidean structural relationships between facial regions, some scholars have attempted to introduce GNN into the FER task to further enhance the model’s performance [17,18].

Although the above-mentioned methods have achieved promising results in FER tasks, they share a common challenge in practical applications: during feature extraction, a large amount of irrelevant information in the image—such as hair and background—is often included, leading to redundant or even distracting features. This not only disperses the attention or feature responses but also weakens the representation ability of key facial regions (e.g., eyebrows, eye corners, mouth corners), thereby affecting the accuracy of expression recognition. As illustrated Figure 1, both CNN and ViT assign certain attention weights to irrelevant areas such as hair and jawline during feature extraction, yet these elements act as noise for facial expression tasks. This indicates that current networks often struggle to effectively distinguish expression-related regions from irrelevant ones during image modeling, resulting in the inclusion of unrelated information in the feature representation and consequently impairing the model’s discriminative accuracy for expressions. This phenomenon reveals the limitations of existing feature extraction mechanisms in region selection. Similarly, graph-based methods usually construct graph structures based on the entire facial feature map, lacking precise localization and modeling of emotion-related regions, making it difficult to effectively suppress the influence of irrelevant regions. This usually leads to interference caused by irrelevant regional features, thereby limiting the model’s performance in complex scenarios.

Figure 1. Panels (a) and (b), respectively, show the attention heatmaps obtained after multi-layer feature extraction by CNN and ViT on two images. From the heatmaps, it can be observed that the regions indicated by the red arrows, although unrelated to facial expressions, still receive relatively high attention weights. This indicates that both CNN and ViT contain redundant information during the feature modeling process. (Note: The original input images are of limited resolution as per the RAF-DB dataset. The heatmaps are best interpreted by observing the relative attention weight distribution rather than fine-grained details. Statement: The RAF-DB dataset used in this study is publicly available at https://www.kaggle.com/datasets/hoanguyensgu/raf-db (accessed on 25 December 2025).

In-depth analysis reveals that this common challenge stems from a deeper and as-yet-unresolved “systemic capability gap”: existing mainstream architectures struggle to jointly achieve the following three key objectives: (1) dynamically screening sparse yet critical expression regions in the early stages of feature modeling to suppress redundancy at the source; (2) performing deep structured modeling of the screened key regions to capture their complex non-Euclidean relationships; (3) deeply integrating the aforementioned fine-grained local representations with the global semantic context of the face to ensure discriminative consistency. To this end, this work proposes a facial expression recognition model based on multi-stage sparse feature learning. To address the aforementioned capability gap, our core idea is to construct a collaborative framework of “screening, modeling, and integration.” A salient feature selection mechanism is first introduced to prioritize the localization of key regions, followed by graph-based structural modeling to enhance spatial relationship learning. Finally, the global guidance mechanism is used to effectively integrate local and global features, thereby significantly improving the model’s perception of expression-relevant regions and its recognition performance. Specifically, the MSAWS module, the Key Region Graph Learning module, and the Global-Guided Local Feature Enhancement (GGLFE) module in the model are designed to address the three core challenges of dynamic screening, structural modeling, and global collaboration, respectively. The main contributions of this article are summarized as follows:

• We propose MFSC, a facial expression recognition model that combines multi-stage feature sparse constraints and key region graph learning. By progressively filtering features and enhancing expression relevant regions, MFSC effectively suppresses local noise, improves sensitivity to subtle expressions, and enhances recognition robustness.
• To eliminate the influence of redundant facial region features and accurately locate the key areas most relevant to emotions, we propose the MSAWS module based on the Transformer architecture to progressively filter facial features.
• We propose a GGLFE module and a graph-based structural modeling of key regions. By constructing a topological region association graph to capture semantic relationships and introducing a cross-attention module, our method enhances local–global interactions and effectively preserves subtle facial expressions.
• We conducted extensive experiments on three public datasets. The results demonstrate that our method achieves performance superior to existing mainstream approaches across three popular facial expression datasets.

The structure of this article is organized as follows: Section 2 provides a brief review of the related research in the field of facial expression recognition. Section 3 presents a detailed exposition of the proposed MFSC model. Section 4 offers an in-depth analysis and discussion of the experimental results. Finally, Section 5 summarizes the article and draws conclusions.

2. Related Work

FER methodologies have evolved significantly, progressing from traditional machine learning to contemporary deep learning paradigms. This evolution can be broadly categorized into three main technical strands: traditional machine learning methods, deep learning methods based on CNN, and more recently, deep learning methods built upon emerging architectures like Transformers and GNN. The latter two represent the dominant and most promising directions in current research. Therefore, this article focuses on reviewing and discussing deep learning-based FER approaches, with particular emphasis on the advancements and integrations beyond standard CNN.

2.1. CNN-Based Facial Expression Recognition

With the advancement of deep learning, CNNs have become the dominant framework in the field of FER, owing to their powerful capability in extracting local features. Conventional CNN architectures, including VGGNet and ResNet, have been extensively employed in the context of FER tasks on static images. Specifically, VGGNet achieves deep feature representation by stacking small convolutional kernels, while ResNet introduces residual connections to effectively alleviate the vanishing gradient problem in deep network training. These architectures have demonstrated strong performance on various FER benchmark datasets [19,20].

Building upon these foundations, researchers have proposed various improved CNN architectures to better accommodate the requirements of FER tasks, particularly in terms of region sensitivity and fine-grained feature modeling. For example, multi-scale convolutional modules have been employed to extract features in parallel, enhancing the network’s scale-awareness capability [21]. DenseNet improves feature propagation and reuse efficiency through densely connected layers, which has been shown to boost expression discrimination performance in FER [22]. IR50, a lightweight variant of ResNet, balances computational efficiency and feature representation capability and has been widely adopted in mobile and real-time facial expression recognition scenarios [15,16].

To address challenges such as significant dynamic variations in facial regions and pose diversity, current research primarily integrates attention mechanisms into CNN architectures to enhance the model’s localization capability for key facial features (e.g., eyebrows, eyes, mouth corners) while suppressing background noise and non-expression-related regions. Specifically, 2S-SGCN [23] introduces attention mechanisms on the basis of CNN to enhance the model’s focus on key facial regions (such as eyebrows, eyes, mouth corners, etc.) while suppressing interference from background and redundant areas. The SFER-MDFAE [24] framework achieves more precise facial feature extraction and robust expression recognition through its innovatively designed multi-scale dual-pooling feature aggregation module and key-region oriented attention mechanism. The FERmc framework employs a multi-branch convolutional structure coupled with attention mechanisms to achieve adaptive enhancement and focus on discriminative regions [25]. In contrast, FER-VMamba establishes a more robust multi-granularity feature representation system in complex scenarios through the synergistic design of multi-scale hybrid convolutions (MixConv) and spatial-channel attention modules (SCAM) [26].

However, CNN-based methods have limited receptive fields due to their network structure, making it difficult to capture long-distance dependency information. Furthermore, when processing complex expressions, they may be affected by redundant areas, resulting in insufficient attention to the expression area.

2.2. Transformer-Based Facial Expression Recognition

The Transformer architecture initially gained widespread attention due to its groundbreaking performance in natural language processing [27]. Its core attention mechanism effectively captures dependencies between distant elements in the input sequence, showcasing powerful global modeling capabilities. Building upon this foundation, researchers proposed the ViT, which partitions input images into fixed-size patches and serializes them as input tokens to the Transformer encoder. Through the use of multi-head self-attention, ViT enables comprehensive global feature modeling of images.

In FER tasks, ViT and its variants have been adopted to address the shortcomings of traditional CNN in capturing long-range dependencies and modeling non-local relationships. Compared with local convolution operations, ViT is capable of modeling global semantic relationships between facial regions, thus effectively improving the model’s ability to perceive complex expression variations [28].

However, the application of ViT in FER still faces several challenges. On the one hand, while a variety of public FER datasets exist, their collective scale is relatively limited compared to the massive datasets that ViT-style models are typically pre-trained on. This discrepancy in data volume can make directly fine-tuned ViT models prone to overfitting on FER tasks. On the other hand, ViT adopts a fixed and uniform patch division strategy, which fails to reflect the importance of key facial expression areas (such as eyebrows, eyes, and corners of the mouth) in modeling. As a result, modeling resources are evenly distributed to redundant areas, reducing the model’s sensitivity to salient expression features. In addition, ViT lacks the inherent local inductive bias of CNN, making it more susceptible to interference from non-informative regions, especially in low-data regimes. To alleviate the aforementioned issues, HLA-ViT [29] introduces a saliency region mask to guide the attention distribution. By incorporating region-aware mechanisms, saliency-guided modules, or attention reallocation strategies within the ViT architecture, the model enhances its focus on key facial regions, thereby improving its ability to capture critical expression features. Global–Local Feature Fusion Transformer (GFFT) [30] proposes cross-patch feature interaction based on self-attention fusion to effectively address challenges posed by facial occlusion and pose variation, thereby enhancing recognition robustness in uncontrolled environments. Y. Li et al. [31] introduced a method combining a lightweight Swin Transformer with a multi-scale feature fusion (EMA) module, which effectively enhances the practicality and real-time performance of the model while maintaining high accuracy.

Based on the above observations, this work proposes an MSAWS mechanism, which recognizes that not all image regions contribute equally to expression discrimination. By learning the saliency distribution of the image, this mechanism selects the most discriminative key regions for subsequent Transformer modeling. This approach enables more precise expression feature construction, reduces computational overhead from irrelevant regions, and enhances the modeling capability of semantically important expression areas.

2.3. Graph-Learning-Based Facial Expression Recognition

In recent years, GNNs have been widely used for computer vision tasks due to their ability to model non-Euclidean structured data. Examples include FER, head pose estimation, human action recognition, and scene graph generation [23,32,33]. In the context of FER, GNNs are primarily employed to model spatial dependencies and semantic interactions among different facial regions. Specifically, researchers typically represent nodes in the graph as feature representations of distinct facial regions, while edges denote structural or semantic relationships between these regions. This graph-based formulation enables effective modeling of complex facial expression variations [32].

To enhance the flexibility of relational modeling among nodes, the GAT has been introduced into FER. By assigning learnable attention weights to neighboring nodes, GAT enables adaptive adjustment of the influence that different regions exert on the central region, thereby allowing for more precise modeling of interactions among key facial areas [34]. This mechanism improves the model’s sensitivity to salient expression regions and enhances its robustness to variations.

However, most existing graph-based modeling approaches still suffer from notable limitations. One class of methods constructs graph structures over the entire feature map. While this allows for global modeling, it often fails to accurately capture the local regions that are closely related to expression variations, leading to the inclusion of substantial redundant information during graph construction. Another class of methods builds graphs based on predefined facial landmarks (e.g., eye corners, mouth corners), which focuses on local regions but is constrained by static layouts. This makes it difficult to adapt to dynamic changes in salient region distributions across individuals or expression states [17]. Such graph construction strategies lack selective attention to expression-relevant semantic regions, potentially allocating modeling resources to ineffective areas and thereby impairing overall recognition performance.

To address the aforementioned issues, some studies have begun integrating regional attention mechanisms with graph-based modeling to enhance the perception and representation of task-relevant regions. Certain researchers employ saliency detection modules to filter graph nodes, constructing graph structures only over key facial regions [23]. Others attempt to classify samples based on uncertainty and incorporate action unit (AU) relationship modeling, which improves the accuracy of emotion label prediction [35]. Although these approaches help mitigate the influence of redundant regions to some extent, such regions often still retain partial weights, resulting in residual noise within the graph structure. This limits the upper-bound performance of these methods, especially in complex facial expression recognition scenarios.

2.4. Summary and Research Gap

To systematically contextualize our work within the existing landscape, Table 1 summarizes the core ideas, advantages, and limitations of major FER approaches.

Table 1. Summary and comparison of mainstream FER methodologies.

Category	Core Mechanism	Advantages	Limitations
CNN-Based	Convolutional feature hierarchy.	Strong local feature extraction; Efficient learning.	Limited receptive field; Redundancy-sensitive.
Transformer-Based	Global self-attention.	Superior long-range context modeling.	High computation cost; Uniform attention dilution.
Graph-Based	Graph-based relation reasoning.	Explicit structural relationship modeling.	Static graph construction; Lack of dynamic focus.

As shown in Table 1, existing methods exhibit complementary strengths yet face inherent limitations. CNN-based methods excel in local feature extraction but lack global awareness. Transformers capture global context but inefficiently allocate attention across all patches, including non-informative regions. GNN model structural relationships but often rely on suboptimal node (region) selection strategies. Synthesizing the above analysis, the respective limitations of CNN, Transformers, and GNN collectively point to an unresolved common challenge and a core research gap: the current lack of a unified framework capable of (1) dynamically and efficiently focusing on key expression regions, akin to an idealized sparse Transformer, to avoid redundant computation; (2) constructing flexible and accurate structural relationship models on this curated, high-quality set of region nodes, akin to an optimized GNN; and (3) deeply integrating the resulting global semantic relations with the finest local features, akin to an enhanced CNN.

Our proposed MFSC model is explicitly designed to bridge this gap. The MSAWS mechanism precisely addresses the first point by performing coarse-to-fine sparse attention. The key-region graph learning module specifically tackles the second point by constructing an adaptive graph over the tokens selected by MSAWS. Finally, the global-guided fusion module satisfactorily resolves the third point by integrating the global relations from the GNN with the local details from the CNN through cross-attention.

3. Proposed MFSC Method

This article proposes an FER method integrating multi-stage feature sparse constraints and key region graph learning. The overall framework is displayed in Figure 2. Starting from the complementary perspectives of global and local information modeling, the method designs three mutually cooperative core modules to achieve precise modeling and fusion of key region features. First, the MSAWS module performs multi-stage local attention computation on the image, progressively selecting discriminative key region features. Second, the GAT module is constructed based on the selected key regions, treating the filtered feature patches as graph nodes and performing structured feature modeling by constructing irregular graph structures combined with graph neural networks, thereby enhancing the modeling of relationships among regions. Finally, a Global Guided Local Feature Enhancement (GGLFE) module is designed to fuse the global salient region features extracted by the graph network with the output of the backbone via a cross-attention mechanism. This complementary integration of local details and global structure improves the overall facial expression classification performance.

Figure 2. The proposed model consists of two branches. The global branch extracts discriminative global features through a multi-stage sparse attention module and a graph neural network module. These features are then fused with those from the local branch via a cross-attention module, and the final output is the predicted facial expression category.

3.1. Notation

We first introduce the mathematical notations used throughout this section. Key symbols are summarized in Table 2, and other general conventions are explained thereafter.

Table 2. Key notations used throughout the methods section.

Symbol	Description
Input, Output and Global Parameters
$\mathbf{I}\in {\mathbb{R}}^{H\times W\times 3}$	Input facial image ($H=224,W=224$).
$\widehat{y},{\mathcal{L}}_{LS}$	Final prediction and the label smoothing loss.
$C,ϵ$	Number of expression classes and the label smoothing factor.
MSAWS Module
${\mathbf{X}}^{\left(s\right)}$	Feature token sequence output from stage s ($s=1,2,3,4$).
${\mathbf{X}}^{final}\in {\mathbb{R}}^{Z\times d}$	Final output of MSAWS: Z selected tokens with dimension d.
${\mathbf{X}}_i,{\mathbf{X}}_i^{sel},{\mathbf{X}}_i^{out}$	Tokens in window i, its selected subset, and residual-enhanced output.
${\mathbf{A}}^{\left(s\right)},{\mathit{\alpha }}^{\left(4\right)}$	Attention weight matrix at stage s and global importance score vector.
${\mathbb{L}}_i$	Index set of selected top-k tokens in window i.
$D,d_k$	Feature dimension (e.g., 256) and attention key dimension ($d_k=D/\#heads$).
${\mathbf{W}}_Q,{\mathbf{W}}_K,{\mathbf{W}}_V$	Learnable projection matrices for Query, Key, Value in attention.
Graph Construction and Learning
$\mathcal{G}=(\mathcal{V},\mathcal{E})$	Graph built from key tokens, with node set $\mathcal{V}$ and edge set $\mathcal{E}$.
${\mathbf{x}}_i\in {\mathbb{R}}^d$	Feature vector of node $v_i$, corresponding to the i-th token in ${\mathbf{X}}^{final}$.
$\mathbf{A}\in {\mathbb{R}}^{Z\times Z}$	(Sparse) adjacency matrix of graph $\mathcal{G}$, based on KNN (K Nearest Neighbors) and cosine similarity.
$sim({\mathbf{x}}_i,{\mathbf{x}}_j)$	Cosine similarity between nodes ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$.
${\mathcal{N}}_K\left({\mathbf{x}}_i\right)$	Set of K nearest neighbors of node ${\mathbf{x}}_i$.
${\mathbf{h}}_i^{\prime }$	Updated feature of node i after GAT propagation.
${\mathit{\alpha }}_{ij}$	Attention weight from node j to node i in GAT.
$\mathbf{W},\mathbf{a}$	Learnable weight matrix and vector in GAT.
${\mathbf{H}}_{GNN}$	Final output node features from the GNN module.
Feature Fusion and Classification
${\mathbf{F}}_l\in {\mathbb{R}}^{K\times D}$	Local feature sequence extracted from IR50 (K patches, dimension D).
${\mathbf{F}}_g\in {\mathbb{R}}^{K\times D}$	Global feature sequence from GNN, aligned with ${\mathbf{F}}_l$.
$\mathbf{Q},\mathbf{K},\mathbf{V}$	Query, Key, Value matrices in the cross-attention fusion module.
${\mathbf{W}}_Q,{\mathbf{W}}_K,{\mathbf{W}}_V$	Learnable projection matrices for cross-attention (distinct from those in MSAWS).
${\mathbf{F}}_{fused}$	Final fused feature after cross-attention.
$p_j$	Predicted probability for class j.

The notations defined in Table 2 adhere to the following typographical conventions throughout this paper. We use bold uppercase letters (e.g., $\mathbf{X}$, $\mathbf{A}$, $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$) to denote matrices and tensors. Bold lowercase letters (e.g., ${\mathbf{x}}_i$, ${\mathbf{h}}_i^{\prime }$) are used for vectors. Calligraphic letters (e.g., $\mathcal{G}$, $\mathcal{V}$, $\mathcal{E}$, ${\mathcal{N}}_K\left(i\right)$, ${\mathbb{L}}_i$) denote sets, graphs, or specific modules. Superscripts in parentheses indicate stage or layer indices (e.g., ${\mathbf{X}}^{\left(s\right)}$, ${\mathbf{A}}^{\left(4\right)}$). Other superscripts, such as ‘$sel$’, ‘$out$’, ‘$final$’, and ‘^⊤’ (transpose), carry specific semantic meanings as defined in the context. Subscripts typically denote indices within a sequence or set (e.g., ${\mathbf{X}}_i$ for the i-th window, ${\mathbf{x}}_i$ for the i-th node, ${\mathbf{A}}_{ij}$ for a matrix element). The subscript in ${\mathbf{A}}^{\left(4\right)}[j,:]$ represents row indexing. Scalar variables and dimensions are typeset in italic (e.g., H, W, C, d, k, $ϵ$). The input image, as a special case, is denoted by the italic uppercase I. Functions and operators (e.g., $\mathrm{Softmax}(·)$, $\mathrm{Concat}(·)$, $\mathrm{TopK}(·)$, $\parallel ·\parallel$) are set in roman typeface.

3.2. Multi-Stage Sparse Attention Weight Selection

In FER tasks, expression variations are often found in specific areas such as the eyebrows, eye corners, and mouth corners, rather than being uniformly distributed across the entire face. Therefore, directly modeling all image patches in a Transformer framework introduces a large amount of redundant information, which is unfavorable for the model’s focus and discrimination on key areas. To address this, this section proposes an MSAWS module, which progressively selects the most discriminative regional features through multi-stage window partitioning and sparse attention mechanisms. This module utilizes ViT as the feature extraction backbone and combines it with multi-stage sparse attention to compress and filter high-dimensional image features. To visually elucidate the core mechanism of the MSAWS module—particularly how spatial locations of tokens are persistently tracked across multiple partitioning and selection stages—we present the detailed pipeline in Figure 3. The module consists of four stages, each of which performs window partitioning, local attention computation, and selection of high-attention tokens to gradually extract features with stronger task relevance.

Figure 3. Pipeline of the proposed MSAWS module. The diagram illustrates the hierarchical process from input image partitioning to the final selection of key tokens through four stages of window-based sparse attention. Crucially, it highlights the persistent position tracking mechanism that maintains spatial index mappings across all stages, ensuring the traceability and spatial coherence of the selected features.

Stage 1: This stage aims to extract high-quality and discriminative local salient regions from the original image, laying the foundation for subsequent multi-stage fine selection and graph modeling. First, the input image is resized to $\mathrm{I}\in {\mathbb{R}}^{H\times W\times 3}(H=224,W=224)$. We adopt ViT as the feature encoder. The input image is first divided into patches of size 16 × 16, which are then embedded into a sequence of 196 patch tokens corresponding to a 14 × 14 grid. These tokens are further grouped into four windows, each with a size of 7 × 7, containing 49 tokens per window, denoted as ${\mathbf{X}}_i\in {\mathbb{R}}^{49\times D},i=1,2,3,4$ (D = 256). Subsequently, a local sparse attention mechanism is independently applied within each window. Specifically, a self-attention weight matrix is constructed for the tokens within each window. This matrix is computed as follows:

$$\begin{array}{c}\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{QK}}^{\top }}{\sqrt{d_k}}}\right)\mathbf{V},\\ \mathbf{Q}={\mathbf{X}}_i{\mathbf{W}}_Q,\mathbf{K}={\mathbf{X}}_i{\mathbf{W}}_K,\mathbf{V}={\mathbf{X}}_i{\mathbf{W}}_V,\end{array}$$

(1)

where mycolor $d_k$ denotes the key dimension. The scaling factor $1/\sqrt{d_k}$ stabilizes gradient propagation by normalizing dot-product magnitudes.

The computed attention matrix $\mathbf{A}\in {\mathbb{R}}^{49\times 49}$ measures the global relevance of each patch within the window. We average each row and select the top $k=36$ token indices ${\mathbb{L}}_{\mathit{i}}$ with the highest attention scores, thereby obtaining the sparsely selected salient regional features:

$$\begin{array}{c}\hfill {\mathbf{X}}_i^{sel}={\mathbf{X}}_i\left[{\mathbb{L}}_i\right],\end{array}$$

(2)

where ${\mathbf{X}}_i^{sel}$ represents the feature representation of sparse salient regions.

The operation is performed in parallel across four windows:

$$\begin{array}{c}\hfill {\mathbf{X}}^{\left(1\right)}=\mathrm{Concat}\left({\mathbf{X}}_1^{sel},{\mathbf{X}}_2^{sel},{\mathbf{X}}_3^{sel},{\mathbf{X}}_4^{sel}\right)\in {\mathbb{R}}^{144\times D},\end{array}$$

(3)

where ${\mathbf{X}}^{\left(1\right)}$ denotes the stage’s output feature obtained by concatenating the outputs from four sparse windows.

To further enhance the representation capability while preserving low-level details from the original patches, we introduce a residual connection mechanism on top of the sparse features. Specifically, the selected indices are mapped back to the original 196 features ${\mathbf{X}}_{orig}$, which are then element-wise added to the output of the attention module, forming a residual-enhanced output:

$$\begin{array}{c}\hfill {\mathbf{X}}_i^{out}={\mathbf{X}}_i^{sel}+{\mathbf{X}}_{orig}\left[{\mathbb{L}}_i\right],\end{array}$$

(4)

where ${\mathbf{X}}_i^{out}$ comprises 144 first-stage tokens, each representing patch features that integrate local saliency, contextual relationships, and residual information, which will be progressively refined in subsequent stages.

Stage 2: After completing the coarse-grained selection of salient regions in Stage 1, the model proceeds to perform two rounds of sparse attention computations at finer scales to further reduce redundancy and enhance the representation of key regions. The input to Stage 2 is the output ${\mathbf{X}}^{\left(1\right)}\in {\mathbb{R}}^{144\times D}$ from Stage 1, which is divided into four windows of size 6 × 6, each containing 36 tokens. Within each window, the attention mechanism from Stage 1 (see Equation (1)) and residual connection are repeated, and the top 25 tokens are selected based on attention scores. The total output is

$$\begin{array}{c}\hfill {\mathbf{X}}^{\left(2\right)}\in {\mathbb{R}}^{100\times D}.\end{array}$$

(5)

Stage 3: The third stage takes ${\mathbf{X}}^{\left(2\right)}$ as input, which is similarly divided into four windows, each of size 5 × 5 containing 25 tokens. The top 16 tokens based on attention scores are selected from each window, resulting in a total output of

$$\begin{array}{c}\hfill {\mathbf{X}}^{\left(3\right)}\in {\mathbb{R}}^{64\times D}.\end{array}$$

(6)

Both stages inherit the attention weight ranking and residual enhancement strategies from the first stage. That is, after selecting tokens, they are fused with the original token features through weighted addition to form semantically richer medium and small-scale features. This multi-stage architecture implements a coarse-to-fine sparse focusing process, progressively enhancing the model’s ability to represent expression-related regions. From the perspective of computational complexity theory, this coarse-to-fine hierarchical selection structure offers significant advantages. Let $N_t$ denote the number of input tokens at stage t, and let the window size be $M\times M$. The complexity of standard global self-attention is $O\left(N_t^2{d}_k\right)$. while the complexity of our window-based sparse attention is

$$\begin{array}{c}\hfill O\left(\left({\displaystyle \frac{N_t}{M^2}}\right)\times \left(M^4{d}_k\right)\right)=O\left(N_t^2{M}^2{d}_k\right).\end{array}$$

(7)

Since M is fixed and relatively small (e.g., 7, 6, 5) and $N_t$ decreases progressively across stages, the overall complexity is significantly lower than that of global attention. In particular, as the sequence length is compressed from 196 in the first stage to k in the final output, the computational cost for subsequent modules such as GNN is reduced by approximately a factor of ${\left({\displaystyle \frac{N}{k}}\right)}^2$.

Stage 4: In the fourth stage, window partitioning is no longer applied. Instead, global attention computation is performed directly on all 64 tokens output from the previous stage:

$$\begin{array}{c}\hfill {\mathbf{A}}^{\left(4\right)}=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{Q}}^{\left(4\right)}{\left({\mathbf{K}}^{\left(4\right)}\right)}^{\top }}{1/\sqrt{d_k}}}\right),\end{array}$$

(8)

where ${\mathbf{Q}}^{\left(4\right)}\in {\mathbb{R}}^{64\times d}$ is the query matrix with each row representing a token’s embedding. ${\mathbf{K}}^{\left(4\right)}\in {\mathbb{R}}^{64\times d}$ is the key matrix encoding positional relationships. $d_k$ denotes the feature dimension and $\sqrt{d_k}$ prevents gradient vanishing. ${\mathbf{A}}^{\left(4\right)}\in {\mathbb{R}}^{64\times 64}$ is the resulting attention weight matrix.

$$\begin{array}{c}\hfill {\mathit{\alpha }}^{\left(4\right)}=\sum _j{\mathbf{A}}^{\left(4\right)}\left[j,:\right],\end{array}$$

(9)

where ${\mathbf{A}}^{\left(4\right)}[j,:]\in {\mathbb{R}}^{1\times 64}$ represents attention weights from token j to all tokens. ${\mathit{\alpha }}^{\left(4\right)}\in {\mathbb{R}}^{64}$ aggregates global importance scores through column-wise summation. The top $\mathit{k}$ tokens with the highest attention weights are selected as the final output:

$$\begin{array}{c}\hfill {\mathbf{X}}^{final}=\mathrm{TopK}\left({\mathbf{X}}^{\left(3\right)},{\mathit{\alpha }}^{\left(4\right)},\mathit{k}\right),\end{array}$$

(10)

where ${\mathbf{X}}^{final}\in {\mathbb{R}}^{k\times D}$ is an Output matrix with top-k most significant tokens selected. TopK(·) is an Operator that filters tokens based on ${\mathit{\alpha }}^{\left(4\right)}$ scores and $\mathit{k}$ criteria.

Based on the aforementioned theoretical foundation, we propose an MSAWS algorithm, as detailed in Algorithm 1.

Algorithm 1: Multi-Scale Attention-based Sparse Window Selection

   Through the aforementioned four-stage progressive compression and selection mechanism, this module effectively filters out the most representative $\mathit{Z}$ high-attention tokens from the original 196 image patches. This process not only reduces computational complexity and memory overhead but also retains the most discriminative visual features associated with facial expression semantics. By discarding redundant and low-information background regions while preserving key facial components (such as eyes, mouth contours, and muscle movement areas), the module constructs a compact yet highly expressive feature subset. This sparse and efficient feature representation serves as a robust foundation for subsequent graph structure modeling, enabling the construction of adaptive relational graphs that capture both local fine-grained dependencies and global semantic correlations among facial regions. Furthermore, the refined token set enhances the effectiveness of cross-attention operations in later fusion stages, facilitating more precise and interpretable interaction between visual features and textual or sequential modalities. Ultimately, this hierarchical token selection strategy contributes to improved model performance in facial expression recognition tasks by focusing computational resources on the most informative regions, thereby increasing both accuracy and operational efficiency.

3.3. Key Region Graph Learning

To further model the structural relationships among key regions in the image, this module constructs a graph based on the $\mathit{Z}$ high-confidence tokens output from the first stage and employs GNN to perform spatial relationship modeling and feature enhancement.

Graph Construction: The output ${\mathbf{X}}^{final}$ from the first module is treated as $\mathbf{z}$ nodes of graph $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$, where each node ${\mathbf{x}}_i\in {\mathbb{R}}^d$ represents a high-dimensional feature vector of an important image region. The edges (adjacency relationships) in the graph are determined based on the feature similarity between nodes. To construct the adjacency matrix $\mathbf{A}\in {\mathrm{R}}^{\mathrm{K}\times \mathrm{K}}$ of the graph, this study defines edge weights by combining the KNN algorithm with Cosine Similarity. First, compute the cosine similarity between any two nodes ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ as follows:

$$\begin{array}{c}\hfill sim\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)={\displaystyle \frac{{\mathbf{x}}_i{\mathbf{x}}_j}{\parallel {\mathbf{x}}_i\parallel ·\parallel {\mathbf{x}}_j\parallel }},\end{array}$$

(11)

for each node ${\mathbf{x}}_i$, the top $\mathit{K}$ most similar nodes among all other nodes are retained as its neighbors, forming the adjacency matrix:

$$\begin{array}{c}\hfill {\mathbf{A}}_{ij}=\left\{\begin{array}{cc}sim\left({\mathbf{x}}_i,{\mathbf{x}}_j\right),\hfill & \mathrm{if}{\mathbf{x}}_j\in {\mathcal{N}}_K\left({\mathbf{x}}_i\right)\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,\end{array}$$

(12)

where ${\mathcal{N}}_K\left({\mathbf{x}}_i\right)$ denotes the $\mathit{K}$ nearest neighbors of ${\mathbf{x}}_i$. The final result is a sparse and symmetric adjacency matrix $\mathbf{A}$, which is used for the propagation operations in the graph neural network.

Graph Learning: GNNs are used to update and aggregate node features based on the adjacency structure. In this study, we adopt the GAT, which introduces an attention mechanism to dynamically learn the weights of neighboring nodes. For each node $x_i$, its feature update is computed as follows:

$$\begin{array}{c}\hfill {\mathbf{h}}_i^{\prime }=\sigma \left(\sum _{j\in {\mathcal{N}}_K\left(i\right)}{\alpha }_{ij}{\mathbf{Wx}}_j\right),\end{array}$$

(13)

where ${\mathbf{Wx}}_j$ denotes the learnable linear transformation matrix. $\sigma$ represents the nonlinear activation function. ${\alpha }_{ij}$ is the attention weight vector, computed as

$$\begin{array}{c}\hfill {\alpha }_{ij}={\displaystyle \frac{exp\left(LeakyReLU\left({\mathbf{a}}^{\top }\left[{\mathbf{Wx}}_i|{\mathbf{Wx}}_j\right]\right)\right)}{{\sum }_{k\in {\mathcal{N}}_K\left(i\right)}exp\left(LeakyReLU\left({\mathbf{a}}^{\top }\left[{\mathbf{Wx}}_i|{\mathbf{Wx}}_k\right]\right)\right)}},\end{array}$$

(14)

where $\mathbf{a}$ is the learnable weight vector in the attention mechanism and ‖ denotes the feature concatenation operation.

To enhance the model’s expressive capability, multiple GAT layers are stacked to achieve deeper relational modeling, with the final output being the node features ${\mathbf{H}}_{GNN}\in {\mathbb{R}}^{Z\times D}$, which contain spatial information between key regions. This module enhances the structural representation capability of features by embedding the high-weight regions output from the Transformer module into the graph structure, and introducing a GAT to model the spatial and semantic dependencies between regions.

3.4. Global-Guided Local Feature Enhancement

To effectively exploit the key regional features modeled by the graph structure, this study introduces a global guidance mechanism that combines the detailed local features extracted by the IR50 model with the global regional features output by the graph neural network. The central idea of this module lies in leveraging the robust feature extraction capabilities of multi-stage sparse attention and the graph network architecture to guide the IR50 model in enhancing the discriminative power of the local facial regions. The IR50 model is a variant of ResNet50, primarily used for feature extraction in facial expression recognition. We extract features from the intermediate layers and convert them into sequential features, denoted as ${\mathbf{F}}_{\mathrm{l}}=\left[f_1^l,f_2^l,\dots ,f_K^l\right]\in {\mathbb{R}}^{K\times D}$, through patch partitioning. The features extracted here need to be consistent in both quantity and dimension with the features extracted by the graph network. For the sake of clarity, we convert the output of the graph network into a sequential form, as shown below:

$$\begin{array}{c}\hfill {\mathbf{F}}_{\mathrm{g}}=\left[f_1^g,f_2^g,\dots f_K^g\right]\in {\mathbb{R}}^{K\times D}.\end{array}$$

(15)

The workflow of the Global Guided Local Feature Enhancement (GGLFE) module is detailed as follows.

Feature Extraction and Alignment: The local features ${\mathbf{F}}_{\mathrm{l}}$ are extracted from the intermediate layers of IR50 and aligned with the global features ${\mathbf{F}}_{\mathrm{g}}$ from the graph network, ensuring both have the same number of tokens K and feature dimension D.

Attention Matrix Construction: The global feature sequence ${\mathbf{F}}_{\mathrm{g}}$ serves as the Query, while the local feature sequence ${\mathbf{F}}_{\mathrm{l}}$ serves as both the Key and the Value. They are transformed through learnable linear projections:

Query matrix:

$$\begin{array}{c}\hfill \mathbf{Q}={\mathbf{F}}_{\mathrm{g}}{\mathbf{W}}_{\mathrm{Q}}\in {\mathbb{R}}^{K\times D}.\end{array}$$

(16)

Key and value matrices:

$$\begin{array}{c}\hfill \mathbf{K}={\mathbf{F}}_{\mathrm{l}}{\mathbf{W}}_{\mathrm{K}}\in {\mathbb{R}}^{K\times D},\mathbf{V}={\mathbf{F}}_{\mathrm{l}}{\mathbf{W}}_{\mathrm{V}}\in {\mathbb{R}}^{K\times D}.\end{array}$$

(17)

Here, ${\mathbf{W}}_{\mathrm{Q}}$,${\mathbf{W}}_{\mathrm{K}}$,${\mathbf{W}}_{\mathrm{V}}$ are learnable linear transformation matrices. The fused output features are ${\mathbf{F}}_{\mathrm{fused}}=\mathbf{AV}\in {\mathbb{R}}^{K\times D}$.

Output: Finally, the fused features are computed by the following formula:

$${\mathbf{F}}_{\mathrm{fused}}=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{QK}}^{\top }}{\sqrt{d_k}}}\right)\mathbf{V},$$

(18)

The resulting feature ${\mathbf{F}}_{\mathrm{fused}}$ incorporates the global structural relationships modeled by the graph network while preserving the local details from IR50, thereby enhancing the representation of discriminative facial regions. This feature will be fed into subsequent network layers to accomplish the final facial expression recognition task.

This module achieves efficient fusion of local and global features through a cross-attention mechanism. By leveraging multi-stage sparse attention and the graph network’s capability to detect global key regions, it guides the local features of IR50 to reinforce the globally discriminative regions. It retains fine-grained details while incorporating structural relational information, enhancing the model’s sensitivity to subtle changes in facial expressions.

3.5. Facial Expression Classification

The features, after being processed by the cross-attention mechanism, form the final global feature vector. The final global feature vector is subsequently input into a multi-layer perceptron (MLP) for the purpose of facial expression classification. The MLP comprises two fully connected layers. The first is employed for feature fusion and nonlinear transformation, while the second generates the final classification results. To suppress overfitting and improve generalization capability, we employ the label smoothing loss function. This loss function [36] effectively mitigates the model’s over-confidence in training labels by softening the hard constraints of the original one-hot labels. The final loss function is formulated as follows:

$${\mathcal{L}}_{LS}=-\sum _{j=1}^C\left[(1-ϵ)y_j+{\displaystyle \frac{ϵ}{C}}\right]log\left(p_j\right),$$

(19)

where C denotes the number of classes, $ϵ$ is the smoothing factor, $y_j$ represents the ground-truth label, and $p_j$ indicates the predicted probability. Finally, we present the complete implementation pipeline of our model in Algorithm 2.

Algorithm 2: FER Integrating Multi-stage Feature Sparse Constraints and Key Region Graph Learning

4. Results

We validated the effectiveness of MFSC on three standard FER datasets: RAF-DB [37], AffectNet [38], and FER2013 [39]. To ensure robustness and statistical significance, we conducted five independent experiments with different random seeds for each dataset. The following sections first introduce the dataset, experimental setup, and parameter configurations, then compare MFSC with the current state-of-the-art facial expression recognition methods based on mean accuracy values from multiple runs, and finally, conduct a series of method analysis and ablation experiments on MFSC.

4.1. Generally Setting

Datasets: We systematically evaluated the performance of the MFSC model on three widely used benchmark datasets: RAF-DB, AffectNet, and FER2013. The specific characteristics of each dataset are as follows:

• RAF-DB is a high-quality large-scale facial expression database that not only includes seven categories of finely annotated basic emotions (happiness, sadness, surprise, fear, disgust, anger, and neutrality) but also covers a variety of complex facial expression variations from real-world scenarios.
• AffectNet is currently the most representative large-scale publicly available dataset in this field. It not only comprehensively covers eight basic emotions but also features a diverse range of sample collection environments and demographic distributions.
• FER2013 provides 35,887 standardized facial images, offering a unified testing benchmark for algorithms. Its well-defined seven basic expression categories enhance the comparability of results.

The combination of these three datasets enables comprehensive validation of the model’s performance across FER tasks from various dimensions. The specific dataset configurations are shown in Table 3.

Table 3. The table displays comprehensive statistics of the experimental dataset, encompassing data partitioning (train/test) and categorical label distribution.

Dataset	Train Size	Test Size	Classes
RAF-DB	12,271	3068	7
AffectNet (7)	280,401	3500	7
AffectNet (8)	283,501	4000	8
FER2013	28,709	7178	7

Evaluation Metrics: In this study, we employ accuracy as the primary evaluation metric for facial expression recognition, which is calculated as

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TP}{TP+TN+FP+FN}},$$

(20)

where TP denotes the number of correctly predicted positive samples, TN denotes the number of correctly predicted negative samples, FP represents the number of incorrectly predicted positive samples, and FN represents the number of incorrectly predicted negative samples.

Implementation Details: In this study, we use the PyTorch (pytorch==1.13.1), deep learning framework [40] to implement end-to-end training and inference of the MFSC model on an NVIDIA RTX 4090 GPU hardware platform. In terms of model architecture, we choose IR50 [41] as the backbone network. The network’s weight parameters have been pre-trained on the large-scale face recognition dataset Ms-Celeb-1M [42], and the weights are updated during our training process.

To systematically validate the effectiveness of the proposed method, experiments are conducted on three popular facial expression recognition benchmark datasets. The experiments adopt a unified training configuration. We set $ϵ$ = 0.12 as this value has been empirically shown to achieve optimal balance between regularization and prediction confidence [43]. Detailed training configurations are shown in Table 4.

Table 4. This table summarizes the key parameters and hyperparameters used during the supervised training of the MFSC model, which was initialized with random weights and optimized end-to-end on RAF-DB, AffectNet and FER2013.

Config	Value
Optimizer	Adam
Base Learning Rate	$4.00\times {10}^{-5}$
Batch Size	64
Training Epochs	200
Learning Rate Schedule	ExponentialLR ($\gamma =0.98$)
Data Augmentation	RandomHorizontalFlip()

4.2. Comparison Methods

We conducted a comparative analysis of the accuracy of our model against previous models to evaluate its performance advantages.

(a)

PSR [11] proposes a pyramid-style super-resolution network architecture and introduces a label smoothing loss function to improve model performance.

(b)

ARM [44] proposes a novel representation refinement module that mitigates the padding effect by reducing the weights of eroded features.

(c)

DAN [45] proposes a Disentangled Attention Network that learns targets with large margins and employs multi-head attention to focus on different facial regions, generating and fusing multiple attention maps to enhance feature robustness and overall representation capability.

(d)

MA-Net [46] proposes a global multi-scale and local attention network to address two key challenges: occlusion and pose variation.

(e)

TransFER [14] primarily employs a multi-attention dropout approach, randomly dropping one attention map to encourage the model to adaptively explore more diverse local regions.

(f)

LHC-Net [47] proposes a novel self-attention module that combines channel-wise self-attention and local attention mechanisms, achieving higher efficiency and stronger local modeling capability.

(g)

AD-Corre [13] proposes a method for guiding network learning of intra-class and inter-class embedded features, thereby improving the discriminative ability of facial expression recognition.

(h)

POSTER [48] effectively combines facial keypoints and image features through a two-stream pyramid cross-fusion architecture.

(i)

EmoNeXt [49] proposes an improved ConvNeXt architecture that integrates spatial transformer networks, SE modules, and self-attention regularization, effectively enhancing facial expression classification performance.

(j)

FST-MWOS [50] proposes a method that integrates Swin Transformer fine-tuning with a multi-weight optimal search strategy, jointly optimizing local and global features to improve facial expression recognition accuracy.

(k)

POSTER++ [16] effectively reduces computational cost and improves facial expression recognition performance by simplifying the two-stream design, introducing a lightweight multi-scale interaction network, and employing a windowed cross-attention mechanism with linear complexity.

(l)

LFNSB [51] proposes a lightweight facial network with spatial bias, incorporating channel fusion operations and depthwise separable convolutions, and designs a novel loss function called cosine coherence loss.

(m)

Mini-ResEmoteNet [52], based on a knowledge distillation framework, designs a lightweight model by progressively reducing the number of feature channels layer by layer, achieving efficient and high-performance facial expression recognition.

(n)

TAPNet [53], enhances facial expression recognition performance in real-world complex scenarios by simulating human facial cognitive mechanisms and collaboratively extracting multi-layered features from global, local, and key facial regions.

(o)

MAFaNet [54] effectively improves facial expression recognition performance in real-world scenarios by collaboratively extracting facial features from global, local, and salient regions, achieving outstanding results across multiple datasets.

4.3. Accuracy Analysis

Comprehensive experimental evaluation on the RAF-DB benchmark. As shown in Table 5, MFSC demonstrates an accuracy of 92.31%, surpassing the performance of some current mainstream methods, such as POSTER++ [16], which reports 92.21%. Notably, unlike mainstream approaches that rely on external keypoint extraction modules (e.g., POSTER [48], POSTER++), MFSC completely abandons dependence on external keypoint networks. Instead, it employs a self-constructed multi-stage feature optimization mechanism that ensures a lightweight architecture while achieving superior performance. This innovative design not only effectively reduces computational resource consumption but also significantly enhances system flexibility and deployment efficiency. Without incorporating keypoint information, MFSC still attains leading recognition accuracy, demonstrating its strong potential to achieve excellent results without auxiliary data. This provides a more efficient and practical solution for facial expression recognition and related visual tasks.

Table 5. Comparison of MFSC with state-of-the-art methods on RAF-DB, AffectNet, and FER2013 datasets. The experimental data for most of the other models are referenced from the results reported in their original articles.

Methods	Year	RAF-DB	AffectNet7	AffectNet8	FER2013
PSR	2020	88.98	63.77	60.68	–
ARM	2021	90.42	65.20	61.33	–
DAN	2021	89.70	65.69	62.09	–
MA-Net	2021	88.42	64.53	60.29	–
TransFER	2021	90.91	66.23	–	–
LHC-Net	2022	–	–	–	74.42
AD-Corre	2022	86.96	63.36	–	72.03
POSTER	2023	92.05	67.31	63.34	–
EmoNeXt	2023	–	–	–	76.12
FST-MWOS	2023	90.38	63.33	–	–
POSTER++	2024	92.21	67.49	63.77	–
LFNSB	2024	91.07	–	–	–
TAPNet	2024	90.77	65.13	61.38	–
Mini-ResEmoteNet	2025	85.00	–	–	70.02
MAFaNet	2025	89.85	65.12	61.41	–
MFSC (Ours)	–	92.31 ± (0.15)	67.35 ± (0.17)	64.28 ± (0.08)	76.21 ± (0.11)

Comprehensive experimental evaluation on the AffectNet benchmark. As shown in Table 5, the performance comparison between MFSC and existing state-of-the-art methods is presented. In the seven-class classification task, MFSC achieves an accuracy of 67.35%. After introducing the “contempt” category to extend the task to eight-class classification, it still achieves a notable accuracy of 64.28%, demonstrating superior generalization and robustness. These results indicate that despite challenges such as class imbalance and ambiguous boundaries between expression categories, MFSC can effectively and stably capture discriminative semantic information through its multi-stage feature optimization mechanism, thereby enhancing model robustness and adaptability in complex expression recognition scenarios. Overall, MFSC not only maintains high recognition accuracy for well-represented majority classes but also exhibits promising performance on under-represented tail classes, verifying its practical applicability and generalization capability in real-world deployments.

Comprehensive experimental evaluation on the FER2013 benchmark. As shown in Table 5, the MFSC model achieves a classification accuracy of 76.21% on the FER2013 dataset, outperforming most existing methods and demonstrating its strong representational capacity in facial expression recognition tasks. The performance achieved by MFSC not only highlights its robustness under complex conditions but also further validates the practicality and generalizability of the model architecture in real-world application scenarios.

Based on the experimental results across the three authoritative benchmarks (RAF-DB, AffectNet, and FER2013), MFSC demonstrates comprehensive and robust accuracy advantages. The achievement of this advantage is realized under the critical premise of completely abandoning reliance on computationally expensive and error-prone external keypoint networks. Compared to high-performing methods such as POSTER++ that depend on keypoint information, MFSC, through its unique internal multi-stage feature optimization mechanism, achieves comparable or even superior recognition accuracy on multiple datasets without utilizing any additional auxiliary supervision signals. This strongly proves the superiority of MFSC’s method itself in feature learning and representational capability. Its advantages do not stem from the support of external complex tools but originate from the efficiency and robustness of its core architecture.

4.4. Computational Efficiency Comparison

As shown in Table 6, MFSC not only achieves superior recognition performance compared to baseline models in facial expression recognition (FER) tasks but also demonstrates higher computational efficiency in terms of the number of parameters, FLOPs, and inference time. Compared with MA-Net, FST-MWOS, TransFER, POSTER, and its enhanced version POSTER++, MFSC maintains competitive accuracy while significantly improving model compactness and inference speed.

Table 6. Comparative Analysis of Computational Efficiency Across State-of-the-Art Facial Expression Recognition Methods.

Methods	#Params	#FLOPs	Time	FPS
MA-Net	63.500 M	3.7 G	17.6 ms	56.7 fps
FST-MWOS	147.900 M	33.82 G	–	–
TransFER	65.200 M	15.3 G	12.3 ms	81.9 fps
POSTER	71.800 M	15.7 G	25.7 ms	38.9 fps
POSTER++	43.710 M	8.4 G	9.1 ms	109.9 fps
MFSC (Ours)	39.764 M	7.6 G	8.9 ms	112.4 fps

Specifically, although MFSC exhibits higher FLOPs compared to MA-Net, it achieves better recognition accuracy and real-time performance, reflecting greater overall efficiency and practicality. When compared with POSTER, MFSC reduces the number of parameters by 32.036 M, FLOPs by 8.1 G, and average inference time by 16.8 ms, significantly lowering the computational burden. Compared to POSTER++, MFSC still offers advantages with a reduction of 3.946 M parameters, 0.8 G fewer FLOPs, and a 0.2 ms decrease in inference time, further enhancing its feasibility for real-world deployment. In summary, MFSC achieves a more optimal balance among recognition accuracy, computational cost, and inference efficiency, making it a promising solution for FER tasks, especially in resource-constrained real-time emotion recognition scenarios.

4.5. Ablation Study

To thoroughly analyze the impact of each key module in the proposed MFSC architecture on performance, this article designed systematic ablation experiments on three major benchmark datasets: RAF-DB, FER2013, and AffectNet. Specifically, we evaluate the impact of the MSAWS and the GNN module. All comparative experiments are built upon the same backbone network (IR50), and the feature output dimensions are kept consistent across models. The following four ablation configurations are designed for comparison:

• Baseline-ViT: This baseline model employs only a ViT to extract global image patch features. Both the MSAWS and GNN modules are removed and replaced with transformer layers of equivalent depth. This serves as the fundamental reference model without the proposed enhancements.
• MFSC w/o MSAWS: In this variant, the MSAWS module is removed, while the ViT and GNN structures are retained. The MSAWS is substituted with standard transformer layers of the same depth to evaluate the impact of the attention-based patch selection mechanism on performance and computational efficiency.
• MFSC w/o GNN: This configuration removes the GNN module, retaining only the ViT and MSAWS components. It is designed to assess the contribution of GNN in modeling local semantic relationships among key regions.
• Full MFSC (Ours): The complete architecture that integrates ViT, MSAWS, and GNN modules, serving as the final proposed solution.

As shown in Table 7, removing the GNN module leads to a slight reduction in FLOPs, indicating lower computational cost. However, this modification results in a general decline in accuracy across all three datasets. This finding aligns with the conclusions drawn in many works that rely solely on global feature modeling [16,45], namely that the lack of explicit modeling of local semantic relationships can limit the model’s representational capacity. However, by introducing GNN-based relational modeling of key regions, our complete model not only overcomes this limitation but also significantly outperforms these methods that use only global features. This demonstrates that although GNN introduces additional computational overhead, it plays a crucial role in modeling the local semantic relationships among features. When the MSAWS module is removed, the model shows a slight improvement in accuracy on the FER2013 dataset. However, its performance drops on both the RAF-DB and AffectNet datasets, and the FLOPs increase significantly, indicating a notable rise in computational cost. The resulting performance inconsistency and higher computational burden underscore that the MSAWS module enhances the model’s generalization ability and ensures robust performance across diverse datasets, while also significantly improving computational efficiency—yielding an approximate 10.7% gain. These findings highlight the importance of MSAWS in strengthening the model’s robustness and efficiency. Consequently, our complete MFSC model, which fully integrates both MSAWS and GNN, achieves the best or near-best accuracy across all three datasets with only 7.6 G FLOPs. This result not only validates the effectiveness of each proposed module but, more importantly, demonstrates their strong synergy in delivering highly efficient and superior performance.

Table 7. Ablation Study on the Performance Impact of Removing MSAWS and GNN Modules.

Model	RAF-DB (%)	FER2013 (%)	AffectNet (%)	FLOPs (G)
Baseline-ViT	90.85	74.12	65.47	8.72
MFSC without MSAWS	91.13	76.84	66.52	8.51
MFSC without GNN	91.46	75.47	67.31	7.35
Full MFSC (Ours)	92.31	76.21	67.35	7.60

4.6. Visualization

4.6.1. Attention Heatmap Analysis

To further validate the effectiveness of the proposed MSAWS mechanism during feature extraction, we visualize and analyze the attention heatmaps from its first three stages. Specifically, we illustrate the spatial distribution and attention intensity of the retained key patches on the original images at each stage. This visualization intuitively reveals the evolving focus on salient regions throughout the multi-stage sparse selection process, helping to understand how the mechanism contributes to constructing discriminative features. Notably, all visualizations are generated after the model training is completed to ensure stability and reliability of the observations.

As shown in Figure 4, during the first stage, the sparse attention mechanism selects a batch of high-weight patches from the original patch set. The highlighted areas are scattered and cover multiple important facial regions, which aligns with the expectation that the initial stage features cover a broad and dispersed attention area. In the second stage, the model further concentrates on regions with higher attention weights, while discarding some patches that performed well in Stage 1 but receive lower attention scores at this stage. This gradually focuses the attention on areas more strongly correlated with expression discrimination. In the third stage, which is the final layer of the sparse attention mechanism, high-weight feature extraction continues; however, unlike Stage 2, the highlighted regions do not further contract. This phenomenon may arise because the third stage is adjacent to the input of the GNN, whose output features directly influence graph modeling and final classification results. Therefore, the model tends to integrate attention information from the first two stages, preserving the most discriminative regions for facial expressions to enhance classification performance.

Figure 4. Attention Visualization Results of the First Three Stages in MFSC Sparse Attention Weight Selection. The attention mechanism progressively refines its focus onto discriminative facial regions (e.g., eyebrows, mouth corners) while effectively pruning redundant areas across processing stages. (Statement: The RAF-DB dataset used in this study is publicly available at https://www.kaggle.com/datasets/hoanguyensgu/raf-db (accessed on 25 December 2025)).

From the overall visualization results, it can be observed that the regions attended to by the model are primarily concentrated around key facial areas such as the eyes, mouth, and cheeks. This attention pattern aligns closely with the perceptual mechanisms of human facial expression recognition, further validating the effectiveness of the proposed multi-stage sparse attention weight selection mechanism in feature extraction and expression representation modeling.

4.6.2. UMAP Clustering Analysis

To further validate the effectiveness of the proposed MSAWS mechanism and GNN in enhancing the discriminability and separability of high-level features, we conducted comparative experiments on the RAF-DB dataset. Specifically, the backbone model is consistently based on IR50, while the MSAWS and GNN modules were replaced with standard ViT layers of the same depth to construct control groups for performance comparison. Figure 5 presents the UMAP visualizations of the high-level feature embeddings from different models.

Figure 5. Visualization of high-level output features using UMAP [55]. The integration of MSAWS and GNN modules yields more separable inter-class distributions and compact intra-class clustering in the feature space.

It can be observed that, compared to the standard ViT, the model integrating MSAWS and GNN exhibits more compact clustering within the “Happy” category, demonstrating stronger intra-class consistency. Moreover, the feature distributions in the “Angry,” “Disgust,” and “Fear” categories show clearer class boundaries and significantly enhanced inter-class separability. These results strongly indicate that the proposed sparse attention mechanism effectively selects key discriminative regions, while the GNN further exploits semantic relationships among these regions. The synergy of both modules improves the model’s feature representation and discrimination capability in complex facial expression classification tasks.

5. Conclusions

This paper proposes MFSC, a novel facial expression recognition framework that integrates multi-stage feature sparsity constraints with graph-based key region learning. Notably, MFSC operates without dependence on external facial keypoints or landmark detection modules, offering greater deployment flexibility. The core of MFSC lies in three synergistic components: the MSAWS module, which reduces the number of parameters and computational complexity of standard Transformers through statistical filtering of attention weights; a GNN module that models spatial structural relationships to enhance connectivity between salient regions; and a cross-attention fusion module that establishes associations between global salient regions and local fine-grained features. Extensive experiments on RAF-DB, AffectNet, and FER2013 datasets validate the efficacy of MFSC. It achieves state-of-the-art or highly competitive accuracies of 92.31%, 67.35% (seven-class), 64.28% (eight-class), and 76.21%, respectively, on these benchmarks. Crucially, MFSC attains this performance with superior efficiency, requiring only 39.8 M parameters and 7.6 G FLOPs, and achieving an inference speed of 112.4 fps. Ablation studies confirm the contribution of each component: the MSAWS module improves generalization and delivers an approximately 10.7% gain in computational efficiency, while the GNN is vital for modeling local semantic relationships. Visualization analyses further show that our model learns to focus adaptively on discriminative facial regions (e.g., eyes, mouth) and produces more separable feature representations. Despite these strengths, the model still exhibits limitations that point to future research directions. First, the trade-off between performance and efficiency requires further optimization: while the MSAWS module enhances efficiency, its dynamic token selection process introduces additional overhead, and the overall model size (39.8 M parameters) may still pose challenges for ultra-low-power devices. Second, the empirical threshold in MSAWS lacks adaptability, as the fixed sparsification threshold may not achieve optimal performance across diverse input conditions or deployment environments. Furthermore, there remains room for improvement in the model’s evaluation metrics. For example, while maintaining high accuracy, the recall rate for minority expression categories could be further optimized, which is crucial for the model’s robustness and practicality in real-world scenarios. Therefore, future work will focus on the following directions: developing a learnable dynamic sparsification strategy to replace the empirical threshold; exploring neural architecture search or knowledge distillation techniques to build lighter versions suitable for edge computing scenarios; and simultaneously enhancing the model’s recall ability for various expressions by introducing class-balanced training mechanisms or designing adaptive threshold adjustment modules, thereby improving its overall performance in complex real-world environments.