Dynamic Occlusion-Aware Facial Expression Recognition Guided by AA-ViT

Abstract

In complex natural scenarios, facial expression recognition often encounters partial occlusions caused by glasses, hand gestures, and hairstyles, making it difficult for models to extract effective features and thereby reducing recognition accuracy. Existing methods often employ attention mechanisms to enhance expression-related features, but they fail to adequately address the issue where high-frequency responses in occluded regions can disperse attention weights (e.g., incorrectly focus on occluded areas), making it challenging to effectively utilize local cues around the occlusions and limiting performance improvement. To address this, this paper proposes a network based on an adaptive attention mechanism (Adaptive Attention Vision Transformer, AA-ViT). First, an Adaptive Attention module (ADA) is designed to dynamically adjust attention scores in occluded regions, enhancing the effective information in features. Next, a Dual-Branch Multi-Layer Perceptron (DB-MLP) replaces the single linear layer to improve feature representation and model classification capability. Additionally, a Random Erasure (RE) strategy is introduced to enhance model robustness. Finally, to address the issue of model training instability caused by class imbalance in the training dataset, a hybrid loss function combining Focal Loss and Cross-Entropy Loss is adopted to ensure training stability. Experimental results show that AA-ViT achieves expression recognition accuracies of 90.66% and 90.01% on the RAF-DB and FERPlus datasets, respectively, representing improvements of 4.58 and 18.9 percentage points over the baseline ViT model, with only a 24.3% increase in parameter count. Compared to existing methods, the proposed approach demonstrates superior performance in occluded facial expression recognition tasks.

1. Introduction

Facial expression is one of the most powerful and universal ways for humans to convey emotions and feelings [1]. The goal of facial expression recognition is to accurately identify different expressions from facial images, such as the basic emotions defined by American psychologists Ekman and Friesen [2], including happiness, sadness, anger, fear, surprise, and disgust. It has now become an important research direction in the fields of artificial intelligence and computer vision, holding significant research value and broad application prospects in domains such as intelligent education, mental health monitoring, medical diagnosis, autonomous driving, and human–computer interaction [3]. In early research on facial expression recognition, studies were predominantly conducted using datasets collected under controlled laboratory conditions (e.g., CK+ [4]), which achieved relatively high performance in scenarios with minimal pixel-level interference. However, facial expression recognition faces numerous challenges in practical applications. For instance, in real-world environments, factors such as occlusion, varying lighting conditions, and pose changes significantly degrade the accuracy of expression recognition. As a result, researchers have begun to focus on facial expression recognition datasets collected in natural scenes, such as RAF-DB [5] and FERPlus [6]. Compared to datasets acquired under controlled conditions, these are more reflective of real-world scenarios and hold greater research significance for the advancement and application of facial expression recognition technology.

Although significant progress has been made in facial expression recognition, critical challenges persist in real-world complex scenarios. This study aims to address the following core problem: In occluded facial expression recognition tasks, mainstream methods based on attention mechanisms often overlook the interference caused by high-frequency responses in occluded regions. This leads to an imbalance in attention weight allocation (e.g., causing the model to incorrectly focus on occluded areas), thereby weakening the model’s ability to utilize locally discriminative features around the occlusions and ultimately limiting performance improvement.

This study is based on the following dual hypotheses:

• Occlusion causes attention dispersion in existing models.
• Current feature extraction and training mechanisms are insufficient for handling complex natural occlusion scenarios. We propose that a systematic solution—comprising an adaptive attention adjustment mechanism, an enhanced feature mapping module, a diversified occlusion simulation strategy, and an improved loss function—can effectively overcome the aforementioned bottlenecks.

To clarify the research approach, this paper defines and measures the following variables:

• Core Dependent Variables: Performance metrics such as expression recognition accuracy and parameter count.
• Key Process Variables: The distribution of attention weights between occluded regions and effective regions.
• Mediating Variables: Feature discriminability and representation capability.
• Controlled Variables:

  (1)

  Core Experimental Conditions (Fixed Factors):

Input data format: Images are uniformly preprocessed to the same resolution (e.g., 224 × 224) and adopt the RGB color space; Baseline network structure: Based on the standard Vision Transformer (ViT), core parameters such as the number of layers and attention heads are kept consistent; Dataset usage: Training and testing are conducted on publicly available natural scene datasets such as RAF-DB and FERPlus; Evaluation metrics: Classification accuracy serves as the core performance evaluation criterion; Basic training setup: The same optimizer, initial learning rate, and batch size are employed.

• (2)

  Controlled Conditions for Comparative Experiments (Baselines for Fair Comparison):

All compared methods are reproduced or their results are referenced under the same data splits (training/validation/test sets); The backbone network and basic training strategy are kept entirely consistent in ablation studies; Performance comparisons are all based on the same experimental environment and computational resources.

However, existing methods mostly employ attention mechanisms to enhance expression-related features, but they fail to fully consider that the high-frequency responses in occluded regions can disperse attention weights (so-called high-frequency responses refer to the fact that occlusion areas, due to more pronounced changes in expression feature information compared to non-occluded areas, tend to be excessively focused on during feature extraction). For example, hair occlusion (long hair can cover the face, especially the forehead, eyes, and cheeks), glasses occlusion (thick frames or reflective lenses may cover the eyes), mask occlusion (people wear masks in winter, which cover most of the face, leaving only the eyes visible), and hand gesture occlusion (when performing certain actions, hands may cover part of the facial expression). All these situations can lead to issues in expression recognition, causing the model to overly focus on the occluded areas while neglecting subtle information in the surrounding regions. This makes it difficult to effectively utilize local cues around the occlusions, thereby limiting performance improvement. Therefore, building upon the application and development of Transformer in the field of computer vision, this paper proposes an Adaptive Attention Vision Transformer (AA-ViT) for occluded facial expression recognition in natural scenes. The network incorporates an Adaptive Attention module (ADA) to enhance the representation of effective feature information, employs a Dual-Branch Multi-Layer Perceptron (DB-MLP) to improve feature expression and model classification capability, and introduces a Random Erasure (RE) strategy to enhance model robustness. The main contributions are as follows:

• To address the issue of model sensitivity to occlusions and insufficient extraction of effective feature information, this paper designs an ADA module. The module utilizes an adaptive attention adjustment mechanism to dynamically perceive occlusion levels across different facial regions and autonomously recalibrate attention weights for occluded areas, thereby significantly enhancing the model’s capacity to capture discriminative expression features.
• To overcome the limited representational capacity of the single linear layer perceptron in conventional Transformers, this paper proposes a DB-MLP. This architecture enables nonlinear mapping of input features, allowing the model to fit complex data distributions while balancing the extraction of local detailed features and global representations. It not only enhances the model’s sensitivity to subtle expression details but also improves overall classification performance.
• To enhance the model’s robustness against random occlusions and improve its generalization capability, this paper proposes a RE strategy. During the training phase, this method randomly generates occlusion masks to simulate various real-world occlusion scenarios, effectively enhancing the model’s robustness and generalization capability against occlusions.
• To address the issue of class imbalance and ensure training stability, a hybrid loss function combining Focal Loss and Cross-Entropy Loss is adopted. Experiments conducted on the in-the-wild datasets RAF-DB and FERPlus demonstrate the effectiveness of the proposed method.

2. Related Work

2.1. Occluded Facial Expression Recognition

Occluded facial expression recognition primarily consists of four stages: facial expression sample collection, data preprocessing, feature extraction, and expression recognition [7]. In real-world scenarios, facial expression images often exhibit varying degrees of occlusion and pose variations. Therefore, effectively extracting discriminative facial expression features is crucial for accurate recognition under such challenging conditions.

Owing to their exceptional feature extraction capabilities, convolutional neural networks (CNNs) have gained widespread popularity among researchers for facial expression feature extraction tasks. The introduction of convolutional neural networks such as AlexNet [8], VGG [9], and GoogleNet [10] has significantly improved model classification accuracy. However, methods that rely on stacking network layers to achieve high recognition efficiency tend to exhibit performance degradation as depth increases, often leading to issues such as gradient vanishing [11]. Therefore, Kaiming He et al. [12] proposed the Residual Neural Network (ResNet) in 2015, which incorporates the concept of residual learning to preserve information integrity and effectively address issues such as information loss, gradient vanishing, and network degradation in convolutional neural networks. Li Yong et al. [13] improved the traditional LeNet convolutional neural network by integrating both shallow and deep features, significantly enhancing the model’s robustness. Although significant improvements have been achieved compared to traditional methods, issues such as overfitting, poor generalization capability, and susceptibility to feature information loss persist. Therefore, Pan et al. [14] fine-tuned the network using pre-trained weights, effectively extracting global features and improving its efficiency in recognizing occluded expressions. However, in occluded facial expression recognition research, the effective utilization of local detailed (local features from non-occluded regions) facial expression features is crucial for recognition accuracy.

To address this, some researchers have incorporated attention mechanisms into networks [15], enabling more effective focus on local detailed features. Li et al. [16] proposed a convolutional neural network embedded with an attention mechanism, which effectively extracts salient features and distinguishes between occluded and non-occluded regions. Chen et al. [17] introduced a dual-branch network with a feature-fused residual attention mechanism, utilizing multi-scale modules to divide extracted feature maps into multiple non-overlapping local feature maps. This approach filters informative features through attention mechanisms and performs classification after feature fusion. Wang et al. [18] proposed a Region Attention Network (RAN), which effectively captures areas affected by occlusion and pose variations in facial expressions. Woo et al. [19] introduced an attention mechanism that integrates both channel and spatial dimensions, adaptively computing feature weights to enhance the model’s feature representation capability. Dai et al. [20] designed a Multi-Scale Channel Attention Mechanism (MS-CAM). This module integrates local details with global contextual features, dynamically optimizes feature fusion weights, and enables autonomous focus on discriminative key features, thereby enhancing the model’s invariant representation against expression feature disturbances in unconstrained environments. Nevertheless, while existing research has made remarkable progress in addressing challenges such as occlusion and pose variations, most methods still fail to adequately account for the issue where high-frequency responses in occluded areas disperse attention weights (e.g., incorrect focus on occluded regions). This limitation hinders the effective utilization of local contextual cues around occlusions and consequently restricts further improvement in model recognition performance. The feature learning capability of current models remains in need of enhancement.

To address the above limitations, Zhang et al. [21] proposed a Selective Feature Enhancement Network (SFEN), which incorporates a learnable feature selection gating mechanism to dynamically enhance the most discriminative expression features. Additionally, the research team led by Wang [22] developed a Multi-scale Feature Interaction Transformer (MFIT). This architecture incorporates a cross-scale feature fusion module and employs a hierarchical attention mechanism to establish connections between features of varying granularities, significantly enhancing the model’s feature learning efficacy. However, in natural occlusion scenarios, existing methods often apply uniform attention to both occluded and non-occluded facial regions during feature learning. This makes it difficult to effectively leverage non-occluded local information for efficient facial expression recognition, thereby limiting further improvement in model performance.

2.2. Visual Transformer

In recent years, Transformer models [23] have successfully expanded from the field of natural language processing to computer vision, leveraging their powerful sequence modeling capacity and global contextual understanding. Dosovitskiy et al. [24] pioneered the Vision Transformer (ViT), marking the first application of a pure Transformer architecture to image classification tasks and opening new directions for computer vision research. With the deepening of research, numerous improved ViT architectures have emerged: Liu et al. [25] proposed the Swin Transformer, which introduces a hierarchical windowed attention mechanism to significantly enhance the model’s ability to capture multi-scale visual features; Yuan et al. [26] developed the Tokens-to-Token ViT (T2T-ViT), adopting a progressive tokenization strategy to better preserve local structural information. In the field of facial expression recognition, Chen et al. [27] proposed the Facial Expression Transformer (FET), which innovatively integrates local regional attention with global relational modeling, effectively enhancing the representational capacity of expression features. Specifically, the Convolutional Transformer Network (CTN), recently developed by Han et al. [28], combines convolutional inductive biases with self-attention mechanisms. This integration preserves the global modeling strengths of Transformer while enhancing local feature extraction capabilities. However, in complex natural environments, facial expression recognition often encounters partial occlusion, which hinders the model from effectively utilizing both global and local features, thereby compromising recognition accuracy. Moreover, insufficient consideration of high-frequency responses in occluded areas may misallocate attention weights, further limiting the effective extraction of informative features from non-occluded regions. Therefore, the proposed AA-ViT model addresses these challenges by designing an ADA module to dynamically adjust attention scores in occluded regions, employing a DB-MLP to enhance feature representation, and incorporating a RE strategy to improve model robustness. These innovations collectively significantly boost the model’s performance in occluded facial expression recognition under complex environments.

3. AA-ViT Model

3.1. Overall Architecture

To achieve robust facial expression recognition in occluded environments, this paper proposes an adaptive attention-based network named AA-ViT. Firstly, to fully suppress interference from occluded regions and enhance discriminative expression features while maintaining model lightweightness and high recognition accuracy, this paper proposes an enhanced Vision Transformer architecture that integrates an ADA module and a DB-MLP. Specifically, to dynamically adjust the attention distribution over occluded regions and enhance the model’s ability to utilize local informative features, an occlusion-aware ADA module is designed. It employs a feature importance reweighting mechanism to suppress high-frequency responses in occluded areas. Furthermore, to enhance the model’s nonlinear representational capacity, a DB-MLP is adopted to replace the conventional single linear layer, improving classification performance through hierarchical feature transformation. Additionally, to significantly improve the model’s robustness in occlusion scenarios, a RE strategy is introduced for data augmentation, enhancing the network’s feature learning and discriminative capabilities. Through the synergistic integration of the aforementioned innovative modules, a highly occlusion-robust AA-ViT network is constructed. The overall network architecture is illustrated in Figure 1.

Figure 1. The overall architecture of the AA-ViT network.

3.2. Improved ADA Module

To address the common challenge of occlusion in visual tasks, this paper designs an improved ADA mechanism. Building upon the standard multi-head self-attention framework, this mechanism innovatively incorporates a dynamic weight adjustment function. This enables the model to adaptively redistribute attention based on input feature content, significantly enhancing robustness in occluded scenarios. The architecture of this module is illustrated in Figure 2. The core architecture of the module comprises two key components: a traditional Query-Key-Value (QKV) projection system and an innovative adaptive weight generation network. During the input processing stage, features are first passed through linear transformations to simultaneously generate the three projections: Query (Q), Key (K), and Value (V), consistent with the standard attention mechanism channels. Unlike traditional approaches, the module additionally introduces a lightweight weight generation network. This network takes the original features as input and employs fully connected layers followed by a Sigmoid activation to produce dynamic weighting coefficients in the range [0, 1] for each attention head. Given an input feature tensor $X\in R^{B\times N\times C}$ (where B denotes batch size, N sequence length, C feature dimension and R represents the set of real numbers), the computational expressions for the standard QKV projection, multi-head splitting process, and weight generation network in the module are as follows:

$$\left[Q,K,V\right]=Linea{r}_{qkv}(X),(Q,K,V\in R^{B\times N\times C})$$

(1)

$$hea{d}_i(Q,K,V)=reshape({\displaystyle \frac{Q_i{K}_i^T}{\sqrt{d_k}}})V_i,d_k=C/h$$

(2)

$$W_{ada}=Sigmoid(Linea{r}_{ada}(X))\in {\left[0, 1\right]}^{B\times N\times head}$$

(3)

where $d_k$ represents the dimension of the key (K) vector in each attention head, $h$ denotes the number of attention heads, and T indicates the transpose operation.

Figure 2. The Structure of the designed adaptive ADA network.

During the attention computation process, the dot-product attention scores between queries and keys are first calculated following the standard procedure. These scores are then element-wise multiplied by the generated adaptive weights, as shown in the equation below. This operation enables the model to dynamically enhance or suppress the importance of different positions and attention heads, resulting in a context-aware adaptive response to feature content.

$$Att{n}_{adjusted}=\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)\otimes W_{ada}^{reshape}$$

(4)

To enhance flexibility, the module also incorporates a random occlusion mask component. When internal occlusion information $M\in {\left\{0,1\right\}}^{B\times N}$ is provided, the module sets the attention scores corresponding to the masked regions to extremely small negative values (−1 × 10⁹), as shown in the equation below. This ensures that these regions receive near-zero attention weights after softmax normalization. This dual capability allows the module to handle implicit occlusion patterns through its internal adaptive mechanism while also fully leveraging prior knowledge when explicit occlusion annotations are available.

$$Att{n}_{Int\_RE}=Att{n}_{adjusted}+(1-M)\cdot (-\infty )$$

(5)

At the output stage, the weighted features from all attention heads are recombined and passed through a final projection layer, preserving the same dimensional structure as the input, as shown in the equation below. The entire computational process maintains the efficient parallel characteristics of the standard attention mechanism while introducing only a minimal number of additional learnable parameters. This ensures the model’s practicality and significantly enhances its feature discrimination capability in complex environments.

$$Output=Concat(hea{d}_1,\cdots ,hea{d}_h)W^O$$

(6)

3.3. Double-Layer Perceptron Design

The enhanced classification head is a key module in the AA-ViT network, specifically optimized for classification tasks. It significantly improves feature discriminability while maintaining high computational efficiency. Unlike the simple single linear layer classification head $Hea{d}_{original}(z)=Wz+b$ in the original ViT (where W is the weight matrix, b the bias term, and z the input feature vector), this design adopts a two-stage nonlinear projection structure. Its mathematical expression is as follows:

$$Hea{d}_{original}(z)=W_2\cdot ReLU(W_{1}z+b_1)+b_2,W_1\in R^{{\displaystyle \raisebox{1ex}{$d$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}\times d},W_2\in R^{C\times {\displaystyle \raisebox{1ex}{$d$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}}$$

(7)

By incorporating an intermediate feature compression layer and activation functions, a more expressive classification decision space is constructed. The core design concept establishes a progressive feature refinement process before the final classification layer: high-dimensional features (embed_dim) are first compressed to half the original dimension (embed_dim//2), followed by nonlinear transformation via a ReLU activation function, and finally mapped to the target class space. This structure resembles the bottleneck layer design in traditional neural networks. However, by carefully controlling the dimensionality reduction ratio (50%), it reduces the number of parameters while avoiding significant information loss. The expression is as follows:

$$\left\{\begin{array}{c}h=ReLU(Linea{r}_{d\to d/2}(z))\\ \widehat{y}=Linea{r}_{d/2\to C}(h)\end{array}\right.$$

(8)

where C denotes the number of categories. The overall computational flow is as follows:

$$Head(z)=Linea{r}_{C\times d/2}(ReLU(Linea{r}_{d/2\times d}(z)))$$

(9)

The network architecture of this module is illustrated in Figure 3. Its innovations are primarily reflected in three aspects: First, the addition of nonlinear projection layers enables the model to learn more complex feature combinations, effectively capturing subtle inter-class differences; Second, the dimensionality compression mechanism acts as a regularization effect, preventing overfitting caused by direct classification in high-dimensional space; Third, the two-stage design results in a smoother feature transformation process, facilitating gradient propagation and optimization stability, which is particularly effective in fine-grained classification scenarios.

Figure 3. DB-MLP Network Architecture Diagram.

3.4. Loss Function

In the training process of deep learning models, the design of the loss function critically impacts model performance. To address the issue of inter-class data imbalance in occluded expression recognition tasks, this paper adopts a hybrid loss function that combines Focal Loss and Cross-Entropy Loss. This hybrid loss function effectively combines the advantages of both loss components: it maintains the training stability of cross-entropy loss while leveraging the capability of focal loss to handle class imbalance. Such a design has demonstrated outstanding performance in multi-class visual tasks. The mathematical expression of this hybrid loss function is as follows:

$$L_{hybrid}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{c=1}^C{y}_{i,c}}}\cdot \left[\lambda {(1-p_{i,c})}^{\gamma }+(1-\lambda )\right]\cdot \log (p_{i,c})$$

(10)

In the formula, N denotes the batch size, C represents the total number of classes (In this paper, the sample classes include ‘Angry’, ‘Disgust’, ‘Fear’, ‘Happy’, ‘Sad’, ‘Surprise’, ‘Neural’, ‘Contempt’), $y_{i,c}\in \left\{0,1\right\}$ corresponds to the ground-truth label encoding (where a value of 1 indicates that sample i belongs to class c, and 0 otherwise), $p_{i,c}\in \left\{0,1\right\}$ signifies the normalized probability predicted by the model for sample i belonging to class c, $\lambda \in \left(0.3,0.7\right)$ serves as the blending weight to balance Focal Loss and Cross-Entropy Loss, $\gamma \in \left(0.5,2.0\right)$ and acts as the focusing factor to adjust the emphasis on challenging samples.

4. Experiments and Analysis

4.1. Dataset and Data Preprocessing

The AA-ViT model is evaluated on two publicly available facial expression datasets: RAF-DB and FERPlus.

RAF-DB [5] is a large-scale real-world facial emotion dataset containing 29,672 facial images, each annotated by 40 trained annotators. The dataset consists of two subsets: basic expressions and compound expressions. This experiment uses only the basic expression subset, which includes 7 emotional categories (anger, disgust, fear, happiness, neutral, sadness, surprise) with a total of 15,339 images. Specifically, the basic expression subset comprises six basic emotions and one neutral emotion. The training set contains 12,271 samples, while the validation set consists of 3068 samples.

FERPlus [6] is an extended and re-annotated version of the FER2013 dataset, containing 35,887 facial images with a resolution of 48 × 48 pixels. The dataset is divided into 28,709 training samples, 3589 validation samples, and 3589 test samples, covering eight expression categories: the seven basic emotions plus “contempt”.

During the data preprocessing stage, it is common practice to enhance model recognition accuracy and generalization capability through optimized training data. This paper employs an improved random masking method [29] to randomly occlude facial expression images, artificially introducing noise. This approach not only enhances data augmentation but also makes the samples more reflective of real-world scenarios. This approach improves the model’s generalization capability to some extent, reduces the risk of overfitting, and thereby enhances recognition efficiency [30]. The random masking process is illustrated in Figure 4.

Figure 4. The image with External Random Erasing: (a) Data processing for RAF-DB; (b) Data processing for FERPlus.

4.2. Experimental Parameter Settings and Evaluation Metrics

4.2.1. Experimental Parameter Settings

This experiment is implemented based on the PyTorch 1.12.1 deep learning framework [31], with training and testing conducted in PyCharm 2025. The hardware environment is configured as follows: Windows 10 operating system, Intel Core i5-12490F (Manufactured by Intel Corporation, headquartered in Santa Clara, CA, USA), (2.9 GHz CPU with 32 GB RAM), and an NVIDIA GeForce RTX 3080 (10 GB) GPU (Manufactured by NVIDIA Corporation, headquartered in Santa Clara, CA, USA).

Since the image dimensions vary across different datasets, this experiment uses the MTCNN network [32] to localize and crop facial regions, standardizing the input image size to 224 × 224 pixels. Across all datasets, the batch size is set to 64, the initial learning rate is 0.0001, and a cosine annealing learning rate decay strategy is adopted with a decay cycle of 5 epochs. The total training epochs are set to 150. The network model is optimized using SGD with a weight decay of 0.0001 and evaluated with a hybrid loss function combining Focal Loss and Cross-Entropy Loss. Detailed experimental parameter settings are summarized in Table 1.

Table 1. Experimental parameter settings.

Parameter	Value
Batch size	64
Initial learning rate	0.0001
Learning rate decay cycle	5
Epochs	150
Weight decay value	0.0001
Optimizer	SGD

In the composite loss function, this paper introduces two key hyperparameters: the mixing weight λ (which balances the Focal Loss and the cross-entropy loss) and the modulation factor γ (which controls the focus on hard vs. easy samples). Their value ranges are respectively set as: $\lambda \in \left(0.3,0.7\right)$, $\gamma \in \left(0.5,2.0\right)$. To validate the rationality of the hyperparameter values, this paper conducted a systematic grid search experiment. The results are shown in the Table 2 below (with the other parameter fixed at the recommended median: λ = 0.5 or γ = 1.0 for testing):

Table 2. Hyperparameter value range ablation experiment.

λ	γ	Sample Recall Rate (%)	Training Stability
0.1	1.0	73.4	Underfitting
0.3	1.0	78.9	Stable
0.5	1.0	82.3	Optimal Stability
0.7	1.0	80.1	Stable
0.9	1.0	75.6	Underfitting
0.5	0.5	78.5	Fast Convergence
0.5	1.0	82.3	Stable
0.5	2.0	83.1	Stable
0.5	3.0	82.7	Slow Convergence
0.5	5.0	79.8	Unstable

As shown in the table, it can be observed that when λ is too small (≤0.3) or too large (≥0.7), a single loss function (pure Focal Loss or pure CE Loss) fails to balance the “focus on hard samples” and the “fitting of the overall distribution.” When λ lies within the range of 0.3–0.7, the model achieves the best performance in sample recall, and the training process remains stable. When γ is too small (≤0.5), Focal Loss degenerates into an approximate CE Loss, leading to a decline in the ability to recognize hard samples. When γ is too large (≥3.0), the training convergence becomes slow and unstable, making it prone to gradient anomalies. When γ falls within the range of 0.5–2.0, the recall rate for hard samples is the highest, and training convergence is smooth. Based on comprehensive theoretical analysis and experimental results, this paper ultimately sets the hyperparameter search ranges as $\lambda \in \left(0.3,0.7\right)$, $\gamma \in \left(0.5,2.0\right)$. This approach not only aligns with the general principles of loss weighting in multi-task learning but has also been validated through systematic ablation experiments, ensuring sound rationality and reproducibility.

4.2.2. Evaluation Indicators

(1)

Accuracy: Accuracy refers to the proportion of correctly predicted samples out of the total samples, which can be expressed as:

$$Acc={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(11)

TP, TN, FP, and FN respectively represent the counts of True Positives, True Negatives, False Positives, and False Negatives, and the sum of these four metrics equals the total number of samples; when organized in a table, they form a confusion matrix used to analyze misclassification patterns for each category.

(2)

Parameters: Model size refers to the number of parameters, which corresponds to the spatial concept and space complexity. Since many models have excessively large parameter counts, they are typically measured in millions (M). This can be expressed as:

$$param=n_{in}\times n_{out}$$

(12)

• where $n_{in}$ and $n_{out}$ represent the input and output channel counts, respectively. The total parameter count is obtained by summing the parameters of all layers.

(3)

F₁-Score: To comprehensively evaluate the recognition performance of the proposed model, the F₁-Score is introduced as an evaluation metric. The F₁-Score takes both precision and recall into account, serving as one of the commonly used evaluation metrics for multi-class classification problems. It provides a more comprehensive reflection of the model’s robustness and accuracy. The formula is expressed as follows:

$$precision=\frac{TP}{TP+FP}$$

(13)

$$recall=\frac{TP}{TP+FN}$$

(14)

$$F_1=2\times \frac{precision·recall}{precision+recall}$$

(15)

4.3. Ablation Study

To validate the impact of individual modules in the AA-ViT model on occluded expression recognition performance, this study employs a controlled variable approach and conducts ablation experiments on both the RAF-DB and FERPlus datasets. This study employs a ViT model as the baseline (PRE denotes fine-tuning with ImageNet [33] pre-trained weights), and incrementally incorporates the random occlusion masking strategy (RE), the Adaptive Dual Attention module (ADA), and the Dual-Branch Multi-Layer Perceptron module (DB-MLP) to evaluate their individual performance gains, with experimental results summarized in Table 3.

Table 3. Performance comparison of different modules of network.

Number	Model					RAF-DB		FERPlus		Parameter /M
	Baseline	PRE	RE	ADA	DB-MLP	Acc./%	F1	Acc./%	F1
a	√	×	×	×	×	86.08	0.676	71.11	0.598	86.00
b	√	√	×	×	×	87.78	0.693	84.45	0.651	86.44
c	√	×	√	×	×	86.67	0.721	73.21	0.672	86.00
d	√	√	√	×	×	88.13	0.779	86.37	0.681	86.44
e	√	√	√	√	×	89.25	0.793	88.97	0.706	101.63
f	√	√	√	×	√	88.68	0.782	88.01	0.687	92.25
g	√	√	√	√	√	90.66	0.809	90.01	0.714	107.44

The “√” indicates the module is applied, and the “×” indicates the module is not applied.

In this study, seven experimental setups (a, b, c, d, e, f, g) were designed to validate the effectiveness of each module. A comparison between experiments a and b shows that when only the PRE module was added, the recognition accuracy improved by 1.7 and 13.34 percentage points on the RAF-DB and FERPlus datasets, respectively, compared to the baseline model, while the number of model parameters increased by only 0.51%. This result indicates that the PRE module, by fine-tuning the network based on pre-trained weights, plays a significant role in improving recognition accuracy. A comparison among experiments b, c, and d reveals that when only the RE module was added, the recognition accuracy did not show significant improvement on either dataset. However, the F₁-score increased substantially. When both the PRE and RE modules were added, the recognition accuracy improved by 0.35 and 1.92 percentage points on the RAF-DB and FERPlus datasets, respectively, compared to using the PRE module alone. Additionally, the F₁-score showed significant improvement, with no increase in model parameters. These results demonstrate that the RE module significantly enhances model robustness when the network is fine-tuned. A comparison between experiments d and e indicates that, under conditions of data preprocessing and network fine-tuning, adding the ADA module improved recognition accuracy by 1.12 and 2.6 percentage points on the RAF-DB and FERPlus datasets, respectively, compared to experiment d, while the number of parameters increased by only 17.57%. This result suggests that the ADA module, by adaptively adjusting attention weight distribution in occluded facial expression regions, significantly enhances the model’s ability to capture effective expression features, thereby improving overall recognition efficiency. A comparison among experiments e, f, and g shows that, under conditions of data preprocessing and network fine-tuning, adding either the ADA module or the DB-MLP module alone improved recognition accuracy to some extent. However, when both the ADA and DB-MLP modules were integrated, recognition accuracy on the RAF-DB and FERPlus datasets improved even further, exceeding the baseline model by 4.58 and 18.9 percentage points, respectively. Moreover, the F₁-scores were higher than those achieved with individual modules, while the increase in model parameters was only 5.72–16.47% compared to experiments e and f. These results confirm the effectiveness of the proposed method in occluded facial expression recognition tasks.

To intuitively verify the reliability of the above analysis conclusions, this paper employs Grad-CAM++ for attention visualization analysis of the AA-ViT model and the baseline model, with the results shown in Figure 5. The deep red areas indicate regions of high attention. The areas of intense focus are precisely the occluded regions. In facial expression recognition tasks, these regions tend to be over-attended due to their significant deviation from baseline expression features, which reduces the extraction of effective information from non-occluded areas and consequently impacts recognition efficiency. Through the AA-ViT model, facial expressions in valid regions can be better recognized by reducing the weight ratio of invalid areas and enhancing the weight ratio of effective regions. When dealing with occlusions such as hair, glasses, masks, hand gestures, and other obstructions, the model enhances recognition weights for effective regions through dynamic weight allocation, thereby improving expression recognition performance. As shown in the figure, AA-ViT can effectively utilize the valid information surrounding occlusions to accurately focus on the mouth and eye regions of the face. Even in images with occlusions or pose variations, it remains capable of precisely locating key facial areas. This effectively addresses the issue of high-frequency responses in occluded regions interfering with attention distribution (e.g., misdirecting focus to occluded areas), enabling the model to fully utilize information from non-occluded regions and thereby significantly improving facial expression recognition efficiency. In extreme cases where facial expressions are completely occluded, the proposed method in this paper can still serve as an image feature extraction model and be integrated with multimodal approaches to achieve efficient recognition of occluded expressions.

Figure 5. Visualization Results of AA-ViT on the RAF-DB Dataset.

To further analyze the classification capability of AA-ViT across different facial expression categories, this paper presents confusion matrices for the model on the RAF-DB and FERPlus datasets, as shown in Figure 6. As shown in the confusion matrix, AA-ViT demonstrates high accuracy for the expressions “happy,” “neutral,” and “surprised” on both datasets (96%, 87%, and 87% on the RAF-DB dataset, and 96%, 92%, and 88% on the FERPlus dataset, respectively). However, the accuracy for “fear” and “disgust” is relatively low (73% and 74% on the RAF-DB dataset, and 79% and 74% on the FERPlus dataset, respectively). Notably, the accuracy for “contempt” in the FERPlus dataset is the lowest, reaching only 68%. The primary reasons for this phenomenon are the imbalanced inter-class distribution in the dataset and the similarity of features among such negative expressions under occlusion, resulting in reduced inter-class distance and thereby hindering accurate expression classification. Although this issue cannot be entirely avoided, the proposed method improves the recognition accuracy of such expressions to some extent and narrows the performance gap compared to other categories. Overall, the proposed method effectively leverages non-occluded feature information and demonstrates strong discriminative capability for occluded facial expressions.

Figure 6. Confusion Matrices of AA-ViT on the RAF-DB and FERPlus Test Sets: (a) Confusion matrix on the RAF-DB dataset; (b) Confusion matrix on the FERPlus dataset.

4.4. Comparative Experiments

To further validate the effectiveness of the proposed method in occluded facial expression recognition tasks, performance evaluations are conducted on the RAF-DB and FERPlus datasets. Comparisons with 16 state-of-the-art methods from recent years are provided, along with an analysis of the improvements achieved by the proposed approach. The results are summarized in Table 4.

Table 4. Comparison of the results of this model with other methods.

Model	RAF-DB		FERPlus		Parameter /M
	Accuracy/%	Improvement	Accuracy/%	Improvement
RAN [18]	86.90	+3.76	89.16	+0.85	89.73
Ad-Corre [34]	86.96	+3.70	-	-	94.50
EAC [35]	88.02	+2.64	87.03	+2.98	-
Face2Exp [36]	88.54	+2.12	-	-	105.32
DFF [37]	89.60	+1.06	88.80	+1.21	-
VTFF [38]	88.14	+2.52	88.81	+1.20	80.10
LPAPViT [39]	-	-	89.16	+0.85	-
PASM [40]	88.68	+1.98	-	-	120.00
AMP-Net [41]	89.25	+1.41	-	-	105.67
SCN [42]	88.14	+2.52	89.35	+0.66	77.70
GE-LA [43]	-	-	89.93	+0.08	133.41
EDGL-FLP [44]	89.90	+0.76	-	-	132.98
KTN [45]	88.07	+2.59	90.49	−0.48	147.05
MRAN [46]	-	-	89.59	+0.42	-
MTAC [47]	90.52	+0.14	-	-	-
POSTER [48]	92.05	−1.39	91.62	−1.61	71.80
Poster++ [49]	92.21	−1.55	-	-	43.70
CEPrompt [50]	92.43	−1.77	-	-	86.19
AA-ViT (ours)	90.66		90.01		107.44

As shown in the table, the facial expression recognition accuracy of the proposed method reaches 90.66% on the RAF-DB dataset. Compared to methods such as RAN, Ad-Corre, EAC, Face2Exp, VTFF, PASM, SCN, KTN, and AMP-Net, the accuracy improves by approximately 1.41% to 3.76%. Notably, when compared to advanced methods like RAN and Ad-Corre, the proposed method demonstrates the most significant advantages. Although it involves a certain increase in the number of parameters, the accuracy is improved by 3.76 and 3.70 percentage points, respectively. Compared to DFF, EDGL-FLP, and MTAC, the accuracy improves by 1.06, 0.76, and 0.14 percentage points, respectively, showing a slight advantage, while the parameter count is reduced by 23.77% compared to the EDGL-FLP method. The POSTER, Poster++, and CEPrompt methods achieve significantly higher accuracy than AA-ViT on the RAF-DB dataset, but their parameter counts are generally lower (achieved through lightweight structural optimization). While lightweight models offer significant advantages in efficiency and real-time performance, their structural simplification often leads to limited generalization and stability in complex scenarios. Therefore, the AA-ViT model demonstrates strong overall performance in occlusion facial expression recognition tasks. On the FERPlus dataset, the facial expression recognition accuracy of the proposed method reaches 90.01%. Compared to methods such as EAC, DFF, and VTFF, the accuracy improves by 2.98, 1.21, and 1.20 percentage points, respectively. When compared to advanced methods like RAN, LPAPViT, SCN, GE-LA, and MRAN, the proposed method also maintains a slight advantage in accuracy, with improvements ranging from 0.08% to 0.85%. While the proposed method slightly lags behind the KTN method in accuracy, the KTN method achieves only 88.07% accuracy on the RAF-DB dataset, indicating relatively poor model generalization. Additionally, the parameter count of the KTN method exceeds that of the proposed method by 36.86%, which hinders the practical application and development of facial expression recognition technology. Although the POSTER method outperforms the AA-ViT method on the FERPlus dataset, compared to the Poster++ and CEPrompt methods, the proposed method still maintains competitive accuracy and achieves stable performance on both the RAF-DB and FERPlus datasets, highlighting the advantages of the proposed design in cross-dataset robustness. In summary, for occlusion facial expression recognition tasks, the proposed method demonstrates certain advantages over some other advanced methods.

5. Conclusions

This paper proposes a network based on an adaptive attention mechanism (Adaptive Attention Vision Transformer, AA-ViT). An adaptive attention module is designed to effectively address the model’s sensitivity to occlusion and enhance the extraction of effective feature information (non-occluded parts). By employing a two-stage perceptron, the method overcomes the limited expressive capacity of single linear layer perceptrons in traditional Transformers, thereby improving the model’s sensitivity to fine-grained features of facial expressions and enhancing overall classification performance. Additionally, an internal and external random occlusion masking strategy is used to strengthen the model’s robustness and generalization capability. Meanwhile, the design of a hybrid loss function effectively addresses the issue of class imbalance in samples, ensuring the stability of model training. Experimental results demonstrate that, in the task of robust facial expression recognition under occluded conditions, the proposed method achieves high recognition accuracy on two public datasets, RAF-DB and FERPlus, with rates of 90.66% and 90.01%, respectively. The F₁ scores also show a significant increase, indicating the model’s strong robustness to occluded expressions and a certain advantage compared to other state-of-the-art methods. However, the proposed method still has room for optimization in terms of model parameter size, which may pose challenges such as high computational resource consumption and limited inference speed in practical application scenarios. Specifically, while the current model structure effectively improves the accuracy and robustness of expression recognition under occluded conditions, its relatively high parameter count may restrict deployment efficiency on edge devices with limited computational capabilities (e.g., mobile terminals, embedded systems). Furthermore, the increased model complexity may also impact training and inference energy consumption, which is not conducive to long-term or large-scale real-time applications. To advance the practical adoption of this technology, future research will focus on achieving model lightweighting while maintaining high recognition accuracy, thereby further enhancing the model’s inference speed and energy efficiency across diverse real-world scenarios.