Joint Learning for Mask-Aware Facial Expression Recognition Based on Exposed Feature Analysis and Occlusion Feature Enhancement

Abstract

Facial expression recognition (FER), applied in fields such as interaction and intelligent security, has seen widespread development with the advancement of machine vision technology. However, in natural environments, faces are often obscured by masks, posture, and body parts, leading to incomplete features, which results in poor accuracy of existing facial expression recognition algorithms. Apart from extreme scenarios where facial features are completely blocked, the key information of facial expression features is mostly preserved in most cases, yet insufficient parsing of these features leads to poor recognition results. To address this, we propose a novel joint learning framework that integrates explicit occlusion parsing and feature enhancement. Our model consists of three core modules: a Facial Occlusion Parsing Module (FOPM) for real-time occlusion estimation, an Expression Feature Fusion Module (EFFM) for integrating appearance and geometric features, and a Facial Expression Recognition Module (FERM) for final classification. Extensive experiments under a rigorous and reproducible protocol demonstrate significant improvements of our approach. On the masked facial expression datasets RAF-DB and FER+, our model achieves accuracies of 91.24% and 90.18%, surpassing previous state-of-the-art methods by 2.62% and 0.96%, respectively. Additional evaluation on a real-world masked dataset with diverse mask types further confirms the robustness and generalizability of our method, where it attains an accuracy of 89.75%. Moreover, the model maintains high computational efficiency with an inference time of 12.4 ms per image. By effectively parsing and integrating partially obscured facial features, our approach enables more accurate and robust expression recognition, which is essential for real-world applications in interaction and intelligent security systems.

1. Introduction

Enhancing the quality of interaction is a crucial goal, and within this pursuit, research in facial expression recognition (FER) plays a pivotal role. As a vital sub-field of computer vision, this discipline is dedicated to discerning human emotional states from facial expressions captured in images or videos. Traditional facial expression models mainly rely on hand-crafted features and machine learning techniques such as HOG [1] and Haar [2]. With the rapid development of deep learning, methods based on convolutional neural networks including CNN [3] and VGG [4] have significantly improved recognition performance.

Recent approaches have further advanced the field by incorporating structural and contextual facial information. For instance, the High-Definition Neural Network (HDNN) [5] excels at capturing relational features within facial images, integrating multi-scale networks to better represent emotional attributes. Similarly, vision transformer-based frameworks have been introduced to model long-range dependencies beneficial for FER. These methods achieve impressive accuracy under controlled conditions with fully visible faces. However, in real-world scenarios, facial expressions are often partially occluded due to obstacles such as masks, sunglasses, or head pose variations, which significantly degrades recognition performance. Figure 1 illustrates a comparison of facial expression recognition (FER) under ideal conditions versus common occlusion factors in natural environments (e.g., masks, poses, objects). Occlusions obscure key features, causing significant drops in recognition accuracy for traditional models—precisely the core challenge addressed by this research.

Several recent works have attempted to address occlusion-aware FER. The Face-mask-aware Face Parsing Model (FFPM) introduces a face parsing module combined with a transformer architecture to improve recognition under mask occlusion [6]. Li et al. proposed a Multi-Angle Feature Extraction (MAFE) method aimed at improving recognition accuracy under occlusion conditions by incorporating multi-scale global features, local fine-grained features, and salient region features [7]. Kim et al. employs a Spatial Transformer Network (STN) with an attention mechanism to leverage specific facial regions that make the primary contribution (or are most relevant) to particular facial expressions [8]. Devasena et al. proposed a novel Twinned-Attention Network (Twinned-Att) for efficient FER in occluded images [9]. Liang et al. proposed a Convolutional-Transformer Dual-Branch Network (CT-DBN) that leverages both local and global contextual information to achieve robust FER against real-word occlusion and head pose variations [10].

While these methods demonstrate progress, they exhibit three primary limitations that our work aims to address. First, methods like FFPM [6] and MVT [11] are primarily designed for and evaluated on standardized mask occlusions, lacking demonstrated gen-eralization to other common, unpredictable occlusions (e.g., hands, hair, accessories). Second, approaches relying solely on implicit attention mechanisms (e.g., STN [8], Twinned-Att [9]) or multi-branch feature extraction (e.g., MAFE [7], CT-DBN [10]) do not explicitly model the occlusion structure itself. They lack a dedicated mechanism to esti-mate the occlusion mask and use it as a structural prior to guide feature learning and re-covery. Third, current methods lack a unified framework that explicitly models occlusion as a first-class structural entity and leverages this explicit understanding to dynamically guide feature extraction and recovery across diverse, unpredictable occlusion patterns. Specifically, reconstruction-based methods become highly speculative when major facial regions are obscured; holistic methods do not adequately model partial occlusion; and sub-region/attention-based approaches rely on implicit, often unreliable, feature re-weighting without explicit structural understanding of the occlusion itself. This fundamental limitation—the inability to explicitly parse the occlusion configuration and use it as a structural prior—forces models to treat occluded and non-occluded regions indistinctly, leading to suboptimal feature utilization, poor generalization beyond trained occlusion types, and significant performance degradation in real-world scenarios where occlusion type, size, and location are highly variable.

In contrast to these works, our proposed framework introduces a novel Facial Occlusion Parsing Module (FOPM) that explicitly estimates a generic occlusion mask in real-time, providing a clear structural prior of the obstruction. This explicit parsing enables dynamic, adaptive guidance for subsequent feature extraction—focusing computation on salient visible regions while utilizing contextual reasoning to infer cues from occluded areas. Furthermore, unlike multi-branch models that process regions independently, our efficient Expression Feature Fusion Module (EFFM) integrates appearance features with geometric landmarks through a gated fusion mechanism, enabling robust and interpretable feature recombination under diverse occlusion patterns. This joint learning strategy of explicit occlusion parsing and guided feature enhancement differentiates our approach by offering a lightweight, plug-and-play solution that is not limited to pre-defined occlusion types and maintains high computational efficiency.

To bridge this gap, we propose a novel and highly practical facial expression recognition framework that emphasizes engineering applicability through its lightweight, plug-and-play occlusion parsing module and efficient feature fusion mechanism. Our approach introduces a Facial Occlusion Parsing Module (FOPM) that explicitly estimates the facial occlusion mask and guides the feature extraction process to focus on salient visible regions while recovering cues from occluded areas via contextual inference. Unlike multi-branch models that process different facial regions independently or attention-based methods that rely solely on implicit feature re-weighting, our FOPM provides an explicit structural prior of occlusion patterns, which is jointly optimized with the expression recognition task. This enables more accurate and interpretable guidance for feature extraction, especially under diverse and unpredictable occlusion scenarios. The FOPM module is computationally efficient and can be readily integrated into existing FER pipelines without significant architectural changes. The framework further incorporates local feature enhancement and global expression representation through a unified fusion mechanism, enabling efficient aggregation of both fine-grained details and high-level semantic information without the need for complex multi-branch coordination or heavy attention computation. Through careful design of the feature fusion components, our method maintains low computational overhead while significantly improving recognition accuracy under various occlusion conditions.

In summary, the main contributions of this work are as follows:

(1)

Our method proposes a lightweight Facial Occlusion Parsing Module (FOPM) that explicitly estimates occlusion patterns and integrates this structural prior to guide feature extraction. This plug-and-play component can be easily deployed in existing FER systems, significantly enhancing their robustness to partial occlusions with low computational cost.

(2)

Our study introduces an efficient end-to-end trainable framework that performs robust occlusion-aware FER by synergistically analyzing both exposed and occluded facial regions. The system employs innovative attention mechanisms that enable practical feature transfer and enhancement without compromising inference speed.

(3)

We conduct a thorough evaluation on publicly available and mask-aware faces, showcasing the superior performance of our proposed method in comparison to various recent methods.

2. Related Works

To better position our contribution within the existing literature and highlight the methodological evolution in this field, this section reviews related work through a structured categorical framework. We organize the discussion into four distinct methodological paradigms that represent the chronological and conceptual progression in occlusion-aware FER research.

2.1. Occlusion Problem in FER

Approaches to tackle occlusion in computer vision are generally categorized into three main types: reconstruction-based, holistic-based, and sub-region-based methods. The majority of FER research to date has concentrated on evaluating these approaches with non-occluded facial images [12,13,14,15,16]. However, there is a growing body of research addressing partial occlusion issues in both two-dimensional and three-dimensional FER [17,18,19].

• Reconstruction-Based Paradigm: The earliest approaches to handling occlusion in FER followed a reconstruction-based paradigm, aiming to restore occluded facial regions before recognition. These methods typically employ generative techniques such as Generative Adversarial Networks (GANs) [20] or autoencoders to reconstruct complete facial images from partially occluded inputs. The underlying assumption is that once reconstructed, standard FER methods can be applied effectively. While conceptually straightforward, these methods face significant limitations: they become highly speculative when major facial regions (e.g., over 50% of the face with masks) are obscured, often introducing artifacts that may mislead subsequent recognition. Furthermore, the reconstruction quality heavily depends on the occlusion pattern and extent, making this approach unreliable for real-world applications with diverse occlusion types.
• Holistic Representation Paradigm: The second paradigm shift moved toward holistic representation methods that leverage sparse signal processing and global feature representations inherently robust to partial occlusion [21,22]. These approaches posit that discriminative expression information is distributed across the face rather than concentrated in specific regions. By learning robust global representations through techniques like sparse coding or deep global descriptors, these methods avoid the need for explicit occlusion detection or reconstruction. However, while demonstrating resilience in occluded object classification generally, these methods often fail to capture the subtle local variations crucial for fine-grained expression discrimination, particularly when critical expression regions (e.g., mouth for happiness, eyes for surprise) are obscured.
• Local Feature Emphasis Paradigm: Recognizing the limitations of global approaches, the field evolved toward local feature emphasis methods that explicitly handle occlusion through region-based processing. This paradigm encompasses several sub-categories: Patch-based Methods segment facial images into smaller patches (overlapping or non-overlapping) and employ attention mechanisms or weighting schemes to emphasize non-occluded regions [23]. The ACNN framework, for instance, combines patch-based and global-local attention to mitigate occlusion effects. Attention-based Methods use learned attention weights to dynamically focus on semantically important and visible regions. The RAN framework [24] introduced relation-attention modules to adaptively capture vital facial regions, while OADN [25] combined landmark-guided attention with regional features to identify non-occluded areas. Landmark-based Methods utilize facial landmarks to guide feature extraction from specific semantic regions, providing structural priors for handling occlusion. While representing significant advancement, these methods still fundamentally treat occlusion as a nuisance to be avoided rather than a structural element to be explicitly modeled.
• Joint Learning Paradigm (Emerging): The most recent evolution in occlusion-aware FER moves toward joint learning frameworks that explicitly model occlusion as an integral part of the recognition process. Rather than merely avoiding or compensating for occlusion, these methods aim to understand the occlusion configuration and leverage this understanding to guide feature extraction and fusion [26]. This emerging paradigm includes: Mask-aware Specialized Methods that specifically address predictable occlusions like face masks [6,27], often using mask detection as an auxiliary task. Geometry-Appearance Fusion Methods that combine geometric facial structure information with appearance features for more robust recognition under occlusion [10]. Explicit Occlusion Modeling Approaches that directly estimate occlusion patterns and use this information to guide adaptive feature processing—the direction our work advances.

2.2. Mask-Aware FER as a Specialized Domain

Face-mask-aware FER represents a specialized and practically important sub-domain within occlusion-aware FER, where the occlusion is more predictable due to the consistent shape and size of face masks [11,27,28,29,30,31,32,33,34,35]. The methodological evolution in this sub-domain mirrors the broader progression described above, from early attempts at reconstruction [20] to recent joint learning approaches that explicitly model mask presence [6,11,27,34,35].

A study that considered face masks in FER focused solely on recognizing emotions from the eye region, evaluating this method on the mask-aware FER-2013 dataset, which encompasses seven distinct emotions [28]. However, this approach neglects other crucial facial areas, such as the forehead, when isolating the eye region using landmark detection methods, thereby reducing the accuracy of the FER system. We have observed that existing landmark detection tools, including Openface2.0 [29], which is utilized in current studies, are less accurate when dealing with facial images that include face masks, further diminishing the detection accuracy of the FER system [30,31,32]. Although some studies have attempted to address the robustness of landmark detection, there remain unresolved issues: there is still a significant gap in landmark detection accuracy between masked and unmasked faces, and the landmarks can only roughly differentiate between the covered and uncovered facial regions.

In prior research, Wang et al. [33] introduced a Self-Cure Network (SCN) to mitigate uncertainties in FER through a straightforward yet effective approach. SCN addresses uncertainty in two key ways: by employing a self-attention mechanism over the FER dataset to assign weights to each training sample based on a ranking regularization, and through a meticulous relabeling mechanism that adjusts the labels of the lowest-ranked samples. This framework effectively realized FER. Subsequently, Yang et al. [27] developed a two-stage, attention-based deep network designed to manage three emotions—positive, negative, and neutral—addressing face mask related challenges in FER. In the initial masked/unmasked binary classification phase, the attention mechanism was integrated into the classifier to generate attention heatmaps for masked areas and reverse attention heatmaps for unmasked regions. In the subsequent mask-aware FER classification stage, the attention mechanism directed the model to focus on the most critical facial areas for FER classification, prioritizing the unmasked region over the masked one. Ma et al. [34] proposed Convolutional Visual Transformers (CVT) to handle FER in uncontrolled environments through two primary steps. First, CVT introduced an attentional selective fusion (ASF) to leverage feature maps produced by dual-branch CNNs. ASF captured discriminative information by fusing multiple features with a combination of global and local attention. The fused feature maps were then flattened and projected into sequences of visual words. Second, drawing inspiration from the success of Transformers in natural language processing, CVT modeled relationships between these visual words using global self-attention. This local and global fused learning approach benefited mask-aware FER. Li et al. [11] were the first to propose a novel pure transformer-based mask vision transformer (MVT) for FER in the wild, comprising two modules: a transformer-based mask generation network (MGN) to create a mask that filters out complex backgrounds and occlusions in face images, and a dynamic relabeling module to correct erroneous labels in wild FER datasets. In 2023, Liu et al. [35] introduced a Patch Attention Convolutional Vision Transformer (PACVT) to address the occlusion FER problem. The backbone convolutional neural network extracted facial feature maps, which were divided into multiple regional patches to capture both local and global features. The PACVT framework achieved commendable performance in mask-aware FER.

Despite this approach showing superior performance compared to other partial occlusion FER methods, there are two main deficiencies that require resolution. The attention mechanism only achieves a relatively high accuracy in distinguishing the face mask region from other image areas. The re-weighting of the detected masked and unmasked regions is not adaptive. Additionally, a limitation of our earlier work was the evaluation of only three emotions.

3. Methods

This section provides a detailed description of our proposed joint learning framework for mask-aware FER. The overall architecture is first outlined, followed by in-depth explanations of its three core components and the joint loss function that enables end-to-end training.

3.1. Framework

As illustrated in Figure 2a, our framework is composed of three integral, cascaded modules that work in concert. Our facial expression recognition framework is designed to accurately interpret and analyze human emotions through facial expressions. It is composed of three integral components, Facial Occlusion Parsing Module (FOPM), Expression Feature Fusion Module (EFFM) and Facial Expression Recognition Module (FERM). They play a crucial role in the overall process of expression analysis.

3.2. Facial Occlusion Parsing Module

The facial occlusion parsing module includes a facial occlusion parsing component and a facial expression feature component under occlusion constraints.

This module is the first step of our framework and is responsible for detecting and parsing any occlusions that may be present in the facial region. Occlusions can be anything from masks to scarves or even hands that might be covering parts of the face, which can impede the accuracy of expression recognition. The facial occlusion parsing module uses advanced algorithms to identify these obstructions and either adjust the analysis to account for them or flag the input as potentially unreliable for further processing. This ensures that our method can handle real-world scenarios where faces might not be fully visible.

The architecture of the FOPM, depicted in Figure 2b, is a lightweight convolutional network designed for efficient feature pre-processing. The input I (with dimensions 3 × 224 × 224) is processed sequentially through three convolutional groups to output the refined feature representation P. The detailed configuration of each group is as follows:

Group 1: This group consists of two convolutional layers. The first layer has a kernel size of 3 × 3, a stride of 2, and padding of 1, converting the input to 64 channels. This is followed by a ReLU activation. The second layer has a kernel size of 3 × 3, a stride of 1, and padding of 1, maintaining 64 channels, and is also followed by a ReLU activation. The output feature map dimensions are 64 × 112 × 112.

Group 2: This group mirrors the structure of Group 1 but increases the channel dimension. It contains two convolutional layers with kernel sizes of 3 × 3, stride of 1, and padding of 1. The first layer in this group increases the channels from 64 to 128, and the second layer maintains 128 channels. Each layer is followed by a ReLU activation. A max-pooling layer (kernel size = 2 × 2, stride = 2) is applied after the activations, resulting in an output feature map size of 128 × 56 × 56.

Group 3: This final group is configured to further refine and prepare the features for subsequent modules. It contains three convolutional layers. The first two layers have kernel sizes of 3 × 3, stride of 1, and padding of 1, increasing channels from 128 to 256 and then maintaining 256 channels. The third layer uses a kernel size of 1 × 1 with a stride of 1 to project the features to the final output dimension of 512 channels. Each layer is followed by a ReLU activation. The output P of this group is a feature map with dimensions 512 × 56 × 56.

The operation of the FOPM can be summarized by the function:

$$P=FOP(I),$$

(1)

where I denotes the input occlusion face image, and FOP(·) represents the function of the FOPM, which is dedicated to the precise pre-processing of facial features to output a refined feature representation P.

3.3. Expression Feature Fusion Module

The expression recognition feature fusion module includes local feature fusion expression and global feature expression. Local feature expression mainly refers to the situation where partial occlusion affects facial features, but the characteristics of facial expressions are still preserved, through the expression of the extracted features. Global feature expression is achieved by parsing facial features to realize a comprehensive expression of features.

Once the facial occlusions have been addressed by the FOPM, the Expression Feature Fusion Module (EFFM) is tasked with integrating the two primary types of facial information: appearance features (texture and color information from the CNN-extracted feature maps) and geometric features (structural information from the predicted facial landmarks). The integration of these complementary information streams is achieved through a dedicated two-branch process within the EFFM, culminating in a gated fusion mechanism.

The EFFM is composed of two sub-networks: the Feature Decomposition Network (FDN) and the Feature Recomposition Network (FRN). The FDN operates on the appearance feature stream. It takes the global feature map P (512 × 56 × 56) from the FOPM and decomposes it into M = 4 distinct regional feature vectors. Each branch passes through a Patch-Gated Unit (PGUnit), which acts as an attention mechanism to generate region-specific weight masks, effectively highlighting salient non-occluded areas.

Concurrently, the geometric feature stream is processed. The facial landmark heat maps (e.g., 68 × 56 × 56), predicted by the U-Net, are average-pooled and projected into a 128-dimensional geometric prior vector. This vector encodes the spatial structure and configuration of the face.

The FRN module performs the explicit integration. Each appearance-based feature vector from the FDN is first modulated by its corresponding attention weights from the PGUnit. The modulated vectors are then processed through a two-layer residual structure. Crucially, the 128D geometric prior vector is concatenated with each of these processed appearance vectors. This combined vector, representing both the local appearance and the global geometric context, is then passed through a gating layer (Sigmoid activation) that learns to dynamically control the flow of information from each branch based on the current geometric configuration. The final output feature map is obtained via a weighted summation of all gated branch features, effectively fusing appearance and geometric information to form a robust and occlusion-invariant representation.

This design ensures that the model does not rely on appearance features alone. The geometric landmarks provide a structural prior that guides the attention mechanism (PGUnit) to focus on semantically important and visible regions (e.g., the eyes and eyebrows when the mouth is occluded), leading to a more informed and robust fusion of features.

As depicted in Figure 2a, our approach firstly predicts the facial masks M from the pre-processing outputs P. In Figure 2c, the image P is processed through a U-Net architecture to generate the facial mask M. This mask serves as the facial mesh, effectively enhancing the preservation of the deformable aspects during the motion process.

In order to achieve precise registration of facial information amidst motion and to effectively capture facial features within the constraints of facial masks like M1 and M2, we have developed two U-Net architectures. These are designed to extract the necessary characteristic features. By individually processing the face frames P1 and P2, our method is capable of extracting the matching kernel estimator and the associated offset estimator. This enables the accurate alignment of facial information across multiple frames. Additionally, both masks M1 and M2 are input into a separate U-Net to generate a mask estimator, which is instrumental in the expression of moving faces.

3.4. Facial Expression Recognition Module

The facial expression recognition module can achieve facial expression recognition by thoroughly parsing feature information and integrating the features.

The final component of our framework is the facial expression recognition module. This is where the actual classification of facial expressions takes place. It takes the enriched feature set produced by the feature fusion module and applies a series of classification algorithms to determine the most likely emotional state of the individual. This could range from basic emotions such as happiness, sadness, anger, surprise, fear, and disgust, to more complex and nuanced expressions. The module is designed to be adaptable and can be trained on diverse datasets to improve its ability to recognize expressions across different demographics and cultural contexts.

The Facial Expression Recognition Module (FERM) consists of one 3-layer convolutional encoder, three stacked ResNet blocks, followed by three additional CNN layers and a ReLU activation layer. The output is passed through a global average pooling (GAP) layer to produce a 512-dimensional feature vector. This vector is then fed into the final classification head, which is implemented as a sequential fully connected (FC) layer that maps the 512-dimensional input to a 7-dimensional output vector (corresponding to the 7 emotion classes), followed by a Softmax activation function to produce the final class probabilities.

This architecture balances depth and efficiency. The residual design ensures stability in training, and the deep feature representations allow the model to differentiate nuanced emotional cues even under occlusion.

As depicted in Figure 2d, DVM facilitates the multiple aligned results {A} to the final multi-frame facial emotion recognition result O. In this module, the inputs A1, A2, and A3 are fed into the network D_Net for the recognition output. The inference process can be expressed as:

$$O=DNet(A),$$

(2)

where {A} = {A1, A2, A3} represents the set of multiple aligned feature maps from the previous module, DNet(·) denotes the function of the recognition network DNet, and O is the final output of the facial emotion recognition result.

This network is organized into the groups. As depicted in Figure 2d, the architecture begins with a CNN structure, which is then followed by the integration of three residual networks. Subsequently, three additional CNN structures are sequentially concatenated, culminating in the inclusion of a ReLU module. The input features are concatenated with the learned residual features to produce the resulting output features. This module facilitates the results of the final emotion features from the alignment results.

3.5. Loss Functions

The setting of the loss function also mainly refers to the design of the network structure, which mainly includes three parts: the loss function for the module FOPM, the loss function for the module EFFM, and the loss function for the module FERM.

The overall loss function is a weighted sum of the losses from the three core modules: FOPM, EFFM, and FERM, enabling joint end-to-end training:

$$L_t=L_p+\beta L_m+\gamma L_d,$$

(3)

where L_p, L_m, and L_d are the loss functions for the FOPM, EFFM, and FERM, respectively. Specifically, L_p is a Mean Squared Error (MSE) loss between the predicted and ground-truth occlusion maps, L_m is an MSE loss for the landmark heatmap regression task within the EFFM, and L_d is the Categorical Cross-Entropy loss applied to the Softmax output of the FERM, calculating the divergence between the predicted emotion probability distribution and the ground-truth one-hot label.

The terms β and γ are balance hyper-parameters that harmonize the contribution of each module’s loss to the total training objective L_t. To determine their optimal values, we conducted a grid search over a range of [0.1, 0.3, 0.5, 0.7, 0.9] for both parameters on the validation set. The combination that yielded the best performance was β = 0.6 and γ = 0.4. This weighting scheme indicates that the occlusion parsing (L_p) and feature fusion (L_m) tasks, which provide crucial structural priors and enriched features, are assigned a higher relative importance during the initial phases of learning compared to the final classification loss (L_d). The average ratios of L_p, L_m, and L_d to the total loss L_t at the end of training were approximately 38%, 25%, and 37%, respectively, demonstrating that all three components contributed significantly to the overall optimization objective.

4. Experiments

To validate the effectiveness and practicality of the proposed model for Facial Expression Recognition (FER) under occlusion settings, we conducted systematic experiments on widely used masked facial expression datasets. This section details the experimental setup, datasets, evaluation metrics, ablation studies, comparisons with state-of-the-art methods, and provides visualizations to further interpret the model’s behavior.

4.1. Datasets

Experiments were conducted on the following two public datasets:

(1)

RAF-DB (Real-world Affective Faces Database) [36] comprises 29,672 facial images, each annotated with one of seven basic expressions: Surprise, Fear, Disgust, Happiness, Sadness, Anger, and Neutral. We strictly adhered to the official dataset split: 12,271 images for training and 3068 images for testing.

(2)

FER+ (Face Emotion Recognition Plus) [37], an enhanced version of FER2013, contains 28,315 training images and 3589 testing images, also labeled with seven expression categories.

To simulate real-world mask occlusions, synthetic mask overlays were generated using the Dlib toolkit [38] and applied only to the detected facial regions. The composite mask overlay is presented in Figure 3. A subset of 10% of the official training set was randomly selected and held out as a validation set for hyperparameter tuning and early stopping in both cases; this validation set was never used for the final evaluation reported in the results. Crucially, all synthetic masks were generated after the official train/test/validation split. Furthermore, we ensured that no identity (if present across different splits) shared the same synthetic mask pattern between the training and test sets, preventing a potential source of bias and data leakage.

Additionally, to validate the model’s generalization capability under real-world occlusion conditions, we curated a RealMask-FER dataset. It consists of 5832 real-world masked facial images collected from public sources, encompassing a wide variety of mask types (medical, cloth, patterned) and wearing styles (e.g., below the nose, fully covered).

4.2. Implementation Details

4.2.1. Training Configuration

(1)

Framework & Hardware: Implemented in PyTorch 1.12 [39], running on Ubuntu 18.04 with 2 × NVIDIA GeForce RTX 3090 GPUs.

(2)

Optimizer: AdamW [40] was used with decay rates β₁ = 0.9 and β₂ = 0.99, and a weight decay of 1 × 10⁻⁴.

(3)

Learning Rate Schedule: The initial learning rate was set to 0.002, and was reduced by a factor of 0.2 every 6 epochs.

(4)

Training Strategy: Models were trained for a maximum of 300 epochs with a batch size of 16. Early stopping was employed with a patience of 20 epochs, monitoring the validation accuracy. The model with the best validation performance was restored for final evaluation.

(5)

Input Resolution: All input images were resized to 3 × 224 × 224 pixels.

4.2.2. Model Complexity and Efficiency

The complete model has approximately 9.8 million parameters:

(1)

The Facial Occlusion Parsing Module (FOPM including the U-Net): ~3.5 M parameters.

(2)

The Expression Feature Fusion and Recognition Modules (EFFM + FERM): ~6.3 M parameters.

Inference performance was evaluated on a single NVIDIA RTX 3090 GPU with a batch size of 16 and an input resolution of 224 × 224. The average inference time per image is 12.4 ms, enabling a near real-time throughput of ~80 frames per second (FPS). The GPU memory consumption during inference is approximately 2.1 GB. For context, the recent PACVT [35] model reports an average inference time of 16.8 ms under similar conditions, demonstrating our model’s superior efficiency.

4.3. Evaluation Metrics

We adopt four widely used metrics for a comprehensive evaluation: Accuracy, Precision, Recall, and F1-Score.

(1)

Accuracy measures the overall correctness of predictions:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},$$

(4)

(2)

Precision assesses the correctness of positive predictions:

$$Precision={\displaystyle \frac{TP}{TP+FP}},$$

(5)

(3)

Recall evaluates the ability to detect positive samples:

$$Recall={\displaystyle \frac{TP}{TP+FN}},$$

(6)

(4)

F1-Score is the harmonic mean of Precision and Recall, providing a balanced measure especially under class imbalance:

$$\mathrm{F}1\text{-}\mathrm{Score}=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}},$$

(7)

where TP, TN, FP, and FN denote True Positives, True Negatives, False Positives, and False Negatives, respectively.

4.4. Ablation Study

To validate the contribution of each proposed module (FOPM, EFFM, FERM), we conducted ablation experiments on both the RAF-DB and FER+ datasets. The results are detailed in

Table 1. The performance AP (Average Precision) of modules FOPM, EFFM, and FERM on the two datasets RAF-DB and FER+.

Methods	FOPM	EFFM	FERM	RAF-DB	FER+
w/o FOPM	×	√	√	88.91%	87.13%
w/o EFFM	√	×	√	90.37%	89.37%
w/o FERM	√	√	×	89.56%	88.75%
Ours	√	√	√	91.24%	90.18%

.

The following observations can be derived:

(1)

Efficacy of FOPM: The FOPM demonstrates the most significant individual impact, improving accuracy by 2.33% and 3.05% on RAF-DB and FER+, respectively, compared to the model without it. This underscores the critical importance of explicit occlusion parsing for recovering discriminative features under occlusion.

(2)

Role of EFFM and FERM: While the individual improvements from EFFM and FERM are relatively smaller, their presence is essential. The cascaded integration of all three modules yields a synergistic effect, where the combined accuracy gain exceeds the sum of individual improvements. This visually demonstrates how the modules work in concert: FOPM identifies usable regions, EFFM performs in-depth analysis of expressive details within these regions, and FERM strategically enhances features potentially weakened by occlusion.

4.5. Comparison with State-of-the-Art Methods

4.5.1. Implementation Details for Compared Methods

To ensure a verifiable, detailed, and fair comparison, we retrained the following representative baseline methods using a standardized and reproducible protocol—RAN [23], SCN [33], CVT [34], MVT [11], and PACVT [35]. Each model was implemented using its official publicly released code, strictly adhering to the architectural details and loss functions specified in the original publications. To adapt these methods to the mask-aware FER task, we replaced the original image inputs with our synthetically masked versions of the RAF-DB and FER+ datasets. All models were trained from scratch under identical hardware and software environments. Key training settings were unified: batch size of 16, AdamW optimizer (β₁ = 0.9, β₂ = 0.99), weight decay of 1 × 10⁻⁴. The initial learning rate was tuned individually for each model via grid search over {0.001, 0.002, 0.005} using the same validation split. Each method was trained for a maximum of 300 epochs with early stopping (patience = 20 epochs) based on validation accuracy. A fixed random seed (42) was set for all experiments to ensure reproducibility.

The same synthetic mask generator (Dlib [38]) was applied uniformly to all methods during both training and testing. Evaluation was performed on the identical masked test sets of RAF-DB and FER+, using the same metrics.

4.5.2. Quantitative Results

The results under this standardized protocol are presented in

Table 2. Performance comparison on the RAF-DB dataset.

Methods	Years	AP	Precision	Recall	F1-Score	Training Protocol
RAN	2018	86.90%	86.45%	86.20%	86.32%	Retrained on our masked data
SCN	2020	87.03%	86.88%	86.70%	86.79%	Retrained on our masked data
CVT	2020	88.14%	87.92%	87.65%	87.78%	Retrained on our masked data
MVT	2021	88.62%	88.40%	88.15%	88.27%	Retrained on our masked data
PACVT	2023	88.21%	87.95%	87.73%	87.84%	Retrained on our masked data
Ours	2025	91.24%	91.05%	90.88%	90.96%	Trained on our masked data

and

Table 3. Performance comparison on the FER+ dataset.

Methods	Years	AP	Precision	Recall	F1-Score	Training Protocol
RAN	2018	87.85%	87.50%	87.20%	87.35%	Retrained on our masked data
SCN	2020	88.01%	87.75%	87.52%	87.63%	Retrained on our masked data
CVT	2020	88.81%	88.55%	88.30%	88.42%	Retrained on our masked data
MVT	2021	89.22%	89.00%	88.75%	88.87%	Retrained on our masked data
PACVT	2023	88.72%	88.45%	88.20%	88.32%	Retrained on our masked data
Ours	2025	90.18%	89.95%	89.70%	89.82%	Trained on our masked data

. The following discussion can be derived from these results:

(1)

Baseline Performance: From Table 2, early influential methods like RAN and SCN achieved accuracies of 86.90% and 87.03%, respectively. These established a foundational performance level but highlighted the difficulty of occlusion handling. From Table 3, Early methods RAN and SCN achieved accuracies of 87.85% and 88.01%, respectively. These established a foundational performance level but indicated room for improvement in handling occlusion.

(2)

Progressive Improvements: From Table 2, subsequent advancements, represented by CVT (88.14%) and the 2021 state-of-the-art MVT (88.62%), demonstrated significant progress. The improvement from 87% to 88.6% underscores the research community’s ongoing efforts to tackle occlusion complexity. PACVT further refined these approaches, achieving 88.21%. From Table 3, Subsequent methods CVT (88.81%) and the then state-of-the-art MVT (89.22%) demonstrated significant progress, pushing accuracy closer to 89%. PACVT further refined these approaches, achieving 88.72%.

(3)

Our Model: Our proposed model achieves a remarkable accuracy of 91.24%, surpassing the previous best (MVT at 88.62%) by a substantial margin of 2.62 percentage points. This result is highlighted as the highest in Table 2. Our proposed model achieves a notable accuracy of 90.18%, surpassing the previous best (MVT at 89.22%) by 0.96 percentage points. This result is highlighted as the highest in Table 3.

(4)

Statistical Significance Analysis: To rigorously validate that the performance improvement of our model is statistically significant and not due to random chance, we conducted the McNemar’s test [41], a widely used non-parametric test for comparing the proportions of matched pairs in classification tasks. The null hypothesis posits that there is no difference in the error rates between our proposed method and the compared model (MVT). The tests were performed on the predictions of our model and the best previous model (MVT) across both RAF-DB and FER+ datasets. The resulting * p *-values were extremely small (* p * < 0.001 for both datasets), allowing us to confidently reject the null hypothesis. This result provides strong statistical evidence that the superior accuracy achieved by our proposed model is statistically significant.

4.5.3. Generalization Evaluation on Real-World Occlusions

To rigorously validate the model’s generalization capability beyond controlled synthetic patterns and address its real-world applicability, we conducted a dedicated quantitative evaluation on the RealMask-FER dataset. This dataset, introduced in Section 4.1, presents a significantly more challenging and realistic scenario due to its diverse spectrum of mask types, wearing styles, and uncontrolled environmental conditions.

Results and Analysis: The quantitative comparison on this challenging real-world set is paramount and the results are presented in

Table 4. Performance comparison on the RealMask-FER dataset.

Methods	Years	AP	Precision	Recall	F1-Score	Training Protocol
MVT	2021	87.92%	87.65%	87.40%	87.52%	Retrained on our masked data
PACVT	2023	87.30%	87.02%	86.75%	86.88%	Retrained on our masked data
Ours	2025	89.75%	89.50%	89.25%	89.37%	Trained on our masked data

. The following discussion can be derived from these results:

(1)

Performance Gap Highlights Real-World Challenge: A consistent performance drop is observed across all models when compared to their results on synthetic masked benchmarks (Table 2 and Table 3). This drop is not a failure but a validation of the RealMask-FER dataset’s complexity and its success in capturing the challenging domain gap between idealized synthetic occlusions and messy real-world scenarios.

(2)

Superior Generalization of Our Model: Our model demonstrates remarkable robustness, maintaining a commanding lead with an accuracy of 89.75%. It outperforms the best baseline (MVT [11]) by a significant margin of 1.83%. This substantial gap underscores that our framework’s core innovation—explicit occlusion parsing and guided feature enhancement—is effective at handling the inherent complexity and variety of real-world occlusions, proving its generalization is superior to methods reliant on implicit attention or multi-branch features alone.

(3)

Effectiveness of Explicit Parsing: The fact that our model, trained solely on synthetically generated data, generalizes effectively to real and varied masks indicates that the Facial Occlusion Parsing Module (FOPM) learns a fundamental, generic representation of “occlusion” as a structural entity, rather than memorizing the texture or pattern of a synthetic mask. This explicit structural prior is crucial for adapting to unseen occlusion patterns.

Conclusion of the Evaluation: This evaluation moves beyond standard benchmarks and provides compelling evidence for the practical viability of our proposed method. The results substantiate that our model offers not only higher accuracy but also superior generalization and robustness, making it highly suitable for real-world applications where occlusion types are unpredictable and diverse. The performance on RealMask-FER is a direct validation of our model’s ability to fulfill its design purpose: performing robust facial expression recognition in natural, unconstrained environments.

4.6. Visualization and Discussion

4.6.1. Confusion Matrix Analysis

Figure 4 and Figure 5 show the confusion matrices of our model on the RAF-DB and FER+ datasets, respectively [42]. The following discussion can be derived from these results:

(1)

High Overall Accuracy: Strong diagonal values across all expressions indicate high per-class recognition accuracy.

(2)

Negative Valence Confusion: A consistent pattern of confusion exists among negative valence emotions (e.g., Sad, Fear, Angry, Disgust), as indicated by significant off-diagonal values between these classes. This is a common challenge in FER due to similar facial muscle activations.

(3)

Performance Extremes: The Happy expression consistently shows the strongest diagonal intensity (highest accuracy), confirming its robust recognition even with masks. Conversely, the Disgust expression shows the weakest diagonal intensity and the most dispersed confusion, often being misclassified as Anger or Sadness.

(4)

Specific Neutral vs. Sad Confusion: Figure 5 highlights a distinct, reciprocal confusion pattern between Neutral and Sad. This suggests that occlusion masks critically obscure key lower-face discriminators for Sadness (e.g., mouth corners drawn downwards), making it difficult to distinguish from a Neutral expression based primarily on upper-face features.

4.6.2. Intermediate Feature Visualization

Figure 6 provides a visual breakdown of our model’s processing pipeline for a partially occluded face. The following discussion can be derived from these results:

(1)

Targeted Attention Allocation: The attention map (Figure 6c) exhibits a clear intensity contrast. Significantly higher weights (visually darker) are concentrated on visible, expression-salient regions (e.g., eyes, forehead), while occluded regions (e.g., mouth covered by the mask) are effectively suppressed (visually lighter).

(2)

Salient Feature Fusion: The fused feature map (Figure 6d) visually reflects this modulation by the attention weights. Stronger activation patterns are predominantly over the unoccluded, high-attention regions identified in Figure 6c, while activations over the occluded areas are minimal. This confirms the model’s ability to dynamically focus on reliable visual cues and suppress noisy information from occluded areas.

4.7. Additional Analysis on Neutral vs. Sad Confusion

The observed confusion between Neutral and Sad expressions is a recognized challenge in occlusion-aware FER. To diagnose this, we conducted a post hoc analysis by visualizing the attention weights for misclassified samples. We found that:

(1)

When a Sad expression was misclassified as Neutral, the model’s attention was often incorrectly focused on parts of the upper face that appeared neutral rather than on the subtle cues (e.g., slight eyebrow lowering, eyelid tension) that might persist even with masks.

(2)

When a Neutral expression was misclassified as Sad, the model sometimes over-interpreted mild upper-face features or the overall context as indicative of negative valence.

This specific confusion highlights a limitation in parsing very subtle geometric and appearance features in the upper face under heavy occlusion. To mitigate this in future work, we propose:

(1)

Incorporating more granular, attention-guided landmark detection specifically around the eyes and eyebrows.

(2)

Employing contrastive learning to explicitly minimize the intra-class variation in Neutral and Sad while maximizing the inter-class separation in the feature space.

4.8. Conclusion of Experiments

The experimental results comprehensively validate the efficacy of the proposed joint learning framework. The model achieves state-of-the-art performance on standard benchmarks (RAF-DB, FER+) under synthetic occlusion and, more importantly, demonstrates strong generalization capability on a real-world masked dataset (RealMask-FER). The ablation studies confirm the critical contribution of each module, and the visualizations provide interpretable evidence of the model’s ability to focus on non-occluded, salient regions. The analysis also pinpoints specific challenges (e.g., Neutral-Sad confusion) that guide future research directions. The model’s high computational efficiency makes it suitable for practical, real-world applications.

5. Conclusions

Our research tackles the challenge of facial expression recognition in natural settings, where occlusions such as masks, posture variations, and other obstructions often lead to incomplete facial information and reduced recognition accuracy. We argue that essential expressive cues remain even under partial occlusion, yet prevailing methods struggle to parse these features effectively. To address this, we propose a novel model that comprehensively extracts and integrates facial expression features alongside occlusion context, significantly enhancing recognition performance. Experimental results validate the efficacy of our approach and demonstrate superior performance compared to existing methods.

Our model demonstrates superior performance, achieving state-of-the-art accuracy on the widely used RAF-DB (91.24%) and FER+ (90.18%) benchmarks with synthetic masks. Critically, to address concerns regarding real-world applicability, an additional quantitative evaluation was conducted on the RealMask-FER dataset, which contains a diverse variety of real-world mask types. Our model attained a leading accuracy of 89.75% on this challenging set, outperforming other methods by a significant margin and substantiating its strong generalization capability beyond controlled synthetic patterns. All comparisons were conducted under a verifiable and detailed protocol to ensure fairness and reproducibility.

Limitations and Future Work: Despite the demonstrated robustness, this study has limitations that point to valuable future research directions. While the evaluation on RealMask-FER confirms performance on real masks, the current framework is primarily optimized for mask-like occlusions. Its performance on other common obstructions (e.g., hands, hair, accessories, or extreme head poses) warrants further investigation. Furthermore, the model’s efficacy across an even broader spectrum of environmental conditions, cultural contexts, and spontaneous occlusions remains to be fully explored.

Building on this work, we plan to extend the framework to explicitly handle a more comprehensive range of partial occlusions. This will involve developing more flexible occlusion parsing modules that can dynamically adapt to various obstruction types without retraining. Additionally, rather than focusing solely on generalization from synthetic data, a crucial future direction involves curating and benchmarking on larger-scale, publicly available real-world occluded FER datasets to drive progress in the field. Finally, we will pursue cross-cultural validation and investigate the influence of factors such as lighting and camera angles to foster more adaptable and inclusive models.