CA-GFNet: A Cross-Modal Adaptive Gated Fusion Network for Facial Emotion Recognition

Abstract

Facial emotion recognition (FER) plays an important role in healthcare, human–computer interaction, and intelligent security systems. However, despite recent advances, many state-of-the-art FER methods depend on computationally intensive CNN or transformer backbones and large-scale annotated datasets while suffering noticeable performance degradation under cross-dataset evaluation because of domain shift. These limitations hinder practical usage in resource-constrained and real-world environments. To address this issue, we propose Cross-Adaptive Gated Fusion Network (CA-GFNet), a lightweight dual-stream FER framework that explicitly combines shallow structural features with deep semantic representations. The proposed architecture integrates domain-robust gradient-based descriptors with compact deep features extracted from a VGG-based backbone. After face detection and normalization, the structural stream captures fine-grained local appearance cues, whereas the semantic stream encodes high-level facial configurations. The two feature streams are projected into a shared latent space and adaptively fused using a gated fusion mechanism that learns sample-specific weights, allowing the model to prioritize the more reliable feature source under dataset shift. Extensive experiments on KDEF along with zero-shot cross-dataset evaluation on CK+ using a strict train-on-KDEF/test-on-CK+ protocol with subject-independent splits demonstrate the effectiveness of the proposed method. CA-GFNet achieves 99.30% accuracy on KDEF and 98.98% on CK+ while requiring significantly fewer parameters than conventional deep FER models. These results confirm that adaptive gated fusion of shallow and deep features can deliver both high recognition accuracy and strong cross-dataset robustness.

1. Introduction

Facial Emotion Recognition (FER) is a fundamental task in affective computing that maps changes in facial appearance to discrete emotional categories [1]. FER supports emotion-aware systems in human–computer interaction [2], cognitive health, and digital health monitoring systems [3], where timely emotion recognition can improve health outcomes and enhance overall intervention workflows [4]. In this context, FER has emerged as a key technology providing natural, universal, and non-intrusive cues for human affect. Compared with other modalities such as Biometric Expression Recognition (BER) and Speech Emotion Recognition (SER), FER offers real-time accessibility and strong applicability in practical environments, making it indispensable for building intelligent and emotion-aware interactive systems [5]. However, despite strong benchmark results, a critical impediment to real-world deployment remains in the form of limited cross-database generalization. Models trained on a source dataset often degrade substantially when evaluated on a different dataset, even when the emotion classes are identical [6]. This degradation is primarily driven by domain shift arising from variations in lighting conditions, emotion intensity, and annotation protocols across datasets. As a result, supervised FER models tend to use dataset-specific statistical features rather than learning emotion-invariant cues, which has motivated recent research on cross-domain and cross-database FER [7,8].

Traditional FER approaches based on hand-crafted descriptors such as LBP, PCA subspaces, and Gabor responses are computationally efficient and can offer partial robustness to local appearance variations. However, their representational capacity is limited, and they often fail to capture higher-level semantics required to handle complex intra-class variation and inter-class similarity. In contrast, deep Convolutional Neural Network (CNN) [9] models learn rich hierarchical representations and achieve high accuracy under train–test conditions; however, they require large labeled data and are prone to overfitting. While recent studies have attempted to improve the robustness and FER systems, several critical limitations remain. López-Echeverry et al. [10] proposed EmotiCloud, a cloud-based FER system for patient monitoring, demonstrating the feasibility of deploying FER in real workflows. Nevertheless, the system relies on a single deep feature representation and was evaluated under matched data conditions, leaving its robustness under cross-database domain shift largely unexplored. Likewise, Cui et al. [11] presented a comprehensive system-level analysis of digital healthcare frameworks that incorporate affective computing components. While their study highlighted the importance of emotion-aware systems for human-centered healthcare applications, the underlying FER components were treated as black-box modules without addressing feature-level robustness or dataset bias. Currently, many existing methods adopt late fusion approaches [12] in which the two streams are trained largely independently and combined only at the classifier. This design limits cross-stream interaction during feature learning.

To address these limitations, we propose Cross-Adaptive Gated Fusion Network (CA-GFNet), a hybrid dual-stream FER framework designed to improve robustness under domain shift. Specifically, CA-GFNet couples a structural stream that extracts low-level gradient cues with a semantic stream that extracts VGG-based deep features. Unlike conventional late-fusion methods, the proposed framework enables early and continuous interaction between the two streams through cross-attention during representation learning. In addition, an adaptive gated fusion module dynamically assigns input-dependent fusion weights, allowing the network to emphasize the more reliable feature source for each sample. This design improves robustness and supports stronger generalization across datasets. In this work, we target cross-database robustness using two controlled FER datasets, Karolinska Directed Emotional Faces (KDEF) [13] and Extended Cohn–Kanade (CK+) [14], under a train-on-KDEF/test-on-CK+ protocol. We further provide comprehensive ablation studies and comparative experiments to quantify the individual contributions of cross-attention coupling and adaptive gated fusion to cross-database generalization.

The following are the key contributions of the proposed study:

• We move beyond conventional late-fusion designs by proposing CA-GFNet, a cross-adaptive based dual-stream FER architecture that enables bidirectional interaction between deep semantic features and shallow structural features during feature learning.
• We establish a cross-dataset evaluation protocol using a train-on-KDEF and test-on-CK+ setting, ensuring no target domain leakage and enabling a fair assessment of cross-database generalization performance.
• We introduce an adaptive gated fusion module that learns input-dependent fusion weights to dynamically combine semantic and structural cues, improving robustness to domain shifts that affect feature modalities differently.
• We conduct extensive experimental evaluations, including ablation studies and cross-dataset comparisons, to quantify the contribution of cross-attention interaction and gated fusion to generalization performance.

The remainder of this paper is organized as follows: Section 2 reviews recent advances in facial emotion recognition; Section 3 presents the proposed CA-GFNet framework; Section 4 reports the experimental results; Section 5 presents the discussion and finally, Section 6 concludes the paper.

2. Literature Review

This section critically reviews existing FER approaches, highlighting their strengths and limitations. We first discuss the evolution of deep learning architectures for FER, followed by recent advances based on attention mechanisms and hybrid models. Finally, we summarize the key research gaps that motivate the proposed CA-GFNet.

2.1. Evolution of Deep Learning Architectures for FER

The field of facial emotion recognition has evolved significantly from early convolutional neural networks to more advanced and hybrid deep learning architectures. Elsheikh et al. [15] introduced the Anti-Aliased Deep Convolution Network (AA-DCN) to address aliasing artifacts caused by downsampling operations in conventional CNNs. By incorporating anti-aliasing filters and enforcing shift invariance, AA-DCN improves spatial feature preservation and achieves strong performance on controlled datasets such as JAFFE. However, the increased training cost and limited scalability restrict its applicability in real-time and unconstrained settings. Huang et al. [16] explored channel-wise feature recalibration using squeeze-and-excitation blocks integrated with residual networks (SE-ResNet). Their approach highlighted the importance of channel dependencies for emotion discrimination and achieved competitive performance on AffectNet and RAF-DB. Nevertheless, the reliance on a single visual modality and a high parameter count limits deployment in resource-constrained environments.

To provide systematic insights into architectural tradeoffs, Qian et al. [17] conducted a comprehensive comparison of ten deep learning architectures on FER2013, including VGG, ResNet, DenseNet, GoogLeNet, MobileNet, EfficientNet, ShuffleNet, and RepVGG. Their study showed that while VGG and ResNet achieve higher accuracy, lightweight architectures suffer notable performance degradation for challenging emotions such as fear and disgust. This analysis clearly exposes the persistent accuracy–efficiency tradeoff in FER systems.

2.2. Advanced Transformer-Based Mechanisms

Vision Transformers (ViTs) have recently gained traction in FER by modeling long-range dependencies and global facial context. Tian et al. [18] proposed a dual-stream global–local fusion architecture that combines CNN-based hybrid local attention with a ViT branch. While this design improves robustness to occlusion and head pose variation, its increased model complexity and reliance on reliable local features limit performance on low-resolution or highly noisy inputs. To address computational constraints, Li et al. [19] introduced a lightweight hybrid ViT using multi-scale attention and patch dropping. Although the model achieved good efficiency and performed well on RAF-DB, its lower accuracy on FER2013 highlights persistent generalization issues. Moreover, aggressive patch dropping risks discarding subtle facial cues that are essential for recognizing fine-grained emotions.

Hybrid CNN–transformer architectures further attempt to exploit complementary strengths. Song and Liu [20] proposed HFE-Net, which integrates convolutional and self-attention mechanisms with multiple specialized modules. Despite improved performance over single-paradigm models, its large number of components increases architectural complexity and susceptibility to background noise. Similarly, Tagmatova et al. [21] presented FERONet, a multimodal transformer combining RGB, optical flow, and depth information. While highly accurate on controlled datasets, its substantial computational and memory requirements hinder lightweight deployment. Other attention-based models incorporate auxiliary information. Nemati et al. [22] integrated facial landmark geometry with Efficient-Net, improving recognition of subtle expressions but introducing dependency on reliable landmark detection. Duongthang et al. [23] proposed FARNet, achieving perfect accuracy on CK+ and OuluCASIA with low parameter count; however, performance dropped sharply on in-the-wild datasets, exposing limited cross-dataset robustness. Similar limitations have been observed in EfficientNet-CBAM models [24], which face GPU memory constraints and moderate real-time performance.

More recent works have explicitly addressed the issue of domain shift. Ma et al. [25] proposed FER-VMamba, leveraging global compact attention and hierarchical feature interaction to handle occlusion and pose variation. Nevertheless, reliance on complex attention and state-space modeling limits the efficiency and cross-dataset scalability of this approach. Shu et al. [26] introduced MDFEF, a multi-domain fusion framework combining semantic, spatial, and frequency branches. While effective under controlled conditions, dependence on multiple deep feature streams without lightweight structural cues constrains its robustness under severe domain shift. Zhu et al. [6] presented FER-DAS, employing active sample selection and domain alignment; however, its iterative adaptation strategy increases training complexity and departs from source-only evaluation settings.

Despite notable progress, several challenges remain unresolved. Cross-dataset generalization is still limited, the accuracy–efficiency tradeoff persists, and shallow structural features such as gradients and textures remain underutilized. Finally, most approaches rely on static training paradigms without adaptive feature fusion tailored to dataset shift. Our proposed CA-GFNet addresses these limitations through a dual-stream design that integrates shallow structural with deep semantic features. By employing adaptive gated fusion, CA-GFNet achieves strong cross-dataset robustness while maintaining computational efficiency.

3. Proposed Method: CA-GFNet

This section presents the architecture and processing pipeline of Cross-Adpative Gated Fusion Network (CA-GFNet), a lightweight dual-stream framework designed for robust facial emotion recognition under cross-dataset domain shift. CA-GFNet integrates shallow structural texture cues with deep semantic representations extracted from a pretrained VGG16 backbone. These complementary features are adaptively fused through a gated mechanism and further refined using a transformer encoder prior to final classification. An overview of the proposed architecture is illustrated in Figure 1.

3.1. Data Preprocessing

Given a raw facial image $I\in {\mathbb{R}}^{H\times W\times C}$, a unified preprocessing pipeline is applied to reduce dataset-specific variability between KDEF and CK+. All images are resized to a fixed spatial resolution and normalized to zero mean and unit variance. During training, lightweight augmentations including horizontal flipping, small rotations, and mild brightness and contrast adjustments are applied to improve robustness to pose and illumination variations. The normalized images are then forwarded to two parallel branches, a deep semantic branch based on VGG16 and a structural branch based on HOG descriptors, to enable complementary feature extraction.

3.1.1. Deep Semantic Features Extraction

Each input image is resized to $224\times 224$ pixels in order to match the input specification of VGG16, then preprocessed using the standard preprocess_input function. As the deep semantic feature extractor, we employ a VGG16 network pretrained on ImageNet with the classification layers removed. Figure 2 shows a graphical representation of the baseline VGG-16 framework. Global average pooling is applied to the final convolutional block to obtain a compact semantic representation:

$${\mathbf{v}}_{\mathrm{raw}}=\mathrm{VGG}16\left(I\right)$$

(1)

where $v_{\mathrm{raw}}\in {\mathbb{R}}^{D_v}$, in which $D_v=512$ denotes the dimensionality of the extracted semantic feature vector. Pretraining on the large-scale ImageNet dataset enables VGG16 to capture high-level facial semantics that are relevant for emotion recognition, providing a strong prior for modeling discriminative expression-related patterns. Representative examples of VGG-based feature preprocessing are shown in Figure 3.

3.1.2. Extraction of Shallow Structural Features

To complement deep semantic representations, the Histogram of Oriented Gradients (HOG) descriptor is employed to capture local structural patterns induced by facial muscle movements. The input image is first converted to grayscale, divided into cells, and represented using normalized histograms of gradient orientations, producing a structural descriptor:

$${\mathbf{h}}_{\mathrm{raw}}=\mathrm{HOG}\left(I_{\mathrm{gray}}\right).$$

(2)

This process yields a feature vector ${\mathbf{h}}_{\mathrm{raw}}\in {\mathbb{R}}^{D_h}$ that represents the local structural texture of the emotional expression. HOG features are inherently robust to illumination variations and provide fine-grained geometric information that is often overlooked by deep networks. Sample structural feature visualizations are shown in Figure 4. Both semantic and structural features are standardized to ensure stable optimization during training.

3.2. Cross-Modal Processing and Adaptive Gated Fusion

Since VGG-based semantic features and HOG-based structural features reside in heterogeneous representational spaces, direct concatenation may lead to suboptimal fusion. To address this issue, each feature stream is projected into a shared latent space of dimension $D_m=512$ using modality-specific fully connected layers with GELU activation, batch normalization, dropout, and layer normalization:

$${\mathbf{x}}_v={\mathcal{P}}_v\left({\mathbf{v}}_{\mathrm{raw}}\right),{\mathbf{x}}_h={\mathcal{P}}_h\left({\mathbf{h}}_{\mathrm{raw}}\right).$$

(3)

An adaptive gated fusion mechanism dynamically balances the contribution of each modality on a per-sample basis. A gating value $g\in [0, 1]$ is computed as follows:

$$g=\sigma \left(W_g\left[\begin{array}{c}{\mathbf{x}}_v\\ {\mathbf{x}}_h\end{array}\right]+{\mathbf{b}}_g\right)$$

(4)

and the fused representation is obtained by

$$\mathbf{z}=g·{\mathbf{x}}_v+(1-g)·{\mathbf{x}}_h.$$

(5)

This formulation enables CA-GFNet to adaptively emphasize semantic or structural cues depending on the input context, improving robustness under dataset shift. Figure 5 shows the gated fusion concept graphically.

Feature Projection to a Common Latent Space

To enable effective fusion, both feature vectors are nonlinearly projected into a shared latent space of the same dimensionality $(D_m=512)$. This projection allows the model to learn a transformed representation of each modality that is more amenable to combination. Each path uses a dedicated subnetwork comprising dense layers, GELU activation (chosen for its smooth nonlinearity), batch normalization (for stable training), and dropout (for regularization). The final LayerNorm produces the projected features $x_v$ and $x_h$:

$$\begin{array}{cc}\hfill {\mathbf{x}}_v^{\left(1\right)}& =\mathrm{GELU}\left(W_v^{\left(1\right)}\mathbf{v}+{\mathbf{b}}_v^{\left(1\right)}\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathbf{x}}_v^{\left(2\right)}& =\mathrm{BatchNorm}\left({\mathbf{x}}_v^{\left(1\right)}\right)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\mathbf{x}}_v^{\left(3\right)}& =\mathrm{Dropout}\left({\mathbf{x}}_v^{\left(2\right)},\mathrm{rate}=0.4\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\mathbf{x}}_v& =\mathrm{LayerNorm}\left(W_v^{\left(2\right)}{\mathbf{x}}_v^{\left(3\right)}+{\mathbf{b}}_v^{\left(2\right)}\right)\hfill \end{array}$$

(9)

where $W_v^{\left(1\right)}\in {\mathbb{R}}^{1024\times D_v}$, $W_v^{\left(2\right)}\in {\mathbb{R}}^{D_m\times 1024}$, and GELU denotes the Gaussian Error Linear Unit activation function. Figure 5 shows this concept in detail.

3.3. Adaptive Gated Fusion Under Domain Shift

When a model is trained on a source domain and evaluated on a target domain, each feature stream experiences a different magnitude of noise due to domain shift. HOG descriptors capture local gradient orientations that are inherently robust to illumination variation and dataset-specific imaging conditions [27], making them relatively stable under domain shift. By contrast, deep VGG16 semantic features encode high-level appearance patterns that may partially overfit to source-domain statistics (i.e., posed studio-lit KDEF images), potentially suffering greater distributional shift when applied to spontaneous CK+ expressions.

Furthermore, not all emotions are equally discernible from both modalities; “anger” may be strongly signaled by deep furrows that are well-captured by HOG gradients, while “disgust” may be better identified by the overall nose and lip configuration encoded by VGG16 semantics. Static late-fusion strategies which assign fixed combination weights to each stream cannot adapt to this asymmetric per-sample degradation. An adaptive gated fusion mechanism addresses this limitation by dynamically emphasizing the less-degraded stream on a per-sample basis [6,28].

Gating Mechanism

Let ${\mathbf{x}}_v\in {\mathbb{R}}^{D_m}$ and ${\mathbf{x}}_h\in {\mathbb{R}}^{D_m}$ denote the projected VGG and HOG feature vectors, respectively, where $D_m=512$. The two projected representations are concatenated and passed through a single sigmoid neuron to produce a per-sample scalar gate:

$$g=\sigma \left({\mathbf{W}}_g\left[\begin{array}{c}{\mathbf{x}}_v\\ {\mathbf{x}}_h\end{array}\right]+b_g\right),g\in [0, 1],$$

(10)

where ${\mathbf{W}}_g\in {\mathbb{R}}^{1\times 2D_m}$, $b_g\in \mathbb{R}$, and $\sigma (·)$ denotes the sigmoid activation. The fused representation is then computed as the convex combination:

$$\mathbf{z}=g·{\mathbf{x}}_v+(1-g)·{\mathbf{x}}_h.$$

(11)

When $g\approx 1$, the model relies predominantly on the deep semantic stream, while when $g\approx 0$ it prioritizes the structural HOG stream. This formulation is a special scalar gate case of the Gated Multimodal Unit [28], adapted here for computational efficiency in a dual-stream setting.

3.4. Transformer Encoder Head

The fused feature vector z is passed through a post-fusion dense layer and then prepared for sequence processing. Although the fused representation z is a single vector, the transformer encoder block is employed as a post-fusion refinement module in which the primary computational benefit comes from the FFN sublayer with residual connections and layer normalization. This sublayer performs deep nonlinear feature refinement. The MHSA component acts as a learned linear self-projection in this single-token setting, consistent with its use in CLS-token classification architectures.

$${\mathbf{z}}^{\prime }=\mathrm{Dropout}\left(\mathrm{GELU}\left(W_p\mathbf{z}+{\mathbf{b}}_p\right),\mathrm{rate}=0.3\right)$$

(12)

To leverage contextual self-attention, ${\mathbf{z}}^{\prime }$ is treated as a single-token sequence and fed into a transformer encoder layer. The encoder employs Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFNs) with residual connections and layer normalization:

$$\begin{array}{cc}\hfill {\mathbf{z}}_0& ={\mathbf{z}}^{\prime }+\mathrm{PE},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{z}}_l^{\prime }& =\mathrm{MHSA}\left(\mathrm{LayerNorm}\left({\mathbf{z}}_{l-1}\right)\right)+{\mathbf{z}}_{l-1},\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathbf{z}}_l& =\mathrm{FFN}\left(\mathrm{LayerNorm}\left({\mathbf{z}}_l^{\prime }\right)\right)+{\mathbf{z}}_l^{\prime }.\hfill \end{array}$$

(15)

where $\mathrm{PE}$ is the positional encoding (trivial here for a single token) and l is the layer index. The output ${\mathbf{z}}_L$ of the transformer encoder is is passed to the final classification layer, which uses a softmax activation to predict the emotion class $\widehat{y}$ from C possible classes:

$$\begin{array}{cc}\hfill \mathbf{p}& =\mathrm{softmax}\left(W_c{\mathbf{z}}_L+{\mathbf{b}}_c\right),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill \widehat{y}& =\arg \max \left(\mathbf{p}\right),\hfill \end{array}$$

(17)

where $W_c\in {\mathbb{R}}^{C\times D_m}$. For better understandings of the findings, Figure 6 shows an architectural comparison between deep-only, late-fusion, domain adaptation, and the proposed framework.

4. Experimentation and Results

This section describes the experimental protocol, datasets, evaluation metrics, and quantitative results used to validate the proposed CA-GFNet framework. The overall experimental workflow is illustrated in Figure 7, highlighting the sequential stages from dataset selection and preprocessing to supervised training on KDEF and zero-shot cross-dataset evaluation on CK+.

4.1. Dataset

Two widely adopted benchmark datasets were employed to evaluate both in-domain performance and cross-dataset generalization: the Karolinska Directed Emotional Faces (KDEF) dataset [13], and the Extended Cohn–Kanade (CK+) dataset [14].

The KDEF dataset consists of facial images from 70 actors (35 male and 35 female), each displaying seven basic emotions: anger, disgust, fear, happiness, sadness, surprise, and neutral. Each expression is captured from five camera viewpoints. In this study, KDEF was used exclusively as the source dataset for supervised training and in-domain evaluation.

The CK+ dataset contains 396 image sequences from 123 subjects, captured under controlled laboratory conditions; each sequence begins with a neutral expression and ends at the peak emotional expression. The same seven emotion categories as KDEF were considered. CK+ was used solely for zero-shot cross-dataset evaluation where models trained on KDEF were directly tested on CK+ without any parameter updates, allowing for strict assessment of generalization under domain shift.

Sample images and dataset statistics are illustrated in Figure 8 and summarized in

Table 1. Dataset statistics for KDEF and CK+.

Emotion Class	KDEF		CK+
	Samples	Total Images
Anger	700	4900	45
Disgust	700		59
Fear	700		25
Happiness	700		69
Neutral	700		87
Sadness	700		28
Surprise	700		83

.

Table 2. Per-class precision, recall, and F1-score for the fused HOG+VGG model on the KDEF and CK+ datasets.

Emotion Class	KDEF (Training)			CK+ (Cross-Dataset Test)
	Precision (%)	Recall (%)	F1 (%)	Precision (%)	Recall (%)	F1 (%)
Anger	99.00	98.71	98.86	97.78	97.78	97.78
Disgust	99.28	99.00	99.14	100.00	98.31	99.15
Fear	98.57	98.43	98.50	92.00	92.00	92.00
Happiness	99.86	99.71	99.79	100.00	100.00	100.00
Neutral	98.57	98.71	98.64	96.63	98.85	97.73
Sadness	98.15	98.71	98.43	96.30	92.86	94.55
Surprise	99.43	99.57	99.50	100.00	100.00	100.00
Macro Average	98.98	98.98	98.98	97.53	97.11	97.32

shows the per-class perfromance of the propsoed study.

4.2. Experimental Setup

All models were first trained on the KDEF dataset using a supervised learning setup. Training was performed for up to 40 epochs with a batch size of 32. The Adam optimizer was used with an initial learning rate of $1\times {10}^{-3}$, ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ=1\times {10}^{-7}$. To improve generalization and reduce overconfidence, label smoothing with factor 0.1 was applied to the sparse categorical cross-entropy loss. Class weights were computed from the training data to mitigate minor class imbalance. A held-out validation split was used for model selection, and the checkpoint with the highest validation accuracy was retained.

To improve robustness to pose and illumination variation, augmentation was applied to the KDEF training subset only, comprising horizontal flipping ($p=0.5$), random rotation ($\pm 10°$), and brightness and contrast adjustments (factor range $[0.8, 1.2]$). No augmentation was applied during validation or cross-dataset evaluation. The complete preprocessing and hyperparameter configuration is summarized in

Table 3. Complete implementation and hyperparameter summary for reproducibility.

Parameter	Value
Preprocessing
Image resize	$224\times 224$
Face detection	None (controlled laboratory datasets)
Landmark alignment	None
Cropping	None (full image used)
HOG configuration
Orientation bins	9
Pixels per cell	$8\times 8$
Cells per block	$2\times 2$
Block normalisation	L2-Hys
image	Grayscale, $224\times 224$ px
Feature dimension	26,244
VGG16
Pre-training	ImageNet
Top layers	Removed
Pooling	Global average pooling
Output dimension	512
Weights frozen	Yes
Training KDEF
Optimiser	Adam
${\beta }_1$, ${\beta }_2$	0.9, 0.999
$ϵ$	$1\times {10}^{-7}$ (framework default)
Learning rate	$1\times {10}^{-3}$
Batch size	32
Max epochs	40
Data split	Stratified 80:20, random seed = 42
Loss function	Sparse categorical cross-entropy
Early stopping	Patience = 10 (monitored: val. accuracy)
LR reduction	Factor = 0.5, patience = 5 epochs
Model selection	Best validation accuracy checkpoint
Augmentation (KDEF training only)
Horizontal flip	$p=0.5$
Rotation range	$\pm 10°$
Brightness factor	$[0.8, 1.2]$
Contrast factor	$[0.8, 1.2]$
Applied to val/test	No
Cross-dataset evaluation—CK+
Protocol	Zero-shot (no parameter updates)
Preprocessing	Identical pipeline to KDEF
Normalisation stats	From KDEF training subset only

.

To evaluate cross-dataset generalization, zero-shot testing was performed on the CK+ dataset. Models trained on KDEF were directly applied to CK+ without any parameter updates or domain adaptation. CK+ images were processed using the same preprocessing pipeline as KDEF, including face cropping, resizing, and normalization. This protocol isolates the effect of dataset shift by evaluating the model exclusively on knowledge learned from the source domain. For both datasets, performance was reported using accuracy, precision, recall, and F1-score.

4.3. Results and Discussion

The proposed CA-GFNet was first evaluated on the KDEF dataset to assess the effectiveness of adaptive feature fusion. Compared to single-modality baselines, the fused HOG+VGG model achieved superior and more stable performance. Specifically, CA-GFNet achieved an accuracy of 97.72% with macro-precision of 97.83%, recall of 97.72%, and F1-score of 97.16%. These results indicate that adaptive gated fusion improves class balance and robustness beyond what is achievable using individual feature streams. The training and validation loss and accuracy curves for KDEF are shown in Figure 9, demonstrating smooth convergence across epochs. Figure 10 shows the confusion matrices of both the datasets.

Zero-shot evaluation on the CK+ dataset further highlights the generalization capability of CA-GFNet. Without any retraining, the fusion model consistently outperforms single-modality baselines and exhibits stable inference behavior. Compared to the zero-shot baseline, CA-GFNet achieves an improvement of approximately 1% in classification accuracy, confirming that adaptive fusion effectively exploits complementary geometric and semantic cues under domain shift.

Figure 11 shows qualitative prediction results of the proposed model on the CK+ test dataset. Each sample shows the input facial expression image with ground truth emotion label and model prediction with confidence percentage. Correctly classified samples demonstrate the model’s ability to generalize across datasets. Figure 12 shows the visualized results, demonstrating that the proposed network effectively localizes the face region, especially in complex data. These results validate the efficacy and reliability of the proposed CA-GFNet for facial emotion classification, particularly when handling complex data.

4.4. Ablation Study

An ablation study was conducted to quantify the contribution of individual feature modalities and assess the effectiveness of their fusion under a zero-shot cross-dataset protocol (

Table 4. Ablation study comparing HOG-only, VGG-only, and fused HOG+VGG configurations on the KDEF and CK+ datasets. ✓ indicates the use of the feature.

Dataset	HOG	VGG	Accuracy	Precision	Recall	F1-Score
KDEF	✓		0.9772	0.9734	0.9772	0.9698
KDEF		✓	0.9592	0.9490	0.9592	0.9504
KDEF	✓	✓	0.9861	0.9865	0.9861	0.9764
CK+	✓		0.9698	0.9451	0.9459	0.9454
CK+		✓	0.9698	0.9555	0.9403	0.9446
CK+	✓	✓	0.9797	0.9472	0.9491	0.9324

). All variants were trained on KDEF and evaluated on both KDEF and CK+. Three configurations were examined: a HOG-only model, a VGG-only model, and the proposed CA-GFNet with adaptive gated fusion.

On KDEF, the HOG-only model achieves 97.72% accuracy, outperforming the VGG-only model (95.92%). This indicates that structural features remain highly discriminative in controlled settings. CA-GFNet delivers the best performance, reaching 98.61% accuracy with consistently higher precision, recall, and F1-score, demonstrating the benefit of integrating complementary shallow and deep representations. Under zero-shot evaluation on CK+, both single-stream baselines generalize reasonably (96.98% accuracy), while CA-GFNet further improves performance to 97.97%, achieving the highest precision and recall. These results confirm that adaptive fusion effectively exploits complementary information and enhances robustness under dataset shift, yielding superior performance in both in-distribution and cross-dataset scenarios.

Model Variants

To isolate the contribution of each feature stream and their interaction, the following model variants were evaluated:

• HOG-only: Uses only Histogram of Oriented Gradients (HOG) descriptors as input to the classifier. The VGG branch and fusion module are disabled, yielding a purely structural-feature baseline.
• VGG-only: Uses only deep features extracted from the final convolutional block of a VGG-16 network. The HOG branch is removed, resulting in a purely deep-semantic baseline.
• HOG–VGG Gated Fusion (Proposed): Activates both feature streams. HOG and VGG features are projected into a shared latent space and adaptively combined using the gated fusion module to produce a unified representation for classification.

All variants share the same classifier head, training protocol, and hyperparameters; only the feature extraction and fusion components differ.

4.5. Comparative Analysis of Accuracy

This section compares the classification accuracy CA-GFNet to recent state-of-the-art FER methods on the KDEF and CK+ datasets.

Table 5. Accuracy comparison of recent FER methods on the KDEF and CK+ datasets.

No.	Method	Year	KDEF (%)	CK+ (%)
1	AA-DCN [15]	2024	96.00	99.26
2	HLA-ViT [18]	2024	96.50	98.50
3	ResNet50+CBAM+TCN [29]	2024	97.08	95.85
4	FARNet [23]	2025	97.50	100.00
5	EfficientNet-XGBoost [24]	2024	98.00	100.00
6	VGGNet16-CBAM [30]	2024	94.00	98.66
7	Multi-Scale Simplicial Transformer [31]	2025	96.00	98.54
8	Hybrid Vision Transformer [19]	2024	95.50	98.00
9	CA-MCNN [32]	2025	96.80	96.50
10	GLA-FERNet [33]	2025	95.00	97.00
11	VGG-19 + ResNet-152 [34]	2025	93.50	96.00
12	YOLOv8-based FER [35]	2025	94.00	95.50
	Proposed CA-GFNet	2026	99.30	98.98

reports the most recent CNN- and transformer-based approaches on KDEF, typically achieving accuracy results between 93.5% and 98.0% [15,18,23,24,29]. In contrast, our proposed CA-GFNet attains 99.30% accuracy on KDEF, surpassing all competing methods and improving upon the best prior result by a clear margin despite relying on lightweight structural features. Where many deep models report near-saturated accuracy results on CK+ under controlled conditions, CA-GFNet achieves 98.98%, remaining within approximately 1% of the best-reported results while using substantially fewer parameters and lower computational complexity. Overall, this comparison demonstrates that CA-GFNet establishes a new state of the art on KDEF and remains highly competitive on CK+, confirming that carefully designed gradient-based structural representations can rival or exceed much heavier deep architectures.

4.6. Comparative Analysis of Precision, Recall, and F1-Score

This section compares CA-GFNet with recent state-of-the-art FER methods in terms of precision, recall, and F1-score on the KDEF and CK+ datasets. On KDEF, most CNN- and transformer-based approaches achieve balanced F1-scores in the range of 0.92–0.96 [15,23,24,29]. In contrast, CA-GFNet attains the highest overall balance, achieving a precision of 0.97, recall of 0.98, and F1-score of 0.97, indicating superior class-wise consistency despite its lightweight design. On CK+, several deep models report near-saturated metrics under controlled conditions; CA-GFNet achieves 0.95 precision, recall, and F1-score, remaining competitive while requiring substantially fewer parameters and lower computational cost than transformer- or NAS-based methods [31,33].

Table 6. Comparison of recent FER methods on the KDEF and CK+ datasets in terms of precision, recall, and F1-score.

No.	Method	Year	KDEF			CK+
			Prec.	Rec.	F1	Prec.	Rec.	F1
1	AA-DCN [15]	2024	0.95	0.94	0.95	0.99	0.98	0.99
2	ResNet50+CBAM+TCN [29]	2024	0.94	0.95	0.94	0.94	0.93	0.94
3	VGGNet16-CBAM [30]	2024	0.92	0.91	0.92	0.97	0.96	0.97
4	HLA-ViT [18]	2024	0.94	0.93	0.94	0.97	0.96	0.97
5	FARNet [23]	2025	0.96	0.95	0.96	0.99	0.99	0.99
6	Multi-Scale Simplicial Transformer [31]	2025	0.93	0.93	0.94	0.97	0.97	0.97
7	EfficientNet-XGBoost [24]	2024	0.95	0.96	0.95	0.99	0.99	0.99
8	Hybrid Vision Transformer [19]	2024	0.93	0.92	0.93	0.96	0.96	0.96
9	CA-MCNN [32]	2025	0.94	0.94	0.94	0.95	0.94	0.95
	Proposed CA-GFNet	2026	0.99	0.99	0.98	0.95	0.95	0.93

demonstrates that CA-GFNet offers a favorable tradeoff between robustness, efficiency, and generalization, making it suitable for practical FER deployment.

5. Discussion

The results summarized in

Table 7. Comparison of model complexity, inference efficiency, and KDEF performance.

Method	Param	FLOPs	Inference	KDEF (%)
LiExNet [36]	42 K (0.04 M)	86 MFLOPs	>530 FPS (Jetson TX2)	88.2
RepVGG + MobileViT + CapsNet [37]	0.95 M	294.6 MFLOPs	Real-time	97.53
Lightweight DCNN (GZS-ConvNet) [38]	Reported	Reduced FLOPs reported	N/A	99.25
FARNet [23]	∼3.05 M	Not reported	N/A	97.5
VGG16 [39]	138.34 M	30.94	Slow	96.0
ResNet50 [40]	25.6 M	4.1 GFLOPs	∼30 ms GPU	∼96–97
EfficientNet-B0 [41]	5.3 M	390 MFLOPs	∼8 ms GPU	96.05%
Our Proposed	5.7 M	4.0 M	11–14 GPU	99.30%

and

Table 8. Cross-domain facial expression recognition studies (source, targets, and reported performance).

No.	Research	Method	Source Data	Target Data	In-Domain Accuracy	Cross-Domain Accuracy	Drop
1	Elsheikh et al. [15]	AA-DCN (Anti-Alias Deep CNN)	CK+	JAFFE/RAF-DB	99.26%	98.00%	1.26%
2	Aly et al. [42]	ResNet50 + 3D CNN (DCNN)	KDEF	CK+	98.53%	97.29%	1.24%
3	Alzahrani et al. [43]	DCD-DAN	FERPlus	RAF-DB AffectNet ExpW SFEW 2.0 JAFFE	92.37%	RAF-DB: 93% AffectNet: 73% ExpW: 78% SFEW 2.0: 64% JAFFE: 62%	−0.81% 18% 14% 27% 29%
4	Guo et al. [44]	USTST	RAF-DB	CK+	87.79%	79.60%	8.19%
5	Guo et al. [45]	POST	RAF-DB	CK+	88.00%	87.00%	1.00%
6	Debnath et al. [46]	Four-layer ConvNet	FER2013	CK+ JAFFE	91.00%	CK+: 98.13% JAFFE: 92.05%	−7.13% −1.05%
7	Ghaedi et al. [47]	GAT-ADA	RAF-DB	FER2013	82.00%	98.00%	−16.00%
8	Zavarez et al. [48]	VGG-Face fine-tuned deep CNN	Multi-source	Cross-database protocol	-	CK+: 88.58% MMI: 67.03% RaFD: 85.97% KDEF: 72.55% JAFFE: 48.67%	-
9	Our Proposed	CA-GFNet	KDEF	CK+	99.00%	98.00%	1.00%

confirm the main contributions of this study. Table 7 shows that many existing FER models achieve strong accuracy but often require high computational cost, large parameter counts, or slower inference, which limits their practicality. In contrast, our proposed CA-GFNet is designed to provide a better balance between efficiency and recognition performance. Table 8 further shows that the proposed method maintains strong cross-database performance under the strict train-on-KDEF/test-on-CK+ protocol, with only a small performance drop when compared to in-domain accuracy. This is important because KDEF and CK+ differ in subjects, image characteristics, and expression patterns, making cross-dataset testing more challenging than matched evaluation. Overall, these findings indicate that combining structural gradient-based cues with deep semantic features through cross-attention and adaptive gated fusion can improve robustness to domain shift while keeping the framework lightweight and practical. Therefore, the proposed CA-GFNet provides an effective and efficient solution for cross-database facial emotion recognition.

6. Conclusions

This study presents a lightweight yet competitive FER framework centered on discriminative HOG features and a compact classifier, with additional analysis of deep VGG features and their adaptive gated fusion within CA-GFNet. Under a strict train-on-KDEF/test-on-CK+ protocol, the proposed model achieves state-of-the-art performance on KDEF (99.30% accuracy) and highly competitive results on CK+ (98.98%) while also attaining the best balance of precision, recall, and F1-score on the source dataset. These results demonstrate that when combined with principled fusion mechanisms, carefully designed shallow structural descriptors can rival or surpass substantially heavier CNN- and transformer-based models while offering superior efficiency, interpretability, and deployability. Ablation studies further confirm that adaptive gated fusion is critical, as naïve feature combination does not consistently improve cross-dataset robustness. Although the current evaluation was limited to controlled laboratory datasets and basic emotion categories, our proposed approach establishes a strong and efficient baseline for FER and highlights the continued relevance of shallow features. Future work will extend this framework to in-the-wild datasets and explore stronger cross-attention and domain alignment strategies to further improve robustness under real-world conditions.