A Hierarchical Attention Synergetic Network for Facial Expression Recognition in Service Robots

Abstract

Facial expression recognition (FER) is crucial for endowing service robots with emotional perception capabilities. Achieving high-performance facial expression recognition hinges on effectively balancing the capture of subtle local textures with the understanding of overall facial configurations. However, coordinating local feature variations with global semantic dependencies in unconstrained environments while maintaining semantic alignment remains a challenge. To address this issue, we propose FER-SDAM, a network architecture based on hierarchical attention collaboration. Through a dual-attention hierarchical collaboration mechanism, this architecture introduces an Attention Consistency Loss (ACL) to explicitly align shallow structural awareness with deep global dependencies. It simultaneously captures structural sensitivity and cross-regional correlations, facilitating the effective fusion of local structural information with global semantics, thereby balancing accuracy, robustness, and computational efficiency. We conducted extensive experiments on AffectNet, RAF-DB, and their subsets containing occlusion and pose variations, achieving accuracy rates of 68.12%, 66.68%, and 88.87% on the AffectNet-7, AffectNet-8, and RAF-DB datasets, respectively. The experimental results demonstrate that FER-SDAM achieves a critical balance between accuracy and efficiency, delivering highly competitive recognition performance while maintaining low computational overhead, making it an ideal solution for real-time deployment in service robots.

1. Introduction

With the continuous advancement of artificial intelligence technology, emotional service robots are gradually transitioning from controlled laboratory environments to complex real-world applications such as educational companionship and medical rehabilitation [1,2]. As the core perception module for human–machine emotional interaction, Facial Expression Recognition (FER) directly determines the naturalness of robotic behavioral decisions and the quality of user experience [3]. However, real-world deployment environments are far from ideal: dynamic viewpoint changes, non-uniform lighting distribution, and frequent occlusions pose severe challenges to algorithmic robustness. Simultaneously, edge-side deployment for service robots must meet stringent real-time requirements (inference speed $>25$ FPS) and low-latency constraints (response time $<100$ ms).

To address these challenges, existing research primarily employs two technical approaches. First, local feature extraction methods based on convolutional neural networks (CNNs) can effectively capture subtle facial textures; the FER-HA model proposed by Nemati et al. [4] dynamically weights emotion-related information through an adaptive attention mechanism. achieving an accuracy improvement of 4.2% compared to the baseline model. However, their limited receptive fields make it difficult to model global semantic structures [5]. Second, while Transformer-based global modeling paradigms can capture long-range dependencies, their high computational cost makes them unsuitable for meeting the real-time interaction requirements of robotic platforms.

To balance these trade-offs, recent studies have explored hybrid architectures that combine Convolutional Neural Networks (CNNs) with Transformers. The HLA-ViT model proposed by Tian et al. [6] adopts a dual-stream architecture that fuses local and global contextual features at the decision level, thereby improving recognition performance and robustness. It achieved accuracy rates of 65.07% and 90.45% on the AffectNet and RAF-DB datasets, respectively. Although such methods have improved performance to some extent, most of them employ loosely coupled strategies (such as feature concatenation or late-stage fusion) and do not explicitly model the synergistic interaction between local and global representations.

A critical yet underexplored issue in these hybrid frameworks is the inconsistency between multi-level attention distributions. In practical scenarios, shallow layers tend to focus on local textures that are sensitive to noise or occlusion, whereas deep layers emphasize global semantic structures. When these attention patterns are not aligned, the model may suffer from unstable predictions. This phenomenon, referred to in this work as attention divergence, can be formally understood as the spatial inconsistency between shallow and deep attention maps, which leads to conflicting optimization objectives during training. To address this issue, we introduce the concept of attention synergy, which aims to enforce spatial consistency and complementary behavior between multi-level attention representations. Specifically, attention synergy is defined as the degree of alignment between shallow structural attention and deep semantic attention. By promoting such alignment, the model can achieve more stable and discriminative feature representations under challenging conditions.

Based on the YOLOv8 network model, this paper designs an architecture with hierarchical collaborative attention modeling and proposes a facial expression recognition network based on a dual-attention mechanism, named FER-SDAM (Facial Expression Recognition Network based on the Synergy Dual Attention Mechanism), which achieves efficient real-time inference. The proposed architecture enhances the perception of local structures in emotion-sensitive regions, such as the eyes and corners of the mouth, through a shallow Multidimensional Collaborative Attention (MCA) [7] module, while modeling global semantic dependencies via a deep Transformer Block Enhancement Module (TBEM) [8]. To explicitly bridge the gap between these heterogeneous representations, we introduce an Attention Consistency Loss (ACL), which constrains the spatial differences in attention maps across different levels, thereby promoting consistent attention patterns.

Extensive experiments are conducted on AffectNet, RAF-DB, and their variants with occlusion and pose perturbations. The results demonstrate that FER-SDAM achieves competitive recognition accuracy with low computational overhead, striking a good balance between accuracy and efficiency. In particular, the proposed method exhibits strong robustness, making it suitable for real-time deployment in service robots.

The main contributions of this paper are summarized as follows:

(1) We propose a hierarchical attention-cooperative FER framework designed specifically for real-time robotics applications, which effectively balances recognition performance and computational efficiency.

(2) We introduce an attention coordination mechanism with explicit consistency constraints by designing an Attention Consistency Loss (ACL). By spatially aligning attention maps of shallow local features with those of deep global semantics, we mitigate attention divergence and enhance robustness under occlusion and pose variations.

(3) Extensive experiments demonstrate that, compared to existing methods, our proposed approach achieves a superior trade-off between accuracy and efficiency and exhibits strong practical value in real-world human–robot interaction scenarios.

2. Related Work

2.1. Hybrid Architectures in FER

With the advancement of deep learning, traditional Convolutional Neural Networks (CNNs) such as ResNet and VGG have been widely applied in Facial Expression Recognition (FER), demonstrating strong local feature extraction capabilities [9]. However, limited by the local receptive field of convolutional operations, CNNs struggle to capture the long-range dependencies between facial key points. To address this limitation, Xiong et al. [10] further refined the ResNet-18 architecture by designing the Res18-LG module. This module achieves deep coupling between local feature recognition and global information inference by embedding multiple attention mechanisms within residual modules and integrating Multi-Header Self-Attention (MHSA). In recent years, Transformer, with its Self-Attention mechanism, has shown excellent performance in global context modeling [11] and has gradually been introduced into the FER field. For instance, methods like FER-former [12] and VTFF [13] attempt to construct hybrid architectures by cascading CNNs and Transformers to balance local features and global semantics.

Nevertheless, most existing hybrid architectures adopt strategies of “serial extraction” or “back-end fusion”, which simply concatenate the features of CNNs and Transformers at the end of the deep network. This loosely coupled combination method often overlooks the semantic consistency between deep global semantics and shallow local features. As a result, when the model faces occlusion or pose changes, the attention distributions at different levels may conflict, making it impossible to form stable recognition decisions.

2.2. Joint Global–Local Feature Modeling

The discriminative information of facial expressions is typically distributed in the coordinated changes between local regions (e.g., the corners of the eyes and mouth) and the overall facial configuration. To enhance the robustness of features, researchers have proposed various global–local joint modeling strategies [14]. For instance, MA-Net [15] and LSGTNet [16] use multi-branch structures to extract and fuse local block features and global face features respectively; some studies utilize attention mechanisms (e.g., CBAM [17], SE-Net [18]) to dynamically weight key regions [19].

Although these methods have improved the recognition accuracy to some extent, most of them treat local and global features as independent optimization targets [20,21], lacking an explicit synergy mechanism to constrain the consistency of their spatial distributions. This independent modeling approach is prone to causing the network to ‘neglect one thing while focusing on another’—that is, local attention focuses on detailed noise, while global attention overlooks micro-expression changes.

Unlike previous RAN [22] approaches that relied on coarse-grained cropping, the core advantage of FER-SDAM lies in ‘cross-layer attention collaboration.’ Compared to Poster++, which employs a computationally intensive dual-stream Transformer architecture, we enhance shallow-layer perception through the MCA module and explicitly align deep- and shallow-layer semantics using the ACL loss. This significantly improves the model’s robustness in occlusion scenarios without substantially increasing the computational burden.

2.3. YOLO-Based Real-Time FER

Emotional service robots require perception systems capable of low-latency inference on embedded platforms with limited computational resources. YOLO (You Only Look Once) has gradually become the mainstream approach due to its excellent balance between speed, accuracy, ease of use, and deployment friendliness [23]. In recent years, researchers have sought to improve the YOLO architecture to suit the tasks of emotional service robots (FERs). For example, PSA-YOLO [24] and ALF-YOLO [25] achieved accuracy improvements of over 2% by introducing lightweight attention modules. Meanwhile, Ren et al. [26] combined Swin Transformer V2 with YOLOv8 for infant facial expression recognition, significantly improving recognition accuracy.

However, the design of FER systems for affective robots is a typical manifestation of the ‘No Free Lunch’ (NFL) principle [27]: blindly increasing model complexity to pursue higher accuracy inevitably leads to increased inference latency, which is unacceptable in real-time human–robot interaction. While existing methods often sacrifice the perception of subtle features for the sake of speed, this research seeks a ‘cost-effective’ balance. By constructing a hierarchical attention cooperation paradigm based on YOLOv8, we optimize the allocation of computational resources. Specifically, we employ a Multi-dimensional Collaborative Attention (MCA) module to selectively enhance shallow-level structural perception, while utilizing a Transformer-based Block Enhancement Module (TBEM) to strengthen deep-level semantics. This strategy enables the network to break the bottleneck between performance and efficiency without significantly increasing the computational burden, achieving robust emotional perception.

In summary, existing works are frequently constrained by the NFL dilemma, where improvements in global modeling capabilities often come at the cost of neglecting local details or imposing excessive computational loads. There remains a significant gap in achieving an explicit alignment between shallow and deep features within a unified and efficient framework. The FER-SDAM proposed in this paper aims to fill this gap. By leveraging a synergetic dual-attention mechanism, we achieve a superior balance between model interpretability and robustness in complex interaction scenarios, effectively providing “more for less” while strictly adhering to the real-time constraints of robot terminals.

3. Proposed Method

3.1. Overall Architecture

To enable emotional service robots to reliably detect facial expressions and respond in real time during real-world interactions, FER-SDAM employs a hierarchical attention-based collaborative modeling architecture built on YOLOv8. As shown in Figure 1, this network architecture takes face regions as the input and divides the process of classifying facial expressions within these regions into three stages: shallow feature extraction, hierarchical attention modeling, and attention coordination and prediction, with the final expression classification label and confidence score as the output. This design ensures inference speed while achieving a step-by-step progression and efficient coordination from shallow texture perception to deep global semantic modeling.

During the feature extraction stage (Figure 1a), the system uses a lightweight, fully convolutional backbone (based on YOLOv8) to extract hierarchical feature maps. Multi-scale feature maps ($P_3,P_4,P_5$) are progressively extracted through hierarchical convolutions and the Concatenate-to-Fuse (C2f) architecture.

In the hierarchical attention modeling stage (Figure 1b), the MCA module is used to selectively enhance emotion-sensitive local regions (e.g., eyes and mouth). These features are then aggregated via a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN) to perform cross-scale fusion. Subsequently, these features are fed into the TBEM to capture long-range dependencies in cross-regional and global contextual semantics.

In the attention synergy and prediction stage (Figure 1c), an Attention Synergy (AS) mechanism is designed. This mechanism structurally regularizes the spatial attention responses of local region attention and global semantic modeling via an Attention Consistency Loss (ACL), establishing a consistent and complementary collaborative relationship and forming attention patterns across different semantic levels. The final prediction head can simultaneously output facial regions and the corresponding probability distributions of facial expressions.

3.2. Hierarchical Attention Modeling

3.2.1. Local Area Enhancement

The discriminative information of facial expressions is often concentrated in specific local areas and exhibits a non-uniform distribution along different spatial and channel dimensions. A single-dimensional attention mechanism is unable to simultaneously capture these fine-grained structural features. Therefore, in the local enhancement stage of this paper, the Multidimensional Collaborative Attention (MCA) is introduced to jointly model the feature responses of the input feature map $F\in R^{C\times H\times W}$ from the channel, horizontal, and vertical dimensions [7]. Its structure is shown in Figure 2.

Specifically, in the spatial dimension, the MCA performs dimensional rearrangement of the feature map along the height axis or width axis, mapping the spatial dependencies in the width and height directions respectively into learnable one-dimensional attention sequences. Subsequently, through the squeeze–excitation operation, by combining global statistical information with convolutional excitation, the corresponding spatial attention weights are generated and fused element-wise with the original features, resulting in the width-enhanced feature $F_W^{″}$ and the height-enhanced feature $F_H^{″}$. In the channel dimension, the MCA models the inter-channel correlation through global pooling and channel excitation, generating the channel-enhanced feature $F_C^{″}$. The outputs of the three attention paths are fused with weights to obtain the final output feature of the MCA.

$$F^{″}={\displaystyle \frac{1}{3}}\otimes \left(F_W^{″}\oplus F_H^{″}\oplus F_C^{″}\right)$$

(1)

Here, ⨂ denotes element-wise multiplication, and ⨁ denotes element-wise addition.

By collaboratively modeling local structural dependencies across multiple dimensions, MCA can effectively enhance the model’s response intensity to key facial expression regions such as the eyes and the corners of the mouth without significantly increasing computational overhead, thereby providing a stable and discriminative local feature foundation for subsequent global semantic modeling.

3.2.2. Global Semantic Collaboration

Although local structure enhancement can highlight key facial expression regions, it is difficult to model the semantic dependencies across regions, such as the coordinated changes between the eyebrows, eyes, and mouth. To address this issue, this paper introduces the Transformer Block Enhancement Module (TBEM) for high-level semantic features to model global dependencies. As shown in Figure 3, TBEM adopts a hierarchical enhancement strategy that first focuses on channels and then on space, and it consists of a Channel Enhancement (CE) block and a Spatial Enhancement (SE) part. The overall calculation process can be summarized as follows:

$$\begin{array}{cc}\hfill A{M}_C& =CE\left(F\right)+F\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill A{M}_S& =SE\left(A{M}_C\right)+A{M}_C\hfill \end{array}$$

(3)

Here, F represents the input feature map, ${AM}_C$ and ${AM}_S$ represent the feature outputs after the Channel Enhancement (CE) block and the Spatial Enhancement (SE) respectively, and ${AM}_S$ serves as the final output of the TBEM.

During the channel enhancement phase, the TBEM models high-level features along the channel dimension and captures global dependencies among different semantic channels through a self-attention mechanism to achieve adaptive reweighting of semantic responses. Subsequently, in the spatial enhancement phase, the features undergo further spatial serialization modeling, explicitly depicting long-range associations among different facial regions to compensate for the deficiencies of convolutional features in global spatial modeling. Joint modeling of semantic consistency and spatial synergy is realized within the high-level semantic space.

3.3. Attention Synergy Mechanism

In hierarchical attention-enhanced architectures, due to the inherent differences between the local feature extractor (MCA) and the global dependency modeler (TBEM) in terms of receptive fields and semantic hierarchy, the two often generate independent attention distributions. This can easily lead to attention fragmentation, meaning the network is unable to form a unified decision-making logic.

To mitigate the negative effects of this discrepancy, we propose the Attention Synergy (AS) mechanism, as shown in Figure 4. Unlike traditional hybrid architectures that rely on loosely coupled feature concatenation or implicit fusion strategies, Attention Synergy refers to a mechanism that, through structured regularization, enables feature modules with different receptive fields and semantic abstraction levels within the network to converge their attention focus on the regions critical for facial expression recognition.

3.3.1. CAM Extraction for Attention

To quantify the attention responses of the MCA and TBEM modules, we extract Class Activation Maps (CAMs) from their output feature maps. Let the outputs of MCA and TBEM be denoted as $F_{\mathrm{MCA}}\in {\mathbb{R}}^{B\times C\times H\times W}$ and $F_{\mathrm{TBEM}}\in {\mathbb{R}}^{B\times C\times H\times W}$, respectively.

The corresponding spatial attention maps are obtained via channel-wise average pooling:

$$A_{\mathrm{MCA}}={\mathrm{AvgPool}}_C\left(F_{\mathrm{MCA}}\right),$$

(4)

$$A_{\mathrm{TBEM}}={\mathrm{AvgPool}}_C\left(F_{\mathrm{TBEM}}\right),$$

(5)

Thus, the attention distributions are given by

$$A_{\mathrm{MCA}},A_{\mathrm{TBEM}}\in {\mathbb{R}}^{B\times 1\times H\times W}.$$

(6)

3.3.2. Attention Divergence

In layered network architectures, attention responses from different layers are typically optimized independently of the task objective, and attention maps may exhibit inconsistent spatial focus. We define this phenomenon as attention divergence, which characterizes the degree of discrepancy between attention distributions across multiple layers. Mathematically, this is expressed as

$${\mathcal{D}}_{att}={\parallel A_{\mathrm{MCA}}-A_{\mathrm{TBEM}}\parallel }_1.$$

(7)

where ${\parallel ·\parallel }_1$ denotes the L1 norm.

The smaller ${\mathcal{D}}_{att}$ is, the smaller the difference between the attention distributions of MCA and TBEM, and the higher the degree of attention synergy. The larger ${\mathcal{D}}_{att}$ is, the greater the difference between the attention distributions of MCA and TBEM; this difference may lead to unstable feature learning, making it difficult to ensure the consistency and coherence of attention patterns.

3.3.3. Attention Consistency Loss

To achieve the goal of attention synergy, we define the attention divergence ${\mathcal{D}}_{att}$ as the Attention Consistency Loss (ACL) and incorporate it into the total objective function ${\mathcal{L}}_{total}$ as a spatial regularization term ${\mathcal{L}}_{acl}$ for joint training. ${\mathcal{L}}_{total}$ is defined as follows:

$${\mathcal{L}}_{acl}={\mathcal{D}}_{att}={\parallel A_{\mathrm{MCA}}-A_{\mathrm{TBEM}}\parallel }_1,$$

(8)

$${\mathcal{L}}_{total}={\mathcal{L}}_{det}+\lambda {\mathcal{L}}_{acl},$$

(9)

where ${\mathcal{L}}_{det}$ denotes the detection loss and $\lambda$ is a sensitivity coefficient that balances the contribution of the attention consistency constraint.

During the network optimization process, ${\mathcal{L}}_{acl}$ encourages the model to align the spatial focus of local structural features with global semantic representations. It is important to note that overemphasizing attention consistency may lead to the problem of over-constrained attention, where deep semantic representations are forced to mimic the responses of shallow-level structures. This may cause local texture noise or background interference in shallow-level features to be propagated into the deep semantic space under alignment constraints, thereby contaminating high-level discriminative features, interfering with global semantic modeling, and resulting in negative transfer. Therefore, selecting an appropriate $\lambda$ to strike a balance between attention alignment and representational independence is crucial for achieving effective hierarchical collaboration.

4. Experiments

4.1. Dataset Settings

To comprehensively evaluate the performance of the FER-SDAM, this paper selects two mainstream ‘in-the-wild’ datasets of FER: AffectNet [28] and RAF-DB [29], as well as their Occlusion and Pose Variant Datasets, specifically Occlusion-AffectNet/RAF-DB and Pose-AffectNet/RAF-DB, for comparison.

4.1.1. AffectNet and RAF-DB

The AffectNet database contains about 1 million facial images collected through the internet, of which approximately 500,000 are manually annotated, covering 8 emotion categories (Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral, Contempt). In this paper, the AffectNet-7 dataset containing seven basic emotions and the AffectNet-8 dataset containing eight basic emotions are respectively employed to maintain consistency with common FER settings [30]. The RAF-DB (Real-world Affective Faces Database) contains about 30,000 facial images collected from the internet and cameras, including 12,271 training samples and 3068 testing samples. The detailed statistical data of the dataset are presented in

Table 1. Statistics of the AffectNet-7, AffectNet-8, and RAF-DB datasets.

Dataset	Sub.	Angry	Disgust	Fear	Happy	Sad	Surprise	Neutral	Contempt	Number Sum
AffectNet-7	Train	24,882	3803	6378	134,415	25,459	14,090	74,874	–	287,401
	Val	500	500	500	500	500	500	500	–	3500
AffectNet-8	Train	24,882	3803	6378	134,415	25,459	14,090	74,874	3750	291,651
	Val	500	500	500	500	500	500	500	500	4000
RAF-DB	Train	705	717	281	4772	1982	1290	2524	–	12,271
	Val	162	160	74	1185	478	329	680	–	3068

.

4.1.2. Occlusion and Pose Variant Datasets

The Occlusion-AffectNet/RAF-DB and Pose-AffectNet/RAF-DB datasets for occlusion and pose variation were designed by Wang et al. [22] based on the validation set of AffectNet and the test set of RAF-DB, with annotations for different occlusion types and different pose degrees. The Occlusion-AffectNet/RAF-DB dataset includes several occlusion types, namely wearing a mask, wearing glasses, objects on the left/right, objects on the upper face, objects on the lower face, and non-occlusion. The Pose-AffectNet/RAF-DB dataset consists of two categories: pitch or yaw angles greater than 30 degrees and greater than 45 degrees. The statistical data of the occlusion and pose variation test datasets are presented in the following

Table 2. Statistics of Occlusion and Pose Variant Datasets.

Dataset	Occlusion	Pose (>$30°$)	Pose (>$45°$)
AffectNet	683	1949	985
RAF-DB	783	1248	558

.

4.2. Experimental Setup

All experiments in this paper were conducted on a unified hardware and software platform. The specific experimental setup is shown in

Table 3. Experimental Platform and Training Configuration.

Category	Item	Configuration/Value
Hardware Platform	Operating System	Windows 10
	Processor	AMD Ryzen Threadripper
		PRO 3945WX (12 cores)
	GPU	NVIDIA RTX-A6000
Software Environment	Deep Learning Framework	PyTorch 1.13
	CUDA Version	CUDA 12.8
	Detection Framework	Ultralytics YOLO
Training Settings	Image Size	320 × 320
	Batch Size	64
	Optimizer	SGD
	lr0	0.003
	Momentum	0.9
	Epochs	400
	Patience	40

. During the experiments, the prediction category with the highest confidence from the detection branches was selected as the final sentiment label for the sample, and the experimental results were reported in the form of ‘Accuracy’. The implementation details are as follows:

All input images were uniformly resized to $320\times 320$ pixels and normalized. To enhance the model’s generalization ability and robustness in complex environments, this study strictly followed the default data augmentation strategies of the Ultralytics YOLOv8 framework, including Mosaic, Mixup, random horizontal flipping, and HSV color space adjustments. To address the severe class imbalance in the AffectNet dataset, the model enabled the framework’s built-in class-weighted loss mechanism, which automatically assigns inversely proportional penalty weights based on the occurrence frequency of each emotion class to ensure the network maintains balanced discriminative power for minority class samples.

All training were initialized with a fixed random seed of 0 to ensure deterministic behavior across the initialization and augmentation processes. The model was optimized using Stochastic Gradient Descent (SGD) with an initial learning rate ($l{r}_0$) of 0.003, a momentum of 0.9, and a weight decay of 0.0005. The maximum number of training epochs was set to 400 with a batch size of 64. To prevent overfitting and enhance training efficiency, we implemented an early stopping strategy with a patience of 40 epochs; specifically, the training process was automatically terminated if the validation performance failed to improve for 40 consecutive epochs. This strategy, combined with the fixed seed, ensures that the model consistently captures the optimal convergence state while maintaining high reproducibility.

5. Experimental Results and Discussion

5.1. Comparison with Current State-of-the-Art Methods

5.1.1. Performance Comparison on the AffectNet and RAF-DB

As shown in

Table 4. Comparison with state-of-the-art methods on RAF-DB, AffectNet-7, and AffectNet-8 datasets.

Methods	Year	Accuracy (%)
		RAF-DB	AffectNet-7	AffectNet-8
HAM [20]	2023	91.92	66.97	63.82
FerNeXt [33]	2023	88.56	64.77	-
POSTER++ [34]	2024	92.21	67.49	63.77
LFNSB [35]	2024	91.07	66.57	63.12
ALDL-LRN [36]	2024	85.96	-	-
MFS [37]	2024	91.45	66.51	62.88
BTN [31]	2025	92.54	67.6	64.29
Ada-DF++ [32]	2025	89.21	66.14	63.75
FER-SDAM (Ours)	-	88.87	68.12	66.68

, FER-SDAM demonstrates a clear competitive edge on the large-scale, “real-world” AffectNet dataset, which is significantly more challenging than RAF-DB due to its vast diversity. Under AffectNet-7 and AffectNet-8 settings, our model achieves accuracy rates of 68.12% and 66.68% respectively, outperforming recent state-of-the-art methods like BTN [31] and Ada-DF++ [32].

Although FER-SDAM’s accuracy (88.87%) on the smaller RAF-DB dataset is slightly lower than that of heavyweight architectures such as POSTER++ [34] (92.21%), as discussed in the performance comparison section, FER-SDAM prioritizes hardware-efficient deployment, achieving a ‘cost-effective’ balance suitable for mobile service robots.

5.1.2. Performance Comparison on Occlusion and Pose Variant Datasets

The experimental results in Table 5 and Table 6 further highlight the robustness and reliability of our FER-SDAM under non-ideal conditions.

Occlusion Robustness (Table 5): FER-SDAM achieves the highest accuracy of 85.63% on Occlusion-RAF-DB, slightly outperforming the current best-performing PCNN network [38]. On the Occlusion-AffectNet dataset, FER-SDAM achieves a accuracy of 69.59%, which is more than 5 percentage points higher than AMP-Net [39] and 11 percentage points higher than the region-based RAN [22]. This indicates that when specific regions are occluded, traditional region-based methods (such as RAN) lose critical information, whereas our ACL effectively aligns shallow-level textures with deep-level global semantics, enabling the network to recover affective cues from the remaining visible regions.

Table 5. Comparison with state-of-the-art methods on Occlusion-RAF-DB and Occlusion-AffectNet datasets.

Methods	Year	Accuracy (%)
		Occlusion-RAF-DB	Occlusion-AffectNet
RAN [22]	2020	82.72	58.5
MA-Net [15]	2021	83.65	59.59
VTFF [13]	2021	83.95	62.98
AMP-Net [39]	2022	85.28	64.27
CRS-CONT [40]	2022	84.63	60.91
MGR3Net [30]	2023	84.43	60.67
SLW-LDT [41]	2025	85.14	61.89
CC-KD [42]	2025	86.12	-
PCNN [38]	2025	85.56	-
FER-SDAM (Ours)	-	86.54	69.59

Table 5. Comparison with state-of-the-art methods on Occlusion-RAF-DB and Occlusion-AffectNet datasets.

Methods	Year	Accuracy (%)
		Occlusion-RAF-DB	Occlusion-AffectNet
RAN [22]	2020	82.72	58.5
MA-Net [15]	2021	83.65	59.59
VTFF [13]	2021	83.95	62.98
AMP-Net [39]	2022	85.28	64.27
CRS-CONT [40]	2022	84.63	60.91
MGR3Net [30]	2023	84.43	60.67
SLW-LDT [41]	2025	85.14	61.89
CC-KD [42]	2025	86.12	-
PCNN [38]	2025	85.56	-
FER-SDAM (Ours)	-	86.54	69.59

Pose Variation Robustness (Table 6): Under extreme pose conditions (yaw angle $>45°$), our method maintains superior performance compared to Pose-RAF-DB (91.77%) and Pose-AffectNet (66.51%), outperforming the current state-of-the-art HAM [20] by approximately 2–3 percentage points. This result demonstrates the effectiveness of our proposed hierarchical attention coordination strategy: by precisely capturing facial geometric deformations through the shallow MCA module and combining it with TBEM for deep-level semantic enhancement, the model successfully overcomes the challenge of feature distortion caused by pose variations.

Table 6. Comparison with state-of-the-art methods on Pose-RAF-DB and Pose-AffectNet datasets.

Methods	Year	Pose-RAF-DB (%)		Pose-AffectNet (%)
		(>$30°$)	(>$45°$)	(>$30°$)	(>$45°$)
RAN [22]	2020	86.74	85.2	53.9	53.19
MA-Net [15]	2021	87.89	87.99	57.51	57.78
VTFF [13]	2021	87.97	88.35	60.61	61
AMP-Net [39]	2022	89.75	89.25	61.37	61.16
MGR3Net [30]	2023	88.07	88.13	57.98	58.45
HAM [20]	2023	92.47	91.76	62.99	63.25
MM-Net [43]	2023	89.74	89.07	-	-
SLW-LDT [41]	2025	89.33	88.71	58.47	57.62
CC-KD [42]	2025	90.15	89.07	-	-
ADMF-ER [44]	2025	90.62	91.13	64.92	65.25
PCNN [38]	2025	89.74	88.38	-	-
FER-SDAM (Ours)	-	90.22	91.77	65.76	66.51

Table 6. Comparison with state-of-the-art methods on Pose-RAF-DB and Pose-AffectNet datasets.

Methods	Year	Pose-RAF-DB (%)		Pose-AffectNet (%)
		(>$30°$)	(>$45°$)	(>$30°$)	(>$45°$)
RAN [22]	2020	86.74	85.2	53.9	53.19
MA-Net [15]	2021	87.89	87.99	57.51	57.78
VTFF [13]	2021	87.97	88.35	60.61	61
AMP-Net [39]	2022	89.75	89.25	61.37	61.16
MGR3Net [30]	2023	88.07	88.13	57.98	58.45
HAM [20]	2023	92.47	91.76	62.99	63.25
MM-Net [43]	2023	89.74	89.07	-	-
SLW-LDT [41]	2025	89.33	88.71	58.47	57.62
CC-KD [42]	2025	90.15	89.07	-	-
ADMF-ER [44]	2025	90.62	91.13	64.92	65.25
PCNN [38]	2025	89.74	88.38	-	-
FER-SDAM (Ours)	-	90.22	91.77	65.76	66.51

In summary, FER-SDAM consistently exhibits reliable recognition capabilities regardless of whether it faces partial occlusion or significant pose deviations. These results validate the network’s exceptional adaptability in complex human–robot interaction scenarios, as well as the theoretical soundness of our attention coordination paradigm.

5.2. Ablation Study

To systematically evaluate the contribution of each individual component within the FER-SDAM framework, we conduct incremental ablation experiments on the AffectNet and RAF-DB datasets. The quantitative results are summarized in

Table 7. Impact of different modules on performance.

Methods	MCA	TBEM	AS	AffectNet-7	AffectNet-8	RAF-DB
YOLOv8s (Baseline)	×	×	×	66.0	62.6	85.5
Variant A	✓	×	×	66.3	63.0	85.2
Variant B	×	✓	×	66.3	62.1	85.3
Variant C	✓	✓	×	66.6	62.3	85.9
FER-SDAM	✓	✓	✓	68.12	66.68	88.87

YOLOv8s (Baseline): Without any enhancement modules. Variant A: With MCA (Multi-dimensional Cooperative Attention). Variant B: With TBEM (Transformer-based Block Enhancement Module). Variant C: With both MCA and TBEM. FER-SDAM: Full model with MCA, TBEM, and AS. ‘✓’ indicates that the module is included, while ‘×’ indicates that the module is excluded.

.

Compared with the YOLOv8s baseline, the independent integration of either the MCA or TBEM module yields performance gains on AffectNet-7. However, the improvements are relatively constrained, with marginal fluctuations observed across AffectNet-8 and RAF-DB. This suggests that a decoupled enhancement of either local textures or global dependencies is insufficient to maintain consistent robustness across diverse data distributions. The simultaneous incorporation of MCA and TBEM outperforms any single-module configuration across all benchmarks. This performance boost validates the complementarity between local multi-scale feature refinement and long-range semantic modeling, effectively bridging the gap between fine-grained expression details and holistic context. The most significant performance leap is observed with the introduction of the AS mechanism. Specifically, the full FER-SDAM model achieves accuracies of 68.12%, 66.68%, and 88.87% on the three datasets, representing a substantial improvement of 2.12%, 4.08%, and 3.37% over the baseline, respectively. This steady advancement demonstrates that the AS mechanism effectively harmonizes heterogeneous attention responses, mitigating semantic conflicts and significantly enhancing the model’s discriminative power and stability in complex HRI scenarios.

5.3. Influence of the Sensitivity Coefficient

To investigate the sensitivity of model performance to the weight coefficient $\lambda$ in the attention consistency loss, we conducted comparative experiments on the RAF-DB, AffectNet-7, and AffectNet-8 datasets, with $\lambda$ varying from 0.0 to 0.7, ensuring that the proposed attention consistency loss $L_{ACL}$ and the detection loss $L_{det}$ are of the same order of magnitude. The results are illustrated in Figure 5.

Overall, the model’s performance exhibits a trend of ‘initial fluctuation, peak attainment, and subsequent degradation’ as $\lambda$ increases. When $\lambda$ is relatively small ($\lambda \le 0.05$), the consistency constraint is insufficient to align the hierarchical attention maps, resulting in limited performance gains. As $\lambda$ increases into the range of 0.1 to 0.3, the recognition accuracy on all three datasets improves significantly. Specifically, RAF-DB achieves its peak accuracy of 88.87% at both $\lambda =0.1$ and $\lambda =0.3$, while AffectNet-7 and AffectNet-8 reach their optimal performance (68.12% and 66.68%, respectively) at $\lambda =0.3$.

It is noteworthy that a slight performance dip occurs at $\lambda =0.2$ across all datasets, which may be attributed to a temporary optimization imbalance between the detection and consistency loss terms. When $\lambda$ exceeds 0.4, the model begins to be affected by an over-constrained attention mechanism, and the accuracy across all datasets shows a sustained decline to varying degrees. At this point, the Attention Consistency Loss excessively influences the optimization process. Under these conditions, the deep semantic layers are overly constrained in order to align with the shallow attention distributions, resulting in reduced flexibility when modeling global dependencies. From an optimization perspective, this over-constraint may lead to negative transfer. Shallow attention often captures detailed but noisy local patterns, which are not always semantically relevant. When forced alignment becomes too strong, these noisy or misleading local signals are propagated to deeper layers, amplifying biases and interfering with the learning of high-level semantic representations. This creates an optimization conflict between the goal of minimizing attention divergence and the goal of maximizing the discriminative power specific to the task, thus affecting the representational capacity of the model and leading to a decrease in performance. Our experimental results indicate that the optimal trade-off is achieved when $\lambda =0.3$. At this value, attention alignment is sufficiently ensured without suppressing semantic diversity, thereby effectively avoiding negative transfer.

5.4. Visualization Analysis

To gain deeper insights into the underlying mechanisms of the hierarchical attention synergy (AS) mechanism, we employed the Grad-CAM algorithm to visualize and compare attention distributions across different model configurations (Figure 6).

The baseline model (YOLOv8s) can roughly locate facial regions, but its attention distribution is relatively diffuse and susceptible to interference from background or irrelevant areas. When only the MCA (Variant A) is integrated, the model’s focus on locally emotion-sensitive regions (such as the corners of the eyes and nostrils) markedly increases while effectively suppressing background noise. Its attention distribution exhibits locally high-response peaks. Conversely, when only the TBEM (Variant B) is introduced to model long-range semantic dependencies, the model’s perception of overall facial structure significantly improves, with broader attention coverage. However, it lacks pronounced peak responses in key emotional regions. Furthermore, when MCA and TBEM are jointly trained but independently optimized (Variant C), the attention distribution stabilizes and concentrates primarily on core regions of emotional expression. However, the true performance leap occurs upon introducing the AS mechanism (FER-SDAM), which explicitly aligns the spatial distribution of shallow structural perception with deep global semantics through Attention Consistency Loss (ACL).

ACL enforces spatial consistency between local and global attention, establishing a strongly coupled co-optimization paradigm. This design effectively overcomes the semantic fragmentation issues caused by traditional “local–global independent optimization.” As shown in Figure 5, FER-SDAM’s heatmap not only preserves the integrity of global facial structures but also exhibits highly focused and intensely activated responses at key expression action units (e.g., brow furrowing, lip corner elevation). This “hierarchically progressive, semantically aligned” attention evolution mechanism constitutes the core mechanism enabling the model’s high robustness under challenging conditions such as occlusion or extreme lighting: even when partial local features are missing due to occlusion, global semantic information can guide attention to dynamically refocus on remaining reliable emotion-sensitive regions via ACL, thereby maintaining stable recognition performance.

5.5. F1-Score and Confusion Matrix

In this study, we compared the F1 scores and normalized confusion matrices of FER-SDAM and the baseline model (YOLOv8s) on the AffectNet and RAF-DB datasets.

As shown in

Table 8. Performance comparison of FER-SDAM and the baseline (YOLOv8s) on AffectNet and RAF-DB datasets.

Method	AffectNet		RAF-DB
	Accuracy (%)	F1-Score	Accuracy (%)	F1-Score
YOLOv8s (Baseline)	62.6	0.62	85.5	0.81
FER-SDAM	66.68	0.63	88.87	0.83

, on the challenging AffectNet dataset, FER-SDAM achieved a 4.08% increase in precision while simultaneously improving the F1 score by 0.1. On the RAF-DB dataset, the F1 score also improved by 0.2. This indicates that FER-SDAM effectively balances precision and recall, ensuring robust performance even under class-imbalanced conditions.

In the normalized confusion matrices shown in Figure 7 and Figure 8, FER-SDAM exhibits a more pronounced diagonal dominance and lower off-diagonal confusion, indicating stronger discriminative power between visually similar facial expressions.

For easily distinguishable emotions such as ‘Happy’, both models achieve high accuracy on both datasets, while FER-SDAM further improves or maintains the performance of several challenging categories. On AffectNet (Figure 7), FER-SDAM yields notable improvements in the recognition of ‘Neutral’, ‘Surprise’, ‘Disgust,’ and ‘Sad’. On RAF-DB (Figure 8), FER-SDAM increases the recognition rates of ‘Surprise’, ‘Fear’, ‘Disgust’, and ‘Sad’ by 3%, 5%, 2%, and 1%, respectively, while simultaneously reducing the confusion among ‘Disgust’, ‘Fear’, ‘Neutral’, and ‘Angry’.

These results indicate that FER-SDAM exhibits stronger discriminative ability in distinguishing visually similar emotion categories, validating the effectiveness of hierarchical attention co-modeling in fine-grained facial expression modeling.

Moreover, the accuracy of ‘Fear’ and ‘Angry’ on AffectNet, as well as those of ‘Angry’ and ‘Neutral’ on RAF-DB, remains comparable between the two models. This suggests that the AS mechanism does not impact the performance of relatively easy categories.

5.6. Comparison of Computational Efficiency

We compare the computational overhead of each method and the results are shown in

Table 9. Computational cost and inference time comparison of different methods.

Methods	RAF-DB Accuracy	Parameters (M)	MACs (G)	Time (ms)	FPS
AMP-Net [39]	0.89	59.44	4.74	26.47 ± 0.52	37.77
POSTER++ [34]	0.92	43.7	8.48	13.09 ± 0.22	74.40
PCNN [38]	0.89	51.00	14.8	10.61 ± 0.27	94.26
MobileNetV2	0.72	3.5	0.30	8.51 ± 0.21	117.24
ShuffleNetV2	0.81	2.3	0.15	7.80 ± 0.18	128.12
FER-SDAM (Ours)	0.89	11.93	1.79	10.29 ± 0.20	97.21

.

The parameters of AMP-Net [39], POSTER++ [34] and PCNN [38] are 59.44 M, 43.7 M and 51.00 M, respectively, while the proposed method is only 11.93 M, which reduces the parameters by about 80% compared with AMP-Net. In terms of computational complexity, the MACs of our method is only 1.79 G, which is about 62.2%, 78.9%and 87.9% lower than that of AMP-Net, POSTER++ and PCNN, respectively, which significantly reduces the computational burden of the model. The average inference time of each image on GPU is 10.29 ± 0.20 ms, which is basically the same as that of PCNN and significantly better than that of AMP-Net and POSTER++. The corresponding frame rate reaches 97.21 FPS, which is the highest among all methods, and fully meets the requirements of real-time and even high-speed scenes.

We further compared FER-SDAM with representative lightweight architectures to assess its edge-side deployment potential. As shown in Table 9, although extreme-lightweight baselines like MobileNetV2 and ShuffleNetV2 achieve higher inference speeds, their recognition accuracy drops significantly. In contrast, FER-SDAM strikes a superior balance: it achieves an accuracy of 0.89—on par with heavyweight models—while maintaining an average inference time of $10.29\pm 0.20$ ms. The corresponding frame rate of 97.21 FPS far exceeds the 25 FPS threshold required for real-time human–robot interaction. These results demonstrate that FER-SDAM offers exceptional efficiency and robustness, making it a highly practical solution for real-world deployment on embedded robotic platforms.

5.7. FER-SDAM Applied to Emotional Robot Recognition

To evaluate the system’s performance on edge devices, this paper conducts a controlled comparison between FER-SDAM and the baseline model YOLOv8s on the robotic platform shown in Figure 9(left). The robotic system integrates a high-resolution camera, an interactive display terminal, a mobile chassis equipped with a control motherboard and drive wheels, and an industrial PC (Intel i5 CPU + NVIDIA GeForce MX250 (2GB) GPU).

Both models were tested in real time using 256 × 256 image inputs under identical human–robot interaction (HRI) conditions, covering both the normal environment shown in Figure 9 and typical non-ideal scenarios involving occlusion, pose changes, low-light conditions, and complex backgrounds. Each scenario was tested independently in 100 interaction sessions, with the recognition results serving as the trigger signal for the robot’s voice-based emotional feedback. Evaluation was based on three key metrics: average recognition accuracy, inference frame rate, and interaction delay (including the delay of the facial detector). The results are shown in

Table 10. Quantitative performance comparison between FER-SDAM and the baseline on the service robot platform.

Methods	Indoor Environment Type	Number of Test	Average Recognition Accuracy (%)	Inference Frame Rate (FPS)	Interaction Delay (ms)
YOLOv8s	Normal	100	92.0	32.5	$48.21\pm 9$
	Complex	500	72.0	29.6	$67.46\pm 12$
FER-SDAM	Normal	100	95.0	29.8	$55.39\pm 3$
	Complex	500	90.0	29.2	$59.68\pm 5$

.

As shown in Table 10, FER-SDAM demonstrates excellent operational stability and perception reliability in real-world environments. Although the baseline model, YOLOv8s, achieves a high frame rate, its interaction latency fluctuates more significantly under complex conditions (up to ±12 ms). In contrast, FER-SDAM maintains a stable latency of 55–64 milliseconds, ensuring more natural and fluid human–computer interaction. More importantly, the proposed method demonstrates superior accuracy in non-ideal scenarios. In environmental tests classified as complex, FER-SDAM maintained an accuracy of 90.0%, which is 18.0% higher than that of the baseline method. This confirms that our hierarchical attention coordination mechanism effectively mitigates environmental interference.

To further evaluate the model’s reliability, Figure 9(right) shows exemplary qualitative results. As demonstrated in Figure 9b,c, even when subjects wear masks or are in cluttered backgrounds, FER-SDAM maintains high-confidence recognition by focusing on visible emotion-sensitive regions (e.g., the eyes) through its collaborative attention mechanism. Under dim lighting (Figure 9d), the model identifies subtle local muscle tension changes in the ‘Fear’ expression, showcasing robust feature extraction capabilities.

Finally, to explore the algorithm’s boundaries, we conducted a quantitative failure analysis focused on head pose variations. The results indicate that FER-SDAM maintains high robustness (accuracy $>85\%$) for yaw angles within $\pm 45°$. However, when the facial yaw angle exceeds 50° (as shown in Figure 9f), the accuracy drops to approximately 62.0%. This degradation is primarily due to extreme geometric distortion and the significant loss of key facial features, reflecting the inherent physical perception limitations of monocular vision systems under drastic pose variations. Overall, these results validate FER-SDAM’s robustness, real-time performance, and suitability for complex real-world interactive scenarios.

6. Conclusions

In this paper, we present FER-SDAM, a synergetic dual-attention network specifically engineered for emotional service robots. By establishing a hierarchical modeling paradigm, the proposed framework successfully bridges the gap between shallow-level structural perception and deep-level global semantic dependencies. The core of our approach, the Attention Synergy (AS) mechanism, effectively harmonizes heterogeneous attention responses through structural regularization. This alignment not only alleviates the semantic conflicts inherent in multi-module learning but also fosters a coherent and interpretable attention pattern, which is fundamental for ensuring the perceptual stability of robots in unconstrained settings.

Experimental results demonstrate that FER-SDAM not only achieves an optimal balance between recognition accuracy and computational efficiency but also exhibits stable and robust performance under challenging conditions such as occlusion and large pose variations, thereby providing an efficient and well-rounded solution for facial expression recognition in service robotics applications. As a highly practical solution, FER-SDAM lays a solid technical foundation for the development of next-generation emotional robots, enabling them to integrate more deeply into social service scenarios and achieve complex human–robot emotional interactions.