Context_Driven Emotion Recognition: Integrating Multi_Cue Fusion and Attention Mechanisms for Enhanced Accuracy on the NCAER_S Dataset

Abstract

In recent years, most conventional emotion recognition approaches have concentrated primarily on facial cues, often overlooking complementary sources of information such as body posture and contextual background. This limitation reduces their effectiveness in complex, real-world environments. In this work, we present a multi-branch emotion recognition framework that separately processes facial, bodily, and contextual information using three dedicated neural networks. To better capture contextual cues, we intentionally mask the face and body of the main subject within the scene, prompting the model to explore alternative visual elements that may convey emotional states. To further enhance the quality of the extracted features, we integrate both channel and spatial attention mechanisms into the network architecture. Evaluated on the challenging NCAER-S dataset, our model achieves an accuracy of 56.42%, surpassing the state-of-the-art GLAMOUR-Net. These results highlight the effectiveness of combining multi-cue representation and attention-guided feature extraction for robust emotion recognition in unconstrained settings. The findings also highlight the importance of accurate emotion recognition for human–computer interaction, where affect detection enables systems to adapt to users and deliver more effective experiences.

1. Introduction

Humans interpret emotions through multiple signals, including facial expression, physical motion and gaits [1,2], human pose [3], environmental context, and speech [4] to interpret emotions, which goes far beyond the monofunctional approach typically used by machines. Despite considerable advances in research, machines still struggle to match our natural ability to recognize nuanced, adaptive emotions. However, extracting human emotions from visual data has become a source of significant interest in computer vision research, leading to advanced applications in sectors such as surveillance, driving monitoring, healthcare, and human–computer interaction systems [5,6,7,8,9].

Emotion recognition also plays a critical role in human–computer interaction (HCI). By detecting emotions from the face, body, and surrounding context, interactive systems can adapt their responses to user needs in real-time. For instance, a tutoring application can recognize when a learner shows signs of frustration and provide additional guidance, or a virtual assistant can identify positive engagement and maintain the current dialogue. Strengthening emotion recognition methods, therefore, contributes directly to improving adaptability, usability, and the overall quality of user experience in human–computer interaction.

Earlier methods [10,11,12,13] focus only on facial expressions and consider them as the main form of non-verbal communication. In this context, datasets like AFEW [14] and FER2013 [15] provide only cropped and aligned facial images. However, these approaches struggle in real-world situations because they overlook crucial contextual details beyond the face.

According to an experimental psychology study, Stefano et al. [16] found that body language conveys affective information that facial expressions alone cannot fully capture. Even more so, methods of emotion recognition demonstrated the importance of facial expressions, body language, movements, postures, and gestures. However, these cues alone are often insufficient in real-world scenarios, where contextual elements also influence emotions. The context includes background information, interactions, and situational cues that provide a more complete and human-like perception of emotions. Several studies [17,18,19] have been incorporating this context to improve emotion recognition, like [17], who combined features extracted from the face and the context, which is the entire image excluding the face. Despite these advances, the challenge of generalizing across diverse environments with heterogeneous elements remains significant. Also [18] combined face, body, and context features to capture different aspects of emotional expression. However, they incorporated the body within the context cue, which introduces redundancy. This overlap causes the model to learn similar information from both cues, leading it to focus on repetitive features and ultimately reducing the network’s overall performance.

To deal with the above problems, we remove both the body and the face from the context image, forcing the model to focus on more important and relevant features. Our goal is to enable the model to learn the key elements necessary for emotion recognition. Additionally, we apply spatial and channel attention mechanisms in the context and face to guide the model’s focus toward critical regions, thereby enhancing its ability to extract meaningful information and improving overall performance.

In this paper, we present a multi-source emotion recognition model that detects human emotions using image data. In our approach, we identified three distinct streams that are exploited to improve classification performance in various emotion categories. These streams are the face-encoding stream, which focuses on capturing facial cues, the body-encoding stream, which analyzes posture, and the context-encoding stream, which extracts scene-level information. The key idea in our approach is the removal of the face and body from the context images, enabling the model to focus more on background details and surrounding elements, which may offer additional emotional cues. Moreover, spatial and channel attention mechanisms are incorporated after each convolutional block in the face- and context-encoding streams, enabling the model to prioritize the most pertinent features for emotion recognition. We considered the work of Willams Costa et al. [18] as a baseline and developed an enhanced pipeline that improves emotion recognition accuracy by integrating these three streams. We summarize our main contributions as follows:

• We employ YOLOv8 to extract the faces and bodies of the main actor with greater precision.
• We enhance the accuracy of the combined model by applying spatial and channel attention mechanisms after each convolution.
• We addressed class imbalances using Focal Loss, which significantly enhanced the model’s ability to learn from underrepresented classes.
• The context image is constructed by removing the face and body of the primary subject, encouraging the model to explore other visual elements within the scene.
• The proposed framework leverages complementary cues from face, body, and context to address the limitations of single-modality approaches.
• Contextual understanding is enhanced by directing the model’s attention toward background objects and secondary actors in the scene.
• Emotion recognition performance is improved by isolating redundant features and forcing the network to learn from diverse visual signals.

To facilitate readability, the main notations and abbreviations used in this paper are summarized in

Table 1. Notations and their meanings, as used in the proposed method.

Notation	Meaning
N	Number of detected faces in an image.
$B_j=[x_j,y_j,w_j,h_j]$	Bounding box of the j-th detected face, defined by top-left corner $(x_j,y_j)$, width $w_j$, and height $h_j$.
$w_j,h_j$	Width and height of the j-th bounding box.
$A_j$	Area of the j-th face bounding box.
$inde{x}_{\max }$	Index of the face with the maximum bounding box area.
$B_f$	Bounding box of the main actor’s face (largest detected face).
$I_{\mathrm{original}}$	Original input image from the dataset.
$I_f$	Cropped face region of the main actor extracted from $I_{\mathrm{original}}$.
$B_i=[x_i,y_i,w_i,h_i]$	Bounding box of the i-th detected body.
$B_{\mathrm{body}}$	Bounding box of the main actor’s body (with largest overlap with $B_f$).
$A_{\mathrm{inter}}(B_f,B_i)$	Intersection area between the face bounding box $B_f$ and a body bounding box $B_i$.
$M_i$	Binary segmentation mask of the i-th detected body.
$M_{\mathrm{body}}$	Segmentation mask corresponding to the main actor’s body.
$I_b$	Body image of the main actor.
$I_c$	Context image obtained by masking the main actor’s body.
F	Input feature map of the attention module.
$M_C\left(F\right)$	Channel attention map.
$F^{\prime }$	Refined feature map after applying channel attention.
$M_S\left(F^{\prime }\right)$	Spatial attention map.
$F^{″}$	Final refined feature map after applying spatial attention to $F^{\prime }$.
$V_f=(f_1,f_2,\dots ,f_{n_f})$	Feature vector extracted from the face encoding stream, where $f_k$ denotes the k-th component.
$V_b=(b_1,b_2,\dots ,b_{n_b})$	Feature vector extracted from the body encoding stream, where $b_k$ denotes the k-th component.
$V_c=(c_1,c_2,\dots ,c_{n_c})$	Feature vector extracted from the context encoding stream, where $c_k$ denotes the k-th component.
$V_{\mathrm{Concat}}$	Concatenated vector of $V_f$, $V_b$, and $V_c$ used for fusion.
$n_f,n_b,n_c$	Dimensions of the face, body, and context feature vectors.
$\sigma$	Sigmoid activation function used in channel and spatial attention.

.

2. Related Work

2.1. Emotion Recognition

The aim of emotion recognition is to accurately understand human emotion, as human–machine interactions are constantly evolving. Most approaches to human emotion recognition began with efforts to analyze facial expressions as the primary source of emotional cues. However, facial expressions alone often fail to capture the complexity of emotions in real-world scenarios, even though the effort to extract meaningful features from the face has advanced [10,13,20,21].

In order to solve this limitation, some methods that use body language have been proposed [22,23]. Mei Si et al. [22] used the features representing posture and body movements to automatically detect people’s emotions in non-gaming scenarios. They focused on four emotions often observed when people play video games: triumph, frustration, defeat, and concentration. Ahmed et al. [23] proposed a two-layer method to select emotion-specific features from a comprehensive list of body movement descriptors. This approach addresses the challenge of identifying relevant features from a vast dataset of human body movements and achieved a very high emotion recognition rate.

Combinations of these features have been also applied to enhance emotion recognition. Multimodal approaches often achieve a better classification performance than uni-modal methods. David Griol et al. [24] introduced an emotion recognition system that integrates both speech and facial cues; they combined these two modalities with a late fusion strategy. G. et al. [25] achieved an improvement in the recognition of eight emotions in ten subjects by fusing the features extracted from facial expressions, body movement, and gestures and speech. Recently, much work has focused on exploring context-sensitive information for emotion recognition. The first challenge involves creating a new dataset of images featuring people in real-world contexts, a novel data set call the “Emotions in Context Database” (EMOTIC) [26], containing people in context in non-controlled environments; this dataset motivates further research [27] to extract meaningful information from both the person and the surrounding scene, which provides contextual information. They then combined the extracted features to enhance the emotion recognition. Lee [17] introduced a novel benchmark for context-aware emotion recognition, called CAER; they also presented a novel deep learning architecture called CAER-Net, which integrates both facial expression analysis and contextual information that hides faces in scenes to focus on other contextual elements using attention mechanisms. Yang, D. et al. [28] identified four contexts to improve emotion recognition: (1) multimodal context combines facial expressions, landmarks, gestures, and gait for representation; (2) scene context uses attention modules to analyse surroundings and extract emotion semantics; (3) surrounding agent context explores emotional influence among agents in the same scene; (4) agent–object context examines interactions between agents and objects. Le et al. [29] proposed a new global local attention mechanism that extracts features from facial and contextual regions independently, then learns them simultaneously using the attention module. Willams Costaa et al. [18] proposed a new direction for emotion recognition based on multiple cues using face and context information and body poses. A comparative summary of the related studies discussed above is presented in

Table 2. Summary of related work on approaches to emotion recognition.

References	Authors	Problems	Solving Methods
[22]	Mei Si et al.	Limited emotion recognition when using only facial expressions.	Used posture and body movement features to detect four emotions (triumph, frustration, defeat, concentration).
[23]	Ahmed et al.	Challenge of selecting relevant features from a large set of body movement descriptors.	Two-layer method to extract emotion-specific features of the body.
[24,25]	David Griol et al. Caridakis et al.	Facial cues alone are insufficient for robust recognition of multiple emotions.	(1) Combined speech and facial cues using a late fusion strategy. (2) Fused facial expressions, body movements, gestures, and speech to improve recognition.
[17,18,26,27]	Kosti et al. Lee et al. Willams Costa et al.	Lack of datasets with contextual information in real-world scenarios.	(1) Created EMOTIC dataset and extracted features from person and context for improved recognition. (2) Proposed CAER benchmark and CAER-Net, integrating facial and contextual features with attention to non-facial elements. (3) Introduced a multimodal approach combining face, context, and body pose cues.
[28,29]	Yang et al. Le et al.	Context in emotion recognition is inherently vast and multifaceted, making it difficult to model consistently across different environments and scenarios.	(1) Defined four context types: multimodal, scene, surrounding agent, and agent object; used attention modules. (2) Proposed global local attention to extract facial and contextual features separately, then learn them jointly.

.

2.2. Attention Mechanisms

Attention mechanisms originated in neural machine translation [30]. Researchers soon generalized this concept, integrating attention into transformer architectures that revolutionized natural language processing [31], and later adapting it to enhance image classification [32,33,34] and image segmentation [35,36] in computer vision. In recent years, using the attentional mechanism,  an important component of neural networks, Aminbeidokhti et al. [37] applied spatial attention to emotion recognition by generating masks that highlight the most informative facial regions. Building on this, Squeeze-and-Excitation blocks later introduced channel-wise recalibration to weight informative feature maps [38]. More recently, researchers have embedded combined channel and spatial attention modules within convolutional backbones to refine feature selection, enabling models to emphasize both the most informative filters and the most informative spatial locations simultaneously and thus bolster performance under occlusion and complex backgrounds [39].

3. Methodology

3.1. Proposed Network Architecture

We describe our proposed method in this section, as illustrated in Figure 1, which utilizes a single dataset called NCAER_S. For each sample, three images are extracted. Our proposed method is structured into three streams: the face-encoding stream, body-encoding stream, and context-encoding stream. The features extracted from these streams are combined and processed through two dense layers; then, the softmax function is used to classify the output into seven emotions. The following subsections present a detailed explanation of all the encoding streams.

3.2. Preprocessing Pipeline

Before we begin training, we first apply a preprocessing step to add some variability between epochs while keeping the important features intact. Each input image, whether from the face-, context-, or body-encoding streams, is resized to a consistent 112 × 112 resolution.

3.3. Face-Encoding Stream

Our approach aims to identify the main actor’s face, which is present in each image. We assume that each scene corresponds to a single emotion; however, the NCAER-S dataset [29] does not clarify which actor is associated with the annotated emotion, making the task particularly complex. The challenge arises from the fact that several faces can appear in the same scene, each with a different emotion.

To address this, we first apply YOLOv8m_200e, an intermediate version of YOLOv8 [40] trained for 200 epochs. The model effectively identifies and localizes faces in complex scenes within the NCAER-S dataset, which includes multiple people, varying light conditions, angles, and occlusions. It generates bounding boxes around the detected faces, denoted by $B_j=[x_j,y_j,w_j,h_j]$ for $j=1,\dots ,N$, where N is the number of detected faces in the image. We assume that the face of the main actor is the one with the largest bounding box. The area $A_j$ is calculated using Equation (1) once the index $inde{x}_{\max }$ of the face with the largest area is found, based on Equation (2). We use this index to retrieve the corresponding bounding box $B_f$ using (3). This bounding box is then used to extract the face region from the original image $I_{\mathrm{original}}$, as shown in (4). Finally, we associate the extracted face with the scene’s emotion annotation. The overall process of detecting and extracting the main actor’s face is illustrated in Figure 2.

$$A_j=w_j\times h_j.$$

(1)

$$inde{x}_{\max }=arg\; max\left(A_j\right).$$

(2)

$$B_{\mathrm{f}}=B_{{\mathrm{index}}_{\max }}.$$

(3)

$$I_{\mathrm{f}}=I_{\mathrm{original}}[y_f:y_f+h_f,x_f:x_f+w_f].$$

(4)

The cropped face of the main actor, denoted as $I_f$, is then used as input for the face-encoding stream, as illustrated in the architecture of Figure 3. This face encoding stream consists of five layers of 5 × 5 convolutional layers, followed by Batch Normalization and rectified linear unit (ReLU) activation. Motivated by [39], our face-encoding stream uses CBAM architecture after each convolution and batch Normalization to direct the model toward the most relevant features at every stage, and eliminates irrelevant information before the next layer processes it to improve both local and global feature extraction.

The CBAM architecture operates by sequentially applying channel attention and spatial attention modules. It first applies channel attention to the input feature map, denoted as F, to refine features based on their importance. The channel attention map $M_C$ is computed by first applying average pooling and maximum pooling operations on the input feature map F to obtain two feature vectors. These vectors are then passed through a multi-layer perceptron (MLP) [41]. Finally, the outputs of the $MLP\left(\mathrm{AvgPool}\right(F\left)\right)$ and $MLP\left(\mathrm{MaxPool}\right(F\left)\right)$ are summed using element-wise summation, and the Sigmoid activation function $\sigma$ is applied to this sum to normalize the values to a range between 0 and 1. In short, the channel attention is computed as follows:

$$M_C\left(F\right)=\sigma (\mathrm{MLP}\left(\mathrm{AvgPool}\left(F\right)\right)+\mathrm{MLP}\left(\mathrm{MaxPool}\left(F\right)\right)).$$

(5)

where $\sigma$ represents the Sigmoid activation function.

The refined feature map $F^{\prime }$ is obtained by multiplying the channel attention map $M_C\left(F\right)$ with the input feature map F to enhance the most important channels and suppress less relevant ones. The specific calculation formula is as follows:

$$F^{\prime }=M_C\left(F\right)·F.$$

(6)

Next, the architecture applies spatial attention to emphasize key regions in the image. The spatial attention map $M_s\left(F^{\prime }\right)$ is computed by first applying average pooling and maximum pooling operations to the feature map $F^{\prime }$. Then, the two pooled feature maps are concatenated and a convolution operation with a 7 × 7 kernel is applied. Finally, the Sigmoid activation function $\sigma$ is applied to the result to normalize the values to a range between 0 and 1. The specific calculation formula is as follows:

$$M_S\left(F^{\prime }\right)=\sigma \left({\mathrm{Conv}2\mathrm{D}}_{7\times 7}\left(\mathrm{Concat}(\mathrm{AvgPool}\left(F^{\prime }\right),\mathrm{MaxPool}\left(F^{\prime }\right))\right)\right).$$

(7)

where $\sigma$ represents the Sigmoid activation function.

Finally, $F^{\prime \prime }$ results from the element-wise multiplication of the spatial attention map and $F^{\prime }$. This process allows the network to focus on significant spatial regions.

$$F^{″}=M_S\left(F^{\prime }\right)·F^{\prime }$$

(8)

As shown in Figure 3, we then use maximum-pooling with a stride of 2, and finally, to avoid overfitting, we use dropout.

3.4. Body-Encoding Stream

Figure 4 illustrates the overall procedure for detecting and extracting the main actor’s body using YOLOv8 [40]. In this process, and in order to incorporate body cues for non-verbal emotion recognition, we relied on the pre-trained YOLOv8 [40] model, which was trained on diverse benchmark datasets to automatically detect and segment human bodies within each scene.

As can be seen in Figure 2, once the main actor’s face with bounding box $B_f$ was identified, we moved on to extract their body. We used YOLOv8 [40] to detect and segment the bodies in the image $I_{\mathrm{original}}$, with each having its own bounding box expressed using Equation (9). Each bounding box is defined by the coordinates of its top-left corner and its width and height.

$$B_i=\left[x_i,y_i,w_i,h_i\right],\mathrm{for}i=1,\dots ,N.$$

(9)

where N is the number of detected bodies in the image $I_{\mathrm{original}}$.

Thus, in order to obtain the main actor’s body, we calculated the intersection area between the face bounding box of the main actor $B_f$ and each bounding box of the detected bodies $B_i$. The formula in Equation  (10) computes the intersection areas and returns the largest intersection area, from which we extract the index of the corresponding bounding box, selected as the box of the body of the primary actor $B_{\mathrm{body}}$.

$$\begin{array}{cc}\hfill A_{\mathrm{inter}}(B_f,B_i)& =max\left(0,min(x_f+w_f,x_i+w_i)-max(x_f,x_i)\right)\hfill \\ \hfill & \times max\left(0,min(y_f+h_f,y_i+h_i)-max(y_f,y_i)\right)\hfill \end{array}$$

(10)

where $A_{\mathrm{inter}}(B_f,B_i)$ represents the maximum area of intersection between the face bounding box $B_{\mathrm{f}}$ and each of the body bounding boxes $B_{\mathrm{i}}$.

Each bounding box is associated with a segmentation mask $M_{\mathrm{i}}$, where $M_{\mathrm{i}}$ is a binary mask indicating the presence of the detected object within the bounding box. After identifying the bounding box of the primary actor’s body denoted as $B_{\mathrm{body}}$, the corresponding segmentation mask $M_{\mathrm{body}}$, is applied to the original image $I_{\mathrm{original}}$, resulting in the image $I_{\mathrm{b}}$, where only the primary actor’s body is isolated and the background is set to black. In short, this can be expressed as:

$$I_{\mathrm{b}}=I_{\mathrm{original}}\times M_{\mathrm{body}}$$

(11)

We used the main actor’s body denoted as $I_b$, as input for the body-encoding stream, as illustrated in our architecture in Figure 1.

As shown in Figure 5, the body-encoding module consists of a six-layer convolutional network with size 128, 32, 128, 128, 96, and 192, followed by batch normalization and rectified linear unit (ReLU) activation functions, with three-layer max pooling with a kernel size of 2. Finally, the output is flattened to prepare for downstream processing.

3.5. Context-Encoding Stream

Figure 6 illustrates the procedure for generating the context image used in our model. To extract the context image, we utilized the previously obtained segmentation mask $M_{\mathrm{body}}$ of the body of the primary actor. The segmented body region was then blacked out in the original image, which can be expressed mathematically as follows:

$$I_{\mathrm{c}}=I_{\mathrm{original}}\times (1-M_{\mathrm{body}})$$

(12)

This operation removed the actor and left only the surrounding environment. The final image, labeled as $I_{\mathrm{c}}$, kept important details like objects, background elements, and other people, providing useful context for the scene. This step ensures that this stream does not learn the duplicated information, as in the body-encoding stream.

In order to extract meaningful contextual information, we removed body regions from the images, and the extracted image, denoted as $I_c$, was used an input to a context-encoding stream. As shown in Figure 7, our stream consists of seven convolutional layers with filter sizes of 32, 32, 64, 64, 128, 128, and 256. Every two convolutional layers except the last one follow Channel Attention [39], which allows the model to focus on important feature channels and assign higher weights to the most relevant ones. The network captures the relationships between different feature maps and ensures that significant details contribute more to the final representation. After this, we integrated Spatial Attention [39] in order to detect crucial regions within the image and directs the model’s focus to the most informative areas. After the attention mechanisms, we added rectified linear unit activation functions, Average Pooling operations to refine feature extraction, and dropout to prevent overfitting.

Both attention mechanisms [39] play a key role in improving accuracy as they help the model prioritize relevant features. The model applies them to filter out irrelevant information and retain critical scene details.

3.6. Adaptive Fusion Networks

According to our proposed architecture, before final classification, it is necessary to combine the three features extracted using the three encoding streams, as shown in Figure 1; the facial features vector $V_f$, the body features vector $V_b$, and the context features vector $V_c$ are concatenated in a single vector $V_{\mathrm{Concat}}$ shown in  (14) and then used as input to the two dense layers; then, the Softmax layer is used to obtain the final classification.

$$V_f=\left(\begin{array}{c}{f}_1\\ f_2\\ ⋮\\ f_{n_f}\end{array}\right)\in {\mathbb{R}}^{n_f},V_b=\left(\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_{n_b}\end{array}\right)\in {\mathbb{R}}^{n_b},V_c=\left(\begin{array}{c}{c}_1\\ c_2\\ ⋮\\ c_{n_c}\end{array}\right)\in {\mathbb{R}}^{n_c}.$$

(13)

$$V_{\mathrm{Concat}}=\left(\begin{array}{c}{f}_1\\ f_2\\ ⋮\\ f_{n_f}\\ b_1\\ b_2\\ ⋮\\ b_{n_b}\\ c_1\\ c_2\\ ⋮\\ c_{n_c}\end{array}\right)\in {\mathbb{R}}^{n_f+n_b+n_c}.$$

(14)

This means that a new vector is created by sequentially placing the components of $V_f$, followed by those of $V_b$, and finally those of $V_c$. The concatenation operation merges these three vectors into a single vector.

4. Experimental Setup

4.1. Datasets

We experimented with our proposed approach using the NCAER-S dataset. Nhat et al. [29] extracted NCAER-S from CAER-S [17] to improve its robustness. CAER-S [17] lacked generalization because many images in its training and test sets came from the same videos. This similarity made the model less effective at handling new data. The new dataset ensured a better separation between training and test images, which allow for more reliable evaluations. According to Figure 8, we can observe that some classes did not have enough images in the test set; therefore, we combined the training, test, and validation datasets and split them again for a balanced distribution.

4.2. Visualization

The following figure represents the distribution of images for each category in NCAER-S dataset. We can observe that the emotion “Neutral” has the highest number of data and “Happy” has the lowest amount of data. We can see from Figure 8 that the original NCAER-S dataset is unbalanced, and some classes have very few images. Also, some test set classes contain less than 10 percent of the total dataset, which makes evaluation unreliable. To fix this, we merged the training, validation, and test sets, then split the dataset again to allocate 10 percent of the total images to the test set. This ensured a sufficient number of evaluation samples but did not fully resolve the imbalance across emotion categories. To mitigate this limitation, we applied data augmentation to the face, body, and context images. After resizing each image to 112 × 112 pixels, we introduced variability by performing random horizontal flipping, which generates mirrored views of body, context, and facial orientations to account for natural variations in posture and head direction, brightness adjustments (up to a maximum delta of 0.1) to emulate different lighting conditions, and contrast adjustments (within a range of 0.9–1.1) to capture variations in visual intensity. These transformations enriched the underrepresented classes while preserving their emotional content. Figure 9 presents the new dataset distribution after this change.

4.3. Implementation Details

We implemented our networks using the TensorFlow 2.10 framework [42] and trained them with the Adam optimizer [43]. All experiments were performed on a workstation equipped with an NVIDIA GeForce RTX 4060 GPU (8 GB VRAM), Intel Core i7-13620H CPU, and 16 GB RAM, running windows 10. The initial learning rate is modulated by a cyclic learning rate schedule based on a cosine function to enhance training dynamics. This strategy dynamically adjusts the learning rate within a predefined range, oscillating smoothly between a minimum value of 0.0001 and a maximum of 0.01 over cycles of 60 epochs. We also use early stopping to avoid overfitting by monitoring the validation loss and terminating training when no improvement is observed within a specified number of epochs. To deal with the class imbalance in our dataset, we use the focal loss function expressed as (15), introduced by Lin et al. in [44]. Unlike standard cross-entropy loss, focal loss reduces the weight of easy-to-classify examples and pushes the model to focus on harder, misclassified ones.

$$\mathrm{FocalLoss}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{c=1}^C\left[-\alpha {\left(1-p_{ic}\right)}^{\gamma }{y}_{ic}ln\left(p_{ic}\right)\right].$$

(15)

where:

• N is the number of samples.
• C is the number of classes.
• $p_{ic}$ is the predicted probability for sample i and class c.
• $y_{ic}$ is the ground-truth label for sample i and class c.
• $\alpha$ is a weighting factor that adjusts the importance of each class, and $\gamma$, which controls the degree of emphasis placed on hard-to-classify examples. In our implementation, we set $\alpha =0.25$ to address the class imbalance, and $\gamma =2$ to direct the model’s attention towards more difficult examples.

The term ${(1-p_{ic})}^{\gamma }$ increases the emphasis on samples where $p_{ic}$ for the correct class is low, while $\alpha$ balances the influence of each class.

5. Results and Discussion

After our experiments on the NCAER_S dataset, we found that integrating YOLOv8 into the face-selector algorithm mentioned in Section 3.3 significantly improves the identification of the main actor compared to Dlib. Dlib, an open-source toolkit widely used for face detection and landmark localization [45], is known for its efficiency, but often struggles in complex scenarios with multiple faces, occlusions, or extreme head poses. This limitation arises because Dlib relies on traditional regression tree methods for face alignment rather than deep learning-based architectures, which restricts its robustness in unconstrained environments. In contrast, YOLOv8, as a modern deep learning detector, reliably captured every visible face in complex scenes and dynamic images of the NCAER_S dataset, which made it much easier to isolate the main actor’s face. This robustness against occlusions and extreme poses directly strengthened our proposed emotion classification pipeline by ensuring that the correct facial region was always selected. Similarly, since the body selector algorithm is based on the face-selector algorithm, we can conclude that it also effectively segments the correct body of the main actor.

Table 3. Ablation experiment results of our proposed emotion recognition network. ✓ indicates that the component is included; x indicates that the component is excluded.

Method	Body & Face Selector Algorithm (YoLov8)	Body Enc. (Ch + Sp Att.) 1	Context Enc. (Ch + Sp Att.) 2	Face Enc. (Ch + Sp Att.) 3	Acc (%d)
Our method	✓	x	x	x	54.28%
	✓	x	x	✓	55.92%
	✓	x	✓	✓	56.62%

¹ Body-encoding stream with channel and spatial attention. ² context-encoding stream with channel and spatial attention. ³ face-encoding stream with channel and spatial attention.

presents the quantitative results of our proposed experiment, which are examined in the subsequent discussion. Following standard practice in the literature [17,46], we evaluated all experiments using the accuracy metric.

Our first proposed experiment focuses on the fusion of three cues: face, body, and context. This combination is essential for achieving the best performance. Each cue plays a unique role in enhancing the results. For example, the body-encoding stream becomes important in complex situations where the context alone may not provide sufficient information. Likewise, the context stream improves the model’s accuracy when the body pose is unavailable.

In addition to the multi-cue fusion, we integrated attention mechanisms within the NCAER_S dataset by applying channel and spatial attention modules after each convolutional block in the face- and context-encoding streams, while leaving the body-encoding stream unchanged. This architecture yielded a significant improvement in classification accuracy. To assess the specific impact of the attention modules on the fusion of these cues, we trained an identical model without them, keeping all other parameters and training conditions constant. As shown in Table 3, the model achieved an accuracy of 56.42% with the attention mechanisms, compared to 54.29% without them. This demonstrates the importance of attention in refining the model’s ability to focus on the most relevant features and further enhancing the fusion of face, body, and context cues. For the context-encoding stream, we deliberately removed the body information and added attention modules to reduce feature redundancy and to encourage the model to extract complementary cues from surrounding actors and objects. As illustrated in Figure 10, our method allows the context-encoding stream to focus on essential contextual details, such as the faces and the bodies of other actors, which might be overshadowed when the body is present. This approach was especially beneficial in cases where multiple individuals appeared in the same scene and exhibited similar emotional expressions, as shown in Figure 11. In contrast, GLAMORNet [29] leave the body inside the context image, which might distract the model from focusing clearly on important background details.

Another experiment focused on addressing class imbalances using focal loss. We observed that its use resulted in a more rapid decrease in loss compared to the standard cross-entropy loss function. This behavior is primarily due to the inherent characteristics of focal loss [44], which places greater emphasis on hard-to-classify examples while downweighting easy ones. In contrast, standard cross-entropy loss, which is used in most existing work in the field of emotion recognition [18,29,47], treats all examples equally. This can cause the model to converge slower, especially when it is confident about easy examples.

The confusion matrix in Figure 12 shows that the model performs best on Fear, which reached 85% accuracy. It also performs well on Disgust (75%) and Sad (61%). In contrast, it struggles to correctly classify Happy (35%), Surprise (35%), Neutral (49%), and Anger (38%). These four emotions are frequently confused with one another, especially Neutral. The model misclassifies Happy as Neutral in 22% of cases, Surprise as Neutral in 21% of cases, and Anger as Neutral in 25% of cases. This pattern suggests both feature overlap and a strong class imbalance. The training set contains over 2500 Neutral examples, as shown in Figure 9, while the Happy and Surprise examples each contain almost half that number. This imbalance is probably behind the default choice of the Neutral model in cases of uncertainty. Anger, which shares certain visual cues with Disgust and Neutral, also suffers from this effect.

The results show that our model achieves the best performance among all evaluated architectures.

Our proposed method was compared with several existing baselines, such as the MCF_NET [48] and GLAMORNet [29] methods. The results, shown in in

Table 4. Comparison of our network with recent state-of-the-art methods on the NCAER-S dataset.

Methods	Accuracy (%)
MCF_NET [48] (Face + Context)	45.59
GLAMOR-Net (Original) [29] (Face + Context)	46.91
GLAMOR-Net (MobileNetV2) [29] (Face + Context)	47.52
GLAMOR-Net (ResNet-18) [29] (Face + Context)	48.40
Our Method (body + Context)	49.89
Our Method (Face + Context + Body)	56.42

, indicate that our model outperforms all baseline architectures. Specifically, our network increases classification accuracy by 8.02% compared to GLAMORNet [29]. The reasons for the high performance of our proposed method are twofold: (1) MCF_NET [48] and GLAMORNet [29] process the entire scene, including body information, through a single context module, without explicitly isolating the body cues. In contrast, our architecture introduces a dedicated body-encoding stream that receives segmented body inputs. This approach directs the model’s attention to gestures, stance, and motion, rather than expecting the context encoder to infer these features indirectly. As a result, the network learns pose-specific representations and subtle body movements that a holistic representation of the context might not capture. (2) Our architecture removes the body from the scene context because it has already extracted the body as input for the body-encoding stream. Moreover, our approach applies channel and spatial attention mechanisms at every convolution layer in order to direct the encoding stream for both context and face to focus on the most relevant features at every stage.

Summary of Achieved Results

The experiments on the NCAER-S dataset demonstrate the effectiveness of the proposed multi-cue emotion recognition architecture. The main findings can be summarized as follows:

• Face and Body Selection: Replacing Dlib with YOLOv8m_200e improved the robustness of actor detection, ensuring accurate face localization even under occlusion and complex poses. This enhancement also benefited the body-extraction stage, as the body region was derived from the detected face.
• Multimodal Fusion: Combining face, body, and context features led to higher recognition accuracy than any unimodal configuration. Each modality contributed complementary information: body cues improved recognition in challenging scenarios, while context compensated for missing or ambiguous body cues.
• Attention Mechanisms: The integration of channel and spatial attention (CBAM) within the face and context streams refined feature representation. The ablation study (Table 3) showed an accuracy gain from 54.29% (without attention) to 56.42% (with attention).
• Comparison with State-of-the-Art: As shown in Table 4, the proposed model surpassed existing multimodal baselines such as MCF_NET and GLAMOR-Net. It achieved an accuracy of 56.42%, representing an improvement of 8.02% over GLAMOR-Net.
• Per-Class Performance: The confusion matrix (Figure 12) revealed strong recognition of emotions such as Fear (85%), and Disgust (75%), but persistent confusion among visually similar categories like Happy, Surprise, and Neutral, largely due to dataset imbalance.
• Loss Function: Employing focal loss instead of cross-entropy accelerated convergence and improved classification for minority classes.

These results confirm that robust actor selection, attention-guided feature refinement, and the adaptive fusion of face, body, and context streams jointly enhance recognition accuracy and robustness. The comparative findings also demonstrate that the proposed architecture provides a significant improvement over prior multimodal approaches.

6. Conclusions

This paper demonstrates that combining face, body, and context cues improves emotion recognition. Each element contributes uniquely: facial expressions reveal detailed emotions, body posture offers additional context, and the surrounding scene helps determine the emotional context. We enhanced feature extraction through spatial and channel attention mechanisms and designed the context stream to exclude the body region, forcing the model to focus on non-redundant scene elements. These design choices led to a clear improvement in classification accuracy. Our approach outperforms existing methods and demonstrates the effectiveness of integrating multiple visual cues with attention-based strategies for robust and scalable emotion recognition in real-world settings. Beyond classification, the proposed framework strengthens the role of emotion recognition in human–computer interaction, where robust emotion detection enables more adaptive, natural, and user-centered experiences.

Our method only applies to static images, which restricts the range of emotional information the model can capture. In the future, we intend to use temporal information coming from video sequences. Also, audio signals are another source of useful information, especially when there is uncertainty in the visual input. In addition, we intend to evaluate our method on a more complex dataset that includes multiple faces and complex scenes. This setup better reflects real-world conditions and enables the model to interpret a wider set of emotional cues that are closer to how humans recognize emotions.