Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features

Abstract

This paper proposes a method for multi-label visual emotion recognition that fuses fore-background features to address the following issues that visual-based multi-label emotion recognition often overlooks: the impacts of the background that the person is placed in and the foreground, such as social interactions between different individuals on emotion recognition; the simplification of multi-label recognition tasks into multiple binary classification tasks; and it ignores the global correlations between different emotion labels. First, a fore-background-aware emotion recognition model (FB-ER) is proposed, which is a three-branch multi-feature hybrid fusion network. It efficiently extracts body features by designing a core region unit (CR-Unit) that represents background features as background keywords and extracts depth map information to model social interactions between different individuals as foreground features. These three features are fused at both the feature and decision levels. Second, a multi-label emotion recognition classifier (ML-ERC) is proposed, which captures the relationship between different emotion labels by designing a label co-occurrence probability matrix and cosine similarity matrix, and uses graph convolutional networks to learn correlations between different emotion labels to generate a classifier that considers emotion correlations. Finally, the visual features are combined with the object classifier to enable the multi-label recognition of 26 different emotions. The proposed method was evaluated on the Emotic dataset, and the results show an improvement of 0.732% in the mAP and 0.007 in the Jaccard’s coefficient compared with the state-of-the-art method.

1. Introduction

Emotion recognition has become increasingly integrated into various aspects of daily life and social activities, such as monitoring student learning states [1], enhancing human–computer interaction [2], driver monitoring [3], and lie detection [4].

In visual-based emotion recognition research, facial expressions have traditionally been regarded as the most effective features [5]. The field has seen rapid advancements in facial expression analysis, thus leading to significant progress. However, in real-world scenarios, individuals are often engaged in diverse social activities within varied environments, which makes it challenging to accurately capture facial information due to factors such as head tilts, occlusions, and uneven lighting. Beyond facial expressions, recent studies began to explore additional visual cues. For instance, Aviezer et al. [6] and Martinez [7] conducted psychological experiments in silent environments where participants’ faces were obscured. Their findings revealed that observers could accurately infer emotions through body posture and environmental context, thus suggesting that these factors are also valuable for emotion recognition. Additionally, there has been limited exploration of foreground information, such as social interactions between individuals, which can offer insights into behaviors through features like distance and proximity, thereby enhancing the emotion recognition accuracy.

Most current research on emotion recognition [8,9,10] focuses on the single-label recognition of a few common emotions, such as happiness, sadness, surprise, neutrality, and anger. However, human emotions are inherently complex and multidimensional, and single-label classification falls short of capturing the full spectrum of emotional expression. Emotion datasets [11] encompass 26 fine-grained emotion categories, and multi-label recognition of these categories can provide a more nuanced and comprehensive description of an individual’s emotional state. In multi-label emotion recognition, some studies [12] overlook the relationships between different emotions, often decomposing the task into independent binary classification tasks, which fails to account for the global correlations between all emotion labels and leads to reduced recognition accuracy. Traditional multi-label classification methods, such as ML-KNN [13] and ML-RBF [14], attempt to learn label correlations through kernel functions, loss functions, and association rules. However, these methods are often inefficient and primarily capture local correlations. The self-attention mechanism in Transformer architectures was also employed for modeling label correlations [15,16], but it is more suited for long sequence data and demands substantial computational resources. Graph-based approaches can model label correlations as well, but the construction of the adjacency matrix is crucial. Consequently, developing a multi-label classifier that effectively captures the global correlations of emotions remains a significant challenge.

To address the above issues, this paper proposes a method of multi-label visual emotion recognition that fuses fore-background features. The contributions of this approach are highlighted in three key aspects as follows:

• A fore-background-aware emotion recognition model (FB-ER) is proposed. The background in which the person is placed and the foreground, such as interactions between different individuals, provide beneficial visual cues for emotion recognition, as shown in Figure 1. FB-ER is a three-branch multi-feature hybrid fusion network. It designs a core region unit (CR-Unit) to effectively extract body features, represents background features as background keywords to help the model understand emotion expressions in specific contexts through semantic data, and captures depth map information to better simulate interactions between different individuals as foreground features. These three types of features are fused both at the feature level and the decision level.
  

  Figure 1. Fore−background features diagram.
• A multi-label emotion recognition classifier model (ML-ERC) is introduced, which utilizes graph convolutional networks (GCNs) to capture label correlations. The nodes in the graph are represented by word-embedding vectors associated with different emotion labels. The edge weights are determined by considering both the label co-occurrence probability matrix, as shown in Figure 2, and the cosine similarity matrix. Information is propagated between nodes by the GCN, which enables the classifier to learn the inter-relational information between emotional labels.
  

  Figure 2. Directed graph between the first five emotions (threshold: 0.2). Dashed lines represent weak correlations, solid lines represent strong correlations.
• An end-to-end network architecture that combines FB-ER and ML-ERC was designed for the multi-label recognition of 26 distinct emotions. This architecture demonstrates generalizability across various environments and emotion categories.

The rest of this paper is organized as follows: Section 2 describes the current related work of visual emotion recognition and multi-label emotion classification. Section 3 gives a description of the proposed FB-ER and ML-ERC. Section 4 introduces the experimental designation and verifies the performance of the proposed method. Finally, Section 5 concludes this paper.

2. Related Work

2.1. Visual Emotion Recognition

Significant progress has been made in the study of facial emotions in recent years. These methods have achieved superior performance by developing effective facial feature extraction networks. Siqueira et al. [8] and Karani et al. [10] optimized their CNN models for computational efficiency, which resulted in models with both high computational efficiency and accuracy. Liao et al. [9] enhanced emotion recognition by incorporating facial optical flow information, while Arabian et al. [17] constructed a facial key point grid and utilized GCN for emotion classification. Kim et al. [18] adopted the designs of VGG, Inception-v1, ResNet, and Xception to propose the CVGG-19 architecture for classifying driver emotions, which offers the advantages of high accuracy and low cost. Oh et al. [19] investigated the impact of noise in image data on classification accuracy and proposed a deep learning model to enhance robustness against noise. Khan et al. [20] proposed a novel temporal shifting approach for a frame-wise transformer-based model using multi-head self/cross-attention (MSCA) to reduce the computational cost of emotion recognition. However, these methods focus solely on the facial region and do not consider other visual information beyond the face.

Several studies [11,21,22,23,24] explored the use of visual information beyond facial features. For instance, Mou et al. [21] employed a dual-stream network, where one branch extracted body features and the other extracted environmental features using a k-NN classifier to independently identify arousal and valence from these feature sets. However, due to the simplicity of the feature extraction network, it could not fully capture the necessary features. Additionally, the fusion of the two branches’ outputs occurred at the decision level, without accounting for the influence of environmental features on emotion recognition. Building on this work, Kosti et al. [11], Lee et al. [22], and Zhang et al. [23] concatenated different visual features for emotion classification. Nonetheless, the recognition accuracy varied greatly across different emotions, particularly those that are more difficult to classify. Ilyes et al. [24] proposed a multi-label focal loss function to address the issue of imbalanced emotion categories, which led to improved emotion recognition performance. However, the networks used in these studies exhibited limited feature extraction capabilities, underutilized background features, and relied on the simple concatenation of features without considering their complementarity. Furthermore, foreground information, such as social interactions between individuals, which can provide valuable insights into behavioral characteristics and enhance emotion recognition accuracy, remains underexplored in the current literature.

2.2. Multi-Label Emotion Classification

Multi-label classification methods can be categorized into three strategies: first order, second order, and high order [12]. The first-order strategy decomposes multi-label classification into multiple independent binary classification tasks while neglecting correlations between labels. The second-order strategy considers pairwise label relationships and distinguishes between related and unrelated labels. However, in real-world scenarios, label dependencies often exceed the limitations of second-order correlations. The high-order strategy addresses the associations between multiple labels and the influence of each label on all others, thus providing a more robust framework for modeling label correlations. Traditional high-order methods include ML-KNN [13], ML-RBF [14], BPMLL [25], and MLC-ARM [26]. ML-KNN [13] generates predictions by considering the relevance of local samples, but it fails to capture the global correlations across all labels. ML-RBF [14] employs a kernel function in feature space to account for label dependencies, while BPMLL [25] introduces a ranking loss function to minimize the dissimilarity between labels. However, as data correlations exceed the second-order level, the effectiveness of both ML-RBF and BPMLL becomes constrained. MLC-ARM [26] leverages association rules mined from sample correlations to perform multi-label classification, but its efficiency is hampered by the high computational costs involved.

In recent years, several approaches adapted modified Transformer architectures to learn correlations between different labels. For instance, methods like Q2L [15], ML-Decoder [16], and SAML [27] utilize enhanced Transformer models and the self-attention mechanism to capture inter-label correlations. However, these methods are better suited for capturing correlations in long sequence data and may not perform optimally when modeling label correlations in image data. Additionally, they demand significant computational resources and storage capacity.

In the field of object detection, several studies [28,29,30,31] constructed simple graph structures to represent the relationships between different objects within a single image, where an edge with a value of 0 denotes that two labels are unrelated, and an edge with a value of 1 indicates that they are related. Graph convolutional networks (GCNs) are then employed to learn local correlations across different images. However, these methods fail to account for the varying strengths of label correlations and overlook the global relationships between labels, and the application of GCNs for modeling correlations has seldom been explored in the domain of emotion recognition.

3. Approach

3.1. Fore-Background-Aware Emotion Recognition (FB-ER)

3.1.1. Relationship between Fore-Background and Emotions

Table 1 presents three different scenarios (derived from the Emotic dataset [11] and Google images): First, facial and postural cues are entirely obscured, which makes it impossible to use these features for emotion recognition. In such cases, the background context becomes crucial for inferring emotions. For example, in Scenario 1, individuals are shown sunbathing; although their faces are not visible, elements like clear water and a bright sky suggest a happy mood. Second, as illustrated in Scenario 2, faces and postures are partially covered. In these instances, emotions are often classified as neutral, which implies neither happiness nor sadness. However, by considering the background, alternative interpretations may arise. For instance, in a wedding setting, more fitting emotions could include affection, esteem, or happiness. Third, when facial expressions are fully visible, as shown in Scenario 3, the same expression can convey different emotions depending on the background context. For example, both expressions in Scenario 3 were initially classified as pain. Yet, with background consideration, one image set in a hospital and the other in a sports stadium, the latter scenario might more accurately reflect emotions such as disquiet, engagement, or excitement.

Table 1. Example table of correlations between background and emotions.

Index	Description	Approximate Proportion in the Dataset	Original Figure	Emotion (without Background)	Background Heat Map	Background Semantics	Emotion (with Background Information)
No. 1	Completely lacking facial information.	16.7%		-		Beach, water park, sunny, boating, swimming	[“Anticipation”, “Engagement”, “Happiness”, “Pleasure”]
				-		Physics, chemistry lab, working, competing, stressful	[“Fatigue”, “Suffering”]
				-		Ski, mountain, snowy, cold, sunny, far-away horizon	[“Engagement”, “Excitement”]
No. 2	Partial facial information.	45%		Neutral		Ballroom, legislative, beauty salon, socializing, congregating	[“Affection”, “Esteem”, “Happiness”]
				Neutral		Bedroom, enclosed area, cloth, warm, dry	[“Peace”, “Happiness”]
No.3	When facial information is complete, the same facial expression can be observed in different backgrounds.	38.3%		Pain		Hospital, operating, working, stressful, medical activity	[“Pain”, “Sadness”, “Suffering”]
				Pain		Martial gym, competing, sports, exercise, congregating	[“Disquiet”, “Engagement”, “Excitement”]

Next, the relationships between the foreground and emotions are examined. In this context, the foreground refers to the interactions between individuals within the image. These interactions can be divided into two scenarios, as depicted in Table 2 (using images from the Emotic dataset [11] and Google): First, when individuals share a common identity or are familiar with one another, their emotional tendencies often converge, as illustrated in Scenario 1. Second, when individuals possess different identities or are unfamiliar with each other, their emotional tendencies may diverge, as shown in Scenario 2. This suggests that emotions can spread rapidly through social interactions, and an individual’s emotional state is significantly influenced by their interactions with others.

Table 2. Example table of correlations between foreground and emotions.

Index	Description	Approximate Proportion in the Dataset	Original Figure	Depth Map	Emotions of Character A	Emotions of Character B
No. 1	Interaction occurs when individuals share a common identity or mutual familiarity.	10.4%			[“Affection”, “Happiness”]	[“Affection”, “Happiness”]
					[“Confidence”, “Engagement”]	[“Confidence”, “Engagement”]
No. 2	Having different identities or being unfamiliar with one another.	19.6%			[“Engagement”]	[“Anticipation”, “Esteem”]
					[“Anticipation”, “Confidence”, “Excitement”]	[“Peace”, “Engagement”]

3.1.2. Emotion Recognition Fusing Fore-Background Features

FB-ER consists of three sub-network branches, as depicted in Figure 3. The first branch is dedicated to the human body, where it extracts essential cues, such as facial expressions and body postures. To achieve this, ResNet18 [32] is utilized as the backbone network for feature extraction. Transfer learning is employed to fine-tune the pre-trained model developed by Krizhevsky et al. [33]. Following the extraction of the body features, a CR-Unit is introduced to emphasize regions that contribute the most to emotion recognition. The structure of the CR-Unit is presented in Figure 4, with its mathematical formulation detailed in Equations (1)–(3):

$$f\left(F_{body},nu{m}_1\right)=BatchNorm\left(Linea{r}_1\left(F_{body},nu{m}_1\right)\right)$$

(1)

$$Core\_weights=Adaptive\left[\sigma \left(Linea{r}_2\left(nu{m}_1,nu{m}_2\right)\right)\right]$$

(2)

$$F_{core}=\mathrm{diag}\left[dot\left(F_{body},Attention\_weights\right)\right]$$

(3)

Figure 3. Overall architecture diagram.

Figure 4. CR-Unit diagram.

In Equation (1), $F_{body}$ represents the body features, which are mapped to length $nu{m}_1$ through $Linea{r}_1$, followed by $BatchNorm$. In Equation (2), the features are mapped from length $nu{m}_1$ to length $nu{m}_2$ by $Linea{r}_2$, and then $Core\_weights$ of the same length as $F_{body}$ is obtained using $Sigmoid$ and $Adaptive$. In Equation (3), $dot$ denotes the dot product operation, and $diag$ represents the extraction of diagonal elements, thus ultimately yielding $F_{core}$. $F_{core}$ focuses more on the regions of $F_{body}$ that are beneficial for emotion classification, thus effectively enhancing the model’s ability to recognize the important parts of the $F_{body}$.

The second sub-network branch was designed for background feature extraction, which provides useful information for understanding the emotions of individuals in the image. The background can be considered as a set of keywords associated with objects and ongoing activities within a scene. To evaluate the scene comprehension, the Places365 model [34] was utilized. The results presented in Table 3 exhibit two different scenes: outdoors and indoors. For the outdoor scene, the recognition keywords include “outdoor”, “cliff”, “climbing”, and “sunny”. For the indoor scene, the keywords include “indoor”, “office”, “working”, and “closed area”. ResNet18 was also employed for extracting background features. Transfer learning techniques were used to fine-tune the pre-trained weights of the Places365 model in this study.

Table 3. Background semantic recognition.

Original Figure	Heat Map	Recognition Results
		Outdoors, cliff, natural, sunny, climbing, rugged scene, far away horizon
		Indoors, office, working, studying, enclosed area, no horizon

The third sub-network branch was designed for foreground feature extraction, which primarily includes the interactions between different individuals. These interactions are valuable for emotion evaluation. This study adopted a method based on depth map extraction to simulate the interaction and proximity between different individuals. The steps are organized as follows:

Step 1—Data preprocessing: all images in the dataset are normalized to RGB three channels.

Step 2—Depth map extraction: MegaDepth [35] is a powerful tool for depth map extraction that was designed to generate depth information from single-view images using advanced stereo matching techniques. Its pre-trained models exhibit a high accuracy and reliability, which enables precise depth map extraction across diverse scenes, lighting conditions, and viewpoints. Therefore, we utilized the MegaDepth pre-trained model to obtain depth maps from the original images.

Step 3—Color rendering: color rendering is applied to the acquired depth map to enhance the visual effect, as illustrated in Figure 5.

Figure 5. Examples of depth maps.

Step 4—Depth matrix calculation: The depth matrix information D of the depth map is calculated using Equation (4) presented below:

$$D=\left[\left(\begin{array}{cccc}D\left(1,1\right)& D\left(1,2\right)& ....& D\left(1,N\right)\\ D\left(2,1\right)& D\left(2,2\right)& ....& D\left(2,N\right)\\ ....& ....& ....& ....\\ D\left(M,1\right)& D\left(M,2\right)& ....& D\left(M,N\right)\end{array}\right)\right]$$

(4)

where D represents a matrix of dimensions $M\times N$, and $D\left(i,j\right)$ denotes the depth value corresponding to column j in row i within the depth map. The third sub-network branch comprises three convolutional layers, three pooling layers, and three fully connected layers, which collectively extracts distinctive features from the depth map.

By extracting the aforementioned features and performing shape adjustment, the following features are obtained: core feature $F_{core}$, background feature $F_{back}$, foreground feature $F_{fore}$, and fusion feature $F_{mix}$:

$$\left\{\begin{array}{c}{F}_{core}\to batchsize\times d_1\hfill \\ F_{back}\to batchsize\times d_2\hfill \\ F_{fore}\to batchsize\times d_3\hfill \end{array}\right.$$

(5)

$$\left\{\begin{array}{c}{F}_{mix}=F_{core}\left|\right|F_{back}\left|\right|F_{fore}\hfill \\ F_{mix}\to batchsize\times \left(d_1+d_2+d_3\right)\hfill \end{array}\right.$$

(6)

where $d_i$ represents the vector length and $\left|\right|$ denotes the concatenation operation. Subsequently, $F_{core}$, $F_{back}$, and $F_{mix}$ are combined with the multi-label object classifier to perform emotion recognition for 26 different types of emotions. The optimal weights of the three emotion recognition results are obtained through training, and different weights are assigned to represent the relative importances of $F_{core}$, $F_{back}$, and $F_{mix}$.

3.2. Multi-Label Emotion Recognition Classifier (ML-ERC)

3.2.1. The Design of the Label Co-Occurrence Probability Matrix

$G=\left\{N,E\right\}$ represents a global label correlation graph structure, where N represents all nodes and E represents all edges within the graph, as illustrated in Equations (7) and (8):

$$\left\{\begin{array}{c}N=\left\{n_1,n_2,n_3,\dots \dots ,n_m\right\}\hfill \\ n_{k\in \left[1,m\right]}=R_k^{\omega }\Rightarrow N={\mathbb{R}}^{m\times \omega }\hfill \\ N={\mathbb{R}}^{m\times \omega }=\left\{R_1^{\omega },R_2^{\omega },R_3^{\omega },\dots \dots ,R_m^{\omega }\right\}\hfill \end{array}\right.$$

(7)

$$\left\{\begin{array}{c}E=\left\{e_{ij}{|}_{i,j\in \mathsf{\Omega }}\right\}\hfill \\ {\left(e_{ij}\right)}_{i,j\in \mathsf{\Omega }}={\left({\theta }_{ij}\right)}_{i,j\in \mathsf{\Omega }}\hfill \end{array}\right.$$

(8)

In Equation (7), each graph node, as denoted by $n_k$, corresponds to an emotion type, where $k\in \left[1,m\right]$, with m representing the number of emotion categories. Each node is represented by a word-embedding vector $R_k^{\omega }$ with a length of $\omega$ using Word2Vec and GloVe. Consequently, all the nodes in the graph structure can be represented as ${\mathbb{R}}^{m\times \omega }$. In Equation (8), $e_{ij}$ represents the edge between node i and node j, Ω represents the set of all edges in the graph structure, and ${\theta }_{ij}$ denotes the weight of the edge between node i and node j. A larger value of ${\theta }_{ij}$ indicates a stronger association between nodes i and j.

In this study, the correlation between different emotions was determined by analyzing the co-occurrence patterns of labels. A represents the label co-occurrence probability matrix, which can be constructed as follows:

Step 1: Compute the co-occurrence count $C{\left(i,j\right)}_{i,j\in \left[1,m\right]}$ of each pair of emotion labels i and j statistically from the dataset.

Step 2: Compute the conditional probability $P\left(i|j\right)$ for each pair of emotion labels in Equation (9):

$$P\left(i|j\right)={\displaystyle \frac{P\left(ij\right)}{P\left(j\right)}}={\displaystyle \frac{C{\left(i,j\right)}_{i,j\in \left[1,m\right]}}{C{\left(j\right)}_{j\in \left[1,m\right]}}}$$

(9)

$C{\left(j\right)}_{j\in \left[1,m\right]}$ represents the frequency of occurrence of emotion label j, while $C{\left(i,j\right)}_{i,j\in \left[1,m\right]}$ represents the frequency of co-occurrence of emotion labels i and j.

Step 3: Construct the conditional probability matrix P:

$$P=\left[\begin{array}{cccc}1& P\left(2|1\right)& \dots & P\left(m|1\right)\\ P\left(1|2\right)& 1& \dots & P\left(m|2\right)\\ \dots & \dots & \dots & \dots \\ P\left(1|m\right)& P\left(2|m\right)& \dots & 1\end{array}\right]$$

(10)

Step 4: According to Equation (11), a judgment is made on whether there is a correlation between different emotional labels. $\tau$ represents the empirical threshold, which indicates the degree of correlation between i and j. Values lower than $\tau$ indicate a weaker correlation and should be ignored. After many experimental verifications, $\tau$ was set to 0.2.

$$\widehat{P}=\left\{\begin{array}{c}0,ifP\left(j|i\right)<\tau \hfill \\ 1,ifP\left(j|i\right)\ge \tau \hfill \end{array}\right.$$

(11)

Step 5: The normalization operation in Equation (12) is used to balance the connection weights between different nodes. The parameter $\lambda$ was set to $1\times {10}^{-6}$, which was used to prevent the denominator from being zero.

$${\tilde{P}}_{ij}=\left\{\begin{array}{c}{\displaystyle \frac{{\widehat{P}}_{ij}}{\lambda +{\sum }_{i=1}^m{\widehat{P}}_{ij}}},i\ne j\hfill \\ 1,i=j\hfill \end{array}\right.$$

(12)

Step 6: To ensure more balanced and stable feature updates in the GCN and improve the model’s performance, further normalization of ${\tilde{P}}_{ij}$ is applied using Equations (13)–(15) based on the normalized Laplacian matrix principle. $D_{ij}$ represents the degree matrix, and A represents the label co-occurrence probability matrix.

$$D_{ij}=\left\{\begin{array}{c}{\sum }_{j=1}^m{\tilde{P}}_{ij},i=j\hfill \\ 0,i\ne j\hfill \end{array}\right.$$

(13)

$${\tilde{D}}_{ij}=D_{ij}^{-{\displaystyle \frac{1}{2}}}$$

(14)

$$A=\left\{\begin{array}{c}{\tilde{D}}_{ij}\tilde{P}{\tilde{D}}_{ij},i\ne j\hfill \\ 1,i=j\hfill \end{array}\right.$$

(15)

3.2.2. The Design of the Cosine Similarity Matrix

Continuous convolution in GCNs reduces the similarity of node representations in the original feature space, as illustrated in Figure 6a,b. Figure 6a shows the Isomap [36] dimensionality reduction plot of the initial node semantic features ${\mathbb{R}}^{m\times \omega }$, where semantically similar emotions are closely positioned, such as “Excitement, Happiness, Pleasure”, “Anger, Annoyance, Aversion”, and “Suffering, Pain”. Figure 6b shows the Isomap dimensionality reduction plot of the node semantic features ${\mathbb{R}}^{m\times \omega }$ after two layers of convolution, where the similarity of node semantic features is disrupted, with semantically similar emotions positioned far apart and dissimilar emotions positioned close together.

Figure 6. (a) Graph of dimension reduction using initial node features with Isomap. (b) Graph of dimension reduction using Isomap with node features after two layers of GCN convolution.

To prevent consecutive convolution operations from disrupting the similarity of node features, a cosine similarity matrix C is introduced to ensure that the semantic similarity of nodes remains unchanged after each convolution. The construction is organized as follows:

Step 1: Obtain the initial cosine similarity matrix $C_{ij}$ according to Equation (16):

$$C_{ij}={\displaystyle \frac{R_i^{\omega }·R_j^{\omega }}{\left|\right|R_i^{\omega }\left|\right|\left|\right|R_j^{\omega }\left|\right|}}$$

(16)

where $\left|\right|\dots \left|\right|$ represents the $L_2$ norm.

Step 2: To ensure that similarity values between different emotion categories are on the same scale, eliminate dimensional discrepancies, and reduce the impact of extreme values, normalize $C_{ij}$ by the Z-score to obtain the cosine similarity matrix C. Figure 7b illustrates the resulting C using ${\mathbb{R}}^{m\times \omega }{|}_{Glove}$ as an example.

Figure 7. (a) Co−occurrence probability matrix of labels. (b) Cosine similarity matrix.

3.2.3. Label Correlation Emotion Classifier

Under the dual constraints of co-occurrence probability matrix A and cosine similarity matrix C, the GCN aggregates each label node to update its representation, thus effectively capturing inter-label association information. This approach results in an object classifier capable of learning the correlations between emotion labels.

The input to the GCN includes the word-embedding vectors of all nodes ${\mathbb{R}}^{m\times \omega }$, the label co-occurrence probability matrix A, and the label cosine similarity matrix C. By stacking multiple GCN layers, the network is able to learn and model the complex relationships between nodes, as described in Equation (17):

$${\left({\mathbb{R}}^{m\times \omega }\right)}^{l+1}=\sigma \left({\displaystyle \frac{A{\left({\mathbb{R}}^{m\times \omega }\right)}^l{W}_a^l+C{\left({\mathbb{R}}^{m\times \omega }\right)}^l{W}_c^l}{2}}\right)$$

(17)

l represents the l-th layer of the GCN, while $W_a^l$ and $W_c^l$ are the weight matrices that need to be learned. Each layer takes the input from the previous layer and updates the output according to Equation (17).

The final output of each GCN node is a classifier $h_i$ for the corresponding emotion label, as shown in Equation (18):

$$H={\left\{h_i\right\}}_{i=1}^m={\left({\mathbb{R}}^{m\times \omega }\right)}^L,h_i={\left(R_i^{\omega }\right)}^L$$

(18)

The final emotion prediction score is denoted as $\widehat{y}$, as shown in Equation (19):

$$\left\{\begin{array}{c}\widehat{y}={\eta }_1{F}_{core}{H}^T+{\eta }_2{F}_{back}{H}^T+{\eta }_3{F}_{mix}{H}^T\hfill \\ {\eta }_1+{\eta }_2+{\eta }_3=1\hfill \end{array}\right.$$

(19)

where $F_{core}$, $F_{back}$, and $F_{mix}$ denote the features of the body’s core region, background features, and fused features mentioned earlier, respectively. ${\eta }_i$ represents the trainable coefficient, which indicates the relative importances of $F_{core}$, $F_{back}$, and $F_{mix}$. Initially, the values were assigned as ${\eta }_1=0.333$, ${\eta }_2=0.333$, and ${\eta }_3=0.333$. After several epochs of iteration, the optimal predictive performance of the model was achieved when ${\eta }_1=0.3$, ${\eta }_2=0.2$, and ${\eta }_3=0.5$. $H^T$ denotes the transpose operation of H. Both feature-level fusion and decision-level fusion are employed for the final emotion identification.

4. Experimental Section

4.1. Dataset

The Emotic dataset [11] was employed in the experiment due to its comprehensive annotations across 26 different emotion categories. The images in this dataset feature complex backgrounds that encompass diverse environments, times, locations, camera viewpoints, and lighting conditions. Therefore, the proposed method demonstrates generalizability across different environments and emotion categories. These inherent characteristics provide the dataset with remarkable diversity, which made the experimental task significantly more challenging.

The dataset was divided into four subsets: Ade20k, Emodb-small, Framesdb, and Mscoco. The sample distribution within the dataset was imbalanced, with varying quantities for each type of emotion. Positive emotions were more prevalent, while negative emotions were less represented. During the experiments, the dataset was partitioned into training, validation, and test sets, as shown in Table 4.

Table 4. Dataset statistics.

Data Distribution	-	Quantity
Sub-dataset distribution	Ade20k	432
	Emodb-small	1374
	Framesdb	4869
	Mscoco	16,510
Sample distribution	Affection	1063
	Anger	209
	Annoyance	368
	Anticipation	5335
	Aversion	168
	Confidence	4059
	Disapproval	659
	Disconnection	325
	Disquietment	1462
	Doubt/Confusion	479
	Embarrassment	152
	Engagement	12,814
	Esteem	851
	Excitment	4394
	Fatigue	538
	Fear	177
	Happiness	5630
	Pain	188
	Peace	1691
	Pleasure	2103
	Sadness	405
	Sensitivity	360
	Suffering	260
	Surprise	417
	Sympathy	718
	Yearning	668
Data partition	Training	23,265
	Validation	3314
	Test	7202

4.2. Loss Function and Evaluation Metrics

Due to the imbalanced sample distribution in the Emotic dataset, we utilized the multi-label focal loss [24] as the loss function in this study, which is expressed as Equation (20):

$$MF{L}_{\alpha ,\gamma }\left(y,\widehat{y}\right)=-\sum _{i=1}^{26}[\alpha {\left(1-{\widehat{y}}_i\right)}^{\gamma }{y}_{i}log\left({\widehat{y}}_i\right)+\left(1-\alpha \right){\left({\widehat{y}}_i\right)}^{\gamma }\left(1-y_i\right)log\left(1-{\widehat{y}}_i\right)]$$

(20)

where ${\widehat{y}}_i$ represents the predicted label of the i-th category, $y_i$ represents the true label of the i-th category, and $\alpha$ and $\gamma$ are two hyperparameters. The balance factor $\alpha$ effectively balances the overall contributions of the positive and negative samples, while the focusing parameter $\gamma$ adjusts the weights of the easy and hard samples. The optimal recognition performance of the model was achieved when $\alpha =0.5$ and $\gamma =0.3$ through multiple experiments.

The evaluation metrics were the average precision (AP), mean average precision (mAP), Jaccard coefficient (JC), total parameters (Params), and floating point operations (FLOPs). mAP and JC are defined as follows in Equations (21) and (22):

$$mAP={\displaystyle \frac{{\sum }_{i=1}^{m}A{P}_i}{m}}$$

(21)

$$JC={\displaystyle \frac{\left|y\cap \widehat{y}\right|}{\left|y\cup \widehat{y}\right|}}$$

(22)

The AP is an approximation of the area under the precision–recall curve and assesses the model’s average precision at all recall levels. In Equation (21), i represents different emotion categories, and m represents the total number of emotion categories. mAP is a global metric that evaluates the average performance of the model across all categories. In Equation (22), y represents the true labels, $\widehat{y}$ represents the predicted results, and the JC measures the degree of overlap between the predicted results and the true labels. The model’s predictive performance value ranged between 0 and 1, with a higher value indicating better performance.

4.3. Comparative Experiments

The methods proposed in this paper were compared with several existing approaches, namely, ML-KNN [13], ML-RBF [14], Kosti et al. [11], Lee et al. [22], Zhang et al. [23], ML-GCN [28], ESRs [8], Ilyes et al. [24], Q2L [15], LGLM [29], RCL-Net [9], ML-Decoder [17], SAML [27], FLNet [30], BHARAT [10], and LFPLM [17]. The emotion precision of these methods is shown in Table A1 in Appendix A. The comparison results of the mAP and JC are shown in Table 5 and Figure 8. The results indicate that the methods proposed in this paper achieved the best performance in terms of the mAP and JC, with the mAP exceeding the state-of-the-art by 0.732% and the JC by 0.007, and there was no significant increase in the model total parameters and FLOPs.

Table 5. The mAP and JC of different methods. The bold parts represent the optimal values of the model.

Experiment	mAP (%)	JC	Params (M)	FLOPs (G)
ML-KNN [13] (2007)	32.531	0.400	11.704	10.240
ML-RBF [14] (2011)	23.665	0.399	11.926	10.470
Kosti et al. [11] (2019)	27.384	0.349	44.820	7.803
Lee et al. [22] (2019)	20.587	0.269	16.010	9.207
Zhang et al. [23] (2019)	27.030	0.353	30.582	9.553
ML-GCN [28] (2019)	35.245	0.372	46.960	8.410
ESRs [8] (2020)	27.602	0.355	20.330	9.436
Ilyes et al. [24] (2020)	28.332	0.360	39.400	23.400
Q2L [15] (2021)	34.243	0.416	18.013	8.310
LGLM [29] (2022)	29.494	0.275	20.956	8.414
RCL-Net [9] (2023)	23.061	0.295	54.881	9.410
ML-Decoder [16] (2023)	35.204	0.386	35.024	8.511
SAML [27] (2023)	34.490	0.443	33.614	6.660
FLNet [30] (2023)	29.026	0.365	35.750	7.440
BHARAT [10] (2024)	33.549	0.418	37.769	3.216
LFPLM [17] (2024)	31.205	0.402	12.810	4.326
Ours	35.977	0.450	33.500	5.500

Figure 8. JCs of different methods [8,9,10,11,13,14,15,16,17,22,23,24,27,28,29,30].

4.4. Ablation Experiments

4.4.1. Ablation Experiments of FB-ER and ML-ERC

To validate the effectiveness of FB-ER and ML-ERC, three ablation experiments were conducted. In the first experiment, only FB-ER was retained, without the inclusion of ML-ERC. In the second experiment, only ML-ERC was used and FB-ER was omitted. In the third experiment, both FB-ER and ML-ERC were utilized. The results of these experiments are presented in Figure 9. The findings demonstrate that both components of the proposed method contributed significantly to its overall effectiveness.

Figure 9. Ablation results of FB-ER and ML-ERC.

4.4.2. Ablation Experiments of FB-ER

Four experiments were conducted using different combinations of branches while keeping all other experimental parameters constant. In the first experiment, only the body information was utilized. The second experiment combined the body information with the background information. The third experiment integrated the body information with the foreground information, and the fourth experiment fused the body information with both the foreground and background information. The resulting average precision is depicted in Figure 10, while the mAP and JC values are shown in Table 6. It can be concluded that the mAP improved by approximately 8.9% and the JC increased by 0.126 when the fore-background information was incorporated.

Figure 10. APs of different branch combinations.

Table 6. Ablation results of mAP and JC on FB-ER model. The bold parts represent the optimal values of the model.

Ablation Types	Experiment	mAP (%)	JC
Different combinations of branches	Body	27.056	0.324
	Body + background	32.238	0.370
	Body + foreground	30.712	0.362
	Body + fore-background	35.977	0.450
Different fusion strategies	Feature-level fusion	33.577	0.416
	Decision-level fusion	31.066	0.364
	Mixed-level fusion	35.977	0.450
With or without CR-Unit	Without CR-Unit	34.150	0.420
	With CR-Unit	35.977	0.450

The ablation experiments with different fusion strategies were conducted to evaluate their effectiveness. The first experiment utilized feature-level fusion, the second employed decision-level fusion, and the third implemented mixed-level fusion. The results for the mAP and JC are presented in Table 6. It can be concluded that compared with feature-level fusion and decision-level fusion, mixed-level fusion improved the mAPs by approximately 2.3% and 4.9%, respectively, and the JCs by approximately 0.034 and 0.066, respectively.

To test the performance of the CR-Unit applied to body features, two ablation experiments were conducted. In the first experiment, the CR-Unit was not incorporated, whereas in the second experiment, the CR-Unit was incorporated. The results for the mAP and JC are presented in Table 6. It can be seen that incorporating the CR-Unit led to an improvement of 1.8% in the mAP and 0.03 in the JC.

4.4.3. Ablation Experiments of ML-ERC

Matrices A and C were applied in the multi-label emotion recognition classifier (ML-ERC) as introduced in Section 3.2.1 and Section 3.2.2. To evaluate their effects, four ablation experiments were conducted. In the first experiment, a traditional threshold decision classifier was used. In the second experiment, only the label co-occurrence probability matrix A was used. In the third experiment, only the cosine similarity matrix C was used. In the fourth experiment, a combination of the label co-occurrence probability matrix A and the cosine similarity matrix C was used. The category precision results are shown in Figure 11, and the mAP and JC results are presented in Table 7. The results demonstrate that the utilization of ML-ERC led to an mAP improvement of 3.5% and an increase in the JC by 0.082.

Figure 11. Ablation results of AP on ML-ERC.

Table 7. Ablation results of mAP and JC on ML-ERC model. The bold parts represent the optimal values of the model.

Ablation Types	Experiment	mAP (%)	JC
Different combinations of matrices	Traditional threshold decision classifier	32.517	0.378
	A	35.367	0.425
	C	33.136	0.386
	$A+C$	35.977	0.450
Different word embedding vectors	GloVe	35.976	0.450
	Word2Vec	35.957	0.450
	Fasttext	35.977	0.447
	Elmo	35.975	0.449

This study investigated the impacts of four distinct word-embedding vectors (GloVe [37], Word2Vec [38], FastText [39], and ELMo [40]) on the performance of the model. The experimental results are presented in Table 7. Based on these results, it can be observed that the mAP and the JC obtained using the four different word-embedding vectors were approximately 35.9% and 0.450, respectively.

4.5. Parameter Experiments

To explore the impact of different thresholds $\tau$ on model performance, the experimental results are shown in Table 8. According to the experiments, the highest mAP and JC were observed when $\tau =0.2$. However, when $\tau =0.1$, the mAP and JC did not reach their optimal state, possibly because the $\tau$ value was set too low, which introduced weakly correlated data and caused interference. Additionally, when $\tau \ge 0.5$, both the mAP and JC decreased, possibly because the $\tau$ value was set too high, which resulted in the exclusion of some relevant information.

Table 8. The impact of different τ values on the mAP. The bold parts represent the optimal values of the model.

$\mathit{\tau }$	A	mAP (%)	JC Graph	JC
$\tau =0.1$		33.16		0.422
$\tau =0.2$		35.977		0.450
$\tau =0.3$		35.96		0.413
$\tau =0.4$		33.85		0.415
$\tau =0.5$		32.31		0.401
$\tau =0.6$		32.39		0.384
$\tau =\left\{\begin{array}{c}\hfill 0.7\hfill \\ \hfill 0.8\hfill \\ \hfill 0.9\hfill \end{array}\right.$		31.98		0.384

4.6. Experimental Visualization Results

The visual results of the ablation experiments for different branch combinations are presented in Figure 12. The visualization in Figure 12 demonstrates that the FB-ER model significantly outperformed the baseline in emotion recognition. It accurately predicted a substantial number of emotional categories, with only a minimal number of incorrect predictions and unpredicted categories.

Figure 12. Visual display of different branch ablation experiments.

Figure 13 shows the visual results of the ablation experiments on the ML-ERC model. Based on the visualization in Figure 13, it could be concluded that the application of the ML-ERC model led to an increase in correct predictions, while the number of incorrect predictions and unpredicted emotional categories decreased.

Figure 13. Visual display of ML-ERC ablation experiment.

5. Conclusions

This paper proposes a method of multi-label visual emotion recognition that fuses fore-background features to address the following issues: multi-label visual emotion recognition overlooks the influence of fore-background features and fails to exploit the correlations between different emotion labels. First, the fore-background-aware emotion recognition model (FB-ER) is proposed to extract human, background, and foreground features. Then, it generates combined features using CR-Unit and hybrid level fusion. Second, multi-label emotion recognition classifier (ML-ERC) is proposed to construct an emotion label graph using the co-occurrence probability matrix and cosine similarity matrix to represent the edges and word-embedding vectors as nodes. The model employs a graph convolutional network to learn the correlations between emotions and obtain a classifier containing the labeled correlations. Finally, visual features are combined with the classifier to achieve multi-label recognition for 26 emotions. To validate this method, detailed ablation and comparative experiments were conducted. The results showed that compared with the state-of-the-art methods, the mAP increased by 0.732% and the JC increased by 0.007.

However, there were still some problems that need to be improved in further studies:

(1) The proposed method exhibited relatively low precision in recognizing difficult-to-identify emotions, such as embarrassment, confusion, and sensitivity.

(2) The use of a static approach to construct the correlation matrix reduced the model’s generalization ability. Future work will consider employing dynamic methods to calculate emotional correlations.