Scene-Speaker Emotion Aware Network: Dual Network Strategy for Conversational Emotion Recognition

Abstract

Incorporating external knowledge has been shown to improve emotion understanding in dialogues by enriching contextual information, such as character motivations, psychological states, and causal relations between events. Filtering and categorizing this information can significantly enhance model performance. In this paper, we present an innovative Emotion Recognition in Conversation (ERC) framework, called the Scene-Speaker Emotion Awareness Network (SSEAN), which employs a dual-strategy modeling approach. SSEAN uniquely incorporates external commonsense knowledge describing speaker states into multimodal inputs. Using parallel recurrent networks to separately capture scene-level and speaker-level emotions, the model effectively reduces the accumulation of redundant information within the speaker’s emotional space. Additionally, we introduce an attention-based dynamic screening module to enhance the quality of integrated external commonsense knowledge through three levels: (1) speaker-listener-aware input structuring, (2) role-based segmentation, and (3) context-guided attention refinement. Experiments show that SSEAN outperforms existing state-of-the-art models on two well-adopted benchmark datasets in both single-text modality and multimodal settings.

1. Introduction

Emotion recognition has been a prominent research area in natural language processing over time [1]. With the rapid proliferation of social media and online platforms, an unprecedented volume of conversational data has become available for computational analysis [2]. This has fueled increasing interest in Emotion Recognition in Conversation (ERC), which is essential for various downstream applications, such as emotion-driven chatbots [3], automated customer service [4,5], and sentiment analysis on social media platforms [6,7].

Recent ERC studies have explored various strategies for modeling speaker-specific emotional states. Some employ recurrence-based methods for speaker modeling [8,9,10,11], utilizing separate recurrent networks to distinguish between speakers and listeners. Others introduce Graph Convolutional Networks (GCNs) [12] to represent utterances and speaker relationships [13,14,15,16,17] by effectively capturing internal emotional inertia and speaker interactions. However, to capture causal information in context, these methods require extracting utterance-level features early on, which include the identity information of different speakers. This shifts the model’s focus away from the emotional dynamics of the specific speaker and, thus, persisting in using these features may lead to confusion.

In addition, some other ERC methods enhance emotional modeling by integrating external knowledge to better address complex speaker-related factors, such as psychological states and speaking motives [18,19]. Notably, classic methods such as COSMIC [20] and TodKat [21] utilize structured commonsense knowledge bases like ConceptNet [22] and ATOMIC [23], or automatically constructed knowledge graphs based on them (e.g., COMET [24]), to enhance emotional reasoning. Results demonstrate that these knowledge-integrated approaches significantly improve the model’s ability to capture nuanced emotional dynamics and uncover implicit speaker intentions, thereby enhancing overall emotion understanding in dialogue.

While a few recent methods attempt to extract additional commonsense knowledge from the inherent knowledge of large language models (LLMs) [25] or reasoning networks [26], the majority still rely on structured commonsense knowledge bases as their primary external source. These knowledge graphs typically offer nine types of relational information. To determine the most effective combination, existing methods often evaluate model performance under various configurations. However, the selected relation types vary considerably across studies. For instance, some methods, such as MKIN-MCL [27], focus on six specific relation types, while CDEA [28] and LECM [29] select six relations based on subject–object roles in event semantics. In contrast, other methods prioritize fewer but more relevant relations—COFFEE [30] selects two based on their correlation with emotion labels, and TG-ERC [31] utilizes three associated with psychological states. However, this trial-and-error process is inefficient and overlooks the potential unreliability of generated external knowledge, often noisy or factually incorrect, which can introduce harmful bias into the model. While some methods aim to suppress unnecessary commonsense knowledge, for instance, the CKCL method [32] leverages contrastive learning to reduce reliance on external knowledge when it aligns with contextual predictions, they fail to address cases where unreliable knowledge misleads otherwise accurate predictions. No existing approach effectively filters and aligns external commonsense knowledge at the utterance level, which undermines its ability to serve as a positive and contextually appropriate supplement.

To improve the efficiency of leveraging multimodal information and external commonsense knowledge in emotion recognition in conversation (ERC), two key challenges must be addressed: (i) Modeling a speaker’s emotional state based on global utterance-level features may introduce noise from mixed speaker identity information, especially in multi-speaker dialogues; (ii) External commonsense knowledge can be unreliable and noisy. There is a lack of effective and universally applicable filtering mechanisms to ensure its relevance and accuracy.

We propose the Scene-Speaker Emotion Aware Network (SSEAN), which employs dual parallel recurrent networks to model both global context and individual speaker emotions. This approach categorizes input information to capture global-level and speaker-specific emotions independently. In multi-turn dialogues, this helps the model capture the continuity and correlation of each speaker’s emotional state across turn transitions. It also enhances the model’s focus on the utterance features that are truly relevant to emotion by alleviating the noise introduced by abrupt speaker identity changes. Furthermore, we introduce a novel dynamic screening module to enhance commonsense knowledge across three levels, including: (1) Structuring compound single- and dual-sentence inputs based on speaker continuity to model speaker-listener dynamics, (2) segmenting and organizing generated commonsense knowledge into speaker and listener paragraphs at the output level, and (3) dynamically filtering role-specific commonsense features using a context-guided attention mechanism. Experiments on the IEMOCAP and MELD datasets demonstrate that our model achieves, and in some cases surpasses, SOTA-level performance in both single-text modality and multimodal settings, underscoring the effectiveness of our approach.

2. Related Work

ERC Methods Focused on Speaker Modeling: The interactive conversational memory network (ICON) [8] pioneered the use of distinct memory networks to handle the interactions between speakers in dyad ic dialogues. The model first utilizes distinct gated recurrent unit (GRU) modules to capture speaker-specific contextual representations for each utterance, which are then integrated through global context modeling. Dialoguernn [9] also added two new GRUs to differentiate the impact of new utterances on speakers and listeners, enabling the model to extend to multi-party dialogues. Inspired by this, our model also employs GRUs to update the emotional states of different speakers. In DialogueGCN [14], directed graph network structures were introduced into dialogue emotion recognition to better model the interactions between speakers and the emotional inertia within individual speakers. They further modeled the speaker-level context by establishing a graph structure of adjacent utterances to the target utterance. I-GCN [13] designed two GCNs to process semantic information at the utterance level and relationship information at the speaker level, respectively, and used an incremental graph structure to capture temporal change information. Concurrently, the Directed Acyclic Graph Network for Conversational Emotion Recognition (DAG-ERC) [17] utilized speaker information and utterance position information to construct a directed acyclic graph neural network to model the dialogue context, enhancing the model’s ability to capture long and short-term sequential information. To address the limitation of recurrent networks in simultaneously modeling dialogue structure and speaker information due to their sequential nature, we employ parallel recurrent networks to capture these two types of information separately.

ERC Methods Focused on Multimodality: Many other approaches focus on multimodal fusion, making full use of effective information across different modalities through the comprehensive application of cross-attention mechanisms and feature decoupling. The multimodal Dynamic Fusion Network (MM-DFN) [16] employs a novel graph-based dynamic fusion module to capture the dynamics of contextual information across different semantic spaces, significantly advancing the state of multimodal emotion recognition in conversations. CFN-ESA [11] incorporates a cross-modal fusion network with emotion-shift awareness, utilizing the textual modality as the primary source. It employs a novel cross-modal encoder module to fully extract complementary and associative information from multimodal data. Li et al. (2022) [10] made improvements in the feature extraction approach. To ensure that features extracted from each modality are more focused on emotional information, they proposed the Emoformer module for extracting emotion vectors to capture the subtle changes in emotions across different modalities, achieving significant performance improvements on two benchmark datasets. Our model draws on the method of extracting the emotional tendencies of each modality using variants of the transformer encoder.

ERC Methods Focused on Commonsense Knowledge: Emotion recognition enhanced by external knowledge mainly relies on two well-established commonsense knowledge bases. The first is ConceptNet [22], which captures commonsense concepts and relationships as a semantic network, covering various aspects of everyday life. The second is ATOMIC [23], centered on events rather than entities, achieving human-competitive results in If-Then reasoning tasks. Building on ATOMIC and ConceptNet, COMET [24], which can automatically construct knowledge graphs, demonstrates the potential to understand and predict emotions, laying a foundation for incorporating commonsense knowledge into emotion recognition. COSMIC [20] and TokDat [21] leverage the COMET model to incorporate commonsense knowledge, thereby enhancing performance on dialogue emotion recognition. COSMIC [20] is a model that uses commonsense knowledge to model various hidden emotional influence factors in conversations, significantly improving the identification of complex emotions. Inspired by COSMIC, we also incorporate commonsense knowledge to model the emotional states of speakers and the complex influences among interlocutors. TokDat [21] combined a topic-augmented language model with commonsense statements, introducing them into a Transformer-based emotion detection model, achieving excellent accuracy. To acquire interpretable and relatively high-quality commonsense knowledge tailored for ERC tasks, we likewise adopt COMET as our external knowledge source. CKCL [32] is a contrastive learning framework designed to determine whether external knowledge is necessary for understanding utterance emotions, thereby avoiding the blind incorporation of knowledge that could hinder model training. It generates pseudo-labels based on the consistency between the original model prediction and the predictions obtained by masking either the context or the knowledge. However, this approach overlooks the fact that, even when the predictions are inconsistent, the incorporated commonsense knowledge may still act as noise and negatively affect the model.

Existing commonsense knowledge in ERC is typically centered around the speaker as the subject, inherently relying on speaker-specific information and requiring intra-speaker sequential modeling. Meanwhile, the generated commonsense knowledge may contradict the fact, which may introduce noise and hinder understanding of speaker’s actual state. However, existing ERC methods rarely filter such knowledge effectively. Our proposed model employs parallel recurrent networks to separately capture scene-level and speaker-level emotional information, enabling better use of commonsense knowledge for speaker modeling and emotional interaction understanding, while preserving global context comprehension. Additionally, a multi-stage attention-based filtering module is introduced to improve the quality of generated commonsense knowledge and identify potential noise.

3. The Proposed Method

We propose the Scene-Speaker Emotion Aware Network (SSEAN), a unified framework for emotion recognition in conversation that jointly models global context and speaker-specific emotional dynamics. The SSEAN employs dual parallel recurrent networks to separately capture global scene-level context and individual speaker emotions, effectively preserving emotional continuity across multi-turn dialogues while reducing noise from abrupt speaker identity shifts. To further enhance reasoning with external knowledge, we design a three-level dynamic commonsense screening module that improves knowledge quality and relevance. The overall architecture of our model is illustrated in Figure 1.

3.1. Problem Definition

Formally, given a conversation C consists of a series of utterances $C=[u_1,u_2\dots ,u_N]$, $u_i=\{u_i^t,u_i^a,u_i^v\}$ where N is the number of utterances in the conversation, and $u_i$ denotes the $i^{\mathrm{th}}$ utterance in the conversation, which contains the representations of two modalities $u_i^t$ (text), $u_i^a$ (audio) and $u_i^v$ (visual). For the conversation C, speakers $P=\{p_1,p_2,\dots ,p_M\}$ participate in the conversation, where M is the number of participants, and a function $p_j=S\left(u_i\right),$$i\in \{1,2,\dots .,N\},j\in \{1,2,\dots ,M\}$ is defined for obtaining the speaker is defined for obtaining the speaker $p_j$ of the utterance $u_i$. The objective of emotion recognition in conversation is to accurately predict the emotion label $y_i$ for each utterance $u_i$ in the given conversation C from a predefined set of emotion labels $Y=[y_1,y_2\dots ,y_k]$, where k is the number of labels.

3.2. Single Modality Feature Extraction

3.2.1. Raw Feature Extraction

First, this study utilizes pre-trained models and tools to extract raw features from individual modalities.

Textual Features

The RoBERTa Large model [33] is utilized for extracting textual representations at the utterance level. BPE tokenized utterances are fed into the model, where the encoder module of RoBERTa is employed for feature extraction, and the decoder module is omitted. In alignment with COSMIC [20], the outputs of the last four hidden layers are averaged to enrich the features with maximal information, resulting in raw text features for enhanced context modeling.

Audio Feature

Following several previous studies [9,34], this paper uses the standard sets ComParE 2016 from the OpenSMILE [35] as a profile for the initial processing of the audio data. ComParE 2016 is the feature set required by The INTERSPEECH 2016 ComParE Challenge, which contains 6373 static features obtained by computing various functions on LLD and is suitable for a variety of downstream tasks including emotion recognition. In this paper, given that the feature dimensions directly output by OpenSMILE are relatively large, a fully connected layer is utilized to reduce the dimensionality of the features, yielding condensed audio raw features.

Visual Feature

Similar to the audio modality, to ensure a fair comparison, we also adopt the approach used in previous studies and employ a 3D-CNN for visual feature extraction. 3D-CNN extracts facial features by leveraging 3D convolutional layers and 3D pooling layers, capturing information across both spatial and temporal dimensions, and is commonly used for facial expression recognition [36]. This is highly relevant to conversational emotion recognition.

After this step, we represent the features of the utterance $u_i$ as $U_i=[U_i^t,U_i^a,U_i^v]$.

3.2.2. Emotion Vector Extraction

To enhance the Scene-Speaker Emotion Aware Network’s emphasis on the continuity and variation of emotional information while minimizing the impact of emotion-irrelevant noise, this study draws inspiration from EmoCaps [10]. It incorporates a structure similar to the Transformer encoder for processing each modality, facilitating the nuanced extraction of emotional features, which has been demonstrated to be feasible in EmoCaps. In the Emotion Vector extraction block, as illustrated in Figure 2, two encoder modules were modified and merged, with the subsequent feed-forward network being replaced by a multi-layer perceptron. Since the self-attention mechanism has a good ability to capture global information from a long sequence, this paper uses it to obtain utterance-level emotion information further. At last, the block aggregates the emotion information through the multilayer perceptron to reduce the feature dimensions and obtain more representative unimodal utterance-level emotion vectors. Since the input single-utterance raw features are unrelated to the dialogue-level context, the emotional vectors extracted at this stage are also independent of the dialogue-level context.

For a given utterance $u_i$, the process of computing the utterance-level emotion vector $E_i$ through the Emotion Vector extraction block can be expressed as follows:

$$\begin{array}{c}{A_1}_i^x=LayerNorm(U_i^x+softmax({\displaystyle \frac{U_i^x{W}_q{U_i^x}^T{W}_k^T}{\sqrt{d}}})U_i^x{W}_v)\\ F_i^x=LayerNorm(U_i^x+RELU(A_i^{x}W+b))\end{array}$$

(1)

where $U{i}^x,x\in \{t,a\}$ is the text or audio component of $U_i$, $W_q,W_k,W_v$, W is the learnable parameter matrix, and b is the bias parameter. For each modality component, we stack two identical self-attention–residual–LayerNorm blocks. and then fed into a multilayer perceptron:

We simplify the feedforward network, which usually consists of two fully connected layers with ReLU activation functions, into one layer, retaining the nonlinear transformation capability while using the multilayer perceptron as a substitution to further improve the nonlinear fitting capability and reducing the feature dimension.

Finally, we concatenation the utterance-level emotion vector $E_i=[E_i^t,E_i^a,E_i^v]$ with the raw feature $U_i=[U_i^t,U_i^a,U_i^v]$ based on modality to obtain the final representation of the new discourse $u_i$ unimodal feature:

$$\begin{array}{c}\hfill T{F}_i=E_i^t\oplus U_i^t\\ \hfill A{F}_i=E_i^a\oplus U_i^a\\ \hfill V{F}_i=E_i^v\oplus U_i^v\end{array}$$

(2)

3.3. Dynamic Screening of Commonsense Knowledge

For a given utterance, we take it as input and use COMET [24] trained on ATOMIC, a knowledge generation model, as the only source to acquire the corresponding commonsense knowledge related to the speaker’s emotion state. ATOMIC is an event-centered knowledge graph that allows for the execution of the corresponding inference task based on the 9 if-then relation types identified as (i) xIntent, (ii) xNeed, (iii) xAttr, (iv) xEffect, (v) xWanted, (vi) xReact, (vii) oEffect, (viii) oWant, and (ix) oReact [23].

Referring to existing work that enhances ERC with commonsense knowledge [20,21], this paper excludes the relation types xNeed, xWant and oWant because they are predictions of character actions before and after the event. Whereas in the dialogue dataset, considering that each dialogue lasts for a shorter period, we do not assume that more actions take place during the conversation. Yet, there is still a controversy about the role of the remaining part of the relationship types for sentiment recognition, existing work [20,21] have experimentally sifted the relation types used in the model species. It has been observed that incorporating additional relation types into the model results in a decline in model performance. However, even within the same dialogue, the applicability of relation types to utterances can vary. For some utterances, all relation categories can provide valid commonsense knowledge, while for other utterances, only some of the relation categories may be able to provide valid commonsense knowledge. Thus, the manual selection method can not make the best use of commonsense knowledge. In this paper, we use the remaining six relational categories for our experiments. The usage of relationship types in related work is shown in

Table 1. Relation types used in related work.

Related Work	xInt	xRea	xAtt	xEff	oRea	oEff
COSMIC	✓	✓		✓	✓	✓
TODKAT	✓	✓			✓
SSEAN	✓	✓	✓	✓	✓	✓

.

In addition, as COMET [37] is built upon a pre-trained GPT-2 model, the knowledge it generates tends to be diverse and includes multiple plausible alternatives. Additionally, the generated commonsense knowledge is inevitably speculative, based on current circumstances, and its reliability requires validation by subsequent factual developments. Directly using raw dialogue utterances as input for COMET further exacerbates these challenges, introducing additional issues that compromise the quality and relevance of the generated knowledge. These speculative, low-quality pieces of commonsense knowledge should not be directly adopted by the model, as they could potentially mislead it.

Therefore, to ensure that the model acquires sufficient and reliable commonsense knowledge, we propose a three-tier dynamic filtering module to refine and enhance the quality of commonsense knowledge. The main structure is shown in Figure 3.

First, most previous studies directly use dialogue utterances as input, which presents two key problems. The first issue is that COMET’s training data consists of descriptive statements with subject–verb–object structures, whereas in dialogues, the speaker and listener are often omitted from the utterances. The second issue arises from the segmentation criterion of dialogue datasets, which is typically based on punctuation marks such as periods. As a result, some utterances contain limited information, and directly inputting them into COMET may generate meaningless or even ambiguous knowledge.

To address these issues at the input level, we adopt a strategy that combines sentence completion with both single- and dual-sentence inputs. Specifically, for each utterance, we complete the missing subjects and verbs following the format of COMET’s training data. Additionally, when a change in speaker occurs, we use the listener of the next utterance as the object and incorporate the subsequent utterance as a response to further enhance the input, thereby forming a dual-sentence input structure. As shown in Algorithm 1.

For each of the 6 relation types containing potentially valid information, the top k most plausible pieces of knowledge are generated in text form as candidates.

At the output level, we implement an initial filtering step. Our experiments reveal that the generated outputs often contain meaningless words or symbols, such as “none”, “.”, “ ”, “y”, “x”, and “n/a”. We first eliminate such outputs and, for utterances that fail to produce meaningful commonsense knowledge, we apply padding using [pad] as a fallback mechanism. After that, by adding descriptive sentence components or subjects, we integrate all candidate knowledge into two general knowledge paragraphs according to subject differences. The detailed methodology is provided in Algorithm 2. The final filtered knowledge output from these steps is treated as input features for our end-to-end trainable model. Therefore, the inclusion of these algorithms does not break the end-to-end trainability of the SSEAN architecture.

To further ensure the reliability of commonsense knowledge across different subjects, we apply an additional filtering and guidance process to the generated commonsense knowledge paragraphs. Specifically, for the commonsense knowledge extracted from the current discourse, we identify the current discourse (for the speaker’s knowledge paragraph) and the subsequent discourse (for the listener’s knowledge paragraph) as valid facts. Commonsense knowledge that exhibits greater similarity to these valid facts is considered more reliable.

Algorithm 1: Speaker-listener-aware input structuring.

Algorithm 2: Generate speaker and listener paragraphs.

Using the same encoder as for the raw textual modality, we obtain utterance features $f_s,f_l$ (current speaker and next-utterance listener) and the corresponding candidate knowledge features $C_s,C_l$ from COMET. For each role $n\in \{s,l\}$, queries, keys and values are computed as

$$Q_n=f_n{W}_q^n,K_n=C_n{W}_k^n,V_n=C_n{W}_v^n,$$

where $W_q^n,W_k^n,W_v^n$ are learnable parameter matrices. The most informative and reliable knowledge representation is obtained through a single cross-attention layer followed by a position-wise feed-forward network:

$$C{F}_i^n=FFN\left(softmax\left(\frac{Q_n{K}_n^{\top }}{\sqrt{d}}\right)V_n\right),n\in \{s,l\}.$$

where the FNN is a two-layer feedforward network containing a ReLU activation layer. Since the attention mechanism is used as a filter here, no residual structure is added. The obtained output $C{F}_i^s,C{F}_i^l$ is the speaker commonsense feature vector and listener commonsense feature vector for the given discourse $u_i$.

3.4. Dual-Strategy Framework

In this paper, we propose the Scene-Speaker Emotion Aware Network (SSEAN), a dialogue emotion recognition framework that employs dual-strategy parallel modeling to distinguish between the global conversational context and speaker-specific context, enabling the simultaneous utilization of multimodal and multi-source information.

Conversational Emotion recognition differs from general emotion recognition tasks in that it is difficult to make correct judgments about emotion by focusing only on utterance-level features. A significant amount of information is contained within the dialogue-level context and the multi-turn interactions among speakers. The components that can reflect the emotions of the utterances require selection and extraction through effective modeling methods.

Fundamentally, based on contextual relevance, we categorize emotional information within dialogues into two mutually exclusive types. The first type exhibits contextual relevance at the global dialogue level but loses this relevance within the same speaker, which involves the textual modality of utterances. Conversely, the second type includes commonsense knowledge about speaker states and the audio modality of utterances, showing contextual relevance within the same speaker while containing a lot of redundant speaker information at the global dialogue level. In this paper, this type involves textual modality of utterances and commonsense knowledge related to the speaker’s state. To minimize the introduction of redundant information for these two types, we adopt different modeling strategies. We propose a parallel structure designed to capture the dialogue-level context, the emotional states of speakers, and multi-turn interactions between speakers independently. This architecture ensures that information about each dialogue participant remains distinct, minimizing redundant speaker-related data. The specific structure is shown in Figure 1.

A significant portion of information in dialogue is often embedded within long-term dependencies. Therefore, global-level contextual relationships can help the model better comprehend the overall progression and state of events throughout the conversation. By maintaining and updating dialogue history across multiple turns, the model gains a deeper understanding of the emotional tone underlying the conversation. Consequently, we refer to the features extracted based on global contextual information as global vectors. The Scene Emotion Vector is utilized to aid the model in understanding the continuity of emotions between adjacent utterances, such as being consistently neutral or negative throughout a particular paragraph. In this paper, we use a Bi-directional Long Short-Term Memory (Bi-LSTM) network [38] to model the global conversational context and extract the Scene Emotion Vector for each utterance from both video and textual modality features.

$$\begin{array}{c}{E}_i^{scene}=LSTM(T{F}_i\oplus V{F}_i^s)\end{array}$$

(3)

Throughout the process, emotional information can naturally be categorized according to the participants of the conversation. For the same event, the identity of the conversation participants might have a significant impact on the emotion of the utterances, which is particularly evident in multi-participant dialogues. Thus, we refer to the features extracted based on the state of conversation participants as Speaker Emotion Vectors. The update mechanism of the Speaker Scene Vector is utilized to help the model understand the inertia of emotions within the same speaker and the emotional interactions triggered between different speakers by the utterances. Due to the model’s structure, information centered on different conversation participants remains independent, preventing cross-interaction. This design minimizes the introduction of speaker-related redundant information and mitigates its negative impact on model performance. To ensure this, we automatically assign independent GRU networks [39] to each speaker based on speaker labels, allowing the model to update the emotional states of individual speakers and obtain a Speaker Emotion Vector for each utterance.

$$\begin{array}{c}{E}_i^{speaker}=e_i^{S\left(u_i\right)}={GRU}_s(e_{i-1}^{S\left(u_i\right)},(A{F}_i\oplus C{F}_i^s))\end{array}$$

(4)

In addition with the help of commonsense knowledge, we have also modeled the interaction between different speakers. Carrying on from the work in the previous section, we extracted the commonsense knowledge features by obtaining two commonsense knowledge feature vectors $C{F}_i^s,C{F}_i^l$ whose subjects are the speaker and the listener of utterance $u_i$, respectively, where $C{F}_i^l$ contains the current speaker’s influence on the listener’s emotion state, which is used to update the listener’s emotion state.

This maintenance of the listener’s state effectively captures the interaction dynamics between speakers in each turn.This mechanism also increases the window in which the model understands changes in emotion, i.e., the emotion of each utterance is judged jointly by information from at least two utterances, and can increase the probability of correct classification when the sentiment state changes.

For the current utterance, we select the GRU network corresponding to the next speaker to update the listener’s emotion state.

$$\begin{array}{c}{e}_i^{S\left(u_{i+1}\right)}=GR{U}_s(e_{i-1}^{S\left(u_{i+1}\right)},C{F}_i^l)\end{array}$$

(5)

The listener here is the chivalrous listener, precisely defined as the speaker of the next time step, expressed as the hearer for ease of understanding. Since the listener knowledge paragraph uses the next utterance as a fact, to ensure the validity of the commonsense knowledge paragraph, when a dialogue consists of more than two participants, the listener’s commonsense knowledge feature is only used for updating the status of the speaker of the next utterance, and the status of the other non-current speakers remains unchanged.

$$\begin{array}{c}{e}_i^{p_j}=e_{i-1}^{p_j},p_j\ne S\left(u_{i+1}\right)\wedge j\in \{1,2,\dots ,M\}\end{array}$$

(6)

Ultimately, we add the Scene Emotion Vector and the Speaker Emotion Vector of the same utterance together and input the result into the softmax layer after going through a linear layer to obtain the final emotion classification of each utterance.

$$\begin{array}{c}{p}_i=softmax(W_{smax}(E_i^{scene}+E_i^{speaker})+b_{smax})\\ \widehat{y_i}=\underset{k}{argmax}\left(p_i\left[k\right]\right)\end{array}$$

(7)

4. Results and Discussion

4.1. Datasets

Our experiments were conducted on two benchmark datasets for conversation emotion recognition tasks: MELD [40] and IEMOCAP [41].These datasets are widely recognized in the community for their diversity in emotional expression, multi-speaker dialogue structures, and rich multimodal annotations (e.g., text, audio, and video), making them well-suited for evaluating models’ ability to understand emotional dynamics in realistic conversational settings.

IEMOCAP consists of dialogue videos performed by ten actors in pairs, making up five sessions in total, including both audio and textual modalities. It comprises 7433 utterances across 151 dialogues, with each utterance labeled with an emotion. The emotions are categorized into six classes: happiness, sadness, neutral, anger, excitement, and frustration. Since IEMOCAP does not have a predefined split for training/validation/testing, to ensure fairness in subsequent comparisons, we follow previous studies [20] by training on the first four sessions and testing on the last session.

MELD features multi-person dialogue videos from the TV show “Friends”, also including corresponding audio and video modalities. It includes 1433 dialogues and 13,708 utterances, with each utterance assigned an emotion label. The labels categorize the emotions into seven classes: anger, disgust, sadness, joy, surprise, fear, or neutral. We adhere to the predefined training/validation/testing split in MELD to maintain the fairness of our experimental results.

The dataset partitioning is summarized in

Table 2. Statistics of datasets.

Dataset	Dialogue			Utterances
	Train	Valid	Test	Train	Valid	Test
IEMOCAP	108	12	31	5163	647	1623
MELD	1038	114	280	9989	1109	2610

, and the corresponding label distribution is illustrated in Figure 4.

4.2. Training Setup

We leverage both audio, visual and textual modalities in the MELD and IEMOCAP datasets. The specific dimensionality of different modalities is shown in

Table 3. Feature dimensionality for different modalities.

Modality	MELD	IEMOCAP
Textual	600	100
Audio	300	100
Visual	300	100
Commonsense Knowledge	600	100

.

The common settings for both datasets include 30 training epochs, a batch size of 64, the Adam optimizer [42], L2 regularization weight $\lambda$ of 0.001, and a dropout rate of 0.2. For the MELD dataset, the learning rate is set to 0.0001, and for the IEMOCAP dataset, it is set to 0.0003.

Both the IEMOCAP and MELD datasets exhibit severe class imbalance issues, as shown in Figure 4.

To address class imbalance, we employ Focal Loss [43] during training. By introducing a focal factor $\gamma$, Focal Loss directs the model to focus on difficult, misclassified examples and prevents over-representation of majority classes in the loss function. For both datasets, $\gamma$ is set to 2.0; since the balancing parameter $\alpha$ only applies to modifying binary classification loss weights, we use the transformed class weights $\widehat{weight}$ as a substitute.

The initial class weight for class i is computed as:

$$\begin{array}{c}{\mathrm{weight}}_i={\displaystyle \frac{1}{log({\mathrm{freq}}_i+ϵ)}}\end{array}$$

(8)

To stabilize training, we apply logarithmic compression and linear scaling to the class weights, which suppresses extreme values while preserving minority class contributions:

$$\begin{array}{c}\mathrm{weight}’={\displaystyle \frac{log(1+\mathrm{weight})}{max(log(1+\mathrm{weight}\left)\right)}}\\ \widehat{weight}={\displaystyle \frac{1}{2}}\times weigh{t}^{\prime }+{\displaystyle \frac{1}{2}}\end{array}$$

(9)

4.3. Evaluation Metrics

Similarly, due to the class imbalance in the datasets, we adopt weighted Average accuracy (WA-Acc) and weighted Average F1-score (WA-F1) as the evaluation metrics for overall model performance. Additionally, for a more detailed assessment and analysis, we compute the F1-score for each individual class separately.

The formula for computing the F1-score for a single classi is:

$$F{1}_i={\displaystyle \frac{2\times {\mathrm{Precision}}_i\times {\mathrm{Recall}}_i}{{\mathrm{Precision}}_i+{\mathrm{Recall}}_i}}$$

(10)

$${\mathrm{Precision}}_i={\displaystyle \frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FP}}_i}}$$

(11)

$${\mathrm{Recall}}_i={\displaystyle \frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FN}}_i}}$$

(12)

The formula for computing accuracy (ACC) for a single classi is as follows:

$${\mathrm{Acc}}_i={\displaystyle \frac{{\mathrm{TP}}_i+{\mathrm{TN}}_i}{{\mathrm{TP}}_i+{\mathrm{TN}}_i+{\mathrm{FP}}_i+{\mathrm{FN}}_i}}$$

(13)

Among them, ${\mathrm{TP}}_i,{\mathrm{TN}}_i,{\mathrm{FP}}_i,{\mathrm{FN}}_i$ represent the True Positives, True Negatives, False Positives, and False Negatives of the predicted class, respectively.

The formulas for weighted average accuracy and weighted average F1-score are as followed:

$$\mathrm{WA}-\mathrm{Acc}=\sum _{i=1}^N{w}_i\times {\mathrm{Acc}}_i$$

(14)

$$\mathrm{WA}-\mathrm{F}1=\sum _{i=1}^N{w}_i\times F{1}_i$$

(15)

4.4. Comparison with Other SOTA Methods

4.4.1. SSEAN-Uni

There are currently few attempts to enhance multimodal models with commonsense knowledge, and these approaches lack representativeness. Therefore, to ensure a fair evaluation while verifying the effectiveness of our proposed model, particularly the three-level commonsense knowledge dynamic filtering module, we first compare our model (SSEAN-Uni, with its structure shown in Figure 5) with other state-of-the-art (SOTA) models using only the text modality enhanced with commonsense knowledge. The results are presented in

Table 4. The overall F1 scores of ERC on the IEMOCAP and MELD datasets.

Models	IEMOCAP	MELD
	WA-F1	WA-F1
COSMIC [20]	65.28	65.21
CauAIN [44]	67.61	65.46
TodKat [21]	61.33	65.47
SKAIG [45]	66.96	65.18
EmotionIC [46]	69.44	66.32
MKFM [47]	68.88	65.66
InstrucERC [48]	71.39	69.27
SSEAN-Uni	72.12	66.17

Bold values indicate the best performance on the corresponding dataset.

.

To verify the stability and robustness of the model, we trained it using five different seeds and computed the mean, standard deviation, and Coefficient of Variation (CV) of the results, shown in

Table 5. Experimental results with different random seeds.

SEED	IEMOCAP	MELD
0	72.44	66.03
42	71.88	65.97
100	72.21	66.12
1000	72.03	66.38
4027	72.12	66.17
mean	72.13	66.13
standard deviation	0.209	0.158
Coefficient of Variation	0.289%	0.239%

. A CV value of less than 1 indicates that the model’s results are stable.

SSEAN-Uni achieved superior F1 performance compared to all state-of-the-art (SOTA) models on the IEMOCAP dataset, attaining a score of 72.12, which achieves the best performance among all baseline models. On the MELD dataset, SSEAN-Uni also outperformed most models, ranking just slightly behind EmotionIC and InstrucERC. The performance gap with InstrucERC is minimal, and through seed adjustments, SSEAN-Uni can achieve results (66.38%) exceeding EmotionIC. However, compared to InstrucERC, there remains a more significant performance gap. Since InstrucERC is a generative multi-task framework based on large language models (LLMs), we hypothesize that the complexity of emotional states in the MELD dataset poses a challenge for SSEAN-Uni, given its model size and the length constraints of single inputs. This limitation may hinder its ability to fully capture long-term emotional dependencies. Nevertheless, SSEAN-Uni offers a substantial advantage in terms of computational efficiency.

4.4.2. SSEAN-Multi

In

Table 6. The F1 results of ERC on the IEMOCAP dataset.

Models	Happy	Sad	Neutral	Angry	Excited	Frustrated	WA-F1	WA-Acc
DialogueRNN [9]	32.2	80.26	57.89	62.82	73.87	59.76	62.89	63.52
Emocaps [10]	71.91	85.06	64.48	68.99	78.41	66.78	71.77	-
DialogueGCN [14]	42.75	84.54	63.54	64.19	63.08	66.99	64.108	65.25
MMGCN [49]	51.57	80.48	57.69	53.95	72.81	57.33	62.89	63.22
MM-DFN [16]	42.22	78.98	66.42	69.77	75.56	66.33	68.18	68.21
BiF-BiAGRU [50]	54.50	72.70	59.40	61.00	66.60	61.60	63.00	62.80
SSEAN-Multi	73.72	87.10	69.09	68.39	79.78	68.54	73.94	73.91

Bold values indicate the best performance on the corresponding dataset.

and

Table 7. The F1 results of ERC on the MELD dataset.

Models	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Angry	WA-F1	WA-Acc
DialogueRNN [9]	76.97	47.69	-	20.41	50.92	-	45.52	57.66	60.31
Emocaps [10]	77.12	63.19	3.03	42.52	57.50	7.69	57.54	64.00	-
DialogueGCN [14]	72.10	41.70	2.80	21.80	44.20	6.70	36.50	54.70	54.90
MMGCN [49]	76.97	47.69	-	20.41	50.92	-	45.52	57.66	60.31
MM-DFN [16]	77.76	50.69	8.00	38.50	54.70	11.80	43.50	60.80	60.80
UniF-BiAGRU [50]	76.40	49.70	11.50	27.00	52.40	14.00	39.40	58.10	60.30
SSEAN-Multi	80.02	58.80	27.27	41.40	64.40	36.07	52.28	66.43	67.04

Bold values indicate the best performance on the corresponding dataset.

, we showcase the performance comparison of the SSEAN-Multi model with other state-of-the-art (SOTA) multimodal models on the emotion recognition in conversation (ERC) task on the IEMOCAP and MELD datasets. The experiment results demonstrate that SSEAN achieves the latest performance benchmarks on both datasets.

On the IEMOCAP dataset, the SSEAN model significantly outperforms all other models, obtaining the highest Weighted Average F1 Score (WA-F1) of 73.95% and an accuracy of 73.96%. This performance is notably robust across all categories except “Angry”, indicating the model’s robustness in recognizing complex emotional states. Particularly, the SSEAN model shows a significant improvement in the “Happy” category, with notable improvements in “Sad” and “Excited” categories as well, highlighting SSEAN’s capability to discern subtle emotional expressions. This outstanding performance can be attributed to SSEAN’s comprehensive contextual and speaker-level modeling, enabling it to more accurately capture emotional dynamics.

In the MELD dataset, SSEAN once again sets a new benchmark, with a WA-F1 score of 66.43% and an accuracy of 67.04%. It exhibits significant improvements across all categories. Notably, SSEAN’s effective utilization of multimodal information and dynamic filtering mechanisms to extract and leverage relevant emotional cues is evidenced by its performance in the “Fear” and “Disgust” categories, which are traditionally challenging to model due to their subtle expressions and dependence on the speaker’s state information.

SSEAN’s performance enhancement stems from the effective utilization of a vast amount of information. The overall leading results affirm the efficacy of our model structure in handling information from multiple sources and modalities and reducing the accumulation of redundant information.

4.5. Ablation Study and Analysis

4.5.1. SSEAN-Uni

To further validate our model’s proficient performance in processing commonsense knowledge and effectively avoiding the introduction of redundant information, we conducted extensive ablation experiments.

The ablation studies shown in

Table 8. The F1 results of Ablation Study.

	IEMOCAP		MELD
	WA-F1	Accuracy	WA-F1	Accuracy
w/o CK dynamic screening module	70.69 $(\downarrow 1.43)$	70.57 $(\downarrow 1.66)$	62.04 $(\downarrow 4.13)$	63.35 $(\downarrow 3.07)$
w/o speaker identity modeling	71.48 $(\downarrow 0.64)$	71.26 $(\downarrow 0.97)$	64.69 $(\downarrow 1.48)$	65.41 $(\downarrow 1.01)$
w/o Speaker Emotion vector	69.82 $(\downarrow 2.3)$	69.53 $(\downarrow 2.7)$	63.22 $(\downarrow 2.95)$	64.03 $(\downarrow 2.39)$
w/o Global Emotion vector	35.40 $(\downarrow 36.72)$	37.79 $(\downarrow 34.44)$	53.24 $(\downarrow 12.93)$	56.82 $(\downarrow 9.6)$
SSEAN	72.12	72.23	66.12	66.56

Bold values indicate the best performance on the corresponding dataset. The ↓ indicate performance decrease compared to the full model.

assessed the impact of components—such as the audio modality, commonsense knowledge, and speaker/scene emotion vectors—on the performance of SSEAN on the IEMOCAP and MELD datasets, highlighting the importance of each component for achieving state-of-the-art emotion recognition in conversation (ERC).

Notably, the most dramatic performance drop is observed when the Global Emotion vector is removed, with WA-F1 scores plummeting by 36.72% and 12.93% on IEMOCAP and MELD, respectively. This decline accentuates the critical role of the fundamental vector in capturing the core emotional tendencies across conversations, underlining its importance in the model’s framework. This finding aligns with conclusions drawn from many other studies.

Specifically, removing the CK dynamic screening module led to a 1.43% drop in WA-F1 on IEMOCAP and a 4.13% drop on MELD. This decrease is comparable to or even greater than the impact of not using commonsense knowledge, emphasizing the three-level dynamic screening module’s crucial role in filtering relevant knowledge and reducing noise in the commonsense knowledge base, thus, ensuring the quality of knowledge integration.

Excluding speaker identity modeling also resulted in significant performance drops, particularly on the multi-party MELD dataset (1.48%). This underscores the necessity of capturing speaker-specific emotional states and interactions to avoid the introduction of redundant or irrelevant information due to speaker changes in multi-turn dialogues.

4.5.2. SSEAN-Muti

We investigated the impact of integrating commonsense knowledge with different modality combinations on model performance, as shown in

Table 9. Performance on IEMOCAP and MELD datasets.

Methods	IEMOCAP		MELD
	WA-F1	WA-Acc	WA-F1	WA-Acc
Text (w CF)	72.12	72.23	66.17	66.42
Text (w CF) + Visual	73.36 ↑	73.32 ↑	65.68	65.54
Text (w CF) + Audio	73.14 ↑	73.19 ↑	65.85	66.73 ↑
Text (w CF) + Audio + Visual	73.94↑	73.91↑	66.43↑	67.04 ↑

Bold values indicate the best performance on the corresponding dataset. The ↑ indicates performance improvement compared to the text-only modality.

. As expected, the tri-modal configuration achieved the best performance compared to bi-modal settings.

For the single-text modality model enhanced with commonsense knowledge, on the IEMOCAP dataset, adding any additional modality consistently improved the model’s WA-F1 and WA-Acc. Similarly, on the MELD dataset, incorporating any new modality led to improvements in WA-Acc. This demonstrates that our Dual-Strategy Framework effectively utilizes multimodal information while mitigating the interference caused by redundant information across different modalities.

In the MELD dataset, although the model’s overall performance slightly declined when only the textual and visual modalities were incorporated, a closer analysis of individual emotion categories (shown in Appendix B) reveals that the visual modality contributes to higher F1 scores in categories such as Neutral, Joy, and Angry.

4.6. Case Study

Commonsense knowledge occasionally incorporates extraneous information that may not align with the actual facts, thereby impeding the training process of models. However, dynamic filtering mechanisms have the capability to mitigate this impact by selectively filtering out such irrelevant information.

Through existing research, we have identified the superior ability of commonsense knowledge in facilitating the transition of emotions in conversations and the classification of similar emotional categories. However, this introduces some challenges. Figure 6 presents a study case from the MELD dataset, where the conversation is entirely neutral. For traditional models that classify emotions solely based on text, this does not pose much of a challenge. However, models incorporating commonsense knowledge often make errors in such scenarios. As shown in Figure 6, the commonsense knowledge generated from the third utterance includes a listener’s reaction that does not match reality. This kind of commonsense knowledge introduces sorrowful emotional information into subsequent judgments, leading to misclassification of the utterance’s emotion. Through the dynamic filtering module for commonsense knowledge, such unrealistic commonsense information is filtered out, thus, removing sorrowful emotional information and avoiding classification errors.

5. Conclusions

In this paper, we proposed the Scene-Speaker Emotion Aware Network (SSEAN) to address key challenges in emotion recognition in conversation (ERC). We introduced a dual-strategy framework that effectively models both global conversational context and speaker-specific emotional dynamics by leveraging two parallel recurrent networks. Our model captures long-term dependencies in dialogue while mitigating the interference of speaker identity information on emotional representation. Furthermore, we designed a three-level dynamic filtering module to refine and enhance the utilization of commonsense knowledge, improving its reliability and effectiveness in ERC tasks.

Our experiment results on the IEMOCAP and MELD datasets demonstrate that SSEAN achieves state-of-the-art (SOTA) performance in both single-text modality and multimodal settings. Further analysis confirms the effectiveness of multimodal integration and commonsense knowledge enhancement in ERC. Additionally, our commonsense knowledge filtering strategy significantly reduced noise from unreliable external knowledge, allowing SSEAN to make more accurate emotion predictions specifically in those classes with few samples. The model’s stability and robustness were further validated through multiple training runs with different random seeds, showing low variance in performance, which underscores the statistical significance of our results.

While our method improves the quality and relevance of integrated commonsense knowledge, it remains limited in modeling long-range knowledge that requires reasoning across multiple dialogue turns. Future work may involve leveraging large language models (LLMs) to enable extended context understanding and more advanced commonsense reasoning.