Hierarchical Cross-Modal Interaction and Fusion Network Enhanced with Self-Distillation for Emotion Recognition in Conversations

Abstract

Emotion recognition in conversations (ERC), which aims to capture the dynamic changes in emotions during conversations, has recently attracted a huge amount of attention due to its importance in providing engaging and empathetic services. Considering that it is difficult for unimodal ERC approaches to capture emotional shifts in conversations, multimodal ERC research is on the rise. However, this still suffers from the following limitations: (1) failing to fully explore richer multimodal interactions and fusion; (2) failing to dynamically model speaker-dependent context in conversations; and (3) failing to employ model-agnostic techniques to eliminate semantic gaps among different modalities. Therefore, we propose a novel hierarchical cross-modal interaction and fusion network enhanced with self-distillation (HCIFN-SD) for ERC. Specifically, HCIFN-SD first proposes three different mask strategies for extracting speaker-dependent cross-modal conversational context based on the enhanced GRU module. Then, the graph-attention-based multimodal fusion (MF-GAT) module constructs three directed graphs for representing different modality spaces, implements in-depth cross-modal interactions for propagating conversational context, and designs a new GNN layer to address over-smoothing. Finally, self-distillation is employed to transfer knowledge from both hard and soft labels to supervise the training process of each student classifier for eliminating semantic gaps between different modalities and improving the representation quality of multimodal fusion. Extensive experimental results on IEMOCAP and MELD demonstrate that HCIFN-SD is superior to the mainstream state-of-the-art baselines by a significant margin.

1. Introduction

Emotion pervades daily human communication [1]. Thus, it is of great significance to automatically understand and recognize human emotions to support related decision-making for providing engaging and empathetic services. As a crucial branch under emotion recognition, emotion recognition in conversations (ERC) has recently attracted a huge amount of attention from both academia and industry due to its promising applications, such as public opinion monitoring [2], emotion-aware dialogue generation [3], intelligent healthcare assistance [4], and so forth. ERC plays an increasingly important role in building an emotion-aware human–computer interaction environment, which is expected to generate appropriate emotional responses during conversations. The task of ERC is to capture the dynamic changes in emotions during conversations based on human communication signals such as audio and text and accurately identify the emotional status of each utterance [5].

Earlier ERC studies have usually relied on a single modality [6,7], which results in relatively poor performance, robustness, and generalization of the ERC system. For instance, if the relevant modality cannot be collected or is of poor quality in some cases, the ERC system cannot work. In addition, it is quite difficult for a single modality to provide enough emotional clues to accurately capture emotional shifts in conversations [5,8], which leads to unsatisfactory emotion recognition performance in unimodal ERC systems. It is known that humans express their emotions through multiple forms such as text, speech, and facial expression in daily communication and interpret emotions of other speakers based on the history of utterances as shown in Figure 1. It indicates that different modalities complement each other in emotion expression and recognition, and capturing the contextual information in conversations is highly critical for accurate emotion identification. Thus, recently, research that leverages multimodal information to achieve ERC is on the rise [8,9,10,11], but many multimodal ERC approaches usually generate multimodal information based on the concatenation strategy without considering cross-modality interaction [9,10,11]. Due to the data heterogeneity of different modalities, how to effectively perform multimodal interaction and fusion for supporting the extraction of comprehensive conversational context to achieve satisfactory emotion recognition performance remains a challenging research topic.

Through a literature review, it is found that existing multimodal ERC research mainly depends on sequence-based algorithms or/and graph neural networks for capturing conversational context. However, they typically fail to fully explore richer multimodal interactions and fusion. They either perform cross-modal interactions and fusion before feeding into sequence-based algorithms or graphs or extract cross-modal complementary information within graphs. For instance, IMAN [12] generates multimodal fusion representations based on the concatenation of cross-modal interactions and, then, employs the gated recurrent units (GRUs) for modeling conversational context. SMFNM [5] initializes the representations of nodes in a graph based on the multimodal fusion representations. In MMGCN [13], which aims to explore cross-modal interactions during information propagation, each node is connected with the nodes which correspond to the same utterance but from different modalities. MM-DFN [14] employs graph convolution operation to explore both intra- and inter-modality interactions. It is known that cross-modal interactions not only are used for simple multimodal fusion but also run through the entire conversational context. Therefore, this study attempts to explore hierarchical cross-modal interaction and fusion.

In addition, many existing ERC approaches cannot work well on modeling speaker-dependent context in conversations. For instance, bc-LSTM [15], MMMU-BA [16], and AGHMN [17] employ sequence-based algorithms to learn contextual information from the neighboring utterances; thus, they can be considered as speaker-independent ERC approaches. Due to the importance of speaker information, speaker-embedding-based approaches, such as Ctnet [18] and MMGCN [13], have been developed for enhancing the performance of emotion recognition in conversations. However, the obtained speaker embedding is static information and ignores the complementary information from other modalities [13,18]. Considering that the speaker dependencies in conversations, including intra-speaker and inter-speaker dependencies, are dynamic and complicated, it is highly crucial to dynamically model speaker-dependent context in ERC. In recent years, some graph-based approaches (such as DialogueGCN [6], MMGCN [13], LR-GCN [19], and GraphCFC [20] have attempted to capture speaker-sensitive contextual information through the pre-defined relation types of edges, but they need to construct fully connected graphs for capturing speaker-dependent context under this explicit declaration of edge relation, which inevitably results in a heavy computation load. Drawing inspiration from the existing research, this work tries to explore speaker-dependent cross-modal interactions at different stages for extracting high-quality conversational context.

Moreover, recent ERC approaches mainly implement geometric manipulation (such as matrix computation) in the feature spaces to learn unified multimodal representations, but they fail to consider the semantic gaps between different modalities during the cross-modal interaction [5,13,21,22]. It may bring the risk of misalignment and further influence the final emotion prediction performance. Thus, how to learn complementary information with more similar semantics from other modalities to address the issue of the semantic gap between different modalities for enhancing the performance of emotion identification is also a great challenge.

To address the above-mentioned issues, we propose a novel multimodal framework for emotion recognition in conversations, called “hierarchical cross-modal interaction and fusion network enhanced with self-distillation (HCIFN-SD)”. In HCIFN-SD, the speaker-dependent context modeling is efficiently integrated with cross-modal interactions via an improved GRU structure for the purpose of better capturing multi-view cross-modal contextual information in conversations. Then, the obtained representations are utilized for constructing three directed graphs to further explore speaker-dependent multimodal interaction and fusion. Finally, the self-distillation mechanism is employed to eliminate the semantic gaps between different modalities for achieving more accurate emotion recognition. The overall structure of HCIFN-SD is illustrated in Figure 2.

To demonstrate the effectiveness of our proposed HCIFN-SD, we conduct extensive experiments on two popular benchmark datasets (i.e., IEMOCAP [23] and MELD [24]). Experimental results show that our HCIFN-SD is superior to existing baselines and sets new state-of-the-art records on ERC tasks. The main contributions of this paper can be summarized as follows:

(1)

We propose a novel multimodal framework for conversational emotion recognition called HCIFN-SD. It effectively integrates hierarchical cross-modal interaction and a fusion network with self-distillation for learning high-quality multimodal contextual representations, filling the gaps in existing approaches for ERC.

(2)

The new GRU structure named MCI-GRU is designed, which not only retains the speaker-dependent cross-modal long-distance conversational context, but also captures local cross-modal conversational context from recent neighboring utterances. The proposed MF-GAT module constructs three directed graphs representing different modality views, which aims to capture both the long-distance conversational context and temporal complementary information from other modalities. In addition, the self-distillation is introduced to minimize the semantic gaps between different modalities for achieving more satisfactory emotion prediction performance.

(3)

Experimental results demonstrate that our proposed HCIFN-SD outperforms the existing mainstream ERC models and achieves new state-of-the-art records on the benchmark datasets. Furthermore, extensive ablation studies can systemically demonstrate the importance and the rationality of each component in HCIFN-SD.

The rest of this paper is organized as follows: The most relevant works about emotion recognition in conversations, multimodal interaction and fusion, and graph neural networks are briefly reviewed in Section 2. The proposed methodology is introduced in Section 3. Section 4 presents the experimental data and setup. The experimental results and related discussion are reported in Section 5. Finally, Section 6 concludes this paper and outlines future work.

2. Related Work

2.1. Emotion Recognition in Conversations

Emotion recognition is an interdisciplinary research field, involving cognitive science, psychology, pattern recognition, artificial intelligence, and so on [25]. ERC, as a crucial branch under emotion recognition, has recently attracted increasing research interest due to its critical role in developing an empathetic human–computer interaction environment. Compared with single-sentence emotion recognition, ERC is more complicated because human emotion is heavily influenced by multiple parties during the conversation process [26]. The influence includes emotion inertia and emotion contagion [11]. As shown in Figure 1, when getting the message that the man will leave, the woman’s emotion is pushed into the frustrated state (u₅), which denotes emotion contagion. And the woman maintains the frustrated emotion state In the following multiple utterances (u₇ and u₈), which is emotion inertia. It means although one party tends to maintain his/her initial emotion during the conversation (i.e., intra-speaker influence), his/her emotional state may also be transferred into a new state due to interactions with the other speakers (i.e., inter-speaker influence). In summary, human emotion in a conversation is largely powered by the intra-speaker and inter-speaker context dependencies of the whole conversation. However, it is also found that many existing ERC approaches cannot work well on modeling intra- and inter-speaker dependencies in conversations [12,13,16,17], which results in relatively poor emotion recognition performance. Therefore, this study proposes the speaker-dependent framework for capturing intra- and inter-speaker context dependencies.

Based on the adopted methods for conversational context modeling, existing ERC research can be categorized into sequence-based approaches and graph-based approaches. Sequence-based Approaches: They usually employ sequence-based algorithms (such as recurrent neural network (RNNs), long short-term memory (LSTM), and gated recurrent units (GRUs)) to extract conversational context dependencies. For example, BC-LSTM [15] utilizes the bidirectional LSTM architecture to learn conversational context for emotion classification. DialogueRNN [6] employs two distinct GRUs to update context and the speaker state, respectively, and then, the updated speaker state is fed into the third GRU to update emotion representation for final emotion classification. HiGRU [27] obtains the individual utterance embedding based on the GRU encoder and then captures the contextual utterance embedding via another GRU encoder. CMN [9] generates multimodal features through concatenation and then employs two distinct GRUs to capture inter-speaker dependency relations. ICON [10] adopts multiple GRUs to learn both intra- and inter-speaker dependencies and uses memory networks to integrate contextual information for final emotion prediction. A-DMN [11] explores both intra- and inter-speaker dependencies in a conversation via RNNs for updating the memory for the current utterance. Different from the above approaches, COSMIC [28] not only utilizes GRUs for conversational context extraction but also integrates commonsense knowledge to interpret the speaker’s intension. To sum up, these sequence-based models typically use recent neighboring utterances for contextual information extraction; thus, it is difficult for them to maintain contextual information from long-distance utterances, but previous ERC research has demonstrated the significance of long-distance emotion dependency [6,18].

Graph-based Approaches: They typically build a graph for a conversation [7], where each node represents one utterance, and the edges denote dependencies among utterances. For instance, DialogueGCN [7] constructs one graph for each conversation and sets a fixed window size for learning contextual information. ConGCN [29] symbolizes the entire conversation as a large graph, where all utterances and all speakers are treated as graph nodes. MMGCN [13] belongs to a multimodal ERC model, which generates three nodes for each utterance to represent different modalities to learn context propagation under the same modality and then integrates the contextual information of different modalities through concatenation. Similar to MMGCN [13], MM-DFN [14] not only constructs three semantic spaces representing different modalities in a graph to learn complementary information among modalities, but also develops a dynamic fusion module to reduce the redundant information accumulated at each graph layer. In general, these graph-based models capture the contextual dependencies among utterances based on the graph structure, but they fail to consider the sequential information in conversations. It is intuitive to combine the advantages of sequence-based and graph-based approaches for capturing the conversational context from both local and long-distance utterances [5,21]. Therefore, this study attempts to combine the advantages of recurrence and graph structures for learning more comprehensive conversational context information.

2.2. Multimodal Interaction and Fusion

Earlier ERC studies mainly rely on unimodal information, especially the text modality. For example, DialogXL [30] employs the pre-trained language model (Xlnet) to extract utterance-level textual features for ERC. COMPM [31], which integrates the speaker’s pre-trained memory with the context model, is a text-based ERC framework. The aforementioned studies [6,7,17,19] also depend on only the textual modality to achieve ERC. However, a single modality is quite difficult to provide enough emotional clues to accurately capture emotional shifts in conversations [5,8], which leads to relatively poor recognition performance and robustness. It is known that humans express their emotions through multiple forms in daily communication, and different signals carry complementary information for emotion expression and recognition. Therefore, recent ERC research efforts [8,9,10,11,13,14] mainly focus on mining complementary information from multimodal signals to obtain high-quality emotional clues for improving emotion recognition performance.

Existing multimodal research can be broadly categorized into feature-level fusion, decision-level fusion, and model-level fusion [32]. Feature-level fusion, also called early fusion, usually generates multimodal representations through concatenation or addition strategies at the input level. For example, CMN [9], ICON [10], and A-DMN [11] obtain utterance-level multimodal representations via feature concatenation. But feature-level fusion cannot efficiently exploit the complementary relations among multiple modalities. Decision-level fusion, which is proposed based on the modality independence assumption, typically aggregates the unimodal decision values through simple voting, averaging, or weighted sum to generate the final decision. For instance, Sahoo and Routray [33] separately utilized speech and facial expressions to obtain the emotion recognition results of the individual modality, respectively, and then adopted the voting strategy at the decision level to achieve final emotion recognition. Nevertheless, this modality independence assumption is invalid in practice, and decision-level fusion cannot capture the cross-modal interactions between different modalities. To sum up, the above-mentioned fusion methods cannot fully exploit the complementary information from different modalities. Model-level fusion, as a trade-off between feature-level and decision-level fusion, has become a promising solution for multimodal information integration. For example, SMFNM [5], CTNet [18], and IMAN [12] employ the multi-head attention or transformer structure to implement cross-modal interactions for learning complementary information to enhance the representation ability of the original modality, and they all demonstrate the importance of learning cross-modal complementary information. Therefore, this study follows the model-level fusion idea to implement multi-stage multimodal fusion for dealing with the data heterogeneity of different modalities on ERC tasks.

2.3. Graph Neural Networks

With the rapid emergence of non-Euclidean data (such as social relations, interaction relationships, and knowledge relations) in various fields, researchers have begun to represent them as graphs, which consist of node, edge, and/or edge weights [34]. Due to the complexity of graph data, it is difficult for traditional neural networks to process them. Thus, graph neural networks (GNNs) have been proposed to meet the demand and have attracted increasing research interest for multiple applications (such as item recommendation [35], opinion detection [36], and knowledge graph construction [37]). GNNs update the node’s representation through iteratively aggregating and propagating neighboring information, followed by a graph pooling operation, to generate a graph representation. Graph convolutional networks (GCNs) [38], as a typical architecture of GNNs, can directly operate convolution on graphs to learn hierarchical representations and update the node’s features. GCNs can be classified into two streams: spectral and spatial. Spectral GCNs transform graph convolution into a frequency domain filtering operation based on the signal processing theory, while spatial GCNs implement graph convolution by information propagation, which is more similar to the convolutional operations in convolutional neural networks (CNNs). Compared with spectral GCNs, spatial GCNs have gained greater attention due to their efficiency and flexibility. In addition, an attention mechanism, as a resource allocation scheme, has also been introduced to GCNs to make the models focus on the task-related information, which is called a graph attention network (GAT) [39]. Multiple studies [19,20] have demonstrated the effectiveness of utilizing the attention mechanism to assign different weights to different neighboring nodes. Therefore, this study also follows the information propagation idea and employs the attention mechanism in the graph to extract multimodal conversational context.

3. Proposed Method

In this section, we first formalize the problem definition and then introduce the proposed HCIFN-SD in detail. The overall structure of HCIFN-SD is illustrated in Figure 2. It shows that our HCIFN-SD consists of three stages: (1) unimodal feature extraction, which aims to extract unimodal alignment features and mark the location information of the speakers; (2) speaker-dependent cross-modal context modeling, which proposes the MCI-GRU module to implement speaker-dependent cross-modal conversational context learning; (3) graph-attention-based multimodal context extraction and fusion, which aims to capture both the long-distance conversational context and temporal complementary information from other modalities based on three directed graphs representing different modality views. Finally, the prediction module introduces the self-distillation strategy to supervise the training of the MCI-GRU module for the purpose of eliminating the semantic gap between different modalities and obtaining better emotion prediction performance.

3.1. Problem Formulation

Assume there are M speakers $s_1,s_2,\dots ,s_M$ in a conversation $C$, where $C$ can be denoted as $C=\left\{u_1,u_2,u_3,\dots ,u_n\right\}$ in temporal sequence. Here, $u_i$ is the i-th utterance in conversation, and n is the total number of utterances. We also define a mapping function $\phi$ to specify the relation between the utterance and its corresponding speaker, i.e., utterance $u_i$ is uttered by speaker $s_{\phi \left(u_i\right)}$. Furthermore, in a multimodal ERC scenario, each utterance involves the textual and acoustic modalities, denoted as $u_i\in \left\{u_i^a,u_i^t\right\}$, where T and A denote the textual and acoustic modalities, respectively. Consequently, the conversation $C$ can be further represented as $C_t=\left\{u_1^T,u_2^T,u_3^T,\dots ,u_n^T\right\}$ and $C_a=\left\{u_1^A,u_2^A,u_3^A,\dots ,u_n^A\right\}$, given the defined emotional labels $Y=\left\{y_1,y_2,..,y_c\right\}$, where $c$ is the number of emotion categories in the conversational dataset. Our HCIFN-SD aims to predict the discrete emotion labels of the constituent utterances in the conversation $C$, i.e., the task of HCIFN-SD is to recognize the emotion state $y_i$ for the utterance $u_i$ based on the captured speaker-dependent multimodal multi-stage context information in conversations.

3.2. Unimodal Feature Extraction

For the textual modality, we employ the pre-trained language model DistilRoberta [40] for textual feature extraction. RoBERTa has 24 stacked transformer layers and 355 M parameters in total [40]. In order to reduce the computation load, speed up the training process, and maintain a certain performance of RoBERTa, knowledge distillation is leveraged during the pre-training phase to obtain the smaller and lighter DistilRoberta model. In addition, the sentence transformer is adopted to obtain utterance-level textual feature $f_i^T$ with 768 dimensions for the utterance $u_i$, which is a 768-dimensional vector. For the acoustic modality, the pre-trained Wav2vec2.0 [41] model is utilized to extract utterance-level acoustic features. In this study, we also fine-tune the Wav2vec2.0 model on the first three sessions of the IEMOCAP [23] dataset, generating a 768-dimensional acoustic feature $f_i^A$ for each utterance.

Then, we employ two distinct 1D convolutional layers to map the extracted unimodal features into a low-dimensional feature space, which can be described as follows:

$${\tilde{U}}_i^m=Conv1D(f_i^m,k_m)$$

(1)

where $m\in \left\{A,T\right\}$, $k_m$ is set to be 1 in this study, denoting the convolutional filter size, and ${\tilde{U}}_i^m\in {\mathbb{R}}^d$, here $d$ represents the hidden layer size and $d=200$.

Meanwhile, inspired by MMGCN [13], we also extract speaker embeddings to represent the identity information of the specific speaker, which can be computed as follows:

$$S_{spk}=SpkEmbedding(S,D)$$

(2)

where S is the set of speakers, and D is the number of speakers in the current conversation. $S_{spk}\in {\mathbb{R}}^d$, which is a feature vector containing the speaker identity information. Then, the speaker embedding is attached to generate utterance-level unimodal representations with the speaker identify information, which can be described as follows:

$$X_i^m=\lambda S_{spk}+{\tilde{U}}_i^m$$

(3)

where $\lambda$ denotes the trade-off parameter, $X_i^m$ represents the unimodal representation enhanced with the speaker identity information, and $X_i^m\in {\mathbb{R}}^d$. Through this addition operation, the model can learn both who is speaking (captured by the speaker embedding) and what is being said (captured by the utterance-level unimodal representations).

3.3. Speaker-Dependent Cross-Modal Context Modeling

3.3.1. Speaker-Dependent Cross-Modal Interactions

In this subsection, the “Speaker-dependent Cross-modal Interactions (SCIs)” component is proposed for capturing both intra- and inter-speaker-dependent cross-modal interactions. Our SCI focuses on extracting speaker-dependent complementary emotion clues from another modality to enhance the representations of the current modality. For instance, when a higher pitch in the acoustic modality appears, the speaker-dependent cross-modal interaction (e.g., text $\to$ audio) can accurately capture this emotional shift in the conversation. However, how to effectively distinguish the different speaker for implementing intra-speaker and inter-speaker cross-modal dependencies is a challenge. It is known that the decoder architecture of the transformer introduces the concept of a mask to effectively control the information flow and block out irrelevant information during the process of attention computation through assigning a minus infinity score for the masked parts [42]. Inspired by this, our SCI component leverages the mask mechanism to distinguish different speakers for capturing intra-speaker and inter-speaker cross-modal dependencies in multi-party conversation scenarios.

Cross-modal Transformers: It follows the encoder architecture of the transformer but utilizes the masked multi-head attention rather than the standard multi-head attention and has a stack of three identical layers for reducing computational load. It indicates that each layer in our cross-modal transformer consists of a masked multi-head attention sublayer and a fully connected feedforward sublayer. The specific mask strategy will be introduced in detail later. In addition, the residual connection is also adopted around each sub-layer, followed by the layer normalization. Specifically, we consider two different modalities $\alpha$ and $\beta$ ($\alpha ,\beta \in \left\{T,A\right\}, \mathrm{a}\mathrm{n}\mathrm{d} \alpha \ne \beta$). The cross-modal interaction from modality $\alpha$ to $\beta$ can be denoted as $CT\left(X^{\beta },X^{\alpha }\right)$, which is mathematically described as follows:

$$CT\left(X^{\beta },X^{\alpha }\right)=Concat\left(hea{d}_1,\dots ,{head}_h\right)W_{CT}$$

(4)

$$hea{d}_i=Attention\left(Q_i^{\beta },K_i^{\alpha },V_i^{\alpha }\right)=soft\mathit{max}(\frac{Q_i^{\beta }(K_i^{\alpha }{)}^T}{\sqrt{d_h}})V_i^{\alpha }$$

(5)

where the query $Q_i^{\beta }=X^{\beta }(W_Q^{\beta }{)}_i$, key $K_i^{\alpha }=X^{\alpha }(W_K^{\alpha }{)}_i$, and $V_i^{\alpha }=X^{\alpha }(W_V^{\alpha }{)}_i$. Here, $(W_Q^{\beta }{)}_i\in {\mathbb{R}}^{d\times d_h}$,$(W_K^{\alpha }{)}_i\in {\mathbb{R}}^{d\times d_h}$, $(W_V^{\alpha }{)}_i\in {\mathbb{R}}^{d\times d_h}$, and $W_{CT}\in R^{h\ast d_h\times d}$are learnable weight matrices. Here, $CT\left(X^{\beta },X^{\alpha }\right)\in {\mathbb{R}}^d$ indicates extracting complementary information from the modality $\alpha$ to enhance the representation ability of the modality $\beta$, $h$ represents the number of heads in the attention mechanism, which means that the feature vector will be divided into $h$ heads for parallel calculation. Based on the above description, we can implement cross-modal interactions. Therefore, the following will introduce different mask strategies to achieve the identification of different cross-modal dependencies.

Intra-speaker Mask: We employ the intra-speaker mask strategy in the multi-head attention to identify the same speaker. It is essentially an attentive masking matrix as shown in Figure 3a, which can be denoted as follows:

$$m_{\mathit{int}ra}=\left\{\begin{array}{cc}-\infty & \mathrm{if} s_{\phi (u_i)}\ne s_{\phi (u_j)}\\ 0& otherwise\end{array}\right.,$$

(6)

where $s_{\phi (u_i)}$ is the mapping function from the utterance to its corresponding speaker. The masking operation needs to be implemented after the multiplication of the query and the key but before the Softmax calculation. Therefore, the intra-speaker mask strategy allows the multi-head attention to concentrate on the cross-modal interaction of the same speaker (i.e., emotion inertia), while blocking information from other speakers.

Inter-speaker Mask: On the contrary, the inter-speaker mask strategy treats other speakers as a unified identity (denoted as “other speakers”) to simplify the inter-speaker modeling process in multi-party conversation scenarios. The illustration of the inter-speaker mask is shown in Figure 3b, which can be described as follows:

$$m_{\mathit{int}er}=\left\{\begin{array}{cc}-\infty & \mathrm{if} s_{\phi (u_i)}=s_{\phi (u_j)}\\ 0& otherwise\end{array}\right.,$$

(7)

Therefore, utilizing the inter-speaker mask can effectively explore cross-modal interaction between different speakers, which means it only takes into account the emotion contagion from other speakers.

Local Information Mask: Given the great importance of local context information in conversational emotion recognition, we also design a local information mask strategy to behave as a context window for exploring the local context from neighboring utterances. The illustration of the local information mask is shown in Figure 3c, which can be described as follows:

$$m_{local}=\left\{\begin{array}{cc}0& if u_i\in \left\{u_i\left|i-p<i<i+p\right.\right\}\\ -\infty & otherwise\end{array}\right.,$$

(8)

where the hyperparameter $p$ denotes the size of the context window. Therefore, the local information mask can selectively focus on important local cross-modal interaction information.

To sum up, based on the different mask strategies, we can obtain multi-view cross-modal interactions, including the intra-speaker interaction $C{T}^{\mathit{int}ra}(X^{\beta },X^{\alpha })$, the inter-speaker interaction $C{T}^{inter}(X^{\beta },X^{\alpha })$, and the local interaction $C{T}^{local}(X^{\beta },X^{\alpha })$.

3.3.2. Multi-View Cross-Modal-Interaction-Enhanced GRU

Considering that the extracted intra-speaker and inter-speaker cross-modal interactions contain long-term context dependencies, we propose a “Multi-view Cross-modal-Interaction-Enhanced GRU (MCI-GRU)” module for learning both long-distance and local conversational context. The illustration of MCI-GRU is shown in Figure 4. Given that the reset gate in GRU specializes in learning local context dependencies in a sequence, we use $C{T}^{local}(X^{\beta },X^{\alpha })$ as the input of GRU for the purpose of leveraging the advantages of the reset gate to extract short-term context information. Meanwhile, due to the powerful ability of the update gate to preserve long-term context dependencies, we utilize both $C{T}^{\mathit{int}ra}(X^{\beta },X^{\alpha })$ and $C{T}^{inter}(X^{\beta },X^{\alpha })$ to participate in the calculation of the update gate in GRU for exploring speaker-dependent long-distance conversational context. This can be described as follows:

$$H_{\alpha \to \beta }=MCI-GRU(C{T}^{local}(X^{\beta },X^{\alpha }),C{T}^{\mathit{int}ra}(X^{\beta },X^{\alpha }),C{T}^{inter}(X^{\beta },X^{\alpha }))$$

(9)

where $\alpha ,\beta \in \left\{T,A\right\},\alpha \ne \beta$, and $H_{\alpha \to \beta }$ denotes the extracted speaker-dependent cross-modal conversational context.

Then, the calculation process of MCI-GRU will be introduced. The multi-view cross-modal interactions (e.g., from $\alpha$ to $\beta$) for the utterance $u_i$ include $C{T}^{local}(X^{\beta },X^{\alpha })$, $C{T}^{\mathit{int}ra}(X^{\beta },X^{\alpha })$, and $C{T}^{inter}(X^{\beta },X^{\alpha })$, respectively, which can be marked as $C{T}_i^{local}$, $C{T}_i^{intra}$, and $C{T}_i^{inter}$ in the subsequent equations in order to simplify the equation expression. Specifically, the computation of the reset gate can be described as follows:

$$H_{\alpha \to \beta }=MCI-GRU(C{T}^{local}(X^{\beta },X^{\alpha }),C{T}^{\mathit{int}ra}(X^{\beta },X^{\alpha }),C{T}^{inter}(X^{\beta },X^{\alpha }))$$

(10)

The update gate is computed based on the simultaneous leverage of $C{T}_i^{intra}$ and $C{T}_i^{inter}$ to extract speaker-dependent long-distance context dependencies, which can be formulated as follows:

$$r_i=\sigma (W_{rl}C{T}_i^{local}+W_{rh}{h}_{i-1}+W_{rs}C{T}_i^{\mathit{int}ra}+W_{rc}C{T}_i^{\mathit{int}er}+b_r)$$

(11)

$$z_i=\sigma (W_{zl}C{T}_i^{local}+W_{zh}{h}_{i-1}+W_{zs}C{T}_i^{\mathit{int}ra}+W_{zc}C{T}_i^{\mathit{int}er}+b_z)$$

(12)

$${\tilde{h}}_i=\mathit{tanh}(W_h(r_i\ast h_{i-1},C{T}_i^{local})+W_{hs}C{T}_i^{\mathit{int}ra}+W_{hc}C{T}_i^{\mathit{int}er})$$

(13)

$$h_i=(1-z_i)\ast h_{i-1}+z_i\ast {\tilde{h}}_i$$

(14)

where $W_{rl}$, $W_{rh}$, $W_{rs}$, $W_{rc}$, $W_{zl}$, $W_{zh}$, $W_{zs}$, $W_{zc}$, $W_h$, $W_{hs}$, $W_{hc}$, $b_r$, and $b_z$ are learnable parameters; $r_i$, $z_i$, ${\tilde{h}}_i$, and $h_i$ denote the output of the reset gate, the output of the update gate, the candidate hidden state, and the hidden state, respectively; $\sigma$ and $tanh$ denote the sigmoid and tanh activation functions, respectively; $\ast$ means the point-wise multiplication. Therefore, the obtained hidden state of MCI-GRU is the captured speaker-dependent cross-modal context representations, which can be denoted as $H_{\alpha \to \beta }$. Considering that $H_{\alpha \to \beta }$ indicates extracting complementary context information from the $\alpha$ modality to enhance the representation of the $\beta$ modality, $H_{\alpha \to \beta }$ is essentially the enhanced representation of the $\beta$ modality. Thus, $H_{A\to T}$ can be denoted as $H_T$, and the enhanced textual repreparation of the utterance $u_i$ can be marked as $h_i^T$. Similarly, $H_{T\to A}$ can be denoted as $H_A$, and the enhanced acoustic repreparation of the utterance $u_i$ can be marked as $h_i^A$.

In addition, we also generate a fusion representation ${\tilde{H}}_{fuse}$ based on the concatenation strategy at this stage for the purpose of supervising the subsequent multimodal fusion, which can be obtained as follows:

$${\tilde{H}}_{fuse}=Concat(C{T}_{A\to T}^{\mathit{int}ra},C{T}_{A\to T}^{\mathit{int}er},G_{A\to T},C{T}_{T\to A}^{\mathit{int}ra},C{T}_{T\to A}^{\mathit{int}er},G_{T\to A})$$

(15)

Then, we propose the “Local Context Enhancement (LCE)” component to further enhance the contextual information of the fusion modality, which can be described as follows:

$$q_i=W_q{\tilde{h}}_i^{fuse}+b_q$$

(16)

$${\alpha }_i=soft\mathit{max}(W_1(q_i\odot k_i)+b_{\alpha }),k_i=\{v_i\left|i-p^{\prime }<i<i+p^{\prime }\right.\}$$

(17)

$$v_i={\alpha }_i{k}_i^T$$

(18)

$$h_i^{fuse}=Norm(\mathit{tanh}(v_i)\odot {\tilde{h}}_i^{fuse}+{\tilde{h}}_i^{fuse})$$

(19)

where ${\tilde{h}}_i^{fuse}$ represents the fusion representation ${\tilde{H}}_{fuse}$for the utterance $u_i$; $W_q$ and$W_1$ are learnable parameters, and $p^{\prime }$ is a hyperparameter; $v_i$ denotes the context information of the fusion modality for the utterance $u_i$; if $i$ = 1, we set the $v_i$ to be ${\tilde{h}}_i^{fuse}$; $Norm$ denotes the layer normalization; and $h_i^{fuse}\in {\mathbb{R}}^d$ indicates the enhanced contextual information representation for the utterance $u_i$. Therefore, the enhanced contextual representation of the fusion modality can be denoted as $H_{fuse}$.

3.4. Graph-Attention-Based Multimodal Interaction and Fusion

3.4.1. Directed Graph Construction

In this stage, three directed graphs, including the $A\to T$ graph, $T\to A$ graph, and the fusion graph, are accordingly constructed, which are denoted as ${\mathcal{G}}^{A\to T}=\{{\mathcal{V}}^A+{\mathcal{V}}^T,E^{A\to T},{\mathcal{W}}^{A\to T}\}$, ${\mathcal{G}}^{T\to A}=\{{{\mathcal{V}}^T+\mathcal{V}}^A,E^{T\to A},{\mathcal{W}}^{T\to A}\}$, and ${\mathcal{G}}^{fuse}=\{{\mathcal{V}}^{fuse},E^{fuse},{\mathcal{W}}^{fuse}\}$, respectively. Here, $\mathcal{V}$ denotes the set of corresponding utterance nodes, $E$ denotes the set of edges between nodes, $R$ means the relation types of edges, and $\mathcal{W}$ represents the set of weights between edges. Specifically, these directed graphs are constructed as follows:

Nodes: The utterance $u_i$ is represented by two nodes $v_i^A$ and $v_i^T$ both in the $A\to T$ graph and the $T\to A$ graph, which are initialized with $h_i^A$ and $h_i^T$ corresponding to the enhanced acoustic and textual modalities. It means that if there are n utterances in a conversation, there are 2n nodes both in the $A\to T$ graph and the $T\to A$ graph accordingly. But each utterance is represented by one node in the fusion graph, initialized with $h_i^{fuse}$. Thus, given a conversation with n utterances, we construct the fusion graph with n nodes.

Edges: The edge between the utterance $u_i$ and the utterance $u_j$ ($i<j$) is denoted as $r_{ij}\in E$, which means the context information in $u_i$ is propagated to $u_j$. Given that a future utterance cannot propagate information backwards, we define that $u_j$ cannot propagate information to its predecessor $u_i$, i.e., $r_{ji}\notin E$, in our directed graphs. If each utterance is connected to all its predecessors, the large number of edges will result in high computational costs. When the speaker speaks utterance $u_{\rho }$, he/she has realized the current context information to express the emotion of utterance${ u}_{\rho }$. Assuming that utterance${ u}_{\rho }$ is the s-th nearest utterance spoken by $s_{\phi \left(u_i\right)}$ before $u_i$, we construct a directed edge $r_{\rho i}\in E$, and $\forall l$ and $\rho <\kappa <i$, the directed edge $r_{\kappa i}\in E$ is also constructed in the graphs. But $\forall \tau <\rho$, we ignore the edge from $u_{\tau }$ to $u_i$ in the graph, i.e., $r_{\tau i}\notin E$. It indicates that the conversational context before utterance${ u}_{\rho }$ is relatively less crucial for identifying the emotional state of utterance $u_i$($\rho <i$), and utterance${ u}_{\rho }$ needs to propagate the learnt conversational context to utterance $u_i$. Therefore, it is significant for the utterance $u_i$ to find the cut-off point ${ u}_{\rho }$ to construct directed edges, for the purpose of reducing computational complexity while preserving the remote context. Here, s is a hyperparameter that needs to be determined during training. Based on the above edge definitions, the edges among the n utterances in a conversation can be constructed in the fusion graph. For example, $v_{\rho }^{fuse}$ is connected with $v_i^{fuse}$ in the fusion graph as shown in Figure 5.

Furthermore, for the $A\to T$ graph and the $T\to A$ graph, we also define that any two nodes under the same modality have no edges, and each node will be connected with the nodes which correspond to its previous utterances but from another modality as shown in Figure 6a,b. It indicates that $v_{\rho }^A$ is connected with $v_i^T$ in the $A\to T$ graph, and $v_{\rho }^T$ will be connected with $v_i^A$ in the $T\to A$ graph, which enables the cross-modal graphs to further explore cross-modal interactions.

Edges types: The relation types of the above defined edges depend on speaker dependency. It means if two nodes ($v_{\rho }^A$ and $v_i^T$) have an edge ($r_{\rho i}^{A\to T}$) and their corresponding utterances are uttered by the same speaker, i.e., $s_{\phi \left(u_{\rho }\right)}=s_{\phi \left(u_i\right)}$, the relation type of $r_{\rho i}^{A\to T}$ is the intra-speaker type. Otherwise, the relation type of the edge is set to be the inter-speaker type. For example, as shown in Figure 6a, supposing there are two speakers in a conversation, the edge type of $r_{12}^{A\to T}$ is the inter-speaker, while the type of $r_{14}^{A\to T}$ is intra-speaker edge.

Edge weights: The edge weights determine the significance of neighboring nodes. In this study, we employ cross-modal graph attention (GAT) [43] to update the edge weights for characterizing the significance of diverse neighboring information. The edge weight of $r_{ij}^{A\to T}$ can be calculated as follows:

$${\alpha }_{ij}^{A\to T}=\frac{\mathit{exp}(\sigma (a^T[W_{\alpha }{h}_i^A\left|\right|W_{\alpha }{h}_j^T]))}{{\sum }_{u_k\in \mathcal{N}(u_i)}\mathit{exp}(\sigma (a^T[W_{\alpha }{h}_i^A\left|\right|W_{\alpha }{h}_k^T]))}$$

(20)

where ${\alpha }_{ij}^{A\to T}$ represents the edge weight between node $v_i^A$ and node $v_j^T$, $h_i^A$ is the enhanced acoustic representation of the utterance $u_i$, $h_i^T$ denotes the enhanced textual representation of the utterance $u_i$, $W_{\alpha }$ and $a^T$ are learnable parameters, || is the concatenation operation, $\sigma$ denotes the non-linear activation function LeakyReLU, and $\mathcal{N}(u_i)$ is the set of neighboring nodes of the utterance $u_i$. Similarly, based on Equation (19), we can also calculate the edge weights in the $T\to A$ graph (${\alpha }_{ij}^{T\to A}$) and the fusion graph (${\alpha }_{ij}^{fuse}$), respectively.

3.4.2. Graph-Attention-Based Multimodal Fusion

In order to further explore the cross-modal context learning and relieve the over-smoothing issue of GNN [44], we have made some improvements to the vanilla GAT, which is called “Graph-Attention-based Multimodal Fusion (MF-GAT)”. Specifically, taking the node $v_i^T$ in the $A\to T$ graph as an example, MF-GAT employs the attention mechanism to aggregate information from its neighboring nodes, which can be described as follows:

$$g_i^{A\to T}={\sum }_{v_j^A\in \mathcal{N}(v_i^T)}{\alpha }_{ij}^{A\to T}{W}_{ij}^{A\to T}{h}_j^A$$

(21)

where ${\alpha }_{ij}^{A\to T}$ is the edge weight between node $v_j^A$ and $v_i^T$, as well as the attention coefficient, $W_{ij}^{A\to T}$ is the learnable parameter, $v_j^A$ is the neighboring node of $v_i^T$, $\mathcal{N}(v_i^T)$ denotes the set of neighboring node of $v_i^T$, and $g_i^{A\to T}$ is the aggregated information of utterance${ u}_i$ in the $A\to T$ graph. Similarly, the aggregated information of utterance${ u}_i$ in both the $T\to A$ graph and the fusion graph can be obtained based on Equation (20), which can be denoted as $g_i^{T\to A}$ and $g_i^{fuse}$, respectively.

Then, we further explore the cross-modal interaction through the above-mentioned MCI-GRU cell to update the representation of utterances under each modality, which can be expressed as follows:

$$m_i^{A\to T(l)}=MCI-GRU(g_i^{A\to T\left(l\right)},g_i^{T\to A\left(l\right)},g_i^{fuse\left(l\right)},m_i^{A\to T(l-1)})$$

(22)

where $g_i^{A\to T(l)}$, $g_i^{T\to A(l)}$, and $g_i^{fuse(l)}$ denote the aggregated information of utterance${ u}_i$ at the layer l in the $A\to T$, $T\to A$, and fusion graphs, respectively; $m_i^{A\to T(l-1)}$ is the updated representation of node $v_i^T$ at the (l − 1) layer; and $m_i^{A\to T(l)}$ indicates the updated representation of node $v_i^T$ at the l layer. Here, the gate mechanism in MCI-GRU is utilized to preserve long-term context dependencies. The specific calculations of MCI-GRU have been described in Equations (10)–(14).

Inspired by ResNet [44] and Transformer, we also propose a new GNN layer in MF-GAT to address the over-smoothing issue and generate the final representation in each graph, which can be formulated as follows:

$${\tilde{m}}_i^{A\to T(l)}=Norm(m_i^{A\to T(l-1)}+m_i^{A\to T(l)})$$

(23)

$$f_i^{A\to T(l)}=Norm(FFN({\tilde{m}}_i^{A\to T(l)})+{\tilde{m}}_i^{A\to T(l)})$$

(24)

$$FFN({\tilde{m}}_i^{A\to T(l)})=Drop(Linear(Drop(\sigma (Linear({\tilde{m}}_i^{A\to T(l)};{\Theta }_1)));{\Theta }_2))$$

(25)

where $Norm$ and $FFN$ denote the layer normalization and feedforward functions, respectively; $Drop$ and $Linear$are the dropout and linear activation functions, respectively; $\sigma$ denotes the non-linear activation function; ${\Theta }_1$ and ${\Theta }_2$ are learnable parameters; and $f_i^{A\to T(l)}\in {\mathbb{R}}^d$ denotes the output representation of the utterance ${ u}_i$ at the l layer in the $A\to T$ graph. Similarly, we can obtain the output representations of the utterance${ u}_i$ at the l layer in the $T\to A$ and the fusion graphs, denoted as $f_i^{T\to A(l)}\in {\mathbb{R}}^d$ and $f_i^{fusion(l)}\in {\mathbb{R}}^d$. Here, l is a hyperparameter that needs to be determined in the training process.

Then, the output representations of the utterance${ u}_i$ are concatenated based on the skip connection [45] to generate the final representations of the utterance${ u}_i$, which can be described as follows:

$$F_i^{A\to T}=Concat(f_i^{A\to T\left(0\right)}, f_i^{A\to T\left(1\right)},f_i^{A\to T\left(2\right)},\cdot \cdot \cdot ,f_i^{A\to T\left(l\right)})$$

(26)

$$F_i^{T\to A}=Concat(f_i^{A\to T\left(0\right)},f_i^{T\to A\left(1\right)},f_i^{T\to A\left(2\right)},\cdot \cdot \cdot ,f_i^{T\to A\left(l\right)})$$

(27)

$$F_i^{fuse}=Concat(f_i^{fuse\left(0\right)},f_i^{fuse\left(1\right)},f_i^{fuse\left(2\right)},\cdot \cdot \cdot ,f_i^{fuse\left(l\right)})$$

(28)

where $F_i^{A\to T}\in {\mathbb{R}}^{d\ast (l+1)}$, $F_i^{T\to A}\in {\mathbb{R}}^{d\ast (l+1)}$, and $F_i^{fuse}\in {\mathbb{R}}^{d\ast (l+1)}$ denote the final obtained representations of the utterance${ u}_i$ in the $A\to T$, $T\to A$, and fusion graphs, respectively.

Finally, we concatenate the representations of the utterance${ u}_i$ learned from three directed graphs to obtain the final multimodal fusion representation $F_i\in {\mathbb{R}}^{d\ast 3(l+1)}$, which can be described as follows:

$$F_i=Concat(F_i^{A\to T},F_i^{T\to A},F_i^{fuse})$$

(29)

3.5. Self-Distillation-Based Emotion Prediction

Emotion Prediction. The final multimodal fusion representation $F_i$ is fed into a fully connected layer followed with a Softmax layer for emotion prediction:

$$p_i=Softmax(W_p(\mathit{Re}LU(W_c{F}_i+b_c))+b_p)$$

(30)

$${\widehat{y}}_i=\mathit{argmax}(p_i)$$

(31)

where $W_c\in R^{d_1\times d_1}$, $b_c\in R^{d_1}$,$W_p\in R^{C\times d_1}$, and $b_p\in R^C$ are trainable parameters, $p_i$ is the predicted probability, and ${\widehat{y}}_i$ is the predicted emotion label for utterance $u_t$. Here, $d_1$= $d\ast 3(l+1)$, and $C$ denotes the pre-defined emotion categories. Thus, $C$ is set to be 6 in IEMOCAP [23], but $C$ is equal to 7 in MELD [24]. We employ the cross-entropy loss with L2 regularization to calculate the emotion classification loss during the training process for the purpose of optimizing all learnable parameters, which can be calculated as follows:

$$L_{cls}=-\frac{1}{{\sum }_{i=1}^{N_C}N}{\sum }_{i=1}^{N_C}{\sum }_{j=1}^N{y}_{ij}\mathit{log}(p_{ij})+{\lambda }_{cls}|{\Theta }_{cls}|$$

(32)

where $N$ is the total number of utterances in the i-th conversation, $N_C$ is the number of conversations in dataset, $y_{ij}$ is the true emotion label for the utterance $u_j$ in the i-th conversation, $p_{ij}$ is the predicted probability, ${\lambda }_{cls}$ is the L2 regularization coefficient, and ${\Theta }_{cls}$ denotes the trainable parameter.

Self-Distillation. However, there are semantic gaps between different modalities during the cross-modal interactions, which may bring the risk of misalignment and further influence the final emotion prediction performance. Therefore, we introduce self-distillation [45] to utilize the soft label ${\widehat{y}}_i$ to supervise the training of each MCI-GRU module to ensure that the learnt representation distributions of each modality are close to each other. Concretely, we treat the main fusion module of the three directed graphs as the teacher module. The three modality outputs in the second stage (i.e., $h_i^T$, $h_i^A$, and $h_i^{fuse}$) are fed into fully connected layers followed with Softmax layers, respectively, which are treated as student modules, denoted as $studen{t}_T$, $studen{t}_A$, and $studen{t}_f$. Taking the $studen{t}_T$ as an example, the predicted probability of $studen{t}_T$ can be expressed as follows:

$$p_i^T=Softmax(W_T(\mathit{Re}LU(W_{ST}{h}_i^T+b_{ST}))+b_T)$$

(33)

where $W_{ST}$, $W_T$, $b_{ST}$, and $b_T$ are learnable parameters, and $p_i^T$ is the obtained prediction probability for the utterance $u_i$ based on $studen{t}_T$. Similarly, we can obtain $p_i^T$ and $p_i^T$ for the utterance $u_i$ based on $studen{t}_A$ and $studen{t}_f$, respectively. Meanwhile, we use the true emotion labels and the cross-entropy loss function to supervise and optimize the training of each student module, which can be described as follows:

$$L_{CE}^T=-\frac{1}{{\sum }_{i=1}^{N_C}N}{\sum }_{i=1}^{N_C}{\sum }_{j=1}^N{y}_{ij}\mathit{log}{p}_{ij}^T$$

(34)

where $y_{ij}$ is the true emotion label for the utterance $u_j$ in the i-th conversation, and $p_{ij}^T$ is the predicted probability based on $studen{t}_T$. Similarly, we can obtain $L_{CE}^A$ and $L_{CE}^f$ for the $studen{t}_A$ and $studen{t}_f$, respectively. Therefore, the total cross-entropy loss of the three student modules can be expressed as follows:

$$L_{CE}={\lambda }_1{L}_{CE}^T+{\lambda }_2{L}_{CE}^A+{\lambda }_3{L}_{CE}^f$$

(35)

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are hyperparameters. In addition, we also introduce the KL divergence to approximate the probability distribution of the teacher module and student modules to further explore high-quality multimodal fusion. It can be calculated as follows:

$$L_{KL}^T=\frac{1}{{\sum }_{i=1}^{N_C}N}{\sum }_{i=1}^{N_C}{\sum }_{j=1}^N{p}_{ij}^T\mathit{log}(\frac{p_{ij}^T}{p_{ij}})$$

(36)

where $p_{ij}$ is the predicted probability for the utterance $u_j$ in the i-th conversation based on the teacher module, and $p_{ij}^T$ is the predicted probability based on $studen{t}_T$. Similarly, we can obtain $L_{KL}^A$ and $L_{KL}^f$ for the $studen{t}_A$ and $studen{t}_f$, respectively. Therefore, the total KL divergence loss of the three student modules can be expressed as follows:

$$L_{KL}={\beta }_1{L}_{KL}^T+{\beta }_2{L}_{KL}^A+{\beta }_3{L}_{KL}^f$$

(37)

where ${\beta }_1$, ${\beta }_2$, and ${\beta }_3$ are hyperparameters.

Therefore, we combine the above $L_{cls}$, $L_{CE}$, and $L_{KL}$ together to generate the final objective function:

$$L={\mu }_1{L}_{cls}+{\mu }_2{L}_{CE}+{\mu }_3{L}_{KL}$$

(38)

where ${\mu }_1$, ${\mu }_2$, and ${\mu }_3$ are hyperparameters. In this study, we set ${\mu }_1$ = ${\mu }_2$ = ${\mu }_3$ = 1.

4. Experimental Datasets and Setup

In this section, the experimental datasets as well as evaluation metrics are first introduced; then, the mainstream ERC baselines are implemented to evaluate the performance of our proposed HCIFN-SD, and finally, the implementation details are reported.

4.1. Experimental Datasets and Evaluation Metrics

(1) Datasets: We evaluate our proposed HCIFN-SD on two widely used multimodal datasets: IEMOCAP [23] and MELD [24], which include textual, acoustic, and visual signals. In this paper, we only use the textual and acoustic modalities for experiments. The basic statistic information of these two datasets is listed in

Table 1. Statistic information of both the IEMOCAP and MELD datasets.

Dataset	Dialogue Count			Utterance Count			Categories	Speaker per Dialogue
	Train	Valid	Test	Train	Valid	Test
IEMOCAP	120		31	5810		1623	6	2
MELD	1039	114	280	9989	1109	2610	7	more than 2

.

IEMOCAP is one of the most widely used multimodal datasets in ERC tasks. It has five conversation sessions under various scenarios involving ten speakers, but each dialogue in IEMOCAP is a dyadic conversation dataset. The recorded multimodal data for approximately 12 h contains 151 conversations, which are further segmented into 7433 utterances. Each utterance is labeled with one of the following six emotion categories, including happiness, sadness, neutral, anger, excitement, and frustration. Following the previous studies [5,13], the first four sessions are utilized for model training and validation, and the remaining session is used for model testing.

MELD is a popular multimodal multi-party conversational dataset, which was collected from the Friends TV shows. It contains a total of 1433 conversations, involving 13,708 utterances uttered by 304 speakers. The dialogue in MELD is relatively short because Table 1 indicates that the average number of utterances per dialogue is 9.6, but each dialogue in MELD involves more than 2 speakers. Each utterance is labeled with one of the following seven emotion labels: neutral, surprise, fear, sadness, joy, disgust, and anger. Experiments on MELD are conducted based on the previous dataset partition rule [5,13].

(2) Evaluation Metrics: Considering that both IEMOCAP and MELD are imbalanced categorical datasets, we use the weighted average accuracy (WAA) and weighted average F1-score (WAF1) as evaluation metrics for the purpose of conveniently comparing our HCIFN-SD with the state-of-the-art ERC approaches. In addition, F1 scores for each emotion category are also reported for comprehensive comparison with the baselines.

4.2. Baseline Approaches

To demonstrate the effectiveness of our proposed HCIFN-SD, we compare it with the following ERC baselines.

• CMN [9] employs two distinct GRUs to explore inter-speaker context dependencies for predicting utterance-level emotions, but it cannot work on multi-party conversation tasks.
• ICON [10], as an improved approach of CMN, employs distinct GRUs to explore both intra- and inter-speaker dependencies for emotion identification, but it still cannot deal with multi-party conversation scenarios.
• BC-LSTM [15] utilizes the bidirectional LSTM architecture to learn conversational context for emotion classification, but it is a speaker-independent model.
• DialogueRNN [6] adopts three distinct GRUs, including Global GRU, Speaker GRU, and Emotion GRU, to capture conversational context, learn speaker identity information, and identify utterance-level emotions, respectively.
• A-DMN [11] is a multimodal sequence-based ERC approach, which utilizes bidirectional LSTM layers to model intra-speaker and inter-speaker influences and then employs the attention mechanism to combine them to update the memory for final emotion prediction.
• DialogueGCN [7] first extracts sequential context based on the Bi-GRU layer, then explores speaker-level dependencies based on the GCN, and finally integrates these two types of context for final emotion recognition. Considering that the original DialogueGCN is a unimodal approach, we implement multimodal DialogueGCN by concatenating features of each modality.
• MMGCN [13] is a multimodal fusion GCN, in which each utterance can be represented by three nodes for distinguishing different modalities to capture both cross-modal dependencies and speaker-dependent dependencies.
• MM-DFN [14] is a graph-based multimodal dynamic fusion approach, which explores the dynamics of conversational context under different modality spaces for the purpose of reducing redundancy and improving complementarity between modalities.
• GraphCFC [20] utilizes multimodal subspace extractors and cross-modal interactions to achieve multimodal fusion. It is a speaker-dependent ERC model.
• GA2MIF [46] is a multimodal fusion approach, involving the textual, acoustic, and visual modalities. It constructs three graphs for modeling intra-modal local and long-distance context as well as cross-modal interactions.
• SMFNM [5] first implements intra-modal interactions based on the semi-supervised strategy, then selects the textual modality as the main modality to guide the cross-modal interaction, and finally explores the conversational context from both multimodal and main-modal perspectives for obtaining more comprehensive context information.

4.3. Implementation Details

We implement HCIFN-SD through the Pytorch framework. The hyperparameters and parameters have been optimized through the grid searching strategy. The hyperparameters are set as follows: The dimension of the unimodal representations enhanced with the speaker identity information is set to be $d=$ 200. The context window size in the local information mask is set to be $p$ = 27 for IEMOCAP and $p$ = 3 for MELD, respectively. Another context window size in the LCE component is set to be $p^{\prime }$ = 4 for both datasets. In the directed graphs, we set $s=$1, which indicates that only the latest intra-speaker dependency is considered as the cut-off point to construct the directed edges for context dependency propagation. The number of graph layers in the MF-GAT module is set to be $l=$2 for IEMOCAP and $l=$1 for MELD, respectively. Therefore, the dimensions of the final multimodal fusion representation are 1800 for the IEMOCAP dataset and 1200 for the MELD dataset, respectively. In addition, we set ${\lambda }_1$ = ${\lambda }_2$= ${\lambda }_{3 }$= 1, ${\beta }_1$=${\beta }_{2 }$=${ \beta }_3$= 1, and ${\mu }_{1 }$= ${\mu }_2$= ${\mu }_3$= 1, respectively.

During training, the Adam optimizer is used with a learning rate of 0.0002 and a maximum of 100 epochs to optimize trainable parameters. The early stopping strategy is employed, and the stop condition is that if the validation loss does not decrease for 5 consecutive epochs, the training will stop. The $L_2$ regularization parameter is set to be 0.00001. The dropout rate is set to be 0.5 to alleviate the overfitting problem.

5. Results and Discussion

In this section, the results of all comparison experiments and ablation studies are reported and discussed in detail to demonstrate the effectiveness and superiority of our proposed HCIFN-SD. Then, a quantitative analysis on the hyperparameters is conducted. Finally, we qualitatively discuss our proposed HCIFN-SD through a case study.

5.1. Comparison with Baseline Models

We compare HCIFN-SD with state-of-the-art baselines on the IEMOCAP and MELD datasets under multimodality features. Experimental results, including the overall performance and F1 scores of each emotion category, are reported in

Table 2. Performance of different approaches on the IEMOCAP testing set under multimodal features. We report F1 scores for each emotion category and overall scores using WAA and WAF1. Best results are highlighted in bold.

Model (& Params)	IEMOCAP
	Happiness	Sadness	Neutral	Anger	Excitement	Frustration	WAA	WAF1
CMN (-)	30.38	62.41	52.39	59.83	60.25	60.69	56.56	56.13
ICON (-)	29.91	64.57	57.38	63.04	63.42	60.81	59.09	58.54
BC-LSTM * (-)	35.60	69.20	53.50	66.30	61.10	62.40	59.80	59.10
DialogueRNN * (8.8 M)	32.90	78.00	59.10	63.30	73.60	59.40	63.40	62.80
A-DMN (-)	50.60	77.20	63.90	60.10	77.90	63.20	64.90	64.10
DialogueGCN * (2.7 M)	36.70	81.90	65.10	64.50	68.80	65.40	66.40	65.80
MMGCN (2.6 M)	45.45	77.53	61.99	66.67	72.04	64.12	65.56	65.71
MM-DFN (2.2 M)	42.22	78.98	66.42	69.77	75.56	66.33	68.21	68.18
GraphCFC (5.4 M)	43.08	84.99	64.70	71.35	78.86	63.70	69.13	68.91
GA2MIF (9.3 M)	41.65	84.50	68.38	70.29	75.99	66.49	69.75	70.00
SMFNM (-)	59.50	81.50	70.00	62.90	74.40	70.10	70.80	70.90
HCIFN-SD-small (4 M)	67.36	73.47	77.60	68.24	74.92	69.03	72.58	72.88
HCIFN-SD (12 M)	68.75	75.51	77.60	69.41	72.91	70.34	73.07	73.42

and

Table 3. Performance of different approaches on the MELD testing set under multimodal features. We report F1 scores for each emotion category and overall scores using WAA and WAF1. Best results are highlighted in bold.

Model (& Params)	MELD
	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Anger	WAA	WAF1
BC-LSTM * (-)	76.40	48.47	0.0	15.60	49.70	0.0	44.50	58.60	56.80
DialogueRNN * (8.8 M)	73.20	51.90	0.0	24.80	53.20	0.0	45.60	58.00	57.00
A-DMN (-)	77.10	57.73	12.06	29.12	57.41	7.16	43.97	—	60.45
DialogueGCN * (2.7 M)	73.30	57.00	16.50	41.10	61.80	20.10	55.30	60.20	60.40
MMGCN (2.6 M)	76.96	49.63	3.64	20.39	53.76	2.82	45.23	59.31	57.82
MM-DFN (2.2 M)	75.80	50.42	—	23.72	55.48	—	48.27	62.49	59.46
GraphCFC (5.4 M)	76.98	49.36	—	26.89	51.88	—	47.59	61.42	58.86
SMFNM (-)	75.00	57.50	16.80	36.80	62.30	25.00	50.30	62.60	62.40
HCIFN-SD-small (4 M)	77.55	62.28	28.00	35.10	66.17	25.00	46.96	64.41	64.05
HCIFN-SD (12 M)	77.87	62.99	34.00	35.10	65.92	26.47	46.67	64.71	64.34

. Here, results with “*” indicate that the baseline is a unimodal approach, but for a fair comparison, we rerun the corresponding open-source codes using our extracted unimodal features to generate multimodal features for emotion prediction; “-” indicates that the results are unavailable. Other baseline results are obtained from the existing literature.

5.1.1. Comparison of Overall Performance

Table 2 shows that our proposed HCIFN-SD performs better than all the baselines and achieves 73.42% on WAF1 and 73.07% on WAA for the IEMOCAP dataset, which achieves an absolute improvement of 2.52% on WAA and 2.27% on WAF1 compared with the currently advanced ERC model (i.e., SMFNM). Table 3 shows that our proposed approach succeeds over the current state-of-the-art ERC approach (i.e., SMFNM) by 1.94% on WAF1 and 2.11% on WAA. To sum up, our proposed HCIFN-SD outperforms all the baseline models in terms of WAF1 and WAA, which demonstrates the superiority of our proposed approach in dealing with multimodal ERC tasks. The significant improvements can be explained by the following reasons:

(1)

Our HCIFN-SD is significantly superior to BC-LSTM on both datasets. It is difficult to accurately identify human emotion in a conversation scenario because the emotional state can be easily influenced by the interlocutor’s behaviors [47]; thus, it is quite significant to model the speaker-sensitive dependencies in conversations [11]. However, BC-LSTM is a typical speaker-independent approach. It only depends on LSTM structures to capture conversational context from the surrounding utterances without considering intra-speaker and inter-speaker influences, which leads to the relatively poor performance of BC-LSTM. All the other models take into account speaker-sensitive modeling. For instance, CMN and ICON utilize two distinct GRUs for two speakers to separately explore intra-speaker dependencies and use another GRU for modeling inter-speaker dependencies. The graph-based models (such as DialogueGCN, MMGCN, MM-DFN, GraphCFC, GA2MIF, and SMFNM) utilize the nodes to represent individual utterances and use the edges between nodes to bridge the intra-speaker and inter-speaker influences. Our proposed HCIFN-SD not only explicitly models the intra-speaker and inter-speaker dependencies based on the proposed different mask strategies but also utilizes the different types of edges in the directed graph construction to comprehensively explore speaker-sensitive conversational context.

(2)

Our proposed approach significantly succeeds over the multimodal sequence-based approaches (such as CMN, ICON, and A-DMN). It can be observed that these approaches directly concatenate multimodal information without incorporating the cross-modal interaction information. However, lots of multimodal research has demonstrated the importance of exploring cross-modal interaction for enhancing the quality of multimodal fusion representation [5,18,48]. In addition, these recurrence-based models tend to utilize local utterances for conversational context extraction and usually ignore long-term context information. To address these issues, our HCIFN-SD designs the improved GRU structure named MCI-GRU, which utilizes the speaker-sensitive dependency information to participate in the calculation of the update gate to explore speaker-dependent long-distance context. Furthermore, our HCIFN-SD proposes multi-stage cross-modal interactions and fusion and employs the self-distillation strategy to enhance the representation ability of multimodal information.

(3)

HCIFN-SD outperforms the multimodal graph-based approaches (such as MMGCN, MM-DFN, GraphCFC, and GA2MIF). It is found that MMGCN and MM-DFN generate three nodes for each utterance to represent different modalities and construct edges for different modality nodes under the same utterance to explore intra-speaker-based complementary information from other modalities. Obviously, this graph construction approach not only results in data redundancy, due to inconsistent data distribution among different modalities, but also fails to extract diverse information in the conversational graph, which all affect the representation ability of the multimodal fusion information. GraphCFC defines different types of edges based on the perspectives of speakers and modalities to extract speaker-dependent cross-modal context information, but it not only fails to capture the sequential information of conversations but also ignores the semantic gaps among modalities. GA2MIF concentrates on cross-modal contextual modeling based on the proposed two modules, but it only depends on the static speaker identity information without considering the in-depth exploration of the speaker-dependent cross-modal context extraction and fusion. To address these issues, our HCIFN-SD proposes the MCI-GRU module to explore speaker-dependent cross-modal context information, utilizes the MF-GAT module to further explore speaker-dependent multimodal interaction and fusion, and finally employs the self-distillation to eliminate the semantic gaps among different modalities.

Therefore, our proposed approach can more adequately extract both long-distance and local conversational context information based on the speaker-sensitive modeling and multi-stage cross-modal interactions and effectively reduce the inconsistent data distribution to eliminate semantic gaps among modalities. Therefore, it is not surprising that compared with baseline models, our HCIFN-SD achieves the best emotion recognition performance.

5.1.2. Comparison for Each Emotion Category

For performance comparison in each emotion category, we also visualize the confusion matrices of HCIFN-SD on the benchmark testing sets as shown in Figure 7.

Table 2 indicates that compared with baseline models, our HCIFN-SD shows remarkably significant improvements for happiness and neutral by 9.25~38.84% and 7.6~24.1%, respectively, on the IEMOCAP dataset. Table 2 also reports that HCIFN-SD slightly outperforms the currently advanced model (i.e., SMFNM) for Frustration and achieves comparable performance for sadness, anger, and excitement on par with the state-of-the-art baselines. Meanwhile, the happiness category is easily confused with excitement and vice versa as shown in Figure 7a, which is consistent with previous research [5]. The possible explanation is that happiness and excitement are quite similar in some cases. Figure 7a presents that the sadness, neutral, and anger emotions can be easily misclassified as frustration, while the frustration emotion can be confused with neutral and anger. These phenomena are strongly related to the class-imbalanced distribution in IEMOCAP, because the frustration and neutral emotions belong to the majority categories [23]. Furthermore, humans have a similar perception and interpretation of negative emotions (such as sadness, anger, and frustration), which can also explain these phenomena.

Table 3 shows that for the MELD dataset, our proposed HCIFN-SD achieves higher F1 scores than baseline models for the majority of emotions, including neutral, joy, disgust, fear, and surprise. Due to the highly imbalanced class distributions in MELD [24], some existing baselines (such as BC-LSTM and DialogueRNN) cannot identify disgust and fear, and some baselines (such as MM-DFN and GraphCFC) do not report F1 scores for these two classes, but the F1 scores of our proposed HCIFN-SD are 26.47% and 34.00%, respectively. These results demonstrate the effectiveness of our HCIFN-SD in recognizing most emotion categories. Table 3 also indicates that our proposed HCIFN-SD achieves a comparable performance for sadness and anger on par with the state-of-the-art baselines. Figure 7b clearly shows that most emotion categories can be easily confused with neutral, which can also be explained by the class-imbalanced issue in MELD.

5.2. Ablation Study

To demonstrate the effectiveness of each component in our proposed approach, we conduct extensive ablation experiments on the two benchmark datasets.

5.2.1. Impact of Different Mask Strategies

In this subsection, we discuss the impact of different mask strategies on the performance of HCIFN-SD. To verify the effectiveness of the intra-speaker mask, inter-speaker mask, and local information mask to model speaker-sensitive cross-modal context information, three comparison settings are used. Experimental results are listed in

Table 4. Ablation results for different mask strategies on IEMOCAP and MELD datasets.

Approaches	IEMOCAP		MELD
	WAA	WAF1	WAA	WAF1
w/o $S_{spk}$ & mask	70.36	70.57	64.13	63.96
w/o intra-speaker mask	72.83	73.00	64.56	64.10
w/o inter-speaker mask	72.58	72.94	64.70	64.30
w/o local information mask	71.90	72.15	64.52	64.14
Ours (HCIFN-SD)	73.07	73.42	64.71	64.34

. Here, “w/o” means removing one specific mask strategy.

Table 4 presents that HCIFN-SD outperforms w/o intra-speaker mask by 0.24~0.42% on WAF1 and 0.15~0.24% on WAA, which verifies the effectiveness of exploring cross-modal interaction based on the intra-speaker mask strategy. Table 4 reports HCIFN-SD exhibits an absolute improvement of 0.14~0.48% WAF1 and 0.01~0.49% on WAA over w/o inter-speaker mask, which demonstrates the effectiveness of exploring cross-modal interaction based on the inter-speaker mask strategy. Therefore, the above results suggest that the speaker-sensitive dependencies in conversations can be well modeled by our proposed intra- and inter-speaker mask strategies. In addition, our HCIFN-SD is superior to w/o local information mask by 0.20~1.27% on WAF1 and 0.19~1.17% on WAA, which suggests the effectiveness of exploring cross-modal interaction based on the local information mask strategy.

Table 4 also shows that removing the intra-speaker mask leads to a slight degradation in performance, but removing the local information mask results in a more significant performance degradation for the IEMOCAP dataset, which indicates that learning local conversational context from neighboring utterances is more important for dealing with IEMOCAP. However, experimental results in Table 4 indicate that exploring local conversational context from neighboring utterances is just as important as exploring intra-speaker dependencies for the MELD dataset. The possible reason for these inconsistent findings is that the speakers of each conversation differ in the two datasets. IEMOCAP consists of dyadic conversations [23], which indicates that plenty of intra-speaker dependencies are contained in neighboring utterances. The average number of speakers per dialogue in MELD is 4.7 [24], which suggests that the neighboring utterances hardly contain self-speaker dependency information. Therefore, it is quite necessary to comprehensively explore cross-modal interactions based on the proposed mask strategies.

5.2.2. Impact of Multi-View Cross-Modal-Interaction-Enhanced GRU

Considering that MCI-GRU utilizes three types of cross-modal interactions to explore speaker dependencies in both long-distance and local conversational contexts, the following four comparison settings are executed for the purpose of discussing the impact of the proposed MCI-GRU module on the performance of HCIFN-SD. Experimental results are listed in

Table 5. Ablation study for the proposed MCI-GRU module on IEMOCAP and MELD datasets.

Approaches	IEMOCAP		MELD
	WAA	WAF1	WAA	WAF1
w/o intra-CI	72.21	72.48	64.71	64.27
w/o inter-CI	71.41	71.77	64.64	63.88
w/o local-CI	72.03	72.31	64.29	63.97
w/o MCI-GRU	71.04	70.93	64.71	64.22
Ours (HCIFN-SD)	73.07	73.42	64.71	64.34

.

Table 5 shows that compared with w/o intra-CI, our HCIFN-SD achieves 0.07~0.94% and 0~0.86% improvements on WAF1 and WAA, respectively, which verifies the effectiveness of exploring intra-speaker cross-modal interaction. Table 5 also reports that compared with w/o inter-CI, using inter-CI (i.e., our HCIFN-SD) can exhibit an enhancement of 0.46~1.65% WAF1 and 0.07~1.66% on WAA, respectively, which means it is important to explore inter-speaker cross-modal interaction. Furthermore, HCIFN-SD performs better than w/o local-CI by 0.37~1.11% on WAF1 and 0.42~1.04% on WAA, which suggests the effectiveness of exploring local-information-based cross-modal interaction. Thus, it is not surprising that w/o MCI-GRU, which is equivalent to removing three types of cross-modal interactions without GRU, achieves the worst performance as shown in Table 5. Therefore, our HCIFN-SD, which proposes the MCI-GRU module to leverage three of cross-modal interactions and the GRU structure to explore conversational context, achieves the best emotion prediction performance on the two benchmark datasets.

Table 5 indicates that modeling the inter-speaker cross-modal interaction has a greater impact on performance, especially for the IEMOCAP dataset, which is consistent with the experiment results in Table 4. The possible reason is that capturing emotion contagion through inter-speaker modeling is more beneficial for identifying emotion shift to infer correct emotions in dyadic conversations. In addition, Table 5 reports that whether to explore intra-speaker modeling has little impact on performance for the MELD dataset. We infer that this is due to the relatively short dialogues in MELD, resulting in relatively few consecutive utterances expressed by each speaker, which inevitably weakens the benefits of exploring intra-speaker dependencies.

5.2.3. Impact of Graph-Attention-Based Multimodal Fusion

The vanilla GAT aggregates information from neighboring nodes, but our MF-GAT utilizes the MCI-GRU cell to further explore cross-modal interactions and designs a new GNN layer to address the over-smoothing issue of GAT to generate the final graph representation. Therefore, the following three comparison settings are implemented to demonstrate the impact of the proposed MF-GAT module on the performance of HCIFN-SD. Here, “w/o MCI-GRU cell” removes the MCI-GRU cell in the MF-GAT module, which indicates that this approach directly uses $g_i^{A\to T(l-1)}$ and $g_i^{A\to T(l)}$ to implement calculation of the GNN layer. “w/o GNN layer” denotes that the output of the MCI-GRU cell (i.e., $m_i^{A\to T(l)}$) is used for generating the final graph representation. “Vanilla GAT” means removing both the MCI-GRU cell and the GNN layer. The corresponding results are listed in

Table 6. Ablation study for the proposed MF-GAT module on IEMOCAP and MELD datasets.

Approaches	IEMOCAP		MELD
	WAA	WAF1	WAA	WAF1
w/o MCI-GRU cell	72.03	72.16	64.70	64.11
w/o GNN layer	71.53	71.47	64.33	63.93
Vanilla GAT	71.53	71.47	64.21	64.03
Ours (HCIFN-SD)	73.07	73.42	64.71	64.34

.

From Table 6, we can see that using the MCI-GRU cell (i.e., our HCIFN-SD) can bring about 0.23~1.26% and 0.01~1.04% improvements in WAF1 and WAA, respectively, which demonstrates the effectiveness of utilizing the MCI-GRU cell to further explore cross-modal interactions. Table 6 indicates that compared with the w/o GNN layer, our HCIFN-SD improves 0.41~1.95% WAF1 and 0.38~1.54% WAA, respectively, which means that it is crucial to utilize our proposed GNN layer to eliminate the over-smoothing issue, especially for the IEMOCAP dataset. Therefore, compared with our HCIFN-SD, the Vanilla GAT approach that removes both the MCI-GRU and GNN layer presents a significant performance degradation. In addition, it is found that compared with the MCI-GRU cell, the GNN layer in the MF-GAT module shows a greater influence on emotion recognition performance. The possible reason for this phenomenon is that GAT updates the node’s information through aggregating information from neighboring nodes; thus, stacking more layers is an intuitive solution to extract remote context information, but this solution also inevitably results in over-smoothing. Therefore, it is quite necessary to address over-smoothing to avoid the potential decline of representation quality.

5.2.4. Impact of Self-Distillation

Considering that there are semantic gaps between different modalities during the cross-modal interactions, which may result in misalignment and further decrease the final emotion prediction performance, our HCIFN-SD introduces the self-distillation strategy for adding two other loss calculations to eliminate the potential semantic gaps among modalities. Therefore, to demonstrate the effectiveness of these loss functions, the following three comparison settings are implemented, and the experimental results are reported in

Table 7. Ablation study for the self-distillation on IEMOCAP and MELD datasets.

${\mathit{L}}_{\mathit{C}\mathit{E}}$	${\mathit{L}}_{\mathit{K}\mathit{L}}$	IEMOCAP		MELD
		WAA	WAF1	WAA	WAF1
w/o	w/o	69.38	69.53	63.87	63.75
w	w/o	70.92	71.16	64.67	64.16
w/o	w	71.72	71.86	64.71	64.25
Ours (HCIFN-SD)		73.07	73.42	64.71	64.34

.

The approach that fails to consider both $L_{CE}$ and $L_{KL}$ (i.e., just using $L_{cls}$ as the final loss function during the training process) achieves the worst prediction performance as shown in the 1st row in Table 7. Then, the approach that incorporates $L_{CE}$ (i.e., the 2nd row in Table 7) can bring about 0.18~2.26% and 0.04~2.15% improvements in WAF1 and WAA, respectively, which indicates that adding the $L_{CE}$ function is beneficial for optimizing the training of each student module in the second stage and further improves the final prediction performance. Similarly, the approach that incorporates $L_{KL}$ (i.e., the 3rd row in Table 7) can improve 0.09~1.56% WAF1 and 0~1.35% WAA, respectively. It suggests that introducing the KL divergence can indeed reduce the difference in the probability distributions of the teacher module and three student modules, which further enhances the final classification performance. In addition, experimental results show that compared with $L_{CE}$, $L_{KL}$ seems more important for enhancing performance. Therefore, our HCIFN-SD that simultaneously introduces both $L_{CE}$ and $L_{KL}$ based on $L_{cls}$ can achieve the best prediction performance on both datasets.

5.3. Analysis on Hyperparameters

The hyperparameter $p$ determines how many neighboring utterances can be considered for exploring local context in conversations, and the hyperparameter $p^{\prime }$ also denotes the size of window in the proposed LCE component. The hyperparameter s determines the selection of cut-off utterance${ u}_{\rho }$, and the number of graph layer l is a hyperparameter, which directly influences the information propagation in conversations. Therefore, we conduct an analysis on these hyperparameters to find the optimal settings.

5.3.1. Analysis of Context Window Size

In this subsection, we discuss the impact of different context window size $p$ on the performance of HCIFN-SD. We conduct comparative experiments for selecting the optimal $p$ for each dataset. Experimental results are presented in Figure 8 and Figure 9.

With the increase in context window size $p$, the performance of HCIFN-SD shows an increasing trend in both datasets as shown in Figure 8 and Figure 9. When $p$ = 27, HCIFN-SD reaches the best prediction performance for 73.07% WAF1 and 73.42% WAA on the IEMOCAP dataset. For the MELD dataset, HCIFN-SD obtains the optimal performance when $p$ = 3. These results indicate that while we just consider the neighboring six utterances (i.e., the past context window size and the future context window size are equal to three) to calculate cross-modal interactions for utterance $u_i$ in MELD, more neighboring utterances need to be considered for IEMOCAP. A possible reason is that the average number of utterances per dialogue is different. The average number of utterances per dialogue is more than 40 in IEMOCAP, but the conversations in MELD are very short. This results in a different range of neighboring utterances that we need to consider when computing local context information in conversations. However, when $p$ > 27 for IEMOCAP or $p$ > 3 for MELD, a decreasing trend in performance can be observed. We infer that too large a context window size means more utterances are involved in context learning, which evolves into a vanilla transformer without a mask to focus on all utterances in conversation. Therefore, the context window size $p$ has an impact on the performance of HCIFN-SD, and we set $p$ = 27 for IEMOCAP and $p$ = 3 for MELD in our experiments.

5.3.2. Analysis of Window Size in LCE

The hyperparameter $p^{\prime }$ influences the selection of neighboring fusion information in the LCE component. Comparative experiments are implemented for selecting the optimal $p^{\prime }$. Experimental results on the IEMOCAP are presented in Figure 10.

Theorem-type environments (including propositions, lemmas, corollaries, etc.) can be formatted as follows:

Figure 10 reports that increasing the window size $p^{\prime }$from 1 to 4 can consistently improve the prediction ability of HCIFN-SD. When $p^{\prime }$ = 4, HCIFN-SD can obtain the optimal prediction performance for 73.07% WAF1 and 73.42% WAA on the IEMOCAP dataset, which indicates that utilizing an appropriate number of neighboring utterances in the LCE component will further enhance the contextual information of the fusion modality and improve the final prediction performance of HCIFN-SD. However, when $p^{\prime }$ > 4, HCIFN-SD suffers from a significant performance decline as shown in Figure 10. It is also found that when setting the inappropriate hyperparameter $p^{\prime }$, the performance of HCIFN-SD is even worse than without using the LCE component. Therefore, the window size in LCE has an impact on the prediction performance of HCIFN-SD.

5.3.3. Analysis of Different Number of Graph Layers

In this subsection, we discuss the impact of different numbers of MF-GAT layers on the performance of HCIFN-SD. We conduct comparative experiments for selecting the optimal $l$ for each dataset. Experimental results on the IEMOCAP dataset are presented in Figure 11.

Figure 11 shows that when l = 2, SMFNM obtains the best prediction performance on the IEMOCAP dataset, which indicates that stacking two graph attention layers can capture the high-quality context dependencies. When l > 2, a slight performance degradation is observed. However, compared with Vanilla GAT, the decrease in magnitude of our HCIFN-SD is relatively gentle as shown in Figure 11, which also verifies the effectiveness of the proposed GNN layer in MF-GAT to relieve the over-smoothing issue. Similarly, the optimal number of graph layers is 1 for MELD. Therefore, the number of graph layers in MF-GAT has an impact on the prediction performance of HCIFN-SD.

5.3.4. Analysis of the Cut-Off Point Selection in Graph Construction

In this subsection, we explore the impact of the cut-off point selection (i.e., s) when constructing the graph. Experimental results on the IEMOCAP dataset are presented in

Table 8. Performance of HCIFN-SD on the IEMOCAP dataset with different numbers of s.

s	IEMOCAP (6-Way)
	WAA	WAF1
1	73.07	73.42
2	71.84	72.12
3	71.29	71.73

.

Table 8 shows that increasing the value of s from 1 to 3 cannot bring performance improvements but instead leads to a significant decline in performance. When s = 1, our HCIFN-SD performs best with 73.07% WAA and 73.42% WAF1, which means that choosing the first nearest utterance spoken by $p\left(u_i\right)$ before $u_i$ (i.e., s = 1) as the cut-off point to construct edges can obtain a good trade-off between context extraction and computational load. Due to the different number of speakers involved in each dialogue, s = 1 determines the different range of utterances, which needs to be considered during the graph construction for different datasets. For instance, for the IEMOCAP dataset, s = 1 suggests that the edges are constructed from the current utterance to the past two utterances for propagating both local and long-distance context. However, when s > 1, the performance of HCIFN-SD shows a significant decline in performance, which indicates that selecting a large range to construct edges may bring some noise and further influence the context propagation. Therefore, the hyperparameter s is set to be 1.

5.4. Case Study

When recognizing utterances with non-neutral emotions, such as “All right, Bye-bye. [happy]”, “Okay, thanks. [happy]”, and “Brian, I need help. [sad]”, many existing ERC models usually misclassify these utterances as neutral, but our proposed HCIFN-SD can identify the correct emotions of these utterances by the multi-stage extraction of speaker-dependent multimodal conversational context. Furthermore, when a conversation contains consecutive utterances with the same emotion label, many baseline ERC models tend to classify the emotion of the next utterance as the same label. Taking the case selected from IEMOCAP as an example, there is an emotion shift from neutral to happy in the fourth utterance as shown in Figure 12. Baseline models cannot correctly identify the emotion, but our proposed HCIFN-SD can accurately capture this emotion shift, demonstrating the superiority of our proposed HCIFN-SD in capturing emotion shift in conversations.

It is known that emotion contagion is a critical factor in ERC systems. To qualitatively demonstrate the effectiveness of the proposed inter-speaker mask, we explore the differences between using and not using an inter-speaker mask in learning cross-modal interactions, as shown in Figure 13. It can be seen that the woman’s emotions were neutral at the beginning, but when the man expressed the frustration emotion due to leaving, the woman’s emotion was also influenced by the frustration emotion. And the emotion contagion influenced the emotions of the subsequent conversation. Figure 13 shows that utilizing the inter-speaker mask to implement cross-modal interactions can accurately detect this emotional contagion.

Like most ERC approaches, our proposed HCIFN-SD also encounters challenges in accurately identifying similar emotions as shown in Figure 7. Specifically, our proposed HCIFN-SD misclassifies anger into frustration with a probability of 25.29%, and wrongly identifies excitement as happiness with a probability of 21.07% on the IEMCAP dataset. However, our proposed method can alleviate this issue to some extent. For instance, our HCIFN-SD identifies happiness as excitement with a probability of 13.19%, while MMGCN and GA2MIF do so with probabilities of 31.94% [13] and 25.0% [46], respectively. Thus, our HCIFN-SD achieves a more promising performance in identifying similar positive emotions.

6. Conclusions

In this paper, we propose a novel ERC approach, called “hierarchical cross-modal interaction and fusion network enhanced with self-distillation (HCIFN-SD)”, to address the research gaps of existing ERC methods. HCIFN-SD first proposes different mask strategies to explore both intra-speaker and inter-speaker emotional dependencies between different modalities and then utilizes speaker-dependent and local cross-modal interactions to extract speaker-dependent long-distance and local conversational context based on the proposed MCI-GRU module. In the MF-GAT module, we first construct three directed graphs to represent different modality spaces, implement cross-modal interactions to update the node’s information based on the utilization of MCI-GR cell, and then design a new GNN layer to alleviate the over-smoothing issue of GAT to generate the final graph representation. Finally, self-distillation is introduced to transfer knowledge from both hard and soft labels, for the purpose of eliminating the semantic gap between different modalities to obtain more outstanding emotion recognition performance. We have demonstrated the effectiveness and rationality of our HCIFN-SD on two benchmark datasets and achieved new state-of-the-art records with 73.42% WAF1 and 73.07% WAA on IEMOCAP and 64.71% WAF1 and 64.34% WAA on MELD. In addition, we analyze and discuss the impact of different settings of hyperparameters on the performance of HCIFN-SD. The case study qualitatively demonstrates the effectiveness of HCIFN-SD.

This study only focuses on the textual and acoustic modalities. In the future, our work will explore how to effectively extract visual information to enhance the representation quality of multimodal information to accurately identify similar emotions in ERC tasks. In addition, we will explore how to efficiently extract high-quality multimodal conversational context to further improve the identification performance for ERC in our future work.