Uni2Mul: A Conformer-Based Multimodal Emotion Classification Model by Considering Unimodal Expression Differences with Multi-Task Learning

Abstract

Multimodal emotion classification (MEC) has been extensively studied in human–computer interaction, healthcare, and other domains. Previous MEC research has utilized identical multimodal annotations (IMAs) to train unimodal models, hindering the learning of effective unimodal representations due to differences between unimodal expressions and multimodal perceptions. Additionally, most MEC fusion techniques fail to consider the unimodal–multimodal inconsistencies. This study addresses two important issues in MEC: learning satisfactory unimodal representations of emotion and accounting for unimodal–multimodal inconsistencies during the fusion process. To tackle these challenges, the authors propose the Two-Stage Conformer-based MEC model (Uni2Mul) with two key innovations: (1) in stage one, unimodal models are trained using independent unimodal annotations (IUAs) to optimize unimodal emotion representations; (2) in stage two, a Conformer-based architecture is employed to fuse the unimodal representations learned in stage one and predict IMAs, accounting for unimodal–multimodal differences. The proposed model is evaluated on the CH-SIMS dataset. The experimental results demonstrate that Uni2Mul outperforms baseline models. This study makes two key contributions: (1) the use of IUAs improves unimodal learning; (2) the two-stage approach addresses unimodal–multimodal inconsistencies during Conformer-based fusion. Uni2Mul advances MEC by enhancing unimodal representation learning and Conformer-based fusion.

1. Introduction

Emotion classification is a crucial subtask that has been extensively studied in domains such as human–computer interaction, healthcare, and medicine. Early research on emotion classification primarily utilized unimodal data, such as text [1,2], audio [3], or visual cues [4], and achieved notable success. Recently, researchers have started incorporating two or more modalities to achieve multimodal emotion classification (MEC) [5,6,7]. MEC (explanations for all abbreviations in Appendix A Table A1) can furnish salient clues to more accurately discern the genuine emotional states of the opinion holder and enhance the precision of outcomes [8]. With the burgeoning of short video applications, MEC has become an active area of research [9].

The two most salient components in MEC tasks are unimodal emotion representation learning and multimodal fusion. In many scenarios, unimodal emotion expression differs from multimodal emotion perception, and multimodal annotation stems from the interaction between each modality. Taking a video clip from the CH-SIMS dataset as an exemplar, as illustrated in Figure 1, the text “Isn’t it about to go bankrupt”, with an emotional annotation of −0.8, indicates a negative emotion, and the audio modality annotation is also −0.8. The visual modality expression includes a smile and corresponds to a positive emotional annotation of 1.0. The combination of these three modalities results in a multimodal emotional annotation of 0.6, indicating a positive emotion for the video clip.

Figure 1. An example of inconsistency between unimodal emotion expression and multimodal emotion perception. Stage One: unimodal emotion recognition, Stage Two: multimodal fusion for emotion recognition. M: Multimodal, V: Vision, A: Audio, T: Text, ⊕: multimodal fusion.

The majority of existing research employs identical multimodal annotations (IMAs) for unimodal model pretraining and multimodal fusion [10,11,12]. The unimodal models trained using IMAs cannot learn satisfactory representations but rather those of forced alignment. The representation of forced alignment will contain certain discrepancies in the distribution of each modality and form an irregular semantic space, as depicted in Figure 2a. Multimodal fusion grounded on this will generate distorted and convoluted classification boundaries (see Figure 2c). If the unimodal models are trained using independent unimodal annotations (IUAs), the representations for each modality are roughly within a unified region and harbor certain differences (see Figure 2b). Leveraging unimodal representations with certain differences such as this for multimodal fusion will contribute to superior fusion outcomes [13]. Meanwhile, the multimodal fusion strategy is also imperative to achieve a comprehensive MEC model. A satisfactory fusion strategy can fully harness the information from each modality and can capture the differences and interactions between modalities. Building upon this, the distributions of each emotion category are uniform, and the classification boundaries are also relatively systematic (see Figure 2d). Most current studies on fusion pay less attention to the unimodal–multimodal emotions’ inconsistency [14,15,16,17] and struggle to learn the differences between different modalities [13].

Figure 2. Hypothesized unimodal and multimodal emotion representation. In (a,b), the markers of different colors represent different modalities, and shapes with dotted curves of different colors represent semantic spaces of respective modalities. The black solid curves indicate the envelope of these colored lines. In (c,d), the markers of different colors represent different emotion categories, and the black solid curves represent classification boundaries. V: Vision, A: Audio, T: Text.

The CH-SIMS dataset encompasses not only IMAs for each clip but also IUAs for each modality [13], which is consistent with our stance and endeavors to train the unimodal model with IUAs, but does not delve deeply enough into the two aforementioned problems: unimodal representation learning and multimodal fusion strategy.

Problem Statement: In most existing multimodal datasets with IMAs:

$$D_1=\left\{X_{\sigma },Y_M\right\},$$

(1)

where $\sigma$ represents different modalities, $X$ is the data, and $Y_M$ is the IMAs. MEC models based on these datasets take $X_{\sigma }$ as input and $Y_M$ as output in the whole training stage. This process can be expressed as

$$X_{\sigma }\stackrel{f}{\Rightarrow }{Y}_M,$$

(2)

where $f$ is responsible for both unimodal representation and multimodal fusion.

We define CH-SIMS as

$$D_2=\left\{X_{\sigma },Y_{\sigma },Y_M\right\},$$

(3)

where $Y_{\sigma }$ represents IUAs and $Y_M$ represents IMAs. We express the training process based on CH-SIMS as

$$X_{\sigma }\stackrel{f_1}{\Rightarrow }{Y}_{\sigma }\stackrel{f_2}{\Rightarrow }{Y}_M,$$

(4)

where $f_1$ is responsible for unimodal representation and $f_2$ for multimodal fusion.

To find optimal $f_1$ and $f_2$, we propose a Conformer-based MEC model (Uni2Mul), which integrates a sub-task to predict IUAs and IMAs. We argue that to achieve a satisfactory unimodal representation, it is necessary to not only design a suitable feature extraction network, but also train it with the correct IUAs rather than IMAs. Moreover, a satisfactory fusion strategy can better leverage the effectiveness of the unimodal representation under a multi-task framework.

The contributions of this work can be summarized as follows:

The key contributions are using IUAs for better unimodal learning (see Figure 2b) and the two-stage approach to account for unimodal–multimodal inconsistencies during fusion (see Figure 2d). Uni2Mul advances MEC by improving unimodal representation learning and fusion.

The remainder of the paper is organized as follows: Section 2 describes the related work, Section 3 discusses the proposed methods and overall architecture of the Uni2Mul model, Section 4 explains the experimental setup and results, Section 5 presents the discussion, and Section 6 summarizes our work.

2. Related Work

2.1. Visual Emotion Classification

Facial expressions are one of the important bases for human emotional expression. The feature extraction methods for facial expressions mainly include traditional methods and deep learning methods.

Traditional methods for facial expression recognition rely on low-level or artificially designed features, necessitating significant human involvement. These methods commonly include global feature extraction methods [18,19], which obtain overall information regarding facial images; local feature extractions [20], which better capture local details; mixed feature extraction methods [21,22], which combine the benefits of global and local features; and optical-flow-based methods [23], which capture the dynamics and changes in facial images. While these traditional, handcrafted methods for feature extraction have achieved noticeable success, they may not be applicable in complex scenes with large amounts of data.

Deep learning networks, such as the convolutional neural network (CNN) [24,25] and recurrent neural network (RNN) [26], have demonstrated strong performance in the recognition of facial expressions, leading researchers to gradually shift away from traditional methods and toward deep learning approaches [27,28,29].

Saravanan et al. [24] noted the effectiveness of CNNs for image recognition tasks as these networks are able to capture spatial features due to their large number of filters. They proposed a model comprising six convolutional layers, two max pooling layers, and two fully connected layers, which outperformed decision trees and feed-forward neural networks on the FER-2013 dataset. Yu et al. [25] proposed a model comprising a face detection module based on the ensemble of three face detectors and a classification module that utilized an ensemble of multiple deep CNNs. To combine the multiple CNN models, they presented two schemes for learning the ensemble weights of the network responses: by minimizing the log likelihood loss and by minimizing the hinge loss. Their model achieved excellent results on both the FER and SFEW datasets.

For modeling the spatiotemporal evolution of visual information, Kahou [26] presented the application of an RNN for modeling this spatiotemporal evolution via the aggregation of facial features to perform emotion recognition in video. Li [30] trained a CNN to extract both geometric and appearance features, and a long short-term memory (LSTM) to capture temporal and contextual information regarding facial expressions. The CNN-LSTM architecture enables a more comprehensive representation of facial expressions as it combines both spatial and temporal information.

To more effectively utilize information regarding the key parts that convey emotion, Ming et al. [31] proposed a facial expression recognition method incorporating an attention mechanism based on a CNN and LSTM. This model mined information regarding important regions more effectively than general CNN-LSTM-based models.

2.2. Audio Emotion Classification

Speech contains rich emotional information, including features such as tone and rhythm, which can convey emotions in addition to the textual information present. Acoustic features may be divided into low-level and high-level categories.

Low-level features refer to those extracted via time and frequency algorithms, which are usually categorized into three types: prosodic features [32], spectral features [33], and voice quality features [34]. Each single feature can only express emotional information from a particular aspect of speech and, as such, cannot effectively represent speech emotions. To improve the recognition of speech emotions, researchers typically fuse multiple single features [35].

High-level features refer to those extracted directly from the raw speech signal or via low-level features [36] using deep learning techniques, such as CNN [37] and LSTM [38,39]. These methods are capable of directly learning optimal feature representations from raw data and forming more abstract high-level features by combining low-level features, thereby effectively capturing hidden features within the data without the cumbersome task of manually extracting features.

Mao et al. [37] proposed the use of CNN for speech emotion recognition (SER) by learning affect-salient features. They divided the CNN training into two stages: the learning of local invariant features and the learning of affect-salient, discriminative features. Their approach led to robust recognition performance in complex scenes and outperformed several well-established SER features, such as spectrogram representation and local invariant features.

Lee et al. [38] proposed an RNN-based speech emotion recognition framework capable of accounting for long contextual effects in emotional speech. The weighted accuracy of this framework improved by up to 12% compared to the baseline. For effective classification and learning of multidimensional complex data (speech features that can be used for analysis, such as pitch, energy, formants, linear predictor coefficients (LPCs), linear frequency cepstral coefficients (LFCCs), MFCCs, and TEO), Kumbhar et al. [39] presented a speech emotion recognition system using the LSTM model and MFCC features. This model yields an observed area under the ROC curve of 0.55.

To reduce the dimensionality of the acoustic data and extract high-level features, Etienne et al. [40] employed a mixed CNN-LSTM architecture. They transformed the audio signal into a spectrogram, which served as input to the convolutional layers followed by recurrent ones. The best results were achieved with a choice of 4 convolutional and 1 Bi-directional LSTM (BiLSTM) layers.

CNN and RNN may lose some context information when dealing with long sequence data and fail to fully capture emotional features. The introduction of an attention mechanism helps these neural networks highlight important information. Atila et al. [41] proposed a novel attention-based 3D CNN-LSTM network for accurate speech-based emotion prediction. The network comprises six 3D convolutional layers, two batch normalization (BN) layers, five Rectified Linear Unit (ReLU) layers, three 3D max pooling layers, one attention layer, one LSTM layer, one flatten layer, one dropout layer, and two fully connected layers. The attention layer is connected to the 3D convolution layers, and the proposed method is highly efficient.

In recent years, some excellent self-supervised models have emerged in the field of speech [42,43,44,45], among which Wav2vec can capture information about the speaker and performs particularly well in ultra-low-resource cases [46].

2.3. Textual Emotion Classification

The Word2Vec model, proposed in 2013 [47,48], realized the distributed representation of words, and the resulting general features of words have been widely used for textual features. Meanwhile, LSTM has been employed to obtain deep emotional representation by capturing long-term context dependencies. Javed et al. [49] proposed two LSTM-based models: simple LSTM and emotion lexicon boost LSTM. The simple LSTM architecture comprised two hidden LSTM layers and a dense layer with softmax activation for emotion analysis. The emotion lexicon boost model enhanced the simple LSTM architecture by incorporating the NRC Hashtag emotion lexicon. Both LSTM networks show promising results. With the emergence of Bidirectional Encoder Representation from Transformers (BERT) [50], BERT has been introduced for textual features in emotion analysis and has achieved good results. Gou et al. [51] generated word-level and sentence-level vectors for text using BERT, inserted the word-level feature into BiLSTM for processing, and connected the output of BiLSTM with sentence-level features for emotion analysis of dialogue. Their method significantly outperformed the baselines.

CNN models can extract local features. Gui [52] learned representations of users and products, reviewed words using heterogeneous network embedding techniques, and employed CNN to detect product review sentiment polarity with the learned representations, achieving state-of-the-art performance. Chen [53] constructed a novel weakly supervised multimodal deep learning framework and trained a discriminative model from cheaply available emoticon annotations for multimodal prediction.

Even with the hybrid approach that leverages the powers of LSTM and CNN, the important information cannot always be better selected from word embeddings. Therefore, the attention mechanism is used with these neural networks. The attention mechanism can highlight important information from contextual information by assigning different weights [54]. Xie et al. [55] calculated attention scores using both the word vectors themselves and the feature of word vectors extracted by the LSTM network. They then integrated the word vectors with attention scores to input them into CNN for calculating emotion intensity. The multi-head attention mechanism from Transformer [56] enables the model to jointly attend to information from different representation subspaces at different positions. Akula et al. [57] enhanced word embeddings of input text with an attention module using L self-attention layers and H heads per layer. They passed the resultant features through a Gated Recurrent Unit (GRU) and a feed-forward layer for sarcasm detection.

2.4. Multimodal Emotion Classification

Unimodal emotion recognition has limitations in accurately detecting human emotions because of its reliance on a single type of sensory input. MEC combines multiple sources of data, such as facial expressions, voice, body posture, and linguistic cues, to more accurately detect and differentiate various emotional states. Multimodal fusion methods aim to combine information from multiple modalities to improve the performance of various tasks. Some popular multimodal fusion methods include early fusion [58,59,60], late fusion [15], and hybrid fusion [14,15].

Early fusion concatenates features from each modality and inputs the resultant joint representation into a classifier for emotion classification. Pérez-Rosas et al. [58] combined linguistic, acoustic, and visual features into a single feature vector, which was used to make a decision about the sentiment orientation of the utterance. Xu et al. [59] used the Object-VGG and Scene-VGG models to detect visual semantic features and extracted words that were important to sentiment with the visual-feature-guided attention mechanism. The visual and textual features were aggregated by using an early fusion layer to obtain the final multimodal representation for MEC.

Late fusion employs and trains a separate classifier for each modality and combines the output of each classifier to obtain the final prediction. Poria et al. [61] presented early and late fusion methods. In the case of early fusion, they concatenated textual and visual features and fed the resulting long vector into a Support Vector Machine (SVM) classifier. In the case of late fusion, they fed features of each modality into separate SVM classifiers and then combined their decisions with weights chosen experimentally. Yu et al. [62] used logistic regression to perform emotion prediction of the text and related images individually and performed late fusion for the probabilistic results using the average strategy.

The mixed fusion method integrates the advantages of early and late fusion methods and considers both the individual features of different modalities and the relevance among them. Cimtay et al. [15] used a hybrid fusion of faces with early fusion on electroencephalogram (EEG) and galvanic skin response (GSR) modalities. EEG and GSR modalities were used for estimating the level of arousal, and late fusion of EEG, GSR, and face modalities was used.

The mixed fusion method requires more computing resources and needs a more refined feature extraction and classifier construction for different modalities, which increases the difficulty and cost of implementation. Recently, many research works have focused on attention-based fusion [63,64,65] and its variants, such as self-attention [66,67], multi-head attention [68,69], and Transformer [70,71]. The attention-based fusion integrates the advantages of early fusion and late fusion and compensates for their shortcomings [64]. Li et al. [63] proposed an attention-based multimodal fusion approach. They first fused the audio and visual modalities, and then fused the resultant fusion result with the text modality. Their approach resolved the problem of asynchronous inputs by using two attention layers to align the inputs and learn their relatedness in an orderly manner. Thao et al. [67] designed a self-attention-based neural network for predicting the affective responses of movie viewers from different input modalities. They implemented a self-attention mechanism using N identical layers, each of which included two sublayers: a multi-head self-attention followed by a feed-forward layer. Each of these sublayers was enclosed by a residual connection, accompanied by layer normalization. Wang et al. [64] proposed a cross-attention asymmetric fusion module, which utilized information matrices of the acoustic and visual modality as weights to strengthen the textual modality. Xie et al. [70] employed a transformer-based cross-modality fusion with a robust deep learning architecture to estimate the emotion during a conversation.

The aforementioned studies reported varying fusion methods and achieved great successes while having a common limitation: insufficient consideration of the differences between IMAs and IUAs, as well as the impacts they may have on emotion classification models. In this paper, we further discuss the impacts of IMAs and IUAs based on existing research and propose a new multimodal fusion method.

3. Methods

Given a video clip that contains multimodal information such as text, audio, and vision, the task aims to predict emotional annotation of the clip, i.e., positive, negative, and neutral. The key is to extract and fuse the representations of each modality. To achieve this, we propose a Conformer-based MEC model (Uni2Mul). Figure 3 provides a detailed illustration of our model. The visual, acoustic, and textual features are extracted from the video data using CLIP (ViT-B/32), Wav2Vec, and BERT (bert-base-chinese), three pre-trained models, respectively. These features are then fed into unimodal neural networks to predict IUAs. Meanwhile, hidden representations are fetched from these unimodal neural networks and fused to predict IMAs. More details about our model are provided below.

Figure 3. Conformer-based MEC model (Uni2Mul). The model contains two stages: emotional classification models for each modality (left part) aim to optimize unimodal representation, and multi-task fusion network with Conformer (right part) performs multimodal fusion and MEC. The different colors correspond to different modalities. Pink, green, and purple represent the visual, acoustic, and textual modalities respectively. $M$: Multimodal, $V$: Vision, $A$: Audio, $T$: Text, $H$: representation from hidden layer, $Y$: emotional annotation, $q$: query, $v$: value, FC: Fully Connected layer. GAP: Global Average Pooling1D, ⊗: operation of multiplication, ⊕: operation of addition.

3.1. Unimodal Neural Networks

3.1.1. Vision

CLIP is a transferable visual model trained from 400 million (image, text) pairs collected from the internet [72]. We use this pre-trained model as a visual feature extractor to make full use of its prior knowledge, which includes not only visual but also textual information. To extract better unimodal emotional representations, we construct an emotion classification model for the visual modality using CNN and LSTM, inspired from [30].

Specifically, we use CLIP to extract features from images and define the visual feature as $X_V$. We then feed $X_V$ into three CNN layers, whose operation process can be formulated as

$$R_{cnn}^V=W_{cnn}^V·X_V+b_{cnn}^V,$$

(5)

where $R_{cnn}^V$ represents the output of CNNs and can be defined as $R_{cnn}^V=[r_0^V,r_1^V,\cdots ,r_t^V]$, where $t$ is the timestep. $W_{cnn}^V$ denotes the weight matrix and $b_{CNN}^V$ represents the bias vector.

Next, the LSTM layer takes the outputs of CNNs as input and performs the following operations:

$$\begin{gathered} i_t^V=\sigma \left(W_i^V\left[h_{t-1}^V:r_t^V\right]+b_i^V\right), \\ {f}_t^V=\sigma \left(W_f^V\left[h_{t-1}^V:r_t^V\right]+b_f^V\right), \\ {o}_t^V=\sigma \left(W_o^V\left[h_{t-1}^V:r_t^V\right]+b_o^V\right), \\ {C}_t^V=f_t^V·C_{t-1}^V+i_t^V·tanh\left(W_c^V\left[h_{t-1}^V:r_t^V\right]+b_c^V\right), \\ {H}_t^V=o_t^V·tanh\left(C_t^V\right), \end{gathered}$$

(6)

where $i_t^V$, $f_t^V$, and $o_t^V$ represent the output of the input gate, forget gate, and output gate, respectively, in the LSTM; $C_t^V$ represents the current moment cell state; and $H_t^V$ represents the hidden output of the LSTM and can be defined as $H_t^V=[h_0^V,h_1^V,\cdots ,h_t^V]$ (where $t$ is timestep). $\sigma (·)$ denotes the sigmoid activation function and $tanh(·)$ is the hyperbolic tangent function. $W_i^V$, $W_f^V,W_o^V,$ and $W_c^V$ represent the weight matrices, while $b_i^V,b_f^V{,b}_o^V$, and $b_c^V$ represent the bias vectors.

Finally, we define the hidden representation as $H_V=H_t^V$, which is fed into a fully connected layer for visual emotional classification. The operation process of the visual emotional classification can be formulated as

$$Y_V=W_V·H_V+b_V,$$

(7)

where $Y_V$ represents the annotation of the visual modality, $W_V$ represents the weight matrix, and $b_V$ represents the bias vector.

3.1.2. Audio

Wav2Vec 2.0 is a self-supervised framework for speech representation learning. It can capture the information about the speaker and performs well in speech recognition tasks, especially in ultra-low-resource cases [46]. Therefore, we use it as a feature extractor for acoustic data. Inspired by Atila et al. [41], we propose a network that consists of a CNN, followed by a self-attention mechanism and a Bi-LSTM.

Firstly, we employ Wav2Vec 2.0 to extract acoustic features from the waveform signal and define the acoustic feature as $X_A$. We then feed $X_A$ into a CNN layer:

$$R_{cnn}^A=W_{cnn}^A·X_A+b_{cnn}^A,$$

(8)

where $R_{cnn}^A$ represents the CNN output, $W_{cnn}^A$ represents the weight matrix, and $b_{cnn}^A$ represents the bias vector.

We use the “Scaled Dot-Product Attention” from [56] as the attention mechanism. The matrix of outputs is computed as

$$\begin{gathered} Qu{e}_A=W_{Que}^A·R_{cnn}^A, \\ Ke{y}_A=W_{Key}^A·R_{cnn}^A, \\ scor{e}_A=softmax(W_{attn}^A·tanh\left(Qu{e}_A·Ke{y}_A\right)), \\ {R}_{attn}^A=scor{e}_A·(W_{Val}^V·Va{l}_A), \end{gathered}$$

(9)

where $Qu{e}_A$, $Ke{y}_A$, and $Va{l}_A$ stand for query, key, and value, respectively. $Va{l}_A$ is equal to $Ke{y}_A$, and $scor{e}_A$ represents the weight on the value. $Softmax$ is the softmax function, and $tanh(·)$ is the hyperbolic tangent function. $R_{attn}^A$ is the output of the attention module and can be defined as $R_{attn}^A=[r_0^A,r_1^A,\cdots ,r_t^A]$, where $t$ is the timestep. $R_{attn}^A$ is fed into BiLSTM to extract deep representations. $W_{Que}^A, W_{Key}^A,W_{attn}^A$, and $W_{Val}^V$ represent the weight matrices. The outputs of BiLSTM can be computed as follows:

$$\begin{gathered} i_t^A=\sigma \left(W_i^A\left[h_{t-1}^A:r_t^A\right]+b_i^A\right), \\ {f}_t^A=\sigma \left(W_f^A\left[h_{t-1}^A:r_t^A\right]+b_f^A\right), \\ {o}_t^A=\sigma \left(W_o^A\left[h_{t-1}^A:r_t^A\right]+b_o^A\right), \\ {C}_t^A=f_t^A·C_{t-1}^A+i_t^A·tanh\left(W_c^A\left[h_{t-1}^A:r_t^A\right]+b_c^A\right), \\ {H}_t^A=o_t^A·tanh\left(C_t^A\right), \end{gathered}$$

(10)

where $i_t^A$, $f_t^A$, and $o_t^A$ represent outputs of the input gate, forget gate, and output gate, respectively, in LSTM; $C_t^A$ represents the current moment cell state; and $H_t^A$ represents the hidden output of LSTM and can be defined as $H_t^A=[h_1^A,\cdots ,h_t^A]$, where $t$ is the timestep. $\sigma (·)$ denotes the sigmoid activation function and $tanh(·)$ is the hyperbolic tangent function. $W_i^A$,$W_f^A, W_o^A,$ and $W_c^A$ represent the weight matrices and $b_i^A,b_f^A{,b}_o^A$, and $b_c^A$ represent the bias vectors.

Finally, we feed the output of BiLSTM into a fully connected layer for acoustic emotional classification. The operation process of the acoustic emotional classification can be formulated as

$$\begin{gathered} H_A=[\overrightarrow{H_t^A}:\overleftarrow{H_t^A}], \\ {Y}_A=W_A·H_A+b_A, \end{gathered}$$

(11)

$H_A$ and $Y_A$ represent hidden representation and annotation of the acoustic modality, respectively. $W_A$ represents the weight matrix, and $b_A$ represents the bias vector.

3.1.3. Text

We use BERT to obtain embeddings from texts. We do not use word segmentation tools, due to the characteristics of BERT. We add two unique tokens to indicate the beginning and the end for each text. We define the textual feature extracted by BERT as $X_T$.

During the training of the unimodal emotional classification models, we find that the performance of the textual modality emotional classification model is superior to the other modalities, so we want to extract more information from the textual modality. Inspired by [54], we design the CNN-attention-based model.

Specifically, we generate the query and key for the textual modality using two 1D CNNs. We use the “Scaled Dot-Product Attention” from [56] as the attention mechanism, too. The matrix of outputs is computed as

$$\begin{gathered} Qu{e}_T=W_{Que}^T·X_T+b_{Que}^T, \\ Ke{y}_T=W_{Key}^T·X_T+b_{Key}^T, \\ scor{e}_T=softmax(W_{attn}^T·tanh\left(Qu{e}_T·Ke{y}_T\right)), \\ {R}_{attn}^T=scor{e}_T·(W_{Val}^T·Va{l}_T), \end{gathered}$$

(12)

where $Qu{e}_T$, $Ke{y}_T$, and $Va{l}_T$ stand for query, key, and value, respectively. $Va{l}_T$ is equal to $Ke{y}_T$, and $scor{e}_T$ represents the weight on the value. $Softmax$ is the softmax function and $tanh(·)$ is the hyperbolic tangent function. $R_{attn}^T$ is the output of the attention module. $W_{Que}^T, W_{Key}^T,W_{attn}^T$, and $W_{Val}^T$ represent the weight matrices. $b_{Que}^T$ and $b_{Key}^T$ represent the bias vectors.

Subsequently, we conduct a global average pooling operation on $Qu{e}_T$ and $R_{attn}^T$, and concatenate them together to formulate the textual representation, $H_T$. $H_T$ is computed as follows:

$$H_T=[GAP\left(Qu{e}_T\right):GAP(R_{attn}^T\left)\right],$$

(13)

where $GAP(·)$ represents the global average pooling operation.

Lastly, we send $H_T$ to a fully connected layer for emotional classification. We formulate the classification operation as follows:

$$Y_T=W_T·H_T+b_T,$$

(14)

where $Y_T$ represents the annotation of the textual modality, $W_T$ represents the weight matrix, and $b_T$ represents the bias vector.

These networks for each modality are trained with IUAs to ensure the difference of representation between the inter-modal information, and are then saved to construct Uni2Mul framework.

3.2. Multi-Task Multimodal Fusion Network with Conformer

Inspired by Gulati et al. [73], we propose the Conformer-based fusion method. The Conformer [73] (see Figure 4) is a convolution-augmented Transformer for speech recognition. It can combine convolution neural networks and Transformers to model both local and global dependencies of audio sequences in a parameter-efficient way. The model exhibits better accuracy with fewer parameters than previous work on the LibriSpeech dataset.

Figure 4. Conformer architecture. (a) is Conformer architecture, (b) is feed-forward module, (c) is convolution module, and (d) is multi-head self-attention module. ⊕: operation of addition.

The Conformer block (see Figure 4a) is composed of four modules stacked together, i.e., a feed-forward module, a self-attention module, a convolution module, and a second feed-forward module at the end. In the multi-head self-attention module (see Figure 4d), we employ pre-norm residual units with dropout, which helps in training and regularizing deeper models. The convolution module (see Figure 4c) is stacked in the order of layernorm layer, convolution layer, gated linear unit (GLU), and so on. Batchnorm is deployed just after the convolution for training deep models. As for the feed-forward module (see Figure 4b), it starts with a layernorm layer, which is followed by two interleaved linear layers and dropout layers. We also apply Swish activation to regularize the network.

Especially, we first load the unimodal models saved above and set these models as untrainable to obtain unimodal representations from their hidden layers. We then concatenate these representations to form the multimodal representation, which can be formulated as

$$X_M=\left[H_V:H_A:H_T\right],$$

(15)

We use a Conformer-based method to fuse the multimodal representation and predict the emotional annotation of video. Especially, we send $X_M$ into the first feed-forward module and define the result as $R_{FF{M}_1}$, which is fed into the multi-head self-attention module. The operation of the multi-head self-attention module in the Conformer can be formulated as follows:

$$\begin{gathered} R_{LN}^M=LN\left(R_{{ffm}_1}\right), \\ Qu{e}_M=W_{Que}^M·R_{LN}^M, \\ Ke{y}_M=W_{Key}^M·R_{LN}^M, \\ scor{e}_M=softmax(W_{attn}^M·tanh\left(Qu{e}_M·Ke{y}_M\right)), \\ {R}_{attn}^M=scor{e}_M·(W_{Val}^M·Va{l}_M), \\ {R}_{mha}^M=W_{MHA}^M·Concat(R_{{attn}_1}^M,\cdots ,R_{{attn}_h}^M), \\ {R}_{DP}^M=DP\left(R_{mha}^M\right), \end{gathered}$$

(16)

where $LN(·)$ is the layer normalization operation, and $Qu{e}_M$, $Ke{y}_M$, and $Va{l}_M$ stand for query, key, and value, respectively. $Va{l}_M$ is equal to $Ke{y}_M$, and $scor{e}_M$ represents the weight on the value. $Softmax$ is the softmax function and $tanh(·)$ is the hyperbolic tangent function. $R_{attn}^M$ is the output of the attention module of one head and $R_{mha}^M$ is the output of the multi-head attention module. $W_{Que}^M, W_{Key}^M,W_{attn}^M$, and $W_{Val}^M$ represent the weight matrices. $Concat\left(·\right)$ represents the concatenate operation and $DP(·)$ represents the dropout operation.

$R_{DP}^M$ is then fed into the convolution module, and the operations in the convolution module can be formulated as

$$\begin{gathered} R_{{cnn}_1}^M=W_{{cnn}_1}^M·LN\left(R_{DP}^M\right)+b_{{cnn}_1}^M, \\ {R}_{{cnn}_2}^M=W_{{cnn}_2}^M·BN\left(GLU\right(R_{{cnn}_1}^M\left)\right)+b_{{cnn}_2}^M, \\ {R}_{CON}=DP\left(R_{{cnn}_2}^M\right)+H_{DP}^M, \end{gathered}$$

(17)

where $LN(·)$ is the layer normalization operation, $GLU(·)$ represents the gated linear unit operation, $BN(·)$ is the batch normalization operation, and $DP(·)$ represents the dropout operation. $R_{{cnn}_1}^M$ and $R_{{cnn}_2}^M$ are convolution operation results. $W_{{cnn}_1}^M$ and $W_{{cnn}_2}^M$ represent the weight matrices, and $b_{{cnn}_1}^M$ and $b_{{cnn}_2}^M$ represent the bias vectors. We feed $H_{CON}$ into the second feed-forward module and define the result as $R_{{ffm}_2}$.

Based on the research results of CH-SIMS, the accuracy of the textual and visual modalities in unimodal emotion recognition models is relatively high, which we refer to as the dominant modalities. In contrast, the accuracy of the acoustic modality is relatively low, which we refer to as the less salient modality. During the process of multimodal fusion, in order to avoid neglecting information from less salient modalities, we utilize a multi-task framework for multimodal fusion. In other words, our multimodal model has four outputs, which include the unimodal classification results for the visual, acoustic, and textual modalities, as well as the overall MEC result. The operation process of the MEC can be formulated as

$$\begin{gathered} Y_M=W_M·GAP\left(R_{{ffm}_2}\right)+b_M, \\ {Y}_V=W_V·H_V+b_V, \\ {Y}_A=W_A·H_A+b_A, \\ {Y}_T=W_T·H_T+b_T, \end{gathered}$$

(18)

where $Y_M$ represents the annotation for each video clip; $W_M, W_V, W_A,$ and $W_T$ represent the weight matrices; and $b_M, b_V, b_A$, and $b_T$ represent the bias vectors. $GAP(·)$ represents the global average pooling operation.

4. Experiment

4.1. Dataset

The CH-SIMS [13] dataset collects 60 videos from movies, TV series, and variety shows, and divides these videos into 2281 video clips. Each video clip has three unimodal annotations and one multimodal annotation. Given that there are only a few datasets containing unimodal annotations currently available, we only conduct experiments on the CH-SIMS dataset.

4.2. Parameters’ Setting

All experiments are carried out using a single NVIDIA (Santa Clara, CA, USA) Quadro P520 card. We adopt Adam as the optimizer with a learning rate of 1 × 10⁻⁴, and set the number of epochs to 30 and batch size to 32. All models are trained using sparse categorical cross-entropy on each softmax output. We evaluate the models with accuracy (Acc.) and F1 score. Acc. is equal to the proportion of correctly classified samples to the total number of samples, and can be formulated as follows:

$$\mathrm{A}\mathrm{c}\mathrm{c}.=\frac{TP+TN}{TP+FN+TN+FP},$$

(19)

$TP$ is true positives, $TN$ is true negatives, $FP$ is false positives, and $FN$ is false negatives.

Precision is the ability of the classifier not to label as positive a sample that is negative, and recall is the ability of the classifier to find all the positive samples. The F1 score can be interpreted as a weighted harmonic mean of the precision and recall, and can be formulated as follows:

$$\begin{gathered} precision=\frac{TP}{TP+FP}, \\ recall=\frac{TP}{TP+FN}, \\ {\mathrm{F}}_1=\frac{2\times precision\times recall}{precision+recall}, \end{gathered}$$

(20)

Higher values of Acc. and F1 score represent better performance. We use the “save_best_only = True” parameter in Keras to save only the best model during training. This parameter ensures that the model weights are saved whenever the monitored metric improves and overwrites the previously saved weights only if there is an improvement. Thus, at the end of training, the saved weights correspond to the best performing model on the validation set.

In addition to the general parameters mentioned above, personalized parameters for unimodal and multimodal models are set as follows:

Vision: In the process of selecting the visual timestep, we conduct several pilot experiments to access the impact of different timesteps on the performance of the visual models. We observe that increasing the timestep results in a larger memory footprint for the feature matrix, without a significant improvement in model performance. To strike a balance between computational complexity and performance, we decide to set the visual timestep to 10. The output length of CLIP is 512. Therefore, the input shape of the visual modality is $10\times 512$. The following three 1D convolution layers have 64 filters, and the kernel sizes are set to 3, 1, and 3, respectively, with strides being set to 1. The subsequent BiLSTM has 32 units.

Audio: Similarly, in line with our previous findings for the visual timestep, our pilot experiments demonstrate that setting the acoustic timestep to 128 not only maintains a good level of performance but also showcases the effectiveness of this parameter. As a result, we set the acoustic timestep to 128. The output length of Wav2Vec 2.0 is 512. Thus, the input shape of the acoustic modality is $128\times 512$. The following 1D convolution layer has 64 filters, and the kernel size and strides are set to 3 and 2, respectively. The subsequent BiLSTM has 32 units, and dropout is used with a rate of 0.5 to prevent overfitting.

Text: The longest sentence in the CH-SIMS dataset has 36 tokens. The output length of BERT used in this paper is 512. Hence, the input shape of the textual modality is $36\times 512$. The subsequent two 1D convolution layers have 64 filters, and both the kernel size and strides are set to 1.

Multimodal: The number of attention heads in our multi-head attention module in Conformer is 2. Dropout is used with a rate of 0.5 to prevent overfitting. The convolution layer in Conformer has 64 filters, and the kernel size is set to 3. The loss weights for multimodal, vision, audio, and text are 0.3, 0.2, 0.2, and 0.3, respectively.

4.3. Experimental Results

We employ the models described in [13] as our initial benchmark. Following established research practices, we conduct experiments involving single-task and multi-task scenarios to evaluate the fusion of multimodal data. We test two variations of unimodal models, namely pre-trained and without pre-training, within both the single-task and multi-task frameworks. The evaluation of these models is based on the Acc. metric, and the outcomes are presented in Table 1, comparing our models to the baseline performance on the CH-SIMS dataset.

Table 1. The results of the baseline and our models on the CH-SIMS dataset. The models with * are multi-task models, extended from single-task models by introducing independent unimodal annotations. “S” in model names stands for “Single-task”, while “M” denotes “multi-task”. The model names with “w/o pre-train” indicate that the corresponding unimodal models have not been pre-trained, while the ones without “w/o pre-train” indicate that the unimodal models have been pre-trained. “Conformer” in model names means the fusion method of the models.

Model		Acc.
baseline	EF-LSTM [74]	51.73
	MFN [75]	63.89
	MULT [76]	65.03
	LF-DNN [13]	66.91
	MLF-DNN * [13]	69.06
	LMF [77]	64.38
	MLMF * [13]	67.70
	TFN [78]	64.46
	MTFN * [13]	69.02
ours	Uni2Mul-S-Conformer (w/o pre-train)	72.21
	Uni2Mul-S-Conformer	76.15
	Uni2Mul-M-Conformer (w/o pre-train)	73.30
	Uni2Mul-M-Conformer	76.81

The results indicate that all four of our models outperform the baseline models mentioned in the CH-SIMS paper in terms of Acc. Particularly, the Uni2Mul-M-Conformer model exhibits an accuracy improvement of 7.75 points compared to the MLF-DNN∗ baseline.

We can observe that the pre-training method in stage one improves the performance of the MEC models. For example, Uni2Mul-S-Conformer demonstrates a higher Acc. than Uni2Mul-S-Conformer (w/o pre-train), and Uni2Mul-M-Conformer shows a higher Acc. than Uni2Mul-M-Conformer (w/o pre-train). These results suggest that pre-trained unimodal models can capture good representations and pass them unchanged to the fusion network for MEC.

Additionally, a multi-task framework can achieve complementary information among multiple related tasks, improving the generalization ability of the MEC models. For example, Uni2Mul-M-Conformer (w/o pre-train) outperforms Uni2Mul-S-Conformer (w/o pre-train), and Uni2Mul-M-Conformer outperforms Uni2Mul-S-Conformer.

These findings demonstrate that Uni2Mul models can achieve superior performance, and our proposed methods for representation extraction and fusion strategy prove effective for MEC.

5. Discussion

5.1. Ablation Study

5.1.1. Unimodal Representation

To validate our hypotheses, we conducted the following experiments to examine the impact of different feature extraction methods and neural network structures on the performance of unimodal emotion classification models. We trained individual models for each modality, utilizing two types of annotations: IMAs and IUAs. The evaluation of these models was based on the Acc. and F1 score. The results are shown in Table 2. To compare the performance differences more clearly between IMA and IUA models, we plotted Figure 5 based on the data in Table 2.

Table 2. Performance of unimodal models in stage one. “ATTN” in the model names represents attention mechanism. Mel is the Mel spectrogram feature.

Modality	Name of Model	Feature	Acc. (IMAs)	F1 (IMAs)	Acc. (IUAs)	F1 (IUAs)
V	CNN	Image	48.36	44.56	45.08	44.18
		CLIP	63.68	63.11	66.96	65.41
	CNN-LSTM	Image	54.49	38.67	51.20	35.06
		CLIP	67.40	62.05	68.05	67.58
	CNN-ATTN-LSTM	Image	54.27	38.18	51.20	34.68
		CLIP	65.43	59.97	67.83	65.60
A	BiLSTM	Mel	51.20	44.35	50.77	42.14
		Wave2Vec	52.52	44.94	50.98	46.17
	CNN-BiLSTM	Mel	51.42	42.70	51.20	42.27
		Wave2Vec	53.61	41.62	53.17	46.84
	CNN-ATTN-BiLSTM	Mel	52.95	43.24	49.45	44.36
		Wave2Vec	54.27	40.48	53.17	50.06
T	BiLSTM	Word2Vec	54.70	43.98	53.39	44.53
		BERT	69.80	65.64	74.40	73.18
	ATTN-BiLSTM	Word2Vec	54.92	43.16	54.05	44.62
		BERT	70.24	65.96	74.62	74.41
	CNN-ATTN	Word2Vec	55.36	42.40	53.39	45.95
		BERT	70.02	66.87	75.27	75.10

Figure 5. Performance comparison chart of unimodal models in stage one. (a–c) depict the model performance metrics, Acc. and F1 score, of three visual models: CNN, CNN-LSTM, and CNN-ATTN-LSTM. (d–f) depict the model performance metrics, Acc. and F1 score, of three acoustic models: BiLSTM, CNN-BiLSTM, and CNN-ATTN-BiLSTM. (g–i) depict the model performance metrics, Acc. and F1 score, of three textual models: BiLSTM, ATTN-BiLSTM, and CNN-ATTN. V: Vision, A: Audio, T: Text.

In the visual modality, we compared our unimodal model, the CNN-LSTM-based model, with the CNN-based and CNN-ATTN-LSTM-based models. Two types of features were tested for each model: video images and the outputs of CLIP. As presented in Table 2 and Figure 5a–c, the Acc. (IUAs) consistently outperformed Acc. (IMAs), and our CNN-LSTM-based model yielded the highest performance among the three models. Specifically, the CNN-LSTM-based model demonstrated a 0.65 percentage point improvement in Acc. (IUAs) and a 5.53 percentage point improvement in F1 (IUAs) when compared to Acc. (IMAs) and F1 (IMAs). Furthermore, CLIP proved to be an effective method for visual feature extraction. Taking the CNN-LSTM model trained using IUAs as an example, employing the output of CLIP as the input feature resulted in a 16.85 percentage point increase in Acc. and a 32.52 percentage point increase in F1 score, compared to using images as the input. This improvement can be attributed to the ability of CLIP to leverage prior knowledge of both vision and text, allowing it to extract textual information along with visual information.

For the acoustic modality, we compared our unimodal model, the CNN-ATTN-BiLSTM-based model, with the BiLSTM-based and CNN-BiLSTM-based models. Two types of features were examined for each model: Mel features and outputs of Wav2vec. As indicated in Table 2 and Figure 5d–f, most models exhibited higher F1 score for IUAs compared to IMAs, and our CNN-ATTN-BiLSTM-based model delivered the best performance among the three models. Additionally, Wav2Vec 2.0 proved to be an effective method for acoustic feature extraction. Although the CNN-ATTN-BiLSTM-based model had a 1.1 percentage point lower Acc. (IUAs) compared to Acc. (IMAs), it demonstrated a 9.58 percentage point higher F1 score for IUAs compared to F1 score for IMAs.

In the textual modality, we compared our unimodal model, the CNN-ATTN-based model, with the BiLSTM-based and ATTN-BiLSTM-based models. Two types of features were tested for each model: outputs of Word2Vec and BERT. As depicted in Table 2 and Figure 5g–i, in most cases, the F1 score for IUAs outperformed the F1 score for IMAs, and in some instances, the Acc. (IUAs) surpassed Acc. (IMAs). Our CNN-ATTN-based model yielded the best performance among the three models. As expected, BERT proved to be an effective method for textual feature extraction.

The results shown in Table 2 and Figure 5 are consistent with our idea: unimodal models trained using IUAs can learn better emotional representation.

5.1.2. Multimodal Fusion

We compared four fusion strategies: concatenate, multi-head attention, Transformer, and Conformer. The evaluation of these fusion strategies was conducted using Acc. (accuracy) and F1 score. The results are presented in Table 3.

Table 3. Results for MEC with different fusion strategies in stage two.

Name of Model	Acc. (IMAs)	F1 (IMAs)	Acc. (IUAs)	F1 (IUAs)
Uni2Mul-S-Concatenate (w/o pre-train)	66.52	65.47	68.71	64.60
Uni2Mul-S-Concatenate	71.99	70.39	73.96	73.17
Uni2Mul-M-Concatenate (w/o pre-train)	70.46	67.95	69.58	68.83
Uni2Mul-M-Concatenate	72.87	70.35	75.05	74.36
Uni2Mul-S-Attention (w/o pre-train)	69.15	65.61	67.61	61.37
Uni2Mul-S-Attention	72.65	71.53	73.96	73.26
Uni2Mul-M-Attention (w/o pre-train)	70.24	64.35	70.90	69.56
Uni2Mul-M-Attention	72.87	71.16	76.37	75.26
Uni2Mul-S-Transformer (w/o pre-train)	68.71	63.64	65.21	59.62
Uni2Mul-S-Transformer	71.77	69.17	75.05	73.87
Uni2Mul-M-Transformer (w/o pre-train)	69.58	63.77	71.77	67.96
Uni2Mul-M-Transformer	73.09	70.06	76.59	74.68
Uni2Mul-S-Conformer (w/o pre-train)	69.37	64.57	72.21	68.56
Uni2Mul-S-Conformer	72.21	70.62	76.15	75.00
Uni2Mul-M-Conformer (w/o pre-train)	71.55	68.40	73.30	71.84
Uni2Mul-M-Conformer	73.09	71.50	76.81	75.08

Based on the overall trend observed in Table 3, Conformer emerged as the most effective fusion method, followed by Transformer, multi-head attention, and concatenate. Specifically, the Uni2Mul-M-Conformer model achieved an Acc. (IUAs) that was 0.22, 0.44, and 1.76 percentage points higher than those of Uni2Mul-M-Transformer, Uni2Mul-M-Attention, and Uni2Mul-M-Concatenate, respectively.

In most cases, Acc. (IUAs) outperformed Acc. (IMAs), and F1 (IUAs) outperformed F1 (IMAs). The Uni2Mul-M-Conformer model demonstrated an Acc. (IUAs) that was 3.72 percentage points higher than that of Acc. (IMAs), while F1 (IUAs) was 3.58 percentage points higher than that of F1 (IMAs). These results align precisely with our expectations.

Furthermore, the multimodal fusion models constructed using pre-trained unimodal models exhibited higher Acc. and F1 score compared to those built without pre-training. Additionally, the multi-task framework outperformed the single-task framework, which is consistent with previous research findings.

5.2. Visualization

5.2.1. Visualization of Hidden Representations

To assess the impact of IMAs and IUAs on model feature extraction, we utilized t-SNE to visualize the hidden representations of our models (refer to Figure 6).

Figure 6. Unimodal and multimodal representation. (a,b) are the results from unimodal models trained using IMAs and IUAs, respectively; (c,d) are the results from models trained using IMAs and IUAs with Conformer-based fusion strategy. In (a,b), the markers of different colors represent different modalities, and shapes filled with different colors represent semantic spaces of different modalities. In (c,d), the markers of different colors represent different emotional categories, and shapes filled with different colors represent classification distribution. V: Vision, A: Audio, T: Text.

For unimodal representations, models trained using IMAs displayed irregular clustering in the semantic space with diffuse distributions (see Figure 6a). Conversely, models trained using IUAs formed more regular, spherical-like clusters (see Figure 6b) that adhered to a uniform Gaussian distribution.

In terms of multimodal representations, the Conformer-based fusion model trained with IMAs were not well concentrated, with multiple overlapping regions, resulting in suboptimal classification performance and limited generalizability (Figure 6c). In contrast, the model trained with IUAs exhibited distinct classification boundaries and minimal overlap in the classification distribution, yielding comparatively superior classification performance and adequate generalizability (see Figure 6d).

5.2.2. Visualization of Attention Weights

In stage one, we obtained a 64-dimensional visual feature vector, a 64-dimensional acoustic feature vector, and a 128-dimensional textual feature vector. In stage two, we concatenated these three vectors into a 256-dimensional multimodal vector and fed it into the fusion network. To explore the differences in fusion methods, we visualized the attention weights of three fusion strategies: multi-head attention (see Figure 7a), Transformer (see Figure 7b), and Conformer (see Figure 7c). In Figure 7a–c, we constructed an attention weight matrix for the aforementioned 256-dimensional multimodal feature vectors. The horizontal axis positions correspond to visual, acoustic, and textual feature components from left to right, and the vertical axis positions correspond to them from top to bottom. The numerical values (different colors) of each position in the matrix represent the degree of correlation of the corresponding positions, and the diagonal represents autocorrelation. The higher the value, the brighter the color. The weights of different heads were averaged.

Figure 7. Attention weights of three fusion methods: (a) for “multi-head attention”, (b) for “Transformer”, and (c) for “Conformer”. The lighter the color, the higher the weight.

In Figure 7a, we observed a few scattered regions with relatively high weights in the distribution, indicating that the multi-head attention fusion model successfully identified important emotional features. In Figure 7b, the weights were predominantly higher along the main diagonal, suggesting that the Transformer model discovered significant intra-modal relationships. Although there were a few high weight regions near the bottom left corner, their weights were lower compared to those along the main diagonal. This implies that the model had less proficiency in fusing inter-modal information and primarily focused on intra-modal relationships. Moving on to Figure 7c, we notice that the main diagonal had a wider area of prominence compared to Figure 7b. Additionally, some vertical highlight regions appeared at the bottom left of the main diagonal. These observations indicate that the Conformer model, benefiting from its convolutional kernel, assigns higher weights to intra-modal relationships and exhibits greater confidence in inter-modal relationships. These findings align with our expectations and suggest that the Conformer network serves as an effective fusion model.

5.2.3. Visualization of Confusion Matrix

To provide a more intuitive representation of the impact of IMAs, IUAs, and fusion strategies on the performance of multi-modal fusion models, we visualize the confusion matrices of these models. The results are shown in Figure 8.

Figure 8. Confusion matrix for MEC models. (a,c,e,g) are from models trained with IMAs, while (b,d,f,h) are from models trained with IUAs. (a,b) belong to the concatenate-based fusion models, (c,d) belong to the multi-head attention-based fusion models, (e,f) belong to the transformer-based fusion models, and (g,h) belong to the Conformer-based fusion models.

Comparing the left side of Figure 8a,c,e,g with the right side Figure 8b,d,f,h, it can be observed that regardless of the fusion strategy employed, models trained with IUAs outperform those trained with IMAs. Models trained with IMAs tend to have higher recognition accuracy for the negative class but lower accuracy for the positive and neutral classes. On the other hand, models trained with IUAs exhibit better recognition capabilities for both the positive and neutral classes, especially those based on the Conformer fusion method, which show improvements in recognition accuracy for all three classes.

This research, however, has several limitations. The first limitation is the lack of additional datasets for extensive validation. The second limitation is that CH-SIMS is a dataset with imbalanced samples, specifically a small number of positive, especially neutral emotion samples. The adaptability of the model to imbalanced data can also be improved.

6. Conclusions

In this paper, we propose a robust Conformer-based MEC model called Uni2Mul, which focuses on optimizing unimodal representation and multimodal fusion. We divide the implementation of Uni2Mul into two stages. In stage one, we construct individual unimodal neural networks for each modality and train them using IUAs to optimize the unimodal representations. This results in pre-trained unimodal models with superior performance. In stage two, we concatenate the hidden representations of these pre-trained unimodal models and feed the concatenated feature into the Conformer-based fusion network. This fusion network includes a sub-task that predicts IUAs for MEC. We also perform ablation experiments for the two stages separately. For stage one, we construct three different neural network structures for each modality and train these networks using IUAs and IMAs, respectively. For stage two, we try four fusion methods: concatenate, multi-head attention, Transformer, and Conformer. We summarize our overall findings as follows:

(1)

Unimodal models trained using IUAs can learn more differentiated information and improve the complementarity between modalities compared to those trained using IMAs.

(2)

The hidden representations of the pre-trained unimodal models serve as effective inputs for the fusion network. This ensures that the differentiated information learned using the unimodal models is passed unchanged to the fusion network.

(3)

The Conformer module, with its multi-head attention mechanism and convolutional kernel, excels in paying attention to important intra-modal information and capturing inter-modal relationships. It is the best among the four fusion strategies mentioned above.