Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

Abstract

In the real world, multimodal sentiment analysis (MSA) enables the capture and analysis of sentiments by fusing multimodal information, thereby enhancing the understanding of real-world environments. The key challenges lie in handling the noise in the acquired data and achieving effective multimodal fusion. When processing the noise in data, existing methods utilize the combination of multimodal features to mitigate errors in sentiment word recognition caused by the performance limitations of automatic speech recognition (ASR) models. However, there still remains the problem of how to more efficiently utilize and combine different modalities to address the data noise. In multimodal fusion, most existing fusion methods have limited adaptability to the feature differences between modalities, making it difficult to capture the potential complex nonlinear interactions that may exist between modalities. To overcome the aforementioned issues, this paper proposes a new framework named multimodal-word-refinement and cross-modal-hierarchy (MWRCMH) fusion. Specifically, we utilized a multimodal word correction module to reduce sentiment word recognition errors caused by ASR. During multimodal fusion, we designed a cross-modal hierarchical fusion module that employed cross-modal attention mechanisms to fuse features between pairs of modalities, resulting in fused bimodal-feature information. Then, the obtained bimodal information and the unimodal information were fused through the nonlinear layer to obtain the final multimodal sentiment feature information. Experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets demonstrated that the proposed approach outperformed other comparative methods, achieving Has0-F1 scores of 76.43%, 80.15%, and 81.93%, respectively. Our approach exhibited better performance, as compared to multiple baselines.

1. Introduction

In the research field of intelligent human–computer interaction, the ability to recognize, analyze, understand, and express emotions is essential for intelligent machines. Therefore, the utilization of computer technology to automatically recognize, understand, analyze, classify, and respond to emotion holds significant value for establishing a harmonious human–machine interaction environment, improving interaction efficiency, and enhancing user experience [1,2,3]. Previous studies [4,5] have primarily focused on sentiment analysis using textual data and have achieved remarkable accomplishments. However, as compared to unimodal analysis, MSA can effectively leverage the coordinated and complementary information from different modalities to enhance emotional understanding and expression capabilities and provide richer information that is more consistent with human behavior.

In recent years, there has been a growing interest in multimodal data for sentiment analysis. MSA aims to utilize the information interaction between texts, images, speech, etc., enabling machines to automatically utilize comprehensive multimodal emotional information for the identification of users’ sentiment tendencies. Early research often employed multimodal fusion [6] through early fusion, directly combining multiple sources of raw features [7,8], or late fusion, aggregating the decisions of multiple sentiment classifiers [9,10,11]. However, the former approach may result in a large number of redundant input vectors, leading to increased computational complexity, while the latter approach may struggle to capture the correlations between different modalities. Therefore, various methods have been proposed for feature fusion in multimodal sentiment analysis. Existing fusion methods include those based on simple operations [12,13], attention-based methods [14,15,16,17], tensor-based methods [18], translation-based methods [19], GAN-based methods [20], routing-based methods [21], and hierarchical fusion [22,23,24]. Although there is a wide range of fusion methods, attention-based fusion methods have shown superior efficiency and performance [25]. However, weighting and summing the features of each modality in the attention mechanism alone may not be able to effectively adapt to the differences in the features across different modalities. Consequently, certain modal features might be disregarded or underestimated, ultimately impacting the accuracy of the fused feature representation. Additionally, complex nonlinear interactions may exist between different modalities, and the attention mechanism may struggle to model such relationships accurately, thereby impacting the effectiveness of feature fusion. Furthermore, previous methods have rarely considered the simultaneous utilization of the interaction information within a single modality and between modalities.

In addition to fusion mechanisms, handling noise in the data is crucial for MSA. While many excellent MSA models have been proposed, they are rarely deployed in the real world. The reason is that during the process of human–machine interaction in real environments, text acquisition can only be achieved through ASR models, which convert speech into texts. Through an in-depth analysis of ASR output, we have discovered that errors in recognizing emotional words in ASR-generated texts directly impair MSA models.

To address these challenges, we propose a MWRCMH framework that reduces speech recognition errors and incorporates cross-modal hierarchical fusion. Specifically, we utilized a language model to capture syntactic and semantic information, enabling us to determine the most likely positions of sentiment words. Furthermore, we leveraged multimodal sentiment information to dynamically complement the emotional semantics in the representation of these positions. The corrected representation of sentiment words was then combined with images and sounds, allowing us to capture the interaction information between pairs of modalities using a cross-modal attention mechanism. The fused bimodal feature information was then concatenated with the feature information from each mode, resulting in a vector that encompassed all the feature information. This vector served as the input, passing through the nonlinear layers for fusion. Finally, we employed nonlinear layers to map the concatenated feature vector to a higher-dimensional space, utilizing the output as the final feature representation for training and prediction in subsequent tasks.

The remaining structure of this paper is as follows: Section 2 provides a review of related literature. In Section 3, we present a detailed description of our proposed method. Subsequently, in Section 4, we elaborate the experimental results. Finally, Section 5 concludes the paper, and we discuss future work.

2. Related Work

Sentiment computing, as an emerging interdisciplinary research field, has been widely studied and explored since its introduction in 1995 [26]. Previous research has primarily focused on unimodal data representation and multimodal fusion. In terms of unimodal data representation, Pang et al. [4] were the first to employ machine learning-based methods to address textual sentiment classification, achieving better results than traditional manual annotation, by using movie reviews as the dataset. Yue et al. [5] proposed a hybrid model called Word2vec-BiLSTM-CNN, which leveraged the feature extraction capability of convolutional neural networks (CNN) and the ability of bi-directional long short-term memory (Bi-LSTM) to capture short-term bidirectional dependencies in text. Their results demonstrated that hybrid network models outperformed single-structure neural networks in the context of short texts. Colombo et al. [27] segmented different regions in image and video data based on features such as color, warmth, position, and size, enabling their method to obtain higher semantic levels beyond the objects themselves. They applied this approach to a sentiment analysis of art-related images. Wang et al. [28] utilized neural networks for the facial feature extraction of images. Bonifazi et al. [29] proposed a space–time framework that leveraged the emotional context inherent in a presented situation. They employed this framework to extract the scope of emotional information concerning users’ sentiments on a given subject. However, the use of unimodal analysis in sentiment analysis has had some limitations since humans express emotions through various means, including sound, content, facial expressions, and body language, all of which are collectively employed to convey emotions.

Multimodal data describe objects from different perspectives, providing richer information, as compared to unimodal data. Different modalities of information can complement each other in terms of content. In the context of multimodal fusion, previous research can be categorized into three stages: early feature fusion, mid-level model fusion, and late decision fusion. Wollmer et al. [30] and Rozgic et al. [31] integrated data from audio, visual, and text sources to extract emotions and sentiments. Metallinou et al. [32] and Eyben et al. [33] combined audio and text patterns for emotional recognition. These methods relied on early feature fusion, which mapped them to the same embedding space through simple concatenation, and the lack of interaction between different modalities. For late-stage decision-fusion methods, internal representations were initially learned within each modality, followed by learning the fusion between modalities. Zadeh et al. [18] utilized tensor-fusion networks to calculate the outer product between unimodal representations, yielding tensor representations. Liu et al. [34] introduced a low-rank multimodal-fusion method to reduce the computational complexity of tensor-based approaches. These methods aimed to enhance efficiency by decomposing the weights of high-dimensional fusion tensors, reducing redundant information, yet they struggled to effectively simulate intermodal or specific modality dynamics. Intermediate model fusion amalgamates the advantages of both early feature fusion and late-stage decision fusion, facilitating the selection of fusion points and enabling multimodal interaction. Poria et al. [35] further extended the combination of convolutional neural networks (CNNs) and multiple kernel learning (MKL). In contrast to Ghosal et al. [36], Poria et al. utilized a novel fusion method to effectively enhance fusion features. Zhang et al. [37] introduced a quantum-inspired framework for the sentiment analysis of bimodal data (texts and images) to address semantic gaps and model the correlation between the two modalities using density matrices. However, these methods exhibited limited adaptability to feature differences and suffered from significant feature redundancy. Concerning hierarchical fusion, Majumder et al. [22] employed a hierarchical-fusion strategy, initially combining two modalities and, subsequently, integrating all three modalities. However, this approach struggled to adequately capture intramodal dynamics. Georgiou et al. [23] introduced a deep hierarchical-fusion framework, applying it to sentiment-analysis problems involving audio and text modalities. Yan et al. [24] introduced a hierarchical attention-fusion network for geographical localization. Nevertheless, these methods overlooked the potential existence of complex nonlinear interactions between modalities. Moreover, many fusion approaches have seldomly considered simultaneously harnessing intramodal and intermodal interactions. Verma et al. [38] emphasized that each modality possesses unique intramodality features, and multimodal sentiment analysis methods should capture both common intermodality information and distinctive intramodality signals.

In addition to considering fusion strategies in the MSA model, it is crucial to address the noise present in modal data. Pham et al. [39] proposed the MCTN model to handle the potential absence of visual and acoustic data. Liang et al. [40] and Mittal et al. [41] also focused on addressing noise introduced by visual and acoustic data, relying on word-level features there were obtained by aligning the audio with the actual text. Xue et al. [42] introduced a multi-level attention-graph network to reduce noise within and between modalities. Cauteruccio et al. [43] introduced a string-comparison metric that could be employed to enhance the processing of heterogeneous audio samples, mitigating modality-related noise. However, these models did not investigate the impact of ASR errors on the MSA model. Notably, Wu et al. [44] utilized a sentiment word position-detection module to determine the most likely positions of sentiment words in text. They dynamically refined sentiment-word embeddings using a multimodal sentiment-word-refinement module that incorporated the improved embeddings as the textual input for the multimodal feature-fusion module. This approach reduced the influence of ASR errors on the MSA model. The sentiment word position-detection module and multimodal sentiment-word-refinement module have proven highly effective, achieving state-of-the-art performance on real-world datasets. However, the original SWRM simply concatenated modalities in feature fusion without capturing intramodal and intermodal features, even when genuine correlations existed.

In this work, we employed a cross-modal hierarchical fusion network for feature fusion. This enabled the model to focus on important features, reduce noisy impacts, and achieve superior fusion results. Thus, our approach enhanced the robustness and performance of the model.

3. Methodology

To enhance the adaptability to feature differences across different modalities and effectively model complex nonlinear interactions, we designed a cross-modal, hierarchical-fusion network that considered the presence of sentiment-word-recognition errors in the texts generated by ASR. The model’s architecture is depicted in Figure 1, and it comprises 4 main components: unimodal-feature extraction, a multimodal-word-refinement module, a multimodal-feature-fusion module, and a prediction network. Initially, we employed BERT to extract textual feature information while Bi-GRU was utilized to extract audio and visual feature information. BERT, rooted in the transformer architecture, served as a pre-trained language model capable of capturing the nuanced semantic information of words within sentences through bidirectional contextual modeling [45]. Bi-GRU, a variant of a recurrent neural network (RNN), was particularly well suited for modeling sequential data [46]. Subsequently, the multimodal-word-refinement module dynamically supplemented missing sentiment semantics by leveraging multimodal-sentiment information, resulting in the generation of new word embeddings. These new word embeddings were then input into the multimodal-feature-fusion module, which utilized a cross-modal hierarchical-fusion network to perform feature fusion across different modalities. Finally, a nonlinear layer was employed to predict the final sentiment-regression labels, facilitating accurate sentiment judgment.

Figure 1. The architecture of our model. where * represents matrix multiplication; × represents line-by-line softmax function. Green represents textual, orange represents visual, and blue represents audio.

3.1. Multimodal Word-Refinement Module (MWRM)

We drew inspiration from the SWRM model of Wu et al. [44]. Firstly, we employed a language model to estimate the potential positions of sentiment words in the ASR transcripts by capturing syntactic and semantic information. For example, given a sentence $\left\{w_1,w_2,\dots ,w_{nt}\right\}$, let us assume that the $i$ word in the sentence was a sentiment word; we replaced the word with a special word (MASK). Then, we used BERT and Bi-GRU to model the word embeddings and the visual, audio, and visual–audio features separately, resulting in $h^i=\{h_l^i:1\le l\le n_i,h_l^i\in {ℝ}^{d_h^i}\},i\in \{t,v,a\}$. In addition, we also used a Bi-GRU network to fuse acoustic and visual features in order to capture advanced sentiment semantics and obtained $h^{va}=\{h_l^{va}:1\le l\le n_i,h_l^{va}\in {ℝ}^{d_h^{va}}\}$. The formulas are presented below:

$$h^t={\mathrm{BERT}}_t(x^t)$$

(1)

$$h^v=B\mathrm{i}GR{U}_v(u^v)$$

(2)

$$h^a=BiGR{U}_a(u^a)$$

(3)

$$h^{va}=BiGR{U}_{va}([u^v;u^a])$$

(4)

where the three modalities of unaligned features, word embeddings, visual features, and audio features are represented as $x^m=\{x_l^m:1\le l\le n_i,x_l^m\in {ℝ}^{d_x^m}\},m\in \{t,v,a\}$. To obtain the corresponding multimodal information, we used a pseudo-alignment method to align the features. We divided the acoustic and visual features into non-overlapping feature groups with lengths of $⌊n_a/n_t⌋$ and $⌊n_v/n_t⌋$, respectively, and took the average of the features in each group to obtain the aligned feature representation ${\mathrm{u}}^i=\{u_l^i:1\le l\le n_l,u_l^i\in {ℝ}^{d_x^i}\},i\in \{v,a\}$.

Then, we used a multimodal, gated network to filter the representation information at the locations of the sentiment word to generate $r^v$. To make the model ignore impossible solutions, we used the gate mask $p$. Therefore, $g^v$ is the gate value obtained by connecting the unimodal contextual perception representation $h_s^t$, $h_s^v$, $h_s^a$, and the bimodal representation $h_s^{va}$ in the position $s$, using a nonlinear network. We then used the word embedding $x^{c_l^s}$ of the candidate word $c_l^s$ and the multimodal representation $h_s^v$, $h_s^a$ and $h_s^{va}$, to obtain the attention score through linear fusion. We fed the attention score to the softmax function to obtain the attention weight, which was then applied to the candidate word embeddings to obtain the sentiment embedding $r^e$, as follows:

$$g^v=Sig\mathrm{mo}id(W_1([h_s^t;h_s^v;h_s^a;h_s^{va}])+b_1)$$

(5)

$$r^v=(1-g^{v}p)x_s^t$$

(6)

$$g_l^e=W_2([x^{c_l^s};h_s^v;h_s^a;h_s^{va}])+b_2$$

(7)

$$r^e=soft\max (g_l^e)\odot x^{c_l^s},$$

(8)

where $W_1\in {ℝ}^{1\times {\Sigma }_{i\in \{t,v,a,va\}}{d}_h^i},b_1\in {ℝ}^1$ are the parameters of the multimodal, gated network, $W_2\in {ℝ}^{1\times (d_x^t+{\Sigma }_{i\in \{v,a,va\}}{d}_h^i)},b_2\in {ℝ}^1$ are the parameters of the multimodal sentiment-word attention network, and ⊙ represents element-wise multiplication.

As the sentiment word we wanted may not have been included in the candidate word list, we also introduced the representation of (MASK) when correcting the word representation, that is $x^{mask}$. Finally, we combined the above representations to obtain the corrected word vector $r^t$, as follows:

$$g^{mask}=Sigmoid(W_3([r^e;x^{mask}])+b_3)$$

(9)

$$r^t=(g^{v}p)(g^{mask}{r}^e+(1-g^{mask})x^{mask})+r^v,$$

(10)

where $W_3\in {ℝ}^{1\times 2d_x^t},b_3\in {ℝ}^1$ are the trainable parameters of the nonlinear network.

3.2. Unimodal Feature Extraction

We utilized the BERT model to extract features from the improved word embeddings $z^t=\{x_1^t,x_2^t,\dots ,r^t,\dots ,x_{nl}^t\}$, which are represented as $s^t$:

$$s^t=BER{T}_{textual}(z^t).$$

(11)

For the visual and audio modalities, we employed two separate Bi-GRU networks to encode the visual and audio features. The representation of the first word was used as the visual representation $s^v$ and acoustic representation $s^a$:

$$s^v=BiGR{U}_{visual}(x^v)$$

(12)

$$s^a=BiGR{U}_{audio}(x^a)$$

(13)

Unimodal features can have different dimensions, i.e., $d_t\ne d_a\ne d_v$. Therefore, they were mapped to the same dimension, such as $d_m$, using a fully connected layer, and computed as follows:

$${\omega }^t=relu(W_t{s}^t+b_t)$$

(14)

$${\omega }^a=relu(W_a{s}^a+b_a)$$

(15)

$${\omega }^v=relu(W_v{s}^v+b_v)$$

(16)

where $W_t\in {ℝ}^{d_m\times d_t},b_t\in {ℝ}^{d_m}$ are the trainable parameters of the textual feature-extraction network, $W_a\in {ℝ}^{d_m\times d_a},b_a\in {ℝ}^{d_m}$ are the trainable parameters of the audio feature-extraction network, and $W_v\in {ℝ}^{d_m\times d_v},b_v\in {ℝ}^{d_m}$ are the trainable parameters of the visual feature-extraction network.

3.3. Cross-Modal Attention (CMA)

In general, the three modalities of word-embedding, vision, and audio had some content that was related to sentiment information. If the interaction of the modality information was not considered, the final multimodal features obtained would inevitably contain a large amount of redundant information that would be irrelevant to the detection task. Therefore, we used a cross-modal attention framework, where the attention function was applied to the representations of the two modalities, i.e., visual–textual, textual–audio, and audio–visual. The cross-modal attention calculation for visual–textual $({\omega }^v,{\omega }^t)$ was as follows:

The feature matrices for visual ${\omega }^v$ and textual ${\omega }^t$ were multiplied to obtain the cross-modal information matrix $B_1,B_2\in {ℝ}^{u\times u}$, where $·$ represents matrix multiplication.

$$B_1={\omega }^v\cdot {({\omega }^t)}^T,B_2={\omega }^t\cdot {({\omega }^v)}^T$$

(17)

Through a line-by-line softmax function, the attention distribution was calculated, $N_1,N_2\in {ℝ}^{u\times u}$, where $N_1(\mathrm{i},\mathrm{j})$ represents the correlation between the $\mathrm{i}$ feature of the word-embedding modality and the $\mathrm{j}$ feature of the visual modality: the larger the value, the stronger the interaction between the two features and the more important the fusion information.

$$N_1={\mathrm{softmax}(\mathrm{B}}_1),N_2={\mathrm{softmax}(\mathrm{B}}_2)$$

(18)

The attention distribution was then multiplied by the feature matrix to obtain the final attention representation matrix, $O_1,O_2\in {ℝ}^{u\times d_m}$.

$$O_1=N_1\cdot {\omega }^t,O_2=N_2\cdot {\omega }^v$$

(19)

Finally, a multiplication gate mechanism was used to obtain the mutual-attention-information matrix of the two modalities $A_1,A_2\in {ℝ}^{u\times d_m}$, where ⊙ represents element-wise matrix multiplication.

$$A_1=O_1\odot {\omega }^v,A_2=O_2\odot {\omega }^t$$

(20)

By transposing $A_1$ and $A_2$ to the desired dimensions and concatenating them, the fused information representation of the word embedding and visual modalities was obtained: $Cm{a}_{tv}\in {ℝ}^{u\times 2d_m}$, where ⊕ represents the concatenation operation of vectors.

$$Cm{a}_{tv}={(A_1)}^T\oplus {(A_2)}^T$$

(21)

Using the same method, the fused features of the textual–audio modality $Cm{a}_{ta}$ and the audio–visual modality could be obtained: $Cm{a}_{av}$.

3.4. Trimodal Fusion

After concatenating the aforementioned cross-modal attention-fused features $Cm{a}_{tv}$, $Cm{a}_{ta}$, and $Cm{a}_{av}$, as well as the unimodal features ${\omega }^t$, ${\omega }^a$, and ${\omega }^v$, we used fully connected layers to fuse features from different sources, enabling the model to fully utilize various feature information. The nonlinear tanh activation function was used to enhance the expressive power of the features. The equation was as follows:

$$s^f=Cm{a}_{tv}\oplus Cm{a}_{ta}\oplus Cm{a}_{av}\oplus {\omega }^t\oplus {\omega }^a\oplus {\omega }^v$$

(22)

$$s^F=\tanh (W_f{s}^f+b_f)$$

(23)

where $W_f\in {ℝ}^{d_F\times d_f},b_f\in {ℝ}^{d_F}$ are the trainable parameter of the trimodal fusion network.

3.5. Regression Analysis

In the regression analysis, we utilized a nonlinear layer to predict the final sentiment regression labels, as shown in the following equation:

$$score=\tanh (W_F{s}^F+b_F)$$

(24)

where $W_F\in {ℝ}^{1\times d_F},b_F\in {ℝ}^1$ are the trainable parameters of the prediction network.

4. Experiments

4.1. Dataset

We used three real-world datasets constructed from the CMU-MOSI [47] dataset, including MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek. The CMU-MOSI dataset is one of the most popular multimodal sentiment analysis datasets and contains 93 monologue clips of YouTube movie-review videos. These clips were divided into 2199 annotated video segments with emotion labels. Sentiment scores ranging from −3 (strongly negative) to 3 (strongly positive) were also annotated. For the multimodal features, we used Facet to extract the visual features. Facet can extract facial-action units, facial landmarks, head poses, and gaze-tracking data from each frame, resulting in a comprehensive representation across 34 dimensions. Facet achieves the visualization of relationships and trends among multiple dimensions by mapping various dimensions of the data onto distinct visual channels. The acoustic features were obtained by applying COVAREP [48], which includes 12 Mel-frequency cepstral coefficients (MFCC) and other low-level features.

However, the text data in the MOSI dataset was sourced from the subtitles of movies and TED Talks as transcriptions and have been expertly corrected and processed for speech. However, obtaining text for real-world applications in this way is unrealistic. To evaluate models in the real world, we manually replaced the raw text data in the dataset with text generated by ASR models. We used SpeechBrain’s ASR model and two widely used commercial APIs to generate text. The ASR model used, as published by Ravanelli et al. [49], was built on a transformer encoder–decoder framework and trained on the Librispeech dataset [50]. The commercial APIs we used were IBM and iFlytek speech-to-text, which are widely used by researchers and software developers. Finally, we transcribed the videos into text using the three ASR models and then built three new datasets: MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek. The percentages of the emotional-word-replacement errors for the MOSI dataset using the 3 ASR APIs are listed in Table 1. The percentage of emotional-word-replacement errors for MOSI-IBM was 17.6%, meaning that about 17 out of 100 utterances had this type of error.

Table 1. The proportion of sentiment-word-replacement errors in three real-world datasets.

Datasets	Percentages
MOSI-SpeechBrain	26.5%
MOSI-IBM	17.6%
MOSI-iFlytek	10.6%

4.2. Implementation Details

To ensure fairness, we used the same parameters for the experiments conducted on all 3 datasets. We used Adam as the optimizer, and the learning rate was 5 × 10⁻⁵. The batch size was 32. The dropout was 0.1. In addition, in Section 3, the values $d_t$, $d_a$, and $d_v$ were 768,128 and 128, respectively. We implemented our entire method based on PyTorch, and all experiments were run on an NVIDIA GeForce RTX 3080 GPU.

4.3. Baseline

To validate the effectiveness of the model, we conducted a comparative analysis of established and representative multimodal sentiment-analysis models, including:

TFN (Zadeh et al., 2017) [18], which uses a tensor-fusion network to create a multi-dimensional tensor to learn unimodal, bimodal, and trimodal information in multimodal data;

LMF (Liu et al., 2018) [34], which decomposes the weight tensor using a low-rank multimodal-fusion approach, which reduces the computational complexity of tensor-based methods, and efficiently fuses multimodal interactions with modality-specific low-rank factors;

MulT (Tsai et al., 2019) [15], which proposes a multimodal transformer that utilizes cross-modal attention to enable interactions between multimodal sequences across different time steps;

MISA (Hazarika et al., 2020) [51], which uses multitask learning to partition each modality into different subspaces and learn related sentiment representations that aid the fusion process;

Self-MM (Yu et al., 2021) [52], which learns rich unimodal representations by training on 1 multimodal task and 3 unimodal subtasks; and

SWRM (Wu et al., 2022) [44], which proposes a multimodal word correction model based on sentiment word perception and utilizes multimodal sentiment information to dynamically complete damaged sentiment semantics.

4.4. Results and Analysis

In this section, we evaluated the performance of our model on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets using various metrics, including accuracy (Acc2), F1-score (F1) in binary classification, mean absolute error (MAE), and correlation (Corr). Specifically, we calculated the Acc2 and F1-scores in 2 ways: negative/non-negative (Non0-Acc, Non0-F1) and negative/positive (Has0-Acc, Has0-F1). Since the predicted results were continuous values, we converted them into emotion classification labels by mapping the emotion scores to the labels. The experimental results for different models are presented in Table 2.

Table 2. Results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets. The best results are in bold. ↑ indicates that the larger the numerical value, the better the model performance; ↓ indicates that the smaller the numerical value, the better the model performance.

Datasets	Models	Evaluation Metrics
		Has0-Acc ↑	Has0-F1 ↑	Non0-Acc ↑	Non0-F1 ↑	MAE ↓	Corr ↑
MOSI-SpeechBrain	TFN	68.98	68.95	69.51	69.57	115.55	48.54
	LMF	68.86	68.88	69.36	69.48	117.42	48.66
	MulT	71.78	71.70	72.74	72.75	109.00	54.69
	MISA	73.79	73.85	74.75	74.66	98.52	65.37
	Self-MM	73.67	73.72	74.85	74.98	90.95	67.23
	SWRM	74.58	74.62	75.70	75.82	90.56	67.47
	Ours	76.53	76.43	78.05	78.03	88.98	68.27
MOSI-IBM	TFN	71.81	71.78	72.13	73.21	109.42	58.19
	LMF	73.06	73.09	74.30	74.41	104.70	59.07
	MulT	75.57	75.54	76.74	76.79	100.32	64.34
	MISA	76.97	76.99	78.08	78.17	91.23	71.30
	Self-MM	77.32	77.37	78.60	78.72	85.65	73.23
	SWRM	78.43	78.47	79.70	79.80	82.91	73.91
	Ours	80.17	80.15	81.86	81.90	79.61	75.32
MOSI-iFlytek	TFN	71.95	72.01	72.62	72.76	107.01	56.52
	LMF	71.98	72.03	72.35	72.49	106.63	59.48
	MulT	77.32	77.05	78.75	78.56	89.84	68.14
	MISA	79.59	79.62	79.82	79.91	85.63	74.53
	Self-MM	80.26	80.26	81.16	81.20	78.79	75.83
	SWRM	80.47	80.47	81.28	81.34	78.39	75.97
	Ours	81.92	81.93	82.93	82.98	75.25	76.31

Based on the Has0-F1 values in the table, we drew the following conclusions: Compared to the tensor-fusion methods TFN and LMF, our model demonstrated a significant improvement of 7.48% and 6.65% on the MOSI-SpeechBrain dataset, respectively; 8.37% and 7.06% on the MOSI-IBM dataset, respectively; and 9.92% and 9.90% on the MOSI-iFlytek dataset, respectively, highlighting the superiority of our model. This superiority stemmed from the fact that TFN and LMF performed tensor fusion at an overall level without adequately considering the interaction modeling between the modalities, whereas our fusion network took this aspect into account.

Additionally, as compared to the attention-based cross-modal model MulT, our model achieved improvements of 4.73%, 4.61%, and 4.88% on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets, respectively. The reason behind this performance gain was that MulT overlooked the feature differences between the different modalities and the potential nonlinear interaction relationships. In contrast, our hierarchical fusion approach incorporated a nonlinear layer for feature fusion, leading to better results. Furthermore, as compared to the modality-invariant and specificity-based fusion model MISA, our model outperformed it by achieving improvements of 2.58%, 3.16%, and 2.31% on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets, respectively. This improvement could be attributed to our model’s focus on capturing the semantic representations of different modalities, resulting in enhanced performance. Moreover, our model, named Self-MM, surpassed the self-supervised and multitasking-representation learning model Self-MM with improvements of 2.71%, 2.78%, and 1.67% on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets, respectively. These findings validated the superior performance of our model, which benefited from its unique design and approach. Lastly, in comparison with the baseline model SWRM, our model achieved improvements of 1.81%, 1.68%, and 1.64% on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets, respectively. This improvement could be attributed to the utilization of our cross-modal hierarchical-fusion network, which enabled better and more effective multimodal fusion, leading to enhanced overall performance.

Figure 2, Figure 3 and Figure 4 show the accuracy of the model regression on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets. Train-Has0-Acc and Train-Has0-F1 were correlated to Has0-Acc and Has0-F1, as generated by the training datasets. Val-Has0-Acc and Val-Has0-F1 were generated by the validation datasets. The models were trained in 10 epochs using 70% of the samples as the training set, and 30% of samples were used as test data.

Figure 2. Train-Has0-Acc, Train-Has0-F1, Val-Has0-F1, and Val-Has0-Acc on the MOSI-SpeechBrain dataset.

Figure 3. Train-Has0-Acc, Train-Has0-F1, Val-Has0-F1, and Val-Has0-Acc on the MOSI-IBM dataset.

Figure 4. Train-Has0-Acc, Train-Has0-F1, Val-Has0-F1, and Val-Has0-Acc on the MOSI-iFlytek dataset.

4.5. Ablation Study

4.5.1. Modal Ablation

The literature [15,24] has demonstrated the superiority of multimodal analysis over unimodal analysis. We also observed the same trend in our experiments, where the trimodal and bimodal classifiers outperformed the unimodal classifier. Among the different modalities, the textual modality exhibited the highest accuracy in the unimodal classification, reaching 73.32%. While the other modalities contributed to enhancing the performance of the multimodal classifier, their impact was relatively small, as compared to the textual modality.

To validate the significant influence of the multimodal information fusion on sentiment-analysis accuracy, we conducted experiments using eight different combinations of modality features: unimodal (T, A, V), bimodal (V + T, T + A, A + V), trimodal (T + V + A), and composite hierarchical fusion (TVA + T + V + A). For the unimodal feature information, we processed the data through a feature-extraction layer and directly utilized the results for sentiment analysis. Similarly, for the bimodal features, we initially fused pairwise combinations of the different modalities and then applied the same processing for the sentiment analysis. Regarding the trimodal feature information, we fused the bimodal features T + A, T + V, and V + T to obtain the trimodal fusion T + A + V, followed by applying the same processing for sentiment analysis. Finally, we employed the fusion processing method proposed in this paper, merging the trimodal information with the unimodal information to obtain the final trimodal features (TVA + T + V + A) for sentiment analysis. The detailed results of sentiment analysis are presented in Table 3.

Table 3. Experimental results of modal ablation on the MOSI-SpeedBrain dataset. ↑ indicates that the larger the numerical value, the better the model performance; ↓ indicates that the smaller the numerical value, the better the model performance.

Tasks	Has0-Acc ↑	Non0-Acc ↑	MAE ↓
T	73.32	75.09	91.22
A	55.69	56.10	137.88
V	55.25	53.66	142.20
V + T	73.91	75.30	89.03
T + A	73.82	74.70	90.41
V + A	56.04	56.89	136.34
T + V + A	75.07	76.37	91.10
TVA + T + V + A	76.53	78.05	88.89

Considering the evaluation metrics for the regression results, the bimodal sentiment analysis generally outperformed the unimodal sentiment analysis, and the best performance was achieved when utilizing the fused trimodal features. Therefore, the effective fusion of textual, acoustic, and visual features contributed to enhancing the performance of sentiment classification.

4.5.2. Model Ablation

We conducted ablation experiments to assess the contributions of each component in our model. Our architecture comprised several different variants. MWRCMH represented the complete model proposed by us. MWRCMH without fusion excluded the cross-modal hierarchical-fusion module and simply concatenated the feature vectors from the different modalities. MWRCMH without MWRM did not employ the multimodal word-correction module and did not consider the impact of the recognition errors of the sentiment words in the ASR transcripts. The results of the model variants are presented in Table 4. Without the multimodal word-correction module, MWRCMH without MWRM yielded inferior results, as compared to MWRCMH, highlighting the significance of identifying appropriate words for refinement. Nevertheless, the results outperformed those of the models in the baseline experiment, thereby demonstrating the effectiveness and superiority of our fusion network. Moreover, the results of MWRCMH without fusion, which lacks the cross-modal hierarchical-fusion module, further substantiated this conclusion.

Table 4. Experimental results of the model ablation. ↑ indicates that the larger the numerical value, the better the model performance; ↓ indicates that the smaller the numerical value, the better the model performance.

Datasets	Models	Has0-Acc ↑	Non0-Acc ↑	MAE ↓
MOSI-SpeechBrain	MWRCMH	76.53	78.05	88.98
	w/o Fusion	75.66	77.29	89.76
	w/o MWRM	75.37	76.62	89.69
MOSI-IBM	MWRCMH	80.17	81.86	79.61
	w/o Fusion	79.74	81.11	80.75
	w/o MWRM	79.61	81.01	81.02
MOSI-iFlytek	MWRCMH	81.92	82.93	75.25
	w/o Fusion	80.61	81.86	76.16
	w/o MWRM	80.66	81.25	76.81

5. Conclusions

This paper introduced a cross-modal hierarchical fusion network-based multimodal sentiment-analysis architecture. Through the incorporation of the multimodal word-correction module and the cross-modal hierarchical-fusion module, our architecture effectively mitigated the sentiment word-recognition errors stemming from speech recognition, augmented the adaptability across diverse modal characteristics, and enhanced the nonlinear modeling capabilities. Our proposed architecture demonstrated strong performance across the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets. However, the architecture disregarded the fine-grained relationships between the textual, audio, and visual information, and it also exhibited certain issues related to feature redundancy.

In forthcoming work, we intend to address the alignment challenges presented by various modalities while concurrently minimizing modal-feature-information redundancy. Furthermore, we will place greater emphasis on rectifying contribution imbalances within multimodal learning.