Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis aims to uncover human affective states by integrating data from multiple sensory sources. However, previous studies have focused on optimizing model architecture, neglecting the impact of objective function settings on model performance. Given this, this study introduces a new framework, DMMSA, which utilizes the intrinsic correlation of sentiment signals and enhances the model’s understanding of complex sentiments. DMMSA incorporates coarse-grained sentiment analysis to reduce task complexity. Meanwhile, it embeds a contrastive learning mechanism within the modality, which decomposes unimodal features into similar and dissimilar ones, thus allowing for the simultaneous consideration of both unimodal and multimodal emotions. We tested DMMSA on the CH-SIMS, MOSI, and MOEI datasets. When only changing the optimization objectives, DMMSA achieved accuracy gains of 3.2%, 1.57%, and 1.95% over the baseline in five-class and seven-class classification tasks. In regression tasks, DMMSA reduced the Mean Absolute Error (MAE) by 1.46%, 1.5%, and 2.8% compared to the baseline.

1. Introduction

Multimodal sentiment analysis (MSA) involves integrating and synergistically parsing diverse heterogeneous data modalities [1,2,3], encompassing many information forms such as text, visual, auditory, and even biometric markers [4]. With the evolution of social media ecosystems and the proliferation of multimedia content, information presentation has evolved from pure text to richly illustrated content, culminating in today’s prevalent video-based information [5,6].

Traditional unimodal sentiment analysis is confined mainly to the textual domain [7]. In contrast, MSA encompasses a comprehensive interpretation of multiple perceptual channels, including visual cues (e.g., facial expressions, scene color, and body movement) and audio characteristics (e.g., pitch amplitude, frequency distribution, and speech tempo) [8,9]. MSA has garnered significant attention in recent research. On the one hand, human emotional expression inherently possesses cross-modal properties, with text, speech, and even haptic cues intricately interwoven to form sentiments, rendering a single modality insufficient to fully unveil the complexity of these sentiments [10,11]. On the other hand, through their deep fusion of multiple signal sources, MSA technologies significantly enhance the accuracy of sentiment recognition and understanding, fulfilling the high-precision sentiment intelligence demands in domains such as intelligent customer service, VR/AR experience optimization, and mental health assessment [12,13].

Existing deep learning-based MSA models typically consist of three core components: a single-modality feature extraction module, a multimodal feature fusion module, and a classification head [14]. The use of pre-trained models as the single-modality feature extraction module has been widely recognized as an effective strategy. Consequently, current researchers often focus on optimizing multimodal feature fusion methods and model training processes to further enhance the overall performance of the models.

Multimodal fusion has emerged as a core technique for understanding video contexts, demonstrating its value across numerous downstream tasks [15,16,17]. Prior research has proposed a series of fusion techniques for MSA. For instance, Yu et al. [18] employed a self-supervised joint learning strategy called Self-MM, which integrates discriminative information learned from individual unimodal tasks with shared similarity information from the multimodal task during the late fusion stage, thereby enhancing model performance.

While multimodal fusion techniques are crucial for models [19,20], setting optimization objectives is equally indispensable in model construction [5]. Suitable optimization objectives effectively guide the model toward continuous performance optimization throughout training [5]. Moreover, as shown in Figure 1, the setting of optimization objectives and model structure optimization focus on different modules, complementing each other. Early researchers typically used only the cross-entropy loss function to supervise the training of MSA models [14]. However, relying solely on single-task training makes it difficult to fully exploit the correlations between different modalities [5]. Therefore, subsequent researchers attempted to use multi-task learning methods, allowing MSA models to complete multiple related tasks simultaneously during training, thereby enhancing the utilization of different modality information. In this approach, the selection of related tasks is crucial, as different tasks can influence each other. Only by choosing the right combination of tasks can the model achieve better performance in multi-task training.

As shown in Figure 2, the sentiment intensity derived solely from analyzing text data is +1. When analyzing audio data alone, the sentiment intensity is −0.2. Similarly, when considering only visual information, the sentiment intensity is −1. However, when taking into account these three modalities of data comprehensively, the overall sentiment intensity of the information is 0. If we ignore the text information, we might conclude that the sentiment is negative. Conversely, if we overlook the visual information, we might infer that the sentiment is positive. In any case, it is impossible to accurately analyze the overall sentiment of the entire piece of information. Therefore, in the MSA task, single-modality sentiment has a direct impact on the overall sentiment [12,21,22]. Nevertheless, many existing methods tend to overlook the significance of unimodal sentiment [18,23]. Thus, enabling MSA models to focus on both uni- and multimodal sentiments concurrently has emerged as a formidable challenge in this field. Another challenge faced by MSA tasks lies in their broad range of sentiment ratings. For example, the MOSI dataset requires models to accurately map samples to a sentiment intensity scale of [−3, +3], which increases the prediction difficulty.

Given these challenges, we propose DMMSA, a Dynamic Tuning and Multi-Task Learning-Based MSA model. DMMSA ensures that the model can capture unimodal signals in detail and integrate multimodal information through the collaborative optimization of unimodal and multimodal tasks. The model is equipped with a text-oriented contrastive learning module to promote feature decoupling and enhance the depth and accuracy of sentiment understanding. Furthermore, incorporating coarse-grained sentiment classification tasks to converge the prediction range improves the accuracy of sentiment intensity determination. We implement Global Dynamic Weight Generation (GDWG) to avoid negative transfer effects and achieve the joint adjustment of model parameters, thereby maximizing overall performance.

The main contributions of this paper can be summarized as follows:

• We propose the Multi-NT-Xent loss to guide the model in decomposing unimodal features and establishing text-centered contrastive relations.
• By employing coarse-grained sentiment analysis tasks, we effectively converge the prediction range, reducing the complexity of modeling sentiment intensity.
• To address the issue of unequal convergence rates among different tasks during multi-task training, we propose the GDWG strategy, effectively mitigating the negative transfer effects arising from such mismatches.

Our model is evaluated on three benchmark datasets: CH-SIMS [18], MOSI [24], and MOSEI [23]. The results show that DMMSA outperforms the baseline method in classification and regression tasks when the model structure remains unchanged and only the optimization objectives are replaced. Additionally, we conduct comprehensive ablation studies, substantiating the efficacy of each component within our proposed architecture.

2. Related Works

2.1. Multimodal Sentiment Analysis

As a core topic in affective computing research, past scholars have primarily focused MSA research on representation learning and multimodal fusion strategies. In representation learning, Wang et al. [25] introduced the Recurrent Attended Variation Embedding Network (RAVEN), tailored for fine-grained structural modeling of non-verbal subword sequences, dynamically adjusting word-level representations in response to non-verbal cues. Regarding multimodal fusion techniques, Zaden et al. [26] designed the Tensor Fusion Network to deeply model intra-modal and inter-modal relationships in online video analysis, addressing the transient variability of spoken languages, sign language, and audio signals. Subsequently, they proposed the Memory Fusion Network [23], employing attention mechanisms for interactive information integration across different views. Sun et al. [4], attentive to heterogeneity issues, proposed an attention-based cross-modal fusion scheme that facilitates modal interactions through attention mechanisms, promoting the effective alignment of distinct modal features. However, these efforts overlooked the potential impact of the settings of optimization objectives on model performance.

Yu et al. [18] recognized the significant influence of unimodal sentiment expressions on overall affective states, leading to the construction of the CH-SIMS dataset, which encompasses both uni- and multimodal sentiment intensity measures. Their study employed the L1 loss function for multimodal and unimodal sentiment analysis as a joint optimization objective. Experimental results revealed that incorporating unimodal sentiment analysis tasks enhanced the model’s accuracy in predicting holistic emotional dispositions. Yang et al. [5] decomposed unimodal representations into similarity and dissimilarity components, utilizing a text-centric contrastive learning approach. However, when implementing multi-task learning, they failed to adequately account for the potential negative transfer effects resulting from pronounced disparities in task convergence rates and loss scales.

In contrast, the DMMSA model incorporates a GDWG mechanism, enabling the model to adaptively adjust task weights based on the relative rates of loss decrease during training, effectively mitigating the detrimental impact of negative transfer on model performance. Moreover, DMMSA incorporates coarse-grained sentiment analysis tasks to constrain the prediction scope.

2.2. Contrastive Learning

Contrastive learning systematically constructs and discriminates between feature differences in positive and negative sample pairs to reveal intrinsic structural relationships within data [8,27,28,29]. This strategy has proven particularly effective in multimodal feature fusion research [5,30]. Specifically, Radford et al. [31] employed multimodal contrastive learning techniques to align image–text pairs, effectively alleviating the inherent data heterogeneity between visual and textual modalities and fostering widespread application in diverse multimodal downstream tasks such as visual question answering and caption generation. Similarly, Akbari et al. [32] trained a vision–audio–text translation model using the same contrastive learning approach, successfully achieving deep alignment among these three modalities.

For performing MSA tasks, Yang et al. [5] devised two contrastive learning mechanisms, intra-modal contrast and inter-modal contrast, to guide the model toward generating features that embody the homogeneity across modalities and capture the heterogeneity between them. This strategy ensures that the model attends equally to commonalities and differences in modal interactions during modeling. Nonetheless, while this method yielded promising results, it did not address the limitation of traditional NT-Xent loss functions, which are tailored for single positive pair settings and are ill suited for scenarios involving multiple positive pairs. The $NT-Xent$ loss is given by

$$\begin{array}{c}\hfill L_{NTX}=-{\sum }_{(a,p)\in P}log{\displaystyle \frac{exp(sim(a,p)/{\tau }_m)}{{\sum }_{(a,k)\in N\cup P}exp(sim(a,k)/{\tau }_m)}}\end{array}$$

(1)

where ${\tau }_m$ is the temperature coefficient that controls the similarity distribution. $(a,p)$ and $(a,k)$ denote positive and negative sample pairs, respectively. N represents the set of negative pairs, and P signifies the set of positive pairs.

However, CL manifestly fails to converge to zero in situations involving more than one positive pair, i.e., $n>1$. In light of this limitation, while leveraging contrastive learning strategies to aid the model in extracting both similarity and dissimilarity features, this paper proposes an improvement to the NT-Xent loss function tailored to accommodate multiple positive instances, namely the Multi-NT-Xent loss:

$$\begin{array}{c}\hfill L_{MNTX}=-log{\displaystyle \frac{{\sum }_{(a,p)\in P}exp(sim(a,p)/{\tau }_m)}{{\sum }_{(a,k)\in N\cup P}exp(sim(a,k)/{\tau }_m)}}\end{array}$$

(2)

Consequently, the model is effectively guided in its contrastive learning tasks, whether faced with a single positive pair or multiple ones.

3. Methodology

In this section, we first provide a brief introduction to the task definition. We then describe the overall operational process of the DMMSA model, and finally, we detail our proposed model training method based on multi-task learning and dynamic tuning.

3.1. Task Definition

MSA aims to decipher sample sentiment states by harnessing multiple signals, encompassing text ($I_t$), visual ($I_v$), and audio ($I_a$) modalities. Task types within this domain are typically categorized into two broad classes: classification and regression. Focusing on the latter, the proposed DMMSA model takes $I_t$, $I_v$, and $I_a$ as inputs, yielding an output sentiment intensity value $y^{*}$, constrained within the actual interval $[-R,R]$, where R defines the upper and lower bounds of the sentiment score.

3.2. Model Architecture

This section primarily introduces the overall architecture of the DMMSA model to facilitate the reader’s understanding. The overall architecture of the DMMSA model is depicted in Figure 3.

In the single-modality feature extraction stage, we employ the pre-trained $BERT$ model to extract text features. $BERT$ is a pre-trained language model based on the transformer architecture. During pre-training, it leverages a vast amount of unsupervised text data to learn rich linguistic knowledge and semantic information. Therefore, compared to models such as RNN, LSTM, and GRU, it can encode text more effectively. For processing visual and audio modality data, we use the transformer model as the visual encoder and audio encoder in DMMSA. The overall formula is as follows:

$$\begin{array}{c}\hfill T=BERT\left(I_t\right)\end{array}$$

(3)

$$\begin{array}{c}\hfill V=Transforme{r}_v\left(I_v\right)\end{array}$$

(4)

$$\begin{array}{c}\hfill A=Transforme{r}_a\left(I_a\right)\end{array}$$

(5)

where T, V, and A represent the text, visual, and audio features extracted by the encoders, respectively. $Transforme{r}_v\left(\right)$ represents the encoder for visual information, and $Transforme{r}_a$ represents the encoder for audio information.

Next, we input the features from different modalities into the feature decomposition module to obtain modality-similar features and modality-dissimilar features. The feature decomposition module consists of six parallel project modules, each of which includes a fully connected layer with a Tanh activation function and layer normalization [5]. The formulas are as follows:

$$\begin{array}{c}\hfill T_s=\sigma \left(Linea{r}_s^T\left(T\right)\right)\end{array}$$

(6)

$$\begin{array}{c}\hfill T_d=\sigma \left(Linea{r}_d^T\left(T\right)\right)\end{array}$$

(7)

$$\begin{array}{c}\hfill V_s=\sigma \left(Linea{r}_s^V\left(V\right)\right)\end{array}$$

(8)

$$\begin{array}{c}\hfill V_d=\sigma \left(Linea{r}_d^V\left(V\right)\right)\end{array}$$

(9)

$$\begin{array}{c}\hfill A_s=\sigma \left(Linea{r}_s^A\left(A\right)\right)\end{array}$$

(10)

$$\begin{array}{c}\hfill A_d=\sigma \left(Linea{r}_d^A\left(A\right)\right)\end{array}$$

(11)

where $T_s$, $V_s$, and $A_s$ represent the modality-similar features extracted by the mapping modules; $T_d$, $V_d$, and $A_d$ represent the modality-dissimilar features; $\sigma$ represents the Tanh activation function; and $Linear$ represents the fully connected layer. The feature decomposition module decomposes features from different modalities into modality-similar and modality-dissimilar features. This is attributed to the incorporation of two tasks, single-modality sentiment classification and contrastive learning, in the training process, which helps the model effectively perform feature decomposition. The details are discussed in the section on multi-task learning.

Finally, we concatenate the extracted modality-similar and modality-dissimilar features and input them into a multi-layer perceptron (MLP) network to obtain the final sentiment prediction result. The formula is as follows:

$$\begin{array}{c}\hfill y^{*}=MLP\left([T_s;T_d;A_s;A_d;V_s;V_d]\right)\end{array}$$

(12)

where $y^{*}$ represents the predicted result and “;” represents concatenation.

3.3. Multi-Task Learning

In this section, we focus on the setup of multi-task learning and how we mitigate negative transfer through dynamic tuning.

As shown in Figure 3, we employ a hard parameter sharing method for multi-task learning. In this approach, all tasks share the hidden layer parameters of the neural network, and each task has its own output layer. This method not only reduces the risk of overfitting but also encourages the model to find a general representation that covers all tasks.

In terms of task setup, we first draw inspiration from previous studies and introduce a unimodal sentiment classification task to help the model better analyze the sentiment of the information [5]. Given that the MSA task requires the model to predict a value from a wide range, which is quite challenging, we further add a coarse-grained sentiment analysis task to help the model narrow down the prediction range and thus improve prediction accuracy. Moreover, to assist the model in better distinguishing between similarity and dissimilarity features, we also incorporate a contrastive learning task. Additionally, by leveraging the collaborative effect of the unimodal sentiment analysis and contrastive learning tasks, we enable the model to accurately extract similarity and dissimilarity features from the three information modalities. This, in turn, enhances the model’s effective utilization of the intrinsic relevance and distinctive emotional inclinations embedded in the sentiment signals of different modalities. The final multi-task learning objective function is given by

$$\begin{array}{c}\hfill L_{MSA}=L_{MSR}+L_{MSC}+{\lambda }_{Uni}{L}_{Uni}+{\lambda }_{CL}{L}_{CL}\end{array}$$

(13)

where $\lambda$ represents the weights generated by the GDWG method, $L_{MSR}$ is the multimodal sentiment regression loss, $L_{MSC}$ is the loss related to multimodal sentiment classification, $L_{Uni}$ is the unimodal sentiment analysis loss, and $L_{CL}$ is the contrastive learning loss.

$L_{MSR}$—Multimodal Sentiment Regression Loss. The purpose of this task is to guide the model in integrating signals from different modalities to estimate the sentiment intensity of samples accurately. Herein, we feed the fused decomposed similarity and dissimilarity features into a multimodal MLP for sentiment intensity prediction, associating its output with the given multimodal sentiment intensity labels via a smooth L1 loss function to derive this loss. The formulaic expression is as follows:

$$\begin{array}{c}\hfill y^{*}=MLP\left([T_s;T_d;A_s;A_d;V_s;V_d]\right)\end{array}$$

(14)

$$\begin{array}{c}\hfill L_{mul}=\left\{\begin{array}{c}0.5\ast {\left({\displaystyle \frac{y^{*}-y}{\phi }}\right)}^2,if\left({\displaystyle \frac{y^{*}-y}{\phi }}\right)<1\hfill \\ (y^{*}-y)-0.5\phi , otherwise\hfill \end{array}\right\}\end{array}$$

(15)

where $y^{*}$ represents the predicted result, “;” represents concatenation, y represents the multimodal sentiment label, and $\phi$ controls the smoothness.

$L_{MSC}$—Multimodal Sentiment Classification Loss. The purpose of this task is to direct the model toward the coarse-grained classification of sentiment, thus narrowing down the prediction scope. Unlike the fine-grained sentiment analysis task, this task does not require the model to precisely predict the exact intensity of sentiment, but rather to predict the range interval to which the sentiment intensity belongs. Initially, we map the sentiment intensity labels of samples to preset sentiment polarity classifications (such as positive, negative, and neutral) in accordance with predefined regulations, thereby creating a sentiment polarity label collection. Next, the disassembled multimodal features are efficiently combined and fed into a sentiment classifier as input, generating the probability distribution of each sample among different sentiment polarities. Eventually, the predicted probability distribution of the classifier is compared with the actual assigned sentiment polarity labels, and the cross-entropy loss function is utilized to measure the loss between the two. The specific formulaic expression is as follows:

$$\begin{array}{c}\hfill y_{MSC}=Classifier\left([T_s;T_d;A_s;A_d;V_s;V_d]\right)\end{array}$$

(16)

$$\begin{array}{c}\hfill L_{MSC}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{c=1}^C{y}_{i,c}log\left(y_{MSC}^{i,c}\right)\end{array}$$

(17)

where N denotes the number of samples and C represents the number of categories.

$L_{Uni}$—Unimodal Sentiment Analysis Loss. The purpose of this task is to guide the model in delving into the sentiment information embedded within each modality. Here, to ensure consistent treatment of modal features, we feed the similarity features ($T_s$, $V_s$, and $A_s$) and dissimilarity features ($T_d$, $V_d$, and $A_d$) of each modality separately into a weight-sharing MLP layer. The MLP layer outputs six sentiment predictions, $u^{*}=MLP\left([T_s/T_d/A_s/A_d/V_s/V_d]\right)$, with the similarity features used to infer the multimodal sentiment label y and the dissimilarity features used to predict the corresponding unimodal sentiment labels $y^{t/v/a}$. In the absence of unimodal labels, the dissimilarity feature prediction task adjusts to instead predict the multimodal label y, maintaining the coherence of model training. Finally, a smooth L1 loss function is employed for each prediction to measure the loss between the prediction and the respective ground-truth label $u=[y/y/y/y^t/y^v/y^a]$. The specific formulaic expression is as follows:

$$\begin{array}{c}\hfill L_{mul}=\left\{\begin{array}{c}0.5\ast {\left({\displaystyle \frac{u^{*}-u}{\phi }}\right)}^2,if\left({\displaystyle \frac{u^{*}-u}{\phi }}\right)<1\hfill \\ (u^{*}-u)-0.5\phi ,otherwise\hfill \end{array}\right\}\end{array}$$

(18)

$L_{CL}$—Contrastive Learning Loss. Contrastive learning tasks help models learn the commonalities among positive examples and the differences from negative examples by comparing positive and negative samples. Through this task, models can extract high-quality similarity and dissimilarity features, thereby paying attention to both unimodal and multimodal sentiment information. Here, considering that text modality data often assume a dominant role in MSA tasks, with other modalities providing auxiliary information to enhance prediction accuracy, we opt to use text data as the reference anchor for constructing positive and negative sample pairs [12]. The specific configuration is as follows:

$$\begin{array}{c}\hfill N=\left\{(T_s,T_d),(T_s,V_d),(T_s,A_d)\right\}\end{array}$$

(19)

$$\begin{array}{c}\hfill P=\left\{(T_s,V_s),(T_s,A_s)\right\}\end{array}$$

(20)

Subsequently, we employ our proposed Multi-NT-Xent loss to guide the model in maximizing the similarity between positive sample pairs while minimizing the similarity between negative sample pairs. The calculation formula for the Multi-NT-Xent loss is as follows:

$$\begin{array}{c}\hfill L_{MNTX}=-log{\displaystyle \frac{{\sum }_{(a,p)\in P}exp(sim(a,p)/{\tau }_m)}{{\sum }_{(a,k)\in N\cup P}exp(sim(a,k)/{\tau }_m)}}\end{array}$$

(21)

where ${\tau }_m$ is the temperature coefficient that controls the similarity distribution; $(a,p)$ and $(a,k)$ denote positive and negative sample pairs, respectively; N represents the set of negative pairs; and P signifies the set of positive pairs.

3.4. Global Dynamic Weight Generation

In multi-task learning scenarios, distinct tasks often exhibit asynchronous convergence patterns, leading specific tasks to stabilize either prematurely or tardily [22,24,33,34]. This inconsistency in convergence rates can engender negative transfer, where the learning process of one task adversarially impacts the performance of other tasks, thereby compromising overall model effectiveness [35]. To address this challenge, we introduce the GDWG mechanism. This mechanism aims to adaptively adjust the relative weights of individual tasks during training. Specifically, it assesses the descent rate of each task’s loss function at every training stage and, based on these assessments, generates weight values for each task. The specific mathematical expression is as follows:

$$\begin{array}{c}\hfill w_k(t-1)={\displaystyle \frac{L_k(t-1)}{L_k\left(1\right)}}\end{array}$$

(22)

$$\begin{array}{c}\hfill {\lambda }_k(t-1)={\displaystyle \frac{exp(w_k(t-1)/\tau )}{{\displaystyle \sum _j^J}exp(w_j(t-1)/\tau )}}\end{array}$$

(23)

where $w_k\left(t\right)$ denotes the relative decay rate of task k at the t-th training stage, ${\lambda }_k\left(t\right)$ represents the weight value assigned to task k at stage t, and $L_k\left(t\right)$ signifies the loss incurred by task k at stage t. J signifies the total number of tasks subject to adjustment, and $\tau$ is a temperature coefficient that governs the magnitude of weight updates, with smaller values indicating greater amplitudes of weight updates. All tasks under consideration are initially assigned equal weights during the model’s initialization phase. Subsequently, their actual loss values at the first training stage, $L_k\left(1\right)$, serve as the respective baseline loss references.

4. Experiments

4.1. Experimental Settings

All experiments in this study were conducted on a 3090 GPU. For the CH-SIMS dataset, the “bert-base-chinese” model was employed, while for the MOSI and MOSEI datasets, the “bert-base-uncased” model was selected for fine-tuning. During fine-tuning, the learning rate was set at 0.00001, with a batch size of 64 and a total of 150 training epochs. In terms of modality information extraction, for the CH-SIMS and MOSI datasets, two single-layer transformers were used to extract audio and visual information. However, for the MOSEI dataset, considering its data characteristics and complexity, a three-layer transformer was adopted to extract information from the visual and audio modalities. Throughout the entire training process, the learning rate was uniformly set at 0.0001, with a batch size of 128 and a total of 300 training epochs.

4.2. Datasets and Baseline Models

To evaluate the performance of the DMMSA model, we selected three representative MSA datasets: CH-SIMS [18], MOSI [24], and MOSEI [23]. CH-SIMS, a resource for MSA in Chinese, comprises 2281 video samples, with sentiment labels expressed as scores within the continuous interval [−1, +1]. MOSI, an English dataset, includes 2199 video clips and employs a [−3, +3] sentiment intensity rating system. MOSEI, an extended English MSA collection derived from MOSI, significantly expands the scale to 22,856 video segments, maintaining the [−3, +3] sentiment intensity scoring range. The specific details of the dataset division are presented in

Table 1. Dataset-specific partitioning details.

Dataset	Training	Valid.	Test	Total
CH-SIMS	1368	456	457	2281
MOSI	1284	229	686	2199
MOSEI	16,326	1871	4659	22,856

.

4.3. Baseline Models

LF-DNN: This model concatenates unimodal features and analyzes sentiment [18].

MFN: This model first employs LSTM for view-specific interaction, then utilizes the attention mechanism for cross-view interaction, and finally summarizes through time with a multi-view gated memory [23].

LMF: By decomposing in parallel tensors and weights, this model utilizes modality-specific low-rank factors to perform multimodal fusion [36].

TFN: This model learns end-to-end dynamics within and across modalities. It utilizes a new multimodal fusion method (tensor fusion) to model the dynamics across modalities [26].

MulT: The core of MulT lies in its cross-modal attention mechanism, which offers a potential cross-modal adaptation by directly attending to low-level features in other modalities to fuse multimodal information [37].

MISA: This model learns modality-invariant and modality-specific representation spaces for each modality to obtain better representations for the fused input [38].

MAG-BERT: This model enhances performance by applying multimodal adaptation gates at different layers of the BERT backbone [14].

Self-MM: This model first utilizes a self-supervised label generation module to obtain unimodal labels and then jointly learns multimodal and unimodal representations based on multimodal labels [39].

ConFEDE: This model first decomposes unimodal features into modality-invariant and modality-specific features through feature decomposition. Subsequently, it utilizes multi-task learning to jointly optimize multimodal sentiment analysis, unimodal sentiment analysis, and contrastive learning tasks [5].

4.4. Evaluation Metrics

We report the model’s performance in classification and regression tasks following prior work. For classification, we computed the accuracy of three-class prediction (Accuracy-3) and five-class prediction (Accuracy-5) on CH-SIMS, as well as the accuracy of two-class prediction (Accuracy-2) and seven-class prediction (Accuracy-7) on MOSI and MOSEI. Here, Accuracy-2 and the F1-score for MOSI and MOSEI are reported in two forms: “negative/non-negative” and “negative/positive” (excluding 0). We present Mean Absolute Error (MAE) and Pearson correlation (Corr) regarding regression. All metrics, except for MAE, are better when higher. Since the output of the model is a specific value, in the context of classification tasks, we primarily determined whether the classification was accurate by checking if this value fell within the correct interval. For example, in the three-class classification task of the CH-SIMS dataset, the accuracy (ACC-3) was divided into the following intervals: [−1, −0.1], (−0.1, 0.1], and (0.1, 1]. For example, suppose the ground truth of an input sample was −0.5, and the model predicted a value of −0.3. In this case, we consider that the model has made a correct classification in the three-class classification task, as it successfully predicted the interval in which the sample’s sentiment lies.

4.5. Comparative Experiments

Table 2. Results of comparative experiment on the CH-SIMS dataset. “Acc” denotes “accuracy”.

Model	Acc-3 (↑)	Acc-5 (↑)	MAE (↓)	Corr (↑)
LF-DNN	66.91	41.62	0.420	0.612
MFN(A)	65.73	39.47	0.435	0.582
LMF	64.68	40.53	0.441	0.576
TFN	65.12	39.30	0.432	0.591
Mult(A)	64.77	37.94	0.453	0.561
Self-MM	64.73	43.15	0.414	0.598
ConFEDE	68.36	43.72	0.3924	0.6351
DMMSA	69.63	46.92	0.3778	0.66

and

Table 3. Results of comparative experiment on the MOSI and MOSEI datasets. In Acc-2 and F1, the left side of “/” represents “negative/non-negative”, and the right side represents “negative/positive”.

Model	MOSI					MOSEI
	Acc-2	F1	Acc-7	MAE	Corr	Acc-2	F1	Acc-7	MAE	Corr
LF-DNN [18]	77.52/78.63	77.46/78.63	34.52	0.955	0.658	80.60/82.74	80.85/82.52	50.83	0.58	0.709
MFN(A) [23]	77.4/-	77.3/-	34.1	0.965	0.632	78.94/82.86	79.55/82.85	51.53	0.573	0.718
LMF [40]	-/82.5	-/82.4	33.2	0.917	0.695	80.54/83.48	80.94/83.36	51.59	0.576	0.717
TFN [26]	-/80.8	-/80.7	34.9	0.901	0.698	78.50/81.89	78.96/81.74	51.60	0.573	0.714
MulT(A) [37]	-/83.0	-/82.8	40.0	0.871	0.698	81.15/84.63	81.56/84.52	52.84	0.559	0.733
MISA(A) [38]	81.8/83.4	81.7/83.6	42.3	0.783	0.776	83.6/85.5	83.8/85.3	52.2	0.555	0.756
MAG-BERT [14]	82.13/83.54	81.12/83.58	41.43	0.790	0.766	79.86/86.86	80.47/83.88	50.41	0.583	0.741
ConFEDE [5]	83.85/85.55	83.83/85.76	43.82	0.725	0.789	80.7/84.38	81.2/84.32	51.96	0.555	0.753
DMMSA	83.97/85.70	83.92/85.70	45.39	0.710	0.793	82.63/86.27	83.04/86.21	53.91	0.527	0.777

summarize the results of the comparative experiments with various methods. The listed results are based on the average of five runs with different random seeds, with the performance data for all baseline models, except for ConFEDE, sourced from published studies.

On the CH-SIMS dataset, DMMSA outperformed all baseline models in classification and regression tasks. Compared to ConFEDE, it achieved increases of 1.27% and 3.20% in Acc-3 and Acc-5, respectively. This is mainly due to the integrated coarse-grained sentiment analysis task, which boosts classification performance. DMMSA also showed significant progress in the MAE and Corr metrics. It effectively captured the interdependencies of uni- and multimodal sentiment analysis and embedded a coarse-grained task that constrained the prediction scope and simplified the analysis. As shown in Figure 4, DMMSA exhibited smaller MAE fluctuations and faster convergence during training than models without coarse-grained analysis, confirming its positive role in optimization.

To further validate our approach, experiments were conducted on the MOSI and MOSEI datasets without unimodal sentiment labels. Table 3 shows that DMMSA surpassed all baselines, demonstrating excellent performance even without unimodal labels. This is mainly due to the design of the contrastive learning task. As shown in Figure 5, the model was still able to identify and separate uni- and multimodal features effectively under the guidance of this task. Notably, DMMSA’s improvements in Acc-5, MAE, and Corr were more significant than those in Acc-2 and Acc-3. This is because complex tasks require higher model performance and feature quality. Simple tasks can yield good results with lower-level features, while DMMSA’s advantage lies in extracting higher-quality features, making its performance gain more prominent as the task difficulty increases.

To confirm this hypothesis, we designed an incremental experiment.

Table 4. Performance of DMMSA under varying amounts of training data.

Data	Acc-2	F1	Acc-7	MAE	Corr
MOSEI	82.63/86.27	83.04/86.21	53.91	0.527	0.777
MOSEI*0.8	82.41/86.15	82.87/86.14	53.39	0.532	0.772
MOSEI*0.6	83.77/86.12	84.01/85.99	52.77	0.538	0.769
MOSEI*0.4	82.78/86.09	83.16/86.03	53.36	0.540	0.767
MOSEI*0.2	80.68/85.03	81.27/85.06	52.37	0.551	0.760
MOSEI*0.1	81.89/85.08	82.33/85.04	52.18	0.552	0.755

shows that DMMSA’s performance in Acc-2 converged when trained with 60% of the data and did not improve with more data. In contrast, its performance in Acc-7 and regression tasks continuously improved as the amount of training data increased.

4.6. Ablation Study and Analysis

We carried out an ablation study on the proposed method to explore the individual contributions of each module to model performance. The results are presented in

Table 5. The ablation experiments on CH-SIMS. “w/o CL” signifies the exclusion of the contrastive learning (CL) task.

Model	Acc-3	Acc-5	MAE	Corr
DMMSA	69.63	46.92	0.3778	0.66
w/o MSC	68.41	44.68	0.3807	0.656
w/o CL	69.41	46.74	0.3828	0.651
w/o GDWG	69.32	46.17	0.3776	0.663

.

We can observe that the model’s performance decreased to varying degrees under the three ablation strategies. The “w/o MSC” strategy resulted in decreases across all performance metrics. The decline can primarily be attributed to the loss of practical constraints on the sentiment prediction range after removing the MSC task. Under the “w/o CL” configuration, the model’s MAE and Corr. indicators significantly decreased. This is mainly because the core objective of CL tasks is to assist the model in effectively distinguishing and extracting similarity and dissimilarity features from single modalities. Once the CL task was removed, the model lost the feature discrimination ability promoted by this mechanism, making it difficult to accurately distinguish and utilize these critical sentiment features; so, its performance was weakened in regression tasks.

In the “w/o GDWG” setting, the model showed a slight upward trend in performance in regression tasks but a significant decline in classification tasks. The reason for this phenomenon is that the model lost the effective regulatory mechanism for gradient convergence rates and magnitude differences among the various tasks during training. As a result, the model tended to over-optimize a single task at the cost of neglecting the learning needs of other tasks, leading to an obvious imbalance in overall performance.

5. Conclusions

This study presents DMMSA, an affective analysis framework. It combines multi-task learning strategies with dynamic tuning mechanisms to improve the accuracy of understanding complex human sentiments. This is achieved by leveraging the inherent correlations between unimodal and multimodal sentiment signals. DMMSA systematically extracts and decomposes sentiment representations from multimodal inputs into similarity and dissimilarity parts. These are then enhanced via coarse-grained sentiment classification tasks and contrastive learning mechanisms that work on the interaction of sentiment representations. To fully assess DMMSA, it is evaluated on three typical MSA datasets: CH-SIMS, MOSI, and MOSEI. Experimental results show that DMMSA outperforms various benchmark models in all overall performance metrics across all datasets. Additionally, a series of ablation experiments further confirms the essential contribution of each component module in DMMSA to the overall performance enhancement, validating the methodological soundness and effectiveness of this design.

6. Discussion

The DMMSA model has achieved remarkable performance across multiple datasets, which can be attributed to several key factors. First, its multi-task learning strategy incorporates single-modality sentiment classification tasks and contrastive learning tasks, enabling the model to gain a more comprehensive understanding and analysis of sentiment signals. This strategy not only enhances the model’s comprehension of single-modality sentiment but also optimizes the feature extraction and decomposition process through the contrastive learning mechanism. Second, the Global Dynamic Weight Generation (GDWG) strategy dynamically adjusts task weights based on the convergence speed of tasks, effectively avoiding the negative transfer issues in multi-task learning. This mechanism ensures that the model balances the learning progress of different tasks during training, thereby improving overall performance. Additionally, by introducing coarse-grained sentiment classification tasks, the DMMSA model more accurately predicts sentiment intensity. This task simplifies the complexity of sentiment analysis by restricting the prediction range, thereby enhancing the model’s prediction accuracy. Compared with existing baseline models, the DMMSA model has achieved significant improvements in multiple key metrics. For instance, on the CH-SIMS dataset, the DMMSA model’s Acc-3 and Acc-5 values were 1.27% and 3.20% higher than those of the ConFEDE model. On the MOSI and MOSEI datasets, the DMMSA model outperformed all baseline models in metrics such as Acc-2, Acc-7, MAE, and Corr. These results indicate that while optimizing the model structure is important, optimizing the training objectives can also effectively enhance model performance.

7. Limitations

Although we have alleviated the negative transfer effects caused by differences in task convergence rates using the GDWG strategy, this problem still exists and is a key factor restricting the performance improvement of DMMSA. Table 4 shows that, as the training sample size increases, the performance of DMMSA in Acc-2 decreases, while in Acc-5, MAE, and Corr, the model shows an upward trend. Therefore, the focus of future research will be on exploring the optimization paths for GDWG. For instance, methods such as dynamic weighting based on meta-learning, dynamic parameter adjustment based on reinforcement learning, and multi-task learning based on transfer learning will all be considered. These methods will help more effectively suppress negative transfer and thereby enhance the overall performance of the model.