DMBT Decoupled Multi-Modal Binding Transformer for Multimodal Sentiment Analysis

Abstract

The performance of Multimodal Sentiment Analysis (MSA) is commonly hindered by two major bottlenecks: the complexity and redundancy associated with supervised feature disentanglement and the coarse granularity of static fusion mechanisms. To systematically address these challenges, a novel framework, the Decoupled Multi-modal Binding Transformer (DMBT), is proposed. The framework first introduces an Unsupervised Semantic Disentanglement (USD) module, which resolves the issue of complex redundancy by cleanly separating features into modality-common and modality-specific components in a lightweight, parameter-free manner. Subsequently, to tackle the challenge of coarse-grained fusion, a Gated Interaction and Fusion Transformer (GIFT) is constructed as the core engine. The exceptional performance of GIFT is driven by two synergistic components. The first is a Multi-modal Binding Transposed Attention (MBTA) that employs a hybrid convolutional and attention model to concurrently perceive both global context and local fine-grained features, and then a Dynamic Fusion Gate (DFG) that performs final, adaptive decision-making by re-weighting all deeply enhanced representations. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that the proposed DMBT framework surpasses existing state-of-the-art models across all key evaluation metrics. The efficacy of each innovative component is further validated through comprehensive ablation studies.

1. Introduction

Multimodal Sentiment Analysis (MSA) emerges as a pivotal tool for interpreting cross-modal affective semantics, demonstrating critical value in applications such as mental health monitoring and immersive interactive systems [1]. As shown in Figure 1, unlike unimodal approaches, MSA enhances the robustness of sentiment understanding by fusing heterogeneous information from language, vision, and audio [2]. In recent years, the Transformer architecture, owing to its powerful context-modeling capabilities, has become the mainstream framework for this task, with seminal works like MulT [3] successfully addressing challenges such as multimodal asynchrony.

Despite this success, Transformer-based approaches in MSA are increasingly confronted with a critical dilemma balancing performance with efficiency. On one hand, many existing fusion strategies suffer from parameter redundancy and computational inefficiency. As highlighted by recent studies, the parameter count in many fusion models can grow exponentially with the number of modalities, severely impeding their training and practical deployment [4]. To facilitate more granular interaction, methods that separate features into modality-common and modality-specific components have become prevalent. However, this has introduced another layer of complexity, as mainstream paradigms rely heavily on cumbersome supervised learning. This approach not only results in convoluted model architectures and training instability but also fails to guarantee the purity of the separated features, thereby compromising the efficacy of subsequent fusion.

Concurrently, the rise in Multimodal Large Language Models (MLLMs) has introduced a new paradigm, demonstrating remarkable flexibility in generating personalized, high-quality content by integrating text and vision [5]. However, their widespread adoption is currently hindered by prohibitive computational costs and the need for massive-scale datasets, making them impractical for many real-world applications [6]. This creates a clear and pressing research gap: there is an urgent need for a framework that strikes a balance between performance, efficiency, and accessibility. Such a framework must achieve lightweight, unsupervised feature disentanglement while simultaneously performing intelligent, dynamic feature fusion, all without relying on the colossal resources required by MLLMs.

To fill this void, this paper proposes a novel Decoupled Multi-modal Binding Transformer (DMBT) framework. To address the dilemma of disentanglement efficiency and purity, an Unsupervised Semantic Disentanglement (USD) module is first introduced. This module discards complex supervised paradigms to achieve clean feature separation in a lightweight, parameter-free manner. Subsequently, to overcome the limitations of coarse and static fusion, a Gated Interaction and Fusion Transformer (GIFT) is constructed. Through its core components—the Multi-modal Binding Transformer Attention (MBTA) and the Dynamic Fusion Gate (DFG)—GIFT synergistically resolves the challenges of fine-grained feature extraction and dynamic decision-making.

The main contributions of this work are summarized as follows:

●

A novel, end-to-end framework for MSA, termed DMBT, is proposed. It systematically addresses the limitations of existing methods through two core innovative modules for feature disentanglement and fusion.

●

An efficient Unsupervised Semantic Disentanglement (USD) module is designed and validated. It integrates an unsupervised, parameter-free paradigm with a language-focused framework, significantly enhancing model efficiency, stability, and disentanglement purity.

●

A new Gated Interaction and Fusion Transformer (GIFT) is constructed, which serves as the core engine of the DMBT model. Its superior performance stems from the synergy between its two integral components: the Multi-modal Binding Transposed Attention (MBTA) and the Dynamic Fusion Gate (DFG).

2. Related Work

2.1. Multimodal Fusion Strategies

The core challenge in MSA is the effective fusion of heterogeneous information [7,8,9]. Foundational research established early (feature-level) and late (decision-level) fusion paradigms, utilizing methods from simple concatenation to more complex tensor fusion to model high-order interactions [9,10,11,12]. Although foundational, these approaches exhibit limitations in managing modal heterogeneity, noise, and dynamic temporal semantics.

To address these limitations, Transformer-based methods have become the mainstream due to their powerful sequence modeling capabilities. MulT [3] stands as a milestone, enabling end-to-end modeling of temporally unaligned sequences via a directional cross-modal attention mechanism. Subsequent research has refined this framework from various perspectives, such as cyclical modality translation [13], hierarchical modeling, and knowledge distillation [14,15]. More recently, a notable trend has been the exploration of language-centric fusion, where the textual modality serves as an anchor to guide the integration of information from other modalities [16,17,18].

However, the increasing sophistication of these models has highlighted a critical trade-off with efficiency, spurring a trend toward lightweight multimodal Transformers. This concern is not unique to MSA. For instance, in medical imaging, MACTFusion proposes a lightweight cross-Transformer that uses a novel cross-axis attention to reduce computational complexity [19]. By employing a cycle-attention mechanism, Cy-Atten achieves a linear O(N) complexity for adding new modalities, demonstrating the feasibility of creating highly efficient yet powerful fusion models [4].

This landscape reveals two primary limitations that our work addresses. First, most specialized frameworks still fuse raw, coupled features, limiting performance. Second, there is a clear need for models that are both performant and computationally efficient. These are the precise challenges that the proposed DMBT framework aims to resolve.

2.2. Disentangled Representation Learning

To address modality heterogeneity at a deeper level, disentangled representation learning, pioneered by Tsai et al. [20], has become a key research area. The goal is to decompose features into modality-invariant (shared) and modality-specific subspaces before fusion.

One major line of work is supervised learning based on constraint optimization. These methods typically employ parallel encoders and a suite of meticulously designed loss functions (e.g., similarity, difference, reconstruction, triplet loss) to enforce the separation of shared and specific features [21,22,23]. However, they often suffer from architectural complexity and training instability.

More recently, contrastive learning has emerged as a powerful paradigm for achieving more robust and fine-grained disentanglement. This trend is evident across different domains. Wei et al. used a cross-temporal contrastive model (CCDM) to disentangle ancient and modern Chinese into shared semantics, shared syntax, and different syntax, effectively bridging the cross-temporal gap. This shows how disentanglement is a fundamental strategy for handling any form of data heterogeneity, be it across time or across modalities [24]. In computer vision, CoDeGAN goes a step further by entirely replacing traditional mutual information objectives with a contrastive loss for unsupervised class disentanglement in GANs, arguing that this leads to more stable training and avoids issues like mode collapse [25].

These cutting-edge works demonstrate that contrastive learning is becoming a mainstream approach for effective disentanglement. However, a common thread among most existing paradigms—whether based on complex supervision like MISA and DLF [21,23] or advanced techniques like contrastive learning is their reliance on parameter-heavy parallel networks and intricate training procedures. This heavy dependence incurs significant computational costs and fails to guarantee complete feature separation, often resulting in residual semantic entanglement. This limitation motivates our exploration of a more lightweight and direct unsupervised disentanglement approach, which is the foundational principle of our USD module.

The design of the DMBT framework is informed by a systematic analysis of the aforementioned literature. It is predicated on the insight that an ideal MSA framework must combine the efficiency and purity of front-end disentanglement with the intelligence and dynamism of back-end fusion. Therefore, this work does not represent an incremental improvement in a single method but rather an organic synthesis of the strengths of different technical routes. For feature disentanglement, inspired by the advantages of alternative, non-learning paradigms, the proposed USD module is parameter-free and similarity-based, ensuring highly efficient and pure feature separation. These separated representations are then processed by the GIFT core, which synergistically resolves the challenges of fine-grained feature extraction and dynamic decision-making through its hybrid attention mechanism and gating controls. In essence, DMBT offers a novel, high-performance solution for MSA by uniquely combining efficient unsupervised disentanglement with a deeply enhanced, dynamically adaptive fusion framework.

3. Proposed Approach

To systematically address the prevalent challenges of supervised disentanglement and static fusion in existing MSA frameworks, a novel Disentangled Multimodal Binding Transformer (DMBT) is proposed. As illustrated in Figure 2, the DMBT architecture is organized as a four-stage, end-to-end pipeline: (1) Feature Extraction, (2) Unsupervised Disentanglement, Language-centric Feature Enhancement, and (4) Gated Fusion and Prediction.

The pipeline commences with standard encoders for initial Feature Extraction. Subsequently, to circumvent the complexities of supervised methods, the features are processed by an Unsupervised Semantic Disentanglement (USD) module. This parameter-free module decomposes the features of each modality into modality-invariant (common) and modality-specific components, establishing a clean foundation for subsequent processing. (3) The core of the framework is the Gated Interaction and Fusion Transformer (GIFT), which orchestrates the enhancement and fusion stages. Within GIFT, a language-centric paradigm is adopted where the textual modality serves as an anchor. The Multimodal Binding Transposed Attention (MBTA) mechanism performs targeted, unidirectional enhancement of the audio and visual features, addressing the issue of insufficient feature granularity. Finally, all common and enhanced specific features are fed into a Dynamic Fusion Gate (DFG). The DFG performs sample-wise dynamic weighting to render a final prediction, thereby overcoming the static fusion bottleneck.

3.1. Initial Feature Representation

Given a multimodal input $X^m$ for modality $\mathrm{m}\in \{\mathrm{t},\mathrm{a},\mathrm{v}\}$, where $\mathrm{t},\mathrm{a},\mathrm{v}$ denote text, audio, and vision, respectively, the raw feature sequences are first projected into a unified dimension, ${\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}$, via modality-specific 1D convolutional layers (${\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}}^{\mathrm{m}}$):

$${\mathrm{F}}^{\mathrm{m}}={\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}}^{\mathrm{m}}\left({\mathrm{X}}^{\mathrm{m}}\right)\in {\mathbb{R}}^{{\mathrm{L}}_{\mathrm{m}}\times {\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}},\hspace{1em}\mathrm{m}\in \{\mathrm{t},\mathrm{a},\mathrm{v}\}$$

(1)

where ${\mathrm{L}}_{\mathrm{m}}$ is the sequence length of modality m, and ${\mathrm{F}}^{\mathrm{m}}$ is the resulting dimensionally aligned feature representation.

3.2. Unsupervised Semantic Disentanglement (USD)

To obviate the complexity and instability of supervised disentanglement, the proposed USD module operates in a parameter-free manner based on geometric similarity. It decomposes each projected feature representation ${\mathrm{F}}^{\mathrm{m}}$ into a modality-common component ${\mathrm{C}}^{\mathrm{m}}$ and a modality-specific component ${\mathrm{S}}^{\mathrm{m}}$.

Specifically, for a given source modality (e.g., text, $\mathrm{t}$), its semantic affinity with a target modality $n\in \{a,v\}$ is quantified by a pairwise cosine similarity matrix:

$${\mathrm{S}\mathrm{i}\mathrm{m}}^{\mathrm{t}\mathrm{n}}={\displaystyle \frac{{\mathrm{F}}^{\mathrm{t}}{\left({\mathrm{F}}^{\mathrm{n}}\right)}^{\mathrm{T}}}{‖{\mathrm{F}}^{\mathrm{t}}‖‖{\mathrm{F}}^{\mathrm{n}}‖}}\in {\mathbb{R}}^{{\mathrm{L}}_{\mathrm{t}}\times {\mathrm{L}}_{\mathrm{n}}}$$

(2)

The underlying principle is that a feature is considered “common” only if it exhibits high semantic relevance to all other modalities. Therefore, a shared weight matrix ${\mathrm{W}}_{\mathrm{c}}^{\mathrm{t}}$ is derived by retaining the minimum similarity score across modalities. This approach is more robust than an average, which could mask low alignment with one modality, or a maximum, which would be overly permissive. The resulting consensus scores are then normalized via a softmax function to produce the final weight matrix:

$${\mathrm{W}}_{\mathrm{c}}^{\mathrm{t}}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\min \left({\mathrm{S}\mathrm{i}\mathrm{m}}^{\mathrm{t}\mathrm{a}},{\mathrm{S}\mathrm{i}\mathrm{m}}^{\mathrm{t}\mathrm{v}}\right)\right)\in {\mathbb{R}}^{{\mathrm{L}}_{\mathrm{t}}\times {\mathrm{L}}_{\mathrm{a}}} \mathrm{o}\mathrm{r} {\mathbb{R}}^{{\mathrm{L}}_{\mathrm{t}}\times {\mathrm{L}}_{\mathrm{v}}}$$

(3)

The weight matrix ${\mathrm{W}}_{\mathrm{c}}^{\mathrm{t}}$ is then used to project the original features ${\mathrm{F}}^{\mathrm{t}}$ onto the modality-invariant subspace, yielding the common component ${\mathrm{C}}^{\mathrm{t}}$. The specific component ${\mathrm{S}}^{\mathrm{t}}$ is subsequently derived via subtraction, which geometrically corresponds to an orthogonal decomposition. The modality-common (${\mathrm{C}}^{\mathrm{t}}$) and modality-specific (${\mathrm{S}}^{\mathrm{t}}$) features are then calculated as:

$$\begin{array}{l}{\mathrm{C}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{c}}^{\mathrm{t}}\cdot {\mathrm{F}}^{\mathrm{t}}\\ {\mathrm{S}}^{\mathrm{t}}={\mathrm{F}}^{\mathrm{t}}-{\mathrm{C}}^{\mathrm{t}}\end{array}$$

(4)

where ${\mathrm{C}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{t}}$ $\in {\mathbb{R}}^{L_t\times d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}}$. This process is applied symmetrically to obtain $\left\{{\mathrm{C}}^{\mathrm{a}},{\mathrm{S}}^{\mathrm{a}}\right\}$ and $\left\{{\mathrm{C}}^{\mathrm{v}},{\mathrm{S}}^{\mathrm{v}}\right\}$.

3.3. Gated Interaction and Fusion Transformer (GIFT)

Following disentanglement, the GIFT module performs deep feature interaction and fusion. It employs a language-centric strategy where the specific textual features ${\mathrm{S}}^{\mathrm{t}}$ serve as an anchor to unidirectionally fuse complementary information from the audio (${\mathrm{S}}^{\mathrm{a}}$) and visual (${\mathrm{S}}^{\mathrm{v}}$) modalities. This asymmetric approach maximizes the semantic guidance of language while mitigating potential redundancy from multi-directional interactions.

3.3.1. Common Feature Stream Processing

The common features $\left({\mathrm{C}}_{\mathrm{t}},{\mathrm{C}}_{\mathrm{a}},{\mathrm{C}}_{\mathrm{v}}\right)$ are processed to form a robust consensus representation. They are first concatenated and then fed into a dedicated shared Transformer encoder to refine and integrate the shared information across modalities:

$$\begin{array}{l}{\mathrm{C}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathrm{C}}_{\mathrm{t}},{\mathrm{C}}_{\mathrm{a}},{\mathrm{C}}_{\mathrm{v}}\right)\\ {\mathrm{C}}_{\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{d}}=\mathrm{S}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\left({\mathrm{C}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}\right)\end{array}$$

(5)

The final state of the encoded sequence is then projected through a two-layer feed-forward network (${\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}}_{\mathrm{c}}$) to yield the final shared representation, ${\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }$:

$${\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }={\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}}_{\mathrm{c}}\left({\mathrm{C}}_{\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{d}}\right)$$

(6)

This representation, ${\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }$, encapsulates the unified, modality-invariant information and is subsequently used for both auxiliary prediction and final fusion.

3.3.2. Specific Feature Stream Processing

The core of GIFT is the Multimodal Binding Transposed Attention (MBTA) which performs deep modeling of the specific features $\left({\mathrm{S}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{a}},{\mathrm{S}}^{\mathrm{v}}\right)$. As shown in Figure 3, MBTA enhances standard multi-head self-attention by integrating a 1D convolutional layer into the Query ($\mathrm{Q}$) and Key ($\mathrm{K}$) paths. This allows for the simultaneous capture of both local temporal patterns and global contextual dependencies. The computation for a single attention head is:

$$\begin{array}{l}{\mathrm{Q}}^{\prime }=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}1\mathrm{d}\left(\mathrm{Q}\right),\hspace{1em}{\mathrm{K}}^{\prime }=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}1\mathrm{d}\left(\mathrm{K}\right)\\ \mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\mathrm{Q},\mathrm{K},\mathrm{V})=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{{\mathrm{Q}}^{\prime }{\left({\mathrm{K}}^{\prime }\right)}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}}\right)\mathrm{V}\end{array}$$

(7)

Within the language-centric paradigm, MBTA executes three key unidirectional enhancement operations:

$$\begin{array}{l}{\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{t}}=\mathrm{M}\mathrm{B}\mathrm{T}\mathrm{A}\left({\mathrm{S}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{t}}\right)\\ {\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{a}\to \mathrm{t}}=\mathrm{M}\mathrm{B}\mathrm{T}\mathrm{A}\left({\mathrm{S}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{a}},{\mathrm{S}}^{\mathrm{a}}\right)\\ {\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{v}\to \mathrm{t}}=\mathrm{M}\mathrm{B}\mathrm{T}\mathrm{A}\left({\mathrm{S}}^{\mathrm{t}},{\mathrm{S}}^{\mathrm{v}},{\mathrm{S}}^{\mathrm{v}}\right)\end{array}$$

(8)

The final enhanced specific representation, ${\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}}$, is the summation of these three outputs:

$${\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}}={\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{t}}+{\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{a}\to \mathrm{t}}+{\mathrm{S}}_{\mathrm{o}\mathrm{u}\mathrm{t}}^{\mathrm{v}\to \mathrm{t}}$$

(9)

3.3.3. Dynamic Fusion Gate (DFG)

The DFG serves as the final decision-making mechanism. It takes the shared representation ${\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }$ and the enhanced specific representation ${\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}}$, concatenates them, and computes a dynamic gating weight $z\in \left[\mathrm{0,1}\right]$ via a Multi-Layer Perceptron (MLP) with a sigmoid activation:

$${\mathrm{F}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime },{\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}},\dots \right)$$

(10)

$$\mathrm{z}=\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}\left(\mathrm{M}\mathrm{L}\mathrm{P}\left({\mathrm{F}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\right)\right)$$

(11)

This gate adaptively weights the fused features, enabling the model to dynamically balance the importance of different information sources on a per-sample basis.

3.4. Hierarchical Prediction and Training Objective

To foster robust intermediate representations, a deep supervision strategy with hierarchical prediction is employed. In addition to the final prediction from the DFG output (${\mathrm{F}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$), auxiliary prediction heads are attached to the intermediate common (${\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }$) and enhanced specific (${\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}}$) representations:

$$\begin{array}{l}{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{c}}={\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{c}}\left({\mathrm{C}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}^{\prime }\right)\\ {\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{s}}={\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{s}}\left({\mathrm{S}}_{\mathrm{e}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}}\right)\\ {\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{f}}={\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{f}}\left({\mathrm{F}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}\right)\end{array}$$

(12)

where ${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{c}}$, ${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{s}}$, and ${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}_{\mathrm{f}}$ are independent predictors. The total training objective, ${\mathrm{L}}_{\mathrm{D}\mathrm{M}\mathrm{B}\mathrm{T}}$, is a weighted sum of the L1 losses computed for each prediction against the ground truth label ${\mathrm{Y}}_{\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}}$:

$$\begin{array}{l}{\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{c}}=\mathrm{L}1\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}\left({\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{c}},{\mathrm{Y}}_{\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}}\right)\\ {\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{s}}=\mathrm{L}1\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}\left({\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{s}},{\mathrm{Y}}_{\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}}\right)\\ {\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{f}}=\mathrm{L}1\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}\left({\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}}_{\mathrm{f}},{\mathrm{Y}}_{\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}}\right)\end{array}$$

(13)

$${\mathrm{L}}_{\mathrm{D}\mathrm{M}\mathrm{B}\mathrm{T}}={\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{f}}+\alpha {\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{c}}+\beta {\mathrm{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\_\mathrm{s}}$$

(14)

where $\alpha$ and $\beta$ are hyperparameters controlling the influence of the auxiliary losses. This hierarchical objective provides direct gradient signals to intermediate layers, mitigating potential gradient vanishing and ensuring that each stage of the model learns task-relevant features. This, in turn, supplies the DFG with high-quality inputs for its final adaptive fusion.

4. Experiment

4.1. Datasets and Evaluation Metrics

Datasets. Experiments are conducted on two standard benchmark datasets for MSA:

CMU-MOSI [26]: A dataset comprising 2199 monologue video clips. Each clip is annotated with a sentiment score from −3 (highly negative) to +3 (highly positive). The standard data partition consists of 1284 training, 229 validation, and 686 test samples.

CMU-MOSEI [27]: A significantly larger dataset containing 22,856 movie review clips, annotated identically to CMU-MOSI. The standard partition includes 16,326 training, 1871 validation, and 4659 test samples.

Evaluation Metrics. For a comprehensive and fair comparison with prior work, model performance is evaluated from both classification and regression perspectives. Classification metrics include 7-class accuracy (Acc-7), 5-class accuracy (Acc-5), binary accuracy (Acc-2), and the binary F1-Score. Regression metrics include the Mean Absolute Error (MAE) and Pearson Correlation Coefficient (Corr). For all metrics except MAE, higher values indicate better performance.

4.2. Implementation Details

Feature Extraction. Following standard protocols, textual features are extracted using a bert-base-uncased model (768-D). Audio and visual features are extracted using the COVAREP and Facet toolkits, respectively.

Model and Training. The DMBT model is implemented in PyTorch 1.13.0 and trained on a single NVIDIA V100 GPU. Hyperparameters are tuned separately for each dataset. The best-performing configuration on MOSI, for instance, utilizes an internal feature dimension (d_model) of 60, 10 attention heads, and 2 layers (n_levels) within the GIFT module. The model is trained using the Adam optimizer with a batch size of 16 and an initial learning rate of $1\times {10}^{-4}$. An early stopping criterion is applied, terminating the training if the validation loss fails to improve for 10 consecutive epochs. To ensure the robustness and reproducibility of our findings, all reported experimental results are the average of five independent runs using different random seeds.

4.3. Main Results and Analysis

The proposed DMBT model is benchmarked against 12 representative baseline models, spanning from early tensor fusion methods to recent Transformer-based approaches, including TFN [10], LMF [11], EF-LSTM [28], LF-DNN [29], MFN [30], Graph-MFN [27], MulT [3], PMR [31], MISA [21], MAG-BERT [32], DMD [33], and DLF [23]. The comparison results on the CMU-MOSI and CMU-MOSEI datasets are presented in

Table 1. Performance comparison of DMBT against baseline models on the CMU-MOSI test set.

Method	Acc-7 (↑)	Acc-5 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
TFN *	34.9	39.39	80.08	80.07	0.698	0.901
LMF *	33.2	38.13	82.5	82.4	0.695	0.917
EF-LSTM †	35.39	40.15	78.48	78.51	0.669	0.949
LF-DNN †	34.52	38.05	78.63	78.63	0.658	0.955
MFN †	35.83	40.47	78.87	78.9	0.67	0.927
Graph-MFN †	34.64	38.63	78.35	78.35	0.649	0.956
MulT	40	42.68	83	82	0.698	0.871
PMR	40.6	-	83.6	83.6	-	-
MISA †	41.37	47.08	83.54	83.58	0.778	0.777
MAG-BERT	43.62	-	84.43	84.61	0.781	0.727
DMD **	44.3	49.34	84.33	84.22	0.769	0.741
DLF	47.08	52.33	85.06	85.04	0.781	0.731
DMBT(Ours)	46.48	52.34	85.52	85.48	0.784	0.718

Bold: The best results for each metric. ^† denotes results from THUIAR’s GitHub page (THUIAR 2024), * denotes results from [21], - denotes results not provided in the original paper, and ** denotes replicated results from public code with hyperparameters provided in the original paper.

and

Table 2. Performance comparison of DMBT against baseline models on the CMU-MOSEI test set.

Method	Acc-7 (↑)	Acc-5 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
TFN *	50.2	53.1	82.5	82.1	0.7	0.593
LMF *	48	52.9	82	82.1	0.677	0.623
EF-LSTM †	50.01	51.16	80.79	80.67	0.683	0.601
LF-DNN †	50.83	51.97	82.74	82.52	0.709	0.58
MFN†	51.34	52.76	82.85	82.85	0.718	0.575
Graph-MFN †	51.37	52.69	83.48	83.43	0.713	0.575
MulT	51.8	54.18	82.5	82.3	0.703	0.58
PMR	52.5	-	83.6	83.4	-	-
MISA †	52.05	53.63	84.67	84.66	0.752	0.558
MAG-BERT	52.67	-	84.82	84.71	0.755	0.543
DMD **	53.68	54.52	85.25	85.11	0.759	0.54
DLF	53.9	55.7	85.42	85.27	0.764	0.536
DMBT(Ours)	53.73	56.27	85.74	85.72	0.77	0.532

Bold: The best results for each metric. ^† denotes results from THUIAR’s GitHub page (THUIAR 2024), * denotes results from [21], - denotes results not provided in the original paper, and ** denotes replicated results from public code with hyperparameters provided in the original paper.

, respectively.

The results indicate that DMBT consistently achieves state-of-the-art or highly competitive performance across all key metrics on both datasets. When compared with recent disentanglement-based models that rely on complex supervision (e.g., MISA, DMD, DLF), DMBT surpasses MISA and DMD on all reported metrics. While DLF shows a slight advantage on the Acc-7 metric for MOSI, which may stem from its complex supervision providing a stronger inductive bias for fine-grained classification, DMBT outperforms it on all other metrics. This suggests that the proposed lightweight, parameter-free USD module achieves a more effective and robust feature disentanglement.

Furthermore, DMBT demonstrates a consistent advantage over other strong Transformer-based baselines (e.g., MulT, MAG-BERT). Notably, on metrics that assess a deeper understanding of sentiment dynamics, such as Correlation (Corr) and Mean Absolute Error (MAE), DMBT sets a new state-of-the-art. This advantage is primarily attributed to the proposed GIFT framework. Its core engine, MBTA, captures richer, fine-grained local features in addition to global context, while its decision-making mechanism, DFG, enables more refined, sample-wise adaptive fusion. The strong performance on the larger and more complex MOSEI dataset further validates the scalability and robustness of the DMBT architecture.

4.4. Ablation Study

To verify the contribution of each key component, a series of ablation studies was conducted on the MOSI dataset by systematically removing the USD, MBTA, and DFG modules from the full DMBT model. The results are presented in

Table 3. Ablation study results on the CMU-MOSI dataset. The best results are in bold.

Method	Acc-7 (%)	Acc-2 (%)	F1 (%)	MAE (↓)
DMBT(Ours)	46.48	85.52	85.48	0.718
Different Modalities
only A	15.45	42.23	25.07	1.468
only V	15.12	42.68	28.14	1.470
only L	44.15	84.45	84.37	0.764
L & A	44.37	84.70	84.61	0.752
L & V	44.31	84.21	84.03	0.758
Different Components
w/o USD	44.02	83.08	83.12	0.765
w/o MBTA	45.61	84.54	83.75	0.781
w/o DFG	44.31	83.62	83.56	0.760

.

Various Modalities Combinations. To assess modality contributions and fusion efficacy, we evaluated DMBT under various input configurations (Table 3). Unimodal analysis confirms the expected dominance of language, which significantly outperforms audio-only or visual-only inputs, establishing it as the primary source of sentiment information. Subsequent bi-modal experiments (L & A, L & V) demonstrate marked performance improvements over the language-only baseline, empirically validating that our language-centric framework effectively extracts and integrates valuable complementary information from the auxiliary visual and audio modalities via the GIFT module. Finally, the superior performance achieved by the full tri-modal model underscores that each modality offers unique contributions, which our proposed architecture successfully synergizes to yield the most accurate sentiment predictions.

Effectiveness of USD: Removing the USD module and reverting to a standard feature fusion approach (as in MulT) leads to significant performance degradation across all metrics. This confirms that the proposed lightweight unsupervised disentanglement provides purer, higher-quality features for downstream fusion.

Effectiveness of MBTA: Replacing the MBTA module with a standard Transformer encoder also results in a consistent performance drop. This indicates that the hybrid convolutional-attentional design of MBTA, which simultaneously models local and global dependencies, is more effective for complex multimodal interactions than standard self-attention alone.

Effectiveness of DFG: Eliminating the DFG and using simple feature concatenation for the final prediction causes a notable decline in performance, particularly on the MAE and Acc-7 metrics. This highlights the importance of the adaptive, sample-wise gating mechanism for intelligent decision-making.

To validate our design choice of using minimum similarity aggregation in the USD module, we conducted a comparative experiment against two direct alternatives: cosine-max and cosine-mean. The results are presented in

Table 4. Ablation study on the aggregation strategy in the USD module on the MOSI dataset. The best results are in bold.

Aggregation Strategy	Acc-7 (%)	Acc-2 (%)	F1 (%)	MAE (↓)
min(Ours)	46.48	85.52	85.48	0.718
max	44.02	83.08	83.12	0.765
mean	45.61	84.54	83.75	0.781

.

The empirical data strongly corroborates our theoretical predictions. The aggregation strategy, which operates on a principle of strict consensus, consistently and significantly outperforms both the mean and max strategies across all key evaluation metrics.

The max strategy, being overly permissive by design, likely incorporates noisy or modality-pair-specific information into the “common” representation, thereby degrading performance.

The mean strategy, while a reasonable compromise, tends to blur the distinction between truly common and partially relevant features, leading to a suboptimal, “polluted” representation.

In contrast, our min strategy successfully identifies the purest set of modality-invariant features. By providing the downstream GIFT module with the cleanest possible inputs, it enables a more effective and robust fusion process, ultimately leading to superior model performance.

In summary, these ablation studies provide compelling evidence that the superior performance of DMBT arises from the synergistic contributions of each of its core architectural innovations.

4.5. Further Analysis

Performance on Sentiment Categories. To dissect the model’s behavior, the 7-class confusion matrix and per-category accuracies on the MOSI test set are visualized in Figure 4. The model demonstrates high accuracy for the five core sentiment categories (N, WN, NT, WP, P). However, a notable performance drop is observed for the extreme categories (HN and HP), which the model tends to confuse with their less intense, adjacent counterparts. This difficulty in discerning subtle intensity differences is a primary reason for the model’s performance on the Acc-7 metric.

Visualization of Disentangled Representations. To investigate the quality of the learned representations, the disentangled feature spaces of DMBT and a strong baseline (DLF) are visualized using t-SNE in Figure 5. The visualization provides strong qualitative evidence for the superiority of the USD module. The feature clusters learned by DMBT exhibit significantly better intra-class compactness and inter-class separability compared to those learned by DLF, which shows considerable overlap between its “Text-Common” and “Text-Specific” representations.

Data-Driven Rationale for Performance on Extremes. The clearer feature separation in Figure 5 also offers a deeper, data-driven explanation for the lower accuracy on extreme sentiment classes. The issue appears to be rooted in the dataset’s intrinsic properties rather than architectural limitations.

Data Imbalance: The “HN” and “HP” classes suffer from a long-tail distribution, with far fewer samples than other categories, which hinders the model’s ability to learn their distinguishing features.

Inherent Semantic Ambiguity: The perceptual difference between “negative” and “highly negative” is inherently subtle across language, visual, and acoustic cues. Consequently, their feature representations are naturally closer in the semantic space. A model with a superior representational capacity, such as DMBT, will more faithfully reflect this intrinsic data ambiguity, which can paradoxically lead to confusion during classification between semantically adjacent but distinct classes.

5. Conclusions

In this paper, we proposed a novel, end-to-end DMBT framework to improve the performance of MSA. The core of DMBT is twofold: The USD module and GIFT with MBTA engine and DFG. Extensive experiments and comprehensive ablation studies demonstrated the superiority of DMBT.

Despite these promising results, we acknowledge several limitations that pave the way for future work. As our analysis revealed, the model’s performance is noticeably lower on extreme sentiment categories, largely due to the long-tail distribution of the training data. Moreover, our current framework assumes the presence of complete modal data, which may not hold true in real-world scenarios where data can be missing or corrupted. Additionally, we lack a formal analysis of the Dynamic Fusion Gate’s stability under perturbation, leaving its potential to overfit to sample-specific noise an open question.

Looking ahead, several tangible directions are planned. To address the challenge of recognizing extreme emotions, we will explore integrating advanced data augmentation techniques or specialized loss functions designed for imbalanced data. To enhance the model’s robustness for real-world deployment, we will focus on extending the DMBT framework to dynamically adapt to missing modalities, ensuring stable performance even with incomplete inputs. Finally, we intend to investigate the generalizability of our core USD and GIFT modules to a broader range of multimodal tasks, with the goal of contributing a more versatile solution to the wider field of multimodal understanding.