LGCDF: Label-Guided Contrastive Disentanglement Fusion of Sensitive Attribute-Free Representations for Fair Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis (MSA) has emerged as a prominent research frontier, enabling a comprehensive understanding of complex human emotions through the synergistic integration of heterogeneous multimodal signals. However, most existing approaches rely on idealized signal distribution assumptions, overlooking the detrimental impact of demographic bias on representation fairness and fusion robustness. This paper proposes a Label-Guided Contrastive Decoupling Fusion (LGCDF) framework that enhances model robustness to demographic bias by learning and fusing multimodal representations invariant to Sensitive Attributes (SAs). Specifically, the proposed LGCDF framework employs gender-sensitive attribute information as modality-level constraints to achieve language-centric cross-modal sentiment alignment, which is accomplished by computing contrastive losses between text–audio and text–visual feature pairs. Moreover, it introduces a SA-guided contrastive decoupling mechanism that decomposes multimodal representations into SA-related and -independent components. The SA-independent components are subsequently fused through a cross-modal attention fusion strategy, thereby facilitating fair sentiment representation and enabling efficient and robust multimodal information fusion. Extensive experimental results demonstrate that the proposed LGCDF framework achieves superior performance in fair representation learning and cross-modal information fusion while maintaining strong robustness under varying gender distribution biases.

1. Introduction

Multimodal Sentiment Analysis (MSA) enhances affective computing by integrating complementary information from textual, acoustic, and visual modalities, generating more robust sentiment representations [1]. This multimodal paradigm has demonstrated significant utility in various scenarios including social media emotion mining [2], video affect recognition [3], and mental health monitoring [4]. Currently, most existing MSA approaches implicitly assume a balanced distribution of Sensitive Attributes (SAs) across multimodal signals. In practical applications, however, MSA tasks often encounter significant imbalances in the distribution of SAs, particularly among different demographic groups. For example, due to socialized gender roles in the real world, women tend to exhibit greater empathy and emotional expressiveness, while men tend to be more assertive and suppress emotional expression [5]. These “emotional expression norms” directly influence observable differences in sentiment intensity, making effective model deployment in real-world applications extremely challenging. Inevitably, this bias compromises the fairness of the learned representations and diminishes the model’s generalization capability. As illustrated in Figure 1 (left), MSA models demonstrate pronounced generalization disparities across demographic subgroups. Specifically, models trained on data dominated by certain demographic attributes in Scenario A achieve substantially higher prediction accuracy when evaluated in Scenario B. Note that the values (0.83, 0.72, 0.53, 0.42) are hypothetical and illustrative, not actual experimental results. They are used to visually demonstrate demographic bias and performance differences in multimodal sentiment analysis. Furthermore, the empirical results presented in Figure 1 (right) reinforce this observation: baseline models such as LMF [6] and MFN [7] exhibit markedly different classification performances under normally distributed (Norm) and skewed (Fair) gender attribute conditions.

Figure 1. Impact of demographic fairness and bias in MSA. (Left): Demographic bias. The abbreviations F, M, A, and C correspond to female, male, adult, and child, respectively. In Scenario A, where female samples are predominant (e.g., female adults: 40%, female children: 20%), the MSA model continues to exhibit demographic bias even under the demographically balanced conditions of Scenario B. Specifically, recognition accuracy remains significantly higher for females (0.83 and 0.72) than for males (0.53 and 0.42). (Right): Performance of the baseline model across varying demographic distributions in the CMU-MOSI dataset. Both LMF [6] and MFN [7] models show considerable performance degradation when applied to imbalanced data. The notations “norm” and “fair” denote the gender attribute distributions in the training, validation, and test sets, corresponding to imbalanced and balanced settings, respectively.

Existing research on demographic bias in MSA has primarily concentrated on two paradigms: bias mitigation through data and representation manipulation and bias control via model fusion and structural optimization [8,9]. Specifically, data-level or representation-level bias mitigation approaches commonly employ techniques such as resampling [10], data augmentation [11], feature disentanglement [12], and adversarial debiasing [13] to intervene either at the data input stage or during feature learning, thereby promoting fair representations across demographic groups. However, unreliable data augmentation strategies and excessive representation manipulation may disrupt the inherent coupling between affective states and demographic attributes, ultimately degrading the model’s fundamental emotion recognition capability [14,15]. In contrast, model fusion- and structure-based bias control methods aim to adjust the relative contributions of different modalities during decision-making to prevent over-reliance on biased dominant modalities. Nevertheless, these approaches typically require strong domain priors and entail substantial computational overhead, which in turn limits the model’s recognition efficiency and scalability [16,17]. In addition, several studies have investigated bias mitigation and fair representation learning from two complementary perspectives: micro-level bias mitigation based on optimization objectives and constraints and macro-level fairness-oriented evaluation [9,18]. The optimization- and constraint-based approaches aim to reduce disparities in prediction performance across demographic groups by introducing regularization terms or adversarial objectives into the training process, thereby penalizing bias-related deviations while maintaining sentiment classification accuracy [19,20]. Fairness-oriented evaluation emphasizes the development of fine-grained fairness assessment frameworks, employing metrics such as Demographic Parity and Equalized Odds to comprehensively measure the effectiveness of bias mitigation in models [21,22,23]. However, both approaches are inherently sensitive to the trade-off between fairness constraints and task performance, leading to unstable model convergence and considerable degradation in recognition accuracy [24].

To address the aforementioned challenges, a Label-Guided Contrastive Disentangled Fusion (LGCDF) framework is proposed, which enhances the model’s robustness to demographic bias by learning and integrating multimodal signal representations that are invariant to sensitive attributes. The LGCDF framework comprises four key stages. First, modality-specific features are projected into a unified representation space through a Multi-Head Self-Attention (MHSA) module. Second, the framework establishes cross-modal sentiment alignment using a Language-centric Bimodal Contrastive (LcBC) module. Third, the framework decomposes the learned representations into SA-related and -independent components through a Label-guided Contrastive Decoupling (LgCD) module. Finally, sentiment prediction is performed by fusing the SA-independent features using a Cross-Modal Attention Fusion (CMAF) module. The main contributions can be summarized as follows:

• This paper presents a novel framework, termed LGCDF, which integrates a label-assisted supervision mechanism and a representation comparison decoupling strategy to achieve fair representation learning for multimodal data and robust cross-modal information fusion under demographic bias.
• A cross-modal semantic alignment module is proposed to establish semantic consistency across modalities by leveraging text-anchored modality pairs and contrastive constraints guided by sensitive attributes.
• A decoupling fusion strategy is proposed to improve the model’s robustness to demographic bias by separating latent representations into components that are dependent on sensitive attributes and those that are independent of them.
• We conducted a comprehensive evaluation of the LGCDF model using the benchmark dataset and its extended variant. The experimental results indicate that the proposed approach consistently achieves high predictive accuracy and exhibits strong robustness across varying demographic bias distributions.

The remainder of this paper is organized as follows. Section 2 surveys related work on demographic fairness and bias mitigation in Multimodal Sentiment Analysis (MSA), contrastive learning-based multimodal feature encoding, and supervised decoupling-based multimodal representation learning. Section 3 elaborates on the proposed methodology, including the problem formulation and overall architecture. Section 4 outlines the experimental setup, encompassing datasets, evaluation metrics, implementation details, and experimental configurations. Section 5 presents and discusses experimental results, focusing on the performance of the LGCDF model across different datasets. Finally, Section 6 concludes this paper, while Section 7 identifies research limitations and suggests potential directions for future work.

2. Related Works

2.1. Demographic Fairness and Bias Mitigation in MSA

Multimodal sentiment analysis models are susceptible to potential sociodemographic biases during representation learning and decision prediction, which may lead to performance disparities across different population groups and exacerbate social discrimination and prejudice in real-world scenarios [25]. Current research on mitigating modality bias in MSA primarily follows two overarching pathways: data-centric processing and model-centric construction [8,9]. These approaches encompass four key strategies: bias elimination in data [26,27], representation learning-based mitigation [28,29], model architecture optimization [30,31], and domain adaptation-based fine-tuning and inference techniques [32,33].

Bias mitigation strategies for multimodal data commonly aim to balance data distributions by adjusting sampling ratios. In contrast, bias mitigation approaches based on fair representation learning focus on alleviating population-level statistical biases through the incorporation of fairness constraints that guide the learning of unbiased and equitable representations. For example, The Decoupling Language Focusing (DLF) framework [12] enhances feature disentanglement by incorporating four geometric metric constraints, which effectively distinguish modality-shared information from modality-specific information. Model architecture optimization methods foster the development of unbiased representations by refining the objective function structure and iteratively updating model parameters [34]. More generally, domain adaptation-based fine-tuning and inference treat groups with different sensitive attributes as distinct domains [33,35]. By employing fine-tuning strategies, these approaches enhance the reasoning ability of multimodal large language models (MLLMs) in fine-grained domains, thereby achieving fair generalization across such domains.

This study integrates the advantages of representation learning and model architecture optimization. By introducing explicit cross-modal alignment constraints and subspace fairness disentanglement constraints, the proposed model extracts gender-aware multimodal sentiment representations, achieving group-level fair representation and bias mitigation with respect to gender attributes.

2.2. Multimodal Feature Encoding Based on Contrastive Learning

The essence of multimodal sentiment analysis lies in guiding models to learn efficient cross-modal representations that capture both the consistency and the complementarity of emotional information across modalities [36]. Traditional multimodal fusion techniques focus on implicit feature fusion, where feature-level integration is achieved through simple concatenation or attention-based fusion mechanisms. With the growing demand for model interpretability in MSA, contrastive learning has emerged as an explicit self-supervised paradigm for sample-level representation learning, providing new perspectives for multimodal sentiment feature encoding [37,38].

The multisource and heterogeneous nature of modal data in MSA has extended contrastive learning into two forms, namely, intra-modal contrast and inter-modal contrast [39]. Intra-modal contrastive methods seek to capture consistent emotional representations within a single modality by identifying core features shared among samples with identical labels. In contrast, inter-modal contrastive methods emphasize the complementarity of emotional representations across modalities [40]. By analyzing how identical emotional labels are manifested in different modalities, these methods facilitate the learning of more robust and discriminative representations. Meanwhile, multimodal feature encoding methods based on contrastive learning have been extensively employed to promote fair representation learning and mitigate bias across modalities [41]. For instance, Gao and Yue et al. [42] proposed a latent bias elimination approach based on contrastive learning. This method mitigates classifier training bias by capturing shared representations among similar samples and enhancing the sparsity of instances that possess the same protected attributes.

This study constructs language-centered bimodal semantic contrastive learning modules based on the improved InfoNCE contrastive loss. By optimizing the bimodal contrastive loss, the model is encouraged to learn modality representations that are semantically consistent with the text modality, thus enabling effective fine-grained cross-modal sample-level semantic alignment.

2.3. Multimodal Representation Learning Based on Supervised Decoupling

Multimodal representations capture not only fundamental emotional cues but also various individual-specific factors, including sensitive attributes. These factors may inadvertently dominate the training process, leading the model to establish spurious associations between such attributes and emotional states [43]. Intuitively, explicitly disentangling relevant factors through external supervision can enhance the robustness of MSA models against sensitive attributes. Currently, supervised disentanglement-based techniques primarily incorporate additional supervisory constraints into the model architecture or objective function to map the multimodal data into independent feature subspaces, achieving the disentanglement of multiple factors [44,45]. In the context of MSA, supervised decoupling-based multimodal representation learning methods are generally categorized into two types according to their decoupling objectives: coarse-grained decoupling and fine-grained decoupling [46,47].

Specifically, Coarse-grained disentanglement primarily targets the feature domain or data style for broad disentanglement. It treats inherent characteristics associated with different datasets, groups, content, or expressive styles as supervisory signals to guide data-level disentanglement [48]. For example, group-level disentanglement can be achieved by using differential characteristics of demographic groups as constraint items [49]. In contrast, fine-grained disentanglement focuses on more specific individual characteristics, such as gender and racial biases, explicitly partitioning multimodal representations into multiple feature subspaces that are task-related (or label-related) and individual-specific [50]. For example, decoupling methods based on multilayer perceptrons [51], variational autoencoders and their derivatives [52,53], and generative adversarial networks and their derivatives have shown superior performance in bias elimination and fine-grained characterization.

Grounded in variational inference theory, this work presents a module designed for fine-grained demographic attribute awareness. This approach harnesses gender attributes as constraints to disentangle representations, subsequently fusing the gender-invariant elements across modalities and curbing the detrimental effects of gender bias.

3. Methodology

3.1. Problem Formulation

Assume the multimodal input is represented as $I_m\in {\mathbb{R}}^{T_m\times d_m}$, where $T_{m\in \{a,v,t\}}$ and $d_{m\in \{a,v,t\}}$ denote the length and dimension of each modalities’ samples, respectively. The goal of multimodal sentiment analysis (MSA) is to predict the sentiment intensity y by extracting, aligning, and fusing sentiment representations across modalities. Formally, this can be expressed as

$$y_i={\mathcal{F}}_f({\mathcal{F}}_a({\mathcal{F}}_e(I_m:\alpha );\theta );\gamma )$$

(1)

where ${\mathcal{F}}_e$, ${\mathcal{F}}_a$ and ${\mathcal{F}}_f$ represent the feature extraction, alignment and fusion modules with parameters $\alpha$, $\theta$ and $\gamma$, respectively. However, demographic distribution shifts across different scenarios result in prediction instability and performance degradation. Considering the demographic bias, the multimodal input $I_m$ is expanded as $S={\{y_i,{\tilde{y}}_i,I_i^m\}}_{i=1}^N$, where N denotes the number of samples and $\{y_i,{\tilde{y}}_i\}$ corresponds to sentiment intensity $y_i$ with gender-sensitive attribute labels ${\tilde{y}}_i$. The MSA task can be updated as

$$y_i={\mathcal{F}}_f({\mathcal{F}}_a({\mathcal{F}}_e(I_m,{\tilde{y}}_i;\alpha );\theta );\gamma ),y\in {\mathbb{R}}^{1\times N},\tilde{y}\in {\mathbb{R}}^{2\times N}$$

(2)

where ${\mathbb{R}}^{2\times N}$ is a binary sensitive attribute label. For example, $\tilde{y}\in \{0,1\}$ represents male ($\tilde{y}=0$) and female ($\tilde{y}=1$) in the gender label, respectively. Although the input conditions differ, the objective functions for both remain identical:

$$\mathcal{L}={\displaystyle \frac{1}{N}}\sum _i^N{(y_i-{\widehat{y}}_i)}^2+\sum _j^k{\lambda }_j{\mathsf{\Phi }}_j$$

(3)

where ${\widehat{y}}_i$ represents the prediction result. $\mathsf{\Phi }$ denotes the constraint and $\lambda$ is the corresponding coefficient. Based on the above procedure, we expand the gender-SA labels and evaluate the performance variations of the proposed model across different datasets to determine the optimal regression parameters ${(\alpha ,\theta ,\gamma )}^{\ast }=\underset{\alpha ,\theta ,\gamma }{argmin}\mathcal{L}$.

3.2. Overall Architecture

This section explains the working principles and implementation details of the core components of the LGCDF model, which include fine-grained feature extraction, cross- modal alignment, and label-guided contrastive disentanglement. The overall architecture and processing pipeline of the proposed LGCDF model are illustrated in Figure 2.

Figure 2. Overall Framework of the LGCDF Model. (A) Architectural overview and data flow. (B) Schematic of LcBC module. (C) Label-guided supervision mechanism in the LgCD module. (D) Composition of contrastive disentanglement loss: disentanglement loss ${\mathcal{L}}_{KL}$, reconstruction loss ${\mathcal{L}}_{rec}$, and representation contrastive loss ${\mathcal{L}}_{CL}^l$.

3.2.1. Unimodal Feature Extraction and Cross-Modal Semantic Alignment

To mitigate the inherent heterogeneity across multi-modal data, we propose a lightweight module that aligns feature dimensions and captures temporal dependencies in intra-modal representations. As illustrated in Figure 3A, the feature mapping module first projects the raw features $I_{m\in \{a,v,t\}}$ from heterogeneous modal spaces into a shared common space, facilitating initial dimensional alignment. Subsequently, the multi-head self-attention $MHS{A}_{m\in \{a,v,t\}}$ is employed to model the aligned features of each modality within a shared feature space and generate modality-specific representations ${\mathcal{M}}_{a,v,t}$ that capture temporal dependencies. Notably, taking the textual modality as an example, the corresponding formulation is given as follows:

$$h_m={\mathcal{W}}_m{I}_m\forall m\in \{t,v,a\}$$

(4)

$${\mathcal{M}}_m=MHS{A}_m\left(h_m\right)\forall m\in \{t,v,a\}$$

(5)

where ${\mathcal{W}}_m$ denotes the mapping matrix. $h_m$ and ${\mathcal{M}}_m$ represent the output of the project module and MHSA module, respectively. To enable subsequent computation, a linear layer with weight parameter ${\mathcal{W}}_l^m$ followed by a ReLU activation is applied, and the output is averaged along the sequence dimension to obtain compressed features.

$${\mathcal{C}}_m=ReLU({\mathcal{W}}_l^m\ast {\mathcal{M}}_m),m\in \{t,v,a\}$$

(6)

Figure 3. Schematic of the LcBC module. (A) Single-modal feature extraction module based on integrated feature mapping and multi-head self attention. (B) Language-centric cross-modal semantic alignment mechanism based on sensitive attribute labels and contrastive learning.

For cross-modal semantic modeling, we first construct text-centered text-audio $I_{ta}$ and text-video $I_{tv}$ modality pairs and compute the similarity matrix ${\mathcal{S}}_P$ for each pair within the batch. Next, the gender-sensitive attribute labels $\tilde{y}$ are incorporated as constraints to generate a corresponding label similarity matrix ${\mathcal{L}}_Y$. To assess cross-modal representation consistency, we compute the similarity between feature similarity matrices and label similarity matrices (see Figure 2B and Figure 3B), ensuring robust feature pairing across sensitive attribute distributions. Specifically, cosine similarity ${\mathcal{F}}_c$ is employed to evaluate the similarity between modality pair features $I_{ta|tv}$, while Gaussian kernel similarity ${\mathcal{F}}_g$ with scaling factor $\sigma$ is used for label vector comparison (Equation (7)). In practice, ${\mathcal{F}}_c$ and ${\mathcal{F}}_g$ are implemented by computing the temperature($\tau$)-scaled cosine similarity across all sample pairs and the Gaussian kernel-based label similarity within the batch samples, respectively (Equation (8)).

$$\left\{\begin{array}{c}{\mathcal{S}}_{P_{tm}}={\mathcal{F}}_c({\mathcal{M}}_t,{\mathcal{M}}_m),m\in \{a,v\}\hfill \\ L_Y={\mathcal{F}}_g(\tilde{y},{\tilde{y}}^T)\hfill \end{array}\right.$$

(7)

$$\left\{\begin{array}{c}{S}_{P_{tm}}=e^{({\displaystyle \frac{{\mathcal{M}}^t}{\left|\right|{\mathcal{M}}^t{\left|\right|}_2}})({\displaystyle \frac{{\mathcal{M}}^m}{\left|\right|{\mathcal{M}}^m{\left|\right|}_2}})/\tau },m\in \{a,v\}\\ L_Y=e^{(-{\displaystyle \frac{{({\tilde{y}}_i-\widetilde{y_j})}^2}{\sigma }})},i,j\in (1,2,\dots ,N)\end{array}\right.$$

(8)

where $S_{P_{tm}}$ represents the cosine similarity of modal feature pairs in the i-th sample. ${\mathcal{L}}_Y$ denotes the Euclidean distance between the label of the i-th sample and that of the j-th sample within a batch. Then the positive sample mask is obtained by comparing the label similarity matrix against a predefined threshold (i.e., $\delta$ = 0.8).

$$P_Y=I[L_{Y_{ij}}>\delta \wedge i\ne j]$$

(9)

where $I[·]$ denotes the indicator function, which returns 1 if the condition inside holds true and 0 otherwise. Based on these matrices, the contrastive loss function defined over modality pairs is formulated as follows:

$${\mathcal{L}}_{CL}^p=-\mathbb{E}\left[log({\displaystyle \frac{{\sum }_m{S}_{P_{tm}}{P}_Y{L}_Y+\epsilon }{{\sum }_m{S}_{P_{tm}}(P_Y{L}_Y+1-P_Y)+\epsilon }})\right]$$

(10)

This loss captures the consistency of multimodal sentiment representations under sensitive attribute conditions. Notably, incorporating similarity between modality pairs and gender-sensitive attribute labels offers a novel perspective for analyzing the correlation between cross-modal sentiment and label distribution, thereby facilitating the disentanglement of sensitive attribute-relevant and -irrelevant features.

3.2.2. Label-Guided Contrastive Disentanglement Fusion

The label-guided contrastive disentanglement fusion module encodes multimodal inputs ${\mathcal{C}}_m$ into modality-specific latent representations by performing representation decoupling (${\mathcal{C}}_m⟼z_m$) and data reconstruction ($z_m⟼{\widehat{\mathcal{C}}}_m$). This module employs two independent encoder ($f_{enc}$)-decoder ($f_{dec}$) architectures for sensitive attribute-related and -irrelevant decoupling, respectively. Notably, the gender-sensitive attribute label ${\tilde{y}}_m$ is embedded in the decoder to guide generation and reconstruction. As illustrated in Figure 2C and Figure 4, the LGCDF module disentangles each input into two independent latent representations: one capturing features associated with sensitive attributes $z_m^{sa}$ and the other preserving attribute-independent information $z_m^{sf}$.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{z}_m^{sa}=f_{enc}\left({\mathcal{C}}_m\right),{\widehat{\mathcal{C}}}_m^{sa}=f_{dec}(z_m^{sa};{\tilde{y}}_m)\hfill \\ z_m^{sf}=f_{enc}\left({\mathcal{C}}_m\right),{\widehat{\mathcal{C}}}_m^{sf}=f_{dec}\left(z_m^{sf}\right)\hfill \\ {\widehat{\mathcal{C}}}_m={\widehat{\mathcal{C}}}_m^{sa}+{\widehat{\mathcal{C}}}_m^{sf}\hfill \end{array}\right.\end{array}$$

(11)

where $z_m^{sa}\sim q\left(z\right|{\mathcal{C}}_m;{\tilde{y}}_m)$ and $z_m^{sf}\sim q\left(z\right|{\mathcal{C}}_m)$. The detailed implementation of latent representation decoupling for the encoder-decoder architecture with embedded sensitive attribute constraints is given in Equation (12):

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mu }_m^{s\ast },{\sigma }_m^{s\ast }=f_{enc}\left({\mathcal{C}}_m\right)\hfill \\ z_m^{s\ast }={\mu }_m^{s\ast }+ϵ\odot e^{({\sigma }_m^{s\ast }/2)}\hfill \\ {\tilde{y}}_m^e=OneHot\left({\tilde{y}}_m\right)\hfill \\ {\widehat{\mathcal{C}}}_m^{s\ast }=\sum f_{dec}\left([z_m^{s\ast }\oplus {\tilde{y}}_m^e]\right)\hfill \end{array}\right.\end{array}$$

(12)

where $\mu$ and $\sigma$ denote the mean and variance of the encoded latent representation, respectively. $s\ast$ indicates the inclusion of constraints on the SAs and $ϵ$ represents a noise vector sampled from a standard normal distribution (i.e., $ϵ\sim \mathcal{N}(0,1)$), which is independent of both the input data and the model parameters. To enhance the effectiveness of representation decoupling and promote independence between SA-related and -independent latent representations, the proposed module integrates three types of losses: disentanglement loss ${\mathcal{L}}_{KL}$, reconstruction loss ${\mathcal{L}}_{rec}$, and SA-exclusive contrastive loss ${\mathcal{L}}_{CL}^l$.

Figure 4. Schematic illustration of the LgCD module. (A) Calculation process of contrastive decoupling loss under sensitive attribute constraints. (B) Implementation of contrastive decoupling mechanism based on encoder-decoder architecture.

Motivated by existing loss functions, we design disentangled and reconstructed loss terms specifically aimed at mitigating demographic biases:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathcal{L}}_{KL}={\mathcal{D}}_{KL}\left(q_{\varphi }\left(z\right|{\mathcal{C}}_m;{\tilde{y}}_m)\right|p\left(z\right))\hfill \\ {\mathcal{L}}_{rec}=\left|\right|{\mathcal{C}}_m-{\widehat{\mathcal{C}}}_m{\left|\right|}^2\hfill \end{array}\right.\end{array}$$

(13)

where $p\left(z\right)\sim \mathcal{N}(0,I)$. In practice, the disentangled loss can be computed using the mean and variance: ${\mathcal{L}}_{KL}={\displaystyle \frac{1}{2}}\sum [{\left({\mu }_m^{s\ast }\right)}^2+{\left({\sigma }_m^{s\ast }\right)}^2-log{\left({\sigma }_m^{s\ast }\right)}^2-1]$. For clarity, we denote the weighted sum of ${\mathcal{L}}_{KL}$ and ${\mathcal{L}}_{rec}$ as ${\mathcal{L}}_{dis}$. As illustrated in Figure 2D, each loss plays a distinct role: the disentanglement and reconstruction losses preserve semantic independence and information completeness of the latent representations, while contrastive loss further enhances the discriminability and independence of these representations with respect to sensitive attributes. Similar to modality contrastive loss, the disentanglement contrastive loss is calculated as the difference between the dot products of the masked and anti-masked label matrices with the latent similarity matrix. This formulation enforces the clear separation of representations associated with ${\tilde{y}}_m$, reinforcing the disentanglement objective.

$$\left\{\begin{array}{c}{\mathcal{S}}_{P_{dis}}={\mathcal{F}}_c({\mathcal{Z}}_m^{sa},{\mathcal{Z}}_m^{sf}),m\in \{a,v\}\hfill \\ L_Y={\mathcal{F}}_g(\tilde{y},{\tilde{y}}^T)\hfill \\ {\mathcal{L}}_{CL}^l=-\mathbb{E}[log({\displaystyle \frac{{\sum }_m{S}_{P_{dis}}{P}_Y{L}_Y+\epsilon }{{\sum }_m{S}_{P_{dis}}(P_Y{L}_Y+1-P_Y)+\epsilon }})]\hfill \end{array}\right.$$

(14)

where $S_{P_{dis}}=exp(-({\displaystyle \frac{z^{sa}}{\left|\right|z^{sa}{\left|\right|}_2}})({\displaystyle \frac{z^{sf}}{\left|\right|z^{sf}{\left|\right|}_2}})/\tau )$. Notably, this work assumes that the SA-related and -irrelevant representations are statistically independent and mutually exclusive. To mitigate the influence of demographic bias, the SA-independent representations are used as input for multimodal feature fusion. Moreover, a Cross-Modal Attention Fusion (CMAF) module is employed for multimodal data fusion. Specifically, the query, key, and value are derived from the SA-independent representations of different modalities. The resulting output matrix is then fed into a fully connected layer f for sentiment intensity prediction.

$$\left\{\begin{array}{c}{h}_{tav}=CMAF(z_a^{sf},z_v^{sf},z_t^{sf})\hfill \\ \widehat{y}={\mathcal{W}}_o{h}_{tav}+b\hfill \\ {\mathcal{L}}_{task}={\displaystyle \frac{1}{N}}\sum _i^N{(y_i-{\widehat{y}}_i)}^2\hfill \end{array}\right.$$

(15)

where ${\mathcal{W}}_o$ denotes the weight of the f. By integrating the loss functions from all modules, the overall objective function of the LGCDF model is defined as

$$\mathcal{L}={\lambda }_p{\mathcal{L}}_{CL}^p+{\lambda }_l{\mathcal{L}}_{CL}^l+{\lambda }_t{\mathcal{L}}_{task}+{\lambda }_d{\mathcal{L}}_{dis}$$

(16)

where ${\lambda }_p$, ${\lambda }_l$, ${\lambda }_t$ and ${\lambda }_d$ denote the weighting coefficients for each loss component. The objective function is minimized through iterative training to obtain the optimal configuration of model parameters.

4. Experiment

4.1. Benchmark Datasets

The CMU-MOSI [54], CMU-MOSEI [55], and CH-SIMS [56] corpora were employed as benchmark datasets for multimodal sentiment analysis. Accordingly, we performed data re-splitting and label augmentation based on sensitive attributes. CMU-MOSI: This dataset consists of 2199 opinion utterances extracted from 93 YouTube monologues (featuring 89 speakers). Each utterance is annotated with sentiment scores (ranging from −3 to +3) and includes aligned textual, visual, and acoustic features. CMU-MOSEI: An expanded version of CMU-MOSI, this dataset comprises over 23,500 utterances from more than 1000 speakers. In addition to sentiment annotations, it introduces fine-grained emotion categories, including happiness, sadness, anger, surprise, disgust, and fear. CH-SIMS: A Chinese multimodal sentiment analysis dataset comprises 2281 natural conversational utterances from 228 speakers. It provides both unimodal and multimodal sentiment scores (ranging from −3 to +3), as well as modality-specific labels, to facilitate detailed analysis.

Building upon a widely used benchmark dataset and its inherent characteristics in the multimodal sentiment analysis (MSA) domain, we enhanced the original dataset through three key processes: augmentation of sensitive attribute labels, construction of imbalanced data partitions, and tailored feature reconstruction. For label augmentation, all data samples in the benchmark dataset were systematically examined, and additional sample-level demographic annotations were incorporated. These annotations include gender (F: female, M: male), age group (A: adult, C: child), and language (C: Chinese, E: English). All labels were formatted as binary tuples, i.e., 0 or 1.

To systematically evaluate the robustness and stability of our model, we constructed two complementary customized datasets, ${\mathcal{D}}_{\mathsf{\Phi }}$ and ${\mathcal{D}}_{\mathsf{\Gamma }}$. Specifically, ${\mathcal{D}}_{\mathsf{\Phi }}$ was built by re-sampling the original dataset according to different sensitive-attribute ratios while retaining the pre-extracted features and their dimensionality. This setup isolates pure sensitive-attribute distribution shift, allowing us to assess the model’s robustness to changes in the demographic composition of the input data. In contrast, ${\mathcal{D}}_{\mathsf{\Gamma }}$ first re-extracts features from the raw multimodal data using a suite of alternative feature extractors and then re-samples these new features based on sensitive-attribute ratios. By introducing both feature extractor shift and distribution shift, ${\mathcal{D}}_{\mathsf{\Gamma }}$ expands the diversity of multimodal sentiment representations and evaluates the model’s domain generalization stability across different feature types.

To improve the transparency and reproducibility of our label augmentation process, we provide a detailed annotation protocol: The labeling was conducted by two independent annotators following a structured guideline, with a high inter-annotator agreement, and any discrepancies were adjudicated by a third senior annotator. Sensitive attributes including gender, age, and language were derived from the original dataset annotations and manually checked by annotators and were annotated using coarse-grained binary labels to maintain consistency with existing fairness research. We further confirm that all data were used in compliance with ethical guidelines, ensuring the privacy and anonymity of individuals. Furthermore, we performed comprehensive data cleaning to ensure the reliability and reproducibility of our experiments. First, we systematically detected and removed duplicate multimodal samples by computing hash values for both text and image features; any sample with a consistent hash was identified as duplicate and excluded. We then carefully checked for potential overlaps between the source datasets by cross-referencing unique identifiers and content hashes. No significant overlaps were observed, and any minor shared samples were resolved by retaining only a single instance. Additional cleaning steps included verifying the consistency of sentiment labels and sensitive attribute annotations via manual inspection, as well as strictly partitioning the dataset into training, validation, and test sets with no data leakage across splits.

The imbalanced dataset ${\mathcal{D}}_{\mathsf{\Phi }}=({\mathcal{D}}_{\mathsf{\Phi }}^{MOSEI}\cup {\mathcal{D}}_{\mathsf{\Phi }}^{MOSI}\cup {\mathcal{D}}_{\mathsf{\Phi }}^{SIMS})$ is derived from an existing benchmark dataset, with its training, validation and test subsets restructured by stratifying samples according to different proportions of sensitive attributes. It is designed to evaluate model performance across varying demographic subgroup distributions. The construction protocol entails repartitioning the original data by adhering to the sample size of the benchmark dataset and predefined splitting ratios for sensitive attributes, thus ensuring consistent sample distributions across the three subsets. For instance, the subset “F2M8” in ${\mathcal{D}}_{\mathsf{\Phi }}$ denotes a 2:8 female-to-male sample ratio that is applied to the training, validation and test subsets.

To simulate the diversity of real-world data and the distributional heterogeneity across distinct demographic subgroups, a feature reconstruction strategy was employed to construct a customized Dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$. This strategy comprised two key steps: (1) re-extracting modality-specific features with dedicated extractors to enhance feature diversity and (2) randomly sampling samples according to a predefined proportion of sensitive attributes to achieve specific data augmentation. Specifically, acoustic, textual and visual features were extracted from the raw data of the benchmark dataset via Librosa, BERT and OpenFace, respectively. Table 1 details the parameter settings of these feature extractors and presents comprehensive dataset statistics. Subsequently, samples were randomly sampled from the entire dataset following predefined sensitive attribute ratios to ensure compositional balance across different demographic subgroups. For instance, the “F2M8” partition in ${\mathcal{D}}_{\mathsf{\Gamma }}$ denotes a 2:8 female-to-male sampling ratio applied to the full dataset for random sample selection. The sample distributions of the dataset under different ratio settings are illustrated in Table 2.

Table 1. Feature Extraction Parameters for Different Tools.

Tool	Params (Name|Value)
Librosa	sample-rate	16,000	spectral-flatness	TRUE	chroma-cqt	TRUE
	n-fft	512	tonnetz	TRUE	chroma-cens	TRUE
	hop-length	940	mfcc-htk	TRUE	spectral-bandwidth	TRUE
	n-mfcc	13	rms	TRUE	spectral-rolloff	TRUE
	spectral-centroid	TRUE	chroma-stft	TRUE	zero-crossing-rate	TRUE
	spectral-contrast	TRUE	poly-features	TRUE	–	–
Bert	bert-base-uncased		TRUE	bert-base-chinese		TRUE
OpenFace	fps	25	facedetScale	0.25	minFaceSize	1
	numFailedDet	10	minTrack	10	cropScale	0.4

Table 2. Sample distribution of the customized dataset under different sensitivity ratios. Numbers indicate sample sizes. Each subset is split into training, test, and validation sets with a ratio of 6:2:2.

Dataset Name	SA	Sample Size
		F0M10	F1M9	F2M8	F3M7	F4M6	F5M5	F6M4	F7M3	F8M2	F9M1	F10M0
${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$	Female	0	78	157	235	314	393	471	550	628	707	786
	Male	1495	1345	1196	1046	897	747	598	448	299	149	0
${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$	Female	0	100	201	302	403	504	605	706	807	908	1009
	Male	1190	1071	952	833	714	595	476	357	238	119	0
${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$	Female	0	814	1629	2443	3258	4072	4887	5701	6516	7330	8145
	Male	14,694	13,224	11,755	10,285	8816	7347	5877	4408	2938	1469	0
${\mathcal{D}}_{\mathsf{\Gamma }}$	Female	0	993	1988	2981	3976	4970	5963	6958	7951	8946	9939
	Male	17,387	15,648	13,909	12,172	10,433	8694	6954	5215	3478	1739	0

These two datasets provide a comprehensive evaluation: ${\mathcal{D}}_{\mathsf{\Phi }}$ tests stability under demographic distribution changes, while ${\mathcal{D}}_{\mathsf{\Gamma }}$ further examines generalization when the feature representation itself varies. The differences between Datasets ${\mathcal{D}}_{\mathsf{\Phi }}$ and ${\mathcal{D}}_{\mathsf{\Gamma }}$ are summarized as follows: (1) The two datasets differ in feature dimensionality. The feature dimensions of Dataset ${\mathcal{D}}_{\mathsf{\Phi }}$ are consistent with those of the standard benchmark dataset, whereas those of Dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$ are determined by the parameter settings of the corresponding feature extractors. Details are presented in Table 3. (2) The overall sample size of Dataset ${\mathcal{D}}_{\mathsf{\Phi }}$ is fixed, and the distribution of male and female samples remains consistent across the training, validation, and test sets. In contrast, the overall sample size of Dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$ varies, and its samples are randomly distributed among the training, validation, and test sets.

Table 3. Feature dimensions of experimental data. The numbers in parentheses represent respectively in order: training sample size/verification sample size/test sample size, sequence length and number of features per frame.

Dataset	Audio Features	Vision Features	Text Features
${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$	(16326/1871/4659, 500, 74)	(16326/1871/4659, 500, 35)	(16326/1871/4659, 50, 768)
${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$	(1284/229/686, 375, 5)	(1284/229/686, 500, 20)	(1284/229/686, 50, 768)
${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$	(1368/456/457, 400, 33)	(1368/456/457, 55, 709)	(1368/456/457, 39, 768)
${\mathcal{D}}_{\mathsf{\Gamma }}$	(1368/456/457, 1601, 70)	(1368/456/457, 232, 177)	(1368/456/457, 39, 768)

4.2. Evaluation Metrics

We employ four standard evaluation metrics: $ACC@2$ measures the binary sentiment polarity classification accuracy. $ACC@5$ evaluates the model’s performance in predicting sentiment categories on a condensed five-point scale ranging from −2 to +2. $ACC@7$ quantifies the classification accuracy of fine-grained sentiment intensity predictions, discretized into a seven-point scale from −3 to +3. $F1$ is the harmonic mean of precision and recall, $CORR$ (pearson correlation coefficient) measures prediction-ground truth alignment, and $MAE$ (mean absolute error) quantifies sentiment intensity prediction deviations.

4.3. Baseline Models

We conduct extensive comparative experiments against state-of-the-art baseline models across multiple benchmark datasets to evaluate the effectiveness of the proposed model. TFN [57] explicitly models inter-modal interactions by fusing cross-modal features through tensor outer products. LMF [6] reduces computational complexity via low-rank decomposition while maintaining the capacity for multimodal interactions. MFN [7] integrates memory networks to dynamically capture temporal cross-modal dependencies, thereby enhancing long-sequence sentiment modeling. EF-LSTM [58] performs early fusion by concatenating multimodal inputs at the input layer and employs a single LSTM to learn joint temporal dynamics. LF-DNN [58] applies late fusion, combining decisions after independent unimodal feature extraction using deep neural networks. Graph-MFN [55] leverages graph neural networks (GNNs) to model inter-modal relationships and capture complex dependency structures. MULT [59] utilizes a transformer-based architecture to uniformly encode multimodal inputs and model global interactions via self-attention mechanisms. MISA [60] disentangles modality-invariant and modality-specific representations to improve generalization and mitigate redundancy. MFM [61] factorizes modality features into a shared latent space to balance complementarity and diversity. MMIM [62] enhances inter-modal consistency by maximizing mutual information between modality-specific representations. KEBR [63] introduces a knowledge-enhanced, self-supervised MSA framework that generates balanced and generalized feature representations.

4.4. Implementation Details

For the benchmark dataset, we employ the pre-extracted multimodal features provided by the MSA framework. For custom datasets, the MSA-FET [64,65] framework is utilized to extract raw multimodal features, where textual, acoustic, and visual modalities are processed using dedicated feature extractors: BERT for 768-dimensional word embeddings (utilizing bert-base-uncased for English and bert-base-chinese for Chinese texts), Librosa for acoustic features, and OpenFace for facial action unit representations. This standardized extraction pipeline ensures feature consistency across different data sources. To improve the reproducibility and methodological clarity of our work, we summarize the weight values for each loss component across different datasets in Table 4.

Table 4. Weights of Loss Components Across Different Datasets.

Dataset	Parameter				Dataset	Parameter
	${\mathit{\lambda }}_{\mathit{t}}$	${\mathit{\lambda }}_{\mathit{p}}$	${\mathit{\lambda }}_{\mathit{l}}$	${\mathit{\lambda }}_{\mathit{d}}$		${\mathit{\lambda }}_{\mathit{t}}$	${\mathit{\lambda }}_{\mathit{p}}$	${\mathit{\lambda }}_{\mathit{l}}$	${\mathit{\lambda }}_{\mathit{d}}$
${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$	0.47	0.1	0.38	0.05	${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$	0.37	0.22	0.37	0
${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$	0.38	0.15	0.09	0.38	${\mathcal{D}}_{\mathsf{\Gamma }}$	0.24	0.27	0.25	0.2

All experiments were conducted on a computer equipped with an Intel(R) Core(TM) i7-14700K CPU, NVIDIA GeForce RTX 4090 GPU, and 125 GB of RAM. The algorithms were implemented in Python 3.12.4 utilizing PyTorch (version 2.4.1+cu121). The operating system was Ubuntu 20.04 LTS. The total number of parameters in the LGCDF model varied across datasets: 211.8 K for the CH-SIMS dataset, 342.4 K for the CMU-MOSI dataset, and 434.3 K for the CMU-MOSEI dataset. Corresponding training durations on this hardware setup were 0.12 h (CH-SIMS), 0.33 h (CMU-MOSI), and 2.13 h (CMU-MOSEI). To enhance the robustness and reproducibility of our findings, we conducted all experiments using five different random seeds (i.e., 1111, 1112, 1113, 1114, 1115) in strict accordance with the seed settings and training methodology of the MSA-FET framework. The results are presented as the mean performance across all repetitions. To ensure rigorous performance comparison, we evaluate all baseline methods and the proposed LGCDF model under the MMSA framework across three datasets: standard-benchmark, imbalanced dataset ${\mathcal{D}}_{\mathsf{\Phi }}$, and customized dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$. All baselines are initialized with their optimal pre-trained hyperparameters and fine-tuned independently for 100 epochs on each dataset. To mitigate stochastic variations, we conduct multiple independent runs with different random seeds and report the most stable configuration based on reproducibility analysis.

4.5. Experimental Results and Analysis

To verify the effectiveness of the LGCDF model in MSA prediction accuracy, bias elimination, and fair representation, this study conducted a series of comparative experiments on the benchmark dataset, the ${\mathcal{D}}_{\mathsf{\Phi }}$ dataset and the ${\mathcal{D}}_{\mathsf{\Gamma }}$ dataset. Experimental visualization results are presented in Section 4.5.1 (Experimental Results based on the benchmark dataset), Section 4.5.2 (Experimental Results based on the ${\mathcal{D}}_{\mathsf{\Phi }}$ dataset) and Section 4.5.3 (Experimental Results based on ${\mathcal{D}}_{\mathsf{\Gamma }}$ dataset). A comprehensive statistical analysis of these results was conducted to further evaluate the model’s ability to mitigate gender bias and ensure demographic representativeness. Heatmaps and bar charts are used for clear presentation of experimental findings, as elaborated upon in Section 4.5.2 and Section 4.5.3.

4.5.1. Experimental Results Based on Benchmark Dataset

This section reports the different model performances on all benchmark datasets. Notably, the experimental results herein demonstrate the LGCDF model’s effectiveness in MSA tasks and provide a benchmark for further analysis: subsequent experiments will evaluate its high performance across diverse data conditions, specifically examining its capacity to mitigate gender bias and achieve robust representation fusion through comparison with benchmark results.

Table 5 presents the experimental results of all baseline models and the proposed LGCDF model on the CMU-MOSEI, CMU-MOSI, and CH-SIMS benchmark datasets. The LGCDF model achieves SOTA performance in both $ACC@2$ and $F1$, obtaining $ACC@2$ = 83.09% and $F1$ = 83.20% on CMU-MOSEI, $ACC@2$ = 82.01% and $F1$ = 82.03% on CMU-MOSI, and $ACC@2$ = 74.62% and $F1$ = 74.47% on CH-SIMS. Furthermore, the LGCDF model demonstrated optimal performance by achieving the highest $CORR$ = 78.01% on the CMU-MOSI dataset and the lowest $MAE$ = 54.83% on the CMU-MOSEI dataset. On the CH-SIMS and CMU-MOSI datasets, the model output a $CORR$ of 58.34% and an $MAE$ of 76.30%, respectively, ranking second only to the best reported values of 58.42% ($CORR$) and 75.28% ($MAE$).

Table 5. Performance comparison on standard benchmark datasets. Bold values indicate the best performance, underlined values denote the second-best performance, italicized values represent the third-best performance, and dark backgrounds denote the performance of the proposed LGCDF model. Note that in the notation ${\mu }_{{\sigma }^2}$, $\mu$ denotes the mean value and ${\sigma }^2$ denotes the variance.

Method	CH-SIMS				CMU-MOSI				CMU-MOSEI
	ACC@2	F1	MAE	CORR	ACC@2	F1	MAE	CORR	ACC@2	F1	MAE	CORR
TFN	72.821.32	72.151.55	43.911.03	57.701.51	76.381.17	76.441.15	97.673.75	66.421.44	78.191.23	78.651.14	57.10.48	71.880.18
LMF	71.641.16	70.381.73	44.691.08	56.552.08	77.640.78	77.550.68	94.373.50	66.321.43	80.431.09	80.90.94	56.510.37	73.260.54
MFM	61.458.8	51.686.63	51.624.46	42.295.56	77.350.98	77.30.86	95.343.75	65.490.88	80.510.98	80.960.86	57.830.4	72.170.59
MFN	71.681.43	70.421.55	44.291.57	56.860.69	78.100.43	78.000.36	92.130.28	66.990.67	79.241.23	79.831.09	57.100.97	72.200.77
MISA	70.291.42	69.371.30	44.590.54	56.041.28	81.720.79	81.700.80	77.361.56	77.491.56	82.681.00	82.900.82	55.210.87	75.750.60
MULT	72.170.76	71.030.90	44.620.22	58.420.90	78.631.05	78.531.00	90.883.15	69.180.88	81.970.86	82.290.84	55.211.49	74.011.05
MMIM	65.176.23	60.124.76	49.174.92	41.905.31	81.360.69	81.390.79	75.281.61	77.670.59	79.422.10	79.911.83	58.211.42	71.421.41
KEBR	70.352.53	69.373.14	58.844.34	7.936.35	58.833.25	58.264.67	137.663.21	17.015.53	79.834.53	64.755.12	80.832.56	35.696.67
LF-DNN	71.251.42	70.001.88	45.660.53	54.430.80	77.960.92	77.821.08	95.141.62	65.871.43	81.723.43	81.592.82	56.011.13	73.231.23
EF-LSTM	54.272.34	38.182.56	59.213.17	0.245.67	76.973.17	76.912.57	97.381.67	63.593.61	77.002.67	77.713.26	59.944.61	68.353.76
GRAPH-MFN	72.211.34	71.431.28	44.830.66	57.291.11	76.790.63	76.710.56	96.612.89	65.260.89	80.271.69	80.771.51	56.430.38	72.670.48
LGCDF	74.620.60	74.471.18	47.521.35	58.241.17	82.011.10	82.031.06	76.302.08	78.011.35	83.090.89	83.200.70	54.830.89	72.760.51

4.5.2. Experimental Results Based on ${\mathcal{D}}_{\mathsf{\Phi }}$ Dataset

To quantitatively evaluate model performance, we computed the ACC@2, F1, ACC@5, and ACC@7 metrics for our proposed LGCDF model and 11 baseline methods across gender-sensitive attribute subsets (F0M10–F10M0). The experimental results are shown in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. Specifically, Figure 5, Figure 7, Figure 9 and Figure 11 present the performance heatmaps of these models on the gender-sensitive subsets of the ${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$, ${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$, ${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$, and ${\mathcal{D}}_{\mathsf{\Gamma }}$ datasets, respectively. In addition, we statistically analyzed the overall performance distribution of each model, including the minimum, maximum, and variance values for the four evaluation metrics. The corresponding results are illustrated in Figure 6, Figure 8, Figure 10 and Figure 12. Notably, the symbols “F*M*” denote data subsets constructed with different proportional configurations, while A–F correspond to the evaluation metrics: $ACC@2$, $F1$, $ACC@5$, $ACC@7$, $MAE$, and $CORR$, respectively. The above heatmap results are obtained by running each model ten times on every subset under optimal parameter settings and averaging the performance scores. The bar chart results are derived from computing the overall statistical metrics (i.e., minimum, maximum, and variance) of each model across all gender-sensitive attribute subsets.

Figure 5. Performance comparison results on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSEI}$. (A): ACC@2, (B): F1, (C): ACC@5, (D): ACC@7, (E): MAE, and (F): CORR.

Figure 6. Statistical Results Based on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSEI}$. The subscripts MIN, MAX, and VAR represent the minimum, maximum, and variance of the corresponding indicators, respectively. The red dashed line represents the mean and the orange bars represent the statistical components of the LGCDF.

Figure 7. Performance comparison results on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSI}$. (A): ACC@2, (B): F1, (C): ACC@5, (D): ACC@7, (E): MAE, and (F): CORR.

Figure 8. Statistical Results Based on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSI}$. The subscripts MIN, MAX, and VAR represent the minimum, maximum, and variance of the corresponding indicators, respectively. The red dashed line represents the mean and the orange bars represent the statistical components of the LGCDF model.

Figure 9. Performance comparison results on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{SIMS}$. (A): ACC@2, (B): F1, (C): ACC@5, (D): ACC@7, (E): MAE, and (F): CORR.

Figure 10. Statistical Results Based on dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{SIMS}$. The subscripts MIN, MAX, and VAR represent the minimum, maximum, and variance of the corresponding indicators, respectively. The red dashed line represents the mean and the orange bars represent the statistical components of the LGCDF model.

Figure 11. Performance comparison results on dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$. (A): ACC@2, (B): F1, (C): ACC@5, (D): ACC@7, (E): MAE, and (F): CORR.

Figure 12. Statistical Results based on ${\mathcal{D}}_{\mathsf{\Gamma }}$. The subscripts MIN, MAX, and VAR represent the minimum, maximum, and variance of the corresponding indicators, respectively. The red dashed line represents the mean and the orange bars represent the statistical components of the LGCDF model.

The experimental results on imbalanced subsets of the CMU-MOSEI dataset are presented in Figure 5. From the perspectives of model category and data subset, the MMIM model demonstrated the best overall performance on four metrics, achieving $ACC@2$ = 92.6% (F0M10), $ACC@5$ = 78.5% (F0M10), $MAE$ = 27.7 (F0M10), and $CORR$ = 89.9 (F0M10), respectively. The MISA model obtained the highest $F1$ = 93.9 (F4M6), while the MFM model achieved the best performance on $ACC@7$, with $ACC@7$ = 79.8 (F0M10). However, all baselines reached their peak performance on the F0M10 subset, and their performance declined to varying degrees as the proportion of male samples decreased. The performance exhibits significant variation across different subsets (as visually indicated by considerable color variation in the performance heatmap), suggesting that the model is highly sensitive to sensitive attributes.

To further assess the robustness of models to variations in sensitive attributes, we performed a statistical analysis on the performance metrics (i.e., $ACC@2$, $F1$, $ACC@5$, $ACC@7$) of each model across varying proportions of sensitive attributes. The statistical analyses (as shown in Figure 6) indicate that the LGCDF model achieves the highest minimum accuracy across all evaluation metrics, with $ACC@2_{MIN}$ = 77.32%, $F{1}_{MIN}$ = 75.56, $ACC@5_{MIN}$ = 51.26%, and $ACC@7_{MIN}$ = 50.26%. The LGCDF model consistently ranks among the top 3 performances in maximum metrics. Moreover, its variance is markedly lower than that of other high-accuracy models, attaining the minimum variance value $ACC@2_{VAR}$ = 6.16. A comprehensive analysis of the specific metrics (Figure 5) and statistical metrics (Figure 6) demonstrates that the performance metrics of the LGCDF model show no significant numerical fluctuations with variations in demographic distributions (with a minimum variance of 6.16), which indicates that LGCDF achieves superior robustness on the CMU-MOSEI dataset. Moreover, the model yields the highest lower bound of accuracy, meaning that even under extreme data distribution scenarios (e.g., F0M10 or F10M0), LGCDF can maintain a high recognition performance without substantial degradation, thus validating the effectiveness of its distributional perturbation resistance.

The experimental results on the CMU-MOSI dataset are presented in Figure 7. The MMIM model consistently outperformed competing approaches across most dataset partitions, achieving accuracies of 94.3%, 94.4%, 49.7%, and 89.2% on $ACC@2$ (F5M5), $F1$ (F5M5), $MAE$ (F0M10), and $CORR$ (F0M10), respectively. In the multi-class classification task, the LMF model attained 76.1% on $ACC@5$ (F0M10) and 67.6% on $ACC@7$ (F0M10). Notably, the performance of the baseline model fluctuates with variations in the male-to-female ratio, showing an initial decline followed by a subsequent rise. This indirectly indicates that baseline models are highly sensitive to shifts in demographic distributions.

As shown in Figure 8, the performance trend of the proposed LGCDF model on the CMU-MOSI subset is largely consistent with that on the CMU-MOSEI subset: the model retains both high accuracy and strong stability, achieving the optimal $ACC@2_{MIN}$ and $ACC@2_{VAR}$ in the polarity classification task. We further ranked all models by their statistical metrics, with $ACC@2$ taken as a representative example. The top 3 rankings across individual statistical components are as follows: LGCDF > MULT > MMIN for $ACC@2_{MIN}$, MMIN > MISA > LGCDF for $ACC@2_{MAX}$, and LGCDF > MULT > KEBR for $ACC@2_{VAR}$. These ranking results further verify that the LGCDF model exhibits prominent advantages in the lower bound stability and variance controllability of recognition accuracy. Notably, the LGCDF model yields slightly inferior performance in fine-grained classification tasks ($ACC@5$ and $ACC@7$), the reasons for which will be elaborated upon in subsequent analysis.

As illustrated in Figure 9, the experimental results on the CH-SIMS dataset show that the MFN and LMF models achieve the highest performance on $ACC@2$ and $F1$, particularly on the F1M9 and F4M6 subsets. Their best results reach approximately $ACC@2$ = 89.18% and $F1$ = 88.88%, respectively. In addition, the GRAPH-MFN model attains the best performance in multi-class classification, achieving 87.1% on both $ACC@5$ (F0M10) and $ACC@7$ (F0M10). The MULT model also exhibits superior performance on regression metrics, with $MAE$ = 15.6 (F0M10) and $CORR$ = 91.8 (F0M10). Overall, the CH-SIMS results reveal a distinct polarization pattern: taking the F5M5 subset as a boundary, the subsets on the left side of the heatmap consistently outperform those on the right across all evaluation metrics. This finding suggests that the baselines display varying degrees of sensitivity to gender-related (male) feature representations.

Experimental results (as shown in Figure 10) on the CH-SIMS dataset subset indicate that the proposed LGCDF model achieves improvements in both polarity and multi-class classification accuracy. Notably, its overall performance metrics are largely consistent with those of the MULT model, generally ranking within the Top-1 or Top-2 across all evaluation metrics. However, the LGCDF model exhibits substantially lower variance in multi-class performance, particularly in $ACC@7_{VAR}$, where its variance is reduced by 17.32% compared with MULT. Combined with Figure 9 and Figure 10, the comparative models exhibit more pronounced performance fluctuations on the ${\mathcal{D}}_{\mathsf{\Phi }}^{SIMS}$b dataset, with a notable decline in overall prediction accuracy. In contrast, the LGCDF model still maintains high accuracy and robust performance, ranking among the top 3 in both the lower bounds of performance accuracy (i.e., $ACC@2_{MIN}$, $F{1}_{MIN}$, $ACC@5_{MIN}$, and $ACC@7_{MIN}$,) and performance stability metrics (i.e., $ACC@2_{VAR}$, $F{1}_{VAR}$, $ACC@5_{VAR}$, and $ACC@7_{VAR}$).

In summary, across all ${\mathcal{D}}_{\mathsf{\Phi }}$ datasets, the LGCDF model delivers performance free of significant gaps or drastic fluctuations with variations in demographic distributions, demonstrating its simultaneous possession of high accuracy and strong robustness. Furthermore, the LGCDF model exhibits two notable characteristics on the ${\mathcal{D}}_{\mathsf{\Phi }}$ datasets: (1) It achieves slightly inferior performance to comparative models in fine-grained classification tasks. For instance, its $ACC@5_{MIN}$ and $ACC@7_{MIN}$ performance in Figure 8 lags behind that of MULT and MMIM. (2) It shows mild sensitivity to specific domain attributes, with its performance declining progressively across ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSEI}$, ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSI}$ and ${\mathcal{D}}_{\mathsf{\Phi }}^{SIMS}$ (${\mathcal{D}}_{\mathsf{\Phi }}^{MOSEI}$ > ${\mathcal{D}}_{\mathsf{\Phi }}^{MOSI}$ > ${\mathcal{D}}_{\mathsf{\Phi }}^{SIMS}$). The underlying causes are explained as follows: (1) The introduction of sensitive attribute labels as constraint information (e.g., the gender one-hot vectors F = [0, 1] and M = [1, 0]) may induce a mismatch between the learning bias of gender supervision signals and the task characteristics of fine-grained sentiment classification. In other words, the fused feature space may have an implicit conflict with the task objective, ultimately leading to a slight deficiency in the model’s capacity to model fine-grained sentiment semantics. (2) The impact of dataset scale on the model is non-negligible, as the model’s performance across different datasets is positively correlated with dataset size. Additionally, the LGCDF model only adopts a single sensitive attribute (e.g., gender) as constraint information and lacks constrained supervision for other sensitive attributes, which results in minor performance variations on language-sensitive attributes.

4.5.3. Experimental Results Based on ${\mathcal{D}}_{\mathsf{\Gamma }}$ Dataset

As illustrated in Figure 11, the experimental results indicate that the MISA model achieves outstanding performance in $ACC@2$, particularly on the F9M1 subset, where it surpasses the second-best model by 5.5%. Across other evaluation metrics, MISA consistently maintains top-tier performance. Likewise, the proposed LGCDF model ranks among the top 3 in most subsets, demonstrating especially strong results on the F0M10 subset, where its $ACC@2$ reaches 80.5%, outperforming MULT and MISA by 0.5% and 3.5%, respectively. Overall, the aggregated results reveal notable differences in model performance across demographic proportions, with LGCDF, MULT, MISA, and MMIM all demonstrating superior results. In contrast to the performance dynamics observed on the ${\mathcal{D}}_{\mathsf{\Phi }}$ dataset, all models show improved accuracy as the proportion of female samples increases, as reflected by the gradual color gradient from left to right in the heatmap.

Statistical results based on the ${\mathcal{D}}_{\mathsf{\Gamma }}$ dataset are presented in Figure 12. All models exhibited minimal performance fluctuations, with the minimum and maximum of various metrics remaining close to their respective averages. The $ACC@5$ and $ACC@7$ metrics, in particular, demonstrated strong consistency across models. However, substantial differences were observed in the variance metrics. Notably, among the models with all accuracy metrics exceeding the mean (i.e., MULT, MISA, MMIM, and LGCDF), the proposed LGCDF model achieved the lowest variance in prediction accuracy, with $ACC@2$ = 9.10, $ACC@5$ = 11.64, and $ACC@7$ = 12.44, respectively.

It is noteworthy that both the LGCDF and MISA models adopt feature decoupling strategies; however, their performance characteristics differ markedly. Specifically, while the MISA model attains a marginally higher upper bound of predictive accuracy, the LGCDF model demonstrates superior stability, as evidenced by its lower performance variance. This discrepancy may be attributed to the distinct decoupling mechanisms employed by the two models. The MISA framework primarily emphasizes the latent correlations between features and sentiment labels, thereby enhancing its potential accuracy. In contrast, the LGCDF model leverages LgCD-based decoupling to generate sentiment representations that are both sensitive and insensitive to demographic attributes. This dual constraint, jointly imposed by sensitive attributes and sentiment labels, enables the LGCDF model to achieve a favorable balance between predictive accuracy and robustness. Furthermore, the consistent high-accuracy trend of the MULT and LGCDF models on the ${\mathcal{D}}_{\mathsf{\Gamma }}$ dataset suggests that compared with conventional single-modal feature extraction and cross-modal fusion methods, integrating multiple attention mechanisms substantially enhances the model’s ability to capture sentiment dependencies within multimodal high-dimensional data.

4.5.4. Significance Testing and Fairness Analysis Across Gender-Sensitive Subgroups

To validate the robustness of our findings, we conducted comprehensive statistical significance testing to evaluate the performance improvements of the proposed LGCDF model against all 11 baseline methods. Specifically, we performed two-tailed t-tests on the ACC@2 results across all fairness-sensitive partitions (gender-sensitive attribute subsets). Each model was evaluated on 11 gender-sensitive attribute subsets within each of the four datasets, resulting in a total of 44 distinct fairness partitions. To account for variability across runs, we executed 10 independent runs for each model on every subset and then calculated the mean performance for each model on each dataset. The results of this analysis are presented in Table 6, which demonstrates that LGCDF achieves statistically significant improvements ($\alpha$ = 0.05) across most comparisons. Notably, on the ${\mathcal{D}}_{\mathsf{\Gamma }}$ dataset, LGCDF outperforms 8 baseline models with p-values well below 0.05, indicating that the performance gains are highly reliable and not due to random fluctuations. For the ${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$ dataset, significant improvements are consistently observed against strong baselines such as LMF, MFN, KEBR, and EF-LSTM, further validating the effectiveness of our approach in this fairness dimension. While some comparisons on ${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$ and ${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$ yield p-values of around 0.1484 (e.g., against GRAPH-MFN), these marginally significant results, combined with the consistent positive T-values across all partitions, indicate that the observed gains are systematic rather than coincidental.

Table 6. Statistical Significance Testing of Accuracy Performance Improvements: LGCDF Model vs. Baselines on Gender-Sensitive Attribute Subsets. × indicates statistical insignificance, while ✔ indicates statistical significance ($\alpha$ = 0.05, two-tailed t-test).

Baseline	${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{sims}}$			${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{mosi}}$			${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{mosei}}$			${\mathcal{D}}_{\mathsf{\Gamma }}$
	T-Value	p-Value	Significant	T-Value	p-Value	Significant	T-Value	p-Value	Significant	T-Value	p-Value	Significant
TFN	1.0389	0.3177	×	1.3546	0.1907	×	1.0389	0.3177	×	7.894	0	✔
LMF	1.0888	0.2968	×	2.4493	0.0237	✔	1.0888	0.2968	×	8.6342	0	✔
MFN	7.5651	0	✔	2.7151	0.0194	✔	7.5651	0	✔	7.2371	0	✔
MFN	1.3424	0.203	×	1.2355	0.231	×	1.3424	0.203	×	3.7608	0.0012	✔
MISA	1.1389	0.2682	×	0.1139	0.9105	×	1.1389	0.2682	×	−0.7507	0.4616	×
MULT	−0.3502	0.7299	×	0.4919	0.6281	×	−0.3501	0.7299	×	1.9828	0.0613	×
MMIM	1.2079	0.2412	×	−0.6552	0.5198	×	1.2079	0.2412	×	1.2707	0.2184	×
KEBR	52.3565	0	✔	17.8928	0	✔	14.7257	0	✔	6.6665	0	✔
LF-DNN	1.7735	0.1007	×	1.4882	0.1523	×	1.7735	0.1007	×	3.4638	0.0025	✔
EF-LSTM	5.0621	0.0001	✔	12.9586	0	✔	5.0621	0.0001	✔	6.8076	0	✔
GRAPH-MFN	1.5326	0.1484	×	1.4433	0.1644	×	1.5326	0.1484	×	5.282	0	✔

To further validate the efficacy of the LGCDF model in mitigating gender bias, we quantified the demographic parity difference (ΔDP) across four customized multimodal sentiment analysis datasets: ${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$, ${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$, ${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$ and ${\mathcal{D}}_{\mathsf{\Gamma }}$. These ΔDP values represent the absolute performance gap between gender-stratified subgroups (F1M9 to F9M1), corresponding to gender distributions ranging from 10% female/90% male to 90% female/10% male. A threshold of 0.1 was adopted as the criterion for demographic parity: values below this threshold indicate that the model performs consistently across different gender subgroups, thereby satisfying the requirement for demographic balance and eliminating significant bias. As shown in Table 7, the LGCDF model consistently maintains gender-based performance differences below the 0.1 threshold across all datasets and gender splits. This demonstrates that the model achieves robust demographic parity in multimodal sentiment analysis, effectively eliminating gender bias. These results establish LGCDF as a reliable and fair solution for deployment in real-world scenarios where equitable performance across demographic groups is critical.

Table 7. Demographic Parity Differences (ΔDP) of LGCDF on Customized Multimodal Sentiment Analysis Datasets with Varying Gender Distribution.

Dataset Name	F1M9	F2M8	F3M7	F4M6	F5M5	F6M4	F7M3	F8M2	F9M1
${\mathcal{D}}_{\mathsf{\Phi }}^{sims}$	0.052	0.013	0.017	0.037	0.029	0.069	0.089	0.054	0.045
${\mathcal{D}}_{\mathsf{\Phi }}^{mosi}$	0.054	0.063	0.056	0.043	0.076	0.091	0.078	0.069	0.074
${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$	0.078	0.087	0.069	0.092	0.089	0.065	0.074	0.062	0.074
${\mathcal{D}}_{\mathsf{\Gamma }}$	0.072	0.084	0.068	0.074	0.081	0.087	0.054	0.062	0.073

Furthermore, we evaluated the structural complexity of the models by comparing their parameter counts and inference run times. Table 8 summarizes the comparison results, reporting the number of parameters (in millions) and average inference time (in seconds) for each model across the four datasets. As shown in Table 8, LGCDF consistently maintains a highly favorable balance between model size and computational cost. In terms of parameter efficiency, LGCDF has only 3.98 million parameters on ${\mathcal{D}}_{\mathsf{\Gamma }}$, which is significantly fewer than resource-heavy models such as MULT (15.76 M), MISA (113.56 M), and MMIM (115.23 M). Even on the largest dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$, LGCDF uses only 4.34 million parameters, demonstrating its lightweight design. In terms of inference speed, LGCDF also exhibits excellent performance. On dataset ${\mathcal{D}}_{\mathsf{\Gamma }}$, it runs in just 90.56 s, which is notably faster than models like MULT (737.84 s). On dataset ${\mathcal{D}}_{\mathsf{\Phi }}^{mosei}$, while the inference time is 267.71 s, this is still competitive given the dataset’s scale and far more efficient than models such as MULT (1546.35 s) and MMIM (1010.10 s). In summary, LGCDF achieves state-of-the-art performance while maintaining a lightweight architecture with a small number of parameters and fast inference speed.

Table 8. Computational complexity comparison in terms of the number of parameters (in millions) and average inference time (in seconds) of LGCDF and state-of-the-art models on four multimodal datasets.

Model	${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{sims}}$		${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{mosi}}$		${\mathcal{D}}_{\mathsf{\Phi }}^{\mathit{mosei}}$		${\mathcal{D}}_{\mathsf{\Gamma }}$
	Parameters (M)	Time (s)	Parameters (M)	Time (s)	Parameters (M)	Time (s)	Parameters (M)	Time (s)
TFN	35.63	27.03	9.49	20.49	50.43	68.95	70.83	13.68
LMF	0.27	8.16	0.51	15.64	0.51	69.75	12.68	13.53
MFM	3.47	34.25	1.42	195.05	5.99	1557.71	32.77	173.19
MFN	0.61	107.68	2.17	45.51	1.27	540.14	11.12	19.09
EF-LSTM	0.22	27.33	0.88	5.30	1.75	59.00	18.96	4.62
LF-DNN	0.63	10.25	0.59	22.22	0.57	81.10	0.54	11.73
GRAPH-MFN	2.72	88.79	2.11	312.23	0.67	1146.64	0.68	4.62
MULT	18.70	81.84	25.72	156.95	0.98	1546.35	15.76	737.84
MISA	123.93	55.69	110.62	73.78	110.91	216.87	113.56	94.36
MMIM	102.64	86.08	109.73	151.03	109.77	1010.10	115.23	87.95
KEBR	103.49	23.31	103.38	44.35	110.55	189.90	110.04	20.38
LGCDF	2.12	12.72	3.42	82.92	4.34	267.71	3.98	90.56

4.6. Ablation Study

To systematically analyze the contribution of each core component and verify the causal effects behind performance improvements, we conduct a comprehensive ablation study on the benchmark dataset. This study consists of two key parts: (1) a module-wise ablation analysis that quantifies the individual importance of each component in our LGCDF framework; (2) a representation disentanglement analysis that explicitly evaluates how effectively our model separates sensitive attribute information from task-relevant information across different demographic groups. While analyzing unimodal contributions can help improve model interpretability and verify the effectiveness of multimodal fusion, we clarify that our method is designed to process complete multimodal inputs (text, audio, and video) and focus on mitigating sensitive attribute bias for fair sentiment analysis, rather than conducting modality comparison or ablation.

4.6.1. Impact of Different Modules

As illustrated in Table 9, the ablation of any individual module leads to varying degrees of performance degradation across all benchmark datasets, demonstrating that each independent component contributes positively to the overall performance enhancement.

Table 9. Ablation studies of module importance. Bold values indicate the best performance and dark backgrounds denote the complete model.

Methods	CH-SIMS				CMU-MOSI				CMU-MOSEI
	ACC@2	F1	MAE	CORR	ACC@2	F1	MAE	CORR	ACC@2	F1	MAE	CORR
w/o MHSA	54.27	38.18	59.11	2.17	77.09	77.08	98.41	66.17	81.29	81.59	57.63	73.15
w/o LcBC	68.32	64.44	49.38	44.30	78.16	78.12	93.22	68.57	82.70	82.81	56.49	74.28
w/o LgCD	54.27	38.18	59.01	4.96	44.75	27.67	147.67	2.49	62.85	48.51	84.13	74.18
w/o CMAF	65.08	58.34	52.44	33.62	78.31	78.22	96.57	66.60	80.65	81.08	57.53	72.85
LGCDF	74.62	74.47	47.52	54.24	82.01	82.03	76.30	78.01	83.09	83.20	54.83	72.76

Due to the heterogeneity of data modalities, the CH-SIMS dataset is derived from Chinese conversational video clips, whereas CMU-MOSI and CMU-MOSEI originate from English monologue video segments, leading different modules to exhibit varying sensitivities to distinct data types. Experimental results demonstrate that the absence of both MHSA (i.e., w/o MHSA) and LgCD (i.e., w/o LgCD) leads to a performance degradation of approximately 20.35% on the CH-SIMS dataset. When only LgCD (i.e., w/o LgCD) is missing, the performance decline reaches 37.26% and 20.24% on the CMU-MOSI and CMU-MOSEI datasets, respectively. When the LcBC module was removed, the model performance declined across all datasets, with the most pronounced drop observed in the F1 score on the CH-SIMS dataset, where performance decreased by 10.03. Furthermore, both the single-modal feature extraction module (MHSA) and the cross-modal fusion module (CMAF) contributed to performance improvements to varying degrees. Among them, the MHSA module exhibited a more pronounced effect, yielding substantial performance gains across multiple datasets and indirectly validating the importance of independent multimodal feature extraction. The overall dynamic changes in the empirical results indicate that inter-modal feature alignment is essential for dialogue-oriented datasets, whereas demographic-attribute-based representation disentanglement improves discriminative feature learning, effectively reducing biases stemming from skewed demographic representations.

4.6.2. Evaluation of Representation Disentanglement Performance

To further validate the efficacy of demographic-aware representation disentanglement, we conducted an ablation study on the benchmark datasets using identical model architectures and hyperparameters across 50 independent experimental runs. Specifically, we separately fed the LGCDF fusion-classification layer with either sensitive-attribute-associated or sensitive-attribute-independent multimodal latent representations as input. The comparative analysis of output results (illustrated in Figure 13 and Table 10) quantitatively demonstrates the effectiveness of representation disentanglement in bias mitigation.

Figure 13. Evaluation of Representation Disentanglement Performance. SA: sensitive-attribute-associated variables; SF: sensitive-attribute-free variables.

Table 10. Statistical Results for Representation Disentanglement Performance Evaluation. Note that in the notation ${\mu }_{\sigma }$, $\mu$ denotes the mean value and $\sigma$ denotes the standard deviation.

Dataset Name	Attribute Type	ACC@2	F1	MAE	CORR
CH-SIMS	SA-Associated	57.770.50	44.761.01	56.770.42	12.272.04
	SA-Free	74.620.59	74.471.24	47.520.58	54.242.52
CMU-MOSI	SA-Associated	75.741.78	75.762.62	99.262.37	65.193.16
	SA-Free	80.321.70	80.272.52	99.332.30	68.353.19
CMU-MOSEI	SA-Associated	81.291.32	81.591.32	57.631.32	73.151.32
	SA-Free	83.091.32	83.201.32	57.831.32	72.761.32

The experimental results across homogeneous datasets demonstrate that the sensitive-attribute-free (SF) decoupled representations consistently outperform their attribute-coupled counterparts across all datasets and evaluation metrics. For instance, the model employing SF features achieved superior performance in $ACC@2$, exceeding the SA-based models by 16.85% on CH-SIMS, 4.58% on CMU-MOSI, and 1.80% on CMU-MOSEI. This robust performance suggests that SF representations encapsulate more effective and generalizable multimodal affective cues.

Furthermore, the overall performance ranking aligns with their respective dataset scales (MOSEI → MOSI → SIMS), implying that data volume partially governs the efficacy of representation disentanglement. Notably, the $CORR$ based on SF feature is slightly lower than that based on SA feature input on the CMU-MOSEI dataset, i.e., $CORR$ = 72.76 (SF) vs. $CORR$ = 73.15 (SA). This phenomenon can be attributed to inherent limitations in demographic-based disentanglement: while SA representations target specific demographic factors, they may inadvertently capture correlated sensitive features (e.g., linguistic, cultural, or geographic variations). These residual correlations become increasingly impactful with larger datasets, ultimately compromising model performance.

5. Discussion

Experimental results, including comparative and ablation studies, validate that our proposed LGCDF model achieves an efficient balance between high accuracy and low variance in sentiment polarity estimation, demonstrating its excellent ability to accurately predict sentiment polarity and eliminate demographic bias. Particularly noteworthy is the model’s outstanding performance in improving the lower bound of accuracy and maintaining low variance. These results indicate that the LGCDF model can learn basic and robust data representations, which is crucial for model deployment and reliable application in real-world scenarios. Notably, our proposed LGCDF framework is inherently general and, in principle, fully compatible with multiple sensitive attributes simultaneously, including correlated or hidden sensitive attributes. The label-guided contrastive disentanglement mechanism can learn unbiased, sensitive-attribute-free representations regardless of whether one or multiple sensitive attributes are involved. Although we focus on gender for simplicity and controlled evaluation, the proposed approach can be naturally extended to jointly handle multiple sensitive attributes such as age, race, and other demographic factors in real-world multimodal sentiment analysis scenarios.

Furthermore, the overall performance dynamics observed in all comparative experiments reveal two key insights: 1. Experimental results comparing benchmark data and imbalanced distribution data show that the performance of the comparative models exhibits varying degrees of degradation during the F0M10 → F10M0 test. The fundamental reason is that the male-to-female sample ratio in the benchmark data is biased (as can be seen from the total number of male and female samples in F0M10 and F10M0 in the figure), resulting in the pre-trained model having a greater advantage in sentiment analysis of male samples. Therefore, as the gender ratio is adjusted, the model performance gradually shows a decline. This phenomenon is consistent with literature findings [66], indicating that initializing sample distribution diversity helps improve model performance, which is particularly important in vertical domain research involving different demographic groups. 2. Although the ablation experiments demonstrate that decoupling SA-oriented representations enhances the bias mitigation and fair representation of the proposed method, the strength of the correlation between the decoupled representations and the ground-truth labels warrants further investigation. As shown in Figure 13, the CORR value of SF features in the CMU-MOSEI dataset is slightly lower than that of SA features. This phenomenon may be attributed to strong correlations among different sensitive attributes. During the decoupling process, multimodal data may retain latent sensitive attribute biases (hidden SA biases) that are inherently aligned with the distribution of demographic characteristics and sentiment labels. Forcibly decoupling a single attribute could disrupt these implicit dependencies. Therefore, integrating multiple strongly correlated attributes during the decoupling process may more effectively mitigate demographic bias in MSA.

6. Conclusions

Against the growing demand for robust and fair multimodal sentiment analysis, this study focuses on resolving cross-modal semantic inconsistency, multimodal representation bias, and unstable generalization under demographically biased distributions. A novel LGCDF model is proposed to enhance the recognition accuracy and generalization robustness of multimodal sentiment analysis models. Experimental results on multiple datasets show that the proposed LGCDF model outperforms the comparison models in terms of classification robustness and lower bound stability under different population distributions. The theoretical contribution of this study resides in the effective integration of sensitive attribute constraints into multimodal fusion, which provides a novel approach to enhancing model robustness against demographic distribution perturbations. In practical applications, the LGCDF model is applicable to real-world multimodal sentiment analysis scenarios with diverse data distributions, thus improving the stability and reliability of sentiment classification systems.

7. Limitations and Future Work

While the LGCDF framework has demonstrated strong capabilities in bias mitigation, fairness representation, and robust information integration, existing research has primarily focused on gender-related bias and has not yet extended to more complex joint biases across sensitive attributes such as age, race, language, and regional culture.

To enhance both the theoretical foundation and practical applicability of the LGCDF, future work will proceed along three main directions. First, we will broaden the applicability of the LGCDF model to address multidimensional demographic biases, thereby enabling more comprehensive investigations into fairness and bias interactions. Second, we plan to integrate the framework with MLLMs to support richer, context-aware emotional content generation and more fine-grained semantic representations. Finally, we intend to apply the enhanced model to adolescent affective disorder research, aiming to advance early screening and diagnosis across diverse regions, genders, age groups, and cultural contexts.