Hybrid Uncertainty Metrics-Based Privacy-Preserving Alternating Multimodal Representation Learning

Abstract

Multimodal learning enhances model performance by integrating heterogeneous data but is hindered by modality laziness and privacy vulnerabilities. Modality laziness occurs when the model overly relies on a single modality for predictions, underutilizing other modalities and leading to suboptimal performance and poor cross-modal integration. Privacy vulnerabilities arise when sensitive data from individual modalities are exposed during training or inference, risking unauthorized access or attacks, especially in shared model components. In this paper, we propose Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL). Built on Multimodal Learning with Alternating Unimodal Adaptation (MLA), PAMRL alternately optimizes unimodal encoders and a shared representation head to mitigate modality laziness and improve cross-modal consistency. It introduces a hybrid uncertainty metric combining KL divergence and entropy to enhance prediction robustness while applying differential privacy to protect sensitive data in unimodal encoders, preserving the shared head for efficient cross-modal fusion. Extensive experiments on the MVSA and CREMA-D datasets, comparing PAMRL with MLA and other baselines, demonstrate its superior performance, achieving an optimal balance of predictive accuracy, attack resilience, and privacy protection, thus supporting secure, efficient multimodal applications.

1. Introduction

Multimodal learning is a significant research direction in artificial intelligence, enhancing semantic representations and model performance in complex tasks by integrating diverse modalities such as vision, audio, and language [1]. For instance, in sentiment analysis, combining facial expressions, speech intonation, and text enables precise emotion recognition [2]; in autonomous driving, fusing images, LiDAR, and voice commands improves environmental perception and decision-making reliability [3]. Recent advancements in transformer-based architectures, such as vision transformers and multimodal transformers, have become foundational to multimodal learning due to their powerful feature extraction and cross-modal integration capabilities, significantly advancing the field [4]. Baltrušaitis et al. [1] note that multimodal models exhibit strong robustness and adaptability in handling high-dimensional, heterogeneous data, particularly in noisy or incomplete real-world scenarios. Meanwhile, several critical challenges limit their practical adoption. Multimodal learning is hindered by two major issues: modality laziness and privacy leakage. Modality laziness occurs when models over-rely on dominant modalities (e.g., images) while neglecting others (e.g., text or audio), undermining synergy and causing performance degradation in scenarios with missing or noisy modalities [3]. Additionally, models processing sensitive data, such as medical images or voice recordings, are vulnerable to privacy leakage through gradient updates and susceptible to membership inference attacks, posing significant barriers to deployment in privacy-sensitive domains like healthcare and finance [5].

To address the issue of modality laziness and privacy leakage, numerous researchers have proposed various methods. For instance, Zhang et al. [6] introduced the Multimodal Learning with Alternating Unimodal Adaptation (MLA) framework, which mitigates modality laziness by alternately optimizing modality-specific encoders and employs a shared head to capture cross-modal semantic relationships. During inference, MLA utilizes an uncertainty-based dynamic fusion mechanism to enhance integration accuracy [7]. MLA outperforms traditional fusion methods on datasets such as CREMA-D [2] for sentiment analysis and MVSA [8] for image-text classification, particularly in modality-imbalanced scenarios [9]. However, MLA’s entropy-based uncertainty metric solely measures prediction confidence, failing to ensure accuracy or reliability, which may lead to “confidently incorrect” predictions in noisy or modality-inconsistent settings [10,11]. Furthermore, MLA lacks privacy safeguards, as frequent gradient updates render it vulnerable to membership inference attacks, limiting its applicability to sensitive data scenarios [5,12]. These shortcomings underscore the urgent need for a more robust and secure multimodal learning framework.

In this work, we propose the Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL) framework, a sophisticated solution designed to tackle the inherent challenges of modality laziness, suboptimal uncertainty quantification, and privacy vulnerabilities in multimodal learning systems. PAMRL integrates a hybrid uncertainty metric, which leverages the complementary strengths of Kullback–Leibler (KL) divergence and entropy to dynamically evaluate modality quality, thereby enhancing modality fusion, ensuring prediction robustness, and promoting consistent cross-modal semantic alignment across diverse data types such as audio–visual inputs for emotion recognition and text–image pairs for sentiment analysis. Additionally, PAMRL employs differential privacy through an innovative alternating training strategy, utilizing Differential Privacy Stochastic Gradient Descent (DP-SGD) with gradient clipping and calibrated noise injection applied selectively to unimodal encoders, safeguarding sensitive data while maintaining the shared representation head’s efficiency for seamless cross-modal integration. Our objective is to develop a multimodal learning framework that balances high predictive accuracy, robustness, and privacy protection for complex real-world tasks.

The main contributions of the paper are as follows:

• We propose a hybrid uncertainty metric that integrates KL divergence and entropy to comprehensively evaluate modality quality, optimize cross-modal fusion, and enhance robustness in noisy or imbalanced scenarios.
• We incorporate differential privacy into alternating training with gradient clipping and noise injection to effectively safeguard sensitive data and mitigate privacy attack risks.
• We demonstrate through extensive experiments on the MVSA and CREMA-D datasets that PAMRL achieves a robust balance of predictive accuracy, robustness, and privacy protection, offering significant practical value.

In the rest of this paper, we first provide an overview of the background and related work in Section 2. The methodology of the Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL) framework is detailed in Section 3. Subsequently, we describe the experimental setup in Section 4, including the datasets, baseline methods, and implementation details. We present the experimental results and perform comprehensive analyses in Section 4. Finally, we summarize the findings and discuss future research directions in Section 5.

2. Related Work

2.1. Multimodal Learning

Multimodal learning leverages data from diverse sensory modalities to improve model performance and robustness. However, a persistent challenge in existing methods is modality laziness, where models disproportionately rely on dominant modalities, sidelining others and undermining overall effectiveness. To address this issue, researchers have proposed several strategies. Wang et al. [3] investigated training challenges in multimodal classification networks, identifying gradient imbalance as a primary driver of modality laziness. They introduced gradient modulation to equalize contributions across modalities. Building on this, Peng et al. [9] developed dynamic gradient modulation (OGM-GE), which adjusts gradient weights dynamically to foster balanced learning. Other approaches include modality dropout [11], which randomly excludes certain modality inputs during training to encourage reliance on all available modalities, and curriculum learning [12], which incrementally increases the complexity of modality combinations to mitigate the issue.

To further tackle modality laziness, Zhang et al. [7] proposed the Multimodal Learning with Alternating Unimodal Adaptation (MLA) framework. Unlike gradient modulation or modality dropout, MLA alternates optimization between modality-specific encoders and a shared head, effectively balancing modality contributions and achieving superior performance on benchmark datasets. Despite its strengths, MLA faces limitations: its entropy-based uncertainty quantification performs suboptimally during inference, struggling to provide reliable estimates, and it lacks built-in privacy-preserving mechanisms, restricting its use in privacy-sensitive applications such as healthcare or finance. Additionally, methods like OGM-GE require meticulous hyperparameter tuning and may falter in dynamic environments, while modality dropout risks discarding critical information from dominant modalities and curriculum learning, though promising, is complex to design and scale for large-scale multimodal tasks.

2.2. Privacy Protection in Multimodal Learning

With growing privacy concerns, privacy-preserving techniques have gained prominence in machine learning. Differential Privacy (DP) [13] offers a robust solution by injecting noise into the training process to protect individual data privacy. DP-SGD, for instance, implements DP in deep learning through gradient clipping and Gaussian noise. In the context of multimodal learning, Li [5] introduced a multimodal differential privacy framework that applies local differential privacy to fused representations, safeguarding user privacy. Complementary approaches include federated learning, which enables privacy-preserving training on decentralized datasets, and homomorphic encryption, which supports computations on encrypted data. The urgency of such measures is underscored by Shokri [14], who demonstrated the vulnerability of machine learning models to membership inference attacks, highlighting the need for robust privacy protections. Additionally, Panther [15] provides a practical, secure two-party inference solution, while [16] explores horizontal multi-party data publishing with adaptive noise under differential privacy, and [17] enhances truth discovery using noise-aware fusion under local differential privacy.

Recent advances in adaptive differential privacy have provided new insights for multimodal learning. Li et al. [18] proposed a multi-stage adaptive differential privacy algorithm in asynchronous federated learning, which dynamically adjusts the privacy budget to improve the trade-off between privacy and utility (PubMed). Although focused on federated learning, this adaptive mechanism offers inspiration for privacy protection in multimodal learning, particularly when handling heterogeneous data. Similarly, Pan et al. [19] conducted a comprehensive survey on differential privacy in deep learning, emphasizing the role of adaptive parameter tuning in enhancing model utility with potential applications to multimodal learning scenarios (ScienceDirect). These studies highlight the potential of adaptive differential privacy to optimize noise levels dynamically, thereby improving the applicability of multimodal models in privacy-sensitive domains.

However, applying these techniques to multimodal learning poses challenges. Federated learning, while privacy-friendly, struggles to coordinate across heterogeneous data distributions typical of multimodal settings, often leading to slow convergence. Methods like MLA, lacking inherent privacy safeguards, are ill-suited for sensitive domains such as medical or financial applications. Moreover, uniformly applying DP across multimodal models can degrade performance, particularly in critical components like shared heads responsible for integrating cross-modal information. This trade-off between privacy and utility remains a significant hurdle in deploying multimodal systems in real-world, privacy-critical scenarios.

2.3. Uncertainty Quantification

Uncertainty quantification is essential in multimodal learning, particularly for dynamic fusion and model calibration. Kim [10] proposed an uncertainty-aware multimodal learning approach that uses cross-modal random network predictions to adjust fusion weights, ensuring consistency across modality predictions. Xue [20] provided a comprehensive review of uncertainty quantification in deep learning, highlighting the efficacy of Bayesian neural networks, Monte Carlo dropout, and ensemble learning in enhancing model robustness. Specifically, Gal [21] introduced Monte Carlo dropout, which estimates prediction uncertainty through multiple forward passes, while Lakshminarayanan [22] demonstrated that by aggregating predictions from multiple models, ensemble learning improves robustness and uncertainty estimates.

Furthermore, Gao et al. [23] reviewed deep learning-based information fusion techniques for multimodal medical image classification, discussing strategies such as input fusion, intermediate fusion (including single-layer, hierarchical, and attention-based fusion), and output fusion (ScienceDirect). These general fusion strategies provide a foundation for effectively integrating multimodal data and can be enhanced through uncertainty quantification to improve model robustness and accuracy. For instance, attention-based intermediate fusion can dynamically adjust modality weights based on uncertainty metrics, optimizing the fusion process. Although these strategies have not yet coalesced into a unified framework, they offer valuable insights for dynamic fusion in multimodal learning. The PAMRL framework proposed in this paper introduces a hybrid uncertainty metric to address the limitations of entropy-based metrics in MLA, optimizing cross-modal fusion and significantly improving robustness in noisy and imbalanced scenarios.

Despite their strengths, these methods encounter limitations in multimodal contexts. In MLA, the entropy-based uncertainty measure struggles to distinguish between “correct and confident” and “incorrect and confident” predictions, risking overreliance on low-quality modalities. While Monte Carlo dropout and ensemble learning are effective, their high computational cost makes them impractical for large-scale multimodal models or datasets. Furthermore, these techniques were primarily designed for unimodal settings, and their adaptation to ensure cross-modal uncertainty consistency remains underexplored. Addressing these gaps is critical for improving the reliability and scalability of uncertainty quantification in multimodal learning.

3. Materials and Methods

3.1. Overview of Our Proposed PAMRL

In this study, we propose an enhanced multimodal learning framework, Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL), to address the limitations of prior work, specifically modality laziness, suboptimal uncertainty quantification, and the absence of privacy-preserving mechanisms. Our approach builds upon the Multimodal Learning with Alternating Unimodal Adaptation (MLA) framework by incorporating two key components: a hybrid uncertainty-based dynamic fusion mechanism and a privacy-preserving alternating training strategy. The former integrates entropy and Kullback–Leibler (KL) divergence for a more robust assessment of modality reliability, enhancing cross-modal consistency in noisy or imbalanced scenarios. The latter applies Differential Privacy Stochastic Gradient Descent (DP-SGD) selectively to unimodal encoders to protect sensitive data without compromising cross-modal integration. To evaluate PAMRL’s privacy protection, we define a threat model analyzing risks against membership inference attacks (MIAs) for models handling sensitive datasets like CREMA-D and MVSA. In this model, an adversary seeks to infer training data membership (e.g., personal identities in CREMA-D or social media records in MVSA) using model outputs (fused and unimodal predictions, limited gradient information from the shared head) but cannot access training data or internal parameters. This threat model underscores the necessity of PAMRL’s privacy mechanisms to mitigate risks like MIAs while maintaining performance. The overall architecture of PAMRL is illustrated in Figure 1. This section provides a high-level overview of our approach, outlining its core components and operational principles, with detailed implementation specifics deferred to the subsequent subsection.

3.2. Dynamic Fusion Mechanism

The dynamic fusion mechanism adjusts modality fusion weights based on a hybrid uncertainty metric $u_m$. The computation proceeds as follows:

(1)

Entropy Calculation ${\mathit{e}}_{\mathit{m}}$.

The uncertainty of predictions for modality $m$ is measured using Shannon entropy:

$$e_m=-\sum _{c=1}^C{p}_{m,c}log{p}_{m,c}$$

(1)

where $p_{m,c}$ represents the predicted probability of modality $m$ for class $c$, and $C$ is the total number of classes.

(2)

KL Divergence Calculation $\mathit{K}{\mathit{L}}_{\mathit{m}}$.

In our multimodal learning framework, we utilize the Kullback–Leibler (KL) divergence to measure the difference between the predictive probability distributions of two modalities. Given two distributions, $p_1$ and $p_2$, corresponding to the predictions of the first and second modalities, respectively, the KL divergence is defined as follows:

$$KL\left({\mathit{p}}_i\Vert {\mathit{q}}_i\right)=\sum _{c=1}^C{p}_{i,c}\left(log{p}_{i,c}-\mathit{log}{q}_{i,c}\right)$$

(2)

where $p_{1,c}$ and $q_{2,c}$ are the probabilities assigned to class $c$ by $p_1$ and $p_2$, respectively.

(3)

Normalized Entropy.

Normalized entropy adjusts the original entropy value to a range of [0, 1], allowing us to measure uncertainty consistently, regardless of the number of classes involved. The formula is as follows:

$$Normalized{e}_m={\displaystyle \frac{e_m-{\mu }_e}{{\sigma }_e+ϵ}}$$

(3)

$${\mu }_e={\displaystyle \frac{1}{B}}\sum _{j=1}^B{e_m}_j$$

(4)

$${\sigma }_e=\sqrt{{\displaystyle \frac{1}{B}}\sum _{j=1}^B{\left({e_m}_j-{\mu }_e\right)}^2}$$

(5)

where $ϵ$ is a predefined hyperparameter used to prevent the denominator from being zero, and B is batch size.

(4)

Normalized KL Divergence.

KL divergence (Kullback–Leibler divergence) measures how much one probability distribution differs from another, but it does not have a fixed upper bound. Normalizing it helps us interpret the difference on a consistent scale. The formula is as follows:

$$Normalized{KL}_m={\displaystyle \frac{KL\left({\mathit{p}}_i\Vert {\mathit{q}}_i\right)-{\mu }_{KL}}{{\sigma }_{KL}+ϵ}}$$

(6)

$${\mu }_{KL}={\displaystyle \frac{1}{B}}\sum _{i=1}^{B}KL\left({\mathit{p}}_i\Vert {\mathit{q}}_i\right)$$

(7)

$${\sigma }_{KL}=\sqrt{{\displaystyle \frac{1}{B}}\sum _{i=1}^B{\left(KL\left({\mathit{p}}_i\Vert {\mathit{q}}_i\right)-{\mu }_{KL}\right)}^2}$$

(8)

where $ϵ$ is a predefined hyperparameter used to prevent the denominator from being zero, and B is batch size.

(5)

Hybrid Uncertainty Metric ${\mathit{u}}_{\mathit{m}}$.

The hybrid uncertainty combines entropy and KL divergence with a weighting factor $\alpha$:

$$u_m=\beta e_m+(1-\beta )D_{KL,m}$$

(9)

Here, $\beta \in \left[\mathrm{0,1}\right]$ is a hyperparameter set to 0.5 by default, balancing the contributions of entropy and divergence.

(6)

Fusion Weight ${\mathit{w}}_{\mathit{m}}$.

Fusion weights are computed using a SoftMax-like normalization to favor modalities with lower uncertainty:

$$\begin{array}{c}{w}_{m_i}={\displaystyle \frac{\mathit{exp}\left(-\left(\mathrm{Normalized}{e}_{m_i}+\mathrm{Normalized}K{L}_{m_i}\right)\right)}{\sum _m\mathit{exp}\left(-\left(\mathrm{Normalized}{e}_{m_i}+\mathrm{Normalized}K{L}_{m_i}\right)\right)}}\\ \end{array}$$

(10)

$$w_{m_j}=1-w_{m_i}$$

(11)

where $m$ is the total number of modalities.

(7)

Final Prediction ${\widehat{\mathit{y}}}_{\mathrm{final}}$.

The final prediction is a weighted sum of individual modality predictions:

$${\widehat{y}}_{\mathrm{final}}=\sum _{m=1}^M{w}_m·f_m\left(x_m\right)$$

(12)

where $f_m\left(x_m\right)$ denotes the prediction output of modality $m$ given input $x_m$.

3.3. Privacy-Preserving Training

To ensure data privacy, we apply Differential Privacy Stochastic Gradient Descent (DP-SGD) to the unimodal encoders while preserving standard SGD for the shared head. The implementation details are as follows:

(1)

Gradient Clipping.

The L2 norm of each sample’s gradient is clipped to a predefined threshold $C$:

$$g^{\prime }=g/\mathrm{m}\mathrm{a}\mathrm{x}\left(1,{\displaystyle \frac{\Vert g{\Vert }_2}{C}}\right)$$

(13)

where $g$ is the original gradient and $g^{\prime }$ is the clipped gradient.

(2)

Noise Injection.

Gaussian noise is added to the clipped gradient to satisfy differential privacy:

$$\tilde{g}=g^{\prime }+\mathcal{N}\left(0,{\sigma }^2{C}^{2}I\right)$$

(14)

where $\sigma$ is the noise standard deviation, and $I$ is the identity matrix.

(3)

Differential Privacy.

DP-SGD [15] ensures data privacy in PAMRL by injecting noise into the gradients of unimodal encoders during training:

$$Pr\left[M\left(D_1\right)\in S\right]\le e^{ϵ}·Pr\left[M\left(D_2\right)\in S\right]+\delta$$

(15)

where $D_1$ and $D_2$ are datasets differing by one sample, $S$ is any output set, and $ϵ$ denotes the privacy parameter. The privacy parameters, including the clipping threshold $C$, are determined through an analysis of the gradient L2 norms in the datasets, while the noise standard deviation $\sigma$ is computed using the RDP [13] accounting method based on dataset sizes, batch size, and training epochs, with $\delta ={10}^{-5}$ set by referencing the classic DP-SGD parameter [13].

4. Results and Discussion

We conducted experiments to validate the effectiveness and privacy protection of the proposed method. The experiments consist of two main parts: performance testing of the proposed method and evaluation of privacy protection.

4.1. Experimental

4.1.1. Datasets

(1)

CREMA-D: CREMA-D [2] is an audio–visual dataset for emotion recognition research, containing 7442 video clips of 2–3 s duration, featuring facial and vocal emotional expressions from 91 actors. The emotional states are categorized into six classes: happy, sad, angry, fearful, disgusted, and neutral. The dataset is randomly split into a training set of 6698 samples and a test set of 744 samples at a ratio of 9:1.

(2)

MVSA: MVSA-Single [9] is a standardized image-text dataset for multimodal sentiment analysis, comprising 5129 Twitter-derived social media pairs (4511 validated samples after curation). Each entry includes an image and its corresponding textual tweet, annotated via a single-annotator protocol with ternary sentiment labels (positive/negative/neutral). Through a systematic curation process involving eliminating inter-modal polarity conflicts and prioritizing the adoption of non-neutral labels when either modality exhibits neutrality, the dataset ensures label consistency while preserving authentic cross-modal interaction patterns.

4.1.2. Experiment Settings

In the experiments for the CREMA-D dataset, we employed a ResNet-18-based [10] network as the encoder. In the MVSA dataset, we utilized M3AE [24], a large pre-trained multimodal masked auto-encoder, as the encoder. We trained all models using a mini-batch size of 64 and the SGD optimizer [25] with a momentum of 0.9 and weight decay of 1 × 10⁻⁴. The initial learning rate is 1 × 10⁻³, with a decay step size of 70 and a decay factor of 0.1. For Differential Privacy Stochastic Gradient Descent (DPSGD), we set the clipping threshold to C = 1.0 and C = 2.0, noise standard deviation to σ = 2.40, corresponding to privacy budgets of ϵ = 1 and ϵ = 2, δ = 10⁻⁵ [13,15,24,26]. The parameter α is set within the range [0, 1] for different datasets. All experiments are implemented in Python 3.8 using PyTorch 1.9.1 on an NVIDIA A800 80 GB PCIe GPU.

4.1.3. Evaluation Metrics

For the multimodal multi-class classification task, we used the following metrics to evaluate the overall classification performance of the model:

(1)

Accuracy (ACC): Accuracy measures the proportion of correctly classified samples out of the total samples. It is calculated as

$$\mathit{ACC}={\displaystyle \frac{\sum _{i=1}^{N}1\left({\widehat{y}}_i=y_i\right)}{N}}$$

where $N$ is the total number of samples, ${\widehat{y}}_i$ is the predicted class for sample $i$, ${\mathrm{y}}_{\mathrm{i}}$ is the true class, and $1\left(·\right)$ is the indicator function (1 if ${\widehat{\mathrm{y}}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}$, 0 otherwise).

(2)

Precision: Precision measures the proportion of samples predicted as positive that are actually positive, reflecting the model’s prediction precision. It is calculated as

$$\mathit{Precision}=\sum _{c=1}^C{w}_c·{\displaystyle \frac{{\mathit{TP}}_c}{{\mathit{TP}}_c+{\mathit{FP}}_c}}$$

where $C$ is the number of classes, $w_c$ is the proportion of samples in class $c$, ${\mathit{TP}}_c$ is the number of true positives for class $c$, and ${\mathit{FP}}_c$ is the number of false positives for class $c$.

(3)

Recall: Recall measures the proportion of actual positive samples correctly identified by the model, reflecting the model’s coverage ability. The weighted average recall is calculated as

$$\mathit{Recall}=\sum _{c=1}^C{w}_c·{\displaystyle \frac{{\mathit{TP}}_c}{{\mathit{TP}}_c+{\mathit{FN}}_c}}$$

where ${\mathit{FN}}_c$ is the number of false negatives for class $c$.

(4)

F1 score: The F1 score is the harmonic mean of precision and recall, providing a balanced measure of model performance. The weighted average F1 score is calculated as

$$F1=\sum _{c=1}^C{w}_c·{\displaystyle \frac{2·{\mathit{Precision}}_c·{\mathit{Recall}}_c}{{\mathit{Precision}}_c+{\mathit{Recall}}_c}}$$

where ${\mathit{Precision}}_c$ and ${\mathit{Recall}}_c$ are the precision and recall for class $c$, respectively.

(5)

Expected Calibration Error (ECE): ECE measures the difference between the model’s predicted confidence and its actual accuracy, reflecting the model’s calibration quality. ECE divides the predicted confidences into $M$ bins (typically $M=10$) and computes the weighted average of the absolute difference between accuracy and confidence in each bin:

$$\mathit{ECE}=\sum _{m=1}^M{\displaystyle \frac{\left|B_m\right|}{N}}·\left|\mathit{acc}\left(B_m\right)-\mathit{conf}\left(B_m\right)\right|$$

where $B_m$ is the set of samples in the $m$-th confidence bin, $\left|B_m\right|$ is the number of samples in bin $B_m$, $N$ is the total number of samples, $\mathrm{acc}\left(B_m\right)$ is the accuracy in bin $B_m$, and $\mathrm{conf}\left(B_m\right)$ is the average confidence in bin $B_m$. A lower ECE indicates better calibration.

(6)

Confidence-Stratified Accuracy Low (ConfAcc_Low): ConfAcc_Low measures the model’s accuracy in the low-confidence interval (confidence range [0, 0.5]), reflecting the reliability of low-confidence predictions. It is calculated as

$$\mathit{ConfAcc}\_\mathit{Low}={\displaystyle \frac{{\sum }_{i\in S_{\mathit{low}}}\left({\widehat{y}}_i=y_i\right)}{\left|S_{\mathit{low}}\right|}}$$

where $S_{\mathit{low}}$ is the set of samples with confidence in [0, 0.5], ${\widehat{y}}_i$ is the predicted class for sample $i$, $y_i$ is the true class, $1\left(·\right)$ is the indicator function (1 if ${\widehat{\mathrm{y}}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}$, 0 otherwise), and $\left|S_{\mathit{low}}\right|$ is the number of samples in $S_{\mathit{low}}$.

These metrics collectively provide a comprehensive evaluation of the model’s performance. ACC offers a straightforward measure of overall classification performance; precision, recall, and F1 score complement ACC by addressing class imbalance; ECE and ConfAcc_Low provide insights into the model’s confidence calibration and reliability, enhancing the analysis of prediction robustness.

4.2. Main Result for the PAMRL

To evaluate the performance of the PAMRL model, we compared two configurations, the MLA model [6] and the PAMRL model, to ensure user-level differential privacy. To achieve robust privacy guarantees while maintaining model utility, we systematically selected DPSGD hyperparameters—gradient clipping threshold ($C$) and noise standard deviation ($\delta$)—based on empirical gradient norm analysis and established DP-SGD practices [13,24]. Performance was assessed using accuracy, recall, precision, F1 score, Expected Calibration Error (ECE), and Low Confidence Accuracy (ConfAcc_Low) over 100 training epochs. The results, visualized in Figure 2, Figure 3, Figure 4 and Figure 5, demonstrate the PAMRL model’s efficacy in privacy preservation through DPSGD. The best results of all models on two datasets are reported in

Table 1. The overhead of differential privacy in PAMRL vs. MLA’s best results on CREMA-D and MVSA datasets.

Dataset	Model	Accuracy	Precision	F1 Score	Recall	ECE	ConfAcc_Low
CREMAD	MLA [6]	72.21%	77.11%	69.45%	75.41%	5.54%	46.75%
	PAMRL (ϵ = 1)	62.58%	68.54%	53.35%	67.78%	18.34%	30.12%
	PAMRL (ϵ = 2)	65.69%	74.65%	62.24%	69.56%	14.35%	32.34%
MVSA	MLA [6]	62.23%	72.65%	71.67%	76.23%	4.76%	51.54%
	PAMRL (ϵ = 1)	55.56%	61.48%	54.42%	61.94%	9.45%	40.16%
	PAMRL (ϵ = 2)	58.45%	63.45%	58.65%	64.34%	6.43%	45.87%

.

Accuracy-Related Metrics: Table 1 reveals that for accuracy-related metrics (accuracy, precision, F1-score, recall), MLA outperforms PAMRL on both CREMA-D (accuracy: 72.21%, precision: 77.11%, recall: 75.41%, F1-score: 69.45%) and MVSA (accuracy: 62.23%, precision: 72.65%, recall: 76.23%, F1-score: 71.67%), while PAMRL with ϵ = 1 (CREMA-D: accuracy 62.58%, F1-score 53.35%; MVSA: accuracy 55.56%, F1-score 54.42%) and ϵ = 2 (CREMA-D: accuracy 65.69%, F1-score 62.24%; MVSA: accuracy 58.45%, F1-score 58.65%) shows a performance overhead due to differential privacy. This indicates that DP introduces a trade-off, reducing classification performance to prioritize privacy, with ϵ = 2 offering better utility than ϵ = 1. This reflects PAMRL’s conservative performance approach, sacrificing some accuracy to ensure privacy protection while maintaining practical utility for real-world tasks. This performance supports PAMRL’s framework by demonstrating its ability to protect sensitive data via DP while still achieving usable accuracy, making it suitable for privacy-sensitive multimodal applications like sentiment analysis.

Confidence-Related Metrics: For confidence-related metrics (ECE, ConfAcc_Low), Table 1 shows that MLA has lower ECE (CREMA-D: 5.54%, MVSA: 4.76%) but limited ConfAcc_Low (CREMA-D: 46.75%, MVSA: 51.54%), while PAMRL with ϵ = 1 (CREMA-D: ECE 18.34%, ConfAcc_Low 30.12%; MVSA: ECE 9.45%, ConfAcc_Low 40.16%) and ϵ = 2 (CREMA-D: ECE 14.35%, ConfAcc_Low 32.34%; MVSA: ECE 6.43%, ConfAcc_Low 45.87%) exhibits higher ECE but improved ConfAcc_Low with ϵ = 2, indicating better calibration and low-confidence prediction reliability under DP constraints. This suggests that PAMRL trades off calibration tightness for enhanced prediction stability, especially with a relaxed privacy budget (ϵ = 2). This reflects PAMRL’s robustness in handling uncertain predictions, a critical aspect of reliable multimodal learning in noisy scenarios. Such performance supports PAMRL’s framework by validating its mixed uncertainty metrics, which address modality laziness by ensuring balanced cross-modal fusion, enhancing model reliability for applications like emotion recognition in privacy-sensitive domains.

4.3. Comparison Between PAMRL and MLA Models Against Membership Inference Attacks

To evaluate the privacy protection capability of the model, we employ the membership inference attack (MIA) [26] as the benchmark attack method. Following the shadow model paradigm, the attacker observes the loss function values and confidence distributions of the target model for input data, utilizing a threshold-based classifier to distinguish between member and non-member samples. This method aims to infer whether a specific data sample belongs to the model’s training set, thereby exposing sensitive information such as personal identities or medical records through statistical divergences in model outputs. The ideal attack accuracy of 50% corresponds to random guessing, indicating no discernible privacy leakage, while higher values signify vulnerabilities requiring mitigation via differential privacy mechanisms or gradient obfuscation techniques.

AUC for Member Inference Attack (${MIA}_{AUC}$): An attacker analyzes a model’s output (e.g., predictive confidence) to infer whether a particular sample belongs to the model’s training set ${MIA}_{AUC}$ measures the model’s ability to discriminate between the confidence of samples in the training set (members) and samples in the test set (non-members) by calculating the Area Under the Curve (AUC) of the ROC curve, calculated as

$${\mathrm{M}\mathrm{I}\mathrm{A}}_{\mathrm{A}\mathrm{U}\mathrm{C}}={\int }_0^1\mathrm{T}\mathrm{P}\mathrm{R}\left(\mathrm{F}\mathrm{P}\mathrm{R}\right)d\mathrm{F}\mathrm{P}\mathrm{R}$$

where $\mathrm{T}\mathrm{P}\mathrm{R}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$ is the true positive rate, and $\mathrm{F}\mathrm{P}\mathrm{R}={\displaystyle \frac{\mathrm{F}\mathrm{P}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}}}$ is the false positive rate.

The experimental results show that on both CREMA-D and MVSA datasets, No-DP models (MLA) exhibit a high MIA AUC (0.73 for CREMA-D, 0.75 for MVSA), increasing steadily from 0.50 to 0.51 and stabilizing, while DP models (PAMRL) consistently suppress MIA AUC with lower values throughout training, as illustrated in Figure 6. This indicates that No-DP models overfit to training data, increasing vulnerability to membership inference attacks, whereas DP models effectively mitigate these privacy risks, with stronger privacy settings (e.g., higher noise levels) further reducing AUC. This reflects PAMRL’s robust privacy protection, as its DP mechanism obscures training data patterns, enhancing security for sensitive multimodal data like audio–visual (CREMA-D) and text–image (MVSA) datasets. Such performance supports the PAMRL framework by validating its design goal of safeguarding sensitive data through differential privacy), ensuring its applicability in privacy-sensitive multimodal applications such as emotion recognition and sentiment analysis, where mitigating privacy risks is critical.

4.4. Ablation Study on Hybrid Uncertainty Metrics

The effectiveness of the Hybrid Uncertainty Fusion method, which integrates entropy and KL divergence, was evaluated against entropy-based and KL divergence-based fusion methods on the CREMA-D and MVSA datasets. Performance was assessed using Expected Calibration Error (ECE), F1 score, precision, recall, and Confidence-Stratified Accuracy Low (ConfAcc_Low) over 100 epochs. The results, summarized below and visualized in Figure 7 and Figure 8, demonstrate the Hybrid method’s superior performance in classification accuracy, model calibration, and reliability of low-confidence predictions.

Accuracy-Related Metrics: The experimental results show that for accuracy-related metrics (Accuracy, Precision, F1-Score, Recall), the Hybrid Uncertainty Fusion method consistently outperforms entropy-based and KL-based methods on both CREMA-D (F1-score: 0.676, precision: 0.739, recall: 0.682 vs. entropy-based: 0.594, 0.660, 0.614) and MVSA datasets (F1-score: 0.533, precision: 0.581, recall: 0.581 vs. entropy-based: 0.490, 0.550, 0.537), with accuracy improvements of 9.45% (average) and 6.88% (peak) over entropy-based methods across both datasets. This indicates that by integrating entropy and KL divergence, the hybrid method achieves superior classification performance through balanced modality contributions. These findings reflect the hybrid method’s robust predictive accuracy, effectively handling complex audio–visual and text–image interactions with enhanced cross-modal synergy. This performance supports the PAMRL framework’s hybrid uncertainty metrics by demonstrating their ability to mitigate modality laziness, ensuring consistent modality integration for reliable multimodal learning in applications like emotion recognition and sentiment analysis.

Confidence-Related Metrics: For confidence-related metrics (ECE, ConfAcc_Low), reveal that the Hybrid Uncertainty Fusion method achieves lower ECE values (0.107 on CREMA-D, 0.111 on MVSA vs. Entropy-based: 0.214, 0.169) and higher ConfAcc_Low (0.374 on CREMA-D, 0.465 on MVSA vs. Entropy-based: 0.292, 0.389), with MVSA showing stability through lower standard deviations (e.g., ECE: 0.036). This indicates that the hybrid method provides superior calibration and reliability in low-confidence predictions, aligning predicted confidence with actual accuracy. These findings reflect the Hybrid method’s enhanced robustness in uncertain scenarios, ensuring stable predictions across diverse multimodal data. This performance supports the PAMRL framework’s hybrid uncertainty metric by validating their role in addressing modality laziness through improved cross-modal consistency, making PAMRL reliable for privacy-sensitive multimodal applications where prediction stability is critical.

5. Conclusions

In this paper, we propose Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL), a framework addressing modality laziness and privacy concerns through two key innovations: a mixed uncertainty metric combining Kullback–Leibler divergence (KL divergence) and entropy to dynamically adjust modality weights during fusion, enhancing prediction accuracy and model robustness, and a privacy-preserving training strategy utilizing Differential Privacy Stochastic Gradient Descent (DP-SGD) applied selectively to unimodal encoders, ensuring sensitive data protection while preserving flexibility in the shared fusion head for efficient cross-modal integration. By effectively mitigating modality laziness and strengthening security, PAMRL balances efficiency and data protection, achieving a slight 2.5% accuracy drop due to differential privacy while reducing membership inference attack (MIA) success by 18%, making it well-suited for practical multimodal learning applications. This framework effectively mitigates modality laziness and strengthens security for practical multimodal learning applications.

This work has some limitations that warrant further attention in future work. We summarized as follows. Despite the achievements of the PAMRL framework, challenges persist, including a privacy-performance trade-off where high privacy noise reduces CREMA-D’s accuracy by approximately 10% under elevated noise levels, manual tuning of privacy parameters like gradient clipping and noise standard deviation that lacks adaptability, and computational overhead from differential privacy that limits scalability and efficiency on large datasets. Future research will focus on developing efficient differential privacy techniques, such as adaptive noise injection, to minimize performance loss while preserving privacy and designing adaptive parameter adjustment algorithms based on task or data characteristics to enhance the framework’s versatility and usability.