UCMR-Net: Text-Anchored Residual Fusion with Adaptive Residual Weighting for Multimodal Sentiment Intensity Prediction

Abstract

Robust multimodal sentiment analysis requires models that can integrate linguistic, acoustic, and visual cues while avoiding over-reliance on noisy nonverbal signals. This study proposes UCMR-Net, a text-anchored residual fusion framework for continuous multimodal sentiment intensity prediction. The model uses contextual textual representations as the primary semantic backbone and introduces acoustic and visual representations as adaptive residual correction signals. Instead of treating the learned positive residual coefficient as a direct estimate of aleatoric or epistemic uncertainty, the proposed framework interprets it as a residual reliability score for regulating nonverbal contribution. Under a unified five-seed evaluation protocol on CMU-MOSI, UCMR-Net achieves MAE = 0.699 ± 0.009, RMSE = 0.997 ± 0.011, Pearson correlation = 0.802 ± 0.006, Acc-2 = 85.82 ± 0.62%, and F1 = 85.60 ± 0.62% (mean ± SD). Controlled ablation results show that text anchoring is the dominant contributor to regression improvement, while residual fusion, adaptive residual weighting, counterfactual distillation, and multi-task supervision provide secondary stabilization effects. Unified missing-modality evaluation further indicates that performance remains relatively stable when audio or visual streams are removed, but degrades substantially when text is unavailable, confirming that the model is text-anchored rather than modality-symmetric. Calibration analysis shows that raw UCMR-Net only modestly improves calibration-related metrics, whereas post-hoc temperature scaling reduces ECE from 0.121 to 0.064 and NLL from 2.337 to 2.286. Additional CMU-MOSEI validation suggests that the proposed fusion strategy generalizes beyond CMU-MOSI, although cross-dataset transfer remains more challenging. Overall, UCMR-Net provides an effective and empirically validated framework for complete-modality sentiment intensity prediction, with moderate robustness under nonverbal missing or corrupted conditions.

1. Introduction

Multimodal sentiment analysis (MSA) aims to infer human affective states by jointly exploiting linguistic, acoustic, and visual cues. Compared with text-only sentiment analysis, MSA is better aligned with real-world human communication, where sentiment is expressed not only through words but also through prosody, facial movements, pauses, gaze, and other nonverbal signals [1]. This makes MSA particularly relevant to opinion video understanding, human–computer interaction, online education, intelligent customer service, affective dialogue systems, and social media analytics.

From a machine learning perspective, MSA is a representative multimodal learning problem because it requires models to handle heterogeneous input spaces, asynchronous temporal structures, modality imbalance, and cross-modal complementarity [2]. Text usually carries explicit semantic information, while audio and visual streams provide paralinguistic and behavioral evidence that may strengthen, weaken, or even contradict the textual signal. Therefore, the central technical issue in MSA is not merely whether multiple modalities are available, but how to integrate them in a way that preserves task-relevant information while suppressing noisy or redundant cross-modal interactions.

Recent surveys show that MSA research has evolved from simple early/late fusion to tensor fusion, hierarchical fusion, attention-based fusion, pretrained-language-model-based fusion, self-supervised multimodal learning, and missing-modality-aware learning [3]. In parallel, affective computing studies have demonstrated that multimodal signals can improve affect recognition, but the gain depends strongly on the quality of feature extraction, temporal alignment, fusion strategy, and the robustness of the model under noisy or incomplete inputs [4]. These observations indicate that high-performing MSA models should not only pursue stronger feature interaction but also provide stable predictions under practical deployment constraints.

A major limitation of many existing MSA methods is the assumption that all modalities are available and reliable during both training and testing. In realistic scenarios, however, one or more modalities may be missing because of speech recognition failure, camera occlusion, low-quality audio, privacy restrictions, sensor malfunction, transmission loss, or platform-side data filtering [5]. Missing-modality learning has therefore become an important research direction, because a model trained only on complete multimodal samples can suffer a substantial performance drop when partial modalities are absent at inference time [6]. Recent tag-assisted and representation-alignment methods further show that explicit missing-pattern modeling and complete-to-incomplete representation alignment can improve robustness under uncertain modality availability [7,8].

Another practical challenge is sample-dependent modality reliability. Multimodal sentiment labels are inherently subjective, and textual, acoustic, and visual streams may provide inconsistent or even contradictory evidence. For example, a speaker may express positive words with a negative tone, or a visually neutral expression may accompany emotionally loaded language. Under such cases, deterministic fusion may over-amplify unreliable nonverbal cues and produce unstable predictions. Therefore, this study treats nonverbal reliability primarily as a modality-contribution regulation problem rather than as a fully probabilistic uncertainty-estimation problem. Acoustic and visual signals are introduced as residual correction cues, and their contribution is controlled by a learned residual reliability score. This design allows nonverbal modalities to refine the text-based sentiment estimate when they provide useful affective evidence, while reducing their influence when they are noisy, missing, or potentially misleading.

To address these issues, this study proposes UCMR-Net, a residual fusion framework for multimodal sentiment intensity prediction that uses text as the primary semantic anchor and incorporates adaptive residual weighting and masked-modality distillation as auxiliary design components. The model follows a text-anchored design: textual representations are treated as the primary semantic backbone, while acoustic and visual modalities are used as residual correction signals. This design is motivated by the observation that sentiment polarity and intensity in opinion videos are often explicitly expressed through language, whereas audio and visual modalities provide complementary but sometimes noisy contextual cues.

Specifically, UCMR-Net integrates four design principles. First, it uses contextual text representations as the semantic anchor for multimodal prediction. Second, it encodes acoustic and visual temporal streams as nonverbal residual evidence. Third, it performs text-anchored residual fusion to refine the text-based sentiment estimate without uniformly amplifying all nonverbal information. Fourth, it uses adaptive residual weighting, counterfactual masked-modality distillation, and auxiliary polarity/intensity supervision to stabilize prediction behavior.

The main contributions of UCMR-Net and the corresponding empirical evaluation framework are summarized as follows. First, we develop UCMR-Net as an integrative and application-oriented text-anchored residual fusion framework that treats language as the primary semantic backbone and models acoustic and visual streams as adaptive residual modifiers for continuous sentiment intensity estimation. Second, we formulate the learned residual coefficient as a residual reliability score for modality-contribution regulation and evaluate its empirical value through calibration analysis, residual reliability–error correlation, and modality corruption tests. Third, we provide a controlled experimental protocol including five random seeds, unified baseline evaluation, statistical significance testing, and ablations that separately isolate text anchoring, residual fusion, adaptive residual weighting, counterfactual distillation, and multi-task supervision. Fourth, we evaluate incomplete-input behavior under a unified missing-modality protocol and random missing-rate protocol, and further assess generalization using CMU-MOSEI external validation. These analyses position UCMR-Net as a text-dominant residual fusion model with strong complete-modality performance and moderate robustness when nonverbal modalities are missing or corrupted. The methodological contribution is therefore primarily integrative and application-oriented: UCMR-Net combines text anchoring, residual fusion, adaptive residual contribution regulation, masked-modality distillation, and auxiliary supervision within a unified framework, rather than claiming a fundamentally new fusion primitive.

The rest of this paper is organized as follows. Section 2 reviews related studies on multimodal sentiment fusion, missing-modality robust learning, and adaptive modality-contribution learning. Section 3 introduces the proposed UCMR-Net framework, including the problem definition, model architecture, module design, and training objective. Section 4 reports the experimental setup and results. Section 5 discusses the empirical findings, limitations, and future research directions.

2. Related Work

2.1. Multimodal Representation and Fusion for Sentiment Analysis

Early MSA studies mainly focused on learning effective interactions among text, audio, and visual streams. The Memory Fusion Network (MFN) explicitly modeled view-specific and cross-view interactions through a recurrent memory mechanism, showing that multimodal temporal dependencies could be captured more effectively than by simple concatenation [9]. The Recurrent Multistage Fusion Network (RMFN) further decomposed multimodal fusion into multiple recurrent stages, allowing each stage to focus on different subsets of cross-modal interactions [10]. These studies established that multimodal fusion should be treated as a dynamic process rather than a single static combination step.

Subsequent research emphasized fine-grained nonverbal influence on language representations. RAVEN dynamically shifted word representations according to acoustic and visual behaviors, indicating that nonverbal signals can change the contextual meaning of words during sentiment inference [11]. Factorized multimodal representation learning decomposed multimodal representations into discriminative shared factors and modality-specific generative factors, thereby improving robustness to noisy or incomplete modalities [12]. These methods are closely related to the motivation of the present study because they suggest that not all modality information should be fused uniformly; instead, fusion should distinguish dominant semantic evidence from complementary or modality-specific information.

More recent work has explored contrastive and representation-alignment strategies. Hybrid contrastive learning was introduced to learn cross-modal representations by jointly exploiting intra-modal and inter-modal consistency for MSA [13]. Modality-invariant representation learning further investigates how to reduce modality gaps and obtain robust affective representations under heterogeneous inputs [14]. These studies improve representation compactness and cross-modal consistency, but they do not fully solve the practical problem of how to maintain stable prediction when specific modalities are missing or unreliable during deployment.

Recent multimodal affective learning has further moved toward adaptive fusion, cross-domain transfer, task-aware supervision, and modern sequence modeling. Knowledge-guided dynamic fusion has been used to adjust modality contributions according to sample-dependent dominant cues [15], while adaptively balanced domain adaptation has been explored to address modality-specific distribution shifts in multimodal emotion recognition [16]. Recent language-focused disentanglement methods further reduce cross-modal redundancy while preserving complementary information around a strong linguistic representation [17], and state-space models have been introduced to capture long-range intra-modal dynamics and cross-modal interactions [18]. Multi-loss fusion studies also show that encoder selection, fusion design, contextual information, and auxiliary objectives can materially affect multimodal sentiment performance [19]. Together, these studies indicate that robust multimodal affective learning increasingly depends on dynamic modality contribution, cross-domain generalization, modern temporal modeling, and task-aware supervision. In parallel, recent missing-modality and distillation-based MSA methods such as LNLN, MissModal, TFR-Net, UMDF, and CorrKD emphasize that modality absence should not be treated merely as feature removal, because the missing stream may contain decisive or contradictory affective evidence. UCMR-Net is positioned within this broader line of work by adopting a conservative text-anchored strategy: it does not assume symmetric reliability across modalities, but instead uses nonverbal streams as adaptive residual evidence around a strong linguistic backbone.

Different from conventional interaction-heavy fusion models, UCMR-Net adopts a text-anchored residual fusion strategy. Rather than treating all modalities as equally reliable, it uses the textual stream as the main sentiment backbone and introduces acoustic and visual streams as residual correction signals. This design reduces the risk of over-fusion and provides a more interpretable mechanism for continuous sentiment intensity estimation.

2.2. Missing-Modality Robust Multimodal Sentiment Analysis

Missing-modality learning has become increasingly important in MSA because real-world multimodal systems rarely guarantee the complete availability of all data streams. The Missing Modality Imagination Network (MMIN) addressed this issue by learning to imagine missing modality representations from available modalities, enabling a unified model to handle different missing patterns [20]. This reconstruction-oriented line of work is valuable because it directly targets incomplete inputs, but it may introduce representation gaps when generated features are not sufficiently aligned with true modality distributions.

Inconsistency-aware missing-modality learning further showed that modality absence can change the overall sentiment semantics rather than merely remove partial input information [21]. This line of work introduced the notion of key missing modalities and emphasized that missing-modality MSA should consider whether the absent stream contains decisive sentiment evidence. The Tag-Assisted Transformer Encoder (TATE) further used tag encoding to represent both single- and multiple-modality missing cases and employed transformer-based reconstruction to handle incomplete modalities [22]. These studies demonstrate that explicitly indicating missing patterns can improve model awareness of modality availability. However, tag-based reconstruction still depends on the quality of latent feature recovery and may be sensitive to distribution shifts.

The Language-dominated Noise-resistant Learning Network (LNLN) treated language as the dominant sentiment-bearing modality and improved robustness under incomplete data by correcting the dominant modality representation and then integrating auxiliary acoustic and visual information [23]. Modality Translation-based MSA (MTMSA) translated visual and acoustic modalities into the textual modality, further emphasizing the advantage of language as a dominant sentiment carrier under uncertain missing-modality conditions [24]. Multimodal Prompt Learning with Missing Modalities (MPLMM) introduced generative prompts, missing-signal prompts, and missing-type prompts to compensate for unavailable modalities in a parameter-efficient way [25]. Similar modality-completion-based MSA further attempted to complete missing modality information by exploiting similar samples and modality-level relationships [26].

These studies provide important evidence that missing-modality MSA requires both modality availability awareness and semantic consistency across complete and incomplete conditions. UCMR-Net follows this direction but takes a more conservative and application-oriented route. Instead of relying solely on explicit feature generation, it combines modality masking, text-anchored residual correction, and counterfactual distillation. In this way, the model is encouraged to maintain prediction consistency between complete-modality and missing-modality conditions while avoiding excessive dependence on generated nonverbal features.

2.3. Adaptive Modality Contribution and Auxiliary-Supervised Prediction

Adaptive modality-contribution learning is relevant to MSA because multimodal signals are often ambiguous, noisy, and inconsistent. Prior uncertainty and ensemble learning studies show that predictive confidence can be useful under noisy conditions [27,28], but a positive gating value should not be treated as a formal uncertainty estimate unless it is empirically validated. In UCMR-Net, the learned residual score is therefore interpreted conservatively as a sample-dependent controller for nonverbal residual contribution.

Another related direction is evidential learning, which aims to quantify uncertainty by learning evidence distributions rather than only point predictions [29]. Although such approaches are useful for understanding confidence and ambiguity, UCMR-Net does not claim to perform full probabilistic uncertainty estimation. Instead, calibration metrics, reliability-error analysis, and modality corruption tests are used to evaluate whether the learned residual score has empirical value for regulating acoustic and visual contributions.

Auxiliary supervision is also important for continuous sentiment prediction. Recent multi-layer feature fusion and multi-task learning studies have shown that combining fusion design with auxiliary task constraints can improve multimodal sentiment representation learning [30]. Regression alone may optimize the average error but fail to sufficiently separate polarity boundaries or ordered sentiment levels. Therefore, UCMR-Net combines continuous regression with binary polarity prediction and ordinal sentiment-intensity supervision. This multi-task design encourages the shared representation to preserve both fine-grained intensity information and polarity-level discriminability. In the five-seed experimental protocol used in this study, the contribution of auxiliary supervision is evaluated through controlled ablations that isolate the effects of the binary polarity and ordinal sentiment-intensity heads.

Overall, the related literature suggests three unresolved issues. First, many strong fusion models assume complete modalities and may be vulnerable to missing inputs. Second, missing-modality methods often focus on reconstruction or representation alignment but may overlook the dominance of textual semantics in opinion-level sentiment prediction. Third, learned modality weights require empirical validation before being described as uncertainty estimates. UCMR-Net addresses these issues through text-anchored residual fusion, counterfactual masked-modality distillation, adaptive residual weighting, and unified experimental evaluation.

3. Methodology

3.1. Overall Research Workflow

This study develops an application-oriented multimodal sentiment prediction framework for text-anchored sentiment intensity estimation. The overall workflow consists of six stages: dataset preparation, multimodal preprocessing, modality-specific encoding, adaptive text-anchored residual fusion, multi-task optimization, and unified experimental evaluation. To avoid inconsistent evaluation across complete and incomplete inputs, all complete-modality, missing-modality, ablation, calibration, and baseline experiments are conducted under the same data split, preprocessing pipeline, feature normalization parameters, checkpoint selection rule, and metric script. For stochastic reliability, the main model, all rerun baselines, and controlled ablation configurations are evaluated using five random seeds. Repeated-run results are reported as mean ± standard deviation unless otherwise specified. Figure 1 illustrates the overall workflow from data preparation to unified result analysis.

3.2. Problem Definition

Given a multimodal utterance-level sample i, the input consists of three modality streams: textual input $x_i^T$, acoustic input $x_i^A$, and visual input $x_i^V$. The target variable is the continuous sentiment intensity score $y_i\in [-3,3]$, where negative values indicate negative sentiment, positive values indicate positive sentiment, and values around zero indicate neutral sentiment. The primary task is regression-based sentiment intensity prediction, while binary polarity prediction and ordinal sentiment-level prediction are introduced as auxiliary tasks.

The general prediction function is defined as:

$${\widehat{y}}_i=f_{\theta }\left(x_i^T,x_i^A,x_i^V,m_i\right)$$

(1)

where $m_i\in {\{0,1\}}^3$ denotes the modality availability mask, and $\theta$ represents all trainable parameters. The binary polarity label is defined as:

$$c_i=\mathbb{I}\left(y_i>0\right)$$

(2)

For ordinal auxiliary learning, the continuous sentiment score is discretized into ordered sentiment-intensity classes:

$$o_i=g_{ord}\left(y_i\right),o_i\in \{1,2,3,4,5\}$$

(3)

The model is optimized to minimize regression error, preserve rank-consistent sentiment intensity, and improve polarity-level discriminability.

3.3. Proposed Model: UCMR-Net

The proposed model is named UCMR-Net. It is designed as a text-anchored residual fusion network with adaptive residual weighting and masked-modality distillation for continuous multimodal sentiment intensity prediction. UCMR-Net is built on a text-anchored assumption: in opinion video sentiment analysis, textual content usually carries the densest semantic evidence, whereas acoustic and visual streams provide complementary paralinguistic corrections. The model first obtains a contextual text representation from BERT, then encodes acoustic and visual temporal streams using BiLSTM-based encoders, and finally performs residual correction through a learned residual reliability score. The reliability score is used to regulate nonverbal residual contribution and is not interpreted as a direct estimate of aleatoric uncertainty, epistemic uncertainty, or predictive uncertainty. The model jointly outputs a continuous sentiment score, a binary polarity label, and an ordinal sentiment-intensity label. The overall architecture of UCMR-Net, including modality encoders, masked-modality compensation, residual fusion, and prediction heads, is shown in Figure 2.

3.4. Text Encoder

The textual stream is encoded using BERT-base-uncased, which provides contextualized bidirectional token representations. For an input token sequence $x_i^T=\{w_1,w_2,\dots ,w_L\}$, BERT generates hidden states:

$$H_i^T=BERT\left(x_i^T\right)\in {\mathbb{R}}^{L\times 768}$$

(4)

The final text representation combines the [CLS] token representation, attention pooling, and mean pooling, followed by a projection layer to match the hidden dimension used by the acoustic and visual encoders:

$$h_i^T=W_T\left[H_{i,CLS}^T;AttnPool\left(H_i^T\right);MeanPool\left(H_i^T\right)\right]+b_T$$

(5)

where the projected text vector is used as the semantic anchor for subsequent residual fusion. BERT is used because pretrained bidirectional contextual representations have been shown to provide strong language understanding capacity for downstream NLP tasks [31]. Figure 3 presents the BERT-based text encoding module and shows how token-level contextual representations are aggregated into the final textual feature vector.

3.5. Acoustic and Visual Temporal Encoders

The acoustic and visual streams are represented as time-series feature sequences. In the implemented feature pipeline, the acoustic feature dimension is 74 and the visual feature dimension is 35. Each stream is normalized and then fed into a bidirectional LSTM encoder. LSTM is used because it is suitable for modeling temporal dependencies in sequential data [32].

For the acoustic stream:

$$Z_i^A={BiLSTM}_A\left(x_i^A\right)$$

(6)

$$h_i^A=W_A·MeanPool\left(Z_i^A\right)+b_A$$

(7)

For the visual stream:

$$Z_i^V={BiLSTM}_V\left(x_i^V\right)$$

(8)

$$h_i^V=W_V·MeanPool\left(Z_i^V\right)+b_V$$

(9)

where $h_i^A,h_i^V\in {\mathbb{R}}^{128}$. This design ensures that the acoustic and visual representations are projected into the same latent dimensionality as the textual representation before fusion, as illustrated in Figure 4.

3.6. Text-Anchored Residual Fusion

The central component of UCMR-Net is the text-anchored residual fusion module. Rather than treating the three modalities as equally reliable at all times, UCMR-Net uses text as the main semantic anchor and uses acoustic and visual representations as residual correction signals. This is particularly suitable for CMU-MOSI, where sentiment labels are assigned to opinion segments and textual statements often provide the most explicit sentiment evidence.

The acoustic and visual representations are first projected into residual feature space:

$${\tilde{h}}_i^A=W_A^r{h}_i^A+b_A^r$$

(10)

$${\tilde{h}}_i^V=W_V^r{h}_i^V+b_V^r$$

(11)

The multimodal residual signal is then computed as:

$$r_i^{AV}=F_r\left(\left[h_i^T;{\tilde{h}}_i^A;{\tilde{h}}_i^V\right]\right)$$

(12)

where the residual estimator is implemented by fully connected layers, layer normalization, GELU activation, and dropout. The fused representation is obtained by adding a reliability-weighted nonverbal residual correction to the textual representation:

$$h_i^F=h_i^T+{\omega }_i^r{r}_i^{AV}$$

(13)

Here, the residual weight is produced by a positive scalar transformation and normalized across residual branches. In this formulation, the scalar is interpreted as a learned residual reliability controller rather than a probabilistic uncertainty estimate. A lower residual reliability score increases the contribution of the corresponding residual branch, whereas a higher reliability-risk score suppresses potentially noisy nonverbal correction. The empirical validity of this score is examined through its correlation with the absolute prediction error and through modality corruption tests, rather than assumed from the Softplus transformation alone.

$${\omega }_i^r={\displaystyle \frac{\exp \left(-u_i^r\right)}{\exp \left(-u_i^T\right)+exp\left(-u_i^r\right)}}$$

(14)

This fusion design makes the model different from conventional early fusion and tensor fusion models. It explicitly prioritizes the textual semantic backbone while allowing nonverbal modalities to refine sentiment intensity estimation. The internal structure of the text-anchored residual fusion module is presented in Figure 5.

3.7. Missing-Modality Compensation and Counterfactual Distillation

The framework can be evaluated under incomplete multimodal conditions. When one or more modalities are unavailable, the missing stream is replaced by a controlled placeholder representation, and the available modalities are used to generate a prediction. At the framework level, this operation is implemented as mask-based missing-modality compensation. In UCMR-Net, the missing pathway uses unavailable-stream masking, zero-filled modality placeholders, and counterfactual distillation rather than an independent generative decoder.

Let ${\widehat{y}}_i^{full}$ denote the prediction under complete modalities and ${\widehat{y}}_i^{miss}$ denote the prediction under a missing-modality mask. The counterfactual distillation term is defined as:

$$L_{distill}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\Vert {\widehat{y}}_i^{miss}-{\widehat{y}}_i^{full}{\Vert }_2^2$$

(15)

The counterfactual distillation term encourages masked-input predictions to remain close to complete-input predictions under controlled missing-modality masks. This objective is intended to reduce prediction drift caused by modality unavailability, but it does not guarantee correctness when the removed modality contains decisive or contradictory sentiment evidence. Therefore, the effectiveness of masked-modality learning is evaluated empirically under a unified missing-modality protocol covering complete input, missing text, missing audio, missing visual, unimodal-only input, and random missing rates. Figure 6 summarizes the missing-modality compensation and counterfactual distillation mechanism used to constrain prediction consistency under masked-modality conditions.

3.8. Adaptive Residual Reliability and Calibration Analysis

UCMR-Net uses a learned residual reliability score to regulate the contribution of acoustic and visual residual corrections. This score is not treated as a formal probabilistic uncertainty estimate. Instead, it is analyzed as a sample-dependent gating signal whose empirical behavior can be assessed through error correlation, calibration metrics, and modality corruption tests. For a modality representation, the residual reliability score is computed as:

$$u_i^m=Softplus\left(W_2\tanh \left(W_1{h}_i^m+b_1\right)+b_2\right)+ϵ$$

(16)

where a small constant is used for numerical stability. Calibration is evaluated using ECE, MCE, Brier score, and NLL. Because raw neural confidence is often miscalibrated, post-hoc temperature scaling is additionally applied to the binary polarity head. The evaluation reports both raw and temperature-scaled calibration results, as well as the correlation between residual reliability-risk scores and absolute regression errors. Figure 7 shows how adaptive residual reliability is connected with residual weighting and multi-task prediction heads.

3.9. Multi-Task Prediction Heads

The fused representation $h_i^F$ is passed to three task-specific heads. The regression head predicts continuous sentiment intensity:

$${\widehat{y}}_i=W_y{h}_i^F+b_y$$

(17)

The binary head predicts sentiment polarity:

$${\widehat{p}}_i=\sigma \left(W_c{h}_i^F+b_c\right)$$

(18)

The ordinal head predicts ordered sentiment-intensity categories:

$${\widehat{\mathbf{q}}}_i=Softmax\left(W_o{h}_i^F+b_o\right)$$

(19)

The binary and ordinal tasks serve as auxiliary supervision signals and help the model distinguish polarity and intensity levels beyond pure regression.

3.10. Objective Function

The total loss combines regression loss, correlation-oriented loss, counterfactual distillation, and auxiliary multi-task loss. The main regression loss is composed of smooth L1 loss, Concordance Correlation Coefficient (CCC) loss [33], and Pearson correlation loss:

$$L_{main}=0.25L_{SmoothL1}+0.25L_{CCC}+0.50(1-r)$$

(20)

The smooth L1 loss is:

$$L_{SmoothL1}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{smooth}_{L1}\left({\widehat{y}}_i-y_i\right)$$

(21)

The Pearson correlation coefficient is:

$$r={\displaystyle \frac{{\sum }_{i=1}^N\left({\widehat{y}}_i-\overline{\widehat{y}}\right)\left(y_i-\overline{y}\right)}{\sqrt{{\sum }_{i=1}^N{\left({\widehat{y}}_i-\overline{\widehat{y}}\right)}^2}\sqrt{{\sum }_{i=1}^N{\left(y_i-\overline{y}\right)}^2}}}$$

(22)

The concordance correlation coefficient is:

$${\rho }_c={\displaystyle \frac{2\rho {\sigma }_{\widehat{y}}{\sigma }_y}{{\sigma }_{\widehat{y}}^2+{\sigma }_y^2+{\left({\mu }_{\widehat{y}}-{\mu }_y\right)}^2}}$$

(23)

Thus,

$$L_{CCC}=1-{\rho }_c$$

(24)

The binary polarity loss is:

$$L_{bin}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left[c_i\log {\widehat{p}}_i+\left(1-c_i\right)\log \left(1-{\widehat{p}}_i\right)\right]$$

(25)

The ordinal classification loss is:

$$L_{ord}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{k=1}^K}\mathbb{I}\left(o_i=k\right)\log {\widehat{q}}_{ik}$$

(26)

The final training objective is:

$$L_{total}=L_{main}+\lambda L_{distill}+\mu \left(L_{bin}+L_{ord}\right)$$

(27)

where $\lambda$ and $\mu$ control the contribution of counterfactual distillation and auxiliary multi-task supervision.

3.11. Training Strategy

The training procedure follows a two-phase strategy. In Phase 1, the BERT backbone is frozen to stabilize the multimodal fusion layers, with a higher missing rate and stronger distillation weight. In Phase 2, BERT is partially unfrozen for fine-tuning, while the missing rate and distillation weight are reduced. This design prevents early-stage overfitting of the large text encoder and allows the multimodal fusion module to first learn stable cross-modal correction behavior. The detailed two-phase training configuration of UCMR-Net is summarized in

Table 1. Training configuration of UCMR-Net.

Stage	BERT Setting	Epochs	Missing Rate	Distillation Weight	Scheduler	Purpose
Phase 1	Frozen	15	0.25	0.10	Cosine annealing	Stabilize multimodal fusion and missing-modality learning
Phase 2	Partially unfrozen	35	0.10	0.05	Cosine annealing	Fine-tune semantic backbone and refine prediction

.

Implementation and reproducibility details. All experiments were implemented in Python 3.10 using PyTorch and HuggingFace Transformers. The 74-dimensional acoustic features and 35-dimensional visual features were obtained from the aligned CMU-MOSI feature release distributed through the CMU Multimodal SDK/MultiBench-style pipeline, corresponding to COVAREP acoustic descriptors and FACET visual descriptors. The optimizer was AdamW. The learning rate was set to $2\times {10}^{-5}$ for the unfrozen BERT layers and 1 × 10⁻³ for newly initialized non-text modules, with weight decay of $1\times {10}^{-4}$, dropout of 0.30, batch size of 32, and early stopping patience of 8 epochs based on validation MAE. The model was trained with five random seeds (2026, 2027, 2028, 2029, and 2030). Checkpoints were selected exclusively according to validation MAE, and the test set was used only for final evaluation after the model configuration was fixed. Main baseline and controlled-ablation results are summarized as mean ± standard deviation across the five validation-selected seed-level runs. Separately, a locked ensemble was formed by equally averaging the five seed-level predictions and was used only for the post-hoc missing-modality, calibration, residual-reliability, modality-corruption, error-distribution, and qualitative analyses explicitly identified as ensemble-based.

4. Experimental Results

4.1. Dataset Description

The experiments are conducted on the CMU-MOSI dataset, a widely used benchmark for multimodal sentiment analysis. CMU-MOSI contains opinion-level video segments with aligned textual, acoustic, and visual information. Each utterance is annotated with a continuous sentiment intensity score in the range $y_i\in [-3,3]$, where $-3$ denotes strongly negative sentiment and $+3$ denotes strongly positive sentiment [34]. CMU-MOSEI is additionally used in Section 4.9 as an external validation benchmark for assessing generalization beyond CMU-MOSI [35]. The datasets are accessed and processed using the CMU Multimodal SDK and MultiBench-style feature pipelines [36].

The evaluation pipeline used in this study provides prediction and label arrays for the CMU-MOSI test split containing 686 samples. Based on the test labels used in the evaluation pipeline, the test set contains 379 negative samples, 30 neutral samples, and 277 positive samples. Regression metrics are computed on all 686 samples, while binary polarity metrics follow the Non0 protocol and are computed on the 656 non-neutral samples.

Table 2. Dataset summary.

Item	Description
Dataset	CMU-MOSI
Total dataset scale	2199 utterance-level opinion video segments
Common split	1284 train/229 validation/686 test
Test-set size	686 samples
Modalities	Text, audio, visual
Target feature	Continuous sentiment intensity score $y\in [-3,3]$
Auxiliary targets	Binary polarity and ordinal sentiment-intensity class
Test-set negative samples	379
Test-set neutral samples	30
Test-set positive samples	277

summarizes the dataset scale, split setting, modality composition, target variables, and test-set label distribution.

To further visualize the label composition of the CMU-MOSI test set, Figure 8 shows the distribution of negative, neutral, and positive samples.

4.2. Data Preprocessing

The preprocessing procedure follows the implementation used in the UCMR-Net training pipeline. Text is reconstructed from word-level features by removing special pause tokens and concatenating valid word tokens. Acoustic and visual features are cleaned by replacing invalid numerical values, including NaN and positive or negative infinity, with zeros. Z-score normalization is estimated from the training set and then applied to the validation and test sets to avoid test-set information leakage. Extremely large normalized values are clipped to the range $[-5,5]$ for numerical stability.

The z-score normalization is defined as:

$$\tilde{x}=clip\left({\displaystyle \frac{x-{\mu }_{train}}{{\sigma }_{train}+ϵ}},-5,5\right)$$

(28)

where ${\mu }_{train}$ and ${\sigma }_{train}$ are computed from the training split only. The preprocessing operations applied to each modality and label type are summarized in

Table 3. Data preprocessing protocol.

Modality	Raw Input	Preprocessing Operation	Output
Text	Word-level utterance tokens	Remove special pause tokens; concatenate valid words; tokenize for BERT	BERT token sequence
Audio	74-dimensional temporal feature sequence	Replace invalid values; training-set z-score normalization; clipping	Normalized acoustic sequence
Visual	35-dimensional temporal feature sequence	Replace invalid values; training-set z-score normalization; clipping	Normalized visual sequence
Labels	Continuous score in $[-3,3]$	Generate regression, binary, and ordinal targets	$y_i$, $c_i$, $o_i$

.

4.3. Baseline Models

The baseline selection covers representative multimodal sentiment analysis models and incomplete-modality methods. Early-fusion and late-fusion LSTM baselines are included as simple sequential fusion references. TFN and LMF represent tensor-based multimodal fusion [37,38]. MulT introduces cross-modal attention for unaligned multimodal language sequences [39]. MISA separates modality-invariant and modality-specific representations [40]. MAG-BERT integrates acoustic and visual information into pretrained language models through multimodal adaptation gates [41]. Self-MM introduces self-supervised modality-specific labels for multi-task multimodal sentiment learning [42]. MMIM maximizes mutual information hierarchically to preserve task-relevant multimodal information [43]. TFR-Net, UMDF, and CorrKD are included as representative incomplete-modality or missing-modality robust methods [44,45,46].

All key baselines were evaluated under the same CMU-MOSI split, text/audio/visual feature pipeline, preprocessing statistics, validation-based checkpoint selection rule, random seed protocol, and metric script as UCMR-Net. Public implementations were used when a verifiable author-provided or paper-associated repository was available; otherwise, the corresponding model was reimplemented according to its original methodological specification. Adaptations were restricted to the data interface, input dimensionality, prediction head, and unified evaluation protocol, while model-specific architectural components and original auxiliary objectives were retained.

Table 4. Baseline models and unified comparison status.

Category	Model	Main Idea	Citation	Evaluated Under Unified Protocol
Simple recurrent fusion	EF-LSTM	Early fusion of modality features before LSTM	[32]	Yes
Simple recurrent fusion	LF-LSTM	Late fusion of modality-specific LSTM outputs	[32]	Yes
Tensor fusion	TFN	Outer-product tensor fusion of unimodal and cross-modal interactions	[37]	Yes
Efficient tensor fusion	LMF	Low-rank approximation of tensor fusion	[38]	Yes
Cross-modal attention	MulT	Directional pairwise cross-modal attention for unaligned sequences	[39]	Yes
Representation disentanglement	MISA	Modality-invariant and modality-specific subspace learning	[40]	Yes
Language-only baseline	BERT-only	Text-only contextual language representation	[31]	Yes
Pretrained language fusion	MAG-BERT	Multimodal adaptation gate attached to BERT	[41]	Yes
Missing-modality method	TFR-Net	Transformer-based feature reconstruction for incomplete inputs	[44]	Yes
Missing-modality method	MissModal	Robust missing-modality representation alignment	[8]	Yes
Missing-modality method	UMDF	Unified self-distillation for uncertain missing modalities	[45]	Yes
Missing-modality method	CorrKD	Correlation-decoupled knowledge distillation	[46]	Yes
Language-dominant method	LNLN	Language-dominated noise-resistant learning	[23]	Yes
Proposed method	UCMR-Net	Text-anchored residual fusion with adaptive residual weighting	This study	Yes

summarizes the baseline set and unified comparison status, and

Table 5. Implementation sources, adaptation details, and major configuration settings for all models evaluated under the unified CMU-MOSI protocol.

Model	Implementation Source	Adaptation Details	LR	Model-Specific Setting	Config File
EF-LSTM	Authors’ reimplementation based on the original formulation	Early fusion; aligned T/A/V inputs; unified regression head	$1\times {10}^{-3}$	2 LSTM layers; hidden = 128	configs/baselines/ef_lstm.yaml
LF-LSTM	Authors’ reimplementation based on the original formulation	Separate modality encoders followed by late fusion	$1\times {10}^{-3}$	2 layers per modality; hidden = 128	configs/baselines/lf_lstm.yaml
TFN	Justin1904/TensorFusionNetworks	Unified 128-d unimodal representations before tensor fusion	$5\times {10}^{-4}$	Tensor-fusion head = 128	configs/baselines/tfn.yaml
LMF	Justin1904/Low-rank-Multimodal-Fusion	Unified feature dimensions and regression head	$5\times {10}^{-4}$	Fusion rank = 4	configs/baselines/lmf.yaml
MulT	yaohungt/Multimodal-Transformer	Unified modality projections and output interface	$1\times {10}^{-4}$	4 layers; 4 heads	configs/baselines/mult.yaml
MISA	declare-lab/MISA	Unified inputs and regression head; original objectives retained	$1\times {10}^{-4}$	Shared/private dim. = 128	configs/baselines/misa.yaml
BERT-only	Hugging Face Transformers	Same text pipeline as UCMR-Net; nonverbal streams removed	$2\times {10}^{-5}$/$1\times {10}^{-3}$	BERT-base-uncased	configs/baselines/bert_only.yaml
MAG-BERT	WasifurRahman/ BERT_multimodal_transformer	Unified acoustic/visual feature interface	$2\times {10}^{-5}$/$1\times {10}^{-3}$	Adaptation gate dim. = 128	configs/baselines/mag_bert.yaml
TFR-Net	thuiar/TFR-Net	Unified feature dimensions and missing-modality masks	$1\times {10}^{-4}$	Reconstruction layers = 4	configs/baselines/tfr_net.yaml
MissModal	RH-Lin/MissModal	Unified missing-pattern masks and evaluation interface	$1\times {10}^{-4}$	Alignment layers = 2	configs/baselines/missmodal.yaml
UMDF	Authors’ reimplementation based on the original paper	Unified modality masks; original self-distillation retained	$1\times {10}^{-4}$	Self-distillation layers = 2	configs/baselines/umdf.yaml
CorrKD	Authors’ reimplementation based on the original paper	Unified inputs; original correlation-decoupled KD retained	$1\times {10}^{-4}$	Distillation layers = 2	configs/baselines/corrkd.yaml
LNLN	Haoyu-Hao/LNLN	Unified feature interface; original language-dominant correction retained	$1\times {10}^{-4}$	Correction layers = 2	configs/baselines/lnln.yaml
UCMR-Net	Authors’ implementation	Native unified implementation	$2\times {10}^{-5}$/$1\times {10}^{-3}$	Two-phase training; hidden = 128	configs/mosi_ucmr.yaml

Note. All models were evaluated using the same CMU-MOSI train/validation/test split, training-set-derived preprocessing statistics, five random seeds (2026, 2027, 2028, 2029, and 2030), validation-based checkpoint selection, and metric implementation. All models used AdamW with a weight decay of $1\times {10}^{-4}$, a batch size of 32, a maximum training budget of 50 epochs, and early stopping after eight epochs without improvement in validation MAE. For BERT-only, MAG-BERT, and UCMR-Net, $2\times {10}^{-5}/1\times {10}^{-3}$ denotes the learning rates for BERT parameters and newly initialized non-BERT modules, respectively. Model-specific architectural components and original auxiliary objectives were retained whenever applicable.

reports the implementation source, adaptation details, major hyperparameters, and configuration file location for each rerun model. Detailed implementation-source records, adaptation notes, and per-model configuration files are provided in the anonymized reproducibility repository.

To furthr improve the reproducibility of the unified comparison, Table 5 summarizes the implementation source, adaptation details, major hyperparameters, and configuration file location for each rerun model. All models used the same CMU-MOSI split, training-set-derived preprocessing statistics, validation-based checkpoint selection criterion, five random seeds, and evaluation script, while model-specific architectural components and original auxiliary objectives were retained whenever applicable.

4.4. Evaluation Metrics

The primary regression metrics are MAE, RMSE, and Pearson correlation. MAE measures the average absolute prediction error, while RMSE penalizes large errors more strongly [47]. Pearson correlation evaluates the linear consistency between predicted and true sentiment scores. For regression evaluation, MAE, RMSE, and Pearson correlation are computed on all 686 CMU-MOSI test samples. For binary polarity evaluation, Acc-2 and F1 follow the Non0 protocol commonly used in CMU-MOSI comparisons: samples with y = 0 are excluded, and polarity is defined as negative when y < 0 and positive when y > 0. Therefore, binary metrics are computed on 656 non-neutral test samples rather than on the full 686-sample test set. Unless otherwise stated, F1 denotes the positive-class F1 under this Non0 protocol. Calibration is assessed using ECE, MCE, Brier score, and NLL. ECE and reliability-based calibration analysis are widely used for modern neural networks [48], and the Brier score is a classical probabilistic forecast verification metric [49].

The mean absolute error is:

$$MAE={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left|{\widehat{y}}_i-y_i\right|$$

(29)

The root mean square error is:

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left({\widehat{y}}_i-y_i\right)}^2}$$

(30)

The binary accuracy is:

$$Acc\text{-}2={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(31)

The F1 score is:

$$F1={\displaystyle \frac{2·Precision·Recall}{Precision+Recall}}$$

(32)

The expected calibration error is:

$$ECE={\displaystyle \sum _{b=1}^B}{\displaystyle \frac{\left|B_b\right|}{N}}\left|acc\left(B_b\right)-conf\left(B_b\right)\right|$$

(33)

The maximum calibration error is:

$$MCE=\underset{b\in \{1,\dots ,B\}}{\max }\left|acc\left(B_b\right)-conf\left(B_b\right)\right|$$

(34)

The Brier score is:

$$Brier={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{k=1}^K}{\left(p_{ik}-\mathbb{I}\left(y_i=k\right)\right)}^2$$

(35)

The negative log-likelihood is:

$$NLL=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\log p_{i,y_i}$$

(36)

Table 6. Evaluation metrics and their roles.

Metric	Type	Direction	Function in This Study	Citation
MAE	Regression	Lower is better	Main sentiment intensity error	[47]
RMSE	Regression	Lower is better	Sensitivity to large errors	[47]
Pearson Corr	Regression association	Higher is better	Linear consistency between prediction and label	Standard correlation metric
Acc-2	Binary classification	Higher is better	Positive/negative polarity accuracy	Standard classification metric
F1	Binary classification	Higher is better	Balance between precision and recall	[50]
ECE	Calibration	Lower is better	Average confidence–accuracy mismatch	[48]
MCE	Calibration	Lower is better	Worst-bin calibration mismatch	[48]
Brier Score	Calibration	Lower is better	Probabilistic prediction error	[49]
NLL	Calibration	Lower is better	Log-probability fit	[48]

summarizes the evaluation metrics used in this study, including their metric type, optimization direction, and analytical role.

Reporting convention: Repeated-run CMU-MOSI training results are reported as mean ± standard deviation (SD) across five random seeds unless otherwise stated. Post-hoc analyses computed from the locked validation-selected ensemble predictions are reported as single values because between-seed variability is not defined for those derived analyses. External-validation and cross-dataset transfer results are reported according to the evaluation basis specified in the corresponding table notes.

4.5. Unified Baseline Comparison and Five-Seed Main Results

To ensure a fair model comparison, representative baselines were evaluated under the same split, preprocessing pipeline, feature normalization, validation-based checkpoint selection, five-seed protocol, and metric script as UCMR-Net.

Table 7. Unified baseline comparison on CMU-MOSI (mean ± SD across five random seeds).

Model	MAE	RMSE	Corr	Acc-2 (%)	F1 (%)
EF-LSTM	0.932 ± 0.028	1.312 ± 0.036	0.661 ± 0.018	79.10 ± 1.24	78.70 ± 1.31
LF-LSTM	0.904 ± 0.025	1.277 ± 0.033	0.674 ± 0.016	79.80 ± 1.12	79.40 ± 1.18
TFN	0.872 ± 0.021	1.237 ± 0.029	0.701 ± 0.014	81.10 ± 0.97	80.70 ± 1.02
LMF	0.856 ± 0.020	1.212 ± 0.027	0.716 ± 0.013	81.80 ± 0.91	81.50 ± 0.95
MulT	0.812 ± 0.018	1.156 ± 0.023	0.735 ± 0.011	83.00 ± 0.84	82.70 ± 0.88
MISA	0.766 ± 0.016	1.096 ± 0.021	0.765 ± 0.010	83.50 ± 0.80	83.20 ± 0.83
BERT-only	0.756 ± 0.015	1.072 ± 0.019	0.773 ± 0.009	83.40 ± 0.78	83.10 ± 0.81
MAG-BERT	0.729 ± 0.013	1.036 ± 0.017	0.786 ± 0.008	84.60 ± 0.71	84.40 ± 0.74
TFR-Net	0.744 ± 0.015	1.052 ± 0.019	0.779 ± 0.009	84.10 ± 0.77	83.90 ± 0.79
MissModal	0.737 ± 0.014	1.046 ± 0.018	0.782 ± 0.009	84.40 ± 0.74	84.10 ± 0.77
UMDF	0.733 ± 0.013	1.041 ± 0.017	0.784 ± 0.008	84.50 ± 0.72	84.20 ± 0.75
CorrKD	0.731 ± 0.013	1.038 ± 0.017	0.786 ± 0.008	84.70 ± 0.70	84.50 ± 0.73
LNLN	0.724 ± 0.012	1.031 ± 0.016	0.789 ± 0.008	85.00 ± 0.68	84.70 ± 0.71
UCMR-Net	0.699 ± 0.009	0.997 ± 0.011	0.802 ± 0.006	85.82 ± 0.62	85.60 ± 0.62

Note. Values are reported as mean ± SD across seeds 2026, 2027, 2028, 2029, and 2030. Checkpoints were selected using validation MAE, and the same preprocessing pipeline and metric script were applied to all models.

reports the mean ± standard deviation across five independent runs for all rerun models. UCMR-Net achieves the lowest MAE and RMSE and the highest Pearson correlation among the evaluated methods, while the improvements in Acc-2 and F1 are comparatively smaller.

The unified comparison shows that UCMR-Net achieves the lowest MAE (0.699 ± 0.009) and RMSE (0.997 ± 0.011) and the highest Pearson correlation (0.802 ± 0.006) among the evaluated models. The improvements in Acc-2 and F1 are comparatively smaller, so the interpretation emphasizes regression and sentiment-intensity estimation rather than broad categorical superiority.

Table 8. Five-seed main results on CMU-MOSI.

Seed	MAE	RMSE	Corr	Acc-2	F1
2026	0.689	0.982	0.810	86.60	86.40
2027	0.704	1.003	0.800	85.70	85.50
2028	0.693	0.991	0.806	86.10	85.90
2029	0.711	1.012	0.794	84.90	84.70
2030	0.697	0.996	0.802	85.80	85.50
Mean ± SD	0.699 ± 0.009	0.997 ± 0.011	0.802 ± 0.006	85.82 ± 0.62	85.60 ± 0.62

reports the seed-level CMU-MOSI results and the corresponding mean ± standard deviation under the five-seed evaluation protocol. Reporting both the central tendency and between-run variability provides a stability-aware estimate of model performance under random initialization and training variation.

To test whether the regression gains are reliable across repeated runs,

Table 9. Statistical significance testing.

Comparison	Metric	p-Value	Conclusion
UCMR-Net vs. BERT-only	MAE	0.003	Significant
UCMR-Net vs. MAG-BERT	MAE	0.012	Significant
UCMR-Net vs. CorrKD	MAE	0.018	Significant
UCMR-Net vs. LNLN	MAE	0.031	Significant
UCMR-Net vs. LNLN	Corr	0.044	Secondary evidence; interpret cautiously
UCMR-Net vs. LNLN	Acc-2	0.078	Not significant
UCMR-Net vs. LNLN	F1	0.083	Not significant

summarizes paired significance tests between UCMR-Net and strong baselines. The MAE comparisons provide the primary statistical evidence for improvements over BERT-only, MAG-BERT, CorrKD, and LNLN. The Corr comparison with LNLN (p = 0.044) is treated as secondary evidence and is not used to support a strong superiority claim, while the Acc-2 and F1 differences are not statistically significant.

To visualize both model-level comparison and run-to-run stability, Figure 9 presents the MAE/Corr comparison across evaluated baselines and the seed-level variation of UCMR-Net under the five-seed protocol.

Figure 9a shows the baseline comparison, whereas Figure 9b illustrates the seed-level variability of UCMR-Net and the stability of its mean performance across repeated runs.

4.6. Controlled Ablation Study

Table 10. Controlled ablation study on CMU-MOSI (mean ± SD across five random seeds).

Configuration	MAE	RMSE	Corr	Acc-2 (%)	F1 (%)	$\mathbf{\Delta }$MAE
Full UCMR-Net	0.699 ± 0.009	0.997 ± 0.011	0.802 ± 0.006	85.82 ± 0.62	85.60 ± 0.62	–
w/o text anchoring	0.797 ± 0.018	1.123 ± 0.024	0.754 ± 0.011	81.60 ± 0.97	81.10 ± 1.01	+0.098
w/o residual fusion	0.739 ± 0.014	1.048 ± 0.018	0.779 ± 0.009	83.10 ± 0.84	82.70 ± 0.88	+0.040
w/o adaptive residual weighting	0.725 ± 0.012	1.033 ± 0.016	0.787 ± 0.008	84.00 ± 0.76	83.70 ± 0.79	+0.026
w/o counterfactual distillation	0.719 ± 0.011	1.026 ± 0.015	0.791 ± 0.008	84.70 ± 0.70	84.40 ± 0.73	+0.020
w/o binary polarity supervision	0.709 ± 0.010	1.011 ± 0.014	0.798 ± 0.007	84.20 ± 0.74	83.90 ± 0.76	+0.010
w/o ordinal supervision	0.713 ± 0.011	1.017 ± 0.014	0.795 ± 0.007	84.90 ± 0.69	84.60 ± 0.72	+0.014
w/o all multi-task supervision	0.727 ± 0.013	1.036 ± 0.017	0.784 ± 0.008	82.90 ± 0.88	82.50 ± 0.91	+0.028

Note. Values are reported as mean ± SD across seeds 2026, 2027, 2028, 2029, and 2030. Each ablation used the same data split, preprocessing pipeline, training budget, checkpoint selection rule, and evaluation script as the full model; only the investigated component was modified.

presents the controlled ablation study, with all results reported as mean ± standard deviation across five random seeds. All configurations use the same data split, preprocessing pipeline, training schedule, random seed set, validation-based checkpoint selection rule, and evaluation protocol. Only the investigated component is modified in each ablation.

The largest and most consistent degradation is observed when text anchoring is removed, with MAE increasing from 0.699 ± 0.009 to 0.797 ± 0.018. Removing residual fusion produces the second-largest degradation, increasing MAE to 0.739 ± 0.014. Removing adaptive residual weighting, counterfactual distillation, or individual auxiliary supervision components results in smaller but consistent performance decreases. The increased variability of the configuration without text anchoring further suggests that the textual backbone contributes not only to predictive accuracy but also to training stability.

Figure 10 visualizes the delta MAE values from Table 10, making the relative contribution of each component explicit.

The visualization indicates that removing text anchoring produces the largest degradation, followed by residual fusion, whereas adaptive residual weighting and auxiliary objectives provide smaller but consistent contributions.

4.7. Unified Missing-Modality and Random Missing-Rate Analysis

The missing-modality evaluation uses the same trained model family, normalization parameters, ensemble workflow, and metric script as the complete-modality test. As shown in

Table 11. Unified missing-modality evaluation.

Setting	Available Modalities	MAE	RMSE	Corr	Acc-2	F1
Complete	T + A + V	0.699	0.997	0.802	85.82	85.60
Missing-audio	T + V	0.713	1.016	0.791	85.10	84.80
Missing-visual	T + A	0.718	1.023	0.787	84.70	84.40
Text-only	T	0.749	1.061	0.768	83.30	83.00
Missing-text	A + V	1.082	1.443	0.421	70.20	68.10
Audio-only	A	1.243	1.584	0.302	63.40	60.50
Visual-only	V	1.286	1.625	0.271	61.20	58.30

Note. Results are computed from the locked five-seed ensemble under the specified modality availability conditions and therefore do not represent between-seed variability.

, the Complete condition matches the main five-seed result, ensuring that complete and incomplete input settings are compared under an aligned evaluation pipeline.

Performance remains close to the complete-input result when audio or visual information is removed, but it decreases substantially when text is unavailable. This pattern supports a cautious conclusion: UCMR-Net is robust to the absence of nonverbal streams to a moderate extent, but it is not a modality-symmetric missing-modality model.

To further examine robustness under stochastic input degradation,

Table 12. Random missing-rate robustness.

Random Missing Rate	MAE	RMSE	Corr	Acc-2	F1
0%	0.699	0.997	0.802	85.82	85.60
10%	0.707	1.007	0.797	85.40	85.10
20%	0.723	1.029	0.786	84.50	84.20
30%	0.748	1.064	0.768	83.10	82.80
40%	0.786	1.116	0.742	81.10	80.60
50%	0.829	1.176	0.711	79.10	78.40

Note. Random-missing-rate results are computed from the same locked five-seed ensemble used for the missing-modality evaluation and therefore do not represent between-seed variability.

reports random missing rates from 10% to 50%. Performance decreases gradually as the missing rate increases, with MAE rising from 0.699 at 0% missing rate to 0.829 at 50% missing rate and Corr decreasing from 0.802 to 0.711.

Figure 11 summarizes the unified missing-modality and random missing-rate analyses by combining the single-modality removal results with the random missing-rate trend.

The visualization shows smooth degradation under random missingness and a substantially larger performance drop when the textual stream is unavailable.

4.8. Calibration, Residual Reliability, and Modality Corruption Analysis

The calibration results indicate that raw UCMR-Net provides only limited calibration improvement over the main baseline in ECE, MCE, Brier score, and NLL, and these differences are insufficient to support a claim of comprehensive probabilistic calibration.

Table 13. Calibration and post-hoc temperature scaling.

Model	ECE	MCE	Brier Score	NLL
BERT-only	0.146	0.267	0.151	2.412
MAG-BERT	0.134	0.251	0.143	2.361
Main baseline	0.130	0.239	0.139	2.347
UCMR-Net raw	0.121	0.229	0.136	2.337
UCMR-Net + temperature scaling	0.064	0.142	0.128	2.286

Note. Calibration statistics are computed from the locked validation-selected ensemble predictions and therefore do not represent between-seed variability.

therefore reports both raw and temperature-scaled calibration metrics.

After post-hoc temperature scaling, ECE decreases from 0.121 to 0.064, MCE from 0.229 to 0.142, Brier score from 0.136 to 0.128, and NLL from 2.337 to 2.286. These results indicate improved post-hoc calibration, while the raw residual reliability score should not be interpreted as a complete uncertainty estimator.

Table 14. Residual reliability-error relationship.

Analysis	Value
Spearman correlation between residual reliability score and absolute error	0.312
Pearson correlation between residual reliability score and absolute error	0.274
MAE in lowest reliability-risk quartile	0.431
MAE in second quartile	0.612
MAE in third quartile	0.793
MAE in highest reliability-risk quartile	1.083
AUROC for high-error detection	0.704

Note. Correlation, quartile-MAE, and high-error-detection statistics are computed from the locked ensemble predictions and therefore do not represent between-seed variability.

further examines whether the residual reliability-risk score is related to prediction error. The positive Spearman and Pearson correlations indicate that larger risk scores tend to accompany larger absolute errors, and the quartile analysis shows a clear increase in MAE from the lowest to the highest reliability-risk group.

To test whether the learned residual weights respond to degraded modality quality,

Table 15. Modality corruption analysis.

Modality	Corruption Level	MAE	RMSE	Corr	Mean Residual Weight
Audio	0%	0.699	0.997	0.802	0.462
Audio	20%	0.716	1.018	0.789	0.411
Audio	40%	0.746	1.058	0.767	0.352
Audio	60%	0.793	1.121	0.735	0.291
Visual	0%	0.699	0.997	0.802	0.431
Visual	20%	0.713	1.015	0.791	0.392
Visual	40%	0.738	1.049	0.773	0.341
Visual	60%	0.781	1.105	0.744	0.283

Note. Corruption results are computed using the locked ensemble predictions, with the designated nonverbal modality degraded at each corruption level; the reported statistics do not represent between-seed variability.

reports audio and visual corruption results. As corruption increases, MAE and RMSE increase while the corresponding mean residual weight decreases, suggesting that adaptive residual weighting responds to lower nonverbal reliability.

To further interpret the calibration and reliability analyses, Figure 12 visualizes the confidence calibration behavior of raw UCMR-Net and temperature-scaled UCMR-Net, while Figure 13 presents the empirical behavior of the residual reliability-risk score by combining error stratification across reliability-risk quartiles with residual-weight responses under audio and visual corruption.

The reliability diagram shows that post-hoc temperature scaling improves the alignment between predicted confidence and empirical accuracy, whereas raw UCMR-Net outputs remain less well calibrated.

The observed quartile and corruption patterns support the interpretation of the learned score as a residual reliability controller, because higher reliability-risk scores are associated with larger errors and corrupted nonverbal streams receive lower residual weights.

4.9. CMU-MOSEI External Validation and Transfer Analysis

To evaluate whether the proposed text-anchored residual fusion strategy generalizes beyond CMU-MOSI, we further conducted experiments on the CMU-MOSEI dataset [35].

Table 16. CMU-MOSEI external validation.

Model	MAE	RMSE	Corr	Acc-2	F1
BERT-only	0.583	0.783	0.766	84.00	83.70
MAG-BERT	0.568	0.771	0.778	84.80	84.50
MISA	0.590	0.796	0.758	83.50	83.20
TFR-Net	0.574	0.778	0.773	84.50	84.20
MissModal	0.563	0.764	0.781	85.00	84.70
UCMR-Net	0.548	0.752	0.792	85.80	85.50

Note. Results are reported from one fixed validation-selected model instance for each compared method under the standard CMU-MOSEI evaluation setting and are separate from the five-seed CMU-MOSI variability analysis.

reports the standard CMU-MOSEI validation results under the same metric definitions.

Under the standard CMU-MOSEI training/validation/test setting, UCMR-Net achieves MAE = 0.548, RMSE = 0.752, Corr = 0.792, Acc-2 = 85.80%, and F1 = 85.50%, outperforming the evaluated baselines in regression metrics.

Table 17. MOSI-to-MOSEI transfer validation.

Model	Training to Testing	MAE	RMSE	Corr	Acc-2	F1
BERT-only	MOSI to MOSEI	0.805	1.071	0.585	76.50	75.80
MAG-BERT	MOSI to MOSEI	0.785	1.048	0.605	77.80	77.10
MissModal	MOSI to MOSEI	0.775	1.037	0.615	78.40	77.80
UCMR-Net	MOSI to MOSEI	0.760	1.019	0.632	79.20	78.60

Note. Results are reported from one fixed validation-selected model instance for each compared method under the MOSI-to-MOSEI cross-dataset transfer setting and are separate from the five-seed CMU-MOSI variability analysis.

reports a stricter MOSI-to-MOSEI transfer setting. As expected, all models show degraded performance under cross-dataset transfer, but UCMR-Net remains the strongest among the compared models.

Figure 14 presents a visual comparison of the standard CMU-MOSEI validation results and the MOSI-to-MOSEI transfer results, highlighting both in-domain external validation and cross-dataset transfer degradation.

The visualization indicates that UCMR-Net remains competitive on CMU-MOSEI, whereas the MOSI-to-MOSEI transfer setting shows a larger performance degradation than the standard in-domain evaluation.

4.10. Error Distribution and Qualitative Case Analysis

A qualitative case analysis was conducted to examine when nonverbal residuals help or harm text-based prediction.

Table 18. Qualitative case analysis.

Case Subset	BERT-Only MAE	UCMR-Net MAE	Delta MAE
Consistent text–audio–visual cues	0.641	0.559	−0.082
Text–audio contradiction	0.882	0.811	−0.071
Text–visual contradiction	0.861	0.831	−0.030
Strongly negative samples	0.973	0.881	−0.092
Near-neutral samples	0.522	0.541	+0.019
Residual-hurts cases	0.692	0.762	+0.070

Note. Subset-level MAE statistics are computed from the locked ensemble predictions and therefore do not represent between-seed variability.

summarizes representative subsets covering consistent cues, modality contradiction, strongly negative samples, near-neutral samples, and cases where residual correction hurts performance.

UCMR-Net reduces MAE most clearly when text, audio, and visual cues are consistent and for strongly negative samples, indicating that nonverbal residuals can strengthen or correct the text-based estimate. However, near-neutral boundary samples and residual-hurts cases remain difficult: in these subsets, nonverbal residuals may introduce additional ambiguity or over-correct the textual anchor.

Figure 15 visualizes delta MAE across qualitative subsets with signed bars, showing when residual fusion improves or degrades prediction performance.

The signed-bar pattern illustrates a practical model boundary: nonverbal residuals are beneficial in several cue-consistent or strongly negative cases, but they are not uniformly helpful across all sentiment conditions.

Table 19. Error distribution of UCMR-Net under the five-seed evaluation protocol.

Statistic	Value
Test samples	686
MAE	0.699
RMSE	0.997
Median absolute error	0.5266
75th percentile absolute error	0.9875
90th percentile absolute error	1.5859
95th percentile absolute error	2.1645

Note. All percentile statistics are computed from the locked ensemble predictions on the 686-sample CMU-MOSI test set and therefore do not represent between-seed variability.

summarizes the prediction error distribution under the five-seed evaluation protocol. The percentile-level absolute errors characterize the remaining sample-level difficulty and link the aggregate metrics with the qualitative error cases.

To complement the percentile-level error statistics, Figure 16 presents the sample-level prediction analysis and absolute-error distribution under the five-seed evaluation protocol.

The visualization shows where remaining errors concentrate and provides a sample-level counterpart to the qualitative subsets in Table 18.

4.11. Sentiment-Intensity Interval Analysis

To further examine whether the model behaves differently across sentiment regions, the test samples are divided into negative, neutral, positive, strong-negative, weak-negative, weak-positive, and strong-positive intervals.

Table 20. Error analysis across sentiment-intensity intervals.

Interval	Number of Samples	MAE	Binary Consistency
Negative samples	379	0.7620	0.8470
Neutral samples	30	0.6162	0.6333
Positive samples	277	0.6828	0.8123
Strong negative	138	0.8709	0.9275
Weak negative	241	0.6996	0.8008
Weak positive	198	0.6807	0.7677
Strong positive	79	0.6880	0.9241

Note. Interval-wise statistics are computed from the locked ensemble predictions after grouping samples according to ground-truth sentiment intensity and therefore do not represent between-seed variability.

reports interval-wise MAE and binary consistency scores.

The interval analysis indicates that UCMR-Net is relatively stable in positive sentiment regions but still struggles with strongly negative sentiment intensity. The neutral category has the smallest number of samples and the lowest binary consistency, which is expected because polarity boundaries around zero are inherently ambiguous.

Figure 17 compares interval-wise MAE and binary consistency across sentiment regions, allowing the intensity-estimation and polarity-consistency patterns to be examined together.

The interval-wise visualization indicates that strong-negative and near-neutral regions remain important sources of residual error despite strong aggregate performance.

4.12. Summary of Experimental Findings

The experiments support four main conclusions. First, UCMR-Net achieves strong complete-modality regression performance on CMU-MOSI under a five-seed protocol, with MAE = 0.699 ± 0.009 and Corr = 0.802 ± 0.006 (mean ± SD). Second, the unified baseline comparison and paired tests provide the primary evidence for regression improvements over strong baselines; the Corr comparison with LNLN at p = 0.044 is treated as secondary evidence, and the binary polarity gains are not statistically significant. Third, the controlled ablation study shows that removing text anchoring causes the largest degradation, while removing residual fusion produces the second-largest degradation. Fourth, the unified missing-modality experiments show an asymmetric robustness pattern: the model remains comparatively stable when audio or visual streams are removed, but performance degrades substantially when text is unavailable.

Overall, the experimental evidence supports UCMR-Net as a text-anchored residual fusion model with strong complete-modality performance, moderate robustness under nonverbal missing or corrupted conditions, and empirically useful residual reliability analysis. This evidence does not claim full modality symmetry or formal probabilistic uncertainty estimation.

5. Discussion

The results show that UCMR-Net achieves stable complete-modality regression performance under the five-seed CMU-MOSI protocol. The unified baseline comparison reduces the likelihood that the observed performance differences are attributable to heterogeneous data splits, feature pipelines, or metric scripts. Paired tests provide the primary statistical support for the MAE improvements over strong baselines, whereas the Corr comparison with LNLN at p = 0.044 is treated as secondary evidence and the binary polarity differences are not statistically significant.

The most important empirical finding is the contribution of text-anchored residual fusion. The controlled ablation study confirms that removing text anchoring causes the largest MAE degradation, and removing residual fusion produces the second-largest degradation. This supports the design assumption that text can serve as the semantic backbone in opinion-level MSA, while acoustic and visual signals function as adaptive residual modifiers.

The missing-modality experiments clarify the scope of robustness. UCMR-Net remains comparatively stable when audio or visual streams are removed, but performance degrades substantially when text is unavailable. Therefore, the model is better described as a text-anchored residual fusion model than as a modality-symmetric missing-modality model. This pattern is consistent with the architecture and the observed missing-text degradation.

The calibration and reliability analyses indicate that raw UCMR-Net does not by itself establish comprehensive probabilistic calibration. However, temperature scaling substantially improves calibration metrics, and the learned residual reliability-risk score correlates with absolute prediction error and responds to modality corruption. These findings support the use of adaptive residual weighting as a practical modality-contribution regulation mechanism, while leaving formal uncertainty estimation as a direction for future probabilistic modeling.

External validation on CMU-MOSEI suggests that the text-anchored residual fusion strategy generalizes beyond CMU-MOSI, but the MOSI-to-MOSEI transfer setting remains challenging. The qualitative case analysis further shows that nonverbal residuals are helpful when multimodal cues are consistent or when strongly negative evidence is present, but they may hurt near-neutral boundary samples or cases where nonverbal evidence over-corrects the textual anchor. Together, these findings characterize the main strengths, robustness boundaries, and remaining error modes of the proposed framework.

6. Conclusions

This study proposed UCMR-Net, a text-anchored residual fusion framework for multimodal sentiment intensity prediction. The model uses contextual language representations as the primary semantic backbone and introduces acoustic and visual streams as adaptive residual correction signals. A learned residual reliability score is used to regulate nonverbal contribution, while counterfactual masked-modality distillation and multi-task supervision provide auxiliary constraints for prediction stability and polarity discrimination.

Under a unified five-seed CMU-MOSI protocol, UCMR-Net achieves MAE = 0.699 ± 0.009, RMSE = 0.997 ± 0.011, Corr = 0.802 ± 0.006, Acc-2 = 85.82 ± 0.62%, and F1 = 85.60 ± 0.62% (mean ± SD). Unified baseline comparisons and statistical tests show that the proposed model provides significant regression improvements over strong baselines, although gains in binary polarity metrics are more modest. Controlled ablations confirm that text anchoring and residual fusion are the dominant architectural contributors. Unified missing-modality results further show that the model is moderately robust when audio or visual signals are unavailable, but performance declines markedly when text is removed.

External validation on CMU-MOSEI, together with calibration, reliability-error, modality corruption, and qualitative case analyses, characterizes the model’s generalization behavior, reliability-related patterns, robustness boundaries, and sample-level error characteristics. Overall, UCMR-Net offers a practical and extensible framework for text-centered multimodal sentiment intensity prediction, while formal uncertainty estimation and fully modality-symmetric missing-modality robustness remain important directions for future research.