A Contrastive and Uncertainty–Aware Framework for Multimodal Named Entity Recognition

Abstract

Multimodal named entity recognition (MNER) aims to improve entity detection in social media texts by leveraging accompanying images, but its performance is often affected by weak text–image alignment, noisy or irrelevant visual content, and limited separation among entity representations. To address these issues, this study proposes CUA-MNER, a contrastive uncertainty–aware framework that combines hierarchical vision–text alignment, variational uncertainty–aware fusion, and token-level contrastive learning. The alignment module models correspondences at token, phrase, and sentence levels, allowing local visual regions and global image context to support textual entity recognition. The fusion module estimates epistemic and aleatoric uncertainty through variational inference and adaptively adjusts the contribution of each modality for different samples. The contrastive objective further encourages entity representations of the same type to be closer while separating different entity types. Experiments on the Twitter2015 and Twitter2017 benchmarks demonstrate that CUA-MNER achieves F1 scores of 76.97% and 89.66%, respectively, outperforming competitive baselines by 0.66 and 1.95 F1 points. Ablation and diagnostic analyses show that the three components provide complementary benefits. These results suggest that modeling modality reliability is useful for robust MNER, while the additional modules also introduce computational overhead and leave cross-domain generalization as an open issue.

1. Introduction

Social media platforms such as Twitter, Weibo, and Instagram increasingly contain posts that combine text with images. This multimodal setting creates challenges for traditional text-only named entity recognition (NER), because the meaning of an entity mention may depend on visual context. For example, the word “Apple” may refer to either a fruit or a technology company. When the post is accompanied by an image of an Apple storefront or logo, the visual cue helps resolve the mention as an organization (ORG), as shown in Figure 1.

Figure 1. Example of visual–text complementarity for entity disambiguation.

Multimodal Named Entity Recognition (MNER) has emerged as a solution to this issue, seeking to improve the localization and classification of named entities by incorporating visual signals as complementary information alongside text. The underlying premise is that image content, whether through explicit depictions or implicit contextual cues, can help models resolve textual ambiguities [1]. Despite notable progress in recent years, three issues remain common in noisy social media MNER:

• Shallow cross-modal alignment leads to inadequate semantic correspondence. Existing methods predominantly rely on simple feature concatenation [2] or unidirectional attention mechanisms [3], which are insufficient for capturing deep semantic associations between textual tokens and image regions. These coarse-grained fusion strategies fail to establish fine-grained cross-modal correspondences, resulting in suboptimal integration of complementary visual and linguistic cues.
• Neglecting modality reliability differences introduces noise interference. Social media posts frequently contain noisy or semantically irrelevant images, with studies reporting that 33.8% of tweets exhibit image–text mismatch or only weak relevance [4]. Existing methods lack mechanisms to quantitatively assess the reliability of each modality, making them vulnerable to misleading visual signals. Prior work such as UAMNer [5] addresses this issue at the sample level through staged label refinement, but does not model continuous, token-level modality reliability, leaving per-token visual contributions unquantified. In extreme cases, this causes multimodal models to underperform their text-only counterparts.
• Over-reliance on supervised signals constrains representation discriminability. Conventional methods trained solely with cross-entropy loss focus on individual sample classification boundaries, overlooking the geometric structure of entity representations in the embedding space. This leads to insufficient intra-class compactness and inter-class separability, which is particularly detrimental when distinguishing fine-grained entity types.

To address these challenges, we propose CUA-MNER, a contrastive and uncertainty–aware framework for MNER. While hierarchical alignment, uncertainty estimation, contrastive learning, and CRF decoding have each been studied in prior work, their joint use in noisy Twitter MNER raises several practical design questions: how to align visual and textual information at different semantic granularities, how to adjust modality contributions when the accompanying image is unreliable, and how to improve entity representation separability without weakening sequence-labeling constraints. CUA-MNER addresses these questions by coordinating three components within a single end-to-end framework: hierarchical visual–text alignment, variational uncertainty–aware fusion, and token-level supervised contrastive learning. The main contributions of this paper are summarized as follows:

• We propose an end-to-end MNER framework that combines hierarchical cross-modal alignment with uncertainty–aware modality fusion. The alignment module captures token-, phrase-, and sentence-level visual–text correspondences, while the fusion module estimates token-level modality reliability and adaptively adjusts the contribution of textual and visual features.
• We introduce a dual-branch uncertainty–aware fusion mechanism that separately models epistemic and aleatoric uncertainty. Compared with fixed or purely attention-based fusion, this design provides a continuous token-level estimate of modality reliability and allows the model to reduce reliance on potentially noisy visual information when the image is weakly related to the text.
• We apply supervised contrastive learning at the entity-token level over fused multimodal representations. This objective complements the CRF-based sequence-labeling loss by encouraging representations with the same BIO-extended entity label to be closer while preserving boundary-sensitive label distinctions.
• We evaluate CUA-MNER on two widely used Twitter MNER benchmarks, Twitter2015 and Twitter2017. The experimental results show that the proposed framework achieves competitive overall performance compared with existing text-only, multimodal, contrastive-learning-based, and uncertainty–aware baselines. Ablation and diagnostic analyses further examine the contribution and limitations of the proposed modules.

2. Related Works

2.1. Multimodal Named Entity Recognition

Since the seminal work by Zhang et al. [1], MNER research has progressively advanced from shallow feature fusion toward deep cross-modal interaction. Lu et al. [6] introduced visual attention mechanisms that compute similarities between text tokens and global image features for cross-modal fusion; however, spatial structure information within images was not taken into account. Zhang et al. [1] subsequently proposed the Adaptive Co-Attention Network, employing gating mechanisms to dynamically modulate modality contributions, though the scalar-valued gating weights preclude fine-grained token-level adaptation. Yu et al. [7] presented a unified multimodal Transformer that generates image- and word-aware representations through dedicated multimodal interaction modules, further incorporating a span detection auxiliary task to guide entity prediction. Zhang et al. [8] designed a multi-granular fusion framework that leverages Graph Neural Networks (GNNs) to model multimodal interactions, constructing a unified multimodal graph with textual words and visual objects as nodes and capturing non-local semantic dependencies via message passing. Wang et al. [9] proposed an improved Transformer alignment module based on multi-head self-attention to model global text–image dependencies, achieving fine-grained cross-modal semantic correspondences through hierarchical feature interaction, albeit at increased computational cost. Chen et al. [10] introduced a multimodal alignment fusion network with a contrastive alignment loss to enforce cross-modal consistency; however, the contrastive objective is defined at the sample level rather than the entity level, leaving entity category information underutilized in guiding representation learning. Chen et al. [11] proposed a hierarchical visual–text network that captures multi-scale visual features via pyramid structures, employing visual prefix and hierarchical feature extraction strategies to enrich multimodal representations.

Despite these advances, existing approaches share two fundamental limitations: (1) the absence of explicit modality reliability assessment renders models fragile under noisy or mismatched image–text conditions; and (2) the exclusive reliance on cross-entropy supervision fails to explicitly optimize the discriminability of entity representations in the embedding space. The proposed CUA-MNER addresses both limitations simultaneously through uncertainty-guided modality fusion and fine-grained contrastive learning.

2.2. Contrastive Learning Methods

Contrastive learning has established itself as a foundational paradigm in self-supervised and semi-supervised representation learning, acquiring discriminative features by pulling together positive pairs while pushing apart negative pairs in the embedding space [12]. SimCLR [13] and MoCo [14] demonstrated strong performance on image classification tasks, while CLIP [15] extended the contrastive learning paradigm to large-scale vision–language pretraining. SupCon [16] incorporated supervised signals by treating same-class samples as positive pairs, demonstrating superior performance over cross-entropy training in classification settings. Targeting the visual object bias problem in MNER, Zhang et al. [17] adopted hard sample mining and debiasing contrastive loss strategies to mitigate discrepancies between visual objects and entity types, implicitly aligning text and image representations in a shared semantic space. Yan et al. [18] proposed a fine-grained ranking-based contrastive learning framework that aligns multimodal text–image features at the global level using Top-K ranking mask strategies to construct positive and negative sample pairs.

Directly adapting existing contrastive methods to MNER, however, presents several non-trivial challenges: (1) entity recognition is a sequence labeling task that requires token-level contrastive objectives rather than sample-level ones; (2) a single sentence may contain multiple entities of different types, necessitating independent optimization with explicit entity boundary awareness; and (3) in multimodal settings, contrastive learning must operate over fused representations that incorporate both textual and visual features. To address these issues, the proposed fine-grained contrastive mechanism operates at the token level for each entity-labeled position, supports multi-entity contrast within a single sample, and applies contrastive computation over post-fusion multimodal representations, making it well-suited to the specific demands of MNER. Recent studies have also examined related strategies for improving robustness. SCANNER [19] uses external knowledge to improve recognition of rare or unseen entities, while GMDA [20] applies generative augmentation to enrich training samples in low-resource settings. These studies are complementary to our work: they focus mainly on data or knowledge enhancement, whereas CUA-MNER focuses on modality reliability and token-level representation learning.

2.3. Uncertainty Estimation

Uncertainty estimation in deep learning aims to quantify the reliability of model predictions, providing confidence indicators that are particularly valuable in safety-critical domains such as healthcare, autonomous driving, and finance. Uncertainty is commonly decomposed into epistemic uncertainty, which arises from model knowledge limitations, and aleatoric uncertainty, which arises from inherent noise in the data [21]. This distinction was later formalized for deep learning by Kendall et al. [22]. While epistemic uncertainty can in principle be reduced with additional training data, aleatoric uncertainty is irreducible. Variational inference [23], Bayesian neural networks [24], and deep ensemble methods [25] constitute the primary toolkit for uncertainty quantification in practice.

In multimodal learning, uncertainty estimation takes on particular significance. Unlike unimodal settings, multimodal systems must address the fundamental question of which modality should be trusted at any given input. Liu et al. [5] proposed UAMNer, an uncertainty–aware MNER framework that applies a two-stage label refinement process to suppress noise from irrelevant images.

UAMNer [5] is the most closely related prior work in uncertainty–aware MNER. It suppresses noise from irrelevant images through a two-stage label refinement strategy, where uncertainty is mainly used to improve sample-level supervision and reduce the influence of noisy visual context. In contrast, CUA-MNER applies uncertainty estimation inside the multimodal fusion process. Specifically, it estimates continuous token-level fusion weights conditioned on both textual features and hierarchically aligned visual features. This allows each token to receive a different degree of visual contribution rather than applying a single sample-level correction.

Another difference lies in how the two types of uncertainty are used. In CUA-MNER, the epistemic branch models the expected fusion preference, while the aleatoric branch captures data-level unreliability caused by weak image–text relevance, image ambiguity, or noisy visual content. The two branches are optimized jointly with the CRF decoding objective and the token-level contrastive objective. This positions CUA-MNER as extending uncertainty–aware MNER toward token-level adaptive fusion, building on rather than replacing the sample-level refinement approach of prior work.

3. Method

The overall architecture of CUA-MNER is shown in Figure 2. The model follows a three-stage design: visual–text alignment, uncertainty–aware fusion, and discriminative representation learning. First, RoBERTa and ResNet-50 extract textual and regional visual features. Second, a hierarchical alignment module models token-, phrase-, and sentence-level correspondences between text and image regions. Third, a variational fusion module estimates modality reliability and adaptively combines textual and visual features. Finally, a token-level contrastive objective improves entity representation separability, and a CRF decoder [26] produces the final label sequence.

Figure 2. Overall architecture of the proposed CUA-MNER model.

3.1. Multimodal Feature Extraction

3.1.1. Text Feature Extraction

Text encoding follows a hierarchical strategy. We employ pretrained RoBERTa [27], a robustly optimized variant of BERT that incorporates dynamic masking, removal of the next-sentence prediction objective, larger batch sizes, longer training sequences, and dynamic mask pattern scheduling. To obtain token-level contextual representations:

$$H^{bert}=RoBERTa\left(X_{text}\right)\in {\mathbb{R}}^{L\times d_{bert}}$$

(1)

where $X_{text}$ denotes the input text sequence, $L$ is the sequence length, and $d_{bert}=768$ is RoBERTa’s hidden dimension.

To further capture long-range sequential dependencies, the RoBERTa output is fed into a BiLSTM network [28], which encodes bidirectional context through forward and backward passes:

$$\overrightarrow{h_i}=LSTM\left(h_i^{bert},\overrightarrow{h_{i-1}}\right)$$

(2)

$$\overleftarrow{h_i}=LSTM\left(h_i^{bert},\overleftarrow{h_{i-1}}\right)$$

(3)

$$h_i^{lstm}=\left[\overrightarrow{h_i};\overleftarrow{h_i}\right]\in {\mathbb{R}}^{L\times {2d}_{lstm}}$$

(4)

where $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$ are the forward and backward hidden states at position $i$, respectively, and $h_i^{bert}$ is the corresponding RoBERTa output feature. The concatenated bidirectional states are then projected to a unified feature space via a linear transformation:

$$H_{text}=W_{proj}^{text}{H}^{lstm}+b_{proj}^{text}\in {\mathbb{R}}^{L\times d}$$

(5)

where $d=512$ is the unified feature dimension.

3.1.2. Image Feature Extraction

We employ pretrained ResNet-50 [29] as the vision encoder because its convolutional structure is well suited to localized visual cues in social media images, such as faces, logos, storefronts, and scene objects. Instead of applying global average pooling, we retain the final 7 × 7 convolutional feature map, which provides 49 region-level visual representations. This design preserves spatial locality and supports token-, phrase-, and sentence-level visual–text alignment without requiring additional region proposal networks. Vision Transformers and vision–language pretrained encoders such as CLIP [15] may provide stronger global semantics, but they also introduce additional pretraining assumptions and computational cost:

$$F_{img}=ResNet50\left(X_{img}\right)\in {\mathbb{R}}^{2048\times 7\times 7}$$

(6)

The 2D feature map is then flattened into a sequence:

$$F_{img}^{seq}=Flatten\left(F_{img}\right)\in {\mathbb{R}}^{49\times 2048}$$

(7)

A projection layer followed by layer normalization maps the regional features to the shared feature space, preserving spatial locality for subsequent fine-grained text–image correspondence:

$$H_{img}=LayerNorm\left(W_{proj}^{img}{F}_{img}^{seq}+b_{proj}^{img}\right)\in {\mathbb{R}}^{49\times d}$$

(8)

Prior to cross-modal fusion, learnable positional embeddings and modality-type embeddings are added to both text and image features to encode sequence order and modality identity. We use RoBERTa and ResNet-50 mainly for comparability with existing MNER baselines. The modular design allows the visual and textual encoders to be replaced without changing the fusion and decoding modules.

3.2. Hierarchical Cross-Modal Alignment

The hierarchical alignment module does not require manually annotated text–image correspondences. Token-region and phrase-region alignments are learned through attention between projected text and image features, while sentence-level alignment is learned by applying a Transformer encoder to the concatenated text and image tokens. The entire module is optimized end-to-end using the NER objective, which avoids the need for region-entity annotations.

3.2.1. Token-Region Alignment

At the finest granularity, we compute for each text token $h_i^{text}$ its relevance to each image region via scaled dot-product attention:

$${\alpha }_{ij}={\displaystyle \frac{\mathit{exp}\left(h_i^{text}\cdot h_j^{img}/\sqrt{d}\right)}{{\displaystyle {\sum }_{k=1}^{49}}\mathit{exp}\left(h_i^{text}\cdot h_k^{img}/\sqrt{d}\right)}}$$

(9)

$$c_i^{token}={\displaystyle \sum _{j=1}^{49}}{\alpha }_{ij}{h}_j^{img}$$

(10)

The scaling factor $\sqrt{d}$ prevents the dot products from entering softmax saturation regions when the feature dimension $d$ is large, ensuring stable gradient propagation.

3.2.2. Phrase-Region Alignment

To incorporate local contextual information without performing explicit span classification, we apply a 1D max-pooling operation with kernel size $k=3$ and padding 1 to the text feature sequence, yielding a phrase-level representation at each position:

$$h_i^{phrase}={\mathit{MaxPool}}_3\left(H_{text}\right)[i]$$

(11)

This phrase feature aggregates context from a local window of three consecutive tokens and is subsequently used to attend over image regions:

$${\beta }_{ij}={\displaystyle \frac{\mathit{exp}\left(h_i^{phrase}\cdot h_j^{img}/\sqrt{d}\right)}{{\displaystyle {\sum }_{k=1}^{49}}\mathit{exp}\left(h_i^{phrase}\cdot h_k^{img}/\sqrt{d}\right)}}$$

(12)

$$c_i^{phrase}={\displaystyle \sum _{j=1}^{49}}{\beta }_{ij}{h}_j^{img}$$

(13)

3.2.3. Sentence-Level Alignment

Global semantic interactions are modeled by concatenating text and image features and processing them through a Transformer encoder:

$$H_{concat}=\left[H_{text};H_{img}\right]\in {\mathbb{R}}^{\left(L+49\right)\times d}$$

(14)

$$H_{cross}=\mathit{TransformerEncoder}\left(H_{concat},M_{attn}\right)$$

(15)

where $M_{attn}\in \{0,1{\}}^{\left(L+49)\times (L+49\right)}$ is the attention mask. The text-side outputs are extracted as $h_i^{cross}=H_{cross}\left[i\right]$ for $i=1,\dots ,L$.

The three alignment signals are fused through a learnable linear combination:

$$c_i^{align}=W_1{c}_i^{token}+W_2{c}_i^{phrase}+W_3{h}_i^{cross}$$

(16)

where $W_1,W_2,W_3\in {\mathbb{R}}^{d\times d}$ are learnable weight matrices. This formulation allows the model to implicitly learn the complementary contributions of each granularity: token-level alignment captures local visual details, phrase-level alignment focuses on intermediate semantic chunks, and sentence-level alignment encodes global context. The optimal trade-off across granularities is learned automatically during training.

3.3. Variational Uncertainty–Aware Fusion

3.3.1. Theoretical Foundation

Inspired by variational inference [23], instead of assigning a fixed fusion weight to each modality, we model the fusion weight as a latent variable. For each token position, the model learns a Gaussian posterior conditioned on the textual feature and the aligned visual feature. The posterior mean represents the expected contribution of each modality, while the posterior variance reflects uncertainty in the fusion decision. This design allows the model to rely more on visual cues when image–text correspondence is strong, and to reduce the visual contribution when the image is noisy or weakly related to the text. The posterior is regularized by a Gaussian prior through the KL term in the training objective. Formally, the fusion weight at position $i$ is drawn from a learned posterior:

$$g\sim q\left(g∣h_i^{text},c_i^{align}\right)$$

(17)

where $h_i^{text}$ is the context-aware text feature from Equation (5) and $c_i^{align}$ is the hierarchical alignment feature from Equation (16).

3.3.2. Uncertainty Estimation Network

Epistemic uncertainty, reflecting model parameter uncertainty, is estimated via a dedicated MLP:

$${\mu }_i^{epi},\mathit{log}{\sigma }_i^{epi}={\mathit{MLP}}_{epi}\left(\left[h_i^{text};c_i^{align}\right]\right)$$

(18)

Aleatoric uncertainty, capturing data-level noise such as image blur or image–text semantic mismatch, is estimated by a separate network:

$${\mu }_i^{ale},\mathit{log}{\sigma }_i^{ale}={\mathit{MLP}}_{ale}\left(\left[h_i^{text};c_i^{align}\right]\right)$$

(19)

The two uncertainty types are combined to obtain the parameters of the fusion weight distribution:

$${\mu }_i=\sigma \left({\mu }_i^{epi}\right)$$

(20)

$${\sigma }_i^2=({\sigma }_i^{epi}{)}^2+({\sigma }_i^{ale}{)}^2$$

(21)

where $\sigma$ denotes the sigmoid function and ${\mu }_i,{\sigma }_i\in {\mathbb{R}}^2$ parameterize the fusion weight distribution for the two modalities. The fused mean ${\mu }_i$ is derived solely from the epistemic branch (Equation (20)), as it encodes the model’s expected fusion preference; the aleatoric mean ${\mu }_i^{ale}$ contributes only through the total variance in Equation (21), inflating uncertainty and triggering the conservative fusion strategy described below. The additive decomposition in Equation (21) follows the variance sum rule under the standard assumption that epistemic and aleatoric uncertainties are conditionally independent given the input [24]. This decomposition enables the model to distinguish between “model incapability” (high epistemic uncertainty, addressable through additional training) and “data inadequacy” (high aleatoric uncertainty, requiring down-weighting of the noisy modality).

3.3.3. Adaptive Fusion

During training, fusion weights are sampled via the reparameterization trick [23]:

$$g_i={\mu }_i+ϵ\odot {\sigma }_i,ϵ\sim \mathcal{N}\left(0,I\right)$$

(22)

At inference, the stochastic term is dropped and the posterior mean is used directly:

$$g_i={\mu }_i$$

(23)

The raw sampled vector $\widetilde{g_i}$ is mapped to the probability simplex via softmax, ensuring that the modality weights are normalized. Each projected modality feature is then scaled by its corresponding weight before summation. Unlike hard selection or discrete switching, this soft gating mechanism keeps both modalities active and allows gradients to flow through the fusion process. During training, posterior sampling encourages the model to explore different fusion weights under high uncertainty. At inference, the posterior mean provides a stable deterministic fusion decision. The fused representation is then computed as:

$$h_i^{fused}=g_{i,1}\cdot W_{text}{h}_i^{text}+g_{i,2}\cdot W_{img}{c}_i^{align}$$

(24)

$$h_i^{final}=\mathit{LayerNorm}\left(h_i^{fused}\right)$$

(25)

where $W_{text},W_{img}\in {\mathbb{R}}^{d\times d}$ are learnable projection matrices. Note that $c_i^{align}$ serves as the visual signal proxy here, as it already incorporates cross-modal Transformer output from Equation (15) rather than raw image features.

3.4. Contrastive Learning

3.4.1. Entity Representation Extraction

Each token position $i$ with a non-O label ($y_i>0$) is treated as an individual entity representation, using its final fused feature directly:

$$z_i=h_i^{final}\in {\mathbb{R}}^d$$

(26)

A two-layer projection head then maps these representations to the contrastive learning space:

$$v_i={\mathit{MLP}}_{proj}\left(z_i\right)\in {\mathbb{R}}^{d_{proj}}$$

(27)

where $d_{proj}=128$ and ${\mathit{MLP}}_{proj}:\mathit{ReLU}\left(W_1{z}_i\right)\to W_2$. This projection head is active only during training on gold-annotated entity tokens and is discarded at inference, where only the CRF decoder is used.

3.4.2. Supervised Contrastive Loss

Motivated by supervised contrastive learning [16], we define a token-level contrastive objective for sequence labeling. For an anchor entity token with label ($y_i$), its positive set contains other entity tokens in the same mini-batch with the same BIO-extended label, such as B-PER or I-PER. Tokens with different entity labels are used as negatives. We retain BIO-extended labels instead of merging B- and I-tags into coarse entity types because boundary information is central to sequence labeling.

Merging B- and I-tags would make the contrastive objective less consistent with the CRF decoding target. A B-tag usually indicates the beginning of an entity span and is strongly affected by the transition from the preceding context, whereas an I-tag represents continuation inside an entity span. If B-PER and I-PER were treated as the same contrastive category, the loss would pull together representations that the CRF still needs to distinguish for boundary prediction. This may blur span boundaries and weaken the transition structure learned by the CRF.

Keeping BIO-extended labels also aligns the geometry of the contrastive space with the final label space. Since the decoder predicts the full BIO label set, the contrastive objective regularizes the same label granularity used during inference. This design is especially important for multi-token entities, where correct boundary detection and correct type classification must be learned together. Although merging BIO labels could increase the number of positive samples for rare entity types, it may do so at the cost of boundary precision. We therefore retain the BIO structure and reduce the dominance of non-entity tokens by excluding O tokens from the contrastive objective. O tokens still contribute to the CRF-based NER loss. The supervised contrastive loss is then:

$${\mathcal{L}}_{contrastive}={\displaystyle \sum _{i=1}^{N_e}}{\displaystyle \frac{-1}{\mid \mathcal{P}\left(i\right)\mid }}\sum _{p\in \mathcal{P}\left(i\right)}\mathit{log}{\displaystyle \frac{\mathit{exp}\left(\mathit{sim}\left(v_i,v_p\right)/\tau \right)}{{\sum }_{j\in \mathcal{A}\left(i\right)}\mathit{exp}\left(\mathit{sim}\left(v_i,v_j\right)/\tau \right)}}$$

(28)

where $\mathcal{A}\left(i\right)=\mathcal{P}\left(i\right)\cup \mathcal{N}\left(i\right)$, $\mathit{sim}(u,v)=u^{\top }v/(\parallel u\parallel \parallel v\parallel )$ is cosine similarity, and $\tau =0.07$ is the temperature hyperparameter. This objective encourages intra-class compactness and inter-class separability on the unit hypersphere, pulling together entity representations of the same type while pushing apart those of different types.

3.5. CRF Decoding and Joint Training

3.5.1. CRF Decoding Layer

A CRF layer [26] is employed as the output decoder to enforce label sequence consistency via learned transition probabilities. Given the fused representations $H^{final}$, emission scores are computed as:

$$E=W_{cls}{H}^{final}+b_{cls}\in {\mathbb{R}}^{L\times K}$$

(29)

where $K=9$ is the total number of labels (O, and B-/I- prefixes for PER, LOC, ORG, MISC). The CRF defines the conditional probability of a label sequence $y$ as:

$$P\left(y|x\right)={\displaystyle \frac{exp\left(Score\left(x,y\right)\right)}{{\sum }_{y^{\prime }\in \gamma }exp\left(Score\left(x,y^{\prime }\right)\right)}}$$

(30)

where $\gamma$ denotes the set of all valid label sequences, and the score function is defined as:

$$Score\left(x,y\right)={\sum }_{i=1}^L{v_i^{⊺}E}_{y_i}+{\sum }_{i=1}^{L-1}{T}_{y_i,y_{i+1}}+T_{start,y_1}+T_{yL,end}$$

(31)

Here $E_{i,y_i}$ is the emission score at position $i$ for label $y_i$, $T\in {\mathbb{R}}^{K\times K}$ is the learnable transition matrix, and $T_{i,j}$ scores the transition from label $i$ to label $j$.

3.5.2. Joint Loss Function

The model is trained under a multi-task objective:

$${\mathcal{L}}_{total}={\mathcal{L}}_{NER}+\lambda \cdot {\mathcal{L}}_{contrastive}+\beta \cdot {\mathcal{L}}_{KL}$$

(32)

The NER loss is the negative log-likelihood of the ground-truth label sequence under the CRF:

$${\mathcal{L}}_{NER}=-\mathit{log}P\left(y^{*}∣x\right)$$

(33)

The KL [30] regularization term imposes a Gaussian prior on the fusion weight distributions:

$${\mathcal{L}}_{KL}={\displaystyle \frac{1}{L}}{\displaystyle \sum _{i=1}^L}\mathit{KL}\left(q\left(g_i∣x\right)\parallel \mathcal{N}\left(0.5,{0.1}^2\right)\right)$$

(34)

where the prior mean of 0.5 encodes a default preference for balanced modality fusion, and the standard deviation of 0.1 controls the allowable degree of uncertainty. The KL divergence between two Gaussians is computed in closed form as:

$$\mathit{KL}\left(q\parallel p\right)={\displaystyle \frac{1}{2}}\left[\mathit{log}{\displaystyle \frac{{\sigma }_p^2}{{\sigma }_q^2}}+{\displaystyle \frac{{\sigma }_q^2+{\left({\mu }_q-{\mu }_p\right)}^2}{{\sigma }_p^2}}-1\right]$$

(35)

The loss weights $\lambda =0.3$ and $\beta =0.01$ are determined via grid search on the validation set.

4. Experiments

4.1. Experimental Settings

4.1.1. Datasets

Experiments are conducted on two widely adopted MNER benchmarks: Twitter2015 [1] and Twitter2017 [6], each consisting of image–text pairs annotated with four entity types (PER, LOC, ORG, MISC). Detailed dataset statistics are provided in Table 1.

Table 1. Statistics of the Twitter2015 and Twitter2017 datasets.

Entity Type	Twitter2015			Twitter2017
	Train	Dev	Test	Train	Dev	Test
PER	2217	552	1816	2943	626	621
LOC	2091	522	1697	731	173	178
ORG	928	247	839	1674	375	395
MISC	940	225	726	701	150	157
Total Entities	6176	1546	5078	6049	1324	1351
Tweets	4000	1000	3257	3373	723	723

4.1.2. Implementation Details and Evaluation Metrics

All experiments are implemented in Python 3.10 with PyTorch 2.6.0 + CUDA 11.8 on a Windows 11 system equipped with an NVIDIA GeForce RTX 3050 GPU. The maximum input length is set to 128, and training uses a batch size of 12 with gradient accumulation over 2 steps. Each CUA-MNER variant is trained for up to 30 epochs. The learning rates are 2.0 × 10⁻⁵ for RoBERTa, 1.0 × 10⁻⁵ for ResNet-50, and 5.0 × 10⁻⁴ for newly initialized modules. A warmup ratio of 0.1 is applied, and mixed precision training is enabled to reduce memory consumption and improve training efficiency. Model performance is evaluated using Precision (P), Recall (R), and F1 score, defined as:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(36)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(37)

$$F1={\displaystyle \frac{2PR}{P+R}}$$

(38)

where TP, FP, and FN denote the numbers of true positive, false positive, and false negative entity predictions, respectively.

4.1.3. Baseline Methods

To comprehensively evaluate CUA-MNER, we compare it against baselines spanning four methodological categories, enabling systematic assessment of the contributions of visual modality, fusion strategy, contrastive learning, and large language model capacity, respectively.

• Text-only methods. These baselines establish the performance ceiling of purely linguistic systems and serve as a reference for quantifying the marginal value of visual information:

CNN-BiLSTM-CRF [31] extracts character-level features via CNN, concatenates them with word embeddings, and feeds the combined representations into a BiLSTM-CRF for sequence labeling.

HBiLSTM-CRF [32] processes character and word embeddings simultaneously through a hierarchical BiLSTM to construct multi-level representations, with CRF decoding for label sequence optimization.

BERT-CRF [2] uses BERT to generate contextualized token representations, followed by a CRF decoder to model label dependencies.

2.

Attention-Based and Graph-Based Multimodal Fusion Methods. These methods represent the mainstream paradigm of cross-modal fusion through attention or graph mechanisms, but without explicit modality reliability estimation:

OCSGA [33] extracts image object labels via Mask R-CNN, converts them into word embeddings, and applies dense co-attention to associate visual objects with textual entities.

AdaCoAtt [1] introduces an adaptive co-attention mechanism on top of the CNN-BiLSTM-CRF backbone, employing gating and filtering operations for cross-modal feature fusion before CRF decoding.

UMT [7] adopts a unified Transformer architecture to fuse text and image features, with an entity span detection auxiliary module to guide named entity boundary prediction.

UMGF [8] extends UMT by constructing a multimodal graph that fuses text and visual information via graph convolutional networks, using targeted visual guidance to focus on task-relevant image regions.

MAF [34] applies a cross-modal matching module to filter irrelevant visual content and aligns text entities with corresponding image regions through a cross-modal alignment module.

HVPNeT [10] employs visual prefix and hierarchical feature extraction strategies to integrate multi-scale visual features into text representations.

3.

Contrastive Learning and Uncertainty–Aware Methods. These methods are most directly comparable to CUA-MNER, as they share one or more of its core technical components:

DebiasCL [17] applies debiasing contrastive learning to enhance cross-modal alignment and mitigate visual object bias in MNER.

TGF [35] constructs a multimodal graph from visual and textual tokens, applies multi-layer Transformer fusion, and combines global and local alignment through multi-objective contrastive learning.

BGA-MNER [36] realizes implicit entity-object alignment via a bidirectional generation process and employs a staged refined context sampler to filter irrelevant information and bridge the cross-modal semantic gap.

FRCL [18] aligns multimodal semantic features at the global level through contrastive learning, leveraging a Top-K ranking mask strategy to learn fine-grained inter-modal interaction representations.

BRCA [37] introduces an adaptive mixed image augmentation strategy to generate augmented training samples, addressing image–text mismatch.

4.

Large Language Model-Based Methods. These baselines represent the emerging paradigm of applying generative LLMs to structured prediction tasks, and are included to contextualize the relative strengths of discriminative versus generative approaches:

Vanilla GPT [38] refers to GPT-3.5-Turbo applied directly to the MNER task without task-specific prompting.

Prompt GPT [38] supplements GPT-3.5-Turbo with a multimodal similar-example perception module and prompt engineering heuristics to provide refined auxiliary knowledge.

GPT-4 [39] denotes the GPT-4 model applied to MNER, offering improved language understanding over GPT-3.5-Turbo.

Visual ChatGPT [40] integrates GPT’s generative capabilities with visual understanding for multimodal named entity recognition.

4.2. Results Analysis

4.2.1. Main Results

Table 2 reports the overall and per-entity-type results on Twitter2015 and Twitter2017. CUA-MNER achieves F1 scores of 76.97% and 89.66% on the two datasets, respectively. Compared with the strongest reported baseline in Table 2, the improvement is 0.66 F1 points on Twitter2015 and 1.95 F1 points on Twitter2017. The smaller margin on Twitter2015 suggests that this dataset remains challenging and that the gain should be interpreted with appropriate caution, while the larger margin on Twitter2017 shows a clearer advantage of the proposed framework.

Table 2. Overall and per-entity-type performance on Twitter2015 and Twitter2017 (%).

Methods	Twitter2015							Twitter2017
	Single Type (F1)				Overall			Single Type (F1)				Overall
	PER	LOC	ORG	MISC	P	R	F1	PER	LOC	ORG	MISC	P	R	F1
CNN-BiLSTM-CRF 1	80.86	75.39	47.77	32.61	66.24	68.09	67.15	87.99	77.44	74.02	60.82	80.00	78.76	79.37
HBiLSTM-CRF 1	82.34	76.83	51.59	32.52	70.32	68.05	69.17	87.91	78.57	76.67	59.32	82.69	78.16	80.37
BERT-CRF 1	84.74	80.51	60.27	37.29	69.22	74.59	71.81	90.25	83.05	81.13	62.21	83.32	83.57	83.44
OCSGA 1	84.68	79.95	56.64	39.47	74.71	71.21	72.92	-	-	-	-	-	-	-
AdaCoAtt 1	81.98	78.95	53.07	34.02	72.75	68.74	70.69	89.63	77.46	79.24	62.77	84.16	80.24	82.15
UMT 1	85.24	81.58	63.03	39.45	71.67	75.23	73.41	91.56	84.73	82.24	70.10	85.28	85.34	85.31
UMGF 1	84.26	83.17	62.45	42.42	74.49	74.85	74.85	91.92	85.22	83.13	69.83	86.54	84.50	85.51
MAF 1	84.67	81.18	63.35	41.82	71.86	75.10	73.42	91.51	85.80	85.10	68.79	86.13	86.38	86.25
HVPNeT 1	-	-	-	-	73.87	76.82	75.32	-	-	-	-	85.84	87.93	86.87
DebiasCL 1	85.97	81.84	64.02	43.38	74.45	76.13	75.28	93.46	84.15	84.42	67.88	87.59	86.11	86.84
TGF 1	85.18	83.06	63.42	42.48	73.88	75.98	74.91	93.84	86.86	84.89	70.70	88.42	86.96	87.70
BGA-MNER 1	86.80	83.62	63.60	42.65	78.60	74.16	76.31	93.71	85.55	85.71	71.05	87.71	87.71	87.71
FRCL 1	86.38	82.51	63.84	42.72	74.74	76.21	75.47	93.92	85.15	85.00	67.70	86.90	87.86	87.38
BRCA 1	-	-	-	-	75.08	76.21	75.62	-	-	-	-	87.23	88.00	87.62
Vanilla GPT 1	-	-	-	-	42.96	75.37	54.73	-	-	-	-	52.19	75.03	61.56
Prompt GPT (N = 10) 1	-	-	-	-	58.57	74.07	65.41	-	-	-	-	72.90	77.65	75.20
GPT4 1	-	-	-	-	-	-	57.98	-	-	-	-	-	-	66.61
Visual ChatGPT 1	-	-	-	-	-	-	41.45	-	-	-	-	-	-	47.24
CUA-MNER (ours)	86.38	82.94	66.95	47.53	75.66	78.33	76.97	94.48	89.33	87.66	76.10	89.17	90.16	89.66

¹ Bold indicates best results. “-” indicates results not reported in original papers.

Compared with text-only methods, CUA-MNER brings consistent improvements. For example, it outperforms BERT-CRF by 5.16 F1 points on Twitter2015 and 6.22 F1 points on Twitter2017. This suggests that visual information can provide useful complementary evidence for MNER. At the same time, not all multimodal baselines outperform text-only models, which indicates that visual features are helpful only when they are properly aligned and filtered. This observation supports the motivation of combining hierarchical alignment with uncertainty–aware fusion.

The per-entity results further show that the gains are not uniform across entity types. On Twitter2015, CUA-MNER performs particularly well on ORG and MISC, reaching 66.95% and 47.53% F1. These categories are often more dependent on external context, such as logos, products, events, and visually grounded social media references. On Twitter2017, CUA-MNER obtains the best F1 scores across all four entity types, with especially clear improvements on MISC and LOC. In contrast, PER and LOC on Twitter2015 show smaller gains, partly because many personal names and location mentions can already be recognized from textual context.

We also note that most baseline results in Table 2 are taken from the corresponding original papers. This allows comparison with a broad set of representative methods, but it does not constitute a fully controlled reimplementation under identical preprocessing, optimization, and checkpoint selection settings. Therefore, the reported comparison should be interpreted as a comparison against published results rather than a completely unified benchmark.

To provide additional reliability evidence where prediction files are available, we further conduct a paired bootstrap significance test between CUA-MNER and our reproduced UMGF baseline on Twitter2015. After label-name normalization, 3187 test instances have identical gold label sequences in both prediction files. Using 10,000 bootstrap resamples, CUA-MNER obtains 77.36% F1 on this aligned subset, while UMGF obtains 74.24% F1. The F1 difference is 3.13 points, with a 95% confidence interval of [2.16, 4.08] points and a one-sided bootstrap p-value below 0.001. This result suggests that the improvement over the reproduced UMGF baseline is statistically reliable under test-set resampling, although it does not replace multi-seed training variance analysis.

4.2.2. Ablation Study

To quantify the contribution of each component, we conduct a systematic ablation study by removing or modifying individual modules of CUA-MNER. The results are summarized in Table 3. The w/o Alignment variant removes the hierarchical cross-modal alignment module and uses global visual features only. The w/o C variant removes the token-level contrastive learning objective. The w/o UA variant replaces uncertainty–aware fusion with conventional gated fusion and removes the KL regularization term. The w/o CUA variant removes both the contrastive objective and the uncertainty–aware fusion module.

Table 3. Ablation results on Twitter2015 and Twitter2017 (%).

Model	Twitter2015			Twitter2017
	P	R	F1	P	R	F1
CUA-MNER	75.66	78.33	76.97	89.17	90.16	89.66
w/o Alignment	75.56	77.90	76.71	87.96	89.19	88.57
w/o C	75.01	77.29	76.13	88.54	88.08	88.31
w/o UA	73.63	77.78	75.65	87.56	88.08	87.82
w/o CUA	76.08	74.49	75.27	87.47	87.86	87.67
w/o KL	75.72	76.10	75.91	87.67	88.38	88.02
Epi-only	76.14	75.89	76.01	87.13	88.23	87.68
Ale-only	75.11	77.21	76.14	86.78	88.90	87.82

Bold indicates best results.

To further examine the uncertainty–aware fusion mechanism, we add three branch-level ablation variants. The w/o KL variant keeps the original dual-branch uncertainty structure but removes the KL regularization term by setting (β = 0). This variant tests whether the Gaussian prior contributes to stable modality weighting. The Epi-only variant retains only the epistemic branch, while the Ale-only variant retains only the aleatoric branch. These two variants are designed to examine whether the two uncertainty branches provide complementary signals rather than redundant estimates.

Table 3 reports the ablation results on Twitter2015 and Twitter2017. Removing the hierarchical alignment module leads to F1 drops of 0.26 and 1.09 points, respectively, showing that multi-granular visual–text correspondence is still useful even with strong textual representations. Removing contrastive learning decreases F1 by 0.84 points on Twitter2015 and 1.35 points on Twitter2017, which indicates that token-level contrastive supervision improves the separability of entity representations. The largest drop among the original component-level variants comes from removing uncertainty–aware fusion, with F1 decreasing by 1.32 and 1.84 points. This suggests that adaptive modality reliability estimation is an important part of CUA-MNER.

The newly added branch-level variants further clarify the uncertainty module. When the KL regularization term is removed, F1 decreases from 76.97% to 75.91% on Twitter2015 and from 89.66% to 88.02% on Twitter2017. This suggests that the Gaussian prior is not merely a regularization formality but plays a substantive role in stabilizing the learned fusion distribution. The Epi-only and Ale-only variants also perform below the full model on both datasets. Epi-only obtains 76.01% and 87.68% F1, while Ale-only obtains 76.14% and 87.82% F1 on Twitter2015 and Twitter2017, respectively. The small but consistent gap between these two variants suggests that the two branches do not behave identically. At the same time, neither single-branch variant matches the full model, indicating that the best performance comes from combining epistemic information for the expected fusion preference with aleatoric information for data-level visual noise. Overall, the ablation results support the complementary roles of alignment, contrastive learning, KL regularization, and dual-branch uncertainty modeling.

4.2.3. Fusion Weight Analysis

Table 4 reports the average modality fusion weights per entity type on the Twitter2015 and Twitter2017 test sets. Figure 3 further visualizes the distribution of image fusion weights using box plots with jittered sample points, showing both the overall distribution and sample-level variation.

Table 4. Average modality fusion weights by entity type.

Entity Type	Twitter2015		Twitter2017
	Text	Image	Text	Image
PER	0.725	0.275	0.515	0.485
LOC	0.720	0.280	0.511	0.489
ORG	0.699	0.301	0.504	0.496
MISC	0.669	0.331	0.512	0.488

Figure 3. Distribution of image fusion weights across entity types on Twitter2015.

The fusion weights reveal clear dataset-level differences. On Twitter2015, CUA-MNER assigns more weight to text, with an average text weight of 0.703. In contrast, the weights on Twitter2017 are closer to balanced. This pattern suggests that the model adjusts its fusion strategy according to the usefulness of visual information. The lower image weights on Twitter2015 are consistent with the higher level of image–text mismatch commonly observed in social media data, whereas the more balanced weights on Twitter2017 indicate that visual cues are more frequently informative. Among the entity types, MISC receives the highest image weight on Twitter2015, suggesting that heterogeneous entities may benefit more from additional visual context.

Figure 3 and Figure 4 further supports these observations. On Twitter2015, the image weights are generally low and concentrated, especially for PER, LOC, and ORG, indicating a text-dominant fusion strategy. MISC shows a wider spread, which is consistent with its more heterogeneous semantics. On Twitter2017, the distributions are closer to 0.5 across entity types, suggesting that visual information is more consistently useful. These patterns indicate that CUA-MNER adjusts fusion weights not only across datasets but also across entity categories.

Figure 4. Distribution of image fusion weights across entity types on Twitter2017.

4.2.4. Uncertainty Validity Analysis

To further examine whether the uncertainty–aware fusion module responds to unreliable visual input, we conduct a controlled image–text mismatch analysis. During evaluation, a proportion of input images is randomly replaced with images from other test samples, while the text and labels remain unchanged. This setting creates additional visual noise without changing the textual entity mentions. We evaluate mismatch ratios of 0.0, 0.3, and 0.6, and report the average image fusion weight, average aleatoric uncertainty, and F1 score in Table 5.

Table 5. Uncertainty behavior under controlled image–text mismatch.

Dataset	Mismatch Ratio	Avg. Image Gate	Avg. Aleatoric Uncertainty	F1
Twitter2015	0.0	0.2897	0.00000152	76.97%
	0.3	0.2898	0.00000146	76.91%
	0.6	0.2893	0.00000152	76.84%
Twitter2017	0.0	0.4927	0.00000004	89.66%
	0.3	0.4927	0.00000004	89.66%
	0.6	0.4927	0.00000004	89.54%

The results show that increasing image–text mismatch causes a small but consistent performance decrease. On Twitter2015, F1 drops from 76.97% to 76.84% as the mismatch ratio increases from 0.0 to 0.6. On Twitter2017, F1 decreases from 89.66% to 89.54%. This suggests that visual information still contributes to recognition, but the effect of random image replacement is limited, especially when the text alone provides strong entity cues.

The fusion statistics are relatively stable across mismatch ratios. The average image gate on Twitter2015 remains around 0.29, while that on Twitter2017 remains around 0.49. The average aleatoric uncertainty also changes only slightly across ratios. One additional observation is that the absolute aleatoric uncertainty values on Twitter2017 (around 4 × 10⁻⁸) are substantially smaller than those on Twitter2015 (around 1.5 × 10⁻⁶). This difference likely reflects the stronger and more consistent image–text alignment in Twitter2017, where the model has less need to signal data-level noise through the aleatoric branch. It does not indicate a malfunction of the branch; rather, it shows that the aleatoric uncertainty estimate is conditioned on the actual input quality of each dataset. Therefore, these results should not be interpreted as evidence of strong uncertainty calibration. Rather, they indicate that the learned fusion strategy is conservative under noisy visual conditions and does not sharply increase visual reliance when images are randomly mismatched.

Overall, the controlled mismatch experiment provides a useful diagnostic complement to the ablation study. It shows that CUA-MNER is only mildly affected by additional visual mismatch, but it also reveals that the current aleatoric uncertainty estimate is not highly sensitive to all forms of image–text inconsistency. Incorporating explicit image–text relevance labels or contrastive alignment signals during training could strengthen the uncertainty module’s sensitivity to mismatched inputs.

4.2.5. Case Study and Error Analysis

To complement the quantitative results, we analyze several representative cases. Figure 5 shows examples where CUA-MNER benefits from visual context and correctly identifies entities that are missed or confused by the baselines. Figure 6 further presents three failure cases, selected from the model’s prediction errors, to show where the framework still struggles.

Figure 5. Qualitative examples of entity prediction results. The ‘Text’ row presents several representative samples from the Twitter2015 test set and their manually labeled entities, while the bottom three rows show the predicted entities by different methods on these samples (‘√’ = correct, ‘×’ = incorrect).

Figure 6. Representative failure cases involving weak visual relevance, MISC ambiguity, and ORG–LOC confusion. The ‘Text’ row presents several representative samples from the Twitter2015 test set and their manually labeled entities, while the bottom row show the predicted entities by CUA-MNER on these samples (‘√’ = correct, ‘×’ = incorrect).

As shown in Figure 5, in case (a), the text-only BERT-CRF model fails to identify “Real Madrid,” whereas CUA-MNER correctly recognizes it as an ORG entity. The accompanying image contains team-related visual cues, which provide useful evidence for organizational disambiguation. In case (b), BERT-CRF misses both entities, and UMGF confuses the entity types. CUA-MNER correctly identifies the PER and LOC entities by combining phrase-level textual context with visual scene information. This case illustrates the benefit of combining hierarchical alignment with contrastive representation learning. In case (c), the sentence contains nested and semantically related location expressions. CUA-MNER correctly identifies both LOC entities, suggesting that sentence-level alignment and contrastive learning help reduce confusion between geographically and organizationally related mentions.

As shown in Figure 6, in case (a), the model misses “#Stoptober” as an ORG entity and incorrectly predicts “Trafalgar Square” as LOC. The image contains a prominent “stop” sign, which is related to the campaign keyword but does not clearly indicate the organizational nature of the hashtag. This example shows that visually salient but semantically incomplete cues may still mislead the model. In case (b), “Mothers Rugby jerseys” is labeled as MISC, and “University of Mary Washington” is labeled as LOC, but both are predicted as ORG-related mentions. This reflects a common ambiguity in social media MNER: team names, sports-related expressions, and university names often carry organization-like meanings, even when their gold labels depend on the annotation context. In case (c), the model predicts “Chicago Public School” as ORG and “Graeme Stewart Elementary” as LOC, reversing the intended labels. The image of a school building provides useful scene information, but it does not resolve the fine distinction between an institution and a place. This type of ORG–LOC confusion remains difficult when textual and visual cues support both interpretations.

These cases suggest that the remaining errors are mainly caused by weak visual relevance, heterogeneous MISC semantics, and overlapping ORG–LOC boundaries. They also indicate that stronger image–text relevance estimation and finer entity-type modeling may further improve CUA-MNER.

4.2.6. Computational Complexity Analysis

Table 6 compares the computational cost of CUA-MNER with two representative multimodal baselines, UMT and UMGF. The comparison focuses on parameter size, training time per epoch, and inference throughput under the same local hardware setting. CUA-MNER contains 162.76 M parameters, which is larger than UMT but smaller than UMGF. This additional cost mainly comes from the hierarchical alignment module and the uncertainty–aware fusion network.

Table 6. Comparison of parameter count and runtime efficiency.

Method	Backbone	Params (M)	Training Time/Epoch (s)	Inference Speed (Samples/s)
UMT	BERT + ResNet152	148.10	783.18	45.97
UMGF	BERT + ResNet152	191.32	~5600 1	21.52
CUA-MNER	RoBERTa + ResNet50 + UA + CL	162.76	468.33	65.81

¹ Estimated from the observed average step time after runtime stabilization.

In terms of runtime, CUA-MNER requires 468.33 s per training epoch, which is lower than UMT (783.18 s) and substantially lower than the estimated training time of UMGF. Its inference speed reaches 65.81 samples/s, compared with 45.97 samples/s for UMT and 21.52 samples/s for UMGF. The higher per-epoch training time of UMT relative to CUA-MNER, despite its smaller parameter count, is likely attributable to the computational overhead of its auxiliary span detection module and the repeated self-attention operations in its unified Transformer, which involve more sequential dependency under the current implementation. CUA-MNER benefits from mixed precision training and the fact that the contrastive projection head is used only during training and is removed during inference, which reduces test-time overhead.

Overall, CUA-MNER is not a lightweight model, but its computational cost remains within a practical range for offline or batch-oriented MNER. The comparison suggests that the performance gains are achieved with a moderate complexity increase rather than by simply adopting a much heavier architecture.

5. Conclusions

This paper presents CUA-MNER, an end-to-end framework for multimodal named entity recognition, and evaluates it through ablation studies, statistical significance testing, controlled mismatch analysis, and failure case examination. The model combines hierarchical cross-modal alignment, variational uncertainty–aware fusion, and token-level contrastive learning to address three challenges in MNER: weak visual–text alignment, unreliable visual inputs, and limited entity representation separability. Experiments on Twitter2015 and Twitter2017 show that CUA-MNER obtains competitive F1 scores among the methods compared in Table 2. Ablation studies further indicate that the three components provide complementary benefits: alignment improves visual–text correspondence, uncertainty–aware fusion reduces the effect of noisy images, and contrastive learning improves representation geometry.

Despite these results, several limitations remain. First, the experiments are conducted on Twitter2015 and Twitter2017, which are widely used but relatively dated MNER benchmarks. These datasets provide a standard basis for comparison, but they cannot fully represent the diversity of current multimodal social media content. Posts from platforms such as Weibo, Instagram, TikTok, or multimodal news streams may differ in language, visual style, entity distribution, and image–text relationship patterns. In particular, memes, screenshots, infographics, and visually edited content may introduce forms of multimodal noise that are not fully covered by the two Twitter datasets. Therefore, the robustness and generalizability claims of this work should be understood within the Twitter MNER setting, and evaluation on newer, multilingual, and cross-platform datasets is an important direction for future work.

Second, CUA-MNER introduces additional computational cost compared with lighter text-only or simple-fusion models. Although the contrastive projection head is used only during training, the hierarchical alignment module and uncertainty–aware fusion module remain active during inference. This makes the model more suitable for offline or batch-oriented information extraction than for strict real-time applications. Future work may explore parameter-efficient alignment modules, lightweight visual encoders, or knowledge distillation to reduce deployment cost.

Third, the current implementation uses ResNet-50 as the visual backbone for comparability with existing MNER systems. Stronger vision–language pretrained encoders, such as CLIP or BLIP-style [41] models, may provide richer semantic visual representations, but they may also introduce additional pretraining assumptions and computational overhead. Future research should examine whether such encoders improve MNER performance under controlled settings and whether their gains justify the added complexity.