MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media

Abstract

The goal of the Multimodal Named Entity Recognition (MNER) job is to identify and classify named entities by combining various data modalities (such as text and images) and assigning them to specified categories. The growing prevalence of multimodal social media posts has spurred heightened interest in MNER, particularly due to its pivotal role in applications ranging from intention comprehension to personalized user recommendations. In the MNER task, the inconsistency between image information and text information and the difficulty of fully utilizing the image information to complement the text information are the two main difficulties currently faced. In order to solve these problems, this study proposes a Multilevel Predictive Cross-Fusion Network (MPCFN) approach for Multimodal Named Entity Recognition. First, textual features are extracted using BERT and visual features are extracted using ResNet, then irrelevant information in the image is filtered using the Correlation Prediction Gate. Second, the hierarchy of visual features received by each Transformer block is controlled by the Dynamic Gate and aligned between image and textual features using the Cross-Fusion Module to align the image and text features. Finally, the hidden layer representation is fed into the CRF layer optimized for decoding using Flooding. Through experiments on TWITTER-2015, TWITTER-2017, and WuKong datasets, our method achieves F1 scores of 76.74%, 87.61%, and 82.35%, outperforming the existing mainstream state-of-the-art models and proving the effectiveness and superiority of our method.

1. Introduction

Named Entity Recognition (NER) is a crucial information extraction method that seeks to recognize and categorize textual entities with particular meanings, such as the names of individuals, locations, businesses, dates, and times, among others [1,2]. By recognizing named entities in unstructured free-form text, NER plays an important role in several domains, including information retrieval, relational extraction, question and answer systems, machine translation, and text categorization [3]. Currently, mainstream text-based NER algorithms produce considerable results when processing well-formed text. They usually use CNN, LSTM, or Transformer as encoders to learn the contextual representation of input words and decode them with softmax or CRF [4,5].

However, using traditional NER methods for named entity recognition of text on social media like Twitter and Facebook is often less effective due to the fact that Twitter tweets usually contain a large number of spelling mistakes, abbreviations, slang, emoticons, and other non-standard languages [6,7]. These noises make it difficult to accurately recognize and classify named entities by traditional NER methods, which usually rely on well-formed text. In addition, the character limitation of Twitter tweets results in very limited contextual information per tweet, and much of the textual content can only be understood in combination with visual context [8].

To address the challenges faced by traditional NER methods in complex textual environments, MNER has been proposed. The objective of MNER is to recognize and classify entities in post by utilizing associated images. Existing MNER methods are broadly categorized into two types: one class of methods inputs both the text and the entire image, encoding the implicit representation of each word together [9,10]; the other class aligns text vectors with visual object features to achieve more comprehensive semantic representations of words [6,11,12,13,14,15]. All these approaches demonstrate that augmenting linguistic representations with visual information enables MNER to achieve outperforms in comparison with traditional NER methods in NER tasks. Figure 1 provides two exemplars of MNER. Specifically, Figure 1a,b represent examples of MNER in English and Chinese, respectively. The objective of MNER is to accurately discern the correct entity types based on the provided image and text. For instance, in Figure 1a, it is expected to recognize that NFL, Patrick Willis, and Silicon Valley belong to the category of organization names (ORG type), person names (PER type), and place names (LOC type), respectively.

Figure 1. Two examples for MNER. (a,b) represent examples of MNER in English and Chinese, respectively. The text in different colors represents different entity categories.

Although prior methodologies have proved the efficacy of MNER, two major problems remain. First, there may be discrepancies between the text and accompanying images, with erroneous image information potentially disrupting entity recognition [16,17]. Second, employing global picture representations ignores the fine semantic alignment between images and text. Additionally, relying solely on object-level features can be problematic, as some images may contain misleading objects or lack explicit objects altogether.

To tackle these issues, we introduce a new MNER method that leverages a hierarchical visual prefix fusion network. Our approach begins with the extraction of unimodal features from text and images using BERT and ResNet, respectively. Subsequently, we employ a Correlation Prediction Gate to screen out irrelevant information in the image, applying adaptive weights to visual features. These weights are dynamically adjusted according to the relationship between the image and the text. Furthermore, a Dynamic Gate mechanism controls the hierarchical processing of visual features within each Transformer block, enabling each block to learn different levels of text and visual information. Finally, the hidden layer representations are fed into a CRF layer, which is optimized using the Flooding method for decoding. We evaluate our method on two public English datasets and one Chinese dataset, and the experimental results demonstrate the effectiveness and advancements of our proposed approach.

The contributions of this paper are summarized as follows:

(1)

We present a Multilevel Predictive Cross-Fusion Network model to enhance MNER’s performance on social media. This model predicts the correlation between text and images to mitigate irrelevant image interference and employs cross-fusion instead of visual prefix fusion for integrating multimodal information.

(2)

To address overfitting associated with model complexity, this study employs a technique known as Flooding to optimize the Conditional Random Field (CRF) layer. This approach effectively mitigates overfitting by modifying the CRF layer’s loss function, thereby enhancing the model’s generalization capability. By incorporating Flooding technology, we further improve the model’s robustness in managing complex sequence annotation tasks.

(3)

Through extensive experiments and analysis, we show that our model performs competitively compared to existing state-of-the-art models.

In Section 2, we provide a comprehensive review of prior research on NER and MNER. We first present a detailed overview of traditional NER methods and their limitations, followed by an analysis of the advantages of multimodal approaches and the findings of previous studies. In Section 3, we provide a detailed description of our proposed MNER method, focusing on the implementation details of correlation prediction between image–text pairs and the multimodal information fusion strategy. Section 4 describes the datasets and evaluation criteria employed in this work, as well as providing a thorough analysis of the experimental results. Finally, Section 5 summarizes the key findings of the research and suggests potential directions for future work.

2. Related Work

NER plays a vital role in natural language processing, with applications in areas such as knowledge graphs, social media analysis, and financial market forecasting. Early research on NER primarily focused on rule-based approaches, where domain experts manually crafted rules and employed pattern-matching techniques to identify entities within posts. For instance, Kim et al. (2000) [3] introduced the Brill rule inference method for processing speech input. However, rule-based NER methods exhibit several limitations, including difficulties in handling complex contexts, high maintenance costs, limited generalizability, and challenges in addressing semantic complexity.

Traditional machine learning methods for NER include the Maximum Entropy Model (MEM) [18], Hidden Markov Model (HMM) [19], Support Vector Machine (SVM) [20], and Conditional Random Fields (CRFs) [21]. These methods rely on manually designed features to extract entities from text. For example, Borthwick et al. (1998) [22] used the MEM method for entity recognition; McCallum et al. (2003) [23] proposed the application of a CRF model in NER. However, these approaches have high demands for feature selection, necessitating the extraction of various task-specific features from the text, and they struggle with handling long-distance dependencies effectively.

To address the limitations of traditional methods, deep learning techniques have been employed to enhance NER accuracy [24]. Researchers have utilized neural networks such as Bidirectional Long Short-Term Memory (BiLSTM) [25] and Convolutional Neural Networks (CNN) [26] to encode words into vectors, subsequently using decoders like CRF [21], LSTM [27], and Transformers [28] to generate labels. The introduction of pre-trained language models like BERT [4] and GPT [29,30] has significantly enhanced NER performance. These models capture comprehensive language features through pre-training on extensive corpora and can be fine-tuned for specific NER tasks. Nevertheless, they still face challenges in comprehending multimodal data, such as informal and ambiguous text found in social media posts.

Previous Research on Multimodal NER

The MNER model has been proposed to improve text-based NER models while also leveraging multimodal data. The MNER task seeks to identify and classify named entities within text that is paired with images. This approach emphasizes aligning textual elements with corresponding regions in images and integrating both textual and visual contexts to improve entity recognition performance.

Some MNER approaches directly utilize global images [31]. For instance, Lu et al. (2018) [9] and Moon et al. (2018) [10] use modal attention networks to combine textual and visual information in the BiLSTM layer. However, it is important to note that not all visual regions in an image contribute equally to enhancing the accuracy of model predictions. To solve this problem, some researchers make use of convolutional architectures (such as ResNet) to evenly segment the entire image into multiple visual regions, and then use multimodal interaction modules with attention mechanisms to select visual regions related to the text. For example, Lu et al. (2018) [9] used a pre-trained ResNet model to extract visual areas, which were subsequently merged into word embeddings using a visual attention algorithm. Zhang et al. (2018) [32] employed an adaptive co-attention network to merge text embeddings with visual region representations. Yu et al. (2020) [8] developed a multimodal interaction module that captures both image-perceived word representations and word-perceived visual representations while also using a pure text entity span detection module as an auxiliary component to reduce visual bias. Chen et al. (2021) [33] obtained the final multimodal representation through attention-guided visual layers using an external knowledge database. Due to the large proportion of unrelated text–image pairs in the dataset, Sun et al. (2021) [34] incorporated text–image relationship classification as a sub-task to evaluate the value of image features and used an improved BERT encoder to collect modality fusion information. Additionally, tackling the challenge of precise semantic matching between objects in images and words in post, Liu et al. (2024) [35] carried out cross-modal semantic interactions between text and vision at different visual granularities and enhanced word representations in post via semantic enrichment.

To more effectively mitigate the influence of irrelevant parts of an image, some researchers employ object detection techniques, such as Mask R-CNN, to extract visual objects from the entire image and treat these objects as local visual information interacting with text. Mask R-CNN was used by Wu et al. (2020) [6] to detect objects. The top k items were embedded into vectors and integrated with text features using dense common attention modules. To address the issue of potentially missing relevant image information caused by relying solely on local images, Chen et al. (2022) [11] incorporated both local and global images obtained through object detection into their model, employing a hierarchical network to fuse text and image information. The interactions between text and image representations are difficult to model; thus, they are frequently trained independently within each modality and out of alignment inside a single space. Wang et al. (2022) [12] and Wang et al. (2022) [36] addressed this issue by utilizing external databases to convert images into text, thereby enhancing the effectiveness of text-based pre-trained models. Due to multigranular cross-modal representation learning and cross-modal semantic interaction between text and vision at different visual granularities, Liu et al. (2024) [35] obtained effective multimodal guided representations for each word, experimentally demonstrating that the results on TWITTER-2015 and TWITTER-2017 outperform the current performance.

In summary, although existing research on Multimodal Named Entity Recognition (MNER) has made significant progress in multimodal fusion, two critical issues still need to be addressed urgently:

(1)

Unbalanced feature alignment granularity: Most models (e.g., RpBERT [34], HVPNeT [11]) either rely solely on global image features or overemphasize local object features. They fail to achieve hierarchical collaboration and dynamic adaptation of “global–local” visual features, which limits the semantic matching accuracy between textual entities and visual regions.

(2)

Insufficient visual noise-filtering capability: Existing methods (e.g., MGCMT [35]) mostly use attention mechanisms in the late stage of feature fusion to select effective visual information, lacking early targeted noise-filtering designs. As a result, irrelevant visual noise tends to propagate continuously during model training, increasing the risk of entity misrecognition.

To address these issues, this paper proposes a Multilevel Predictive Cross-Fusion Network (MPCFN). Through an innovative architecture of “early noise filtering based on a Correlation Prediction Gate—dynamic gating for hierarchical visual feature allocation—cross-fusion for bidirectional modal alignment”, this network specifically addresses the aforementioned research gaps.

3. Methodology

We start this section by describing the MNER problem and provide an overview of our suggested approach. Then, we give a thorough explanation of the precise implementation procedure of our system, utilizing the picture and sequence shown in Figure 2 as an example.

Figure 2. Overall framework of our proposed methodology: Stage 1: Extract text features by using BERT and visual features by using ResNet. Stage 2: Utilize a Correlation Prediction Gate to predict the relationships between text and image, and apply adaptive weights to the visual features based on these predictions. Stage 3: Predict a normalized vector $G$ to control the level of visual features each block receives via the Dynamic Gate. Stage 4: Aligning image and text features through cross-fusion. Stage 5: A CRF layer optimized with Flooding receives the representation from the hidden layer for decoding.

Task Definition: Given an input pair consisting of a text sentence $T$ and an associated image $V$, the goal of MNER is to extract a set of entities from $T$ and classify each extracted entity into one of the pre-defined types. Similar to most existing MNER approaches, we frame this task as a sequence-labeling problem. Let $T=\left(t_1,t_2,\dots ,t_n\right)$ denote the sequence of input tokens, and $y=\left(y_1,y_2,\dots ,y_n\right)$ represent the corresponding sequence of labels.

3.1. Overall Framework

The overall architecture of our model is shown in Figure 2. For clarity, the model is divided into 5 main components: (1) text and visual feature extraction, (2) correlation prediction, (3) the Dynamic Gate, (4) cross-fusion, (5) CRF and Flooding layer.

Firstly, we employ the BERT model for comprehensive feature extraction from the text data. Utilizing the visual grounding toolkit, we extract local object information from the raw images. Subsequently, we encode these global images and local objects using the ResNet architecture to derive rich visual features. Additionally, we have designed and implemented an adaptive matrix that dynamically learns during the model training process, filtering visual features based on the relevance between the image and text pairs. In addition, we maintain a dynamically updated normalized vector $G$, which assigns filtered visual and textual features to each Transformer layer with varying weights. This approach facilitates hierarchical integration and fusion of visual and textual information. During the decoding stage, the CRF layer is employed to further refine the output. To mitigate overfitting due to high model complexity, we have incorporated a technique called Flooding, enhancing the model’s generalization and stability. Through these meticulously designed methods, our model achieves exceptional performance in the joint representation learning of images and text.

3.2. Stage 1: Text and Visual Feature Extraction

In this work, each text sequence $X=\left\{x_0,x_1,\dots ,x_n\right\}$ is fed into a pre-trained BERT [4] to obtain the sequence representations $T=\left\{T_0,T_1,\dots ,T_n\right\}$, where $T_i\in {\mathbb{R}}^d$ is the extracted word representation for $x_i$, and $x_0$ and $x_n$ denote the two inserted special tokens “[CLS]” and “[SEP]” that denote the start and end position of a sentence.

$$T_i=BERT\left(x_i;{\theta }^{\mathrm{bert} }\right)\in {\mathbb{R}}^d$$

(1)

The BERT parameter is represented by ${\theta }^{bert}$. In particular, $T_i$ is derived by summing these sub-tokens if the tokenizer divides $x_i$ into numerous sub-tokens.

We use the visual grounding tools to extract the regions of a picture that contain entities given the image [11]. Then, we transform the initial image and the divided image to $224\times 224$ pixels as the hierarchical image $O=\left\{o_1,o_2,\dots ,o_n\right\}$. Visual representations are extracted through the pre-trained ResNet.

$$v_i=ResNet\left(o_i;{\theta }^{\mathrm{res} }\right)\in {\mathbb{R}}^d$$

(2)

where ${\theta }^{\mathrm{res} }$ is the ResNet parameter and $o_i$ is a 224 × 224 pixels image for the $i$-th hierarchical image.

3.3. Stage 2: Correlation Prediction

Visual cues that are not directly related to the text can introduce uncertainty or even negatively impact multimodal model learning. Therefore, prior to incorporating visual features, we filter them based on their relevance to the corresponding text. Given a hierarchical image feature $v_i$, we integrate it with the text feature $T$ and input the combined feature into BERT to derive the joint feature [34]. The Correlation Prediction Gate is defined as a softmax function, which inputs the joint features into the gate to obtain a relationship matrix that controls the visual feature weights.

$$G_{matrix}=softmax\left([v_i, T]\right)$$

(3)

where $v_i$ is the visual features, $T$ is the text features, [⋅,⋅] indicates the concatenation operation, and $G_{matrix}$ indicates the relational matrix.

Then, we apply weights to visual features $v_i$ using the relational matrix $G_{matrix}$ to derive visual representations that are pertinent to the textual content. The visual representations correlated with the text, denoted as $V^{\prime }$, are computed through the following method:

$$V^{\prime }=\left\{v_i{G}_{matrix} \right| i=0, \dots ,n\}= \{ v_i^{\prime } | i=0, \dots ,n\}$$

(4)

where $v_i$ indicates initial visual features.

3.4. Stage 3: Dynamic Gate

Path decision-making can be viewed as this module’s function, and the Dynamic Gate’s prediction of normalized vector G governs the hierarchy of visual features that each Transformer block extracts. First, we create the gate signals’ logits, ${\alpha }^{\left(l\right)}$ [11]:

$${\alpha }^{\left(l\right)}= f(W_l({\displaystyle \frac{1}{c}}\sum _{i=1}^{c}P\left(V_i\right)))$$

(5)

where $f\left(\cdot \right)$ represents the activate function Leaky_ReLU, and $P$ represents the global average pooling layer. We use the MLP layer $W_l$ to reduce the feature dimension by $c$ and implement a soft gate by generating continuous values as path probabilities. The calculation formula for $G$ is as follows:

$$G=\left\{ softmax\left({\alpha }^{\left(l\right)}\right)\right| l=0, \dots ,12\}=\{ G_l | l=0, \dots ,12\}$$

(6)

where $G_l$ indicates the probability vector of $l$-th Transformer block.

In order to assign appropriate visual features to each Transformer layer, we use the Dynamic Gate to link the visual features ${V_i}^{\prime }$ of each image in $V^{\prime }$ to all Transformer layers, and apply adaptive weights $G_i$ during the linking process. The visual feature $V_{Gate}$ selected by the Dynamic Gate is calculated as follows:

$$V_{Gate}=\left\{ \left[{v_0^{\prime }, v_1^{\prime }, \dots , v}_n^{\prime }\right]G_i \right| G_i\in G\}$$

(7)

where $v_i$ is the visual features, and [⋅, ] indicates the concatenation operation.

3.5. Stage 4: Cross-Fusion

While Transformer layers are capable of capturing which contextual words are more relevant to the prediction of input words, they fail to incorporate relevant visual context, resulting in underutilization of image information. To address this, we employ a cross-fusion approach that ensures equal consideration of text and image information from the initial fusion stage. This approach allows each word to learn not only a text-based representation informed by visual context but also a visual representation enhanced by textual information, thereby improving the model’s integrated understanding of both text and visual data. Consequently, the model can more accurately predict words closely associated with each visual segment, thereby enhancing the effectiveness of multimodal information fusion.

Image-Aware Word Representation: To enhance word representation through associated images, we utilize a multihead cross-modal attention mechanism [8]. This mechanism treats $V_{Gate}\in {\mathbb{R}}^{d\times 49}$ as queries, and $T\in {\mathbb{R}}^{d\times \left(d\times 49\right)}$ as keys and values:

$${CA}_i(V_{Gate},T)=softmax\left({\displaystyle \frac{{\left[W_{q_i}{V}_{Gate}\right]}^{\top }\left[W_{k_i}T\right]}{\sqrt{d/m}}}\right){\left[W_{v_i}T\right]}^{\top }$$

(8)

$$CA(V_{Gate},T)=W^{\prime }{\left[{CA}_1(V_{Gate},T),\dots ,{CA}_{12}(V_{Gate},T)\right]}^{\top }$$

(9)

where $\{W_{q_i}, W_{k_i}, W_{v_i}\}$ and $W^{\prime }$ indicate the weight matrices for the query, key, value, and multihead attention, respectively. ${CA}_i$ denotes the $i$-th head of cross-modal attention. We then add three further sub-layers on top of that:

$$\begin{array}{c}\widehat{TC}=LN(V_{Gate}+CA(V_{Gate},T\left)\right)\end{array}$$

(10)

$$TC=LN(\widehat{TC}+FFN(\widehat{TC}\left)\right)$$

(11)

where FFN refers to the feed-forward network, LN denotes layer normalization [37,38], and $TC$ is the output representations of the cross-modal Transformer layer.

Word-Aware Visual Representation: To capture visual representations aligned with each word, we also employ the CMT layer, which uses $T$ as queries and $V_{Gate}$ as keys and values, creating a symmetric variant of the previously described CMT layer. This approach yields a word-specific visual representation, denoted as $VC$, effectively aligning each word with its closely related visual blocks.

3.6. Stage 5: CRF and Flooding Layer

CRF Layer: To combine textual and visual representations, we concatenate the textual representations $TC$ and the visual representations $VC$ to form the final hidden representation $H=\left(t_0,t_1,\dots ,v_0,v_1,\dots \right)$, where $t_i$ denotes the $\mathit{\text{i\_th}}$ textual representation and $v_i$ denotes the $\mathit{\text{i\_th}}$ visual representation [39]. We input $H$ into a standard Conditional Random Field layer that defines the probability of a sequence of labels y given an input sentence $S$ and its associated image $V$:

$$P\left(y∣S,V\right)={\displaystyle \frac{\mathit{exp}\left(\mathit{score}\left(H,y\right)\right)}{\sum _{y^{\prime }} \mathit{exp}\left(\mathit{score}\left(H,y^{\prime }\right)\right)}}$$

(12)

$$score(H,y)=\sum _{i=0}^n T_{y_i,y_{i+1}}+\sum _{i=1}^n E_{h_i,y_i}$$

(13)

where $T_{y_i,y_{i+1}}$ is denoted as the transition score from label $y_i$ to label $y_{i+1}$, and $E_{h_i,y_i}$ is denoted as the emission score for the i-th word corresponding to label $y_i$.

Flooding Layer: Over-parameterized deep networks can cause the training loss to continue to approach zero, making the model overconfident and degrading test performance. To address this problem, we use a solution called Flooding [40], where a baseline $b$ is set, and, when the training loss reaches the value of the baseline, the gradient rises, preventing further reduction of the training loss. If the original learning objective is $L$, the proposed modified learning objective $L^{\prime }$ is

$$L^{\prime }\left(\theta \right)=\left|L\right(\theta )-b|+b$$

(14)

where $b>0$ denotes the flood level set by the user, and $\theta$ represents the model parameter.

4. Experiments

4.1. Dataset

In our experiments, we use three datasets to evaluate the performance: two English benchmark MNER datasets TWITTER-2015 (Zhang et al., 2018 [32]) and TWITTER-2017 (Lu et al., 2018 [9]) and one Chinese dataset WuKong (Gu et al., 2022 [41]).

TWITTER-2015 and TWITTER-2017: TWITTER-2015 and TWITTER-2017 contain 8257 and 4819 tweet data, respectively, and the dataset sequence annotation method is the BIO tagging scheme. The statistical details of the dataset are provided in Table 1.

Table 1. The basic statistics of our three datasets.

Entity Type	TWITTER-2015			TWITTER-2017			WuKong
	Train	Dev	Test	Train	Dev	Test	Train	Dev	Test
PER	2217	552	1816	2943	626	621	7780	2144	2088
LOC	2091	522	1697	731	173	178	1381	400	366
ORG	928	247	839	1674	375	395	6381	1677	1684
MISC/GPE	940	225	726	701	150	157	6865	1765	1854
No. of entities	6176	1546	5078	6049	1324	1351	22407	5986	5992
No. of tweets	4000	1000	3257	3373	723	723	36024	9699	9700

WuKong: This Chinese dataset is constructed from data sourced from the Chinese Internet, featuring 55,423 image–text pairs labeled using the BMES tagging scheme. It includes four entity types: Person, Location, Organization, and Geo-Political Entity. Statistical details of the dataset are provided in Table 1.

4.2. Evaluation Metric

We use precision (P), recall (R), and F1 score (F1) as measures, and we apply exact match assessment, where a named entity is considered correctly identified only if its borders and type match the ground truth [42]. Based on the numbers of true positives ($TP$), false positives ($FP$), and false negatives ($FN$), precision, recall, and F1 score are computed. The detailed calculation is as follows:

$$\mathrm{Precision} = {\displaystyle \frac{TP}{TP+FP}}$$

(15)

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(16)

Precision assesses how well the NER system identifies only the correct entities, while recall assesses the system’s capability to identify all entities within the corpus.

$$\text{F1-score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(17)

The harmonic mean of recall and precision is the F1 score. Given that the dataset includes multiple entity types, we use the micro-averaged F-score to ensure all entities are treated equally.

4.3. Baselines

We performed a thorough comparison of our model against various baseline models to highlight its superiority. The models compared are categorized into two types: text-based approaches and multimodal approaches.

Text-based approaches:

• We utilize BERT (Devlin et al., 2018 [4]) as a language model for MNER with a softmax decoder, as it has been pre-trained on a substantial number of unlabeled text data.
• An extension of BERT, BERT-CRF uses a normal CRF layer as the decoder rather than a softmax layer.

Multimodal approaches:

• UMT (Yu et al., 2020 [8]): In order to acquire both Image-Aware Word Representations and visual representations inspired by the words, UMT (Yu et al., 2020) uses a multimodal interaction module.
• RpBERT (Sun et al., 2021 [34]) adopts a method of text–image relation propagation to select visual clues.
• UMGF (Zhang et al., 2021 [43]) presents a unified multimodal graph fusion approach for MNER.
• HVPNeT (Chen et al., 2022 [11]) introduces a dynamic gated aggregation method to obtain hierarchical multiscaled visual features, which are used as visual prefixes for fusion.
• GMNER (Yu et al., 2023 [44]) introduces a hierarchical indexing framework called H-Index, which generates entity–type–region triples hierarchically using a sequence-to-sequence model.
• MGCMT (Liu et al., 2024 [35]) improves word representation through semantic enhancement and cross-modal interaction at various levels, achieving effective multimodal guidance for each word.

4.4. Experiment Configuration

For the two English datasets, TWITTER-2015 and TWITTER-2017, the hyperparameters are the same for us. The batch size, the input text sequence’s maximum length, the dropout rate, the learning rate, and epochs are, respectively, set to 8, 147, 0.1, 3 × 10^–5, and 50. During the training process, the weight decay parameters for the text, image, and CRF are set to 1 × 10^–2, 5 × 10^–2, 1 × 10^–2, respectively. The text representations $T$ are encoded using the BERT–base–uncased model pre-trained by Devlin et al. (2018) [4]. The visual representations $\mathit{V}$ are encoded with ResNet-101. When applying the Flooding method to the TWITTER-15 and TWITTER-17 datasets, we designated the parameter $b$ as 0.3 and 0.5, correspondingly.

For the WuKong dataset, we adjusted the learning rate, the number of epochs, and $b$ to 1 × 10^–5, 10, and 0.6, respectively, and used BERT–base–Chinese as the text encoder. Other parameters were consistent with TWITTER-2015 and TWITTER-2017. When testing other baseline models on the WuKong dataset, we utilized the default parameters of the original models, merely substituting the English pre-trained models with their Chinese counterparts. This modification was implemented to assess the performance variance of the models in different linguistic contexts.

5. Result

5.1. Ablation Study

To examine the contributions of key components in our model, we conducted a component study. This study presents a comparative analysis of the full model and its variants with different combinations of the Correlation Prediction (CP), Cross-Fusion (CF), and Flooding Modules integrated (i.e., analyzing the model performance when each component is included or excluded). As shown in Table 2, we observed the following:

Table 2. Experimental results of the ablation study.

Methods	TWITTER-2015			TWITTER-2017			WuKong
	P	R	F1	P	R	F1	P	R	F1
Baseline model	73.76	75.52	74.63	86.96	87.05	86.50	77.00	82.88	79.83
+CP	74.05	75.94	74.98	87.32	87.19	87.26	80.86	83.13	81.98
+Flooding	74.87	74.47	74.67	88.09	86.53	87.30	79.36	83.36	81.31
+CF	74.43	75.44	74.94	84.82	86.45	85.63	82.31	81.04	81.67
+CP + Flooding	76.08	75.67	75.88	88.38	86.68	87.52	80.41	83.66	82.01
+CP + CF	73.37	76.46	74.88	85.83	86.97	86.40	83.45	80.96	82.19
+Flooding + CF	76.64	75.98	76.31	88.74	85.71	87.20	80.34	83.29	81.79
MPCFN (ours)	76.84	76.65	76.74	88.34	86.90	87.61	81.82	82.89	82.35

(1)

When the model is not integrated with all components, its performance is lower compared to the full model that includes all components. However, even in such partial-component integration scenarios, the model’s F1 score still exceeds that of the text-only baseline model, as well as that of most existing MNER methods. This finding fully demonstrates that each component in our model plays an indispensable key role in enhancing the final entity recognition results.

(2)

For the English datasets TWITTER-15 and TWITTER-17, the model performance significantly drops when the Flooding module is not integrated, indicating that integrating the Flooding module plays a crucial role in model training. In contrast, for the Chinese dataset WuKong, the performance decline (when the Flooding module is excluded) is less pronounced. This is because the WuKong dataset is over three times larger than the TWITTER datasets, making the data more diverse and reducing the likelihood of overfitting. Consequently, the contribution of integrating the Flooding module is less significant in this case.

(3)

The model experiences a not insignificant performance degradation when neither the Correlation Prediction Module nor the Cross-Fusion Module is integrated. This result suggests that integrating the Correlation Prediction Module (for filtering extraneous images from the data) and integrating the Cross-Fusion Module (for image fusion) are both beneficial for the NER task.

5.2. Analysis of Results Compared to Existing Models

Table 3 presents the experimental results of the MPCFN and all baselines on three datasets. From the experimental results, we can observe the following:

Table 3. Performance results of different MNER approaches.

Modality	Methods	TWITTER-2015			TWITTER-2017			WuKong
		P	R	F1	P	R	F1	P	R	F1
Text	BERT	68.30	74.61	71.32	82.19	83.72	82.95	76.82	81.12	78.91
	BERT-CRF	69.22	74.59	71.81	83.32	83.57	83.44	78.37	79.39	78.88
Text + Image	UMT	71.67	75.23	73.41	85.28	85.34	85.31	79.06	80.46	79.75
	RpBERT	73.29	75.23	74.25	85.86	86.75	86.30	78.44	82.09	80.23
	UMGF	74.49	75.21	74.85	86.54	84.50	85.51	78.22	74.62	76.38
	HVPNeT	73.87	76.82	75.32	85.84	87.93	86.87	79.98	80.12	80.05
	GMNER	61.65	62.03	61.27	65.62	66.54	64.72	-	-	-
	MGCMT	73.57	75.59	74.57	86.03	86.16	86.09	75.28	80.56	77.83
	MPCFN (ours)	76.84	76.65	76.74	88.34	86.90	87.61	81.82	82.89	82.35

(1)

Comparing SOTA multimodal methods with text-based unimodal approaches reveals that multimodal methods generally perform better, suggesting that incorporating additional visual information is often beneficial for NER tasks.

(2)

Comparing current multimodal models with text-based unimodal methods shows that multimodal approaches generally perform better on English datasets like TWITTER-2015 and TWITTER-2017, with a maximum F1 score improvement of about 3.5%. However, on the Chinese dataset WuKong, the maximum performance improvement is approximately 1% (for instance, when comparing RpBERT with BERT), and, in some cases, the performance of certain multimodal models (such as UMGF and MGCMT) is not superior to text-based methods. This highlights the need for further enhancement of multimodal models.

(3)

When comparing text-based unimodal methods with existing multimodal approaches, the proposed Multilevel Predictive Cross-Fusion Network demonstrates substantial performance improvements across three distinct datasets. Specifically, our model achieves improvements of 4.93%, 4.17%, and 3.44% over text-based unimodal methods on these datasets. Furthermore, relative to current state-of-the-art methods, our model yields performance gains of 1.42%, 0.74%, and 2.12%, respectively.

The observed improvement in model performance can be attributed to several factors. First, unlike models such as RpBERT that rely solely on global image features, our approach incorporates additional fine-grained image features for interaction with text, enhancing the alignment of text and image features. Second, whereas models like HVPNeT introduce a Visual Gate to regulate visual features only in the later stages of training, our model employs correlation prediction to filter visual object features at all levels early in the training process. This early-stage screening effectively reduces irrelevant information and minimizes the risk of inadvertently discarding useful visual features by applying a layered filtering approach. These strategies collectively contribute to the exceptional performance of the Multilevel Predictive Cross-Fusion Network in multimodal learning tasks.

5.3. Parameter Sensitivity Analysis

The choice of b in Flooding layer. During training, the parameter $b$ is crucial for setting the lower bound of the loss function. If $b$ is too high, the model may struggle to achieve convergence. Conversely, if $b$ is too low, it may impair the model’s ability to mitigate overconfident predictions, thereby impacting performance on the test set. In order to balance this contradiction, we conducted a series of experiments and parameterized three different datasets with the experimental results. Specifically, we set $b$ to 0.3, 0.5, and 0.6, respectively. As shown in Table 4, these parameter settings are based on a careful analysis of multiple rounds of experiments.

Table 4. The test for different values of parameter $b$ in our model.

Parameter b	TWITTER-2015			TWITTER-2017			WuKong
	P	R	F1	P	R	F1	P	R	F1
0.2	76.07	75.58	75.82	87.09	85.86	86.47	82.01	80.59	81.30
0.3	76.84	76.65	76.74	87.90	87.12	87.51	81.19	83.39	82.28
0.4	76.88	76.05	76.46	86.93	85.64	86.28	80.33	83.28	81.78
0.5	77.32	75.60	76.45	88.34	86.90	87.61	81.08	83.29	82.17
0.6	76.39	75.50	75.94	87.89	85.94	86.90	81.82	82.89	82.35
0.7	75.80	76.49	76.15	89.10	85.27	87.14	81.12	83.24	82.17
0.8	75.51	75.25	75.38	87.42	85.42	86.41	80.79	83.18	81.97

Unimodal learning. This study is dedicated to exploring the specific contribution of different data modalities to the performance of MNER models. By evaluating the unimodal learning effects on text, global images, and local objects, we find that they all have a significant impact on model performance. Table 5 shows that removing either the global image or local objects causes a notable drop in model performance, with similar performance declines observed for both cases. This finding further validates that, in visual scenes, the relative position information of local objects and the global image information complement each other and together play a key role in improving the model performance.

Table 5. The test for different modes in the unimodal learning.

Methods	TWITTER-2015			TWITTER-2017			WuKong
	P	R	F1	P	R	F1	P	R	F1
MPCFN	76.84	76.65	76.74	88.34	86.90	87.61	81.82	82.89	82.35
w/o uni-text	73.76	75.52	74.63	86.96	87.05	86.50	77.00	82.88	79.83
w/o uni-obj	75.05	75.88	75.47	87.59	86.75	87.17	81.22	83.36	82.28
w/o uni-img	76.81	75.21	76.01	88.46	86.23	87.33	80.52	83.11	81.79

5.4. Case Study

Figure 3 presents a qualitative comparison of our proposed MPCFN model against two baseline methods, UMT and MGCMT, in the MNER task. The case studies illustrate how the MPCFN effectively addresses common error types and demonstrates superior recognition capabilities by leveraging multimodal information.

Figure 3. The case comparisons among UMT, MGCMT, and the MPCFN (ours).

Key advantages of the MPCFN include the following:

Accurate Boundary Recognition. In the first case, the MPCFN correctly identifies the full organization name “JKF Ag” as an ORG, while the baseline models only recognize the partial entity “JKF”. This demonstrates the MPCFN’s precision in determining entity boundaries.

Robustness to Complex Entities. For the phrase “Saudi Oil Minister”, the MPCFN accurately recognizes “Saudi” as an LOC, whereas the baselines misclassify the entire phrase as an ORG. This shows the MPCFN’s ability to disambiguate and classify compound entities.

Accurate Entity Identification with Visual Context. In the case of “Sandy from kfc”, the accompanying image contains a prominent, clear KFC logo in the background, which features the iconic portrait of Colonel Sanders. The MPCFN is the only model that correctly identifies “kfc” as an organization (ORG), by effectively utilizing this decisive visual cue. In contrast, MGCMT misclassifies it as a location (LOC), and UMT fails to recognize it as an entity (O). In the tweet “RT @BiebsVogue: justin is probably looking at his twitter like this”, the MPCFN successfully identifies “justin” as a PER (person) by effectively utilizing the accompanying image. In contrast, MGCMT fails to recognize this entity due to insufficient visual–text alignment. This highlights the MPCFN’s superior ability to integrate visual cues for accurate entity recognition.

Accurate entity and non-entity judgment. The last example, featuring the song title “Drag Me Down”, provides a critical test of the model’s ability to leverage contextual cues. While the baseline models (UMT, MGCMT) treated it as a non-entity, our MPCFN model correctly classified it as MISC. This case underscores the MPCFN’s advantage in interpreting nuanced and non-standard entity mentions.

These examples validate that the MPCFN achieves more precise and robust performance by effectively fusing visual and textual information, leading to fewer errors in entity boundary recognition, type classification, and identification compared to existing approaches.

5.5. Attention Analysis Between Textual Entities and Visual Objects

To further reveal the multimodal fusion mechanism of the Cross-Fusion Module, this section analyzes the cross-modal attention matrices generated during the module’s core operation: Image-Aware Word Representation (IAWR). Notably, the matrix is derived from swapping the query (Q) matrices in the multihead cross-modal attention (Equations (8) and (9)), directly reflecting how the model establishes semantic associations between textual entities and visual regions.

As shown in Figure 4, this heatmap visually demonstrates the core function of the cross-fusion module: for the personal name entity “Richard Howarth” in the text (corresponding to a word segment in the text sequence), regions R37 and R46–R47 in the heatmap appear dark blue (attention weight > 0.08), precisely aligned with the facial region of the person in the image; the R43-R47 region corresponding to “Simon Woodings” also shows a high-weight response, matching the visual region of the person. This strong “entity word–visual region” correlation proves that the Cross-Fusion Module achieves fine-grained alignment of text and image features through a query matrix interchange mechanism.

Figure 4. Heatmap of the query-swapped attention matrix (Image-Aware Word Representation phase in cross-fusion). The left panel shows the 7 × 7 segmented input image, with regions labeled R11–R77 (row × column corresponding to grid coordinates). The middle heatmap visualizes the attention matrix (text tokens × visual regions), and only the portion with non-zero attention intensity is displayed—regions R51 and beyond in the image are not included in the intercepted heatmap. The right panel is the weight intensity legend, indicating the correspondence between color depth and attention weight value. The bottom text presents the original social media post content, where entities like “Richard Howarth” and “Simon Woodings” are marked in red with their categories (B-PER/I-PER denotes the beginning/middle part of a person name entity), and the table maps text tokens to their x-axis indices.

For non-entity words in the text, such as the preposition “with”, the conjunction “and”, and even the general noun “morning”, the model significantly reduces the attention weight (lighter colors) given to the person regions in the image. This demonstrates that the model is able to effectively distinguish key entity information from background information in the text. Instead of paying attention uniformly to the entire image, it focuses attention resources on the most relevant visual areas, suppressing the interference of irrelevant information.

5.6. Generalization Study

Model generalization capability describes how well a model adapts when dealing with data that are different from the training data; models with high generalization capability are able to learn general patterns and laws from the training data and apply this knowledge to new data. We only trained on TWITTER2015 and TWITTER-2017 and validated on each other’s test sets, avoiding the WuKong dataset because it is a Chinese dataset with a significant gap compared to the other two English datasets. The results are displayed in Table 6, where TWITTER-2017 → TWITTER-2015 indicates that models trained on the TWITTER-2017 dataset are used to evaluate the TWITTER-2015 dataset and vice versa. From the results in Table 6, it is obvious that our model outperforms other models in terms of generalization ability. We hypothesize that this advantage may be attributed to the crucial role of the Flood layer, which effectively mitigates the overfitting phenomenon by limiting the continuous decrease in the training loss, thus enhancing the model’s adaptability to new data.

Table 6. Performance comparison of generalization ability.

Methods	TWITTER-2017 → TWITTER-2015			TWITTER-2015 → TWITTER-2017
	P	R	F1	P	R	F1
UMT	64.67	63.59	64.13	67.80	55.23	60.87
MGCMT	67.67	63.28	65.40	66.41	58.11	61.98
MPCFN (ours)	73.71	62.84	67.84	65.79	59.22	62.33

5.7. Low-Resource Experiment

In this study, we aimed to evaluate the model’s performance in resource-constrained environments. To accomplish this, we implemented a strategy of randomly sampling between 10% and 50% of the training samples from three distinct datasets, thereby creating a series of training sets under low-resource conditions. On this basis, we conducted a systematic performance comparison of the proposed MPCFN model against the existing UMT and HVPNeT models. The results of this comparison are detailed in Figure 5.

Figure 5. Performance comparison in low-resource setting.

The experimental results demonstrate that the MPCFN significantly outperforms both UMT and HVPNeT in the English datasets TWITTER-2015 and TWITTER-2017, regardless of the low-resource configurations. In addition, on the Chinese dataset WuKong, the MPCFN similarly demonstrates superior performance over the other two models under all conditions except the 10% sample configuration. These results consistently validate the efficiency and robustness of the MPCFN in the face of data scarcity, demonstrating its significant potential and broad applicability for named entity recognition in low-resource environments.

6. Conclusions and Future Work

To cope with the problem where traditional text-based NER methods find it difficult to deal with texts with ambiguous meanings and the challenges of existing MNER methods in terms of data alignment and visual information interference, in this paper we propose an innovative MPCFN for the MNER task. We introduce a relevance prediction module to filter non-essential details in the image and align the features of the image and the text through the Cross-Fusion Module. To enhance model training further, we combine the CRF layer with the Flooding layer to optimize the generalization ability of the network. Through extensive experiments on two benchmark English datasets and one Chinese dataset, our experimental results confirm the excellent performance of the MPCFN on multimodal data in different languages.

Although the MPCFN model proposed in this paper has been fully validated on three benchmark datasets (TWITTER-2015, TWITTER-2017, and WuKong), a detailed observation of the experimental process and in-depth analysis of the results reveal two limitations of the model: (1) Limited coverage of data scenarios: The model is only designed and optimized for the basic social media scenario of “single text–single image”, and has not been adapted to more complex multimodal input forms such as “multiple images associated with a single text” and “text + short video frame sequences”. (2) Weak cross-domain transfer performance: The effectiveness of the model has only been verified on social media datasets, and no adaptive optimization or sufficient testing has been conducted for non-social media multimodal data such as news, e-commerce, or medical data.

This work opens up several avenues for future research: on the one hand, testing and optimization on more linguistic datasets to enhance the performance of the MPCFN in multilingual environments and to address the differences and specific challenges between different languages; on the other hand, how to apply the MPCFN model to multimodal tasks in other domains, including multimodal sentiment analysis and multimodal machine translation, etc., to validate its generalization and scalability in different tasks.

The code, data, and best-performing models are available at https://github.com/kuyuanchou/MPCFN.