Joint Fine-Grained Entity-Type Disambiguation and Robust Grounding via Multi-Level Knowledge Integration for FMNERG

Abstract

Fine-grained Multimodal Named-Entity Recognition and Grounding (FMNERG) aims to extract entity–type–object triples from image–text pairs. However, existing methods still struggle with two major challenges: distinguishing semantically similar labels in an expanded fine-grained label space and achieving robust entity–object alignment under complex image–text relations in social media. To address these issues, we propose a Multi-level Knowledge Integration Framework (MKIF) for FMNERG, which leverages multi-level knowledge to jointly improve fine-grained entity-type disambiguation and robust entity grounding in a unified manner. Specifically, we employ multimodal large language models (MLLMs) to obtain global-level knowledge, enriching the semantics of raw multimodal inputs and serving as a shared semantic anchor across subtasks. To improve fine-grained entity-type discrimination, we further derive conceptual and discriminative descriptions for each type as label-level knowledge and utilize both global- and label-level knowledge as supervision via contrastive learning. In addition, we discover potential visual concepts in the dataset as object-level knowledge and design a dynamic gating mechanism to incorporate the global- and object-level knowledge into object embeddings for robust entity–object alignment. Comprehensive experiments demonstrate that MKIF outperforms state-of-the-art methods. Furthermore, detailed analyses validate the effectiveness of each core component.

1. Introduction

Multimodal named-entity recognition (MNER) has attracted increasing attention in information extraction and multimodal learning. It aims to extract named entities from unstructured text with auxiliary images [1,2,3,4,5,6,7,8,9]. However, existing MNER studies divide entities into coarse-grained entity types (i.e., person, location, organization, and miscellaneous) and only extract entities from text without explicitly linking them to their corresponding regions in the image, restricting the applications of MNER in real-world scenarios. To address these limitations, Fine-grained Multimodal Named-Entity Recognition and Grounding (FMNERG) [10] has recently been proposed. FMNERG includes two subtasks of MNER and Entity Extraction and Grounding (EEG), aiming to extract triples that contain named entities, their fine-grained types and grounded bounding boxes in the image. Compared to MNER, FMNERG has wider applications, including multimodal knowledge graph construction [11], human–machine dialogue [12], visual question answering [13], and privacy-aware multimodal analysis [14]. Despite the rapid progress of multimodal large language models (MLLMs), directly applying them to FMNERG remains nontrivial [5,15,16], especially when structured multimodal extraction requires both fine-grained semantic discrimination and reliable entity–object grounding.

The major challenges of FMNERG stem from the entity–type and entity–object gaps in entity–type–object triple extraction. The first gap arises from the difficulty of associating entities with their fine-grained types as the number of candidate types increases. Fine-grained type hierarchies contain semantically similar subtypes under the same coarse-grained category, resulting in blurred type boundaries that make the subtypes difficult to distinguish [17]. As shown in Figure 1a, the entity Francona can be easily classified into the person category under the coarse-grained setting. However, in FMNERG, the model needs to further recognize it as a coach rather than other semantically similar fine-grained categories, such as athlete, because the coaches are often seen on the sidelines of a field analyzing performance, while the athletes are seen in action like running or jumping in a stadium or track field. This requires the model not only to effectively leverage the multimodal cues in the tweet but also to capture the conceptual and discriminative features of fine-grained entity types for accurate entity-type extraction. The second gap lies in the complex and diverse relationships between textual entities and visual content on social media, where image–text pairs may exhibit varying degrees and forms of semantic relevance [18,19], making it difficult to reliably localize the visual regions corresponding to the entities. Figure 1b demonstrates this challenge. Given several entities which belong to the band category, the visual concepts associated with their corresponding regions include logos, group photos, posters and text, which are diverse in the feature space. Addressing such complex entity–object relations requires robust entity–object alignment in EEG.

Figure 1. (a) An example of FMNERG. None indicates that the entity MLB has no corresponding bounding box in the image. In the coarse-grained setting, categories including person, location, organization and miscellaneous are semantically well separated, making it easy to recognize Francona as person. By contrast, in the fine-grained setting, the model must further distinguish among semantically similar candidate subtypes such as coach, athlete, director, and musician, and correctly classify Francona as coach. (b) Illustration of complex entity–object relations in FMNERG. The same fine-grained entity type may correspond to diverse visual concepts on social media, such as text, logo, poster, and group photo for the type band, which increases the difficulty of aligning textual entities with their corresponding visual objects.

In response to these challenges, two-stage methods [5,20] first extract entity–type pairs and then perform entity-aware visual grounding in a pipeline paradigm, which may lead to error propagation since MNER and EEG are treated as separate tasks. Therefore, unified methods have recently attracted attention. Sequence-to-sequence-based methods H-Index [11] and TIGER [10] directly generate the target entity–type–objects sequence from the multimodal input via pre-trained sequence-to-sequence models. MQSPN [6] further reformulates the FMNERG task as set prediction to learn intra- and inter-entity relationships of multimodal entities, which addresses the exposure bias of sequence-to-sequence-based methods. However, these methods focus on predicting fine-grained labels through autoregressive generation of entity–type–region sequences or set prediction guided by type-grained queries, without explicitly modeling the semantic differences among similar fine-grained types. GEM [21] then exploits external knowledge and multimodal large language model (MLLM)-based multimodal reranking for fine-grained type disambiguation, but it still establishes the associations between entities and visual regions only in feature space, which fails to handle the complex entity–object relations on social media.

To address both issues of distinguishing semantically similar entity types and robustly modeling entity–object relations, we draw inspiration from MLLMs. Recently, MLLMs [22,23,24,25] have demonstrated remarkable capabilities of multimodal understanding [26,27], making it possible to obtain semantically rich knowledge for multimodal inputs and fine-grained entity-type discrimination. Building on this insight, we leverage MLLMs to acquire both global- and label-level knowledge, providing sample-specific context and type-specific semantic cues, respectively. To further improve entity–object alignment, we discover dataset-level visual concepts as object-level knowledge to augment object representations. In this way, multi-level knowledge can be incorporated to support different subtasks and jointly bridge the two gaps of FMNERG.

In this paper, we propose a novel Multi-level Knowledge Integration Framework (MKIF) for FMNERG, as shown in Figure 2. This framework aims to jointly improve fine-grained entity-type discrimination and robust entity–object alignment by leveraging multi-level knowledge in a unified manner. Specifically, we first utilize the global-level knowledge acquired from MLLMs to enrich the semantics of multimodal inputs and serve as a shared semantic anchor across subsequent subtasks. We further design a knowledge-enhanced span representation module with multimodal interaction to obtain the entity span embeddings. Next, to address the first issue of disambiguating fine-grained entity types, we acquire label-level knowledge by generating conceptual and discriminative descriptions for entity types via MLLMs and use both global- and label-level knowledge as supervision through contrastive learning to enhance the model’s ability to discriminate between fine-grained entity types. Then, to tackle the second challenge of robust entity–object alignment, we discover potential visual concepts in the dataset as object-level knowledge and enhance the object representations by dynamically fusing global- and object-level knowledge through a learnable gating mechanism, thereby enabling robust entity–object association. We conduct experiments on the widely used Twitter-FMNERG dataset, and the results demonstrate that our proposed MKIF outperforms both unimodal and multimodal state-of-the-art baselines. Our main contributions are summarized as follows:

• We propose a novel Multi-level Knowledge Integration Framework (MKIF), which incorporates multi-level knowledge to jointly improve multimodal semantic understanding, fine-grained type discrimination, and entity–object alignment for FMNERG in a unified manner. This framework offers a general paradigm for incorporating heterogeneous knowledge in FMNERG.
• We design dual-level knowledge-guided alignment and knowledge discovery-enhanced fusion to operationalize the proposed framework, where the former enhances fine-grained entity-type discrimination with dual-level knowledge-guided supervision and the latter strengthens entity grounding through knowledge-enhanced object representation.
• Extensive experiments demonstrate the superior performance of MKIF compared to both unimodal and multimodal state-of-the-art methods. Furthermore, detailed ablation studies and analyses validate the effectiveness of each component of our framework and provide key insights into its advantages.

Figure 2. Overview of our proposed MKIF. MKIF first obtains entity span embeddings through the knowledge-enhanced span representation module. For MNER, label-level knowledge is incorporated through the dual-level knowledge-guided alignment (DKA) module to improve fine-grained entity-type disambiguation. For EEG, potential visual concepts are discovered as object-level knowledge. Both object- and global-level knowledge are fused with object embeddings via the knowledge discovery-enhanced fusion (KDF) module, enabling robust associations between entities and objects. The heterogeneous knowledge sources are functionally coordinated within MKIF to bridge the entity–type and entity–object gaps in FMNERG.

Figure 2. Overview of our proposed MKIF. MKIF first obtains entity span embeddings through the knowledge-enhanced span representation module. For MNER, label-level knowledge is incorporated through the dual-level knowledge-guided alignment (DKA) module to improve fine-grained entity-type disambiguation. For EEG, potential visual concepts are discovered as object-level knowledge. Both object- and global-level knowledge are fused with object embeddings via the knowledge discovery-enhanced fusion (KDF) module, enabling robust associations between entities and objects. The heterogeneous knowledge sources are functionally coordinated within MKIF to bridge the entity–type and entity–object gaps in FMNERG.

2. Related Works

In this section, we review representative MNER studies beginning in 2018 and FMNERG studies beginning in 2023, together with subsequent developments in both tasks.

2.1. Multimodal Named-Entity Recognition

Conventional named-entity recognition (NER) extracts entities and predicts their types based solely on textual context. However, in multimodal social media posts, short text may provide insufficient evidence for these predictions [2]. To address this limitation, MNER emerged in 2018, aiming to extract textual entities and their types with the help of images [1,2]. Since more complementary information from images is integrated, MNER has been shown to outperform unimodal NER [1,2,3,4,5,9,15,28]. Existing MNER methods mainly focus on exploiting visual information and integrating it into text through modality interaction. To better utilize the image modality, prior studies have explored various forms of visual representations, including visual objects [29], object labels [4], image captions [30,31], OCR texts [31], and scene graphs [32]. For modality interaction, UMT [3] and CAT-MNER [33] propose multimodal interaction modules based on Transformer [34] to capture inter-modality interactions, showing the effectiveness of attention-based fusion for multimodal semantic understanding. UMGF [35] represents the input text and image using a multi-modal graph and proposes a graph-based multimodal fusion module to perform semantic interactions. MAF [36] proposes a cross-modal matching method to determine the proportion of image information that should be retained and exploits contrastive learning [37] to align the representations between the two modalities. However, social media posts are not always informative enough for MNER due to their short textual content and weakly relevant images [38,39,40,41]. As a result, relying solely on in-sample information may lead to suboptimal MNER performance. To tackle this challenge, knowledge-based methods acquire auxiliary external knowledge from multimodal knowledge graphs [42], knowledge corpus [43], or large language models (LLMs) [44] to enrich the semantics of raw samples. In this work, we utilize MLLMs instead of knowledge bases or unimodal LLMs to obtain high-quality external knowledge, thereby enhancing the comprehension of multimodal inputs.

2.2. Fine-Grained Multimodal Named-Entity Recognition and Grounding

FMNERG [10] extends MNER by introducing fine-grained entity types and entity-aware visual grounding. It simultaneously requires extracting entities, predicting their fine-grained types and grounded bounding boxes in the image, which makes the task more challenging than MNER and better aligned with real-world applications. Existing FMNERG methods can be generally divided into two categories. The two-stage methods [5,20] separately process MNER and entity-aware visual grounding, resulting in error propagation. Therefore, unified methods have gained attention. Yu et al. [11] and Wang et al. [10] first propose the sequence-to-sequence-based methods H-Index and TIGER, respectively. They use pre-trained sequence-to-sequence models [45,46] to encode the multimodal input and decode the indices of entities, types, and groundable or ungroundable indicators, and introduce a visual output layer to predict the distribution over candidate visual regions for entity grounding. Bao et al. [47] further propose a multi-level alignment pre-training method based on H-Index to improve congruence between multimodal data. To address the exposure bias issue of sequence-to-sequence-based methods, MQSPN [6] introduces the set prediction paradigm into FMNERG to adaptively learn intra-entity relationships and establish inter-entity relationships from a global optimal matching view. Despite their advancements, the above methods neglect to learn the discrimination of fine-grained entity types explicitly. Recently, GEM [21] exploited LLMs to obtain external knowledge and employed MLLMs to rerank the predicted fine-grained entity categories for disambiguation, followed by a large visual language model (LVLM)-based implicit grounder for EEG. However, GEM still models entity–object relations in feature space, which is insufficient for handling the complex entity–object relationships on social media. In contrast, our approach focuses on leveraging multi-level knowledge to bridge both the entity–type and entity–object gaps.

3. Methodology

In this section, we first formulate the FMNERG task and then present the details of our proposed MKIF. Figure 2 shows the architecture of MKIF, which contains three key components: (1) The knowledge-enhanced span representation module acquires global-level knowledge and generates shared multimodal span representations. (2) The dual-level knowledge-guided alignment (DKA) module aligns the entity span representations with dual-level knowledge by contrastive learning to improve fine-grained entity-type disambiguation in MNER. (3) The knowledge discovery-enhanced fusion (KDF) module discovers the potential visual concepts in the dataset as object-level knowledge and integrates global- and object-level knowledge into object embeddings via a dynamic gating mechanism to facilitate robust entity–object alignment in EEG.

3.1. Problem Formulation

Given a text sequence $T=\left(t_1,t_2,\dots ,t_{N_t}\right)$ of $N_t$ tokens and its associated image I, the goal of FMNERG is to identify all named entities in T, and to predict both their fine-grained entity types from the entity-type set $\mathcal{Y}$ and their corresponding grounded bounding boxes in I. In Twitter-FMNERG, $\mathcal{Y}$ comprises 51 predefined fine-grained entity types organized under 8 coarse-grained categories. The detailed category hierarchy and label distribution are provided in Appendix A. The output Y is a set of triples:

$$Y={\left\{\left(e_i,c_i,b_i\right)\right\}}_{i=1}^m,$$

(1)

where m denotes the number of named entities in T, $e_i$ denotes the ith entity, $c_i\in \mathcal{Y}$ denotes the fine-grained entity type of $e_i$, and $b_i$ denotes the grounded bounding box of $e_i$ in I. If there is no grounded region of $e_i$, $b_i$ is None, otherwise it is a 4-dimensional vector containing the top-left and bottom-right coordinates of the bounding box:

$$b_i=\left\{\begin{array}{cc}\left(x_i^1,y_i^1,x_i^2,y_i^2\right),\hfill & \mathrm{if}{e}_i\mathrm{is}\mathrm{grounded},\hfill \\ None,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(2)

where $\left(x_i^1,y_i^1\right)$ and $\left(x_i^2,y_i^2\right)$ denote the top-left and bottom-right coordinates of $b_i$, respectively.

3.2. Knowledge-Enhanced Span Representation

To enrich the semantics of multimodal inputs and enhance the model’s understanding of them, we first design a knowledge-enhanced span representation module with multimodal interaction. Motivated by the impressive capabilities of MLLMs in multimodal understanding [26,27], we employ MLLMs instead of unimodal LLMs and knowledge corpora to obtain global-level knowledge for each multimodal input. Specifically, we selected GPT-4o-mini [22] as a representative MLLM for global-level knowledge acquisition, given its strong performance in multimodal understanding and cost efficiency. We input the text T and image I into GPT-4o-mini, using the instruction template “Here is the text of a tweet: T, and the image of this tweet I is uploaded above. Please provide brief background knowledge of this tweet, tell me the named entities in the text, and tell me the visual concepts in the image briefly.” to obtain the global-level knowledge as:

$$K=\left(k_1,k_2,\dots ,k_{N_k}\right)=MLLM(T,I),$$

(3)

where $k_i$ denotes the ith knowledge token, and $N_k$ is the number of tokens in K. We concatenate the global-level knowledge K with text T as the final textual input:

$$T_k=\left(t_1,t_2,\dots ,t_{N_t},\left[\mathrm{SEP}\right],k_1,k_2,\dots ,k_{N_k}\right),$$

(4)

where $\left[\mathrm{SEP}\right]$ is the separator token (taking BERT [48] for example).

We use pre-trained BERT as the text encoder. Given the textual input $T_k$, we obtain its contextualized representation as $\mathit{T}=BERT\left(T_k\right)=\left({\mathit{t}}_1,{\mathit{t}}_2,\dots ,{\mathit{t}}_{N_x}\right)$, where $N_x$ is the total length of $\mathit{T}$ and ${\mathit{t}}_i\in {\mathbb{R}}^d$ is the contextualized embedding of the ith token. For the image modality, we follow previous works [10,11,47] to detect all visual objects of each image via the pre-trained VinVL [49] model, and rank them based on the detection probabilities, obtaining the top-$N_o$ object candidates as $\mathit{O}=\left({\mathit{o}}_1,{\mathit{o}}_2,\dots ,{\mathit{o}}_{N_o}\right)\in {\mathbb{R}}^{N_o\times 2048}$, where ${\mathit{o}}_i\in {\mathbb{R}}^{2048}$ is the VinVL representation of the ith object. We then project $\mathit{O}$ into the same dimension as $\mathit{T}$ using a trainable linear layer ${\mathit{W}}_R\in {\mathbb{R}}^{2048\times d}$ as the visual encoder, obtaining $\mathit{R}=\mathit{O}{\mathit{W}}_R=\left({\mathit{r}}_1,{\mathit{r}}_2,\dots ,{\mathit{r}}_{N_o}\right)\in {\mathbb{R}}^{N_o\times d}$.

Next, to effectively integrate visual information with textual representations, we design a Transformer [34,50]-based multimodal interaction module for fine-grained token-object interactions in FMNERG. Specifically, we feed $\mathit{T}$ and $\mathit{R}$ into cross-attention transformers (CATs) to calculate the interactions, including text-guided visual attention and image-guided textual attention. In this way, the model can dynamically select complementary visual cues for each token while preserving the contextualized textual semantics. $CAT({\mathit{H}}_x,{\mathit{H}}_y)$ is calculated as:

$$\left\{\begin{array}{c}{\mathit{H}}_x^{\prime }=LN\left(MHA({\mathit{H}}_x,{\mathit{H}}_y,{\mathit{H}}_y)+{\mathit{H}}_x\right),\hfill \\ CAT\left({\mathit{H}}_x,{\mathit{H}}_y\right)=LN\left(FFN\left({\mathit{H}}_x^{\prime }\right)+{\mathit{H}}_x^{\prime }\right).\hfill \end{array}\right.$$

(5)

where MHA, LN and FFN are multi-head attention, layer normalization and feed-forward network, respectively. We calculate the text-guided visual interaction and image-guided textual interaction as ${\mathit{R}}^{\prime }=CAT(\mathit{R},\mathit{T})$ and ${\mathit{T}}^{\prime }=CAT(\mathit{T},\mathit{R})$.

We then concatenate ${\mathit{T}}^{\prime }$ and ${\mathit{R}}^{\prime }$ and feed them into a self-attention transformer (SAT). $SAT\left({\mathit{H}}_x\right)$ is calculated as:

$$\left\{\begin{array}{c}{\mathit{H}}_x^{\prime }=LN\left(MHA({\mathit{H}}_x,{\mathit{H}}_x,{\mathit{H}}_x)+{\mathit{H}}_x\right),\hfill \\ SAT\left({\mathit{H}}_x\right)=LN\left(FFN\left({\mathit{H}}_x^{\prime }\right)+{\mathit{H}}_x^{\prime }\right).\hfill \end{array}\right.$$

(6)

We denote the combination of the above bidirectional cross-attention and subsequent self-attention as the cross-modal interaction (CMI) module. The output of CMI is:

$${\mathit{H}}_{\mathrm{CMI}}=SAT\left([{\mathit{T}}^{\prime };{\mathit{R}}^{\prime }]\right)\in {\mathbb{R}}^{(N_x+N_o)\times d}.$$

(7)

The resulting matrix ${\mathit{H}}_{\mathrm{CMI}}$ jointly encodes the textual sequence and visual objects after cross-modal interaction.

Finally, we adopt a span-based prediction approach [51,52] to extract entities and their corresponding types. We enumerate all possible $n_s$ spans (up to a maximum width of $L_{max}$) in the text as $\mathcal{S}={\left\{s_i\right\}}_{i=1}^{n_s}$ and calculate their representations as:

$${\mathit{h}}_e\left(s_i\right)=\left[{\mathit{h}}_{START(i)};{\mathit{h}}_{END(i)};{\varphi }_e\left(s_i\right)\right],$$

(8)

where ${\mathit{h}}_{START(i)}$ and ${\mathit{h}}_{END(i)}$ are the representations of the start and end tokens of $s_i$ from the first $N_t$ rows of ${\mathit{H}}_{\mathrm{CMI}}$, and ${\varphi }_e\left(s_i\right)$ is the span length embedding obtained from a learnable look-up table. Following prior span-based entity extraction methods [51,52], we formulate span typing as single-label multiclass classification and adopt the cross-entropy loss as the training objective for MNER:

$${\mathcal{L}}_{span}=-\sum _{s_i\in \mathcal{S}}\log P_e\left(y_e^{\ast }\left(s_i\right)\mid s_i\right),$$

(9)

where $y_e^{\ast }\left(s_i\right)$ denotes the gold span label from set ${\mathcal{Y}}^{\prime }=\mathcal{Y}\cup \left\{ϵ\right\}$, and $ϵ$ denotes a non-entity span.

3.3. Dual-Level Knowledge-Guided Alignment

To address the first gap of insufficient discrimination of semantically similar fine-grained entity types, we propose a dual-level knowledge-guided alignment (DKA) module. DKA captures the conceptual and discriminative features of fine-grained entity types as label-level knowledge and then integrates them with global-level knowledge to enhance the model’s discriminative capability through contrastive learning.

Inspired by Zhang et al. [53], we first generate both conceptual and discriminative descriptions for each fine-grained entity type using the MLLM. To be specific, given an FMNERG dataset containing M entity types, we query GPT-4o-mini [22] to generate two kinds of descriptions for each category, as shown in Table 1.

Table 1. Details of conceptual and discriminative descriptions for entity types.

	Conceptual	Discriminative
Definition	Descriptions that summarize the concept of each entity type.	Descriptions that distinguish semantically similar entity types
Prompt	Please describe both the textual and visual concepts of [class].	Please provide useful features for distinguishing the original class [class_1] from the similar class [class_2].

Note: Bold text identifies the column and row headings.

Conceptual Description. For each entity type c, we obtain one conceptual description $d_{con}^c$ using the following query: “Please describe both the textual and visual concept of each given word: [athlete, musician, …]”. This kind of description summarizes the concept of each entity type.

Discriminative Descriptions. For each entity type c, we define the other entity types that share the same coarse-grained category with it as its similar label set ${\mathcal{S}}_c$. Formally, let ${\mathcal{C}}_{coarse}\left(c\right)$ denote the coarse-grained category of c. Then, ${\mathcal{S}}_c=\{c^{\prime }\mid c^{\prime }\ne c,{\mathcal{C}}_{coarse}\left(c^{\prime }\right)={\mathcal{C}}_{coarse}\left(c\right)\}$. We then generate the discriminative description set ${\mathcal{D}}_{dis}^c$ for c using the following prompt: “[fine_class_1] and [fine_class_2] belong to the category [coarse_class]. Please provide useful features for distinguishing the original class [fine_class_1] from the similar class [fine_class_2].”.

${\mathcal{D}}_{dis}^c$ contains $|{\mathcal{S}}_c|$ discriminative descriptions, with each description characterizing the distinguishing features between c and one of its similar types in ${\mathcal{S}}_c$. These discriminative descriptions provide informative cues for distinguishing semantically similar fine-grained entity types.

In this way, given an entity type c, we obtain its label-level knowledge ${\mathcal{D}}^c=\left\{d_{con}^c\right\}\cup {\mathcal{D}}_{dis}^c$, which contains one conceptual description and $\left|{\mathcal{S}}_c\right|$ discriminative descriptions. All descriptions are generated offline before model training. We then encode all descriptions using BERT to obtain $|{\mathcal{D}}^c|=1+\left|{\mathcal{S}}_c\right|$ label embeddings, denoted as ${\mathit{E}}_{\mathcal{D}}^c\in {\mathbb{R}}^{|{\mathcal{D}}^c|\times d_l}$, where $d_l=d$ is the embedding dimension. The final label-level knowledge embedding ${\mathit{e}}_{lk}^c\in {\mathbb{R}}^{d_l}$ is obtained by averaging all embeddings as:

$${\mathit{e}}_{lk}^c=\mathrm{adaptive}\_\mathrm{avgpool}\left({\mathit{E}}_{\mathcal{D}}^c\right).$$

(10)

However, the obtained label embeddings mainly encode category-level general characteristics, which are independent of the input sample. Therefore, we employ cross-attention to incorporate global-level knowledge as complementary contextual information, enabling the model to enrich the label semantics with sample-related cues and obtain the dual-level knowledge embedding of c as:

$${\mathit{e}}_c=CAT({\mathit{e}}_{lk}^c,{\mathit{e}}_{gk}),$$

(11)

where ${\mathit{e}}_{gk}$ is the global-level knowledge embedding obtained from BERT. Notably, the BERT model used for knowledge encoding shares its parameters with the BERT model described in Section 3.2, and the parameters are frozen in this step.

Finally, motivated by Zhang et al. [53], we design a contrastive loss to improve fine-grained entity type discrimination under the supervision of dual-level knowledge. For each entity span s, the dual-level knowledge embedding of its gold label $y_s$ serves as the positive embedding, while the dual-level knowledge embeddings of all the other entity types are treated as negative embeddings. Following normalized temperature-scaled contrastive formulations for representation learning [37], we define the loss as follows:

$${\mathcal{L}}_{dka}=-\sum _{s\in {\mathcal{S}}_{gold}}\log {\displaystyle \frac{\exp \left(sim\left({\mathit{h}}_e\left(s\right),{\mathit{e}}_{y_s}\right)/\tau \right)}{{\sum }_{c=1}^M\exp \left(sim\left({\mathit{h}}_e\left(s\right),{\mathit{e}}_c\right)/\tau \right)}},$$

(12)

where $sim(·,·)$ denotes the normalized cosine similarity, $\tau$ is a temperature parameter, ${\mathcal{S}}_{gold}$ is the set of gold entity spans in the input text, $y_s$ is the gold fine-grained type of span s, and ${\mathit{e}}_c$ represents the dual-level knowledge embedding for the cth entity type. Accordingly, the denominator contrasts the span representation against the embeddings of all entity types, encouraging the span to be close to the embedding of its gold label and far from those of incorrect labels, thereby facilitating fine-grained entity-type disambiguation.

3.4. Knowledge Discovery-Enhanced Fusion

To robustly model entity–object relations, we introduce a knowledge discovery-enhanced fusion (KDF) module, which discovers potential visual concepts from the dataset as object-level knowledge and adaptively integrates the object-level and global-level knowledge into object embeddings through a learnable gating mechanism, thereby facilitating robust entity–object alignment in EEG.

The first step is visual concept discovery. After detecting objects in all training images using VinVL (as described in Section 3.2), we employ the Vision Transformer (ViT) [54] encoder of the pre-trained CLIP model [55] to extract their feature embeddings. Benefiting from large-scale vision-language pre-training, CLIP captures broad visual semantics and exhibits strong transferability across diverse visual recognition tasks, enabling efficient feature extraction from a large number of heterogeneous objects in social media images. Based on the extracted embeddings, we then apply an unsupervised clustering algorithm to divide these objects into V clusters and treat the cluster centers as potential visual concept embeddings. For each object, we take the ID of its nearest cluster center as its corresponding visual concept v and introduce a learnable lookup table ${\varphi }_v$ to obtain the object-level knowledge embedding ${\mathit{e}}_{ok}={\varphi }_v\left(v\right)$, where the parameters of ${\varphi }_v$ are initialized with the ViT features of the cluster centers.

Next, we introduce a learnable gating mechanism to adaptively control the contribution of global- and object-level knowledge to object representations as:

$$\mathit{g}=\sigma \left({\mathit{W}}_h{\mathit{r}}_i+\tilde{\mathit{e}}\right),$$

(13)

where $\tilde{\mathit{e}}={\mathit{W}}_e[{\mathit{e}}_{gk};{\mathit{e}}_{ok}]$ denotes the projected knowledge embedding, ${\mathit{W}}_h$ and ${\mathit{W}}_e$ are learnable projection matrices, and $\sigma (·)$ denotes the Sigmoid function. The final output of KDF is calculated as:

$${\mathit{r}}_i^k={\mathit{r}}_i+\mathit{g}\odot \tilde{\mathit{e}},$$

(14)

where ⊙ is the element-wise product operation. We denote the knowledge-enhanced object embedding set as: ${\mathit{R}}^k=\left({\mathit{r}}_1^k,{\mathit{r}}_2^k,\dots ,{\mathit{r}}_{N_o}^k\right)$.

During EEG, for each span, we first determine whether it is groundable in the image, which is formulated as a binary classification task:

$${\mathcal{L}}_{groundable}=-\sum _{s_i\in \mathcal{S}}\log P_g\left(y_g^{\ast }\left(s_i\right)\mid s_i\right),$$

(15)

where $y_g^{\ast }\left(s_i\right)$ denotes the gold label indicating whether $s_i$ is groundable. For each groundable span s, we calculate its similarity distribution $p\left(\mathit{z}\right)$ over $N_o$ knowledge-enhanced object embeddings as:

$$p\left(\mathit{z}\right)=Softmax\left({\mathit{R}}^k{\mathit{h}}_e\left(s\right)\right).$$

(16)

Following prior work [10], we use the Intersection over Union (IoU) scores as the supervision signals. For each visual region, we compute its IoU scores with regard to all ground-truth bounding boxes of a given entity and take the maximum value as its final IoU score. Notably, if the final IoU score of a visual region is lower than 0.5, we follow the setting in [56] and set it to 0. We take the normalized IoU score distribution $g\left(\mathit{z}\right)$ as the object supervision of each groundable entity. The training objective is to minimize the Kullback–Leibler Divergence (KLD) [57] loss between the similarity distribution $p\left(\mathit{z}\right)$ and object supervision $g\left(\mathit{z}\right)$, formulated as:

$${\mathcal{L}}_{eeg}={\displaystyle \frac{1}{NG}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{k=1}^G}g\left({\mathit{z}}_k^j\right)\log {\displaystyle \frac{g\left({\mathit{z}}_k^j\right)}{p\left({\mathit{z}}_k^j\right)}},$$

(17)

where N is the number of samples, and G is the number of groundable entities.

3.5. Model Training and Inference

We train MKIF by jointly optimizing the following losses:

$$\mathcal{L}={\mathcal{L}}_{span}+{\mathcal{L}}_{dka}+{\mathcal{L}}_{groundable}+{\mathcal{L}}_{eeg}.$$

(18)

During inference, we first extract entities and predict their types using $y_e\left(s\right)=\arg {\max }_{c\in {\mathcal{Y}}^{\prime }}{P}_e\left(c\mid s\right)$, and we select ${\mathcal{S}}_{ent}=\left\{s:y_e\left(s\right)\ne ϵ\right\}$ as the set of entity spans. Then, we predict whether the entities in ${\mathcal{S}}_{ent}$ are groundable by computing $y_g\left(s\right)=\arg {\max }_{b\in \{0,1\}}{P}_g\left(b\mid s\right)$. For the ith ungroundable entity $s_i$, we predict its triple as $(s_i,y_e\left(s_i\right),None)$. For the jth groundable entity $s_j$, we select the visual object with the highest probability in the similarity distribution as its predicted region $b_{pred}=\left(x_{pred}^1,y_{pred}^1,x_{pred}^2,y_{pred}^2\right)$, and its predicted triple is $(s_j,y_e\left(s_j\right),b_{pred})$.

4. Experiments

In this section, we evaluate the overall effectiveness of MKIF and the contributions of its proposed components. Following the experimental setup in Section 4.1, Section 4.2 compares MKIF with representative baselines to assess its performance in FMNERG, MNER, and EEG. Section 4.3 examines the contributions of DKA, KDF, and the individual knowledge sources through ablation studies. We also evaluate the contribution of each information request in the global-level knowledge generation prompt through ablation studies. Section 4.4 further investigates fine-grained type disambiguation, visual concept discovery, and the sensitivity of MKIF to the choice of knowledge-generating MLLM. Finally, Section 4.5 compares model predictions on representative samples selected from the test set to illustrate the effectiveness of MKIF in fine-grained type discrimination and robust entity–object alignment.

4.1. Setup

4.1.1. Dataset

We conducted experiments on the widely used Twitter-FMNERG dataset [10]. It contains 10,000 tweet–image pairs and 16,772 entities, including 6770 visually groundable entities associated with 8148 bounding boxes. Table 2 shows the statistics of Twitter-FMNERG, while Appendix A presents the detailed entity–type hierarchy and coarse-grained category distribution.

Table 2. Statistics of Twitter-FMNERG dataset. “#” denotes the number of items.

Statistics	Train	Valid	Test	Total
#Tweet	7000	1500	1500	10,000
#Entity	11,779	2450	2543	16,772
#Groundable Entity	4733	991	1046	6770
#Box	5723	1171	1254	8148

4.1.2. Baselines

To evaluate the effectiveness of our method, we compared it with several representative unimodal and multimodal baselines.

First, we selected the following unimodal methods which only exploit the text modality: HBiLSTM-CRF-None [2], BERT-None [48], BERT-CRF-None [10] and T5-Paraphrase-None [10]. None denotes that we only detected the entity–type pairs for the MNER subtask and set the visual object prediction to the majority class $None$.

Next, we selected several competitive multimodal baselines, including GVATT-VinVL-EVG [2], UMT-VinVL-EVG [3], UMGF-VinVL-EVG [35], ITA-VinVL-EVG [31], H-Index [11], MMT5-VinVL-EVG [10], TIGER [10], MQSPN [6] and GEM [21]. EVG means that we used MNER methods to extract entity–type pairs and then predicted the bounding box for each pair via entity-aware visual grounding (EVG) proposed in [10].

4.1.3. Implementation Details

We conducted experiments on a single NVIDIA A100 GPU with PyTorch 1.10.1. In the knowledge-enhanced span representation module, we used the pre-trained bert-base-uncased model to encode the textual inputs. The maximum length of spans $L_{max}$ was set to eight as in [52]. Following previous work [10], the number of object candidates $N_o$ detected by VinVL was 32. The temperature parameter $\tau$ in the contrastive loss was set to 0.1, informed by [37]. These settings were kept fixed during hyperparameter selection. In the KDF module, we used the pre-trained CLIP ViT-B/32 model for object encoding, and used the k-means algorithm [58] to discover the potential visual concepts. The number of potential visual concepts V was 100.

We selected task-specific hyperparameters using validation FMNERG F1, varying one parameter group at a time while keeping the others fixed. Starting with a batch size of eight, one attention layer in each of CMI and DKA, and a maximum of 50 epochs, we sequentially evaluated $V\in \{50,100,200,300,400,500\}$, five joint CMI/DKA attention layer configurations $\left\{\right(1,1),(2,2),(4,4),(1,2),(2,1\left)\right\}$, and batch sizes $\{4,8,16,32\}$. Each configuration was compared using its best validation checkpoint. The resulting settings were $V=100$, one attention layer in each of the CMI and DKA modules, and a batch size of 8. After selecting the final configuration, we set its training limit to 40 epochs based on the validation learning curve from a separate run using the same configuration but a different random seed. Detailed results and the learning curve are provided in Appendix E. The test set was excluded from configuration and checkpoint selection.

We used the AdamW optimizer [59] to minimize the training loss function. The learning rate was set to $2\times {10}^{-5}$ for the pre-trained BERT, as evaluated in its original fine-tuning study [48], and $2\times {10}^{-4}$ for the other trainable parameters, consistent with the learning rate ranges considered in [60]. Both learning rates were fixed rather than included in our hyperparameter search.

4.1.4. Metrics

For a fair comparison, we followed previous works [6,10,21] and took the F1 score as the metric to evaluate FMNERG performance. To be specific, in the MNER subtask, if the span boundary and entity type of a predicted entity were both correct, we considered it a correct prediction. In the EEG subtask, if an entity was not groundable, it was a correct prediction only if the EEG prediction was $None$. In contrast, for a groundable entity, it was a correct prediction if the IoU score between the predicted visual object and any of the ground-truth bounding boxes was greater than 0.5 [61]. This process is demonstrated as follows:

$$C_e/C_t=\left\{\begin{array}{cc}1,\hfill & p_e/p_t=g_e/g_t,\\ 0,\hfill & \mathrm{otherwise},\end{array}\right.$$

(19)

$$C_r=\left\{\begin{array}{cc}1,\hfill & p_r=g_r=\mathrm{None},\\ 1,\hfill & \max \left({\mathrm{IoU}}_1,\dots ,{\mathrm{IoU}}_j\right)>0.5,\\ 0,\hfill & \mathrm{otherwise},\end{array}\right.$$

(20)

where $C_e$, $C_t$, and $C_r$ denote whether the entity, type, and grounding result are correctly predicted, respectively. $p_e$, $p_t$, and $p_r$ represent the predicted entity, type, and visual region, while $g_e$, $g_t$, and $g_r$ denote their corresponding gold annotations. ${\mathrm{IoU}}_j$ is the IoU score between the predicted region $p_r$ and the jth gold bounding box.

In this way, we can calculate the Precision (Pre), Recall (Rec) and F1 score (F1) as:

$$correct=\left\{\begin{array}{cc}\hfill 1,& C_e\mathrm{and}{C}_t\mathrm{and}{C}_r,\\ \hfill 0,& \mathrm{otherwise},\end{array}\right.$$

(21)

$$Pre={\displaystyle \frac{\#correct}{\#predict}},Rec={\displaystyle \frac{\#correct}{\#gold}},F1={\displaystyle \frac{2\times Pre\times Rec}{Pre+Rec}},$$

(22)

where $\#correct$ is the number of predicted triples that match the gold triples, $\#predict$ and $\#gold$ are the numbers of predicted and gold triples, respectively.

4.2. Main Results

To assess the overall effectiveness of MKIF, we compared it with representative unimodal and multimodal baselines on Twitter-FMNERG. Table 3 reports the results on FMNERG and its two subtasks, MNER and EEG.

Table 3. Comparison of MKIF with baselines on Twitter-FMNERG. “-” indicates that the corresponding results are not reported in the original paper.

		FMNERG			MNER			EEG
Modality	Methods	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1
Text	HBiLSTM-CRF-None	34.86	32.38	33.57	62.31	56.55	59.29	49.25	43.27	46.07
	BERT-None	33.28	34.28	33.77	58.91	60.05	59.47	46.27	47.65	46.94
	BERT-CRF-None	34.41	35.51	34.95	60.06	61.38	60.72	46.93	48.44	47.67
	T5-Paraphrase-None	37.38	37.29	37.33	64.83	65.32	65.07	49.03	48.91	48.97
Text + Image	GVATT-VinVL-EVG	42.02	38.75	40.32	63.08	57.85	60.35	55.27	53.46	54.35
	UMT-VinVL-EVG	40.67	41.99	41.32	61.24	62.01	61.63	53.58	55.32	54.43
	UMGF-VinVL-EVG	41.73	42.11	41.92	61.68	61.90	61.79	54.51	55.00	54.75
	ITA-VinVL-EVG	43.05	42.51	42.78	63.80	62.64	63.21	57.63	56.90	57.26
	H-Index	46.83	46.28	46.55	65.25	64.45	64.84	60.82	60.10	60.46
	MMT5-VinVL-EVG	45.35	45.08	45.21	66.46	66.77	66.61	58.35	58.01	58.18
	TIGER	47.57	46.85	47.20	64.43	65.40	64.91	62.44	61.49	61.96
	MQSPN	-	-	48.57	-	-	67.09	-	-	62.64
	GEM	-	-	50.54	-	-	68.09	-	-	63.59
	Ours	53.97	52.18	53.06	71.13	68.86	69.97	66.41	64.22	65.29

Note: Bold and underlined values indicate the best and second-best results, respectively.

The main results show that multimodal approaches outperform unimodal ones, highlighting the necessity of visual information for joint entity recognition and grounding. The unified methods are generally superior to the EVG-based pipeline methods due to the alleviation of error propagation. Compared with state-of-the-art methods, MKIF achieves a relative F1 improvement of 4.99% over GEM, as well as gains of 2.76% and 2.67% on the MNER and EEG subtasks, respectively. Our method also achieves higher precision and recall than the baseline on FMNERG, MNER and EEG.

To further assess the robustness of MKIF’s gains over TIGER, we evaluated both models across five random seeds. For each of FMNERG, MNER, and EEG, we applied a two-sided paired t-test to the seed-wise F1 differences, testing whether their mean was zero. We used a Bonferroni-adjusted significance threshold of $0.05/3=0.0167$ for the three comparisons. As shown in Figure 3, MKIF outperforms TIGER on FMNERG under every seed, with a mean gain of 5.30 F1 points (95% CI: [4.59, 6.00]; $p=3.13\times {10}^{-5}$). The mean gains on MNER and EEG are 3.93 and 3.53 F1 points, respectively, and both are significant under the adjusted threshold $p=1.06\times {10}^{-4}$ and $p=1.51\times {10}^{-3}$. These results support the consistency of MKIF’s performance gains across the evaluated seeds. Appendix B provides the detailed results for each seed and the full statistical report.

Figure 3. FMNERG F1 scores of MKIF and TIGER across five matched random seeds.

We further evaluated our method on the grounded multimodal named entity recognition (GMNER) [11] task, which is the coarse-grained version of FMNERG and includes only four entity types in the MNER subtask (i.e., person, location, organization and miscellaneous). Note that we removed DKA as its role in enhancing fine-grained entity-type discrimination is not required in this setting. We supplemented the evaluation with two additional GMNER baselines, MNER-QG [62] and Bao et al.’s method [47], and report the results on Twitter-GMNER in Table 4. From the table, we can observe that MKIF achieves the highest F1 scores on both GMNER and EEG, and the second-highest F1 on MNER, further confirming its effectiveness.

Table 4. Results on Twitter-GMNER. “-” indicates that the corresponding results are not reported in the original paper.

	GMNER			MNER			EEG
Methods	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1
MNER−QG	53.02	54.84	53.91	78.16	78.59	78.37	58.48	56.59	57.52
H-Index	56.16	56.67	56.41	79.37	80.10	79.73	60.90	61.46	61.18
TIGER	55.84	57.45	56.63	79.88	80.70	80.28	60.72	61.81	61.26
Bao et al.	57.93	58.56	58.24	79.65	80.55	80.10	-	-	-
MQSPN	59.03	58.49	58.76	81.22	79.66	80.43	61.86	62.94	62.40
GEM	-	-	59.83	-	-	83.15	-	-	63.16
Ours	62.29	61.58	61.93	82.22	81.38	81.80	65.91	65.16	65.53

Note: Bold and underlined values indicate the best and second-best results, respectively.

All these results demonstrate the effectiveness of MKIF. We attribute its superior performance over the baselines to the following reasons: (1) MKIF acquires high-quality global-level knowledge from the MLLM to enrich the semantics of raw samples, enabling a better understanding of multimodal inputs than non-knowledge-based methods such as TIGER and MQSPN. (2) MKIF leverages both global- and label-level knowledge to explicitly address the insufficient disambiguation of semantically similar entity types, leading to substantially improved MNER performance. Moreover, our method does not require additional instruction tuning of MLLM as in GEM, which improves computational efficiency. (3) By discovering visual concepts and performing gated fusion between knowledge and objects, MKIF effectively captures complex entity–object relations, outperforming the baselines that capture such relationships only in the feature space during EEG.

Additionally, we report the F1 scores of each method on coarse-grained entity types in Table 5. The results show that for coarse-grained categories person, building, product and art, our method achieves F1 improvements of 14.80%, 2.61%, 2.99% and 6.94% over MQSPN, respectively. MKIF also achieves the second-best performance on location, organization and other. The results on Twitter-FMNERG and Twitter-GMNER support the effectiveness and generalizability of MKIF across fine- and coarse-grained entity-type settings.

Table 5. F1 scores of our methods and baselines on FMNERG with respect to each coarse-grained entity type.

Entity Type	H-Index	TIGER	MQSPN	Ours
Person	45.13	43.78	48.64	55.84
Location	62.33	67.69	71.03	68.75
Building	32.88	40.00	44.78	45.95
Organization	46.68	46.75	49.03	47.33
Product	28.19	27.38	29.46	30.34
Art	38.89	43.27	42.84	45.92
Event	45.56	48.39	51.88	48.28
Other	41.81	48.28	43.49	43.80
All	46.55	47.20	48.57	53.06

Note: Bold and underlined values indicate the best and second-best results, respectively. Italics indicate entity type labels.

To complement the task performance, we also compare the online inference efficiency of MKIF and TIGER. The measurement setup and detailed results for inference latency, FLOPs, and peak GPU memory are provided in Appendix F.

4.3. Ablation Study

In this section, we examine the contributions of the proposed components in MKIF and the information requested in the global-level knowledge generation prompt through ablation experiments.

First, we compared the full model with variants that removed DKA, KDF, or individual knowledge sources to assess their respective contributions to performance.

The results are presented in Table 6, which includes the following settings: (1) We removed DKA (w/o DKA). We further assessed the contribution of each component within DKA by individually removing global-level knowledge (w/o Glob.), conceptual descriptions (w/o Conc.), and discriminative descriptions (w/o Disc.). (2) We removed KDF during EEG (w/o KDF) and further removed the global-level knowledge (w/o Glob.) and object-level knowledge (w/o Obj.) individually during fusion. (3) We removed both DKA and KDF (w/o DKA and KDF).

Table 6. Ablation study of MKIF.

	FMNERG			MNER			EEG
Methods	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1
MKIF	53.97	52.18	53.06	71.13	68.86	69.97	66.41	64.22	65.29
w/o DKA	52.79	51.00	51.88 ($\downarrow 1.18$)	70.45	68.15	69.28 ($\downarrow 0.69$)	65.41	63.19	64.28 ($\downarrow 1.01$)
w/o Glob.	52.64	51.71	52.17	70.14	68.98	69.55	65.25	64.10	64.67
w/o Conc.	53.75	51.00	52.34	71.41	67.84	69.57	66.10	62.72	64.37
w/o Disc.	52.59	51.87	52.23	70.02	69.13	69.57	64.87	63.98	64.42
w/o KDF	52.46	51.51	51.98 ($\downarrow 1.08$)	70.08	68.90	69.49 ($\downarrow 0.48$)	65.16	63.98	64.56 ($\downarrow 0.73$)
w/o Glob.	53.07	51.95	52.50	70.43	69.02	69.72	65.49	64.10	64.79
w/o Obj.	53.27	51.87	52.56	70.44	68.66	69.54	65.63	63.90	64.75
w/o DKA & KDF	51.73	51.20	51.46 ($\downarrow 1.60$)	69.41	68.78	69.09 ($\downarrow 0.88$)	64.12	63.47	63.79 ($\downarrow 1.50$)

Note: Bold values indicate the best results. ↓ indicates the decrease in F1 points relative to the full MKIF model.

The detailed discussions are as follows:

• Effectiveness of DKA. As demonstrated in Table 6, removing DKA leads to a 2.22% relative decrease in F1 on FMNERG, indicating its contribution to distinguishing semantically similar fine-grained categories. The results also validate that global-level knowledge helps filter out sample-irrelevant information. Conceptual descriptions provide category-level semantics, while discriminative descriptions further highlight inter-class differences. Such complementary information provides more effective supervision for fine-grained type disambiguation.
• Effectiveness of KDF. As presented in Table 6, aligning the entities only with raw object embeddings results in a 2.04% relative decrease in F1 on FMNERG. Moreover, adding either global-level or object-level knowledge individually improves performance. This shows that global-level knowledge provides sample-specific contextual cues and object-level knowledge captures concept-level visual semantics. Meanwhile, the gated fusion mechanism enables the model to adaptively integrate the two knowledge sources into object representations, thereby learning complex entity–object relations robustly.

Second, we conducted an ablation study to evaluate how removing specific information requests from the global-level knowledge generation prompt for GPT-4o-mini affected the performance of MKIF. Specifically, based on the full prompt, we constructed three ablated variants by removing one information requirement at a time, including background knowledge, named entities, and visual concepts. The F1 scores of FMNERG under different prompt templates are reported in Table 7.

Table 7. Ablation study of different prompt templates for obtaining global-level knowledge.

Prompt Templates for Obtaining Global-level Knowledge	F1
Please provide brief background knowledge of this tweet, tell me the named entities in the text, and tell me the visual concepts in the image briefly. (full prompt)	53.06
Please tell me the named entities in the text, and tell me the visual concepts in the image briefly. (w/o background knowledge)	50.64
Please provide brief background knowledge of this tweet, and tell me the visual concepts in the image briefly. (w/o named entities)	52.05
Please provide brief background knowledge of this tweet, and tell me the named entities in the text. (w/o visual concepts)	50.71
TIGER (non-knowledge-based baseline)	47.20

Note: Bold and underlined F1 scores indicate the best and second-best results, respectively. Bold italic text highlights the prompt settings and the baseline.

As shown in Table 7, the full prompt yields the highest F1 score, while all ablated versions still achieve strong performance. The results suggest that the effectiveness of MKIF is not sensitive to a particular prompt template, and MKIF can consistently benefit from MLLM-generated knowledge under different prompt formulations, verifying the robustness of our approach. Meanwhile, removing background knowledge, named entities, or visual concepts each causes a performance drop to different extents, which indicates that the three queried information types are complementary rather than redundant in global-level knowledge construction. In particular, removing background knowledge or visual concepts from the instruction leads to a significant performance decline, indicating the importance of sample-level contextual semantics, and demonstrating that the image-side concepts provide useful cross-modal cues for enhancing entity–object alignment. By contrast, removing the named-entity-related instruction results in a relatively smaller performance drop, indicating that MKIF can effectively capture entity-related information from the original textual input and span representations, rather than relying on entity mentions provided by the MLLM.

4.4. Analysis and Discussion

To better understand the effectiveness of MKIF, we evaluated the contribution of DKA to distinguishing semantically similar entity types, the semantic characteristics of the discovered visual concepts and the sensitivity of MKIF to the number of the visual concepts, and the performance of MKIF with different MLLMs as knowledge sources. These aspects were investigated through the following research questions.

• ▸RQ1: Does DKA improve the disambiguation of semantically similar fine-grained entity types in MNER?

A: To provide an intuitive illustration of DKA’s contribution, we visualize several test instances of both MKIF and MKIF w/o DKA on the MNER subtask in Figure 4.

Figure 4. MNER predictions of MKIF and its ablated version. We visualize the predicted probability distributions for several test samples, showing the top-5 classes. In case #1, the conceptual descriptions enable the model to infer that the tweet content is inconsistent with sports team, thereby correctly predicting the entity as city. In case #2, the discriminative descriptions assist the model in distinguishing the semantically similar entity types athlete and artist.

In case #1, the entity LuverneMN belongs to the city category, but the model without DKA incorrectly recognizes it as a sports team. In contrast, by leveraging the conceptual description, the full model captures the concept that a sports team is “a group of players competing in sports, with visuals including uniforms, team logos, and group huddles”, which is inconsistent with the visual evidence. Consequently, the full model decreases the prediction probability of class sports team and correctly recognizes the entity as a city. In case #2, removing DKA results in ambiguous predictions between athlete and artist when extracting the entity Gwen, although the input image suggests an athlete. This occurs because the ablated model fails to capture the discrimination that athletes are characterized by physical fitness and competition, whereas artists focus on creative expression and artistic techniques. These observations indicate that DKA plays a crucial role in mitigating ambiguous predictions by exploiting dual-level knowledge supervision through contrastive learning.

• ▸RQ2: How does visual concept discovery in KDF contribute to the effectiveness of MKIF?

A: We first illustrate representative visual concepts discovered from the training set. As shown in Figure 5, unsupervised clustering identifies coherent visual concepts, which provide complementary semantic cues for enriching object representations during gated fusion.

Figure 5. Representative visual concepts discovered from the training set via unsupervised clustering.

Next, we varied the number of visual concepts V from 50 to 500 and report the performance of MKIF under different settings in Figure 6. From the results, we can observe that MKIF achieves the best performance when $V=100$, while F1 decreases when V becomes higher or lower. This can be explained as follows: when V is too small, the potential visual concepts in the dataset cannot be adequately discovered, leading to semantic inconsistency among objects within the same cluster; but when V is too large, the excessively fine-grained clustering may split semantically similar objects into different clusters, introducing redundant information.

Figure 6. The performance of MKIF under different numbers of visual concepts V discovered from the dataset.

To complement our analysis of the number of visual concepts, we further analyzed the sensitivity of MKIF to different clustering algorithms in Appendix D.

• ▸RQ3: How does the choice of MLLM affect the effectiveness of MKIF?

A: We selected two representative open-source MLLMs as alternative sources of both global- and label-level knowledge. LLaVA-1.5-7B [24] is a widely used baseline for visual instruction tuning with a simple and reproducible design, while InternVL-2.5-8B [63,64] represents more recent advances through dynamic-resolution image processing and training on diverse vision-language tasks. As shown in Table 8, we compared all knowledge-based variants with the non-knowledge-based method TIGER.

Table 8. Performance comparison with different MLLMs in global- and label-level knowledge generation.

Model	FMNERG			MNER			EEG
	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1
TIGER	47.57	46.85	47.20	64.43	65.40	64.91	62.44	61.49	61.96
Ours (LLaVA-1.5-7B)	50.89	49.63	50.25	68.75	67.13	67.93	63.95	62.37	63.15
Ours (InternVL-2.5-8B)	52.21	51.12	51.66	69.04	67.68	68.35	65.14	63.78	64.46
Ours (GPT-4o-mini)	53.97	52.18	53.06	71.13	68.86	69.97	66.41	64.22	65.29

Note: Bold values indicate the best results.

The results show that all knowledge-based variants consistently outperform TIGER, demonstrating the effectiveness of incorporating global- and label-level knowledge. Although the performance varies with the adopted MLLM, MKIF remains effective across different knowledge sources, indicating that its gains are not tightly coupled to a specific MLLM. Compared with GPT-4o-mini, LLaVA-1.5-7B and InternVL-2.5-8B reduce FMNERG F1 by 2.81 and 1.40 percentage points, respectively. These performance gaps reflect differences in the relevance and semantic richness of global-level knowledge for each input image–text pair, as well as differences in the conceptual and discriminative quality of label-level knowledge. Therefore, we use GPT-4o-mini as the default knowledge source in our final model.

More capable MLLMs may further improve the quality of generated knowledge. However, as shown in Table 2, Twitter-FMNERG contains 10,000 image–text pairs, and generating sample-specific global-level knowledge across this dataset makes API cost an important consideration in model selection. GPT-4o-mini therefore offers a practical balance between the performance demonstrated above and knowledge-generation cost. Moreover, the design of MKIF allows more capable MLLMs to serve as sources of both global- and label-level knowledge without requiring architectural changes.

4.5. Case Study

To gain deeper insights into the superiority of MKIF in distinguishing semantically similar fine-grained entity types and robust entity–object alignment, we compare its predictions with those of TIGER on representative test samples in Figure 7.

Figure 7. Cases selected from the test set of Twitter-FMNERG. The wrong predictions are highlighted in red.

In case #1, both MKIF and TIGER correctly detect the corresponding region of entity Shego in the image, but compared to TIGER, our model correctly recognizes it as a character rather than a musician. Such improvement can be attributed to DKA, which captures the distinction that characters are fictional figures in stories, animations or games, while musicians are involved in creating and performing music. Consequently, our model correctly extracts the triple from a tweet about cartoons. In case #2, although TIGER correctly recognizes the entity Morehead State University and its type, it fails to link the entity to its corresponding visual object. Moreover, in the more challenging case #3, TIGER incorrectly matches MV-22 with a human and links the country Dutch to plain text. In contrast, our method successfully captures the implicit relation between the entity in the educational institution category and the visual concept logo, as well as the correlation between product and transportation. Therefore, MKIF accurately detects the regions of Morehead State University and MV-22 in the images.

These cases illustrate that MKIF is more effective than TIGER in resolving semantic ambiguities among fine-grained entity types and robustly capturing complex entity–object relations, thereby addressing the key challenges of FMNERG.

5. Conclusions

In this paper, we proposed MKIF, a novel framework that leverages multi-level knowledge to jointly address two key challenges in FMNERG: distinguishing semantically similar fine-grained entity types and robustly aligning textual entities with visual objects. We first acquired global-level knowledge and designed a knowledge-enhanced span representation module to generate knowledge-enriched entity span embeddings. To further improve the discrimination of fine-grained entity types, we proposed a dual-level knowledge-guided alignment (DKA) module, which exploits both global- and label-level knowledge as supervision via contrastive learning. Moreover, we proposed a knowledge discovery-enhanced fusion (KDF) module that integrates object-level knowledge with global-level knowledge into object embeddings through a dynamic gating mechanism, enabling robust modeling of entity–object relations. Comprehensive experiments demonstrated that MKIF consistently outperformed state-of-the-art methods, and detailed analyses validated the effectiveness of its core components. In future work, we plan to equip our framework with continual learning capabilities, enabling better adaptation to the dynamically evolving content on social media, and to explore privacy-aware FMNERG pipelines that protect the privacy of multimodal content while maintaining downstream performance.