Mix-Persona Comment Generation and Geographically Enhanced Context Retrieval for LLM Fine-Tuning in Multimodal Crisis Post Classification

Abstract

Social media has become a vital source for humanitarian organizations to gather information during crises. However, existing multimodal classification methods operate primarily as isolated systems, while neglecting external references crucial for accurate judgment. Furthermore, while user comments can provide valuable context, they are often scarce during the early stages of a crisis. To address these limitations, we propose a framework named Mix-Persona Comment Generation with Geographically Enhanced Context Retrieval for LLM Instruction Fine-tuning (MPCG-GECR). To mitigate comment scarcity, we employ a Synthetic Persona Generator (SPG) that prompts LLMs to adopt diverse mix-personas, generating synthetic comments that simulate multi-perspective public discourse. To incorporate external references, we introduce a Geographically Enhanced Context Retrieval (GECR) module. Unlike standard retrieval approaches, GECR utilizes a hybrid re-ranking strategy to identify samples that are both multimodally similar and geographically consistent, serving as reliable reference anchors for the LLM. By integrating these social perspectives and geographic references into a unified instruction-tuning format, we transform the classification task into a context-aware text generation problem and fine-tune the LLM using Low-Rank Adaptation (LoRA). Extensive experiments on the CrisisMMD and DMD datasets demonstrate that MPCG-GECR effectively overcomes data scarcity and context isolation, significantly outperforming existing methods.

1. Introduction

The widespread adoption of social media platforms has rendered them a crucial channel for humanitarian organizations to obtain information about crisis events [1]. Accurately and rapidly analyzing these highly time-sensitive multimodal posts can effectively assist humanitarian organizations in planning relief operations [2].

Existing methods for multimodal crisis post classification primarily rely on multimodal feature fusion, as illustrated in Figure 1a. These approaches employ various pre-trained models to extract features from different modalities, feeding them into downstream fusion modules for feature integration [3,4,5,6]. Although progress has been made, these methods operate within a closed system, relying solely on the model’s internal parametric knowledge of the models. While Lin et al. [7] introduced Large Language Models (LLMs) to extract event-related information, they restricted the LLM’s role to simple querying, failing to leverage its reasoning capabilities effectively or connect the posts to broader external contexts.

To enhance the semantic depth of social media posts, some existing methods (though not specifically designed for crisis post analysis) incorporate user comments as auxiliary text [8,9,10], as shown in Figure 1b. However, relying on real-world comments presents a significant bottleneck: during the early stages of a crisis, immediate user engagement is often scarce. Many posts contain no comments due to the urgency of the situation or limited visibility. Specifically, not all posts attract user engagement, and the time constraints of crisis data collection further limit the window for users to post reactions. This “comment scarcity” renders methods dependent on crowd-sourced feedback ineffective precisely when they are most needed.

Furthermore, current approaches overlook a fundamental aspect of human cognition in crisis assessment: reference via retrieval. When experts analyze a crisis post, they do not examine it in isolation; they intuitively compare it with similar historical events or environmental contexts to make informed judgments [11,12]. In the broader field of Natural Language Processing (NLP), Retrieval-Augmented Generation (RAG) has achieved remarkable success by grounding LLM responses in external non-parametric knowledge, thereby reducing hallucinations and improving accuracy [13,14,15]. Despite its promise, RAG has not yet been effectively adapted for multimodal crisis post analysis. A naive application of standard retrieval methods is insufficient because crisis posts are characterized by unique geographic semantics. Merely retrieving similar text and images without considering geographic consistency (e.g., retrieving a flood image from a tropical zone to match a flood query in an arid zone) can introduce noise and mislead the model.

To address these limitations—specifically the scarcity of user interaction and the lack of geographically grounded reference information—we propose the Mix-Persona Comment Generation with Geographically Enhanced Context Retrieval for LLM Instruction Fine-tuning (MPCG-GECR) framework. Our pipeline is illustrated in Figure 1c. First, to solve the comment scarcity issue, we employ a Synthetic Persona Generator (SPG) that prompts the LLM to act as hundreds of diverse mix-personas, generating varied user comments that simulate multi-perspective public discourse. Second, to incorporate high-value external context, we introduce a Geographically Enhanced Context Retrieval (GECR) module. Unlike traditional retrieval methods, GECR considers both multimodal feature similarity and geographic semantics (e.g., location names, functional zones). By performing a hybrid re-ranking strategy, GECR retrieves samples that serve as “Geographic Reference Anchors,” providing the LLM with objective evidence to ground its reasoning. Finally, we integrate these synthetic subjective perspectives and retrieved objective references to construct a rich instruction fine-tuning dataset. We employ Low-Rank Adaptation (LoRA) to efficiently fine-tune the LLM, transforming the classification task into a context-aware generation task.

Crucially, the SPG and GECR modules are active not only during training but also during the inference phase. When handling new, unseen tweets, the system dynamically augments them with synthetic viewpoints and retrieved historical context before classification. This ensures that the model operates on a consistent data distribution across both training and testing phases. Furthermore, a critical distinction exists between the two modules: while SPG synthesizes subjective social perspectives to simulate public discourse, GECR strictly retrieves objective real-world historical data to serve as factual grounding.

By organically combining diverse subjective perspectives (via SPG) with objective geographic references (via GECR), our framework enables the LLM to mine underlying crisis information with unprecedented depth. Experiments demonstrate that MPCG-GECR outperforms other methods on the CrisisMMD dataset [16] and the DMD dataset [17].

The main contributions of this paper are as follows:

• We propose the MPCG-GECR framework, a novel approach that merges synthetic social media user comments and retrieval-augmented geographic context to address the data scarcity and context isolation problems in multimodal crisis post classification. Extensive experiments show that our method outperforms existing baselines.
• We introduce the Geographically Enhanced Context Retrieval (GECR) module, representing the first attempt to integrate geographic semantics into the RAG paradigm for crisis post analysis. This module provides the LLM with interpretable geographic reference anchors, improving classification reliability.
• We demonstrate the effectiveness of prompting LLMs to generate diverse personas and comments for data augmentation, offering a new direction for handling low-resource crisis post analysis tasks.

2. Related Work

2.1. Traditional Multimodal Crisis Post Classification

Early research in multimodal crisis post classification predominantly relied on multimodal feature fusion techniques. These methods typically employ separate encoders for text and image modalities, followed by fusion modules to integrate cross-modal representations. For instance, Abavisani et al. [3] aimed to address data inconsistency and overfitting through two key mechanisms: Stochastic Shared Embeddings regularization for robust training and Cross-Attention-based fusion of projected image and text embeddings. Qian et al. [4] utilized a multimodal masked transformer network to capture cross-modal semantic relationships, while filtering out redundant information. More recently, Wang et al. [5] adopted a multi-layer architecture combining autoencoders and graph convolutional networks to extract comprehensive information from multimodal data. Yu et al. [6] incorporated uncertainty estimation into the multimodal fusion process, treating encoder outputs as subjective opinions to learn more reliable representations.

While these approaches have demonstrated utility, they operate within a closed-system paradigm, relying exclusively on the parametric knowledge embedded in the models. This restricts their capacity to incorporate real-time or contextually relevant external knowledge, which is often critical for understanding evolving crisis situations.

2.2. Comment-Augmented Social Media Analysis

Beyond crisis-specific applications, several studies in related domains, such as fake news detection and rumor verification have explored user comments as valuable auxiliary signals. Zheng et al. [8] proposed the Multi-modal Feature-enhanced Attention Networks, which integrate textual, visual, and social graph features into a unified framework. Their approach specifically focuses on aligning and complementing relationships across different modalities to enhance rumor detection. Similarly, Su et al. [10] constructed a dual-layer graph comprising both a news layer and a user layer. By mining multi-relations and user interaction features, their model effectively captures user credibility signals to assist in distinguishing fake news. Nan et al. [9] introduced a teacher–student framework named CAS-FEND. This method transfers knowledge from a comment-aware teacher model trained on historical data to a content-only student model, designed to preserve detection accuracy even when social contexts are absent in newly emerging news.

However, these methods depend on the availability of sufficient user engagement or established historical social contexts. This reliance becomes a significant limitation in crisis contexts where posts often receive few or no comments due to urgency, low visibility, or the nascent stage of the event. Consequently, the problem of comment scarcity makes approaches dependent on crowd-sourced feedback or social graph accumulation ineffective during the critical early phases of crisis response.

2.3. LLM Applications and Retrieval-Augmented Generation

Recent advancements have demonstrated the potential of LLMs and Retrieval-Augmented Generation (RAG) in analyzing complex social media content, particularly in the domain of fake news detection. Zheng et al. [18] proposed the Explainable Adaptive Rationale-Augmented Multimodal framework. They utilized Large Vision-Language Models to generate analyses and employed task-specific small models to extract useful rationales, thereby enhancing both generalization and explainability. To exploit the synergy between different model sizes, Zhou et al. [19] introduced the Multi-Round Collaboration Detection framework. This approach features a two-stage retrieval module to select relevant demonstrations and integrates the generalization abilities of LLMs with the specialized functionalities of small models through iterative learning. Similarly, Irnawan et al. [20] emphasized the importance of factual grounding. They proposed a method that assesses claim veracity primarily based on retrieved factual evidence, utilizing the common-sense reasoning mechanisms of LLMs only when explicit evidence is absent.

Despite their success in general NLP and fake news detection, RAG and LLMs have seen limited application in multimodal crisis settings. Standard retrieval methods often ignore the critical role of geographic consistency. This oversight can lead to misleading matches, for example, retrieving posts from unrelated regions that lack relevant geographic context. Moreover, existing LLM-based crisis post analysis approaches, such as that of Lin et al. [7], typically treat LLMs as passive knowledge bases for simple information extraction. They fail to fully leverage the generative and reasoning capabilities of LLMs in an integrated manner crucial for the accurate analysis of crisis posts.

2.4. Geographic Information in Social Media Analysis

The integration of geographic information has become a pivotal enhancement in social media analysis, particularly for crisis management. Unlike general text classification, crisis-related posts are inherently tied to specific spatio-temporal contexts. Recent studies have demonstrated that incorporating geographic attributes can significantly improve the granularity and accuracy of social media mining.

Several approaches focus on the precise extraction and inference of geographic locations from unstructured data. Wei et al. [21] fused the mention and retweet graphs with a gating mechanism to infer user locations. Similarly, Han et al. [22] employed fine-tuning on open-source LLMs to extract crisis geospatial intelligence and structured spatial entity attributes from Chinese social media texts.

Beyond mere location extraction, geographic information serves as a critical bridge for multi-task learning and multi-source data fusion. Zou et al. [23] introduced a framework that integrates toponym extraction with damage identification. By designing toponym-enhanced weights, their model utilizes geographic references to improve the representation of crisis severity. Zhu et al. [24] combined social media texts with microwave remote sensing images. Their framework matches textual traffic information with basic geographic data to assess road damage levels under extreme flood conditions. Furthermore, Zorenböhmer et al. [25] applied an aspect-based emotion analysis to wildfire tweets, revealing that emotion patterns, such as fear and sadness, exhibit distinct variations correlated with the distance to wildfire perimeters.

These studies collectively underscore a fundamental argument: geographic information is not merely a metadata tag but a semantic anchor that grounds social media content in physical reality. In crisis analysis, the interpretation of a post often depends on its location. However, existing methods primarily treat geographic information as an extraction target or a visualization tool. They rarely utilize geographic consistency as a retrieval signal to guide the reasoning process of the model. This limitation motivates our approach, where we leverage geographic semantics to retrieve contexts, thereby enabling the LLM to make more informed and grounded judgments.

3. Method

This paper proposes a Mix-Persona Comment Generation with Geographically Enhanced Context Retrieval for LLM Instruction Fine-tuning (MPCG-GECR) framework, as illustrated in Figure 2. The framework comprises four primary components: Image Information Extraction, User Comment Generation, Geographically Enhanced Context Retrieval, and Instruction Dataset Construction with LLM Fine-tuning. Crucially, our framework integrates “Subjective Generation” and “Objective Retrieval” within a unified inference pipeline. The User Comment Generation module synthesizes diverse social personas to address the scarcity of interaction. In contrast, the Geographically Enhanced Context Retrieval (GECR) module grounds reasoning in reality by retrieving verifiable real-world historical samples based on geographic and multimodal similarities. Both components are utilized during inference to construct a comprehensive context for the LLM. The following sections provide a detailed description of the proposed MPCG-GECR framework.

3.1. Image Information Extraction

To enable the joint processing of images and text using LLMs, we first employ a Multimodal Large Language Model (MLLM) to extract image information, while simultaneously incorporating common-sense knowledge related to the image content provided by the MLLMs. Specifically, we utilize LLaVA [26,27] as the MLLM, replacing the original visual encoder of LLaVA with FG-CLIP [28] to enhance its capability in fine-grained image understanding. FG-CLIP is a powerful pre-trained multimodal model that achieves fine-grained visual and textual alignment.

Subsequently, we use a fixed prompt template to instruct the MLLM to generate detailed descriptions of the image. The prompt template is shown in Figure 3:

Here, [Tweet text] and [Tweet image] denote the original content of the tweet. We store the response generated by the MLLM as [Detailed image caption].

The primary objective of this process is to bridge the modality gap by translating visual content into textual descriptions. Since LLMs currently exhibit superior reasoning and instruction-following capabilities compared to MLLMs, converting images into text allows us to leverage the robust analytical power of LLMs in the subsequent stages. Furthermore, this process serves to explicitly externalize fine-grained visual details and implicit common-sense knowledge into a unified textual format, providing the model with a richer context for accurate analysis. It is worth noting that this stage focuses solely on visual content extraction. The verification of geographic entities and consistency alignment is rigorously addressed in the Geographically Enhanced Context Retrieval (GECR) module, as detailed in Section 3.3.

3.2. Diverse Comment Generation

We design a Synthetic Persona Generator (SPG) to generate sufficiently diverse personas. Specifically, we select five distinct attributes of social media users: Gender, Age, Education, Occupation, and Location. The Location attribute is particularly relevant to crisis events. Categorizing users by their geographic relationship to the crisis area is designed to activate the LLM’s understanding of the differing perspectives between affected and non-affected populations. This is a key aspect closely aligned with our task. The other four attributes are typical user characteristics. Prior research suggests that differences in these attributes lead people to express divergent views in their comments [29,30]. Gender differences reflect varied interests and perspectives; age reflects differences in cognitive maturity; education level corresponds to disparities in knowledge acquisition; while occupation represents disparities in professional expertise and skills. Prior research in crisis informatics suggests that user demographics significantly influence information-sharing behaviors [31,32,33,34]. For instance, professional backgrounds (Occupation) often correlate with the sharing of actionable situational awareness, while educational levels can impact the linguistic formality and logical structure of the text. By explicitly modeling these attributes, we enable the SPG to simulate the semantic divergence between `emotional noise’ and `actionable intelligence,’ thereby aiding the classifier in learning robust decision boundaries.

The specific attribute categories used are as follows:

• Gender: Non-binary, Male, Female;
• Age: Minors, Young Adulthood, Middle Age, Old Age;
• Education: Primary Education, Secondary Education, Undergraduate Education, Graduate Education;
• Occupation: Humanitarian Worker, Domain expert, Engineer, Medical Staff, Journalist, General public;
• Location: Resident of the local crisis area, Resident of the surrounding crisis areas, Resident of the non-affected area.

We construct diverse mix-personas by combining different attributes, as illustrated by User 1 in Figure 2. Some combinations are inherently contradictory, such as having both Minors and Graduate Education. To prevent the LLM from generating hallucinations, we filter out such clearly inconsistent combinations during the persona construction process, such as [Male, Minors, Graduate Education, Domain expert, Resident of the non-affected area]. Specifically, the inclusion of the `General Public’ category in the Occupation attribute is designed to model the common scenario where specific professional expertise is absent or unknown, ensuring the model covers the baseline perspective of the average user.

After obtaining the mix-persona, we generate comments using the prompt template shown in Figure 2. Specifically, we decompose the mix-persona into individual attribute descriptions to guide the LLM in accurately simulating the specified character. Meanwhile, we assign the LLM a specific task and instruct it to mimic real Twitter users when generating comments. We provide the LLM with tweet text, image caption, and comment list to generate new comments while taking into account the existing comments.

Notably, we adopt a multi-round iterative strategy to generate the comment list, which aligns closely with the Chain-of-Thought (CoT) paradigm [35,36]. In each round, the LLM is assigned only one new persona that it has not previously simulated and is prompted to produce a corresponding comment, rather than generating a full comment list for multiple personas at once. This CoT-like strategy helps prevent the LLM from becoming confused by multiple conflicting persona attributes, thereby reducing the risk of introducing additional hallucinations during comment generation. Due to the context length limitations of the LLM, each [Comment list] contains only a limited number of comments. To accommodate a large volume of diverse comments, we generate multiple [Comment list] instances for the same tweet. We also ensure that the persona attributes differ across various [Comment list] instances for the same tweet to maintain diversity. Although we do not explicitly model temporal dynamics, the generation is conditioned on the tweet’s content, which serves as an implicit temporal anchor (e.g., descriptions of ‘shaking ground’ trigger immediate reactions, while ‘rescue arrival’ triggers post-event comments). This allows the model to dynamically adapt to the crisis stage depicted in the post.

In summary, the implementation of the Synthetic Persona Generator (SPG) aims to simulate the multifaceted nature of public discourse during a crisis. By varying key demographic attributes—particularly Location—we enable the model to capture the distinct cognitive perspectives of both affected populations and remote observers. The primary benefit of this approach, reinforced by our multi-round iterative generation strategy, is the creation of a high-quality heterogeneous comment corpus that minimizes hallucinations while maximizing the diversity of social viewpoints.

However, while the SPG simulates geographic perspectives via persona attributes, it relies on the internal knowledge of the LLM and lacks connection to specific real-world external data. To complement this simulated social diversity with objective data-driven evidence, we introduce the Geographically Enhanced Context Retrieval (GECR) module in the following section.

3.3. Geographically Enhanced Context Retrieval

To assist the LLM in incorporating broader information for the accurate analysis of crisis-related posts, we adopt a Retrieval-Augmented Generation (RAG) approach to collect high-value reference information. Specifically, we introduce the Geographically Enhanced Context Retrieval (GECR) module, marking its first application in crisis post analysis. Unlike existing methods that perform retrieval based solely on feature information, our GECR module incorporates a retrieval process targeting geographic semantics. This allows us to simultaneously retrieve high-value samples that share both similar multimodal features and geographic semantic information with the target sample, thereby providing the model with more explicit references. Unlike the synthetic comments generated in Section 3.2, the geographic context here is retrieved from real-world historical data (training set) to serve as objective grounding.

The GECR process consists of three steps: Geographic Entity Extraction and Indexing, Multimodal Feature Retrieval, and Hybrid Re-ranking with Geographic Semantics.

3.3.1. Geographic Entity Extraction and Indexing

We first employ lightweight NER tools (BERT-NER [37,38]) and OCR tools (EasyOCR [39]) to process multimodal data. Specifically, for the target sample and all samples in the training set, we use the OCR tool to extract all textual information from images. Subsequently, we utilize the NER tool to extract Geo-Entities (e.g., Countries, States, Cities, Towns) and Location-Entities (e.g., river names, street names, landmarks, functional zones).

This extracted information is inserted into the original training set data and stored under two new distinct tags: Geo and Location. This transformation converts unstructured user-generated content into structured geographic metadata, successfully constructing a retrieval database enriched with geographic semantics. This step serves as the foundation for the subsequent geographic semantic retrieval process.

3.3.2. Multimodal Feature Retrieval

We utilize the FG-CLIP encoder to extract text and image features from the target sample and all training set samples. We then calculate the multimodal similarity between the target sample and other samples, respectively. Specifically, for a multimodal target sample $X_i=\{T_i,V_i\}$ and a training set sample $X_j=\{T_j,V_j\}$, we denote the unimodal features extracted by FG-CLIP as $t_i,v_i,t_j,v_j$. We use $\cos (·)$ to represent the normalized cosine similarity between two features. For example, the similarity between $t_i$ and $t_j$ is expressed as

$${\mathrm{Cos}}_{t_i,t_j}={\displaystyle \frac{t_i·t_j}{\parallel t_i\parallel \parallel t_j\parallel }}.$$

(1)

Here, $\parallel ·\parallel$ indicates the L2 norm, used to normalize the feature vectors. Given the strong cross-modal alignment capabilities of FG-CLIP, potential associations between cross-modal contents are already considered during encoding. Therefore, we employ a cross-modal similarity fusion method to comprehensively measure multimodal similarity from four different cross-modal directions. Finally, the average similarity $S_{i,j}$ between sample $X_i$ and $X_j$ is calculated as

$${\mathrm{S}}_{i,j}={\displaystyle \frac{{\mathrm{Cos}}_{t_i,t_j}+{\mathrm{Cos}}_{t_i,v_j}+{\mathrm{Cos}}_{v_i,t_j}+{\mathrm{Cos}}_{v_i,v_j}}{4}}.$$

(2)

Using the similarity calculation method above, we match the features of the target sample $X_i$ against all samples in the retrieval database (training set). We select the Top-K samples with the highest similarity as candidate samples following this coarse-grained multimodal feature retrieval. These K candidates similar to $X_i$ form the sample candidate pool $R_i=\{X_{i_1},X_{i_2},\dots ,X_{i_k}\}$.

In this step, we establish a preliminary candidate pool based on multimodal features; these candidates are already similar to the target sample at the feature level, providing a solid basis for the subsequent fine-grained geographic re-ranking.

3.3.3. Hybrid Re-Ranking with Geographic Semantics

Upon obtaining the candidate pool $R_i$, we further employ a Hybrid Re-ranking Strategy based on geographic semantics to select high-value samples that provide additional context for the LLM’s reasoning. This strategy comprises two distinct branches: Explicit Alignment and Implicit Inference. Regardless of whether the target sample possesses explicit geographic semantic tags, the dual guarantee provided by these branches ensures that the hybrid re-ranking strategy effectively retrieves the most discriminative samples from the candidate pool.

Before re-ranking, we first check the geographic semantic information of the target sample. If the target sample contains geographic semantic information (i.e., it has Geo and Location tags), we execute the Explicit Alignment strategy.

Explicit Alignment strategy. We first divide the candidate pool into two groups based on the target sample’s Geo and Location tags: those that share the same tags and those that do not.

• Same-Tag Group: We employ the FG-CLIP encoder to extract feature embeddings for the Geo and Location tags of both the target and candidate samples, subsequently computing the geographic semantic similarity between them. Based on this metric, we re-rank all candidate samples in descending order. In cases where the geographic semantic similarity between two candidate samples and the target sample is identical, their relative order is determined by their multimodal similarity. Finally, we retrieve the Top-2 samples from the re-ranked list. Here, ‘Top-2’ refers to the two candidates with the highest similarity scores in the sorted list, serving as the most reliable geographic reference anchors.
• Different-Tag Group: To supplement extra diversity, we also retrieve information from the group with different tags. Exposure to diverse data enables models to learn essential patterns, mitigate overfitting, and develop a more holistic reasoning capability [40]. This method of supplementing diverse information has been proven beneficial in fields such as Visual Question Answering (VQA), enabling models to make more comprehensive and accurate inferences [41]. For this group, we rank based on multimodal similarity and retrieve the Top-2 samples.

In cases where the “Same-Tag” group contains fewer than two samples, the deficit is filled by retrieving additional samples from the “Different-Tag” group.

Ultimately, we select four high-value samples from the candidate pool $R_i$ to constitute the final retrieved set $R_i^{\prime }$. This set includes not only samples exhibiting explicit environmental similarity and phenomenological consistency but also supplementary samples that share similar multimodal features yet possess distinct geographic semantics. The choice of a total retrieval size of $K=4$ and the specific $2+2$ allocation strategy are grounded in our experimental findings (see Section 4.6) and a design philosophy of balance. Specifically, retrieving two samples from the “Same-Tag” group provides a robust reference average to mitigate the risk of outliers inherent in single-sample retrieval. Simultaneously, allocating the remaining two slots to the “Different-Tag” group prevents the marginalization of diversity benefits that would occur with a skewed (e.g., 3-1) split. This strategy ensures that the retrieved context maintains clear environmental similarity and phenomenological consistency while simultaneously offering valuable supplementary information.

If the target sample lacks geographic semantic information (i.e., no Geo or Location tags), we execute the Implicit Inference strategy. This serves as a data-driven form of latent geographic inference.

Implicit Inference strategy. We implement two distinct processing strategies depending on the availability of event type information in the training set:

• If event type information is available in the training set (e.g., the CrisisMMD dataset [16] annotates specific event names), we group the samples in the candidate pool by event type and rank them within each group by multimodal similarity (high to low). We then select the Top-2 samples from the largest group, and the Top-1 sample from the second and third largest groups, respectively. If only two groups exist, we select the Top-2 from each; if only one group exists, we simply select the Top-4.
• If event type information is unavailable (e.g., the DMD dataset [17] only annotates sample categories), we implement a latent semantic clustering strategy. We employ the K-Means++ algorithm [42] to cluster samples based on the multimodal features of the target and candidate pool. The construction of a multimodal feature involves three steps:
  • Pre-normalization: We first apply L2 normalization to the single-modal text and image features extracted by FG-CLIP. This step is crucial to address the modality gap phenomenon [43]. Without this, the modality with naturally larger feature magnitudes (typically vision) would dominate the distance calculation, causing the model to ignore textual semantics [44].
  • Concatenation: We concatenate the normalized multimodal features to form a fused vector, ensuring an equal contribution (1:1) from both modalities.
  • Post-normalization: Finally, we perform a second L2 normalization on the fused features. This ensures that the Euclidean distance used in K-Means effectively approximates Cosine Similarity, which is the native metric for CLIP-based embeddings.

• The number of clusters K is set to a small constant (e.g., $K=4$) to impose a structured diversity on the retrieved candidates, forcing the selection algorithm to sample from distinct latent semantic sub-groups rather than over-focusing on a single dominant pattern. From the clustering results, we select the Top-2 samples from the cluster containing the target sample and the Top-1 sample from each of the two clusters with the nearest centroids. In the specific case where the target sample forms a cluster of its own (i.e., a singleton cluster), we identify the nearest clusters by calculating the Euclidean distance between the target sample and the centroids of all other clusters. We then select the Top-2 samples from each of the two clusters with the smallest centroid distances. In cases where $K\ne 4$, we continue to retrieve the Top-2 samples from within the target sample’s cluster, while selecting the two samples with the highest similarity from the nearest neighboring clusters.

Through this implicit inference, we ensure that, even when the target sample lacks specific geographic semantic information, we can implicitly infer it through event categories or feature similarity and form the retrieved set $R_i^{\prime }$. This strategy effectively retrieves context with environmental similarity and phenomenological consistency based on environmental morphology and crisis characteristics, while also supplementing diversity rather than limiting results to samples that are merely highly similar to the target.

In summary, by introducing the GECR module, we overcome the limitations of traditional retrieval methods that ignore geographic semantics. Whether through explicit alignment or implicit inference, this module ensures that retrieved samples remain maximally similar to the target in terms of geospatial and environmental context without losing diversity. These samples not only share similar text and images with the target, but also serve as critical geographic reference anchors, providing bases for the LLM to evaluate the content of crisis posts. The complete workflow of the proposed GECR module is summarized in Algorithm 1.

Algorithm 1 Geographically Enhanced Context Retrieval (GECR)

Require:  Target sample $X_i=\{T_i,V_i\}$, Training set $\mathcal{D}$, Retrieval size K, Final size $N=4$. Ensure:  Retrieved context set ${\mathcal{R}}_i^{\prime }$.   1: // Step 1: Multimodal Feature Retrieval   2: Extract features $f_i=\mathrm{FG}-\mathrm{CLIP}\left(X_i\right)$, $f_j=\mathrm{FG}-\mathrm{CLIP}\left(X_j\right),\forall X_j\in \mathcal{D}$.   3: Calculate similarity $S_{i,j}$ via Equation (2) for all $X_j\in \mathcal{D}$.   4: ${\mathcal{R}}_i\leftarrow \mathrm{TopK}(\mathcal{D},S_{i,j},K)$. {Initial Candidate Pool}   5: // Step 2: Hybrid Re-ranking Strategy   6: if $X_i$ has Geo/Location tags then   7:     Branch A: Explicit Alignment   8:     Split ${\mathcal{R}}_i$ into ${\mathcal{G}}_{same}$ (share tags) and ${\mathcal{G}}_{diff}$ (different tags).   9:     Sort ${\mathcal{G}}_{same}$ by tag similarity (descending). 10:     Sort ${\mathcal{G}}_{diff}$ by multimodal similarity $S_{i,j}$ (descending). 11:     Select top samples from ${\mathcal{G}}_{same}$: ${\mathcal{S}}_1\leftarrow {\mathcal{G}}_{same}[0:\min (2,|{\mathcal{G}}_{same}\left|\right)]$. 12:     Calculate remaining quota: $N_{rem}=4-\left|{\mathcal{S}}_1\right|$. 13:     Select top samples from ${\mathcal{G}}_{diff}$: ${\mathcal{S}}_2\leftarrow {\mathcal{G}}_{diff}[0:N_{rem}]$. 14:     ${\mathcal{R}}_i^{\prime }\leftarrow {\mathcal{S}}_1\cup {\mathcal{S}}_2$. 15: else 16:     Branch B: Implicit Inference 17:     if Event Types are available in $\mathcal{D}$ then 18:         Group ${\mathcal{R}}_i$ by Event Type. 19:         Select samples based on group size (Top-2 from largest, Top-1 from others). 20:     else 21:         Construct fused features F via Norm-Concat-Norm strategy. 22:         Perform K-Means++ on ${\mathcal{R}}_i\cup \left\{X_i\right\}$ with F ($K=4$). 23:         Identify target cluster $C_{target}$ and nearest clusters $C_{near}$. 24:         Select Top-2 from $C_{target}$ and Top-1 from each $C_{near}$. 25:     end if 26:     ${\mathcal{R}}_i^{\prime }\leftarrow$ Selected samples. 27: end if 28: return  ${\mathcal{R}}_i^{\prime }$

With the successful acquisition of high-value retrieval samples ($R_i^{\prime }$), we have obtained all the necessary information to construct the instruction fine-tuning dataset. These retrieved samples $R_i$, containing both their content, ground-truth labels, and Geo/Location tags, serve as few-shot demonstrations to guide the LLM’s reasoning process.

3.4. Instruction Dataset Construction

We reformulate the multimodal crisis classification task as a text generation task, requiring the LLM to predict the category of a post via text generation conditioned on the provided multimodal context. To optimize this mapping and unlock the reasoning potential of the LLM, we employ instruction fine-tuning. To this end, we reconstruct the original dataset by integrating heterogeneous information modules—tweet text, visual descriptions, synthetic comments, and the retrieved geographic context—into a unified instruction format.

Specifically, for the i-th multimodal post $X_i=\{T_i,V_i\}$ with the ground-truth label $Y_i$, we aggregate the following augmented components:

• The [Detailed image caption] $D_i$, generated by the MLLM to externalize image details;
• The [Comment List] $C_{i,j}$, generated by the LLM. Each list comprises n distinct comments (denoted as $c_{i,j,1},\dots ,c_{i,j,n}$), corresponding to varying social perspectives derived from different simulated personas;
• The [Retrieved Samples] $R_i^{\prime }$, consisting of high-value reference samples obtained through the GECR module, which provides geographic semantics and reasoning anchors.

During the construction of the instruction dataset, we transform the original sample $X_i$ into a series of augmented instruction instances. Since we generate multiple distinct comment lists for a single post to maximize diversity, a single original sample $X_i$ yields j instruction-tuning instances, defined as $I_{i,j}=\{T_i,D_i,C_{i,j},R_i^{\prime }\}$.

Finally, these components are populated into a structured prompt template (as shown in Figure 2). This template functions as a structured framework guiding the LLM to synthesize the textual content ($T_i$), visual context ($D_i$), social sentiment ($C_{i,j}$), and geographic references ($R_i^{\prime }$) to infer the correct crisis category. The [Class] field in the output component of the template is replaced with the category name corresponding to $Y_i$.

By constructing the dataset in this manner, we effectively simulate a “Human-in-the-Loop” reasoning process where the model is provided with objective content (tweet text and image), subjective public opinion (comments), and comparative context (retrieved samples). This holistic data construction strategy ensures that the LLM is not merely memorizing labels but is learning to reason across multiple dimensions of the crisis event. In the next section, we will detail how we utilize this enriched dataset to fine-tune the LLM for optimal performance.

3.5. LoRA-Based LLM Instruction Fine-Tuning

To effectively adapt LLMs to the nuances of crisis analysis without the prohibitive computational cost of full fine-tuning, we employ Low-Rank Adaptation (LoRA) [45]. LoRA facilitates efficient fine-tuning by freezing the pre-trained weight matrices of the LLM and injecting a small number of trainable low-rank decomposition matrices into the transformer layers. We specifically integrate LoRA into the query and value projection layers of the attention blocks within the LLM. The forward pass in a LoRA-adapted layer is expressed as

$$h=W_{0}x+\Delta Wx=W_{0}x+MNx,$$

(3)

where x is the input, $W_0\in {\mathbb{R}}^{d\times k}$ denotes the frozen pre-trained weight matrix, and $\Delta W$ represents the trainable adapter. The adapter is decomposed into two low-rank matrices $M\in {\mathbb{R}}^{d\times r}$ and $N\in {\mathbb{R}}^{r\times k}$, where the rank $r\ll \min (d,k)$.

During the fine-tuning process, the model is optimized by minimizing the negative log-likelihood loss, which drives it to learn how to autoregressively generate the correct textual response based on the given instruction. Crucially, the input instruction $X_I$ is constructed by populating the comprehensive template that integrates distinct informational elements: the [Tweet text] $T_i$, the MLLM-generated [Detailed image caption] $D_i$, the [Comment List] $C_{i,j}$, and the [Retrieved Samples] $R_i^{\prime }$. The loss function is defined as

$$\mathcal{L}=-\sum _{s=1}^S\log p_{\theta }(y_s\mid X_I,y<s),$$

(4)

where y$={\left\{y_s\right\}}_{s=1}^S$ denotes the target output sequence (i.e., the textual representation of the ground-truth crisis category).

Through this instruction fine-tuning strategy, we transform the LLM from a general-purpose generator into a specialized reasoning agent for crisis informatics. Consequently, The LLM learns not only to classify textual and visual details but also to perform a holistic assessment: integrating retrieved geographic evidence with social comment to provide accurate and context-aware crisis post classification.

4. Results

4.1. Dataset

We conducted experiments on the publicly available multimodal crisis post classification dataset, CrisisMMD [16]. This dataset comprises seven natural disaster events from 2017, including Hurricane Irma, Hurricane Maria, Hurricane Harvey, Mexico earthquake, Iraq–Iran earthquakes, Sri Lanka floods, and California wildfires. A protocol is applied for multi-image tweets: if a tweet has two or more images, we process each text–image pair as an independent sample while maintaining identical textual content across pairs. In the experiments, we used tweets with consistent multimodal labels and evaluated the performance on both Task 1 and Task 2:

• Task 1 aims to assess whether a tweet contains valuable humanitarian aid information, requiring classification into two categories: Informative and Not Informative.
• Task 2 further categorizes tweets into five fine-grained classes: “Affected individuals” (including “Injured or dead people” and “Missing or found people”), “Rescue, volunteering, or donation effort”, “Infrastructure and utility damage” (including “Vehicle damage”), “Other relevant information”, and “Not humanitarian”.

The CrisisMMD dataset is split into training, validation, and test sets with a ratio of 75/13/12%.

In addition, we conducted experiments on the DMD dataset [17]. The DMD dataset comprises multimodal posts sourced from the Instagram platform, covering five distinct damage categories in addition to a “No damage” category: Infrastructural damage, Natural landscape (such as landslides, avalanches, and fallen trees), Fires, Floods, and Human damage (human casualties or deaths). In our experiments, only multimodal posts containing both images and text were utilized. The DMD dataset is split into training, validation, and test sets with a ratio of 80/10/10%. Notably, the DMD dataset does not contain specific event labels, providing only the sample content itself and the corresponding ground-truth labels. It is important to note that the classification task on the DMD dataset involves discriminating among these six specific categories. Therefore, it represents a fine-grained multi-class classification task, analogous to Task 2 in the CrisisMMD dataset.

4.2. Implementation Details

We utilized LLaVA-1.5-7B [26,27] as the MLLM, replacing its original visual feature extractor with FG-CLIP [28]. Mix-persona simulation and diverse comment generation are conducted using DeepSeek-3.1 [46]. We utilized Llama 3-8B [47] during the fine-tuning stage and employed AdamW [48] as the optimizer. The batch size was set to 12. To ensure statistical reliability, we conducted experiments with five random seeds and reported the average performance. The model is trained for a total of 20 epochs, incorporating an early stopping strategy with a patience value of 5. During comment generation, we restricted the maximum length of DeepSeek output to 50 tokens, while limiting LLaVA’s output to 120 tokens. The maximum number of comments n in each [Comment list] was set to 30.

4.3. Baselines

We compare the proposed MPCG-GECR with numerous prior studies. We select ResNet [49] and BERTweet [50] as unimodal baselines and choose SCBD [3], AT-CAVE [51], OWSEC [4], MFEK [7], CEFN [6], MMDG [5], OWDII [52], CG-PG [53], and CrisisSpot [54] as multimodal baselines, primarily focused on improving multimodal fusion. Additionally, we include DeepSeek-V3 [46], DeepSeek-R1 [55], LLama 3-8B [56], and LLaVA-1.5-7B [26,27] as representative LLMs and MLLMs to evaluate their performance on downstream zero-shot classification tasks. Since LLMs have not been widely adopted in crisis tweet analysis, we select MFEK as the representative model that utilizes LLMs. Detailed explanations of the selected baselines are provided below.

• ResNet [49]: This classical image processing model has shown robust performance in various computer vision tasks.
• BERTweet [50]: A large-scale pre-trained language model for English tweets, which shares the same architecture as BERT and uses the RoBERTa pre-training procedure.
• SCBD [3]: This crisis tweet classification model addresses data inconsistency and overfitting through two key mechanisms: Stochastic Shared Embeddings (SSE) regularization for robust training and Cross-Attention-based fusion of projected image and text embeddings. These mechanisms enhance model stability while maintaining multimodal integration.
• AT-CAVE [51]: An adaptive transformer-based conditioned variational autoencoder network for incomplete multimodal tweet classification. It jointly models the textual information, visual information and label information into a unified deep model, which can generate more discriminative latent features and enhance the performance in missing-modality scenarios.
• OWSEC [4]: An open-world multimodal classification model that combines fine-grained semantic interaction. The model utilizes a multimodal mask transformer architecture to establish cross-modal semantic relation, filter redundant information through dynamic masking, and generate virtual mixed samples for training a separate open-world classifier.
• MFEK [7]: A knowledge-enhanced multimodal architecture that addresses out-of-distribution challenges via image-guided textual enhancement, multi-source knowledge extraction from Wikipedia and GPT-3.5 Turbo, and a co-attention-based fusion mechanism of external knowledge with text and image features.
• CEFN [6]: An evidential fusion framework grounded in subjective logic theory that explicitly models uncertainty during multimodal integration. The network treats encoder outputs as subjective opinions, enabling direct uncertainty quantification and more reliable representation learning.
• MMDG [5]: A multilayer deep graph model that constructs text and image graphs for multimodal representation learning. The architecture combines Graph Convolutional Networks (GCN) with autoencoder components to systematically extract and integrate heterogeneous graph-structured information.
• OWDII [52]: A multimodal model for categorizing social media posts related to disasters in an open-world environment, utilizing a multitask (closed-world and open-world) classifier and a sample generation strategy that models the distribution of unknown samples using known data.
• CG-PG [53]: A complementary graph learning and prompt-based cross-modal generation network for missing-modality cases in the fake news detection field. It explores structural complementary information in image and text graphs and generates representations of the missing modality from available modalities.
• CrisisSpot [54]: A graph neural network architecture designed to model complex cross-modal relationships by jointly analyzing textual–visual content correlations and social context features (user-centric and content-centric). Its inverted dual embedded attention mechanism simultaneously captures both complementary and contradictory data patterns.
• DeepSeek-V3 [46]: DeepSeek-V3 is a powerful Mixture-of-Experts (MoE) language model with 671B total parameters, employing innovative Multi-head Latent Attention (MLA) and DeepSeekMoE architectures for efficient training and inference.
• DeepSeek-R1 [55]: DeepSeek-R1 is an advanced iteration of DeepSeek-R1-Zero that addresses readability and language consistency issues through multi-stage training with cold-start data prior to reinforcement learning. This enhanced version achieves more robust performance while maintaining powerful logical behaviors.
• LLama 3-8B [56]: LLama 3-8B is a open-source LLM with 8 billion parameters. This model employs a decoder-only transformer architecture with Grouped-Query Attention (GQA) and an efficient 128K-token tokenizer. Pre-trained on over 15 trillion tokens and instruction-tuned, it delivers strong performance, particularly in reasoning and coding, establishing itself as a leading model in its class.
• LLaVA-1.5-7B [26,27]: LLaVA-1.5 is an MLLM using CLIP-ViT-L-336px with a multilayer perceptron projection enhanced by academic-task-oriented VQA data with response formatting prompts.

4.4. Experimental Results

4.4.1. CrisisMMD Dataset Results

Table 1. Performance comparison (in percentage, %) on the CrisisMMD dataset. All values are percentages, and the best performance is highlighted in bold.

	Task 1		Task 2
	Accuracy	Macro-F1	Accuracy	Macro-F1
ResNet (2016, [49])	81.85	79.14	83.58	60.61
BERTweet (2020, [50])	85.48	81.40	86.58	66.96
SCBD (2020, [3])	89.75	88.45	91.44	68.85
AT-CAVE (2022, [51])	91.69	89.42	91.89	70.54
OWSEC (2023, [4])	92.09	90.43	92.75	73.79
MFEK (2024, [7])	92.69	90.83	92.95	73.83
CEFN (2025, [6])	91.32	89.73	92.47	72.54
MMDG (2025, [5])	92.64	90.38	92.98	73.44
OWDII (2025, [52])	92.35	90.62	92.79	73.90
CG-PG (2025, [53])	92.16	90.82	92.62	73.67
CrisisSpot (2025, [54])	93.11	91.41	93.54	74.45
DeepSeek-V3 (2024, [46])	81.25	78.05	61.20	44.45
DeepSeek-R1 (2025, [55])	83.17	79.25	63.21	46.14
LLama 3-8B (2024, [56])	65.68	54.48	54.95	39.97
LLaVA-1.5-7B (2024, [26,27])	68.94	61.08	64.36	63.10
MPCG-GECR (Ours)	95.72	94.04	95.21	76.17

presents the experimental results of our proposed MPCG-GECR framework on the CrisisMMD dataset. The results indicate that multimodal methods generally surpass unimodal approaches, demonstrating the importance of jointly processing information from both modalities in multimodal tweet analysis.

MPCG-GECR outperforms all baseline methods across all tasks and metrics. Specifically, on the accuracy metric, it surpasses the state-of-the-art method CrisisSpot by 2.61% in Task 1 and 1.67% in Task 2, while the performance gap further widens to 2.63% and 1.72% in terms of the Macro-F1 score, respectively. By generating a large number of diverse user comments, MPCG-GECR significantly enriches the original dataset with external information that was previously absent. Through instruction fine-tuning, the LLM successfully learns to capture the relationship between tweet content and user comments. A key differentiator of MPCG-GECR compared to other methods lies in its ability to supplement and leverage missing user comments for crisis tweets, which is also the crucial factor driving its performance advantage.

Furthermore, a critical observation from Table 1 is the performance disparity between our fine-tuned framework and general-purpose Large Models (e.g., DeepSeek-V3, DeepSeek-R1, LLaMA 3) or Multimodal LLMs (LLaVA-1.5) in zero-shot settings. Despite their immense parameter scale and generalization capabilities, these models exhibit suboptimal performance, significantly lagging behind supervised baselines. For instance, LLaVA-1.5 achieves only 68.94% accuracy on Task 1, which is far lower than the 95.72% achieved by MPCG-GECR. This underscores that without task-specific adaptation, general LLMs struggle to grasp the nuanced definitions of humanitarian categories and the specific boundaries of crisis informatics. Even DeepSeek-R1, which possesses advanced reasoning capabilities, fails to match the performance of specialized models in this domain. We clarify the motivation for comparing our fine-tuned framework with zero-shot General LLMs. This comparison is not meant to establish a trivial superiority of supervised training, but to investigate whether the massive parametric knowledge of modern LLMs can supersede the need for task-specific adaptation in crisis informatics.

This comparison highlights the paradigm shift represented by MPCG-GECR: moving from feature fusion (as seen in methods like CrisisSpot and MMDG) to generative reasoning. By transforming the classification task into a generative instruction-following task and grounding it with synthesized comments and retrieved geographic contexts, our method effectively aligns the vast knowledge of the LLM with the specific requirements of crisis analysis, achieving a new state of the art.

For an assessment of the incremental value of our proposed modules against a standard Supervised Fine-Tuning (SFT) baseline, we refer readers to the Ablation Study in Section 4.5. The comparison there explicitly isolates the contributions of the SPG and GECR modules over the base Llama 3-8B architecture.

4.4.2. DMD Dataset Results

Table 2. Performance comparison (in percentage, %) on the DMD dataset (Multi-class Classification, analogous to Task 2 of the CrisisMMD dataset). All values are percentages, and the best performance is highlighted in bold.

	Accuracy	Macro-F1
ResNet (2016, [49])	86.53	86.37
BERTweet (2020, [50])	85.89	85.64
SCBD (2020, [3])	92.36	92.27
AT-CAVE (2022, [51])	92.67	92.58
OWSEC (2023, [4])	93.22	93.14
MFEK (2024, [7])	93.51	93.40
CEFN (2025, [6])	93.03	92.91
MMDG (2025, [5])	93.33	93.26
OWDII (2025, [52])	92.90	92.59
CG-PG (2025, [53])	93.14	92.98
CrisisSpot (2025, [54])	93.79	93.60
DeepSeek-V3 (2024, [46])	80.19	79.95
DeepSeek-R1 (2025, [55])	82.12	81.88
LLama 3-8B (2024, [56])	75.60	72.51
LLaVA-1.5-7B (2024, [26,27])	82.62	82.14
MPCG-GECR (Ours)	94.83	94.68

presents the experimental results of our proposed MPCG-GECR framework on the DMD dataset.

On the DMD dataset, our proposed MPCG-GECR framework likewise achieved the best performance, with an accuracy of 94.83% and a Macro-F1 score of 94.68%, outperforming all baseline models. This result is consistent with the observations on the CrisisMMD dataset, robustly validating the effectiveness and robustness of our method across different data distributions and crisis types. Specifically, compared to the current state-of-the-art model, CrisisSpot, MPCG-GECR achieved a gain of 1.04% in accuracy and 1.08% in Macro-F1. Although the margin of improvement is slightly smaller than that on the CrisisMMD dataset, it demonstrates that our method can still achieve stable and statistically significant performance gains on the DMD dataset, a dataset where baseline models already exhibit strong performance, proving its capability to address the bottleneck.

4.5. Ablation Study

To validate the efficacy of various designs in MPCG-GECR, we conducted ablation studies testing the following variants:

• w/o GECR and w/o SPG: The standard fine-tuning baseline excluding both modules, utilizing only tweet text and image captions.
• w/o GECR: The variant removing the retrieval module but retaining synthetic comments.
• Random retrieval: The variant replacing GECR with random retrieval.
• w/o [Attribute]: Variants removing specific persona attributes.
• Full-list generation at once: The variant generating comments without the iterative strategy.

Table 3. Results of the ablation study.

	Task 1		Task 2
	Accuracy	Macro-F1	Accuracy	Macro-F1
MPCG-GECR	95.72	94.04	95.21	76.17
w/o GECR and w/o SPG	90.31	88.62	90.25	69.45
w/o GECR	94.21	92.66	94.03	74.78
Random retrieval	92.59	90.20	92.14	72.05
w/o Gender	94.92	93.11	94.42	75.81
w/o Age	94.60	92.72	94.35	75.23
w/o Education	94.79	92.85	94.36	75.39
w/o Occupation	94.26	92.69	94.16	74.95
w/o Location	94.25	92.72	94.04	74.75
Full-list generation at once	94.88	93.08	94.71	75.53

shows the test results of MPCG-GECR and its variants.

First, we establish a baseline using the standard supervised fine-tuning approach (w/o GECR and w/o SPG), which achieves an accuracy of 90.31% (Task 1). This indicates the base capability of Llama-3 when adapted to the crisis domain using only intrinsic parametric knowledge. Upon introducing the Synthetic Persona Generator (w/o GECR), we observe a performance gain of roughly 3.90% (reaching 94.21%), confirming that simulating diverse public discourse effectively bridges the semantic gap in isolated tweets. Finally, the complete MPCG-GECR framework achieves the highest accuracy of 95.72%.

Significant performance degradation is observed when the GECR module is removed, underscoring its pivotal role in our framework. By retrieving high-value samples aligned with geographic semantics, GECR provides the LLM with effective reference anchors, enhancing reasoning beyond purely generative augmentation.

Furthermore, replacing GECR with a random retrieval strategy results in a notable decline in performance (92.59%), which is even lower than the baseline without any retrieval (94.21%). This finding directly answers the question of whether performance gains stem from mere information volume or semantic consistency. It demonstrates that blindly introducing external context acts as noise, impairing the model’s judgment. The improvement observed in MPCG-GECR is strictly attributable to the geographic consistency of the retrieved samples, which validates their role as reliable reference anchors rather than merely from the presence of additional few-shot demonstrations.

It can be observed that removing any persona attribute leads to varying degrees of performance degradation, indicating that each attribute possesses distinctive characteristics that help emphasize differences among users. Notably, we find that the performance decline is most pronounced when the “Occupation” and “Location” attributes are excluded, highlighting the critical importance of these two dimensions. We attribute this to the specific semantic value they carry: the “Occupation” attribute introduces variations in professional expertise and domain knowledge, while the “Location” attribute encapsulates distinct cognitive perspectives derived from spatial proximity to the crisis. Consequently, these attributes exert a substantial influence on the model’s ability to accurately parse crisis information.

A noticeable performance drop occurs when the LLM is required to generate the entire comment list at once. This confirms that our iterative generation strategy effectively improves the quality of comment lists by producing more accurate user comments. We posit that this iterative approach also helps mitigate LLM hallucinations. When the model is prompted to simulate multiple conflicting personas and generate a dialogue simultaneously (i.e., the Full-list generation at once strategy), it is prone to context confusion and role-mixing. However, it is important to acknowledge a trade-off: the strategy significantly reduces generation latency, particularly when a large volume of comments is required. Therefore, in scenarios where timeliness is prioritized over accuracy, the Full-list generation at once strategy remains a viable recommendation.

4.6. Hyperparameter Sensitivity

In the MPCG-GECR framework, the primary hyperparameters include n, which controls the number of comments per list, and K, which determines the number of retrieved samples in the GECR module. We investigated the impact of these hyperparameters on Task 1 of the CrisisMMD dataset.

For hyperparameter n, we tested six different values: 5, 10, 20, 30, 40, and 50. The corresponding results are presented in Figure 4. As depicted, the model performance generally exhibits a trend of initial improvement, followed by a plateau as the number of comments per list increases. Specifically, performance peaks at $n=30$ and begins to decline at $n=40$. We posit that at $n=30$, the model is equipped with a sufficiently diverse set of social perspectives; thus, further increasing this number does not yield additional performance gains.

For hyperparameter K, we evaluated five values: 2, 3, 4, 5, and 6. The results, shown in Figure 5, indicate that the model performance first increases and then decreases, achieving its optimum at $K=4$. This suggests that providing the model with more retrieved context does not invariably aid the decision-making process. Conversely, excessive retrieved items may introduce noise, thereby impairing judgment. Similarly, an insufficiently diverse set of retrievals (i.e., with excessively high similarity to the target sample) is also suboptimal, as it hinders the LLM’s ability to assess the target content from a broad and comprehensive perspective.

4.7. Computational Efficiency

In the previous subsection, we discussed the impact of the hyperparameter n on model performance. We further evaluated the average inference time per sample in the test set under different n values, using an Nvidia 4090 GPU and a batch size of 12.

The inference latency of MPCG-GECR comprises the following components: the image captioning latency from LLaVA-1.5-7B, the comment generation latency, the retrieval latency from GECR, and the inference latency from Llama 3-8B. Among these, the image captioning latency (∼1.3 s) and GECR retrieval latency (∼0.3 s) are independent of n. The inference latency of Llama 3-8B exhibited negligible variation across different n values and was relatively short overall, with a minimum of ∼1.6 s at n = 5, ∼1.9 s at n = 30, and a maximum of ∼2.1 s at n = 50. This marginal increase is attributable to the relative insensitivity of the LLM to input length, as its inference latency is primarily influenced by the length of the generated output.

The comment generation latency for different n values is shown in Figure 6.

As illustrated in Figure 6, the comment generation latency increases rapidly with n, as the LLM requires increased time to generate the larger volume of content. Considering the critical requirement for timeliness in crisis scenarios, we recommend selecting the appropriate comment generation strategy and hyperparameter n based on the specific accuracy requirements.

4.8. Case Study and Discussion

To qualitatively evaluate the effectiveness of different models, we selected several examples from the CrisisMMD dataset, as shown in Figure 7.

In all three cases, the proposed MPCG-GECR succeeded, whereas CrisisSpot and MFEK failed sometimes. The specific analysis is as follows:

• Case 1: The primary challenge lies in accurately discerning the meaning of the image and jointly analyzing it with the text. CrisisSpot and MFEK likely failed to correctly interpret the visual content, exhibiting a bias towards classifying the image as “Not informative,” which led to their failure. In contrast, MPCG-GECR leverages the MLLM to effectively understand the image information, ultimately enabling the LLM to make the correct classification.
• Case 2: The difficulty here involves accurately inferring the implicit information in the text. The sample presents a headline from a news weekly that contains an informative image. The external knowledge introduced by MFEK backfired, causing it to erroneously over-index on the textual content while overlooking the image, leading to a misclassification. Both CrisisSpot and MPCG-GECR achieved correct multimodal fusion, resulting in successful classification.
• Case 3: The textual content in this sample strongly biased both CrisisSpot and MFEK towards interpreting it as containing other informational content. Additionally, the human present in the image introduced further distraction. Only MPCG-GECR correctly comprehended that the image lacked substantive crisis-related content, leading to the accurate classification of “Not humanitarian”.

5. Discussion

5.1. Paradigm Shift: From Feature Fusion to Generative Reasoning

MPCG-GECR outperforms state-of-the-art baselines due to a paradigm shift from traditional feature fusion to generative reasoning. While conventional “closed-system” fusion methods are limited by pre-trained parametric knowledge, our approach aligns the LLM’s vast world knowledge with domain requirements via an “augment, retrieve, and fine-tune” pipeline. The stark performance disparity between our fine-tuned model and general-purpose LLMs (e.g., DeepSeek-V3, LLaVA-1.5) in zero-shot settings offers a critical insight: model parameter scale alone is insufficient for crisis informatics. Instead, instruction fine-tuning with augmented context is essential to capture the nuanced definitions and specific boundaries of crisis informatics.

5.2. Mechanism Analysis: Anchors and Perspectives

Ablation studies validate the distinct roles of our two core modules. First, the significant drop in performance when removing the GECR module (or replacing it with random retrieval) validates the hypothesis that geographic semantics are paramount. Unlike random few-shot examples which may introduce noise, GECR retrieves samples that share spatial and environmental semantics with the target. Second, the SPG module confirms that crisis perception is heavily influenced by cognitive perspectives, specifically “Location” and “Occupation.” By synthesizing these diverse viewpoints, the model overcomes the information scarcity intrinsic to isolated posts.

5.3. Trade-Offs Between Accuracy and Latency in Deployment

A practical implication of our study lies in the trade-off between computational efficiency and classification accuracy, a crucial consideration for real-time crisis response systems. Our efficiency analysis indicates that iterative comment generation acts as the primary bottleneck. While the iterative strategy yields higher accuracy by minimizing context confusion, it incurs a latency penalty that scales linearly with n. However, the “full-list generation at once” strategy offers a viable alternative for time-critical scenarios. Although it sacrifices a degree of accuracy due to potential role-mixing, it dramatically reduces the inference time. Consequently, we propose an adaptive deployment strategy: using the rapid mode for initial screening and the iterative mode for high-priority analysis, with optimal resource points at $n=30$ and $K=4$.

5.4. Quality of Synthetic Comments

To assess the reliability of the comments generated by the SPG module, we conducted a manual review of 50 randomly selected samples. We focused on checking for Label Bias (whether the generated comments contradict the ground-truth category of the post) and Logical Consistency (whether the comments align with the assigned persona). The review indicated that 96% of the generated comments were logically consistent with the post content and persona attributes. Importantly, we did not observe significant label bias that could mislead the model’s classification. The generated comments primarily reflected diverse emotional reactions and subjective viewpoints typical of social media discourse, rather than introducing factual discrepancies. Furthermore, regarding the risk of ‘echo chambers’ or self-reinforcement loops, our Mix-Persona strategy serves as an active countermeasure. By explicitly enforcing diversity in attributes, we force the model to generate perspectives from conflicting standpoints, such as those of local residents and remote observers. This ensures the classifier is exposed to a broad spectrum of social discourse rather than a single reinforced viewpoint.

5.5. Limitations and Future Directions

A primary limitation is the GECR module’s dependence on historical data, which may underperform in regions with sparse digital footprints. Future research should focus on two directions:

• Cross-Regional Transfer Learning: Investigating how geographic knowledge learned from data-rich regions can be adaptively transferred to data-sparse regions, potentially using domain adaptation techniques.
• Dynamic Persona Adaptation: Moving beyond static attribute lists to dynamically generate persona profiles based on the real-time demographic and cultural characteristics of the impacted region, thereby generating more culturally context-aware comments.

6. Conclusions

In this paper, we address the dual challenges of user comment scarcity and contextual isolation in multimodal crisis post analysis. To overcome these limitations, we propose the Mix-Persona Comment Generation with Geographically Enhanced Context Retrieval for LLM Instruction Fine-tuning (MPCG-GECR) framework, which combines subjective social simulation with objective geographic retrieval.

Our key contribution lies in establishing a novel methodology that transforms crisis classification from a feature discrimination task into a context-aware generative reasoning task. By leveraging the Synthetic Persona Generator (SPG) to simulate diverse public discourse and the Geographically Enhanced Context Retrieval (GECR) module to provide grounded environmental references, we successfully bridge the gap between data-driven deep learning and knowledge-driven reasoning. Extensive experiments on the CrisisMMD and DMD datasets confirm that our approach not only achieves state-of-the-art performance but also offers robust predictions. We believe this framework sets a new baseline for leveraging Large Language Models in multimodal crisis post classification.