A Domain-Specific Memory Bank-Enhanced Framework for Remote Sensing Image Captioning

Highlights

What are the main findings?

• On the test sets, the full-bank variant (R1) and clustered variant (R4) yield CIDEr gains of 1.87% on RSICD and 11.7% on Sydney, respectively, relative to B.
• Using 39.1% of the full-bank entries, R4 retains 99.3% of the mean validation CIDEr of R1.

What are the implications of the main finding?

• Memory-bank replacement with lightweight adaptation enables domain transfer while keeping the backbone frozen.
• The benefits are dataset dependent, and the reported confidence intervals for the two test set gains include zero.

Abstract

Remote sensing image captioning enables automated interpretation of remote sensing data for applications such as environmental monitoring and urban planning. However, general-purpose multimodal models exhibit suboptimal adaptability on remote sensing imagery due to domain discrepancies in data distribution, resolution, and rotation invariance. We propose a domain-specific memory bank-enhanced framework for remote sensing image caption, built upon the lightweight EVCap architecture with three core innovations: (1) a remote sensing-specific visual memory bank providing domain-representative visual–semantic alignment; (2) a staged transfer training strategy that leverages pretrained knowledge while mitigating overfitting from limited remote sensing data; and (3) a K-means clustering-based small-sample memory bank that preserves performance while reducing memory footprint for resource-constrained deployment. Extensive experiments on the NWPU, RSICD, and Sydney datasets demonstrate substantial improvements over baselines. On the test sets, the domain-specific memory bank (R1) improves CIDEr by 1.87% on RSICD (0.3255 to 0.3316), and the clustering-based variant (R4) by 11.7% on Sydney (0.1422 to 0.1589) using only 39% of the full bank, despite decreases on NWPU and RSICD, with about 3.97 million trainable parameters.

1. Introduction

Remote sensing image captioning, as one of the important tasks in remote sensing intelligent interpretation, leverages deep learning and large language models to enhance the automated processing and analysis of remote sensing imagery. This task exhibits distinctive characteristics including automation, efficiency, interpretability, diversity, and semantic understanding. Generating natural language descriptions for remote sensing images has become a key approach to bridging the gap between visual data and human-readable information [1,2,3]. It finds wide applications across multiple specialized domains, including scene understanding and classification, geospatial intelligence analysis and generation, urban planning, and illegal construction detection.

With the rapid advancement of multimodal large language models (MLLMs) such as ChatGPT, GPT-4V, and Qwen-VL-Max, significant progress has been achieved in general-purpose image captioning. Domain-specific models for remote sensing, including RSGPT [4], EarthGPT [5], RS-ChatGPT [6], SkyEyeGPT [7], and RemoteCLIP [8], have also been developed.

However, the behavior of MLLMs on remote sensing image captioning is more nuanced than a uniform performance drop. On the one hand, general-purpose MLLMs are pretrained predominantly on natural images; the unique characteristics of remote sensing imagery—multi-scale objects, complex backgrounds, and domain-specific semantic concepts—induce a substantial domain gap upon transfer, causing inaccurate visual–semantic alignment and weak fine-grained scene recognition [3,9,10,11,12,13,14]. On the other hand, remote sensing domain models fine-tuned on in-domain corpora can be highly competitive: RSGPT attains a CIDEr of 1.0294 on RSICD after fine-tuning the Q-Former and projection layers of its 7B backbone on RSICap [4], while EarthGPT and GeoChat show similar capabilities through large-scale in-domain instruction tuning [5,15].

Early remote sensing image captioning methods were predominantly based on encoder–decoder frameworks comprising convolutional neural networks (CNNs) combined with recurrent neural networks (RNNs) or long short-term memory (LSTM) networks [11,12,13,14], along with improvements to these frameworks [2,16,17], and the introduction and optimization of Transformer-based approaches [18,19,20,21]. While these model designs for remote sensing image caption achieved certain levels of success, they suffer from limitations including task singularity, poor generalization capability, and heavy dependence on annotated datasets specific to particular scenarios. In recent years, multimodal large models have demonstrated superior image processing and semantic understanding capabilities, achieving outstanding results through training on large-scale remote sensing datasets.

Nevertheless, representative remote sensing domain models such as EarthGPT [5], GeoChat [15], and RSGPT [4] similarly depend on massive computational resources and large-scale annotated datasets, which constrains their deployment in resource-constrained scenarios. The lightweight memory-augmented multimodal model EVCap [22] performs well on general image caption tasks but exhibits insufficient domain adaptability for remote sensing imagery.

The limitations of existing research can be summarized into three aspects.

• Domain Mismatch in Visual Memory Banks: General-purpose visual memory banks lack representativeness for the remote sensing domain, leading to inaccurate visual–semantic alignment;
• Inefficient Training Strategies: Training models from scratch on remote sensing data often results in overfitting due to insufficient data volume;
• High Computational Costs: State-of-the-art multimodal large models require massive computational resources, limiting their practical deployment.

We propose a domain-specific memory-bank augmentation method for remote sensing image caption that addresses these limitations and achieves model training and adaptation through three integrated components. First, we construct a large-scale remote sensing-specific memory bank to address the issue of retrieval bias caused by the mismatch between the abundant natural objects in the LVIS of general-purpose models and the distribution of remote sensing scenes and objects. We construct a structurally compatible and feature-aligned remote sensing memory bank by integrating four datasets (NWPU [23], RSICD [13], RSICap [4], and UCM [24]) and extracting features with the frozen ViT and Q-Former of EVCap. Second, we employ a staged transfer training strategy, leveraging pretrained knowledge to mitigate training complexity and mitigate overfitting risks. Third, we identify a critical limitation of naive memory-bank expansion: while increasing knowledge volume, it inevitably introduces substantial redundant and noisy samples, which impairs retrieval efficiency and potentially interferes with generation quality. To address this issue, we propose a K-means clustering-based memory-bank optimization strategy that selects 15,000 high-quality, most representative samples as “core memories” to maintain performance while reducing computational costs. Experiments on three benchmark datasets of NWPU[23], RSICD [13], and Sydney [25], demonstrate the effectiveness of the proposed method, providing a novel low-cost, high-performance solution for remote sensing image caption.

The main contributions of this work are as follows:

• Remote Sensing-Specific Memory-Bank Construction. We propose a remote sensing domain-specific memory-bank construction method to address the domain mismatch problem of visual memory banks, achieving improved model performance on remote sensing datasets. We demonstrate through comparative experiments that replacing the general LVIS memory bank with a remote sensing domain-specific memory bank maintains training set performance while significantly enhancing cross-domain generalization.
• Staged Transfer Training Strategy. We introduce a staged transfer training strategy to address the low efficiency of training strategies, achieving effective utilization of pretrained knowledge while reducing training difficulty and overfitting risk. We show through experimental analysis that freezing the Q-former and only updating the external memory bank is a more robust adaptation strategy on remote sensing datasets, providing a valuable empirical reference for fine-tuning large models in remote sensing.
• Semantic Clustering-based Representative Memory Selection Mechanism. We propose a K-means clustering-based representative high-quality small-sample memory bank selection mechanism to address the high computational cost problem, achieving improved model generalization while reducing the number of stored samples by 61%. We demonstrate through experimental results that high-quality selected memory banks outperform coarse large-scale memory banks, and data quality has a decisive impact on model performance.

2. Methods

The domain-specific memory bank-enhanced remote sensing image captioning framework comprises two technical pipelines: data preparation and model training, with the core logic centering on remote sensing external memory-bank construction and model fine-tuning. In this section, we first introduce the EVCap baseline as preliminaries, then detail three main components: the proposed framework overview, remote sensing-specific memory-bank construction, and staged transfer training strategy. An overview of the proposed methodology is illustrated in Figure 1.

Figure 1. Overview of the proposed methodology. This figure consists of three components: the construction pipeline of the remote sensing-specific memory bank, the model architecture, and the staged transfer learning technique.

Preliminaries on EVCap. EVCap [22] augments a frozen LLM with an external visual–name memory from LVIS v1: across the 1203 categories, one to ten real and five synthetic images yield $M=14,596$ image–name pairs, encoded by the frozen BLIP-2 encoder, the averaged embedding as key and the name as value. At inference, the identically encoded query retrieves the most similar keys; the retrieved names and features pass through an attentive fusion module and a trainable linear layer into the frozen Vicuna-13B decoder with 3.97M trainable parameters in total. Trained solely on COCO, its memory is replaceable without backbone retraining. Our implementation keeps this architecture and retrieval pipeline unchanged, modifying only the memory content, metadata, and data pipeline.

2.1. Model Architecture

Our work is built upon the EVCap framework [22], with the core module comprising three major components: a visual encoder (Vision Transformer, ViT), a Q-Former alignment module, and a language decoder (large language model, LLM). This architecture implements an end-to-end processing pipeline from visual feature extraction to natural language generation. Specifically, the visual encoder is responsible for converting input images into structured visual feature representations; the Q-Former module aligns visual features with the language feature space; and the language decoder generates semantically coherent image captions based on the aligned multimodal features.

• Visual Encoder: This study employs the frozen EVA ViT-g of BLIP-2 as the visual encoder, with all parameters frozen during training to preserve its large-scale general visual feature extraction capability. The visual encoder maps input images into 1408-dimensional feature vectors, providing foundational visual representations for subsequent multimodal alignment.
• Q-Former: As a lightweight Transformer architecture, the Q-Former bridges the feature gap between visual and language modalities. In baseline experiments, the Q-Former is fully frozen to mitigate overfitting on the limited remote sensing datasets. To further examine its impact on model performance, an additional unfreezing group is included in the ablation study, where the Q-Former is unfrozen to systematically evaluate how fine-tuning affects vision–language feature alignment.
• Language Decoder: We use Vicuna-13B-v1.5-16K with AWQ 4-bit quantization as the language decoder, which generates captions from the aligned multimodal representations.

Differences from EVCap. Relative to the EVCap baseline, our work introduces three core modifications.

1.

External Memory Bank: Replaces the general LVIS memory bank with a domain-specific remote sensing memory bank (Ext-Memory-RS) built from training sets of four remote sensing datasets.

2.

Retrieval Mechanism: Adapts feature retrieval to a full remote sensing memory bank of 38,385 samples, a randomly sampled bank of 14,596 samples, and a K-means-selected bank of 15,000 samples while keeping the EVCap retrieval protocol unchanged and isolating the memory content as the only variable instead of the single-stage training used in the original EVCap.

2.2. Remote Sensing-Specific Memory-Bank Construction

To mitigate the limited adaptability of general-purpose vision–language models when applied to remote sensing data, we propose a remote sensing-specific memory bank (Ext-Memory-RS) to facilitate adaptive vision–language learning in remote sensing scenarios, enabling lightweight, scalable, and efficient object knowledge storage. The construction of the remote sensing-specific memory bank adheres to three core principles: feature normalization, structural compatibility and quality verifiability. By implementing a four-stage pipeline encompassing multi-source data integration, feature alignment extraction, structured storage, and quality verification, we establish a remote sensing memory bank of 38,385 uniformly distributed, semantically distinct samples. This memory bank directly replaces the LVIS general-purpose memory bank and provides remote sensing domain knowledge for image caption tasks.

The composition of the memory bank is as follows: NWPU training set: 25,200 samples (not the full 31,500), RSICD training set: 8500 samples (8734 images in the official training split, retained after removing near-duplicates within the set), UCM full dataset: 2100 samples and RSICap full dataset: 2585 samples.

Data Usage Protocol and Evaluation Independence.The 38,385 memory-bank pairs are drawn only from non-evaluation sources: the NWPU-RESISC45 and RSICD training partitions, the full UCM, and RSICap; their validation/test sets and the entire Sydney benchmark contribute nothing to the bank, and no parameters are updated on memory-bank data. UCM and RSICap serve exclusively as auxiliary retrieval knowledge, and RSICap is annotated on DOTA-v1.5 crops, disjoint from the evaluation benchmarks. Perceptual-hash deduplication between the bank and all evaluation sets removes 234 near-duplicate pairs, ensuring the reported gains stem from domain knowledge rather than evaluation leakage. Evaluation on RSICD therefore involves retrieval-time access to RSICD training images and captions stored in the memory bank; accordingly, we do not claim zero-shot or unseen-dataset generalization, and a leave-one-dataset-out evaluation is left for future work.

In addition, based on the Ext-Memory-RS, two types of small-sample memory banks are further constructed to achieve a balance among memory-bank scale, inference speed, and representation quality. The first type is the RS small-sample memory bank (Ext-Memory-RS-Random), constructed through random sampling without replacement from Ext-Memory-RS, which directly reduces the number of feature–text pairs while preserving the original data distribution characteristics, thereby lowering storage and computational costs. The second type adopts the K-means clustering method to construct a representative small-sample memory bank (Ext-Memory-RS-Cluster).Through the hierarchical construction strategy comprising Ext-Memory-RS, the randomly downsampled small-sample memory bank, and the clustering-based representative small-sample memory bank, a foundation is established for subsequent comparisons of the effects of different memory-bank scales and sample quality on model performance.The technical pipeline for remote sensing-specific memory-bank construction is illustrated in Figure 2.

Figure 2. Technical pipeline for remote sensing-specific memory-bank construction. Three memory banks are illustrated: Ext-Memory-RS, the full-scale remote sensing memory bank extracted from NWPU, RSICD, RSICap, and UCM datasets; Ext-Memory-RS-random, the small-sample memory bank obtained by random downsampling; and Ext-Memory-RS-cluster, the high-quality small-sample memory bank constructed via K-means clustering.

2.2.1. Remote Sensing Dataset Integration

This study integrates four remote sensing datasets, NWPU [23], RSICap [4], RSICD [13], and UCM [24], to construct a foundational data source with scale and diversity. The core characteristics and contributions of each dataset are summarized in Table 1.

Table 1. Comparison of core features of multi-source remote sensing datasets.

Datasets	Size	Used	Cat.	Main Contributions
NWPU [23]	31,500	25,200	45	Scene-level coverage with large-scale samples
RSICap [4]	2585	2585	24	Fine-grained textual descriptions
RSICD [13]	10,921	8500	30	Object-level category subdivision
UCM [24]	2100	2100	21	Typical land-cover scenes
Total	47,106	38,385	–	–

• Data Integration and Preprocessing. Multi-source remote sensing data were integrated following strict normalization protocols, which included resolution unification, format conversion, and label standardization.
• Feature Extraction and Compatibility.We adopt the ViT and Q-Former architecture from EVCap for feature extraction. By preserving the original textual descriptions and classification labels, we ensure full compatibility with the structure of the LVIS.
• Retrieval Mechanism. We propose a land-cover category-guided retrieval strategy to improve retrieval precision. Category labels play three distinct roles. Memory-bank construction: for NWPU, RSICD, and UCM, the scene/land-cover annotations shipped with the datasets (45, 30, and 21 categories, respectively) are attached to every memory entry as ground-truth metadata and are used to organize the memory bank and to analyze its category coverage (Section 2.2.6); for RSICap, which does not ship scene-level annotations, a single placeholder label is assigned to all entries so that the coverage statistics remain well defined. Training: category labels are used only for memory-bank organization and quality analysis; no label-supervised loss is introduced into model training. Inference: retrieval follows the unchanged EVCap protocol and selects relevant memory entries by global feature similarity; at this stage the benefit of category guidance is embodied in the organized and quality-analyzed memory content that retrieval draws upon.

A retrieved context is formed at the query-token level: each of the 32 learnable Q-Former queries retrieves its top-5 most similar entries by cosine similarity (inner product over L2-normalized features), yielding 160 candidates per image; similarity-based filtering retains 32 candidates, and order-preserving deduplication keeps the top-9 entries, which are concatenated with “[SEP]” tokens as the retrieval context. This protocol is identical to EVCap.

2.2.2. Feature Extraction and Structured Storage

By adopting the pretrained EVCap backbone, we implement an “end-to-end alignment + structured storage” framework for memory-bank construction. The process is detailed as follows.

• Visual Feature Extraction: We utilize the frozen EVA-CLIP ViT-g/14 backbone of BLIP-2 [22] to extract 1408-dimensional visual features from remote sensing images, preserving the capability for general visual feature extraction.
• Cross-Modal Alignment: The frozen Q-Former compresses the 1408-dimensional visual features into 32 learned embeddings of size 1 × 768, mapping them into the linguistic feature space and realizing deep alignment between visual and language modalities.
• Structured Storage: The aligned features, textual descriptions, and classification labels are serialized using the Pickle format into the file ext-memory-rs-final.pkl. This format is fully compatible with the LVIS structure, allowing for direct model loading and inference.

We establish a specialized remote sensing memory bank (Ext-Memory-RS) consisting of 38,385 samples after cleaning and deduplicating the raw remote sensing datasets. Each sample contains 768-dimensional aligned features, natural language descriptions, and semantic labels, thereby providing the model with rich remote sensing domain knowledge.

2.2.3. Memory Bank Quality Analysis

We conducted a comparative verification of Ext-Memory-RS and LVIS across three dimensions: data scale, category breadth, and semantic representativeness.

• Data Dimensions.Compared to the LVIS generic memory bank (14,596 samples), Ext-Memory-RS (38,385 samples) expanded data volume, feature matrix, and file size by approximately 2.5 to 2.7 times.More importantly, it achieved a substantial increase in remote sensing sample density.
• Domain Specificity Analysis: The comparative analysis of keyword clouds highlighting key categories and descriptive content between Ext-Memory-RS and the LVIS is presented in Figure 3. The core keywords in the remote sensing memory bank were highly concentrated on typical remote sensing land-cover objects. These ranged from natural landscapes (e.g., forest, mountain, river) to man-made facilities (e.g., airport, building, stadium). In contrast, the keywords in the LVIS primarily consisted of common daily objects.
• Semantic Representativeness: Ext-Memory-RS achieved balanced coverage across both common and rare categories. The samples were derived from well-balanced open-source datasets, effectively mitigating the issues of overfitting on frequent categories and underfitting on rare categories, thus ensuring consistent model performance across all categories.

Figure 3. Comparison of keyword clouds between the RS memory bank and the LVIS. The color intensity corresponds to word frequency, with darker colors denoting higher frequencies.

2.2.4. Small-Scale Remote Sensing-Specific Memory Bank Based on Random Sampling Without Replacement

To enable fair comparison with the baseline LVIS and control for sample quantity as a confounding variable, we propose a fixed-size random sampling without replacement strategy based on the full-scale remote sensing memory bank, constructing a small-scale remote sensing memory bank (Ext-Memory-RS-Random) with sample size strictly matched to the LVIS (14,596 Samples). This approach preserves the original data distribution while ensuring consistent memory-bank scale across experiments, adhering to controlled experimental conventions in vision–language domain adaptation research. One subtle point to keep in mind is that the clustering-based bank contains 15,000 samples while the random-sampling bank holds 14,596, a difference of 404 samples or roughly 2.8 percent; as such, the comparison between the two is best understood as approximate in capacity rather than strictly size-matched. The method is implemented in PyTorch (version 1.12.1) together with Python (version 3.9). standard libraries, encompassing four key stages: reproducibility constraints, data validation and loading, random index sampling, and memory-bank reconstruction with serialized storage.

• Reproducibility Constraints: A global random seed (seed = 2024) is fixed, and both the random and torch random number generators are initialized synchronously, ensuring that the sampling results are fully reproducible and conform to the norms of scientific experimentation.
• Data Validation and Loading: The standardized full-scale remote sensing memory bank (List structure: [features, metadata]) is loaded, with file integrity and data format validated to ensure structural compatibility with LVIS.
• Random Indexing with Modeled Sampling: From the $N=\mathrm{38,385}$ full-scale samples, $K=\mathrm{14,596}$ indices are drawn without replacement and sorted to enable efficient slicing of feature tensors. Let the full-scale remote sensing memory bank be defined as a standard dual structure as follows.

  $$M_{\mathrm{full}}=⟨F,C⟩,$$

  (1)

  where $F\in {\mathbb{R}}^{N\times D}$ is the remote sensing image visual feature tensor, N = 38,385 is the total number of full-scale samples, and $D=768$ is the fixed feature dimension. $C=\{{\mathbf{c}}_1,{\mathbf{c}}_2,\dots ,{\mathbf{c}}_N\}$ is the text metadata collection corresponding one-to-one with the features. K = 14,596 is the target number of sampled instances. Core Sampling Formula: Uniform random sampling without replacement is employed to draw a fixed number of samples from the full-scale sample set. The sampling index set is defined as

  $$L=\mathrm{RandomSample}\left(\right\{1,2,\dots ,N\},K),$$

  (2)

  where $\mathrm{RandomSample}(·)$ denotes random sampling without replacement, ensuring that all samples have equal probability of being selected with no redundant sampling. Based on the generated indices, the small-scale memory bank is constructed while preserving the original data structure:

  $$M_{\mathrm{sub}}=⟨F_L,C_L⟩,$$

  (3)

  where $F_L$ denotes the feature tensor sliced by the index set, and $C_L$ denotes the corresponding metadata subset.
• Small-Scale Memory-Bank Reconstruction and Serialized Storage: The 768-dimensional feature tensors are filtered by the index set, and the corresponding text metadata is synchronously matched, strictly preserving the storage format of the original memory bank. The downsampled memory bank is persisted using pickle for seamless model loading.

2.2.5. K-Means Clustering-Based High-Quality Memory Selection

We propose a K-means clustering-based strategy for representative memory sample selection to address the issues of information redundancy and representation degradation inherent in random downsampling. This approach constructs a compact remote sensing-specific memory bank (Ext-Memory-RS-Cluster) characterized by low redundancy, high discriminability, and strong domain specificity.Specifically, we perform unsupervised clustering on the L2-normalized 768-dimensional Q-Former features of all 38,385 remote sensing samples, minimizing the within-cluster sum of squared errors over k-means++ initialization, a single run (n_init = 1), at most 300 iterations, and a convergence tolerance of ${10}^{-4}$, with a fixed random seed of 2024; because Euclidean distances are computed on L2-normalized features, they correspond to cosine dissimilarity. The clustering partitions the samples into 15,000 clusters, and the sample closest to each cluster centroid is selected as the representative instance, so all 15,000 entries originate from the full bank. Compared with the random downsampling, this strategy maximally preserves the global feature distribution and semantic information of remote sensing scenes while maintaining strict memory-bank scale control. Consequently, it enhances domain knowledge representation, providing external support for vision–language model adaptation.

This method retains the reproducibility constraints, data validation and serialized storage protocols from the random sampling pipeline, replacing only the core selection mechanism with the K-means algorithm [26] formulated as follows:

• K-Means Clustering Objective Function: Using the squared Euclidean distance as the feature similarity metric, the optimization objective of K-means is to minimize the within-cluster sum of squared errors (SSE) between samples and their corresponding cluster centers.

  $$\min {\displaystyle \sum _{k=1}^K}\sum _{{\mathit{f}}_i\in S_k}{\parallel {\mathit{f}}_i-{\mathit{\mu }}_k\parallel }_2^2,$$

  (4)

  where $S_k$ denotes the sample set of the k-th cluster, ${\parallel ·\parallel }_2$ is the $L_2$ Euclidean distance, and ${\mathit{f}}_i$ is the 768-dimensional feature vector of the i-th sample.
• Representative Sample Selection Criterion: For each cluster k, the Euclidean distance from all samples within the cluster to the cluster center ${\mathit{\mu }}_k$ is computed, and the sample with the minimum distance is selected as the representative sample of that cluster.

  $$i_k^{*}=\arg \underset{i\in S_k}{\min }{\parallel {\mathit{f}}_i-{\mathit{\mu }}_k\parallel }_2.$$

  (5)
  Finally, the representative samples from all clusters are aggregated to construct the high-quality small-scale memory bank.

  $$M_{\mathrm{cluster}}=⟨F_{L^{*}},C_{L^{*}}⟩,$$

  (6)

  where $L^{*}=\{{\mathbf{i}}_1^{*},{\mathbf{i}}_2^{*},\dots ,{\mathbf{i}}_K^{*}\}$ is the filtered global index set, and $F_{L^{*}}$ and $C_{L^{*}}$ denote the filtered feature tensor and metadata collection, respectively.

2.2.6. Statistical Characterization of the Memory Banks

We characterize the random bank (R3) and the clustered bank (R4) along three representative dimensions (Appendix A, Table A1). (i) Category coverage: both banks cover all 88 categories (entropy 6.20 vs. 6.12 bits), but K-means reorganizes the per-category composition (frequency correlation with the full bank: 0.691 vs. 0.996). (ii) Feature-space utilization: the clustered bank is less redundant (participation ratio 2.53 vs. 2.06) and better substitutes the non-selected samples at retrieval time (NN coverage 0.99916 vs. 0.99875). (iii) Distributional fidelity: random downsampling better preserves the full-bank distribution (Fréchet $7\times {10}^{-6}$ vs. $5.7\times {10}^{-3}$). Overall, K-means trades a bounded distribution shift for retrieval substitutability and feature-space utilization.

2.3. Staged Transfer Training Strategy

To achieve efficient transfer of general-purpose pretrained knowledge to the remote sensing domain and mitigate overfitting, we propose a two-stage progressive transfer training framework that combines parameter freezing, lightweight adapter fine-tuning, and GPU memory optimization strategies, achieving substantial improvements in training efficiency while preserving model performance. The core of this framework employs a progressive learning paradigm of “general knowledge distillation–domain adaptation fine-tuning,” enabling the model to retain general vision–language alignment capability while adaptively adjusting to domain-specific feature distributions in remote sensing scenarios.

2.3.1. Stage One: General Data Pretraining and Knowledge Distillation

The first stage adopts the original EVCap framework [22], which employs the LVIS large-scale visual memory bank and the COCO image caption dataset [27] for training in distillation of general knowledge, with the objective of allowing the model to learn the general paradigm of cross-modal alignment. During training, all parameters of the ViT encoder and the Q-Former alignment module are strictly frozen, and only the vision–language alignment adaptation layer between the memory bank and the language decoder is trained, preventing the degradation of underlying general feature extraction capability. The training strategy is as follows.

• The total number of training epochs is set to 30, and a cosine annealing learning rate scheduler is employed to dynamically adjust the learning rate as defined in (7):

  $${\eta }_t={\eta }_{\max }·{\displaystyle \frac{1+\cos \left(\frac{t·\pi }{T}\right)}{2}},$$

  (7)

  where ${\eta }_t$ is the learning rate at epoch t, ${\eta }_{\max }=1\times {10}^{-4}$ is the initial maximum learning rate, and $T=30$ is the total number of training epochs.
• During training, the validation loss and BLEU-4 metric are monitored in real time. Early stopping is triggered when the validation performance shows no improvement for 5 consecutive epochs to prevent overfitting. The cross-entropy loss is computed as follows:

  $$L=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{k=1}^K}{y}_{ik}\log {\widehat{y}}_{ik},$$

  (8)

  where N is the batch size, K is the vocabulary size, $y_{ik}$ is the ground-truth label, and ${\widehat{y}}_{ik}$ is the model-predicted probability.

After 30 epochs of training, the BLEU-4 metric of the model on the COCO validation set reaches 0.387, representing an 8.2% improvement over the baseline model, which indicates the effectiveness of general knowledge distillation.

2.3.2. Stage Two: Remote Sensing Data Adaptive Fine-Tuning and GPU Memory Optimization

Based on the pretrained weights from Stage One, the second stage employs the RS-memory remote sensing-specific memory bank and the NWPU [23] for domain adaptation fine-tuning. The training epochs are set to 20, with a cumulative total of 50 epochs across both stages, focusing on optimizing the model’s semantic understanding capability for remote sensing terrain features. To address the deployment challenge of the 13B-parameter model in a single-GPU environment, this study adopts a GPU memory saving scheme combining gradient optimization and model quantization.

• Gradient Optimization: By enabling gradient checkpointing, a 50% reduction in GPU memory occupancy is achieved at the cost of a 20% increase in training time; by limiting the text generation length to 128 tokens, the GPU memory consumption of the attention mechanism is reduced; meanwhile, a dynamic gradient zeroing strategy is adopted to avoid unnecessary GPU memory usage.
• Model Quantization Strategy: The AWQ (Activation-aware Weight Quantization) method is used to perform 4-bit quantization on the Vicuna-13B-v1.5-16K model [28]. Combined with the configurations of low_resource = True and device_8bit = 0, the GPU memory occupancy of the model during Stage-Two training is reduced from the conventional 26 GB+ to the 10 GB level, as expressed in (9):

  $${\mathrm{Memory}}_{\mathrm{quantized}}={\displaystyle \frac{\mathrm{Parameters}\times \mathrm{BitWidth}}{8\times {1024}^3}}+\mathrm{Overhead},$$

  (9)

  where Parameters denotes the total number of model parameters, BitWidth is the quantization bit width, and Overhead accounts for the additional storage overhead introduced by quantization metadata.

3. Experiments and Analysis

3.1. Experimental Setup

3.1.1. Dataset Selection and Preprocessing

The NWPU dataset is adopted as the training dataset in this experiment, comprising a total of 31,500 images covering 45 scene categories, of which 25,200 images are used for training [23]. The validation datasets include the NWPU validation set, the RSICD validation set [13], and the Sydney validation set [25], consisting of 3150, 1094, and 58 images, respectively. The test datasets include the NWPU test set, the RSICD test set, and the Sydney test set, consisting of 3150, 1093, and 58 images, respectively.

Evaluation Metrics: This experiment employs standard evaluation metrics for the image caption task: BLEU-1/4: n-gram Bilingual Evaluation Understudy Score (n = 1 to 4) [29]; METEOR: Metric for Evaluation of Translation with Explicit ORdering [30]; ROUGE-L: Recall-Oriented Understudy for Gisting Evaluation based on Longest Common Subsequence [31]; CIDEr: Consensus-based Image Description Evaluation [32].

3.1.2. Experimental Group Design

To quantify the contribution of each module to remote sensing image caption performance, this study designs seven progressive comparative experiments using the controlled variable method. By maintaining a fixed baseline configuration and introducing single-variable changes, the independent effects and synergistic interactions of different factors are precisely evaluated. All experimental groups adopt a unified optimizer configuration (AdamW, ${\beta }_1=0.9$, ${\beta }_2=0.999$, weight decay = 0.05), cosine annealing learning rate scheduling strategy (initial learning rate $1\times {10}^{-4}$), and evaluation metric system (BLEU-1/4, METEOR, ROUGE-L, CIDEr). During inference, captions are generated deterministically by beam search with a beam width of 5 and a maximum generation length of 32 tokens, using length-normalized sequence scores and terminating on newline or period tokens; a global seed of 42 is fixed, and these decoding settings are identical across all compared configurations. The specific configurations of each experimental group are presented in Table 2.

Table 2. Experimental group configurations.

Group	Name	ViT	Q-Former	Memory Bank	Training Strategy	Research Objective
B	Baseline	Frozen	Frozen	LVIS	Fine-tuning for 20 epochs based on Stage One pretrained weights	Providing performance baseline
G1	General Memory Bank + Q-Former Fine-tuning	Frozen	Fine-tuned	LVIS	Training from scratch for 50 epochs	Serving as a from-scratch training reference with a general-purpose memory bank
R1	Full-Scale RS Memory Bank + Frozen Q-Former	Frozen	Frozen	Ext-Memory-RS	Fine-tuning for 20 epochs based on Stage One pretrained weights	Evaluating the domain adaptation effect of the RS-specific memory bank
R2	Full-Scale RS Memory Bank + Q-Former Fine-tuning	Frozen	Fine-tuned	Ext-Memory-RS	Fine-tuning for 20 epochs based on Stage One pretrained weights	Evaluating the synergistic optimization of the memory bank and Q-Former
R3	Random Small-Sample RS Memory Bank	Frozen	Frozen	Ext-Memory-RS-random	Fine-tuning for 20 epochs based on Stage One pretrained weights	Evaluating the performance limit under small-sample conditions
R4	Clustering-Based High-Quality Small-Sample Memory Bank	Frozen	Frozen	Ext-Memory-RS-cluster	Fine-tuning for 20 epochs based on Stage One pretrained weights	Evaluating the decisive effect of memory-bank quality on model performance
R5	Random Small-Sample RS Memory Bank + Q-Former Fine-tuning	Frozen	Fine-tuned	Ext-Memory-RS-random	Fine-tuning for 20 epochs based on Stage One pretrained weights	Evaluating the effect of Q-Former fine-tuning under varying memory-bank quality

Note on trainable components. In all groups, the ViT-g visual encoder and the Vicuna-13B LLM decoder (AWQ 4-bit) remain frozen. B, R1, R3, and R4 share the same recipe—Stage-One weights, 20 epochs of NWPU fine-tuning, and only the memory-to-LLM adaptation layer updated—differing solely in the memory bank (LVIS, Ext-Memory-RS, RS-random, and RS-cluster, respectively). R2 and R5 additionally unfreeze the Q-Former. G1 randomly initializes the trainable modules and trains for 50 epochs on NWPU with the LVIS bank, additionally unfreezing the full Q-Former.

Statistical Robustness. Given the 58-image Sydney and 1093-image RSICD test sets, we quantify two sources of randomness. (i) Test set sampling: for B vs. R1 on RSICD and B vs. R4 on Sydney, paired bootstrap over the per-image CIDEr scores with 10,000 iterations yields 95% CIs of [−0.0069, +0.0199] and [−0.0133, +0.0463]; point estimates are positive yet include zero, consistent with the modest gains. (ii) Memory-bank sampling: three sampling seeds alter the bank membership while only 9.5% of entries are shared, and leave its distributional properties essentially unchanged as documented in Section 2.2.6 and Appendix A; these properties drive the downstream differences, so the comparisons do not hinge on a particular draw. Multi-seed retraining remains a limitation.

Model Selection Protocol. All early-stopping and checkpoint-selection decisions are based exclusively on validation performance: for each evaluation dataset, the checkpoint is selected by the validation CIDEr of NWPU-val, RSICD-val, or Sydney-val, respectively. Test sets are never used for training, early stopping, or hyperparameter selection. Because this per-target selection applies equally to all compared groups, it does not alter the relative comparisons among them.

3.2. Results and Analysis

To comprehensively evaluate the effectiveness of each strategy, comparative experiments were conducted on three standard test sets: NWPU [23], RSICD [13], and Sydney [25]. As shown in Table 3, B achieved the optimal CIDEr score of 0.9810 on the NWPU test set, which shares the same data source as the training set, demonstrating the strong baseline generation capability of the original model. However, the performance of group B exhibits noticeable degradation on the cross-domain RSICD and Sydney datasets.

Table 3. Comparison of experimental results on the test sets.

Grp	Data	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	CIDEr
B	NWPU	0.7664	0.5827	0.4534	0.3607	0.6085	0.3181	0.9810
	RSICD	0.5101	0.2504	0.1434	0.0882	0.3184	0.2080	0.3255
	Sydney	0.4301	0.2086	0.1218	0.0726	0.2600	0.1965	0.1422
R1	NWPU	0.7630	0.5724	0.4432	0.3516	0.6018	0.3132	0.9667
	RSICD	0.5153	0.2586	0.1506	0.0947	0.3290	0.2097	0.3316
	Sydney	0.4222	0.2137	0.1306	0.0763	0.2677	0.1960	0.1429
R4	NWPU	0.7533	0.5652	0.4370	0.3463	0.5965	0.3104	0.9410
	RSICD	0.5013	0.2410	0.1372	0.0850	0.3107	0.2029	0.3099
	Sydney	0.4366	0.2320	0.1421	0.0842	0.2774	0.2046	0.1589
R5	NWPU	0.7227	0.5097	0.3659	0.2682	0.5409	0.2851	0.7816
	RSICD	0.5034	0.2441	0.1380	0.0847	0.3150	0.1998	0.3079
	Sydney	0.4109	0.2037	0.1200	0.0710	0.2686	0.1848	0.1240

The proposed R4 demonstrates favorable cross-domain generalization robustness. On the Sydney test set, R4 achieved a CIDEr score of 0.1589, representing relative improvements of 11.7% and 11.2% over group B (CIDEr: 0.1422) and R1 with the full-scale remote sensing memory bank (CIDEr: 0.1429), respectively. Given that the Sydney test set contains only 58 images with an extremely small data scale and substantial scene discrepancies from the training set, this improvement indicates that the clustering-based representative remote sensing memory bank can provide more precise domain prior knowledge for the model, mitigating the cross-domain distribution shift problem.

3.2.1. Validation Set Results

To comprehensively investigate the optimal adaptation strategy for the remote sensing image captioning task, this study conducts rigorous ablation experiments centered on two core dimensions: external memory-bank domain adaptation and Q-Former parameter fine-tuning. Table 4 compares the validation results of each experimental group on three datasets with different scales and data distributions: NWPU, RSICD, and Sydney.

Table 4. Comparison of validation set results.

Group	NWPU	RSICD	Sydney
	(CIDEr/BLEU-4)	(CIDEr/BLEU-4)	(CIDEr/BLEU-4)
B	0.9867/0.3540	0.2735/0.0864	0.1312/0.0878
G1	0.4529/0.1568	0.1592/0.0590	0.1349/0.0985
R1	0.9608/0.3443	0.2825/0.0914	0.1339/0.0842
R2	0.8016/0.2813	0.2694/0.0831	0.1105/0.0850
R3	0.9644/0.3480	0.2777/0.0882	0.1443/0.0913
R4	0.9537/0.3440	0.2662/0.0834	0.1472/0.0867
R5	0.7655/0.2600	0.2667/0.0847	0.1185/0.0812

Comparing group B with R1, when the general-purpose LVIS is replaced with the remote sensing-specific memory bank, the CIDEr score on the RSICD improves from 0.2735 to 0.2825, representing a 3.3% increase, and improves by 2.1% on the Sydney dataset. Meanwhile, the performance on the NWPU decreases from 0.9867 to 0.9608, a decline of only 2.6%. This indicates that the remote sensing-specific memory bank can significantly enhance cross-dataset generalization capability while maintaining performance on the training set.

3.2.2. Effectiveness of Remote Sensing Memory-Bank Domain Adaptation

• Evaluation of Domain Memory-Bank Replacement Effect. Comparing group B with R1, the effect of “domain Memory-Bank replacement” can be directly evaluated as shown in Table 3. With the visual encoder and Q-Former frozen and only the lightweight memory–LLM adaptation layer updated, replacing the LVIS with Ext-Memory-RS (R1) results in minimal performance fluctuation on the NWPU test set, with the CIDEr score decreasing from 0.9810 to 0.9667, a relative decline of 1.5%. However, on the cross-domain RSICD test set [13] the CIDEr score improves from 0.3255 to 0.3316, achieving superior performance over the baseline. These results indicate that the extensive natural object knowledge contained in the LVIS is not essential for remote sensing image captioning. Furthermore, introducing a domain-specific remote sensing memory bank does not compromise the model’s inherent generation capability while effectively injecting domain-specific knowledge.
  R1, which incorporates Ext-Memory-RS, demonstrates the most stable performance in large-scale classification scenarios on the NWPU validation set, with both CIDEr and BLEU-4 decreasing by less than 3% relative to the baseline (Table 4 and Table 5). This configuration is suitable for applications requiring extremely high precision and sufficient computational resources.R3 with Ext-Memory-RS-random performs slightly better than Ext-Memory-RS on the Sydney validation set (+7.8% CIDEr, Table 5), but exhibits a 3.5% performance degradation on the RSICD validation set, indicating that random sampling introduces noisy samples. R4 with Ext-Memory-RS-cluster achieved a 9.9% CIDEr improvement on the Sydney validation set (0.1339 to 0.1472), confirming that representative samples more effectively capture semantic features of remote sensing scenes. Even with only approximately 39% of the full-sample bank size, R4 retains 99.3% of the validation-set CIDEr and 98.9% of the BLEU-4 of the full bank, achieving efficient knowledge utilization.
• Comparative Analysis of Adaptation Paths. Regarding adaptation effectiveness, using CIDEr on the NWPU validation set as the evaluation metric, Q-Former fine-tuning caused severe performance degradation across all experimental configurations. As shown in Table 6, all three comparison groups exhibited reductions exceeding 16%, with more severe degradation observed on smaller datasets. In contrast, the external memory-bank replacement strategy (B → R1) results in only a 2.6% CIDEr decrease on the NWPU validation set (0.9867 → 0.9608) while achieving a 3.3% improvement on the cross-domain RSICD validation set.

Table 5. Performance comparison of different memory bank types on the validation sets.

Group	Ratio	NWPU	RSICD	Sydney
		BLEU-4	BLEU-4	CIDEr
LVIS (B)	–	0.3540	0.0864	0.1312
Ext-Memory-RS (R1)	100%	0.3443	0.0914	0.1339
Ext-Memory-RS-Random (R3)	38.0%	0.3480 (+1.1%)	0.0882 (−3.5%)	0.1443 (+7.8%)
Clustered (R4)	39.1%	0.3440 (−0.1%)	0.0834 (−8.8%)	0.1472 (+9.9%)

Table 6. Comparison of adaptation pathway effectiveness.

Adaptation Pathway	NWPU	RSICD	Sydney	NWPU Change
B → R1	0.9867 → 0.9608	0.2735 → 0.2825	0.1312 → 0.1339	−2.6%
B → G1	0.9867 → 0.4529	0.2735 → 0.1592	0.1312 → 0.1349	−54.1%
R1 → R2	0.9608 → 0.8016	0.2825 → 0.2694	0.1339 → 0.1105	−16.6%
R3 → R5	0.9644 → 0.7655	0.2777 → 0.2667	0.1443 → 0.1185	−20.6%

From the adaptation cost perspective and in terms of trainable parameters, Q-Former fine-tuning requires updating all cross-modal alignment parameters, imposing high demands on both training data quantity and distribution diversity. Currently, approximately 25,200 labeled images are available in the remote sensing domain, which is insufficient to meet these requirements. For external memory-bank updates, trainable parameters are limited to the adaptation layer between the memory bank and the language decoder, resulting in far fewer parameters than full Q-Former fine-tuning. Furthermore, the update process does not disturb the cross-modal alignment space, ensuring higher training stability.

A comparison of test results between B and R1 on the NWPU is presented in Figure 4. Green highlights indicate accurate and detail-rich captions, while red highlights denote inaccurate or hallucinated descriptions.The proposed method generated more fine-grained and accurate image captions.

Figure 4. Comparison of test results between groups B and R1 on the NWPU. Green highlights indicate accurate and detailed descriptions, while red highlights denote inaccurate or hallucinated descriptions.

In summary, different types of memory banks exhibited significant adaptation differences in remote sensing scenarios, with the representative small-sample memory bank (Ext-Memory-RS-Cluster) achieving the optimal balance between performance and efficiency ratio in complex scenarios.

3.2.3. Trade-Off Between Memory-Bank Scale and Quality

As shown in Table 7, the performance and cost comparisons of the three memory-bank types exhibited significant differences. With a reduction in sample size of approximately 61%, R4 achieved performance comparable to R1 on the NWPU and even outperformed R1 on the Sydney dataset.These results indicated substantial information redundancy within the full-scale remote sensing memory bank, suggesting that blind expansion of knowledge scale yields no substantial improvement in model generalization. Using only 39% of the original samples, R4 achieved a 99.3% CIDEr retention and a 98.9% BLEU-4 retention on the validation sets while reducing memory-bank construction time and GPU memory consumption. Notably, R4 attained a CIDEr score of 0.1589 on the Sydney dataset and maintained competitiveness on the NWPU dataset, validating the effectiveness of the clustering-based redundancy removal strategy.

Table 7. Comparison of memory-bank scale, construction cost, and performance retention.

Metric Dimension	Full (R1)	Random (R3)	Clustered (R4)	Change Ratio (%)
Number of Samples	38,385	14,596	15,000	−60.9
Construction Time (h)	12.5	2.6	3.2	−74.4
GPU Memory (GB)	8.6	2.1	2.1	−75.6
Avg. BLEU-4	0.1733	0.1758	0.1714	—
Avg. CIDEr	0.4591	0.4621	0.4557	—
BLEU-4 Retention (%)	100.0	101.5	98.9	—
CIDEr Retention (%)	100.0	100.7	99.3	—

Construction Time/GPU Memory: memory-bank construction only; features extracted once and shared, excluding full-bank feature extraction. Retention = metric(R_x)/metric(R1) × 100%, no intermediate rounding; single NVIDIA A100 GPU (Santa Clara, CA, USA).

Collectively, the clustering-based selection strategy enhanced sample utilization efficiency in low-resource scenarios while reducing computational costs by preserving the most representative samples. Furthermore, in vision–language retrieval-augmented generation tasks, the “semantic coverage and purity” of the knowledge base are more critical than its “absolute quantity.”

3.2.4. Analysis of Q-Former Fine-Tuning Failure and Catastrophic Forgetting

To investigate the impact of Q-Former fine-tuning on remote sensing domain adaptation, three experimental groups R1, R2, and R5 were compared. As shown in Table 8, unfreezing the Q-Former resulted in a significant performance degradation under all datasets. On the NWPU validation set, the fine-tuned groups R2 and R5 achieved BLEU-4 scores of 0.2813 and 0.2600, respectively, lower than their corresponding frozen counterparts (R1: 0.3443; R3: 0.3480). This suggests that unfreezing the Q-Former parameters under small-sample conditions is prone to causing overfitting, thereby impairing the model’s domain adaptation capability.

Table 8. Analysis of the impact of Q-Former fine-tuning.

Group	Strategy Type	NWPU	RSICD	Sydney	Avg.
		BLEU-4	BLEU-4	CIDEr	Perf
R1	Q-Former Frozen + Full-Sample	0.3516	0.0947	0.1429	0.1964
	RS Memory Bank
R2	Q-Former Fine-Tuned + Full-Sample	0.2813	0.0831	0.1105	0.1583
	RS Memory Bank
R5	Q-Former Fine-Tuned +	0.2682	0.0847	0.1240	0.1589
	Small-Sample

This performance degradation is consistent with two commonly observed failure modes of fine-tuning under limited data. First, the Q-Former has already established a robust vision–language alignment space during pretraining; the remote sensing training dataset—comprising only 25,200 images—is likely insufficient to support learning of new stable representations when the alignment parameters are unfrozen. Second, fine-tuning on a domain with a comparatively narrow semantic distribution may bias the alignment space toward frequent patterns of that domain, a behavior akin to the catastrophic forgetting reported in continued training of large vision–language models.

This finding indicates that for retrieval-augmented vision–language models trained on moderately sized datasets, keeping the Q-Former frozen while optimizing the external knowledge base constitutes a safer and more efficient adaptation strategy than fine-tuning internal model parameters.

3.2.5. Comparison with State-of-the-Art Methods

Table 9 places the proposed framework within the broader RSIC landscape, with scores of prior methods reproduced from their original publications. These numbers serve as contextual reference rather than a strict ranking, for two reasons. First, the protocols differ fundamentally: the in-domain methods (SD-RSIC [14], PKG-Transformer [21], and DP-RSCap [33]) are trained and evaluated on RSICD itself, whereas all variants of this work are trained solely on the NWPU training split—a considerably more demanding cross-domain setting. Second, the CIDEr values cited from the original papers are not mutually comparable: SD-RSIC (0.794) and RSGPT (1.0294) differ by only 29.7%, whereas PKG-Transformer (2.7401) and DP-RSCap (3.0470)—likewise in-domain specialists—exceed RSGPT by a factor of 2.7–3.0, far beyond any plausible performance gap within this class. CIDEr is highly sensitive to evaluation conventions (the IDF corpus, caption tokenization, reference preprocessing, and scale factors), and the templated, short-reference statistics of RSICD amplify such differences to a multiplicative level; inconsistent scale conventions are documented in this literature (e.g., RSGPT reports 102.94 on a 0–100 scale). We therefore omit CIDEr from Table 9 and cite the original values in text, comparing methods on the BLEU, METEOR, and ROUGE-L scores computed with the standard pycocoevalcap toolkit. Read with these caveats, our best variant R1 attains a CIDEr of 0.3316 on RSICD under this unified protocol—below the cited values of the in-domain specialists, to which it is not directly comparable. R1 requires no parameter update on RSICD data (the training-split images enter only the retrieval memory bank) and updates only approximately 3.97 million parameters, more than an order of magnitude fewer than the roughly 10⁸ updated in RSGPT, the most directly comparable 7B-class RS-MLLM. The lower BLEU scores are consistent with the known bias of n-gram metrics against longer captions generated by LLM decoders; our METEOR scores (0.31–0.33), which account for synonymy and paraphrase, remain competitive. Accordingly, the main conclusions of this work rest on the controlled internal comparisons, where all variants share an identical backbone, training data, and evaluation implementation.

Table 9. Comparison with state-of-the-art methods on RSICD test set.

Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L
SD-RSIC [14]	0.644	0.474	0.369	0.300	0.249	0.523
PKG-Transformer [21]	0.6967	0.5830	0.5045	0.4431	0.3332	0.6078
DP-RSCap [33]	0.817	0.718	0.632	0.558	0.398	0.717
RSGPT [4]	0.7032	0.5423	0.4402	0.3683	0.3010	0.5334
Ours (B)	0.5101	0.2504	0.1434	0.0882	0.3184	0.2080
Ours (R1)	0.5153	0.2586	0.1506	0.0947	0.3290	0.2097
Ours (R4)	0.5013	0.2410	0.1372	0.0850	0.3107	0.2029

Prior-method scores are reproduced from their original publications. All scores of our variants are computed with the standard pycocoevalcap toolkit.

3.2.6. Analysis of Cross-Dataset Test Set Results

We compared the comprehensive evaluation results of experimental groups B, R1, R4, and R5. As shown in Figure 5, Ext-Memory-RS-cluster achieved superior performance under complex scenes, while the synergistic effect between the Ext-Memory-RS and Q-Former fine-tuning requires further optimization.

Figure 5. Comparison of metric results across experimental groups on the test datasets. This bar chart compares the metric scores of four experimental groups (B, R1, R4, and R5) evaluated on the NWPU, RSICD, and Sydney test datasets, where higher scores indicate better performance.

Baseline B maintained the best performance, achieving a CIDEr score of 0.9810 on the NWPU as shown in Figure 5. R1, which incorporates Ext-Memory-RS, exhibits a slight decrease in CIDEr to 0.9667 but demonstrates improved cross-dataset generalization capability. R4, which incorporates Ext-Memory-RS-cluster, retains a CIDEr score of 0.9410 while utilizing only approximately 39.1% of the samples in Ext-Memory-RS.

Group R4 shows a clear generalization advantage on the domain-distant Sydney dataset. Baseline B achieved a CIDEr of 0.3255 on the RSICD dataset. R1, which incorporates Ext-Memory-RS, improved CIDEr to 0.3316, representing a 1.87% increase and validating the effectiveness of the domain-specific memory bank.

On the Sydney dataset, baseline B achieved a CIDEr score of 0.1422. R4, equipped with Ext-Memory-RS-cluster, achieved the best CIDEr performance at 0.1589, corresponding to an 11.7% improvement over Group B. Comparative results of B, R1, and R4 on the Sydney dataset are presented in Figure 6. The cross-domain generalization capability of R4 indicates that the clustering-based selection strategy can be advantageous in small-sample settings, as observed on Sydney, the most domain-distant evaluation set in our experiments; the generality of this observation is examined in Section 4.4. On the NWPU and RSICD datasets, R4 performs slightly worse than R1, which aligns with expected outcomes. A plausible explanation is that the clustering strategy preserves global distribution representativeness while trading off local details; whether this trade-off uniformly benefits cross-domain transfer is examined in Section 4.4.

Figure 6. Comparison of results of experimental groups B, R1, and R4 on the Sydney test set. Green highlights indicate accurate and detailed descriptions, while red highlights denote inaccurate or hallucinated descriptions.

In summary, R4 exhibited differentiated performance across the three datasets in terms of in-domain fidelity and out-of-domain generalization. In near-domain scenarios, the knowledge coverage advantage of the full-sample memory bank is difficult to replace, while on Sydney, the most domain-distant evaluation set, R4 showed the largest advantage, which we hypothesize reflects a purity–generality effect (Section 4.4). These observations provide an experimental basis for selecting memory-bank strategies, with applicability boundaries delineated in Section 4.4.

4. Discussion

4.1. Trade-Off Between Memory-Bank Efficiency and Quality in Remote Sensing Domain

The experimental results revealed a complex trade-off relationship between memory-bank scale and quality. Comparison between Ext-Memory-RS and Ext-Memory-RS-Random showed that the random bank matched the full bank on the validation sets (BLEU-4 retention 101.5%, CIDEr retention 100.7%), indicating a substantial proportion of redundant samples within the full memory bank. This finding provided a theoretical basis for subsequent intelligent sampling strategies.

The K-means semantic clustering-based memory bank achieved effective memory-bank compression while maintaining performance comparable to random downsampling. R4, which incorporates Ext-Memory-RS-cluster, outperformed R1 with Ext-Memory-RS on the Sydney dataset validation set (CIDEr: 0.1472 vs 0.1339), suggesting that clustering-based sample selection enhanced memory efficiency and domain-distant transfer, while its generalization varied across datasets (Section 4.4). This improvement can be attributed to the feature-space representativeness of the K-means selection: the clustered bank is less redundant (participation ratio 2.53 vs. 2.06) and better substitutes the non-selected samples at retrieval time, although it reorganizes the per-category composition and thus introduces a bounded distribution shift (Section 2.2.6); its uneven cross-dataset behavior is analyzed in Section 4.4.

From the perspective of memory efficiency, both downsampling methods achieved an approximately 61% reduction in the number of stored bank entries, with proportionally lower storage and bank-construction cost; end-to-end deployment efficiency was not separately benchmarked.

4.2. Effectiveness of the Staged Transfer Training Strategy

Our findings revealed that the fine-tuning strategy for Q-Former parameters substantially impacts model performance. G1, trained from scratch on NWPU with the general-purpose LVIS memory bank and with the Q-Former unfrozen (Table 2), exhibited severe performance degradation, with CIDEr on the NWPU validation set dropping to 0.4529, a reduction of 54.1%. Even under the pretraining plus fine-tuning framework (R2 and R5), Q-Former unfreezing consistently produced negative effects. In contrast, R1, employing the staged transfer training strategy, effectively leveraged pretrained knowledge, maintaining a CIDEr score of 0.9608.

The core principle of this strategy is to preserve the cross-modal alignment capability pretrained in the Q-Former while achieving domain adaptation through external memory-bank updates. This approach ensures model stability while enabling effective knowledge transfer. Experimental results validated the effectiveness of this strategy, which is particularly crucial for remote sensing tasks with limited labeled data.

4.3. Superior Efficiency of External Memory-Bank Adaptation over Fine-Tuning

Comparison between R2 and R4 demonstrated that for remote sensing image captioning tasks, updating the external memory bank achieved higher adaptation efficiency than fine-tuning internal model parameters during domain adaptation of retrieval-augmented vision–language models. Under conditions of moderately sized remote sensing labeled data, the external memory-bank adaptation path outperformed internal parameter fine-tuning in both performance stability and training cost, providing a more reliable paradigm for domain adaptation of retrieval-augmented vision–language models.

4.4. Dataset-Dependent Generalization of the Clustered Memory Bank

Table 3 shows R4 trailing R1 by 2.66% on NWPU and 6.54% on RSICD in test set CIDEr but leading by 11.2% on Sydney. We offer three hypotheses: K-means minimizes visual-feature reconstruction rather than captioning objectives like rare categories, counts, and spatial relations, plausibly discarding tail patterns; centroid-nearest samples are lexically more generic, limiting near-domain richness, while the full bank retains in-domain variety. None is directly tested; “semantic purity” was not quantified. The clustered bank is thus a design choice for tight storage budgets, pending target-distribution validation, not a presumed monotonic domain-distance gain. Future work will quantify per-category performance, tail coverage, retrieval relevance, and equal-size seed effects.

5. Conclusions

We introduced a domain-specific memory-augmented multimodal model for remote sensing image captioning. By constructing a remote sensing visual memory bank, adopting a staged transfer training strategy, and optimizing small-sample memory-bank construction, our approach addresses domain mismatch, low training efficiency, and high computational cost—key limitations of existing remote sensing image captioning models. Extensive experiments on three benchmark datasets validated the effectiveness of the proposed method: the domain-specific visual memory bank improved the CIDEr score on the RSICD test set by 1.87% (0.3255 to 0.3316); the staged transfer training strategy reduced overfitting and training complexity; and the representative small-sample memory bank achieved an 11.7% CIDEr score improvement on the Sydney test set (0.1422 to 0.1589) while using only about 39% of the samples in the full memory bank (a 61% reduction), at the cost of moderate decreases on the NWPU and RSICD test sets.

Despite these advances, two limitations remain: first, the K-means-based clustering strategy exhibited inconsistent generalization across datasets due to varying visual distributions, warranting exploration of more robust clustering approaches; second, only three memory-bank configurations were evaluated (full-sample, random downsampling, clustering-based), leaving the exact functional relationship between memory scale and performance unestablished. Future work could conduct systematic scale-gradient experiments to quantify this relationship rigorously.