A Review of Visual Grounding on Remote Sensing Images

Abstract

Remote sensing visual grounding, a pivotal technology bridging natural language and high-resolution remote sensing images, holds significant application value in disaster monitoring, urban planning, and related fields. However, it faces critical challenges due to the inherent scale heterogeneity, semantic complexity, and annotation scarcity of remote sensing data. This paper first reviews the development history of remote sensing visual grounding, providing an overview of the basic background knowledge, including fundamental concepts, datasets, and evaluation metrics. Then, it categorizes methods by whether they employ large language models as a pedestal, and provides in-depth analyses of the innovations and limitations of Transformer-based and multimodal large language model-based methods. Furthermore, focusing on remote sensing image characteristics, it discusses cutting-edge techniques such as cross-modal feature fusion, language-guided visual optimization, multi-scale, and hierarchical feature processing, open-set expansion and efficient fine-tuning. Finally, it outlines current bottlenecks and proposes valuable directions for future research. As the first comprehensive review dedicated to remote sensing visual grounding, this work is a reference resource for researchers to grasp domain-specific concepts and track the latest developments.

1. Introduction

The leapfrog advancement of remote sensing technology has propelled remote sensing image quality and resolution into the sub-meter-level era, where massive high-precision earth observation data provides unprecedented informational support for urban planning, environmental monitoring, national defense, and related domains. To overcome the efficiency bottleneck of traditional manual interpretation, researchers have dedicated efforts to establishing intelligent interaction bridges between natural language and remote sensing images. In remote sensing image captioning [1], natural language processing techniques enable the automated generation of precise and descriptive textual summaries, facilitating rapid comprehension of scene-level characteristics. For cross-modal image–text retrieval [2], alignment algorithms such as Contrastive Language-Image Pretraining (CLIP) [3] and Bootstrapping Language–Image Pretraining (BLIP) [4] empower bidirectional querying: retrieving images via textual descriptions or extracting textual metadata from visual content, thereby significantly enhancing data mining efficiency. In visual question answering (VQA) [5], algorithms demonstrate capabilities to resolve complex queries about object locations, area estimations, and other geospatial attributes within remote sensing scenes, offering intelligent decision-making support for resource management and disaster assessment. Despite these achievements, the more practical and challenging task of Remote Sensing Visual Grounding (RSVG) remains underexplored, constrained by theoretical and technical limitations.

RSVG aims to localize target objects in remote sensing images through natural language queries (phrases or sentences) and output corresponding bounding boxes. Unlike conventional object detection [6], which identifies all instances of predefined categories, RSVG simulates real-world referential dialogs, addressing complex localization demands in specialized scenarios. Compared to visual grounding in natural images, RSVG has three domain-specific characteristics:

• Scale Heterogeneity: Remote sensing images span square-kilometer-scale urban clusters to sub-meter-scale individual targets, where small objects (e.g., ships, vehicles) coexist with large structures (e.g., airports, ports). This diversity challenges traditional detectors with fixed receptive fields.
• Semantic Complexity: Due to resolution limits in remote sensing imagery, small targets often have complex textures and insufficient edge detail [7]. This causes bidirectional ambiguity in visual–language mapping, where the complexity of accurate semantic expression can lead to information loss and attention drift during feature extraction, while simple expressions can easily cause target confusion in dense scenes. Additionally, target extraction is heavily impacted by background interference.
• Annotation Scarcity: Restricted data accessibility and expertise-dependent annotation result in significantly smaller datasets than natural-scene benchmarks, severely undermining model generalization capabilities.

While visual grounding in natural images has evolved over a decade, progressing from two-stage [8] proposal matching, one-stage [9] end-to-end regression, and transformer-based cross-modal encoding to the Multimodal Large Language Model (MLLM) [10], direct adaptation to remote sensing suffers significant performance degradation. The fundamental contradiction lies in the essential difference between natural images and remote sensing data: the former has prominent subjects and straightforward semantics, while the latter needs to parse compound semantics, and the dense and small targets of remote sensing images are easily submerged in the complex background. The introduction of Geospatial Visual Grounding (GeoVG) [11] in 2022 marked the initial effort to adapt visual grounding to remote sensing by constructing geospatial relational graphs to compress search spaces, albeit limited by coarse-grained feature alignment. Subsequent work [12] improved accuracy through multiscale cross-modal fusion but struggled with missed detections of small targets and inadequate parsing of complex linguistic descriptions.

As MLLM breaks through the scalinglLaw [13] of traditional machine learning, it injects new momentum into the solution of remote sensing visual grounding tasks that require cross-modal understanding by virtue of its hundreds of billions of parameters and massive cross-modal pretraining data. GeoChat [14] pioneered conversational interaction with high-resolution remote sensing images, enabling coordinate outputs via natural language instructions, though its precision lags behind dedicated models. GeoGround [15] unified horizontal bounding boxes (HBBs), oriented bounding boxes (OBBs), and segmentation masks through text-mask serialization, yet its computational complexity hinders real-time deployment.

Although the Transformer-based method and the MLLM-based method have made a stage breakthrough in the field of remote sensing visual grounding, they still face three new types of challenges: First, the semantic fragmentation of heterogeneous data from multiple sources may lead to the inefficiency of cross-modal graphic alignment, and it is difficult for the existing methods to rely on the annotation system dominated by optical remote sensing images to support the generalization requirements of open-set scenarios. Secondly, the static one-time-phase model cannot capture the spatial and temporal heterogeneity of dynamic processes such as flood inundation and urban expansion, and the decoupling of temporal semantic description and spatial localization capability seriously restricts the decision-making effectiveness of key applications such as disaster warning. Finally, the harsh resource constraints of the starboard/aircraft platforms form a sharp contradiction with the demand for high-resolution image processing, and the bottleneck of the real-time nature of the existing models needs to be broken through urgently. To address these challenges, this study systematically dissects three core issues—intelligent annotation of multisource data, spatiotemporal modeling for dynamic scenes, and lightweight edge deployment—and proposes knowledge-driven approaches, spatiotemporal coupling strategies, and computational resource coordination mechanisms, offering novel insights to advance RSVG research and practical implementation. Our contributions can be summarized as follows: First, we collected and tracked literature related to RSVG on Web of Science and Google Scholar, defined RSVG, traced its development history, and classified methods based on technical details. Then, we discussed RSVG benchmark datasets and evaluation metrics, analyzed and compared the performance of existing methods on classic datasets, and summarized research trends. Second, we focused on the characteristics of visual grounding tasks in the field of remote sensing and analyzed the innovations and discussions proposed by scholars regarding the characteristics of RSVG. Finally, we integrated current research challenges and provided valuable directions for future research to inspire subsequent researchers. To the best of our knowledge, this is the first systematic review of RSVG, and it aims to offer strategic and detailed references for researchers in this field.

This review systematically examines the technological framework of RSVG. To identify high-quality publications relevant to RSVG, we adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [16], as illustrated in Figure 1. We first collected relevant articles through Web of Science, Google Scholar, and IEEE Xplore, using keywords such as “remote sensing visual grounding,” “remote sensing referring expression comprehension,” and “remote sensing phrase grounding” to search for all articles up to 10 May 2025. We then established exclusion and inclusion criteria to screen records and full-text articles. The exclusion criteria were as follows:

• Duplicate work.
• Articles without publicly available reproducible code or peer review.
• Articles where the full text could not be obtained from the publisher.

The inclusion criteria were as follows:

• Articles written in English.
• Images used in the study must be remote sensing images (RSI) and cannot be other types of images.
• Methods that integrate complete sentence text (rather than individual or multiple discrete words).

A total of 62 papers and articles were initially identified. After applying the eligibility criteria and manually screening out redundant papers and articles, 28 papers and articles were ultimately selected for this review study. The remaining sections are structured as follows: Section 2 outlines the background, including concept definition, mainstream datasets, and evaluation metrics. Section 3 deconstructs existing methodologies into Transformer-driven and MLLM-driven paradigms based on architectural differences, offering in-depth critiques of their design philosophies and performance limitations. Section 4 addresses domain-specific innovations in RSVG, exploring advancements in cross-modal feature fusion, language-guided visual optimization, multi-scale and hierarchical feature processing, open-set expansion and efficient fine-tuning. Section 5 is based on application scenarios, reveals the core challenges, and provides an outlook.

2. Background

2.1. Concept Definition

Remote sensing visual grounding, also known as remote sensing referring expression comprehension [17], aims to achieve accurate spatial localization of specific targets in remote sensing images through natural language descriptions. The technique maps unstructured textual instructions to bounding box coordinates in the image by establishing cross-modal associations between visual features and semantic descriptions.

Compared with traditional object detection methods, remote sensing visual grounding is fundamentally different in the task paradigm. Traditional object detection [6] is usually based on supervised learning with a closed set of categories, and its performance is limited by the completeness of predefined categories and labeled data. Figure 2a shows the localization results of a ‘vehicle’ target in a remote sensing image. Remote sensing visual grounding uses open vocabulary descriptions as inputs, and is capable of handling compound semantic queries and dynamic parsing of implicit semantic constraints through multimodal feature fusion. As shown in Figure 2b, the visual grounding task can not only locate the ‘vehicle’ target, but also filter out the targets that are more suitable for the target based on the modifiers ‘white’ and ‘in the sunlight’. This openness makes the visual grounding more suitable for emergency response, environmental monitoring, and other fields that require flexible queries.

In particular, RSVG differs conceptually from Referring Remote Sensing Image Segmentation (RRSIS). While both involve remote sensing images and text descriptions, RSVG provides coarse target localization via bounding boxes, whereas RRSIS generates pixel-level masks for precise contour extraction, as shown in Figure 2c. Technically, RSVG focuses on cross-modal reasoning efficiency and often employs region recommendation networks based on the attention mechanism to quickly filter candidate regions through spatial relationship modelling, while referring image segmentation is oriented towards more accurate pixel-level feature extraction, which needs to solve problems such as edge blurring and target sticking, and usually relies on language-guided cascade segmentation architectures to handle fine-grained features.

Table 1. A comparison of traditional object detection, visual grounding, and referring image segmentation.

	Traditional Object Detection	Visual Grounding	Referring Image Segmentation
Input Modality	visual modality	visual modality+ linguistic modality	visual modality+ linguistic modality
Output Form	bounding boxes	Bounding boxes	pixel-level masks
Target Definition	Predefined closed categories	Open-vocabulary descriptions	Fine-grained semantic descriptions

compares the significant differences between the three tasks of traditional object detection, visual grounding, and referring image segmentation.

2.2. Datasets and Benchmark

The construction and annotation of relevant datasets have become critical factors driving advancements in this field. Sun et al. established the first remote sensing visual grounding dataset, the RSVG dataset (RSVGD) [11], laying the foundation for research in this domain. DIOR-RSVG [12] created with manually validated automatic generation algorithms based on the large-scale object detection dataset detection in optical remote sensing (DIOR) [18], covers 20 different target categories with high inter-class similarity and intra-class diversity, providing rich and reliable annotation data for visual grounding experiments. Lan et al. [19] proposed the RSVG-HR dataset, focusing on high-resolution remote sensing images, which contains 2650 image–text pairs. By re-annotating high-resolution images in the RSVGD dataset using an absolute-position and relative-position schema, RSVG-HR offers more challenging task scenarios. Li et al. [20] constructed the OPT-RSVG dataset by collecting 25,452 images from High-Resolution Remote Sensing Detection (HRRSD) [21], DIOR, and swimming pool and car detection (SPCD) [22] datasets and creating 48,952 image–text pairs. It introduces more complex scenes, a wider spatial resolution span, and richer object categories, providing more challenging data resources for RSVG tasks. To address the limitation of datasets being confined to optical remote sensing images, Li et al. [23] proposed the visual grounding for high-resolution synthetic aperture radar images (SARVG) dataset, focusing on visual grounding for synthetic aperture radar (SAR) images. Containing 2465 high-resolution SAR images and 7617 image–text pairs, it provides a valuable resource for visual grounding studies of SAR images, but the dataset contains only a single target, the power transmission tower, and target diversity is lacking.

Current instruction-tuning datasets for multimodal large models are primarily constructed from established benchmarks such as RSVG, DIOR-RSVG, and OPT-RSVG. Notably, SkySenseGPT [24] extends visual grounding to object reasoning tasks, annotating over 210,000 targets in rotated bounding box format with coordinate precision elevated to the millimeter level (0.01), significantly advancing spatial localization accuracy. Benchmarks designed to evaluate multimodal large models are evolving toward more complex scenarios, safety-oriented requirements, and large-scale scene challenges. For instance, the Versatile vision-language Benchmark for Remote Sensing image understanding (VRSBench) [25] proposed by Li et al. supports both horizontal and rotated bounding box localization across 26 object categories. While the DIOR-based detection dataset contains an average of 3.3 instances per image, the DOTA-v2 [26] subset escalates instance density to 14.2 instances per image, creating highly challenging scenarios for dense small-object localization. The Comprehensive and Objective Benchmark for Evaluating the Remote Sensing Capabilities of Large Vision-Language Models (COREval) [27] addresses potential data leakage risks by constructing test sets from six satellite platforms, such as Landsat-8 and Sentinel-1/2, to ensure objective generalization assessment. Wang et al. [28] developed a benchmark for ultra-large-scale scenes, manually annotating 12,619 visually grounded instances with unique attributes (e.g., color, shape, position, size, relative location, etc.) from ultra-high-resolution images with an average size of 8500 × 8500. Some objects in this dataset occupy as few as five pixels, establishing new standards for microscopic-level localization and anti-background interference testing in massive scenes.

Table 2. Major datasets for remote sensing visual grounding tasks.

Datasets	Type	Image Sources	Ann. Format	Total Images	Total Objects	Avg. Length	Image Size
RSVGD [11]	RGB	DIOR	HBB	4239	7933	28.33	1024 × 1024
DIOR-RSVG [12]	RGB	DIOR	HBB	17,402	38,320	7.47	800 × 800
RSVG-HR [19]	RGB	DIOR	HBB	2650	2650	19.6	1024 × 1024
OPT-RSVG [20]	RGB	HRRSD, DIOR, SPCD	HBB	25,452	48,952	10.10	-
RSSVG [29]	RGB	FAIR1M, CGWX, DIOR-RSVG	HBB	11,157	25,237	9.77	-
SARVG-T [23]	SAR	CAPELLA, GF-3, ICEYE SAR	HBB	2465	7617	-	512 × 512
SARVG-S [29]	SAR	SAR-ship-Dataset	HBB	43,798	54,429	7.72	-
Benchmark
VRSBench [25]	RGB	DOTA-v2, DIOR	OBB	29,614	52,472	14.31	512 × 512
COREval [27]	RGB	Google Earth	HBB/OBB	-	200	-	800 × 800
XLRSBench [28]	RGB	DOTA-v2, ITCVD	OBB	-	12,619	-	8500 × 8500

Since both Li et al. [23] and Chen et al. [29] proposed SARVG datasets, SARVG-T denotes the dataset with the power transmission tower as the target, while SARVG-S denotes the dataset with the ship as the target, for distinction purposes.

provides a comparative analysis of mainstream datasets for RSVG tasks. Table 2 summarizes major datasets for remote sensing visual grounding tasks. Appendix B comprehensively documents the partitioning methodologies employed by the datasets cataloged in Table 2.

2.3. Evaluation Metrics

The evaluation framework for remote sensing visual grounding tasks typically adopts the classic assessment paradigm proposed by Zhan et al. [12]. For precision-based metrics (Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, Pr@0.9), they measure the accuracy when the intersection-over-union (IoU) between the predicted bounding box and the ground-truth box exceeds thresholds of 0.5, 0.6, 0.7, 0.8, and 0.9, respectively. These metrics intuitively reflect the model’s prediction accuracy under different overlap requirements. Additionally, mean intersection-over-union (meanIoU) calculates the arithmetic mean of IoU values for all image–language pairs in the dataset, as shown in Equation (1), evaluating the model’s localization capability across diverse scenarios. Cumulative intersection-over-union (cumIoU) is defined as the ratio of the total intersection area to the total union area across all samples, as shown in Equation (2), emphasizing the model’s performance under the overall data distribution.

$$meanIoU={\displaystyle \frac{1}{M}}\sum _{t=1}^M{\displaystyle \frac{I_t}{U_t}}$$

(1)

$$cumIoU={\displaystyle \frac{{\sum }_t{I}_t}{{\sum }_t{U}_t}}$$

(2)

where $t$ is the sample index, M is the dataset size, and $I_t$ and $U_t$ denote the intersection and union areas between the predicted and ground-truth boxes, respectively.

3. Evolutionary Trajectory

Early visual grounding research for natural images began to emerge in 2014, with the proposal of two-stage [8] and one-stage [9] architectures using Long Short-Term Memory (LSTM) [30] as the language encoder and a Convolutional Neural Network (CNN) as the visual encoder to address cross-modal understanding requirements for visual grounding tasks. Two-stage visual grounding methods often treat visual grounding as an object retrieval task, identifying the object with the highest semantic match to the text expression from a set of object proposals. This approach consists of two stages: the first stage generates sparse object region proposals using object detection methods or unsupervised methods; the second stage matches object regions with text expressions to select the optimal object as the prediction result. A common practice is to encode candidate objects and text expressions using CNN and LSTM, respectively, and determine the output by calculating similarity. However, the second stage of the two-stage method relies heavily on the object detection results of the first stage. If the target region is not accurately detected in the first stage, the second stage cannot perform matching and localization effectively. Additionally, the computation of cross-modal similarity between numerous object region proposals and text expressions consumes substantial computational resources.

One-stage methods overcome the dependency on the first stage’s results through end-to-end training. This approach directly performs visual–language fusion in the intermediate layers of the object detector and outputs the bounding box with the highest score on predefined dense anchors. Inspired by YOLOv3 [31], Yang et al. [32] utilized DarkNet and Bidirectional Encoder Representation of Transformer (BERT) to extract visual and text features, respectively, and integrated text features into the YOLOv3 detector for visual grounding. Researchers have also made diverse attempts to improve model performance: Sadhu et al. [33] extended the task to zero-shot localization; Liao et al. [34] transformed visual grounding into a correlation filtering problem; Yang et al. [9] designed a recursive subquery construction module to handle long and complex sentences; Ye et al. [35] proposed a filter-based cross-modal fusion network to filter visual feature maps using structured knowledge and context. Although one-stage methods simplify the training process and reduce computational costs, they rely on manually designed mechanisms and complex modules for multimodal reasoning, making them prone to overfitting on scenario-specific datasets and lacking generality and generalization.

In 2020, the Transformer model was first applied to image classification tasks [36] and achieved better results than CNN models. This application inspired researchers to explore the potential of using Transformer architectures to unify visual–text feature representations, bringing new insights to cross-modal tasks like visual grounding. Deng et al. [17] proposed a Transformer-based object coordinate regression model that performs cross-modal fusion through stacked Transformer encoder layers and predicts target locations; Du et al. [37] utilized text-guided self-attention to mine semantic information and improve coordinate regression performance. Transformer-based visual grounding methods surpass traditional approaches by utilizing attention mechanisms to develop efficient feature extraction modules. These modules are adept at capturing cross-scale visual information and improving contextual relationships [38], demonstrating high versatility and effectiveness. Their improved performance in object detection tasks and wide applicability have gradually replaced earlier two-stage and one-stage methods.

Due to significant differences in target characteristics and scale between remote sensing images and natural images, both two-stage and one-stage methods exhibit limited applicability. Following the success of Transformer-based visual grounding in natural image processing, Sun et al. first adapted this approach for remote sensing in October 2022 [12], as shown in Figure 3. Later, Zhan et al. [12] advanced the field by releasing the DIOR-RSVG benchmark dataset and open-sourcing their Multistage Synergistic Aggregation Module implementation, providing essential resources for deep learning-based visual grounding. Although RSGPT appeared in 2023 as the first multimodal large language model for remote sensing, it did not achieve region-level localization capabilities. A key breakthrough occurred in November 2023 with GeoChat [14], which successfully integrated visual grounding into the MLLM framework. This milestone laid the groundwork for subsequent MLLM-based methods, driving faster development and notable performance gains from 2024 onward.

This section categorizes remote sensing visual grounding methods into Transformer-based architectures and MLLM-based methods, depending on whether they utilize large language models as foundational backbones. Representative frameworks of both paradigms are introduced, with their architectural differences comparatively illustrated in Figure 4.

3.1. Transformer-Based Methods

Transformer has become the mainstream architecture for remote sensing visual grounding tasks due to its powerful parallel processing capabilities and modeling of long-range dependencies. Transformer-based methods typically use ResNet-50 or DarkNet-53 as the visual encoder and BERT as the language encoder during the encoding stage, as shown in

Table 3. Summary of Transformer-based and MLLM-based methods: architectural configurations, dataset, and performance on DIOR-RSVG benchmark. Appendix A compiled the GitHub links corresponding to the Transformer-based and MLLM-based methods.

Methods	Visual Enc.	Text Enc./LLM	Params.	Training Set	Test Set	Pr@0.5	mIoU
Transformer-based methods
GeoVG [11]	-	-	-	26,991	7500	57.78	-
MGVLF [12]	ResNet-50	BERT	152.5	26,991	7500	76.78	68.04
LQVG [19]	ResNet-50	BERT	166.3	26,991	7500	83.41	74.02
APMOR [40]	ResNet-101	BERT	-	26,991	7500	79.37	68.86
Eff-Gounding DINO [41]	ResNet-50	BERT	169.3	26,991	7500	83.05	73.41
RINet [42]	DarkNet-53	BERT	-	26,991	7500	64.14	-
CrossVG [43]	ViT-B/16	BERT	-	26,991	7500	77.51	70.56
VGRSS [29]	ResNet-50	BERT	-	26,991	7500	83.01	74.85
MSANet [39]	DarNet-53	BERT	-	26,991	7500	74.23	64.88
VSMR [44]	ResNet-50	BERT	-	15,328	19,160	78.24	68.88
QAMFN [45]	ResNet-50	BERT	128.4	15,328	19,160	81.67	71.48
MSVG [46]	ResNet-101	BERT	-	15,328	19,160	83.61	72.87
LPVA [20]	ResNet-50	BERT	156.2	15,328	19,160	82.27	72.35
FQRNet [55]	ResNet-50	BERT	-	15,328	19,160	77.23	68.35
CSDNet [47]	ResNet-101	BERT	154.64	27,133	7422	80.92	70.88
TACMT [23]	ResNet-50	BERT	150.9			-	-
b. MLLM-based Methods
GeoChat [14]	CLIP-ViT	Vicuna-v1.5	~7 B	GeoChat-Instruction	555	-	-
SkyEyeGPT [48]	EVA-CLIP	LLaMA2	~7 B	SkyEye-968k	7500	88.59	-
EarthGPT [49]	DINO-ViT+ CLIP-ConNeXt	LLaMA2	~7 B	MMRS-1M	7500	76.65	69.34
SkySenseGPT [24]	CLIP-ViT	Vicuna-v1.5	~7 B	FIT-RS	-	-	-
LHRS-Bot [50]	CLIP-ViT	LLaMA2	~7 B	LHRS-Instruct	7500	88.10	-
VHM [51]	CLIP-ViT	Vicuna-v1.5	~7 B	VariousRS-Instruct	-	-	-
GeoPix [52]	CLIP-ViT	Vicuna-v1.5	~7 B	GeoPixInstruct	-	-	-
EarthDial [53]	InternViT	Phi-3-mini	~4 B	EarthDial-Instruct	-	-	-
GeoGround [15]	CLIP-ViT	Vicuna-v1.5	~7 B	refGeo	7500	77.73	-

For Transformer-based methods, Training set refers to the number of training instances from DIOR-RSVG. For MLLM-based methods, Training set refers to the fine-tuning dataset used.

. They construct visual–text interaction modules based on self-attention mechanisms to enhance alignment between visual and language modalities, thereby improving visual grounding accuracy. The basic architecture is shown in Figure 4a.

Sun et al. [11] pioneered the introduction of visual grounding to the remote sensing field in 2022 with their GeoVG method, which uses a graph-based strategy to establish relationships between visual and language features. The numerical context module in the language encoder represents complex expressions as geospatial relationship graphs, reducing the search space in large-scale scenes and improving localization accuracy. However, creating relationship graphs requires high computational costs and tends to overlook relationships between small target objects. Wang et al. [39] proposed an autoregressive discrete coordinate sequence generation method to explore interactions between direct regression features and encoded multimodal features, achieving a mean IoU of 64.88% on RSVG. The novel frequency and query refinement network (FQRNet) [55] captures global structural information from remote sensing data using Fourier transforms and enhances spatial features with frequency information. Li et al. [23] designed a cross-modal encoder for SAR data in their TACMT model to strengthen text-guided scattering feature extraction, improving accuracy by 6.2% on the SARVG1.0 dataset.

In summary, Transformer-based methods exhibit strong feature extraction capabilities and flexible architectural designs, performing well in handling complex backgrounds and small-target localization. However, there is still room for optimization in multimodal feature fusion, addressing non-salient target features, and mitigating background interference.

3.2. MLLM-Based Methods

With the breakthrough of large language models (LLMs), multimodal large language models have provided new opportunities for automated analysis of earth observation data. By integrating visual and language modality information, MLLMs can more naturally process human language instructions and enhance open-scene understanding through large-scale data training and instruction tuning. General MLLMs support a wide range of tasks, including classification, detection, captioning, question answering, and visual reasoning. As an important downstream task, visual grounding has also achieved breakthroughs with the development of large models. These models typically rely on pretrained large language models and use specific visual encoders to align image and language features for unified multimodal task processing. Figure 4b illustrates the basic framework of MLLM-based visual grounding.

GeoChat [14] proposed by Zhou et al. first demonstrated remote sensing visual grounding capabilities in the MLLM. It uses CLIP-ViT as the visual backbone to align visual and language modalities, inserts positional encoding, and scales image input sizes to handle larger images. However, it has obvious limitations in localizing small targets, with an acc@0.5 value of only 2.9%. Zhan et al. [48] designed the SkyEyeGPT, which maps remote sensing visual features to the language domain through a simple projection layer, significantly improving the precise localization of small objects.

Compared to the massive data scale in natural scenes that supports the development of multimodal large models, the lack of training data in remote sensing has become a bottleneck. Muhtar et al. [50] pretrained on the large 4-million-image–text-pair dataset LHRS-Align and fine-tuned on the 30,000-instruction-pair dataset LHRS-Instruct, combining multi-level visual–language alignment strategies to unleash the potential of MLLMs. EarthDial [53], proposed by Soni et al., supports natural language dialog for multispectral, multitemporal, and multiresolution remote sensing data, using 11.11 million instruction pairs containing RGB, Sentinel-2, SAR, near-infrared, and infrared data for comprehensive instruction tuning to achieve stronger generalization.

Additionally, some scholars have introduced visual prompt models or combined rotated bounding boxes and masks to localize finer-grained targets. SkySenseGPT [24] extends visual grounding to object reasoning tasks, using rotated bounding boxes to improve the accuracy of target fitting compared to traditional horizontal bounding boxes. EarthMarker [54], proposed by Zhang et al., first introduced visual prompt learning into remote sensing multimodal large models, allowing users to interact with AI using prompts such as boxes, points, and free-form shapes, breaking the limitations of language instructions and enhancing flexibility. GeoPix [52] extends visual grounding to the pixel level and introduces a Class-wise Learnable Memory (CLM) module to dynamically extract and store category-specific geographic context, improving the model’s understanding of diverse instances in complex remote sensing scenes. However, when user instructions for referring segmentation and visual context analysis accidentally include instance location queries, task interference can occur, causing errors in segmentation masks in visual grounding tasks, as shown in Figure 5. GeoGround [15] leverages the powerful multi-task learning capabilities of LLMs, combining prompt-assisted learning (PAL) and geometry-guided learning (GGL) to unify visual grounding tasks with OBB, HBB, and mask annotations, allowing flexible output choices.

MLLM-based methods facilitate alignment and interaction between visual and language modalities by embedding text and images into a unified semantic space. Built on large-scale pretraining data, they exhibit excellent zero-shot generation capabilities and open-domain adaptability. MLLMs often inherit the reasoning capabilities of LLMs, supporting deep semantic understanding and reasoning for complex language instructions, with more flexible output formats. The strong language generation and multimodal fusion capabilities of MLLMs provide new approaches for remote sensing visual grounding. Nonetheless, the substantial parameter count of these models imposes substantial demands on both training costs and cross-modal data acquisition. Furthermore, the autoregressive mechanism inherent in LLMs, when utilized for coordinate generation, introduces a conflict between their sequential processing nature and the parallel processing capabilities demanded by dense object detection tasks. This, along with the high computational complexity and the mismatch between the model’s output structure and the requirements of object detection, collectively constrain the performance of MLLMs in localizing small objects.

Table 4. Comparative analysis of Transformer-based vs. MLLM-based approaches.

	Transformer-Based	MLLM-Based
Visual Encoder	ResNet-50/DarkNet-53	ViT
Text Encoder	BERT	-
Strengths	Computational efficiency Task specificity Low training cost	Strong cross-modal alignment Open-world adaptability Advanced semantic reasoning
Weaknesses	Limited generalization Shallow semantic understanding Coarse alignment	High resource demands Localization constraints from text regression Data dependency

compares the Transformer-based and MLLM-based methods.

4. Characteristics and Innovations

Compared to generic visual grounding, the development of remote sensing visual grounding has consistently centered on three core characteristics: scale heterogeneity, semantic complexity, and annotation scarcity.

4.1. Scale Heterogeneity

Acquired from nadir satellite perspectives, remote sensing imagery exhibits extensive spatial coverage, capturing objects ranging from square-kilometer-scale landforms to sub-meter-level fine targets. This target diversity challenges conventional detectors with fixed receptive fields, which struggle to accommodate multiscale objects and suffer severe background noise interference. Consequently, researchers have pursued multiscale hierarchical feature processing to mitigate scale heterogeneity and background clutter in large-scale scenes, thereby expanding dynamic perceptual coverage. Zhan et al. [12] proposed an MGVLF module, which integrates multiscale visual features and multi-granularity textual embeddings to adaptively filter irrelevant noise, effectively tackling challenges posed by significant scale variations and cluttered backgrounds. Further advancing this approach, Wang et al. [39] introduced a Multistage Synergistic Aggregation Module (MSAM), achieving multi-scale contextual fusion through generative coordinate sequence prediction. This method attained an accuracy of 83.61% on the DIOR-RSVG dataset. To resolve confusion between targets and similar objects, Qiu et al. [40] developed a learnable attribute prompter that adaptively explores diverse attribute information based on common object characteristics in remote sensing images. RINet [42] adopted a local-to-object scheme to progressively localize target regions via a Regional Indication Generator (RIG), enhancing localization capabilities for small targets. Ding et al. [44] designed an Adaptive Feature Selection (AFS) module to suppress noise and combined it with a multistage decoder to iteratively infer target attributes, achieving a precision of 78.24% in complex scenarios. While innovations in multi-scale and hierarchical feature processing effectively mitigate scale variations and background interference, challenges persist in slow model convergence and insufficient validation of detection performance in high-density scenarios.

4.2. Semantic Complexity

Remote sensing images contain diverse and numerous targets, with small objects such as ships, vehicles, and buildings constituting a significant proportion. A single target may correspond to varied textual descriptions, while identical expressions can refer to different objects. Precise semantic expressions are often lengthy and complex, frequently encountering issues like description forgetting and attention drift during feature extraction and matching. Conversely, oversimplified expressions risk ambiguity, particularly in large-scale scenes with densely clustered targets, where object confusion severely impedes precise localization. To enhance semantic consistency between visual and textual features, researchers employ multilevel alignment and dynamic interaction mechanisms to optimize cross-modal feature fusion. Early studies achieved coarse-grained alignment by constructing geospatial relation graphs but struggled to parse complex descriptions. Zhan et al. [12] proposed the MGVLF module to integrate multiscale visual features and multi-granularity textual embeddings. Lan et al. [19] introduced the Language Query-based Visual Grounding (LQVG) framework, which retrieves multiscale visual features using textual features as queries and incorporates a Multistage Cross-Modal Alignment (MSCMA) module to strengthen semantic correlations. Their approach achieved accuracies (Pr@0.5) of 83.41% and 87.37% on the DIOR-RSVG and RSVG-HR datasets, respectively, though with higher computational complexity. Choudhury et al. [43] simplified traditional multimodal fusion modules in their CrossVG model, relying solely on Sstacking Transformer encoder layers to achieve efficient cross-modality interaction. Multidimensional Semantic-Guidance Visual Grounding (MSVG) [45] further employed a Multidimensional Text–Image Alignment Module (MTAM) to increase the relevance between visual features and textual descriptions. Overall, cross-modality fusion strategies—such as constructing geospatial relation graphs and multi-granularity feature fusion—effectively narrow the search space and improve semantic matching capabilities.

Furthermore, language-guided visual optimization can mitigate attention drift and enhance focus on target regions by embedding textual semantics into the visual feature extraction process. Li et al. [45] proposed a Query-Guided Visual Attention (QGVA) module for the visual encoder, which dynamically focuses on language-described regions by injecting textual semantics into the visual encoding process. This method achieved a 4.98% accuracy improvement over MGVLF on the DIOR-RSVG dataset. LPVA [20] innovatively adopted a channel-spatial dual-dimensional dynamic weight adjustment strategy, combined with a Multilevel Feature Enhancement (MFE) decoder to suppress background interference and enhance feature distinctiveness. It achieved an accuracy of 82.27% on DIOR-RSVG, though its capability remains limited to one-object localization and it lacks proficiency in multi-target matching. For scenarios involving lengthy and complex linguistic expressions, RINet [42] introduced a word contribution learner to evaluate the importance of each word in the language description. Through iterative fine-tuning, it improved the comprehension of intricate linguistic information (Figure 6 illustrates the workflow of RINet). In summary, this method leverages attention mechanisms and domain-specific training data to enhance the robustness of complex description parsing and guide visual optimization.

4.3. Annotation Scarcity

Specialized datasets for RSVG are limited by domain knowledge and scarce data. Commonly used RSVG datasets, typically derived from annotated DIOR object detection benchmarks, contain at most 50,000 instances, significantly fewer than natural scene counterparts like RefCOCO (142,209 referring expressions for 50,000 objects). On one hand, the professional interpretation requirements for multi-modal remote sensing imagery pose a significant challenge to annotation quality. For example, SAR targets exhibit varying signatures due to incidence angles and polarization modes, requiring annotators to possess electromagnetic scattering knowledge and cross-modal interpretation skills, thereby escalating annotation costs. On the other hand, data access restrictions further exacerbate annotation scarcity: publicly available remote sensing data sources are primarily concentrated in civilian scenarios, while military or sensitive targets remain inaccessible due to security protocols, limiting dataset diversity and scenario coverage. This dual scarcity undermines model generalization for complex scenes or novel targets. Consequently, researchers explore open-set methods to expand candidate categories. Hu et al. adapted Grounding DINO to RSVG via a Multi-scale Image-to-Text Fusion Module (MSITFM) and Text Confidence Matching (TCM), optimizing cross-modal interaction and label assignment for robust open-world localization. Multimodal foundation models address scarcity by integrating multisensor data into unified instruction frameworks. EarthDial constructed an 11.11 million-pair multi-modal instruction dataset, leveraging complementary cross-modal information from RGB, SAR, near-infrared, etc., to drive the model’s learning of universal visual–language mapping relationships and reduce reliance on single-modal annotations. GeoGround unifies HBB, OBB, and mask tasks through prompt-assisted and geometry-guided learning, converting annotations into reusable instruction–response pairs, as shown in Figure 7. While multitask interference remains a concern, these approaches establish new paradigms for mitigating RSVG data limitations. To more comprehensively analyze whether each RSVG model has been improved to address the three typical challenges of scale heterogeneity, semantic complexity, and label scarcity,

Table 5. Comparative analysis of innovations across RSVG methods.

Methods	S.H.	S.C.	A.S.	Innovations
GeoVG [11]	√	√	×	The numeric context module represents complex expressions as geospatial relation graphs. The adaptive region attention module extracts key visual content.
MGVLF [12]	√	×	×	Multi-scale visual features and multi-granularity text embeddings are utilized to learn more discriminative representations. Irrelevant noise is adaptively filtered, and salient features are enhanced.
VSMR [44]	√	√	×	The multimodal enhancer and adaptive feature selection module focuses visual feature attention on language-related regions. The multistage decoder (MSD) reduces ambiguity in reasoning by continuously considering visual and language information and performing iterative queries.
LQVG [19]	√	√	×	Sentence-level text features are utilized as language query features for target retrieval. The MSCMA module enhances semantic relevance.
QAMFN [45]	√	√	×	The QGVA mechanism enhances visual features. A text-semantic attention-guided masking (TAM) module filters redundant information.
MSVG [46]	√	√	×	The MTAM enhances the correlation between visual features and text descriptions. A visual enhancement fusion module (VEFM) strengthens feature relevance through contextual information. Multistage decoding achieves final feature fusion and visual grounding.
LPVA [20]	√	√	×	A progressive attention (PA) module dynamically generates multi-scale weights and biases to enable the visual backbone to gradually focus on features related to language expressions. The MFE decoder aggregates visual contextual information to enhance the distinctiveness of target object features.
FQRNet [55]	√	√	×	A spatial–frequency fusion strategy based on language query refinement addresses challenges of scale variation and blurred boundaries. A frequency-guided spatial (FGS) module enhances spatial representation using spectral features. A query-aware original attention (QOA) mechanism enables deep multimodal fusion.
MSANet [39]	√	×	×	The MSAM aggregates multi-scale contextual information through a stacking strategy. A generative paradigm is introduced to directly generate discrete coordinate sequences, enhancing interaction between the regression process and encoded features.
CrossVG [43]	√	√	×	A cross-modal guidance encoder (CMGE) uses visual features to guide multi-granularity text embeddings. A cross-modal decoder explores word-level attributes to improve target recognition accuracy.
APMOR [40]	×	√	×	A learnable attribute prompter dynamically explores rich attribute information in remote sensing images. An attribute-prompting multimodal fusion encoder establishes fine-grained interaction between visual and language features. A multimodal progressive object reasoning decoder gradually queries more comprehensive object features.
TACMT [23]	√	√	×	A text-aware query selection module optimizes decoder queries. A cross-scale fusion module handles features of different scales.
RINet [42]	√	√	×	A local-to-object strategy is adopted to locate target regions via a regional indication generator. A word contribution learner evaluates the importance of each word in language expressions. A multi-round fine-tuning process fully utilizes complex language information.
Eff-Gounding DINO [41]	√	√	√	A multi-scale image-to-text fusion module (MSITFM) updates text features via self-attention and uses scale-specific cross-attention for multi-scale visual feature fusion to reduce learning complexity. A text confidence matching (TCM) mechanism introduces IoU-based confidence in label assignment to reduce mismatches.
VGRSS [29]	√	√	×	The Language-Guided Visual Feature Enhancement (LVFE) module enhances visual features through text guidance before feature fusion to address the problem of insufficient utilization of text information. The Visual–Language Fusion (VLF) module preserves spatial information through non-compressive stacking fusion and residual mechanisms. The EIoU loss function is introduced into bounding box regression, and geometric constraints are utilized to improve the convergence accuracy of the model, which is especially suitable for the multi-scale characteristics of ship targets.
CSDNet [47]	√	√	×	The Text-aware Fusion Module (TFM) modulates visual features using textual cues aggregated from image context to reduce target feature confusion. The Context-Enhanced Interaction Module (CIM) harmonizes the differences between visual and textual features by modeling multimodal contexts. The Text-Guided Sparse Decoder (TSD) addresses the issue of surface information redundancy.
GeoChat [14]	√		√	Supports image-level and region-level conversations.
SkyEyeGPT [48]	√		√	A high-quality RS instruction fine-tuning dataset with 968,000 instances enhances instruction fine-tuning of different granularities via a two-stage adjustment method.
EarthGPT [49]	√	√	√	The visual-enhanced perception mechanism refines and integrates coarse-scale semantic and detailed perceptual information. The cross-modal mutual comprehension method enhances interactions between visual perception and language understanding to deepen multimodal comprehension. Optimization is conducted using the unified instruction-following dataset MMRS-1M.
SkySenseGPT [24]	√	×	√	Fine-grained instruction tuning with the high-quality FIT-RS dataset significantly improves the complex scene.comprehension ability of remote sensing multimodal models.
LHRS-Bot [50]	√	×	√	The large-scale weakly-labeled LHRS-Align dataset trains the visual perception module in the pretraining stage, followed by multi-task and instruction fine-tuning.
VHM [51]	√	×	√	The large-scale high-quality VersaD dataset has detailed in-context examples, coupled with a fine-grained prompt framework and quality inspection mechanism.
GeoPix [52]	√	√	√	The class-wise learnable memory (CLM) module stores and retrieves intra-class shared geographic context to enhance model understanding of diverse instances in complex RS scenes. The two-stage training strategy mitigates conflicts between generation and segmentation tasks.
EarthDial [53]	√	√	√	The adaptive high-resolution module meets the requirements of high-resolution RS imagery. The data fusion module processes multi-band or multi-temporal data streams. The three-stage training strategy integrates RGB pretraining, temporal fine-tuning, and multi-band optimization.
GeoGround [15]	√	×	√	The text-mask paradigm compresses mask information into compact text sequences for efficient learning by VLMs. Hybrid supervision integrates PAL and GGL to fine-tune models using three types of signals.

S.H.: Scale Heterogeneity; S.C.: Semantic Complexity; A.S.: Annotation Scarcity. √ indicates that the model has been improved for the corresponding issue, while × indicates no improvement.

provides a systematic summary of the relevant content and clearly identifies the innovative features of each model.

5. Challenges and Outlook

Remote sensing visual grounding has achieved notable progress, yet critical bottlenecks persist in multisource data fusion, dynamic scene understanding, and edge computing adaptability. Future research needs to focus on the following directions to carry out an in-depth exploration:

5.1. Intelligent Annotation Agent for Multisource Heterogeneous Data

Current RSVG datasets predominantly rely on optical imagery, resulting in weak generalization across modalities such as SAR, hyperspectral, multispectral, and LiDAR point clouds. This limitation stems from the differences in physical properties among multiple data sources. For example, SAR images are imaged by an electromagnetic wave scattering mechanism, and their texture features have no intuitive correspondence with the RGB channels; hyperspectral data contains hundreds of bands with high spectral redundancy, which will lead to dimensional disaster when directly inputted into the model; and the laser point cloud can accurately characterize the three-dimensional geometric structure, but it lacks semantic labels and has significant alignment errors in a large scale. Traditional manual annotation methods require specialized knowledge and consume a lot of time, and cross-modal graphical alignment relies on empirical rule design, which makes it difficult to support the training needs of open-set models. Thus, a knowledge-driven [56] annotation agent can be designed to integrate geographic knowledge mapping and a domain macrolanguage model, to embed professional terms into the text description generation process; to synthesize multi-source data-text pairs using the Diffusion [57] model to realize the scale of the training set to scale up, to satisfy the training demand of multi-modal large models, and to explore the potential of the model in visual grounding tasks.

5.2. Cross-Temporal Perception Modeling for Dynamic Scenarios

Existing remote sensing visual grounding methods mainly focus on static one-temporal-phase data, with which is difficult to capture dynamic processes such as flood inundation and urban sprawl, resulting in insufficient model temporal correlation capability. This is mainly due to the fact that multi-temporal phase remote sensing data often have significant heterogeneity, coupled with the lack of quantitative representation models for temporal semantics in natural language. Although TEOChat [58] tries to construct a temporal visual language large model for resolving the temporal evolution law, it fails to establish an accurate mapping between spatiotemporal coordinates and textual commands. Future work could adopt memory-augmented networks with deformable temporal attention mechanisms to dynamically correlate historical feature maps with current observation data. Multitemporal image–text pairs with temporal adverbs could enhance cross-temporal tracking, enabling autonomous change trajectory identification and disaster spread prediction, ultimately achieving a “change perception–precise localization–decision feedback” loop.

5.3. Edge Computing-Oriented Lightweight Deployment

High-resolution remote sensing image processing depends on large-parameter models to create effective representations. However, satellite and airborne platforms face multiple constraints, including limited computing resources, radiation-hardened requirements, and environmental adaptability, which severely restrict computational and storage capabilities. This makes supporting in-orbit training and real-time inference of complex models difficult. Research shows that even on specialized embedded AI hardware like the Jetson series or Movidius Myriad 2, inference times for unoptimized models are still too slow to meet the high-timeliness needs of tasks like disaster response. Although techniques like model quantization can reduce computational load, they often cause significant accuracy drops (mAP). This limits their usefulness in scenarios with strict real-time demands, such as military reconnaissance and disaster monitoring. Future research must take a dual approach: first, systematically compress models, reduce computational complexity, and improve inference efficiency and accuracy retention on edge platforms by integrating optimization methods like lightweight network design [59], model pruning [60], knowledge distillation [61], and low-bit-width quantization [62]; second, develop a federated edge learning framework [63] based on low-orbit satellite constellations. Through multi-node collaboration and knowledge sharing, this approach aims to overcome the data sample and model generalization bottlenecks of single satellites, gradually building distributed core models. Only in this way can we bridge the gap between laboratory models and practical deployment, enabling a real-time perception-decision loop of “what you see is what you get” in extreme, dynamic scenarios like battlefield perception and rapid disaster area assessment.

6. Conclusions

In this review, we systematically track and summarize the research progress and future path of remote sensing visual grounding technology in the last five years. First, we establish the basic concept of RSVG by comparing the scope boundaries of object detection, referring image segmentation, compiling the mainstream datasets such as DIOR-RSVG and RSVG-HR, and evaluation metrics. Subsequently, two major approaches to remote sensing visual grounding, namely, the Transformer-based methods and the multimodal large language model-based methods, are introduced in detail and analyzed comparatively. In addition, we discuss in depth the innovative techniques of remote sensing visual grounding based on the key features in the remote sensing images. Finally, we summarize the challenges faced by the research on remote sensing visual grounding and propose valuable directions for future research. This paper is suitable for both beginners and experienced researchers in the field of remote sensing visual grounding, and serves as a valuable resource for tracking the latest research progress.