SAREval: A Multi-Dimensional and Multi-Task Benchmark for Evaluating Visual Language Models on SAR Image Understanding

Highlights

What are the main findings?

• We establish SAREval, the first comprehensive benchmark specifically designed for evaluating Vision-Language Models on SAR image understanding, comprising 20 tasks across perception, reasoning, and robustness dimensions with 14,950 expert-verified image–text pairs.
• Current VLMs demonstrate significant limitations in SAR image interpretation, particularly in fine-grained target discrimination and physical-attribute reasoning tasks, while exhibiting unexpected performance improvements under certain noise conditions that challenge conventional robustness understanding.

What are the implications of the main findings?

• The benchmark provides a standardized platform for developing and evaluating SAR-specific VLMs, addressing the critical gap in specialized evaluation frameworks for radar remote sensing and enabling fair comparison across different model architectures.
• Our findings reveal fundamental domain adaptation challenges in VLMs and establish a new paradigm for constructing multimodal benchmarks in specialized remote sensing domains, with direct implications for maritime surveillance, infrastructure monitoring, and environmental observation applications.

Abstract

Vision-Language Models (VLMs) demonstrate significant potential for remote sensing interpretation through multimodal fusion and semantic representation of imagery. However, their adaptation to Synthetic Aperture Radar (SAR) remains challenging due to fundamental differences in imaging mechanisms and physical properties compared to optical remote sensing. SAREval, the first comprehensive benchmark specifically designed for SAR image understanding, incorporates SAR-specific characteristics, including scattering mechanisms and polarization features, through a hierarchical framework spanning perception, reasoning, and robustness capabilities. It encompasses 20 tasks from image classification to physical-attribute inference with over 10,000 high-quality image–text pairs. Extensive experiments conducted on 11 mainstream VLMs reveal substantial limitations in SAR image interpretation. Models achieve merely 25.35% accuracy in fine-grained ship classification tasks and demonstrate significant difficulties in establishing mappings between visual features and physical parameters. Furthermore, certain models exhibit unexpected performance improvements under certain noise conditions that challenge conventional robustness understanding. SAREval establishes an essential foundation for developing and evaluating VLMs in SAR image interpretation, providing standardized assessment protocols and quality-controlled annotations for cross-modal remote sensing research.

1. Introduction

Intelligent algorithms have recently achieved extensive progress in remote sensing image interpretation. Dedicated models tailored for specific tasks, based on Convolutional Neural Networks (CNNs) [1], Recurrent Neural Networks (RNNs) [2], and self-attention mechanisms [3], have attained high accuracy and efficiency across a range of perception tasks, including aircraft detection [4], ship detection [5], and sea–land segmentation [6]. However, such models are typically designed for unimodal, single-task visual processing, lacking the capability to understand natural language instructions. Consequently, they struggle to meet the demands of complex tasks such as open-domain Visual Question Answering (VQA) [7], fine-grained semantic description [8], and Referring Expression Comprehension (REC) [9]. In this context, Vision-Language Models (VLMs) [10], a class of multimodal intelligent models that fuse visual features from imagery with semantic information from natural language, have ushered in a transformative advance for remote sensing interpretation. Distinct from traditional dedicated models, VLMs can simultaneously process both image and text information, effectively bridging the semantic gap between imagery and language. This enables the deep integration of visual perception and semantic reasoning, reduces reliance on expert manual interpretation, and offers a more flexible and intelligent interpretation paradigm for applications in Earth observation, environmental monitoring, disaster prevention and mitigation, and land use planning. Ultimately, VLMs facilitate the transition of remote sensing data from being merely “accessible” to becoming “understandable and actionable.”

To address early cross-modal generation needs, initial research typically employed separate encoders to extract visual and textual features independently, coupled with specially designed fusion modules for feature interaction. This approach enabled basic tasks such as image captioning and visual grounding, thereby validating the feasibility of VLMs in remote sensing. Within these models, CNNs [11] and Fully Convolutional Networks (FCNs) [12] were commonly used to extract spatial visual features, while RNNs and their variants, such as Long Short-Term Memory (LSTM) [13] networks, were responsible for encoding textual sequence information. For instance, in remote sensing image captioning, Shi and Zou [12] introduced FCNs to perform multi-scale ground object semantic extraction and decomposed the captioning process into subtasks like instance detection and environmental analysis. Zhang et al. [14] designed a visual aligning loss function to explicitly constrain the similarity between attention weights and word vector spaces, thereby enhancing modal alignment. In visual grounding, the RSVG [15] combined a language encoder to construct geospatial relation graphs, an adaptive region attention visual encoder, and a fusion module, achieving an accuracy of 59.4% at IoU@0.5. After end-to-end optimization on specific tasks and datasets, such as UCM-caption [16] and RSVQA [17], these models can achieve competitive performance metrics. Nevertheless, these models were generally designed for a single task, requiring independent training and maintenance of a dedicated set of parameters for each task. Their capabilities are relatively closed, making it difficult to transfer to new scenarios or novel object categories and fundamentally lacking the flexibility to respond to open-domain natural language instructions. Furthermore, their performance is highly dependent on training data, resulting in limited generalization ability.

The success of VLMs pre-trained on large-scale image–text pairs in the natural image domain has paved the way for a unified visual-language modeling paradigm. Models such as Contrastive Language–Image Pre-training (CLIP) [18] and Align Before Fuse (ALBEF) [19] align visual and linguistic modalities through contrastive learning, providing a powerful representational foundation for downstream tasks. Subsequently, Multimodal Large Language Models (MLLMs) represented by Large Language and Vision Assistant (LLaVA) [20] and BLIP-2 [21] have further integrated visual encoders with large language models, demonstrating remarkable zero-shot generalization and reasoning capabilities, thereby extending model functionality from single-task processing to multi-dimensional tasks. Advances in efficient fine-tuning techniques and the emergence of high-quality remote sensing image–text training datasets have created opportunities for applying VLMs in the vertical domain of remote sensing. Optical remote sensing, being the most widely used and data-rich modality, has taken the lead in exploring and deploying remote sensing VLMs. For example, GeoChat [22], building upon the LLaVA framework and infused with remote sensing domain knowledge, supports multi-task dialogue and zero-shot understanding of high-resolution optical imagery, establishing itself as the first unified multimodal large model for multiple remote sensing tasks. EarthMarker [23] introduced visual prompt learning to enable multi-granularity interpretation of optical imagery at the image, region, and point levels. SkySenseGPT [24] proposed a large-scale instruction-tuning dataset, FIT-RS, covering complex comprehension tasks from relation understanding to scene graph generation, and constructed a model excelling in multiple tasks through instruction tuning. EarthDial [25] extended support to multi-sensor, multi-temporal, and multi-resolution remote sensing data, achieving broader task coverage and stronger generalization. TEOChat [26] designed spatiotemporal reasoning modules for temporal optical data, expanding model applications to spatiotemporal reasoning tasks such as change detection and disaster damage assessment. Remote sensing VLMs are typically constructed upon powerful visual encoders and language decoders, and are aligned through supervised fine-tuning or reinforcement learning using large-scale, high-quality, and task-diverse remote sensing instruction datasets. This enables a single model to respond to open-ended natural language instructions and adapt to multiple downstream tasks—such as scene classification, VQA, image captioning, visual grounding, object detection, and change analysis—in a zero-shot or few-shot manner. Consequently, these models significantly enhance generality, interactivity, and knowledge transferability, marking a paradigm shift toward unified, semantically aware remote sensing intelligence.

In contrast, the development of SAR remote sensing VLMs remains in its early stages. Although SAR offers unique advantages in cloudy and rainy regions, nighttime observation, and military reconnaissance due to its all-weather imaging capability and sensitivity to surface dielectric properties, its coherent imaging mechanism introduces speckle noise and geometric distortions [27], resulting in significant visual differences between SAR and optical imagery. As a result, traditional VLMs designed for optical features are not directly applicable to SAR data. Until recently, only a few studies have explored the integration of SAR imagery with VLMs. TACMT [28] investigated natural language-guided target detection in SAR images for visual grounding, but it was limited to a single category of objects, with limited generalization ability. EarthGPT [29] adapted VLMs using the multimodal instruction-tuning dataset MMRS-1M, achieving preliminary extension beyond optical data. However, SAR target samples constitute only a small portion of the dataset, and the covered tasks are basic, making it insufficient for training and evaluating models on complex SAR-specific tasks.

Compared to gradual technological progress, the development of SAR-oriented evaluation benchmarks has lagged significantly, severely hindering objective assessment and equitable exploration of VLM applicability to SAR data. As shown in

Table 1. Comparison of existing benchmark datasets with SAREval.

Dataset	SAR	Robustness Test	Answer Type	Dimension	Annotation Method
RSIEval [30]	×	×	FF	6	M
LHRS-Bench [34]	×	×	MCQ	11	M
FIT-RSFG [24]	×	×	MCQ, BBox, FF	11	A + M
GeoChat-Bench [22]	×	×	FF, BBox	6	A + M
SARChat-Bench [33]	√	×	BBox, FF	6	A + M
VRSBench [31]	×	×	BBox, FF	3	A + M
XLRS-Bench [35]	×	×	MCQ, BBox, FF	16	A + M
CHOICE [36]	√	×	MCQ, BBox, Seg	23	A + M
VLEO-Bench [37]	×	×	MCQ, BBox, FF	6	A + M
GeoBench-VLM [32]	√	×	MCQ, BBox, Seg	31	A + M
REOBench [38]	×	√	BBox, Seg, FF	6	A + M
SAREval	√	√	MCQ, BBox, Seg, FF	20	A + M

Abbreviations adopted: MCQ for multiple-choice questions, FF for free form, BBox for bounding box, and Seg for segmentation masks; in annotation formats, M for manual annotation, and A denotes automatic annotation.

, existing remote sensing benchmarks are predominantly designed for optical imagery, with established benchmarks such as RSI-Eval [30], FIT-RSFG [24], and VRSBench [31] focusing exclusively on optical data. Although some benchmarks, such as GeoBench-VLM [32], have attempted to include limited SAR data, they only support restricted tasks such as land use classification and object counting, and fail to incorporate evaluation criteria tailored to core SAR characteristics such as microwave scattering mechanisms and geometric distortion. SARChat-Bench [33], as the first dedicated SAR visual-language benchmark, partially addresses the scarcity of SAR data in terms of quantity, but its framework lacks diversity and comprehensiveness. Overall, existing SAR-oriented benchmarks exhibit notable shortcomings, largely due to three major technical challenges:

• Limited Support for SAR-VLM Alignment Needs: Existing benchmarks do not adequately meet the alignment requirements for SAR-VLM evaluation. For instance, they lack image–text pairs that reflect SAR-specific characteristics such as microwave scattering mechanisms (e.g., metallic targets appearing as bright spots due to strong reflection, while water bodies appear dark due to weak reflection). Textual descriptions also often fail to balance technical accuracy with semantic readability, which makes conventional optical image–text alignment paradigms unsuitable for SAR data.
• High Cost of Professional SAR Annotation: Interpreting SAR imagery requires deep domain expertise. Manual annotation involves not only delineating object boundaries but also providing specialized information such as scattering mechanisms and imaging parameters, leading to low annotation efficiency and difficulty in maintaining consistency.
• Lack of SAR-Adapted Task Paradigms: General-purpose benchmarks designed for optical imagery cannot fully address the interpretation needs of the SAR domain. Meanwhile, SAR-specific tasks—such as polarization mode inference, speckle noise robustness, and geometric distortion correction—have not yet established mature evaluation paradigms or quantifiable metrics, further hindering the development of SAR benchmarks.

To address these challenges, this paper introduces SAREval, a multimodal benchmark for SAR imagery. The main contributions are as follows:

• We construct a multi-level evaluation benchmark centered on perception, reasoning and robustness dimensions, deeply integrating SAR-specific characteristics. Unlike existing benchmarks that focus primarily on optical data or include only limited SAR tasks, SAREval is designed specifically for SAR data, with tasks closely aligned with SAR imaging mechanisms and scattering characteristics. It covers 20 specialized tasks ranging from image classification and object detection to physical interpretation and sensor parameter inversion. This offers a standardized platform for fine-grained assessment of model adaptability to SAR’s unique properties.
• We propose a task-adaptive hybrid annotation framework and introduce an innovative distractor generation strategy based on “physical constraints + gradient divergence.” To tackle the challenges of SAR image–text alignment and high annotation cost, we combine a label-driven method, LLM-assisted method, and expert verification into a unified data generation pipeline. Specifically, for numerical reasoning tasks, we design a three-tier distractor system based on physical constraints and gradient divergence, ensuring that distractors are both physically plausible and capable of discriminating between different levels of reasoning ability. Under this framework, we produce over ten thousand high-quality test samples across diverse scenarios, balancing professionalism and scale.
• We conduct systematic benchmarking of multiple state-of-the-art VLMs and provide critical performance insights and analysis. We comprehensively evaluate 11 mainstream models on SAREval, quantifying their performance differences across various tasks and deeply analyzing their limitations in specialized tasks such as geometric measurement and physical state reasoning. This offers empirical evidence to guide future development and optimization of VLMs for SAR data.

2. Related Works

2.1. Remote Sensing Multimodal Benchmarks

The advancement of remote sensing Visual Language Models has brought the objective and fair evaluation of model performance to the forefront of scholarly attention, necessitating the design and development of specialized evaluation benchmarks. Early remote sensing VLM benchmarks primarily focused on assessing single tasks or foundational capabilities. While they laid the groundwork for domain evaluation, they suffered from significant limitations, including incomplete task coverage, limited data scale, and modality specificity. RSI-Eval [30], as the first benchmark in this field, innovatively designed quantitative evaluation metrics for remote sensing scenarios, providing an assessment basis for fundamental generative tasks like image captioning and Visual Question Answering. However, its evaluation scope was confined to these basic tasks and did not extend to complex capabilities such as object localization and semantic reasoning. Its relatively small data scale also limited its ability to comprehensively characterize overall model performance. LHRS-Bench [34] enriched the granularity of foundational task evaluation by constructing multi-dimensional Q&A samples from limited imagery, improving the discriminative power of assessments. FIT-RSFG [24] focused on understanding object semantic relationships in complex scenes, enhancing the evaluation of precise alignment between visual features and textual semantics in optical imagery, an approach aligned with the contemporaneous FIT-RSRC for object relation understanding in remote sensing. However, its data relies entirely on optical imagery and does not incorporate SAR or other remote sensing modalities with distinct imaging mechanisms, resulting in significant limitations in its evaluative scenarios. Although VRSBench [31] significantly increased data scale and expanded the diversity of basic tasks, its task design remains confined to image captioning and dialogue, and it lacks coverage of SAR modalities, thus failing to support cross-modal evaluation needs. GEOBench-VLM [32] attempted to break the single-modality constraint by including SAR imagery in its evaluation system. However, SAR samples constitute less than 20% of its dataset and only support basic tasks like land use classification and ship object counting. It cannot test models’ understanding of SAR-specific microwave scattering mechanisms or their interpretation capabilities under unique scenarios involving geometric distortions. CHOICE [36], although establishing a hierarchical evaluation system, suffers from insufficient pixel-level task assessment, a limited sample size for advanced reasoning tasks, and low compatibility with visual grounding, which restricts its evaluation scope. SARChat-Bench [33], as the first dedicated image–text benchmark dataset for SAR imagery, constructed 2 million annotated samples across six typical tasks, representing a preliminary exploration of SAR-specific benchmarks. However, it emphasizes dialogue generation tasks, features a monolithic benchmark evaluation architecture, and lacks completeness in its task system design. Notably, the vast majority of existing benchmarks focus solely on model performance under ideal conditions, overlooking the pervasive issues of data perturbation and quality degradation common in real-world application scenarios. Although REOBench [38] provides valuable insights for robustness evaluation in optical remote sensing, overall, current benchmarks still exhibit significant limitations requiring breakthroughs, particularly in terms of modality coverage completeness, task type diversity, and adaptability to the unique technical challenges posed by SAR modalities.

2.2. Strategies for Constructing Visual Language Benchmark Data

Visual language benchmark data typically appears in the form of question–answer pairs, encompassing both image and text modalities. The construction of multimodal benchmarks relies on high-quality image–text matching and systematic task design. Current strategies for building these benchmark datasets primarily revolve around three paradigms: meticulous manual annotation, automated rule-based generation, and large model-assisted generation. Early datasets like RSIEval [30] and LHRS-Bench [34] were constructed using manual annotation, involving domain experts to label or meticulously curate data to ensure accuracy and objectivity. However, this approach incurs high temporal and economic costs, making it difficult to scale to very large dataset construction. This is particularly challenging for highly specialized remote sensing modalities like SAR, where annotators need not only a solid foundation in radar remote sensing theory but also an understanding of physical principles like microwave scattering mechanisms and polarization characteristics. The scarcity of such expert resources further exacerbates the difficulty of data acquisition, rendering purely manual annotation inadequate for large-scale benchmark construction. Automated rule-based generation strategies create images or question–answer pairs in bulk using predefined algorithms or rules in a structured manner. CHOICE [36] generates answers from predefined labels and creates distractors based on labels from other samples. GeoBench-VLM [32] converts detection data into questions about specific object counts in counting tasks, providing rules for generating correct counts and alternative options with controlled deviations to automatically produce Multiple-Choice Question (MCQ) options. This strategy significantly improves annotation efficiency and generates data with a uniform format and strong controllability. It is particularly suitable for building benchmarks that require specific distributions or aim to test particular capabilities, enabling the generation of large-scale, standardized data samples. Its limitation lies in the heavy reliance on deep domain knowledge for rule design, and the generated data may lack diversity and semantic complexity. For SAR-specific physical phenomena like speckle noise, geometric distortion, and polarization sensitivity, rules traditionally designed for optical imagery are difficult to transfer directly, requiring the reconstruction of a generation framework adapted to SAR imaging mechanisms, which places higher demands on the expertise of rule designers. LLM-assisted generation is currently the mainstream paradigm for efficient large-scale dataset construction. VRSBench [31] uses attribute information extracted from detection labels to design prompts that guide GPT-4 in generating annotations. This strategy leverages the inherent capabilities of large models to generate data and is the predominant method for efficient, large-scale dataset creation today. Subsequent datasets like SARChat-Bench [33] and XLRS-Bench [35] have all utilized LLM to generate multi-turn dialogues and complex instructions, effectively expanding dataset scale and task coverage, and greatly enhancing the efficiency and diversity of data production. However, the quality of the generated content depends on the capability of the base model used, and it carries the unavoidable risk of hallucination and potential error propagation. Therefore, it is often combined with human verification to ensure data quality. In summary, balancing generality and specialization, scale and cost, and constructing a hybrid data construction framework that deeply integrates domain knowledge while maintaining both scale and quality represents a key challenge for advancing the development of SAR visual language models.

3. SAREval

3.1. Hierarchical Capability Taxonomy

As illustrated in Figure 1, SAREval is structured around three core capability dimensions, comprising a total of 20 fine-grained tasks. This architecture is designed to comprehensively evaluate the overall performance of Vision-Language Models in interpreting SAR imagery. A detailed description of each dimension and its constituent tasks is provided below.

3.1.1. Perception Capability Dimension

Perceptual capability forms the foundation of remote sensing image interpretation. It requires models to identify various ground objects within an image, distinguish their physical properties, and provide support for harder reasoning tasks. Since SAR imagery, based on coherent imaging principles, lacks intuitive visual features such as color and texture found in optical imagery, object discrimination relies heavily on microwave scattering characteristics and scene structure. To systematically and comprehensively evaluate the perceptual capabilities of VLMs on SAR data, this dataset constructs evaluation tasks across three levels: Image-level Understanding, Object-level Recognition, and Pixel-level Segmentation. Each level is closely aligned with typical SAR remote sensing applications, such as ocean monitoring and land use mapping. To intuitively illustrate the design of each subtask in the Perception dimension, Figure 2 presents visualization examples of key tasks across Image-level Understanding, Object-level Recognition, and Pixel-level Segmentation.

Image-level Understanding: These tasks focus on the model’s ability to grasp the overall theme and central content of an entire SAR image. SAREval selects natural environments (e.g., farmland, forest, and water bodies) and man-made environments (e.g., airports, ports, refineries, urban areas), covering the most critical application scenarios for SAR remote sensing, thereby providing a comprehensive assessment of a model’s practical potential. Representative tasks include Typical Scene Classification and Image Description: (1) Scene Classification: This task focuses on coarse-grained theme identification, requiring the model to accurately distinguish core categories of natural versus man-made environments. It simulates the practical need for rapidly screening imagery for specific scenes, testing the model’s ability to capture overall scene characteristics. This provides technical support for the preliminary classification and filtering of large-scale remote sensing datasets. (2) Image Description: This task requires the model to generate natural language text describing the scene type, key objects, and their spatial relationships. Since SAR imagery lacks color information, descriptions must incorporate scattering characteristics and spatial layout. This tests the model’s ability in multimodal semantic fusion and professional expression, aiding non-experts in quickly understanding SAR image content.

Object-level Recognition: These tasks focus on the recognition, counting, and localization of high-value ground objects, such as aircraft, ships, oil tanks, bridges and vehicles in SAR imagery, which are often central to scenarios like maritime surveillance, military reconnaissance, and infrastructure monitoring. They evaluate the model’s progressive capability from basic recognition to complex semantic localization, meeting fine-grained information needs across different scenarios. (1) Object Detection: Beyond foundational multi-category detection, this task emphasizes fine-grained classification of three high-value targets, including vessels, aircraft, and vehicles. For vessels, it accommodates varying difficulty levels: a moderate-difficulty task defines 14 relatively general categories, while a high-difficulty task refines these into 81 more discriminative fine-grained categories. Aircraft detection covers 13 common categories. Vehicles are meticulously divided into 22 major and 50 minor subcategories. This design thoroughly and precisely validates the model’s ability to distinguish between object categories at different levels of granularity. (2) Object Counting: Building upon detection, this task requires precise quantification of target numbers. It primarily assesses the model’s recognition stability and precision in scenes with dense targets or complex backgrounds. The absence of color and texture features in SAR imagery makes small targets prone to being missed and clutter prone to false detection, placing higher demands on the quantitative accuracy of counting results. (3) Visual Grounding: Differing from traditional object detection, this task requires the model to process complex natural language queries in an open-vocabulary context. It must not only identify predefined categories but also localize targets and judge attributes based on natural language descriptions. This tests the model’s multimodal semantic fusion and implicit constraint resolution capabilities, closely aligning with real-world scenarios where users query specific targets using natural language.

Pixel-level Segmentation: These tasks demand a higher level of fine-grained understanding from the model. SAR imagery covers areas ranging from square-kilometer-scale urban agglomerations to sub-meter individual objects, with small targets like ships and vehicles often coexisting with large-scale features like airports and ports in the same scene. Such tasks require the model to generate pixel-level masks for precise contour extraction, testing its ability to handle challenges like edge blurring and object adhesion in fine-grained segmentation. (1) Land use Segmentation: This task requires the model to perform precise pixel-level segmentation for eight land cover categories—agricultural land, bare land, buildings, developed land, grassland, roads, forest, and water bodies—using pixel-level masks. The model must ensure the fineness of object boundary extraction, providing detailed spatial distribution information for precise land use planning, management, and ecological environment monitoring. (2) Marine Oil Spill Detection: This task directly addresses environmental monitoring and disaster emergency response scenarios. It requires the model to accurately identify areas of changed sea surface roughness caused by oil spills in SAR imagery. Typically, when an oil spill occurs, an oil film forms on the sea surface, significantly reducing its backscatter intensity and creating dark areas in the image that contrast markedly with the surrounding water. Identifying and extracting oil spill areas provides critical support for marine pollution emergency response, oil spill source identification, and cleanup operation planning, effectively mitigating losses from marine pollution.

3.1.2. Reasoning Capability Dimension

Reasoning capability represents the core advanced skill of VLMs, building upon the foundational perception of SAR imagery. Interpreting SAR data requires not only visual features but also integration of specialized knowledge, such as microwave scattering physics and the geometric distortion principles of side-looking imaging, to deduce physical essence and quantitative parameters from image phenomena, thereby providing decision support for quantitative remote sensing applications. This dimension designs three categories of progressive tasks to systematically test the model’s deep reasoning and analytical abilities, as shown in Figure 3.

Object Attribute Inference: These tasks require the model to fully understand the microwave scattering characteristics of objects during the SAR imaging process, combining visual features and domain knowledge to infer the dynamic physical states and related attributes of targets. They have practical applications in maritime supervision and energy management. (1) Ship Motion Direction Estimation: A crucial requirement for maritime dynamic monitoring and safety control, this task evaluates the model’s ability to infer a target’s motion state based on the texture features of its wake and ocean dynamics principles. In SAR images, ship movement generates features like turbulent wakes and narrow-V wakes, which have definite physical relationships with the ship’s dynamics. Accurately inferring the ship’s motion direction enables direct application in various key scenarios, such as real-time vessel track monitoring in maritime traffic management, identifying suspicious vessels deviating from routes, judging illegal fishing in restricted zones combined with current data, and predicting future positions for search and rescue or oil spill dispersion warnings. (2) Oil Storage Tank Volume Estimation: This task requires the model to leverage the dihedral corner reflector effect inherent in SAR imaging mechanisms. By estimating the height difference between the tank roof and the floating roof surface, it infers the oil storage volume, enabling timely detection of anomalies like oil theft or leakage. This provides crucial support for energy supply chain control and oil depot safety monitoring.

Imaging Parameter Estimation: These tasks aim to evaluate the model’s ability to invert key sensor imaging parameters from SAR imagery. Parameter estimation directly reflects SAR image quality and object scattering characteristics, holding significant importance for multi-source SAR data fusion, image quality assessment, and enhancement of specific targets. They test whether the model has developed a deep physical understanding of SAR imaging geometry and its interaction with ground objects. (1) Target Azimuth Inference: Utilizing a dataset built from SAR images of the same vehicle target at different azimuth angles under the same polarization, this task assesses whether the model can capture scattering differences caused by orientation changes, evaluating its ability to infer the imaging azimuth angle based on local target scattering characteristics. This can serve applications like dynamic target tracking. (2) Polarization Orientation Inference: This task is designed based on the differing reflection characteristics of objects under various polarization modes, facilitating the effective use of multi-polarization SAR data and enhanced identification of specific targets like metallic objects and vegetation. (3) Incidence Angle Inference: Useful for cross-scenario data fusion, this task critically assesses whether the model can move beyond superficial cognition of object brightness and infer the imaging geometry from the global intensity gradient of the image. Once the model accurately infers the incidence angle for different images, it can standardize the intensity values of images from different sensors based on the physical relationship between incidence angle and scattering intensity, thereby eliminating fusion interference caused by geometric differences. This provides a crucial geometric anchor for collaborative analysis of SAR data across time, platforms, and sensors. (4) Imaging Resolution Inference: This task requires the model to infer the resolution parameter based on imaging detail and dimensional accuracy, providing a fundamental basis for extended applications like imaging mission planning and super-resolution reconstruction.

3.1.3. Robustness Capability Dimension

The design of the robustness dimension originates from the inherent interference in real SAR imagery and the practical need for application robustness. Numerous interference factors in reality severely affect imaging quality, constraining model performance in perception and reasoning. Constructing capabilities in this dimension aim to comprehensively evaluate the stability of VLMs under non-ideal conditions, exposing model overfitting issues that are difficult to detect on clean data, and prompting models to perform stably and reliably in complex, challenging environments. It specifically focuses on in-depth investigation of two primary interference types, evaluating the model’s Robustness against Interference. Figure 4 provides comparative visualizations for the two core interference types in the Robustness dimension, clearly showing SAR image changes before and after interference addition and the corresponding task evaluation scenarios.

Interference caused by Coherent Mechanisms: Speckle noise, stemming from the unique coherent imaging mechanism of SAR systems, is an unavoidable intrinsic ailment of SAR imagery. When the coherent electromagnetic waves emitted by the radar illuminate a target, the scattered echoes from the target interfere with each other at the sensor. Because a resolution cell contains multiple scatterers, the phase of each scatterer’s echo varies randomly due to different distances from the sensor, causing the composite amplitude to fluctuate. This finally manifests as granular noise with alternating bright and dark spots in the image. Speckle noise obscures fine object textures, blurs target contours, and alters the image’s statistical distribution. Robustness evaluation based on speckle noise effectively tests the essential invariance of the model’s feature representations, helping to uncover VLMs with better generalization performance.

Interference caused by Background Clutter: Clutter noise originates from complex echoes generated by numerous random scatterers in the scene background. It is highly non-stationary and scene-dependent, exhibiting significant spatiotemporal dynamic characteristics. Unlike global speckle noise, clutter is a localized interference strongly correlated with the background environment. In maritime scenes, sea clutter intensity is influenced by sea state, wind speed, radar frequency, and polarization mode, exhibiting non-Gaussian distribution characteristics that change dynamically with waves, forming irregular textures of alternating brightness in the image. In terrestrial scenes, clutter characteristics are determined jointly by topography, object types, and radar frequency. For example, building walls and roofs in urban areas cause multipath scattering, easily generating strong clutter regions, while the periodic textures of flat farmland form regular clutter patterns. Clutter noise significantly reduces the local signal-to-clutter ratio, masks real targets, and increases the false alarm probability, posing major challenges for target detection and recognition tasks. Robustness evaluation based on clutter noise examines the model’s ability to discriminate object features and resist confusion.

3.2. Dataset Construction and Quality Control

The construction of SAREval is structured around the Perception, Reasoning, and Robustness dimensions. Leveraging multi-source open datasets and integrating commercial model assistance with expert manual verification, it establishes a comprehensive benchmark encompassing multiple formats, including multiple-choice questions, subjective open-ended questions, and objective label outputs. This provides a unified, fair, and comprehensive evaluation standard for large Vision-Language Models on SAR image understanding tasks, effectively facilitating performance comparison and capability assessment across different models and task types.

3.2.1. Data Source

The SAREval dataset is built upon publicly available datasets. For the diverse tasks under the three core capability dimensions, specialized SAR datasets with corresponding technical characteristics were selected or constructed to suit varying task requirements. The Perception dimension utilizes data sources such as MSAR-1.0 [39], SAR-Airport [40], and OpenEarthMap-SAR [41], covering typical scenes including harbors, airports, oil refineries, urban areas, and forests. It integrates object detection datasets to form a labeling system encompassing 13 aircraft categories, 14 major and 81 fine-grained ship categories, and 22 major and 50 fine-grained vehicle categories. Finely annotated datasets like OpenEarthMap-SAR [41] and Deep-SAR Oil Spill [30] are used to implement land use classification and marine oil spill detection tasks. The Reasoning dimension utilizes geospatial datasets (e.g., FAIR-CSAR [42] and BHdataset [43] ) that are stored in standard geoTIFF format with fine annotations and high bit-depth, compatible with GIS software such as ArcGIS. These alongside datasets with prominent feature attributes such as OpenSARWake [44] and ATRNet-STAR [45], to construct tasks for geometric measurement, attribute judgment, and imaging parameter inference. The Robustness dimension is constructed using image samples from the multi-class object detection task and simulated noise data introduced into image samples from the ship classification detection task. Notably, to prevent potential data leakage during model training, all data are exclusively drawn from the predefined test splits of the public source datasets, ensuring the objectivity and fairness of evaluation results. Specific data sources are detailed in Appendix A.1.

3.2.2. Task-Adaptive Hybrid Annotation Framework

To balance data scale, annotation efficiency, and professional accuracy requirements, the SAREval benchmark employs a hybrid annotation framework combining label-driven methods, manual annotation, and LLM-assisted approaches, as illustrated in Figure 5. Tailored to task characteristics, it generates question–answer pairs adapted for three formats: MCQs, free form, and objective label output.

Label-driven Structured Annotation Method: This method centers on the annotation metadata of SAR images. It generates standardized question–answer pairs through key field extraction and rule-based mapping, serving foundational tasks with definitive objective answers, such as scene classification, land use classification, object detection, and polarization mode determination. Objective labels typically serve as the ground truth in answers for MCQs and objective label output tasks. MCQs convert open-ended answers into predefined option sets, prompting the model to select the correct letter from the given choices. This format is suitable for tasks with a single, objectively correct answer. Objective Label Output tasks primarily visual grounding and pixel-level segmentation. Models are required to output the spatial location of targets within the image, either as rotated bounding box coordinates [x1, y1, x2, y2, x3, y3, x4, y4] or as masks.

In designing MCQs, the strategic construction of distractors is crucial for enhancing the assessment’s discriminative power. Accordingly, SAREval employs differentiated distractor generation strategies based on task characteristics. For non-numerical tasks like scene classification, object detection, polarization orientation inference, and ship motion direction estimation, distractors are randomly selected from predefined label sets. For numerical reasoning tasks like object counting, target dimension estimation, incidence angle inference, and target azimuth inference, distractors must align with realistic SAR imaging scenarios, simulating potential cognitive confusion in practical applications while effectively identifying random model selection. Thus, a “Physical Constraints + Gradient Divergence” generation strategy is adopted, constructing a three-tier distractor system comprising Adjacent Distractor, Mid-Range Distractor, and Special Distractor based on ground truth values. This ensures physical plausibility while utilizing differentiated numerical values to precisely evaluate model reasoning depth.

(1) Object Counting: Adjacent Distractors are true count ±1–3, simulating minor errors from visual fatigue or partial occlusion in manual counting; Mid-Range Distractors are true count ±3–6, simulating counting deviations from target overlap or edge blurring in dense scenes; Special Distractors are set to 0 or values exceeding scene capacity, eliminating model reliance on shallow default counting range memorization while ensuring distractors reflect actual counting uncertainties and accurately evaluate precise target quantity reasoning capability.

(2) Target Size Measurement: True values for target dimensions are calculated from object detection bounding boxes annotated with Oriented Bounding Boxes (OBBs). Adjacent interference items are set to ±5–15% of the true area, with step sizes dynamically adjusted based on actual area size: 0.1-meter increments for small targets and 1.0-meter increments for large targets. This simulates common minor errors in practical annotation, such as rounding discrepancies and pixel-level measurement inaccuracies, testing the model’s ability to distinguish subtle area variations and preventing misjudgments due to numerical precision issues. Med-range interference parameters are set at ±30% to ±60% of the actual area, simulating estimation deviations caused by varying target specifications in real scenarios. Special interference items employ a multidimensional strategy incorporating extreme values (±80–120% of true area), typical proportional values (e.g., 0.5×, 2×), and integer approximations. This approach eliminates reliance on shallow memory while precisely evaluating the model’s deep reasoning capabilities regarding area essence and its interference resistance through physically plausible yet cognitively biased numerical values.

(3) Incidence Angle Inference: Distractor generation strictly follows SAR imaging physics, ensuring angles within a conventional range (10–60°). Adjacent Distractors are true value ±3–5°, simulating subtle sampling interval variations; range-edge items are plausible values near 10° lower bound or 60° upper limit, testing global incidence angle range understanding; Special Distractors are set near attitude angles, preventing confusion between incidence/attitude angle physical meanings and reinforcing fundamental parameter comprehension.

(4) Target Azimuth Inference: Distractor design leverages ATRNet-STAR [45] acquisition characteristics (airborne SAR, 5° sampling interval). Adjacent Distractors are true value ±5–10°, aligning with minimal practical sampling differences; Mid-Range Distractors differ by 30–60°, representing significant orientation changes; Special Distractors utilize 360° azimuth periodicity, verifying model understanding of angular periodicity.

LLM-Assisted Generative Annotation Method: To significantly enhance annotation efficiency and scale, SAREval employs an LLM-assisted strategy for generating labels, aiding in question generation, prompt generation for visual grounding, and answer generation for image description. For question generation, the task definition and requirements for the model under test are first determined. DeepSeek is then used to generate 10–20 sets of question variants with different phrasings. Finally, five unambiguous questions are selected as the final prompts for each task.

For the complex needs of visual grounding prompt generation and image description answer generation, this study utilizes GPT-4o, combined with paired visible-light and SAR image data, to improve the accuracy of SAR image description generation. Firstly, basic information, such as image source, resolution, polarization mode and attribute information, including target category, bounding box, location, and size, are extracted from paired visible-SAR datasets like OpenEarthMap-SAR [41], assessing target uniqueness within the image to build accurate reference descriptions. Secondly, structured prompts are designed based on the reference descriptions and fed into GPT-4o along with the images, generating descriptions for both the SAR and visible-light images that cover global scene context and local details, with outputs organized in JSON format. Finally, professionals cross-compare, filter, and integrate the outputs, focusing on identifying common issues like hallucinations and attribute errors, to produce accurate and credible final descriptions, thus implementing quality control. This process yielded 3605 pairs for the visual grounding task and 395 pairs for the image description task. This method leverages the generative power of large models to drastically reduce manual annotation costs, while relying on attribute extraction and human verification to ensure data quality, effectively balancing the dual demands of scale and precision.

Expert Annotation Method for High-Difficulty Specialized Reasoning: To thoroughly test models’ understanding of SAR imaging mechanisms, this study innovatively introduced challenging reasoning tasks like ship motion direction detection and oil tank volume estimation. As public datasets lack corresponding professional annotations for such tasks, and their accuracy directly impacts the assessment of models’ understanding of SAR target physical properties, an expert-driven manual fine annotation approach was employed. Specifically, three professionals with over five years of experience in remote sensing were rigorously selected for annotation. Two annotators independently labeled the same target without communication, while a third reviewer performed verification. A data point was directly adopted only if the annotations from the first two annotators were consistent. For disputed samples, the third reviewer made the final judgment based on professional knowledge and image characteristics. This process resulted in 404 valid data points for ship motion direction judgment and 386 for oil tank volume estimation. This “dual independent annotation + third-party review” quality control process minimizes subjective bias, ensures annotation reliability, and provides high-quality benchmark data for subsequent performance evaluation of models on these challenging reasoning tasks. Quality control details are provided in Appendix A.4.

4. Evaluation Based on SAREval

4.1. Experimental Setup

To systematically evaluate the adaptability of mainstream VLMs for SAR image understanding tasks, this study conducts comprehensive multi-dimensional assessments of 11 representative open-source VLMs with the SAREval benchmark dataset. The selected models include Qwen2.5-VL-7B, DeepSeek-VL2-tiny, Phi-3.5-vision, along with models from the LLaVA series (LLaVA-1.5-7B, LLaVA-Next, LLaVA-OneVision) and the Intern series (InternVL2-4B, Mono-InternVL-2B, InternVideo2.5-8B, InternVL2.5-4B, InternVL3-9B). Detailed comparisons of model architectures and parameter sizes are provided in Appendix A.3. To ensure fair comparisons, all VLMs were evaluated under a zero-shot setting with standardized prompts, while accommodating specific input requirements of different models. Image inputs were uniformly resized to 512 × 512 pixels throughout the experiments.

For tasks employing MCQs, accuracy serves as the primary evaluation metric. This is defined as the ratio of correctly predicted samples to the total number of test samples. The metric precisely matches model outputs against ground truth answers, effectively quantifying model performance across various SAR image understanding capabilities. To eliminate potential bias from option ordering, correct answers for all multiple-choice questions were randomly distributed within option sequences during experimental design. This ensured equal probability for each option position to contain the correct answer, thereby preventing evaluation distortion that might arise from model learning positional preferences. For the Visual Grounding tasks, accuracy at specific Intersection over Union (IoU) thresholds—specifically Acc@0.5 and Acc@0.25—serves as the core evaluation metric. Standardized prompts guided models to output target coordinate information, which was then converted into rectangular bounding boxes within a unified pixel coordinate system. These predicted boxes were compared against manually annotated ground truth boxes using IoU calculations. A prediction was considered correct if the IoU value between the predicted and ground truth boxes exceeded the specified threshold. The number of correctly localized samples was then counted to calculate localization accuracy.

For the Image Description task, we employ three established automated metrics to quantify textual similarity from complementary perspectives. BLEU (Bilingual Evaluation Understudy) [46] measures n-gram precision (1 g to 4 g), emphasizing lexical exactness between generated and reference descriptions. ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation) [47] focuses on the longest common subsequence, capturing sentence-level structural similarity and fluency. BERTScore [48] computes semantic similarity using contextual embeddings from pre-trained language models, aiming to better correlate with human judgment by recognizing paraphrases and semantic equivalence beyond surface lexical overlap. Concurrently, we introduce a LLM-as-a-Judge [49] subjective assessment using DeepSeek, prompted to score descriptions across three SAR-relevant dimensions: accuracy, professionalism, and completeness. The system generates quantitative scores ranging from 0 to 100 based on these criteria. This paradigm leverages the superior semantic understanding and domain adaptability of modern LLMs, offering an evaluation more aligned with expert human judgment for complex, technical descriptions. The specific prompts and scoring rubric for the LLM judge are detailed in Appendix A.5.

Additionally, for robustness evaluation, model performance degradation (ΔAccuracy) between the original test set and noise-corrupted versions is calculated to quantify interference resistance, providing a comprehensive assessment of model stability in practical applications.

4.2. Main Results

4.2.1. Perception Capability Assessment

Perceptual capability constitutes the foundation of remote sensing image interpretation. Our experiments focused on fundamental visual tasks, including object detection, counting, and scene classification, primarily evaluating models’ capacity for extracting and recognizing SAR visual features. Results are presented in Figure 6 and

Table 2. Performance (%) of VLMs on the Perception Dimension tasks.

Model Name	AC	ShipC	OTC	BC	TCC	VC	ACD	SCD-M	SCD-H	VCD-M	VCD-H	MCOD	SC
Phi-3.5-vision	22.68	24.03	23.66	14.59	16.67	17.19	27.48	36.62	25.35	23.40	20.38	24.42	44.46
Qwen2.5-VL-7B	14.43	20.16	16.79	40.00	25.56	17.97	23.87	47.89	22.54	18.87	17.74	34.81	45.09
Deepseek-VL2-tiny	14.43	10.85	9.92	41.62	20.00	11.72	17.12	16.20	19.72	12.45	11.32	23.48	20.89
Mono-InternVL-2B	28.87	31.78	21.37	34.59	26.67	23.44	27.48	38.03	21.83	19.62	20.00	32.83	50.79
InternVL2-4B	20.62	20.93	19.85	15.14	6.67	17.19	7.66	30.99	9.15	7.17	2.64	28.27	46.99
InternVL2.5-4B	29.90	22.48	27.48	35.14	22.22	24.22	27.03	31.69	19.01	24.91	19.25	38.55	58.70
InternVideo2.5-8B	20.62	19.38	23.66	25.41	6.67	2.34	12.16	18.31	6.34	8.30	2.64	36.57	44.78
InternVL3-9B	20.62	21.71	30.53	16.76	11.11	25.00	22.97	23.94	16.20	18.87	16.23	38.43	19.46
LLaVA-1.5-7B	14.43	20.93	24.43	16.76	18.89	18.75	27.48	39.44	21.83	23.77	18.87	43.34	47.78
LLaVA-Next	9.28	2.33	0.76	13.51	13.33	8.59	16.67	38.73	22.54	22.26	17.36	38.32	38.61
LLaVA-onevision-qwen2-7b	26.80	20.93	14.50	24.86	12.22	21.09	24.32	38.03	24.65	27.92	23.02	46.03	61.71

Abbreviations adopted: AC for Aircraft Counting; ShipC for Ship Counting; OTC for Oil Tank Counting; BC for Bridge Counting; TCC for Tower Crane Counting; VC for Vehicle Counting; ACD for Aircraft Category Detection; SCD for Ship Category Detection; VCD for Vehicle Category Detection; MCOD for Multi-Class Object Detection; SC for Scene Classification. “M” denotes medium-difficulty tasks; “H” denotes high-difficulty tasks. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

,

Table 3. Scores of VLMs on the Image Description task. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

Model Name	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	BERTScore	Subjective Score
Phi-3.5-vision	0.2462	0.1037	0.0445	0.0244	0.1745	0.583	48.0739
Qwen2.5-VL-7B	0.2386	0.114	0.0523	0.0291	0.1759	0.5948	50.9989
Deepseek-VL2-tiny	0.1832	0.0861	0.0376	0.0206	0.1519	0.5563	51.393
Mono-InternVL-2B	0.0868	0.0429	0.0202	0.0114	0.0962	0.5062	34.9366
InternVL2-4B	0.1759	0.0844	0.0372	0.0204	0.1552	0.5775	47.4545
InternVL2.5-4B	0.2286	0.1111	0.0526	0.0297	0.1787	0.5936	48.7487
InternVideo2.5-8B	0.2772	0.1292	0.0582	0.0335	0.1955	0.5883	50.7744
InternVL3-9B	0.2093	0.101	0.0481	0.0254	0.1671	0.5947	58.0316
LLaVA-1.5-7B	0.2409	0.1179	0.0543	0.0316	0.1908	0.5555	54.8532
LLaVA-Next	0.187	0.0839	0.0344	0.0185	0.1589	0.5689	56.9065
LLaVA-onevision-qwen2-7b	0.1925	0.0905	0.0412	0.0226	0.1624	0.5829	46.0764

and

Table 4. Performance (%) of VLMs on the Visual Grounding task. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

Model Name	Acc@0.25	Acc@0.5
Phi-3.5-vision	0.88	0.05
Qwen2.5-VL-7B	0.44	0.15
Deepseek-VL2-tiny	1.42	0.20
Mono-InternVL-2B	0.06	0.00
InternVL2-4B	1.72	0.39
InternVL2.5-4B	0.83	0.10
InternVideo2.5-8B	1.96	0.59
InternVL3-9B	2.01	0.44
LLaVA-1.5-7B	5.10	1.37
LLaVA-Next	6.08	2.89
LLaVA-onevision-qwen2-7b	0.64	0.15

.

In image-level understanding tasks, results for Typical Scene Classification revealed that the LLaVA-OneVision-Qwen2-7B model achieved the best performance, followed by InternVL2.5-4B and Mono-InternVL-2B. These models demonstrated strong capabilities in grasping overall features for coarse-grained scene classification, meeting practical requirements for rapid screening of specific scene imagery. For Image Description (Table 3), InternVL3-9B achieved the highest subjective LLM evaluation score, showcasing advantages in natural language generation and professional semantic integration. However, all models scored notably low on automated evaluation metrics such as BLEU and ROUGE-L, indicating significant challenges remain in generating text descriptions that conform to SAR imagery expertise.

In object-level recognition tasks, LLaVA-onevision-qwen2-7b achieved the highest accuracy of 46.03% in Multi-class Object Detection, representing a 22.55% improvement over DeepSeek-VL2-tiny. For fine-grained classification and counting tasks, model performance exhibited a clear gradient effect as task difficulty increased. Qwen2.5-VL-7B performed best on the medium-difficulty Ship Classification Detection task, achieving 47.89%, whereas all models showed significant performance degradation on the high-difficulty task, with the highest accuracy reaching only 25.35%. This indicates limited capability in distinguishing highly similar fine-grained categories. In Object Counting tasks, bridges demonstrated the best overall performance due to their strong scattering characteristics and regular geometric shapes. DeepSeek-VL2-tiny achieved 41.62% accuracy in Bridge Counting but only 6.67% in Tower Counting (TCC), reflecting uneven adaptability to different ground object features. This imbalance particularly manifests in insufficient extraction capability for small targets and weak scattering features, directly attributable to SAR imagery’s lack of texture information and reliance solely on scattering intensity variations for fine category discrimination.

As shown in Table 4, performance in Visual Grounding tasks remained generally low, with all models achieving Acc@0.5 values below 3%. LLaVA-Next performed best at 2.89%, less than one-tenth of the optimal performance observed in optical image localization tasks. This phenomenon stems from SAR imagery’s lack of texture and color anchors, making it difficult for models to establish precise mappings between target visual features and linguistic descriptions. Additionally, geometric distortions caused by side-looking imaging further exacerbate localization deviations, indicating significant capability gaps in VLMs for fine-grained interpretation tasks like SAR target localization.

4.2.2. Reasoning Capability Assessment

The reasoning dimension focuses on deriving target physical attributes and SAR imaging parameters, primarily evaluating models’ logical mapping capabilities from visual features to deep semantics. Results are presented in

Table 5. Performance (%) of VLMs on the Reasoning Dimension tasks.

Model Name	IAI	TAI	POI	IRI	SMDE	BHE	TDM
Phi-3.5-vision	7.32	29.56	18.00	29.52	26.73	29.56	20.10
Qwen2.5-VL-7B	13.66	30.77	11.50	20.26	37.62	0.02	19.66
Deepseek-VL2-tiny	18.05	27.35	31.00	29.07	26.73	20.59	31.56
Mono-InternVL-2B	19.51	23.93	16.50	17.62	16.83	23.39	23.28
InternVL2-4B	25.37	25.64	22.00	21.15	6.93	29.84	7.84
InternVL2.5-4B	13.66	9.40	18.50	22.47	30.69	11.05	0.74
InternVideo2.5-8B	39.02	33.33	18.00	25.55	33.66	29.56	22.06
InternVL3-9B	22.44	24.79	16.00	23.35	29.70	28.84	32.35
LLaVA-1.5-7B	22.44	33.33	19.00	25.11	16.83	23.04	28.26
LLaVA-Next	46.34	38.46	18.00	23.35	9.90	12.58	1.72
LLaVA-onevision-qwen2-7b	43.41	28.21	20.50	17.62	19.80	19.61	31.28

Abbreviations adopted: IAI for Incidence Angle Inference; TAI for Target Azimuth Angle Inference; POI for Polarization Orientation Inference; IRI for Image Resolution Inference; SMDE for Ship Motion Direction Estimation; BHE for Building Height Estimation; TDM for Target Dimension Measurement. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

.

In Imaging Parameter Estimation tasks, models demonstrated relatively prominent performance in Incidence Angle Inference (IAI) and Target Azimuth Inference (TAI). LLaVA-Next achieved the best accuracy of 46.34% in IAI, while Qwen2.5-VL-7B reached the highest accuracy of 38.46% in TAI. However, overall performance in Polarization Orientation Inference (POI) remained low, with only DeepSeek-VL2-tiny exceeding 30%. This reflects insufficient model comprehension of multi-polarization scattering mechanisms, directly attributable to the lack of polarimetric SAR data in existing VLMs’ pre-training.

In Object Attribute Inference tasks, Ship Motion Direction Estimation showed the best performance. Qwen2.5-VL-7B effectively combined wake texture features with hydrodynamic principles for dynamic state inference, achieving the best performance of 37.62% in this task. However, this model performed extremely poorly in Building Height Estimation, revealing uneven reasoning capabilities across different physical attributes. In contrast, models like InternVL2-4B and Phi-3.5-vision could establish height correlations using features like layover and shadows, achieving nearly 30% accuracy. Meanwhile, InternVL3-9B, LLaVA-OneVision-Qwen2-7B, and DeepSeek-VL2-tiny all exceeded 30% accuracy in Target Dimension Measurement, indicating preliminary capability to establish pixel-to-physical size mappings and derive key spatial parameters of ground objects.

4.2.3. Robustness Capability Assessment

The robustness dimension focuses on evaluating interference resistance, primarily assessing VLMs’ stability in maintaining target recognition and classification performance when SAR images are affected by noise or clutter interference. Testing centered on two core SAR interference sources—speckle noise from coherent mechanisms and background clutter—using Multi-class Object Detection and Ship Classification Detection as benchmark tasks. Results are presented in

Table 6. Performance (%) of VLMs on Interference caused by Background Clutter task.

Model Name	No Speckle Noise	Add Speckle Noise	ΔAccuracy
Phi-3.5-vision	24.42	25.47	1.05
Qwen2.5-VL-7B	34.81	36.33	1.52
Deepseek-VL2-tiny	23.48	38.67	15.19
Mono-InternVL-2B	32.83	23.28	−9.54
InternVL2-4B	28.27	35.63	7.36
InternVL2.5-4B	38.55	26.05	−12.50
InternVideo2.5-8B	36.57	28.86	−7.71
InternVL3-9B	38.43	35.63	−2.80
LLaVA-1.5-7B	43.34	43.69	0.35
LLaVA-Next	38.32	33.64	−4.67
LLaVA-onevision-qwen2-7b	46.03	43.81	−2.22

This scenario uses the Multi-Class Object Detection task as the benchmark. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

and

Table 7. Performance (%) of VLMs on Interference caused by Background Clutter task.

Model Name	SCD-M			SCD-H
	No Background Clutter	Add Background Clutter	ΔAccuracy	No Background Clutter	Add Background Clutter	ΔAccuracy
Phi-3.5-vision	36.62	35.21	−1.41	25.35	23.94	−1.41
Qwen2.5-VL-7B	47.89	19.01	−28.87	22.54	4.93	−17.61
Deepseek-VL2-tiny	16.20	23.24	7.04	19.72	16.90	−2.82
Mono-InternVL-2B	38.03	19.01	−19.01	21.83	22.54	0.70
InternVL2-4B	30.99	50.00	19.01	9.15	50.70	41.55
InternVL2.5-4B	31.69	28.87	−2.82	19.01	11.97	−7.04
InternVideo2.5-8B	18.31	38.73	20.42	6.34	19.01	12.68
InternVL3-9B	23.94	35.92	11.97	16.20	20.42	4.23
LLaVA-1.5-7B	39.44	42.25	2.82	21.83	23.24	1.41
LLaVA-Next	38.73	40.85	2.11	22.54	22.54	0.00
LLaVA-onevision-qwen2-7b	38.03	42.25	4.23	24.65	21.83	−2.82

This scenario uses the Ship Category Detection task as the benchmark. “M” denotes medium-difficulty tasks, “H” denotes high-difficulty tasks. Values in bold red indicate the maximum; values with a blue underline indicate the minimum.

, and Figure 7.

In speckle noise interference tests based on Multi-class Object Detection (Table 6), we compared model performance before and after noise introduction. Models exhibited three distinct response patterns: InternVL2.5-4B, Mono-InternVL-2B, and InternVideo2.5-8B suffered severe performance degradation, showing extreme sensitivity to coherent speckle noise. In contrast, LLaVA-1.5-7B and LLaVA-OneVision-Qwen2-7B demonstrated exceptional stability, with performance fluctuations confined within ±2% under noise interference, making them ideal candidates for deployment in stable operational environments. Most strikingly, DeepSeek-VL2-tiny showed a significant +15.19% performance improvement after speckle noise introduction.

This phenomenon similarly manifested in background clutter interference tests benchmarked against Ship Classification Detection, as shown in Table 7. LLaVA series models maintained optimal stability across different difficulty levels, with variation amplitudes generally controlled within ±3%. Meanwhile, InternVL2-4B, InternVideo2.5-8B, and DeepSeek-VL2-tiny again demonstrated performance improvements under noise conditions, with InternVL2-4B achieving a remarkable 41.55% improvement in high-difficulty tasks.

5. Discussion and Analysis

Based on the experimental data from the SAREval, this chapter conducts an in-depth discussion across three dimensions, namely performance disparity analysis, critical bottlenecks, and future directions. Through cross-dimensional correlation analysis and cross-model comparative performance assessment, it reveals the current technical limitations of SAR-VLMs and outlines future development pathways. This study provides both theoretical foundation and practical guidance for developing reliable and high-performance visual-language models in SAR remote sensing.

5.1. Performance Disparity Analysis

Cross-dimensional correlation analysis reveals a capability fragmentation phenomenon in SAR-VLMs, where significant disparities exist between perceptual and reasoning capacities, while certain models exhibit unexpected pseudo-robustness characteristics. In the perception dimension, model capabilities degrade significantly along the “scene—object—pixel” hierarchy. For Scene Classification tasks, LLaVA-OneVision-Qwen2-7B and InternVL2.5-4B have reached practically useful levels, benefiting from the relative stability of scene-level scattering distribution patterns. However, performance remains generally poor in fine-grained Object Detection and Visual Grounding tasks, highlighting models’ inadequate adaptation to SAR-specific geometric distortions and scattering details. The unique side-looking imaging geometry of SAR systems imposes particularly demanding requirements on models’ spatial perception capabilities.

The reasoning dimension shows the weakest overall performance and exhibits a unique decoupling from model parameter scale. For instance, InternVL3-9B and DeepSeek-VL2-tiny achieve comparable performance in Target Dimension Measurement tasks, indicating that current models generally lack an internalized understanding of SAR physical mechanisms. Reasoning tasks typically require models to establish mappings from image appearances to physical realities, such as estimating building heights from layover features or inferring ship motion directions from wake textures, where deep domain knowledge dependency transcends purely data-driven learning paradigms.

In robustness evaluation, certain models exhibit anomalous performance improvement rather than degradation. This counterintuitive phenomenon, particularly DeepSeek-VL2-tiny’s significant improvement under speckle noise interference, challenges conventional understanding of robustness. Our analysis suggests this does not reflect genuine generalization capability but may stem from overfitting to specific noise patterns in training data. Since SAR speckle inherently contains scattering mechanism information, models might leverage memorized noise patterns for response matching, exhibiting a form of pseudo-robustness to specific noise types. This interpretation finds support in the LLaVA series’ stable performance—their open-vocabulary training paradigm may more effectively drive models to learn noise-invariant features, pointing toward a promising direction for developing genuinely robust SAR VLMs.

Through cross-model comparative analysis, we systematically reveal the capability boundaries and characteristic differences in various VLM architectures in SAR image interpretation, demonstrating a decoupling phenomenon between model scale and performance outcomes. Parameter scaling does not universally guarantee performance improvement. The LLaVA series models demonstrate robust overall performance in Image Description, Object Detection, and Visual Grounding tasks, showing particular strengths in semantic generation and spatial alignment. Their architecture combining CLIP visual encoders with dual-layer projectors provides a solid foundation for cross-modal alignment. However, this series shows weaker performance in inversion tasks involving SAR physical mechanisms, such as Imaging Parameter Estimation and geometric measurement, indicating their visual encoders lack prior knowledge of SAR-specific scattering mechanisms and imaging geometry.

The Intern series models possess advantages in parameter scale and training data volume, with InternVL3-9B and InternVideo2.5-8B demonstrating particularly strong cross-modal reasoning capabilities. InternVL3-9B leads in subjective Image Description ratings, while InternVideo2.5-8B excels in Ship Motion Direction Estimation. Through large-scale multimodal pre-training, these models develop more generalizable visual representations. However, the series shows unremarkable performance in certain foundational perception tasks, revealing imperfect balancing between high-level semantic understanding and low-level visual feature extraction.

Notably, as shown in Figure 8b, lightweight models like Mono-InternVL-2B achieve excellent performance in certain Object Counting tasks, even surpassing larger models. Meanwhile, the largest model InternVL3-9B fails to demonstrate advantages commensurate with its parameter count, suggesting that moderate model scales facilitate more efficient feature alignment in data-scarce specialized domains. This finding provides crucial guidance for model selection under resource constraints.

5.2. Critical Bottleneck

Data Perspective: Optical Prior Bias and Annotation Scarcity. With optical imagery constituting over 95% of general VLM training data, models exhibit significant optical-prior bias, typically manifesting as semantic mapping errors. As illustrated by the qualitative failure cases in Appendix A.6, models consistently apply erroneous optical analogies: strong, localized radar scatterers are misclassified as “white lines”, the inherent grayscale nature of SAR leads to incorrect inferences about scene illumination (e.g., “daytime”), and geometric structures exhibiting strong double-bounce reflections are mistaken for familiar optical objects like “bridges.” These are not mere recognition errors but fundamental misinterpretations stemming from a lack of physical understanding of SAR imaging mechanisms. Furthermore, the scarcity of professional SAR annotations exacerbates this issue. High-quality SAR interpretation requires expertise in microwave physics and imaging geometry, resulting in prohibitively high annotation costs. Consequently, existing annotated SAR data exhibits significantly lower density and precision compared to optical imagery, limiting models’ learning of fine-grained scattering features.

Model Perspective: Architectural Design and SAR Feature Misalignment. Current multimodal fusion architectures remain unoptimized for SAR characteristics. During visual encoding, ViT architectures designed for optical imagery struggle to effectively capture SAR-specific features like strong scatterer distributions and textural periodicity. Cross-modal alignment modules lack semantic understanding of technical terminology, often treating concepts like “incidence angle” and “polarization” as common vocabulary, resulting in deficient physical reasoning capabilities. Additionally, model input resolution limitations conflict with SAR’s high-resolution nature. Most VLMs support maximum input resolutions ≤ 1024 × 1024, while practical applications often use SAR imagery with 0.5–1 m resolution. During feature downsampling, substantial details of strong scatterers and small targets are lost, causing inadequate fine-grained recognition performance.

Evaluation Perspective: The Gap in Domain-Adapted Evaluation for SAR Descriptions. A critical challenge in evaluating open-ended tasks like Image Description is the significant divergence between traditional automated metrics (e.g., BLEU, ROUGE) and LLM-based subjective scores. This stems from a fundamental mismatch between general Natural Language Processing (NLP) evaluation paradigms and the specialized nature of SAR imagery. First, metrics relying on exact lexical matching [50] penalize legitimate technical paraphrasing, such as describing the same scatterer as a “bright signature” or “strong backscatter,” thereby underestimating semantically correct outputs. Second, they are domain-agnostic [51,52], unable to assess whether key SAR attributes are captured, conflating wording variation with factual error. In addition, they lack sensitivity to technical completeness, as a description may score highly by matching generic scene words while missing essential physical mechanisms, or be penalized for including qualifying details that actually enhance accuracy [53]. While LLM-based evaluation advances semantic understanding, it currently lacks standardized, fine-grained frameworks, and results can vary with prompt design and model knowledge. The absence of a domain-adapted evaluation suite that combines automated efficiency with expert-level semantic judgment hinders reliable performance measurement and constrains the deployment of generative VLMs in mission-critical SAR applications where descriptive precision is paramount.

5.3. Future Directions

Addressing the critical bottlenecks facing SAR VLMs, future research should pursue breakthroughs in data construction, model architecture, and evaluation frameworks to advance SAR image understanding from superficial correlation to mechanistic cognition.

Developing Physics-Aware Training Data Frameworks. Current VLMs pre-trained on optical imagery exhibit severe optical prior bias, necessitating SAR-specific multimodal training data. We recommend a dual-track strategy combining expert-guided semi-automatic annotation and physics-driven data augmentation to ensure data quality while scaling for large-model training. Specifically, synthetic data generation integrating electromagnetic simulation and physical rendering can simulate SAR imaging under multi-angle and multi-polarization conditions, establishing complete mappings from scattering mechanisms to image features. Simultaneously, constructing SAR-optical paired datasets through multi-modal data alignment in unified geographic coordinate systems establishes cross-domain semantic bridges, mitigating domain shift-induced cognitive biases. For data augmentation, introducing SAR-specific degradations like speckle noise models and geometric distortion transformations enhances model adaptability to real imaging environments.

Advancing Physics-Informed Model Architectures. Existing architectures suffer from inaccurate feature capture and misaligned cross-modal correspondence, hindering effective utilization of SAR physical priors. These limitations undermine model performance on complex tasks. At the visual encoder level, small convolutional kernels could capture local scattering features, while polarization attention mechanisms dynamically fuse multi-channel information, and geometric distortion invariance modules alleviate side-looking imaging effects. For cross-modal fusion, encoding physical priors like microwave scattering models and imaging geometry constraints as semantic vectors to participate in multimodal alignment would strengthen physical concept-visual feature associations. Simultaneously, exploring chain-of-thought reasoning could help models establish proper inference processes, while reinforcement learning mechanisms with carefully designed reward functions could guide models to uncover deep relationships in SAR imaging, establishing inference pathways from image features to physical parameters for deep comprehension of SAR imaging mechanisms.

Establishing Physics-Guided Evaluation Frameworks. Current evaluation metric development primarily migrates from natural language processing, remaining largely confined to semantic matching with untapped potential for visual-linguistic alignment in specialized domains. For SAR imaging, developing physically consistent evaluation metrics that quantify alignment between model outputs and electromagnetic scattering principles constraints, or establishing domain-knowledge-infused assessment mechanisms that incorporate microwave physics priors for judging output plausibility, could more objectively measure generative models’ application potential in vertical domains.

6. Conclusions

The advancement of VLMs presents new opportunities for intelligent interpretation of SAR imagery. Addressing the current lack of systematic evaluation benchmarks for SAR image interpretation, SAREval innovatively constructs 14,950 high-quality image–question pairs utilizing SAR data from diverse maritime, terrestrial, and urban scenarios through a hybrid annotation strategy combining a label-driven method, LLM-assisted generation, and expert verification. This benchmark systematically integrates 20 fine-grained tasks across three core capabilities and comprehensively evaluates model performance from basic object recognition to advanced semantic reasoning. Through comparative analysis of 11 mainstream vision-language models on the SAREval, this study reveals that current VLMs exhibit inherent optical prior bias due to modality imbalance in pre-training, demonstrating inadequate adaptation to geometric distortions and scattering characteristics in SAR imagery, thereby limiting their effectiveness in fine-grained object recognition and complex reasoning tasks. The development of SAREval establishes a reliable testing platform for the research and performance evaluation of SAR-oriented vision-language models, while its systematic evaluation framework and high-quality annotated data will accelerate the practical deployment of SAR-VLMs in real-world applications, including maritime surveillance, energy infrastructure management, and land resource monitoring.

We acknowledge, however, that the current construction of SAREval is inevitably shaped by the scope of publicly available datasets. This results in a benchmark that predominantly covers StripMap and TOPSAR imaging modes, while offering limited representation of other critical modalities. Furthermore, the existing task suite is currently constrained to maritime and terrestrial object understanding, indicating significant potential for expansion into broader application domains. To transcend these limitations and guide the evolution of SAR-VLM evaluation, we outline corresponding pathways for future work. First, to address the gap in task diversity, future iterations should introduce complex tasks such as multi-temporal SAR change detection and cross-modal (SAR-optical) collaborative understanding, directly responding to the core operational needs of dynamic monitoring and data fusion. Second, to overcome the constraints of data modality and scenario coverage, the benchmark will integrate continuous satellite observation streams and high-resolution polarimetric SAR data to enhance coverage of underrepresented regions (e.g., polar zones) and support more sophisticated land-cover inversion tasks. Third, to ensure the evaluation framework remains robust and practically relevant, the design of anti-interference tasks requires continuous refinement. This involves incorporating a wider array of realistic imaging degradation scenarios, thereby better aligning the benchmark with the challenges of real-world deployment and more effectively steering the technological iteration of SAR-VLMs.