Multimodal Prompt Learning for Spatial Reasoning in Remote Sensing Image Scene

Highlights

What are the main findings?

• A Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework is proposed, which realizes effective fusion of visual and text modalities to enhance remote sensing scene spatial relationship reasoning.
• A novel Geo-RSSG dataset with detailed ground object annotations, precise spatial relationships, and rich attributes is constructed, and UVSTPL achieves state-of-the-art performance on this dataset.

What are the implications of the main findings?

• The UVSTPL framework promotes the integration of vision-language multimodal learning and GeoAI, providing a new approach for intelligent interpretation of remote sensing data.
• The Geo-RSSG dataset bridges the gap in high-quality multimodal benchmarks for remote sensing reasoning tasks and provides a reliable foundation for future research in this field.

Abstract

A remote sensing scene graph (RSSG) enables machines to interpret interactions among ground objects in remote sensing images and supports semantic reasoning and description, thus making it a fundamental technique in the field. However, most existing scene reasoning approaches cannot fully utilize multimodal information, resulting in limited performance when inferring spatial relationships among ground objects. To this end, we propose a Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework, which integrates visual features with matched geospatial object labels, leverages a prompt learning module for multimodal feature extraction, and employs a refined UVTransE model to predict spatial relationships. The core principle of UVSTPL is to enhance semantic feature extraction and improve relationship prediction performance via the collaborative fusion of visual and linguistic modalities. To strengthen the model’s ability to reason about the spatial relationships among ground objects in images, a novel Geo-RSSG dataset is constructed, which includes precise annotations of geographic entities, spatial relationships, and attributes. Extensive experiments demonstrate that the proposed UVSTPL method outperforms benchmark models on the spatial relationship prediction task. In comparison with the best baseline method, our approach improves prediction precision by 1.85%, mean precision by 8.49%, mean recall by 17.46%, and mean F1-score by 12.97%. This study offers valuable insights for advancing the understanding and cognitive capabilities of remote sensing scenes.

1. Introduction

Remote sensing images serve as core data products for airborne Earth observation, and also act as a vital carrier for humans to interpret terrestrial geographical phenomena and analyze human–environment interactions. Essentially, such images capture spatial scenes of specific land areas from a top-down perspective, recording the attributes and spatial relationship characteristics of ground objects (Figure 1a). Human interpretation of remote sensing images involves two complementary levels: pixel-level and semantic-level understanding. Pixel-level analysis targets object recognition, boundary localization, and instance segmentation, delivering fundamental structural information of the scene (Figure 1b). As a higher-order cognitive process, semantic-level understanding aims to explicate spatial relationships and semantic attributes of ground objects, thereby forming structured and interpretable geographic knowledge (Figure 1c).

Currently, computers can effectively adopt large model fine-tuning and Transformer architecture to perform low-level understanding tasks for remote sensing images, including object detection [1,2,3], scene classification [4,5,6], and semantic segmentation [7,8,9]. Nevertheless, remote sensing images are captured from a unique top-down perspective and exhibit strong spatial heterogeneity, making high-level tasks—which demand precise analysis of geographic objects and their intricate spatial relationships—still highly challenging.

A remote sensing scene graph adopts a graph representation, where nodes denote ground objects or attributes, and edges stand for relational connections. It explicitly models geographic entities, their attributes, and inter-object spatial relationships within remote sensing imagery. As a structured knowledge representation paradigm, it supports fine-grained semantic understanding of remote sensing scenes and provides fundamental knowledge for downstream tasks such as image interpretation [10,11,12], visual question answering [13,14], and natural language description [15,16,17].

Scene graphs for natural images [18,19,20] have been thoroughly investigated in computer vision. Relevant studies mainly build upon mainstream datasets, including Microsoft COCO [21] and Visual Genome (VG) [22], to implement visual relationship prediction [23,24]. However, remote sensing images contain diverse ground objects, complex spatiotemporal relationships, and distinct scene characteristics. Existing scene graph reasoning approaches designed for natural images cannot be directly adapted to the remote sensing domain, and research on remote sensing scene graph reasoning remains relatively limited. Most existing algorithms yield unsatisfactory performance on remote sensing data, mainly due to two major challenges. First, publicly available remote sensing scene graph datasets remain scarce, and most existing datasets are originally developed for object detection. Ground objects in remote sensing images are complex and distributed, with large ground objects often enclosing smaller ones within their bounding boxes. Traditional bounding box localization struggles to accurately capture the morphological characteristics of ground objects, which inevitably introduces noise, impedes spatial relationship extraction, and ultimately degrades the accuracy and reliability of scene graphs. Second, existing datasets generally lack attribute annotations and only provide basic relationship labels, limiting the semantic richness of the resulting scene graphs.

To address the above issues and empower large models to conduct accurate fine-grained spatial cognition and reasoning, we propose a remote sensing image scene graph spatial relationship reasoning framework named UVSTPL (Unified Visual-Semantic Triple Prompt Learning). Based on a finely annotated scene graph dataset, integrating joint feature encoding and a dedicated relationship prediction network can effectively improve the accuracy of relationship prediction. Our main contributions are summarized as follows:

• We constructed Geo-RSSG, a large-scale remote sensing scene graph dataset, including detailed annotations for various ground objects, precise spatial relationships, and rich attribute information.
• We propose the UVSTPL framework for spatial relationship reasoning. It combines multimodal features with prompt learning and adopts UVTransE for robust relationship prediction, thereby improving the reliability and accuracy of spatial relationship prediction for remote sensing images.

2. Related Work

2.1. Remote Sensing Image Scene Graph Relationship Reasoning

To improve the capability of machines to recognize and understand remote sensing images, constructing specialized datasets is indispensable for verifying the efficacy of proposed approaches. Current datasets largely concentrate on labeling objects and their relationships within remote sensing scenes. GRTRD [25] contains 19,904 annotated geographic objects and 18,602 geospatial relationships. RSSGD [26] is a remote sensing scene graph dataset built based on the descriptive sentences of RSICD [27], consisting of objects, attributes, regional coordinates, and inter-objects relationships. Lin et al. [28] developed a scene graph dataset by selecting six object categories and twelve relationship types from the captions of three remote sensing image captioning datasets: UCM-CAPTIONS [29], Sydney-CAPTIONS [29], and RSICD [27], with relative positions of up, down, left, and right in this dataset. Derived from high-resolution remote sensing images, RSG [30] contains more than 210,000 objects and 400,000 triplets. It encompasses 11 complex geospatial scene types, 48 object categories annotated with oriented bounding boxes (OBB), and 58 relationship categories. SSGD [31] is automatically generated by a spatial relationship calculation model. It requires no manual annotation but involves complex computations. Most existing remote sensing scene graph datasets rely on bounding boxes for object annotation. However, ground objects in remote sensing images are mostly scattered and densely mixed, hindering accurate discrimination. Additionally, directional relationships in these datasets are represented by relative positions rather than actual geographic locations, and detailed attribute information is generally missing.

In terms of remote sensing image scene graph relationship prediction, Cui et al. [32] proposed the Multi-Scale Remote Sensing Image Interpretation Network (MSRIN) to achieve end-to-end recognition of remote sensing objects and spatial relationships. Li et al. [26] utilized the Multi-Scale Semantic Fusion Network (MSFN) to fuse and refine multi-scale semantic context, thereby enhancing the ability to understand remote sensing scenes. Chen et al. [25] put forward a novel method based on an object-relationship message-passing mechanism to effectively predict geospatial relationships in high-resolution remote sensing scenes. Lin et al. [28] designed a segmentation-driven model to generate more comprehensive and accurate remote sensing image scene graphs. In a further study, Lin et al. [33] integrated contextual information from all objects and statistical knowledge based on an adjusted Transformer architecture. Rui et al. [34] proposed a ship group relationship description (SGRD) method based on scene graph generation with a global and local context fusion network.

2.2. Scene Reasoning of Remote Sensing Images Based on Visual Language Models

Visual Language Models (VLMs) bridge computer vision and natural language processing. These models process and understand visual and textual information in a unified manner and have achieved remarkable performance in various computer vision tasks. Radford et al. [35] proposed a simple yet effective visual language model that combines image and text representations with contrastive learning. The model achieved remarkable success in various downstream tasks and shows strong generalization on unseen data. Subramanyam et al. [36] proposed CREPE (CLIP Representation Enhanced Predicate Estimation), which validated the effectiveness of CLIP for relationship prediction. Qiu et al. [37] employed a pre-trained CLIP model to extract features from remote sensing images, achieving impressive scene classification results with scarce annotations. Applenet [38] leverages multi-scale visual content and style information from the CLIP visual encoder to learn prompt tokens, addressing few-shot recognition and generalization of optical remote sensing scenes. As the first foundation visual language model dedicated to remote sensing, RemoteCLIP [39] learns robust visual features with rich semantics and well-aligned text embeddings. It supports a set of remote sensing tasks such as scene classification, cross-modal retrieval, and object counting. GeoRSCLIP [40] is a CLIP model fine-tuned on RS5M, the first large-scale remote sensing image–text pairing dataset. This work successfully migrates pre-trained VLMs to the remote sensing domain. Wang et al. [41] developed a continuously pre-trained CLIP model, which substantially outperforms vanilla CLIP and other baselines in three remote sensing downstream tasks: zero-shot scene classification, fine-grained attribute classification, and cross-modal retrieval. Meng et al. [42] proposed a remote sensing image semantic description model that combines CLIP latent variables with a multi-scale grouped Transformer. Despite the above progress, research on VLMs—and the CLIP model in particular—for spatial relationship reasoning in remote sensing scene graphs remains underexplored.

3. Geo-RSSG Dataset

3.1. Dataset Description

Most existing remote sensing scene graph datasets adopt object detection boxes and mainly focus on annotating objects and relationships. From a geographical perspective, we constructed a fine-grained geographical scene graph dataset based on semantic segmentation. It contains diverse geographical objects, authentic spatial relationships, and comprehensive attribute information. Geo-RSSG includes five major categories: water systems, residential areas and facilities, transportation, topography, and vegetation and soil, along with 45 subcategories. Detailed descriptions of these categories are presented in

Table 1. Geographical object types.

Major Category	Subcategory
Water Systems	river, ocean, lake, pond, reservoir
Topography	mountain, beach, island,
Vegetation and Soil	farmland, forest, meadow, greenbelt, bare land, paved surface
Residential Areas and Facilities	residential area, commercial area, industrial area, stadium, athletics track, basketball court, football field, tennis court, school, park, storage tank, greenhouse, container, swimming pool, shed
Transportation	parking lot, airport, railway station, harbor, wharf, runway, railway, highway, freeway, intersection, overpass, viaduct, bridge, roundabout, gas station, toll station

.

Geographic relationships include topological relationships and directional relationships, as summarized in

Table 2. Relationship types.

Major Category	Subcategory
Topological relationships	overlap, contained, surround, partially surround, parallel, touch, adjoin
Direction relationships	north, south, west, east, northwest, northeast, southwest, southeast, center

. We define ten types of topological relationships based on the nine-intersection model, with illustrative examples visualized in Figure 2. Different from existing scene graph datasets, directional relationships in our dataset are described using geographic cardinal directions (East, South, West, and North). This is determined by the fixed acquisition posture of remote sensing imaging platforms. For attribute information, we adopt a hybrid qualitative and quantitative labeling scheme. Qualitative attributes cover object color, shape, and regional properties (see

Table 3. Attribute types.

Major Category	Subcategory
Color	green, blue, white, black, gray, yellow, red, brown
Shape	straight, curved, round, oval, rectangular, square, striped, triangular, irregular, neatly arranged, scattered, high-rise, low-rise, long, short
Region	wide, narrow, big, small

), while quantitative attributes include area, width, and other measurable geometric metrics.

3.2. Data Annotation

The entire annotation flow includes the following steps:

(1)

Data Collection: To meet the needs of practical applications, high-resolution satellite images in the Geo-RSSG dataset were collected from Google Earth, with a spatial resolution of 0.3 m per pixel. The original exported images are 20,350 × 20,225 pixels in RGB format. Valid regions without borders or watermarks were cropped into non-overlapping patches of 512 × 512 pixels, preserving the original spatial resolution. A total of 3587 remote sensing images from two relatively representative cities in China were selected, including 2249 images of the coastal city of Guangzhou and 1338 images of the plain city of Zhengzhou. These images cover three types of scenes: urban areas, suburban areas, and rural areas.

(2)

Scene Graph Annotation: We designed a standardized dataset annotation process, as shown in Figure 3. In the object annotation stage, valuable image patches are extracted from remote sensing images, with latitude and longitude coordinate information retained, and instance segmentation is performed on geographic objects. All instance segmentation masks are manually annotated by professional annotators with reference to the original imagery. For the attribute annotation and relationship annotation stages, we developed an annotation tool, AR-annotation, to annotate the topological and directional relationships between adjacent ground objects.

3.3. Data Analysis

The finalized dataset contains 86,000 ground object labels, 150,000 relationship labels, and 246,000 attribute labels, with annotation examples visualized in Figure 4. The distribution of the five major categories of ground objects is presented in Figure 5, which presents a typical long-tail distribution consistent with the real-world geographic characteristics. Common geographic objects, including residential areas, roads, and greenbelts, dominate the dataset, while rare categories such as airports and railway stations appear less frequently.

The distribution of topological relationships is shown in Figure 6. Relationships including adjoin, touch, and parallel occur more frequently, whereas contained and overlap relations are relatively scarce. In contrast, directional relationships exhibit a relatively balanced quantity distribution. Figure 7 shows the distribution of each attribute category in the Geo-RSSG dataset. The annotation of attribute information, such as color and shape, not only enriches the information of geographic objects but also facilitates accurate geographic object recognition. For example, football fields possess representative attributes, including green color and rectangular shape, which align with human common sense.

4. UVSTPL Framework

Drawing on the relationship prediction network of CREPE [36], we proposed the UVSTPL framework for remote sensing spatial relationship reasoning. The overall architecture of UVSTPL is depicted in Figure 8, which consists of the following three steps: subject-object pairing, triplet prompt learning, and relationship classification.

Subject-object pairing: We extract the subject and object features of images through instance segmentation, instead of traditional bounding box annotations. All adjacent ground objects in the instance segmentation labels are combined into different subject-object pairs, which are fed into the CLIP text encoder. Meanwhile, the corresponding regions of adjacent ground objects are cropped from the original remote sensing images as the input for the CLIP image encoder.

Triplet prompt learning: A token embedding module is used to transform the image features of subjects, objects, and their combined regional areas into unified vector representations. The input to the network includes subject features, object features, and joint features of subject-object overlapping regions.

Relationship classification network: By subtracting the subject and object features from the previous joint region features, relationship features are obtained. These features are further fed into a fully connected network to acquire the probability distribution of relationship predictions, thereby enabling relationship prediction. Relational features are obtained by subtracting independent subject and object features from the extracted joint regional features.

4.1. Subject-Object Pairing

In this step, instance segmentation is performed on the original image I to obtain the text labels of the subject and object, and the regions corresponding to the entities in the segmentation results are extracted, as shown below:

$$\mathrm{S}= \left\{\mathrm{R}\left({\mathrm{E}}_1\right),\mathrm{R}\left({\mathrm{E}}_2\right),\dots ,\mathrm{R}\left({\mathrm{E}}_{\mathrm{n}}\right)\right\},$$

(1)

where ${\mathrm{E}}_{\mathrm{n}}$ represents the n-th entity, and $\mathrm{R}\left({\mathrm{E}}_{\mathrm{n}}\right)$ represents the region of the entity in the original image I.

The next step is to combine the entity regions in pairs, which can be expressed as:

$$\mathrm{U}\left({\mathrm{E}}_{\mathrm{i}},{\mathrm{E}}_{\mathrm{j}}\right)=\mathrm{R}\left({\mathrm{E}}_{\mathrm{i}}\right)\cup \mathrm{R}\left({\mathrm{E}}_{\mathrm{j}}\right),$$

(2)

where $\mathrm{U}\left({\mathrm{E}}_{\mathrm{i}},{\mathrm{E}}_{\mathrm{j}}\right)$ denotes the combination of the regions of two adjacent entities ${\mathrm{E}}_{\mathrm{i}}$ and ${\mathrm{E}}_{\mathrm{j}}$ in the same image I.

Text labels and combined region images are input into CLIP’s text and image encoders to obtain the subject features, object features, and image features of the combined regions.

4.2. Triplet Prompt Learning

To better extract relationship features from combined features, we use a learnable prompt combination structure. The structure is shown in Figure 9.

First, the input subject and object are tokenized using the token module, which converts original words or phrases into a series of token identifiers. In this step, each ground object is represented as a unique token ID. For example, “River” is represented as 2473, and “Grassland” as 5922. The relationship we intend to predict is marked as the same token for n times. All embeddings are bounded by the start-of-sequence [SOS] and end-of-sequence [EOS] tokens, with a maximum length of 77 tokens. Next, these tokens are converted into their corresponding vector representations through the token embedding module. During this process, each token is embedded into a fixed-length vector in the high-dimensional space, while preserving its semantic information.

To enhance the model’s adaptability to remote sensing scenarios, we introduce a dynamic adjustment function $∆(\cdot )$, which adjusts the initial token embedding ${\mathrm{M}}_{\mathrm{f}}$ based on the combined image feature ${\mathrm{I}}_{\mathrm{f}}$. The adjusted token embedding ${\mathrm{M}}_{\mathrm{\theta }}$ can be expressed as:

$${\mathrm{M}}_{\mathrm{\theta }}=∆\left({\mathrm{M}}_{\mathrm{f}}+{\mathrm{I}}_{\mathrm{f}}\right)$$

(3)

The dynamic adjustment function $∆(\cdot )$ is implemented as a multilayer perceptron (MLP) that takes the element-wise sum of the initial token embedding ${\mathrm{M}}_{\mathrm{f}}$ and the combined region image feature ${\mathrm{I}}_{\mathrm{f}}$ as inputs. The MLP consists of three hidden layers with ReLU activations: a linear layer mapping the 512-dimensional input to 256, a second layer mapping 256 to 128, a third layer mapping 128 back to 256, and a final linear layer projecting 256 to 512. The output of the MLP is an adjustment vector of dimension 512. This adjustment vector is then added element-wise to the original token embedding, producing the final adjusted token embedding ${\mathrm{M}}_{\mathrm{\theta }}$. The MLP contains approximately 0.33 million trainable parameters and is trained jointly with the prompt learning network. This design allows the model to dynamically adapt the token embedding based on the visual content of the subject-object combined region while preserving the original semantic information through the additive residual connection.

By combining the adjusted token embedding with the subject and object token embeddings, the triplet token embedding ${\mathrm{U}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ is formed, as described below:

$${\mathrm{U}}_{\mathrm{emb}}=\left\{{\mathrm{S}}_{\mathrm{emb}}+{\mathrm{M}}_{\mathrm{\theta }}+{\mathrm{O}}_{\mathrm{emb}}\right\},$$

(4)

herein, ${\mathrm{S}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ and ${\mathrm{O}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ respectively denote the token embeddings of the subject and object, while ${\mathrm{M}}_{\mathrm{\theta }}$ represents the combination of the adjusted and refined token embeddings.

We employ simple element-wise addition in Equation (4), drawing inspiration from the translation-based paradigm of TransE, in which relationships are modeled as vector offsets. Complex interactions among subject, predicate, and object are explicitly captured by the subsequent UVTransE classifier (Section 4.3), which computes a non-linear residual ${\mathrm{P}}_{\mathrm{f}} = \mathrm{MLP} ({\mathrm{U}}_{\mathrm{f}}-{\mathrm{S}}_{\mathrm{f}}-{\mathrm{O}}_{\mathrm{f}})$ based on the encoded joint text feature. Furthermore, our ablation study (Section 5.4) demonstrates that replacing additive fusion with a more sophisticated MLP-based non-linear fusion brings only marginal performance gains (<0.5% MF1). This validates that the simple additive formulation is empirically sufficient. Accordingly, we retain this simple yet effective design.

To construct hard negative samples, we compute pairwise cosine distances between predicate embeddings in the original 512-dimensional CLIP text space. For each positive predicate, we select the predicate with the smallest cosine distance (excluding itself) as the negative sample. This strategy forces the model to discriminate between semantically similar relationships, improving fine-grained reasoning. Figure 10 visualizes the predicate embeddings using t-SNE for illustration only; the actual negative sampling is based on distances in the original feature space. Then, the embedded triplet token ${\mathrm{U}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ is input into CLIP’s text encoder to obtain a unified text feature vector ${\mathrm{U}}_{\mathrm{f}}$, which integrates information about the subject, object, and relationship. For example, when the ground truth label is <residential area_1, adjoin, bare_land_3>, we construct <residential area_1, touch, bare_land_3> as a negative sample, ensuring that the negative sample has similar semantic features to the positive samples.

In the computation process, the features of the composite region image processed by the CLIP image encoder are represented as ${\mathrm{I}}_{\mathrm{f}}$. Thus, the composite feature vector can be expressed as:

$${\mathrm{U}}_{\mathrm{f}} ={ \mathrm{CLIP}}_{\mathrm{txt}}\left(\mathrm{\phi }\left({\mathrm{S}}_{\mathrm{e}\mathrm{m}\mathrm{b},}{\mathrm{I}}_{\mathrm{f},}{\mathrm{O}}_{\mathrm{e}\mathrm{m}\mathrm{b}}\right)\right),$$

(5)

herein, ${\mathrm{S}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ and ${\mathrm{O}}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ respectively denote the token embeddings of the subject and object, ${\mathrm{I}}_{\mathrm{f}}$ represents the combined region image feature, $\mathrm{\phi }$ denotes the learnable context vector, and ${\mathrm{U}}_{\mathrm{f}}$ stands for the joint feature vector.

Through self-supervised learning, the contrastive loss enables the model to capture relationship-specific features. It effectively distinguishes visually similar yet semantically distinct relationships and boosts the accuracy of fine-grained relationship prediction. Thus, the following contrastive loss function is used to optimize the learnable parameters:

$${\mathrm{L}}_{\mathrm{LTD}}=-\log {\displaystyle \frac{\exp \left(\mathrm{sim}\left({\mathrm{I}}_{\mathrm{f}},{\mathrm{U}}_{\mathrm{f}}\right)\right)}{\exp \left(\mathrm{sim}\left({\mathrm{I}}_{\mathrm{f}},{\mathrm{U}}_{\mathrm{f}}\right)\right)+\exp \left(\mathrm{sim}\left({\mathrm{U}}_{\mathrm{f}},{\widehat{\mathrm{U}}}_{\mathrm{f}}\right)\right)}},$$

(6)

where sim denotes cosine similarity, and $\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{I}}_{\mathrm{f}},{\mathrm{U}}_{\mathrm{f}}\right)$ represents the cosine similarity between the combined region image feature and the joint text feature. ${\widehat{\mathrm{U}}}_{\mathrm{f}}$ indicates the negative sample most similar to the ground truth label, and $\mathrm{e}\mathrm{x}\mathrm{p}$ stands for the exponential function.

The dynamic adjustment function $∆(·)$ is parameterized by learnable parameters $\mathrm{\phi }$. During training, it dynamically calibrates the context vector and prompt insertion positions by combining image features to achieve image-aware prompt generation. This mechanism strengthens the model’s representation capability for complex relationships and improves the efficacy of contrastive learning.

4.3. UVTransE Classifier

The joint text feature vector ${\mathrm{U}}_{\mathrm{f}}$ generated during the prompt learning phase, together with the subject ${\mathrm{S}}_{\mathrm{f}}$ and object feature vectors ${\mathrm{O}}_{\mathrm{f}}$, are fed into the UVMLP network for relation prediction. The network architecture of UVMLP is illustrated in Figure 11. Here, ${\mathrm{S}}_{\mathrm{f}}$ represents the subject feature vector, ${\mathrm{O}}_{\mathrm{f}}$ refers to the object feature vector, ${\mathrm{U}}_{\mathrm{f}}$ denotes the joint text feature vector, ${\mathrm{P}}_{\mathrm{f}}$ is the relation feature vector, ${\mathrm{P}}_{\mathrm{fgt}}$ is the relation feature vector for the true label, ${\mathrm{P}}_{\mathrm{logits}}$ represents the prediction probability of the relationship, and ${\mathrm{P}}_{\mathrm{score}}$ indicates the probability weight.

The relational feature representation is obtained by subtracting subject and object features from the joint text embedding. The relational feature vector is then fed into a fully connected network to calculate the probability distribution of the relationship. The Softmax function converts the probabilities into normalized weights, yielding the final relational prediction.

The loss function of the UVTransE network is:

$${\mathrm{L}}_{\mathrm{t}}= {\mathrm{L}}_{\mathrm{cross}}+{\mathrm{L}}_{\mathrm{cosine}}$$

(7)

${\mathrm{L}}_{\mathrm{cross}}$ is the cross-entropy loss function, and the formula is as follows:

$${\mathrm{L}}_{\mathrm{cross}}=-{\sum }_{\mathrm{x}}\mathrm{p}\left(\mathrm{x}\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(\mathrm{q}\left(\mathrm{x}\right)\right),$$

(8)

herein, $\mathrm{p}\left(\mathrm{x}\right)$ denotes the probability distribution of the ground truth relationship category, and $\mathrm{q}\left(\mathrm{x}\right)$ represents the predicted distribution of the model.

${\mathrm{L}}_{\mathrm{cosine}}$ is the cosine similarity loss:

$${\mathrm{L}}_{\mathrm{cosine}}=1-\cos \left(\mathrm{\theta }\right)=1-{\displaystyle \frac{{\mathrm{P}}_{\mathrm{f}}·{\mathrm{P}}_{\mathrm{fgt}}}{‖{\mathrm{P}}_{\mathrm{f}}‖\times ‖{\mathrm{P}}_{\mathrm{fgt}}‖}}$$

(9)

herein, $\mathrm{c}\mathrm{o}\mathrm{s}\left(\mathrm{\theta }\right)$ denotes the cosine similarity. Since we usually minimize the loss rather than directly maximize the similarity, we define the loss function using a negative value.

${\mathrm{L}}_{\mathrm{cross}}$ contributes to higher classification accuracy. However, when using the cross-entropy loss function alone, the UVTransE network suffers from unsatisfactory convergence, and the similarity between the predicted relationship features and the ground truth relationship features remains low. The loss balances the predicted relationship feature vector and the ground truth relationship feature vector ${\mathrm{P}}_{\mathrm{f}\mathrm{g}\mathrm{t}}$, which further improves the performance of the UVTransE network.

5. Experiments and Results

5.1. Experimental Setup

For the spatial reasoning task, we primarily focus on relationship prediction accuracy. When comparing against baseline models, we focus our analysis on the prediction of different relationship types. Therefore, we use precision as the metric for overall prediction performance. Additionally, we adopt mean $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}$, $\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}$, and $\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}$ to evaluate the model’s average performance across all relationship types.

Formally, we define the metrics as follows:

$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}$is the ratio of correctly predicted relationship triplets to all predicted triplets:

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(10)

For a single relationship category, $\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}$ is defined as:

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(11)

Mean precision ($\mathrm{M}\mathrm{P}$) is the average precision over all relationship categories:

$$\mathrm{M}\mathrm{P}={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}{\mathrm{n}}_{\mathrm{i}}$$

(12)

Mean recall ($\mathrm{M}\mathrm{R}$) is the average recall over all relationship categories:

$$\mathrm{M}\mathrm{R}={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\mathrm{R}\mathrm{ecal}{\mathrm{l}}_{\mathrm{i}}$$

(13)

Mean F1-score ($\mathrm{M}\mathrm{F}1$) is the harmonic mean of mean precision and mean recall:

$$\mathrm{M}\mathrm{F}1=2\times {\displaystyle \frac{\mathrm{M}\mathrm{P}\times \mathrm{M}\mathrm{R}}{\mathrm{M}\mathrm{P}+\mathrm{M}\mathrm{R}}}$$

(14)

where $\mathrm{T}\mathrm{P}$ = true positives, $\mathrm{F}\mathrm{P}$ = false positives, $\mathrm{F}\mathrm{N}$ = false negatives, and $\mathrm{N}$ denotes the total number of relationship categories.

The remote sensing scene graph dataset is divided into an 8:2 ratio, with 80% for training and 20% for testing. Model training is conducted on a computing service equipped with five NVIDIA A100 GPUs (80 GB memory each). The image and text encoders of the CLIP model are frozen during training; only the prompt learning module and the UVTransE classifier are updated. For the prompt learning network, the number of epochs is set to 50, the learning rate to 0.01, the batch size to 64, and the optimizer is SGD. For the classifier network, the number of epochs is 150, while other parameter settings are the same as those of the prompt learning network.

To evaluate the computational efficiency of UVSTPL, we report its number of trainable parameters, floating-point operations (FLOPs), and inference latency. Although multimodal prompts may introduce extra computation, our UVSTPL incurs only 2.20M additional trainable parameters since the CLIP backbone is kept frozen. The UVTransE classifier consumes 1.87 MFLOPs and incurs an inference latency of 7.67 ms per sample on an NVIDIA A100 GPU. Combined with the frozen CLIP encoder, the full model requires approximately 13.0 GFLOPs with an inference latency of 22.7 ms per sample. This modest increase yields a 12.97% absolute improvement in MF1 score over the optimal baseline (Remote CLIP-seg), demonstrating a favorable accuracy–efficiency trade-off.

5.2. Comparison Experiments

To comprehensively evaluate the proposed UVSTPL, we compare it against a set of representative baselines, spanning both general visual relationship prediction methods and approaches tailored for remote sensing imagery. These baselines are briefly described below.

CREPE-box [36]: A CLIP-based visual relationship prediction model that extracts subject-object regions via bounding boxes and adopts CLIP pre-trained priors to predict relationship predicates.

RSSGG_CS-seg [33]: A remote sensing scene graph generation method proposed in 2022. It fuses contextual information and statistical knowledge, and takes segmentation masks as input. This model serves as a competitive baseline for remote sensing scene graph tasks.

CLIP-box and CLIP-seg: We build two simple yet effective baselines based on the vanilla CLIP (ViT-B/32). For CLIP-box, bounding boxes are utilized to crop subject and object regions; for CLIP-seg, instance segmentation masks are adopted instead. A linear classifier is trained on the Geo-RSSG training set to identify relationship categories.

RemoteCLIP [39] and GeoRSCLIP [40]: These are recently released vision-language foundation models pre-trained on large-scale remote sensing image-text pairs. Compared with vanilla CLIP, the two models are more specialized for the remote sensing domain. We treat them as strong upper-bound baselines to verify whether our UVSTPL can surpass state-of-the-art domain-specific foundation models.

For a fair comparison, all baseline models are trained on the Geo-RSSG training set with their image encoders frozen. Only the linear classifier or task-specific prediction head is updated during training. Consistent with this setting, our UVSTPL also freezes the pre-trained CLIP encoder, and only optimizes the prompt learning module and UVTransE classifier.

We evaluate all methods in terms of precision, MP, MR, and MF1, and the quantitative results are presented in

Table 4. Precision and recall of different models.

Method	Precision	MP	MR	MF1
CREPE-box	50.56	38.22	33.24	35.73
RSSGG_CS-seg	61.83	49.32	38.73	44.03
CLIP-box	88.44	44.01	40.77	42.39
CLIP-seg	88.44	54.87	42.01	48.44
RemoteCLIP-box	88.52	49.56	41.44	45.50
RemoteCLIP-seg	88.43	56.65	42.20	49.43
GeoRSCLIP-box	88.51	53.78	43.04	48.41
GeoRSCLIP-seg	88.25	56.22	41.48	48.85
UVSTPL-box	90.23	61.67	50.18	55.93
UVSTPL-seg	90.28	65.14	59.66	62.40

Note: Bold indicates the best performance for each metric.

. In this work, the suffix “-box” indicates that subject-object combined regions are extracted using bounding boxes, while “-seg” denotes regions derived from segmentation masks. Among all baselines, RemoteCLIP-seg achieves the best overall performance, with a precision of 88.43%, MP of 56.65%, MR of 42.20%, and MF1 of 49.43%.

Our proposed UVSTPL-seg outperforms Remote CLIP-seg by a clear margin across all metrics: precision increases by 1.85% (90.28 vs. 88.43), MP by 8.49% (65.14 vs. 56.65), MR by 17.46% (59.66 vs. 42.20), and MF1 by 12.97% (62.40 vs. 49.43). When adopting bounding box inputs, UVSTPL-box also exceeds the top-performing baseline GeoRSCLIP-box, yielding performance gains of 1.72% in precision (90.23 vs. 88.51), 7.89% in MP (61.67 vs. 53.78), 7.14% in MR (50.18 vs. 43.04), and 7.52% in MF1 (55.93 vs. 48.41).

Meanwhile, the precision rates of different models in each relationship are compared (see

Table 5. Precision of different models on individual relationships.

Method	Overlap	Adjoin	Intersect	Contained	Partially Surround	Parallel	Touch	Cross	Cover	Surround
CREPE-box	0	76.11	53.24	18.07	9.56	80.56	86.11	43.24	0	15.32
RSSGG_CS-seg	0	75.54	60.73	53.41	17.23	88.83	92.54	65.73	0	37.23
CLIP-box	0	82.11	64.89	48.39	0	90.99	95.17	58.59	0	0
CLIP-seg	0	82.45	64.60	53.92	27.34	91.70	95.11	71.67	0	51.85
RemoteCLIP-box	0	82.94	64.62	68.18	45.24	90.98	95.23	48.39	0	0
RemoteCLIP-seg	0	82.48	63.33	57.78	43.10	92.06	95.31	67.19	0	52.78
GeoRSCLIP-box	0	83.06	63.76	53.49	45.00	91.92	95.13	50.90	0	54.55
GeoRSCLIP-seg	0	83.08	66.05	36.54	43.86	89.30	95.38	67.12	0	46.42
UVSTPL-box	0	90.23	74.91	19.99	8.93	94.03	98.47	60.66	0	46.67
UVSTPL-seg	0	90.28	76.62	43.22	29.65	94.84	97.95	81.15	25.00	56.00

Note: Bold indicates the best performance for each metric.

for results). Overall, the prediction precision rate using the -seg method is higher than that using the -box method. Although the UVSTPL-box method has an unsatisfactory prediction effect on small sample relations, its prediction accuracy for common relations is higher than that of the baseline model. The UVSTPL-seg method has reached the highest precision in the prediction of other relationships except for “contained” and “partially surround”. In terms of the small sample relationship, especially for “cross”, “cover” and “surround”, the prediction accuracy of our model is much higher than that of other models.

Figure 12 presents the prediction results of representative models for different relational categories. To balance clarity and comprehensiveness, we select CREPE-box, RSSGG_CS-seg, CLIP-seg, RemoteCLIP-seg, GeoRSCLIP-seg, UVSTPL-box, and UVSTPL-seg for visualization. Consistent with other models, our method fails to make correct predictions for the “overlap” category. However, our method well differentiates similar topological relations, e.g., between “cross” and “intersect”, as well as “contained” and “surround”. Notably, it gains obvious advantages on challenging spatial relations: it reaches 25.00% precision on “cover”, whereas nearly all baselines score 0% for this category. Full precision values of all models, including CLIP-box, RemoteCLIP-box, and GeoRSCLIP-box, which are not displayed in the figure, are summarized in Table 5.

5.3. Results Analysis

5.3.1. Analysis of Model Relationship Prediction Performance

As shown in

Table 6. Model evaluation under different data inputs and different contexts.

Method	n_ctx	Precision	MP	MR	MF1
UVSTPL-box	2	90.22	59.07	51.64	55.36
	4	90.23	61.67	50.18	55.93
	6	89.94	61.91	56.42	59.17
UVSTPL-seg	2	89.44	63.66	58.08	60.87
	4	90.28	65.14	59.66	62.40
	6	89.94	51.43	50.19	50.81

Note: Bold indicates the best performance for each metric.

, we compare the precision, mean precision, mean recall, and mean F1-score of the UVSTPL-seg and UVSTPL-box models as n_ctx (context size) is set to 2, 4, and 6. It can be observed that different context lengths have a certain impact on the model’s prediction performance, and the models achieve the best performance when n_ctx = 4. This indicates that appropriate context information can improve model performance. Meanwhile, segmentation-based models maintain a relatively high mean precision and mean recall. Since relationships in our dataset are usually expressed as one or two words, UVSTPL can better predict masked relationships when n_ctx = 4.

5.3.2. Analysis of the Distribution of Model Relationship Prediction Precision

Under different n_ctx values, the precision distribution of UVSTPL-seg and UVSTPL-box across 10 relationship types is analyzed in the form of confusion matrices and radar charts. For the confusion matrix, the horizontal axis represents predicted labels, the vertical axis represents true labels, and the depth of color indicates the sample count. Figure 13 uses segmented region features, Figure 14 uses detected bounding box region features, and the same patterns apply to Figure 15a and Figure 15b respectively.

The model predicts common relationships (e.g., “adjoin”, “intersect”, “parallel”, “touch”) consistently well, and also achieves high precision with uncommon ones (e.g., “contained”, “cross”, “cover”, “surround”). In contrast, the prediction for “overlap” is notably poor—all models obtain zero precision in this category (see Table 5 and Figure 12a). Figure 16 shows a typical failure case where <meadow1, overlap, pond1> is misclassified as <meadow1, adjoin, pond1>. The underlying reasons for this complete failure are analyzed in Section 5.5.1.

When the context size is set to 2 or 4, the two methods produce fairly similar relationship predictions. However, at a context size of 6, their predictions diverge considerably. This is because UVSTPL-seg extracts more accurate object contours from segmented regions and therefore requires less contextual information. In contrast, the UVSTPL-box, which relies on bounding boxes, tends to include background noise and thus benefits from additional context. Notably, UVSTPL-seg can predict the rare relationship “cover” with 25.00% precision, whereas UVSTPL-box almost entirely fails to do so (0% precision).

5.3.3. Clustering Analysis of the Impact of Context Length

We adopt the t-SNE dimensionality reduction method to visualize the results on the test set. As shown in Figure 17 and Figure 18, we feed both bounding boxes and segmented regions into UVSTPL and compare its ability to distinguish among different relationship types. The results indicate that both methods achieve the best separation among relationship types when the context length is set to 4. At a context length of 2, the clustering performance remains satisfactory. However, at a context length of 6, the clarity and separation of clusters degrade notably, reflecting reduced discriminability. In Figure 17c, at a context length of 4, the separation among relationship categories is clearest, with relationships such as “parallel”, “touch”, and “adjoin” forming distinct clusters. Across all context lengths and both input types, the segmentation-based method consistently outperforms the bounding-box-based method in distinguishing relationships, particularly when the context length is 4. This indicates that the fine-grained spatial information provided by segmentation masks enhances the model’s ability to discriminate subtle spatial relationships, especially when attending to local contexts.

Our experiments show that the discriminative performance of UVSTPL depends on the input modality and requires a suitable context length. Moreover, incorporating excessive contextual information can introduce noise or irrelevant cues that may obscure the distinctive features of individual relationships. The strong performance of the segmentation-based method, especially at a context length of 4, underscores the importance of combining fine-grained spatial information with focused local context to accurately identify relationships in complex remote sensing scenes.

5.4. Ablation Experiments

5.4.1. Effect of Network Sub-Modules

We adopt the best-performing UVSTPL-seg variant, which takes segmented regions as input with a context length of 4. The performance of different sub-networks is compared in terms of precision, MP, MR, and MF1 (see

Table 7. Performance of the UVSTPL-seg model with different sub-networks.

Sub-Network		Precision	MP	MR	MF1
PL	UV
√	×	86.77	55.58	43.36	49.47
×	√	88.84	59.42	41.22	50.32
√	√	90.28	65.14	59.66	62.40

Note: “PL” refers to the prompt learning network, while “UV” denotes the UVTransE-based classifier network. “√” denotes experiments with the corresponding sub-network integrated, and “×” denotes those without the sub-network. Bold indicates the best performance for each metric.

). Additionally, confusion matrices and t-SNE dimensionality reduction feature maps are used to analyze the relationship prediction capabilities of the different sub-networks.

When the model uses both the prompt learning network and the UVTransE classifier simultaneously, it achieves the best prediction performance. Without the UVTransE classifier, the model’s precision drops to 86.77%, and its mean precision and mean recall fall to 55.58% and 43.36%, respectively. When the prompt learning module is removed, precision decreases only slightly. However, both mean precision and mean recall drop, with recall falling to just 41.22%. These results indicate that the UVTransE classifier contributes more to precision improvement, while the prompt learning module plays a critical role in few-shot relationship prediction.

The confusion matrix, shown in Figure 19, illustrates the classification performance of the UVSTPL-seg model. As shown in Figure 19a, removing the UVTransE classifier results in a large number of misclassifications, particularly for categories such as “intersect”, “contained”, and “partially surround”. This indicates that prompt learning alone cannot improve precision across multiple categories. As shown in Figure 19b, the introduction of UV improves the prediction results for common relationships, particularly in terms of identifying the “intersect” relationship. However, the prediction results for “contained” and “partially surround” (which have a small number of samples) are still not ideal. The most significant performance improvement is observed in Figure 19c. The synergistic combination of PL and UV significantly improves the precision for all relationship categories, especially in distinguishing fine-grained spatial relationships such as “contained” and “partially surround”. Collectively, these results highlight the complementarity of PL and UV in enhancing the UVSTPL-seg model’s ability to distinguish subtle spatial relationships in complex remote sensing scenarios.

In Figure 20, we illustrate the impact of different sub-modules within the UVSTPL framework on distinguishing relationship features. The t-SNE visualization shows the distribution of 10 relationship categories under various configurations (with a context length of 4). In Figure 20a, the model with only the PL network achieves good clustering performance only for the “adjoin” and “parallel” relationships. In contrast, the model with only UV (Figure 20b) exhibits poor clustering performance. This indicates that without contextual prompts, the model struggles to distinguish among different relationship categories. Figure 20c demonstrates the most significant improvement, where the integration of both prompt learning and UVTransE leads to distinct clusters for most relationship categories. This configuration yields excellent class separation, particularly for similar relationships such as “contained”, “adjoin”, and “partially surround”. The clear feature separation observed under this setting highlights the synergistic effect of combining the prompt learning module with the UVTransE classifier. Overall, UVSTPL substantially enhances the ability to discriminate subtle spatial relationships in complex geographic scenes.

5.4.2. Effect of Triplet Fusion Strategies

To investigate whether additive fusion imposes performance limitations, we substitute it with a two-layer MLP that operates on the concatenated embeddings of the subject, predicate, and object. As presented in

Table 8. Effect of triplet fusion strategies.

Triplet Fusion	Precision	MP	MR	MF1
Additive (ours)	90.28	65.14	59.66	62.40
MLP-based (concatenation)	90.45	65.38	59.82	62.65

, the MLP-based fusion yields only a negligible MF1 gain of 0.25 percentage points (62.65% vs. 62.40%). This empirical evidence demonstrates that additive fusion is not a bottleneck for our model, as complex relational interactions are sufficiently modeled by the subsequent UVTransE module. Accordingly, we adopt the additive formulation for its simplicity and efficiency.

5.4.3. Effect of Negative Sampling Strategies

To evaluate the contribution of hard negative sampling, we compare random sampling with our cosine-distance-based hard negative sampling. As shown in

Table 9. Effect of negative sampling strategies.

Negative Sampling	Precision	MP	MR	MF1
Random sampling	87.62	60.53	52.47	56.13
Hard negative sampling (cosine distance, ours)	90.28	65.14	59.66	62.40

, hard negative mining consistently outperforms random sampling across all metrics, with an absolute gain of +6.27% in MF1 (62.40% vs. 56.13%). This verifies that the proposed strategy well distinguishes semantically similar relationships. Therefore, we adopt this strategy in our framework.

5.5. Discussion

5.5.1. Failure on the “Overlap” Relationship

As shown in Table 5 and Figure 12a, all models (including the proposed UVSTPL and the strong remote sensing foundation baselines) achieve zero precision on the “overlap” relationship. The universal failure across all approaches indicates that this limitation is not model-specific, but originates from inherent topological ambiguity, semantic coupling, and extreme data imbalance in remote sensing scene interpretation.

First, extreme sample scarcity. The dataset contains only 28 “overlap” instances, accounting for less than 0.02% of all relational samples. Such severe class imbalance restricts the model from learning stable and discriminative features for this tail category.

Second, high semantic similarity. The cosine similarity between “overlap” and “adjoin” in the CLIP text embedding space reaches 0.92, which is substantially higher than the average pairwise similarity of 0.68. The tight semantic alignment causes inherent language-level confusion, hindering vision-language differentiation of the two relations.

Third, inherent visual topological ambiguity (core cause). The fundamental challenge lies in the inconsistent pixel-level manifestation of abstract topological rules in remote sensing imagery. Geometrically, “adjoin” describes boundary sharing without area intersection, while “overlap” denotes partial area overlap. However, narrow overlapping regions lack visible edge cues, as the upper object occludes underlying surfaces and removes boundary contours. In contrast, “adjoin” scenarios consistently present clear and continuous edges, forming a strong visual prior for the model.

Faced with edge-free overlap samples and overwhelmingly dominant “adjoin” cases, the model lacks contradictory cues to override its learned prior and, thus, statistically defaults to “adjoin” predictions. Combined with terrain undulation, variable imaging angles, fragmented land cover, and mixed-pixel interference, subtle overlap and adjacency become visually indistinguishable. This ambiguity is an intrinsic limitation of remote sensing visual interpretation rather than a simple dataset defect.

Essentially, this failure stems from the inherent mismatch between abstract topological definitions and ambiguous visual presentations of remote sensing scenes. Even with sufficient samples, pure vision-language models struggle to distinguish these fine-grained relations without explicit geometric constraints. For future improvement, we will integrate geometric prior information (e.g., IoU metrics), generate synthetic overlap samples for tail-class augmentation, and adopt reweighted loss functions and few-shot learning strategies to alleviate long-tail learning bias.

5.5.2. Few-Shot Relationships Remain Challenging

Although UVSTPL achieves notable improvements on rare relations such as “cross” and “surround” (e.g., 81.15% vs. 67.19% for “cross”; 56.00% vs. 52.78% for “surround”), the absolute accuracy for “cover” (25.00%) and “surround” (56.00%) remains moderate. As shown in Table 5, all baseline models, including RemoteCLIP and GeoRSCLIP, perform poorly on these tail categories, and most attain 0% precision on “cover”.

The long-tail distribution is alleviated but not fully resolved. Apart from extreme class imbalance, the core challenge stems from the ambiguous visual boundaries of these complex spatial relations in remote sensing imagery. The visual manifestation of the “cover” relation often overlaps with partial overlap and adjacency. Large-area coverage shares similar visual patterns with object intersection, while partial coverage is difficult to distinguish from simple adjacency due to mixed pixels and shadow interference. Likewise, the topological difference between “surround” and other adjacent relations is easily obscured by irregular layouts of ground objects in complex land cover scenes.

To tackle these challenges, we put forward several future research directions: generating augmented samples for rare and visually ambiguous relations via synthetic data creation, adopting reweighted loss functions to balance category learning, and developing few-shot learning strategies dedicated to remote sensing scene graph tasks to boost feature representation for tail classes.

5.5.3. Choice of CLIP Backbone

In this study, we adopted the generic CLIP (ViT-B/32) as the visual language backbone instead of remote-sensing-specific variants (RemoteCLIP, GeoRSCLIP). This is a deliberate design choice: using a backbone that has not been pre-trained on remote sensing data provides a more challenging test for our prompt learning and UVTransE modules. It allows us to demonstrate that the observed improvements come from the proposed architectural components, not from a stronger pre-trained encoder. Moreover, the UVSTPL is “backbone-agnostic”. It can be directly applied to any CLIP-style encoder without modification. Comparison experiments indicate that replacing the generic CLIP with RemoteCLIP yields an additional performance gain of about 1–2% in MF1, which we plan to investigate systematically in future work.

5.5.4. Geographic Transferability

For remote sensing spatial relationship recognition, geographic transferability describes a model’s ability to retain stable performance on unseen regions with different geographic features, a property that defines its practical applicability.

The Geo-RSSG dataset employed in this work consists of remote sensing images from two Chinese cities: Guangzhou, a coastal city with complex terrain, and Zhengzhou, a representative plain city. While the dataset includes urban, suburban, and rural scenes and covers two typical geographic types, the model’s performance on mountainous areas and overseas cities with unique land-use patterns remains unquantified. Several critical factors limiting cross-region transferability are discussed as follows.

First, regions differ substantially in land cover and object appearance. Noticeable variations exist in architectural styles, building materials, vegetation, road networks, and spectral characteristics. For example, residential buildings in European countries show distinct roof designs and colors compared with those in China. Our vision-language backbones are pre-trained on massive global data and can learn generalizable representations to alleviate domain shift. Nevertheless, fine-tuning on data limited to the two domestic cities introduces scene bias, making the model overly adapted to local visual characteristics.

Second, spatial relationships follow region-specific statistical distributions. Divergent urban planning results in varying occurrence frequencies and visual forms of topological relations, such as “adjoin” and “contained”. Models trained on compact urban scenes often struggle with sparse suburban areas and rugged mountainous regions. Terrain undulation, shadow occlusion, and fragmented ground objects in mountainous areas further change how relations like “adjoin” and “surround” appear visually. Such domain shifts break the consistency of data distribution, leading to performance drops when the model is applied to new regions.

Third, image scale and resolution present additional challenges. The entire dataset uses a fixed ground sampling distance (GSD) of 0.3 m, matching common high-resolution satellite data. When processing images from different sensors or with varying zoom scales, resampling operations or scale-invariant feature learning are required to maintain reliable results.

To overcome these limitations, we plan the following future work. We will first construct and annotate a multi-region test set for quantitative transferability assessment. We will also explore domain adaptation methods, including adversarial feature alignment and test-time adaptation to bridge cross-domain gaps. Moreover, we will exploit unlabeled cross-region data through semi-supervised learning and decouple relational semantics from scene-specific visual features to improve model robustness across geographic scenarios.

6. Conclusions

To enhance the ability of large models to understand remote sensing images, this study proposes a remote sensing image scene graph spatial relationship reasoning method based on joint visual-semantic triplet prompt learning. We construct Geo-RSSG, a fine-grained remote sensing scene graph dataset that defines comprehensive entity and relationship types and provides rich geographic semantic information. Built upon semantic segmentation results, the proposed method integrates the CLIP model, prompt learning, and the UVTransE classifier to perform spatial relationship reasoning. The experimental results demonstrate that the UVSTPL model significantly outperforms baseline methods in both precision and recall. This research advances the understanding of remote sensing scenes and is of considerable value for improving spatial cognition in remote sensing imagery.

However, several limitations remain. Constrained by natural and anthropogenic geographic factors, the ground objects and relationships in our dataset follow a heavily long-tailed distribution, which limits the model’s ability to learn complex relationships such as “contained and surround” and “adjoin and overlap”. Moreover, the dataset scale needs to be further expanded, which can be addressed through data augmentation and annotation, assisted by pre-trained models. This study focuses on the prediction of topological relationships, as directional relationships can achieve very high accuracy through spatial relationship calculations, but such calculations usually involve high computational complexity and time costs. Therefore, based on the constructed dataset, the inference of directional relationships can be realized by fine-tuning multimodal large models. Additionally, while the method proposed in this paper focuses on relationship reasoning within scene graphs, a promising extension is to convert scene graph datasets into question–answer pairs and leverage visual question answering (VQA) to jointly predict entities, relationships, and attributes, thereby enabling automatic generation of remote sensing scene graphs.

In future work, we plan to further enrich the Geo-RSSG dataset, leverage large vision-language models to infer directional relationships, and explore the application of scene graphs to scene reasoning tasks such as image captioning and VQA.