SplitGround: Long-Chain Reasoning Split via Modular Multi-Expert Collaboration for Training-Free Scene Knowledge-Guided Visual Grounding

Abstract

Scene Knowledge-guided Visual Grounding (SK-VG) is a multi-modal detection task built upon conventional visual grounding (VG) for human–computer interaction scenarios. It utilizes an additional passage of scene knowledge apart from the image and context-dependent textual query for referred object localization. Due to the inherent difficulty in directly establishing correlations between the given query and the image without leveraging scene knowledge, this task imposes significant demands on a multi-step knowledge reasoning process to achieve accurate grounding. Off-the-shelf VG models underperform under such a setting due to the requirement of detailed description in the query and a lack of knowledge inference based on implicit narratives of the visual scene. Recent Vision–Language Models (VLMs) exhibit improved cross-modal reasoning capabilities. However, their monolithic architectures, particularly in lightweight implementations, struggle to maintain coherent reasoning chains across sequential logical deductions, leading to error accumulation in knowledge integration and object localization. To address the above-mentioned challenges, we propose SplitGround—a collaborative framework that strategically decomposes complex reasoning processes by fusing the input query and image with knowledge through two auxiliary modules. Specifically, it implements an Agentic Annotation Workflow (AAW) for explicit image annotation and a Synonymous Conversion Mechanism (SCM) for semantic query transformation. This hierarchical decomposition enables VLMs to focus on essential reasoning steps while offloading auxiliary cognitive tasks to specialized modules, effectively splitting long reasoning chains into manageable subtasks with reduced complexity. Comprehensive evaluations on the SK-VG benchmark demonstrate the significant advancements of our method. Remarkably, SplitGround attains an accuracy improvement of 15.71% on the hard split of the test set over the previous training-required SOTA, using only a compact VLM backbone without fine-tuning, which provides new insights for knowledge-intensive visual grounding tasks.

1. Introduction

Scene Knowledge-guided Visual Grounding (SK-VG) is a recently proposed knowledge-driven detection task that requires a model to leverage scene knowledge to comprehend more natural and contextually referring expressions [1]. As shown in the example in Figure 1, the core distinctions between SK-VG and traditional visual grounding (VG) tasks lie in two key aspects: In terms of input structure, SK-VG operates on a triple modality (image, query, and scene knowledge), distinguishing it from the dual modality (image and query) employed by traditional VG. In terms of semantic content, the queries in SK-VG are typically context-dependent, requiring supplementary scene knowledge for unambiguous target object localization. The traditional VG task exhibits significant differences from natural human referring habits when applied to human–computer interaction scenarios. On one hand, VG employs relatively cumbersome descriptive referring expressions, while user interactions tend to be more abstract and concise. On the other hand, traditional VG tasks typically fail to achieve fluent contextual interaction due to the lack of scene knowledge. Building upon the traditional grounding task, SK-VG formulates its task to be more oriented toward practical human–computer interaction scenarios, thereby addressing these limitations.

As the queries in SK-VG are typically insufficient to unambiguously localize the target object in the image, the object detection process exhibits strong dependency on knowledge comprehension and reasoning. Particularly, since the queries in SK-VG seldom directly describe the visual attributes of the targets, the model usually has to execute a multi-step reasoning chain based on the scene knowledge, such as the following: (1) identifying potential candidates, (2) determining their appearance features, and (3) localizing the target in the image. This process presents significant challenges for models, demanding sophisticated long-chain reasoning capabilities across different modalities.

Previous research has achieved remarkable success in conventional grounding tasks [2,3,4,5,6,7]. For instance, Grounding DINO incorporates language modeling into closed-set detectors to enable open-set concept generalization [2], while unified frameworks like UNINEXT and OFA demonstrate outstanding performance across various grounding tasks [3,4]. However, these models primarily focus on grounding capabilities while remaining deficient in knowledge understanding and referencing, leading to suboptimal target localization under the information-insufficient conditions inherent in SK-VG tasks. Although recent models specifically trained for SK-VG such as KeViLI and LeViLM enhance information fusion and feature matching between scene knowledge, images, and queries, they still exhibit limitations in handling deeper-level reasoning processes [1].

Compared to traditional models, Vision–Language Models (VLMs) support flexible input formats and demonstrate superior scene knowledge comprehension and reasoning capabilities due to extensive textual pretraining [8,9,10,11,12]. Their multi-modal alignment proficiency, acquired through training on image–caption pairs [13], further endows them with significant advantages in SK-VG tasks. Nevertheless, VLMs suffer from hallucination issues during reasoning processes, particularly in multi-hop or long-chain scenarios [14,15]. As shown in Figure 1a, directly employing a single VLM to address the SK-VG problem requires the model to internally execute a reasoning chain with multiple discrete stages of semantic analysis and logical deduction before the eventual grounding operation. Such a process, i.e., compressing an interdependent reasoning pipeline into an end-to-end step, manifests in inaccurate contextual relationship capture and consequential referential confusion. This performance bottleneck becomes more pronounced when compact backbones with limited model capacity are used.

In this paper, we propose SplitGround, a novel framework for the SK-VG task that synergistically integrates knowledge reasoning and visual grounding through hierarchical decomposition of complex inference chains leveraging the advantages of VLMs. Our approach establishes a collaborative multi-expert architecture where specialized agents handle modularized subtasks through atomic reasoning operations. This systematic decomposition addresses long-chain reasoning bottlenecks by shortening inference pathways while enhancing system reliability through transparent intermediate processes, thereby improving both detection accuracy and interpretability.

Fundamentally, the inherent gap between queries and images in SK-VG arises from the ambiguous alignment between context-dependent referring expressions and the visual content. To bridge this gap through knowledge integration, SplitGround hierarchically decomposes the task into two layers (left/right part in Figure 1b). The lower layer performs knowledge-enhanced processing of raw inputs: it resolves query–image semantic discrepancies by transforming both modalities into knowledge-anchored representations. This refinement substantially reduces cross-modal misalignment, enabling the upper layer to execute precise visual grounding through direct matching between the two representations, thus operating within a minimized semantic gap environment.

To validate the effectiveness of our approach, we conduct extensive experiments on the SK-VG dataset [1]. Results demonstrate that SplitGround achieves state-of-the-art performance through training-free implementation, attaining an accuracy of 72.99% on the test set, surpassing the fully supervised best-performing model LeViLM by 0.42%. Compared with the single VLM model used in SplitGround [12], our accuracy has increased by 3.89%. Furthermore, the modular architecture of SplitGround demonstrates broad adaptability. It achieves generalized performance improvements across diverse VLMs while maintaining particularly strong effectiveness in lightweight implementations.

The contributions of our work can be summarized as follows:

• We propose SplitGround for SK-VG tasks, which is a training-free solution that harnesses the cross-modal grounding and knowledge reasoning capabilities of VLMs.
• We design a novel framework featuring multi-expert collaboration for fine-grained image annotation and referring expression conversion. This decomposition of complex long-chain reasoning into stepwise processes enhances overall grounding performance.
• Comprehensive experiments on the SK-VG dataset demonstrate that our training-free approach establishes new SOTA results, significantly outperforming previous baselines. Simultaneously, experimental analyses validate the effectiveness of the inherent design and highlight the adaptability of the proposed method.

2. Related Work

In Section 2.1 and Section 2.2, we briefly review existing visual grounding and SK-VG methods, respectively. The two subsections summarize the representative studies in recent years and analyze the limitations of current approaches. Then, in Section 2.3, from the methodology perspective, we investigate techniques that are related to SplitGround, focusing on the application paradigms of VLMs in multi-modal scenarios.

2.1. Visual Grounding

Visual grounding is a multi-modal task that aims to align the referred object in the query with specific regions in images. A classical categorization divides visual grounding models into two-stage [16,17,18,19,20,21] and one-stage methods [22,23,24,25,26,27]. Two-stage approaches typically first generate candidate regions from the image and subsequently evaluate these regions based on textual descriptions. As a representative example, Mao et al. [28] fine-tuned VGGNet to generate candidate bounding boxes, which were then matched with textual features extracted by LSTM [29]. Chen et al. [30] proposed the Ref-NMS module to enhance generated proposals using expression information. Such two-stage methods offer straightforward implementation and strong interpretability. On the other hand, one-stage methods perform dense prediction on fused multi-modal features. A representative example is FAOA proposed by Yang et al. [24], which injects text embeddings together with spatial information into YOLOv3 [31], enabling efficient and accurate region prediction.

In addition to the traditional taxonomy mentioned above, the recent prevalence of transformer-based architectures has established a new category which utilizes encoder–decoder frameworks [32,33,34,35,36,37,38,39,40]. Pioneering works like TransVG [26] introduced a [REG] token to fuse vision–language features, achieving state-of-the-art performance with a concise structure. This has inspired numerous DETR-like [41] end-to-end detectors, including the current SOTA models in the DINO series [2,42,43,44]. Concurrently, utilizing vision–language pretraining (VLP) models like CLIP [45] as the image/text encoder has become mainstream in visual grounding [45,46,47,48,49]. Leveraging such foundation models with their powerful generalization capabilities, researchers have pursued data-efficient optimization strategies for VG tasks, including zero-shot/few-shot adaptation [50] and unsupervised or weakly supervised training paradigms [51,52,53]. Meanwhile, researchers also explore open-vocabulary detection tasks based on the conventional VG task [2,50,54,55], making it more useful in real-world scenarios.

However, even though traditional VG models already demonstrate remarkable grounding capabilities, with some models even unifying various multi-modal tasks [3,4,56], they still exhibit limitations in knowledge comprehension and struggle to achieve effective alignment between knowledge and other modal information, resulting in suboptimal performance on SK-VG.

2.2. Scene Knowledge-Guided Visual Grounding

In recent years, researchers have also expanded the traditional VG task from the perspective of knowledge enhancement, proposing the Scene Knowledge-guided Visual Grounding (SK-VG) task [1]. This task employs knowledge to guide the detection process, enabling the VG framework to accommodate more diverse expression forms, thereby enhancing its practicality for user applications. In an SK-VG task, the insufficient information contained in queries and images often proves inadequate for locating target regions. Compared with a conventional VG task, SK-VG places greater emphasis on the knowledge comprehension and reasoning capabilities of the models, highlighting the crucial role of knowledge for precise grounding. Representative approaches for SK-VG include KeViLI and LeViLM. KeViLI adopts a one-stage approach utilizing transformer layers [57] for multi-modal information extraction and fusion. LeViLM builds upon the pretrained GLIP [55], integrating queries and scene knowledge through prompt engineering and then achieving alignment and fusion of visual and textual features for region-level localization.

Existing SK-VG models have attained basic knowledge understanding capabilities, yet still show notable deficiencies in long-chain reasoning and leave room for improvement on hard samples. To address these issues, our proposed SplitGround framework leverages VLMs [8,12] for knowledge comprehension and processing, adopting a multi-agent collaborative architecture that decomposes complex reasoning into multiple simpler subtasks. We present comparative results between our model and other approaches in Section 4.2.

2.3. VLMs for Detection Tasks

VLMs possess robust multi-modal comprehension and reasoning capabilities, and have gained significant attention in multi-modal detection tasks such as visual grounding or other applied tasks [15,58,59]. Compared to traditional small-scale models, VLMs often achieve remarkable performance in multi-modal tasks, including higher accuracy and less need for additional training. Some works specifically optimize VLM foundation models for grounding tasks [5,55,60,61,62,63]. For instance, LLaVA-Grounding [60] proposed by Zhang et al. integrates the grounding and chat functionality by incorporating external models to create a specialized VLM. CogVLM [62] bridges the gap between pretrained language models and image encoders by introducing a trainable visual expert module, providing a powerful foundational model. Other works employ LLMs and VLMs as agents to create workflows or implement multi-round conversations to achieve task planning, tool invocation, image processing, and multi-modal understanding [64,65,66,67,68,69]. For example, Zhao et al. designed three agents in LLM-Optic [68] based on VLMs, responsible for query refinement, candidate bbox annotation, and bbox selection. Li et al. leveraged the open-vocabulary understanding capability of VLMs in SeeGround [69] to conduct landmark parsing, perspective selection, and image rendering, introducing a novel zero-shot 3D visual grounding model.

In SplitGround, we also adopt a multi-agent collaboration framework, designing diverse VLM-based agents. To accommodate the knowledge-driven nature of the SK-VG task, all agents in SplitGround perform stepwise reasoning by referencing knowledge content. Through agentic workflow collaboration, the framework decomposes the complex task and integrates knowledge into queries and images, thereby reducing the reasoning burden on the grounding process.

3. Method

3.1. Overview

The Scene Knowledge-guided Visual Grounding (SK-VG) task requires the model to localize the referred object in an image I by interpreting a context-dependent query Q under the guidance of scene knowledge K. Formally, this task can be formulated as:

$$\mathcal{B}={\mathcal{F}}_g(Q,I,K),$$

(1)

where $\mathcal{B}\in {\mathbb{R}}^4$ denotes the predicted bounding box, and ${\mathcal{F}}_g(·)$ represents the grounding model. Unlike conventional VG tasks, SK-VG inherently requires scene knowledge mediation to address the fundamental gap between context-dependent queries and visual content. This gap necessitates multi-step inference such as entity resolution, appearance description, and visual localization, which introduces significant challenges.

To address this challenge, as shown in Figure 2, we propose SplitGround, a hierarchical framework that splits sequential multi-step reasoning into single-step operations via multi-expert collaboration. Specifically, we utilize a multi-layer framework in order to bridge the fundamental gap of SK-VG. The lower layer employs specialized modules to transform raw inputs into knowledge-anchored representations: the image I is enriched with spatially grounded entity annotations $I^{\prime }$ while the query Q undergoes lexical normalization to distill contextual references into canonical noun phrases $Q^{\prime }$. This refinement enables direct contextual alignment under shared knowledge K, effectively reducing explicit knowledge dependency during subsequent processing. In the upper layer, the grounding agent operates on these pre-aligned representations $Q^{\prime }$ and $I^{\prime }$, where the simplified semantic matching mitigates error propagation across reasoning steps while enhancing localization precision through compressed cognitive processing.

Specifically, in the lower layer, the two core modules of SplitGround for processing inputs are as follows:

• Agentic Annotation Workflow (AAW) (Section 3.2): Integrates scene knowledge into visual representations by annotating entities on I.
• Synonymous Conversion Mechanism (SCM) (Section 3.3): Translates context-dependent queries Q into entity-centric expressions using scene knowledge K.

Formally, the framework can be formulated as:

$$\begin{array}{cc}\hfill Q^{\prime }& ={\mathcal{F}}_{SCM}(Q,K),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill I^{\prime }& ={\mathcal{F}}_{AAW}(I,K),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \mathcal{B}& ={\mathcal{F}}_g(Q^{\prime },I^{\prime },K),\hfill \end{array}$$

(4)

where ${\mathcal{F}}_{SCM}(·)$ and ${\mathcal{F}}_{AAW}(·)$ denote the SCM and AAW, respectively. ${\mathcal{F}}_g(·)$ employs an off-the-shelf VLM, acting as our grounding backbone. By leveraging VLMs in a training-free paradigm, SplitGround capitalizes on their zero-shot capabilities in cross-modal understanding and localization.

3.2. Agentic Annotation Workflow

AAW implements an annotation–inspection mechanism to explicitly mark entities from knowledge text K on image I. This entity-aware annotation creates direct correspondences between contextual information and visual contents, providing direct visual cues that simplify subsequent grounding operations. In order to obtain more accurate annotations, the framework primarily involves three agents:

Named Entity Recognizer: Extracts human entities from K:

$${\mathcal{E}}_{\mathrm{human}}={\left\{e_i\right\}}_{i=1}^N={\mathcal{F}}_R\left(K\right),$$

(5)

where ${\mathcal{F}}_r(·)$ identifies person-type entities.

Annotator: Labels entities on I using point annotations:

$$L_A={\left\{(x_i,y_i,e_i)\right\}}_{i=1}^N={\mathcal{F}}_A({\mathcal{E}}_{\mathrm{human}},I,K),$$

(6)

where $(x_i,y_i)$ indicates the location coordinates of entity $e_i$. Compared to the form of bbox, point annotations are better for precise grounding as shown in the experiments. More details will be discussed in Section 4.3. Furthermore, simultaneous annotation of multiple entities enables the model to maintain a holistic perspective, thereby better capturing inter-entity relationships and relative spatial configurations.

Inspector: Inspects and adjusts annotations given by Annotator, as well as gives feedback on omitted entities:

$$L_I,F_{omit}={\mathcal{F}}_I(L_A,I,K),$$

(7)

where $L_I$ indicates the annotations after inspecting and $F_{omit}={\{e_j,des{c}_j\}}_{j=1}^{omitNum}$ indicates the feedback that includes omitted entities and the descriptions of them. The Inspector operates through two primary mechanisms. First, it verifies and rectifies the correspondence between existing coordinates and labels—preserving correct annotations, modifying erroneous labels, and deleting meaningless coordinates. Second, it identifies omitted entities and provides descriptive feedback about their visual characteristics to facilitate subsequent supplementary annotation.

To address potential spatial relation confusions and improve the quality of VLM-based annotations, an additional annotation–inspection process is implemented. The complete workflow is outlined in Algorithm 1. The second-round annotation incorporates results from the initial round and Inspector feedback, while the final inspection validates newly added annotations by retaining correct ones and removing errors. We adopt a two-round refinement strategy that balances efficiency and accuracy, as our empirical results demonstrate diminishing benefits beyond two iterations. A detailed discussion is conducted in Section 4.3.

Algorithm 1 Agentic Annotation Workflow

Input: Image I, Scene knowledge K Output: Annotated image $I^{\prime }$  1: $\mathcal{E}\leftarrow \mathrm{E}\mathrm{X}\mathrm{T}\mathrm{R}\mathrm{A}\mathrm{C}\mathrm{T}\mathrm{E}\mathrm{N}\mathrm{T}\mathrm{I}\mathrm{T}\mathrm{I}\mathrm{E}\mathrm{S}{\left(K\right)}_{\mathrm{type}=“\mathrm{Person}”}$  2: $I_{initial}\leftarrow \mathrm{A}\mathrm{N}\mathrm{N}\mathrm{O}\mathrm{T}\mathrm{A}\mathrm{T}\mathrm{E}(I,\mathcal{E},K)$  where $I_{initial}$ is annotated by points $\{e_i:(x_i,y_i)|e_i\in \mathcal{E}\}$  3: $I_{corrected},F\leftarrow \mathrm{I}\mathrm{N}\mathrm{S}\mathrm{P}\mathrm{E}\mathrm{C}\mathrm{T}(I_{initial},K)$  where $\mathrm{Feedback}F=\left\{(e_{omit},{desc}_{omit})\right\}$  4: if  $F=\varnothing$ then  5:    $I^{\prime }\leftarrow I_{corrected}$  6: else  7:    $I_{final}\leftarrow \mathrm{A}\mathrm{N}\mathrm{N}\mathrm{O}\mathrm{T}\mathrm{A}\mathrm{T}\mathrm{E}(I_{corrected},F,K)$  8:    $I^{\prime }\leftarrow \mathrm{I}\mathrm{n}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}{(I_{final},K)}_{\mathrm{scope}=\mathrm{new}\mathrm{annotations}}$  9: end if

Throughout this workflow, multi-agent collaboration through information interaction enables autonomous image annotation based on scene knowledge, effectively reducing grounding model dependencies on scene knowledge and complex reasoning.

3.3. Synonymous Conversion Mechanism

SCM is designed to reformulate input queries by leveraging large language models (LLMs) for semantic comprehension and reasoning. It systematically converts context-dependent referring expressions into shortest entity-centric representations by recognizing entities in the knowledge and resolving synonym variations, thereby reducing cognitive complexity for downstream grounding tasks. Formally, given an input query q, the transformation is defined as:

$$Q^{\prime }={\mathcal{F}}_{\mathrm{SCM}}(Q,K)=\left\{\begin{array}{ccc}\mathrm{Name}\left(e\left(Q\right)\right)\hfill & \hfill & \mathrm{if}e\left(Q\right)\in {\mathcal{E}}_{\mathrm{named}}\hfill \\ \mathrm{the}\left[\mathrm{type}\right]:Q\hfill & \hfill & \mathrm{otherwise}\hfill \end{array},\right.$$

(8)

where ${\mathcal{E}}_{\mathrm{named}}$ represents person-type entities with predefined names in K, and $e\left(Q\right)$ extracts the entity that query Q refers to.

SCM resolves the entity referenced in the original query through a pre-designed pipeline where the queries are converted into standard forms. Specifically, if an entity corresponds to a person with a predefined name in the scene knowledge K, SCM reformulates the query into its corresponding entity name. For other entities (e.g., unnamed persons or objects), their types (person or object) are explicitly prepended to the query in the format “the [type]: [query]”, where [type] denotes the entity category. For instance, the query “The manager of Caleb’s rival company” might be transformed into “Jackson”, while the query “the black car in front of William” will be transformed into “the object: the black car in front of William”.

This standardized naming can also align with the labels annotated by AAW on the image, enabling the grounding model to directly localize specified entities via name-to-position mapping, thereby reducing the reasoning complexity of the final grounding process.

4. Experiments and Discussion

We conduct extensive experiments on the SplitGround framework. After explaining the basic experimental configurations in Section 4.1, we compare the performance of SplitGround on the SK-VG dataset with existing SK-VG models and representative traditional VG models in Section 4.2. Additionally, we conduct comprehensive tests on the performance of VLMs with different scales before and after deploying the SplitGround framework, followed by a systematic analysis of the observed patterns. Section 4.3 presents ablation studies on key modules and design components to demonstrate the effectiveness of our design in SplitGround. Finally, Section 4.4 provides qualitative case studies through typical examples to visualize the effectiveness of our framework. Additional analytical experiments are provided in Appendix B.1.

4.1. Configurations

Dataset. The SK-VG dataset [1] contains 4000 frames extracted from movie scenes, partitioned into training (60%), validation (20%), and test (20%) sets. Most frames maintain a resolution width of 1920 pixels. Each image is supplemented with two different textual accounts serving as scene knowledge annotations. These sentences combine precise visual observations about depicted characters with non-visual contextual metadata, such as names, identities, and social connections.

Every piece of scene knowledge is associated with five distinct query–box pairs. These queries exhibit linguistic diversity across hundreds of referenced objects while maintaining strict referential uniqueness to specific image elements. The test set is systematically categorized through dual taxonomies: (1) bbox scale (small/medium/large) using size thresholds of 64 × 64 and 128 × 128 pixels, and (2) knowledge dependency (easy/medium/hard). In most easy cases, the referent can be discerned through visual features described in the query. Conversely, hard cases necessitate knowledge-based reasoning to resolve referential ambiguities, where the absence of such contextual information significantly makes it difficult to distinguish or recognize the referent. This hierarchical evaluation framework enables a multi-dimensional assessment of the model’s capabilities.

An input example with three queries of different levels are shown in Figure 3.

Metrics. The evaluation adheres to established guidelines of top-1 accuracy (%) [26]. Predictions are classified as correct when the intersection-over-union (IoU) ratio between the predicted bounding box and the ground truth exceeds 0.5.

Implementation Details. Because SplitGround adopts a training-free paradigm (i.e., VLMs are not fine-tuned on the SK-VG dataset to preserve their generalization capabilities and reduce the training cost), all experiments are conducted on the test set of the SK-VG dataset. Considering that Qwen2.5-VL [12] exhibits superior grounding capabilities while GPT-4o [8] excels in cross-modal reasoning tasks [12,70], in the AAW module, GPT-4o [8] serves as the Recognizer and Inspector to handle reasoning and verification, while Qwen2.5-VL-72B-instruct acts as the Annotator for knowledge-based image annotation. For the relatively simpler reasoning tasks in the SCM module, the GPT-4o-mini model is utilized. And we employ Qwen2.5-VL-7B-instruct as the primary model for grounding tasks. The temperature of all models is set to 0.1 during execution.

When evaluating traditional VG models, since they do not support triplet input formats containing scene knowledge (i.e., image–query–knowledge triplet), we test two alternative input formats: the original image–query pairs and the merged queries formatted as “query: {Q} \n knowledge: {K}”. The VG models exclusively leverage pretrained visual grounding capabilities, with no fine-tuning or training conducted on SK-VG domain data.

Detailed prompts for each module and control groups are provided in Appendix A.

4.2. Comparative Study

Comparisons to the State of the Art.

Table 1. Comparative results on the test set of the SK-VG dataset [1]. All scores are given in percentage (%). The best and 2nd best scores are highlighted in bold and underlined.

Method	Venue	Model Task	w/o Training	Input	Overall Acc	Difficulty Level
						Easy	Medium	Hard
TransVG [26]	ICCV’21	VG	✓	$Q,I$	23.29	23.32	22.87	23.71
				$Q,K,I$	21.05	19.22	21.01	24.28
MDETR [71]	ICCV’21	VG	✓	$Q,I$	31.52	37.29	29.60	23.54
				$Q,K,I$	20.48	17.24	20.79	25.77
OFA [4]	ICML’22	multi-task	✓	$Q,I$	35.33	41.91	31.95	27.44
				$Q,K,I$	24.89	23.88	25.05	26.46
UNINEXT(H) [3]	CVPR’23	multi-task	✓	$Q,I$	41.86	54.29	35.18	27.27
				$Q,K,I$	20.96	16.31	22.05	27.90
Florence-2(L) [56]	CVPR’24	multi-task	✓	$Q,I$	34.63	46.53	31.29	17.45
				$Q,K,I$	0.02	0.00	0.00	0.06
Grounding DINO(B) [2]	ECCV’24	open-set	✓	$Q,I$	29.51	46.04	20.24	10.51
		detection		$Q,K,I$	1.44	1.23	2.04	1.34
KeViLI [1]	CVPR’23	SK-VG	✗	$Q,K,I$	30.01	33.75	26.55	27.14
LeViLM [1]	CVPR’23	SK-VG	✓	$Q,K,I$	7.55	13.08	4.38	1.26
			✗		72.57	84.08	65.52	59.95
SplitGround	Ours	SK-VG	✓	$Q,K,I$	72.99(+0.42)↑	73.15(−10.93)↓	70.19(+4.67)↑	75.66(+15.71)↑

demonstrates the merits of SplitGround against traditional VG models and other SK-VG models with various training strategies. The results reveal critical insights about model performance in SK-VG tasks. First, conventional VG models exhibit limited accuracy, with best-performing architectures UNINEXT(H) [3] achieving merely 41.86%. Notably, these models all demonstrate distinct performance degradation when handling the $Q,K,I$ input form compared to the $Q,I$ input, similar to that in the VG task, whether they specialize in grounding tasks (“VG” in the table) or are applicable to multiple multi-modal tasks (“multi-task” in the table). Despite extensive pretraining on VG datasets, these models exhibit persistent underperformance in knowledge-intensive SK-VG scenarios, revealing a critical mismatch between their abilities and the contextual reasoning demands in this task.

Meanwhile, our framework achieves a substantial breakthrough, attaining an accuracy of 72.99%—surpassing even the best-performing baseline LeViLM, which is pretrained on SK-VG datasets and obtained an overall accuracy of 72.57%. This margin demonstrates the superiority of SplitGround in this task.

Further analysis across varying difficulty levels reveals the advantages of SplitGround. Even though many of the other tested models exhibit marked performance degradation with increasing difficulty, SplitGround maintains consistent accuracy across the three levels. Notably, in hard samples that require multi-hop reasoning, it achieves an accuracy of 75.66%, demonstrating an absolute improvement of 15.71% over the suboptimal approach. These comparative results empirically validates the effectiveness of our multi-expert collaborative architecture, where the decomposition of long reasoning chains enables robust handling of complex cognitive processes that overwhelm other monolithic models.

Enhancement to VLMs. We further investigate the effectiveness of SplitGround in enhancing the performance of the original VLMs on the SK-VG task. Experiments are conducted using Qwen2.5-VL models with 3B, 7B, and 32B parameter scales, and the results are illustrated in Figure 4. In the bar chart, light colors (left) correspond to the results of the original VLMs, while dark colors (right) represent the performance after being enhanced by SplitGround. The line graph shows the trend of improvement as the model scale increases.

A clear observation from the figure is that SplitGround improves the accuracy of VLMs to varying degrees. This suggests that the shortened reasoning chains generated through precise annotation and query conversion can effectively assist VLMs in their overall reasoning process, ultimately leading to higher accuracy on grounding tasks. Notably, such improvements are particularly pronounced for smaller, lightweight VLMs with limited capacity and relatively weaker reasoning abilities. For the 3B-parameter model, grounding accuracy improves dramatically from 65.67% to 71.22% (5.55% absolute improvement). Moreover, mid-sized 7B models achieve more moderate yet significant improvement from 69.10% to 72.99% (3.89% absolute improvement). In contrast, larger VLMs, which inherently possess stronger reasoning capabilities, are to some extent sufficient to handle cross-modal long-range reasoning processes independently, resulting in a reduced reliance on the annotation information and reasoning chain simplification provided by the SplitGround framework. Consequently, the improvement gains diminish as the model scale increases (from 72.29% to 75.02% in the 32B model). This progressively narrowing trajectory empirically establishes a negative correlation between the effectiveness of the SplitGround framework and model scale, validating that the reasoning-chain simplification in SplitGround optimally compensates for capability gaps in compact architectures while remaining compatible with larger models.

4.3. Ablation Study

Module-Level Ablation. Using the Qwen2.5-VL-7B-instruct model as the backbone, we conduct an ablation study on SplitGround over the test set to analyze the contributions of individual modules. The experimental results, shown in Figure 5, reveal that compared to the VLM itself, integrating the AAW module and SCM module individually improves accuracy by 2.06% and 0.85%, respectively, with the AAW module yielding a larger enhancement. Furthermore, combining the SCM module with the AAW-enhanced VLM achieves an additional 1.83% accuracy gain, surpassing the performance of integrating SCM alone with the original VLM. This improvement can be attributed to the fact that the SCM-converted queries, transformed into minimal proper noun forms, align directly with the image annotations generated by AAW. Such synergy between the image annotation expert and the query conversion expert highlights the advantages of the collaborative multi-expert paradigm in SplitGround.

To further analyze the functional mechanisms of the two modules, we conduct thorough experiments across easy, medium, and hard difficulty levels, with quantitative results detailed in

Table 2. Module-level ablation results at various difficulty levels. The best scores are highlighted in bold.

Method	Difficulty Level
	Easy	Medium	Hard
Original VLM	73.38	65.81	65.10
+AAW	71.63	69.31	72.27
+SCM	74.14	65.65	67.16
SplitGround	73.15	70.19	75.66

. As evidenced in Table 2, the AAW module demonstrates substantial performance gains (+7.17%) on hard cases requiring multi-step reasoning chains. This validates that the image annotation strategy effectively reduces reasoning complexity for intricate tasks. However, AAW causes a slight accuracy drop (−1.75%) on easy cases where a simpler reasoning process is required, likely due to visual distraction introduced by on-image annotations. Meanwhile, the SCM module delivers consistent improvements across easy and hard difficulty levels (+0.76%, +2.06%). Crucially, the collaborative integration of AAW and SCM in SplitGround achieves dual benefits: amplifying performance gains on hard samples (+10.56%) while alleviating the interference of AAW in easy cases.

The difficulty-specific results in Table 2 elucidate the scaling trends observed in Figure 4. SplitGround exhibits more pronounced performance gains on lightweight VLMs, where the AAW annotation effectively compensates for limited inherent reasoning capacity. For larger VLMs with stronger inherent reasoning abilities, the enhancements from AAW and SCM may diminish on hard samples, while the annotation interference of AAW persists in easy cases. This explains the reduced overall improvement margin when scaling up backbone models.

Complementing the quantitative analysis, Figure 6 further provides qualitative visualizations of the operational mechanisms in SplitGround, which will be thoroughly analyzed in Section 4.4.

Submodule-Level Ablation. In

Table 3. Ablation results of different configurations in AAW and SCM. The best acc is highlighted in bold.

Configuration	Module	Test Acc
Default*	-	72.99
One-round	AAW	72.81(−0.18)↓
Three-round	AAW	73.02(+0.03)↑
Bbox	AAW	68.60(−4.39)↓
w/o the [type]	SCM	71.84(−0.97)↓

, we further ablate specific design choices within the AAW and SCM modules. To evaluate the impact of different configurations, we test the following variants:

—

Default*. The configuration adopted by SplitGround in Table 1, serving as the baseline for the experiments in Table 3.

—

One-round. Only a single round of annotation–inspection is performed in AAW.

—

Three-round. The AAW module conducts an additional round of annotation–inspection (three rounds in total).

—

Bbox. The Annotator employs bounding boxes for annotation rather than points.

—

w/o the [type]. The SCM module retains the original query without adding “the [type]:” when name-based transformation is infeasible.

The results in Table 3 demonstrate that the default two-round annotation–inspection process improves accuracy by 0.18% compared to the one-round setting, while achieving comparable performance to the three-round configuration. This indicates that the two-round setting optimally balances effective supplementation of initial annotations with avoidance of redundant cycles.

Other ablated configurations exhibit varying degrees of performance degradation relative to the default. Bbox annotations introduce more visual clutter that may cause attention diversion of the VLM from intrinsic visual content, occasionally reducing the overall accuracy. The explicit type prompting in the SCM module enables precise semantic disambiguation of target categories, providing positive assistance for VLM grounding. These submodule-level ablation results highlight the rationality and effectiveness of the design paradigm in SplitGround.

4.4. Case Study

To visually showcase the effectiveness of SplitGround and the critical role of its multi-expert collaboration mechanism in splitting reasoning chains in different cases, we present a case study in Figure 6 (ground truth, correct prediction, and incorrect prediction bounding boxes shown in yellow, green, and red, respectively; AAW-processed images and SCM-processed queries highlighted in purple and green). Representative samples from different difficulty levels in the SK-VG dataset are selected to compare the grounding results of the original VLM (the top line) and SplitGround (the bottom line), as well as show the effects of the AAW and SCM modules (the middle two rows). This qualitative result can sufficiently align with the previous results shown in Table 2.

For some easy examples (column 1), the baseline VLM demonstrates adequate grounding capability. However, the VLM occasionally confuses grounding targets in some scenarios, as exemplified in column 2 where it misidentifies the carried objects (the glasses) as its human carriers. Our SCM resolves this through type-specific query augmentation (e.g., prepending “the [type]:” prompts), effectively reducing such semantic ambiguities. When processing images containing multiple similar entities (column 3), the VLM often leads to the incorrect target due to its limited discriminative power. The AAW addresses this by visually annotating entity categories, enabling direct visual reference for more precise target identification. Furthermore, the VLM exhibits spatial reasoning limitations (e.g., left–right confusion in column 4), which SplitGround can mitigate through its structured reasoning paradigm.

In summary, the case study highlights that in SplitGround, annotated images explicitly label key entity names, while transformed queries simplify reasoning chains by converting them into forms that align with image annotations. For simple cases, the model can more easily identify the relevant entity in annotated images and localize corresponding objects based on these references. For hard cases, the query conversion significantly reduces reasoning complexity, enabling the model to accurately localize targets in annotated images with minimal effort.

5. Conclusions

In this paper, we present SplitGround, a novel training-free framework for Scene Knowledge-guided Visual Grounding. It fills the most significant gap in existing SK-VG models, namely the challenge that grounding models face when addressing complex long-chain reasoning. By designing expert modules for hierarchical task decomposition, SplitGround effectively integrates knowledge information into the originally uninformative queries and images. Our Agentic Annotation Workflow (AAW) achieved named entity annotation in images through a two-round annotation–inspection process, while the Synonymous Conversion Mechanism (SCM) optimized context-dependent queries to reduce subsequent reasoning costs. These two modules collaboratively split the reasoning chain into multiple sub-chains, significantly simplifying the original lengthy multi-hop reasoning steps and enhancing the accuracy of final grounding results. We conducted extensive experiments on the SK-VG dataset to evaluate the performance of our SplitGround from both qualitative and quantitative perspectives, with a detailed analysis of the inherent mechanism of each component. Experimental results demonstrated that SplitGround outperformed previous methods and achieved competitive performance compared with supervised models.

Further improvements remain possible for SplitGround. From the perspective of structural improvement, the current multi-expert discussion framework still exhibits limitations in real-time performance. Potential enhancements include simplifying the discussion workflow within the AAW module or integrating the SCM and AAW modules. From the perspective of practical application, SplitGround primarily faces real-world human–computer interaction scenarios where context information will typically not be given as a long-text scene knowledge. It might be necessary for the model to learn contextual information in an online pattern. Additionally, the practical effectiveness of SplitGround also requires further validation with more real-world data.