KNA-SG: Keyframe–Node-Associated Open-Vocabulary 3D Scene Graphs from RGB Sequences

Abstract

3D scene graphs organize objects and their relationships in a scene into structured representations, providing an interpretable and queryable foundation for relational reasoning and object grounding. Existing open-vocabulary 3D scene graph construction methods primarily focus on object-level feature representation and open-ended edge reasoning. However, they often lack explicit and retrievable associations between object nodes and keyframes, making it difficult to recall relevant visual evidence for target disambiguation and relationship verification in complex queries. Moreover, pre-constructed edges are inherently limited in their ability to cover the diverse linguistic expressions encountered in downstream tasks. To address these limitations, we propose KNA-SG, a framework for constructing open-vocabulary 3D scene graphs from RGB sequences with explicit keyframe–node associations. Built upon instance-grounded 3D reconstruction, KNA-SG represents each object instance as a graph node and uses a unique instance identifier to associate the node with the keyframes in which the instance is observed. The ID-annotated keyframes guide MLLM-based open-vocabulary semantic parsing, enabling semantic attributes to be assigned to these graph nodes. This design transforms keyframes into retrievable visual evidence for target disambiguation and relationship verification during query reasoning. Verified relationships are further written back into the scene graph as reusable relational memory to support subsequent queries. To ensure the effectiveness of selected keyframes, we design a two-stage keyframe selection strategy that combines visual quality assessment with semantic redundancy removal, preserving a set of clear keyframes that provide comprehensive scene coverage. Experimental results show that KNA-SG outperforms existing methods on open-vocabulary 3D semantic segmentation and 3D object grounding tasks.

1. Introduction

Language-guided 3D object grounding is a representative downstream capability for embodied agents to understand and execute instructions in human-centered environments [1,2,3]. Natural-language instructions typically specify a target object through its attributes and relationships with surrounding entities. Although 3D reconstruction and object detection provide scene geometry and candidate objects, they do not directly support relational reasoning required for complex referring expressions. By representing object instances as nodes and typed relations as edges, 3D scene graphs (3DSGs) provide a compact and queryable scene abstraction for language-guided object grounding [4,5].

Early 3DSG approaches relied on closed-set vocabularies and predefined relation schemas, constraining both node categories and edge types to manually specified label spaces [6,7,8]. Recent advances in foundation models for visual and 3D understanding [9,10,11,12,13] have shifted 3DSG research toward open-vocabulary object representation and open-set relation modeling [4,14,15,16,17]. Existing work on node semantics can be broadly categorized into two lines of research. The first projects open-vocabulary visual features extracted from 2D images, such as contrastive language–image pre-training (CLIP) features, onto 3D representations, including point clouds, voxels, and Gaussian primitives, thereby enabling open-vocabulary object retrieval in 3D scenes. However, the aggregation and compression of features across multiple views may lead to the loss of fine-grained semantic information. The second selects representative views from multi-view observations, extracts object-centric image crops, and uses multimodal large language models (MLLMs) to generate textual descriptions. This approach can provide richer object-level attributes, but its reliance on isolated crops may remove scene-level context that is useful for interpreting object attributes relative to surrounding entities and for inferring inter-object relations.

In parallel, relation modeling has evolved from fixed rules based primarily on geometric heuristics toward more flexible approaches that leverage large language models (LLMs) to integrate object descriptions, 2D observations, and 3D spatial cues for relation inference. These developments have expanded 3DSGs from closed-set symbolic structures into open-vocabulary representations that connect 3D perception with language-based reasoning. Nevertheless, even with open-set relation prediction and precomputed graph edges [4,18], precomputing spatial relations during graph construction presents an inherent trade-off. Exhaustively instantiating pairwise relations can result in quadratic growth in the number of graph edges, many of which may be redundant or irrelevant to downstream tasks, thereby increasing the costs of graph storage, retrieval, and reasoning. Conversely, restricting the graph to a predefined subset of relations makes it difficult to anticipate the diverse and query-specific relations required by open-ended language queries. Open-ended object grounding therefore benefits from query-time relation inference and verification, particularly when the relevant relations depend on visual appearance, scene context, or a query-specific spatial reference.

Such verification requires reusable visual evidence that remains traceable to individual object instances. Recent multimodal 3DSGs have begun to preserve image observations within graph representations, organizing them around co-observed object pairs or hierarchical spatial contexts rather than discarding them after graph construction [19,20]. To support efficient downstream reasoning, the retained evidence should be both compact and informative. Continuous RGB sequences contain redundant content, repeated viewpoints, and frames of uneven visual quality. Keyframe selection can therefore condense these sequences into a representative set of observations while reducing storage and multimodal reasoning costs [21,22,23]. Building on this line of research, explicitly associating individual 3D object nodes with the keyframes in which they are observed provides an object-centric and traceable organization of visual evidence for fine-grained attribute disambiguation and query-time relation verification.

Motivated by these observations, we propose KNA-SG, a training-free open-vocabulary 3D scene graph framework with explicit keyframe–node associations. KNA-SG first selects clear and representative keyframes from an RGB sequence through clarity-aware filtering and feature-based redundancy reduction. The selected keyframes are processed by an instance-aware dense multi-view reconstruction model that jointly estimates scene geometry and cross-view-consistent instance representations. Together, these outputs support 3D object instance recovery and aligned per-view mask extraction. Consistent identifiers are marked on the corresponding object regions, establishing bidirectional associations between each node and the keyframes observing that object. These ID-marked keyframes enable an MLLM to assign open-vocabulary names and fine-grained attributes to their corresponding object nodes. Beyond graph construction, the associated keyframes provide visual evidence for query-time relation verification alongside node semantics and 3D spatial cues. Verified relations are written back as relation memory to avoid repeated inference and support knowledge reuse without exhaustively precomputing all potential relations.

The main contributions of this work are summarized as follows.

• We propose KNA-SG, an open-vocabulary 3D scene graph representation with explicit keyframe–node associations. Consistent instance identifiers establish bidirectional links between reconstructed 3D object nodes and their observations in ID-marked keyframes. This design unifies 3D geometry, open-vocabulary semantics, and traceable visual evidence within the graph.
• A two-stage keyframe selection strategy is developed for constructing compact and reliable visual evidence from continuous RGB sequences. Clarity-aware filtering removes degraded observations, while DINOv2-based feature comparison suppresses redundant viewpoints. The retained keyframes provide clear and representative observations for 3D instance reconstruction, node semantic parsing, and downstream visual reasoning.
• We introduce a query-adaptive relation verification mechanism with relation memory. Instead of exhaustively precomputing pairwise relations, KNA-SG verifies query-relevant relation hypotheses using node semantics, 3D spatial cues, and associated keyframe evidence. The validated relations are written back into the graph, enabling relational knowledge to accumulate and be reused across subsequent queries.

2. Related Work

2.1. 3D Scene Graphs

3DSGs abstract 3D environments into metric–semantic graph structures, representing scene entities as nodes and their relationships as edges. This formulation provides a compact, interpretable, and queryable description of the scene. Traditional approaches rely on closed-set 2D instance segmentation or pre-segmented 3D object instances to construct graph nodes. Armeni et al. [24] introduces a hierarchical representation that organizes buildings, rooms, objects, and camera poses within a unified structure grounded in 3D geometry. Wald et al. [8] predict object categories and inter-object relations from instance-segmented 3D point clouds. SceneGraphFusion [25] incrementally construct 3D semantic scene graphs from RGB-D observations, updating nodes and edges as new frames arrive. In robotics, Kimera [6] integrate hierarchical 3D scene graphs with metric–semantic mapping, while Hydra [7] advances hierarchical scene abstraction for real-time robotic perception. These methods establish the foundations of 3D scene graph construction, although their node semantics and relation types are generally restricted to predefined vocabularies.

Recent studies introduce vision–language foundation models to support open-vocabulary scene representation. ConceptGraphs [4] forms persistent 3D object nodes by associating class-agnostic masks across posed RGB-D observations. Each node aggregates open-vocabulary visual features and object-level captions, while graph edges encode geometric or language-inferred relationships. HOV-SG [15] organizes floors, rooms, and objects within a hierarchical open-vocabulary graph for large-scale object retrieval and navigation. Open3DSG [17] aligns point-cloud graph features with vision–language representations to predict open-vocabulary object classes and open-set relationships. GaussianGraph [16] investigates scene graph generation on 3D Gaussian representations, where Gaussian primitives are grouped into object-level entities. Some methods have also introduced richer relational reasoning and visual context into graphs. BBQ [5] combines metric and semantic relations with deductive graph reasoning for language-guided object grounding. MSGNav [19] stores co-observation images as multimodal relational evidence for navigation reasoning. KeySG [20] augments hierarchical scene graphs with representative room-level keyframes and retrieves task-relevant context through hierarchical retrieval-augmented generation (RAG). Beyond metrically grounded 3DSGs, DyGEnc [26] encodes sequences of textual scene graphs into spatio-temporal representations for question answering. It preserves object-relation structures and temporal dependencies for subsequent LLM reasoning.

Overall, existing 3DSGs have substantially advanced open-vocabulary object representation and structured relational reasoning. Recent studies have begun to retain selected visual observations, while explicit associations between object nodes and their supporting observations, together with query-time verification of task-relevant relations, remain less explored.

2.2. RGB-Based Multi-View 3D Scene Reconstruction

Continuous RGB sequences captured via camera motion in real-world scenarios typically contain redundant perspectives and blurry frames. Classical simultaneous localization and mapping (SLAM) systems select keyframes according to geometric criteria such as camera pose changes, feature overlap, or covisibility structure [21,27]. These criteria allow the retained frames to provide sufficient spatial constraints for camera tracking and structure recovery.

Beyond geometry-driven selection, recent methods increasingly consider visual informativeness and computational efficiency when filtering long video sequences [28,29,30]. KeySG [20] selects representative keyframes at the room level to improve visual coverage and support hierarchical multimodal retrieval. These studies highlight the need to suppress redundant or unreliable observations while preserving a compact set of views for subsequent scene reconstruction and understanding.

Given multi-view RGB observations, recent feed-forward models directly recover scene geometry. DUSt3R [31] formulates dense reconstruction as point-map regression, allowing image pairs to be lifted into a shared 3D coordinate space without requiring a complete classical reconstruction pipeline. MASt3R [32] advances this direction by integrating 3D geometric reconstruction with learnable matching descriptors, ensuring more robust multi-view correspondence under extreme viewpoint changes. SLAM3R [33] extends feed-forward reconstruction to RGB video by combining local reconstruction with sliding-window registration. VGGT [34] further unifies camera parameter estimation, depth prediction, point-map prediction, and point tracking in a transformer-based multi-view framework. These methods make RGB-based geometry recovery more direct and provide a useful foundation for downstream 3D scene understanding.

Geometry alone, however, does not explicitly identify object instances across views. OV3R [35] incorporates open-vocabulary semantics into RGB video reconstruction through 2D–3D semantic enhancement, supporting dense reconstruction and open-vocabulary 3D segmentation. IGGT [36] introduces instance-grounded geometry learning to produce cross-view consistent object instances from RGB inputs. These developments extend RGB-based multi-view reconstruction from geometric recovery toward object-centric scene understanding.

2.3. Prior Work on Language-Guided 3D Object Grounding

Language-guided 3D object grounding aims to identify instances described by natural-language expressions. ScanRefer [2] introduced free-form referring expressions for objects in reconstructed indoor scenes, while ReferIt3D [1] emphasized fine-grained disambiguation among same-category instances through attributes and spatial relations. Early methods commonly generated object proposals before aligning their geometric and semantic features with language. InstanceRefer [37] achieves robust grounding by cooperatively modeling fine-grained target attributes, multi-instance spatial relations, and global scene localization. TransRefer3D [38] leverages a transformer-based architecture to explicitly match visual entities and pairwise spatial relations with linguistic cues through entity-and-relation-aware attention. More recently, zero-shot and open-vocabulary approaches leverage large language models to handle open-ended queries. LLM-Grounder [39] exemplifies this paradigm through LLM-driven query decomposition and modular coordination of off-the-shelf 3D perception tools. SORT3D [40] couples LLM-based sequential reasoning with an object-centric spatial toolbox built on 2D-derived attributes. SeeGround [41] bridges 3D scenes and 2D vision–language models through query-aligned views and spatially enriched textual descriptions.

Scene-graph-based methods provide a persistent object-centric representation for grounding. OVSG [14] parses free-form language into query graphs and grounds entities through subgraph matching. BBQ [5] constructs object-centric 3D scene graphs with metric and semantic spatial edges and performs deductive reasoning for grounding. However, the underlying visual observations are typically compressed into node-level embeddings or textual descriptions, rather than retained as traceable frame-level evidence associated with individual object nodes.

3. Method

3.1. Overview

The overall framework of KNA-SG is shown in Figure 1. Given an RGB sequence, KNA-SG constructs an initial open-vocabulary 3D scene graph that supports downstream language-guided object grounding through query-time reasoning. The construction process begins with a two-stage keyframe selection procedure consisting of clarity screening and global visual feature difference selection. The patch tokens corresponding to the selected keyframes are subsequently processed by IGGT [36] to predict multi-view geometry and dense instance features. Feature clustering is then performed to obtain object-level geometric information and cross-view consistent instance masks. Each instance is assigned a unique ID and associated with the keyframes in which it is observed. The instance IDs are placed on the corresponding keyframes according to the visible masks, producing ID-marked visual evidence. These keyframes are provided to an MLLM together with structured prompts to parse open-vocabulary attributes. The resulting nodes contain 3D geometry, semantic attributes, and retrievable visual evidence, and are organized with the initial physical relations into a sparse 3D scene graph. When a natural-language query is provided, KNA-SG parses it into structured target, anchor, and relation constraints. Candidate nodes are retrieved using the stored semantic attributes and, when spatial constraints are present, further evaluated through geometry-first reasoning. A unique high-confidence candidate may be returned directly without invoking the MLLM. If either semantic retrieval or geometric reasoning remains ambiguous, the associated ID-marked keyframes are used for MLLM-based visual verification. Relations verified during this process are written back to the graph as reusable relational memory for subsequent queries.

The following sections present the main components of KNA-SG in three parts. Section 3.2 introduces the two-stage keyframe selection strategy. Based on the object instances reconstructed by IGGT, Section 3.3 describes keyframe–node association and ID-guided open-vocabulary node parsing. Section 3.4 then presents scene graph initialization and query-time reasoning.

3.2. Two-Stage Keyframe Selection

Long RGB sequences often contain low-quality observations and substantial temporal redundancy. Directly processing all frames increases the computational cost of global multi-view aggregation, while degraded observations may affect instance reconstruction, and highly similar views provide limited additional scene information. In KNA-SG, the selected keyframes are used not only as inputs to the reconstruction backbone but also as persistent visual evidence associated with object nodes for subsequent semantic parsing and query-time verification. To obtain a compact and informative subset of frames, we adopt a two-stage keyframe selection strategy. The first stage performs clarity-aware candidate selection to filter low-quality observations while preserving coarse temporal coverage. The second stage uses the DINOv2 [CLS] token features from the IGGT front end to measure global visual differences between candidate frames, thereby suppressing redundant views.

3.2.1. Clarity-Aware Candidate Selection

The first stage operates directly on the input RGB sequence. Its goal is to construct a temporally distributed candidate set while filtering out frames affected by motion blur or unstable imaging conditions. For sequences acquired under stable conditions, candidate frames can be obtained through interval-based sampling. For unconstrained RGB sequences, the clarity score for each frame $I_t\in W_k$ is defined as

$$Q\left(I_t\right)={\displaystyle \frac{1}{2}}log\left(1+V_{\mathrm{L}}\left(I_t\right)\right)+{\displaystyle \frac{1}{2}}log\left(1+E_{\mathrm{T}}\left(I_t\right)\right).$$

(1)

where $V_{\mathrm{L}}$ is the variance of the Laplacian, and $E_{\mathrm{T}}$ is the Tenengrad energy. The Laplacian term captures high-frequency structural information, while the Tenengrad term reflects gradient strength. Their combination favors frames with clear edges and stable local structures. The input sequence is divided into non-overlapping temporal segments, and the frame with the highest clarity score in each segment is selected as a local candidate:

$$I_k^{*}=arg\underset{I_t\in W_k}{max}Q\left(I_t\right).$$

(2)

Local candidates below a sequence-level relative clarity threshold are discarded, and a minimum temporal interval is further enforced between retained candidates to avoid local oversampling. This procedure filters relatively blurred observations while preserving coarse temporal coverage for subsequent feature-based selection. Figure 2 illustrates an example of clarity-aware selection where blurred observations are excluded while clearer frames are retained.

3.2.2. Global Visual Feature Difference Selection

The second stage reduces redundancy among the candidate frames. It operates on the [CLS] token features produced by the DINOv2 [10] encoder in the IGGT front end, before global multi-view aggregation, and therefore introduces no additional feature extraction network. The [CLS] token is used as a compact descriptor of frame-level global visual content.

For two adjacent frames with descriptors ${\mathbf{f}}_{t-1}$ and ${\mathbf{f}}_t$, their feature difference is defined as

$$d_t=1-\mathrm{sim}({\mathbf{f}}_{t-1},{\mathbf{f}}_t),$$

(3)

where $\mathrm{sim}(·,·)$ denotes cosine similarity. A sliding window is used to estimate the local mean ${\mu }_t$ and standard deviation ${\sigma }_t$ of $d_t$, and the adaptive threshold is defined as

$${\tau }_t=max({\mu }_t+0.5{\sigma }_t,{\tau }_{\mathrm{base}}),$$

(4)

Let ${\mathcal{F}}_{\mathrm{sel}}$ denote the set of descriptors of selected keyframes. For each candidate frame with descriptor ${\mathbf{f}}_t$, its novelty with respect to the selected set is measured as

$${\delta }_t=1-\underset{\mathbf{f}\in {\mathcal{F}}_{\mathrm{sel}}}{max}\mathrm{sim}({\mathbf{f}}_t,\mathbf{f}).$$

(5)

The frame is retained in the final keyframe set only if

$${\delta }_t>{\tau }_t.$$

(6)

This stage keeps frames that provide additional global visual information relative to the selected set, resulting in a compact keyframe set with reduced view redundancy. Since the selection is performed before global multi-view aggregation, the reconstruction backbone processes fewer repeated observations while preserving representative views for instance-aware reconstruction and subsequent scene graph construction. Figure 3 visualizes an example of the selection where candidate frames similar to the selected keyframes are removed and frames with larger feature differences are retained.

The effectiveness of the two-stage strategy is evaluated in Section 4.4. The ablation results show that the selected keyframes improve object-level visibility coverage, especially for small objects, and provide more representative observations for open-vocabulary 3D semantic aggregation under matched frame budgets.

3.3. ID-Guided Open-Vocabulary Node Attribute Parsing

The dense instance features predicted by IGGT are post-processed and clustered to obtain cross-view consistent instance masks. These masks are then applied to the corresponding depth maps to recover the geometry of each instance. KNA-SG subsequently assigns open-vocabulary semantic attributes to each reconstructed object. The semantic parsing module needs to associate language descriptions with the corresponding instances. Directly prompting an MLLM with unmarked keyframes may lead to ambiguous references, especially when multiple visually similar objects appear in the same view. We therefore introduce explicit visual IDs to distinguish instances during MLLM-based node attribute parsing.

For each reconstructed instance, its unique visual ID is overlaid on the corresponding keyframe observations. To place the ID inside the visible instance region, we use the center of the maximum inscribed circle of the instance mask. Given an instance mask $M_i$, let $\partial M_i$ denote its boundary. The ID position $p_i^{*}$ is defined as

$$p_i^{*}=arg\underset{p\in M_i}{max}\mathrm{dist}(p,\partial M_i),$$

(7)

where $\mathrm{dist}(p,\partial M_i)$ denotes the Euclidean distance from pixel p to the mask boundary. This rule selects the point farthest from the visible mask boundary, reducing the risk that the ID is placed outside the object or close to ambiguous boundary regions caused by irregular shapes, partial occlusion, or truncation. Figure 4 compares this strategy with placement at the center of the minimum bounding rectangle.

The ID-marked keyframes are then used as visual prompts for the MLLM. As shown in Figure 5, a structured prompt is designed to guide instance-level semantic parsing with explicit ID references. The prompt specifies the visible ID list and requires the model to assign attributes to each ID consistently across viewpoints. For each visible ID, the MLLM outputs a structured set of attributes, including category, material, color, and function. The output is organized in an ID-indexed format so that each semantic description can be directly assigned to the corresponding reconstructed instance. This design provides an explicit mapping between visual observations and graph nodes.

3.4. 3D Scene Graph Initialization and Query-Time Reasoning

3.4.1. Sparse 3D Scene Graph Initialization

Guided by the parsed semantic attributes, we normalize the reconstructed scene to establish a consistent coordinate system. The floor normal determines the vertical axis and provides a common height reference, while the dominant wall direction defines the horizontal orientation. The reconstructed scene is then transformed into this canonical coordinate system. When an absolute metric scale is required in practical applications, a scene-scale prior is needed to recover the global scale. The normalized coordinate system provides a common reference for computing object-level geometry descriptors and spatial relations.

KNA-SG initializes a 3D scene graph as

$${\mathcal{G}}^{\left(0\right)}=(\mathcal{V},{\mathcal{E}}^{\left(0\right)}),$$

(8)

where $\mathcal{V}$ denotes the set of object nodes and ${\mathcal{E}}^{\left(0\right)}$ denotes the set of initial relation edges. Each node stores its instance identity, open-vocabulary semantic attributes, lightweight 3D geometric descriptors, including its 3D bounding box and centroid, and the indices of the keyframes in which the instance is visible. These keyframe indices are ranked in descending order by the visible mask area of the corresponding instance.

Instead of precomputing dense spatial edges between all object pairs, KNA-SG initializes only a compact set of persistent physical relations, including support, attachment, and hanging. The ID-marked keyframes and the corresponding prompt are provided to the MLLM. Following the predefined relation rules, the MLLM identifies the directly associated object for each non-structural instance. Structural elements, including the floor, walls, ceiling, doors, and windows, are excluded only as relation subjects, and they may still serve as supporting nodes. The proposed relations are subsequently checked using relation-specific geometric cues, including bounding-box extents, relative height, vertical gap, and projected overlap between the involved objects. The relations that satisfy the corresponding geometric conditions form the initial edge set ${\mathcal{E}}^{\left(0\right)}$. Continuous spatial relations are not precomputed densely but are evaluated from the stored node geometry when required by a query.

3.4.2. Structured Query Parsing and Candidate Retrieval

Given a free-form natural-language query q, KNA-SG aims to identify the object node that best satisfies the query: $v\in \mathcal{V}$.

An LLM first parses the query into a structured representation $Q=(t,S,o)$, where t contains the target category and explicitly described attributes, $S={\left\{s_k\right\}}_{k=1}^K$ denotes an ordered sequence of query constraints, including unary target constraints and target–anchor relational constraints. A binary relation step is represented as $s_k=(r_k,a_k)$, where $r_k$ denotes the relation operator and $a_k$ denotes the corresponding anchor description. Relations involving two anchors, such as “between,” are represented as $s_k=(r_k,a_1^k,a_2^k)$. o specifies the observer position and viewing direction when required. For view-dependent expressions, the parsed observer position and viewing direction define the local reference frame in which left–right and front–behind relations are evaluated.

The target description is semantically matched against the open-vocabulary attributes of the nodes in $\mathcal{V}$, yielding the candidate set $C_t$. The LLM is constrained to select existing graph nodes. Each anchor description is processed in the same manner, producing an anchor candidate set $C_a^k$ for relation $r_k$. When the target category is not explicitly stated but the query contains executable geometric constraints, non-anchor object nodes are retained as initial target candidates.

3.4.3. Geometric Reasoning

Given the candidate sets, KNA-SG evaluates each target candidate against both unary and relational geometric constraints. Unary constraints describe properties of the target itself or its location in the scene. For example, comparative expressions such as “the largest table” are resolved by comparing the 3D bounding-box dimensions of all tables, while expressions such as “the table in the middle of the room” are evaluated using the candidate size and its distance to the scene center. The physical relations are queried directly from ${\mathcal{E}}^{\left(0\right)}$. Other spatial relations, including proximity, relative height, direction, and view-dependent relations, are evaluated dynamically from the stored centroids and 3D bounding boxes. The final geometry score of a target candidate $v_i$ is defined as

$$S\left(v_i\right)=\sum _{s_k\in S_u}{s}_k\left(v_i\right)+\sum _{s_k\in S_r}\underset{v_j\in C_a^k}{max}{s}_k(v_i,v_j),$$

(9)

where $S_u$ and $S_r$ denote the unary and relational geometric constraints extracted from S, respectively. ${\mathcal{C}}_a^m$ denotes the candidate anchor set for the m-th relation, and ${\widehat{s}}_{r_m}$ is the normalized score indicating how well $v_i$ and $v_j$ satisfy relation $r_m$.

The geometry-based prediction is

$$v_g=arg\underset{v_i\in {\mathcal{C}}_t}{max}S\left(v_i\right).$$

(10)

If only one candidate remains, it is returned directly. Otherwise, KNA-SG assess whether the geometry ranking provides sufficient evidence. Let $s_1$ and $s_2$ be the highest and second-highest geometry scores. The absolute and relative margin are defined as

$$m=s_1-s_2,\rho ={\displaystyle \frac{m}{max\left(\right|s_1|,ϵ)}}.$$

(11)

The geometry ranking is regarded as ambiguous when the relative margin falls below the threshold, indicating that multiple candidates satisfy the parsed geometric constraints to a similar degree.

3.4.4. Keyframe-Based Visual Verification

KNA-SG invokes keyframe-based visual verification only when complementary visual evidence is required. Specifically, the visual branch is activated when the geometry ranking is ambiguous according to the above margin criterion, when the query is state-aware, or when the target or anchor candidates cannot be reliably resolved from the scene-graph attributes.

When reliable candidates are available, KNA-SG constructs a restricted visual candidate set from the target candidates and retrieves their associated ID-marked keyframes. A restricted visual candidate set $C_v$ is constructed from the highest-ranked geometry candidates. The ID-marked keyframes associated with the selected target and anchor candidates are then retrieved. When available, keyframes jointly observing the target and anchor candidates are preferred because they provide direct visual evidence for the queried relation. For relational queries, keyframes jointly observing the target and anchor candidates are preferred because they provide direct visual evidence for the queried relation. If the target or anchor candidates cannot be reliably resolved, the system expands the visual search to the object IDs appearing in the available ID-marked keyframes and uses scene-level keyframes as contextual evidence.

Let $C_v$ denote the final set of candidate IDs supplied to the MLLM, and let $K_v$ denote the corresponding keyframe set. The visual verification process is expressed as

$$(v_{vl},c_{vl},h_{vl})=\mathrm{MLLM}(q,C_v,K_v).$$

(12)

where $v_{vl}$ is the selected target node, $c_{vl}$ indicates whether the prediction is certain, and $h_{vl}$ indicates whether an explicit relation in the query is visually supported. By restricting the output to object IDs grounded in the supplied keyframes, the MLLM uses visual evidence to distinguish object states, verify appearance descriptions, and resolve ambiguous target–anchor configurations.

The visual prediction is accepted only when the returned ID belongs to $C_v$, the prediction is marked as certain, and any explicit relation in the query is confirmed. If these checks are not satisfied, KNA-SG retains the geometry-based prediction when a valid geometry result is available; otherwise, the query is marked as unresolved.

The confirmed grounding result is further used to incrementally update the scene graph. Specifically, a target–relation–anchor tuple validated during query-time reasoning is added to the relation set. Each newly added edge records whether it is supported by geometric reasoning alone or further verified using the associated keyframes. Query attributes that are confirmed by the available evidence may also be attached to the resolved node. In this way, the sparse scene graph is progressively enriched with relation and attribute evidence obtained during grounding.

4. Experiments

4.1. Datasets and Experimental Setup

Experiments were conducted on the Replica [42] and ScanNet [43] datasets. Replica provides realistic indoor 3D scene reconstructions ranging from individual rooms to larger office environments, while ScanNet contains instance-level indoor RGB-D data acquired in unconstrained real-world scenes. These two datasets together support the evaluation of the proposed framework under both relatively clean observations and more challenging capture conditions. For consistency with prior studies and to ensure fair comparison, experiments on ScanNet were performed on scene0011_00, scene0030_00, scene0046_00, scene0086_00, scene0222_00, scene0378_00, scene0389_00, and scene0435_00. Experiments on Replica were conducted on office0–office4 and room0–room2.

For language-guided 3D object grounding, we use Nr3D and Sr3D datasets [1], which provide referring expressions grounded in ScanNet scenes. Nr3D provides natural human-written object descriptions, whereas Sr3D provides synthetically generated spatial referring expressions with more controlled relational structures. Following prior 3DSG-based grounding studies, the evaluations are conducted on the same ScanNet scene subset listed above.

We evaluate KNA-SG on open-vocabulary 3D semantic segmentation, language-guided 3D object grounding, and ablation studies of the key design components. All experiments were carried out on a server equipped with an Intel Xeon Gold 6132 CPU, 256 GB RAM, and one NVIDIA A100 GPU with 40 GB memory. Qwen3-VL-8B [44] was used as the multimodal perception model on ID-tagged keyframes. It performs open-vocabulary attribute parsing for object nodes and infers initial support relations between visible instances during sparse scene graph initialization. And Qwen3-4B was used as the language reasoning model for target localization, including query parsing and constrained matching between query phrases and the node vocabulary.

4.2. 3D Open-Vocabulary Semantic Segmentation

We evaluate the quality of the constructed semantic scene representation using open-vocabulary 3D semantic segmentation. Following prior work, we adopt mean accuracy (mAcc), mean intersection over union (mIoU), and frequency-weighted mIoU (F-mIoU) as evaluation metrics. As shown in

Table 1. Quantitative comparison of open-vocabulary 3D semantic segmentation on Replica and ScanNet.

Methods	Replica			ScanNet
	mAcc	mIoU	F-mIoU	mAcc	mIoU	F-mIoU
ConceptGraphs [4]	0.36	0.18	0.15	0.52	0.26	0.29
HOV-SG [15]	0.30	0.23	0.39	0.47	0.30	0.37
Open3DSG [17]	0.35	0.31	0.45	0.52	0.36	0.31
BBQ-CLIP [5]	0.38	0.27	0.48	0.56	0.34	0.36
GaussianGraph [16]	0.49	0.31	-	0.49	0.31	-
Ours	0.51	0.39	0.69	0.58	0.44	0.50

, our method achieves the best overall performance across all metrics on both datasets. On Replica, our method obtains an mIoU of 0.39 and an F-mIoU of 0.69, outperforming the strongest baseline scores of 0.31 and 0.48, respectively. The larger gain in F-mIoU indicates that the constructed semantic representation provides more consistent predictions for spatially dominant regions. On ScanNet, our method achieves an mIoU of 0.44 and an F-mIoU of 0.50, compared with the best baseline scores of 0.36 and 0.37. This shows that the improvement is maintained in real-world indoor scenes with noisier observations and more complex layouts.

Compared with previous open-vocabulary methods, the improvement is observed not only in region-level overlap but also in category-level prediction. For example, our method improves mAcc from the best baseline score of 0.49 to 0.51 on Replica and from 0.56 to 0.58 on ScanNet. These results suggest our framework improves both semantic coverage and spatial consistency in the constructed 3D scene representation. The performance gain is related to the object-centric perception design of KNA-SG. The two-stage keyframe selection reduces redundant and low-quality observations before multi-view aggregation while retaining representative views for semantic propagation. In addition, ID-guided visual prompting provides explicit object references for MLLM-based node parsing, reducing ambiguity when multiple similar instances appear in the same scene.

4.3. Language-Guided 3D Object Grounding

We evaluate whether the constructed open-vocabulary 3D scene graph can support language-guided 3D object grounding. This evaluation measures the ability of the proposed representation to perform semantic filtering and relation-aware target grounding under natural-language queries. Following standard 3D referring expression grounding protocols, we report grounding accuracy under 3D bounding-box IoU thresholds, A@0.1 and A@0.25. A prediction is considered correct when the IoU between the predicted target box and the ground-truth box exceeds the corresponding threshold. Results are reported under the overall, easy, hard, view-dependent, and view-independent settings to assess different levels of linguistic and spatial complexity.

As shown in

Table 2. Language-guided 3D object grounding results on the Nr3D dataset. A@0.1 and A@0.25 denote grounding accuracy under 3D bounding-box IoU thresholds of 0.1 and 0.25, respectively.

Methods	Overall		Easy	Hard	View Dep.	View Indep.
	A@0.1	A@0.25	A@0.1	A@0.1	A@0.1	A@0.1
ConceptGraph [4]	16.0	7.2	18.7	9.1	12.7	17.0
GaussianGraph [16]	17.2	8.6	20.7	10.9	-	-
BBQ [5]	28.3	19.0	30.5	22.8	23.6	29.8
Open3DSG [17]	31.4	22.5	32.3	25.2	24.3	33.5
KNA-SG	43.8	29.2	50.4	26.9	37.6	45.7

, KNA-SG achieves the best overall performance among the compared methods on Nr3D, reaching 43.8% under A@0.1 and 29.2% under A@0.25. Compared with Open3DSG [17], our method improves overall A@0.1 from 31.4% to 43.8% and A@0.25 from 22.5% to 29.2%. The improvement is also observed across all evaluated splits. In particular, KNA-SG achieves 50.4% on the easy split and 37.6% on the view-dependent split under A@0.1, indicating that explicit node attributes and query-time relation reasoning are effective for both direct object descriptions and expressions involving observer-dependent spatial cues. Figure 6 presents qualitative examples of language-guided 3D object grounding on Nr3D.

Table 3. Language-guided 3D object grounding results on the Sr3D dataset. A@0.1 and A@0.25 denote grounding accuracy under 3D bounding-box IoU thresholds of 0.1 and 0.25, respectively.

Methods	Overall		Easy	Hard	View Dep.	View Indep.
	A@0.1	A@0.25	A@0.1	A@0.1	A@0.1	A@0.1
ConceptGraph [4]	13.3	6.2	13.0	16.0	15.2	13.1
GaussianGraph [16]	18.2	7.4	19.1	16.3	-	-
BBQ [5]	34.2	22.7	34.3	33.3	32.9	34.4
Open3DSG [17]	37.3	25.8	36.5	36.1	36.1	36.6
KNA-SG	51.4	37.2	54.1	30.7	45.6	52.2

further reports results on Sr3D. KNA-SG obtains the best overall accuracy, with 51.4% under A@0.1 and 37.2% under A@0.25. The method also performs strongly on view-dependent and view-independent queries, achieving 45.6% and 52.2% under A@0.1, respectively. These results suggest that the proposed graph representation generalizes to synthetic spatial referring expressions and can support grounding through a combination of object-level semantics, geometric relation computation, and keyframe-based verification. On the hard split, KNA-SG remains slightly behind Open3DSG, suggesting that cases with multiple similar candidate objects still pose challenges for the current semantic and spatial reasoning pipeline.

Overall, the grounding results show that KNA-SG provides a useful interface between open-vocabulary 3D scene representation and natural-language object grounding. The gains over other methods indicate that explicitly associating object nodes with semantic attributes, lightweight geometry, and keyframe observations improves the execution of language queries in reconstructed 3D scenes.

4.4. Ablation Study of Keyframe Selection

4.4.1. Object-Level Visibility Coverage

To evaluate whether the selected frames provide sufficient object-level observations, we measure visibility coverage at the instance level. Structural background categories such as walls, floors, and ceilings are excluded from this evaluation. For each non-structural object instance, we compute the ratio between its visible surface area covered by the selected frames and its total surface area. Obj@0.1 reports the percentage of non-structural objects whose visible coverage is at least 10%, while SmallObj@0.1 applies the same criterion to small non-structural objects. MeanCov measures the average visible coverage of observed non-structural objects. These metrics evaluate not only whether objects are included in the selected frames, but also whether they are observed with sufficient surface coverage.

Table 4. Ablation study of frame sampling strategies for object-level visibility coverage. MC denotes maximum-coverage sampling.

Sampling Strategy	3D Info	Obj@0.1	SmallObj@0.1	MeanCov
MC	Y	100.00	100.00	86.21
Uniform	N	99.19	98.44	81.26
Ours	N	99.52	100.00	82.83

compares three frame sampling strategies. Maximum-coverage sampling (MC) selects frames with a target coverage ratio of 95% and a maximum frame number of 32. Since MC relies on 3D information during sampling, it is used as an upper-bound reference. Uniform sampling selects one frame every 100 frames and serves as a fixed-interval baseline. Ours denotes the proposed two-stage keyframe selection strategy that does not rely on 3D information during sampling. As expected, MC achieves the highest overall coverage, reaching 100.00% on both Obj@0.1 and SmallObj@0.1, with a MeanCov of 86.21%. Compared with Uniform, the proposed method improves Obj@0.1 from 99.19% to 99.52% and increases MeanCov from 81.26% to 82.83%. More importantly, SmallObj@0.1 improves from 98.44% to 100.00%, reaching the same level as maximum-coverage sampling. This suggests that the proposed strategy better preserves small-object observations, which are often missed by fixed-interval sampling due to limited visible area. The improvement arises from the complementary roles of clarity-aware candidate selection (CCS) and feature difference selection (FDS). CCS selects visually clear candidates within temporal windows while enforcing minimum frame intervals to avoid densely sampled observations. FDS then evaluates feature-level diversity among candidates using DINOv2 global representations and suppresses redundant views while retaining meaningful viewpoint changes. Together, they provide more representative object-level evidence for reconstruction and scene graph construction.

4.4.2. Effect on Open-Vocabulary 3D Semantic Segmentation

In addition to object-level visibility coverage, we further evaluate whether the proposed keyframe selection strategy improves open-vocabulary 3D semantic segmentation. Unlike the visibility coverage experiment, where different sampling methods follow their own selection rules, this ablation adopts a scene-wise frame-budget control protocol. For each scene, the number of selected frames is first determined by the full CCS and FDS strategy, and all ablated variants are constrained to use the same frame number. This protocol removes the influence of different input sizes and allows us to focus on the effectiveness of the sampling criteria themselves.

Table 5. Ablation study of feature difference selection (FDS) on Replica.

FDS	mAcc	mIoU	F-mIoU
×	41.62	29.85	66.40
✓	50.91	39.15	68.73

reports the ablation results on Replica. Since Replica contains relatively clean and stable indoor observations, we mainly evaluate the effect of FDS. Compared with the baseline without FDS, introducing FDS improves mAcc from 41.62% to 50.91% and mIoU from 29.85% to 39.15%, while F-mIoU increases from 66.40% to 68.73%. Under the same frame budget, this result indicates that selecting semantically diverse frames provides more representative observations for subsequent instance-grounded reconstruction and semantic aggregation than simply using frames selected without semantic diversity refinement.

Table 6. Ablation study of clarity-aware candidate selection (CCS) and feature difference selection (FDS) on ScanNet.

CCS	FDS	mAcc	mIoU	F-mIoU
×	×	54.10	39.67	45.85
×	✓	54.34	40.30	43.96
✓	✓	59.43	44.06	49.87

further evaluates the combined effect of CCS and FDS on ScanNet. Under the matched-budget setting, using FDS alone brings only limited improvement, increasing mAcc from 54.10% to 54.34% and mIoU from 39.67% to 40.30%, while F-mIoU decreases from 45.85% to 43.96%. This suggests that semantic diversity alone is insufficient when visually degraded frames remain in the candidate set. When CCS is combined with FDS, the full two-stage strategy achieves consistent improvements across all metrics, increasing mAcc to 59.43%, mIoU to 44.06%, and F-mIoU to 49.87%. Since all variants use the same scene-wise frame budget, these gains are not caused by using more input frames. Instead, they indicate that CCS provides a more reliable candidate set by suppressing low-quality observations, while FDS further improves the semantic representativeness of the selected frames before global multi-view aggregation.

4.5. Effectiveness of ID-Based Visual Prompts

We evaluate whether explicit visual IDs improve MLLM-based instance-level object perception. The baseline uses raw RGB keyframes as input, where the MLLM identifies and counts visible object instances from the scene content without explicit object references. In contrast, the ID-based setting renders unique visual IDs on keyframes, with ID positions determined from cross-view consistent instance masks to ensure that each ID is placed within the corresponding visible object region. These IDs provide spatially anchored references, allowing the MLLM to associate its responses with specific visible instances. Structural background categories, including walls, floors, and ceilings, are excluded so that the evaluation focuses on the foreground object. We adopt two instance-counting-based metrics, namely mean instance recall (mIR) and mean count consistency score (mCCS). For each category c, instance recall is defined as

$${IR}_c={\displaystyle \frac{{\sum }_{s}min\left({\widehat{N}}_{s,c},N_{s,c}\right)}{{\sum }_s{N}_{s,c}}},$$

(13)

where ${\widehat{N}}_{s,c}$ and $N_{s,c}$ denote the predicted and ground-truth instance counts of category c in scene s, respectively. The $min(·)$ operation prevents over-counting from increasing recall, so this metric measures the coverage of ground-truth instances. The final mIR is obtained by averaging ${IR}_c$ over all evaluated foreground categories. To evaluate counting consistency, we define the score for category c as

$${CCS}_c=max\left(0,1-{\displaystyle \frac{{\sum }_s\left|{\widehat{N}}_{s,c}-N_{s,c}\right|}{{\sum }_s{N}_{s,c}}}\right).$$

(14)

This metric penalizes both missed instances and redundant predictions. The final mCCS is computed by averaging ${CCS}_c$ over all evaluated foreground categories.

As shown in

Table 7. Effect of ID-based visual prompts on instance-level object perception. The “with ID” setting renders unique visual IDs on selected keyframes according to cross-view consistent instance masks, while the baseline uses raw RGB keyframes without explicit object references. Structural background categories are excluded. mIR denotes mean instance recall, and mCCS denotes mean count consistency score.

Dataset	With ID	mIR	mCCS
Replica	×	38.61	37.88
	✓	51.31	47.42
ScanNet	×	47.96	40.36
	✓	67.99	47.47

, explicit visual IDs improve instance-level perception on both datasets. On Replica, ID-based prompting increases mIR from 38.61% to 51.31% and mCCS from 37.88% to 47.42%. On ScanNet, mIR increases from 47.96% to 67.99%, while mCCS improves from 40.36% to 47.47%. These results indicate that explicit visual IDs help the MLLM associate semantic responses with spatially anchored instances, instead of relying only on scene-level interpretation from raw RGB images.

The gain is more pronounced for mIR, particularly on ScanNet. This suggests that explicit IDs mainly improve the completeness of instance perception in cluttered real-world scenes. Without explicit references, the MLLM may focus on salient objects while omitting less prominent instances or merging multiple similar objects into one response. ID-based visual prompts reduce this ambiguity by providing stable references for visible instances, resulting in more complete instance coverage and more consistent object counting.

4.6. Ablation Study of Query Execution Strategies

We further evaluate the effect of different query execution strategies on language-guided 3D object grounding. As shown in

Table 8. Ablation study of query execution strategies on the Nr3D dataset. A@0.1 denotes grounding accuracy under a 3D bounding-box IoU threshold of 0.1. Avg. Time reports the average inference time per query.

Strategy	Overall	Easy	Hard	View Dep.	View Indep.	Avg. Time (s)
MLLM-based selection	42.8	50.4	23.4	34.6	45.3	41.7
Geometry-only reasoning	37.3	44.6	18.8	31.5	39.1	9.6
KNA-SG	43.8	50.4	26.9	37.6	45.7	25.4

, we compare three variants on the Nr3D dataset. The first variant, MLLM-based selection, parses the query, retrieves candidate nodes from the scene graph, and uses the MLLM to select the target from the candidates based on their associated ID-tagged keyframes. The second variant, geometry-only reasoning, parses the query and resolves relation constraints using 3D geometric descriptors. The KNA-SG combines geometry-based reasoning with keyframe-based MLLM verification.

The MLLM-only variant achieves competitive overall accuracy, reaching 42.8%, but requires the longest average inference time of 41.7 s per query. This indicates that keyframe-based visual verification is useful for resolving candidate ambiguity, but applying it to all queries introduces substantial computational cost. In contrast, the geometry-only variant is much faster, with an average inference time of 9.6 s, but its overall accuracy drops to 37.3%. This suggests that geometry alone is insufficient for queries requiring fine-grained visual or view-dependent disambiguation. KNA-SG achieves the highest overall accuracy among the three strategies, reaching 43.8%, while reducing the average inference time from 41.7 to 25.4 s compared with MLLM-based selection. Compared with geometry-only reasoning, KNA-SG improves the hard split from 18.8% to 26.9% and the view-dependent split from 31.5% to 37.6%, showing that keyframe-based verification helps resolve difficult and viewpoint-sensitive queries. Compared with MLLM-only verification, KNA-SG achieves higher accuracy with lower inference time, indicating that the geometry-first decision strategy avoids unnecessary MLLM calls while retaining visual verification for ambiguous cases.

These results support the design of query-time relation verification in KNA-SG. Geometry-based reasoning provides an efficient first-stage filter, while ID-tagged keyframes and MLLM-based verification improve robustness when geometry is ambiguous or insufficient. The hybrid strategy therefore balances grounding accuracy and inference efficiency.

5. Conclusions

In this paper, we propose KNA-SG, an open-vocabulary 3D scene graph framework constructed from RGB sequences. It associates reconstructed object nodes with retrievable keyframe observations. Experimental results demonstrate the effectiveness of KNA-SG on open-vocabulary 3D semantic segmentation and language-guided 3D object grounding. In particular, language-guided grounding serves as a representative downstream task for evaluating whether the constructed scene graph can support semantic retrieval, geometric relation reasoning, and keyframe-based visual verification. The ablation studies further show that two-stage keyframe selection helps preserve useful object-level observations, ID-guided node parsing reduces ambiguity in instance-level semantic assignment, and geometry-first query execution with keyframe-based verification improves the balance between grounding accuracy and inference cost. These results indicate that reliable object-level visual evidence is important for connecting 3D reconstruction, semantic parsing, and language-driven reasoning. A remaining limitation is that the current graph is constructed for static scenes, where object nodes and verified relations are assumed to remain valid after construction. Future work will extend the framework to dynamic scenes by updating object nodes and accumulated spatial relations when objects move or disappear.