HOSG-Nav: Hierarchical Open-Vocabulary Semantic Graph Navigation for Language-Guided Global Planning in 3D Gaussian Scenes

Abstract

Natural-language-driven robot navigation in complex indoor environments requires the joint capability of high-fidelity scene representation, structured semantic reasoning, and executable path planning. To address this challenge, this paper proposes HOSG-Nav, a unified framework for natural-language-driven global navigation that integrates open-vocabulary 3D Gaussian scene representation, hierarchical semantic scene graph construction, and large-language-model-driven planning. First, an open-vocabulary 3D Gaussian field is constructed to jointly encode scene geometry, appearance, and semantic information, where compressed CLIP features are lifted into continuous 3D space and depth supervision is introduced to enhance geometric stability and metric-scale consistency. Second, the optimized Gaussian primitives are further abstracted into a semantic scene graph with a region–object hierarchical structure and traversable topological relations to support structured environment understanding. Finally, for natural language instructions, hierarchical semantic parsing is performed with the assistance of a large language model, and executable global navigation paths are generated through cross-modal target retrieval and graph-search-based planning. Experimental results on the Replica dataset demonstrate that HOSG-Nav achieves competitive performance in scene representation, semantic target retrieval, and global navigation, validating the effectiveness of jointly integrating multimodal 3D representation, hierarchical semantic abstraction, and language-guided planning.

1. Introduction

Robotic navigation is a fundamental capability for embodied agents operating in indoor environments such as warehouse logistics, service robotics, inspection, and human–robot collaboration. Classical navigation systems primarily address localization, obstacle avoidance, and geometric path planning—that is, determining where the robot is and how it can reach a target location [1,2,3]. However, as embodied intelligence moves toward task-oriented interaction in open environments, navigation is no longer a purely geometric problem. Robots must increasingly understand semantic targets, resolve region-level context, interpret human instructions, and make decisions conditioned on objects, regions, and their relations. This shift requires navigation systems to move beyond geometric reachability and support semantic grounding and structured reasoning. Advanced autonomous navigation systems in other open-environment domains, such as autonomous driving and autonomous UAV navigation, also rely heavily on structured environment representations. Autonomous vehicles commonly use high-definition maps, bird’s-eye-view (BEV) semantic representations, occupancy grids, lane-level topology, semantic/vectorized maps, and dynamic object states to support localization, perception, prediction, and rule-constrained planning [4,5,6,7]. Similarly, autonomous UAVs, especially in GPS-denied or partially unknown environments, often employ SLAM-based metric maps, occupancy or voxel maps, Euclidean signed distance fields, frontier maps, and topological abstractions for collision avoidance and trajectory generation [8,9,10]. These representations are highly effective for metric localization and safety-critical motion planning, but they are usually designed around geometric traversability, traffic rules, or mission-level waypoints, rather than open-vocabulary semantic grounding from free-form natural language instructions. In contrast, our work focuses on indoor language-guided global navigation, where the key requirement is to connect continuous 3D scene representation with region–object semantic abstraction and language-conditioned graph search.

A key prerequisite for such capability is an environment representation that is both semantically expressive and planning-friendly. Early approaches relied mainly on occupancy grids, topological maps, or dense geometric maps [2,11]. Although effective for collision avoidance and route search, these representations provide limited support for object semantics, functional regions, and high-level task goals. More recently, vision–language models have promoted navigation-oriented mapping from closed-set semantics to open-vocabulary scene representation [12,13,14,15]. In parallel, continuous 3D representations such as neural radiance fields (NeRFs) and 3D Gaussian Splatting (3DGS) have improved the fidelity and expressiveness of scene modeling [16,17,18,19]. In particular, 3DGS provides an efficient continuous representation that can jointly encode geometry, appearance, and semantics in 3D space. This makes it a promising foundation for navigation in open environments.

Despite these advances, continuous 3D semantic representations alone are insufficient for navigation-oriented reasoning. A Gaussian scene comprises a large number of low-level primitives that support reconstruction and rendering yet remain unsuitable for direct semantic retrieval, region reasoning, or graph-based path planning. For practical navigation, an embodied agent requires a more compact and interpretable intermediate structure that can connect continuous scene modeling with discrete decision making. In this respect, 3D scene graphs provide a natural representation, as they explicitly organize objects, regions, and their relations through nodes and edges [20,21,22,23]. Such structured representations are better suited to semantic querying, hierarchical target grounding, and planning over navigable topology.

A second challenge is connecting structured environment representations with natural language instructions. Vision-and-language navigation (VLN) has shown that language can effectively guide navigation behavior [24,25,26]. However, most existing VLN-style approaches rely on local observation sequences or implicit policy representations, and therefore offer limited support for stable environmental memory and explicit long-horizon reasoning. Recent advances in large language models (LLMs) have demonstrated capabilities in semantic parsing, task decomposition, commonsense reasoning, and high-level planning [27,28,29,30]. This creates an opportunity to decompose free-form navigation instructions into structured semantic constraints and align them with environment representations that explicitly encode regions, objects, and navigable relations. Nevertheless, a unified framework that tightly integrates open-vocabulary continuous 3D scene representation, hierarchical semantic abstraction, and LLM-driven planning is still lacking.

To address these challenges, we propose HOSG-Nav, a unified framework for natural-language-driven global navigation in complex indoor environments. The method first constructs an open-vocabulary 3D Gaussian scene representation that jointly models geometry, appearance, and semantics. It then abstracts the optimized Gaussian field into a hierarchical semantic scene graph with a region–object structure, transforming low-level scene primitives into a structured representation suitable for semantic reasoning and navigation planning. Finally, an LLM parses free-form natural language instructions into hierarchical semantic queries, which are combined with cross-modal target retrieval and graph-based search to generate executable global navigation paths. In this way, HOSG-Nav establishes a unified pipeline from continuous scene representation to structured semantic abstraction and language-guided global planning.

Unlike advanced autonomous driving and UAV navigation systems, which mainly rely on HD maps, BEV/occupancy representations, metric SLAM maps, or topological abstractions for safety-critical motion planning, HOSG-Nav specifically targets natural language-guided indoor global navigation by coupling open-vocabulary continuous 3D representation with hierarchical semantic graph reasoning. The main contributions of this work are summarized as follows:

• We propose HOSG-Nav, a unified framework for natural-language-driven global navigation in complex indoor environments. The framework integrates open-vocabulary 3D scene representation, hierarchical semantic scene graph construction, and LLM-driven planning within a single pipeline, bridging continuous 3D scene modeling and structured navigation reasoning. Unlike prior methods that address isolated stages—such as open-vocabulary 3D mapping without structured planning, or scene-graph-based reasoning without continuous 3D semantic representation—HOSG-Nav tightly couples all three levels into an end-to-end pipeline.
• We develop an open-vocabulary 3D Gaussian scene representation and a navigation-oriented hierarchical scene graph construction method. Compressed CLIP features are lifted into a continuous 3D Gaussian field, while depth supervision is introduced to improve geometric stability and metric consistency. Based on the optimized field, Gaussian primitives are further abstracted into a region–object hierarchical scene graph through spatial–semantic clustering, semantic naming, functional region formation, and traversable topology modeling. This direct abstraction from a jointly optimized continuous Gaussian field to a discrete hierarchical graph differs from prior scene graph methods that build on discrete segmentation or online observation aggregation.
• We introduce an LLM-driven hierarchical planning strategy for natural language navigation. The proposed method decomposes free-form instructions into region-level, object-level, and attribute-level constraints, and combines hierarchical cross-modal retrieval with graph-search-based planning to generate executable global paths. The planning module is explicitly aligned with the region–object graph structure, enabling tighter coupling between language reasoning and structured scene abstraction than prior approaches that plan over flat object graphs or implicit policy representations. Extensive experiments demonstrate the effectiveness of HOSG-Nav in open-vocabulary scene representation, semantic target grounding, and global navigation.

2. Related Work

2.1. Semantic 3D Mapping for Navigation

Early navigation systems primarily relied on occupancy grids and topological maps to represent environmental structure [1,2,11], while effective for obstacle avoidance and geometric path planning, such representations provide limited support for high-level semantic reasoning (for example, identifying target objects, distinguishing functional areas, or resolving ambiguous language descriptions). To address this limitation, semantic cues have gradually been incorporated into 3D mapping, giving rise to semantic 3D mapping [31,32,33]. This direction enables robots to reason not only about where they can move, but also about what semantic entities different locations correspond to.

Existing semantic 3D mapping methods have followed several main directions. Some enrich maps by learning semantic features from sensor observations [34], while others adopt object-centric formulations by associating pre-built object models with geometric maps [35]. A widely used strategy is to lift 2D detections or semantic predictions into 3D through back-projection or geometric instantiation [36,37,38,39]. Semantic information has also been integrated into dense SLAM systems to jointly optimize geometry and semantics online [40,41]. Despite their effectiveness, most of these methods remain based on closed-set semantic categories and therefore generalize poorly to open-vocabulary instructions and abstract region-level queries.

The emergence of vision–language foundation models, particularly CLIP [12], has advanced semantic 3D mapping toward open-vocabulary scene representation. Recent methods lift or aggregate vision–language features into 3D space, allowing scene entities to be indexed by free-form text [13,14,15,42,43]. Compared with conventional closed-set maps, these approaches offer greater flexibility for unseen targets and natural language descriptions. However, many of them still depend heavily on 2D feature lifting and multi-view aggregation, making the final 3D semantic representation sensitive to observation quality and fusion consistency.

In parallel, continuous 3D representations such as NeRF [16] and 3DGS [17] have provided a new basis for semantic map construction. By jointly modeling geometry, appearance, and semantics in continuous space, these methods offer a finer-grained representation for language-conditioned perception and navigation. In particular, Gaussian-based SLAM and mapping systems have shown competitive performance in reconstruction quality, efficiency, and rendering speed [18,19,44,45]. More recent studies further optimize semantic features directly in the mapping process [46,47,48,49], making 3DGS an increasingly promising foundation for navigation-oriented scene understanding.

Overall, semantic 3D mapping has evolved from geometry augmented with category labels to open-vocabulary continuous scene representations. However, most existing methods focus primarily on enriching the map itself, rather than constructing structured intermediate representations that are suitable for semantic reasoning and graph-based navigation. Building navigation-oriented hierarchical structures on top of continuous 3D semantic representations remains an important problem.

2.2. Semantic 3D Scene Graphs for Navigation

Scene graphs provide a structured representation in which scene entities, attributes, and relations are explicitly encoded as nodes and edges [50]. Originally developed for visual relation modeling and high-level vision tasks such as visual question answering and image captioning [51,52,53], scene graphs have gradually been extended to 3D perception and robotic mapping [20]. Compared with point clouds, voxel maps, or continuous fields, 3D scene graphs are more interpretable and compositional, and are therefore better suited to semantic querying and structured reasoning.

Prior work has shown that 3D scene graphs can serve as compact object-centric representations for large-scale environments [54,55], while also supporting semantic querying, relation modeling, and high-level task reasoning [56,57]. To enable practical robotic deployment, several studies have investigated automatic graph construction and incremental updating. For example, SceneGraphFusion [21] moves 3D scene graphs from offline representations toward online perceptual mapping. Nevertheless, semantic hierarchy and object relations alone are insufficient for navigation. A navigation-oriented graph must additionally encode region hierarchy, traversability, and topological connectivity so that it can support not only scene understanding but also path planning.

Motivated by this need, recent work has increasingly explored semantic 3D scene graphs for navigation. Dynamic Scene Graphs [58] explicitly incorporate spatial structure for planning and decision making. Situational Graphs [59], G2o [60], and S-Graphs [61] more tightly couple scene graphs with SLAM graphs for real-world robotic navigation. ConceptGraphs [22] further introduce open-vocabulary vision–language features into 3D scene graphs, enabling language querying and high-level semantic planning. HOV-SG [23] constructs a hierarchical scene graph over floors, rooms, and objects on top of open-vocabulary 3D maps and explicitly targets language-guided indoor navigation. These studies collectively indicate that 3D scene graphs are becoming an important bridge between scene representation, semantic retrieval, and navigation planning.

Despite this progress, several limitations remain. Methods such as HOV-SG construct graphs on top of discrete open-vocabulary maps rather than on a jointly optimized continuous 3D Gaussian field; SayPlan and SayNav employ LLMs for planning but do not tightly integrate continuous open-vocabulary 3D representation with hierarchical semantic graph abstraction. Many existing methods are still built on discrete segmentation results or online observation aggregation, and therefore make limited use of the joint coupling between geometry and semantics available in continuous 3D scene representations. In addition, although hierarchical organization is sometimes introduced, support for region-level semantics, hierarchical target retrieval, and tight integration with global path planning is often still limited. Therefore, constructing navigation-oriented hierarchical scene graphs directly from continuous open-vocabulary 3D representations remains an important open problem.

2.3. Language-Guided Planning and Navigation

Language-guided navigation seeks to enable an embodied agent to perform target search, path planning, and task execution from natural language instructions. A representative paradigm is vision-and-language navigation (VLN) [24], in which an agent predicts action sequences or local waypoints from egocentric visual observations and text instructions [25,26]. This line of research has significantly advanced the integration of language understanding and visual perception, allowing agents to follow routes and reach targets specified in natural language [62]. However, most VLN-style methods rely on online observations or implicit policy representations, and therefore lack a stable and reusable explicit 3D semantic environment model. This limitation becomes more evident in long-horizon navigation, open-set target grounding, and generalization to complex environments.

With the rapid development of large language models (LLMs) [27], increasing attention has been paid to their role in navigation-oriented reasoning and planning. One direction introduces chain-of-thought reasoning into the navigation process. For example, Open-Nav [29] adopts a spatio-temporal chain-of-thought for continuous VLN, NavCoT [30] proposes a trainable decoupled chain-of-thought for future imagination, visual filtering, and action prediction, CL-CoTNav [63] applies hierarchical chain-of-thought reasoning to object navigation, and VoroNav [64] combines spatial topology and scene description to guide waypoint selection. These studies show that LLMs can improve instruction understanding and high-level decision making beyond conventional end-to-end navigation policies.

Another important line of work couples LLMs with semantic maps and structured environment representations for navigation. Zhong et al. [65] proposed a language-driven navigation strategy that integrates semantic maps with LLMs, using visual foundation models for map construction and LLMs for instruction interpretation, while closely related, this approach relies on 2D semantic maps, whereas HOSG-Nav constructs a continuous open-vocabulary 3D Gaussian representation and abstracts it into a region–object hierarchical graph for global planning. More broadly, Igelbrink et al. [66] reviewed online knowledge integration for 3D semantic mapping, emphasizing the role of scene graphs and language models in incorporating symbolic knowledge into mapping pipelines. This supports the importance of unifying geometric mapping, semantic abstraction, and language-level reasoning for modern robot navigation.

Another important direction couples LLMs with structured environment representations, especially scene graphs, for planning and navigation. SayPlan [57] uses GPT-4 [67] to iteratively search scene graphs and generate executable plans under feasibility constraints. SayNav [68] combines scene graph reasoning with short-range navigation in previously unseen environments. Related work has further explored richer graph–LLM integration for embodied tasks. For example, MoMa-LLM [69] feeds scene graphs into GPT-4 for mobile manipulation, while GRID [70] combines scene graphs, graph neural networks, and LLM-based encodings to predict action sequences. These studies suggest that scene graphs provide an effective interface between high-level language reasoning and low-level action execution.

Overall, language-guided navigation is evolving from instruction following over observation streams toward a structured paradigm combining language reasoning, explicit environment representation, semantic mapping, and high-level planning [65,66]. However, many existing approaches still rely on local semantic memory built during online exploration or on implicit decision making over observation sequences, rather than on a stable continuous 3D semantic scene representation. Moreover, even when scene graphs are introduced, many methods focus primarily on local target reasoning or subtask planning, and do not fully unify open-vocabulary 3D semantic mapping, hierarchical scene graph retrieval, and global path planning within a single framework. Bridging these components into a unified language-driven navigation pipeline therefore remains an important research direction.

2.4. Representations in Advanced Autonomous Navigation Systems

Beyond indoor robot navigation, autonomous driving and autonomous UAV navigation provide important references for environment representation and planning. In autonomous driving, the most widely used abstractions include high-definition maps, BEV representations, 3D occupancy grids, semantic/vectorized maps, lane graphs, and dynamic object states [4,5,6,7]. HD maps provide high-precision geometric and semantic priors such as lane boundaries, road topology, traffic signs, and traffic lights, while BEV and occupancy representations offer structured top–down descriptions for perception, prediction, and planning. These representations are particularly suitable for road-constrained navigation, where motion is strongly regulated by lanes, traffic rules, and dynamic interactions with other agents.

Autonomous UAV navigation often uses a different set of abstractions due to the three-dimensional motion space and the frequent need to operate in GPS-denied, cluttered, or partially unknown environments. Common representations include SLAM-based metric maps, occupancy grids, voxel maps, signed distance fields, frontier-based exploration maps, and topological graphs [8,9,10]. These maps mainly support localization, obstacle avoidance, trajectory optimization, and safe exploration. When semantic information is introduced, it is usually used to improve scene understanding, landing-site selection, or mission-level reasoning.

Compared with these advanced navigation systems, HOSG-Nav addresses a complementary problem. Rather than focusing on lane-level driving policies, high-speed dynamic avoidance, or low-level UAV trajectory control, our goal is to generate executable global navigation paths from free-form natural language instructions in indoor 3D scenes. Therefore, the proposed representation combines three levels of abstraction: a continuous open-vocabulary 3D Gaussian field for geometry, appearance, and semantics; a region–object hierarchical scene graph for semantic reasoning; and a graph-search interface for language-conditioned global planning. Current Gaussian-based representations in autonomous driving are mainly explored for reconstruction, rendering, simulation, or map updating [71], whereas our work further abstracts Gaussian primitives into a navigation-oriented semantic graph. In this sense, HOSG-Nav can be viewed as complementary to the metric and topological abstractions used in autonomous vehicles and UAVs, with a stronger emphasis on open-vocabulary language grounding and hierarchical semantic planning.

3. Methods

3.1. Overview

We propose HOSG-Nav, a unified framework for natural-language-driven global navigation built upon open-vocabulary 3D scene representation and hierarchical semantic reasoning. As illustrated in Figure 1, the framework consists of three tightly coupled stages that read from left to right. Stage 1 (left) takes multi-view RGB-D observations and optimizes a continuous 3D Gaussian field that jointly encodes geometry, appearance, and semantics. Stage 2 (center) clusters the Gaussian primitives into objects and regions, forming a hierarchical scene graph with semantic and topological relations. Stage 3 (right) accepts a natural language instruction, parses it via an LLM into hierarchical queries, retrieves the target through cross-modal matching, and plans an executable global path using graph search.

In this way, HOSG-Nav establishes a unified pipeline from continuous 3D scene modeling to structured semantic abstraction and language-guided navigation planning. Section 3.2 introduces the open-vocabulary 3D scene representation, Section 3.3 describes the hierarchical semantic scene graph construction, and Section 3.4 presents the LLM-driven global planning strategy.

3.2. Open-Vocabulary 3D Gaussian Scene Representation

To support natural-language-driven navigation, the scene representation must simultaneously preserve geometric fidelity and enable open-vocabulary semantic grounding. To this end, we extend 3DGS with semantic feature lifting and depth-aware supervision, yielding a continuous 3D representation that jointly models appearance, structure, and semantics.

3.2.1. 3DGS Preliminaries

Standard 3DGS [17] represents a scene as a set of anisotropic Gaussian primitives. Each primitive i is parameterized by its center ${\mathit{\mu }}_i\in {\mathbb{R}}^3$, covariance matrix ${\mathbf{\Sigma }}_i\in {\mathbb{R}}^{3\times 3}$, opacity $o_i\in [0,1]$, and view-dependent color ${\mathit{c}}_i$ encoded by spherical harmonics. To ensure that the covariance remains positive semi-definite during optimization, it is decomposed as

$${\mathbf{\Sigma }}_i={\mathit{R}}_i{\mathit{S}}_i{\mathit{S}}_i^T{\mathit{R}}_i^T,$$

(1)

where ${\mathit{R}}_i$ and ${\mathit{S}}_i$ denote the rotation and scaling matrices, respectively.

Given a camera pose, the rendered color at pixel $\mathit{p}$ is obtained by differentiable $\alpha$-blending over the depth-sorted overlapping Gaussians:

$$C\left(\mathit{p}\right)={\displaystyle \sum _{i=1}^N}{\mathit{c}}_i{\alpha }_i{\displaystyle \prod _{j=1}^{i-1}}(1-{\alpha }_j),$$

(2)

where ${\alpha }_i\left(\mathbf{p}\right)$ is the alpha blending weight of Gaussian i at pixel $\mathbf{p}$, computed as the product of its learnable opacity $o_i$ and the value of its 2D projected Gaussian evaluated at $\mathbf{p}$.

We initialize the Gaussian field using a Structure-from-Motion pipeline, specifically COLMAP [72], which provides camera poses and a sparse point cloud. The Gaussian centers ${\mathit{\mu }}_i$ and base colors are initialized from these sparse points, thereby providing a geometric prior for subsequent joint optimization.

3.2.2. 2D Semantic Feature Extraction and Compression

To endow the 3D scene with open-vocabulary semantics, we extract 2D semantic features from multi-view RGB images using a pre-trained CLIP model [12]. Directly storing and rendering the original 512-dimensional CLIP features, however, would incur substantial memory and computational overhead. We therefore introduce a lightweight linear projection layer with learnable weights ${\mathit{W}}_{comp}\in {\mathbb{R}}^{512\times 32}$ to compress the features into a 32-dimensional latent space:

$${\mathit{F}}_{32}^{2D}\left(\mathit{p}\right)={\mathit{F}}_{CLIP}^{2D}\left(\mathit{p}\right){\mathit{W}}_{comp}.$$

(3)

This compressed representation preserves the semantic structure required for language grounding while remaining efficient for downstream rendering, retrieval, and planning.

3.2.3. Joint Optimization with Semantic and Depth Supervision

To lift the compressed 2D semantic features into 3D space, we assign each Gaussian primitive an additional learnable semantic feature vector ${\mathit{f}}_i\in {\mathbb{R}}^{32}$. Unlike appearance parameters, which are more sensitive to initialization, the semantic features are initialized to zero and learned through multi-view supervision. The rendered semantic feature at pixel $\mathit{p}$ is computed using the same accumulation rule as color rendering:

$$F\left(\mathit{p}\right)={\displaystyle \sum _{i=1}^N}{\mathit{f}}_i{\alpha }_i{\displaystyle \prod _{j=1}^{i-1}}(1-{\alpha }_j).$$

(4)

To align the rendered 3D semantic features with the compressed 2D CLIP features, we define the semantic consistency loss as the average cosine distance:

$${\mathcal{L}}_{semantic}={\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{\mathit{p}\in \mathcal{P}}\left(1-{\displaystyle \frac{F{\left(\mathit{p}\right)}^T{\mathit{F}}_{32}^{2D}\left(\mathit{p}\right)}{{\parallel F\left(\mathit{p}\right)\parallel }_2{\parallel {\mathit{F}}_{32}^{2D}\left(\mathit{p}\right)\parallel }_2}}\right),$$

(5)

where $\mathcal{P}$ denotes the set of supervised pixels, and ${\parallel ·\parallel }_2$ denotes the Euclidean (L2) norm.

In addition, RGB supervision alone is often insufficient in weakly textured indoor regions, where geometry may become unstable or locally oversmoothed. To improve structural consistency, we introduce explicit depth supervision. The rendered depth at pixel $\mathit{p}$ is defined as

$$D\left(\mathit{p}\right)={\displaystyle \sum _{i=1}^N}{z}_i{\alpha }_i{\displaystyle \prod _{j=1}^{i-1}}(1-{\alpha }_j),$$

(6)

where $z_i$ denotes the depth of the Gaussian center in the camera coordinate system. The depth loss is then given by

$${\mathcal{L}}_{depth}={\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{\mathit{p}\in \mathcal{P}}{∥D\left(\mathit{p}\right)-D_{GT}\left(\mathit{p}\right)∥}_1,$$

(7)

where $D_{GT}\left(\mathit{p}\right)$ is the sensor depth, and ${\parallel ·\parallel }_1$ denotes the L1 norm.

Following standard 3DGS practice, we also employ a photometric loss ${\mathcal{L}}_{photo}$ between the rendered and ground-truth RGB images, typically consisting of an $L_1$ term and a D-SSIM term. The overall training objective is

$${\mathcal{L}}_{total}={\mathcal{L}}_{photo}+{\lambda }_{sem}{\mathcal{L}}_{semantic}+{\lambda }_D{\mathcal{L}}_{depth}.$$

(8)

where the weights ${\lambda }_{sem}$ and ${\lambda }_D$ are chosen to balance the photometric, semantic, and depth objectives so that neither supervision term dominates the optimization, while both contribute meaningfully to the final representation.

The resulting 3D Gaussian field serves as a unified intermediate representation with consistent geometry, appearance, and open-vocabulary semantics. This representation forms the basis for the subsequent hierarchical scene graph construction and language-guided navigation planning.

3.3. Hierarchical Semantic Scene Graph Construction

Although the semantic 3D Gaussian field provides a continuous and expressive scene representation, it remains suboptimal for high-level reasoning and graph-based planning. LLM-guided navigation requires a structured representation with discrete semantic units, explicit relations, and navigable topology. We therefore abstract the dense Gaussian field into a hierarchical semantic scene graph that supports both semantic retrieval and global planning.

Formally, the scene graph is defined as

$$\mathcal{S}=(\mathcal{V},\mathcal{E}),$$

(9)

where $\mathcal{V}$ and $\mathcal{E}$ denote the node and edge sets, respectively. The node set is divided into region-level nodes ${\mathcal{V}}_{reg}$ and object-level nodes ${\mathcal{V}}_{obj}$:

$$\mathcal{V}={\mathcal{V}}_{reg}\cup {\mathcal{V}}_{obj}.$$

The edge set includes three relation types:

$$\mathcal{E}={\mathcal{E}}_{reg\text{-}reg}\cup {\mathcal{E}}_{reg\text{-}obj}\cup {\mathcal{E}}_{obj\text{-}obj},$$

where ${\mathcal{E}}_{reg\text{-}reg}$ denotes region adjacency, ${\mathcal{E}}_{reg\text{-}obj}$ denotes region–object containment, and ${\mathcal{E}}_{obj\text{-}obj}$ denotes local traversability among object nodes.

3.3.1. Spatial–Semantic Joint Clustering

We first aggregate the Gaussian field $\mathcal{G}={\left\{g_i\right\}}_{i=1}^N$, where each $g_i$ denotes a Gaussian primitive parameterized by center ${\mathit{\mu }}_i\in {\mathbb{R}}^3$, covariance ${\mathbf{\Sigma }}_i\in {\mathbb{R}}^{3\times 3}$, opacity $o_i\in [0,1]$, color ${\mathit{c}}_i$, and semantic feature ${\mathit{f}}_i\in {\mathbb{R}}^{32}$ (Section 3.2), into object candidates. Since low-opacity Gaussians often correspond to noise or floating artifacts, we retain only primitives satisfying $o_i\ge {\tau }_o$, where ${\tau }_o$ is a minimum opacity threshold chosen to filter out near-transparent floating artifacts while preserving valid scene structure; in practice, most foreground Gaussians have opacities well above this level, whereas transient noise clusters near zero. To jointly account for geometric continuity and semantic consistency, we define a spatial–semantic distance between any two Gaussian primitives $g_i$ and $g_j$:

$$D(i,j)={\gamma }_1{d}_{geo}(i,j)+{\gamma }_2{d}_{sem}(i,j),$$

(10)

where

$$d_{geo}(i,j)={\displaystyle \frac{\parallel {\mathit{\mu }}_i-{\mathit{\mu }}_j{\parallel }_2}{r_{scene}}},$$

(11)

and

$$d_{sem}(i,j)=1-{\displaystyle \frac{{\mathit{f}}_i·{\mathit{f}}_j}{\parallel {\mathit{f}}_i{\parallel }_2{\parallel {\mathit{f}}_j\parallel }_2}}.$$

(12)

where $r_{scene}$ is a scene-scale normalization factor, while ${\gamma }_1$ and ${\gamma }_2$ balance geometric and semantic similarity. The notation ${\parallel ·\parallel }_2$ denotes the Euclidean (L2) norm.

Based on this joint metric, we apply DBSCAN [73] to obtain instance-level clusters:

$$\mathcal{C}=\{{\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_K\},$$

(13)

where ${\mathcal{C}}_k\subseteq \mathcal{G}$ denotes the set of Gaussian primitives assigned to the k-th object cluster. This step transforms the dense Gaussian field into a set of semantically coherent and spatially compact object candidates. The complementary behavior of the two distance terms is key to reliable clustering. Geometric proximity alone tends to merge nearby but semantically distinct objects, while semantic similarity alone may group spatially separated instances of the same category. By combining both terms with ${\gamma }_1=0.6$ and ${\gamma }_2=0.4$, the joint metric $D(i,j)$ favors clusters that are simultaneously spatially contiguous and semantically consistent, so that DBSCAN with $\epsilon =0.12$ recovers coherent object candidates even in cluttered scenes. These clustering parameters exhibit moderate sensitivity: increasing ${\gamma }_1$ favors spatial compactness over semantic coherence, which can merge nearby but semantically distinct objects; increasing $\epsilon$ expands the neighborhood radius and merges more distant Gaussians into single clusters. The chosen values strike a balance that recovers coherent object candidates across diverse cluttered indoor scenes.

3.3.2. Object-Level Node Construction

Each object cluster ${\mathcal{C}}_k$ is abstracted into an object node $v_{obj}^k\in {\mathcal{V}}_{obj}$ with geometric, semantic, and spatial attributes.

The object center is defined as the opacity-weighted average of Gaussian centers:

$${\overline{\mathit{\mu }}}_k={\displaystyle \frac{{\sum }_{i\in {\mathcal{C}}_k}{o}_i{\mathit{\mu }}_i}{{\sum }_{i\in {\mathcal{C}}_k}{o}_i}}.$$

(14)

The object-level semantic feature is obtained by aggregating and normalizing the Gaussian semantic features:

$${\overline{\mathit{f}}}_k={\displaystyle \frac{{\sum }_{i\in {\mathcal{C}}_k}{\mathit{f}}_i}{{∥{\sum }_{i\in {\mathcal{C}}_k}{\mathit{f}}_i∥}_2}}.$$

(15)

To describe the occupied spatial extent, we construct an axis-aligned bounding box (AABB) [74]:

$$B_k=\left[\underset{i\in {\mathcal{C}}_k}{min}{\mathit{\mu }}_i,\underset{i\in {\mathcal{C}}_k}{max}{\mathit{\mu }}_i\right].$$

(16)

To obtain an explicit semantic label for language-level reasoning, we match the object feature ${\overline{\mathit{f}}}_k$ against a candidate text label set ${\mathcal{T}}_{obj}$. Let ${\mathit{e}}_t$ denote the CLIP text embedding of label $t\in {\mathcal{T}}_{obj}$. The object label is assigned by maximum similarity:

$${\ell }_k^{obj}=arg\underset{t\in {\mathcal{T}}_{obj}}{max}\mathrm{sim}({\overline{\mathit{f}}}_k,{\mathit{e}}_t),$$

(17)

where $\mathrm{sim}(·,·)$ denotes cosine similarity.

Accordingly, the k-th object node is represented as

$$v_{obj}^k=\left\{{\overline{\mathit{\mu }}}_k,{\overline{\mathit{f}}}_k,B_k,{\ell }_k^{obj}\right\}.$$

(18)

3.3.3. Region-Level Node Construction

Object nodes alone are insufficient for region-aware language grounding. Many navigation instructions refer not only to objects, but also to higher-level functional areas such as offices, kitchens, or reception zones. We therefore further group object nodes into region-level nodes to form a hierarchical region–object structure.

Let ${\mathcal{V}}_{obj}=\{v_{obj}^1,v_{obj}^2,\dots ,v_{obj}^K\}$, and let ${\pi }_{xy}\left({\overline{\mathit{\mu }}}_k\right)$ denote the projection of the object center onto the ground plane. Two object nodes are considered region-adjacent if they are spatially close (within a proximity threshold ${\tau }_r$) and belong to the same connected free space: this threshold is set according to the expected spatial scale of functional regions in indoor environments, large enough to group objects within the same room but small enough to avoid merging distinct functional areas.

$$A_{ij}^{reg}=\left\{\begin{array}{cc}1,\hfill & \parallel {\pi }_{xy}\left({\overline{\mathit{\mu }}}_i\right)-{\pi }_{xy}\left({\overline{\mathit{\mu }}}_j\right){\parallel }_2<{\tau }_r\mathrm{and}\mathrm{Conn}(i,j)=1,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(19)

where $\mathrm{Conn}(i,j)$ indicates that no major non-traversable structure separates the two objects. Based on this relation, spatially and topologically connected object nodes are merged into candidate region clusters:

$${\mathcal{R}}_m\subseteq {\mathcal{V}}_{obj},m=1,2,\dots ,M.$$

(20)

For each region cluster ${\mathcal{R}}_m$, the region center is defined as

$${\overline{\mathit{\mu }}}_m^{reg}={\displaystyle \frac{1}{|{\mathcal{R}}_m|}}\sum _{v_{obj}^k\in {\mathcal{R}}_m}{\overline{\mathit{\mu }}}_k,$$

(21)

and the aggregated region semantic feature is

$${\overline{\mathit{f}}}_m^{reg}={\displaystyle \frac{{\sum }_{v_{obj}^k\in {\mathcal{R}}_m}{\overline{\mathit{f}}}_k}{{∥{\sum }_{v_{obj}^k\in {\mathcal{R}}_m}{\overline{\mathit{f}}}_k∥}_2}}.$$

(22)

To assign a functional semantic label, we match the semantic composition of the region against a candidate region label set ${\mathcal{T}}_{reg}$. Let ${\mathit{e}}_c$ denote the text embedding of region label $c\in {\mathcal{T}}_{reg}$. The matching score is defined as

$$s(c,{\mathcal{R}}_m)=\sum _{v_{obj}^k\in {\mathcal{R}}_m}{w}_k·\mathrm{sim}({\mathit{e}}_{{\ell }_k^{obj}},{\mathit{e}}_c),$$

(23)

where ${\mathit{e}}_{{\ell }_k^{obj}}$ is the text embedding of object label ${\ell }_k^{obj}$ and $w_k$ is the object importance weight. Intuitively, this score measures how well the semantic composition of a region—weighted by its constituent objects—matches a candidate functional label such as ‘kitchen’ or ‘office’. The region label is then assigned by

$${\ell }_m^{reg}=arg\underset{c\in {\mathcal{T}}_{reg}}{max}s(c,{\mathcal{R}}_m).$$

(24)

Figure 2 provides an intuitive illustration of the region labeling process, where the weighted semantic composition of object nodes votes for the most compatible functional region label.

The resulting region node is represented as

$$v_{reg}^m=\left\{{\overline{\mathit{\mu }}}_m^{reg},{\overline{\mathit{f}}}_m^{reg},{\ell }_m^{reg},{\mathcal{R}}_m\right\}.$$

(25)

This process yields a hierarchical representation in which object-level semantics are elevated to region-level functional abstraction, providing a more suitable structure for language-guided retrieval.

3.3.4. Topology Relation Generation

After constructing the region and object nodes, we establish the relations required for global planning.

If an object node $v_{obj}^k$ belongs to region cluster ${\mathcal{R}}_m$, we create a containment edge

$$e_{m,k}^{inc}\in {\mathcal{E}}_{reg\text{-}obj}.$$

(26)

For navigable relations, we first project the Gaussian field to a 2D occupancy grid. For each planar location $\mathit{p}=(x,y)$, the occupancy score is defined as

$$M_{occ}\left(\mathit{p}\right)=\sum _{i\in \mathcal{G}}{o}_i·\mathbb{I}\left({\pi }_{xy}\left({\mathit{\mu }}_i\right)\in \mathrm{grid}(x,y)\right),$$

(27)

where $\mathbb{I}(·)$ is the indicator function. After thresholding with occupancy threshold ${\tau }_{occ}$, the grid is partitioned into free and occupied space. This threshold is selected based on the projected Gaussian opacity distribution to reliably distinguish navigable free space from occupied regions.

For any two object nodes u and v, let $L(u,v)$ denote the discrete line segment connecting their projected centers. We define the collision indicator as

$$\delta (u,v)=\left\{\begin{array}{cc}1,\hfill & \exists \mathit{p}\in L(u,v)\mathrm{s}.\mathrm{t}.M_{occ}\left(\mathit{p}\right)>{\tau }_{occ},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(28)

where the corresponding edge weight is

$$w_{u,v}=\left\{\begin{array}{cc}\parallel {\overline{\mathit{\mu }}}_u-{\overline{\mathit{\mu }}}_v{\parallel }_2,\hfill & \delta (u,v)=0\mathrm{and}\parallel {\overline{\mathit{\mu }}}_u-{\overline{\mathit{\mu }}}_v{\parallel }_2<d_{max},\hfill \\ \infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(29)

where the parameter $d_{max}$ denotes the maximum distance for direct edge construction. It is chosen to ensure that only physically plausible connections are encoded, avoiding edges that span disconnected spaces or jump across non-traversable obstacles. If $w_{u,v}<\infty$, a traversable edge is created:

$$e_{u,v}^{nav}\in {\mathcal{E}}_{obj\text{-}obj}.$$

(30)

Analogously, coarse region-level adjacency edges are constructed to form the navigation skeleton:

$${\mathcal{E}}_{reg\text{-}reg}.$$

Conceptually, as illustrated in Figure 3, this process transforms the dense 3D Gaussian field into a sparse navigable roadmap: the occupancy grid identifies walkable free space, collision detection prunes invalid shortcuts, and the remaining edges form a traversable graph that connects object nodes through collision-free paths. The final scene graph therefore combines semantic abstraction and traversable topology, providing a structured mid-level representation for instruction grounding and graph-based navigation.

3.4. LLM-Driven Global Navigation Planning

Given the hierarchical semantic scene graph $\mathcal{S}=(\mathcal{V},\mathcal{E})$, the next step is to map a free-form natural language instruction to a target region, a target object, and an executable path. We address this problem with a collaborative pipeline that combines LLM-based [67] semantic parsing, hierarchical cross-modal retrieval, and graph search.

3.4.1. Instruction Parsing and Hierarchical Query Generation

Given a natural language instruction, L, such as go to the wooden dining chair in the kitchen, the system first decomposes it into three components:

$$\mathcal{Q}\left(L\right)=\{L_{reg},L_{obj},L_{att}\},$$

(31)

where $L_{reg}$ denotes the region-level description, $L_{obj}$ denotes the object-level target, and $L_{att}$ denotes optional attribute constraints.

To enable cross-modal retrieval, these textual components are encoded into the same semantic space as the scene graph nodes. Using the CLIP text encoder ${\mathrm{\Phi }}_{text}$ and the same projection matrix ${\mathit{W}}_{comp}$ introduced in Section 3.2, each textual prompt $L_x\in \{L_{reg},L_{obj},L_{att}\}$ is mapped as

$${\mathit{f}}_{text}^x={\displaystyle \frac{{\mathrm{\Phi }}_{text}\left(L_x\right){\mathit{W}}_{comp}}{\parallel {\mathrm{\Phi }}_{text}\left(L_x\right){\mathit{W}}_{comp}{\parallel }_2}}.$$

(32)

This produces region, object, and attribute embeddings that are directly comparable with node semantic features in the hierarchical graph.

3.4.2. Hierarchical Region–Object Retrieval

Directly matching the full instruction against all graph nodes is both inefficient and prone to ambiguity. We therefore adopt a hierarchical retrieval strategy aligned with the graph structure.

At the region level, the similarity between the region prompt and each region node is computed as

$$S_{reg}(v_{reg}^m,L)={\overline{\mathit{f}}}_m^{reg}·{\mathit{f}}_{text}^{reg}.$$

(33)

The target region is selected by

$$v_{reg}^{*}=arg\underset{v_{reg}^m\in {\mathcal{V}}_{reg}}{max}{S}_{reg}(v_{reg}^m,L).$$

(34)

The retrieval space is then restricted to the object nodes contained in the selected region:

$${\mathcal{V}}_{obj}^{sub}=\left\{v_{obj}^k\mid v_{obj}^k\in {\mathcal{R}}_{*}\right\},$$

(35)

where ${\mathcal{R}}_{*}$ denotes the object set associated with $v_{reg}^{*}$. For each candidate object node, the object-level matching score is

$$S_{obj}(v_{obj}^k,L)={\overline{\mathit{f}}}_k·{\mathit{f}}_{text}^{obj}.$$

(36)

If attribute information is available, it is further incorporated to refine the ranking. The final target node is determined by

$$v_{target}=arg\underset{v_{obj}^k\in {\mathcal{V}}_{obj}^{sub}}{max}\left(S_{obj}(v_{obj}^k,L)+{\lambda }_{att}{S}_{att}(v_{obj}^k,L)\right),$$

(37)

where $S_{att}(v_{obj}^k,L)$ denotes the attribute-consistency score and ${\lambda }_{att}$ controls its contribution. This weight is chosen to give moderate emphasis to attribute-level matching without overwhelming the primary object-level semantic score.

This hierarchical retrieval scheme reduces the ambiguity commonly encountered in open-vocabulary indoor scenes with repeated object categories distributed across different regions.

3.4.3. Hierarchical Graph-Based Path Planning

Once the target node $v_{target}$ has been identified, the system plans a global path over the hierarchical scene graph. Let the current agent pose be $T_{curr}\in SE\left(3\right)$ and its position be ${\mathit{p}}_{curr}$. The nearest reachable graph node is first selected as the start node $v_{start}$.

Planning is then performed hierarchically. A coarse path is first searched on the region graph from the current region to the target region $v_{reg}^{*}$. A finer path is then computed within the object subgraph of the target region from the local entry point to the target object node $v_{target}$. Conceptually, as shown in Figure 4, this two-level search mimics human wayfinding: one first decides which functional regions to traverse (coarse region-level planning), and then determines the precise path to the target object within the destination region (fine object-level planning). At both levels, we employ A* search [75] with cost

$$f\left(n\right)=g\left(n\right)+h\left(n\right),$$

(38)

where $g\left(n\right)$ is the accumulated path cost and $h\left(n\right)$ is the heuristic term. We use the Euclidean distance between node centers as the heuristic:

$$h\left(n\right)=\parallel {\overline{\mathit{\mu }}}_n-{\overline{\mathit{\mu }}}_{target}{\parallel }_2.$$

(39)

The planner outputs a waypoint sequence

$$P=({\mathit{p}}_{curr},{\overline{\mathit{\mu }}}_{start},{\overline{\mathit{\mu }}}_1,\dots ,{\overline{\mathit{\mu }}}_{target}),$$

(40)

which can be directly passed to a low-level controller for execution.

3.4.4. Online Replanning and Pose Correction

To improve robustness in dynamic environments, we incorporate online replanning. If a local sensor detects that an edge $e_{u,v}$ has become blocked, its weight is updated as

$$w_{u,v}=\infty ,$$

(41)

and graph search is re-triggered from the current node. This enables local path revision without reconstructing the entire graph.

In addition, during long-horizon execution, pose drift may accumulate. To mitigate this effect, the frozen 3D Gaussian map can be used for lightweight online pose correction. Specifically, the current RGB image, depth map, and projected semantic feature map are aligned with the rendered observation under the current pose estimate through differentiable rendering, while keeping all Gaussian scene parameters fixed. By minimizing the resulting appearance, geometry, and semantic inconsistencies, the system refines the current pose online and improves localization stability during navigation.

Overall, the proposed planning module integrates LLM-based instruction parsing, hierarchical semantic retrieval, and graph-based path planning into a unified framework, mapping open-form natural language commands to executable global navigation paths.

4. Experiments

4.1. Datasets

We evaluate HOSG-Nav on the Replica dataset [76], a photo-realistic indoor benchmark widely used in neural scene representation research. Following the common evaluation protocol in related works [77], we use eight pre-rendered indoor sequences covering representative office- and room-level environments. Replica provides high-quality RGB-D observations, accurate camera trajectories, and fine-grained semantic annotations, making it well suited for evaluating open-vocabulary 3D scene modeling, hierarchical scene graph construction, and language-driven global navigation in a controlled setting. We also conducted preliminary experiments on ScanNet (5 real-world scenes); however, the results were less stable due to sensor noise, calibration drift, and incomplete geometry, which are known challenges for Gaussian-based scene reconstruction in uncontrolled environments. We therefore restrict the main evaluation to Replica and leave systematic real-world benchmarking to future work.

4.2. Implementation Details

All experiments are conducted on a workstation equipped with an NVIDIA L40 GPU. For open-vocabulary semantic encoding, we adopt OpenCLIP ViT-H/14 (laion2B-s32B-b79K) as the shared vision–language backbone. For instruction parsing, GPT-4 is used to decompose each navigation command into a region-level query, an object-level query, and optional attribute constraints.

For open-vocabulary 3D scene representation, the semantic and depth loss weights are set to ${\lambda }_{sem}=0.5$ and ${\lambda }_D=1.0$, respectively. These values provide a stable balance between semantic alignment and geometric regularization in indoor scenes. Optimization is performed using Adam for 30,000 iterations.

For hierarchical scene graph construction, Gaussian primitives with opacity lower than ${\tau }_o=0.1$ are removed before clustering. In DBSCAN, the geometric and semantic weights are set to ${\gamma }_1=0.6$ and ${\gamma }_2=0.4$, respectively, with neighborhood radius $\epsilon =0.12$ and minimum samples set to 20. For region-level grouping, the planar aggregation threshold is set to ${\tau }_r=1.5$ m. In topology generation, the occupancy grid resolution is 0.05 m and the maximum distance for direct edge construction is $d_{max}=2.0$ m. Unless otherwise stated, all experiments use these settings.

4.3. Evaluation of Open-Vocabulary Scene Representation

We first evaluate the proposed scene representation from two perspectives: novel-view rendering and open-vocabulary semantic understanding. The former assesses whether the semantic and depth extensions preserve the rendering capability of 3DGS, while the latter evaluates whether the learned representation improves semantic discriminability for downstream retrieval and navigation reasoning. We compare primarily against 3DGS and HOV-SG because they represent the most directly comparable approaches in the open-vocabulary 3D Gaussian representation and hierarchical scene graph navigation literature, respectively.

4.3.1. Novel-View Rendering Quality

We compare HOSG-Nav with vanilla 3DGS using PSNR, SSIM, and LPIPS. As shown in

Table 1. Quantitative evaluation of open-vocabulary scene representation on Replica. Higher PSNR/SSIM, mIoU, F-mIoU, and mAcc are better, while lower LPIPS is better. – indicates that the method is not designed for the corresponding evaluation. Values represent mean ± standard deviation across three independent training runs per scene. Bold indicates the best results.

	Novel-View Rendering			Open-Vocabulary Semantics
Method	PSNR	SSIM	LPIPS	mIoU	F-mIoU	mAcc
3DGS	28.79 ± 0.05	0.943 ± 0.003	0.065 ± 0.004	–	–	–
HOV-SG	–	–	–	0.231 ± 0.004	0.386 ± 0.006	0.304 ± 0.005
HOSG-Nav	34.66 ± 0.08	0.973 ± 0.002	0.096 ± 0.005	0.244 ± 0.005	0.455 ± 0.007	0.362 ± 0.006

, HOSG-Nav achieves 34.66 PSNR and 0.973 SSIM, outperforming 3DGS (28.79 and 0.943, respectively). This indicates that the proposed representation improves reconstruction fidelity and structural consistency rather than degrading the original rendering capability of 3DGS. Although 3DGS yields a slightly lower LPIPS value (0.065 vs. 0.096), HOSG-Nav provides a better overall balance between appearance quality and structural accuracy.

The qualitative results in Figure 5 and Figure 6 support this observation. Compared with the baseline, HOSG-Nav produces clearer boundaries, more coherent local structures, and more stable depth recovery in challenging regions such as wall surfaces, window frames, furniture contours, and occlusion transitions. These improvements are particularly important for subsequent scene graph abstraction and navigation planning, both of which rely on stable geometry and consistent spatial layout.

Overall, these results show that the proposed open-vocabulary 3D representation preserves and, in several respects, improves the rendering quality of the original 3DGS framework.

4.3.2. Open-Vocabulary Semantic Understanding

We next compare HOSG-Nav with HOV-SG on open-vocabulary semantic understanding using mIoU, F-mIoU, and mAcc. As shown in Table 1, HOSG-Nav consistently outperforms HOV-SG across all three metrics, improving mIoU from 0.231 to 0.244, F-mIoU from 0.386 to 0.455, and mAcc from 0.304 to 0.362. The gain in F-mIoU is particularly notable, indicating improved object-level semantic discrimination and cleaner foreground segmentation.

The qualitative results in Figure 7 further illustrate this advantage. HOSG-Nav produces more coherent predictions for major scene components and more complete boundaries for foreground objects, with fewer errors near object–background interfaces. This confirms that jointly optimizing semantics within the 3D Gaussian field yields a more stable and discriminative representation for downstream retrieval and navigation tasks.

Taken together, the results in Table 1 indicate that HOSG-Nav provides a unified 3D representation that is effective for both high-quality reconstruction and open-vocabulary semantic reasoning.

4.4. Evaluation of Hierarchical Semantic Scene Graph

We next evaluate the hierarchical semantic scene graph constructed on top of the continuous 3D scene representation. Since its main role is to provide an interpretable and navigable intermediate structure for downstream reasoning, we focus on qualitative analysis of its abstraction quality, hierarchical organization, and topological utility.

Figure 8 shows the graph construction results in the Office0 scene, illustrating a four-step abstraction process. Step 1: Starting from the continuous semantic 3D scene in Figure 8a, the dense Gaussian field preserves geometry, appearance, and semantics but is too granular for reasoning. Step 2: The method abstracts major physical entities into object-level nodes, as shown in Figure 8b. These nodes preserve the dominant semantic anchors of the scene while significantly reducing the redundancy of the dense Gaussian representation. Step 3: The object nodes are grouped into region-level nodes according to spatial proximity, connectivity, and semantic co-occurrence. As shown in Figure 8c, the resulting region graph captures functional areas such as workspace, display, and passage zones, while the region-level edges provide a coarse navigation skeleton. Step 4: Figure 8d visualizes the full hierarchical scene graph, in which region–object containment and inter-region adjacency are jointly modeled. Together, these four panels provide a step-by-step illustration of how HOSG-Nav transforms dense Gaussian primitives into a compact hierarchical graph suitable for language-guided navigation.

Overall, the visualization results demonstrate that HOSG-Nav transforms a continuous open-vocabulary 3D scene into a structured representation with explicit objects, functional regions, and navigable topology, providing a practical basis for language-guided target retrieval and global planning.

4.5. Global Planning and Navigation Performance

We further evaluate HOSG-Nav on natural-language-driven global planning and navigation. The goal is to assess whether the proposed framework can reliably map free-form language instructions to target regions, target objects, and executable navigation paths.

At the front end of the planning pipeline, GPT-4 is used for hierarchical instruction parsing. As illustrated in Figure 9, the parser converts diverse natural language expressions into structured region–object query pairs. This enables a unified interface for downstream retrieval and planning, regardless of whether the instruction emphasizes region, object, or both.

For quantitative evaluation, we use Retrieval-SR10 [%] and Navigation-SR [%]. Retrieval-SR10 measures whether the correct target can be recalled within the top-10 candidates, while Navigation-SR measures whether the system generates a correct and executable global path. Higher values indicate better performance.

As shown in

Table 2. Comparison of global planning and navigation performance. Higher values indicate better performance. Values represent mean ± standard deviation across three independent training runs per scene. Bold indicates the best results.

Method	Retrieval-SR10 [%]	Navigation-SR [%]
HOV-SG	31.48 ± 0.5	40.41 ± 0.6
HOSG-Nav	33.17 ± 0.8	42.26 ± 0.9

, HOSG-Nav outperforms HOV-SG on both metrics, improving Retrieval-SR10 from 31.48% to 33.17% and Navigation-SR from 40.41% to 42.26%. Each reported value represents the mean across three independent training runs per scene; standard deviations are provided in Table 2. The gain is consistent across both retrieval and navigation metrics, and it is meaningful given that HOV-SG is already a competitive hierarchical scene graph baseline. HOV-SG is selected as the primary baseline because it is the most directly comparable existing method that jointly addresses open-vocabulary 3D mapping, hierarchical scene graph construction, and language-guided indoor navigation. Expanding the comparison to include additional scene representation methods (e.g., LERF, OpenScene) and end-to-end navigation policies (e.g., SayPlan, SayNav) would further strengthen the empirical assessment.

The improvement stems from two factors. First, HOSG-Nav is built upon a jointly optimized open-vocabulary 3D Gaussian scene representation, where geometry, appearance, and semantics are learned in a unified field before graph abstraction. This leads to stronger local geometric–semantic consistency and reduces ambiguity in downstream target grounding. Second, the proposed region–object graph is explicitly aligned with the hierarchical decomposition of language instructions, enabling a tighter planning pipeline from language parsing to region retrieval, object retrieval, and graph search. This alignment is particularly beneficial for indoor object-centric navigation, where repeated object categories frequently occur across multiple functional areas.

These results confirm that HOSG-Nav more tightly integrates continuous 3D scene modeling, structured semantic abstraction, and language-guided planning into a unified global navigation framework.

4.6. Ablation Study

We finally conduct ablation experiments to analyze the contribution of the main components in HOSG-Nav, including the open-vocabulary 3D representation, the hierarchical scene graph, and the hierarchical planning strategy.

4.6.1. Ablation on Open-Vocabulary 3D Scene Representation

We first examine the effect of semantic and depth supervision in the 3D scene representation. Specifically, we compare three variants: w/o Semantic, which removes ${\mathcal{L}}_{semantic}$; w/o Depth, which removes ${\mathcal{L}}_{depth}$; and the Full model.

As reported in

Table 3. Ablation study on open-vocabulary 3D scene representation. Higher PSNR/SSIM, mIoU, F-mIoU, and mAcc are better, while lower LPIPS is better. Bold indicates the best results.

	Novel-View Rendering			Open-Vocabulary Semantics
Variant	PSNR	SSIM	LPIPS	mIoU	F-mIoU	mAcc
w/o Semantic	34.12	0.970	0.091	0.214	0.401	0.318
w/o Depth	33.41	0.967	0.103	0.236	0.432	0.347
Full	34.66	0.973	0.096	0.244	0.455	0.362

, removing semantic supervision primarily reduces semantic understanding performance, with clear drops in mIoU, F-mIoU, and mAcc. Removing depth supervision primarily affects reconstruction quality and also slightly weakens semantic performance, indicating that stable geometry benefits semantic lifting and multi-view consistency. The full model achieves the best overall balance, confirming that semantic and depth supervision are complementary.

4.6.2. Ablation on Hierarchical Scene Graph Construction

We next analyze the contribution of the graph design by comparing three variants: Geo-only Clustering, which constructs object nodes using only geometric distance; Flat Object Graph, which uses object-level nodes only; and the Full Hierarchical Graph.

Table 4. Ablation study on hierarchical scene graph construction. Bold indicates the best results.

Variant	Retrieval-SR10 [%]	Navigation-SR [%]
Geo-only Clustering	30.92	39.18
Flat Object Graph	32.11	40.87
Full Hierarchical Graph	33.17	42.26

shows that the full hierarchical graph performs best. Using geometry alone for clustering leads to the largest performance drop, suggesting that semantic consistency is necessary for robust object abstraction in open-vocabulary scenes. The flat object graph performs better, but still underperforms the full model, indicating that region-level abstraction provides useful coarse semantic context and reduces ambiguity during retrieval and planning.

4.6.3. Ablation on Hierarchical Planning Strategy

Finally, we evaluate the planning pipeline itself. We compare Flat Retrieval + A*, which directly matches the instruction to all object nodes; Region→Object Retrieval w/o Attribute, which adopts hierarchical retrieval but removes attribute-level constraints; and the Full Planning strategy.

As shown in

Table 5. Ablation study on hierarchical language-guided planning. Bold indicates the best results.

Variant	Retrieval-SR10 [%]	Navigation-SR [%]
Flat Retrieval + A*	31.36	39.95
Region → Object Retrieval w/o Attribute	32.74	41.63
Full Planning	33.17	42.26

, flat retrieval yields the weakest performance, confirming that direct matching over all object nodes is more susceptible to ambiguity. Introducing hierarchical region-to-object retrieval improves both retrieval and navigation success by narrowing the candidate space before object matching. Attribute-aware disambiguation further improves performance, particularly in cases where multiple similar objects exist within the same region.

Overall, the ablation results consistently support the design of HOSG-Nav. The performance gain is not attributable to a single module, but rather to the interaction among the open-vocabulary 3D representation, the hierarchical scene graph, and the structured planning strategy. We note that a systematic parameter sensitivity analysis—varying clustering thresholds, loss weights, and graph construction hyperparameters independently—would provide deeper insight into the robustness of each design choice. Such an analysis is left for future work, given the significant computational cost of repeated training runs per parameter configuration.

5. Discussion

HOSG-Nav demonstrates that tighter coupling among continuous 3D scene representation, hierarchical semantic abstraction, and structured planning yields competitive gains in language-guided global navigation. The improvement is not attributable to any single module; rather, semantic supervision improves feature discriminability, depth supervision stabilizes geometry, hierarchical graph construction reduces retrieval ambiguity, and LLM-based parsing aligns instructions with the region–object structure of the graph.

Nevertheless, several challenges remain. Replica serves as the primary benchmark in this study owing to its photo-realistic ground truth and widespread adoption in comparable works; nevertheless, the transfer of these findings to real-world scenes with sensor noise and incomplete geometry warrants further quantitative investigation. Preliminary trials on ScanNet (5 scenes) showed that real-world sensor noise and incomplete geometry degrade Gaussian reconstruction stability, which propagates to semantic lifting and planning. This aligns with the broader literature: comparable methods such as HOV-SG, ConceptGraphs, and LERF also report primary results on synthetic datasets. Cross-dataset generalization remains an open challenge. In addition, the empirical comparison is intentionally focused on HOV-SG and 3DGS, which are the most directly comparable methods in the open-vocabulary Gaussian and hierarchical scene-graph navigation literature; broadening this comparison to encompass alternative scene representations and end-to-end planners would further consolidate the findings. The framework also focuses on global planning without tight integration of local control, near-field perception, or manipulation-oriented execution. In addition, the reported ablations validate the relative contribution of each module under a consistent hyperparameter setting; a more exhaustive parameter sweep would be valuable for deployment in highly variable environments. Prioritizing real-world deployment, cross-dataset validation, expanded baseline comparisons, and exhaustive parameter sweeps should be the next step.

6. Conclusions

This paper proposed HOSG-Nav, a unified framework for natural-language-driven global navigation in complex indoor environments. The key distinction from prior work is the tight coupling of three levels: a continuous open-vocabulary 3D Gaussian field that jointly encodes geometry, appearance, and semantics; a region–object hierarchical scene graph abstracted directly from that field; and an LLM-driven planning module that performs hierarchical retrieval and graph search over the structured graph. The method integrates these components within a single pipeline, bridging continuous 3D scene modeling and structured semantic reasoning for navigation.

Experimental results on Replica demonstrate that HOSG-Nav achieves competitive performance in open-vocabulary scene representation, hierarchical semantic abstraction, and language-guided global navigation. The ablation results further confirm the effectiveness of semantic and depth supervision, hierarchical graph construction, and structured language-guided planning. These results establish a proof of concept for the proposed integration; extending them to larger real-world datasets, a broader set of baselines, and exhaustive parameter regimes will be essential for broader adoption.

Overall, HOSG-Nav takes a meaningful step toward unifying continuous 3D semantic mapping, hierarchical scene reasoning, and natural-language-driven navigation within a common framework.