A Visual Target Navigation Method for Quadrotor Based on Large Language Model in Unknown Environment

Highlights

What are the main findings?

• We propose a hierarchical prompt engineering framework that systematically integrates task instructions, scene prior knowledge, role definitions, and demonstration examples into structured prompt templates.

What are the implications of the main findings?

• We propose a novel integration method that bridges LLMs with quadrotor visual target navigation tasks, addressing the long-standing gap between natural language understanding and autonomous quadrotor navigation.

Abstract

This paper proposes a novel Large Language Model (LLM)-based visual target navigation framework for quadrotors in unknown environments. Leveraging the semantic knowledge of LLMs, our method enables autonomous exploration based on natural language instructions. We design an intelligent planner using specialized prompt templates that operates in two phases: first, deriving global search sequences via probabilistic inference; second, dynamically generating sub-goal waypoints by fusing visual observations with statistical priors and LLM-derived scene relevance metrics. The quadrotor then executes a progressive search via path planning algorithms. Simulation results indicate that our fused method outperforms single-modality baselines by approximately 20%. Furthermore, physical flight experiments demonstrate success rates of 56% in Cross-layout and 48% in T-shaped layout scenarios. These results, while reflecting the inherent challenges of perceptual occlusion and planning uncertainty, validate the feasibility and potential of the proposed framework in real-world applications.

1. Introduction

Compared to ground robots, quadrotors offer distinct advantages in conducting search missions in unknown environments, including rapid response, and superior maneuverability. These unmanned aerial vehicles (quadrotors) typically rely on onboard sensors (e.g., cameras) for environmental perception in navigation scenes. When deployed to explore hazardous unknown areas, such quadrotors require explicit instructions defining search tasks, such as target objects and designated destinations. Nevertheless, enabling quadrotors to locate arbitrarily specified targets based on human instructions in unknown environments remains a significant challenge. This difficulty arises fundamentally from the requirement for quadrotors to possess not only autonomous exploration capabilities but also advanced language comprehension and semantic reasoning skills [1,2].

Humans, by comparison, demonstrate remarkable superiority in searching unknown indoor environments. Leveraging rich prior knowledge of the physical world and robust logical reasoning capabilities, humans can quickly identify probable target locations and develop efficient search strategies. For example, when searching for specific items, humans prioritize relevant areas (e.g., refrigerators in kitchens or sofas in living rooms), which significantly reduces search time—an ability rooted in a profound understanding of environments and commonsense knowledge regarding object distribution. During navigation to target areas, humans perform scene matching via visual observations to confirm arrival before conducting localized searches. If the target remains undetected, they proceed to the next most probable location until the search is successful.

In the current quadrotor field, conventional approaches for achieving exploration in unknown environments typically construct environmental maps from visual observations prior to planning navigation routes based on geometric representations [3]. Owing to the absence of prior scene knowledge, such methods often entail extensive area-wide searches. In contrast, learning-based methods—designed to emulate human exploration of uncharted territories—leverage deep reinforcement learning (DRL) to directly optimize navigation strategies for locating targets of specified categories using real-time visual inputs [4]. While these learning-based approaches demonstrate superior performance in environmental exploration and target navigation once accurate target information is obtained, they are unable to fully exploit prior knowledge of the physical world to enhance search efficiency.

The emergence of large language models (LLMs) such as BERT [5], GPT-3 [6], and GPT-4 [7] has rendered this limitation addressable. These models not only demonstrate robust language comprehension capabilities to effectively interpret human intentions and generate contextually appropriate responses but also embed extensive prior knowledge of the physical world that provides valuable guidance for decision-making. Consequently, recent research has begun integrating LLMs with robotic systems—including quadrotors—to enhance their intelligent autonomy: specifically in understanding complex human instructions for task planning [8], and in converting natural language-described targets into structured planning languages to generate executable solutions [9,10]. However, most existing paradigms focus on ground-based robots constrained to 2D planes. Adapting these to quadrotors requires addressing unique 3D spatial characteristics. Unlike ground robots, quadrotors offer superior maneuverability for vertical exploration but face significantly higher spatial complexity, necessitating specialized adaptation of LLM planners for aerial agility.

This study focuses on LLM-based autonomous decision-making for quadrotor planning in unknown environments, leveraging LLMs’ inherent prior knowledge as directional guidance. Specifically, we first integrate vision-language models (VLMs) to generate real-time environmental descriptions, then adopt an LLM-driven active search decision-making mechanism to dynamically generate sub-target waypoints for quadrotor navigation. Our approach selects search directions using linguistic models and prior probabilities as knowledge bases, thereby reducing learning costs while enhancing the generalizability of scene prior knowledge.

However, we explicitly acknowledge the current limitations of this approach. In our physical flight experiments, the system achieved a maximum success rate of 56%. This modest performance highlights the significant difficulty of zero-shot exploration in unknown environments. The main challenges arise from the instability of LLM reasoning and the difficulty of perceiving small objects in complex 3D spaces. Despite these limitations, our work demonstrates that fusing LLM reasoning with statistical priors offers a viable pathway for autonomous aerial navigation.

Our contributions are summarized as follows:

• We propose a hierarchical prompt engineering framework that systematically integrates task instructions, scene prior knowledge, role definitions, and demonstration examples into structured prompt templates. This design effectively guides LLMs in task comprehension and reliably meets downstream quadrotor navigation requirements.
• We develop a novel active search decision-making approach that integrates dataset-derived prior probabilities with LLMs. By jointly leveraging these two knowledge sources for search direction selection, our method achieves efficient target search in unknown environments, and its effectiveness is further validated through physical experiments.
• We propose a novel integration method that bridges LLMs with quadrotor visual target navigation tasks, addressing the long-standing gap between natural language understanding and autonomous quadrotor navigation. This method is elaborated in Section 3, and its effectiveness is experimentally demonstrated in Section 4.

2. Related Works

2.1. LLM for Robot Control

Natural language generation, a core capability of large language models (LLMs), is widely used in task planning for its interpretability. However, natural language’s ambiguity can lead to unclear instructions, potentially causing task failures. To address this, Kim et al. [11] introduced a sub-goal planner that extracts key sub-goals and objects from high-level instructions, reducing ambiguity through hierarchical decomposition and external validation. Hu et al. [12] leveraged GPT-4 [7] and chain-of-thought reasoning to generate detailed plans for long-term tasks, utilizing LLMs’ world knowledge and reasoning. Zhu et al. [13] proposed a dual-system approach, distinguishing fast thinking for simple actions (e.g., grasping) from slow thinking for complex reasoning (e.g., rearrangement), enhancing both responsiveness and execution quality.

Large language models (LLMs) can generate planning formats ranging from structured Planning Domain Definition Language (PDDL) to flexible natural language. Structured formats like PDDL offer precision and interpretability, facilitating robot execution, while natural language provides adaptability but may require complex controllers for interpretation. Liu et al. [8] proposed converting planning problems into PDDL via LLMs, parsing them with a PDDL planner, and translating results back into natural language, enhancing interpretability and usability. Yang et al. [14] extended this by transforming parsed PDDL into finite state automata, creating a full planning-control loop. Zhou et al. [15] introduced an iterative optimization method, using LLMs to generate, score, and refine PDDL-based plans, improving solution quality and reliability through automated evaluation.

2.2. Visual Target Navigation

Recent studies have also applied LLMs to visual target navigation tasks, demonstrating their effectiveness in this context. Majumdar et al. [16] employed the BERT model to construct a scoring function between language instructions and paths, aiding embodied agents in navigation. Shah et al. [17] introduced LM-NAV, which extracts landmark names from instructions using a language model, matches these landmarks in the world through an image-language model, and then navigates to them using a navigation model. Xie et al. [18] improved the path generation of Rapidly-exploring Random Tree (RRT) by introducing LLM for semantic relevance measurement to achieve outdoor navigation for quadrotors. Chen et al. [19] utilized the prior knowledge of LLM for semantic inference to obtain scene labels, achieving 3D scene understanding for robots. Yu et al. [20] introduced LLM into the problem of visual target navigation, enabling the use of language to find relevant boundaries from semantic maps as long-term goals and effectively explore the environment.

3. Proposed Method

3.1. Overview

To address the challenge of quadrotor visual target navigation guided by natural language instructions in unknown environments, we propose an active search behavior decision-making method. In this work, a large language model (LLM) serves as a search sequence planner to determine the order of exploration. The LLM’s output combined with prior probabilistic knowledge and semantic relevance metrics is used to guide the quadrotor’s search directions.

Figure 1 illustrates the overall framework of our approach. Consider a scenario where a quadrotor searches for a specified target inside a house. The user provides a natural language instruction containing the target name. Based on this instruction, the quadrotor generates a task plan to find the target: it first constructs an initial search sequence using prior map information and the target name, then autonomously selects search directions by fusing real-time visual observations with semantic guidance from the LLM. Corresponding sub-goal points are generated accordingly, and the quadrotor iteratively navigates toward them using a path planning algorithm until the target is located. Within this framework, there are two extremely important modules that support the autonomous object-seeking and exploration of quadrotor.

(1) Observation Module. To enable autonomous navigation in unknown environments, it is crucial to perceive raw environmental observations and extract critical information for downstream reasoning. This necessitates two key perceptual capabilities: scene recognition, which identifies objects within the scene, and mapping, which locates these objects to facilitate path planning.To achieve these capabilities, the framework integrates two locally deployed visual language models:

Recognize-anything Model [21]: This model constructs scene descriptions by identifying object names from RGB images.

GroundingDINO Model [22]: This model detects object names and their corresponding bounding boxes from RGB images and a list of object names.

The module leverages these models to obtain object names and bounding boxes, combining this information with depth images to calculate object positions through coordinate transformation. Its primary function is to construct a semantic map using RGB and depth images, recording discovered objects and their positions.

(2) Memory Module. This module is designed to store essential information and maintain data acquired within the scene. It manages both static and dynamic data: static information includes prompt templates and prior probability lists, while dynamically updated information encompasses the quadrotor’s historical path data, semantic maps, and records of previously encountered objects.

3.2. Priori Information for Search Order

This section focuses on the decision-making problem of search sequence in unknown indoor environments. By leveraging the commonsense reasoning capabilities of large language models (LLMs), we guide the model to infer the most probable regions where the target object may be located and generate a priority-based search list. This approach enables the quadrotor to prioritize searching high-probability regions, avoiding resource wastage from blind exploration. It provides a critical decision-making foundation for determining search directions and optimizing efficiency.

To effectively guide the large language model (LLM) in generating a realistic search sequence tailored to indoor search tasks, a structured prompt was designed. Details of the prompt design can be found in Appendix A.1. The prompt consists of two key elements: role description and example description, using a single-interaction mode without requiring a continuous dialogue mechanism. The prompt construction method is as follows:

1. Role description design

The role description includes task scenario definition and capability boundary constraints, clearly specifying the quadrotor system’s responsibilities and decision-making authority.

(1) Task scenario description

The task scenario serves as a critical foundation for prompt construction. By clarifying task requirements and contextual background, it provides a systematic execution framework for subsequent experiments. A comprehensive task scenario definition should include the following core elements:

$$Tas{k}_{-}Scene=\left\{R_{Des},A_{Sce}\right\}$$

(1)

where $R_{Des}$ is the role definition, which specifies the function of the large language model in the task, and $A_{Sce}$ is the application scenario, which defines the specific experimental environment.

For the task requirement of generating global search sequences, the role is defined as a sequencing assistant; the application scenario focuses on indoor environments to evaluate the spatial distribution probability of search targets.

(2) Capability boundary constraints

These constraints guide the LLM’s behavior, mitigate potential risks, and ensure system reliability. A complete capability boundary definition includes:

$$Capabilit{y}_{-}Boundaries=\left\{O_{Per},O_{Pro}\right\}$$

(2)

where $O_{Per}$ is the permitted operation, which clearly defines the operation behaviors that can be invoked by the large language model when solving the target task, and ensures that all the operations comply with the system safety specification and task requirements. $O_{Pro}$ is the prohibited operation, which strictly restricts the operation behaviors that may cause safety hazards.

For the task of generating a global search order, the permitted operation is limited to the use of predefined region labels in the input list, and the output result must be limited to the set of input region labels and include all the labels; the prohibition operation prohibits the generation of any region labels outside the list, the omission of the predefined region labels, or hypothetical reasoning beyond the description of the environment. This design not only fully utilizes the reasoning advantages of the large language model, but also effectively avoids its potential risk of “illusion”.

2. Example description design

The example description establishes the interaction paradigm between the user and the LLM. A complete interaction format includes:

$$Interactio{n}_{-}Protocol=\left\{I_{Spe},O_{For}\right\}$$

(3)

where $I_{Spe}$ is the input specification, which explicitly defines the type of requirement contained in the user’s instruction and ensures that there is no obvious ambiguity in the instruction, and $O_{For}$ is the output format, which serves the task requirements.

For the task of generating global search sequences, the input specifications clearly define the format and definitions of input elements, enabling the large language model to distinguish between the target object names and search region label lists in the input components. The output format is designed for automated parsing, providing an easily extractable structure to guide the model’s outputs while facilitating region label extraction and reducing the likelihood of program errors. An example output format is shown below:

Region Label 1 - Region Label 2 - … - Region Label n

After generating the search sequence, the quadrotor follows the priority order to scan each region. Within each region, the quadrotor uses its visual perception module to identify the target object. If the target is not found, the quadrotor proceeds to the next higher-priority region until the search is complete.

3.3. Search Direction Decision-Making

Upon obtaining the regional search sequence, the quadrotor needs to determine the optimal moving direction based on current perceptual information until reaching the target location. To achieve this, this paper adopts two complementary methods to evaluate which direction within the current field of view is most likely to lead to the target area: data-driven prior probability analysis and semantic-understanding-based LLM reasoning.

(1) Prior probability analysis: Based on large-scale datasets, this method assigns prior probabilities to each region label through statistical methods, representing the likelihood of the target object appearing in that region. During the quadrotor’s search process, by detecting object categories in the current field of view and combining them with pre-calculated regional probability distributions, it can compute regional relevance scores for each direction.

(2) LLM reasoning: To compensate for the limitations of pure probability analysis in complex scenarios, this paper introduces LLM for semantic relevance assessment. By inputting detected object categories (such as “refrigerator”, “sofa”, “desk”, etc.) from various directions, the LLM can determine the most probable corresponding region labels based on its rich semantic knowledge base, and output scores indicating the likelihood of target object presence. This method fully utilizes the LLM’s deep understanding of semantic relationships between objects and scenes, effectively handling special scenarios or complex environments not covered in the dataset. By combining prior probability analysis with LLM reasoning, comprehensive evaluation of search directions can be achieved from both statistical and semantic understanding dimensions.

The pseudo-code of the search direction autonomous decision-making method is shown in the Algorithm 1. The method first determines the search order of the region based on the list of search targets and region names, and then acquires visible light images in different directions. In this paper, we mainly use the Recognize-anything model for target recognition. Based on the recognition results, the scene likelihood ranking and relevance score under the a priori probability and the scene likelihood ranking and relevance score generated by the large language model are calculated respectively. Finally, the results of the a priori probability and the large language model are fused to determine the prioritized search direction, and the subgoal point locations are computed and generated by Equation (4), which provide inputs for the subsequent local planning of the search path.

$$subgoa{l}_p=cu{r}_p+\lambda \ast senso{r}_r\ast \overrightarrow{D}$$

(4)

where $cu{r}_p$ is the current position, $subgoa{l}_p$ is the subgoal position, $\lambda$ is a fixed value, $senso{r}_r$ is the sensor sensing range, and $\overrightarrow{D}$ is the search direction vector per unit length.

Algorithm 1 Search Direction Decision-Making

1: input: $img=[im{g}_0,im{g}_1,im{g}_2]$,$targe{t}_{-}name$,$x_{satrt}$ 2: output:  $searc{h}_{-}direction$ 3: initialize $are{a}_{-}list$ 4: $searc{h}_{-}order=ll{m}_{-}pr{e}_{-}order(are{a}_{-}list,targe{t}_{-}name)$ 5: for each $im{g}_i$ in $img$ do 6:     $nam{e}_{-}list=recognize\left(im{g}_i\right)$ 7: end for 8: $p_{-}score,p_{-}order=P_{-}compute\left(nam{e}_{-}list\right)$ 9: $l_{-}order=ll{m}_{-}order\left(nam{e}_{-}list\right)$ 10: $l_{-}score=ll{m}_{-}score\left(nam{e}_{-}list\right)$ 11: $f_{-}order,f_{-}score=fusio{n}_{-}order(p_{-}score,p_{-}order,l_{-}order,l_{-}score)$ 12: $searc{h}_{-}direction=directio{n}_{d}ecide(f_{-}order,f_{-}score,searc{h}_{-}order)$ 13: return  $searc{h}_{-}direction$

Where $are{a}_{-}list$ is a predefined list of candidate region categories defining the semantic boundaries of the environment, $p_{-}score$ and $p_{-}order$ are the semantic relevance scores and search order, respectively, for the current candidate direction, obtained based on prior probabilities, while $l_{-}score$ and $l_{-}order$ are generated by the LLM. $f_{-}order$ is the final output sequence of search directions, sorted based on the fused decision metrics.

3.3.1. Prior Probabilities for Search Direction

To quantify the correlation between region categories and target objects, we construct a “Region Category - Object Name” prior probability list based on the ADE20K dataset [23]. Unlike previous methods that rely on small-scale custom data or unstable model predictions, we leverage the dense, pixel-level expert annotations from ADE20K to ensure statistical reliability. Given that our task focuses on indoor environments, using the entire dataset would introduce noise. Therefore, we applied a semantic filter to the ADE20K dataset to construct a specialized subset. We selected 3000 images specifically annotated with indoor scene labels (e.g., kitchen, bedroom, living room, office). This subset covers the majority of common indoor structures and household objects (e.g., refrigerator, bed, sofa, desk).

Based on the ground-truth annotations of these 3000 images, we calculate the prior probability using statistical co-occurrence frequency. Let a denote a target object and b denote a region category. The prior probability $p(a\mid b)$, representing the likelihood of object a appearing in region b, is calculated as:

$$p(a\mid b)={\displaystyle \frac{N_{a\cap b}}{N_b}}$$

(5)

where $N_b$ is the total number of images labeled as region b in our subset, and $N_{a\cap b}$ is the number of those images that also contain the object a. This data-driven approach allows the quadrotor to prioritize search directions based on robust commonsense knowledge extracted from real-world distributions.

After obtaining scene descriptions from different directions, to reduce the impact of common objects (e.g., doors and windows) on scene recognition, this study adopts the information entropy filtering method proposed in ref. [20] to calculate the entropy value of each object category. The objects with the highest information content are selected based on their entropy values, which are then used to compute the scene’s posterior probability. The entropy calculation formula is as follows:

$$H_{o_i}=-\sum _{t_j\in L_T}p\left(t_j\mid o_i\right)logp\left(t_j\mid o_i\right)$$

(6)

where $o_i\in L_o$ represents the target object category (e.g., refrigerator, bed), and $t_j\in L_T$ denotes the scene category (e.g., kitchen, bedroom). $p(t_j\mid o_i)$ indicates the conditional probability of target object $o_i$ appearing in scene $t_j$, obtained through normalization processing based on current scene categories.

As can be seen from Equation (6), objects with greater information content have lower entropy values $H_{o_i}$. The entropy $H_{o_i}$ measures the distribution uncertainty of target object $o_i$ across different scenes. A lower entropy value indicates higher concentration of the object in specific scenes and stronger scene discriminative power; conversely, higher entropy reflects more uniform distribution across different scenes and weaker scene differentiation.

Based on the above method, this paper selects several key objects with the lowest information entropy (i.e., strongest discriminative power) and calculates the likelihood scores for each scene category through their conditional probability distributions, namely the posterior probability:

$$P\left(t_j\right)=\prod _{k=1}^{K}p\left(t_j\right|o_k)$$

(7)

where $\left\{o_1,o_2,\dots ,o_K\right\}$ represents the set of key objects selected through information entropy filtering. Since the number of objects detected from different viewpoints may vary, the posterior probability $P\left(t_j\right)$ needs to be normalized using Equation (8) to obtain the final likelihood score $S\left(t_j\right)$. Based on these likelihood scores, the scene probability rankings for the three viewpoints are determined as $p{o}_1$, $p{o}_2$, and $p{o}_3$.

$$S\left(t_j\right)={\displaystyle \frac{P\left(t_j\right)}{{\displaystyle \sum _{t_i\in L_T}}P\left(t_i\right)}}$$

(8)

3.3.2. LLM for Search Direction

After obtaining the regional search order, the corresponding region labels for each direction are determined through scene matching and ranked based on matching scores. If the region labels differ, the next search direction is determined according to the predefined order; if they are the same, it is considered that the target region has been reached, and further distinction between directions is made using relevance scores (measuring the probability of the target object’s presence). This process involves two key tasks: region label matching score ranking and relevance score calculation, leveraging the semantic understanding capabilities of large language models. Similar to Section 3.2, the prompt design consists solely of role descriptions and example descriptions. Details of the prompt design can be found in Appendix A.2.

1. Region label matching score ranking

(1) Role description design

(a) Task scenario description

The task scenario consists of two elements: role definition and application scenario. In this paper, the role definition is a sorting assistant; the application scenario is oriented toward scenario matching, evaluating the matching degree between scenario descriptions and regional tags.

(b) Capability boundary constraints

The capability boundary includes two elements: permitted operations and prohibited operations. In this paper, permitted operations are limited to using predefined regional tags from the input list, and the output results must be confined to the input set of regional tags while including all tags; prohibited operations forbid generating any regional tags outside the list or omitting predefined regional tags.

(2) Example description design

The example description consists of two elements: input specifications and output format, clarifying the interaction form with the large language model. In this paper, the input specifications enhance the definition of input elements, enabling the large language model to distinguish between the scene description list and the region label list in the input elements of this method. The output format serves the automated parsing of downstream tasks, providing an output format that facilitates extraction. An example of the output format is as follows:

Direction n: Region Label 1 - Region Label 2 - …… - Region Label n.

2. Relevance score calculation

(1) Role description design

(a) Task scenario description

The task scenario consists of two elements: character definition and application scenario. The character definition is a scoring assistant; the application scenario is to measure the relevance level of the searched target to the current scenario. After the quadrotor arrives at the area, when the region labels in each direction are determined to be consistent, this score is used to determine the quadrotor’s next search direction.

(b) Capability boundary constraints

The capability boundary includes two elements: permitted operations and prohibited operations. In this paper, permitted operations are limited to measurement by scoring only; prohibited operations forbid the use of any form of measurement other than scoring.

(2) Example description design

The example description consists of two elements: input specifications and output format, which clarify the interaction form with the large language model. The input specifications enable the large language model to distinguish between the target object name and scene descriptions in different directions within the input elements of this method. The output format serves automated parsing, measured by a score between [0, 10]. An example of the output format is as follows:

Direction $n:x,x\in [0,10]$.

From an implementation perspective, the two aforementioned issues can be combined into a single prompt to guide the large language model’s reasoning. However, in practice, this approach presents significant problems. Firstly, the merged prompt may make it difficult for the large language model to handle both tasks simultaneously, often resulting in outputs that fail to meet requirements—either due to irregular formatting or incomplete answers. Secondly, complex prompts increase the model’s reasoning burden, reducing the accuracy and consistency of its outputs. Therefore, it is preferable to address these two issues separately in the interaction. This step-by-step approach not only ensures that the large language model focuses on a single task, producing more standardized results, but also enhances the efficiency and reliability of the interaction. By sequentially obtaining match rankings and relevance scores, this information can be integrated more precisely, thereby providing more dependable support for subsequent decision-making.

3.3.3. Fusing Prior Probabilities with LLM for Search Direction

Both methods exhibit notable limitations when used independently. The prior probability-based approach has two key drawbacks: (1) Its predefined “region-object” probability coverage is limited, struggling with unfamiliar environments or uncertain targets; (2) It fails to distinguish between identical region labels, providing only categorical probabilities without direct object associations. The LLM-based method, while demonstrating superior semantic understanding, lacks quantitative discrimination for identical labels. Its relevance scores offer limited practical value due to weak correlation with region labels.

To address these limitations, we propose a fusion strategy: the prior probability method provides quantitative baselines, while the LLM method supplements judgment for uncovered scenarios. Crucially, this hybrid enables relevance-based navigation decisions. The fused scoring formula is:

$$s=a\ast are{a}_{-}scor{e}_{-}p+b\ast are{a}_{-}scor{e}_{-}l$$

(9)

where s is the fusion probability score, $area\_score\_p$ is the probability score $S\left(t_j\right)$ from Section 3.3.1, and $area\_score\_l$ is the ranking-based score from Section 3.3.2. Regarding the fusion weights, we empirically set $a=0.4$ and $b=0.6$. We assign a higher weight b to the LLM because it performs real-time reasoning based on the specific visual context, whereas the prior probability a serves as a statistical baseline to stabilize the decision.

Algorithm 2 outlines the fusion strategy, which generates labeled regions and relevance scores for each direction. Inputs include prior probabilities and LLM-derived rankings with scores, while outputs are fused rankings and scores. If the prior and LLM labels match, they are adopted directly; otherwise, the highest-scoring fused label is selected. The fused results guide the search direction, with sub-goal points generated based on sensor range, advancing the search through local path planning.

Algorithm 2 Fusion Strategy

1: $$\begin{gathered} \mathbf{input}\mathbf{:} p_{-}score=[p{s}_0,p{s}_1,p{s}_2],p_{-}order=[p{o}_0,p{o}_1,p{o}_2],l_{-}order= \\ [l{o}_0,l{o}_1,l{o}_2],l_{-}score=[l{s}_0,l{s}_1,l{s}_2] \end{gathered}$$ 2: output:  $f_{-}order=[f{o}_0,f{o}_1,f{o}_2],f_{-}score=[f{s}_0,f{s}_1,f{s}_2]$ 3: initialize $are{a}_{-}list,are{a}_{-}scor{e}_{-}p,are{a}_{-}scor{e}_{-}l,are{a}_{-}scor{e}_{-}f,a,b$ 4: $are{a}_{-}scor{e}_{-}p=index(are{a}_{-}list,p_{-}score)$ 5: $are{a}_{-}scor{e}_{-}l=assig{n}_{-}score(are{a}_{-}list,l_{-}order)$ 6: for  $i=0:2$  do 7:     if $same(p{o}_{i0},l{o}_{i0})$ then 8:         $f{o}_i=p{o}_{i0}$ 9:         $f{s}_i=l{o}_{i0}$ 10:   else 11:      for each $s_i$ in $are{a}_{-}scor{e}_{-}f$ do 12:          $s_i=a\ast are{a}_{-}scor{e}_{-}{p}_i+b\ast are{a}_{-}scor{e}_{-}{l}_i$ 13:      end for 14:       $f{o}_i=ma{x}_{-}score(are{a}_{-}list,are{a}_{-}scor{e}_{-}f)$ 15:       $f{s}_i=indx(f{o}_i,l{o}_i)$ 16:   end if 17: end for 18: return  $f_{-}order,f_{-}score$

4. Experiments

To comprehensively evaluate the proposed method, we employed a multi-stage framework with heterogeneous localization setups serving distinct validation purposes. The simulation phase utilizes ground-truth positioning to benchmark LLM reasoning capabilities and decision logic in a noise-free environment. In contrast, physical experiments leverage both a high-precision motion capture system and a LiDAR-based odometry system to verify the framework’s adaptability across different sensor modalities. While variations in accuracy and latency across these setups preclude direct cross-scenario comparison, this design demonstrates that our high-level semantic decision-making remains effective regardless of the underlying localization hardware. Consequently, all performance comparisons are conducted strictly within each specific experimental setting.

4.1. Airsim-Based Simulation Experiments

We conducted simulations using AirSim, an open-source, cross-platform simulator based on Unreal Engine. Supporting both software- and hardware-in-the-loop simulations with PX4/ArduPilot, AirSim excels in physical and visual simulations for robots like quadrotors, leveraging game-engine rendering for superior visual effects. Our study employs a simulated quadrotor, as shown in Figure 2.

Thanks to the extensive environmental resource library of Unreal Engine, AirSim offers a variety of simulation scenarios. For this paper, we selected an indoor simulation environment, as illustrated in Figure 3.

The Figure 4 shows an example of searching for a computer without location hint. Based on prior room category information, a pre-determined search order is generated. Then, based on visual observations, the degree of matching with room categories is used to generate sub-goal points. Through path planning algorithms, the quadrotor gradually approaches and finally find the target.

Due to the occasional instability in the output of LLM, which can lead to undefined or unexecutable generated code, and an increasing error rate as task complexity increases, we evaluate performance using both code executability and task success rates. To more comprehensively evaluate task performance, we have introduced two new evaluation metrics: Success weighted by Path Length (SPL) and Distance to Goal (DTG) at the end of the task. The definition of SPL is as follows [24]:

$${\displaystyle \frac{1}{N}}\sum _{i=1}^N{S}_i{\displaystyle \frac{l_i}{max(l_i,p_i)}}$$

(10)

where N is the total number of experiments, $S_i$ indicates whether the i-th experiment was successful (1 for success, 0 for failure), $l_i$ is the Euclidean distance between the starting position and the success position in the i-th experiment, and $p_i$ is the actual trajectory length of the i-th experiment. It is important to note that we have adjusted the definition of $l_i$ in this paper. In the original definition, $l_i$ was the shortest trajectory length between the starting position and the success position. However, in real-world scenarios, obstacles may exist between the two points, making it difficult to calculate the shortest path. Therefore, we use the Euclidean distance as a substitute. The DTG is calculated as the average Euclidean distance between the quadrotor and the target at the end of the task:

$${\displaystyle \frac{1}{N_s}}\sum _{i=1}^{N_s}{D}_i$$

(11)

where $N_s$ is the total number of successful experiments, and $D_i$ denotes the Euclidean distance between the quadrotor and the target position at the end of the i-th experiment.

The task instruction was “go to find the X,” where X represents an object in the experimental scene. Each method was tested 25 times, and the results are shown in

Table 1. Performance comparison of the fusion method.

Method	Success Rate	SPL	DTG (m)
Qwen2 + Prior probabilities (Proposed)	60%	0.3816	2.4618
Qwen2	44%	0.2750	2.9901
Prior probabilities	36%	0.2196	2.7473
Ref. [25]	40%	0.2752	2.8949

. The fused method achieved approximately a 20% higher success rate compared to the pure LLM-based, prior-based, and ref. [25] approaches, demonstrating its superior effectiveness. These comparisons effectively serve as an ablation study for the fusion weights in Equation (9): the prior-based baseline represents the extreme case of $a=1,b=0$ (36% success), while the pure LLM-based baseline represents $a=0,b=1$ (44% success). The superior performance of the fused method ($a=0.4,b=0.6$, 60% success) validates the necessity of balancing statistical stability with real-time semantic reasoning. Additionally, the SPL and DTG metrics indicate that the fused method provides better search-direction decisions, resulting in shorter paths and closer proximity to the target.

The performance of the method from ref. [25] aligns closely with our LLM-only baseline, as both rely heavily on the semantic reasoning capabilities of LLMs. However, a key distinction lies in the prompt engineering: ref. [25] employs a flat, conversational role-playing prompt (e.g., “Picture yourself as a robot…”) to generate actions directly, which can be prone to instability in complex scenarios. In contrast, our method utilizes a Hierarchical Prompt Framework with explicit capability boundaries to output structured relevance scores. While this structured prompting yields a slight performance edge over ref. [25] in the pure LLM setting, the decisive advantage of our Fused Method (60%) stems from integrating these semantic scores with statistical prior probabilities. This fusion mechanism effectively stabilizes the decision-making process, compensating for the hallucinations and uncertainties inherent in both ref. [25] and pure LLM approaches.

To investigate the impact of large language models on the method, this study compared two different models: Alibaba’s Qwen2.5-VL and Baidu’s ERNIE-3.5. Note that since ERNIE-3.5 is primarily designed for Chinese applications, Chinese prompts were used during interaction. Accordingly, all prompt content was translated into Chinese, and the output of the perception module was also adapted to Chinese. Each method was tested 25 times, with the experimental results shown in

Table 2. Success Rates with Different Language Models.

Model	Success Rate	SPL	DTG (m)
Qwen2	60%	0.3816	2.4618
Qwen2.5-VL	64%	0.3859	2.2545
ERNIE-3.5	52%	0.3064	2.6239

.

The results show that Qwen2 and Qwen2.5-VL performed similarly, with the latter slightly outperforming the former. In comparison, ERNIE-3.5’s results were slightly worse. Analysis revealed that this difference stems from variations in the ranking outputs between the Qwen series and ERNIE-3.5, leading to different subgoal point generation under the same conditions. Since individual decisions cannot be directly evaluated, only the overall task results could be analyzed. Thus, when using Chinese prompts, ERNIE-3.5 performed slightly worse than the Qwen series with English prompts. This discrepancy may arise from semantic deviations during translation, affecting decision accuracy, or from differences in model capabilities. The semantic deviations mainly originate from variations in object name translations between Chinese and English. For example, the Chinese word is a general term, while English distinguishes between “desk” and “table,” leading to inconsistent scene inferences and thus affecting decision outcomes.

To further validate the independent contribution of the execution module (local path planning) to the overall system performance, we conducted comparative ablation experiments in the simulation environment. Keeping the LLM-based decision module constant, we compared our proposed step-by-step RRT algorithm against a standard Probabilistic Roadmap Method (PRM) [26] and a No-Planning baseline using the carrot-chasing algorithm [27] for direct flight control.

As shown in

Table 3. Results of Active Search Strategies under Different Path Planning Algorithms.

	Success Rate	SPL	DTG
$RR{T}_{-}sbs$	60%	0.3816	2.4618
$PRM$	40%	0.2464	5.3537
$\mathit{carro-chasing}$	24%	0.2051	4.6973

, the step-by-step RRT method achieved a success rate of 60%, significantly outperforming PRM (40%) and the No-Planning baseline (24%). The results indicate that even with correct high-level decisions from the LLM, a robust local planner is critical for obstacle avoidance and flight continuity.

4.2. Physical Flight Experiments

4.2.1. Customized Indoor Scenes

The actual quadrotor built for the experiments in this paper is shown in Figure 5, with a wheelbase of 250 mm in length, a height of 298 mm, and a weight of 0.984 kg (without battery).

The quadrotor is equipped with CUAV’s V6X flight control module, offering high stability and scalability, along with a high-performance processor and precision sensors. The algorithm module runs on a Jetson Orin NX host (RTSO-3002 carrier board), an ARM-based system known for its low power consumption, compact design, high computational power, and developer-friendly features. Prior to use, the host requires system flashing with Jetpack 5.1.2, compatible with Ubuntu 20.04 and Python 3.8, meeting the minimum requirements for large model operations. For imaging, the Intel Realsense D435i camera (Intel Corporation, Santa Clara, CA, USA) is employed, providing high-precision depth data and high-resolution color images in a compact, energy-efficient design. It features a USB-C 3.1 interface for fast data transfer, fulfilling all experimental needs.

In the real experiments, the Nokov system was used to achieve the positioning of the quadrotor. Due to the use of a motion capture system for positioning, the flight experiment area must remain within the camera’s field of view. Consequently, various indoor scenarios were constructed within the motion capture system’s experimental area. Indoor environments often contain numerous objects, some of which are difficult to incorporate due to their complexity. To address this, virtual background images were used as substitutes, enhancing the richness of the physical experiment scenarios and providing additional contextual information. For this study, four indoor scenarios were created, representing a kitchen, a reading room, an office, and a bathroom. Small physical objects and virtual background images were used to construct these environments, with walls (1.8 m high and 0.6 m thick) installed to separate the scenes. Two layouts were designed for the physical experiments, as shown in Figure 6, arranging the four rooms in a cross layout and a T-shaped layout, while maintaining consistent interior arrangements across all rooms.

The experiment utilized Alibaba’s Qwen2 model for human-machine interaction. Due to the model’s substantial size making local deployment impractical, we established a local area network via hotspot to enable remote API access from the onboard computer for Qwen2 interaction. The workflow, shown in Figure 7, involves launching two terminals for text instruction reception and LLM interaction. A task instruction, such as “go to find the coffee cup”, is entered and published to the ROS topic $/inpu{t}_{-}stream$. The LLM interaction module subscribes to this topic, generates a prompt, and sends it to Qwen-2. Upon receiving a response, it publishes long-term control commands to $/contro{l}_{-}cmd$. The task management module executes these commands, prioritizing areas based on the likelihood of containing the target item and using a path planning algorithm to search each area. For example, the quadrotor first searches the kitchen, then moves to the office, where it successfully locates the coffee cup, ending the task.

Following the outlined experimental flight procedure, this study conducted 25 target search trials for each of the two layout configurations, using common indoor objects including coffee cups, oranges, and computers as search targets. As depicted in Figure 8, the schematic diagrams present the coffee cup search experiments for both layouts. Each schematic provides comprehensive flight path visualization, explicitly indicating the actual flight trajectory, reference path planning, and target object location. Additionally, the diagrams incorporate critical decision-making points throughout the search process, categorized into three critical phases: (1) Global Search Sequence Planning, (2) Local Direction Decision-Making, and (3) Target Identification and Task Termination.

Based on the aforementioned experimental procedure, multiple flight experiments were conducted to search for different items within the scenarios. This study evaluates the flight experiments using three metrics: success rate, Success weighted by Path Length (SPL), and Distance to Goal (DTG). The experimental results are presented in

Table 4. Physical Flight Experiment Result.

Layout	Success Rate	SPL	DTG
Cross	56%	0.3069	1.7741
T-shaped	48%	0.3782	1.6327

. The success rate is determined by whether the target object is identified during the flight. SPL, which is strongly correlated with the success rate, is also influenced by the flight distance. DTG, independent of success, measures the distance between the quadrotor and the target object when the task execution stops. In our flight experiments, the success rate is affected by the accuracy of object recognition and the ability to avoid obstacles. Instances of collisions with walls or unstable quadrotor posture during recognition, which degrade image quality, can lead to task interruptions or failure to identify the target. SPL, on the other hand, is more influenced by the search order. For example, as shown in Figure 7, certain search orders can result in longer flight paths. Both layouts achieve a certain level of success rate, but compared to the cross layout, the T-shaped layout shows a noticeable advantage in SPL, primarily due to its shorter flight paths in most cases.

The success rate in our flight experiments was constrained by the following factors:

(1)

Decision Program Anomalies: The large language model occasionally generated abnormal navigation decisions beyond predefined area labels.

(2)

Small-Target Detection Limits: When distant and small targets (e.g., oranges) were undetected by the perception module, premature region-completion decisions misled the quadrotor to the next area.

(3)

Localization System Failures: Near walls, motion capture system occlusion caused signal degradation or loss, risking collisions.

As shown in Table 4, an interesting divergence appears: the T-shaped layout yields a lower success rate (48%) but a higher SPL (0.3782) compared to the Cross-shaped layout (56%, 0.3069). This stems from topological differences. The high connectivity of the Cross layout allows the quadrotor to backtrack and recover from navigational errors, leading to more successes but longer trajectories. In contrast, the restrictive T-shaped layout limits backtracking; consequently, successful runs tend to be direct and highly efficient, while incorrect decisions often lead to immediate failure, resulting in a “high-efficiency or failure” pattern.

4.2.2. Indoor Structured Environment

This paper conducts additional physical flight experiments in an indoor structured environment, which excludes virtual backgrounds and consists entirely of real objects. The actual scene and its schematic diagram are shown in the Figure 9. The experimental environment comprises two distinct zones: a laboratory area and an office area. The laboratory area primarily contains various tools for assembling quadrotor hardware platforms, while the office area is furnished with typical work supplies such as computers and keyboards.

Since the indoor environment lacks a motion capture system for positioning, we integrated a MID-360 LiDAR (Livox Technology Company Limited, Shenzhen, China) as the primary sensor for localization data acquisition. Figure 10 shows the modified quadrotor configuration with the added LiDAR module. For localization, we employed the Fast-LIO algorithm [28], which was specifically configured for compatibility with this LiDAR system.

The flight procedure for this task is essentially consistent with that used in the indoor customized environment. Based on this experimental setup, flight tests were conducted to locate a keyboard in the office area and a drilling machine in the testing area. The figures respectively illustrate schematic diagrams of the search operations for the keyboard and drilling machine in the structured indoor environment. Elements in Figure 11 maintain consistency with those in the indoor customized scenario.

4.3. Failure Analysis

To further enhance the reproducibility of the proposed method, we conducted a qualitative analysis of the failed trials observed during physical experiments. Since the global search order follows a predefined sequence, failures were not due to path looping but primarily stemmed from two distinct sources.

First, Perception Limitations accounted for a significant portion of incomplete tasks. In scenarios involving small-scale targets, such as the coffee cup, the vision-language model occasionally failed to detect the object from a long distance or under partial occlusion. This led the agent to erroneously conclude that the target was not present in the current view and move to the next search configuration.

Second, Semantic Reasoning Inconsistency in the local decision-making process contributed to failures. In specific instances, the LLM failed to establish the correct semantic correlation between the observed scene and the target object. For example, when searching for a “keyboard,” the LLM occasionally assigned a low relevance score despite the agent facing a “desk” area. This underestimation caused the agent to classify the direction as irrelevant and prematurely abandon the search in the correct region, resulting in a missed target. Additionally, there were rare cases of LLM Hallucination, where the model generated output labels outside the predefined region list, triggering exception handling routines that interrupted the autonomous flow.

5. Conclusions

This paper presents a novel framework that integrates large language models (LLMs) with visual target navigation tasks. Specifically, our research investigates how LLMs can serve as pre-planners in scenarios lacking explicit positional cues, demonstrating their capability to accurately interpret task contexts and generate predefined search sequences for exploration guidance. The proposed method dynamically generates sub-target waypoints by synergistically combining real-time visual observations with LLM-derived prior probabilities and relevance metrics, enabling autonomous decision-making for search direction selection in unknown environments. Extensive experimental validation confirms the superior performance of our approach in addressing visual target navigation challenges, with additional verification conducted on physical robotic platforms. For future work, we identify three key improvement directions: (1) optimizing semantic map construction, (2) enhancing spatial information representation to strengthen semantic relevance measurements, and (3) improving obstacle avoidance capabilities through advanced perception algorithms. Additionally, to address the limitation of utilizing printed scenery in current physical experiments, we plan to validate the system in fully realistic indoor environments. While the VLMs employed in this study demonstrate robust semantic recognition capabilities on 2D representations, which are sufficient for validating the high-level planning logic, future deployment with real-world objects will further test the perception module’s robustness against complex geometric occlusions and lighting variations.