Large Language Models for UAV Autonomy from a Perception–Cognition–Action Perspective

Highlights

What are the main findings?

• This survey organizes recent LLM-enabled UAV research using a P–C–A framework, clarifying the roles of LLMs and multimodal variants across the autonomy stack.
• The review finds that deployment readiness depends not only on model capabilities, but also on verifiable interfaces that translate semantic outputs into representations usable by downstream flight modules.

What are the implications of the main findings?

• LLM-enhanced UAV systems should combine semantic reasoning with structured output contracts, deterministic verification, safety filtering, and fallback mechanisms.
• Future research should prioritize closed loop validation and the integration of LLM components with established perception, planning, and flight control modules.

Abstract

Deployable autonomy remains a key challenge for unmanned aerial vehicles (UAVs) operating in open-ended missions. Large language models (LLMs) and their multimodal variants, which can process visual and other sensory inputs, have introduced new capabilities for semantic perception, task reasoning, and language-conditioned control. However, these capabilities do not by themselves produce flight-ready autonomy. We structure our analysis around a Perception–Cognition–Action (P–C–A) framework. At each layer, we identify the capabilities contributed by LLM-based components and examine how they connect to existing flight modules through input specifications, output representations, architectural coupling patterns, and safety mechanisms. Across the surveyed systems, LLMs extend UAV autonomy beyond fixed perception categories, scripted task plans, and pre-programmed controllers. However, field deployment depends on whether model outputs can be transformed into representations that downstream modules can parse, verify, and safely execute. Without adequate validation, captions, task plans, code, waypoints, and control commands may become failure points that propagate across the P–C–A loop. Our analysis highlights structured output contracts, independent safety barriers, and deterministic fallback mechanisms as key design elements for the reliable integration of LLM capabilities into UAV platforms.

1. Introduction

Unmanned aerial vehicles (UAVs) have become increasingly prevalent in logistics, precision agriculture, public safety, infrastructure inspection, and environmental monitoring [1,2,3]. Despite this rapid adoption, most operational UAV systems still require substantial human supervision. Human operators are commonly responsible for mission specification, route planning, progress monitoring, and intervention when environmental conditions or mission requirements change. Consequently, UAV autonomy remains largely limited to well-defined scenarios, while a substantial gap persists between high-level human instructions and executable machine-level commands [4].

Recent advances in large language models (LLMs), including multimodal large language models (MLLMs), offer new opportunities to narrow this gap. These models can interpret natural-language instructions, reason over textual and visual information, decompose complex tasks, and produce structured outputs. In UAV applications, such capabilities may enable operators to specify missions at a higher level of abstraction while allowing the system to adapt its interpretation, planning, and execution to changing tasks and environments. Rather than relying exclusively on pre-programmed behaviours and manually designed decision rules, LLM-enabled UAV systems can potentially support more flexible and context-aware interactions between human operators and autonomous flight platforms.

Several recent surveys have examined the emerging intersection of LLMs and UAVs [5,6,7,8,9]. Existing reviews have organized the literature according to model capabilities, application domains, task types, operational workflows, or agent architectures. These perspectives provide valuable overviews of the field; however, three important gaps remain. First, existing surveys seldom focus explicitly on how LLM-generated outputs are represented and how such representations are connected to conventional UAV modules. Second, although LLMs introduce language-conditioned flexibility into UAV autonomy, the interface contracts required to make their outputs reusable, interpretable, verifiable, and executable have not been systematically analyzed. Third, safety is often discussed as an independent issue rather than as a cross-layer property of the interfaces linking perception, cognition, and action. In particular, limited attention has been paid to how errors introduced in one layer propagate to downstream modules and where they can be detected, constrained, or corrected.

To address these gaps, this survey investigates two central questions: What capabilities do LLMs and MLLMs introduce to UAV perception, cognition, and action? And how are these capabilities transformed into outputs that can be parsed, verified, and executed by existing UAV flight stacks? To answer these questions, we organize the literature using a P–C–A framework. This framework captures the major functional stages through which LLM-based components influence UAV autonomy: perception interprets sensory observations and environmental context; cognition supports task understanding, reasoning, and planning; and action translates high-level intent into executable commands and flight-control behaviours. Importantly, the framework also makes it possible to examine the representations and interfaces that connect these layers to established UAV perception, planning, and control modules.

This survey makes three main contributions: (1) We organize recent research on LLM-enabled UAVs using the P–C–A framework, with emphasis on the capabilities introduced at each layer, the output representations they produce, and their connections to existing flight-system modules. (2) We catalogue representative coupling patterns between LLM-based components and conventional UAV modules, including free-form natural language, semi-structured annotations, and machine-readable formal representations. (3) We examine how safety risks can propagate across perception, cognition, and action interfaces, and discuss the roles of deterministic verification, safety filtering, recovery mechanisms, and multi-tier validation in evaluating deployment readiness.

Figure 1 maps the full structure of this survey onto the P–C–A framework. The rest of the paper is organised as follows. Section 2 introduces the relevant background and compares existing surveys. Section 3 reviews LLM-enhanced perception, from open-vocabulary recognition to structured output schemas. Section 4 examines LLM-based task planning and reasoning architectures. Section 5 reviews execution mechanisms that translate high-level intent into low-level flight-control actions. Section 6 discusses safety considerations across all three layers. Section 7 identifies broader themes and future directions, and Section 8 concludes. Table 1 lists the abbreviations used in this survey.

Figure 1. Structure of this survey. The P–C–A framework organises the analysis across three layers, with cross-layer safety discussed separately.

Table 1. Abbreviations used in this survey.

Abbreviation	Definition
API	Application Programming Interface
BLEU	Bilingual Evaluation Understudy
CBF	Control Barrier Function
CCSP	Continuous Constraint Satisfaction Problem
CIDEr	Consensus-based Image Description Evaluation
CLIP	Contrastive Language–Image Pre-training
CoT	Chain-of-Thought
DSL	Domain-Specific Language
HFACS	Human Factors Analysis and Classification System
HPDL	Hierarchical Planning Domain Definition Language
HTN	Hierarchical Task Network
IoDT	Internet of Drones Things
LLM	Large Language Model
MARL	Multi-Agent Reinforcement Learning
MLLM	Multimodal Large Language Model
mAP	mean Average Precision
MPC	Model Predictive Control
NMPC	Nonlinear Model Predictive Control
OWL	Web Ontology Language
P–C–A	Perception–Cognition–Action
PDDL	Planning Domain Definition Language
RAG	Retrieval-Augmented Generation
RGB	Red–Green–Blue
RL	Reinforcement Learning
ROS	Robot Operating System
S0–S3	Validation tiers: Simulation (S0), Software-In-The-Loop (S1), Real-world flight (S2), Operational deployment (S3)
SDF	Signed-Distance Function
UAV	Unmanned Aerial Vehicle
VLA	Vision–Language–Action Model
VLM	Vision–Language Model
VLN	Vision–Language Navigation
VQA	Visual Question Answering

2. Background

2.1. LLM Variants

LLMs have evolved beyond their original role as next-token predictors. In robotics and autonomous systems, language models are increasingly combined with visual encoders, multimodal fusion mechanisms, external tools, and action-generation modules. These extensions enable models to process not only textual instructions but also images, sensor observations, environmental states, and, in some cases, action representations. This survey considers four model categories that frequently appear in LLM-enabled UAV research: text-only LLMs, vision–language models (VLMs), MLLMs, and vision–language–action models (VLAs).

LLMs are typically Transformer-based models trained on large-scale text corpora to model token sequences. Following instruction tuning, alignment, or task-specific adaptation, they can interpret natural-language prompts, generate text, perform structured reasoning, and interact with external knowledge sources or software tools [10]. In UAV systems, text-only LLMs are commonly used for mission interpretation, task decomposition, high-level planning, code generation, and tool orchestration.

VLMs combine language models with visual encoders to jointly process images and textual inputs. They support tasks such as image captioning, visual question answering, visual grounding, referring-expression comprehension, and open-vocabulary recognition [11,12]. For UAV applications, VLMs can help transform aerial images or video frames into semantic descriptions that are relevant to mission execution.

MLLMs extend language–vision processing by incorporating additional modalities, such as depth images, thermal imagery, audio, inertial measurements, LiDAR data, or point clouds [13]. Depending on the architecture, these modalities may be encoded separately and aligned with a language model through projection layers, cross-modal attention, fusion modules, or unified token representations. In UAV autonomy, MLLMs are particularly relevant when mission understanding depends on heterogeneous sensor observations rather than RGB imagery alone.

VLAs connect visual observations and language instructions to action representations [14]. Depending on the system design, these outputs may correspond to discrete actions, action tokens, motion primitives, skill selections, waypoint commands, trajectory parameters, or low-level control signals. VLAs therefore provide a possible route from high-level language-conditioned perception to embodied execution. However, in safety-critical UAV applications, their outputs commonly require additional parsing, constraint checking, planning, or control-layer verification before execution.

2.2. LLMs Meet UAVs

The integration of artificial intelligence into UAV systems has progressed through two broad stages. Earlier UAV autonomy systems relied primarily on hand-crafted rules and task-specific machine-learning models. Typical examples include object detectors trained for a fixed target category, path planners designed for a particular operational environment, and controllers calibrated for a specific airframe or flight condition [4]. Such approaches can achieve strong performance within their intended operating conditions, but they often require retraining, redesign, or expert reconfiguration when mission objectives, target classes, platforms, or environmental conditions change. Human–UAV interaction has therefore often remained limited to predefined commands, graphical control interfaces, and telemetry-based supervision.

More recently, LLMs and their multimodal variants have been introduced into UAV autonomy pipelines [15,16,17]. These models can provide several capabilities that complement conventional task-specific components. First, natural-language understanding and generation allow operators to express mission objectives in more intuitive and flexible forms. Second, zero-shot and few-shot generalization can reduce the need to create a separately trained model for every new object category, task description, or operational scenario. Third, multimodal reasoning enables the joint interpretation of textual instructions, images, video streams, maps, and other sensor data. Fourth, tool use, program synthesis, and structured-output generation can enable LLMs to interact with existing planners, simulators, databases, APIs, and flight-software modules.

These capabilities shift the central research question from merely improving an individual perception, planning, or control component to determining how general-purpose language-based reasoning can be incorporated into a UAV autonomy stack. This shift also introduces important systems challenges: LLM outputs must be represented in forms that downstream modules can parse; these outputs must be validated against platform, mission, and safety constraints; and errors must be prevented from propagating from high-level reasoning to physical vehicle behaviour.

2.3. Related Surveys

Several recent surveys have examined the intersection of LLMs and UAVs from different perspectives. Javaid et al. [5] provide an early overview of the field and discuss potential future research directions. Cidjeu et al. [6] conduct a systematic mapping study of research published between 2021 and 2024 and propose a multi-criteria taxonomy. Chen et al. [7] adopt an empirical perspective by analysing academic publications, GitHub repositories, and practitioner perspectives, identifying nine common tasks across four UAV workflows. Koubaa and Gabr [8] focus on agent-based UAV systems, tool use, and cognitive reasoning. Tian et al. [9] provide a broad review covering UAV modules, foundation models, datasets, tasks, applications, and future directions for agent-based UAV systems. Table 2 compares the scope, organization, and primary focus of these surveys.

Table 2. Overview of recent surveys on LLM-enabled UAV systems.

Survey	Year	Main Focus	Organisation	Interfaces	Validation
Javaid et al. [5]	2024	LLM capabilities for UAVs	Capability oriented	No	Limited
Cidjeu et al. [6]	2025	Systematic mapping of LLM and UAV studies	Multi-criteria taxonomy	No	Limited
Chen et al. [7]	2025	Research and industrial practice	Workflows and tasks	No	Empirical analysis
Koubaa and Gabr [8]	2025	Agent-based UAV systems	Agent architecture	Partial	Not central
Tian et al. [9]	2025	Broad LLM-enabled UAV survey	Modules, tasks, and applications	Partial	Not central

Most existing surveys organize the literature according to model capabilities, task categories, application domains, operational workflows, or agent architectures. These perspectives are valuable for describing a rapidly evolving field. However, they do not always distinguish between systems that generate fundamentally different forms of outputs for different consumers in the autonomy stack. For example, two systems may both be categorized as UAV planning systems, while one produces a natural-language recommendation for a human operator and another generates machine-readable waypoints for a flight controller. Their differences concern not only the planning task itself, but also the representation of the generated output, the point at which it enters the autonomy loop, and the validation required before downstream execution.

This survey addresses this distinction by focusing on the interfaces between LLM-based components and conventional UAV modules. In particular, it examines the role of LLM outputs at the Perception, Cognition, and Action layers; the representations used to encode these outputs; and the mechanisms through which they are interpreted, verified, and executed. This perspective is informed by established work on UAV guidance, navigation, and control, including safety-aware planning and obstacle-avoidance frameworks developed under environmental and model uncertainty [4].

2.4. P–C–A Framework and Validation Tiers

Perception, cognition, and action are widely used concepts in intelligent and embodied systems. In cognitive robotics, they describe the processes through which an agent receives information from its environment, forms an interpretation of that information, and selects or generates behaviour. Haazebroek et al. [18] describe perception, action, and cognition as interacting elements of a closed loop rather than as fully independent or strictly sequential stages. More recently, PCA-Bench [19] adopted a related three-part perspective to evaluate MLLMs on embodied decision-making tasks.

This survey adopts the P–C–A framework as an organizing lens for the LLM-enabled UAV literature. A typical UAV autonomy stack may include sensing, state estimation, mapping, mission management, planning, guidance, flight control, and actuation. The P–C–A framework provides a higher-level abstraction for identifying where LLM-based components are introduced within this stack and for comparing systems that employ LLMs at different stages of the autonomy process.

In this review, Perception refers to the use of sensor observations to generate mission-relevant semantic information. Typical outputs include textual scene descriptions, captions, object labels, bounding boxes, segmentation masks, scene graphs, grounded references, and answers to visual questions. Cognition refers to the interpretation of mission intent and the generation of decisions, task specifications, or high-level plans. Its outputs may include PDDL descriptions, behaviour trees, DSLs, API call sequences, symbolic task plans, or program fragments. Action refers to the transformation of a decision or task specification into platform-level execution commands. Typical outputs include waypoint sequences, trajectories, motion primitives, controller parameters, control programs, and, in some cases, direct control commands.

These categories characterize the role of an LLM-generated output rather than the complete functionality of a UAV subsystem. A single system may span multiple layers. For example, a multimodal model may identify a target from aerial imagery, use this information to generate a search or inspection plan, and subsequently produce waypoints that are passed to an existing trajectory planner or flight controller. In many systems, the LLM does not directly control the UAV. Instead, it generates an intermediate representation that is parsed, interpreted, checked, refined, or executed by conventional components in the flight stack.

The relationships among the three layers are also central to safety. An incorrect perception output may lead to an erroneous task interpretation, while an invalid task specification may result in an unsafe trajectory or control command. Safety should therefore be considered not only within individual modules, but also across the interfaces that connect Perception, Cognition, and Action. This interface-oriented perspective supports the analysis of error propagation, validation mechanisms, fallback strategies, and the conditions required for safe deployment.

To characterize the experimental maturity of reviewed systems, this survey assigns each work to one of four validation tiers:

S0—Simulation-only evaluation: The LLM-based method is evaluated in a simulation environment without integration into a realistic flight-software stack.

S1—Software-in-the-loop evaluation: The LLM-based component is connected to a simulated UAV, autopilot, or flight-software stack, enabling closed-loop software-level assessment.

S2—Controlled real-flight evaluation: The system is tested using a physical UAV in a controlled indoor or outdoor environment.

S3—Operationally relevant deployment: The system is evaluated in a realistic application setting involving extended field operation, operational constraints, or deployment-level procedures.

These validation tiers do not by themselves establish system safety or operational readiness. Instead, they provide a consistent basis for comparing the maturity of LLM-enabled UAV systems and for identifying the gap between conceptual demonstrations, simulated evaluations, controlled flight tests, and practical deployment.

3. Perception

Perception converts sensor observations into the semantic and geometric information that downstream autonomy modules consume. Classical UAV perception relies on detectors and segmenters trained for fixed categories, which works well when targets are known in advance but cannot accommodate instructions such as “find the person carrying a red backpack.” LLMs and vision–language models loosen this boundary by enabling language-conditioned recognition and scene understanding.

The practical value of these capabilities depends on how the resulting semantic content is represented. Section 3.1 reviews the fixed-category assumptions of classical perception. Section 3.2 examines open-vocabulary recognition, scene description, and visual querying as language-conditioned extensions. Section 3.3 then compares the output representations these systems expose—unstructured natural language, semi-structured geometric labels, and fully structured schemas—and asks which forms can be trusted and consumed by the cognition and action layers.

3.1. Traditional Perception

Perception is the entry point of the P–C–A pipeline. Classical UAV visual perception has concentrated on tasks such as object detection and segmentation. Datasets such as DOTA [20], VisDrone [21], UAVid [22], and UAVDT [23] have made these tasks easier to compare and have pushed detectors and segmentation models toward practical use in inspection and agricultural observation [24].

For this survey, the important limitation is not simply whether these models are accurate, but how narrowly their tasks are defined. A detector trained only on classes such as person and vehicle has no direct way to search for a “damaged ambulance” or an “individual carrying a red backpack” unless the category set is expanded or extra rules are added. Boxes and masks also describe geometry more readily than intent: they can mark objects, but they do not by themselves say which object matters for the mission or how two objects are related. In aerial imagery, small objects, top-down viewpoints, and cluttered backgrounds make this fixed-task assumption harder to maintain.

LLMs and VLMs change this boundary mostly at the task-specification stage. Operators can describe targets in natural language instead of selecting category indices or drawing fixed search regions. The model can then extract attributes, constraints, and termination conditions and use them as a query or task condition for a perception module [8,25]. This should be understood as semantic augmentation rather than replacement: UAVs still need conventional perception and estimation modules to provide spatial evidence.

3.2. Semantic Perception

Recent work uses LLMs and vision–language models to loosen the fixed-task boundary of classical perception. The most visible change is that the perception task can be specified after deployment, often in language rather than as a predefined class index. Open-vocabulary recognition allows the system to search for text-specified targets outside a closed label set [11,26,27,28]. Scene description turns visual observations into human-readable summaries, mainly for operator awareness [29,30,31]. Visual querying changes the interaction pattern. Instead of passively receiving captions, the operator can ask a specific question about the scene [32,33,34,35]. These directions all add semantic meaning to UAV perception, but they differ in one important respect. Some outputs are meant to inform a human operator, while others must be grounded before another module can use them.

3.2.1. Open-Vocabulary Perception

Conventional object detectors are fixed to the categories they were trained on. In dynamic missions, operators often need to search for targets outside that list, which creates a bottleneck. What open-vocabulary perception adds is run-time target specification. The UAV can be asked to look for a phrase such as “damaged ambulance” without adding that phrase as a new detector class. Figure 2 contrasts the two paradigms. Closed-vocabulary detectors are restricted to the fixed categories in their training set, whereas open-vocabulary models accept free-text queries at run time and let operators specify targets without retraining.

Figure 2. Closed-vocabulary versus open-vocabulary perception for UAVs.

Recent systems address this limit along two main lines. One line designs UAV-specific open-vocabulary models. UAV-OVD [28], for example, uses a multi-level text-guided fusion decoder for aerial unknown-category detection. The other line applies general vision–language models to aerial imagery and studies how much domain adaptation is needed. Grounding DINO [26] leverages vision–language pre-training for spatial localisation. YOLO-World [27] emphasises real-time throughput, with an accuracy trade-off in aerial scenes. CAGE [36] makes this domain gap visible. After UAV-domain pre-training, YOLO-World-v2-L reaches only 13.9 mAP on VisDrone in zero-shot evaluation. mAP averages detection precision across recall levels and object categories; it is the standard metric for comparing object detectors. This improves on the 8.59 mAP of the OGC-pretrained baseline, but remains well below supervised closed-set performance. On the efficiency side, CAGE runs at 22.9 ms per frame on an NVIDIA Jetson Orin NX. This result suggests that lightweight open-vocabulary models can approach real-time edge deployment, even though billion-parameter VLMs remain impractical onboard.

The same shift appears in segmentation and tracking for aerial platforms. Text-conditioned segmentation models use a phrase to select pixel-level regions rather than a fixed semantic class [37,38]. AeroReformer [39], for example, segments objects in UAV imagery from natural-language descriptions through vision–language cross-attention. In tracking, language conditioning allows the target to be initialised or revised by description. CloudTrack [40] reports that semantic constraints can reduce drift under cluttered backgrounds and viewpoint changes. Li et al. [41] further show that target descriptions can be updated during flight.

3.2.2. Scene Description

Scene description generates natural-language summaries of image content, so operators can understand a situation without watching every frame. Unlike bounding boxes or masks, captions present scene content in terms humans can read directly. Early work such as CapERA [42] showed that descriptive sentences could be generated from UAV footage, but used conventional encoder decoder models. More recently, RS-LLaVA [30] used a large vision–language model for joint captioning and visual question answering on remote-sensing imagery, finding that unified architectures can outperform task-specific pipelines. de Curtò and de Zarzà [31] showed that LLM-based pipelines, which combine CLIP prefix captioning with GPT-3 refinement, can produce zero-shot descriptive text for aerial frames without domain-specific training.

Fluency alone is not enough for task-oriented systems. Metrics such as BLEU and CIDEr measure syntactic similarity to reference captions, but high scores do not mean that the caption is operationally useful. PCSFTr [43] reaches a CIDEr score of 398.7 on UCM-caption, a large improvement over earlier baselines. Even so, caption quality measured in this way may not reflect whether mission-critical information is present. A caption may describe background scenery while omitting a small damaged structure or stranded vehicle. In top-down views where targets are small and sparse, the gap between fluent language and operational utility is especially sharp. A caption may be fluent while giving a path planner no grounded target to act on.

3.2.3. Visual Querying

VQA allows operators to ask natural-language questions and receive semantic answers tailored to the task, turning passive viewing into active querying, as illustrated in Figure 3. RSGPT [33] fine-tunes on remote-sensing image–text pairs to support VQA and semantic access at multiple granularities for overhead imagery. GeoChat [34] adds grounding to the query process, enabling region-level referring expression reasoning. But aerial VQA is fragile. Small changes in query phrasing, scene distribution, or viewpoint can shift answers [35]. High-resolution overhead imagery, sparse small targets, and perspective ambiguity further reduce reliability. For operator support, an occasional wrong answer may be corrected by human judgement.

Figure 3. Visual question answering for UAV interaction.

For closed-loop use, the same instability is harder to tolerate. For UAV use, a VQA answer has to be more than plausible. It must arrive in time and remain stable across similar queries. This is difficult in aerial scenes because many VLMs are pretrained mainly on natural images, while UAV imagery contains overhead views and tiny objects. Small details may also be lost when a high-resolution frame is compressed into visual tokens. If the answer is generated autoregressively, latency can vary from one query to the next. Such delay may be acceptable for operator awareness, but it is harder to use in a planner that expects timely updates.

3.3. Perception Interfaces

A caption that helps an operator understand the scene may give a path planner nothing to act on. A text-specified target becomes actionable only when returned with spatial evidence—a grounded box or mask. Whether a perceptual output is useful depends on who consumes it. Table 3 catalogues representative systems by the capability they provide and by the output format exposed to downstream modules. The same semantic content can appear in different forms. A target mentioned in a caption may also be exposed as a grounded object record or as part of a scene graph, and this choice determines whether later stages can use the result directly.

Table 3. Perception layer: capabilities, output morphologies, and validation tiers. ^† Preprint; ^‡ Peer-reviewed.

System	Input Modality	LLM/VLM Enhancement	Output Representation	Downstream Module	Validation Tier
Unstructured natural language
De Curtó & de Zarzá 2023 ‡ [31]	Aerial imagery	CLIP Prefix + GPT-3	NL scene description	Human operator	S0
RS-LLaVA ‡ [30]	RS imagery	Fine-tuned VLM (LLaVA-1.5)	Caption/VQA answer	Human/planner	S0
RSGPT ‡ [33]	RS imagery	Fine-tuned VLM	VQA answer/caption	Human/planner	S0
GeoChat ‡ [34]	Overhead imagery	Grounded VLM	Grounded VQA answer	Human/planner	S0
Semi-structured
Li et al. 2024 ‡ [41]	Aerial video + NL	VLM-guided tracking	Tracked target boxes + NL labels	Tracking module	S0
Fully structured
UAV-OVD ‡ [28]	Aerial img + NL text	Multi-level text-guided fusion decoder	Bounding boxes + confidence	Detection pipeline	S0
Grounding DINO ‡ [26]	Aerial img + NL	Vision–language grounding	Grounded bounding boxes	Detection pipeline	S0
YOLO-World ‡ [27]	Aerial img + NL	Open-vocabulary alignment	Detection boxes (real-time)	Edge detector	S0
CAGE † [36]	Aerial img + NL	UAV-domain pre-training	Detection boxes (zero-shot)	Edge detector (Jetson)	S0
CloudTrack ‡ [40]	Aerial video + NL	VLM semantic grounding	Tracked boxes + semantic labels	Tracker	S0
AeroReformer ‡ [39]	Aerial img + NL expr.	MLLM-generated descriptions	Segmentation masks	Segmentation module	S0
SceneLLM ‡ [44]	Aerial img + NL	LLM scene-graph generation	Scene-graph triplets	Planner/reasoning	S0

3.3.1. Natural-Language Outputs

The simplest way to deliver semantic meaning is free-form natural language. Remote-sensing VLMs [33,34], scene-description frameworks [31], and VQA benchmarks [35] use this pattern because it is easy to present to humans and fits standard NLP metrics. For human operators, this is often the right format. It lets them understand a scene without specialised tools.

Natural language is a poor machine-to-machine interface. Captions often lack spatial anchors such as coordinates or distances. Paraphrases preserve information inconsistently [5]. Most downstream modules cannot use free text directly without an extraction layer. The same generative process that produces fluent captions can also produce hallucinations [25]. These problems are practical as well as semantic. If captions are produced at high rate, they are awkward for embedded systems because each sentence has to be parsed before it can affect planning. A caption that is wrong but grammatically complete may also pass silently because there is no schema to violate. When such an error reaches the cognitive layer, it propagates as semantic misinterpretation. In current UAV pipelines, there is no widely used standard filter that can reliably catch unstructured caption errors before they enter downstream reasoning.

3.3.2. Geometric Outputs with Labels

Semi-structured representations impose lightweight syntactic rules that later stages can parse without full NLP. In perception, this pattern appears when outputs combine geometric primitives with natural-language labels. The geometry is machine-readable, but the label space remains open and may still require interpretation. Li et al. [41] report tracked target boxes augmented with NL descriptions, so a tracker receives both spatial extents and task-relevant labels without a full captioning pipeline. These formats are easier to parse than free text, yet they still create integration risks. A missing coordinate, an ambiguous label, or an unexpected field name may break an adapter written for one system’s output style. Engineers therefore face a double burden. They must write adapters for each system-specific output format, and they must add validation layers to catch malformed outputs that lightweight parsing does not catch.

3.3.3. Machine-Readable Schemas

Fully structured outputs expose perception results through fixed, machine-readable schemas. In UAV perception, representative examples include bounding boxes with confidence scores (UAV-OVD [28], Grounding DINO [26], YOLO-World [27], and CAGE [36]), tracked boxes with class labels (CloudTrack [40]), segmentation masks (AeroReformer [39]), and scene-graph triplets (SceneLLM [44]). Unlike free-form captions or geometric outputs paired with open-ended labels, these formats define explicit fields that downstream software can parse without a separate language-understanding step.

Each schema makes a different trade-off between spatial precision and semantic richness. Bounding boxes provide compact object locations and confidence values that detection, tracking, and planning modules can readily consume. However, a high-confidence box may still be spatially misaligned, causing a downstream module to reason about an incorrect target location. Tracked boxes add temporal identity and are useful for pursuit or monitoring tasks, but identity switches and tracking drift can cause the wrong object to be passed to later stages. Segmentation masks preserve object shape and extent more accurately than boxes, which is useful for obstacle boundaries and landing-area assessment, but boundary errors can distort the estimated obstacle footprint. Scene graphs provide the richest semantic representation by encoding relations such as vehicle–next to–building; however, missing or incorrect relations may hide mission-relevant proximity or collision constraints.

These formats reduce syntactic ambiguity, but they do not eliminate semantic error. A planner may correctly parse a bounding box, mask, or scene graph while still receiving a false detection, an inaccurate location, or an incomplete relation. Their practical use also depends on update rate and inference latency. CAGE reports 22.9 ms inference on a Jetson Orin NX [36], indicating that low-latency structured detection can be feasible on edge hardware. More semantically expressive outputs, such as multi-frame tracking results or scene graphs, may be more difficult to update at the rate required for time-critical flight decisions. Thus, fully structured outputs are the most directly usable interface for downstream autonomy modules, but their reliability depends on both the correctness and timeliness of the underlying perception result.

4. Cognition

Cognition is the part of the autonomy stack where mission intent is interpreted and turned into a plan. Classical planners work well when the goal, state, and constraints are specified in advance. Their assumptions become less reliable when the environment or mission conditions change. LLMs can add a semantic layer to this process by interpreting vague mission language, inferring implicit goals and success criteria, and revising plans when new information becomes available.

A language level plan is not yet an executable UAV plan. Before it can be passed to the action layer, the plan must be expressed in a form that an executor can parse and verify. This section follows the main steps involved in this translation. Section 4.1 begins with intent formalisation and task decomposition, since an underspecified command must first be converted into concrete goals and subtasks. Section 4.2 then examines how physical and situational constraints are incorporated into the plan. Section 4.3 considers the role of commonsense reasoning and external knowledge. Commonsense reasoning provides general background knowledge, while retrieval and knowledge bases supply information that is current, domain specific, or platform specific. Section 4.4 addresses the interface between cognitive reasoning and execution by examining representations such as PDDL, behaviour trees, domain specific languages, and API calls. Finally, Section 4.5 extends the discussion from a single UAV to coordination among multiple vehicles.

New observations, execution feedback, and changes in vehicle availability may require the system to return to an earlier stage and revise the plan. Throughout the section, we therefore consider two related questions: what LLMs contribute to UAV cognition, and how their outputs are structured so that they can be passed to the action layer without losing semantic intent or verifiability.

4.1. Task Planning

4.1.1. Intent Formalisation

Task objectives are rarely complete enough for classical planners. They arrive as natural-language commands, structured mission specifications, or perceptual events, but they almost always leave gaps that must be resolved before planning can begin. These gaps concern mission meaning, not scene content. Some gaps are lexical: “north slope” and “as soon as possible” carry no physical anchor. Others are contextual, because operators rely on unstated background knowledge. For example, “search and rescue” conventionally implies open terrain and battery reserves. The most consequential gap, however, is the omission of success criteria. Telling a UAV to “find the survivors” does not say whether visual detection suffices or whether coordinates and imagery must be relayed. Classical systems force operators or engineers to resolve all of these by hand before takeoff, translating open-ended intent into fixed parameter sets that inevitably distort the original objective [5,8]. LLMs automate this formalisation step. UAV-VLA [45] and SPINE [46] infer termination conditions, success criteria, and priority constraints directly from high-level mission descriptions. CityNavAgent [47] attaches reachability and observability constraints to subgoals, anchoring abstract objectives to concrete environmental entities.

4.1.2. Task Decomposition

Once an objective has been interpreted, it must be broken into subtasks. Traditional hierarchical task network (HTN) and symbolic planning methods provide explicit structure, but their decomposition templates are usually hand-authored and mission-family-specific. They therefore struggle when free-form intent changes the ordering, granularity, or dependency structure of a mission.

LLM-based systems introduce a more task-conditioned decomposition pattern [48]. CityNavAgent [47] segments navigation tasks into landmark-level, object-level, and motion-level subgoals. Neusis [49] activates decomposition layers from perceptual states, reordering visits as its belief map updates. TPML [50] varies decomposition strategy with task semantics: monitoring favours synchronous decomposition for consistency, while search-and-rescue favours asynchronous decomposition for rapid response. Decomposition must also resolve the dependencies among these subtasks. SPINE [46] infers implicit ordering from task semantics when instructions omit intermediate steps, validating the result through execution feedback.

The granularity of decomposition must match the execution format. If a subtask is already concrete enough for the controller, further decomposition adds overhead; if it is too abstract, it must be translated into code or trajectories. The granularity of these subtasks also determines which formalisation interface is appropriate later. TypeFly’s MiniSpec [51] compresses subtask representation to reduce latency, while CityNavAgent distinguishes high-level landmark planning from low-level waypoint generation.

4.1.3. Replanning and Adaptation

A decomposition is a hypothesis that assumes the world will stay as planned. Aerial missions consistently prove that hypothesis wrong. Perceptual updates, vehicle failures and changing weather all require the plan to be revised online. Replanning and adaptation are therefore part of task planning, not a separate phase.

Neusis [49] treats replanning as a belief-update problem. Instead of restarting the mission from scratch, it re-evaluates subgoal ordering as its belief map over target locations shifts, redirecting search effort to the most likely areas. SPINE [46] validates inferred task ordering against execution outcomes. The LLM derives ordering from task semantics and then checks it against actual outcomes; mismatches trigger re-decomposition. TACOS [52] handles swarm-level replanning triggered by changes in force structure. A Coordinator-Supervisor hierarchy distributes planning: the Coordinator produces high-level task plans, while the Supervisor LLM issues temporally ordered API calls from real-time telemetry. When an agent fails or joins, the Coordinator automatically updates the plan so the swarm remains consistent with available assets.

4.2. Constraint Handling

Classical planners encode constraints as numerical boundaries. That works when constraints are static and precisely modelled, but in flight they are dynamic and situational, and unforeseen combinations can outrun classical response. LLMs complement this by interpreting soft and situational constraints through world knowledge, but hard physical limits still require symbolic or optimisation-based verification.

The reviewed systems treat this as interpretation of meaning rather than numerical checking. Sezgin et al. use retrieval-augmented generation (RAG) to produce semantic decisions such as “return to base” under battery or communication failure [53]. Shibli et al. unify weather and battery failure as semantic factors influencing task priority in an Internet of Drones Things (IoDT) scheduling framework, reporting a 95% completion rate and a 42% reduction in completion time under coupled scenarios [54]. The gain is cross-domain reasoning through a unified model; the cost is that hard-limit certification must come from a separate verifier.

4.3. Knowledge-Enhanced Reasoning

4.3.1. Commonsense Reasoning

Traditional UAV decision-making relies on sensor measurements and fixed rules; it cannot reason about situations it has not been explicitly programmed for. LLMs introduce commonsense reasoning by drawing on pretrained world knowledge rather than deriving every inference from current sensor data. This knowledge covers things that are never written into the mission spec: thermal imaging at night, wildfire risk from smoke plumes, or headwind adjustments before landing. These inferences require no task-specific training, and Section 4.4 examines how cognitive interfaces can recover verifiability without sacrificing this flexibility.

Aerial vision-and-language navigation (VLN) serves as a canonical testbed for this capability. Liu et al. first defined the city-scale AerialVLN task, revealing the limitations of traditional methods in handling long-horizon trajectories and complex 3D spatial relations [55]. Commonsense reasoning further extends to multi-UAV collaboration and active search. AeroDuo [56] employs a high-altitude Pilot-LLM for global reasoning and target-probability prediction, while a low-altitude UAV executes fine-grained navigation.

4.3.2. Knowledge Enhancement

LLM knowledge acquired during pretraining is static and generic. Real-time weather, airspace notices and platform-specific performance data did not exist at training time, leaving systematic gaps in temporal validity and domain precision. RAG fills this gap by retrieving relevant external documents during inference and reasoning over them.

In safety-critical UAV missions, RAG has improved decision reliability. Sezgin et al. associate real-time telemetry with historical mission logs to produce semantic-level emergency responses under battery-failure and communication-loss scenarios [53]. Hao et al. integrate domain knowledge bases through hybrid retrieval and reranking for bio-inspired UAV swarm control [57]. Rather than simply providing more information, RAG converts fragmented operational data into structured reasoning context. When knowledge requires cross-fact association, knowledge graphs offer deeper structure than text retrieval. Duan et al. encode scenario, resource, task, and performance knowledge as entity-relation triples for mission planning [58]. Yan et al. map UAV accident narratives onto a four-tier causal taxonomy using the Human Factors Analysis and Classification System (HFACS), achieving a macro-F1 of 0.76 on 200 incident reports [59].

4.4. Cognition Interfaces

Cognitive outputs must eventually cross the boundary from language-level reasoning to executable commands. The systems in this subsection make that crossing explicit by generating formal action specifications, combining LLM reasoning with classical verification, and exposing the trade-offs different output forms impose.

4.4.1. Formalisation Routes

Current LLM-based cognitive planning produces outputs in three broad forms—PDDL, behaviour trees, and DSLs. Each route imposes a different contract between the cognitive layer and downstream executors.

(i) PDDL/hierarchical PDDL (HPDL) generation. The LLM translates natural-language mission descriptions into algorithmically solvable planning domains, including hierarchical task-network variants. Liu et al. [60] and Xie et al. [61] demonstrate iterative generation-and-verification loops in which the LLM emits PDDL specifications, a symbolic solver attempts a solution, and errors trigger domain refinement. Moon et al. [62] established an early pre-LLM pipeline from Web Ontology Language (OWL)/XML scene graphs to PDDL planning that remains the architectural reference for structured visual-to-planning pipelines. This route offers strong formal guarantees because the plan is verifiably correct with respect to the domain axioms. But the PDDL route is brittle. Minor syntactic errors in PDDL output render the file unparseable, and state-space explosion limits scalability to complex aerial missions with continuous variables.

(ii) Behaviour-tree generation. Rather than full symbolic planning, the LLM assembles hierarchical control nodes (sequence, selector, action) that support reactive interrupts. This interface is more robust to syntactic variation than PDDL because tree nodes are coarse-grained and self-contained, yet it sacrifices optimality: behaviour trees encode “good enough” reactive policies rather than globally optimal plans, and their coupling with low-level controllers depends on hand-crafted action-node implementations. In the surveyed UAV literature, behaviour-tree generation is less prevalent than PDDL or DSL routes, reflecting the field’s stronger emphasis on trajectory optimality over reactive interrupts.

(iii) DSL generation. Rather than PDDL or behaviour trees, a DSL constrains the LLM to a fast-to-parse, verifiable command set. TypeFly’s MiniSpec [51] trades expressiveness for certifiability and sub-500 ms latency. The route occupies a middle ground: more structured than free-form code, more flexible than full symbolic planning.

4.4.2. Hybrid Architectures

LLMs excel at discrete contextual reasoning and high-level strategy, but they are weak at continuous optimisation, hard physical-constraint satisfaction, and millisecond-level real-time control. Traditional optimisers and model-predictive controllers offer rigorous mathematical guarantees, but lack contextual understanding. The prevailing trend is to combine them in architectures that keep each component in its strength domain.

In one pattern the LLM generates candidate action hypotheses and a symbolic verifier filters physically feasible solutions—the LLM-Modulo framework. Say’n’Fly [63] applies this to search-and-rescue, using the LLM to exclude low-feasibility actions and a heuristic search to select the next action. A second pattern inserts a safety filter between the LLM and the physical plant. SwarmGPT [64] uses an optimisation-based filter to minimally correct trajectories, while Sanyal and Roy [65] use scene-aware control barrier functions (CBFs) for formal guarantees. A third pattern casts the LLM as a meta-decision maker above reinforcement learning (RL) or model predictive control (MPC). Wu et al. [66] use an LLM to select among specialised policies on the Pareto frontier of speed, safety and energy; RALLY [67] fuses LLM offline priors with multi-agent reinforcement learning (MARL) online policies, and SkyVLN [68] combines LLM semantic navigation with nonlinear model predictive control (NMPC) tracking.

LLMs are therefore kept at discrete high-level decisions, while classical methods remain responsible for continuous-space computation and rigid constraints.

4.4.3. Verification Gaps

These specification routes share a common vulnerability: the LLM generates the specification, but the translation’s correctness depends on the LLM’s ability to faithfully map operator intent into logical axioms. AutoTAMP [69] addresses this by positioning the LLM as a translator and checker: the model generates candidate PDDL domains, then a symbolic planner validates them before execution. But even this generation–verification separation cannot eliminate the risk that input errors propagate to output plans when the LLM misunderstands the mission’s constraint structure or omits implicit preconditions.

The probabilistic nature of LLM generation precludes formal guarantees for rigid constraints such as collision avoidance and dynamical feasibility. SwarmGPT addresses this via an optimisation-based safety filter that minimally corrects LLM-generated trajectories, intervening only when rigid constraints are violated [64]. This “semantic generation + symbolic verification” architecture defines a cognitive boundary. The LLM handles situational reasoning about why and how to adapt; the symbolic layer checks whether physical limits are breached. But the boundary between these two layers remains under-specified in most systems. No standard protocol exists for communicating which constraint was violated, why the correction was applied, or how the planner should revise its intent.

4.4.4. Output Taxonomy and Operational Trade-Offs

Table 4 synthesises the representation-design patterns observed across the cognitive layer. It catalogues representative systems along two orthogonal dimensions: (1) The Formal column records the formalisation strictness of the cognitive output, whether it admits algorithmic verification (Hard: PDDL, certified trajectories) or imposes syntactic discipline without logical proof (Soft: subgoals, directed acyclic graphs (DAGs), MiniSpec) and (2) the Val. column records the physical validation tier (S0–S2) achieved by the system; these two dimensions are independent—a soft-structured output such as TypeFly’s MiniSpec can reach S2 through extensive onboard flight testing, while a hard-formalised PDDL domain may remain at S0 if it has only been evaluated in simulation. The choice of formalisation depth governs whether downstream action modules can execute without further semantic interpretation, whereas validation tier governs whether that execution has been demonstrated in physical flight.

Table 4. Cognition layer: formalisation routes, output forms, and validation tiers. ^‡ Peer-reviewed.

System	Intent Input	Formalisation Route	Output Form	Formal.	Downstream Module	Validation Tier
Soft: structured but not formally verifiable
AerialVLN ‡  [55]	Aerial img + NL nav.	VLN alignment	8-class discrete actions (4-DoF)	Soft	Discrete action executor	S0
CityNavAgent ‡  [47]	Long-horizon NL nav.	Hierarchical decomposition	Landmark subgoals + waypoints	Soft	Motion planner/controller	S0
Neusis ‡  [49]	Belief-map updates + NL	Belief-driven factor graph	Dynamic subgoal sequences	Soft	Coverage planner	S0
TPML ‡  [50]	NL task + swarm context	Task-agent matching	Synchronous/asynchronous dispatch plan	Soft	Multi-UAV scheduler	S0
SPINE ‡  [46]	Under-specified NL mission	LLM implicit inference + validation	Behaviour sequences + NL justifications	Soft	Validator → executor	S0
TypeFly ‡  [51]	NL mission description	DSL generation (MiniSpec)	MiniSpec program tokens	Soft	MiniSpec interpreter	S2
UAV-VLA ‡  [45]	Satellite img + NL text	VLM + GPT fusion	Flight path-and-action plans	Soft	ArduPilot Mission Planner	S0
Hard: formally verifiable
Say’n’Fly ‡ [63]	NL SAR mission	LLM-Modulo + heuristic search	Verified action sequence	Hard	Symbolic executor	S0
AutoTAMP ‡  [69]	NL task + motion spec.	PDDL generation + symbolic check	Validated PDDL domain + plan	Hard	PDDL solver/motion planner	S0
SwarmGPT ‡  [64]	NL choreography instr.	LLM + optimisation certification	Certified safe trajectories	Hard	Swarm executor	S2
Moon 2019 ‡  [62]	Visual scene graph	OWL/XML → PDDL instantiation	PDDL problem + POPF plan	Hard	POPF planner	S0

Operational implications. Planning latency ranges from hundreds of milliseconds for MiniSpec generation to several seconds for iterative PDDL verification; the output contract must specify whether the downstream executor waits or proceeds with a stale plan. Communication overhead scales with output verbosity—a PDDL domain file may contain thousands of tokens, whereas a behaviour-tree skeleton compresses the same intent into tens of nodes, a critical distinction for bandwidth-constrained multi-UAV links. Error recovery depends on formality: soft-structured outputs permit graceful degradation through replanning, while a single syntax error in a hard-formalised PDDL file halts execution entirely. These trade-offs explain why S2-validated systems such as TypeFly (soft DSL) and SwarmGPT (hard certified trajectory) occupy opposite ends of the latency–certifiability trade-off.

The soft–hard distinction in Table 4 reflects a trade-off between flexibility and verifiability. Soft outputs, such as subgoals, behaviour sequences, and MiniSpec programs, are easier to generate and tolerate approximate intent. They fit missions where the environment changes quickly or where a human operator remains in the loop, because a loosely structured plan can be repaired or reinterpreted at runtime. The weaker cost guarantees that although a soft output is syntactically valid, it can still violate implicit physical constraints, and its meaning can drift between the LLM and the executor. Hard outputs, such as PDDL domains and certified trajectories, carry a proof of correctness with respect to stated axioms or constraints. They suit safety-critical steps where collision avoidance, dynamical feasibility, or geofence limits must be checked before actuation. The price is brittleness and scale: PDDL domains may explode in state space when continuous aerial dynamics are involved, and a single syntax error can halt execution. In practice, the most field-ready systems often mix the two, using soft generation for intent interpretation and hard verification for the final command envelope.

4.5. Multi-UAV Coordination

Multi-UAV coordination extends reasoning and task allocation from a single vehicle to a group of vehicles. Classical task allocation and cooperative scheduling therefore enter the review here, rather than under Task Planning. Each agent must know its own task, the tasks assigned to other agents, and how to respond when the swarm changes.

4.5.1. Language Based Task Allocation and Role Assignment

Language models contribute to coordination through role assignment and task-agent matching. RALLY [67] switches roles among Commander, Coordinator, and Executor through a two-stage LLM pipeline, allowing the swarm to reassign responsibilities as the mission evolves. TPML [50] maps natural language to multi-UAV task dispatch and supports both synchronous and asynchronous task-agent matching according to the mission requirements. These systems do not replace classical allocation optimisers. Instead, they add a semantic layer that represents assignments and adapts them when the situation changes.

4.5.2. LLM Decision Making and Swarm Collaboration

At the formation and distributed reasoning level, LLMs translate high-level spatial descriptions into geometric behaviours and distribute computation across agents. FlockGPT [70] compiles natural language shape commands into signed distance function formation code, allowing non-experts to specify shapes such as “form a sphere.” SwarmGPT [64] couples LLM reasoning with temporal coordination for large synchronised sequences and uses an optimisation-based safety filter to check physical feasibility. As the swarm or the model becomes larger, the computational load may exceed the resources available on a single UAV. SwarmChain [71] distributes LLM computation across agents through tensor parallelism, achieving a 1.9–2.3× speedup and up to 52% latency reduction.

The main difference between LLM-based coordination and conventional swarm methods lies in how mission intent is represented and distributed. Classical swarm methods rely on local interaction rules, consensus algorithms, and artificial potential fields to produce group behaviour without requiring a single agent to maintain a complete global plan. They are computationally efficient and suitable for formation keeping and collision avoidance, but they do not normally provide a general mechanism for converting an open-ended instruction such as “surround the building in a hexagon while keeping cameras pointed inward” into coordinated vehicle actions. LLM-based coordination adds a semantic layer for this translation. The advantage is greater flexibility and easier interaction with human operators; the costs include additional latency, communication requirements, and difficulties in verifying the safety of updated plans. Effective UAV swarms will likely combine both approaches, using LLMs for mission-level role and formation allocation and classical swarm controllers for low-level formation keeping and collision avoidance.

5. Action

The cognitive layer returns task sequences, waypoints, behaviour trees, or PDDL plans. These are still symbols: they do not move the aircraft until something converts them into thrust, roll rate and yaw torque. A classical UAV autonomy stack runs a rigid hierarchy of path planning, trajectory optimisation and attitude control, but that hierarchy has no semantic layer. A command like “fly around the red building while avoiding pedestrians” cannot become motion directly. The action layer has to close that gap by turning language-conditioned intent into control set-points while keeping every generated command inside the platform’s dynamic envelope, real-time deadlines and safety limits. The subsections that follow trace this translation from intent to motion. Section 5.1 examines how high-level decisions become trajectories, Section 5.2 reviews language-to-code compilation, Section 5.3 discusses execution feasibility and safety filtering, and Section 5.4 catalogues the output formats and coupling strategies summarised in Table 5. Across these topics, the central concern is how much authority the LLM is given over motion, and what representation guarantees are needed before its outputs can be executed safely.

5.1. Trajectory Generation

The safest and most common pattern keeps the generative model outside the real-time control loop. The LLM or VLM acts only as an interpretation layer. It does not send motor commands. Instead, it reads mission intent and returns structured numbers: target coordinates, semantic obstacle labels, or updated cost maps. These values feed into a conventional planner. The NMPC or geometric controller below it keeps full authority over feasibility, collision avoidance, and actuator limits. UAV-VLRR [72] shows this split in emergency search-and-rescue. Its VLM-LLM pipeline reads aerial imagery and operator instructions. It outputs target coordinates and semantic obstacle classes. Those outputs become time-varying references and hard constraints for a point-to-point NMPC. Indoor arena tests showed 33.75% faster mission completion than an off-the-shelf autopilot and 54.6% faster than human teleoperation. The gain comes from semantically informed setpoints, not from replacing the controller. Zhong et al. [73] distinguish between a real-world vision-based planning stack and an LLM-integrated simulation example. In the physical quadrotor system, NanoDet-based detection, Hungarian matching, and Kalman filtering are used to estimate the states and short-term trajectories of dynamic obstacles. These estimates are incorporated into a B-spline-based trajectory search and optimisation pipeline with collision, velocity, and acceleration constraints. The authors validate this non-LLM planning stack in both simulation and indoor real-world flight experiments. Separately, their LLM-integrated simulation uses ChatGPT to translate high-level user requests into task-specific code built from predefined UAV APIs; a GPT-enhanced EGO-Planner is demonstrated for executing a user-specified trajectory. Accordingly, the LLM is better characterised as a high-level language-to-code interface than as the source of dynamic-obstacle prediction.

LLM-aided execution outputs differ in granularity. They range from sparse waypoints to probability maps to dense continuous trajectories. Finer granularity gives more manoeuvre freedom, but it also pushes more responsibility for dynamic feasibility onto the tracking layer. LEVIOSA [74] turns natural-language instructions into deterministic 3D waypoint sequences. It uses hierarchical prompting and a multi-critic consensus. The sparse setpoints favour mission legibility and inter-vehicle separation over aggressive attitude changes. That makes the approach useful for multi-UAV coordination, where schedules and formation constraints matter more than tracking every attitude change exactly. Hierarchical vision–language–action models such as Hi Robot [75] push the same idea into open-ended instruction following. These models break high-level commands into reusable manipulation primitives. Aerial platforms have not explored this direction much. AeroDuo [56] uses a target-location probability map to seed A* sub-goals from its centroid. Sun et al. [76] treat belief-state uncertainty as a first-class planning input. That lets the cognitive layer produce contingency-aware subgoals when observations diverge from expectations.

At the finest granularity, OpenUAV [77] feeds continuous SE(3) poses (six-degree-of-freedom rigid-body configurations) into the PX4/ArduPilot autopilot stack as time-varying setpoints. Its contribution is to show that standard autopilot firmware can track continuous, language-conditioned trajectories in software-in-the-loop simulation. No hand-engineered intermediate discretisation is needed.

The controller must reject tracking errors, handle actuator saturation and recover from references that exceed transient limits.

5.2. Language-to-Code

An alternative approach treats the LLM as a compiler that outputs executable control programs rather than trajectory parameters. FlockGPT [70] takes a natural-language shape description such as “a spherical formation” and compiles it into Python and signed-distance-function (SDF) flocking code. The SDF encodes spatial constraints as distance fields from the formation boundary. A user can generate complex 3-D formation laws without learning control theory. LLVM-Drone [78] does something similar. It generates embedded Python function calls that map high-level visual tasks to UAV behaviours. Choutri et al. [79] translate bilingual speech—English and Arabic—directly into pyparrot API commands. They add a lightweight safety state-machine. Before deployment, it checks generated code against hard limits on altitude, velocity, and battery. If the code violates a limit, the system rewrites it into a permitted equivalent or falls back to hover. In all three cases, the LLM writes only predefined library calls. That restriction prevents the infinite loops, API misinvocations, and state-machine violations that open-ended code generation can produce.

The most immediate obstacle is inference latency. An LLM forward pass takes tens to hundreds of milliseconds, while aerial attitude control runs at hundreds of hertz. Action-chunking policies spread decoding cost across longer execution windows. That is one mitigation route [80]. In ground robotics, action-chunking transformers such as ACT show that compact policy networks can reproduce fine manipulation trajectories from few demonstrations [81]. The same compression should be possible for UAV motion primitives. TypeFly [51] attacks the problem with MiniSpec, a token-efficient DSL. It reportedly halves output tokens and planning latency compared with unconstrained Python generation. Its stream interpreter starts issuing flight-control instructions token-by-token before generation finishes. A query skill offloads dynamic high-level decisions to lightweight remote callbacks. Real-time LLM-aided execution needs both language-level compression and system-level pipeline engineering.

Once code is generated, the execution layer still needs closed-loop monitoring and replanning. TACOS [52] uses a Coordinator-Supervisor hierarchy. The Coordinator turns high-level commands into task plans. The Supervisor LLM issues temporally ordered API calls such as goto and start_tracking from real-time telemetry. The ATOMICA trajectory planner handles the low-level collision-free trajectories. When an agent fails or joins, the Coordinator replans automatically. Execution thus becomes online closed-loop control rather than a static script.

Because the LLM emits executable programs, the execution format needs nested guards. Syntax confinement keeps code valid within a predefined library, as MiniSpec does by design. Semantic validation enforces platform state-machine invariants, such as no take-off without pre-flight checks [79]. General semantic verification is still open. Aggregate dynamics monitoring certifies the dynamic feasibility of a code sequence before actuation.

5.3. Execution Feasibility

LLMs generate flight commands without knowing the UAV dynamics, airspace limits, or current environment. The execution layer needs mechanisms that stop invalid outputs before they reach the controller.

Wang et al. [82] propose GSCE, a prompt framework with enhanced reasoning for reliable LLM-driven drone control, at the soft end. It uses chain-of-thought reasoning and self-consistency checks to turn natural-language instructions into structured waypoints with confidence scores. Zhang et al. [83] use vision–language reasoning to recognise obstacle categories and adjust safe corridors during indoor navigation. These methods improve output reliability, but they do not prove physical feasibility.

Even syntactically valid action sequences can fail physically if preconditions are missing or state estimates drift. Joublin et al. [84] introduce CoPaL, a corrective-planning framework for robot actions with LLMs, which detects mismatches between planned actions and the current world state and prompts an LLM for a corrective replan. Raman et al. [85] propose CaPE, synthesising alternative actions when preconditions are violated. Both were demonstrated on ground robots; their direct transfer to UAVs is still limited. The principle still applies: when an LLM command conflicts with telemetry, trigger a local replan.

The strongest guarantee comes from formal certification before execution. Curtis et al. [86] frame LLM-generated parameterised skills as a Continuous Constraint Satisfaction Problem (CCSP) in their PRoC3S framework. Optimisation then certifies kinematic and geometric feasibility before execution. Qian et al. [87] close the perception-action loop for micro aerial vehicles by encoding dense occupancy maps and object-level semantics into CBFs inside an NMPC pipeline, yielding a formally certifiable safety bound at the control layer. These barriers link semantic outputs to constraint verification and classical safety filtering. Beyond the barrier itself, the coupling between representation and control loop determines what can go wrong. Control-loop frequency ranges from below 1 Hz for LLM-generated API calls in TACOS to 100–400 Hz for NMPC tracking. The output contract must state whether the LLM produces a one-shot reference or a continuously updated trajectory. Safety-filter latency adds delay between plan reception and actuation; SwarmGPT’s optimisation-based filter may need tens of milliseconds to certify a trajectory, and that time must fit inside the controller’s stability margin. Error propagation differs by format: a single wrong waypoint is spatially bounded, whereas a wrong API call sequence can trigger unbounded state-machine transitions. That is why two systems at the same validation tier—TypeFly and SwarmGPT, both S2—can have very different failure modes and containment properties.

5.4. Action Interfaces

The action layer’s output formats are catalogued in Table 5. The structured specifications from cognitive systems—subgoals, PDDL domains, or MiniSpec programs—become execution commands here. Table 5 compares representative systems by execution format, coupling strategy and validation tier. The table covers a spectrum from sparse waypoints to probability maps, continuous trajectories, and executable programs.

Table 5 arranges action-layer systems along a spectrum of coupling tightness. At the loose end, open-loop waypoints and trajectories keep the LLM outside the real-time control loop; the model produces a reference and a conventional planner or controller enforces feasibility. This is the most common pattern in S2-validated systems because the downstream module retains full authority over dynamics and safety. Moving inward, DSL compilation and hierarchical API calls give the LLM more control over the vocabulary of commands, but they confine that vocabulary to a restricted grammar or state machine so that every emitted token can be checked before execution. These approaches trade some flexibility for predictability and are the main route to keeping an LLM close to the control loop without giving it direct motor access. At the tight end, VLA end-to-end models map vision and language directly to velocity or primitive commands. They remove intermediate hand-engineered interfaces, but the coupling leaves little room for independent verification; safety relies on model consistency checks or pseudo-depth heuristics rather than on hard constraints. The validation tiers line up roughly with this spectrum: open-loop systems with explicit safety filters account for most S2 results, while the tightest coupling and the loosest code-generation pipelines remain at S0 or S1.

Table 5. Action layer: execution formats, coupling strategies, and validation tiers. ^† Preprint; ^‡ Peer-reviewed.

System	Semantic Input	Execution Format	Coupling	Platform	Safety Filter	Validation Tier
LEVIOSA ‡  [74]	NL mission description	3D waypoint sequences	Open-loop	Multi-UAV	None	S0
AeroDuo ‡  [56]	NL task + aerial img.	Prob. map + A* waypoints	Open-loop	Low-altitude UAV	None	S0
OpenUAV ‡  [77]	NL nav. instruction	Continuous 6-DoF poses	Open-loop	PX4/ArduPilot (SITL)	None	S1
UAV-VLRR †  [72]	Aerial img + NL instr.	Coords + sem. obstacles	Open-loop	NMPC controller	Soft (controller-native)	S2
TypeFly ‡  [51]	NL mission description	MiniSpec tokens	DSL compilation	Onboard interpreter	Soft (syntax)	S2
FlockGPT ‡  [70]	NL shape description	Python + SDF formation code	Open-loop	Multi-UAV swarm	Soft (library)	S2
TACOS ‡  [52]	High-level NL commands	API calls (goto, track)	Hierarchical	Swarm + ATOMICA	Soft (external)	S0
Choutri et al., 2025 ‡  [79]	Bilingual speech (EN/AR)	pyparrot API commands	State-machine validation	Consumer drone	Soft (state-machine)	S0
Zhong et al., 2024 ‡  [73]	NL task	GPT-generated code via UAV APIs	LLM-to-code + planner	Quadrotor UAV (simulation)	KF prediction + B-spline planning	S0
VLFly ‡  [83]	Aerial img + NL instr.	Continuous velocity commands	VLA end-to-end	Tello Edu (egocentric RGB)	Soft (VLM consistency)	S2
AutoFly ‡  [88]	Aerial img + NL instr.	High-level action primitives	VLA end-to-end	Multi-platform (lab + forest)	Soft (pseudo-depth)	S2
Neuro-LIFT ‡  [89]	Event stream + NL	Binary maneuver signal + PID	Event-based planner	Parrot Bebop 2 (event camera)	Hard (physics constraints)	S2
SwarmGPT ‡  [64]	NL choreography instr.	Certified safe trajectories	Open-loop	Swarm sim. + real	Hard (opt. filter)	S2

Table 5. Action layer: execution formats, coupling strategies, and validation tiers. ^† Preprint; ^‡ Peer-reviewed.

System	Semantic Input	Execution Format	Coupling	Platform	Safety Filter	Validation Tier
LEVIOSA ‡  [74]	NL mission description	3D waypoint sequences	Open-loop	Multi-UAV	None	S0
AeroDuo ‡  [56]	NL task + aerial img.	Prob. map + A* waypoints	Open-loop	Low-altitude UAV	None	S0
OpenUAV ‡  [77]	NL nav. instruction	Continuous 6-DoF poses	Open-loop	PX4/ArduPilot (SITL)	None	S1
UAV-VLRR †  [72]	Aerial img + NL instr.	Coords + sem. obstacles	Open-loop	NMPC controller	Soft (controller-native)	S2
TypeFly ‡  [51]	NL mission description	MiniSpec tokens	DSL compilation	Onboard interpreter	Soft (syntax)	S2
FlockGPT ‡  [70]	NL shape description	Python + SDF formation code	Open-loop	Multi-UAV swarm	Soft (library)	S2
TACOS ‡  [52]	High-level NL commands	API calls (goto, track)	Hierarchical	Swarm + ATOMICA	Soft (external)	S0
Choutri et al., 2025 ‡  [79]	Bilingual speech (EN/AR)	pyparrot API commands	State-machine validation	Consumer drone	Soft (state-machine)	S0
Zhong et al., 2024 ‡  [73]	NL task	GPT-generated code via UAV APIs	LLM-to-code + planner	Quadrotor UAV (simulation)	KF prediction + B-spline planning	S0
VLFly ‡  [83]	Aerial img + NL instr.	Continuous velocity commands	VLA end-to-end	Tello Edu (egocentric RGB)	Soft (VLM consistency)	S2
AutoFly ‡  [88]	Aerial img + NL instr.	High-level action primitives	VLA end-to-end	Multi-platform (lab + forest)	Soft (pseudo-depth)	S2
Neuro-LIFT ‡  [89]	Event stream + NL	Binary maneuver signal + PID	Event-based planner	Parrot Bebop 2 (event camera)	Hard (physics constraints)	S2
SwarmGPT ‡  [64]	NL choreography instr.	Certified safe trajectories	Open-loop	Swarm sim. + real	Hard (opt. filter)	S2

6. Safety and Risk Propagation

Risk grows at every boundary of the P–C–A pipeline. Traditional UAV stacks are deterministic and analysable module by module. LLM-enhanced systems add probabilistic generators whose failures are statistical and hard to bound. This section traces how risk propagates through the closed loop and where it can be intercepted. Figure 4 shows the chain: perceptual deception creates a wrong world state; cognitive hallucination and constraint neglect turn that into an infeasible plan; execution latency and dynamic violation turn the plan into physical harm. Each mitigation checkpoint in the figure is positioned at a layer boundary so that risk can be intercepted before it reaches the physical plant. We then examine governance—interpretability, human oversight, and regulatory compliance—across the whole stack. No single module guarantees safety. Safety must come from the architecture of boundaries that constrains, verifies, and overrides generative outputs before they reach the physical plant.

Figure 4. Closed-loop failure propagation chain along the P–C–A pipeline.

The mitigations in Figure 4 each have a representative implementation in the reviewed literature, although their maturity on real UAVs varies. VLM ensembles, together with input filtering and validation, are used to suppress perceptual false positives under domain shift. Cross-modal consistency checks compare language reports against visual evidence. Symbolic verification and recovery have been demonstrated in ground-robot systems such as CAPE/COPAL [84,85] and adapted to aerial planners through AutoTAMP [69]. CBF and NMPC hard constraints are the most mature in real flight, with physical demonstrations in UAV navigation [65,87]. Deterministic fallback modes are standard in certified autopilots, but their integration with LLM failure detection remains largely at the design stage. Most of the higher-level mitigations, such as ensembling, cross-modal checking, and symbolic recovery, have been evaluated only in simulation or on ground robots. Only the lowest-level hard constraints currently have extensive S2 flight evidence.

6.1. Perceptual Hallucination

The risk chain starts at the sensor-to-perception boundary. VLM-enhanced perception produces open-ended semantic errors that are harder to bound than classical detection failures. It may confidently report “rescue workers” where there is only vegetation, or miss mission-critical targets because domain shift hurts small-object reliability. The cognitive layer then plans against these hallucinations as if they were ground truth.

Attackers can exploit these weaknesses with physical-world attacks [90]. Defence at the P→C boundary mixes statistical robustness with semantic consistency. VLM ensembles and input filtering cover the statistical side; cross-modal alignment checks cover the semantic side [91]. But these defences are post hoc. A VLM output is a probability, not a measurement. The planner simply cannot tell a hallucination from a real target.

6.2. Cognitive Hallucination

When a wrong world state reaches the cognitive layer, LLM-based planners show several failure patterns [92,93]. Capability fabrication means claiming sensors that do not exist [94]. Constraint neglect means dropping airspace or energy limits [82]. State misjudgment means overconfident inference when observations are incomplete. Adversaries can make these failures worse by manipulating the language input. Prompt injection and jailbreaks hijack decision chains without changing model weights [95,96,97,98,99]. Coupled with UAV actuation, such attacks can move from information theft to physical harm.

Errors compound through the closed loop. Minor deviations can grow into catastrophic action sequences [100]. Current defences use hybrid generation–verification architectures, such as AutoTAMP [69] and CAPE/COPAL recovery [84,85], plus algorithmic Self-Refine [101]. But these are mostly post hoc. They do not remove hallucinations at the source, and they cannot fix physically infeasible execution commands.

6.3. Execution Infeasibility

The action layer is where high-level abstractions meet physical dynamics. Failures here fall into two categories: those caused by latency and those caused by dynamic infeasibility. Even a sound plan can fail if its trajectory violates the platform’s dynamical envelope or if inference latency destabilises the control loop. A generative-model forward pass takes tens to hundreds of milliseconds. Aerial attitude control runs at hundreds of hertz [5]. Algorithmic acceleration, such as speculative decoding and lookahead verification, can cut per-token latency [102,103,104]. But these methods are still hard to deploy on resource-constrained aerial platforms. VLA systems bridge the gap with action chunking and System 1/2 decoupling, pairing fast System 1 inference with slower System 2 deliberation. But chunk-boundary discontinuities and communication jitter can still push delays past stability margins [105,106,107]. Hard barriers must catch infeasible commands before they reach the motors. PRoC3S’s CCSP certification and CBF/NMPC safety bounds work independently of LLM inference.

6.4. Cross-Layer Governance

LLMs are black boxes, which makes their decisions unverifiable by current certification frameworks [108,109,110]. The EU AI Act and EASA concept paper classify AI-enabled aviation as high-risk, explicitly requiring explainable and predictable behaviour. Explainability must therefore be built into every layer of P–C–A. Perception layers must justify obstacle alerts [91]. Cognition layers need faithful chain-of-thought audit trails [111]. Action layers need semantic annotations that map raw telemetry back to high-level task descriptions. Current stacks largely lack this capability. Explainable AI for autonomous driving faces the same challenge: turning sensor-level decisions into human-understandable rationales [112]. The fluency of LLM outputs also makes human supervisors less effective.

Human–machine collaborative supervision is therefore a forward-looking requirement rather than an afterthought. A useful oversight mechanism must let a human operator monitor mission intent, inspect the LLM’s chain of reasoning, and override decisions at a semantic level. This goes beyond emergency stop buttons, because it requires interfaces that show why the system chose a particular plan, what alternatives were rejected, and what assumptions the plan rests on. Shared autonomy research in ground robotics has explored similar ideas, gradually transferring control authority between human and machine as confidence changes, but comparable frameworks for UAVs are still rare. For aviation, the EU AI Act and EASA concept paper explicitly demand meaningful human oversight with the ability to intervene when the operational-design domain is exceeded. Designing these interfaces is itself a research problem: too much information overwhelms the operator, while too little hides the failure modes that matter most.

Regulations impose architectural forcing functions. The EU AI Act [113] and the EASA concept paper [114] classify AI-enabled aviation as high-risk. They require documented operational-design-domain limits and deterministic recovery modes. The FAA roadmap [115] demands “explainable and predictable behaviour” before type certification. These requirements must be built into the P–C–A pipeline from the start, not added later. Table 6 summarises the closed-loop risk chain. The table shows that mitigation belongs at layer boundaries, not inside single modules.

Table 6. Closed-loop safety risk mapping along the P–C–A pipeline.

Risk Type	Source	Manifestation	Mitigation Strategy	Interface
Perceptual hallucination	Perception	VLM false detection/omission under domain shift	VLM ensemble + cross-modal consistency	P→C
Visual adversarial	Perception	Physical lighting/patch attacks on VLM	Ensemble defence + input filtering	Sensor→P
Capability fabrication	Cognition	LLM claims non-existent sensors	Symbolic verification (AutoTAMP)	C→A
Constraint neglect	Cognition	Hard constraints omitted in plan	Safety filter + Recovery (CAPE/COPAL) †	C→A
State misjudgment	Cognition	Overconfident inference under partial obs.	Self-Refine + Runtime monitoring	C→A
Prompt injection	Cognition	Malicious NL hijacks decision chain	Input filtering + instruction hierarchy	Human→C
Latency violation	Action	Chunk-boundary discontinuity, jerky motion	System 1/2 + speculative decoding	C→A
Comm. jitter	Action	Edge delay exceeds control stability margin	Delay-aware MPC + deterministic fallback	Network→A
Accountability gap	Cross-layer	Black-box decisions resist audit	CoT logging + semantic annotation	Full stack

^† CAPE/COPAL recovery principles demonstrated in ground robotics; direct UAV validation remains limited. Most symbolic-verification and cross-modal-consistency mitigations in the table have been evaluated in simulation or on ground robots, while CBF/MPC hard constraints and deterministic autopilot fallbacks have stronger real-flight evidence.

7. Discussion

The previous sections treated perception, cognition, and action separately. In practice, the boundaries blur: latency in action exposes weaknesses in perception, and an under-specified cognitive plan can break an otherwise sound controller. This section discusses the cross-layer themes that emerge, assesses deployment maturity, and lists open challenges.

7.1. Deployment Trade-Offs

Figure 5 catalogues the seven S2-validated systems identified in the reviewed corpus, organised by their principal layer contribution and annotated with platform and safety mechanism. Across these systems, conventional autopilot and control components retain low-level flight stabilisation.

Figure 5. S2-validated LLM-enabled UAV systems with flight-stack integration.

Richer language understanding usually needs larger models and longer inference. UAV platforms, however, need control decisions in milliseconds. The main mitigations are System 1/2 decoupling, algorithmic acceleration, and communication offloading. How well they work depends on the mission profile.

The reviewed literature has a marked validation-tier imbalance. Of the 42 primary systems, most stay at S0. Only seven reach S2: SwarmGPT [64], UAV-VLRR [72], TypeFly [51], FlockGPT [70], VLFly [83], AutoFly [88], and Neuro-LIFT [89]. No S3 system appeared in the corpus. The S2 systems do not always use larger models than S0 systems. Their advantage comes from rigorous output contracts: structured output formats, explicit safety filters, and deterministic recovery paths. This pattern suggests that explicit output contracts, safety filters, and deterministic fallback mechanisms are recurring features of field-tested systems. Representation maturity may therefore be an important contributor to deployment readiness, alongside model scale, task specificity, platform constraints, and the operational design domain. Table 7 summarises the S1–S2 systems and their deployment interfaces.

Table 7. Deployment-oriented summary of systems with S1–S2 validation or direct flight-stack integration.

System	Layer	Deployment Interface	Platform Coupling	Safety Mechanism	Validation Tier
OpenUAV [77]	A	Continuous 6-DoF references	PX4/ArduPilot SITL	Autopilot tracking constraints	S1
UAV-VLRR [72]	A	Coordinates and semantic obstacle labels	NMPC controller	Controller-native constraints	S2
VLFly [83]	A	Continuous velocity commands	Tello Edu (egocentric RGB)	VLM goal-retrieval consistency	S2
AutoFly [88]	A	High-level action primitives	Multi-platform (lab + forest)	Pseudo-depth obstacle avoidance	S2
Neuro-LIFT [89]	A	Binary go/no-go + PID trajectory	Parrot Bebop 2 (event camera)	EV-Planner physics constraints	S2
TypeFly [51]	C/A	MiniSpec command tokens	Onboard interpreter	DSL syntax confinement	S2
FlockGPT [70]	A	Python and SDF formation code	Multi-UAV swarm platform	Library-level command bounds	S2
SwarmGPT [64]	C/A	Certified safe trajectories	Swarm executor	Optimisation-based safety filter	S2

7.2. Simulation–Reality Gap

Bridging S0 and S2 requires output representations that survive real-world conditions, not just more training data. The usual sim-to-real view treats the gap as distribution shift in sensor data. For LLM-enhanced UAV stacks the deeper problem is representation abstraction mismatch: the same scene graph that works in simulation collapses in reality. Simulation assumes ideal sensors and deterministic physics. Reality brings motion blur, sensor degradation, and time-varying latency. These effects can degrade perceptual outputs from structured scene graphs to vague natural-language descriptions. The “scene graph in, PDDL out” contract then breaks.

The skew toward simulation is not accidental. Real UAV experiments are expensive, risky, and hard to reproduce across platforms and weather conditions. There are few public benchmarks that pair aerial imagery with language annotations and flight logs, so researchers often validate ideas in synthetic environments where the LLM’s fluent output looks correct because the simulation itself is forgiving. The consequence is an overestimation of capability: a system that works in a simulator may still fail when sensor noise, wind gusts, and communication jitter are present. It also hides risk-propagation paths. In simulation, a wrong caption may be harmless because the planner can still access ground-truth state; in flight, the same caption becomes the planner’s only evidence. Bridging this gap therefore requires more than better models; it requires validation protocols that deliberately stress the interfaces between perception, cognition, and action under realistic disturbances.

Future validation should include representation-stress tests: perceptual hallucination injection, cognitive constraint deletion, and action-layer latency spikes. These tests should target layer boundaries, both separately and together, rather than single modules. One useful idea is to keep the LLM output fixed as an invariant semantic contract and let a lower-level adaptor map it to platform-specific dynamics through online system identification. Air–ground collaborative systems already show how language-specified missions can be shared across heterogeneous platforms in unknown environments [116].

7.3. Regulatory Compliance

A central question is whether LLM semantic reasoning can internalise constraints or whether hard symbolic barriers are always required. The evidence is mixed. Soft constraints, such as weather priority and mission reordering, can be interpreted and weighted by the LLM, but their effect on the plan must remain auditable and subject to operator override or downstream verification. Hard constraints, such as collision avoidance and geofence boundaries, need symbolic verification [64,65].

Regulations force design-time architectural choices. The EU AI Act [113] and EASA [114] treat AI-enabled aviation as high-risk. They require conformity assessments and human oversight with meaningful semantic-level override. The FAA Roadmap [115] requires “explainable and predictable behaviour.” For LLM systems this means behavioural containment inside a verifiable envelope. The certification target shifts from the LLM itself to the safety filter and to the P–C–A interface where probabilistic reasoning hands off to deterministic execution. Compliance must be built in at design time. Post hoc safety filters break easily when prompting changes shift the LLM output distribution.

7.4. Open Challenges

Several challenges stand out for the next three to five years. We organise them below and link each one back to the risks discussed in Section 6.

7.4.1. Latency–Capability Trade-Off

The latency–capability trade-off remains unresolved. No current architecture combines billion-parameter semantic depth with aerial real-time determinism. Domain-specific distilled models shrink the model. FPGA or neuromorphic accelerators shrink inference time. Delay-aware MPC treats LLM inference as a time-varying parameter. The hard part is compressing semantic intent into temporally deterministic representations without losing task-relevant nuance. This challenge maps directly to the execution-infeasibility risk in Section 6.3: a planner that is too slow or too coarse can send stale or dynamically infeasible commands to the controller.

7.4.2. Onboard Deployability Under Extreme Constraints

Onboard deployability under extreme constraints is still open. Tactical operations in contested airspace cannot rely on the cloud or powerful edge GPUs. Sub-watt inference engines, deterministic middleware such as ROS 2 real-time extensions, and minimal semantic representations that still keep task safety within kilobyte-scale onboard memory are all needed. Lightweight platforms like the Crazyflie ecosystem have succeeded in multi-robot research for years [117], showing that aerial autonomy does not need heavyweight compute if layer contracts are lean enough. This challenge links to the latency-violation and communication-jitter risks in Section 6.3: tighter onboard resources make every millisecond of inference and every byte of inter-agent traffic safety-relevant.

7.4.3. From Prototypes to Operational Deployment

Moving from simulation demonstrations to operational deployment is still the largest open challenge. The reviewed literature has produced many prototypes, but they differ widely in how deeply the LLM is coupled to the physical platform. Open-loop waypoint and trajectory generation keeps the LLM outside the control loop and is the easiest to validate, which is why most S2 systems adopt this pattern. DSL compilation and hierarchical API calls trade some flexibility for predictability by restricting the LLM’s output vocabulary. VLA end-to-end approaches remove hand-engineered interfaces but are the hardest to certify because failures are buried inside the model rather than isolated at a verifiable boundary. The right coupling for a given mission therefore depends on the operational design domain, the acceptable risk, and the availability of human oversight. Turning these prototypes into fielded systems will require not only better models, but also clearer validation protocols, regulatory acceptance of split-responsibility architectures, and human–machine interfaces that let operators understand and override decisions when the situation exceeds the system’s certified envelope. This challenge connects to the cross-layer governance and accountability-gap risks in Section 6.4: without an auditable boundary between probabilistic reasoning and deterministic execution, no single module can be held responsible when a failure propagates through the pipeline.

This survey is limited by the current literature landscape, which is heavily skewed toward vision-and-language navigation and simulation-dominated benchmarks. Real-world flight validation (S2) remains sparse, and no large-scale operational deployment (S3) was identified in the reviewed literature. Future surveys should track progress toward sustained operational validation as the field matures.

8. Conclusions

In summary, LLMs add capability to UAVs at the perception, cognition, and action layers, but safety must be enforced at the boundaries between them. How well this works depends as much on representation structure and layer contracts as on model scale. The detailed layer-by-layer analysis and open challenges are presented in Section 3, Section 4, Section 5, Section 6 and Section 7. Across the reviewed systems, representation maturity, including structured output formats, explicit safety filters, and deterministic recovery paths, appears to be an emerging predictor of field readiness, although model size and task specificity also matter.

Moving from prototypes to operation means shifting from demonstrating capability to engineering trust. Future certifiable autonomy stacks should treat the LLM–symbolic boundary as a certification boundary. Components upstream of this boundary may remain probabilistic and data-driven, while components downstream should be formally analysable or statistically validated against operational design domain limits. Achieving this will require the interdisciplinary collaboration outlined in Section 7 and a shared commitment to the principle that, in LLM-enhanced aerial robotics, the boundary between semantic generation and physical execution is part of the safety-critical system.