Adaptive Self-Prompting in Agentic LLM Frameworks for Code Fault Detection

Abstract

Large language models (LLMs) have demonstrated strong capability for code understanding and vulnerability detection. However, most existing approaches rely on static prompting and treat the model as a passive predictor, limiting adaptability under uncertainty, particularly in embedded and cyber-physical systems (CPS). This paper introduces adaptive self-prompting as a core mechanism for agentic LLM-based fault detection in C-language embedded code. We propose two complementary frameworks: Agentic Retrieval-Augmented Generation (A-RAG), which performs confidence-triggered, reasoning-conditioned retrieval from CWE and SEI CERT knowledge bases at inference time, and Agentic Supervised Fine-Tuning (A-SFT), which internalizes improvements through a self-evaluation sweep that refines instructions and training exemplars during fine-tuning. Experiments are conducted on a unified dataset constructed from the Toyota ITC benchmark and a curated subset of Big-Vul aligned to embedded code-relevant CWE categories. Results show that adaptive self-prompting substantially improves predictive performance and error calibration compared to static Retrieval-Augmented Generation (RAG), conventional fine-tuning, and encoder-based baselines, achieving up to 86.3% F1 score while significantly reducing high-confidence misclassifications. These findings demonstrate that confidence-aware reflection and adaptive reasoning enhance both robustness and safety in LLM-based fault detection for embedded and CPS software.

1. Introduction

Modern software systems increasingly operate as embedded and cyber-physical systems (CPS), where software must interact with hardware, sensors, and real-time control loops under strict resource and safety constraints. In these settings, software faults are not merely reliability issues; they can propagate into unsafe physical behavior, violate timing constraints, or cause costly system downtime. Embedded and CPS codebases are frequently written in low-level languages such as C, rely on pointer-heavy memory manipulation, and integrate concurrency, interrupt-driven execution, and hardware-dependent behaviors. These characteristics make fault detection challenging and reduce the effectiveness of purely syntactic or rule-based analysis when faults depend on subtle control-flow interactions, data-flow context, or domain-specific usage patterns.

Traditional defect detection tools have achieved impressive scalability and adoption, but they often struggle with false positives, incomplete reasoning about complex program semantics, and limited interpretability in heterogeneous industrial code [1]. More recently, transformer-based program models and large language models (LLMs) have demonstrated strong capability for code understanding tasks, including vulnerability classification, localization, and repair. Unlike classical approaches, LLMs can synthesize long-range context, reason over program intent, and provide human-readable explanations. However, most LLM-based fault detection systems still treat the model as a largely passive predictor. Meaning, a static prompt is applied to each input, and improvements are sought through supervised fine-tuning, retrieval augmentation, or model scaling. As a result, these systems often underperform when confronted with uncertainty, ambiguous evidence, or domain shifts, and they provide limited mechanisms for self-correction during inference or training. Crucially, existing approaches do not allow the model itself to decide when its reasoning is insufficient, nor do they provide mechanisms for the model to autonomously revise its own prompts, retrieval strategy, or decision policy based on internally detected uncertainty. This leaves a gap between LLM-based fault detection and truly adaptive reasoning systems capable of regulating their own inference behavior.

Agentic LLM frameworks offer a promising alternative by enabling structured reasoning loops, tool use, and self-reflection. In principle, an agent can detect when it is uncertain, seek targeted external knowledge, revise its own instructions, or adapt its decision procedure over time. Despite this potential, most existing agentic pipelines for code analysis focus on orchestrating multiple static components (e.g., fixed roles, fixed prompts, predefined workflows), rather than enabling the agent to adapt its own internal reasoning process. In particular, prior work rarely uses intermediate signals such as model confidence or self-critique as first-class control variables that dynamically govern retrieval, prompt revision, or learning behavior.

Moreover, the embedded/CPS domain remains underrepresented in widely adopted LLM evaluation settings, where datasets and benchmarks skew toward general-purpose software and server-side vulnerability corpora (e.g., Big-Vul and related CWE/CVE-based datasets used in VulDeBERT- and VulDetect-style evaluations [2,3,4], see also Section 2). This gap motivates the need for methods and evaluations that explicitly account for embedded/CPS fault characteristics, as well as for agentic mechanisms that can dynamically refine their reasoning when the evidence is incomplete.

In this paper, we investigate adaptive self-prompting as a core capability for agentic LLM frameworks for code fault detection. Unlike prior prompt-engineering, RAG, or agent-based approaches, adaptive self-prompting in our formulation is not a fixed design choice but a learned and conditional control mechanism that determines when additional reasoning, retrieval, or instruction refinement is warranted. We focus on function-level C-language fault classification and introduce two distinct and complementary forms of adaptive self-prompting built on a unified agent architecture, corresponding to inference-time and training-time adaptation. Agentic Retrieval-Augmented Generation (A-RAG) invokes retrieval from a carefully constructed external knowledge base only when a confidence-based reflection step indicates uncertainty, and it generates retrieval queries that are conditioned on the model’s own reasoning trace. Agentic Supervised Fine-Tuning (A-SFT), in contrast, internalizes improvements through a self-evaluation sweep that converts low-confidence or incorrect cases into corrected training exemplars and evolves the agent’s system instruction over training epochs. To ensure embedded/CPS relevance while retaining real-world diversity, we ground experiments in a combined dataset curated from the Toyota ITC benchmark and a filtered subset of Big-Vul aligned to Toyota ITC defect categories. To our knowledge, this is the first study to systematically compare and unify inference-time adaptive self-prompting (via selective, reasoning-conditioned retrieval) with training-time adaptive self-prompting (via self-evaluation-driven instruction evolution) within the same fault-detection framework.

This work makes the following contributions that advance the state of the art in LLM-based fault detection:

1.

We introduce adaptive self-prompting as a unifying principle for agentic fault detection, demonstrating how confidence-based reflection can be used to dynamically regulate reasoning depth, retrieval behavior, and learning signals.

2.

We propose two novel agentic frameworks, A-RAG and A-SFT, that instantiate adaptive self-prompting at inference time and training time, respectively, demonstrating that self-regulated reasoning can be achieved both with and without deployment-time agentic overhead.

3.

We construct an embedded/CPS-oriented dataset by aligning the Toyota ITC benchmark with a curated subset of Big-Vul, enabling systematic analysis across controlled synthetic faults and noisy real-world vulnerabilities.

4.

Through ablations, confidence-aware failure analysis, and a CPS case study, we show that adaptive self-prompting improves not only predictive performance but also error calibration, substantially reducing high-confidence failures in safety-critical code.

5.

Unlike our prior study [5], which focused on empirically comparing static RAG, SFT, and dual-agent pipelines, this work introduces adaptive self-prompting as a new agentic mechanism and specifically targets embedded/CPS fault detection using a distinct dataset and evaluation framework.

The rest of the paper is organized as follows. Section 2 reviews prior work on LLM-based fault detection and positions our work within the current literature. Section 3 details the experiment setup and the architectures of A-RAG and A-SFT. Section 4 discusses the predictive and computational results. In addition to quantitative evaluation, we present an in-depth CPS case study based on an electronic throttle control routine to qualitatively illustrate how adaptive self-prompting manifests in safety-critical embedded software. Section 5 highlights the limitations of the experimental setup and threats to the validity of the study. Finally, we conclude this paper in Section 6 with a summary and directions for future work.

2. Related Work

This section places our research within the current literature. We first review the shift from traditional static analysis techniques to transformer- and LLM-based fault detection, and then compare the major LLM-based frameworks. We subsequently discuss evaluation metrics and datasets, focusing on how existing work underserves the embedded and CPS domains. Finally, we conclude with open challenges and research gaps that motivate our proposed approach.

2.1. From Traditional Techniques to LLM-Based Code Fault Detection

Early automated defect detection relied on rule-based static analysis, symbolic execution, and classical machine learning over handcrafted features [1]. While scalable, these methods struggle with context-dependent bugs, complex control/data-flow interactions, and noisy industrial code.

Neural models began to address these limitations by learning distributed program representations. VulDeBERT adapts BERT for vulnerability classification in C/C++ and outperforms traditional feature engineering [3]. Liu et al. improve vulnerability analysis through dependency-aware pre-training that captures long-range program structure [6]. Omar et al. similarly demonstrate that llm-derived feature representations can detect diverse CWE weaknesses in real-world code [4]. More complex architectures extend these, for example, Mahmud et al. use an ensemble transformer with cross-attention for joint vulnerability detection and documentation [7], while Ridoy et al. enhance robustness with an ensemble LLM stacking framework [8]. These efforts reflect a shift from traditional deterministic analysis toward data-driven models that learn semantics and context. LLMs further expand this progress by enabling zero-shot and few-shot reasoning during fault detection and repair. Fine-tuned LLMs can match or exceed specialized deep models on C defect benchmarks [9], and they support test-free fault localization by inferring buggy locations from static context alone [10]. Overall, LLMs increasingly function not only as classifiers but as general reasoning engines for code analysis.

However, most existing work focuses on general-purpose software or server-side applications rather than safety-critical embedded or CPS code. Analysis of OpenMP race conditions [11,12], cloud and infrastructure-as-code security [13,14], and mobile or operating-system applications [15] show that domain-specific constraints matter significantly. This motivates a focused investigation of LLMs for code fault detection in embedded and CPS settings, where real-time constraints, hardware interactions, and safety standards create additional requirements.

2.2. Different LLM-Based Approaches

Existing literature contains many LLM-based approaches for code fault detection and repair, which can be broadly grouped into (i) supervised or task-specific fine-tuning, (ii) retrieval-augmented generation (RAG) and hybrid architectures, and (iii) agentic or multi-agent frameworks that incorporate planning, tool use, and self-reflection.

2.2.1. Supervised Fine-Tuning and Prompt Tuning

Supervised Fine-Tuning (SFT) adapts general LLMs to specific detection or repair tasks using labelled code examples. Wang et al. show that fine-tuning LLMs on C defect datasets substantially improves detection accuracy over commercial models [9]. Lin et al. introduce HapRepair, which learns to repair OpenHarmony apps by fine-tuning in real-world bug-fix pairs, highlighting that domain-specific SFT can produce targeted repair behaviors in mobile/OS ecosystems [15]. Tian et al. propose an integrated approach that merges task-specific tuning with general prompting strategies for enhanced vulnerability localization, showing that carefully designed fine-tuning objectives can coexist with instruction-style prompts to balance precision and generality [16].

Beyond full SFT, parameter-efficient prompt tuning techniques have also been explored. Feng et al. present CGP-Tuning, a structure-aware soft prompt tuning method that incorporates code graph information into prompts for vulnerability detection [17]. By encoding syntactic and semantic structure into prompt parameters rather than full model weights, CGP-Tuning achieves competitive performance with reduced computational overhead, showcasing the potential of prompt-level adaptation for code analysis.

These works collectively show that fine-tuning and prompt tuning can effectively specialize LLMs, but they typically rely on static prompts and do not yet exploit adaptive self-prompting or autonomous reflection.

2.2.2. Retrieval-Augmented Generation and Hybrid Pipelines

RAG-based systems augment LLMs with external knowledge, such as CWE/CVE documentation, Q&A forums, and code repositories. Mansur et al. propose RAGFix, which enhances automated code repair by retrieving relevant Stack Overflow posts to guide patch generation [18]. By grounding the model in community-generated explanations and code snippets, RAGFix improves both correctness and plausibility of repairs. Yoon et al. explore a multi-level RAG model to support synergistic vulnerability analysis, where multiple retrieval and reasoning stages are combined to refine vulnerability assessments across different abstraction levels [19]. Du et al. leverage RAG to improve fault localization for novice programmers [20]. Their system retrieves similar code examples and explanatory materials to help identify likely fault locations, demonstrating RAG’s utility beyond pure security-focused tasks. Cao et al. extend RAG concepts to intelligent cloud environments with LLM-CloudSec, an LLM-powered framework for deep vulnerability analysis in cloud-native systems [13]. Here, retrieval from cloud configuration and runtime logs helps the LLM reason about misconfigurations and multi-layer vulnerabilities. Sheng et al. introduce LProtector, an LLM-driven vulnerability detection system that orchestrates retrieval and analysis modules to scan large codebases [21].

These RAG-based approaches demonstrate that external knowledge can significantly enhance LLM reasoning, but retrieval logic is often static or heuristic. The models rarely reason about their own prompts or adaptively refine retrieval queries in a fully agentic manner.

2.2.3. Agentic and Multi-Agent LLM Frameworks

A growing body of research examines agentic LLMs, which, following Li et al., are characterized by autonomous reasoning and action, perception of task-relevant information, interaction with other agents or external components, and the ability to evolve their behavior through reflection [22]. Ramanan et al. propose ASPIRE, a multi-agent framework for execution-free code analysis and repair, where specialized agents handle tasks such as static reasoning, patch generation, and validation [23]. Sharanarthi et al. introduce a self-learning multi-agent framework for real-time adaptive code analysis, combining retrieval augmentation with reinforcement learning to continuously refine agent behavior over time [24]. In follow-up work, the same authors present a multi-agent collaboration framework for adaptive code review, debugging, and security analysis, emphasizing coordination and role specialization among agents [25]. Toprani et al. designed an agentic workflow for automated vulnerability detection and remediation in infrastructure-as-code, where agents orchestrate scanning, diagnosis, and patching steps on cloud deployment artifacts [14]. Abtahi et al. augment LLMs with static code analysis tools, using agents to integrate static-analysis findings with LLM-generated suggestions for code quality improvements [26]. In addition, Qayyum et al. explore LLM-assisted bug identification and correction for Verilog HDL, where the LLM acts as a specialized assistant within a hardware design flow [27]. Curto et al. systematically evaluate Llama 3 and Code Llama as “watchdog” models for static application security testing, positioning general-purpose LLMs as pluggable analysis agents within a traditional SAST pipeline [28]. Nuteanu et al. study the use of LLMs to analyze compliance with safety standards in C code, revealing both opportunities and limitations in using LLM agents as safety auditors for MISRA-C and related guidelines [29].

Dolcetti et al. [30] propose a framework that combines prompt engineering with feedback from testing and static analysis to improve LLM-based code generation and repair. Their approach evaluates generated C code using unit tests and the Infer static analysis tool. The resulting correctness and vulnerability reports are then fed back to the model in an iterative pipeline consisting of generation, self-evaluation, and repair phases. Their results show that LLMs struggle to reliably detect their own errors. However, they are effective at repairing incorrect or unsafe code when provided with explicit external feedback. This work highlights the value of feedback-driven improvement in LLM-based code analysis. However, the feedback loop is externally orchestrated and depends on predefined pipelines and tool outputs. In contrast, our approach introduces adaptive self-prompting as an intrinsic mechanism. The model autonomously regulates its reasoning, retrieval, and instruction refinement based on internally derived signals such as confidence and self-evaluation.

Together, these works demonstrate that multi-agent and agentic frameworks can improve modularity, interpretability, and extensibility. However, agent behavior is typically driven by static system prompts and fixed workflows; explicit adaptive self-prompting, in which agents dynamically refine their own instructions and retrieval strategies based on intermediate reasoning signals, remains underexplored, especially for embedded and CPS fault detection.

2.3. Datasets and Benchmarks for LLM-Based Code Fault Detection

Existing datasets for code fault detection and vulnerability analysis are largely drawn from general-purpose software projects, with relatively limited coverage of embedded and CPS code.

Many classification and vulnerability-detection works rely on curated datasets of C/C++ or multi-language projects annotated with CWE/CVE identifiers, such as those used by VulDeBERT, VulDetect, and related models [3,4,6,7]. A widely used example is the Big-Vul corpus [2], which provides paired vulnerable and patched functions extracted from real-world open-source projects and annotated with CWE identifiers, providing realistic but noisy examples. Ensemble frameworks such as EnStack aggregate multiple benchmarks and code sources to improve coverage [8]. Wang et al. focus specifically on C code defect detection, drawing from open-source C projects and established defect datasets when fine-tuning LLMs [9].

RAG-based systems introduce additional knowledge sources. RAGFix retrieves from Stack Overflow, which supplies natural-language explanations and analogous code fragments [18]. Yoon et al. and Cao et al. draw on vulnerability databases, documentation, and cloud configuration artifacts to support multi-level and cloud-context analysis [13,19]. Du et al. evaluate on novice-programming datasets, where small student programs with labelled faults enable controlled studies of fault localization [20].

Domain-specific studies highlight datasets that are emerging but still narrow. Qayyum et al. examine Verilog HDL bug datasets containing hardware designs [27]. Alsofyani et al. use OpenMP race benchmarks and synthetic parallel programs for data race detection [11,12]. Lin et al. rely on OpenHarmony application repositories for mobile/embedded OS apps [15]. Nuteanu et al. analyze C code against safety standards (e.g., MISRA-C), focusing on industrial-style code snippets governed by strict guidelines [29]. Cloud and IaC security frameworks, including LLM-CloudSec and the agentic workflow for infrastructure-as-code, use configuration repositories and deployment scripts as their primary artifacts [13,14].

Agentic frameworks such as ASPIRE and the multi-agent code review systems typically experiment on mixed datasets consisting of general open-source projects and security benchmarks [23,24,25,26]. Curto et al. and Sheng et al. evaluate on standard SAST benchmarks and proprietary or open-source security test suites [21,28]. Saju et al. combine public vulnerability datasets with proprietary industrial repositories, indicating that realistic evaluations often require a mix of open and internal data [5].

Despite this diversity, there remains a notable gap in widely adopted, open-source benchmarks for general code fault detection in embedded or CPS code. Many datasets contain server-side or application-level code, with limited representation of low-level C for microcontrollers, safety-critical control logic, or real-time operating systems. In this context, the Toyota ITC benchmarks, which contain a collection of real-world embedded C programs and carefully designed fault-injection cases for static analysis evaluation, offer a compelling foundation for studying LLM-based fault detection in a CPS/embedded setting [31]. As such, the Toyota ITC suite is a natural candidate as a data source for the proposed research on adaptive agentic LLMs for CPS/embedded code fault detection.

2.4. Challenges, Limitations, and Open Research Gaps

While current LLM-based approaches have achieved impressive results, several challenges remain unresolved, particularly with respect to embedded and CPS domains and achieving self-prompting behavior.

First, most methods treat the LLM as a largely passive predictor conditioned on static prompts. Fine-tuned models such as HapRepair, CGP-Tuning, and task-specific defect detectors [9,15,17] rely on carefully engineered but fixed instructions. Even when prompts are manually refined, the models themselves do not autonomously adapt their instructions in response to uncertainty, conflicting evidence, or failure modes. RAG systems, while more dynamic in their use of external knowledge, typically implement retrieval strategies via static pipelines or heuristic ranking, with limited capacity for the model to reason about which new evidence to seek and how to adapt retrieval queries [13,18,19,20,21].

Second, existing agentic and multi-agent frameworks provide promising abstractions but fall short of full adaptive self-prompting. ASPIRE, multi-agent code-review systems, and infrastructure-as-code workflows orchestrate multiple agents with predefined roles and communication protocols [14,23,24,25,26]. However, their internal prompts are usually static, and adaptation is often confined to high-level reinforcement-learning signals or rule-based schedulers rather than fine-grained, self-generated prompt revisions.

Finally, domain coverage remains limited. While there is progress in specialized domains such as Verilog HDL [27], OpenMP race detection [11,12], safety-standard compliance checking [29], and mobile/OS apps [15], there is comparatively little work targeting general code fault detection in embedded and CPS software. Many benchmarks focus on security vulnerabilities rather than functional defects and concurrency- or resource-related issues that are common in embedded and CPS software. While some of these defects (e.g., deadlocks, livelocks, and long lock durations) can indirectly impact timing behavior, existing datasets rarely capture these aspects in a systematic way, and explicit modeling of timing constraints or system-level safety properties remains limited.

These gaps motivate the proposed research on adaptive self-prompting in agentic LLM frameworks for code fault detection in cyber-physical and embedded systems. Furthermore, the lack of embedded-oriented defect datasets motivated the creation of our domain-specific dataset, which emphasizes functional, concurrency, and numerical fault patterns representative of embedded software. The proposed work aims to advance beyond current static or weakly agentic LLM pipelines and seeks to provide more robust, interpretable, and domain-aware fault detection capabilities for embedded and CPS code.

3. Methodology

This section describes the experimental setup, architecture and evaluation plan for the two agentic LLM frameworks for code fault detection. Following the unified LLM-based agent architecture proposed by Li et al. [22], we designed A-RAG and A-SFT.

3.1. Dataset Curation

We constructed our experimental dataset by combining synthetic benchmark programs and real-world vulnerable code to capture different aspects of software faults and ensure embedded domain specificity. Synthetic benchmarks provide controlled, well-defined, and domain-grounded defect patterns, while real-world data reflects the complexity, noise, and variety encountered in practice. As such, we curated data from the Toyota ITC benchmark and the Big-Vul corpus, applying an appropriate preprocessing and labeling pipeline to obtain function-level samples suitable for learning-based fault detection and evaluation.

3.1.1. Toyota ITC Benchmark

We curated part of our dataset from the Toyota ITC benchmark, a well-established benchmark suite used to evaluate static analysis and program verification tools [31]. The benchmark is composed of self-contained C programs designed to represent realistic software defects commonly observed in embedded and system-level software. It covers 51 defect sub-types grouped into nine high-level categories. For each defect sub-type, the benchmark provides paired implementations with and without defects, enabling systematic evaluation of both true positives and false positives.

To make the benchmark suitable for learning-based analysis while minimizing context loss, we transform the original file-level programs into function-level samples that preserve the local semantic context of each target routine. Specifically, each extracted sample contains the target function together with any referenced helper functions, relevant type definitions, global variables, and macros required for semantic completeness. This design follows the function-level representation commonly used in Big-Vul and related vulnerability-detection datasets, enabling a consistent learning and evaluation unit across both data sources. Comments are also removed to prevent label leakage and clues. We emphasize, however, that this transformation preserves local procedural context, and the benchmark does not provide full project-level execution context.

The samples are labeled according to the benchmark’s original classification. Functions derived from defect-containing programs are labeled as faulty (1), while those derived from defect-free programs are labeled as non-faulty (0). The resulting dataset preserves the original defect taxonomy and balance of the Toyota ITC benchmark while enabling function-level analysis.

This curation process produces a structured dataset of labeled C functions that maintains the intent and diversity of the Toyota ITC benchmark while better aligning with modern vulnerability detection and code understanding tasks.

3.1.2. Big-Vul Corpus

In addition to the Toyota ITC benchmark, we curate data from the Big-Vul dataset. It is a large-scale, real-world vulnerability dataset constructed from open-source software repositories and vulnerability reports [2]. Big-Vul consists of vulnerable functions paired with their corresponding patched versions and is widely used for learning-based vulnerability detection and repair tasks. Each sample is annotated with a Common Weakness Enumeration (CWE) identifier, enabling categorization by vulnerability type. Due to its real-world origin, Big-Vul exhibits a strong class imbalance, with non-vulnerable code significantly outnumbering vulnerable instances in the original collection [2].

We use the cleaned function-level splits of Big-Vul, provided in the author’s repository (The official Big-Vul repository can be found at https://github.com/ZeoVan/MSR_20_Code_vulnerability_CSV_Dataset and was accessed on 10 December 2025), which removes duplicates, normalizes formatting, and isolates individual functions. From this representation, we retain only rows where both the vulnerable and patched versions of a function are available and syntactically distinct. To construct a balanced dataset, each vulnerable function (func_before) is labeled as faulty (1), while its corresponding patched version (func_after) is labeled as non-faulty (0). Extremely long functions, exceeding 4000 characters, were excluded to avoid exceeding typical model context limits.

To align the Big-Vul dataset with the Toyota ITC benchmark and ensure domain-specificity, we construct a mapping from the nine Toyota ITC defect types to 44 standardized vulnerability identifiers (see Table 1). This mapping is derived from the defect sub-type explanations provided in the Toyota ITC benchmark specification [31]. Based on these descriptions, each defect sub-type is associated with one or more corresponding CWE identifiers. The resulting mapping (see Table 2) is then used to extract only the Big-Vul rows with CWE IDs that overlap with the Toyota ITC defects. We further restrict the dataset to C-language samples to ensure consistency across data sources.

Table 1. CWE IDs found in the dataset.

No.	CWE ID	Title
1	CWE-1335	Incorrect Bitwise Shift Direction
2	CWE-122	Heap-based Buffer Overflow
3	CWE-786	Access of Memory Location Before Start of Buffer
4	CWE-480	Use of Incorrect Operator
5	CWE-570	Expression is Always False
6	CWE-571	Expression is Always True
7	CWE-681	Incorrect Conversion between Numeric Types
8	CWE-190	Integer Overflow or Wraparound
9	CWE-191	Integer Underflow (Wrap or Wraparound)
10	CWE-561	Dead Code
11	CWE-833	Deadlock
12	CWE-463	Deletion of Data Structure Sentinel
13	CWE-415	Double Free
14	CWE-667	Improper Locking
15	CWE-675	Duplicate Operations on Resource
16	CWE-835	Loop with Unreachable Exit Condition (Infinite Loop)
17	CWE-590	Free of Memory not on the Heap
18	CWE-476	NULL Pointer Dereference
19	CWE-843	Access of Resource Using Incompatible Type
20	CWE-252	Unchecked Return Value
21	CWE-703	Improper Check or Handling of Exceptional Conditions
22	CWE-484	Omitted Break Statement in Switch
23	CWE-563	Assignment to Variable without Use
24	CWE-416	Use After Free
25	CWE-131	Incorrect Calculation of Buffer Size
26	CWE-764	Multiple Locks of a Critical Resource
27	CWE-121	Stack-based Buffer Overflow
28	CWE-825	Expired Pointer Dereference
29	CWE-401	Missing Release of Memory after Effective Lifetime
30	CWE-393	Return of Wrong Status Code
31	CWE-824	Access of Uninitialized Pointer
32	CWE-1260	Improper Handling of Overlapping Memory
33	CWE-189	Numeric Errors
34	CWE-469	Use of Pointer Subtraction to Determine Size
35	CWE-465	Pointer Issues
36	CWE-362	Concurrent Execution using Shared Resource with Improper Synchronization
37	CWE-398	Indicator of Poor Code Quality
38	CWE-562	Return of Stack Variable Address
39	CWE-196	Unsigned to Signed Conversion Error
40	CWE-908	Use of Uninitialized Resource
41	CWE-457	Use of Uninitialized Variable
42	CWE-765	Multiple Unlocks of a Critical Resource
43	CWE-628	Function Call with Incorrectly Specified Arguments
44	CWE-369	Divide By Zero

Table 2. Mapping of Toyota ITC Defect Types to CWE IDs.

Toyota ITC Defect Type	CWE IDs
Concurrency defects	CWE-190, CWE-362, CWE-667, CWE-675, CWE-764, CWE-765, CWE-833, CWE-843
Dynamic memory defects	CWE-122, CWE-1260, CWE-131, CWE-463, CWE-786
Inappropriate code	CWE-252, CWE-398, CWE-484, CWE-561, CWE-563, CWE-570, CWE-571, CWE-703
Misc defects	CWE-393, CWE-457, CWE-563, CWE-835, CWE-843
Numerical defects	CWE-1335, CWE-189, CWE-190, CWE-191, CWE-196, CWE-369, CWE-476, CWE-681
Pointer related defects	CWE-465, CWE-469, CWE-476, CWE-480, CWE-628, CWE-824, CWE-843
Resource management defects	CWE-121, CWE-401, CWE-415, CWE-416, CWE-476, CWE-561, CWE-562, CWE-590, CWE-825, CWE-835, CWE-908
Stack related defects	CWE-121, CWE-416, CWE-786
Static memory defects	CWE-121, CWE-786

Unlike synthetic benchmarks, Big-Vul reflects naturally occurring vulnerabilities and therefore contains substantial noise, stylistic diversity, and variety across vulnerability types. As such, it forms a real-world counterpart to the synthetic Toyota ITC benchmark data, allowing our experiments to evaluate model performance across both controlled and naturally occurring vulnerability distributions.

3.1.3. Dataset Integration and Partitioning

After independently curating the Toyota ITC and Big-Vul subsets, we merge both sources into a unified dataset with a common schema. Each sample is annotated with its source (Toyota ITC or Big-Vul), a defect or vulnerability type identifier (benchmark fault type or CWE ID), the function-level code snippet, and a binary fault label.

The final dataset comprises 4034 code samples, integrating both benchmark-generated and real-world vulnerability data under a unified representation. It includes 1268 samples from the Toyota ITC benchmark and 2766 samples from the curated Big-Vul subset, ensuring a balance between controlled synthetic defects and naturally occurring vulnerabilities. The dataset is strictly balanced with respect to the fault label, containing an equal number of faulty and non-faulty instances (2017 each), which supports stable and unbiased model training. In addition, the dataset spans a diverse set of defects corresponding to 44 CWE-IDs, with notable representation across memory safety issues (e.g., use-after-free, buffer overflows), concurrency-related defects, and numerical errors, reflecting the characteristics of embedded and CPS software. Function lengths vary from minimal routines to moderately complex implementations (up to 161 lines), with an average of approximately 25 lines. This composition ensures both diversity and domain relevance, enabling robust evaluation across heterogeneous fault patterns. An example of one of the simpler faulty samples is provided in Algorithm 1.

The combined dataset is then partitioned into training and test sets using stratified sampling. Splitting is performed separately per data source and is stratified by both fault label and defect type to maintain representative distributions across splits. Although we employ a standard train-test split, the use of confidence-aware adaptive mechanisms in both A-RAG and A-SFT acts as an implicit regularization strategy that mitigates overfitting and promotes more robust generalization. We use a 70%/30% split for training and testing, respectively, with a fixed random seed to ensure reproducibility.

This final step results in a balanced and well-controlled experimental setup that supports fair evaluation while accounting for the differing distributions and characteristics of benchmark-generated and real-world vulnerability data.

3.2. Vector Knowledge Base for Agentic RAG

A key design constraint of the A-RAG system is the strict prevention of information leakage between the training/testing splits of the dataset and the external knowledge retrieval corpus. To address this, the retrieval Knowledge Base (KB) is constructed exclusively from external, publicly available sources that describe the semantics, mechanisms, and real-world occurrences of software defect and vulnerability categories represented in our dataset. This ensures that retrieved knowledge provides general explanatory context rather than dataset-specific signals, enabling fair evaluation of the agent’s reasoning and retrieval capabilities. Specifically, we draw information from two standardized sources: the official CWE XML distribution (The CWE List Version 4.19.1 used in this study can be found at https://cwe.mitre.org/data/index.html) and the SEI CERT C Coding Standard (The SEI CERT C rules were extracted from https://wiki.sei.cmu.edu/confluence/spaces/c/pages/87152044/SEI+CERT+C+Coding+Standard). Both sources were last accessed on 12 December 2025.

Algorithm 1 Example Faulty Sample from Our Dataset

1: Source: Toyota-ITC 2: Type: Dynamic Buffer Overrun, CWE-122: Heap-based Buffer Overflow 3: Label: Faulty (1)   4: Code:   extern volatile int vflag; void dynamic_buffer_overrun_008() {   int **buf = (int**) calloc(5, sizeof(int*));   int i, j;   for (i = 0; i < 5; i++)     buf[i] = (int*) calloc(5, sizeof(int));   for (i = 0; i < 5; i++) {     for (j = 0; j <= 5; j++) {       *(*(buf + i) + j) = i;   // Out-of-bounds write     }     free(buf[i]);   }   free(buf);  }

Similar to the relevant CWEs mapped earlier, we also extracted a collection of 34 SEI cert rules relevant to our dataset (see Table 3 and Table 4). Unlike CWE, which provides a fixed and standardized taxonomy, the SEI CERT C coding standard comprises a broad and evolving set of rules. To construct Table 3 and Table 4, we adopt a systematic filtering and mapping procedure. We begin with the full set of SEI CERT C rules and retain only those that are (i) applicable to C-language embedded and CPS contexts and (ii) semantically aligned with the CWE categories represented in our dataset. For each retained CWE category, all corresponding SEI rules are included to ensure exhaustive coverage within this scope. Consequently, Table 3 and Table 4 are complete with respect to the intersection of SEI CERT rules, embedded/CPS-relevant defects, and the curated dataset, rather than the entirety of the SEI standard.

Table 3. SEI CERT C Rules found in the dataset.

No.	Rule ID	Title
1	INT34-C	Do not shift an expression by a negative number of bits
2	ARR30-C	Do not form or use out-of-bounds pointers or array subscripts
3	MEM35-C	Allocate sufficient memory for an object
4	EXP34-C	Do not dereference null pointers
5	MSC12-C	Detect and remove dead code
6	INT31-C	Ensure that integer conversions do not result in lost or misinterpreted data
7	INT35-C	Use correct integer precisions
8	INT30-C	Ensure that unsigned integer operations do not wrap
9	INT32-C	Ensure that operations on signed integers do not overflow
10	MSC07-C	Detect and remove unused code
11	CON35-C	Ensure that shared objects are synchronized consistently
12	STR31-C	Guarantee that storage for strings has sufficient space for character data and the null terminator
13	MEM30-C	Do not access freed memory
14	MEM31-C	Free dynamically allocated memory exactly once
15	MEM34-C	Only free memory allocated dynamically
16	EXP36-C	Do not cast pointers into more strictly aligned pointer types
17	EXP37-C	Call functions with the correct number and type of arguments
18	ERR33-C	Detect and handle standard library errors
19	EXP20-C	Do not depend on the order of evaluation for side effects
20	ERR30-C	Set errno to zero before calling functions that set it
21	MSC13-C	Avoid deadlock by locking in a predefined order
22	DCL40-C	Do not create incompatible declarations of the same function or object
23	CON38-C	Ensure signals are handled consistently
24	MSC37-C	Ensure that control never reaches the end of a non-void function
25	FLP32-C	Prevent or detect domain and range errors in math functions
26	FLP34-C	Ensure that floating-point conversions are within range
27	ARR36-C	Do not subtract or compare pointers that do not refer to the same array
28	ARR37-C	Ensure that array indices are within valid bounds
29	CON33-C	Avoid race conditions when using library functions
30	CON43-C	Do not allow data races in multithreaded code
31	DCL30-C	Declare objects with appropriate storage durations
32	CON34-C	Declare objects shared between threads as atomic
33	EXP33-C	Do not read uninitialized memory
34	INT33-C	Ensure that division and remainder operations do not result in divide-by-zero errors

Table 4. Mapping of Toyota ITC Defect Types to SEI CERT C Coding Standard Rules.

Toyota ITC Defect Type	SEI CERT C Rules
Concurrency defects	CON33-C, CON35-C, CON38-C, CON43-C
Dynamic memory defects	ARR30-C, MEM35-C, STR31-C
Inappropriate code	ERR30-C, ERR33-C, EXP20-C, MSC07-C, MSC12-C, MSC13-C
Misc defects	DCL40-C, EXP33-C, MSC12-C, MSC13-C, MSC37-C
Numerical defects	FLP32-C, FLP34-C, INT30-C, INT31-C, INT32-C, INT33-C, INT34-C, INT35-C
Pointer related defects	ARR36-C, ARR37-C, EXP33-C, EXP34-C, EXP36-C, EXP37-C, MEM34-C
Resource management defects	DCL30-C, ERR33-C, EXP33-C, MEM30-C, MEM31-C, MEM34-C, MEM35-C
Stack related defects	ARR30-C, CON34-C, DCL30-C, MEM35-C
Static memory defects	ARR30-C

For the KB, we only extract information corresponding to these CWE IDs and SEI CERT rules. From the CWE repository, we extract structured textual descriptions, extended explanations, consequences, mitigations, and detection methods. From the SEI CERT C Coding Standard, we extract rule descriptions as well as compliant and non-compliant code examples.

All extracted documents are cleaned and split into semantically coherent chunks using a sentence-level segmentation strategy with a maximum chunk length of 1500 characters. Each chunk is embedded using the nomic-embed-text model, producing 768-dimensional dense vectors. The embeddings are indexed in FAISS using an inner-product index normalized to approximate cosine similarity.

3.3. Agentic Framework Foundations

Li et al. [22] define an LLM agent using five components: profile, perception, self-action, mutual interaction, and evolution. Both A-RAG and A-SFT exhibit these components, ensuring structural consistency.

• Profile: Each agent is assigned a persistent role, such as “expert static-analysis agent”, through instructional prompts.
• Perception: The agent receives code snippets, retrieved knowledge (A-RAG), or self-generated reasoning traces (A-SFT).
• Self-action: The agents perform fault classification, confidence estimation, retrieval-query generation or refinement, or prompt revision.
• Mutual interaction: The agents interact with external modules such as a vector retrieval store (A-RAG) or a fine-tuning dataset buffer (A-SFT).
• Evolution: Both frameworks implement adaptive self-prompting, where the model revises and adapts instructions, retrieval queries, or reasoning strategies based on reflection, aligning with Li et al.’s definition of agent evolution.

3.4. LLM Backbone Model

Both frameworks use Llama 3.1 8B deployed via Ollama as the LLM backbone. The Llama 3.1 8B model was selected as the backbone due to its strong balance between reasoning capability and computational efficiency. Recent advances in large language models have shown that models in the 7B–13B parameter range can achieve competitive performance on reasoning and code-related tasks, particularly when enhanced with instruction tuning and task-specific adaptation [32]. Furthermore, parameter-efficient fine-tuning methods such as LoRA enable effective adaptation of such models without requiring full model retraining, significantly reducing computational cost while maintaining strong downstream performance [33]. This makes mid-sized models particularly suitable for iterative experimentation and deployment in constrained environments. This aligns with the goals of this research, which target embedded and CPS systems with strict resource, latency, and energy constraints. In contrast, larger models (e.g., 70B+) incur substantially higher computational and memory costs, with diminishing efficiency returns relative to their scaling, making them less practical for real-time or resource-constrained deployment scenarios [34].

3.5. Agentic RAG Framework

The A-RAG framework, as shown in Figure 1, integrates autonomous reasoning, adaptive query generation, and contextual grounding through the external vector KB. While the dataset provides concrete faulty and corrected program instances, the CWE and SEI KBs supply the model with semantic and conceptual information, enabling more interpretable and defect-aware decision-making. A-RAG operates on the five agentic components of Li et al. [22] through an iterative loop that progressively improves classification confidence by invoking retrieval only when necessary.

Figure 1. Agentic Retrieval-Augmented Generation (A-RAG) framework.

3.5.1. Initial Detection and Confidence Estimation

Given an input code snippet, the agent first performs a standalone classification using a structured baseline prompt. This prompt defines the agent’s role, the required output fields, and the expected reasoning style:

To quantify the reliability of this initial classification, the agent evaluates its own certainty using a dedicated numerical confidence prompt:

We define confidence as the model’s probability assigned to the predicted class. This signal is used to trigger adaptive self-prompting. If the confidence score exceeds the predefined threshold $\tau =0.75$, the agent accepts its decision without invoking retrieval. Otherwise, it engages in an adaptive retrieval cycle.

We selected this value of $\tau$ as a conservative mid-to-high operating point for separating predictions that appear reliable for single-pass classification from those that need additional evidence gathering. This choice is consistent with recent adaptive RAG literature, where retrieval decisions are commonly governed by confidence- or uncertainty-based thresholds [35,36,37,38,39]. However, the specific cutoff is treated as a task- and model-dependent hyperparameter rather than a universal constant. In our setting, $\tau =0.75$ was therefore used as a reasonable fixed decision boundary for this initial study of confidence-triggered adaptive prompting. Since the primary objective of this work is to investigate whether adaptive self-prompting can improve fault detection over static mechanisms, we did not delve into exhaustive threshold optimization or calibration analysis.

3.5.2. Adaptive Query Generation and Retrieval

When uncertainty is detected, the agent shifts from pure reasoning to guided retrieval. Instead of using a fixed retrieval prompt, A-RAG generates a context-aware, adaptive retrieval query that focuses on the specific aspects of the snippet that caused uncertainty. This process forms the core of the agent’s reflective evolution.

The prompt reformulates the agent’s previous output trace into an explicit search query aligned with CWE and SEI semantics:

The resulting query is embedded and used to retrieve the top-5 most similar chunks from the unified CWE+SEI vector index. These chunks typically contain defect explanations, unsafe usage patterns, compliant/noncompliant examples, and relevant diagnostic guidance.

3.5.3. Context-Augmented Re-Evaluation

The retrieved knowledge provides the agent with explicit guidance on known vulnerability classes, typical triggering conditions, and canonical examples relevant to the snippet. The agent integrates these elements into a second-stage evaluation, where the original snippet and retrieved context are jointly analyzed.

A second confidence estimation is produced, and if it remains below $\tau$, the adaptive retrieval loop may be repeated once more. This two-stage adaptive cycle ensures that retrieval is invoked only when necessary, preserves efficiency, and avoids over-reliance on external knowledge.

3.5.4. Algorithmic Formulation

To formalize the adaptive self-prompting mechanism in A-RAG, Algorithm 2 summarizes the end-to-end inference procedure. The pseudocode highlights the conditional control flow driven by model confidence and the iterative refinement of reasoning through retrieval-triggered self-prompting.

3.6. Agentic SFT Framework

The A-SFT framework, as shown in Figure 2, strengthens the model’s internal decision-making through autonomous self-evaluation, instruction refinement, and parameter-efficient supervised fine-tuning. A-SFT, therefore, complements A-RAG by improving the underlying model parameters rather than relying on external context. Consistent with the unified agent architecture of Li et al. [22], A-SFT instantiates the five agentic components through a closed-loop learning process that evolves the agent’s reasoning strategies over time.

Algorithm 2 Agentic Retrieval-Augmented Generation (A-RAG)

Require:  Code snippet x, confidence threshold $\tau =0.75$, maximum retrieval rounds $R=2$ Ensure:  Final classification label $\widehat{y}$ and reasoning trace $\mathcal{R}$ 1:                 ▹ Phase 1: Initial Detection and Confidence Estimation 2: $(\widehat{y},\mathcal{R})\leftarrow \mathrm{INITIALDETECTIONPROMPT}\left(x\right)$ 3: $c\leftarrow \mathrm{CONFIDENCEESTIMATIONPROMPT}(x,\widehat{y},\mathcal{R})$ 4: if  $c\ge \tau$  then 5:         return $(\widehat{y},\mathcal{R})$ 6: end if 7: $r\leftarrow 0$ 8: while  $c<\tau$  and  $r<R$  do 9:                  ▹ Phase 2: Adaptive Query Generation and Retrieval 10:     $q\leftarrow \mathrm{ADAPTIVEQUERYPROMPT}(x,\widehat{y},\mathcal{R},c)$ 11:     $\mathcal{K}\leftarrow \mathrm{VECTORRETRIEVE}\left(q\right)$ 12:                      ▹ Phase 3: Context-augmented re-evaluation 13:     $(\widehat{y},\mathcal{R})\leftarrow \mathrm{CONTEXTAUGMENTEDPROMPT}(x,\mathcal{K})$ 14:     $c\leftarrow \mathrm{CONFIDENCEESTIMATIONPROMPT}(x,\widehat{y},\mathcal{R})$ 15:     $r\leftarrow r+1$ 16: end while 17: return  $(\widehat{y},\mathcal{R})$

Figure 2. Agentic Supervised Fine-Tuning (A-SFT) Framework. The component enclosed in the dotted box is an expanded view of the Self-Evaluation Sweep.

3.6.1. Adaptive Self-Evaluation Sweep

The A-SFT pipeline begins with an independent inference sweep over the dataset’s training split. For each example, the agent produces:

1.

an initial classification and explanation using the same structured baseline prompt employed in A-RAG’s initial detection stage,

2.

a numerical confidence score derived from the confidence-estimation prompt, and

3.

a reflective self-evaluation of its own reasoning when confidence score is less than the threshold $\tau =0.75$ or classification is incorrect.

The self-evaluation stage uses the following template:

The final reasoning (original or corrected) generated for each sample during the self-evaluation sweep over the dataset is later used in the training sample for fine-tuning. The self-critiques are stored for system instruction adaptation.

3.6.2. Instruction Adaptation

After completing the detection and self-evaluation sweep, A-SFT aggregates the agent’s critiques to identify systematic reasoning weaknesses. These patterns drive a targeted refinement of the agent’s system instruction using the following prompt:

The resulting instruction serves as a persistent role refinement for the agent and is appended to the supervised training set. This mechanism instantiates the agent’s adaptive component by enabling the model to update its own guidance policies across training epochs.

3.6.3. Supervised Fine-Tuning

The final fine-tuning dataset integrates both the corrected reasonings and the revised instructions. Each training sample adheres to a structured format consisting of:

1.

the input C code snippet,

2.

the updated system instruction,

3.

the actual binary fault label, and

4.

a concise explanation aligned with the defect or vulnerability categories.

This ensures that fine-tuning reinforces not only correct classifications but also the refined reasoning principles developed during the self-evaluation sweep.

Fine-tuning is performed using the Ollama runtime with LoRA adapters applied to the Llama 3.1 8B backbone. This parameter-efficient configuration maintains training stability while accommodating long code snippets and detailed explanations. The full hyperparameter configuration is provided in Table 5.

Table 5. LoRA fine-tuning hyperparameters for the A-SFT framework.

Hyperparameter	Value
Base model	Llama 3.1 8B
Adapter type	LoRA
LoRA rank (r)	8
LoRA $\alpha$	16
Dropout	0.05
Learning rate	$2 \times {10}^{-5}$
Batch size	4
Sequence length	4096 tokens
Optimizer	AdamW
Weight decay	0.01
Loss function	BCE + explanation consistency
Training epochs	3
Precision	bfloat16
Gradient clipping	1.0
Warmup ratio	0.03

3.6.4. Algorithmic Formulation

To formalize the adaptive self-prompting mechanism in A-SFT, Algorithm 3 summarizes the closed-loop training procedure. In contrast to A-RAG, which adapts inference-time reasoning through conditional retrieval, A-SFT concentrates all adaptive behavior in the training phase. The pseudocode highlights how confidence, self-critique, and instruction refinement govern training exemplar construction and parameter updates.

Algorithm 3 Agentic Supervised Fine-Tuning (A-SFT)

Require:  Code dataset $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^N$, initial instruction ${\mathcal{I}}_0$, confidence threshold $\tau =0.75$ Ensure:  Fine-tuned model parameters ${\theta }^{★}$ and adapted instruction ${\mathcal{I}}^{★}$ 1:  Initialize model parameters $\theta$ 2:  $\mathcal{E}\leftarrow Ø$                        ▹ Corrected exemplar buffer 3:  $\mathcal{C}\leftarrow Ø$                             ▹ Self-critique buffer 4:                       ▹ Phase 1: Adaptive Self-Evaluation Sweep 5:  for each $(x_i,y_i)\in \mathcal{D}$ do 6:        $({\widehat{y}}_i,{\mathcal{R}}_i)\leftarrow \mathrm{INITIALDETECTIONPROMPT}(x_i,{\mathcal{I}}_0)$ 7:        $c_i\leftarrow \mathrm{CONFIDENCEESTIMATIONPROMPT}(x_i,{\widehat{y}}_i,{\mathcal{R}}_i)$ 8:        if $c_i\ge \tau$ and ${\widehat{y}}_i=y_i$ then 9:             $\mathcal{E}\leftarrow \mathcal{E}\cup \left\{(x_i,y_i,{\mathcal{R}}_i)\right\}$ 10:      else 11:           $({\mathcal{S}}_i,{\stackrel{\sim }{y}}_i,{\stackrel{\sim }{\mathcal{R}}}_i)\leftarrow \mathrm{SELFEVALUATIONPROMPT}(x_i,{\widehat{y}}_i,y_i,{\mathcal{R}}_i,c_i)$ 12:           $\mathcal{C}\leftarrow \mathcal{C}\cup \left\{{\mathcal{S}}_i\right\}$ 13:           $\mathcal{E}\leftarrow \mathcal{E}\cup \left\{(x_i,y_i,{\stackrel{\sim }{\mathcal{R}}}_i)\right\}$ 14:      end if 15:  end for 16:                          ▹ Phase 2: Instruction Adaptation 17:  ${\mathcal{C}}_{\mathrm{agg}}\leftarrow \mathrm{AGGREGATECRITIQUES}\left(\mathcal{C}\right)$ 18:  ${\mathcal{I}}^{★}\leftarrow \mathrm{INSTRUCTIONADAPTATIONPROMPT}\left({\mathcal{C}}_{\mathrm{agg}}\right)$ 19:                         ▹ Phase 3: Supervised Fine-Tuning 20:  Construct augmented dataset ${\mathcal{D}}^{★}$ using:      $(x_i,y_i,{\mathcal{R}}_i)$ for high-confidence correct samples      $(x_i,y_i,{\stackrel{\sim }{\mathcal{R}}}_i)$ for corrected samples      updated instruction ${\mathcal{I}}^{★}$ 21:  ${\theta }^{★}\leftarrow \mathrm{LORAFINETUNE}(\theta ,{\mathcal{D}}^{★})$ 22:  return  $({\theta }^{★},{\mathcal{I}}^{★})$

A-SFT thus provides an internal complement to A-RAG. While A-RAG injects external semantic guidance when uncertainty is detected, A-SFT improves the model’s intrinsic classification and reasoning capabilities through iterative self-improvement. Another key consequence of this design is that all adaptive reasoning and self-prompting in A-SFT are confined to training time. At deployment, the fine-tuned policy model operates with a fixed system instruction and performs fault detection in a single forward pass, without confidence estimation, retrieval, self-critique, or multi-stage control flow. In contrast to agentic inference-time frameworks such as A-RAG, A-SFT therefore shifts computational overhead entirely to training, enabling deterministic and low-latency inference suitable for deployment in resource-constrained embedded and CPS environments.

3.7. Theoretical Grounding

This section provides theoretical and conceptual foundations for why adaptive self-prompting improves reasoning in our agentic fault-detection setting. We use adaptive self-prompting as an umbrella term for (i) confidence-triggered reflection that modifies the inference process (e.g., deciding whether to retrieve, how to retrieve, and how to re-evaluate), and (ii) training-time self-evaluation that produces corrective feedback and updates persistent system instructions. Although our task domain is function-level C fault detection, the mechanisms we employ are instances of general LLM reasoning principles: eliciting intermediate structure, allocating computation/attention where uncertainty is highest, and introducing self-verification signals that reduce overconfident failure.

3.7.1. Reasoning as Structured Intermediate Computation

A central result in prompt-based reasoning is that LLM performance on multi-step tasks improves when the model generates explicit intermediate steps rather than attempting a single-shot answer. Chain-of-thought (CoT) prompting demonstrates that providing or eliciting intermediate reasoning steps can unlock latent capabilities in sufficiently large models and improve accuracy on tasks that require multi-hop inference, arithmetic, and symbolic manipulation [40,41]. Subsequent work shows that how intermediate reasoning is generated matters. Sampling multiple reasoning paths and selecting the most consistent outcome improves robustness over greedy decoding because complex problems admit multiple valid latent reasoning paths [42]. Related decomposition strategies, such as least-to-most prompting, explicitly break a complex problem into simpler subproblems to enable easy-to-hard generalization [43].

Our adaptive self-prompting mechanisms instantiate these same principles in a code-analysis setting. Fault detection often requires multiple latent sub-decisions (e.g., checking boundary conditions, validating arithmetic ranges, and reasoning about concurrency ordering). Static prompting forces these sub-decisions to be compressed into a single forward pass, increasing the likelihood of premature commitment to an incorrect hypothesis. In contrast, adaptive self-prompting creates structured intermediate computation:

• Reflection: this extracts uncertainty into an explicit variable (a confidence score) that can control the inference policy.
• Adaptive query generation: this decomposes the task by translating uncertain code behaviors into targeted retrieval intents (similar to forming subqueries).
• Re-evaluation: this encourages a second-pass inference conditioned on grounded evidence, analogous to multi-sample or multi-path reasoning, but with knowledge grounding rather than only stochastic decoding.

Thus, adaptive self-prompting takes advantage of the general theoretical claim from CoT-style work, forcing intermediate structure, which reduces the complexity of the next prediction step [40,43].

3.7.2. Adaptive Compute Allocation Under Uncertainty

A second foundation is that reasoning improvements often come from allocating additional computation selectively to hard cases. In active-learning inspired prompting, uncertainty estimates are used to identify inputs where additional supervision or improved prompting yields the largest gains. Active-Prompt formalizes this idea by selecting uncertain instances for annotation, demonstrating that uncertainty is a meaningful proxy for where extra reasoning resources are most valuable [44]. More generally, modern agentic LLM studies frame problem solving as an adaptive loop where the model decides when to think more, ask itself subquestions, or consult tools [45,46].

Our A-RAG design follows this principle via confidence-triggered retrieval. Retrieval is not uniformly beneficial. Injecting irrelevant or weakly related context can distract the model, increase prompt length, and introduce spurious correlations or contradictions. Selective retrieval can therefore improve reasoning by (i) avoiding unnecessary context when the model is already confident, and (ii) providing additional grounding precisely when uncertainty indicates missing knowledge or ambiguous evidence. This aligns with self-reflective retrieval frameworks that explicitly learn or prompt the model to retrieve on demand and to critique the usefulness of retrieved passages [35]. In our setting, this theory predicts an improvement not only in aggregate performance (e.g., F1), but also in risk calibration, where uncertain cases should be more likely to trigger retrieval and be converted into correct predictions or, at least, less likely to remain confidently wrong.

3.7.3. Self-Verification and Error-Correction as Test-Time Learning

A third foundation is that iterative self-critique can function as a lightweight form of test-time learning. Self-Refine shows that an LLM can improve output quality by generating an initial attempt, critiquing it, and refining it iteratively without weight updates [47]. Reflexion similarly frames self-reflection as a mechanism for learning from mistakes through linguistic feedback stored as memory, improving subsequent decisions [48]. These methods support a general interpretation: LLM outputs can be treated as hypotheses, and self-generated critiques act as approximate gradients that steer the next generation away from known failure modes.

A-RAG uses self-verification implicitly as the model first commits to a hypothesis (faulty vs. non-faulty), then estimates confidence, then conditionally gathers evidence and re-evaluates the hypothesis. This mirrors verification-driven reasoning, where a second pass checks whether the initial reasoning is consistent with grounded principles (here, CWE and SEI CERT guidance). Essentially, this converts some error modes from confident misclassification into either correct predictions or explicitly uncertain outputs, improving the safety profile of the system for CPS/embedded settings where overconfident errors are operationally costly.

3.7.4. Why Adaptive Self-Prompting Improves Reasoning in A-SFT

While A-RAG improves reasoning primarily at inference time, A-SFT internalizes the same benefits into model parameters and persistent instructions. The theoretical motivation is that self-evaluation produces targeted hard examples and policy updates:

• Hard-example mining via uncertainty and self-revision: Cases that are low-confidence or revised during self-critique represent regions where the model’s implicit decision boundary is underspecified. This is analogous to selecting informative points in uncertainty-based active learning [44].
• Error-pattern extraction and instruction revision: Aggregating critiques yields a distribution over recurring reasoning failures (e.g., missing integer promotion rules, ignoring ownership/lifetime signals, confusing safe vs. unsafe pointer arithmetic). Updating the system instruction encodes these patterns as explicit decision heuristics, similar to decomposition prompts and structured reasoning policies [40,43].

From a learning perspective, A-SFT reduces the gap between knowing and using relevant heuristics as the critique and refined instruction make latent constraints explicit and repeatedly reinforced during fine-tuning. This can be interpreted as a form of bootstrapped reasoning supervision, where the model generates rationales and corrections that are then used to shape subsequent behavior [42,49].

3.8. Performance Evaluation

We evaluated the proposed agentic frameworks using a set of rigorous baseline comparisons designed to isolate the contribution of agentic self-prompting, confidence-based reflection, and adaptive reasoning. All experiments are conducted using the same dataset, tasks, and hardware platform (NVIDIA GeForce RTX 3070 Ti 8 GB GPU, 16 GB system RAM, Intel i7-12800H CPU) to ensure fair and controlled comparisons.

3.8.1. Comparison Baselines

For model-level baselines, we compare against the isolated Llama 3.1 8B backbone used within our frameworks and also CodeLlama-7B. These comparisons help distinguish improvements due to framework design rather than underlying model capacity.

Next, we include component-level ablations derived directly from the A-RAG and A-SFT pipelines. We use a RAG-only baseline which retains the same vector knowledge base, retrieval mechanism, and generation model used in A-RAG, but removes all agentic components. In particular, the confidence-based reflection, adaptive self-prompting, and iterative retrieval stages are omitted. The model, therefore, performs a single-pass retrieval followed by response generation, allowing us to isolate the impact of agentic control beyond retrieval augmentation alone.

In addition, we use an SFT-only baseline, which applies supervised fine-tuning to the base LLaMA model without the agentic self-evaluation sweep used in A-SFT. The confidence estimation, adaptive prompt refinement, and iterative reasoning mechanisms are removed, resulting in a conventional parameter-efficient supervised fine-tuning setup. This comparison isolates the contribution of agentic self-prompting and reflection beyond standard SFT.

Other than decoder-only LLM baselines, we include GraphCodeBERT as a strong encoder-based baseline for fault detection. GraphCodeBERT is a Transformer encoder pretrained on source code using both masked language modeling and structure-aware objectives, and it has been widely adopted in prior work on learning-based vulnerability detection, including evaluations on Big-Vul-style datasets [6,7,8].

For our experiments, we fine-tune the pretrained graphcodebert -base model for binary function-level fault classification using a standard sequence classification head. Each code snippet fed to the model is tokenized and truncated to a maximum sequence length of 512 tokens to ensure compatibility with the encoder architecture and to maintain efficient training on the available hardware. GraphCodeBERT serves as a non-agentic, encoder-only reference point, allowing us to compare conventional pretrained code representation learning approaches against our agentic LLM frameworks.

3.8.2. Prediction and Computation Metrics

We evaluate all models in a binary classification setting, where each code sample is labeled as faulty (1) or non-faulty (0). We report accuracy, precision, recall, and F1 score, with F1 serving as the primary metric due to its balance between precision and recall.

To further analyze model behavior, we examine the confusion matrices of A-RAG and A-SFT. This analysis provides insight into the distribution of true positives, false positives, true negatives, and false negatives, enabling a more nuanced understanding of how each model handles faulty versus non-faulty samples. In particular, confusion matrices allow us to identify systematic error patterns, such as a tendency toward false negatives (missed faults) or false positives (over-flagging safe code), which are critical in safety-sensitive embedded and CPS contexts. By comparing these distributions across frameworks, we assess not only overall predictive performance but also the reliability and risk profile of each approach.

Since the dataset encompasses a diverse set of fault categories, we additionally report per-CWE F1 scores of the two frameworks for the 10 most and 10 least occurring CWE IDs in the dataset. This breakdown enables fine-grained analysis of the model’s capability to detect memory, pointer, numerical, and concurrency-related bugs individually, highlighting semantic strengths and weaknesses across fault types. For the least frequent CWE IDs, results are averaged over 10 independent runs to mitigate the high variance introduced by small sample sizes and to obtain more stable performance estimates. Despite this, these results should be interpreted with caution due to the limited number of samples per category.

In addition to predictive performance, we report inference runtime (average time per sample) and peak GPU memory usage to assess computational efficiency and deployment feasibility. For fine-tuned frameworks, we also report the overall training time to assess the complexity of learning necessary parameters for domain-specific tuning.

3.8.3. Statistical Significance Testing

To determine whether performance differences between frameworks are statistically significant, we conduct paired statistical significance testing on the test set. Because all models are evaluated on the same function-level samples in a binary classification setting, paired comparison tests are used.

We apply McNemar’s test to compare models based on their paired prediction outcomes (correct versus incorrect) for each test sample. McNemar’s test is specifically designed for comparing two classifiers evaluated on the same dataset and does not assume normality of the performance distribution. Statistical significance is assessed at a significance level of $\alpha =0.05$.

Although the F1 score is reported as the primary evaluation metric, McNemar’s test is applied to paired prediction correctness because it directly evaluates whether two classifiers differ significantly in their error distributions on the same test samples. In binary classification, F1 is a deterministic function of the confusion matrix (TP, FP, FN), and any statistically significant difference in paired misclassification rates necessarily reflects a change in the underlying error structure from which F1 is computed. Therefore, McNemar’s test provides a statistically sound basis for supporting conclusions drawn from F1 improvements while avoiding distributional assumptions.

Pairwise comparisons are performed between each proposed agentic framework (A-RAG and A-SFT) and all baseline models. Since 11 pairwise comparisons are performed between the proposed agentic frameworks and baseline models, Bonferroni correction is applied by dividing the significance level $\alpha =0.05$ by the number of comparisons (11), resulting in an adjusted threshold of $\alpha =0.0045$ to control the family-wise error rate.

3.8.4. Confidence and Error Taxonomy

Beyond aggregate classification metrics, we perform a fine-grained error analysis to characterize the qualitative behavior and risk profile of each model. Predictions are categorized based on correctness, error type, and model-reported confidence, resulting in six distinct taxonomy classes:

• Confident Correct (CC): Correct predictions with high confidence.
• Uncertain Correct (UC): Correct predictions with low confidence.
• Confident False Positive (CFP): Non-faulty code incorrectly classified as faulty with high confidence.
• Uncertain False Positive (UFP): Non-faulty code incorrectly classified as faulty with low confidence.
• Confident False Negative (CFN): Faulty code incorrectly classified as non-faulty with high confidence.
• Uncertain False Negative (UFN): Faulty code incorrectly classified as non-faulty with low confidence.

Model confidence is defined as the normalized probability of the predicted class. For encoder-based classifiers, this probability is obtained from the softmax output of the classification head. For generative LLM-based classifiers, confidence is computed from token-level log-probabilities by normalizing the likelihood of generating the predicted label string against the alternative label. A fixed confidence threshold of 0.75 is applied uniformly across all models to categorize predictions as confident or uncertain, ensuring consistent comparison and isolating differences attributable to reasoning behavior rather than probability calibration.

The confidence values used in the taxonomy correspond to model self-reported probabilities. No external calibration (e.g., temperature scaling) was applied, and formal calibration metrics such as Expected Calibration Error were not computed. Therefore, the analysis evaluates relative internal certainty and its alignment with correctness rather than formally calibrated probabilities. Improvements observed in the agentic frameworks reflect reduced high-confidence misclassification and stronger alignment between certainty and correctness.

3.8.5. Failure-Cases

We further conduct an in-depth failure-case analysis focusing on high-risk error categories. Failure cases are identified from the test split by collecting all misclassified samples and stratifying them by CWE ID and confidence category as defined in the confidence-aware error taxonomy. We additionally inspect intermediate artifacts, retrieval queries, and retrieved context for A-RAG, and self-critiques for A-SFT, to determine whether failures arise from missing evidence, incorrect reasoning, or limitations in representation and training signal. This qualitative inspection is used to derive failure patterns rather than to exhaustively analyze individual samples.

3.8.6. Illustrative CPS Case Study

The quantitative evaluation examines the impact of adaptive self-prompting on fault detection performance across both synthetic and real-world benchmarks. However, benchmark metrics alone do not reveal how agentic LLM frameworks reason about faults in safety-critical CPS, where numerical computation, control-flow logic, and safety supervision mechanisms interact in non-trivial ways.

To complement these aggregate metrics, we also conduct a focused qualitative case study that examines how agentic self-prompting manifests in a realistic CPS setting based on a single, synthetic automotive control routine. Rather than providing an exhaustive evaluation of CPS behaviors, this case study serves as an illustrative walkthrough designed to expose the internal reasoning processes of the proposed agentic frameworks when applied to realistic embedded control logic. The objective is to demonstrate how confidence estimation, reflection, and adaptive reasoning manifest in a CPS context, rather than to characterize all possible failure modes.

4. Results and Discussion

This section discusses the results of the performance evaluation of the proposed agentic frameworks and presents a CPS-focused case study to qualitatively examine agentic self-prompting in safety-critical embedded systems.

4.1. Prediction Analysis

Figure 3 compares prediction performance across all models and highlights the impact of agentic reasoning, adaptive learning, and external knowledge integration.

Figure 3. Prediction Metrics Comparison Heatmap.

The isolated Llama 3.1 8B and CodeLlama-7B achieve the lowest F1 scores (around 70%), reflecting limited domain specialization and the absence of structured reasoning. GraphCodeBERT improves performance to approximately 76.7% F1 due to its structure-aware pretraining, but its fixed encoder context and lack of adaptive mechanisms restrict its effectiveness on long, function-level samples from Big-Vul and Toyota ITC.

RAG-only improves performance to roughly 78.8% F1, confirming the value of CWE and SEI CERT knowledge for fault detection. However, uniform retrieval introduces unnecessary context in confident cases. SFT-only further improves F1 to about 80.3% by adapting model parameters to the dataset, but static instructions limit its ability to correct recurring reasoning errors.

A-RAG achieves a substantial gain, reaching approximately 84.5% F1. Its confidence-based reflection and adaptive query generation ensure retrieval is invoked only when needed and is tightly aligned with the code’s uncertain behaviors. A-SFT delivers the best performance at approximately 86.3% F1, benefiting from its closed-loop self-evaluation and instruction refinement, which internalize corrected reasoning directly into model parameters.

Overall, Figure 3 shows that agentic control, not retrieval or fine-tuning alone, is the primary driver of performance gains. Among all approaches, A-SFT provides the most robust generalization across heterogeneous fault distributions.

4.2. Confusion Matrix Analysis

Figure 4 displays the confusion matrices of both the proposed frameworks. They provide insight into the error characteristics of both frameworks. A-RAG correctly identifies 522 faulty and 507 non-faulty samples, but yields 88 false negatives and 103 false positives. In contrast, A-SFT improves performance with 533 true positives and 519 true negatives, while reducing errors to 77 false negatives and 91 false positives.

Figure 4. Confusion Matrices of the Proposed Frameworks.

Overall, A-SFT demonstrates consistent improvements in both fault detection and false alarm reduction, indicating a better balance between sensitivity and specificity. As summarized in earlier Figure 3, these gains translate into higher accuracy, precision, recall, and F1-score, confirming the effectiveness of the agentic supervised fine-tuning approach over A-RAG.

4.3. Per-CWE Performance Analysis

We report the F1 scores for the ten most frequent CWE categories in the dataset in Table 6. We also report the average F1 scores for the ten least frequent CWE categories across 10 independent runs in Table 7. This is to examine how the two frameworks behave across vulnerability types with varying data availability.

Table 6. F1 performance on the 10 most frequent CWE categories in the dataset.

CWE ID	Count	A-RAG F1 (%)	A-SFT F1 (%)
CWE-416	618	86.2	88.1
CWE-189	600	85.4	87.3
CWE-190	598	85.1	87.0
CWE-476	530	84.3	86.2
CWE-362	512	83.6	85.4
CWE-415	166	80.5	82.6
CWE-121	154	79.4	81.7
CWE-786	118	78.6	80.8
CWE-835	94	77.8	79.9
CWE-122	86	76.9	78.7

Table 7. Average F1 performance on the 10 least frequent CWE categories in the dataset across 10 runs.

CWE ID	Count	A-RAG F1 (%)	A-SFT F1 (%)
CWE-764	2	65.0	64.0
CWE-1260	4	66.2	65.1
CWE-465	4	66.0	65.4
CWE-469	4	67.1	66.0
CWE-480	4	67.4	66.3
CWE-562	4	68.0	66.9
CWE-463	6	68.6	67.8
CWE-393	8	69.2	68.5
CWE-484	8	69.6	68.9
CWE-833	10	70.1	69.4

For high-frequency CWEs, both A-RAG and A-SFT achieve strong performance, with A-SFT consistently outperforming A-RAG. Frequently occurring categories such as CWE-416, CWE-189, and CWE-190 exhibit recurring syntactic and semantic patterns that are well represented in the training data. Through supervised fine-tuning with corrected exemplars and refined instructions, A-SFT internalizes these patterns, leading to higher and more stable F1 scores. In contrast, A-RAG does not update model parameters and relies on in-context reasoning and conditional retrieval, which limits performance gains on common CWEs.

For low-frequency CWEs, such as CWE-764, CWE-1260, and CWE-465, both A-RAG and A-SFT exhibit lower and more variable F1 scores due to the extremely small number of evaluation samples. In this setting, per-CWE F1 is highly sensitive to individual prediction errors, and small absolute differences should not be interpreted as systematic performance advantages despite averaging the scores across multiple runs. While A-SFT is further constrained by limited supervised exposure to these rare vulnerability patterns during fine-tuning, A-RAG does not perform additional training and is therefore not directly affected by data sparsity in the same manner. However, A-RAG’s performance on rare CWEs remains subject to evaluation variance rather than reflecting a consistent architectural advantage. Overall, these results indicate that per-CWE analysis for rare vulnerabilities should be interpreted qualitatively, with aggregate metrics providing a more reliable comparison between the two frameworks.

Overall, the per-CWE analysis highlights a complementary trade-off. A-SFT is most effective for frequent CWEs such as CWE-416 and CWE-190, where sufficient data supports parameter learning, while A-RAG remains competitive for rare and underrepresented CWEs such as CWE-764 by leveraging adaptive, knowledge-grounded reasoning without additional training.

4.4. Computational Analysis

Figure 5 reports inference runtime, peak GPU memory usage, and training cost across all evaluated models. These results reflect the architectural and learning differences described in Section 3, as well as the constrained hardware environment (8 GB GPU, 16 GB RAM, Intel i7-12800H CPU).

Figure 5. Computation Performance Comparison.

GraphCodeBERT achieves the lowest inference latency and memory usage due to its encoder-only architecture and short input length (512 tokens). Decoder-only baselines (Llama 3.1 8B and CodeLlama-7B) incur higher inference costs as a result of autoregressive generation over long code sequences, with similar computational profiles driven primarily by model scale.

The RAG-only baseline increases inference time due to mandatory embedding computation and vector retrieval for every sample. A-RAG further increases runtime, as shown in Figure 5, due to its agentic design with confidence estimation, adaptive query generation, and conditional multi-stage reasoning. Since retrieval is triggered only when confidence falls below $\tau =0.75$, the reported cost reflects an average of single-pass and iterative executions. Peak GPU memory remains comparable to the base model, indicating computation-bound rather than memory-bound overhead.

In contrast, A-SFT shows inference efficiency similar to the base LLM and the SFT-only baseline. Once LoRA adapters are trained, inference requires a single forward pass with no retrieval or agentic loops. This efficiency is achieved at the cost of increased training time, as A-SFT incorporates self-evaluation, critique aggregation, and instruction refinement during training. This demonstrates that the benefits of agentic self-prompting in A-SFT are realized without incurring additional deployment-time overhead.

Overall, Figure 5 highlights complementary trade-offs where A-RAG favors adaptive, knowledge-grounded reasoning with higher inference cost, while A-SFT shifts computation to training to enable efficient deployment-time inference under limited hardware resources.

4.5. Statistical Significance

Figure 6 visualizes the statistical significance of all pairwise comparisons using McNemar’s test with Bonferroni correction. Comparisons exceeding the threshold are clearly separated from those that do not. Exact p-values for all comparisons are reported in Table 8. Both proposed agentic frameworks significantly outperform the non-agentic baselines (Llama 3.1 8B, CodeLlama-7B, and GraphCodeBERT), with p-values below the Bonferroni-adjusted threshold. These results confirm that the observed improvements in aggregate F1 are not attributable to random variation in test-set predictions.

Figure 6. Pairwise the statistical significance based on McNemar’s test. Bars show $-{log}_{10}\left(p\right)$, and the dashed line indicates the Bonferroni-corrected threshold ($\alpha =0.0045$).

Table 8. McNemar test results for pairwise comparisons.

Model A	Model B	p-Value	Significant
A-RAG	Llama 3.1 8B	0.0004	Yes
A-SFT	Llama 3.1 8B	0.00001	Yes
A-RAG	CodeLlama-7B	0.0002	Yes
A-SFT	CodeLlama-7B	0.000008	Yes
A-RAG	GraphCodeBERT	0.0007	Yes
A-SFT	GraphCodeBERT	0.00009	Yes
A-RAG	RAG-only	0.0015	Yes
A-SFT	RAG-only	0.0002	Yes
A-RAG	SFT-only	0.0042	Yes
A-SFT	SFT-only	0.0018	Yes
A-SFT	A-RAG	0.0027	Yes

Similarly, the proposed frameworks significantly outperform the component-level ablations (RAG-only and SFT-only). This outcome suggests that both inference-time adaptive retrieval and training-time adaptive self-prompting provide meaningful gains over static retrieval and conventional supervised fine-tuning.

Direct comparisons among the proposed frameworks further clarify their relative contributions. A-SFT significantly outperforms A-RAG, indicating that training-time adaptive self-prompting and instruction refinement yield measurable improvements beyond inference-time retrieval alone.

Overall, the statistical analysis supports the central claim of this work. Agentic control mechanisms, particularly adaptive self-prompting, produce statistically significant improvements over non-agentic baselines and static retrieval or fine-tuning pipelines, with A-SFT accounting for the majority of the observed performance gains.

4.6. Confidence-Aware Error Taxonomy Analysis

Classification metrics aside, we analyze model behavior through a confidence-aware error taxonomy that captures not only correctness, but also the certainty with which decisions are made. Table 9 reports the distribution of predictions across six categories, distinguishing correct and incorrect outcomes while separating confident decisions from uncertain ones using a fixed confidence threshold of 0.75.

Table 9. Error Taxonomy Across Models.

Models	CC (%)	UC (%)	CFP (%)	UFP (%)	CFN (%)	UFN (%)
Llama 3.1 8B	48.3	22.9	7.6	7.6	7.4	6.1
CodeLlama-7B	44.5	24.2	9.9	6.6	9.0	6.0
GraphCodeBERT	49.6	26.8	4.4	8.2	3.9	7.2
RAG only	56.5	21.8	5.3	6.4	4.5	5.4
SFT only	60.1	20.0	3.7	6.9	3.7	5.6
A-RAG	71.6	12.6	1.3	7.2	1.4	5.8
A-SFT	75.8	10.3	0.9	6.6	1.0	5.4

Across all models, strong performance is characterized by a high proportion of Confident Correct (CC) predictions and a low incidence of Confident False Negatives (CFN) and Confident False Positives (CFP), which represent the most operationally risky failure modes. The non-agentic LLM baselines exhibit relatively modest CC rates (48.3% for Llama 3.1 8B and 44.5% for CodeLlama-7B), coupled with higher levels of confident misclassification. In particular, both models produce nearly symmetric rates of confident false positives and false negatives, indicating limited self-awareness when reasoning about ambiguous or edge-case code patterns.

GraphCodeBERT improves over decoder-only baselines in terms of error calibration, reducing confident errors (CFP + CFN = 8.3%) while increasing the share of uncertain correct predictions. This behavior suggests that structure-aware pretraining encourages caution, but also leads to a larger fraction of correct decisions being made with low confidence, reflecting limited expressiveness when handling long or semantically complex functions.

Introducing retrieval or fine-tuning in isolation yields further improvements. The RAG-only and SFT-only variants both increase CC rates (56.5% and 60.1%, respectively) and reduce confident errors relative to base models. However, both approaches still retain a non-trivial proportion of uncertain predictions, indicating that static retrieval pipelines and conventional supervised fine-tuning do not fully resolve uncertainty in difficult cases.

The proposed agentic frameworks exhibit a qualitatively different error profile. A-RAG achieves a substantial increase in confident correct predictions (71.6%) while sharply reducing both CFP (1.3%) and CFN (1.4%). This shift demonstrates the effect of confidence-triggered retrieval: uncertain cases are selectively augmented with external knowledge, converting what would otherwise be confident errors into either correct decisions or explicitly uncertain outcomes. Notably, A-RAG slightly increases the proportion of uncertain false positives compared to SFT-only, reflecting a conservative bias where the agent prefers uncertainty over overconfident misclassification.

A-SFT yields the most favorable distribution overall, with the highest CC rate (75.8%) and the lowest combined confident error rate (CFP + CFN = 1.9%). By internalizing corrected reasoning patterns through self-evaluation and instruction refinement, A-SFT reduces both uncertainty and overconfidence at inference time. Compared to A-RAG, A-SFT further shifts predictions from the uncertain categories (UC and UFN) into confident correct outcomes, indicating that adaptive self-prompting during training leads to more stable and decisive reasoning at deployment.

Overall, this confidence-aware error analysis reveals that agentic self-prompting improves not only accuracy but also the risk profile of model decisions. While non-agentic models frequently exhibit overconfident failures, the proposed agentic frameworks systematically suppress confident misclassification and promote either correct high-confidence decisions or explicit uncertainty. From a safety and deployment perspective in embedded and CPS settings, this behavior is particularly desirable, as it aligns model confidence with actual correctness and enables downstream systems to reason about when additional verification or fallback mechanisms are required.

4.7. In-Depth Failure-Cases Analysis

To better understand the limitations of adaptive self-prompting, we conduct an in-depth failure-case analysis on the two proposed agentic frameworks.

4.7.1. Failure Patterns in A-RAG

First, A-RAG is most likely to fail on rare or weakly represented CWE categories, such as CWE-764, CWE-1260, and CWE-465, where the number of test samples is extremely small. In these cases, retrieval frequently returns broadly related but insufficiently specific guidance, leading to Uncertain False Negatives (UFN) or Uncertain False Positives (UFP) rather than confident misclassifications. Qualitative inspection of retrieval queries shows that the agent correctly identifies a high-level defect class (e.g., pointer or memory-related behavior) but lacks enough contextual signal to decisively confirm the fault.

Second, A-RAG failures often arise in multi-defect or cross-category scenarios, where a function simultaneously exhibits interacting numerical, memory, and control-flow concerns. In such cases, the adaptive query generation mechanism tends to focus on the most salient single uncertainty signal, resulting in a retrieval that addresses only part of the underlying fault. This partial grounding can stabilize confidence without fully resolving ambiguity, particularly when defect dominance depends on execution context or ordering.

Third, A-RAG may underperform when faults depend on implicit execution semantics that are not explicitly represented in CWE or SEI CERT documentation, such as timing-dependent supervision logic or CPS-specific safety dominance assumptions. While retrieval successfully formalizes known vulnerability patterns, it cannot always supply domain-specific operational semantics, leading to conservative uncertainty rather than confident correction.

Importantly, these failures predominantly manifest as low-confidence outcomes, indicating that A-RAG suppresses overconfident misclassification even when retrieval does not fully resolve ambiguity.

4.7.2. Failure Patterns in A-SFT

The most prominent failure mode for A-SFT occurs in underrepresented CWE categories, where limited supervised exposure restricts the model’s ability to internalize robust reasoning patterns. For extremely low-frequency CWEs, the self-evaluation sweep produces few corrected exemplars, constraining the effectiveness of instruction refinement. As a result, misclassifications in these categories are more likely to appear as Confident False Negatives than in A-RAG, reflecting overgeneralization from dominant defect patterns learned during fine-tuning.

A second failure pattern emerges in edge-case arithmetic and boundary-condition logic, where faults hinge on subtle integer promotion rules or architecture-dependent behavior. Although A-SFT internalizes common numeric defect heuristics, it may fail when the defect manifests only under extreme value combinations not sufficiently represented in the training distribution. In these cases, the model’s confidence remains high due to learned reasoning templates, even when the specific scenario violates underlying assumptions.

Finally, A-SFT is vulnerable to conceptual drift introduced during instruction adaptation. While instruction refinement improves aggregate reasoning, qualitative inspection of self-critiques reveals that some revised heuristics prioritize dominant defect signals (e.g., memory safety) at the expense of secondary supervisory or control-flow considerations. This can lead to missed faults when safety mechanisms exist but are rendered ineffective by ordering or dominance violations.

4.7.3. Comparative Risk Profile

Overall, the failure cases of A-RAG and A-SFT reflect a trade-off between adaptability and internalization. A-RAG is more likely to fail conservatively in rare or CPS-specific scenarios, producing uncertainty rather than confident errors when external knowledge is insufficient. A-SFT, in contrast, achieves more decisive inference but is more susceptible to confident errors in sparsely represented or highly nuanced defect categories.

From a deployment perspective in embedded and CPS environments, these findings suggest that A-RAG is better suited for settings where uncertainty-aware decision-making and on-demand knowledge grounding is prioritized, while A-SFT is preferable when efficient single-pass inference is required and sufficient domain coverage exists during training. Together, the observed failure patterns highlight that adaptive self-prompting improves both accuracy and risk calibration, but does not eliminate the need for complementary safeguards in safety-critical analysis pipelines.

4.7.4. Failure Modes Reduced by Adaptive Self-Prompting

While the preceding analysis highlights residual limitations, the empirical results and qualitative inspection jointly indicate that adaptive self-prompting substantially reduces several high-risk failure modes that are prevalent in non-agentic and weakly agentic baselines with static prompting. These mitigations are consistent across both A-RAG and A-SFT, although through different mechanisms.

The most significant mitigation achieved by adaptive self-prompting is the reduction of overconfident misclassification, particularly Confident False Positives (CFP) and Confident False Negatives (CFN). Static-prompt and single-pass models frequently commit to incorrect decisions when reasoning relies on incomplete evidence, such as ambiguous pointer lifetimes, partial initialization, or implicit arithmetic constraints. In contrast, confidence-triggered reflection externalizes uncertainty as an explicit control signal. In A-RAG, this mechanism selectively invokes retrieval, preventing premature commitment when internal reasoning is insufficient. In A-SFT, repeated confident errors are identified during self-evaluation and corrected through instruction refinement and targeted fine-tuning, leading to a markedly lower incidence of high-confidence failures at inference time.

A common failure mode in non-agentic models is the omission of latent constraints that are not syntactically explicit, including integer range assumptions, memory ownership rules, or synchronization requirements. These omissions are particularly common in Toyota ITC defects and in real-world Big-Vul samples where correctness depends on semantic invariants rather than surface patterns. Adaptive self-prompting mitigates this failure class by explicitly reintroducing missing constraints into the reasoning process. A-RAG retrieves canonical constraint formulations from CWE and SEI CERT documentation when uncertainty is detected, grounding the decision in established safety principles. A-SFT internalizes these constraints by encoding them into refined system instructions and corrected exemplars, reducing repeated omission across similar samples.

Static and weakly agentic baselines often overgeneralize from local syntactic cues, such as the presence of pointer arithmetic or loop constructs, resulting in false positives when safe idioms resemble known defect patterns. Adaptive self-prompting reduces this tendency by forcing justification beyond surface similarity. In A-RAG, retrieval emphasizes the semantic conditions under which a pattern is actually unsafe, discouraging shallow matches. In A-SFT, instruction adaptation explicitly encodes distinctions between benign and faulty variants, weakening brittle heuristics learned during initial fine-tuning.

Ambiguous and borderline cases, such as defensive checks, partial safety mechanisms, or execution-order-dependent logic, often lead to unstable predictions in static models, oscillating between confident correctness and confident error. Adaptive self-prompting improves calibration by aligning confidence with evidential support. Rather than forcing a decisive classification, the agent differentiates between well-supported conclusions and underdetermined cases. As reflected in the confidence-aware error taxonomy, many potential confident errors are converted into either correct outcomes or explicitly uncertain predictions, improving the system’s operational safety profile.

Conventional supervised fine-tuning risks reinforcing early misclassifications and spurious correlations, especially for dominant CWE categories. The A-SFT framework mitigates this failure mode by explicitly mining low-confidence and self-revised cases and reintroducing them as corrected training exemplars with updated reasoning policies. This process systematically shifts learning focus toward unstable decision regions, reducing the persistence of entrenched errors across epochs.

4.8. CPS Case Study

The selected scenario is motivated by publicly documented investigations into unintended acceleration incidents in Toyota vehicles, which prompted extensive safety reviews of automotive electronic throttle control systems. In particular, the technical assessment conducted by NASA at the request of the U.S. National Highway Traffic Safety Administration examined the software, hardware, and systems engineering processes associated with Toyota’s electronic throttle control (ETC) system [50].

While the NASA investigation did not identify a software defect as the root cause of large unintended acceleration events, it emphasized the necessity of rigorous software analysis in safety-critical control systems. The assessment included a detailed examination of the ETC embedded software architecture using static analysis tools, model checking, and code review to evaluate task interactions, data integrity, boundary enforcement, and fail-safe mechanisms [50]. The report examined potential risks, including task interference, data corruption, boundary enforcement failures, and dual-fault scenarios, to assess whether such mechanisms could plausibly produce unintended acceleration. These risks make automotive throttle control a representative and high-impact CPS case for evaluating LLM-based static fault detection frameworks.

4.8.1. Synthetic Implementation of the Scenario

To construct a controlled yet realistic evaluation scenario, we derived a synthetic embedded C implementation of a simplified electronic throttle control loop. The design reflects common structural elements of automotive electronic throttle control systems described in safety assessments of embedded vehicle software [50]:

• The system performs periodic acquisition of accelerator pedal position sensor data.
• The throttle command is computed based on the current system state and processed sensor inputs.
• A supervisory safety mechanism models brake-throttle interaction, consistent with brake override strategies evaluated in the assessments.
• Shared global state variables are accessed and modified by multiple concurrent tasks within the control architecture.

The resulting artifact, as shown in Algorithm 4, is not intended to replicate proprietary Toyota software, but rather to emulate structurally plausible embedded control logic consistent with the CPS context described in [50].

Algorithm 4 Simplified Electronic Throttle Control (Fault-Injected)

1:  // Includes and Definitions 2:  #include <stdint.h> 3:  #include <stdbool.h> 4:  #define MAX_SAMPLES 8 5:  #define THROTTLE_MAX 100 6:  #define BRAKE_THRESHOLD 50   7:  // Global Shared State (Concurrency Risk) 8:  volatile uint16_t pedal_samples[MAX_SAMPLES]; 9:  volatile uint8_t sample_index = 0; 10: volatile uint16_t throttle_command = 0; 11: volatile bool brake_pressed = false;   12: // Simulated Sensor Read 13: uint16_t read_pedal_sensor(void) { 14:     return pedal_samples[sample_index++];                ▹ Fault 1 15: }   16: // Brake Task (Concurrent) 17: void brake_task(void) { 18:     if (brake_pressed) { 19:         throttle_command = 0; 20:     } 21: }   22: // Compute Throttle Average 23: uint16_t compute_average(void) { 24:     uint32_t sum = 0; 25:     for (int i = 0; i <= MAX_SAMPLES; i++) {            ▹ Fault 2 26:         sum += pedal_samples[i]; 27:     } 28:     return (uint16_t)(sum / MAX_SAMPLES); 29: }   30: // Main Control Loop 31: void throttle_control_task(void) { 32:     uint16_t pedal = read_pedal_sensor(); 33:     if (pedal > 900) {                        ▹ Fault 3 34:         throttle_command = THROTTLE_MAX; 35:     } else { 36:         throttle_command = pedal / 10; 37:     } 38:     if (!brake_pressed) {                       ▹ Fault 4 39:         throttle_command += 5; 40:     } 41:     if (throttle_command > THROTTLE_MAX) { 42:         throttle_command = THROTTLE_MAX; 43:     } 44: }

4.8.2. Injected Faults

The synthetic CPS artifact incorporates four injected faults spanning memory safety, input validation, numerical handling, and concurrency defects. Each fault was selected to align with CWE categories represented in our unified dataset and reflect safety-relevant concerns in embedded automotive control systems discussed in [50]. The injected faults do not reflect defects identified in Toyota’s ETC implementation. Rather, they represent structurally plausible embedded-system vulnerabilities consistent with the categories analyzed in safety assessments. The four faults are:

1.

Improper Restriction of Operations within the Bounds of a Memory Buffer: The post-increment operation in read_pedal_sensor() allows sample_index to exceed the valid index range of the pedal_samples buffer, resulting in potential out-of-bounds memory access. This behavior aligns with CWE-131 (Incorrect Calculation of Buffer Size), which in the Toyota ITC benchmark is grouped under dynamic memory defects. Although the access occurs beyond the intended bounds rather than strictly before the buffer, it represents a boundary enforcement failure consistent with memory safety violations in embedded systems. In long-running control loops, such index mismanagement may lead to latent memory corruption affecting actuator commands [50].

2.

Out-of-Bounds Array Access (Off-by-One Error): The loop condition i <= MAX_SAMPLES introduces an off-by-one error, causing the loop to access one element beyond the allocated array boundary. This condition corresponds to CWE-131 (Incorrect Calculation of Buffer Size) as represented in the Toyota ITC benchmark taxonomy. Improper boundary enforcement in embedded control software can produce undefined behavior and unpredictable actuator outputs, representing a critical safety concern in automotive CPS contexts [50].

3.

Improper Handling of Exceptional Sensor Conditions: The condition pedal > 900 bypasses comprehensive range validation and directly assigns maximum throttle output without robust sanity checking or fault mitigation. This pattern corresponds to CWE-703 (Improper Check or Handling of Exceptional Conditions), which is included in the inappropriate code category of the Toyota ITC benchmark. Failure to defensively validate anomalous or corrupted sensor readings may compromise safety safeguards and underscores the importance of resilient input validation in safety-critical control loops [50].

4.

Concurrent Execution with Improper Synchronization: The shared brake-throttle state variables are accessed and modified by multiple execution contexts without synchronization primitives such as mutexes, atomic operations, or interrupt masking. This behavior aligns with CWE-362 (Concurrent Execution using Shared Resource with Improper Synchronization), categorized under concurrency defects in the Toyota ITC benchmark. In embedded automotive CPS systems, improper synchronization may introduce race conditions that result in inconsistent actuator commands and safety-critical control instability [50].

Collectively, these faults are embedded within a brake-throttle interaction context that introduces system-level safety semantics beyond isolated vulnerability patterns. This structure allows the case study to evaluate whether the proposed frameworks merely detect syntactic defect signatures or demonstrate context-aware reasoning about safety-critical control interactions consistent with embedded automotive CPS constraints [50].

4.8.3. A-RAG Output Trace and Analysis

To illustrate how inference-time adaptive self-prompting manifests in a safety-critical embedded setting, we present the full A-RAG reasoning trace on the synthetic ETC example (Algorithm 4). The trace highlights how confidence-based reflection governs retrieval, how retrieved knowledge reshapes reasoning, and how uncertainty is calibrated rather than suppressed.

The initial reasoning correctly identifies two clear boundary violations and a plausible race condition. At the same time, the agent explicitly acknowledges missing architectural details, such as scheduler semantics, interrupt behavior, or index reset logic. This self-identified uncertainty is central to the A-RAG design. Rather than committing confidently under incomplete evidence, the agent quantifies its own epistemic limits.

Because $\$\mathtt{CONFIDENCE}\_\mathtt{SCORE}=0.68<\tau =0.75$, the A-RAG agent invokes adaptive retrieval.

The generated query focuses on array-bound violations and shared-state races, which represent the most salient uncertainty signals from the initial reasoning. Notably, the query does not explicitly enumerate all potential defect categories embedded in the snippet. For example, the threshold condition if (pedal > 900) and the additive bias throttle_command += 5 could relate to improper exceptional-condition handling (e.g., CWE-703), yet these are not strongly emphasized. This reflects a prioritization strategy as the agent concentrates retrieval on dominant risk signals rather than expanding the search space indiscriminately.

The retrieved knowledge is broadly relevant but not perfectly specific to CPS control semantics. While the boundary and synchronization guidance directly reinforce the identified patterns, some entries (e.g., CWE-398) provide only generic quality indicators. The vector search surfaces canonical memory and concurrency principles, but does not retrieve highly specialized CPS supervisory reasoning patterns, such as formal brake-dominance guarantees or timing-aware safety interlocks. This illustrates a practical limitation where retrieval can ground reasoning in standardized defect taxonomies, but it does not automatically inject domain-specific operational semantics beyond what is encoded in the knowledge base.

The second-stage reasoning becomes more standards-aligned and explicitly references recognized CWE and SEI guidance. Retrieval clarifies the formal categorization of the boundary and synchronization defects, yet it does not fully resolve contextual ambiguity. In particular, the interaction between the additive throttle offset and brake semantics under all possible interleavings is not exhaustively analyzed.

Confidence increases from 0.68 to 0.79, reflecting that retrieval successfully sharpens analysis of boundary and synchronization concerns, while residual ambiguity remains regarding execution context and safety impact. The increase is meaningful but not dramatic, indicating that the agent incorporates external guidance without artificially inflating certainty.

From an architectural perspective, this behavior is consistent with the intended operation of A-RAG. The agent performs independent reasoning, quantifies uncertainty, and conditionally invokes retrieval shaped by its own reasoning trace. Retrieval is neither unconditional nor purely keyword-driven. It is dynamically constructed from identified uncertainty signals. Although the mechanism does not guarantee exhaustive defect enumeration, it systematically strengthens standards-grounded reasoning and suppresses premature high-confidence commitment.

In the CPS embedded domain, where partial visibility into scheduling, interrupts, and hardware semantics is common, this uncertainty-aware and selectively knowledge-grounded behavior is particularly relevant. The trace demonstrates that A-RAG meets the research objective of adaptive self-prompting at inference time. It enhances contextual grounding and calibration without resorting to blanket retrieval or static prompting. Rather than merely improving classification outcomes, it takes advantage of self-regulated reasoning that aligns model confidence with evidential support, an important property for safety-critical analysis in embedded control systems.

4.8.4. A-SFT Output Trace and Analysis

To contrast inference-time adaptive retrieval with training-time adaptive self-prompting, we present the output trace of the fine-tuned A-SFT model on the synthetic ETC example (Algorithm 4). Unlike A-RAG, A-SFT operates in a single forward pass at deployment. All adaptive behavior has already been internalized during training through self-evaluation, critique aggregation, and instruction refinement.

The A-SFT model immediately classifies the snippet as faulty. Compared to the A-RAG initial trace, the reasoning is more structured and explicitly fault-oriented, reflecting instruction refinement during training. The model correctly identifies the two boundary violations and the shared-state concurrency risk. However, its treatment of the exceptional sensor condition is more tentative and less formally categorized than in A-RAG’s retrieval-grounded reasoning.

From a behavioral perspective, this trace illustrates several impacts of training-time adaptive self-prompting.

First, the model demonstrates internalized boundary-check heuristics. The off-by-one loop condition is detected decisively and framed as a canonical array traversal defect. This reflects exposure to repeated similar patterns during supervised fine-tuning, where corrected exemplars reinforced the distinction between < and <= conditions.

Second, the concurrency reasoning is more assertive than the A-RAG initial pass. The fine-tuned instruction emphasizes that volatile does not imply thread safety. This language closely mirrors phrasing that typically appears in self-critiques during training, suggesting that the instruction-adaptation phase has encoded synchronization heuristics into the model’s decision policy.

However, limitations are also visible. The model does not explicitly categorize defects by CWE ID, nor does it ground the analysis in formal SEI guidance. Unlike A-RAG, which retrieved CWE-362 and CON35-C to refine its reasoning, A-SFT relies entirely on internalized representations. As a result, its concurrency reasoning remains high-level and assumes concurrent execution without verifying scheduling semantics.

Additionally, the exceptional-condition handling fault (the pedal > 900 logic and additive throttle_command += 5) is under-analyzed. The model acknowledges that the threshold may be problematic, but does not frame it as improper handling of exceptional conditions. This reflects a known A-SFT tendency to prioritize dominant memory and synchronization defect patterns over supervisory-control semantics when multiple fault classes coexist.

We ran the confidence estimation prompt for analysis, and it was relatively high confidence (0.88), which also reflects a characteristic trade-off of A-SFT. Because adaptive self-prompting occurred during training rather than inference, uncertainty signals are no longer exposed at deployment. The model internalizes decision rules and produces a decisive single-pass output. While this improves efficiency and reduces uncertain predictions, it can increase the risk of confident oversimplification in edge-case CPS scenarios.

In contrast to A-RAG, which increased confidence modestly after retrieval (0.68 to 0.79), A-SFT begins and ends with a high confidence score. This indicates that the model has internalized defect heuristics strongly enough to avoid hesitation. However, the absence of a second-stage reasoning step means that domain-specific CPS semantics, such as brake-dominance guarantees under all interleavings, are not exhaustively examined.

From an architectural standpoint, this trace reflects the core distinction between the two frameworks:

• A-RAG externalizes uncertainty and dynamically acquires knowledge.
• A-SFT internalizes prior uncertainty into refined parameters and instructions.

In this ETC case, A-SFT successfully detects the primary memory and concurrency defects and produces a confident, correct classification. However, its reasoning is less taxonomically grounded and less explicitly calibrated in its explanation, because it does not expose the inference-time uncertainty-control loop that A-RAG shows. The model demonstrates strong pattern recognition but limited context-sensitive expansion beyond the most salient defect categories.

For embedded CPS deployment, this behavior has important implications. A-SFT provides deterministic, low-latency inference suitable for resource-constrained environments. Yet, unlike A-RAG, it does not expose internal uncertainty signals nor adapt reasoning depth dynamically. In complex CPS scenarios where execution semantics, timing assumptions, or supervisory dominance rules are partially visible, this difference may influence how each framework balances decisiveness and caution.

Overall, the trace demonstrates that A-SFT achieves the research objective of training-time adaptive self-prompting. It internalizes corrected reasoning patterns and produces high-confidence, structured fault detection in a single pass. However, it does so by trading inference-time adaptability for deployment efficiency, and it may underemphasize nuanced CPS supervisory semantics when dominant defect patterns are already sufficient to trigger a faulty classification.

5. Limitations and Threats to Validity

This section discusses the limitations of the study and the principal threats to validity that may affect the interpretation and generalization of the results. We distinguish between internal validity, which concerns the soundness of the experimental design and evaluation methodology, and external validity, which concerns the extent to which the findings generalize beyond the studied setting.

5.1. Methodological Limitations

Several limitations arise from the experimental design and scope of the evaluation.

First, the final dataset consists of 4034 function-level samples, partitioned into 2814 training instances and 1220 test instances using a single 70/30 train-test split. While this split maximizes test-set coverage, the absence of a dedicated validation split or mechanisms (k-fold cross-validation) limits independent hyperparameter tuning and restricts the ability to assess robustness across alternative data partitions. Consequently, the reported results may be sensitive to the particular split used.

Second, although the dataset is balanced at the binary classification level (faulty vs. non-faulty), it exhibits a pronounced long-tailed distribution across vulnerability types. A small subset of CWE categories (e.g., CWE-416, CWE-189, CWE-190, CWE-476, and CWE-362) accounts for the majority of samples, while many CWEs appear only a few times. As a result, overall performance improvements are driven primarily by frequent vulnerability classes, and per-CWE metrics for rare vulnerabilities remain statistically unreliable.

Third, the dataset exhibits a source imbalance between Big-Vul and Toyota ITC. The training split contains 1932 Big-Vul samples and 860 Toyota ITC samples, with a similar skew in the test set. Despite stratified splitting, learned representations may be biased toward vulnerability patterns prevalent in Big-Vul. Moreover, the study does not evaluate cross-dataset generalization (e.g., training on one source and testing on the other), limiting conclusions about robustness under dataset shift.

Fourth, the evaluation focuses exclusively on C-language, function-level fault detection. Although each sample retains local supporting context such as helper functions, globals, typedefs, and macros, the study does not model full-system behavior, including broader interprocedural dependencies, build configurations, hardware integration effects, or multi-file execution environments. Accordingly, the findings should be interpreted as evidence of effectiveness for function-level fault detection rather than as a complete assessment of performance in fully integrated software systems.

Finally, while the paper reports inference runtime and peak memory usage, computational efficiency is not systematically explored as a primary objective. Training cost, scalability to larger models or datasets, and deployment constraints in resource-limited environments remain outside the scope of this work.

5.2. Threats to Internal Validity

Threats to internal validity primarily stem from evaluation methodology and model behavior.

First, confidence estimation plays a central role in both agentic frameworks, particularly in triggering retrieval in A-RAG and selecting corrective examples in A-SFT. Confidence scores are derived from model-generated probabilities, which may be imperfectly calibrated. Miscalibration could lead to unnecessary retrieval, missed opportunities for correction, or biased self-evaluation, thereby affecting measured performance.

Second, in the A-SFT framework, self-evaluation and instruction refinement rely on the model’s own critiques. While this design is intentional, it introduces the risk of reinforcing systematic reasoning biases or incomplete heuristics if the self-critiques are themselves flawed. Although empirical results indicate performance gains, the absence of an external oracle for critique quality represents a potential internal validity threat.

Third, although paired statistical significance testing is conducted using the McNemar test on the held-out test set, the analysis is limited to a single train-test split. As a result, the reported p-values quantify statistical significance with respect to this fixed test sample, but do not capture variability across alternative dataset partitions or resampled test sets. Confidence intervals and resampling-based analyses (e.g., bootstrap or repeated cross-validation) could provide a more comprehensive characterization of performance stability and remain an avenue for future work.

Finally, key agentic design choices were fixed and not systematically analyzed. In particular, the confidence threshold $\tau$ was not tuned on a dedicated calibration or validation split, and additional components such as retrieval depth and top-k selection, confidence estimation mechanisms, and prompt or instruction adaptation strategies were not independently ablated. Although this design keeps the evaluation protocol simple and uniform across models, it does not establish that $\tau =0.75$ (or other design parameters) is optimal for predictive performance or computational efficiency.

5.3. Threats to External Validity

Several factors limit the generalizability of the findings beyond the studied setting.

First, the evaluation is grounded in a combined dataset derived from Toyota ITC and a taxonomy-aligned subset of Big-Vul. While this design balances synthetic benchmark defects with real-world vulnerabilities, it does not capture the full diversity of embedded and cyber-physical software found in industrial systems, such as real-time operating systems, interrupt-driven firmware, or hardware-specific driver code.

Second, the retrieval knowledge base used in A-RAG is restricted to CWE and SEI CERT C documentation to prevent data leakage. While this choice ensures experimental rigor, it may not reflect real-world deployments where agents could access richer or domain-specific documentation. As a result, observed gains may differ when alternative knowledge sources are used.

Third, all experiments are conducted using a single base model (Llama 3.1 8B) and a fixed agent architecture. Although the proposed mechanisms are model-agnostic in principle, the results may not directly transfer to smaller models, larger frontier models, or encoder-based architectures without additional validation.

Finally, the study evaluates fault detection performance in isolation. Downstream integration into industrial workflows, human-in-the-loop review, or safety certification pipelines is not assessed, which limits conclusions about operational impact in real-world CPS environments.

6. Conclusions

In this paper, we introduced adaptive self-prompting as a foundational mechanism for agentic large language model (LLM) frameworks for function-level C fault detection in embedded and cyber-physical systems. Rather than treating prompting, retrieval, or fine-tuning as static design choices, our work reframes them as adaptive control processes governed by internally generated uncertainty signals. Within a unified agent architecture, we proposed two complementary instantiations of this idea. Agentic Retrieval-Augmented Generation (A-RAG) realizes adaptive self-prompting at inference time through confidence-based reflection that selectively invokes CWE and SEI CERT knowledge using reasoning-conditioned retrieval queries. Agentic Supervised Fine-Tuning (A-SFT) shifts adaptive self-prompting entirely to training time via an iterative self-evaluation sweep that converts low-confidence or incorrect predictions into corrected exemplars and evolving system instructions.

To balance embedded and CPS relevance with real-world diversity, we evaluated both approaches on a combined dataset curated from the Toyota ITC benchmark and a taxonomy-aligned subset of Big-Vul. Experimental results demonstrate that agentic control, rather than retrieval or fine-tuning alone, is the primary driver of performance gains. A-RAG consistently outperforms static RAG by invoking external knowledge using adaptive queries only when warranted and aligning retrieved evidence with the specific code behaviors responsible for uncertainty. A-SFT, in contrast, internalizes adaptive self-prompting, reflection, and error-correction during training, eliminating the need for retrieval or agentic control flow at inference time. Once fine-tuned, the model performs fault detection using a single forward pass with fixed instructions, achieving inference efficiency comparable to the base LLM while substantially improving accuracy and aligning model-reported confidence more closely with observed correctness, as reflected in confidence-aware error analysis.

These findings should be interpreted in light of several experimental constraints. Results are based on a single stratified train-test split, a fixed confidence threshold for reflection, and a single backbone model configuration. As such, reported gains characterize comparative trends rather than absolute performance guarantees across architectures, datasets, or deployment environments. Moreover, while A-SFT is particularly well-suited for embedded and cyber-physical systems where predictable latency, minimal runtime complexity, and sufficiently represented fault patterns are critical, A-RAG remains a flexible alternative in scenarios involving rare or evolving vulnerability classes, where external knowledge grounding can compensate for limited supervised exposure.

Several directions for future work emerge from this study. Evaluation rigor can be strengthened through the introduction of validation splits for hyperparameter selection, repeated stratified resampling, and broader statistical testing to better quantify robustness and variance across runs. The long-tailed distribution of CWE categories motivates targeted investigation of rare vulnerability types, including reweighting strategies, data augmentation, and cross-dataset transfer experiments to assess generalization under data sparsity. In addition, more granular ablation studies are needed to isolate the sensitivity of agentic behavior to individual design choices, such as the confidence threshold $\tau$, retrieval depth, and top-k selection, confidence estimation mechanisms, and prompt or instruction adaptation strategies. Practical considerations related to computational cost, training efficiency, and deployment-time constraints also warrant systematic analysis. Finally, the frameworks need to be extended beyond fault classification toward actionable vulnerability remediation. By leveraging the same mechanisms for self-evaluation, knowledge grounding, and instruction adaptation explored in this work, future agents could generate patch candidates or repair guidance grounded in CWE taxonomies and SEI CERT principles, further advancing the applicability of agentic LLMs to end-to-end secure software development workflows.