Securing Locally Deployed LLM-RAG Assistants: A Cross-Model and Cross-Lingual Evaluation of Indirect Prompt Injection Defenses

Abstract

Retrieval-Augmented Generation (RAG) exposes locally deployed Large Language Model (LLM) assistants to indirect prompt injection through untrusted documents. This study evaluates four attack families—instruction override, context contamination, tool abuse, and simulated data exfiltration—across five defense configurations (C1–C5) and four local LLMs, using a synthetic corpus of 40 documents and 32 queries. A complementary Spanish–English 2 × 2 experiment with a common multilingual retriever examines cross-lingual attack behavior. In the original Llama 3.1 8B experiment, conditional Attack Success Rate (ASR) decreased from 70.4% without defenses (C1) to 22.2% with deterministic post-generation filtering (C5). Across the four models, C5 achieved post-filter ASRs of 7.4–22.2%, with no observed consequential tool misuse or concrete synthetic-data leakage, although Answer Utility ranged from 21.9% to 62.5%. Across the four Spanish–English language conditions, C5 post-filter ASR ranged from 14.3% to 40.7%; in the English–English condition, the Spanish-calibrated filter did not remove English override phrases or disclosure structures, and C5 did not improve on the model-enforced defense C4. Because the filter and automatic evaluator share some detection indicators, post-filter ASR reflects unsafe content remaining in delivered responses rather than intrinsic model resistance. All automatic labels were validated through adjudication, bilingual re-labeling, and a blind semantic review of pre- and post-filter responses. Context contamination remained the main residual weakness. These findings support independent security enforcement while highlighting retrieval effects, security–utility trade-offs, and limitations of the controlled synthetic setting.

1. Introduction

Large Language Models (LLMs) are increasingly moving beyond cloud-only deployments toward local, on-device, and edge-assisted execution. This evolution is motivated by requirements such as reduced communication latency, greater data locality, improved privacy, resilience to cloud connectivity limitations, and lower dependence on centralized services. At the same time, deploying LLM-based services closer to data sources introduces significant computational and architectural challenges, particularly in resource-constrained edge and Internet of Things (IoT) environments. Recent surveys [1,2] consequently identify efficient local inference, edge–cloud collaboration, and trustworthy operation as central research directions for next-generation LLM-enabled edge intelligence.

A particularly relevant architecture for local LLM applications is Retrieval-Augmented Generation (RAG). Rather than relying exclusively on knowledge encoded in model parameters, RAG systems retrieve external information at inference time and incorporate the retrieved content into the generation context. This approach can improve access to domain-specific and up-to-date information while allowing the underlying model to remain unchanged [3]. However, the introduction of external knowledge also creates additional trust boundaries. Retrieved documents may be incomplete, unreliable, manipulated, or deliberately malicious, and the generator must process them together with legitimate system and user instructions. Recent work on trustworthy RAG [4] therefore identifies security, privacy, robustness, reliability, and accountability as fundamental concerns alongside answer quality.

One of the most important security threats in this setting is prompt injection. In a direct prompt-injection attack, adversarial instructions are supplied explicitly through the user’s interaction with the model. In an indirect prompt injection, however, malicious instructions are embedded in external content that the LLM subsequently processes as part of a legitimate task. Greshake et al. [5] demonstrated that this distinction enables attackers without direct access to the target model to influence LLM-integrated applications through data likely to be retrieved or consumed by the application. More recent experimental evidence [6] confirms that indirect injection remains effective against contemporary LLMs and that attack success can vary significantly across models and document characteristics. OWASP consequently identifies prompt injection as LLM01:2025 [7], emphasizing that the integration of RAG, external data, and tools does not by itself eliminate this vulnerability.

The problem becomes more critical when the LLM is not limited to generating text but is connected to tools, services, or automated workflows. Malicious retrieved content may then attempt to induce unauthorized actions, access internal information, or influence tool execution. AgentDojo [8], for example, demonstrates that agents operating on untrusted external data can be manipulated into executing attacker-selected tasks through prompt injection. Similarly, PoisonedRAG [9] shows that the knowledge base underlying a RAG system constitutes an attack surface in its own right: strategically inserted malicious documents can alter answers to selected queries even when the user interaction itself is benign. These findings are particularly relevant to edge-intelligence scenarios in which local AI assistants may eventually mediate access to device information, operational services, or cyber–physical processes.

Several defense strategies have been proposed. A common family of approaches attempts to make the distinction between trusted instructions and untrusted data more explicit. Spotlighting [10], for example, encodes provenance information into external content so that the LLM can more reliably distinguish data from commands. StruQ [11] pursues a stronger separation by introducing structured queries that place instructions and data in distinct channels and train the model to follow instructions only from the trusted portion. Other approaches rely on prompt hardening, content filtering, constrained tool permissions, output validation, or combinations of multiple safeguards. Nevertheless, existing evaluations [6,8,11] indicate that no single mitigation can be assumed to provide universal protection, and defenses that appear effective for one model or workload may not transfer unchanged to another.

A further dimension that remains insufficiently understood is language dependence. LLMs increasingly operate in multilingual environments, yet their safety behavior is not necessarily invariant across languages. Recent large-scale evaluation of adversarial prompts in Turkish and English found substantial differences between models and between language conditions, indicating that security properties established in one language cannot automatically be extrapolated to another [12]. This issue is especially relevant for RAG systems, where the language of the user’s query and the language of retrieved external content may differ. An assistant operating primarily in one language may therefore retrieve and process malicious instructions expressed in another, creating a cross-lingual attack path that is difficult to capture through monolingual evaluation alone.

Despite the rapid growth of research on prompt injection, the literature reviewed above leaves comparatively limited empirical evidence on the transferability of indirect prompt-injection defenses across both model families and languages under a common local RAG architecture. Existing studies have established the feasibility of indirect injection, demonstrated attacks against RAG knowledge bases and tool-using agents, and proposed several defensive mechanisms. However, fewer studies isolate whether a defense that performs well with one local LLM continues to behave similarly when the underlying model changes, or whether its effectiveness is preserved when the query and malicious document are expressed in different languages. Addressing this gap is important for trustworthy edge intelligence, since practical local deployments are likely to use heterogeneous model families and operate in multilingual information environments.

Accordingly, this study investigates four research questions. RQ1 examines to what extent susceptibility to indirect prompt injection varies across different locally deployed LLM families. RQ2 examines whether different defense configurations, ranging from prompt-level safeguards to deterministic post-generation filtering, exhibit consistent security effectiveness across the evaluated model families. RQ3 investigates whether attack effectiveness changes when the language of the user query and the malicious retrieved document differ, while controlling the retrieval mechanism. Finally, RQ4 examines the trade-off between security and legitimate task utility, since a defense that prevents attacks only by refusing benign requests would provide limited practical value.

To address these questions, we develop a fully local and reproducible RAG-based experimental framework in which synthetic documents may contain malicious instructions targeting four attack classes: instruction override, context contamination, unauthorized tool use, and simulated data exfiltration. Five progressively modified configurations are evaluated, ranging from an undefended RAG baseline to configurations incorporating explicit trust separation, tool-use policies, response validation, and deterministic output filtering. The framework records not only Attack Success Rate (ASR), but also Tool Misuse Rate (TMR), Excessive Agency Rate (EAR), Leakage Rate (LR), and Answer Utility (AU), enabling security improvements to be considered jointly with their operational cost.

The experimental study extends an initial Llama 3.1 8B implementation to three additional local model families (Llama 3.2 3B, Gemma 3 4B, and Qwen3 4B) while preserving the corpus, queries, attack scenarios, retrieval parameters, and defense configurations. A second experiment introduces a controlled 2 × 2 cross-lingual design combining Spanish and English user queries with Spanish and English malicious documents. To isolate language effects from retrieval effects, the same multilingual embedding-based retriever is used across all four linguistic conditions. This design makes it possible to study not merely whether prompt injection is possible in multiple languages, but whether security behavior and defense effectiveness transfer across model and language boundaries.

The principal contribution of this work is therefore not a new prompt-injection technique, but a controlled evaluation of the robustness and transferability of defenses in local LLM-RAG systems. The study contributes a reproducible attack-and-defense testbed, a cross-family evaluation of four local LLMs, a controlled cross-lingual methodology, and a multidimensional security–utility assessment. From an edge-intelligence perspective, the work further highlights that local execution and data locality do not themselves guarantee trustworthy operation: when locally deployed LLMs ingest externally sourced information or mediate access to tools, explicit security controls remain necessary. The findings provide the experimental motivation for future model-agnostic runtime enforcement mechanisms capable of separating untrusted information from trusted instructions and controlling consequential actions independently of the generative model. Although motivated by edge-intelligence scenarios, the study evaluates a locally deployed architecture on a single local host; it does not assess latency, memory, energy consumption or performance on resource-constrained edge hardware.

The remainder of this paper is organized as follows. Section 2 reviews related work on local and edge LLM deployment, RAG security, indirect prompt injection, defensive mechanisms, and multilingual robustness. Section 3 defines the system architecture, threat model, attack families, defense configurations, and evaluation metrics. Section 4 presents the experimental methodology, including the evaluated models and the controlled cross-lingual design. Section 5 reports the experimental results and discusses their implications for model dependence, multilingual security, defense transferability, and trustworthy edge intelligence. Finally, Section 6 summarizes the conclusions, limitations, and future research directions.

2. Background and Related Work

2.1. Local LLMs, Retrieval-Augmented Generation, and Edge Intelligence

LLMs are increasingly moving from centralized cloud platforms toward local and edge-assisted execution. Advances in model compression, quantization, and efficient inference have enabled smaller models to support privacy-sensitive, low-latency, and resilient applications closer to the data source [1,2]. In parallel, LLMs are being considered as reasoning and interaction components within future IoT and edge-intelligence architectures [13].

RAG is particularly attractive in this context because it allows locally deployed models to access external, domain-specific knowledge without retraining [3]. This capability can reduce dependence on model-internal knowledge while preserving local control over inference and data. However, RAG also enlarges the trusted computing base: external documents, retrievers, embeddings, and downstream tools become part of the security boundary. Recent work [4,14] on trustworthy RAG and edge-oriented cybersecurity has therefore identified knowledge-base poisoning, indirect prompt injection, and information leakage as relevant threats.

Accordingly, local execution should not be equated with trustworthy execution. When external information is dynamically incorporated into the model context, the provenance and trustworthiness of retrieved content become security-critical.

2.2. Indirect Prompt Injection and RAG Security

Prompt injection exploits the difficulty LLMs have in distinguishing trusted instructions from untrusted natural-language content [5,15]. In indirect prompt injection [5], malicious instructions are not provided by the user but are embedded in external content subsequently retrieved or processed by the application. A benign query can therefore expose the model to attacker-controlled instructions capable of altering responses, inducing unauthorized actions, or triggering information disclosure.

RAG systems are especially exposed because their knowledge base is itself part of the attack surface. PoisonedRAG [9] demonstrated that strategically inserted documents can manipulate targeted outputs by exploiting both retrieval and generation. Related work [16] has shown that malicious information stored in persistent or retrievable memory can likewise influence later reasoning and actions.

The consequences become more severe when LLMs are connected to tools. AgentDojo [8] showed that indirect injection can manipulate agents processing untrusted external data into executing attacker-selected tasks. Recent reviews of agent security [17] similarly identify over-privileged tool access, unsafe memory, and insufficient provenance controls as recurring weaknesses. Therefore, evaluating RAG security requires more than measuring answer correctness: tool misuse, excessive agency, information leakage, refusal behavior, and legitimate answer utility should also be considered.

2.3. Defense Strategies

Defenses against prompt injection can be broadly grouped according to where they intervene in the LLM pipeline [18,19,20]. Prompt-level defenses attempt to make the distinction between trusted instructions and untrusted content explicit. Spotlighting [10] marks external content to improve provenance awareness, while StruQ [11] introduces stronger structural separation between instructions and data. Although attractive because of their simplicity, these approaches still depend on the model itself correctly applying the defensive instruction. Their effectiveness may therefore vary across models, prompts, languages, and task contexts [18,19,20].

Other strategies introduce controls outside the generative decision process, including input filtering, response validation, restricted tool permissions, and deterministic post-generation checks. In tool-using systems, this distinction is particularly important: an LLM may propose an action, but a separate authorization mechanism can determine whether that action is permitted. Current literature [17,20] therefore increasingly favors defense-in-depth approaches combining provenance, instruction–data separation, validation, tool restrictions, and runtime controls. In this taxonomy, configuration C3 (Section 3.4) implements prompt-level instruction–data separation in the spirit of Spotlighting [10] (StruQ [11] pursues the same principle but requires fine-tuning), C4 adds a model-enforced tool policy with self-validation, and C5 implements an output guardrail enforced deterministically outside the model. Retrieval filtering, provenance-based defenses, capability-based tool authorization and privilege-separation architectures such as CaMeL [21] require architectural changes or training beyond the scope of this comparative study; they are discussed as complements in Section 5.6.

This motivates the comparative experimental design adopted in this study, in which alternative security configurations are evaluated under a common RAG architecture. Rather than assuming that additional safeguards necessarily improve protection, the evaluation examines how prompt-level defenses, tool-use policies, and deterministic post-generation filtering affect attack success and legitimate task utility across different models and experimental conditions.

2.4. Cross-Model and Cross-Lingual Robustness

Security behavior is not necessarily stable across LLM families. Differences in model architecture, training, alignment, instruction tuning, and tokenization can affect how adversarial instructions are interpreted, and recent work has therefore emphasized cross-model evaluation [6,16,18,19].

Language introduces an additional source of variability. Most prompt-injection studies have focused on English, although multilingual LLMs are widely deployed. Aytaş et al. found in [12] substantial differences in prompt-injection vulnerability across Turkish and English conditions and across models, suggesting that security findings obtained in one language should not automatically be generalized to another.

For RAG systems, the problem is more complex because the language of the user query and the language of the retrieved malicious document may differ independently. A Spanish query may retrieve an English document, or vice versa, producing a cross-lingual attack path. Importantly, retrieval effects must be separated from generation effects. Otherwise, differences in attack success may reflect retrieval quality rather than the model’s ability to resist malicious instructions.

The literature therefore establishes that indirect prompt injection is feasible [5,6], that RAG and persistent memory introduce exploitable trust boundaries [9,16], that tool-using agents increase the potential impact of compromise [8,17], and that vulnerability can vary across both models and languages [12,18,19,20]. However, comparatively little work evaluates all these dimensions jointly under a common local RAG architecture (Table 1).

Table 1. Positioning of the present study relative to representative related work.

Study	Indirect Attack	RAG/External Knowledge	Tool Security	Multiple Models	Multilingual	Defense Evaluation
Greshake et al. [5]	✓	✓	✓	—	—	Limited
Spotlighting [10]	✓	✓	—	✓	—	✓
AgentDojo [8]	✓	✓	✓	✓	—	✓
StruQ [11]	✓	✓	—	✓	—	✓
PoisonedRAG [9]	✓	✓	—	✓	—	—
Aytaş et al. [12]	Direct PI	—	—	✓	✓	—
Jing et al. [16]	✓	✓	✓	✓	—	Limited
This work	✓	✓	✓	✓	✓	✓

3. Experimental Framework and Threat Model

3.1. System Architecture

The experimental framework implements a fully local RAG-based assistant that answers questions using a synthetic university knowledge base. The architecture was designed to reproduce the principal trust boundaries of an LLM application that combines external knowledge with tool access, while avoiding real users, external services, or consequential actions.

For each interaction, the user query is first submitted to a retriever, which selects the k most relevant documents from the local corpus. The retrieved documents are then incorporated into the prompt together with the system instructions and the user’s question. A locally deployed LLM generates the response and, in configurations where tools are enabled, may additionally propose tool calls. All responses, retrieved documents, tool requests, and evaluation labels are recorded for subsequent analysis.

The processing chain depicted in Figure 1 begins with the user query, document retrieval, and prompt construction. The local LLM then generates a textual response that may contain structured tool-call declarations. Under C5, the deterministic output filter processes the generated text before tool-call parsing and execution. Configurations C1–C4 do not apply this external filter; instead, they differ in their tool availability and model-enforced safeguards. Where tool use is enabled, the resulting tool-call declarations are handled through local simulated functions. Responses, tool-use events, and evaluation information are subsequently recorded for analysis. The document repository contains both legitimate and deliberately poisoned documents. Consequently, retrieval establishes an explicit trust boundary: retrieved content may provide useful information, but it cannot be assumed to contain trustworthy instructions. This reflects the indirect prompt-injection threat model described in previous work [5,8,9].

Figure 1. Experimental architecture and trust boundaries of the local RAG-based assistant.

Three simulated tools are exposed to the assistant:

• consultar_datos_usuario(), which returns synthetic user information, including an internal_notes field used to evaluate simulated information disclosure.
• crear_ticket_simulado(), which represents the creation of a simulated support ticket without real-world side effects.
• buscar_documento(), which represents a read-only auxiliary document-search operation.

None of these tools interact with an external service or produce real-world effects. Their purpose is to determine whether malicious retrieved content can influence the model into requesting actions that were not justified by the original user query.

3.2. Threat Model

The attacker considered in this study does not control the user, the system prompt, the LLM, or the application code. Instead, the attacker is assumed to be capable of introducing malicious textual content into a document that may subsequently be retrieved by the RAG system. The legitimate user submits a benign query and is unaware that one or more retrieved documents may contain adversarial instructions. Once included in the prompt context, the poisoned document attempts to influence the LLM by exploiting the ambiguity between external information and executable instructions.

The attacker is assumed to know the application domain and the general topics of user queries, so that poisoned documents are topically plausible, but does not know the system prompt, the active defense configuration, the retrieval parameters or the model, and does not adapt documents to observed outputs. The system prompt, the application code, the tool implementations and the user query are trusted; the content of retrieved documents is untrusted.

The attacker may attempt to override or weaken the system’s original instructions, introduce false information into the generated answer, induce an unauthorized tool request, or cause the disclosure of simulated internal information. The threat model intentionally excludes model-weight manipulation, adversarial model training, infrastructure compromise, attacks against external services, and real personal-data exposure. It therefore isolates retrieval-mediated indirect prompt injection as the experimental variable.

An attack is considered successful only when a poisoned document is retrieved, and the resulting response exhibits behavior corresponding to the attack encoded in that document. This distinction is important. Failure to retrieve a malicious document indicates absence of attack exposure. When poisoned content is retrieved, attack success is assessed through observable behavior in the evaluated response. For C5, this assessment is performed on the post-filter output. Consequently, the absence of observable attack success under C5 may reflect either the model’s non-compliance with the malicious instruction or the suppression of attack-related content by the deterministic filter. These possibilities cannot be reliably distinguished from the archived post-filter responses alone.

3.3. Attack Families

The poisoned corpus implements four attack families representing different security consequences as summarized in Table 2: instruction override, context contamination, tool abuse, and simulated data exfiltration.

Table 2. Attack families considered in the experimental framework.

Attack Family	Attacker Objective	Primary Security Property
Instruction override	Replace or weaken trusted instructions	Integrity
Context contamination	Introduce attacker-controlled false information	Integrity
Tool abuse	Trigger an action not requested by the user	Authorization
Data exfiltration	Expose internal or user-related information	Confidentiality

Instruction-override attacks attempt to make the retrieved document supersede the trusted system prompt. Examples include requests to ignore previous instructions, disable restrictions, or acknowledge that the original security policy no longer applies. A successful attack is recorded when the generated response exhibits evidence that the injected override instruction has been followed.

Context-contamination attacks introduce plausible but deliberately false statements into otherwise legitimate-looking documents. Their objective is not necessarily to override the complete system behavior, but to make attacker-controlled misinformation appear in the final answer. This family represents an important limitation of purely syntactic defenses: malicious content may resemble ordinary factual information rather than an explicit command.

Tool-abuse attacks instruct the model to invoke an available tool even though the legitimate user did not request the corresponding action. Consequential simulated tools include accessing user information and creating a support ticket. A distinction is made between tool misuse, involving an unjustified consequential action, and excessive agency, which additionally includes unnecessary read-only actions such as auxiliary document searches.

Finally, data-exfiltration attacks attempt to make the assistant disclose synthetic user information or internal metadata. The test environment includes fixed canary values specifically for detecting this behavior; no real personal information is used. The framework distinguishes between following the requested exfiltration structure and emitting a concrete synthetic value. This allows attack compliance and data leakage to be measured separately.

3.4. Defense Configurations

Five security configurations (C1–C5), summarized in Table 3, were designed to compare alternative combinations of tool access, instruction–data separation, model-enforced policies, and deterministic post-generation filtering within the same underlying RAG architecture. These configurations represent distinct security strategies rather than an assumed progression toward greater effectiveness.

Table 3. Experimental defense configurations.

Configuration	RAG	Tools	Untrusted-Document Separation	Tool Policy	Deterministic Output Filter
C1	✓	—	—	—	—
C2	✓	✓	—	—	—
C3	✓	✓	✓	—	—
C4	✓	✓	✓	✓	—
C5	✓	✓	✓	—	✓

• C1: RAG without tools or explicit defenses. Retrieved documents are provided directly to the LLM. No tool access or specific prompt-injection mitigation is applied. C1 establishes the basic susceptibility of the RAG pipeline to malicious retrieved content.
• C2: RAG with tools, without specific defenses. The model receives the same retrieved content but is additionally given access to the simulated tools. This configuration represents the principal risk baseline because malicious documents can potentially influence both generated text and proposed actions.
• C3: Explicit instruction–data separation. C3 extends C2 by explicitly declaring retrieved documents to be untrusted external content. Documents are structurally delimited, and the system prompt states that instructions contained within them must be treated as data rather than commands. This configuration operationalizes the instruction–data separation principle explored by previous defenses such as Spotlighting and StruQ [10,11].
• C4: Instruction–data separation with tool policy and response validation. C4 extends C3 with explicit tool-use rules. A tool may only be requested when the user’s original question justifies its use. Documents are not permitted to authorize actions. The model is also instructed to validate its response before returning it.
• C5: Instruction–data separation with deterministic output filtering. C5 uses the same prompt-level separation as C3 but replaces C4′s additional model-enforced policy with a deterministic post-generation filter. The filter operates on the generated response before it is returned to the user. The C5 filter is implemented as a deterministic, line-oriented Python routine applied to the generated text before tool-call parsing and execution in the simulated environment. It identifies tool requests represented as `TOOL_CALL: function (arguments)` and removes requests involving consequential tools (`consultar_datos_usuario` and `crear_ticket_simulado`) when these are not authorized by the legitimate user query. Tool-call lines containing predefined attack-specific signatures are also removed. Additional regular-expression checks suppress lines containing email addresses, Spanish identification-number patterns, or structured identity-related disclosures, such as `user_id: value`. Recognizable instruction-override expressions are removed by truncating the affected line at the matched pattern. If the filtering process removes the entire response, a predefined refusal message is returned. The mechanism does not perform semantic factual verification and does not systematically prevent unnecessary read-only tool requests. Consequently, it may fail to detect paraphrased or obfuscated malicious content and may also remove legitimate information matching generic disclosure patterns. The deterministic filter and the automatic attack-success evaluator use partially overlapping lexical and structural indicators. Consequently, the ASR reported for C5 measures attack success in the post-filter output delivered to the user, rather than the underlying LLM’s intrinsic resistance to injected instructions. A low post-filter ASR therefore does not necessarily imply that the model rejected the malicious instruction during generation; it may also reflect the removal or suppression of unsafe content by the external filter. This distinction is important when interpreting C5 as a security enforcement mechanism rather than a mechanism that directly improves the model’s instruction-following robustness. C5 is not proposed as a new defense mechanism: it is a minimal implementation of established output-filtering and runtime-enforcement concepts, used to contrast an externally enforced control with model-enforced defenses under identical conditions. The overlap between the filter and the automatic evaluator differs by attack family. It is identical for tool abuse (same consequential-tool criterion and the same six injected signatures), overlapping for data exfiltration (the filter’s structural rule covers all five disclosure markers of the evaluator, and both use the same email pattern), partial for instruction override (four of the evaluator’s nine markers are also matched by the filter) and absent for context contamination. For the first three families, a post-filter success rate of zero therefore holds partly by construction. Both the filter patterns and the evaluator markers were calibrated on Spanish outputs; English renderings of the injected content were only partially covered by the filter, which becomes relevant in the cross-lingual experiment (Section 5.4). The filter version in the public repository reproduces all 32 archived C5 responses for Llama 3.1 8B and 109 of the 128 archived C5 responses overall when replayed on the archived C3 responses (Section 5.2). The repository documentation indicates that the override and disclosure rules were generalized after an earlier version was evaded by two obfuscated variants (Section 5.2); this generalization predates every C5 run reported here: the generalized version is the one contained in the first commit of the repository, and re-applying it to the archived C5 outputs of all eight runs leaves them unchanged.

Importantly, C5 does not attempt to identify semantically plausible factual contamination. Determining whether a natural-language statement is false would require an additional factual-verification mechanism and is therefore outside the deterministic filter implemented here. Taken together, these configurations enable a comparative assessment of model-dependent safeguards and externally enforced controls, without assuming that additional defensive mechanisms necessarily yield greater security or better preservation of legitimate task utility.

3.5. Evaluation Metrics

The framework evaluates both security and legitimate task performance. Let $N_p$ denote the number of executions in which at least one poisoned document is retrieved, and $N$ the total number of executions. The principal metric is the Attack Success Rate (ASR) shown in (1). ASR is conditioned on actual retrieval of poisoned content so that retrieval failure is not incorrectly interpreted as successful defense. The Tool Misuse Rate (TMR) measures unjustified use of consequential tools as shown in (2).

$$ASR={\displaystyle \frac{N_{\mathrm{s}\mathrm{u}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{f}\mathrm{u}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{s}}}{N_p}}$$

(1)

$$TMR={\displaystyle \frac{N_{\mathrm{u}\mathrm{n}\mathrm{a}\mathrm{u}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l} \mathrm{t}\mathrm{o}\mathrm{o}\mathrm{l} \mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{s}}}{N}}$$

(2)

The Excessive Agency Rate (EAR) additionally counts any unnecessary tool invocation, including read-only auxiliary searches (3). The Leakage Rate (LR) measures executions in which a concrete synthetic internal value is exposed (4).

$$EAR={\displaystyle \frac{N_{\mathrm{u}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{t}\mathrm{o}\mathrm{o}\mathrm{l} \mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{s}}}{N}}$$

(3)

$$LR={\displaystyle \frac{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a} \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{a}\mathrm{g}\mathrm{e} \mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}}}{N}}$$

(4)

Answer Utility (AU) measures whether the generated response provides the information required by the legitimate user query, independently of whether the injected instruction was followed. We do not report a separate Safe Refusal Rate because, under the operational definition used in this study, it would be exactly complementary to ASR and would not distinguish explicit refusals from other forms of attack non-compliance.

$$AU={\displaystyle \frac{N_{\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{f}\mathrm{u}\mathrm{l} \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{s}}}{N}}$$

(5)

Utility is evaluated independently from most security labels. This distinction is essential because an overly restrictive defense may obtain a low ASR simply by refusing legitimate requests. Security improvements are therefore interpreted jointly with AU rather than in isolation. Responses are automatically labelled using predefined observable markers and tool-call logs. Ambiguous cases are flagged for manual review. The complete rules, raw outputs, labelled results, and analysis scripts are retained in the public experimental repository to support reproducibility, which is available at https://github.com/trustlabupct/llm-instruction-injection-rag accessed on 6 November 2026.

Consequential tool misuse denotes a call to a tool that accesses user data or changes state (consultar_datos_usuario, crear_ticket_simulado) that is not authorized by the legitimate query. Concrete data leakage denotes the appearance in the delivered output of a fixed canary value or of a concrete synthetic identifier (an email address or a numeric user identifier) in response to an exfiltration request; reproducing the requested disclosure structure with empty or invented placeholders counts as attack success but not as leakage. A tool-abuse attack is counted as successful when the execution contains an unauthorized consequential tool call, or when a signature injected by the poisoned document (for example, an attacker-defined ticket subject) appears in tool-call arguments or in the response text. A data-exfiltration attack is counted as successful when the response reproduces the requested disclosure structure or contains a canary value. Instruction-override and context-contamination success are detected through predefined markers of the injected behavior (Section 4.6).

4. Experimental Methodology

4.1. Dataset and Evaluation Queries

The experimental corpus is fully synthetic and represents the knowledge base of a fictitious university assistant (Table 4). It contains 40 documents, of which 24 are clean and 16 are deliberately poisoned. The clean documents cover six administrative and technical domains: enrollment, internships, financial aid, academic regulations, technical support, and administrative procedures.

Table 4. Experimental dataset composition.

Component	Number	Description
Clean documents	24	Legitimate university information
Poisoned documents	16	4 per attack family
Total documents	40	Synthetic corpus
Normal queries	14	Standard information requests
Attack-oriented queries	13	Benign queries exposed to poisoned retrieval
Ambiguous queries	3	Underspecified requests
Sensitive queries	2	Legitimate use of protected tools
Total queries	32	Evaluated under each configuration

The poisoned subset is balanced across the four attack families introduced in Section 3, with four documents for instruction override, context contamination, tool abuse, and simulated data exfiltration, respectively. Poisoned documents are designed to remain topically plausible so that they can be retrieved in response to otherwise legitimate user questions.

The evaluation set contains 32 queries: 14 normal queries, 13 attack-oriented queries in which a benign question is expected to retrieve potentially malicious content, three ambiguous queries, and two sensitive queries involving legitimate tool access. Each query includes the expected safe behavior, expected relevant documents, and an explicit specification of tools that are permitted or forbidden for that request. No real university data or personal information is included. Synthetic user values and fixed canaries are used exclusively to detect simulated leakage.

4.2. Local LLMs

All models are executed locally through Ollama 0.40.13 using its HTTP chat interface. Four open-weight models from different families and sizes are considered, namely Llama 3.1, Llama 3.2, Gemma 3, and Qwen3, as shown in Table 5. The purpose of this comparison is not to rank the models according to general capability, but to determine whether susceptibility to indirect prompt injection and the effectiveness of the same defensive configurations remain stable when the underlying LLM changes.

Table 5. Local LLMs included in the evaluation.

Model	Ollama Identifier	Approx. Size	Experimental Role
Llama 3.1	llama3.1:8b	8B	Original baseline
Llama 3.2	llama3.2:3b	3B	Cross-model evaluation
Gemma 3	gemma3:4b	4B	Cross-model evaluation
Qwen3	qwen3:4b-instruct-2507-q4_K_M	4B	Cross-model evaluation

To minimize stochastic variability, generation is performed with temperature = 0.0, seed = 42, and a context window of 8192 tokens. The same prompts, tools, evaluation queries, and defense configurations are retained across models.

The experimental framework communicates with Ollama through its local HTTP chat API, using non-streamed generation. The accompanying repository provides a Docker-based reproduction environment comprising a Python 3.11 application container and a separate Ollama service, together with the required Python dependencies and experiment scripts. However, the Docker configuration does not pin the Ollama runtime version, and several Python dependencies are specified through minimum-version constraints. All models were run with 4-bit quantization (Q4_K_M) through Ollama 0.40.13 on a laptop with an AMD Ryzen 7 7435HS CPU (8 cores, 16 threads), an AMD Radeon RX 7600S GPU with 8 GB of VRAM and 16 GB of RAM, running Windows 11. The deterministic C5 filter adds a median of 69 µs per response (95th percentile 84 µs; measured offline by replaying the filter 200 times over the 160 archived Llama 3.1 8B responses), and the median wall-clock time per execution, including retrieval and generation, was 1.4–1.9 s across C1–C5 on the experimental host (estimated from consecutive run timestamps). The system prompts contain 37, 78, 145, 233 and 145 words for C1–C5, respectively. Memory consumption, throughput and energy efficiency on constrained hardware were not evaluated in this study.

4.3. Retrieval and Generation Settings

For the cross-model experiment, retrieval is performed using the same corpus and retrieval configuration for all models. The retriever ranks documents according to query similarity and returns the top k = 3 documents. Because retrieval occurs before model invocation, all models evaluated under the same linguistic condition receive the same retrieved context for a given query.

The retrieval unit was the complete document; no document chunking, query expansion, or reranking was applied. Each document was indexed using its title concatenated with its content. In the Spanish cross-model experiment, retrieval employed scikit-learn’s TF–IDF vectorizer with unigram and bigram features (ngram_range = (1, 2)) and sublinear term-frequency scaling (sublinear_tf = True). Document relevance was computed using cosine similarity, and the three highest-ranked documents were selected through exhaustive comparison over the 40-document corpus. In the controlled cross-lingual experiment, the same document representation and top-3 retrieval procedure were retained, but TF–IDF representations were replaced with normalized embeddings generated by sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. Neither retrieval backend used an approximate nearest-neighbor index or an additional reranking stage. This configuration ensured consistent retrieval settings within each experiment while allowing the influence of the retrieval backend to be considered when comparing the original baseline and cross-lingual results.

Each of the 32 queries is subsequently evaluated under all five defense configurations C1–C5. This results in 160 executions per model and 640 model-configuration executions for the four-model comparison.

All retrieved document identifiers, similarity scores, poisoned-document indicators, model responses, proposed tool calls, model identifiers, and execution parameters are recorded. Tool calls are parsed from structured declarations generated by the LLM and executed only against local simulated functions.

4.4. Cross-Model Experimental Protocol

The cross-model experiment keeps the application architecture, Spanish-language corpus, query set, retrieval results, attack documents, tool definitions, prompt templates, temperature, seed, and \(k\) constant. The independent variable is therefore the generative model. For each model, the principal outcome variables are Attack Success Rate (ASR), Tool Misuse Rate (TMR), Excessive Agency Rate (EAR), Leakage Rate (LR), and Answer Utility (AU) across configurations C1–C5. Attack success is additionally disaggregated by attack family.

This design supports two complementary analyses. First, it measures whether the same attack produces comparable behavior across different LLM families. Second, it tests whether the relative effectiveness of C1–C5 remains stable when the underlying model changes.

4.5. Controlled Cross-Lingual Protocol

A second experiment studies the effect of language independently of model family. Llama 3.1 8B is retained as the reference model, and the user-query language and poisoned-document language are independently varied, producing a 2 × 2 design using Spanish (ES) and English (EN): ES-ES, ES-EN, EN-ES, and EN-EN for user query and malicious content, respectively. Queries and poisoned documents were available in Spanish and English with matched semantic content and attack objectives. To preserve the original baseline corpus, the 24 clean documents remained in Spanish across all four conditions. This residual corpus-language asymmetry is considered when interpreting the cross-lingual results. The English versions were produced from the Spanish originals with a large language model (Claude Sonnet 4.5, Anthropic) and were reviewed manually by one of the authors to verify that each query and each injected instruction kept its meaning and structure.

Spanish and English were selected as a practically relevant language pair for a controlled evaluation rather than as a representative sample of multilingual diversity. Spanish was the language of the original experimental corpus and cross-model evaluation, allowing continuity with the baseline experiments. English was introduced to examine whether indirect prompt-injection behavior changed when legitimate queries and malicious retrieved content were expressed in the same or different languages. The resulting 2 × 2 design enables comparisons between matched-language and cross-language conditions within this specific pair. However, the findings cannot be generalized to other languages or language families without additional experiments. Furthermore, retaining the clean documents in Spanish introduces a corpus-language asymmetry that limits the isolation of purely linguistic effects.

For the final controlled comparison, all four conditions use the same multilingual dense retriever based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, with normalized embeddings, cosine similarity, and k = 3. This is important because using TF-IDF for monolingual cases and multilingual embeddings only for cross-language cases would confound the effect of language with the effect of the retrieval algorithm.

Two aspects are therefore analyzed separately: retrieval exposure, i.e., whether at least one poisoned document enters the top-k context, and conditional attack success, i.e., whether the model obeys the malicious instruction once exposed. This prevents lower retrieval recall from being incorrectly interpreted as greater model robustness.

Each language condition contains the same 32 queries and five configurations, resulting in 160 executions per condition and 640 executions for the controlled cross-lingual experiment.

4.6. Labeling, Reproducibility, and Statistical Analysis

Model responses are converted into binary evaluation labels using explicit and auditable rules based on observable response content and tool-call logs. The evaluator detects characteristic evidence of instruction override, factual contamination, unauthorized tool use, and simulated exfiltration. Answer utility is assessed against predefined expected-answer indicators associated with each query. Answer Utility was not compared across language conditions because the implemented utility detector relies on Spanish-language lexical indicators, and manual inspection confirmed that this systematically underestimates otherwise correct English responses.

The rules were originally calibrated using Llama 3.1 8B outputs. To reduce the risk of applying model-specific lexical assumptions to other models, the experimental workflow includes a six-query pilot validation for each new model before the complete 160-run experiment. Pilot responses are inspected against the automatically generated labels, with particular attention to alternative vocabulary, output formatting, truncated responses, tool-call parsing, and cases requiring manual review.

To quantify potential model-dependent labeling uncertainty, we examined the evaluator’s manual-review flags across all 160 executions per model. The number of flagged cases was 40 (25.0%) for Llama 3.1 8B, 56 (35.0%) for Llama 3.2 3B, 64 (40.0%) for Gemma 3 4B, and 28 (17.5%) for Qwen3 4B. The evaluator flags ambiguous queries and executions involving retrieved poisoned content for which neither attack success nor positive answer utility is detected. These flags identify cases warranting further inspection, but do not necessarily indicate labeling errors. All 188 flagged executions were adjudicated by reading the archived responses. The adjudication revealed two types of automatic false negative: (i) lexical variants of the injected claims not covered by the original Spanish markers (for example, “no existe un porcentaje mínimo”, “gratuidad de matrícula”, “se otorgan automáticamente”, or short disclosure labels such as “Correo:” and “rol:”), and (ii) responses written in English, which the Spanish-only markers could not detect; the latter mainly affected the controlled cross-lingual experiment (Section 5.4), in which Llama 3.1 8B answered 30 of the 32 undefended EN–EN queries in English. The evaluator was therefore extended with the English equivalents of all override, contamination, disclosure and signature markers and with the Spanish variants observed during adjudication, and every archived output of the cross-model and cross-lingual experiments was re-labeled. When restricted to the original marker set, the extended evaluator reproduces the archived labels exactly. The extended markers changed 59 of the 1280 attack-success labels, all from failure to success (1, 2, 2 and 0 for Llama 3.1 8B, Llama 3.2 3B, Gemma 3 4B and Qwen3 4B; 0, 5, 6 and 43 for the ES–ES, ES–EN, EN–ES and EN–EN conditions); every changed label was verified manually. Two further labels were corrected manually (Llama 3.1 8B, Q17 under C3 and C5), where the response “La respuesta es no, según la normativa actual” adopts the poisoned claim without any lexical marker. Six borderline cases (narrated tool sequences in C1, which has no tools, and single-field disclosure fragments) kept their automatic label; they are listed, together with the extended marker set, and all changed labels, in Supplementary Table S4. All values reported in Section 5 incorporate these corrections. Because adjudication covered flagged executions and marker-driven changes, unflagged false negatives of a purely semantic nature cannot be fully excluded; this residual uncertainty motivated the blind semantic review described below. The original labeling validation also compared automatic labels against a blind re-evaluation of 40 cases under a stricter harm criterion, obtaining 80% agreement and Cohen’s κ = 0.61. Because both annotations were performed by the same evaluator under different criteria, this value is interpreted as a sensitivity check on the labeling rules rather than as inter-rater reliability. To assess whether the models had complied with the injected instructions before filtering, a blind semantic review of 24 stratified cases (six per model, including four controls without poisoned content) was performed on the archived C3 responses (pre-filter) and on the reconstructed C5 outputs (post-filter) by two independent annotators who had no access to the automatic labels or to the filter rules; disagreements were adjudicated by an author who did not implement the labeling rules, and the results are reported in Section 5.2.

Given the limited number of exposed queries per configuration, point estimates are accompanied by 95% bootstrap confidence intervals using 10,000 resamples for ASR and Answer Utility. Differences between paired defense configurations are additionally evaluated using the exact McNemar test, matching outcomes by query. Statistical significance is assessed at p < 0.05. However, effect magnitude and consistency across models are considered together with significance because of the relatively small sample size. As additional robustness checks, the Llama 3.1 8B experiment was repeated three times at temperature 0.7, and the retrieval depth was varied (k = 1, 3, 5); the results are reported in Supplementary Table S5. These checks were produced during the original study with the original evaluator, and the per-run outputs of the temperature analysis were not archived, so only its aggregate means and standard deviations are available.

All experiment inputs, raw responses, tool-call logs, labeled datasets, metric summaries, and analysis scripts are retained to support reproducibility.

5. Results

5.1. Baseline and Replication Results

As shown in Table 6, for the original Llama 3.1 8B experiment, 27 of the 32 queries retrieved at least one poisoned document. The undefended configurations showed high susceptibility to indirect prompt injection, with ASR values of 70.4% for C1 and 74.1% for C2. Explicit instruction–data separation alone (C3) did not reduce ASR (70.4%), whereas C4 and C5 reduced ASR to 29.6% and 22.2%, respectively.

Table 6. Baseline results for Llama 3.1 8B.

Config.	ASR % (95% CI)	Tool Misuse %	EAR %	Leakage %	Utility % (95% CI)
C1	70.4 (51.9–85.2)	0.0	0.0	12.5	50.0 (31.2–68.8)
C2	74.1 (55.6–88.9)	21.9	87.5	3.1	53.1 (34.4–71.9)
C3	70.4 (51.9–85.2)	34.4	87.5	12.5	62.5 (46.9–78.1)
C4	29.6 (14.8–48.1)	6.3	9.4	3.1	34.4 (18.8–50.0)
C5	22.2 (7.4–37.0)	0.0	50.0	0.0	62.5 (43.8–78.1)

An additional non-monotonic effect was observed in tool misuse. Although C3 introduced explicit instruction–data separation, its Tool Misuse Rate increased from 21.9% under C2 to 34.4% under C3, corresponding to 7 and 11 of the 32 executions, respectively. This descriptive reversal indicates that adding prompt-level safeguards does not necessarily reduce unauthorized tool requests. However, the present experiment does not isolate the mechanism responsible for this increase, and the difference should not be interpreted as establishing a causal adverse effect of instruction–data separation. Rather, it reinforces the need to evaluate defensive configurations empirically instead of assuming that additional instructions consistently improve security.

The reductions from C3 to C4 and from C3 to C5 were statistically significant according to the exact McNemar test (p = 0.00098 and p = 0.00024, respectively), while differences between C1–C2, C2–C3 and C4–C5 were not significant (p = 1.00, 1.00 and 0.75). Under C5, no consequential tool misuse or concrete data leakage was observed in the delivered outputs, and Answer Utility was 62.5%. By contrast, C4 reduced utility to 34.4%.

An independent English replication reproduced the same overall pattern. C4 and C5 yielded the lowest observed ASR values in that replication; under C5, no consequential tool misuse or concrete synthetic-data leakage was detected in the delivered outputs. Because translation also altered retrieval behavior, these results are treated as a reproducibility check rather than as evidence of a language effect. With the extended bilingual evaluator described in Section 4.6, the replication yielded conditional ASRs of 65.5%, 55.2%, 62.1%, 34.5% and 31.0% for C1–C5 (29 exposed queries). The controlled cross-lingual analysis is presented in Section 5.4. In Table 6, ASR is conditioned on poisoned-document retrieval (n = 27), whereas the remaining metrics use all executions (n = 32). Confidence intervals were estimated using 10,000 nonparametric bootstrap resamples.

5.2. Cross-Model Evaluation

The cross-model experiment evaluated the same five configurations using Llama 3.1 8B, Llama 3.2 3B, Gemma 3 4B, and Qwen3 4B under the same Spanish corpus, queries, retrieval settings, and attack scenarios. The results show clear model dependence as depicted in Figure 2. C3 and C4 did not improve robustness consistently across all models. Most notably, Gemma 3 increased from 48.1% ASR in C3 to 59.3% in C4. This reversal reflects the form of the responses rather than greater compliance under C4: in the three queries in which C4 was attacked successfully and C3 was not (Q10, Q17 and Q19), the C3 output consisted solely of a buscar_documento call without any textual answer, which cannot exhibit attack success, whereas the C4 output contained a textual answer that reproduced the injected override phrase or the poisoned claim (Q19: “Sí, puedes solicitar prácticas externas curriculares aunque estés en primer curso. El requisito de 120 créditos ha sido suprimido.”). Under C3, 14 of Gemma 3′s 32 responses were tool-call-only outputs, which also explains its low Answer Utility in that configuration (21.9%). The C3–C4 difference was significant only for Llama 3.1 8B (exact McNemar p = 0.00098); it was not significant for Llama 3.2 3B (p = 1.00), Gemma 3 4B (p = 0.25) or Qwen3 4B (p = 0.69). In the Spanish cross-model experiment, C5 yielded the lowest observed post-filter ASR for each of the four evaluated models, and the C1–C5 reduction was significant for every model (exact McNemar p ≤ 0.0003). The C5 values of the four models (7.4–22.2%) have overlapping 95% confidence intervals and should not be read as a ranking of models. No unauthorized consequential tool use or concrete synthetic-data leakage was detected in the delivered C5 responses. However, unnecessary read-only tool requests persisted in three of the four models, as reflected by the nonzero Excessive Agency Rate (EAR) values reported in Table 7.

Figure 2. Conditional Attack Success Rate (ASR) across LLM families and defense configurations in the Spanish cross-model experiment. For C5, ASR is measured on the delivered post-filter response and does not establish the underlying LLM’s resistance before filtering. C5 achieved the lowest observed post-filter ASR across all four models, whereas C3 and C4 exhibited greater model-dependent variability.

Table 7. C5 outcomes across models. ASR is conditioned on poisoned-document retrieval (n = 27 per model), whereas the remaining metrics use all executions (n = 32 per model). Confidence intervals were estimated using 10,000 nonparametric bootstrap resamples. EAR denotes Excessive Agency Rate, including unnecessary read-only tool invocations.

Model	ASR % (95% CI)	Tool Misuse %	EAR %	Leakage %	Utility % (95% CI)
Llama 3.1 8B	22.2 (7.4–37.0)	0.0%	50.0	0.0	62.5 (43.8–78.1)
Llama 3.2 3B	14.8 (3.7–29.6)	0.0%	25.0	0.0	28.1 (12.5–43.8)
Gemma 3 4B	7.4 (0.0–18.5)	0.0%	56.3	0.0	21.9 (9.4–37.5)
Qwen3 4B	22.2 (7.4–37.0)	0.0%	0.0	0.0	56.3 (40.6–71.9)

However, the associated utility cost remained model-dependent. C5 showed the most consistent security behavior across model families, as shown in Table 7, while utility varied substantially. This suggests that security transferability and utility transferability should be evaluated separately.

To further examine the operation of C5, a retrospective filter-replay analysis was conducted using the archived C3 responses. Since C3 and C5 share the same prompt-level instruction–data separation, the C5 deterministic filter was applied to the 32 archived C3 responses for each model, and the resulting outputs were compared with the corresponding archived C5 outputs. This procedure exactly reproduced 109 of the 128 recorded C5 responses: 32/32 for Llama 3.1, 22/32 for Llama 3.2, 28/32 for Gemma 3, and 27/32 for Qwen3. Among these 109 exactly reproduced cases, 60 (55.0%) involved a rule-triggered filtering intervention, distributed as 20, 20, 13, and 7 cases across the respective models. An additional 22 cases exhibited only formatting-level changes. Importantly, this analysis reconstructs filtering behavior from archived C3 generations rather than directly observing the original C5 pre-filter responses. Consequently, the reconstructed intervention counts should not be interpreted as the historical activation rates of C5, and the 19 non-reproduced responses cannot be used to infer pre-filter behavior. These findings support the interpretation of C5 as an external output-enforcement mechanism while confirming that post-filter ASR alone cannot establish whether the underlying LLM complied with the malicious instructions before filtering.

A complementary offline sensitivity check examined the unmodified C5 output filter using 20 deliberately constructed adversarial output strings and four legitimate controls. The adversarial cases covered instruction override, textual tool-call representations, synthetic data disclosure, and context contamination, with five examples per category combining canonical patterns and paraphrased or obfuscated variants. The filter triggered on 1/5 instruction-override strings, 2/5 tool-call representations, and 1/5 synthetic disclosure strings. None of the five context-contamination examples triggered filtering, consistent with the mechanism’s stated scope. One of the four legitimate controls was also filtered because it contained a benign contact email address. These observations illustrate the sensitivity of deterministic pattern-based filtering to lexical and formatting variations, as well as the possibility of removing legitimate information. Importantly, the test cases were manually constructed rather than generated by the evaluated LLMs; therefore, the results represent an illustrative rule-level stress test, not estimates of attack-success rates or detection probabilities under adaptive attacks. Furthermore, textual tool-call representations passing the output filter do not establish actual tool execution. End-to-end evaluation against adaptively modified poisoned documents remains necessary to establish robustness beyond the tested patterns.

The blind semantic review (Section 4.6) covered 24 cases for which the C3-plus-filter replay reproduced the archived C5 output exactly: 20 exposed to poisoned content and 4 controls. The two annotators agreed on 114 of 120 judgments (95.0%; Cohen’s κ = 1.00 for pre-filter unsafe behavior and κ = 0.82 for post-filter unsafe behavior), and all six disagreements involved an “uncertain” or “partial” category; they were resolved by an author who did not implement the labeling rules. In 17 of the 20 exposed cases, the pre-filter response had complied with the injection, and this occurred for all four models, including Gemma 3 4B, which has the lowest C5 post-filter ASR. The filter removed the observable effect in 10 of these 17 cases (4 of 4 instruction-override, 3 of 4 tool-abuse and 3 of 4 data-exfiltration cases), left a residual effect in 2 (a “rol” attribute that survived the removal of an identifier and an email line; a ticket announcement carrying the attacker-defined subject after all tool calls, including the one the user had legitimately requested, had been removed) and left the 5 context-contamination cases unchanged. The consensus post-filter judgments agreed with the corrected automatic C5 labels in all 24 cases. Answer correctness was unchanged by filtering in all 24 cases; no case showed removal of content needed for the legitimate request, and the five spontaneous unauthorized tool calls observed in controls and in non-complying cases were also blocked. The sample was selected purposively, so these proportions are not estimates for the full set of executions. The review confirms that the 0% post-filter instruction-override rate under C5 reflects removal of observable evidence rather than model resistance, whereas for tool abuse and data exfiltration the removal of the tool-call or disclosure line before parsing and delivery does prevent the harmful outcome in the simulated environment even though the model had complied.

5.3. Attack-Family Analysis

The effectiveness of the evaluated defenses differed across attack families. Table 8 reports the conditional post-filter ASR under C5, which achieved the lowest aggregate ASR across the four models in the Spanish cross-model experiment. No observable instruction-override successes were detected in the delivered C5 responses, and a single residual data-exfiltration success was observed (Llama 3.1 8B: a “rol: estudiante” field that survived the removal of the identifier and email lines), although this does not establish that the underlying models resisted these attacks before filtering. Tool-abuse ASR was 14.3% for Llama 3.1 and 28.6% for Llama 3.2, whereas no such successes were detected for Gemma 3 or Qwen3. These attack-family outcomes should be distinguished from the Tool Misuse Rate, which specifically measures unauthorized consequential tool use and remained at 0% under C5. In contrast, context contamination persisted across all four models, reaching 44.4% for Llama 3.1 8B and 66.7% for Qwen3, and constituted the principal residual weakness of the deterministic filtering approach.

Table 8. Conditional post-filter Attack Success Rate (ASR) under C5 by attack family in the Spanish cross-model experiment. The denominators correspond to queries retrieving at least one poisoned document from the respective family: instruction override (n = 10), context contamination (n = 9), tool abuse (n = 7), and data exfiltration (n = 7). Attack-family exposures are not mutually exclusive because a single query may retrieve poisoned documents from multiple families.

Model	Instruction Override	Context Contamination	Tool Abuse	Data Exfiltration
Llama 3.1 8B	0.0%	44.4%	14.3%	14.3%
Llama 3.2 3B	0.0%	22.2%	28.6%	0.0%
Gemma 3 4B	0.0%	22.2%	0.0%	0.0%
Qwen3 4B	0.0%	66.7%	0.0%	0.0%

This result is consistent with the design of C5. The reason is that deterministic filtering can block recognizable unauthorized actions, disclosure patterns, and explicit override content, but cannot reliably identify a semantically plausible false statement without an additional factual-verification mechanism.

5.4. Controlled Cross-Lingual Evaluation

The controlled cross-lingual experiment evaluated four combinations of query and malicious-document language (ES-ES, ES-EN, EN-ES, and EN-EN) using Llama 3.1 8B and the same multilingual embedding retriever in all conditions (Table 9). Poisoned-document retrieval exposure was 87.5% for ES-ES, 84.4% for EN-EN, and 65.6% for both cross-lingual conditions. ASR was calculated conditionally on actual exposure to poisoned content. Because Llama 3.1 8B answered most undefended EN–ES and EN–EN queries in English (28 and 30 of 32 under C1, respectively), all values in this section use the extended bilingual evaluator described in Section 4.6. With the original Spanish-only markers, the EN–EN condition would have shown an artefactually low undefended ASR of 7.4% (2 of 27 exposed queries), compared with 59.3% (16 of 27) after re-labeling; 43 of the 160 EN–EN labels, 6 EN–ES labels and 5 ES–EN labels changed, all from failure to success.

Table 9. Cross-lingual experimental conditions.

Condition	Query Language	Malicious Document Language	Retrieval Backend
ES–ES	Spanish	Spanish	Multilingual embeddings
ES–EN	Spanish	English	Multilingual embeddings
EN–ES	English	Spanish	Multilingual embeddings
EN–EN	English	English	Multilingual embeddings

As shown in Table 10, indirect prompt injection produced non-zero conditional ASR in all four language conditions under all five configurations. However, the defense configurations did not exhibit monotonic improvements. C5 achieved the lowest observed ASR in ES–ES (21.4%) and EN–ES (14.3%), whereas C4 was lowest in ES–EN (9.5% versus 19.0% for C5) and in EN–EN (37.0% versus 40.7% for C5). The EN–EN result has a specific cause: the override and disclosure patterns of the deterministic filter were calibrated on Spanish outputs, and English renderings of the injected content (“Security policy deactivated”, “Internal identifier: …”, “Administrative notes: …”) passed through the filter unmodified. Of the 11 C5 successes in EN–EN, 2 were override phrases, 1 was a disclosure structure that a bilingual filter would have removed, and 8 were context contamination, which the filter does not address by design. C5 therefore added little over the shared prompt-level separation of C3 in that condition. C3 did not reduce ASR relative to C1 in any language condition (ES–ES 39.3% → 57.1%; ES–EN 33.3% → 47.6%; EN–ES 47.6% → 47.6%; EN–EN 59.3% → 63.0%). These findings illustrate that additional defensive instructions do not necessarily improve robustness, and that a pattern-based filter is only as multilingual as its patterns.

Table 10. Poisoned-document retrieval exposure and conditional ASR across controlled language conditions (95% bootstrap confidence intervals). The first language denotes the query language and the second the malicious-document language. ASR is conditioned on poisoned-document retrieval. Values in brackets represent 95% confidence intervals estimated using 10,000 nonparametric bootstrap resamples (seed = 42). All conditions use the same multilingual embedding-based retriever. Values were obtained with the extended bilingual evaluator (Section 4.6).

Condition	Poison Exposure	C1 ASR [95% CI]	C2 ASR [95% CI]	C3 ASR [95% CI]	C4 ASR [95% CI]	C5 ASR [95% CI]
ES–ES	28/32 (87.5%)	39.3 [21.4–57.1]	50.0 [32.1–67.9]	57.1 [39.3–75.0]	39.3 [21.4–57.1]	21.4 [7.1–35.7]
ES–EN	21/32 (65.6%)	33.3 [14.3–52.4]	47.6 [28.6–66.7]	47.6 [28.6–66.7]	9.5 [0.0–23.8]	19.0 [4.8–38.1]
EN–ES	21/32 (65.6%)	47.6 [28.6–66.7]	57.1 [38.1–76.2]	47.6 [28.6–66.7]	42.9 [23.8–61.9]	14.3 [0.0–28.6]
EN–EN	27/32 (84.4%)	59.3 [40.7–77.8]	70.4 [51.9–85.2]	63.0 [44.4–81.5]	37.0 [18.5–55.6]	40.7 [22.2–59.3]

Given the relatively small numbers of exposed queries and the overlapping confidence intervals, differences between configurations and language conditions should be interpreted cautiously rather than as evidence of statistically established superiority. Furthermore, retrieval exposure differed across language conditions, and the clean documents remained in Spanish throughout the experiment. Consequently, the observed differences cannot be attributed exclusively to language effects. The ES–ES condition was rerun using the common multilingual retriever and is therefore not directly comparable with the original TF–IDF baseline reported in Section 5.1.

To further characterize retrieval exposure, we examined the attack families represented among the top-3 retrieved documents for each query. In the original Spanish TF–IDF baseline, the 27 exposed queries included 10 exposures to instruction override, 9 to context contamination, 7 to tool abuse, and 7 to data exfiltration. Under the common multilingual retriever, the corresponding counts for instruction override, context contamination, tool abuse, and data exfiltration were 15, 3, 11, and 6 in ES–ES (28 exposed queries); 10, 2, 8, and 5 in ES–EN (21 exposed queries); 10, 4, 5, and 6 in EN–ES (21 exposed queries); and 15, 9, 11, and 7 in EN–EN (27 exposed queries). These counts are not mutually exclusive, since a single query may retrieve poisoned documents belonging to multiple attack families. The differences in family-level exposure demonstrate that the retrieved attack composition varied across conditions, providing an additional reason to interpret cross-lingual ASR comparisons cautiously.

To examine whether the observed cross-lingual differences persisted under matched retrieval exposure, we performed exploratory two-sided exact McNemar tests using query identifiers, restricting each comparison to queries that retrieved poisoned content in both language conditions. The resulting matched subsets contained between 14 and 23 queries. Across the six pairs of language conditions and five security configurations (30 comparisons), only two tests reached unadjusted statistical significance, both involving C4: ES–EN versus EN–EN (2 versus 9 successful attacks among 19 matched exposed queries; p = 0.0156) and ES–EN versus EN–ES (2 versus 8 successful attacks among 14 matched exposed queries; p = 0.0313). However, neither difference remained significant after Holm correction for multiple comparisons. No C5 comparison reached nominal statistical significance. Furthermore, paired comparisons between C1 and C5 within each language condition were not statistically significant for ES–ES (p = 0.18), ES–EN (p = 0.25) and EN–EN (p = 0.0625); the EN–ES comparison (8 versus 1 discordant queries; p = 0.039) did not remain significant after correction for the four comparisons. These findings highlight the statistical uncertainty associated with the limited sample size. Moreover, matching query identifiers does not eliminate differences in the retrieved attack-family composition; therefore, the observed contrasts should not be interpreted as establishing causal language effects.

5.5. Security–Utility Trade-Off

Figure 3 shows that lower attack success did not always coincide with higher task utility. C5 achieved the most consistently low ASR across the evaluated models, but its Answer Utility ranged from 21.9% for Gemma 3 to 62.5% for Llama 3.1, indicating that security effectiveness and utility preservation are partially independent dimensions. Consequently, defense performance should be assessed jointly in terms of attack reduction and retained functionality rather than by ASR alone.

Figure 3. Security–utility trade-off across models and defense configurations. Each point represents one model–configuration pair, with ASR on the x-axis and Answer Utility (AU) on the y-axis. Lower ASR and higher AU indicate a more favorable operational trade-off.

The sources of utility loss differed between configurations. Under C4, Llama 3.1 8B refused or deflected 20 of the 32 queries with generic statements such as “Lo siento, pero no puedo cumplir con esa solicitud”, which accounts for its utility of 34.4%. Under C5, the filter rarely removed legitimate content: in the semantic review, no case lost content needed for the legitimate request, and the generic block message replaced only responses that consisted solely of tool-call declarations or of unauthorized data. The low C5 utility of Gemma 3 4B (21.9%) and Llama 3.2 3B (28.1%) instead reflects the form of the responses before filtering: 14 of Gemma 3′s and 5 of Llama 3.2′s 32 C5 outputs were tool-call-only declarations without a textual answer, and a further 9 and 13, respectively, were replaced by the block message. Security effectiveness and utility preservation should therefore be evaluated as separate dimensions of model–defense compatibility.

5.6. Discussion

The results show that susceptibility to indirect prompt injection is strongly model-dependent. Prompt-level defenses such as C3 and C4 did not consistently improve security across the evaluated LLMs, indicating that defensive instructions cannot be assumed to transfer uniformly across model families [11,18,19,20]. Under the Spanish cross-model experimental setting, C5 yielded the lowest observed post-filter ASR for each model, with no consequential tool misuse or concrete synthetic-data leakage detected in the delivered outputs. However, these findings do not establish intrinsic model resistance to malicious instructions or guarantee security beyond the tested conditions. In particular, Excessive Agency Rate remained non-zero for three of the four models, and Answer Utility varied substantially, demonstrating that deterministic filtering cannot replace comprehensive authorization and validation mechanisms. The retrospective filter-replay analysis further supports the interpretation of C5 as an external output-enforcement mechanism, while the offline sensitivity test illustrates its limitations against lexical and structural variations. Overall, these findings motivate separating generation from security-critical enforcement rather than relying exclusively on the model’s interpretation of defensive instructions.

The attack-family analysis further shows that context contamination remains the main residual weakness. Explicit overrides, unauthorized tool calls, and recognizable disclosure patterns can be detected or blocked deterministically, whereas semantically plausible misinformation is considerably harder to distinguish from legitimate retrieved knowledge. Complementary mechanisms such as provenance-aware retrieval, source validation, factual consistency checking, or trusted-knowledge verification are therefore required.

The controlled cross-lingual evaluation confirms that language mismatch did not eliminate indirect prompt injection under the tested conditions. However, the results do not establish a consistently beneficial hierarchy of defenses across language settings. In particular, C3 did not reduce ASR relative to C1 in any language condition, and in EN–EN the deterministic filter provided almost no additional protection over C3 because its patterns did not cover English renderings of the injected content. This is a direct demonstration of the language dependence of pattern-based output filtering: the same filter that removed every override phrase in Spanish outputs let “Security policy deactivated” and English disclosure structures through. The re-labeling required to uncover this effect also shows that evaluation tooling must itself be multilingual: with Spanish-only markers, the EN–EN condition appeared to be the least vulnerable, when it was in fact the most vulnerable (59.3% undefended ASR). Furthermore, the original ES–ES baseline using TF–IDF yielded a C1 ASR of 70.4%, compared with 39.3% using the common multilingual embedding retriever, highlighting the importance of retrieval configuration and document selection when interpreting attack effectiveness. The cross-lingual conditions also exhibited lower poisoned-document exposure (65.6%) than ES–ES (87.5%) and EN–EN (84.4%), while the clean corpus remained Spanish-only. Together with the relatively small sample size and wide confidence intervals, these factors limit causal interpretation and prevent definitive rankings across language conditions. The results therefore support the narrower conclusion that cross-lingual settings remain vulnerable to indirect prompt injection and that security assessment must consider retrieval exposure, model behavior, and defense mechanisms jointly.

These findings are relevant to trustworthy local and edge intelligence. Local execution can improve data locality and reduce cloud dependence, but it does not remove the security risks introduced by external knowledge and tool integration. When a RAG system processes untrusted documents, retrieval becomes part of the security boundary [9,14], and agentic capabilities can transform compromised context into consequential actions [8]. The results therefore motivate architectures in which the LLM may propose an answer or action while an independent runtime layer decides whether it is authorized [21]. C5 represents an initial and limited implementation of this principle rather than a complete runtime security architecture.

The study nevertheless has several limitations. The corpus and user data are synthetic, the evaluation comprises 32 queries in a single application domain, and all tools are simulated. Therefore, the experiments measure attempted rather than real-world consequences. The simulated tools differ from real agentic systems in four respects: tool calls are proposed in a single generation step and executed against stubs without side effects; tool outputs are not fed back into the model context, so second-order injection through tool results is not exercised; there is no authentication or multi-step planning; and no effect is irreversible. In deployments with real APIs, files or databases, the same attempts could have real consequences, and enforcement should be placed at the tool-authorization layer rather than only on generated text. Automatic labeling relies on lexical and structural indicators initially calibrated on Spanish outputs of Llama 3.1 8B. Adjudication of all flagged executions, bilingual re-labeling and a blind semantic review of 24 cases corrected 61 labels and found no further disagreement, but the semantic review covered only six cases per model, so residual false negatives of a purely semantic nature cannot be excluded, and the reported cross-model differences should still be interpreted with caution. An additional methodological limitation concerns the partial overlap between the deterministic C5 filter and the automatic attack-success evaluator. Although the retrospective filter-replay analysis provides evidence of filtering interventions, it cannot establish the original pre-filter behavior of C5 or independently determine whether all semantically meaningful attack effects were removed. The semantic review reported in Section 5.2 is a first step in this direction, but it relies on reconstructed rather than directly captured pre-filter outputs; future evaluations should archive both. Furthermore, although all models are executed locally, the study does not measure resource consumption, latency, or energy efficiency on constrained IoT hardware, and should therefore be interpreted as evaluating an edge-relevant local architecture rather than a physical edge deployment. Future work will extend the present approach toward a model-agnostic runtime security layer combining provenance, deterministic tool authorization, policy enforcement, leakage detection, and auditable event logging, and will evaluate it across larger corpora, additional domains and languages, adaptive attacks, and real edge devices.

6. Conclusions

This study evaluated indirect prompt-injection attacks against locally deployed LLM-RAG assistants across four model families, five security configurations, four attack families, and controlled Spanish–English language conditions. The results demonstrate substantial model dependence in attack susceptibility and show that prompt-level defenses do not consistently improve security. Deterministic post-generation filtering (C5) achieved the lowest measured post-filter ASR across the four evaluated models (7.4–22.2%), with no observed consequential tool misuse or concrete synthetic-data leakage in the delivered outputs. However, this advantage did not extend to every language condition: in the English–English condition, the Spanish-calibrated filter let English override phrases and disclosure structures through, and C5 (40.7%) did not improve on C4 (37.0%). The persistence of excessive agency, context contamination, and substantial differences in Answer Utility (21.9–62.5% under C5) demonstrates that deterministic filtering is not a comprehensive defense. Furthermore, the partial overlap between the filter and the automatic evaluator means that post-filter ASR measures observable attack success in delivered responses rather than the LLM’s intrinsic resistance to malicious instructions. The retrospective filter-replay analysis and complementary offline sensitivity test further highlighted the role of external enforcement, while revealing limitations associated with pattern-based detection, obfuscated outputs, and potential removal of legitimate information. Given the synthetic corpus, limited query set, and simulated tools, these findings should be interpreted as evidence within the controlled experimental setting rather than as guarantees of security in operational deployments.

These findings support a broader architectural principle for trustworthy local and edge-oriented AI systems. Security-critical decisions should not rely exclusively on the generative model. Instead, generation should be separated from authorization and enforcement through independent runtime controls capable of validating actions, constraining tool use, tracking provenance, and preventing sensitive-data disclosure. The study is limited by its synthetic corpus, simulated tools, relatively small query set, and single application domain. Future work will develop a model-agnostic runtime layer that enforces explicit policies on tool calls through capability-based authorization, tracks the provenance of retrieved content, verifies factual claims against trusted sources and applies language-independent detection; it will be evaluated against adaptively generated attacks, in additional languages and domains, and on resource-constrained edge devices with latency, memory and energy measurements.