Recent Advances and Open Challenges in Mitigating Inference-Time Attacks on Large Language Models

Abstract

The rapid integration of Large Language Models into high-stakes domains has elevated inference-time attacks into a primary security concern for production deployments. These attacks are adversarial techniques that exploit models exclusively through their input–output interface. The existing survey literature lacks a dedicated and structured treatment that jointly maps the attack surface and systematically evaluates the mitigation strategies developed against it. This paper addresses this gap through two original taxonomic contributions. First, LLM vulnerabilities are organized into a three-layer attack surface taxonomy stratified by lifecycle stage, establishing the theoretical primacy of the inference time category. Second, to directly address how these attacks can be mitigated, a defense taxonomy spanning three axes, namely prompt-level, inference-time, and training-time interventions, is proposed, within which 30 mitigation mechanisms published from 2024 onwards are systematically analyzed. Building on this taxonomy, an intersectional comparative analysis is conducted across three dimensions: defense-attack coverage, security-utility-latency tradeoffs, and white-box versus black-box applicability, in order to evaluate how effectively current mitigation strategies neutralize each attack category. These dimensions are further synthesized into a practitioner decision framework that maps deployment constraints to concrete defense configurations and identifies two structural coverage gaps that persist regardless of access level or latency budget. The resulting Defense-Attack Coverage Matrix demonstrates that no single defense mechanism provides comprehensive protection, and that robust deployment mandates layered, complementary strategies. The analysis further reveals that the fundamental unresolved tension limiting effective mitigation is the trade-off between adversarial robustness and model utility, with over-refusal and capability degradation constituting the primary practical barriers to deploying these defenses. Finally, open challenges related to multimodal attack surfaces, agentic LLM security, and the absence of standardized evaluation frameworks are identified, together with concrete future research directions. The taxonomies and analyses presented are intended to serve as an actionable reference for both researchers and practitioners tasked with mitigating inference-time attacks in secure LLM deployments.

1. Introduction

The unpredictable surge in the adoption of Large Language Models (LLMs) across individual and corporate sectors, particularly their rapid proliferation in high-risk domains such as healthcare [1], law [2], and financial analysis [3], brings forth significant security concerns alongside their inherent benefits. For numerous institutions integrating these models into business processes and individuals incorporating them into their daily lives, the resilience of these models against adversarial attacks has become a critical issue. Early research on LLM security primarily focused on neutralizing harmful content through alignment techniques applied during the training phase [4]. Consequently, the operational reality of deployed systems has shifted the research paradigm toward the inference stage, where systems must defend against a distinct and equally urgent category of adversarial exploits. Unlike training-time vulnerabilities, inference-time attacks do not require access to model weights, training pipelines, or deployment infrastructure. Instead, they operate solely through the standard input–output interface provided during user interactions. The unrestricted nature of this interface, along with increasingly complex Retrieval-Augmented Generation (RAG), tool integration, and multimodal LLM deployments, renders inference-time attacks a paramount security risk.

Our review restricts its scope to on inference-time attacks and the defense strategies developed to mitigate them. The prominence of this category is underscored by the OWASP Top 10 for LLM Applications (2025) [5], the most widely adopted vulnerability taxonomy for production LLM systems: seven of the ten listed risks manifest at inference time, while the remaining three relate to training, supply chain, and infrastructure concerns. Beyond this empirical prevalence, inference-time attacks are particularly critical for three additional reasons. First, they require no access to model weights or training pipelines [6]. Second, as LLMs are increasingly deployed in production systems with real-world consequences, the attack surface grows continuously with each new tool integration and retrieval source. Third, unlike training-time mitigations that require retraining, inference-time attacks can emerge and evolve after deployment, making them harder to anticipate and patch. Training-time and infrastructure threats, while equally important, are addressed in dedicated surveys [7,8]. A dedicated and structured survey that maps the full scope of inference-time threats alongside a systematic taxonomy of corresponding defenses, and that critically evaluates their mutual coverage, trade-offs, and limitations, is currently absent in the literature. The existing survey literature on LLM security, while valuable, does not adequately serve the specific need of a practitioner or researcher seeking to mitigate inference-time attacks. Security surveys such as [9,10,11] catalog the full spectrum of LLM vulnerabilities, including training-time and infrastructure-level threats, but treat mitigation strategies at a coarse granularity and do not offer a dedicated taxonomy of inference-time defenses. As a result, no existing survey jointly maps the inference-time attack surface and systematically evaluates the mitigation strategies developed specifically to counter it, at the level of granularity, currency (2024 onward), and comparative rigor required to guide real-world defense selection. This study aims to fill this gap with the following concrete contributions, each directly oriented toward the mitigation of inference-time attacks.

• LLM vulnerabilities are classified within a three-layered attack surface taxonomy based on the lifecycle stage in which they emerge, advocating for the practical and theoretical priority of the inference-time category.
• An original defense taxonomy organized across three axes is proposed, within which 30 defense mechanisms published from 2024 onwards are systematically examined for their effectiveness against specific inference-time attack families.
• An intersectional comparative analysis covering defense–attack coverage gaps, security-utility-latency trade-offs, and white-box versus black-box applicability is conducted to provide actionable guidance for practitioners regarding defense configuration selection in real-world deployments.
• A deployment-oriented defense framework that translates the comparative findings of this survey into practical defense recommendations for representative LLM deployment scenarios. Unlike existing surveys that primarily compare defense techniques individually, the proposed framework maps deployment constraints to appropriate defense stacks while explicitly highlighting their residual security gaps.

In constructing this review, the temporal scope was deliberately restricted to publications from 2024 onwards, as existing survey literature predominantly covers research conducted prior to this period; given the rapid pace of advancement in LLM security, this boundary was established to address the resulting gap in up-to-date coverage. Research articles on LLM security were collected by searching primary databases for the literature review. From the retrieved results, candidate publications were filtered according to three criteria: (i) the originality of the proposed approach relative to existing methods, (ii) the diversity of the defense technique with respect to those already included in the review, and (iii) the rigor and realism of the evaluation methodology, including reported success rates against established attack benchmarks. Application of these criteria yielded the final set of 30 defense mechanisms systematically analyzed throughout this paper.

The remainder of the paper is organized as follows: Section 2 presents the background regarding the LLM inference pipeline, the attack surface taxonomy, and traditional defense foundations. Section 3 and Section 4 detail inference-time attack families and defense strategies, respectively. Section 5 provides a comparative analysis across coverage, trade-offs, and access settings. Section 6 examines evaluation benchmarks and metrics. Section 7 identifies open challenges and future research directions, and Section 8 concludes the study.

2. Background

2.1. LLM Architecture and Inference Pipeline

Modern LLMs are predominantly built upon the Transformer architecture [12], which relies on a self-attention mechanism to capture dependencies across input tokens. Given query (Q), key (K), and value (V) projections, attention scores are computed as:

$$Attention\left(Q, K, V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V.$$

Here, the term ${QK}^T$ represents the dot-product similarity between query and key vectors. The scaling factor $\sqrt{d_k}$ is used to prevent large values in high-dimensional spaces from pushing the softmax function into regions with extremely small gradients. The softmax operation normalizes these similarity scores into a probability distribution, which is then used to compute a weighted sum over the value vectors V, producing a context-aware representation.

Contemporary LLMs adopt a decoder-only design, consisting of a stack of layers each comprising masked multi-head self-attention and a position-wise feed-forward network (FFN), and are trained autoregressively to predict the next token given all preceding ones. Before any computation, raw text is segmented into sub-word tokens by a tokenizer (e.g., BPE) and mapped to continuous embeddings augmented with positional encodings [13].

At inference time, the pipeline proceeds in two phases [14]. During the prefill phase, all prompt tokens are processed in parallel, producing the initial key-value (KV) cache that stores intermediate attention states for reuse. During the subsequent decode phase, output tokens are generated one at a time: at each step the model attends to the full KV cache and samples the next token, appending it to the sequence until a stopping criterion is met. Prefill is compute-bound; decoding is memory-bound due to the continuously growing KV cache [15].

Production deployments frequently augment this pipeline with RAG, tool-use interfaces, and multi-modal inputs [16]. Each of these extensions introduces additional untrusted input channels, such as retrieved documents, tool outputs, or visual content, that expand the attack surface and motivate the inference-time threat taxonomy developed in Section 3.

2.2. Attack Surface Taxonomy

Attacks against large language models can be systematically organized into three categories according to when and where they occur within the model lifecycle: training and development time attacks, inference-time attacks, and infrastructure and operational attacks [8,17]. This taxonomy is shown in Figure 1.

Training and development time attacks target the model before it is deployed. The attack surface consists of only the dataset and components used for model training at that moment. Data poisoning corrupts the training corpus to skew model behavior, either by flipping labels or by manipulating gradients during optimization. Backdoor and trojan attacks embed hidden triggers into model weights during fine-tuning so that the model behaves maliciously only when a specific input pattern is present [7]. Supply chain attacks exploit the distribution of pretrained model checkpoints, embedding malicious behavior directly into publicly released weights before downstream user’s fine-tune or deploy them. Model/Architecture Manipulation introduces system vulnerabilities by tampering with model architectures, hyperparameters, or training pipelines via compromised training scripts or malicious checkpoints. Annotation/Labeling Attacks systematically mislabel data within crowdsourced or outsourced labeling pipelines to intentionally skew model behavior.

Inference-time attacks exploit the model at the point of user interaction. At this stage, attackers only have a model to which they can provide input. Prompt injection attacks aim to hijack model behavior. These attacks can be direct, where the user submits malicious instructions, or indirect, where untrusted content in retrieved documents or tool outputs carries hidden commands [18,19]. Jailbreaking attacks attempt to circumvent safety alignment through template-based strategies such as roleplay and persona manipulation, optimization-based techniques such as GCG and AutoDAN [20], or multi-turn decomposition that spreads a harmful request across several conversation turns. Adversarial input attacks manipulate the model at the token level via unicode homographs or at the semantic level via meaning-preserving paraphrases [21]. Sensitive Information Disclosure extracts training data, system prompts, proprietary instructions, or Personally Identifiable Information (PII) through crafted queries such as prompt leaking and extraction attacks [22]. Insecure Output Handling exploits downstream systems by leveraging unsanitized LLM outputs to execute generated code directly or perform SQL and command injections. Excessive Agency manipulates LLMs within agentic setups to execute unauthorized actions or escalate privileges by driving connected tools, APIs, or plugins beyond their intended scope. Unbounded Consumption refers to the risk wherein LLM-powered applications allow uncontrolled utilization of computational resources, enabling adversaries to exhaust system budgets through excessive token consumption and GPU-intensive operations with minimal request overhead.

Infrastructure and operational attacks target the broader deployment environment rather than the model’s inference behavior directly. The attack surface is formed by the environment and components in which the model is installed. The model’s operation and maintenance process is also included. Model extraction attacks reconstruct a functional approximation of a proprietary model through repeated querying [23]. Membership inference attacks determine whether a specific data record was part of the training set, and model inversion attacks attempt to recover sensitive training data from model outputs [24]. RAG pipeline poisoning spans two distinct attack vectors. First, offline data tampering occurs when an adversary injects malicious content into the external knowledge base (e.g., a vector database) prior to deployment, which falls under Infrastructure and Operational Time. Second, runtime dynamic manipulation arises when the model retrieves a compromised document during inference, leading to the incorporation of attacker-controlled content (e.g., indirect prompt injection), which falls under Inference Time. Denial-of-service attacks abuse the API to exhaust computational resources.

2.3. Traditional Defense Methods for Inference Time Attacks

Early efforts to defend LLMs against inference-time attacks relied on a small set of lightweight, model-agnostic techniques that operate on the input or output stream without modifying model weights. Rule-based input filtering applies keyword blacklists and regular-expression patterns to reject prompts containing known harmful terms or structures; while computationally inexpensive, this approach is easily bypassed through synonym substitution, paraphrase, or encoding tricks. Perplexity-based detection exploits the observation that gradient-optimized adversarial suffixes often exhibit anomalously high perplexity relative to natural language. It uses a language model’s log-probability estimates to flag suspicious inputs. However, this approach is less effective against semantic-level attacks and LLM-generated jailbreaks, which produce fluent, low-perplexity prompts. Input preprocessing defenses, including paraphrasing, retokenization, and random perturbation, aim to destroy adversarial structure in the prompt before it reaches the target model. However, they can be circumvented once an attacker is aware of the transformation and incorporates it into the attack objective. Output moderation classifiers, such as the OpenAI Moderation API and Llama-Guard, screen generated responses against a set of harm categories and block policy-violating outputs; single-turn filtering of this kind has been shown to fail under multi-turn attacks, in which each individual turn appears benign while the cumulative conversation steers the model toward harmful outputs [25]. Taken together, these methods share a fundamental limitation: they are static and reactive, operating on surface-level signals and remaining brittle against adaptive adversaries who can optimize attacks with knowledge of the defense.

3. Inference-Time Attacks

Although Inference-Time attacks are critically vital, the model has now been refined and deployed to a production environment, leaving the model inputs as the only area where attackers can intervene. While these attacks differ in the techniques they employ, they all are fundamentally based on manipulating the model inputs.

3.1. Prompt Injection Attacks

Prompt injection is ranked LLM01:2025 by OWASP [5], reflecting broad consensus that it constitutes a fundamental architectural vulnerability rather than an implementation flaw. The attack exploits the inability of current LLMs to reliably distinguish between trusted instructions and untrusted user-supplied or externally retrieved content [6]. This type of attack is examined under two headings. In direct prompt injection (DPI), the adversary submits a crafted user turn that overrides or augments the system prompt, for example by instructing the model to ignore prior directives and adopt an alternative persona [18]. In indirect prompt injection (IPI), malicious instructions are embedded in content that the model ingests from external sources such as retrieved documents, visited web pages, processed emails, or tool outputs; the victim need not interact with the attacker directly, since the model executes the hidden commands autonomously when it encounters the poisoned content [26]. The consequences of a successful injection range from unauthorized data exfiltration and system prompt leakage to privilege escalation and the execution of unintended actions in tool-augmented or agentic deployments. Difference between direct and indirect injections is shown in Figure 2. While DPI involves an attacker providing malicious prompts directly as input, IPI occurs when the model is directed to an external web source from which it retrieves the malicious instructions.

3.2. Jailbreaking Attacks

Jailbreaking (JB) refers to a class of attacks that aim to bypass the safety alignment of an LLM and elicit policy-violating outputs, without necessarily hijacking the model’s functional behavior as in prompt injection. Template-based attacks construct hand-crafted prompts that exploit known weaknesses in alignment. Common strategies include role-playing scenarios, such as instructing the model to act as an uncensored persona like “DAN”, hypothetical framings, and cipher or encoding tricks that obscure the harmful intent of a request [27]. Optimization-based attacks automate the search for adversarial inputs: the Greedy Coordinate Gradient (GCG) method [20] appends a gradient-optimized token suffix to a prompt so that the model is steered toward affirmative responses, achieving near-perfect attack success rates on open-weight models; adaptive variants using random search on log-probabilities have similarly achieved 100% success rates across a wide range of frontier models [28]. Multiturn attacks distribute a harmful request across several conversation turns, exploiting the model’s tendency to maintain conversational coherence; agent-driven variants of this strategy, in which a large reasoning model autonomously plans and executes a persuasive dialogue, have been shown to achieve an overall success rate of 97% across nine widely used target models [29].

3.3. Adaptive Attacks

Adaptive attacks (AA) represent a dynamic category of adversarial techniques where the adversary iteratively refines their strategy based on the feedback or defenses encountered during interaction with the model. Unlike static, “one-shot” attacks, adaptive variants are specifically engineered to bypass active defense mechanisms such as input filters, perplexity-based detection, or output sanitizers. These methods often employ gradient-free optimization or random search strategies on log-probabilities to navigate the model’s decision boundaries. Research has demonstrated that even when defensive layers are implemented, adaptive variants can achieve a 100% success rate across a wide range of frontier models by co-evolving with the defense [28]. The primary threat of adaptive attacks lies in their ability to identify “blind spots” in the model’s safety alignment that static benchmarks fail to capture, making them a critical consideration for robust security evaluations [20].

3.4. Adversarial Input Attacks

Adversarial input attacks (AIA) manipulate the surface form of a prompt to alter model behavior while preserving its semantic content, thereby evading detection mechanisms that operate on lexical or syntactic features. Token-level attacks operate on individual tokens: homograph substitution replaces standard characters with visually identical unicode equivalents that map to different token identities, and synonym replacement identifies the most influential tokens in a prompt and substitutes them with semantically similar alternatives that mislead the model while remaining human-readable [30]. Figure 3 illustrates an attack example in which semantically similar alternative words are used. Semantic-level attacks preserve the harmful intent of a request through meaning preserving paraphrase or low-resource language translation, exploiting the uneven safety alignment coverage across languages and linguistic registers [27]. Suffix and prefix attacks append or prepend an adversarially crafted token sequence to a prompt; the GCG method introduced gradient-based suffix optimization and demonstrated that the resulting strings transfer across models, eliciting harmful outputs even on closed-source APIs not used during optimization. Subsequent work on output-prefix attacks showed that enforcing the first few tokens of the model’s response to match an affirmative snippet is sufficient to bypass safety filters at the API level, requiring no gradient access [31]. A unifying property of this attack family is transferability: adversarial inputs optimized against one model frequently retain their effectiveness when applied to other models, amplifying their practical threat [20].

3.5. Information Disclosure Attacks

Information disclosure attacks (IDA) aim to compromise the confidentiality of the LLM’s internal state, its operational constraints, or the data used during its training and fine-tuning processes. This category is primarily bifurcated into prompt leaking and training data extraction. Prompt leaking involves crafting inputs that trick the model into revealing its underlying system prompts, which often contain proprietary logic or sensitive operational boundaries [6]. Training data extraction exploits the model’s propensity to memorize specific sequences from its training corpus; by using targeted prefix probing, attackers can force the model to output PII, confidential documents, or copyrighted material embedded within its weights [32,33]. These vulnerabilities highlight a fundamental tension in LLM deployment: the model’s capacity to retrieve information often fails to distinguish between public knowledge and the sensitive data it was intended to keep private.

3.6. Insecure Output Handling

Insecure output handling (IOH) occurs when an LLM-integrated system or application fails to properly sanitize or validate the model’s generated output before processing or rendering it. This vulnerability is significant because it enables downstream output-related attacks, thereby facilitating broader LLM-based web application exploits. Given that LLMs can generate content that secondary systems inherently trust, the lack of rigorous validation at the destination can lead to severe security compromises. Mitigating this risk requires developers to treat LLM-generated content as untrusted data, ensuring that any output passed to web interfaces, databases, or execution environments is subjected to standard security sanitization processes [34].

3.7. Excessive Agency

Excessive agency (EA) refers to a vulnerability where an LLM is granted too much autonomy, allowing it to perform unauthorized or unintended actions through its integrated tools. This risk is particularly prevalent in AI agent systems where models are empowered to execute multi-step sequences or interact with external APIs. When an LLM possesses excessive agency, it may be manipulated into performing harmful operations, such as unauthorized data access or system modifications, primarily because the application lacks sufficient constraints on the model’s decision-making capabilities [35]. Mitigation requires a comprehensive defense-in-depth approach, specifically utilizing a multi-layered security architecture that includes rigorous tool-registry authentication and credential scrubbing. By enforcing strict boundaries on the agent’s operational scope and requiring explicit authentication for tool execution, organizations can significantly limit the potential damage caused by over-privileged models.

3.8. Unbounded Consumption

Unbounded consumption (UC) is an attack category where an adversary exploits an LLM’s resource requirement to induce system degradation or a denial of service. By forcing the model into excessively long, complex, or repetitive multi-step sequences, attackers can exhaust computational budgets, increase operational latency, or drive-up infrastructural costs. This threat is distinct from other vulnerabilities because it targets the operational sustainability of the system rather than the integrity of its data. The primary mitigation strategy involves implementing strict token-budget controls. These controls function as automated termination mechanisms, halting multi-step sequences once a predefined execution limit is reached, thereby ensuring that the LLM cannot consume indefinite resources. Unlike refusal-based filters, which may be bypassed via adversarial paraphrasing, budget controls remain effective because they enforce deterministic operational limits on the inference process.

4. Defense Strategies Against Inference-Time Attacks

The detection of inference-time attacks against LLMs constitutes a critical research domain for ensuring model security and reliability. While the literature offers various defense mechanisms for this purpose, these approaches diverge significantly in their methodologies. To mitigate the existing complexity and categorize these works, this section introduces a novel taxonomy that contributes to the literature by analyzing 30 distinct research papers published from 2024 onwards. In this study, current defense strategies are examined under three primary headings: prompt-level, inference-time, and training-time interventions. The proposed taxonomy is illustrated in Figure 4. It is important to clarify the precise technical watershed separating these categories. Prompt-level defenses operate as static, input-only filtering mechanisms applied before the text reaches the target model’s core processing loop, whereas inference-time defenses actively monitor or intervene during the autoregressive generation or decoding process itself, for instance by analyzing intermediate activation values or leveraging self-consistency loops.

4.1. Prompt Level Defenses

Prompt-level defenses are based on the analysis and filtering of user-provided inputs either prior to or during processing by the model. The primary objective is to intercept malicious prompts before they reach the model or to sanitize them to neutralize potential threats. This approach not only prevents the unnecessary expenditure of computational resources and time on malicious requests but also mitigates attack scenarios that could compel the model to generate unauthorized or harmful outputs. At this level of defense, there is only a single prompt available for inspection and decision-making, while the model’s subsequent response remains unknown. Consequently, this fundamentally constitutes a text classification problem, and the corresponding solution approaches have been formulated with this awareness.

4.1.1. Machine Learning (ML) Based Classifier

ML techniques are frequently employed in text classification tasks. For classification, a comprehensive dataset containing both malicious and benign prompts is first constructed. This text-based dataset is then transformed into a vector representation to enable algorithmic processing, after which a classification model is trained. For word vectorization, techniques such as TF-IDF, Word2Vec, and N-grams are commonly utilized [36]. To determine whether incoming prompts are malicious, traditional ML algorithms such as Random Forest (RF), XGBoost, or Logistic Regression (LR) are employed [37]. These approaches are widely adopted in the literature and are relatively lightweight compared to deep learning-based methods, offering faster inference times and making them suitable for real-time detection. However, these methods also have certain limitations. In particular, since they rely on static or predefined patterns, they struggle to detect complex and obfuscated attacks or emerging threats such as indirect prompt injection, often resulting in higher false negative rates [38].

4.1.2. Embedding Based Classifier

Text representations are not only used for classification with ML algorithms but also for capturing the relationship between malicious prompts and user inputs. Studies adopting this approach have transformed text-based inputs into dense vector representations using models such as OpenAI text-embedding, GTE-large, or MiniLM, and subsequently classified these vectors [39,40]. Vector databases and similarity searches have been utilized to detect anomalies by analyzing the semantic relationship between user intent and external content [41]. This approach is significantly more robust against surface-level lexical variations (e.g., the use of synonyms) and achieves high accuracy by capturing the linguistic context. Furthermore, it is model agnostic as it requires no modifications to the underlying LLM architecture; however, it necessitates greater computational overhead compared to traditional ML methods. Another limitation is that its detection capabilities remain confined to predefined attack vectors.

4.1.3. Pre-Processing

In contrast to standard pre-processing that optimizes text inputs, this defensive strategy transforms the input prior to model submission by intentionally degrading potentially malicious patterns. It encompasses retokenization, paraphrasing, masking, the sanitization of invisible characters and Unicode manipulations [42], and structural cleaning techniques such as CCFC (Core & Core-Full-Core) that isolate the primary semantics of the prompt [43]. While the method is model agnostic, requires no additional training, and maintains low computational overhead, it may inadvertently distort the input’s original meaning, thereby compromising the model’s utility or response quality in benign tasks. Moreover, multi-perturbation methods requiring supplementary processing stages can escalate inference latency and operational costs.

4.1.4. LLM Based Classifier

Studies adopting this approach utilize auxiliary language models, such as Llama or GPT, acting as an ‘LLM-as-a-judge’ to audit prompt security [44,45,46]. These models are specifically fine-tuned to assess whether an input or output is malicious based on predefined safety policies [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]. Furthermore, known-answer detection and game-theoretic detection algorithms also leverage external LLMs to identify anomalies within the prompts [48]. By harnessing the sophisticated contextual understanding and reasoning capabilities of these models, this approach offers superior detection efficacy against both direct and indirect injections. However, employing an external large language model for detection purposes entails significant operational costs and increased latency.

4.1.5. Hybrid Classifier

Hybrid classifiers, deeming the use of individual prompt-level detection methods as sole decision-makers insufficient, adopt ensemble learning approaches based on consensus or majority voting mechanisms. For instance, Sentra-Guard integrates language translation, SBERT embedding-based semantic search (FAISS), fine-tuned transformer classifiers, and zero-shot LLM inference [49]. Similarly, SHIELD employs a multi-layered architecture comprising rule-based algorithms, machine learning, and LLMs [37]. These studies rely on the principle that distinct security layers compensate for each other’s shortcomings; consequently, simple known attacks are rapidly filtered, while sophisticated attacks are identified through semantic or behavioral analysis [36,37,38]. However, the system architecture of this approach is highly complex. Running multiple models concurrently or sequentially escalates maintenance overhead, integration challenges, and hardware requirements. Figure 5 illustrates a generalized high-level prompt-level hybrid classification framework. The prompt is classified using multiple approaches and then subjected to a voting process for the final decision, after which it is assigned, a label based on the collective outcome.

4.2. Inference-Time Defenses

The inference phase encompasses the generation process, as well as the intervals immediately preceding and succeeding output production. Within this stage, defensive measures are implemented either by leveraging the model’s intrinsic capabilities to ensure cost-efficiency or by employing a multi-layered framework to analyze the impact of the prompt on the model’s internal behavior. At this stage, the model output, in addition to the input prompt, becomes available for further analysis.

4.2.1. Self-Consistency

Self-Consistency approach aims to foster self-reflection, self-evaluation, or step-by-step reasoning within the model’s generation process. Defenses such as PARDEN identify inconsistencies in malicious outputs by requiring the model to repeat its responses [50]. While Chain of Thought (CoT) methodologies encourage the model to engage in self-deliberation and reflection before arriving at a decision [51], proxy evaluator agents are employed to filter potentially risky responses or mislead attackers with decoy outputs before they are generated [52]. By utilizing the inherent reasoning and instruction-following capabilities of LLMs without requiring external security models, additional hardware, or further training, this approach enables cost-efficient defense implementation. However, as it involves multiple generation stages or redundant outputs (e.g., PARDEN), it increases token consumption and extends response latency.

4.2.2. Input/Output Classification

These consist of dynamic approaches that concurrently or through a multi-layered framework verify both the model’s inputs and its directly generated outputs [53]. Representative examples include LeakSealer, which audits potential PII leakage in the output [54], and Llama Guard integration [47]. Methods such as Attn-Defense monitor attention weights during inference to detect input anomalies, aiming to intervene by pinpointing the concentration of the attack [55]. The most distinguishing feature of this approach is its ability to intercept the final output before it reaches the user, even if an indirect prompt injection attack successfully compromises the generation. Furthermore, methods analyzing attention weights enable granular sanitization of toxic instructions while preserving the user’s original intent. Conversely, output analysis necessitates that the model has already completed the generation process. Additionally, operations based on internal states, such as attention mechanisms or activation analysis under white-box access, typically require open-source or weight-accessible models.

4.3. Training-Time Defenses

Defensive strategies at this stage aim to render models fundamentally resilient against attacks by directly modifying their weights through techniques such as fine-tuning or reinforcement learning. In this paradigm, the model lacks any intermediate layers or external controllers at either the input or output stages; instead, the model directly confronts the adversary. Enhancing the model’s intrinsic resistance enables it to maintain its own security independently, bypassing the need for supplementary defense mechanisms.

4.3.1. Model Unlearning

This methodology focuses on the systematic removal (unlearning) of harmful information and undesirable behavioral patterns from large language models. For instance, SafeLLM monitors malicious content at the level of FFN activations and irreversibly disables harmful knowledge pathways through targeted optimizations. Because these behaviors are purged from the model’s parameters and attention pathways, the model becomes exceptionally resilient even to sophisticated adaptive jailbreak attacks, thereby establishing permanent security [56]. While this approach constructs a robust defensive barrier, it presents a significant caveat: overly aggressive unlearning processes during the training phase may compromise the model’s general fluency and its performance in benign domains.

4.3.2. Adversarial Training

Adversarial Training entails augmenting the training dataset with malicious and deceptive prompts to optimize the model toward generating accurate and secure responses. By leveraging continuous and discrete attacks derived from algorithms such as GCG or AutoDAN, models are hardened to achieve greater robustness (e.g., CAT or R2D2 algorithms) [57]. Furthermore, the ReFAT algorithm enables more efficient adversarial training by simulating self-deception scenarios through the targeted suppression of refusal features [58]. Transformer architectures are then fine-tuned on this data for specialized tasks [59]. This methodology provides the model with proactive and inherent robustness against both encountered and novel attack vectors, enabling the identification of malicious intent at an intrinsic level. However, generating adversarial examples and optimizing these workflows necessitates substantial computational overhead. Additionally, these defenses remain susceptible to circumvention if they fail to generalize across increasingly diverse attack patterns.

5. Comparative Analysis

Comparing the defense approaches proposed in the literature is of paramount importance for selecting optimal defense methodologies. In particular, this comparison should consider their interrelationships, efficacy against various attack vectors, security-utility-latency trade-offs, and applicability across different model access levels such as white-box and black-box settings. The comprehensive analysis presented in this section constitutes one of the key contributions of this study to the existing literature. Building on this analysis, Section 5.1, Section 5.2 and Section 5.3 are subsequently synthesized in Section 5.4 into a unified decision framework for practitioners.

5.1. Defense-Attack Coverage Matrix

No single defense strategy for LLMs provides exhaustive protection against the entire spectrum of attack vectors.

Table 1. Attack coverage matrix: Defended inference-time attacks across evaluated literature.

Publication (Year)	Defense Method	DPI	IPI	JB	AA	AIA	IDA	IOH	EA	UC
Hu et al. (2025) [43]	Input Pre-processing	•	•	•		•	◦
Pingua et al. (2024) [41]	Embedding Based Classifier	•		•		•	◦
Ayub & Majumdar (2024) [40]	ML Based Classifier	•		•		◦	◦
Galinkin & Sablotny (2024) [60]	ML Based Classifier	•		•		◦	◦
Alamsabi et al. (2026) [39]	ML Based Classifier	•		•	◦	◦	◦
Panebianco et al. (2025) [54]	ML Based Classifier	•		•		◦	◦
Thaqi et al. (2025) [46]	ML Based Classifier	•		•		◦	◦		◦
Aswin & Ilavendhan (2025) [47]	ML Based Classifier	•		•		◦	◦
Jacob et al. (2025) [44]	LLM Based Classifier	•		•		◦	◦
Zhang et al. (2025) [55]	LLM Based Classifier	•		•		◦	◦
Tien & Van Huong (2024) [59]	LLM Based Classifier	•		•		◦	◦
Pan et al. (2025) [45]	LLM Based Classifier	•		•		◦	◦
Esugo et al. (2025) [37]	Hybrid Classifier	•		•		•	◦
Hasan et al. (2025) [49]	Hybrid Classifier	•		•	◦	◦	◦
Kokkula (2024) [36]	Hybrid Classifier	•		•		•	◦
Mishra et al. (2025) [42]	Hybrid Classifier	•		•		•	◦
Lan et al. (2025) [61]	Hybrid Classifier	•		•		•	◦
Cao et al. (2024) [51]	Self-consistency	•		•		◦
Lin et al. (2025) [62]	Self-consistency	•		•		•
Zhang et al. (2024) [50]	Self-consistency	•		•		◦
Lan et al. (2025) [63]	Input/Output Classification	•	◦	•	◦	◦	•	•
Liu et al. (2025) [48]	Input/Output Classification	•	◦	•	•	•	•	•
Prakash et al. (2026) [38]	Input/Output Classification	•	•	•	•	•	•	•	◦	◦
Schwarz (2025) [53]	Input/Output Classification	•	•	•	◦	•	•	•
Zhao et al. (2026) [52]	Input/Output Classification	•	◦	•	•	◦	•	•
Zhao & Li (2025) [64]	Input/Output Classification	•	•	•	◦	•	•	◦	◦	◦
Li et al. (2025) [56]	Model Unlearning	•		•		◦	◦
Wang et al. (2025) [10]	Adversarial Training	•		•	•	•	◦
Xhonneux et al. (2024) [57]	Adversarial Training	•		•	•	•	◦
Yu et al. (2025) [58]	Adversarial Training	•		•	◦	•	◦

(•) completely addressed, (◦) partially addressed, and (blank) completely unaddressed. Abbreviations: DPI: Direct Prompt Injection, IPI: Indirect Prompt Injection, JB: Jailbreaking, AA: Adaptive Attacks, AIA: Adversarial Input Attacks, IDA: Information Disclosure Attacks, IOH: Insecure Output Handling, EA: Excessive Agency, UC: Unbounded Consumption.

illustrates the specific attack vectors targeted by each defense methodology, categorized according to the taxonomy proposed in this study. Based on the proposed taxonomy, the analyzed studies target substantially similar attack vectors, a finding that validates the robustness of the proposed taxonomic approach. A comprehensive analysis of the matrix yields significant insights regarding the coverage areas of the respective defense mechanisms. Primarily, all evaluated defense approaches encompass DPI and Jailbreaking attacks. Because prompt level defenses exclusively analyze input data, they fail to mitigate AA, IOH and IDA attacks. Conversely, inference-time defense approaches, particularly studies utilizing Input/Output Classification, facilitate bidirectional detection and thus enable the identification of AA, IOH and IDA attacks. Furthermore, since the detection of IPI attacks requires an external detection layer, such threats can only be identified within inference time defenses.

The fact that EA and UC are largely ignored in most existing studies, and only partially addressed in a few, reveals a critical research gap in the literature and presents clear avenues for future research. Rather than stemming from a lack of standardized evaluation metrics, these two gaps arise from a more fundamental issue. Specifically, existing LLM defense research has focused predominantly on model-level threats, leaving system-level and agent-level vulnerabilities largely unaddressed. Consequently, future research must transcend the model itself to establish a holistic defense framework that encompasses both the agent orchestration layer and the deployment infrastructure.

5.2. Trade-Offs: Security, Utility, Latency

For a defense system to be practically viable, a delicate equilibrium between security, model utility, and latency must be maintained. Overly stringent security measures often lead to the over-defense phenomenon, where LLMs inadvertently reject benign and legitimate user requests [44]. For instance, excessively aggressive model unlearning techniques can result in catastrophic forgetting, thereby compromising the model’s general-purpose capabilities. The utility degradation problem, arising from training procedures designed to strengthen the defensive capabilities of the model, has been evaluated only in research adopting training-time defense strategies among the surveyed works. Across these studies, the reported performance degradations are stated as 4% [10], 4.8% [56], and 5% [58], respectively. Consequently, to preserve utility performance, methods such as ReFAT, which meticulously optimize refusal features, are preferred.

Latency remains a critical bottleneck in real-time LLM deployments. Training-time defenses do not introduce any additional evaluation layer to the model and therefore incur no inference latency. However, the situation differs for other categories of defenses. In general, prompt-level defenses tend to impose lower latency compared to inference-time defenses, as they typically rely on a single-pass evaluation. Nevertheless, hybrid classifiers may constitute an exception in certain studies, particularly when they do not employ parallel evaluation mechanisms. The approaches that introduce the highest latency are those performing classification on both the input and output channels.

Table 2. Latency comparison of defense approaches.

Paper	Defense Approach	Latency (s)
[39]	ML Based Classifier	0.006
[46]	ML Based Classifier	0.22
[53]	Input/Output Class.	0.45
[49]	Hybrid Classifier	0.47
[48]	Input/Output Class.	1.5
[63]	Input/Output Class.	1.9

presents the latency of the examined approaches, categorized by their classification strategies, reported in seconds. Furthermore, invoking large-scale LLM-based classifiers via APIs for every input results in an exponential increase in operational expenditures [46]. To mitigate this issue, hybrid architectures employ low-cost rule-based filters and machine learning algorithms as primary defenses, activating resource-intensive LLM-based controllers only in suspicious scenarios to achieve optimal cost and performance efficiency [37,38].

5.3. White-Box, Black-Box Setting Efficacy

The efficacy of defense mechanisms is directly contingent upon the level of access the defender has to the target model. White-box defenses pertain to scenarios where the defender possesses full access to the LLM’s weights, internal activations, and attention mechanisms. According to our proposed taxonomy, Training-Time defenses fall within this category. However, implementing such defenses in closed-source commercial cloud environments is unfeasible. Conversely, black-box defenses are those where the defender only has access to the model’s inputs and outputs, allowing them to function in a model-agnostic manner. In our taxonomy, Prompt-Level and Inference-Time defenses belong to this category. Although black-box defenses are remarkably easy to deploy, they are more susceptible to being bypassed via novel obfuscation techniques and adaptive attacks compared to white-box methodologies [48].

A notable caveat arises within this classification. A subset of Inference-Time defenses, particularly state-monitoring approaches relying on intermediate activation values or attention maps, effectively require white-box-level access despite their black-box classification. Since most production systems rely on proprietary, closed-source APIs such as OpenAI and Anthropic, where such internal information remains inaccessible, these defenses are rendered unfeasible in commercial settings, and their utility is largely restricted to open-source models.

5.4. Deployment-Oriented Defense Framework

While Section 5.1, Section 5.2 and Section 5.3 compare defense mechanisms from complementary analytical perspectives, practitioners ultimately require guidance on how these techniques should be combined under different operational constraints. Existing survey papers typically evaluate defenses individually but rarely translate these observations into deployment recommendations. Based on the comparative analyses presented throughout this review, we derive a deployment-oriented defense framework that maps common LLM deployment scenarios to appropriate defense stacks while explicitly identifying their remaining security limitations.

To bridge the gap between theoretical attack taxonomies and practical system design,

Table 3. Deployment-oriented defense framework for secure LLM systems.

Deployment Scenario	Deployment Constraints	Baseline Defense	Escalation Layer	Design Rationale	Residual Gaps
Public Customer Chatbot	Closed-source API; low latency; high user traffic	ML classification or embedding-based detector	LLM-as-a-Judge for suspicious prompts	Direct prompt injection and jailbreak attacks dominate public-facing systems. Lightweight filtering provides an effective balance between security, latency, and operational cost.	Adaptive prompt injection, indirect prompt injection, and output manipulation remains partially unresolved.
Enterprise RAG Assistant	Retrieved documents; moderate latency acceptable	Embedding validation and prompt classifier	Input/Output classifier	Retrieved documents introduce indirect prompt injection risks that cannot be detected solely through prompt inspection. Multi-stage validation improves robustness.	Tool misuse and orchestration-layer attacks require external access control mechanisms.
Internal Enterprise Assistant	Higher security; moderate latency	Hybrid detector (Rule + ML + Embedding)	LLM-based evaluator	Enterprise environments process confidential information, making higher computational overhead acceptable in exchange for stronger semantic attack detection.	Previously unseen attack strategies may bypass static detection models.
Self-Hosted Open-Source LLM	White-box access Available	Adversarial training or model unlearning	Activation-based runtime monitoring	White-box access enables intrinsic robustness techniques that are unavailable for proprietary APIs. Runtime monitoring complements training-time defenses.	System-level vulnerabilities and external tool abuse remain outside model-level protection.
High-Security Deployments	Security prioritized over latency	Hybrid defense pipeline	Human approval and runtime monitoring	False negatives are substantially more costly than increased latency. A defense-in-depth architecture provides the highest level of resilience.	Zero-day attacks and sophisticated semantic jailbreaks require continuous monitoring and policy updates.
Edge/On-Device Local LLMs	Low memory/compute availability; offline operation; sub-second latency	Lightweight ML classifier or on device rule-based regex engine	Quantized LLM-as-a-Judge for suspicious prompts	Edge deployments cannot offload safety checks to heavy external cloud LLMs due to latency and connectivity constraints.	Complex semantic obfuscation and adaptive token-level attacks easily bypass low-footprint classifiers.
Agentic LLM Systems	Autonomous tool execution; external APIs	Prompt validation and input/output monitoring	Tool permission control, sandboxing, and credential isolation	Security failures increasingly originate from autonomous tool execution rather than prompt manipulation alone. Therefore, model-centric defenses must be combined with system-level security controls.	Excessive Agency and orchestration-layer attacks remain difficult to mitigate using model-level defenses alone.

presents a context-aware defense framework tailored to real-world LLM deployment scenarios. Rather than advocating a one-size-fits-all security mechanism, the framework maps distinct operational contexts, which range from high-traffic public chatbots to autonomous agentic systems, against their specific deployment constraints, primary threat vectors, and trade-off requirements. To achieve robust defense-in-depth, each scenario prescribes a two-tier architectural strategy comprising a lightweight Baseline Defense for initial filtering and an escalated Escalation Layer for deep semantic or state monitoring. Furthermore, by explicitly delineating the Design Rationale and identifying the unresolved Residual Gaps for each configuration, this table provides practitioners with an actionable decision matrix while highlighting where supplementary system-level orchestration controls (e.g., tool sandboxing, privilege boundaries, and token budgets) must compensate for model-centric limitations.

6. Benchmarks and Evaluation

Accurately quantifying the efficacy of attacks against LLMs and the defensive mechanisms developed to counter them constitutes a fundamental cornerstone of AI security research. This section examines the datasets, evaluation metrics, and inherent limitations of current evaluation methodologies utilized in the reviewed literature, while also addressing the reliability of LLM-based judges.

6.1. Current Datasets and Benchmarks

In research conducted on LLM security, the requirement for standardized datasets and benchmarks is of critical importance for the objective comparison of proposed defense mechanisms, the assurance of reproducibility, and the evaluation of models across both security and utility axes. Researchers utilize purpose-built datasets to systematically examine model behaviors against various attack vectors. It is essential for researchers to quantitatively measure the extent to which their developed defenses protect the model from malicious exploitation (e.g., via the Attack Success Rate—ASR). For this purpose, diverse test scenarios are required. Datasets such as AdvBench, HarmBench, and JailbreakBench are employed to evaluate how effectively defense mechanisms and red-teaming strategies can thwart targeted jailbreak attacks within a standardized and comprehensive framework [10,43,52,58].

Beyond direct user inputs, a distinct data structure is necessary to understand how application-integrated and RAG-based systems can be manipulated by data retrieved from the external world (e.g., web pages, documents, and emails). Researchers leverage datasets such as deepset/prompt-injections and BIPIA to measure the detection performance of obfuscated instructions that redirect the model toward unintended actions or lead to data exfiltration [39,47,62].

One of the most significant challenges in the security domain is the potential degradation of the model’s core capabilities (mathematics, coding, logical reasoning) or the inadvertent blocking of benign requests during the hardening process. Consequently, it is mandatory to test whether defenses adversely impact model utility. To this end, standard language and capability benchmarks such as MMLU, MT-Bench, and AlpacaEval are central to studies aiming to quantify performance loss following adversarial training [56,57,58]. Furthermore, datasets such as XSTest are specifically utilized by researchers to test prompts that appear harmful but are actually safe, thereby calibrating the false positive rates and sensitivity settings of defense mechanisms.

Table 4. Datasets used in defense models.

Dataset	Papers	Definition
AdvBench [20]	[10,43,50,52,56,58]	A benchmark dataset consisting of adversarial prompts designed to test LLM security by forcing models to generate harmful content.
HarmBench [65]	[10,49,52,58,63]	A safety evaluation dataset designed to measure the defense performance of LLM’s against harmful requests across various categories.
ToxicChat [66]	[45,54,60]	A safety dataset compiled from user-chatbot interactions, designed to detect toxic content and abusive language to which AI systems are exposed.
deepset/ prompt-injections [67]	[36,38,47,61,62]	A classic classification dataset designed to detect direct and indirect prompt injection attacks.
JailbreakBench [68]	[10,49,52]	A safety evaluation dataset established to test Jailbreak attacks and the defense mechanisms developed against them.
DAN [69]	[51,60]	A pioneering Jailbreak dataset template in the literature, imposing a fictional persona on the model that exempts it from any restrictions.
BIPIA [39]	[39]	A pioneering dataset designed to detect and analyze indirect prompt injection attacks aimed at exploiting model security.
AttnDefence [70]	[55]	A safety dataset designed to detect attacks that specifically manipulate the attention mechanism and to evaluate defense methods.
Alpaca/AlpacaEval [71]	[44,56,62]	A pioneering fine-tuning dataset consisting of instruction-response pairs, used to endow LLM’s with instruction-following capabilities.
MMLU & MT-Bench [72]	[57,58]	Benchmark datasets containing multiple-choice questions in academic and professional subjects, designed to evaluate the general knowledge and problem-solving abilities of language models.
SST2 [73]	[48,62]	A classic NLP dataset consisting of single-sentence movie reviews, widely used to measure how accurately sentiment analysis models can distinguish positive or negative sentiment in text.

illustrates the datasets used in the defense studies reviewed in this work, along with their respective frequencies of utilization.

6.2. Evaluation Metrics

The most frequently utilized metric for measuring the efficacy of attacks against LLMs and the corresponding defensive mechanisms is the ASR, which is employed for security and vulnerability assessments. ASR represents the percentage of malicious prompts that successfully generate the intended illicit output, and the primary objective of defenses is to drive this rate toward zero. Particularly within training-time defenses, model unlearning and adversarial training approaches have demonstrated superior performance in radically reducing ASR against complex jailbreak attacks [56,57,58]. In the context of inference-time interventions, attention-based methods providing targeted mitigation, such as Attn-Defense, and pre-processing defenses like CCFC stand out as prominent strategies that successfully minimize overall ASR without altering the core model parameters [43,55].

For classification and detection-oriented defense approaches, fundamental machine learning metrics, such as Accuracy, Precision, Recall, and F1-Score, are preferred to evaluate the performance of algorithms designed to filter malicious prompts and data leaks. Studies indicate that the combination of embedding-based classifiers with tree-based machine learning algorithms, such as XGBoost or Random Forest, yields exceptionally high results in terms of F1-Score and Accuracy (e.g., a 97.7% F1-Score) [39,40]. Similarly, Hybrid Classifiers (e.g., Sentra-Guard and SHIELD) that integrate multiple techniques and semantic analyses achieve near-perfect results in F1-Score and Recall metrics against complex and obfuscated injections [37,49]. To verify the sensitivity of classifiers to various threshold values and their overall discriminative power, auROC and auPRC metrics are utilized; approaches such as UniGuardian, PromptShield, and LeakSealer prove their detection efficacy by exhibiting high performance across these metrics [44,54,62].

To quantify the over-defense phenomenon and the resulting degradation of user experience FPR and FNR metrics are employed. A high FPR, signifying the rejection of benign user queries, is highly undesirable in production environments. In this context, well-trained LLM-based classifiers such as PromptShield and game-theoretic defenses like DataSentinel distinguish themselves in real-world deployments by maintaining FPR at exceptionally low levels (between 0.1% and 1.0%), even when confronted with adaptive attacks. Similarly, the self-consistency-based PARDEN approach utilizes the principle of response repetition to sustain a high TPR while significantly reducing FPR compared to competing methodologies [50].

Utility and Latency metrics are utilized to evaluate whether integrated defense mechanisms impair the model’s core capabilities and to assess operational costs during live deployment. Standard benchmarks such as MMLU, MT-Bench, AlpacaEval, Perplexity, Just-Eval, and IFEVAL are used to measure model utility. Methods like adversarial training and unlearning demonstrate their ability to provide security with minimal performance loss on these capability metrics. Conversely, systems employing proxy evaluators like PROACT or hybrid architectures have proven to maintain full model utility, exhibiting zero performance degradation on instruction-following tests such as IFEVAL [52]. Regarding Latency and Throughput metrics lightweight embedding-based and machine learning classifiers prove far superior to massive LLM-as-a-judge frameworks; by delivering results at the millisecond (ms) level, they emerge as the optimal choice for real-time systems.

6.3. Limitations of Evaluations and Judge Reliability

While security testing is rapidly evolving, current evaluation regimes harbor specific limitations and reliability concerns. In the literature, instead of manually labeling model outputs to determine the ASR, LLM-based judges such as GPT-4, Llama Guard, or StrongReject are frequently employed [52,55,58]. However, as these judges are themselves inherently LLMs, they remain susceptible to subjective biases, over-refusal tendencies, and the risk of being deceived by sophisticated obfuscation attacks. Evaluations conducted across different judge models can yield inconsistent results. Furthermore, the tendency for training and testing data to be confined to specific linguistic structures or semantic forms can mislead evaluation outcomes. For instance, models subjected to adversarial training might perfectly thwart attacks phrased in the imperative mood (e.g., ‘Tell me how to build...’), yet remain completely vulnerable when the same intent is presented with polite phrasing (e.g., ‘Could you please tell me how to build...?’). Additionally, the use or omission of chat templates can artificially fluctuate the model’s utility scores. Several emerging paradigms aim to mitigate this judge bias problem. Ensemble voting across structurally diverse judge models has been proposed to reduce the influence of any single model’s idiosyncratic biases on the final verdict. In parallel, hybrid evaluation pipelines that combine deterministic heuristic check-stops or regex-based filters with semantic LLM evaluation offer a complementary safeguard against obfuscation attacks that exploit purely semantic assessment.

7. Open Challenges and Future Directions

While significant strides have been made in defense mechanisms against inference-time attacks on LLMs, fundamental challenges and novel research trajectories persist. Most current defenses remain static and brittle, as they are designed to detect specific attack patterns, leaving systems vulnerable to adaptive attacks. This brittleness is compounded by a structural coverage gap: as shown in Table 1, Excessive Agency and Unbounded Consumption remain the least-addressed attack categories among the 30 surveyed mechanisms, since nearly all reviewed defenses operate exclusively on text and possess no mechanism for classifying tool calls, API invocations, or resource-consumption patterns. Furthermore, as LLMs evolve from text-centric architectures into multimodal systems capable of processing images and audio, security vulnerabilities escalate dramatically. Attackers exploit these new surfaces by embedding malicious instructions typographically within images or concealing them within audio files to bypass conventional text filters. This new attack surface can render existing text-oriented detection mechanisms entirely obsolete. Consequently, there is a critical need for future architectures to incorporate Multimodal Input Sandboxes capable of isolating and analyzing inputs to detect cross-modal threats before they reach the core model.

Agentic LLMs, which do not merely generate text but also orchestrate external tools, APIs, and autonomous operations, elevate security threats to a new dimension. In these systems, a successful prompt injection can lead to high-level vulnerabilities that directly jeopardize system infrastructure, such as Remote Code Execution (RCE), Server-Side Request Forgery (SSRF), or SQL Injection [61]. Moreover, multi-agent systems face ‘prompt infection’ scenarios, where a compromised agent propagates malicious instructions to other agents, leading to a cascading system failure [53]. Therefore, new defense models are required to implement dynamic privilege restriction for tool access to mitigate indirect prompt injections originating from external data sources, such as web searches.

A fundamental philosophical and practical challenge in AI safety is the inherent trade-off between Security and Model Utility. Overly rigid alignment against attacks often causes the system to erroneously reject benign requests or results in a degradation of reasoning quality. This trade-off is not merely theoretical. Section 5.2 reports a consistent average utility degradation of approximately 5% across the training-time defenses. The theoretical solution to this problem lies in a deeper understanding of security parameters within the model’s semantic space. Defining the mathematical boundaries for achieving maximum protection without sacrificing the utility and fluency of LLMs remains the ultimate objective of secure artificial intelligence research.

8. Conclusions

This study systematically examines inference-time attacks against LLMs, and the defense strategies developed to counter them, presenting a structured dual taxonomy that organizes both threat vectors and mitigation methods along principled axes. By analyzing 30 defense mechanisms published since 2024, it is demonstrated that while the field has made significant progress in terms of scope, several persistent structural limitations remain. Comparative analysis of the defensive side shows that no single mechanism can provide comprehensive protection across the entire attack surface. Prompt-level classifiers offer rapid and model-agnostic filtering; however, they exhibit vulnerability to semantic obfuscation and adaptive evasion. Inference-time defenses, such as self-consistency methods and attention-based monitors, add a dynamic and context-aware layer of protection without requiring access to model weights, yet this protection introduces measurable latency and token costs. Training-time interventions, particularly adversarial training and model unlearning, embed resilience directly into model parameters and provide the strongest empirical resistance against complex attacks; nevertheless, they impose a significant computational burden and risk degrading general-purpose capabilities if not meticulously calibrated. Our Defense-Attack Coverage Matrix (Table 1) clearly reveals that robust deployment requires layered and complementary defenses rather than reliance on any single strategy.

The balance between security and model utility remains the fundamental unresolved tension in the field. Overly aggressive defenses consistently lead to the phenomenon of over-refusal, degrading user experience and downstream task performance. A trade-off is that this review quantifies at an average utility degradation of approximately 5% across the training-time defenses that reported this metric, underscoring that mitigating inference-time attacks cannot be evaluated in isolation from the utility cost it imposes. The theoretical resolution of this tension is regarded as the most fundamental open problem in LLM security research. Looking ahead, two research directions are particularly prominent. First, extending existing defense taxonomies to multimodal inputs is of critical importance; as visual and audio channels introduce entirely new attack surfaces, text-oriented detection mechanisms require a fundamental redesign rather than simple adaptation. Second, the security of agentic LLM deployments, where a successful inference-time attack can trigger real-world actions with irreversible consequences, requires specialized threat models and privilege-restriction architectures that current frameworks do not sufficiently address, as reflected in the persistently limited coverage of Excessive Agency and Unbounded Consumption across the surveyed literature. Third, the pronounced latency gap between lightweight prompt-level classifiers and the broader-coverage bidirectional Input/Output Classification methods motivates further development of cascaded, escalation-based architectures that reserve costly inference-time analysis for inputs already flagged as suspicious. In summary, the inference-time attack surface will continue to expand in proportion to the capabilities and deployment scope of LLMs. Sustainable progress in this domain requires moving beyond the paradigm of designing defenses against known attack patterns toward architectures capable of reasoning about adversarial intent at the semantic and representational levels. It is anticipated that the taxonomies, analyses, and open challenges presented in this review will provide a useful foundation for this transformation.