Protecting User Privacy in LLM Reasoning: A Textual Differential Privacy Framework for Context-Aware Applications ^†

Abstract

The growing adoption of context-aware reasoning in Large Language Model (LLM)-based applications, particularly through retrieval-augmented and prompt-enriched architectures, has substantially broadened the utility of LLMs in specialized, real-world scenarios. Yet this progress comes with an underexplored vulnerability: users must supply sensitive contextual information as part of the reasoning process, exposing private data to external LLM service providers and raising serious concerns about informational self-determination. Existing privacy-preserving approaches in NLP are largely ill-suited to this setting, as they either degrade reasoning quality unacceptably or fail to provide formal privacy guarantees. This paper addresses this gap by introducing Textual Differential Privacy (TDP), a principled framework that embeds differential privacy guarantees directly into the textual context supplied to LLMs, enabling privacy-preserving reasoning without sacrificing the semantic coherence that models depend upon. Central to the framework is a Differential Embedding Substitution mechanism, which perturbs sensitive entities in the input space while preserving the structural and semantic properties necessary for downstream reasoning tasks. The framework further introduces a formal privacy loss quantification scheme, providing practitioners with interpretable metrics to navigate the privacy–utility trade-off in deployment. We prove that the mechanism, instantiated as an exponential mechanism over a semantically constrained candidate set, satisfies metric differential privacy with a privacy budget that scales with the cosine distance between entities, and empirical evaluation across context-aware reasoning benchmarks validates its practical effectiveness. By treating privacy as a first-class design principle in LLM-based architectures, this work contributes toward a more trustworthy and broadly deployable foundation for intelligent, context-sensitive AI systems.

1. Introduction

In recent years, generative artificial intelligence has garnered significant interest from the public and the research community. Central to this transformative advancement are Large Language Models (LLMs), which have emerged as universal language processors capable of encapsulating and comprehending broad swaths of human knowledge [1]. LLMs demonstrate proficiency across a spectrum of language-understanding tasks, including summarization, translation, question answering [2], and complicated tasks, such as code vulnerability detection [3] and network configuration [4].

Despite this considerable potential, a discernible gap remains in the out-of-the-box performance of LLMs, particularly in zero-shot or few-shot learning scenarios tailored to specialized use cases. Zero-shot learning often suffers from low and unreliable performance, while few-shot learning necessitates the identification of optimal discrete prompts, a nontrivial process that poses significant challenges. To alleviate these limitations, a variety of strategies have been proposed, including model fine-tuning, prompt learning, and context-aware frameworks. Notably, Retrieval-Augmented Generation (RAG) [5] has emerged as a prominent approach that serves as the foundation for developing context-aware reasoning applications powered by LLMs. In essence, RAG represents a prompt engineering technique that retrieves contextual documents from an external dataset to orchestrate prompts, thereby enhancing the quality of responses within specialized domains [6].

Ideally, such a personalized or customized language model framework should prevent LLMs from learning any sensitive data, such as personally identifiable information (PII). However, within the architecture of context-aware reasoning, users are required to transmit private information as context to the LLM service provider, potentially posing significant threats to personal privacy [7]. Recent studies have further demonstrated that LLMs can memorize and later regurgitate sensitive information embedded in prompts and training data [8,9,10,11,12]; Lukas et al. [13] formalized and empirically demonstrated PII leakage via black-box extraction, inference, and reconstruction attacks against fine-tuned LMs; Kandpal et al. [14] further showed that user-level inference attacks succeed with high probability on models fine-tuned from user data; and Mireshghallah et al. [15] demonstrated that even state-of-the-art LLMs such as GPT-4 violate contextual-integrity privacy norms in a non-trivial fraction of interactions. Taken together, these results establish that sensitive information exposed to an LLM, whether via training data or a prompt, is at material risk of later disclosure. The privacy risk is particularly acute in deployment settings in which third-party LLM APIs are used: users are effectively compelled to place sensitive, often regulated, data into the hands of an external honest-but-curious provider in order to obtain high-quality reasoning.

Mitigating this risk is difficult because the privacy mechanism must operate in the plaintext input space. Unlike centralized training or fine-tuning, where differentially private optimization [16,17,18,19,20] or forward-pass noise injection [21] can be applied, context-aware reasoning requires transmitting a concrete textual prompt that an external LLM can interpret. Standard approaches that add noise to token embeddings or apply perturbations at the representation level [22,23,24,25,26] cannot be used directly, because the model only accepts tokenized text. More recent work has begun to address privacy at the prompt level via differentially private in-context learning [27], prompt-engineering with differential privacy guarantees [28], and privacy-preserving prompt tuning [29,30], but these approaches largely concern few-shot demonstration selection or prompt parameter training rather than the on-the-fly sanitization of retrieved context required by RAG pipelines. Conversely, coarse strategies such as replacing sensitive entities with pseudo-values (e.g., [PERSON_1]) break the semantic coherence that LLMs rely upon and frequently fail to satisfy any formal notion of indistinguishability [31]. The result is a gap: existing privacy-preserving techniques in NLP either destroy utility, target the wrong computational stage (training or prompt-engineering rather than online context sanitization), or lack formal guarantees in the textual input space.

To address this gap, this paper introduces a paradigm Textual Differential Privacy (TDP) derived from our prior work [32], that embeds differential privacy guarantees directly into the textual context supplied to LLMs, while preserving the structural and semantic properties necessary for downstream reasoning. The proposed paradigm tackles the inherent challenges of protecting user privacy in LLM-based context-aware reasoning without compromising the high-quality contextual inference derived from the language model. Central to the framework is the Differential Embedding Substitution (DES) (In our work [32], this mechanism was named the Differential Embedding Hash (DEH). We rename it to Differential Embedding Substitution (DES) throughout this version, because the operation is a similarity-constrained, randomized substitution of named entities and involves no cryptographic hashing.) algorithm, which anonymizes sensitive named entities by replacing them with type-compatible syntactically compatible substitutes, drawn from a candidate set governed by a cosine-similarity threshold. Moreover, we introduce a formal privacy-loss quantification scheme, providing practitioners with interpretable metrics to navigate the privacy–utility trade-off in deployment. Through rigorous analysis and comprehensive experimentation, we demonstrate the effectiveness and robustness of our approach in mitigating privacy risks associated with context-aware reasoning tasks.

Relative to our prior work [32], we (i) substantially expand the related work with contemporary developments in differentially private language modeling and privacy-preserving text analytics, (ii) formalize the TDP framework with a rigorous metric differential privacy analysis and provide theoretical guarantees for the Differential Embedding Hash mechanism (Section 4), (iii) enrich the empirical evaluation with additional ablation and sensitivity analyses across multiple entity categories, privacy budgets, and LLM backbones, (iv) systematically compare against representative baselines (hash-based anonymization and pseudo-value replacement via Presidio), and (v) examine the privacy–utility trade-off, limitations, and avenues for future research.

Our contributions can be summarized as follows:

• We introduce Textual Differential Privacy (TDP), a novel paradigm aimed at preserving user privacy in context-aware reasoning tasks based on large language models while concurrently ensuring the performance of inference.
• We propose the Differential Embedding Substitution algorithm, which facilitates the anonymization of sensitive data within the framework of context-aware reasoning, preserving reasoning capabilities without compromising privacy.
• We derive a theoretical method to assess privacy loss in context-aware reasoning for a principled understanding of the balance between privacy protection and utility.
• We conduct a comprehensive experimental evaluation, including ablations and baseline comparisons, to validate the efficacy and resilience of TDP in mitigating privacy threats associated with context-aware reasoning.

The remainder of this paper is organized as follows. Section 2 reviews related work on context-aware reasoning, differential privacy, and privacy-preserving LLMs. Section 3 introduces the adversary model and the technical preliminaries underlying our framework. Section 4 presents the TDP framework in detail, followed by a theoretical analysis. Section 5 reports the experimental evaluation and comparison with representative baselines. Section 6 discusses the privacy–utility trade-off, limitations, and future directions. Section 7 concludes the paper.

2. Related Work

2.1. Context-Aware Reasoning with LLMs

To address the challenges inherent to context-dependent conversations, particularly those pertaining to token limitations and the intricate modeling of entity relations within the contextual scope provided to LLMs, recent work has advocated for integrating both prior queries and contextual information into the model input, thereby enhancing coherence across interactions [33]. Retrieval-Augmented Generation (RAG) has emerged as the dominant framework for this purpose, designed to enhance the quality of responses generated by LLMs by augmenting their internal representation with external sources of knowledge.

In a seminal paper published in 2020, Lewis et al. [5] introduced the RAG framework as a means of furnishing LLMs with access to supplementary information beyond their training corpus. RAG empowers LLMs to leverage specialized knowledge domains, thereby improving the precision of responses. More recent surveys [6] consolidate advances in retrieval strategies, indexing schemes, and prompt orchestration patterns.

Implementation of RAG within LLM-based question-answering systems offers dual advantages. First, it ensures that the model accesses up-to-date and trustworthy facts, improving response accuracy. Second, by granting users visibility into the retrieval sources, RAG fosters transparency and allows users to verify claims made by the model, thereby engendering trust in its outputs [6].

2.2. Differential Privacy

Privacy-preserving data release is a critical concern in contemporary data persistence [34] and processing systems [35,36,37]; protocol-level solutions alone [38] fail to protect privacy at the data source. Differential Privacy (DP), through rigorous mathematical definitions and formal models, provides strong privacy guarantees for individuals during both data release and analysis [39,40].

Definition 1

(Approximate differential privacy [40]). Let $\mathcal{X}$ be a dataset domain equipped with an adjacency relation ∼ (i.e., $D\sim D^{\prime }$ if D and $D^{\prime }$ differ in one record). A randomized mechanism $\mathcal{M}:\mathcal{X}\to R$ satisfies $(\epsilon ,\delta )$-differential privacy if, for every $D\sim D^{\prime }\in \mathcal{X}$ and every measurable $S\subseteq R$,

$$Pr[\mathcal{M}(D)\in S]\le e^{\epsilon }·Pr[\mathcal{M}(D^{\prime })\in S]+\delta .$$

(1)

The parameter $\epsilon \ge 0$ quantifies the privacy loss; $\delta \in [0,1]$ allows a relaxation that permits the pure-DP guarantee [39] to fail with probability at most δ.

Intuitively, $(\epsilon ,\delta )$-DP enforces that the output distributions of a mechanism on any two adjacent inputs are statistically close, rendering it infeasible for any adversary, with any prior or side information, to infer the presence or absence of an individual record from the mechanism’s output.

However, the requirement of transmitting sensitive data in unstructured form is not well captured by the binary Hamming-style adjacency of Definition 1: textual records differ not merely by “one record” but by a graded semantic distance. Chatzikokolakis et al. [41] therefore introduced metric differential privacy ($d_{\chi }$-privacy), which generalizes the adjacency relation to a pseudometric $d_{\chi }:\mathcal{X}\times \mathcal{X}\to {\mathbb{R}}_{\ge 0}$.

Definition 2

(Metric differential privacy [41]). A randomized mechanism $\mathcal{M}:\mathcal{X}\to R$ satisfies $(\epsilon ,\delta )$-$d_{\chi }$-privacy if, for every $D,D^{\prime }\in \mathcal{X}$ and every measurable $S\subseteq R$,

$$Pr[\mathcal{M}(D)\in S]\le e^{\epsilon ·d_{\chi }(D,D^{\prime })}·Pr[\mathcal{M}(D^{\prime })\in S]+\delta .$$

(2)

Definition 2 recovers Definition 1 as the special case $d_{\chi }(D,D^{\prime })\in \{0,1\}$. Its key advantage in the textual setting is that the privacy loss between two inputs scales with their semantic distance, so that two semantically distant inputs need not be rendered indistinguishable [22,24,42].

In the field of natural language processing (NLP), standard DP typically introduces noise into the word embedding space [16]. However, for scenarios involving explicit de-identification of plaintext, obfuscation-based methods remain the dominant approach [43], albeit with notable deficiencies such as non-provable privacy and high precomputation overhead [44]. Work on differentially private text analytics has further explored generalized metrics [24], calibrated multivariate perturbations [22], hierarchical representations [23], natural text sanitization [45], auto-encoder-based rewriting [46,47], and Mahalanobis-metric perturbations [25]. While illuminating, these methods operate in the embedding space and often exhibit semantically degraded outputs under heightened privacy constraints [31], limiting their applicability to plaintext context-aware reasoning.

It is instructive to contrast our mechanism explicitly with the metric-DP-in-embedding-space line of work, since both exploit the exponential/Laplace machinery of $d_{\chi }$-privacy. Weggenmann and Kerschbaum [44] and Feyisetan et al. [22,23] perturb every token by sampling calibrated noise in the embedding space and projecting back to the nearest word, thereby providing word-level $d_{\chi }$-privacy for the entire document. Two differences are central to our setting. First, those mechanisms presuppose that the downstream consumer either accepts perturbed embeddings directly or tolerates the substantial lexical drift that whole-document perturbation induces; in our deployment the consumer is a black-box third-party LLM that accepts only fluent tokenized text and whose reasoning quality collapses under indiscriminate perturbation. We therefore confine perturbation to NER-identified sensitive spans and require substitutes to be type-compatible (a person for a person), preserving the syntactic and discourse scaffolding the LLM relies on. Second, whereas those methods calibrate noise to a user-chosen $\epsilon$ over the full vocabulary, we couple the mechanism to a semantically constrained candidate set $C_{\theta }(·)$, which both bounds the worst-case utility loss and yields the interpretable accounting of Section 4.5. We regard a head-to-head empirical comparison against these embedding-perturbation baselines, as well as against text-rewriting mechanisms such as DP-BART [47], ADePT [46], and natural-text sanitization [45], as an important and natural next step, which we discuss in Section 6.

2.3. Privacy-Preserving Large Language Models

Privacy concerns specific to LLMs have attracted growing attention in recent years [7]. Carlini et al. [8,9] demonstrated that LLMs memorize and can leak training data, with memorization scaling log-linearly in model capacity, duplication rate, and prompt context length. Mireshghallah et al. [11] analyzed memorization in fine-tuned autoregressive models, while Huang et al. [12] specifically studied the leakage of personally identifiable information such as email addresses via memorization-vs-association attacks. ProPILE [10] formalized probing attacks for PII leakage, and Lukas et al. [13] introduced rigorous game-based definitions together with black-box extraction, inference, and reconstruction attacks that can extract up to an order of magnitude more PII than prior baselines. At the session and deployment level, Kandpal et al. [14] designed user-inference attacks against fine-tuned LLMs with near-perfect success rates on some configurations, and Mireshghallah et al. [15] introduced the ConfAIde benchmark grounded in contextual integrity, showing that even GPT-4 violates information-flow norms in 39% of interactions. More recently, self-sovereign identities are leveraged in decentralized LLM agents [48] for secure and privacy-preserving communications. Together, these results establish that sensitive information exposed to an LLM, whether via training data or a prompt, is at material risk of later disclosure.

A rich body of work addresses privacy during training or fine-tuning of language models. DP-SGD [16] provides the foundational training-time mechanism; recent adaptations [17,18,19,20] have shown that LLMs can be trained or fine-tuned with strong differential privacy guarantees while retaining usable quality. Du et al. [21] complement this line by injecting noise in the forward pass via an analytic matrix-Gaussian mechanism, thereby mitigating embedding–inversion and attribute-inference attacks that DP-SGD alone does not address. At the prompt level, Duan et al. [30] introduced differentially private prompt learning via a PATE-style ensemble; Li et al. [29] proposed privacy-preserving prompt tuning (RAPT) with privatized token reconstruction; Tang et al. [27] generated DP few-shot demonstrations for in-context learning; and Hong et al. [28] proposed DP-OPT, which transfers differentially private discrete prompts from a local LLM to a cloud LLM. Mattern et al. [31] critically examined the limits of word-level DP in text analytics.

By contrast, relatively little work has addressed the inference-time privacy of prompts transmitted to a third-party LLM for context-aware reasoning with retrieval. Presidio [49], a widely used de-identification tool, substitutes detected PII with pseudo-values (e.g., [PERSON_1]), an approach that lacks formal DP guarantees and often disrupts downstream reasoning [31]. DP-BART [47] and ADePT [46] investigate privatized text rewriting, and Yue et al. [45] propose natural text sanitization under metric DP, but these methods are not tailored to RAG or context-aware prompts and typically incur substantial semantic drift. Most related prompt-level DP methods [27,28,29,30] assume control over the demonstration set or learn soft/discrete prompts rather than sanitizing the retrieved context that a RAG pipeline synthesizes at query time. To the best of our knowledge, no existing paradigm is specifically designed to provide formal, quantifiable privacy guarantees for context-aware reasoning prompts consumed by large language models while preserving their reasoning effectiveness. This paper directly fills that gap.

3. Preliminaries

3.1. Adversary Model

Within the paradigm of context-aware reasoning, Local Differential Privacy (LDP) is the more appropriate setting, as it assumes that the central party is honest but curious. In LDP, each participant locally perturbs their data before sharing it with the central party, thereby preserving privacy while permitting meaningful analysis. The assumption of honesty implies that the central party faithfully follows the protocol and does not tamper with the reported data. However, the central party is characterized as curious, meaning that it may attempt to infer sensitive information from the perturbed data [50].

Under the conventional context-aware reasoning framework, users transmit their sensitive data in plaintext form as contextual prompts to an honest-but-curious LLM service, potentially resulting in significant privacy breaches. To mitigate these vulnerabilities, the incorporation of differential privacy metrics becomes imperative. DP offers a quantitative framework for reasoning about privacy, enabling the elucidation of a principled trade-off between privacy preservation and analytical efficacy. In our setting, we assume the following adversary:

• The adversary is the LLM service provider (or any entity able to observe the transmitted prompts), is honest-but-curious, and is computationally unconstrained in the sense that the metric-DP guarantee of Theorem 1 is information-theoretic and holds against an adversary of arbitrary computational power, so no amount of computation applied to the released prompt circumvents the output-distribution bound under the stated metric.
• The adversary has auxiliary knowledge about possible entity types (e.g., candidate person names, organization names) but does not know which specific entity corresponds to a given user.
• The adversary observes the anonymized prompt $x^{\prime }$ and the LLM response; it does not observe the user’s local lookup dictionary used for re-identification.

In this setting, we posit that when no entity names are present in the context, privacy concerns are absent. Conversely, when sensitive entity names are included, the semantic distance between the replacement and the actual entity determines the degree of privacy protection: a greater semantic distance corresponds to stronger privacy. The metric of privacy loss, denoted $\epsilon$, serves as an overall measure for evaluating the extent of privacy protection in text. A lower $\epsilon$ indicates a higher risk of PII being revealed during release, while a higher $\epsilon$ indicates that the anonymized text diverges meaningfully from the original.

3.2. Named-Entity Recognition

Named-Entity Recognition (NER) is a foundational task in privacy-preserving text processing. NER identifies and classifies named entities, such as persons, organizations, and locations, within a given text. In the context of reasoning with LLMs, NER plays a significant role in preserving privacy while processing text data containing sensitive information. The process consists of two sub-steps.

• Named-Entity Recognition. A NER algorithm is applied to the input text to identify named entities. Formally, $\mathrm{NER}(x)=\{e_1,e_2,\dots ,e_k\}$, where x is the input text and $e_i$ denotes the i-th identified entity.
• Differential Embedding Substitution. Once named entities are identified, they are replaced using the Differential Embedding Substitution (DES) mechanism to anonymize the data. Let $\mathrm{Sub}(e_i)$ denote the substitute drawn for the i-th entity $e_i$. Formally, the anonymized text is

  $$\mathrm{Anonymized}(x)=\mathrm{Replace}\left(x,e_i,\mathrm{Sub}(e_i)\right),$$

  (3)

  where $\mathrm{Replace}(·)$ denotes the in-place substitution of entity $e_i$ with the drawn substitute $\mathrm{Sub}(e_i)$. Here $\mathrm{Sub}:{\mathcal{E}}_t\to {\mathcal{E}}_t$ is a randomized map that returns a replacement entity of the same type t, not a cryptographic hash or a vector; it samples from the candidate pool $C_{\theta }(e_i)$ of Equation (7) according to the distribution specified in Section 4.3. We retain the historical name “embedding hash” only as a label; the operation is similarity-based substitution. The processed prompt encapsulates relevant context and enforces privacy preservation during inference. By replacing named entities with anonymized substitutes, this process ensures privacy while enabling context-aware reasoning and personalized interactions.

3.3. Metric Differential Privacy for Text

We now specialize Definition 2 to the textual setting in which a sample is an ordered sequence of words. Let W denote a finite vocabulary and $W^{\le L}$ the set of strings of length at most L over W. Let $\varphi :W\to {\mathbb{R}}^n$ be a word embedding model (e.g., Word2Vec [51] or GloVe [52]) satisfying $\varphi (w)\ne \mathbf{0}$ for every $w\in W$. For two vectors $\mathbf{a},\mathbf{b}\in {\mathbb{R}}^n\backslash \{\mathbf{0}\}$, their cosine similarity is

$${\mathrm{S}}_{cos}(\mathbf{a},\mathbf{b})={\displaystyle \frac{{\mathbf{a}}^{\top }\mathbf{b}}{{\parallel \mathbf{a}\parallel }_2{\parallel \mathbf{b}\parallel }_2}}\in [-1,1],$$

(4)

and the induced cosine distance is $d_{cos}(\mathbf{a},\mathbf{b}):=1-{\mathrm{S}}_{cos}(\mathbf{a},\mathbf{b})\in [0,2]$.

Definition 3

(Textual cosine distance). For two strings $x=w_1{w}_2\cdots w_l$ and $x^{\prime }=w_1^{\prime }{w}_2^{\prime }\cdots w_l^{\prime }$ of equal length $l\le L$, the textual cosine distance is

$$D(x,x^{\prime })={\displaystyle \sum _{i=1}^l}{d}_{cos}\left(\varphi (w_i),\varphi (w_i^{\prime })\right)={\displaystyle \sum _{i=1}^l}\left(1-S_{cos}\left(\varphi (w_i),\varphi (w_i^{\prime })\right)\right).$$

(5)

In our mechanism the two strings being compared are always the original prompt x and its anonymized release $x^{\prime }$, which are token-aligned by construction: $x^{\prime }$ is obtained from x by substituting individual entity tokens in place, so $|x|=|x^{\prime }|$ and position i in $x^{\prime }$ corresponds to position i in x. The general definition for arbitrary $x,x^{\prime }$ of unequal length is therefore not needed for the privacy analysis; when one nonetheless wishes to compare unequal-length strings, we pad the shorter one to the length of the longer with a designated null token ⊥ for which $\varphi (\perp )$ is fixed and $d_{cos}(\varphi (w),\varphi (\perp ))$ is defined to be the maximal value 2, so that insertions or deletions are charged the worst-case per-token distance. This convention preserves non-negativity and symmetry and is used only as a fallback.

Proposition 1

(Pseudometric). The textual cosine distance $D(·,·)$ is a pseudometric on $W^{\le L}$: it is non-negative, symmetric, satisfies $D(x,x)=0$, and obeys the triangle inequality, because each summand $d_{cos}(·,·)$ is a pseudometric on ${\mathbb{R}}^n\backslash \{\mathbf{0}\}$. It fails to be a metric only because distinct words with identical embedding directions may receive distance zero.

We stress that the pseudometric property is sufficient for our purposes: metric differential privacy (Definition 2) is defined with respect to a pseudometric $d_{\chi }$, and the exponential-mechanism guarantee of Theorem 1 requires only non-negativity, symmetry, and the triangle inequality, all of which $D(·,·)$ satisfies. The only consequence of D being a pseudometric rather than a metric is benign in the privacy direction: two distinct entities $w\ne \tilde{w}$ with $\varphi (w)\Vert \varphi (\tilde{w})$ satisfy $d_{cos}(\varphi (w),\varphi (\tilde{w}))=0$ and are therefore required to be perfectly indistinguishable, which is the strongest possible guarantee rather than a gap in it. Such exact collinearity of two distinct in-vocabulary entity embeddings is essentially never observed in practice with Word2Vec, but the analysis remains valid even when it occurs.

Instantiating Definition 2 with $d_{\chi }=D$ yields $(\epsilon ,\delta )$-D-privacy for textual data. We will refer to this specialization simply as textual metric DP. For an LLM $\mathcal{M}$ used for context-aware reasoning, our objective is to design a local mechanism that, for each sensitive token $w_i$, releases a perturbed substitute $w_i^{\prime }$ such that the overall release satisfies $(\epsilon ,\delta )$-D-privacy. Because the mechanism acts independently on each sensitive position, the composition is parallel over disjoint positions, and the overall privacy loss over a sample decomposes additively into per-token contributions (see Theorem 1 below).

Two quantities will be central to our analysis. First, the local sensitivity of the textual release operator $\mathcal{R}$ at position i is

$${\Delta }_i(\mathcal{R})=\underset{w,w^{\prime }\in W}{sup}{d}_{cos}\left(\varphi (w),\varphi (w^{\prime })\right),$$

(6)

which is bounded by 2 by construction. Second, for a similarity threshold $\theta \in (0,1]$ and entity $w\in W$, the candidate pool is

$$C_{\theta }(w)=\left\{w^{\prime }\in {\mathcal{E}}_t:{\mathrm{S}}_{cos}\left(\varphi (w),\varphi (w^{\prime })\right)<\theta \right\},$$

(7)

where ${\mathcal{E}}_t$ is the set of dictionary entities of the same type t as w. The candidate pool consists of all type-compatible entities whose (signed) cosine similarity to w is smaller than $\theta$, and therefore whose cosine distance is greater than $1-\theta$. We use the signed similarity rather than its absolute value: a substitute lying in the opposite embedding direction (negative ${\mathrm{S}}_{cos}$, hence cosine distance approaching 2) is maximally far from w and is exactly the kind of distant substitute the mechanism should be free to select; thresholding on $|{\mathrm{S}}_{cos}|<\theta$ would needlessly exclude such candidates. Because Word2Vec entity similarities are predominantly non-negative, the two conventions coincide on the bulk of the dictionary; the signed form is the principled choice and is what we adopt. It is the minimum semantic-distance “floor” that our mechanism guarantees on every release.

4. The TDP Framework

To address the above challenges, we propose the utilization of Metric DP to achieve a trade-off between privacy preservation and data utility. Metric DP serves as the cornerstone of LDP frameworks and protocols, offering a robust mechanism to quantify and manage privacy concerns within context-aware reasoning environments. The overall procedural workflow of the TDP framework is depicted in Figure 1. By leveraging Metric DP, stakeholders can systematically evaluate the degree of privacy protection afforded to sensitive data while simultaneously gauging the analytical utility derived from such data. This approach enhances transparency and accountability in privacy-preserving practices and empowers decision-makers to navigate the trade-offs inherent in balancing privacy with utility.

Figure 1. Diagram of fine-grained differentially private context-aware reasoning. User data undergoes transformation through an embedding model to construct a vector database (“PK” denotes the primary key). Queries prompt similarity search operations over this database, facilitating the retrieval of proximal contextual matches and prompt orchestration. Subsequently, Named-Entity Recognition processes the contextual prompts, yielding named entities and their positions. To uphold privacy, the Differential Embedding Hash algorithm anonymizes this information and generates anonymized contextual prompts. These are then fed into the LLM for context-aware reasoning, culminating in response derivation. Reverse hashing locally re-identifies anonymized entities in the response prior to presentation to the user.

The framework unfolds as a systematic approach toward fine-grained differential privacy in context-aware reasoning, orchestrating several interdependent processes. It commences with the transformation of user data through an embedding model to construct a vector database. Subsequent queries prompt tailored search operations within this database, facilitating the retrieval of contextually proximal matches. NER then identifies salient entities and their positional attributes within the contextual prompts. To safeguard user privacy, the Differential Embedding Hash algorithm is employed to anonymize and replace sensitive information extracted through NER, thereby generating anonymized contextual prompts. These prompts serve as inputs for context-aware reasoning with the LLM, culminating in the derivation of responses. The methodology incorporates reverse hashing mechanisms to discern anonymized responses before their presentation to the user.

4.1. Context-Aware Prompt Construction

RAG serves as a pivotal framework for realizing context-aware prompt functionalities, providing LLMs with additional information from an external knowledge source. This facilitates more accurate and contextual answers while reducing hallucinations. In most RAG deployments, the LLM serves as the central party providing services, while context-aware prompts encapsulate user privacy data, transmitted in plaintext to the central party. This presents significant risks of privacy breach. To mitigate these risks and protect user privacy, it is imperative to enact privacy-preserving operations on context-aware prompts. The core functionality of RAG revolves around prompt orchestration, a procedure frequently executed locally and often involving sensitive data and PII. Therefore, the focus of privacy-preserving measures in our framework is the de-identification of the prompts generated by RAG, aiming to preserve the inference capabilities of LLMs while undergoing anonymization.

4.2. Named-Entity Recognition Integration

Failure to anonymize data may expose sensitive information such as names, addresses, contact numbers, and other identifying details associated with individuals, which could be exploited for malicious purposes. By obscuring or eliminating this PII, data can be utilized without jeopardizing individuals’ privacy rights or violating data protection legislation.

Intuitively, to anonymize PII contained within context-aware prompts while ensuring the effectiveness of LLM inference, we replace PII with substitutes that do not materially alter the overall semantic meaning. This requires integrating a privacy layer that de-identifies context-aware prompts prior to their transmission to the LLM, a procedure referred to as the anonymization process. The initial step involves identifying data fields containing PII, through two stages: entity extraction and entity classification. In entity extraction, a NER model scans the data, identifying words qualifying as entities through techniques such as BIOES tagging. The subsequent stage categorizes these entities into predefined classes.

Flair [53] is an advanced NLP framework with notable strengths in NER. One of its primary advantages is the use of contextual string embeddings, which capture nuanced semantic relationships and dependencies crucial for accurate NER. Furthermore, Flair provides pre-trained NER models across multiple languages. To enhance Flair’s performance on unseen or out-of-domain data, we employ the FLERT framework for fine-tuning, which provides a structured methodology to adapt Flair to diverse data domains and to improve its generalization capacity [54].

In the context of NER, numerical labels are assigned to specific entity types according to a standardized convention. Each tag is prefixed with either B- or I-, signifying whether the token is the beginning or an intermediate part of an entity span. This labeling scheme offers a structured approach to annotating entities in text, with flexibility and extensibility that facilitates customized fine-tuning of models according to specific requirements.

As illustrated in Figure 2, within the Flair framework, a sentence is transformed into a character sequence and fed into a bidirectional character language model pretrained on extensive unlabeled corpora. Contextual embeddings are extracted for each word by capturing the states of the first and last characters. These embeddings are then consumed by a BiLSTM-CRF sequence labeler [55]. To further optimize NER performance, we adopt a strategy known as surgical fine-tuning, which involves targeted adjustment of a pretrained model to enhance its effectiveness on a specific dataset [56]. Surgical fine-tuning modifies a pre-trained model by adding a new class to its classification layer while retaining the original weights on the layer subset relevant to the task, enabling customization to NER tasks or domains without losing the knowledge gained during pre-training. Following identification, PII is replaced with pseudo values or codes as detailed next.

Figure 2. The workflow of Named-Entity Recognition within the TDP framework.

4.3. Differential Embedding Substitution

As the second step of anonymization, replacement substitutes PII with pseudo values or codes, with the aim of minimizing any adverse impact on the inference and analysis capabilities of LLMs. Conventional encryption algorithms are unsuitable for these substitutes, since encrypted data can impede the language model’s ability to grasp the inherent meaning or context. Existing approaches therefore often resort to pseudo values such as [NAME_GIVEN_1] or [ORGANIZATION_1] to replace PII [49].

However, according to the principles of DP [35], a mechanism’s output should be statistically indistinguishable between a dataset and an adjacent one, even with arbitrarily small changes to individual records. Resorting to fixed pseudo values clearly contradicts this principle, because a dataset where entities have been replaced by patterned pseudo-values is trivially discernible. To overcome this, we propose the Differential Embedding Substitution (DES) algorithm, which replaces PII with substitutes drawn from a candidate set according to an exponential mechanism that satisfies metric DP, while additionally enabling quantitative privacy accounting.

4.3.1. Candidate Selection

The central idea of DES is to anonymize each PII entity by replacing it with another named entity of the same type, sampled by a randomized mechanism drawn from the candidate pool $C_{\theta }(w_i)$ of Equation (7). For each identified entity $w_i$, the algorithm performs an inverse similarity search: it computes the cosine similarity between $\varphi (w_i)$ and $\varphi (w^{\prime })$ for every labeled entity $w^{\prime }$ of the same category, and retains those with ${\mathrm{S}}_{cos}<\theta$. A substitute is then drawn according to the exponential mechanism

$$p_{{\epsilon }_b}(v\mid w_i)={\displaystyle \frac{exp\left(-\frac{{\epsilon }_b}{2}{d}_{cos}(\varphi (w_i),\varphi (v))\right)}{{\sum }_{u\in C_{\theta }(w_i)}exp\left(-\frac{{\epsilon }_b}{2}{d}_{cos}(\varphi (w_i),\varphi (u))\right)}},v\in C_{\theta }(w_i),$$

(8)

where ${\epsilon }_b>0$ is the privacy budget, fixed ex ante by the practitioner. The exponential weighting is what determines the per-position output-distribution ratio scale with the cosine distance between candidate originals, and hence yields the metric-DP guarantee of Theorem 1. Two special cases are worth noting: as ${\epsilon }_b\to 0$, $p_{{\epsilon }_b}$ tends toward the uniform distribution over $C_{\theta }(w_i)$ (maximum entropy, the simple instantiation used in our experiments of Section 5); as ${\epsilon }_b$ grows, mass concentrates on the closest in-pool candidates, trading privacy for utility. To preserve the one-to-one correspondence between $w_i$ and $w_j^{\prime }$ within a session, the selected $w_j^{\prime }$ is removed from the candidate pool for the duration of that dialogue with the LLM. If $C_{\theta }(w_i)=\varnothing$ (a rare entity with no sufficiently distant in-type candidate), the mechanism applies a deterministic, data-independent fallback: it emits a fixed generic placeholder for type t (identical for every empty-pool entity of that type), so that the fallback event reveals nothing beyond the entity type that NER has already exposed.

4.3.2. Privacy Accounting

DES admits a closed-form privacy accountant. Let $x=w_1\cdots w_l$ be an input containing $m\le l$ sensitive entities, and let $x^{\prime }=w_1^{\prime }\cdots w_l^{\prime }$ denote the anonymized release produced by DES; by construction, $w_i=w_i^{\prime }$ on non-sensitive positions and $w_i^{\prime }\in C_{\theta }(w_i)$ on the m sensitive positions. We distinguish two quantities that the original presentation conflates. (i) The formal privacy budget is the parameter ${\epsilon }_b$ of Equation (8), which by Theorem 1 controls the metric-DP guarantee ex ante, uniformly over all outputs and all pairs of adjacent inputs. (ii) The realized semantic shift of a particular release is the textual cosine distance actually incurred. The textual cosine distance of Equation (5) collapses to a sum over sensitive positions,

$$D(x,x^{\prime })=\sum _{i\in {\mathcal{I}}_{\mathrm{sens}}}{d}_{cos}\left(\varphi (w_i),\varphi (w_i^{\prime })\right)>m(1-\theta ),$$

(9)

where ${\mathcal{I}}_{\mathrm{sens}}\subseteq \{1,\dots ,l\}$ indexes the sensitive positions and the strict inequality follows from $w_i^{\prime }\in C_{\theta }(w_i)$. We report the normalized realized shift

$$\overline{\epsilon }:={\displaystyle \frac{D(x,x^{\prime })}{\theta }}>{\displaystyle \frac{m(1-\theta )}{\theta }},$$

(10)

as an interpretable, a posteriori diagnostic of how far the release landed from the original; it is not the differential-privacy parameter and should not be read as one. Its monotone dependence on m and on $1/\theta$ nonetheless makes it a convenient per-condition utility-side summary, which is why we tabulate it in Section 5. The pseudocode of DES is presented in Algorithm 1.

Algorithm 1 Differential Embedding Substitution (DES)

1: Input: Sample $x=w_1{w}_2\cdots w_l$; labeled named entity collection $D_e=\{w_1^{\prime },w_2^{\prime },\dots ,w_n^{\prime }\}$; threshold $\theta$; budget ${\epsilon }_b$.  2: Output: Anonymized string $x^{\prime }$; realized semantic shift $\overline{\epsilon }$.  3: Procedure:  4:    $\{e_1,e_2,\dots ,e_k\}\leftarrow \mathrm{NER}(x)$                ▹ Named-entity recognition  5:    for $i=1$ to k                 ▹ Loop through each named entity  6:       $C_{\theta }(e_i)\leftarrow \varnothing$; Compute similarity $S\leftarrow {\mathrm{S}}_{\cos \mathrm{ine}}(\varphi (e_i),\varphi (w_j^{\prime }))$ for each type-t candidate $w_j^{\prime }\in D_e$  7:       if $S<\theta$ then add $w_j^{\prime }$ to $C_{\theta }(e_i)$  8: $$  9:       if $C_{\theta }(e_i)=\varnothing$ then $\mathrm{Sub}(e_i)\leftarrow$ fixed generic placeholder for type t ▹ deterministic fallback 10:       else $\mathrm{Sub}(e_i)\leftarrow v$ sampled from $C_{\theta }(e_i)$ by Equation (8); remove v from $D_e$ for this session 11:       $x\leftarrow \mathrm{Replace}(x,e_i,\mathrm{Sub}(e_i))$ 12:    $x^{\prime }\leftarrow x$; compute $\overline{\epsilon }$ by Equation (10) 13: $$ 14: Return $x^{\prime }$, $\overline{\epsilon }$

Through the TDP paradigm, we can anonymize the input text and account for its privacy under the metric-DP budget ${\epsilon }_b$ while reporting the realized semantic shift $\overline{\epsilon }$, facilitating a more refined quantitative assessment of the privacy–utility balance.

4.3.3. Tokenization and Positional Stability

The mechanism identifies entities at the word level with Flair, whereas the downstream LLM consumes sub-word tokens. Replacing a single-token word with a multi-token substitute (or the reverse) therefore shifts the token count and the absolute positions of all subsequent tokens, and such a “structural” perturbation that is distinct from any semantic change could in principle degrade reasoning by displacing the positional scaffolding on which the transformer attends. We did not design the candidate selection of Equation (8) to control token length, so we quantified the realized drift directly. Across our evaluation the per-substitution change in tokenizer length, measured with the same tokenizer used for prompt accounting, was modest: the mean absolute change was $|\Delta \mathrm{tokens}|=0.41$, and $73.8\%$ of substitutions left the token count of the prompt unchanged. The remaining substitutions changed the count by a small number of tokens, well within the range that ordinary paraphrasing or formatting variation introduces and to which instruction-tuned LLMs are robust. This is consistent with the high exact-match accuracies of Table 1, which would be the first quantity to fall were positional drift breaking reasoning. The effect is therefore small in our regime but not zero, and it suggests a concrete refinement: augmenting the candidate score in Equation (8) with a length-matching term, so that, among equally distant substitutes, the mechanism prefers one whose sub-word length matches the original. We note this as future work in Section 6.

Table 1. Model performance under TDP for multiple-choice QA on the Cosmos QA dataset. CG: control group; PER: persons; LOC: locations; ORG: organizations. The reported $\overline{\epsilon }$ is the realized aggregate semantic shift $D(x,x^{\prime })/\theta$ averaged over the evaluated prompts (Equation (10)), an a posteriori diagnostic, not the formal metric-DP budget ${\epsilon }_b$. Accuracies are exact-match over the 200 evaluated items. Equal accuracy values across conditions denote an equal number of correctly answered items, not necessarily the same set: anonymization can shift which specific items are missed while leaving the total count unchanged, so equal entries reflect stable aggregate accuracy rather than identical per-item outcomes.

Model	Accuracy (%)
	CG	PER $\overline{\mathit{\epsilon }}\mathbf{=}\mathbf{0.64}$	PER&ORG $\overline{\mathit{\epsilon }}\mathbf{=}\mathbf{0.69}$	PER&LOC $\overline{\mathit{\epsilon }}\mathbf{=}\mathbf{0.88}$	PER&ORG&LOC $\overline{\mathit{\epsilon }}\mathbf{=}\mathbf{1.03}$
Llama 2 [57]	92.50	92.50	92.50	92.00	92.00
ChatGPT-4 [58]	98.50	98.50	98.50	98.50	98.50
Gemini [59]	97.00	97.00	96.00	96.00	95.50

4.4. Response Re-Identification

While the above anonymization is effective, it can be misleading for human interpretation: a user should ultimately see references to the real entities, not their anonymized substitutes. To this end, we re-identify anonymized entities in the LLM’s response.

Re-identification requires maintenance of a local dictionary, populated during the DES phase, that records the bijection between each $w_i$ and its replacement $w_j^{\prime }$. Using this dictionary, anonymized entities in the LLM’s response are located and substituted back with the original PII, producing the final response for the user. This re-identification step is performed entirely locally, ensuring that the mapping never leaves the trusted client boundary.

We note an important robustness caveat. Re-identification by dictionary lookup is reliable only when the substitute $w_j^{\prime }$ appears verbatim in the response. If the LLM paraphrases or omits the substitute, e.g., rewriting “Tim Henman” as “the incoming manager”, the lookup will fail to recover the corresponding original entity. Empirically this failure mode is markedly rarer for DES than for pseudo-value baselines such as Presidio, because a fluent in-vocabulary substitute (a real name) is far more likely to be reproduced verbatim than an artificial placeholder such as [PERSON_1] (Section 5.5); nonetheless, it is not eliminated. A practical mitigation, which we leave to future work, is to perform fuzzy span-level alignment between the anonymized prompt and the response (e.g., via the LLM’s own token alignment or a lightweight local matcher) so that paraphrased substitutes can still be mapped back. We make this limitation explicit in Section 6.

4.5. Theoretical Analysis

We now analyze the privacy guarantees provided by DES. Throughout, let ${\mathcal{E}}_t$ denote the set of labeled entities of type t (e.g., Person) and let $C_{\theta }(w)$ denote the candidate pool of Equation (7). Write ${\mathcal{M}}_{\mathrm{DES}}^{(i)}$ for the per-position DES sub-mechanism of Equation (8) at sensitive position i, and ${\mathcal{M}}_{\mathrm{DES}}={⨂}_{i\in {\mathcal{I}}_{\mathrm{sens}}}{\mathcal{M}}_{\mathrm{DES}}^{(i)}$ for the parallel composition over the m sensitive positions of a prompt.

The privacy claim of the conference version was stated too loosely: the quantity $D(x,x^{\prime })/\theta$ is a property of the realized release rather than an ex ante bound on the output-distribution ratio, and a uniform draw over a candidate set yields a ratio governed by the pool cardinalities, not by the cosine distance. We address both points directly. The mechanism we analyze is the exponential mechanism of Equation (8), whose distance-aware weighting is exactly what converts the cardinality-ratio bound into a genuine metric-DP guarantee; the uniform draw is recovered as the ${\epsilon }_b\to 0$ limit and is treated separately in Remark 1.

Theorem 1

(Metric DP of DES). Fix a budget ${\epsilon }_b>0$ and let ${\mathcal{M}}_{\mathrm{DES}}^{(i)}$ be the per-position exponential mechanism of Equation (8) with full support ${\mathcal{E}}_t$. Then for every $w,\tilde{w}\in {\mathcal{E}}_t$ and measurable $S\subseteq {\mathcal{E}}_t$,

$$Pr\left[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(w)\in S\right]\le e^{{\epsilon }_b{d}_{cos}(\varphi (w),\varphi (\tilde{w}))}Pr\left[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(\tilde{w})\in S\right].$$

(11)

Consequently, for the parallel composition ${\mathcal{M}}_{\mathrm{DES}}$ over the sensitive positions of any two inputs $x,\tilde{x}\in W^{\le L}$ that differ only on ${\mathcal{I}}_{\mathrm{sens}}$,

$$Pr[{\mathcal{M}}_{\mathrm{DES}}(x)\in S]\le e^{{\epsilon }_{b}D(x,\tilde{x})}·Pr[{\mathcal{M}}_{\mathrm{DES}}(\tilde{x})\in S],$$

(12)

so ${\mathcal{M}}_{\mathrm{DES}}$ satisfies ${\epsilon }_b$-D-privacy (Definition 2 with $\delta =0$), where $D(·,·)$ is the textual cosine distance of Equation (5).

Proof. 

Write $d(·,·)=d_{cos}(\varphi (·),\varphi (·))$ and $Z(w)={\sum }_{u\in {\mathcal{E}}_t}exp(-\frac{{\epsilon }_b}{2}d(w,u))$. For any single output $v\in {\mathcal{E}}_t$,

$${\displaystyle \frac{Pr[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(w)=v]}{Pr[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(\tilde{w})=v]}}=exp\left(\frac{{\epsilon }_b}{2}\left(d(\tilde{w},v)-d(w,v)\right)\right)·{\displaystyle \frac{Z(\tilde{w})}{Z(w)}}.$$

By the triangle inequality $d(\tilde{w},v)-d(w,v)\le d(w,\tilde{w})$, the first factor is at most $exp(\frac{{\epsilon }_b}{2}d(w,\tilde{w}))$. For the normalizers, the same inequality gives $d(\tilde{w},u)\le d(w,u)+d(w,\tilde{w})$ for every u, hence term-by-term $Z(\tilde{w})\le exp(\frac{{\epsilon }_b}{2}d(w,\tilde{w}))Z(w)$, so the second factor is also at most $exp(\frac{{\epsilon }_b}{2}d(w,\tilde{w}))$. Multiplying the two bounds yields a per-output ratio of at most $exp({\epsilon }_{b}d(w,\tilde{w}))$, and integrating over S gives Equation (11). Because the sub-mechanisms act on disjoint positions, parallel composition of metric DP multiplies the per-output bounds, giving the exponent ${\epsilon }_b{\sum }_{i\in {\mathcal{I}}_{\mathrm{sens}}}d(w_i,{\tilde{w}}_i)={\epsilon }_{b}D(x,\tilde{x})$ and establishing Equation (12). □

Theorem 1 is the corrected statement: the differential-privacy parameter is the design-time budget ${\epsilon }_b$, the guarantee holds uniformly over all outputs and all pairs of adjacent inputs, and the privacy loss between two specific inputs scales with their cosine distance $D(x,\tilde{x})$ exactly as metric DP requires. The realized shift $\overline{\epsilon }=D(x,x^{\prime })/\theta$ of Equation (10) is, by contrast, an a posteriori utility-side diagnostic of one release and is not claimed to be the DP parameter.

Proposition 2

(Exact indistinguishability within a candidate-pool class). Fix $\theta \in (0,1]$ and let $w,\tilde{w}\in {\mathcal{E}}_t$ satisfy $C_{\theta }(w)=C_{\theta }(\tilde{w})$ with $\varphi (w),\varphi (\tilde{w})$ equidistant to every pool member. Let ${\mathcal{M}}_{\mathrm{DES}}^{(i)}$ be run in its uniform (${\epsilon }_b\to 0$) configuration. Then for every measurable $S\subseteq {\mathcal{E}}_t$, $Pr[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(w)\in S]=Pr[{\mathcal{M}}_{\mathrm{DES}}^{(i)}(\tilde{w})\in S]$; in particular ${\mathcal{M}}_{\mathrm{DES}}^{(i)}$ is $(0,0)$-DP on that class.

Proof. 

In the uniform configuration the output is $\mathrm{Unif}(C_{\theta }(·))$; equal pools imply equal supports with identical mass $1/|C_{\theta }(w)|$ per atom, and integrating over S yields the claim. □

We have deliberately demoted this statement from a stand-alone theorem to a corollary-style proposition, because exactly equal pools are a measure-zero event for dense Word2Vec embeddings and the proposition alone would say little about the typical case of nearly equal pools. That typical case is precisely what Theorem 1 covers: two entities with nearly identical pools have small $d_{cos}(\varphi (w),\varphi (\tilde{w}))$ and are therefore ${\epsilon }_b{d}_{cos}$-indistinguishable with a small exponent, degrading gracefully rather than discontinuously.

Remark 1

(Uniform instantiation and the cardinality-ratio bound). The simple uniform-over-$C_{\theta }$ mechanism used in our experiments is the ${\epsilon }_b\to 0$ limit truncated to $C_{\theta }(w)$. Truncation reintroduces a support mismatch: for $v\in C_{\theta }(w)\cap C_{\theta }(\tilde{w})$ the ratio of probabilities is $|C_{\theta }(\tilde{w})|/|C_{\theta }(w)|$, which depends on pool cardinalities rather than on cosine distance, while outputs in $C_{\theta }(w)\backslash C_{\theta }(\tilde{w})$ are absorbed by a relaxation term δ. Formally the truncated uniform mechanism is $({\epsilon }_{\mathrm{card}},\delta )$-DP with ${\epsilon }_{\mathrm{card}}={max}_{w,\tilde{w}}\left|log(|C_{\theta }(\tilde{w})|/|C_{\theta }(w)|)\right|$ over adjacent inputs. This is exactly the cardinality-dependence noted above, and it is the reason we adopt the distance-aware exponential mechanism of Equation (8) when a metric-DP guarantee calibrated to semantic distance is required; the uniform variant remains a convenient, maximum-entropy baseline whose utility we report.

Remark 2

(A fixed $\delta$ and a non-leaking fallback). In the conference version δ was defined as the data-dependent probability that some candidate pool is empty. We replace this with a fixed design constant. For the full-support mechanism of Theorem 1, $\delta =0$. For the θ-truncated mechanism, δ is the (input-independent) tail mass that the full-support exponential mechanism would place outside the shared support; the practitioner sets θ and ${\epsilon }_b$ so that δ does not exceed a target constant (we use $\delta ={10}^{-5}$ as a reference). Crucially, the empty-pool event is handled by the deterministic, data-independent fallback of Algorithm 1 (a fixed type-generic placeholder), so it leaks nothing beyond the entity type already revealed by NER and does not signal that a rare entity was present; this closes the “opt-out” loophole present in the conference version, in which the empty-pool case released public statistics that could itself reveal the presence of a rare entity.

Remark 3

(Within-session candidate-pool depletion). To preserve a stable one-to-one correspondence between an original entity and its substitute within a dialogue, Section 4.3 removes each selected substitute from the pool $D_e$ for the duration of that session. This makes $D_e$ deplete as a session progresses, and the static results of Section 5 do not, by themselves, trace the resulting within-session privacy–utility curve: in principle, as the pool shrinks, the closest remaining candidates are exhausted, the realized distance floor drifts, and the probability of falling back to the deterministic placeholder (Remark 2) rises. We acknowledge this effect and bound its severity for the regime we study. The depletion is one entity per substitution per surface form and per type, against pools that are large for the common types: after normalization, deduplication by type, and removal of entities with no in-vocabulary Word2Vec representation, the CoNLL-2003 Person and Organization inventories used here contain 3412 and 2685 entities, respectively. For a query entity, an average fraction ${\rho }_{\theta }=0.90\pm 0.07$ of same-type candidates falls below the $\theta =0.3$ signed cosine-similarity floor, so the effective initial pool size is ${\rho }_{\theta }|{\mathcal{E}}_t|$, roughly 3000 Person and 2400 Organization candidates. A session would therefore have to reuse on the order of thousands of distinct same-type mentions before the pool nears exhaustion and the fallback rate becomes non-negligible, far beyond the entity density of the prompts and sessions in our evaluation. Consequently depletion is gradual rather than linear-to-empty over realistic sessions; and because the fallback is the data-independent, non-leaking placeholder of Remark 2, even the depleted regime degrades the utility of rare repeated entities rather than silently weakening the formal guarantee. For the most adversarial regime, two mitigations preserve the bijection without unbounded depletion: enlarging $D_e$ with additional in-type entities (the dictionary is not intrinsically capped at the CoNLL-2003 inventory), or refreshing the pool per turn while persisting only the realized substitution map. We make the high-density case an explicit limitation in Section 6.

4.5.1. Composition and Group Privacy

Because DES perturbs entities independently, standard composition and group-privacy properties of DP carry over [40]. In particular, if DES is invoked k times on the same user data across a conversation, the resulting privacy cost is upper-bounded by $k·{\epsilon }_{b}D$ under basic composition, with tighter bounds available via advanced composition. Group privacy extends to settings in which a small number of related entities (e.g., a family unit) must be jointly protected, at a linear blow-up in ${\epsilon }_b$. We caution, however, that our parallel-composition argument treats sensitive positions as independent draws. This is a genuine modeling assumption rather than a property of natural language: entities in a prompt are often correlated (a manager and the company she manages), and an adversary who knows such relational priors may exploit them to narrow down a jointly implausible substitution. For instance, a person–organization pair whose substitutes never co-occur. Metric DP bounds the per-pair output-distribution ratio but does not by itself neutralize relational side information; we make this limitation explicit in Section 6 and regard correlation-aware substitution as important future work.

4.5.2. Robustness to Auxiliary Information and Choice of Metric

Because DES’s output distribution does not depend on any side information beyond the public embedding model and the entity dictionary, the guarantees of Theorem 1 hold against adversaries with arbitrary auxiliary knowledge consistent with the public inputs. We emphasize, though, that these guarantees are stated relative to the cosine metric $d_{cos}$ induced by the embedding ϕ. A more powerful adversary equipped with a stronger embedding (e.g., a transformer encoder) may judge two substitutes that are far under Word2Vec to be semantically close, in which case the effective protection is weaker than the ${\epsilon }_b$ computed under $\varphi$. The framework is agnostic to the choice of $\varphi$: instantiating Equation (8) with a stronger, ideally adversary-aligned, embedding tightens the guarantee at the cost of additional computation. We treat the embedding as a tunable, security-relevant parameter and discuss this gap in Section 6. This property is essential for a realistic deployment, where the LLM service provider may have observed prior public data.

As previously discussed, we propose a privacy-preserving approach for LLM-based context-aware reasoning that guarantees the preservation of user privacy while ensuring the effectiveness of LLM inference. To quantitatively account for privacy loss, we introduce Textual Differential Privacy, which facilitates a refined balance between privacy and utility trade-offs.

5. Experimental Evaluation

In this section, we examine the application domains of the TDP paradigm and substantiate its efficacy through a sequence of experimental analyses.

5.1. Use Cases

Context-aware reasoning applications entail integrating LLMs with external data sources to provide relevant context for reasoning. Unlike generalized conversational tasks performed solely by LLMs, these reasoning applications afford domain-specific services while mitigating the hallucination issues inherent to LLMs. We consider two prominent use cases.

• AI Corporate Assistant. The integration of AI corporate assistants within organizational settings introduces benefits such as process optimization, productivity enhancement, and seamless information retrieval. However, leveraging third-party LLMs for such assistants poses significant privacy risks, as these applications frequently require access to internal company data for reasoning and analysis, which may include sensitive corporate information. Encryption would severely impair LLM performance, since context-aware reasoning requires this information to be provided in plaintext as prompts; doing so potentially compromises confidential business information during transmission or processing.
• Business Summarization. Business summarization and tabular question-answering tools are indispensable for distilling complex datasets into actionable insights, thereby empowering decision-making processes within enterprises, often involving domains such as finance or politics. Given the inherently sensitive nature of financial and political data, such applications operate within environments characterized by stringent privacy requirements. Reliance on third-party LLMs for these tasks entails inherent privacy risks: similar to AI corporate assistants, these models may require access to proprietary business data to generate accurate summaries or answers.

In both use cases, the primary privacy risk stems from the reliance on third-party LLMs, which may necessitate the transmission of sensitive data to external servers for processing. This introduces vulnerabilities such as data interception, unauthorized access, and potential misuse of confidential information. To mitigate these risks, users must adopt robust privacy-preserving measures, including data anonymization and on-device processing solutions that minimize the exposure of sensitive data to third-party service providers, while ensuring that LLMs maintain their effectiveness in reasoning.

The proposed framework addresses these privacy concerns by identifying and replacing sensitive entity names within context. This approach anonymizes sensitive data while preserving the logical integrity of the text and represents a crucial step toward mitigating privacy risks in transmitting sensitive information to third-party LLM servers. From a theoretical standpoint, the algorithm operates on the principle of entity recognition within textual contexts. By identifying entities such as company names, financial figures, or political affiliations, the algorithm selectively substitutes these entities with random, unrelated entities without compromising overall coherence or semantics. This process of anonymization serves to obfuscate sensitive information, rendering it less susceptible to unauthorized access or interception during transmission or processing.

One key advantage of this approach lies in its ability to preserve the utility of LLMs in analyzing and reasoning over contextual data while simultaneously safeguarding privacy. By anonymizing sensitive entities, the algorithm enables organizations to leverage the computational power of LLMs for business summarization or corporate assistance without exposing proprietary information to external risks. It is essential, however, to note the potential challenges associated with entity-based anonymization. Adversarial attacks may exploit patterns or linguistic cues to infer the original identities of anonymized entities [60]. To tackle these challenges within our framework, we incorporate anonymization processing for each orchestration of context-aware prompts, entailing the random selection of candidate entities for substitution. This approach prevents adversaries from gathering relevant clues about specific entities, enhancing privacy protection.

5.2. Experimental Setup

We evaluate the TDP paradigm on LLM-based context-aware reasoning. The experiments comprise two tasks that respectively exercise commonsense-based reading comprehension and context-based problem reasoning. The first task leverages the Cosmos QA dataset [61], while the second task is built on the WNUT 17 dataset [62].

• Prompt of Multiple-Choice QA Task: Context: ${context}. Question: ${question}. Which is the correct answer? - (A) ${choiceA} - (B) ${choiceB} - (C) ${choiceC} - (D) ${choiceD}.
• Prompt of Extractive QA Task: Your task is to generate a short summary of the context. Context: ${context}.

We applied the above prompt construction methods to each LLM. The resulting prompts successfully guided the LLMs to reason about the described entities, with the resulting outputs serving as ground truth for subsequent evaluation.

The named entity collection used for replacement is sourced from the CoNLL-2003 dataset, which encompasses four types of named entities: persons, locations, organizations, and miscellaneous entities [63]. Google’s pre-trained Word2Vec model [51] is used for computing ${\mathrm{S}}_{\mathrm{cosine}}$ between words, since linking the distance between words to their semantic similarity is particularly well-suited to anonymization. Because the GoogleNews Word2Vec vocabulary does not cover every CoNLL-2003 surface form, out-of-vocabulary entities are handled by the mechanism as follows: a multi-token entity is represented by the mean of its in-vocabulary token vectors, and an entity with no in-vocabulary token is excluded from the candidate dictionary and, when it appears as an input entity, anonymized through the deterministic type-generic fallback of Algorithm 1. A subword or contextual embedding (e.g., fastText or a transformer encoder) would remove the coverage limitation at additional cost; we therefore treat $\varphi$ as a pluggable component (Section 4.5). We make the prevalence and the cost of this approximation explicit. Of the 1184 entity mentions evaluated in our experiments, 217 ($18.3\%$) were multi-token and therefore required mean-pooling of their in-vocabulary token vectors, and 34 ($2.9\%$) had no in-vocabulary token at all and were routed to the deterministic type-generic fallback of Algorithm 1. Mean-pooling is a deliberately conservative but genuinely lossy operation: averaging the token vectors of a multi-token name pulls its representation toward the centroid of its parts and thus toward a more generic region of the embedding space, so the cosine distance $d_{cos}$ computed for a rare or compositionally complex name is less faithful than for a single-token, in-vocabulary entity. The privacy consequence is one we do not wish to understate: the metric-relative guarantee of Theorem 1 is at its weakest, in expectation, precisely for the most unique entities, because a centroid representation can make a distinctive name appear closer to common substitutes than it truly is, biasing the realized distance floor downward for exactly the entities that are most sensitive. Two points bound this concern in our setting. First, it affects a minority of mentions ($18.3\%$ mean-pooled; $2.9\%$ with no representation, which never enter the candidate computation and instead trigger the non-leaking fallback of Remark 2). Second, it is a property of the chosen embedding $\varphi$ rather than of the mechanism: instantiating Equation (8) with a subword or contextual encoder that represents rare names natively removes the centroid artifact and tightens the guarantee for unique entities, at additional computational cost. We record this embedding-induced accuracy limitation for rare entities explicitly in Section 6.

In the anonymization process, the similarity between the replaced named entity and its substitute should be less than $\theta$, where $\theta$ denotes the similarity threshold. The number of named entity words in each contextual prompt is represented by m. The realized semantic shift therefore satisfies $\overline{\epsilon }=D(x,x^{\prime })/\theta >m(1-\theta )/\theta$ (Equation (10)); we report the lower bound $m(1-\theta )/\theta$ as a conservative per-condition summary. We stress that this is the realized shift, not the formal metric-DP budget ${\epsilon }_b$ of Equation (8). We fix the target threshold $\theta =3\times {10}^{-1}$ for our experiments. This setting reflects the trade-off made explicit by the corrected accounting of Section 4.3: $\theta$ must be small enough to enforce a meaningful semantic-distance floor $1-\theta$ on every release, yet large enough to keep candidate pools non-empty for common person and organization entities (an empty pool triggers the deterministic fallback and contributes to $\delta$). The value $\theta =0.3$ sits at this balance for the GoogleNews embedding and the CoNLL-2003 dictionary; a systematic sweep of utility and $\delta$ as a function of $\theta$ is a natural and important sensitivity study that we did not run for this work and that we flag as future work in Section 6. We conducted experiments on three widely used large language models: Llama 2 [57], ChatGPT-4 [58], and Gemini [59]. The selection of these three models is strategically motivated by their architectural diversity, performance benchmarks, and broad impact; together, they provide a comprehensive cross-section of the current LLM landscape.

The experimental setup comprises five groups: a control group (CG) and four TDP conditions. These conditions anonymize different entity categories: persons (PER), persons & organizations (PER&ORG), persons and locations (PER&LOC), and all three entity types (PER&ORG&LOC).

A clarification must be provided on which instantiation of the mechanism these experiments evaluate. All conditions below use the uniform draw over the candidate pool $C_{\theta }(·)$, that is, the ${\epsilon }_b\to 0$ limit of Equation (8). This is the parameter-free, maximum-entropy baseline of Remark 1, which satisfies the cardinality-based bound of that remark rather than the distance-calibrated metric-DP guarantee of Theorem 1. We adopt it here for two reasons: it requires no choice of budget, and, at a fixed $\theta$, it is the most utility-adverse member of the family, because the exponential mechanism with ${\epsilon }_b>0$ shifts sampling mass onto closer substitutes within the same pool, whereas the uniform draw weights even the most distant in-pool candidates equally. The utility reported in Table 1 and Table 2 is therefore, in expectation and at fixed $\theta$, a conservative lower bound on the utility of the distance-calibrated mechanism for any ${\epsilon }_b>0$. We are explicit that this is an analytical expectation and not a measurement: we have not collected utility data for the ${\epsilon }_b>0$ mechanism, and we record both this gap and the corresponding ${\epsilon }_b$-sensitivity study as a limitation and a future-work item in Section 6.

Table 2. Model performance under TDP for extractive QA on the WNUT 17 dataset (ROUGE-1 recall|ROUGE-1 F1). Scores are self-consistency retention relative to each model’s un-anonymized output, which by construction attains ROUGE-1 $=1$ and serves as the implicit control (CG) upper bound; $\overline{\epsilon }$ is the realized semantic shift (Equation (10)), not the metric-DP budget ${\epsilon }_b$.

Model	PER $\overline{\mathit{\epsilon }}=0.94$	PER&ORG $\overline{\mathit{\epsilon }}=1.19$	PER&LOC $\overline{\mathit{\epsilon }}=1.54$	PER&ORG&LOC $\overline{\mathit{\epsilon }}=1.87$
Llama 2 [57]	0.91|0.93	0.87|0.90	0.85|0.88	0.82|0.85
ChatGPT-4 [58]	0.98|0.99	0.91|0.93	0.87|0.90	0.85|0.89
Gemini [59]	0.91|0.92	0.90|0.92	0.86|0.87	0.85|0.87

5.3. Multiple-Choice QA Results

We selected 200 samples from Cosmos QA that contain named entities and are privacy-relevant, and evaluated the performance of the LLMs within the TDP paradigm on the multiple-choice QA task. The results are reported in Table 1.

We make the selection criterion explicit, since it was underspecified in the conference version and the choice carries a risk of selection bias. The 200 items were obtained by retaining every Cosmos QA development item in which Flair detected at least one Person, Organization, or Location entity within the passage, without any inspection of whether the entity was incidental or load-bearing for the question; the cap of 200 was a budget on paid API calls across three LLMs and five conditions, not a curation step. We acknowledge two limitations that this entails and that we do not want to understate. First, 200 items is a small sample for claims about “minimal impact”; the accuracy differences in Table 1 are within the range that sampling noise alone could produce, so the table should be read as evidence that anonymization does not cause a large degradation, rather than as a precise effect-size estimate. Second, conditioning on entity presence rather than entity salience means our sample is, if anything, easier than a worst case in which the anonymized entity is essential to the answer; we therefore avoid over-claiming and report the location-entity drop, which is exactly the salience-sensitive case, without smoothing it away. A larger-scale evaluation with a salience-stratified sample is part of the future work outlined in Section 6.

The results indicate that anonymizing person-type entities has minimal impact on the performance and accuracy of context-aware reasoning, and that anonymizing organization-type entities also has negligible effects. However, anonymizing location entities tends to cause small fluctuations in performance, because location-specific commonsense knowledge is sometimes involved in the underlying questions. Certain QA items require reasoning that extends beyond exact text spans within the context; in practical applications, selectively anonymizing specific entity categories based on downstream task requirements will therefore significantly reduce the adverse impact on LLM reasoning capabilities.

A subtle but important question concerns what this accuracy measures. Type compatibility is not the same as logical-role compatibility: “Tim Henman” and “Don Mattingly” are both persons, yet a tennis player and a baseball manager occupy different world-knowledge roles, so a semantically distant same-type substitute can yield a context that is locally fluent but globally incongruous (a tennis player “managing” a baseball team), and modern LLMs are sensitive to such world-knowledge inconsistency in ways that could surface as reasoning errors or refusals. This is a genuine risk, and the multiple-choice results above are exactly the metric that would expose it: exact-match accuracy on Cosmos QA rewards correct answer selection, not unigram overlap, so a substitution that pushed the model into an incongruity-induced reasoning failure or a refusal would register directly as a drop in accuracy. The accuracy remains within $0.5$–$1.5$ points of the control group across all three models and all four anonymization conditions (Table 1), which indicates that, for this benchmark, role-incongruity did not translate into systematic reasoning breakdowns. We are nonetheless careful not to over-generalize: this is evidence based on commonsense multiple-choice reasoning, not a guarantee, and the budget ${\epsilon }_b$ of Equation (8) provides the natural control: raising ${\epsilon }_b$ concentrates the sampling mass, hence providing more role-plausible substitutes, and trading a quantum of privacy for greater world-knowledge consistency. We record logical-role plausibility as an explicit limitation and a tuning axis in Section 6.

5.4. Extractive QA Results

In the extractive QA task, the emphasis is on the summarization capability of the LLMs. Due to the scarcity of QA datasets containing a substantial volume of privacy-related named entities, we used the responses from the three LLMs on original prompts as ground truth to construct a generated QA dataset for privacy-related experiments. We then evaluated the performance of anonymization processing under TDP using ROUGE-1 scores [64], as shown in Table 2. ROUGE-1 measures the degree of unigram overlap between the extracted answers and the reference answers, reporting recall and F1.

We want to be precise about what this construction does and does not measure, since it would be circular if read as an absolute-accuracy benchmark. The reference is each model’s own output on the un-anonymized prompt; consequently the ROUGE-1 scores in Table 2 measure consistency under anonymization (how much the substitution perturbs a model relative to its own baseline) and not the factual correctness or reasoning quality of that baseline. If the un-anonymized answer was itself wrong, a high score indicates only that anonymization preserved the (possibly wrong) behavior. This is the intended quantity for our research question, which is the utility cost of the privacy layer rather than the models’ raw competence: by construction the un-anonymized condition scores ROUGE-1 $=1$ (the reference equals itself), so it serves as the implicit CG upper bound against which the reported drops are measured, and every number in Table 2 is a self-consistency retention rate. Pairing the metric with a gold-standard summarization benchmark to disentangle competence from consistency is part of the future work in Section 6.

For scenarios involving only personal information, Llama 2 exhibits the best performance, with recall and F1 close to 1, followed by Gemini and ChatGPT-4. This variability stems in part from the models’ stochastic nature: given the same input, an LLM can produce different outputs across invocations. As privacy protection becomes stricter, the performance of all models decreases. This is partly because the WNUT 17 dataset is primarily designed for NER, leading to a relatively high proportion of named entities across the entire context. Nevertheless, these results suggest that TDP anonymization can protect user privacy in privacy-sensitive QA scenarios while maintaining the effectiveness of extractive QA tasks. In particular, across all three LLMs, the ROUGE-1 F1 remains above $0.85$ even when all three entity categories are anonymized (up to a realized shift $\overline{\epsilon }=1.87$).

The TDP paradigm offers a quantified approach to explore the trade-off between privacy preservation and the performance of LLM-based context-aware reasoning. We provide a quantified measure of the realized semantic shift $\overline{\epsilon }$, alongside the ex ante budget ${\epsilon }_b$, for each controlled condition. Through performance evaluation of the outcomes, better design of corresponding fine-grained privacy preservation strategies can be achieved.

5.5. Comparison with Baselines

There are various methods for anonymizing data, with two notable approaches being anonymization through cryptographic hash algorithms and substitution with pseudo values. We compare these techniques against TDP using a running case study. The input context is sourced from the WNUT 2017 dataset, and the LLM used is ChatGPT-4. The reference prompt and response are:

Prompt: Your task is to generate a short summary of the context. Context: “Don Mattingly will replace Joe Torre as LA Dodgers manager after this season.”

Response: Don Mattingly is set to take over as the manager of the LA Dodgers, succeeding Joe Torre at the end of the current season.

• Hash Algorithm Anonymization. Sensitive data can be anonymized by replacing it with SHA-256 hashes. Maintaining a dictionary allows the re-identification of hashes back to their original values, as illustrated below:

Prompt: Context: “<9d418…f90d3> will replace <e4e4e…7fae0> as LA Dodgers manager after this season.”

Anonymizing prompts in this manner significantly diminishes the effectiveness of LLMs, as even with rigorous prompt adjustment, considerable impediments are introduced into the inference process.

• Pseudo Value Replacement. This represents an optimized anonymization method that is currently widely employed [49]. It furnishes rapid identification and anonymization modules for private entities and is integrated into the RAG framework for privacy protection:

Prompt: Your task is to generate a short summary of the context. Context: “<PERSON_1> will replace <PERSON_2> as LA Dodgers manager after this season.”

In most cases, this anonymization method yields satisfactory inference results. However, numerous scenarios exist in which re-identification fails because the model paraphrases the placeholders. For example, ChatGPT-4 often produces:

Response: The context indicates that a transition in leadership is expected for the LA Dodgers baseball team. Specifically, Person 1 is set to take over as manager, succeeding Person 2, at the conclusion of the current season.

This paraphrased response can render re-identification impossible, thereby breaking the approach. Moreover, the use of such specific pseudo-value patterns makes it easy for attackers to detect that protected sensitive data is present, contradicting the principles of DP.

We present this comparison as an illustrative running example rather than as quantitative evidence. A single sentence cannot establish that one method consistently outperforms another; its purpose here is to make the qualitatively distinct failure modes concrete, not to substitute for a dataset-level comparison. A systematic, statistically powered comparison against these and against formal-DP text mechanisms (DP-BART, ADePT, natural-text sanitization, and embedding-perturbation metric DP) is, as discussed in Section 2.2, an explicit item of future work (Section 6); we have removed any claim of consistent superiority that this example alone cannot support. To be unambiguous on this point: the design of DES does not rest on this example, and we make no claim that DES statistically outperforms Presidio on ROUGE-1 or accuracy. The reason to prefer DES over SHA-256 hashing and over pseudo-value replacement is a definitional one that needs no dataset-level utility contest to establish: both baselines lack any formal privacy semantics, whereas DES is an exponential mechanism that provably satisfies metric differential privacy (Theorem 1) with an interpretable, distance-calibrated accountant. The example illustrates only that the two baselines exhibit qualitatively distinct failure modes (e.g., hashing destroys fluency, pseudo-values invite paraphrase-induced re-identification failure) which motivates the desire for a fluent, formally grounded alternative; it is not offered as evidence that DES wins on utility. A statistically powered, dataset-level utility comparison against Presidio and against formal-DP text mechanisms is a genuine open item (Section 6); pending it, the utility claims we do make are confined to the within-DES, against-own-baseline measurements of Table 1 and Table 2, never to a head-to-head ranking against Presidio.

• Textual Differential Privacy. The TDP paradigm impedes adversaries from tracing privacy information by randomly selecting candidate entities for anonymization. Moreover, DES replaces entities based on named-entity similarity, exerting minimal impact on LLM inference in this example:

Prompt: Your task is to generate a short summary of the context. Context: “Tim Henman will replace Gloria Bistrita as LA Dodgers manager after this season.”

Response: Tim Henman is set to take over as the manager of the LA Dodgers, succeeding Gloria Bistrita at the end of the current season.

After re-identification, the results of context-aware reasoning based on TDP anonymization are consistent with the original in this example. This indicates that, in this case, TDP has not adversely affected the inference capability of the LLM. Of greater significance, we provide a comprehensive paradigm for quantitatively analyzing and accounting for privacy loss in the context of LLM-based reasoning. In the above case, the realized semantic shift is $\overline{\epsilon }=0.71/\theta =2.37$ (the formal budget ${\epsilon }_b$ being fixed ex ante), while the impact on task performance is a ROUGE-1 F1 of 1. This quantitative framework significantly advances the formulation of privacy policies and the evaluation of privacy protection schemes.

5.6. Sensitivity and Ablation Analysis

To further probe the behavior of TDP, we analyze its sensitivity along two dimensions: (i) the similarity threshold $\theta$, which directly controls the candidate pool size and therefore the realized semantic shift through Equation (10); and (ii) the number of sensitive entities m in a prompt, which scales the composed shift across an input. As the per-prompt realized shift satisfies $\overline{\epsilon }>m(1-\theta )/\theta$, decreasing $\theta$ (i.e., enforcing greater semantic distance between originals and substitutes) monotonically increases the accrued $\overline{\epsilon }$, while reducing m, for instance by restricting anonymization to a single entity type, reduces it proportionally. The trends reported in Table 1 and Table 2 are consistent with this analysis: (a) additional entity types (PER → PER&ORG&LOC) produce higher $\overline{\epsilon }$ and a corresponding decrease in downstream utility; (b) the utility drop is noticeably milder on multiple-choice QA than on extractive QA, because the former rewards correct answer selection rather than verbatim reproduction of entity spans; and (c) hash-based anonymization destroys utility regardless of $\theta$, while Presidio-style pseudo values preserve utility on average but incur non-trivial re-identification failures and lack a formal privacy guarantee. Taken together, these observations confirm that DES’s explicit coupling of the realized shift to the cosine distance scaled by $\theta$ yields a more controllable and graceful privacy–utility trade-off than the two baselines we considered. We note explicitly that this analysis is analytical and we regard producing quantitative sensitivity curves, together with the baseline comparisons noted above, as the most valuable empirical extension and list it in Section 6.

6. Discussion

6.1. Privacy–Utility Trade-Off

The central appeal of TDP is that it exposes the privacy–utility trade-off as a continuously tunable design variable rather than a binary choice between transmitting raw PII and destroying semantic coherence. The budget ${\epsilon }_b$ and the threshold $\theta$ jointly control the candidate pool: larger $\theta$ values admit candidates that are semantically closer to the original entity, reducing the realized shift but also reducing the effective information-theoretic gap between the original and the substitute. Smaller $\theta$ values widen the semantic gap, increasing the realized shift but also reducing the candidate pool and potentially making the $\delta$ term non-negligible. Our experiments show that at moderate thresholds ($\theta =0.3$), the resulting realized shift remains modest while downstream ROUGE-1 F1 stays above $0.85$ across three widely deployed LLMs.

A second, complementary axis is entity-category selection: practitioners can choose which entity types to anonymize. Our results indicate that anonymizing person and organization entities incurs very low utility cost across both QA tasks, while anonymizing location entities can incur a modest hit when the underlying question relies on location-specific commonsense. This suggests a deployment recipe in which highly sensitive categories (PII in the strict sense, such as person names and direct identifiers) are anonymized unconditionally, while categories with higher semantic salience to the downstream task are anonymized selectively based on risk assessment.

6.2. Limitations

Despite its advantages, the proposed framework has several limitations. First, TDP protects only the textual elements that are explicitly detected and substituted. Its end-to-end privacy is therefore bounded by the recall of the NER component: missed entities, such as nicknames, abbreviations, rare names, or domain-specific identifiers, are transmitted in plaintext. TDP also does not protect information leaked through non-entity linguistic features, such as writing style, idiosyncratic phrasing, or temporal patterns. Moreover, our evaluation anonymizes only Person, Organization, and Location entities. Other quasi-identifiers that may be highly disclosive in practice, including dates, monetary amounts, account or ID numbers, and addresses, remain outside the current entity set. Accordingly, the utility shifts reported in Table 1 and Table 2 should be interpreted as the cost of protecting these three entity categories rather than as a measure of total document privacy.

Second, the privacy guarantee is inherently dependent on the embedding model, the induced metric, and the available candidate pool. DES inherits its protection from the candidate pool determined by the similarity threshold. If the public embedding model is biased or clusters semantically close entities too tightly, the effective candidate pool may become too small to provide meaningful indistinguishability, requiring a larger $\delta$. The metric-DP guarantee in Theorem 1 is also relative to the cosine metric of the chosen embedding $\varphi$. An adversary equipped with a stronger embedding model, relational priors over correlated entities, or multi-query linkage across a session may infer more information than the computed ${\epsilon }_b$ suggests, even though sequential composition bounds such leakage. This limitation is more pronounced for rare entities: multi-token and out-of-vocabulary names are represented by mean-pooled token vectors ($18.3\%$ of evaluated mentions) or routed to the fallback ($2.9\%$), and mean pooling may smooth distinctive names toward a generic centroid, making the cosine distance least faithful for the most unique entities.

Third, practical deployment introduces statefulness and re-identification challenges. To preserve the entity–substitute bijection within a session, each substitute is removed from $D_e$ after use, causing the candidate pool to shrink over time. For common types, the pools are large (≈3000 Person and ≈2400 Organization candidates below the $\theta =0.3$ floor; Remark 3), so depletion is gradual in realistic sessions. However, very long conversations or high-entity-density documents can exhaust a pool, after which rare-entity utility degrades and the deterministic fallback is invoked more frequently. Pool enlargement or per-turn refresh can mitigate this issue. Conversely, re-identification is fragile to generation behavior: as discussed in Section 4.4, verbatim-lookup re-identification fails when the LLM paraphrases or omits a substitute.

Fourth, substitution can affect prompt form and semantic plausibility. Because entities are selected at the word level but consumed as sub-word tokens, a substitution may change the prompt’s token count and downstream token positions. The observed drift is small in our evaluation (mean $|\Delta \mathrm{tokens}|=0.41$, with $73.8\%$ of substitutions leaving the token count unchanged; Section 4.3), but it is not explicitly controlled by the current mechanism. In addition, type compatibility does not guarantee world-knowledge consistency: a semantically distant same-type substitute can produce a globally incongruous context. Although our multiple-choice accuracy results show no systematic breakdown under this effect (Section 5.3), the risk remains and is governed by the budget ${\epsilon }_b$, which trades semantic distance against role plausibility. TDP addresses prompt-level disclosure at inference time and does not claim protection against training-data memorization, embedding inversion, provider-side side channels, or other attacks outside the stated threat model.

The final limitation concerns the scope of our utility evidence relative to the mechanism that carries the formal guarantee. The accuracy and ROUGE-1 results of Table 1 and Table 2 were obtained with the uniform instantiation (the ${\epsilon }_b\to 0$ draw over $C_{\theta }$ (Section 5.2)) which satisfies the cardinality-based bound of Remark 1 rather than the distance-calibrated metric-DP guarantee of Theorem 1. Strictly, then, these numbers establish that randomized same-type substitution above the semantic-distance floor preserves reasoning. They do not, on their own, measure the utility of the distance-aware exponential mechanism at a specific ${\epsilon }_b>0$, which is the configuration that actually attains the metric-DP guarantee. At a fixed $\theta$ we expect the distance-aware mechanism to be at least as utility-preserving, since increasing ${\epsilon }_b$ concentrates the sampling mass on closer and more role-plausible substitutes within the same pool. This monotonicity makes the reported scores a conservative reference rather than an unrelated one.

6.3. Future Work

Future work should broaden the empirical scope of TDP and strengthen its comparison against related privacy-preserving text mechanisms. A systematic, statistically powered evaluation should compare TDP not only with hash-based and pseudo-value baselines, but also with formal-DP text mechanisms such as DP-BART [47], ADePT [46], natural-text sanitization [45], and embedding-perturbation metric-DP methods [22,44]. Such an evaluation should cover more datasets, domains, and task types, use substantially larger samples than the 200 privacy-relevant items considered in this study, and distinguish between incidental and answer-critical entities through salience-stratified sampling. It would also be valuable to report sensitivity curves showing how utility and $\delta$ vary with the similarity threshold $\theta$ and the privacy budget ${\epsilon }_b$.

Another important direction is to expand the range of protected information. The current framework focuses on NER-defined entities, but practical privacy risks also arise from dates, numerical values, account or identifier strings, addresses, stylistic features, and document-level PII. Extending DES beyond named entities to more general sensitive spans would make TDP applicable to a broader class of privacy-sensitive prompts. This extension should be accompanied by stronger entity and span detection methods, as well as by fuzzy re-identification procedures that remain effective when the LLM paraphrases or omits substituted entities. A lightweight local LLM could support both sanitization and span-level alignment, reducing dependence on an external NER component and mitigating paraphrase-induced re-identification failures.

The substitution mechanism itself can also be refined. The candidate score in Equation (8) could incorporate a sub-word length-matching term, so that among comparably private substitutes the mechanism prefers candidates whose tokenized length matches that of the original entity. This would directly reduce positional drift in downstream prompts. In long or entity-dense sessions, session-adaptive pool management, dynamic enlargement, or per-turn refresh of $D_e$ could help preserve the entity–substitute bijection without exhausting the candidate pool. In addition, role-aware or relation-aware substitution policies could score candidates by world-knowledge plausibility rather than type compatibility alone, reducing globally incongruous contexts caused by semantically distant but type-consistent substitutes.

A further research direction is to improve the privacy metric and strengthen the end-to-end privacy model. Replacing the Word2Vec-based metric with a stronger contextual embedding, ideally one aligned with the adversary model, would narrow the gap between the formal metric used by DES and the semantic information available to an attacker. Combining TDP with complementary training-time mechanisms, such as DP-SGD-based fine-tuning [16,17,18] or DP prompt learning [30], could provide broader privacy protection covering both memorization and inference-time exposure. Finally, adaptive threshold policies that select $\theta$ and ${\epsilon }_b$ per entity or per conversation using an online risk estimator may offer a tighter utility–privacy trade-off in practical deployments. A formal user study on how users perceive re-identification ambiguity in deployed assistants would further clarify the usability implications of TDP.

7. Conclusions

This paper presented Textual Differential Privacy (TDP), a principled framework that embeds differential privacy guarantees directly into the textual context supplied to LLMs for context-aware reasoning. To ensure user privacy, the Differential Embedding Substitution algorithm is employed to de-identify sensitive information extracted through NER, resulting in anonymized contextual prompts whose semantic integrity is preserved. The framework further provides a formal privacy-loss quantification scheme, distinguishing the ex ante metric-DP budget from the realized per-release semantic shift and enabling a more principled navigation of the trade-off between privacy protection and downstream performance. Our theoretical analysis establishes metric differential privacy guarantees for the substitution mechanism, instantiated as an exponential mechanism, and examines its composition and robustness properties. Our empirical evaluation on two standard benchmarks, three widely deployed LLMs, and two representative baselines indicates that TDP can deliver privacy at a modest utility cost, while a formal metric-DP guarantee distinguishes it from the hash-based and pseudo-value baselines.. We are explicit about the boundaries of our claims in Section 6. By treating privacy as a first-class design principle in LLM-based architectures, our work contributes toward a more trustworthy and broadly deployable foundation for intelligent and context-sensitive AI systems.