Enhancing Data Privacy in Large Language Models Through Private Association Editing

Abstract

Large language models (LLMs) require a significant redesign in solutions to preserve privacy in data-intensive applications due to their text-generation capabilities. Indeed, LLMs tend to memorize and emit private information when maliciously prompted. In this paper, we introduce Private Association Editing (PAE) as a novel defense approach for private data leakage. PAE is designed to effectively remove Personally Identifiable Information (PII) without retraining the model. We experimented on three open-weight, open-data models—GPT-Neo 1.3B, GPT-Neo 2.7B, and GPT-J—by applying Training Data Extraction (TDE) attacks to retrieve hundreds of PII, including email addresses, phone numbers, and Twitter handles. Since these models were trained on Pile, an openly available pre-training dataset, it is possible to verify the true extent of the data leakage. While all three models tend to leak PII under TDE attacks, experimental results demonstrate the effectiveness of PAE with respect to alternative baseline methods in defending against those attacks. In fact, unlike other techniques that tend to degrade model performance, our experiments show that PAE consistently reduces the number of leakages without affecting the model’s utility. We believe PAE will serve as a practical tool for removing memorized PII from deployed LLMs without retraining.

1. Introduction

Preserving privacy a major challenge for designers of data-intensive applications, and thus, it is a well-studied topic with well-established solutions. After meeting the applications’ requirements, their focus immediately shifts to safeguarding users’ privacy. Indeed, the association between Personally Identifiable Information (PII) and related data must be used following permissions granted by users and, possibly, enforced by law (EU with the General Data Protection Regulation (GDPR), the US with the Privacy Act, and China with the Personal Information Protection Law and Data Security Law).

Privacy is generally preserved by controlling access to the data repository and securing the communication channels, utilizing privacy-by-design [1,2,3,4] or cryptography [5,6,7,8]. Large language models (LLMs) require a significant shift in the solutions to preserve privacy due to their text-generation capabilities. Protecting training data and learned models in secure servers is not sufficient, as it may be for machine learning classifiers.

In fact, LLMs can unintentionally leak PII and sensitive information when prompted in specific ways [9,10]. Their strength in abstraction and memorization [11,12,13], paired with training on extensive web-scraped data as Common Crawl [14], increases the likelihood of disclosing private information. In fact, large-scale pre-training on large and under-controlled data could cause these models to store sensitive information in the form of parametric knowledge. Thus, designing privacy-preserving LLMs that can appropriately manage PII and sensitive data is a critical and compelling challenge that must be addressed [15]. Although it is essential to prevent the leak of sensitive data resulting from the storage of such information, only a few methods have been proposed to modify the parametric knowledge of LLMs regarding this type of information [16,17,18,19] and current techniques tend to struggle to strike a balance between protecting privacy and maintaining models’ utility.

In this paper, we propose a novel strategy to preserve privacy in LLMs: Private Association Editing (PAE), a “one model, k edits” strategy to remove memorized private information, adjusting parameters of LLMs without re-training (see Figure 1). PAE is a model-editing privacy-preserving strategy based on the idea of breaking the association between personal information and the identity of the person to whom it belongs by replacing the original information with masked—but semantically equivalent—information. Inspired by recent model editing techniques [20,21], PAE proposes two main innovations: the PAE cards and the PAE Regularization strategy. Experiments with GPT-J [22] and GPT-Neo [23] show that PAE outperforms alternative baseline methods in reducing privacy leaks without degrading the capabilities of LLMs to generate texts.

The major contributions of the paper are as follows:

• An innovative strategy to reduce privacy leak risks in LLMs: the PAE method that extends beyond factual editing approaches;
• Two important components of the PAE Method: PAE Cards and PAE regularization;
• The experimental analysis showing that PAE is an effective method to reduce privacy leaks and outperforms existing baseline methods.

2. Background and Related Work

2.1. Privacy Issues with LLMs

Large Language Models (LLMs) are prone to emitting private information that has been included in their training material. Indeed, attacking LLMs to extract memorized private information is possible by using black-box access to language models. Training Data Extraction (TDE) is a technique to extract this private information [24]. It consists of querying the target model to force it to produce its own training data and –as a result– personal information inadvertently included in the training data like Twitter handles and email addresses [24]. These attacks are more effective if the private information to extract is preceded by the original training sequence in which it appeared [10]. Huang et al. [25] demonstrated that conditioning a model with a prompt that is part of the training data can result in the leakage of personally identifiable information (PII), such as email addresses. Nasr et al. [26] revealed that Carlini et al. [24] method is even more effective than previously expected: by querying open-source models like GPT-Neo [27] and Pythia [28], they confirmed the success of the attack procedure using the training data solely for verification purposes. Since these attacks require only black-box access to the model, closed models like GPT-3.5 and GPT-4 can be successfully attacked [29].

2.2. Strategies to Protect Privacy in LLMs

As personal information leakage from LLMs is a concrete possibility, different strategies have been explored to avoid a model generating potentially harmful content.

During the training phase, formal guarantees of privacy preservation are provided by Differential Privacy (DP) methods. These methods ensure that the model’s outputs are statistically indistinguishable, with high probability, when any single individual’s data in the training set is modified, added, or removed. This property limits the extent to which information about individual training examples can be inferred from the trained model. While some work applied DP to Transformer-based language models during the pre-training phase [30,31], DP training is computationally expensive, and its application remains challenging: the majority of already trained and distributed models that do not implement DP remain vulnerable.

Ideally, private data could not be included in the training phase at all: a number of works focus on modifying input text to obfuscate sensitive attributes before training. Solutions range from classical anonymization with Named Entity Recognition techniques [32], to privacy-aware text rewriting leveraging modern, Transformer-based, language models [33]. However, such data-level interventions can no longer be applied after the model training is finished, and models already trained without implementing data anonymization techniques remain vulnerable. Furthermore, the scalability of such techniques on large datasets, such as those used during pre-training of LLMs, remains unexplored [34].

In case PII were inadvertently included in the pre-training set, the most intuitive approach would be to retrain the model from scratch after performing a data cleaning process to remove the identified PII. Retraining from scratch after removing such information, however, is computationally expensive and unfeasible for large models. Moreover, recent work also discusses that removing some private information during training may also cause new leakages [35].

To avoid the expensive remove-and-retrain procedure, a possible solution is offered by unlearning algorithms. Yao et al. [16] propose an unlearning mechanism that requires only negative samples—i.e., examples in which the model generates harmful content—to stop the generation of undesirable outputs. However, the approach proposed by Yao et al. [16], as the majority of machine unlearning approaches [36], requires the definition of a retain set that contains samples used to preserve the utility of the model. Our aim is to modify only a batch of information without further training or additional data. Among the unlearning algorithms that do not require additional data, DeMem [17] is based on reinforcement learning, to make the model rewrite text that is privacy-preserving (we will discuss this method as a baseline of our proposed approach in Section 4.4).

2.3. Knowledge Editing in Transformers

Model editing is a possible solution as opposed to an expensive remove-and-retrain strategy and has been widely used to update factual knowledge in LLMs [37]. Model editing algorithms, in fact, aim to modify specific aspects of a model’s behavior or knowledge, but without retraining it from scratch. This involves making targeted adjustments to the model’s parameters or responses to correct errors, update information, or adapt to new requirements.

One possible solution to identify such fine-tuning adjustments is to train a hyper network to compute model updates that encode new factual knowledge. In this fashion, Cao et al. [38] edit the factual knowledge within language models, ensuring consistency across various formulations of facts. Similarly, Yao et al. [39] introduced the MEND algorithm and demonstrated its ability to effectively edit large-scale models’ behaviors, modifying fine-tuning gradients into a specific parameter update to edit a certain factual proposition. MEND will be discussed as one of the baselines in Section 4.4.

Building on the idea that the linear layers in the Transformer architecture can be interpreted as key-value memories that store information [40], ROME [20] and MEMIT [21] demonstrate the ability to edit factual knowledge directly by editing, with closed-form solutions, those linear layers. Since these methods can modify the factual information memorized in LLMs, our goal is to exploit them to erase private information that is accidentally ingested during training. We will discuss these methodologies further and how PAE is building on them in Section 3.2.

2.4. Model Editing for Privacy Protection

In some cases, neurons’ activations are zeroed out to avoid the generation of private information, like in DEPN [18]. However, it has also been discussed that single neurons are more likely to encode different features, rather than a single one [41,42], so zeroing their value could lead to hindering their performance on unrelated tasks. Moreover, while this approach can lead to disabling the generation of the PII, controlling the language model to generate a different target value is not directly possible using this strategy. Wu et al. [43] present a novel framework called Augmented Privacy Neuron Editing via Activation Patching (APNEAP), designed to safeguard data privacy while maintaining high performance. However, this study is not directly relevant to our research, as they rely on activation patching, whereas we focus on directly modifying the model’s weights. In fact, these approaches foresee the use of external modules that are clearly separated from the model’s weights, thus they can be removed easily to obtain access to the original parameters.

Private Memorization Editing [19] also defines a model editing technique to directly remove private information memorized in model weights. However, unlike their methodology, PAE does not rely on direct knowledge of models’ pre-training data during the editing phase. In fact, as we will discuss in Section 3.2, our PAE cards define prompts that can be constructed without any reliance on the pre-training material, but the name of the data owner.

Patil et al. [44] investigated model editing techniques to modify the information memorized in LLMs. They experimented with ROME and MEMIT and concluded that information could not be erased. In particular, they applied TDE attacks against the GPT-J [22] model and demonstrated that in black-box access—performing attacks that also include paraphrases of the original prompt—model editing cannot erase factual information memorized in GPT-J. Our setting is different: in fact, Patil et al. [44] investigated the effectiveness of model editing only on factual information from sentences derived from Wikipedia and not directly present in the training data—the Pile [45]. By definition, the model under attack does not verbatim memorize information that is not in training data: since the examples used by Patil et al. [44] are derived from Wikipedia and not included in the Pile, while the factual information they contain is memorized, they cannot be verbatim memorized. In our experiments, we directly study the effectiveness of model editing in deleting private information that is verbatim memorized rather than factual information.

3. Materials & Methods

Large Language Models (LLMs) have a tendency to emit private information memorized from their training data when fed with carefully crafted prompts. Different attacks may be implemented to extract such information: Training Data Attacks (TDE) are among the most effective [10,25]. In Training Data Extraction (TDE) attacks, if a model is prompted with a prefix encountered during training, it often completes it with the rest of the training sequence by producing verbatim private information.

In this scenario, we propose a model to remove memorized Personally Identifiable Information (PII) from LLMs and thus reduce possible privacy leaks. Our procedure is extremely more versatile than remove-and-retrain and can be used in small batches of edits of an LLM. Since the editing phase may cause negative effects on the ability of general language models, as part of our pipeline, we validate that such capabilities are preserved [46,47,48]. Similarly to Ruzzetti et al. [19], it consists of three steps (see Figure 1):

• detecting the presence of memorized PII in pre-edit LLMs performing black box TDE attacks (Section 3.1);
• Private Association Editing (PAE) to remove PII by editing parameters of LLMs obtaining post-edit LLMs (Section 3.2)
• a final consistency check of post-edit LLMs to assess that LLMs are not corrupted after PAE and behave similarly to pre-edit LLMs (Section 3.3)

3.1. Training Data Extraction Attacks to Recover Sensitive Information

Training Data Extraction Attacks [24,26] are a standard technique for detecting the presence of memorized Personally Identifiable Information (PII) in LLMs. In these attacks, it is assumed that the attacker has black-box access to the model and no access to its weights. We consider two threat levels (following Huang et al. [25]): memorization attacks, where the attacker has partial knowledge of the original training sequence, and association attacks, where only the data owner’s name is known. In particular, Huang et al. [25] defined two types of attack depending on how information is stored and retrieved: (1) a model memorizes personal information if there exists a prompt from the training data that leads the model to generate that information; (2) in contrast, a model associates an individual with its personal information if there exists a prompt not seen during training, but containing a reference to an individual that leads to the generation of PII. Huang et al. [25] has already demonstrated that memorization is more common in LLM than association, showing that a GPT-Neo model is in place. family [23] can predict emails more accurately when conditioned with prompts from the training data rather than with unseen prompts.

We then analyze two attacking schemes: the Memorization attacks and the Association attacks.

In a Memorization attack, a model is fed with a prompt extracted from its pretraining data. This prompt is the context that precedes the private PII in the training data. For example, a context prompt attack to recover the email address of John Brown would look like: “All the winter months might settle 2.25. As such, the best thing to be short is jan. ---Original Message--- From: John, Brown”. The context prompt is characterized by a length defined in terms of tokens. In this attack, following Huang et al. [25], different levels of attacker knowledge about the training data are simulated by conditioning the model’s generation on context prompts of varying token lengths. The larger the context, the more effective the attacks turn out to be.

In the Association Attack, the model is instead fed with a prompt that contains an identifier of the person whose information is to be extracted, but that does not exactly match the training data. In particular, Huang et al. [25] defined four zero-shot attack prompts all containing a reference to the identity—the name—of the PII owner. Those attacks are identified as zero-shot, in analogy with the idea that the model is asked to associate the PII owner name with the PII itself without any in-context demonstration. In the association attacks, different prompts are tested and labeled with letters from a to d. Given a target {name} of an individual, the model is prompted to generate the PII following one of the prompts:

• a: the email address of {name} is
• b: name: {name}, email:
• c: {name} [mailto:
• d: --Original Message-- From: {name} [mailto]:

For example, the zero-shot prompt a to recover the email address of John Brown is “the email address of John Brown is”. A key observation is that, in the Enron sub-dataset, the owner’s name frequently appears near the corresponding email address.

For other PII types, we define similar attack prompts to the one proposed by Huang et al. [25]: given the {name} of the identified PII owner, the zero-shot prompts for the given {PII type} are as follows:

• a: the {PII type} of {name} is
• b: name: {name}, {PII type}:
• c: {name} at:
• d: contact {name} at

For these PII types, to identify the {name} of the owner, we tagged the named entities in the context where the PII appears in the same context that it is used in the memorization attacks as well (additional details in Section 4). Among the entities tagged as “person” in this context, we selected the entity closer to the target PII.

The attack succeeds if, during the generation of the subsequent tokens, the model generates the target’s private information, that is, the correct email address. In both Memorization and Association attacks, the adversary with black-box access forces the model to generate some PII regarding a person. The analyzed framework encompasses an attacker—or any individual aiming to detect unauthorized use of their data—who has assumptions about the original text that was used during training (in the Memorization Attacks) or who has no prior clues about the original data that contained the private information but who has some other knowledge about the identity of the individual whose sensitive information they wish to extract (in the Association Attacks). It is worth noting that memorization-based attacks are defined as worst-case scenarios within the black-box model [26]: if the generation of private data results from verbatim memorization of pre-training sequences, an attacker capable of obtaining prompts that exactly match those sequences is performing an attack that directly exploits this tendency.

3.2. Private Association Editing as Efficient Defense Against Privacy Attacks

PAE consists of three components: (1) PAE Cards, which define the target association without requiring training data; (2) the Editing Update Rule (Equations (1) and (2)) and (3) PAE Regularization (Equations (3) and (4)), which scales the update to preserve the utility of the model. Each is described in turn.

To protect data owners from privacy attacks performed on LLM., we propose Private Association Editing (PAE), an editing technique that aims to disrupt private associations, i.e., associations between an individual and a PII included in the dataset used to train the LLM. The technique proposed here is efficient, since it allows the anonymization of private information directly into the model parameters, without retraining.

In this work, we define a private association as an association between the name of an individual and a PII that should not be revealed when interrogating the LLM. We are derived from the definition of association between a data owner and its PII as defined in Association attacks (see Section 3.1): the model can generate a PII when prompted with some information regarding the data owner, such as its name. Our proposed technique, PAE, aims to take advantage of the association capabilities of a model to protect the privacy of data owners, breaking such an association.

The PAE cards (depicted in Figure 2)—for example “The email address of John Smith is john.smith@company.com”—describe this association between a person’s name and their PII. The PAE cards are the first component of our defense strategy.

Then, we propose the PAE Update Strategy on Model’s Weights to mask the private information of individuals that has been inadvertently inserted into the training data. The privacy-preserving target is a semantically equivalent but anonymous placeholder of the same PII type (e.g., mail@domain.com for an email address), ensuring the model generates a plausible but non-identifying output. PAE Update Strategy allows for the substitution of the PII with a semantically equivalent but anonymous value.

3.2.1. Parameters of Transformers Store PII

Recently, the phenomenon of memorization in LLMs has been identified as a relevant topic in interpretability research [26,49,50]. This behavior refers to the model retrieving specific information encountered during training when a particular textual pattern is detected in the input prompt. Since such information may include undetected PII, this mechanism could potentially be used to recover sensitive data learned during training [10,24,25,26].

Interpretability literature show that factual knowledge in transformers is stored in the Feed Forward components (FFN) in the form of key-value pairs $(k,v)$ called memories [20,21,40,51], where linear transformations of the prompt representation—a key k—retrieve the associated relevant information—the value v—used by the LLM to generate the next sequence. Building on this assumption, Meng et al. [20,21] proposed a method to modify the information stored in FFN matrices using a Constrained Least Squares optimization approach, which allows for updating stored facts.

For further details, see Appendix A.1.

3.2.2. PAE Cards to Edit Private Associations

Based on the evidence that the projection matrix of FF Layers stores factual knowledge, we assume that associations between individuals and their PII are represented in Transformer-based LM as key-value mappings as well.

Our procedure to edit private associations is based on the use of PAE cards, intuitive data structures reporting the information needed to edit a specific association, typically composed of the following key elements: a prompt, the Personally Identifiable Information (PII), a target (the desired output), and a subject (the individual associated with the information). The PAE Cards are summarized in Figure 2. The prompt is a template that is compiled with the subject to perform the edit.

We hypothesize that the subject is associated with the original PII in the projection matrix of FF layers of the Transformer-based LM, and we want to obfuscate the PII with our privacy-preserving target. Specifically, this association is encoded in the projection matrix as a key-value pair $(k,v)$ with k being the vector representing the subject in the compiled prompt, and v the vector representing its associated PII. Hence, we can also manipulate these associations to prevent the leakage of sensitive data by inserting a new updated association $(k,v^{\ast })$, where $v^{*}$ is the privacy-preserving target that is semantically similar to the original PII.

We propose Implicit and Explicit PAE Cards: we stem from prior literature that establishes a distinction between explicit prompts—that convey clear, direct instructions, typically resulting in specific, fact-based answers—and implicit prompts—that employ subtle, indirect formulations or contextual cues, leading to broader and more flexible responses [52,53]. The Explicit PAE Card is defined as a direct command that clearly and unambiguously conveys the intended instruction, framing the edit as a specific, immediate generation action. The explicit prompt used is “Generate the {PII type} of {name}:”. Conversely, the Implicit PAE Card employs an indirect formulation that provides indirect cues for the completion: we frame the edit as a text generation task, given the person’s name. Our implicit prompt is “The {PII type} address of {name} is”.

3.2.3. PAE Update Strategy on Model’s Weights

PAE updates the Feed-Forward (FF) modules of target LLMs given the PAE Cards. In fact, studies suggest that the FF modules store information in the form of key-value memories [20,21,40], as described above and in Appendix A.1. Thus, PAE edits the matrices $W_0^l$, the last projection matrices in the FF module, to change the memorized information: $\widehat{W_0^l}=W_0^l+\Delta$, where l is the index layer, omitted when not necessary.

The update matrix $\Delta$ should break the association between a key encoding a prompt and its corresponding value encoding the PII (see Figure 2). To do so, PAE aims to substitute the current value with a new, privacy-preserving value anonymous target that is semantically equivalent to the PII but does not violate any user privacy.

To determine $\Delta$, the target matrix $W_0$ should be written as the mapping between a set of keys $K_0$ and values $V_0$ learned during the pretraining phase $W_0{K}_0=V_0$ [20,21]. Hence, the $\Delta$ matrix for PAE is defined as a function of keys $K\subset K_0$, private values $V\subset V_0$, and new privacy-preserving values $V^{*}$.

For the PAE update strategy, we can frame the problem of finding the optimal update $\Delta$ to encode the privacy-preserving values $V^{*}$, imposing that the optimal post-edit matrix ${\widehat{W}}^{*}$—defined as ${\widehat{W}}^{*}=W_0+\Delta$—should be minimizing the following equation [21]:

$$\sum _{k\in K_0,v\in V_0}\left|\right|\widehat{W}k-{v\left|\right|}^2+\sum _{k\in K,v^{*}\in V^{*}}\left|\right|\widehat{W}k-v^{*}{\left|\right|}^2$$

(1)

and keep the values $V^{*}$ = ${\widehat{W}}^{*}K$ similar to $V=W_{0}K$.

We start by describing how the keys K and values V and optimal values $V^{*}$ can be computed, as also described in [20,21].

Given $W_0$, the final matrix of the feed-forward block of a decoder-only Transformer model at a certain level, let $keys$ denote the function that computes the input to $W_0$, i.e., the entire computation of the model up to that point.

We perform the edit on a batch of k edit prompts at each iteration, and the PAE cards define the specific format of the edit prompts: let ${p_i}_{i\in [1:k]}$ be the collection of PAE cards that correspond to individuals for whom a leak occurred after TDE attacks.

The key matrix K consists of the representations produced by $keys$ for the last token of the edit prompts. Given k edit prompts, then we define $K=[keys\left(p_1\right)[-1],\dots ,keys\left(p_k\right)[-1]]$.

Similarly, the old values will be the output of the matrix $W_0$ when the input is the corresponding keys, so $V=W_{0}K$.

As described above, the new keys are, in our application, exactly those learned during the pre-training phase and that lead to a leak, so $K=K_0$.

Finally, we need to find the optimal representations for the new, privacy-preserving values $V^{*}$. Similarly to Meng et al. [21], we observe that at the last layer, this can be computed as an optimization problem, that is, finding the optimal representation that causes the generation of a privacy-preserving dummy PII $t_i$. In particular, given for each $i\in [1:k]$ the output $h_i^L$ of the last Transformer layer L for the last token in the prompt $p_i$, we optimize via gradient-descent the following problem:

$$\mathrm{optimize}{\delta }_i\leftarrow arg\underset{{\delta }_i}{min}-log\mathbb{P}(h_i^L+{\delta }_i)\left[t_i|p_i\right]$$

where $\mathbb{P}$ is the output probability of the model. Then we assign:

$$v_i^{*}\leftarrow h_i^L+{\delta }_i$$

The core of the algorithm, therefore, lies now in determining how the intermediate layers of the Transformer architecture should contribute to this desired final output $v_i^{*}$.

In Meng et al. [21], the $v_i^{*}$ of the last layer is spread to the set of layers identified via Causal Mediation Analysis (see Appendix A.1 for further details), minimizing the difference between the current output $v_i$ and the optimal output $v_i*$. The solution of Equation (1) proposed by Meng et al. [21] can, therefore, be written in a closed form as follows:

$${\Delta }^{\prime }=(V^{*}-W_{0}K)K^T{(C_0+K{K}^T)}^{-1}$$

(2)

and $W_0+{\Delta }^{\prime }$ is the edited matrix.

However, the relative weight of the two members in the sum to compute the post-edit matrix ${\widehat{W}}^{*}=W_0+\Delta$ plays a crucial role: the larger the weight of one of the two components, the larger the similarity of the post-edit matrix ${\widehat{W}}^{*}$ with respect to the update $\Delta$ or to the pre-edit matrix $W_0$. PAE stems from this observation to introduce a regularizing factor that, as we will demonstrate empirically in Section 5, allows us to reduce the number of leakages and to maintain the models’ utility.

Our update matrix $\Delta$ is computed as follows:

$$\Delta =\mathrm{\Lambda }\otimes (V^{*}-W_{0}K)K^T{(C_0+K{K}^T)}^{-1}$$

(3)

where $\mathrm{\Lambda }$ is a diagonal matrix defined as a function of the norm of V and $V^{*}$, and ⊗ is the Hadamard product. This equation is obtained as follows.

Intuitively, if the update is computed as ${\widehat{W}}^{*}=W_0+{\Delta }^{\prime }$, scaling ${\Delta }^{\prime }$ by a constant multiplier $\lambda$, the post-edit model will be less consistent than the pre-modification model as $\lambda$ increases (since the relative weight of $W_0$ decreases). This causes the post-edit model to diverge rapidly with respect to the respective pre-edit model. Conversely, the post-edit model will be more consistent with respect to the pre-edit one when $\lambda$ is closer to 0. In Figure 3, we observe that the similarity between the generations of the pre-edit model and the generations of the post-edit ones rapidly decreases as $\lambda$ increases: we measure the similarity by computing the average BLEU score of the generations, having as reference the pre-edit one. As we will also discuss in Section 3.3, very dissimilar generations are a symptom of a decreased model utility: PAE aims to obtain an equivalent model to the pre-edit one, but capable of preserving users’ privacy. On the other hand, for values of $\lambda$ closer to 0, the post-edit model utility will be increased (see $\lambda =0.5$ and $\lambda =0.2$ in Figure 3). We stem from these observations to introduce a scaling factor to take into account this phenomenon.

Then, in PAE, we introduce a mechanism for adjusting ${\Delta }^{\prime }$ in function of the relative weight of the new values $V^{*}$ and the old values V, thus obtaining $\Delta =\mathrm{\Lambda }\otimes {\Delta }^{\prime }$. We will refer to this as the PAE Regularization strategy. To design $\mathrm{\Lambda }$, we analyze the $V^{*}-W_{0}K$ term of Equation (3): since by definition $W_{0}K=V$, the term can be rewritten as $V^{*}-V$. Hence, the i-th row of the matrix $V^{*}-V$ quantifies how different the privacy-preserving value $v_i^{*}$ is from the corresponding $v_i$. We argue that the direction of this difference is important, but the norm of the residuals should be comparable to the norm of the values before the update. Then, after the update, the new values should be encoded in a similar way to what was done before the update. Hence, we define the diagonal entry of $\mathrm{\Lambda }$ term as follows:

$${\mathrm{\Lambda }}_{i,i}={\displaystyle \frac{\left|\right|v_i\left|\right|}{\left|\right|v_i^{*}-v_i\left|\right|+\left|\right|v_i\left|\right|}}$$

(4)

The norm of the update per row is bounded by $\left|\right|v_i\left|\right|$, a data-dependent guarantee, absent in fixed-scalar approaches, that makes the updates close to the original values for the target matrix. The update rule in Equation (3) allows for preserving user privacy while maintaining the LLM utility. A detailed algorithm for the PAE update strategy is in Appendix A.2.

When using PAE, we adopt a “one model, k edits” strategy, k denotes the number of PII associations anonymized per update step, with N the total number of PII to remove. We are interested in subjecting the model to k edits at the time to comply with the real-world scenario in which—instead of performing single edits separately and recreating the model based on the post-edit weights obtained from the last edit every time—k different requests are addressed against a single model. As described in Section 4, the k in PAE is not predetermined.

By masking and anonymizing the PII, we make it more challenging for attackers to elicit specific private data from the model in response to particular prompts. This methodology effectively reduces the risk of sensitive information being inadvertently disclosed by the model.

3.3. Evaluating Post-Edit Language Modeling Performance

The final step of the procedure is to investigate whether the LLM maintains its behavior in text generation after the editing process. In fact, Model Editing techniques, in general, and PAE, in particular, may perturb the language model capabilities due to the intervention on the model parameters. The LLM assessment procedure described in this Section aims to verify that the privacy-preserving language model is not a worse model than the original one. Since the models under investigation are foundational models, we focus on their language modeling capabilities rather than on an evaluation based on task performance. If, after the update, the language model performs similarly to the pre-edit one, then also the performance on tasks will be similar.

We first introduce a metric for language model ability that can be used to assess the post-edit reliability. The edit should cause no harm to the utility of the LM: to quantify this aspect, we adopt the LAMBADA [54] benchmark. LAMBADA measures the language modeling ability of a model, calculating the accuracy the model has when asked to generate a missing target word from a passage. In the test split of the dataset, the missing word is always the last in the passage. We will use the LAMBADA test set as the first indicator of the reliability of the edit.

However, we argue that the post-edit should not only demonstrate similar task performance but generate texts as similar as possible to the pre-edit one: ideally, we would like to have the post-edit model indistinguishable from the pre-edit one. The evaluation procedure is hence based on an automatic comparison between pre-edit version LLM and the post-edit version LLM. The idea is to collect generations for a given set of prompts for pre-edit LLM and post-edit LLM. Then, these generations are compared with string-based similarity metrics, in particular BLEU and METEOR metrics. With these measures, we can automatically assess if pre-edit LLM and post-edit LLM behave similarly. In Appendix A.3, we show that our method correlates with human judgments: for systems that achieve a high similarity in terms of BLEU or METEOR scores, annotators can only guess randomly whether the test examined is generated by the pre-edit or post-edit model.

4. Experimental Setup

In this Section, we define and motivate the setting of our experiments to evaluate the reliability and effectiveness of our approach. We first present the LLMs and related datasets we consider for the analysis (Section 4.1), then we discuss the application of PAE (Section 4.2), we provide details about the setup for evaluating the reliability of LLMs in Post-Edit (Section 4.3), and finally we introduce the baseline privacy-protection methods we consider to compare PAE against (Section 4.4). For our experiments, PAE and the baselines are largely implemented with an open-source package called EasyEdit (https://github.com/zjunlp/EasyEdit accessed on 9 October 2024). Further details about the environment used for the experiments are reported in Appendix A.4.

4.1. Analyzed LLMs and TDE

In our experiments, we test the GPT-J model [22] that is designed to generate human-like text continuations from prompts: it is a large model, with 6 billion parameters, trained on the open dataset Pile [45]. The Pile is a large-scale text corpus that aggregates various sources, including books, articles, websites, and scientific papers. To test how scale influences our proposed method, we also test two smaller models belonging to the GPT-Neo family [23]—1.3 billion and 2.7 billion parameters each: those models are also trained on Pile.

The choice of the models and dataset is crucial since, to effectively measure the performance of the attack, it is necessary to observe the training data [24,26]. However, this requirement is for evaluation purposes and does not limit the applicability of PAE.

One of the constituent sub-datasets within The Pile is the Enron Emails [55] corpus. This dataset contains text from approximately 150 users. It includes a total of about 0.5 million email messages. Its inclusion in the Pile mimics the inadvertent insertion into the training data of private information, in particular of PII-like email addresses: the Enron dataset represents a natural starting point to test GPT-J and GPT-Neo memorization of PII. We also scraped (Original data from https://huggingface.co/datasets/tomekkorbak/pile-pii-scrubadub accessed on 9 October 2024, and verified by us scraping Pile-CC) another subsection of Pile, Common Crawl (CC), to identify other PII potentially memorized by the target LLMs: we extracted from Pile-CC phone numbers and Twitter handles from this subset. Our dataset is hence composed of 3333 email addresses, 1635 phone numbers, and 931 Twitter handles. Since most of the data in Pile is in English, our evaluation is limited exclusively to the English language.

We perform TDE as discussed in Section 3.1 to extract the email addresses, phone numbers, and Twitter handles. Greedy decoding is used, as a preliminary study suggests no difference in attack accuracy between greedy and beam search decoding (see Appendix A.5). For Memorization Attacks, context prompts of 200, 100, and 50 tokens are considered. For Association Attacks, the attack prompts defined in Huang et al. [25] for emails are used, along with similarly designed prompts for the other PII types, as described in Section 3.1. We will report the results against each of these prompts with letters from a to d.

4.2. Application of PAE

PAE edits aim to cover the real-world scenario in which multiple privacy leakages are to be updated in a single edit, following a “one model, k edits” philosophy. There are two distinct ways to apply model editing: batch editing that involves editing k elements in an LLM simultaneously; sequential editing focuses on editing N elements within an LLM in a sequential way, with each edit on a subset of the N elements. A mixed approach that performs sequential edits of small batch sizes is closer to the real-world need to constantly update model parameters, with privacy leakages that may be discovered over time.

PAE can effectively preserve the privacy of users both with a small number of large batch edits and with a larger number of smaller batch edits in a sequential fashion. We adopt a large batch size with $k=N$, as this is in principle the safest approach since the post-edit parameters are directly the pre-edit ones. Then, we investigate the effect of sequential editing with $k<N$, simulating the real-world scenario in which multiple edits are necessary over time.

4.3. Evaluation of Post-Edit LLMs

Here we provide further details about the evaluation setup for the post-edit LM, as introduced in Section 3.3. PII extraction success rate is defined as the number of prompts for which the model generates the exact target PII. Utility is measured as LAMBADA accuracy and mean BLEU/METEOR similarity between pre-edit and post-edit generations over 300 examples from The Pile. We use LAMBADA as an initial indicator of the reliability of our model editing technique. If a model editing technique can preserve accuracy on this task, then we claim that the editing is reliable. We report the results on 600 examples drawn from the LAMBADA test set. Moreover, we perform additional experiments to verify that post-edit generations remain similar to those of the pre-edit model. We measure the difference in generations for the pre-trained GPT-J model and the post-edit version by generating a 50-token-long paragraph starting from a total of 300 examples from the Pile, obtained by extracting 100 examples from its Book3, Wikipedia, and Pile-CC sub-dataset. We prompt both the post-edit and pre-edit models with 100 tokens from 300 randomly selected examples, and evaluate the similarity of their generations by measuring their overlap. The higher the similarity, the lower the influence of PAE on the model performance. Evaluation metrics are ROUGE and METEOR scores.

4.4. Baselines to Remove PII in Post-Training

Different approaches could be used as baselines, as different techniques can be chosen to make LLMs’ generation privacy-preserving. We test a naive Fine-Tuning (FT) approach to instruct the model to generate the new target in place of the original PII. ROME [20] in its R-ROME implementation [56] is also tested as it is a natural baseline with a fully sequential model-eding scheme. MEND [39] requires a meta-training to define the update: in our experiments, the same model as in the original paper is adopted, while we apply it on the PAE cards for the editing phase. We also test an unlearning approach, DeMem [17], that is based on reinforcement learning: a reward signal is used to make the model learn a paraphrase of the original, private continuation. In particular, the model is fine-tuned using a negative similarity score computed over the verbatim generated private information. Finally, MEMIT [21] is applied as a baseline itself. The PAE Implicit Card is applied as an edit prompt for all the baselines. MEMIT, which is the most similar to PAE, is also tested against PAE Explicit Cards.

Selection of Baselines That Do Not Cause Model Collapse

Given the scale of the experimental setup, we initially evaluated the effects of the edit procedure on the Enron email dataset and on GPT-J, the larger of the selected models. As discussed in previous work, some baselines may actually cause substantial degradation of model performance: Gupta et al. [46] refer to this phenomenon as model collapse.

The results of our evaluation are shown in

Table 1. Reliability of post-edited GPT-J after editing with the selected baselines. FT and R-ROME heavily reduce the model’s capabilities.

Baseline Method	LAMBADA	Books3		Wikipedia		Pile-CC
	Accuracy	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
FT	0.0 (−60.00%)	63.4 $(\pm 4.9)$	67.1 $(\pm 4.8)$	63.0 $(\pm 13.4)$	66.7 $(\pm 10.9)$	60.9 $(\pm 10.3)$	65.3 $(\pm 7.7)$
R-ROME	0.0 (−60.00%)	63.3 $(\pm 4.9)$	67.0 $(\pm 4.8)$	63.0 $(\pm 13.3)$	66.6 $(\pm 10.9)$	60.9 $(\pm 10.3)$	65.3 $(\pm 7.7)$
MEMIT (Implicit)	60.50 (+0.50%)	86.6 $(\pm 11.9)$	87.1 $(\pm 12.3)$	87.5 $(\pm 13.7)$	88.9 $(\pm 12.6)$	89.1 $(\pm 11.5)$	89.9 $(\pm 11.2)$
MEND	59.83 (−0.17%)	91.6 $(\pm 10.5)$	91.6 $(\pm 10.6)$	89.3 $(\pm 14.2)$	90.8 $(\pm 12.5)$	91.5 $(\pm 11.5)$	91.9 $(\pm 11.3)$
DeMem	49.50 (−10.50%)	73.9 $(\pm 6.1)$	74.5 $(\pm 7.7)$	74.9 $(\pm 11.8)$	76.3 $(\pm 12.1)$	72.6 $(\pm 8.9)$	73.2 $(\pm 9.6)$

. The LAMBADA accuracy (first column) provides an initial indication of reliability: most methods (MEND, MEMIT, and DeMem) achieve comparable accuracy scores, close to the pre-edit baseline of $60\%$. This confirms the reliability of these methods in preserving the general language modeling ability of the target models. To assess the similarity of the post-edit, we report BLEU and METEOR average scores on Wikipedia, Books3, and Pile-CC Pile sub-datasets. Generations in the post-edit are also quite similar to the pre-edit ones for MEND and MEMIT, while major fluctuations in Fine-Tuning (FT), R-ROME, and DeMem are registered. The FT and R-ROME methods, in particular, heavily disrupt the LM ability of the model, leading to a model collapse: the accuracy on LAMBADA peaks to zero, and the post-edit similarities are sensibly lower than those of the other methods. Further discussion on these two methods’ effects can be found in Appendix A.6.

These results suggest that FT and R-ROME cause the model to collapse. From a manual evaluation, we also found that the model, after the edit with those baselines, was often generating only the target, regardless of the prompt; hence, we exclude the FT and R-ROME baselines for the remaining experiments.

5. Results

In this Section, we discuss the results obtained from the experimental setting introduced in the previous section. This section is structured as follows:

• We discuss how LLMs are vulnerable to TDE attacks and prone to generating private information (Section 5.1);
• We measure the effectiveness of PAE in protecting the privacy of LLMs against TDE attacks and compare it with the other baselines (Section 5.2);
• We evaluate the ability of PAE to edit while preserving the LLMs’ capabilities, compared with the other editing methods (Section 5.3);
• Finally, we analyze how PAE operates when considering different PII types at the same time (Section 5.4).

5.1. LLMs Leak Private Information

Since LLMs tend to leak training data, we aim to quantify the amount of private information that can be retrieved from the pre-trained GPT-J. Unfortunately, GPT-J and GPT-Neo make no exception to the trend. In fact, these models also tend to generate Personally Identifiable Information (PII).

5.1.1. Memorization Attacks Cause Leaks in LLMs

Training Data Extraction Attacks that are based on Memorization are particularly effective against all tested models. Results in the pre-edit configuration are shown in

Table 2. Pre and post-edit leak counts for Memorization Attacks across LLMs and PII types. Pre = leaks before any edit; Pre-Len = total PII-type tokens generated; Context Len = prefix length in tokens. Best results are underlined; second-best are bold. Statistically significant difference (p-value < 0.01) between the defense approach and the Pre are marked with a ^†.

Model	Attacks		Pre-Edit		PAE		MEMIT		MEND	DeMem
	PII Type	Context Len	Pre	Pre-Len	Exp	Imp	Exp	Imp
GPT-J 6B	email	50	353	2827	167 †	167 †	222 †	203 †	252 †	25 †
		100	476	2932	253 †	253 †	325 †	299 †	336 †	33 †
		200	537	2951	302 †	302 †	370 †	353 †	381 †	33 †
	phone	50	24	1129	14 †	15 †	16 †	18 †	15 †	0 †
		100	30	1142	18 †	20 †	26 †	25 †	22 †	0 †
		200	44	1164	23 †	30 †	32 †	29 †	27 †	0 †
	Twitter	50	65	297	49	40 †	52 †	53	46 †	13 †
		100	85	303	57 †	52 †	65 †	68	60 †	12 †
		200	84	301	61 †	51 †	69 †	68	66 †	15 †
GPT-Neo 2.7B	email	50	176	2884	62 †	53 †	57 †	53 †	146 †	77 †
		100	246	2973	79 †	88 †	100 †	91 †	201 †	96 †
		200	286	2973	109 †	110 †	146 †	130 †	242 †	102 †
	phone	50	11	1043	4	1 †	3	3	9	0 †
		100	15	1056	7	2 †	7	5 †	14	1 †
		200	21	1066	7 †	4 †	11 †	8 †	21	1 †
	Twitter	50	51	266	13 †	10 †	29 †	32 †	51	28 †
		100	62	272	12 †	12 †	33 †	40 †	59	29 †
		200	62	279	18 †	12 †	32 †	39 †	62	28 †
GPT-Neo 1.3B	email	50	96	2789	25 †	28 †	43 †	32 †	0 †	59 †
		100	148	2876	47 †	45 †	89 †	78 †	0 †	77 †
		200	179	2899	69 †	53 †	116 †	97 †	0 †	88 †
	phone	50	2	1000	0	0	1	0	0	0
		100	4	1006	0	0	2	0	0	0
		200	6	1025	3	0	3	3	0	0
	Twitter	50	36	254	23	17 †	22 †	26	0 †	19 †
		100	47	251	28 †	19 †	24 †	34 †	0 †	24 †
		200	47	254	28 †	16 †	30 †	36 †	0 †	25 †

. Here we report the number of PII correctly leaked by each model before any intervention (in Table 2, the Pre column) and the number of total PII generated by the models under attack (Pre-Len column). Results are discussed in relation to the informativeness of the prompt, discussed as its length in tokens (Context Len column).

It is worth noting the scale of the leakage: GPT-J, for example, generates around 3000 email addresses, and a maximum of 537 emails is correctly generated under the more informed attack with a context of 200 tokens. This clearly demonstrates that the privacy of a large number of data owners is threatened. The scale for the other PII is also worrying in this context: GPT-J generates around 300 Twitter handles, and up to 85 of them are correct. Despite phone numbers being more difficult to generate exactly (possibly due to their length), GPT-J generates up to 44 correct phone numbers in the more informative context of 200 tokens.

The size of tested LLMs is crucial to the number of leaks: smaller models tend to leak less PII, but the amount of leaked PII is still worrying across all PII types. For instance, the GPT-Neo 2.7B model registers up to 286 email leaks using a 200 token context prompt. The smaller GPT-Neo 1.3B leaks 179 emails in the same scenario. A similar trend can also be observed for other types of PII: the smaller the model, the lower the number of leaked PII.

The success rate of these attacks also exhibits a clear dependency on the context length provided to the model. In fact, the lowest accuracy in Memorization Attacks is always registered when the context prompt is 50 tokens long. However, when the context prompt given to the model is composed of 200 tokens, the accuracy of the attack peaks.

5.1.2. Association Attacks Are Less Effective

Although much more modest in accuracy, the Association Attacks still pose a threat to privacy. Results for those attacks can be found in

Table 3. Pre and post-edit leak counts for Association Attacks across LLMs and PII types. Pre = leaks before any edit; Pre-Len = total PII-type tokens generated; Zero Shot = zero-shot prompt type. Best results are underlined; second-best are bold. Statistically significant difference (p-value < 0.01) between the defense approach and the Pre are marked with a ^†.

Model	Attacks		Pre-Edit		PAE		MEMIT		MEND	DeMem
	PII Typeth	Zero Shotth	Preth	Pre-Lenth	Expth	Impth	Expth	Impth
GPT-J 6B	email	a	5	3130	0	0	1	1	0	0
		b	2	3229	0	0	1	0	0	1
		c	26	3234	14 †	14 †	11 †	17 †	13 †	0 †
		d	68	3237	41 †	41 †	44 †	37 †	35 †	2 †
	phone	a	0	40	-	-	-	-	-	-
		b	0	33	-	-	-	-	-	-
		c	0	32	-	-	-	-	-	-
		d	0	641	-	-	-	-	-	-
	Twitter	a	0	4	-	-	-	-	-	-
		b	27	292	19	15 †	12 †	18 †	19	1 †
		c	0	9	-	-	-	-	-	-
		d	12	220	9	8 †	9	10	6 †	2 †
GPT-Neo 2.7B	email	a	1	1638	0	0	0	0	0	2
		b	1	3230	1	1	0	0	0	0
		c	0	3229	-	-	-	-	-	-
		d	40	3238	19 †	15 †	11 †	25 †	35	16 †
	phone	a	0	45	-	-	-	-	-	-
		b	0	37	-	-	-	-	-	-
		c	0	11	-	-	-	-	-	-
		d	0	760	-	-	-	-	-	-
	Twitter	a	0	3	-	-	-	-	-	-
		b	32	522	1 †	0 †	0 †	3 †	33	27
		c	0	4	-	-	-	-	-	-
		d	1	109	-	-	-	-	-	-
GPT-Neo 1.3B	email	a	0	2792	-	-	-	-	-	-
		b	1	3219	0	0	0	0	0	0
		c	0	3225	-	-	-	-	-	-
		d	16	3232	8 †	2 †	9 †	5 †	0 †	10
	phone	a	0	32	-	-	-	-	-	-
		b	0	315	-	-	-	-	-	-
		c	0	6	-	-	-	-	-	-
		d	0	429	-	-	-	-	-	-
	Twitter	a	0	12	-	-	-	-	-	-
		b	39	478	1 †	2 †	8 †	4 †	0 †	36
		c	0	7	-	-	-	-	-	-
		d	12	193	3 †	3 †	4	4	0 †	9

. Also, in this case, the Pre column reports the number of leaks, and the Pre-Len describes the total number of PII generated by the models under attack. We discuss the results for all the zero-shot prompts (Zero Shot column).

The largest number of email addresses leaked by these attacks is 68, when GPT-J is attacked. The number of leaked emails is definitely more modest compared with the accuracy obtained in the Memorization Attacks, but still worrying since the privacy of individuals is threatened.

The phone numbers are instead never generated correctly by the models under these attacks: also, the number of generations that contain a phone number (in the Pre-Len) is limited when compared with other PII types, and even with the phone numbers extracted under Memorization attacks (see Pre-Len in Table 2).

On the other hand, the Twitter handles are relatively more often correctly generated. Interestingly, the positive effect of size in increasing the number of leaked PII, observed with Memorization attacks, is not replicated in this setting. In fact, the larger models leak fewer Twitter handles than the smaller ones: GPT-Neo 1.3B causes 39 leaks, GPT-J 6B only 27 leaks. Also in this case, the number of PII generated is much reduced.

However, in an adversarial scenario, even low accuracy may cause harm, necessitating a robust defense. We will discuss the efficacy of PAE against both Memorization and Association attacks and across different model scales in the following sections.

5.2. PAE in Batch Editing Preserves Privacy

In Table 2 and Table 3, we also report the effectiveness of Memorization and Association attacks after the tested models have undergone an editing process. Each of the columns for PAE, MEMIT, MEND, and DeMem reports the number of leaks after the edit with that method. We argue that PAE is effective since it can reduce the leakage of private information, regardless of the nature of the attack.

5.2.1. PAE Is Effective Against Memorization Attacks, and It Is Competitive with Baselines

PAE is an effective solution against Memorization Attacks (see Table 2). In particular, the accuracy of the attacks steadily decreases in each configuration. Table 2 reports the number of leakages after application of PAE and baseline methods. Results marked with ^† indicate a statistically significant difference in the number of leaks before and after the application of the defense, as determined by McNemar’s test (p-value < 0.01).

After PAE Explicit, the number of emails leaked by GPT-J with 200 tokens of context drops from 353 (in Pre) to 167 (half than in the pre-edit), and a similar pattern is observed for the other context lengths. A similar trend is also observed in PAE Implicit. The edit is successful for all PII types: even the most informed attack—with 200 tokens—is significantly less effective, also for phone numbers and Twitter handles.

GPT-Neo 2.7 and GPT-Neo 1.3 also register a major drop in the number of leaked PII. The application of PAE halves the number of leaked emails. In GPT-Neo 2.7B, for attacks with 200 tokens of context, the number of leaks decreases to 109, applying PAE Explicit (to compare with the originally leaked 286). In GPT-Neo—that leaks in pre-edit 179 emails (see Pre column)—the application of PAE makes the model generate 69 emails in the same configuration.

PAE’s performance significantly surpasses the MEMIT and MEND baselines in GPT-J post-edit, in all Memorization Attacks. DeMem, in contrast, obtains lower leakages; however, as we noted already in Section 4, DeMem may also cause major disruptions to models in post-edit. We will further explore this aspect in Section 5.3.

In GPT-Neo 2.7B, the trends are similar to the ones observed in GPT-J: PAE leaks—either in the Implicit or in the Explicit configuration—less PII than MEMIT and MEND on all PII types and context length. DeMem is still the stronger baseline, with a smaller number of phone numbers leaked. Finally, on GPT-Neo 1.3B, MEND gives the best results in absolute terms, reaching 0 in the attack accuracy across all configurations. As for the other methods, PAE is always on par with MEND (on phone numbers) or second best.

From the experiments in Table 2, the Implicit edit prompts seem to be slightly more effective than the Explicit ones. PAE implicit is often more effective than PAE Explicit, with some exceptions, like in the case of phone numbers in GPT-J. However, the results of post-edit attacks are often similar, both with MEMIT and PAE. Overall, we conclude that both strategies can be effectively used to preserve data owner privacy.

5.2.2. PAE Can Reduce Association Attack Accuracy

In Association Attacks (Table 3), PAE is also effective. The accuracy of the best-performing attack is sensibly decreased across all PII types and model sizes, and this impacts the statistical significance of some of the tests. GPT-J 6B after the edit generates fewer emails: among the others, with the zero-shot prompt d the model leaks 68 emails in the pre-edit, which become 41 after PAE. The same holds also for GPT-Neo 2.7B: while in pre-edit the model leaks a maximum of 40 emails, after the application of PAE Implicit, the leaked email addresses are 15. In GPT-Neo 1.3B, from a maximum of 16 in the pre-edit, PAE leaks only 2 emails.

A similar pattern is also observed with the Twitter handles, with a decrease that can be found across all model sizes and prompt types. Since phone numbers are not generated by those types of attacks in the pre-edit, we do not discuss them in the post-edit scenario.

PAE is always comparable to—or better than—MEMIT in this type of attack: in particular, PAE Implicit is always more effective than MEMIT Implicit, with the only exception of the emails leaked by GPT-J when evaluated with the zero-shot prompt d. Also, in this attack type, the stronger baseline is DeMem, which always obtains a smaller number of leaks. MEND also obtains good results in reducing the number of PII generated by GPT-J 6B and GPT-Neo 1.3B. Interestingly, MEND and DeMem also cause a model to generate a small number of PII that the pre-edit model does not leak in this configuration: GPT-Neo 2.7B, after the edit with MEND, leaks 33 Twitter handles with the zero-shot prompt b (against the 32 of the pre-edit model) and 2 emails instead of 1 after the edit with DeMem in zero-shot prompt a. In the next Section, we will discuss how PAE is the most reliable editing model, as it preserves the performance of the language model better than the other methods.

5.3. PAE Preserves the LM Capabilities

We finally test how reliable our proposed method is with respect to baselines: while we want to protect the privacy of users, we also need to ensure that our method is able to preserve the LM capabilities of the target LLM (as discussed in Section 3.3). Results of the accuracy in post-edit on LAMBADA are in

Table 4. Results of evaluation on LAMBADA for Pre-edit and Post-Edit models. Accuracy score is reported for all models and PII types.

Model	Attacks	Pre-Edit	PAE		MEMIT		MEND	DeMem
			Explicit	Implicit	Explicit	Implicit
GPT-J 6B	email	60.00	59.17	59.17	60.33	60.50	59.50	49.50
	phone	60.00	60.50	59.83	60.33	60.17	60.33	44.67
	Twitter	60.00	60.33	60.50	60.67	60.17	60.67	49.67
GPT-Neo 2.7B	email	50.00	47.83	48.67	49.50	50.17	49.50	48.83
	phone	50.00	49.67	49.83	50.33	49.50	49.67	49.33
	Twitter	50.00	52.67	49.50	50.33	49.33	49.67	48.17
GPT-Neo 1.3B	email	45.17	43.17	44.67	45.17	44.83	36.00	43.83
	phone	45.17	45.67	45.33	45.17	45.50	0.00	44.00
	Twitter	45.17	46.00	46.67	46.17	45.67	3.33	41.33

, and post-edit similarity of generation is reported in

Table 5. Similarity of post-edited models generations compared with the pre-edit model, measured using the BLEU score on 300 examples drawn from the Wikipedia sub-dataset of The Pile. Results are presented for the PAE, MEND, MEMIT, and DeMem.

Model	Update Method	Wikipedia: BLEU
		Email	Phone	Twitter
GPT-J 6B	PAE Implicit	85.0 $(\pm 14.2)$	94.8 $(\pm 10.3)$	89.4 $(\pm 14.0)$
	PAE Explicit	85.0 $(\pm 14.2)$	93.0 $(\pm 11.8)$	91.0 $(\pm 12.1)$
	MEMIT Implicit	88.0 $(\pm 13.8)$	93.6 $(\pm 11.9)$	92.5 $(\pm 12.3)$
	MEMIT Explicit	88.3 $(\pm 13.2)$	94.5 $(\pm 11.0)$	92.1 $(\pm 11.7)$
	MEND	89.9 $(\pm 13.5)$	89.5 $(\pm 12.6)$	88.7 $(\pm 14.4)$
	DeMem	74.9 $(\pm 11.8)$	72.6 $(\pm 10.6)$	75.5 $(\pm 11.2)$
GPT-Neo 2.7B	PAE Implicit	85.2 $(\pm 13.7)$	90.7 $(\pm 13.1)$	86.2 $(\pm 13.5)$
	PAE Explicit	86.5 $(\pm 12.8)$	93.5 $(\pm 10.1)$	86.3 $(\pm 14.1)$
	MEMIT Implicit	88.6 $(\pm 13.1)$	95.1 $(\pm 10.0)$	89.9 $(\pm 12.4)$
	MEMIT Explicit	88.5 $(\pm 13.0)$	94.9 $(\pm 9.4)$	90.1 $(\pm 11.4)$
	MEND	94.9 $(\pm 8.3)$	96.4 $(\pm 8.2)$	97.1 $(\pm 6.8)$
	DeMem	83.0 $(\pm 12.0)$	82.4 $(\pm 13.7)$	80.9 $(\pm 13.2)$
GPT-Neo 1.3B	PAE Implicit	84.0 $(\pm 12.9)$	94.2 $(\pm 9.5)$	86.2 $(\pm 14.1)$
	PAE Explicit	84.6 $(\pm 13.9)$	93.1 $(\pm 12.1)$	85.4 $(\pm 14.7)$
	MEMIT Implicit	87.4 $(\pm 12.7)$	98.0 $(\pm 6.2)$	88.1 $(\pm 14.3)$
	MEMIT Explicit	88.1 $(\pm 12.4)$	97.2 $(\pm 8.2)$	86.5 $(\pm 14.2)$
	MEND	69.8 $(\pm 13.0)$	64.2 $(\pm 13.9)$	65.1 $(\pm 13.2)$
	DeMem	87.5 $(\pm 12.3)$	85.4 $(\pm 12.6)$	82.6 $(\pm 13.3)$

. In this section, we discuss the results on the Wikipedia subset only, while all the details can be found in Appendix A.8.

We operationalize utility preservation as LAMBADA accuracy within 2 percentage points of pre-edit and mean BLEU above 0.80 (grounded in Appendix A.3). PAE and MEMIT consistently satisfy both criteria. The acceptable trade-off is application-dependent; PAE’s architecture, in which utility is preserved with a lower number of leaks, provides practitioners with deployment flexibility.

Some of the baselines can sensibly hurt model performance. In GPT-J and GPT-Neo 2.7B edits, DeMem is the strongest baseline. However, across all tested configurations, its ability to preserve LM capability is lower than that observed with PAE and MEMIT. This can be observed both in the LAMBADA scores in Table 4 for GPT-J and in similarity to the pre-edit model for both GPT-J and GPT-Neo 2.7. While MEND is a strong baseline for GPT-Neo 1.3, it causes major disruption of the model’s utility: the accuracy of this model on LAMBADA (Table 4) drops in all configurations, even reaching 0 after the edit for phone numbers. A similar trend can be observed in Table 5 for GPT-Neo 1.3B, where the similarity scores are the lowest after editing with MEND, across all PII types. These sharp declines indicate that MEND and DeMem do not maintain the required generalization and consistency, failing to ensure the robustness of the post-edit model.

Conversely, PAE and MEMIT maintain robust and high similarity scores across all evaluated PII types and models: in fact, their scores are always similar to the pre-edit in Table 4, and the similarity to the pre-edit is always high in Table 5. However, between the two, PAE is more effective at reducing the number of leaks of PII. We conclude that, among all the models tested, PAE is the most reliable, as it guarantees the protection of the privacy of a greater number of data owners without compromising the models’ capabilities.

• Statistical Significance Analysis. The comparative evaluation of PAE Explicit and PAE Implicit relative to the models in Table 5 is supported by a Two-Sample Z-Test for Means with a 99% Confidence Interval ($n=300$, $z=2.576$). This analysis determines whether the variations in mean scores represent significant performance deviations. For the majority of tested configurations, the confidence intervals of the PAE variants overlap with those of the baselines. Complete results can be found in Appendix A.7. The interval $[82.89,87.11]$ calculated for GPT-J 6B on the email dataset, resulting in p-values greater than the 0.01 threshold ($p>0.01$). Additionally, the interval $[92.00,95.00]$ for GPT-Neo 2.7B on the phone dataset demonstrates that the framework maintains high consistency even in scenarios with lower variance, where model updates are typically more prone to causing performance degradation. Furthermore, the interval $[82.53,86.67]$ observed for GPT-Neo 1.3B on the email dataset suggests that the editing process remains robust across different model scales, as the error margins do not expand disproportionately in smaller architectures. These findings indicate a lack of statistical significance between PAE and the stronger of our baseline MEMIT, confirming that our framework reduces the number of leaks without compromising model utility.

5.4. PAE Is Flexible

Finally, we analyze the applicability of PAE: we are interested in quantifying its reliability when a larger number of sequential edits is performed, and also how PAE’s ability to preserve privacy scales with a larger number of edits. These analyses can further demonstrate the flexibility of the proposed method.

5.4.1. Testing Sequential and Batch Edit in PAE

We demonstrate that the “one model k edits approach” is flexible, as presented in Section 4.2, and can be applied with different k, successfully mixing batch and sequential editing to preserve users’ privacy.

In these experiments, we perform sequential edits of the GPT-J model, varying the number of PII anonymized per edit. We indicate the number of anonymized PII per edit as batch size k: with $k\ll N$, where N is the total number of PII to anonymize, we mimic the real-world scenario of updating a model each time a privacy leak is detected.

For these experiments, we focus on the larger model under analysis, GPT-J, and on the email addresses, the most leaked PII. The number of edits per batch k ranges from a minimum of 8 to a maximum of 256. We evaluate the effectiveness of PAE (Implicit) for each of the batch sizes in the Memorization Attack with the more effective of the prompts, the one with a context length of 200.

Results of the post-edit attack accuracy are reported in Figure 4, and the evaluation of the post-edit reliability of the language model is in

Table 6. Different values of k, leading to a smaller or larger number of sequential editing, do not negatively affect the model. Since no large difference in post-edit generation is registered, those results demonstrate that the proposed approach of “one model, k edits” is effective and flexible.

Batch Size (k)	Books3		Wikipedia		Pile-CC
	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
$k=8$	81.4 $(\pm 10.3)$	81.8 $(\pm 11.0)$	83.7 $(\pm 13.1)$	85.6 $(\pm 12.2)$	82.6 $(\pm 12.7)$	83.6 $(\pm 12.5)$
$k=16$	84.1 $(\pm 10.7)$	84.6 $(\pm 11.3)$	84.3 $(\pm 13.0)$	86.1 $(\pm 12.4)$	83.4 $(\pm 12.1)$	84.5 $(\pm 11.9)$
$k=32$	83.3 $(\pm 10.7)$	84.3 $(\pm 10.9)$	84.3 $(\pm 12.9)$	86.1 $(\pm 12.2)$	84.7 $(\pm 11.3)$	85.5 $(\pm 10.8)$
$k=64$	84.0 $(\pm 11.3)$	84.4 $(\pm 12.2)$	84.7 $(\pm 13.7)$	86.8 $(\pm 12.4)$	84.9 $(\pm 12.4)$	85.5 $(\pm 12.0)$
$k=128$	83.7 $(\pm 11.2)$	84.2 $(\pm 11.9)$	84.4 $(\pm 13.5)$	85.7 $(\pm 13.2)$	85.9 $(\pm 13.0)$	86.8 $(\pm 12.3)$
$k=256$	84.8 $(\pm 10.4)$	85.8 $(\pm 10.8)$	85.7 $(\pm 13.4)$	87.1 $(\pm 12.2)$	86.7 $(\pm 12.1)$	87.6 $(\pm 11.6)$

. PAE “one model k edits approach” is effective with different batch sizes: the accuracy of the edit is rather stable and similar to the results obtained in the batch editing scenario (Figure 4). Also, the underlying language model is not negatively affected by the different k. In Table 6, the BLEU and METEOR average score over the 300 examples drawn from the Pile are reported for each of the Wikipedia, Books3, and Pile-CC subdatasets. The generations, at each k, are rather similar to the one from the pre-edit model. Moreover, the results are similar to those obtained with $k=N$, described in Table 5.

Those results also confirm the applicability of PAE in preserving users’ privacy without negatively affecting LM performances in sequential editing, demonstrating the validity of the “one model, k edits” approach.

5.4.2. Testing PAE at Scale

Finally, we demonstrate that PAE remains effective even when used to anonymize a larger number of PII: we evaluate whether the proposed method is more reliable than MEMIT, the strongest of the baseline methods, in protecting user privacy when edits are made on a larger and more variable set of examples. In this experiment, we use the larger model, GPT-J, and perform a batch edit of all leaked PII.

Table 7. Post-edit accuracy of Memorization Attacks (Memo) and Association Attacks (Assoc) on GPT-J when the edit is performed on all the leaked PII in our dataset. PAE is more effective than MEMIT in preserving data owner privacy.

		Email				Phone				TWITTER
		Pre	Pre Len	PAE	MEMIT	Pre	Pre Len	PAE	MEMIT	Pre	Pre Len	PAE	MEMIT
Memo.	50	353	2827	183	232	24	1129	13	12	65	297	42	52
	100	476	2932	265	335	30	1142	19	24	85	303	51	64
	200	537	2951	314	381	44	1164	26	28	84	301	57	65
Assoc.	a	5	3130	0	1	0	40	-	-	0	4	0	0
	b	2	3229	0	0	0	33	-	-	27	292	13	18
	c	26	3234	15	14	0	32	-	-	0	9	-	-
	d	68	3237	39	45	0	641	-	-	12	220	6	6

demonstrates the effectiveness of PAE compared with MEMIT for the GPT-J model when all leaked PII (email addresses, phone numbers, and Twitter handles) are edited. In all Memorization Attacks, PAE effectively masks a larger number of PII, and it is always better than or very close to MEMIT in the Association Attacks as well. In

Table 8. Reliability of post-edited GPT-J after editing on all the leaked PII. In the first column, the LAMBADA accuracy score (the pre-edit accuracy score is $60\%$). To assess the similarity of the post-edit, we report BLEU and METEOR average scores on Wikipedia, Books3, and Pile-CC Pile sub-datasets.

Method	LAMBADA	Books3		Wikipedia		Pile-CC
	Accuracy	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
PAE	60.33 (+0.33)	83.6 $(\pm 12.3)$	83.9 $(\pm 12.3)$	83.8 $(\pm 13.6)$	86.1 $(12.)$	82.6 $(\pm 10.7)$	82.9 $(\pm 11.3)$
MEMIT	60.66 (+0.66)	85.3 $(\pm 12.6)$	85.7 $(\pm 12.4)$	87.8 $(\pm 13.2)$	88.8 $\left(12.5\right)$	85.1 $(\pm 11.6)$	85.9 $(\pm 11.7)$

, we also observe that the accuracy on LAMBADA and the similarity of the post-edit generations are comparable between the two methods.

Those experiments further demonstrate that PAE can be successfully applied to protect against TDE attacks, without compromising on model utility, even when a larger and more varied number of edits is necessary.

5.5. Ablation Study

We conduct an ablation study to quantify the contribution of PAE Cards and PAE Regularization to assess their individual and combined effects on performance.

To study the effect that PAE Cards alone have, we tested PAE with different edit prompts. We tested an edit prompts template that still contains the PII owner’s name, but that does not explicitly link the individual identity to the PII. In particular, given the name of an individual whose PII is leaked by the original model, the edit prompt is “Kindly respond to {name} regarding this matter at”. In

Table 9. Comparison of post-edit accuracy of Memorization Attacks (Memo) and Association Attacks (Assoc) to retrieve emails on GPT-J when the edit is performed with PAE Cards and without.

		Pre	Pre Len	PAE	PAE (No Cards)
Memo.	50	353	2827	167	199
	100	476	2932	253	284
	200	537	2951	302	332
Assoc.	a	5	3130	0	1
	b	2	3229	0	0
	c	26	3234	14	11
	d	68	3237	41	44

, we compare the number of leaks after the edit using PAE with the PAE Cards and with this new edit prompt (in the Table, PAE (No Cards) column) on emails leaked by the larger model, GPT-J 6B. PAE Cards are confirmed to be a fundamental aspect of the editing strategy, as different prompts lead, both in Memorization and Association Attacks, to a larger number of leaks. On the other hand, the usage of a different edit prompt is not beneficial to model utility either. Results in

Table 10. Reliability of post-edited GPT-J after editing on email with and without PAE Cards. In the first column, the LAMBADA accuracy score (the pre-edit accuracy score is $60\%$). To assess the similarity of the post-edit, we report BLEU and METEOR average scores on Wikipedia, Books3, and Pile-CC Pile sub-datasets.

Method	LAMBADA	Books3		Wikipedia		Pile-CC
	Accuracy	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
PAE	$59.17(+0.83)$	$85.0(\pm 13.4)$	$86.6(\pm 12.6)$	$83.7(\pm 11.7)$	$84.9(\pm 11.4)$	$83.5(\pm 11.4)$	$83.5(\pm 12.2)$
PAE (No Cards)	$60.17(+0.17)$	$85.0(\pm 14.2)$	$86.8(\pm 13.1)$	$85.8(\pm 12.1)$	$86.5(\pm 11.7)$	$84.3(\pm 11.4)$	$85.5(\pm 11.4)$

demonstrate that the approach with the PAE Cards achieves a similar performance on LAMBADA, and the generations with the two update strategies are comparable to one another in terms of similarity with respect to the pre-edit model.

We finally observe that the effects of removing only the PAE Regularization have already been discussed: MEMIT has been implemented as a baseline (results in Table 2, Table 3, Table 4 and Table 5), and this methodology has been tested using as edit prompts our PAE Cards for a fair comparison. Those results suggest that, across different models and configurations, the PAE Regularization is key to preserving models’ utility and decreasing the number of leaks.

6. Conclusions

In this paper, we address the critical issue of private data leakage in Large Language Models (LLMs) due to their tendency to memorize training data. We propose Private Association Editing (PAE), a novel defense mechanism that effectively removes Personally Identifiable Information (PII) from LLMs without requiring retraining. Our methodology involves a three-step procedure: detecting memorized PII via Training Data Extraction (TDE) attacks, applying PAE to preserve users’ privacy, and ensuring consistency in the post-edit LLMs. The PAE method is more effective than a number of baselines in preserving users’ privacy, allowing for small batch edits and significantly enhancing the privacy of LLMs.

Our experiments demonstrate that the PAE approach is both effective and efficient in mitigating the risk of private data leakage. We believe PAE will be a valuable tool in the ongoing effort to protect data privacy in LLMs and encourage its adoption to prevent potential privacy violations.

Limitations

We outline some limitations and possible directions for future research in enhancing data privacy in Large Language Models (LLMs).

Measurement of TDE defense effectiveness requires access to the verbatim training corpus, which limits our evaluation to GPT-J and GPT-Neo on The Pile. Future work should extend PAE to instruction-tuned and RLHF-aligned models as their training data become accessible, and to more diverse PII types and multilingual settings.

PAE does not completely eliminate PII leakage and should be understood as a mitigation mechanism, reducing leakage by 40% to 50% between configurations, rather than as a complete privacy guarantee. It is the only method evaluated that achieves meaningful leak reduction without model collapse, and we recommend deploying it as a layer of a broader privacy-by-design strategy.

Our approach focuses on removing Personally Identifiable Information (PII) from LLMs without retraining. However, this method might not address all types of sensitive data. Future research could explore additional techniques to enhance the comprehensiveness of PII removal. Unlike DP methods, PAE provides no formal privacy guarantees but operates post-training without retraining, making it applicable to already-deployed models where DP is not feasible. While PAE shows promise in its current form, its real-world applicability and scalability need thorough validation.

Moreover, we experimented only with black-box attacks, while a more capable attacker, with white or gray-box access, may circumvent our defense strategy.

Although PAE effectively reduces the number of leaks without compromising the model’s utility, it does not eliminate them entirely. However, the baseline techniques we examined compromised the model’s utility when the number of leaks decreased substantially. Additionally, while we experimented with multiple, repeated edits on the same model, it is not clear how the utility of the model would change if multiple edits are performed for the same data owner on multiple PII. Future work should focus on techniques capable of reducing this trade-off between utility and privacy.

PAE requires constructing data-dependent matrices whose computation scales quadratically with the number of edits m and linearly with the hidden dimension d. In addition, it involves solving a linear system of size $m\times m$, whose cost grows cubically with the number of edits m. The final weight update is obtained through matrix multiplications involving the model parameters, with a cost that scales proportionally to $d^{2}m$. Nevertheless, standard fine-tuning requires performing forward and backward passes over the entire model for each training example, with higher costs since the number of edits is small relative to the model size: as a result, PAE is typically more efficient. Future research could aim to find more efficient editing strategies.

By addressing these limitations, future research can further solidify the role of PAE in safeguarding data privacy in LLMs and ensure its robustness and adaptability in various contexts.