Privacy Auditing in Differential Private Machine Learning: The Current Trends

Abstract

Differential privacy has recently gained prominence, especially in the context of private machine learning. While the definition of differential privacy makes it possible to provably limit the amount of information leaked by an algorithm, practical implementations of differentially private algorithms often contain subtle vulnerabilities. Therefore, there is a need for effective methods that can audit $(ϵ,\delta )$ differentially private algorithms before they are deployed in the real world. The article examines studies that recommend privacy guarantees for differential private machine learning. It covers a wide range of topics on the subject and provides comprehensive guidance for privacy auditing schemes based on privacy attacks to protect machine-learning models from privacy leakage. Our results contribute to the growing literature on differential privacy in the realm of privacy auditing and beyond and pave the way for future research in the field of privacy-preserving models.

1. Introduction

In today’s data-driven world, more and more researchers and data scientists are using machine learning to develop better models or more innovative solutions for a better future. These models often tend to use sensitive (e.g., health-related personal data and proprietary data) [1] or private data (e.g., personal identifiable information, such as age, name, and user input data), which can lead to privacy issues [2]. When using data containing sensitive information, the individual’s right to privacy must be respected, both from an ethical and legal perspective [3]. The functionality of privacy modeling for the privacy landscape ranges from descriptive queries to training large machine-learning (ML) models with millions of parameters [4]. Moreover, ML subsets of deep-learning algorithms can analyze and process large amounts of data collected from different users or devices to detect unusual patterns [5]. On the other hand, ML systems are exposed to several serious vulnerabilities. This logically leads to the consideration that training ML models are vulnerable to privacy attacks. Therefore, it is crucial for the practical application of ML models and algorithms to protect the privacy of input datasets, training data, or data that must be kept secret during inference.

Numerous works have shown that data and parameters of ML models can leak sensitive information about their training, for example, in statistical modeling [6,7,8,9]. There are several causes of data leakage, such as overfitting and influence [10], model architecture [11], or memorization [12]. If your personal or important data are used to train an ML model, you may want to ensure that an intruder cannot steal your data. To measure and reduce the likelihood of sensitive data leakage, there are various mitigation and protection strategies.

A robust framework for protecting sensitive data in statistical databases, especially through mechanisms such as noise addition and gradient clipping, is a proven mathematical framework called differential privacy (DP) [2,13]. The core idea of DP is to add noise to the data or model parameters to obscure an individual’s influence on a data release [14], where the unit of privacy characterizes what you are trying to protect. To clarify DP, it must be provably guaranteed that an attacker is not able to reliably predict whether or not a particular individual is included in the dataset. Consequently, such an approach can provide a strong privacy guarantee for individuals. In this context, DP is a powerful privacy-preserving tool to prevent sensitive information about an individual from being revealed in a variety of ML models and to analyze the privacy of/from the training data.

The integration of DP methods into ML models makes them more robust to privacy attacks and paves the way for differential private machine learning (DPML) [15,16,17,18]. In this regard, ML models using DP algorithms could guarantee that each user’s contribution to the dataset does not result in a significantly different model [19]. However, the advantages of ML models’ accuracy with DP’s strong privacy guarantees and ease of decentralization [20] come with a price [21], especially when aiming for a low-privacy parameter [22]. For example, models trained with the Differentially Private Stochastic Gradient Descent (DP-SGD) privacy algorithm [23] show a significant decrease in accuracy compared to non-DP models [24,25]. The main reason for this could be that the privacy analysis of existing DPML methods and algorithms (e.g., DP-SGD) is overly cautious in real-world scenarios.

Ensuring privacy in DPML raises the following key questions: How can we guarantee the privacy of the model? Does our model reveal private information? What level of differential privacy does an algorithm satisfy? Answering these questions is crucial, because overestimating the privacy guarantee leads to a decrease in the accuracy of the model, while underestimating it leads to a privacy leakage [26].

To prevent privacy leakage from ML models, we use a DP framework that adds a calculated amount of noise or randomness to hide each individual’s contribution to the data, thus reducing the risk of privacy leakage from small changes in a dataset [27]. A common approach is to add noise to the data during the training process [28]. The process of determining how to add noise is called mechanism in the context of DP and can be influenced by several factors, including the specific noise distribution (e.g., Laplacian and Gaussian mechanisms), the desired level of the privacy, and the type of query. DP can also facilitate effective data-partitioning strategies when sensitive information is distributed across multiple datasets or partitions. By ensuring that each portion adheres to DP standards, organizations can analyze aggregated data without compromising individual privacy. These strategies are used, for example, when data cannot be centralized due to privacy concerns (e.g., federated learning) [29]. There are other approaches where noise is added to the inputs, outputs, ground truth labels, or even to the whole model [30]. As a result, the algorithm can still learn from the data and make accurate predictions about decisions. Adding noise provides a strong worst-case privacy guarantee for ML algorithms [31]. Moreover, there is a crucial technique in the DP context—gradient clipping, which is used in training ML models. It helps to ensure that the contribution of individual data points to the model’s gradients remains bounded, improving privacy guarantees while preserving the performance of the model. The purpose of gradient clipping is twofold. First, bounding the gradients reduces the sensitivity of the model output to individual training examples, as doing so is essential for ensuring DP. Second, gradient clipping helps prevent overfitting by avoiding extreme updates that could lead to the memorization of specific data points. Although DP is a formalization stating that a query should not reveal that an individual is present in a trained dataset. It should be noted that there are recent approaches in which ML models are trained non-privately and their predictions are de-noised before being released to satisfy DP [32]. This means that DP gives the user a quantitative guarantee of how distinguishable an individual’s information can be to a potential attacker.

Differential privacy [2,33] ensures that running the algorithm on two adjacent datasets, $D$ and $D^{\prime }$, results in two approximately equal distributions that differ in one data point, and that the two distributions are approximately equal. The privacy level is often characterized by the privacy parameters (also known as privacy risk): $ϵ$, i.e., the privacy loss; and $\delta$, i.e., the probability of deviation from the privacy guarantee. Together, these parameters form a mathematical framework for quantifying privacy and allow the fine-tuning the privacy level to balance data utility and privacy concerns. Choosing appropriate privacy parameters is challenging but crucial, as weak parameters can lead to excessive privacy leakage, while strong parameters can compromise the utility of the model utility [34]. A small $ϵ$ ensures that an attacker cannot reliably distinguish whether the algorithm has processed $D$ or $D^{\prime }$; that is, it provides strong privacy but less accuracy. Meanwhile, a large $ϵ$ provides weaker privacy guarantees [35,36]. This parameter controls the trade-off between privacy and utility. Since there are no guidelines on how to set the right amount of ϵ and δ in practice, this can be a challenging process. Even when implemented correctly, there are several known cases where published DP algorithms with miscalculated privacy guarantees incorrectly report a higher level of privacy [37,38]. In order to provide the expected privacy guarantees for the DPML model, the privacy auditing must be used.

Privacy auditing—the process of testing privacy guarantees—relies on multiple model training runs in different privacy configurations to effectively detect privacy leakage [39,40]. There are many reasons why one would want to audit the privacy guarantees of a privately differentiated algorithm. First, if we check and the audited value of ϵ is greater than the (claimed) upper bound, the privacy proof is false, and there is an error or bug in our algorithm [34,41]. Second, if we audit and the audited value of ϵ matches, then we can say that our privacy proof is a tight privacy estimate or tight auditing, and our privacy model does not need to be improved [42]. Tight auditing refers to the process of empirically estimating the privacy level of a DP algorithm in a way that closely matches its theoretical privacy guarantees. The goal is to obtain an accurate estimate of the actual privacy provided by the algorithm when applied to real-world data. Existing auditing scenarios for DP suffer from the limitation that they provide narrow estimates under implausible worst-case assumptions and that they require thousands or millions of training runs to produce non-trivial statistical estimates of privacy leakage [43]. Third, if we are unable to rigorously prove how private our model is, then auditing provides a heuristic measure of how private it is [44].

In practice, the differential privacy audit [45,46,47,48,49,50] of ML modeling has been proposed to empirically measure and analyze the privacy leakage through the DPML algorithm. To investigate and audit the privacy of data and models, you must first apply a specific type of attack, called privacy attack, to a DP algorithm and then perform an analysis, for example, a statistical calculation. To evaluate data leakage in ML, we categorize privacy attacks into membership inference attacks [51,52,53], data-poisoning attacks [54,55], model extraction attacks [56,57], model inversion attacks [58,59], and property inference attacks [60]. In addition, assumptions must be made about the attacker’s knowledge and ability to access the model in either black-box or white-box settings. Finally, the attacker’s success is converted into an $ϵ,\delta$ estimate using an attack’s evaluation procedure. The privacy attack, together with the privacy assessment, forms an auditing scheme. For example, most auditing schemes [46,47,61,62] have been developed for centralized settings.

Motivation for the research. The aim of this review paper is to provide a comprehensive and clear overview of the study of privacy auditing schemes issued in the context of differential private machine learning. The following aspects are considered:

• The implementation of differential privacy in consumer-use cases makes greater privacy awareness necessary, thus raising both data-related and technical concerns. As a result, privacy auditors are looking for scalable, transparent, and powerful auditing methods that enable accurate privacy assessment under realistic conditions.
• Auditing methods and algorithms have been researched and proven effective for DPML models. In general, auditing methods can be categorized according to privacy attacks. However, the implementation of sophisticated privacy auditing requires a comprehensive privacy-auditing methodology.
• Existing privacy-auditing techniques are not yet well adapted to specific tasks and models, as there is no clear consensus on the privacy loss parameters to be chosen, such as ϵ, algorithmic vulnerabilities, and complexity issues. Therefore, there is an urgent need for effective auditing schemes that can provide empirical guarantees for privacy loss.

Contributions. This paper provides a comprehensive summary of privacy attacks and violations with practical auditing procedures for each attack or violation. The main contributions can be summarized as follows:

• We systematically present types and techniques for privacy attacks in the context of differential privacy machine-learning modeling. Recent research on privacy attacks for privacy auditing is categorized into five main categories: membership inference attacks, data-poisoning attacks, model inversion attacks, model extraction attacks, and property inference.
• A structured literature review of existing approaches to privacy auditing in differential privacy is conducted with examples from influential research papers. The comprehensive process of proving auditing schemes is presented. An in-depth analysis of auditing schemes is provided, along with an abridged description paper of the papers.

The rest of this article is organized as follows: The following section provides an overview of the relevant background of the theoretical foundations of differential privacy, including its mathematical definitions and basic properties. Section 3 describes the types of privacy attacks on ML models before evaluating privacy leakage in the context of differential privacy. Section 4 presents various privacy auditing schemes based on privacy attacks and privacy violations, along with some influential paper examples. Section 5 discusses the manuscript and provides future research trends.

2. Preliminaries

In this section, the brief theoretical and mathematical foundations for differential privacy are presented.

2.1. Differential Privacy Fundamentals

An individual’s privacy is closely related to intuitive notions of privacy, such as release of privacy unit and privacy loss [63]. The privacy unit (e.g., person) quantifies how much influence a person can have on the dataset. The privacy loss quantifies how recognizable the data release is. The formalization of differential privacy is defined in relation to the privacy unit and the privacy loss. The DP framework can ensure that the insertion or deletion of a record in a dataset has no effect on the query results, thus ensuring privacy. To satisfy the DP, a random function called “mechanism” is used. Any function can be a mechanism as long as we can mathematically prove that the function satisfies the given definition of differential privacy. The relevant definitions, proof, and theorems are presented below.

DP mechanism: DP relies on rigorous mathematical proofs to ensure privacy guarantees. These foundations help us to understand the behavior of DP models and determine the privacy loss [64,65]. DP is defined in terms of privacy unit (input) and privacy loss (output) of a randomized function. The description of this function that satisfies DP is called the mechanism $\mathcal{M}(·)$ [66].

Adjacent datasets: Two datasets, $D$ and $D^{\prime }$, are adjacent if $D\subset D^{\prime }$ differs by the change in a single individual $\left|D\right|\pm 1$. To determine whether your data analysis is a DP data analysis, you must provide data transformations that contain each function from a dataset to a dataset. For example, if you are using functions to help you understand your data, the properties or statistics you are using are statistical queries.

Inbounded and bounded DP [67,68]: If the dataset is not known, you are operating under unbounded DP (e.g., the sets of possible datasets is of any size). In contrast, if the dataset is known, you are operating under bounded DP (e.g., the sets of possible datasets are known size).

Pure DP [2]: In the original DP, a mechanism, $\mathcal{M},$ satisfies $ϵ$-DP if for all pairs of adjacent datasets, $D$ and $D^{\prime }$, differing by one individual, and for all possible sets of outputs, $\mathit{S}$, of the algorithm, the following identity is as shown:

$$\mathrm{Pr}\left[\mathcal{M}\left(D\right)\in \mathit{S}\right]\le e^{ϵ}\times \mathrm{P}\mathrm{r}\left[\mathcal{M}\left(D^{\prime }\right)\in \mathit{S}\right],$$

(1)

where $\mathrm{P}\mathrm{r}$ denotes probability, $ϵ$ is the privacy budget (also known as a privacy risk or a privacy loss parameter) representing the degree of privacy protection, and $e^{\epsilon }$ is amount of information leakage or the maximum difference between the outcomes of two transformations.

Approximate DP: In approximate DP, a small failure probability of error, $\delta$, is added to pure DP to relax the constraint:

$$\mathrm{Pr}\left[\mathcal{M}\left(D\right)\in \mathit{S}\right]\le e^{ϵ}\times \mathrm{P}\mathrm{r}\left[\mathcal{M}\left(D^{\prime }\right)\in \mathit{S}\right]+\delta$$

(2)

This makes it easier to design practical algorithms that keep the privacy guarantees perfect with higher utility, especially when the dataset is large. If $\delta =0$, we can achieve a stricter notation of $ϵ$-differential privacy.

The privacy loss [36,69]: Let $\mathcal{M}(·)$ be a mechanism and $D$ and $D^{\prime }$ adjacent datasets, the privacy loss for a given output, $y:$

$$L\left(\mathcal{M}\left(D\right),\mathcal{M}\left(D^{\prime }\right), y\right)=\ln \left({\displaystyle \frac{\mathrm{Pr}\left[\mathcal{M}\left(D\right)=y\right]}{\mathrm{Pr}\left[\mathcal{M}\right(D^{\prime })=y]}}\right)=\le ϵ$$

(3)

The privacy loss quantifies how assure a potential attacker can be based on the odds ratio of the two possibilities. In this way, the distance between the output distributions for a given $y$ can be measured. In other words, the pair of output distributions provides the distinguishability of the mechanisms. If the loss is zero, the probabilities match and the attacker have no advantage. If the loss is positive, the attacker chooses dataset $D$. If the loss is negative, the attacker chooses dataset $D^{\prime }$. If the loss magnitude is large, there is a privacy violation.

Hypothesis test interpretation of DP [70]: DP can be interpreted as a hypothesis test with the null hypothesis that $\mathcal{M}$ was trained on $D$ and the alternative hypothesis that $\mathcal{M}$ was trained on $D^{\prime }$. False-positive result (type-I error) is the probability of rejecting the null hypothesis when it is true, while false-negative result (type-II errors) is the probability of failing to reject the null hypothesis when the alternative hypothesis is true. For example, Kairouz et al. (2015) characterised $(ϵ,\delta )$-DP in terms of the false-positive rate (FPR) and false-negative rate (FNR) that can be achieved by an acceptance region. This characterisation enables the estimation of the privacy parameters as follows:

$${\displaystyle \frac{TPR-\delta }{FPR}}<e^{ϵ},$$

(4)

Furthermore, from the hypothesis testing perspective of hypothesis testing [66] (Balle et al., 2020), the attacker can be viewed as performing the following hypothesis testing problem given the output of either $\mathcal{M}\left(D\right)$ or $\mathcal{M}\left(D^{\prime }\right):$

H₀: The underlying dataset is $D$.

H₀: The underlying dataset is $D^{\prime }$.

In other words, for a fixed type I error, $\alpha$, the attacker tries to find a rejection rule, $\varphi$, that minimises the type II error, $\beta$.

Private prediction interface [71]: A prediction interface, $M$, is $ϵ$-differential private if for any interactive query generating algorithm, $Q$, the output ($Q\leftrightarrows M\left(F\right)$) is $ϵ$-differential private with respect to model $F$, where ($Q\leftrightarrows M\left(F\right)$) denotes the sequence of queries and responses generated in the interaction of $Q$ and $M$ on model $F$.

Rényi DP (RDP) [72]: This DP extends the standard concept of DP by allowing for a continuum level based on the Rényi divergence. In RDP a randomized mechanism, $\mathcal{M}$, satisfies $(\alpha ,\overline{ϵ})$-RDP if for all neighbouring datasets, $D$ and $D^{\prime }$, the Rényi divergence of order $\alpha$ between the distribution of the outputs of the algorithm on $D$ and $D^{\prime }$ is bounded by $ϵ$:

$${\mathcal{D}}_{\alpha }\left(\mathcal{M}\right(D\left)\right|\left|\mathcal{M}\right(D^{\prime })\le \overline{ϵ}$$

(5)

The global sensitivity of a function [66]: The sensitivity, $∆f$, of a function is the maximum absolute distance between scalar outputs, $∆f\left(D\right)$ and $f\left(D^{\prime }\right)$, over all possible adjacent datasets, $\left(D\right)$ and $\left(D^{\prime }\right)$:

$$∆f={max}_{D,D^{\prime }}|∆f(D)-f(D^{\prime }\left)\right|$$

(6)

If the query dimension of the function $d=1,$ the sensitivity of function $f$ is the maximum difference between the values that $f$ may take on a pair of the adjacent datasets.

Differential privacy mechanisms. One way to achieve $ϵ$-DP and $(ϵ,\delta )$-DP is to add noise sampled from Laplace and Gaussian distributions, respectively, where the noise is proportional to the sensitivity of the mechanism. In general, there are three main mechanisms for adding noise to data used in DP, namely the Laplace mechanism [2,73,74], the Gaussian mechanism [66,75,76], and the exponential mechanism [13]. It should be noted that the Laplace mechanism provides $ϵ$-DP and focuses on tasks that return numeric results. The mechanism achieves privacy via output perturbation, i.e., modifying the output with Laplace noise. The Laplace distribution has two adjustable parameters, its centre and its width, $b$. The Gaussian mechanism yields $(ϵ,\delta )$-DP. Considering the proximity of the original data, the Laplace mechanism would be a better choice than the Gaussian mechanism, which has a more relaxed definition. It should be noted that the exponential mechanism is usually used more for non-numerical data and performs tasks with categorical outputs. When ϵ is small, the transformation tends to be private. The exponential mechanism is used to privately select the best-scoring response from a set of candidates. The mechanism associates each candidate, $c$, via a scoring function, $q(x,c)$.

Upper bound and lower bound [77]: The DP algorithm is accompanied by a mathematical proof that gives an upper bound for the privacy parameters, $ϵ$ and $\delta$. In contrast, a privacy audit provides a lower bound for the privacy parameters.

2.2. Differential Privacy Composition

Three core properties of differential privacy are defined for the development of suitable algorithms to preserve privacy and fulfil data protection guarantees. They play a central role in understanding the net privacy cost of a combination of DP algorithms. A crucial property of differential privacy is the composition of differentially private queries [50,78] bounds on privacy guarantees.

Sequential composition [2,79] is the most fundamental, in which a series of queries are computed and released in a single batch. It limits the total privacy cost of obtaining multiple results on DP mechanisms with the same input data. Suppose a sequence of randomized algorithm, ${\{\mathcal{M}}_1\left(D\right), {\mathcal{M}}_2\left(D\right),\dots , {\mathcal{M}}_k\left(D\right)\}$, which consists of $k$ sequential steps, which is performed with the privacy budget, $\{{ϵ}_1,{ϵ}_2,\dots ,{ϵ}_k\}$, on the same given dataset, $D$, then the output mechanism is $G\left(D\right)={(\mathcal{M}}_1\left(D\right), {\mathcal{M}}_2\left(D\right),\dots , {\mathcal{M}}_k\left(D\right))$, which satisfies ${\sum }_{i=1}^k{ϵ}_i$-DP.

Parallel composition [13] is a special case of DP composition when different queries are applied to disjoint subsets of the dataset. If $\mathcal{M}\left(D\right)$ satisfies $ϵ$-DP and dataset $D$ is divided into $k$ disjoint parts, such as $d_1\cup ,\dots ,\cup d_k=D$, then the mechanism that releases all results, $\mathcal{M}(d_1,\dots ,d_k)$, satisfies $ϵ$-DP. In this case, the privacy loss is not the sum of all, $ϵ$, but rather the ${ϵ}_{max}$.

Postprocessing immunity [80] means that you can apply transformations (either deterministic or random) to the DP release and know that the result is still differentially private. If $\mathcal{M}\left(D\right)$ satisfies $ϵ$-DP, then for any randomised or deterministic function $g=\left(\mathcal{M}\left(D\right)\right)$ satisfies $ϵ$-DP. Postprocessing immunity guarantees that the output of DP mechanism can be used arbitrarily without additional privacy leakage [81]. Since postprocessing of DP outputs does not decrease privacy [61], we can choose a summary function $\tau$ to preserve as much information about $\mathcal{M}$ as possible.

2.3. Centralized and Local Models of Differential Privacy

The two main common models [82,83] for ensuring data privacy, each with different applications and mechanisms, are the central model and the local models. In differential privacy, the user datasets are noised either at the data center after receiving client’s data or it is noised by each user of the data locally.

The classic centralized version of DP requires a trusted curator who is responsible for adding noise to the data before distributing or analyzing. In centralized differential privacy, the data and model are collocated, and the noise is added to the original dataset after it has been aggregated in a trusted data center. A major problem with centralized differential privacy setting is that users still need to trust a central authority, namely the administrator of the dataset, to maintain their privacy [84]. There is also the risk of a hostile curator [85].

In the local differential privacy model, the data are made differentially private before they come under the control of the curator of the dataset [86]. Noise is added directly to the user’s dataset [87] before it is transmitted to the data center for further processing [88]. In the trade-off between privacy and accuracy, both centralized and local paradigms of DP can reduce the overall accuracy of the converged model due to the randomization of information shared by users [85].

Another taxonomic approach is the general distinguish DP into single-party learning and the multi-party learning [89]. Single-party learning means that the training data of each data owner is stored in a central place. In multi-party learning, on the other hand, there are several data owners who each keep their own datasets locally and are often unable to exchange raw data for data protection reasons.

2.4. Noise Injecting

In the ML pipeline, there are multiple stages at which we can insert noise (perturbation) to achieve DP: (1) on the training data (the input level), (2) during training, (3) on the trained model, or (4) on the model outputs [90].

Input perturbation. At the input level, we distinguish between central or local settings [91]. Input perturbation is the simplest method to ensure that an algorithm satisfies the DP. It refers to the introduction of random noise into the data (into the input of the algorithm) itself. If a dataset is $D=\left\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots {\mathbf{x}}_n\right\}$ and each record ${\mathbf{x}}_i$ is a d-dimensional vector, then a differential private $\mathbf{x}$ is denoted as $\tilde{\mathbf{x}}=\mathbf{x}+noise$, where $noise$ is a random d-dimensional vector.

Output perturbation. Another common approach is output perturbation [92], which obtains DP by adding random noise is introduced to the intermediate output or the final model output [50]. By intermediate output, we mean the middle layers of the neural networks, while the final model output implies the optimal weight obtained by minimizing the loss function. The differential private layer is denoted as $\tilde{h}=h+noise$, where $h$ represents the hidden layers in a neural network.

Objective perturbation. In objective perturbation, random noise is introduced to the underlying objective function of the machine-learning algorithm [50]. As the gradient is dependent on the privacy-sensitive data, randomization is introduced at each step of the gradient descend. We can imagine that the utility of the model changes slightly when the noise is added to the objective function. A differential private objective function is represented as $\tilde{L}\left(D,w\right)=L(D,w)+noise$.

Gradient perturbation. In gradient perturbation [19], noise is introduced into the gradients during the training process to solve the optimal model parameter using gradient descent methods. The differential private gradient descent is $w_{t+1}=w_t+{\eta }_t(\lambda w_t+\nabla L\left(x_i,w_t\right)+noise)$, where $\lambda$ is a regularization parameter, and $\eta$ is a learning rate.

Each option provides privacy protection at different stages of the ML development process, with privacy protection being weakest when the DP is introduced at the prediction level and strongest when it is introduced at the input level. Keeping the input data private in different ways means that any model trained on that data will also have DP guarantees. If you introduce DP during training, only that particular model will have DP guarantees. DP at the prediction level means that only the model’s predictions are protected, but the model itself is not differentially private. Note that if perturbations (noise) are added to data to protect privacy, the magnitude of this noise is often controlled using norms (also called scaling perturbations with norms).

3. Privacy Attacks

In this section, the alternative privacy attacks related to differential privacy auditing are presented, and their classification is proposed, evaluating the privacy guarantees of the DP mechanisms and algorithms.

3.1. Overview

The assets that are sensitive and potentially threatening the ML models target either the training data, the model, its architecture, its parameters, and/or the hyperparameters. They can take place either in the training phase or in the inference phase. During model training, the attacker attempts to infer and actively modify the training process or the model. While privacy attacks provide a qualitative assessment of DP, they do not provide quantitative privacy guarantees, nor do they detect exact differential privacy violations with respect to the desired ϵ [84].

Based on a comprehensive overview of the current State of the Art in privacy-related attacks and the proposed threat models in DP, the different types of attacks for DPML auditing typically include (1) black-box and white-box attacks (also known as settings), (2) the type of attack, and (3) the centralized or local DP settings. There is an extensive body of literature tailoring attacks to specific ML problems [93,94,95].

3.2. White-Box vs. Black-Box Attacks

Depending on the capability and knowledge of the attacker and the analysis of potential information leaks in DPML models, they are generally divided into two main areas: black-box and white-box attacks (also known as settings) [84,95]. If the attacker has full access to the target model, including its architecture, training algorithm, model parameters, hyperparameters, gradients, and data distribution, as well as outputs and inputs, we speak of white-box attacks. On the other hand, if an attacker evaluates the privacy guarantees of differential private mechanisms without accessing their internal workings and only has access to the output of the model with arbitrary inputs, it is a black-box attack [26,48]. In this type of attack, the attacker can only query the target model and obtain the outputs, typically confidence scores or class labels.

White-box privacy attacks [96,97], in the context of DP, involve scenarios where the attacker has full access to the model parameters, architecture, and training data. The attacker can exploit this detailed knowledge to create an attack model that predicts whether specific records were part of the training dataset based on internal model behavior. Usually, the attacker tries to identify the most vulnerable space (e.g., the feature space) of the target model by using the available information and modifying the inputs. They may also analyze gradients, loss values, or intermediate activations to derive insights of information status. Wu et al. [96], for example, focus on implementing a white-box scenario where the attacker has full access to the weights and outputs of the target model. If this is the case, we can speak of the strong capability of the attacker on the model. Steinke et al. [48] implement white-box auditing in their auditing framework, where the attacker has access to all intermediate values.

Black-box privacy attacks [84,95] involve scenarios in which the attacker has limited access to a ML model, typically only being able to observe and retrieve its output for specific inputs. This means that the attacker has no knowledge of the internals of the target model. In this scenario, the vulnerabilities of the model are identified using information about the past input/output pairs. Most black-box attacks require the presence of a predictor [84]. In black-box environments, only the predicted confidence probabilities or hard labels are available [97]. In privacy auditing through black-box access, the attacker only sees the final model weights or can only query the model [96]. Black-box attacks are also used to detect privacy violations that exploit vulnerabilities in differential privacy algorithms and lead to privacy violations [26].

Grey-box privacy attacks in DPML represent a middle ground between white-box and black-box attacks, where the attacker has limited access to the model’s internals. This type of attack occurs when an attacker has some knowledge of the model, such as to specific parameters or layers, but not to the complete internal workings.

Attacks on collaborative learning assume access to the model parameters or the gradients during training or deal with attacks during inference. In cases where the attacker has partial access, these are called grey-box attacks (partial white-box attacks) [98,99]. We consider attackers to be active if they interfere with the training in any way. On the other hand, if the attackers do not interfere with the training process and try to derive knowledge after the training, they are considered passive attackers. It is important to add here that most work assumes that the expected input is fully known, although some preprocessing may be required.

3.3. Type of Attacks

In the ML context, the attacker attempts to gain access to the model (e.g., to the parameters) or intends to violate the privacy of the individuals in the training data or perform an attack on the dataset used for inference and model evaluation. In this study, privacy attack techniques are categorized into five types: membership inference attack, data-poisoning attacks, model inversion attacks, model extraction attacks, and property inference attacks [46,62]. The most general form corresponding to the assessment of a data leakage is membership inference: the inference of whether a particular data point was part of a model training set or not [52]. Far more powerful attacks, such as model inversion (e.g., attribute inference [100] or data extraction [101,102], aim to recover partial or even complete training samples by interacting with an ML model. In the ML context, inference attacks that aim to infer private information from data analysis tasks are a significant threat to privacy-preserving data analysis.

3.3.1. Membership Inference Attack

A membership inference attack (MIA; also known as training data extraction attack) is used to determine whether a particular data point is part of the training dataset or not [10,51,52,103,104]. In other words, MIA tries to measure how much information about the training data leaks through the model [34,90,105]. The success rate of these attacks is influenced by various factors, such as data properties, model characteristics, attack strategies, and the knowledge of the attacker [106]. This type of attack is based on the attacker’s knowledge in both white-box and black-box settings [96]. The earlier works in MIAs use average-case success metrics, such as the attacker’s accuracy in guessing the membership of each sample in the evaluation set [52]. The MIAs typically consist of four main types:

• White-box membership inference.
• Black-box membership inference.
• Label-only membership inference.
• Transfer membership inference.

In white-box MIAs, the attacker (also known as a full-knowledge attacker) [84,107] has access to the internal parameters of the target model (e.g., gradients and weights), along with some additional information [10]. For example, the attacker has access to the internal weights of the model and thus to the activation values of each layer [108]. The goal of the attacker is to gain access to the model parameters and gradients to identify differences in how the model processes training and non-training data. The main techniques are gradient-based approaches, which exploit the gradients of ML models [41,109] to infer whether a specific data point was part of the training dataset, which can reveal training data by examining gradients for target data points, as the gradients for such data can often differ from the activations during training; and activation analysis [61], as activations for training data may differ from activations for non-training data in certain layers, which can be used as a signal to detect membership. In addition, introducing white-box MIAs in the ML model, it may also be possible to analyse internal gaps of the model and the use of features by exploiting the internal parameters and hidden layers of a model, as these often reveal training data [96].

In the black-box setting, the attacker simply queries the ML model with input data and observes the output, which can return either confidence-based scores, hard labels, or class probabilities. The attacker exploits the differences in the ML model between training data and unseen data, often leveraging high confidence scores for training data points as a signal of membership. In a black-box setting this type of attack is carried out through techniques such as shadow model training [90] or confidence-based score analysis [110].

By training a series of shadow models—local surrogate models—attacker obtains an attack model that can infer the membership of a particular record in the training dataset [111]. This is performed by training a model that has the same architecture as the target model but uses its own data samples to approximate the training set of the target model. The attack only requires the exploitation of output prediction vector of the model and is well feasible against supervised machine-learning models. Instead of creating many shadow models [112], use only one to model the loss or logit distributions for members and non-members.

In confidence score analysis (also known as confidence-based attacks), the attacker analyzes the confidence scores returned by the model by comparing the confidence in the trained samples with the untrained samples (unseen data). The attacker has access to the labels and prediction vectors to obtain confidence scores (probabilities) for the queried input. This approach is mainly investigated in works such as [52,113]. Carlini et al. [103] use a confidence-based analysis that is maximized at low false-positive rates (FPRs). To improve performance at low FPRs, Tramèr et al. [49] introduce data poisoning during training. However, these attacks can be computationally expensive, especially when used together with shallow models to stimulate the behavior of the target models [114].

In the label-only MIAs, the attacker only has access to the model’s labels to determine whether a specific data sample was part of the model’s training set. The attacker uses only the predicted labels to infer membership under input perturbation [20,115], often by leveraging inconsistencies in the model’s predictions between training and non-training data. Standard label-only MIAs often require a high number of queries to assess the distance of the target sample from the model’s decision boundary, making them less effective [54,116]. There are two main techniques in label-only MIAs: adaptive querying, where the inputs are slightly modified to see if the model changes the label, which could indicate that the data were part of the training set; and meta-classification, which means that a secondary model is trained to distinguish between the labels of training and non-training data to infer membership.

The transfer membership [117] is a case where direct access to the target model is restricted. The attacker can train an independent model with a similar dataset or use publicly available models that have been trained with similar data. The attacker’s goal is to train a local model that approximates the behavior of the target model and use this local model to launch MIAs. There are two main techniques: model approximation [118,119], which means that the attacker approximates the decision boundary of the target model and uses black-box settings to infer the membership of the surrogate model; and adversarial examples [120], meaning that the attacker can generate adversarial examples that are more likely to be misclassified by non-training points to improve the accuracy of membership inference.

MIA is widely researched in the field of ML and could serve as a basis for stronger attacks or be used to audit different types of privacy leaks. Most MIAs follow a common scheme to quantify the information leakage of ML algorithms over training data. For example, Ye et al. [113] compare different strategies for selecting loss thresholds. Yeom et al. [10] compares the use of MIA for privacy testing with the use of a global threshold, τ, for all samples. Later, attack threshold calibration was introduced to improve the attack threshold, as some samples are more difficult to learn than others [22,103]. Another approach to MIA is defined by an indistinguishably game between a challenger and an adversary (i.e., privacy auditor) [105]. The challenger tries to find out whether a particular data point or a sample of data points was part of the training dataset used to train a particular model. A list of privacy attacks is shown in

Table 1. A list of alternative attacks to evaluate the privacy guarantees.

Attack	DPML Stages Impact	Type of Attack	Attack Techniques
Membership inference	Training data	White-box membership inference attack	Gradient-based approaches: Exploiting gradients whether specific data points were part of the training dataset.
			Activation analysis: Exploiting the activations for training data based on the assumption that they differ in certain layers from the activations for non-training data in certain layers.
		Black-box membership inference attack	Training shadow models: Creating and training a set of models that mimic the behavior of the targeted model.
			Confidence score analysis: Construct and analyze confidence scores or confidence intervals.
		Label-only membership inference attack	Adaptive querying: Modifying the inputs to answer queries that are individually selected, where each query depends on the answer to the previous query when the model changes the label.
			Meta-classification: Training a secondary model to distinguish between the labels of training and non-training data.
		Transfer membership inference attack	Model approximation: Using approximation algorithms to test the decision boundaries of the target model.
			Adversarial examples: Using adversarial techniques to evaluate privacy guarantees.
Data poisoning	Training phase/model, data	Gradient manipulation attack	The gradients are intentionally altered during the model training process.
		Targeted label flipping	Label modification of certain data points in the training data without changing the data themselves.
		Backdoor poisoning	Inserting a specific trigger or “backdoor”.
		Data injection	Injecting malicious data samples that are designed to disrupt the model’s training.
		Adaptive querying and poisoning	Injecting a slightly modified version of data points and analyzing how these changes affect label predictions.
Model inversion	Model	White-box inversion attacks	The attacker uses detailed insights into the model’s structure and parameters (e.g., model weights or gradients) to recover private training data.
		Black-box inversion attacks	The attacker iteratively queries the model and uses the outputs to infer sensitive information without access to the model’s internals.
		Inferring sensitive attributes from the model	Balancing the privacy budget for sensitive and non-sensitive attributes.
		Gradient-based inversion attacks	The attacker tries to recover private training data from shared gradients.
Model extraction	Model	Adaptive Query-Flooding Parameter Duplication (QPD) attack	Allow the attacker to infer model information with black-box access and no prior knowledge of model parameters or training data.
		Equation-solving attack	Targets regression models by adding high-dimensional Gaussian noise to model coefficients.
		Membership-based property inference	Combines membership inference with property inference, targeting specific subpopulations with unique features.

.

3.3.2. Data-Poisoning Attack

In data-poisoning attacks, malicious data are injected into the training set in order to influence the behavior of the model. These attacks, whether untargeted (random) or targeted [54,121], are a form of undermining the functionality of the model. Common approaches are to either reduce the accuracy of the model (random) or manipulate the model to output a label specified by the attacker (targeted) to reduce the performance of the model or cause targeted misclassification or misprediction. If the attacker tries to elicit a specific behavior from the model, the attack is called targeted. A non-targeted attack, on the other hand, means that the attacker is trying to disrupt the “overall functionality of the model”. Targeted or non-targeted poisoning attacks can include both model poisoning and data-poisoning attacks. The impact of poisoning attacks, for example, causes the classifier to change its decision boundary and achieve the attacker’s goal of violating privacy [121].

In the context of DP, during data-poisoning attacks [55,122], the attacker manipulates and falsifies the model at the time of its training or during the inference time of the model by injecting adversarial examples into the training dataset [54]. In this way, the behavior of the model is manipulated, and meaningful information is extracted. Poisoning attacks are not only limited to training data points; they also target model weights. Among these threats, data poisoning stands out due to its potential to manipulate and undermine the integrity of AI-driven systems. It is worth noting that this type of attack is not directly related to data privacy but still poses a threat to ML modeling [123]. In model poisoning, target model poisoning aims to misclassify selected inputs without modifying them. This is achieved by manipulating the training process. Data-poisoning attacks are relevant for DP auditing as they can expose potential vulnerabilities in privacy-preserving models. The DPs typically consist of five main types (as shown in Table 1):

Gradient manipulation attacks: In gradient manipulation attacks, especially gradient inversion attacks [96,124], the attacker manipulates the gradient update by injecting false or poisoned gradients that either distort the decision boundary of the model or lead to overfitting. These attacks allow the attacker to reconstruct private training data from shared gradients and undermine the privacy guarantees of ML models. This approach also aims to investigate whether gradient clipping and noise addition can effectively protect against excessive influence of individual gradients.

Targeted label flipping: This involves modification of the labels of certain data points in the training data without changing the data themselves, especially those in sensitive classes [125]. The attacker then checks whether this modified information can be restored from the model.

Influence limiting: To assess how DP mechanisms limit the influence of any single data point, poisoned records are inserted into the training data to see if their impact on the model predictions and accuracy can be detected [126].

Backdoor poisoning attacks: This type of attack aims to insert a specific trigger or “backdoor” [127,128,129] that later manipulates the behavior of the model when it is activated in the testing or deployment phase. If the model is influenced by the backdoor pattern in a way that compromises individual data points, this may indicate vulnerability to targeted privacy risks. These types of attacks are often evaluated against specific target perturbation learners. The attacker intentionally disrupts some training samples to change the parameter distribution [130]. Among these threats, data poisoning stands out due to its potential to manipulate and undermine the integrity of ML-driven systems. These attacks were developed for image datasets [129]. In the original backdoor attacks, the backdoor patterns are fixed [131], e.g., a small group of pixels in the corner of an image. More recent backdoor attacks can be dynamic [132] or semantic [133]. Backdoor attacks are a popular approach to poison ML classification models. DP can help prevent backdoor attacks by ensuring that the training process of the model includes noise addition or privacy amplification.

Data injection: In this type of attack, an attacker injects malicious data samples that are designed to disrupt the model’s training [134]. This is different to backdoor attacks in that it may not involve a specific trigger pattern but simply serves to corrupt the model’s decision making. Adding random, noisy samples into the training set can skew the model’s weight, leading to suboptimal performance.

3.3.3. Model Inversion Attack

These attacks exploit the released model to predict sensitive attributes of individuals using available background information. Existing DP mechanisms struggle to prevent these attacks while preserving the utility of the model [108,135,136].

The model inversion attack is a technique in which the attacker attempts to recover the training dataset from learned parameters. For example, Zhang et al. [58] use model inversion attacks to reconstruct training images from a neural network-based image recognition model. In these attacks, the attackers use the released model to infer sensitive attributes of individuals in the training data or the outputs of DPML models. These attacks allow attackers to infer sensitive attributes of individuals by exploiting the outputs of the model [29,135,136].

The idea of model inversion [137] is to invert a given pre-trained model, $f_{\theta }$, in order to recover a private dataset, $D$, such as images, texts, and graphs. The attacker who attempts to use the model inversion attack [28,138,139,140] queries the model with different inputs and observes the outputs. By comparing the outputs for different inputs, the attacker identifies and recognizes patterns. By testing each feature, the attacker can consequently infer the pattern in the original training data, resulting in a data leakage.

A common approach for this type of attack is to reconstruct the input data from the confidence score vectors predicted by the target model [100]. The attacker trains a separate attack model on an auxiliary dataset that acts as the inverse of the target model [141]. The attack model takes the confidence score vectors of the target model as input and tries to output the original data of the target model [113]. Formally, let $f_{\theta }$ be the target model and $g_{{\theta }^{\prime }}$ be the attack model. Given a data record $(x,y)$, the attacker inputs $x$ into $f_{\theta }$ and receives $f\left(x\right)$ and then feeds into $g_{{\theta }^{\prime }}$ and receives $g\left(f\right(x\left)\right)$, which is expected to obtain $g\left(f\right(x\left)\right)\approx x$; that is, the outputs of the target models and the attacks model could be very similar. These attacks can be categorized as follows:

• Learning-based methods.
• White-box inversion attacks.
• Black-box inversion attacks.
• The gradient-based inversion attacks.

Learning-based methods: These methods can reconstruct diverse data for different training samples within each class. Recent advances have improved their accuracy by regularizing the training process with semantic loss functions and introducing counterexamples to increase the diversity of class-related features [142].

White-box inversion attacks: In such attacks, the attackers have full access to the structure and parameters of the model. Auditors use the white-box inversion as a worst-case scenario or use the model’s parameters of the model [143] to assess whether the DP mechanisms preserve privacy even with highly privileged access.

Black-box inversion attacks: In such attacks, the attackers only have access to the output labels of the model [144] or obtain the confidence vectors. Black-box attacks simulate typical model usage scenarios, allowing auditors to assess how much information leakage occurs purely through interactions with the model’s interface.

Gradient-based inversion attacks (also known as input recovery from gradients): The attacker accesses gradients shared during the training rounds (especially in FL) and uses these to infer details about the training [145]. Auditors use these attacks by masking sensitive information, particularly in collaborative and decentralized working environments.

Traditional DP mechanisms often fail to prevent model inversion attacks while maintain model utility. Model inversion attacks are a significant challenge for DP mechanisms, especially for regression models and graph neural networks (GNNs). These attacks allow attackers to infer sensitive attributes of individuals by exploiting the released model and some background information [135,136].

Model inversion enables an attacker to fully reconstruct private training samples [97]. For visual tasks, the model inversion attack is formulated as an optimization problem. The attack uses a trained classifier to extract representations of the training data. A successful model inversion attack generates diverse and realistic samples that accurately describe each class of the original private dataset.

3.3.4. Model Extraction Attack

Model extraction attacks (also known as reconstruction attacks) aim to steal the functionality, replicate ML models, and expose sensitive information of well-trained ML models [146]. The attacker can approach or replicate a target model by sending numerous queries to infer model parameters or hyperparameters and observing its response. In model extraction attacks [95,147,148], in the context of DP auditing, attackers attempt to derive a victim model by extensively querying model parameters or training data in order to train a surrogate model. The attacker learns a model to try to extract information and possibly fully reconstruct a target model by creating a new duplicate model that resembles the target model in a way that behaves very similarly to the attacked model. This type of attack only targets the model itself and not the training data.

This type of attack can be categorized into two classes [149]: (i) accuracy extraction attacks, i.e., focusing on replacing the target model; and (ii) fidelity extraction attacks, i.e., aim to closely match the prediction of target model.

This threat can increase the privacy risk as a successful model extraction can enable a subsequent next threat, such as a model inversion. There are two approaches to creating a surrogate model [62,147]. First, the surrogate model fits the target model to a set of input points that are not necessarily related to the learning task. Second, create a surrogate model that matches the accuracy of the target model on the test set that is related to the learning task and comes from the distribution of the input data.

Model extraction attacks pose a significant security threat to ML models, especially those provided via cloud services or public APIs. In these attacks, an attacker repeatedly queries a target model to train a surrogate model that mimics the functionality of the target’s model.

3.3.5. Property Inference Attacks

Property inference attacks (also called distribution inference) [150,151,152,153] aim to infer global, aggregate properties of the training data used in machine-learning models, rather than details of individual data points. These sensitive properties are often based on ratios, such as the ratio of male to female records in a dataset. The attack attempts to understand the statistical information of a training dataset from an ML model. In contrast to privacy attacks that focus on individuals in a training dataset (e.g., membership inference), PIAs aim to extract population-level features from trained ML models [60]. Existing property inference attacks can be broadly categorized as follows:

The attacker attacks the training dataset and attempts to leak sensitive statistical information related to the dataset or a subset of the training dataset, such as specific attributes, which can have a significant impact on privacy in the model [150]. The attacker can also exploit the model’s ability to memorize explicit and implicit properties of the training data [154]. This can be achieved by poisoning a subset of the training data to increase the information leakage [152]. There is an option where an attacker can maliciously control a portion of the training data to increase the information leakage. This can lead to a significant increase in the effectiveness of the attack and is then referred to as a property inference poisoning attack [155].

Differential privacy (DP) auditing uses property inference attacks to test whether the DP mechanisms are robust against the leakage of information about specific properties, features, or patterns within the dataset.

4. Privacy Auditing Schemes

In this section, the privacy auditing schemes for differential privacy auditing are presented.

4.1. Privacy Auditing in Differential Privacy

Testing and evaluating DPML models are important to ensure that they effectively protect privacy while retaining their utility. Since differential privacy is always a trade-off between privacy and utility, evaluating the privacy of the model helps in choosing an appropriate privacy budget. A privacy budget that is high enough to ensure sufficient accuracy and a budget that is low enough to ensure acceptable privacy. Which level of accuracy and/or privacy is sufficient depends on the application of the model [130]. To address the security-critical privacy issues [156] and detect potential privacy violations and biases, the privacy auditing procedures can be implemented to empirically evaluate DPML.

Privacy auditing is a set of techniques for empirically verifying the privacy leakage of an algorithm to ensure that it fulfils the defined privacy guarantees and standards. Privacy auditing of DP models [45,46,47,157,158] aims to ensure that privacy-preserving mechanisms are effective, reliable and provide the privacy guarantees of DPML models and algorithms. For example, one approach is to use a probabilistic automation model to track privacy leakage bounds [159]; and another uses canaries to audit ML algorithms [160], efficiently detects $\left(ϵ,\delta \right)$-violations [48], estimates privacy loss during a single training run [118], or transforms $\left(ϵ,\delta \right)$ into Bayesian posterior belief bounds [34]. To ensure robust privacy auditing in differential privacy (DP), it is important to follow the key steps that rigorously attack the machine-learning model and verify privacy guarantees:

Define the scope of the privacy audit: Establish the objectives and purposes of the audit. This includes determining which specific mechanisms or algorithms are to be evaluated and determining the privacy guarantees $(ϵ,\delta )$ that are relevant for the audit.

Clear delineation of the privacy guarantees expected by the DP model (differential privacy mechanisms), the definition of data protection requirements that are tailored to the sensitivity of the data, compliance with standards, and the justification of the privacy parameters, $ϵ,\delta$ (upper bound), are required. For example, the authors of [161] describe a privacy auditing pipeline that is divided into two components: the attacker scheme and the auditing scheme.

Perform privacy attacks and vulnerability implementation: Implement privacy attacks (e.g., membership inference, model extraction, and model inversion) to evaluate the robustness of the DP mechanism or DPML algorithm. It means to providing robust privacy guarantees for a DP mechanism that effectively limit the amount of information that can be inferred about individual data points, regardless of a potential attacker’s strategy or the configuration of the dataset. For example, simulate black-box or white-box membership inference [162,163] to assess the impact of model access on privacy leakage. This gives us the opportunity to test the resilience of the model and measure the success/failure rates of these attacks.

Analyze and interpret the audit results: The final step is to empirically estimate the privacy leakage from a DPML model, denoted as ${ϵ}_{emp}$, and compare it with the theoretical privacy budget, $ϵ$ [81]. An important goal of this process is the assessment of the tightness of the privacy budget. The audit is considered tight if the privacy parameter is ${ϵ}_{emp}\approx ϵ$. Such an approach can be used to effectively validate DP implementations in the model or to detect DP violations in case of ${ϵ}_{emp}>ϵ$ [164,165,166].

4.2. Privacy Auditing Techniques

Privacy auditing techniques in differential privacy are essential to ensure that privacy guarantees are met in practical implementations. Empirical auditing techniques establish practical lower bounds on privacy leakage, complimenting the theoretical upper bounds provided by DP [32]. Before we address privacy auditing schemes, it is necessary to explain the main auditing techniques (Birhane et al., 2024) that have been used to evaluate the effectiveness of DP mechanisms and algorithms against privacy attacks in ML models.

Canary-based audits: Canary-based auditing is a technique for assessing the privacy guarantees of DPML algorithms by introducing specially designed examples, known as canaries, into the dataset [43,160]. The auditor then tests whether these canaries are included in the outputs of the model and distinguishes between models trained with different numbers of canaries. An effective DP should limit the sensitivity of the model to the presence of these canaries, minimizing the privacy risk. Canaries must be carefully designed to ensure that they can detect potential privacy leaks without jeopardizing overall privacy guarantees. Canary-based auditing often requires dealing with randomized datasets, which enables the development of randomized canaries. The introduction of Lifted Differential Privacy (LiDP) [160] by distinguishing between models trained with different numbers of canaries can leverage statistical tests and novel confidence intervals to improve sample complexity. There are several canary strategies: (1) a random sample from a dataset distribution with a false label, (2) the use of an empty sample, (3) an adversarial sample, and (4) the canary crafting approach [42]. The disadvantage of canaries is that an attacker must have access to the underlying dataset and knowledge of the domain and model architecture.

Statistical auditing: In this context, statistical methods are used to empirically evaluate privacy guarantees [167]. These include influence-based attacks and improved privacy search methods that can be used to detect privacy violations and understand information leakage in datasets, thus greatly improving the auditing performance of various models, such as logistic regression and random forest [62].

Statistical hypothesis testing interpretation: The aim of this approach is to find the optimal trade-off between type I and type II errors [168]. This means that no test can effectively determine whether a specific individual’s data are included in a dataset, ensuring that both high power and high significance are simultaneously unattainable simultaneously [169,170]. It is used to derive the theoretical upper bound and is very useful in deriving the tight compositions [171] and has even motivated a new relaxed notion of DP called f-DP [76].

Single training run auditing (also known as one-shot auditing): It enables privacy auditing during a single training run, eliminating the need for multiple retraining sessions. This technique utilizes the parallelism of independently adding or removing multiple training examples and enables meaningful empirical privacy reduction with only one training run of the model [43]. The technique is efficient and requires no prior knowledge of the model architecture or DP algorithm. This method is particularly useful in FL settings and provides accurate estimates of privacy loss under the Gaussian mechanism [118].

Empirical privacy estimation: In this technique, the actual privacy loss of an algorithm is evaluated by practical experiments rather than theoretical guarantees [172]. This technique is used to audit the implementations of DP mechanisms or claims about the models trained with DP [42]. They are also useful to estimate the privacy loss in cases where a tight analytical upper bound on ϵ is unknown.

Post hoc privacy auditing: This technique traditionally establishes a set of lower bounds for privacy loss (e.g., thresholds). However, it requires sharing intermediate model updates and data with the auditor, which can lead to high computational costs [173].

Worst-case privacy check: In the context of differential privacy, the worst-case privacy auditing [32,102] refers to the specific data points or records in a dataset that, if added, removed or altered from the dataset, could potentially have the greatest impact on the output of a differential privacy mechanism. Essentially, these are the most “sensitive” records where the privacy guarantee is most at risk.

4.3. Privacy Audits

When we focus on auditing DPML models, we first need to know whether we have enough access to information to perform a white-box audit or a black-box audit. A white-box auditing can be difficult to perform on a large scale in practice, as the algorithm to be audited needs to be significantly modified, which is not always possible [109]. Nevertheless, auditing DPML models in a white-box environment requires minimal assumptions about the algorithms [43]. Instead, black-box audits are more realistic in practice, as our attacker can only observe the final trained model.

Privacy auditing schemes empirically evaluate the privacy leakage of a target ML model, or its algorithm trained with DP [42,46,47,62,96]. Such schemes use the DP definition a priori to formalize and quantify the privacy leakage [174]. Currently, most auditing techniques are based on simulating different types of attacks [114] to determine a $(ϵ,\delta )$ lower bound on the privacy loss of a ML model or algorithm [62]. Privacy auditing can be performed using different attacker schemes (processes), which can be broadly categorized as follows (

Table 2. A list of privacy auditing schemes.

Privacy Auditing Scheme	Privacy Attack	Auditing Methodology
Membership inference audits	White-box membership inference auditing	Auditors analyze gradients, hidden layers, intermediate activations measuring how training data influences model behavior.
	Black-box membership inference auditing	Auditors observe probability distributions and confidence scores by analyzing these outputs and assessing the likelihood that certain samples were part of the training data.
	Shadow model membership auditing	Auditors use “shadow models” to mimic the behavior of the target model.
	Label-only membership inference auditing	Auditor evaluates the privacy guarantee leveraging only output labels, training shadow models, generating a separate classifier, and quantifying true-positive rate and accuracy.
	Single-training membership inference run auditing	Auditor leverages the ability to add or remove multiple training examples independently during the run. This approach focuses on estimating the lower bounds of the privacy parameters without the need for extensive retraining of the models.
	Metric-based membership inference auditing	Auditor assesses privacy guarantees directly evaluating metrics and statistics derived from the model’s outputs (precision, recall, and F1-score) on data points.
	Data augmentation-based auditing	Auditor generates or augmented data samples similar to training set, testing whether these samples reveal membership risk.
Data poisoning auditing	Influence-function analysis	Evaluate privacy by introduction malicious data.
	Gradient manipulation in DP training	Auditor alters the training data using back-gradient optimization, gradient ascent poisoning, etc.
	Empirical evaluation of privacy loss	Auditor conducts quantitative analyses of how the privacy budgets is affected.
	Simulation of worst-case poisoning scenarios	Auditor constructs approximate upper bounds on the privacy loss.
Model inversion auditing	Sensitivity analysis	Auditor quantifies how much private information is embedded in the model outputs.
	Gradient and weight analyses	Auditor attempts to recreate input features or private data points form model outputs using gradient-based or optimization methods.
	Empirical privacy loss	Auditor calculates the difference between theoretical and empirical privacy losses.
	Embedding and reconstruction test	Auditor examines whether latent representations or embeddings could be reversed to obtain private data.
Model extraction auditing	Query analysis	Auditors simulate extraction attacks by extensively querying the model and analyzing how well they can replicate its outputs or decision boundaries.
Property inference auditing	Evaluating property sensitivity with model outputs.	The auditor performs a test to infer whether certain properties can be derived from the model and whether the privacy parameters are sufficient to obscure such data properties.

):

• Membership inference auditing.
• Poisoning auditing.
• Model inversion auditing.
• Model extraction auditing.
• Property inference auditing.

In summary, privacy auditing schemes leverage various techniques to strike a balance between privacy, data utility and auditing efficiency. The comprehensive privacy auditing methodology, privacy guarantees with references can be found in Appendix A. We review the most important works, starting with the use of black-box and white-box privacy attacks.

4.3.1. Differential Privacy Auditing Using Membership Inference

Membership inference audits: These audits test the resilience of the model against membership attacks, where an attacker tries to determine whether certain data points were included in the training set. The auditor performs MIAs to estimate how much information about individual records may have been leaked. This category is divided into the following subcategories:

• Black-box inference membership auditing: This approach relies solely on assessing the privacy guarantees of machine-learning models by evaluating their vulnerability to membership inference attacks (MIAs) without accessing the internal workings of the model.

Song et al. [175] examine how robust training, including a differential privacy mechanism, affects the vulnerability to black-box MIAs. The success of MIAs is measured using metrics such as attack accuracy and the relationship between model overfitting and privacy leakage. It also investigates MIAs under attacker robustness and differential privacy conditions and shows that DP models are also vulnerable under black-box conditions. Carlini et al. [103] present a DP audit method related to black-box threshold MIAs by proposing a first-principles approach. The authors introduce the likelihood ratio attack (LiRA). It analyzes the most vulnerable points in the model predictions. The authors question the use of existing methodologies that rely on average-case accuracy metrics to evaluate empirical privacy, which do not adequately capture an attacker’s ability to identify the actual members of the training dataset. They propose to measure the attacker’s ability to infer membership of a dataset using the true-positive rate (TPR) and the low false-positive rate (FPR) at very low positive rates (e.g., <0.1%). The authors offer a $ϵ$—maximization strategy. The authors conclude that even a powerful DP mechanism can sometimes be vulnerable to carefully constructed black-box accesses. Lu et al. [172] present Eureka, a novel method for estimating relative DP guarantees in black-box settings, which defines a mechanism’s privacy concerning a specific input set. At its core, Eureka uses a hypothesis testing technique to empirically estimate privacy loss parameters. By computing outputs on adjacent datasets, the potential leakage and thus the degree of privacy guarantee is determined. The authors use MIAs based on classifiers to audit $(ϵ,\delta )$-DP algorithms. They demonstrate that Eureka achieves tight accuracy bounds in estimating privacy parameters with relatively low computational cost for large output space.

Kazmi et al. [174] present a black-box privacy auditing method for ML target models based on a MIA using both training data (i.e., “members” (true positives)) and generated data not included in the training dataset (i.e., non-members (true negatives)). This method leverages membership inference as primary method to audit datasets used in the training of ML models without retraining them (ensembled membership auditing (EMA)). EMA aggregates the membership scores on a data-by-data basis based on individual data, using statistical tests. The method, which authors call PANORAMIA, quantifies privacy leakage for large-scale ML models without controlling the training process or retraining of the model. Koskela et al. [176] use the total variation distance (TVD), a statistical measure that quantifies the difference between two probability distributions, between the output distributions of a model when trained on two neighboring datasets. The authors suggest that TV distance can serve as a robust indicator of privacy guarantee when examining the outputs of a DP mechanism. The auditing process compares two TVD how much they differ from each other across different outputs generated from adjacent datasets to approximate the privacy parameters. It is directly related to the privacy parameter, $ϵ$, and provides a tangible way to evaluate privacy loss. The auditing process utilizes a small hold-out dataset that has not been exposed during training. Their approach allows for the use of an arbitrary hockey-stick divergence to measure the distance between the score distributions of audit training and test samples. This work fits well with FL scenarios.

White-box inference membership auditing: White-box audits leverage full access to the internal parameters of a model, including gradients and weights. They are often used in corporate ML research, where the internal parameters of the model are available and allow a detailed analysis of the DP efficiency.

Leino and Fredrikson [107] propose a calibrated white-box membership inference attack by evaluating the resulting privacy risk, which also leverages the intermediate representations of the target model. The work investigates how MIAs exploit the tendency of deep networks to memorize specific data points, leading to overfitting. They linearly approximated each layer, launched a separate attack on each layer, and trained the target model (DP-SGD) that combines the outputs of the layer-wise attacks. The high-precision calibration ensures that the attack can confidently identify whether a data point was part of the training set. Chen et al. [177] evaluate a differential private convolutional neural network (CNN) and Lasso regression model with and without sparsity using a MIA on high-dimensional training data, using genomic data as an example. They show that sparsity of the model in contrast to the non-private setting can improve the accuracy of the model in the non-private setting. By applying a regularization technique (e.g., Lasso), the study demonstrates that sparsity can complement DP efforts.

There are seminal works that use both white-box and black-box settings: Nasr et al. [47] extended the study of [46] Jagielski et al. (2020) on empirical privacy estimation techniques by analyzing DP-SGD through an increasing series of black-box membership inference to white-box poisoning attacks. They are the first to audit DP-SGD tightly. To do so, they use attacker-crafted datasets and active white-box attacks that insert canary gradients into the intermediate steps of DP-SGD. Tramèr et al. [49] propose a method for auditing backpropagation clipping algorithm (a modification of the DP-SGD algorithm), assuming that it works in black-box or white-box settings. The goal was to empirically evaluate how often the outputs of $M\left(D\right)$ and $M\left(D^{\prime }\right)$ are distinguishable. The auditor’s task with MIA is to maximize the FPR/TPR ratio to assess the strength of the privacy mechanism. Nasr et al. [42] follow up on their earlier work (Nasr et al., 2021) and design an improved auditing scheme for testing DP implementations in black-box and white-box settings for DP-SGD with gradient canaries or input space canaries. This method provides a tight privacy estimation that significantly reduces the computational cost by leveraging tight composition theorems for DP. The authors check each individual step of the DP-SGD algorithm; that is, they do not convert each obtained lower bound obtained into a guarantee for $ϵ$, which is given after compiling over all training steps with understanding of the privacy understanding of the end-to-end algorithm.

Shadow model auditing: Shadow model membership auditing is a technique used to assess the privacy of machine-learning models by replicating the target models that are trained on similar datasets. They allow the auditor to infer information about the target model’s training data without direct access to it. In this method, multiple shadow models are created that mimic the behavior of the target model so that the auditor can infer the membership status based on the outputs of the shadow models. The primary purpose of using shadow models is to facilitate MIAs determining whether specific data points were part of the training dataset.

The groundbreaking work Shokri et al. [52] evaluates the membership inference attack in a black-box environment in which the attacker only has access to the target model via a query. The attack algorithm is based on the concept of shadow models. The attacker trains shadow models that are similar to the target model and uses these shadow models to train a membership inference model. MIA is modeled as a binary classification task for an attack model that is trained using the prediction of shadow models on the attacker dataset. Salem et al. [112] utilize data augmentation to create shadow models and analyze privacy leakage. It provides insights into the impact of data transformations on inference accuracy in both black-box and white-box settings.

Yeom et al. [10] investigate overfitting in ML model auditing using a threshold membership inference attack as a primary method and attribute inference attack based on distinction between training and testing per-instance losses. The authors provide an upper bound on the success of MIA as a function of the parameters in DP. By training shadow models, the authors demonstrate how models that memorize training data are more susceptible to MIAs, especially when DP techniques are not optimally applied. They conclude that overfitting is sufficient for an attacker to perform MIA. Sablayrolles et al. [22] focus on the comparison of black-box attacks and white-box attacks by effectively estimating the model loss for a data point. The authors use the shadow model technique to demonstrate MIAs across architectures and training methods. They also investigate Bayes optimal strategies for MIAs that leverage knowledge of model parameters in white-box settings. Their findings suggest that white-box attacks do not require specific information about model weights and losses, but can still be performed effectively using probabilistic assumptions, and that optimal attacks depend on the loss function and thus black-box attacks are as good as white-box attacks. The authors introduce the Inverse Hessian attack (IHA), which utilizes model parameters to enhance the effectiveness of membership inference. By computing inverse-Hessian vector products, these attacks can exploit the sensitivity of model output to specific training examples.

Label-only membership auditing: label-only membership inference auditing in differentially private machine learning is a privacy assessment method where an auditor attempts to deduce whether a particular data point was part of the training dataset based on the model’s predicted labels (without access to probabilities or other model details). This form of auditing is particularly relevant for real-world scenarios.

Malek et al. [178] adapt a heuristic method to evaluate label differential privacy (Label DP) in different configurations where privacy of the labels associated with training examples is preserved, while features may be publicly accessible. The authors propose two primary approaches: Private Aggregation of Teacher Ensembles (PATE) and Additive Laplace with Iterative Bayesian Inference (ALBI). They apply noise exclusively to the labels in the training data, leading to the development of different label-DP mechanisms, and investigate model accuracy and estimate lower bounds for the privacy parameter $ϵ$ values. They trained several models with and without a training point, while the rest of the training set remained unchanged. Choquette-Choo et al. [179] focus on a black-box attack, where only the labels of the model, and not the full probability distribution, are available to the attackers. They investigate privacy leakage in four private prediction algorithms: PATE, CaPC, PromptPATE, and Private-KNN. The authors showed that DP provides the strongest protection against privacy violation in both the average-case and worst-case scenarios and when the model is trained with overconfidence. However, this may come at the expense of the model’s test accuracy. The authors show that an effective defense against label-only MIAs involves DP and strong regularization, which significantly reduces the leakage of private information.

Single-run membership auditing: It is a technique using a single execution of the audit process.

Steinke et al. [43] propose a novel auditing scheme that uses only a single training of the model and can be evaluated using a one-time model output, making audits feasible in practical applications. The authors apply their auditing scheme specifically to the DP-SGD algorithm. After training, auditors select a set of canary data points (auditing examples) and apply MIA thresholds and model parameter tuning to maximize audit assurance from a single model output. The attack estimates the sensitivity of the model to individual data, which is an indication of the privacy risk of the model. By adjusting the MIA parameters and interpreting the model’s response to the canary points, the auditor approximates the empirical privacy loss, ${ϵ}_{emp}$. This empirical estimate provides information on how closely the model’s practical privacy matches the theoretical DP guarantees. Their analysis uses parallelism to add or remove multiple data points (training examples independently) in a single training run of the algorithm and statistical generalization. This auditing scheme requires minimal assumptions about the underlying algorithm, making it applicable in both black-box and white-box settings. Andrew et al. [118] propose a novel one-shot auditing framework that enables efficient auditing during a single training run without a priori knowledge of the model architecture, tasks, or DP training algorithm. The method is proven to provide provably correct estimates for privacy loss under the Gaussian mechanism, demonstrating its performance on FL benchmark datasets. The method they propose is model and dataset agnostic, so it can be applied to any local DP task.

Annamalai et al. [109] present a one-shot nearly tight black-box auditing scheme for the privacy guarantees of the DP-SGD algorithm to investigate empirical vs. theoretical aspects. The main idea behind the auditing is to craft worst-case initial model parameters, since DP-SGD is agnostic to the choice of initial model parameters that can yield to tighter privacy audits. The model was initialized with the average-case initial parameters. The authors empirically estimate the privacy leakage from DP-SGD by using the gradient-based membership inference attacks approach. The authors’ key finding is that by crafting worst-case initial model parameters, more realistic privacy estimates can be obtained to address the limitations of the theoretical privacy analysis of DP-SGD.

Loss-based membership inference auditing: It is a technique that measures privacy leakage in differentially private models by evaluating the differences in the model’s loss values generating during the training of ML models when predicting training data versus non-training data.

Wang et al. [111] introduced a novel randomized approach to privacy accounting, which aims to improve on traditional deterministic methods by achieving tighter bounds on privacy loss. The method leverages the concept of Privacy Loss Distribution (PLD) to more accurately measure and track the cumulative privacy loss over a sequence of computations. This approach is particularly beneficial for large-scale data applications where the privacy budget is strict.

Confidence score membership auditing: In this type of audit, the vulnerability of the model is assessed on the basis of the confidence scores of the predictions. Higher confidence in the predictions for training data points compared to non-training points often indicates a leak. By examining the confidence values for predictions in a large sample, auditors can determine whether training points have a higher confidence than non-training points and thus estimate membership leakage.

Yeom et al. [10] establish a direct link between overfitting and membership inference vulnerabilities by analyzing confidence scores. It is shown that high-confidence predictions are often associated with data memorization, which increases privacy risks, especially when attackers exploit confidence scores.

Metric-based membership inference auditing: This refers to the use of various metrics and statistics calculated from a ML model’s outputs to assess the privacy risks in DPML systems. This approach applies membership inference techniques and calculates metrics and statistics that allow a quantitative assessment of privacy leakage. It is often used to compare different models and privacy parameters.

Rahman et al. [180] evaluates DP mechanisms against MIAs and uses accuracy and F-score as a privacy leakage metrics to measure the privacy loss on models trained with DP algorithms. Jayaraman and Evans [21] evaluate the private mechanism against both membership inference and attribute inference attacks. They used balanced prior data distribution probability. Note that if the prior probability is skewed, the above-mentioned methods are not applicable. Liu et al. [169] evaluate DP mechanism using a hypothesis testing framework. They connect precision, recall, and F-score metrics to the DP parameters $(ϵ,\delta )$. Based on the attacker’s background knowledge, they give insight into choosing these parameter values. Balle et al. [170] explain DP through a statistical hypothesis testing interpretation, where conditions for a privacy definition based on statistical divergence are identified, allowing for an improved conversion rules between divergence and differential privacy. Carlini et al. [102] investigate how neural networks unintentionally memorize specific training data. The authors develop an attack methodology to quantify unintended memorization by evaluating how easy it is to reconstruct specific data points (e.g., training examples with private information) from the trained model. This study uses metric-based approaches to measure memorization and unintended data retention, which are both critical components in determining membership inference. The research identifies factors contributing to memorization, including model size, training duration, and dataset characteristics.

Humphries et al. [181] investigate the effectiveness of DP in protecting against MIAs in ML. The authors perform an empirical evaluation by varying the $ϵ$ values in DP-SGD and observing the effect on the success rate of MIAs including back-box and white-box attacks. The authors suggest that DP needs to be complemented by other techniques that specifically target membership inference risk. Ha et al. [41] evaluate the impact of privacy parameters by adjusting $ϵ$ on the effectiveness of DP in mitigating gradient-based MIAs. The authors recommend specific DP parameter settings and training procedures to improve privacy without sacrificing model utility. Askin et al. [182] explore statistical methods for quantifying and verifying differential privacy (DP) claims. This method includes estimators and confidence intervals for the optimal privacy parameter ϵ of a randomized algorithm and avoids the complex process of event selection, which simplifies the implementation, provides estimators and confidence intervals for the optimal privacy parameter ϵ of a randomized algorithm. Liu and Oh [158] report on extensive hypothesis testing using DPML using the Neyman–Pearson criterion. They give guidance on setting the privacy budget on assumption about the attacker’s knowledge considering different types of auxiliary information that an attacker can obtain (to strengthen the MIA such as probability distribution of data, record correlation, and temporal correlation).

Aerni et al. [183] design adaptive membership inference attacks based on the LiRA framework [103], which frames membership inference as a hypothesis testing problem. Given the score of the victim model on $x$, the attack then applies a likelihood ratio test to distinguish between the two hypotheses. To estimate the score distribution, multiple shadow models must be trained by repeatedly sampling a training set and training models.

Data augmentation-based auditing: This form of auditing involves generating synthetic or modified versions of the data in order to assess and improve privacy guarantees. This approach is useful in evaluating models with overfitting tendencies, where small perturbations could reveal privacy weaknesses.

Kong et al. [184] present a notable connection between machine unlearning and MIA. Their method provides a mechanism for privacy auditing without modifying the model. By leveraging forgeability (creating new, synthetic data samples), data owners can construct Proof-of-Repudiation (PoR) concept that allows a model owner to refute claims made by MIAs to enhance privacy protection and mitigate privacy risks.

Recently, there has been works on auditing lower bounds for differential Rènyi differential privacy (RDP). Balle et al. [170] investigate the relationship between RDP and its interpretation in terms of a statistical hypothesis testing interpretation. The authors investigate the conditions for a privacy definition based on statistical divergence, which allows for an improved conversion rules between divergence and differential privacy. They provide precise privacy loss bounds under RDP and interpret these in terms of type I and type II errors in hypothesis testing. Kutta et al. [185] develop a framework to estimate lower bounds for RDP parameters (especially for $ϵ-\mathrm{R}\mathrm{D}\mathrm{P}$) by investigating a mechanism in a black-box manner. Their framework allows auditors to derive minimal privacy guarantees without requiring internal access to the mechanism. Their goal was to observe how much the outputs deviate for small perturbations in the inputs. Domingo-Enrich et al. [186] propose an auditing procedure of DP with the regularized kernel Rènyi divergence (KRD) to define the regularized kernel Rènyi differential privacy (KRDP). Their auditing procedure can estimate from samples even in high dimensions for $ϵ$-DP, $\left(ϵ,\delta \right)$-DP, and $(\alpha ,ϵ)$-Rényi DP. Their proposed auditing method does not suffer from curse of dimension and has parametric rates in high dimension. However this approach requires knowledge of the covariance matrix of the underlying mechanisms, which is impractical for most mechanisms other than Laplace and Gaussian mechanisms and inaccessible in black-box settings.

Kong et al. [40] introduce a family of function-based testers for Rènyi DP (also for pure and approximate DP). The authors introduce DP-Auditorium, a DP auditing library implemented in Python that allows to test DP guarantees from the black-box access to the mechanism. DP-Auditorium facilitates the development and execution of privacy audits, allowing researchers and practitioners to evaluate the robustness of DP implementations against various adversarial attacks, including membership inference and attribute inference. The library also supports multiple privacy auditing protocols and integrates configurable privacy mechanisms, allowing for testing across different privacy budgets and settings. Chadha et al. [32] proposes a framework for auditing private predictions with different poisoning and querying capabilities. They investigate privacy leakage in terms of Rényi DP of four private prediction algorithms PATE, CaPC, Prompt PATE and Private-KNN. The experiments have shown that there are algorithms that are easier to poison and lead to much higher privacy leakage. Moreover, the privacy leakage is significantly lower for attackers without query control than for attackers with full control. The authors have shown that privacy leakage is lower for attackers without query control.

4.3.2. Differential Privacy Auditing with Data Poisoning

Data-poisoning auditing: In data poisoning, “poisoned” data are introduced into the training dataset to observe whether they influence the model predictions and worsens the data protection guarantees. The auditor simulates various data poisoning scenarios by inserting manipulated samples that distort the data distribution. The main scenarios that have been considered in the data poisoning auditing literature are adversarial injection of data points, influence function analysis, manipulation of gradients in DP training, empirical evaluation of privacy loss ϵ, simulation of worst-case poisoning scenarios, and privacy violations.

Influence function analysis is a statistical tool that helps to identify whether specific data points have an excessive influence on the model used to measure the effect on model’s predictions, indicating possible poisoning. They provide a way to estimate how much specific training samples influence the model’s behavior without needing to retrain the model. The seminal work by Koh and Ling [187] provides a robust framework for influence functions to analyze and audit the predictions made by black-box ML models. It introduces techniques for measuring the influence of individual training points on model predictions, setting the stage for analyzing poisoning attacks. The authors utilize first-order Taylor approximations to derive influence functions. This method is particularly useful for diagnosing issues related to model outputs. Lu et al. [61] audit the tightness of DP algorithms using influence-based poisoning to detect privacy violations and understand information leakage. They manipulate the training data to influence the output of the model and thus violate privacy guarantees. Their main goal is to verify the privacy of a known mechanism whose inner workings may be hidden.

To understand how poisoned gradients can influence privacy guarantees, the gradient manipulation is used in DP training. By monitoring gradients, auditors can detect anomalies due to poisoned inputs, as these may cause the differentially private model to exhibit non-robust behavior. Chen et al. [188] investigate the potential for reconstructing training data from gradient leakage analysis during the training of neural networks. The reconstruction problem is formulated as a series of optimization problems that are solved iteratively for each layer of the neural network. An important contribution that this work makes is the proposal of a metric to measure the security level of DL models against gradient-based attacks. The seminal paper Xie et al. [189] investigate the impact of gradient manipulation on both privacy guarantees and model accuracy relevant to DP auditing in federated learning. Liu and Zhao [190] focus on the interaction of gradient manipulation with privacy and proposed ways to improve the robustness of the model under these attacks. Ma et al. [54] establish complementary relationships between data poisoning and differential privacy by using a small-scale and a large-scale data-poisoning attack based on the gradient ascend of logistic regression parameters $\theta$ with respect to $x$ to reach target ${\theta }^{*}$. They evaluate the attack algorithms on two private learners targeting an objective perturbation and an output perturbation. They show that differentially private learners are provably resistant to data-poisoning attacks, with the protection deceasing exponentially as the attacker poisons more data. Jagielski et al. [46] investigate privacy vulnerabilities in DP-SGD, focusing on the question of whether the theoretical privacy guarantees hold under real-world conditions. The DP-SGD algorithm was audited by simulating a model-agnostic clipping-aware poisoning attack (ClipBKD) in black-box settings on logistic regression and fully connected neural network models. The models were initialized, such that the initial parameters were set for the average case. The empirical privacy estimates are derived from Clopper–Pearson confidence intervals of the FP and FN rates of attacks. The authors provide a $ϵ$-maximisation strategy to obtain a lower bound on the privacy leakage.

Empirical evaluation of privacy loss evaluates how the privacy budgets are affected by poisoned data. Auditors measure the effective privacy loss, or empirical epsilon, by feeding in poisoned data and calculating whether the privacy budget remains within acceptable bounds. Steinke and Ullman [191] introduce auditing mechanisms that track the empirical privacy losses and provide insights into the impact of poisoned data on privacy guarantees in real-world applications. The authors clarify the relationship between pure and approximate DP by establishing quantitative bounds on privacy loss under different conditions and introducing adaptive data analysis. Kairouz et al. [192] develop empirical privacy assessment methods applicable to DP-SGD in high-risk inversion settings. This method allows a detailed examination of how shuffling affects privacy guarantees. The authors evaluate the different parameters, such as batch size and privacy budget, in terms of privacy leakage.

Privacy violation: The first work in the field of DP auditing, Li et al. [193] consider relaxing the DP notations to cover different types of privacy violations, such as unauthorized data collection, sharing and targeting. The authors outline key dimensions that influence privacy, such as individual factors (e.g., awareness and knowledge), technological factors (e.g., data processing), and contextual factors (e.g., legal framework). However, the data leakage is not assessed. Hay et al. [194] evaluate the existing DP implementations for correctness of implementation. The authors create a privacy evaluation framework, named DPBench. This framework is designed to evaluate, test and validate privacy guarantees. Recent work proposes efficient solutions for auditing simple privacy mechanisms for scalar or vector inputs to detect DP violations (Ding et al., 2018; Bichsel et al., 2021). For each neighboring input pair, the corresponding output is determined, and Monte Carlo probabilities are measured to determine privacy. Ding et al. [45] were the first to propose practical methods for testing privacy claims in black-box access to a mechanism. The authors designed StatDP, a hypothesis testing pipeline for checking DP violations in many classical DP algorithms, including noisy argmax and for identifying ϵ-DP violations in sparse algorithms, such as the spare vector technique and local DP algorithms. Their work focuses on univariate testing of DP and evaluates the correctness of existing DP implementations.

Wang et al. [195] offer a code analysis-based tool CheckDP to generate or prove counterexamples for a variety of algorithms, including spares variety vector algorithm. Barthe et al. [196] investigate the decidability of DP. CheckDP and DiPC can not only detect violations of privacy claims, but can also be used for explicit verification. Bichsel et al. [26] present a privacy violation detection tool, DP-Sniper, which shows that a black-box setting can effectively identify privacy violations. It utilizes two strategies: (1) classifier training to train a classifier that predicts whether an observed output is likely to have been generated from one of two inputs; and (2) optimal attack transformation, where this classifier is then transformed into an approximately optimal attack on differential privacy. DP-Sniper is particularly effective at exploiting floating-point vulnerabilities in naively implemented algorithms and detecting significant privacy violations

Niu et al. [165] present DP-Opt(mizer), a disprover that attempts to find for counterexamples whose lower bounds on differential privacy exceed the claimed level of privacy guaranteed by the algorithm. The authors focus exclusively on $ϵ$-DP. They train a classifier to distinguish between the outputs $M\left(a\right)$ and $M\left(a^{\prime }\right)$ and create an attack based on this classifier. The statistical guarantees for the attack found are given. They transform the search task into an improved optimization objective that takes the empirical error into account and then solve it using various off-the-shelf optimizers. Lokna et al. [48] present a black-box attack detection privacy violations of DP to detect potential $\left(ϵ,\delta \right)-$ violations by grouping $\left(ϵ,\delta \right)-$ pairs based on the perception that many pairs can be grouped together because they are due to the same algorithm. The key technical insight of their work is that many (ϵ, δ) differentially private algorithms combine ϵ and δ into a single privacy parameter ρ. By directly measuring the robustness or degree of privacy failure $\rho$, one can audit multiple privacy claims simultaneously. The authors implement their method in a tool called Delta-Siege.

4.3.3. Differential Privacy Auditing with Model Inversion

This is a DPML model evaluation scheme that examines how much information about individual data records can be inferred from the outputs of the trained model (usually confidence score values or gradients) to understand the level of privacy leakage. The model is inverted to extract information. The auditing might be a white-box with access to gradients or internal layers or a black-box that only accesses the output labels. The main challenge in detecting model inversion attacks in differential privacy auditing is the need to prevent the inference of sensitive attributes of individuals from the shared model, especially in the context of black-box scenarios. The main scenarios that have been considered in the literature for model inversion auditing are sensitivity analyses, gradient and weight analyses, empirical privacy loss, and embedding and reconstruction tests.

Sensitivity analyses quantify how much private information is embedded in the model’s outputs that could potentially be reversed. Auditors evaluate gradients or outputs to determine how well they reflect the characteristics of the data. This analysis often involves running a series of model inversions to assess how DP mechanisms (e.g., DP-SGD) protect against the disclosure of sensitive attributes.

Frederikson et al. [100] present a seminal paper that introduces model inversion attacks that use confidence scores from model predictions to reconstruct sensitive input data. It explores how certain types of models, even when protected with DP, can be vulnerable to model inversion attacks by revealing certain features. Wang et al. [136] analyze the vulnerability of existing DP mechanisms. They use a functional mechanism method that perturbs the coefficients of the polynomial representations of the objective function balancing the privacy budget between sensitive and non-sensitive attributes to mitigate model inversion attacks. Hitaj et al. [197] focuses primarily on collaborative learning settings, which is relevant to DP as it shows how generative adversarial networks (GANs) can be used for model inversion to reconstruct sensitive information, providing insights into potential vulnerabilities in DP-protected models. Song et al. [198] discuss how machine-learning models can memorize training data in a way that allows attackers to perform inversion attacks. The authors analyze scenarios in which DP cannot completely prevent leakage of private data features through inversion techniques. Fang et al. [135] examine the vulnerability of existing DP mechanisms using a functional mechanism method. They propose a differential privacy allocation model. They optimize the regression model by adjusting the allocation of the privacy budget allocation within the objective function. Cummings et al. [199] introduce an individual sensitivity metric technique like smooth sensitivity and sensitivity preprocessing to improve the accuracy of private data by reducing sensitivity, which is crucial for mitigating model inversion risk.

Gradient and weight analyses show whether and how gradients expose sensitive attributes. By auditing gradients and weights, privacy auditors can check whether protected data attributes can be inferred directly or indirectly. Since model inversion often leverages gradients for black-box attacks, gradient clipping in DP-SGD helps mitigate exposure.

Works such as that of Phan et al. [200] investigate how model inversion can circumvent the standard DP defense by exploiting subtle dependencies in the model parameters. There are works that use gradient-inversion attacks. Zhu et al. [201] show that gradient information, i.e., minimizing the difference between the observed gradients and those that would be expected from the true input data commonly used in DP or federated learning, can reveal sensitive training data by inversion. It is shown how even with DP mechanisms, gradient-based inversion attacks can reconstruct data and thus pose a privacy risk. Huang et al. [202] align the gradients of dummy data with the actual data, making the dummy images resemble the private images. The paper describes in detail how gradient inversion attacks work by recovering training patterns from model gradients shared during federated learning. Wu et al. [203] use gradient compression to reduce the effectiveness of gradient inversion attacks. Zhu et al. [204] introduce a novel generative gradient inversion attack algorithm (GGI) where the dummy images are generated from low-dimension latent vectors through the pre-trained generator.

Empirical privacy loss approach calculates the difference between theoretical and empirical privacy losses in inversion scenarios. Auditors measure the privacy loss, $ϵ$, by performing a model inversion on a DP-protected model and comparing the result with the theoretical privacy budget. Large deviations indicate a possible weakness of DP in protecting against inversion.

Yang et al. [205] investigate defense mechanisms against model inversion and propose prediction purification techniques, which involves modifying the outputs of the model to obscure sensitive information while still providing useful predictions. It shows how adding additional processing to predictions can mitigate the effects of inversion attacks. Zhang et al. [206] apply DP to software defect prediction (SDP) sharing models and investigate privacy disclosure through model inversion attacks. The authors introduce class-level and subclass-level DP and use DPRF (differential private random forest) as a part on enhancing DP mechanism.

Embedding and reconstruction test: It examines whether latent representations or embeddings could be reversed to obtain private data. The auditors question whether embeddings of DP models are resistant to inversion by attempting to reconstruct data points from compressed representations.

Manchini et al. [207] show that stricter privacy restrictions can lead to a strong bias in inferential, affecting the statistical performance of the model. They propose an approach to improve the data privacy in regression models under heteroscedasticity. In addition, there are some methods such as Graph Model Inversion (GraphMI) that are specifically designed to address the unique challenges of graph data [208]. Park et al. [139] recover the training images from the predictions of the model to evaluate the privacy loss of a face recognition model and measure the success of model inversion attacks based on the performance of an evaluation model. The results have shown that even a high privacy budget $ϵ$ = 8 can provide protection against model inversion attacks.

4.3.4. Differential Privacy Auditing Using Model Extraction

When auditing DPML models using model extraction attacks, auditors evaluate how resistant a DP-protected model is to extraction attacks, where an attacker attempts to replicate or approximate the model by querying it multiple times and using the outputs to train a surrogate. This form of auditing is essential to verify that DP implementations truly protect against unintentional model replication, which can jeopardize privacy by allowing the attacker to learn sensitive information from the original model. The main scenarios that have been considered in the literature for auditing model extractions are query analyses.

Query analyses measure the extent to which queries can reveal model parameters or behaviors. Auditors simulate extraction attacks by extensively querying the model and analyzing how well they can replicate its outputs or decision boundaries.

Carlini et al. [101] show that embeddings can reveal private data, advancing research on the robustness of embeddings for DP models. Dziedzic et al. [209] require users to perform computational tasks before accessing model predictions. The proposed calibrated Proof of Work Mechanism (PoW) can deter attackers by increasing the model extraction effort and creating a balance between robustness and utility. Their work contributes to the broader field of privacy auditing by proposing proactive defenses instead of reactive measures in ML applications. Li et al. [210] investigate a novel personalized local differential mechanism to defend against the equation-solving attack and query-based attacks (solving for model parameters through multiple queries) by querying the model multiple times to solve for the model parameters. The authors concluded that this method is particularly effective against regression models and can be mitigated by adding high-dimensionality Gaussian noise to model coefficients. Li et al. [146] use the active verification two-stage privacy auditing method to detect suspicious users based on the query patterns and verifying if they are attackers. By analyzing how well the queries cover the feature space of the victim model, it can detect potential model extraction. Once suspicious users are identified, an active verification module is employed to confirm whether these users are indeed attackers. Their proposed method is particularly useful for object detection models through innovative use of feature space analysis and perturbation strategies. Zheng et al. [211] propose a novel privacy preserving mechanism, Boundary Differential Privacy ($ϵ$-BDM), which modifies the output layer of the model. BDP is designed to introduce carefully controlled noise around the decision boundary of the model. This method guarantees that an attacker cannot learn the decision boundary of two classes with a certain accuracy, regardless of the number of queries. The special layer, the so-called Boundary DP layer that applies differential privacy principles was implemented in the ML model. By integrating BDP into this layer, the model produces an output that preserves privacy around the boundary and effectively obscures information that could be exploited in extraction attacks. This boundary randomized response algorithm was developed for binary models and can be generalized to multiclass models. The extensive experiments (Zheng et al., 2022) [18] have shown that BDP obscures the prediction responses with noise and thus prevents attackers from learning the decision boundary of any two classes, regardless of the number of queries issued.

Yan et al. [212] propose an alternative to caching the BDP layer in which privacy loss is adapted accordingly. The authors propose an adaptive query-flooding parameter duplication (QPD) extraction attack that allows the auditor to infer model information with black-box access and without prior knowledge of the model parameters or training data. A defense strategy called monitoring-based DP (MDP) dynamically adjusts the noise added to the model responses based on real-time evaluations, providing effective protection against QPD attacks. Pillutla et al. [160] introduce a method in which multiple randomized canaries are added to audit the privacy guarantees by distinguish between models trained with different numbers of canaries in the dataset. By distinguishing between models trained with different numbers of canaries, the authors introduce Lifted Differential Privacy (LiDP) can effectively audit differentially private models. The introduction of novel confidence intervals that adapt to empirical high-order correlations to improve the accuracy and reliability of the auditing process.

4.3.5. Differential Privacy Auditing Using Property Inference

Auditing DP with property inference attacks typically focuses on extracting global features or statistical properties of a dataset used to train ML model, such as average age, frequency of diseases, or frequency of geographic locations, rather than specific data records to reveal sensitive information. The goal is to ensure that the DP mechanisms effectively prevent attackers from inferring sensitive characteristics even if they have access to the model outputs. The auditor checks whether the model reveals statistical properties of the training data that could violate privacy. The literature review on the property inference for differential privacy auditing presents different scenarios, such as evaluating property sensitivity with model outputs and attribute-based simulated worst-case scenarios.

Evaluating property sensitivity with model outputs test how well the DP obscures statistical dataset properties. Auditors analyze the extent to which an attacker could infer information at the aggregate or property level by examining model outputs across multiple queries. For example, changes in the distribution of outputs when querying specific demographics can reveal hidden patterns. An attribute-based simulation of a worst-case scenario is a case in which an attacker has partial information on certain attributes of the dataset. Auditors test the DP model by combining partially known data (e.g., location or age) with the model’s predictions to see if the model can reveal other attributes. This type of adversarial testing helps validate DP protections against more informed attacks.

Suri et al. [213] introduce the concept of distribution inference attacks in both white- box and black-box models, which motivated later DP studies to counteract these vulnerabilities. This type of inference attack aims to uncover sensitive properties of the underlying training data distribution, potentially exposing private information about individuals or groups within the dataset. The authors discuss auditing information disclosure at three granularity levels: distribution, user, and record level. This multifaceted approach allows for a comprehensive evaluation of privacy risks associated with ML models. Ganju et al. [214] show how property inference attacks on attributes can reveal the characteristics of datasets even when neural networks use DP. The authors introduce an approach for inferring properties that a model inadvertently memorizes, using both synthetic and real-world datasets.

Melis et al. [215] focus on collaborative learning using property inference in the context of shared model updates, focusing in particular on how unintended feature leakage can jeopardize privacy. By analyzing the model’s outputs, the author identify which features can leak information and which features contribute to the privacy risks. They introduce a method for inferring sensitive attributes that may only apply to subgroups, thereby revealing potential privacy vulnerabilities. Property inference attacks, in this case, rely on the detection of sensitive features in the training data. Attackers can exploit the linear property of queries to obtain multiple responses from DP mechanisms, leading to unexpected information leakage [215]. Huang and Zhou [215] address critical concerns about the limits of differential privacy (DP), especially in the context of linear queries. They show how the inherent linear properties of certain queries can lead to unexpected information leaks that undermine the privacy guarantees that DP is supposed to provide. Ben Hamida et al. [216] investigate how the implementation of differential privacy can reduce the likelihood of successful property inference attacks by obscuring the relationships between the model parameters and the underlying data properties. Song et al. [217] provide a comprehensive evaluation of privacy attacks on ML models, including property inference attacks. The authors propose attack strategies that target unintended model memorization, with empirical evaluations on DP-protected models. A list of privacy auditing schemes is shown in Table 2.

5. Discussion and Future Research

This paper presents the current trends in privacy auditing in DPML using membership inference, data poisoning, model inversion attacks, model extraction attacks, and property inference.

We consider the advantages of using membership inference for privacy auditing in DPML models through quantification of privacy risk, empirical evaluation, improved audit performance, and guidance for privacy parameter selection. MIAs can effectively quantify the privacy risk (the amount of private information) that a model leaks about individual data points in its training set. This makes them a valuable tool for auditing the privacy guarantees of DP models [41]. They provide a practical lower bound on inference risk complementing the theoretical upper bound of DP [16]. MIAs enable an empirical evaluation of privacy guarantees in DP models, helping to identify potential privacy leaks and implementation errors [160]. They can be used to calculate empirical identifiability scores that enable a more accurate assessment of privacy risks [34]. Advanced methods that combine MIAs with other techniques, such as influence-based poisoning attacks, have been shown to provide significantly improved audit performance compared to previous approaches [61]. MIAs can help in selecting appropriate privacy parameters (ε, δ) by providing insights into the trade-off between privacy and model utility [11,163]. These attacks require relatively weak assumptions about the adversary’s knowledge, making them applicable in different scenarios. This flexibility allows for broader applicability in real-world settings where attackers may have limited information about the mode [46].

We consider the drawbacks of using membership inference for privacy auditing in DPML models as impacting model utility, parameter selection complexity, and non-uniform risk across classes. Implementing DP to defend against MIAs often leads to a trade-off in which increased privacy leads to lower model accuracy [217]. Excessive addition of noise, which is required for strong privacy guarantees, can significantly degrade the utility of the model, especially in scenarios with imbalanced datasets. Choosing the right privacy parameters is challenging due to the variability of data sensitivity and distribution, making it difficult to effectively balance privacy and utility [35]. Legal and social norms for anonymization are not directly addressed by different privacy parameters, adding to the complexity. Some MIA methods, especially those that require additional model training or complex computations, can entail a significant computational overhead [96]. The development of robust auditing tools that can provide empirical assessments of privacy guarantees in DP models is crucial. These tools should take into account real-world data dependencies and provide practical measures of privacy loss [34]. Future research should focus on adaptive privacy mechanisms that can dynamically adjust privacy parameters based on the specific characteristics of the training data and the desired level of privacy.

Using data poisoning to audit privacy in DP provides valuable insight into vulnerabilities and helps quantify privacy guarantees, thus improving the understanding of the robustness of models. Data-poisoning attacks can reveal vulnerabilities in DP models by showing how easily an attacker can manipulate training data to influence model outputs. This helps identify vulnerabilities that are not obvious through standard auditing methods [46]. Data poisoning can help evaluate the robustness of DP mechanisms against attacks by attackers. By understanding how models react to poisoned data, we can improve their design and implementation [61]. By using data-poisoning techniques, auditors can quantitatively measure the privacy guarantees of differentially private algorithms [2]. This empirical approach complements theoretical analyses and provides a clearer understanding of how privacy is maintained in practice [43]. The use of data poisoning for auditing can be generalized to different models and algorithms, making it a versatile tool for evaluating the privacy of different machine-learning implementations [61].

However, it also poses challenges in terms of complexity, potential misuse, and the limits of the scope of application, which must be carefully considered in practice. Conducting data-poisoning attacks requires significant computational resources and expertise. Developing effective poisoning strategies can be complex and may not be feasible for all organizations [43,61]. Data-poisoning attacks may not cover all aspects of privacy auditing. While they may reveal certain vulnerabilities, they may not address other types of privacy breaches or provide a comprehensive view of the overall security of a model [43]. Techniques developed for data poisoning in the context of audits could be misused by malicious actors to exploit vulnerabilities in ML models. This dual use raises ethical concerns about the impact of such research [43]. The effectiveness of data-poisoning attacks can vary greatly depending on the specific characteristics of the model being audited. If the model is robust against certain types of poisoning attacks, the auditing process may lead to misleading results regarding its privacy guarantees.

Future research could focus on developing more robust data-poisoning techniques that can effectively audit DP models. By refining these methods, auditors can better assess the resilience of models to different types of poisoning attacks, leading to improved privacy guarantees. As federated learning becomes more widespread, interest in how data-poisoning attacks can be used in this context is likely to grow. Researchers could explore how to audit federated learning models for DP, taking into account the particular challenges posed by decentralized data and model updates. The development of automated frameworks that utilize data poisoning for auditing could streamline the process of evaluating differentially private models. Such frameworks would allow organizations to routinely assess the privacy guarantees of their models without the need for extensive manual intervention. There is a trend towards introducing standardized quantitative metrics for evaluating the effectiveness of DP mechanisms using data poisoning [63]. This could lead to more consistent and comparable assessments across different models and applications.

Model inversion attacks can expose vulnerabilities in DP models by showing how easily an attacker can reconstruct sensitive training data from the model outputs. This helps identify vulnerabilities that may not be obvious through standard auditing methods [58]. The model inversion serves as a benchmark for evaluating the effectiveness of different DP mechanisms. By evaluating a model’s resilience to inversion attacks, auditors can assess whether the models fulfil the privacy guarantees, enhancing trust in privacy protection technologies [200]. By using model inversion techniques, auditors can quantitatively measure the privacy guarantees provided by DP algorithms [199]. This empirical approach provides a better understanding of how well privacy is maintained in practice. The insights gained from the model inversion can inform developers about necessary adjustments to strengthen privacy protection. This iterative feedback loop can lead to continuous improvement of model security against potential attacks. Model inversion techniques can be applied to different types of ML models, making them versatile tools for evaluating the privacy of different implementations.

Performing model inversion audits can be computationally expensive and time-consuming, requiring significant resources for both implementation and analysis. This can limit accessibility for smaller organizations or projects with limited budgets [48]. Model inversion methods often rely on strong assumptions about the attacker’s capabilities, including knowledge of the architecture and parameters of the model [198]. This may not reflect real-world scenarios where attackers have limited access, which can lead to an overestimation of privacy risks. Practical implementations of algorithms with varying levels of privacy often contain subtle vulnerabilities, making it difficult to audit at scale, especially in federated environments [118]. The results of model inversion audits can be complex and may require expert interpretation to fully understand their implications. This complexity can hinder the effective communication of results to stakeholders who may not have a technical background [48]. While model inversion attacks are effective in detecting certain vulnerabilities, they may not cover all aspects of privacy auditing. While DP is an effective means of protecting the confidentiality of data, it has problems preventing model inversion attacks in regression models [136]. Other types of privacy violations may not be captured by this method, resulting in an incomplete overview of the overall security of a model.

Future research could investigate the implementation of DP at the class and subclass level to strengthen defenses against model inversion attacks. These approaches could enable more granular privacy guarantees that protect sensitive attributes related to specific data classes while enabling useful model outputs [58]. The use of the stochastic gradient descent (SGD) algorithm as a strategic approach to address the challenge of selecting an appropriate value for the privacy budget shows a possible future application of model inversion to optimize privacy budget selection. There may be a trend towards dynamic privacy budgeting, where the privacy budget is adjusted in real time based on the context and sensitivity of the data being processed. This could help to better balance the trade-off between privacy and utility, especially in scenarios that are prone to model inversion attacks [136].

Model extraction attacks pose a significant privacy risk, even when DP mechanisms are used [101]. These attacks aim to replicate the functionality of a target model by querying it and using the answers to infer its parameters or training data. Model extraction attacks can derive the parameters of a machine-learning model through public queries [209]. Even with DP, which adds noise to the model outputs to protect privacy, these attacks can still be effective. For example, the adaptive query-flooding parameter duplication (QPD) attack can infer model information with black-box access and without prior knowledge of the model parameters or training data [212].

Current trends in privacy auditing in the context of DPML show that the focus is on developing efficient, effective frameworks and methods for evaluating privacy guarantees. As the field continues to advance, ongoing research is critical to refine these auditing techniques and schemes, address the challenges related to the privacy–utility trade-off, and improve the practical applicability of DPML systems in real-world settings. We hope that this article provides insight into privacy auditing in both local and global DP.