DSAK: Distillation of Self-Adaptive Knowledge for Membership Privacy Protection

Abstract

The utilization of machine learning models is extensive in a wide array of significant applications. However, their vulnerability to security and privacy attacks is a serious concern, for example, for the protection of financially sensitive data such as account flow. Particularly troubling is the threat of membership inference, which enables attackers to determine whether a given data sample is included in the training set of a targeted machine-learning model. Existing knowledge distillation techniques have shown promise in balancing model performance with data privacy. However, achieving superior privacy during the training process of the target model is challenging due to the teacher model’s performance limitations and the scarcity of unlabeled benchmark data. To address this issue, we propose a novel framework called Distillation of Self-Adaptive Knowledge (DSAK). DSAK utilizes self-duplicated teacher and noise-generative models to introduce specialized self-adaptive noise for privacy training in the target model. By incorporating new data features derived from this noise, DSAK improves model performance and reduces the risk of memorizing member data. Experimental results demonstrate DSAK’s effectiveness in defending against existing attack schemes across multiple datasets while surpassing other membership inference defense schemes in terms of efficiency.

1. Introduction

In recent years, significant progress has been made in developing machine learning (ML) models. These models have demonstrated impressive performance across a range of domains, including image recognition [1], speech recognition [2,3], natural language processing [4], music understanding [5], blockchain [6], and cryptocurrency transactions [7]. This trend has spurred heightened interest in the utilization of ML in real-world scenarios [8,9]. Despite advances in ML algorithms, effectively training them often requires large datasets, which may contain sensitive information from domains such as finance and healthcare. This necessitates handling data with the utmost discretion. As a result, concerns have been raised about the ability of ML algorithms to ensure robust privacy protection in such situations [10]. Unfortunately, recent studies [11,12] indicate that ML models are vulnerable to privacy breaches. These models can inadvertently reveal or retain sensitive information during training, making them susceptible to various privacy attacks such as dataset reconstruction [13], attribute inference [14], and membership inference attacks [15]. Membership inference attacks (MIAs) have drawn significant attention due to their ability to determine whether a specific data sample was included in the victim model’s training data. MIAs exploit various characteristics, including output confidence [16], prediction loss [17], prediction entropy [18,19], and data robustness [20,21], to make precise inferences about the membership status of a data sample. In black-box ML service scenarios, the ease with which attackers can obtain these model properties makes an MIA a significant threat to the privacy of sensitive data with minimal effort [22].

To prevent the unauthorized leakage of private data by ML services, research in ML privacy protection focuses on developing effective defenses against MIAs. The core of existing defensive strategies involves altering the output of ML models to make it difficult for adversaries to differentiate between sensitive and non-sensitive data. These methods can be pursued in two distinct directions: (1) Influencing the training process of the model through techniques such as differential privacy [23,24], relaxed training [25], adversarial regularization [26], and knowledge distillation [27,28]. (2) Influencing the inference process of the model, for instance, through confidence score masking [29]. These methods aim to adjust the model’s output or operational characteristics during the inference phase to enhance privacy safeguards. Nevertheless, the effectiveness of these defense mechanisms is limited by the challenge of achieving a sufficient balance between privacy and practicality. Moreover, these methods may not be adequate to protect against all forms of attacks.

Our work proposes a cutting-edge framework for the protection against MIAs, named Distillation of Self-Adaptive Knowledge (DSAK). It is an effective combination of knowledge distillation and generative networks. The majority of contemporary cutting-edge knowledge distillation methods leverage teacher models to filter confidential training data, enabling the training of secure target models without the necessity for manual labeling [22]. These methods reduce the vulnerability to MIAs by restricting the target model’s capacity to retain excessive details from the training data. Nonetheless, they often necessitate additional training datasets [30] or may not achieve the desired performance levels in unprotected environments [31], as the teacher model can act as a bottleneck in the distillation process and fail to supply the student model with extra descriptive insights. Diverging from existing knowledge distillation methods, DSAK introduces several key innovations to address the limitations of privacy leakage and performance degradation: (a) self-adaptive noise generation, which leverages a noise-generative model to perturb private training data, thereby mitigating membership inference risks while eliminating the reliance on additional datasets and enhancing data diversity; (b) dynamic teacher updating, which employs a parameter-passing mechanism to continuously optimize the teacher model, overcoming the fixed-teacher bottleneck and improving knowledge transfer efficiency; and (c) joint privacy-utility optimization, which integrates adaptive data perturbation with co-evolving teacher–student learning to break the traditional trade-off between privacy protection and model performance. The distinct differences between DSAK and prior methods are illustrated in Figure 1. This paper’s principal contributions are delineated in the following points:

• We put forward a defense mechanism aimed at thwarting black-box membership inference attacks, called Distillation of Self-Adaptive Knowledge (DSAK), which innovatively introduces noise generative models that enable the high-quality privacy training of machine learning models without introducing additional data.
• We introduce a special type of data noise, called self-adaptive noise, which can provide additional data features to the training process of the machine learning model and circumvent its excessive memory of the membership data.
• We perform a detailed examination of DSAK to evidence its contemporary balance of data privacy with model correctness. For example, when considering an increased level of membership privacy, DSAK achieves significantly higher classification task accuracies, ranging from 30% to 65.3%, compared to state-of-the-art defense mechanisms across various classification tasks.

Figure 1. The specific variations that mark the contrast between DSAK and previous methodologies in knowledge distillation [30].

Figure 1. The specific variations that mark the contrast between DSAK and previous methodologies in knowledge distillation [30].

2. Related Work

2.1. Membership Inference Attack

Membership inference attacks (MIAs) refer to a collection of privacy attacks that aim to deduce whether a specific data sample is part of the training dataset of a machine learning model. Various MIAs have been proposed, which can be categorized into three distinct types based on the information obtained from the targeted model, as detailed below.

2.1.1. Label Information-Based Methods

In actual implementation scenarios, assaults that depend on label-related information are assessed to occur more frequently. Precedent academic projects [20,32] have developed an array of membership inference offensives based on these premises. For example, Label-Leaks [33] successfully investigated the utilization of label information alone in executing MIAs. This implies that the vulnerability of machine learning models to privacy attacks may be greater than previously assumed, and attackers can potentially leverage limited information to deduce membership data.

2.1.2. Partial Output Information-Based Methods

Reza Shokri et al. [16] introduced a classical MIA that utilizes partial output information to distinguish between members and non-members. This attack relies on the predicted output confidence of the shadow model. Subsequently, Ml-leaks [34] refined the technique by easing the restrictions on model architecture, training datasets, and the number of shadow models employed. This innovation facilitates the inference of membership in access data by utilizing the highest output confidence. Building upon these findings, Liwei Song et al. [19] have further contributed to the field by advancing the attack methodology through the establishment of a range of thresholds for different class labels.

2.1.3. Total Output Information-Based Methods

In this type of assault, adversaries are capable of executing MIAs by gaining access to statistical information from the target model. A widely used strategy is the prediction loss-based approach, first introduced by Samuel Yeom et al. [21]. Ml-leaks [34] subsequently showcased the potency of employing prediction entropy to gauge the model’s predictive uncertainty for attack purposes. In more recent developments, Liwei Song et al. [19] put forward an advanced attack strategy that integrates prediction entropy and takes into account the correlation of training data and the inter-point distances within the feature space of the victim model.

2.2. Defenses Against Membership Inference Attack

The emergence of MIAs in the privacy of the machine learning field has prompted a surge of research into developing defense methods. The subsequent list outlines a summary of prevalent defense strategies.

2.2.1. Noise Disturbance-Based Methods

Noise perturbation is a widely adopted method for privacy preservation, implemented as Differential Privacy and Confidence Score Perturbation. Differential Privacy ensures privacy for individual samples and protects against MIAs [24,35,36]. However, achieving the optimal balance between privacy and performance is challenging. To address performance issues, the defense mechanism introduces noise perturbations to the confidence score vector [37,38,39,40]. Perturbing confidence scores, while easier to execute, has demonstrated efficacy in practical applications, although it is not foolproof against all forms of attacks.

2.2.2. Suppressing Overfitting-Based Methods

In the realm of ML, overfitting presents a significant challenge and increases susceptibility to MIAs. Addressing overfitting has become a crucial area of research in developing countermeasures against MIAs, with strategies such as Regularization Training and Relax Learning being implemented. Adversarial regularization methods, as discussed by Deep Learning [23] and Ear [41], bolster model generalization by incorporating adversarial instances. Meanwhile, the Relax Loss approach, introduced by Junjie Chen et al. [42], aims to improve generalization through the adoption of an innovative loss function. While these defenses help to mitigate overfitting and reduce vulnerability to MIAs in ML models, achieving an optimal balance between preserving privacy and maintaining performance remains a complex task.

2.2.3. Knowledge Distillation-Based Methods

Knowledge distillation functions as a model compression strategy, aiming to transfer insights from a comprehensive, high-accuracy model (the teacher) to a more compact model (the student). This process aims to reduce computational requirements while preserving the model’s accuracy. As noted by Kd-gan [43], this method has been observed to mitigate vulnerabilities to MIAs. Furthermore, DMP [30] introduced a novel protective framework, DMP, which incorporates a knowledge distillation approach. This approach utilizes a subset of membership data to educate a teacher model, subsequently employing this teacher model to guide the unsupervised training of the target model. Recent work [44] on multi-branch alignment for vision-language tasks further highlights the role of collaborative representation learning in improving knowledge transfer and model robustness. The implementation of these strategies has demonstrated that defenses based on knowledge distillation provide robust protection against MIAs.

3. Preliminaries

This section offers an overview of the terminology and preliminary knowledge essential for describing this paper. Various terminologies are detailed in Table 1.

Table 1. DSAK notations [28].

Notations	Description	Notations	Description
$x_i,y_i$	Membership Data Samples	${x_i}^{\prime },{y_i}^{\prime }$	Non-Membership Data Samples
D	Private Membership Dataset	$D^A$	Shadow Membership Dataset
$D^{\prime }$	Private Non-Membership Dataset	${D^A}^{\prime }$	Shadow Non-Membership Dataset
f	Target Classifier Model	B	Batch Size of Data Samples
t	Teacher classifier Model	C	Number of Output Classes
g	Self-adaptive Noise Generative Model	K	Number of Iterations
$\alpha$	Label Weights of TSG in DSAK	$\beta$	Distant Weights of TSG in DSAK
$\lambda$	Probability of Member Data Application of SKD in DSAK	∇	Amount of Variation of Model Parameters
$E_{pre}$	Training Epochs of PTG in DSAK	$E_t$	Parameter Transfer Epochs of TTM in DSAK
$\theta$, $\delta$, $\eta$	Parameters of Model f, t and g	$l_f$, $l_t$, $l_g$	Learning Rates of Model f, t and g

3.1. Membership Data and Membership Inference Attack

The data pairs $(x,y)$ used to train the machine learning model are referred to as membership data, and the set of these data samples is defined as the membership dataset: $D={\left\{(x_i,y_i)\right\}}_{i=1}^N$. In addition, data pairs $(x^{\prime },y^{\prime })$ that are not used in the training process are referred to as non-membership data and form the non-membership dataset $D^{\prime }={\left\{(x_i^{\prime },y_i^{\prime })\right\}}_{i=1}^N$. Where N represents the total number of data samples. Due to the privacy properties specific to membership data, MIAs are devoted to determining whether a given data sample, x or $x^{\prime }$, belongs to the membership dataset D or the non-membership dataset $D^{\prime }$.

3.2. Machine Learning Classification

The central objective of this research is to devise a protective strategy for machine learning models deployed in scenarios involving supervised learning and classification tasks. In the context of our analysis, we denote these models as f. The objective of f is to learn a parameter vector $\theta$ representing a mapping from input x to output y. f generates a C-dimensional vector output, where each dimension represents the probability of the input belonging to a specific class.

3.3. Machine Learning Generativation

The noise generation model used in this study is a component of machine learning generation, a technique in which a machine learning model learns from existing data to generate novel output. In the context of our analysis, we denote these models as g.

In our research, the goal of function g is to acquire a set of parameters $\eta$ that define the transformation from input data x to the resultant n, which represents the self-adaptive noise generated based on the features of this input sample. Notably, the variables n and x are deliberately designed to possess identical data dimensions, facilitating seamless data-superimposed operations.

4. Our New Method

This research introduces a novel training framework, DSAK, aimed at providing an advanced privacy training approach to mitigate the risk of membership inference for the target class classifier model. The mechanism of DSAK is visually represented in Figure 2 and Algorithm 1. It initiates with a pre-training phase, PTG, and continues with three phases of knowledge distillation: TTM, TSG, and SKD. For clarity, the input data samples for all models discussed in this section are symbolized by x, while their accurate labels are symbolized by y.

Algorithm 1 Distillation of Self-Adaptive Knowledge

Input: Dataset D Parameter: Training epochs E, PTG training epochs $E_{pre}$, TTM parameter transfer epochs $E_t$, learning rates $l_f$, $l_t$, $l_g$, weighting parameters $\alpha$, $\beta$, $\lambda$ Output: Optimum model $f\left(\theta \right)$   1: Initialize model $f\left(\theta \right)$;   2: Initialize model $t\left(\delta \right)$;   3: Initialize model $g\left(\eta \right)$;   4: for $epoch$ in {1,…, $E_{pre}$} do   5:    for i in {1,…, K} do   6:      Get sample batch ${\left\{(x_i,y_i)\right\}}_{i=1}^B$   7:      Compute pre-training loss $l_{ptg}\left(\eta \right)$ of g   8:            $l_{ptg}\left(\eta \right)={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{l}_{ptg}(g,x_i)$   9:      Update parameter $\eta$ 10:            $\eta \leftarrow \eta +l_g·\nabla l_{ptg}\left(\eta \right)$ 11:    end for 12: end for 13: for $epoch$ in {1,…, E} do 14:    for i in {1,…, K} do 15:      Get sample batch ${\left\{(x_i,y_i)\right\}}_{i=1}^B$ 16:      Compute distillation training loss $l_{ttm}\left(\delta \right)$ of t 17:            $l_{ttm}\left(\delta \right)={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{l}_{ttm}(t,x_i,y_i)$ 18:      Update parameter $\delta$ 19:            $\delta \leftarrow \delta +l_t·\nabla l_{ttm}\left(\delta \right)$ 20:      Compute distillation training loss $l_{tsg}\left(\eta \right)$ of g 21:            $l_{tsg}\left(\eta \right)={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{l}_{tsg}(g,t,x_i,y_i)$ 22:      Update parameter $\eta$ 23:            $\eta \leftarrow \eta +l_g·\nabla l_{tsg}\left(\eta \right)$ 24:      Compute distillation training loss $l_{skd}\left(\theta \right)$ of f 25:            $l_{skd}\left(\theta \right)={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{l}_{skd}(f,g,t,x_i,y_i)$ 26:      Update parameter $\theta$ 27:            $\theta \leftarrow \theta +l_g·\nabla l_{skd}\left(\theta \right)$ 28:    end for 29:    if $epoch$ % $E_t$ == 0 then 30:      $\delta$ ← $\theta$ 31:    end if 32: end for

Figure 2. The illustration of DSAK.

4.1. Pre-Training of Self-Adaptive Noise Generation Model

The primary objective of the Pre-Training of Self-Adaptive Noise Generation Model (PTG) phase is to reduce the variability in the noise generation process and minimize its potential impact on the classifier model f at the outset of knowledge distillation training.

To facilitate this, PTG pre-trains the self-adaptive noise-generative model, denoted as g, for $E_{pre}$ epochs before initiating the knowledge distillation training of DSAK. This training aims to minimize the generation noise, such that the data samples superimposed with noise could be converged to their original samples. Inspired by Arcface [45], this phase adopts the application of cosine similarity to quantify the similarity values between the two data samples, as illustrated in Equation (1).

$$l_{ptg}(g,x)=\sum _{x\in D}{\displaystyle \frac{\left(g\right(x)+x)·x}{\sqrt{{(g\left(x\right)+x)}^2}·\sqrt{x^2}}}.$$

(1)

4.2. Training of Teacher Model

The intent behind the Training of Teacher Model (TTM) phase is to train a teacher model t to guide the training of the target classifier model f and the self-adaptive noise-generative model g. Given that the primary role of the teacher model is to provide training guidance for the other models in the system, its training is solely constrained by the loss of the classification task, as defined in Equation (2).

$$l_{ttm}(t,x,y)=\sum _{x,y\in D}t\left(x\right)·log\left(y\right).$$

(2)

Notably, if the teacher model t is trained solely on private training data samples, it would hinder the effectiveness of training guidance if overfitting occurs. To mitigate this situation, during the knowledge extraction training of DSAK, the teacher model t parameters are substituted with the target classifier model f parameters at regular intervals of $E_t$. This substitution is achieved by applying the Exponential Moving Average (EMA) [46] technique. In cases where the model consists solely of linear layers, a complete replacement of parameters is performed instead.

4.3. Training of Self-Adaptive Noise-Generative Model

The primary objective of the Training of Self-Adaptive Noise-Generative Model (TSG) phase is to train the generative model g for generating a special kind of data noise, which is denoted as self-adaptive noise, guided by the teacher model t. The self-adaptive noise aligns the superimposed noisy data samples with key features of the private training data samples while maximizing distance with non-key features. The former purpose seeks to minimize Equation (3), while the latter seeks to minimize Equation (4).

$$l_{tsg-f}(g,t,x,y)=\sum _{x,y\in D}t(g\left(x\right)+x)·log\left(y\right).$$

(3)

$$l_{tsg-l}(g,x)=-\sum _{x\in D}log\left({\displaystyle \frac{\left(g\right(x)+x)·x}{\sqrt{{(g\left(x\right)+x)}^2}·\sqrt{x^2}}}\right).$$

(4)

The DSAK framework utilizes the aforementioned superimposed noise data samples to train the classifier model f, thereby mitigating the issue of overfitting at the data level. Ultimately, the TSG phase seeks to minimize Equation (5).

$$l_{tsg}(g,t,x,y)=\alpha ·l_{tsg-f}(g,t,x,y)+\beta ·l_{tsg-l}(g,x),$$

(5)

where $\alpha$ and $\beta$ represent the weights of branch losses $l_{tsg-f}$ and $l_{tsg-l}$ in the total loss $l_{tsg}$ of the TSG phase.

4.4. Self-Knowledge Distillation

The Self-Knowledge Distillation (SKD) serves as the concluding phase in DSAK training, with its primary objective being the thorough distillation of distant self-knowledge from the classifier model f. This process is facilitated by collaborating the teacher model t and the noise-generative model g. In this phase, the target classifier model f is trained using the self-adaptive noise overlay data, the visualization of which is presented in Figure 3. The TSG phase seeks to minimize Equation (6).

$$\begin{array}{c}\hfill l_{skd}(f,g,t,x,y)=\sum _{x,y\in D}f(g\left(x\right)+x)·log\left(t\left(x\right)\right)\\ \hfill +\lambda ·f\left(x\right)·log\left(t\right(x\left)\right),\end{array}$$

(6)

where $\lambda$ represents the probability of participating in the training utilizing private training data samples directly in the training of the SKD phase.

Figure 3. Visualization of CIFAR100 data samples with superimposed self-adaptive noise.

4.5. Datasets

Our research undertakes the classification challenge across three distinct datasets, specifically starting with the principal benchmark dataset widely recognized in the field of machine learning, CIFAR100 [47]. In addition, we incorporate two supplementary validation datasets: Texas100 and Purchase100. Notably, these ancillary datasets are streamlined adaptations made available by the original authors [16] and were initially used to conduct the pioneering membership inference assaults on ML models.

CIFAR100. This dataset is a widely acknowledged benchmark for testing the effectiveness of image recognition software. It consists of a comprehensive set of 60,000 images, each sized at 32 × 32 pixels. The images are divided into 100 separate classes, with each class representing a specific object type.

Texas100. This dataset comprises hospital discharge records, featuring in-depth inpatient data from a multitude of medical centers, as disclosed by the Texas Department of State Health Services. It includes comprehensive details on injury causation, diagnoses, patient behaviors, and general demographics, including sex, age, race, hospital identification number, and stay duration. With 67,330 records and 6170 binary indicators for the 100 most frequent medical operations, the dataset segments the records into 100 classes, each indicative of a specific patient profile.

Purchase100. This dataset is based on Kaggle’s “Get Valuable Shoppers” challenge. The dataset includes the shopping records of several thousand people. The challenge aims to find discounts to attract new shoppers to buy new products. The dataset contains 197,324 data records. Each data record corresponds to one customer and has 600 binary features (each corresponding to one item). Each feature reflects whether the item was purchased by the customer or not. The data are aggregated into 100 classes, and the task is to predict the category of each customer.

Dataset Partition

The datasets under consideration are partitioned according to the prescribed format laid out in Table 2. The confidential data are divided into two subsets, denoted as D and $D^{\prime }$, whereas the adversarial data, simulated by a hypothetical attacker, are allocated as $D^A$ and ${D^A}^{\prime }$. It should be noted that the datasets are entirely disjoint. While dataset D has been deployed for training the victim classification model f, $D^A$ has been earmarked for training the shadow models of MIA. During the validation of the classification performance metrics, the efficacy of the defense model on dataset $D^{\prime }$ serves as the preferred performance metric. Simultaneously, the probability of D successfully being inferred is leveraged to evaluate the privacy risk of the defense model.

Table 2. The setup includes the number of data samples used in the experiment. The privacy data comprises datasets D and $D^{\prime }$, while the shadow data consists of datasets $D^A$ and ${D^A}^{\prime }$. D is utilized to train the target classifier models, whereas $D^A$ is used to train the shadow models. $D^{\prime }$ is regarded as non-membership data in the privacy data, and ${D^A}^{\prime }$ is considered non-membership data in the shadow data.

Dataset	Private Data		Shadow Data
	$-\mathit{D}-$	$-{\mathit{D}}^{\prime }-$	$-{\mathit{D}}^{\mathit{A}}-$	$-{{\mathit{D}}^{\mathit{A}}}^{\prime }-$
CIFAR100	$2.5\times {10}^3$	$5\times {10}^3$	$2.5\times {10}^3$	$5\times {10}^3$
Texas100	$10\times {10}^3$	$5\times {10}^3$	$10\times {10}^3$	$5\times {10}^3$
Purchase100	$10\times {10}^3$	$10\times {10}^3$	$10\times {10}^3$	$10\times {10}^3$

5. Experimental Design and Implementation

We conducted experiments to validate our method and explain the design in this section.

5.1. Models

To ensure an impartial evaluation of the effectiveness of various defense mechanisms, we have implemented separate baseline classifier models for each of the datasets considered, which are AlexNet [48] and DenseNet [49] for the CIFAR100 dataset, 4-layer fully connected neural network (NN Net) for the Texas100 dataset, and 3-layer NN Net for the Purchase100 dataset. Moreover, the RenseNet [50] and 4-layer NN Net are employed as the self-adaptive noise-generative model of DSAK in the image datasets(CIFAR100) and others (Texas100, Purchase100).

5.2. Attack Methods

The following is a concise overview of the four distinct membership inference attack techniques employed for our validation process:

Attack 1 ($I_{corr}$): Stolen memories [51] developed a MIA technique that hinges on predictive accuracy. If the target model does not generalize well, it can correctly predict the training data but make incorrect predictions on the test data. This method can easily infer the membership of a sample, deciding if it is a member or non-member, by evaluating the model’s predictive success on the input sample.

Attack 2 ($I_{conf}$): Liwei Song et al. [19] introduced an approach that employs distinct thresholds for various categories, aiming to enhance the success rate of MIA. The researchers propose the hypothesis that class imbalance within the dataset could lead to varying degrees of model confidence across different classes. Membership is inferred if the sample’s prediction confidence exceeds a predetermined threshold, while it is deemed a non-member if it falls below.

Attack 3 ($I_{entr}$): In their research, Liwei Song et al. [19] introduced an MIA that relies on predicted entropy, akin to the attack based on confidence scores. A data sample is deemed a member if its predicted entropy falls below an established threshold; conversely, it is classified as a non-member if the entropy surpasses this threshold.

Attack 4 ($I_{mentr}$): In their research, Liwei Song et al. [19] found that directly using the predicted entropy is flawed, thus, devised a method for calculating the predicted entropy correction that considers the actual class labels.

Attack 5 ($I_{lira}$): Carlini et al. [52] proposed a likelihood ratio-based membership inference attack, which estimates the likelihood difference between member and non-member samples by modeling their output distributions. This method provides a more powerful inference capability than traditional threshold-based attacks.

5.3. Settings of DSAK

DSAK sets different parameters for different data forms in pursuit of excellent training results. Table 3 shows the recommended parameter settings.

Table 3. Parameter settings for DSAK.

Dataset	Model	${\mathit{E}}_{\mathit{t}}$	${\mathit{l}}_{\mathit{f}}$	${\mathit{l}}_{\mathit{t}}$	${\mathit{l}}_{\mathit{g}}$	$\mathit{\alpha }$	$\mathit{\beta }$	$\mathit{\lambda }$
CIFAR100	AlexNet	4	$1\times {10}^{-4}$	$1\times {10}^{-4}$	$1\times {10}^{-4}$	1	10	0
	DenseNet	1	$5\times {10}^{-4}$	$5\times {10}^{-4}$	$5\times {10}^{-4}$	1	50	0
Texas100	NN	5	$5\times {10}^{-4}$	$5\times {10}^{-4}$	$5\times {10}^{-4}$	1	1	$5\times {10}^{-1}$
Purchase100	NN	16	$1\times {10}^{-4}$	$1\times {10}^{-4}$	$1\times {10}^{-4}$	1	1	$5\times {10}^{-1}$

5.4. Methods of Comparison

To verify the effectiveness and sophistication of DSAK, we display an experimental comparison against a spectrum of current best-in-class defense approaches for MIA protection, analyzing different viewpoints, which are, in detail, as follows.

5.4.1. Noise-Based Methods

This series of defense methods focuses on privacy training by adding noise to the model’s training process. We select the following methods for comparison: (i) Differentially Private Stochastic Gradient Descent (DP-SGD) [35]: It employs a combination of stochastic gradient descent and differential privacy techniques to ensure the privacy of sensitive data during the training process. DP-SGD introduces controlled random noise into gradient updates, which prevents the leakage of individual data samples while preserving the utility and accuracy of the trained model. (ii) MemGuard [37]: It incorporates randomness into the certainty metrics generated by the target model, while keeping the assigned labels for the input data unchanged.

5.4.2. Suppressing Overfitting Methods

This series of defense methods mainly aims to prevent overfitting in the target model through specific training techniques, ultimately achieving effective defense against attacks. We select the following methods for comparison: (i) Adversarial Regularization (AdvReg) [53]: It integrates adversarial examples into the training process, forcing the model to learn more generalized and robust representations to fortify the models against adversarial attacks. (ii) Relax Loss [25]: It introduces a relaxation parameter that allows the model to learn from both correct and incorrect predictions within a certain margin, avoiding overfitting predictions.

5.4.3. Methods in Knowledge Distillation

To conduct an effective comparison with existing state-of-the-art knowledge distillation techniques, we selected Distillation for Membership Privacy (DMP) [30] as the comparison method. DMP involves introducing a teacher model for the unlabeled training of the target model. This allows the target model to perform well without direct exposure to samples from the membership dataset. Notably, to achieve a fair comparison of defense effectiveness and to simulate the actual ML service environment as much as possible, an additional equal amount of unlabeled data will not be introduced for DMP execution in this validation as in DMP [30], but instead the labels of the membership dataset D are removed and used to achieve the unlabeled training of the target model after the teacher model training is completed.

6. Results and Discussion

6.1. Privacy Training Effectiveness of DSAK

Table 4 presents the performance and membership inference risk analysis results of the classifier model before and after the implementation of DSAK. The application of DSAK yields a significant reduction in the risk of privacy loss, as evidenced by a notable decrease in attack accuracy in multiple MIA across various datasets, resulting in an accuracy level of nearly 50%, which is equivalent to random guessing. Furthermore, the integration of DSAK exerts only a slight negative impact on the model’s operational effectiveness. In fact, it enhances the classification accuracy of DSAK, especially within the CIFAR100 and Texa100 classification tasks. These observations validate the superior performance and privacy-safeguarding attributes of DSAK as a protective measure.

Table 4. This analysis compares the training and testing accuracy rates, along with the variations in membership inference risk, for models both incorporating and excluding the DSAK. The table presents all accuracy percentages, with the DSAK’s performance metrics distinctively indicated in gray. Among them, ↑ indicates that a higher value of this metric is better (and ↓ indicates the opposite). Additionally, we have highlighted the best accuracy achieved in the classification task in blue, and the reduction in membership inference attack success post-defense in green.

Dataset	Model	Defense	Task Accuracy (%) ↑		Membership Inference Attack Accuracy ↓
			Train	Test	Avg.	${\mathit{I}}_{\mathit{corr}}$	${\mathit{I}}_{\mathit{conf}}$	${\mathit{I}}_{\mathit{entr}}$	${\mathit{I}}_{\mathit{mentr}}$	${\mathit{I}}_{\mathit{ilra}}$
CIFAR100	AlexNet	w/o	99.3	32.7	81.3	83.3	80.2	80.3	71.4	91.2
		w	46.5	33.4	52.1–29.2	56.6	50.0	50.1	50.8	53.1
	DenseNet	w/o	98.9	53.7	75.3	72.6	73.2	73.1	77.4	80.4
		w	69.7	54.1	55.7–19.6	57.8	54.4	54.2	56.7	55.2
Texas100	NN Net	w/o	82.2	53.2	62.0	64.5	56.9	55.6	62.8	70.2
		w	63.2	53.9	52.7–9.3	54.6	52.0	52.1	52.5	52.1
Purchase100	NN Net	w/o	97.7	76.5	62.1	60.6	60.3	60.3	64.0	65.4
		w	90.4	76.8	53.9–8.2	56.8	52.7	51.2	53.7	55.3

Compared with existing defense mechanisms, this improvement can be attributed to the integration of self-adaptive noise generation and parameter transfer in DSAK. Specifically, the noise-generative model produces sample-wise adaptive perturbations, which effectively obscure membership-related signals while preserving essential semantic information. Meanwhile, the parameter transfer mechanism enables the joint optimization of the teacher and target models during training, mitigating overfitting and preventing the teacher from becoming a static bottleneck. The combination of these two components reduces the distinguishability between member and non-member samples while maintaining effective knowledge transfer, thereby enhancing resistance to membership inference attacks.

6.2. Comparative Analysis of the Noise-Based Methods

The comparison in Table 5 positions DSAK against noise-based defense strategies, DP-SGD, and MemGuard. DP-SGD’s analysis is restricted to the CIFAR100 classification challenge with AlexNet, in accordance with differential privacy theory [35]. DP-SGD is adept at countering MIA, reducing their accuracy to approximately 50%, but this reduction comes at the expense of the target model’s classification precision. In contrast, MemGuard preserves classification accuracy and is highly effective against confidence-based MIA. However, it encounters difficulties with attacks based on the model’s output labels. Comparatively, DSAK substantially lowers the success rate of MIA across all assessed metrics while either maintaining or augmenting classification performance. These observations validate DSAK’s superior proficiency in balancing the dual objectives of membership inference defense and classification task accuracy.

Table 5. Analysis of the defensive capabilities under various data conditions and against a range of attacks. All accuracy values presented in this table are in percentage form. Among them, ↑ indicates that a higher value of this metric is better (and ↓ indicates the opposite). The protective efficacy of our approach serves as a reference point, distinctively marked in gray. Concurrently, $Tes{t}^{+}$ denotes the proportional enhancement in $Test$ accuracy for DSAK relative to other methods, while ${\mathit{Avg.}}^{-}$ illustrates the proportional decrease in $\mathit{Avg.}$ accuracy for DSAK in comparison to them. Improvements are accentuated with green, whereas decrements are indicated with red.

Dataset	Model	Defense	Task Accuracy (%) ↑			Membership Inference Attack Accuracy (%) ↓
			Train	Test	${\mathit{Test}}^{+}$	Avg.	${\mathit{Avg.}}^{-}$	${\mathit{I}}_{\mathit{corr}}$	${\mathit{I}}_{\mathit{conf}}$	${\mathit{I}}_{\mathit{entr}}$	${\mathit{I}}_{\mathit{mentr}}$	${\mathit{I}}_{\mathit{ilra}}$
CIFAR100	AlexNet	DP-SGD	28.1	20.2 ± 0.2	+65.3%	52.2 ± 0.1	$-\mathbf{1}$%	53.0	50.1	50.1	54.7	53.2
		MemGuard	99.3	32.7 ± 0.2	+21%	57.1 ± 0.1	+9%	83.3	50.0	50.0	50.1	52.1
		AdvReg	76.4	32.5 ± 0.8	+2.8%	57.2 ± 0.2	+9%	71.9	50.1	50.1	54.3	59.5
		Relax Loss	37.2	31.8 ± 0.3	+5.0%	52.0 ± 0.1	+2%	54.8	50.4	50.3	52.1	52.6
		DMP	50.4	30.4 ± 0.2	+9.9%	54.4 ± 0.1	+4%	60.0	54.2	53.2	52.4	52.4
		DSAK (ours)	46.5	33.4 ± 0.2		52.1 ± 0.1		56.6	50.0	50.1	50.8	53.1
	DenseNet	MemGuard	98.9	53.7 ± 0.1	+0.7%	55.7 ± 0.1	+0%	72.6	50.0	50.0	51.1	54.8
		AdvReg	71.7	40.8 ± 0.6	+32.6%	55.3 ± 0.3	+1%	65.4	51.0	51.8	52.0	56.2
		Relax Loss	83.5	53.9 ± 0.5	+0.4%	58.0 ± 0.3	+4%	64.8	56.5	54.7	59.7	54.4
		DMP	59.5	53.1 ± 0.2	+1.9%	59.2 ± 0.1	+6%	65.1	51.2	59.6	59.8	60.2
		DSAK (ours)	69.7	54.1 ± 0.2		55.7 ± 0.1		57.8	54.4	54.2	56.7	55.2
Texas100	NN Net	MemGuard	82.2	53.2 ± 0.1	+1.3%	53.4 ± 0.1	+1%	64.5	50.0	50.0	50.6	52.1
		AdvReg	76.0	52.8 ± 0.5	+2.1%	57.7 ± 0.6	+9%	64.6	54.4	53.8	58.3	57.2
		Relax Loss	62.4	52.4 ± 0.5	+2.9%	53.3 ± 0.4	+1%	55.3	51.7	52.0	52.3	55.3
		DMP	71.1	51.9 ± 0.2	+3.9%	57.0 ± 0.2	+8%	59.6	55.0	55.1	57.9	57.6
		DSAK (ours)	63.2	53.9 ± 0.2		52.7 ± 0.1		54.6	52.0	52.1	52.5	52.1
Purchase100	NN Net	MemGuard	97.7	76.5 ± 0.1	+0.4%	52.9 ± 0.2	$-\mathbf{2}$%	60.6	50.0	50.0	51.3	52.7
		AdvReg	95.3	70.3 ± 0.5	+9.2%	56.6 ± 0.5	+5%	60.9	55.1	55.2	55.2	56.8
		Relax Loss	90.8	73.1 ± 0.4	+5.1%	53.6 ± 0.2	$-\mathbf{1}$%	57.6	51.5	50.4	52.7	55.9
		DMP	87.3	72.4 ± 0.3	+6.1%	56.2 ± 0.1	+4%	57.5	55.0	54.5	57.7	56.2
		DSAK (ours)	90.4	76.8 ± 0.3		53.9 ± 0.2		56.8	52.7	51.2	53.7	55.3

6.3. Comparative Analysis of the Suppressing Overfitting Methods

Table 5 compares DSAK with defenses based on suppressing overfitting, namely, AdvReg and Relax Loss. Both AdvReg and Relax Loss effectively reduce the risk of MIAs by mitigating overfitting, but they also result in noticeable declines in classification task accuracy compared to the original model. AdvReg defense can reduce performance by 12.9% (DenseNet in CIFAR100), while Relax Loss defense reduces performance by 3.4% (NN Net in Purchase100). AdvReg defense may rely too much on the attack model, leading to a compromised balance between defense effectiveness and classification accuracy. Relax Loss hinders the model from reaching optimal classification accuracy due to its relaxed learning mechanism.

In comparison, DSAK offers a potent defense against MIA and upholds the target model’s classification capabilities, as demonstrated in the ${Test}^{+}$ column within Table 5. Under extreme conditions, DSAK’s classification accuracy outstrips AdvReg by 32.6% and exceeds Relax Loss by 5.1%. Additionally, DSAK shows a commendable resistance to overfitting, evidenced by the gains in training and validation accuracy, demonstrated in Figure 4a,b.

Figure 4. Accuracy variation curve of AlexNet in the CIFAR100 classification task. (a) Precision enhancement trend of target model without DSAK defense. (b) Precision enhancement trend of target model within DSAK defense. (c) Precision enhancement process of teacher model in DSAK. (d) Generalization error trend of teacher and target model with or w/o DSAK.

6.4. Comparative Analysis of the Knowledge Distillation Methods

Table 5 compares DSAK with the knowledge distillation defense method, DMP, which uses no additional data. DMP utilizes teacher models to transmit membership data features, preventing the target model from memorizing unnecessary details and reducing adverse effects on classification accuracy. DMP adeptly maintains data privacy, decreasing the accuracy of MIA to below 60%, while the decline in classification accuracy relative to the unprotected model remains within 5%. DSAK builds upon this by incorporating parameter transfer mechanisms and self-adaptive noise, providing additional benefits. Figure 4c demonstrates that using parameter passing to update the teacher model mitigates teacher model overfitting, enabling more effective training guidance and controlled overfitting during target model training, thus reducing the risk of privacy leakage. Figure 4d shows the training correlation between the teacher and target classifier models through the generalization error trend. Due to these advantages, DSAK consistently outperforms DMP in mitigating MIAs (by at least 4.0% on average compared to DMP) while preserving classification task accuracy more effectively.

7. Conclusions

Our work introduces the Distillation of Self-Adaptive Knowledge (DSAK), a novel training framework that integrates knowledge distillation and data noise generation techniques to mitigate membership inference attacks. DSAK achieves excellent training and privacy preservation efforts by introducing generative models to create adaptive noise in knowledge distillation and parameter transfer from the student to the teacher model. Empirical findings from our evaluation confirm that DSAK is proficient in countering membership inference attacks, exceeding the performance of existing defenses, and securing an advantageous harmony between the utility of the model and the privacy of membership information.