Differentially Private Generative Modeling via Discrete Latent Projection

Abstract

Deep generative models trained on sensitive data pose significant privacy risks, yet enforcing differential privacy (DP) in high-dimensional generators often leads to severe utility degradation. We propose Differentially Private Vector-Quantized Generation (DP-VQG), a three-stage generative framework that introduces a discrete latent bottleneck as the interface for privacy preservation. DP-VQG separates geometric structure learning, differentially private discrete latent projection, and non-private prior modeling, ensuring that privacy-induced randomness operates on a finite codebook aligned with the decoder’s effective support. This design avoids off-support degradation while providing formal end-to-end DP guarantees through composition and post-processing. We provide a theoretical analysis of privacy and utility, including explicit bounds on privacy-induced distortion. Empirically, under the privacy budget of $\epsilon =10$, DP-VQG attains Fréchet Inception Distance (FID) scores of 18.21 on MNIST and 77.09 on Fashion-MNIST, surpassing state-of-the-art differentially private generative models of comparable scale. Moreover, DP-VQG produces visually coherent samples on high-resolution datasets such as Flowers102, Food101, CelebA-HQ, and Cars, demonstrating scalability beyond prior end-to-end DP generative approaches.

1. Introduction

Deep generative models are increasingly deployed for data synthesis in privacy-sensitive domains such as medical imaging, biometrics, and personalized content generation. Their ability to model complex, high-dimensional data distributions, however, also raises serious privacy concerns. A growing body of work has shown that high-capacity generators may memorize training samples and enable membership inference or data extraction attacks, particularly in low-data or overparameterized regimes [1,2,3,4,5]. These risks highlight the need for principled privacy guarantees in generative modeling, beyond heuristic anonymization or ad hoc sanitization.

To illustrate the practical urgency of this issue, consider a real-life scenario in cross-institutional medical research. Hospitals often seek to share patient X-ray or MRI datasets to train robust diagnostic models. However, strict regulations (e.g., GDPR, HIPAA) prohibit sharing raw scans, and traditional de-identification methods (such as removing metadata or blurring faces) fail to obscure biometric markers embedded in bone structures or organ textures. In such cases, a privacy-preserving generative model can serve as a crucial enabler by synthesizing high-fidelity, anonymous cohorts that retain the statistical properties of pathologies without exposing the biometric identity of any specific patient. This necessitates a solution that offers rigorous, mathematically quantifiable privacy guarantees rather than relying on empirical obfuscation.

Differential Privacy (DP) has established itself as the de facto standard for privacy-preserving machine learning [6]. In contrast to heuristic techniques such as data anonymization or masking, which are prone to reconstruction attacks in high-dimensional feature spaces, DP offers a rigorous, information-theoretic guarantee against re-identification that remains robust to auxiliary knowledge and linkage attacks. Furthermore, DP satisfies the post-processing property [7]: any computation performed on the output of a DP mechanism incurs no additional privacy loss. This property is particularly instrumental for modular generative architectures, as it allows for the optimization of downstream components on sanitized representations without consuming further privacy budget.

In deep learning, the dominant approach to enforcing DP is Differentially Private Stochastic Gradient Descent (DP-SGD), which clips per-sample gradients and injects calibrated noise during optimization [8]. While DP-SGD has been successfully applied to discriminative models, its application to modern generative architectures remains highly challenging. High-dimensional generators typically require long training horizons and fine-grained signal reconstruction, both of which are acutely sensitive to gradient noise. As a result, end-to-end DP training often leads to severe degradation in sample quality and diversity in practice [9,10].

Existing approaches generally treat private generation as a monolithic task, applying noise uniformly across all model parameters. We argue that this strategy suffers from a fundamental representation—privacy mismatch. Modern latent generative models suggest that synthesis can be decoupled into two components: a universal “machinery” that reconstructs signal fidelity (e.g., textures, edges), and a specific “blueprint” that determines semantic content. Consider a biological analogy: the cellular machinery required to express proteins (e.g., ribosomes) is universal across a species, whereas the genetic code (DNA) is unique to the individual. In privacy-sensitive contexts, the “machinery” of generation, such as knowing how to render an eye or an organ structure, is often generic knowledge learnable from public proxies. The true privacy sensitivity lies in the “DNA”, which represents the specific combination of features that constitutes a private record. Current end-to-end methods effectively waste valuable privacy budget “encrypting the ribosome” alongside the DNA, rather than focusing protection solely where it is needed.

Complementing this semantic perspective, we adopt a geometric view grounded in the Manifold Hypothesis of representation learning [11]. This theory posits that high-dimensional data concentrate near a low-dimensional non-linear manifold embedded in the ambient space. Consequently, decoders are well-conditioned only in neighborhoods supported by this training manifold. When privacy mechanisms interact with continuous interfaces—for example, through noisy optimization trajectories, stochastic latent sampling, or direct perturbation of continuous representations—privacy-induced randomness can easily push the generative process away from these well-supported regions. In high-dimensional settings, even modest perturbations may force the decoder to extrapolate beyond its effective domain, resulting in blurry, unstable, or semantically incoherent samples [12]. We refer to this phenomenon as an off-support (or informally, off-manifold) failure mode. This contrast between continuous and discrete perturbation mechanisms is illustrated in Figure 1.

Figure 1. Comparison of continuous and discrete latent perturbations under differential privacy.

In this work, we propose Differentially Private Vector-Quantized Generation (DP-VQG), a framework that leverages a discrete privacy bottleneck to resolve these challenges. By shifting privacy mechanisms from continuous parameters to a discrete code selection process, we decouple high-fidelity reconstruction from privacy-preserving semantic generation. This architecture supports two strategies: FP-VQG (Fully Private), which utilizes quantization to prevent off-manifold geometric drift even without public data, and PP-VQG (Public-Prior), which employs auxiliary data to learn the generative “machinery” cost-free, focusing the privacy budget entirely on the semantic “DNA”. Crucially, since the final high-capacity generator operates purely on sanitized discrete codes, it functions as a post-processing step that incurs no additional privacy cost while preserving sample quality.

The contributions of this paper are summarized as follows:

• We propose DP-VQG, a novel framework that integrates vector quantization with differential privacy. It introduces a discrete bottleneck that inherently resists off-manifold degradation and enables the efficient decoupling of universal features from private semantics.
• We provide a theoretical analysis of the privacy–utility trade-off, formalizing the error decomposition of discrete bottlenecks and proving that our approach avoids the high-dimensional distortion scaling typical of continuous mechanisms.
• Extensive experiments on standard image benchmarks demonstrate that DP-VQG significantly outperforms state-of-the-art end-to-end DP generative models in both sample fidelity and downstream utility.

The remainder of the paper is organized as follows. Section 2 reviews related work on differentially private generative modeling. Section 3 introduces background on differential privacy, vector quantization, and discrete mechanisms. Section 4 presents the DP-VQG framework and its theoretical analysis. Section 5 reports experimental results, and Section 6 concludes the paper.

2. Related Work

2.1. Generative Modeling Paradigms

Generative modeling has undergone several major paradigm shifts over the past decade. Early progress was dominated by Generative Adversarial Networks (GANs), which formulate data generation as a minimax game between a generator and a discriminator [13]. Despite their strong empirical performance, GAN-based models are known to suffer from training instability, mode collapse, and sensitivity to architectural and optimization choices, particularly in high-dimensional image domains.

To address these limitations, likelihood-based models such as autoregressive models and normalizing flows were proposed, offering exact or tractable likelihoods at the cost of restrictive architectural assumptions or sequential sampling procedures [14,15,16]. More recently, diffusion-based and score-based generative models have emerged as a dominant paradigm, replacing adversarial objectives with iterative denoising or score matching processes and achieving state-of-the-art sample quality across a wide range of modalities [17,18].

Alongside these model-centric advances, a parallel line of research has emphasized structured or two-stage generative architectures. Rather than modeling the data distribution directly in pixel space, these approaches first learn a compact representation via an autoencoder, tokenizer, or vector quantization module, and subsequently fit a generative prior over the latent space. Representative examples include variational autoencoders with expressive priors [19], vector-quantized generative models [20], latent diffusion models [21], and masked token-based generative models [22,23,24]. This decomposition separates representation learning from density estimation, leading to improved computational efficiency, modularity, and scalability.

The present work follows this multiple-stage generative modeling tradition by adopting a discrete latent interface learned through vector quantization. While prior work primarily motivates such architectures from a modeling or computational perspective, our focus is on how this structural decomposition can be exploited to support privacy-preserving generation under differential privacy constraints.

2.2. Differentially Private Generative Models

Research on Differentially Private Generative Models (DPGMs) have progressed from classical statistical synthesis toward modern neural generative frameworks.

Early privacy-preservation approaches often relied on heuristic techniques such as data anonymization (e.g., k-anonymity, l-diversity), data anatomization, or graph-based de-identification [25,26]. While these methods have proven effective for low-dimensional tabular or structured data, they are ill-suited for high-dimensional unstructured data such as images. For instance, graph-based notation is difficult to define for pixel grids without losing semantic integrity, and simple anonymization is vulnerable to linkage and reconstruction attacks when dealing with high-dimensional feature spaces. Consequently, Differential Privacy (DP) has emerged as the de facto standard for modern generative tasks. Unlike heuristic methods, DP offers a formal, information–theoretic bound on privacy leakage that is robust to post-processing and independent of an attacker’s background knowledge [6], making it the optimal choice for training deep generative models on sensitive distributions.

Building on DP principles, early work on private synthesis focused on estimating low-order statistics or structured distributions, such as marginals and Bayesian networks [27,28]. These methods offer strong theoretical guarantees and interpretability, but their expressiveness is limited when applied to complex perceptual data.

More recent studies have explored the use of expressive neural generators for private data synthesis. This line of work has been systematically reviewed in recent surveys, which organize existing methods by model family, privacy mechanism, and application domain [29]. From a broader perspective, these developments align with emerging theoretical views that regard randomized learning algorithms as a viable means of reconciling statistical performance with normative requirements such as privacy [30].

Within neural generative modeling, existing approaches can be broadly categorized by how differential privacy is enforced. One prominent class builds on the PATE framework [31], in which an ensemble of teacher models trained on disjoint data partitions provides privatized supervision to a student generator. This paradigm has been instantiated in several generative settings, including both tabular and image synthesis [32,33]. While effective in certain regimes, PATE-based methods often require large ensembles or auxiliary unlabeled data, which may limit scalability and practical deployment.

Another major category consists of generative models trained directly with differentially private stochastic gradient descent [8]. DP-SGD has been applied to a variety of architectures, including GANs and diffusion-based models [10,34,35]. Although this approach provides a general and principled privacy mechanism, its application to high-capacity generators frequently leads to a substantial degradation in sample quality under strict privacy budgets, due to the accumulation of noise across high-dimensional parameter spaces.

Beyond these dominant paradigms, a number of works explore alternative formulations of private generative modeling, including gradient sanitization strategies, functional mechanisms, and other hybrid approaches [36,37,38,39]. These methods illustrate the diversity of design choices in differentially private generation, but also highlight the lack of a unified structural principle for mitigating privacy-induced utility loss in complex generative models.

Recent work combines task aligned public pretraining with DP fine tuning to achieve strong generation quality [40]. This line of approaches typically assumes close alignment between public and private data distributions and is often tailored to image synthesis with caption conditioned pipelines, which introduces additional supervision requirements and a more complex multi stage procedure.

Our work builds on this line of research while adopting a distinct perspective. Instead of enforcing privacy solely through end-to-end optimization or ensemble-based supervision, we introduce a discrete latent bottleneck that separates privacy preservation from generative modeling. This representation-level decoupling complements existing DPGM approaches and provides an alternative pathway for improving the privacy–utility trade-off in high-dimensional data generation.

3. Preliminaries

In this section we introduce the mathematical background necessary for the analysis of our privacy-preserving generative framework. We review the formulation of differential privacy, the DP-SGD training mechanism, vector-quantized latent representations, and the exponential mechanism.

3.1. Differential Privacy

Let $\mathcal{X}$ be the data domain, and let $D,D^{\prime }\in {\mathcal{X}}^n$ be two adjacent datasets differing in exactly one entry. A randomized mechanism $\mathcal{M}:{\mathcal{X}}^n\to \mathcal{Y}$ satisfies $(\epsilon ,\delta )$-differential privacy if, for all measurable sets $S\subseteq \mathcal{Y}$,

$$\mathrm{Pr}[\mathcal{M}\left(D\right)\in S]\le e^{\epsilon }\mathrm{Pr}[\mathcal{M}\left(D^{\prime }\right)\in S]+\delta .$$

(1)

The parameters $\epsilon$ and $\delta$ quantify the privacy guarantee: small $\epsilon$ and $\delta$ enforce indistinguishability of outputs between adjacent datasets.

A fundamental property of differential privacy is post-processing invariance: if $\mathcal{M}$ is $(\epsilon ,\delta )$-DP, then for any deterministic mapping h, the composed mechanism $h\circ \mathcal{M}$ is also $(\epsilon ,\delta )$-DP. Thus, once privacy is ensured by a mechanism, subsequent transformations of its outputs incur no additional privacy cost.

Another key property is sequential composition: if mechanisms ${\mathcal{M}}_1$ and ${\mathcal{M}}_2$ are applied in sequence, with ${\mathcal{M}}_1$ being $({\epsilon }_1,{\delta }_1)$-DP and ${\mathcal{M}}_2$ being $({\epsilon }_2,{\delta }_2)$-DP, then the composed mechanism $({\mathcal{M}}_1,{\mathcal{M}}_2)$ is $({\epsilon }_1+{\epsilon }_2,{\delta }_1+{\delta }_2)$-DP. This result underlies our analysis of training-time DP and sampling-time DP mechanisms.

3.2. DP-SGD Training Mechanism

Let $\theta \in \Theta$ denote the parameters of a generative model, and let $\mathcal{l}(\theta ;x)$ be a per-sample loss function. DP-SGD optimizes $\theta$ via iterative updates while ensuring differential privacy with respect to the dataset D. At iteration t, given a mini-batch $B_t\subseteq D$, the per-sample gradients $g_i^{\left(t\right)}={\nabla }_{\theta }\mathcal{l}({\theta }^{\left(t\right)};x_i)$ are clipped to a maximum norm $C>0$, averaged, and perturbed with Gaussian noise:

$${\tilde{g}}^{\left(t\right)}={\displaystyle \frac{1}{|B_t|}}\sum _{i\in B_t}\mathrm{clip}\left(g_i^{\left(t\right)},C\right)+\mathcal{N}(0,{\sigma }^2{C}^{2}I).$$

(2)

The parameters are updated as

$${\theta }^{(t+1)}={\theta }^{\left(t\right)}-{\eta }_t{\tilde{g}}^{\left(t\right)}.$$

Using Rényi differential privacy (RDP) [41] or Privacy Random Variables (PRVs) [42] accounting, one can track the cumulative privacy loss across iterations and convert it into a final $({\epsilon }_{\mathrm{train}},\delta )$-DP guarantee for the mapping $D\mapsto \theta \left(D\right)$.

3.3. Generative Models and Latent Spaces

We consider generative models represented as measurable mappings

$$G_{\theta }:\mathcal{Z}\to \mathcal{X},$$

where $\mathcal{Z}\subseteq {\mathbb{R}}^d$ is a latent space. The latent variable z may be sampled from a prior distribution $P_Z$ or obtained from a deterministic encoder applied to some conditioning input. The specific architectural form of $G_{\theta }$ is not important for our analysis; in particular, $G_{\theta }$ may represent a feed-forward decoder, a normalizing flow, or the denoising trajectory of a diffusion sampler initialized from a latent vector.

The key structural assumption required for our privacy mechanism is that $G_{\theta }$ can operate on a finite set of latent codes, introduced below.

3.4. Vector-Quantized Latent Representations

Let $\mathcal{C}=\{c_1,\dots ,c_K\}\subset {\mathbb{R}}^d$ be a finite codebook. Given a latent embedding $z\in {\mathbb{R}}^d$, we define its quantization via the nearest-code assignment

$$q\left(z\right)\in arg\underset{k\in \{1,\dots ,K\}}{min}{\parallel z-c_k\parallel }_2.$$

(3)

The generator is trained to accept codes from $\mathcal{C}$ as inputs, so that for any k, the image $G_{\theta }\left(c_k\right)$ lies on a region of the data manifold consistent with training. This structure enables latent perturbation to be performed in a discrete space, which is essential for applying the exponential mechanism.

Vector-quantized representations provide a robust interface between continuous latent embeddings and finite code sets. Throughout this work, we assume the codebook $\mathcal{C}$ has bounded diameter and is fixed during sampling.

3.5. The Exponential Mechanism

The exponential mechanism [43] is a fundamental tool for achieving differential privacy over discrete output spaces. Given a score function $u:\mathcal{Z}\times \mathcal{C}\to \mathbb{R}$ and privacy parameter $\epsilon >0$, the mechanism selects an element $c_k\in \mathcal{C}$ with probability

$$\mathrm{Pr}[{\mathcal{M}}_{\exp }\left(z\right)=c_k]={\displaystyle \frac{exp\left(\frac{\epsilon }{2\Delta u}u(z,c_k)\right)}{{\sum }_{j=1}^{K}exp\left(\frac{\epsilon }{2\Delta u}u(z,c_j)\right)}},$$

(4)

where $\Delta u$ is the sensitivity of the score function:

$$\Delta u=\underset{z,z^{\prime }\in \mathcal{Z}}{sup}\underset{k\in \{1,\dots ,K\}}{sup}|u(z,c_k)-u(z^{\prime },c_k)|.$$

The mechanism guarantees $\epsilon$-DP under changes in its input z.

4. Methodology: The DP-VQG Framework

We propose Differentially Private Vector-Quantized Generation (DP-VQG), a three-stage framework that synthesizes high-fidelity samples under formal differential privacy (DP) guarantees. The core design principle is an operator-level decomposition of generation into (i) geometric structure learning and (ii) density modeling on a sanitized discrete bottleneck.

4.1. Formulation of Private Discrete Bottlenecks

4.1.1. PrivateData and Adjacency

Let $\mathcal{X}$ denote the data domain and $D_{\mathrm{priv}}={\left\{x_i\right\}}_{i=1}^N\in {\mathcal{X}}^N$ a private dataset. We adopt the standard record-level adjacency: two datasets $D,D^{\prime }\in {\mathcal{X}}^N$ are adjacent (denoted $D\sim D^{\prime }$) if they differ in exactly one record.

4.1.2. Mechanisms as (Random) Operators

Rather than learning $p_{\theta }\left(x\right)$ via an end-to-end DP training of a high-dimensional generator, we model the overall release as a composition of three operators:

$$\mathcal{G}\left(D_{\mathrm{priv}}\right)={\mathcal{M}}_{\mathrm{prior}}\left({\mathcal{M}}_{\mathrm{encode}}(D_{\mathrm{priv}};\Theta ),\Theta \right),\Theta \sim {\mathcal{M}}_{\mathrm{structure}}\left(\mathcal{D}\right),$$

(5)

where $\mathcal{D}$ denotes the data source used for structure learning. Concretely:

1.

Structure learning ${\mathcal{M}}_{\mathrm{structure}}$: a (possibly randomized) training operator outputting parameters $\Theta$ that define an encoder–decoder manifold and a codebook. We write $\Theta =(E,G,\mathcal{C})$, where $E:\mathcal{X}\to {\mathbb{R}}^d$ is an encoder, $G:{\mathbb{R}}^d\to \mathcal{X}$ a decoder, and $\mathcal{C}=\{c_1,\dots ,c_K\}\subset {\mathbb{R}}^d$ a finite codebook.

2.

Latent encoding ${\mathcal{M}}_{\mathrm{encode}}$: a randomized discrete projection that maps each private record $x_i$ to a code index (or code vector) in $\mathcal{C}$, producing a sanitized discrete latent dataset $\tilde{Z}={\left\{{\tilde{z}}_i\right\}}_{i=1}^N$.

3.

Prior modeling ${\mathcal{M}}_{\mathrm{prior}}$: a (non-private) learning operator that fits a generative model over $\tilde{Z}$ and induces a sampler for discrete latents, followed by decoding through G.

This decomposition isolates privacy consumption to the releases $(\Theta ,\tilde{Z})$; any subsequent training or sampling based solely on $(\Theta ,\tilde{Z})$ is DP-preserving by post-processing.

4.1.3. Privacy Composition for Discrete Bottlenecks

We state a sufficient end-to-end condition via sequential composition and post-processing.

Lemma 1

(Privacy via Discrete Bottleneck Composition). Assume ${\mathcal{M}}_{\mathrm{structure}}$ is $({\epsilon }_1,{\delta }_1)$-DP w.r.t. $D_{\mathrm{priv}}$ when $\mathcal{D}=D_{\mathrm{priv}}$. Conditioned on Θ, assume ${\mathcal{M}}_{\mathrm{encode}}(·;\Theta )$ is $({\epsilon }_2,{\delta }_2)$-DP w.r.t. $D_{\mathrm{priv}}$ and releases $\tilde{Z}$. Then, any (possibly randomized) operator ${\mathcal{M}}_{\mathrm{prior}}$ trained solely on $(\Theta ,\tilde{Z})$ yields an overall release that is $({\epsilon }_1+{\epsilon }_2,{\delta }_1+{\delta }_2)$-DP w.r.t. $D_{\mathrm{priv}}$.

Proof. 

By sequential composition, releasing $\Theta$ and then $\tilde{Z}$ yields $({\epsilon }_1+{\epsilon }_2,{\delta }_1+{\delta }_2)$-DP. Since ${\mathcal{M}}_{\mathrm{prior}}$ accesses only $(\Theta ,\tilde{Z})$, the post-processing property implies no additional privacy loss. □

4.1.4. Geometric Rationale: Anchoring on a Discrete Support

A key choice is the latent domain. Continuous DP mechanisms (e.g., Gaussian perturbation in ${\mathbb{R}}^d$) often produce off-support latent inputs, where the decoder may be poorly conditioned. DP-VQG instead constrains the DP mechanism to output a valid code in the finite set $\mathcal{C}$.

Assumption 1

(Anchored Decoder Regularity). There exists $L_G>0$ such that the decoder is Lipschitz on the codebook support:

$$\parallel G\left(c\right)-G\left(c^{\prime }\right)\parallel \le L_G\parallel c-c^{\prime }\parallel ,\forall c,c^{\prime }\in \mathcal{C}.$$

Under Assumption 1, any privacy-induced randomness that changes the selected code remains on-support and admits a controlled error propagation from latent distortion to reconstruction distortion.

4.2. Three-Stage Stochastic Instantiation

We instantiate Equation (5) into three training stages as shown in Figure 2.

Figure 2. The proposed DP-VQG framework. Privacy is consumed only when accessing $D_{\mathrm{priv}}$: optionally in Stage I (FP-VQG) and necessarily in Stage II. Stage III operates solely on the sanitized latent dataset $\tilde{Z}$ and is thus purely post-processing.

4.2.1. Stage I: Manifold Learning (Implementation of ${\mathcal{M}}_{\mathrm{structure}}$)

Stage I learns a discrete generative manifold $\Theta =(E,G,\mathcal{C})$ through a VQ-VAE tokenizer, where the codebook $\mathcal{C}={\left\{c_k\right\}}_{k=1}^K\subset {\mathbb{R}}^d$ induces a finite latent support on which all subsequent operations are constrained. A critical practical requirement for DP-VQG is that the decoder G remain stable under stochastic perturbations of code indices: downstream stages may not always recover the deterministic nearest-code assignment, and rare/inactive codes can lead to catastrophic decoding artifacts if they were never sufficiently exercised during training. Standard VQ-VAE training, however, is typically fragile in this respect because (i) it relies on deterministic top-1 nearest-neighbor assignments, which do not expose G to code mismatch, and (ii) it often yields highly imbalanced code usage, leaving a portion of the codebook weakly trained.

To explicitly improve robustness to discrete latent perturbations, we adopt a stochastic quantization scheme and a code-usage regularizer during tokenizer training.

Stochastic Top-k Code Assignment

Given an encoder feature map $z=E\left(x\right)\in {\mathbb{R}}^{d\times H\times W}$, we flatten spatial locations into vectors ${\left\{z_{\mathcal{l}}\right\}}_{\mathcal{l}=1}^{HW}$ with $z_{\mathcal{l}}\in {\mathbb{R}}^d$. For each location, define squared distances to code vectors

$$d_{\mathcal{l}}\left(k\right)={\parallel z_{\mathcal{l}}-c_k\parallel }_2^2,k\in \left[K\right].$$

(6)

During training, instead of using the deterministic nearest neighbor, we sample a code index from a temperature-controlled distribution. Let

$${\pi }_{\tau }(k\mid z_{\mathcal{l}})={\displaystyle \frac{exp\left(-d_{\mathcal{l}}\left(k\right)/\tau \right)}{{\sum }_{j=1}^{K}exp\left(-d_{\mathcal{l}}\left(j\right)/\tau \right)}},$$

(7)

where $\tau >0$ is a temperature. To reduce variance while preserving stochasticity, we optionally restrict sampling to the top-k nearest candidates: let ${\mathcal{N}}_k\left(z_{\mathcal{l}}\right)\subset \left[K\right]$ be the set of k smallest-distance codes under $d_{\mathcal{l}}\left(·\right)$, and sample

$${\widehat{k}}_{\mathcal{l}}\sim \mathrm{Cat}\left({\left\{{\pi }_{\tau }(j\mid z_{\mathcal{l}})\right\}}_{j\in {\mathcal{N}}_k\left(z_{\mathcal{l}}\right)}\right),z_q\left(z_{\mathcal{l}}\right)=c_{{\widehat{k}}_{\mathcal{l}}}.$$

(8)

At evaluation time, we revert to deterministic quantization ${\widehat{k}}_{\mathcal{l}}=arg{min}_k{d}_{\mathcal{l}}\left(k\right)$. This stochastic assignment can be viewed as discrete data augmentation: it trains the decoder to tolerate moderate code mismatch by exposing it to plausible alternative codes during training.

Entropy Regularization for Balanced Code Usage

To avoid degenerate solutions where only a small subset of codes is active, we encourage balanced utilization of the codebook. Let ${\pi }_{\tau }(·\mid z_{\mathcal{l}})$ be the full soft assignment distribution in Equation (7), and define the batch-averaged usage distribution

$$\overline{p}\left(k\right)={\displaystyle \frac{1}{HWB}}\sum _{b=1}^B\sum _{\mathcal{l}=1}^{HW}{\pi }_{\tau }(k\mid z_{\mathcal{l}}^{\left(b\right)}),k\in \left[K\right].$$

(9)

We maximize the entropy $H\left(\overline{p}\right)=-{\sum }_{k=1}^K\overline{p}\left(k\right)log\left(\overline{p}\left(k\right)\right)$ (equivalently, minimize $-H\left(\overline{p}\right)$), which favors a more uniform activation of codes and mitigates the risk of “dead” or rarely-used codes.

Training Objective

Let $z=E\left(x\right)$ and $z_q$ be the (stochastically) quantized latent map produced by Equation (8). We optimize the tokenizer parameters $\Theta =(E,G,\mathcal{C})$ via

$$\underset{\Theta }{\min }{\mathbb{E}}_{x\sim \mathcal{D}}[\underset{\mathrm{reconstruction}}{\underbrace{\parallel x-G\left(z_q\right){\parallel }_2^2}}+\underset{\mathrm{VQ}+\mathrm{commitment}}{\underbrace{\parallel sg\left[z\right]-z_q{\parallel }_2^2+\beta {\parallel z-sg\left[z_q\right]\parallel }_2^2}}+{\lambda }_{\mathrm{ent}}\underset{\mathrm{code}-\mathrm{usage}\mathrm{regularization}}{\underbrace{\left(-H\left(\overline{p}\right)\right)}}],$$

(10)

where $\beta >0$ is the commitment weight, ${\lambda }_{\mathrm{ent}}\ge 0$ controls the entropy regularization strength, and $sg\left[·\right]$ is the stop-gradient operator. In practice, we implement Equation (8) using top-k sampling with a temperature $\tau$ and optimize Equation (10) with the straight-through estimator.

The above objective defines a generic procedure for learning a robust discrete manifold. Depending on whether the data used for structure learning contains private records, we consider two instantiations that differ only in the training protocol and privacy accounting of this stage.

Variant A: Fully Private VQG (FP-VQG). If $\mathcal{D}=D_{\mathrm{priv}}$, we train Equation (10) using DP-SGD to ensure $({\epsilon }_1,{\delta }_1)$-DP for releasing $\Theta$. Since the Exponential Moving Average (EMA) codebook updates bypass per-sample gradient clipping/accounting, we optimize the codebook embeddings $\mathcal{C}$ via explicit gradients so that all updates are tracked by the privacy accountant.

Variant B: Public-Prior VQG (PP-VQG). If a public dataset $D_{\mathrm{pub}}$ is available with $D_{\mathrm{pub}}\cap D_{\mathrm{priv}}=\varnothing$, we train Equation (10) on $\mathcal{D}=D_{\mathrm{pub}}$ using standard SGD, incurring zero privacy cost w.r.t. $D_{\mathrm{priv}}$ (i.e., ${\epsilon }_1={\delta }_1=0$).

The two instantiations of Stage I are designed to address complementary data regimes. FP-VQG provides a fully self-contained solution with end-to-end differential privacy guarantees and is particularly suitable for small- to medium-scale datasets whose distributions differ substantially from available public data. In contrast, PP-VQG leverages auxiliary public datasets to learn generic visual structures and low-level statistics without privacy constraints, introducing differential privacy only when modeling the distributional characteristics specific to the private dataset. This separation is especially advantageous for high-dimensional and complex data, as it avoids the excessive noise and blurring effects that can arise when training the encoder–decoder components under DP-SGD in FP-VQG, while still ensuring rigorous privacy protection for sensitive distributional information.

4.2.2. Stage II: DP Discrete Projection via the Exponential Mechanism (Implementation of ${\mathcal{M}}_{encode}$)

We freeze $\Theta$ and encode $D_{\mathrm{priv}}$ through an ${\epsilon }_2$-DP discrete projection operator.

Bounded Representation

To ensure bounded utility sensitivity, we define the ${\mathcal{l}}_2$-ball projection ${\mathrm{\Pi }}_R\left(·\right)={\mathrm{\Pi }}_{B(0,R)}\left(·\right)$ and use

$$\overline{E}\left(x\right)={\mathrm{\Pi }}_R\left(E\left(x\right)\right),{\overline{c}}_k={\mathrm{\Pi }}_R\left(c_k\right).$$

Then, $\parallel \overline{E}{\left(x\right)\parallel }_2\le R$ and $\parallel {\overline{c}}_k{\parallel }_2\le R$ for all k.

Utility and Sensitivity

We set the utility

$$u(x,k)=-\parallel \overline{E}\left(x\right)-{\overline{c}}_k{\parallel }_2^2.$$

(11)

Since $\parallel \overline{E}\left(x\right)-{\overline{c}}_k{\parallel }_2\le 2R$, we have $u(x,k)\in [-4R^2,0]$ and hence, the global sensitivity satisfies

$$\Delta u=\underset{x,x^{\prime },k}{\max }|u(x,k)-u(x^{\prime },k)|\le 4R^2.$$

(12)

DP Discrete Projection Operator

Define ${\mathrm{\Pi }}_{{\epsilon }_2}:\mathcal{X}\to \left[K\right]$ by the exponential mechanism:

$$\mathrm{Pr}\left({\mathrm{\Pi }}_{{\epsilon }_2}\left(x\right)=k\right)={\displaystyle \frac{exp\left(\frac{{\epsilon }_2}{2\Delta u}u(x,k)\right)}{{\sum }_{j=1}^{K}exp\left(\frac{{\epsilon }_2}{2\Delta u}u(x,j)\right)}}.$$

(13)

Applying ${\mathrm{\Pi }}_{{\epsilon }_2}$ independently to each record yields a discrete latent dataset $\tilde{Z}={\left\{{\tilde{z}}_i\right\}}_{i=1}^N$ (indices or code vectors). We denote this record-wise randomized encoding as ${\mathcal{M}}_{\mathrm{encode}}(D_{\mathrm{priv}};\Theta )$.

It is worth noting that our privacy guarantee is defined at the record level: the encoding mechanism takes one private record x and release its entire discrete latent representation (a code index map in ${\left[K\right]}^{H\times W}$ or the corresponding code vectors) as a single DP output. In practice, this can be implemented either (i) as a joint exponential-mechanism release over the whole code map, or (ii) by distributing the total budget ${\epsilon }_2$ across spatial locations so that the overall release remains $({\epsilon }_2,0)$-DP for each record. Throughout the paper, the reported ${\epsilon }_2$ refers to this total record-level privacy budget.

Lemma 2

(DP of the Exponential-Mechanism Quantizer). With utility Equation (11) and sensitivity bound Equation (12), the operator ${\mathrm{\Pi }}_{{\epsilon }_2}$ defines an $({\epsilon }_2,0)$-DP record-level mechanism that maps a single private record x to its entire discrete latent representation. Consequently, releasing $\tilde{Z}$ satisfies $({\epsilon }_2,0)$-DP under record-level adjacency, where ${\epsilon }_2$ denotes the total privacy budget for the record-level encoding release.

Proof. 

This follows directly from the standard guarantee of the exponential mechanism with sensitivity $\Delta u$. We emphasize that ${\mathrm{\Pi }}_{{\epsilon }_2}$ is defined at the record level: it releases the entire discrete latent representation associated with one record as a single DP output. Practical implementations may realize this operator by composing multiple location-wise mechanisms, but such implementations are calibrated to satisfy the same overall $({\epsilon }_2,0)$ record-level guarantee.    □

Geometric Interpretation: Soft Voronoi Tessellation

Deterministic VQ partitions ${\mathbb{R}}^d$ into Voronoi cells; Equation (13) induces a Gibbs distribution over cell candidates, yielding a softening of boundaries that masks the precise location of $\overline{E}\left(x\right)$, as shown in Figure 3.

Figure 3. Comparison of deterministic vs. probabilistic latent quantization. The background colors visualize the decision regions.

4.2.3. Stage III: Masked Generative Modeling on Sanitized Latents (Implementation of ${\mathcal{M}}_{prior}$)

Stage III learns a generative prior over the sanitized discrete lattice and is purely post-processing.

Let each latent map be $z\in {\left[K\right]}^{H\times W}$, and let M denote a special mask token with index $K+1$. We discretize time into $t\in \{1,\dots ,T\}$ with a masking schedule ${\gamma }_t\in [0,1]$ (cosine in our implementation). Define the forward corruption kernel $q_t(z_t\mid z_0)$ as an i.i.d. masking process:

$$q_t(z_t\mid z_0)=\prod _{u,v}\left((1-{\gamma }_t)\mathbf{1}\{z_t^{(u,v)}=z_0^{(u,v)}\}+{\gamma }_t\mathbf{1}\{z_t^{(u,v)}=M\}\right).$$

(14)

Equivalently, for each site $(u,v)$, we sample $m_t^{(u,v)}\sim \mathrm{Bernoulli}(1-{\gamma }_t)$ and set

$$z_t^{(u,v)}=m_t^{(u,v)}{z}_0^{(u,v)}+(1-m_t^{(u,v)})M.$$

(15)

We parameterize $p_{\varphi }(z_0\mid z_t,t)$ using a bidirectional Transformer that outputs token probabilities at masked sites. The training objective minimizes the masked negative log-likelihood:

$$\underset{\varphi }{min}{\mathbb{E}}_{z_0\sim \tilde{Z}}{\mathbb{E}}_{t\sim \mathrm{Unif}\left(\right\{1,\dots ,T\left\}\right)}{\mathbb{E}}_{z_t\sim q_t(·\mid z_0)}\left[-\sum _{(u,v):z_t^{(u,v)}=M}log{p}_{\varphi }\left(z_0^{(u,v)}\mid z_t,t\right)\right].$$

(16)

Since $\tilde{Z}$ is DP, optimizing Equation (16) incurs no additional privacy cost.

Inference as an Iterative Reconstruction Map

We initialize $z^{\left(0\right)}$ as fully masked and iterate S steps. At step s, the model predicts token probabilities for all masked sites; a confidence score (e.g., maximum probability) determines which sites are fixed and which remain masked, following a decreasing remasking schedule aligned with $\left\{{\gamma }_t\right\}$.

Computational Remark

Each inference iteration evaluates the Transformer once on an $HW$-length sequence; hence, the sampling complexity scales as $O\left(S·{\mathrm{Cost}}_{\varphi }\left(HW\right)\right)$, where ${\mathrm{Cost}}_{\varphi }$ depends on the Transformer architecture (typically quadratic or linear-time variants in sequence length).

The overall training pipeline, which integrates these three stages to enforce end-to-end differential privacy, is formally outlined in Algorithm 1.

Algorithm 1 Differentially Private Vector-Quantized Generation (DP-VQG)
Require: Private dataset $D_{\mathrm{priv}}$, public dataset $D_{\mathrm{pub}}$ (optional), budgets $({\epsilon }_1,{\epsilon }_2)$, codebook size K, temperature $\tau$.
Ensure: Generative prior $\varphi$, Tokenizer $\Theta =(E,G,\mathcal{C})$.
1: Stage I: Robust Manifold Learning
2: Initialize tokenizer $\Theta$
3: if Variant is FP-VQG (No public data) then
4:  $\mathcal{D}\leftarrow D_{\mathrm{priv}}$, Optimizer ← DP-SGD$\left({\epsilon }_1\right)$	▹ Private training
5: else	▹ PP-VQG (Public Prior)
6:  $\mathcal{D}\leftarrow D_{\mathrm{pub}}$, Optimizer ← Standard SGD	▹ Zero privacy cost
7: end if
8: while not converged do
9:  Sample batch $x\sim \mathcal{D}$
10:  Compute latents $z=E\left(x\right)$ and distances $d_{\mathcal{l}}\left(k\right)={\parallel z_{\mathcal{l}}-c_k\parallel }_2^2$
11:  Sample ${\widehat{k}}_{\mathcal{l}}\sim \mathrm{Softmax}(-d_{\mathcal{l}}/\tau )$ restricted to Top-k	▹ Stochastic Sampling
12:  $\mathcal{L}\leftarrow {\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{\mathrm{VQ}}-{\lambda }_{\mathrm{ent}}H\left(\overline{p}\right)$	▹ Entropy Reg. (Equation (10))
13:  Update $\Theta$ using selected Optimizer
14: end while
15: Stage II: DP Discrete Projection
16: Freeze $\Theta$, initialize sanitized dataset $\tilde{Z}\leftarrow \varnothing$
17: for each record $x_i\in D_{\mathrm{priv}}$ do
18:  $\overline{E}\left(x_i\right)\leftarrow {\mathrm{\Pi }}_R\left(E\left(x_i\right)\right)$	▹ Bounded embedding
19:  Define utility $u(x_i,k)=-{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_k\parallel }_2^2$
20:  Sample ${\widehat{k}}_i\propto exp\left({\displaystyle \frac{{\epsilon }_{2}u(x_i,k)}{2\Delta u}}\right)$	▹ Exp. Mechanism (Equation (13))
21:  Add ${\tilde{z}}_i={\overline{c}}_{{\widehat{k}}_i}$ to $\tilde{Z}$
22: end for
23: Stage III: Prior Modeling (Post-Processing)
24: Initialize Masked Transformer $\varphi$
25: while not converged do
26:  Sample sanitized latents $z_0\sim \tilde{Z}$
27:  Generate masked input $z_t\leftarrow \mathrm{Mask}(z_0,{\gamma }_t)$
28:  Update $\varphi$ minimizing NLL$(z_0\mid z_t)$	▹ Equation (16)
29: end while
30: return Generator $(\varphi ,G)$

4.3. Theoretical Privacy Analysis and Error Bounds

4.3.1. Privacy Guarantees for Instantiation Variants

We restate end-to-end privacy guarantees as corollaries of Lemmas 1 and 2.

Proposition 1

(Privacy of FP-VQG). If Stage I is $({\epsilon }_1,{\delta }_1)$-DP (e.g., DP-SGD on $D_{\mathrm{priv}}$) and Stage II uses ${\mathrm{\Pi }}_{{\epsilon }_2}$, then the overall FP-VQG release is $({\epsilon }_1+{\epsilon }_2,{\delta }_1)$-DP.

Proof. 

Stage II is pure ${\epsilon }_2$-DP (Lemma 2), and Stage III is post-processing; hence, Lemma 1 yields $({\epsilon }_1+{\epsilon }_2,{\delta }_1)$-DP. □

Proposition 2

(Privacy of PP-VQG). If Stage I is trained on $D_{\mathrm{pub}}$ with $D_{\mathrm{pub}}\cap D_{\mathrm{priv}}=\varnothing$ (thus ${\epsilon }_1={\delta }_1=0$ w.r.t. $D_{\mathrm{priv}}$), and Stage II uses ${\mathrm{\Pi }}_{{\epsilon }_2}$, then PP-VQG is $({\epsilon }_2,0)$-DP w.r.t. $D_{\mathrm{priv}}$.

Proof. 

Stage I is independent of $D_{\mathrm{priv}}$. The only access to $D_{\mathrm{priv}}$ is Stage II, which is pure ${\epsilon }_2$-DP. Stage III is post-processing. □

4.3.2. Utility Analysis of the Discrete Barrier

A key advantage is the scaling of privacy-induced distortion. Continuous Gaussian perturbation typically yields ${\mathbb{E}\parallel \eta \parallel }^2\propto d{\sigma }^2$, while discrete selection yields only logarithmic dependence on K.

Theorem 1

(Expected Utility Loss of Discrete Selection). Let $k_i^{*}=\arg {\min }_{k\in \left[K\right]}{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_k\parallel }_2^2$ be the optimal code associated with record $x_i$, and let ${\widehat{k}}_i={\mathrm{\Pi }}_{{\epsilon }_2}\left(x_i\right)$ be selected by Equation (13) with $\Delta u\le 4R^2$. Then, the expected excess squared latent distortion satisfies

$$\mathbb{E}\left[\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{{\widehat{k}}_i}{\parallel }_2^2\right]-{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}\parallel }_2^2\le {\displaystyle \frac{8R^2}{{\epsilon }_2}}\left(lnK+1\right).$$

(17)

Proof. 

The proof relies on the fundamental utility analysis of the exponential mechanism. Let $\mathcal{R}=\{1,\dots ,K\}$ denote the set of possible outcomes (code indices). The mechanism ${\mathrm{\Pi }}_{{\epsilon }_2}\left(x\right)$ selects an outcome $k\in \mathcal{R}$ with probability proportional to $exp\left({\displaystyle \frac{{\epsilon }_{2}u(x,k)}{2\Delta u}}\right)$.

A standard result in differential privacy [6] states that for the exponential mechanism, the expected difference between the optimal utility and the utility of the selected element is bounded by:

$$\mathbb{E}\left[\underset{j\in \mathcal{R}}{\max }u(x,j)-u(x,{\mathrm{\Pi }}_{{\epsilon }_2}\left(x\right))\right]\le {\displaystyle \frac{2\Delta u}{{\epsilon }_2}}\left(ln\left|\mathcal{R}\right|+1\right).$$

Substituting our specific utility definition: The optimal utility is ${\max }_{j}u(x,j)={\max }_j\left(-\parallel \overline{E}\left(x\right)-{\overline{c}}_j{\parallel }_2^2\right)=-{min}_j\parallel \overline{E}\left(x\right)-{\overline{c}}_j{\parallel }_2^2=-{\parallel \overline{E}\left(x\right)-{\overline{c}}_{k^{*}}\parallel }_2^2$. The utility of the selected element is $u(x,\widehat{k})=-{\parallel \overline{E}\left(x\right)-{\overline{c}}_{\widehat{k}}\parallel }_2^2$.

Therefore, the expected utility loss corresponds exactly to the expected excess distortion:

$$\mathbb{E}\left[-\parallel \overline{E}\left(x\right)-{\overline{c}}_{k^{*}}{\parallel }_2^2-\left(-\parallel \overline{E}\left(x\right)-{\overline{c}}_{\widehat{k}}{\parallel }_2^2\right)\right]=\mathbb{E}\left[\parallel \overline{E}\left(x\right)-{\overline{c}}_{\widehat{k}}{\parallel }_2^2\right]-{\parallel \overline{E}\left(x\right)-{\overline{c}}_{k^{*}}\parallel }_2^2.$$

We derived the global sensitivity bound $\Delta u\le 4R^2$ in the previous section. Substituting $\Delta u=4R^2$ and $\left|\mathcal{R}\right|=K$ into the standard bound yields:

$$\mathbb{E}\left[\mathrm{Excess}\mathrm{Distortion}\right]\le {\displaystyle \frac{2\left(4R^2\right)}{{\epsilon }_2}}(lnK+1)={\displaystyle \frac{8R^2}{{\epsilon }_2}}(lnK+1).$$

This completes the proof. □

The bound in Theorem 1 characterizes the distortion introduced purely by the differentially private discrete projection, and in particular, isolates the effect of the codebook size K on the DP-induced selection noise. In practice, however, the choice of K also affects the intrinsic representational capacity and quantization accuracy of the tokenizer itself. To bridge this gap between the formal DP analysis and practical design, we consider the following representation-level distortion proxy.

Practically, the $lnK$ term implies an explicit design trade-off when choosing the codebook size. Increasing K improves the representational granularity and typically decreases the deterministic quantization error (i.e., the geometric approximation term induced by nearest-code assignment). Meanwhile, under a fixed record-level budget ${\epsilon }_2$, the DP discrete projection contributes an additional distortion upper-bounded by Theorem 1. This suggests using the following representation-level distortion proxy to interpret the dependence on K:

$${\mathcal{D}}_{\mathrm{proxy}}\left(K\right)=\underset{\mathrm{quantization}/\mathrm{geometric}\mathrm{term}}{\underbrace{\mathbb{E}\left[\parallel \overline{E}\left(X\right)-{\overline{c}}_{k^{*}\left(X\right)}{\parallel }_2^2\right]}}+\underset{\mathrm{DP}\mathrm{selection}\mathrm{term}(\mathrm{Theorem}1)}{\underbrace{{\displaystyle \frac{8R^2}{{\epsilon }_2}}\left(lnK+1\right)}}.$$

(18)

The first term typically decreases with K, whereas the second term increases with $lnK$, which predicts a non-monotonic dependence on K. We emphasize that ${\mathcal{D}}_{\mathrm{proxy}}\left(K\right)$ is introduced only as an explanatory proxy (not as a new bound on the end-to-end error) to highlight the competing effects of representation capacity and privacy-induced selection distortion.

Corollary 1

(Record-Level Reconstruction Error under Anchored Regularity). Under Assumption 1 (Anchored Decoder Regularity), the corresponding expected excess reconstruction distortion is bounded as:

$$\mathbb{E}\left[\parallel G\left({\overline{c}}_{{\widehat{k}}_i}\right)-G\left({\overline{c}}_{k_i^{*}}\right){\parallel }^2\right]\le L_G^2\left(4\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}{\parallel }_2^2+{\displaystyle \frac{16R^2}{{\epsilon }_2}}(lnK+1)\right).$$

Proof. 

By Assumption 1, the decoder G is $L_G$-Lipschitz continuous on the codebook support $\mathcal{C}$. This implies that for any two code vectors $c,c^{\prime }\in \mathcal{C}$, the distance in the data space is bounded by the distance in the latent space:

$$\parallel G\left(c\right)-G\left(c^{\prime }\right){\parallel }_2\le L_G{\parallel c-c^{\prime }\parallel }_2.$$

Squaring both sides preserves the inequality since the norms are non-negative:

$$\parallel G\left(c\right)-G\left(c^{\prime }\right){\parallel }_2^2\le L_G^2{\parallel c-c^{\prime }\parallel }_2^2.$$

Let $c={\overline{c}}_{{\widehat{k}}_i}$ (the selected code) and $c^{\prime }={\overline{c}}_{k_i^{*}}$ (the optimal code). Taking the expectation over the randomness of the privacy mechanism ${\widehat{k}}_i\sim {\mathrm{\Pi }}_{{\epsilon }_2}\left(x_i\right)$:

$$\mathbb{E}\left[\parallel G\left({\overline{c}}_{{\widehat{k}}_i}\right)-G\left({\overline{c}}_{k_i^{*}}\right){\parallel }_2^2\right]\le L_G^2\mathbb{E}\left[\parallel {\overline{c}}_{{\widehat{k}}_i}-{\overline{c}}_{k_i^{*}}{\parallel }_2^2\right].$$

To rigorously bound the latent distance $\mathbb{E}\parallel {\overline{c}}_{{\widehat{k}}_i}-{\overline{c}}_{k_i^{*}}{\parallel }_2^2$, we employ the triangle inequality relative to the projected embedding $\overline{E}\left(x_i\right)$. Using ${\parallel a-b\parallel }^2\le {2\parallel a-c\parallel }^2+2{\parallel b-c\parallel }^2$:

$$\parallel {\overline{c}}_{{\widehat{k}}_i}-{\overline{c}}_{k_i^{*}}{\parallel }_2^2\le 2\parallel {\overline{c}}_{{\widehat{k}}_i}-\overline{E}\left(x_i\right){\parallel }_2^2+2{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}\parallel }_2^2.$$

Taking expectations on both sides and substituting the bound from Theorem 1 (which states $\mathbb{E}\parallel {\overline{c}}_{{\widehat{k}}_i}-\overline{E}\left(x_i\right){\parallel }_2^2\le {\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}\parallel }_2^2+\mathrm{Bound}$):

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\overline{c}}_{{\widehat{k}}_i}-{\overline{c}}_{k_i^{*}}{\parallel }_2^2& \le 2\left(\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}{\parallel }_2^2+{\displaystyle \frac{8R^2}{{\epsilon }_2}}(lnK+1)\right)+2{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}\parallel }_2^2\hfill \\ \hfill & =4\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}{\parallel }_2^2+{\displaystyle \frac{16R^2}{{\epsilon }_2}}(lnK+1).\hfill \end{array}$$

Here, the first term $4\parallel \overline{E}\left(x_i\right)-{\overline{c}}_{k_i^{*}}{\parallel }_2^2$ represents the quantization error, and the second term represents the privacy-induced error. In the regime where privacy noise dominates quantization error, the bound is controlled by the $O\left({\displaystyle \frac{1}{{\epsilon }_2}}\right)$ term. □

Increasing codebook richness (K) incurs only a logarithmic penalty, contrasting the linear-in-d scaling typical for continuous Gaussian perturbation.

In theoretical analysis, ${\mathrm{\Pi }}_R$ acts as a hard clipping operator. However, to ensure numerical stability and avoid information loss from post-hoc clipping, our practical implementation structurally enforces this constraint. We modify the final layer of the encoder to include a bounding activation (e.g., $L_2$-normalization or tanh). For instance, applying $L_2$-normalization ensures ${\parallel E\left(x\right)\parallel }_2=1$ (implying $R=1$), or using tanh ensures ${\parallel E\left(x\right)\parallel }_2\le \sqrt{d}$. This architectural constraint guarantees that the sensitivity bound $\Delta u\le 4R^2$ is always satisfied by design, stabilizing the differential privacy accounting.

4.3.3. End-to-End Distributional Error Decomposition

We characterize how the discrepancy between the data distribution and the generated distribution can be decomposed into (i) representation-induced geometric approximation, (ii) distortion introduced by the differentially private discrete projection, and (iii) prior modeling error.

Let $X\sim P_{\mathrm{data}}$ denote a random data point from the underlying population distribution on $\mathcal{X}$, and let its latent representation be understood at the record level. Define the projected embedding $\overline{E}\left(X\right)={\mathrm{\Pi }}_R\left(E\left(X\right)\right)$ and the projected codebook $\overline{\mathcal{C}}={\left\{{\overline{c}}_k\right\}}_{k=1}^K$ with ${\overline{c}}_k={\mathrm{\Pi }}_R\left(c_k\right)$. Let

$$k^{*}\left(X\right)=\arg \underset{k\in \left[K\right]}{\min }{\parallel \overline{E}\left(X\right)-{\overline{c}}_k\parallel }_2^2,Z^{*}={\overline{c}}_{k^{*}\left(X\right)}$$

denote the deterministic nearest-code representation. Let $\widehat{k}={\mathrm{\Pi }}_{{\epsilon }_2}\left(X\right)$ be the index selected by the differentially private projection in Stage II, and let $\tilde{Z}={\overline{c}}_{\widehat{k}}$ denote the corresponding embedded latent representation. Finally, let $P_{\varphi }$ denote the distribution on ${\overline{\mathcal{C}}}^{H\times W}$ induced by the learned prior $p_{\varphi }$, and let $P_{\tilde{Z}}$ and $P_{Z^{*}}$ denote the distributions of $\tilde{Z}$ and $Z^{*}$, respectively.

Proposition 3

(Distributional Error Decomposition in $W_2$). Assume $P_{\mathrm{data}}\in {\mathcal{P}}_2\left(\mathcal{X}\right)$ and that the projected codebook $\overline{\mathcal{C}}\subset {\mathbb{R}}^d$ is bounded. Let $P_{\mathrm{gen}}:=G_{\#}{P}_{\varphi }$ be the generated distribution obtained by sampling $Z\sim P_{\varphi }$ and decoding $G\left(Z\right)$. Under Assumption A1, the Wasserstein-2 distance satisfies

$$W_2(P_{\mathrm{data}},P_{\mathrm{gen}})\le \underset{\mathit{geometric}\mathit{approximation}}{\underbrace{W_2(P_{\mathrm{data}},G_{\#}{P}_{Z^{*}})}}+\underset{\mathit{DP}\mathit{projection}\mathit{distortion}}{\underbrace{W_2(G_{\#}{P}_{Z^{*}},G_{\#}{P}_{\tilde{Z}})}}+\underset{\mathit{prior}\mathit{modeling}\mathit{error}}{\underbrace{W_2(G_{\#}{P}_{\tilde{Z}},G_{\#}{P}_{\varphi })}}.$$

(19)

Moreover, the distortion introduced by the DP projection admits the bound

$$W_2(G_{\#}{P}_{Z^{*}},G_{\#}{P}_{\tilde{Z}})\le L_G{\left(4\mathbb{E}\parallel \overline{E}\left(X\right)-Z^{*}{\parallel }_2^2+{\displaystyle \frac{16R^2}{{\epsilon }_2}}(lnK+1)\right)}^{1/2}.$$

(20)

Proof. 

The decomposition in Equation (19) follows from the triangle inequality in the Wasserstein-2 distance. By Assumption 1, the decoder G is $L_G$-Lipschitz on the support of $\overline{\mathcal{C}}$. Hence, for any distributions P and Q on ${\overline{\mathcal{C}}}^{H\times W}$,

$$W_2(G_{\#}P,G_{\#}Q)\le L_G{W}_2(P,Q).$$

(21)

To bound $W_2(P_{Z^{*}},P_{\tilde{Z}})$, we consider the coupling induced by sampling $X\sim P_{\mathrm{data}}$ and generating $(Z^{*},\tilde{Z})$ via nearest-code assignment and the DP projection applied to the same X. This yields

$$W_2^2(P_{Z^{*}},P_{\tilde{Z}})\le \mathbb{E}\left[\parallel Z^{*}-\tilde{Z}{\parallel }_2^2\right].$$

(22)

Using ${\parallel a-b\parallel }_2^2\le {2\parallel a-c\parallel }_2^2+2{\parallel b-c\parallel }_2^2$ with $c=\overline{E}\left(X\right)$, we obtain

$$\mathbb{E}\left[\parallel Z^{*}-\tilde{Z}{\parallel }_2^2\right]\le 2\mathbb{E}\left[\parallel \overline{E}\left(X\right)-Z^{*}{\parallel }_2^2\right]+2\mathbb{E}\left[\parallel \overline{E}\left(X\right)-\tilde{Z}{\parallel }_2^2\right].$$

(23)

By Theorem 1,

$$\mathbb{E}\left[\parallel \overline{E}\left(X\right)-\tilde{Z}{\parallel }_2^2\right]\le \mathbb{E}\left[\parallel \overline{E}\left(X\right)-Z^{*}{\parallel }_2^2\right]+{\displaystyle \frac{8R^2}{{\epsilon }_2}}(lnK+1).$$

(24)

Substituting into Equation (23) gives

$$\mathbb{E}\left[\parallel Z^{*}-\tilde{Z}{\parallel }_2^2\right]\le 4\mathbb{E}\left[\parallel \overline{E}\left(X\right)-Z^{*}{\parallel }_2^2\right]+{\displaystyle \frac{16R^2}{{\epsilon }_2}}(lnK+1).$$

(25)

Combining Equations (21), (22) and (25) yields Equation (20). □

Proposition 3 separates the distributional discrepancy induced by the discrete DP projection from the geometric approximation determined by the learned representation and the modeling error of the non-private prior. Only the middle term depends explicitly on the privacy budget ${\epsilon }_2$.

4.4. Computational Complexity

We analyze the asymptotic time complexity of the proposed framework to demonstrate its scalability. The computational cost is dominated by the three processing stages, all of which operate in polynomial time with respect to the input dimensions.

• Stage I (Tokenizer): The VQ-VAE relies on standard convolutional operations. For an input of size $H\times W$, the encoding cost scales as $O\left(H·W·C_{\mathrm{channels}}\right)$. The stochastic code selection adds a term $O\left(H·W·K·d\right)$, which is linear in the number of pixels.
• Stage II (DP Projection): The exponential mechanism computes utility scores for all K codes for each record. For a dataset of N samples, this step scales as $O\left(N·H·W·K·d\right)$, which is linear in both dataset size and resolution.
• Stage III (Prior): The Transformer prior employs self-attention, scaling quadratically with the latent sequence length $L=H\times W$. The complexity is $O\left(L^2·d_{\mathrm{model}}\right)$. Since L is fixed by the downsampled latent resolution (e.g., $1/4$ or $1/8$ of pixel resolution), this remains highly efficient for standard image benchmarks compared to pixel- space autoregression.

5. Experiments

In this section, we provide an extensive empirical evaluation of the proposed DP-VQG framework.

5.1. Experimental Setup

5.1.1. Datasets

We evaluate our method on a diverse set of benchmarks. First, we use three standard datasets to compare against existing baselines:

• MNIST [44]: A dataset of 70,000 grayscale handwritten digits ($28\times 28$), serving as a baseline for mode coverage.
• Fashion-MNIST [45]: A dataset of 70,000 grayscale images of clothing items ($28\times 28$), testing the capture of structural details.
• CIFAR-10 [46]: A dataset of 60,000 color images ($32\times 32$) across 10 classes, representing a challenging scenario due to high variance.

Additionally, to demonstrate the scalability of DP-VQG to high-resolution image generation, we conduct experiments on CelebA-HQ [47], Oxford 102 Flowers [48], Food-101 [49] and Stanford Cars [50].

5.1.2. Evaluation Metrics

To quantitatively assess the performance, we employ the following metrics:

• Fréchet Inception Distance (FID): Measures the distance between the distribution of real and synthetic images in the feature space of an Inception-V3 network. Lower FID indicates better image quality and diversity. We compute FID using the clean-fid [51] library in Python 3.12.3.
• Inception Score (IS): Evaluates the clarity and diversity of generated images by measuring the KL-divergence between the conditional class distribution and the marginal class distribution. Consistent with prior work in this domain, we compute IS using specific CNN classifiers trained on the respective original training datasets, rather than the standard ImageNet-pretrained network.
• Downstream Classification Accuracy (Acc): We train a standard two-layer CNN classifier on the synthetic data generated by our method and evaluate it on the real test set. Higher accuracy indicates better preservation of semantic utility.

5.1.3. Baselines

We compare DP-VQG against several state-of-the-art differentially private generative models:

• DP-GAN [9]: A differentially private Generative Adversarial Network trained with DP-SGD.
• G-PATE [33]: A PATE-based approach that trains a student generator by aggregating private gradients from an ensemble of teacher discriminators.
• DP-Merf [52]: A method utilizing random feature representations of kernel mean embeddings to enforce differential privacy with analytic sensitivity guarantees.
• GS-WGAN [36]: Gradient-Sanitized Wasserstein GAN, a robust DP-GAN variant.
• DataLens [53]: A scalable generative framework utilizing the TopAgg mechanism for compressive gradient aggregation to handle high-dimensional data within a PATE-like setting.
• DP-Sinkhorn [37]: An optimal transport-based method that minimizes the Sinkhorn divergence under privacy constraints, avoiding adversarial training instability.
• PEARL [54]: A framework employing characteristic functions for one-shot data sanitization, allowing model training without iterative privacy budget degradation.
• DP-LFlow [35]: A flow-based model that performs generation in a lower-dimensional latent space to enable effective DP training on high-dimensional images.
• DPDM [10]: A standard DDPM trained directly on pixel space using DP-SGD. Due to its significantly larger model size and computational costs compared to other baselines, we exclude it from comparisons on MNIST and Fashion-MNIST for fairness.

5.1.4. Implementation Details

We implement DP-VQG using PyTorch 2.7.0 and Opacus 1.5.4 [55] for differential privacy accounting. All experiments are conducted on NVIDIA A100 GPUs. The specific hyperparameters and architectural choices are detailed below to ensure reproducibility.

Privacy Settings

For all privacy-preserving experiments, we fix the target privacy parameter $\delta ={10}^{-5}$ and vary the total budget $\epsilon$ to analyze the privacy–utility trade-off. Privacy costs are tracked using the PRVs [42] accountant. For FP-VQG, the total privacy budget $\epsilon$ is explicitly split between Stage I and Stage II. In all reported experiments, we allocate the budget in a fixed ratio of ${\epsilon }_1/{\epsilon }_2$ = 4:1 (allocating 80% to structure learning and 20% to discrete projection). This asymmetric design is theoretically grounded in the robustness of disentangled representations [56]. We invest the majority of the budget in Stage I to ensure the learning of a high-fidelity semantic manifold, as the quality of the manifold support is critical for generalization [11]. Once a robust geometry is established, the discrete latent space naturally performs semantic clustering [20], making the representation tolerant to local perturbations. This geometric robustness allows Stage II to operate effectively, even under the higher noise levels implied by the lower budget share, preventing the signal degradation typical of continuous DP mechanisms.

Stage I: Structure Learning and Tokenizer

The VQ-VAE consists of a lightweight encoder–decoder network enhanced with Residual Blocks. To align with the theoretical requirements of the exponential mechanism, we enforce specific architectural constraints:

• Sensitivity Control (R): To satisfy the boundedness requirement used in Stage II and in the sensitivity bound of Equation (12), we explicitly enforce an ${\mathcal{l}}_2$-ball constraint on both encoder embeddings and codebook vectors, i.e., $\parallel \overline{E}{\left(x\right)\parallel }_2\le R$ and $\parallel {\overline{c}}_k{\parallel }_2\le R$. Concretely, for any latent vector $v\in {\mathbb{R}}^d$ (including each spatial token embedding output by E and each codebook entry $c_k$), we apply the deterministic projection

  $${\mathrm{\Pi }}_R\left(v\right)=v·min\left(1,{\displaystyle \frac{R}{{\parallel v\parallel }_2+ϵ}}\right),$$

  with a small $ϵ$ for numerical stability (which is set to ${10}^{-8}$), and use $\overline{E}\left(x\right)={\mathrm{\Pi }}_R\left(E\left(x\right)\right)$ and ${\overline{c}}_k={\mathrm{\Pi }}_R\left(c_k\right)$ in Stage II. This guarantees $\parallel \overline{E}\left(x\right)-{\overline{c}}_k{\parallel }_2\le 2R$ and thus deterministically bounds the global sensitivity of the utility in Equation (11) by $\Delta u\le 4R^2$. We treat R as a dataset-dependent hyperparameter and set it to the 99-th percentile of the empirical ${\mathcal{l}}_2$ norms of the encoder token embeddings on the training data (and enforce the same bound for the learned codebook).
• Stochastic Quantization: During training, we employ stochastic top-k sampling with $k=10$ and a Gumbel-Softmax temperature of $\tau =1.0$. This stochasticity prevents the decoder from overfitting to deterministic codes.
• Loss Weights: We set the commitment loss weight to $\beta =0.25$ and the entropy regularization weight to ${\lambda }_{\mathrm{ent}}=0.1$ to encourage high codebook utilization.
• Hyperparameters: We use a codebook size of $K=1024$ across all datasets. The latent dimension is set to $d=32$ for MNIST/Fashion-MNIST, $d=128$ for CIFAR-10 and $d=256$ for other high-resolution datasets.

For FP-VQG, we train the VQ-VAE from scratch on the private dataset $D_{\mathrm{priv}}$ using DP-SGD. We use a batch size of 256 and a learning rate of 1 ×${10}^{-3}$. Crucially, we disable standard Exponential Moving Average (EMA) updates for the codebook and instead optimize the embeddings $\mathcal{C}$ via explicit gradients to ensure valid privacy tracking.

For PP-VQG, we train a custom high-capacity VQGAN on the public imagenet [57] dataset $D_{\mathrm{pub}}$ using standard SGD (${\epsilon }_1=0$). Unlike off-the-shelf pre-trained models, this implementation strictly incorporates our specific architectural constraints to ensure compatibility with the privacy accounting in Stage II. Furthermore, leveraging the absence of privacy constraints in this stage, inspired by taming-transformers [58], we augment the training with a patch-based adversarial discriminator and perceptual loss to maximize geometric fidelity and sharpness before private data encoding.

Stage II: DP Discrete Projection

We freeze the encoder E and codebook $\mathcal{C}$. For each private record $x_i$, we compute the utility scores $u(x_i,k)=-{\parallel \overline{E}\left(x_i\right)-{\overline{c}}_k\parallel }_2^2$ for all $k\in \left[K\right]$, and sample the discrete code index using the exponential mechanism with sensitivity $\Delta u=4$. This stage consumes the allocated privacy budget ${\epsilon }_2$.

In practice, the latent representation is a spatial grid of discrete tokens. We instantiate the exponential mechanism independently at each spatial location and account for the resulting privacy loss using Privacy Random Variable (PRV) composition [42]. We calibrate the per-location mechanisms using PRVs such that their composed privacy loss matches the target record-level budget ${\epsilon }_2$, following standard practice. All privacy budgets reported in the experiments refer to this final record-level guarantee. The detail parameters are shown in Table 1.

Table 1. Token grid sizes and privacy accounting for Stage II.

Dataset	Token Grid ($\mathit{H}\times \mathit{W}$)	Accountant	Record-Level ${\mathit{\epsilon }}_2$
MNIST	$7\times 7$	PRVs	${\epsilon }_2$
Fashion-MNIST	$7\times 7$	PRVs	${\epsilon }_2$
CIFAR-10	$8\times 8$	PRVs	${\epsilon }_2$
High-resolution datasets	$16\times 16$	PRVs	${\epsilon }_2$

Stage III: Masked Generative Modeling

We employ a MaskGIT-based [22] bidirectional Transformer as the generative prior trained via Masked Visual Token Modeling (MVTM). To adapt to the varying complexity of different datasets, we scale the Transformer architecture and training configurations accordingly. These dataset-specific hyperparameters are summarized in Table 2. Across all experiments, we use a cosine learning rate schedule decaying from $1\times {10}^{-4}$ to $5\times {10}^{-6}$, and the inference sampling steps are fixed at $T=16$.

Table 2. Hyperparameters for Stage III (Masked Generative Modeling) across different datasets. The Transformer architecture is detailed in terms of embedding dimension ($d_{\mathrm{model}}$), number of layers (L), and attention heads (h).

Dataset	Dim (${\mathit{d}}_{\mathbf{model}}$)	Layers (L)	Heads (h)	Epochs	Batch Size	Cond.	CFG Scale
MNIST/Fashion-MNIST	384	6	6	120	120	Class	4.0
CIFAR-10	384	6	6	200	256	Class	3.0
Food-101/Oxford Flowers	384	6	6	300	128	Class	2.5
Stanford Cars	384	6	6	300	128	Uncond.	–
CelebA-HQ	512	8	8	300	128	Uncond.	–

5.2. Main Results: Quality and Utility

Table 3 summarizes the quantitative performance on MNIST and Fashion-MNIST under a moderate privacy budget ($\epsilon =10$). Across both datasets, DP-VQG consistently outperforms prior DP generative models, including GAN-based approaches (e.g., DP-GAN, GS-WGAN) and flow-based methods such as DP-LFlow. On MNIST, our method achieves an FID of 18.21, improving upon DP-LFlow by a substantial margin and attains downstream classification accuracy close to that obtained from real data.

Table 3. Quantitative comparison on MNIST and Fashion-MNIST under privacy budget $\epsilon =10$, $\delta ={10}^{-5}$. We report Fréchet Inception Distance (FID ↓), Inception Score (IS ↑), and Downstream Accuracy (Acc ↑). Bold indicates the best privacy-preserving performance.

Method	MNIST			Fashion-MNIST
	FID ↓	IS ↑	Acc ↑	FID ↓	IS ↑	Acc ↑
DP-GAN [9]	179.16	4.76	63.0%	243.80	3.48	53.0%
G-PATE [33]	177.20	3.85	51.0%	205.80	3.32	50.0%
DP-Merf [52]	116.30	2.91	63.0%	132.60	2.28	62.0%
GS-WGAN [36]	58.13	8.83	80.0%	131.34	5.43	65.0%
DataLens [53]	173.50	5.78	80.7%	167.70	3.12	70.6%
DP-Sinkhorn [37]	48.40	8.12	83.2%	128.30	4.56	71.1%
Pearl [54]	116.0	8.12	78.8%	102.00	3.1	81.2%
DP-LFlow [35]	25.40	8.26	94.8%	80.80	5.43	80.7%
DP-VQG (Ours)	18.21	9.13	97.22%	77.09	5.68	81.7%
Real Data	-	9.76	99.4%	-	7.79	92.0%

We note that some baseline results are taken from their respective publications and may follow slightly different evaluation protocols; these numbers are therefore used as reference points rather than strict, protocol-matched comparisons. Nevertheless, the overall performance gap is consistent with the visual comparisons in Figure 4, indicating that discrete latent projection preserves fine-grained structure more effectively than pixel-space perturbation under the same privacy budget.

Figure 4. Qualitative comparison of generated samples across four benchmarks under formal privacy constraints ($\epsilon$ = 10.0). Baseline samples are sourced from their respective original publications for fair comparison. On standard grayscale datasets (MNIST and Fashion-MNIST), our FP-VQG generates samples with higher realism and structural fidelity compared to prior arts. On challenging high-dimensional datasets (CIFAR-10 and CelebA), where existing methods typically suffer from severe blurring or noise, our PP-VQG significantly outperforms baselines, breaking the quality barrier to produce sharp, high-fidelity images with coherent semantics.

Beyond generative fidelity, DP-VQG demonstrates exceptional semantic preservation. On MNIST, the downstream classification accuracy reaches 97.22%, closely approaching the upper bound of real data (99.4%) and surpassing the previous best by a clear margin. Even on the more structurally complex Fashion-MNIST dataset, our method maintains this advantage, achieving an FID of 77.09 and an accuracy of 81.7%. These results indicate that the discrete codebook in Stage I effectively captures and preserves class-discriminative features, ensuring high data utility while robustly managing differential privacy noise.

5.3. Privacy–Utility Trade-Off Analysis

To comprehensively evaluate the robustness of DP-VQG under varying privacy constraints, we conduct a sensitivity analysis on MNIST and Fashion-MNIST by sweeping the privacy budget $\epsilon$ from strict ($\epsilon =0.2$) to relaxed ($\epsilon =10$) settings, while fixing $\delta ={10}^{-5}$. Figure 5 illustrates the Fréchet Inception Distance (FID) trends compared against state-of-the-art baselines.

Figure 5. FID comparison under different privacy budgets $\epsilon$. (Left) MNIST. (Right) Fashion-MNIST.

The results demonstrate that DP-VQG consistently maintains the lowest FID scores across the entire privacy spectrum, exhibiting exceptional robustness in high-privacy regimes. Specifically, at extremely low privacy budgets ($\epsilon \le 1.0$), traditional GAN-based methods (e.g., DP-GAN, GS-WGAN) suffer from severe mode collapse, resulting in FID scores exceeding 300. In contrast, our method effectively disentangles privacy preservation from generative quality via the latent codebook, maintaining FID scores below 100 even at $\epsilon =0.5$. Furthermore, while the competitive baseline DP-LFlow shows strong performance at higher budgets, our DP-VQG demonstrates faster convergence to optimal utility. As observed in the curves, our performance gain is most pronounced in the strict privacy interval, after which the FID metric stabilizes around $\epsilon =2$, suggesting that our model is highly budget-efficient and can generate high-fidelity samples without requiring excessive privacy costs.

5.4. High-Resolution Image Synthesis with Public Priors

Existing differentially private generative models struggle to scale to high-resolution image synthesis, as protecting high-dimensional pixel or continuous latent representations requires injecting excessive noise, often leading to severe perceptual degradation.

We address this challenge with the PP-VQG variant, which leverages a public VQGAN pre-trained on ImageNet as a strong perceptual prior. By decoupling low-level visual representation from semantic and compositional modeling, PP-VQG restricts privacy-sensitive learning to the discrete latent code space. Differential privacy is applied only to the selection of discrete codes, while the encoder and decoder—responsible for encoding and rendering general visual primitives—are trained entirely on public data.

Figure 6 presents qualitative samples generated under a moderate privacy budget of $\epsilon =10$ on five datasets, CIFAR-10, Oxford-102 Flowers, Food-101, CelebA-HQ, and Stanford Cars, all synthesized at a resolution of $256\times 256$. Across these diverse domains, PP-VQG produces visually coherent and photorealistic images, preserving global structure and fine-grained details.

Figure 6. Images synthesized by PP-VQG under $\epsilon =10$ using a VQGAN pre-trained on ImageNet. All datasets are synthesized at a resolution of $256\times 256$, except for CIFAR-10, which is generated at its native resolution of $32\times 32$.

The robustness of PP-VQG to privacy noise stems from its discrete latent formulation. Since privacy perturbations affect only code selection, errors primarily manifest as localized structural or compositional deviations, while overall perceptual quality remains stable. This contrasts with pixel-space DP models, where noise directly corrupts continuous representations and often results in global blurring.

Table 4 reports FID and Kernel Inception Distance (KID) [59] scores under $\epsilon =10$ on a set of higher-resolution datasets. At this resolution, these metrics serve primarily as coarse distributional references rather than definitive indicators of sample quality, as quantitative evaluation under differential privacy is rarely reported beyond low-resolution benchmarks. Existing DP generative models are typically restricted to $32\times 32$ or $64\times 64$ images, for example, DPDM [10] reporting an FID of 97.7 on CIFAR-10 and 78.3 on CelebA downsampled to $64\times 64$, or DP-LFlow [35] exhibiting substantially higher FID values at comparable scales. In contrast, our results show that DP-VQG remains stable when applied to $256\times 256$ datasets, producing coherent samples while maintaining formal privacy guarantees. These experiments highlight the scalability of discrete latent projection under differential privacy, complementing the low-resolution quantitative gains reported earlier.

Table 4. Quantitative evaluation of samples generated by PP-VQG under a formal differential privacy constraint with privacy budget $\epsilon =10$. Lower FID and KID indicate better generative quality.

Metric	CIFAR-10	Food-101	Flowers-102	CelebA-HQ	Stanford Cars
FID ↓	47.99	87.70	110.99	98.27	68.60
KID $\times {10}^3\downarrow$	20.67	38.54	72.88	39.41	39.69

While PP-VQG leverages a public dataset to learn the encoder–decoder and codebook, the resulting tokenizer is lightweight compared to end-to-end DP generative baselines. In particular, the model size and training cost of our public VQGAN tokenizer are substantially smaller than pixel-space DP diffusion models such as DPDM [10], which require large-scale models and multi-GPU training even on low-resolution datasets. In practice, PP-VQG enables fast single-GPU training for downstream private datasets once the public tokenizer is learned.

Overall, these results demonstrate that PP-VQG enables practical high-resolution image synthesis under differential privacy by exploiting public visual priors and shielding low-level visual representations from privacy noise.

5.5. Parameter Experiments

We investigate the effect of the codebook size K on generative performance under a fixed privacy budget. Figure 7 reports the Fréchet Inception Distance (FID) obtained with different values of K, while keeping all other components of DP-VQG unchanged, including the privacy parameter $\epsilon$, the tokenizer training protocol, and the generative prior.

Figure 7. Effect of codebook size K on FID under a fixed privacy budget $\epsilon$. (Left) MNIST. (Right) Fashion-MNIST.

As K increases from small values, the generation quality improves substantially. This regime reflects the benefit of a richer discrete representation: a larger codebook enables a finer partition of the latent space and reduces the intrinsic quantization error, leading to more accurate geometric approximation of the data manifold. However, beyond an intermediate range, further enlarging the codebook results in degraded performance.

This non-monotonic behavior is consistent with our theoretical analysis. On the one hand, increasing K improves the deterministic nearest-code approximation and thus reduces the geometric error term in the end-to-end decomposition. On the other hand, under a fixed privacy budget, the expected distortion introduced by the differentially private discrete projection scales logarithmically with K. When K becomes too large, this privacy-induced selection noise dominates the representation gain, leading to a degradation in generation quality.

We further note that the optimal range of K is closely related to the intrinsic complexity of the underlying dataset. Datasets with low visual diversity and simple geometric structure such as MNIST can be well represented with a relatively small codebook, as the geometric approximation error is already small. In contrast, datasets with richer visual diversity and higher-dimensional latent manifolds require larger K to adequately capture the underlying structure and suppress quantization artifacts. However, for any fixed privacy budget $\epsilon$, excessively increasing K inevitably amplifies the DP-induced selection distortion through the logarithmic $lnK$ term, which limits the practically optimal codebook size even for complex datasets.

Overall, these results reveal a clear trade-off between representation capacity and privacy robustness, and highlight the importance of choosing an appropriate codebook size that balances geometric fidelity with the stability of private discrete selection. In practice, we select K by jointly monitoring the tokenizer reconstruction error and the effective codebook utilization under the target privacy budget, choosing the smallest K that sufficiently suppresses quantization artifacts while avoiding the regime where privacy-induced selection noise dominates.

5.6. Ablation Study and Robustness Analysis

5.6.1. The “Privacy–Fidelity Gap” in Continuous Latent Spaces

To examine the effect of where privacy noise is injected, we construct a controlled comparison between continuous and discrete latent perturbations. Both variants share the same encoder–decoder architecture, latent dimensionality, and privacy budget; the only difference lies in whether noise is applied directly to continuous latent vectors or to discrete code selection. This comparison is not intended to represent all continuous DP generative methods, but rather to isolate the practical impact of noise placement under an otherwise matched setup.

We conducted a “stress test” by relaxing the privacy budget $\epsilon$ from strict ($1.0$) to non-private levels (1000). As visualized in Figure 8, the continuous baseline suffers from a “signal drowning” effect. At standard privacy budgets ($\epsilon \le 10$), the reconstruction is dominated by noise, resulting in unrecognizable outputs. Remarkably, the continuous model requires an impractical budget of $\epsilon \ge 100$ to produce recognizable digits. In sharp contrast, our DP-VQG achieves high-fidelity reconstruction with sharp structural details at a strict budget of $\epsilon =1.0$. These results indicate that, under this controlled setting, discrete latent projection provides greater robustness to privacy noise than direct perturbation of continuous latents.

Figure 8. Reconstruction results on the MNIST dataset with different perturbation strategies on latent representations. Using a DP-trained VQ-VAE ($\epsilon =2.0$), the comparison highlights the superior robustness of discrete codebook selection (b) over continuous noise injection (a).

5.6.2. Component Analysis of Robust Discrete Representation

We perform a component-wise ablation study to verify the necessity of the two mechanisms introduced in our robust tokenizer: Stochastic Code Sampling and Entropy Regularization. We aim to disentangle their individual contributions towards mitigating the noise sensitivity of the discrete bottleneck. Table 5 summarizes the FID scores on MNIST and Fashion-MNIST under a fixed privacy budget ($\epsilon =10$). We compare four variants of the tokenizer training strategy against a non-private reference baseline.

Table 5. Ablation study of robust tokenizer components on MNIST and Fashion-MNIST. We report FID scores (lower is better) under the same differential privacy budget. The first row provides a non-private reference (lower bound) using our robust model with deterministic code selection. The symbols 3 and 7 denote the presence and absence of components, respectively. Bold values highlight the best results among the privacy-preserving methods.

Method	Stochastic	Entropy	DP	FID ↓
	Sampling	Reg.	Choice	MNIST	F-MNIST
Reference (Deterministic)	✓	✓	✗	12.9	56.4
Standard VQ-VAE	✗	✗	✓	48.5	141.3
Entropy-Only	✗	✓	✓	19.8	96.4
Stochastic-Only	✓	✗	✓	20.0	101.1
Robust VQ-VAE (Ours)	✓	✓	✓	18.2	77.1

Vulnerability of Standard VQ-VAE

The Standard VQ-VAE (trained with deterministic argmin and no regularization) exhibits catastrophic degradation when subjected to differential privacy noise. As shown in Table 5, the FID spikes to 48.5 on MNIST and 141.3 on Fashion-MNIST. This confirms that without robust training, the decoder overfits to precise codebook vectors and fails to generalize to the stochastic perturbations introduced by the DP mechanism, rendering the generated images structurally incoherent.

Individual Contributions

Deploying either mechanism individually yields substantial improvements. Stochastic-Only training forces the decoder to tolerate code selection jitter, reducing FID to 20.0 on MNIST. Similarly, Entropy-Only training prevents codebook collapse and ensures a dense latent space, achieving an FID of 19.8. While both variants significantly outperform the standard baseline, neither achieves optimal performance in isolation, suggesting that addressing noise sensitivity and codebook utilization separately is insufficient.

Synergistic Effect

The proposed Robust VQ-VAE combines both mechanisms to achieve the best trade-off, lowering the FID to 18.2 on MNIST and 77.1 on Fashion-MNIST. This demonstrates a synergistic effect: entropy regularization ensures a rich and utilized representation space, while stochastic sampling explicitly prepares the decoder for the inevitable noise injection during the privacy-preserving stage. Comparing these results to the Reference (Deterministic) baseline (Row 1), our method successfully closes the gap caused by DP noise, whereas the standard approach suffers from an unacceptable loss of utility.

5.7. Empirical Validation of Theoretical Analysis

Several key components of our theoretical analysis rely on structural assumptions and worst-case bounds. In this subsection, we provide empirical validation and diagnostic analyses to quantify how these theoretical components manifest in practice.

5.7.1. Empirical Verification of Anchored Decoder Regularity

Assumption 1 postulates that the decoder G is $L_G$-Lipschitz on the support of the projected codebook $\overline{\mathcal{C}}$. Since exact Lipschitz constants of deep networks are difficult to compute, we adopt an empirical proxy to characterize the local regularity of G on the discrete latent support.

Specifically, for randomly sampled code pairs $(c_i,c_j)\in \overline{\mathcal{C}}$, we estimate the ratio

$${\widehat{L}}_G(i,j)={\displaystyle \frac{\parallel G\left(c_i\right)-G\left(c_j\right){\parallel }_2}{\parallel c_i-c_j{\parallel }_2}},$$

(26)

and report summary statistics over random pairs and nearest-neighbor pairs on the codebook graph. This proxy directly measures the effective Lipschitz behavior of G restricted to the latent support used at generation time.

Table 6 reports the empirical distributions of ${\widehat{L}}_G$ across datasets. We observe that low-dimensional datasets such as MNIST exhibit smaller and more concentrated ${\widehat{L}}_G$ values, whereas higher-resolution datasets (e.g., CelebA-HQ) yield larger and more dispersed ratios. These results support the dataset-dependent nature of $L_G$ and empirically validate Assumption 1 in all evaluated settings.

Table 6. Empirical Lipschitz proxy of the decoder on the projected codebook support.

	Random Pairs				Nearest-Neighbor Pairs
Dataset	Max	P99	Mean	Median	Max	P99	Mean	Median
MNIST	22.10	15.54	6.85	6.22	21.98	16.80	7.22	6.56
Fashion-MNIST	74.58	53.98	21.89	19.64	84.94	68.89	23.66	18.84
CelebA-HQ	44.29	18.88	9.11	8.73	47.99	22.07	8.25	7.25

5.7.2. Empirical Decomposition of the $W_2$ Error Bound

Proposition 3 decomposes the end-to-end Wasserstein-2 discrepancy into three conceptual components: geometric approximation, DP projection distortion, and prior modeling error. Since Equation (19) is an upper bound, it does not define a unique additive partition of $W_2(P_{\mathrm{data}},P_{\mathrm{gen}})$. To provide quantitative insight into which component dominates across datasets and privacy budgets, we introduce computable empirical proxies aligned with the three theoretical sources of error.

Geometric Approximation Proxy

We measure representation-induced distortion by the non-private nearest-code quantization error:

$${\mathcal{E}}_{\mathrm{geo}}={\mathbb{E}}_{x\sim P_{\mathrm{data}}}\left[\parallel \overline{E}\left(x\right)-Z^{*}{\left(x\right)\parallel }_2^2\right].$$

(27)

DP Projection Distortion Proxy

We quantify the additional distortion introduced by the exponential-mechanism projection as the expected excess distortion:

$${\mathcal{E}}_{\mathrm{dp}}={\mathbb{E}}_{x\sim P_{\mathrm{data}}}\left[\parallel \overline{E}\left(x\right)-\tilde{Z}{\left(x\right)\parallel }_2^2-{\parallel \overline{E}\left(x\right)-Z^{*}\left(x\right)\parallel }_2^2\right].$$

(28)

Prior Modeling Proxy

To approximate the modeling discrepancy between sanitized latents and the learned prior $p_{\varphi }$, we compare their empirical token distributions on the $H\times W$ lattice via the Jensen–Shannon divergence:

$${\mathcal{E}}_{\mathrm{prior}}=\mathrm{JSD}\left(\mathrm{Hist}\left(\tilde{Z}\right),\mathrm{Hist}\left(Z_{\varphi }\right)\right).$$

(29)

Table 7 reports $({\mathcal{E}}_{\mathrm{geo}},{\mathcal{E}}_{\mathrm{dp}},{\mathcal{E}}_{\mathrm{prior}})$ across datasets and privacy budgets. We observe that the geometric approximation and DP projection terms are of comparable magnitude, while the prior modeling proxy is smaller in most settings. Moreover, under record-level budgeting over dense token grids, the DP-induced excess distortion varies only mildly within practical privacy ranges, indicating that geometric quantization error dominates the end-to-end discrepancy beyond moderate privacy budgets. These empirical findings are consistent with the conservative nature of the worst-case bound in Proposition 3.

Table 7. Empirical proxies aligned with the three components in Proposition 3: geometric approximation ${\mathcal{E}}_{\mathrm{geo}}$, DP-induced excess distortion ${\mathcal{E}}_{\mathrm{dp}}$, and prior modeling mismatch ${\mathcal{E}}_{\mathrm{prior}}$ (Jensen–Shannon divergence between token histograms).

Dataset	${\mathit{\epsilon }}_2$	${\mathcal{E}}_{\mathbf{geo}}$	${\mathcal{E}}_{\mathbf{dp}}$	${\mathcal{E}}_{\mathbf{prior}}$
MNIST	10	0.592	0.377	0.233
MNIST	2	0.592	0.370	0.233
Fashion-MNIST	10	0.623	0.400	0.214
CelebA-HQ	10	0.043	0.068	0.642
CelebA-HQ	2	0.043	0.070	0.641

6. Conclusions

In this work, we presented DP-VQG, a differentially private generative framework that enforces privacy through a discrete latent bottleneck rather than through end-to-end noise injection in high-dimensional continuous spaces. By decomposing the generative pipeline into (i) representation learning, (ii) differentially private discrete latent projection, and (iii) non-private generative modeling, DP-VQG isolates privacy-sensitive operations to a compact and structured interface while treating the subsequent generative process as pure post-processing.

From a conceptual perspective, DP-VQG highlights the importance of aligning privacy mechanisms with the structure of modern generative models. Our analysis shows that, when privacy is enforced via discrete latent selection, the resulting utility degradation scales logarithmically with the codebook size rather than with the ambient latent dimension. This property contrasts sharply with continuous perturbation mechanisms, which often suffer from severe off-support effects under realistic privacy budgets. Theoretical results formalize this distinction through explicit bounds on latent and reconstruction-space distortion, while the end-to-end Wasserstein decomposition clarifies how privacy-induced error interacts with geometric approximation and prior modeling.

Empirically, DP-VQG achieves a strong privacy–utility trade-off across a range of benchmarks. On standard low-resolution datasets, it consistently outperforms existing differentially private generative baselines in both sample quality and downstream utility. Moreover, the public–prior variant demonstrates that high-resolution image synthesis under formal differential privacy is feasible when privacy noise is confined to a discrete latent interface, enabling visually coherent generation in regimes where prior end-to-end DP approaches struggle to scale.

While DP-VQG provides a principled and scalable alternative to continuous DP perturbations, several open questions remain. In particular, for very large generative models, the required codebook size may increase by one or two orders of magnitude. In these regimes, it becomes increasingly critical to investigate the computational cost and privacy-induced distortion associated with differentially private discrete selection. Exploring hierarchical, structured, or adaptive codebooks that preserve the advantages of discrete privacy mechanisms at scale is a promising direction for future work.

Overall, this work suggests that discrete latent interfaces offer a robust foundation for privacy-preserving generative modeling, providing a favorable balance between formal privacy guarantees and practical sample quality. We hope that DP-VQG will encourage further research on structurally aligned privacy mechanisms for large-scale generative models beyond end-to-end noisy training.