Local Differential Privacy Image Generation Using Flow-Based Deep Generative Models

Abstract

Diagnostic radiologists need artificial intelligence (AI) for medical imaging, but access to medical images required for training in AI has become increasingly restrictive. To release and use medical images, we need an algorithm that can simultaneously protect privacy and preserve pathologies in medical images. To address this, we introduce DP-GLOW, a hybrid that combines the local differential privacy (LDP) algorithm with GLOW, one of the flow-based deep generative models. By applying a GLOW model, we disentangle the pixelwise correlation of images, which makes it difficult to protect privacy with straightforward LDP algorithms for images. Specifically, we map images to the latent vector of the GLOW model, where each element follows an independent normal distribution. We then apply the Laplace mechanism to this latent vector to achieve $ϵ$-LDP, which is one of the LDP algorithms. Moreover, we applied DP-GLOW to chest X-ray images to generate LDP images while preserving pathologies. The $ϵ$-LDP-processed chest X-ray images obtained with DP-GLOW indicate that we have obtained a powerful tool for releasing and using medical images for training AI.

1. Introduction

Diagnostic radiologists need artificial intelligence (AI) for medical imaging to reduce workloads and enhance productivity. However, access to medical images required for the training of AI has become increasingly restrictive owing to the increasing demands for the protection of personal information in each country. Moreover, data use agreements bind the purpose of use for even medical image datasets open to researchers upon request (e.g., [1]). Additionally, if someone specifies personal identities in medical images, irreversible leakage of personal information may occur.

Here, we assume a situation in which we anonymize test datasets when training datasets are open to the public. We further assume that probabilistic distributions of medical images for the training and test datasets are similar. For example, the training dataset can be a large-scale dataset already open to the public from a hospital, and the test dataset can be a dataset privately held by another hospital.

Differential privacy (DP) algorithms [2] have recently emerged as tools with a provable privacy protection guarantee and usefulness. We specifically focus on local DP (LDP) [3], which adds noise to each image so that we cannot specify any identity in an image. Because one can anonymize the image upstream of image processing using LDP algorithms, we can use the processed images without limiting the purpose of use. At the same time, LDP can retain valuable information in the original image, e.g., lung opacity suggesting pneumonia in chest X-ray (CXR) images.

Because CXR images are the most representative medical images, in this study, we adopt CXR images from the Radiological Society of North America (RSNA) dataset. However, we can extend the proposed method to natural images and medical images of arbitrary modality and dimensions.

To summarize, our contributions are as follows:

• We adopt GLOW to disentangle image pixels and realize the $ϵ$-LDP algorithm for images (DP-GLOW).
• We generate $ϵ$-LDP-processed CXR images.
• We evaluate the usefulness of $ϵ$-LDP-processed CXR images using a pneumonia detection model.
• We visually confirm how the proposed method obscures identities in medical images.

2. Related Works

Abadi et al. [4] proposed a differentially private stochastic gradient descent (DP-SGD) method, in which a controlled noise is added to the gradient of parameters and then clipped during the training of a deep model. Ziller et al. [5] proposed the training of a segmentation network for CXR images using a discriminative model trained with DP-SGD. Kossen et al. [6] proposed the generation of differentially private time-of-flight magnetic resonance angiography (TOF-MRA) images using generative adversarial networks (GANs) trained with DP-SGD. For DP-SGD, the theoretical guarantee that images generated using GANs trained with DP-SGD satisfy $ϵ$-LDP is not apparent.

Fan [7] adopted LDP (although it is not explicitly stated in the paper) for pixelized images. Image pixelization has the effect of reducing the global sensitivity of the LDP algorithm. Fan [8] proposed another LDP algorithm using pixelization and Gaussian blur. However, these methods can significantly degrade the quality of the original images. Croft et al. [9], Liu et al. [10], and Li and Clifton [11] almost simultaneously proposed another LDP algorithm for images. Liu et al. [10] showed a concrete implementation using GANs, whereas Croft et al. [9] showed an abstract formulation. The implementation of the LDP algorithm for images by Li and Clifton is similar to that by Liu et al., but Li and Clifton adopt clipping so that the generated LDP images are within the probabilistic distribution of training images. Finally, Croft et al. [12] experimented with an LDP algorithm for facial obfuscation. On the other hand, we propose an algorithm for local differential privacy using GLOW [13], one of the flow-based deep generative models. We adopt GLOW as it is a renowned and representative flow-based deep generative model. While it is challenging to map real images to latent space without degrading them in GANs, it is easily achievable with flow-based deep generative models.

In previous studies [7,8,9,10,11,12], LDP was not adopted for medical images. Therefore, previous studies did not evaluate the usefulness of LDP-processed images being able to contain valuable information for AI for medical imaging; in this study, we aim to evaluate the usefulness experimentally and quantitatively. As a metric of usefulness, we adopt the area under the curve (AUC) for pathology detection (we assume pneumonia detection in this study), which is essential in AI for medical imaging.

3. Materials and Methods

3.1. Dataset for CXR Images

We took CXR images from the RSNA Pneumonia Detection Challenge dataset [14]. This dataset comprises 30,000 frontal-view CXR images, with each image labeled as “Normal”, “No Opacity/Not Normal”, or “Opacity” by one to three board-certified radiologists. The Opacity group consists of images with suspicious opacities suggesting pneumonia, and the No Opacity/Not Normal group consists of images with abnormalities other than pneumonia.

We show the composition of the CXR image sets in Table 1. We randomly sampled CXR images for the three datasets. We then trained GLOW, one of the flow-based deep generative models (DGMs) using ${\mathcal{S}}_{\mathrm{mixture}}^{\mathrm{train}}$. The goal was to ensure that lung opacity, which suggests pneumonia, was not overly obfuscated for the DP-GLOW algorithm. For testing purposes, we used images from ${\mathcal{S}}_{\mathrm{unknown}}^{\mathrm{test}}$ to generate $ϵ$-LDP-CXR images with DP-GLOW. Subsequently, we detected pneumonia from these $ϵ$-LDP-CXR images. Additionally, we utilized ${\mathcal{S}}_{\mathrm{normal}}^{\mathrm{train}}$ to train a separate model specifically designed to detect pneumonia from CXR images (see Appendix A for details).

Table 1. Composition of datasets. ${\mathcal{S}}_{\mathrm{normal}}^{\mathrm{train}}$ and ${\mathcal{S}}_{\mathrm{mixture}}^{\mathrm{train}}$ share 6529 normal CXR images.

Set	Normal	Abnormal
${\mathcal{S}}_{\mathrm{normal}}^{\mathrm{train}}$	7808	0
${\mathcal{S}}_{\mathrm{mixture}}^{\mathrm{train}}$	6553	6631
${\mathcal{S}}_{\mathrm{unknown}}^{\mathrm{test}}$	1358	13,863

3.2. Preliminary for GLOW

The probability distribution from which images arise is complex and difficult to handle. Thus, flow-based deep generative models (DGMs) [13,15,16] transform this probability distribution into more manageable distributions, e.g., the elementwise independent normal distribution, by requiring the neural network to be bijective. Specifically, using the change of variables formula, we obtain the following:

$$\begin{array}{c}\hfill logp\left(\mathit{x}\right)=logp\left(\mathit{z}\right)+log\left|\frac{\partial \mathit{z}}{\partial \mathit{x}}\right|,\end{array}$$

(1)

where $\mathit{z}$ is a random variable vector that follows a Gaussian distribution independent for each element, and $\mathit{x}$ is a random variable vector sampled from the probability distribution to which images belong. During this transformation process, the logarithm of the determinant appears with respect to the neural network. To efficiently compute this, the neural network is decomposed and represented as a product of functions where the determinant is easy to compute. Specifically, we define the relationship between $\mathit{z}$ and $\mathit{x}$:

$$\begin{array}{ccc}\hfill \mathit{z}& =& {\mathit{G}}_{\mathit{\theta }}^{-1}\left(\mathit{x}\right),\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill & =& {\mathit{g}}_K^{-1}\circ {\mathit{g}}_{K-1}^{-1}\cdots {\mathit{g}}_1^{-1}\left(\mathit{x}\right),\hfill \end{array}$$

(3)

where ${\mathit{G}}_{\mathit{\theta }}^{-1}$ is a trainable (parameters $\mathit{\theta }$) invertible map between a probabilistic distribution of images and a tractable probabilistic distribution (elementwise independent normal distribution in our settings) and ${\mathit{g}}_K^{-1}\cdots {\mathit{g}}_1^{-1}$ are decomposed functions. Using Equation (1), the flow-based DGMs maximize the average logarithm likelihood of training images ($\mathcal{L}$) during training:

$$\begin{array}{c}\hfill \mathcal{L}=\frac{1}{\left|\mathcal{D}\right|}\sum _i^{\mathcal{D}}logp\left({\mathit{x}}_i\right),\end{array}$$

(4)

where $\mathcal{D}$ represents image dataset, ${\mathit{x}}_i$ indicates an image in the dataset, and $\left|\mathcal{D}\right|$ is number of images in the dataset. After the training, the flow-based DGMs can generate fake but realistic images using Equation (3) (sampling) and can explicitly compute the value of the probabilistic density function of images using Equation (1) (density estimation).

GLOW [13] is one of the flow-based DGMs. The deep network in GLOW recursively contains the actnorm, coupling, and permutation layers. The actnorm layer normalizes the data. The coupling layer contains deep convolutional neural networks while it guarantees the invertibility of the layer. The permutation layer ensures that processing in coupling layers affects all the elements of data and is implemented using $1\times 1$ convolution. Moreover, GLOW adopts a multiscale architecture, which can contain multiple deep networks of different recursive levels. This approach can reduce memory requirements and computational costs without significant compromise to image quality. For a detailed understanding of GLOW, we refer the reader to [13].

3.3. Our Framework: DP-GLOW

We represent a gray-scale CXR image as a vector $\mathit{x}\in {\mathbf{R}}^{H\times W}$, where H and W are the height and width of the image, respectively. We assume that the abovementioned GLOW has already been trained with many CXR images. Moreover, as a result of the training, we assume that we obtained an invertible map ${\mathit{G}}_{\mathit{\theta }}^{-1}$ between an elementwise independent normal distribution and a probabilistic distribution of CXR images.

Our objective is to generate another image $\tilde{\mathit{x}}$ from $\mathit{x}$, which satisfies the definition of $ϵ$-LDP:

$$\begin{array}{ccc}\hfill logp\left(\tilde{\mathit{x}}\right|\mathit{x})-logp\left(\tilde{\mathit{x}}\right|{\mathit{x}}^{\prime })& \le & ϵ,\hfill \end{array}$$

(5)

where $ϵ(\ge 0)$ is the privacy budget, ${\mathit{x}}^{\prime }$ is an arbitrary image taken from a probabilistic distribution of CXR images, $p\left(\tilde{\mathit{x}}\right|\mathit{x})$ is the posterior probability to obtain $\tilde{\mathit{x}}$ when we already obtained $\mathit{x}$, and $p\left(\tilde{\mathit{x}}\right|{\mathit{x}}^{\prime })$ is the posterior probability to obtain $\tilde{\mathit{x}}$ when we already obtained ${\mathit{x}}^{\prime }$. Specifically, even if we simply add noise that follows a Laplace distribution to the image itself, the image satisfies local differential privacy. However, especially when the privacy budget is small, the intensity of the noise increases, and the class of the image (in this case, CXR images) is not retained. In such cases, the utility drastically decreases. To avoid this, in this study, we add noise following the Laplace distribution to the image mapped to the latent space and then revert the perturbed latent space vector back to the image space. To this end, we introduce a trained invertible vector function ${\mathit{G}}_{\mathit{\theta }}^{-1}$, which maps $\mathit{x}$ into another vector $\mathit{z}$ (a latent space vector), each element of which does not have a correlation with other elements. This vector function is, in general, dependent on the probabilistic distribution of images we adopt (e.g., CXR, head computed tomography, and mammography images). Therefore, we explicitly represent this dependence as parameters $\mathit{\theta }$. We further define ${\mathit{z}}^{\prime }\equiv {\mathit{G}}_{\mathit{\theta }}^{-1}\left({\mathit{x}}^{\prime }\right)$, and $\tilde{\mathit{z}}\equiv {\mathit{G}}_{\mathit{\theta }}^{-1}\left(\tilde{\mathit{x}}\right)$. As we will prove later, we can add noise that follows the Laplace distribution to the latent space vector:

$$\begin{array}{ccc}\hfill {\tilde{\mathit{z}}}_k& =& {\mathit{z}}_k+{\mathcal{N}}_k,\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill {\mathcal{N}}_k& \sim & \mathrm{Lap}\left({\mu }_k=0,{\sigma }_k=\frac{\Delta z_k}{\frac{ϵ}{H·W}}\right),\hfill \end{array}$$

(7)

where ${\mu }_k$ and ${\sigma }_k$ are, respectively, the expectation (a scalar) and scale (a scalar) of the Laplace distribution, $ϵ$ is the user-defined privacy budget, and $\Delta z_k$ is the sensitivity for the element k defined as:

$$\begin{array}{ccc}\hfill \Delta z_k& :=& \underset{\mathit{z},{\mathit{z}}^{\prime }:\mathit{z}\ne {\mathit{z}}^{\prime }}{max}|z_k-z_k^{\prime }|,\hfill \end{array}$$

(8)

where $\mathit{z}$ and ${\mathit{z}}^{\prime }$ run latent vectors of all the training images. Moreover, we set a bound for each element of the latent space vector so that $\mathit{x}$ and $\tilde{\mathit{x}}$ are in the probabilistic distribution of CXR images:

$$\begin{array}{ccc}\hfill z_k& \leftarrow & \mathrm{clip}\left(z_k,c_k-\frac{w_k}{2},c_k+\frac{w_k}{2}\right),\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill {\tilde{z}}_k& \leftarrow & \mathrm{clip}\left({\tilde{z}}_k,c_k-\frac{w_k}{2},c_k+\frac{w_k}{2}\right),\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill w_k& =& \alpha ·\left({\max }_{\mathit{z}}{z}_k-{\min }_{\mathit{z}}{z}_k\right),\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill c_k& =& \frac{\left({\max }_{\mathit{z}}{z}_k+{\min }_{\mathit{z}}{z}_k\right)}{2},\hfill \end{array}$$

(12)

where min and max operators, respectively, find the minimum and maximum values over all the latent space vectors $\mathit{z}$ for training images, the scalar function $\mathrm{clip}(x,a,b)$ bounds the value of x such that $a\le x\le b$, and the subscript k means the element number (runs all the elements of the latent space vector). We empirically set $\alpha =0.4$ in this study. An overly large $\alpha$ can cause significant collapse of the input images, while an excessively small $\alpha$ tends to overly normalize them. Now we can generate a differentially private image $\tilde{\mathit{x}}$ for the budget $ϵ$,

$$\begin{array}{c}\hfill \tilde{\mathit{x}}={\mathit{G}}_{\mathit{\theta }}\left(\tilde{\mathit{z}}\right).\end{array}$$

(13)

We prepare the vector function ${\mathit{G}}_{\mathit{\theta }}^{-1}$ by training one of the flow-based DGMs, i.e., GLOW, with many CXR images prior to executing this locally differential private algorithm. We summarize our DP-GLOW algorithm to generate $ϵ$-LDP images below:

• Train GLOW (maximize the average logarithm likelihood) with many CXR images to obtain ${\mathit{G}}_{\mathit{\theta }}^{-1}$.
• Set the privacy budget $ϵ$.
• Compute sensitivity from the training CXR images following Equation (8).
• Compute image-dependent clipping parameters following Equations (11) and (12) from the training CXR images.
• Map an image $\mathit{x}$ from the test CXR images set onto the latent vector $\mathit{z}$.
• Clip $\mathit{z}$ following Equation (9).
• Add noise following Equation (6) to obtain $\tilde{\mathit{z}}$.
• Clip $\tilde{\mathit{z}}$ following Equation (10).
• Map the clipped latent vector $\tilde{\mathit{z}}$ onto the CXR image space by ${\mathit{G}}_{\mathit{\theta }}$ to obtain a $ϵ$-LDP CXR image.

In Figure 1, we visually illustrate the DP-GLOW algorithm to clarify the step where we introduce noise using the Laplace mechanism.

Figure 1. Overview of the DP-GLOW algorithm.

Finally, we prove that our framework indeed satisfies $ϵ$-LDP. We can obtain the following equations using the change of variable formula:

$$\begin{array}{ccc}\hfill logp\left(\tilde{\mathit{x}}\right|\mathit{x})& =& logp(\tilde{\mathit{x}},\mathit{x})-logp\left(\mathit{x}\right)\hfill \\ & =& logp(\tilde{\mathit{z}},\mathit{z})+log\left|det\left(\frac{\partial (\tilde{\mathit{z}},\mathit{z})}{\partial (\tilde{\mathit{x}},\mathit{x})}\right)\right|\hfill \end{array}$$

(14)

$$\begin{array}{ccc}& -& logp\left(\mathit{z}\right)-log\left|det\left(\frac{\partial \mathit{z}}{\partial \mathit{x}}\right)\right|\hfill \\ & =& logp\left(\tilde{\mathit{z}}\right|\mathit{z})+log\left|det\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}\right|\hfill \end{array}$$

(15)

$$\begin{array}{ccc}& +& log\left|det\frac{\partial \mathit{z}}{\partial \mathit{x}}\right|-log\left|det\frac{\partial \mathit{z}}{\partial \mathit{x}}\right|\hfill \end{array}$$

(16)

$$\begin{array}{ccc}& =& logp\left(\tilde{\mathit{z}}\right|\mathit{z})+log\left|det\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}\right|,\hfill \end{array}$$

(17)

where $p\left(\mathit{x}\right)$ is the same as in Equation (1), and we used the following modification:

$$\begin{array}{ccc}\hfill log\left|det\left(\frac{\partial (\tilde{\mathit{z}},\mathit{z})}{\partial (\tilde{\mathit{x}},\mathit{x})}\right)\right|& =& log\left|\begin{array}{cc}\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}& \frac{\partial \mathit{z}}{\partial \tilde{\mathit{x}}}\\ \frac{\partial \tilde{\mathit{z}}}{\partial \mathit{x}}& \frac{\partial \mathit{z}}{\partial \mathit{x}}\end{array}\right|\hfill \end{array}$$

(18)

$$\begin{array}{ccc}& =& log\left|\begin{array}{cc}\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}& \frac{\partial \mathit{z}}{\partial \tilde{\mathit{x}}}\\ \mathbf{0}& \frac{\partial \mathit{z}}{\partial \mathit{x}}\end{array}\right|\hfill \end{array}$$

(19)

$$\begin{array}{ccc}& =& log\left|det\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}\right|+log\left|det\frac{\partial \mathit{z}}{\partial \mathit{x}}\right|.\hfill \end{array}$$

(20)

We have $\frac{\partial \tilde{\mathit{z}}}{\partial \mathit{x}}=\mathbf{0}$ because vector $\mathit{z}$ is fixed and there is no correlation between probabilistic vector $\mathcal{N}$ and any image vector $\mathit{x}$.

Therefore, we have:

$$\begin{array}{c}\hfill logp\left(\tilde{\mathit{x}}\right|\mathit{x})=logp\left(\tilde{\mathit{z}}\right|\mathit{z})+log\left|det\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}\right|,\end{array}$$

(21)

and

$$\begin{array}{c}\hfill logp\left(\tilde{\mathit{x}}\right|{\mathit{x}}^{\prime })=logp\left(\tilde{\mathit{z}}\right|{\mathit{z}}^{\prime })+log\left|det\frac{\partial \tilde{\mathit{z}}}{\partial \tilde{\mathit{x}}}\right|.\end{array}$$

(22)

Substituting those equations, we have:

$$\begin{array}{ccc}\hfill logp\left(\tilde{\mathit{x}}\right|\mathit{x})-logp\left(\tilde{\mathit{x}}\right|{\mathit{x}}^{\prime })& =& logp\left(\tilde{\mathit{z}}\right|\mathit{z})-logp\left(\tilde{\mathit{z}}\right|{\mathit{z}}^{\prime }).\hfill \end{array}$$

(23)

Combined with the Laplace mechanism, we can ensure:

$$\begin{array}{ccc}\hfill logp\left(\tilde{\mathit{x}}\right|\mathit{x})-logp\left(\tilde{\mathit{x}}\right|{\mathit{x}}^{\prime })& =& logp\left(\tilde{\mathit{z}}\right|\mathit{z})-logp\left(\tilde{\mathit{z}}\right|{\mathit{z}}^{\prime })\hfill \\ & =& \sum _k^{H·W}logp\left({\tilde{z}}_k\right|z_k)\hfill \end{array}$$

(24)

$$\begin{array}{ccc}& -& \sum _k^{H·W}logp\left({\tilde{z}}_k\right|z_k^{\prime })\hfill \\ & =& \sum _k^{H·W}-\frac{|{\tilde{z}}_k-z_k|}{\frac{H·W·\Delta z_k}{ϵ}}\hfill \end{array}$$

(25)

$$\begin{array}{ccc}& +& \frac{|{\tilde{z}}_k-z_k^{\prime }|}{\frac{H·W·\Delta z_k}{ϵ}}\hfill \end{array}$$

(26)

$$\begin{array}{ccc}& \le & \frac{ϵ}{H·W}·\sum _k^{H·W}\frac{|-z_k^{\prime }+z_k|}{\Delta z_k}\hfill \end{array}$$

(27)

$$\begin{array}{ccc}& \le & ϵ,\hfill \end{array}$$

(28)

where k runs each element of vectors.

3.4. Hyperparameters

We show the hyperparameters for the training of CXR image sets ${\mathcal{S}}_{\mathrm{normal}}^{\mathrm{train}}$ and ${\mathcal{S}}_{\mathrm{mixture}}^{\mathrm{train}}$ in Table 2. The ‘learn-top’ option determines whether we train the means and variances of the latent space in GLOW. The minibatch size refers to the number of images trained simultaneously during batch learning. To train GLOW, we used Tensorflow 1.12.0 [17]. The versions of CUDA and cuDNN were 9.0 and 7.4, respectively. We carried out all processes in one computing node of the Reedbush-L supercomputer system, Rackable C1102-GP8, SGI, Mountain View, CA, USA, in the Information Technology Center, The University of Tokyo. The system consists of 64 computing nodes equipped with two Xeon E5-2695v4 processors, Intel, Santa Clara, CA, USA, 256 GB memory, and four GPUs (Tesla P100 SXM2 with 16 GB memory, NVIDIA, Santa Clara, CA, USA).

Table 2. Hyperparameters for GLOW.

Coupling Layer	Affine
Learn-top option	False
Flow permutation	1 × 1 convolution
Minibatch size	4
Number of training samples per epoch	50,000
Network levels	7
Depth per level	32
Image size (in pixel)	H512 × W512 × C1
Total epochs	200
Learning rate in steady state	${10}^{-3}$

To obtain $ϵ$-LDP-CXR images and to detect pneumonia in CXR images, we used Tensorflow 1.15.5. The versions of CUDA and cuDNN were 9.0 and 8.1, respectively. We carried out all processes in one computing node of the Wisteria/B-DEC01 supercomputer system, PRIMERGY GX2570 M6, FUJITSU, Tokyo, Japan in the Information Technology Center, The University of Tokyo. The system (Wisteria-Aquarius) consists of 45 computing nodes equipped with two Xeon 8360Y processors, Intel, and eight GPUs (A100 with 40 GB memory, NVIDIA).

4. Results

4.1. $ϵ$-LDP-Processed CXR Images

In Figure 2, we show $ϵ$-LDP-processed CXR images of four clinical cases obtained with the image domain LDP, which directly imposes the Laplace mechanism on the input image, with different privacy budgets together with the original images. Figure 3 shows four $ϵ$-LDP-processed CXR images of clinical cases obtained with DP-GLOW and different privacy budgets together with the original images. In case 1 for DP-GLOW, there is decreased permeability in the bilateral hilar regions. Although this hilar opacity tends to be preserved with a larger privacy budget, the entire image is degraded when the privacy budget becomes ${10}^1·H·W$. A similar tendency is observed in the images of all the four cases for DP-GLOW; for example, in case 4 with $ϵ={10}^1·H·W$, the lung opacity suggesting pneumonia in the right lower lung field is well preserved, while the entire image is degraded.

Figure 2. $ϵ$-LDP-processed CXR images (we applied the Laplace mechanism in the image domain). (a) Original, case 1; (b) $ϵ={10}^3·H·W$, case 1; (c) $ϵ={10}^2·H·W$, case 1; (d) $ϵ={10}^1·H·W$, case 1; (e) Original, case 2; (f) $ϵ={10}^3·H·W$, case 2; (g) $ϵ={10}^2·H·W$, case 2; (h) $ϵ={10}^1·H·W$, case 2; (i) Original, case 3; (j) $ϵ={10}^3·H·W$, case 3; (k) $ϵ={10}^2·H·W$, case 3; (l) $ϵ={10}^1·H·W$, case 3; (m) Original, case 4; (n) $ϵ={10}^3·H·W$, case 4; (o) $ϵ={10}^2·H·W$, case 4; (p) $ϵ={10}^1·H·W$, case 4.

Figure 3. $ϵ$-LDP-processed CXR images obtained with DP-GLOW. (a) Original, case 1; (b) $ϵ={10}^3·H·W$, case 1; (c) $ϵ={10}^2·H·W$, case 1; (d) $ϵ={10}^1·H·W$, case 1; (e) Original, case 2; (f) $ϵ={10}^3·H·W$, case 2; (g) $ϵ={10}^2·H·W$, case 2; (h) $ϵ={10}^1·H·W$,case 2; (i) Original, case 3; (j) $ϵ={10}^3·H·W$, case 3; (k) $ϵ={10}^2·H·W$, case 3; (l) $ϵ={10}^1·H·W$, case 3; (m) Original, case 4; (n) $ϵ={10}^3·H·W$, case 4; (o) $ϵ={10}^2·H·W$, case 4; (p) $ϵ={10}^1·H·W$, case 4.

4.2. Qualitative Assessment of LDP-Processed CXR Images

Here, we assume two possible privacy leakage scenarios. To CXR images, we intentionally add features that can lead to the re-identification of the subject appearing in a CXR image. The first feature is an artificial block marker. The second feature is a rare anatomical abnormality known as situs inversus, simulated by flipping a CXR image along the vertical axis. Figure 4a,c show CXR images with the artificial block marker. Figure 5a shows a flipped CXR image to represent a case of situs inversus. We applied DP-GLOW to these CXR images. In Figure 4b,d, the image domain LDP fails to obfuscate the artificial block marker with a moderate privacy budget. In contrast, in Figure 6b,d, DP-GLOW successfully obfuscated the artificial block marker with the moderate privacy budget. On the other hand, the anatomical shape of the chest and the abnormal opacity (hilar regions in the case 1) are preserved. In Figure 5b, we observed that the right edge of the heart does not become obfuscated with the image domain LDP. In contrast, in Figure 7b, we observed that the right edge of the heart becomes obfuscated and the heart appears at the center of the thoracic cage with DP-GLOW. However, DP-GLOW with this privacy budget is insufficient to almost completely erase the feature of situs inversus.

Figure 4. Block obfuscation with the image domain LDP. (a) Original, case 1 with a block. (b) $ϵ={10}^2·H·W$, case 1. (c) Original, case 2 with a block. (d) $ϵ={10}^2·H·W$, case 2.

Figure 5. Flip obfuscation with the image domain LDP. (a) Original, case 5 (simulated situs inversus). (b) $ϵ={10}^2·H·W$, case 5.

Figure 6. Block obfuscation with DP-GLOW. (a) Original, case 1 with a block. (b) $ϵ={10}^2·H·W$, case 1. (c) Original, case 2 with a block. (d) $ϵ={10}^2·H·W$, case 2.

Figure 7. Flip obfuscation with DP-GLOW. (a) Original, case 5 (simulated situs inversus). (b) $ϵ={10}^2·H·W$, case 5.

4.3. Pneumonia Detection in $ϵ$-LDP-Processed CXR Images

Table 3 shows the area under the curve (AUC) with different privacy budgets for $ϵ$-LDP-processed CXR images obtained with the image domain LDP and DP-GLOW. For details of AUC computation, see Appendix A.

Table 3. AUC for pneumonia detection (ID-LDP: Image Domain LDP).

$\mathit{ϵ}$	AUC (ID-LDP)	AUC (DP-GLOW)
∞ (without clip)	0.807	0.807
${10}^3·H·W$	0.813	0.679
${10}^2·H·W$	0.559	0.665
${10}^1·H·W$	0.643	0.539

5. Discussion

We showed an algorithm (DP-GLOW) for generating useful images for diagnosis and medical AI, while $ϵ$-LDP is guaranteed against any image in the training distribution. Furthermore, this is the first study to apply an $ϵ$-LDP algorithm against a medical image itself. Additionally, we validated the usefulness of the $ϵ$-LDP CXR images generated by AI for pneumonia detection: we showed the AUC as a function of the privacy budget. Finally, this is the first work to adopt flow-based DGMs for LDP processing.

For DP-GLOW, the AUCs for pneumonia detection significantly change from 0.539 to 0.807, while the privacy budget varies from ${10}^1·H·W(=2,621,440)$ to ∞. This means that this range of the privacy budget is indeed meaningful, whereas the privacy budget is very large compared with usual values of $ϵ$-LDP for scalar quantities. This finding implies that we must normalize the privacy budget so that we can consistently handle $ϵ$-LDP for vector quantities. To normalize the budget, we compute the privacy budgets per image pixel. To this end, we intentionally indicated the privacy budget to have a common factor $H·W$. Therefore, the actual privacy budgets per image pixel in this study are from ${10}^1$ to ∞, which are not much larger than commonly accepted privacy budgets.

Most of the approximate forms in CXR images are preserved and privacy is not protected with the image domain LDP. On the other hand, given a low privacy budget, DP-GLOW deforms the image so much that individuals cannot be identified. However, the AUCs for pneumonia detection are similar with the low privacy budget between DP-GLOW and the image domain LDP.

This study has several future directions. First, we adopted CXR images but one can adopt other kinds of image including nonmedical images. Second, we can readily extend this method to three-dimensional (3D) flow-based DGMs to generate $ϵ$-LDP 3D images, such as CT (Computed Tomography) and MR (Magnetic Resonance) images. Third, we can use other flow-based DGMs different from GLOW. Fourth, although we have formulated our LDP algorithm using flow-based deep generative models, we are now attempting to adapt it for denoising diffusion probabilistic models [18].

This study has a limitation—we must train GLOW with many medical images to generate $ϵ$-LDP-processed medical images using DP-GLOW. The training is very difficult when medical images of interest are not available. However, once the GLOW is trained with the medical images of interest, we can distribute the GLOW model to further release medical images of the same kind to the public using DP-GLOW. Empirically, we know that at least 1000 images are needed to train the GLOW model satisfactorily. However, the number of images required may vary if we use a different flow-based deep generative model. Additionally, due to memory limitations, it is expected that we cannot process large-sized 3D images. Furthermore, DP-GLOW cannot handle medical images with varying resolutions.

6. Conclusions

We proposed DP-GLOW, the $ϵ$-LDP algorithm for images built upon the flow-based DGMs, which can simultaneously ensure provable privacy protection and usefulness, e.g., the preservation of pathologies, with a controllable privacy budget. The $ϵ$-LDP-processed CXR images obtained with DP-GLOW indicate that we have obtained a powerful tool to release and use medical images for training AI. Furthermore, DP-GLOW could benefit other areas such as images obtained from surveillance cameras and those uploaded to social networking services.