# GDP vs. LDP: A Survey from the Perspective of Information-Theoretic Channel

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Preliminaries

#### 2.1. GDP and LDP

**Definition**

**1**(GDP)

**.**

**Definition**

**2**(LDP)

**.**

#### 2.2. Information-Theoretic Channel and Metrics

## 3. Privacy Threat Model on Information-Theoretic Channel

## 4. Information-Theoretic Channel Models and Definitions of GDP and LDP

#### 4.1. Information-Theoretic Channel Models of GDP and LDP

#### 4.2. Information-Theoretic Definitions of GDP and LDP

## 5. Privacy-Utility Metrics of GDP and LDP under Information-Theoretic Channel Models

## 6. Properties of GDP and LDP under Information-Theoretic Channel Models

## 7. GDP and LDP Mechanisms under Information-Theoretic Channel Models

## 8. Differential Privacy Synthetic Data Generation

#### 8.1. Differential Privacy Synthetic Data Generation with Generative Adversarial Network

#### 8.2. Differential Privacy Synthetic Data Generation with Federated Learning

## 9. Open Problems

## 10. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Dwork, C.; McSherry, F.; Nissim, K.; Smith, A. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Theory of Cryptography Conference, New York, NY, USA, 4–7 March 2006; pp. 265–284. [Google Scholar]
- Kasiviswanathan, S.P.; Lee, H.K.; Nissim, K.; Raskhodnikova, S.; Smith, A. What can we learn privately? SIAM J. Comput.
**2011**, 40, 793–826. [Google Scholar] [CrossRef] - Liu, H.; Wu, Z.; Peng, C.; Tian, F.; Lu, L. Bounded privacy-utility monotonicity indicating bounded tradeoff of differential privacy mechanisms. Theor. Comput. Sci.
**2020**, 816, 195–220. [Google Scholar] [CrossRef] - Dobrota, B. Measuring the Quantity of Data Privacy and Utility Tradeoff for Users’ Data: A Visualization Approach. Master Thesis, Utrecht University, Utrecht, The Netherlands, 2021. [Google Scholar]
- Kairouz, P.; Oh, S.; Viswanath, P. Extremal mechanisms for local differential privacy. J. Mach. Learn. Res.
**2016**, 4, 492–542. [Google Scholar] - Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; John Wiley & Sons Inc.: Hoboken, NJ, USA, 2006. [Google Scholar]
- Alvim, M.S.; Andrés, M.E.; Chatzikokolakis, K.; Degano, P.; Palamidessi, C. Differential privacy: On the trade-off between utility and information leakage. In Proceedings of the 8th International Workshop on Formal Aspects of Security and Trust, Leuven, Belgium, 12–14 September 2011; pp. 39–54. [Google Scholar]
- Barthe, G.; Olmedo, F. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In Proceedings of the 40th International Colloquium Automata, Languages, and Programming, Riga, Latvia, 8–12 July 2013; pp. 49–60. [Google Scholar]
- Dwork, C.; Roth, A. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci.
**2014**, 9, 211–407. [Google Scholar] [CrossRef] - Fehr, S.; Berens, S. On the conditional Rényi entropy. IEEE Trans. Inf. Theory
**2014**, 60, 6801–6810. [Google Scholar] [CrossRef] - Mironov, I. Rényi differential privacy. In Proceedings of the 30th IEEE Computer Security Foundations Symposium, Santa Barbara, CA, USA, 21–25 August 2017; pp. 263–275. [Google Scholar]
- Wang, W.; Ying, L.; Zhang, J. On the relation between identifiability, differential privacy, and mutual-information privacy. IEEE Trans. Inf. Theory
**2016**, 62, 5018–5029. [Google Scholar] [CrossRef] [Green Version] - Cuff, P.; Yu, L. Differential privacy as a mutual information constraint. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 24–28 October 2016; pp. 43–54. [Google Scholar]
- Mir, D.J. Information-theoretic foundations of differential privacy. In Proceedings of the 5th International Symposium on Foundations and Practice of Security, Montreal, QC, Canada, 25–26 October 2012; pp. 374–381. [Google Scholar]
- Smith, G. On the foundations of quantitative information flow. In Proceedings of the 12th International Conference on Foundations of Software Science and Computational Structures, York, UK, 22–29 March 2009; pp. 288–302. [Google Scholar]
- Barthe, G.; Köpf, B. Information-theoretic bounds for differentially private mechanisms. In Proceedings of the 24th IEEE Computer Security Foundations Symposium, Cernay-la-Ville, France, 27–29 June 2011; pp. 191–204. [Google Scholar]
- Duchi, J.C.; Jordan, M.I.; Wainwright, M.J. Local privacy and statistical minimax rates. In Proceedings of the 54th Annual IEEE Symposium on Foundations of Computer Science, Berkeley, CA, USA, 26–29 October 2013; pp. 429–438. [Google Scholar]
- Duchi, J.C.; Jordan, M.I.; Wainwright, M.J. Privacy aware learning. J. ACM
**2014**, 61, 38:1–38:57. [Google Scholar] [CrossRef] - Jiang, B.; Li, M.; Tandon, R. Context-aware data aggregation with localized information privacy. In Proceedings of the 6th IEEE Conference on Communications and Network Security, Beijing, China, 30 May–1 June 2018; pp. 1–9. [Google Scholar]
- Song, H.; Luo, T.; Li, J. Common criterion of privacy metrics and parameters analysis based on error probability for randomized response. IEEE Access
**2019**, 7, 16964–16978. [Google Scholar] [CrossRef] - Lopuhaä-Zwakenberg, M.; Tong, H.; Skoric, B. Data sanitisation protocols for the privacy funnel with differential privacy guarantees. arXiv
**2020**, arXiv:2008.13151. [Google Scholar] - Lopuhaä-Zwakenberg, M.; Goseling, J. The privacy-utility tradeoff of robust local differential privacy. arXiv
**2021**, arXiv:2101.09139. [Google Scholar] - Bun, M.; Steinke, T. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Proceedings of the 14th Theory of Cryptography Conference, Beijing, China, 31 October–3 November 2016; pp. 635–658. [Google Scholar]
- Asoodeh, S.; Liao, J.; Calmon, F.P.; Kosut, O.; Sankar, L. Three variants of differential privacy: Lossless conversion and applications. IEEE J. Sel. Areas Inf. Theory
**2021**, 2, 208–222. [Google Scholar] [CrossRef] - Chaudhuri, K.; Imola, J.; Machanavajjhala, A. Capacity bounded differential privacy. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; pp. 3469–3478. [Google Scholar]
- Calmon, F.P.; Fawaz, N. Privacy against statistical inference. In Proceedings of the 50th Annual Allerton Conference on Communication, Control, and Computing, Allerton Park & Retreat Center, Monticello, IL, USA, 1–5 October 2012; pp. 1401–1408. [Google Scholar]
- Makhdoumi, A.; Fawaz, N. Privacy-utility tradeoff under statistical uncertainty. In Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing, Allerton Park & Retreat Center, Monticello, IL, USA, 2–4 October 2013; pp. 1627–1634. [Google Scholar]
- Holohan, N.; Leith, D.J.; Mason, O. Extreme points of the local differential privacy polytope. Linear Algebra Appl.
**2017**, 534, 78–96. [Google Scholar] [CrossRef] [Green Version] - De, A. Lower bounds in differential privacy. In Proceedings of the 9th Theory of Cryptography Conference, Taormina, Sicily, Italy, 19–21 March 2012; pp. 321–338. [Google Scholar]
- Edwards, T.; Rubinstein, B.I.P.; Zhang, Z.; Zhou, S. A graph symmetrization bound on channel information leakage under Blowfish privacy. IEEE Trans. Inf. Theory
**2022**, 68, 538–548. [Google Scholar] [CrossRef] - Rogers, R.M.; Roth, A.; Smith, A.D.; Thakkar, O. Max-information, differential privacy, and post-selection hypothesis testing. In Proceedings of the 57th Annual Symposium on Foundations of Computer Science, Hyatt Regency, New Brunswick, NJ, USA, 9–11 October 2016; pp. 487–494. [Google Scholar]
- Padakandla, A.; Kumar, P.R.; Szpankowski, W. The trade-off between privacy and fidelity via Ehrhart theory. IEEE Trans. Inf. Theory
**2020**, 66, 2549–2569. [Google Scholar] [CrossRef] - Ayed, F.; Battiston, M.; Camerlenghi, F. An information theoretic approach to post randomization methods under differential privacy. Stat. Comput.
**2020**, 30, 1347–1361. [Google Scholar] [CrossRef] - Sarwate, A.D.; Sankar, L. A rate-disortion perspective on local differential privacy. In Proceedings of the 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton Park & Retreat Center, Monticello, IL, USA, 30 September–3 October 2014; pp. 903–908. [Google Scholar]
- Kalantari, K.; Sankar, L.; Sarwate, A.D. Robust privacy-utility tradeoffs under differential privacy and Hamming distortion. IEEE Trans. Inf. Forensics Secur.
**2018**, 13, 2816–2830. [Google Scholar] [CrossRef] [Green Version] - Xiong, S.; Sarwate, A.D.; Mandayam, N.B. Randomized requantization with local differential privacy. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Shanghai, China, 20–25 March 2016; pp. 2189–2193. [Google Scholar]
- Kalantari, K.; Sankar, L.; Sarwate, A.D. Optimal differential privacy mechanisms under Hamming distortion for structured source classes. In Proceedings of the IEEE International Symposium on Information Theory, Barcelona, Spain, 10–15 July 2016; pp. 2069–2073. [Google Scholar]
- Wang, S.; Huang, L.; Nie, Y.; Zhang, X.; Wang, P.; Xu, H.; Yang, W. Local differential private data aggregation for discrete distribution estimation. IEEE Trans. Parallel Distrib. Syst.
**2019**, 30, 2046–2059. [Google Scholar] [CrossRef] - Lopuhaä-Zwakenberg, M.; Skoric, B.; Li, N. Information-theoretic metrics for local differential privacy protocols. arXiv
**2019**, arXiv:1910.07826. [Google Scholar] - McSherry, F.; Talwar, K. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science, Providence, RI, USA, 20–23 October 2007; pp. 94–103. [Google Scholar]
- Duchi, J.C.; Wainwright, M.J.; Jordan, M.I. Local privacy and minimax bounds: Sharp rates for probability estimation. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–8 December 2013; pp. 1529–1537. [Google Scholar]
- Holohan, N.; Leith, D.J.; Mason, O. Optimal differentially private mechanisms for randomised response. IEEE Trans. Inf. Forensics Secur.
**2017**, 12, 2726–2735. [Google Scholar] [CrossRef] [Green Version] - Erlingsson, Ú.; Pihur, V.; Korolova, A. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 21st ACM SIGSAC Conference on Computer and Communications Security, Scottsdale, AZ, USA, 3–7 November 2014; pp. 1054–1067. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.C.; Bengio, Y. Generative adversarial nets. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 2672–2680. [Google Scholar]
- Frigerio, L.; de Oliveira, A.S.; Gomez, L.; Duverger, P. Differentially private generative adversarial networks for time series, continuous, and discrete open data. In Proceedings of the 34th IFIP TC 11 International Conference on ICT Systems Security and Privacy Protection, Lisbon, Portugal, 25–27 June 2019; pp. 151–164. [Google Scholar]
- Mukherjee, S.; Xu, Y.; Trivedi, A.; Patowary, N.; Ferres, J.L. privGAN: Protecting GANs from membership inference attacks at low cost to utility. Proc. Priv. Enhancing Technol.
**2021**, 3, 142–163. [Google Scholar] [CrossRef] - Hayes, J.; Melis, L.; Danezis, G.; De Cristofaro, E. LOGAN: Membership inference attacks against generative models. Proc. Priv. Enhancing Technol.
**2019**, 1, 133–152. [Google Scholar] [CrossRef] [Green Version] - Liu, K.S.; Xiao, C.; Li, B.; Gao, J. Performing co-membership attacks against deep generative models. In Proceedings of the 19th IEEE International Conference on Data Mining, Beijing, China, 8–11 November 2019; pp. 459–467. [Google Scholar]
- Hilprecht, B.; Härterich, M.; Bernau, D. Monte Carlo and reconstruction membership inference attacks against generative models. Proc. Priv. Enhancing Technol.
**2019**, 4, 232–249. [Google Scholar] [CrossRef] [Green Version] - Chen, D.; Yu, N.; Zhang, Y.; Fritz, M. GAN-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 27th ACM SIGSAC Conference on Computer and Communications Security, Virtual Event USA, 9–13 November 2020; pp. 343–362. [Google Scholar]
- Hu, H.; Pang, J. Stealing machine learning models: Attacks and countermeasures for generative adversarial networks. In Proceedings of the 37th Annual Computer Security Applications Conference, Virtual Event USA, 6–10 December 2021; pp. 1–16. [Google Scholar]
- Abadi, M.; Chu, A.; Goodfellow, I.J.; McMahan, H.B.; Mironov, I.; Talwar, K.; Zhang, L. Deep learning with differential privacy. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 24–28 October 2016; pp. 308–318. [Google Scholar]
- Fan, L. A survey of differentially private generative adversarial networks. In Proceedings of the 1st AAAI Workshop on Privacy-Preserving Artificial Intelligence, New York, NY, USA, 7 February 2020. [Google Scholar]
- Cai, Z.; Xiong, Z.; Xu, H.; Wang, P.; Li, W.; Pan, Y. Generative adversarial networks: A survey toward private and secure applications. ACM Comput. Surv.
**2021**, 54, 132:1–132:38. [Google Scholar] [CrossRef] - Lu, P.-H.; Yu, C.-M. POSTER: A unified framework of differentially private synthetic data release with generative adversarial network. In Proceedings of the 24th ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA, 30 October–3 November 2017; pp. 2547–2549. [Google Scholar]
- Xie, L.; Lin, K.; Wang, S.; Wang, F.; Zhou, J. Differentially private generative adversarial network. arXiv
**2018**, arXiv:1802.06739. [Google Scholar] - Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein GAN. arXiv
**2017**, arXiv:1701.07875. [Google Scholar] - Torkzadehmahani, R.; Kairouz, P.; Paten, B. DP-CGAN: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 16–20 June 2019; pp. 98–104. [Google Scholar]
- Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv
**2014**, arXiv:1411.1784. [Google Scholar] - Liu, Y.; Peng, J.; Yu, J.J.Q.; Wu, Y. PPGAN: Privacy-preserving generative adversarial network. In Proceedings of the 25th IEEE International Conference on Parallel and Distributed Systems, Tianjin, China, 4–6 December 2019; pp. 985–989. [Google Scholar]
- Ha, T.; Dang, T.K. Investigating local differential privacy and generative adversarial network in collecting data. In Proceedings of the 14th International Conference on Advanced Computing and Applications, Quy Nhon, Vietnam, 25–27 November 2020; pp. 140–145. [Google Scholar]
- Chen, D.; Orekondy, T.; Fritz, M. GS-WGAN: A gradient-sanitized approach for learning differentially private generators. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems, virtual, 6–12 December 2020; pp. 12673–12684. [Google Scholar]
- Yang, R.; Ma, X.; Bai, X.; Su, X. Differential privacy images protection based on generative adversarial network. In Proceedings of the 19th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, Guangzhou, China, 29 December 2020–1 January 2021; pp. 1688–1695. [Google Scholar]
- Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A.C. Improved training of Wasserstein GANs. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5767–5777. [Google Scholar]
- Beaulieu-Jones, B.K.; Wu, Z.S.; Williams, C.; Lee, R.; Bhavnani, S.P.; Byrd, J.B.; Greene, C.S. Privacy-preserving generative deep neural networks support clinical data sharing. Circ. Cardiovasc. Qual.
**2019**, 12, e005122:1–e005122:10. [Google Scholar] [CrossRef] - Odena, A.; Olah, C.; Shlens, J. Conditional image synthesis with auxiliary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 6–11 August 2017; pp. 2642–2651. [Google Scholar]
- Fan, L.; Pokkunuru, A. DPNeT: Differentially private network traffic synthesis with generative adversarial networks. In Proceedings of the 35th Annual IFIP WG 11.3 Conference on Data and Applications Security and Privacy, Calgary, AB, Canada, 19–20 July 2021; pp. 3–21. [Google Scholar]
- Zhang, S.; Ni, W.; Fu, N. Differentially private graph publishing with degree distribution preservation. Comput. Secur.
**2021**, 106, 102285:1–102285:17. [Google Scholar] [CrossRef] - Bojchevski, A.; Shchur, O.; Zügner, D.; Günnemann, S. NetGAN: Generating graphs via random walks. In Proceedings of the 35th International Conference on Machine Learning, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018; pp. 609–618. [Google Scholar]
- Li, A.; Fang, J.; Jiang, Q.; Zhou, B.; Jia, Y. A graph data privacy-preserving method based on generative adversarial networks. In Proceedings of the 21st International Conference on Web Information Systems Engineering, Amsterdam, The Netherlands, 20–24 October 2020; pp. 227–239. [Google Scholar]
- Neunhoeffer, M.; Wu, S.; Dwork, C. Private post-GAN boosting. In Proceedings of the 9th International Conference on Learning Representations, Virtual Event Austria, 3–7 May 2021. [Google Scholar]
- Hardt, M.; Rothblum, G.N. A multiplicative weights mechanism for privacy-preserving data analysis. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science, Las Vegas, NV, USA, 23–26 October 2010; pp. 61–70. [Google Scholar]
- Indhumathi, R.; Devi, S.S. Healthcare Cramér generative adversarial network (HCGAN). Distrib. Parallel. Dat.
**2021**, 39, 1–17. [Google Scholar] [CrossRef] - Imtiaz, S.; Arsalan, M.; Vlassov, V.; Sadre, R. Synthetic and private smart health care data generation using GANs. In Proceedings of the 30th International Conference on Computer Communications and Networks, Athens, Greece, 19–22 July 2021; pp. 1–7. [Google Scholar]
- Jordon, J.; Yoon, J.; van der Schaar, M. PATE-GAN: Generating synthetic data with differential privacy guarantees. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA, 6–9 May 2019. [Google Scholar]
- Zhang, X.; Ding, J.; Errapotu, S.M.; Huang, X.; Li, P.; Pan, M. Differentially private functional mechanism for generative adversarial networks. In Proceedings of the IEEE Global Communications Conference, GLOBECOM 2019, Waikoloa, HI, USA, 9–13 December 2019; pp. 1–6. [Google Scholar]
- Ho, S.; Qu, Y.; Gu, B.; Gao, L.; Li, J.; Xiangy, Y. DP-GAN: Differentially private consecutive data publishing using generative adversarial nets. J. Netw. Comput. Appl.
**2021**, 185, 103066:1–103066:11. [Google Scholar] [CrossRef] - Papernot, N.; Abadi, M.; Erlingsson, Ú.; Goodfellow, I.J.; Talwar, K. Semi-supervised knowledge transfer for deep learning from private training data. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Li, C.; Xu, T.; Zhu, J.; Zhang, B. Triple generative adversarial nets. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4088–4098. [Google Scholar]
- McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1273–1282. [Google Scholar]
- Augenstein, S.; McMahan, H.B.; Ramage, D.; Ramaswamy, S.; Kairouz, P.; Chen, M.; Mathews, R.; Arcas, B.A. Generative models for effective ml on private, decentralized datasets. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 26–30 April 2020. [Google Scholar]
- Zhang, L.; Shen, B.; Barnawi, A.; Xi, S.; Kumar, N.; Wu, Y. FedDPGAN: Federated differentially private generative adversarial networks framework for the detection of COVID-19 pneumonia. Inf. Syst. Frontiersz
**2021**, 23, 1403–1415. [Google Scholar] [CrossRef] [PubMed] - Nguyen, D.C.; Ding, M.; Pathirana, P.N.; Seneviratne, A.; Zomaya, A.Y. Federated learning for COVID-19 detection with generative adversarial networks in edge cloud computing. IEEE Internet Things
**2021**. [Google Scholar] [CrossRef] - Xin, B.; Geng, Y.; Hu, T.; Chen, S.; Yang, W.; Wang, S.; Huang, L. Federated synthetic data generation with differential privacy. Neurocomputing
**2022**, 468, 1–10. [Google Scholar] [CrossRef] - Triastcyn, A.; Faltings, B. Federated generative privacy. IEEE Intell. Syst.
**2020**, 35, 50–57. [Google Scholar] [CrossRef] - Yang, G.; Wang, S.; Wang, H. Federated learning with personalized local differential privacy. In Proceedings of the 6th IEEE International Conference on Computer and Communication Systems, Chengdu, China, 23–26 April 2021; pp. 484–489. [Google Scholar]
- Murakami, T.; Kawamoto, Y. Utility-optimized local differential privacy mechanisms for distribution estimation. In Proceedings of the 28th USENIX Security Symposium, Santa Clara, CA, USA, 14–16 August 2019; pp. 1877–1894. [Google Scholar]
- Ren, X.; Yu, C.-M.; Yu, W.; Yang, S.; Yang, X.; McCann, J.A.; Yu, P.S. LoPub: High-dimensional crowdsourced data publication with local differential privacy. IEEE Trans. Inf. Forensics Secur.
**2018**, 13, 2151–2166. [Google Scholar] [CrossRef] [Green Version] - Wang, N.; Xiao, X.; Yang, Y.; Zhao, J.; Hui, S.C.; Shin, H.; Shin, J.; Yu, G. Collecting and analyzing multidimensional data with local differential privacy. In Proceedings of the 35th IEEE International Conference on Data Engineering, Macao, China, 8–11 April 2019; pp. 638–649. [Google Scholar]
- Ren, X.; Xu, J.; Yang, X.; Yang, S. Bayesian network-based high-dimensional crowdsourced data publication with local differential privacy. Sci. Sin. Inform.
**2019**, 49, 1586–1605. [Google Scholar] - Du, R.; Ye, Q.; Fu, Y.; Hu, H. Collecting high-dimensional and correlation-constrained data with local differential privacy. In Proceedings of the 18th Annual IEEE International Conference on Sensing, Communication, and Networking, Rome, Italy, 6–9 July 2021; pp. 1–9. [Google Scholar]
- Hemkumar, D.; Ravichandra, S.; Somayajulu, D.V.L.N. Impact of data correlation on privacy budget allocation in continuous publication of location statistics. Peer-to-Peer Netw. Appl.
**2021**, 14, 1650–1665. [Google Scholar] [CrossRef] - Cao, Y.; Yoshikawa, M.; Xiao, Y.; Xiong, L. Quantifying differential privacy in continuous data release under temporal correlations. IEEE Trans. Knowl. Data Eng.
**2019**, 31, 1281–1295. [Google Scholar] [CrossRef] [Green Version] - Jorgensen, Z.; Yu, T.; Cormode, G. Conservative or liberal? Personalized differential privacy. In Proceedings of the 31st IEEE International Conference on Data Engineering, Seoul, Korea, 13–17 April 2015; pp. 1023–1034. [Google Scholar]
- Chen, Y.; Machanavajjhala, A.; Hay, M.; Miklau, G. PeGaSus: Data-adaptive differentially private stream processing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA, 30 October–3 November 2017; pp. 1375–1388. [Google Scholar]
- Niu, B.; Chen, Y.; Wang, B.; Wang, Z.; Li, F.; Cao, J. AdaPDP: Adaptive personalized differential privacy. In Proceedings of the 40th IEEE Conference on Computer Communications, Vancouver, BC, Canada, 10–13 May 2021; pp. 1–10. [Google Scholar]
- Pathak, M.A.; Rane, S.; Raj, B. Multiparty differential privacy via aggregation of locally trained classifiers. In Proceedings of the 24th Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 6–9 December 2010; pp. 1876–1884. [Google Scholar]
- Wei, K.; Li, J.; Ding, M.; Ma, C.; Yang, H.H.; Farokhi, F.; Jin, S.; Quek, T.Q.S.; Poor, H.V. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans. Inf. Forensics Secur.
**2020**, 15, 3454–3469. [Google Scholar] [CrossRef] [Green Version] - Sun, L.; Qian, J.; Chen, X. LDP-FL: Practical private aggregation in federated learning with local differential privacy. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, Montréal, QC, Canada, 19–20 August 2021; pp. 1571–1578. [Google Scholar]
- Wu, S.; Yu, M.; Ahmed, M.A.M.; Qian, Y.; Tao, Y. FL-MAC-RDP: Federated learning over multiple access channels with renyi differential privacy. Int. J. Theor. Phys.
**2021**, 60, 2668–2682. [Google Scholar] [CrossRef] - Niu, C.; Zheng, Z.; Wu, F.; Tang, S.; Gao, X.; Chen, G. Unlocking the value of privacy: Trading aggregate statistics over private correlated data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, London, UK, 19–23 August 2018; pp. 2031–2040. [Google Scholar]
- Gu, X.; Li, M.; Cheng, Y.; Xiong, L.; Cao, Y. PCKV: Locally differentially private correlated key-value data collection with optimized utility. In Proceedings of the 29th USENIX Security Symposium, Boston, MA, USA, 12–14 August 2020; pp. 967–984. [Google Scholar]
- Cheng, X.; Tang, P.; Su, S.; Chen, R.; Wu, Z.; Zhu, B. Multi-party high-dimensional data publishing under differential privacy. IEEE Trans. Knowl. Data Eng.
**2020**, 32, 1557–1571. [Google Scholar] [CrossRef] - Zheng, X.; Cai, Z. Privacy-preserved data sharing towards multiple parties in industrial IoTs. IEEE J. Sel. Areas Commun.
**2020**, 38, 968–979. [Google Scholar] [CrossRef]

Privacy Type | Advantage | Disadvantage |
---|---|---|

GDP | Better data utility | Needing trusted data collector |

Suitable for dataset of any scale | ||

LDP | Without needing trusted data collector | Poor data utility |

Not applicable to small scale dataset |

Symbol | Description |
---|---|

x | Dataset |

$\mathcal{M}$ | Randomized mechanism |

$\epsilon $ | Privacy budget |

$\delta $ | Probability without satisfying differential privacy |

X | Input random variable of information-theoretic channel |

Y | Output random variable of information-theoretic channel |

$p\left(y\right|x)$ | Channel transition probability matrix |

$p\left(x\right)$ | Probability distribution on source X |

$q\left(x\right)$ | Another probability distribution on source X |

${D}_{\alpha}\left(p\left(x\right)\right|\left|q\left(x\right)\right)$ | Rényi divergence |

${H}_{\alpha}\left(X\right)$ | Rényi entropy |

$H\left(X\right)$ | Shannon entropy |

${H}_{\infty}\left(X\right)$ | Min-entropy |

${H}_{\alpha}\left(X\right|Y)$ | Conditional Rényi entropy |

$H\left(X\right|Y)$ | Conditional Shannon entropy |

${H}_{\infty}\left(X\right|Y)$ | Conditional min-entropy |

$I(X;Y)$ | Mutual information |

${I}_{\infty}(X;Y)$ | Max-information |

${I}_{\infty}^{\beta}(X;Y)$ | $\beta $-approximate max-information |

${D}_{\mathrm{KL}}\left(p\left(x\right)\right|\left|q\left(x\right)\right)$ | Kullback–Leibler divergence |

${\mathsf{\Delta}}_{f}(p\left(x\right),q\left(x\right))$ | f-divergence |

$\left|\right|p\left(x\right)-{q\left(x\right)\left|\right|}_{TV}$ | Total variation distance |

${D}_{\infty}\left(p\left(x\right)\right|\left|q\left(x\right)\right)$ | Max-divergence |

${D}_{\infty}^{\delta}\left(p\left(x\right)\right|\left|q\left(x\right)\right)$ | $\delta $-approximate max-divergence |

$\overline{D}$ | Expected distortion |

$d({x}_{i},{y}_{j})$ | Single symbol distortion |

${p}_{E}$ | Error probability |

$\mathcal{H}$ | A class of functions |

$\mathsf{\Gamma}$ | A divergence |

${D}_{\mathsf{\Gamma}}^{\mathcal{H}}(p\left(x\right),q\left(x\right))$ | $\mathcal{H}$-restricted $\mathsf{\Gamma}$-divergence |

${D}_{f}^{\mathcal{H}}(p\left(x\right),q\left(x\right))$ | $\mathcal{H}$-restricted f-divergence |

${D}_{R,\alpha}^{\mathcal{H}}(p\left(x\right),q\left(x\right))$ | $\mathcal{H}$-restricted Rényi divergence |

Privacy Type | Data Type | Input | GDP and LDP Mapping | Real Output | Random Output | Adjacent Relationship | |
---|---|---|---|---|---|---|---|

GDP [7] | Numerical data | Dataset | X | $\{p\left(z\right|x):p\left(z\right|x)\le {e}^{\epsilon}p\left(z\right|{x}^{\prime})\}$ | Y | Z | x and ${x}^{\prime}$ are adjacent datasets. |

LDP | Categorical data | Data item | X | $\{p\left(z\right|x):p\left(z\right|x)\le {e}^{\epsilon}p\left(z\right|{x}^{\prime})\}$ | X | Z | x and ${x}^{\prime}$ are different. |

Existing Work | Privacy Type | Information-Theoretic Metric | Formula | Description |
---|---|---|---|---|

DP [7] | $\epsilon $-DP | Channel transition probability | $p\left(z\right|x)\le {e}^{\epsilon}p\left(z\right|{x}^{\prime})$ | The transition probability matrix is used as the GDP mapping. |

DP [8] | $(\epsilon ,\delta )$-DP | f-divergence | ${\mathsf{\Delta}}_{{e}^{\epsilon}}=max{d}_{{e}^{\epsilon}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))$ ${d}_{{e}^{\epsilon}}=max\{p\left(z\right|x)-{e}^{\epsilon}p\left(z\right|{x}^{\prime}),p\left(z\right|{x}^{\prime})-{e}^{\epsilon}p\left(z\right|x),0\}$ ${\mathsf{\Delta}}_{{e}^{\epsilon}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))\le \delta $ | f-divergence includes KL-divergence. |

DP [9] | $\epsilon $-DP | Max-divergence | ${D}_{\infty}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))\le \epsilon $ ${D}_{\infty}(p\left(z\right|{x}^{\prime})\left|\right|p\left(z\right|x))\le \epsilon $ | Since the max-divergence is not symmetric and does not satisfy triangular inequality, the reciprocal of equation must be true. |

$(\epsilon ,\delta )$-DP | ${D}_{\infty}^{\delta}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))\le \epsilon $ ${D}_{\infty}^{\delta}(p\left(z\right|{x}^{\prime})\left|\right|p\left(z\right|x))\le \epsilon $ | |||

$(\alpha ,\epsilon )$-RDP [11] | $\epsilon $-DP | Rényi divergence | ${D}_{\alpha}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))\le \epsilon $ | When $\alpha \to \infty $, $(\alpha ,\epsilon )$-RDP is $\epsilon $-DP according to max-divergence. If $\mathcal{M}$ is $\epsilon $-DP, then $\mathcal{M}$ is $(\alpha ,\frac{1}{2}{\epsilon}^{2}\alpha )$-RDP [23]. |

$(\epsilon +\frac{log\frac{1}{\delta}}{\alpha -1},\delta )$-DP | If $\mathcal{M}$ is $(\alpha ,\epsilon )$-RDP, then it also satisfies $(\epsilon +\frac{log\frac{1}{\delta}}{\alpha -1},\delta )$-DP. | |||

Capacity bounded DP [25] | $(\epsilon ,\delta )$-DP | $\mathcal{H}$-restricted divergence | ${D}_{\mathsf{\Gamma}}^{\mathcal{H}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))\le \epsilon $ | An adversary cannot distinguish between $p\left(z\right|x)$ and $p\left(z\right|{x}^{\prime})$ beyond $\epsilon $ in the function class $\mathcal{H}$, where $\mathsf{\Gamma}$ is the f-divergence. |

$(\alpha ,\epsilon )$-RDP | An adversary cannot distinguish between $p\left(z\right|x)$ and $p\left(z\right|{x}^{\prime})$ beyond $\epsilon $ in the function class $\mathcal{H}$, where $\mathsf{\Gamma}$ is the Rényi divergence. |

Existing Work | Information-Theoretic Privacy Definition | Formula | Description | Relationship to GDP | Stronger or Weaker than GDP |
---|---|---|---|---|---|

[26] | $\epsilon $-information privacy | ${e}^{-\epsilon}\le \frac{p\left(x\right|z)}{p\left(x\right)}\le {e}^{\epsilon}$ | When the output is given, the posterior and prior probabilities of the input x do not change significantly. | $\epsilon $-information privacy ⇒$2\epsilon $-DP | $\epsilon $-information privacy is stronger than $2\epsilon $-DP. |

[27] | $\epsilon $-strong DP | ${sup}_{z,x,{x}^{\prime}}\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}\le {e}^{\epsilon},\forall z,x,{x}^{\prime}$ | $\epsilon $-strong DP relaxes the adjacent datasets assumption. | $\epsilon $-strong DP ⇒$\epsilon $-information privacy $\epsilon $-information privacy ⇒$2\epsilon $-strong DP $\epsilon $-information privacy ⇒$\frac{\epsilon}{H\left(X\right)}$-worst-case divergence privacy $\frac{\epsilon}{H\left(X\right)}$-worst-case divergence privacy ⇒$\frac{\epsilon}{H\left(X\right)}$-divergence privacy $\epsilon $-DP ⇒$(\epsilon ,\delta )$-DP | $\epsilon $-strong DP is stronger than $\epsilon $-information privacy. $\epsilon $-information privacy is stronger than $2\epsilon $-DP. $\epsilon $-DP is stronger than $(\epsilon ,\delta )$-DP. |

$\epsilon $-information privacy | The same as above. | The same as above. | |||

Worst-case divergence privacy | $H\left(S\right)-{min}_{z}H\left(S\right|Z=z)=\epsilon H\left(S\right)$ | Some private data S are correlated with some non-private data X. | |||

[12] | $\epsilon $-identifiability | $p\left(x\right|z)\le {e}^{\epsilon}p\left({x}^{\prime}\right|z)$ | Two adjacent datasets cannot be distinguished from the posterior probabilities after observing the output dataset, which makes any individual’s data hard to identify. | $\epsilon $-identifiability ⇒$[\epsilon -maxln\frac{p\left(x\right)}{p\left({x}^{\prime}\right)},\epsilon ]$-DP $\epsilon $-DP ⇒$[\epsilon ,\epsilon +2maxln\frac{p\left(x\right)}{p\left({x}^{\prime}\right)}]$-mutual-information privacy | $\epsilon $-identifiability is stronger than $[\epsilon -maxln\frac{p\left(x\right)}{p\left({x}^{\prime}\right)},\epsilon ]$-DP. $\epsilon $-DP is stronger than $[\epsilon ,\epsilon +2maxln\frac{p\left(x\right)}{p\left({x}^{\prime}\right)}]$-mutual-information privacy. |

$\epsilon $-mutual-information privacy | $I(X;Z)\le \epsilon $ | Mutual-information privacy measures the average amount of information about X contained in Z. | |||

[13] | $\epsilon $-mutual-information DP | ${sup}_{p\left({x}_{i}\right)}I({x}_{i};Z|{X}^{-i})\le \epsilon $ | The same as $\epsilon $-mutual-information privacy in work [12] above, and ${X}^{-i}$ represents the input dataset except the i-th element. | $\epsilon $-DP⇒$\epsilon $-mutual information DP ⇒$(\epsilon ,\delta )$-DP | $\epsilon $-DP is stronger than $\epsilon $-mutual-information DP. $\epsilon $-mutual-information DP is stronger than $(\epsilon ,\delta )$-DP. |

Existing Work | Information-Theoretic Privacy Definition | Formula | Description | Relationship to LDP | Stronger or Weaker than LDP |
---|---|---|---|---|---|

[12,19] | $\epsilon $-mutual-information privacy | The same as Table 5 above. | The same as Table 5 above. | $\epsilon $-local information privacy ⇒$\epsilon $-mutual-information privacy $\epsilon $-local information privacy ⇒$2\epsilon $-LDP $\epsilon $-LDP ⇒$\epsilon $-local information privacy | $\epsilon $-local information privacy is stronger than 2$\epsilon $-LDP. $\epsilon $-LDP is stronger than $\epsilon $-local information privacy. |

[19,26] | $\epsilon $-local information privacy | ||||

[21,26] | $\epsilon $-local information privacy | The same as Table 5 above. | The same as Table 5 above. | $\epsilon $-LDP ⇒$\epsilon $-local information privacy $\epsilon $-local information privacy ⇒$2\epsilon $-LDP $\epsilon $-SRLIP ⇒$\epsilon $-local information privacy | $\epsilon $-LDP is stronger than $\epsilon $-local information privacy. $\epsilon $-local information privacy is stronger than 2$\epsilon $-LDP. |

[21] | $\epsilon $-SRLIP | ${e}^{-\epsilon}\le \frac{p\left(z\right|s,{x}_{1},\dots ,{x}_{m})}{p\left(z\right|{x}_{1},\dots ,{x}_{m})}\le {e}^{\epsilon}$ | SRLIP satisfies $\epsilon $-LIP for the attacker accessing some data $\{{x}_{1},\dots ,{x}_{m}\}$ of a user and does not leak sensitive data s beyond the knowledge the attacker gained from the side channel. |

Existing Work | Privacy Metric | Formula | Description | Bound |
---|---|---|---|---|

[16] | Maximal leakage | $ML\left(p\left(z\right|x)\right)={max}_{p\left(x\right)}({H}_{\infty}\left(X\right)-{H}_{\infty}\left(X\right|Z))$ | The maximal leakage of channel $p\left(z\right|x)$ is the maximal reduction in uncertainty of X when Z is given, which is taken by maximizing over all input distributions of the attacker’s side information. | $ML\left(p\left(z\right|x)\right)\le d\epsilon {log}_{2}e+{log}_{2}m$ with spheres $\{x\in {\{0,1\}}^{n}|d(x,{x}_{i})\le d\}$ of radius d and center ${x}_{i}$. |

[7] | Min-entropy leakage | ${I}_{\infty}(X;Z)={H}_{\infty}\left(X\right)-{H}_{\infty}\left(X\right|Z)$ | The min-entropy leakage corresponds to the ratio between the probabilities of attack success with a priori and a posterior. | ${I}_{\infty}(X;Z)\le n{log}_{2}\frac{\upsilon {e}^{\epsilon}}{\upsilon -1+{e}^{\epsilon}}$ with $\upsilon \ge 2$ |

Worst-case leakage | ${C}_{\infty}={max}_{p\left(x\right)}{I}_{\infty}(X;Z)$ | The same as maximal leakage above. | ||

[29] | Mutual information | $I(X;Z)$ | The mutual information denotes the amount of information leaked on X given Z. | $I(X;Z)\le 3\epsilon n$$I(X;Z)\ge n(1-2\eta )$ with $\delta \ge {2}^{-C(\epsilon ,\eta )n}$, $0<\u03f5,\eta <1$, and a constant $C(\epsilon ,\eta )>0$ |

[30] | Min-entropy leakage | The same as above. | The same as above. | ${I}_{\infty}(X;Z)\le log({\sum}_{t=1}^{q}exp\left(\epsilon {d}_{t}\right))$, where q is the number of connected components of induced adjacency graph, and ${d}_{t}$ is the diameter of the t-th connected component. |

[14] | Mutual information | $I(X;Z)$ | The same as above. | – |

[13] | $\alpha $-mutual-information | ${I}_{\alpha}(X;Z)={min}_{p\left(z\right)}{D}_{\alpha}\left(p\left(z\right|x)\right|\left|p\left(z\right)p\left(x\right)\right)$ | The notion of $\alpha $-mutual-information is the generalization of mutual information using Rényi information measures. | ${sup}_{p\left({x}_{i}\right)}{I}_{\alpha}({x}_{i};Z|{X}^{-i})\le \epsilon $ |

[31] | Max-information | ${I}_{\infty}(X;Z)={log}_{2}{sup}_{x,z\in (X,Z)}\frac{p(x,z)}{p\left(x\right)p\left(z\right)}$ | Maximum information is a correlation measure, similar to mutual information, which allows to bound the change of the conditional probability of an event relative to prior probability. | ${I}_{\infty}(X;Z)\le {log}_{2}e\xb7\epsilon n$ and ${I}_{\infty}^{\beta}(X;Z)\le {log}_{2}e\xb7({\epsilon}^{e}\frac{n}{2}+\epsilon \sqrt{\frac{nln\frac{2}{\beta}}{2}})$ for $\epsilon $-DP ${I}_{\infty}^{\beta}(X;Z)=O(n{\epsilon}^{2}+n\sqrt{\frac{\delta}{\epsilon}})$ for $(\epsilon ,\delta )$-DP |

$\beta $-approximate max-information | ${I}_{\infty}^{\beta}(X;Z)={log}_{2}{sup}_{O\subseteq (X\times Z),p\left(\right(x,z)\in O)>\beta}\frac{p\left(\right(x,z)\in O)-\beta}{p\left(x\right)p\left(z\right)}$ | |||

[11] | Rényi divergence | ${D}_{\alpha}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))$ | A natural relaxation of GDP based on the Rényi divergence. | – |

[25] | $\mathcal{H}$-restricted divergences | ${D}_{\mathsf{\Gamma}}^{\mathcal{H}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))$ | The privacy loss is measured in terms of a divergence $\mathsf{\Gamma}$ between output distributions of a mechanism on datasets that differ by a single record restricted to functions in $\mathcal{H}$. | ${D}_{\mathrm{KL}}^{\mathcal{H}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))\le 8\sqrt{\epsilon}$ |

[32,33] | Privacy budget | $\epsilon =ln\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}$ | The privacy budget represents the level of privacy preserving. | – |

Existing Work | Utility Metric | Formula | Description | Bound |
---|---|---|---|---|

[7] | Expected distortion | $U(Y,Z)={\sum}_{y}{\sum}_{z}p\left(y\right)p\left(z\right|y)d(y,z)$ | How much information about the real answer can be obtained from the reported answer to average. | $U(Y,Z)\le \frac{{e}^{\epsilon n}(1-{e}^{\epsilon})}{{e}^{\epsilon n}(1-{e}^{\epsilon})+c(1-{e}^{\epsilon n})}$ with $\left|\right\{z\left|d\right(y,z)=d\}|=c$ |

[14] | Expected distortion | ${\sum}_{x}{\sum}_{z}p\left(x\right)p\left(z\right|x)d(x,z)$ | The same as above. | – |

[32] | Fidelity | $\left|\right|\xb7{\left|\right|}_{1}$ | The fidelity of a pair of transition probability distributions is ${\mathbb{L}}_{1}$-distortion metric. | – |

[33] | Mutual information | $I(X;Z)$ | Mutual information captures the amount of information shared by two variables, that is to say, quantifying how much information can be preserved when releasing a private view of the data. | – |

Existing Work | Privacy Metric | Formula | Description | Bound |
---|---|---|---|---|

[17] | KL-divergence | ${D}_{\mathrm{KL}}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))$ | The general result bounds the KL-divergence between distributions $p\left(z\right|x)$ and $p\left(z\right|{x}^{\prime})$ by the privacy budget $\epsilon $ and the total variation distance between $p\left(x\right)$ and $q\left(x\right)$ of the initial distributions of the X. | ${D}_{\mathrm{KL}}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))+{D}_{\mathrm{KL}}(p\left(z\right|{x}^{\prime})\left|\right|p\left(z\right|x))\le 4{({e}^{\epsilon}-1)}^{2}\left|\right|p\left(x\right)-q\left(x\right){\left|\right|}_{TV}^{2}$ |

[34,35] | Mutual information | $I(X;Z)$ | The same as Table 7 above. | – |

[4,5,37,38] | Privacy budget | $\epsilon =ln\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}$ | The same as Table 7 above. | – |

Average privacy [39] | Conditional entropy | $\frac{H\left(X\right|Z,P)}{H\left(X\right|P)}$ | Privacy metric is the fraction of sensitive information that is retained from the aggregator with prior knowledge P. | – |

Existing Work | Utility Metric | Formula | Description | Bound |
---|---|---|---|---|

[34,35,37] | Expected Hamming distortion | $E\left[d(x,z)\right]=p(x\ne z)={\sum}_{x}{\sum}_{z}p\left(x\right)p(z\ne x|x)$ | Hamming distortion measures the utility of a channel $p\left(z\right|x)$ in terms of the worst-case Hamming distortion over source distribution $p\left(x\right)$. | – |

[5] | f-divergence | ${D}_{f}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))={\sum}_{x}p\left(z\right|{x}^{\prime})f\left(\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}\right)$ | f-divergence measures statistical discrimination between distributions $p\left(z\right|x)$ and $p\left(z\right|{x}^{\prime})$ by the privacy budget $\epsilon $ and the total variation distance between $p\left(x\right)$ and $q\left(x\right)$ of the initial distributions of the X. | ${D}_{\mathrm{KL}}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))+{D}_{\mathrm{KL}}(p\left(z\right|{x}^{\prime})\left|\right|p\left(z\right|x))\le \frac{2(1+\delta ){({e}^{\epsilon}-1)}^{2}}{{e}^{\epsilon}+1}\left|\right|p\left(x\right)-q\left(x\right){\left|\right|}_{TV}^{2}$ |

Mutual information | $I(X;Z)$ | The same as Table 8 above. | $I(X;Y)\le \frac{1}{2}(1+\delta )P\left(T\right)P\left({T}^{c}\right){\epsilon}^{2}$ with $T\in arg{min}_{A\subseteq X}|P\left(A\right)-\frac{1}{2}|$ for a given distribution P and partitioning X into two parties T and ${T}^{c}$ | |

[36] | Expected distortion | ${\sum}_{x}{\sum}_{z}p\left(x\right)p\left(z\right|x)d(x,z)$ | A channel $p\left(z\right|x)$ yields a small distortion between input and output sequences with respect to a given distortion measure. | – |

Average error probability [20] | Expected Hamming distortion | ${p}_{E}={\sum}_{x}p\left(x\right){\sum}_{x\ne z}p\left(z\right|x)$ | The average error probability is defined to be the expected Hamming distortion between the input and output data based on maximum a posterior estimation. | ${p}_{E}=\frac{n-1}{n-1+{e}^{\epsilon}}$ |

[38] | Mutual information | $I(X;Z)$ | The same as Table 8 above. | ${sup}_{p\left(z\right|x)}I(X;Z)={max}_{k=\lfloor \beta \rfloor}^{\lceil \beta \rceil}\{\frac{k\xb7{e}^{\epsilon}log\frac{m\xb7{e}^{\epsilon}}{k\xb7{e}^{\epsilon}+m-k}log\frac{m}{k\xb7{e}^{\epsilon}+m-k}}{k\xb7{e}^{\epsilon}+m-k}$} with $\beta =\frac{(\epsilon {e}^{\epsilon}-{e}^{\epsilon}+1)m}{{({e}^{\epsilon}-1)}^{2}}$ |

Distribution utility [39] | Mutual information | $\frac{I(Z;P)}{I(X;P)}$ | Utility metric is the fraction of relevant information after accessing to prior knowledge P or tally vector $T={\left({T}_{x}\right)}_{x\in X}$ and ${T}_{x}=\left|\{i:{x}_{i}=x\}\right|$. | – |

Tally utility [39] | Entropy Mutual information | $\frac{I(Z;T)}{H\left(T\right)}$ |

Existing Work | Privacy Type | Privacy Property | Information-Theoretic Metric | Formal Description |
---|---|---|---|---|

[16] | GDP | Sequential composition | Maximal leakage | $ML({C}_{1}+{C}_{2})\le ML\left({C}_{1}\right)+ML\left({C}_{2}\right)$ for the sequential composition ${C}_{1}+{C}_{2}$ of channels ${C}_{1}$ and ${C}_{2}$. When ${C}_{1}$ is ${\epsilon}_{1}$-DP and ${C}_{2}$ is ${\epsilon}_{2}$-DP, ${C}_{1}+{C}_{2}$ is ${\epsilon}_{1}+{\epsilon}_{2}$-DP. |

Parallel composition | $ML({C}_{1}\times {C}_{2})=ML\left({C}_{1}\right)+ML\left({C}_{2}\right)$ for the parallel composition ${C}_{1}\times {C}_{2}$ of channels ${C}_{1}$ and ${C}_{2}$. When ${C}_{1}$ is ${\epsilon}_{1}$-DP and ${C}_{2}$ is ${\epsilon}_{2}$-DP, ${C}_{1}\times {C}_{2}$ is $max\{{\epsilon}_{1},{\epsilon}_{2}\}$-DP. | |||

[8] | GDP | Sequential composition | f-divergence | ${\mathsf{\Delta}}_{\alpha {\alpha}^{\prime}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))\le {\mathsf{\Delta}}_{\alpha}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))+{max}_{x}{\mathsf{\Delta}}_{{\alpha}^{\prime}}(p\left(z\right|x),p\left(z\right|{x}^{\prime}))$, where ${\mathsf{\Delta}}_{\alpha}$, the same as Table 4 above. |

[11] | RDP | Post-processing | Rényi divergence | If there is a randomized mapping $g:R\to {R}^{\prime}$, then ${D}_{\alpha}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))\ge {D}_{\alpha}(g\left(p\left(z\right|x)\right)\left|\right|g\left(p\left(z\right|{x}^{\prime})\right))$. |

Group privacy | If $\mathcal{M}:x\to R$ is $(\alpha ,\epsilon )$-RDP, $g:{x}^{\prime}\to x$ is ${2}^{c}$-stable and $\alpha \ge {2}^{c+1}$, then $\mathcal{M}\circ g$ is $(\frac{\alpha}{{2}^{c}},{3}^{c}\epsilon )$-RDP. | |||

Sequential composition | If ${\mathcal{M}}_{1}:x\to {R}_{1}$ is $(\alpha ,{\epsilon}_{1})$-RDP and ${\mathcal{M}}_{2}:{R}_{1}\times x\to {R}_{2}$ is $(\alpha ,{\epsilon}_{2})$-RDP, then the mechanism $({\mathcal{M}}_{1},{\mathcal{M}}_{2})$ satisfies $(\alpha ,{\epsilon}_{1}+{\epsilon}_{2})$-RDP. | |||

[25] | Capacity bounded DP | Post-processing | $\mathcal{H}$-restricted divergences | $\mathcal{H}$, $\mathcal{G}$, and $\mathcal{I}$ are function classes such that for any $g\in \mathcal{G}$ and $i\in \mathcal{I}$, $i\circ g\in \mathcal{H}$. If mechanism $\mathcal{M}$ is $(\mathcal{H},\mathsf{\Gamma})$-capacity bounded DP with $\epsilon $, then $g\circ \mathcal{M}$ is also $(\mathcal{I},\mathsf{\Gamma})$-capacity bounded DP with $\epsilon $ for any $g\in \mathcal{G}$. |

Convexity | ${\mathcal{M}}_{1}$ and ${\mathcal{M}}_{2}$ are two mechanisms which have the same range and provide $(\mathcal{H},\mathsf{\Gamma})$-capacity bounded DP with $\epsilon $. If $\mathcal{M}$ is a mechanism which executes mechanism ${\mathcal{M}}_{1}$ with probability $\pi $ and ${\mathcal{M}}_{2}$ with probability $1-\pi $, then $\mathcal{M}$ is $(\mathcal{H},\mathsf{\Gamma})$-capacity bounded DP with $\epsilon $. | |||

Sequential composition | $\mathcal{H}$ is the function class $\mathcal{H}=\{{\mathcal{H}}_{1}+{\mathcal{H}}_{2}|{h}_{1}\in {\mathcal{H}}_{1},{h}_{2}\in {\mathcal{H}}_{2}\}$. If ${\mathcal{M}}_{1}\left(x\right)$ and ${\mathcal{M}}_{2}\left(x\right)$ are $({\mathcal{H}}_{1},\mathsf{\Gamma})$ and $({\mathcal{H}}_{2},\mathsf{\Gamma})$ capacity bounded DP with ${\epsilon}_{1}$ and ${\epsilon}_{2}$, respectively, then the combination $({\mathcal{M}}_{1},{\mathcal{M}}_{2})$ is $(\mathcal{H},\mathsf{\Gamma})$ capacity bounded DP with ${\epsilon}_{1}+{\epsilon}_{2}$. | |||

Parallel composition | $\mathcal{H}$ is the function class $\mathcal{H}=\{{\mathcal{H}}_{1}+{\mathcal{H}}_{2}|{h}_{1}\in {\mathcal{H}}_{1},{h}_{2}\in {\mathcal{H}}_{2}\}$. If ${\mathcal{M}}_{1}\left({x}_{1}\right)$ and ${\mathcal{M}}_{2}\left({x}_{2}\right)$ are $({\mathcal{H}}_{1},\mathsf{\Gamma})$ and $({\mathcal{H}}_{2},\mathsf{\Gamma})$ capacity bounded DP with ${\epsilon}_{1}$ and ${\epsilon}_{2}$ respectively, and the datasets ${x}_{1}$ and ${x}_{2}$ are disjoint, then the combination $({\mathcal{M}}_{1}\left({x}_{1}\right),{\mathcal{M}}_{2}\left({x}_{2}\right))$ is $(\mathcal{H},\mathsf{\Gamma})$ capacity bounded DP with $max\{{\epsilon}_{1},{\epsilon}_{2}\}$. | |||

[3] | GDP LDP | Privacy-utility monotonicity | Mutual information | The mutual information decreases as the decreasing of the privacy budget, and vice versa |

Existing Work | Privacy Type | Model | Objective Function | Constraint Condition | Mechanism | Solution | Description | ||
---|---|---|---|---|---|---|---|---|---|

[7] | GDP | Maximal utility | Expected distortion | $U(Y,Z)={\sum}_{y}{\sum}_{z}p\left(y\right)p\left(z\right|y)d(y,z)$ | Min-entropy leakage | ${I}_{\infty}(X;Z)={H}_{\infty}\left(X\right)-{H}_{\infty}\left(X\right|Z)$ | $p\left(z\right|y)=\frac{\alpha}{{\left({e}^{\epsilon}\right)}^{d}}$, where $d=d(y,z)$, $\alpha =\frac{{\left({e}^{\epsilon}\right)}^{n}(1-{e}^{\epsilon})}{{\left({e}^{\epsilon}\right)}^{n}(1-{e}^{\epsilon})+c(1-{\left({e}^{\epsilon}\right)}^{n})}$, and c the same as Table 8 above. | Graph symmetry induced by the adjacent relationship of adjacent datasets. | Optimal randomization mechanism provides the better utility while guaranteeing $\epsilon $-DP. |

[14] | GDP | Risk-distortion | Mutual information | ${inf}_{p\left(z\right|x)}I(X;Z)$ | Expected distortion | $\sum p\left(x\right)\sum p\left(z\right|x\left)d\right(x,z)\le D$ | $p\left(z\right|x)=\frac{p\left(z\right)exp(-\epsilon d(x,z\left)\right)}{Z(x,\epsilon )}$, where $Z(x,\epsilon )$ is a normalization function. | Lagrangian multipliers. | Conditional probability distribution is DP mapping, which minimizes the privacy risk given a distortion constraint. |

[33] | GDP | Constrained maximization program | Mutual information | $maxI(X;Z)$ | GDP | $sup\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}\le exp\left(\epsilon \right)$ |
$$p\left(z\right|x)=\left\{\begin{array}{cc}p(z=x|x),\hfill & x=z\hfill \\ \frac{1-p(z=x|x)}{m-1},\hfill & x\ne z\hfill \end{array}\right.$$
| Definition of GDP. | When x is transformed into z and $z=x$, the conditional transition probability is $p(z=x|x)$. When $z\ne x$, the conditional transition probability is $\frac{1-p(z=x|x)}{m-1}$ under strongly symmetric channel. |

Existing Work | Privacy Type | Model | Objective Function | Constraint Condition | Mechanism | Solution | Description | |||
---|---|---|---|---|---|---|---|---|---|---|

[34,35,37] | LDP | Rate-distortion function | Mutual information | ${min}_{p\left(z\right|x)}I(X;Z)$ | Expected Hamming distortion | ${\sum}_{x}{\sum}_{z}p\left(x\right)p\left(z\right|x)d(x,z)\le D$ | Binary channel | $\epsilon =log\frac{1-D}{D}$ | Memoryless symmetric channel. | LDP is just a function of the channel, and the worst-case Hamming distortion on source distribution $p\left(x\right)$ measures the utility of a channel $p\left(z\right|x)$. |

Discrete alphabet | $\epsilon =log(m-1)+log\frac{1-D}{D}$ | |||||||||

[5] | LDP | Constraint maximization problem | KL-divergence Mutual information | ${max}_{p\left(z\right|x)}{D}_{\mathrm{KL}}\left(p\left(z\right|x)\right||p\left(z\right|{x}^{\prime}))$${max}_{p\left(z\right|x)}I(X;Z)$ | LDP | $\epsilon =ln\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}$ | Binary randomized response |
$$p\left(z\right|x)=\left\{\begin{array}{cc}\frac{{e}^{\epsilon}}{1+{e}^{\epsilon}},\hfill & x=z\hfill \\ \frac{1}{1+{e}^{\epsilon}},\hfill & x\ne z\hfill \end{array}\right.$$
| Solving the privacy-utility maximization problem is equivalent to solving finite-dimensional linear program. | The binary and multivariate randomized response mechanisms are universally optimal in the low and high privacy regimes and well approximate the intermediate regime. The quaternary randomized mechanism satisfies $(\epsilon ,\delta )$-LDP. |

Multivariate randomized response |
$$p\left(z\right|x)=\left\{\begin{array}{cc}\frac{{e}^{\epsilon}}{n-1+{e}^{\epsilon}},\hfill & x=z\hfill \\ \frac{1}{n-1+{e}^{\epsilon}},\hfill & x\ne z\hfill \end{array}\right.$$
| |||||||||

Quaternary randomized response |
$$\begin{array}{cc}& \hspace{1em}0\text{}1\hspace{1em}\text{}2\hspace{1em}\hspace{1em}\text{}3\hfill \\ \begin{array}{c}0\\ 1\end{array}& \left(\begin{array}{cccc}\delta & 0& \frac{1-\delta}{1+{e}^{\epsilon}}& \frac{(1-\delta ){e}^{\epsilon}}{1+{e}^{\epsilon}}\\ 0& \delta & \frac{(1-\delta ){e}^{\epsilon}}{1+{e}^{\epsilon}}& \frac{1-\delta}{1+{e}^{\epsilon}}\end{array}\right)\end{array}$$
| |||||||||

[38] | LDP | Maximize utility | Mutual information | ${sup}_{p\left(z\right|x)}I(X;Z)\le {I}_{\beta}$${I}_{k}=\frac{k\xb7{e}^{\epsilon}log\frac{m\xb7{e}^{\epsilon}}{k\xb7{e}^{\epsilon}+m-k}log\frac{m}{k\xb7{e}^{\epsilon}+m-k}}{k\xb7{e}^{\epsilon}+m-k}$$\beta =\frac{(\epsilon {e}^{\epsilon}-{e}^{\epsilon}+1)m}{{({e}^{\epsilon}-1)}^{2}}$ | LDP | $\epsilon =ln\frac{p\left(z\right|x)}{p\left(z\right|{x}^{\prime})}$ | k-subset mechanism |
$$p\left(Z\right|x)=\left\{\begin{array}{cc}\frac{n{e}^{\epsilon}}{\left({\textstyle \genfrac{}{}{0pt}{}{n}{k}}\right)(k{e}^{\epsilon}+n-k)},\hfill & \left|Z\right|=k,x\in Z\hfill \\ \frac{n}{\left({\textstyle \genfrac{}{}{0pt}{}{n}{k}}\right)(k{e}^{\epsilon}+n-k)},\hfill & \left|Z\right|=k,x\notin Z\hfill \\ 0,\hfill & \left|Z\right|\ne k\hfill \end{array}\right.$$
| This problem maximizes mutual information when x is a sample according to the uniform distribution with probability $\frac{1}{n}$. | The mutual information bound is used as a universal statistical utility measurement, and the k-subset mechanism is the optimal $\epsilon $-LDP mechanism. |

Existing Work | Attack Target | Attack Type | Attack Method | Characteristic | Attack Effect |
---|---|---|---|---|---|

[47] | Generative models | Membership inference | The discriminator can learn the statistical difference of distribution, detect overfitting and recognize the input as part of the training dataset. | The proposed attack has low running cost, does not need information about the attacked model, and has good generalization. | Defenses are either ineffective or lead to a significant decline in the performance of the generative models in terms of training stability or sample quality. |

[48] | Generative models | Co-membership inference | The membership inference of the target data x is used as the optimization of the attacker’s network to search for potential codes to reproduce x, and the final reconstruction error is used to judge whether x is in the training data. | When the generative models are trained with large datasets, the co-membership inference attack is necessary to achieve success. | The performance of attacker’s network is better than that of previous membership attacks, and the power of co-membership attack is much greater than that of a single attack. |

[49] | Generative models | Membership inference | The membership inference attack based on Monte Carlo integration only considers the small distance samples in the model. | This attack allows membership inference without assuming the type of generative models. | The success rate of this attack is better than that of previous studies on most datasets, and there are only very mild assumptions. |

[50] | Generative models | Membership inference | This work proposed a general attack model based on reconstruction for which the model is suitable for all settings according to the attacker’s knowledge about the victim model. | This work provides a theoretically reliable attack calibration technology, which can continuously improve the attack performance in different attack settings, data modes, and training configurations in all cases. | This attack reveals the information of the training data used for the victim model. |

[51] | GAN | Model extraction | This work studied the model extraction attack based on target and background knowledge from the perspectives of fidelity extraction and accuracy extraction. | Model extraction based on transfer learning can enable adversaries to improve the performance of their GAN model through transfer learning. | Attack model stealing the most advanced target model can be transferred to new fields to expand the application scope of extraction model. |

Existing Work | GAN Type | Clipping Strategy | Perturbation Strategy | Privacy Loss Accountant |
---|---|---|---|---|

[55] | GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[56] | WGAN | Clipping weight | Gradient perturbation | Moment accountant |

[45] | GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[58] | CGAN | Clipping gradient | Gradient perturbation | RDP accountant |

[60] | GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[61] | GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[62] | WGAN | Clipping gradient | Gradient perturbation | RDP accountant |

[63] | WGAN-GP | Clipping gradient | Gradient perturbation | Moment accountant |

[65] | AC-GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[67] | GAN | Clipping gradient | Gradient perturbation | Moment accountant |

[68] | NetGAN | Clipping gradient | Gradient perturbation | Privacy budget composition [9] |

[70] | GAN | – | Data perturbation | – |

[71] | GAN | – | Data perturbation | Advanced composition [9] |

[73] | GAN | – | Data perturbation | – |

[74] | GAN | – | Data perturbation | – |

[75] | GAN | – | Label perturbation | Moment accountant |

[76] | GAN | – | Objective function perturbation | Advanced composition |

[77] | GAN | – | Differential privacy identifier | Privacy budget composition |

Existing Work | GAN Type | Clipping Strategy | Perturbation Strategy | Privacy Loss Accountant | Training Method |
---|---|---|---|---|---|

[81] | GAN | Clipping weight | Weight perturbation | RDP accountant | FedAvg algorithm |

[62] | WGAN | Clipping gradient | Gradient perturbation | RDP accountant | FedAvg algorithm |

[82] | GAN | Clipping weight | Gradient perturbation | Moment accountant | FedAvg algorithm |

[83] | GAN | – | Gradient perturbation | – | FedAvg algorithm |

[84] | GAN | Clipping gradient | Gradient perturbation | RDP accountant | Serial training |

[85] | GAN | – | Differential average-case privacy | – | FedAvg algorithm |

**Table 17.**Open problems of GDP and LDP from the perspective of different types of information-theoretic channel.

Scenario | Data Type | Privacy Type | Open Problem | Method | Information-Theoretic Foundation |
---|---|---|---|---|---|

Data collection | Categorical data | LDP | Personalized privacy demands | Rate-distortion framework | Discrete single symbol information-theoretic channel |

Poor data utility | |||||

Information-theoretic analysis of existing LDP mechanisms | |||||

High-dimensional (correlated) data collection | Categorical data | LDP | Poor data utility | Rate-distortion framework Joint probability Markov chain | Discrete sequence information-theoretic channel |

Continuous (correlated) data releasing | Numerical data | GDP | Information-theoretic analysis of existing GDP mechanisms | Rate-distortion framework Joint probability Markov chain | Continuous information-theoretic channel |

RDP mechanisms | |||||

Personalized privacy demands | |||||

Poor data utility | |||||

Multiuser (correlated) data collection | Numerical data Categorical data | GDP LDP | Privacy leakage risk | Rate-distortion framework | Multiple access channel Multiuser channel with correlated sources |

Multi-party data releasing | Broadcast channel | ||||

Synthetic data generation | Numerical data Categorical data | GDP LDP | Poor data utility | GAN GAN with federated learning | Information-theoretic metrics |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Liu, H.; Peng, C.; Tian, Y.; Long, S.; Tian, F.; Wu, Z.
GDP vs. LDP: A Survey from the Perspective of Information-Theoretic Channel. *Entropy* **2022**, *24*, 430.
https://doi.org/10.3390/e24030430

**AMA Style**

Liu H, Peng C, Tian Y, Long S, Tian F, Wu Z.
GDP vs. LDP: A Survey from the Perspective of Information-Theoretic Channel. *Entropy*. 2022; 24(3):430.
https://doi.org/10.3390/e24030430

**Chicago/Turabian Style**

Liu, Hai, Changgen Peng, Youliang Tian, Shigong Long, Feng Tian, and Zhenqiang Wu.
2022. "GDP vs. LDP: A Survey from the Perspective of Information-Theoretic Channel" *Entropy* 24, no. 3: 430.
https://doi.org/10.3390/e24030430