A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy
Abstract
:1. Introduction
- (1)
- A personalized FL framework combining knowledge distillation and DP is proposed to cope with non-IID data environments, incorporating knowledge distillation in the training of local models to enhance the model’s generalization capabilities and applying differential privacy mechanisms to noise perturbation of the output parameters;
- (2)
- A bidirectional feedback mechanism is proposed that adaptively adjusts the knowledge distillation and DP parameters according to the performance of the model and the user’s privacy requirements, balancing the relationship between privacy protection and model performance to achieve the optimal model performance under the demanded privacy intensity;
- (3)
- It is experimentally demonstrated that FedKADP improves the robustness and accuracy of data processing in non-IID data environments, maximizes model performance with the same privacy-preserving strength, and significantly reduces the communication cost.
2. Relevant Definitions
2.1. Federated Learning
2.2. Knowledge Distillation
2.3. Rényi Differential Privacy
3. FedKADP-Specific Implementation
3.1. FedKADP Framework
3.2. Bidirectional Feedback Mechanism
- (1)
- From DP to knowledge distillation: the temperature parameter of knowledge distillation is dynamically adjusted based on the actual impact of noise added by DP on model performance;
- (2)
- From knowledge distillation to DP: the noise parameter of the DP based on the actual impact of knowledge distillation on model performance is adjusted to ensure that model learning is maximized without sacrificing too much privacy.
3.3. Knowledge Distillation Optimization
3.4. Adaptive Differential Privacy
3.5. FedKADP-Algorithm Description
Algorithm 1. FedKADP Client Algorithm. |
Input: is the number of local training iterations, represents the model parameters, is the learning rate, is the batch size, is the Gaussian noise parameter, is the temperature parameter, and is the cropping threshold. |
1: |
2: |
3: into batches of size ) |
4: for do |
5: do |
6: |
7: |
8: |
9: |
10: |
11: |
12: |
13: end for |
14: return |
Algorithm 2. FedKADP Server Algorithm. |
Input: is the privacy budget, is the total number of clients, and is the client sampling rate. |
1: |
2: for do |
3: |
4: clients) |
5: for in parallel do |
6: Algorithm 1 7: end for |
8: |
9: (Privacy overhead calculating) |
10: if |
11: break |
12: end if |
13: ) |
14: end for |
15: return |
4. Theoretical Analysis
4.1. Privacy Overhead Calculation
4.2. Privacy Analysis
5. Experiment
5.1. Visual Analysis of the Bidirectional Feedback Mechanism
5.2. Influence of Relevant Parameters on FedKADP
5.2.1. Client Sampling Rate ()
5.2.2. Initial Noise Parameter ()
5.3. Membership Inference Attack Experiments
5.4. Comparison Experiments
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Waring, J.; Lindvall, C.; Umeton, R. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artif. Intell. Med. 2020, 104, 101822. [Google Scholar] [CrossRef] [PubMed]
- Lin, W.Y.; Hu, Y.H.; Tsai, C.F. Machine learning in financial crisis prediction: A survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 2011, 42, 421–436. [Google Scholar]
- McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. Artif. Intell. Statist. 2017, 54, 1273–1282. [Google Scholar]
- Singh, P.; Singh, M.K.; Singh, R.; Singh, N. Federated Learning: Challenges, Methods, and Future Directions; Springer International Publishing: Cham, Switzerland, 2022; pp. 199–214. [Google Scholar]
- Fredrikson, M.; Jha, S.; Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, 12–16 October 2015; pp. 1322–1333. [Google Scholar]
- Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H.B.; Mironov, I.; Talwar, K.; Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 24–28 October 2016; pp. 308–318. [Google Scholar]
- Noble, M.; Bellet, A.; Dieuleveut, A. Differentially private federated learning on heterogeneous data. Int. Conf. Artif. Intell. Statist. 2022, 151, 10110–10145. [Google Scholar]
- Wei, K.; Li, J.; Ding, M.; Ma, C.; Yang, H.H.; Farokhi, F.; Jin, S.; Quek, T.Q.S.; Poor, H.V. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans. Inf. Forensics Secur. 2020, 15, 3454–3469. [Google Scholar] [CrossRef]
- Tan, A.Z.; Yu, H.; Cui, L.; Yang, Q. Towards personalized federated learning. IEEE Trans. Neural Netw. Learn. Syst. 2022, 34, 9587–9603. [Google Scholar] [CrossRef] [PubMed]
- Zhao, Y.; Li, M.; Lai, L.; Suda, N.; Civin, D.; Chandra, V. Federated learning with non-iid data. arXiv 2018, arXiv:1806.00582. [Google Scholar] [CrossRef]
- Zhang, X.; Hong, M.; Dhople, S.; Yin, W.; Liu, Y. Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Trans. Signal Process. 2021, 69, 6055–6070. [Google Scholar] [CrossRef]
- Jeong, E.; Oh, S.; Kim, H.; Park, J.; Bennis, M.; Kim, S.L. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv 2018, arXiv:1811.11479. [Google Scholar]
- Fallah, A.; Mokhtari, A.; Ozdaglar, A. Personalized federated learning: A meta-learning approach. arXiv 2020, arXiv:2002.07948. [Google Scholar]
- Tursunboev, J.; Kang, Y.-S.; Huh, S.-B.; Lim, D.W.; Kang, J.M.; Jung, H. Hierarchical Federated Learning for Edge-Aided Unmanned Aerial Vehicle Networks. Appl. Sci. 2022, 12, 670. [Google Scholar] [CrossRef]
- Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv 2015, arXiv:1503.02531. [Google Scholar]
- Dwork, C.; Roth, A. The algorithmic foundations of differential privacy. Found. Trends® Theor. Comput. Sci. 2014, 9, 211–407. [Google Scholar] [CrossRef]
- McSherry, F.; Talwar, K. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), Providence, RI, USA, 21–23 October 2007; pp. 94–103. [Google Scholar]
- McSherry, F.D. Privacy integrated queries: An extensible platform for privacy-preserving data analysis. In Proceedings of the SIGMOD/PODS ‘09: International Conference on Management of Data, Providence, RI, USA, 29 June–2 July 2009; pp. 19–30. [Google Scholar]
- Mironov, I. Rényi differential privacy. In Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium (CSF), Santa Barbara, CA, USA, 21–25 August 2017; pp. 263–275. [Google Scholar]
- Liu, F. Generalized gaussian mechanism for differential privacy. IEEE Trans. Knowl. Data Eng. 2018, 31, 747–756. [Google Scholar] [CrossRef]
- Wang, Y.X.; Balle, B.; Kasiviswanathan, S.P. Subsampled rényi differential privacy and analytical moments accountant. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Okinawa, Japan, 16–18 April 2019; pp. 1226–1235. [Google Scholar]
- Wu, C.; Wu, F.; Lyu, L.; Huang, Y.; Xie, X. Communication-efficient federated learning via knowledge distillation. Nat. Commun. 2022, 13, 2032. [Google Scholar] [CrossRef] [PubMed]
- Fu, J.; Chen, Z.; Han, X. Adap dp-fl: Differentially private federated learning with adaptive noise. In Proceedings of the 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), Wuhan, China, 9–11 December 2022; pp. 656–663. [Google Scholar]
Epochs | Batch Size | Learning Rate | Learning Rate Decay | Reg | |
---|---|---|---|---|---|
Shadow model | 50 | 128 | 0.001 | 0.96 | 1 × 10−4 |
Attack model | 50 | 10 | 0.001 | 0.96 | 1 × 10−7 |
Method | Label | Precision | Recall | F1-Score | Support |
---|---|---|---|---|---|
FedKADP | Member | 0.48 | 0.03 | 0.06 | 15,000 |
Fedavg | Member | 0.50 | 0.18 | 0.26 | 15,000 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jiang, Y.; Zhao, X.; Li, H.; Xue, Y. A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy. Electronics 2024, 13, 3538. https://doi.org/10.3390/electronics13173538
Jiang Y, Zhao X, Li H, Xue Y. A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy. Electronics. 2024; 13(17):3538. https://doi.org/10.3390/electronics13173538
Chicago/Turabian StyleJiang, Yingrui, Xuejian Zhao, Hao Li, and Yu Xue. 2024. "A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy" Electronics 13, no. 17: 3538. https://doi.org/10.3390/electronics13173538
APA StyleJiang, Y., Zhao, X., Li, H., & Xue, Y. (2024). A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy. Electronics, 13(17), 3538. https://doi.org/10.3390/electronics13173538