Dual-Layer Bidirectional Contrastive Federated Learning: Enhancing Feature Extraction and Personalization Simultaneously
Abstract
:1. Introduction
- We propose a strategy that divides the model into a feature extraction layer and a classification layer, and uses the abstract representations output by these two layers for contrastive training, without relying on data from other clients as negative samples;
- We introduce a novel method that improves the model’s feature extraction capability by minimizing the distance between the client’s feature extraction layer and the global model, while enhancing the model’s personalization capability by maximizing the distance between the client’s classification layer and the global model;
- We conduct extensive experiments on three datasets, and the results demonstrate that our method achieves competitive performance.
2. Related Work
2.1. Personal Federated Learning
2.2. Contrastive Learning
3. Preliminary
3.1. Problem Formulation
3.2. Motivation
4. Method
4.1. Framework Overview
Algorithm 1 FDCL-Server |
Input: global round T, all clients set c, client local steps Q, initial model , dataset size Output: a trained model
|
Algorithm 2 FDCL-Client |
Input: all clients c, client learning rate , local epoch E, temerature , client’s dataset , local model w, global model Output: a trained model
|
4.2. Model Partition
4.3. Bidirectional Contrastive Learning
- The feature extraction layer of the global model is denoted as , and the classification layer is ;
- The feature extraction layer of the current version of the client model is , and the classification layer is ;
- The feature extraction layer of the previous version of the client model is , and the classification layer is .
5. Experiment
5.1. Experimental Setup
- FedAVG: FedAVG is a fundamental federated learning algorithm that performs global model averaging updates by training locally on the client and uploading updates to the server;
- DITTO: DITTO achieves fairness and robustness in heterogeneous environments by introducing a globally regularized multi-task learning framework;
- MOON: MOON uses contrastive learning to enhance the representation ability of global models and conducts personalized learning between clients through contrastive loss functions;
- FedPer: FedPer addresses the issue of non-independent and identically distributed data by learning personalized models for each client, thereby improving personalized performance;
- FedCP: FedCP dynamically learns a conditional policy to disentangle global and personalized information from its features, followed by independent processing through a global processing module and a personalized processing module.
5.2. Performance Comparison
5.3. The Impact of Participation Rate
5.4. Effectiveness of the Number of Classification Heads
5.5. The Impact of Hyperparameter
5.6. The Impact of Local Epochs
5.7. Influence of Client Quantity
5.8. Ablation Study
6. Analysis
6.1. Communication Overhead
6.2. Scalability Under Low-Resource Conditions
6.3. Convergence Analysis
6.3.1. Assumptions
- Smoothness: , , and are L-smooth;
- Strong Convexity: is -strongly convex and is -strongly convex if and are convex;
- Bounded Heterogeneity: ;
- Bounded Contrastive Gradient: .
6.3.2. Local Update
6.3.3. Local Deviation
6.3.4. Global Aggregation
6.3.5. Convergence Rate
7. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Kairouz, P.; McMahan, H.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A.; Bonawitz, K.; Charles, Z.; Cormode, G.; Cummings, R.; et al. Advances and Open Problems in Federated Learning; Georg Thieme: Stuttgart, Germany, 2021; Volume 14, pp. 1–210. [Google Scholar] [CrossRef]
- Liu, J.; Huang, J.; Zhou, Y.; Li, X.; Ji, S.; Xiong, H.; Dou, D. From distributed machine learning to federated learning: A survey. Knowl. Inf. Syst. 2022, 64, 885–917. [Google Scholar] [CrossRef]
- Gao, W.; Xu, G.; Meng, X. FedCon: Scalable and Efficient Federated Learning via Contribution-Based Aggregation. Electronics 2025, 14, 1024. [Google Scholar] [CrossRef]
- Chen, Z.; Zhou, C.; Jiang, Z. One-Shot Federated Learning with Label Differential Privacy. Electronics 2024, 13, 1815. [Google Scholar] [CrossRef]
- Hsu, T.M.H.; Qi, H.; Brown, M. Federated Visual Classification with Real-World Data Distribution. In Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Gernany, 2020; pp. 76–92. [Google Scholar] [CrossRef]
- Koutsoubis, N.; Yilmaz, Y.; Ramachandran, R.P.; Schabath, M.; Rasool, G. Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation. arXiv 2024, arXiv:2406.12815. [Google Scholar]
- Li, Z.; Li, Q.; Zhou, Y.; Zhong, W.; Zhang, G.; Wu, C. Edge-cloud Collaborative Learning with Federated and Centralized Features. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, Taipei, China, 23–27 July 2023; ACM: New York, NY, USA, 2023; pp. 1949–1953. [Google Scholar] [CrossRef]
- Li, Z.; Lin, T.; Shang, X.; Wu, C. Revisiting weighted aggregation in federated learning with neural networks. In Proceedings of the 40th International Conference on Machine Learning, ICML’23, Honolulu, HI, USA, 23–29 July 2023; JMLR.org: Brookline, MA, USA, 2023. [Google Scholar]
- Liu, Y.; Huang, A.; Luo, Y.; Huang, H.; Liu, Y.; Chen, Y.; Feng, L.; Chen, T.; Yu, H.; Yang, Q. FedVision: An Online Visual Object Detection Platform Powered by Federated Learning. arXiv 2020, arXiv:2001.06202. [Google Scholar] [CrossRef]
- Hu, M.; Zhou, P.; Yue, Z.; Ling, Z.; Huang, Y.; Li, A.; Liu, Y.; Lian, X.; Chen, M. FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation. In Proceedings of the 2024 IEEE 40th International Conference on Data Engineering (ICDE), Utrecht, The Netherlands, 13–16 May 2024; pp. 2137–2150. [Google Scholar] [CrossRef]
- Hu, M.; Cao, Y.; Li, A.; Li, Z.; Liu, C.; Li, T.; Chen, M.; Liu, Y. FedMut: Generalized Federated Learning via Stochastic Mutation. Proc. AAAI Conf. Artif. Intell. 2024, 38, 12528–12537. [Google Scholar] [CrossRef]
- Hu, M.; Yue, Z.; Xie, X.; Chen, C.; Huang, Y.; Wei, X.; Lian, X.; Liu, Y.; Chen, M. Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, Barcelona, Spain, 25–29 August 2024; ACM: New York, NY, USA, 2024; pp. 1096–1107. [Google Scholar] [CrossRef]
- McMahan, H.B.; Moore, E.; Ramage, D.; Hampson, S.; y Arcas, B.A. Communication-Efficient Learning of Deep Networks from Decentralized Data. arXiv 2023, arXiv:1602.05629. [Google Scholar]
- Li, T.; Sahu, A.K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; Smith, V. Federated Optimization in Heterogeneous Networks. arXiv 2020, arXiv:1812.06127. [Google Scholar]
- Mendieta, M.; Yang, T.; Wang, P.; Lee, M.; Ding, Z.; Chen, C. Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning. arXiv 2022, arXiv:2111.14213. [Google Scholar]
- Zhang, N.; Li, Y.; Shi, Y.; Shen, J. A CNN-Based Adaptive Federated Learning Approach for Communication Jamming Recognition. Electronics 2023, 12, 3425. [Google Scholar] [CrossRef]
- Zhou, Y.; Duan, G.; Qiu, T.; Zhang, L.; Tian, L.; Zheng, X.; Zhu, Y. Personalized Federated Learning Incorporating Adaptive Model Pruning at the Edge. Electronics 2024, 13, 1738. [Google Scholar] [CrossRef]
- Jiang, Y.; Zhao, X.; Li, H.; Xue, Y. A Personalized Federated Learning Method Based on Knowledge Distillation and Differential Privacy. Electronics 2024, 13, 3538. [Google Scholar] [CrossRef]
- Li, T.; Hu, S.; Beirami, A.; Smith, V. Ditto: Fair and Robust Federated Learning Through Personalization. arXiv 2021, arXiv:2012.04221. [Google Scholar]
- Dinh, C.T.; Tran, N.H.; Nguyen, T.D. Personalized Federated Learning with Moreau Envelopes. arXiv 2022, arXiv:2006.08848. [Google Scholar]
- Fallah, A.; Mokhtari, A.; Ozdaglar, A. Personalized Federated Learning: A Meta-Learning Approach. arXiv 2020, arXiv:2002.07948. [Google Scholar]
- Li, X.; Hu, C.; Luo, S.; Lu, H.; Piao, Z.; Jing, L. Distributed Hybrid-Triggered Observer-Based Secondary Control of Multi-Bus DC Microgrids Over Directed Networks. IEEE Trans. Circuits Syst. I Regul. Pap. 2025, 72, 2467–2480. [Google Scholar] [CrossRef]
- Hu, Z.; Su, R.; Veerasamy, V.; Huang, L.; Ma, R. Resilient Frequency Regulation for Microgrids Under Phasor Measurement Unit Faults and Communication Intermittency. IEEE Trans. Ind. Inform. 2025, 21, 1941–1949. [Google Scholar] [CrossRef]
- Arivazhagan, M.G.; Aggarwal, V.; Singh, A.K.; Choudhary, S. Federated Learning with Personalization Layers. arXiv 2019, arXiv:1912.00818. [Google Scholar]
- Finn, C.; Abbeel, P.; Levine, S. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv 2017, arXiv:1703.03400. [Google Scholar]
- Zhang, J.; Hua, Y.; Wang, H.; Song, T.; Xue, Z.; Ma, R.; Guan, H. FedALA: Adaptive Local Aggregation for Personalized Federated Learning. Proc. AAAI Conf. Artif. Intell. 2023, 37, 11237–11244. [Google Scholar] [CrossRef]
- Zhang, M.; Sapra, K.; Fidler, S.; Yeung, S.; Alvarez, J.M. Personalized Federated Learning with First Order Model Optimization. arXiv 2021, arXiv:2012.08565. [Google Scholar]
- Yang, M.S.; Sinaga, K.P. Federated Multi-View K-Means Clustering. IEEE Trans. Pattern Anal. Mach. Intell. 2025, 47, 2446–2459. [Google Scholar] [CrossRef] [PubMed]
- Duan, H.; Zhao, N.; Chen, K.; Lin, D. TransRank: Self-Supervised Video Representation Learning via Ranking-Based Transformation Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 3000–3010. [Google Scholar]
- Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. arXiv 2020, arXiv:2002.05709. [Google Scholar]
- He, K.; Fan, H.; Wu, Y.; Xie, S.; Girshick, R. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv 2020, arXiv:1911.05722. [Google Scholar]
- Grill, J.B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.H.; Buchatskaya, E.; Doersch, C.; Pires, B.A.; Guo, Z.D.; Azar, M.G.; et al. Bootstrap your own latent: A new approach to self-supervised Learning. arXiv 2020, arXiv:2006.07733. [Google Scholar]
- Khosla, P.; Teterwak, P.; Wang, C.; Sarna, A.; Tian, Y.; Isola, P.; Maschinot, A.; Liu, C.; Krishnan, D. Supervised Contrastive Learning. arXiv 2021, arXiv:2004.11362. [Google Scholar]
- Li, Q.; He, B.; Song, D. Model-Contrastive Federated Learning. arXiv 2021, arXiv:2103.16257. [Google Scholar]
- Zhuang, W.; Wen, Y.; Zhang, S. Divergence-aware Federated Self-Supervised Learning. arXiv 2022, arXiv:2204.04385. [Google Scholar]
- Mu, X.; Shen, Y.; Cheng, K.; Geng, X.; Fu, J.; Zhang, T.; Zhang, Z. FedProc: Prototypical Contrastive Federated Learning on Non-IID data. arXiv 2021, arXiv:2109.12273. [Google Scholar] [CrossRef]
- Guo, Y.; Tang, X.; Lin, T. FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction. arXiv 2023, arXiv:2205.13462. [Google Scholar]
- Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images; Technical Report; University of Toronto: Toronto, ON, Canada, 2009. [Google Scholar]
- Wang, Z.; Fan, X.; Peng, Z.; Li, X.; Yang, Z.; Feng, M.; Yang, Z.; Liu, X.; Wang, C. FLGo: A Fully Customizable Federated Learning Platform. arXiv 2023, arXiv:2306.12079. [Google Scholar]
- Deng, L. The MNIST Database of Handwritten Digit Images for Machine Learning Research. IEEE Signal Process. Mag. 2012, 29, 141–142. [Google Scholar] [CrossRef]
- Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv 2017, arXiv:1708.07747. [Google Scholar]
- Debnath, A.K.; de Compadre, R.L.L.; Debnath, G.; Shusterman, A.J.; Hansch, C. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity. J. Med. Chem. 1991, 34, 786–797. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 3–6 December 2012; Pereira, F., Burges, C., Bottou, L., Weinberger, K., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2012; Volume 25. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar] [CrossRef]
- Glorot, X.; Bordes, A.; Bengio, Y. Deep Sparse Rectifier Neural Networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 11–13 April 2011; Volume 15, pp. 315–323. [Google Scholar]
- Song, L.; Fan, J.; Chen, D.R.; Zhou, D.X. Approximation of Nonlinear Functionals Using Deep ReLU Networks. arXiv 2023, arXiv:2304.04443. [Google Scholar] [CrossRef]
- Daubechies, I.; DeVore, R.; Foucart, S.; Hanin, B.; Petrova, G. Nonlinear Approximation and (Deep) ReLU Networks. arXiv 2019, arXiv:1905.02199. [Google Scholar] [CrossRef]
- Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S. How Powerful are Graph Neural Networks? In Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA, 6–9 May 2019. [Google Scholar]
- Zhang, J.; Hua, Y.; Wang, H.; Song, T.; Xue, Z.; Ma, R.; Guan, H. FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, Long Beach, CA, USA, 6–10 August 2023; ACM: New York, NY, USA, 2023; pp. 3249–3261. [Google Scholar] [CrossRef]
- Karimireddy, S.P.; Kale, S.; Mohri, M.; Reddi, S.J.; Stich, S.U.; Suresh, A.T. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML’20, Vienna, Austria, 12–18 July 2020; JMLR.org: Brookline, MA, USA, 2020. [Google Scholar]
Methods | MNIST | FMNIST | CIFAR-10 | CIFAR-100 | MUTAG | ||||
---|---|---|---|---|---|---|---|---|---|
FedAVG | 92.43 | 96.74 | 78.95 | 79.32 | 43.36 | 57.41 | 18.76 | 20.17 | 68.42 |
DITTO | 92.64 | 96.88 | 76.19 | 77.20 | 36.97 | 55.88 | 17.45 | 19.47 | 73.68 |
FedPer | 92.98 | 97.03 | 78.90 | 80.19 | 41.74 | 48.92 | 18.67 | 20.33 | 63.16 |
MOON | 93.05 | 97.35 | 77.57 | 79.62 | 42.06 | 57.06 | 18.82 | 20.97 | 78.95 |
FedCP | 89.85 | 94.49 | 72.57 | 73.21 | 37.13 | 39.57 | 13.49 | 14.54 | 69.81 |
FDCL (Ours) | 93.12 | 97.83 | 79.27 | 80.68 | 43.89 | 57.43 | 19.12 | 20.63 | 84.21 |
20% | 30% | 50% | 70% | 100% | |
---|---|---|---|---|---|
FedAVG | 37.35 | 40.49 | 43.36 | 43.97 | 44.38 |
DITTO | 35.79 | 36.34 | 36.97 | 38.96 | 41.41 |
FedPer | 36.45 | 39.08 | 41.74 | 41.93 | 43.36 |
MOON | 37.84 | 40.23 | 42.06 | 42.33 | 42.50 |
FedCP | 34.81 | 35.52 | 37.13 | 38.08 | 38.57 |
FDCL (Ours) | 38.42 | 41.54 | 43.89 | 44.62 | 44.92 |
0.05 | 0.1 | 0.3 | 0.5 | 0.7 | 1.0 | 2.0 | |
Acc | 40.52 | 42.06 | 39.60 | 35.73 | 34.31 | 36.23 | 28.55 |
5 | 20 | 50 | |
---|---|---|---|
FedAVG | 40.76 | 38.55 | 35.98 |
FedPer | 41.08 | 38.73 | 36.32 |
MOON | 41.22 | 39.45 | 37.04 |
FDCL (Ours) | 42.67 | 40.32 | 38.73 |
CIFAR-10 | CIFAR-100 | ||
---|---|---|---|
✗ | ✗ | 36.63 | 16.06 |
✗ | ✓ | 38.87 | 18.22 |
✓ | ✗ | 39.45 | 18.31 |
✓ | ✓ | 40.32 | 18.48 |
Method | CIFAR-10 | CIFAR-100 | MNIST |
---|---|---|---|
FedAVG | 53.76 s | 87.37 s | 56.47 s |
FedPer | 56.41 s | 92.40 s | 54.75 s |
Ditto | 89.83 s | 116.09 s | 82.01 s |
MOON | 66.77 s | 95.26 s | 64.24 s |
FDCL (Ours) | 68.32 s | 96.95 s | 67.47 s |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, M.; Fang, F. Dual-Layer Bidirectional Contrastive Federated Learning: Enhancing Feature Extraction and Personalization Simultaneously. Electronics 2025, 14, 2256. https://doi.org/10.3390/electronics14112256
Liu M, Fang F. Dual-Layer Bidirectional Contrastive Federated Learning: Enhancing Feature Extraction and Personalization Simultaneously. Electronics. 2025; 14(11):2256. https://doi.org/10.3390/electronics14112256
Chicago/Turabian StyleLiu, Moxuan, and Feifei Fang. 2025. "Dual-Layer Bidirectional Contrastive Federated Learning: Enhancing Feature Extraction and Personalization Simultaneously" Electronics 14, no. 11: 2256. https://doi.org/10.3390/electronics14112256
APA StyleLiu, M., & Fang, F. (2025). Dual-Layer Bidirectional Contrastive Federated Learning: Enhancing Feature Extraction and Personalization Simultaneously. Electronics, 14(11), 2256. https://doi.org/10.3390/electronics14112256