Heterogeneous Federated Learning via Knowledge Transfer Guided by Global Pseudo Proxy Data

Abstract

Federated learning with data free knowledge distillation enables effective and privacy-preserving knowledge aggregation by employing generators to produce local pseudo samples during client-side model migration. However, in practical applications, data distributions across different institutions are often non-independent and identically distributed (Non-IID), which introduces bias in local models and consequently impedes the effective transfer of knowledge to the global model. In addition, insufficient local training can further exacerbate model bias, undermining overall performance. To address these challenges, we propose a heterogeneous federated learning framework that enhances knowledge transfer through guidance from global proxy data. Specifically, a noise filter is incorporated into the training of local generators to mitigate the negative impact of low-quality pseudo proxy samples on local knowledge distillation. Furthermore, a global generator is introduced to produce global pseudo proxy samples, which, together with local pseudo proxy data, are used to construct a cross attention matrix. This design effectively alleviates overfitting and underfitting issues in local models caused by data heterogeneity. Extensive experiments on publicly available datasets with heterogeneous data distributions demonstrate the superiority of the proposed framework. Results show that when the Dirichlet distribution coefficient is 0.05, our method achieves an average accuracy improvement of 5.77% over popular baselines; when the coefficient is 0.1, the improvement reaches 6.54%. Even under uniformly distributed sample classes, our model still achieves an average accuracy improvement of 7.07% compared to other methods.

1. Introduction

With the rapid advancements in the internet, mobile devices, and high-performance computing, artificial intelligence (AI) has become a powerful tool across various domains. However, concerns about data ownership, privacy, and cybersecurity pose significant barriers to sharing intelligent models, particularly in sensitive areas. To address these issues, Google introduced Federated Learning (FL) in 2016 [1], enabling distributed model training without directly accessing raw data. This approach allows decentralized clients to collaboratively train a global model and has been widely applied in healthcare, biometrics, and natural language processing [2,3]. FL can be categorized into horizontal and vertical types based on the consistency of local model structures. In horizontal FL, client models with the same architecture are trained on local datasets and aggregated using weighted averaging to form a global model. This practical and scalable approach has gained traction in industry [4]. Nevertheless, FL still faces challenges arising from disparities in data collection, preprocessing, and model construction across clients. A critical issue, particularly in industrial settings, is the non-IID nature of user data, which leads to locally biased models and degrades global model performance and generalization. Local data heterogeneity remains a major obstacle to achieving high-accuracy federated models [5,6].

Knowledge distillation techniques have therefore been proposed to transfer knowledge by having the student model mimic the teacher model’s output distribution, reducing the impact of data heterogeneity without altering model parameters [7]. In conventional FL with knowledge distillation, clients act as teacher models, transmitting soft predictions to the server, which aggregates them to update the global model and distills the knowledge back to clients. However, this approach relies on high-quality proxy datasets, which conflicts with the FL principles of data privacy and local data isolation. Furthermore, the aggregated global knowledge may not effectively guide client-side training. A major challenge is achieving knowledge aggregation without accessible proxy datasets. To overcome this, recent work has introduced data-free knowledge distillation for FL, enabling knowledge transfer between clients and the server without requiring actual data, thus enhancing privacy [8]. The first architecture for data-free distillation, DAFL, was proposed by Chen et al. in 2019 [8], using a generative adversarial network (GAN) to synthesize data. Following this, Pan et al. developed Meta-KD in 2020 [9], integrating meta-learning to optimize the data generation distribution. These efforts sparked a surge in research on replacing real datasets with generated proxy data in FL. Zhu et al. proposed FedGen [10], combining data-free distillation with FL by training a generator at the server based on client predictions, which is then broadcast to clients to assist local model updates. While FedGen advances data-free distillation, it remains vulnerable to noise in generated pseudo data, which can hinder knowledge transfer and degrade global model performance. In addition, in scenarios with significant local data heterogeneity, reliance on local labels can introduce model bias, limiting global to local knowledge transfer.

To address these limitations, we propose a novel federated learning framework for heterogeneous data, FedKDG (Federated Knowledge Distillation with Global Pseudo Data Guidance). This framework leverages the global model’s knowledge to guide local model training, reducing the impact of data heterogeneity on local model bias. First, the global server uses real local labels as auxiliary data to train a generator that integrates knowledge from multiple clients. Clients then receive the global model and generator to train local generators, incorporating filters to suppress noisy generated samples. A cross attention matrix is created between global and local synthetic samples to enhance the diversity and effectiveness of pseudo data in global to local distillation. Finally, a global distillation loss function is introduced, utilizing local real sample labels to further facilitate knowledge transfer from the global model.

The main contributions of this paper are summarized as follows:

(1)

Global to local knowledge transfer to mitigate heterogeneity data induced bias: We propose a global knowledge-guided local model optimization module that effectively transfers knowledge from the global model to local models using global pseudo-data, thereby addressing the classification bias caused by data heterogeneity.

(2)

Noise filtered generation for robust pseudo data construction: We design an optimization and filtering mechanism for pseudo data generation, which mitigates the negative impact of noisy samples and ensures the fidelity of transferred knowledge.

(3)

Extensive empirical validation under heterogeneous settings: We validate the proposed approach on widely used benchmark datasets and demonstrate superior performance in terms of federated classification accuracy compared to state-of-the-art models under non-IID data distributions.

The remaining sections of this paper are organized as follows: Section 2 provides a detailed introduction and analysis of classic and widely adopted related works. Section 3 provides the three parts of the process framework of the FedKDG (our method). Section 4 presents the comparative experimental results and the analysis of the FedKDG algorithm parameters. Section 5 presents the conclusion.

2. Related Work

Federated Learning (FL) trains local models on client devices and aggregates their outputs to form a global model. Classical models, like Federated Averaging (FedAvg) [2], focus on weight-level aggregation but face weight divergence issues. PFNM [11] addresses this by aligning client parameters via weight matching and Bayesian nonparametrics for improved global integration. FedMA [12] enhances aggregation through CNN-LSTM combinations and alignment mechanisms. Recent approaches explore feature-level and hybrid aggregation. FedBE [13] integrates Bayesian ensembles with FL to aggregate high-quality models, while Fed2 [14] adapts feature structures and employs model fusion to improve convergence by capturing data distributions.

2.1. Heterogeneous Federated Learning

While existing methods can effectively aggregate local models into a global model, data heterogeneity arising from diverse edge devices poses significant challenges to knowledge transfer. Heterogeneous federated learning (FL) models, which can be categorized into data, model, and system heterogeneity, have emerged, with data heterogeneity being the most extensively addressed. Early solutions combined multi-task learning with Bayesian frameworks. For instance, MOCHA [3] applied alternating optimization for multi-task FL but is limited to convex models. To handle non-convex settings, VIRTUAL [15] introduced a hierarchical Bayesian network-based approach. Jiang et al. [16] integrated meta-learning to optimize local learning rates and server-side optimizers for personalized models. Astraea [17] improved model discriminability through data augmentation but relies on access to client data distributions, increasing vulnerability to security risks such as backdoor attacks. With the growing prominence of network attacks, achieving stable and secure knowledge aggregation has become an emerging and critical challenge in federated learning system design [18,19].

To balance global collaboration and local personalization, regularization and contrastive learning approaches have been explored. Acar et al. proposed Personalized Federated Learning (PFL) [20], which personalizes local training via gradient correction but is prone to catastrophic forgetting. Shoham et al. introduced FedCurv [21], leveraging Elastic Weight Consolidation (EWC) to preserve learned knowledge. Li et al. proposed FedProx [22], which adds a proximal term to FedAvg to improve convergence. Dinh et al. presented pFedMe [23], using the Moreau envelope to decouple personalized and global model optimization. Li et al. further introduced MOON [24], employing model contrastive learning to align local training via representation similarity. Building upon this, Mu et al. proposed FedProc [25], applying prototype-based contrastive learning to further mitigate heterogeneity. However, transmitting unprocessed outputs from client models poses privacy risks. To address this, Yoon et al. proposed FedMix [26], employing Mixup-based data augmentation to enhance privacy and data diversity. Xu et al. [27] used K-means clustering to filter malicious client data based on label trustworthiness.

2.2. Federated Learning with Knowledge Distillation

Previous approaches typically adjust model parameters to mitigate heterogeneity, but this can lead to knowledge loss or noise. Knowledge distillation (KD) has gained attention as an effective solution for knowledge transfer [7]. Sattler et al. proposed CFD [28] to reduce communication overhead using KD. He et al. introduced FedGKT [29], which transfers knowledge from lightweight CNNs on edge devices to larger global CNNs, easing computational load. Fang et al. proposed RHFL [30], using KD on unrelated public data, though it may not fully leverage local predictions. Huang et al. [31] introduced a latent embedding module with adversarial learning to distinguish private and public domains for better domain adaptation. To address data heterogeneity, Li et al. developed FedMD [32], combining KD with transfer learning to derive global knowledge from local predictions. Chang et al. proposed Cronus [33] to improve FL accuracy on public datasets, while Ozkara et al. [34] integrated KD to enhance training efficiency across heterogeneous data sources.

Despite progress, traditional KD methods rely heavily on the availability of high-quality proxy datasets, which may be impractical in real-world scenarios. To address this, Lin et al. proposed FedDF [35], an ensemble distillation framework that accelerates convergence via entropy minimization. Zhu et al. introduced FedGen [9], eliminating the need for real data by training a generator on client predictions. Zhang et al. developed FedFTG [36], a conditional generator-based approach to minimize global-local loss for effective distillation. DENSE [37], a two-stage data-free FL method, enables server-side training without data exchange. To tackle heterogeneity, Heinbaugh et al. proposed FedCVAE-Ens and FedCVAE-KD [38], using conditional variational autoencoders (CVAE) to reconstruct local tasks. Wu et al. introduced FedDKC [39], reducing inter-client knowledge variance through refinement strategies. However, many methods face catastrophic forgetting during iterative knowledge transfer. Zhang et al. proposed TARGET [40], mitigating forgetting by simulating global data transfer from past tasks. Zhu et al. introduced edTAD [41], a topology-aware data-free KD framework ensuring consistency in knowledge transfer, while Wang et al. presented DFRD [42], using an exponential moving average of generators to counter forgetting and preserve client knowledge.

Although promising, these methods still rely on generator-based pseudo-data, which can cause instability and hinder collaboration. Zhao et al. proposed FedF2DG [43], a generator-independent pseudo-data strategy that adaptively adjusts label distributions to reduce bias. However, it faces challenges with noisy pseudo-data and semantic mismatches. Additionally, most methods treat the server as the sole student model, overlooking model heterogeneity across clients.

3. Methodology

Existing data-free knowledge distillation methods in federated learning are constrained by the quality of the generator, which limits effective knowledge transfer. In addition, pseudo proxy datasets mainly focus on aggregating knowledge from local models into a global model, while the potential for the global model to support local updates is largely overlooked. This limitation reduces the ability to mitigate issues arising from Non-IID and heterogeneous data distributions. To address these challenges, we propose a heterogeneous federated learning framework that leverages globally generated pseudo proxy data to guide knowledge transfer. By training a global generator, the server produces pseudo-proxy data to facilitate global to local knowledge migration, enabling the server to play an active role that goes beyond simple parameter aggregation.

Our approach enables collaborative optimization between global and local models, eliminates reliance on real proxy data, and enhances knowledge transfer. It also reduces local model bias and mitigates global performance degradation caused by data heterogeneity. The model architecture is shown in Figure 1.

3.1. Problem Definition

The proposed federated learning framework consists of K clients, denoted as $k=1,2,\dots ,K$. Each client k holds a local dataset $D_k={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^{n_k}$, where $x_i\in X\subseteq R^d$, $y_i\in Y=\left\{0,1,\dots ,C\right\}$ represents the class label for a classification task, and $n_k$ is the number of samples held by client k. Given the heterogeneous nature of local data distributions in this setting, each client’s data follows a non-IID distribution $\widehat{p}\left(y_k\right)\sim Dir\left(\alpha \right)$, where $\alpha$ controls the sparsity of the Dirichlet distribution. The global data distribution is denoted by $p_k\left(x,y\right)$, and the heterogeneity across clients can be formally expressed as $\forall k\ne j,\widehat{p}\left(y_k\right)\ne \widehat{p}\left(y_j\right)$, and $\widehat{p}\left(y_k\right)\ne \widehat{p}\left(y_g\right)$.

In each communication round, the central server receives local model parameters ${\theta }_k$ and true labels Y from client data. A global generator $G_g$ is then trained to synthesize pseudo samples, which are used to facilitate knowledge transfer across clients. This results in an updated global server state and a globally aggregated model that captures rich shared information. Subsequently, the server distributes the updated global model parameters ${\theta }_g$ and the generator $G_g$ to all clients. These are used to refine local model parameters ${\theta }_k$, which are biased due to training on heterogeneous data. The federated learning process then proceeds iteratively, gradually optimizing toward a global model with improved generalization.

3.2. Localized Personalized Knowledge Transfer

Due to data privacy constraints, the global server cannot access the local data distributions used for training client models. To enable effective knowledge transfer from the global model to local models, we adopt a data-free knowledge distillation approach based on the DAFL framework. Specifically, a global generator $G_g$ corresponding to the global model is constructed and optimized iteratively on the server to produce pseudo-proxy samples with refined feature distributions, denoted as $D_g=\left(d_{g_1},d_{g_2},\dots ,d_{g_n}\right)$. These pseudo samples serve as the basis for knowledge distillation between the global model $M_g$ and each client model $M_k$, facilitating the extraction and transfer of global knowledge. To further enhance sample quality and reduce redundancy, a noise filter $F_g$ is introduced to eliminate irrelevant or noisy information during the generation process. To ensure the effectiveness of the generator $G_g$, the global server is allowed to observe the label distributions of training samples used by each client. Importantly, only the label Y is shared, not the raw data $D_k$, thereby preserving privacy while enabling the server to condition the generator on diverse local label distributions. The global knowledge transfer process is illustrated in Figure 2.

First, a random noise vector $z_g\sim N\left(0,I\right)$ is introduced as the input to the generator, and the distribution of the true labels $p\left(Y\right)$ from users’ local data is used as supervision to train the global generator $G_g$. This phase adopts the DAFL training paradigm to produce a pseudo sample set $D_g=G_g\left(z_g,p\left(Y\right)\right)$. To ensure diversity in the generated pseudo samples $D_g$, a diversity loss function $L_{divg}$ is introduced. By minimizing the distributional discrepancy among pseudo samples generated from different noise inputs, this loss promotes sample diversity while preventing excessive discriminative bias. The formulation is given as follows:

$$L_{divg}=E_{\left(z_{g_1},z_{g_2}\right)}\left[{∥G_g\left(z_{g_1},p\left(Y\right)\right)-G_g\left(z_{g_2},p\left(Y\right)\right)∥}_2^2\right]$$

(1)

where $z_{g1},z_{g2}$ denotes different noise inputs. For each client k, the pseudo samples $D_g$ generated by the global generator $G_g$ are fed into the local model for training. A cross-entropy loss is then constructed between the local model’s predictions $p_{{\theta }_k}$ and the client’s true label $y_i$, denoted as $L_{ce}$. In practice, each client’s cross-entropy loss is further weighted by the proportion of its true label distribution to account for data imbalance. The overall cross-entropy loss $L_{ce}$ is defined as the weighted sum of individual client losses, as formulated below:

$$L_{ceg}=-\sum _{k=1}^K{\displaystyle \frac{n_k}{N}}{E}_{\left(D_g,y_i\right)}\left[log{p}_{{\theta }_k}\left(y_i|D_g\right)\right]$$

(2)

The global generator $G_g$ is trained using a combined loss function that incorporates both the diversity loss $L_{divg}$ and the cross-entropy loss $L_{ceg}$. The total loss function $L_{G_g}$ is defined as follows:

$$L_{G_g}=L_{divg}+L_{ceg}$$

(3)

Second, since the pseudo samples $D_g$ generated by the global generator $G_g$ may contain substantial noise, a pseudo sample filter $F_g$ is introduced to ensure the effectiveness of the generated data. This selector filters out noisy or irrelevant information, making the retained pseudo samples ${D_g}^{\prime }=\left({d_{g_1}}^{\prime },{d_{g_2}}^{\prime },\dots ,{d_{g_n}}^{\prime }\right)$ more representative of the clients’ real data $D_k$. The selection process is formulated as follows:

$$F_g={\displaystyle \frac{\nabla f\left(D_g\right)-min\left(\nabla f\left(D_g\right)\right)}{max\left(\nabla f\left(D_g\right)\right)-min\left(\nabla f\left(D_g\right)\right)}}$$

(4)

where $\nabla f\left(·\right)$ is gradient function of the model, and $\nabla f\left(D_g\right)=\left[{\displaystyle \frac{\partial f}{\partial d_{g_1}}},{\displaystyle \frac{\partial f}{\partial d_{g_2}}},\dots ,{\displaystyle \frac{\partial f}{\partial d_{g_n}}}\right]$. Simultaneously, the global generator $G_g$ is trained as follows:

$${L_{ceg}}^{\prime }=-\sum _{k=1}^K{\displaystyle \frac{n_k}{N}}{E}_{\left({D_g}^{\prime },y_i\right)}\left[log{p_{{\theta }_k}}^{\prime }\left(y_i|{D_g}^{\prime }\right)\right]$$

(5)

where ${D_g}^{\prime }=F_g^T·D_g^T$. At this stage, the generator training loss can be expressed as:

$${L_{G_g}}^{\prime }=L_{divg}+{L_{ceg}}^{\prime }$$

(6)

Finally, in the current communication round, the server broadcasts the trained global generator $G_g$, global model parameters ${\theta }_g$, and the generated proxy dataset ${D_g}^{\prime }$ to each local client k. The global model parameters ${\theta }_g$ are directly assigned to initialize the local models, serving as the starting point for local training: ${\theta }_k^t={\theta }_g^t$.

3.3. Local Model Optimization Guided by Global Knowledge

Each client k receives the global parameters ${\theta }_g$ along with the trained global generator $G_g$. To ensure effective extraction and transfer of global knowledge, a local pseudo-random sample set $D_l=\left(d_{l_1},d_{l_2},\dots ,d_{l_n}\right)$ is generated by training a local generator $G_l$ using client-specific data. This local pseudo dataset, together with the global pseudo samples generated by $G_g$, is used to construct a cross-attention matrix that optimizes the distribution of global pseudo samples and facilitates global knowledge transfer. Simultaneously, a noise filter $F_l$ is incorporated during the local generator training to remove redundant or noisy information, further refining the distribution of the locally generated pseudo dataset. The detailed process is illustrated in Figure 3 and described as follows:

(1) Global Knowledge Transfer to Mitigate Local Data Heterogeneity.

First, a local generator $G_l$ is constructed for each client. Random noise vectors $z_l\sim N\left(0,I\right)$ and the distribution of the client’s local true labels $p\left(Y\right)$ are fed into the local generator for training, resulting in a local pseudo sample set $D_l=G_l\left(z_l,p\left(Y\right)\right)$. To ensure diversity in the generated samples $D_l$, a diversity loss function $L_{divl}$ is introduced, defined as follows:

$$L_{divl}=E_{\left(z_{l_1},z_{l_2}\right)}\left[{∥G_l\left(z_{l_1},p\left(Y\right)\right)-G_l\left(z_{l_2},p\left(Y\right)\right)∥}_2^2\right]$$

(7)

where $z_{l1},z_{l2}$ represents different initial noise vectors. The pseudo samples $D_l$ generated by the local generator $G_l$ are fed into the local model for training, and a cross-entropy loss $L_{cel}$ is constructed between the local model’s predictions ${p_{{\theta }_k}}^{″}$ and the user’s true labels $y_k$, as formulated below:

$$L_{cel}=-E_{\left(D_l,y_k\right)}\left[log{p_{{\theta }_k}}^{″}\left(y_k|D_l\right)\right]$$

(8)

The local generator $G_l$ is trained using a combined loss function that integrates the diversity loss $L_{divl}$ and the cross-entropy loss $L_{cel}$. The total loss function $L_{G_l}$ is defined as follows:

$$L_{G_l}=L_{divl}+L_{cel}$$

(9)

Since the pseudo samples $D_l$ generated by the local generator $G_l$ contain substantial noise, a pseudo sample filter $F_l$ is introduced to ensure the authenticity of the generated samples. This selector filters out noisy information, making the pseudo samples ${D_l}^{\prime }=\left({d_{l_1}}^{\prime },{d_{l_2}}^{\prime },\dots ,{d_{l_n}}^{\prime }\right)$ generated by $G_l$ more representative of the user’s real data, and ${D_l}^{\prime }=F_l^T·D_l^T$. The local generator $G_l$ is retrained as follows:

$${L_{cel}}^{\prime }=-E_{\left({D_l}^{\prime },y_k\right)}\left[log{p^{‴}}_{{\theta }_k}\left(y_k|{D_l}^{\prime }\right)\right]$$

(10)

At this stage, the generator training loss can be expressed as:

$${L_{G_l}}^{\prime }=L_{divl}+{L_{cel}}^{\prime }$$

(11)

After training, the local generator $G_l$ produces a local pseudo sample set ${D_l}^{\prime }$, which is used to facilitate knowledge transfer from the global model to the local model. Second, to enable global knowledge transfer and mitigate bias caused by local data heterogeneity during local model training, the local model inputs a batch of the user’s true label distribution $p\left(y_b\right)$ into the global generator $G_g$ to obtain pseudo samples $P_i=G_g\left(z_i,p\left(y_b\right)\right)$. These pseudo samples $P_i$ are then fed into the trained local model. A KL divergence loss is constructed between the local model’s predictions ${\ddot{p}}_{{\theta }_k}$ on the real batch and on the generated pseudo samples $p_{{\theta }_g}$. This loss $L_{lat}$ serves as a latent constraint for training the local model, guiding it with knowledge extracted from the global model on the server side via the generated pseudo samples. The formulation is as follows:

$$L_{lat}={\displaystyle \frac{1}{B}}\sum _{i=1}^B{D}_{KL}\left({\ddot{p}}_{{\theta }_k}\left(·|P_i\right)\left|\right|p_{{\theta }_g}\left(·|P_i\right)\right)$$

(12)

Simultaneously, to adapt the global model to the local data distribution, each user samples from their true label set $y_i$ and inputs the resulting label distribution $p\left(y_i\right)$ into the global generator $G_g$ to generate pseudo samples $P_j=G_g\left(z_j,p\left(y_i\right)\right)$. These pseudo samples $P_j$ are then used to train the local model by constructing a cross-entropy loss $L_{tea}$, which measures the discrepancy between the local model’s predictions ${\stackrel{⃛}{p}}_{{\theta }_k}$ and the user’s true labels $y_i$. The formulation is as follows:

$$L_{tea}=-\sum _{c=1}^{C}p\left(y_i=c\right)log{\stackrel{⃛}{p}}_{{\theta }_k}\left(y_i=c|P_j\right)$$

(13)

We enhance the personalized representation of global pseudo samples at the local level by leveraging cross-attention between globally and locally generated pseudo samples, thereby facilitating global knowledge transfer. Specifically, the true label distribution $p\left(y_b\right)$ of each user batch is projected onto a uniform distribution $p\left(y_a\right)$. A batch of labels sampled $y_{ab}$ from this uniform distribution is then input separately into both the global generator $G_g$ and the local generator $G_l$, producing global pseudo sample outputs $P_g=G_g\left(y_{ab}\right)$ and local pseudo sample outputs $P_l=G_l\left(y_{ab}\right)$, respectively. A global-local cross-attention matrix $A={\left(W_l·P_l\right)}^T\left(W_g·P_g\right)$, where $W_l$ and $W_g$ are linear projection matrices used to align the dimensions of the output matrices. Meanwhile, obtain the normalized matrix N:

$$Nor=softmax\left({\displaystyle \frac{A}{\sqrt{d}}}\right)$$

(14)

where the Softmax function is defined as $softmax\left(x_i\right)={\displaystyle \frac{e^{x_i}}{{\displaystyle \sum _{i=1}^N}{e}^{x_i}}}$. The normalized matrix $Nor$ is used as the final attention matrix, which is multiplied by the output of the global generator $P_g$ to construct a residual matrix. The resulting output $P_A=Nor{P}_g+P_g$ is then fed into the prediction model. Therefore, inspired by the attention mechanism, this study assigns attention weights to the outputs of the generative models, enabling dynamic focus on critical global information while disregarding less relevant parts. This approach helps the results closely approximate the global generator’s outputs, thereby enhancing the accuracy and efficiency of the detection model.

Finally, the samples $P_A$ are input into the global model and local model, respectively, for training. The server is the teacher model, while the client k is the student model. A KL divergence loss is constructed between the local model’s predictions ${\tilde{p}}_{{\theta }_k}$ and the global model’s predictions ${p_{{\theta }_g}}^{\prime }$, serving as the distillation loss for training the local model. This allows the powerful global model to guide the training of the local model. The formulation is as follows:

$$L_{disg}=D_{KL}\left({\tilde{p}}_{{\theta }_k}\left(·|P_A\right)\left|\right|{p_{{\theta }_g}}^{\prime }\left(·|P_A\right)\right)$$

(15)

(2) Construction of Personalized Loss for Local Model. Each client inputs a batch of real local data $x_b$ and corresponding true labels $y_i$ into the local model k for training. The negative log-likelihood (NLL) loss is employed to minimize the prediction output of the local model ${\dot{p}}_{{\theta }_k}$, serving as the prediction loss $L_{pre}$ for training. This enhances both the training speed and stability of the model. The loss is defined as follows:

$$L_{pre}=-{\displaystyle \frac{1}{B}}\sum _{i=1}^{B}log{\dot{p}}_{{\theta }_k}\left(y_i|x_b\right)$$

(16)

where B denotes the batch size.

Meanwhile, a batch of the user’s real data $x_b$ is simultaneously input into both the global model and the local model for training. In this process, the server functions as the teacher model, and the client serves as the student model. A KL divergence loss is constructed between the prediction outputs of the local ${\stackrel{⌢}{p}}_{{\theta }_k}$ and global models ${p_{{\theta }_g}}^{″}$, serving as the distillation loss for training the local model. This facilitates knowledge transfer from the server to the client using each batch of real user data for auxiliary training. The formulation is as follows:

$$L_{dis}={\displaystyle \frac{1}{B}}\sum _{j=1}^B{D}_{KL}\left({\stackrel{⌢}{p}}_{{\theta }_k}\left(·|x_b\right)\left|\right|{p_{{\theta }_g}}^{″}\left(·|x_b\right)\right)$$

(17)

(3) Unified Training of the Local Model. To obtain a personalized local model that mitigates the effects of data heterogeneity, we employ a combination of the prediction loss $L_{pre}$, latent construct loss $L_{lat}$, cross-entropy loss $L_{tea}$, generative distillation loss $L_{disg}$, and distillation loss $L_{dis}$. These components are integrated into a final total loss function $L_k$ used to train the local model k, as formulated below:

$$L_k=L_{pre}+L_{lat}+L_{tea}+L_{disg}+L_{dis}$$

(18)

The overall loss function in this work is composed of multiple components, as shown in the above equation. During practical implementation of the algorithm, it is necessary to ensure that all loss components are maintained at a comparable scale, so as to prevent the training process from being dominated by any single loss term.

3.4. Global Aggregation of Local Models

Each local user shares their true label distribution $y_i$ and the optimized local model parameters ${\theta }_k^{t+1}$ with the server S. The global model parameters ${\theta }_g$ on the server are updated by performing a weighted average of all local models, where the weights are determined by the proportion of samples held by each user k. The update rule is defined as follows:

$${\theta }_g^{t+1}=\sum _{k=1}^K{\displaystyle \frac{n_k}{N}}{\theta }_k^{t+1}$$

(19)

where $N={\displaystyle \sum _{k=1}^K}{n}_k$.

Meanwhile, if any local model is updated, knowledge is transferred from the local model to the global model using the data-free distillation method described in Section 3.2. This enables the iterative process illustrated in Figure 1.

4. Experiments

4.1. Dataset Description

(1) 

Benchmark Datasets

To evaluate the effectiveness of the proposed model, four benchmark datasets widely used in the field of federated learning are selected: MNIST (Modified National Institute of Standards and Technology), EMNIST (Extended MNIST), CIFAR-10 (Canadian Institute for Advanced Research), and CIFAR-100. These datasets are representative in terms of data complexity, application scenarios, and experimental design. All experimental datasets employed in this paper are classic public datasets.

a. MNIST. MNIST is a benchmark dataset for handwritten digit recognition, containing 60,000 images. Each 28 × 28 grayscale image represents one of 10 digit classes (0–9). At about 170 MB, MNIST is ideal for quickly validating federated learning algorithms. It is often used to simulate non-IID scenarios through random partitioning or label-specific allocation.

b. EMNIST. EMNIST extends MNIST by adding 62 classes of handwritten characters, including digits and both uppercase and lowercase letters. It contains 814,255 samples, with the same 28 × 28 grayscale image format as MNIST. The visual similarity between certain letter classes, such as uppercase ‘C’ and lowercase ‘c’, increases classification difficulty, offering a more challenging task for evaluating model robustness.

c. CIFAR-10. CIFAR-10 contains 60,000 32 × 32 RGB images across 10 object categories, with a 5:1 train-test split. It introduces greater variability in pose, background, and lighting, increasing challenges for feature extraction. As a result, complex models like Convolutional Neural Networks (CNNs) are often necessary for effective learning.

d. CIFAR-100. CIFAR-100, released by the Canadian Institute for Advanced Research in 2009, contains 100 fine-grained object categories, including animals, plants, and vehicles. It comprises 60,000 RGB images (32 × 32 pixels), with 50,000 for training (500 per class) and 10,000 for testing (100 per class). The 100 classes are grouped into 20 superclasses (e.g., mammals, flowers), enabling multi-granularity classification tasks.

Table 1. Configuration of Key Parameters.

Parameter Name	Value
Number of Clients	20
Global Training Rounds	200
Local Training Rounds	20
Local Training Batch Size	32
Local Training Learning Rate	0.01
Generator Batch Size	32
Backbone Network	ResNet-18

summarizes the roles of these datasets in federated learning research.

(2) 

Dirichlet-based Non-IID Datasets

In federated learning, data distribution modeling and partitioning are crucial for model performance. The Dirichlet distribution, a multivariate probability distribution, is widely used to simulate non-IID scenarios due to its ability to capture uncertainty in probability vectors. The concentration parameter $\alpha$ controls distribution skewness: a small $\alpha$ (e.g., $\alpha \to 0$) results in high heterogeneity, while a larger $\alpha$ (e.g., $\alpha \to 1$) yields a more uniform distribution, approximating IID.

In this study, we evaluate the impact of three different Dirichlet concentration parameters $\alpha$ on the partitioning of MNIST, EMNIST, CIFAR-10, and CIFAR-100 datasets among 20 users. The comparative experimental results under these settings are illustrated in Figure 4, Figure 5 and Figure 6, which visualize the label distributions across users for varying $\alpha$ values. In each figure, subfigure (a) corresponds to a small $\alpha =0.05$, subfigure (b) to a moderate $\alpha =0.1$, and subfigure (c) to a large $\alpha =1$, showing the progressive reduction in data heterogeneity. The horizontal axis represents user IDs (1–20), and the vertical axis denotes label IDs (0–9 for MNIST and CIFAR-10, 0–25 for EMNIST, where a 26-class subset is used). The size of each dot indicates the quantity of samples a user holds for a particular label—the larger the dot, the more samples assigned. Overall, when $\alpha =0.05$ is small (e.g., $\alpha \to 0$), the data distribution is extremely non-IID: some users may hold hundreds of times more samples for a given label than others, while some labels may be entirely absent for certain users. This high degree of heterogeneity poses a significant challenge for federated learning. When $\alpha$ is moderate (e.g., $\alpha =1$), the heterogeneity is mitigated to some extent, though the distribution still deviates from a fully IID scenario.

In all experiments, 65% of each dataset was allocated as the training set, and 35% as the testing set, to be partitioned among users under the different Dirichlet settings.

4.2. Experimental Setting and Evaluation Metrics

The experimental environment and parameter configuration in this study strictly adhere to the latest research standards in the field of federated learning, ensuring reproducibility and scientific rigor. The hardware platform is equipped with an NVIDIA GeForce RTX 4070 GPU, which provides substantial acceleration for parallel computation in deep learning. The operating system is Windows 10, and PyTorch 1.10 is adopted as the deep learning framework, offering flexible support for distributed training in federated learning. GPU acceleration is enabled through CUDA and cuDNN. The source code for the proposed model has been made publicly available at: https://github.com/LNNU-computer-research-526/FedKDG, accessed on 11 December 2025.

In this study, the ResNet-18 architecture serves as the base model for network construction. A total of 20 clients participate in the federated training process. Detailed training parameters are presented in Table 1.

To fairly evaluate the effectiveness of the proposed model, classification accuracy is employed as the primary evaluation metric for performance assessment and comparative analysis. Classification accuracy is a commonly used metric that measures a model’s ability to correctly classify samples. It is defined as the ratio of the number of correctly predicted samples to the total number of samples in the test dataset. This can be expressed by the following formula:

$$accuracy={\displaystyle \frac{TP+TN}{TP+FP+FN+TN}}$$

(20)

where $TP$ is the true rate, which is predicted to be a positive example and is actually a positive example. $FP$ is a false positive example, which is predicted to be a positive example but is actually a negative example. $FN$ is a false negative example, which is predicted to be a negative example but is actually a positive example. $TN$ is a true negative example.

4.3. Comparative Analysis of Experimental Results

To evaluate the effectiveness of the proposed federated learning method FedKDG, five representative and widely adopted federated learning approaches FedAvg, FedProx, FedDistill, FedEnsemble, and FedGen are selected for comparative analysis.

Table 2. Accuracy Comparison of Mainstream Federated Learning Methods on MNIST Dataset under Different Data Heterogeneity (Bold for the best result).

Method	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=1$
FedAvg	88.43%	90.31%	94.73%
FedProx	87.94%	90.21%	94.55%
FedDistill	60.29%	60.89%	80.06%
FedEnsemble	89.35%	91.50%	94.70%
FedGen	94.52%	95.52%	97.29%
FedKDG	97.08%	98.83%	99.27%

summarizes the classification accuracy results of FedKDG and the five baseline methods under three different data heterogeneity scenarios on the MNIST dataset. As shown in the comparison, the proposed FedKDG method effectively mitigates local model heterogeneity by incorporating globally balanced distribution information. Even under severe data heterogeneity, FedKDG achieves a classification accuracy of 97.08%, demonstrating a substantial improvement over the widely used FedAvg approach. Moreover, under scenarios with moderate and low levels of local heterogeneity, FedKDG maintains high accuracy rates of 98.83% and 99.27%, respectively. These results highlight the model’s strong ability to overcome the challenges posed by data heterogeneity, outperforming other popular federated learning methods.

This study also reports comparative experimental results on the EMNIST dataset, which features more similar sample characteristics, as shown in

Table 3. Accuracy Comparison of Mainstream Federated Learning Methods on EMNIST Dataset under Different Data Heterogeneity (Bold for the best result).

Method	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=1$
FedAvg	66.89%	70.46%	78.53%
FedProx	65.93%	69.67%	77.71%
FedDistill	40.52%	45.15%	60.26%
FedEnsemble	67.33%	70.69%	78.65%
FedGen	75.06%	78.53%	84.20%
FedKDG	88.47%	93.00%	94.61%

. On this extended version of the MNIST dataset, the proposed method continues to demonstrate strong performance. Under a high degree of data heterogeneity, FedKDG achieves an accuracy of 88.47%. The EMNIST dataset contains a larger number of hard samples with similar feature representations across different classes, posing a greater challenge for model training. As a result, the classification accuracy of the five baseline methods drops significantly. In contrast, the proposed method, which incorporates global knowledge from the global model, effectively distinguishes between different classes.

Under moderate heterogeneity, FedKDG reaches an accuracy of 93.00%, outperforming previous methods by nearly 20 percentage points. Even in scenarios with balanced data distribution, where all models show improved performance, FedKDG still achieves the highest accuracy of 94.61%. These results demonstrate that the proposed method not only overcomes the challenges of severe data heterogeneity but also performs competitively on datasets with a larger number of classes.

To thoroughly evaluate and analyze the effectiveness of the proposed model, this study compares the accuracy of federated learning models under three different data heterogeneity scenarios on the color image dataset CIFAR-10, as shown in

Table 4. Accuracy Comparison of Mainstream Federated Learning Methods on CIFAR-10 Dataset under Different Data Heterogeneity (Bold for the best result).

Method	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=1$
FedAvg	33.29%	39.85%	48.46%
FedProx	33.20%	39.38%	48.05%
FedDistill	41.79%	33.19%	28.84%
FedEnsemble	36.81%	42.13%	49.30%
FedGen	40.89%	45.29%	53.42%
FedKDG	45.85%	51.21%	64.48%

. As observed in the table, FedKDG achieves accuracy rates of 45.85% and 51.21% under two heterogeneous settings, both outperforming other popular methods. Furthermore, under the third scenario, FedKDG attains an accuracy of 64.48%, demonstrating superior capability in federated knowledge transfer.

In addition, we validate and compare the effectiveness of the proposed model on the more challenging CIFAR-100 dataset, which includes a larger number of categories, as shown in

Table 5. Accuracy Comparison of Mainstream Federated Learning Methods on CIFAR-100 Dataset under Different Data Heterogeneity (Bold for the best result).

Method	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=1$
FedAvg	14.35%	17.56%	21.07%
FedProx	14.31%	17.35%	20.89%
FedDistill	18.01%	14.62%	12.54%
FedEnsemble	15.87%	18.56%	21.43%
FedGen	17.63%	19.95%	23.23%
FedKDG	19.76%	22.42%	28.04%

. The results demonstrate that under a heterogeneity level of 0.05, FedKDG achieves a classification accuracy of 19.76%, outperforming other models in terms of recognition accuracy. When the heterogeneity level increases to 0.1, FedKDG attains an accuracy of 22.42%, and under a highly heterogeneous setting with a level of 1, the accuracy significantly improves to 28.04%, representing an average improvement of approximately 7% over baseline methods.

These findings indicate that FedKDG exhibits strong robustness in multi-channel color image classification tasks. Even in the presence of highly non-IID data, the model maintains stable knowledge transfer capability and generalization performance.

Based on the overall statistical results, FedKDG demonstrates outstanding performance across all datasets. Specifically, it achieves an average accuracy of 98.34% under three levels of data heterogeneity on the MNIST dataset, 91.98% on the EMNIST dataset, 53.74% on the CIFAR-10 dataset, and 23.41% on the CIFAR-100 dataset. FedKDG consistently delivers satisfactory results across all evaluated scenarios.

To facilitate intuitive observation and analysis, convergence curves under various heterogeneity settings across different datasets are plotted. Figure 7 illustrates the performance of FedKDG and the reproduced results of the five baseline federated learning methods under varying degrees of heterogeneity for each dataset. As the training rounds progress, the figures clearly show the trend in prediction accuracy for each method. Notably, FedKDG quickly surpasses the accuracy of all other methods under every heterogeneity setting on the MNIST dataset. Moreover, all models converge stably to their respective optimal values.

For the EMNIST dataset, the proposed model still converges rapidly to its optimal performance, as illustrated in Figure 8. Moreover, compared to other methods, FedKDG demonstrates a clear advantage in test accuracy under highly heterogeneous conditions.

CIFAR-10 is a more challenging color image dataset compared to the others. Under conditions of high heterogeneity, the proposed model’s performance is initially limited by the quality of the generator and does not rapidly reach the optimal results. However, as the number of training rounds increases, the model effectively mitigates noise through the pseudo-data information filtering mechanism and addresses data heterogeneity via global knowledge transfer. As shown in Figure 9, the model ultimately achieves the best experimental results.

To further analyze the federated learning performance of the proposed model on heterogeneous datasets, we examined the number of classes per client in the EMNIST and CIFAR-10 datasets, as illustrated in Figure 10 and Figure 11. As shown in Figure 10, despite EMNIST containing a total of 26 classes, most clients have training data encompassing fewer than 10 classes. Even under such highly heterogeneous training conditions, the proposed model achieves higher accuracy compared to the popular FedGen model, as demonstrated in Figure 10. This result highlights the superior generalization ability and robustness of our approach. Similarly, although CIFAR-10 contains only 10 classes, its significant heterogeneity results in most clients having local training data comprising only 3 to 4 classes, as depicted in Figure 11. Under these circumstances, our model still attains better federated learning accuracy, achieving up to 74% accuracy even for clients with training data from only 2 classes, as shown in Figure 11.

Therefore, the proposed model demonstrates satisfactory accuracy, generalization, and robustness in heterogeneous data environments compared to existing popular federated learning methods.

Based on the above, in the domain of federated learning with heterogeneous data, the proposed FedKDG method demonstrates significant improvements over other federated learning approaches in terms of both accuracy and convergence performance. This indicates that the involvement of the global model in the training process plays a substantive role, reflecting the strength of the global model and its effective aggregation of beneficial knowledge for client models. The use of dual generators combined with a global model in a data-free knowledge distillation federated learning framework clearly offers greater advantages compared to traditional federated learning methods. This suggests that deep convolutional neural networks can effectively capture hidden, highly nonlinear relationships, and when paired with a knowledge-rich global model, both accuracy and convergence can be further enhanced.

4.4. Analysis of Module Effectiveness

Based on the experimental and comparative results presented in the previous section, the proposed model achieves superior recognition accuracy. This improvement fundamentally stems from the local model training process, where the cross-attention module integrates global information while preserving local personalized features, thereby enhancing local recognition performance. To better illustrate this, we visualize the feature distributions of samples generated by the local and global generators, as shown in Figure 12, Figure 13 and Figure 14. In each figure, panel (a) depicts the feature visualization of samples generated by the global generator, showing diverse distributions that represent multiple classes with relatively balanced class distributions. Panel (b) shows the feature visualization of samples generated by the local generator, where the feature distribution is highly imbalanced. Panel (c) presents the feature distribution of samples after global feature correction. It is evident that these samples not only retain the personalized representations of the local model but also exhibit a more balanced class distribution and reduced heterogeneity. Using such samples for knowledge distillation helps to mitigate local model bias and improve recognition accuracy.

Furthermore, from Figure 12, Figure 13 and Figure 14, it can be observed that under different levels of data heterogeneity, the proposed method effectively alleviates local information heterogeneity bias. Even when local heterogeneity is pronounced, the model optimizes local information through the global model, generating pseudo-samples containing rich class information. These pseudo-samples are then used in subsequent knowledge distillation, enhancing the local model’s accuracy and generalization ability.

4.5. Ablation Study

To further validate the effectiveness of FedKDG in heterogeneous federated learning, this section conducts ablation experiments under the heterogeneity setting of the CIFAR-10 dataset. The goal is to verify the indispensability of each component in the proposed method. The experimental results are summarized in

Table 6. Ablation study results on the CIFAR-10 dataset.

Ex	Global Generator	Filter	Global Distillation	Local Distillation	Accuracy
(1)	✗	✗	✗	✗	33.29%
(2)	✓	✗	✗	✗	41.64%
(3)	✓	✓	✗	✗	41.88%
(4)	✓	✓	✓	✗	41.50%
(5)	✓	✓	✓	✓	45.85%

.

(1)

Train the local model solely based on the prediction loss between the local model’s output and the true labels from a single batch of user data.

(2)

Building on experiment (1), a global generator is trained on the server side. The final total loss for training the local model includes both the cross-entropy loss between the local model’s predictions and all true user labels, and a latent loss measuring the discrepancy between the local model’s predictions on real user data and those on pseudo-samples generated by the global generator. No pseudo-sample filter is applied here; the global generator attempts to produce pseudo-samples that approximate the user’s real data as closely as possible.

(3)

Based on experiment (2), a pseudo-sample filter is added to assist the training of the global generator. This filter removes noisy information from the generated samples, resulting in more realistic pseudo-samples.

(4)

Building on experiment (3), knowledge distillation from the global model to the local model is incorporated during local training. Specifically, the KL divergence between the local model’s and global model’s predictions is added as a distillation loss with a weighting factor of 0.01, enabling knowledge transfer between the server and clients.

(5)

Extending experiment (4), a local generator is randomly assigned to one user for training, analogous to the global generator. A pseudo-sample filter is also applied to the local generator to remove noise. During local model training, an attention mechanism is introduced to focus the generator’s output on key global information. Additionally, a KL divergence loss between the local model’s and global model’s predictions is constructed as a generative distillation loss with a weight of 0.01. This enables the powerful global model to guide the training of the local model, with knowledge distillation from experiment (4) serving as auxiliary supervision.

5. Conclusions

To address the challenges of local model training bias and ineffective knowledge transfer in federated learning under non-independent and identically distributed (Non-IID) data environments, this paper proposes a novel federated learning framework based on data-free knowledge distillation. During the transfer of local models to the global model, a noise sample filtering mechanism is introduced to mitigate the negative impact of low-quality pseudo proxy samples generated by local generators. Meanwhile, during the global to local model transfer phase, a global pseudo proxy supervision module is constructed to alleviate overfitting and underfitting issues caused by data heterogeneity across clients, thereby enhancing model robustness and generalization.

Extensive experiments under varying degrees of data heterogeneity demonstrate that the proposed method outperforms mainstream federated learning baselines in task accuracy. However, the current framework has the following limitations: (1) it is designed specifically for knowledge aggregation and federated learning of local models in image classification tasks and has not been extended to local models in natural language processing or time series prediction; (2) it primarily addresses federated learning with non-balanced local data distributions, without fully considering heterogeneity in local model architectures; (3) its application assumes a secure network environment without attacks and does not adequately account for model performance and robustness under network attacks. Based on these limitations, future research will focus on addressing the above challenges. In particular, for the third point, we plan to leverage low-quality samples in the generated pseudo-proxy dataset, constructing them as attack datasets to train the global model, thereby enhancing its stability and robustness.