Enhancing Privacy and Communication Efficiency in Federated Learning Through Selective Low-Rank Adaptation and Differential Privacy

Abstract

Federated learning (FL) enables collaborative model training without centralizing raw data, but its application to large-scale vision models remains constrained by high communication cost, data heterogeneity, and privacy risks. Furthermore, in real-world applications such as autonomous driving and healthcare, model updates can inadvertently expose sensitive information even without direct data sharing. This highlights the need for frameworks that balance privacy, efficiency, and accuracy. The current approach to addressing information exposure involves encrypting data by incorporating additional encoding. However, such approaches to encrypting data significantly increase communication costs. In this paper, we propose Federated Share-A Low-Rank Adaptation with Differential Privacy (FedSA-LoRA-DP), a parameter-efficient and privacy-preserving federated learning framework. The framework combines selective aggregation of low-rank parameters with Differential Privacy (DP), ensuring that only lightweight components are shared while formally bounding individual data influence. Since DP simply perturbs the numeric values of existing parameters without altering their dimensionality or structure, it does not increase communication cost. This design allows FedSA-LoRA-DP to provide strong privacy guarantees while maintaining communication efficiency and model accuracy. Experiments on CIFAR-100, MNIST, and SVHN datasets demonstrate that the proposed framework achieves accuracy comparable to non-private counterparts, even under heterogeneous non-independent and identically distributed data and partial client participation. These results demonstrate that integrating differential privacy into low-rank adaptation enables privacy-preserving and communication-efficient federated learning without sacrificing model performance across heterogeneous environments.

1. Introduction

Modern applications in fields such as autonomous driving, healthcare, and smart-city infrastructure depend on vast amounts of data that are spread across different devices and organizations. In the case of autonomous vehicles, for instance, every car constantly records sensory information that could, if jointly utilized, improve perception and driving-assistance models and thus contribute to safer roads [1,2,3,4]. However, centralizing such data is often infeasible due to concerns about privacy, data protection laws, and practical limitations.

Federated Learning (FL) enables distributed training without centralizing raw data [5,6,7]. Each client trains a local model on its private dataset and transmits only model updates to a central server, which aggregates them to form a global model. Although this approach mitigates privacy risks, traditional algorithms such as Federated Averaging (FedAvg) still require exchanging the information of the entire model at every communication round, leading to significant communication overhead [4,5].

To address these challenges, Parameter-Efficient Fine-Tuning (PEFT) has been introduced as a means to reduce communication and computation costs [8,9,10]. Low-Rank Adaptation (LoRA) inserts small trainable matrices $\mathit{A}$ and $\mathit{B}$ into pretrained layers, enabling fine-tuning with fewer parameters [11].

Although the FL does not involve the exchange of raw data during the update process, it remains vulnerable to privacy leakage from shared updates. Gradient inversion attack [12,13,14] and membership inference attack [15,16] have shown that private information can be reconstructed from gradients. Consequently, it is crucial to protect not only data but also shared parameters.

Differential Privacy (DP) provides a formal privacy guarantee by injecting calibrated noise into gradients, thus bounding the influence of any individual data sample [17,18,19,20]. Recent DP-based FL frameworks, adaptive local mechanisms, and feedback-controlled approaches aim to achieve a balance between privacy protection and model accuracy [21,22,23]. Although these approaches improve privacy protection, they often add computational complexity or reduce model accuracy, especially under strict privacy budgets. Furthermore, data heterogeneity in non-i.i.d. settings causes the global model to converge toward an average representation that fits no client well, resulting in degraded overall accuracy [24,25].

In this paper, we propose FedSA-LoRA-DP, a novel FL framework that integrates DP and leverages LoRA for efficient model adaptation. In the framework, we share only the $\mathit{A}$ matrices between clients and the server and apply the DP to them to ensure secure aggregation. This design maintains the same amount of transmitted data even when differential privacy noise is added, as the noise only perturbs the numeric values of the $\mathit{A}$ matrices. Through experiments conducted under various conditions, we confirm that the proposed framework maintains a high accuracy of the model even in a non-independent and identically distributed (non-i.i.d.) environment, demonstrating its robustness and practical effectiveness.

We summarize our contributions as follows:

• We design a selective aggregation strategy that aggregates only the LoRA ${\mathit{A}}_i$ matrices while keeping the ${\mathit{B}}_i$ matrices local to each client, achieving both communication efficiency and personalization.
• We incorporate DP into the shared $\mathit{A}$ matrices, providing formal privacy guarantees without increasing communication overhead.
• We demonstrate through experiments on multiple datasets (CIFAR-100, MNIST, and SVHN) that FedSA-LoRA-DP achieves comparable accuracy to non-private baselines even under heterogeneous and partial participation conditions.

This paper is organized as follows. Section 2 reviews related work on PEFT, FL, and DP. Section 3 introduces our proposed method, which integrates selective Aggregation and LoRA with DP. Section 4 presents the experimental setup and performance evaluation under various conditions. Section 5 discusses the implications of the results, including the effects of data heterogeneity, LoRA rank, dataset characteristics, and models. Finally, Section 6 concludes the paper.

2. Related Work

2.1. Federated Learning and Communication Efficiency

To address high communication cost and client heterogeneity, several strategies have been developed to reduce the amount of data exchanged while maintaining model accuracy. Sattler et al. [4] proposed sparse update compression to reduce bandwidth usage. Li et al. [26] proposed FedProx, an additional regularization term to stabilize optimization under non-i.i.d. conditions, while Wang et al. [27] introduced FedNova, in which client updates are normalized to mitigate the impact of heterogeneous local training. Other communication-efficient approaches leverage gradient quantization and periodic aggregation to reduce transmission overhead and maintain scalability in federated systems [28,29,30].

While these methods improve scalability, they still transmit large parameter sets and may underperform in highly heterogeneous settings.

2.2. Parameter-Efficient Fine-Tuning

PEFT has emerged as a complementary solution to reduce communication and computational load by limiting the number of trainable parameters. Instead of updating all parameters, methods such as Adapter-Tuning [8], Prefix-Tuning [9], and Prompt-Tuning [10] learn lightweight modules that preserve most of the pretrained model’s knowledge. LoRA [11] has attracted particular attention by introducing trainable low-rank matrices $\mathit{A}$ and $\mathit{B}$ into existing layers, achieving both parameter and communication efficiency without degrading performance.

Aggregating the LoRA matrices $\mathit{A}$ and $\mathit{B}$ in an FL setting presents significant challenges. If the server directly aggregates the $\mathit{A}$ and $\mathit{B}$ matrices and broadcasts them to all clients, aggregation errors occur. Specifically, in an FL task with K clients, the model update of each client can be represented by two low-rank matrices, ${\mathit{A}}_i$ and ${\mathit{B}}_i$, introduced by LoRA. After aggregation and broadcasting on the server, the model update for each client is given as follows:

$${\displaystyle \frac{1}{K}}({\mathit{B}}_1+{\mathit{B}}_2+\dots +{\mathit{B}}_K){\displaystyle \frac{1}{K}}({\mathit{A}}_1+{\mathit{A}}_2+\dots +{\mathit{A}}_K),$$

(1)

which is different from the proper model update ${\displaystyle \frac{1}{K}}({\mathit{B}}_1{\mathit{A}}_1+{\mathit{B}}_2{\mathit{A}}_2+\dots +{\mathit{B}}_K{\mathit{A}}_K){\mathit{A}}_0$.

To address this challenge, some methods have been explored [31,32,33]. For instance, Sun et al. [32] proposed the method that updates only $\mathit{B}$ and freezes the $\mathit{A}$. Thus, the local update on each client is ${\displaystyle \frac{1}{K}}({\mathit{B}}_1+{\mathit{B}}_2+\dots +{\mathit{B}}_K){\mathit{A}}_0$ where $A_0$ denotes the initialized and fixed weights. Guo et al. [31] proposed the method that aggregates only the $\mathit{A}$ matrices across clients while keeping the $\mathit{B}$ matrices local. Thus, the local update on each client i is ${\mathit{B}}_i{\displaystyle \frac{1}{K}}({\mathit{A}}_1+{\mathit{A}}_2+\dots +{\mathit{A}}_K)$.

Despite these frameworks significantly reducing communication overhead, most have yet to achieve a well-balanced integration of model accuracy, privacy preservation, and efficiency. Ensuring consistent performance while maintaining strong privacy guarantees remains an open challenge.

2.3. Differential Privacy in LoRA-Based Federated Learning

Recent advances have explored combining DP with PEFT in FL to enhance privacy and communication efficiency. Liu et al. [34] proposed the method which integrates LoRA and DP to fine-tune large language models in a distributed manner. While this approach effectively reduces communication cost and provides formal privacy guarantees, it aggregates both LoRA matrices $\mathit{A}$ and $\mathit{B}$ across clients, thereby limiting client-level personalization and introducing potential bias when data distributions are non-i.i.d.

Sun et al. [32] proposed the method which freezes the shared $\mathit{A}$ matrix and updates only $\mathit{B}$ to simplify communication and avoid aggregation inconsistency. Although this design improves communication efficiency, it constrains model expressivity since the shared representation $\mathit{A}$ remains fixed throughout training.

Overall, while these DP-integrated LoRA frameworks have made progress in improving privacy and communication efficiency in federated settings, most still face limitations in balancing personalization, model expressivity, and scalability under heterogeneous data conditions. A more comprehensive approach that jointly achieves privacy preservation, communication efficiency, and robustness to client diversity remains an open challenge.

2.4. Privacy Attacks and Defense Mechanisms

Although FL reduces direct data sharing, it does not fully eliminate privacy risks. There are studies that have shown that sensitive information can be extracted from shared model updates. For example, gradient inversion attack [12,13,14] reconstructs input samples by exploiting gradients. Membership inference attacks [15,16] reveal whether specific data points were included in training. These studies argue that even without central data aggregation, model parameters themselves can act as unintended information channels.

To mitigate such risks, several defense strategies have been proposed. For instance, blockchain-assisted FL has been investigated as a way to ensure transparency and accountability across decentralized systems [35,36]. Nevertheless, most of these methods either introduce high computational overhead or compromise model utility when applied to large-scale models.

DP provides a rigorous framework to quantify privacy guarantees by bounding the influence of any individual data sample [17,18,19]. Within FL, DP has been applied to protect model updates from inference and reconstruction attacks. Since DP simply perturbs the numeric values of existing parameters without altering their dimensionality or structure, it does not increase communication cost, in contrast to cryptographic approaches that require additional encoded data [37,38].

Recent research has also provided empirical evidence that the theoretical guarantees of differential privacy can effectively mitigate practical privacy attacks. For example, Mironov [19] introduced Rényi Differential Privacy (RDP), which provides a tighter accounting of cumulative privacy loss, while Jayaraman and Evans [39] empirically demonstrated that DP significantly reduces the success rate of membership inference and data reconstruction attacks in machine learning models. Similarly, Dong et al. [40] established refined analytical bounds for the Gaussian mechanism, confirming its robustness against adversarial inference under realistic settings. These studies collectively support the use of DP as a principled defense framework that provides both theoretical and empirical privacy protection.

Despite extensive research efforts on communication-efficient, parameter-efficient, and privacy-preserving federated learning, an effective framework that simultaneously achieves high model accuracy, strong privacy protection, and low communication overhead has yet to be established.

3. Proposed Method

We propose FedSA-LoRA-DP, a federated learning framework that integrates Selective Aggregation and LoRA with DP to achieve communication efficiency, privacy preservation, and personalization simultaneously. An overview of our proposed framework is shown in Figure 1. To clarify the operation of our proposed framework, the main steps of FedSA-LoRA-DP are summarized as follows.

Step 1: 

Broadcast: The server distributes the aggregated $\mathit{A}$ matrices to all clients, which combine them with their frozen pretrained weights $\mathit{W}$ and local ${\mathit{B}}_i$.

Step 2: 

Local training: Each client i performs local training on its private dataset $D_i$ to update ${\mathit{A}}_i$ and ${\mathit{B}}_i$.

Step 3: 

Adding noise: After training, differential privacy noise is applied to the gradients of ${\mathit{A}}_i$ through gradient clipping and Gaussian perturbation, altering only their numeric values without changing the communication size.

Step 4: 

Upload: The clients upload the differentially private ${\mathit{A}}_i$ matrices to the server, while keeping ${\mathit{B}}_i$ local to preserve personalization.

Step 5: 

Aggregation: Finally, the server aggregates the received ${\mathit{A}}_i$ matrices using a weighted average based on the local sample sizes, and the updated global $\mathit{A}$ is broadcast to all clients for the next round.

Specifically, FedSA-LoRA-DP exploits the structural asymmetry between LoRA’s low-rank matrices $\mathit{A}$ and $\mathit{B}$ by aggregating the ${\mathit{A}}_i$ matrices into a shared $\mathit{A}$ matrix on the server while keeping the ${\mathit{B}}_i$ matrices local to each client. The overall training and aggregation procedure of the proposed framework is summarized in Algorithm 1.

Let $\mathit{W}\in {\mathbb{R}}^{m\times n}$ be a frozen pretrained weight matrix. In LoRA, a low-rank matrix $\Delta \mathit{W}$ is parameterized as $\Delta \mathit{W}=\mathit{B}\mathit{A}$, where $\mathit{B}\in {\mathbb{R}}^{m\times r}$ and $\mathit{A}\in {\mathbb{R}}^{r\times n}$. In each client i, the pretrained weight matrix ${\mathit{W}}_i$ is augmented with a LoRA module parameterized by low-rank matrices ${\mathit{A}}_i$ and ${\mathit{B}}_i$. The model weights are updated as

$${\mathit{W}}_i^{t+1}={\mathit{W}}_i^t+{\mathit{B}}_i^t{\mathit{A}}_i^t,$$

(2)

where t denotes the learning round. Only ${\mathit{A}}_i$ and ${\mathit{B}}_i$ are trainable, leading to a significant reduction in the number of trainable and transmitted parameters compared to full fine-tuning. In LoRA clients can collaboratively train global knowledge via ${\mathit{A}}_{\mathbf{i}}$ while retaining personalization through local ${\mathit{B}}_{\mathbf{i}}$ [41].

Algorithm 1 FedSA-LoRA-DP

Require:  Pretrained model $\mathit{W}$, LoRA rank r, learning rate $\eta$, clipping norm C, noise multiplier $\sigma$, rounds T, clients K Ensure:  Personalized models for all clients Server: Insert LoRA modules $\{\mathit{A},\mathit{B}\}$ into $\mathit{W}$, freeze backbone, broadcast ${\mathit{A}}^0$ for$t=1$ to T do     for each client $i=1,\dots ,K$ in parallel do         Receive ${\mathit{A}}^{t-1}$ and update:             Apply DP to ${\mathit{A}}_i^t$: $${\tilde{\mathit{A}}}_i^t\leftarrow {\mathit{A}}_i^t-\eta {\tilde{g}}_i^t$$               where ${\tilde{g}}_i^t$ is obtained by clipping and adding Gaussian noise             Update ${\mathit{B}}_i^t$         Upload ${\tilde{\mathit{A}}}_i^t$ and $n_i$ to the server     end for     Server: Aggregate $${\mathit{A}}^t\leftarrow \sum _{i=1}^K{\displaystyle \frac{n_i}{N}}{\tilde{\mathit{A}}}_i^t$$     Broadcast ${\mathit{A}}^t$ to all clients     Privacy accounting: Update ${\epsilon }_t$ using RDP end for

For each client i, let ${\mathcal{B}}_i={\left\{x_j\right\}}_{j=1}^B$ be a minibatch and $g_j={\nabla }_{\mathit{A}}\ell \left(x_j\right)$ the gradient for sample $x_j$. We apply DP to the shared parameters $\mathit{A}$ as follows. Each gradient is clipped:

$${\overline{g}}_j={\displaystyle \frac{g_j}{max\left(1,\frac{\parallel g_j{\parallel }_2}{C}\right)}},$$

(3)

and then perturbed with Gaussian noise:

$${\tilde{g}}_i={\displaystyle \frac{1}{{\mathcal{B}}_i}}\sum _{j=1}^B{\overline{g}}_j+\mathcal{N}(0,{\sigma }^2{C}^2\mathbf{I}),$$

(4)

where C is the clipping norm, $\sigma$ is the noise multiplier and $\mathbf{I}$ is the identity matrix. The ${\mathit{A}}_i$ parameters are then updated by

$${\tilde{\mathit{A}}}_i={\mathit{A}}_i-\eta {\tilde{g}}_i,$$

(5)

where $\eta$ is the learning rate. In contrast, ${\mathit{B}}_i$ is updated without noise. Following the privacy composition framework of Abadi et al. [18], the noise multiplier $\sigma$ is computed based on the specified privacy budget $(\epsilon ,\delta )$, the total number of iterations T, where N represents the total number of training samples.

Consider a situation with K clients where each client i holds a local dataset $D_i$. Each client takes a sample $N_i$ from their respective dataset $D_i$; we set $N={\sum }_{i=1}^K{N}_i$. Let $x_i$ denote the number of samples on client i. After local updates, only the updated ${\mathit{A}}_i$ is uploaded to the server. The server aggregates them using a weighted average:

$$\mathit{A}=\sum _{i=1}^K{\displaystyle \frac{n_i}{N}}{\tilde{\mathit{A}}}_{\mathbf{i}}.$$

(6)

The aggregated $\mathit{A}$ is then broadcast to all clients for the next round.

We use Opacus, a PyTorch library for differential privacy, to implement DP and track the cumulative privacy loss during training [20]. Specifically, we employ the built-in RDP accountant to compute the privacy spending $(\epsilon ,\delta )$ across multiple communication rounds, following the composition rules of RDP [19].

4. Experiment and Evaluation

4.1. Experiment Setup

Experiments were conducted on the CIFAR-100 dataset [42], using an input resolution of $224\times 224$ and following the standard preprocessing pipeline for BiT models [43]. Samples of the datasets are as shown in Figure 2 The dataset was partitioned into 10 clients to simulate a federated learning environment, and both i.i.d. and non-i.i.d. settings were examined. In the non-i.i.d. scenario, data were distributed across clients according to a Dirichlet distribution with $\alpha =0.5$, introducing statistical heterogeneity to reflect realistic federated learning conditions better. A local batch size of 64 was used for client-side training.

For the model architecture, BiT-s R50 × 1 and BiT-s R101 × 1, both pretrained on ImageNet-21k, were employed as backbones. In all experiments, the backbone networks were kept frozen, and LoRA modules were inserted into all $1\times 1$ convolution layers to enable parameter-efficient fine-tuning. Federated training was performed for 100 communication rounds. During each round, only the LoRA ${\mathit{A}}_i$ matrices were aggregated on the server, while the ${\mathit{B}}_i$ matrices remained local to each client. Client-side training was conducted for three epochs per round using Stochastic Gradient Descent (SGD) with a cosine learning rate schedule.

For experiments involving privacy preservation, differential privacy was applied by performing DP exclusively on the shared LoRA $\mathit{A}$ matrix, while local parameters were left unperturbed [18]. The main hyperparameters used in the federated learning setup are summarized in

Table 1. Main hyperparameter configuration for the federated learning experiments, including DP settings. The same configuration was applied across all datasets unless otherwise specified.

Parameter	Symbol	Value
Federated learning setup
Number of rounds	T	100
Number of clients	N	10
Local epochs	E	3
Learning rate	$\eta$	$1.0\times {10}^{-4}$
Optimizer	-	SGD
DP configuration
Privacy budget	$\epsilon$	8, 2
Failure probability	$\delta$	$1.0\times {10}^{-5}$
Clipping norm	C	0.5
Batch size	$\mathcal{B}$	64
Noise multiplier	$\sigma$	1.4 ($\epsilon =8$), 3.75 ($\epsilon =2$)

.

To ensure accurate privacy accounting, we adopted RDP [19] as implemented in Opacus [20]. RDP provides a tighter composition bound compared to standard $(\epsilon ,\delta )$-DP by tracking the cumulative privacy loss across multiple training iterations. In our setup, each client performed per-sample gradient clipping with norm $C=0.5$ and added Gaussian noise with a multiplier $\sigma \in \{1.4,3.75\}$ to the gradients of the LoRA ${\mathit{A}}_i$ parameters during local training. The sampling rate is determined by the mini-batch size ($\mathcal{B}=64$) relative to the local dataset size, and the cumulative privacy loss $\epsilon$ was computed from the RDP accountant after $T=100$ communication rounds at a fixed $\delta ={10}^{-5}$. We report the final privacy budgets $\epsilon =8$ and $\epsilon =2$, corresponding to the noise multipliers $\sigma =1.4$ and $\sigma =3.75$, respectively.

4.2. Analysis

This section provides an in-depth analysis of the proposed FedSA-LoRA-DP framework by examining how different factors, such as data heterogeneity, client participation rate, and LoRA rank, affect the overall performance. In real-world federated learning scenarios, it is unlikely that all clients participating in training will have the same data distribution, and it is also difficult for the same clients to always participate in training. For this reason, we compared experiments in i.i.d. and non-i.i.d. environments, as well as experiments where the client participation rate was changed. Furthermore, we evaluated whether the proposed method could maintain high accuracy in spite of lighter parameter configuration to confirm its applicability in resource-constrained federated environments. We conducted experiments with LoRA ranks set to $r=8$ and $r=4$, and verified the trade-off between model weight reduction and accuracy.

Unless otherwise specified, the parameters listed in

Table 2. Default hyperparameter settings used throughout the experiments unless otherwise specified.

Hyperparameter	Value
Data heterogeneity	i.i.d.
Client participation rate	$1.0$
LoRA rank	8

are used throughout the experiments. Since the primary source of client diversity in our setting arises from differences in dataset size and class balance, clients with larger and more balanced data naturally achieve higher accuracy. Therefore, instead of reporting individual client metrics, we summarize results at the global level to focus on the overall performance trends. All reported results in this section denote the mean and standard deviation of test accuracies across all clients.

4.2.1. Effect of Data Heterogeneity

To analyze the impact of data heterogeneity, we conducted experiments under both i.i.d. and non-i.i.d. (Dirichlet $\alpha =0.5$) data distributions and compared the resulting model performance. The results of this analysis are summarized in

Table 3. Test accuracy (%) of FedSA-LoRA and FedSA-LoRA-DP under i.i.d. and non-i.i.d. settings (Dirichlet $\alpha =0.5$) on CIFAR-100. Both BiT-S R50 × 1 and R101 × 1 architectures are evaluated to assess the impact of data heterogeneity on model performance. Results show that FedSA-LoRA-DP achieves nearly identical accuracy to FedSA-LoRA in both data distributions, demonstrating that differential privacy has minimal influence on convergence or generalization even under heterogeneous client data.

Method	BiT-S R50 × 1		BiT-S R101 × 1
	i.i.d.	non-i.i.d. ($\mathit{\alpha }$ = 0.5)	i.i.d.	non-i.i.d. ($\mathit{\alpha }$ = 0.5)
FedSA-LoRA	74.47 ± 0.29	55.60 ± 2.14	78.32 ± 0.29	59.33 ± 2.55
FedSA-LoRA-DP ($\epsilon$ = 8)	74.07 ± 0.31	55.32 ± 2.12	77.88 ± 0.23	59.09 ± 2.45
FedSA-LoRA-DP ($\epsilon$ = 2)	74.06 ± 0.29	55.33 ± 2.12	77.88 ± 0.22	59.09 ± 2.44

.

When comparing the i.i.d. and non-i.i.d. settings for the BiT-S R50 × 1 model, FedSA-LoRA exhibited a performance gap of 18.87 points, whereas FedSA-LoRA-DP showed gaps of 18.75 points and 18.73 points for $\epsilon =8$ and $\epsilon =2$, respectively. Similarly, for the BiT-S R101 × 1 model, the accuracy gap between i.i.d. and non-i.i.d. settings was 18.99 for FedSA-LoRA, compared to 18.79 points for both $\epsilon =8$ and $\epsilon =2$ in FedSA-LoRA-DP. Furthermore, when comparing FedSA-LoRA and FedSA-LoRA-DP, the accuracy difference was only 0.40–0.41 points for the R50 × 1 model and 0.44 points for the R101 × 1 model.

These results demonstrate that the proposed FedSA-LoRA-DP framework maintains stable learning performance even under data heterogeneity. The minimal performance difference between FedSA-LoRA and FedSA-LoRA-DP indicates that the introduction of differential privacy does not significantly compromise model accuracy. A detailed discussion of the limitations under more extreme non-i.i.d. conditions is provided in Section 5.1.

As shown in Figure 3, all methods exhibit similar convergence behavior during training, indicating that the introduction of differential privacy does not hinder the learning dynamics or delay convergence. The performance difference appears primarily in the final accuracy, suggesting that DP noise affects only the fine-tuning stage rather than the overall optimization trajectory. Furthermore, when comparing $\epsilon =8$ and $\epsilon =2$, there is almost no observable difference in the learning dynamics, indicating that the injected DP noise does not affect convergence stability. This indicates that the injected DP noise is sufficiently small compared to the model’s inherent stochasticity, resulting in nearly identical convergence behavior and learning stability across different privacy levels.

4.2.2. Effect of Client Participation Rate

To verify whether the model can maintain high accuracy even in scenarios where not all clients are able to participate in every training round, we investigate the framework’s robustness to partial client participation by comparing full ($f=1.0$), where all clients join every round, and partial ($f=0.3$), where 30% of clients are randomly selected for each round. To maintain comparable privacy levels across different participation settings, the noise multiplier $\sigma$ was set to 1.1 for $\epsilon =8$ and 3.0 for $\epsilon =2$. The results evaluating this effect are summarized in

Table 4. Test accuracy (%) of FedSA-LoRA and FedSA-LoRA-DP under different client participation ratios ($f=1.0,0.3$) using BiT-S R50 × 1 and R101 × 1 backbones on CIFAR-100. This experiment evaluates robustness to partial participation, a common scenario in real-world federated learning. The results indicate that accuracy degradation remains about 0.5 points even when only 30% of clients participate, showing that FedSA-LoRA-DP maintains stable performance despite reduced client availability.

Method	BiT-S R50 × 1		BiT-S R101 × 1
	$\mathit{f}\mathbf{=}\mathbf{1.0}$	$\mathit{f}\mathbf{=}\mathbf{0.3}$	$\mathit{f}\mathbf{=}\mathbf{1.0}$	$\mathit{f}\mathbf{=}\mathbf{0.3}$
FedSA-LoRA	74.47 ± 0.29	73.97 ± 0.33	78.32 ± 0.29	77.77 ± 0.33
FedSA-LoRA-DP ($\epsilon$ = 8)	74.07 ± 0.31	73.65 ± 0.43	77.88 ± 0.23	77.34 ± 0.26
FedSA-LoRA-DP ($\epsilon$ = 2)	74.06 ± 0.29	73.66 ± 0.42	77.88 ± 0.22	77.35 ± 0.27

.

When comparing the client participation ratios $f=1.0$ and $f=0.3$ for the BiT-S R50 × 1 model, FedSA-LoRA exhibited an accuracy difference of 0.50 points, whereas FedSA-LoRA-DP showed gaps of 0.42 points and 0.40 points for $\epsilon =8$ and $\epsilon =2$, respectively. Similarly, for the BiT-S R101 × 1 model, the accuracy gap between the two participation settings was 0.55 points for FedSA-LoRA, 0.54 points for FedSA-LoRA-DP ($\epsilon =8$), and 0.53 points for FedSA-LoRA-DP ($\epsilon =2$).

The largest gap between FedSA-LoRA and FedSA-LoRA-DP was observed in the comparison of the BiT-S R101 × 1 model with $\epsilon =8$, where the accuracy difference reached 0.43 points. These results suggest that FedSA-LoRA-DP maintains comparable accuracy to FedSA-LoRA even when the number of participating clients is reduced, demonstrating robustness against partial participation and communication variability in federated environments.

4.2.3. Effect of LoRA Rank

To investigate the model’s ability to maintain high accuracy while being trained with a smaller number of parameters, we analyze the effect of the LoRA rank by comparing configurations with $r=8$ and $r=4$. The results are summarized in

Table 5. Test accuracy (%) of FedSA-LoRA and FedSA-LoRA-DP with different LoRA ranks ($r=4,8$) on CIFAR-100 using BiT-S R50 × 1 and BiT-S R101 × 1 backbones. The upper two columns correspond to the BiT-S R50 × 1 results, and the right two columns to the BiT-S R101 × 1 results. This comparison evaluates the trade-off between model compactness and performance, showing that reducing the LoRA rank from 8 to 4 leads to less than a 0.2-point drop in accuracy for both backbones. Furthermore, FedSA-LoRA-DP maintains comparable accuracy to FedSA-LoRA across all configurations, indicating that differential privacy has minimal impact even under stronger parameter compression.

Method	BiT-S R50 × 1		BiT-S R101 × 1
	$\mathit{r}\mathbf{=}\mathbf{8}$	$\mathit{r}\mathbf{=}\mathbf{4}$	$\mathit{r}\mathbf{=}\mathbf{8}$	$\mathit{r}\mathbf{=}\mathbf{4}$
FedSA-LoRA	74.47 ± 0.29	74.28 ± 0.31	78.32 ± 0.29	78.27 ± 0.29
FedSA-LoRA-DP ($\epsilon$ = 8)	74.07 ± 0.31	73.94 ± 0.30	77.88 ± 0.23	77.86 ± 0.29
FedSA-LoRA-DP ($\epsilon$ = 2)	74.06 ± 0.29	73.93 ± 0.31	77.88 ± 0.22	77.87 ± 0.29

.

When comparing LoRA ranks $r=8$ and $r=4$ for the BiT-S R50 × 1 model, FedSA-LoRA showed a performance difference of 0.19 points, while FedSA-LoRA-DP reported gaps of 0.13 points for both $\epsilon =8$ and $\epsilon =2$. For the BiT-S R101 × 1 model, the accuracy differences were 0.05 points for FedSA-LoRA and only 0.02 and 0.01 points for FedSA-LoRA-DP ($\epsilon =8$ and $\epsilon =2$), respectively. Furthermore, when comparing FedSA-LoRA and FedSA-LoRA-DP, the accuracy gap was 0.40–0.44 points for rank $r=8$, whereas it decreased to 0.34–0.41 points for rank $r=4$.

The smallest accuracy gap was observed for the BiT-S R50 × 1 model with $r=4$, indicating that the proposed method maintains high accuracy even under stronger parameter compression. These results demonstrate that the proposed framework is robust to changes in LoRA rank, and that privacy preservation can be achieved with negligible loss in accuracy. A more detailed discussion on the relationship between LoRA rank and noise robustness is provided in Section 5.2.

4.3. Generalizability Across Datasets

The goal of this analysis is to evaluate the generalizability of the proposed FedSA-LoRA-DP framework across datasets of varying complexity and domain characteristics. We employed three datasets: CIFAR-100, MNIST [44], and SVHN [45], each differing in image dimensionality, visual diversity, and class structure. Samples of MNIST and SVHN are as shown in Figure 4.

All experiments in this subsection were conducted under the i.i.d. setting with a LoRA rank of $r=8$ and client participation $f=1.0$, to ensure consistent comparison across datasets. The results of test accuracy are shown in Figure 5a,b, corresponding to the BiT-S R50 × 1 and R101 × 1 architectures, respectively.

For the BiT-S R50 × 1 model, FedSA-LoRA achieved 74.47% accuracy on CIFAR-100, while FedSA-LoRA-DP achieved 74.07% for $\epsilon =8$ and 74.06% for $\epsilon =2$, resulting in only a 0.4-point difference. For MNIST, FedSA-LoRA achieved 95.38%, and FedSA-LoRA-DP achieved 93.91% for both $\epsilon =8$ and $\epsilon =2$, corresponding to a 1.47-point reduction. For SVHN, FedSA-LoRA achieved 52.63%, and FedSA-LoRA-DP achieved 50.24% for $\epsilon =8$ and 50.28% for $\epsilon =2$, showing a 2.3-point difference.

For the BiT-S R101 × 1 model, FedSA-LoRA achieved 78.32% accuracy on CIFAR-100, while FedSA-LoRA-DP achieved 77.88% for both $\epsilon =8$ and $\epsilon =2$, resulting in only a 0.4-point difference. For MNIST, FedSA-LoRA achieved 94.99%, and FedSA-LoRA-DP achieved 94.11% for $\epsilon =8$ and 94.35% for $\epsilon =2$, corresponding to reductions of 0.88 and 0.64 points. For SVHN, FedSA-LoRA achieved 51.03%, and FedSA-LoRA-DP achieved 46.47% for $\epsilon =8$ and 46.49% for $\epsilon =2$, showing differences of 4.56 points and 4.54 points.

These results confirm that the proposed FedSA-LoRA-DP framework maintains high accuracy across datasets with different visual and structural complexities. A deeper discussion of the dataset-dependent privacy–utility trade-off is provided in Section 5.3.

4.4. Communication Cost

To quantitatively validate the communication efficiency of our proposed methods, we measured both the per-round and total communication costs for FedSA-LoRA, FedSA-LoRA-DP, and the baseline FedAvg.

As shown in

Table 6. Comparison of communication costs per round among different federated learning methods. FedSA-LoRA and FedSA-LoRA-DP achieve approximately 99% reduction in communication overhead compared with standard FedAvg, since only LoRA ${\mathit{A}}_i$ matrices are transmitted. The integration of DP introduces no additional communication cost, demonstrating the efficiency of the proposed design.

Method	Per-Round Cost (MB)
FedAvg	90.428
FedSA-LoRA	0.6375
FedSA-LoRA-DP	0.6375

, the per-round communication cost of FedSA-LoRA and FedSA-LoRA-DP is only 0.6375 MB, whereas FedAvg requires 90.428 MB per round, corresponding to an approximate 99% reduction in communication overhead.

The substantial reduction in communication cost stems from the structural difference between FedAvg and our LoRA-based approaches. In FedAvg, the entire set of model parameters $\mathit{W}$ must be transmitted between clients and the server in each communication round, resulting in a payload proportional to the full model size. In contrast, our methods employ LoRA, which decomposes the parameter update into two small matrices, $\mathit{A}$ and $\mathit{B}$. During training, only $\mathit{A}$ is communicated, while the pretrained base weights $\mathit{W}$ remain frozen. This design effectively transforms federated learning into a transfer learning paradigm, where clients fine-tune lightweight adaptation modules rather than the entire model. Consequently, the per-round communication cost scales linearly with the low-rank dimension r, leading to a 99% reduction compared to standard FedAvg. Even when differential privacy is applied, the communication cost remains unchanged, as DP noise is injected locally without altering message size.

Therefore, our method achieves substantial communication savings while maintaining competitive global accuracy.

5. Discussion

5.1. Limitations Under Extreme Data Heterogeneity

Table 7. Test accuracy (%) of FedSA-LoRA and FedSA-LoRA-DP under varying data heterogeneity levels on the CIFAR-100 dataset. Both BiT-S R50 × 1 and R101 × 1 architectures are evaluated under i.i.d. and non-i.i.d. settings with Dirichlet parameters $\alpha \in \{0.5,0.1\}$. A smaller $\alpha$ value indicates stronger heterogeneity, producing more imbalanced class distributions and less data overlap among clients. The results show that both FedSA-LoRA and FedSA-LoRA-DP remain stable under moderate heterogeneity ($\alpha =0.5$) but experience a marked accuracy drop when $\alpha =0.1$.

Method	BiT-S R50 × 1			BiT-S R101 × 1
	i.i.d.	$\mathit{\alpha }$ = 0.5	$\mathit{\alpha }$ = 0.1	i.i.d.	$\mathit{\alpha }$ = 0.5	$\mathit{\alpha }$ = 0.1
FedSA-LoRA	74.47 ± 0.29	55.60 ± 2.14	30.10 ± 2.81	78.32 ± 0.29	59.33 ± 2.55	31.62 ± 2.91
FedSA-LoRA-DP ($\epsilon$ = 8)	74.07 ± 0.31	55.32 ± 2.12	30.03 ± 2.83	77.88 ± 0.23	59.09 ± 2.45	31.55 ± 2.88
FedSA-LoRA-DP ($\epsilon$ = 2)	74.06 ± 0.29	55.33 ± 2.12	30.03 ± 2.83	77.88 ± 0.22	59.09 ± 2.44	31.54 ± 2.88

presents the results under more extreme non-i.i.d. conditions, where the Dirichlet parameter is set to $\alpha =0.1$. As $\alpha$ decreases, the class distributions among clients become highly imbalanced, and the overlap between local datasets diminishes substantially. Consequently, the global model struggles to generalize across clients, resulting in a marked drop in overall test accuracy. This degradation can be attributed to two key structural factors of the proposed framework.

First, the low-rank structure of LoRA inherently limits the representational capacity of each client model. When client data distributions differ significantly, the restricted trainable subspace may lead to underfitting, as the low-rank decomposition cannot fully capture the diversity of local feature representations. This structural limitation becomes more pronounced as the degree of data heterogeneity increases, leading to diminished generalization across clients.

Furthermore, previous work has shown that the method performs well even under non-i.i.d. conditions in large-scale language models, where a substantially higher-dimensional [31,32] parameter space is used. However, in our image classification setting, the total number of trainable parameters is relatively small, making the expressive power of LoRA more constrained. As a result, it becomes more challenging for low-rank adaptation to capture client-specific feature variations, which exacerbates the performance drop under strong heterogeneity.

Second, since only the ${\mathit{A}}_i$ matrices are aggregated on the server while ${\mathit{B}}_i$ remain local, the amount of shared information across clients is substantially reduced. Although this selective aggregation strategy greatly enhances communication efficiency, it can intensify model divergence when client distributions are highly dissimilar. Under such conditions, the aggregated $\mathit{A}$ updates fail to represent a coherent global direction, limiting the benefits of collaborative training.

In summary, while FedSA-LoRA-DP maintains robustness under moderate heterogeneity (e.g., $\alpha =0.5$), its performance deteriorates sharply under extreme data skewness, revealing an inherent structural limitation of the selective aggregation mechanism. Future work could address this issue by adopting clustering-based aggregation or personalized model fusion techniques to further enhance robustness in highly non-i.i.d. federated environments.

5.2. Impact of LoRA Rank on Noise Robustness

Table 5 demonstrates an interesting trend; the performance degradation caused by DP noise becomes smaller as the LoRA rank decreases. This observation indicates that lower-rank configurations inherently exhibit stronger robustness to DP-induced noise. The following factors may explain this behavior.

First, reducing the LoRA rank effectively decreases the number of trainable parameters, thereby limiting the dimensionality of the parameter space affected by the injected noise. In lower-rank settings, the gradient perturbations caused by Gaussian noise are distributed over a smaller subspace, leading to more stable optimization and less distortion of the learned representations. In contrast, higher-rank LoRA modules expose a larger number of parameters to DP noise, increasing the likelihood of performance degradation due to noisy gradient updates.

Second, in the proposed FedSA-LoRA-DP framework, DP is applied only to the ${\mathit{A}}_i$ matrices. When the rank r is smaller, the size of these matrices—and consequently the number of perturbed parameters—is also reduced. As a result, the total noise magnitude introduced per round decreases, further mitigating accuracy loss. This selective protection mechanism, combined with low-rank parameterization, thus achieves a natural form of noise resilience.

Finally, these results highlight a trade-off between model capacity and privacy robustness. While higher ranks provide greater representational power, lower ranks yield improved stability against DP noise. This implies that LoRA-based federated systems can exploit rank as a controllable hyperparameter to balance privacy and performance, enabling efficient and privacy-preserving learning even with constrained model capacity.

5.3. Impact of Dataset Characteristics on Privacy–Utility Trade-Off

The variations in performance observed across CIFAR-100, MNIST, and SVHN can be attributed to differences in dataset complexity and representational diversity. Unlike the preceding quantitative analysis, this section focuses on the underlying factors that influence the privacy–utility trade-off in federated LoRA-based training.

First, the dimensionality and representational complexity of each dataset play a critical role in determining the sensitivity to noise injection. CIFAR-100 consists of high-resolution and semantically rich images, where the LoRA modules can capture diverse feature patterns even with low-rank adaptation. As a result, the Gaussian noise added to the gradients of the ${\mathit{A}}_i$ matrices has only a marginal impact on the overall optimization, leading to a minimal accuracy gap of approximately 0.4 points.

In contrast, MNIST is a low-dimensional and homogeneous dataset with limited intra-class variation. In such cases, the model’s representational subspace is narrow, and the DP-induced perturbation in the low-rank $\mathit{A}$ updates may directly disrupt the already compact feature space. Consequently, a larger accuracy drop of about 1.5 points was observed. This slightly larger gap than in the CIFAR-100 case suggests that DP-induced noise has a more noticeable effect on datasets with limited feature diversity and simpler visual representations.

For SVHN, which contains real-world digit images with substantial illumination and background variability, both the low-rank approximation and the heterogeneous data distributions among clients contribute to training instability. Because the shared $\mathit{A}$ matrix captures only a small portion of the total feature variation across clients, the model becomes more sensitive to gradient variance, leading to less stable updates and lower overall accuracy. This explains the relatively lower accuracy, around 2.3 points below FedSA-LoRA, despite consistent privacy guarantees. Also, this effect is inherent to the dataset’s complex visual structure rather than the privacy mechanism itself.

Additionally, these observations suggest that the effectiveness of the proposed method depends on the expressive capacity of the LoRA modules relative to dataset complexity. In large-scale models such as LLMs, where parameter space is extremely high-dimensional, low-rank adaptation can still capture rich feature interactions even under strong privacy constraints. However, for smaller-scale models or simpler tasks like image classification, the limited number of trainable parameters may not fully compensate for DP-induced noise, making the performance degradation more noticeable.

Overall, these findings highlight that the privacy–utility trade-off in federated LoRA-based learning is inherently dataset-dependent. Datasets with richer and higher-dimensional representations tend to absorb privacy noise more effectively, while simpler or noisier datasets experience more pronounced degradation. Future work could explore adaptive noise scaling or rank-aware privacy calibration strategies to balance the noise magnitude according to dataset complexity.

5.4. Future Work

In future work, we plan to verify the extent to which the proposed method is resistant to membership inference attacks and gradient leakage attacks. We demonstrated that FedSA-LoRA-DP effectively balances model accuracy and privacy through differential privacy in this paper. However, further investigation is needed to evaluate its empirical resilience against direct privacy attacks.

Theoretical guarantees of differential privacy have been extensively established in prior studies [17,18], ensuring formal protection against information leakage in expectation. Nevertheless, it would be valuable to empirically validate how effectively FedSA-LoRA-DP withstands practical privacy attacks, such as membership inference or gradient inversion, under realistic federated conditions. This line of research would complement the theoretical analysis presented in this work and provide a more comprehensive assessment of the method’s robustness.

In addition, future work will extend the proposed framework to large-scale and domain-specific applications such as autonomous driving, medical imaging, and natural language processing. Evaluating FedSA-LoRA-DP under these complex, heterogeneous, and data-rich environments will be essential to verify its scalability, practical effectiveness, and robustness to extreme non-i.i.d. conditions. Such experiments will also enable a more direct connection between the theoretical contributions and real-world deployment scenarios.

Furthermore, we plan to expand our experimental evaluation to include comparisons with representative DP-FL baselines. These benchmarks will help quantify the relative advantages of FedSA-LoRA-DP in terms of accuracy retention, communication efficiency, and privacy–utility trade-offs. Such analysis will provide a more comprehensive empirical perspective on how our method aligns with and differs from existing adaptive or personalized DP-FL frameworks.

Depending on future requirements, we will consider integrating cryptographic techniques such as secure aggregation or homomorphic encryption to further strengthen privacy protection. Therefore, a systematic empirical evaluation against these attack scenarios remains an important direction for future work to confirm whether FedSA-LoRA-DP provides robust privacy protection beyond formal differential privacy guarantees.

6. Conclusions

In this paper, we proposed FedSA-LoRA-DP, a novel FL framework that integrates DP and LoRA for efficient model adaptation. This method protects privacy during the training process while maintaining communication efficiency and model accuracy.

Evaluation across various experimental conditions on the CIFAR-100 dataset confirmed that the proposed method minimizes accuracy degradation compared to its non-private counterpart. Furthermore, additional experiments using the MNIST and SVHN datasets confirmed that the proposed method demonstrates high versatility across different data domains. These results demonstrate that FedSA-LoRA-DP is an effective method for achieving both privacy and performance.

As future work, we plan to quantitatively evaluate the actual degree of privacy protection using attack methods such as Membership Inference Attack and Gradient Leakage Attack. In addition, we aim to extend the proposed FedSA-LoRA-DP framework to large-scale, real-world applications such as autonomous driving and healthcare, to further validate its scalability, robustness, and practical effectiveness in complex federated environments. We also plan to include comparative evaluations with representative DP-FL baselines to more comprehensively assess the privacy–utility trade-off and communication efficiency of the proposed framework.