Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

Abstract

With the growing scale of pre-trained language models (PLMs), full parameter fine-tuning becomes prohibitively expensive and practically infeasible. Therefore, parameter-efficient adaptation techniques for PLMs have been proposed to learn through incremental updates of pre-trained weights, such as in low-rank adaptation (LoRA). However, LoRA relies on heuristics to select the modules and layers to which it is applied, and assigns them the same rank. As a consequence, any fine-tuning that ignores the structural information between modules and layers is suboptimal. In this work, we propose structure-aware low-rank adaptation (SaLoRA), which adaptively learns the intrinsic rank of each incremental matrix by removing rank-0 components during training. We conduct comprehensive experiments using pre-trained models of different scales in both task-oriented (GLUE) and task-agnostic (Yelp and GYAFC) settings. The experimental results show that SaLoRA effectively captures the structure-aware intrinsic rank. Moreover, our method consistently outperforms LoRA without significantly compromising training efficiency.

1. Introduction

With the scaling of model and corpus size [1,2,3,4,5], large language models (LLMs) have demonstrated an ability for in-context learning  [1,6,7] in various natural language processing (NLP) tasks, that is, learning from a few examples within context. Although in-context learning is now the prevalent paradigm for using LLMs, fine-tuning still outperforms it in task-specific settings. In such scenarios, a task-specific model is exclusively trained on a dataset comprising input–output examples specific to the target task. However, full parameter fine-tuning, which updates and stores all the parameters for different tasks, becomes impractical when dealing with large-scale models.

In fact, LLMs with billions of parameters can be effectively fine-tuned by optimizing only a few parameters [8,9,10]. This has given rise to a branch of parameter-efficient fine-tuning (PEFT) techniques [11,12,13,14,15,16] for model tuning. These techniques optimize a small fraction of the model parameters while keeping the rest fixed, thereby significantly reducing computational and storage costs. For example, LoRA [15] introduces trainable low-rank decomposition matrices into LLMs, enabling the model to adapt to a new task while preserving the integrity of the original LLMs and retaining the acquired knowledge. Fundamentally, this approach is built upon the assumption that updates to the weights of the pre-trained language model have a lower rank during adaptation to specific downstream tasks [8,9]. Thus, by reducing the rank of the incremental matrices, LoRA optimizes less than $0.5\%$ of the additional trainable parameters. Remarkably, this optimization achieves comparable or even superior performance to that of full parameter fine-tuning.

However, despite its advantages, LoRA also comes with certain limitations that warrant consideration. One limitation lies in LoRA’s reliance on heuristics to select the modules and layers to which it is applied. Though heuristics can be effective under specific circumstances, their lack of generalizability is a concern. This lack of generalizability can result in suboptimal performance, or even complete failure, when applied to new data. Another limitation is the assignment of the same rank to incremental matrices across different modules and layers. This tends to oversimplify the complex structural relationships and important disparities that exist within neural networks. This phenomenon is illustrated in Figure 1.

Figure 1. Fine-tuning performance of LoRA across different modules and layers with varying ranks on MRPC.

In this paper, we propose a novel approach called structure-aware low-rank adaptation (SaLoRA), which adaptively learns the intrinsic rank of each incremental matrix by removing rank-0 components. As shown in Figure 2, we introduce a diagonal gate matrix $\mathit{G}$ = $\mathrm{diag}(g_1,\dots ,g_r)$ for each incremental matrix. The modified incremental matrix can be represented as $\Delta$$\mathit{W}$ = $\mathit{B}\mathit{G}\mathit{A}$. The incremental matrix is divided into triplets, where each triplet ${\mathcal{T}}_i$ contains the i-th column of $\mathit{B}$, the i-th gate mask of $\mathit{G}$ and the i-th row of $\mathit{A}$. Here, $g_i$ represents the binary “gate” that indicates the presence or absence of the i-th triplet. Although incorporating the active triplet count as a penalty term in the learning objective is unfeasible, we employ a differentiable relaxation method to selectively remove non-critical triplets by considering the $L_0$ norm [17,18]. The $L_0$ norm is equal to the number of non-zero triplets and encourages the model to deactivate less essential triplets. This strategy assigns a higher rank to crucial incremental matrices to capture task-specific information. Conversely, less significant matrices are pruned to possess a lower ranks preventing overfitting. However, $\mathit{A}$ and $\mathit{B}$ are not orthogonal, implying potential dependence among the triplets. Removing these triplets can result in a more significant deviation from the original matrix. To enhance training stability and generalization, we introduce orthogonality regularization for $\mathit{B}$ and $\mathit{A}$. Furthermore, we integrate a density constraint and leverage Lagrangian relaxation [19] to control the number of valid parameters.

Figure 2. Structure-aware low-rank adaptation.

We conduct extensive experiments on a wide range of tasks and models to evaluate the effectiveness of SaLoRA. Specifically, we conduct experiments on the General Language Understanding Evaluation [20] benchmark in a task-oriented setting to assess the model’s performance. In addition, we evaluate the model’s performance in a task-agnostic setting by fine-tuning LLaMA-7B with a 50K cleaned instruction-following dataset [21], and then perform zero-shot task inference on two text style transfer tasks: sentiment transfer [22] and formality transfer [23]. The experimental results demonstrate that SaLoRA consistently outperforms LoRA without significantly compromising training efficiency.

2. Backgound

Transformer Architecture. The Transformer [24] is primarily constructed using two key submodules: a multi-head self-attention (MHA) layer and a fully connected feed-forward (FFN) layer. The MHA is defined as follows:

$$\begin{array}{c}MHA(Q,K,V)=Concat(hea{d}_1,\dots ,hea{d}_h)W^O,\hfill \\ hea{d}_i=Atention(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)\hfill \end{array}$$

(1)

where $Q,K,V\in {\mathbb{R}}^{n\times d}$ are input-embedding matrices; $W^O\in {\mathbb{R}}^{d\times d}$ is an output projection; $W_i^Q,W_i^K,W_i^V\in {\mathbb{R}}^{d\times d_k}$ are query, key and value projections of head i, respectively; n is sequence length; d is the embedding dimension; h is the number of heads and $d_k=d/h$ is the hidden dimension of the projection subspaces. The FFN consists of two linear transformations separated by a ReLU activation:

$$FFN\left(x\right)=ReLU(x{W}^U+b^U)W^D+b^D$$

(2)

where $W^U\in {\mathbb{R}}^{d\times d_m}$ and $W^D\in {\mathbb{R}}^{d_m\times d}$.

Parameter-Efficient Fine-Tuning. With the growing size of models, recent works have developed three main categories of parameter-efficient fine-tuning (PEFT) techniques. These techniques optimize a small fraction of model parameters while keeping the rest fixed, thereby significantly reducing computational and storage costs [10]. For example, addition-based methods [11,12,13,25,26] introduce additional trainable modules or parameters that are not part of the original model or process. Specifcation-based methods [14,27,28] specify certain parameters within the original model or process as trainable, whereas the others remain frozen. Reparameterization-based methods [15,16,29], including LoRA, reparameterize existing parameters into a parameter-efficient form by transformation. In this study, we focus on reparameterization-based methods, with particular emphasis on LoRA.

Low-Rank Adaptation. LoRA, as introduced in the work of Hu et al. [15], represents a typical example of a reparameterization-based method. In LoRA, some pre-trained weights of LLMs’ dense layers are reparameterized by injecting trainable low-rank incremental matrices. This reparameterization only allows low-rank matrices to be updated, while keeping the original pre-trained weights frozen. By reducing the rank of these matrices, LoRA effectively reduces the number of parameters during the fine-tuning process of LLMs. Consider a pre-trained weight matrix $\mathit{W}$$\in {\mathbb{R}}^{d\times k}$, accompanied by a low-rank incremental matrix $\Delta$$\mathit{W}$ = $\mathit{BA}$. For $\mathit{h}$ = $\mathit{Wx}$, the modified forward pass is as follows:

$$\mathit{h}=\mathit{Wx}+\frac{\alpha }{r}\Delta \mathit{Wx}=\mathit{Wx}+\frac{\alpha }{r}\mathit{BAx}$$

(3)

where $\mathit{B}$$\in {\mathbb{R}}^{d\times r}$, $\mathit{A}$$\in {\mathbb{R}}^{r\times k}$, with the rank $r\ll min(d,k)$, and $\alpha$ is a constant scale hyperparameter. The matrix $\mathit{A}$ adopts a random zero-mean Gaussian initialization, while the matrix $\mathit{B}$ is initialized as a zero matrix. Consequently, the product $\Delta \mathit{W}=\mathit{BA}$ is initially set to zero at the beginning of training. Let ${\mathit{B}}_{\ast j}$ and ${\mathit{A}}_{j\ast }$ denote the j-th column of $\mathit{B}$ and the j-th row of $\mathit{A}$, respectively. Using this notation, $\Delta \mathit{W}$ can be expressed as $\Delta \mathit{W}={\sum }_{j=1}^r{\mathit{B}}_{\ast j}{\mathit{A}}_{j\ast }$.

3. Method

In this section, we will first give a brief introduction to parameter-efficient fine-tuning, and then discuss our proposed model based on the problem definition.

3.1. Problem Formalization

We consider the general problem of efficiently fine-tuning LLMs for specific downstream tasks. Firstly, let us introduce some notations. Consider a training corpus $\mathcal{D}={(x_i,y_i)}_{i=1}^N$, where N represents the number of samples. Each sample consists an input, $x_i$, and its corresponding output, $y_i$. We use the index i to refer to the incremental matrix, i.e., $\Delta {\mathit{W}}_i={\mathit{B}}_i{\mathit{A}}_i$ for $i=1,\dots ,K$, where K is the number of incremental matrices. However, LoRA’s assumption of identical ranks for each incremental matrix overlooks structural relationships and the varying importance of weight matrices across different modules and layers during fine-tuning. This oversight can potentially impact overall model performance. Our objective is to determine the optimal ${\left\{ran{k}^{\ast }\left(\Delta {\mathit{W}}_i\right)\right\}}_{i=1}^K$ on the fly. The optimization objective can be formulated as follows:

$$\begin{array}{cc}\hfill & \hfill \underset{\mathcal{W}}{min}\mathcal{R}\left(\mathcal{W}\right)\triangleq \frac{1}{N}\left(\sum _{i=1}^N\mathcal{L}(f(x_i;\mathcal{W}),y_i)\right)\hfill \\ \hfill & \hfill s.t.rank\left(\Delta {\mathit{W}}_i\right)\le r,k=1,\dots ,K.\hfill \end{array}$$

(4)

where $\mathcal{W}=\{\Delta {\mathit{W}}_i,\dots ,\Delta {\mathit{W}}_K\}$ represents the sets of trainable parameters and $\mathcal{L}$ corresponds to a loss function, such as cross-entropy for classification. Note that $rank\left(\Delta {\mathit{W}}_i\right)\in \{0,1,\dots ,r\}$ is an unknown parameter that needs to be optimized.

3.2. Structure-Aware Intrinsic Rank Using $L_0$ Norm

To find the optimal ${\left\{ran{k}^{\ast }\left(\Delta {\mathit{W}}_i\right)\right\}}_{i=1}^K$ on the fly, with minimal computational overhead during training, we introduce a gate matrix $\mathit{G}$ to define the structure-aware intrinsic rank:

$$\Delta \mathit{W}=\mathit{B}\mathit{G}\mathit{A}=\sum _{j=1}^r{g}_j{\mathit{B}}_{\ast j}{\mathit{A}}_{j\ast }$$

(5)

where the $g_j\in \{0,1\}$ serves as a binary “gate”, indicating the presence or absence of the j-th rank. The gate matrix $\mathit{G}$ = $\mathrm{diag}(g_1,\dots ,g_r)$ is a diagonal matrix consisting of the pruning variables. By learning the variable $g_j$, we can control the rank of each incremental matrix individually, rather than applying the same rank to all matrices. To deactivate non-critical rank-0 components, the ideal approach would be to apply $L_0$ norm regularization to the gate matrix $\mathit{G}$:

$${\left|\right|\mathit{G}\left|\right|}_0=\sum _{j=1}^r{g}_j$$

(6)

where r is the rank of incremental matrices. The $L_0$ norm measures the number of non-zero triplets; thus, optimizing $L_0$ would encourage the model to deactivate less important incremental matrices.

Unfortunately, the optimization objective involving ${\left|\right|\mathit{G}\left|\right|}_0$ is computationally intractable due to its non-differentiability, making it impossible to directly incorporate it as a regularization term in the objective function. Instead, we use a stochastic relaxation approach, where the gate variables g are treated as continuous variables distributed within the interval $[0,1]$. We leverage the reparameterization trick [30,31] to ensure that g remains differentiable. Following prior studies [17,19], we adopt the Hard-Concrete (HC) distribution as a continuous surrogate for random variables g, illustrated in Figure 3. The HC distribution applies a hard-sigmoid rectification to s, which can easily be sampled by first sampling $u\in U(0,1)$ and then computing as follows:

$$\begin{array}{cc}\hfill s& =Sigmod\left(\frac{log\frac{u}{1-u}+log\theta }{\tau }\right)\times \left(\zeta -\gamma \right)+\gamma \hfill \\ \hfill g& =min\left(1,max\left(0,s\right)\right)\hfill \end{array}$$

(7)

where $\theta$ is the trainable parameter of the distribution and $\tau$ is the temperature. The interval $(\gamma ,\zeta )$, with $\gamma <0$ and $\zeta >1$, enables the distribution to concentrate probability mass at the edge of the support. The final outputs g are rectified into $[0,1]$. By summing up the probabilities of the gates being non-zero, the $L_0$ norm regularization can be computed via a closed form, as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[\right|\left|\mathit{G}\right|{|}_0]& =\sum _{j=1}^r\mathbb{E}\left[g_j>0\right]\hfill \\ \hfill & =\sum _{j=1}^{r}Sigmod\left(log{\theta }_j-\tau log\frac{-\gamma }{\zeta }\right)\hfill \end{array}$$

(8)

Figure 3. Hard-Concrete distribution with different parameters.

As g now represents the output of the parameterized HC distribution function and serves as an intermediate representation for the neural network, gradient-based optimization methods can perform gradient updates for $\mathit{\theta }$ = $\{{\theta }_1,\dots ,{\theta }_r\}$. For each training batch, we sample the gate mask and then share it across the training examples within the batch to enhance sampling efficiency.

3.3. Enhanced Stability Using Orthogonal Regularization

In deep networks, orthogonality plays a crucial role in preserving the norm of the original matrix during multiplication, preventing signal vanishing or exploding [32]. However, in LoRA, where $\mathit{B}$ and $\mathit{A}$ are not orthogonal, the dependence can lead to larger variations when removing certain columns or rows through $L_0$ regularization. This, in turn, leads to training instability and the potential for negative effects on generalization [16]. For this, we turn to orthogonal regularization, which enforces the orthogonality condition:

$${\mathcal{R}}_{orth}(\mathit{B},\mathit{A})=\left|\right|{\mathit{B}}^T\mathit{B}-{\mathit{I}\left|\right|}_F^2+\left|\right|\mathit{A}{\mathit{A}}^T-\mathit{I}{\left|\right|}_F^2$$

(9)

where $\mathit{I}$ is the identity matrix.

Now, let us substitute Equations (8) and (9) into Equation (4) to derive the new training objective:

$$\begin{array}{c}\hfill \underset{\mathcal{W},\mathsf{\Theta }}{min}\mathcal{R}(\mathcal{W},\mathsf{\Theta })\triangleq \frac{1}{N}\left(\sum _{i=1}^N\mathcal{L}(f(x_i;\mathcal{W}),y_i)\right)+\lambda \sum _{i=1}^K\mathbb{E}\left[\right||{\mathit{G}}_i{\left|\right|}_0]+\beta \sum _{i=1}^K{\mathcal{R}}_{orth}({\mathit{B}}_i,{\mathit{A}}_i)\end{array}$$

(10)

where $\mathsf{\Theta }=\{{\mathit{\theta }}_i,\dots ,{\mathit{\theta }}_K\}$ represents the sets of trainable parameters, and $\lambda$ and $\beta$ are two constant hyperparameters.

3.4. Controlled Budget Using Lagrangian Relaxation

If we only rely on Equation (10) to learn the intrinsic rank for each incremental matrix, the resulting parameter budget cannot be directly controlled. This limitation becomes problematic in many real-world applications that require a specific model size or parameter budget. To address this issue, we further introduce an additional density constraint on $\mathcal{R}\left(\mathcal{W},\mathsf{\Theta }\right)$ to guide the network towards achieving a specific desired budget.

$$\begin{array}{c}\hfill \underset{\mathcal{W}}{min}\mathcal{R}\left(\mathcal{W}\right)\triangleq \frac{1}{N}\left(\sum _{i=1}^N\mathcal{L}(f(x_i;\mathcal{W}),y_i)\right)+\beta \sum _{i=1}^K{\mathcal{R}}_{orth}({\mathit{B}}_i,{\mathit{A}}_i)\\ \hfill s.t.C\left(\mathsf{\Theta }\right)\triangleq \sum _{i=1}^K\frac{\mathbb{E}\left[\left|\right|{\mathit{G}}_i{\left|\right|}_0\right]\times \left(d_i+k_i\right)}{\#\left({\mathit{B}}_i\right)+\#\left({\mathit{A}}_i\right)}=b\end{array}$$

(11)

where b represents the target density and $\#\left(x\right)$ counts the total number of parameters in matrix x. $\Delta {\mathit{W}}_i={\mathit{B}}_i{\mathit{A}}_i$, where ${\mathit{B}}_i$ is of $d_i\times r_i$, and ${\mathit{A}}_i$ is of $r_i\times k_i$. However, lowering the density constraint poses a challenging and (not necessarily strictly) constrained optimization problem. To tackle this challenge, we leverage Lagrangian relaxation as an alternative approach, along with the corresponding min-max game:

$$\underset{\lambda }{max}\underset{\mathcal{W},\mathsf{\Theta }}{min}\mathcal{L}(\mathcal{W},\mathsf{\Theta },\lambda )\triangleq \mathcal{R}\left(\mathcal{W},\mathsf{\Theta }\right)+\lambda {\left(C\left(\mathsf{\Theta }\right)-b\right)}^2$$

(12)

where $\lambda \in \mathbb{R}$ is the Lagrangian multiplier, which is jointly updated during training. The updates to $\lambda$ would increase the training loss unless the equality constraint is satisfied, resulting in the desired parameter budget. We optimize the Lagrangian relaxation by simultaneously performing gradient descent on $(\mathcal{W},\mathsf{\Theta })$ and projected gradient ascent (to ${\mathbb{R}}^{+}$) on $\lambda$, as demonstrated in previous works [19,33]. During the experiments, we observed that the term $\lambda {\left(C\left(\mathsf{\Theta }\right)-b\right)}^2$ converged quickly. To enhance training efficiency, we only optimize $(\mathsf{\Theta },\lambda )$ between $T_{start}$ and $T_{end}$ time steps. We provide a summarized algorithm in Algorithm 1.

Algorithm 1 SaLoRA

Input: Dataset $\mathcal{D}$; total iterations T; target density b; hyperparameters $\tau ,\gamma ,\zeta ,\beta ,{\eta }_p,{\eta }_c$. Output: The fine-tuned parameters {$\mathcal{W},\mathsf{\Theta }$}.    for $t=1,\dots ,T$ do        Sample a mini-batch from $\mathcal{D}$        if $T_{start}\le t<T_{end}$ then           Sample a gate mask set $\mathcal{G}$ from HC distribution and share it across the mini-batch           Compute the gradient $\mathcal{L}(\mathcal{W},\mathsf{\Theta },\lambda )$           Update ${\mathcal{W}}^{(t+1)}={\mathcal{W}}^{\left(t\right)}-{\eta }_p{\nabla }_{\mathcal{W}}\mathcal{L}({\mathcal{W}}^{\left(t\right)},{\mathsf{\Theta }}^{\left(t\right)},{\lambda }^{\left(t\right)})$           Update ${\mathsf{\Theta }}^{(t+1)}={\mathsf{\Theta }}^{\left(t\right)}-{\eta }_c{\nabla }_{\mathsf{\Theta }}\mathcal{L}({\mathcal{W}}^{\left(t\right)},{\mathsf{\Theta }}^{\left(t\right)},{\lambda }^{\left(t\right)})$           Update ${\lambda }^{(t+1)}={\lambda }^{\left(t\right)}+{\eta }_c{\nabla }_{{\lambda }^{\left(t\right)}}\mathcal{L}({\mathcal{W}}^{\left(t\right)},{\mathsf{\Theta }}^{\left(t\right)},{\lambda }^{\left(t\right)})$       else           Compute the gradient $\mathcal{L}\left(\mathcal{W}\right)$           Update ${\mathcal{W}}^{(t+1)}={\mathcal{W}}^{\left(t\right)}-{\eta }_p{\nabla }_{\mathcal{W}}\mathcal{L}\left({\mathcal{W}}^{\left(t\right)}\right)$       end if end for return The fine-tuned parameters {${\mathcal{W}}^{\left(T\right)},{\mathsf{\Theta }}^{\left(T\right)},{\lambda }^{\left(T\right)}$ }.

3.5. Inference

During training, the gate mask $g_i$ is a random variable drawn from the HC distribution. At inference time, we first calculate the expected value of each $g_i$ in $\mathit{G}$. If the value of $g_i$ is greater than 0, we retain the corresponding i-th low-rank triplet. This procedure enables us to obtain the deterministic matrices $\mathit{B}$ and $\mathit{A}$.

4. Experiments

We evaluated the effectiveness of the proposed SaLoRA on RoBERTa [34] and LLaMA-7B in both task-oriented and task-agnostic settings.

Baselines. We compared SaLoRA with the following methods:

• Fine-tuning (FT) is the most common approach for adaptation. To establish an upper bound for the performance of our proposed method, we fine-tuned all parameters within the model.
• Adapting tuning, as proposed by Houlsby et al. [25], incorporates adapter layers between the self-attention module (and the MLP module) and the subsequent residual connection. Each adapter module consists of two fully connected layers with biases and a nonlinearity in between. This original design is referred to as Adapter^H. Recently, Pfeiffer et al. [11] introduced a more efficient approach, applying the adapter layer only after the MLP module and following a LayerNorm. We call it Adapter^P.
• Prefix-tuning (Prefix) [12] prepends a sequence of continuous task-specific activations to the input. During tuning, prefix-tuning freezes the model parameters and only backpropagates the gradient to the prefix activations.
• Prompt-tuning (Prompt) [13] is a simplified version of prefix-tuning, allowing the additional k tunable tokens per downstream task to be prepended to the input text.
• LoRA, introduced by Hu et al. [15], is a state-of-the-art method for parameter-efficient fine-tuning. The original implementation of LoRA applied the method solely to query and value projections. However, empirical studies [16,35] have shown that extending LoRA to all matrices, including $W^Q$, $W^K$, $W^V$, $W^O$, $W^U$ and $W^D$, can further improve its performance. Therefore, we compare our approach with this generalized LoRA configuration to maximize its effectiveness.
• AdaLoRA, proposed by Zhang et al. [16], utilizes singular value decomposition (SVD) to adaptively allocate the parameter budget among weight matrices based on their respective importance scores. However, this baseline involves computationally intensive operations, especially for large matrices. The training cost can be significant, making it less efficient for resource-constrained scenarios.

4.1. Task-Oriented Performance

Models and Datasets. We evaluated the performance of different adaptive methods on the GLUE benchmark [20] using pre-trained RoBERTa-base (125M) and RoBERTa-large (355M) [34] models from the HuggingFace Transformers library [36]. See Appendix A for additional details on the datasets we used.

Implementation Details. For running all the baselines, we utilized a publicly available implementation [37]. We evaluated the performance of LoRA, AdaLoRA and SaLoRA at $r=8$. To maintain a controlled parameter budget, we set the desired budget ratio (b) to $0.50$ for both SaLoRA and AdaLoRA. During training, we used the AdamW optimizer [38], along with the linear learning rate scheduler. During our experiments, we observed that using a larger learning rate (${\eta }_c$) significantly improved the learning process for both the gate matrices and Lagrange multiplier. Therefore, we set ${\eta }_c$ to $0.01$ for all conducted experiments. We fine-tuned all models using an NVIDIA A100 (40 GB) GPU. Additional details can be found in Appendix B.

Main Results. We compared SaLoRA with the baseline methods under different model scale settings, and the experimental results on the GLUE development set are presented in Table 1. We can see that SaLoRA consistently achieved better or comparable performance compared with existing approaches for all datasets. Moreover, it even outperformed the FT method. SaLoRA’s superiority was particularly striking when compared with LoRA, despite both models having a similar parameter count of 1.33 M/3.54 M for base/large model scales. After training, SaLoRA effectively utilized only $0.5\times 1.33$ $\mathrm{M}/0.5\times 3.54$ parameters, yet still attained superior performance. This observation emphasizes the effectiveness of our method in learning the intrinsic rank for incremental matrices.

Table 1. Results with RoBERTa-base and RoBERTa-large on GLUE development set. We report the Pearson correlation for STS-B, Matthew’s correlation for CoLA, and accuracy for other tasks. We report the mean and maximum deviation of 5 runs using different random seeds. The best results are shown in bold. † indicates numbers published in prior works.

Model and Method	# Trainable	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	STS-B	ALL
	Parameters	ACC	ACC	Mathew	ACC	ACC	ACC	ACC	Pearson	Avg
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{FT}\right)$†	125.00 M	87.6	94.8	63.6	91.9	92.8	78.7	90.2	91.2	86.4
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{Prefix}\right)$	1.33 M	$82.{58}_{\pm 0.24}$	$91.{65}_{\pm 0.44}$	$47.{45}_{\pm 4.47}$	$85.{98}_{\pm 0.11}$	$85.{91}_{\pm 0.44}$	$60.{47}_{\pm 0.54}$	$88.{09}_{\pm 1.37}$	$87.{49}_{\pm 0.63}$	78.70
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{Prompt}\right)$	0.62 M	$79.{14}_{\pm 0.99}$	$88.{33}_{\pm 1.44}$	$42.{35}_{\pm 5.12}$	$73.{30}_{\pm 5.84}$	$80.{51}_{\pm 1.64}$	$55.{81}_{\pm 2.67}$	$69.{75}_{\pm 0.59}$	$76.{94}_{\pm 3.59}$	70.77
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{LoRA}\right)$	1.33 M	$87.{49}_{\pm 0.44}$	$94.{77}_{\pm 0.53}$	$61.{22}_{\pm 2.09}$	$91.{39}_{\pm 0.18}$	$92.{85}_{\pm 0.28}$	$79.{24}_{\pm .21}$	$89.{46}_{\pm 1.23}$	$90.{89}_{\pm 0.18}$	85.91
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{AdaLoRA}\right)$	1.33 M	$\mathbf{87}.{\mathbf{93}}_{\pm \mathbf{0}.\mathbf{20}}$	$94.{59}_{\pm 0.21}$	$59.{29}_{\pm 1.06}$	$90.{94}_{\pm 0.13}$	$92.{61}_{\pm 0.10}$	$76.{39}_{\pm 1.30}$	$87.{35}_{\pm 0.39}$	$90.{87}_{\pm 0.15}$	85.00
${\mathrm{RoB}}_{\mathrm{base}}\left(\mathrm{SaLoRA}\right)$	1.33 M	$87.{83}_{\pm 0.04}$	$\mathbf{95}.{\mathbf{14}}_{\pm \mathbf{0}.\mathbf{73}}$	$63.{39}_{\pm 1.79}$	$91.{46}_{\pm 0.09}$	$\mathbf{92}.{\mathbf{99}}_{\pm \mathbf{0}.\mathbf{21}}$	$\mathbf{81}.{\mathbf{01}}_{\pm \mathbf{0}.\mathbf{87}}$	$\mathbf{90}.{\mathbf{20}}_{\pm \mathbf{0}.\mathbf{74}}$	$91.{13}_{\pm 0.17}$	86.64
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{FT}\right)$†	356.05 M	90.2	96.4	68.0	92.2	94.7	86.6	90.9	92.4	88.9
${\mathrm{RoB}}_{\mathrm{large}}\left({\mathrm{Adapt}}^{\mathrm{P}}\right)$†	4.05 M	$90.2_{\pm 0.3}$	$96.1_{\pm 0.3}$	$68.3_{\pm 1.0}$	$91.9_{\pm 0.1}$	$94.8_{\pm 0.2}$	$83.8_{\pm 2.9}$	$90.2_{\pm 0.7}$	$92.1_{\pm 0.7}$	88.4
${\mathrm{RoB}}_{\mathrm{large}}\left({\mathrm{Adapt}}^{\mathrm{H}}\right)$†	7.05 M	$89.5_{\pm 0.5}$	$96.2_{\pm 0.3}$	$66.5_{\pm 4.4}$	$92.1_{\pm 0.1}$	$94.7_{\pm .2}$	$83.4_{\pm 1.1}$	$88.7_{\pm 2.9}$	$91.0_{\pm 1.7}$	87.8
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{Prefix}\right)$	3.02 M	$88.{61}_{\pm 0.12}$	$94.{70}_{\pm 0.44}$	$60.{06}_{\pm 1.44}$	$87.{57}_{\pm 0.25}$	$89.{60}_{\pm 0.37}$	$77.{33}_{\pm 1.37}$	$89.{85}_{\pm .88}$	$89.{97}_{\pm 3.37}$	84.71
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{Prompt}\right)$	1.09 M	$85.{65}_{\pm 2.54}$	$93.{95}_{\pm 0.83}$	$58.{34}_{\pm 2.63}$	$83.{98}_{\pm 1.45}$	$84.{92}_{\pm 3.28}$	$58.{70}_{\pm 4.83}$	$74.{22}_{\pm 2.65}$	$80.{47}_{\pm 0.71}$	77.53
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{LoRA}\right)$	3.41 M	$89.{96}_{\pm 0.12}$	$96.{10}_{\pm 0.11}$	$\mathbf{68}.{\mathbf{76}}_{\pm \mathbf{1}.\mathbf{75}}$	$88.{67}_{\pm 0.88}$	$94.{86}_{\pm 0.07}$	$85.{49}_{\pm 1.51}$	$90.{93}_{\pm 0.74}$	$92.{25}_{\pm 0.17}$	88.38
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{AdaLoRA}\right)$	3.54 M	$\mathbf{90}.{\mathbf{84}}_{\pm \mathbf{0}.\mathbf{03}}$	$96.{29}_{\pm 0.19}$	$67.{61}_{\pm 0.12}$	$91.{12}_{\pm 0.26}$	$94.{82}_{\pm 0.11}$	$86.{28}_{\pm 0.36}$	$89.{89}_{\pm 0.31}$	$92.{27}_{\pm 0.16}$	88.64
${\mathrm{RoB}}_{\mathrm{large}}\left(\mathrm{SaLoRA}\right)$	3.54 M	$90.{67}_{\pm 0.07}$	$\mathbf{96}.{\mathbf{63}}_{\pm \mathbf{0}.\mathbf{28}}$	$68.{37}_{\pm 0.34}$	$91.{95}_{\pm 0.08}$	$\mathbf{94}.{\mathbf{98}}_{\pm \mathbf{0}.\mathbf{09}}$	$\mathbf{87}.{\mathbf{80}}_{\pm \mathbf{1}.\mathbf{29}}$	$\mathbf{91}.{\mathbf{81}}_{\pm \mathbf{1}.\mathbf{57}}$	$\mathbf{92}.{\mathbf{43}}_{\pm \mathbf{0}.\mathbf{18}}$	89.33

4.2. Task-Agnostic Performance

Models and Datasets. We present the experiments conducted to evaluate the performance of the self-instruct tuned LLaMA-7B models on instruction-following data [21]. Our objective was to assess their capability in comprehending and executing instructions for arbitrary tasks. We evaluated model performance on two text style transfer datasets: Yelp [22] and GYAFC [23]). Text style transfer refers to the task of changing the style of a sentence to the desired style while preserving the style-independent content. The prompts used in these experiments can be found in Appendix C.

Implementation Details. We tuned the learning rate ${\eta }_p$ from $\{8\times {10}^{-5},3\times {10}^{-5},1\times {10}^{-4},3\times {10}^{-4},8\times {10}^{-4},1\times {10}^{-3}\}$ and kept the following hyperparameters fixed: $r=8,b=0.5,{\eta }_c=0.01$, $\beta =0.1$, $\tau =1.0$, $\gamma =-0.1$ and $\zeta =1.1$. All models were fine-tuned on an NVIDIA A800 (80 G) GPU.

Furthermore, we compared the performance of SaLoRA with dataset-specific style transfer models, including StyTrans [15], StyIns [16] and TSST [17]. In contrast to SaLoRA, these models were trained on a specific dataset. To evaluate the performance of style transfer models, we used the following metrics: (1) Transfer accuracy (ACC) using a fine-tuned BERT-base [39] classifier with each dataset. (2) Semantic similarity to human references via BLEU [40] score. (3) Sentence fluency (PPL) via perplexity, as measured by KenLM [41].

Main Results. Table 2 presents our experimental results on the Yelp and GYAFC datasets. Compared with LoRA, our method SaloRA achieved better or comparable performance across all directions on both datasets. This demonstrates the effectiveness of our method. In the negative-to-positive transfer direction, though SaloRA’s transfer accuracy was lower than the dataset-specific models (e.g., StyIns achieved 92.40 compared with SaloRA’s 71), it still aligned with the human reference accuracy of 64.60. Furthermore, SaloRA exhibited a lower perplexity (PPL) compared with dataset-specific models. These results show that SaLoRA (including LoRA) aligns more closely with human writing tendencies. In the formal-to-informal transfer direction, we also observed that our transfer accuracy was lower than dataset-specific models. This disparity may be attributed to the inherent bias of a large model for generating more formal outputs. This can be verified from the fact that SaLoRA exhibited a significant improvement in the transfer accuracy compared with dataset-specific models.

Table 2. Automatic evaluation results on Yelp and GYAFC datasets. ↑ indicates that higher values mean better performance, and vice versa.

Model and Method	Yelp						GYAFC
	Negative to Positive			Positive to Negative			Informal to Formal			Formal to Informal
	ACC↑	BLEU↑	PPL↓	ACC↑	BLEU↑	PPL↓	ACC↑	BLEU↑	PPL↓	ACC↑	BLEU↑	PPL↓
Reference	64.60	100	102.62	93.80	100	77.53	88.44	100	66.86	87.63	100	105.28
StyTrans	88.40	25.85	173.35	94.20	24.90	141.88	32.81	54.91	144.15	80.86	27.69	201.78
StyIns	92.40	25.98	116.01	89.60	26.08	105.79	54.73	60.87	96.53	80.57	30.25	132.54
TSST	91.20	28.95	112.86	94.40	28.83	101.92	65.62	61.83	87.04	85.87	33.54	128.78
${\mathrm{LaA}}_{7\mathrm{B}}$	2.20	33.58	208.69	0.80	31.12	156.14	12.01	60.18	189.78	7.75	34.61	145.43
${\mathrm{LaA}}_{7\mathrm{B}}\left(\mathrm{LoRA}\right)$	71.00	25.96	82.20	92.80	31.83	83.03	89.34	61.06	68.52	34.45	41.59	82.96
${\mathrm{LaA}}_{7\mathrm{B}}\left(\mathrm{SaLoRA}\right)$	73.20	24.76	76.49	94.60	31.96	87.41	89.63	61.76	67.53	39.04	40.91	79.54

4.3. Analysis

The Effect of Rank r. Figure 4 illustrates the experimental results of fine-tuning RoBERTa-large across different ranks. We see that the rank r significantly influenced the model’s performance. Both large and small values of r led to suboptimal results. This observation emphasizes that selecting the optimal value for r through heuristic approaches is not always feasible. Notably, SaLoRA consistently improved performance across all ranks when compared with the baseline LoRA. This suggests that SaLoRA effectively captured the “intrinsic rank” of the incremental matrix.

Figure 4. Fine-tuning performance under different ranks.

The Effect of Sparsity b. Figure 5 shows the experimental results of fine-tuning RoBERTa-large across various levels of sparsity. Remarkably, SaLoRA consistently exhibited enhanced performance across all sparsity levels compared with the baseline. This result suggests that SaLoRA’s modifications facilitated the acquisition of the “intrinsic rank” of the incremental matrix under different sparsities. It is noteworthy that SaLoRA’s performance even surpassed the results of LoRA under low sparsity conditions ($0.125$). The fact that SaLoRA can outperform LoRA even under low sparsity conditions highlights its capacity to capture and leverage parameters with a constrained budget. Consequently, SaLoRA’s efficacy could be expanded on a limited budget, making it a versatile method with a broader range of applications.

Figure 5. Fine-tuning performance under different sparsity levels.

Ablation Study. We investigated the impact of Lagrangian relaxation and orthogonal regularization in SaLoRA. Specifically, we compared SaLoRA with the following variants: (i) SaLoRA${}_{\lambda =0}$: SaLoRA without Lagrangian relaxation; (ii) SaLoRA${}_{\beta =0}$: SaLoRA without orthogonal regularization. These variations involved the fine-tuning of the RoBERTa-base model on the CoLA, STS-B, and MRPC datasets. The target sparsity was set to 0.5 by default. SPS represented the expected sparsity of the incremental matrix. From Table 3, we see that:

Table 3. Ablation studies on Lagrangian relaxation and orthogonal regularization.

Method	MRPC		STS-B		CoLA
	ACC	SPS	ACC	SPS	ACC	SPS
SaLoRA	$\mathbf{90}.{\mathbf{20}}_{\pm \mathbf{0}.\mathbf{74}}$	$0.{51}_{\pm 0.00}$	$91.{13}_{\pm 0.17}$	$0.{52}_{\pm 0.00}$	$63.{39}_{\pm 1.79}$	$0.{52}_{\pm 0.00}$
SaLoRA${}_{\lambda =0}$	$89.{95}_{\pm 0.49}$	$0.{37}_{\pm 0.02}$	$\mathbf{91}.{\mathbf{20}}_{\pm \mathbf{0}.\mathbf{12}}$	$0.{42}_{\pm 0.03}$	$\mathbf{63}.{\mathbf{65}}_{\pm \mathbf{3}.\mathbf{39}}$	$0.{43}_{\pm 0.02}$
SaLoRA${}_{\beta =0}$	$90.{00}_{\pm 0.94}$	$0.{51}_{\pm 0.00}$	$90.{78}_{\pm 0.27}$	$0.{52}_{\pm 0.00}$	$62.{89}_{\pm 2.16}$	$0.{52}_{\pm 0.00}$

• Without Lagrangian relaxation, the parameter budget was uncontrollable, being $0.37$, $0.42$ and $0.43$ on the three datasets, respectively. Such results highlight the pivotal role that Lagrangian relaxation plays in controlling the allocation of the parameter budget. Nonetheless, it is worth noting that omitting Lagrange relaxation may lead to slight enhancements in performance. However, given the emphasis on control over the parameter budget, this incremental enhancement should be disregarded.
• Without orthogonal regularization, the performance of SaLoRA degenerated. These results validate that incorporating orthogonal regularization into SaLoRA ensures the independence of doublets from one another, leading to a significant enhancement in its performance.

Visualization of Four Components. We plotted the visualization of expected sparsity $\widehat{b}$, the Lagrangian multiplier $\lambda$ and $\left|\right|A^{T}A-I{\left|\right|}_F^2$ and $\left|\right|B^{T}B-I{\left|\right|}_F^2$ to show whether these four components were regularized by Lagrangian relaxation and orthogonal regularization, respectively. Specifically, we fine-tuned the RoBERTa-base using SaLoRA on the CoLA, STS-B and MRPC datasets. The initial Lagrangian multiplier $\lambda$ was 0 and the target sparsity b was 0.5. From Figure 6, we see that:

Figure 6. Visualization of expected sparsity $\widehat{b}$ and the Lagrangian multiplier $\lambda$ under Lagrangian relaxation, and $\left|\right|A^{T}A-I{\left|\right|}_F^2$ and $\left|\right|B^{T}B-I{\left|\right|}_F^2$ under orthogonal regularization: (a) expected sparsity $\widehat{b}$; (b) Lagrangian multiplier $\lambda$; (c) A of $W^O$ at the first layer; and (d) B of $W^O$ at the first layer.

• The expected sparsity $\widehat{b}$ decreased from 0.92 to about 0.50, and the Lagrangian multiplier $\lambda$ kept increasing during training. The results indicate that the SaLoRA algorithm placed more emphasis on satisfying the constraints, eventually reaching a trade-off between satisfying the constraints and optimizing the objective function.
• The values of $\left|\right|A^{T}A-I{\left|\right|}_F^2$ and $\left|\right|B^{T}B-I{\left|\right|}_F^2$ could be optimized to a highly negligible level (e.g., 0.001). Therefore, this optimization process enforced orthogonality upon both matrices A and B, guaranteeing the independence of doublets from one another.

Comparison of Training Efficiency. We analyzed the efficiency of SaLoRA in terms of memory and computational efficiency, as shown in Table 4. Specifically, we selected two scales of the RoBERTa model, that is, $Ro{B}_{base}$ and $Ro{B}_{large}$, and measured the peak GPU memory and training time under different batch sizes on an NVIDIA A100 (40 GB) GPU. From Table 4, we see that:

Table 4. Comparison of training efficiency between LoRA and SaLoRA on the MRPC dataset.

Model	BS	Method	GPU Mem	Time
${\mathrm{RoB}}_{\mathrm{base}}$	16	LoRA	3.54 GB	15 min
		SaLoRA	3.54 GB	20 min
	32	LoRA	5.34 GB	14 min
		SaLoRA	5.35 GB	15 min
	64	LoRA	9.00 GB	13 min
		SaLoRA	9.00 GB	14 min
${\mathrm{RoB}}_{\mathrm{large}}$	16	LoRA	7.44 GB	44 min
		SaLoRA	7.46 GB	53 min
	32	LoRA	12.16 GB	40 min
		SaLoRA	12.18 GB	44 min
	64	LoRA	21.80 GB	38 min
		SaLoRA	21.82 GB	41 min

• The GPU memory usages of both methods were remarkably similar. Such results demonstrate that SaLoRA does not impose significant memory overhead. The reason behind this is that SaLoRA only introduces gate matrices in contrast to LoRA. The total number of parameters was $r\times L\times M$. In this experiment, r denotes the rank of the incremental matrix (set at 8), L corresponds to the number of layers within the model (12 for $Ro{B}_{base}$ and 24 for $Ro{B}_{large}$) and M stands for the number of modules in each layer (set at 6).
• The training time of SaLoRA increased by 11% when using a batch size of 32 compared with LoRA. This suggests that the additional computational requirements introduced by SaLoRA are justified by its notable gains in performance. This is because SaLoRA is only utilized during a specific training phase ($T_{start}$ to $T_{end}$) comprising 30% of the overall training time. With the remaining 70% being equivalent to LoRA, the overall impact on training time remains manageable.

The Resulting Rank Distribution. Figure 7 shows the resulting rank of each incremental matrix obtained from fine-tuning RoBERTa-base with SaLoRA. We observed that SaLoRA always assigned higher ranks to modules ($W^U$, $W^O$ and $W^V$) and layers (4, 5, 6 and 7). This aligns with the empirical results shown in Figure 1, indicating that modules ($W^U$, $W^O$ and $W^V$) and layers (4, 5, 6 and 7) play a more important role in model performance. Hence, these findings not only validate SaLoRA’s effective prioritization of critical modules and layers, but also emphasizes its capacity to learn the structure-aware intrinsic rank of the incremental matrix.

Figure 7. The resulting rank of each incremental matrix obtained from fine-tuning RoBERTa-base on MRPC with SaLoRA. The initial rank is set at 8, and the target sparsity is 0.5. The x-axis is the layer index and the y-axis represents different types of modules.

5. Conclusions

In this paper, we present SaLoRA, a structure-aware low-rank adaptation method that adaptively learns the intrinsic rank of each incremental matrix. In SaLoRA, we introduced a diagonal gate matrix to adjust the rank of the incremental matrix by penalizing the $L_0$ norm based on the count of activated gates. To enhance training stability and model generalization, we orthogonally regularized $\mathit{B}$ and $\mathit{A}$. Furthermore, we integrated a density constraint and employed Lagrangian relaxation to control the number of valid ranks. In our experiments, we demonstrated that SaLoRA effectively captures the structure-aware intrinsic rank and consistently outperforms LoRA without significantly compromising training efficiency.