Parameter-Efficient Fine-Tuning via General Linear Structural Regularization for High-Rank Adaptation

Abstract

Parameter-efficient fine-tuning (PEFT) enables large language models to adapt to downstream tasks with low computational cost. As a representative high-rank PEFT method, MoRA (High-Rank Updating for Parameter-Efficient Fine-Tuning) improves update expressiveness through a compression–transformation–decompression reparameterization mechanism. However, its bottleneck subspace is still modeled using a freely learned linear transformation. In addition, grouped compression may project information from different original directions into shared bottleneck coordinates. This may reduce subspace separability and lead to inefficient utilization of the effective update space. To address this limitation, we propose GL-log-MoRA, which introduces a learnable general linear transformation into the MoRA bottleneck subspace and applies log-determinant regularization to encourage a more balanced spectral structure. In this way, the proposed method improves directional coordination and subspace expressiveness without imposing hard structural constraints or causing noticeable memory overhead. We evaluate GL-log-MoRA on five benchmarks: LogiQA, Financial PhraseBank, GSM8K, FinQA, and HotpotQA. The results show that GL-log-MoRA achieves the best performance on these downstream tasks and yields small but consistent improvements over MoRA under the same parameter budget. Compared with MoRA, GL-log-MoRA improves LogiQA from 42.50% to 45.45% and Financial PhraseBank from 81.60% to 83.02%. It also improves GSM8K from 63.1% to 64.6%, FinQA from 10.02% to 10.23%, and HotpotQA from 70.6% to 70.8%. Meanwhile, the average empirical effective-rank indicator increases from 1.05 to 2.80. Peak GPU memory changes only slightly, from 18.21 GB to 18.28 GB.

1. Introduction

Large language models (LLMs), such as GPT and LLaMA, have achieved remarkable success across a wide range of natural language processing tasks [1,2]. However, their rapidly increasing parameter scale has made full-parameter fine-tuning prohibitively expensive in both storage and computation, especially in resource-constrained scenarios. As a result, parameter-efficient fine-tuning (PEFT) has become an important research direction for both practical deployment and efficient model specialization. PEFT adapts downstream models by training only a small subset of parameters while keeping the backbone frozen.

Among existing PEFT methods, Adapter, Prompt Tuning, and LoRA (Low-Rank Adaptation) represent three widely studied paradigms. In particular, LoRA has become one of the most influential approaches because of its simplicity, low memory overhead, and compatibility with large-scale training. By decomposing the update matrix into two low-rank factors, LoRA significantly reduces the number of trainable parameters. However, its strict low-rank structure also limits the expressive capacity of parameter updates. This limitation may become problematic for tasks requiring complex high-dimensional transformations and strong reasoning ability.

To alleviate the limitation of low-rank updates, recent studies have explored high-rank PEFT methods. As a representative high-rank PEFT approach, MoRA (High-Rank Updating for Parameter-Efficient Fine-Tuning) replaces conventional low-rank factorization with a compression–transformation–decompression mechanism. By doing so, it increases the theoretical rank upper bound of the update matrix under a similar parameter budget. This design provides stronger expressive potential than standard LoRA and demonstrates the promise of high-rank adaptation. Nevertheless, improving the theoretical rank alone does not guarantee that the bottleneck subspace can be effectively utilized during optimization.

More specifically, the bottleneck transformation in MoRA is still modeled as a freely learned linear mapping without explicit structural guidance inside the compressed subspace. In addition, the grouped compression mechanism may mix information from different original directions into shared bottleneck coordinates. As a result, the learned bottleneck transformation may exhibit spectrum concentration, direction redundancy, and effective-rank degeneration in practice. In other words, although MoRA expands the potential update space, it does not explicitly regulate how different bottleneck directions are organized or coordinated. This limitation suggests that improving the internal structure of the bottleneck subspace may be as important as increasing the theoretical rank itself in high-rank PEFT.

Motivated by this observation, we propose GL-log-MoRA (General Linear Structural Regularization for High-Rank Parameter-Efficient Fine-Tuning). It is a structure-guided high-rank PEFT method that enhances bottleneck subspace utilization through soft general linear regularization. Instead of imposing a hard geometric constraint, we introduce a learnable general linear transformation into the MoRA bottleneck space and apply a log-determinant regularization term to encourage a more balanced singular-value structure. In this way, the proposed method preserves the expressive flexibility of MoRA while improving directional coordination and structural efficiency inside the bottleneck subspace.

The main contributions of this work are as follows:

•

We propose GL-log-MoRA, a high-rank PEFT framework that improves bottleneck subspace modeling by introducing a learnable general linear transformation into MoRA.

•

We design a log-determinant regularization mechanism to alleviate spectrum concentration and effective-rank degeneration. This mechanism encourages more balanced utilization of bottleneck directions during optimization.

•

We conduct experiments on multiple reasoning and classification benchmarks and provide efficiency and spectral analyses. The results show that GL-log-MoRA consistently improves over LoRA and MoRA under the same parameter budget, with only negligible additional memory cost.

2. Related Work

2.1. Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) aims to adapt large pre-trained models by updating only a small subset of parameters, thereby reducing optimization cost, memory consumption, and storage overhead. Among existing PEFT methods, LoRA is one of the most influential approaches. It represents the weight update as the product of two low-rank matrices while freezing the backbone parameters, so that downstream adaptation can be achieved with a small number of trainable parameters and the learned update can be merged back into the original model after training [3]. Its main advantage lies in its simplicity and efficiency, but the strict low-rank structure may limit the expressive capacity of parameter updates, especially for tasks requiring complex transformations.

In addition to LoRA, several PEFT paradigms have been proposed to reduce adaptation costs. Adapter inserts lightweight bottleneck modules into Transformer blocks, but it introduces additional inference modules and may increase latency [4]. Prompt Tuning and Prefix Tuning optimize continuous task-specific vectors while keeping the backbone frozen, although their expressive capacity may be limited on complex reasoning tasks and prefix tokens may increase sequence length [5,6]. IA3 further reduces parameter cost by learning element-wise scaling vectors for attention and feed-forward activations, but its structural flexibility is relatively limited [7]. Several LoRA-based extensions further improve the parameterization of low-rank updates. AdaLoRA dynamically reallocates the rank budget across layers according to parameter importance [8]. DoRA decomposes weight adaptation into magnitude and direction components [9]. QLoRA combines LoRA with quantized backbone weights to reduce memory consumption [10]. ReLoRA periodically merges low-rank updates to accumulate a higher-rank effective update [11]. These methods demonstrate the effectiveness of PEFT. However, most of them either remain within a low-rank update framework or primarily focus on efficiency. As a result, the structural organization of higher-rank update spaces remains insufficiently explored.

2.2. High-Rank PEFT

Because strict low-rank constraints may limit representation capacity, recent studies have increasingly explored high-rank or full-rank PEFT. MoRA is a representative method in this direction. Instead of factorizing the update into two low-rank matrices as in LoRA, MoRA adopts a compression–transformation–decompression mechanism and uses a square bottleneck matrix to increase the theoretical rank upper bound of the effective update under a similar parameter budget [12]. This design alleviates the low-rank bottleneck and shows that, under the same number of trainable parameters, increasing the effective update rank can improve adaptation capability.

Several recent methods further expand this line of research. RandLoRA performs full-rank PEFT by learning linear combinations of fixed random low-rank basis matrices, where the trainable part is restricted to diagonal scaling parameters. This study shows that full-rank updates can reduce the performance gap between LoRA and full fine-tuning. In some settings, this gap can be nearly eliminated, especially on more demanding multimodal tasks [13]. HiRA introduces a Hadamard-product-based high-rank adaptation mechanism for large language models. It couples the learned update with the frozen pre-trained weight through multiplicative modulation. This design increases update expressiveness while keeping the parameter budget low, thereby demonstrating the effectiveness of Hadamard-style high-rank adaptation [14]. ABBA further improves expressiveness by modeling the update as the Hadamard product of two independently learnable low-rank matrices. Unlike HiRA, ABBA fully decouples the update from the frozen pre-trained weight. This allows both components to be optimized freely. ABBA reports stronger empirical performance on arithmetic and commonsense reasoning benchmarks [15]. HyperAdapt takes a different route and performs high-rank adaptation by applying row-wise and column-wise diagonal scaling to the pre-trained weight matrix, requiring only n + m trainable parameters for an n × m layer. This method shows that high-rank transformations can be induced with extremely small parameter overhead while remaining competitive with full fine-tuning and strong PEFT baselines [16]. These studies suggest that high-rank PEFT is a promising direction for overcoming the limitations of conventional LoRA-style low-rank updates. However, existing high-rank PEFT methods mainly focus on enlarging the effective update rank or redesigning the update form itself. Their primary emphasis is therefore on expanding expressive capacity. By contrast, even when the theoretical rank becomes much larger, the internal organization of bottleneck directions may still be poorly conditioned in practice, leading to spectrum concentration and ineffective utilization of subspace directions.

2.3. Structure-Guided PEFT

Beyond rank expansion, another important research direction is to exploit structural information to guide parameter updates. SVFT is a representative example: it uses the singular vectors of pre-trained weights to construct update directions and learns only a small number of coefficients, thereby injecting structural priors from the original model into PEFT [17]. This line of work indicates that adaptation quality may be improved not only by increasing the number of trainable parameters, but also by designing better update directions.

Related ideas also appear in methods that impose explicit geometric constraints. Orthogonal LoRA constrains low-rank matrices on an orthogonal manifold, aiming to stabilize the update space and reduce redundancy among learned directions [18]. These studies collectively show that structural priors can help prevent degenerate update patterns and make PEFT more stable and effective.

Nevertheless, these structure-guided approaches differ from the problem considered in this work. Methods such as SVFT rely on fixed structural bases extracted from pre-trained weights [17], while orthogonality-based methods impose relatively hard geometric constraints on the optimization space [18]. Existing high-rank methods, including RandLoRA, HiRA, ABBA, and HyperAdapt, mainly improve expressiveness by increasing the rank of the effective update or redesigning the parameterization [13,14,15,16]. However, they do not explicitly regulate the internal spectral organization of the bottleneck transformation itself. Therefore, the key research gap is that the internal structure of the high-rank bottleneck subspace remains insufficiently constrained, even when the update rank is enlarged. To address this gap, we propose GL-log-MoRA, which introduces a learnable general linear transformation into the MoRA bottleneck subspace and applies log-determinant regularization as a soft structural prior. Unlike methods that merely enlarge rank, our method explicitly improves the internal organization of bottleneck directions. Unlike methods based on fixed singular-vector bases, it learns the structural transformation jointly with task adaptation. In addition, unlike hard-constraint methods, it preserves optimization flexibility while encouraging a more balanced spectral structure.

For clarity, Table 1 summarizes the representative PEFT methods, high-rank PEFT methods, and structure-guided adaptation approaches discussed above.

Table 1. Comparison of representative PEFT methods, high-rank PEFT methods, and structure-guided adaptation approaches.

Category	Method	Core Idea	Limitation
PEFT	Adapter [4]	Insert small bottleneck layers into Transformer blocks for parameter-efficient adaptation	Introduces additional modules and increases inference latency
	Prompt Tuning [5]	Optimize continuous prompt embeddings while freezing backbone parameters	Limited expressive capability for complex reasoning tasks
	Prefix Tuning [6]	Introduce learnable prefix tokens into attention layers	Increases sequence length and computational cost
LoRA-based methods	LoRA [3]	Decompose weight updates into low-rank matrices to reduce trainable parameters	Strict low-rank constraint limits representation capability
	AdaLoRA [8]	Dynamically allocate rank according to parameter importance during training	Additional complexity in rank scheduling
	DoRA [9]	Decompose weight magnitude and direction to improve adaptation stability	Introduces additional optimization complexity
	ReLoRA [11]	Periodically merges low-rank updates into the model to accumulate a higher-rank effective update.	Requires repeated merging and reinitialization, increasing optimization complexity.
High-rank PEFT	MoRA [12]	Use bottleneck mapping to obtain higher effective update rank under fixed parameter budget	Lack of structural constraint in bottleneck subspace
	HiRA [14]	Introduces Hadamard-product-based high-rank adaptation to enhance update expressiveness under a low parameter budget.	Emphasizes rank expansion rather than explicit spectral regulation of subspace directions.
	ABBA [15]	Uses the Hadamard product of two independently learnable low-rank matrices to obtain a highly expressive update.	Improves expressiveness, but does not explicitly constrain structural organization inside the update space.
	HyperAdapt [16]	Induces high-rank adaptation through row-wise and column-wise diagonal scaling with very few trainable parameters.	Focuses on high-rank parameterization efficiency rather than bottleneck spectral conditioning.
	RandLoRA [13]	Forms full-rank updates by learning combinations of fixed random low-rank basis matrices.	Improves rank expressiveness, but does not explicitly regulate internal bottleneck structure.
Structure-guided methods	SVFT [17]	Constructs update directions from singular vectors of pre-trained weights and learns only a few coefficients.	Relies on fixed structural bases extracted from pre-trained weights.
	Orthogonal subspace learning [18]	Constrain low-rank matrices on orthogonal manifold	Hard constraints may reduce flexibility
Ours	GL-log-MoRA	Introduce a learnable general linear transformation with log-determinant regularization	Requires additional regularization design and hyperparameter tuning

Table 1. Comparison of representative PEFT methods, high-rank PEFT methods, and structure-guided adaptation approaches.

Category	Method	Core Idea	Limitation
PEFT	Adapter [4]	Insert small bottleneck layers into Transformer blocks for parameter-efficient adaptation	Introduces additional modules and increases inference latency
	Prompt Tuning [5]	Optimize continuous prompt embeddings while freezing backbone parameters	Limited expressive capability for complex reasoning tasks
	Prefix Tuning [6]	Introduce learnable prefix tokens into attention layers	Increases sequence length and computational cost
LoRA-based methods	LoRA [3]	Decompose weight updates into low-rank matrices to reduce trainable parameters	Strict low-rank constraint limits representation capability
	AdaLoRA [8]	Dynamically allocate rank according to parameter importance during training	Additional complexity in rank scheduling
	DoRA [9]	Decompose weight magnitude and direction to improve adaptation stability	Introduces additional optimization complexity
	ReLoRA [11]	Periodically merges low-rank updates into the model to accumulate a higher-rank effective update.	Requires repeated merging and reinitialization, increasing optimization complexity.
High-rank PEFT	MoRA [12]	Use bottleneck mapping to obtain higher effective update rank under fixed parameter budget	Lack of structural constraint in bottleneck subspace
	HiRA [14]	Introduces Hadamard-product-based high-rank adaptation to enhance update expressiveness under a low parameter budget.	Emphasizes rank expansion rather than explicit spectral regulation of subspace directions.
	ABBA [15]	Uses the Hadamard product of two independently learnable low-rank matrices to obtain a highly expressive update.	Improves expressiveness, but does not explicitly constrain structural organization inside the update space.
	HyperAdapt [16]	Induces high-rank adaptation through row-wise and column-wise diagonal scaling with very few trainable parameters.	Focuses on high-rank parameterization efficiency rather than bottleneck spectral conditioning.
	RandLoRA [13]	Forms full-rank updates by learning combinations of fixed random low-rank basis matrices.	Improves rank expressiveness, but does not explicitly regulate internal bottleneck structure.
Structure-guided methods	SVFT [17]	Constructs update directions from singular vectors of pre-trained weights and learns only a few coefficients.	Relies on fixed structural bases extracted from pre-trained weights.
	Orthogonal subspace learning [18]	Constrain low-rank matrices on orthogonal manifold	Hard constraints may reduce flexibility
Ours	GL-log-MoRA	Introduce a learnable general linear transformation with log-determinant regularization	Requires additional regularization design and hyperparameter tuning

3. Method

In this section, we first review the formulation of LoRA and MoRA, and analyze the structural limitations of MoRA. Based on this analysis, we then introduce the proposed GL-log-MoRA method together with its regularization mechanism and theoretical insights.

3.1. Basic Form of MoRA

To clarify the structural difference between LoRA and MoRA, we briefly review their formulations. LoRA performs parameter-efficient adaptation by introducing a low-rank update to the frozen pre-trained weight matrix, whereas MoRA replaces the conventional low-rank factorization with a compression-transformation-decompression mechanism.

Given a frozen pre-trained weight matrix $W_0$, LoRA computes the output as:

$$y = W_{0}x + ∆Wx = W_{0}x + BAx,$$

(1)

where ΔW = BA denotes the low-rank update matrix, $A \in R^{r\times k}$ and$B \in R^{d\times r}$ are trainable matrices. As a result, the rank of the update matrix satisfies rank (ΔW) ≤ r. To alleviate this limitation, MoRA introduces a structured reparameterization mechanism. Specifically, the update can be written as

$$y=W_{0}x + f_{\mathit{decomp}}\left(M{f}_{\mathit{comp}}\left(x\right)\right),$$

(2)

Here, $f_{comp}$ and $f_{decomp}$ denote the compression and decompression operators, respectively, and $M \in R^{r\times r}$ is a trainable square matrix.

Unlike LoRA, which explicitly restricts the update matrix to a low-rank factorization, MoRA constructs a square bottleneck transformation and combines compression and decompression operators to map the low-rank update into a higher-dimensional internal transformation space. Under the same parameter budget, this design significantly increases the theoretical rank upper bound of the update matrix. The effective update rank can be approximately expressed as

$$\widehat{r} \approx \sqrt{\left(d+k\right)r},$$

(3)

where d and k denote the input and output dimensions of the weight matrix, respectively. This relationship indicates that even with a relatively small external rank $r$, the structured bottleneck transformation in MoRA can lead to a much higher effective update rank. However, a higher theoretical rank does not necessarily guarantee efficient utilization of the bottleneck subspace during optimization. This motivates a further analysis of the structural limitations of MoRA.

3.2. Structural Limitations of MoRA

Although MoRA increases the theoretical rank of parameter updates, its bottleneck transformation is still modeled as a freely learned linear mapping without explicit structural guidance. As a result, the singular values of the bottleneck matrix may become highly imbalanced during training, leading to spectrum concentration and reduced effective-rank utilization. Moreover, the compression stage uses grouped aggregation. As a result, information from different original dimensions can be mixed into the same bottleneck coordinates. This may weaken the separability of subspace directions and produce redundant updates. These observations suggest that increasing the theoretical rank alone is not sufficient, and motivate the introduction of additional structural regularization within the bottleneck subspace.

3.3. The Proposed GL-Log-MoRA Method

To enhance structural expressiveness inside the bottleneck subspace, we introduce a learnable general linear transformation matrix Q before the MoRA update matrix. The matrix $Q \in R^{r\times r}$ has the same dimensionality as the bottleneck subspace and is initialized as the identity matrix. The original compression and decompression operators remain unchanged, so the proposed modification only acts inside the low-dimensional bottleneck space. The structural comparison between MoRA and GL-log-MoRA is shown in Figure 1.

Figure 1. Structural comparison between MoRA and GL-log-MoRA.

Figure 1. Structural comparison between MoRA and GL-log-MoRA.

Let $x \in R^d$ denote the input hidden representation and $W_0 \in R^{d\times d}$ denote the frozen pre-trained weight matrix. The forward computation of the adapted layer can then be written as

$$y = W_{0}x + f_{\mathit{decomp}}\left(QM{f}_{\mathit{comp}}\left(x\right)\right),$$

(4)

where $f_{comp}$ and $f_{decomp}$ denote the compression and decompression operators, respectively. The matrices $Q,M \in R^{r\times r}$ are trainable matrices that operate on the compressed bottleneck representation, where $r$ denotes the bottleneck dimension. The second term in Equation (4) represents the parameter-efficient update introduced by the proposed method, which can be equivalently written as:

$$f_{\mathit{decomp}}\left(QM{f}_{\mathit{comp}}\left(x\right)\right)=\mathsf{\Delta }Wx,$$

(5)

where $\mathsf{\Delta }W$ denotes the effective weight update applied to the input representation.

In practice, the compression operator reshapes the input vector into a sequence of low-dimensional blocks. Specifically, the input vector $x \in R^d$ is divided into $n$contiguous segments of length $r$, such that $d=nr$, and reshaped into a matrix $X \in R^{n\times r}$, where each row of $X$ corresponds to a segment of the original vector $x$. The compression operator can therefore be expressed as

$$X= f_{\mathit{comp}}\left(x\right)=\left[x_{1:r},x_{\mathrm{r}\text{:}2\mathrm{r}},\cdots ,x_{(\mathrm{n}-1)\mathrm{r}\text{:}\mathrm{nr}}\right].$$

(6)

After compression, the same bottleneck transformation $QM$ is applied to each block, where $Q,M \in R^{r\times r}$ and thus $QM \in R^{r\times r}$. The decompression operator then restores the original dimensional layout by concatenating the transformed blocks:

$$f_{\mathit{decomp}}\left(QM{f}_{\mathit{comp}}\left(x\right)\right)=concat\left(QM{f}_{\mathit{comp}}\left(x\right)\right),$$

(7)

Here, $\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\cdot )$ denotes concatenating the rows of the transformed block matrix into a vector, thereby mapping the block representation back to the original space $R^{\mathrm{d}}$. Under this block-wise compression-transformation-decompression implementation, the resulting update matrix $\mathsf{\Delta }W$ can be interpreted as a block-diagonal matrix composed of repeated transformation blocks:

$$\mathsf{\Delta }W = diag\left(QM, QM, \dots , QM\right) = \left[\begin{array}{cccc} QM& 0& \cdots & 0\\ 0& QM& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & QM \end{array}\right].$$

(8)

As a result, the proposed GL-log-MoRA enhances the structural expressiveness of the bottleneck subspace while preserving the efficient parameterization mechanism of MoRA.

Here, Equation (8) describes the effective update matrix $\mathsf{\Delta }W$, rather than the frozen pre-trained weight $W$. In the general case, if the input hidden representation has last dimension $k$, the effective update should satisfy $\mathsf{\Delta }W \in R^{d\times k}$ so that $\mathsf{\Delta }Wx$ is well-defined and has the same output dimension as the original linear transformation. The block-diagonal form in Equation (8) corresponds to a simplified square case with $d = k$, introduced only to clarify the block-wise compression–transformation–decompression mechanism.

To improve the clarity of the proposed method, Figure 2 illustrates the overall block-wise update architecture of GL-log-MoRA.

Figure 2. Overall architecture of GL-log-MoRA. The compression and decompression operators remain unchanged, while the bottleneck transformation is modified from M to QM within the bottleneck subspace.

To make the training process of the proposed method clearer, Algorithm 1 presents the optimization procedure of GL-log-MoRA.

Algorithm 1. GL-log-MoRA: Training Procedure with General Linear Structural Regularization

1: procedure GL-log-MoRA($\mathcal{M}$, $\mathcal{X}$, fcomp, fdecomp, Q, M, λ, δ, E, η) 2:           Freeze backbone model $\mathcal{M}$ 3:           Insert GL-log-MoRA adapters into target layers 4:           for epoch = 1 to E do 5:                      for each mini-batch (x_t, y_t) ∼ $\mathcal{X}$ do 6:                                 z_t ← fcomp(x_t) 7:                                 h_t ← Q M z_t 8:                                 ΔW x_t ← fdecomp(h_t) 9:                                 ŷ_t ← $\mathcal{M}$(x_t) + ΔW x_t 10:                              Ltask ← $ℒ$(ŷ_t, y_t) 11:                              Lreg ← − λ log det(QᵀQ + δI) 12:                              Ltotal ← Ltask + Lreg 13:                              Update Q and M using AdamW with learning rate η 14:                     end for 15:           end for 16:           return fine-tuned model $\mathcal{M}$ 17: end procedure

3.4. Motivation for General Linear Structural Constraints

To address the above limitations, we introduce a learnable general linear transformation Q into the bottleneck subspace. Rather than imposing a hard geometric constraint, Q serves as a flexible structural transformation that acts only in the compressed space. This design has two advantages. First, it preserves the expressive capacity of the original MoRA transformation by avoiding overly restrictive constraints on the trainable update. Second, it provides an explicit mechanism for coordinating different subspace directions before task-specific adaptation. In this way, Q is responsible for structural organization, while M remains responsible for expressive adaptation. This separation allows the model to improve subspace utilization without sacrificing the flexibility of the original bottleneck transformation.

3.5. Log-Determinant Regularization

The choice of log-determinant regularization is motivated by the goal of discouraging spectral collapse while preserving the high-rank adaptation capacity of the bottleneck transformation. In contrast, a Frobenius-norm penalty mainly controls the overall parameter magnitude, whereas a nuclear-norm penalty tends to favor low-rank structure, which is not aligned with the motivation of MoRA-style high-rank adaptation. In this work, $\lambda$ is set as a mild regularization coefficient so that the structural prior does not dominate task optimization, while $\delta$ is introduced only to ensure numerical stability in the log-determinant computation. To further stabilize the spectral structure of the learnable transformation Q, we apply a log-determinant regularization term:

$$L_{reg}= - \lambda \log \det \left(Q^{T}Q + \delta I\right),$$

(9)

where $\lambda$ denotes the regularization coefficient and $\delta$ is a small positive constant. Since $Q^{T}Q+ \delta I$ is symmetric positive definite for $\delta > 0$, the regularization term can also be interpreted from the perspective of singular values. Let ${\sigma }_i$ denote the singular values of Q. The regularization term can then be written as

$$L_{reg}=-\lambda {\Sigma }_i\log \left({\sigma }_i^2+\delta \right).$$

(10)

This regularization encourages a more balanced singular-value spectrum, preventing excessive concentration of update energy on a few dominant directions and helping maintain a well-conditioned transformation matrix.

3.6. Method Analysis

GL-log-MoRA decouples structural regulation from task-specific adaptation within the bottleneck subspace. Specifically, Q acts as a learnable structural transform that regulates interactions among bottleneck directions, while M remains a free trainable matrix that preserves adaptation flexibility. This design improves subspace utilization and numerical stability by alleviating degenerate mappings and spectral concentration without sacrificing expressive capacity. In addition, the log-determinant term serves as a soft structural prior, avoiding the projection or retraction steps required by hard orthogonality constraints. Consequently, the proposed method encourages a more balanced singular-value structure and more effective use of the high-rank update space.

4. Experimental Results and Analysis

4.1. Experimental Settings

Experiments were conducted using the LLaMA-3.1-8B-Instruct pre-trained model. All methods inserted PEFT modules into the Transformer attention projection matrices, including q_proj, k_proj, v_proj, and o_proj. The remaining backbone parameters were frozen. For a fair comparison, the compared methods were trained under the same model scale and training configuration. We used the AdamW optimizer with a learning rate of 2 × 10⁻⁴ for the bottleneck transformation matrix $M$ and 3 × 10⁻⁴ for the structural transformation matrix $Q$, together with a batch size of 4 and a maximum sequence length of 1024. All experiments were conducted on a single NVIDIA A800 GPU using FP16 mixed-precision training.

The above settings were chosen according to a combination of common PEFT practice, fairness of comparison, and computational constraints. The target modules were fixed to the attention projection layers for all compared methods. This follows common LoRA-style PEFT practice [3,9,12] and ensures consistency. The external bottleneck rank was fixed to $r = 8$, following a standard small-rank PEFT setting used in prior PEFT studies [3,12], while also enabling a fair comparison under the same parameter budget. The batch size and sequence length were selected to balance context coverage and GPU memory usage on a single A800 device. For GL-log-MoRA, a slightly larger learning rate was used for $Q$ than for $M$in order to facilitate the optimization of the newly introduced structural transformation. In the structural regularization term, the log-det coefficient $\lambda$ was set to 0.01 as a mild regularization strength so that the structural prior would not dominate the task objective. To evaluate engineering efficiency, we measured the average training throughput in tokens per second. For the attention projection layers of LLaMA-3.1-8B, the external bottleneck rank $r = 8$ corresponds to an internal transformation dimension of approximately 256 under the MoRA compression–decompression design, which already provides substantial expressive capacity.

For clarity and reproducibility, the main experimental settings are summarized in Table 2.

Table 2. Summary of the main experimental settings.

Setting	Value
Base model	LLaMA-3.1-8B-Instruct
Compared methods	LoRA, DoRA, MoRA, HyperAdapt, RandLoRA, GL-log-MoRA
Target modules	q_proj, k_proj, v_proj, o_proj
Backbone parameters	Frozen
Optimizer	AdamW
Learning rate for M	2 × 10−4
Learning rate for Q	3 × 10−4
Batch size	4
Maximum sequence length	1024
Training precision	FP16 mixed precision
Hardware	Single NVIDIA A800 GPU
$\mathrm{External} \mathrm{bottleneck} \mathrm{rank} r$	8
Internal transformation dimension in MoRA	Approximately 256
$\mathrm{Log}-\det \mathrm{regularization} \mathrm{coefficient} \lambda$	0.01
$\mathrm{Log}-\det \mathrm{regularization} \mathrm{coefficient} \delta$	0.001
Throughput metric	Tokens/s
Main benchmark types	Reasoning and classification

4.2. Main Results

Overall, the experimental results show three consistent patterns. First, GL-log-MoRA achieves stable gains over MoRA across multiple reasoning and classification benchmarks under the same parameter budget. Second, these improvements are obtained with negligible additional memory costs. Third, the performance gains are accompanied by better effective-rank utilization in the bottleneck subspace, suggesting that the proposed structural regularization improves not only task performance but also the internal update structure.

As shown in Table 3, under the unified setting of r = 8, GL-log-MoRA achieves consistent improvements over MoRA on the main downstream benchmarks. Specifically, the performance increases from 42.50% to 45.45% on LogiQA and from 63.10% to 64.60% on GSM8K. It also increases from 10.02% to 10.23% on FinQA, from 81.60% to 83.02% on Financial PhraseBank, and from 70.60% to 70.80% on HotpotQA. In addition, compared with the newly added PEFT baselines, including DoRA, HyperAdapt, and RandLoRA, GL-log-MoRA achieves the best performance on all five downstream benchmarks. These results suggest that the proposed structural regularization can improve the effectiveness of high-rank adaptation across different task settings.

Table 3. Performance comparison of different PEFT methods on downstream benchmarks under r = 8.

Method	LogiQA	FPB	GSM8K	FinQA	HotpotQA	MMLU 0	MMLU 5
LoRA	42.19%	82.55%	59.9%	9.02%	69.4%	55.30%	63.90%
DoRA	42.38%	82.49%	62.6%	9.12%	69.8%	55.30%	63.80%
HyperAdapt	42.86%	82.36%	62.4%	9.61%	70.1%	54.86%	63.36%
RandLoRA	43.08%	82.61%	62.9%	9.82%	70.3%	55.19%	63.83%
MoRA	42.50%	81.60%	63.1%	10.02%	70.6%	54.80%	64.60%
GL-log-MoRA	45.45%	83.02%	64.6%	10.23%	70.8%	54.40%	64.10%

Among these benchmarks, GSM8K [19], LogiQA [20], FinQA [21], and HotpotQA [22] are reasoning-oriented tasks, while Financial PhraseBank [23] is a financial sentiment classification task. The relatively clear gains on LogiQA and GSM8K suggest that structurally constrained high-rank updates are beneficial for logical and arithmetic reasoning. FinQA adopts a strict exact-match metric, resulting in relatively low absolute scores for all methods, but GL-log-MoRA still achieves the best result. The improvement on Financial PhraseBank further indicates that the proposed method is not limited to reasoning tasks and can also be effective in classification settings.

For HotpotQA, the performance gap between different methods is relatively small. This is mainly because the oracle supporting-facts setting provides the supporting evidence in advance, reducing the task difficulty from full multi-hop reasoning to answer extraction. Therefore, the representational demand on the adaptation subspace is relatively limited. Nevertheless, GL-log-MoRA still maintains the best performance, showing that the proposed structural design remains stable without introducing performance degradation.

The MMLU results are used as a supplementary sanity check under different prompting settings. Here, MMLU 0 denotes zero-shot evaluation, while MMLU 5 denotes five-shot evaluation. The MMLU scores of different methods remain within a relatively close range, suggesting that downstream fine-tuning does not severely damage the model’s general knowledge capability. Although GL-log-MoRA does not obtain the highest MMLU score, its results remain comparable to the baseline methods.

It is also important to note that in MoRA-style adaptation, the external rank parameter r is not identical to the true rank of the final update matrix. Through the compression and decompression operators, a small external rank can correspond to a much larger internal transformation space. Therefore, when MoRA already possesses a relatively high effective rank, improving the internal organization of the subspace update may be more effective than simply increasing the external rank.

The experiments in this paper are intended to validate the proposed structural mechanism rather than to exhaustively optimize hyperparameters for each baseline. Formal statistical significance testing over multiple independent seeds was not feasible because full multi-task fine-tuning is computationally expensive. Nevertheless, we conducted repeated fine-tuning and evaluation runs on representative benchmarks under the same configuration. The models were fine-tuned at least three times and evaluated more than three times using deterministic greedy decoding. The reported scores remained unchanged or showed the same overall improvement trend. Therefore, the reported gains should be interpreted as stable empirical improvements under the current setting rather than statistically proven superiority.

4.3. Efficiency Analysis

As shown in Table 4, LoRA achieves the highest training throughput (2482.40 tokens/s) and the lowest peak memory usage (18.16 GB), reflecting the efficiency advantage of conventional low-rank adaptation. MoRA remains relatively close to LoRA in memory footprint. Its peak GPU memory increases only from 18.16 GB to 18.21 GB. However, its throughput decreases to 2271.49 tokens/s due to the additional compression–transformation–decompression operations. Compared with MoRA, GL-log-MoRA further reduces the training throughput to 1426.60 tokens/s. In contrast, peak memory changes only slightly from 18.21 GB to 18.28 GB. This corresponds to an increase of about 0.076 GB, or approximately 76 MB, which is less than 0.5%.

Table 4. Training throughput and peak GPU memory of LoRA, MoRA, and GL-log-MoRA.

Method	Throughput (Tokens/s)	Peak Memory (GB)
LoRA	2482.40	18.16
MoRA	2271.49	18.21
GL-log-MoRA	1426.60	18.28

A stride-based aggregation optimization was used in the implementation, and the results indicate that it has limited impact on memory footprint under the current high-rank setting. Overall, GL-log-MoRA preserves nearly the same memory profile as the original MoRA, which supports its engineering practicality. The throughput reduction mainly comes from the extra matrix operations and gradient propagation introduced by the additional bottleneck-space transformation, rather than from cache growth. In terms of parameter count, GL-log-MoRA introduces only one additional subspace matrix $Q \in R^{r\times r}$. When $r = 8$, this adds only 64 trainable parameters, which is negligible relative to the original model size.

These results indicate a clear accuracy–efficiency trade-off. LoRA remains the most efficient choice when training speed is the primary concern, whereas GL-log-MoRA is more suitable for scenarios where memory budget is tight and modest, but stable performance gains are valuable. MoRA can be regarded as an intermediate option that provides higher-rank adaptation with only limited efficiency loss relative to LoRA.

Therefore, the main overhead of GL-log-MoRA comes from additional computation in the bottleneck transformation rather than memory growth. This confirms that the proposed method trades training speed for improved bottleneck-subspace utilization and modest but stable performance gains.

4.4. Hyperparameter Sensitivity Analysis

To further examine the robustness of the proposed method, we conducted a hyperparameter sensitivity analysis on LogiQA, focusing on the log-det regularization coefficient $\lambda$and the numerical stability constant $\delta$.

As shown in Table 5, the performance remains relatively stable over a moderate range of $\lambda$ values, while removing the regularization ($\lambda = 0$) leads to a noticeable performance drop.

Table 5. Sensitivity analysis of the log-det regularization coefficient $\lambda$on LogiQA.

λ	0	1 × 10−4	1 × 10−3	1 × 10−2	5 × 10−2	1 × 10−1
EM	42.97	44.53	44.07	45.45	44.84	45.45

Table 6 further shows that varying $\delta$ under the default setting λ = 0.01 causes only limited performance fluctuation, which is consistent with its role as a numerical-stability term rather than a primary performance-controlling hyperparameter. These results support the reasonableness and robustness of the default hyperparameter setting.

Table 6. Sensitivity analysis of the numerical stability constant δ on LogiQA when λ = 0.01.

$\mathit{\delta }$	1 × 10−6	1 × 10−4	1 × 10−3	1 × 10−2	1 × 10−1
EM	44.38	44.68	45.45	44.53	44.22

4.5. Spectral Analysis

To further analyze the impact of the proposed method on subspace structure, we examined the singular-value distributions of the bottleneck update matrices at q_proj, k_proj, v_proj, and o_proj. Specifically, we counted the number of singular values larger than 0.1 and used it as an effective-rank indicator. The same threshold was applied to all methods, and the results were averaged over 32 layers.

Table 7 shows a clear progressive trend from LoRA to MoRA and further to GL-log-MoRA. LoRA yields the lowest effective-rank values across all projection layers, with an average of only 0.21. This indicates that, under strict low-rank parameterization, only a very limited number of update directions contribute substantially after training. MoRA increases the average effective-rank indicator to 1.05, suggesting that its high-rank bottleneck design indeed expands the usable update subspace relative to LoRA. However, the post-training spectrum still indicates that only a limited number of directions are effectively utilized in practice.

Table 7. Average effective rank of update matrices in attention projection layers.

Method	q_proj	k_proj	v_proj	o_proj	Avg.
LoRA	0.45	0.20	0.03	0.16	0.21
MoRA	2.59	0.97	0.09	0.56	1.05
GL-log-MoRA	4.03	2.12	1.75	3.28	2.80

By contrast, GL-log-MoRA further raises the average effective-rank indicator to 2.80 and achieves higher values in all four projection layers. Compared with MoRA, the proposed method alleviates singular-value concentration and makes subspace directions more evenly utilized. These results suggest that increasing the theoretical rank alone is not sufficient; the internal structural organization of the bottleneck subspace is also critical. After introducing log-det regularization, the singular-value distribution becomes smoother, which improves subspace expressiveness and aligns with the observed performance gains.

4.6. Additional Analysis and Discussion

4.6.1. Training Loss Dynamics

To provide a supplementary view of the training process under the same training budget, we plot the training loss curves of different methods on HotpotQA [22] under the experimental settings in Table 2. Since training on HotpotQA is computationally expensive, we present the loss values only over a limited range of training steps rather than over the full convergence process. Due to time and computational constraints, the displayed curves reflect training behavior for only a subset of the total training steps. Therefore, Figure 3 is mainly used to illustrate the stability of loss reduction during training, rather than to serve as strong evidence of superior final task performance.

As shown in Figure 3, the training losses of LoRA, MoRA, and GL-log-MoRA all decrease steadily without obvious oscillation or divergence, indicating stable optimization behavior for all three methods under the current setting. Compared with LoRA and MoRA, GL-log-MoRA exhibits slightly lower training loss over the displayed training stage. However, because only a limited number of training steps are shown, the loss has not yet fully flattened, and this should not be interpreted as evidence that the models have fully converged.

Meanwhile, the final HotpotQA performance difference between GL-log-MoRA and MoRA is marginal (70.8% vs. 70.6%). Therefore, Figure 3 should be interpreted cautiously. Although smoother loss reduction may indicate stable optimization behavior, it does not necessarily translate into better final task performance. This is because final performance is also affected by generalization ability and possible overfitting in later training stages. For example, a baseline such as LoRA may continue to reduce training loss in a later epoch, while its downstream performance may no longer improve accordingly because of overfitting. Therefore, Figure 3 serves as a supplementary illustration of training stability and intermediate optimization behavior, rather than direct evidence of a substantial improvement in final task performance on HotpotQA.

Figure 3. Training loss curves of different methods on HotpotQA from step 500 onward.

Figure 3. Training loss curves of different methods on HotpotQA from step 500 onward.

4.6.2. Ablation Study and Comparison with Alternative Structural Constraint Strategies

To further examine the effect of different structural constraints on the bottleneck transformation matrix $Q$, we compared GL-log-MoRA with several alternative variants on LogiQA. As shown in Table 8, introducing a learnable $Q$ without log-det regularization improves the EM score from 42.50% to 42.97%, indicating that the additional bottleneck transformation itself contributes to task adaptation.

Table 8. Comparison of different structural constraint strategies on LogiQA.

Method	Constraint on (Q)	LogiQA EM (%)
MoRA	None	42.50
GL-MoRA w/o log-det	Learnable (Q) without regularization	42.97
Hard-Stiefel-MoRA	Hard orthogonal constraint	43.14
Soft Orthogonal MoRA	Soft orthogonal regularization	44.22
GL-log-MoRA	Log-det regularization	45.45

We then evaluated two orthogonality-based variants applied to $Q$. Hard-Stiefel-MoRA constrains $Q$ through a hard orthogonal projection, while Soft-Orthogonal-MoRA introduces a soft regularization term that encourages $Q^{\mathrm{T}}Q$ to approach the identity matrix. These two variants achieve 43.14% and 44.22% EM, respectively, both outperforming the original MoRA baseline. This result suggests that imposing structural constraints on $Q$ can improve the utilization of the bottleneck subspace.

However, both orthogonality-based variants are still lower than GL-log-MoRA, which achieves the best result of 45.45%. This indicates that strict or approximate orthogonality can provide useful structural regularization, but it may also restrict the flexibility of the bottleneck transformation. In contrast, the proposed log-det regularization provides a softer spectral constraint. Instead of forcing $Q$ to be orthogonal, it encourages a more balanced spectrum and helps avoid spectral collapse while preserving the expressive flexibility of $Q$.

In preliminary experiments, we also attempted to directly impose an orthogonality constraint on the original MoRA update matrix $M$. However, this design led to severe performance degradation, suggesting that directly constraining $M$ may interfere with its role in task-specific adaptation. Therefore, our main comparison focuses on structural constraints applied to $Q$, while keeping $M$ as the original MoRA update matrix.

Overall, these results show that the performance gain of GL-log-MoRA does not simply come from adding an extra transformation matrix. Rather, the log-det regularization provides a more effective structural induction than hard or soft orthogonality constraints, leading to better bottleneck subspace utilization and stronger task performance.

5. Practical Implications

GL-log-MoRA is practically useful for memory-constrained fine-tuning scenarios that require modest performance gains without a substantial increase in GPU memory usage. Since the method introduces only a small additional bottleneck-space matrix, it is suitable for domain-specific adaptation of large language models in industrial intelligent systems, financial analysis, question answering, and decision-support applications. However, because the method reduces training throughput, it is more suitable for scenarios that prioritize memory efficiency and task performance over maximum training speed.

In addition, GL-log-MoRA may be useful for complex reasoning tasks, such as multi-hop question answering and numerical reasoning, where models need to coordinate multiple intermediate representations. By introducing structured regularization through Q, the proposed method improves bottleneck-subspace utilization and can therefore support domain-specific reasoning applications such as question answering, dialog systems, and automated decision-support systems.

Furthermore, GL-log-MoRA has the potential to be extended to multimodal learning and larger-scale adaptation scenarios. In these settings, both computational resources and model stability are important considerations. Future work will explore these directions to further evaluate the applicability of GL-log-MoRA in practical and high-stakes environments. The integration of external knowledge sources and more advanced reasoning strategies may also be explored to address complex reasoning tasks in real-world applications.

6. Conclusions

In this work, we proposed GL-log-MoRA, a parameter-efficient fine-tuning method that incorporates soft general linear structural constraints to enhance bottleneck subspace utilization. By introducing a learnable general linear transformation Q together with log-determinant regularization, GL-log-MoRA improves subspace expressiveness while maintaining computational efficiency and low memory overhead.

Our experiments on reasoning and classification benchmarks, including LogiQA, GSM8K, FinQA, FPB, and HotpotQA, show that GL-log-MoRA consistently outperforms LoRA and MoRA. It also remains competitive with the newly added representative baselines under the same parameter budget. The improvements are small but stable across all benchmarks, particularly on tasks requiring strong reasoning ability, such as LogiQA and GSM8K. Moreover, GL-log-MoRA achieves these gains with negligible additional memory cost. These characteristics make it suitable for deployment in memory-constrained environments.

The main contributions of this work are twofold. First, we introduced a soft structural regularization technique to improve bottleneck subspace expressiveness. Second, we showed that this technique can lead to modest but consistent improvements in task performance across multiple datasets.

Future work will focus on improving the computational efficiency of the proposed method. We will also evaluate its scalability on larger language models and multimodal tasks, and further examine its robustness across broader downstream settings.