Abstract
Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet uniform rank assignment ignores the varying contribution of layers to a downstream task. Adaptive methods such as AdaLoRA allocate ranks via gradient and singular-value importance, but they do not exploit the representation change induced by the task. We introduce FLowRA, a calibration-based rank-allocation framework that combines a lightweight flow-change descriptor (velocity, acceleration, and directional alignment of hidden representations) with gradient activity to solve a constrained integer-budget problem. FLowRA complements existing LoRA variants, requires only two user-configurable parameters, and supports multiple architectures through architecture-specific implementations. On GLUE, SuperGLUE, instruction-following, code, vision, and vision-language benchmarks, FLowRA consistently outperforms uniform LoRA, DoRA, and AdaLoRA under low trainable-parameter budgets, reducing the gap to full fine-tuning by 48.5% on GLUE relative to the strongest baseline. We compare FLowRA with AdaLoRA using sensitivity-score correlations and report component ablations to assess the contribution of flow-guided allocation.
1. Introduction
Fine-tuning large pre-trained models has become the dominant paradigm for adapting foundation models to downstream tasks. However, the vast parameter count of modern architectures makes full fine-tuning prohibitively expensive. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), address this by updating only a small fraction of parameters. A central, unresolved challenge in LoRA-based methods is how to distribute a fixed adaptation budget across layers and within weight matrices, a decision that remains largely heuristic and suboptimal.
The impressive achievements of deep learning have come at the cost of an exponential increase in the scale of neural network models [1]. Modern architectures now contain hundreds of millions, and in some cases, hundreds of billions of parameters, making them extremely resource-intensive both during training and in real-world deployment [2]. For example, language models like GPT-4 [3] or LLaMA 3 [4] require vast computational clusters composed of specialized accelerators [5] to function effectively. The associated costs include not only the necessary time and energy for training but also significant operational expenses related to their deployment [6], from high electricity consumption in data centers to technical limitations when integrating such models into mobile devices [7] or embedded devices [8].
Given these growing demands for resources and the associated costs, optimizing neural network models has become a critical challenge for the AI research and engineering community [9,10,11]. Reducing computational requirements without sacrificing model quality is essential to ensure accessibility, ecological sustainability, and large-scale practical deployment of AI technologies [12,13,14,15,16]. Numerous optimization strategies have been proposed in this context, including quantization [17,18,19,20], weight factorization [21,22], low-bitwidth representations [23,24,25], and hardware accelerators [26,27,28]. However, most of these approaches face limitations in terms of universality, implementation complexity, or inevitable trade-offs in model accuracy.
In addition to optimizing data processing using neural networks, efficient adaptation of pre-trained neural networks is an equally important challenge. This task is very labor-intensive on modern big datasets [22,29,30,31,32]. For this purpose, multiple tuning methods are often used [22,31,32,33,34], which allow one to not train a highly parameterized model completely, but to partially change the parameter values until the target values of quality metrics are achieved. The paradigm of fine-tuning large pre-trained models has become the cornerstone of modern deep learning [35,36,37,38,39]. Among PEFT techniques, LoRA and its variants have achieved dominant popularity due to their simplicity and strong empirical performance [22]. The core premise of LoRA is the low-rank intrinsic dimensionality hypothesis: weight updates during fine-tuning reside on a low-rank subspace. By injecting trainable low-rank matrices in parallel with frozen pre-trained weights, LoRA achieves performance competitive with full fine-tuning at a fraction of the memory and storage cost [22].
Existing LoRA variants can be grouped into three categories: (i) rank/target selection methods (e.g., AdaLoRA [34]), (ii) structural modifications (DoRA [32], VeRA [35], LoRA-FA [40]), and (iii) scalar modulation techniques [41]. While effective, these methods either rely on costly hyperparameter search or use importance signals that overlook how representations themselves change on task data. For instance, AdaLoRA scores components by singular value and gradient magnitude but does not ask: does this layer dramatically reshape activations for the target task?
Several interconnected challenges follow from this state of the field. First, the core hyperparameters (the rank r, the set of target layers, and scaling factors ) are selected via expensive grid search or rule-of-thumb, with no grounding to predict their optimal values for a given task-model pair [22,34]. This makes application to novel architectures or tasks a process of trial and error. Second, current LoRA variants treat adaptation as a uniform, layer-local operation and ignore the functional sensitivity within the network’s computational graph: adapting a layer in a critical bottleneck may have an outsized impact compared to one in a redundant pathway. Third, methods are highly specialized to specific architectural motifs, primarily the linear layers of transformers; applying LoRA to diverse topologies, such as convolutional networks or recurrent layers, requires ad hoc engineering [42]. Fourth, multi-task adapter composition remains challenging, since the naive merging of independently trained LoRA modules leads to interference and performance degradation [43].
We hypothesize that layers exhibiting large, task-specific representation changes are stronger candidates for adaptation. FLowRA implements this idea without any information-theoretic claim: it computes a simple, calibration-phase flow-change descriptor from finite differences of hidden states (velocity, acceleration, alignment). Together with gradient magnitudes, this descriptor feeds a greedy knapsack allocator that assigns ranks to layers and sub-blocks under a strict parameter budget. FLowRA’s design is deliberately modular: the flow descriptor can be used with any allocation scheme, and the resulting ranks can be realized with any LoRA variant (standard LoRA, DoRA, etc.). The framework requires only two user-configurable parameters and adapts to diverse architectures (transformers, CNNs, RNNs) without manual tuning.
In our experiments, we compare FLowRA with AdaLoRA under the evaluated parameter configurations, analyze correlations between flow-guided and SVD-based sensitivity scores, and perform component ablations to assess the contribution of the flow-change descriptor. The results show that flow-guided allocation improves performance over uniform and gradient-based allocation in the evaluated settings. We also discuss scenarios where the flow descriptor may provide limited additional benefit, such as when existing importance measures already identify the critical modules or when the calibration data are insufficient. The main contributions of this work are as follows:
- 1.
- A lightweight, calibration-based flow-change descriptor (velocity × acceleration × alignment) that captures task-specific representation change without training;
- 2.
- A formulation of rank allocation as a constrained integer budget-allocation problem, addressed using a lightweight greedy heuristic;
- 3.
- A systematic comparison with AdaLoRA through sensitivity-score correlation analysis, complemented by component ablations that quantify the contribution of the flow-change descriptor beyond gradient-based allocation;
- 4.
- A unified rank allocation framework that supports transformers, CNNs, residual networks, and gated RNNs through architecture-specific handlers.
The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 states the problem, describes FLowRA, and details the experimental setup. Section 4 reports the results, Section 5 discusses them together with limitations and failure cases, and Section 6 concludes the paper. The implementation, configuration files, detailed per-architecture algorithms, and extended experimental results are publicly available in the project repository [44].
2. Related Work
2.1. Low-Rank Adaptation and Adaptive Rank Allocation
Low-Rank Adaptation (LoRA) [22] freezes pre-trained weights and trains small low-rank update matrices in selected modules, achieving competitive performance with a fraction of the trainable parameters. The simplicity of this mechanism has prompted a wide family of variants. DoRA [32] decomposes weight updates into magnitude and direction components, providing finer control over adaptation without increasing parameter count. VeRA [35] and LoRA-FA [40] push parameter efficiency further by sharing random projections across layers. QLoRA [33] combines LoRA with 4-bit quantized base models, enabling fine-tuning of very large models on consumer hardware. Initialization-based methods such as PiSSA [45] and OLoRA [46] improve adaptation by aligning the initial low-rank subspace with the principal directions of the pre-trained weight matrix. Despite these advances, the choice of rank and target modules in most LoRA variants remains governed by uniform rules or manual search. FLowRA is designed as an allocation mechanism that can, in principle, be combined with several of these variants, decoupling the question of where and how much to adapt from the question of how the adapter is parameterized. In particular, structural modifications such as DoRA (magnitude/direction decomposition) and VeRA (shared random projections) are orthogonal to rank allocation, and FLowRA’s descriptor can be combined with any of them.
AdaLoRA [34] dynamically allocates rank during training by parameterizing weight updates with an SVD-like decomposition and pruning singular components with low importance, measured as . This yields a powerful allocation mechanism that already outperforms uniform LoRA. FLowRA’s key insight is that a pre-training, calibration-phase descriptor of representation change provides a complementary signal: layers that strongly transform task-specific activations may benefit from higher rank even if their current singular values are not large. Empirically, the Spearman correlation between and AdaLoRA’s SVD-based importance is only (Section 4.3), indicating that the two signals capture different properties. By combining both signals (flow descriptor and gradient activity), FLowRA achieves better allocation under the evaluated parameter configurations. The flow descriptor can be used as an additional input to any allocation scheme, not only the greedy knapsack presented here.
2.2. Task-Aware PEFT and Adapter Composition
Several methods study which modules to adapt given task or architecture characteristics. Prefix tuning [47] and prompt tuning [48] add learnable tokens to the input without touching weights. Adapter layers [38] insert small bottleneck modules between transformer sub-layers; while effective, they add inference latency and require architecture-specific placement. Recent work on adapter routing and mixture-of-adapters explores dynamic per-input allocation, which is complementary to FLowRA’s calibration-initialized rank allocation with training-stage refinement. FLowRA’s block-level allocation within a weight matrix is a meaningful distinction from all of the above: rather than choosing which modules receive adapters at module granularity, it also determines which sub-blocks within a module’s weight matrix receive capacity, targeting the most sensitive parameter regions.
Composing independently trained adapters for multi-task deployment is an active research area. Task arithmetic [49] adds adapter weight vectors, which are simple but lead to interference from overlapping subspaces. TIES-Merging [50] reduces interference by trimming and resolving sign conflicts before merging. LoRAHub [51] and adapter fusion methods [52] learn weighted combinations at inference time, at the cost of additional parameters or per-task forward passes. FLowRA’s flow-guided allocation naturally encourages adapters to occupy distinct subspaces per task, providing an allocation-level complement to these merging strategies. We evaluate composition quantitatively in Section 4.7, comparing against task arithmetic and TIES-Merging under matched per-task budgets.
2.3. PEFT for Modern Foundation Models
Evaluations based solely on LLaMA-2 or BERT-era NLU benchmarks do not fully reflect the diversity of modern foundation models. Recent surveys note that the relative performance ordering of PEFT methods can shift on instruction-tuned checkpoints and reasoning-heavy tasks [37]. The present study focuses primarily on LLaMA-2 checkpoints and includes an additional evaluation on LLaMA-3-8B (Section 4.1); results on other large language model families are outside the scope of this study.
3. Materials and Methods
3.1. Problem Statement
Let be a frozen pre-trained model with weight matrices for . Given a calibration set from the target task and a total trainable parameter budget B, we must assign an integer rank to each layer and, optionally, sub-block ranks within each matrix, such that
and the expected downstream task performance is maximized. The central difficulty is to define a utility function computable before training that predicts which modules will benefit most from adaptation.
The sensitivity and contribution of different parameters within a weight matrix to a specific downstream task are highly non-uniform. There may exist critical subspaces or localized regions within the pre-trained weight matrices that are disproportionately significant for task adaptation, corresponding to specific feature detectors, neuron subsets, or attention heads that are crucial for the target domain. Applying a global low-rank approximation across the entire matrix forces the model to distribute its limited adaptation capacity uniformly, potentially diluting expressiveness for the most critical, task-specific features. The challenge therefore decomposes into two interconnected sub-problems. The first is significance assessment: given a pre-trained weight matrix and a target task, how can we efficiently estimate an adaptation significance score for its parameters or coherent structural blocks (e.g., rows, columns, or subspaces) before extensive adaptation training? The second is resource-aware allocation: given a significance map and a strict total parameter budget B, how can we allocate a non-uniform low-rank budget across different regions of the weight matrices, concentrating more parameters on high-significance regions while using a minimal budget for less critical ones?
FLowRA constructs a layer-wise utility from two signals: a flow-change descriptor (computed purely from frozen activations) and the expected gradient magnitude . The allocation is then solved greedily. This formulation cleanly separates the where from the how.
3.2. Proposed Method: FLowRA
This section details the Flow-guided Low-Rank Adaptation (FLowRA) framework. We present a structured algorithm that uses a calibration-based flow-change descriptor to guide automated, task-aware adapter configuration. The general architecture is presented in Figure 1: the flow-change descriptor is estimated on the frozen model, and low-rank adapters are placed on the parameter blocks with the highest descriptor values, while the remaining parameters stay frozen.
Figure 1.
FLowRA architecture.
3.2.1. Flow-Change Descriptor
We seek a task-aware scalar that reflects how strongly a layer’s activations change on the target data. Intuitively, layers that substantially reshape representations are plausible candidates for adaptation. To capture this, we define a discrete flow-change descriptor that combines the first-order change (velocity) and second-order change (acceleration) of the hidden representations across consecutive layers. This descriptor is used purely as an empirical calibration feature; we make no claim of a formal information-theoretic interpretation. For practical implementation with discrete layers, we compute layer-wise divergence using finite differences on activations :
Each component of captures a distinct aspect of representation dynamics. The acceleration term measures the second-order change in activations and indicates where the transformation rate itself is changing rapidly, while the velocity term measures the first-order change, i.e., the basic transformation rate. The directional alignment is the cosine similarity between the acceleration and velocity vectors: when they are aligned (), transformations intensify along the current direction, and when they are opposed (), the flow is sharply turning or decelerating. Since the directional alignment term in Equation (1) can be negative, the signed descriptor is converted to its non-negative magnitude before rank allocation. This transformation retains the magnitude of representation change while ensuring that the allocation scores are non-negative. Large values therefore indicate layers where representations change strongly between consecutive processing steps. Its utility as a calibration feature is validated through ablation in Section 4.3 and in our public repository [44].
3.2.2. The FLowRA Algorithm
Algorithm 1 presents the complete FLowRA procedure. It consists of three sequential phases: profiling, where the model’s intrinsic information dynamics are measured; blueprint generation, where the adapter structure is computed; and injection and training, where the adapters are instantiated and fine-tuned. The algorithm is model-agnostic and requires only forward/backward passes and gradient computations. In the module-level implementation used for the main experiments, the block-level sensitivity and allocation steps are omitted, and ranks are assigned directly to target modules.
The flow-guided allocation determines the initial adapter ranks before training. During dynamic rank refinement, the calibration-based descriptor remains fixed, while gradient sensitivity is updated at predefined training steps. The RefineRanks procedure redistributes ranks according to the updated criticality scores, preserving the total trainable-parameter budget. Existing adapter components are retained where possible, and newly required components are initialized before training continues. Disabling refinement keeps the initial ranks fixed.
The profiling phase measures the model’s information dynamics before any adaptation. A small, representative calibration set is sufficient, typically samples, because we measure statistical properties rather than learn. Computing the flow divergence requires storing three consecutive activation tensors (); for transformer layers, is the output of the entire layer (attention + FFN). Gradients are computed with respect to the downstream task loss, which makes the profiling task-aware, and in multi-task scenarios, they can be averaged across tasks. From these quantities, FLowRA computes the layer criticality
which weights the flow descriptor by gradient activity and layer scale, and, for each block of a partitioned weight matrix, the block sensitivity
| Algorithm 1 FLowRA: Flow-Guided Low-Rank Adaptation |
|
The blueprint generation phase translates these measurements into an adapter architecture. The total budget is first distributed across layers proportionally to , and the remaining budget is then allocated greedily to the layer with the highest marginal utility , which provides a heuristic refinement of the integer rank assignment. Within each layer that receives a nonzero rank, blocks are processed in decreasing order of and receive rank proportional to . Blocks whose rank falls below the threshold (default 1) are pruned. The procedure prioritizes blocks with higher sensitivity scores and leaves any remaining budget unused if no feasible rank increment is possible. Sorting blocks costs per layer, so the entire blueprint generation takes , which is negligible compared to the profiling phase.
In the final phase, the blueprint specifies where low-rank adapters are injected. The block-wise structure ensures memory locality and allows efficient batched matrix multiplication. Only the injected adapter parameters are trained, while the frozen pre-trained weights act as a fixed feature extractor, and the adapters learn task-specific adjustments.
3.2.3. Properties of the Allocation Procedure
Overall, the rank allocation is a two-stage greedy procedure: layer budgets are assigned proportionally to , with the remaining capacity distributed according to marginal utility, and within each layer, the budget is allocated to blocks according to their sensitivity scores .
The allocation strategy is a heuristic motivated by diminishing returns in adaptation capacity, which can be illustrated by a concave surrogate utility such as . However, the layer criticality is an empirical proxy for task-specific utility, and the proposed allocation procedure does not establish the conditions required for a theoretical approximation guarantee. We therefore do not claim a worst-case approximation ratio or global optimality. Instead, the effectiveness of the allocation strategy is evaluated empirically in Section 4.3 and in our public repository [44].
The allocation rule favors layers with higher criticality relative to their parameter cost, although integer rounding and budget constraints do not guarantee a monotone ordering of the assigned ranks. Finally, the hyperparameters and control allocation sparsity explicitly. As , the budget concentrates on the single most critical layer; as , the rank within a layer concentrates on the single most sensitive block; and provides hard thresholding that prunes blocks with .
3.2.4. Implementation and Architecture-Specific Instantiations
The finite-difference computation in Equation (1) requires consecutive hidden representations to have compatible dimensions. When their shapes differ, the architecture-specific COMPUTE DIVERGENCE handler maps the representations to a common shape before evaluating the flow-change descriptor. Several dimension-alignment strategies are supported, including linear projections for vector-valued representations, convolutions for channel mismatches, and zero-padding for differences in channel or spatial dimensions.
In the default configuration, zero-padding is used to align the compared activation tensors to a common shape. This operation is applied only during calibration and does not introduce additional trainable parameters or modify the pre-trained model. Architecture-specific projection operators can be used when padding is not appropriate.
The algorithm relies on three architecture-specific handlers: ComputeDivergence evaluates Equation (1) for a given layer type, Partition defines how weight matrices are split into blocks (e.g., by input/output channels for convolutions or by heads for attention), and ExtractBlock extracts a block from a weight matrix for adapter injection. For very large models, storing the activations can be memory-intensive, so we use gradient checkpointing for the backward pass, process sequentially rather than in true batches, and quantize stored activations to FP16. When memory allows, profiling over can be batched, block sensitivity computation within a layer can be parallelized, and adapter training uses standard data parallelism. The general procedure of Algorithm 1 is instantiated for each layer type through these handlers; we summarize the key formulas below, and the complete per-architecture algorithms are available in the project repository [44].
For a fully connected layer with , the flow divergence is computed directly from and , and is partitioned into blocks (by default, , ). The sensitivity of block is proportional to its contribution to both information flow and gradient propagation
where and . Blocks with high sensitivity identify the input-output feature interactions that are most critical for task adaptation; under balanced partitioning, the adapter overhead reduces to .
For a convolutional layer, the kernel is flattened to and partitioned across input/output channels, and the flow divergence is computed from the feature maps as
where is the input to the previous layer. Channel-wise partitioning respects the convolutional inductive bias, so high-sensitivity blocks correspond to the input-output channel mappings most critical for spatial feature extraction.
For a residual block , the divergence and criticality focus on the residual function
where are the parameters of . Since skip connections implement identity mappings, the budget is allocated to the transformative component , with only an overhead over the underlying layer types.
For multi-head attention with H heads, a head-specific divergence and criticality are computed as
Head ranks are allocated as ; within each head, a fraction of the rank is assigned to the value and output projections and the remainder to the query and key projections, reflecting the dominant role of and in task adaptation. In a full transformer block, separate divergences and are computed for the attention and FFN outputs, and the budget is split according to
so that the allocation between token-level interactions (attention) and feature transformations (FFN) adapts to the task.
For gated recurrent units (GRU, LSTM), gate-specific divergences are computed over time steps, e.g., for the update gate (similarly for the reset gate and the candidate state). Gate criticalities determine the ranks , so capacity goes preferentially to the most dynamically active gates.
Table 1 summarizes the resulting costs. Across all architectures, FLowRA follows the same principles: flow divergence as the signal, criticality-weighted allocation proportional to , block-structured adaptation within parameter matrices, and partitioning that respects the architectural inductive bias. In all cases, the overhead scales linearly with the adaptation rank. FLowRA thus provides a complete, automated pipeline for deriving and deploying parameter-efficient adaptations across heterogeneous architectures.
Table 1.
Computational cost of FLowRA across architectures.
3.3. Experimental Setup
We evaluate FLowRA as a unified framework for rank allocation, architectural heterogeneity, dynamic refinement, and multi-task composition. Our experiments are organized to answer the following questions:
- 1.
- Does FLowRA outperform strong PEFT baselines across NLU/NLG/vision/multimodal tasks under the reported trainable-parameter budgets?
- 2.
- Does flow-based sensitivity score predict where rank is actually needed (and where it is wasted)?
- 3.
- Does progressive subspace refinement improve convergence and final quality vs. static low-rank updates?
- 4.
- Does polymorphic adaptation generalize across heterogeneous modules (attention, MLP, norms, conv/Vision Transformers)?
- 5.
- Is FLowRA robust to hyperparameters and data scarcity?
- 6.
- How does FLowRA’s multi-task composition compare with standard adapter merging methods in terms of interference?
For each downstream task, the calibration subset is selected from the corresponding training split and is retained for subsequent adapter training. We use up to 512 calibration examples, with the calibration set size limited by the number of available training examples in few-shot settings. During calibration, the pre-trained backbone remains frozen, and task-loss gradients are used only to estimate layer sensitivity and guide rank allocation. Validation and test examples are not used for calibration.
Unless otherwise specified, the main experiments reported in Section 4 use module-level LoRA adapters, with ranks allocated across target modules rather than individual weight-matrix sub-blocks. The block-level allocation described in Algorithm 1 represents a more fine-grained formulation of the framework and is not used in the default experimental configuration.
We report results on NLU benchmarks, namely GLUE [53], SuperGLUE [54], and question answering (SQuAD v1.1/v2.0 [55,56], Natural Questions [57], TriviaQA [58]); on NLG and reasoning benchmarks, GSM8K [59] and MATH [60]; on code generation, HumanEval [61] and MBPP [62]; and on vision and vision-language tasks, including ImageNet [63], CUB-200 [64], ADE20K [65], and VQA v2 [66], together with few-shot CUB-200 to measure data efficiency. Primary language model results use LLaMA-2 (7B, 13B, 70B) [67], while cross-modal results use ViT-B/16 and ViT-L/16 [68], ResNet-50 [69], and BLIP-2 [70]. We compare FLowRA with a frozen backbone, LoRA [22], AdaLoRA [34], DoRA [32], BitFit [71], prefix-tuning [47], and adapters [38]. The trainable parameter fractions vary across methods and are reported in comparison tables.
FLowRA exposes only two user-configurable parameters: the trainable-parameter budget B (e.g., 0.05–2%) and the composition threshold (default 0.3), used when combining task-specific adapters. The remaining algorithmic settings, including , , , and the refinement schedule , are determined internally rather than requiring manual configuration by the user. The block-level parameters and are not used in the default module-level implementation. In the default module-level configuration, the allocation exponent is fixed to 1, corresponding to rank allocation proportional to layer criticality. The refinement schedule is determined automatically from the total number of training steps T. An initial warm-up period of of T is followed by rank refinement every 100 training steps. Table 2 summarizes the key configuration parameters of the compared PEFT methods and provides a qualitative comparison of their tuning difficulty.
Table 2.
Configuration parameters and tuning complexity of PEFT methods.
Unless explicitly stated otherwise, numbers are averaged across 3 seeds, and ± indicates standard deviation. The gap to full fine-tuning (“Gap to FT”) is computed relative to full fine-tuning on the same benchmark; for improvements below one point, we note whether the difference is within one standard deviation of the baseline. At least one setting where FLowRA does not outperform AdaLoRA is discussed in Section 5. The complete experimental setup, configuration files, training scripts, implementation details, and extended experimental results are available in the project repository [44].
The comparisons in Table 4 are conducted in the low trainable-parameter regime but do not use strictly identical parameter budgets. The reported parameter fractions are 0.84% for LoRA, 1.21% for AdaLoRA, 0.92% for DoRA, and 0.87% for FLowRA.
Method-specific adaptation mechanisms and target-module selection strategies are preserved; therefore, identical adaptation configurations are not assumed.
To complement these comparisons, the results under a strictly matched trainable-parameter budget of 0.05% for all PEFT methods are reported in the Section 4.4.
4. Results
4.1. Additional Evaluation on LLaMA-3-8B
The main experimental evaluation uses LLaMA-2 checkpoints [67]. Table 3 additionally reports completed GLUE results on LLaMA-3-8B (mean ± std over 3 seeds). FLowRA reaches 88.1 with 0.87% trainable parameters and reduces the gap to full fine-tuning to −1.3 points, compared with −2.7 for DoRA, −3.0 for AdaLoRA, and −3.5 for LoRA. These results provide complementary evidence that the flow-guided allocation approach is effective on the evaluated LLaMA-3-8B checkpoint.
Table 3.
GLUE results on LLaMA-3-8B.
4.2. Main Results
4.2.1. Natural Language Understanding
Table 4 reports average scores (%, mean ± std over 3 seeds; the per-learning-rate std is given in Section 4.4 ). FLowRA reaches 87.4 on GLUE and 82.7 on SuperGLUE with only 0.87% trainable parameters, outperforming LoRA and DoRA at similar budgets. Critically, FLowRA closes the gap to full fine-tuning to −1.7 points, which corresponds to a 48.5% reduction in the remaining gap vs. the strongest baseline (DoRA at −3.3). The aggregate GLUE scores in Table 4 are computed as the arithmetic mean of the nine task-level scores, including WNLI, using the original unrounded values. Table 5 reports the corresponding task-level results rounded to two decimal places.
Table 4.
NLU results on GLUE and SuperGLUE.
Table 5.
GLUE task breakdown.
FLowRA improves not only the easy/high-signal tasks but also the brittle ones (CoLA, MRPC, RTE), where low-rank targeting mistakes typically show up first. Table 5 reports detailed GLUE scores, showing the largest gains over DoRA on CoLA (+1.4), MRPC (+1.7) and WNLI (+9.0), consistent with FLowRA allocating higher rank to bottleneck projections and task-specific FFN directions.
The gains carry over to extractive and open-domain question answering (Table 6): FLowRA improves over LoRA by 1.6 to 2.3 F1 points on SQuAD v1.1/v2.0, Natural Questions, and TriviaQA, remaining within 1.1 to 1.2 F1 points of full FT.
Table 6.
Question-answering results.
4.2.2. Instruction Following and Code Generation
Instruction following benefits disproportionately from FLowRA under complexity, where gradient directions become more anisotropic and adapter interference becomes more visible. On Alpaca [72], FLowRA nearly matches full FT on complex instructions (75.9 vs. 76.3) while outperforming DoRA by +6.2 (see Table 7).
Table 7.
Instruction following on Alpaca.
For code, FLowRA improves both pass@1 and pass@10 on HumanEval/MBPP, yielding the strongest PEFT average (46.6 vs. 44.9 for DoRA), approaching full FT (see Table 8).
Table 8.
Code generation results.
4.2.3. Mathematical Reasoning and Convergence
On LLaMA-2 13B with few-shot chain-of-thought prompting (8-shot GSM8K, 4-shot MATH), FLowRA improves GSM8K and MATH while also helping code benchmarks, giving the best PEFT average (35.7 vs. 34.3 for DoRA). Table 9 summarizes end performance.
Table 9.
Reasoning and code results on LLaMA-2 13B.
Beyond final scores, FLowRA converges faster. Table 10 shows GSM8K accuracy (%) vs. training steps: FLowRA reaches DoRA’s final accuracy roughly 1000 steps earlier and continues improving with diminishing but consistent gains.
Table 10.
GSM8K accuracy during training.
4.2.4. Vision and Vision-Language
FLowRA generalizes to heterogeneous architectures without hand-crafted targeting rules: it achieves 81.3 on ImageNet (ViT-B/16), 84.6 on CUB-200 (ResNet-50), 47.5 mIoU on ADE20K (ViT-L/16), and 71.4 on VQA v2 (BLIP-2), consistently reducing the gap to full FT to under 1 point across tasks. Table 11 reports these scores (%) together with the gap to full FT in parentheses; for comparison, the gaps of LoRA, adapters, and prefix-tuning range from 1.7 to 5.4 points.
Table 11.
Vision and vision-language results.
FLowRA is also substantially more data-efficient than LoRA in low-shot regimes, suggesting that flow-based allocation avoids wasting rank on low-sensitivity subspaces that overfit early. Table 12 reports few-shot accuracy (%) on CUB-200, together with the gap reduction, i.e., the fraction of the full FT–LoRA gap closed by FLowRA. At 32-shot, FLowRA improves over LoRA by +7.2 points, and the gap reduction ranges from 60.0% at 1-shot to 78.7% with the full training set.
Table 12.
Few-shot results on CUB-200.
A detailed empirical and analytical validation of the FLowRA mechanisms is available in the project repository [44], including an in-depth analysis of flow sensitivity patterns, progressive rank refinement, subspace alignment dynamics, and convergence behavior on reasoning benchmarks.
4.3. Ablations
We ablate FLowRA components on the GLUE development sets. Table 13 reports per-task scores together with the average drop relative to the full model. In the “w/o info flow allocation” variant, the flow-change descriptor is removed from the module-level criticality score, yielding . Gradient-based sensitivity, dynamic rank refinement, and the remaining adaptation mechanisms are retained. The experiments in Table 13 use module-level LoRA; therefore, the block-level sensitivity score in Equation (3) is not involved in this ablation. In the “w/o dynamic rank refinement” variant, the initial flow-guided rank allocation is retained, but the assigned ranks remain fixed throughout training. The “w/o polymorphic adapters” variant replaces polymorphic adapters with standard LoRA adapters while preserving flow-guided rank allocation. Orthogonal adapter composition is not activated during standard single-task GLUE training and is therefore not included in the single-task component ablations in Table 13. Multi-task adapter composition is examined separately in Section 4.7.
Table 13.
Component ablations on GLUE dev.
We additionally evaluate two reference configurations. In “Static uniform rank ()”, all adapted modules receive the same fixed rank of 16, without flow-guided rank allocation or dynamic rank refinement, while polymorphic adapters are retained. In contrast, the “LoRA baseline ()” uses conventional LoRA adapters without the additional FLowRA mechanisms. Thus, these two configurations share the same rank but differ in their adapter implementation.
As shown in Table 13, removing the flow-based component produces the largest performance decrease among the component ablations, reducing the average GLUE score from 87.0 to 82.8 (4.2 percentage points).
To quantify what additional signal the flow descriptor contributes, Table 14 compares normalized sensitivity scores per layer type for the flow descriptor , the gradient term , AdaLoRA’s SVD-based importance, and the combined FLowRA criticality on LLaMA-7B with a 512-sample MNLI calibration set. The Spearman correlation between and the SVD-based importance is only 0.38, whereas the combined score correlates with it at 0.71, indicating that the flow descriptor adds information that SVD/gradient scores do not capture, while the combined score remains consistent with them. The values presented in Table 14 are aggregated by layer type, whereas the reported Spearman correlations were calculated using individual layer-level observations. Therefore, the correlations cannot be directly reproduced from the five aggregated rows. The complete layer-level sensitivity scores underlying Table 14 and the corresponding correlation calculations are available in the project repository [44].
Table 14.
Sensitivity scores of FLowRA and AdaLoRA on LLaMA-7B with a 512-sample MNLI calibration set.
4.4. Robustness to Hyperparameters and Ultra-Low Budgets
FLowRA remains stable over a wide LR band and peaks near ; performance degrades gracefully at overly large LR. In the evaluated settings, FLowRA achieves its highest scores at a learning rate of . Performance decreases at higher learning rates (see Table 15).
Table 15.
Learning rate sensitivity.
The GLUE average in this learning-rate sensitivity experiment is computed over the six development tasks used in Table 13, rather than the nine-task aggregate reported in Table 4.
FLowRA also benefits consistently from additional budget but already dominates at ultra-low budgets, where uniform-rank methods waste capacity. Table 16 shows that with only 0.05% trainable parameters, FLowRA reaches 81.3 on GLUE, compared with 72.1 for LoRA and 76.2 for DoRA, reducing the gap to full fine-tuning from 17.0 points for LoRA to 7.8 points for FLowRA.
Table 16.
Adaptation with 0.05% trainable parameters.
For AdaLoRA, the trainable-parameter budget was capped at 0.05% by setting the initialization rank to 3 and the target rank to 2, with adaptation restricted to the query, key, value, and output attention projections. For LLaMA-2-7B on MNLI, the initial configuration contains 3,158,400 trainable parameters, including the classification head, out of 6,607,355,904 base-model parameters. This corresponds to approximately 0.0478%, satisfying the specified 0.05% budget ceiling.
4.5. Efficiency
FLowRA introduces a one-time calibration stage to estimate flow statistics before adapter training. On LLaMA-7B (4×A100 80 GB), calibration requires 5.1 h and adapter training takes 33.4 h, resulting in a total runtime of 38.5 h (Table 17). This is approximately 6.9% longer than standard LoRA (36.0 h), but shorter than AdaLoRA (41.0 h, including calibration) and full fine-tuning (42.3 h).
Table 17.
Training and end-to-end runtime efficiency on the MNLI benchmark using LLaMA-7B.
Despite this additional computational overhead, FLowRA provides a favorable trade-off between adaptation quality and runtime. Separately, the NLU evaluations in Table 4 show that, at comparable trainable-parameter budgets (0.87% for FLowRA and 0.84% for LoRA), FLowRA improves GLUE performance from 84.7 to 87.4 and SuperGLUE performance from 78.9 to 82.7. In the LLaMA-7B efficiency evaluation (Table 17), its adapter-training stage is shorter than that of LoRA (33.4 vs. 36.0 h), and it reaches convergence in 9 K steps compared with 12 K for LoRA. The runtime measurements in Table 17 were obtained on the MNLI benchmark using LLaMA-7B.
The calibration stage accounts for approximately 13% of the total runtime on LLaMA-7B. Thus, FLowRA prioritizes adaptation quality and parameter efficiency rather than minimizing the runtime of a single fine-tuning run. Inference latency and throughput remain within approximately 1% of the full-model baseline (Table 18). Inference efficiency was evaluated on the Alpaca benchmark using LLaMA-7B. The throughput values in Table 18 represent aggregate token throughput across concurrent inference requests rather than single-request generation speed.
Table 18.
Inference latency, aggregate token throughput, and GPU memory consumption of LLaMA-7B on the Alpaca benchmark.
Figure 2 summarizes these trade-offs against DoRA, LoRA, and AdaLoRA. Panel (a) gives a normalized comparison across GLUE, SuperGLUE, multi-task and few-shot performance, gap to full FT, hyperparameter simplicity, training speed, and parameter efficiency; panel (b) shows the normalized efficiency profile in terms of parameter and memory efficiency, training and convergence speed, hyperparameter simplicity, and quality retention; and panel (c) shows quality as a percentage of full FT on GLUE, SuperGLUE, ImageNet, HumanEval, and GSM8K.
Figure 2.
Comparison of FLowRA with DoRA, LoRA, and AdaLoRA: (a) overall; (b) efficiency; (c) quality relative to full FT.
4.6. Case Studies
We analyze cross-architecture transfer from ImageNet-pre-trained ViT models to fine-grained bird classification on CUB-200. Without any manual per-architecture tuning, FLowRA’s polymorphic adapters match a specialized hand-tuned adapter baseline (84.6), while LoRA lags notably (80.2), and adapters reach 82.9. This supports the claim that flow profiles are sufficient to select the appropriate adaptation type across heterogeneous components (see Table 19).
Table 19.
Cross-architecture transfer.
We next consider catastrophic forgetting under sequential adaptation to four GLUE tasks, measuring forgetting as the average performance drop across tasks. Sequential FLowRA tuning reduces forgetting to 1.7% compared to 5.4% for sequential LoRA and 8.2% for sequential full fine-tuning, and FLowRA with orthogonal composition further reduces forgetting to 0.5% while keeping the same average score as full fine-tuning (86.9), consistent with the allocation encouraging distinct subspaces per task (see Table 20).
Table 20.
Sequential learning on four GLUE tasks using LLaMA-7B. Forget. = Forgetting.
4.7. Multi-Task Composition
We evaluate simultaneous adaptation to eight GLUE tasks, excluding WNLI, using adapter merging, with 12.5% of the parameters per task for all merging methods. Naive merging degrades substantially due to subspace overlap, and task arithmetic and TIES-Merging progressively reduce interference, i.e., the degradation caused by negative task interactions. FLowRA composition achieves the best performance among merging methods (87.9 versus 88.0 for independent tuning) and the lowest measured interference, with a maximum per-task drop of 1.7 points compared with 5.1 for TIES-Merging and 19.1 for naive merging (see Table 21). At the task level, it retains 98.1–99.2% of the independently tuned scores on all tasks except CoLA, where it even exceeds independent tuning (see Table 22). We attribute the improvement to flow-guided allocation that concentrates each adapter’s capacity on task-sensitive modules, naturally reducing cross-task parameter overlap.
Table 21.
Multi-task composition on eight GLUE tasks.
Table 22.
Task-level multi-task retention on eight GLUE tasks, excluding WNLI.
4.8. Summary of Findings
Across benchmarks, FLowRA achieves strong performance among the evaluated PEFT baselines with 0.87% trainable parameters (see Figure 2). It reaches 87.4 on GLUE and 82.7 on SuperGLUE, i.e., +2.7 and +3.8 points over LoRA and +1.6 and +2.6 points over DoRA, respectively. In vision and vision-language tasks, it stays within 0.5–0.8 points of full fine-tuning. It also converges faster and supports multi-task composition with reduced interference (a maximum per-task drop of 1.7 vs. 19.1 for naive merging) while retaining about 98–99% of per-task performance. Table 23 summarizes average performance across language understanding, vision, code, and multi-modal benchmarks: FLowRA reduces the average gap to full FT to points, compared with −2.8 for DoRA and −3.5 for LoRA.
Table 23.
Average performance by domain.
These results support the central hypothesis that task-specific changes in hidden representations provide a useful signal for allocating low-rank adaptation capacity across layers and modules.
5. Discussion
Beyond empirical gains, FLowRA offers practical value by providing an interpretable, calibration-based explanation of layer- and block-level adaptation requirements in PEFT, reducing the need for manual rank and target-module selection by exposing only two user-configurable parameters, as summarized in Table 2. The calibration stage introduces additional computational overhead, resulting in a slightly longer end-to-end runtime than standard LoRA in the evaluated setting. Nevertheless, FLowRA achieves higher downstream performance while requiring a similarly small fraction of trainable parameters. This trade-off makes the method particularly relevant when adaptation quality is prioritized over minimizing the runtime of a single fine-tuning run. More broadly, FLowRA demonstrates that lightweight calibration-based profiling can guide rank allocation more effectively than uniform heuristics, without requiring expensive grid search.
The method also has several limitations. Its reliability depends on the quality of the calibration data: if the calibration set (512 samples by default) does not represent the target task, the flow descriptor becomes unreliable, and performance degrades noticeably when calibration size drops below 256 samples (see the project repository [44]). This is most problematic for highly specialized domains, such as low-resource languages or rare medical sub-specialties, where even 512 calibration samples may be insufficient. Moreover, the flow descriptor does not always add information. On tasks where SVD-based importance already identifies the critical modules (e.g., SST-2, where accuracy exceeds 95%), the flow descriptor may provide only limited additional benefit. Conversely, some layers exhibit high representation change because they perform generic computations that are not task-specific, and without gradient filtering, such layers may receive inflated rank budgets. The criticality score partially mitigates this by weighting flow by gradient activity, but the multiplicative form is heuristic and is ablated in our public repository [44].
There are also practical and conceptual constraints. Fine-grained block allocation can cause memory fragmentation, so our implementation uses module-level LoRA by default, and true parameter savings require architecture-specific engineering. Because the descriptor is computed on a frozen base model, a heavily fine-tuned checkpoint (e.g., an instruction-tuned model) may have a shifted representation geometry that reduces the descriptor’s utility. Finally, FLowRA uses the calibration-based flow descriptor to determine the initial rank allocation. The assigned ranks can subsequently be refined during adapter training, as evaluated in the ablation study in Section 4.3. This distinguishes the initial flow-guided allocation from the training-stage rank refinement mechanism.
6. Conclusions
FLowRA shows that a simple, calibration-phase flow-change descriptor can meaningfully improve low-rank allocation, complementing gradient- and SVD-based importance. By automating rank allocation and reducing reliance on grid search, FLowRA lowers the barrier to efficient fine-tuning across architectures. Open questions include the behavior of the descriptor on instruction-tuned checkpoints, its compatibility with quantized and state-space models, and the efficiency and robustness of dynamic rank refinement during training.
Author Contributions
Conceptualization, A.S. (Aleksei Samarin); methodology, A.S. (Aleksei Samarin), A.N. and A.T.; software, A.N., A.T., D.N. and A.S. (Alexander Savelev); validation, A.S. (Alexander Savelev), A.D., E.M. and V.M.; formal analysis, E.M., E.K., V.M. and J.A.; investigation, A.S. (Aleksei Samarin), A.N., D.N., A.S. (Alexander Savelev), A.D. and J.A.; resources, A.S. (Aleksei Samarin), A.N. and A.M.; data curation, E.M., E.K., V.M. and A.M.; writing—original draft preparation, A.N., D.N., E.K. and J.A.; writing—review and editing, A.S. (Aleksei Samarin) and E.K.; visualization, A.T. and E.K.; supervision, A.S. (Aleksei Samarin); project administration, A.S. (Aleksei Samarin) and E.K. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by a grant, provided by the Ministry of Economic Development of the Russian Federation (agreement dated 20 June 2025 No. 139-15-2025-011, identifier 000000C313925P4G0002).
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
The implementation, configuration files, training scripts, detailed per-architecture algorithms, and extended experimental results are available in the project repository at https://github.com/user321459/flowra [44] (accessed on 15 October 2026). All benchmarks used in this study are publicly available.
Conflicts of Interest
The authors declare no conflicts of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| CNN | Convolutional neural network |
| FFN | Feed-forward network |
| FLowRA | Flow-guided low-rank adaptation |
| FT | Fine-tuning |
| LLM | Large language model |
| LoRA | Low-rank adaptation |
| LR | Learning rate |
| NLG | Natural language generation |
| NLU | Natural language understanding |
| PEFT | Parameter-efficient fine-tuning |
| QA | Question answering |
| RNN | Recurrent neural network |
| SVD | Singular value decomposition |
| ViT | Vision Transformer |
| VQA | Visual question answering |
References
- Bernstein, L.; Sludds, A.; Hamerly, R.; Sze, V.; Emer, J.; Englund, D. Freely scalable and reconfigurable optical hardware for deep learning. Sci. Rep. 2021, 11, 3144. [Google Scholar] [CrossRef] [Scilit]
- Lee, J.; Mukhanov, L.; Molahosseini, A.S.; Minhas, U.; Hua, Y.; Martinez del Rincon, J.; Dichev, K.; Hong, C.H.; Vandierendonck, H. Resource-Efficient Convolutional Networks: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques. ACM Comput. Surv. 2023, 55, 276. [Google Scholar] [CrossRef] [Scilit]
- OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F.; Almeida, D.; Altenschmidt, J.; Altman, S.; et al. GPT-4 Technical Report. arXiv 2023. [Google Scholar] [CrossRef] [Scilit]
- Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. The llama 3 herd of models. arXiv 2024, arXiv:2407.21783. [Google Scholar]
- Kindratenko, V.; Wilhelmson, R.; Brunner, R.; Martinez, T.; Hwu, W.-m. High-Performance Computing with Accelerators. Comput. Sci. Eng. 2010, 12, 12–16. [Google Scholar] [CrossRef] [Scilit]
- Baresi, L.; Quattrocchi, G. Training and Serving Machine Learning Models at Scale. In Service-Oriented Computing; Springer: Cham, Germany, 2022; pp. 669–683. [Google Scholar] [CrossRef] [Scilit]
- Cai, H.; Lin, J.; Lin, Y.; Liu, Z.; Tang, H.; Wang, H.; Zhu, L.; Han, S. Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications. ACM Trans. Des. Autom. Electron. Syst. 2022, 27, 20. [Google Scholar] [CrossRef] [Scilit]
- Peccia, F.N.; Bringmann, O. Embedded Distributed Inference of Deep Neural Networks: A Systematic Review. arXiv 2024, arXiv:2405.03360. [Google Scholar]
- Kallimani, R.; Pai, K.; Raghuwanshi, P.; Iyer, S.; Alcaraz López, O. TinyML: Tools, Applications, Challenges, and Future Research Directions. Multimed. Tools Appl. 2023, 83, 29015–29045. [Google Scholar] [CrossRef] [Scilit]
- Sanh, V.; Debut, L.; Chaumond, J.; Wolf, T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv 2019. [Google Scholar] [CrossRef] [Scilit]
- Kurtic, E.; Campos, D.; Nguyen, T.; Frantar, E.; Kurtz, M.; Fineran, B.; Goin, M.; Alistarh, D. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 4163–4181. [Google Scholar] [CrossRef] [Scilit]
- Patterson, D.; Gonzalez, J.; Hölzle, U.; Le, Q.; Liang, C.; Munguía, L.M.; Rothchild, D.; So, D.; Texier, M.; Dean, J. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink. TechRxiv 2022. [Google Scholar] [CrossRef]
- Wu, C.J.; Raghavendra, R.; Gupta, U.; Acun, B.; Ardalani, N.; Maeng, K.; Chang, G.; Aga, F.; Huang, J.; Bai, C.; et al. Sustainable AI: Environmental Implications, Challenges and Opportunities. arXiv 2021. [Google Scholar] [CrossRef] [Scilit]
- Shoukourian, H.; Wilde, T.; Labrenz, D.; Bode, A. Using Machine Learning for Data Center Cooling Infrastructure Efficiency Prediction. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW); IEEE: New York, NY, USA, 2017; pp. 954–963. [Google Scholar] [CrossRef] [Scilit]
- Osondu, J. Red AI vs. Green AI in Education: How Educational Institutions and Students Can Lead Environmentally Sustainable Artificial Intelligence Practices. Res. Prepr. 2025. [Google Scholar] [CrossRef]
- Li, B.; Samsi, S.; Gadepally, V.; Tiwari, D. Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis; Association for Computing Machinery: New York, NY, USA, 2023; pp. 1–15. [Google Scholar] [CrossRef] [Scilit]
- Gholami, A.; Kim, S.; Zhen, D.; Yao, Z.; Mahoney, M.; Keutzer, K. A Survey of Quantization Methods for Efficient Neural Network Inference. In Low-Power Computer Vision; Chapman and Hall/CRC: Boca Raton, FL, USA, 2022; pp. 291–326. [Google Scholar] [CrossRef] [Scilit]
- Liu, Z.; Wang, Y.; Han, K.; Ma, S.; Gao, W. Post-Training Quantization for Vision Transformer. arXiv 2021, arXiv:2106.14156. [Google Scholar]
- Lin, Y.; Zhang, T.; Sun, P.; Li, Z.; Zhou, S. FQ-ViT: Fully Quantized Vision Transformer without Retraining. arXiv 2021. [Google Scholar] [CrossRef] [Scilit]
- Xiao, G.; Lin, J.; Seznec, M.; Demouth, J.; Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv 2022. [Google Scholar] [CrossRef] [Scilit]
- Sainath, T.; Kingsbury, B.; Sindhwani, V.; Arisoy, E.; Ramabhadran, B. Low-rank matrix factorization for Deep Neural Network 698 training with high-dimensional output targets. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing; IEEE: New York, NY, USA, 2013; pp. 6655–6659. [Google Scholar] [CrossRef] [Scilit]
- Hu, E.J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. arXiv 2021, arXiv:2106.09685. [Google Scholar]
- Wang, P.; Cheng, Y.; Peng, Q.; Dong, B.; Li, S. Low-Bitwidth Convolutional Neural Networks for Wireless Interference Identification. IEEE Trans. Cogn. Commun. Netw. 2022, 8, 557–569. [Google Scholar] [CrossRef] [Scilit]
- Simons, T.; Lee, D.J. A review of binarized neural networks. Electronics 2019, 8, 661. [Google Scholar] [CrossRef] [Scilit]
- Dettmers, T.; Zettlemoyer, L. The case for 4-bit precision: k-bit Inference Scaling Laws. arXiv 2022. [Google Scholar] [CrossRef] [Scilit]
- Reuther, A.; Michaleas, P.; Jones, M.; Gadepally, V.; Samsi, S.; Kepner, J. AI Accelerator Survey and Trends. In 2021 IEEE High Performance Extreme Computing Conference (HPEC); IEEE: New York, NY, USA, 2021; pp. 1–9. [Google Scholar] [CrossRef] [Scilit]
- Burhanuddin, M. Efficient Hardware Acceleration Techniques for Deep Learning on Edge Devices: A Comprehensive Performance Analysis. KHWARIZMIA 2023, 2023, 103–112. [Google Scholar] [CrossRef] [Scilit]
- Tuli, S.; Jha, N. AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers. arXiv 2023. [Google Scholar] [CrossRef] [Scilit]
- Chen, H.; Garner, P.N. A Bayesian Interpretation of Adaptive Low-Rank Adaptation. arXiv 2025, arXiv:2409.10673. [Google Scholar]
- Brown, T.B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20; Curran Associates Inc.: Red Hook, NY, USA, 2020. [Google Scholar]
- Hayou, S.; Ghosh, N.; Yu, B. LoRA+: Efficient Low Rank Adaptation of Large Models. arXiv 2024, arXiv:2402.12354. [Google Scholar]
- Liu, S.Y.; Wang, C.Y.; Yin, H.; Molchanov, P.; Wang, Y.C.F.; Cheng, K.T.; Chen, M.H. DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv 2024, arXiv:2402.09353. [Google Scholar]
- Dettmers, T.; Pagnoni, A.; Holtzman, A.; Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv 2023, arXiv:2305.14314. [Google Scholar]
- Zhang, Q.; Chen, M.; Bukharin, A.; Karampatziakis, N.; He, P.; Cheng, Y.; Chen, W.; Zhao, T. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv 2023, arXiv:2303.10512. [Google Scholar]
- Kopiczko, D.J.; Blankevoort, T.; Asano, Y.M. VeRA: Vector-based Random Matrix Adaptation. arXiv 2024, arXiv:2310.11454. [Google Scholar]
- Li, Y.; Yu, Y.; Liang, C.; He, P.; Karampatziakis, N.; Chen, W.; Zhao, T. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models. arXiv 2023, arXiv:2310.08659. [Google Scholar]
- Wang, L.; Chen, S.; Jiang, L.; Pan, S.; Cai, R.; Yang, S.; Yang, F. Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies. arXiv 2025, arXiv:2410.19878. [Google Scholar]
- Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; de Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; Gelly, S. Parameter-Efficient Transfer Learning for NLP. arXiv 2019, arXiv:1902.00751. [Google Scholar]
- Xu, Y.; Xie, L.; Gu, X.; Chen, X.; Chang, H.; Zhang, H.; Chen, Z.; Zhang, X.; Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arXiv 2023, arXiv:2309.14717. [Google Scholar]
- Zhang, L.; Zhang, L.; Shi, S.; Chu, X.; Li, B. LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. arXiv 2023, arXiv:2308.03303. [Google Scholar]
- Liu, H.; Tam, D.; Muqeeth, M.; Mohta, J.; Huang, T.; Bansal, M.; Raffel, C. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22; Curran Associates Inc.: Red Hook, NY, USA, 2022. [Google Scholar]
- He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; Neubig, G. Towards a Unified View of Parameter-Efficient Transfer Learning. arXiv 2022, arXiv:2110.04366. [Google Scholar]
- Tang, A.; Shen, L.; Luo, Y.; Yin, N.; Zhang, L.; Tao, D. Merging multi-task models via weight-ensembling mixture of experts. In Proceedings of the 41st International Conference on Machine Learning, ICML’24; JMLR.org: Norfolk, MA, USA, 2024. [Google Scholar]
- FLowRA: Flow-based Low-Rank Adaptation Framework. Available online: https://github.com/user321459/flowra (accessed on 15 October 2026).
- Meng, F.; Wang, Z.; Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. Adv. Neural Inf. Process. Syst. 2024, 37, 121038–121072. [Google Scholar] [CrossRef] [Scilit]
- Büyükakyüz, K. Olora: Orthonormal low-rank adaptation of large language models. arXiv 2024, arXiv:2406.01775. [Google Scholar]
- Li, X.L.; Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers); Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 4582–4597. [Google Scholar]
- Lester, B.; Al-Rfou, R.; Constant, N. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic; Moens, M.F., Huang, X., Specia, L., Yih, S.W.t., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 3045–3059. [Google Scholar] [CrossRef] [Scilit]
- Ilharco, G.; Ribeiro, M.T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; Farhadi, A. Editing models with task arithmetic. arXiv 2022, arXiv:2212.04089. [Google Scholar]
- Yadav, P.; Tam, D.; Choshen, L.; Raffel, C.; Bansal, M. TIES-Merging: Resolving Interference When Merging Models. In Proceedings of the Advances in Neural Information Processing Systems; Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2023; Volume 36, pp. 7093–7115. [Google Scholar] [CrossRef] [Scilit]
- Huang, C.; Liu, Q.; Lin, B.Y.; Pang, T.; Du, C.; Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv 2023, arXiv:2307.13269. [Google Scholar]
- Pfeiffer, J.; Kamath, A.; Rücklé, A.; Cho, K.; Gurevych, I. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online; Merlo, P., Tiedemann, J., Tsarfaty, R., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 487–503. [Google Scholar] [CrossRef] [Scilit]
- Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; Bowman, S.R. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP; Association for Computational Linguistics: Stroudsburg, PA, USA, 2018; pp. 353–355. [Google Scholar] [CrossRef] [Scilit]
- Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32; Curran Associates, Inc.: Red Hook, NY, USA, 2019. [Google Scholar]
- Rajpurkar, P.; Zhang, J.; Lopyrev, K.; Liang, P. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2016; pp. 2383–2392. [Google Scholar]
- Rajpurkar, P.; Jia, R.; Liang, P. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers); Association for Computational Linguistics: Stroudsburg, PA, USA, 2018; pp. 784–789. [Google Scholar]
- Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. Natural questions: A benchmark for question answering research. Trans. Assoc. Comput. Linguist. 2019, 7, 453–466. [Google Scholar] [CrossRef] [Scilit]
- Joshi, M.; Choi, E.; Weld, D.S.; Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Association for Computational Linguistics: Stroudsburg, PA, USA, 2017; pp. 1601–1611. [Google Scholar]
- Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. Training Verifiers to Solve Math Word Problems. arXiv 2021, arXiv:2110.14168. [Google Scholar]
- Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; Steinhardt, J. Measuring Mathematical Problem Solving with the MATH Dataset. arXiv 2021, arXiv:2103.03874. [Google Scholar]
- Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H.P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. Evaluating Large Language Models Trained on Code. arXiv 2021, arXiv:2107.03374. [Google Scholar]
- Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. Program Synthesis with Large Language Models. arXiv 2021, arXiv:2108.07732. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition; IEEE: New York, NY, USA, 2009; pp. 248–255. [Google Scholar]
- Wah, C.; Branson, S.; Welinder, P.; Perona, P.; Belongie, S. The Caltech-UCSD Birds-200-2011 Dataset; Technical Report CNS-TR-2011-001; California Institute of Technology: Pasadena, CA, USA, 2011. [Google Scholar]
- Zhou, B.; Zhao, H.; Puig, X.; Xiao, T.; Fidler, S.; Barriuso, A.; Torralba, A. Semantic Understanding of Scenes through the ADE20K Dataset. arXiv 2018, arXiv:1608.05442. [Google Scholar]
- Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; Parikh, D. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. arXiv 2017, arXiv:1612.00837. [Google Scholar]
- Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. Llama 2: Open foundation and fine-tuned chat models. arXiv 2023, arXiv:2307.09288. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv 2021, arXiv:2010.11929. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: New York, NY, USA, 2016; pp. 770–778. [Google Scholar] [CrossRef] [Scilit]
- Li, J.; Li, D.; Savarese, S.; Hoi, S. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv 2023, arXiv:2301.12597. [Google Scholar]
- Zaken, E.B.; Goldberg, Y.; Ravfogel, S. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers); Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 1–9. [Google Scholar]
- Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; Hashimoto, T.B. Stanford Alpaca: An Instruction-Following LLaMA Model. 2023. Available online: https://github.com/tatsu-lab/stanford_alpaca (accessed on 15 October 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.

