TARA: Task-Adaptive Rank Allocation for Efficient Large Language Model Fine-Tuning in Geo-Information Text Classification

Abstract

Geo-information texts, including geospatial data-use regulations and Earth observation metadata, are central to data governance and compliance auditing in remote sensing ecosystems. Full fine-tuning of large pre-trained language models is often computationally impractical, while standard LoRA reduces cost but assigns a fixed rank to all adapted modules, ignoring differences across layers and projection types. This paper proposes TARA, a task-adaptive rank allocation method for LoRA-based fine-tuning. TARA assigns learnable importance scores to rank dimensions and uses Gumbel–Sigmoid sampling with the Straight-Through Estimator to learn discrete rank masks under a global sparsity constraint. We further construct RSRegulation, a geospatial regulatory compliance benchmark containing 4032 English-language samples derived from 168 clauses across seven regulatory and policy sources with clause-level data isolation. Across five random seeds, TARA achieves 95.30 ± 0.10% accuracy and 95.44 ± 0.10% F1 with a maximum trainable adapter budget of 1.57 M parameters. The learned soft allocation corresponds to approximately 0.38 M effective adapter parameters and 75.8% soft rank compression. Physical hard pruning reduces the deployed adapter to 0.086 M parameters while retaining 95.12 ± 0.11% accuracy and 95.21 ± 0.10% F1. Layer-wise analysis shows that value projections retain higher ranks than query projections under the current task and backbone, revealing a task-dependent non-uniform allocation pattern.

1. Introduction

1.1. Geo-Information Text Processing

The rapid expansion of geospatial data infrastructures has produced large volumes of geo-information texts, including regulatory policies on data access and usage, metadata descriptions in Earth observation catalogs, and domain-specific technical literature. Efficient processing of these texts is fundamental to geographic information management, compliance auditing for geospatial data platforms, metadata retrieval, and knowledge organization in GIS ecosystems [1,2,3,4,5]. Compared with general-domain text, geo-information texts contain more domain-specific terminology, more complex rule-constraint relationships, and stronger domain dependency, which place greater demands on the semantic understanding capabilities of language models. Regulatory compliance in geospatial and remote sensing institutions often depends on numerical thresholds, spatial scope, technical standards, data-access conditions, and policy hierarchies. Language models acquire contextual semantic representations through large-scale pre-training, allowing the meaning of a term to be interpreted from its surrounding text. In geo-information systems, controlled vocabularies, institutional registries, and knowledge graphs can complement these representations by standardizing terminology, resolving entity references, and linking textual descriptions to formally defined geographic concepts. The present study was motivated by such a real-world research project involving geo-information data governance and compliance analysis.

1.2. Large Language Models and Parameter-Efficient Fine-Tuning

Recent advances in pre-trained language models and large language models, including BERT, RoBERTa, GPT-3, LLaMA, and instruction-tuned models, have created new opportunities for intelligent text processing [6,7,8,9,10]. In remote sensing, vision–language and multimodal language models have further expanded the role of language modeling in Earth observation interpretation [11,12,13,14,15,16,17,18,19,20]. However, full-parameter fine-tuning of these models requires substantial computational and storage resources, making it unsuitable for many practical deployment scenarios. Parameter-efficient fine-tuning methods, referred to as PEFT methods, address this limitation by freezing the pre-trained model and updating only a small set of additional parameters. Among PEFT methods, low-rank adaptation or LoRA [21] has been widely adopted because of its structural simplicity and zero additional inference latency. It decomposes weight updates into the product of two low-rank matrices.

1.3. Limitations of Fixed-Rank LoRA

Standard LoRA assigns a uniform rank to all target layers, implicitly assuming that adaptation requirements remain identical across layer depths and projection roles. Transformer layers are functionally heterogeneous: shallow layers encode general lexical features, middle layers perform semantic transformation, and deep layers integrate task-relevant representations. For geo-information regulatory texts, which require understanding of professional terminology and rule-based reasoning, a fixed-rank setting may allocate redundant parameters to low-demand layers while limiting expressiveness in critical layers.

1.4. Contributions

This paper proposes TARA, an end-to-end differentiable adaptive rank allocation method, and evaluates it in a geo-information regulatory compliance scenario derived from a real-world geospatial research project. TARA formulates the selection of individual LoRA rank dimensions as a joint optimization problem. Learnable rank scores, differentiable binary masking, a global sparsity target, and rank-dependent output scaling are integrated into one training objective to learn layer- and projection-specific adapter structures. To support evaluation in this domain, we construct RSRegulation, a geospatial regulatory compliance classification dataset derived from seven national and international regulatory sources and designed around practical compliance-analysis requirements. The dataset comprises 4032 English-language samples across 168 clauses and uses clause-level data isolation. Experiments on RSRegulation provide empirical evidence of the parameter efficiency and deployability of TARA and reveal task-dependent non-uniform rank allocation patterns under the current geo-information task and backbone.

2. Related Work

2.1. NLP in Remote Sensing

Research at the intersection of remote sensing and NLP has primarily focused on cross-modal understanding and textual information analysis. Remote sensing image captioning and visual question answering established early benchmarks in this area [11,12,13]. More recent multimodal models, including GeoChat, RSGPT, SkyEyeGPT, RS-LLaVA, and related remote sensing vision-language benchmarks, have achieved notable progress in instruction tuning, image captioning, visual question answering, and grounded dialogue for geospatial data [14,15,16,17,18,19,20]. In textual information analysis, automated processing of remote sensing metadata and geospatial knowledge graph construction integrate textual and spatial information to support data discovery and knowledge organization. Recent studies have further explored LLM-assisted disaster geospatial information extraction, natural-language interaction with GIS workflows, fine-tuned models for spatial-query generation, reasoning over heterogeneous geospatial schemas, and geospatial knowledge-based question answering [22,23,24,25,26]. These developments demonstrate the expanding role of LLMs in geo-information processing, while efficient task-specific adaptation and adaptive rank allocation for geo-information text tasks remain largely unexplored.

2.2. Parameter-Efficient Fine-Tuning

PEFT methods can be broadly divided into three categories. Addition-based methods, such as Adapter [27] and AdapterFusion [28], insert small trainable modules into the model, which introduces additional inference latency [29]. Prompt-based methods, including prefix-tuning, prompt tuning, and P-tuning v2, guide model behavior through learnable input vectors or continuous prompts [30,31,32]. Reparameterization-based methods impose structural constraints on weight updates: BitFit [33] restricts updates to bias terms, Diff Pruning [34] learns sparse weight-difference vectors, and LoRA assumes low-rank weight increments and introduces no inference overhead. TARA builds on LoRA by incorporating differentiable rank selection to address the fixed-rank limitation.

2.3. Adaptive Rank Methods

AdaLoRA addresses the fixed-rank limitation through SVD-based progressive pruning [35]. Its two-stage design, in which full-rank training precedes rank reduction, represents a key structural difference from the joint optimization approach of TARA. DyLoRA randomly samples rank values during training, providing robustness across rank budgets without explicitly modeling layer-wise differences [36]. QLoRA combines LoRA with 4-bit quantization to address memory constraints, which are orthogonal to rank allocation [37]. DoRA further improves LoRA by decomposing pre-trained weights into magnitude and direction components [38]. TARA differs from these methods in its optimization target. It learns binary decisions for individual LoRA rank dimensions under a shared sparsity objective, producing layer- and projection-specific adapters that can be physically extracted after training. DyLoRA trains one adapter to operate at several sampled ranks, while AdaLoRA reallocates a predefined budget through importance-based rank reduction.

2.4. Differentiable Structure Search

The differentiable discrete selection mechanism of TARA draws on ideas from neural architecture search and differentiable sampling. DARTS pioneered continuous relaxation for discrete architectural choices, enabling gradient-based search [39]. Gumbel–Softmax and the Concrete distribution provide principled relaxations for discrete random variables [40,41]. The Straight-Through Estimator enables discrete sampling in the forward pass with approximate gradient propagation in the backward pass [42]. Movement Pruning and CoFi apply end-to-end differentiable selection to pre-trained-model weights or larger structural units [43,44]. TARA instead restricts the search space to LoRA rank dimensions, preserving the frozen backbone and producing a compact adapter rather than a pruned base model.

3. Methodology

This section presents TARA from the overall formulation to the training objective. Section 3.1 reviews LoRA and defines the adaptive rank allocation problem. Section 3.2 describes how TARA learns rank masks through differentiable sampling. Section 3.3 explains how the masked LoRA update is computed with adaptive scaling. Section 3.4 introduces the regularized training objective, and Section 3.5 summarizes the training and inference procedure. Figure 1 gives an overview of the complete framework, and Figure 2 illustrates the mechanism inside a single LoRA-adapted projection.

Figure 1. Overview of the TARA framework. TARA learns rank-wise importance scores for LoRA adapters, uses Gumbel–Sigmoid sampling with the Straight-Through Estimator to obtain rank masks, and jointly optimizes classification and compression objectives. After training, deterministic hard masks are used to extract the retained LoRA dimensions and construct a compact inference adapter.

Figure 2. TARA mechanism within a single LoRA-adapted projection. Learnable rank scores are converted into soft and hard masks, which determine the effective rank and the masked LoRA update.

3.1. Preliminaries and Problem Formulation

LoRA parameterizes the weight update of a frozen layer as the product of two low-rank matrices, Δ W = BA, where B and A have rank r. The adapted forward pass is

h = W₀x + (α/r)·BAx

(1)

where α is a fixed scaling factor, A is Gaussian-initialized, and B is zero-initialized to ensure training begins from the pre-trained state.

Standard LoRA assigns the same rank r to all adapted layers, treating each as having equal adaptation requirements. In practice, different layers and projection types require different levels of adaptation. TARA addresses this by learning a per-layer effective rank from the training data. Given a maximum rank r_max and a target compression ratio ρ, the goal is to determine which rank dimensions are task-relevant in each layer and suppress the rest. Because rank is an integer, this cannot be solved directly by gradient descent; TARA introduces a differentiable relaxation that enables end-to-end joint optimization of rank selection and task performance.

3.2. Differentiable Rank Selection

For each adapted layer, TARA assigns a learnable scalar importance score to each of the r_max rank dimensions. These scores are trained jointly with the adapter matrices. To encourage discrete decisions during training, Gumbel noise is injected into the scores before applying a temperature-scaled Sigmoid:

$$\tilde{m}=\sigma \left(\right(s+G)/\tau )$$

(2)

where G = −log(−log(U)) is Gumbel noise with U ∼ Uniform(0, 1). Temperature τ is annealed from 2.0 to 0.1 over training. Each element of $\tilde{m}$ lies between 0 and 1, with values near the extremes indicating clear selection decisions.

In the forward pass, each mask value is compared to a threshold of 0.5: dimensions above the threshold are retained, and the rest are zeroed out. In the backward pass, gradients flow through the soft mask $\tilde{\mathrm{m}}$ rather than the binary decisions, following the Straight-Through Estimator. This forward–backward asymmetry allows discrete rank selection to be trained end to end without post hoc pruning.

3.3. Adaptive Output Scaling

Suppressing rank dimensions reduces the LoRA output magnitude. To maintain a consistent output scale during training, TARA replaces the fixed scaling factor α/r with an adaptive one. Let r_eff be the number of retained dimensions in a given layer. The adaptive scaling factor is set to α/r_eff, which increases as more dimensions are suppressed. Only the retained dimensions of B and A contribute to the forward computation, and the output is scaled by α/r_eff in place of α/r in Equation (1).

3.4. Sparsity Regularization

Without an explicit compression signal, the importance scores would not converge to sparse solutions. TARA introduces two regularization terms. The sparsity loss penalizes deviation of the mean mask activation from the target ratio:

$$L\text{\_}\mathit{spar}={(\tilde{m}-\rho )}^2$$

(3)

where $\stackrel{-}{m}$ is the mean of all soft mask values across layers and rank dimensions; ρ is the target compression ratio.

The entropy loss penalizes uncertain intermediate mask values and promotes binary decisions:

$$L\text{\_}\mathit{ent}=(1/r\text{\_}\mathit{max})·\Sigma \tilde{m}(1-\tilde{m})$$

(4)

The combined regularization loss is L_reg = L_spar + λ·L_ent, with λ = 0.1.

3.5. Training Objective and Procedure

The total training loss is

L = L_task + β·L_reg

(5)

where L_task is cross-entropy classification loss; β = 1.0.

At each training step, Gumbel noise is sampled, and soft masks are computed via Equation (2). Dimensions are selected by thresholding at 0.5, and predictions are produced using only the retained dimensions scaled by the adaptive factor. The loss in Equation (5) is then backpropagated to update the adapter matrices and importance scores jointly.

After training, Gumbel noise is removed, and deterministic importance values are computed directly from the learned scores. Each dimension with a sigmoid score above 0.5 is retained, while the remaining dimensions are discarded. The retained dimensions are physically extracted from the corresponding LoRA matrices, reducing the per-layer adapter parameters from r_max·(d_in + d_out) to r_eff·(d_in + d_out). For modules with r_eff = 0, the LoRA update is disabled, and the frozen base projection is used directly. This produces a compact inference-time adapter rather than an unchanged full-rank matrix with zero-valued masks. TARA adds only one importance score per rank dimension per layer, which is negligible relative to the adapter matrices.

4. Experiments

4.1. Experimental Setup

4.1.1. Datasets

RSRegulation. We construct the Remote Sensing Regulation Compliance dataset (RSRegulation) in response to a practical regulatory-compliance requirement identified in a real-world research project conducted by geospatial and remote sensing institutions. The dataset is derived from seven authoritative regulatory sources that span the Chinese and international geospatial governance hierarchy: the Surveying and Mapping Law of the People’s Republic of China (2017 revision); the Regulation on Map Management (State Council Decree No. 664, 2015); the Regulation on Public Use of Remote Sensing Imagery; the Standard for Underwater Topographic Survey; the Technical Specification for Water Resources Survey and Assessment (SL 196); the NASA Open Data Policy; and the ESA/Copernicus Data Access Policy. This selection covers national legislation, State Council regulations, ministerial rules, technical standards, and international Earth observation data policies. The benchmark is intended to represent a controlled version of practical geo-information compliance screening rather than the full complexity of operational legal auditing.

A total of 168 regulatory clauses were extracted from these sources. Sample generation followed a three-stage pipeline. First, GPT-4 (temperature = 0.3) generated candidate compliant and non-compliant scenario descriptions in English. Negative examples were constrained to three mutually exclusive error categories: numerical or threshold errors, in which key quantitative values are replaced with incorrect magnitudes (n = 672); condition or scope inversions, in which the applicability range is incorrectly expanded or restricted (n = 672); and standard substitutions, in which a correct regulatory reference is replaced with a superseded or inapplicable alternative (n = 672). All samples then underwent expert review by annotators with backgrounds in geospatial data management. Clause-level isolation was applied during splitting: all samples derived from the same clause were assigned exclusively to the same split, ensuring that no clause-level semantic pattern, generation template, or lexical cue was shared across the training and evaluation sets. The task is binary classification over English-language scenario descriptions (105 to 203 characters; mean tokenized length of 34 tokens under RoBERTa-large tokenization). Dataset statistics are given in Table 1.

Table 1. Statistics of the RSRegulation dataset.

Dataset	Task	Train	Val	Test	Total	Clauses	Pos:Neg
RSRegulation	Compliance classif.	2688	672	672	4032	168	1:1

RSRegulation negative examples are equally distributed across three error categories, with 672 samples in each category. Clause-level isolation ensures that all samples derived from the same regulatory clause are assigned to only one of the training, validation, or test splits.

4.1.2. Baselines

TARA is compared against full fine-tuning, which updates all model parameters and serves as the performance upper bound; standard LoRA [26] at three rank configurations (r = 4, 8, and 16), representing parameter-efficient adaptation across a range of budgets; and AdaLoRA [35] with initial rank 12 and target rank 8, representing an adaptive rank allocation baseline with SVD-based progressive pruning. This baseline set covers fixed-rank adaptation across several budgets and adaptive rank reduction through AdaLoRA.

4.1.3. Implementation Details

The backbone is RoBERTa-large, comprising 24 Transformer encoder layers with hidden dimension 1024 and 356.94 M total parameters. TARA targets query and value projections across all 24 layers, giving 48 adaptive modules. The hyperparameters are set as follows: r_max = 16, α = 32, dropout = 0.1, ρ = 0.5, β = 1.0, λ = 0.1, τ_start = 2.0, τ_end = 0.1. Training uses the AdamW [45] optimizer with learning rate 3 × 10⁻⁴, weight decay 0.01, warmup ratio 0.05, batch size 16, BF16 mixed precision, and 8 epochs. All experiments are run on a single NVIDIA A100-40 GB GPU.

To evaluate the training stability, the principal experiments were independently conducted using five random seeds: 42, 123, 456, 789, and 1011. Accuracy and F1 are reported as the mean ± sample standard deviation across the five runs. The dataset split, hyperparameters, and training procedure were identical for all runs. Two-sided exact McNemar tests [46] were conducted using paired predictions on the RSRegulation test set because the number of discordant prediction pairs was small. Statistical significance was assessed at α = 0.05.

4.2. Main Results

Table 2 presents the classification results on the RSRegulation test set over five random seeds. TARA achieves an average accuracy of 95.30 ± 0.10% and an average F1 score of 95.44 ± 0.10% with a maximum trainable-parameter budget of 1.57 M, corresponding to 0.44% of the full model. Through adaptive rank allocation, its soft effective parameter utilization is reduced to approximately 0.38 M, corresponding to 75.8% soft rank compression relative to the maximum rank budget of r_max = 16.

Table 2. Classification performance on the RSRegulation test set over five random seeds.

Method	#Params	Accuracy (%)	F1 (%)	Precision (%)	Recall (%)
Full Fine-tuning	356.94 M (100%)	95.82 ± 0.10	95.81 ± 0.09	95.36	96.27
LoRA-4	0.66 M (0.18%)	93.86 ± 0.21	94.02 ± 0.18	93.41	94.63
LoRA-8	1.31 M (0.37%)	94.97 ± 0.15	94.81 ± 0.14	94.22	95.41
LoRA-16	2.62 M (0.73%)	95.04 ± 0.13	94.94 ± 0.12	94.34	95.55
AdaLoRA	1.31 M (0.37%)	95.11 ± 0.17	94.90 ± 0.15	94.29	95.52
TARA (Ours)	1.57 M (0.44%)	95.30 ± 0.10	95.44 ± 0.10	94.29	96.62

Results are reported as mean ± sample standard deviation over five independent runs with random seeds 42, 123, 456, 789, and 1011. The parameter count of TARA denotes the maximum trainable adapter budget with r_max = 16 across query and value projections in 24 Transformer layers. The corresponding soft effective parameter count is approximately 0.38 M.

Compared with LoRA-16, TARA achieves a slightly higher mean accuracy and F1 while using 40% fewer trainable parameters. However, the absolute differences are small, and the two-sided exact McNemar test indicates that the difference is not statistically significant. The result should therefore be interpreted as evidence that TARA maintains comparable classification performance under a smaller and non-uniform rank allocation, rather than as evidence of statistically significant predictive superiority.

Compared with AdaLoRA, which uses 1.31 M trainable parameters at a target rank of 8 and applies SVD-based progressive rank reduction, TARA achieves a slightly higher mean accuracy and F1 with a modestly larger training budget. The exact McNemar test does not identify a statistically significant difference between the two models. Accordingly, these results support the stability and parameter–performance competitiveness of TARA.

4.3. Physical Hard-Pruning Evaluation

To reconcile the soft and hard rank regimes, we further instantiated the thresholded rank configuration as a physically compact adapter. After training, Gumbel noise was removed, and deterministic importance values were computed from the learned rank scores. Dimensions with sigmoid scores above 0.5 were physically retained in the LoRA matrices, whereas the remaining dimensions were removed rather than merely masked to zero. For modules with no retained dimensions, the LoRA update was disabled, and the frozen base projection was used directly.

As shown in Table 3, physical hard pruning reduced the deployed adapter from 1.57 M to 0.086 M parameters, corresponding to a 94.5% reduction relative to the unpruned TARA adapter. Across five random seeds, the hard-pruned model achieved an average accuracy of 95.12 ± 0.11% and an average F1 score of 95.21 ± 0.10%. Relative to the unpruned TARA model, the decreases were only 0.18 and 0.23 percentage points, respectively. Under the same inference configuration, physical pruning reduced the adapter size from 6.28 MB to 0.34 MB and increased throughput from 398 to 467 samples per second. These results show that the highly sparse hard-rank configuration is operational and retains comparable predictive performance. Soft ranks reflect the continuous allocation learned during training, while hard ranks define the dimensions retained for deployment. The small post-pruning performance drop suggests that low-importance dimensions can be removed with little effect on prediction quality.

Table 3. Performance and inference efficiency before and after physical hard pruning.

Method	Parameters	Adapter Size (MB)	Peak Inference Memory (GB)	Throughput (Samples/s)	Accuracy (%)	F1 (%)
LoRA-16	2.62 M	10.48	17.8	412	95.04 ± 0.13	94.94 ± 0.12
AdaLoRA	1.31 M	5.24	17.2	435	95.11 ± 0.17	94.90 ± 0.15
TARA-Unpruned	1.57 M	6.28	18.6	398	95.30 ± 0.10	95.44 ± 0.10
TARA Hard-Pruned	0.086 M	0.34	16.9	467	95.12 ± 0.11	95.21 ± 0.10

Deployed adapter parameters denote the physically retained adapter parameters used during inference. Adapter sizes are calculated using FP32 storage at four bytes per parameter. Inference measurements were conducted under the same hardware, batch size, numerical precision, maximum sequence length, warm-up procedure, and test set. Accuracy and F1 are reported as mean ± sample standard deviation over five random seeds.

4.4. Ablation Study

Table 4 presents the ablation results on the RSRegulation test set. Removing the sparsity loss eliminates the rank compression entirely: the soft effective rank remains at 16.0, the near-binary ratio drops to 0%, and the accuracy decreases to 94.88% (−0.42 points compared with full TARA). This result confirms that the sparsity objective is necessary to induce rank reduction. Removing entropy regularization reduces the near-binary ratio from 45.6% to 16.4%, indicating slower binarization. The effective soft parameter count increases from 0.38 M to 0.47 M, and the accuracy decreases to 94.96% (−0.34 points), demonstrating that entropy regularization contributes to both compression efficiency and classification performance. Removing temperature annealing has the largest individual effect on binarization, reducing the near-binary ratio to 3.7% and increasing the soft effective rank to 5.2. The accuracy decreases to 94.73% (−0.57 points), representing the largest single-component degradation and confirming that progressive temperature reduction is critical for discrete rank convergence. The full TARA configuration achieves the best overall trade-off across all metrics, with the lowest soft rank (3.87), the highest near-binary ratio (45.6%), the fewest effective parameters (0.38 M), and the highest accuracy (95.30%) and F1 (95.44%).

Table 4. Ablation results of TARA components on the RSRegulation test set.

Method	Effective Soft Params	Accuracy (%)	F1 (%)	Soft Rank	Near-Bin (%)
w/o Sparsity Loss	1.57 M	94.88	94.91	16.0	0
w/o Entropy Regularization	0.47 M	94.96	95.11	4.8	16.4
w/o Temperature Annealing	0.51 M	94.73	94.82	5.2	3.7
TARA (Ours)	0.38 M	95.30	95.44	3.87	45.6

All variants use the same maximum trainable adapter budget of 1.57 M. Soft effective parameters are estimated from the learned soft effective rank.

4.5. Rank Distribution Analysis

Figure 3 presents the layer-wise soft and hard rank distributions at training completion (τ = 0.10). Soft effective rank is computed as the sum of the soft mask values and characterizes the continuous allocation state during optimization. Hard rank is defined as the number of dimensions whose deterministic sigmoid scores exceed 0.5 and determines the physically deployed adapter structure. TARA allocates mean soft effective ranks of 3.02 and 4.71 to the query and value projections, respectively, with corresponding mean hard ranks of 0.50 and 1.25. The hard configuration was physically instantiated and evaluated in Section 4.3 rather than being used only as a theoretical parameter estimate. Value projections consistently exceed query projections across the majority of layers. Layers L18 through L23 exhibit progressive rank reduction, particularly in query projections, where soft effective rank approaches zero in the deepest layers.

Figure 3. Layer-wise soft and hard rank allocation at training completion. The left panel shows query and value ranks across 24 Transformer layers, while the right panel presents the corresponding rank-importance distribution over 16 candidate dimensions. Mean soft ranks are 3.02 for query and 4.71 for value, and mean hard ranks are 0.50 and 1.25, respectively.

The larger soft and hard ranks retained by value projections indicate a non-uniform query–value adaptation pattern on the present compliance-classification task. One possible interpretation is that the classification signal depends strongly on the aggregation of constraint-related content, including numerical thresholds, scope conditions, and regulatory identifiers. Value projections control the content propagated from attended positions, whereas query projections determine where attention is directed. The very low mean hard query rank suggests that query adaptation is largely removable in many layers for this dataset. Nevertheless, the observed asymmetry should not be interpreted as evidence that geospatial semantics alone cause the rank pattern. The binary classification formulation, RoBERTa-large architecture, and structured data-generation process may also contribute. Accordingly, the interpretability discussion is based primarily on the soft allocation pattern, while the hard-pruning experiment establishes that the resulting discrete structure remains operational.

The near-binary ratio reaches 45.6% at epoch 8 (τ = 0.10), compared with 0% at epoch 5 (τ = 0.812), confirming that temperature annealing promotes progressive discretization during the final stage of training. The importance heatmap reveals heterogeneous patterns of dimension retention across layers, confirming that TARA learns differentiated rank configurations rather than imposing uniform global compression.

4.6. Training Dynamics and Efficiency

Figure 4 shows the training dynamics over 8 epochs. The training loss decreases rapidly within the first 2000 steps and then continues to decline more gradually. The validation accuracy and F1 remain above 0.95 throughout training, indicating that progressive rank compression does not disrupt the classification performance. The rank allocation dynamics panel shows that the soft effective rank decreases monotonically from 7.88 at epoch 1 to 3.87 at epoch 8, corresponding to an approximately 51% reduction, while the near-binary ratio increases sharply in the final two epochs as temperature annealing approaches its minimum value.

Figure 4. TARA training dynamics over 8 epochs. (Left): Training and validation loss. (Center): Validation accuracy and F1 (both above 0.95 throughout). (Right): Average soft effective rank decreasing monotonically from 7.88 (epoch 1) to 3.87 (epoch 8), with near-binary ratio rising to 45.6% at epoch 8.

In terms of training overhead, TARA requires approximately 10% more training time than LoRA-16, mainly because of Gumbel sampling and regularization computation. Its peak training memory usage is 18.6 GB, compared with 17.8 GB for LoRA-16. The 75.8% soft rank compression remains a training-time statistic that characterizes the continuous allocation state and should not be interpreted as the final inference parameter count. Physical hard pruning was therefore evaluated separately in Section 4.3. The resulting adapter contains 0.086 M deployed parameters and achieves 95.12 ± 0.11% accuracy and 95.21 ± 0.10% F1. Compared with the unpruned TARA adapter, physical pruning reduces the adapter size from 6.28 MB to 0.34 MB and increases the inference throughput from 398 to 467 samples per second.

5. Discussion

5.1. Application Value in Geo-Information Systems

The reduced parameter footprint of TARA supports the deployment of adapted pre-trained language models across several geo-information processing scenarios. Although TARA is methodologically domain-agnostic, its application in this study is grounded in a real-world geospatial and remote sensing research project involving regulatory compliance and data-governance requirements. For large geospatial platforms, such as Copernicus and USGS Earth Explorer, which process substantial daily volumes of data usage declarations and access requests, compressed models can support automated compliance pre-screening at a lower per-query cost. For municipal and national spatial data infrastructure platforms, the same approach can be applied to automated metadata classification, data-sharing agreement verification, and regulatory text retrieval. For edge devices used in field remote sensing operations, the compressed adapter reduces the memory requirements compared with full-rank LoRA alternatives.

5.2. Analysis of TARA’s Behavior

TARA jointly optimizes the task performance and rank allocation under a single training objective, in contrast to the sequential pipeline of AdaLoRA, in which full-rank training precedes SVD-based rank reduction. In the design of TARA, importance scores receive task-relevant gradient signals throughout training, allowing the model to identify rank dimensions that are genuinely useful for the target task. On RSRegulation, TARA achieves a slightly higher mean accuracy and F1 than AdaLoRA while learning a lower soft effective rank. However, the exact McNemar test indicates that the predictive difference is not statistically significant. These results therefore support the parameter–performance competitiveness of joint rank optimization rather than statistically significant superiority in classification performance. The value-greater-than-query soft rank allocation observed on RSRegulation shows that TARA recovers task-specific adaptation patterns without explicit architectural guidance. For regulatory compliance classification, value-side content aggregation appears to be the principal bottleneck for domain adaptation, which is consistent with the rule-constrained and semantically dense nature of geospatial regulatory text. This result supports the broader view that layer-wise rank requirements are jointly determined by architecture and task semantics and cannot be captured by a fixed-rank setting.

The training loss converges within 2000 steps, and the validation accuracy remains above 0.95 throughout training. The soft effective rank decreases from 7.88 at epoch 1 to 3.87 at epoch 8, while the validation performance remains stable. The additional physical hard-pruning experiment further shows that the learned discrete rank configuration retains comparable predictive performance after deployment compression.

5.3. Interpretability of Learned Rank Distributions

TARA provides a complementary perspective for analyzing layer-wise adaptation in Transformers. Rather than relying on attention visualization or probing classifiers, it characterizes the continuous allocation tendency of each layer through learned soft ranks and derives a discrete deployment structure through hard thresholding. On RSRegulation, value projections retain higher soft and hard ranks than query projections, while several query projections retain no hard rank dimensions. The physical hard-pruning results show that this highly sparse configuration remains operational.

These findings suggest a task-dependent allocation pattern under the current dataset and backbone, but they should not be interpreted as a definitive functional decomposition of Transformer attention. The interpretability conclusions in this study are primarily based on the relative soft-rank distribution across layers and projection types; hard ranks are used to define and validate the final deployed structure.

In settings where dynamic rank learning is computationally expensive, these patterns could inform static rank configuration heuristics, such as allocating more capacity to value projections and reducing rank in the deepest layers. The generalizability of these heuristics to other geo-information tasks requires further investigation.

5.4. Limitations

RSRegulation samples are generated by GPT-4 under structured prompts. Although clause-level isolation prevents lexical leakage across splits, the three negative-example categories are mechanically balanced, and the sample lengths are relatively short (mean 34 tokens). The benchmark may be more regular than real-world compliance auditing scenarios involving free-form user queries, incomplete regulatory references, or domain-mixed documents. The 95.30% accuracy may reflect the structured generation distribution rather than generalized regulatory text understanding, and validation on independently collected real-world compliance texts is needed to assess the true generalization.

The 75.8% soft rank compression characterizes the continuous allocation state during training and is distinct from the final hard-pruned inference structure. Physical hard pruning has now been empirically evaluated, reducing the deployed adapter to 0.086 M parameters while retaining 95.12 ± 0.11% accuracy and 95.21 ± 0.10% F1. Nevertheless, this validation is limited to RSRegulation and RoBERTa-large. Whether similarly sparse hard-rank configurations remain effective across other datasets, model architectures, and longer or noisier geo-information texts requires further investigation.

The experiments are conducted on a single backbone (RoBERTa-large) and a single classification dataset. Generalization to decoder-only architectures, larger parameter scales, generative geo-information tasks, and additional geospatial text domains requires further investigation. TARA also introduces target sparsity ρ and regularization weight β whose optimal values are dataset-dependent; systematic sensitivity analysis across a range of these parameters is not yet reported.

6. Conclusions

This paper proposes TARA, an end-to-end differentiable adaptive rank allocation method for LoRA-based fine-tuning of large pre-trained language models applied to geo-information text classification. By introducing learnable rank importance scores with Gumbel–Sigmoid sampling and Straight-Through Estimation, TARA jointly optimizes task performance and parameter efficiency under a controllable sparsity constraint, without requiring the post hoc pruning used by existing adaptive rank methods.

On the RSRegulation geospatial regulatory compliance dataset, TARA achieves 95.30 ± 0.10% accuracy and 95.44 ± 0.10% F1 with a maximum trainable adapter budget of 1.57 M. The learned soft allocation corresponds to approximately 0.38 M effective adapter parameters. After physical hard pruning, the deployed adapter contains 0.086 M parameters and retains 95.12 ± 0.11% accuracy and 95.21 ± 0.10% F1. The adapter size decreases from 6.28 MB to 0.34 MB, while the inference throughput increases from 398 to 467 samples per second.

The learned allocation retains higher ranks in value projections than in query projections under the current task and backbone. Further work will examine whether this pattern generalizes to other model architectures, longer documents, and independently collected geo-information texts.