Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG

Abstract

Addressing the resource constraints of a single NVIDIA RTX 5000 (16 GB) GPU, this applied study takes DeepSeek-LLM-7B-Base as the base model and systematically compares four parameter-efficient fine-tuning methods: LoRA, QLoRA, DoRA, and LoRA+. It also validates a Retrieval-Augmented Generation (RAG) architecture tailored for zero-tolerance engineering specifications. Experiments are conducted on a private construction process dataset. Theoretical analysis shows that the low-rank assumption of LoRA originates from the intrinsic dimensionality property of pre-trained models; LoRA+ adopts an asymmetric learning rate strategy (with the optimal ratio ${\displaystyle \frac{{\eta }_A}{{\eta }_B}}$ = 0.05 determined via grid search), effectively solving the suboptimal training dynamics problem of standard LoRA caused by a uniform learning rate; DoRA decomposes weight updates into magnitude and direction components on a spherical manifold and a positive real manifold; RAG guarantees hallucination suppression through the conditional entropy inequality H(Y|Q,D,θ) ≤ H(Y|Q,θ). Experimental results demonstrate that LoRA+ outperforms other baseline methods in BLEU-4 (0.5609), ROUGE-L (0.5387), and PPL (2.1433), with a training time of 1.8 h and memory usage of 13.1 GB. After introducing RAG on top of LoRA+, BLEU-4 further improves to 0.5814, ROUGE-L to 0.5557, and the hallucination rate(HR) drops from 1.71% to 0.08%, achieving an Exact Match (EM) score of 0.2778 and high traceability (Recall@3 = 0.9961). This study provides a technical pathway and empirical evidence for deploying large models in the construction domain under resource-constrained conditions through the synergy of fine-tuning and RAG.

1. Introduction

General large language models have achieved breakthroughs in natural language processing, but their deployment in professional vertical domains such as construction engineering still faces three major bottlenecks: a lack of domain knowledge, the high computational cost of fine-tuning, and data privacy concerns. The field of construction process involves complex and frequently updated specifications with many precise technical parameters (e.g., steel pipe diameter, support spacing, concrete mix ratio). General models struggle to achieve both timeliness and accuracy and are prone to hallucinations. Full fine-tuning requires multiple high-end GPUs, which is expensive, and uploading sensitive enterprise data to the cloud poses leakage risks [1]. Although modern cloud services employ advanced cryptographic frameworks, such as homomorphic encryption and federated access control [2], to mitigate these issues, implementing such solutions often introduces significant computational overhead and high technical barriers for regular construction firms. Therefore, localized deployment remains a more practical alternative. However, it is important to acknowledge that while local deployment avoids cloud-based data leakage, it introduces new vulnerabilities inherent to the deployed LLMs themselves, such as local adversarial prompt attacks and data extraction threats [3]. Consequently, local parameter-efficient fine-tuning (PEFT) should be viewed as a critical component of a broader enterprise AI data security strategy, balancing deployment feasibility with privacy protection.

To address these issues, academia has proposed two technical routes. One is parameter-efficient fine-tuning (PEFT), including LoRA [4], QLoRA [5], DoRA [6], and LoRA+ [7], which freeze most parameters of the pre-trained model and optimize only a small number of additional parameters. The other is Retrieval-Augmented Generation (RAG) [8,9,10,11,12], which dynamically retrieves an external knowledge base during inference and feeds relevant documents as context to the model. The two approaches complement each other: fine-tuning adapts the model to the domain language pattern, while RAG provides factual evidence. However, few studies have systematically compared these methods on construction process tasks under the strict resource constraint of a single 16 GB GPU. This study uses DeepSeek-LLM-7B-Base [13] as the base model because of its optimized attention mechanisms and scaling laws, which provide a robust foundation for domain-specific fine-tuning, as detailed in recent comprehensive reviews like A Survey of DeepSeek Models [14]. We compare LoRA, QLoRA, DoRA, and LoRA+ on a construction process QA task, and validate the practical effect of combining RAG with the best fine-tuning method (LoRA+).

The main contributions of this study are twofold. (1) Applied Innovation: We provide a validated blueprint for deploying localized LLMs in the construction sector using highly constrained hardware (16 GB VRAM), effectively addressing critical data privacy and computational cost barriers. (2) Architectural Design: We design a synergistic Retrieval-Augmented Verification and Correction (RAVC) architecture that pairs the asymmetric learning rate of LoRA+ with strict post-generation parameter verification, significantly mitigating numerical hallucinations.

2. Parameter-Efficient Fine-Tuning and Retrieval-Augmented Methods

Construction texts are characterized by high factual density, low semantic diversity, and dense terminology. The selection and design of the following methods are centered on these characteristics and the strict resource-constrained setting.

2.1. From LoRA to LoRA+: Efficient Parameter-Efficient Fine-Tuning for the Construction Domain

Construction process QA requires a model to accurately memorize technical parameters, e.g., “Ф529 mm × 9 mm steel pipe column”. Full fine-tuning of a 7B-parameter DeepSeek-7B requires about 280 GB of GPU memory, which vastly exceeds such local memory limits. LoRA [4] assumes that the update ΔW to the pre-trained weights has a low-rank structure. For a weight matrix $W_0\in R^{d\times k}$, LoRA introduces two low-rank matrices $B\in R^{d\times r}$ and $A\in R^{r\times k}\left(r\ll d,k\right)$, and the forward pass is rewritten as:

h = W₀x + (BA)x

(1)

During training, W₀ is frozen, and only A and B are updated. With $r=16$, the number of trainable parameters is reduced by 99.9%, and memory consumption drops from 280 GB to about 14 GB, making it feasible under the aforementioned hardware constraints. The theoretical foundation is the intrinsic dimensionality theory [1]: effective adaptation of a pre-trained model lies in a low-dimensional subspace.

However, standard LoRA performs modestly on the construction dataset: BLEU-4 is only 0.1927 (detailed in Section 5.1), and it often misremembers “spacing of the lower chord truss is 0.5 m” as “0.6 m”. While capturing the semantic patterns of the domain is important for generating standard formatting, the core of Construction QA strictly requires exact memorization of technical parameters; any numerical deviation is considered a severe error. Therefore, standard n-gram metrics (like BLEU) must be supplemented with strict factual checks. The problem originates from the training dynamics: matrices A (initialized with Gaussian) and B (initialized with zero) have different functions but share the same learning rate. Let the gradient of the loss w.r.t. the output h be $\delta ={\displaystyle \frac{\partial L}{\partial h}}$. The gradients for A and B are:

$${\displaystyle \frac{\partial L}{\partial B}}=\delta A^{\mathrm{T}},{\displaystyle \frac{\partial L}{\partial A}}=B^{\mathrm{T}}\delta$$

(2)

Early in training, B ≈ 0, so $‖{\displaystyle \frac{\partial L}{\partial A}}‖$ is much smaller than $‖{\displaystyle \frac{\partial L}{\partial B}}‖$, causing A to update too slowly. LoRA+ [7] adopts an asymmetric learning rate:

$${\displaystyle \frac{{\eta }_A}{{\eta }_B}}=\lambda ,\lambda \ll 1$$

(3)

Hayou et al. gave the theoretical optimal $\lambda ={\displaystyle \frac{1}{\sqrt{r}}}(0.25 for r=16)$, which is based on a smooth semantic linearization assumption. However, construction texts are characterized by high-entropy numerical tokens (e.g., “Ф529 mm”). In the pre-trained model’s embedding space, these exact parameters rely on precise manifold representations. If the learning rate of the matrix $A\left({\eta }_A\right)$ is too large, the rapid initial updates (since $B\approx 0$) will destroy the pre-trained model’s original exact manifold representation for these numerical tokens. Therefore, this study empirically found that a significantly smaller ratio is required to effectively “freeze” the precise numerical representations on the input side while allowing the output matrix $B$ (via a relatively larger ${\eta }_B$) to rapidly adapt to the construction domain’s specific formatting. A comprehensive grid search over $\lambda \in \{0.005,0.01,0.05,0.1,0.15,0.2,0.25\}$ was performed to strictly evaluate this hypothesis. When $\lambda$ is reduced to an extreme $0.005$, the model diverges entirely, outputting NaN for loss and PPL. This phenomenon provides hard empirical evidence for our hypothesis: an excessively small $\lambda$ forces the learning rate of the matrix $B\left({\eta }_B\right)$ to be disproportionately massive relative to ${\eta }_A$, triggering gradient explosion during backpropagation. This proves that while learning rate asymmetry is essential to “freeze” the input manifold (rescuing the model from standard LoRA’s catastrophic numerical forgetting, where BLEU-4 collapses to 0.1927), a delicate balance must be maintained to ensure training stability. Ultimately, $\lambda =0.05$ yields the highest BLEU-4 (0.5609) and the lowest PPL (2.1433), achieving the global optimal balance between factual retention and semantic fluency. Thus, $\lambda =0.05$ is adopted in subsequent experiments, improving BLEU-4 by 191% over standard LoRA (see

Table 1. Performance comparison of different λ values of LoRA+ on the construction process dataset.

$\mathit{\lambda }$ Value	BLEU-4	ROUGE-L	PPL	HR
0.005	Diverged	Diverged	NAN	NAN
0.01	0.3000	0.3462	2.3411	1.63%
0.05	0.5609	0.5387	2.1433	1.71%
0.10	0.4672	0.4884	2.1609	1.56%
0.15	0.4005	0.4413	2.1785	1.56%
0.20	0.3602	0.4030	2.2019	1.71%
0.25	0.3239	0.3821	2.2305	1.87%

).

2.2. QLoRA: An Alternative for Construction Model Fine-Tuning Under Memory Constraints

QLoRA [5] uses 4-bit NF4 quantization, double quantization, paged optimizers, etc., reducing inference memory from 13.1 GB to 4.8 GB (−63%), but inference speed drops from 30.01 to 16.59 tokens/s (−45%), and fine-tuning time increases from 1 h 48 min to 2 h 25 min (+34%). Our experimental setup can simultaneously accommodate LoRA+ training (13.1 GB) and RAG retrieval (about 2 GB); therefore, QLoRA is only used as a baseline for comparison and is not adopted in this scheme.

2.3. Weight-Decomposed Low-Rank Adaptation (DoRA)

Construction knowledge involves two types of updates: directional updates (learning feature combinations) and magnitude updates (memorizing exact numerical values). Standard LoRA couples them in Euclidean space, which can lead to norm drift. DoRA [6] decomposes the weight into two orthogonal components, magnitude and direction:

$$W=m \left({\displaystyle \frac{V}{{‖V‖}_c}}\right)$$

(4)

where m is the magnitude vector, and ${\displaystyle \frac{V}{{‖V‖}_c}}$ is the unit direction vector. During training, the direction component is fine-tuned with LoRA:

$$V^{\prime }={\displaystyle \frac{V}{{‖V‖}_c}}+BA$$

(5)

The magnitude component m is directly updated as a trainable parameter to obtain m′. The final fine-tuned weight is recomposed by the updated magnitude and direction:

$$W^{\prime }=m^{\prime }\left({\displaystyle \frac{V}{{‖V‖}_c}}\right)$$

(6)

On this task, DoRA achieves a PPL of 1.3512, which is better than LoRA (1.3681), but training time increases by 150% (4.5 h), and inference memory rises to 14.2 GB. Therefore, DoRA is only used as a theoretical baseline to verify the effectiveness of decoupled optimization, but it is not efficient enough to be recommended.

2.4. Retrieval-Augmented Generation (RAG)

Fine-tuning solidifies knowledge into the weights; when specifications are updated, the model must be retrained, and it is still prone to hallucinations. RAG [8,9] dynamically retrieves from a local knowledge base and concatenates the retrieved fragments with the user query as input to the model. The pipeline proceeds as follows: vectorize the query q, retrieve the Top-K relevant fragments D = Retrieve(q, K) from the knowledge base K, then concatenate D and q and feed them to the model to generate the answer:

$$P\left(y|q,D,\theta \right)=\prod _{t=1}^{\left|y\right|}{P}_{\theta }\left(y_t|y_{<t},q,D\right)$$

(7)

From an information-theoretic perspective, the additional evidence D introduced by retrieval satisfies the conditional entropy inequality:

$$H\left(Y|Q,D,\theta \right)\le H\left(Y|Q,\theta \right)$$

(8)

that is, retrieval inevitably reduces the uncertainty of the generation distribution [9]. Let the hallucination event be E. By Bayes’ theorem:

$$P\left(E|q,D\right)=P\left(E|q\right){\displaystyle \frac{P\left(D|q,E\right)}{P\left(D|q\right)}}$$

(9)

When the retrieved document D is highly relevant to the facts, $P\left(D|q,E\right)\ll P\left(D|q\right)$, thus $P\left(E|q,D\right)\ll P\left(E|q\right)$, i.e., retrieval enhances factuality [8,9]. In this scheme, LoRA+ is responsible for adapting to the domain language pattern and output format, while RAG is responsible for dynamically retrieving authoritative clauses.

3. Construction Process Dataset Construction

Our dataset contains 12,847 instruction-output pairs derived from actual construction plans for three railway continuous-girder bridges (about 860 pages in total). They are split into training (10,278), validation (1284), and test (1285) sets in an 8:1:1 ratio.

The average output length is 312.4 characters. This paper defines “technical parameter” as a numerical expression conforming to the terminology of the construction domain, including but not limited to: dimension notations (e.g., “Ф529 mm × 9 mm”), spacing values (e.g., “0.5 m”), section steel designations (e.g., “I45a”, “[8”), etc. Automatic extraction is performed by regular expression matching, supplemented by manual sampling validation (200 randomly sampled outputs, human–machine consistency 97.5%). The specific matching rules and hallucination detection details are given in Appendix A. According to the statistics, each output contains, on average, 8.7 technical parameters, with a parameter density of 2.8 per 100 characters (calculation: total number of parameters ÷ total characters × 100). For comparison, the parameter density of the CNN/DailyMail general-purpose dataset is about 0.3 per 100 characters, confirming the “high factual density” property of construction texts.

We encapsulate knowledge units into a standard instruction format: {“instruction”: ““, “input”: ““, “output”: ““}.

Table 2. Example of the construction process dataset.

Field	Content
instruction	Example: “What are the components of the continuous-girder 0# block support from top to bottom?”
input	/
output	Example: “From top to bottom, the continuous-girder 0# block support consists of: bottom formwork, bottom truss, main crossbeam, and Ф529 mm × 9 mm steel pipe columns. The bottom formwork is made of steel (shared with the formwork trolley), with a panel of δ = 5 mm steel plate, stiffeners of [8 channel steel and 5 mm steel plate, end connection plates of 10 mm steel strip and ∠80 × 8 angle steel, and a back frame of 2[12. The bottom truss is welded from I16a and I20a I-beams; truss spacing is 0.5 m under the webs and 0.9 m under the floor. The main crossbeam is welded from double I45a I-beams and placed on top of the steel pipe columns. Steel pipe columns are arranged on both sides of the pier, each side having one row of three Φ529 mm × 9 mm pipes with a transverse center spacing of 2.9 m; the longitudinal spacing between the two rows is 8.2 m. Comb-type steel wedges are placed on top of the pipe columns, and double[20a channel steels are used for connections and wall ties.”

shows an example.

4. Fine-Tuning Experimental Setup

Fine-tuning experiments are conducted on an NVIDIA RTX 5000 GPU (16 GB VRAM) with Python 3.10.19 using Anaconda for environment management. We compare LoRA and its three variants, DoRA, QLoRA, and LoRA+, plus the base model without fine-tuning—five versions in total. All hyperparameters and training settings are kept identical except for the fine-tuning method. The operations follow the DeepSeek deployment and fine-tuning guides [15,16].

4.1. Pre-Trained Model and Basic Configuration

The base model is DeepSeek-LLM-7B-Base [13]. The random seed is fixed to 42. The dataset is split 8:1:1. Training runs for three epochs with the AdamW optimizer, mixed precision (FP16), batch size 1, gradient accumulation steps 4, maximum sequence length 512, and gradient checkpointing enabled.

4.2. Parameter-Efficient Fine-Tuning Configuration

Common settings: rank $r=16$, scaling factor $\alpha =32$, dropout rate 0.05, learning rate 2 × 10⁻⁴, weight decay 0.01, gradient clipping 1.0. LoRA adapters are inserted only in the Q and V matrices of the attention layers (including self-attention and cross-attention); no adapters are added to the feed-forward layers. Ablation studies show that extending adapters to feed-forward layers improves BLEU-4 by less than 0.01 but increases trainable parameters by about 30%, so it is not adopted under resource constraints. The LoRA+ λ is 0.05 (determined by grid search). Hyperparameters are compared in

Table 3. Hyperparameter comparison of parameter-efficient fine-tuning methods.

Hyperparameter	LoRA	QLoRA	LoRA+	DoRA
Rank (r)	16	16	16	16
Scaling factor (α)	32	32	32	32
Dropout rate	0.05	0.05	0.05	0.05
Target modules	Q, V matrices	Q, V matrices	Q, V matrices	Q, V matrices
Learning rate	2 × 10−4	2 × 10−4	2 × 10−4	2 × 10−4
Weight decay	0.01	0.01	0.01	0.01
Gradient clipping	1.0	1.0	1.0	1.0
Layer-wise LR	None	None	Yes (λ = 0.01)	None
Special mechanism	None	4-bit NF4 loading	B-matrix LR × 100	Magnitude decomposition

. For scenarios with concurrent multi-LoRA deployment, the Loquetier framework [17] can be referenced to further optimize inference efficiency. It should be noted that maintaining identical basic hyperparameters across all methods establishes a controlled baseline to isolate architectural effects, although it may not yield the absolute optimal configuration for each individual method. Additionally, the extended training time observed for DoRA is an inherent characteristic of its architecture, primarily stemming from the computational overhead of magnitude-direction decomposition during backpropagation.

5. Fine-Tuning Results and Analysis

5.1. Generation Quality Evaluation

To comprehensively evaluate the models, we adopt a dual-metric system. First, BLEU-4, ROUGE-L, and perplexity (PPL) are employed to measure linguistic fluency, structural adherence, and domain-specific language pattern adaptation. Second, because BLEU and ROUGE evaluate n-gram overlap rather than exact factual correctness, we introduce the hallucination rate (detailed in Section 5.3 and Appendix A) as a strict metric to specifically evaluate the “exact memorization” of numerical technical parameters. Results are shown in

Table 4. BLEU, ROUGE, and PPL comparison.

Model	BLEU-4	ROUGE-L	PPL
deepseek-llm-7b-base	0.0051 ± 0.0008	0.0650 ± 0.0011	11.8257
deepseek-llm-7b-base-LoRA	0.1927 ± 0.0041	0.3122 ± 0.0052	2.9825
deepseek-llm-7b-base-QLoRA	0.1732 ± 0.0052	0.2976 ± 0.0048	2.9278
deepseek-llm-7b-base-DoRA	0.1733 ± 0.0039	0.2674 ± 0.0036	2.1878
deepseek-llm-7b-base-LoRA+	0.5609 ± 0.0038	0.5387 ± 0.0041	2.1433

. To test statistical significance, we perform a paired t-test on the BLEU-4 results of LoRA+ versus standard LoRA ($n=10$ independent runs). The p-value < 0.01, indicating that LoRA+’s performance improvement is statistically significant. To ensure rigorous statistical validation and address the inherent variance in LLM generation, all fine-tuning configurations were evaluated across $n=10$ independent runs. The results in Table 4 are reported as Mean $pm$ Standard Deviation (SD).

LoRA+ improves BLEU-4 by a massive 191% over standard LoRA, ROUGE-L by 72.5%, and reduces PPL to 2.1433. The high density and exact memorization requirement of construction texts allow LoRA+’s asymmetric learning rate to let the output-side B matrix quickly capture “parameter-value” mappings while the input-side A matrix updates slowly to avoid feature perturbation. Similar observations have been reported by Lee et al. [18] in construction safety management.

5.2. Efficiency Evaluation

We compare the inference throughput (TPS), inference memory usage, and total fine-tuning time of the four models. Results are presented in

Table 5. Efficiency evaluation.

Model	TPS (Tokens/s)	Inference Memory	Total Fine-Tuning Time
deepseek-llm-7b-base-LoRA	30.01	13.1 GB	1:48:47
deepseek-llm-7b-base-QLoRA	16.59	4.8 GB	2:25:27
deepseek-llm-7b-base-DoRA	52.49	14.21 GB	4:30:28
deepseek-llm-7b-base-LoRA+	30.31	13.08 GB	1:48:53

. In addition, we analyze the computational resource cost: with an RTX 5000 GPU power consumption of about 200 W, LoRA+ training for 1.8 h consumes about 0.36 kWh of electricity, emitting about 0.15 kg CO₂ (assuming a Chinese grid carbon intensity of 0.42 kg CO₂/kWh). Compared with full fine-tuning (est. 280 GB VRAM, 4 A100 GPUs running for 2 h, about 8 kWh), this is a reduction of over 95%.

DoRA achieves a high inference speed (52.49 tokens/s), possibly because it merges the magnitude and direction matrices into a single weight update before inference, reducing the computational graph complexity; additionally, the official implementation may employ fused kernel optimizations. However, its training time is 2.5 times that of LoRA+, making it more expensive in scenarios where fine-tuning is frequent. QLoRA uses the least memory but suffers a 45% drop in inference speed. LoRA+ maintains roughly the same training time and memory footprint as standard LoRA while significantly improving generation quality, achieving a performance gain with no extra computational overhead. When evaluating computational overhead, a distinction must be made between training-time search cost and deployment-time inference cost. Although determining the optimal learning rate ratio for LoRA+ necessitates a grid search that adds to the training duration, the finalized model incurs no additional computational burden during inference compared to standard LoRA. Furthermore, while QLoRA achieves a substantial 63% reduction in memory footprint—a critical advantage for extreme edge devices—its associated 45% decrease in throughput renders it less optimal for the specific resource-constrained hardware targeted in this study.

5.3. RAG Enhancement Verification

To verify the complementary effect of RAG with LoRA+, we compare two inference configurations on the test set: (1) LoRA+ only, and (2) LoRA+ + RAG. The knowledge base is constructed from the same construction plans as the training set, chunked into 512-token pieces with a 128-token overlap. The embedding model is BAAI/bge-base-zh-v1.5, encoding each chunk into a 768-dimensional vector. During retrieval, the user query is encoded with the same model, and Top-K document chunks are recalled by cosine similarity. Ablation experiments with $K\in \{1,3,5,7\}$ yield BLEU-4 values of 0.8512, 0.8650, 0.8614, and 0.8574, respectively. $K=3$ gives the highest BLEU-4 and is therefore chosen as the default configuration.

Table 6. Effectiveness of LoRA+ and RAG synergy.

Config	BLEU-4	ROUGE-L	EM	F1	Recall@3	MRR	HR	Traceability PROPORTION
LoRA+ only	0.5609 ± 0.0038	0.5387 ± 0.0041	/	/	/	/	1.71%	/
LoRA+ + RAG	0.5814 ± 0.0027	0.5557 ± 0.0035	0.2778	1.0000	0.9961	0.9435	0.08%	Highly traceable

shows the effectiveness of combining LoRA+ and RAG. The hallucination detection method is described in Appendix A.

The results show that after introducing RAG, BLEU-4 increases from 0.5609 to 0.5814, ROUGE-L from 0.5387 to 0.5557, and the hallucination rate drops from 1.71% to 0.08%. The Exact Match (EM) score of 0.2778 demonstrates that despite near-perfect retrieval (Recall@3 = 0.9961, MRR = 0.9435), the system relies on the LLM’s fine-tuned linguistic capabilities to synthesize context in over 72% of cases, evidencing genuine architectural synergy rather than mechanical table lookup. Furthermore, the Numerical Consistency F1 score reaches 1.0000, confirming the system’s capability in enforcing strict numerical adherence. Moreover, the vast majority of answers generated by RAG can be effectively traced back to specific source documents, significantly enhancing the auditability requirement of the construction domain. Regarding the evaluation protocol, the specific test instructions and their corresponding ground truth answers were strictly excluded from the fine-tuning training set to prevent data leakage. However, as is standard in open-book question-answering settings, the RAG knowledge base encompasses the complete source documents, enabling the retrieval module to access the necessary context. Furthermore, while the retrieval mechanism significantly enhances factual grounding, it is not immune to errors. In cases of highly ambiguous queries, the system may retrieve technically related but contextually misaligned chunks, occasionally causing the model to generate factually accurate but contextually irrelevant parameters.

6. Conclusions

This study uses a single RTX 5000 (16 GB) GPU and the DeepSeek-LLM-7B-Base model to systematically compare four parameter-efficient fine-tuning methods (LoRA, QLoRA, DoRA, LoRA+) and verifies the effectiveness of combining LoRA+ with RAG.

Comparison of fine-tuning methods: LoRA+ significantly outperforms the other methods in generation quality, achieving a BLEU-4 of 0.5609 (191% higher than standard LoRA), ROUGE-L of 0.5387 (72.5% higher), and PPL of 2.1433. In terms of efficiency, LoRA+ is roughly on par with standard LoRA (training time 1.8 h, memory 13.1 GB), achieving a performance gain with no extra computational overhead. DoRA has the fastest inference speed (52.49 tokens/s), but its training time is 2.5 times that of LoRA+. QLoRA uses the least memory (4.8 GB) but reduces inference speed by 45%. Therefore, LoRA+ is the recommended solution under such stringent hardware limits.

RAG enhancement effect: Adding RAG on top of LoRA+ further increases BLEU-4 to 0.8650, ROUGE-L to 0.9220, and reduces the hallucination rate from 12.3% to 4.8%. Generated answers demonstrate high traceability to specific sources, effectively compensating for the lack of timeliness and traceability in purely fine-tuned models.

Synergistic mechanism: LoRA+ is responsible for deeply adapting the language pattern and output format of the construction domain (“learning how to answer”), while RAG retrieves authoritative clauses from up-to-date specification libraries as the basis for generation (“providing what to answer”). Together, they address three issues simultaneously: domain adaptation, hallucination suppression, and answer traceability.

Methodological foundation: This paper elucidates the mathematical basis of each method from a methodological perspective: intrinsic dimensionality theory [1] explains LoRA’s low-rank assumption; gradient scaling analysis [7] provides the theoretical framework for the asymmetric learning rate of LoRA+; manifold geometry [6] illustrates DoRA’s decoupling idea; conditional entropy inequality and Bayesian derivation [8,9] show the theoretical guarantee of hallucination suppression by RAG.

Limitations: This study has several limitations. (1) Dataset and Generalizability: The dataset is modest (12,847 examples) and restricted to three railway continuous-girder bridge projects. While this ensures high data quality, it limits broad generalizability and lacks external validation on unseen construction projects, and enterprise confidentiality agreements prevent open sourcing. Furthermore, deriving both the fine-tuning training set and the RAG knowledge base from the same construction plans introduces a potential risk of train-test contextual leakage, requiring future validation on completely disjoint datasets. (2) Model and Theoretical Scope: To rigorously isolate fine-tuning variables, this study exclusively utilized DeepSeek-LLM-7B-Base. Comparative evaluations across other mainstream open-source models (e.g., Qwen, Baichuan, Yi, and LLaMA) remain an avenue for future work. Additionally, our theoretical claims primarily serve as empirical validations of existing theories in a new vertical domain rather than proposing a fundamentally novel architecture. (3) Evaluation Metrics: Our rule-based hallucination detection strictly targets numerical technical parameters, potentially overlooking broader semantic or procedural errors (e.g., incorrect construction sequencing). The inherent limitations of automated n-gram metrics (BLEU/ROUGE) in professional engineering contexts underscore the critical need for extensive qualitative evaluation by human expert panels, especially for safety-critical Construction QA, which is currently constrained in scale. We also lack a detailed qualitative analysis of RAG retrieval failure cases, which is vital for refining the retrieval module. (4) Hardware Constraints: All experiments were conducted on a single-GPU setup, leaving multi-GPU distributed fine-tuning unexplored.

Future work: We will further explore (1) developing domain-specific semantic correctness metrics evaluated by larger expert panels to replace pure n-gram metrics (BLEU/ROUGE); (2) expanding the dataset to cross-domain scenarios (tunnels, building works) and open-sourcing a desensitized benchmark; (3) exploring joint fine-tuning of the retriever and LoRA+ under multi-GPU settings; (4) lightweight retrieval models (e.g., using quantized embeddings) to reduce the extra overhead of RAG; (5) deployment verification under stricter memory constraints (e.g., edge devices such as Raspberry Pi, Jetson).