An Adaptive Multi-Source Retrieval-Augmented Generation Framework Integrating Query Complexity Awareness and Confidence-Aware Fusion

Abstract

Retrieval-Augmented Generation (RAG) has been observed to encounter challenges in heterogeneous query scenarios characterised by varying evidence requirements and reasoning depths. In order to address this limitation, the present paper puts forward a proposal for an Adaptive Multi-Source RAG framework (AMSRAG) that integrates query complexity awareness with confidence-aware fusion. The framework performs query complexity classification with a pretrained language model, calibrates the classification confidence to guide the dynamic scheduling of retrieval paths and the adjustment of fusion weights, and enables a controllable balance between answer quality and retrieval efficiency through hierarchical path selection and cross-source weighting. The experiments conducted on multiple open-domain question-answering datasets demonstrate that the query complexity classifier achieves an accuracy of 85.9% and a Macro-F1 score of 85.4%. These outcomes indicate the potential for the classifier to generate a reliable decision signal, which can subsequently be utilised to guide the process of adaptive retrieval and fusion. The proposed framework demonstrates a marked improvement in terms of both answer accuracy and retrieval relevance when compared to the fixed-pipeline RAG. In scenarios involving high-confidence queries, the system has been shown to effectively avoid redundant retrieval, thereby reducing the average number of retrievals. In instances of low-confidence complex queries, the system has been shown to enhance evidence coverage and completeness of answers through multi-source retrieval and confidence-weighted fusion. This study proposes a novel methodology for enhancing the adaptability and resource efficiency of RAG systems in response to heterogeneous query conditions.

1. Introduction

Retrieval-Augmented Generation (RAG) [1] is a seminal paradigm that has been shown to be effective in mitigating the limitations of large language models (LLMs) with regard to knowledge, timeliness and coverage. By integrating external retrieval with generation, RAG effectively extends the model’s knowledge boundary and enhances its ability for cross-document reasoning [2,3]. Representative methods include Fusion-in-Decoder (FiD), which fuses multi-document evidence during decoding to improve the utilization of long-text information [4]; Retrieval-Enhanced Transformer (RETRO), which deeply embeds retrievable corpora into the Transformer architecture to support large-scale knowledge invocation [5]; and GraphRAG, which organizes cross-document entity relations through graph structures to strengthen multi-hop reasoning [6]. Meanwhile, researchers have combined Dense Passage Retrieval (DPR) with generative models to propose end-to-end RAG frameworks [4,7], and introduced differentiable retrieval during pretraining (REALM, Retrieval-Augmented Language Model Pre-Training) to internalize external knowledge within models [2]. It is evident that these developments have collectively resulted in a substantial advancement in the domains of knowledge augmentation and reasoning capability. However, most frameworks still use a unified and fixed retrieval generation process, applying the same strategy to queries of varying complexity, and thus cannot reach a good tradeoff between speed and the coverage of evidence.

In terms of Query Complexity analysis and adaptive routing, recent studies have evolved from heuristic statistical features (e.g., length, inverse document frequency distribution, Query Clarity) toward semantic modeling and dynamic decision-making. Self-RAG introduces a Critic Token during generation to determine whether retrieval is necessary [8], while Adaptive-RAG employs a lightweight model to predict Query Complexity and switches among “no-retrieval,” “single-step,” and “iterative” modes [9]. These explorations verify the value of complexity and uncertainty signals for retrieval triggering and path selection. However, their impact remains limited to single-stage switches, and a systematic framework that spans “complexity modeling–confidence calibration–end-to-end scheduling” is still lacking. Furthermore, given that the probabilistic outputs of complexity prediction frequently lack calibration, routing decisions demonstrate instability in the presence of distribution shifts. Therefore, how to elevate complexity and uncertainty signals into interpretable, controllable, and transferable scheduling variables throughout the entire process remains an open question.

Regarding multi-source information and confidence levels, the fusion modeling of both factors is crucial for enhancing overall retrieval performance. Early approaches often used score-based linear fusion methods (CombSUM, CombMNZ) [10] and rank-based fusion methods such as Reciprocal Rank Fusion (RRF) [11], among which RRF is widely used for its simplicity and robustness. Subsequent research incorporated learning-to-rank, reinforcement learning, and pseudo-relevance feedback mechanisms [12,13,14] to more precisely capture the importance and complementarity of retrieval sources, while employing Maximal Marginal Relevance (MMR) to suppress redundancy and enhance evidence coverage [15].

Meanwhile, confidence calibration methods such as Temperature Scaling, Platt Scaling, and Isotonic Regression [16,17,18] have been applied to alleviate model confidence bias, forming the foundation for uncertainty-based fusion and scheduling. Beyond post hoc calibration, uncertainty-aware learning can incorporate uncertainty directly into training objectives or regularizers. For example, Incremona et al. [19] propose a differentiable uncertainty-aware mutual-information regularizer, whereas AMSRAG uses calibrated confidence as a lightweight, model-agnostic signal for routing and fusion without retraining the underlying LLM. Despite progress, these methods mainly adopt static weighting or single-source confidence correction and still lack a mechanism that dynamically integrates Query Complexity and global confidence. How to realize an integrated modeling framework of “Complexity Awareness–Confidence Calibration–Fusion Decision” remains an open research problem. This framework should enable the system to balance information efficiency and confidence fusion across different query types.

To address the above three issues, this paper proposes an Adaptive Multi-Source RAG Framework (AMSRAG) based on Query Complexity awareness and confidence fusion. This framework takes the joint “complexity–confidence” signal as the core scheduling variable, connecting retrieval triggering, path selection, and multi-source fusion into a unified and dynamically controllable process.

AMSRAG systematically integrates complexity modeling and confidence estimation into the RAG pipeline. This integration enables the model to balance efficiency and evidence coverage under different query conditions and offers a novel mechanism for improving the adaptability of RAG systems in heterogeneous query environments.

2. Methods

As shown in Figure 1, the overall architecture of the AMSRAG framework consists of four components: the Query Complexity Classifier (C), the Adaptive Router (R), the set of Multi-Source Retrievers (RS), and the Confidence-Aware Fusion Engine (CAF). The framework first uses C to output the query complexity category c and its calibrated confidence $\alpha$, then R determines the appropriate retrieval path based on this information, selecting the retrievers to be activated in RS to obtain candidate evidence. Finally, CAF performs cross-source fusion and weighting of retrieval results, and the fused evidence set is passed to the generation model to produce the final answer. Here, a retrieval path denotes a routing option (Strategy A/B/C) that specifies the retrieval budget and fusion configuration, rather than a multi-hop evidence path.

The workflow replaces static strategies with query-driven dynamic scheduling: for low-complexity and high-confidence queries, it suppresses unnecessary retrieval calls; for low-confidence or high-complexity queries, it expands the activated retriever set according to the complexity and confidence signals.

Given a query q, the complexity classifier C outputs a pair $\left(c,\alpha \right),$ where c denotes the predicted query complexity category and $\alpha$ represents the calibrated confidence. Guided by this joint signal, the retriever controller R dynamically activates a subset $R{S}^{*}\subseteq RS$ to perform retrieval and construct the candidate evidence set. The CAF module then weights and constrains the diversity of cross-source results according to the complexity and confidence signals, producing the fused context $D^{*}$. Finally, the large language model generates the final answer conditioned on the query and the fused evidence:

$$a=f_{LLM}\left(q,D^{*}\right)$$

(1)

$$D^{*}=CAF\left(R{S}^{*}\left(q|c,\alpha \right)\right)$$

(2)

where $c\in \left\{\text{zero-hop},\text{one-hop},\text{multi-hop}\right\}$, $\alpha \in \left[0,1\right]$, and $R{S}^{*}$ denote the dynamically activated subset of retrievers conditioned on the joint signal $\left(c,\alpha \right)$.

2.1. Query Complexity Modeling and Confidence Estimation

This section formalizes the query complexity taxonomy (zero-hop, one-hop, and multi-hop) and introduces a ModernBERT-based classifier that predicts the complexity label and estimates the associated confidence score for downstream strategy selection and fusion.

Query complexity reflects the span of external evidence required to answer a question. It is divided into 3 categories: zero-hop, one-hop, and multi-hop. Zero-hop queries can be directly answered using the model’s internal knowledge; one-hop queries require a single evidence fragment; multi-hop queries involve cross-entity or cross-document reasoning that depends on multiple evidence sources. The definition of query complexity corresponds to different levels of evidence demand and lays the foundation for subsequent classification and strategy selection.

The proposed three-level taxonomy is grounded in the structural requirements of evidence retrieval and reasoning depth: zero-hop queries rely on the internal parametric knowledge of the LLM; one-hop queries require a single piece of external evidence; and multi-hop queries necessitate cross-document reasoning. The discretisation process provides a deterministic mapping from a query’s information demand to an optimised retrieval strategy. This ensures a principled balance between computational efficiency and context sufficiency.

To automatically determine the category of query complexity, a classification model based on ModernBERT is trained. ModernBERT is an improved version of the BERT architecture [20,21], which enhances both representational capacity and computational efficiency, particularly suitable for modeling queries with large semantic spans. Given an input query q, the model first obtains a sentence-level representation vector $h_{\left[CLS\right]}$ through the encoder and maps it through a linear classification layer to a probability distribution over the three complexity categories:

$$p_c=soft\max {\left(W{h}_{\left[CLS\right]}+b\right)}_c$$

(3)

$$\widehat{c}=\arg \underset{c}{\max }{p}_c$$

(4)

$$\alpha =\underset{c}{\max }{p}_c$$

(5)

where $p_c$ denotes the predicted probability that q belongs to complexity category c; W and b are the parameters of the classification layer; $\widehat{c}$ is the predicted label corresponding to the category with the maximum probability; $\alpha$ is the probability of the corresponding category, which is used as the calibrated confidence after post-processing. During training, the classifier adopts Parameter-Efficient Fine-Tuning (PEFT), and in subsequent experiments, the classifier’s accuracy and confidence calibration results are reported.

2.2. Progressive Multi-Source Retrieval Strategy

In order to achieve a dynamic balance between answer quality and retrieval cost, a layered progressive multi-source retrieval scheduling mechanism is designed based on the classifier’s confidence output. The system is organised in such a way that three retrieval paths (Strategies A, B, and C) are defined with different retrieval scales and resource allocations. Strategy A is suitable for high-confidence scenarios (such as explicit zero-hop or one-hop queries), where the system employs only the optimal retriever. For zero-hop cases, the model can directly respond with internal knowledge to reduce computational overhead. Strategy B addresses medium-confidence scenarios where a single path may exhibit insufficient recall. The system therefore activates two complementary retrievers in parallel (e.g., dense retrieval and sparse keyword retrieval) to balance semantic coverage and precision, improving recall quality and robustness. Strategy C targets low-confidence or multi-hop queries, simultaneously invoking semantic, keyword, and structured graph retrieval. This multi-path redundancy enhances evidence coverage, providing robust support for complex reasoning.

These 3 strategies form a progressive transition from high-confidence streamlined retrieval to low-confidence multi-source coverage, enabling the system to adaptively schedule retrieval resources while maintaining answer quality.

To formalize the above strategies as an executable decision function, two confidence threshold parameters, ${\theta }_h,{\theta }_l\in \left(0,1\right)$ are introduced, where ${\theta }_l<{\theta }_h$. Based on the interval of the classification confidence $\alpha$, the query is mapped to the corresponding retrieval strategy:

$$S\left(\alpha ,c\right)=\left\{\begin{array}{c}A,\alpha \ge {\theta }_h\\ B,{\theta }_l\le \alpha <{\theta }_h\\ C,\alpha <{\theta }_l\end{array}\right.$$

(6)

Among these, c denotes the complexity label used to guide the specific selection and configuration of retrievers within the same strategy level. By properly setting ${\theta }_h$ and ${\theta }_l$, the higher threshold emphasizes precision, whereas the lower threshold ensures recall, enabling the system to balance retrieval coverage and accuracy. During implementation, thresholds are statistically determined based on the calibration set to optimize overall performance across most query scenarios.

To mitigate the impact of potential misclassifications, AMSRAG incorporates a confidence-aware buffer mechanism. The routing logic is described by the decision process in Equation (6), where the final selection is modulated by the calibrated confidence $\alpha$ rather than relying solely on the predicted label c. In scenarios where the classifier exhibits uncertainty, such as queries positioned at the boundary between one-hop and multi-hop levels, the system prioritizes a more comprehensive retrieval path. This design ensures that the framework remains robust to prediction noise, as path redundancy prevents information deficiency even in the presence of marginal classification errors.

2.3. Confidence-Aware RRF

Conventional RRF treats all retrieval sources equally, with a fixed smoothing constant k, and tends to favor documents ranked high across multiple retrievers. This approach fails to capture source heterogeneity and complexity characteristics, and it may reduce diversity among the fused results. The conventional RRF formula is expressed as:

$$RRF\left(d\right)={{\displaystyle \sum }}_{r\in RS}{\displaystyle \frac{1}{k+ran{k}_r\left(d\right)}}$$

(7)

where RS denotes the set of retrievers, $ran{k}_r\left(d\right)$ is the rank position of document d in retriever r, and k is the smoothing constant.

To address the limitations of conventional RRF in modeling source differences and adaptivity, this paper introduces Confidence-Aware Reciprocal Rank Fusion (CA-RRF) by incorporating both query complexity and confidence signals into the fusion stage. The improved fusion form is defined as:

$$CA-RRF\left(d\right)={{\displaystyle \sum }}_{r\in RS}{\displaystyle \frac{w_r\left(c,\alpha \right)}{k_r\left(c\right)+ran{k}_r\left(d\right)}}$$

(8)

where c represents the complexity category, and α is the calibrated confidence. The smoothing term $k_r\left(c\right)$ is adaptively set according to both the retrieval source scale and the query complexity: a smaller $k_r\left(c\right)$ in low-complexity scenarios emphasizes highly relevant results, while a larger $k_r\left(c\right)$ in high-complexity scenarios preserves more candidate documents. The weighting function $w_r\left(c,\alpha \right)$ employs confidence-based monotonic interpolation to balance stability and dominance across different confidence intervals:

$$w_r\left(c,\alpha \right)=\left(1-\rho \left(\alpha \right)\right){\displaystyle \frac{1}{\left|R_S\right|}}+\rho \left(\alpha \right){\beta }_r^{\left(c\right)}$$

(9)

$$\rho \left(\alpha \right)=\sigma \left(\kappa \left(\alpha -{\alpha }_0\right)\right)$$

(10)

Here, ${\beta }_r^{\left(c\right)}$ denotes the prior weight of retriever r under complexity c, satisfying ${{\displaystyle \sum }}_{r\in RS}{\beta }_r^{\left(c\right)}=1$, which characterizes the source-importance distribution across different complexity scenarios. $\sigma$ denotes the sigmoid function, and $\kappa$ controls its transition steepness. ${\alpha }_0$ is the neutral confidence threshold. $\kappa$ and ${\alpha }_0$ are treated as fixed design constants that determine the transition smoothness and neutral point in confidence-aware weighting; the corresponding values are reported in the experimental settings. $\kappa$ and ${\alpha }_0$ are fixed to stabilize the confidence-to-weight mapping and to reduce the risk of overfitting additional coupled hyperparameters given the limited calibration split. The mapping function $\rho \left(\alpha \right)\in \left(0,1\right)$ smoothly transforms the model-predicted confidence $\alpha$ into interpolation coefficients: when $\alpha$ is low, the fusion leans toward uniform weighting for stability; when $\alpha$ is high, prior weights are enhanced to highlight dominant retrieval sources.

To prevent the fusion results from being overly concentrated on a single source, MMR [15] is further introduced as a diversity constraint. When selecting the next document from the candidate set, the optimization objective is:

$$d^{\star }=\arg \underset{d\notin L}{\max }CA-RRF\left(d\right)-\lambda \underset{d^{\prime }\in L}{\max }Sim\left(d,d^{\prime }\right)$$

(11)

where L is the set of selected documents, $Sim\left(d,d^{\prime }\right)$ represents document–document similarity, and $\lambda \in \left[0,1\right]$ is the diversity control parameter balancing relevance and redundancy. In our implementation, $\lambda$ is fixed to a standard default value, and the specific setting is reported in the experimental settings. This constraint suppresses highly similar results while preserving overall relevance, thereby enhancing the diversity of content sources in the fused results.

By jointly incorporating complexity information, confidence signals, and diversity control into the fusion stage, CA-RRF achieves integration of adaptive multi-source evidence without requiring additional supervision, effectively improving the relevance and robustness of the fused results.

3. Experiment

This section presents the experimental protocol and analyses used to evaluate AMSRAG under heterogeneous query conditions. We first introduce the data preparation and annotation procedure, then report the evaluation and calibration results of the query complexity classifier, and finally present end-to-end QA performance along with additional analyses, including ablation studies and case studies.

3.1. Data Preparation and Annotation

3.1.1. Datasets

For data preparation, to evaluate the applicability of the proposed framework, two open-domain question answering datasets, MS MARCO and HotpotQA [22,23], are selected. The former is based on real search logs, primarily featuring factual questions covering zero-hop and single-hop scenarios. The latter emphasizes multi-hop reasoning across paragraphs, requiring the integration of evidence from multiple segments. Together, they cover diverse information needs ranging from simple factual queries to complex reasoning problems.

3.1.2. Complexity Annotation

For data annotation, this paper employs the Large Language Model (GPT-4) to assist in labeling query complexity and performs manual verification to improve label reliability [24]. The GPT-4 model is first provided with clear definitions and examples of zero-hop, one-hop, and multi-hop queries. It then independently reasons over each query and assigns complexity ratings. To reduce randomness in model judgments, a self-consistency strategy is adopted: multiple reasoning outputs are generated for the same query and voted upon [25]. If the majority results agree, the label is accepted; if significant disagreement occurs, the query is marked as “to be reviewed” and subsequently checked by humans. This strategy reduces manual workload while ensuring annotation reliability.

GPT-4 functions as a pre-labeling aid within the fixed hop-based taxonomy of query complexity, categorised into zero-hop, one-hop, and multi-hop. In order to monitor the quality of the annotations, a lightweight auditing protocol is applied. This involves the random selection of 50 queries from every 1000 annotated queries for human verification by two to three reviewers. Discrepancies are resolved through majority adjudication. In the subset that was audited, approximately 8% of queries were the subject of disagreement between reviewers. Across a dataset comprising over 25,000 queries, the estimated human override rate of model-generated labels is approximately 1.0% to 1.5%, corresponding to approximately 10 to 15 incorrect labels per 1000 queries. The complexity classifier is trained on the human-audited labels defined by this taxonomy rather than on raw model outputs. This is intended to reduce potential bias introduced by LLM-assisted pre-labeling.

3.1.3. Data Splits and Statistics

The annotated queries for complexity classification are divided into three distinct splits: training, calibration, and test. These splits are utilised for specific purposes, namely the training of the complexity classifier, the calibration and tuning of confidence-related hyperparameters (including the selection of thresholds), and the reporting of classifier performance, respectively. Concurrently, an end-to-end QA evaluation set is constructed for the purpose of comparing AMSRAG with baselines and conducting ablation studies under the same evaluation protocol. The following section provides a synopsis of the dataset statistics and complexity-class distributions that have been utilised in the present study. Please refer to

Table 1. Dataset statistics and complexity-class distributions used in this study.

Usage	Split/Set	Queries	Zero-Hop (25%)	One-Hop (40%)	Multi-Hop (35%)
Complexity classifier	Train (80%)	18,991	4748	7596	6647
	Calibration (10%)	2374	594	950	830
	Test (10%)	2374	594	950	830
AMSRAG	Evaluation	300	75	120	105

Note: The calibration split is used for confidence calibration and threshold search (${\theta }_h$, ${\theta }_l$). The end-to-end QA evaluation set is separate from the classifier splits and is used for baseline comparison and ablation experiments unless otherwise stated. The end-to-end corpus contains 1892 documents, and the 300 queries are drawn from MS MARCO (162) and HotpotQA (138).

for further information.

3.1.4. Hyperparameter Settings

All routing and fusion hyperparameters are determined on the calibration split and then frozen for all reported results. We select ${\theta }_h$ and ${\theta }_l$ once on the calibration split with ${\theta }_h<{\theta }_l$, while $\kappa$ ${\alpha }_0$ and $\lambda$ are fixed design defaults. We performed a lightweight grid search for the routing thresholds on the calibration split. Specifically, ${\theta }_h\in \{0.70,0.75,0.80,0.85,0.90\}$ and ${\theta }_l\in \{0.50,0.55,0.60,0.65,0.70\}$ with the constraint ${\theta }_l<{\theta }_h$, yielding 15 feasible pairs, and selected $({\theta }_h,{\theta }_l)=(0.90,0.60)$.

Table 2. Hyperparameter settings for routing and fusion.

Hyperparameter	Value	Selection Protocol
${\theta }_h$	0.90	selected once on the calibration split
${\theta }_l$	0.60	selected once on the calibration split
$\kappa$	2.0	fixed design constant
${\alpha }_0$	0.5	fixed design constant
$\lambda$	0.5	fixed default (MMR)

reports the final settings.

3.2. Evaluation of Query Complexity Classifier

To evaluate the discriminative ability and confidence reliability of the proposed ModernBERT-based query complexity classifier on zero-hop, one-hop, and multi-hop queries, two types of baselines are introduced: random and heuristic rule-based classifiers as lower-bound references, and BERT and RoBERTa variants as model-based references [20,26]. To ensure fair comparisons, all methods use consistent data splits and evaluation procedures. All experiments are conducted on an Ubuntu 22.04 system using Python 3.12 and PyTorch 2.6.0, with CUDA 12.4, running on a single NVIDIA RTX 4090 GPU with 24 GB memory.

3.2.1. Evaluation Metrics and Composite Measure

Accuracy and Macro-F1 are used to assess overall correctness and class balance. The Expected Calibration Error (ECE) quantifies the reliability of predicted probabilities. Additionally, a Reliability-Adjusted Accuracy (RAA) metric is defined to jointly consider accuracy and calibration:

$$RAA=Accuracy-\lambda ECE$$

(12)

where Accuracy denotes overall classification accuracy, ECE represents the expected calibration error, and $\lambda$ is the penalty coefficient. In this experiment, $\lambda =1$, balancing the contributions of Accuracy and ECE. A higher RAA indicates better overall performance in both accuracy and confidence reliability.

As shown in

Table 3. Performance comparison of query complexity classification models.

Model	Accuracy (%) ↑	Macro-F1 (%) ↑	ECE (%) ↓	RAA (%) ↑
Random	32.4	32.2	19.8	12.6
Rule-based	46.6	47.2	27.2	19.3
BERT	84.7	84.2	7.4	77.2
RoBERTa	84.7	84.6	4.1	80.5
ModernBERT (Ours)	85.9	85.4	1.9	84.0

, the ModernBERT classifier achieves the best performance in Accuracy, Macro-F1, and ECE, with RoBERTa following closely. Overall, pretrained fine-tuned models significantly outperform heuristic and random baselines, indicating that simple rules or random partitioning fail to capture complexity-level differences effectively.

As demonstrated in Figure 2, the confusion matrix provides a more detailed insight into the classifier’s performance across categories. The multi-hop class demonstrated the highest recognition accuracy of 94.4%, followed by the one-hop class with 83.8%. In contrast, the zero-hop queries exhibited relatively lower distinguishability, with an accuracy of 77.2%. This finding suggests that the model can reliably utilise semantic span and structural cues in multi-hop reasoning yet encounters difficulty differentiating between zero-hop and one-hop queries due to their comparable surface expressions.

Analysis of the confusion matrix in Figure 2 reveals that the system is primarily characterized by ‘upward’ misclassifications, such as the categorization of zero-hop queries as one-hop. While such errors marginally increase latency due to redundant retrieval, they do not compromise the final answer quality. Conversely, ‘downward’ errors that could potentially lead to information loss are significantly less frequent. This asymmetric error profile, combined with the low Expected Calibration Error (ECE) of 1.9% reported in Section 3.2.3, demonstrates that the downstream routing and fusion processes remain stable and effective despite minor classification fluctuations.

From an efficiency perspective, such confusion may route a subset of true zero-hop queries to the one-hop branch, thereby triggering unnecessary retrieval. However, compared with fixed pipelines, correctly identified zero-hop queries still skip retrieval entirely. This helps preserve the overall efficiency gains.

3.2.2. Results and Discussion

By observing misclassifications, it was found that multi-hop questions are mainly confused with one-hop ones (5.6%), reflecting the continuity of information demand between complex reasoning and single-evidence queries. Zero-hop questions are more often misclassified as one-hop (22.5%), showing that the boundary between them is inherently fuzzy and represents the most difficult region for classification.

3.2.3. Calibration Effect

To ensure that the classifier’s output confidence can serve as a reliable decision basis for downstream routing and fusion, a temperature-based probability calibration is applied. The temperature parameter is fitted on the calibration split by minimizing the negative log-likelihood with respect to the human-audited labels. Expected Calibration Error is computed using fifteen fixed-width confidence bins over the interval from 0 to 1. Figure 3 illustrates the reliability curves before and after calibration.

Before calibration, the Expected Calibration Error (ECE) is approximately 7.7%, indicating an overconfidence issue. After applying Temperature Scaling, ECE decreases significantly to about 1.9%, while the classification accuracy and Macro-F1 remain largely unchanged. After calibration, the predicted confidence aligns more closely with the empirical accuracy, confirming the reliability of the confidence signal. Because calibration is applied to the classifier probabilities, it is decoupled from the downstream retriever pool and generator choice. The temperature can be re-estimated on a held-out calibration split when the deployment setting changes.

3.3. End-to-End Experiments

After evaluating and calibrating the upstream classifier, the proposed “Complexity-Aware–Progressive Retrieval–Confidence-Aware Fusion” framework is applied to open-domain QA tasks for end-to-end evaluation. For answer generation, we adopt Qwen-Turbo as the large language model, selected to balance answer quality and computational cost across all end-to-end experiments. The fixed-pipeline RAG employs static retrieval and fusion (global top-k and fixed RRF weights), while AMSRAG conducts adaptive retrieval and dynamic fusion parameter adjustment via CA-RRF based on the combined complexity–confidence signals. The complete process forms a closed loop of complexity prediction, progressive retrieval, confidence-aware fusion, and answer generation.

Considering both output quality and system efficiency, 4 indicators are adopted: Exact Match (EM) and token-level F1 for answer accuracy and consistency, nDCG@5 for retrieval relevance ranking, and retrieval latency (ms) measured as the average time from query trigger to fused retrieval result [27,28].

As demonstrated in

Table 4. End-to-end results on open-domain QA (95% bootstrap CIs in brackets).

Framework	EM (%)	F1 (%)	nDCG@5	Retrieval Latency (ms)
Fixed-pipeline RAG	47.1 [44.3, 49.9]	60.3 [57.8, 62.8]	0.802 [0.779, 0.825]	460
Self-RAG	49.6 [46.8, 52.4]	62.4 [60.0, 64.8]	0.835 [0.813, 0.857]	540
Adaptive-RAG	50.1 [47.3, 52.9]	62.9 [60.5, 65.3]	0.838 [0.816, 0.860]	505
AMSRAG (Ours)	50.9 [48.1, 53.7]	63.5 [61.2, 65.8]	0.846 [0.825, 0.867]	520

Note: Confidence intervals are estimated via non-parametric bootstrap over evaluation queries (B = 1000; percentile method).

, the adaptive frameworks typically enhance EM, F1, and nDCG@5 in comparison to the fixed-pipeline baseline, accompanied by a modest augmentation in retrieval latency that remains within a feasible range for open-domain QA. Across the evaluated settings, AMSRAG demonstrates competitive performance relative to the other adaptive baselines (EM 50.9, F1 63.5, nDCG@5 0.846), with a moderate latency increase relative to the fixed pipeline (520 ms vs. 460 ms). The findings in EM and F1 indicate that more precise responses are obtained on average. Concurrently, the elevated nDCG@5 is indicative of the intended consequence of confidence-aware fusion, namely the enhancement of the quality of top-ranked evidence across sources. The observed latency overhead can be attributed to adaptive scheduling, which activates additional retrieval for lower-confidence or higher-complexity queries. This is in addition to the extra computation introduced by multi-source fusion (CA-RRF) and diversity-aware reranking (MMR). In order to assess the statistical reliability of the data, AMSRAG improves EM over the fixed-pipeline baseline by +3.8%, with a 95% confidence interval of [1.2%, 6.4%] and p < 0.01 under a paired bootstrap test on per-query EM differences.

Using the same measurement protocol as Table 4 (from query trigger to the fused retrieval result), profiling shows that AMSRAG spends 25 ms on complexity classification (ModernBERT + calibration), 10 ms on CA-RRF, and 15 ms on MMR reranking per query on average. The remaining time is dominated by retrieval execution (including any associated I/O) under the selected strategy, which also accounts for the small residual difference in end-to-end latency relative to the fixed pipeline.

We vary the routing thresholds $({\theta }_h,{\theta }_l)$ around the calibrated setting (0.90, 0.60) while keeping all other components fixed. As shown in

Table 5. Sensitivity to routing thresholds.

${\mathit{\theta }}_{\mathit{h}}/{\mathit{\theta }}_{\mathit{l}}$	EM (%)	F1 (%)	Latency (ms)
0.85/0.55	50.3 [47.5, 53.1]	62.8 [60.5, 65.1]	492
0.90/0.60	50.9 [48.1, 53.7]	63.5 [61.2, 65.8]	520
0.92/0.62	50.7 [47.9, 53.5]	63.2 [60.9, 65.5]	538
0.95/0.65	49.9 [47.1, 52.7]	62.3 [60.0, 64.6]	565

, the observed differences in EM/F1 across the tested threshold pairs are small and the 95% confidence intervals overlap substantially, while retrieval latency increases with stricter thresholds.

3.4. Ablation Study

The present section is concerned with the quantification of the contributions of the principal components in AMSRAG, via a controlled ablation study. It is evident that the predicted complexity and calibrated confidence jointly control both the process of routing and the process of fusion. In order to address this, the present study employs signal-level ablations that disable or fix how the decision signal is applied. This is achieved while ensuring that the remainder of the pipeline remains unaltered.

All variants are evaluated in accordance with the same protocol as outlined in Table 4. They share the same retriever pool, index/corpus, prompt template, decoding configuration, and answer generator. The measurement of retrieval latency is consistently defined in Table 4. Fixed-routing variants employ the non-adaptive policy of the fixed pipeline, whereas adaptive-routing variants utilise the same progressive retrieval thresholds as in the primary experiments.

The following five variants were evaluated: In the context of vanilla RRF, A0 denotes the fixed baseline. In the context of fixed retrieval, A1 substitutes RRF for CA-RRF. A2 facilitates adaptive progressive retrieval with vanilla RRF. A3 facilitates adaptive retrieval with CA-RRF whilst simultaneously disabling MMR. A4 is the full AMSRAG system with progressive retrieval, CA-RRF, and MMR.

The component-wise effects are obtained through the implementation of paired comparisons. The process of routing is measured by the comparison of A2 with A0, fusion by the comparison of A1 with A0 and A3 with A2, and diversification by the comparison of A4 with A3. The results of the ablation process are presented in

Table 6. Ablation results of AMSRAG components.

Variant	Routing	Fusion	MMR	EM	F1	nDCG@5	Retrieval Latency (ms)
A0	Fixed	RRF	Off	47.1	60.3	0.802	460
A1	Fixed	CA-RRF	Off	48.2	61.1	0.824	466
A2	Adaptive	RRF	Off	49.6	62.3	0.816	505
A3	Adaptive	CA-RRF	Off	50.5	63.1	0.842	512
A4	Adaptive	CA-RRF	On	50.9	63.5	0.846	520

.

3.5. Case Study

In order to provide further elucidation regarding the adaptive scheduling and fusion mechanisms of the AMSRAG framework when faced with different query types, this section will present three test cases that are representative of the framework’s functionality. The first of these is a high-confidence zero-hop query, the second is a low-confidence multi-hop query, and the third is a mis-classified-induced failure scenario, as illustrated in Figure 4.

In situations where confidence is high and the direct path is known (i.e., zero-hop cases), the system has been shown to accurately identify low complexity and respond directly using internal knowledge without the need for external retrieval. This suggests an enhancement in efficiency and precision. The process of avoiding redundant retrieval is achieved through the utilisation of complexity awareness and confidence estimation, thereby ensuring the maintenance of correctness.

In instances of low confidence pertaining to multi-hop cases, the system is able to detect multi-entity reasoning requirements and automatically trigger multi-source parallel retrieval. Various retrievers (sparse, dense, and graph-based) collectively contribute to evidence aggregation, thereby producing more complete and coherent answers. This finding suggests that the framework is effective in leveraging confidence-driven multi-source fusion for complex reasoning tasks.

Conversely, in instances of failure attributable to complexity misclassification, the system erroneously identifies a multi-hop question as one-hop, resulting in the activation of inadequate retrieval paths and the generation of incomplete answers. This observation indicates that classification accuracy and confidence thresholds have a significant impact on overall performance, as errors at the front end can propagate and amplify downstream.

Beyond misclassification failures, AMSRAG may also fail in two cases: high-confidence but wrong routing under distribution shift, which can reduce evidence coverage; and fusion imbalance when one retriever dominates, suppressing complementary sources. Near-duplicate passages can further worsen this via reranking. Simple safeguards such as confidence-aware fallback or cross-source coverage constraints may improve robustness.

The framework demonstrates the capacity to balance retrieval efficiency and answer quality; however, further optimisation of complexity boundary recognition and dynamic threshold adjustment is required to enhance stability and generalisation in real-world scenarios.

4. Conclusions and Future Work

This paper proposed AMSRAG, an Adaptive Multi-Source Retrieval-Augmented Generation framework designed to improve adaptability under heterogeneous query conditions in retrieval-augmented generation. The framework can adaptively select the appropriate retrieval path according to the query’s complexity and confidence, improving both retrieval efficiency and answer quality.

Specifically, AMSRAG establishes a unified scheduling mechanism through complexity classification and confidence calibration, enabling dynamic selection of retrieval paths and fusion strategies. This design overcomes the rigidity of fixed “retrieval-generation” pipelines.

Moreover, to balance retrieval efficiency and evidence coverage across different complexity levels, a progressive multi-source retrieval strategy is proposed. The framework suppresses redundant retrieval in high-confidence scenarios to enhance efficiency, while expanding recall under low-confidence conditions to ensure evidence completeness.

In addition, a CA-RRF method is introduced to fully exploit complexity and confidence signals during fusion. By leveraging complexity and confidence signals with MMR-based diversity control, the fusion results achieve improved relevance and robustness.

The experimental results demonstrate that AMSRAG exhibits consistent superiority over fixed-pipeline RAG systems in open-domain question-answering tasks, enhancing both the quality of the answers provided and the relevance of the retrieved information. In scenarios where confidence is high, the retrieval process is rendered redundant, thereby leading to an optimised trade-off between the quality of the response and its efficiency.

Despite these promising results, several aspects still warrant further exploration. The identification of complexity-boundary cases and the flexibility of threshold strategies require refinement. Additionally, constrained by the types of retrievers and dataset scales employed, the generalization ability of AMSRAG in cross-domain and multimodal scenarios remains to be validated, including the stability of calibration and routing signals under distribution shifts and component changes. Future research may explore dynamic threshold optimization and framework extension to enhance overall robustness and universality.

Although AMSRAG is validated on open-domain QA benchmarks (MS MARCO and HotpotQA), deployment in specialized domains such as medical, legal, or technical QA may require adaptation. The indexed corpus and retriever pool should be replaced with in-domain resources, and calibration parameters and routing thresholds should be re-tuned under distribution shift. In high-stakes settings, stricter grounding and evidence validation may be needed to reduce harmful hallucinations.