An LLM-Based Multi-Path Question Answering System with XGBoost Routing and Threshold-Based Refusal

Abstract

In conventional question answering systems, general-purpose large language models (LLMs), despite their strong capabilities in language understanding and generation, exhibit notable limitations in scenarios with stringent factuality requirements. Their outputs often lack explicit evidential grounding, making them prone to hallucinations and inconsistent responses. Moreover, LLMs do not inherently guarantee determinism when performing operations over structured data—such as aggregation, conditional filtering, and cross-field constraints—thereby undermining result consistency and reliability. To address these issues, we propose an internal-data-first framework for controllable question answering in high-risk scenarios. The framework categorizes knowledge sources into unstructured documents and structured data, enabling evidence-constrained generation via retrieval-augmented generation (RAG) and database-backed, verifiable query execution via restricted, read-only structured queries. In addition, an UNK branch is introduced as a safe degradation mechanism that triggers refusal when inputs lack sufficient evidence, exceed system capability boundaries, or fail to meet confidence requirements, thereby suppressing hallucinations and unauthorized generation. To enable controlled selection among the two execution pathways (RAG/SQL) and the safety degradation branch (UNK) at the system level, we design a learned router based on XGBoost with confidence-thresholded selective prediction, which preferentially activates UNK refusals for low-confidence or out-of-distribution inputs. We validate the proposed framework using a graduate admissions consultation system as an exemplar application, constructing both a document knowledge base and structured score tables, and conducting controlled comparisons across multiple system variants with multi-metric evaluations. Experimental results indicate that, under the current controlled evaluation setting, the proposed framework exhibits relatively stable behavior under complex query formulations and demonstrates practical engineering potential in high-risk vertical-domain question answering scenarios.

1. Introduction

In recent years, large language models (LLMs) have substantially advanced the language understanding and generation capabilities of question answering (QA) systems [1]. However, in vertical-domain QA, the evaluation focus has gradually shifted from generation fluencyto answer reliability and behavior controllability. Driven by requirements on data security, regulatory compliance, and consistency of institutional knowledge, vertical-domain QA systems are typically deployed in controlled environments and rely primarily on governable internal data assets, including internal document repositories and business databases. In such settings, existing solutions commonly adopt retrieval-augmented generation (RAG) to support document-centric explanatory questions by retrieving relevant internal materials as contextual evidence. Meanwhile, for tasks involving explicit structured data assets, systems often incorporate restricted read-only SQL capabilities to support verifiable query needs such as numerical aggregation and field-level retrieval, thereby improving the stability, consistency, and auditability of fact-oriented answers [2,3].

Despite their widespread adoption, vertical-domain QA systems still face several critical challenges in practical deployment. First, answer generation may still partially rely on parametric memory and subjective completion of LLMs, leading to inconsistent conclusions and limited verifiability of supporting evidence, which in turn increases the risk of over-assertion. Second, vertical-domain knowledge assets are typically highly heterogeneous, encompassing both unstructured materials (e.g., PDF and Word documents) and structured data (e.g., tables and field records). Without explicit differentiation and governance of knowledge modalities, mixing heterogeneous content into a single knowledge base or execution pathway can easily introduce retrieval noise, inconsistent granularity, and mismatched processing chains, ultimately degrading end-to-end system stability. Third, the selection between execution pathways such as RAG and restricted read-only SQL is often unstable: semantically similar questions expressed in different forms may trigger different pipelines, resulting in misrouting and output variance. This issue becomes particularly severe for boundary cases or low-quality inputs. In the absence of confidence-aware gating and a unified degradation strategy, systems may continue execution and output seemingly definitive answers even when evidence is insufficient, thereby amplifying the risks of tool misuse and hallucinated generation. These challenges highlight the urgent need for a systematic mechanism that explicitly delineates factual boundaries, distinguishes knowledge modalities, and enables stable decision-making and risk gating prior to execution in vertical-domain QA systems [4].

To address the above challenges, this work adopts an internal-data-first design paradigm at the system level, constraining answerable facts to internal document repositories and structured databases. Under this premise, the system further distinguishes execution boundaries for different types of queries. In this setting, the system is required not only to determine whether a query can be answered reliably, but also to decide, prior to execution, whether it should enter the document-grounded RAG pathway, the restricted read-only SQL pathway, or a conservative refusal pathway.

Accordingly, multi-tool invocation is organized as a “decision-before-execution” multi-path orchestration process. Specifically, an entry-side routing module first determines the appropriate pathway for each query, followed by confidence-threshold gating to filter low-confidence inputs, thereby enabling explicit control over tool usage boundaries and risk screening prior to execution. Compared with multi-tool orchestration methods that emphasize flexible tool use in open environments, this work places greater emphasis on execution-boundary control and selective risk management in controlled vertical-domain scenarios.

Existing approaches typically rely on large language models to complete tool selection implicitly during reasoning or generation. In contrast, this work focuses more explicitly on execution-boundary constraints in high-risk vertical domains. Specifically, pathway selection is reformulated as an explicit routing problem prior to execution, and risk filtering is performed through confidence-based gating before execution. Meanwhile, UNK is modeled as a first-class routing category, rather than being treated merely as a post hoc fallback after execution failure. Therefore, the primary goal of this work is not to extend general-purpose agent capabilities in open environments, but rather to improve controllability and risk management in high-risk vertical domains through explicit pathway decision-making, restricted execution, and selective refusal. Based on this design, we further formalize pathway boundaries and restricted execution policies, and establish multi-path baselines together with reproducible evaluation protocols, thereby providing a reference for system design and evaluation in similar controlled scenarios.

The main contributions of this work are summarized as follows:

1.

For question answering in high-risk vertical domains, we propose an internal-data-first, “decision-before-execution” multi-path orchestration framework that constrains answerable facts to internal document repositories and structured databases, and explicitly defines answer boundaries, execution boundaries, and refusal boundaries.

2.

Under this framework, we design an entry-side explicit routing mechanism based on XGBoost, which unifies RAG, restricted read-only SQL, and UNK within a single decision space, and enables pre-execution refusal gating for low-confidence inputs through a confidence threshold.

3.

We construct a hybrid question answering dataset covering RAG, SQL, and UNK intents and validate the effectiveness of the proposed framework in controlled vertical-domain scenarios through automatic evaluation, human evaluation, and supplementary cross-scenario experiments.

2. Related Work

2.1. Large Language Models

LLMs have substantially advanced natural language understanding and generation through large-scale pretraining and alignment, making natural language a unified interaction interface for QA systems. Recent generations of models, exemplified by GPT-4, demonstrate marked improvements across a wide range of professional and academic benchmarks, strengthening their general capabilities in complex query understanding and structured generation, and thereby providing a more powerful foundation for vertical-domain QA [1]. To further improve instruction following and alignment with user intent and output preferences, instruction tuning and reinforcement learning from human feedback (RLHF) have become key alignment techniques. Representative works such as InstructGPT show that human-feedback-based alignment can significantly enhance instruction adherence and output quality while also improving factuality-related performance to some extent [5]. More recent surveys systematically summarize data construction strategies, training paradigms, and evaluation protocols for instruction tuning, offering methodological guidance for adapting LLMs to vertical-domain applications [6].

However, vertical-domain QA typically demands consistent answers, verifiable evidence, and enforceable boundaries, rendering generation fluency a secondary evaluation criterion. Prior studies have shown that LLMs may still produce unsupported content (i.e., hallucinations) under insufficient evidence or ambiguous boundaries. The underlying causes span multiple stages, including data collection, model training, and inference, and have motivated extensive follow-up research on reliability evaluation and risk governance [4]. Consequently, vertical-domain scenarios often require combining generative capabilities with governable data sources, together with system-level execution constraints and risk control mechanisms, to suppress overconfident answering under low-certainty conditions.

2.2. Retrieval-Augmented Generation

To reduce LLM reliance on parametric knowledge and improve evidence-grounded answering, a line of research has focused on retrieval-augmented generation (RAG), which introduces external, updatable corpora as non-parametric memory to anchor generation in retrieved evidence. Seminal RAG work integrates dense retrievers with generative models and proposes a unified framework combining parametric and non-parametric memory, demonstrating substantial gains in factual generation on knowledge-intensive tasks [2]. Subsequent approaches, such as REALM, incorporate retrieval directly into the pretraining stage, enabling models to leverage retrieval modules consistently during pretraining, fine-tuning, and inference, thereby improving modularity and knowledge updateability [7]. Atlas and related models further validate the robustness of retrieval-augmented approaches in few-shot settings and highlight the engineering advantages of updatable indexes for system maintainability [8]. In recent years, research on “RAG + LLM” has continued to expand. Related reviews summarize key issues such as retrieval, fusion, generation, and evaluation from the perspective of system components and architectural evolution while discussing advances in context compression, robustness, and evaluation benchmarks [3,9,10,11,12]. Concurrently, comprehensive benchmarks like CRAG provide a more systematic measurement framework for comparative assessments of RAG systems in complex knowledge tasks [13].

Despite improved evidence grounding, the end-to-end performance of RAG systems remains highly dependent on retrieval quality and knowledge base governance. When retrieval noise is high, evidence is insufficient, or the question type mismatches the evidence modality, systems may still exhibit output variance and unstable answers [3,9]. More importantly, RAG does not inherently address the explicit decision problem of when to perform retrieval-based generation, when to execute structured queries, and when to refuse or degrade. In particular, continuing to apply retrieval-based generation to structured factual queries may introduce unnecessary generation errors and interpretation overhead. These limitations indicate that vertical-domain QA requires a more systematic mechanism that, beyond evidence grounding, explicitly defines pathway boundaries, performs pre-execution routing decisions, and applies risk gating under low-confidence conditions to support stable and reproducible deployment.

2.3. Multi-Tool Orchestration and Agent-Style Dispatching

In recent years, tool use and multi-tool orchestration around large language models have become an important research direction. Representative methods such as ReAct interleave reasoning, action, and observation during generation, enabling the model to dynamically decide whether to invoke external tools based on intermediate reasoning results, thereby improving interactive problem-solving ability in complex tasks. Methods such as Self-Ask adopt intermediate sub-question decomposition to realize lightweight tool invocation, thereby enhancing information acquisition in multi-step reasoning scenarios. More broadly, works such as Gorilla and ToolLLM target wider settings of tool use or API invocation, emphasizing the ability of large language models to call external tool collections, generalize across tools, and adapt to different tasks. Overall, such approaches typically rely on implicit decision-making during generation to complete tool selection, with a primary focus on improving flexible tool use and complex task solving in open environments.

However, high-risk vertical-domain question answering differs substantially from the open-ended tool-use scenarios described above. In settings such as admissions consultation, healthcare, and finance, the system is required not only to invoke tools, but also to determine prior to execution whether a query should be executed at all, which pathway should be used, and when refusal is more appropriate. If pathway selection is delegated entirely to implicit judgments during generation, boundary cases, low-confidence inputs, or evidence-insufficient queries may easily lead to inappropriate tool invocation and overreaching responses.

Compared with existing multi-tool agent-style dispatching frameworks, high-risk vertical-domain question answering places stricter requirements on execution-boundary control and pre-execution decision-making. These differences suggest the need for more explicit pathway-selection mechanisms and more conservative handling of uncertain cases, rather than relying entirely on implicit tool decisions during generation.

2.4. XGBoost for Learned Routing

When a system is equipped with multiple execution capabilities—such as retrieval-based generation and structured querying—the problem of pathway selection can be formulated as a supervised classification task, where input query features are used to predict the most appropriate execution path, thereby improving overall system stability and controllability. Gradient Boosted Decision Trees (GBDTs) are widely adopted in industrial classification tasks due to their strong ability to model nonlinear decision boundaries, compatibility with heterogeneous features, and low inference latency. XGBoost, as an efficient and scalable implementation of GBDT, incorporates second-order optimization, explicit regularization, sparsity-aware splitting, and approximate quantile sketching. These design choices enable strong training efficiency and generalization performance in large-scale and sparse-feature settings, making XGBoost a widely used and competitive baseline in practice [14]. In multi-path systems, XGBoost can naturally output class probabilities and interface with threshold-based strategies, providing a direct mechanism for confidence-aware gating and safe degradation under low-certainty conditions, thereby reducing cascading risks caused by misrouting.

It is worth noting that the stability of learned routing depends not only on the classifier itself, but also on feature construction, probability calibration, and distribution shift. As a result, practical systems typically require complementary threshold calibration, unified degradation policies, and reproducible evaluation protocols to systematically validate routing performance and gating behavior.

3. System Design

3.1. Overall Framework

We propose an internal-data-first, multi-path intelligent question answering framework for scenarios involving hybrid knowledge modalities. As illustrated in Figure 1, the framework follows a decision-before-execution orchestration paradigm [15,16,17] and decomposes the system into two main components: a decision layer and an execution layer. Specifically, Stage I and Stage II correspond to the entry-side decision process, while Stage III corresponds to pathway execution. The decision layer is responsible for pathway selection and risk gating, whereas the execution layer performs retrieval, querying, and answer generation within strictly controlled capability boundaries. This design enables structured facts to be handled through restricted read-only queries with improved consistency and verifiability, unstructured knowledge to be generated under explicit evidence constraints, and conservative refusal strategies to be adopted under uncertainty or out-of-scope conditions.

Figure 1. System architecture overview.

At the decision layer, an incoming user query is first subjected to unified preprocessing and representation construction, and then fed into a learned router for three-way classification (Stage I: learned routing with confidence screening). The router outputs a pathway decision among RAG, restricted read-only SQL, and UNK (Stage II: pathway determination). In this work, XGBoost is employed as the routing decision model, producing a probability distribution over the three candidate pathways. To suppress misrouting and overconfident answering under low certainty, the decision layer further introduces a confidence threshold $\tau$ to enable selective prediction. When the maximum class probability falls below $\tau$, the system does not proceed to the RAG or SQL execution pipelines; instead, it directly routes the query to the UNK branch, returning a conservative refusal or guidance response. Only when the confidence requirement is satisfied is the corresponding execution pathway activated. Through this XGBoost-based routing with $\tau$-gated control, pathway selection and risk handling are explicitly front-loaded, providing a stable and controllable entry point for subsequent execution and reducing erroneous operations under low-confidence inputs.

At the execution layer, the system performs controlled execution according to the pathway decision produced at Stage II (Stage III: internal-data-first execution). When the selected pathway is RAG, the system retrieves relevant evidence fragments from the local document knowledge base and generates answers under explicit evidence constraints. When the pathway is SQL, the system executes restricted read-only queries over structured data tables to perform filtering, aggregation, and constraint-based computation, and then generates response formulations based on the database execution results, thereby improving the consistency and verifiability of fact-oriented answers. When the pathway is UNK, the system outputs a unified refusal or guidance response to cover low-confidence, unverifiable, or out-of-scope queries. By adopting a three-path execution mechanism under a unified entry point, the framework enables differentiated processing for queries over heterogeneous knowledge modalities, while ensuring that risks associated with uncertain inputs are mitigated prior to execution.

3.2. XGBoost-Based Routing and Threshold-Based Gating

This section describes the design of the decision layer at the system entry. Unlike conventional QA components that directly generate answers, the decision layer is responsible for producing executable routing decisions. Specifically, it routes each input query to one of three execution pathways—RAG, restricted read-only SQL, or UNK—and activates conservative gating under uncertainty to prevent inappropriate tool invocation and low-confidence forced answering. Overall, the decision layer realizes a “learned routing + threshold gating” mechanism that enforces the decision-before-execution principle, ensuring that retrieval or structured querying is triggered only when controllable conditions are satisfied.

We formulate pathway selection as a supervised multi-class classification problem with the label space:

$$\mathcal{R}=\{\mathrm{RAG},\mathrm{SQL},\mathrm{UNK}\}.$$

Given an input query q, we first encode it using a sentence embedding model (bge-small-zh-v1.5), mapping the query into a 512-dimensional dense vector, followed by normalization to obtain a stable feature scale. Based on labeled routing data, an XGBoost-based multi-class classifier is trained using a softmax cross-entropy objective to produce a posterior probability distribution over the three candidate pathways:

$$\begin{array}{cc}\hfill p\left(q\right)& =\left[p(\mathrm{RAG}\mid q),p(\mathrm{SQL}\mid q),p(\mathrm{UNK}\mid q)\right],\hfill \\ \hfill & \sum _{r\in \mathcal{R}}p(r\mid q)=1.\hfill \end{array}$$

(1)

At inference time, the pathway with the maximum posterior probability is selected as the candidate route:

$$\widehat{r}=arg\underset{r\in \mathcal{R}}{max}p(r\mid q),$$

(2)

where $p(r\mid q)$ denotes the router’s confidence that query q belongs to pathway r, and $\widehat{r}$ represents the predicted execution route prior to threshold gating. The $argmax$ rule provides a deterministic candidate decision and serves as the basis for subsequent confidence-based control.

A key design choice in our framework is adopting three-way classification rather than binary RAG/SQL routing. Explicitly modeling UNK as a first-class category allows inputs that are unsafe, unsupported, or ambiguous to be handled conservatively. When information is missing, semantic boundaries are unclear, or the query falls outside the system’s coverage, the router can assign higher probability to UNK, preventing the system from being forced into retrieval or querying pipelines and thereby reducing the risk of invalid retrieval, failed queries, or unsupported generation.

On top of routing prediction, we further introduce a selective prediction mechanism with confidence gating. We define the routing confidence as the maximum posterior probability:

$$p_{max}\left(q\right)=\underset{r\in \mathcal{R}}{max}p(r\mid q).$$

(3)

If $p_{max}\left(q\right)<\tau$, where $\tau$ is a predefined confidence threshold, the system directly activates the UNK pathway and outputs a unified refusal or guidance response. Only when $p_{max}\left(q\right)\ge \tau$ is the candidate route $\widehat{r}$ accepted and passed to the execution layer. This threshold-based gating strategy introduces minimal additional complexity while enabling early rejection of low-confidence inputs and providing a tunable trade-off between coverage and risk control.

The threshold $\tau$ is selected on a validation set via grid search by jointly evaluating routing performance and refusal behavior. Macro-F1 is used as the primary optimization metric, while the activation rate of UNK serves as a constraint reflecting system conservativeness. Intuitively, smaller $\tau$ values increase coverage but may amplify misrouting risk, whereas larger $\tau$ values enforce more conservative behavior at the cost of higher refusal rates. The final $\tau$ is fixed for all subsequent end-to-end evaluations and comparative experiments.

3.3. Internal-Data-First Multi-Path Execution and Safe Degradation

The execution layer operates strictly according to the final routing decision produced by the decision layer and performs retrieval, querying, or response generation within predefined capability boundaries. To avoid unsupported generative inference over structured facts and ungrounded interpretation of unstructured knowledge, the system adopts an internal-data-first execution mechanism with three mutually exclusive pathways: RAG, restricted read-only SQL, and UNK. Each query is committed to exactly one pathway prior to execution, and no cross-path switching is allowed during execution. If the selected pathway fails to produce verifiable results, the system degrades to UNK rather than forcing an answer.

RAG Pathway. When the selected pathway is RAG, the execution layer addresses explanatory and non-structured queries by retrieving evidence from a local document corpus $\mathcal{D}={\left\{d_i\right\}}_{i=1}^N$. Semantic retrieval is performed to identify the top-k evidence fragments most relevant to the query:

$${\mathcal{E}}_k\left(q\right)=\mathrm{Top}\text{-}{k}_{d_i\in \mathcal{D}}cos\left(f\left(q\right),f\left(d_i\right)\right),$$

(4)

where $f(·)$ denotes the text embedding function. The retrieved evidence fragments are used as the primary context for generation, and the model is constrained to summarize, reorganize, and explain content strictly within the evidence scope. Free-form generation detached from retrieved materials is explicitly discouraged. If the retrieved evidence is insufficient to support a verifiable conclusion or no relevant fragments can be identified, the system returns a conservative response indicating insufficient information rather than producing speculative answers.

SQL Pathway. When the selected pathway is SQL, the execution layer handles fact-oriented queries that can be verified through structured data by executing restricted read-only structured queries. In implementation, SQL queries are generated by the same large language model used in the main experiments. The SQL-generation prompt consists of six components: task instruction, target database schema, field-description information, read-only constraint specification, output-format requirement, and the user query. The schema is serialized in the form of “table name–field name–field description”, while the read-only constraint explicitly prohibits modification operations such as INSERT, UPDATE, DELETE, ALTER, and DROP, and requires the model to output only SQL statements that satisfy these restrictions. To improve reproducibility, an example SQL prompt template is provided in the Appendix A. The generated SQL is first subjected to syntactic parsing and keyword-based rule validation, and query results are returned only when the SQL satisfies the read-only constraints and can be executed successfully; otherwise, the system uniformly degrades to the UNK pathway. The validated SQL is then executed in a restricted environment to obtain structured results, which are subsequently formatted under a unified response convention and verbalized into natural language. By delegating filtering, aggregation, and numerical computation to the database engine, the SQL pathway reduces generative inference over factual results and improves the consistency, auditability, and execution stability of fact-oriented answers. It should be noted, however, that the consistency and stability claimed in this work mainly refer to the stronger verifiability of the database execution and result computation stages relative to purely generative answering. Since the SQL statements themselves are still generated by a large language model, this pathway does not constitute a strictly end-to-end deterministic process. If the generated SQL fails to satisfy the read-only constraints, cannot be parsed, or fails during execution, the system does not force a speculative answer but instead degrades to the UNK pathway in order to reduce the risks of erroneous execution and unsupported outputs [18,19].

UNK Pathway. When the selected pathway is UNK, the execution layer directly outputs a unified refusal or guidance response and terminates further retrieval or querying. The UNK pathway is activated under two primary conditions. First, internal knowledge coverage is insufficient: no relevant evidence can be retrieved from the document corpus, or no matching records exist in the structured database to support a verifiable conclusion. Second, decision uncertainty is detected: the routing confidence does not exceed the threshold $\tau$, triggering conservative gating at the decision layer. In both cases, the system avoids inappropriate tool invocation and low-confidence forced answering by explicitly declining execution.

Overall, the three execution pathways are clearly separated in responsibility and constraints. The RAG pathway focuses on evidence-grounded explanatory tasks, the SQL pathway specializes in verifiable structured filtering and computation, and the UNK pathway provides system-level safe degradation and risk containment. Together with threshold-based routing at the decision layer, these components form a closed-loop, end-to-end controllable QA framework that prioritizes reliability and safe deployment in vertical-domain settings.

4. System Evaluation

4.1. Experimental System Background

We validate the feasibility and effectiveness of the proposed framework using a graduate admissions consultation intelligent QA system as an exemplar application. Graduate admissions consultation exhibits typical characteristics of high-frequency interaction, strict statement consistency, and high risk sensitivity. During the admissions season, candidates and their families frequently inquire about interview score thresholds, adjustment rules, material requirements, contact information, and admission statistics. Any factual deviation or inconsistency in responses can easily lead to misinformation and complaint risks. Consequently, the core requirements of this scenario are not expressive fluency, but rather verifiable facts, controllable behavior, and reproducible results. The system must be able to clearly ground answers in authoritative sources, avoid over-assertive generation under uncertainty, and maintain stable outputs across varied phrasings and noisy inputs [20,21,22,23,24,25].

Graduate admissions consultation is selected as the validation scenario primarily because it naturally encompasses two knowledge modalities and three categories of task boundaries, providing a representative and high-trust evaluation environment. First, admissions brochures, interview policies, and adjustment rules are predominantly released as unstructured documents, and corresponding questions typically require citation of authoritative textual evidence. Second, scores and admission outcomes are mainly stored as structured records, where common queries involve conditional filtering, aggregation, and cross-field constraint computation, with a stronger emphasis on the verifiability, reproducibility, and consistency of query results. Third, real-world consultations include a substantial number of queries related to trend prediction, subjective judgment, or individualized admission likelihood, for which definitive commitments are inappropriate. Such queries require explicit behavioral boundaries and robust risk control strategies.

Based on these characteristics, we present in the subsequent sections the construction procedures for internal structured databases and policy document repositories and organize a labeled evaluation benchmark covering RAG, SQL, and UNK intents. This benchmark supports routing evaluation and end-to-end comparative experiments under realistic, high-trust requirements.

4.2. Dataset Collection and Preparation

This subsection describes the internal data assets and evaluation dataset construction procedures used for system validation. Specifically, we construct a structured score and admission database and a policy document corpus and organize a labeled dataset covering RAG, SQL, and UNK intents. The resulting datasets are used for routing model training, confidence threshold selection, and end-to-end comparative evaluation.

4.2.1. Structured Score and Admission Database

A structured dataset was constructed based on publicly available graduate admissions score reports and imported into a locally deployed MySQL database. The data were primarily collected from admissions information and score sheets released by multiple schools of a normal university over the past six years. Through information extraction and field standardization, approximately 10,000 structured records were generated, covering core data for more than 30 graduate program tracks, as illustrated in Table 1. Based on this database, we further curated a structured QA test set consisting of 4296 question–answer pairs. These samples cover typical structured query tasks such as admission quotas, enrollment plans, and program-level distributions, providing reliable data support for precise question answering via the SQL pathway.

Table 1. Schema of the structured admission database.

Year	School	Program Code	Program Name	Initial Score	Re-Exam Score	Final Score	Study Mode	Admission Status
2025	School of Physics and Electronic Engineering	085410	Artificial Intelligence	310	70.08	65.23	Full-time	Admitted
2024	School of Education	045118	Preschool Education	370	71.20	72.88	Part-time	Admitted
2023	School of International Education	045300	Chinese International Education	350	90.05	78.02	Full-time	Admitted
2025	School of Primary Education	045115	Primary Education	336	82.66	73.38	Full-time	Admitted

To ensure compliance in data usage and stable system operation, we applied systematic desensitization and cleaning procedures to the raw data, including:

• Removal or anonymization of personal identifiers: Sensitive attributes such as names, identification numbers, and contact information are deleted or obfuscated to prevent privacy leakage.
• Normalization of sensitive fields: Fields such as geographic regions and institution names are standardized using encoded representations to avoid precise identification and ambiguity.
• Outlier detection and removal: Records with logical errors, missing values, or extreme anomalies are identified and filtered to ensure data quality.
• Field minimization: Only fields directly relevant to the QA tasks are retained, while redundant or irrelevant attributes are removed.

4.2.2. Unstructured QA Dataset

Unstructured data are primarily collected from PDF documents released by a normal university over multiple years, including graduate admissions brochures, training plans, and re-examination regulations. We first standardize document formats and apply OCR to convert them into machine-readable text. The documents are then segmented using structural cues such as headings and paragraph breaks, resulting in semantically coherent text chunks of approximately 100–300 words.

Each text chunk is encoded into a semantic vector representation and stored in a vector database, together with metadata such as source document, page number, and paragraph index. This design supports semantic retrieval, evidence localization, and provenance tracing during the retrieval stage, thereby providing explicit textual grounding for generative answers and reducing hallucination risks.

Based on the processed document corpus and vector index, we construct 2148 unstructured QA samples covering typical topics such as re-examination score thresholds, scholarship policies, examination subjects, and admission requirements. Queries and reference answers are generated under authoritative textual constraints and manually spot-checked. This dataset is used for training and validating the RAG pathway, with a focus on evaluating semantic matching, evidence extraction, and answer consistency.

4.2.3. Routing and Evaluation Dataset

Building upon the SQL-type and RAG-type QA samples described in Section 4.2.1 and Section 4.2.2, we further supplement UNK-type queries to construct a unified labeled dataset covering RAG, SQL, and UNK intents. This dataset is used for learned router training and subsequent end-to-end system evaluation. UNK queries are mainly derived from high-frequency boundary questions observed in real admissions consultations, together with paraphrased and extended variants. These queries emphasize scenarios such as trend prediction, subjective school selection, individualized admission likelihood estimation, missing critical information, and cases exceeding internal knowledge coverage. To ensure labeling consistency, each UNK candidate undergoes answerability verification. If a query can be answered through read-only database queries and yield a verifiable conclusion, it is relabeled as SQL; if sufficient authoritative evidence can be retrieved from official policy documents to form a verifiable conclusion, it is relabeled as RAG. Only queries that cannot be reliably answered using internal documents or databases, or for which definitive commitment is inappropriate, are retained as UNK.

For unified evaluation and automated assessment, all UNK samples adopt a fixed refusal template. Table 2 presents representative examples of the three categories. Following this process, the final dataset contains 4296 SQL samples, 2148 RAG samples, and 1938 UNK samples, totaling 8382 queries. The dataset is stratified by category and split into training, validation, and test sets with an 8:1:1 ratio. The training set is used exclusively for router learning; the validation set is used for hyperparameter tuning and confidence threshold $\tau$ selection; after fixing all configurations, the test set serves as a held-out benchmark for final reporting, ensuring fair comparison and reproducibility.

Table 2. Examples of SQL, RAG, and Unk queries.

Query	Label	Reference Answer
How many students were admitted to the Computer Technology program of the School of Information Science in 2024?	SQL	In 2024, 28 students were admitted to the Computer Technology program of the School of Information Science.
What was the highest initial exam score for the Artificial Intelligence program in 2025?	SQL	The highest initial exam score for the Artificial Intelligence program in 2025 was 342.
How is the re-examination score threshold determined?	RAG	The re-examination score threshold is typically determined based on the annual admission plan, score distribution, and relevant institutional policies, and is officially announced by the university.
What is the contact information of the Graduate Admissions Office of the School of Marxism?	RAG	The contact phone number of the Graduate Admissions Office of the School of Marxism is 0898-65812980.
Is the Computer Technology program difficult to get into this year?	Unk	This question exceeds the scope of verifiable answers based on official documents and the score database, or involves predictive judgment. A definitive conclusion cannot be provided.
If my initial exam score is not ideal, do I still have a chance to be admitted?	Unk	This question exceeds the scope of verifiable answers based on official documents and the score database, or involves individualized decision-making and prediction. A definitive conclusion cannot be provided.

4.3. Experiments and Results Analysis

This section’s experiments and results analysis comprise two parts: First, we evaluate the load-balancing stability and denial-of-service control effectiveness of learning-based routing with confidence threshold $\tau$ on the validation/test set. Subsequently, we conduct end-to-end evaluations with a fixed decision strategy to compare the final response quality and overall consistency across different system variants.

4.3.1. Hyperparameter Sensitivity Analysis

To obtain a stable configuration for the three-class routing task in admissions consultation, we first conduct hyperparameter sensitivity experiments on the XGBoost router using the validation set. While keeping data splits and feature construction fixed, we focus on two key hyperparameters: learning_rate, which controls the step size of gradient boosting, and max_depth, which determines the complexity of individual trees. We perform grid search over predefined candidate ranges and evaluate performance primarily using Macro-F1, with Accuracy as a secondary metric. Since the routing task involves class imbalance among RAG, SQL, and UNK, Accuracy alone may be dominated by majority classes and obscure minority performance, especially for UNK. Macro-F1 assigns equal weight to each class and better reflects balanced routing quality at answerable–unanswerable boundaries.

As shown in Figure 2, Macro-F1 stabilizes once the learning rate exceeds a certain range, with only minor fluctuations thereafter. In contrast, variations in max_depth have a relatively limited impact, indicating low sensitivity to tree depth under the current feature and data scale. Considering Macro-F1 stability, Accuracy for tie-breaking, and inference efficiency, we select learning_rate = 0.20 and max_depth = 3 as the default configuration for subsequent experiments.

Figure 2. Hyperparameter sensitivity of the XGBoost router on the validation set.

4.3.2. Confidence Threshold $\mathbf{\tau }$ Analysis

After fixing the router configuration, we scan the confidence threshold $\tau$ on the validation set to determine an appropriate operating point for low-confidence gating. The threshold controls refusal strength: higher values lead to more queries being gated as UNK.

To characterize the trade-off between routing quality and refusal intensity, we jointly analyze Macro-F1 and the UNK ratio as functions of $\tau$. Macro-F1 reflects balanced routing performance across the three classes, while the UNK ratio directly measures refusal intensity and inversely indicates answerable coverage.

As illustrated in Figure 3, the UNK ratio increases in a stepwise manner as $\tau$ grows, whereas Macro-F1 remains stable over a relatively wide range before dropping sharply at high thresholds. This indicates that moderate threshold increases can strengthen conservative gating without significantly degrading routing balance, while overly large thresholds lead to excessive refusals and reduced coverage.

Figure 3. Effect of confidence threshold $\tau$ on Macro-F1 and UNK ratio.

Based on the joint trends of the two curves, we select on the validation set a working point located within the performance plateau and before the refusal rate begins to rise rapidly. The default threshold for subsequent experiments is therefore fixed at $\tau =0.562$, and it is used for final test-set reporting and end-to-end comparative evaluation. It should be noted that the primary purpose of Section 4.3.2 is to determine the threshold on the validation set and then keep the selected threshold fixed for test-set evaluation. Since the main focus of this study is to validate the overall effectiveness of the proposed framework at a fixed operating point, we do not further conduct end-to-end sensitivity analysis around neighboring threshold values on the test set. Nevertheless, the validation-set threshold sweep shows that Macro-F1 varies relatively smoothly around the optimal threshold, without abrupt fluctuations, which to some extent suggests a certain degree of robustness to threshold perturbation. A more systematic threshold sensitivity analysis is left for future work.

4.3.3. Routing Decisions and Refusal Statistics on the Test Set

After fixing the router configuration and confidence threshold $\tau$, we evaluate routing stability and refusal controllability on the test set. Test samples are grouped by gold routing labels into RAG, SQL, and UNK, where UNK denotes inputs that are unanswerable or lack sufficient evidence.

Given the safety semantics of UNK, we adopt an answerability-aware scoring rule. When the gold label is RAG or SQL (answerable), predicting UNK is counted as an error (false refusal). When the gold label is UNK (unanswerable or insufficient evidence), predicting UNK is counted as correct (true refusal). For each gold category, we report the total number of samples (total), the number of correct predictions (correct), the number of predictions routed to UNK (pre_unk), the number of incorrect predictions (wrong), and the accuracy defined as $\mathtt{correct}/\mathtt{total}$. The ALL row summarizes overall performance.

Table 3 presents the routing and refusal statistics on the test set. Overall, the router achieves an accuracy of 0.9857 on 838 test samples, indicating high routing stability at the selected operating point. For RAG queries, 211 out of 215 samples are routed correctly (accuracy 0.9814), with 4 samples gated to UNK, corresponding to false refusals on answerable inputs. For SQL queries, 424 out of 428 samples are routed correctly (accuracy 0.9907), with only 4 samples misrouted to UNK, demonstrating strong stability for structured queries. For UNK queries, 191 out of 195 samples are correctly refused (accuracy 0.9795), indicating effective coverage of unanswerable inputs; the remaining 4 cases correspond to missed refusals.

Table 3. Routing and refusal statistics on the test set.

Gold	Total	Correct	Pred. as UNK	Wrong	Accuracy
RAG	215	211	4	4	0.9814
SQL	428	424	4	4	0.9907
UNK	195	191	191	4	0.9795
All	838	826	198	12	0.9857

Across the entire test set, 198 samples are predicted as UNK, the majority of which originate from gold UNK cases (true refusals), while only a small fraction come from RAG or SQL (false refusals). These results suggest that the selected threshold $\tau$ achieves a balanced trade-off between refusal safety and answerable coverage.

4.3.4. Automatic End-to-End Evaluation

To further compare the final answer quality of different systems from an end-to-end perspective, we conduct automatic evaluation based on the default router configuration and the operating threshold $\tau$ determined in the decision experiments. To ensure fair horizontal comparison, all compared methods are evaluated on the test set using the same input queries, reference answers, internal data sources (the official document corpus and the structured score database), and evaluation pipeline. The methods differ only in pathway decision strategy, tool invocation mechanism, and whether low-confidence refusal control is enabled.

In addition to the original internal comparison settings, we further introduce two external baseline methods to simulate representative paradigms of multi-tool orchestration and tool use in the existing literature. Specifically, the ReAct-style baseline is designed to reflect the dynamic “reasoning–action–observation” tool invocation mechanism used in ReAct-like methods, while the Self-Ask baseline is intended to reflect the intermediate sub-question decomposition and lightweight tool invocation strategy used in Self-Ask-like methods. The final comparison therefore includes the following six system variants:

• LLM-only: No retrieval or database access is used; answers are generated directly by the LLM based solely on the input query.
• LLM-Orchestrator: The LLM performs routing decisions (RAG/SQL/UNK) and triggers the corresponding execution pipelines.
• Ours without $\mathbf{\tau }$: The proposed system with XGBoost-based routing and multi-path execution, but without confidence threshold gating; the route with maximum posterior probability is always executed.
• ReAct-style baseline: a dynamic tool-use baseline that alternates among reasoning, tool invocation, and observation, where the same large language model decides during generation whether to call the document retrieval tool or the structured query tool. For fairness, it shares the same document knowledge base, structured database, and final response templates as our method. The maximum number of tool-use steps is limited to three in order to control inference cost and maintain comparability across methods. This setting is also consistent with the relatively bounded nature of the current task, in which most queries can be handled within a small number of tool interactions. Relaxing this limit may allow additional exploration, but could also introduce longer trajectories and greater instability, which we leave for future study.
• Self-Ask baseline: a decomposition-based tool-use baseline that first determines whether external information is required and then generates intermediate sub-questions to trigger tool invocation. It shares the same knowledge sources and output constraints as our method, but does not employ an explicit pathway classifier or a separate confidence-threshold refusal mechanism; when no valid tool invocation result can be produced, a default refusal response is returned.
• Ours: The full system, including learned routing, restricted execution, and confidence-based refusal with threshold $\tau$.

End-to-end evaluation is conducted on final system outputs. For each sample, the system output is denoted as A and the reference answer as G. To assess output quality from complementary perspectives, we adopt an automatic evaluation suite that combines coverage, semantic consistency, and surface-level consistency.

Keyword Coverage (Keyword-Cov). This metric measures the extent to which key information in the reference answer is covered by the system output. For each sample, a keyword set $K\left(G\right)$ is extracted from G using tokenization, stopword removal, and domain-specific keyword lists. Keyword coverage is defined as:

$$\mathrm{Keyword}\text{-}\mathrm{Cov}(A,G)={\displaystyle \frac{\left|K\right(G)\cap K(A\left)\right|}{\left|K\right(G\left)\right|}}.$$

(5)

This metric focuses on whether the essential points are addressed and is relatively robust to paraphrasing.

BERTScore-F1. To mitigate limitations of surface-level matching, we adopt BERTScore as a semantic alignment metric and report its F1 score. BERTScore computes contextual token-level similarity between A and G based on pretrained language representations:

$$\begin{array}{cc}\hfill R_{\mathrm{BERT}}& ={\displaystyle \frac{1}{\left|X\right|}}\sum _{x_i\in X}\underset{s_j\in S}{max}{x}_i^{\top }{s}_j,\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill P_{\mathrm{BERT}}& ={\displaystyle \frac{1}{\left|S\right|}}\sum _{s_j\in S}\underset{x_i\in X}{max}{x}_i^{\top }{s}_j,\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill F_{\mathrm{BERT}}& ={\displaystyle \frac{2P_{\mathrm{BERT}}{R}_{\mathrm{BERT}}}{P_{\mathrm{BERT}}+R_{\mathrm{BERT}}}},\hfill \end{array}$$

(8)

where X and S denote the token embedding sets of the reference and generated texts, respectively.

Cleaned-Content-F1. To reduce the influence of fixed templates, disclaimers, and formatting artifacts, we normalize both A and G through text cleaning (e.g., removing boilerplate phrases, normalizing numbers and units). Precision P and recall R are computed at the word or token level, and F1 is calculated as:

$$\mathrm{Cleaned}\text{-}\mathrm{Content}\text{-}\mathrm{F}1(A,G)={\displaystyle \frac{2PR}{P+R}}.$$

(9)

This metric reflects strict content consistency and reproducibility.

Sentence Cosine Similarity (SentCos). We further evaluate sentence-level semantic similarity by encoding A and G into sentence embeddings and computing cosine similarity:

$$\mathrm{SentCos}(A,G)=cos(e_A,e_G),$$

(10)

which provides a stable complement to token-level semantic metrics.

Based on these metrics, we compute per-sample scores and report averaged results over the test set. The comparative results are summarized in Table 4.

Table 4. Automatic end-to-end evaluation results.

Variant	Keyword-Cov	BERT-F1	Cleaned-Content-F1	SentCos
LLM-only	0.6944	0.6129	0.1825	0.7185
LLM-Orchestrator	0.6596	0.7780	0.6591	0.8142
Ours without $\tau$	0.8942	0.9462	0.8856	0.9524
ReAct-style baseline	0.8301	0.8503	0.8043	0.8521
Self-Ask baseline	0.8211	0.7586	0.7060	0.7370
Ours	0.9257	0.9596	0.9167	0.9677

From Table 4, clear differences can be observed in end-to-end answer quality across the compared systems. Overall, LLM-only performs the worst, and is substantially inferior to the other methods particularly in terms of BERT-F1, Cleaned-Content-F1, and SentCos. This suggests that when answers rely solely on parametric memory without external knowledge support or tool invocation, the system may still cover some surface-level keywords but remains limited in factual alignment, content completeness, and semantic consistency. Compared with LLM-only, LLM-Orchestrator shows improvement in semantic consistency and content consistency, indicating that direct tool orchestration by a large language model can improve answer quality to some extent. However, its relatively low Keyword-Cov suggests that when pathway selection is determined purely through implicit generative judgment, critical information may still be omitted, and tool invocation or degradation behavior may remain unstable.

Compared with the above methods, the introduction of an explicit multi-path execution mechanism leads to a clear overall performance gain. Both the ReAct-style baseline and the Self-Ask baseline outperform the purely generative methods, indicating that tool use is beneficial for fact-oriented question answering. However, these two methods still rely mainly on implicit decisions made during the reasoning process for tool selection and invocation and therefore continue to exhibit a certain degree of variation across different question types. Among them, the ReAct-style baseline performs better overall than the Self-Ask baseline, suggesting that dynamic tool use is more adaptable than lightweight decomposition-based strategies in the current task, although its results still remain below those of our method.

Ours without $\tau$ achieves relatively high scores across all four metrics, indicating that explicit pathway selection based on learned routing together with restricted execution can more stably dispatch different queries to appropriate processing pathways, thereby improving key-point coverage, semantic alignment, and content consistency. On this basis, the full method further achieves the best results across all four metrics, showing that the introduction of the confidence threshold $\tau$ reduces misrouting and inappropriate execution under low-confidence inputs, and thus further improves end-to-end output quality. Overall, the coordinated design of explicit pathway decision-making, restricted execution, and threshold-based refusal gating more effectively balances answer coverage and risk control under the current experimental setting, yielding the best overall performance.

It should be noted that, because the UNK samples adopt a fixed refusal template, aggregated automatic metrics on the full test set may to some extent be influenced by template matching. To more accurately reflect the quality of system answers on genuinely answerable queries, we further conduct a separate evaluation on the answerable subset after removing UNK samples (i.e., RAG + SQL), as shown in Table 5.

Table 5. Automatic evaluation results on the answerable subset (RAG + SQL).

Variant	Keyword-Cov	BERT-F1	Cleaned-Content-F1	SentCos
LLM-only	0.6944	0.6129	0.1825	0.7185
LLM-Orchestrator	0.6498	0.7683	0.6506	0.8100
Ours without $\tau$	0.9064	0.9527	0.8972	0.9653
ReAct-style baseline	0.8287	0.8734	0.8474	0.8799
Self-Ask baseline	0.7811	0.7869	0.7692	0.7674
Ours	0.9287	0.9592	0.9190	0.9753

Table 5 reports the automatic evaluation results on the answerable subset (RAG + SQL). Overall, all methods that incorporate external tool use outperform LLM-only, indicating that external knowledge access and structured querying play a clear role in improving performance on answerable queries. Both the ReAct-style baseline and the Self-Ask baseline show improvements over the purely generative method, but still remain below our method overall, suggesting that implicit tool selection during generation is still insufficient for stable question-type identification and pathway matching. In contrast, our method achieves the best performance on Keyword-Cov, BERT-F1, Cleaned-Content-F1, and SentCos, indicating that explicit pre-execution pathway decision-making together with restricted execution more effectively improves answer quality and consistency on answerable queries.

Taken together, the results on the full test set and the answerable subset show that our method still maintains strong performance after removing UNK samples. This suggests that the observed improvement is not mainly due to superficial matching with a fixed refusal template, but more likely arises from the coordinated design of learned routing, restricted execution, and confidence-threshold gating.

4.3.5. Human Evaluation

Although automatic evaluation metrics can assess system output quality from perspectives such as keyword coverage, semantic similarity, and content consistency, they remain proxy measures and cannot fully capture factual correctness or the appropriateness of refusal behavior. To provide complementary evidence regarding the system’s practical performance in real question answering scenarios, we further conducted a human evaluation study.

Specifically, we randomly sampled 80 instances from the test set, including 25 RAG cases, 25 SQL cases, and 30 UNK cases, and invited three evaluators with relevant background knowledge to score the system outputs. Each evaluator performed the assessment independently based only on the input questions and the corresponding system responses, and the final score for each sample was obtained by averaging the three ratings. We observed generally consistent scoring trends across evaluators. A 10-point rating scale was adopted. For answerable samples (RAG and SQL), the evaluation mainly considers three aspects: answer correctness, information completeness, and clarity of expression. For unanswerable samples (UNK), the evaluation focuses primarily on whether the refusal is appropriate, namely whether the response clearly explains that the system cannot currently provide a reliable conclusion and avoids unsupported or misleading answers.

The human evaluation results are summarized in Table 6. Our method achieves average scores of 8.9, 8.6, and 9.1 on the RAG, SQL, and UNK subsets, respectively, with an overall average score of 8.9. After normalization by the full score of 10, the corresponding normalized scores are 0.89, 0.86, 0.91, and 0.89.

Table 6. Human evaluation results.

Type	Number of Samples	Average Score	Normalized Score
RAG	25	8.9	0.89
SQL	25	8.6	0.86
UNK	30	9.1	0.91
Overall	80	8.9	0.89

These results indicate that the proposed method can provide relatively accurate and reasonably complete answers for answerable queries while also generating appropriate refusal responses in unanswerable cases with comparatively stable behavior. Combined with the automatic evaluation results reported above, the findings suggest that the proposed framework performs well not only in terms of semantic and content consistency but also in terms of practical usability and boundary control as reflected in human evaluation.

It should be noted that the present human evaluation remains a small-scale sampled validation and is intended primarily as supplementary evidence for the automatic evaluation results. Future work may extend this analysis to larger sample sizes and more diverse application scenarios and incorporate multi-model human comparisons to provide a more comprehensive assessment of system performance.

4.3.6. Supplementary Cross-Scenario Experiment in an Enterprise FAQ Setting

To examine whether the proposed method is overly dependent on a single graduate admissions consultation scenario, we further construct a small-scale supplementary validation dataset in an enterprise FAQ setting and conduct cross-scenario testing on different methods. The purpose of this experiment is not to claim that the proposed framework already possesses strong cross-domain generalizability, but rather to provide supplementary evidence for the main experimental findings by offering a preliminary observation of its applicability in another mixed-task setting involving document knowledge, structured information, and unanswerable queries.

The supplementary dataset is built around common enterprise business question answering scenarios and contains three categories of queries. The first category consists of document-based questions that rely on FAQ texts or business instruction documents and corresponds to the RAG setting. The second category includes structured-field queries involving items such as fees, limits, and processing time, corresponding to the SQL setting. The third category contains queries that exceed knowledge coverage, lack sufficient evidence, or are inappropriate for definitive answers, corresponding to the UNK setting. The total dataset size is approximately 150 samples. To maintain consistency with the main experiment as much as possible, all compared methods reuse the model configurations, routing settings, and evaluation procedures determined in the main experiment, and no additional tuning is performed for this supplementary dataset.

Using the same automatic evaluation metrics as those adopted in the main experiment, the supplementary results are reported in Table 7.

Table 7. Automatic evaluation results on the enterprise FAQ supplementary dataset.

Variant	Keyword-Cov	BERT-F1	Cleaned-Content-F1	SentCos
LLM-only	0.2616	0.5922	0.3502	0.5842
LLM-Orchestrator	0.7138	0.7784	0.7272	0.7750
Ours without $\tau$	0.8604	0.8361	0.8177	0.8047
ReAct-style baseline	0.6739	0.7038	0.7607	0.6994
Self-Ask baseline	0.8055	0.7613	0.7212	0.7447
Ours	0.8927	0.9180	0.8633	0.9091

As shown in Table 7, although the absolute performance of each method exhibits some variation relative to the main experiment, the overall trend remains broadly consistent. Specifically, LLM-only still performs the worst, indicating that in the absence of external knowledge support and tool invocation mechanisms, the model struggles to stably handle queries with factual constraints. LLM-Orchestrator, ReAct-style baseline, and Self-Ask baseline all outperform LLM-only, suggesting that the introduction of tool usage and external knowledge access can improve response quality. However, these methods still rely primarily on implicit decision-making during generation for pathway selection and therefore continue to exhibit a certain degree of instability under cross-scenario conditions.

In contrast, the proposed method achieves the best performance across all four evaluation metrics. Moreover, Ours further improves over Ours without $\tau$, indicating that explicit pathway routing and confidence-threshold gating remain beneficial in this supplementary setting. To some extent, these results suggest that the proposed “learned routing + restricted execution + threshold-based refusal” framework is effective not only in the graduate admissions consultation scenario, but also shows promising applicability in another mixed-knowledge question answering task.

It should be noted, however, that this supplementary experiment remains limited in dataset scale, and that the question distribution and knowledge organization in the enterprise FAQ setting are still relatively simple compared with those in the main scenario. Therefore, the current results are better interpreted as preliminary supplementary evidence regarding cross-scenario feasibility, rather than as support for stronger conclusions about cross-domain generalization. Future work should further evaluate the transferability and robustness of the proposed framework on larger and more diverse datasets.

5. Conclusions and Future Work

This paper proposes an internal-data-first, multi-path framework for controllable question answering in high-risk scenarios and systematically validates its effectiveness through a hybrid vertical-domain question answering dataset and comparative experiments across multiple system variants. The framework treats governable internal data as the sole source of factual commitment and explicitly separates knowledge sources into official institutional documents (unstructured textual evidence) and a score database (structured records). Through a unified entry point, a learned router performs three-way classification over RAG, restricted read-only SQL, and UNK: document-grounded queries are handled via retrieval-augmented generation, queries that can be verified through structured querying are resolved through restricted read-only SQL, and inputs that are insufficient, out-of-scope, or predictive in nature trigger safe refusal. In addition, a confidence-threshold gating mechanism is introduced to front-load risk control, thereby reducing cascading errors caused by misrouting and low-confidence forced answering. Under the current study setting, the proposed framework demonstrates better routing stability and end-to-end output consistency than both purely generative baselines and baselines relying solely on large-language-model orchestration, suggesting that the coordinated design of learned routing, restricted execution, and threshold-based refusal helps improve system controllability and result consistency. Despite these advantages, several limitations remain. The confidence-threshold gating mechanism introduces an inherent trade-off between refusal safety and answerable coverage, and the use of a fixed refusal template for UNK may bias certain surface-level automatic metrics. In addition, the query embedding model adopted in this work, bge-small-zh-v1.5, is designed for Chinese semantic matching, and thus the current findings are primarily grounded in Chinese-language settings. Deployment in non-Chinese scenarios would require replacing the embedding model with a language-appropriate alternative and may also require retraining or adjusting the router and related prompt templates.

Future work can be pursued along the following directions:

1.

Enhancing decision-layer reliability (routing and confidence control).Beyond the current three-class routing formulation, future work may incorporate confidence calibration and out-of-distribution (OOD) detection to further disentangle low-confidence inputs from intrinsically unanswerable or out-of-scope queries. Cost-sensitive learning and risk-aware optimization can also be explored to enable adaptive threshold selection that remains robust under shifts in input distributions and class proportions.

2.

Strengthening execution-layer controllability and evidence constraints (multi-path consistency and fallback protocols). For the RAG, SQL, and UNK pathways, stricter consistency checks and failure recovery mechanisms can be introduced, such as coverage and consistency validation of retrieved evidence, citation anchoring and version management for documents, semantic equivalence verification and read-only sandboxing for SQL execution, and unified degradation strategies with interpretable feedback. These measures can further reduce risks of biased answers or unsupported generation while improving auditability.

3.

Improving evaluation protocols and reproducible benchmarks (reliability–coverage–cost trade-offs). Beyond existing automatic metrics, future evaluation can incorporate layered protocols that explicitly assess evidence correctness, refusal rationality, and routing cost. Human audit sampling and boundary stress-test sets can be introduced to systematically characterize trade-offs among reliability, coverage, and computational cost, supporting more comprehensive conclusions and guiding subsequent framework iterations.