Agentic Hallucination Risk Scoring for Medical LLMs via Uncertainty Quantification and Clinical Knowledge Injection

Abstract

Large Language Models (LLMs) have witnessed significant adoption across numerous domains since 2020, but their proclivity to hallucinate creates unacceptable dangers in high-risk environments like healthcare, where wrong outputs can directly jeopardize human safety. While present systems focus on pre-generation mitigation strategies, they cannot ensure the safety of individual outputs during inference. We provide a post hoc Hallucination Risk Scoring (HRS) methodology that intercepts questionable outputs before they reach patients via an agentic pipeline. Given a medical question, a domain-specific LLM generates an initial response from which five complimentary uncertainty signals are computed, which are then separated into a decision layer that governs escalation and a guidance layer that directs clinical knowledge injection by a GPT. The framework is tested using three biological question-answering datasets of various complexity: PubMedQA-Labeled, PubMedQA-Artificial, and BioASQ Task B. The results show an up to 38% safety increase at the most sensitive threshold configuration, zero deterioration across all experimental configurations enforced by the Revert Baseline method, and complexity-aware escalation rates that scale organically with dataset difficulty. Tunable thresholds allow physicians to calibrate system behavior based on deployment requirements, providing a practical safety–accuracy trade-off. Statistical research finds entropy as the primary uncertainty signal separating escalated from non-escalated situations across all datasets. These findings provide a deployable, interpretable, and configurable post hoc safety paradigm for reliable medical AI implementation.

1. Introduction

The rapid adoption of Large Language Models (LLMs) [1] after 2020 has created revolutionary opportunities in multiple sectors. However, the restriction of LLMs known as hallucination [2] poses unacceptable risks in high-stakes sectors like healthcare due to models producing reasonable but wrong outputs. Medical question-responding requires a high level of reliability [3], which current LLMs cannot offer.

In a clinical scenario, a patient presents a medical report to an LLM and inquires about their lung cancer diagnosis. If the model inaccurately responds yes, it can cause psychological suffering, unneeded medical treatments, and erode trust in AI-assisted healthcare. False negative responses can delay essential diagnoses. Undetected hallucinations pose a direct threat to patient safety, highlighting the need to address this issue in the medical field [4].

1.1. Motivation

Currently, hallucination mitigation methods focus on pre-generation processes such as model fine-tuning, retrieval-augmented generation, and instruction tuning [5]. Although helpful in minimizing hallucinations, these systems have a major limitation: they cannot ensure the safety of individual output. Even a fine-tuned model can hallucinate on rare or complex medical issues, and presenting an inaccurate response to the user can cause significant injury.

Fine-tuning large medical language models needs significant computational resources and domain-specific training data, making them unsuitable for quick deployment in many clinical settings. Our work is inspired by the discovery that effective prompt engineering can improve the performance of current models without the computational overhead of retraining [6]. Targeted clinical knowledge injection can improve a model’s performance by lowering uncertainty at inference time, without affecting model weights [7].

Our post hoc safety approach involves intercepting uncertain outputs before they reach patients, either through agentic prompt optimization or escalating cases to clinical experts, rather than attempting to eliminate hallucinations at the model level. We prioritize patient safety over reaction speed in clinical deployments; we accept a tiny latency overhead for a meaningful safety assurance.

1.2. Contributions

This work makes the following contributions:

• Hallucination Risk Signal (HRS) Framework: We propose a two-layer decision and guiding architecture that incorporates five complementary uncertainty signals (entropy, perplexity, ambiguity, rarity, and token length) to quantify post hoc hallucination risk in medical LLMs.
• Agentic Safety Pipeline: Our iterative agentic pipeline uses a GPT orchestrator to input clinical knowledge and reduce model uncertainty through targeted quick improvement, rather than retraining.
• Zero Degradation Guarantee: We found zero degradation in nine runs across three datasets and three threshold configurations, indicating that the pipeline never transforms a correct response to an incorrect one.
• Tunable Safety–Accuracy Tradeoff: We give an empirical characterization of the safety–accuracy tradeoff across several threshold settings. This allows physicians to calibrate system behavior based on the deployment scenario.
• Multi-Layer Complexity Validation: We validate escalation decisions using two separate layers. ML classifier performance analysis and statistical signal validation using Mann–Whitney U tests [8] provide model-independent evidence that escalated cases represent genuine medical perplexity.

1.3. Research Questions

This work is guided by the following research questions:

• RQ1: What HRS signals distinguish escalated from non-escalated instances, and how does their value differ among datasets of varying complexity?
• RQ2: Is there a correlation between HRS escalation rate and dataset complexity? Can this be proven using an independent ML classifier performance analysis?
• RQ3: What impact does the quality of the orchestrator model have on the safety–accuracy tradeoff, as well as the costs and latency for clinical deployment?
• RQ4: Do ML classifiers support HRS escalation choices by showing worse performance on escalated cases, validating true medical uncertainty across models?

1.4. Paper Organization

The remainder of this paper is organized as follows. Section 2 reviews related work on hallucination detection, uncertainty quantification, and agentic AI systems. Section 3 describes the datasets, HRS signal design, agentic pipeline, and evaluation metrics. Section 4 presents experimental results across threshold sensitivity, orchestrator comparison, statistical validation, and complexity analysis. Section 5 discusses key findings and limitations. Section 6 concludes with directions for future work.

2. Related Work

2.1. Hallucination in Medical LLMs

Hallucination in LLMs poses a significant concern in medical contexts, since inaccurate answers might have a direct impact on patient safety. Research on hallucinations in therapeutic settings has taken many approaches, including benchmark construction, ethical governance, and taxonomic survey. The lack of a deployable, inference-time safety mechanism leaves a major gap between detecting and preventing hallucinations in patients.

Pal et al. [9] developed Med-HALT, a benchmark and dataset for evaluating hallucinations in medical LLMs. Med-HALT, based on international medical examinations, identifies two types of hallucination tests: reasoning- and memory-based, which assess problem-solving and knowledge retrieval capacities, respectively. Med-HALT is an evaluation system that recognizes hallucinations but does not provide a runtime technique to prevent or rectify them. The benchmark uses a single-pass approach, with no iterative refining or agentic interaction. Our study uses an agentic orchestration loop to detect uncertainty and inject targeted clinical knowledge at inference time. This approach is data-agnostic and applies beyond preset benchmark settings.

Kozlakidis et al. [10] approached medical LLM hallucination from an ethical and governance perspective, arguing that five major stakeholder groups (developers, employers, end-users, publishing platforms, and regulators) have different responsibilities in managing hallucination risk in clinical settings. Although the concept is useful for developing accountability structures, it does not address identifying, decreasing, or managing hallucination at the system level. Our work demonstrates the ethical responsibility of developers to reduce hallucination risk through technical safeguards and rigorous testing. We use an agentic pipeline with interpretable uncertainty signals and clinical knowledge injection to implement this obligation.

Zhu et al. [11] conducted the most thorough survey on medical hallucination, establishing a single taxonomy across LLMs and large vision-language models. They analyzed causes at the data, training, and inference levels, and reviewed improvements in detection, evaluation, and mitigation. However, their challenges section highlights problems that our work directly solves. Existing mitigation strategies rely on finetuning, which is computationally expensive and not suitable for quick clinical application. Our approach uses a prompt-based agentic strategy to improve model behavior during inference without adjusting weights. Second, existing benchmarks identified in the survey are heavily concentrated on healthcare queries and CT imaging reports, limiting generalization. We assessed our approach using biomedical question-answering datasets from several medical subdomains. They also found no studies that use hallucination risk rating based on model-internal signals for detection and intervention. Our framework evaluates clinically motivated safety metrics such as safety gain, auto-answer accuracy, degradation rate, and escalation rate, providing a more meaningful assessment of system trustworthiness in clinical deployment. Standard evaluations rely on surface-level metrics like BLEU [12] and ROUGE [13].

All the works demonstrate the clinical severity of hallucination, identify ethical obligations, and map the solution landscape. However, none suggest an inference-based system to intercept ambiguous outputs before they reach the user. Our proposed methodology is motivated by the lack of interpretable uncertainty signals, iterative intervention, and clinically grounded safety criteria in previous efforts.

2.2. Uncertainty Quantification

The ability to quantify uncertainty in LLM outputs [14] is a necessary prerequisite for hallucination reduction, as overconfident but wrong responses are precisely what make hallucinations dangerous in clinical settings. Liu et al. [15] addressed this in a comprehensive survey of uncertainty quantification methods for LLMs, introducing a new taxonomy that divides UQ approaches into two categories: computational efficiency and uncertainty dimensions, specifically input, reasoning, parameter, and prediction uncertainty. Their work assessed existing methodologies and compiled benchmarks and metrics for the UQ landscape. However, their challenges section identifies a critical interpretability deficit. Users are unable to distinguish whether uncertainty is caused by ambiguous inputs, knowledge gaps, or decoding stochasticity, advocating for future systems to incorporate agentic behavior and precompute uncertainty signals before committing to a final output. Our research directly addresses all three of these shortcomings. Each of our five Hallucination Risk Signals corresponds to a separate uncertainty dimension in their taxonomy: Entropy and Perplexity HRS measure prediction uncertainty at the decision border and reasoning level, respectively. Ambiguity HRS measures input uncertainty via Jensen-Shannon divergence [16] over numerous runs, Rarity HRS captures parameter uncertainty by measuring semantic distance from the training distribution, and Token Length HRS acts as a surrogate for reasoning uncertainty. Rather than producing a single opaque confidence score, our multi-signal architecture makes uncertainty understandable and actionable, directly addressing the interpretability gap they found. Furthermore, our agentic orchestration loop and Baseline Gate method operationalize their request for precomputed uncertainty-aware intervention before responses reach the user.

Sah [17] conducted a thorough systematic analysis of uncertainty in LLMs, reviewing over 45 works to connect theoretical foundations with cutting-edge methods. Their study divides uncertainty into aleatoric and epistemic forms, detailing probabilistic modeling, confidence assessment, and calibration procedures, with example case studies based on medical diagnosis. Critically, Sah suggests numerous future study possibilities that our work directly addresses. First, they advocate for agentic inference as a crucial advancement for uncertainty-aware systems; our approach replies by implementing a comprehensive agentic orchestration loop that iteratively monitors uncertainty signals and intervenes before responses reach the user. Second, they argue that perturbation strategies alone are insufficient for comprehensive risk assessment and advocate for optimized prompting; our clinical knowledge injection mechanism exemplifies this perfectly, using targeted fact injection rather than generic rephrasing to reduce measurable uncertainty across iterations. Third, they identify trustworthy AI as an open challenge in diagnosing uncertainty and mitigating bias; our five HRS signals directly address this by capturing distinct behavioral dimensions of model uncertainty, targeting different sources of unreliability rather than combining everything into a single opaque confidence score. Finally, our study of three datasets of diverse difficulty and three orchestrator configurations responds to their request for scalable and robust deployment, indicating that our framework maintains its safety guarantees across different clinical complexity levels.

Chen et al. [18] investigate uncertainty quantification for language models used to perform EHR activities in both white-box and black-box contexts [19]. In the white-box environment, when model parameters and output logits are available, they use multi-tasking prediction prompts and ensemble approaches to quantify and minimize model uncertainty. They then apply the ensembling-based technique to closed, black-box models like GPT-4. While their methodology demonstrates the usefulness of ensemble-based uncertainty reduction in clinical prediction tasks, it is not without limits. First, their technique is limited to EHR prediction tasks and does not apply to larger biomedical question-answering domains, whereas our platform has been tested across different biomedical QA datasets of varying complexity levels. Second, their method requires access to numerous model instances for assembly, whereas our system uses focused prompt-based clinical knowledge injection on a single model, making it better suited for lightweight clinical deployment. Third, and most importantly, their approach focuses entirely on assessing and lowering uncertainty, without intercepting or escalating uncertain outputs before they reach the user. Our agentic pipeline immediately tackles this gap by intercepting uncertain responses and referring confusing cases to clinical experts, putting patient safety ahead of system performance.

The reviewed works contribute to our understanding of hallucination detection, ethical governance, taxonomy classification, and uncertainty quantification in medical LLMs. However, a persistent gap emerges across all works: no one proposes an inference-time, agentic, post hoc safety mechanism that intercepts questionable outputs before they reach the user. Our work is directly inspired by this fundamental limitation, and we propose a paradigm that combines interpretable uncertainty signals, iterative clinical knowledge injection, and escalation-based safety assurances. The following sections provide a detailed description of each component’s design and implementation.

Recent survey works such as [20,21] provide a comprehensive overview of large language model architectures, training paradigms, and alignment strategies. These studies highlight advances in reasoning, efficiency, and safety mechanisms, and provide a broader context for understanding the limitations of current models. In contrast to approaches focused on improving generation quality and alignment during training, our work addresses post hoc inference-time safety through uncertainty-based intervention.

Recent approaches such as retrieval-augmented generation (RAG), self-consistency, and tool-augmented LLMs aim to improve factual correctness by enhancing the generation process. These methods primarily operate at the pre-generation or generation stage by incorporating external knowledge or multiple reasoning paths. In contrast, our framework focuses on post hoc inference-time safety by intercepting uncertain outputs before they reach the user. Therefore, these approaches are complementary and can be integrated with the proposed framework rather than directly compared.

3. Materials and Methods

This section provides an extensive discussion of the proposed framework. We begin by discussing the datasets used in this work, explaining their properties, intended purpose, and the reasoning behind selecting various datasets of varying complexity. We then present the HRS signals, which are the work’s main contribution, in detail, including each signal’s description and role in quantifying model uncertainty. We then show the agentic pipeline, which serves as the operational backbone of our post hoc hallucination mitigation technique, in which uncertainty signals are employed iteratively to direct clinical knowledge injection before committing to an accurate answer. Finally, we specify the metrics used to evaluate the safety, accuracy, and efficiency of the proposed system.

3.1. Datasets

We tested our approach on three biomedical question-answering datasets of various complexity to ensure generalizability and robustness across difficulty levels while adhering to the medical domain. Each dataset uses a defined answer format that aligns with our framework’s yes/no/maybe categorization objective. Table 1 summarizes critical statistics and label distribution for each dataset.

Table 1. Dataset Statistics and Label Distribution.

Dataset	Difficulty	Source	Total	Train (80%)	Test (20%)	Yes (%)	No (%)	Maybe (%)	Label Format
PQA-L	Hard	Expert Annotated	1000	800	200	55.2%	33.8%	11%	Yes/No/Maybe
PQA-A	Easy	Auto Generated	211,269	800	200	93.6%	6.4%	N/A	Yes/No/Maybe
BioASQ-B	Medium	Expert Curated	1464	800	200	73.1%	26.9%	N/A	Yes/No

3.1.1. PubMedQA-Labeled (PQA-L)

PQA-L [22] is a collection of 1000 expert-annotated biomedical research questions from PubMed abstracts. Clinical domain experts provide yes/no/maybe labels to each question. This dataset is the most difficult tier in our evaluation, with fewer but higher-quality annotations that require advanced clinical reasoning. PQA-L is the primary standard for threshold sensitivity and orchestrator comparisons due to its clinical validity and complexity, assuring consistency across all analyses. PQA-L is the only dataset in our review with a three-class label scheme that includes maybe responses, highlighting the inherent ambiguity in expert-annotated clinical research questions.

3.1.2. PubMedQA-Artificial (PQA-A)

PubMedQA-A [22] is a collection of 211,269 expert-annotated biomedical research questions from PubMed abstracts. Clinical domain experts provide yes/no labels to these questions. This dataset is the most demanding in our study, with fewer but higher-quality annotations that require deep clinical reasoning. PQA-L is the fundamental standard for threshold sensitivity and orchestrator comparisons, assuring consistency across all analyses due to its clinical validity and complexity.

3.1.3. BioASQ Task B (BioASQ-B)

BioASQ-B [23] is a manually curated collection of 1464 yes/no questions from PubMed literature, confirmed by biomedical specialists. BioASQ-B differs from PubMedQA in that it has just yes/no labels and no “maybe” category. To ensure pipeline consistency, the framework’s output instruction was changed from yes/no/maybe to yes/no for this dataset only, without affecting other components. BioASQ-B adds a medium difficulty tier and a unique annotation methodology, demonstrating that our framework is applicable beyond the PubMedQA dataset family.

3.1.4. Dataset Splits and Preprocessing

All three datasets had a consistent 80/20 train-test split, resulting in 800 training and 200 test samples each. The fixed split size was chosen to ensure comparability among datasets, especially for PQA-L, which is limited in size. To address class imbalance, stratified sampling [24] was used during splitting. This ensured that the label distribution in training and test sets accurately reflected the total dataset distribution, as shown in Table 1. Section 4.6 provides empirical justification for categorizing datasets as hard, medium, or easy. Six independent machine learning classifiers show a direct correlation between dataset difficulty, model uncertainty, and escalation rate.

3.2. Hallucination Risk Signal (HRS) Design

LLMs for medical question-answering present the difficulty of producing confident but wrong responses, known as hallucination. To address this, we designed five complementary HRSs, each expressing a unique level of model uncertainty during inference. We use many signals instead of a single measure because LLM uncertainty can show in various forms, such as decision-level ambiguity and topic-level rarity. A single signal cannot capture all of these dimensions simultaneously.

3.2.1. Entropy HRS

Entropy HRS measures the model’s uncertainty at the final yes/no/maybe decision border. LLMs are probabilistic systems, therefore each generated token has a probability distribution throughout the vocabulary [25]. To calculate binary entropy, we extract the probability values assigned to yes and no tokens during the initial generation stage as shown below:

$$H_{entropy}=-\left(p_{yes}{log}_2{p}_{yes}+p_{no}{log}_2{p}_{no}\right)$$

(1)

$p_{yes}$ and $p_{no}$ represent the normalized probability of yes and no tokens, respectively. A high entropy value suggests uncertainty at the decision boundary, indicating the need for escalation. The entropy score is normalized to the range of $[0,1]$.

3.2.2. Perplexity HRS

Entropy HRS measures uncertainty at the decision token, while Perplexity HRS [26] assesses total confidence in the generated response, including final decision and supporting reasoning. A high perplexity score suggests that the model generates tokens with low confidence, indicating unclear reasoning. Token-level perplexity is calculated using generation scores and normalized to $[0,1]$, as follows:

$$H_{perplexity}=1-{\displaystyle \frac{1}{PPL}}$$

(2)

where $PPL=exp\left(-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^{N}logp\left(t_i\right)\right)$ and $p\left(t_i\right)$ is the probability of the i-th generated token.

3.2.3. Ambiguity HRS

LLMs are probabilistic systems, meaning they can create different outcomes based on the same input in multiple runs [27]. Ambiguity HRS takes advantage of this trait by repeating the same query three times with a sample temperature of 0.3 and calculating the difference between the returned token probability distributions. We utilize Jensen–Shannon (JS) divergence instead of Kullback–Leibler (KL) divergence [28] because it is symmetric and bounded, making it ideal for comparing various distributions.

$$H_{ambiguity}={\displaystyle \frac{1}{3}}\left(JS(P_1\parallel P_2)+JS(P_1\parallel P_3)+JS(P_2\parallel P_3)\right)$$

(3)

$P_1$, $P_2$, and $P_3$ denote the token probability distributions from three independent runs. A high ambiguity score suggests that the model gives inconsistent results, indicating confusion regarding the correct answer.

3.2.4. Rarity HRS

The proportion of medical queries varies dramatically between training programs. Questions regarding rare conditions or odd clinical scenarios may be beyond the model’s reliable knowledge boundaries, raising the chance of hallucinations. The Rarity HRS calculates the cosine distance [29] between a test question’s embedding and the centroid of all training embeddings to determine its distance from the training data distribution.

$$H_{rarity}=1-cos\left({\mathbf{e}}_{test},{\overline{\mathbf{e}}}_{train}\right)$$

(4)

${\mathbf{e}}_{test}$ is the BioMistral-7B hidden state embedding of the test question, while ${\overline{\mathbf{e}}}_{train}$ is the mean embedding computed across all training cases. A high rarity score suggests that the test question is not semantically related to the training distribution, indicating a lack of dependable information for confident answers.

3.2.5. Token Length HRS

The length of a model’s response implies confidence. When a model is uncertain about an answer, it generates lengthier and more intricate responses to develop a reason, similar to a student writing extensively when doubtful [30]. Token Length HRS measures the normalized length of generated reasoning:

$$H_{token}=min\left(1,{\displaystyle \frac{|{\mathrm{tokens}}_{reason}|}{L_{max}}}\right)$$

(5)

where $|{\mathrm{tokens}}_{reason}|$ represents the number of tokens in the generated reasoning, and $L_{max}=50$ indicates the maximum generation length.

3.2.6. Two-Layer Architecture

The five HRS signals are divided into two functional layers based on statistical significance, as validated by Mann–Whitney U tests (see Section 4.3):

Decision Layer comprises Entropy HRS and Perplexity HRS, which act as the principal escalation gates. These signals determine whether a case should be escalated to human review or proceed via the agentic loop based on the threshold condition.

$$\mathrm{Escalate}\mathrm{if}{H}_{entropy}\ge {\tau }_e\mathrm{AND}{H}_{perplexity}\ge {\tau }_p$$

(6)

The decision-layer rule requires both entropy and perplexity to exceed predefined thresholds before escalation is triggered. This design is motivated by the complementary roles of the two signals: entropy captures uncertainty at the final decision boundary, while perplexity reflects uncertainty across the generated reasoning sequence. Requiring both conditions ensures that escalation occurs only when the model is uncertain both in its prediction and its reasoning, reducing false positives caused by noise in a single signal.

The thresholds for entropy and perplexity are selected through empirical sensitivity analysis, as shown in Section 4.1, and are treated as tunable hyperparameters. This allows the system to adapt to different deployment requirements by controlling the trade-off between safety (higher escalation) and efficiency (lower escalation). In practical settings, these thresholds can be calibrated based on acceptable risk levels and clinical workload constraints.

Guidance Layer includes Ambiguity, Rarity, and Token Length HRS. Instead of influencing the escalation decision, these signals provide context to the GPT orchestrator, guiding clinical knowledge injection. A high Rarity score indicates an unusual medical issue and prompts the orchestrator to provide definitional background. A high Ambiguity score indicates inconsistent model behavior and prompts the orchestrator to offer additional grounding facts. Section 4.3 provides a statistical rationale for the two-layer separation.

3.3. Agentic Pipeline

3.3.1. Overview

Our suggested methodology relies on the agentic pipeline to mitigate post hoc uncertainty before responding to users. The pipeline assesses the initial LLM output and uses targeted clinical knowledge injection to limit the likelihood of hallucinations. Figure 1 depicts the overall system design.

Figure 1. Overview of the Agentic HRS System Architecture for Evaluating Hallucination Risk in Medical LLMs. The pipeline consists of two major components: (1) HRS Signal Computation, in which BioMistral-7B derives five uncertainty signals from model internals, and (2) the LLM Orchestrator, which receives all signal values and repeatedly injects clinical knowledge to minimize uncertainty. When confidence thresholds are met, the algorithm exits with SUCCESS, whereas ESCALATE refers the case to a clinician after maximum repetitions. The Revert Baseline protection system stresses patient safety by preventing any deterioration.

The pipeline overcomes a drawback of single-pass LLM inference, where a model may return a right answer with low confidence or an inaccurate response with high confidence. Neither outcome is appropriate in clinical settings. The agency loop monitors uncertainty signals and intervenes when confidence levels are not met, either by refining the question or escalation to a human expert for cases that cannot be reliably predicted by automation.

3.3.2. Pipeline Flow

The pipeline operates as follows. BioMistral-7B provides an initial yes/no/maybe response based on the user’s medical inquiry and context. All five HRS signals are computed from the model’s internal states, as detailed in Section 3.2. This is the first pipe in the system.

The generated signals are tested against decision layer thresholds. If both Entropy HRS and Perplexity HRS are below their respective thresholds ${\tau }_e$ and ${\tau }_p$, the answer is regarded sufficiently confident and provided straight to the user without entering the agentic loop—a process we call the Baseline Gate. This early exit strategy is critical from a deployment perspective, as it prevents unnecessary latency for cases where the model is already confident.

If the thresholds are not satisfied, the GPT orchestrator receives all five HRS signal values, including the current question and iteration history. The orchestrator optimizes prompts by assessing risk patterns and injecting clinical information to create better questions. The orchestrator performs one of two actions:

• CONTINUE: The orchestrator adds clinical facts to the inquiry and improves the prompt for the following iteration.
• ABSTAIN: The orchestrator escalates the situation when the query is deemed unanswerable based on clinical assumptions.

This procedure has a maximum of five iterations. If the decision layer thresholds are reached at any point, the pipeline will leave with a success status. If the maximum number of iterations is reached without fulfilling the thresholds, the issue will be escalated to a clinician.

3.3.3. Knowledge Injection vs. Rephrasing

Our orchestrator distinguishes between information injection and question rephrasing, which is an important design decision. Rephrasing simply changes the wording of a query while maintaining its semantic substance, without adding any new information to the model. Rephrasing does not reduce entropy or perplexity since the model’s uncertainty is caused by insufficient clinical knowledge, not language ambiguity.

Knowledge injection involves incorporating clinically confirmed facts into a question, such as mechanisms of action, epidemiological evidence, or clinical guidelines. This provides the model with additional information to reason about. Our data show that information injection consistently reduces entropy, whereas rephrasing does not.

The knowledge injection process is implemented through structured prompt-based augmentation, where the orchestrator introduces clinically relevant facts, assumptions, or definitions into the query to guide the downstream model toward a more confident response. This knowledge is not retrieved from external databases but is generated using the model’s internal medical knowledge. While this approach enables flexible and lightweight deployment without external dependencies, it may introduce variability depending on the model’s internal representations and reasoning behavior.

3.3.4. Baseline Gate

The Baseline Gate is an early exit mechanism that assesses the initial model response before entering the agentic loop. Formally, if:

$$H_{entropy}<0.5\mathrm{AND}{H}_{perplexity}<0.4$$

(7)

the user receives the answer directly without the orchestrator’s involvement. This approach serves two purposes: it decreases computational cost for simple scenarios and avoids unwanted intervention on issues that the model can confidently answer, preserving efficiency and accuracy.

3.3.5. Revert Baseline Protection

Our pipeline includes Revert Baseline protection, which prevents the agentic loop from degrading a proper baseline answer. Consider the scenario where the model correctly replies yes in the first iteration, but after orchestrator intervention, produces no in subsequent iterations. Without protection, the system would accept the incorrect post-intervention answer.

The Revert Baseline mechanism monitors the baseline response and its accuracy throughout the loop. If the post-intervention answer differs from the right baseline, the system will revert to the original answer. This approach enforces zero degradation through the Revert Baseline mechanism, ensuring that the agentic pipeline does not worsen a valid answer, which is crucial for clinical implementation.

3.3.6. Escalation

Cases are escalated to human expert review under two conditions: (1) the GPT orchestrator provides an ABSTAIN action, indicating the question is truly unanswerable, or (2) the maximum number of iterations is achieved without satisfying the decision layer thresholds. The system prioritizes patient safety over throughput by returning an ESCALATED status rather than an automated response. The escalation rate can be adjusted by setting the thresholds ${\tau }_e$ and ${\tau }_p$, as explained in Section 4.1.

3.3.7. Algorithmic Formulation

To formally describe the proposed framework, Algorithm 1 summarizes the agentic Hallucination Risk Scoring (HRS) pipeline. The procedure captures the interaction between uncertainty estimation, decision-layer evaluation, and iterative-knowledge-guided refinement.

Algorithm 1 Agentic HRS Pipeline

1. Input: Medical query q 2. Generate initial response $r_0$ using the base LLM. 3. Compute HRS signals $\{H_e,H_p,H_a,H_r,H_t\}$. 4. If $H_e<{\tau }_e$ and $H_p<{\tau }_p$, return $r_0$ (Baseline Gate). 5. For $i=1$ to K: (a) Use the orchestrator to inject clinical knowledge and refine the query. (b) Generate updated response $r_i$. (c) Recompute HRS signals. (d) If $H_e<{\tau }_e$ and $H_p<{\tau }_p$, return $r_i$. 6. If no iteration satisfies the thresholds, return Escalate and refer the case to a clinician.

The maximum number of iterations K is selected based on a trade-off between uncertainty reduction and computational latency. Increasing the number of iterations can further reduce uncertainty; however, empirical observations indicate diminishing returns beyond a small number of iterations, while latency increases linearly due to repeated LLM inference. Therefore, a fixed upper bound (e.g., $K=5$) is used to balance performance improvement and practical deployment efficiency.

The computational complexity of the proposed pipeline is $\mathcal{O}(K·C)$, where K is the maximum number of iterations and C denotes the cost of a single LLM inference and HRS computation. Since K is bounded by a small constant, the overall complexity remains linear in the cost of inference, making the framework suitable for real-time or near-real-time clinical applications.

3.4. Evaluation Metrics

Our framework is evaluated based on five measures that consider both safety and efficiency during clinical implementation. Medical AI evaluation requires more than just standard accuracy. A system that confidently answers ambiguous questions poses a higher danger than one that appropriately escalates uncertain cases [31]. Unlike standard classification settings where every instance receives a prediction, our framework introduces a selective decision mechanism that may escalate uncertain cases instead of producing an automated response. As a result, traditional metrics such as F1-score, AUROC, and calibration error (ECE) do not fully capture the safety-oriented behavior of the system. Therefore, we adopt evaluation metrics specifically designed for this setting, focusing on the trade-off between safety, accuracy, and escalation.

Our metrics evaluate the framework’s capacity to prioritize patient safety while ensuring practical deployment.

3.4.1. Safety Gain

The Safety Gain metric identifies and escalates false baseline answers, preventing them from reaching the user:

$$\mathrm{Safety}\mathrm{Gain}={\displaystyle \frac{|\mathrm{Escalated}\cap \mathrm{Wrong}\mathrm{Baseline}|}{\left|\mathrm{Wrong}\mathrm{Baseline}\right|}}$$

(8)

A higher Safety Gain means that the framework intercepts more potentially damaging wrong replies before presenting them to the user. This metric aligns with our framework’s key safety purpose of preventing incorrect replies from reaching patients.

3.4.2. Auto-Answer Accuracy

Auto-Answer Accuracy assesses prediction accuracy for non-escalated cases, when the framework provides an automatic response:

$$\mathrm{Auto}-\mathrm{Answer}\mathrm{Accuracy}={\displaystyle \frac{{\left|\mathrm{Correct}\mathrm{Predictions}\right|}_{\mathrm{non}-\mathrm{escalated}}}{\left|\mathrm{Non}-\mathrm{Escalated}\mathrm{Cases}\right|}}$$

(9)

Escalated cases are deliberately excluded from this metric because they are referred to clinicians rather than answered automatically. Auto-Answer Accuracy measures the framework’s reliability on specific circumstances, providing a more accurate assessment of system performance than global accuracy.

3.4.3. Escalation Rate

Escalation Rate measures the proportion of test cases referred to human expert review:

$$\mathrm{Escalation}\mathrm{Rate}={\displaystyle \frac{\left|\mathrm{Escalated}\mathrm{Cases}\right|}{\left|\mathrm{Total}\mathrm{Cases}\right|}}$$

(10)

Interpreting escalation rates requires balancing patient safety and clinical workload considerations. Increasing the escalation rate can improve safety by assigning more complex cases to clinicians, but it can also increase the workload for medical professionals. Lowering the escalation rate improves system autonomy and efficiency, but may result in automated responses for ambiguous instances. The ideal escalation rate depends on deployment situation and is determined by the tunable thresholds ${\tau }_e$ and ${\tau }_p$, as described in Section 4.1.

3.4.4. Degradation Rate

Degradation Rate measures the proportion of cases where the agentic loop converted a correct baseline answer into an incorrect final answer:

$$\mathrm{Degradation}\mathrm{Rate}={\displaystyle \frac{{|\mathrm{Correct}\mathrm{Baseline}\cap \mathrm{Incorrect}\mathrm{Final}|}_{\mathrm{non}-\mathrm{escalated}}}{\left|\mathrm{Total}\mathrm{Cases}\right|}}$$

(11)

Degradation in medical AI poses a significant risk, as a system that occasionally transforms right responses to incorrect ones cannot be trusted in clinical settings, regardless of overall accuracy. Our Revert Baseline protection system, outlined in Section 3.3.5, ensures 0% degradation under all experimental scenarios. This is our strongest safety assurance, as it guarantees that the agentic pipeline will never make a correct solution worse.

3.4.5. Cost per Call and Latency

In addition to accuracy and safety metrics, practical deployment also demands consideration of computational efficiency. Our report includes two efficiency measures for each orchestrator configuration:

Cost per Call metric calculates the average API cost per orchestrator invocation based on OpenAI API input and output token counts:

$$\mathrm{Cost}={\displaystyle \frac{N_{input}\times P_{input}+N_{output}\times P_{output}}{{10}^6}}$$

(12)

The average token counts are $N_{input}$ and $N_{output}$, while the pricing rates per million tokens for each model are $P_{input}$ and $P_{output}$, respectively.

Latency calculates the average wall-clock time per test example, encompassing BioMistral inference, HRS calculation, and orchestrator calls. Cost per Call and Latency give a comprehensive cost-performance profile for orchestrator configurations, allowing for informed decision-making for clinical implementation.

4. Results and Quantitative Analysis

This section gives experimental results based on four main analyses: threshold sensitivity, orchestrator comparison, statistical signal validation, and dataset complexity validation. All trials use the ideal threshold configuration of ${\tau }_e=0.95$ and ${\tau }_p=0.30$, proving the framework enforces zero degradation through the Revert Baseline mechanism described in Section 3.3.5.

4.1. Threshold Sensitivity Analysis

The decision layer thresholds ${\tau }_e$ and ${\tau }_p$ determine the balance between patient safety and system autonomy. Table 2 summarizes our evaluation of three threshold configurations across three datasets to understand the trade-off.

Table 2. Threshold Sensitivity Analysis Across Three Datasets. Degradation Rate = 0 across all configurations.

Dataset	Threshold (${\mathit{\tau }}_{\mathit{e}}$/${\mathit{\tau }}_{\mathit{p}}$)	Safety Gain	Escalation Rate	Auto-Answer Acc.	Baseline Acc.
PubMedQA-L	0.80/0.40	36%	24%	71%	64%
PubMedQA-L	0.95/0.30	14%	8%	68%	64%
PubMedQA-L	0.98/0.30	6%	3%	66%	64%
PubMedQA-A	0.80/0.40	38%	20%	96%	94%
PubMedQA-A	0.95/0.30	8%	3%	95%	94%
PubMedQA-A	0.98/0.30	15%	2%	95%	94%
BioASQ-B	0.80/0.40	32%	16%	90%	84%
BioASQ-B	0.95/0.30	6%	4%	88%	84%
BioASQ-B	0.98/0.30	2%	3%	86%	84%

The results show a consistent safety–accuracy trade-off across all datasets and threshold setups. Stricter thresholds reduce escalation rate but decrease safety gain, while loosened thresholds increase safety interception at the expense of higher escalation. The auto-answer accuracy consistently exceeds baseline accuracy in all configurations, indicating that the framework improves on the base model regardless of threshold selection.

Applying the same criterion ${\tau }_e=0.95$ and ${\tau }_p=0.30$ across all three datasets results in varying escalation rates: 8% for PubMedQA-L, 4% for BioASQ-B, and 3% for PubMedQA-A. The link between dataset difficulty and escalation rate shows that the framework prioritizes difficult data over easier data, rather than a predetermined escalation policy based on query difficulty. Section 4.6 provides empirical validation for the complexity ranking.

Based on these results, we select ${\tau }_e=0.95$ and ${\tau }_p=0.30$ as the optimal configuration, offering a balanced trade-off between safety gain and escalation rate across all three datasets.

4.2. Orchestrator Comparison

To assess how orchestrator quality affects framework performance, we compared three GPT-family models as orchestrators while leaving all other components the same. Table 3 summarizes the results for PubMedQA-L at the ideal thresholds of ${\tau }_e=0.95$ and ${\tau }_p=0.30$.

Table 3. Orchestrator Comparison on PubMedQA-L (${\tau }_e=0.95$, ${\tau }_p=0.30$).

Model	Safety Gain	Esc. Rate	Auto Acc.	Avg Input	Avg Output	Cost/Call	Latency
GPT-4o-mini	14%	8%	68%	1050	110	$0.000224	9.73 s
GPT-4o	12%	10%	66%	1088	125	$0.003976	9.31 s
o3-mini	11%	8%	68%	1098	540	$0.003585	11.77 s

GPT-4o-mini outperforms all other orchestrators in terms of safety gain (14%), escalation rate (8%), and auto-answer accuracy (68%). Additionally, it has the lowest cost per call ($0.000224).

o3-mini, a reasoning-optimized model, produces 540 output tokens per call, five times more than GPT-4o-mini (110 tokens), but does not enhance safety or accuracy metrics significantly. Extensive chain-of-thought reasoning is ineffective for structured clinical knowledge injection tasks. Instead, brief and targeted fact injection is more effective. GPT-4o-mini is chosen as the default orchestrator for all future trials.

4.3. Statistical Signal Validation

To validate individual HRS signals, we use Mann–Whitney U tests to compare signal values between escalated and non-escalated patients across all three datasets. The null hypothesis $H_0$ indicates no significant change in signal values between escalated and non-escalated situations. The findings are summarized in Table 4.

Table 4. Mann–Whitney U test results ($p<0.05$ significance level). ✓ indicates statistical significance, and × indicates non-significance. Esc = escalated mean; Non = non-escalated mean.

Signal	PubMedQA-L			PubMedQA-A			BioASQ-B
	Esc	Non	Sig	Esc	Non	Sig	Esc	Non	Sig
Entropy	0.99	0.64	✓	0.99	0.65	✓	0.97	0.58	✓
Perplexity	0.21	0.20	×	0.21	0.16	×	0.27	0.16	✓
Ambiguity	0.51	0.56	×	0.70	0.50	×	0.54	0.39	×
Rarity	0.29	0.28	×	0.24	0.25	×	0.26	0.28	×
Token Length	0.81	0.83	×	0.86	0.87	×	0.80	0.71	×

While entropy is the only consistently significant signal across all datasets, this fact supports our two-layer approach. Entropy (and perplexity) are the major decision signals for escalation, whereas ambiguity, rarity, and token length are intended to guide the orchestrator. These signals are not designed to drive escalation decisions on their own, but rather to provide contextual information during iterative refining as described in Section 3.2.

4.4. Robustness Validation via Multi-Seed Analysis

To address concerns about statistical robustness and reproducibility, we carry out more trials with multiple random seeds. We analyze the proposed framework using five independent runs (seeds = 30, 42, 100, 126, and 999), reporting the mean and standard deviation for all essential evaluation criteria.

Table 5 summarizes the results across three datasets with varied complexity: PubMedQA-L (hard), BioASQ-B (middle), and PubMedQA-A (easy). The results show that the proposed architecture remains stable across many stochastic initializations, with minimal variance in key metrics like Auto-Answer Accuracy and Escalation Rate.

Table 5. Multi-Seed Robustness Analysis Across Datasets.

Dataset	Safety Gain	Auto-Acc	Baseline	Esc. Rate	Input Tok.	Output Tok.	Latency (s)
PubMedQA-L	0.099 ± 0.030	0.657 ± 0.017	0.635 ± 0.020	0.078 ± 0.022	655.1 ± 15.8	91.3 ± 1.82	14.60 ± 1.40
BioASQ-B	0.154 ± 0.065	0.860 ± 0.023	0.832 ± 0.028	0.061 ± 0.010	661.1 ± 8.13	91.4 ± 2.41	12.44 ± 0.80
PubMedQA-A	0.113 ± 0.069	0.950 ± 0.016	0.939 ± 0.015	0.040 ± 0.014	658.0 ± 13.5	96.1 ± 2.08	11.41 ± 0.53

We observe that the standard deviation for key measures like Auto-Answer Accuracy and Escalation Rate is relatively low across all three datasets, demonstrating that the proposed framework remains stable across multiple random seeds. This implies that the model’s performance is consistent and insensitive to random fluctuations in generation.

Furthermore, across all datasets, the suggested technique consistently outperforms the baseline model in terms of auto-answer accuracy, confirming the agentic framework’s usefulness in enhancing dependable predictions. All experiments use a fixed threshold configuration (${\tau }_e=0.95$, ${\tau }_p=0.30$), which ensures that observed behavior reflects inherent model properties rather than threshold adjustment.

Beyond stability, we investigate how the system performs on datasets of varied complexity. We see a definite trend in escalation rates, with PubMedQA-L (hard) having the highest escalation, then BioASQ-B (medium), and PubMedQA-A (easy) having the lowest. This illustrates that the framework’s escalation behavior changes depending on the difficulty of the dataset, demonstrating its complexity-aware design.

4.5. Ablation Study of HRS Signals

We conduct an ablation investigation utilizing a leave-one-signal-out (LOSO) method to examine the contribution of individual Hallucination Risk Signals (HRSs) as shown in Table 6. In every experiment, one signal is eliminated while the other elements remain unaltered. The PubMedQA-L dataset is used for all experiments, and the same threshold configuration (${\tau }_e=0.95$, ${\tau }_p=0.30$) is used. The findings are presented as mean and standard deviation across several seeds.

Table 6. Ablation Study on HRS Signals (PubMedQA-L).

Configuration	Safety Gain	Auto-Acc	Esc. Rate
Full Model	0.099 ± 0.030	0.657 ± 0.017	0.078 ± 0.022
No Entropy	0.000 ± 0.000	0.650 ± 0.004	0.000 ± 0.000
No Perplexity	0.065 ± 0.040	0.659 ± 0.010	0.047 ± 0.012
No Ambiguity	0.075 ± 0.043	0.657 ± 0.007	0.062 ± 0.023
No Rarity	0.089 ± 0.008	0.658 ± 0.005	0.073 ± 0.012
No Token Length	0.080 ± 0.021	0.659 ± 0.006	0.062 ± 0.013

The ablation investigation shows that entropy has a significant impact on both Safety Gain and Escalation Rate. Specifically, removing entropy causes the escalation mechanism to completely collapse and eliminates safety improvements, demonstrating that entropy is the framework’s fundamental driving signal.

Removing perplexity results in a significant reduction in both Safety Gain and Escalation Rate, indicating that it serves a crucial supportive function in the decision layer. In contrast, deleting ambiguity, rarity, and token length resulted in relatively slight performance losses, implying that these signals play an auxiliary role.

These findings are consistent with the proposed two-layer design, in which entropy and perplexity comprise the decision layer responsible for escalation, and ambiguity, rarity, and token length serve as guidance signals to the orchestrator. Overall, the ablation results demonstrate that the multi-signal framework acts in complementary roles rather than with equal contributions from all signals.

4.6. Dataset Complexity Validation

To provide model-agnostic validation that our dataset difficulty categorization reflects genuine data complexity rather than LLM-specific behavior, we evaluate six independent ML classifiers on BioMistral-7B embeddings across all three datasets. This analysis provides model-independent validation that the escalation decisions produced by the HRS framework are aligned with intrinsic dataset complexity, rather than being driven by arbitrary or fixed thresholds. The results are summarized in Table 7.

Table 7. ML Classifier Performance Across Datasets. Average reflects mean across all six classifiers.

Dataset	Model	Accuracy	Precision	Recall	F1-Score
PubMedQA-L (Hard)	Logistic Regression	0.485	0.478	0.485	0.480
	Random Forest	0.545	0.303	0.545	0.389
	Gradient Boost	0.535	0.448	0.535	0.462
	SVM	0.545	0.440	0.545	0.446
	KNN	0.505	0.466	0.505	0.480
	MLP	0.535	0.504	0.535	0.512
	Average	0.525	0.440	0.525	0.461
BioASQ-B (Medium)	Logistic Regression	0.755	0.742	0.755	0.746
	Random Forest	0.725	0.526	0.725	0.609
	Gradient Boost	0.780	0.799	0.780	0.730
	SVM	0.800	0.829	0.800	0.759
	KNN	0.770	0.755	0.770	0.756
	MLP	0.760	0.753	0.760	0.756
	Average	0.765	0.734	0.765	0.726
PubMedQA-A (Easy)	Logistic Regression	0.915	0.887	0.915	0.890
	Random Forest	0.915	0.837	0.915	0.874
	Gradient Boost	0.915	0.837	0.915	0.874
	SVM	0.915	0.837	0.915	0.874
	KNN	0.900	0.857	0.900	0.874
	MLP	0.915	0.857	0.915	0.874
	Average	0.913	0.849	0.913	0.877

The results show that PubMedQA-L has the lowest average accuracy (52.5%), followed by BioASQ-B (76.5%) and PubMedQA-A (91.3%). The consistent ranking across six separate classifiers demonstrates that the observed difficulty variations are due to dataset complexity, not LLM-specific behavior.

The complexity ranking matches the escalation rates reported at the ideal threshold (${\tau }_e=0.95$, ${\tau }_p=0.30$): 8% for PubMedQA-L, 4% for BioASQ-B, and 3% for PubMedQA-A. Our framework accurately adjusts escalation based on data complexity, as evidenced by the negative link between ML accuracy and HRS escalation rate across models.

4.7. Escalated Cases Analysis

To ensure that escalated cases are truly ambiguous medical problems, we evaluated ML classifier performance on HRS-escalated cases from all three datasets, as reported in Table 8.

Table 8. ML Model Performance on HRS-Escalated Cases.

Model	PQA-L	BioASQ	PQA-A
Logistic Regression	54.5%	77.8%	88.9%
Random Forest	18.2%	44.4%	77.8%
Gradient Boost	45.5%	77.8%	77.8%
KNN	36.4%	88.9%	77.8%
MLP	63.6%	77.8%	88.9%
Average	43.6%	73.3%	82.2%

ML classifier performance on escalated cases follows the same complexity hierarchy as the whole dataset analysis. Classifiers acquire just 43.6% average accuracy on PubMedQA-L elevated examples, falling below the random chance criterion for a balanced three-class problem. This confirms the ambiguity of the cases, even for independent ML approaches. Random Forest, the best-calibrated uncertainty estimator among our classifiers, reaches just 18.2% on PubMedQA-L escalated instances, indicating that HRS escalation accurately identifies cases beyond automated prediction.

The continuous worsening of ML performance on escalated cases across all three datasets confirms that HRS escalation decisions are based on true medical ambiguity rather than conservative system behavior, regardless of model. Our objective is to use AI to assist patients, not mislead them. This means withholding uncertain responses and avoiding false confidence.

5. Discussion and Limitations

5.1. Discussion

This work introduces an agentic architecture to reduce post hoc hallucination risk in medical question answering, filling a crucial gap in current medical AI deployment. Our method to preventing hallucinations involves intercepting questionable outputs before they reach patients and escalating ambiguous situations to clinical professionals, rather than attempting to prevent them at the model level, which is a challenging task. This post hoc safety paradigm is a practical and deployable strategy to ensure trustworthy medical AI.

Zero Degradation Guarantee: This study’s key discovery is the consistent 0% degradation rate across nine runs across three datasets and threshold setups because of the revert baseline mechanism, which was explicitly designed to prevent the agentic pipeline from converting the correct response into the incorrect response during the iterative refinement. In clinical contexts, a system that periodically transforms accurate answers to incorrect ones poses an unacceptable danger, regardless of overall accuracy. Therefore, this design acts as a safety constraint, ensuring that the system preserves valid baseline answers while still allowing improvements in uncertain cases. This mechanism aligns with the primary objective of minimizing harmful outputs in medical AI deployment.

Knowledge Injection over Rephrasing: GPT-4o-mini outperforms GPT-4o and o3-mini as an orchestrator, despite being the smallest and lowest-priced device tested. This surprising finding highlights a key point concerning clinical knowledge injection: more knowledge does not always equal better injection. A model with broad knowledge may introduce facts that transcend the downstream model’s reasoning capacity, resulting in complication rather than clarity. GPT-4o-mini’s tailored fact injection outperforms extensive reasoning chains in guiding medical LLM reactions.

Complexity-Aware Escalation: Our framework accurately adjusts escalation based on data complexity, as evidenced by the negative association between ML classifier accuracy and HRS escalation rate across three datasets of increasing difficulty. This conclusion has significant implications for future medical AI systems, as a single fixed safety threshold is unsatisfactory in various clinical contexts. Expert-annotated research questions demand greater escalation rates compared to consumer-level queries. Our configurable threshold architecture enables context-aware deployment.

Entropy as Universal Signal: Entropy HRS is the most significant predictor of escalation across all three datasets (p < 0.001), making it the dominant uncertainty signal for medical yes/no/maybe question answering. This observation confirms the two-layer signal architecture and provides a solid foundation for future threshold design.

5.2. Limitations

Single Modality: The current architecture relies solely on textual input. Medical diagnosis relies on multiple data sources, such as imaging, test results, and clinical notes. Extending the HRS framework to a multimodal agentic system is a promising path for future research.

Answer Format Constraints: The framework focuses on answering yes/no/maybe questions. HRS signals are derived using the model’s long-form reasoning, but escalation decisions are based on binary classification uncertainty. To accommodate open-ended answers, the entropy and perplexity signal computations would need to be redesigned.

Dataset Limitations: The current evaluation is conducted on structured biomedical question-answering datasets, which may not fully reflect real-world clinical workflows involving long-form reasoning, multimodal inputs, or noisy data. While these datasets provide a controlled environment for evaluating hallucination risk and uncertainty, future work should extend the framework to more realistic clinical settings to assess its applicability in practice.

Base Model Selection: BioMistral-7B was chosen as the foundation model due to its open-source nature and proven performance in biomedical benchmarking. The goal of this work was to show the usefulness of the agentic safety framework, rather than optimizing base model performance. Using larger or more capable medical LLMs, such as MedLLaMA or Clinical-Camel, may result in distinct escalation characteristics and provide a potential route for further research.

Clinical Deployment Validation: Our system has demonstrated high empirical performance across many datasets. However, real-world clinical deployment requires validation of the entire pipeline under clinician supervision. Before deployment, it is crucial to conduct a prospective evaluation in a clinical context, including assessing medical professionals’ treatment of escalated cases. This is an important direction for future development.

Limitations of Knowledge Injection: The current framework relies on prompt-based knowledge injection derived from the model’s internal knowledge rather than standardized or externally validated clinical sources. This may introduce variability in the injected content and potential bias depending on model behavior. While the approach is effective for reducing uncertainty in a lightweight and deployable manner, future work should explore integration with retrieval-based systems or curated medical knowledge bases to improve consistency, reliability, and clinical grounding.

6. Conclusions and Future Work

This paper presents an agentic post hoc hallucination risk mitigation paradigm for medical question answering, addressing a fundamental gap in current medical LLM deployment in which questionable outputs reach patients without being intercepted. Our methodology uses an LLM orchestrator to repeatedly inject clinically validated knowledge into uncertain inquiries, driving a domain-specific medical LLM towards confident and reliable solutions under configurable safety thresholds. Our framework’s zero degradation property, enforced by the Revert Baseline method, ensures that the agentic pipeline never changes a correct response into a wrong one across all experimental configurations. Furthermore, our complexity-aware escalation mechanism shows that the system dynamically adapts escalation rates based on dataset difficulty, as evaluated independently by six machine learning classifiers across three biological datasets. Statistical examination of the five HRSs demonstrated entropy to be the dominating and universal uncertainty signal across all datasets, laying a solid foundation for future threshold design in clinical deployments.

Looking ahead, there are several promising paths for further development. Extending the paradigm from single-modal text input to multimodal clinical data, including imaging and lab findings, is a logical next step. Expanding open-ended responses beyond the existing yes/no/maybe approach might increase clinical usefulness. Evaluating the framework with larger and more competent base models like MedLLaMA and Clinical-Camel would provide more information about scalability. Exploring integrating calibration-based evaluation alongside safety-oriented metrics would be beneficial. Most crucially, prospective clinical validation under physician supervision is still the critical approach to real-world implementation.