GuidelineGuard: An Agentic Retrieval-Augmented Generation Framework with Sentence-Level Citation Auditing for Guideline-Grounded Question Answering

Abstract

Background: Large language models (LLMs) can produce clinically plausible recommendations that are not adequately supported by authoritative evidence. Objectives: We introduce GuidelineGuard, a modular multi-agent retrieval-augmented generation pipeline in which a separate Auditor verifies claim–sentence support before a recommendation is surfaced. Methods: The original evaluation used a 73-sentence guideline corpus and GG-Bench-60, with replication across three open-weight backbones. In response to reviewer concerns about benchmark size and selective evaluation, we added a source-traceable GG-Bench-200 stress test and the complete 500-case held-out PQA-L test split of PubMedQA. The revision experiments compare single-pass RAG, a paired multi-agent no-Auditor ablation, and GuidelineGuard; the paired runner is designed to share the Planner–Retriever–Clinician draft so that the Auditor is the only intervention. Checkpoint verification confirmed an identical observable pre-audit state for all 200 GG-Bench cases and 496/500 PubMedQA cases; four PubMedQA cases were regenerated after quota-interrupted resumption and were correct commitments in both arms. Because the originally used hosted Llama endpoints became unavailable after the initial experiments, the expanded runs use openai/gpt-oss-20b for generation and openai/gpt-oss-120b for the Auditor. Results: On GG-Bench-200, single-pass RAG achieved 0.970 operational accuracy, while the no-Auditor and GuidelineGuard arms achieved 0.955 and 0.925, respectively. GuidelineGuard committed on 186/200 cases (coverage 0.930) and was correct on 185/186 commitments (selective accuracy 0.995); all 186 commitments cited at least one gold evidence identifier. Relative to the paired no-Auditor arm, the gate rejected six otherwise-correct commitments and no incorrect commitment. On PubMedQA-500, single-pass RAG achieved 0.644 operational accuracy at 0.950 coverage, the no-Auditor arm 0.638 at 0.896 coverage, and GuidelineGuard 0.550 at 0.736 coverage. Selective accuracy increased across those operating points from 0.678 to 0.712 to 0.747. Within the 496 PubMedQA cases with verified-identical observable pre-audit state, the gate rejected 36 incorrect and 45 correct pre-audit commitments, demonstrating both error enrichment and a substantial false-rejection cost. Conclusions: The expanded results support GuidelineGuard as a selective claim–evidence verification mechanism, not as a universally more accurate generator. Its value is the explicit, auditable coverage–risk trade-off; the appropriate verification threshold is task- and cost-dependent and requires prospective clinical validation.

1. Introduction

Clinical practice guidelines (CPGs) synthesize the best available evidence into actionable recommendations and are used as the standard against which clinical decisions are judged in audit, education, and litigation [1,2]. An established finding in implementation research is that guidelines and routine practice often drift apart [3,4]. The reasons are familiar: time pressure, fragmented information systems, and the difficulty of recalling the right recommendation at the right moment. The cost is measurable. Patients are under-prescribed evidence-based therapies, kept on contraindicated drug combinations, or escalated to higher levels of care too late.

Large language models (LLMs) seem like a good fit for this problem at first. Tools like Med-PaLM-2, GPT-4, and MedGemma [5,6,7] have shown strong results on medical exam-style questions, such as the USMLE. Because of this, people have started testing them in areas like clinical decision support, discharge summaries, patient triage, and documentation. Some surveys even suggest that a noticeable number of clinicians are already using general-purpose LLMs in their work, often without formal approval or oversight [8].

Several recent studies report that LLMs often produce medical answers that are unsupported, made up, or do not match established guidelines [8,9,10]. These “guideline hallucinations” are not merely a quality problem; in clinical settings, they can create direct safety risks. In other words, a confident but incorrect recommendation can directly affect patient care, and unlike errors in a research draft, clinicians may not catch these mistakes in real time. The problem is compounded by fluent, confident language that can obscure the absence of adequate evidentiary support [9].

Retrieval-augmented generation (RAG) is a common way to improve LLM outputs by grounding them in retrieved documents [11]. However, single-pass RAG still has several issues, especially in high-stakes settings. The model may ignore the retrieved evidence or change its meaning during generation [12], cite sources that do not actually support the claim [13], or combine pieces of information in ways that no single source actually recommends [14]. In clinical decision support, this is particularly risky because the presence of a citation can make an answer appear trustworthy even when the cited evidence does not support it. Under time pressure, clinicians seldom verify a cited answer against the original source, which is a tendency known as automation bias [15]. Recent surveys also place RAG within the broader evolution of AI text generation, retrieval, and detection technologies [16]. In clinical settings, however, retrieval alone does not guarantee that a generated claim is entailed by its cited source, motivating explicit claim–evidence verification.

We argue that guideline adherence should not depend on a single LLM call getting it right. It needs to be enforced by the system itself. To draw an analogy, hospitals do not rely on individual prescribers to be perfectly accurate; they rely on pharmacy checks, electronic alerts, and double-signatures. The verification of a recommendation is performed by a different actor with a different objective from the one who proposed it. We adopt the same principle in our system design.

We introduce GuidelineGuard, a multi-agent architecture in which the act of recommending and the act of verifying that the recommendation is supported by a specific guideline sentence are performed by separate agents with different objectives, separate contexts, and separate prompts. A claim is surfaced only if an independent verifier agent affirms that a specific cited sentence entails it, otherwise the claim is dropped or surfaced as a flagged fallback.

Contributions

This paper makes the following contributions.

• A verifier-in-the-loop architecture. A modular four-agent pipeline (Planner, Retriever, Clinician, Auditor), with a temperature-zero LLM Synthesizer, that makes guideline-citation faithfulness a structural property of the system. The Auditor sees only the claim and the candidate sentence, nothing else. This avoids the contamination that arises in reflection-based methods like Self-RAG [17], where the same model generates and then critiques its own output.
• An open guideline corpus. A curated, machine-readable corpus of 73 sentence-level clinical recommendations sourced from publicly available guidelines across 23 clinical topics, with full provenance metadata. The corpus is structured to enable both retrieval and sentence-level evidence adjudication.
• Source-traceable benchmark and expanded stress test. GG-Bench-60 contains 60 author-written multiple-choice cases with answer and sentence-level gold-evidence identifiers. For the major revision, we preserve those 60 cases verbatim and add 140 deterministic, source-traceable template cases to form GG-Bench-200. The expansion is a coverage and robustness stress test, not clinician-authored clinical ground truth; no independent clinician participated in construction or adjudication.
• Empirical evaluation. A head-to-head comparison against five baselines, one of which removes the Auditor to isolate its contribution and one of which (Self-RAG-style self-critique) tests whether the safety property requires separating the verifier from the generator. The results quantify the Auditor’s selective-gate trade-off rather than assuming that auditing improves raw citation faithfulness, and replicate the operating-point behavior across model scale and model family (Llama-3.1-8B, Llama-3.3-70B, and Qwen3.6-27B). We additionally validate the pipeline on an external, public benchmark, PubMedQA, that we did not author, where its calibrated-abstention behavior is preserved.
• Structured error analysis. We decompose GuidelineGuard’s accuracy gap into abstentions, where the auditor declines to provide a confident answer, and genuine misclassifications, where the generated answer disagrees with the ground truth. Our results show that the majority of the gap arises from abstentions, indicating a controllable trade-off rather than an inherent limitation.
• Open-weight, deployment-ready implementation. The pipeline depends only on open-weight LLMs and openly licensed guideline data, with no proprietary model or private dataset in the trust path. This removes the two principal procurement and compliance barriers to clinical CDS deployment, particularly in low- and middle-income settings, and makes every component of the safety chain inspectable by the deploying institution.

2. Background and Related Work

2.1. Guideline Adherence and the CDS Gap

A substantial amount of research has shown consistent gaps between clinical practice guidelines and routine care [3,4]. Computerized clinical decision support (CDS) systems were proposed to address this, but their effectiveness has been mixed. Rule-based systems can improve specific processes, but they are rigid and difficult to maintain, while alert-based systems often experience high override rates and contribute to alert fatigue mentioned by [18]. As a result, there is a demand for CDS systems that can handle flexible, free-text clinical queries while ensuring that their outputs are grounded in clearly identifiable and authoritative sources.

2.2. LLMs in Clinical Decision Support

Med-PaLM-2 [5], GPT-4 [6] and MedGemma [7] have driven rapid progress on medical question answering benchmarks, including USMLE-style multiple-choice exams. Almanac [19] and ClinicalGPT [20] explore retrieval-grounded clinical assistants. Most of this work focuses on making answers more accurate or improving reasoning, not on whether those answers can be traced back to a guideline; the question “where in the guideline does this come from?” is typically not part of the evaluation. A small but growing strand of work explicitly studies guideline-grounded or retrieval-grounded clinical question answering [19,21] and motivates the present paper.

2.3. Hallucination in Clinical LLMs

Recent benchmarks and analyses [8,9,10,22] document that even retrieval-augmented clinical LLMs produce unsupported claims at non-trivial rates. Med-HALT [9] and MedHallu [22] introduce systematic evaluations of medical hallucination; Goodman et al. [8] demonstrate clinically relevant errors in LLM answers to common patient-facing questions. We extend this work by focusing on citation faithfulness: whether a claim is actually supported by the exact sentence cited. Unlike general factuality, this is a concrete, measurable target.

2.4. Retrieval-Augmented Generation and Its Limits

RAG was introduced as a general way to ground open-domain generation in retrieved evidence [11]. Its weaknesses are by now well documented: long-context “lost in the middle” effects [12], fabricated citations [13], and retrievals that are faithful but off-topic [14]. In medicine, MedRAG [23] set up a benchmark for medical RAG and showed clear gains from retrieval, but also a stubborn rate of unsupported claims. Self-RAG [17] added reflection tokens so the model can critique itself at generation time. The catch is that the critic and the generator are the same model looking at the same context, so the check inherits whatever biases produced the original answer.

2.5. Multi-Agent LLM Systems for Medicine

Recent work on clinical abstention distinguishes uncertainty-driven from safety-driven abstention and frames the decision as a cost-sensitive trade-off [24]. We adopt this distinction and therefore do not equate abstention with safety; instead, we evaluate the independent Auditor as a selective claim–evidence gate and report its false-rejection trade-off.

Multi-agent debate [25], MedAgents [26], AgentClinic [27], and MDAgents [28] have shown that role specialization helps in medical reasoning. These systems mostly tackle collaborative accuracy: several agents talk until they agree on an answer. Our setup differs in two ways. First, success means the answer can be traced back to a written guideline, not that the agents agree. Second, we deliberately restrict what the Auditor sees, so its verdicts do not depend on the Clinician’s reasoning. Most multi-agent systems share a single context window across all agents; here, we close that shortcut.

2.6. Guideline-Grounded NLP

Prior work has explored semantic knowledge representations for clinical guideline recommendations [29] and guideline-based question answering with extractive machine reading comprehension [21]. GuideQA-style efforts target span extraction from a guideline document for a given question. We are not aware of prior CDS pipelines that enforce sentence-level guideline citation directly in the architecture with an independent verifier stage and an evaluation set that includes gold guideline-sentence annotations. We therefore position GuidelineGuard at the intersection of three traditions (clinical CDS, faithful RAG, and multi-agent LLM systems) and introduce both the architecture as well as the evaluation methodology for it.

3. Methods

3.1. Problem Formulation

Let q denote a free-text clinical query (e.g., a vignette and a multiple-choice question) and let $G=\{s_1,\dots ,s_n\}$ denote a corpus of guideline sentences, each with a stable identifier $\mathrm{id}\left(s_i\right)$, a source label, a topic, and the verbatim sentence text. A CDS response is a structured object $R={\left\{(c_i,e_i,a_i)\right\}}_{i=1}^m$ where each tuple comprises a recommendation claim $c_i$, an evidence sentence $e_i\in G$, and an audit verdict $a_i\in$ {Entails, Partially, Not}. The system also produces a final user-facing answer string $A\left(q\right)$ derived deterministically from the set of claims with verdict Entails (or, in fallback mode, Partially). Citation faithfulness, the central quality property of R, is the fraction of $(c_i,e_i)$ pairs for which $e_i$ entails $c_i$ under an independent verifier; we treat this as a structural requirement rather than as a model-internal score.

3.2. Guideline Corpus

We built an open corpus of 73 clinical recommendations, each a single sentence, drawn from publicly available CPGs with licenses that allow redistribution for research. We pulled the sentences in verbatim where possible, or lightly reworded them so they stand on their own, and kept a pointer from each one back to the section of the guideline it came from.

Table 1. Composition of the GuidelineGuard guideline corpus (73 sentence-level recommendations across 23 topics; selected sources shown).

Topic Group	Sources	n
Hypertension and cardiology	ACC/AHA, ESC	11
Diabetes and metabolic	ADA, KDIGO	8
Pulmonary (asthma, COPD)	GINA, GOLD	5
Infectious disease	IDSA/ATS, CDC	9
Sepsis and critical care	Surviving Sepsis Campaign	6
Preventive services and screening	USPSTF, CDC	6
Stroke and neurology	AHA/ASA	4
Anticoagulation and VTE	CHEST, ASH	4
Obstetrics/pregnancy	ACOG, WHO	3
Paediatrics	WHO IMCI, AAP	4
Renal and electrolyte	KDIGO	3
Pain and opioid prescribing	CDC	3
Anaphylaxis and allergy	WAO	2
Psychiatry and sleep	APA	3
COVID-19 and respiratory	WHO, NIH	2

summarizes the topical composition. Each entry has a stable alphanumeric identifier (e.g., HTN-001) used throughout the pipeline for citation.

3.3. Retrieval

We use a hybrid sparse plus dense retrieval pipeline. Sparse retrieval uses Okapi BM25 [30] (implementation rank_bm25) and produces a ranked top-20 list. Dense retrieval uses BAAI/bge-small-en-v1.5 [31] sentence embeddings (384-dimensional, 80 MB on disk) computed via sentence-transformers on CPU; cosine similarity over a brute-force index (the corpus is small) produces a second top-20 list. The two ranked lists are fused with Reciprocal Rank Fusion [32] ($k=60$) and the top-8 fused sentences per sub-query are passed forward. Index construction takes under two seconds on a typical laptop and is performed once at startup.

3.4. Agent Architecture

GuidelineGuard executes four LLM agents per query, each implemented as an independent LLM call with a role-specific system prompt, followed by a temperature-zero LLM Synthesizer. Figure 1 summarizes the data flow.

• Planner.

The Planner receives the original query q and outputs a JSON list of up to four focused sub-queries, each tagged with one intent from {diagnosis, workup, treatment, dosing, contraindication, prevention, screening, monitoring}. A single dense search over a multi-part question often pulls evidence for one part and misses the others. Splitting the query up front fixes this.

• Retriever.

For each sub-query, the Retriever fetches the top 8 fused guideline sentences. The union over all sub-queries (deduplicated, capped at 15 sentences) is passed to the Clinician.

• Clinician.

The Clinician receives the original query and the retrieved evidence set, but not the Auditor’s verdicts. Its system prompt requires it to (i) select one answer choice, (ii) express the rationale as concise recommendation claims, (iii) attach a citation identifier to every claim, and (iv) cite only identifiers present in the retrieved evidence set. It is instructed not to introduce guideline facts that cannot be linked to a retrieved sentence and to return structured JSON containing the answer choice and claim–citation pairs. The Clinician is a generation component rather than a clinical expert or human-in-the-loop reviewer; its name denotes the role assigned in the prompt. It is not presented as a substitute for clinician judgment.

• Auditor.

For each (claim, candidate sentence) pair in the Clinician’s draft, the Auditor performs natural language inference with a prompt focused solely on verification, the Auditor receives only the claim and the candidate sentence text, not the query, not the retrieved set, not the Clinician’s reasoning, and not the rest of the draft. It emits exactly one verdict from {Entails, Partially, Not} together with a one-sentence rationale. The independence of the Auditor’s context is the architectural property that distinguishes GuidelineGuard from reflection-based approaches.

• Synthesizer.

A temperature-zero LLM Synthesizer composes the verified claims into a clinician-facing answer with bracketed source identifiers (e.g., “initiate first-line antihypertensive therapy [HTN-001]”). If no Entails claims are produced, the Synthesizer falls back to Partially supported claims, prepends a flag (“partial support”), and proceeds. If neither is available, the system abstains rather than guessing. The choice of strict-, standard- or coverage-mode is a single parameter, allowing the same system to be tuned for different deployment risk profiles (Section 5.1).

The full pipeline is summarized in

Table 2. The GuidelineGuard pipeline.

Input: query q, guideline corpus G, retrieval depth $k=8$, modes $\in \{\mathrm{strict},\mathrm{standard},\mathrm{coverage}\}$.
Output: answer string A, claim set R.
1.    Q ← Planner(q)	// decompose into intent-tagged sub-queries
2.    $E\leftarrow {\bigcup }_{q^{\prime }\in Q}$ Retrieve(q′, G, k)	// hybrid BM25 + dense + RRF, dedup, cap 15
3.    D ← Clinician(q, E)	// JSON draft: list of (claim, cited_id)
4.    for each $(c_i,e_i)\in D$:
5.       $a_i\leftarrow$Auditor$(c_i,\mathrm{text}\left(e_i\right))$	// verification-only context
6.    $R\leftarrow \left\{(c_i,e_i,a_i)\right\}$
7.    if $\mathrm{mode}=\mathrm{coverage}$: surface all R
8.    else if $\exists i:a_i=$ Entails: surface those;
9.       else if mode=standard and $\exists i:a_i=$ Partially: surface those (flagged)
10.    else: abstain
11.    A ← Synthesize(surfaced R)
12.    return $A,R$

.

3.5. Models and Infrastructure

All four agents use the open-weight Llama-3.1-8B-Instant [33] model served via the Groq Cloud inference endpoint. Embeddings use BAAI/bge-small-en-v1.5 (Sentence-Transformers). Sparse retrieval is implemented with rank_bm25 0.2. Orchestration is a thin custom Python loop with strict JSON I/O and schema validation; no agent framework dependency is required. The retrieval, audit, and synthesis components have no GPU dependency and can be hosted on-premise alongside an institution’s existing infrastructure, which is a deliberate design choice rather than a constraint: it keeps protected health information inside the institutional boundary and confines external calls to the LLM endpoint, which can itself be swapped for an on-premise open-weight deployment without code changes.

3.6. Baselines

We compare GuidelineGuard against five baselines that span the design space:

• Zero-shot LLM (same Llama-3.1-8B-Instant, no retrieval, no agents).
• Chain-of-thought (zero-shot with an explicit “think step by step” instruction).
• Single-pass RAG (top-10 fused retrieved sentences concatenated; one LLM call asked to answer and cite source IDs).
• Self-RAG (self-critique) [17]-style baseline. The same LLM drafts a cited answer and then critiques each (claim, cited-sentence) pair from its own draft, filtering out pairs it judges not entailed. Crucially, the critic is the same model that wrote the draft, so there is no separation between generator and verifier; this is precisely the bias GuidelineGuard’s Auditor is designed to avoid.
• Multi-agent without Auditor (Planner + Retriever + Clinician + Synthesizer; no Auditor). This ablation isolates the verifier’s contribution.
• GuidelineGuard (full pipeline, standard mode).

For systems other than GuidelineGuard and Self-RAG, citation-faithfulness judging is performed post hoc by the secondary LLM (Llama-3.3-70B-Versatile) using a verification-only prompt. This is not a strictly like-for-like comparison with GuidelineGuard, whose CF is produced by the operational Auditor during inference, nor with Self-RAG, where self-judgment is part of the baseline. Consequently, cross-system CF differences may reflect evaluator strictness and evaluation granularity as well as system behavior. We therefore treat those CF contrasts as descriptive and do not use them as evidence that the Auditor is superior; a fully common independent verifier across all systems would be required for a strict head-to-head CF comparison.

3.7. Benchmark: GG-Bench-60

We constructed GG-Bench-60, an evaluation set of 60 four-option multiple-choice clinical cases. Each case was authored and gold-annotated by the sole author, who has clinical-ML research experience, using the topic taxonomy of the corpus. The annotation records (a) the correct answer letter and (b) the identifier(s) of the guideline sentence(s) judged to justify it. No non-author clinician participated in case construction or annotation. We therefore treat GG-Bench-60 as a transparent, author-constructed benchmark rather than as an independently curated clinical examination set, and we use the external PubMedQA evaluation as a complementary check against benchmark-construction bias. The 60 cases span all 23 represented topics and cover both positive recommendations (e.g., first-line therapy for newly diagnosed hypertension) and contraindication knowledge (e.g., ACE inhibitors in pregnancy, beta-blockers in active asthma). A total of 12 cases (20%) test contraindication recognition, which we expected to be especially hard for systems that need to stay faithful to their citations.

3.8. Major-Revision Expanded Evaluation

To address the reviewers’ concerns about the size of the in-domain and external evaluations, we added two prespecified expanded experiments without altering the original reported runs. First, GG-Bench-200 preserves the original 60 cases verbatim and adds 140 deterministic, source-traceable template cases derived from the same 73-sentence corpus. The expanded set contains 200 unique four-option items, is balanced across answer letters (50 A, 50 B, 50 C, 50 D), and exercises every one of the 73 evidence identifiers at least twice. These added cases are synthetic and were not independently clinician-authored or clinician-validated; they are therefore interpreted as a controlled coverage/robustness stress test rather than as a measure of clinical realism.

Second, we replaced the earlier 100-case PubMedQA convenience subset with the 500-case held-out PQA-L test split. We reproduce the PubMedQA authors’ label-stratified two-fold split logic with random seed 0 and use the 500-item test fold. Each question’s source abstract is segmented into sentence-level evidence units with stable identifiers. PubMedQA supplies expert yes/no/maybe answer labels but not sentence-level gold rationale identifiers, so reference-based citation metrics such as Gold-Evidence Recall are not defined for this benchmark.

The expanded experiments focus on the three systems needed for the reviewer-critical ablation: single-pass RAG, multi-agent without Auditor, and GuidelineGuard. The revision runner is designed to build the Planner–Retriever–Clinician state once whenever either agentic arm is missing, then synthesize the no-Auditor output directly and audit the same state for GuidelineGuard. Post-run checkpoint verification found identical observable pre-audit state (Planner sub-queries, retrieved identifiers, and claim count) for all 200 GG-Bench-200 cases and 496/500 PubMedQA cases. Four PubMedQA cases were regenerated after quota-interrupted resumption because one arm had already checkpointed before the interruption; all four were correct commitments in both arms. Gate-transition counts below are therefore computed on the 496 verified-paired PubMedQA cases, while benchmark-level performance uses all 500 completed outputs. All calls use temperature 0. Because the hosted llama-3.1-8b-instant and llama-3.3-70b-versatile endpoints used in the original experiments were no longer available when the major-revision runs were executed, the expanded experiments use openai/gpt-oss-20b for Planner, Clinician, and Synthesizer and openai/gpt-oss-120b for the independent Auditor. We therefore report the expanded results as a post-deprecation successor-model replication rather than pooling them with the original Llama results.

For the expanded analysis, a commitment requires a parseable answer choice and no explicit ABSTAIN token. Operational accuracy counts abstentions and unparseable outputs as incorrect. Selective accuracy is correctness among commitments. On GG-Bench-200, Gold-Supported Commitment Rate (GSCR) is the fraction of commitments citing at least one benchmark gold-evidence identifier, and Retrievable-Evidence Rejection (RER) is the fraction of cases with retrieved gold evidence on which the system does not commit. These are reference-alignment proxies, not human entailment judgments. For the Auditor, we additionally report categorical operating points based on maximum support (Entails only; Entails-or-Partially; ungated paired draft) and the observed-range trapezoidal AURC. Because only three discrete thresholds are available, this AURC is a descriptive operating-range summary rather than a calibrated continuous selective-risk estimate.

3.9. Metrics

Table 3. Evaluation metrics used in this study. We distinguish answer performance, evidence coverage, and verification-gate behavior so that abstention is not conflated with citation faithfulness.

Metric	Definition
Accuracy	Top-1 multiple-choice accuracy on GG-Bench-60. Abstentions count as incorrect.
Selective Accuracy (Sel.Acc)	Top-1 accuracy only on committed cases (parseable answer and no explicit abstention).
Coverage	Fraction of cases on which the system commits to an answer. Abstention rate is $1-\mathrm{Coverage}$.
Citation Faithfulness (CF)	Macro-average across cases of the fraction of audited claims whose cited evidence sentence is judged Entails. A case with no audited claim contributes 0; CF is therefore a joint coverage/faithfulness diagnostic, not a pure conditional precision measure.
Conditional Claim Faithfulness (CCF)	Fraction of audited claims judged Entails, restricted to committed cases. This separates evidence quality among surfaced answers from the cost of abstention.
Citation Validity (CV)	Fraction of emitted citation identifiers that exist in the applicable evidence corpus; a fabrication check. Cases that emit no citation do not enter the CV denominator and are represented separately by Citation Coverage.
Citation Coverage	Fraction of cases emitting at least one citation identifier.
Gold-Evidence Recall (GER)	Fraction of cases citing at least one human-annotated gold evidence sentence.
Gold-Supported Commitment Rate (GSCR)	Among committed cases, fraction citing at least one gold evidence sentence. We report its complement as an operational false-acceptance rate (FA = 1 − $\mathrm{GSCR}$).
Retrievable-Evidence Rejection Rate (RER)	Among cases where at least one gold evidence sentence was retrieved, fraction for which GuidelineGuard abstained. This is an operational false-rejection proxy, not a claim that the Auditor’s decision is clinically incorrect.
Risk–coverage/AURC	Selective risk $1-\mathrm{Sel}.\mathrm{Acc}$ as a function of coverage. For GuidelineGuard, the ranking score is the maximum Auditor support score per case (Entails = 1, Partially = 0.5, Not = 0); AURC is the discrete area under this risk–coverage curve.
Mean latency	Mean per-query wall-clock latency in seconds.

represents the details of the metrics and their respective definitions used in this study.

3.10. Statistical Analysis

We report 95% confidence intervals using the non-parametric percentile bootstrap with 2000 resamples for all proportion-valued metrics. All systems are evaluated on the same 60 cases within each backbone, enabling paired comparison. Because the Auditor changes the decision rule, we do not interpret selective accuracy without its corresponding coverage. We therefore report selective accuracy, coverage, gold-supported commitment rate, retrievable-evidence rejection rate, and a risk–coverage analysis together. For the focal comparison against the multi-agent ablation, the aggregate claim-level tally underlying CF is 45/70 verified claims for GuidelineGuard versus 54/68 for the ablation; this difference is interpreted as evidence that the Auditor is a stricter gate, not as evidence that it increases raw citation faithfulness. With $n=60$ cases, CIs are correspondingly wide; we discuss this in Limitations. For binary per-case outcomes (accuracy and Gold-Evidence Recall), we run two-sided exact McNemar tests separately within each backbone. We do not pool discordant pairs across backbones because the same 60 benchmark items are repeated under different backbones and therefore do not constitute independent replicated observations. Cross-backbone consistency is summarized qualitatively from the backbone-specific effect directions and exact p-values. We make no accuracy-superiority claim that the paired tests do not support.

3.11. Reproducibility, Ethics, and Safety

All clinical vignettes in the benchmark were written by the author and are fully synthetic. No patient-identifiable data were used, and no clinical record was looked at while building the corpus or the benchmark. The guideline corpus only includes guidelines whose licenses allow redistribution for research. GuidelineGuard is a research prototype and is not meant for clinical use. Any real deployment would need local validation against the institution’s own guideline set, clinician-in-the-loop testing, and regulatory review appropriate to the jurisdiction and indication. For reproducibility of the hosted-model results, all backbones were queried at temperature 0 in deterministic non-reasoning mode (no hidden “thinking” tokens), and we record the exact model identifiers used (llama-3.1-8b-instant, llama-3.3-70b-versatile, and qwen/qwen3.6-27b, all served via Groq) together with the run date. Because hosted endpoints can be updated or deprecated, the released reproducibility package additionally ships the complete raw per-case outputs, so every reported number can be regenerated offline even if a provider later changes or withdraws a model.

4. Results

We report aggregated results across the six systems in

Table 4. Main results on GG-Bench-60 ($n=60$, Llama-3.1-8B-Instant backbone). 95% CIs are non-parametric bootstrap intervals (2000 resamples). CF is a joint case-level faithfulness/coverage diagnostic; Self-RAG CF is self-judged, whereas RAG and the multi-agent ablation use an independent LLM judge. UCR is $1-\mathrm{CF}$.

System	Acc. [95% CI]	CF [95% CI]	GER [95% CI]	UCR	Cov.	Lat. (s)
Zero-shot	0.933 [0.867, 0.983]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	1.000	1.000	1.51
Chain-of-thought	0.950 [0.883, 1.000]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	1.000	1.000	2.98
Single-pass RAG	0.983 [0.950, 1.000]	0.781 [0.675, 0.872]	0.883 [0.800, 0.950]	0.219	1.000	5.96
Self-RAG (self-critique)	0.983 [0.950, 1.000]	0.850 [0.750, 0.933] †	0.900 [0.817, 0.967]	0.150 †	1.000	8.14
Multi-agent (no Auditor)	0.867 [0.767, 0.950]	0.842 [0.742, 0.925]	0.983 [0.950, 1.000]	0.158	0.900	13.57
GuidelineGuard (ours)	0.850 [0.750, 0.933]	0.731 [0.617, 0.833]	0.967 [0.917, 1.000]	0.269	0.867	12.69

^† Self-RAG CF/UCR is computed by the same model that produced the answer; the independent NLI cross-check (Section 4.8) is applicable only to systems with an independently scored Auditor and is not reported for Self-RAG.

. All metrics are computed on the same 60 cases, with 95% bootstrap confidence intervals (2000 resamples) reported for proportion-valued quantities. Figure 2 visualizes the principal metric trade-offs across systems.

4.1. Per-Metric Findings

• Citation faithfulness.

GuidelineGuard achieved a CF of 0.731, versus 0.781 for single-pass RAG (Δ = −5.0 pp) and 0.842 for the multi-agent ablation without Auditor (Δ = −11.1 pp). At first glance the ablation appears to dominate the full pipeline on CF; however, the ablation achieves this by surfacing every claim the Clinician proposes, including borderline paraphrases that an independent verifier would not endorse. The Auditor applies a stricter standard and downgrades a small fraction of these to Partially or Not, which lowers CF on the population of surfaced claims but converts what would have been silent unsupported claims into explicit abstentions (visible in the lower Coverage of 0.867 and the 13.3 pp accuracy gap that is fully recovered under Selective Accuracy; see

Table 5. Selective Accuracy on GG-Bench-60: accuracy computed only on cases for which the system committed to an answer (i.e., produced a parseable choice and did not explicitly abstain). 95% CIs are non-parametric bootstrap intervals (2000 resamples) over the committed subset. This metric disentangles the precision/coverage trade-off of GuidelineGuard from its underlying reasoning quality.

System	n	Committed	Correct	Sel. Acc. [95% CI]
Zero-shot	60	60	56	0.933 [0.867, 0.983]
Chain-of-thought	60	60	57	0.950 [0.883, 1.000]
Single-pass RAG	60	60	59	0.983 [0.950, 1.000]
Self-RAG (self-critique)	60	60	59	0.983 [0.950, 1.000]
Multi-agent (no Auditor)	60	54	52	0.963 [0.907, 1.000]
GuidelineGuard (ours)	60	52	51	0.981 [0.942, 1.000]

and Section 5.1).

• Gold-evidence recall.

GuidelineGuard cited a human-annotated gold guideline sentence in 96.7% of cases, versus 88.3% for single-pass RAG (a difference of +8.3 pp). The multi-agent ablation reached 98.3%. The Planner–Retriever–Clinician trio is already very good at surfacing the right evidence on its own. What the Auditor adds is precision on top of a retrieval stack that is already strong on recall.

• Citation validity and citation coverage.

We separate two properties that a coarser analysis can conflate. Citation Validity (CV) asks whether every emitted identifier actually exists in the corpus. On GG-Bench-60, no system emitted a single non-existent identifier, so CV = 1.000 for all systems, including single-pass RAG; we therefore do not claim that any baseline fabricates identifiers. Citation Coverage asks how often a system attaches at least one citation to its output, and here the systems differ: single-pass RAG cited in 88.3% of cases, Self-RAG in 91.7%, GuidelineGuard in 96.7%, and the multi-agent ablation in 100.0%. The lower coverage of the retrieval baselines is driven by cases in which they committed to an answer without attaching any evidence, not by invalid identifiers. GuidelineGuard’s design keeps identifiers as structured fields tied to the retrieved set, which guarantees validity by construction and yields high coverage without free-text identifier generation.

• Accuracy and structured error decomposition.

GuidelineGuard’s accuracy (0.850) was lower than single-pass RAG (0.983); the gap of 13.3 pp reflects an explicit precision/coverage trade-off rather than a reasoning deficiency. Of the 9 cases on which GuidelineGuard was scored incorrect, 8 were abstentions (the Auditor refused to surface a confident answer) and only 1 was a genuine misclassification (the Synthesizer surfaced an answer that disagrees with the gold). All 9 cases on which GuidelineGuard was scored incorrect were cases on which single-pass RAG was correct. These abstentions are not assumed to be inherently safer than commitment. Their desirability depends on the relative downstream cost of unsupported commitment versus unnecessary rejection; the expanded selective analysis below makes that trade-off explicit.

• Selective accuracy.

To disentangle the precision/coverage trade-off from underlying reasoning quality, we additionally report Selective Accuracy (Table 5): the top-1 accuracy of each system computed only on cases for which it committed to an answer. On this metric, GuidelineGuard reached 0.981 (51/52), statistically indistinguishable from single-pass RAG (0.983, 59/60); the 95% confidence intervals overlap fully. The 13.3 pp gap on raw accuracy is attributable to GuidelineGuard’s eight additional abstentions relative to single-pass RAG. The committed-case point estimate is similar to RAG, but the benchmark is too small to establish equivalence or to rule out differences in the underlying generator.

• Coverage and selective risk.

GuidelineGuard committed on 52/60 cases (86.7% coverage) and achieved 51/52 selective accuracy (98.1%), whereas single-pass RAG committed on all 60 cases and achieved 59/60 accuracy (98.3%). The two systems therefore occupy different operating points rather than having directly comparable accuracy alone. GuidelineGuard’s selective risk was 1.9%, versus 15.0% when its eight abstentions are counted as incorrect at full coverage. The risk–coverage analysis (Figure 3) yields an AURC of 0.114 using the Auditor support score, showing that higher-support cases are preferentially surfaced.

• Latency and cost.

Mean per-query wall-clock latency was 12.7 s for GuidelineGuard, versus 6.0 s for single-pass RAG, 3.0 s for chain-of-thought, and 1.5 s for zero-shot. The latency overhead of the full pipeline is acceptable for offline or batched CDS use cases (e.g., pre-visit summarization, audit) and could be reduced for synchronous use by parallelizing the Auditor calls across claims.

4.2. Selective-Gate Diagnostics

To make the Auditor’s contribution explicit, we evaluated it as a selective gate rather than assuming that abstention is intrinsically safer. On the 8B GG-Bench-60 run, GuidelineGuard committed on 52 cases and all 52 committed cases cited at least one gold evidence sentence, giving a Gold-Supported Commitment Rate of 1.000 and an operational false-acceptance rate of 0.000. By comparison, single-pass RAG cited gold evidence in 53/60 committed cases (0.883; FA = 0.117), while the multi-agent ablation did so in 53/54 committed cases (0.981; FA = 0.019). These rates are based on the benchmark’s sentence-level gold annotations and should not be interpreted as exhaustive evidence correctness.

The corresponding operational false-rejection proxy is the fraction of cases in which gold evidence was retrieved but GuidelineGuard abstained. Gold evidence was retrieved for 59/60 cases; six of those 59 were abstentions, giving a Retrievable-Evidence Rejection Rate of 0.102. This result directly addresses the concern that abstention can reject a supportable answer: in this benchmark, most abstentions were not caused by retrieval failure but by the Auditor’s stricter claim–evidence matching criterion. We therefore treat this rejection rate as a limitation and calibration target rather than as evidence of clinical safety.

Finally, we computed a risk–coverage curve using the maximum Auditor support score for each case, mapping Entails, Partially, and Not to 1, 0.5, and 0, respectively. Sorting cases by this score produced an AURC of 0.114. At the manuscript’s standard operating point, coverage was 0.867 and selective risk was 0.019. The curve makes explicit that the central contribution is selective verification: the system can trade coverage for lower observed risk by changing the acceptance threshold, but this trade-off requires task-specific calibration and does not by itself establish clinical benefit.

Table 6. Selective-gate diagnostics on GG-Bench-60 with the Llama-3.1-8B-Instant backbone. FA is the complement of Gold-Supported Commitment Rate. RER is defined only for GuidelineGuard because the baselines do not expose an abstention gate tied to the same evidence-verification mechanism. AURC is computed from the GuidelineGuard Auditor support score.

System	Committed	GSCR	FA	GER	RER
Single-pass RAG	60	0.883	0.117	0.883	–
Multi-agent (no Auditor)	54	0.981	0.019	0.983	–
GuidelineGuard	52	1.000	0.000	0.967	0.102

GuidelineGuard AURC = 0.114; coverage = 0.867; selective risk = 0.019 at the reported operating point.

summarizes the corresponding selective-gate diagnostics.

4.3. Per-Topic Robustness

Performance was heterogeneous across the 23 topics in the benchmark (

Table 7. Per-topic accuracy (cases correct/cases) across the 23 represented topics in GG-Bench-60. ZS: zero-shot; CoT: chain-of-thought; RAG: single-pass retrieval-augmented generation; MA: multi-agent without Auditor; GG: GuidelineGuard. Self-RAG is omitted here for space; its aggregate metrics appear in Table 4 and Table 5.

Topic	n	ZS	CoT	RAG	MA	GG
acs	4	4/4	4/4	4/4	4/4	3/4
afib	1	1/1	1/1	1/1	1/1	1/1
anaphylaxis	2	2/2	2/2	2/2	2/2	2/2
antibiotics	3	3/3	3/3	3/3	3/3	3/3
anticoagulation	2	2/2	2/2	2/2	2/2	2/2
asthma	2	2/2	2/2	2/2	0/2	0/2
copd	2	2/2	2/2	2/2	2/2	2/2
covid	2	2/2	2/2	2/2	2/2	2/2
diabetes	5	5/5	5/5	5/5	4/5	4/5
hypertension	4	3/4	4/4	4/4	4/4	3/4
immunization	3	3/3	3/3	3/3	3/3	3/3
pain	2	2/2	2/2	2/2	2/2	2/2
pediatrics	3	3/3	3/3	2/3	2/3	2/3
pneumonia	2	2/2	2/2	2/2	2/2	2/2
pregnancy	2	2/2	2/2	2/2	2/2	2/2
psychiatry	2	2/2	2/2	2/2	2/2	2/2
renal	2	2/2	2/2	2/2	1/2	1/2
screening	3	3/3	2/3	3/3	3/3	3/3
sepsis	5	4/5	5/5	5/5	5/5	5/5
sti	3	3/3	2/3	3/3	2/3	2/3
stroke	2	2/2	2/2	2/2	2/2	2/2
uti	2	1/2	1/2	2/2	1/2	2/2
vte	2	1/2	2/2	2/2	1/2	1/2

). In particular, the sampled asthma cases yielded 0/2 correct for GuidelineGuard, so the earlier statement that no topic showed complete failure was incorrect and has been removed. Because several topics contain only one or two cases, the per-topic estimates are descriptive rather than evidence of topic-level robustness. The largest gaps versus single-pass RAG correspond to topics where the Auditor was strict on borderline paraphrases (e.g., asthma step-up therapy). The per topic table is intended to flag topics that would benefit from corpus expansion or auditor calibration, and to support the journal’s expectation that aggregate metrics be supplemented with disaggregated evidence.

4.4. Robustness Across Model Scale

To confirm that the qualitative ordering is not an artifact of the 8B backbone, we replicated all systems on Llama-3.3-70B-Versatile (served via the same Groq endpoint).

Table 8. Cross-backbone replication on GG-Bench-60. All six systems were evaluated on the 8B backbone; four completed on the 70B backbone (Llama-3.3-70B-Versatile, served via Groq) within the same rate-limit budget. Self-RAG is a single-model self-critique baseline that we evaluate on the 8B backbone only, and the 70B multi-agent run did not complete because the provider rate-limited the longer agentic trajectories; both are therefore absent from the 70B block. The qualitative ordering is preserved: GuidelineGuard ties single-pass RAG on Selective Accuracy and dominates on Gold-Evidence Recall, while the abstention rate falls to 1.7% as the underlying model becomes more able to surface a defensible answer.

Backbone	System	Acc	Sel. Acc. (n com.)	Abst.%	CF	GER
8B	Zero-shot	0.933	0.933 (60)	0.0	–	–
8B	Chain-of-thought	0.950	0.950 (60)	0.0	–	–
8B	Single-pass RAG	0.983	0.983 (60)	0.0	0.781	0.883
8B	Self-RAG (self-critique)	0.983	0.983 (60)	0.0	0.850	0.900
8B	Multi-agent (no Auditor)	0.867	0.963 (54)	10.0	0.842	0.983
8B	GuidelineGuard	0.850	0.981 (52)	13.3	0.731	0.967
70B	Zero-shot	0.967	0.967 (60)	0.0	–	–
70B	Chain-of-thought	0.933	0.933 (60)	0.0	–	–
70B	Single-pass RAG	1.000	1.000 (60)	0.0	–	1.000
70B	GuidelineGuard	0.983	1.000 (59)	1.7	0.817	0.983

CF entries marked “–” indicate that LLM-judge CF was not separately recomputed for the 70B backbone; the 8B CF column (Table 4) is the principal comparison and is not expected to change materially with backbone size.

summarizes the cross-scale comparison. On 70B, GuidelineGuard reached an accuracy of 0.983 (59/60) and a Selective Accuracy of 1.000 (59/59), tying single-pass RAG on both metrics; the single non-correct case was an explicit abstention rather than a misclassification, dropping the abstention rate to 1.7%. Gold-Evidence Recall remained at 0.983 and Citation Faithfulness improved from 0.731 (8B) to 0.817 (70B). The principal conclusion of the 8B experiments is therefore preserved at scale: GuidelineGuard either matches or marginally trails the strongest baseline on raw accuracy, recovers fully under Selective Accuracy, and dominates on Gold-Evidence Recall.

4.5. Robustness Across Model Families

The scale replication in Section 4.4 varies model size within a single family (Meta Llama). To test whether the findings also transfer across model families (the stronger generalization claim), we replicated the pipeline on Qwen3.6-27B (Alibaba), an architecturally and organizationally independent backbone, served through the same Groq endpoint in deterministic non-reasoning mode (temperature 0, no hidden “thinking” tokens) so that the run is byte-for-byte reproducible. All five systems evaluated on this backbone completed 60/60 with zero error stubs.

Table 9. Cross-family replication on GG-Bench-60 with Qwen3.6-27B (Alibaba), an independent model family, run in deterministic non-reasoning mode. All five systems completed 60/60 with zero error stubs and zero fabricated identifiers. The qualitative ordering established on the Llama backbones is preserved: GuidelineGuard ties the strongest baselines on Selective Accuracy, abstains rather than misclassifying, and guarantees citation validity.

System	Acc.	Sel. Acc. (n com.)	Abst.%	CF	GER
Zero-shot	1.000	1.000 (60)	0.0	–	–
Chain-of-thought	0.933	0.933 (60)	0.0	–	–
Single-pass RAG	1.000	1.000 (60)	0.0	0.857	1.000
Multi-agent (no Auditor)	0.967	1.000 (58)	3.3	0.892	0.967
GuidelineGuard	0.967	1.000 (58)	3.3	0.850	0.950

reports the results.

The cross-family results broadly reproduce the within-family pattern. GuidelineGuard reaches 0.967 accuracy and 1.000 Selective Accuracy (58/58 committed), tying single-pass RAG and the multi-agent ablation on Selective Accuracy; its two non-correct cases are both explicit abstentions rather than misclassifications, so the abstention rate is 3.3% and there is not a single guideline misclassification. Gold-Evidence Recall is 0.950 and Citation Faithfulness is 0.850, again marginally below the multi-agent ablation (0.892) on raw CF while retaining the two structural guarantees the ablation lacks: a guaranteed-valid citation for every committed answer (Citation Validity = 1.000) and the ability to abstain. The property that most directly addresses hallucination holds across all three backbones and both families: zero fabricated identifiers were produced by any system on any backbone. The qualitative ordering (GuidelineGuard matches or marginally trails the strongest baseline on raw accuracy, recovers fully under Selective Accuracy, guarantees citation validity, and abstains rather than guessing) is therefore not an artifact of the Llama family.

4.6. External-Benchmark Validation on PubMedQA

GG-Bench-60 is a benchmark we constructed, which the reviewer could reasonably view as favorable to our own system. To test the central claims on data we did not build, we additionally evaluated all five systems on PubMedQA [34], a public expert-annotated biomedical QA benchmark in which each item pairs a research question with its source abstract (the grounding context) and an expert yes/no/maybe label. We drew a seeded, label-stratified subset of $n=100$ items from the expert-labeled PQA-L split (proportional to the full distribution: 55 yes, 34 no, 11 maybe), reformulated each as a three-way multiple-choice question (A/B/C), and exposed each item’s abstract as a per-question mini-corpus of sentences that the retrieval and citation-audit machinery operates over unchanged. We used the Llama-3.1-8B backbone (identical to our primary configuration), so the external numbers are directly comparable to Table 4. All five systems completed 100/100 cases with zero fabricated sentence identifiers.

Table 10. Corrected external-benchmark validation on a seeded, label-stratified $n=100$ subset of PubMedQA (PQA-L), Llama-3.1-8B backbone. CV is identifier-level Citation Validity over emitted citations; Cit.Cov is case-level Citation Coverage. All emitted identifiers from the three grounded systems exist in the corresponding per-question abstract corpus, so CV is 1.000 for each. CF is retained as a descriptive metric but is not strictly comparable across systems because the verifier/evaluation unit differs (see text). Brackets after Sel.Acc give the number of committed cases.

System	Acc.	Sel. Acc. (n com.)	Abst.%	CV	Cit.Cov	CF
Zero-shot	0.570	0.570 (100)	0.0	–	–	–
Chain-of-thought	0.610	0.610 (100)	0.0	–	–	–
Single-pass RAG	0.710	0.710 (100)	0.0	1.000	0.990	0.714
Multi-agent (no Auditor)	0.710	0.710 (100)	0.0	1.000	1.000	0.415
GuidelineGuard	0.610	0.726 (84)	16.0	1.000	0.840	0.425

reports the results with 95% bootstrap confidence intervals (2000 resamples).

The external benchmark reproduces the paper’s central pattern rather than a raw-accuracy win. On overall accuracy, the two retrieval baselines lead (single-pass RAG and the multi-agent ablation both $0.71$, 95% CI $[0.62,0.80]$), while GuidelineGuard scores $0.61$ ($[0.51,0.71]$), because GuidelineGuard is the only system that abstains, declining to commit on $16\%$ of items where it cannot verify an answer against the abstract. When scored only on the cases it does commit to, GuidelineGuard attains the highest Selective Accuracy of any system ($0.726$, on 84 committed cases), matching or slightly exceeding both retrieval baselines ($0.71$; confidence intervals overlap, so we claim parity, not superiority). In other words, the calibrated-abstention behavior that GuidelineGuard was designed to provide transfers intact to an external benchmark: it withholds an answer on its least-supported cases and is at least as accurate as strong RAG on the rest, whereas every baseline is forced to guess. Re-audit of the raw citation identifiers confirmed that Citation Validity is 1.000 for all three citation-bearing systems: all 200 identifiers emitted by single-pass RAG, all 180 emitted by the multi-agent ablation, and all 133 emitted by GuidelineGuard exist in the corresponding per-question abstract corpus. The previously reported values of 0.990 and 0.840 arose because cases with no emitted citation were incorrectly entered as zero-validity cases, thereby conflating Citation Validity with Citation Coverage. Citation Coverage is reported separately in the corrected Table 10 (0.990, 1.000, and 0.840, respectively).

Two honest caveats bound this result. First, PubMedQA provides no sentence-level gold rationales, so Gold-Evidence Recall is undefined and is not reported for this benchmark. Second, the Citation Faithfulness columns are not a strict head-to-head comparison. GuidelineGuard’s CF is generated by its operational Auditor over atomic claim–citation pairs, whereas the citation-bearing baselines are scored post hoc by the secondary LLM judge over their emitted answer–citation pairs. Even though both use verification-only evidence, differences in evaluator and evaluation granularity can contribute to the observed CF values. We therefore report CF descriptively, do not attribute cross-system CF differences solely to system quality, and regard common independent rescoring of all systems as the appropriate design for a future verifier-focused comparison. The takeaway of the external validation is thus the same as in-domain and unembellished: GuidelineGuard’s contribution is guaranteed-valid citation and calibrated abstention, not raw-accuracy dominance, and that contribution holds on a public benchmark we did not construct.

4.7. Statistical Significance of the Paired Comparisons

Because every system within a backbone is evaluated on the same 60 GG-Bench cases, GuidelineGuard-versus-baseline contrasts are paired binary comparisons. Following the Round-2 reviewer recommendation, we report two-sided exact McNemar tests separately for each backbone and no longer pool discordant counts across backbones. Repeating the same benchmark items under different backbones creates repeated measurements, so treating the backbone-specific outcomes as independent pooled observations would overstate the effective sample size.

Table 11. Backbone-specific two-sided exact McNemar tests on GG-Bench-60. Each cell is $b:c$ (exact p), with b denoting GuidelineGuard-only successes and c baseline-only successes. “–” denotes a comparison not available for that backbone. The incomplete rate-limited 70B no-Auditor run is not used. No across-backbone pooled significance test is reported.

Metric	Baseline	Llama-3.1-8B	Llama-3.3-70B	Qwen3.6-27B
Accuracy	Zero-shot	4:9 (0.267)	2:1 (1.000)	0:2 (0.500)
Accuracy	Chain-of-thought	3:9 (0.146)	4:1 (0.375)	4:2 (0.688)
Accuracy	Single-pass RAG	1:9 (0.021)	0:1 (1.000)	0:2 (0.500)
Accuracy	Self-RAG	1:9 (0.021)	–	–
Accuracy	Multi-agent, no Auditor	1:2 (1.000)	–	0:0 (1.000)
GER	Zero-shot	58:0 (<$0.001$)	59:0 (<$0.001$)	57:0 (<$0.001$)
GER	Chain-of-thought	58:0 (<$0.001$)	59:0 (<$0.001$)	57:0 (<$0.001$)
GER	Single-pass RAG	7:2 (0.180)	0:1 (1.000)	0:3 (0.250)
GER	Self-RAG	6:2 (0.289)	–	–
GER	Multi-agent, no Auditor	0:1 (1.000)	–	0:1 (1.000)

reports $(b:c)$, where b is the number of cases on which GuidelineGuard has the target property and the baseline does not, and c is the converse.

The backbone-specific results support a more limited interpretation than the former pooled analysis. For raw accuracy, single-pass RAG and Self-RAG outperform GuidelineGuard on the 8B backbone ($p=0.021$), but the single-pass RAG contrast is not significant on 70B ($p=1.000$) or Qwen ($p=0.500$). All other available accuracy contrasts are non-significant within their respective backbones. Thus, there is no consistent cross-backbone accuracy-superiority claim for GuidelineGuard; the 8B retrieval-baseline difference is driven primarily by GuidelineGuard abstentions. In contrast, for Gold-Evidence Recall, GuidelineGuard exceeds zero-shot and chain-of-thought prompting on every backbone ($p<0.001$ for each), while none of the available comparisons against citation-bearing retrieval baselines is significant. We therefore describe the evidence-grounding advantage over ungrounded prompting as consistent across backbones, while describing retrieval-baseline differences qualitatively rather than through an invalid pooled test.

4.8. Auditor Cross-Check Against a Deterministic NLI Verifier

A natural concern with reporting Citation Faithfulness using an LLM-based Auditor is that the LLM, even when given verification-only context, could systematically over-credit its own claims. To probe this, every (claim, evidence) pair that the Auditor scored on GG-Bench-60 was independently re-judged with a deterministic, non-LLM verifier: a publicly available DeBERTa-v3-base cross-encoder fine-tuned on NLI (cross-encoder/nli-deberta-v3-base), with the three NLI labels mapped to Entails/Partially/Not as the entailment/neutral/contradiction classes. This second verifier never sees the Auditor’s prompt, the Clinician’s draft, or the question, and is deterministic up to floating-point arithmetic, so it cannot share the LLM’s systematic biases.

On the 68 audited pairs for the 8B backbone, raw three-way agreement was 0.706 (Cohen’s $\kappa =0.418$, moderate); on the ENTAILS-vs-not binary, the agreement was 0.779 ($\kappa =0.522$, moderate). On the 61 pairs for the 70B backbone, three-way agreement was 0.754 ($\kappa =0.282$) and binary agreement 0.770 ($\kappa =0.317$). The asymmetry that matters for our claim is the NLI confirmation rate of LLM Entails verdicts: when the LLM Auditor said the evidence entailed the claim, the independent NLI verifier confirmed that judgment in 80.0% (36/45) of cases on 8B and 83.7% (41/49) of cases on 70B, and 83.6% (46/55) of cases on the cross-family Qwen3.6-27B backbone (Table 9). Where the two verifiers disagreed, the NLI verifier most often returned Partially rather than Not (9/9 disagreements on 8B; 7/8 disagreements on 70B; 9/9 disagreements on Qwen3.6-27B), i.e., the disagreement is a strictness gradient, not a contradiction. The reported Citation Faithfulness values are therefore not artifacts of self-evaluation by a single model; they are supported by an independent classifier of a different family.

4.9. Ablation: Contribution of the Auditor

The multi-agent ablation (Planner + Retriever + Clinician + Synthesizer, no Auditor) reached CF of 0.842 against 0.731 for the full pipeline (Δ = −11.1 pp). Accuracy was 0.867 versus 0.850. The ablation cites in every case (citation coverage 1.000) and reaches very high gold-evidence recall (0.983), so the Planner–Retriever–Clinician trio is doing most of the grounding work on its own. The Auditor sits on top of that and enforces a stricter, independently checked standard. The price is a measurable abstention rate. We therefore interpret the Auditor as a selective gate rather than as a component that improves the generator’s intrinsic faithfulness.

4.10. Qualitative Example

Figure 4 illustrates GuidelineGuard’s output for a representative hypertension case from GG-Bench-60. The Planner decomposes the vignette into “treatment” and “threshold” sub-queries; the Retriever surfaces the AHA/ACC threshold sentence (HTN-001); the Clinician proposes an antihypertensive-initiation claim with HTN-001 as evidence; the Auditor, seeing only the claim and the sentence text, returns Entails; the Synthesizer surfaces the cited recommendation.

4.11. Error Analysis

GuidelineGuard got 9 of the GG-Bench-60 cases wrong. Eight of those nine were abstentions. In each one, the Auditor judged every candidate (claim, sentence) pair as Not or only Partially entailing, so the Synthesizer returned nothing at all. Looking at these cases, a pattern stands out: the corpus actually contains the right knowledge, but the guideline sentence and the answer choice are worded differently enough that the Auditor cannot link them. The clearest example is asthma step-up therapy, where the guideline talks about “escalation” and the answer choice says “increase ICS dose.” Only one of the nine was a genuine misclassification, a diabetes vignette in which the Synthesizer committed to an answer that disagreed with the gold label rather than abstaining. Two fixes follow from this. The first is to expand the corpus so that the same recommendation appears in more than one wording, which directly targets the eight wording-mismatch abstentions. The second is to let the Clinician and Auditor exchange one round of feedback, giving the Clinician a single chance to redraft when its first attempt is rejected. We leave both for future work.

4.12. Major-Revision Expanded Results

Table 12. Expanded controlled evaluation on GG-Bench-200 using the revision successor models. Operational accuracy counts abstentions and unparseable outputs as incorrect. GSCR is a benchmark-reference alignment proxy among committed cases; RER is the rejection rate among cases where at least one gold evidence identifier was retrieved.

System	Commit	Op. Acc.	Coverage	Sel. Acc.	GSCR	RER
Single-pass RAG	199/200	0.970	0.995	0.975	0.879	0.005
Multi-agent, no Auditor	192/200	0.955	0.960	0.995	1.000	0.015
GuidelineGuard	186/200	0.925	0.930	0.995	1.000	0.046

reports the 200-case controlled stress test. Single-pass RAG committed on 199/200 cases and was correct on 194 commitments (operational accuracy 0.970; selective accuracy 0.975). The paired no-Auditor arm committed on 192/200 and was correct on 191 (operational accuracy 0.955; selective accuracy 0.995). GuidelineGuard committed on 186/200 and was correct on 185 (operational accuracy 0.925; selective accuracy 0.995). Thus, the Auditor did not improve answer accuracy on this benchmark; it reduced coverage by 3.0 percentage points relative to the paired no-Auditor arm while leaving committed-case accuracy essentially unchanged.

The paired ablation makes the gate’s cost directly visible. Of the no-Auditor arm’s 192 commitments, 191 were correct and one was incorrect. GuidelineGuard retained 185 of the correct commitments and the one incorrect commitment, while rejecting six correct commitments. The expanded GG-Bench therefore provides no evidence that the gate improves answer discrimination in this highly controlled setting. It does show strict reference alignment: every one of GuidelineGuard’s 186 committed cases cited at least one gold evidence identifier. Across all 198 audited claim–sentence pairs, the Auditor returned 183 Entails, 9 Partially, and 6 Not verdicts. The reference-ID alignment of GuidelineGuard’s emitted citations was 0.984; this quantity should not be confused with human-adjudicated semantic entailment.

Table 13. Expanded external evaluation on the complete 500-case held-out PubMedQA PQA-L test split. PubMedQA does not provide sentence-level gold rationale identifiers, so GSCR/GER are not reported.

System	Correct Commitments	Op. Acc.	Coverage	Sel. Acc.
Single-pass RAG	322/475	0.644	0.950	0.678
Multi-agent, no Auditor	319/448	0.638	0.896	0.712
GuidelineGuard	275/368	0.550	0.736	0.747

reports the complete 500-case PubMedQA test evaluation. Here, the selective trade-off is more pronounced. Single-pass RAG achieved 0.644 operational accuracy at 0.950 coverage (322/475 committed answers correct). The paired no-Auditor arm achieved 0.638 at 0.896 coverage (319/448 correct), while GuidelineGuard achieved 0.550 at 0.736 coverage (275/368 correct). Selective accuracy increased from 0.678 for RAG to 0.712 for the no-Auditor arm and 0.747 for GuidelineGuard, but the higher committed-case accuracy came at a 16.0 percentage-point coverage reduction relative to its paired no-Auditor draft.

Among the 496 PubMedQA cases with verified-identical observable pre-audit state, the no-Auditor arm committed on 444 cases. The Auditor gate rejected 81 of those commitments: 36 were incorrect and 45 were correct according to the expert answer label. The rejected subset was therefore enriched for errors (36/81 = 0.444) relative to the verified-paired ungated committed set (129/444 = 0.291), indicating that the gate contains useful error signal, but its false-rejection burden is substantial. This is exactly why abstention cannot be described as inherently safe: whether the trade-off is desirable depends on the relative downstream cost of unsupported commitment versus unnecessary rejection.

The categorical risk–coverage operating points are shown in Figure 5. On GG-Bench-200, Entails-only, Entails-or-Partially, and ungated paired-draft points had coverage/risk of (0.890, 0.0056), (0.930, 0.0054), and (0.960, 0.0052), respectively; the observed-range discrete AURC was 0.00038. On PubMedQA-500, the corresponding points were (0.522, 0.249), (0.734, 0.253), and (0.896, 0.288), with observed-range discrete AURC 0.0971. These sparse-threshold AURCs are descriptive and are not compared numerically with the earlier continuous-score AURC because the scoring constructions differ.

5. Discussion

5.1. The Precision/Coverage Trade-Off

One of the more useful things we observed is that adding a separate verifier creates a precision/coverage trade-off that a standard accuracy benchmark cannot see. The Auditor labels each claim–sentence pair as Entails, Partially, or Not; the standard synthesis rule prefers Entails claims and uses Partially supported claims only when no Entails claim is available. This changes the set of surfaced claims and can reduce coverage; it does not, by itself, imply higher raw Citation Faithfulness. In the original ablation, raw CF was lower with the Auditor, so we interpret the gate through coverage and selective risk rather than as an intrinsically faithfulness-improving component. The pipeline exposes this trade-off through a single tunable parameter, with three operating modes:

• Strict mode (Entails only): the deployment surfaces only fully entailed recommendations and abstains otherwise. Appropriate for high-stakes settings (oncology, prescribing, pediatric dosing) where missing information is preferable to unsupported information.
• Standard mode (Entails, with Partially fallback): partially entailed claims are surfaced as a flagged fallback when no fully entailed claim is available. This is the configuration evaluated in our main results.
• Coverage mode (Clinician output without Auditor): all Clinician-produced claims are surfaced, accepting a higher unsupported-claim rate in exchange for higher coverage. Appropriate for educational, exploratory or audit settings where downstream review is mandatory.

Choosing between these modes is a deployment decision, not an architectural one. Teams can make it based on a local estimate of what matters more: the cost of false reassurance or the cost of abstaining. CDS systems should report this trade-off openly, since a single accuracy number hides it.

5.2. Why Structural Enforcement Matters

Existing safety techniques for clinical LLMs largely operate at training time (RLHF, instruction tuning, domain-specific fine-tuning) or at the prompt level (chain-of-thought, system messages, “be careful, this is medical”). Both leave guideline adherence as an emergent property of a single forward pass: the model is encouraged to behave safely but is not structurally prevented from behaving otherwise. GuidelineGuard reframes guideline adherence as a structural property of the system: an output cannot be returned without an independent verifier signing off on its grounding in a named source sentence. This is consistent with how clinical workflows themselves achieve safety through everyday mechanisms such as checklists, pharmacy verifications, second readings, and physician–pharmacist double-signatures, rather than relying on any single clinician’s intent. By aligning the system’s safety architecture with the discipline’s own safety architecture, GuidelineGuard makes verification a property the deploying institution can audit, rather than a property they must trust the model to internalize.

5.3. Practical Implications

The full pipeline is small enough to run on a clinician’s laptop and uses only open data and open weights. No proprietary model or private dataset sits in the trust path. This matters for low- and middle-income settings where neither GPT-4-class APIs nor private clinical datasets are accessible to either the developer or the deployer; the system is, in principle, deployable in a primary care clinic anywhere the underlying open-weight models are permitted. Even where private datasets and large compute budgets are available, GuidelineGuard demonstrates that the principal safety property of interest (sentence-level citation faithfulness) can be obtained without them. By releasing GG-Bench-60 and per-case adjudications we additionally provide a reusable yardstick for guideline-grounded CDS development.

5.4. Comparison with Prior Work

Compared with grounded generation based on self-reflection (Self-RAG [17]), which we evaluated as a direct baseline (Table 4, line “Self-RAG (self-critique)”), GuidelineGuard’s main difference is the strict separation of the verifier’s context from the generator’s. Self-RAG matches single-pass RAG on accuracy (0.983) and has a reported CF of 0.850 versus 0.731 for GuidelineGuard, but those CF values arise from different evaluation paths: Self-RAG self-judges, whereas GuidelineGuard uses its operational Auditor. The independent NLI cross-check in Section 4.8 probes the GuidelineGuard Auditor only and does not make the cross-system CF values directly comparable. A common independent verifier would be needed to attribute a CF difference to the systems themselves rather than to evaluator strictness. Compared with collaborative multi-agent systems (MedAgent [26], MDAgents [28]), GuidelineGuard targets a different success criterion: not inter-agent agreement, but verifiable adherence to a normative source. Compared with citation-aware generation work in general domain QA (Gao et al. [13]), our contribution is the application of the citation-faithfulness target to a high-stakes clinical setting with a benchmark built on held-out ground-truth data, evaluated under an independent verifier and replicated across model scale.

5.5. Generalizability

GuidelineGuard is corpus-agnostic: The same architecture can run over an institution’s local order set, a national guideline set, or a payer policy document, as long as recommendations can be broken into sentence-level units with stable identifiers. The Auditor does not depend on any one model either. An institution that cannot use a third-party LLM API can swap in an on-premise NLI model. The Synthesizer is an LLM call run at temperature zero and is constrained to the claims passed by the gate; it has no independent retrieval stage. These properties become important at deployment, where regulators expect institutions to control each component. Empirically, the qualitative ordering of GuidelineGuard against its baselines held across two model families and a $3\times$ range of model sizes (Section 4.4 and Section 4.5), evidence that the mechanism, structural citation enforcement, does not depend on a particular backbone.

6. Limitations

We note the following limitations of the present work, in order of importance.

• Corpus size and breadth. The corpus contains 73 sentence-level recommendations across 23 topics; specialized practice domains (e.g., hematology-oncology, transplant medicine) are not covered. Deployment in additional specialties will require corpus expansion, ideally via automated ingestion of full guideline PDFs with sentence-level provenance.
• Benchmark construction and clinical validation. The original GG-Bench-60 is author-constructed, and the 140-case expansion to GG-Bench-200 is deterministic and source-traceable rather than independently clinician-authored. The expansion improves evidence coverage and sample size but does not establish clinical realism. No independent clinician panel adjudicated the benchmark or Auditor verdicts; this remains an important limitation despite the auxiliary NLI cross-check and the expert answer labels in PubMedQA.
• Cross-system CF comparability. The original evaluation does not apply one common independent verifier to every system. GuidelineGuard CF comes from its operational Auditor, Self-RAG is self-judged, and the other citation-bearing baselines are scored post hoc by a secondary LLM. Accordingly, cross-system CF differences are descriptive and can reflect verifier strictness or evaluation granularity in addition to system behavior. A future verifier-focused study should normalize outputs to a common claim–citation representation and rescore all systems with the same blinded independent verifier.
• Auditor verifier as an LLM. The Auditor is itself an LLM and can make mistakes; we mitigate this with verification-only context and structured outputs, and we cross-checked every Auditor decision against a deterministic DeBERTa-v3 NLI verifier of a different model family (Section 4.8). This cross-check supports the internal robustness of the GuidelineGuard Auditor but does not remove the cross-system CF-comparability limitation above. Substantive agreement (binary $\kappa =0.52$ on 8B, 0.32 on 70B; NLI confirmation rate of LLM Entails 80–84% across three backbones) indicates that the reported GuidelineGuard CF values are not solely a self-evaluation artifact, but a fully deterministic on-premise verifier (e.g., a fine-tuned medical-NLI head) remains the preferred end-state for regulated deployment.
• Model coverage. The full pipeline was evaluated on three open-weight backbones spanning two model families and a $3\times$ size range, namely, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, and Qwen3.6-27B (Alibaba), and the qualitative ordering held on all three (Section 4.4 and Section 4.5); we have not evaluated on closed-API models, and the relative ordering of agentic-versus-monolithic systems on proprietary backbones remains to be confirmed.
• English-only. Both the corpus and the benchmark are in English. Extension to other languages, particularly to settings where the local guideline corpus is not in English, is left for future work.
• No prospective deployment. Clinical utility must be established in subsequent prospective work with clinician users, including measurement of override behavior, alert fatigue, and downstream patient outcomes.
• MCQ format. Our benchmark uses multiple-choice questions; while this is the dominant evaluation format in the clinical LLM literature, it does not test free-text generation quality, which an open-ended QA extension is in preparation to evaluate.
• Synthetic vignettes. The vignettes are author-written and do not capture the noise, missingness, and idiosyncrasy of real-world clinical narrative; this is necessary for an openly releasable benchmark but is a real generalizability constraint.

7. Future Work

Five directions seem most promising.

• Corpus expansion via automated ingestion of full guideline PDFs (WHO, NICE, USPSTF) with sentence-level provenance and license-aware redistribution.
• Faster auditing by replacing the LLM-based Auditor with a small fine-tuned guideline-NLI model for deterministic, latency-bounded verification; this would reduce GuidelineGuard’s per-query latency by approximately 50% on our profile.
• Iterative redrafting. A bounded one-shot exchange in which the Clinician is given the rejected (claim, sentence, verdict) tuples and asked to re-draft; this targets the abstention category of errors documented in Section 4.11.
• Multimodal extension. Coupling GuidelineGuard with a multimodal text-plus-tabular RAG pipeline for patient-specific reasoning over de-identified electronic health records.
• Prospective clinical evaluation using the system on real point-of-care queries, measuring guideline adherence for the recommendations they actually follow.

8. Conclusions

We introduced GuidelineGuard, a multi-agent pipeline in which a separate Auditor gates claim–evidence pairs before synthesis. The original experiments and the major-revision expansions lead to a deliberately narrower conclusion than the initial submission: the Auditor should be understood as a selective verification gate, not as a component that universally improves raw accuracy or citation faithfulness. On GG-Bench-200, GuidelineGuard reached 0.995 selective accuracy at 0.930 coverage, but the paired no-Auditor arm already reached 0.995 selective accuracy at 0.960 coverage; the gate rejected six correct commitments and retained the single incorrect commitment. On the complete PubMedQA-500 test split, GuidelineGuard increased selective accuracy from 0.712 in the no-Auditor arm to 0.747 while reducing coverage from 0.896 to 0.736; within the 496 verified-paired cases, the rejected subset was enriched for errors but also contained more correct than incorrect answers. These results make the central deployment question explicit: a verification gate is useful only when its reduction in unsupported commitments justifies its false-rejection cost. Future work should therefore focus on threshold calibration, independent clinician adjudication, larger and more realistic guideline corpora, and prospective evaluation rather than treating abstention itself as a safety endpoint.