Helpful or Harmful? Re-Evaluating Frugality in Retrieval-Augmented Generation for Medical Question Answering

Abstract

Medical question answering (QA) systems and conversational agents have attracted growing interest as tools that can assist clinicians, support medical students, and help patients navigate complex information sources. However, existing evaluations of retrieval strategies largely overlook the cost–benefit balance—here referred to as frugality, under realistic computational constraints. This work introduces a frugality-based evaluation framework that jointly assesses accuracy improvements and computational cost to determine when retrieval-augmented generation is beneficial in medical question answering, rather than evaluating retrieval effectiveness through accuracy alone. This study addresses these gaps through a systematic comparative framework that evaluates retrieval relevance, computational efficiency, and knowledge base composition across multiple biomedical QA tasks. We employ open-source LLMs (LlaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-7B-Chat) across three benchmark medical QA datasets, including MedMCQA, MedQA-USMLE, and PubMedQA. In addition to that, we evaluate a dataset with larger contexts to simulate model distraction across the CliniQG4QA dataset using additional models (Meditron-7B, Qwen2.5-7B-Medical, Medgemma-4B, Phi-3-mini-4k-Instruct, and GPT4o-Mini). We examine how retrieval design choices alter the accuracy–latency trade-off, examining how relevance, corpus design, and hardware constraints interact in medical retrieval-augmented generation (RAG) systems. Our comprehensive results demonstrate when retrieval is genuinely beneficial versus when it imposes unnecessary computational costs, highlighting interactions between relevance and corpus designs in QA.

1. Introduction

1.1. Motivation

Artificial Intelligence (AI) continues to transform healthcare, from clinical decision support to medical education and biomedical research. The COVID-19 pandemic further highlighted the need for scalable tools that can deliver reliable medical information under time and resource constraints [1]. In this context, medical question answering (QA) systems and conversational agents have attracted growing interest as tools that can assist clinicians, support medical students, and help patients navigate complex information sources.

Large Language Models (LLMs) underpin much of this progress. Models such as Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-7B-Chat demonstrate strong reasoning and natural language capabilities across a wide range of biomedical tasks. Despite these advances, LLMs remain limited by two structural issues in their architecture and training paradigm. First, their parametric knowledge is fixed at training time, making them vulnerable to outdated or incomplete medical information [2]. Second, they are prone to hallucinations, in other words, fluent but factually incorrect statements, which can undermine trust and pose risks in high-stakes medical decision-making [3,4].

Retrieval-Augmented Generation (RAG) has become the standard approach for addressing these limitations. By conditioning model outputs on evidence retrieved from an external knowledge base, RAG can provide more verifiable and up-to-date responses [5,6,7]. However, retrieval is not a guaranteed benefit. Sparse retrieval methods such as BM25 rely on lexical overlap and may miss semantically relevant passages [8]. Dense retrievers, including GTR-T5 and biomedical encoders such as BioBERT and S-BioBERT [9,10], improve semantic recall, but are more computationally costly. Hybrid retrieval seeks to merge lexical precision with semantic depth [11]. Conversely, large contexts with irrelevant, but semantically similar passages can lead to the model getting distracted, and providing incorrect answers [12]. Yet, two central issues have received comparatively limited examination in existing medical RAG studies.

A key open question concerns whether RAG improves performance because it retrieves genuinely relevant evidence, or simply because it adds extra text to the model’s input. This issue arises at two levels: the relevance of the specific passages selected during retrieval, and the relevance of the underlying knowledge base from which retrieval occurs. Even when each query receives the same number of retrieved passages, the informational content of those passages can differ substantially depending on the corpus, raising the possibility that gains may reflect the model’s sensitivity to added context rather than genuine evidence alignment. Prior work rarely includes control conditions such as a range of retrieval methods, dataset-provided context, or enriched knowledge bases (external knowledge base merged with dataset-provided context) that would allow these effects to be separated. Without such baselines, it is difficult to establish whether improvements arise from retrieval relevance or corpus composition, or are partly driven by the model’s sensitivity to additional context length. Distinguishing these effects is therefore essential for understanding when and why RAG provides benefits in medical QA.

The second gap concerns efficiency. Retrieval introduces latency, increases memory usage, and expands the input prompt, all of which add computational overhead. These costs are especially critical in real-world settings such as hospitals or training institutions, where models often run on modest single-GPU hardware [13,14,15,16]. Evaluating accuracy alone is insufficient without jointly examining efficiency.

Taken together, these gaps suggest that the central question is not which retrieval strategy performs best in isolation, but when retrieval is worth its computational cost in medical question answering. We therefore evaluate retrieval through a frugality lens that considers effectiveness and efficiency jointly across evidence-grounded, parametric-knowledge-dominant, and distraction-heavy medical QA settings.

1.2. Scope and Contributions

This paper addresses these gaps through a systematic evaluation of retrieval strategies under realistic computational constraints. Experiments are conducted using three open-source LLMs (LlaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-7B-Chat) across three benchmark medical QA datasets which include MedMCQA [17], MedQA-USMLE [18], and PubMedQA [19]. Alongside standard sparse, dense, and hybrid retrieval methods, we introduce three diagnostic baselines designed to isolate the contribution of retrieval relevance, knowledge base composition and context alignment:

• A novel metric used to assess the importance of RAG in the scope of data, models and retrieval;
• A dataset-context baseline that supplies PubMedQA’s ground-truth background passages to approximate an upper bound of performance under perfectly aligned evidence; and
• An enhanced knowledge base condition in which PubMedQA’s native contexts are merged into the external corpus (knowledge base) to examine whether retrieval algorithms can still surface relevant passages when they are guaranteed to exist within a substantially larger pool of distractor material.

Additionally, we examine the effects of distraction on a wider range of eight models; seven open-source (Llama-3-8B-Instruct, Medgemma-4B-It, Medical-Coding-LLM, Meditron-7B, Qwen2.5-7B-Medical, Mistral-7B -Instruct-v0.3, and Deepseek-7B-chat) and one closed-source LLM (GPT-4o-Mini) across the MIMIC-III QA dataset [20]. The same RAG methods are used to determine whether the matching trends resurface for a dataset with larger medical contexts. These experiments evaluate the models on a MCQA modification of the CliniQG4QA dataset in three main paths: (1) using only the MIMIC-III medical notes as context, (2) using medical notes along with retrieval, and (3) providing no context at all, only what is discernable from the questions and answers. These configurations allowed us to systematically examine how model performance changes with the amount of noise in the prompt.

Across these configurations, we jointly quantify effectiveness (accuracy and macro-F1) and efficiency (latency and input length), examining how retrieval design choices alter the accuracy–latency trade-off. By analysing both dimensions, this study provides a principled understanding of when retrieval is genuinely beneficial, when it imposes unnecessary computational cost, and how relevance, corpus design, and hardware constraints interact in medical QA. The key contributions of this paper include:

• A systematic benchmark of sparse, dense, and hybrid retrieval strategies under controlled single-GPU conditions (INT8 quantisation) using three medical QA datasets. This includes a joint evaluation of accuracy, F1, latency, and input length, providing clear insights into the trade-offs between retrieval quality and efficiency.
• Novel diagnostic baselines that isolate retrieval relevance, context alignment, and the impact of knowledge base composition.
• An adverse benchmark demonstrating cases where retrieved context degrades performance, showing how irrelevant or weakly aligned evidence can distract the model and reduce answer quality compared to LLM-only inference.
• Practical guidance for developing efficient and reliable RAG systems suitable for deployment in resource-constrained clinical and educational environments. This includes a novel metric (FrugalityScore) to calculate the cost-effectiveness of RAG-based prompts to LLMs.

2. Related Work

The application of LLMs to medical question answering has demonstrated both remarkable potential and significant limitations. Modern LLMs built on the Transformer architecture [21] have shown strong generalisation across diverse language tasks [22,23]. Kung et al. [24] and Nori et al. [25] showed that models like GPT-4 could approach or exceed passing scores on medical licensing examinations, while Singhal et al. [26] demonstrated strong performance on clinical reasoning tasks. However, these studies also highlighted the persistent challenge of hallucination [2,3] and parametric knowledge limitations [2], particularly concerning rapidly evolving medical guidelines.

The RAG framework, introduced by [5], addresses these limitations by integrating external knowledge retrieval during inference. Subsequent work by Izacard and Grave [6] with Fusion-in-Decoder and Gao et al. [7] in their comprehensive survey demonstrated improved factual grounding across domains. In biomedical contexts, RAG has been shown to reduce hallucinations and enhance transparency [27], though retrieval effectiveness remains dependent on passage relevance. RAG improves timeliness because indices can be refreshed without retraining and enhances data efficiency by shifting adaptation from parameters to retrieval [28,29]. However, RAG introduces significant costs: retrieval and reranking add computational overhead; longer prompts increase latency; and system complexity grows with additional components to implement, tune, and monitor [30,31].

Traditional retrieval approaches include sparse methods like BM25 [8,32], dense retrievers using general-purpose encoders [11] or domain-specific models like BioBERT [9] and PubMedBERT [10], and hybrid strategies that combine both approaches [11]. Sparse retrieval performance can be improved via query expansion and normalisation [33,34], with Bolat and Sen [34] reporting approximately 25% improvement in top-20 precision through biomedical query expansion. The MIRAGE benchmark [35] shows that well-tuned sparse retrieval can be competitive with denser pipelines on some medical tasks.

Dense retrieval maps queries and documents into shared semantic spaces via vector embeddings using dual encoders [29], with biomedical adaptations through continued pre-training, contrastive learning, and sentence-level fine-tuning [36,37,38,39]. Recent pre-training approaches, such as representation bottlenecks, further enhance dense retriever effectiveness while reducing storage overhead [40]. Cross-encoders provide stronger relevance estimates as rerankers [41,42,43], while late interaction models like ColBERT preserve accuracy with dual-encoder efficiency [44]. Although recent advances such as ColBERTv2 [45] demonstrate that late interaction models can narrow the efficiency–accuracy gap in general-domain retrieval, systematic evaluations of such methods in medical RAG pipelines remain scarce.

While dense retrieval methods improve recall, they introduce computational overhead that becomes critical in constrained environments. Recent work has addressed efficiency through inverted index optimisation [46], lightweight reranking [45], quantisation techniques [13,47,48], and sparse context selection during inference, though comprehensive efficiency–accuracy trade-off analyses remain scarce. More recent work by Meta addresses further limitations of RAG, by acknowledging wasted computational resources in computing cross-passage attention [49]. The construction of effective knowledge bases for medical RAG remains an active area of research. While some approaches rely on large-scale general medical corpora [35], others emphasise curated or hybrid domain-specific collections [35]. Recent frameworks include MKRAG for fact extraction [50], BIOMEDRAG for learned chunk scoring [51], RAG² for rationale-guided retrieval [52], and graph-aware retrieval methods that leverage structured biomedical knowledge graphs to improve evidence selection [53,54]. Retrieval optimisation also involves diversity-aware selection to reduce redundancy [55] and strategies to mitigate the “lost-in-the-middle” effect in long contexts [54]. However, few studies systematically evaluate the interaction between general medical knowledge and task-specific context, particularly whether their combination provides complementary benefits for different types of medical QA tasks.

The notion of distraction in LLMs also remains to be an active area of research. Amiraz et al. [56] show that strong retrievers that rank irrelevant passages higher than relevant ones increase the problem of distraction. Medical LLMs, specifically, are more prone to getting distracted because they lack the ability to filter out noise from the input [57]. Distraction is an issue, because it leads to unsafe and unstable performance within high-stakes environments such as within medical knowledge [12].

Despite these advances, critical gaps remain in the literature. First, the assumption that retrieval improvements arise from the relevance of retrieved evidence, rather than from the simple addition of extra context, has rarely been examined. Second, most existing evaluations focus heavily on accuracy while giving limited attention to the latency and memory costs introduced by retrieval, even though practical deployments in clinical settings often operate on modest hardware. Third, little is understood about how different types of knowledge bases interact, such as broad medical corpora and dataset-specific context, or whether combining them provides complementary benefits or introduces distracting information. Finally, there is a lack of robust guidelines for when RAG is beneficial and when it is harmful. This study addresses these gaps by framing retrieval as a cost–benefit decision and introducing a unified frugality-based evaluation that jointly considers effectiveness and efficiency across biomedical QA tasks.

Finally, we address more general gaps in the literature pointed out by medical RAG survey papers. Abo El-Enen et al. [58] and Gargari and Habibi [59] note that there is a need for standardised evaluation frameworks, and domain-specific benchmarks that assess RAG across multiple dimensions. We address this through a Frugality metric across multiple medical datasets, assessing such systems across efficiency and cost. Additionally, He et al. [60] identify that latency is an issue for many RAG-based AI systems; however, they do not find a way to quantify this cost. The authors also find that there is a need to explore hybrid RAG approaches within the medical domain, but they do not assess their disadvantages. We aim to fill this gap by quantitively comparing hybrid RAG approaches against dense and sparse ones.

Miao et al. [61] highlight that future work should explore and optimise the balance between retrieval, precision, and efficiency in a healthcare setting. Our work proposes a tunable metric that can quantify this balance specifically for real-world data. Finally, there is a specific need within literature to explore the use of reranking, and integrating domain-specific embeddings such as BioBERT [62], which we address as part of our methodology that evaluates a range of RAG methods, including the use of re-ranking for hybrid approaches.

3. Methods

We employ a controlled comparative framework evaluating multiple retrieval strategies across three medical QA datasets and three instruction/chat-tuned 7–8B models (LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-7B-Chat). We additionally, evaluate a dataset with larger contexts to simulate model distraction across the CliniQG4QA dataset, using eight models (Llama-3-8B-Instruct, Medgemma-4B-It, Medical-Coding-LLM, Meditron-7B, Qwen2.5-7B-Medical, Mistral-7B-Instruct-v0.3, Deepseek-7B-Chat, and GPT4o-Mini). All experiments in this study were conducted under single-GPU INT8 quantisation to simulate realistic clinical constraints. The experimental design isolates the effects of retrieval strategy, knowledge base composition, and model architecture on both accuracy and efficiency metrics. Figure 1 provides an overview of the pipeline, from datasets through retrieval to inference, and also illustrates the relationships between datasets and knowledge sources.

Figure 1. Overview of the experimental pipeline, showing the flow from benchmark datasets through the retrieval modules (sparse, dense, and hybrid) and the alternative corpus conditions (external knowledge base, dataset context, and enhanced knowledge base). Retrieved or dataset-provided evidence forms the Background Information block used during prompt assembly. The constructed prompt is processed by one of the LLM generators (LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, DeepSeek-7B-Chat, MedGemma-4B, GPT4o-mini, Meditron-7B, Qwen2.5-Medical, or Medical-Coding-LLM), and the system records retrieval time, input word budget, end-to-end latency, and prediction accuracy/F1, awlongside a visual representation of relationships between datasets and knowledge sources used in the experiments.

The models evaluated in this paper can be grouped into two main categories: (i) models that have been medically tuned, either during pretraining in the case of Meditron-7B [63] or fine-tuned such as MedGemma-4B-It [64], and Medical-Coding-LLM which is a Phi-3-Mini-4k based model (https://huggingface.co/Kavyaah/medical-coding-llm [accessed on 3 December 2025]), and (ii) models that have not been trained directly on medical knowledge—general purpose LLMs. The performance of individual models often cannot be directly explained by the level of domain adaptation, or model size, since our experiments used models of similar sizes around 8B parameters. Instead, they emerge from a combination of (a) pre-training data scale and diversity (studies comparing medical LLMs with their base models show that models with a broader medical pre-training can have notable accuracy gains [65,66]); (b) methods for domain adaptation, such as LoRA or QLoRA, which affect accuracy without changing model size [67] and alignment methods used [68]; and (c) architectural decisions made during training and designing the model, such as choice of attention algorithm [69].

3.1. Datasets and Knowledge Base Construction

We evaluate on three complementary medical QA benchmarks that represent diverse reasoning requirements in clinical and biomedical contexts, and one less structured and more challenging dataset:

• MedMCQA [17]: A multi-subject dataset of 194,000 multiple-choice questions, from which we use 1001 validation questions spanning Surgery, Medicine, and Pathology to maintain scale parity with the other datasets while ensuring specialty diversity.
• MedQA-USMLE [18]: Comprising 1273 USMLE-style clinical vignettes that require multi-step reasoning across patient presentations, diagnostic indicators, and treatment options, with answer choices normalised to A–E.
• PubMedQA [19]: A research-focused dataset of 1000 yes/no/maybe expert-annotated questions, where each question is explicitly grounded in PubMed abstracts, testing evidence-based reasoning from biomedical literature.
• CliniQG4QA [20]: A set of 1287 diverse extractive questions derived from the MIMIC-III dataset [70], consisting of an admission note as context, and questions.

Our primary external knowledge base comprises 23 credible medical sources totalling approximately 14 million words (119 MB), including standard medical textbooks, clinical handbooks, and peer-reviewed open-access journals (see Appendix B Table A1).

Documents were processed using LangChain’s unstructured document loader with Tesseract OCR for image-based PDFs; we adopt fixed-length character-based chunking (500 characters with 50-character overlap) as a tokeniser-agnostic preprocessing strategy. This approach has some limitations: it may split biomedical terms, citations, or abbreviations, and it makes the semantic content of a “chunk” dependent on the formatting density of the corpus. We therefore do not claim that character-based chunking is optimal in terms of retrieval quality.

Character-level chunking was chosen for two main practical reasons. First, it ensures consistent chunk sizes across all datasets and retrievers without the need for model-specific tokenisers, which may use different algorithms or introduce additional variables when comparing retrievers. Second, it enables reproducible, deterministic preprocessing across all retrievers, ensuring that all methods receive identical evidence. This chunking approach makes our benchmark more robust to noisy segmentation.

While token-based chunking could preserve semantic boundaries more faithfully, it introduces a dependency on model-specific tokenisers, making the notion of “k retrieved passages” less comparable across different retrieval variations. Character-based chunking therefore reflects a trade-off between semantic accuracy and experimental control.

For PubMedQA experiments, we additionally construct dataset-context passages only (approx. 212K words, 1.3 MB) and an enhanced knowledge base obtained by merging the external medical corpus with the dataset’s native context passages (approx. 14.3 million words, 120 MB), enabling direct investigation of knowledge base composition effects. This size asymmetry is important for interpreting the results in the Section 4, where differences across external retrieval, dataset-context only retrieval, and the enhanced merged knowledge base can be related not only to accuracy and latency but also to the scale and alignment of the evidence available. Figure 1 summarises the corpus statistics, and visually depicts datasets and corpus relationships.

For CliniQG4QA, a script (https://github.com/MeRichard123/medical-rag-evaluation/blob/36618628852fea3bfc1c5da79b795e97bbd93c99/data/mimic-annotated-QA/convert_to_mcqa.py [accessed on 10 December 2025]) is used to generate an MCQA dataset from the initial open-ended questions. This was done by appending three random options into the answers array. CliniQG4QA was originally intended to be open-ended; as a result, the answers appear within the context creating an extractive task for the model. For the MIMIC-III evaluation, we mask out the answer from the context to force the model to use the surrounding knowledge of the question and context rather than a lookup.

3.2. Theory and Formulation

We can formalise Figure 1 if we consider $\mathcal{D}$ as the dataset used for evaluation, and let ${\mathcal{D}}_c\subset \mathcal{D}$ such that ${\mathcal{D}}_c$ contains the dataset context for datasets where those are present; i.e., ${\mathcal{D}}_c$ is the supporting evidence for each question. Note that $\mathcal{D}$ is a generalisation over any of the evaluated datasets, and ${\mathcal{D}}_c$ is the dataset context for either PubMedQA or CliniQG4QA.

Prompt context is retrieved by a retriever $\mathcal{R}$. Given a query question q, we retrieve a list of candidate contexts from a document d from ${\mathcal{D}}_c$.

The BM25 method computes a relevant score s using the TF-IDF algorithm, which uses double normalisation for stability. It can be defined as

$$\begin{array}{c}\hfill s_{\mathrm{BM}25}(q,d)=\sum _{t\in q}\mathrm{IDF}\left(t\right)·{\displaystyle \frac{f(t,d)(k_1+1)}{f(t,d)+k_1(1-b+b\frac{\left|d\right|}{\mathrm{avgdl}})}}\end{array}$$

(1)

where

• $f(t,d)$ = the frequency of term t in document d
• $\left|d\right|$ = length of document d
• avgdl = average document length in ${\mathcal{D}}_c$
• $k_1$ = term saturation parameter
• b = length normalisation parameter
• $\mathrm{IDF}\left(t\right)=log{\displaystyle \frac{N_d-n_t+0.5}{n_t+0.5}}$
• $N_d=\left|{\mathcal{D}}_c\right|,n_t$ = number of documents containing t

We define retrieval functions in terms of score ${\mathcal{R}}_{\mathrm{BM}25}(q,{\mathcal{D}}_c,k)={\mathrm{TopK}}_{d\in {\mathcal{D}}_c}\left(s_{\mathrm{BM}25}(q,d)\right)$, which returns the top-k most relevant contexts $\{d^{\left(1\right)},\dots ,d^{\left(k\right)}\}$, with $k=5$ in most of our experiments. Returned contexts are used by the generator within the prompt.

Dense Retrieval similarly creates a retriever using dense vectors. Let $E_q\left(q\right)\in {\mathbb{R}}^d$ and $E_d\left(d\right)\in {\mathbb{R}}^d$ be encoder functions that map a query and a context to a dense vector; as such, we use the cosine similarity score: $s_{\mathrm{dense}}(q,d)=cos(E_q\left(q\right),E_d\left(d\right))$. Such a similarity score can be approximated in a more computationally efficient manner as $\mathrm{FAISS}(E_q\left(q\right),E_d\left(d\right))$. The dense retriever can then be defined in terms the dense score

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{dense}}(q,{\mathcal{D}}_c,k)& ={\mathrm{TopK}}_{d\in {\mathcal{D}}_c}\left(s_{\mathrm{dense}}(q,d)\right)\hfill \end{array}$$

(2)

Hybrid Retrieval is a ranked combination of $s_{\mathrm{BM}25}(q,d)$ and $s_{\mathrm{dense}}(q,d)$, which uses an Ensemble Retriever that computes the Reciprocal Rank Fusion algorithm [71]:

$$\begin{array}{c}\hfill s_{\mathrm{hybrid}}(d\in {\mathcal{D}}_c)={\displaystyle \frac{1}{k+r_{\mathrm{BM}25}\left(d\right)}}+{\displaystyle \frac{1}{k+r_{\mathrm{dense}}\left(d\right)}}\end{array}$$

(3)

Retrieval is then defined as ${\mathcal{R}}_{\mathrm{hybrid}}={\mathrm{TopK}}_{d\in {\mathcal{D}}_c}\left(s_{\mathrm{hybrid}}(s,d)\right)$; the Top-K candidates. This candidate pool can be denoted as $\mathcal{C}$.

Hybrid retrieval returns a larger number of documents due to fusion between BM25 and Dense retrieval. As such, there is a need to rerank documents using a reranker $r(q,d)$ to obtain a second score $s_{\mathrm{rerank}}(q,d)$.

Prompting was used to obtain an output from the model. Let a prompt P be defined as the concatenation of three components $P=⟨P_f,q,\mathcal{C}⟩$; few-shot examples $P_f$, a query q, and retrieved background context $\mathcal{C}$ defined by the retrieved documents from a retrieval operator $\mathcal{R}(q,{\mathcal{D}}_c,k)=\{d^{\left(1\right)},\dots ,d^{\left(k\right)}\}$.

The Prompt is passed to an LLM $\mathbf{Generator}$ $G_{\theta }$ with parameters $\theta$. We define a generated answer vector $y=(y_1,\dots ,y_T)$ with probability

$$\begin{array}{c}\hfill p_{\theta }\left(y\right|P)=\prod _{t=1}^T{p}_{\theta }\left(y_t\right|y_{<t},P),\end{array}$$

(4)

which is decoded using the Hugging Face pipeline $\widehat{y}=\mathrm{Decode}(G_{\theta },P;\mathrm{\Omega })$, where $\mathrm{\Omega }$ is the decoding configuration such as max tokens, temperature parameter of zero (for deterministic outputs), and scored output $\widehat{y}\in \{A,B,C,D\}\left|\{A,B,C,D,E\}\right|\{\mathrm{Yes},\mathrm{No}\}$.

3.3. Retrieval Strategies and Configurations

Retrieval strategies and baselines are designed not to maximise accuracy, but to isolate the effects of retrieval relevance, corpus composition, and context length on both model performance and computational cost.

This study evaluates retrieval under controlled, single-GPU conditions by implementing three principal retrievers, which are sparse, dense, and hybrid, together with three diagnostic baselines designed to isolate the contribution of relevance, context specificity, and retriever sensitivity. All retrieval systems are constrained to return k = 5 passages per query to ensure methodological comparability across datasets. Table 1 summarises all experimental configurations.

Table 1. Summary of experimental configurations.

Variant	Retrieval	Evidence Source	k	Reranker	Notes
LLM-only	None	N/A	N/A	N/A	Baseline without external context.
RAG (BM25) + LLM	Sparse (BM25)	Top-k chunks from BM25	5	None	Lexical match; typically shorter, higher-overlap snippets.
RAG (Dense) + LLM	Dense (GTR-T5-Large)	Top-k FAISS neighbours	5	None	Strong general semantic recall.
RAG (Dense) + LLM	Dense (S BioBERT)	Top-k FAISS neighbours	5	None	Domain-adapted embeddings for biomedical terminology.
RAG (Hybrid) + LLM	Sparse + Dense (GTR-T5-Large)	Fused candidates from BM25 + GTR-T5	5	MiniLM	Keep top $N=5$ after reranking.
RAG (Hybrid) + LLM	Sparse + Dense (S BioBERT)	Fused candidates from BM25 + S BioBERT	5	MiniLM	Keep top $N=5$ after reranking.
PubMedQA dataset context + LLM	None (direct insertion)	Contexts field from dataset	N/A	N/A	Ground-truth context: perfectly aligned evidence.
Dataset-Context Injection (PubMedQA)	Sparse/Dense/ Hybrid	External KB + dataset context merged	5	Same as corresponding retriever	Checks retriever ability to surface known-relevant evidence within a larger corpus.

The sparse retriever uses the classical BM25 inverted index [8,32]. Sparse methods remain highly efficient on curated medical corpora due to strong lexical precision and well-behaved term distributions. BM25 therefore represents a robust, interpretable baseline for assessing retrieval from authoritative biomedical sources.

The dense retriever performs exact inner-product similarity searching using FAISS (IndexFlatIP), operating over L2-normalised embeddings. Two encoders are employed: Google’s GTR-T5-Large (https://huggingface.co/sentence-transformers/gtr-t5-large [accessed on 2 December 2025]), which provides broad semantic recall across heterogeneous text, and pritamdeka/S-BioBert-snli-multinli-stsb, denoted as S-BioBERT (https://huggingface.co/pritamdeka/S-BioBert-snli-multinli-stsb [accessed on 2 December 2025]), a domain-adapted encoder with improved handling of biomedical synonymy and terminology variation. Approximate nearest-neighbour indexing is deliberately avoided to eliminate stochastic approximation noise and ensure deterministic comparisons across variants.

The hybrid retriever integrates these complementary signals through a two-stage process. First, BM25 and dense retriever scores are linearly fused with equal weights, producing a joint candidate pool. Second, the top-10 fused candidates are evaluated by a compact cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2, denoted as MiniLM (https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2) [accessed on 2 December 2025]), and the final top-5 passages are retained. This pattern reflects contemporary multi-stage retrieval pipelines, which achieve higher precision at small evidence budgets while containing latency.

Beyond these core retrievers, the study introduces three diagnostic baselines that probe different methodological questions fundamental to RAG. First, the dataset-context condition, used for PubMedQA, replaces external retrieval entirely with the dataset’s own ground-truth background passage. This condition acts as an upper-bound reference for retrieval quality because the supplied context is perfectly aligned with the question. Evaluating an LLM with this curated context reveals the maximum attainable performance when retrieval is ideal, providing a benchmark relative to which external retrieval strategies can be interpreted.

Second, the study evaluates a dataset-context-injection condition, a sensitivity test designed to assess whether retrieval algorithms can surface highly relevant evidence when it is embedded within a large external corpus. In this setting, the PubMedQA abstracts are merged into the external knowledge base, effectively guaranteeing the presence of relevant evidence among millions of distractor tokens. The sparse, dense, and hybrid retrievers are then executed normally. This allows us to test whether a retriever is capable of identifying relevant passages when they exist in the corpus, thereby distinguishing retrieval-algorithm sensitivity from limitations imposed by corpus composition. This condition also reflects real-world biomedical settings, where relevant evidence is often present but embedded in large, noisy repositories.

Finally, assessing the performance under increased distractive circumstances, MIMIC-III is based on real-world unstructured data including admission notes, medications, laboratory notes and doctor observations [70], resulting in large contexts with significant levels of distractor material. Along with retrieved context, this creates an increased amount of text for the model to decide its relevance.

Collectively, the three principal retrievers and the three diagnostic baselines provide a comprehensive framework for evaluating retrieval quality, retrieval relevance, retrieval sensitivity, and the interaction between context alignment and RAG efficiency.

Across all datasets and retrieval variants, the study evaluates 113 controlled experimental configurations. For MedMCQA, MedQA-USMLE each model is assessed under six conditions: LLM-only, sparse retriever (BM25), dense retrievers (GTR, S-BioBERT), hybrid (GTR, S-BioBERT). For PubMedQA, and CliniQG4QA each model is evaluated under seven conditions, combining the LLM-only and primary retrieval methods with the dataset-context condition and the dataset-context-injection experiments. With three models evaluated across these settings, the experimental design yields 57 configurations, with an additional eight models for distractor material across the seven conditions yielding an additional 56 configurations, enabling systematic analysis of retrieval relevance, corpus sensitivity, distraction, and efficiency.

3.4. LLM Configuration and Inference Framework

All retrieval conditions feed into one of three 7B-8B class instruction/chat-tuned models selected for their diverse architectural characteristics and alignment strategies: (i) LLaMA-3-8B-Instruct: optimised for factual accuracy and reasoning with grouped-query attention, (ii) Mistral-7B -Instruct: balances performance with computational efficiency through sliding-window attention, and (iii) DeepSeek-7B-Chat: emphasises conversational fluency with Reinforcement Learning from Human Feedback (RLHF) alignment. Models are deployed using INT8 weight-only quantisation with FP16 activations [13] to support single-GPU deployment. Inference runs are conducted on an NVIDIA RTX 4070 SUPER (12 GB VRAM) workstation equipped with an Intel i5-14600KF CPU and 32 GB RAM. We employ greedy decoding ($\mathrm{\Omega }=\{$do_sample=False, max_new_tokens=10}) to enforce discrete outputs, eliminate sampling variance, and minimise latency inflation from verbose generations. Each query is processed individually with warm-up iterations excluded from timing measurements. It is important to note that no fine-tuning was applied in this research; and example prompts and model outputs can be found in Appendix D. Configuration of each model is shown in Table 2. For clarity and interpretability, figures and tables in the main manuscript focus on the three core 7–8B models and primary datasets used to establish the central contribution of this work. Additional models and datasets introduced later are used as diagnostic extensions to examine robustness under distraction and extreme context misalignment and are therefore not included in all figures and tables.

Table 2. Configuration per model.

Parameter/Setting	LLaMA-3-8B-Instruct	Mistral-7B-Instruct	DeepSeek-7B-Chat
Precision	INT8 (weights), FP16 activations	Same as LLaMA-3-8B	Same as LLaMA-3-8B
Quantisation method	LLM.int8	LLM.int8	LLM.int8
Max new tokens	10	10	10
Sampling (do_sample)	False (greedy decoding)	False (greedy decoding)	False (greedy decoding)
Temperature	0 (deterministic)	0	0
Top-p/nucleus sampling	Not used	Not used	Not used
Batch size	1 (single query)	1	1
Device	Single NVIDIA GPU (12 GB VRAM)	Same	Same
Tokeniser library version	HuggingFace Transformers v4.53	Same	Same
License (summary)	Llama-3-8B Community License	Apache 2.0	MIT

Distraction LLM Configuration

In addition to the models used for RAG, extra models following the same configuration were evaluated to assess how well different models can discern the correct answers from large contexts: (i) Meditron-7B, Qwen2.5-7B-Medical, Medical-Coding-LLM(Phi-3-mini-4k), and Medgemma-4B—all are pretrained on a form of medical data and therefore have medical knowledge in their initial knowledge base, pertaining to enhanced performance on medical tasks [72]. (ii) GPT4o-Mini, a proprietary LLM, from OpenAI, in order to assess the state-of-the-art alongside open-source LLMs.

3.5. Evaluation Metrics

We adopt a comprehensive evaluation framework assessing both effectiveness and efficiency dimensions. Our evaluation pipeline ensures consistent measurement across all experimental conditions. Effectiveness is quantified using Accuracy (i.e., the proportion of correct predictions) and Macro-F1, computed as

$$P_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c}},R_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}},{F1}_c={\displaystyle \frac{2P_c·R_c}{P_c+R_c}}.$$

Macro-F1 is the unweighted mean over all classes C, Macro-F1=${\displaystyle \frac{1}{\left|C\right|}}{\sum }_{c\in C}F{1}_c$, where $TP$ = true positives, $FP$ = false positives, $FN$ = false negatives, P = precision, and R = recall.

Efficiency is evaluated using two complementary indicators:

• End-to-end latency, measuring total elapsed time (in seconds) per query from retrieval initiation to post-generation parsing. This metric captures the cumulative cost of the full RAG pipeline, including retrieval, reranking, and LLM prefill/decoding, presented as $T_{\mathrm{Latency}}=T_{\mathrm{Retrieval}}+T_{\mathrm{Reranking}}+T_{\mathrm{Prefill}}+T_{\mathrm{Decode}}$. While transformer attention incurs a theoretical $O\left(n^2\right)$ cost in sequence length [73], our experiments are not intended to isolate generation complexity. Instead, it captures system-level efficiency, where non-LM components like retrieval are allowed to dominate latency depending on retrieval strategy.
• Average input word length is calculated as the mean whitespace-delimited word count of the model input (instruction + question + context). Although Transformer cost scales with tokens, word count is used as a tokeniser-agnostic indicator of input length for consistent cross-model comparison. It is therefore treated as a descriptive correlate of prompt size rather than a direct proxy for prefill cost. End-to-end latency remains the primary efficiency metric, with word length used to characterise relative prompt expansion across retrieval conditions.

For statistical analysis, paired non-parametric Wilcoxon signed-rank tests are applied to compare Accuracy, Macro-F1, and latency across matched dataset × model pairs. Pearson correlation coefficients are additionally computed between prompt length and latency to quantify efficiency dependencies. Where sample size (N < 9) precludes significance testing, results are reported descriptively. All subsequent analyses are interpreted primarily through the frugality framework introduced in Section 5, with accuracy and latency treated as contributing factors rather than standalone endpoints.

Metrics for Extended Models

In addition to F1 and Accuracy metrics, exact match and mean reciprocal rank is used to assess each model. Exact Match (EM) assesses the percentage of samples where the model output exactly matches the gold standard answer; in most cases, this is equivalent to accuracy. Mean Reciprocal Rank (MRR) assesses partial correctness via the average rank at which the first relevant answer appears in a set of candidate answers, computed as

$$\begin{array}{c}\hfill MRR={\displaystyle \frac{1}{\left|Q\right|}}\sum _{i=1}^{\left|Q\right|}{\displaystyle \frac{1}{ran{k}_i}},\end{array}$$

(5)

where $\left|Q\right|$ is the number of queries of ground truths, and $ran{k}_i$ is the rank position or index of the correct answer.

3.6. Implementation Environment and Reproducibility

All experiments were executed on a single workstation equipped with an NVIDIA RTX 4070 SUPER (12 GB VRAM), an Intel i5-14600KF CPU, and 32 GB RAM. The full pipeline including dataset preprocessing, knowledge base construction, retrieval backends, prompt assembly, and evaluation harness was implemented in Python 3.11 using open-source libraries such as transformers, torch, sentence-transformers, faiss-cpu, pyserini, and langchain. Deterministic decoding (greedy, no sampling) and pinned library versions were used to ensure reproducibility and to prevent silent CPU fallbacks. For each query, the system logs the ground-truth label, model prediction, end-to-end latency, and input length, with aggregated metrics exported as CSV summaries. All code, documentation, and configuration files required to reproduce the experiments are publicly available (https://github.com/FiyinOloyede/Efficient_RAG_with_LLMs_for_Medical_Question_Answering [accessed on 12 December 2025]).

4. Results

4.1. Retrieval Experiments

Table 3 summarises the performance of the three models across configurations, with their means at the end of each column, and Table 4 shows the latencies of each method with respect to its input length. Across all retrieval experiments, Llama-3-8B ranked highest overall, with Mistral-7B coming second and DeepSeek-7B last.

Table 3. Accuracy and F1 scores across datasets, retrieval variants, and models. Bold values indicate dataset-level means across all variants; PubMedQA excludes dataset_context to ensure parity.

Dataset	Variant	Accuracy			F1
		LLaMA3-8B	Mistral-7B	DeepSeek-7B	LLaMA3-8B	Mistral-7B	DeepSeek-7B
MedMCQA	LLM-only	0.6284	0.4935	0.3816	0.5007	0.4850	0.2828
	bm25	0.5864	0.4505	0.3586	0.5803	0.3523	0.2715
	dense-gtr	0.6054	0.4915	0.3736	0.5976	0.3835	0.2912
	dense-sbiobert	0.6244	0.4805	0.3616	0.6187	0.3786	0.2793
	hybrid-gtr	0.6094	0.4695	0.3716	0.6035	0.3667	0.2895
	hybrid-sbiobert	0.6024	0.4845	0.3686	0.5968	0.3770	0.3634
	Mean	0.6094 $\pm 0.014$	0.4783 $\pm 0.015$	0.3693 $\pm 0.008$	0.5829 $\pm 0.038$	0.3909 $\pm 0.043$	0.2963 $\pm 0.031$
MedQA-US	LLM-only	0.5562	0.4400	0.3213	0.4615	0.4338	0.2969
	bm25	0.5240	0.2954	0.2647	0.5170	0.2330	0.2476
	dense-gtr	0.5389	0.3865	0.2781	0.5357	0.3845	0.2706
	dense-sbiobert	0.5216	0.3488	0.2757	0.5151	0.3435	0.2653
	hybrid-gtr	0.5334	0.3346	0.2773	0.5287	0.3264	0.2662
	hybrid-sbiobert	0.5326	0.2899	0.2749	0.5302	0.2277	0.2584
	Mean	0.5345 $\pm 0.011$	0.3492 $\pm 0.052$	0.2820 $\pm 0.018$	0.5147 $\pm 0.025$	0.3248 $\pm 0.075$	0.2675 $\pm 0.015$
PubMedQA	LLM-only	0.4860	0.2780	0.4660	0.2550	0.1687	0.2717
	dataset-context	0.7250	0.5960	0.4160	0.4994	0.3489	0.2769
	bm25	0.5530	0.3080	0.3110	0.3346	0.2144	0.2031
	dense-gtr	0.5740	0.3500	0.3210	0.3542	0.2149	0.2153
	dense-sbiobert	0.5660	0.3400	0.2840	0.3604	0.2093	0.1934
	hybrid-gtr	0.5580	0.3410	0.3170	0.2575	0.2094	0.2147
	hybrid-sbiobert	0.5540	0.3340	0.3200	0.3381	0.2121	0.2165
	Mean	0.5636 $\pm 0.067$	0.3639 $\pm 0.098$	0.3479 $\pm 0.062$	0.3427 $\pm 0.076$	0.2254 $\pm 0.053$	0.2274 $\pm 0.031$
Overall		0.5726 $\pm 0.052$	0.3954 $\pm 0.087$	0.3338 $\pm 0.053$	0.4705 $\pm 0.116$	0.3089 $\pm 0.090$	0.2516 $\pm 0.066$

Table 4. Average latency and input length per prompt across datasets, retrieval strategies, and models (lower values are better). Bold values indicate dataset-level means across all variants.

Dataset	Variant	Avg. Latency (s)			Avg. Num. of Words
		LLaMA3-8B	Mistral-7B	DeepSeek-7B
MedMCQA	LLM-only	0.90	1.04	0.45	52
	bm25	1.85	2.15	1.47	542
	dense-gtr	2.83	1.39	0.66	559
	dense-sbiobert	1.05	1.43	0.60	537
	hybrid-gtr	1.87	2.27	1.45	562
	hybrid-sbiobert	1.86	2.27	1.43	551
	Mean	1.7267 $\pm 0.6946$	1.7583 $\pm 0.5360$	1.0100 $\pm 0.4446$	467.167 $\pm 203.615$
MedQA-US	LLM-only	0.89	1.05	0.59	164
	bm25	7.06	7.67	7.05	745
	dense-gtr	3.78	1.54	0.76	717
	dense-sbiobert	1.09	1.45	0.75	704
	hybrid-gtr	7.33	7.76	7.10	733
	hybrid-sbiobert	7.19	7.62	6.80	726
	Mean	4.5567 $\pm 3.0646$	4.5150 $\pm 3.4749$	3.8417 $\pm 3.4436$	631.50 $\pm 229.45$
PubMedQA	LLM-only	0.36	0.60	0.20	33
	dataset-context	1.01	0.55	0.23	369
	bm25	1.66	1.64	0.93	462
	dense-gtr	2.59	1.03	0.32	490
	dense-sbiobert	1.03	0.94	0.28	472
	hybrid-gtr	2.56	1.64	0.97	481
	hybrid-sbiobert	1.69	1.62	0.95	472
	Mean	1.6483 $\pm 0.8674$	1.2450 $\pm 0.4490$	0.6083 $\pm 0.3765$	401.667 $\pm 180.856$
Total Mean		2.6439 $\pm 2.2502$	2.5061 $\pm 2.4245$	1.8200 $\pm 2.4065$	500.11 $\pm 217.244$

In terms of efficiency, DeepSeek-7B consistently achieved the lowest average latency across datasets, making it a better choice for performance-critical applications. RAG offered clear improvements on PubMedQA, but remained marginal for MedMCQA, MedQA-US.

The retrieval methods provide only modest gains in model performance. Across the experiments, Dense retrieval achieved the highest performance, followed closely by Hybrid, with BM25 trailing by a small margin. In contrast, latency analysis highlighted that certain retrievers were preferable due to faster response times.

RAG appeared to be most beneficial for literature-grounded datasets like PubMedQA, where the LLM-only accuracy ($0.4860$) was significantly lower than the average RAG performance ($0.5883\pm 0.0615$). In particular, allowing the model to also fetch the data context during retrieval resulted in $72.50\%$ accuracy.

4.1.1. Retrieval Strategy Comparison

Across all dense encoders, GTR-T5 shows marginal improvement over S-BioBERT regarding Accuracy and F1 score. GTR-T5 achieves an average accuracy of $0.4526\pm 0.119$ compared to $0.4225\pm 0.1281$ from S-BioBERT (Table A2). Unlike accuracy, there is a significant efficiency gain between the two models in terms of latency, with GTR-T5 taking an average of $1.6556\pm 1.1629$ s (s) and S-BioBERT taking on average $0.9578\pm 0.3754$ s (Table A3).

The statistical significance of the results can be confirmed using a Wilcoxon signed-rank test (Table 5). This was done using R studio, where zero-differences are ignored and ties are averaged and ranked. The test results show that the accuracy and F1 differences are statistically insignificant, unlike the difference in latency, which showed statistical significance. Since the accuracy difference between the two encoders was marginal, S-BioBERT is favoured due to its faster retrieval latencies.

Table 5. Wilcoxon signed-rank test results for dense retrieval and hybrid retrieval (significant in bold).

Metric	N	V(${\mathit{W}}^{+}$)	$\mathit{p}<0.05$
	Dense retrieval methods
Accuracy	9	38	0.0663
Macro-F1	9	32	0.1731
Avg. Latency (s)	9	42	0.0208
	Hybrid retrieval methods
Accuracy	9	33.5	0.1917
Macro-F1	9	18	0.5940
Avg. Latency (s)	8	36	0.0115

The hybrid retrievers show a similar pattern, with Hybrid-GTR attaining a higher accuracy, but lower F1 when compared to Hybrid-SBioBERT. The latency difference was a lot lower for the hybrid retrievers, yet still statistically significant (Table 5).

A hierarchy emerges when we compare the three retrievers (BM25, Dense S-BioBERT, and Dense GTR). BM25 performs the worst overall, with the lowest average accuracy: $0.4057\pm 0.1241$, lowest average F1: $0.3282\pm 0.1356$ (Table A2), and second highest average latency: $3.4978\pm 2.8456$ (Table A3). Dense S-BioBERT exhibits statistically equivalent performance to Dense GTR in terms of Accuracy (Wilcoxon signed rank, p = 0.0663), while achieving significantly lower retrieval latency (p = 0.0208), indicating a better efficiency–effectiveness trade-off.

Across all datasets and models, GTR-T5 achieved a marginally higher average accuracy ($0.4526\pm 0.119$) than S-BioBERT ($0.4225\pm 0.1281$), with a similarly small difference in macro-F1. However, S-BioBERT was substantially more efficient, reducing average latency from $1.6556\pm 1.1629$ to $0.9578\pm 0.3754$. A Wilcoxon signed-rank test (Table 6) confirms that the accuracy and F1 differences are not statistically significant in most cases; there was a significant difference between BM25 and Hybrid-SBioBERT, while the latency reduction is significant. Given the negligible performance gap and consistent efficiency gains, S-BioBERT is favoured as the dense encoder for subsequent experiments.

Table 6. Wilcoxon signed-rank test results for retrieval (significant in bold).

Comparison	Metric	N	V(${\mathit{W}}^{+}$)	$\mathit{p}<0.05$
BM25 vs. Dense-SBioBERT	Accuracy	9	6	0.0506
	Macro-F1	9	7	0.0663
	Avg. Latency (s)	9	45	0.0077
BM25 vs. Hybrid-SBioBERT	Accuracy	9	2	0.0152
	Macro-F1	9	4	0.0284
	Avg. Latency (s)	9	22	0.952765
Dense S-BioBERT vs. Hybrid-SBioBERT	Accuracy	9	26	0.6784
	Macro-F1	9	24	0.8590
	Avg. Latency (s)	9	0	0.0077

Overall, these statistical tests and results demonstrate that dense retrieval, in particular with a biomedical encoder like S-BioBERT, is the most effective and efficient approach to retrieve context. Hybrid approaches offer little advantage over dense ones, and BM25 consistently underperformed.

4.1.2. Model Comparison

Although the three models are similar in scale (7–8 billion parameters), their behaviour differs substantially. Llama-3-8B consistently achieved the best performance across all three datasets, followed by Mistral-7B and DeepSeek-7B. The performance of Mistral-7B and Llama-3-8B remained similar, where Mistral-7B maintained a middle ground between the two models.

The models differed in terms of response times. DeepSeek-7B achieves the lowest average latencies: $1.8200\pm 2.4065$ s compared to $2.6439\pm 2.2502$ s and $2.5061\pm 2.4245$ for Llama-3-8B and Mistral-7B, respectively, on the MedMCQA dataset. Increased latency correlated with a larger prompt; this was reflected in MedQA-US, where the average latencies were $3.8417\pm 3.44$ s, $4.5150$ s $\pm$ 3.47, and $4.5567$ s $\pm$ 3.06 for DeepSeek-7B, Mistral-7B and Llama-3-8B respectively. The same trend is seen on PubMedQA: $0.6083\pm 0.38$ s, $1.2450\pm 0.45$ s, and $1.6483\pm 0.87$ s, in the same order as previously listed. These differences illustrate DeepSeek’s ability to process prompts efficiently. Literature suggests this may be driven by low-level system optimisation that improves the computational throughput of the model, such as the use of grouped-query attention [74], use of BF16 at inference for all layers (Appendix E Figure A10), and adjusting the layer sizes to make the model deeper rather than wider, which, according to DeepSeek-AI et al. [75], helps facilitate pipeline partitioning to speed up inference.

4.2. Distraction Experiments

The experiments in this section introduce additional models and datasets as diagnostic extensions to evaluate the frugality framework under extreme context misalignment, where large and weakly relevant inputs increase the risk of model distraction. The CliniQG4QA dataset constructs an extractive task, and as a result, the question is framed around the answer. This means when converted to MCQA, the answer is often discernable from the question itself. Table 7 presents the performance of each model having been given no context. Most models are successfully able to figure out the answer based on the question and keyword overlap. GPT4o-Mini performs the best achieving 96% accuracy, and Qwen2.5-7B-Medical has a similar performance of 94%; the majority of the models fall in the 80–90% accuracy range, and finally, Meditron-7B has performance comparable to that of RAG and experiments with context included, presented in Table 8. Given a non-context baseline, medical LLMs perform on average better ($0.904\pm 0.031$, without the anomalous result of Meditron-7B) than non-medical LLMs ($0.889\pm 0.045$).

Table 7. CliniQG4QA with no context: providing the model the question and options only.

Model	Accuracy	F1	Exact Match
Meditron-7B	0.2626	0.0833	0.2626
Qwen2.5-7B-Medical	0.9425	0.9416	0.9425
Medgemma-4B	0.9021	0.9014	0.9021
Medical-Coding-LLM(Phi-3-mini-4k)	0.8664	0.8653	0.8664
Llama-3-8B	0.8640	0.8651	0.8640
Mistral-7B	0.8827	0.8848	0.8827
DeepSeek-7B	0.8454	0.8454	0.8454
GPT4o-Mini	0.9650	0.9648	0.9650

Table 8. Accuracy and F1 scores across datasets, retrieval variants, and medical models.

Model	Variant	Accuracy	Exact Match	MRR	F1
Medgemma-4B	LLM-only	0.2743	0.2743	0.2641	0.1352
	bm25	0.2517	0.2517	0.2517	0.1006
	dataset-context	0.2347	0.2347	0.2347	0.0950
	dense-gtr	0.2486	0.2486	0.2486	0.0996
	dense-sbiobert	0.2378	0.2378	0.2378	0.0960
	hybrid-gtr	0.2455	0.2455	0.2455	0.0986
	hybrid-sbiobert	0.2517	0.2517	0.2517	0.1006
	Mean	0.2492 $\pm 0.0129$	0.2492 $\pm 0.0129$	0.2477 $\pm 0.0098$	0.1037 $\pm 0.0141$
Meditron-7B	LLM-only	0.2440	0.2440	0.2440	0.0981
	bm25	0.2541	0.2541	0.2541	0.1013
	dataset-context	0.2354	0.2354	0.2354	0.0953
	dense-gtr	0.2782	0.2782	0.2782	0.1088
	dense-sbiobert	0.2580	0.2580	0.2580	0.1025
	hybrid-gtr	0.2688	0.2688	0.2688	0.1059
	hybrid-sbiobert	0.2618	0.2618	0.2618	0.1038
	Mean	0.2572 $\pm 0.0145$	0.2572 $\pm 0.0145$	0.2572 $\pm 0.0145$	0.1022 $\pm 0.0046$
Qwen2.5-7B-Medical	LLM-only	0.2587	0.2587	0.2440	0.1328
	bm25	0.4141	0.4141	0.4466	0.3701
	dataset-context	0.2300	0.2300	0.2300	0.0935
	dense-gtr	0.2502	0.2502	0.2502	0.1001
	dense-sbiobert	0.2393	0.2393	0.2393	0.0966
	hybrid-gtr	0.2448	0.2448	0.2448	0.0983
	hybrid-sbiobert	0.2510	0.2510	0.2510	0.1003
	Mean	0.2697 $\pm 0.0642$	0.2697 $\pm 0.0643$	0.2723 $\pm 0.0772$	0.1417 $\pm 0.1016$
Medical-Coding-LLM(Phi-3-mini-4k)	LLM-only	0.2712	0.2712	0.2649	0.1247
	bm25	0.2379	0.2378	0.2378	0.0960
	dataset-context	0.2370	0.2370	0.2378	0.0958
	dense-gtr	0.2549	0.2549	0.2549	0.1015
	dense-sbiobert	0.2331	0.2331	0.2331	0.0945
	hybrid-gtr	0.2432	0.2432	0.2432	0.0978
	hybrid-sbiobert	0.2541	0.2541	0.2541	0.1013
	Mean	0.2473 $\pm 0.0135$	0.2473 $\pm 0.0135$	0.2490 $\pm 0.0113$	0.1017 $\pm 0.0105$
Overall Mean		0.2559 $\pm 0.0335$	0.2559 $\pm 0.0335$	0.2565 $\pm 0.0390$	0.1123 $\pm 0.0516$

As discussed previously, there are many reasons some models perform better than others. Qwen2.5 Medical likely outperforms the other models, for three main reasons. Firstly, Qwen2.5-7B-Medical is built on the Qwen2.5 family which uses a larger transformer depth, optimised for attention patterns. We speculate this aids the model in better encoding medical terminology, which is needed for discerning subtle context clues that BM25 surfaces [76]. Secondly, the addition of literature-grounded medical-pretraining makes the model domain-aware and able to provide more accurate answers [76]. Thirdly, we speculate that Qwen may have a stronger synergy with retrieval than other models, allowing it to interpret sparse keyword matches better than other models; this again can be seen as an architectural benefit [77].

GPT-4o, on the other hand, has been pretrained on a large multi-domain corpus, presumably far larger than most open-source medical models, totalling trillions of tokens, compared to billons from Open-Source models [78]. Additionally, GPT-4 and its derivative models have been instruction-turned with human feedback to a greater extent than open models. In addition to post-training, these models support very large context windows of up to 128,000 [79], which is far beyond that of open-source models, which range from 4–16k tokens. This allows the model to digest large amounts of text and discern noisy inputs without losing focus on the task, unlike open-source models that often truncate or chunk inputs, making them more vulnerable to missing information.

Table 8 presents the results of distraction experiments on medical LLMs, and Table 9 presents the same task on general-purpose LLMs.

Table 9. Accuracy and F1 scores across datasets, retrieval variants, and non-medical models.

Model	Variant	Accuracy	Exact Match	MRR	F1
Llama-3-8B	LLM-only	0.2681	0.2681	0.2595	0.1219
	bm25	0.2517	0.2517	0.2517	0.1006
	dataset-context	0.2525	0.2525	0.2525	0.1008
	dense-gtr	0.2953	0.2953	0.4726	0.1950
	dense-sbiobert	0.2432	0.2432	0.2432	0.0978
	hybrid-gtr	0.3652	0.3652	0.4672	0.3229
	hybrid-sbiobert	0.2852	0.2852	0.4830	0.1902
	Mean	0.2802 $\pm 0.0420$	0.2802 $\pm 0.0420$	0.3471 $\pm 0.1191$	0.1613 $\pm 0.0827$
Mistral-7B	LLM-only	0.2828	0.2828	0.2757	0.1327
	bm25	0.2642	0.2642	0.2642	0.1045
	dataset-context	0.2455	0.2455	0.2455	0.0986
	dense-gtr	0.2688	0.2688	0.2688	0.1059
	dense-sbiobert	0.2486	0.2486	0.2486	0.0966
	hybrid-gtr	0.2347	0.2347	0.2347	0.0965
	hybrid-sbiobert	0.2385	0.2385	0.2385	0.0963
	Mean	0.2547 $\pm 0.0176$	0.2547 $\pm 0.0176$	0.2537 $\pm 0.0159$	0.1044 $\pm 0.0131$
DeepSeek-7B	LLM-only	0.2549	0.2549	0.2486	0.1198
	bm25	0.2580	0.2580	0.2580	0.1025
	dataset-context	0.2603	0.2603	0.2603	0.1033
	dense-gtr	0.2517	0.2517	0.2517	0.1006
	dense-sbiobert	0.2533	0.2533	0.2533	0.1011
	hybrid-gtr	0.2587	0.2587	0.2587	0.1028
	hybrid-sbiobert	0.2401	0.2401	0.2401	0.0968
	Mean	0.2539 $\pm 0.0068$	0.2539 $\pm 0.0068$	0.2538 $\pm 0.0068$	0.1038 $\pm 0.0074$
GPT4o-Mini	LLM-only	0.9837	0.9837	0.4130	0.9838
	bm25	0.9821	0.9821	0.4707	0.9822
	dataset-context	0.9821	0.9821	0.4604	0.9822
	dense-gtr	0.9838	0.9838	0.5359	0.9836
	dense-sbiobert	0.9858	0.9858	0.5979	0.9858
	hybrid-gtr	0.9617	0.9617	0.4995	0.9614
	hybrid-sbiobert	0.9639	0.9639	0.4456	0.9644
	Mean	0.9776 $\pm 0.0102$	0.9776 $\pm 0.0102$	0.4890 $\pm 0.0619$	0.9776 $\pm 0.0102$
Overall Mean		0.4416 $\pm 0.3161$	0.4416 $\pm 0.3161$	0.3359 $\pm 0.1170$	0.3368 $\pm 0.3796$
Overall (excluding GPT)		0.2629 $\pm 0.3232$	0.2629 $\pm 0.3161$	0.2849 $\pm 0.0799$	0.1232 $\pm 0.0537$

When compared to the performance of LLMs with no context (Table 7), the models performed significantly worse with RAG, suggesting the models struggled to ignore the irrelevant information from the context and from the retrieved context. Notably, Qwen2.5-7B-Medical with a BM25 retriever surpasses the other Med LLMs by double, achieving $41.41\%$ accuracy.

Table 8 contains the results for the medical LLMs. The average performance of medical LLMs in terms of accuracy is $0.2559\pm 0.0335$, compared to $0.2629\pm 0.3232$ for open-source non-medical LLMs. This confirms, through decreased performance, that medical LLMs tended to get slightly more easily distracted than non medical LLMs [57].

The listed non-medical sum excludes GPT4o-Mini on purpose, and for a reason. It is treated as an anomaly since it is a stronger closed-source model. GPT achieves an accuracy of $\approx$98% with RAG context; this is a $2\%$ improvement from its non-context result, suggesting that the additional context may have kept the model more grounded on the question being asked.

4.2.1. Dataset Context Benchmark

MIMIC-III and PubmedQA are accompanied by structured “context” passages, drawn from admission notes and PubMed abstracts. These include a background to summarise the study or hospital admission, often providing clues towards the correct answer.

Results show that these contexts are able, in some cases, to improve calibration compared to LLM-only, achieving higher F1 and more stable accuracies; they also achieved lower latencies, since no concatenation of multiple passages was required. Across models, using dataset context yielded higher average F1 scores of $0.3751\pm 0.114$ across all models than LLM-only ($0.2318\pm 0.055$) and RAG with external knowledge bases ($0.249\pm 0.062$).

In other cases like the distraction experiments (see Section 4.2), the dataset context showed minimal improvement in model performance due to the need to process an increased number of tokens.

4.2.2. Latency–Length Correlation

Table 10 presents the latencies for the distraction experiments; following a similar trend to the other datasets, DeepSeek-7B had the lowest latency, followed closely by Mistral-7B. Medgemma-4B achieved similar latencies to GPT4o-Mini, although they were not directly comparable due to the nature of calling proprietary models through the OpenAI python library.

Table 10. Latencies across the CliniQG4QA dataset.

Variant	Llama3-8B	Mistral-7B	DeepSeek-7B	Medgemma-4B	Meditron-7B	Medical-Coding-LLM(Phi-3-mini-4k)	Qwen2.5-7B-Medical	GPT4o-Mini
LLM-only	1.7889	1.7410	1.6180	3.8846	1.7351	2.0579	1.6693	1.3389
Dataset context	2.2261	2.1465	2.0404	4.6182	2.1344	2.5789	2.0878	5.2483
BM25	2.1900	2.0863	2.0289	4.4190	2.1240	2.4911	4.1315	5.7130
Dense-GTR	4.5033	1.9701	1.6873	4.3657	1.8196	2.3336	1.7121	4.1176
Dense-SBioBERT	1.9497	2.1194	1.7944	4.8142	1.8485	2.2782	1.6986	4.5714
Hybrid-GTR	6.1605	2.1403	1.9805	4.4669	2.0976	2.5485	2.0393	5.7061
Hybrid-SBioBERT	4.5770	2.2470	1.9877	5.3221	2.1192	2.7439	2.0999	5.6184
No context	1.7889	0.9342	0.8156	3.6982	1.6719	2.2896	1.5866	1.3487
Total	3.1481 $\pm 1.6840$	1.9244 $\pm 0.4285$	1.7441 $\pm 0.4085$	4.4486 $\pm 0.5092$	1.9461 $\pm 0.1919$	2.4152 $\pm 0.2162$	2.1281 $\pm 0.8354$	4.2078 $\pm 1.8566$

Figure 2 shows the scatter plots and regression lines (code available at: https://gitlab.com/MeRichard123/frugality-data-vis [accessed on 10 January 2026]) for the three models used for RAG. We plot the correlation between mean prefill word counts (a tokeniser-agnostic correlate for input length) against the mean latency across dataset and RAG variant pairs. The results show clear positive relationships for Llama-3-8B (r = 0.67, p = 0.0017), Mistral-7B (r = 0.60, p = 0.0071), and DeepSeek-7B (r = 0.57, p = 0.0118). All three correlations indicate statistical significance under conventional thresholds. A similar trend is seen for MIMIC-III shown in Figure 3, plotted separately for ease of visibility of clustered points.

Figure 2. Scatterplot of latency against input length (measured as average input word count, used as a tokeniser-agnostic indicator of prompt size) with regression lines (per model) for the smaller datasets. Plots Generated in R Studio (Version 2025.09.2+418).

Figure 3. Scatterplot of latency against input length (measured as average input word count, used as a tokeniser-agnostic indicator of prompt size) with regression lines (per model) for the larger MIMIC-III dataset. Plots Generated in R Studio (Version 2025.09.2+418).

We empirically observe a positive relation between input word count and latency. Prior work suggests two main mechanisms underlying this trend. First, the computational complexity of self-attention grows with sequence length, with a theoretical complexity of $O\left(n^2\right)$, since attention matrix scales quadratically with the number of tokens [80,81]. While this theoretical bound is quadratic, our analysis only fits a linear regression model to illustrate the relationship between the sequence length and latency; we do not claim to empirically estimate or validate quadratic scaling. Secondly, model KV-caches grow for each token, expanding linearly. Loading these pages on the GPU dominates the memory access path, causing larger memory-bandwidth consumption and latency [82,83].

4.2.3. RAG vs. LLM-Only Experiments

The differences between LLM-only runs and those that used RAG show that RAG has its benefits and drawbacks. For MedMCQA, the LLM-only run performed marginally better than using retrieval in accuracy ($0.50\pm 0.12$ versus $0.48\pm 0.10$, on average), with F1 remaining almost the same ($0.423\pm 0.121$ versus $0.423\pm 0.132$, also on average). LLM-only runs also demonstrated quicker processing times than its RAG-based counterpart due to shorter contexts. A similar trend occurs with MedQA-USMLE, where the non-retrieval method resulted in improved accuracy ($0.44\pm 0.12$ versus $0.38\pm 0.12$).

RAG was, however, advantageous in some cases; for the PubMedQA dataset, RAG was useful and resulted in slightly higher accuracy than the LLM-only alternative; $0.41\pm 0.11$ in LLM-only and $0.43\pm 0.14$ with RAG. Although these differences are modest, they provide some utility in the use of RAG.

4.2.4. Effectiveness of Dataset Context

The uses of original dataset context improved performance on the PubMedQA; these question-specific contexts allowed the model to ground itself closer to the answer. In the cases where dataset context was available, the performance was significantly improved, with Llama-3-8B achieving the highest performance improvement with a $72.5\%$ accuracy.

5. Frugality

5.1. Proposed Metric

Frugality measures whether the utility gained from improved accuracy outweighs the computational cost introduced by retrieval. As such, frugality provides an interpretable decision framework for evaluating retrieval under constrained deployment settings. In order to assess the effectiveness of RAG, we propose a Frugality Score, defined as

$$\begin{array}{cc}\hfill \mathrm{FrugalityScore}& =tanh(U_{acc}-U_{cost})\hfill \end{array}$$

(6)

where

$$\begin{array}{cc}\hfill U_{acc}& ={\displaystyle \frac{exp((AC{C}_{rag}-AC{C}_{\mathrm{LLM}-\mathrm{only}})/t_{acc})-1}{exp((AC{C}_{rag}-AC{C}_{\mathrm{LLM}-\mathrm{only}})/t_{acc})+1}}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill U_{cost}& ={\displaystyle \frac{exp((COS{T}_{rag}-COS{T}_{\mathrm{LLM}-\mathrm{only}})/t_{cost})-1}{exp((COS{T}_{rag}-COS{T}_{\mathrm{LLM}-\mathrm{only}})/t_{cost})+1}}\hfill \end{array}$$

(8)

In the Equations above, the temperature parameters $t_{acc}$ and $t_{cost}$ correspond to the standard deviations of the accuracy differences ($AC{C}_{rag}-AC{C}_{\mathrm{LLM}-\mathrm{only}}$) and cost differences ($COS{T}_{rag}-COS{T}_{\mathrm{LLM}-\mathrm{only}}$), respectively.

Frugality is bound to $[-1,+1]$ where positive values indicate RAG becoming more frugal and negative values suggest that the cost (latency, defined as the end-to-end time, shown mathematically in Section 3.5 outweighs the accuracy benefit. The frugality is computed using two utilities. First, the normalised accuracy utility ($U_{acc}$) indicates how much RAG affected the accuracy. Second, the normalised cost penalty ($U_{cost}$) indicates how expensive RAG becomes.

Whilst the well-known hyperbolic tangent function $tanh$ [84] forms the foundation of our metric by specifying values in the range $[-1,+1]$, exponential transformations are required for two reasons: (a) to handle unbounded values, as is the case for cost values measured in seconds, and (b) to scale performance differences more effectively than raw values. Without exponential values and temperature parameters, the frugality scores would become clustered around zero.

The use of the tanh function provides greater variance ($\sigma =0.1312$) than the $sigmoid$ ($\sigma =0.0034$). Figure A11 visualises this difference in the form of a box plot, which indicates that the spread of values using the tangent function is much broader. As a result, the differences between systems which use $sigmoid$ appear much more compressed and harder to compare.

Moreover, two different temperature parameters are required because accuracy and cost are expressed in different units. Furthermore, the temperature parameters are automatically set to the standard deviations of the accuracy and cost differences—this mitigates the need for manually specified values, although such values can still be provided if required in different scenarios. Finally, while we use accuracy in terms of exact match and cost in terms of seconds of inference per prompt, other metrics can be employed, making our proposed $FrugalityScore$ highly flexible for its application to other scenarios.

While Pareto-based evaluation has seen many applications in literature—for example, Behera and Poosapati [85] and Dallaire [86] describe the Pareto front of RAG systems with regard to accuracy–latency trade-offs, while Barker et al. [87] computes Pareto optimal parameters for the hyperparameter tuning of LLM and RAG systems—these approaches lack a scalar decision criterion for assessing whether the added costs of retrieval are justified by the accuracy gains. Our Frugality score complements Pareto analysis by applying it into a bounded utility function, and allowing for direct comparisons and systems-oriented decision making. Additionally, allowing for tuning the model via temperature parameters could open the model to alternative approaches such as learning the parameters from data.

Figure 4 visualises the application of tanh to the FrugalityScore, showing that the score (Y-axis) is bounded to $[-1,+1]$, which makes it robust to outliers by preventing extreme gains or penalties from dominating the metric; the X-axis shows the unnormalised score.

Figure 4. Mapping from net utility margin $\Delta$U to Frugality Score via tanh. Points indicate empirical results across datasets and models, illustrating the saturation for extreme trade-off. Figure generated in R Studio (Version 2025.09.2+418).

5.2. Sensitivity

Figure 5 examines how the frugality score changes when varying the temperature parameters, which control the scaling of the accuracy gains and cost increases in the utility function. The plot demonstrates that the frugality score, or relative ordering of methods within the contour space, does not change significantly under moderate perturbations of these parameters. This observation suggests that increasing or decreasing the cost systematically penalises RAG-based systems while keeping the overall ranking of models stable across a wide range of values.

Figure 5. Sensitivity analysis of the Frugality metric to the temperature parameters. Contours represent iso-levels of the Frugality score over the utility space of accuracy gain ($\Delta ACC$) and cost ($\Delta Cost$). The baseline represents initial parameters (standard deviations of the difference between cost and accuracy). Each panel corresponds to a different scaling of temperature, illustrating how the geometry of the utility mapping deforms under temperature variations. Points denote observed values from each dataset.

To confirm the values are not the same across perturbations, the values plotted are reported in Appendix A Figure A4 and Figure A5. The contour exhibits a tan-like shape, which implies diminishing returns for both Cost and Accuracy. Even though $\Delta COST$ is penalised to a greater extent, the changes eventually saturate, which prevents extreme trade-offs from dominating the metric. This ensures stability across scales.

Overall, these results demonstrate that conclusions drawn from Frugality are robust to differing assumptions about the weightings of accuracy and cost.

6. Concluding Remarks

This work demonstrates that retrieval-augmented generation is neither universally helpful nor universally harmful in medical question answering. Its usefulness depends on dataset uncertainty, evidence alignment, and computational constraints, which are jointly captured by the proposed frugality framework.

Looking closely at Figure 6, model-specific trends become apparent; Llama-3-8B appears to gain the biggest advantage from retrieval, given that more points appear above the trade-off plane. Both Llama-3-8B and Mistral-7B are able to take more advantage of RAG than DeepSeek-7B; this may be because chat-tuned models like DeepSeek-7B prioritise conversational coherence over evidence use.

Figure 6. Frugality trade-off plane across all datasets between normalised accuracy gain ($U_{acc}$) and cost penalty ($U_{cost}$). Points above the diagonal $U_{acc}=U_{cost}$ correspond to a positive frugality score, indicating that the accuracy improvement outweighs the cost. The colours indicate the final score after binding the range using tanh. Figure created in R Studio.

Section 4.2.2 explains the link between length and latency, where all models show increased latency with longer prompts. DeepSeek-7B scales better for larger prompts, since Llama-3-8B and Mistral-7B incur higher latency as context grows; however, Llama-3-8B and Mistral-7B are able to convert the extra latency cost into meaningful accuracy gains. Llama-3-8B and Mistral-7B are, as a result, more frugal than DeepSeek-7B.

Additionally, frugality is not an intrinsic property of RAG, but rather dataset-dependent. PubMedQA presents with the largest accuracy gain from RAG compared to the other models, with the largest number of points (see Appendix A Figure A2) in the positive region indicating that RAG is frugal. This may be the case because PubMedQA is grounded by medical literature from PubMed, and therefore can make better use of retrieved context [19]. Conversely, CliniQG4QA (see Appendix A Figure A3), MedQA-USMLE (see Appendix A Figure A1), and MedMCQA (see Appendix A Figure A4) are not literature-grounded and therefore RAG is either neutral in the case CliniQG4QA, or harmful in the case of MedQA. In terms of distraction, Appendix A Figure A5 shows that, in cases where the model was distracted, RAG was also not frugal and even harmful. RAG was only frugal when the uncertainty of the dataset could not be discerned from parametric knowledge.

Figure 6 also labels different retrieval methods for each point. High latency retrieval with marginal accuracy gains is consistently mapped to negative frugalities: hybrid methods tend to add cost without any utility, and BM25 is mostly harmful, while Dense SBioBERT achieves the best balance in accuracy and cost.

Our results indicate that RAG is neither universally helpful nor harmful in medical QA. Under the Frugality framework, RAG is helpful for evidence-grounded data such as PubMedQA, where accuracy gains consistently outweigh the costs, but become harmful or neutral for knowledge-heavy benchmarks such as MedMCQA or MedQA-USMLE and further distractive for large-context datasets such as CliniQG4QA.

6.1. Frugality Guidelines for RAG

Use RAG when:

1.

Questions are literature-dependent.

2.

Parametric knowledge is insufficient.

3.

Latency budget is flexible, for low latency situations simpler models, and non-hybrid retrieval is recommended.

Avoid RAG when:

1.

Benchmarks resemble closed-book exam questions.

2.

Prompts already contain enough signal.

3.

Real-time constraints matter.

Frugality Score as a Selection Criterion

Optimising accuracy in isolation systematically favours high cost pipelines, which may be impractical in a clinical setting. Our proposed $FrugalityScore$ metric is arguably useful for selecting a model-retriever-dataset combination, which provides additional insights beyond accuracy metrics alone.

6.2. Significance of Our Work

The novelty of this paper is fourfold:

1.

We use a set of eight LLMs in this study—four general-purpose and four medical—which have not been used in a focused RAG-based evaluation before.

2.

The set of datasets in our study, including MedMCQA, MedQA-USMLE, PubMedQA, and CliniQG4QA, have not previously been used together in this specific frugality-focused evaluation.

3.

The proposed $FrugalityScore$ metric is novel and can benefit future evaluations—even beyond question answering and more generally for other (data-driven) AI systems.

4.

Our work creates a robust benchmark across multiple datasets, evaluated across two dimensions, addressing gaps in RAG research mentioned previously.

6.3. Limitations and Future Work

This study is limited in its exhaustiveness, and should continue to experiment on the full range of datasets across all models to confirm a wider picture of recommendation. Additionally, latency computations are limited to the hardware used in the experiment, and so a more general set of results across a range of hardware may provide more accurate averages. The use of latency does not give an overall picture of which parts of the pipeline are more costly, therefore we also suggest varying the latency within the metric, so as to decompose latency into its constituent parts, attributing Frugality to RAG cost only for comparison. Hyperparameters were purposely restricted (fixed chunk size, top-k = 5, greedy decoding) to isolate the effects of RAG, and further accuracy gains could be attained by the intricate tuning of parameters, or full-sentence context retrievals.

Future work could also focus on employing additional fine-tuning strategies to the models, tuning the frugality metric using machine learning approaches to assess learnt temperature parameters, and evaluating a wider range of RAG techniques and different types of questions. The latter should be evaluated not only on text data but also on multimodal data. Whilst evaluations using data generated by domain experts are useful to advance AI models, real-world data and evaluations will be needed to provide more compelling evidence on the actual levels of readiness of medical question answering systems.

Finally, research dedicated to the theoretical and mathematical background is needed to fully explore how RAG aids models in downstream tasks.