Designing an Architecture of a Multi-Agentic AI-Powered Virtual Assistant Using LLMs and RAG for a Medical Clinic

Abstract

This paper presents the design, implementation and evaluation of an agentic virtual assistant (VA) for a medical clinic, combining large language models (LLMs) with retrieval-augmented generation (RAG) technology and multi-agent artificial intelligence (AI) frameworks to enhance reliability, clinical accuracy and explainability. The assistant has multiple functionalities and is built around an orchestrator architecture in which a central agent dynamically routes user queries to specialized tools for retrieval-augmented question answering (Q&A), document interpretation and appointment scheduling. The implementation combines LangChain and LangGraph with interactive visualizations to track reasoning steps, prompts using Gemini 2.5 Flash defines tool usage and strict formatting rules, maintaining reliability and mitigating hallucinations. Prompt engineering has an important role in the implementation and thus, it is designed to assist the patient in the human–computer interaction. Evaluation through qualitative and quantitative metrics, including ROUGE, BLEU, LLM-as-a-judge and sentiment analysis, confirmed that the multi-agent architecture enhances interpretability, accuracy and context-aware performance. Evaluation shows that the multi-agent architecture improves reliability, interpretability and alignment with medical requirements, supporting diverse clinical tasks. Furthermore, the evaluation shows that Gemini 2.5 Flash combined with clinic-specific RAG significantly improves response quality, grounding and coherence compared with earlier models. SBERT analyses confirm strong semantic alignment across configurations, while LLM-as-a-judge scores highlight the superior relevance and completeness of the 2.5 RAG setup. Although some limitations remain, the updated system provides a more reliable and context-aware solution for clinical question answering.

1. Introduction

The integration of conversational AI into healthcare systems has expanded rapidly in recent years, driven by the need for efficient, accessible and accurate communication between patients and healthcare providers. Virtual Assistants (VAs) and chatbots are deployed to handle administrative tasks, provide medical information and support patient engagement beyond traditional clinical settings [1]. While early voice-based assistants, such as Siri and Alexa, demonstrated limited clinical accuracy, recent advancements in LLMs have enabled the development of highly capable conversational agents that can interpret complex queries, deliver relevant information and assist in healthcare workflows with improved precision and contextual awareness [2].

AI is increasingly transforming healthcare across multiple domains, including medical imaging, diagnostics, virtual patient care, electronic health record management, drug discovery, rehabilitation and administrative workflows, demonstrating its capacity to improve patient engagement. For instance, AI applications have been instrumental in early diagnosis and outbreak control during the COVID-19 pandemic, highlighting their potential in crisis response [3]. In medical education, particularly in radiotherapy, AI-driven chatbots are emerging as specialized tools. A proposed framework emphasizes the importance of privacy and ethical compliance in developing chatbots. Challenges identified include content accuracy, bias and system integration, underscoring the need for advancements in natural language processing (NLP), personalized learning and immersive technologies [4].

Hybrid technical approaches that combine RAG technology with domain-specific knowledge are increasingly being explored to enhance both reliability and clinical utility. In radiology, Ref. [5] demonstrated that a GPT-4-based RAG chatbot, grounded in 96 curated gastrointestinal imaging documents, correctly identified the primary differential diagnosis in 78% of cases and included the true diagnosis among the top three suggestions in 90% of cases, compared to 54% and 74%, respectively, for a generic GPT-4 model, highlighting substantial gains in diagnostic accuracy and clinician trust. Similarly, Ref. [6] evaluated a context-aware RAG chatbot for medication-related osteonecrosis of the jaw, reporting that retrieval augmentation improved answer relevance and scientific correctness in over 85% of queries, while also providing explicit source excerpts to support transparency and trustworthiness. Systematic analyses further confirm that RAG-based approaches consistently outperform non-augmented LLMs, with Ref. [7] reporting median accuracy gains of 9–10% across a wide range of medical question-answering tasks. These studies illustrate that integrating retrieval mechanisms, domain-specific knowledge and graph-based reasoning substantially enhances the accuracy, transparency and clinical applicability of AI-driven medical assistants.

Multi-agent systems (MAS) consist of multiple autonomous entities, known as agents, that interact within a shared environment to achieve individual or collective objectives. These agents are capable of reasoning and taking actions independently or collaboratively, which allows them to handle complex, dynamic tasks more efficiently than single-agent systems. MAS are particularly well-suited for environments that require task decomposition, decentralized decision-making and coordination among specialized components. By assigning distinct roles and responsibilities to each agent, MAS can optimize problem-solving, enhance scalability and improve robustness in a wide range of applications [8]. In healthcare, MAS are usually applied to address the complexity of patient care and clinical workflows. For instance, Ref. [8] describes MAS frameworks in which agents are assigned roles analogous to clinical staff, such as data collection, diagnostic reasoning, treatment recommendation and resource coordination, enabling layered oversight and multi-step decision-making. Such systems have demonstrated the potential to improve diagnostic accuracy, optimize treatment plans and enhance patient monitoring while reducing cognitive load on clinicians.

In this paper, our goal is to design and evaluate an agentic VA for a medical clinic, capable of handling multiple patient-facing tasks through advanced LLM capabilities. The system’s ‘agentic’ nature is defined by its ability to dynamically select and orchestrate appropriate tools based on user intent. Rather than following rigid decision trees, the central agent analyzes user queries in real time and autonomously routes them to the most suitable tool. The system integrates a RAG pipeline to ensure that answers to patient questions are generated only from verified clinic data scraped from its official website, guaranteeing factual accuracy and domain relevance. A built-in evaluation module applies response quality metrics, including factual alignment, completeness and clarity, before any output is returned to the user, mitigating the risk of hallucinations in high-stakes medical contexts.

Beyond conversational Q&A, the assistant is equipped to interpret laboratory results by extracting and reasoning over structured and unstructured data from attached PDF reports. Additionally, it offers NL-based appointment management, enabling patients to schedule, reschedule or cancel visits using conversational instructions.

The contributions of our paper consist of:

• Domain-constrained RAG-based Q&A. We implement a retrieval pipeline that limits the assistant’s knowledge base exclusively to curated medical clinic data, with a response evaluation layer to ensure reliability.
• Lab result interpretation from PDFs. We integrate document parsing and LLM reasoning to analyze patient laboratory reports, extract relevant metrics and provide patient-friendly explanations.
• NL-based appointment management. We develop a dialogue-driven scheduling interface enabling end-to-end appointment handling without manual intervention.
• Agentic orchestration. We propose an agentic framework to dynamically select and chain specialized tools (retrieval, PDF parsing, scheduling) based on the user’s intent.
• Performance evaluation. Benchmarking of the system’s capabilities is provided using metrics for accuracy in simulated patient interaction scenarios.

This research advances the field of medical conversational AI by demonstrating the viability of agentic VAs that combine retrieval, reasoning and action-taking capabilities within a single LLM-powered architecture. The paper is structured into five sections. The first section, Introduction, outlines the motivation and problem definition, emphasizing the growing need for accurate, always-available patient support tools. The second section, Literature review, examines prior work in medical chatbots, RAG systems, PDF understanding and conversational appointment scheduling. The Methodology section details the agentic architecture, including diagrams, tool orchestration, retrieval design, PDF processing pipeline and evaluation framework. The Results section presents empirical findings, demonstrating improvements in factual accuracy, reduced response latency and high user satisfaction compared to baseline LLM implementations. Finally, the Conclusions section summarizes the contributions and suggests avenues for future work, including multimodal input expansion and integration with electronic health record systems.

2. Literature Review

2.1. Previous Works

The existing studies on conversational AI highlight how LLMs have begun to enhance patient–clinician communication and improve accessibility to medical information. RAG has emerged as a powerful method to enhance the capabilities of LLMs by combining generative abilities with external knowledge retrieval. RAG allows a model to query structured or unstructured data sources during the generation process, thereby grounding its outputs in factual information rather than relying solely on pre-trained weights of the LLM. This approach improves accuracy, mitigates hallucinations and facilitates handling of specialized or evolving knowledge domains, making it particularly valuable for contexts where correctness and domain specificity are important. Recent research highlights that RAG systems consistently outperform baseline generative models across diverse NLP tasks, from open-domain Q&A to technical knowledge extraction, as they can dynamically incorporate relevant documents, studies or knowledge graphs during response generation [9].

In the medical domain, the advantages of RAG are essential for chatbot applications, where reliability, scientific correctness and context-awareness are paramount. Further, Ref. [10] underscores several advantages of RAG in high-stakes medical domains: (1) when retrieval provides useful and relevant information, accuracy rates reach 97–98%, showing that RAG applications give highly precise and trustworthy answers, performing much better than models that rely only on their built-in knowledge when questions are asked in a consistent way; (2) RAG’s reliance on externally retrieved documents means that even as adversarial evidence enters the retrieval pool, the presence of correct evidence helps preserve fidelity of responses, a kind of redundancy that generative-only models lack; (3) prompt design in retrieval significantly impacts system reliability. This implies that RAG systems give more tools for improving safety and correctness, via curation of retrieval sources, bias-aware ranking and prompt design, than systems that generate purely from internal model knowledge. Although this approach increased response time and cost, the gains in accuracy and reliability highlight the clear advantages of RAG in clinical decision support [5]. Similarly, in a comparative study by researchers developing a Patient Information Assistant (PIA) for pre-CT informed consent, the RAG-based chatbot significantly reduced physician consultation time without sacrificing clarity or comprehension [11,12]. These outcomes highlight RAG’s value in grounding responses in domain-relevant documents while conserving clinician resources. Additionally, these results indicate that incorporating retrieval mechanisms into medical VAs enables more precise, trustworthy and context-aware interactions, positioning RAG as a foundational approach for next-generation healthcare chatbots.

Traditionally, performance in medical Q&A tasks has been assessed using lexical overlap metrics such as ROUGE, BLEU and Exact Match. While these metrics provide a straightforward way to compare generated answers with a ground truth, they have notable limitations:

(1)

They emphasize surface-level lexical overlap (n-gram matching) and thus fail to recognize answers that are clinically correct but use different wording, synonyms or paraphrases. In Ref. [13], it is shown that BLEU and ROUGE often fail to penalize critical errors, e.g., misclassifying “benign” vs. “malignant”, even when lexical overlap remains high.

(2)

They lack semantic nuance: they miss subtle meaning and often fail to account for the quality of reasoning or how well the answer is supported by evidence, uncertainty expression, partial correctness or adherence to clinical guidelines. Moreover, Ref. [14] highlights that LLMs which score well under BLEU/accuracy often underperform when examined for more demanding facets of medical knowledge (comparison, verification, discrimination). A review of evaluation methods in the medical LLM literature finds that few works reliably include physician ratings or guideline concordance, and those that do report that BLEU/ROUGE often fail to capture what experts consider most important (safety, clarity, appropriateness) [15].

To address these gaps, researchers are increasingly adopting LLM-as-a-judge frameworks, in which a more powerful LLM is used to evaluate chatbot outputs along multiple dimensions (accuracy, coherence, faithfulness to sources, conciseness, etc.). In this context, Ref. [16] demonstrated that LLM-as-a-judge aligns more closely with expert evaluation in extractive Q&A compared to ROUGE or BLEU, providing a richer assessment of correctness and reasoning quality. Likewise, Ref. [17] introduced LLMEval-Med, a benchmark that incorporates physician-guided evaluation criteria and checklist-based scoring, showing that LLM-as-a-judge can reliably capture subtle differences in factual correctness, clinical safety and relevance that traditional metrics overlook.

Multimodal LLMs integrate various data types, such as text, images and structured data, to provide a more comprehensive understanding of medical information. This integration enables VA to perform complex tasks like medical report generation, visual Q&A and patient triage. The Lingshu model [18], introduced in mid-2025, represents a significant advancement in multimodal medical understanding. Trained on a vast dataset encompassing medical images, text and general-domain data, it addresses several limitations of previous models, such as limited medical knowledge coverage and susceptibility to hallucinations. Its performance across tasks like multimodal Q&A, text-based Q&A and medical report generation surpasses prior open-source baselines, demonstrating its potential in enhancing clinical decision-making processes. These advancements suggest that combining image, chart and report data with text permits richer, more context-aware interaction and interpretation in medical VA applications.

The movement from single LLMs toward more agentic AI systems is accelerating in medicine, enabling more modular, autonomous and context-aware tools. Thus, Ref. [19] introduced a hierarchical framework in which agents were assigned roles similar to clinical staff (nurse, physician, specialist), enabling decisions to be checked and improved at different stages of the process. By distributing responsibilities across tiers, the system achieved more than 3.2% improvement in safety scores compared to flat, single-tier agentic models, and it outperformed other multi-agent baselines by as much as 8.2% in complex healthcare safety benchmarks, highlighting the benefit of structured oversight. Also, Ref. [20] compared agent-based systems across 20 studies and it found that all agent systems outperformed baseline LLMs in clinical task accuracy, often by large margins; median improvements of around 53 percentage points were reported for single-agent tool-calling systems. Similarly, Ref. [21] analyzed 18 studies and reported robust gains not only in diagnostic accuracy and treatment planning but also in real-time patient monitoring and interdepartmental coordination, emphasizing that multi-agent approaches extend beyond individual task accuracy to broader workflow optimization in clinical environments.

2.2. Research Gap and Our Contribution

The research shows a rapid evolution from text-only, hallucination-prone chatbots toward robust, RAG-enhanced systems that prioritize accuracy, explainability and trustworthiness [6,7,18]. At the same time, multimodal approaches and privacy-aware architectures are expanding the boundaries of what healthcare chatbots can achieve, pointing toward a future of clinically reliable, context-sensitive and patient-centered conversational AI [18,22,23].

Research on RAG demonstrates its value in ensuring that chatbot responses remain accurate and grounded in verified medical knowledge, significantly improving factual alignment and interpretability across clinical tasks [24]. In parallel, recent work on multimodal LLMs shows promising progress in interpreting unstructured clinical data such as radiology reports and laboratory results, enabling richer and safer decision support [25,26]. Furthermore, appointment management systems have largely remained siloed and disconnected from domain-specific Q&A or document interpretation functionalities. Despite these advancements, most solutions remain fragmented, each addressing a narrow task in isolation. To the best of our knowledge, no prior research has proposed a single VA that simultaneously integrates (1) RAG-based medical Q&A grounded in verified clinic data, (2) interpretation of laboratory results from PDF documents, and (3) NL-based appointment scheduling.

This paper addresses this gap by proposing an agentic VA designed specifically for medical clinics. Our system unifies these three capabilities within a single, explainable architecture powered by Gemini 2.5 Flash, enabling accurate, grounded and context-aware patient–clinic interaction. Through this integration, we aim to advance the practical deployment of comprehensive, domain-specialized AI assistants in healthcare environments.

3. Research Methodology

To address the objectives mentioned in the introduction, our research adopts a design-oriented methodology. The approach focuses on building and evaluating an agentic VA for a medical clinic, integrating RAG, document interpretation and NL-based appointment management. The methodology is structured into several steps as illustrated in Figure 1 and presented in

Table 1. Methodological steps.

Step	Description
Step 1: System design	- The application is designed as an orchestrator architecture, where a main agent routes user queries according to the detected intent. - The main agent is connected to multiple tools, each responsible for specific functionalities (e.g., retrieval, document interpretation, scheduling). This orchestration is implemented using a zero-shot reasoning approach, enabling the agent to provide answers without requiring explicit prior examples. The system employs a main orchestrator agent and three specialized tool agents. The orchestrator analyzes user queries, identifies intent and routes requests to the appropriate tool. The three specialized agents are: (1) the retrieval agent, which searches the clinic’s knowledge base using RAG to answer medical questions; (2) the lab interpretation agent, which extracts laboratory test results from uploaded PDFs and provides patient-friendly explanations; (3) the appointment scheduling agent, which manages booking, rescheduling and cancelation by extracting entities such as dates, doctors and services. It defines the architecture with three main pages: (1) Home: secure login and first user contact (application not accessible without an account). (2) Chatbot: patient-facing interface with three core functionalities: retrieval-augmented Q&A, lab report/document interpretation and appointment scheduling. (3) Evaluation dashboard: admin-only interface to observe chatbot sessions, visualize analytics and monitor usage through charts and reports.
Step 2: Data collection and preprocessing	The second step involves collecting and preparing the knowledge base that supports the chatbot’s responses. Since the chatbot will be used to support a medical clinic, the data sources include medical articles, laboratory test descriptions and information on medical conditions extracted from the clinic’s website. The clinic’s website contains scientific articles written in Romanian, posted in an official capacity by the clinic, with the aim to respond to patient’s questions or inform on specific topics. The appointment schedule was built by extracting the names and the focus area of real doctors from the website, information publicly available which was later used to create the appointments on a mocked availability. The following stages are defined: - Stage 1. Data acquisition To gather the necessary materials, a web scraping process is implemented using Playwright. This framework enabled automated browsing, interaction with dynamic web elements and the extraction of relevant content across heterogeneous page formats. By adopting it, the dataset remained both up-to-date and adaptable to the diverse structure of medical resources. Periodic scraping is essential to ensure the accuracy of the chatbot’s medical knowledge. - Stage 2. Data structuring The extracted information is stored in JSON format, containing fields such as URL, title and content. In order to make the data suitable for RAG, the text is divided into overlapping chunks of approximately 800 characters, using LangChain’s RecursiveCharacterTextSplitter. This approach enhances retrieval efficiency while preserving contextual continuity, as the overlapping regions help maintain meaning across adjacent segments. - Stage 3. Embedding and indexing Each chunk of text is transformed into a vector representation using OpenAI’s text-embedding-3-small model, which produces embeddings of 1536 dimensions. These embeddings are subsequently stored in a Pinecone vector database, comprising 12,757 vectors stored within a dedicated clinic_info namespace, making it possible to perform semantic search and achieve fast, contextually relevant document retrieval during conversations with the chatbot. - Stage 4. Quality assurance Preprocessing also incorporates a quality control layer through the “LLM-as-a-judge” technique. This framework introduces evaluation metrics such as relevance, grounding, completeness, coherence and conciseness. These metrics function both as safeguards against hallucinations, ensuring that generated answers remain anchored in verified medical sources, and as analytic tools for monitoring and improving the chatbot’s performance in real-world interactions.
Step 3: Implementation	The implementation also follows several stages: (1) LLMChain creation-establishes the core reasoning flow by linking the LLM to a customized prompt. (2) ZeroShotAgent definition-allows the agent to dynamically select and apply the most appropriate tools to solve complex user tasks. (3) OutputFixingParser integration-automatically corrects formatting errors in model outputs, ensuring coherent and valid responses. (4) AgentExecutor deployment-runs the agent in real time, combining the agent, its tools and a memory module that maintains conversational context. This component also manages parsing errors and provides visibility into intermediate reasoning steps. (5) Interactive visualization-a StreamlitCallbackHandler connects the agent’s execution to a Streamlit container, allowing users to observe the agent’s reasoning process in real time. (6) Execution configuration-the handler is embedded within a RunnableConfig, which passes execution parameters to the agent at runtime. (7) Final response generation-the chatbot is triggered with the command agent_executor.invoke(…), which processes the user prompt, generates the response and then formats it for clear presentation to the end user. An extract from the implementation is presented below:  # Build the LLM agent chain with tools, memory and output parsing   llm_chain = LLMChain(llm = llm, prompt = prompt_custom)   agent = ZeroShotAgent(llm_chain = llm_chain, tools = tools, verbose = True)   output_parser = OutputFixingParser.from_llm(llm, parser = StrOutputParser())   agent_executor = AgentExecutor.from_agent_and_tools(     agent = agent,     tools = tools,     memory = memory,     handle_parsing_errors = True,     return_intermediate_steps = True,     verbose = True,     output_parser = output_parser) # Set up Streamlit callbacks for real-time interaction   callback_handler = StreamlitCallbackHandler(st.container(), expand_new_thoughts = False)   config = RunnableConfig(callbacks = [callback_handler]) # Run the agent and process the final output   response = agent_executor.invoke({“input”: prompt}, config = config)   last_output = extract_last_tool_output(response)     if last_output:       response_text = clean_response(last_output)     elif response and “output” in response:       response_text = clean_response(response.get(“output”)) if response else None
Step 4: Prompt engineering	The orchestration relies on a zero-shot reasoning approach, which allows the agent to respond to user queries without requiring pre-defined examples for each case. The model used for all the prompts is Gemini 2.5 Flash, released by Google. While the choice of model is important to the performance of the system, the design of the prompts ultimately guides the direction of the conversation. Prompts are very important in this use case and they are used not only for task classification but also for the evaluation of responses through an LLM-as-a-judge mechanism that generates metrics on answer quality. Each model presents specific characteristics and as such, prompt engineering requires iterative testing and refinement. Despite the existence of general guidelines, this process is not strictly rule-based but rather experimental, involving continuous validation across diverse scenarios. As a best practice, the main prompt includes guidance via several structured elements: role and persona of the agent a detailed explanation of the task available tools and usage logic formatting rules for reasoning placeholders for input parameters These components ensure that the system can provide reliable, context-aware and domain-specific assistance in the medical setting. Part of the main prompt is provided below. **Role:** You are an advanced AI assistant agent for a medical clinic. **Task:** Your primary goal is to accurately understand the user’s current request, **considering the ongoing conversation stored in {chat_history}**, and select the single most appropriate tool to fulfill it. You must then use the tool’s output to formulate a helpful, human-readable final answer. **Available Tools & Selection Priority:** … **Tool Selection Logic:** … **CRITICAL FORMATTING RULES:** 1. You **MUST** strictly follow the Thought/Action/Action Input/Observation sequence. 2. **EVERY** ‘Thought:’ **MUST** be immediately followed by an ‘Action:’ and ‘Action Input:’. **NO EXCEPTIONS**, except for the final ‘Thought:’ before the ‘Final Answer:’. 3. **ALWAYS** use a tool first. Do **NOT** answer from memory. 4. Use **plain text only** for SQL queries in ‘Action Input:’. **NEVER** use markdown, backticks (′), or code blocks. **FINAL ANSWER FORMATTING RULES:** 1. Format the ‘Final Answer:’ clearly and concisely for the user. 2. Use bullet points (‘-’) for lists (e.g., appointments, doctors, services). 3. Appointment details in the final answer should ideally include: Date, Time, Doctor Name, Service Name, Location Name. Patient name might be omitted if context implies it’s the user asking about their own appointments. 4. Explanations from ‘retrieval_qa’ should be informative but **always include a disclaimer recommending consultation with a healthcare professional** for definitive medical advice or diagnosis. **Previous Conversation:** {chat_history} **(Now, begin processing the latest input below based on the history and your rules)**
Step 5: Assessment	The evaluation of the proposed system is conducted through an Evaluation Dashboard, designed to monitor, log and analyze chatbot conversations by providing filtering, visualization and analytic functionalities that facilitated both qualitative and quantitative assessments. The assitant provides: (1) Integration of five qualitative metrics, implemented using the “LLM-as-a-judge” approach. Each generated answer is automatically evaluated and visualized on the dashboard, with scores distributed across five possible levels. The metrics included: - Relevance-assessed how closely the answer addresses the user’s query based on retrieved sources. - Grounding-measured the degree to which the response is factually supported by the underlying medical documents. - Completeness-evaluated whether the response incorporates all necessary information from the available sources. - Coherence-focused on the logical flow and clarity of the response. - Conciseness-analyzed whether the response conveyes information effectively without unnecessary complexity or redundancy. (2) Tracking the frequency of source usage, by aggregating URLs across conversations, administrators could identify which medical articles, laboratory test descriptions or clinical guidelines are most frequently retrieved and used by the assistant. (3) Sentiment analysis of both questions and responses-using an LLM-based judgment framework, each interaction is labeled with a sentiment category, scored numerically and accompanied by an explanatory rationale from the model. The results are displayed in tabular format with four columns: User Question, Sentiment Label, Score and Model Justification. This analysis offered deeper insights into the emotional dynamics of conversations, which can be essential for patient-centered applications in healthcare. (4) Standard NLP metrics-to complement qualitative measures, standard benchmarking metrics are applied to interactions where ground truth answers are available. These include: - ROUGE-1, measuring unigram (word-level) overlap between the generated and reference answers. - ROUGE-L, evaluating the longest common subsequence to capture structural similarity. - BLEU, assessing n-gram overlap precision between predicted and reference answers. The distribution of scores for each metric is visualized through dedicated graphs, alongside mean performance values, providing a quantitative benchmark.

, covering system design, data collection and preprocessing, implementation and evaluation.

Figure 2 presents the architecture in short focusing more on the RAG process which is part of 2 tools out of the 3 ones.

4. Results

The development and evaluation of the medical VA provide several important findings that reflect both the feasibility of the proposed architecture and the challenges that accompany such a system in a clinical context. These findings emerge not only from the technical implementation but also from the assessment that combines qualitative and quantitative evaluation metrics. Gemini 2.5 Flash is used in the whole chatbot, for the main prompt as well as for each LLM-as-a-judge metric. Gemini 2.5 Flash is selected due to its balanced performance in reasoning accuracy, low latency and cost efficiency, which are important requirements for real-time clinical assistant systems. The model demonstrated reliable instruction following and structured output generation. Its stable behavior under constrained prompting supports the delivery of grounded, consistent and clinically appropriate responses, making it well suited for deployment in a medical virtual assistant context.

4.1. Technical Feasibility of the Orchestrator-Based Architecture

The adoption of an orchestrator-based architecture, supported by a ZeroShotAgent, proves to be an effective design choice for managing the diverse tasks expected of a medical VA. The system is able to dynamically route user queries to specialized tools for knowledge retrieval, laboratory report interpretation and appointment scheduling, all without the need for extensive fine-tuning or pre-programmed examples. This modular approach enhances flexibility and also ensured that each function is handled by the most appropriate component, thereby improving overall dependability.

Moreover, the integration of real-time execution with Streamlit visualization adds an important layer of transparency by making the assistant’s intermediate reasoning steps visible to administrators. In a medical context, where accountability and trust are essential, the ability to trace how an answer is generated is a significant advantage, as it allows both developers and clinicians to better understand, evaluate and refine the assistant’s decision-making processes.

The findings also demonstrate the effectiveness of the RAG pipeline in supporting knowledge retrieval and document interpretation within the medical assistant. By combining embeddings, a Pinecone vector database and LLM integration, the system is able to consistently retrieve relevant information from the clinic’s resources and deliver responses that scored highly in terms of relevance, grounding and completeness. The important factors in this performance are:

-

The use of overlapping chunking ensures that contextual meaning is preserved across adjacent passages. As a result, the assistant is able to provide more coherent and accurate answers, maintaining the continuity of medical information that is often spread across lengthy or complex documents.

-

The model chosen, when testing with Gemini 2.0 Flash Lite, the performance is poor not only for the main orchestrator agent, but also for the LLM guardrails which are either returning an evaluation that is not making complete sense or it fails to stick to the structured output.

-

Prompt engineering, the use of a structured prompt format proves important in guiding the system to produce trustworthy outputs, ensuring both linguistic consistency and domain-appropriate responses. A particularly effective element of this design is the strict requirement that the assistant generate answers exclusively based on the information extracted from the clinic’s website, rather than memory when generating answers. This constraint significantly reduced the risk of hallucinations and reinforced the importance of controlled orchestration in healthcare, where unsupported or fabricated medical information could have serious consequences.

-

Testing shows that these strict control mechanisms involve certain trade-offs: while they enhance reliability and ensured adherence to established rules, they sometimes constrain the natural flow of conversation, causing the assistant to appear less adaptable when faced with open-ended or ambiguous queries. This highlights the ongoing challenge of designing prompts that both preserve medical accuracy and provide a smooth, user-friendly conversational experience.

Figure 3 illustrates an example of the chatbot’s response to a patient query regarding uploaded laboratory test results. In this instance, the assistant extracts and presents the numerical value from the document and also provides a personalized interpretation tailored to the logged-in user. The underlying knowledge sources used for retrieval are in Romanian, and the chatbot is explicitly configured to generate outputs in Romanian. For clarity and reproducibility within this paper, the responses have been translated into English and, as shown in the example, the laboratory parameter “VSH” corresponds to the erythrocyte sedimentation rate (ESR) test. Complementarily, Figure 4 demonstrates how the reference intervals for the ESR test would typically be displayed in the absence of interpretation. This comparison underscores the added value of the assistant’s reasoning layer, which enhances raw laboratory data with contextualized, patient-centered explanations.

Figure 5 presents the evaluation metrics used to determine whether a generated response is appropriate for delivery to the end user. Each of the three metrics is accompanied by both a categorical level and a textual justification, which provide a transparent basis for later analysis through the visualization dashboard. The metrics are structured on a five-point ordinal scale. For example, in the case of the relevance criterion, the possible levels range from “Very Relevant” and “Relevant” to intermediate assessments such as “Partially Relevant” and further to “Slightly Relevant” or “Not Relevant”. Similar structured scales are applied to the other two metrics, ensuring consistent interpretability. For a response to be released to the user, the assigned levels across all three metrics must fall within the top three categories of the respective scale. This safeguard ensures that only answers meeting predefined quality thresholds in terms of relevance, grounding and completeness are ultimately communicated, while less reliable outputs are filtered for further review.

4.2. Evaluation Metrics and Performance Insights

The evaluation framework integrates multiple complementary metrics to enable a comprehensive assessment of the assistant’s performance across distinct quality dimensions. Lexical overlap metrics, namely ROUGE and BLEU, provide standardized quantitative measures of surface-level correspondence with reference answers, facilitating consistent comparison across experimental configurations. However, recognizing their limitations in capturing semantic nuance and clinical appropriateness, we incorporate SBERT-based semantic similarity analysis to measure whether responses convey equivalent meaning despite variations in phrasing or structure. Finally, the LLM-as-a-judge framework addresses gaps that purely automated metrics cannot capture, evaluating responses along clinically relevant dimensions such as grounding, completeness, coherence and relevance. This approach ensures that evaluation accounts for lexical precision, semantic consistency and domain-specific quality requirements, providing a more robust and interpretable assessment than any single metric could achieve in isolation.

The evaluation of the medical VA provides a detailed picture of its performance across multiple dimensions. The LLM-as-a-judge metrics indicate that the assistant generally performs well in relevance, grounding and completeness. The relevance pie chart from Figure 6 shows that most responses fall into the “partially relevant” and “relevant” categories, with only a small proportion considered “very relevant” or “not relevant” suggesting the assistant reliably retrieves useful information but still has room to optimize precision in certain queries. Grounding appears particularly strong, with the vast majority of responses classified as “well grounded” and a small fraction as “not grounded”, confirming that the assistant predominantly bases its answers on verified sources. Completeness exhibits greater variability, with a mix of “fully complete”, “mostly complete” and “partially complete” responses, indicating that while the assistant often provides sufficient information, complex or multifaceted queries can sometimes result in partial coverage of all relevant details.

Regarding coherence and conciseness, the charts reveal that most responses are rated as “fair” or “good,” with a smaller proportion achieving “excellent” ratings. This suggests that while the answers are generally understandable and structured, there is variability, particularly when handling more complex which required reliable extraction of entities information. The combination of these qualitative insights underscores that the assistant can consistently support patient queries, though careful prompt design and iterative improvements may further be needed to optimize readability and efficiency.

For the evaluation process, a subset of benchmark questions is selected from the HealthSearchQA dataset [25], which comprises a total of 3173 medical questions. To ensure a representative yet computationally manageable sample, the queries that are extracted and used for testing and analysis are filtered out based on a high search match score.

All benchmark questions (in English) are translated into Romanian using a translation pipeline to ensure terminological fidelity, while the clinic’s knowledge base remained in Romanian to reflect the real deployment environment. Benchmark questions are translated into Romanian via Gemini 2.5 Flash using a domain-aware translation prompt that preserves medical terminology. A second Gemini invocation operates as an LLM-as-judge to verify fidelity, flagging errors or ambiguities. Only translations judged “OK” are used for evaluation with our Romanian-based knowledge base; flagged ones are manually reviewed or excluded to ensure semantic alignment and terminological accuracy.

Table 2. Examples of the explained output.

English	Romanian	Judge
Are benign brain tumors serious?	Sunt tumorile cerebrale benigne grave?	Verdict: OK Explanation: Corrected Romanian: -
Are bone cysts serious?	Sunt chisturile osoase grave?	Verdict: OK Explanation: Corrected Romanian: -
Are cold sores a herpes virus?	**Este herpesul labial (cold sores) un virus herpetic?**	Verdict: ERROR Explanation: The parenthetical “(cold sores)” is redundant and unnecessary, making the translation less natural and professional. “Herpesul labial” is the standard and well-understood term in Romanian for cold sores. Corrected Romanian: Este herpesul labial un virus herpetic?
Are floaters in eye serious?	**Sunt corpi flotanți în ochi gravi?**	Verdict: ERROR Explanation: The term “corpi flotanți” is technically correct but less idiomatic for “floaters” in a patient’s question; “musculițe zburătoare” is the more common and natural term. Additionally, “gravi” (masculine plural) refers directly to the “corpi,” whereas “grave” (feminine plural) would be more appropriate when referring to “musculițele zburătoare” and the implied seriousness of the condition. Corrected Romanian: Sunt grave musculițele zburătoare din ochi?

shows examples of the explained output, the “judge” column is later parsed to take the correct translation.

We compare the outputs in three configurations structured to isolate the contribution of two primary factors, the RAG mechanism and the change in the LLM model:

(A) with clinic-specific retrieval enabled and Gemini 2.5 Flash;

(B) as a standalone model with Gemini 2.5 without external context;

(C) with clinic-specific retrieval enabled and Gemini 2.0 Flash.

All models are queried using the same set of user questions and are instructed using the same prompt template (in the case of no RAG, the prompt is changed to remove the mention of the chunks) to ensure comparability. For each question, all three models generate one response, resulting in triplets of responses for direct comparison.

The Sentence-BERT (SBERT) model is employed as the metric for response comparison. SBERT is utilized to quantify the semantic similarity between responses, focusing on shared meaning rather than lexical overlap. This approach is achieved by extending the Bidirectional Encoder Representations from Transformers (BERT) architecture to generate fixed-length sentence embeddings. The semantic relationship between any two responses is then calculated using cosine similarity on their embeddings. Given that higher cosine similarity values show a stronger semantic overlap, SBERT is well-suited for assessing whether different outputs convey the same underlying information, even when surface-level phrasing or structural elements vary [26].

Figure 7 identifies the three pairwise semantic similarity comparisons:

• A vs. B (2.5 RAG vs. 2.5 No RAG): This comparison isolates the effect of retrieval augmentation within the same model generation.
• A vs. C (2.5 RAG vs. 2.0 RAG): This comparison captures the impact of the model version upgrade under identical retrieval settings.
• B vs. C (2.5 No RAG vs. 2.0 RAG): This comparison primarily reflects the differences driven by the model version when RAG is either entirely absent (in the case of B) or present in the older model (C).

The box plot in Figure 7 indicates that all three comparisons exhibit moderate to high semantic similarity, with median cosine similarity values generally ranging between 0.65 and 0.75. The A vs. B comparison (Gemini 2.5 Flash with RAG vs. Gemini 2.5 Flash without RAG) shows the highest central tendency, with most scores concentrated in the upper similarity range. This suggests that, within the same model version, the inclusion of RAG preserves the core semantic intent of the responses. The presence of several low-value outliers, however, indicates that RAG can occasionally lead to more pronounced semantic differences, likely due to the integration of additional contextual or clinically grounded information.

The A vs. C comparison (Gemini 2.5 Flash with RAG vs. Gemini 2.0 Flash with RAG) exhibits a lower median and broader dispersion, reflecting greater variability in semantic similarity when both the model version and its underlying capabilities differ. In contrast, the B vs. C comparison (Gemini 2.5 Flash without RAG vs. Gemini 2.0 Flash with RAG) displays a more compact distribution with fewer extreme values, indicating more consistent semantic alignment across questions. Overall, the chart shows that while semantic similarity remains high across all configurations, differences in model version and retrieval usage influence the stability and variability of generated responses.

Figure 8 indicates that for all configurations, most values cluster between 0.6 and 0.85, confirming a generally high level of semantic alignment between model outputs. The A vs. B comparison (Gemini 2.5 Flash with RAG vs. Gemini 2.5 Flash without RAG) exhibits a noticeable concentration toward higher similarity values, but also a left tail extending toward lower scores, indicating that while most responses remain close in meaning, RAG occasionally introduces more distinct, content-rich variations.

The A vs. C comparison (Gemini 2.5 Flash with RAG vs. Gemini 2.0 Flash with RAG) shows a broader spread across the similarity range, with more mass in the mid-range (approximately 0.5–0.7). This suggests increased variability when moving across model versions, even when retrieval is consistently applied. In contrast, the B vs. C comparison (Gemini 2.5 Flash without RAG vs. Gemini 2.0 Flash with RAG) displays a more compact distribution centered around similar values, indicating more stable semantic behavior. Overall, the histogram reinforces that semantic similarity is high across all pairs, while differences in retrieval usage and model version primarily affect the spread and consistency of responses rather than their core meaning.

Figure 9 summarizes the average semantic similarity and variability across the three model comparisons. It shows that B vs. C and A vs. B have similarly high mean semantic similarity (≈0.71), indicating strong overall alignment between these model pairs. In contrast, A vs. C has a lower mean (0.65), suggesting greater semantic divergence when comparing Gemini 2.5 Flash with RAG to Gemini 2.0 Flash with RAG. Variability differs across comparisons: A vs. B exhibits the highest standard deviation, highlighting that RAG can introduce more diverse responses within the same model version, while B vs. C is the most stable, with the lowest variability and higher minimum similarity.

The LLM-as-a-judge evaluation results from Figure 10 and Figure 11 demonstrate a dramatic and consistent improvement in response quality across all key metrics. The LLM-as-a-judge evaluation comparing Figure 11 (Configuration A: Gemini 2.5 Flash with RAG) and Figure 10 (Configuration C: Gemini 2.0 Flash with RAG) revealed a pronounced and consistent advantage for the responses generated by the newer Gemini 2.5 model. This configuration (Figure 11) demonstrates significant enhancements across critical quality metrics, achieving near-perfect outcomes in Relevance and Grounding by generating mostly “very relevant” and “fully grounded” responses, effectively eliminating the quality deficiencies seen in Figure 10. Further performance gains are evident in Completeness and Coherence, where the Gemini 2.5 RAG setup produces a higher proportion of “fully complete” and “excellent” responses, confirming its superior ability to provide comprehensive and logically structured medical information within the RAG framework. For the grounding chart, the “well grounded” category has a different color scheme in Figure 10, compared to Figure 6 and Figure 11 because partial grounding is also present as a category.

This indicates that the Gemini 2.5 Flash model, when integrated with the clinic-specific RAG system (Figure 11), significantly outperforms the Gemini 2.0 Flash model (Figure 10), demonstrating better consistency and quality across four out of five metrics. The overall decisive shift towards higher quality in Figure 11, especially in the critical areas of Grounding and Relevance, underscores the success of the newer LLM version in enhancing the trustworthiness and contextual appropriateness of the system’s generated output.

4.3. Limitations

Several limitations of the system are identified during evaluation. While the retrieval mechanism ensured that responses are generally well grounded in the clinic’s resources, subtle nuances are sometimes simplified or omitted, which can be a common challenge with specialized domains like healthcare, where accuracy depends not only on factual correctness but also on attention to medical details and precise terminology. The use of multiple agents, each responsible for a distinct function such as retrieval, interpretation or scheduling, helped mitigate this issue by distributing tasks, yet it also revealed that domain-specific refinement remains necessary for more complex cases.

Furthermore, the current system is evaluated exclusively within a text-based conversational interaction setting. While this scope is appropriate for the clinic’s immediate requirements, multimodal interaction scenarios, such as processing medical images alongside textual queries or supporting voice-based interfaces, remain beyond the present implementation and represent important directions for future work.

Another limitation becomes evident in the appointment scheduling process which required accurate extraction of entities such as dates, doctors and services while producing outputs in a structured format that could be easily parsed. Stronger models and carefully engineered, rule-based prompts significantly improve performance, but ambiguous or loosely phrased user queries still posed difficulties. This highlights the need for prompts that not only guide the assistant to produce safe and structured responses but also enforce strict formatting rules that maintain robustness under uncertainty.

The dependability of the system depends largely on the accuracy of its knowledge base. Since the assistant’s expertise is drawn from the clinic’s website and related resources, periodic re-scraping and updating are required to avoid outdated or inaccurate responses. This creates an ongoing maintenance burden and poses challenges for scalability in the absence of automated update mechanisms. Moreover, while the multi-agent design enhances flexibility, it also means that each agent’s performance ultimately relies on the continued relevance of the shared knowledge base.

5. Conclusions

Our research focused on the design and implementation of an AI-driven VA tailored for a medical clinic. The system was structured around an orchestrator-centered system design in which a central agent dynamically routed queries to specialized tools responsible for tasks such as retrieval-augmented Q&A, document interpretation and appointment scheduling. This setup was embedded in an application with three main interfaces: a secure home page, a patient-facing chatbot and an evaluation dashboard that allowed administrators to track interactions, review analytics and monitor usage patterns.

The assistant was supported by a carefully constructed knowledge base derived from the clinic’s website. Data were collected through automated scraping, structured into JSON format, divided into overlapping text chunks and embedded into a Pinecone vector atabase for efficient semantic retrieval. To safeguard accuracy, preprocessing included a quality control mechanism using an “LLM-as-a-judge” framework, which scored responses for relevance, grounding, completeness, coherence and conciseness. These steps ensured that the retrieval-augmented pipeline could provide contextually appropriate and factually anchored medical information.

Implementation brought these components together through the integration of LLMChains, a ZeroShotAgent for tool selection and an AgentExecutor that maintained conversational context while handling parsing errors. Output consistency was reinforced with an OutputFixingParser and transparency was enhanced by linking reasoning steps to an interactive Streamlit visualization. Finally, prompt engineering played a central role, with structured instructions defining the assistant’s role, task and formatting rules. By enforcing strict tool use, while continuously refining prompts through iterative testing, the system balanced flexibility in conversation with safeguards that reduced hallucinations and maintained medical trustworthiness.

The main insight and conclusions drawn from our research are:

(a)

The study shows that employing a multi-agent architecture is an effective way to balance retrieval, reasoning and safe response generation. By delegating distinct responsibilities, such as knowledge retrieval, report interpretation and scheduling, to specialized agents, the system was able to handle diverse clinical support tasks with greater dependability. Evaluation metrics confirmed that this modular approach reduced risks such as hallucination and insufficient source anchoring and also ensured that outputs remained interpretable and aligned with medical requirements.

(b)

The findings highlight strong potential for deploying such assistants in real clinical environments, provided robust safeguards are in place. Human oversight remains essential to validate outputs in complex or ambiguous cases, while disclaimers ensure that patients clearly understand the assistant’s supportive, not diagnostic, role. Continuous monitoring through evaluation dashboards and automated checks is equally important for maintaining trust, accuracy and clinical relevance over time.

(c)

Several limitations need to be addressed before large-scale deployment. Domain adaptation proved difficult, with subtle clinical nuances occasionally oversimplified. Appointment scheduling required precise entity extraction and rigid formatting, which was vulnerable to ambiguous phrasing despite the use of strict prompting. Furthermore, the system’s accuracy depends on periodic re-scraping and updating of knowledge sources, creating a maintenance burden that threatens scalability without automated update mechanisms.

(d)

The assistant illustrates how AI can strengthen patient communication and streamline administrative processes, offering tangible benefits in clinical workflows. At the same time, it reinforces the necessity of ethical safeguards, transparent design and continuous evaluation to ensure safe integration into healthcare contexts. Importantly, such systems should be understood as augmenting professional practice rather than replacing it, providing reliable support in routine tasks while leaving clinical responsibility firmly with medical experts.

The evaluation demonstrates that integrating Gemini 2.5 Flash with a clinic-specific RAG pipeline substantially enhances response quality, semantic consistency and source-based accuracy in the medical domain. The translation and validation workflow ensured high-fidelity Romanian benchmark queries, enabling accurate testing in a realistic deployment setting. Across SBERT-based comparisons, all model configurations produced strongly aligned responses, though the introduction of RAG increased variability while enriching contextual relevance. The LLM-as-a-judge assessment further confirmed that Gemini 2.5 Flash with RAG delivered the most relevant, coherent, complete and evidence-supported responses, far surpassing the 2.0 version under identical retrieval conditions. Despite residual challenges, such as nuanced medical phrasing, strict formatting for scheduling tasks and the need for continuous maintenance of the knowledge base, the overall results show that the upgraded model and retrieval framework meaningfully improve dependability, usefulness and domain alignment in a healthcare assistant setting.