Large Language Models in Clinical Decision Support: A Critical Review of Progress, Limitations, and the Path Toward Sustainable Healthcare

Abstract

Large language models such as GPT-4, Med-PaLM 2, and BioMedLM are being increasingly applied to clinical decision support tasks including diagnostic reasoning, clinical documentation, medical question answering, and patient-facing health communication, with several studies reporting performance comparable to trained clinicians on structured benchmarks. However, the current pace of adoption has outrun the critical evaluation of whether these models are genuinely safe, equitable, and sustainable for deployment in real-world healthcare environments. This narrative review argues that the existing literature has focused disproportionately on benchmark performance while giving insufficient attention to the conditions required for these models to function as trustworthy long-term clinical tools. Drawing on published evidence across medical informatics, clinical artificial intelligence, and natural language processing, we synthesise model performance across four clinical task areas and examine documented limitations including factual hallucination, demographic bias, and poor generalisability outside controlled settings that are already present in current deployments. We further identify five structural sustainability challenges that obstruct responsible deployment at scale, and propose a corresponding five-pillar framework alongside a priority research agenda for closing the gaps this review identifies. This review contributes a critically grounded perspective to the growing discourse on artificial intelligence for sustainable healthcare, with direct relevance to Sustainable Development Goal 3 (Good Health and Well-Being) and Sustainable Development Goal 10 (Reduced Inequalities), and offers concrete guidance for developing clinical models that are safe, inclusive, and viable across diverse global health contexts.

1. Introduction

Clinical decision-making sits at the heart of every patient encounter, and yet the conditions under which clinicians make those decisions have become increasingly difficult to sustain. Physicians today face an extraordinary volume of information: electronic health records that can run to hundreds of pages per patient, a medical literature that doubles roughly every 73 days, and time pressures that leave little room for careful reflection [1,2]. These conditions have real consequences. A landmark study by Singh et al. estimated that approximately 12 million adults in the United States experience a diagnostic error in outpatient settings every year, with around half of those errors carrying the potential for patient harm [3]. On a global scale, the World Health Organisation reports that more than one in ten patients is harmed during medical care, and that adverse events cause approximately 2.6 million deaths annually in low- and middle-income countries alone [4]. These are not failures of individual clinicians working carelessly; they are largely the product of systems that ask people to do more than is humanly possible without adequate support.

The workforce carrying this load is under serious strain. Surveys conducted by the American Medical Association found that nearly half of all US physicians reported at least one symptom of burnout in 2023, a figure that, while lower than the peak of 62.8% recorded during the COVID-19 pandemic, remains substantially higher than rates seen in the general working population [5]. A significant driver of this strain is not the difficulty of clinical work itself but the administrative burden that surrounds it: time spent on documentation, prior authorizations, and navigating fragmented information systems rather than caring for patients [6]. The result is a workforce that is increasingly stretched and a system that, by its own metrics, is failing a meaningful proportion of the people it is supposed to help.

It is within this context that artificial intelligence, and large language models in particular, have entered clinical medicine. Unlike earlier AI tools developed for narrow, well-defined tasks such as detecting a specific lesion type in a radiology image or flagging a particular drug interaction, large language models (LLMs) are general-purpose systems capable of processing and generating clinical text across a wide range of tasks [2,7]. They can read a patient history and suggest a differential diagnosis, draft a discharge summary from raw clinical notes, answer a physician’s question about drug dosing, or explain a diagnosis to a patient in plain language. This breadth of capability is genuinely new, and it has generated considerable interest from clinicians, health systems, and researchers who see an opportunity to address some of the most persistent challenges in healthcare delivery. Models such as GPT-4, Med-PaLM 2, and BioMedLM have demonstrated performance on structured clinical benchmarks that, in some evaluations, approaches or matches that of trained clinicians [8,9]. These results have attracted wide attention, and the pace of clinical deployment has accelerated accordingly.

What has not kept pace is critical evaluation. The majority of published work on LLMs in clinical settings focuses on benchmark performance, measuring how well a model answers multiple-choice medical questions or scores on standardized examination tasks. Far less attention has been paid to whether these models are safe and reliable when used by real clinicians with real patients, whether they perform equitably across different demographic groups, or whether their deployment can be sustained responsibly over time within the complex institutional, regulatory, and ethical landscape of healthcare. This review argues that the field has moved too quickly toward enthusiasm and too slowly toward accountability, and that addressing this imbalance is not a technical afterthought but a prerequisite for building clinical AI that is genuinely useful and sustainable over the long term.

1.1. Defining the Scope

This review focuses specifically on the application of large language models to clinical decision support tasks: those activities in which a clinician or a patient-facing system uses an LLM to assist with forming, communicating, or acting on a clinical judgment. The four task areas included in this review are diagnostic reasoning, clinical documentation, medical question answering, and patient-facing applications. These categories represent the areas where LLM deployment in clinical settings is most active and where published evidence, both supportive and critical, is most available [7,10,11]. Table 1 summarises the full scope boundaries applied throughout this review. A broader taxonomy of six task areas, including treatment recommendation and clinical summarisation, is mapped in the Table in Section 2.3 to situate these four within the wider landscape of clinical large language model research.

Table 1. Scope boundaries for this narrative review of large language models in clinical decision support.

	In Scope	Out of Scope (with Justification)
Tasks	Diagnostic reasoning; clinical documentation; patient triage; medical question answering; treatment recommendation	Administrative scheduling; billing and coding automation; supply chain optimization Reason: No direct clinical judgment involved
Models	GPT-4; Med-PaLM 2; BioMedLM; LLaMA clinical fine-tunes; Med-Gemini	General chatbots without clinical evaluation; models with no published clinical assessment Reason: Insufficient evidence base for critical appraisal
AI Type	Text-generating LLMs and multimodal LLMs with NLP as the primary decision interface	Pure imaging AI (radiology/pathology CNNs); genomics sequence models; robotic surgery systems Reason: Different technical basis and separate review literature
Pipeline	Clinical decision support across all care settings (primary, secondary, tertiary, patient-facing)	Drug discovery; molecular property prediction; protein structure modeling Reason: Distinct from clinical practice and governed by separate development standards

The models considered in this review include both general-purpose LLMs that have been evaluated in clinical settings and models specifically developed or fine-tuned for healthcare applications. The former group includes GPT-4 (OpenAI) and its predecessors, which have been tested extensively across medical benchmarks and in simulated clinical scenarios [9,12]. The latter group includes Med-PaLM 2 (Google), which was developed with clinical knowledge explicitly incorporated into its training and instruction tuning [8]; BioMedLM (Stanford CRFM and MosaicML), a domain-specific model trained entirely on biomedical literature from PubMed [13]; LLaMA-based clinical fine-tunes, which represent the growing open-source ecosystem of medically adapted models [14]; and Med-Gemini (Google DeepMind), a multimodal model built on the Gemini architecture and evaluated across fourteen clinical tasks including long-context medical record processing [15]. Together, these models cover the dominant proprietary and open-weight systems currently shaping the LLM-in-healthcare literature, and they provide a representative basis for the critical analysis that follows.

Several important and adjacent areas of clinical AI are excluded from this review, and the reasoning for each exclusion is worth stating clearly to avoid misreading the scope as a gap in coverage. AI systems applied to medical imaging, such as deep learning models for radiology or pathology, are excluded unless they incorporate a substantial natural language processing component that connects to clinical decision-making [16]. These imaging models operate on a fundamentally different technical basis and have their own established review literature. Drug discovery pipelines, including models applied to molecular property prediction, protein structure, or candidate compound generation, are similarly outside the scope of this review, as are genomics-only applications where the LLM component, if present at all, is secondary to sequence modeling [7]. Finally, AI tools used purely for administrative purposes, such as scheduling optimization, billing code suggestion, or supply chain management, are excluded because they do not directly influence clinical judgments or patient care decisions, which is the specific sustainability concern this review is examining.

This distinction between clinical and administrative AI is worth holding carefully. Both have value, but the risks attached to them are different in kind. An error in a scheduling algorithm creates operational inefficiency; an error in a diagnostic recommendation or a treatment suggestion can directly harm a patient. This review is concerned with the latter category and with the question of whether LLMs are being developed and deployed in that category in a way that is genuinely safe and sustainable over time.

1.2. The Central Argument

This review advances one central argument: large language models hold real potential for clinical decision support, but the way they are currently developed, evaluated, and deployed is not sustainable, and until that changes, their clinical value will remain theoretical rather than realized. This is not an argument against LLMs in healthcare. It is an argument about what responsible progress in this area actually requires, and about the specific gap between where the field is now and where it needs to be.

The evidence behind this argument is already accumulating. A systematic review of 519 published studies on LLMs in healthcare found that fewer than 5% used real patient care data, while 44.5% relied on medical licensing examination questions as the primary evaluation task [17]. A parallel analysis of the 100 most cited papers on medical LLMs found that 60% assessed models exclusively on public exam-based benchmarks, with no consensus on what constitutes valid evaluation for clinical deployment [18]. Perhaps most telling, a systematic review of 39 clinical LLM benchmarks published between 2017 and 2025 confirmed that, despite LLMs now routinely exceeding human performance on medical licensing examinations, no device using an LLM had received regulatory clearance from the US Food and Drug Administration as of 2024 [19]. The field has become very good at measuring what LLMs can do on tests. It has paid far less attention to whether they are safe, equitable, and governable in the places where medicine actually happens.

This matters for reasons that extend well beyond clinical performance. The United Nations Sustainable Development Goals, and SDG 3 on good health and well-being in particular, rest on the principle that health interventions should be accessible, safe, and effective across all populations and all settings, not only in well-resourced environments with robust infrastructure and regulatory oversight [20]. An AI tool that performs well for English-speaking patients at a large academic medical center, but fails for patients in rural or low-income settings, or that produces confident but incorrect outputs without any mechanism for accountability, does not advance sustainable health outcomes. It compounds existing inequalities under the appearance of progress. The sustainability framing adopted in this review is therefore not a rhetorical device for fitting the scope of a Special Issue. It is the substantive lens through which the evidence is assessed, because a clinical AI tool that cannot be trusted, explained, or held accountable over time is, by any reasonable definition, not a sustainable clinical tool.

Several reviews of large language models in healthcare and clinical medicine have been published in recent years. Existing systematic reviews have catalogued evaluation methodologies across hundreds of studies, mapped task-specific performance metrics, and documented application trends across clinical specialties [17,19]. Others have focused specifically on real-world clinical workflow integration, patient-facing applications, or the technical architectures underlying clinical model development. These contributions have been valuable in establishing the empirical landscape of clinical large language model research.

The present review differs from this body of work in three respects. First, it adopts sustainability as its primary organising lens rather than performance or application scope. Where existing reviews ask what large language models can do in clinical settings, this review asks whether the conditions for safe, equitable, and accountable deployment currently exist and, if not, what would be required to create them. Second, rather than cataloguing a broad range of studies, this review critically evaluates the evidence across four core clinical task areas and uses that evaluation to identify structural gaps that cross-cut all of them. Third, it advances a concrete five-pillar framework linking documented limitations directly to actionable recommendations for developers, clinicians, regulators, and funders—a constructive contribution that moves beyond description toward guidance. No existing review of clinical large language models has organised its analysis around the Sustainable Development Goals or proposed a framework that integrates clinical validation, explainability, equity, regulatory readiness, and governance as interdependent conditions for trustworthy deployment.

Against the context established above, this review pursues four explicit objectives.

• To map the current landscape of LLM applications in clinical decision support. This objective describes which models are being applied, across which clinical tasks, and with what degree of published evidence behind them. It is addressed in Section 2 and Section 3, and provides the factual foundation on which the critical analysis rests.
• To critically evaluate the performance limitations of clinical LLMs. This objective goes beyond reporting what models can do and examines where and how they fail in clinical contexts, including factual hallucination, demographic bias, generalisation failure, and the gap between benchmark performance and real-world clinical reasoning. It is addressed within Section 3, which closes with a synthesis of the documented limitations that cut across all four task areas.
• To identify the core sustainability challenges that obstruct responsible clinical LLM deployment. This objective examines the structural conditions that prevent LLMs from functioning safely and equitably at scale in real health systems, including explainability deficits, regulatory immaturity, inequitable global access, environmental cost, and governance failures. It is addressed in Section 4.
• To propose a framework and research agenda for sustainable clinical LLMs. This objective translates the critical analysis into constructive guidance: a five-pillar framework for what responsible LLM deployment requires, and a set of specific priority research directions for the field. It is addressed in Section 5.

The remainder of this paper is organised as follows. Section 2 provides the technical and historical context needed to situate LLMs within the broader trajectory of clinical decision support. Section 3 synthesises the published evidence on LLM performance across four clinical task areas including diagnostic reasoning, clinical documentation, medical question answering, and patient-facing applications and closes with a synthesis of the documented limitations that cut across all four. Section 4 examines the structural challenges that currently prevent responsible deployment at scale: explainability, regulatory immaturity, global equity, environmental cost, and governance. Section 5 proposes a five-pillar framework for sustainable clinical LLM development and sets out the priority research agenda needed to close the gaps this review identifies. Section 6 draws the findings together and returns to the central argument.

1.3. Literature Search Strategy

This narrative review was conducted in accordance with the principles of the Scale for the Assessment of Narrative Review Articles (SANRA), which emphasises a clearly stated rationale, explicit review objectives, a justified literature search strategy, and transparent scientific reasoning [21]. A structured but non-systematic search of PubMed, Scopus, Web of Science, and arXiv was performed for literature published between January 2018 and March 2025. The start date was chosen to capture the emergence of transformer-based language models following the publication of the foundational Vaswani et al. [22]; arXiv was included because a substantial proportion of the most recent and methodologically relevant work in this field appears first as preprints before journal publication.

Search terms were applied in combinations and included: “large language models,” “clinical decision support,” “artificial intelligence,” “healthcare AI,” “GPT,” “Med-PaLM,” “BioMedLM,” “LLaMA,” “diagnostic reasoning,” “clinical documentation,” “medical question answering,” “hallucination,” “explainability,” “AI governance,” “health equity,” and “sustainable AI.” Reference lists of retrieved articles were also examined to identify additional relevant studies not returned by the primary search.

Studies were included if they reported empirical evaluations of large language model performance on clinical tasks, examined failure modes or limitations of clinical AI systems, addressed regulatory or governance frameworks for healthcare AI, or proposed evaluation methodologies for clinical decision support tools. We prioritised peer-reviewed publications but included preprints where they represented the most current or methodologically relevant evidence on a specific question. Non-English publications, studies examining narrow AI or traditional machine learning tools without comparison to large language models, and purely theoretical papers without empirical grounding were excluded. Given the rapid pace of publication in this field, we did not apply a strict study design filter; instead, study quality was assessed narratively, with the level of evidence and risk of bias noted where relevant in the text. The final corpus of included studies is reflected in the reference list.

2. Background

2.1. From Rule-Based Systems to Foundation Models

Clinical decision support has been an explicit goal of medical AI since the 1970s. Early systems such as MYCIN, developed at Stanford University to recommend antibiotic treatments for bloodstream infections, operated through hand-crafted rule sets encoding expert knowledge as conditional logic [23]. These systems could reason within narrow, well-defined domains, but they did not learn from data and could not generalise beyond the rules their designers had explicitly programmed. The broader problem they exposed was one of knowledge acquisition: capturing the full depth and flexibility of clinical reasoning in a finite set of rules proved intractable, and by the 1990s, most rule-based clinical AI had failed to achieve sustained clinical adoption [24]. The following two decades brought a shift toward statistical machine learning, with supervised models applied to tasks such as diabetic retinopathy screening, sepsis risk prediction, and cancer detection in radiology [25]. These narrowly scoped models achieved strong performance within their intended tasks but remained brittle outside them: a model trained to detect a specific pathology in chest radiographs from one institution could perform poorly on images from another, and no single model could reason across the breadth of clinical tasks a practising clinician encounters in a single shift. Figure 1 illustrates the key milestones in this trajectory.

Figure 1. Historical milestones in the development of artificial intelligence for clinical decision support, from rule-based expert systems to large language model foundation models. Era shading reflects the dominant technical paradigm in each period.

The paradigm shifted fundamentally with the introduction of the transformer architecture in 2017 [22]. By replacing recurrent sequence processing with self-attention mechanisms, transformers enabled models to be trained at a scale previously impractical, and pre-training on large corpora of unlabelled text produced representations that transferred across tasks without task-specific redesign. BERT (2018), GPT-2 (2019), and GPT-3 (2020) demonstrated that a single pre-trained model could be adapted to diverse language tasks through fine-tuning or prompting alone [26]. The clinical implications became apparent quickly. By 2022, models such as Med-PaLM 2 and BioMedLM had been trained or fine-tuned on biomedical literature and clinical text, and GPT-4, released in 2023, demonstrated performance on standardised medical examinations that exceeded prior benchmarks by a substantial margin [8,9]. What distinguishes these foundation models from all earlier clinical AI is not simply their scale but their generality: a single model can draft a clinical note, answer a pharmacology question, and explain a diagnosis to a patient, without requiring separate training pipelines for each task. It is this generality that makes LLMs qualitatively different from earlier clinical AI tools, and it is also what makes their responsible deployment considerably more complex.

2.2. What Makes Large Language Models Different

Understanding what distinguishes LLMs from earlier clinical AI tools is necessary for interpreting both their performance and their limitations. Earlier systems, whether rule-based expert systems or narrowly scoped supervised models, were designed to solve one task on one data type in one setting. An LLM is structurally different. At its core, it is a transformer-based neural network [22] trained to predict the next token in a sequence of text across an enormous and diverse corpus. Through this process, the model develops rich internal representations of language, facts, reasoning patterns, and contextual relationships, without being explicitly programmed to acquire any of them. The key mechanism is self-attention, which allows the model to weigh the relevance of every word in an input against every other word when generating a response, enabling nuanced contextual understanding that earlier sequence models could not achieve. A useful way to picture this for a clinical audience is to think of the model as having read an extraordinarily large portion of the world’s published text and developed a flexible capacity to reason and write across the topics it has encountered, including medicine.

What distinguishes LLMs further is how they are adapted for deployment. Pre-training on a broad corpus produces a general foundation. That foundation can then be fine-tuned on domain-specific data, adjusting the model’s weights toward a target task or field, as in Med-PaLM 2 and BioMedLM [8,13]. Alternatively, the same pre-trained model can be guided at inference time through prompting, where the task, constraints, and examples are supplied in the input text rather than encoded into the model’s weights, as demonstrated by GPT-4’s clinical performance without any medical fine-tuning [9]. This distinction matters clinically: a fine-tuned model may perform more reliably within its target domain but may be less adaptable outside it, while a prompted general model is more flexible but harder to validate systematically [27].

A third adaptation strategy, increasingly common in clinical deployments, is retrieval-augmented generation (RAG), in which the model’s response is grounded by dynamically retrieving relevant external documents, such as clinical guidelines, formularies, or patient records, before generating output [28]. RAG reduces but does not eliminate the risk of the model generating plausible-sounding but incorrect information, a limitation examined in detail in Section 3.2. Together, these three approaches, fine-tuning, prompting, and RAG, represent the main technical pathways through which LLMs are being adapted for clinical use, and each carries distinct trade-offs for safety, reliability, and sustainability.

2.3. The Clinical Decision Support Landscape

LLMs are being applied across a range of clinical tasks that differ substantially in their cognitive demands, data requirements, and risk profiles. Mapping these tasks clearly before reviewing the evidence matters for two reasons. First, performance varies considerably across task types, and conflating them produces misleading conclusions about what LLMs can and cannot yet do in clinical settings. Second, the sustainability challenges examined in Section 4 play out differently depending on whether a model is helping a clinician draft a note or recommending a treatment for a critically ill patient.

Table 2 maps the principal clinical task areas to which LLMs have been applied. This review examines four of these in depth in Section 3: diagnostic reasoning, clinical documentation, medical question answering, and patient-facing applications. These four were selected because they represent the task areas with the strongest published evidence base and the most direct implications for clinical decision support at the point of care.

The tasks range from those where LLMs operate on structured, well-defined inputs to those requiring integration of ambiguous, real-world clinical data and multi-step reasoning. Diagnostic reasoning and treatment recommendation sit at the higher end of cognitive complexity and patient risk; clinical documentation and summarisation sit at the lower end of patient risk but carry their own hazards when errors propagate into the clinical record. Patient triage occupies a particularly sensitive position: errors in urgency classification carry direct consequences for patient outcomes, particularly in emergency and resource-limited settings [29,30]. Medical question answering spans both clinician-facing and patient-facing contexts, with different risk profiles for each. These distinctions are essential for interpreting the evidence in Section 3 and for understanding which deployment contexts demand the highest standard of validation before any LLM tool is considered ready for clinical use.

Table 2. Taxonomy of clinical decision support tasks addressed by large language models in the reviewed literature, with a representative published study for each task area. The final column indicates which task areas are examined in depth in Section 3; the remaining task areas are included to map the broader landscape but fall outside the scope of this review.

Task Area	Description	Representative Study	Reviewed in Section 3
Diagnostic reasoning	Generation of differential diagnoses from patient history, physical examination findings, and symptom presentation; identification of rare or atypical conditions.	Kanjee et al. (2023) evaluated GPT-4 on complex clinical cases from NEJM case records, reporting diagnostic accuracy comparable to experienced clinicians on structured vignettes [31].	Yes
Clinical documentation	Automated generation of clinical notes including SOAP notes, discharge summaries, referral letters, and operative reports from structured or unstructured clinical input.	Patel and Lam (2023) assessed ChatGPT for automated discharge summary generation, identifying both efficiency benefits and safety risks from hallucinated content [32].	Yes
Medical question answering	Responding to structured or open-ended clinical questions from physicians or patients; performance on standardised medical licensing examinations used as a common evaluation proxy.	Ayers et al. (2023) compared physician and ChatGPT responses to patient questions on a public health forum, finding LLM responses rated higher on quality and empathy in a blinded evaluation [12].	Yes
Patient-facing applications	Large language models deployed directly with patients including symptom checkers, mental health support chatbots, and health information tools; distinct risk profile from clinician-facing tools given direct access by potentially vulnerable users.	Hua et al. (2025) reviewed 160 studies of LLM-based mental health chatbots and found only 16% had undergone clinical efficacy testing [33].	Yes
Treatment recommendation	Guideline-aligned drug selection, dosing guidance, therapeutic sequencing, and management plan generation based on patient-specific clinical data.	Hager et al. (2024) tested state-of-the-art LLMs on 2400 real patient cases and found they did not accurately follow diagnostic or treatment guidelines [18].	No
Clinical summarisation	Compression of lengthy patient records, laboratory trends, radiology reports, or multi-visit clinical histories into concise and actionable summaries for clinical use.	Van Veen et al. (2024) found that adapted LLMs were rated equivalent or superior to medical expert summaries by ten physician reviewers in a blinded reader study [34].	No

Table 2. Taxonomy of clinical decision support tasks addressed by large language models in the reviewed literature, with a representative published study for each task area. The final column indicates which task areas are examined in depth in Section 3; the remaining task areas are included to map the broader landscape but fall outside the scope of this review.

Task Area	Description	Representative Study	Reviewed in Section 3
Diagnostic reasoning	Generation of differential diagnoses from patient history, physical examination findings, and symptom presentation; identification of rare or atypical conditions.	Kanjee et al. (2023) evaluated GPT-4 on complex clinical cases from NEJM case records, reporting diagnostic accuracy comparable to experienced clinicians on structured vignettes [31].	Yes
Clinical documentation	Automated generation of clinical notes including SOAP notes, discharge summaries, referral letters, and operative reports from structured or unstructured clinical input.	Patel and Lam (2023) assessed ChatGPT for automated discharge summary generation, identifying both efficiency benefits and safety risks from hallucinated content [32].	Yes
Medical question answering	Responding to structured or open-ended clinical questions from physicians or patients; performance on standardised medical licensing examinations used as a common evaluation proxy.	Ayers et al. (2023) compared physician and ChatGPT responses to patient questions on a public health forum, finding LLM responses rated higher on quality and empathy in a blinded evaluation [12].	Yes
Patient-facing applications	Large language models deployed directly with patients including symptom checkers, mental health support chatbots, and health information tools; distinct risk profile from clinician-facing tools given direct access by potentially vulnerable users.	Hua et al. (2025) reviewed 160 studies of LLM-based mental health chatbots and found only 16% had undergone clinical efficacy testing [33].	Yes
Treatment recommendation	Guideline-aligned drug selection, dosing guidance, therapeutic sequencing, and management plan generation based on patient-specific clinical data.	Hager et al. (2024) tested state-of-the-art LLMs on 2400 real patient cases and found they did not accurately follow diagnostic or treatment guidelines [18].	No
Clinical summarisation	Compression of lengthy patient records, laboratory trends, radiology reports, or multi-visit clinical histories into concise and actionable summaries for clinical use.	Van Veen et al. (2024) found that adapted LLMs were rated equivalent or superior to medical expert summaries by ten physician reviewers in a blinded reader study [34].	No

3. Large Language Model Performance in Clinical Decision Support

This section synthesises published evidence on LLM performance organised by the six clinical task areas defined in Section 2.3. For each task area the evidence is first summarised and then critically evaluated. The intention is not to catalogue every study but to assess what the available evidence actually demonstrates, identify where it is strong, and be honest about where it falls short.

3.1. Diagnostic Reasoning

The evidence on LLMs in diagnostic reasoning is, at first reading, striking. Kanjee and colleagues tested GPT-4 on a series of complex clinical cases drawn from the Massachusetts General Hospital case records and found diagnostic accuracy comparable to that of experienced clinicians on structured vignettes [31]. Cabral and colleagues carried out a more rigorous comparison: they recruited 21 attending physicians and 18 residents, each working through twenty sequential clinical cases using a validated clinical reasoning score (the revised-IDEA instrument), and found that GPT-4 outperformed both groups [35]. In a retrospective study of 100 real emergency department admissions, GPT-4 also outperformed resident physicians in diagnostic accuracy across cardiovascular, endocrine, and gastrointestinal presentations [36]. Among the proprietary models, Med-PaLM 2 achieved performance approaching specialist physician level on the MultiMedQA benchmark, and Med-Gemini subsequently set a new high of 91.1% accuracy on the MedQA (USMLE) task, representing the strongest reported performance on a standardised medical examination at the time of publication [8,15].

These results warrant careful interpretation for two reasons. First, many high-performing evaluations use published clinical cases that may already appear in LLM training data, artificially inflating apparent accuracy. Rutledge directly demonstrated this problem by comparing GPT-4’s performance on published versus unpublished challenging cases, finding that performance on published cases was substantially higher and recommending that published case evaluations be interpreted with caution [37]. The benchmark validity literature has reinforced this point, showing that real-world clinical cases contain a significantly larger volume of irrelevant and ambiguous detail than the curated vignettes most LLMs are tested on, and that LLMs are vulnerable to distraction by this noise [18]. Second, diagnostic performance is highly task-dependent. A 2025 evaluation using 1453 real patient cases found that while LLMs exceeded 85% accuracy on simple diagnostic tasks when sufficient examination data were provided, performance dropped substantially on examination recommendation tasks and further still on treatment planning, with the best models achieving only around 30% precision on treatment recommendations [38]. Critical reasoning steps were frequently missing even when factual accuracy was maintained.

The proprietary versus open-source performance gap, which was pronounced in earlier evaluations, is narrowing. In a real-world emergency department study using 2000 actual patient cases, open-source and closed-source LLM pipelines achieved comparable diagnostic match rates of 62% to 72%, with no statistically significant difference between model families [39]. Open-source models offered particular advantages for sourcing and interpretability, which are clinically relevant considerations independent of raw accuracy. The emerging competitiveness of open-source models matters for sustainability: local deployment of open-weight models can preserve patient privacy and reduce reliance on commercial infrastructure, two factors that are especially significant in low-resource clinical settings. The weight of evidence on diagnostic reasoning therefore supports a qualified position: LLMs can assist with generating differential diagnoses in structured settings, but their performance under real-world clinical conditions, with incomplete, messy, and longitudinal data, remains substantially less well evidenced than benchmark results suggest.

Beyond research settings, machine learning models applied to administrative claims data have demonstrated the potential for outcome prediction in real-world clinical populations, though performance disparities across demographic subgroups remain a consistent challenge [40]. In resource-limited settings, deep learning approaches have shown promise for automating diagnosis of high-burden diseases such as tuberculosis from chest radiographs, a task area directly relevant to health systems in low-and middle-income countries [41,42].

3.2. Clinical Documentation

Clinical documentation is the area where LLM deployment in real healthcare settings is currently most widespread. The appeal is direct: clinicians in many health systems spend more time on electronic health record tasks than on direct patient care, and the documentation burden is a documented driver of burnout [6]. Ambient AI scribes, which record and transcribe clinical encounters and generate structured notes automatically, represent the most commercially mature application of LLMs in medicine. Studies report that these tools can reduce documentation time by 20% to 30%, with one quality improvement study of 45 clinicians across 17 specialties finding a median reduction of 2.6 min per appointment and a 29.3% decrease in after-hours electronic health record work [43]. In summarisation tasks specifically, Van Veen and colleagues demonstrated that adapted LLMs matched or exceeded the quality of medical expert summaries in 81% of cases across four distinct clinical summarisation tasks including radiology reports, progress notes, and doctor-patient dialogue, as assessed by a blinded reader study of ten physicians [34]. ChatGPT has also been evaluated for automated production of patient clinic letters and discharge summaries, with studies identifying both efficiency gains and meaningful concerns about content accuracy [32,44].

The critical problem in clinical documentation is not that LLMs perform poorly on average but that their errors occur unpredictably and are difficult to detect without careful clinical review. Asgari and colleagues conducted the most rigorous error analysis to date, annotating 12,999 clinician-labelled sentences from LLM-generated clinical notes and identifying a hallucination rate of 1.47% and an omission rate of 3.45% [45]. These figures sound low but are clinically meaningful: in a note of 200 sentences, a 1.47% hallucination rate means approximately three fabricated statements may be present, any one of which could involve a medication dose, allergy, or clinical finding. Omissions are arguably more dangerous than hallucinations in documentation because they are absent from the note and therefore invisible to the reviewing clinician. A further risk specific to ambient scribing is speaker attribution error, where systems misattribute statements between clinician and patient in the transcript, with evidence showing significantly higher transcription error rates for speakers with non-standard accents, introducing a demographic bias into the documentation process before any LLM processing begins [46].

The pace of ambient AI scribe adoption across health systems has outrun both clinical validation and regulatory oversight [30]. Most published evidence comes from survey-based or single-site quality improvement studies rather than prospective controlled evaluations with patient outcome endpoints. LLM-generated notes also tend to be more verbose than clinician-authored notes, and there is early evidence of automation bias, where clinicians accept AI-generated content with reduced scrutiny when under time pressure, precisely the condition under which documentation tools are most likely to be used. The efficiency benefits of LLM-assisted documentation are real and potentially significant for clinician well-being; the evidence that this assistance is consistently safe at the scale of current deployment is not yet adequate.

3.3. Medical Question Answering

The numbers here are hard to ignore. GPT-3.5 passed the USMLE threshold for the first time in 2022 with an accuracy of around 60% on MedQA [47]. Within a year, GPT-4 had reached 86%, Med-PaLM 2 had matched it at 86.5%, and Med-Gemini pushed the figure further to 91.1% [15,48]. BioMedLM, at 2.7 billion parameters and trained exclusively on biomedical literature, achieved 50.3% on the same benchmark, a result that, given its much smaller scale, hinted at the potential for efficient domain-specific models as an alternative to infrastructure-heavy proprietary systems [13]. Figure 2 provides a visual comparison of these scores across the major models. The pace of improvement has been striking, and it has generated understandable enthusiasm about what LLMs might eventually be capable of in clinical settings.

Figure 2. Comparison of large language model accuracy on the MedQA (USMLE-style) benchmark across major proprietary and open-weight models. Scores reflect best-reported results from published studies at the time of model release. The dashed line indicates the approximate USMLE passing threshold (≈60%). See [9,13,15,47,48] for primary sources.

The problem is that these benchmarks are not what they appear to be. Passing the USMLE is a legitimate achievement, but it is a multiple-choice test of pattern recognition and recall over a fixed body of knowledge. Clinical competence is something different, it requires updating a diagnosis when new information changes the picture, communicating uncertainty honestly to a patient, managing ten competing clinical priorities simultaneously, and knowing when not to act. None of that is captured by selecting the correct answer from four options on a structured vignette. Agentically evaluated models, where the LLM must gather information through conversation rather than receive all case details upfront, show substantially lower accuracy: one study found GPT-4’s diagnostic accuracy dropped from 82% to 62.7% when the model had to ask questions to obtain information rather than being given it directly [49]. The field has largely failed to notice how wide this gap is because the benchmarks that dominate the literature are the ones that are easiest to run and most flattering to current models. A systematic review of 519 clinical AI studies found that 44.5% relied on medical licensing exam questions as their primary evaluation [17], and a review of 39 medical LLM benchmarks concluded that the majority still cannot reliably predict performance on real-world clinical tasks [19].

A practical problem compounds this: benchmark results are not reliably reproducible. Performance changes meaningfully across model versions, even when the model name does not change, and is sensitive to prompt wording in ways that developers do not always disclose. Rutledge documented this directly, showing that GPT-4 performed substantially better on published clinical cases that were likely in its training data than on cases that were not [37]. This means that a model’s headline score on a widely used benchmark may reflect what it has memorised rather than what it can reason about—a distinction that matters enormously in a clinical context and very little in a benchmark leaderboard.

None of this means the progress is illusory. A model that can accurately answer the majority of medical licensing questions clearly encodes a substantial amount of clinical knowledge, and that knowledge has genuine value when deployed appropriately. The issue is that the field has allowed exam performance to stand as a proxy for clinical readiness, and the two are not the same thing.

3.4. Patient-Facing Applications

When a patient types their symptoms into a chatbot at two in the morning, they are not interacting with a clinical decision support system in the way researchers and developers typically imagine it. They are probably anxious, possibly in pain, and almost certainly unaware of what the system they are using is actually capable of or where its limits are. This is the context in which LLMs are increasingly operating as patient-facing tools, and it is a context that the field has been slow to take seriously.

The range of applications is growing quickly. LLMs are now embedded in symptom checkers, mental health support chatbots, health information tools, and post-discharge patient communication systems. Ayers and colleagues demonstrated that ChatGPT responses to patient questions posted to a public health forum were rated higher in quality and empathy than physician responses by a blinded evaluation panel, a finding that generated considerable media coverage and pointed to genuine potential for LLMs in accessible health communication [12]. In mental health specifically, the global treatment gap is real: the majority of people with mental health conditions receive no care at all, and LLM-based chatbots have attracted interest as a scalable, low-cost bridge. A systematic review of 160 studies from 2020 to 2024 found that LLM-based chatbot architectures surged to 45% of new studies in 2024, reflecting rapid growth in research attention. However, only 16% of LLM-based chatbot studies had undergone clinical efficacy testing, with 77% still in early validation stages [33].

The safety concerns in this space are not hypothetical. Stanford researchers tested five therapy chatbots on vignettes containing explicit signals of suicidal ideation and found that models responded to a query about tall bridges in New York with bridge height information rather than crisis support, an example that is not a technical edge case but a basic failure of safety reasoning [50]. A 2025 Lancet commentary reported that dangerous interactions between LLM chatbots and users in crisis had already been documented, and that a comparison of three leading LLMs with expert suicidologists found inconsistent performance and an upward bias in judging their own responses as appropriate [51]. For adolescents, who are developmentally more susceptible to over-disclosure and who already use these tools extensively for emotional support, the risks are compounded by the absence of formal risk management protocols: a 2024 health technology audit found that fewer than 20% of AI-integrated mental health deployments had formal risk management plans addressing adolescent use [52].

There is also an important informed-consent concern. Patients may not know when AI systems are being used to support decisions about their care. Investigative reporting has shown that AI-enabled decision tools are often used in hospitals without explicit disclosure, and legal analyses suggest that current consent standards do not clearly require such disclosure. As a result, hospitals may have little practical incentive to seek patient consent for the use of these systems [53,54].

A large survey of 3000 US adults found that the overwhelming majority expected to be informed and involved in decisions about AI use in their care, suggesting that current opt-out approaches fall short of patient expectations [55]. In South Korea, a study of 1000 patients found that when AI is involved in clinical decision-making, patients rated information about that use as equally or more important than information about short-term treatment effects: in other words, they want to know [56]. The gap between what patients expect and what they currently receive is not a minor implementation detail. It is a fundamental breach of the transparency conditions that make clinical relationships trustworthy, and it matters directly for the sustainability of LLM deployment in patient-facing contexts over the long term.

Taken together, the evidence across these four task areas reveals a consistent pattern. LLMs perform most reliably when the task is structured, the input is clean, and performance is measured against a fixed reference standard. When those conditions do not hold (and in real clinical settings they rarely do), performance drops in ways that are hard to predict and harder to detect. A model that accurately answers standardised examination questions may fabricate a drug interaction in a clinical note. A system that handles common diagnostic vignettes confidently may produce unreliable outputs when the patient is elderly, speaks a language under-represented in training data, or presents with an atypical constellation of symptoms. The same tools that show promise in reducing documentation burden introduce a new category of clinical risk when their errors go unnoticed inside accepted notes. And the patients interacting directly with LLM-based tools are largely doing so without being informed that the system they are using is not a clinician. These are not incidental problems that better engineering will resolve in the next model release. They reflect structural conditions in how LLMs are currently built, evaluated, and deployed, conditions that go well beyond technical performance and raise questions about explainability, regulation, equity, and governance. It is those structural conditions that Section 4 examines.

4. Sustainability Challenges for Clinical Large Language Model Deployment

The evidence in Section 3 documents what LLMs can and cannot do across clinical task areas. This section addresses a different but related question: even where performance is acceptable, are the conditions in place for LLMs to be deployed responsibly, sustainably, and equitably over time? The answer, across five interconnected dimensions, is that they largely are not. The challenges described below are not arguments against LLMs in healthcare. They are arguments about what the field needs to prioritise before clinical AI can fulfill the promise suggested by its best benchmark results.

4.1. Explainability and Clinical Trust

When a clinician acts on an AI recommendation, they take on professional and legal responsibility for that decision. They cannot defend a clinical choice by pointing to a model’s output unless they can explain why that output was trustworthy. This is not a philosophical problem; it is a practical one that plays out every time a clinician must justify a decision to a colleague, a patient, or a review board. LLMs, as currently constructed, offer no reliable mechanism for that justification. They generate text that is plausible and often accurate, but the reasoning process that produced it is opaque by design. A model cannot tell a physician why it suggested one antibiotic over another in terms that connect to clinical evidence or patient-specific factors in an auditable way [57,58].

The main approaches available for explaining AI outputs in clinical settings (including SHapley Additive Explanations (SHAP), Local Interpretable Model-agnostic Explanations (LIME), and attention visualisation) were not designed for LLMs and do not translate cleanly to them. SHAP and LIME are post-hoc methods that generate approximate, local explanations after a prediction is made; they work reasonably well for tabular data in traditional ML models but struggle with the high-dimensional, sequential nature of LLM outputs [59,60]. Attention weights, sometimes offered as a window into what a model is “focusing on,” have been shown to be poor proxies for causal importance and are often misleading when used as clinical explanations [61]. Chain-of-thought prompting, which encourages models to show reasoning steps, produces text that looks like reasoning but is not a faithful representation of the underlying computation [62]. In short, the tools the field currently has for explaining LLM outputs are approximations at best, and in clinical contexts an approximate explanation can be more dangerous than no explanation, because it creates a false sense of transparency.

The regulatory environment is making this gap increasingly difficult to ignore. Under Article 13 of the EU AI Act (Regulation (EU) 2024/1689), high-risk AI systems (a category that explicitly encompasses clinical decision support tools) must be designed to ensure sufficient transparency for deployers to interpret system outputs and use them appropriately [63]. The Act became binding law in August 2024, with full application of its high-risk provisions phased in over the following 36 months. An LLM deployed in a clinical decision support role that cannot provide clinicians with an auditable, interpretable account of its outputs would not meet these requirements. For LLM developers targeting European health systems, explainability is no longer an aspirational design principle; it is a legal condition for deployment. Sustainable clinical AI requires meeting this condition in practice, not just on paper.

4.2. Regulatory Immaturity

As of December 2024, the US Food and Drug Administration (FDA) had cleared or approved 1016 AI and machine learning-enabled medical devices [64]. None of them is a large language model deployed for clinical decision support. This is not an oversight. It reflects a genuine regulatory classification problem that current frameworks have not resolved.

The FDA regulates AI in healthcare through its Software as a Medical Device (SaMD) pathway. Traditional clinical decision support tools (those that help clinicians make decisions based on patient data) are subject to premarket review when they meet device criteria. LLMs present a specific difficulty: their general-purpose nature means a single model can produce outputs that would qualify as a medical device in one interaction and general health information in the next, depending only on how the user phrases their question. A study testing GPT-4 and Llama 3 found that both models readily produced device-level clinical decision support outputs even when explicitly instructed to remain compliant with FDA guidance, leading the authors to conclude that new regulatory paradigms will be needed to constrain generative AI in clinical settings [65]. The FDA itself has acknowledged this challenge, noting plans to develop methods to identify and tag LLM-based devices in its authorized device database in a future update [64]. The 2024 finalization of Predetermined Change Control Plans for AI-enabled devices represents progress on managing adaptive algorithms, but these plans were designed for narrow ML systems and do not straightforwardly extend to models that can generate open-ended clinical text [66].

The European Union has moved further with the AI Act (Regulation (EU) 2024/1689), which entered into force in August 2024 and classifies clinical decision support systems as high-risk AI, mandating requirements around risk management, data governance, transparency, and human oversight [63]. This is a more explicit framework than the FDA’s current approach, but its full application to high-risk medical systems is phased in over 36 months, and specific guidance on how LLM-based clinical tools should meet conformity requirements has not yet been published. The UK’s Medicines and Healthcare Products Regulatory Agency (MHRA) operates a software and AI as a medical device framework that broadly parallels the FDA’s approach, with active work to harmonise with the EU position post-Brexit. Outside these three jurisdictions, regulatory frameworks for clinical AI are either nascent or absent, leaving health systems in most of the world operating in a governance vacuum as LLM tools enter clinical use through commercial channels with no formal oversight. Table 3 summarises the regulatory landscape across key jurisdictions.

Table 3. Global regulatory landscape for large language models in clinical decision support across selected jurisdictions. Gaps reflect the status as of the end of 2024.

Jurisdiction	Regulatory Body	Current Framework	LLM-Specific Gap
USA	FDA (CDRH)	SaMD pathway; AI/ML Action Plan (2021, updated 2024); Predetermined Change Control Plans finalized December 2024	No LLM cleared as a clinical decision support device; classification ambiguous for general-purpose models; open-ended text generation evades narrow device criteria
European Union	European Commission/ AI Office	EU AI Act (Reg. 2024/1689): clinical decision support classified as high-risk AI; Article 13 transparency requirements; phased implementation 2024–2026	LLM-specific conformity guidance not yet published; high-risk AI obligations apply from August 2026; interaction with EU Medical Devices Regulation unresolved
United Kingdom	MHRA	Software and AI as a Medical Device framework; joint Good Machine Learning Practice principles with FDA and Health Canada (2021)	Post-Brexit divergence from EU AI Act creates compliance complexity for pan-European developers; no LLM-specific guidance issued
China	NMPA	AI medical device regulation introduced 2021; requires clinical trial evidence for Class III AI medical devices	Limited international transparency on LLM evaluation standards; domestic models subject to separate Generative AI regulations (2023)
LMICs (most)	Variable/absent	No dedicated AI medical device regulatory framework in most low- and middle-income countries	LLM tools enter clinical use through commercial channels without formal safety assessment; no mandatory post-market monitoring

The core sustainability risk here is deployment speed outpacing regulatory readiness. LLM-based tools are already in clinical use through ambient scribing products, patient-facing chatbots, and EHR-integrated assistants, across geographies where no regulatory body has formally assessed their safety and effectiveness. Closing this gap requires not just adapting existing device frameworks but developing new evaluation paradigms that can handle the inherently general-purpose and probabilistic nature of large language models.

4.3. Equity and the Global Access Gap

Inequitable large language model performance across patient subgroups is first and foremost a clinical safety problem: a model that produces systematically higher error rates for patients of a particular age, sex, ethnicity, or language background is not equally safe for all patients, regardless of its aggregate accuracy score. The global access gap examined in this section is the structural condition that produces and entrenches this safety problem at scale, and it cannot be addressed without understanding the geographic and linguistic concentration of clinical large language model development.

The countries generating the most clinical LLM research are not the countries carrying the greatest burden of preventable illness and death. The USA, UK, and China together dominate the global landscape of AI health startups and published clinical AI studies, while low- and middle-income countries, which account for the vast majority of the world’s unmet health needs, contribute less than 1% of global generative AI usage in healthcare [67,68]. This geographic concentration is not a coincidence. Developing and deploying frontier LLMs requires infrastructure (servers, reliable electricity, broadband connectivity, and integrated electronic health record systems) that is simply absent in much of sub-Saharan Africa, South Asia, and rural settings across Latin America and Southeast Asia. Even where connectivity exists, the models themselves were largely trained on English-language clinical text from high-income health systems, meaning their outputs may reflect clinical norms, disease prevalences, and medication formularies that bear little resemblance to the settings where the global disease burden is heaviest. A model trained predominantly on data from academic medical centers in Boston or London is not a neutral clinical tool when deployed in a district hospital in Malawi or rural Indonesia [69].

The consequences of this mismatch go beyond performance degradation. Because most clinical LLMs are trained without meaningful representation of LMIC populations, languages, and health contexts, deploying them in those settings risks causing harm in ways that will not appear in any published evaluation. A model that confidently recommends a drug that is not available on the national formulary, or that fails to recognise disease presentations common in tropical settings, does not simply underperform; it can actively mislead a clinician who has no way of knowing that the tool was never designed for their context. There is also a structural risk that does not receive enough attention: AI investment in LMICs tends to arrive through external partnerships or commercial channels focused on scalable, revenue-generating applications, rather than through locally driven development that accounts for actual disease priorities and health system realities [67,70]. Figure 3 illustrates this mismatch between where clinical LLM development is concentrated and where the health need is greatest.

Figure 3. The global clinical LLM access gap: countries where large language model research and development is concentrated (predominantly USA, UK, EU, and China) bear a disproportionately low share of the global burden of preventable illness, while countries with the highest health need have the least access to validated clinical AI tools.

This gap is not hypothetical. Deep learning tools for tuberculosis detection, a disease whose burden falls overwhelmingly on low-income countries, have been developed and evaluated primarily using datasets from a narrow range of settings, with known limitations in generalisability to the diverse clinical and imaging environments of high-burden regions [41]. Demographic performance disparities have been documented even in well-designed clinical ML studies applied to real-world data [40], reinforcing that inequitable performance is not unique to LLMs but is a systemic pattern in clinical AI development.

These disparities connect directly to the United Nations Sustainable Development Goals. SDG 3, which commits to ensuring healthy lives and promoting well-being for all at all ages, cannot be achieved if the AI tools shaping the future of clinical care are designed for and validated in only a subset of the world’s health systems. SDG 10, which calls for reduced inequalities within and among countries, is actively undermined if clinical AI widens the performance gap between well-resourced and under-resourced health systems rather than narrowing it. A two-tier global health AI landscape (in which high-income settings benefit from well-validated, locally trained clinical LLMs while low-income settings receive either nothing or tools that do not work reliably for their populations) is not an abstract risk. It is the trajectory the field is currently on, and changing it requires deliberate choices about how clinical LLMs are built, for whom, and with whose data and expertise [4,20].

4.4. Computational and Environmental Sustainability

Computational and environmental costs are not direct metrics of clinical effectiveness; they are constraints on deployment feasibility and sustainability that affect which health systems can access clinical large language model tools and at what scale. This subsection addresses them briefly as supporting context rather than primary clinical arguments.

Training frontier LLMs demands substantial computational resources with measurable environmental consequences. Strubell and colleagues first quantified this systematically in 2019, calculating that training a large transformer model with neural architecture search could produce carbon emissions roughly equivalent to the lifetime mileage of five American cars [71]. GPT-3, with approximately 175 billion parameters, generated an estimated 552 metric tons of CO₂ equivalent during training [72]. Published estimates for GPT-4, which is considerably larger, suggest training costs of approximately 6900 to 15,000 metric tons of CO₂ equivalent, depending on the carbon intensity of the electricity grid used—a range that reflects both the scale of the undertaking and the degree to which training location influences environmental impact [73]. These are one-time training costs, but inference costs accumulate continuously at scale: estimates suggest that a single ChatGPT query produces approximately 4 g of CO₂ equivalent, roughly twenty times the footprint of a standard web search, and projections for GPT-4o’s annual inference in 2025 exceed the electricity consumption of 35,000 US homes [74].

For a paper concerned with sustainable healthcare AI, the honest position is that this is relevant but not yet decisive. The evidence base for net environmental impact (weighing the costs of training and inference against the carbon savings from reduced unnecessary clinical interventions, shorter hospital stays, or more efficient resource allocation at scale) remains genuinely underdeveloped. There are plausible mechanisms by which clinical AI could reduce the healthcare system’s environmental footprint: fewer repeat investigations, earlier accurate diagnoses, reduced length of stay. But these pathways have not been quantified in peer-reviewed studies with sufficient specificity to support strong claims. Two things can be said with confidence. First, the energy cost of clinical LLMs is real and should be factored into deployment decisions, particularly when choosing between lighter open-weight models and large proprietary systems for tasks where the performance gap is narrow. Second, for the global equity argument made in Section 4.3, the infrastructure demands of frontier LLMs (power, cooling, and hardware) are themselves a barrier to access in low-resource settings, meaning environmental cost and access cost are not separate concerns but the same concern expressed in different terms.

4.5. Governance, Accountability, and the Human-in-the-Loop Problem

Consider a clinical scenario that is not hypothetical in its structure, even if it has not been formally documented: a physician in a busy emergency department queries an LLM-integrated clinical decision support tool for guidance on a complex medication interaction. The tool returns a confident recommendation. The physician, managing multiple patients simultaneously, accepts it without reviewing the underlying evidence. The recommendation is incorrect. A patient is harmed. Who is responsible? The clinician, who accepted AI output without verification? The hospital, which deployed the tool without establishing a validation protocol? The developer, whose model generated a plausible-sounding error? Current legal and regulatory frameworks in most jurisdictions offer no clear answer to this question, and that ambiguity is itself a sustainability problem. A clinical AI ecosystem in which accountability for errors is undefined is one in which the incentives for rigorous pre-deployment validation are weak and the mechanisms for post-deployment learning from failures are absent [75,76].

The human-in-the-loop assumption that underpins most current deployment guidance deserves direct challenge. The premise is that clinicians will critically evaluate AI outputs before acting on them, and that this oversight relationship makes deployment safe even when models are imperfect. The evidence suggests this premise is optimistic in real clinical conditions. Automation bias (the tendency to accept AI recommendations without adequate critical scrutiny) as a clinically significant risk in AI-driven decision support, noting that it manifests both in errors of commission (accepting an incorrect recommendation) and errors of omission (failing to act because the system did not prompt action) [77]. A study of pathologists found that time pressure increased the severity of automation bias even when it did not increase its frequency, with incorrect AI output leading to pronounced performance declines under workload conditions [78]. In a randomised controlled trial of physicians who had completed formal AI literacy training, exposure to erroneous LLM recommendations still degraded diagnostic accuracy compared to exposure to correct AI advice, suggesting that even specifically trained clinicians are not reliably protected against over-reliance on LLM outputs when those outputs are wrong [79]. The mechanism is not irrationality. It is cognitive economics: in an environment of high patient volume, time pressure, and decision fatigue, accepting an AI recommendation that appears authoritative is the rational response to an unsustainable workload.

Sustainable deployment therefore cannot rely on individual clinician vigilance as its primary safety mechanism. It requires structural governance: pre-deployment validation standards that include prospective clinical testing, audit trails that log AI outputs and clinician responses for post-market monitoring, clear incident reporting pathways for AI-related adverse events, and defined accountability frameworks that specify which party bears responsibility when things go wrong. Article 14 of the EU AI Act mandates human oversight for high-risk AI systems, including requirements for trained oversight personnel and intervention protocols, but these provisions have not yet been translated into clinical-specific implementation guidance for LLMs [63]. The gap between regulatory intent and operational reality is where most deployed clinical LLMs currently sit. Closing it is not a technical problem. It is a governance problem, and governance problems require sustained institutional commitment, not model updates.

5. A Framework for Sustainable Clinical LLMs

The challenges identified across Section 3 and Section 4 are not independent problems requiring separate interventions. They share a common root: clinical LLMs are being developed and deployed within a framework designed for static, narrow AI tools, applied to systems that are general-purpose, probabilistic, and deployed in environments where errors carry direct consequences for patient safety. Addressing this requires more than incremental model improvements. It requires a deliberate reorientation of how clinical AI is built, evaluated, and governed. This section proposes a five-pillar framework for sustainable clinical LLM development and identifies the research priorities that the field needs to close the gaps this review has documented.

5.1. The Five-Pillar Framework

The framework presented here was derived directly from the evidence reviewed in Section 3 and Section 4 rather than constructed independently. Each pillar corresponds to a documented gap: where the evidence identified a pattern of failure, the corresponding pillar names the condition that would need to be in place for that failure to be addressed. Pillar 1 responds to the finding that fewer than 5% of clinical LLM studies used real patient care data [17], and that performance under real-world conditions diverges substantially from benchmark results Section 3. Pillar 2 responds to the finding that current post-hoc explainability methods are proxies that do not provide clinically actionable reasoning, and that the EU AI Act now mandates interpretability for high-risk clinical AI systems Section 4.1. Pillar 3 responds to the documented geographic and linguistic concentration of clinical LLM development and the evidence that fewer than 1% of generative AI usage in healthcare occurs in low-income countries Section 4.3. Pillar 4 responds to the finding that no large language model had received regulatory clearance as a clinical decision support device as of 2024, despite widespread commercial deployment Section 4.2. Pillar 5 responds to the evidence that automation bias persists even among specifically trained clinicians [79], and that accountability for AI-related clinical errors is currently undefined in most jurisdictions Section 4.5. Table 4 provides a concise summary of this derivation, mapping each documented limitation to its corresponding pillar and primary recommended action. The pillars are not sequential; a weakness in any one of them undermines the others, and together they describe the minimum conditions for responsible deployment at scale.

Table 4. Mapping of documented clinical large language model limitations to the five-pillar framework. Each pillar was derived from evidence reviewed in Section 3 and Section 4.

Documented Limitation	Evidence Source	Pillar	Primary Recommended Action
Benchmark-only evaluation; fewer than 5% of studies use real patient data	Section 3; [17]	1. Clinical Validation	Prospective multi-site trials with patient outcome endpoints; mandatory subgroup reporting
Post-hoc explainability methods are proxies; chain-of-thought reasoning is unfaithful	Section 4.1; [62]	2. Explainability by Design	Interpretable rationale generation built into model development; evaluate explanations for clinical utility
LLM development concentrated in high-income countries; fewer than 1% of usage in LMICs	Section 4.3; [68]	3. Equity-Centred Development	Diversify training data; audit subgroup performance; prioritise lightweight open-weight models
No LLM cleared as clinical decision support device despite widespread deployment	Section 4.2; [64]	4. Regulatory Readiness	Early regulatory engagement; apply medical device standards from development stage
Automation bias persists in AI-trained clinicians; accountability for errors undefined	Section 4.5; [77,79]	5. Governance and Accountability	Mandatory audit trails; incident reporting systems; explicit accountability frameworks

Pillar 1: Clinical Validation. The evidence reviewed in Section 3 showed that the field evaluates LLMs primarily on structured benchmarks that do not reflect the complexity of real clinical environments. Sustainable clinical AI requires prospective, multi-site validation studies with patient outcome endpoints, not physician satisfaction scores or benchmark accuracy rates. This means designing evaluation studies that include diverse patient populations, non-English clinical contexts, and settings that range from well-resourced academic centers to district hospitals in low-income countries. Developers, academic medical centers, and funders share responsibility for this shift. It requires investment in clinical trial infrastructure for AI, not just engineering resources for model development. Acceptable primary endpoints should be specified by task: diagnostic accuracy, time-to-diagnosis, and missed diagnosis rate for reasoning tasks; note error rate, omission rate, and clinician correction frequency for documentation tasks; and patient-reported outcomes, harmful advice rate, and appropriate escalation rate for patient-facing tools. Internal validation alone is insufficient; external validation on held-out datasets from non-development institutions is a minimum credibility requirement before deployment claims are made.

Pillar 2: Explainability by Design. Section 4.1 established that current post-hoc explanation methods are proxies at best and can create false confidence. Explainability in clinical LLMs cannot be an afterthought added during deployment, it must be designed into the model development pipeline from the outset. In practice, this means developing architectures that can generate clinically meaningful rationales alongside their outputs, investing in evaluation frameworks that measure whether explanations are actually useful to clinicians rather than just technically present, and treating the EU AI Act’s Article 13 transparency requirements as a minimum standard rather than a compliance ceiling. Measurable indicators for explainability should include: explanation faithfulness, assessed by testing whether the stated rationale predicts model behaviour under counterfactual inputs; evidence grounding rate, defined as the proportion of rationale claims traceable to specific clinical guidelines or cited evidence; and clinician decision quality with versus without explanation, evaluated through task-based experimental studies that compare diagnostic accuracy under both conditions. A clinically useful explanation is one that changes or confirms a clinician’s decision in a measurable and appropriate direction, not one that simply accompanies a model output.

Pillar 3: Equity-Centred Development. The geographic and linguistic concentration of clinical LLM development documented in Section 4.3 will not self-correct through market forces. Training datasets must be actively diversified to include clinical records from LMICs, non-English languages, and health system contexts that differ substantially from US academic medical centers. Model performance must be audited across demographic subgroups before deployment, and lightweight open-weight models that can be deployed locally without cloud dependency must be prioritised for low-resource settings. This is not a secondary concern, it is a prerequisite for the field’s relevance to global health. Fairness audits must specify subgroups in advance rather than post hoc. Minimum required subgroups include age group, sex, race and ethnicity, primary language, health literacy level, and geographic setting (urban versus rural; high-income versus low-income country). An acceptable performance gap threshold should be prespecified; as a reference standard, no subgroup should show a greater than ten percentage point difference in primary endpoint performance relative to overall model performance without triggering a mandatory investigation before deployment. This threshold is not arbitrary; it mirrors the disparity standard applied in fairness audits of clinical risk prediction models and is consistent with emerging regulatory guidance on algorithmic equity in healthcare AI [64].

Pillar 4: Regulatory Readiness. Section 4.2 documented that LLMs are entering clinical use faster than regulatory frameworks can assess them. Developers targeting clinical applications should engage with regulatory bodies early and design their systems to meet medical device standards from the outset rather than arguing after deployment that their tools fall outside device classification. Regulatory bodies in turn need to accelerate the development of evaluation frameworks specifically suited to general-purpose, generative AI systems—frameworks that go beyond static performance metrics to assess safety under real-world conditions, across diverse populations, and over time as models are updated.

Pillar 5: Governance and Accountability. Section 4.5 showed that the human-in-the-loop assumption does not hold under real clinical conditions and that accountability for AI-related errors is currently undefined. Sustainable deployment requires clear institutional governance: audit trails that log model outputs and clinical decisions, incident reporting systems that capture AI-related adverse events, and accountability frameworks that specify responsibility across the developer-deployer-clinician chain. This is institutional infrastructure work, not engineering work, and it requires investment from health systems and policymakers as much as from the AI development community.

Table 5 maps each pillar to the problem it addresses, the current gap, and the recommended actions, and Figure 4 provides a visual overview of how the five pillars interrelate and converge toward the shared goal of trustworthy clinical AI. Connecting this framework to the United Nations Sustainable Development Goals, Pillar 1 supports SDG 3 by ensuring tools genuinely improve health outcomes; Pillar 3 advances SDG 10 by reducing AI-driven health inequalities; and Pillars 4 and 5 reinforce SDG 16 by building strong, accountable governance institutions for health AI.

Table 5. Five-pillar framework for sustainable clinical large language model deployment. Each pillar addresses a structural sustainability challenge identified in Section 4 and maps to specific actions for developers, clinicians, and policymakers.

Pillar	Problem Addressed	Current Gap	Recommended Actions
1. Clinical Validation	Benchmark-only evaluation fails to predict real-world safety	Most evidence from curated vignettes and exam-based benchmarks; fewer than 5% of studies use real patient care data	Prospective multi-site trials with patient outcome endpoints; mandatory demographic reporting in all evaluation studies; dedicated funding for clinical AI trial infrastructure
2. Explainability by Design	Black-box outputs prevent clinician accountability and trust	Post-hoc XAI methods are proxies; chain-of-thought explanations are not faithful to model reasoning	Develop architectures with interpretable clinical rationale generation; evaluate explanations for clinical utility, not just technical presence; treat EU AI Act Art. 13 as minimum standard
3. Equity-Centred Development	Models trained on HIC data underperform in LMIC and non-English settings	Less than 1% of global generative AI usage in healthcare occurs in low-income countries; demographic auditing rare	Diversify training datasets by geography, language, and health system context; audit performance across subgroups before deployment; prioritise lightweight open-weight models for low-resource settings
4. Regulatory Readiness	LLMs deployed clinically before regulatory frameworks can assess them	No LLM cleared as a clinical decision support device by FDA; EU AI Act LLM-specific guidance not yet published	Early regulatory engagement by developers; accelerate evaluation frameworks for general-purpose generative AI; apply medical device standards from development, not post-deployment
5. Governance and Accountability	Accountability for AI-related errors is undefined; automation bias is underestimated	No standard audit trail or incident reporting system for clinical LLM deployments; liability frameworks absent in most jurisdictions	Mandatory audit trails for clinical AI outputs; incident reporting systems for AI-related adverse events; explicit accountability frameworks across developer, deployer, and clinician; fund AI governance infrastructure alongside model development

Figure 4. A five-pillar framework for sustainable clinical large language model deployment. Each pillar addresses one structural sustainability challenge identified in Section 4. The pillars are interdependent: a weakness in any one undermines the others, and the framework as a whole is directed toward the shared goal of trustworthy clinical AI that is safe, equitable, and governable across diverse global health settings.

5.2. Priority Research Directions

The framework in Section 5.1 describes what sustainable clinical LLM deployment requires. This subsection identifies the specific research gaps that must be closed to get there. The priorities listed here are not general calls for more work; each maps directly to a documented limitation in Section 3 and Section 4.

Prospective clinical trials with patient outcome endpoints. The most consequential methodological gap in the current literature is the near-total absence of prospective, controlled evaluations of large language models in clinical settings where the primary outcome is what happens to patients, not what physicians think about the tool or how it scores on a benchmark. Future trials should adopt multi-centre designs that include sites across different health system types, resource levels, and geographic regions, to establish whether findings generalise beyond the academic medical centres where most current evaluations are conducted. Primary endpoints should be specified in advance according to the clinical task being evaluated: for diagnostic support, appropriate endpoints include diagnostic accuracy, time-to-diagnosis, and rate of missed or delayed diagnoses; for documentation tools, relevant endpoints include clinician time savings, error rate in generated notes, and patient safety events attributable to documentation failures; for patient-facing applications, endpoints should include patient-reported outcomes, inappropriate escalation rates, and harm events. External validation on held-out datasets from institutions not involved in model development is a minimum requirement before deployment claims can be considered credible; internal validation alone is insufficient given the documented risk of training data contamination in published evaluations [37]. Subgroup analyses should be prespecified and must include demographic variables (age, sex, race and ethnicity, and socioeconomic status) as well as linguistic variables (primary language, health literacy level, and dialect or regional variation), since performance gaps across these groups have direct implications for clinical safety and equity [40]. Protocols should be preregistered, and all trials should report the model version, prompt template, and evaluation dataset composition in sufficient detail to permit independent replication.

Beyond specifying trial designs, the field needs clarity on what evaluation levels are required for clinical large language models, distinguishing between what is currently measured and what actually matters for patient care. At the model level, relevant metrics include diagnostic top-k accuracy, treatment guideline concordance rate, medication contraindication error rate, and hallucination rate. At the level of clinical decision quality, evaluation should assess whether the tool reduces missed diagnoses, inappropriate treatments, or documentation errors compared to standard care. Patient outcome metrics, including time-to-diagnosis, adverse event rates, readmission, and length of stay, represent the highest standard of evidence and should be primary endpoints in prospective trials wherever feasible. Workflow impact metrics, including clinician time savings, alert fatigue, and automation bias incidence, capture the real-world consequences of deployment that are invisible to benchmark evaluation. Safety metrics, including harmful recommendation rate, critical omission rate, and unsafe escalation failure rate, should be reported in all clinical evaluations regardless of task type. Finally, calibration, defined as the alignment between a model’s expressed confidence and its actual accuracy, is a prerequisite for safe clinical use: a model that does not know when it does not know is fundamentally unsuitable for clinical deployment regardless of its average accuracy score.

Clinical hallucination benchmarks for real-world complexity. Section 3.2 documented specific hallucination rates in clinical note generation, but these figures come from controlled settings and do not capture the full range of error types that arise when LLMs process the kind of long, ambiguous, multi-visit patient records that clinicians actually work with. The field needs hallucination benchmarks built from real electronic health record data (not curated vignettes) that classify errors by clinical consequence rather than by linguistic category. This is methodological infrastructure work that is unglamorous but essential. Fairness auditing standards specific to clinical LLMs. The evidence on demographic bias in clinical AI is growing, but there is no agreed standard for what a fairness audit of a clinical LLM should include, what subgroups it should examine, or what level of performance disparity is acceptable before deployment. The field needs to develop and adopt minimum reporting requirements for demographic performance data (analogous to the CONSORT guidelines for clinical trials) that apply before any clinical LLM is made available for use in patient care. These standards should be co-developed by clinicians, ethicists, and communities most affected by AI health inequities, not designed unilaterally by AI developers.

Lightweight open-weight models for low-resource settings. The global access gap documented in Section 4.3 will not close unless models designed for clinical use can be deployed locally, without cloud connectivity, in settings with limited infrastructure. This requires deliberate investment in smaller, efficient model architectures trained on locally representative data, as well as evaluation frameworks that test performance under the conditions of actual LMIC deployment (intermittent connectivity, non-standard EHR systems, and multilingual patient populations). Open-weight models that can be fine-tuned locally without proprietary data sharing offer the most tractable path to equitable clinical AI.

Longitudinal studies on automation bias and clinical skill. The studies cited in Section 4.5 examined automation bias over short experimental periods. What happens to clinical reasoning skills over months and years of routine LLM-assisted practice is unknown. There is a plausible risk that over time, clinicians who regularly accept AI recommendations without critical evaluation may develop degraded independent reasoning capacity in the task areas where they rely on LLM support. This is a patient safety concern that cannot be assessed by any current study design in the literature. Longitudinal cohort studies comparing clinical performance across groups with different levels of AI-assisted practice are needed. Minimum reporting standards for clinical LLM research. A recurring frustration in synthesising this literature is that studies do not report the information needed to evaluate or replicate them. Model version, exact prompt template, patient population demographics, exclusion criteria, and hallucination or error rates are frequently absent. The field should adopt minimum reporting standards for clinical LLM publications analogous to the TRIPOD+AI guidelines for clinical prediction models (specifying what must be reported about the model, the evaluation dataset, and the demographic composition of both). Without this, the evidence base will remain impossible to compare systematically, and the field will continue making progress that cannot be accumulated.

These six priorities are not exhaustive, but they represent the areas where research investment would have the highest leverage on the sustainability challenges this review has identified. Progress on any one of them would strengthen the case for the five-pillar framework proposed in Section 5.1. Progress on all of them is what responsible clinical AI development actually requires.

5.3. Limitations

This review has several limitations that should be acknowledged. As a narrative rather than systematic review, the literature search was structured but not exhaustive, and the selection of studies reflects the authors’ judgement about relevance and methodological quality rather than a prespecified protocol with formal quality appraisal. While the SANRA framework was followed to enhance transparency and rigour, the absence of a PRISMA-compliant selection process means that some relevant studies may not have been captured, and the possibility of selection bias cannot be excluded.

The evidence base itself carries limitations that propagate into this review. The majority of included studies were conducted in high-income, English-language settings, which constrains the generalisability of conclusions drawn about performance in low-resource or non-English clinical environments. The rapid pace of publication in this field means that some findings reported here may already have been superseded by newer model releases or evaluation studies published after the search cutoff of March 2025. Several key citations rely on preprint evidence that had not undergone peer review at the time of writing; these are identified in the reference list and should be interpreted with appropriate caution.

Finally, the five-pillar framework proposed in Section 5.1 is a conceptual contribution derived from narrative synthesis rather than a validated instrument. Its utility will depend on uptake by developers, clinicians, regulators, and funders, and empirical evaluation of whether applying the framework improves deployment outcomes is itself an important direction for future work, as detailed in Section 5.2.

6. Conclusions

Large language models have entered clinical medicine at a pace that has outrun the field’s capacity to evaluate them honestly. The evidence synthesised in this review confirms that several of the claims made on their behalf (that benchmark performance translates to clinical competence, that human oversight is a reliable safety net, that deployment can precede validation) do not hold up when examined against what actually happens when these systems are used in real clinical environments. Models that pass medical licensing examinations can fabricate drug interactions in clinical notes. Systems that perform well in controlled evaluations may fail systematically for patients from demographic groups not well represented in training data. Tools deployed at scale without governance frameworks leave no mechanism for detecting errors, learning from failures, or assigning accountability when patients are harmed.

This is not an argument against LLMs in healthcare. The potential is genuine. LLMs can reduce documentation burden, support clinical reasoning under information overload, and extend access to health information in settings that would otherwise have none. The argument is about sequence and conditions. Realising that potential requires the field to treat real-world sustainability (safety, explainability, equity, regulatory compliance, and governance) as the primary evaluation standard for clinical AI, not as desirable features to be added after deployment has begun.

The five-pillar framework proposed in Section 5 and the research agenda in Section 5.2 describe what this shift looks like in practice. For developers, it means building explainability and demographic fairness into model development rather than treating them as post-hoc concerns. For clinicians and health systems, it means demanding evidence of real-world validation before adopting LLM-based tools, and establishing governance infrastructure alongside deployment. For regulators, it means accelerating evaluation frameworks capable of handling general-purpose generative AI in high-stakes clinical context. For funders, it means directing investment toward prospective clinical trials, LMIC-appropriate model development, and governance research, not only toward benchmark improvement.

The world’s most pressing health challenges do not sit in the populations best represented in the training data of current clinical LLMs. Building AI that is genuinely useful across the full spectrum of global health need is the defining challenge for this field, and meeting it is inseparable from the broader mission of using artificial intelligence as a force for sustainable development. That is the standard this review has held the current evidence against, and it is the standard the field must now hold itself to.