Leave as Fast as You Can: Using Generative AI to Automate and Accelerate Hospital Discharge Reports

Abstract

Clinical documentation, particularly the hospital discharge report (HDR), is essential for ensuring continuity of care, yet its preparation is time-consuming and places a considerable clinical and administrative burden on healthcare professionals. Recent advancements in Generative Artificial Intelligence (GenAI) and the use of prompt engineering in large language models (LLMs) offer opportunities to automate parts of this process, improving efficiency and documentation quality while reducing administrative workload. This study aims to design a digital system based on LLMs capable of automatically generating HDRs using information from clinical course notes and emergency care reports. The system was developed through iterative cycles, integrating various instruction flows and evaluating five different LLMs combined with prompt engineering strategies and agent-based architectures. Throughout the development, more than 60 discharge reports were generated and assessed, leading to continuous system refinement. In the production phase, 40 pneumology discharge reports were produced, receiving positive feedback from physicians, with an average score of 2.9 out of 4, indicating the system’s usefulness, with only minor edits needed in most cases. The ongoing expansion of the system to additional services and its integration within a hospital electronic system highlights the potential of LLMs, when combined with effective prompt engineering and agent-based architectures, to generate high-quality medical content and provide meaningful support to healthcare professionals. Hospital discharge reports (HDRs) are pivotal for continuity of care but consume substantial clinician time. Generative AI systems based on large language models (LLMs) could streamline this process, provided they deliver accurate, multilingual, and workflow-compatible outputs. We pursued a three-stage, design-science approach. Proof-of-concept: five state-of-the-art LLMs were benchmarked with multi-agent prompting to produce sample HDRs and define the optimal agent structure. Prototype: 60 HDRs spanning six specialties were generated and compared with clinician originals using ROUGE with average scores compatible with specialized news summarizing models in Spanish and Catalan (lower scores). A qualitative audit of 27 HDR pairs showed recurrent divergences in medication dose (56%) and social context (52%). Pilot deployment: The AI-HDR service was embedded in the hospital’s electronic health record. In the pilot, 47 HDRs were autogenerated in real-world settings and reviewed by attending physicians. Missing information and factual errors were flagged in 53% and 47% of drafts, respectively, while written assessments diminished the importance of these errors. An LLM-driven, agent-orchestrated pipeline can safely draft real-world HDRs, cutting administrative overhead while achieving clinician-acceptable quality, not without errors that require human supervision. Future work should refine specialty-specific prompts to curb omissions, add temporal consistency checks to prevent outdated data propagation, and validate time savings and clinical impact in multi-center trials.

1. Introduction

1.1. Hospital Discharge Reports, a Necessary Resource Blackhole

Hospital discharge reports (also known as discharge summaries, HDRs in this paper) are vital for ensuring continuity of care when patients transition from hospital to home or other care settings [1]. They serve as the primary communication tool conveying a patient’s hospital course, diagnoses, treatments, and follow-up plan to the next healthcare providers or even as a base for continuity in care in transnational collaboration [1]. Studies show that having a discharge summary available at the first post-hospital visit can significantly impact patient outcomes [2,3]. For example, when a discharge summary was available to the outpatient physician, one study observed a trend toward 26% lower readmission risk [4].

The HDR is generated based on clinical course notes and emergency department reports. The clinical course compiles all information recorded by various physicians throughout the patient’s hospital stay, including progress notes, diagnostic test results, and clinical observations [1,4,5]. These data are contributed by multiple professionals, often in diverse styles and formats, resulting in lengthy and complex documents that can be challenging to manage. A multi-hospital study on heart failure patients exemplified this variability: even the highest-performing hospital’s discharge summaries were deemed insufficient in timeliness, transmission, and content, indicating room for improvement across the board [6]. Common deficiencies include missing key details (such as incomplete medication lists, absent test results, or vague follow-up instructions) and lack of clarity in the information provided [4]. The quality can depend on individual physician practices; without a standardized template, some may omit elements that others include. This inconsistency poses risks to patient safety and continuity [7], as the receiving provider cannot always count on a uniform or coherent set of data.

Beyond the quality challenges, preparing discharge summaries can be time-consuming, and inefficiencies in this process may lead to delays in patient discharge or communication gaps. The task of compiling all relevant information at discharge contributes to physician workload; one analysis noted that physicians spend roughly 2 h on documentation (like writing notes and summaries in the electronic health record) for every 1 h of direct patient care [8]. Furthermore, the administrative burden associated with manually preparing these documents can hinder workflow efficiency in clinical practice [9] and lead to burnout by clinicians [10]. Automating parts of this process can alleviate administrative workload, allowing clinicians to focus more on patient care [10]. A significant reduction in documentation time can enhance physician satisfaction and improve interactions with patients [11].

1.2. The Potential of Leveraging AI for HDR Generation

Digital technologies have been employed for some time to automate and enhance the accuracy and efficiency of clinical documentation [12] and clinical support systems [13]. Recent advancements in AI, particularly the development of large language models (LLMs), have shown the ability to generate human-like text and have been explored for clinical documentation purposes [13]. These models can assist in summarizing patient interactions and ensuring that critical clinical information is accurately captured and easily communicated [14]. Moreover, healthcare professionals consider these models especially useful in supportive roles, such as drafting preliminary documents, aiding in decision-making [15], or making information more understandable [14,16].

Nonetheless, automation of HDR generation has historically been challenging due to the necessary steps in the generation process: extraction and abstraction [8]. While extraction focuses more on gathering the existing data from the hospital stay and summarizing it, abstraction requires a more creative approach and capacity to make sense of the data to produce a coherent document. Most artificial intelligence cases before the popularization of GenAI focused on extraction—and did so without very promising results—leaving the professionals with the burden of adding the information into the HDR and producing the final document [8]. And even after the rise of GenAI popularity, up until recently, data intake constrains of GenAI models acted as a bottleneck in the development of automated clinical data processing systems and services [17].

Yet nowadays GenAI is particularly adept at processing larger amounts of text according to predefined guidelines and producing appealingly sound arguments, especially when given a very specific and uncreative task that involves a limited set of data, such as that of generating an HDR [8,18,19,20,21,22]. Given the possibility for a standardized structure of an HDR [1,23,24], their strong correlation with clinical progression, and the necessity of managing extensive textual data from medical records, modern GenAI with advanced reasoning capabilities is well suited to automating this essential yet labor-intensive task without risking quality.

Several studies have demonstrated the potential of GenAI in this domain. Janota and Janota [19] utilized GenAI to generate psychiatric discharge reports. A blind review conducted by professionals revealed that GPT-4 (one of OpenAI’s GenAI model) produced summaries that were more coherent and structured than those developed by clinicians. However, concerns regarding accuracy and reliability necessitated professional oversight. Similarly, Clough et al. [20] investigated the feasibility of AI-generated HDRs using ChatGPT (https://chatgpt.com/, accessed on 15 May 2025) (late 2023 to early 2024) on 25 synthetic cases. Their findings indicated that GPT-generated documentation was of equivalent quality to that produced by junior doctors [20]. Further improvements in HDR accuracy were achieved by Pal et al. through the use of fine-tuned models targeting specific report sections, demonstrating the feasibility of automatically generating precise HDRs from nurses’ notes when restricting GenAI involvement to selected parts of the report [5].

Although most research on GenAI in HDR generation focuses on assessing output quality and its potential to partially replace human labor, the underlying motivation for such implementations is the promise of increased resource efficiency [20]. In a case study on orthopedic discharge documents, researchers found that ChatGPT generated reports in one-tenth of the time required by a human while maintaining equivalent quality and exhibiting fewer hallucinations [21]. This significant reduction in time spent on documentation suggests notable economic savings while allowing healthcare professionals to dedicate more time to direct patient interactions.

Nevertheless, existing research is predominantly centered on English-language healthcare services. In many regions, English is neither the primary language of healthcare professionals nor of patients, creating potential disadvantages unless equivalent systems are developed for local languages. Some progress has been made in this area; for instance, as early as 2019, Chinese researchers generated HDRs in Chinese using AI with satisfactory results [25] and more recently researchers proved that generic GenAI services such as ChatGPT also provided close-to-human results in Italian HDRs [22]. However, challenges persist for minor languages. A Peruvian case study indicated that while Spanish-language AI services for HDRs may become feasible, implementation in indigenous languages such as Quechua and Aymara remains unlikely [16].

Finally, none of the identified studies examined the systemic integration of GenAI into hospital software, as most research has focused on feasibility assessments based primarily on GPT within ChatGPT’s interface and using synthetic data. As a result, no existing solutions fully address the outlined challenges in a comprehensive manner within electronic health record (EHR) systems while including privacy protections for real patients. Modern multi-agent approaches, which facilitate collaboration between multiple AI models while structuring data systematically, offer promising capabilities for automating HDR generation. Such advancements suggest that the potential for a fully integrated and effective AI-driven HDR system is substantial.

In this paper, the authors aim to showcase an example of the preclinical testing of a real-life implementation of such solutions where data protection, hallucination control, EHR integration, and multilingual support are considered. The case is sought to be in a real-world preclinical stage; as fully fledged rollout and validations are not yet completed, the solution is still under development, and it will take a few iterations and clinical validations before it is implemented system-wide and its benefits and handicaps can be systematically evaluated. Nonetheless, the tool design already aims towards a ready-to-use tool within the hospital’s EHR, hinting at how GenAI-generated HDRs could be integrated into the clinical workflow, and what iterative evaluation could look like as opposed to previous research that simply proves AI’s ability to perform the task through generic tools and interfaces, and leaving the prompting to the user.

2. Materials and Methods

2.1. Objective

The primary objective of this study is to enable an integrated GenAI solution to generate HDRs within a hospital’s infrastructure with the necessary safeguards and quality control mechanisms for deployment in an operative setting. This application should facilitate the writing process of HDRs by taking a digital clinical course and producing a properly formatted draft that only requires medical validation and minor editing.

This objective involves demonstrating that AI can generate a structured, preliminary, and high-quality hospital discharge summary based on clinicians’ records, which include the patient’s clinical course during hospitalization, results of complementary tests, and emergency department reports (which, as we outlined in the introduction, the literature demonstrates is possible) in a real-life setting and within the established clinical workflows. Such implementation shall also demonstrate that GenAI is suited to offer multilingual support in Catalan and Spanish. Additionally, this study assesses healthcare professionals’ satisfaction with the implementation of this technology.

2.2. Solution Development and Implementation

The widespread adoption of generic GenAI services has significantly lowered the barriers to AI products’ development, enabling rapid and cost-effective design, testing, and production [26,27]. A key factor in this transformation is the availability of base models as services, which shift development efforts from extensive research and technological innovation to a more streamlined process of customization, design, and iterative testing. This shift facilitates an agile design approach, allowing for the rapid iteration of proofs of concept and prototypes. Given the abductive nature of design processes [28], this methodology prioritizes experimental and exploratory approaches over structured research, aligning with grounded theory principles [29]. As a result, both the knowledge generated through this study and the AI-driven HDR service itself emerge as products of an iterative process in which experiments are quickly performed and afterwards evaluated to substantiate conclusions.

To evaluate the feasibility of this approach, the authors adopted an iterative, design-driven methodology [30], implementing the solution within a hospital’s information system for real-world testing. This environment closely resembles a living lab [31], where collaboration among the private sector, public sector, and academia encourages innovation beyond conventional laboratory settings, development, testing, and real-world experiences are prioritized over technical trials.

Following the design cycle—comprising research, ideation, testing, and evaluation—the project evolved through multiple iterations. It progressed from unstructured concept testing to the development of a functional prototype and, ultimately, to a deployable service in the phases shown in Figure 1. This paper presents the process and findings from this iterative development, including the proof of concept, prototype implementation, and an initial limited deployment within the hospital’s information system through the prototype.

2.2.1. Proof of Concept

The initial proof-of-concept phase aimed to identify the optimal prompting strategy and the most suitable LLM for generating HDRs. This phase employed an iterative, exploratory approach, consistent with design-science research methodologies, which emphasize rapid prototyping and iterative refinement to address complex, real-world problems [33]. The process involved testing a range of prompting strategies, from pattern-structured prompting configurations [34] to multi-agent frameworks tailored for specific sub-tasks in least-to-most and maieutic prompting approaches [35,36]. This approach aligns with the principles of design thinking, which advocate for flexibility, non-linear workflows, and iterative ideation to foster innovation [30].

The evaluation focused on generating three HDRs, each corresponding to a distinct medical specialty: cardiology, pediatrics, and general medicine. These specialties were selected based on input from participating clinicians, ensuring relevance and practical applicability. The selection of standard, commonly encountered discharge reports as test cases was informed by the need to establish a baseline for evaluating the system’s performance in routine clinical scenarios [37]. Each HDR generation trial utilized two primary inputs: an anonymized clinical course summary and a clinician-authored HDR, which served as a reference standard for comparison.

The primary objective of this phase was to generate AI-authored HDRs that comprehensively captured essential clinical information while minimizing inaccuracies and adhering to the established structural format defined by the hospital guidelines. Evaluation was conducted through a triangulated subjective assessment involving two developers and a designer [38,39], focusing on the efficiency and feasibility of the prompting strategies rather than exhaustive precision analysis for each trial. This approach was necessitated by the high volume of iterations, which precluded granular evaluation at this stage and the need to balance resources efficiency in development and knowledge production [33]. Subsequent phases (see Section 2.2.2) were planned to focus on system optimization, quality assessment, and formal validation.

Five LLMs were evaluated during this phase (at the date of autumn 2024): Gemini Pro 1.5 (Google, Mountain View, CA, USA), Claude 3.5 Sonnet (Antropic, San Francisco, CA, USA), GPT-4o (OpenAI, San Francisco, CA, USA), Mistral (Mistral, Paris, France), and Llama 3.1 (Meta, Meno Park, CA, USA). Model selection was integrated into the exploration of prompting strategies, guided by five key requirements derived from the clinical context and operational constraints:

• Multilingual support: Given the bilingual nature of the clinical setting (which might include documents in Catalan and Spanish, often mixed in the same clinical course), the system was optimized for both languages. Language-specific adjustments were implemented to ensure effective comprehension and accurate report generation.
• Perceived output quality: The variability in clinical courses and discharge reports necessitated a meticulous analysis to capture the full spectrum of linguistic and contextual nuances inherent in medical documentation. In this initial test, assessments were performed by the developers as medical professionals would intervene later on to improve results.
• Service stability: Continuous availability during testing was a critical operational requirement. Due to recurrent service interruptions, Claude was excluded from further consideration, underscoring the importance of reliability in deploying AI systems in clinical environments.
• Price: A good balance between cost and effectivity is key in LLM selection; if similar results are obtained with different costs, the cheaper model provides a competitive advantage, while if a high-cost model is perceived to perform more poorly than others, it shall be discarded. Price was calculated based on the average of multiple generations during the testing period.
• Generation time: While AI can generate documentation much faster than humans, considering the time spent by multiple LLMs is relevant in choosing a good balance between price, speed, and quality. Time was calculated based on the average of multiple generations during the testing period.

These necessary requirements enabled a more structured selection of LLM models, where some performed poorly in Catalan while others failed at times to fulfill the requests, failing to offer a stable service. This early-stage experimentation established a foundation for refining prompting strategies, paving the way for a more structured and rigorous evaluation framework.

2.2.2. Prototyping

The second phase focused on developing a functional prototype capable of generating a higher volume of HDRs beyond single-trial outputs. This advancement aimed to facilitate the model’s repetitive use within a system optimized for backend deployment. The primary objective was not only to scale the model but also to validate the quality of AI-generated HDRs and implement iterative improvements as necessary.

During this phase, 60 HDRs were generated across six medical specialties based on data provided by the hospital: cardiology, pediatrics, general medicine, pneumology, surgery, and gynecology. Of these, 27 were manually validated through qualitative comparative analysis between AI-generated and clinician-authored HDRs. This manual assessment involved a content comparison for each of the HDR requirements. Additionally, all generated HDRs underwent evaluation using the ROUGE test where they were compared to HDRs written by clinicians.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a widely used text evaluation metric in automatic translation and summarizing that assesses the quality of generated text by comparing it to a human-written reference [40,41]. This similarity assessment is based on word overlap (ROUGE-1), bigram overlap (ROUGE-2), and longest common subsequence matching (ROUGE-L) [40]. The ROUGE evaluation criteria included the following:

• Precision: The generated text is considered accurate if it contains only the original information without introducing extraneous content, that is, the amount of AI-generated text that is present in the reference document.
• Recall: A high recall score indicates that the generated text captures nearly all relevant content from the reference, that is, the amount of reference text that is present in the AI-generated text.
• F-score: This measure balances precision and recall, ensuring that the text is both comprehensive and of high quality.

ROUGE results were analyzed by averaging the scores across each specialty and language pair (combination of languages found in the clinical course data). This comparison was critical for refining the model’s design based on the structural and informational characteristics of HDRs within each specialty. F1-F2 scores are also provided as a form of balancing between F1 and F2 scores.

The combined use of ROUGE metrics and partial manual evaluation helped mitigate the expected limitations of ROUGE, particularly its reduced sensitivity to synonym usage and variations in phrase structure [42]. These linguistic variations are common in clinical documentation where differing style might favor more narrative or synthetic approaches to the writing, which can also include non-standardized abbreviations, making qualitative assessment a valuable complement to automated evaluation.

2.2.3. Limited Implementation in a Relevant Setting

The final phase of testing centered on deploying the AI-generated hospital discharge report (HDR) service in a real-world hospital environment. Designed for adaptability and scalability, the AI-HDR system accommodates the diverse technological infrastructures and organizational requirements of the Catalan healthcare network. Functioning as an external module, it seamlessly interfaces with hospital information systems. Upon clinician activation, the hospital software backend encrypts the clinical course and emergency report of the patient and transfers it securely to the AI-HDR tool. User interaction remains straightforward: from the ward’s bed management interface, the physician selects a patient and activates the AI-HDR function (“IAIA” in Catalan), triggering a report generation process that typically completes within about two minutes. The physician then reviews and, if necessary, edits the generated report before recording it in the existing hospital information system (Figure 2). Figure 3 shows an example of the user interface.

Following report generation, physicians evaluated the resulting HDR using binary answers to four parameters:

• Factual errors;
• Missing information;
• Writing/style issues;
• Correct (only minor edits required).

Additionally, clinicians were asked to provide a description of the issues they encountered with the report generation and the time they spent editing it. The four parameters were analyzed with system percentages given the small sample, and greater focus was given to the written evaluations as they provide greater information for system improvement. Unstructured written data were clustered into emerging groups and where possible groups that matched the original four-parameter classification. In total, six groups of comments emerged, covering factual errors, missing information, style issues, too lengthy or short sections, duplicated data, and outdated data. The number of HDRs with comments on each section is also reported with percentages of the analyzed reports. The differences between the self-reported parameters and those found in the written comments are discussed in Section 4.2. This pilot phase spanned over the whole month of February and partially during March 2025, during which 47 HDRs were generated through AI-HDR and assessed under these criteria. Researchers excluded time from the analysis as only a few clinicians reported their time spent on editing HDRs.

2.3. Ethical and Legal Framework

The implementation of the AI-generated HDR system in the public Catalan healthcare sector presents significant challenges in data protection, ethics, and security [43]. To ensure regulatory compliance, patient protection, and system integrity, key mechanisms have been established across four main areas [43]:

• Regulatory Compliance: AI HDR generation will adhere to European (GDPR), national (LOPDGDD and Law 41/2002), and regional (Decree 105/2000 and the CatSalut Security Plan) regulations [43]. This includes measures such as data anonymization, processing agreements, and security protocols.
• Technical Security: High-level protection mechanisms will be deployed, including advanced encryption (TLS 1.3 and AES-256), controlled processing environments, encryption of data at rest, and regular audits to ensure system robustness.
• Ethical Governance: A multidisciplinary committee will oversee the deployment of AI-generated HDRs to ensure alignment with bioethical principles, transparency in informed consent, and the prevention of algorithmic bias [43].
• Post-Market Surveillance: Continuous monitoring of system performance will be conducted through the collection of clinical data, incident management, and iterative reviews to enhance accuracy and adapt to evolving regulations [43].

This comprehensive approach ensures that AI-HDR is implemented with security, fairness, and transparency within the Catalan healthcare framework.

3. Results

3.1. Proof of Concept

The initial proof of concept consisted of identifying which tools performed better for the project requirements and could, therefore, deliver quality HDRs against the documentation written by clinicians.

Table 1. LLM model evaluation in early abductive trials. “+” and “-“ respond to a Harry’s profile perceptive qualitative evaluation of positive and negative points. Empty spots are left when there is no available data due to exclusion of the model before such requirement was evaluated.

Model	Google’s Gemini Pro 1.5	Anthropic’s Claude 3.5 Sonnet	OpenAI’s GPT-4o	Mistral	Llama 3.1
Summaries Performance	+++	++	+	+-	+-
Multilingual Support	Yes	Yes	No	No	No
Service Stability	Yes	No
Average Price (EUR)	0.25	0.22	0.42	0.22	0.30
Average Generation Time (s)	147	133	215	>300	200

shows the performance of each model for each selection criteria in order from most suitable to least. The models that were deemed not to perform well during iterative testing were not even considered for multilingual support and service stability, resulting in a lack of data.

3.2. Prototype

During the prototype validation phase, 60 HDRs were generated from six specialties combining multilingual inputs and outputs. The AI-generated outputs were evaluated against reports written by professionals through a ROGUE test. The results of the evaluation are presented in

Table 2. ROUGE results for the generation of HDR with AI in 6 disciplines.

Speciality	R1-Precision	R1-Recall	R1-F1	R2-Precision	R2-Recall	R2-F2	RL-Precision	RL-Recall	RL-F
Gynecology	0.16	0.49	0.24	0.06	0.19	0.09	0.08	0.26	0.12
Vascular surgery	0.18	0.49	0.25	0.07	0.19	0.10	0.10	0.28	0.15
Urology	0.22	0.57	0.31	0.11	0.27	0.15	0.14	0.36	0.19
Surgery	0.31	0.60	0.40	0.13	0.26	0.17	0.14	0.29	0.18
Cardiology	0.52	0.43	0.44	0.21	0.19	0.19	0.24	0.21	0.21
Pneumology	0.55	0.61	0.57	0.31	0.35	0.33	0.31	0.35	0.33

for each of the specialties and in

Table 3. ROUGE results for the generation of multilingual HDR with 3 language pairs in the clinical course and the HDR.

Language Pair	R1-Precision	R1-Recall	R1-F1	R2-Precision	R2-Recall	R2-F2	RL-Precision	RL-Recall	RL-F	Nº of Cases
es-es	0.39	0.57	0.43	0.20	0.28	0.22	0.22	0.33	0.24	35
ca-es	0.24	0.46	0.29	0.08	0.17	0.10	0.11	0.24	0.14	16
ca-ca	0.18	0.52	0.26	0.07	0.20	0.10	0.09	0.26	0.13	9

for each linguistic combination. Language pairs are the combination of languages found in the clinical documentation. All HDRs were generated in the same language as the reference HDR authored by clinicians for accurate evaluation.

General scores are generally higher for R1 (unigrams) than R2 (pairs) and RL (longest combination), which is perfectly normal behavior, as further discussed in detail in Section 4.1. In terms of specialties, pneumology scored remarkably high, while Gynecology and Vascular scores were rather low, but within an acceptable range. Further on, language pairs showed great disparities between different language combinations, yet beyond the Catalan–Spanish combination, single-language documentation and reports offered results within the expected ranges of specially trained summarization models in the respective language. This is again further on discussed in Section 4.1. It is worth highlighting the generally lower scores of precision metrics, especially when compared to the high scores for recall.

Due to the limited analysis possible from ROGUE scores, 27 of the 60 HDRs were further qualitatively analyzed in a comparative approach (between the human- and AI-generated HDR) following the contents of the HDR structure. Divergences were consistently found (

Table 4. Qualitative comparison of human- and AI-generated HDRs.

HDR Domain	Mismatched Cases	Patterns Observed
Admission chronology	12/27 (44%)	Date or level-of-care pathway (e.g., ED vs. ward) wrong or missing in AI report. Examples: Case 4 (01-06 vs. 31-05), Case 5 (04-05 vs. 03-05), Case 6 (three different dates documented).
Discharge date	6/27 (22%)	AI usually matched; human had the occasional extra day noted (e.g., Case 11).
Social/baseline context	14/27 (52%)	Human versions added living situation, occupation, or functional status; AI also included some details, but not always the same as the clinician and omitted information (e.g., Case 6 and Case 2), sometimes not found in the course documentation.
Diagnostic labels	11/27 (41%)	Humans frequently added comorbid or situational diagnoses (e.g., SARS-CoV-2, acidosis, and psychiatric history) that AI left out or phrased more broadly (e.g., Cases 3, 14, and 25).
Medication and dosage	15/27 (56%)	Divergences in dose (amoxicillin 1 g q8h vs. 500 mg q12h), omission of gastro-protection, or additional heparin/NSAIDs only in human report (Cases 4, 7, 10).
Investigations	10/27 (37%)	AI gave a list without dates/values, human-specified results, or added studies (extra ECG and sensitivity panel) (Cases 6, 10, 22).
Follow-up plans	12/27 (44%)	Human follow-ups were generally vague while AI added additional details, in some cases hallucinated. (e.g., Case 6, 25)

), although HDR pairs were overall highly aligned. Of special relevance in the analysis is the high number of discrepancies (generally errors but not necessary in all cases) in the medication and dosage and in the social and baseline context, which favored human-made reports where data unavailable in the clinical course were found.

3.3. Limited Implementation in a Relevant Setting

The limited implementation in a relevant setting followed a progressive implementation of HDRs in the disciplines of pneumology, surgery, and early testing in Pediatrics. Professionals from the participating hospital in this study were given access to the tool within the system and prompted with a survey for result evaluation (

Table 5. HDR flagged by professionals on each of the expected issues an AI HADR could contain.

Score	Reports (n)	Percentage of HDR
Factual errors	22	46.8%
Missing information	25	53.2%
Writing/style issues	13	27.7%
Correct (only minor edits needed)	2	4.3%

). Throughout February and March 2025, a total of 47 HDRs were generated. In the evaluation survey, more than half (53.2%) of reports were reported to have missing information, while slightly less than half had some sort of factual errors. Only two reports (4.3%) were correct enough to not include factual errors, missing information, or style issues. It is worth noting that an HDR could be flagged with multiple issues as they are not mutually exclusive—except for the reports that were correct. It is worth noting that these HDRs were never compared to a clinician-written HDR as professionals were using the AI tool to generate the HDR.

Results were slightly different for the qualitative analysis of the comments made by professionals during evaluation (

Table 6. Qualitative analysis of medical professional comments on the AI-generated HDR.

Score	Reports (n)	Percentage of HDR
Factual errors	9	19.1%
Missing information	22	46.8%
Lengthy or too short evolution	6	12.8%
Writing/style issues	5	10.6%
Duplicated data	3	6.4%
Outdated data	1	2.1%

). In comments, they reported a far lower number of factual errors (19.1%) and 46.8% of cases with missing information. On the other hand, they reported additional issues such as too lengthy or too short explanations of patients’ evolution (12.8%), outdated data presented as current (2.1%), and duplicated data throughout the HDR (6.4%).

4. Discussion

This study demonstrates the feasibility of applying Generative AI models to the automatic generation of HDRs with success in both lab and real-world settings. While the results do not demonstrate that the service is ready for deployment to replace clinician’s tasks—which was never expected—they offer a sound base to accelerate professionals’ workflows. We wish to illustrate some limits to the analysis methodologies and prove that although results might appear significantly negative, the tool is perceived to be beneficial in the professional workflows.

4.1. Analysis of ROUGE Evaluation and Its Limitations

The ROUGE family of metrics—unigram overlap (ROUGE-1), bigram overlap (ROUGE-2), and longest common subsequence (ROUGE-L)—was designed to quantify lexical similarity between a candidate summary and a human reference [42]. In our multilingual clinical setting, this yields a quick, reproducible proxy for content fidelity, yet it also exposes blind spots:

• Stylistic heterogeneity in clinical notes: Clinical course notes oscillate between terse, checklist-like phrasing and richer narrative prose, often within the same service; these affect the AI-generated contents, while humans have better training at extracting the actually useful information and standardizing its format in a concise manner. Moreover, each physician has their own preferences when it comes to HDR writing styles, which affects the reference documents against which the AI-generated reports are compared. When the reference is highly narrative and the AI output is concise, recall drops and precision rises; the reverse occurs when the AI is more expansive than the reference. Our results with overall higher scores on recall suggest that reference texts are more succinct while AI-generated texts are more detailed in the explanations. This analysis is in fact substantiated by the qualitative evaluation of 27 HDRs performed along with the ROUGE tests. Generally, doctors’ HDRs were more telegraphic and factual, whereas the model tended to elaborate, especially around the daily evolution narrative of the patient, which is an expected behavior of general LLM models when they are provided with superfluous information—as can be the case in hospital clinical courses.
• Semantic adequacy beyond surface overlap: As ROUGE compares groups of letters (tokens) rather than factual information, harmless synonyms or equivalent expressions (abbreviations or differing descriptive wording) are penalized as false positives/negatives. Complementary semantic metrics such as BERTScore [44] or domain-adapted COMET [45] are recommended for future iterations, and the multilingual FRESA framework has already demonstrated a better correlation with human judgment in Spanish and Catalan [46]. In our case, we opted for a complementary human qualitative review.

Regardless of the mentioned limitations, ROUGE scores suggest positive results when benchmarked against other ROUGE evaluations in prose texts such as news. The PEGASUS model was developed with a corpus of text from CNN/Daily Mail and in English news summarization offers similar scores to our es-es (Spanish) results [47]. The PEGASUS model scored R1-F1 0.44, R2-F 0.22, and RL-F 0.41 [47], while our implementation scored R1-F1 0.43, R2-F 0.22, and RL-F 0.24. This comparison suggests that our performance is aligned with what can be expected from more specialized models. Even when it comes to the ca-ca pairs, although scores are considerable lower, they are also aligned with what can be expected for specialized models in Catalan [48]. Ahuir et al. reported results of between 0.26 and 0.29 for R1-F1, 0.10 and 0.12 for R2-F, and 0.20 and 0.23 for RL-F [48]. To what degree the lower scores for Catalan are due to language structure or lack of training data for the models is unclear to the researchers.

On the other hand, when it comes to the different disciplines, lower ROUGE scores for gynecology and vascular cardiology stem from depressed precision (i.e., the system adds terminology absent from the clinician’s HDR). This hints at discipline-specific narrative conventions: certain services favor tightly templated HDRs, so a general LLM with prompt structures designed for mixed-discipline notes may “over-generate”. Further analysis on differing HDR structures for these disciplines would be required as results are on the lower end of what is perceived as acceptable; rather, a custom processing flow with specific prompts targeting their specific HDR structures should attenuate this mismatch.

In sum, while ROUGE metrics provide a valuable baseline for evaluating content fidelity, they are insufficient on their own. The results, although generally satisfactory, leave room for interpretive ambiguity and require triangulation with qualitative analyses to ensure the accuracy and contextual appropriateness of AI-generated clinical documentation.

4.2. Context of Qualitative Evaluations

The quantitative analyses presented earlier—particularly the ROUGE-based comparison of 60 AI-generated HDRs with clinician originals—show that lexical overlap is, at best, an incomplete proxy for clinical adequacy. This limitation became clear once we submitted a purposive subsample of 27 HDR pairs to close qualitative scrutiny and complemented that exercise with bedside feedback on 47 prospectively generated drafts. The richer evaluation produced two convergent insights: first, that both LLMs and clinicians can introduce facts that have no textual antecedent in the clinical course (“phantom data”); second, that LLMs reproduce values that have been explicitly corrected later in the record, thereby perpetuating errors. Both phenomena explain why clinicians, despite recognizing the productivity gain afforded by the tool, still flagged more than half of the drafts for missing or incorrect information during routine use (53% and 47%, respectively).

The comparative analysis during the prototyping phase aligned each AI draft with its human reference along the standard HDR sections—admission chronology, diagnostics, social context, medications, investigations, and follow-up plan. Divergences were ubiquitous, affecting at least one domain in every pair; the most frequent mismatches concerned social or baseline context (52%), medication details (56%), and admission chronology (44%). In parallel, free-text comments written by ward physicians during a one-month limited deployment revealed a somewhat different hierarchy of concerns: only one-fifth of reports were criticized for factual errors, whereas nearly half were said to lack pertinent information; additional complaints targeted excessive or insufficient narrative length, data duplication, and the presence of outdated values. That is not to say that clinicians misreported factual errors or missing data, rather that the combination of evaluations provides a better picture and that, possibly, missing data or errors were not “critical”, although further analysis would be required to validate this hypothesis. In fact, missing data are not necessarily bad per se or an error.

A striking proportion of human–AI discrepancies arose because professionals added information in the report that was not present in the clinical course, as some authors also noted [49,50], while the AI also added hallucinations, flagging them (at times) for further revision by the professional. These insertions most often concerned social circumstances (e.g., living arrangements and occupational status) that clinicians traditionally dictate from memory or consult in ancillary systems not included in the prompt. From a technical perspective, the model behaves consistently with what is expected from the HDR structure: it predicts that a statement about family context or comorbidity should follow, and therefore fabricates one when the input is silent or makes its own decision on what data are relevant, which is not necessarily aligned with that of the clinician. For example, on a report, the clinician reported the patient to live in a rural area with two cats, while the LLM reported the user to live in a flat without an elevator. While neither were untrue, lack of data in the clinical report and differing assessments generated a discrepancy in the report.

The second recurrent pattern concerned the treatment of explicit corrections in the course. The following example illustrates the mechanism: an early entry listed a weight trajectory, but a note written 25 min later read “Correction: weight is from another patient”. Because the prototype concatenates all daily notes into a single text block, temporal relationships might be missed by the LLM, which copied the obsolete weight into the discharge summary. Similar cascades were documented for antibiotic doses and date-of-admission fields, producing factual errors in the documentation. In our audit, 44% of HDR pairs disagreed on admission chronology and 56% on medication doses, proportions that mirror the frequency of free-text complaints of outdated or duplicated data during bedside use. These propagated errors are particularly insidious because they appear authoritative—the authoring clinician has to be fully aware of the course contents in order to identify these misses as without that information one would not suspect the error.

Two remedial avenues follow directly from these findings. First, prompts should instruct the model to surface uncertainty by leaving placeholders—e.g., “[social context not documented]”—instead of inventing information. The use of agents can be revised to improve the ability to check for factual errors and amend them. Alternatively, the pipeline can be broadened to ingest structured demographics and nursing hand-over notes, bringing the model’s context in line with clinicians’ cognitive workspace. Second, a lightweight information-extraction agent should precede generation, tagging entities with timestamps and detecting corrective phrases (regular expressions for “correction”, “error”, etc.). Such pre-processing would enable the model to privilege the latest mention of a variable, which could flag physically implausible outputs before the draft reaches the editor.

The qualitative strand of this study exposes failure modes that evade standard summarization metrics yet carry significant clinical risk. Nonetheless, it is important to contextualize these errors in the stage of development of the tool and the expected outcome by the clinicians whom in conversations were generally positive regarding the use of the tool. In fact, beyond the specific reported discrepancies the general outcome was of high quality, in many cases more precise than that of the clinicians. From an experiential point of view, the use of the tool reduces their task to fact checking a report and performing some modifications instead of having to study the clinical course and craft a new HDR from scratch.

4.3. Overall Study Limitations and Expansion of the Research

In addition to the specific evaluation methods employed in the advanced stages of service development, several broader limitations emerge from the overall design of this case study. Firstly, the initial design process was inherently iterative and somewhat unstructured—as is often characteristic of early-stage design—which, while beneficial for rapid prototyping and refinement, limited the capacity to objectively determine the most appropriate model. Model selection might have been strengthened through a more detailed, case-by-case qualitative analysis of output quality and multilingual capabilities. However, such an approach was complicated by the evolving nature of prompt structures and agent configurations throughout this study. The system’s progression from a focus on prompt engineering to the integration of agent-based architectures enhanced both output quality and adaptability, yet introduced additional complexity to comparative evaluations.

Advanced prompting strategies—including few-shot learning, chain-of-thought reasoning, and domain-specific knowledge integration—proved useful in identifying effective methodologies for clinical documentation generation. Still, more formalized comparative approaches, such as model pair comparisons inspired by A/B testing, could have yielded more reliable insights. These methods could also be valuable during later stages of deployment, where clinical staff could contribute systematically to iterative service improvements, especially as underlying technologies continue to evolve.

Additionally, the inclusion of metrics such as the number of successful outputs and completed tests would have allowed for a more nuanced and objective assessment of system stability. Expanding the range of models tested, including those suitable for local deployment, may also have been advantageous. While local deployment offers greater institutional sovereignty, the choice to offload AI services to third-party providers in early stages was primarily driven by agility, faster access to model updates, and cost efficiency [51,52]. Nevertheless, fine-tuned small language models, although requiring higher upfront investment in infrastructure and training, could potentially offer comparable or superior performance with reduced long-term operational costs [51,52].

Economic feasibility remains a critical constraint, encompassing costs associated with model usage, server infrastructure, licensing, processing time, post-deployment maintenance, and marketing. A comparative analysis of these expenditures against the costs of human-generated clinical documentation—and the corresponding benefits of freeing clinician time—would enhance the understanding of long-term viability.

This study’s limitations are further evident in its small-scale deployment, involving a single hospital and a limited number of reports per phase. The true effectiveness and scalability of the system will require evaluation across multiple sites and disciplines. Future research should address the challenge of limited clinician participation in system evaluation, despite widespread use and apparent time savings. Even simplified evaluation instruments, such as the four-parameter tool used in this study, encountered participation barriers. For instance, data on the time clinicians spent editing AI-generated hospital discharge reports (HDRs)—though explicitly requested—were rarely submitted. To address this in future large-scale implementations, hospital systems should incorporate automated tracking mechanisms to log the time elapsed between the generation of an AI output and the clinician’s final upload to the electronic health record.

5. Conclusions

The AI-HDR prototype (IAIA in Catalan) illustrates the potential for improving healthcare professionals’ efficiency by automating bureaucratic processes and partially offloading routine documentation tasks to AI systems. This automation could potentially free up time for higher-value clinical activities, although the present study does not assess the direct benefits of AI-generated HDRs. The project’s development required approval and collaboration from three key hospital stakeholder groups, administration, digital system managers, and, most critically, medical professionals, whose acceptance was essential not only for data acquisition but also for its evaluation, practical implementation in real-world settings, and ethical approval.

Despite promising results, several limitations indicate areas for future research. First, adapting the system to languages beyond Catalan and Spanish may pose challenges, particularly in maintaining terminological accuracy. Even within the studied languages, performance assessments revealed a decline in quality when processing Catalan-language documentation, highlighting the need for more robust multilingual solutions. Second, the considerable variability in the format and style of clinical courses and discharge reports across medical specialties suggests that an initial focus on a single specialty may be necessary to optimize system performance. Additionally, the choice of LLM could influence the approach’s generalizability to different healthcare settings. Future work should also prioritize full integration with electronic health records (EHRs) and conduct pilot studies across multiple clinical environments to evaluate the system’s practical impact and reliability.

In conclusion, this study highlights the potential of Generative AI and LLMs as valuable tools for supporting clinical documentation. While large-scale implementation in hospitals is feasible, it requires a tailored approach for each medical specialty to ensure effective HDR design. This research lays the foundation for future advancements in healthcare management, contributing to the optimization of medical workflows and improved efficiency in clinical settings.