Fine-Tuning LLaMA2 for Summarizing Discharge Notes: Evaluating the Role of Highlighted Information

Abstract

This study investigates whether incorporating highlighted information in discharge notes improves the quality of the summaries generated by Large Language Models (LLMs). Specifically, it evaluates the effect of using highlighted versus unhighlighted inputs for fine-tuning LLaMA2-13B model for summarization tasks. We fine-tuned LlaMA2-13B in two variants using MIMIC-IV-Ext-BHC dataset: one variant fine-tuned with the highlighted discharge notes (H-LLaMA), and the other on the same set of notes without highlighting (U-LLaMA). Highlighting was performed automatically using a Cardiology Interface Terminology (CIT) presented in our previous work. H-LLaMA and U-LLaMA were evaluated on a randomly selected test set of 100 discharge notes using multiple metrics (including BERTScore, ROUGE-L, BLEU, and SummaC_CONV). Additionally, LLM-based judgment via ChatGPT-4o rated coherence, fluency, conciseness, and correctness, alongside a manual completeness evaluation on a random sample of 40 notes. H-LLaMA consistently outperformed U-LLaMA across all metrics. H-summaries, generated using H-LLaMA, in comparison to U-summaries, generated using U-LLaMA, achieved higher BERTScore (63.75 vs. 59.61), ROUGE-L (23.43 vs. 21.82), BLEU (10.4 vs. 8.41), and SummaC_CONV (67.7 vs. 40.2). Manual review also showed improved completeness for H-summaries (54.8% vs. 47.6%). All improvements were statistically significant (p < 0.05). Moreover, LLM-based evaluation indicated higher average ratings across coherence, correctness, and conciseness.

1. Introduction

Electronic Health Records (EHRs) contain a wide range of clinical information, including details of a patient’s hospital stay, progress notes, medical history, medications, vital signs, and diagnostic reports [1]. The exponential growth of EHRs has revolutionized medical data management, enhancing access to patient records, interoperability, and aiding healthcare professionals in making informed decisions [2,3].

However, EHRs are often perceived as cluttered and difficult to navigate, which can significantly hinder healthcare providers’ ability to efficiently extract relevant insights, potentially impacting clinical decision-making and patient safety [4,5]. Several factors contribute to this complexity, including the vast amount of abbreviations and medical jargon, variability in how different healthcare providers document data in EHRs, copy–pasting practices that clutter records with redundant information, and the design of EHR systems themselves.

Summarization of EHRs can solve this problem to a great extent by enabling healthcare providers to quickly access the most relevant information, reducing errors and supporting informed clinical decisions [2,6].

Given the complexity and volume of EHR data, automated summarization methods have become increasingly important. There has been a notable shift from traditional text summarization methods to techniques that use Large Language Models (LLMs) [7,8,9]. These models have evolved from pre-training and fine-tuning approaches to prompt-based methods [10,11,12]. While LLMs demonstrate strong potential in summarization tasks, several challenges persist. One critical issue is “hallucination,” where LLMs generate plausible but factually incorrect information. In the healthcare domain, such inaccuracies can lead to misdiagnoses, inappropriate treatment plans, or worse [13,14]. Additionally, LLMs may overlook crucial clinical details, potentially compromising patient care [15].

Fine-tuning LLMs for EHR summarization offers a promising solution to mitigating these challenges by enhancing its understanding of medical terminology and context [16,17,18]. Fine-tuning is a process that allows LLMs to adapt their general knowledge to specific domains by training them on smaller, domain-relevant datasets, enabling them to learn task-specific patterns and terminology. During fine-tuning, the model’s pre-trained weights are updated to optimize its performance for the target task, allowing it to learn task-specific features and improve accuracy [19,20].

By adapting LLMs to healthcare-specific datasets, fine-tuned models can better grasp domain-specific terminologies and contextual nuances, leading to improved performance in summarization tasks [21]. Moreover, fine-tuning can reduce hallucinations by reinforcing factual content through curated training data [17,18,22,23]. This approach also ensures the inclusion of essential clinical details that might otherwise be overlooked in generic LLM outputs.

The process of fine-tuning LLMs for EHR summarization relies on high-quality datasets [24]. Studies have shown that training models on domain-specific data significantly enhances their ability to interpret and summarize clinical narratives effectively [25,26]. The quality of the dataset used for fine-tuning directly impacts the model’s summarization performance [27].

The MIMIC-IV-BHC dataset, a curated collection of discharge notes that are a key component of EHRs, and corresponding Brief Hospital Course (BHC) summaries, has been introduced [28,29]. This study benchmarked five models (GPT-3.5, GPT-4, Clinical-T5-Large, Llama2-13B, and FLAN-UL2) using both prompting-based and fine-tuning adaptation strategies. The reported results demonstrate that fine-tuned Llama2-13B achieved the highest quantitative scores (BLEU, BERT-Score, ROUGE-L).

However, although fine-tuning can help reduce the frequency of hallucinations by training the model on domain-specific data, it does not entirely eliminate the risk of generating inaccurate information, as the underlying architecture of LLMs can still lead to hallucinations [30]. These studies [30,31] emphasize that while fine-tuning LLMs can improve their performance, there are still challenges, including the potential loss of previously acquired knowledge during the fine-tuning process, which can lead to missing critical information in the generated summaries.

In our previous study [32], we demonstrated that providing discharge notes with automatically highlighted fine-granular clinical details, using the Cardiology Interface Terminology (CIT) curated through machine learning-supported methods [33,34,35], improves the accuracy of LLM-generated summaries when used at inference time via prompt engineering. Highlighted inputs reduced both hallucination and omission errors compared to unhighlighted inputs. Motivated by these findings, the present study examines whether incorporating the same highlighting directly into the fine-tuning process enables the model to internalize these patterns rather than relying solely on prompt-level cues. To this end, we fine-tune LLaMA2-13B (via LoRA) on the MIMIC-IV-BHC dataset using either highlighted or unhighlighted discharge notes and compare the resulting summary quality across multiple evaluation dimensions.

To evaluate our approach, we analyze the resulting summaries, from both approaches, using BLEU, BERT-Score, and ROUGE-L metrics. Additionally, we assess the completeness of the summaries through manual review and employ LLMs as judges to evaluate summaries’ quality based on coherence, fluency, conciseness, and correctness. We aim to enhance the clarity and usability of clinical records by systematically examining how highlighting input notes affects the performance of a fine-tuned LLaMA2-13B model in discharge note summarization. This, in turn, can support better healthcare decision-making and improve patient outcomes.

To guide this investigation, we address the following research questions. (1) Does incorporating automatically highlighted detailed information into the fine-tuning data improve the quality of discharge note summaries generated by LLaMA2-13B? (2) How does highlighting affect different dimensions of summary quality, including semantic similarity (BERTScore), content overlap (ROUGE-L, BLEU), factual consistency (SummaC-CONV), and completeness? (3) Does highlighted fine-tuning also lead to perceptibly higher-quality summaries when evaluated by LLM-based judgments?

We selected LLaMA-2-13B for this study for three reasons. First, prior benchmark results on the MIMIC-IV-Ext-BHC dataset show that fine-tuned LLaMA-2-13B outperforms other open-source models, including Clinical-T5-Large and FLAN-UL2, and achieves performance closest to proprietary GPT-4 [28]. Second, LLaMA-2-13B offers an optimal balance between model size and computational feasibility for local fine-tuning on protected clinical data, whereas newer models (e.g., LLaMA-3 and Qwen [36]) require substantially more resources for reproducible fine-tuning.

2. Background

2.1. Text Summarization and Related Work

Text summarization techniques can be broadly classified into three categories: extractive summarization, abstractive summarization, and LLM-based summarization [37].

Extractive summarization [38] selects key sentences or phrases directly from the original text without modifying their wording. It relies on ranking mechanisms to identify the most informative parts of the content. However, such techniques contend with some disadvantages, including a potential lack of coherence, as extracted sentences may not flow naturally, and the inability to paraphrase or generalize information [37].

Unlike extractive methods, abstractive summarization [39] generates new text that conveys the key ideas of the original content in a more concise and coherent manner. It offers enhanced fluency and improved contextual understanding, but it also carries the risk of introducing factual inconsistencies, increasing computational costs, and requiring large-scale training datasets to ensure accuracy [37].

With the advent of LLMs, and due to their strong performance in summarization tasks [7,40,41,42], there has been a significant shift toward using LLMs for text summarization. LLM-based summarization utilizes pre-trained transformer models to generate summaries based on prompts or fine-tuning. These models can perform both extractive and abstractive summarization, adapting their output based on the task requirements. As a result, numerous research efforts have focused on developing automated summarization methods for medical texts.

Five LLMs were tested in generating summaries for discharge notes using MIMIC-IV-Ext-BHC dataset [28]. The study covers both open-source models (Clinical-T5-Large, FLAN-UL2, and Llama2-13B) and proprietary models (GPT-3.5 and GPT-4). These models were adapted using different strategies, including fine-tuning and in-context learning. The results showed that fine-tuned Llama2-13B achieved the best performance among open-source models based on quantitative metrics such as BLEU and BERT-Score. However, GPT-4 with in-context learning demonstrated the most robustness across varying input lengths and was preferred over human summaries in a clinical reader study, where five clinicians compared its summaries to those written by human experts. The results also emphasized that while open-source models like Llama2-13B performed well and could match human-written summaries, proprietary models, like GPT-4, had a clear edge in producing summaries that clinicians preferred.

Furthermore, a framework for radiology report summarization, using ChatGPT-3.5, has been proposed [41]. In-context learning and iterative optimization were used to improve the Automatic Impression Generation (AIG) task, which involves summarizing the “Findings” section of a radiology report into the “Impression” section. Instead of fine-tuning the model, prompts using similar reports retrieved via a similarity search technique were dynamically constructed. These retrieved examples provide contextual information, allowing ChatGPT to better generate relevant summaries. The method was evaluated on MIMIC-CXR and OpenI datasets, demonstrating state-of-the-art performance in radiology report summarization without requiring additional training data.

Finally, a system for EHR summarization, using the Google Flan-T5 model, has also been proposed [2] to generate clinician-focused summaries based on clinician-specified topics. Flan-T5 were fine-tuned on an EHR question-answering dataset formatted in the Stanford Question Answering Dataset (SQuAD) style. The fine-tuning process utilized the Seq2SeqTrainer from the Hugging Face Transformers library [43], with optimized hyperparameters to enhance performance. The results achieved are an Exact Match (EM) score of 81.81%, ROUGE scores (ROUGE-1: 96.03%, ROUGE-2: 86.67%, ROUGE-L: 96.10%), and a BLEU score of 63%.

Recent work in clinical NLP has also explored knowledge-guided summarization methods that use clinical concepts, ontologies, or highlight signals to guide LLMs [44,45,46,47]. Examples include models that use Unified Medical Language System (UMLS) [48] or SNOMED tags [49], entity-aware attention, or concept constraints to reduce hallucinations and improve factual accuracy. Highlight-based approaches mark details of the note to steer the model toward clinically relevant content. Our study builds on this line of work by examining whether automatically generated highlights, using CIT, can be incorporated directly into the fine-tuning process to improve summarization quality.

2.2. Dataset

The MIMIC-IV-Ext-BHC dataset [28,29] is a collection of Brief Hospital Course (BHC) summaries paired with their corresponding discharge notes extracted from the MIMIC-IV-Note database [50,51]. MIMIC-IV-Ext-BHC is created by preprocessing discharge summaries from MIMIC-IV-Note, which contains 331,794 de-identified clinical notes from 145,915 patients admitted to the Beth Israel Deaconess Medical Center between 2008 and 2019. Both datasets are hosted on PhysioNet [29,50] and can be accessed after signing a Data Use Agreement and completing the required training. The collection of patient information and the creation of these research resources were reviewed by the Institutional Review Board (IRB) [52] at the Beth Israel Deaconess Medical Center, which granted a waiver of informed consent and approved the data-sharing initiative.

The MIMIC-IV-Ext-BHC dataset is designed to facilitate research on hospital course summarization, addressing the challenge of extracting concise and relevant information from lengthy clinical narratives. This dataset covers a diverse patient population reflective of the broader MIMIC-IV cohort, which includes various age groups, genders, and medical conditions.

To create the MIMIC-IV-Ext-BHC dataset, the MIMIC-IV notes are preprocessed through tokenization, section identification, normalization, and cleaning. This process separated the BHC from the rest of the clinical note.

To validate data quality, a manual review of 100 randomly sampled clinical notes was conducted and a clinical team reviewed 30 note–summary pairs, reporting no significant issues with the extracted content.

The MIMIC-IV-Ext-BHC dataset, consisting of 270,033 discharge note–BHC pairs, contains the columns note_id, input, target, input_tokens, and target_tokens. Note_id is the unique identifier for each row in the dataset, which matches the note_id column from the original MIMIC-IV-Note dataset. The input field includes preprocessed discharge note texts excluding the BHC section. The target field includes the standardized and cleaned BHC text, providing a concise summary of the patient’s hospital course. Input_tokens and target_tokens columns store the tokenized lengths of the clinical notes and BHC summaries.

2.3. Automatic Highlighting

In our previous work [33,34,35], we presented a multi-stage method for curating Cardiology Interface Terminology (CIT), tailored through an automatic method for highlighting information in discharge notes of cardiology patients. Highlighting details in discharge notes enables fast skimming, enhances summarization and simplification, and improves research and interoperability [3,32,33,34,35,53,54,55,56]. This process of CIT curation consists of two main phases. In the first stage, we constructed an initial version of the CIT, referred to as ICIT, by incorporating concepts from 11 cardiology-related subhierarchies of SNOMED CT [57]. However, ICIT alone did not sufficiently capture all detailed information present in the discharge notes. To address this, in the second stage, we employed a semi-automatic, iterative approach to enrich ICIT. Specifically, we mined fine-granularity phrases from discharge notes that contained ICIT concepts. All the mined phrases are reviewed both automatically and manually, and the legitimate ones are added to CIT, forming a new version of CIT. All the illegitimate phrases, structurally or semantically, are added to a reject list (R). In each iteration, we used the latest version of CIT to highlight the build dataset and evaluated its performance by measuring two metrics: coverage and breadth. Coverage is defined as the percentage of total number of words highlighted in a note; and breadth is defined as the average number of words of highlighted concepts.

This iterative process continued until further improvements in coverage became negligible. The output of the second stage serves as the training data for the third stage. All concepts included in the resulting CIT at the end of stage two were considered as positive samples (labeled as 1), while all phrases in the rejection list R were considered as negative samples (labeled as 0). The third stage proceeded as follows: first, we embedded the labeled phrases of training data using Clinical BioBERT [58], and then we trained a neural network (NN) classifier on the embedded dataset. After conducting a grid search [59] to fine-tune the hyperparameters, we ended up having an NN model consisting of a single hidden layer with 100 neurons, ReLU activation function [60], and Adam optimizer [61]. Once trained, the model was used to classify newly extracted phrases from discharge notes as either legitimate concepts or illegitimate phrases. Phrases classified as legitimate were added to CIT, resulting in the final terminology, referred to as CIT_ML. CIT_ML demonstrated a coverage of 74.21% and a breadth of 1.68 for the test dataset. Figure 1 shows an excerpt of a discharge note highlighted by CIT_ML, with information highlighted in blue background color.

3. Method

For this study, we fine-tuned two variants of the LLaMA 2 (13B) model separately: one using highlighted discharge notes, which we refer to as H-LLaMA, and the other one using the same notes without highlighted content, which we refer to as U-LLaMA. Fine-tuning was performed on the High-Performance Computing (HPC) cluster at the New Jersey Institute of Technology (NJIT), located in Newark, NJ, USA. The fine-tuning procedure was identical in both settings, with the only difference being whether the input included highlighted information. We will delve into the details of data preparation and the fine-tuning procedure in the following sections.

3.1. Data Preparation

As described in the Section 2, the MIMIC-IV-Ext-BHC dataset consists of clinical note–summary pairs, where each pair represents a complete discharge note text and its corresponding condensed BHC summary. Each row in this dataset includes a note_id which is a unique identifier matching the note_id column from the original MIMIC-IV-Note database.

In our previous work, we developed CIT to highlight the detailed information of discharge notes of cardiology patients. Therefore, to identify cardiology-related records in MIMIC-IV-Note, we focused on two Intensive Care Units (ICUs) associated with cardiology: Coronary Care Unit (CCU) and Cardiac Vascular Intensive Care Unit (CVICU). We queried the “discharge” table in MIMIC-IV-Note for patients admitted to CCU or CVICU. This resulted in approximately 18,600 records, from which we extracted their note_id values. Next, we filtered MIMIC-IV-Ext-BHC to retain only the rows where the note_id matched those from patients admitted to CCU or CVICU in MIMIC_IV. This resulted in approximately 14,000 records.

From the 14,000 cardiology-related records, we randomly selected 1000 records for fine-tuning the model, which we refer to as training data, and another 100 records for evaluating summaries generated by the fine-tuned model, which we refer to as test data.

We have highlighted the discharge notes of training data and test data using the automatic highlighting method we proposed and reported on in [33,34,35], which automatically generates highlights for each note into an HTML file. The highlighted information is enclosed in <span> tags with the background color #ADD8E6. To prevent the model from being confused by HTML tags, each HTML file was converted into plain text, and the highlighted information was enclosed within square brackets (‘[’ and ‘]’). The brackets are treated as additional tokens by the tokenizer, allowing the model to learn that bracketed spans correspond to clinically important information rather than noise, consistent with our earlier findings.

Please note that although multiple CIT concepts may overlap in the raw terminology, text highlighting by nature cannot display overlapping or nested mappings. Therefore, in such cases, we must choose one of the potential CIT concept mappings for the highlight. In [3,62], we describe an algorithm that selects a highlight which maximizes the coverage and breadth measures. This approach prioritizes longer, multi-word concepts, as they tend to better capture the semantics of the sentence.

We prepared four datasets for model training and evaluation:

Unhighlighted Training Set (UTrain): This set consists of 1000 original discharge notes without any highlighted information, each paired with its corresponding BHC summary.

Unhighlighted Test Set (UTest): This set includes 100 original, unhighlighted discharge notes along with their corresponding BHC summaries and was used for evaluation purposes.

Highlighted Training Set (HTrain): This set contains the same 1000 discharge notes, those used in UTrain, but with detailed information highlighted. The highlighted content was enclosed within square brackets (‘[’ and ‘]’). Each note is paired with its corresponding BHC summary.

Highlighted Test Set (HTest): This set includes 100 discharge notes, the same ones used in UTest, with highlighted information enclosed within square brackets. Each note is paired with its associated BHC summaries.

We converted both UTrain and HTrain into JSON format. Each training example was represented as a JSON object with three fields, “instruction”, “input”, and “output”, resulting in two separate JSON datasets.

The UTrain dataset used the instruction: “Summarize the clinical note into a brief hospital course.”

The HTrain dataset used the instruction: “Summarize the clinical note into a brief hospital course, focusing on the information enclosed within ‘[’ and ‘]’.”

For both datasets, the prompt (i.e., instruction + input) and the output (i.e., BHC) were tokenized, concatenated, and padded or truncated to a fixed length of 4096 tokens. As in MIMIC-IV-Ext-BHC dataset, the input token length averaged 2267 ± 914 and the output token length averaged 564 ± 410 [29], the combined prompt and output length typically remained within this limit, and truncation was rarely needed.

3.2. Fine-Tuning Procedure

For fine-tuning, we used the HuggingFace Transformers library [43] in combination with PEFT (Parameter-Efficient Fine-Tuning) [63] and LoRA (Low-Rank Adaptation) [64]. The model was initialized from a pre-trained LLaMA-2 13B checkpoint in HuggingFace format. The tokenizer was also loaded from the same checkpoint and extended to include the custom discharge note section tags. Special tags of the input discharge notes such as <SEX>, <SERVICE>, <ALLERGIES>, <CHIEF COMPLAINT>, etc., were explicitly added to the tokenizer as special tokens. These tags preserved semantic structure and helped guide the model’s understanding of discharge note organization.

The model was fine-tuned on our institution’s High-Performance Computing (HPC) cluster. Jobs were executed using a SLURM scheduler on a single computer node configured with 2 NVIDIA GPUs, 66 CPU cores, and 256 GB RAM. This environment ensured secure, local processing of clinical text and provided sufficient computational resources for stable fine-tuning of the 13B-parameter LLaMA-2 model. Fine-tuning was performed using LoRA with rank = 8, α = 32, and dropout = 0.10. LoRA adaptation was applied specifically to the query and value projection layers of the attention mechanism, referred to as q_proj and v_proj in the model implementation. Training was conducted for nine epochs using the AdamW optimizer [65] with a learning rate of 2 × 10⁻⁴ and weight decay of 0.005. We used a per-device batch size of 5, and gradient accumulation over 8 steps. The dataset was split into 90% for training and 10% for evaluation.

The selected hyperparameters were chosen based on established best practices for fine-tuning LLaMA-2 models and to ensure stable training on long clinical notes. LoRA settings (rank = 8, α = 32, dropout = 0.10) follow commonly adopted configurations shown to balance parameter efficiency and preservation of base-model knowledge in prior clinical NLP studies [66,67,68]. The learning rate (2 × 10⁻⁴), weight decay (0.005), and batch/gradient accumulation settings were selected after small-scale exploratory runs that demonstrated stable loss curves without overfitting. While more extensive hyperparameter tuning is possible, it is computationally expensive for 13B-parameter models and was beyond the scope of this study.

3.3. Generating Summary from Fine-Tuned Models

After fine-tuning, each of the two fine-tuned LoRA adaptors were separately merged with the base LLaMA-2 13B model to produce standalone fine-tuned models for inferences. Inferences used generation prompt structures consistent with the training format. Following best practices in recent clinical text generation literature [69,70,71], inference was performed using nucleus sampling [72] with temperature = 0.7 and top-p = 0.9. We refer to the summaries generated by U-LLaMA as U-summaries, and those from H-LLaMA as H-summaries.

3.4. Evaluation Metrics

Evaluation involves assessing the quality of the generated summaries in relation to the golden standard summary, in this case, BHC. Summaries should be evaluated from different perspectives. In most related studies, extensive manual work has been conducted to assess summary quality alongside automatic metrics. We performed evaluation in three different categories:

Manual evaluation, automatic evaluation, and LLM-based evaluation. Automatic and LLM-based evaluations were conducted on all 100 notes in the test dataset, and a manual evaluation was performed on a random subset of 40 notes.

3.4.1. Manual Evaluation

Completeness: Measures how well the summary captures the information from the reference BHC summary. A high completeness score means the summary includes more information with fewer missing critical details. Completeness is calculated using Equation (1).

$$completenss={\displaystyle \frac{\# important information in the generated summary}{\# important information in the refrence summary}}\ast 100$$

(1)

We evaluate the completeness of the generated summaries based on the reference summary, rather than the entire discharge note. This is because much of the information in the discharge note is not expected to appear in the summary. When gold-standard reference summaries are available, we assume that only the information contained in the reference summary is necessary to include in the generated summary.

3.4.2. Automatic Evaluation

BERTScore: BERTScore [73] uses contextual embeddings from a pre-trained BERT model to compute semantic similarity between the generated and reference summaries. Unlike traditional n-gram overlap methods, BERTScore compares the similarity of words in the generated and reference texts based on their meaning rather than exact word matches.

BERTScore provides three main scores:

Precision: How much of the generated text is semantically supported by the reference.

Recall: How much of the reference is captured by the generated text.

F1 score: The harmonic mean of precision and recall, often used as the main BERTScore. In this study we also report F1 score as the BERTScore.

ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation—Longest Common Subsequence): ROUGE metrics [74] measures how much of the reference summary appears in the generated summary by measuring the overlap of n-grams, word sequences, and longest common subsequences between the generated summary and reference summaries. One common variant of ROUGE is ROUGE-L, which measures the longest common subsequence to capture fluency and coherence. Its ability to reflect holistic similarity rather than just local n-gram matches makes it more suitable for evaluating summary-level similarity.

BLEU (Bilingual Evaluation Understudy): BLEU metric [75] measures how much of the generated summary appears in the reference summary by measuring n-gram overlap between the generated summary and reference summary. BLEU is widely used for assessing machine translation and text generation tasks, including summarization.

Sumac_CONV: SummaC-CONV [76], specifically designed for the summarization task, is a factual consistency evaluation metric that leverages Natural Language Inference (NLI) [77] to assess whether a generated summary is factually consistent with its reference text. Unlike naive sentence-level approaches, SummaC-CONV does not assume a one-to-one alignment between source and summary sentences. Instead, it compares each summary sentence with multiple relevant sentences from the reference text, aggregating entailment scores. By mapping each summary sentence to multiple sentences, it correctly evaluates cases where multiple facts are summarized into a single sentence or appear in a different order than in the original document.

3.4.3. LLM-Based Evaluation

LLMs have shown great potential as evaluators for LLM-generated summaries, evaluating various aspects of the generated texts [78,79]. In this study, we use ChatGPT 4o (through Azure) to evaluate generated summaries of each discharge note. We crafted and refined a prompt to instruct ChatGPT 4o to evaluate each summary based on four criteria, and then assign a score from 1 to 5 for each, where 1 indicates the lowest and 5 the highest. Here is the final version of the prompt:

“Act as a cardiologist and read a discharge note (A) alongside its reference summary (B) and two LLM-generated summaries (C and D). Your task is to grade both summaries on a scale of 1 to 5 for the following four metrics:

Coherence: Does the summary maintain a logical flow of ideas and present information clearly?

Fluency: Is it grammatically well-formed and easy to read?

Conciseness: Does it effectively condense information, keeping only the most relevant clinical details?

Correctness: Does the summary accurately reflect the content of the discharge note (A) and the reference summary (B), without introducing false or misleading information?

Assign a score (1–5) for each metric for both summaries (C and D). A score of 1 indicates the lowest quality, while 5 indicates the highest.”

4. Results

Figure 2a shows an example of a BHC summary of one discharge note, containing 181 words, alongside the H-summary (Figure 2b) and U-summary of its corresponding discharge note (Figure 2c), which contain 126 and 135 words, respectively.

In this example, six key pieces of information are highlighted in green in Figure 2a. Among these green highlights, the blue highlights in Figure 2b indicate information that is present in the original note (Figure 2a) but missing from the U-summary (Figure 2c). Conversely, yellow highlights denote information present in the U-summary (Figure 2c) but absent from the H-summary (Figure 2b). Pink highlights in Figure 2a indicate information that does not appear in either summary. All unhighlighted pieces of information in (Figure 2a) correspond to information that appears in both summaries. The H-summary achieves a completeness score of 56.6%, while the U-summary achieves a completeness score of 47.8%.

Manual Evaluation: For random 40 notes, the average completeness of the U-summaries is 47.6%, while the H-summaries demonstrated a higher average completeness of 54.8%, an increase of 6 percentage points. Additionally, for 33 notes, the completeness of the H-summary is higher than the corresponding U-summary, for 4 notes, the completeness is equal, and for 3 notes, the completeness of U-summary is higher. Using Fisher Exact test [80,81], we compared the completeness of H-summary group (33 notes) and the U-summary group (7 notes) with a significance level of 0.05. The statistical value was less than 0.00001, indicating a statistically significant result (p < 0.05).

Automatic evaluation: As shown in

Table 1. The average results of automatic evaluations for U_summaries (U_S) and H_summaries (H_S).

Word Count			BERTScore		ROUGE-L		BLEU		Sumac_CONV
HCB	U_S	H_S	U_S	H_S	U_S	H_S	U_S	H_S	U_S	H_S
393.19	300.38	331.67	59.61	63.75	21.82	23.43	8.41	10.4	40.2	67.7

, the average BERTScore of the U-summaries for 100 notes is 59.61, while that of the H-summaries is 63.75, with 91 notes having higher BERTScore for H-summaries than U-summaries. Using Fisher Exact test, we compared the BERTScore of H-summary group (91 notes) and the U-summary group (9 notes) with a significance level of 0.05. The statistical value was less than 0.00001, indicating a statistically significant result (p < 0.05).

The average ROUGE-L score for the U-summaries is 21.82, whereas it is 23.43 for the H-summaries, with 90 notes having higher ROUGE-L score for H-summaries than U-summaries. Using Fisher Exact test, we compared the ROUGE-L score of H-summary group (90 notes) and the U-summary group (10 notes) with a significance level of 0.05. The statistical value was less than 0.00001, indicating a statistically significant result (p < 0.05).

Also, the average BLEU score of the U-summaries is 8.41, compared to 10.4 for the H-summaries, with 81 notes having higher BLEU score for H-summaries than U-summaries. Using Fisher Exact test, we compared the BLEU score of H-summary group (81 notes) and the U-summary group (19 notes) with a significance level of 0.05. The statistical value was less than 0.00001, indicating a statistically significant result (p < 0.05).

Finally, the average Sumac_CONV score is 40.2 for the U-summaries and 67.7 for the H-summaries, with 98 notes having higher Sumac_CONV score for H-summaries than U-summaries. Using Fisher Exact test, we compared the Sumac_CONV score of H-summary group (98 notes) and the U-summary group (2 notes) with a significance level of 0.05. The statistical value was 0.0002, indicating a statistically significant result (p < 0.05).

LLMs evaluation: ChatGPT 4o generally rated both U_summaries and H_summaries highly across all four criteria (Coherence, Fluency, Conciseness, and Correctness), indicating that both types of summaries are overall well written. However, H_summaries received slightly higher average scores than U_summaries across three criteria (Coherence, Conciseness, and Correctness), and for Fluency, the averages were almost equal.

The average scores for Coherence are 4.84 (U_summaries) and 4.91 (H_summaries), for Fluency are 4.91 (U_summaries) and 4.92 (H_summaries), for Conciseness are 4.65 (U_summaries) and 4.81 (H_summaries), and for Correctness are 4.79 (U_summaries) and 4.91 (H_summaries). When considering the overall average across all criteria, H_summaries scored 4.88, slightly outperforming U_summaries, which scored 4.79.

Also, as shown in

Table 2. The results of LLM evaluation for U_summaries (U_S) and H_summaries (H_S).

Coherence		Fluency		Conciseness		Correctness		Average
U_S	H_S	U_S	H_S	U_S	H_S	U_S	H_S	U_S	H_S
4.84	4.91	4.91	4.92	4.65	4.81	4.79	4.91	4.79	4.88

, the average length of BHC summaries for these 100 notes is 393.19, while the average length of U-summaries is 300.38 and that of H-summaries is 331.67, indicating that H-summaries tend to include more information.

5. Discussion

In this study, we fine-tuned LLaMA 2 (13B) twice, separately, under the same conditions: once using unhighlighted discharge notes and once more using the same notes but highlighted. By keeping everything else constant, we aimed to isolate and measure the effect of highlighting on fine-tuning. Results (

Table 3. Minimum, maximum, and standard deviation for evaluation metrics for U_summaries (U_S) and H_summaries (H_S).

Metric	Summary	Min	Max	STD
Completeness	U_S	23.2	73.3	14.01
	H_S	29.3	80.0	12.36
BERTScore	U_S	57.8	69.2	2.86
	H_S	59.3	69.3	2.75
ROUGE-L	U_S	14.1	34.3	7.19
	H_S	19.2	39.3	5.54
BLEU	U_S	1.2	13.7	5.35
	H_S	4.2	22.4	4.97
Sumac	U_S	7.5	63.4	15.13
	H_S	60.1	89.3	8.21
Coherence	U_S	4	5	0.36
	H_S	4	5	0.22
Fluency	U_S	4	5	0.36
	H_S	4	5	0.22
Conciseness	U_S	4	5	0.43
	H_S	4	5	0.40
Correctness	U_S	4	5	0.40
	H_S	4	5	0.30

) show that using highlighted discharge notes for fine-tuning improves the quality of the generated summaries compared to fine-tuning with unhighlighted notes across all evaluation metrics. For the completeness, BERTScore, ROUGE-L, BLEU, and SummaC_CONV metrics, we achieved a statistical significance of (p < 0.05) using Fisher Exact test with a significance level of 0.05.

While the improvements in ROUGE-L and BLEU (lexical similarity measures) and BERTScore (a semantic similarity measure) are numerically modest, the much larger gain in SummaC_CONV suggests that highlighting may have a stronger effect on improving factual correctness than on general similarity. SummaC_CONV assesses whether a summary omits or contradicts clinical details, which may explain why it shows a clearer distinction between the two models.

For statistical testing, we converted each metric into a per-note comparison (whether the H-summary scored higher than the U-summary for that specific note) and applied Fisher Exact test to the resulting win/loss counts. We chose this approach because it makes no assumptions about score distributions and is reliable for small sample sizes. However, we acknowledge that this method is less conventional for continuous metrics such as BERTScore or ROUGE, because it only captures whether H > U, not how large the difference is. Therefore, the test reflects consistency of improvement rather than effect size. Even so, the overall conclusion is unchanged when looking directly at the average scores and the large number of per-note wins, which both strongly favor the highlighted model.

In our previous study [32], we demonstrated that summarizing discharge notes with highlighted information, using a prompt engineering strategy, improves the accuracy of summaries compared to summarizing unhighlighted discharge notes. Unlike our previous work, where highlighting was provided only at inference time through the prompt, the present study incorporates highlight signals directly into the fine-tuning data. This allows the model to internalize and learn from highlighted patterns during training, rather than relying solely on external prompt cues. Demonstrating improvements under this training regime shows that highlight information is beneficial not just as contextual guidance during prompting, but also as an embedded feature that shapes the model’s learned representation.

The BHC section in discharge notes offers a concise narrative of the patient’s clinical journey. Composing this section is widely recognized as a cognitively demanding and time-intensive task for clinicians since it requires synthesizing a large volume of notes and reports generated throughout the patient’s hospitalization into a coherent summary [82,83]. This process is not only laborious but also susceptible to errors, given the high documentation burden [82,84]. Furthermore, BHCs exhibit significant variability in both style and content. Authored by different clinicians, they reflect diverse writing habits, and they frequently alternate between extractive and abstractive summarization strategies [83]. Prior research has shown that discharge summaries and their BHC sections may omit critical information or introduce excessive, redundant, or even erroneous content [84,85]. Also, sometimes new or summary-only information is added to BHC that are not documented elsewhere clearly.

Although BHCs are commonly used as reference summaries for training summarization models, they are inherently noisy. Their variability in coherence, completeness, and potential misrepresentation of clinical facts introduce significant challenges for model training and evaluation [83,85]. After reviewing several examples, we noted that some BHCs are very short, shallow, and lack important details, while others are overly long and include too much information. There is no consistent structure or style among them. Indeed, the BHC targets have a mean token length of 564 with a standard deviation of 410 [29], which is notably high, indicating substantial variation in target length. This inconsistency makes it harder for the language model to learn what kind of summary it is supposed to generate. If all BHCs followed a similar pattern, both the highlighted and unhighlighted models would have a clearer target during training.

Another factor that impacts the quality of the generated summaries is the presence of errors or mismatches between the original discharge notes and their corresponding BHCs. In some cases, the BHC includes information that does not appear in the original note. For example, in note “11874424-DS-9,” the “atorvastatin 80 mg” is mentioned in the BHC, but the original discharge note only refers to a prior dose of atorvastatin 40 mg, with no indication of a change to 80 mg. Similarly, the original note states that the patient was accompanied by his son, while the BHC refers to a daughter. These kinds of inaccuracies or additions in the BHCs negatively affect the quality of the summaries generated by the LLM. Since the model generates summaries based only on the original note, it should not include information that was never there. As a result, when these summaries are evaluated against BHCs, the evaluation scores decrease, not because the model failed, but because the reference summaries are flawed.

While LLM-based evaluation provides a scalable and cost-effective complement to manual review, it also introduces potential biases and limitations, particularly in clinical or safety-critical domains. LLMs may overestimate summary quality because they are sensitive to fluency and coherence, sometimes rewarding well-written text even when factual omissions are present. Their judgments can also reflect the biases of the underlying training data or misinterpret specialized clinical details. Furthermore, LLMs may implicitly favor summaries that resemble their own generative style, creating an alignment bias rather than an objective assessment. For these reasons, LLM-based scores should be interpreted as supportive evidence rather than definitive clinical validation, and future work should incorporate more extensive clinician-based evaluation to mitigate these limitations.

This work should be interpreted as a proof-of-concept study rather than a full-scale evaluation. The training set (1000 notes) and test set (100 notes) are relatively small compared to the full MIMIC-IV corpus and come exclusively from cardiology units (CCU and CVICU). This domain specificity limits the ability to generalize findings to other specialties, where documentation patterns, terminology, and summary structures may differ substantially. The reduced dataset size also increases the risk of overfitting, even with parameter-efficient fine-tuning, and may amplify the influence of noise or inconsistencies in the BHC targets. Therefore, while the observed improvements from highlighted fine-tuning are robust within this subset, broader validation across larger and more diverse clinical domains is necessary to assess generalizability.

Future work: For future work, we plan to fine-tune the model using a larger dataset and to reduce variability in the training data. In particular, we will stratify samples based on the BHC-to-note length ratio or other style consistency criteria, allowing us to select example BHCs that follow a more uniform structure. This will help reduce heterogeneity, improve representativeness, and make it easier to evaluate model performance. We also aim to conduct a preliminary evaluation of BHCs using LLMs, assessing them based on their corresponding discharge notes to determine which are more comprehensive and less erroneous. This evaluation will help identify BHCs suitable to serve as gold-standard summaries. BHCs with higher ratings, along with their corresponding discharge notes, form the training dataset for fine-tuning the model. We also plan to conduct a hybrid training where we fine-tune LLaMA using a mixed dataset that includes both highlighted and non-highlighted inputs. This will allow us to observe how the model behaves when trained with both types of data. Also, we plan to increase the test set for manual evaluation to 100 notes in order to achieve a more accurate assessment. Additionally, the clinical significance of the observed metric improvements remains uncertain, and future work will include clinician-based evaluation and deeper analysis of why factual consistency measures show larger gains than lexical overlap or semantic similarity metrics.

6. Conclusions

This study investigates the effect of highlighting information in discharge notes on the summaries generated by Large Language Models (LLMs). Highlighting is performed automatically using a Cardiology Interface Terminology (CIT) proposed in our previous work. To carry out our experiment, we fine-tuned the LLaMA2-13B model twice using the MIMIC-IV-Ext-BHC dataset: once with the highlighted discharge notes and once more with the same set of discharge notes without highlighting.

Our results demonstrate that incorporating highlighted information into the fine-tuning process improves the quality of discharge note summarization. That is, summaries generated from highlighted inputs consistently outperformed those from unhighlighted inputs across all evaluation metrics, including BERTScore, ROUGE-L, BLEU, SummaC_CONV, completeness, and LLM-based judgments.

This work provides a scalable framework for improving discharge notes’ summarization using fine-tuned LLMs. By enhancing the clarity and completeness of clinical summaries, this approach has the potential to support more effective healthcare delivery and better informed decision-making.