Predicting Mortality and Readmission in Obstructive Sleep Apnea via LLM-Expanded Clinical Concepts

Abstract

Obstructive Sleep Apnea (OSA) is a common sleep disorder associated with serious health risks. This study leverages large language models (LLMs) to process and interpret clinical narratives in electronic health records. It develops clinically meaningful lexicons for predicting mortality and readmission risk, as well as for multiclass diagnostic classification in OSA patients. Using LLM-expanded lexicons, logistic regression models achieved ROC–AUC scores of 0.844 for 6-month all-cause post-discharge mortality, 0.817 for 1-year all-cause post-discharge mortality, and 0.729 for all-cause hospital readmissions following the first discharge. Diagnostic performance was highest with smaller n-gram representations, indicating that additional contextual length did not improve performance. Compared with frequency-based n-gram models, LLM-expanded lexicons yielded sparser feature sets with lower computational cost and comparable performance. Our findings highlight the potential of LLM-expanded lexicons to enhance OSA diagnosis and clinical risk stratification.

1. Introduction

Obstructive sleep apnea (OSA) is a sleep-related breathing disorder characterized by complete (apnea) or partial (hypopnea) obstruction of the upper airways, leading to sleep disruptions and intermittent hypoxemia, i.e., a low level of oxygen in the blood. In the United States, the overall prevalence is 32.4% among adults aged 20 years and older, with 39.1% among males and 26.0% in females, after adjusting for obesity [1], and almost one billion people are affected globally [2]. In addition, OSA is associated with an increased risk of various cardiovascular (CV) [3,4,5,6], metabolic [5], and neurological conditions [7,8].

The substantial morbidity and mortality associated with OSA and the cost of treatment require accurate risk stratification to optimize patient care and resource allocation. Current clinical risk assessment tools, such as the STOP-BANG questionnaire [9] and the NoSAS score [10], are designed primarily for OSA screening rather than mortality or readmission prediction. Although traditional polysomnographic parameters such as the apnea-hypopnea index (AHI) provide diagnostic information, they have limited prognostic value for predicting adverse clinical outcomes in OSA patients.

Previous machine learning approaches for OSA outcome prediction have predominantly relied on structured patient data, such as demographics, vital signs, and laboratory values. However, using solely structured Electronic Health Records (EHR) data can lead to biased results and suboptimal predictive performance, as substantial clinical information often resides in unstructured clinical narratives [11]. Recent studies have shown that unstructured clinical notes improve the prediction of mortality and hospital readmission [12,13]. Artificial intelligence (AI)-based medical systems increasingly use natural language processing (NLP) with pretrained language models to process and interpret clinical narratives in EHRs [14,15,16].

Large language models (LLMs) offer several advantages over traditional machine learning (ML) and conventional NLP approaches for healthcare applications. First, LLMs learn contextualized representations that capture semantic meaning, nuanced language variability, and complex clinical phenomena—including long-range dependencies, negation, and temporality [17,18,19]. Second, through large-scale pretraining on biomedical and clinical corpora, clinical LLMs enable effective transfer learning and improved performance on downstream healthcare tasks with limited labeled data, outperforming classical ML models that rely heavily on manual feature engineering or fixed vocabularies [20,21]. Finally, LLMs have shown the ability to generalize across diverse medical tasks without requiring task-specific architectures and training, providing a flexible framework for clinical NLP applications [22].

These advantages motivated our use of pretrained clinical language models to enrich the representation of clinical concepts. Starting from physician-provided seed terms for categorizing OSA and its comorbidities, we expanded the lexicon by identifying additional relevant medical terms based on their semantic similarity to the seed terms using LLMs. Unlike traditional NLP pipelines, which typically rely on fixed vocabularies, rule-based mapping, or task-specific feature engineering, this approach identifies synonymous, abbreviated, and context-dependent expressions commonly found in discharge documentation. Thus, the main advantages of our approach include broader coverage of clinical terminology, reduced reliance on manual feature curation, and leveraging knowledge from pretrained language models.

Similar terminology expansion strategies using distributional semantics and embedding-based nearest-neighbor retrieval have been studied in clinical text [23,24,25]. However, prior work has not systematically combined expert-curated seed terms with LLM-based expansion and applied the resulting lexicon to OSA-focused predictive analyses. Our contributions include (i) a framework for using LLMs to expand clinical lexicons and then use them to represent unstructured discharge notes; (ii) demonstrating LLM-expanded lexicons transfer across task—from diagnosis to outcome prediction; and (iii) showing that LLM-expanded lexicons enable sparser, more efficient models than traditional frequency-based n-gram approaches.

The remainder of this paper is organized as follows. Section 2 reviews related work, Section 3 describes the dataset, LLM-based lexicon expansion process, and modeling framework, Section 4 presents experimental results and comparisons with n-gram baselines, and Section 5 discusses the main findings, limitations, and future directions.

2. Related Work

We review prior work on LLMs in healthcare in Section 2.1 and NLP methods for predicting clinical outcomes in Section 2.2. While LLMs have been widely applied to clinical prediction, few studies have examined whether domain-specific LLM-expanded lexicons can transfer across tasks. This study assesses whether lexicons developed for OSA diagnosis can effectively predict mortality and hospital readmission.

2.1. LLMs in Health Care

LLMs are a type of AI model that is trained on vast amounts of typically unlabeled data [26]. While traditional AI models are often single-task systems, foundation models (FMs) which are LLMs trained on large datasets, can be subsequently finetuned to perform many different downstream tasks. FMs represent a paradigm shift in AI model development [26]. This allows a single LLM to be reused across a range of tasks with minimal adaptation or retraining. However, LLMs typically have a substantially greater number of parameters than traditional AI models—sometimes in the hundreds of billions. This requires significant computational resources for training [27].

Recent advances in LLMs, the exponential growth of medical literature, and the widespread availability of large-scale EHRs have set the stage for clinical LLMs to revolutionize medical practice. Noteworthy applications of LLMs in healthcare include named entity recognition and relation extraction (e.g., BioBERT [19] and BlueBERT [28]), medical question answering and inference (e.g., GatorTron [20] and Med-PaLM [21]), discharge summaries (e.g., ChatGPT-3.5 [29]), diagnosis classification (e.g., ClinicalBERT [18]), and various others [20,30]. Recent systematic reviews and studies demonstrate LLMs’ efficacy in interview dialogue summarization [31] and disease diagnosis and treatment [32], and highlight the need for human-centric LLMs for personalized medicine and equitable development and access [33].

Embeddings are numerical representations of words, phrases, or sentences that capture contextual information and understand relationships within large segments of text. They have been used in various tasks, such as text retrieval and ranking [34], text classification [35], sentiment analysis [36], clinical concept extraction [37], and patient risk stratification [38].

Building on previous work [39], this study focuses on embeddings extracted from LLMs. Starting with a set of initial medical terms (“seed terms”) for categorizing OSA and its associated comorbidities, we aim to expand the lexicon using LLMs. Specifically, we identify additional relevant medical terms by computing the cosine similarity between their embeddings and the seed terms.

2.2. Predicting Mortality and Hospital Readmission Through NLP Techniques

Recent studies have explored the application of NLP techniques to predict mortality and hospital readmission in healthcare settings. Some approaches used unstructured clinical notes only [40,41,42,43]. For example, Huang et al. pretrained BERT on clinical notes and fine-tuned it for improved 30-day hospital readmission prediction.

Yet, others integrate clinical text, vital signs, time series measurements, and imaging to create a comprehensive profile of a patient’s health status for more accurate predictions [44,45,46,47,48]. For example, Jin et al. performed named entity extraction and negation detection on clinical notes and trained a multimodal neural network that integrated time series signals and unstructured clinical text representations for predicting in-hospital mortality risk in ICU patients.

More recently, LLM ensembles have been applied directly to mortality prediction using unstructured medical notes from MIMIC-IV [49]. LLMs have also been used to annotate and extract structured variables from unstructured clinical narratives to support downstream outcome prediction [15,16]. These approaches treat LLMs as domain-aware information extractors, highlighting a shift from using LLMs as black-box generators to clinically-guided feature engineering tools.

Despite prior work on leveraging LLMs in clinical prediction tasks, few studies have explored whether domain-specific concepts—automatically expanded using LLMs—can be repurposed across clinical tasks such as outcome prediction. In this study, we aim to assess the predictive effectiveness of LLM-expanded medical terms, originally developed for OSA diagnosis, in predicting mortality and hospital readmission risks.

3. Materials and Methods

This section describes the data, methods, and evaluation framework. We present the MIMIC-IV dataset in Section 3.1, the software platform in Section 3.2, the LLM-based lexicon expansion and classification workflow in Section 3.3, and model training procedures in Section 3.4.

3.1. Dataset

The Medical Information Mart for Intensive Care (MIMIC)-IV database is utilized for this project [50]. It comprises deidentified EHR for patients admitted to the Beth Israel Deaconess Medical Center Intensive Care Unit or ICU between 2008 and 2019. MIMIC-IV v2.2, released in January 2023, consists of records for 299,712 patients and 431,231 admissions.

In addition to OSA, we examined the following associated comorbidities: diabetes mellitus type 2 (T2DM), hypertension (HTN), heart failure (HF), and atrial fibrillation (AF).

Table 1. Summary of ICD Codes and Seed Terms.

Condition	ICD Codes, N	Seed Terms, N
OSA	9	38
T2DM	157	13
HTN	68	10
HF	61	15
AF	16	12
Total	311	88

Note: The full list of ICD codes is provided in Supplementary Materials S1, and the complete list of seed terms in Supplementary Materials S2.

shows the number of International Classification of Diseases (ICD) codes and seed terms identified by physicians for each health condition.

As an example, the following ICD codes are used to identify patients with OSA: 327.20 (Organic sleep apnea, unspecified), 327.23 (Obstructive sleep apnea [adult, pediatric]), 327.29 (Other organic sleep apnea), 780.51 (Insomnia with sleep apnea), 780.53 (Hypersomnia with sleep apnea), 780.57 (Sleep apnea [NOS]), G4730 (Sleep apnea, unspecified), G4733 (Obstructive sleep apnea [adult, pediatric]), and G4739 (Other sleep apnea). A patient is considered to have a positive diagnosis for a specific health condition if they possess at least one corresponding ICD code.

Table 2. Demographic Data for Patients with OSA and Comorbidities.

Characteristic	OSA	T2DM	HTN	HF	AF
Patients, N	13,942	21,666	74,080	21,076	25,743
Discharge Notes, N	29,892	53,446	161,245	49,479	55,418
Women, N (%)	5628 (40.4)	10,088 (46.6)	36,486 (49.3)	9944 (47.2)	11,215 (44.0)
White, N (%)	9895 (71.0)	13,802 (63.7)	51,238 (69.2)	15,269 (72.4)	19,980 (77.6)
Black, N (%)	1966 (14.1)	3688 (17.0)	9959 (13.4)	2476 (11.7)	1776 (6.9)
Other, N (%)	2081 (14.9)	4176 (19.3)	12,883 (17.4)	3331 (15.9)	3967 (15.5)

provides basic demographic data of patients of interest.

The dataset described in this section serves as the source population for all analyses in this study. Task-specific cohorts for mortality prediction, readmission prediction, and the n-gram size analysis are derived from this shared population using outcome-based inclusion criteria and are described in detail in Section 4.2, Section 4.3, and Section 4.4, respectively.

3.2. Software and Platform

The major software packages used in this study include pandas 2.3.3, NumPy 2.1.2, matplotlib 3.10.6, mpi4py 4.1.0, PyTorch 2.8.0+cu129, and scikit-learn 1.7.2. Pretrained biomedical language models included BlueBERT, GatorTron-medium, and BioClinicalBERT, all used as released without additional fine-tuning. Computing resources were provided by the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory, which supported large-scale language model inference, complex machine learning workloads, and processing of EHR data.

3.3. Process Flow

Our previous study demonstrated LLM-based lexicon development for OSA sub-phenotyping [51]. In this study, we focus on applying these LLM-expanded lexicons for mortality and readmission prediction. The flowchart in Figure 1 outlines the sequence of tasks or processes that are executed to achieve the objective described in Section 1. Next, we describe these tasks.

3.3.1. Bi/Tri/Four-Grams Extracted from Discharge Notes

The initial step is to extract bigrams (pairs of consecutive words), trigrams (triplets of consecutive words), and four-grams (four consecutive words) from all patient discharge notes to capture commonly used phrases. MIMIC-IV v2.2 contains 331,794 discharge notes. The mean number of characters per note is 10,551. The longest and shortest discharge notes have 60,381 and 353 characters, respectively. The numbers of bigrams, trigrams, and four-grams extracted from this step are 3,096,096, 5,407,839, and 4,792,806, respectively. These n-grams (i.e., bigrams, trigrams, and four-grams) are candidates for expanding lexicons in this study.

3.3.2. Seed Terms Provided by Physicians

Physicians involved in this study provided seed terms, i.e., relevant medical terms or phrases, for OSA and comorbidities of interest. The number of seed terms per condition is listed in Table 1. For example, the following terms are among the 38 terms for OSA: poorly refreshing sleep, obstructive sleep apnea (OSA), obesity hypoventilation syndrome (OHS), unrefreshing sleep, sleepiness, excessive daytime sleepiness (EDS), and snoring.

3.3.3. Expanding Lexicon via LLMs

The goal of this step is to identify informative n-grams from discharge notes for categorizing OSA and its associated comorbidities. The general approach to selection is by comparing how similar these n-grams are to the seed terms of corresponding conditions. The similarity between an n-gram and a seed term is measured by the cosine similarity between their embeddings, which are semantic representations extracted from an LLM. Specifically, given an n-gram t and a seed term s, the cosine similarity between t and s is measured using

$$S_{\mathrm{c}}(t,s)={\displaystyle \frac{V\left(t\right)·V\left(s\right)}{2\times \parallel V\left(t\right)\parallel ·\parallel V\left(s\right)\parallel }}$$

(1)

where $V\left(t\right)$ and $V\left(s\right)$ are LLM embedding vectors of the n-gram t and seed term s, respectively, and $\parallel ·\parallel$ denotes the Euclidean norm.

3.3.4. Expanded Lexicon

For each n-gram, there are 88 similarity scores, corresponding to the 88 seed terms (Table 1). The importance or relevance of each n-gram to a specific health condition (OSA or comorbidity) is measured by the average of all similarity scores between the n-gram and the seed terms associated with that condition. As a result, each n-gram has five similarity scores, one for each health condition.

The similarity scores of n-grams are then ranked individually for each condition, and the rankings of bigrams, trigrams, and four-grams are separated as well. Thus, each health condition yields three distinct ranked lists, from which a number of n-grams are selected as textual features for prediction tasks.

3.3.5. Patient Discharge Notes Extracted Using ICD Codes

Discharge notes for patients with OSA and/or comorbidities are extracted based on ICD codes (See Table 2 for summary). The process involved merging information from multiple tables or files. Discharge notes are long-form narratives that describe the reason for a patient’s admission to the hospital, their hospital course, and any relevant discharge instructions. Each discharge note corresponds to a single hospital stay, and a patient may have multiple discharge notes if he or she has more than one hospital stay.

3.3.6. Classifying with Logistic Regression

Logistic Regression (LR) is a statistical technique in machine learning used to model the relationship between a set of independent variables and a categorical dependent variable. LR estimates the probability of an observation belonging to each class, handling both binary and multiclass classification through multinomial extensions. It is selected for this study due to its simplicity and its ability to provide interpretable insights into the relative importance of different text features (i.e., n-grams) for predicting class membership.

Each discharge note was labeled based on the study-specific classification. For example, in the mortality study, each discharge note was labeled based on the patient’s status as either alive or deceased.

To represent each discharge note as input features for LR, a “bag-of-ngrams” encoding was applied, treating each n-gram as a binary presence/absence feature variable. The selection of n-grams was determined by the health conditions under study. For instance, in a study involving patients with OSA and HF using trigrams, the feature set comprised the top-n trigrams from each condition’s sorted trigram list, merged with duplicates removed. Each discharge note was then encoded as a feature vector based on the presence or absence of these selected trigrams.

3.3.7. Prediction Results

The expanded lexicons for characterizing OSA and associated comorbidities were evaluated for their predictive power in three classification tasks: (1) mortality prediction (alive vs. deceased), (2) readmission prediction (readmitted vs. not readmitted), and (3) n-gram size impact study using diagnostic labels (OSA only, HF only, and OSA & HF), as assessed in Section 4.

3.4. Model Training and Evaluation

LR hyperparameters were optimized using repeated stratified 5-fold cross-validation, which ensures balanced class distribution across folds and prevents data leakage. Grid search evaluated regularization strength ($C\in \{0.001, 0.01, 0.05, 0.1, 1\}$), penalty (L2), solver (LBFGS and SAGA), and class weighting (None or balanced), selecting the configuration that maximized the mean area under the receiver operating characteristic curve (ROC-AUC) for binary tasks and weighted AUROC (wAUC) for multiclass tasks across validation folds. wAUC was selected over macro AUC because it weights each class by its prevalence, providing a clinically relevant measure that reflects actual patient distribution, whereas macro AUC treats all classes equally regardless of size. The same hyperparameter search space, random seed (random_state = 42), and cross-validation splits were held constant across all classification tasks and feature representations to ensure reproducibility. Final models were trained using scikit-learn’s LogisticRegression with the selected hyperparameters.

Pretrained language models were used only for medical lexicon expansion; no fine-tuning or task-specific training of LLM parameters was performed.

4. Results and Discussion

We present results for three prediction tasks: mortality in Section 4.2, hospital readmission in Section 4.3, and diagnostic classification examining n-gram size effects in Section 4.4. Section 4.1 provides an overview of the LLMs, cohort construction, and model comparison approach used across experiments.

4.1. Overview

The following LLMs were used for lexicon expansion. For mortality, we used GatorTron Medium (3.9 B parameters, trained on EHR data from the University of Florida Health system, PubMed, and MIMIC) and BlueBERT (336 M parameters, trained on MIMIC and PubMed) to generate trigrams. BlueBERT was also used to generate four-grams for readmission. Additionally, BioClinicalBERT (110 M parameters, initialized from BioBERT and trained on MIMIC-III) was used to examine the impact of n-gram size on diagnostic performance for OSA. These models were selected to balance model complexity and compatibility with MIMIC-IV data.

As outlined in Section 3.3.6, for each task-specific study cohort, an equal number of LLM-selected n-grams were chosen from OSA and related comorbidities, then merged into a unified feature set. These n-grams were treated as binary features based on their presence in discharge notes. Models were trained and evaluated following the same process as described in Section 3.4.

To further investigate the impact of LLM-expanded n-grams on mortality and readmission prediction, we include a generic n-gram baseline, referred to as the Top-ngram approach. Rather than using LLM-selected trigrams or four-grams, this method relies on the most frequently occurring trigrams or four-grams in discharge notes from the MIMIC-IV dataset. We compared the LLM-expanded n-gram approach with the Top-ngram model in both predictive performance and computational characteristics. Model complexity is assessed using three indicators: total number of n-gram count, non-zero coefficients after model fitting, and empirical runtime. These measures represent feature-space dimensionality, model sparsity, and computational cost, respectively.

4.2. Mortality Prediction

To build the cohort, we extracted patients and their hospital admissions related to OSA, T2DM, or HTN using ICD codes determined by physicians.

Table 3. Patient Composition for Mortality Prediction: OSA, T2DM, and HTN.

Group	Patients (N)	Hospital Admissions (N)
OSA Only (w/o T2DM and HTN)	6392	11,266
T2DM Only (w/o OSA and HTN)	5561	9372
HTN Only (w/o OSA and T2DM)	56,111	107,081
OSA & T2DM & HTN	2828	6000
Other +	23,809	49,432
Total	81,096 *	183,151

⁺ Includes patients with two of the three health conditions. * A patient with multiple admissions or discharge notes may appear in more than one of the categories listed above at different points in time.

summarizes the number of patients and corresponding hospital admissions for each group. Note that each admission is associated with a single discharge note.

For the mortality study cohort, only the last hospital admission for each patient was included. Discharge notes were labeled as deceased if the patient died within either 6 months or 1 year following discharge. The counts reported in

Table 4. The Mortality Study Cohort.

Status	6-Month Post-Discharge	1-Year Post-Discharge
Alive	66,813	61,623
Deceased	9606	14,796

represent the number of alive and deceased cases across the different patient groups.

The mortality cohort was evaluated using stratified 5-fold cross-validation with three repeats. Because only one admission per patient was included, the evaluation is patient-independent by construction. The same splits were applied consistently across all models and feature representations.

Two approaches were explored, as described in the Overview section. The Top-ngram approach selected the 200,000 most frequent trigrams from all discharge notes as features. In contrast, the LLM-based approach (i.e., GatorTron Medium and BlueBERT) merged top-ranked trigrams from each condition-specific LLM-expanded list, yielding 206,858 features for GatorTron Medium and 204,003 features for BlueBERT.

Table 5. Mortality Prediction Results.

Approach	6-Month Post-Discharge			1-Year Post-Discharge
	AUC	Total Trigrams	Non-Zero N (%)	AUC	Total Trigrams	Non-Zero N (%)
Top-ngram	0.899	200,000	199,936 (99.97)	0.871	200,000	199,939 (99.97)
GatorTron Medium	0.844	206,858	163,867 (79.22)	0.817	206,858	164,123 (79.34)
BlueBERT	0.821	204,003	159,980 (78.42)	0.803	204,003	159,763 (78.31)

AUC: ROC-AUC.

shows that the Top-ngram approach outperformed both the GatorTron Medium and BlueBERT models in predicting 6-month and 1-year mortality. To better understand these results, we further examined the model parameters. Specifically, we analyzed the number of unique trigrams used in each model (i.e., Total Trigrams) and the number of trigrams with non-zero coefficients in the fitted LR models (i.e., Non-zero) (Table 5). A non-zero coefficient indicates that the trigram contributes to the model’s predictions.

The Top-ngram approach retained nearly all 200,000 trigrams (99.9% non-zero coefficients), whereas GatorTron Medium and BlueBERT produced sparser models with only 78–79% non-zero coefficients. Higher sparsity (more coefficients set to zero) is desirable as it indicates the model relies on a smaller, semantically focused feature subset, improving interpretability and computational efficiency. Empirically, models using LLM-expanded lexicons were approximately twice as fast as the Top-ngram approach, consistent with their sparser feature representations.

Despite substantial class imbalance (12.6–19.4% mortality) (Table 4),

Table 6. Additional Performance Metrics for Mortality Prediction.

Outcome	Approach	Accuracy	Precision	Recall
6-month Post-discharge	Top-ngram	0.881 ± 0.003	0.520 ± 0.007	0.696 ± 0.010
	GatorTron	0.846 ± 0.002	0.421 ± 0.006	0.611 ± 0.013
	BlueBERT	0.834 ± 0.003	0.390 ± 0.008	0.574 ± 0.012
1-year Post-discharge	Top-ngram	0.808 ± 0.003	0.502 ± 0.008	0.760 ± 0.008
	GatorTron	0.786 ± 0.004	0.463 ± 0.008	0.656 ± 0.009
	BlueBERT	0.776 ± 0.003	0.446 ± 0.008	0.639 ± 0.009
Outcome	Approach	Specificity	F1-Score
6-month Post-discharge	Top-ngram	0.907 ± 0.003	0.595 ± 0.007
	GatorTron	0.879 ± 0.002	0.499 ± 0.007
	BlueBERT	0.871 ± 0.003	0.465 ± 0.008
1-year Post-discharge	Top-ngram	0.819 ± 0.004	0.605 ± 0.007
	GatorTron	0.817 ± 0.004	0.543 ± 0.008
	BlueBERT	0.809 ± 0.003	0.525 ± 0.009

shows all approaches achieved good performance with precision 3× above baseline prevalence and high specificity (0.81–0.91). LLM approaches achieved comparable ROC-AUC with better computational efficiency while Top-ngram showed better precision–recall balance, which may be important in clinical settings where both unnecessary interventions and missed high-risk patients carry costs.

4.3. Hospital Readmission Prediction

To build the readmission cohort, we extracted patients with OSA or AF using physician-assigned ICD codes.

Table 7. Patient Composition for Hospital Readmission: OSA and AF.

Group	Patients (N)	Discharge Notes (N)
OSA Only (w/o AF)	11,287	22,698
AF Only (w/o OSA)	23,539	48,224
OSA & AF	3405	7194
Total	38,231	78,116

provides information on the patient composition for each group.

As part of the preprocessing for the readmission analysis, hospital admissions were first chronologically ordered for each patient to establish a timeline of visits. The final admission for each patient was identified, and a distinction was made between intermediate and last admissions. To ensure that the analysis focused only on admissions where readmission was possible, records corresponding to a patient’s final admission were excluded if the patient had died following that visit. This step helps avoid bias by removing cases where readmission was not a possibility due to death. Patients’ discharge notes were labeled as 1 if they were readmitted and labeled as 0 if not readmitted or deceased. The counts reported in

Table 8. The Readmission Study Cohort.

Readmitted	Not Readmitted	Total
42,124	25,037	67,161

represent the number of discharge notes across the different patient groups.

For readmission prediction, we applied stratified group 5-fold cross-validation (three repeats) with patient-level grouping to prevent data leakage. Model training and hyperparameter optimization followed the procedures described in Section 3.4.

Two approaches were used to generate four-gram features for logistic regression. The Top-ngram approach selected the 200,000 most frequent four-grams from all discharge notes as features. The LLM-based approach merged top-ranked four-grams from each condition-specific BlueBERT-expanded list, yielding 208,455 features after removing duplicates. Figure 2 shows the predictive power of four-grams in readmission prediction improves for longer windows, indicating they are potentially better in capturing chronic disease burden and long-term risk. “Anytime” here refers to hospital admissions that occur at any time after the initial hospital discharge.

Table 9. Readmission Prediction Results.

Approach	AUC	Total Four-Grams, N	Non-Zero, N (%)
Top-ngram	0.736	200,000	199,629 (99.81)
BlueBERT	0.729	208,455	150,376 (72.13)

AUC: ROC-AUC.

shows that Top-ngram and BlueBERT approaches achieved comparable performance (ROC-AUC 0.736 vs. 0.729). As observed in the mortality study, the BlueBERT approach yielded a sparser model (72.13% non-zero coefficients vs. 99.83% for Top-ngram). The computational cost also differed substantially: the BlueBERT-expanded model was approximately five times faster than the Top-ngram model.

With a more balanced class distribution (62.7% readmission rate),

Table 10. Additional Performance Metrics for Readmission Prediction. ROC-AUC values are reported in Table 9.

Approach	Accuracy	Precision	Recall	Specificity
Top-ngram	0.684 ± 0.004	0.763 ± 0.006	0.719 ± 0.006	0.624 ± 0.008
BlueBERT	0.673 ± 0.003	0.767 ± 0.005	0.687 ± 0.006	0.649 ± 0.007
Approach			F1-Score
Top-ngram			0.741 ± 0.004
BlueBERT			0.725 ± 0.004

shows both approaches performed well. Precision was 0.76–0.77 and F1-scores were 0.72–0.74. Top-ngram showed higher recall (0.719 vs. 0.687) but slightly lower specificity (0.624 vs. 0.649).

4.4. Comparing the Impact of N-Gram Size

The goal of this study is to analyze how the size of LLM-expanded n-grams influences diagnostic performance when the language model is held fixed. BioClinicalBERT was selected based on our prior systematic evaluation of six pretrained language models for OSA classification [51], which identified it as best balancing representational quality and computational efficiency. BioClinicalBERT [18] is used throughout the study to isolate the effect of n-gram size.

Similar to the mortality and readmission studies, the patient cohort was extracted based on OSA and HF ICDs (

Table 11. Patient Cohort for OSA and HF.

Diagnosis Label	Discharge Notes (N)
OSA Only (w/o HF)	21,929
HF Only (w/o OSA)	41,516
OSA & HF	7963
Total	71,408

).

An equal number of bi/tri/four-grams was chosen from the ranked bi/tri/four-gram lists of OSA and HF. Similar to the previous sections, each selected n-gram serves as a unique feature for the bag-of-words classification model, capturing the presence or absence of each n-gram in a given discharge note. The discharge notes were labeled with OSA only (i.e., without HF), HF only (i.e., without OSA), or both OSA & HF based on the ICD code. LR was then employed for a 3-class 1-to-many classification. Classification performance was measured by wAUC rather than macro AUC to account for class imbalance (Table 11), as wAUC weights classes by prevalence to better reflect the actual patient distribution.

Stratified 5-fold cross-validation (three repeats) at admission-level was applied to this task-specific cohort, consistently across all n-gram sizes. Figure 3 shows the wAUC range from approximately 0.74 to 0.9. It also suggests that the wAUC scores for all three types of n-grams increase as the number of n-grams grows. The bigram-based model generally outperforms the trigram- and four-gram-based models, indicating that the additional context provided by trigrams and four-grams does not contribute to higher predictive performance for this analysis. This is somewhat unexpected, as trigram- and four-gram-based models were initially assumed to offer richer contextual information due to their longer phrase structure.

For diagnosis prediction, we applied stratified patient-level grouped 5-fold cross-validation (three repeats) to prevent data leakage from the same patient. Model training and hyperparameter optimization followed the procedures described in Section 3.4.

Table 12. Comprehensive Performance Metrics for OSA/HF Diagnosis Across N-gram Types.

N-Gram	wAUC a	mAUC b	Accuracy	Precision c	Recall c
Bigrams	0.863 ± 0.004	0.836 ± 0.004	0.751 ± 0.004	0.645 ± 0.008	0.644 ± 0.005
Trigrams	0.820 ± 0.005	0.793 ± 0.005	0.699 ± 0.007	0.579 ± 0.010	0.582 ± 0.007
Four-grams	0.803 ± 0.005	0.775 ± 0.005	0.688 ± 0.006	0.566 ± 0.006	0.565 ± 0.004
N-Gram	F1 c	Precision d	Recall d	F1 d
Bigrams	0.642 ± 0.006	0.746 ± 0.005	0.751 ± 0.004	0.747 ± 0.004
Trigrams	0.579 ± 0.008	0.695 ± 0.007	0.699 ± 0.007	0.696 ± 0.007
Four-grams	0.563 ± 0.005	0.680 ± 0.008	0.688 ± 0.006	0.683 ± 0.007

^a wAUC: weighted AUC. ^b mAUC: macro AUC. ^c Macro-averaged metrics. ^d Weighted metrics (account for class imbalance).

shows wAUC is consistently higher than macro AUROC across all n-gram types, reflecting class imbalance and supporting the use of wAUC as the primary metric. Bigrams outperform trigrams and four-grams across all measures, including wAUC (0.863 vs. 0.820 vs. 0.803) and weighted F1 (0.747 vs. 0.696 vs. 0.683). The gap between weighted and macro-averaged metrics indicates difficulty in predicting minority classes, and the monotonic decline in performance with increasing n-gram size confirms that additional context from longer n-grams does not improve diagnostic performance.

5. Conclusions

Research indicates that OSA increases the risk of cardiovascular and metabolic complications. This study used LLMs and NLP to develop a lexicon specific to OSA and its associated comorbidities for predicting patient mortality and hospital readmission risk, as well as for performing multiclass diagnostic classification of OSA and HF.

5.1. Major Findings

LLMs can identify informative lexicons for predicting mortality and hospital readmission in OSA patients, achieving ROC-AUC scores of 0.844 for 6-month post-discharge mortality (Table 5) and 0.729 for all-cause hospital readmissions following the first discharge (Table 9). In the mortality study, the GatorTron Medium-expanded lexicon performed slightly better than the lexicon expanded with BlueBERT, which could be because more complex models are more adept at selecting higher-quality n-grams, leading to a more accurate characterization of health status.

The Top-ngram approach achieved comparable or better results but required greater computational cost. This increased computational cost stems from using nearly all available n-grams, whereas LLM-expanded lexicons focus on a smaller, semantically informed feature subset (Table 5 and Table 9). Although the LLM approach began with a similar number of n-grams, only a smaller fraction had non-zero coefficients, yielding sparser models. This sparsity likely explains why LLM models ran approximately two to five times faster than Top-ngram models.

LLMs can identify informative lexicons for OSA diagnosis, achieving wAUC scores of 0.9 or slightly higher (Figure 3). wAUC scores for all three n-grams increase as the number of n-gram counts grow, with the bigram model outperforming trigram and four-gram models. This suggests the extra context provided by trigrams and four-grams did not help with predictive performance.

5.2. Limitations and Future Work

In this study, we explored the effectiveness of employing LLM-expanded lexicons for predicting patient outcomes and performing multiclass diagnostic classification of OSA and HF. Several limitations should be acknowledged.

Data and Labeling Limitations. First, discharge notes for patients with OSA and comorbidities were labeled using ICD codes, primarily designed for billing purposes and not necessarily indicative of the patient’s final diagnosis. Second, our analysis relied solely on data from a single dataset (MIMIC-IV), which may limit generalizability to other healthcare settings with different patient populations, documentation practices, or coding conventions. To enhance model validation, we plan to incorporate two or more instances of diagnostic codes for a specific health condition when labeling [52], collaborate with physicians to create a ground truth dataset [53], and validate our approach on other datasets pending data availability and access agreements.

Model and Evaluation Limitations. We employed logistic regression to support interpretability, but this approach may not capture complex nonlinear relationships between features. In addition, our models use only unstructured clinical notes. While we reported standard performance metrics, we did not assess clinical utility or real-world deployment feasibility. Future work will explore more advanced models, combine structured data (e.g., demographics, time series) with unstructured data from EHRs, and conduct real-world evaluations.

Generalizability and N-gram Size. The n-gram size study demonstrated that bigrams outperform trigrams and four-grams for OSA/HF diagnosis, but this finding may not generalize to other patient populations or prediction tasks. We plan to systematically investigate how n-gram size affects performance across different comorbidity combinations (OSA with T2DM, HTN, AF) and prediction tasks (mortality, readmission). Additionally, we will explore lexicons using combinations of n-gram sizes rather than single n-gram types, and develop methods to automatically select optimal n-gram sizes based on task characteristics.

Interpretability and Clinical Adoption. While sparse models offer some interpretability through feature weights, understanding which specific clinical concepts drive predictions and how they interact remains challenging. We plan to develop explainable methods that improve interpretability of model outputs, including feature attribution techniques and visualization tools for clinicians, and conduct user studies with physicians to evaluate clinical acceptability and actionability of model predictions.