Identifying Pre-Existing Diabetes at ICU Admission with Machine Learning on Public GOSSIS Data

Abstract

Background: Pre-existing diabetes mellitus is prevalent among critically ill adults and can influence initial glycemic targets, therapeutic decisions, and early risk stratification in the intensive care unit (ICU). However, diabetes status may be distributed across heterogeneous electronic health record (EHR) sources and may be incomplete at the time of ICU admission, particularly for inter-facility transfers. Methods: Using the public WiDS Datathon 2021 tabular release derived from the Global Open-Source Severity of Illness Score (GOSSIS) initiative, we conducted a retrospective machine-learning benchmarking study for admission-time identification of documented diabetes status in ICU patients. Candidate predictors included demographics, admission characteristics, anthropometrics, day-1 physiologic and laboratory summaries, APACHE-related variables, comorbidity indicators, and site descriptors. We compared CatBoost, random forest, tuned XGBoost, tuned LightGBM, histogram-based gradient boosting, and a soft-voting ensemble combining XGBoost, LightGBM, and histogram-based gradient boosting. Because class imbalance was a central concern, the final workflow emphasized model-intrinsic class weighting and threshold-aware evaluation rather than synthetic oversampling. Results: In the primary leakage-mitigated random validation split, the voting ensemble achieved the highest overall balance, with AUROC 0.8539, precision 0.5671, recall 0.6690, and F1-score 0.6138. Tuned LightGBM was the most sensitivity-oriented individual model, achieving recall 0.7677 and AUROC 0.8537, although with lower precision and a less favorable Brier score. Ablation analyses clarified the source of this performance: removing leakage-prone and APACHE-related variables caused only modest decreases in discrimination, whereas the strict reduced model that also excluded glucose-like predictors produced a marked decline, with LightGBM AUROC falling to 0.7432 and the voting ensemble AUROC falling to 0.7448. These findings, together with SHAP analyses identifying day-1 glucose maximum, day-1 glucose minimum, BMI, age, hemoglobin, and related clinical variables as major contributors, indicate that glucose-related admission variables remained the dominant predictive signal. In grouped hospital validation, tuned LightGBM maintained recall of 0.7684 while AUROC decreased modestly to 0.8443, indicating preserved case detection under stricter site separation but reduced precision. Precision–recall analysis further showed that average precision decreased from 0.622 under random validation to 0.551 under grouped validation; at a high-sensitivity grouped-site operating point, a probability threshold of 0.4537 achieved recall of 0.8001 with precision of 0.4314. Calibration curves and Brier scores showed that predicted probabilities were imperfectly calibrated. Conclusions: Although the dominance of glucose-related predictors is clinically plausible for identifying documented diabetes status, early glycemic measurements in critically ill patients may also partly capture acute stress physiology, treatment-related effects, monitoring intensity, or other forms of acute dysglycemia rather than chronic diabetes status alone. Therefore, these findings support gradient-boosted and ensemble models as reproducible tools for ICU admission-time phenotyping of documented diabetes status, but the proposed system should be interpreted primarily as a screening-oriented phenotyping aid for chart review, cohort enrichment, or workflow support, not as a stand-alone diagnostic tool. Further external validation, recalibration, threshold selection matched to intended use, and clinical review are needed before deployment.

1. Introduction

Diabetes mellitus and dysglycemia are highly prevalent in critical care, where acute illness frequently disrupts glucose homeostasis through counter-regulatory hormone release, inflammatory signaling, medication exposure, nutritional support, and other iatrogenic factors. Both hyperglycemia and hypoglycemia, as well as increased glycemic variability, have been associated with adverse outcomes in critically ill populations, including infection, longer ICU stay, and increased mortality risk.

Contemporary evidence has shifted ICU glycemic management away from intensive glucose-normalization strategies after large multicenter randomized trials demonstrated net harm, including increased mortality and hypoglycemia risk, under intensive glucose targets compared with more moderate control [1]. Reflecting this evidence base, major consensus recommendations for most critically ill adults favor initiating insulin for persistent hyperglycemia, commonly at or above approximately 180 mg/dL, and then targeting a moderate glucose range centered around approximately 140–180 mg/dL rather than pursuing tighter normalization [2,3].

Accurate recognition of diabetes status at or near ICU admission is clinically meaningful because early hyperglycemia is etiologically heterogeneous. Elevated glucose may reflect documented pre-existing diabetes, previously unrecognized chronic dysglycemia, stress hyperglycemia caused by acute physiologic stress, medication-related hyperglycemia, nutrition-associated hyperglycemia, or treatment effects during the early ICU course. This distinction influences how clinicians interpret admission glucose values, balance insulin titration against hypoglycemia risk, and decide whether elevated inpatient glucose should trigger outpatient diabetes evaluation after discharge. In the present study, the target label is therefore interpreted as documented diabetes status available in the source records, not as de novo diagnosis of diabetes during ICU stay, and not as direct prediction of stress hyperglycemia or incident diabetes.

Increasing attention has been directed toward contextualizing acute hyperglycemia relative to baseline glycemic status. Measures such as the stress hyperglycemia ratio, commonly defined using admission glucose divided by an HbA1c-derived estimated average glucose, have been proposed as pragmatic markers of relative hyperglycemia and have been associated with illness severity and mortality across cohorts [4,5].

Related constructs, including the glycemic gap, have also been explored for risk stratification and outcome prediction in ICU populations [6]. Importantly, observational work suggests that the prognostic meaning of hyperglycemia and the optimal glucose exposure range may differ between patients with and without diabetes, reinforcing the importance of identifying baseline diabetes status when studying ICU glycemic control and safety tradeoffs [7].

Despite its importance, diabetes status at ICU admission is often incomplete, delayed, or fragmented across heterogeneous electronic health record (EHR) sources. Relevant evidence may be distributed across problem lists, billing or diagnosis codes, prior encounter histories, medication records, laboratory archives, and transfer documentation.

These sources may be unavailable when patients arrive from another facility, when longitudinal history is not accessible, or when documentation practices differ across hospitals. Under these real-world constraints, clinicians and researchers often need a computable method to infer likely documented diabetes status using structured variables available at or near admission, while acknowledging uncertainty from incomplete capture and heterogeneous documentation.

Computable phenotyping operationalizes clinical definitions using EHR data through rule-based algorithms, statistical models, or machine-learning approaches. Great collaborative efforts have disseminated phenotype definitions and workflows, including PheKB and eMERGE-derived algorithms, demonstrating the feasibility of diabetes identification using combinations of diagnosis codes, medications, and laboratory criteria [8,9]. However, phenotype portability across institutions remains challenging because of differences in coding practices, data models, missingness patterns, laboratory availability, and clinical workflows. Even when the same clinical definition is intended, implementation burden and site-specific customization can be substantial [10,11].

These challenges have motivated calls for clearer reporting of phenotype complexity, implementation requirements, validation design, and maintenance considerations so that end users can judge whether an algorithm is fit for purpose in their target setting and timeframe [12]. In parallel, reporting and appraisal frameworks for prediction modeling and artificial intelligence methods emphasize transparency in dataset construction, label definition, preprocessing order, feature selection, validation strategy, calibration, interpretability, and intended use case [13,14].

These principles are particularly important for ICU phenotyping, where apparent model performance can be inflated by leakage-prone variables, site identifiers, duplicated preprocessing across train and validation data, or validation designs that allow patients from the same hospital to appear in both training and testing folds. A clinically useful diabetes phenotype should therefore be evaluated not only by discrimination, but also by threshold-dependent performance, probability calibration, robustness to feature exclusions, and transportability across hospitals.

Public critical-care datasets have created important opportunities to develop reproducible phenotyping and prediction pipelines at scale. The WiDS Datathon 2021 tabular release, derived from the Global Open-Source Severity of Illness Score (GOSSIS) initiative, provides a large public multi-center ICU dataset suitable for benchmarking structured EHR machine-learning workflows [15,16]. In this study, we use the labeled WiDS 2021 training file to evaluate admission-time identification of documented diabetes status among adult ICU stays. The separate unlabeled challenge file is not used to fit preprocessing parameters, select thresholds, or estimate validation performance.

Prior work has demonstrated the broad utility of machine-learning and deep learning approaches for diabetes prediction, clinical disease classification, ICU outcome modeling, and healthcare operations. Recent reviews have highlighted the use of tree-based learners, neural networks, recurrent architectures, convolutional models, and Transformers across EHR, sensor, wearable, administrative, and operational datasets, while also emphasizing persistent limitations related to fragmented data sources, bias, privacy, validation, and reproducibility [17]. In contrast to these broader applications, the present study focuses specifically on admission-time phenotyping of documented diabetes status in a public multi-center ICU cohort. Rather than optimizing only discrimination, we evaluate leakage mitigation, feature-scenario robustness, grouped hospital transportability, precision–recall behavior under class imbalance, probability calibration, and SHAP-based interpretability.

A closely related GOSSIS-based study by Sánchez-Gómez et al. [18] applied an AdaBoost ensemble classifier to diabetes-related prediction in ICU patients using more than 90 structured clinical features, including demographic variables, vital signs, laboratory values, and comorbidities. Their model reported an AUROC of 0.83 and accuracy of 83.28%, suggesting good overall discrimination, but the threshold-dependent results showed an important imbalance between specificity and sensitivity, with high specificity of 93.95% and lower recall of 41.95%. The authors identified glucose, body mass index, age, creatinine, and bicarbonate as important predictors, supporting the clinical relevance of metabolic, renal, and demographic features in ICU diabetes-related modeling [18]. Their findings are consistent with the present study in showing that structured ICU variables can support machine-learning-based diabetes classification in public critical-care data. However, our study differs in objective and evaluation scope: rather than focusing on a single AdaBoost model, we benchmark multiple tree-based classifiers for admission-time phenotyping of documented diabetes status, evaluate leakage-mitigated feature scenarios, assess grouped hospital transportability, examine precision-recall behavior under class imbalance, evaluate probability calibration, and use SHAP-based interpretation to characterize model behavior.

We benchmarked several tree-based machine-learning classifiers selected for their suitability to structured tabular clinical data with nonlinear associations, mixed feature types, and clinically meaningful missingness patterns. The evaluated models included CatBoost, random forest, tuned XGBoost, tuned LightGBM, histogram-based gradient boosting, and a soft-voting ensemble combining tuned XGBoost, tuned LightGBM, and histogram-based gradient boosting [19,20,21,22].

The workflow emphasized split-aware preprocessing, including training-derived imputation, one-hot encoding based only on training categories, correlation pruning based only on the training matrix, removal of high-frequency one-hour variables, removal of explicit site identifiers from the model matrix, and harmonization of admission-source variables.

Because class imbalance is a central concern in diabetes phenotyping, the final workflow emphasized model-intrinsic class weighting and threshold-aware evaluation rather than synthetic oversampling, consistent with recent concerns that SMOTE-generated synthetic medical records may introduce clinically questionable examples and distort model interpretation, and with recent clinical machine-learning approaches that combine class weighting with threshold optimization to improve recall while limiting artifacts from oversampling [23].

Model performance was assessed using complementary fixed-threshold metrics, including accuracy, precision, recall, and F1-score; probability-based metrics, including AUROC and the Brier score; and precision–recall analysis, which is especially informative for evaluating binary classifiers under class imbalance [24,25,26,27]. To evaluate robustness and possible leakage dependence, the full modeling framework was repeated across four prespecified feature scenarios: full feature set, leakage-mitigated original, exclude all APACHE, and strict reduced model. This scenario-based ablation design was used to determine whether model performance depended on leakage-prone predictors, APACHE-derived variables, site-related structure, or glucose-like proxies, consistent with recommendations to detect and mitigate data leakage, assess predictor-related risk of bias, and transparently report validation and feature-handling decisions in clinical machine-learning prediction studies [13,28,29,30,31]. To estimate transportability to unseen clinical environments, we also performed grouped site-stratified validation using hospital identifiers so that no hospital appeared in both training and validation folds. Finally, calibration plots and Brier scores were used to assess whether the predicted probabilities were reliable as risk estimates, while SHAP-based interpretability was used to examine whether model behavior was clinically coherent, transparent, and explainable at both global and patient-specific levels [32,33,34,35,36,37,38].

Accordingly, the objective of this study was to develop and evaluate a leakage-aware, reproducible machine-learning benchmark for ICU admission-time phenotyping of documented diabetes status using the public WiDS 2021/GOSSIS release. By combining model benchmarking with ablation analysis, grouped hospital validation, calibration assessment, and SHAP-based interpretation, this work aims to clarify the performance, robustness, and implementation tradeoffs of machine-learning-based diabetes identification in public multi-center critical-care EHR data, consistent with recommended practices for benchmarking critical-care models, preventing leakage, transparently reporting clinical prediction models, assessing probability reliability, and explaining tree-based model behavior [14,30,33,36,39,40,41].

2. Materials and Methods

2.1. Experimental Setup and Environment

All computational analyses and modeling were conducted using Python 3.9.6. Data manipulation, cleaning, and structuring were performed using the pandas and numpy libraries. The core machine learning pipeline was constructed using scikit-learn, which provided robust modules for data splitting, missing value imputation, and performance evaluation metrics. For the predictive modeling, we implemented an ensemble of advanced gradient boosting frameworks, specifically CatBoost, LightGBM, and XGBoost, alongside standard Random Forest and Histogram-based Gradient Boosting classifiers. Furthermore, model interpretability and feature importance were extracted using the SHAP (SHapley Additive exPlanations) library, while interactive and static visualizations were rendered using plotly and matplotlib.

2.2. Data Source and Study Design

We conducted a retrospective machine-learning benchmarking study using the public WiDS Datathon 2021 tabular release derived from MIT’s Global Open-Source Severity of Illness Score (GOSSIS) initiative. The raw datasets used in this study were ingested into the computational environment using the pandas library. The primary dataset was segmented into a training cohort consisting of 130,157 patient records with 181 features, and an unlabeled testing cohort comprising 10,234 records with 180 features. This initial data ingestion phase provided a quick structural inspection of the data and served as the foundation for all subsequent preprocessing and predictive modeling [15,16].

2.3. Outcome Definition

The primary outcome was a binary indicator of pre-existing diabetes mellitus documented at or prior to ICU admission. This label reflects historical diabetes status as represented in the source database (e.g., comorbidity fields and/or admission documentation) rather than a de novo diagnosis made during the ICU stay. Accordingly, the modeling task is framed as phenotyping/identification of an existing condition, not detection of stress hyperglycemia or prediction of incident diabetes.

2.4. Candidate Predictors and Feature-Scenario Design

Candidate predictors were drawn from the public WiDS 2021 release and included demographic variables, admission characteristics, anthropometrics, day-1 physiologic and laboratory summaries, APACHE-related variables, comorbidity indicators, and site descriptors. The analysis is designed to address indirect leakage and site memorization by evaluating four prespecified feature scenarios:

• Full feature set: routine cleaning only, with grouping identifiers removed from the model matrix.
• Leakage-mitigated original: removal of variables most likely to encode label-adjacent information, including APACHE diagnosis fields and selected chronic comorbidity indicators, while retaining other clinically relevant predictors.
• Exclude all APACHE: removal of all APACHE-related variables in addition to the leakage-mitigated exclusions.
• Strict reduced model: a conservative predictor set that further excluded APACHE-related variables, site identifiers, and glucose-like predictors.

2.5. Data Preprocessing and Feature Engineering

To prepare the raw WiDS 2021/GOSSIS files for predictive modeling, we implemented a split-aware preprocessing and feature-engineering pipeline in which all data-driven transformations were learned only from the training portion of the labeled dataset.

The unlabeled challenge file was not used to fit imputers, define categorical encoding levels, derive engineered variables, or determine correlation-pruning rules. This prevented information from validation or challenge records from influencing the preprocessing parameters used during model development.

Formally, after initial row filtering, the adult labeled cohort was denoted by $D_{\mathrm{l}\mathrm{a}\mathrm{b}}=\{\left(x_i,y_i\right){\}}_{i=1}^N$, where $x_i$ is the predictor vector for ICU stay $i$ and $y_i\in \{0,1\}$ is the binary diabetes label. The labeled data were partitioned into an 80/20 stratified random split using a fixed seed of 40:

$$\left(X_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},X_{\mathrm{v}\mathrm{a}\mathrm{l}},y_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},y_{\mathrm{v}\mathrm{a}\mathrm{l}}\right)=\mathrm{S}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}\mathrm{S}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\left(X,y;\mathrm{test\_size}=0.20, \mathrm{stratify}=y, \mathrm{seed}=40\right)$$

(1)

All preprocessing parameters were then estimated from the training split alone and applied without refitting to the validation and unlabeled challenge sets:

$$\Theta ={\mathrm{f}\mathrm{i}\mathrm{t}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\left(\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\right), X_{\mathrm{v}\mathrm{a}\mathrm{l}}^{\mathrm{*}}={\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}}_{\Theta }\left(X_{\mathrm{v}\mathrm{a}\mathrm{l}}\right), X_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\mathrm{*}}={\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}}_{\Theta }\left(X_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}\right)$$

(2)

The first row-level transformation restricted the analysis to adult ICU stays aged 16 years or older:

$$D_{\mathrm{a}\mathrm{d}\mathrm{u}\mathrm{l}\mathrm{t}}=\{\left(x_i,y_i\right)\in D_{\mathrm{l}\mathrm{a}\mathrm{b}}:ag{e}_i\ge 16\}$$

(3)

Schema-level harmonization then standardized admission-source variables. Multiple hospital and ICU admission categories were recoded to a smaller and more consistent vocabulary to reduce spelling inconsistencies and ontology fragmentation. Missing hospital_admit_source values were filled with “Other,” and missing icu_admit_source values were backfilled from hospital_admit_source whenever that variable was available.

Logical consistency checks were next applied to all paired day-level summary variables ending in _min and _max. Whenever the recorded minimum exceeded the corresponding maximum, the two values were exchanged according to the following rule:

$$x_{ij}^{\mathrm{m}\mathrm{i}\mathrm{n}}>x_{ij}^{\mathrm{m}\mathrm{a}\mathrm{x}}: \left(x_{ij}^{\mathrm{m}\mathrm{i}\mathrm{n}},x_{ij}^{\mathrm{m}\mathrm{a}\mathrm{x}}\right)=\left(x_{ij}^{\mathrm{m}\mathrm{a}\mathrm{x}},x_{ij}^{\mathrm{m}\mathrm{i}\mathrm{n}}\right)$$

(4)

A limited engineered physiologic feature was added for the $P_i^{AO2}/F_i^{IO2}$ ratio. Let $R_i^{PF}$ denote d1_pao2fio2ratio_max, $P_i^{AO2}$ denote pao2_apache, and $F_i^{IO2}$ denote fio2_apache. When the day-1 ratio was missing, both source variables were observed, and the denominator was nonzero, the ratio was computed as:

$$R_i^{PF}={\displaystyle \frac{P_i^{AO2}}{F_i^{IO2}}}, F_i^{IO2}\ne 0$$

(5)

All h1_prefixed variables were removed to eliminate extremely granular one-hour measures. The code also dropped near-duplicate APACHE variables whenever an equivalent day-1 representation was available, thereby prioritizing directly observed day-1 measurements over redundant APACHE summaries. Potentially non-predictive identifiers, including encounter_id, readmission_status, and the unnamed index column, were excluded from the predictor matrix. The grouping variables hospital_id and icu_id were retained separately for grouped validation and robustness analyses but were excluded from the model feature space under all feature scenarios.

Missing-data handling was fully split-aware. For continuous predictors, let $o_{ij}=1$ indicate that the value was observed in the raw training-derived matrix and $o_{ij}=0$ indicate that it was missing. Missing continuous values were replaced with the training-set mean:

$${\mu }_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}={\displaystyle \frac{1}{n_{j,\mathrm{o}\mathrm{b}\mathrm{s}}^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}}}\sum _{i\in \mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}: o_{ij}=1}{x}_{ij}$$

(6)

For categorical predictors, missing values were replaced by the most frequent training-set category. If $c_{ij}$ denotes the categorical value for record $i$ and categorical predictor $j$, and $m_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}$ denotes the training-set mode, the imputation rule was:

$$m_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}=\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\{c_{ij}:i\in \mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}, o_{ij}=1\}$$

(7)

A more specific imputation rule was implemented for anthropometric variables. For height, weight, and body mass index (BMI), the pipeline first computed sex-specific means within the training split and then fell back to the overall training mean if the sex-stratified mean was unavailable. Let $g_i$ denote the recorded gender for patient $i$, let $A=\{\mathrm{h}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t},\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t},\mathrm{B}\mathrm{M}\mathrm{I}\}$, and let $a_{j{g}_i}=1$ indicate that a training-derived gender-specific mean existed for anthropometric variable $j$ and gender group $g_i$:

$${\tilde{x}}_{ij}=\left\{\begin{array}{ll}{x}_{ij},& o_{ij}=1\\ {\mu }_{j\mid g_i}^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)},& o_{ij}=0, a_{j{g}_i}=1\\ {\mu }_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)},& o_{ij}=0, a_{j{g}_i}=0\end{array}\right., j\in A$$

(8)

This two-stage strategy preserved more physiologically plausible imputations for anthropometric variables than a single global mean. In addition, ethnicity and gender themselves were filled using training-derived modal values, with code fallback defaults of “Other/Unknown” and “M,” respectively, when no empirical mode was available.

After imputation, categorical variables were one-hot encoded using only the levels observed in the training data. For a categorical predictor $x_j$ with training categories $C_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}=\{c_1,\dots ,c_K\}$, the dummy representation was defined as:

$$z_{ik}^{\left(j\right)}=\mathbf{1}\left(x_{ij}=c_k\right), c_k\in C_j^{\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\right)}$$

(9)

Validation and challenge matrices were then aligned to the resulting training-derived column set. To reduce redundancy and instability, the code computed pairwise absolute Pearson correlations on the training design matrix only. For features $a$ and $b$, the training-set Pearson correlation was:

$$r_{ab}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^{n_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}}}\left(x_{ia}-{\bar{x}}_a\right)\left(x_{ib}-{\bar{x}}_b\right)}{\sqrt{{\displaystyle {\sum }_{i=1}^{n_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}}}{\left(x_{ia}-{\bar{x}}_a\right)}^2}\sqrt{{\displaystyle {\sum }_{i=1}^{n_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}}}{\left(x_{ib}-{\bar{x}}_b\right)}^2}}}$$

(10)

A feature was flagged for removal whenever it appeared as the second member of any upper-triangular feature pair whose absolute correlation exceeded the threshold of 0.80:

$$C_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}=\{b:\exists a<b, \left|r_{ab}\right|>0.80\}$$

(11)

The same dropped columns were removed from the validation and challenge matrices. Finally, feature names were sanitized by replacing non-alphanumeric characters with underscores, and both validation and test matrices were realigned one final time to the exact training-derived column set. This sequence ensured a stable and identical feature space across all downstream models and validation scenarios. Refer to

Table 1. Preprocessing operations implemented directly.

Component	Rule Implemented in Code
Age restriction	Keep records with age ≥ 16
Split strategy	80/20 stratified train/validation split
Random seed	random_state = 40
Grouping variables	hospital_id and icu_id preserved for robustness checks but removed from the predictor matrix
Identifiers dropped	Unnamed: 0, encounter_id, readmission_status
Row-consistency correction	Swap any _min and _max pair when _min > _max
Engineered feature	Fill d1_pao2fio2ratio_max using pao2_apache/fio2_apache when eligible
Explicit column removal	All h1_ variables removed
APACHE duplicate handling	Drop near-duplicate APACHE variables when equivalent day-1 variables exist
Numeric imputation	Training-set mean
Categorical imputation	Training-set mode
Anthropometric imputation	Sex-stratified training mean for height, weight, and BMI, then global training mean fallback
Categorical encoding	One-hot encoding using training categories only
Correlation pruning	Drop columns with absolute Pearson correlation > 0.80, determined from the training matrix only
Final column handling	Sanitize feature names and align validation/test matrices to training columns

and

Table 2. Scenario-dependent exclusion rules applied after preprocessing but before model fitting.

Scenario	Feature-Exclusion Rule
Full feature set	Basic cleaning only; group identifiers removed from predictor matrix
Leakage-mitigated original	Remove obvious leakage-prone comorbidity and diagnosis fields; retain other APACHE variables
Exclude all APACHE	Remove all APACHE-related variables in addition to the leakage-mitigated exclusions
Strict reduced model	Remove APACHE-related variables, group identifiers, and glucose-like predictors

for details.

2.6. Model Development

We compared five individual tree-based classifiers selected for their strong performance on structured tabular clinical data: CatBoost, random forest, tuned XGBoost, tuned LightGBM, and Histogram-based Gradient Boosting (HGB). These algorithms were trained on the final split-aware, one-hot encoded, correlation-pruned feature matrices described in Section 2.5. For each ICU stay $i$, the transformed predictor vector was denoted by $x_i^{\star }$ and the binary outcome was $y_i\in \{0,1\}$, where $y_i=1$ indicated documented diabetes status. Each classifier learned a mapping from the final predictor space to an estimated probability of diabetes:

$$p_i^{\left(m\right)}=\mathrm{P}\mathrm{r}\left(y_i=1\mid x_i^{\star },M_m\right)=f_m\left(x_i^{\star }\right)$$

(12)

where $M_m$ denotes the $m$-th trained model and $f_m\left(\cdot \right)$ is the model-specific probability function. Predicted class labels were obtained by applying a probability threshold $t$ to the estimated probabilities:

$${\widehat{y}}_i^{\left(m\right)}\left(t\right)=\mathbb{I}\left(p_i^{\left(m\right)}\ge t\right), \mathrm{with} \mathrm{default} t=0.50 \mathrm{unless} \mathrm{otherwise} \mathrm{specified}$$

(13)

This formulation allowed the analysis pipeline to evaluate both fixed-threshold classification performance and threshold-aware behavior in the later precision-recall analysis. The modeling strategy intentionally emphasized probability estimation, not only hard classification, because subsequent sections evaluate AUROC, Brier score, precision-recall behavior, and calibration.

CatBoost and random forest served as robust baseline models. CatBoost was implemented with $100$ boosting iterations and a fixed random seed, while random forest was implemented with $100$ trees, parallel processing, and the same study-level random state. The random forest model can be represented as an average over $B$ independently trained decision trees:

$$p_i^{\left(\mathrm{R}\mathrm{F}\right)}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{T}_b\left(x_i^{\star }\right), B=100$$

(14)

where $T_b\left(x_i^{\star }\right)$ is the probability estimate from the $b$-th tree. This averaging structure reduces variance relative to a single decision tree and provides a conservative benchmark against the boosting-based models.

The gradient-boosted tree models used additive ensembles in which each new tree updated the current prediction function. In general form, the boosted scoring function after $K$ iterations can be written as:

$$F_K\left(x_i^{\star }\right)=F_0\left(x_i^{\star }\right)+{\displaystyle \sum _{k=1}^K}\eta h_k\left(x_i^{\star }\right)$$

(15)

where $h_k\left(\cdot \right)$ is the $k$-th regression tree and $\eta$ is the learning rate. XGBoost and LightGBM represented tuned gradient-boosted tree approaches optimized for this benchmark. The tuned XGBoost configuration used $200$ estimators, maximum depth $8$, learning rate $0.05$, subsample $0.8$, minimum child weight $10$, $\mathrm{colsample\_bytree}=0.8$, a fixed random state, and log-loss evaluation. The tuned LightGBM configuration used gradient boosting decision trees with binary objective, AUC metric, learning rate $0.007$, subsample $1.0$, $\mathrm{colsample}\text{\_}\mathrm{bytree}=0.2$, $\mathrm{reg\_alpha}=3$, $\mathrm{reg\_lambda}=1$, $\mathrm{scale\_pos\_weight}=4$, $10,000$ boosting iterations, unrestricted tree depth, fixed random state $100$, and force_col_wise enabled.

Because class imbalance was a central methodological concern, the final analysis pipeline emphasized model-intrinsic class weighting and threshold-aware evaluation rather than synthetic oversampling. In the LightGBM model, the positive diabetes class was weighted more heavily through $\mathrm{scale\_pos\_weight}=4$. Conceptually, the weighted binary learning objective can be expressed as:

$$\mathcal{L}={\displaystyle \sum _{i=1}^n}{w}_{y_i}\hspace{0.17em}\mathcal{l}\left(y_i,p_i\right)+\Omega \left(F\right)$$

(16)

where $\mathcal{l}\left(y_i,p_i\right)$ is the binary classification loss, $w_{y_i}$ is the class-specific observation weight, and $\Omega \left(F\right)$ represents regularization terms that penalize excessive model complexity. This weighting scheme increased the cost of misclassifying diabetes-positive cases during training while preserving the original empirical distribution of the data. Therefore, the workflow avoided synthetic generation of minority-class records and instead evaluated the sensitivity-precision trade-off directly through model probabilities and threshold-dependent metrics.

In addition to the five individual models, we trained a soft-voting ensemble that combined tuned XGBoost, tuned LightGBM, and histogram-based gradient boosting. The ensemble assigned weights of $1:3:1$ to XGBoost, LightGBM, and HGB, respectively, giving greater influence to the sensitivity-oriented LightGBM model while still incorporating the more conservative probability estimates from XGBoost and HGB. The ensemble probability was calculated as the weighted average:

$$p_i^{\left(\mathrm{e}\mathrm{n}\mathrm{s}\right)}={\displaystyle \frac{1\cdot p_i^{\left(\mathrm{X}\mathrm{G}\mathrm{B}\right)}+3\cdot p_i^{\left(\mathrm{L}\mathrm{G}\mathrm{B}\mathrm{M}\right)}+1\cdot p_i^{\left(\mathrm{H}\mathrm{G}\mathrm{B}\right)}}{1+3+1}}$$

(17)

Final ensemble labels were then obtained by applying the same thresholding rule in Equation (13). This design provided a more balanced operating point than relying on LightGBM alone: the ensemble retained much of LightGBM’s sensitivity while improving precision by tempering extreme probability estimates through the additional boosted-tree models.

Table 3. Model-development configuration.

Model	Role in Analysis	Key Code-Level Configuration
CatBoost	Boosting baseline	100 estimators; random_state = 40; verbose output suppressed
Random Forest	Bagging baseline	100 trees; random_state = 40; parallel processing enabled
XGBoost (Tuned)	Tuned gradient-boosted tree model	200 estimators; max_depth = 8; learning_rate = 0.05; subsample = 0.8; min_child_weight = 10; colsample_bytree = 0.8; eval_metric = logloss
LightGBM (Tuned)	Sensitivity-oriented tuned model	GBDT binary objective; AUC metric; learning_rate = 0.007; colsample_bytree = 0.2; reg_alpha = 3; reg_lambda = 1; scale_pos_weight = 4; n_estimators = 10,000
HistGradientBoosting	Additional tree-boosting comparator	scikit-learn histogram-based gradient boosting; random_state = 40
Soft-voting ensemble	Combined probability model	XGBoost, LightGBM, and HGB combined with soft voting and weights 1:3:1

summarizes the model-development choices.

2.7. Evaluation, Ablation Analysis, and Grouped Site-Stratified Validation

Model performance was evaluated on held-out validation data generated by the split-aware preprocessing workflow described in Section 2.5. For a given validation set $V=\left\{\right(x_i^{\star },y_i),i=1,\dots ,n_V\}$, $x_i^{\star }$ denotes the final transformed predictor vector and $y_i\in \{0,1\}$ denotes the observed diabetes label. For model $m$, the predicted probability $p_i^{\left(m\right)}$ was obtained from the fitted probability function defined in Equation (12), and thresholded class labels were obtained according to Equation (13).

The primary model-comparison tables used the model-default binary prediction, corresponding to a probability threshold of $t=0.50$ for the probabilistic binary classifiers unless a separate threshold analysis was explicitly reported.

For each model and validation fold, the threshold-specific confusion-matrix counts were computed from the observed and predicted labels as follows:

$$\begin{array}{c}TP\left(t\right)={\displaystyle \sum _{i\in V}}\mathbb{I}\left(y_i=1\right)\mathbb{I}\left({\widehat{y}}_i\left(t\right)=1\right)\\ FP\left(t\right)={\displaystyle \sum _{i\in V}}\mathbb{I}\left(y_i=0\right)\mathbb{I}\left({\widehat{y}}_i\left(t\right)=1\right)\\ TN\left(t\right)={\displaystyle \sum _{i\in V}}\mathbb{I}\left(y_i=0\right)\mathbb{I}\left({\widehat{y}}_i\left(t\right)=0\right)\\ FN\left(t\right)={\displaystyle \sum _{i\in V}}\mathbb{I}\left(y_i=1\right)\mathbb{I}\left({\widehat{y}}_i\left(t\right)=0\right)\end{array}$$

(18)

These four quantities were used to calculate the fixed-threshold classification metrics reported in the primary and ablation tables:

$$\begin{array}{c} n_V\left(t\right)=TP\left(t\right)+TN\left(t\right)+FP\left(t\right)+FN\left(t\right)\hfill \\ \mathrm{Accuracy}\left(t\right)={\displaystyle \frac{TP\left(t\right)+TN\left(t\right)}{n_V\left(t\right)}}\hfill \\ \mathrm{Precision}\left(t\right)={\displaystyle \frac{TP\left(t\right)}{TP\left(t\right)+FP\left(t\right)}}\hfill \\ \mathrm{Recall}\left(t\right)={\displaystyle \frac{TP\left(t\right)}{TP\left(t\right)+FN\left(t\right)}}\hfill \end{array}$$

(19)

$$\begin{array}{c}\mathrm{F}1\left(t\right)=2\cdot {\displaystyle \frac{\mathrm{Precision}\left(t\right)\cdot \mathrm{Recall}\left(t\right)}{\mathrm{Precision}\left(t\right)+\mathrm{Recall}\left(t\right)}}\hfill \end{array}$$

(20)

In the implementation, precision, recall, and F1-score were computed using the scikit-learn metric functions with $zero\_division=0$, so undefined ratios caused by an empty denominator were assigned a value of zero rather than producing unstable estimates. Recall was interpreted as sensitivity for the diabetes-positive class, whereas precision represented the positive predictive value among ICU stays flagged as likely diabetes-positive.

Probability-based performance was evaluated separately from fixed-threshold classification. Probability accuracy was summarized using the Brier score, which measures the mean squared deviation between the predicted probability and the observed binary label:

$$\mathrm{Brier}={\displaystyle \frac{1}{n_V}}\sum _{i\in V}{\left(p_i^{\left(m\right)}-y_i\right)}^2$$

(21)

Discrimination was assessed with the receiver operating characteristic curve and the area under that curve (AUROC). At each threshold t, the true-positive rate and false-positive rate were defined as:

$$\begin{array}{c}TPR\left(t\right)={\displaystyle \frac{TP\left(t\right)}{TP\left(t\right)+FN\left(t\right)}}\\ FPR\left(t\right)={\displaystyle \frac{FP\left(t\right)}{FP\left(t\right)+TN\left(t\right)}}\end{array}$$

(22)

The AUROC was therefore interpreted as a threshold-independent ranking measure. Equivalently, for n1 positive cases and n0 negative cases in the validation fold, AUROC estimates the probability that a randomly selected diabetes-positive ICU stay receives a higher predicted probability than a randomly selected diabetes-negative ICU stay:

$$\begin{array}{c}\mathrm{AUROC}={\displaystyle \frac{1}{n_1{n}_0}}{\displaystyle \sum _{i:y_i=1}}{\displaystyle \sum _{j:y_j=0}}{S}_{ij}\\ S_{ij}=\mathbb{I}\left(p_i>p_j\right)+0.5\hspace{0.17em}\mathbb{I}\left(p_i=p_j\right)\end{array}$$

(23)

Because the positive class represented a minority of the cohort, receiver operating characteristic analysis was supplemented with precision-recall analysis. The prevalence of the positive class in a validation fold provides the no-skill precision baseline:

$${\pi }_V={\displaystyle \frac{1}{n_V}}\sum _{i\in V}{y}_i$$

(24)

Precision-recall curves were generated by sweeping the classification threshold over the predicted probabilities. Average precision (AP) summarized the area under the precision-recall curve using the stepwise interpolation implemented by scikit-learn:

$$\mathrm{AP}={\displaystyle \sum _{k=1}^K}\left(R_k-R_{k-1}\right)P_k$$

(25)

where $P_k$ and $R_k$ are the precision and recall values at the $k$-th threshold operating point. To support clinically interpretable threshold selection, we also examined operating points that preserved high sensitivity. For a prespecified recall target $r=0.80$, the selected threshold was defined as the largest threshold that still achieved at least the target recall:

$$t_r=\mathrm{m}\mathrm{a}\mathrm{x}\{t:\mathrm{Recall}\left(t\right)\ge r\}, \mathrm{with} r=0.80$$

(26)

The primary evaluation was conducted under the leakage-mitigated original feature scenario using the 80/20 stratified random validation split. For each trained model m in the set of evaluated classifiers, the analysis pipeline stored the predicted class labels, predicted probabilities, and metric vector:

$$q_{\left(s,m\right)}={\left[\mathrm{Accuracy},\mathrm{Precision},\mathrm{Recall},\mathrm{F}1,\mathrm{AUROC},\mathrm{Brier}\right]}_{\left(s,m\right)}$$

(27)

where s indexes the feature scenario and m indexes the model. This notation reflects the implementation in which the same train_eval_predict routine was applied to CatBoost, random forest, tuned XGBoost, tuned LightGBM, histogram-based gradient boosting, and the soft-voting ensemble. The leakage-mitigated original scenario served as the primary benchmark because it removed variables most likely to encode label-adjacent information while retaining clinically interpretable admission and day-1 predictors.

To quantify robustness to potential leakage and feature-set dependence, the full modeling framework was repeated across the four prespecified feature scenarios: full feature set, leakage-mitigated original, exclude all APACHE, and strict reduced model. For a given metric component k, the scenario-specific change relative to the primary leakage-mitigated benchmark was defined as:

$${\Delta }_{\left(s,m\right)}^{\left(k\right)}=q_{\left(s,m\right)}^{\left(k\right)}-q_{\left(\mathrm{primary},m\right)}^{\left(k\right)}$$

(28)

where $q_{\left(\mathrm{primary},m\right)}^{\left(k\right)}$ is the value of metric k for model m under the leakage-mitigated original scenario. This ablation framework allowed the analysis to determine whether performance was driven primarily by APACHE-derived variables, site-related structure, glucose-like predictors, or more general clinical and physiologic information.

In addition to the row-level random split, a grouped site-stratified validation analysis was conducted to estimate transportability to unseen hospitals. The grouped experiment reconstructed the processed adult labeled cohort under the same leakage-mitigated feature scenario and then applied GroupShuffleSplit with one split, test_size = 0.20, random_state = 40, and hospital_id as the grouping variable. Let $h_i$ denote the hospital identifier for ICU stay $i$, and let $G_{\mathrm{train}}$ and $G_{\mathrm{val}}$ denote the selected training and validation hospital sets. The grouped split enforced hospital-level separation:

$$\begin{array}{c}{G}_{\mathrm{train}}\cap G_{\mathrm{val}}=⌀.\hfill \\ D_{\mathrm{train}}^{\left(\mathrm{group}\right)}=\{i:h_i\in G_{\mathrm{train}}\}\hfill \\ D_{\mathrm{val}}^{\left(\mathrm{group}\right)}=\left\{i:h_i\in G_{\mathrm{val}}\right\}\hfill \end{array}$$

(29)

Thus, no hospital contributed records to both the grouped training and grouped validation partitions. The grouped analysis used the tuned LightGBM model because it was the sensitivity-oriented model emphasized for minority-class detection and because it provided the basis for subsequent calibration and SHAP-based interpretability analyses. To compare random-split and grouped-site behavior, the grouped validation metric change was summarized as:

$${\Delta }_{\mathrm{group}}^{\left(k\right)}=q_{\left(\mathrm{group},\mathrm{LGBM}\right)}^{\left(k\right)}-q_{\left(\mathrm{random},\mathrm{LGBM}\right)}^{\left(k\right)}$$

(30)

A decrease in AUROC, precision, or F1-score, or an increase in Brier score after site separation, would indicate reduced transportability to unseen hospitals, whereas stable recall would indicate that the model continued to identify diabetes-positive ICU stays despite a stricter validation design.

Together, as shown in

Table 4. Evaluation, ablation, and grouped-validation workflow.

Evaluation Component	Code-Level Implementation	Purpose in Analysis
Primary feature scenario	leakage_mitigated_original with 80/20 stratified split	Main benchmark after removing obvious leakage-prone fields
Model set	CatBoost, random forest, tuned XGBoost, tuned LightGBM, HGB, and soft-voting ensemble	Compare baseline, boosted, and ensemble classifiers under identical feature processing
Fixed-threshold metrics	Accuracy, precision, recall, and F1-score from predicted labels	Summarize discrete classification performance at the default operating point
Probability metrics	AUROC and Brier score from predicted probabilities	Measure ranking discrimination and probability accuracy
Precision-recall analysis	precision_recall_curve and average_precision_score	Characterize minority-class behavior under class imbalance
Target-recall threshold	Largest threshold satisfying Recall(t) ≥ 0.80	Identify a high-sensitivity operating point for screening-oriented use
Ablation analysis	Repeat full model set across four prespecified feature scenarios	Assess dependence on leakage-prone, APACHE-related, site, and glucose-like predictors
Grouped validation	GroupShuffleSplit by hospital_id with test_size = 0.20 and random_state = 40	Estimate generalizability to hospitals not observed during training

, the random split, scenario ablations, precision-recall analysis, and grouped hospital validation provided complementary evidence about discrimination, probability reliability, class-imbalance behavior, robustness to leakage-prone predictors, and site-level generalizability.

2.8. Calibration Assessment and SHAP-Based Interpretability

We implemented discrimination-based evaluation with probability calibration assessment and SHAP-based interpretability. Calibration was examined for the tuned LightGBM model under both the primary random stratified validation split and the stricter grouped site validation split. This distinction was important because AUROC and average precision evaluate ranking behavior, whereas calibration evaluates whether predicted probabilities are numerically reliable as risk estimates.

Let s ∈ {random, group} index the validation strategy, and let $V_s$ denote the corresponding validation set. For ICU stay $i$ in $V_s$, the tuned LightGBM model produced an estimated probability of documented diabetes status:

$$p_i^{\left(s\right)}=\mathrm{P}\mathrm{r}\left(y_i=1\mid x_i^{\star },M_{\mathrm{LGBM}}^{\left(s\right)}\right), i\in V_s$$

(31)

A perfectly calibrated probabilistic classifier would satisfy the condition that, among patients assigned predicted risk $p$, the observed event frequency is also $p$:

$$\mathbb{E}\left[y_i\mid p_i^{\left(s\right)}=p\right]=p, 0\le p\le 1$$

(32)

Calibration curves were generated using the scikit-learn calibration_curve function using 10 equal-width bins and strategy = ‘uniform’. Thus, the interval [0, 1] was divided into 10 equal-width probability bins. For bin $b$, the validation records assigned to that bin were defined as:

$$B_b^{\left(s\right)}=\left\{\begin{array}{ll}\left\{i\in V_s: {\displaystyle \frac{b-1}{10}}\le p_i^{\left(s\right)}<{\displaystyle \frac{b}{10}}\right\},& b=1,\dots ,9,\\ \left\{i\in V_s: 0.90\le p_i^{\left(s\right)}\le 1\right\},& b=10.\end{array}\right.$$

(33)

For each nonempty bin, the analysis pipeline computed the mean predicted probability and the observed fraction positive:

$$\begin{array}{c}{\bar{p}}_b^{\left(s\right)}={\displaystyle \frac{1}{\left|B_b^{\left(s\right)}\right|}}{\displaystyle \sum _{i\in B_b^{\left(s\right)}}}{p}_i^{\left(s\right)}\\ {\bar{o}}_b^{\left(s\right)}={\displaystyle \frac{1}{\left|B_b^{\left(s\right)}\right|}}{\displaystyle \sum _{i\in B_b^{\left(s\right)}}}{y}_i\end{array}$$

(34)

The plotted calibration curve was therefore the set of bin-level pairs displayed against the diagonal perfect-calibration reference line:

$$\begin{array}{c}{C}_s=\{\left({\bar{p}}_b^{\left(s\right)},{\bar{o}}_b^{\left(s\right)}\right):\left|B_b^{\left(s\right)}\right|>0\}\\ \mathrm{Perfect} \mathrm{calibration}: {\bar{o}}_b^{\left(s\right)}={\bar{p}}_b^{\left(s\right)}\end{array}$$

(35)

Points below the diagonal indicate probability overestimation, whereas points above the diagonal indicate probability underestimation. The Brier score defined in Equation (21) was retained as the numerical summary of probability accuracy for both validation settings. No post hoc recalibration model, such as Platt scaling or isotonic regression, was fitted; the purpose of this step was to diagnose probability reliability rather than to recalibrate the model.

Interpretability was assessed using SHAP rather than relying only on model-native feature-importance rankings (SHapley Additive exPlanations) [33]. SHAP analysis was anchored to the grouped-site LightGBM model because that model was trained and evaluated under hospital-level separation and therefore provided the more conservative explanation target. For computational efficiency, SHAP values were computed on the grouped validation set, with random subsampling applied only when the grouped validation set exceeded 3000 records:

$$S_{\mathrm{SHAP}}=\left\{\begin{array}{ll}{V}_{\mathrm{group}},& \left|V_{\mathrm{group}}\right|\le 3000,\\ \mathrm{a} 3000\text{-}\mathrm{record} \mathrm{sample} \mathrm{from} V_{\mathrm{group}} \mathrm{without} \mathrm{replacement},& \mathrm{otherwise}.\end{array}\right.$$

(36)

For transformed predictor vector $x_i^{\star }$, the explanation takes the additive form:

$$F_G\left(x_i^{\star }\right)={\varphi }_0+{\displaystyle \sum _{j=1}^p}{\varphi }_{ij}$$

(37)

where $F_G\left(x_i^{\star }\right)$ is the grouped-site LightGBM output on the TreeExplainer model-output scale, ${\varphi }_0$ is the expected baseline output, $p$ is the number of final predictors, and ${\varphi }_{ij}$ is the contribution of predictor $j$ for ICU stay $i$. Conceptually, each SHAP value is a weighted average of the marginal contribution of feature j across possible feature subsets:

$$\begin{array}{c}{\varphi }_{ij}={\displaystyle \sum _{S\subseteq F_{-j}}}w\left(S\right)\left[f_G\left(x_{i,S+j}^{\star }\right)-f_G\left(x_{i,S}^{\star }\right)\right]\\ w\left(S\right)={\displaystyle \frac{\left|S\right|!\left(p-\left|S\right|-1\right)!}{p!}}\end{array}$$

(38)

Global feature influence was summarized by the mean absolute SHAP value, which measures the average magnitude of a predictor’s contribution across the SHAP explanation sample:

$$I_j={\displaystyle \frac{1}{\left|S_{\mathrm{SHAP}}\right|}}\sum _{i\in S_{\mathrm{SHAP}}}\left|{\varphi }_{ij}\right|$$

(39)

The global mean absolute SHAP bar plot ranked predictors by $I_j$, while the SHAP beeswarm summary plot displayed the distribution and direction of feature contributions across patients. Higher or lower feature values could therefore be interpreted in terms of whether they pushed the grouped-site LightGBM output toward or away from the diabetes-positive class.

To examine nonlinear predictor effects, dependence plots were generated for the most influential predictors. For a selected feature $j$, the dependence plot consisted of the observed transformed feature value paired with its SHAP contribution across the explanation sample:

$$D_j=\{\left(x_{ij}^{\star },{\varphi }_{ij}\right):i\in S_{\mathrm{SHAP}}\}$$

(40)

The top SHAP features were identified by sorting Equation (39), and dependence plots were generated for the six highest-ranked predictors. These plots were used to assess whether dominant predictors contributed monotonically, showed threshold-like changes, or saturated at high or low values.

Finally, local SHAP waterfall plots were generated for representative cases selected from the grouped validation confusion-matrix categories. Using the grouped LightGBM predicted labels, the case sets were defined as:

$$\begin{array}{c}TP=\{i:y_i=1,{\widehat{y}}_i=1\}, FP=\{i:y_i=0,{\widehat{y}}_i=1\}\\ TN=\{i:y_i=0,{\widehat{y}}_i=0\}, FN=\{i:y_i=1,{\widehat{y}}_i=0\}\end{array}$$

(41)

We selected the first available example from each nonempty set and plotted the cumulative SHAP decomposition for that patient. If features are ordered by decreasing absolute local contribution, the waterfall trajectory after the first $k$ displayed features is:

$$W_i\left(k\right)={\varphi }_0+{\displaystyle \sum _{r=1}^k}{\varphi }_{i,j_r}, \left|{\varphi }_{i,j_1}\right|\ge \left|{\varphi }_{i,j_2}\right|\ge \cdots$$

(42)

This local decomposition showed how individual predictors moved the grouped-site LightGBM output away from the baseline value and toward either the diabetes-positive or diabetes-negative class. Together, the calibration diagnostics and SHAP analyses addressed two complementary concerns: whether the model probabilities were reliable enough to interpret, and whether the model’s behavior could be explained at both global and patient-specific levels using the leakage-mitigated, site-separated modeling workflow, as shown in

Table 5. Calibration and SHAP-based interpretability workflow.

Component	Code-Level Implementation	Purpose in Analysis
Calibration target	Tuned LightGBM evaluated under both random stratified validation and grouped site validation	Assess probability reliability beyond threshold-independent discrimination
Calibration curve	calibration_curve with n_bins = 10 and strategy = ‘uniform’	Compare observed diabetes frequency with mean predicted probability across risk bins
Calibration reference	Diagonal line representing observed fraction positive = mean predicted probability	Identify systematic probability overestimation or underestimation
Probability score	Brier score reported for random and grouped LightGBM probability outputs	Summarize probability accuracy as a lower-is-better proper scoring rule
Interpretability model	Grouped-site LightGBM model	Explain the model evaluated under the stricter hospital-separated design
SHAP explainer	shap.TreeExplainer applied to the grouped-site LightGBM model	Decompose predictions into additive feature-level contributions
SHAP sample	Grouped-validation records, sampled to at most 3000 rows with RANDOM_STATE if needed	Limit computational burden while keeping the explanation sample reproducible
Global outputs	Mean absolute SHAP bar plot and SHAP beeswarm summary plot	Rank influential predictors and show directional contribution patterns
Dependence plots	Top six predictors ranked by mean absolute SHAP value	Assess nonlinear, threshold-like, or saturated predictor effects
Local outputs	Waterfall plots for representative TP, FP, TN, and FN cases	Explain individual correct and incorrect predictions

.

2.9. Reporting and Reproducibility

Methods and results reporting were structured to align with core elements of TRIPOD, including explicit definitions of outcome and predictors, transparent preprocessing, clear separation of model development from evaluation, and complete reporting of performance metrics. For full reproducibility, release of the complete preprocessing pipeline (including feature lists and imputation rules), the final tuned hyperparameters, and software/package versions is provided.

3. Results

3.1. Cohort Composition and Validation Design

The analytic cohort was derived from the labeled WiDS 2021 training file, which contained 130,157 ICU stays. This labeled file should be distinguished from the model-training partition generated after splitting. The separate unlabeled challenge file contained 10,234 ICU stays and was not used to estimate validation performance, select thresholds, or fit preprocessing parameters. After applying the adult eligibility restriction (age ≥ 16 years), 125,139 labeled ICU stays remained for model development and evaluation. Thus, 5018 records were excluded during cohort definition, and 96.1% of the labeled cohort was retained in the adult analytic cohort.

The primary internal validation design used the 80/20 stratified random partition described in the Methods. This split allocated 100,111 adult records to training and 25,028 records to validation, corresponding almost exactly to 80.0% and 20.0% of the adult analytic cohort. Because the split was stratified by diabetes_mellitus, the validation partition preserved the minority-class structure of the outcome and provided the principal row-level benchmark for comparing CatBoost, random forest, tuned XGBoost, tuned LightGBM, histogram-based gradient boosting, and the soft-voting ensemble.

A second validation design was used to evaluate transportability across clinical sites rather than performance on randomly sampled patient rows alone. In this grouped site-stratified experiment, hospital_id was used as the grouping variable, yielding 95,233 records for model training and 29,906 records for validation. The resulting row allocation was 76.1% training and 23.9% validation, rather than exactly 80/20, because the split operated at the hospital level and therefore had to assign all records from a given hospital to the same fold.

Hospital-level separation was complete. A total of 163 hospitals were assigned to the grouped training partition, and 41 hospitals were assigned to the grouped validation partition, for a total of 204 hospitals with no overlap between folds. Thus, approximately 79.9% of hospitals were used for grouped training and 20.1% were held out for grouped validation. This design intentionally created a stricter generalizability assessment than the row-level random split by reducing the possibility that performance estimates were inflated by hospital-specific coding patterns, admission practices, laboratory ordering behavior, or site-level population structure appearing in both training and validation data. Consequently, the random split was interpreted as the primary internal benchmark, whereas the grouped split was interpreted as a more conservative assessment of performance at previously unseen hospitals, as shown in

Table 6. Cohort retention and validation-design summary.

Component	Count	Percentage/Interpretation
Labeled WiDS 2021 file	130,157 ICU stays	Initial labeled source cohort used for model development and validation
Unlabeled challenge file	10,234 ICU stays	Held out from all validation-performance estimation and threshold selection
Excluded by adult restriction	5018 ICU stays	3.9% of labeled cohort excluded because age < 16 years or did not meet the adult eligibility rule
Adult analytic cohort	125,139 ICU stays	96.1% of labeled cohort retained after age restriction
Primary random split	100,111 training/25,028 validation records	80.0%/20.0% row-level stratified split of the adult analytic cohort
Grouped site split	95,233 training/29,906 validation records	76.1%/23.9% row allocation; proportions differ from 80/20 because hospitals, not individual rows, were split
Grouped hospitals	163 training/41 validation hospitals	79.9%/20.1% hospital allocation with no hospital overlap between training and validation folds

Note. Percentages are rounded to one decimal place.

.

3.2. Primary Model Performance in the Leakage-Mitigated Random Split

Table 7. Primary model metrics in the leakage-mitigated feature scenario.

Model	Accuracy	Precision	Recall	F1 Score	AUC	Brier Score
Voting Ensemble	0.8151	0.5671	0.6690	0.6138	0.8539	0.1300
LightGBM (Tuned)	0.7815	0.5017	0.7677	0.6069	0.8537	0.1508
XGBoost (Tuned)	0.8288	0.6557	0.4642	0.5436	0.8505	0.1197
HistGradientBoosting	0.8251	0.6445	0.4543	0.5330	0.8458	0.1214
CatBoost	0.8227	0.6317	0.4629	0.5343	0.8403	0.1236
Random Forest	0.8154	0.6645	0.3225	0.4342	0.8252	0.1286

Note. Brier score is a lower-is-better measure of probability accuracy. All other metrics are higher-is-better.

summarizes the primary model-comparison results in the leakage-mitigated scenario. Two distinct operating profiles were apparent. The tuned LightGBM model achieved the highest recall (0.7677) and nearly the highest AUROC (0.8537), indicating that it was the most aggressive minority-class detector. However, this sensitivity came at the expense of precision (0.5017), overall accuracy (0.7815), and the least favorable Brier score (0.1508), indicating poorer probability reliability. In contrast, the voting ensemble preserved essentially the same AUROC (0.8539), improved precision to 0.5671, and achieved the highest F1-score (0.6138) among the evaluated models. XGBoost (tuned) showed the lowest Brier score (0.1197), suggesting the best raw probability fit of the individual models, but its recall remained moderate at 0.4642. Random forest produced the highest precision (0.6645) but the lowest recall (0.3225), illustrating the cost of using a comparatively conservative classifier in an imbalanced screening task.

Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 provide the model-specific confusion matrices and top-15 feature-importance summaries for the six primary models. Taken together, these panels show that performance differences were driven less by fundamentally different predictor families than by how each algorithm traded sensitivity against false-positive burden. Across all models, day-1 glucose maxima dominated the feature rankings, and day-1 glucose minima, BMI, age, creatinine, pre-ICU length of stay, hemoglobin, and blood urea nitrogen repeatedly appeared among the most influential variables. The consistency of these rankings across algorithms supports the clinical plausibility of the pipeline and shows that the leakage mitigation step succeeded in removing hospital_id and icu_id from the dominant predictor set.

In Figure 1A, CatBoost correctly classified 18,046 non-diabetes stays and 2545 diabetes stays, while producing 1484 false positives and 2953 false negatives. This pattern is consistent with a balanced but not highly sensitive classifier. Figure 1B shows that d1_glucose_max clearly dominated the CatBoost ranking, followed by d1_glucose_min, age, BMI, and d1_creatinine_max, indicating that CatBoost relied primarily on metabolic and renal markers rather than on any residual site proxy.

Figure 2 illustrates the conservative operating profile of the random forest model. It produced the fewest false positives among the primary models (895) but missed a large number of diabetes cases (3725 false negatives), which explains its very low recall. Its feature ranking remained consistent with the other models, with d1_glucose_max by far the strongest signal and d1_glucose_min, age, BMI, pre_icu_los_days, and d1_creatinine_max forming the next tier of contributors.

The tuned XGBoost model in Figure 3 delivered a stronger balance between specificity and sensitivity than CatBoost or random forest. It correctly identified 2552 diabetes stays while keeping false positives to 1340, which explains its comparatively strong accuracy and the lowest Brier score among the individual-tuned models. Its importance profile again centered on d1_glucose_max as APACHE-related variables such as ventilated_apache, arf_apache, gcs_unable_apache, and intubated_apache, along with BMI and age.

Figure 4 highlights why LightGBM emerged as the most sensitivity-oriented primary model. It correctly identified 4221 diabetes stays and reduced false negatives to 1277, but this gain was accompanied by 4192 false positives. Panel B shows that the same core predictors remained dominant, with d1_glucose_max and d1_glucose_min well above the other variables, followed by BMI, d1_creatinine_max, pre_icu_los_days, age, d1_wbc_max, and several vital-sign summaries.

As shown in Figure 5, the histogram-based gradient boosting model behaved similarly to CatBoost and tuned XGBoost. It delivered moderate sensitivity, with 2498 true positives and 3000 false negatives, while maintaining false positives at 1378. Its feature ranking was again led by d1_glucose_max, followed by age, BMI, d1_creatinine_max, and selected laboratory and respiratory features, reinforcing the stability of the signal structure across boosting methods.

Figure 6 demonstrates why the voting ensemble provided the most balanced overall operating point. The ensemble correctly identified 3678 diabetes stays and misclassified 2808 non-diabetes stays as positive, thereby improving precision relative to tuned LightGBM while retaining substantially higher recall than the more conservative baseline models. Its aggregated importance ranking remained anchored by d1_glucose_max, with d1_glucose_min, BMI, d1_creatinine_max, age, and pre_icu_los_days forming the next strongest contributors.

3.3. Ablation Analyses

Table 8. Ablation metrics across the four feature scenarios.

Scenario	Model	Accuracy	Precision	Recall	F1 Score	AUC	Brier
Full feature set	CatBoost	0.8250	0.6371	0.4720	0.5423	0.8435	0.1227
Full feature set	Random Forest	0.8178	0.6784	0.3247	0.4392	0.8270	0.1280
Full feature set	XGBoost (Tuned)	0.8300	0.6605	0.4653	0.5459	0.8525	0.1189
Full feature set	LightGBM (Tuned)	0.7842	0.5058	0.7672	0.6097	0.8565	0.1491
Full feature set	HistGradientBoosting	0.8270	0.6496	0.4616	0.5397	0.8486	0.1203
Full feature set	Voting Ensemble	0.8159	0.5685	0.6721	0.6159	0.8566	0.1286
Leakage-mitigated original	CatBoost	0.8227	0.6317	0.4629	0.5343	0.8403	0.1236
Leakage-mitigated original	Random Forest	0.8154	0.6645	0.3225	0.4342	0.8252	0.1286
Leakage-mitigated original	XGBoost (Tuned)	0.8288	0.6557	0.4642	0.5436	0.8505	0.1197
Leakage-mitigated original	LightGBM (Tuned)	0.7815	0.5017	0.7677	0.6069	0.8537	0.1508
Leakage-mitigated original	HistGradientBoosting	0.8251	0.6445	0.4543	0.5330	0.8458	0.1214
Leakage-mitigated original	Voting Ensemble	0.8151	0.5671	0.6690	0.6138	0.8539	0.1300
Exclude all APACHE	CatBoost	0.8228	0.6335	0.4591	0.5324	0.8377	0.1247
Exclude all APACHE	Random Forest	0.8163	0.6616	0.3354	0.4451	0.8239	0.1288
Exclude all APACHE	XGBoost (Tuned)	0.8256	0.6447	0.4587	0.5360	0.8478	0.1210
Exclude all APACHE	LightGBM (Tuned)	0.7788	0.4978	0.7634	0.6026	0.8508	0.1529
Exclude all APACHE	HistGradientBoosting	0.8232	0.6377	0.4523	0.5293	0.8438	0.1224
Exclude all APACHE	Voting Ensemble	0.8111	0.5587	0.6661	0.6077	0.8514	0.1314
Strict reduced model	CatBoost	0.7830	0.5179	0.1737	0.2601	0.7252	0.1533
Strict reduced model	Random Forest	0.7831	0.5748	0.0489	0.0902	0.7036	0.1558
Strict reduced model	XGBoost (Tuned)	0.7911	0.5999	0.1470	0.2361	0.7402	0.1491
Strict reduced model	LightGBM (Tuned)	0.6868	0.3786	0.6641	0.4823	0.7432	0.1986
Strict reduced model	HistGradientBoosting	0.7896	0.5998	0.1268	0.2093	0.7362	0.1498
Strict reduced model	Voting Ensemble	0.7634	0.4597	0.4394	0.4493	0.7448	0.1656

Note. Brier score is lower-is-better. The strict reduced model excluded glucose-like predictors in addition to APACHE-related variables and site descriptors.

reports the full ablation results across all four feature scenarios. The most important observation is that performance fell only modestly when leakage-prone or APACHE-related variables were removed, but deteriorated sharply under the strict reduced model that also excluded glucose-like predictors. For LightGBM, AUROC declined only from 0.8565 in the full-feature scenario to 0.8537 in the leakage-mitigated scenario and to 0.8508 after removing all APACHE variables. The voting ensemble showed a similar pattern, decreasing from 0.8566 to 0.8539 and then to 0.8514. These changes were small enough to support the claim that the models retained meaningful discriminative ability after reasonable leakage mitigation. In contrast, the strict reduced model caused a much larger collapse in performance. LightGBM AUROC fell to 0.7432, and its Brier score worsened to 0.1986; the voting ensemble dropped to 0.7448 AUROC and 0.1656 Brier. Thus, the ablation analysis suggests that the model was not primarily dependent on explicit site or APACHE identifiers, but it was strongly dependent on glucose-related admission information, which is clinically plausible given the task of identifying documented diabetes.

Figure 7 complements Table 8 by making the ablation trends visually explicit. The full-feature, leakage-mitigated, and exclude-all-APACHE scenarios form a tight cluster for most models, whereas the strict reduced model produces an obvious downward shift in recall-weighted performance and AUC. Importantly, Brier score is the only metric in Figure 7 for which lower values are better; the strict reduced bars rise because probability accuracy worsened materially once glucose-like predictors were removed. The figure therefore supports a nuanced conclusion: modest variable exclusions designed to reduce leakage do not materially destabilize performance, but removing the dominant glycemic signal substantially changes the task and sharply reduces discriminative utility.

3.4. Grouped Site-Stratified Validation

Table 9. Tuned LightGBM performance under random stratified versus grouped site-stratified validation.

Validation Strategy	Accuracy	Precision	Recall	F1 Score	AUC	Brier Score
Random Stratified Split	0.7815	0.5017	0.7677	0.6069	0.8537	0.1508
Grouped Site Split	0.7727	0.4546	0.7684	0.5712	0.8443	0.1596

Note. The grouped split used hospital_id so that no hospital appeared in both training and validation folds.

compares the tuned LightGBM model under the original random stratified split and the stricter grouped site split. The grouped split reduced accuracy from 0.7815 to 0.7727, precision from 0.5017 to 0.4546, F1-score from 0.6069 to 0.5712, AUROC from 0.8537 to 0.8443, and Brier score from 0.1508 to 0.1596. Recall, however, remained essentially unchanged (0.7677 versus 0.7684). This is an important pattern. It suggests that some of the precision and ranking advantage seen in the random split depended on within-hospital regularities that do not fully transfer to unseen sites, but the model’s ability to find likely diabetes cases remained stable even after hospital-level separation.

Figure 8A shows the corresponding grouped-site confusion matrix: 18,581 true negatives, 5432 false positives, 1365 false negatives, and 4528 true positives. Because the grouped validation fold was larger than the random validation fold, the raw counts are not directly comparable, but the error pattern is informative. The grouped-site model continued to identify many diabetes cases while producing a relatively larger false-positive burden, which is consistent with the drop in precision documented in Table 9. Figure 8B shows that the grouped-site feature ranking remained clinically coherent. The leading variables d1_glucose_max, d1_glucose_min, BMI, d1_creatinine_max, pre_icu_los_days, and age.

3.5. Precision–Recall Behavior Under Class Imbalance

Because documented diabetes represented a minority outcome in the validation cohorts, precision–recall analysis was used to complement AUROC-based discrimination. Precision–recall curves directly summarize the trade-off between sensitivity and positive predictive value and are therefore useful for evaluating classifier behavior when the positive class is less frequent. In this analysis, the tuned LightGBM model was evaluated under both the primary random stratified validation split and the grouped hospital validation split. Horizontal reference lines were added to represent the diabetes-positive prevalence in the corresponding validation fold, which serves as the no-skill precision baseline.

The random-split LightGBM model achieved an average precision of 0.622, whereas the grouped-site LightGBM model achieved an average precision of 0.551. The decrease in average precision under grouped validation is consistent with the lower grouped-site precision reported in Table 9 and indicates that hospital-level separation increased the false-positive burden. However, the grouped-site precision–recall curve remained above the prevalence baseline across much of the recall range, showing that the model retained useful minority-class ranking ability even when evaluated on hospitals not seen during training.

The precision–recall curves also clarify the clinical implications of threshold selection. In the grouped-site validation set, a probability threshold of 0.4537 achieved recall of 0.8001 with precision of 0.4314. Thus, the model can be tuned to preserve high sensitivity for diabetes-positive ICU stays at previously unseen hospitals, but this operating point requires accepting a substantial number of false positives. This trade-off may be appropriate for a screening-oriented phenotyping workflow in which the model prompts chart review, medication reconciliation, or cohort enrichment. However, it would be less appropriate for automated diagnostic labeling or interruptive clinical alerts without additional confirmation. These findings reinforce that the model should be interpreted as a computable phenotyping aid rather than a stand-alone diagnostic tool.

In Figure 9, the blue curve shows the primary random stratified validation split, and the orange curve shows the grouped hospital validation split. Average precision was 0.622 for the random split and 0.551 for the grouped split. Horizontal reference lines indicate the diabetes-positive prevalence in the corresponding validation fold and represent the no-skill precision baseline. The grouped curve shows reduced precision relative to the random split but remains above the baseline over much of the recall range, indicating preserved minority-class ranking performance under hospital-level separation.

3.6. Calibration Assessment

Calibration analysis further evaluated whether the tuned LightGBM probabilities were numerically reliable as risk estimates rather than only useful for ranking patients. Figure 10 shows the calibration curves for the random stratified and grouped hospital validation splits. In both validation settings, the model curve fell below the diagonal reference line across much of the probability range, indicating that predicted probabilities tended to overestimate the observed frequency of documented diabetes, especially in the mid-to-high predicted-risk ranges. The random-split LightGBM model had a Brier score of 0.1508, whereas the grouped-site LightGBM model had a Brier score of 0.1596. This increase under grouped validation indicates that probability reliability worsened slightly when the model was evaluated on hospitals not observed during training. These findings are important because AUROC and average precision measure ranking behavior, but they do not establish whether the predicted probabilities are clinically reliable as absolute risk estimates. Therefore, although tuned LightGBM preserved strong recall and reasonable discrimination under both validation strategies, its probabilities would likely require post hoc recalibration and prospective monitoring before use in a decision-support workflow where the predicted probability itself is interpreted clinically.

Figure 10A shows calibration on the random stratified validation split; Figure 10B shows calibration on the grouped hospital validation split. The diagonal reference line represents perfect calibration. In both panels, the model curve falls below the diagonal over much of the probability range, indicating probability overestimation. Calibration worsened slightly under grouped-site validation, consistent with the higher grouped-site Brier score.

3.7. SHAP-Based Interpretability

Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 present SHAP-based interpretability analyses for the grouped-site LightGBM model. This choice is important because the grouped-site model provides a more conservative estimate of generalizability. The SHAP outputs allow global, nonlinear, and patient-level interpretation of the model’s behavior.

Figure 11A quantifies the overall influence of each feature on the grouped-site LightGBM output. d1_glucose_max was by far the dominant feature, with a mean absolute SHAP value substantially larger than all other predictors. The next strongest contributors were BMI, age, d1_glucose_min, d1_hemoglobin_max, and urineoutput_apache. Figure 11B adds directional information. High values of d1_glucose_max and BMI were generally associated with positive SHAP values that pushed the model toward diabetes, whereas lower values tended to push predictions downward. Age showed a more graded pattern, and lower-ranked variables clustered closer to zero, indicating more modest global influence. Importantly, site identifiers were absent from the SHAP ranking, which supports the success of the leakage-mitigation strategy.

Figure 12 shows that the effects of the dominant predictors were nonlinear rather than purely linear. In Figure 12A, the contribution of d1_glucose_max rose sharply as maximum glucose increased from approximately 100 to 250 mg/dL, then plateaued at very high values, indicating saturation rather than unbounded growth. In Figure 12B, BMI crossed from negative to positive contribution around the high-20s and then climbed steadily through the obese range, again with some flattening at the upper extreme. In Figure 12C, younger age was associated with negative SHAP values; the contribution increased through middle and older adulthood and then flattened or slightly tapered in the oldest age bands. These dependence plots show that the model captured clinically plausible thresholds rather than relying on simple monotonic linear rules.

Figure 13 reveals a more complex interaction structure in the secondary predictors. The d1_glucose_min plot in Figure 13A displayed an inverse and clearly non-monotonic pattern: lower glucose minima tended to increase the model output, whereas higher minima progressively reduced it. This behavior likely reflects interaction with d1_glucose_max and treatment response rather than a simple isolated glucose effect. Figure 13B shows that d1_hemoglobin_max contributed positively at lower-to-mid values and then became increasingly negative at higher values, suggesting that this variable acted as a contextual severity marker rather than a direct diabetes proxy. Figure 13C indicates that urineoutput_apache had a more diffuse contribution with relatively small SHAP magnitudes overall; it functioned as a secondary contextual feature rather than a dominant driver.

Figure 14 and Figure 15 show four local SHAP waterfall explanations generated from representative cases drawn from the true-positive, false-positive, true-negative, and false-negative sets in the analysis pipeline. The most striking recurring pattern is the dominant role of d1_glucose_max. In the positively scored cases, high d1_glucose_max created the single largest upward shift in model output, often overwhelming smaller downward contributions from hemoglobin, age, or glucose-minimum features. In the negatively scored cases, low or less extreme d1_glucose_max exerted the largest downward pull, often accompanied by lower BMI or other context variables that further reduced the diabetes score. These local explanations are valuable because they show that the model’s final decision for an individual patient was rarely driven by one feature in isolation; instead, a strong glycemic signal was modulated by anthropometrics, age, hematologic markers, renal markers, and pre-ICU context. These local explanations provide patient-level transparency beyond global feature-importance rankings alone.

Overall, the results lead to three main conclusions. First, the ensemble provided the most balanced overall operating profile, whereas tuned LightGBM remained the most sensitivity-oriented model. Second, reasonable leakage mitigation and grouped site validation reduced performance only modestly, supporting the robustness of the pipeline. Third, the calibration and SHAP analyses show that the final model is interpretable and still useful as a screening-oriented phenotyping tool, but that its probabilities remain imperfectly calibrated and would require further refinement before deployment.

4. Discussion

In this public, multi-center ICU cohort derived from the WiDS Datathon 2021/GOSSIS release, leakage-aware tree-based machine-learning models showed strong performance for admission-time identification of documented diabetes status. The analysis was intentionally designed as a benchmarking and phenotyping study rather than a stand-alone diagnostic model. The workflow used split-aware preprocessing, avoided fitting preprocessing steps on validation or challenge data, excluded explicit site identifiers from the model matrix, evaluated prespecified feature-ablation scenarios, used model-intrinsic class weighting rather than synthetic oversampling, and assessed performance under both random and grouped hospital validation. This design strengthens the credibility of the results because it reduces the risk that apparent discrimination was driven by direct leakage, site memorization, or preprocessing choices informed by held-out data.

The primary leakage-mitigated random-split results showed two clinically meaningful operating profiles. The soft-voting ensemble achieved the most balanced overall performance, with an AUROC of 0.8539, a precision of 0.5671, a recall of 0.6690, and an F1-score of 0.6138. Tuned LightGBM was the most sensitivity-oriented individual model, achieving recall of 0.7677 and AUROC of 0.8537, although this came with a lower precision of 0.5017 and a less favorable Brier score of 0.1508. These findings indicate that model choice should depend on intended use. LightGBM may be preferable when the goal is high-sensitivity screening and missed diabetes-positive cases are costly, whereas the ensemble may be preferable when the burden of false positives, chart review, or alert fatigue is a major concern.

The ablation analyses clarify the source of model performance and support a cautious interpretation. Removing leakage-prone and APACHE-related variables caused only modest reductions in discrimination. For LightGBM, AUROC declined from 0.8565 in the full-feature scenario to 0.8537 in the leakage-mitigated scenario and to 0.8508 after excluding all APACHE variables. The voting ensemble showed a similar pattern, with AUROC decreasing from 0.8566 to 0.8539 and then to 0.8514. These small changes suggest that the models were not primarily dependent on explicit APACHE-derived variables, site identifiers, or obvious leakage-prone predictors. In contrast, the strict reduced model that also removed glucose-like predictors produced a much larger decline in performance: LightGBM AUROC fell to 0.7432, and the ensemble AUROC fell to 0.7448. Thus, glucose-related admission variables remained the dominant predictive signal. This finding is clinically plausible for identifying documented diabetes status, but it is also the central interpretive limitation of the study. Early glycemic values in critically ill patients can reflect chronic diabetes, but they can also capture acute stress physiology, medication exposure, insulin or dextrose treatment, nutritional support, illness severity, and monitoring intensity. Therefore, the model should not be interpreted as learning chronic diabetes status alone.

Grouped hospital validation provided a stricter assessment of generalizability to unseen clinical sites. When tuned LightGBM was evaluated under hospital-level separation, AUROC decreased from 0.8537 in the random split to 0.8443, precision decreased from 0.5017 to 0.4546, and Brier score worsened from 0.1508 to 0.1596. Recall, however, remained essentially stable at 0.7684. This pattern suggests that some precision and ranking advantage in the random split may reflect within-hospital regularities that do not fully transfer to unseen hospitals. At the same time, the model retained useful case-finding ability after site separation. For multi-center ICU phenotyping, this distinction is important: row-level random validation estimates internal performance, whereas grouped hospital validation provides a more conservative estimate of transportability across sites.

The precision–recall analysis further emphasizes that the system is best interpreted as a screening-oriented phenotyping aid. Tuned LightGBM achieved an average precision of 0.622 under random validation and 0.551 under grouped hospital validation, showing that hospital-level separation increased the false-positive burden. At a grouped-site probability threshold of 0.4537, the model achieved recall of 0.8001 but precision of only 0.4314. This high-sensitivity operating point may be acceptable when the model is used to prompt chart review, support medication reconciliation, enrich research cohorts, or identify patients needing confirmation of diabetes history. However, it would be inappropriate for automated diagnostic labeling, treatment decisions, or interruptive clinical alerts without additional clinical confirmation. The model should therefore be framed as a computable phenotyping aid, not as a stand-alone diagnostic tool.

The clinical relevance of this distinction is especially important in critical care. Admission-time diabetes status can help clinicians interpret early hyperglycemia and contextualize glycemic targets, but hyperglycemia in the ICU is etiologically heterogeneous. In a patient with established diabetes, elevated glucose may reflect chronic dysglycemia or baseline metabolic disease. In a patient without known diabetes, the same glucose value may reflect stress hyperglycemia, acute inflammatory and hormonal responses, sepsis physiology, glucocorticoid or vasopressor exposure, enteral or parenteral nutrition, dextrose-containing fluids, or early treatment effects. Thus, although glucose-related variables are informative for identifying documented diabetes status, their dominance means that model predictions may partly encode acute dysglycemia and clinical management patterns rather than chronic diabetes alone. This reinforces the need for cautious use and clinical review.

Calibration results also show why discrimination alone is insufficient. Although tuned LightGBM maintained strong recall and reasonable AUROC under both random and grouped validation, calibration curves and Brier scores showed that predicted probabilities were not fully reliable as absolute risk estimates. In both validation settings, predicted probabilities tended to overestimate the observed frequency of documented diabetes, particularly in the mid-to-high probability range, and the grouped-site Brier score was worse than the random-split Brier score. These findings indicate that the model can rank patients usefully while still producing probabilities that are not deployment-ready. If predicted probabilities are used clinically, post hoc recalibration, prospective monitoring, and periodic site-specific reassessment would be necessary.

The SHAP-based interpretability analysis provides additional support for both clinical plausibility and cautious interpretation. In the grouped-site LightGBM model, day-1 maximum glucose was the dominant predictor, followed by BMI, age, day-1 minimum glucose, hemoglobin, urine output, and other physiologic or laboratory variables. These predictors are clinically coherent because glycemic extrema, anthropometrics, age, renal and hematologic markers, and pre-ICU context may all relate to documented diabetes status or to the conditions under which diabetes is recognized and recorded. Importantly, hospital_id and icu_id were absent from the dominant predictor set because they were removed from the model matrix by design, supporting the leakage-mitigation strategy. However, SHAP results also reinforce the same limitation identified by ablation analysis: the dominant glycemic signal is clinically meaningful but not specific to chronic diabetes. Early glucose values may be shaped by both baseline disease and acute ICU physiology.

The SHAP dependence and local waterfall plots further showed that model behavior was nonlinear and context-dependent. The contribution of maximum glucose increased sharply across clinically meaningful glucose ranges and then plateaued at high values. BMI and age showed graded or threshold-like effects, whereas secondary predictors such as minimum glucose, hemoglobin, and urine output showed more complex patterns that may reflect interactions with illness severity, renal function, treatment response, or monitoring intensity. Local explanations demonstrated that individual predictions were rarely determined by one variable alone; rather, strong glycemic signals were modulated by anthropometric, renal, hematologic, and clinical-context variables. This improves auditability and interpretability, but it does not eliminate the need for clinical confirmation when distinguishing chronic diabetes from stress hyperglycemia or treatment-related dysglycemia.

The use of model-intrinsic class weighting and threshold-aware evaluation was appropriate for this imbalanced clinical phenotyping task. Rather than generating synthetic minority-class records, class weighting preserved the empirical patient distribution while increasing the training penalty for misclassifying diabetes-positive cases. In this study, positive-class weighting helped LightGBM achieve high sensitivity, while the soft-voting ensemble moderated the false-positive burden by combining sensitivity-oriented and more conservative boosted-tree models. These results support class weighting and explicit threshold selection as transparent, reproducible approaches for imbalanced ICU phenotyping. Nevertheless, threshold selection should be matched to the intended workflow rather than treated as a universal operating point.

Several limitations should be considered. First, the outcome label represents documented diabetes status in the source data, not adjudicated chronic diabetes confirmed by longitudinal outpatient records, HbA1c, medication history, or manual chart review. As a result, the model may learn correlates of diabetes documentation as well as correlates of true chronic disease. Second, although grouped hospital validation is stronger than a row-level random split, it remains internal validation within the same public data release. External validation in independent ICU datasets is necessary before general clinical use. Third, the strong influence of glucose-like predictors creates a clinical ambiguity: early glucose values are highly informative for diabetes phenotyping, but they may also reflect stress hyperglycemia, treatment exposure, acute illness severity, or monitoring patterns. Fourth, calibration was diagnostic only; no recalibration model was fitted. Therefore, predicted probabilities should not be interpreted as deployment-ready clinical risk estimates without recalibration and prospective assessment.

Future work should prioritize external validation in independent ICU datasets, prospective evaluation across hospitals and time periods, recalibration when probabilities are used clinically, and threshold selection matched to specific workflow goals. Improved reference labeling is also needed. Ideally, future studies should distinguish documented chronic diabetes from previously undiagnosed diabetes, stress hyperglycemia, medication-related hyperglycemia, nutrition-associated dysglycemia, and treatment-related glucose changes using longitudinal EHR data, HbA1c, outpatient medication history, problem lists, diagnosis codes, and manual adjudication where feasible. Future extensions could also compare the present leakage-aware tree-based benchmark with deep tabular architectures, including TabNet, TabTransformer, FT-Transformer, or other Transformer-based models, under the same grouped validation, calibration, ablation, and interpretability framework.

Overall, this study provides a reproducible and clinically interpretable benchmark for ICU admission-time phenotyping of documented diabetes status in public multi-center EHR data. The findings show that leakage-aware gradient-boosted and ensemble tree models can achieve strong discrimination, that reasonable leakage mitigation and grouped-site validation do not eliminate useful case-finding performance, and that model evaluation should extend beyond AUROC to include precision–recall behavior, calibration, ablation analysis, and interpretability. At the same time, the dominance of glucose-related predictors requires a careful clinical interpretation. The proposed system should be used primarily as a screening-oriented phenotyping aid for chart review, cohort enrichment, workflow support, or research classification, not as a stand-alone diagnostic tool. Clinical deployment would require external validation, recalibration, workflow-specific thresholding, and confirmatory clinical review to distinguish chronic diabetes from acute stress physiology and treatment-related dysglycemia.

5. Conclusions

Using the public WiDS Datathon 2021 tabular release derived from the GOSSIS initiative, this study developed and evaluated leakage-aware machine-learning models for identifying documented diabetes status among adult ICU stays at or near admission. The workflow emphasized split-aware preprocessing, removal of explicit site identifiers from the model matrix, prespecified feature-scenario ablations, model-intrinsic class weighting, threshold-aware evaluation, grouped hospital validation, calibration assessment, and SHAP-based interpretability. Together, these design choices provide a reproducible benchmark for ICU admission-time diabetes phenotyping in public multi-center EHR data.

In the primary leakage-mitigated random validation split, gradient-boosted and ensemble tree models achieved strong discrimination. The soft-voting ensemble provided the most balanced operating profile, with AUROC 0.8539, precision 0.5671, recall 0.6690, and F1-score 0.6138. Tuned LightGBM was the most sensitivity-oriented individual model, achieving recall 0.7677 and AUROC 0.8537, although with lower precision and a less favorable Brier score. Grouped hospital validation showed that tuned LightGBM retained case-finding ability at unseen hospitals, with recall remaining stable at 0.7684, while AUROC decreased to 0.8443 and precision declined to 0.4546. Precision–recall analysis further confirmed the intended-use trade-off: a grouped-site threshold of 0.4537 preserved high recall of 0.8001 but reduced precision to 0.4314. These findings indicate that LightGBM is most appropriate for high-sensitivity screening, whereas the ensemble may be preferable when false-positive burden is a major concern.

Ablation and interpretability analyses clarify the source and limitations of model performance. Removing leakage-prone and APACHE-related variables caused only modest performance reductions, supporting the robustness of the leakage-aware pipeline. In contrast, the strict reduced model that also excluded glucose-like predictors produced a marked decline in discrimination, confirming that glucose-related admission variables remained the dominant predictive signal. SHAP analysis was consistent with this finding, identifying day-1 maximum glucose, BMI, age, day-1 minimum glucose, hemoglobin, urine output, and related physiologic or laboratory variables as major contributors. Although these predictors are clinically plausible for identifying documented diabetes status, early glycemic measurements in critically ill patients may also partly capture acute stress physiology, treatment-related effects, monitoring intensity, nutritional support, or other forms of acute dysglycemia rather than chronic diabetes status alone.

Calibration results further emphasize that the model should not be interpreted as a deployment-ready diagnostic instrument. Calibration curves and Brier scores showed that predicted probabilities were not fully reliable as clinical risk estimates without further recalibration. Therefore, the proposed system should be interpreted primarily as a screening-oriented phenotyping aid for chart review, medication reconciliation, cohort enrichment, or workflow support, not as a stand-alone diagnostic tool.

Future work should prioritize external validation in independent ICU datasets, prospective evaluation across hospitals and time periods, workflow-specific threshold selection, post hoc recalibration when probabilities are used clinically, and improved reference labeling that better distinguishes chronic diabetes from stress hyperglycemia and treatment-related dysglycemia. Clear documentation of intended use remains essential for safe and reproducible application in critical care research and decision-support workflows.